hub v0.143.0 (code): the kernel lane — the day-before household mail, the night instruction, the operator's kernel set (R-836, decision 172)
gates / gates (push) Successful in 2m47s
gates / gates (push) Successful in 2m47s
KernelDue / KernelNotify (09-20 h Budapest, one per 20 h, max 3, registered
address, only an accepted mail counts) / os_update.kernel {kver, tonight}
(no mail, no step) / layer kernel ingest + operator events / Approve kernel
set after every ring-0 box booted it healthily after a night stage / two
System page cells. 11 §5.11 written; §5.10 status corrected (proven).
Installer uninstall knows the two GRUB generators (unreleased).
Evidence: audits/kernel-lane-2026-10-07/ (red-proofs, boot timing).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -469,7 +469,7 @@ Decision 88 (R-851): *"Yes, but maybe not indefinitely."* Evidence `audits/os-do
|
||||
|
||||
---
|
||||
|
||||
### 5.10 The Proxmox package lane — BUILT 2026-10-07, unreleased, not yet proven live (R-812 option A, `09` §3 decision 163) `[FACT]`
|
||||
### 5.10 The Proxmox package lane — BUILT 2026-10-07 (agent v0.151.0, hub v0.142.0), proven on demo-felhom (R-812 option A, `09` §3 decision 163) `[FACT]`
|
||||
|
||||
**What it updates:** the host's Proxmox USERSPACE packages (pve-manager, qemu-server, pve-container, lxc-pve,
|
||||
libpve-*, proxmox-widget-toolkit, the ceph client libraries …) — the 77–78 packages behind on the demo boxes on
|
||||
@@ -498,9 +498,81 @@ libpve-*, proxmox-widget-toolkit, the ceph client libraries …) — the 77–78
|
||||
when every ring-0 box ran the set in `DockerNightsEffective` (2) healthy night steps since it was first seen and none
|
||||
was unhealthy — „healthy" is the box's own verdict above (no memory-kill check; that is the Docker engine's, R-528).
|
||||
An approval nudges no box: ring 1 takes the set only by a signed `os_pve_step` (signed per box, as `os_docker_step`).
|
||||
- **Not yet:** the live proof on demo-felhom (ring 0); the undo runbook (by hand: `apt-get install <name>=<old>` —
|
||||
- **Proven live 2026-10-07** on demo-felhom (ring 0, a signed `os_pve_step`): 65 packages in 70 s, pve-manager
|
||||
9.2.2 → 9.2.21, healthy, every container kept its id, the hub logged the report (`audits/day-2026-10-07/B/RESULT.md`).
|
||||
- **Not yet:** the undo runbook (by hand: `apt-get install <name>=<old>` —
|
||||
Proxmox keeps 30–66 old versions, C2).
|
||||
|
||||
### 5.11 The kernel lane — BUILT 2026-10-07 (agent v0.152.0, hub v0.143.0; R-836, `09` §3 decisions 164, 171, 172) `[FACT]`
|
||||
|
||||
**The ruling (decision 172):** design option A (a one-shot flag on the ESP) with option C (lockup → panic) on the one-shot
|
||||
entry. A box may restart at night for a kernel update; **the household is mailed the day before** with what to do if
|
||||
the box is not back by morning; each kernel set is approved by the operator after ring 0 ran it; **a frozen new kernel
|
||||
that needs a person's power cycle is accepted** for the first customers. Why A: measured in the spike on the Tester 1
|
||||
VM, demo-felhom and demo-hp (Secure Boot on) — the ESP flag and UEFI `BootNext` both boot a kernel once and fall back
|
||||
after a panic; a hardware watchdog armed during the reboot never fired on any of the three
|
||||
(`audits/kernel-spike-2026-10-07/DESIGN-kernel-lane.md`). A writes nothing to firmware.
|
||||
|
||||
**The boot side** (config bundle, two GRUB generators): `/etc/grub.d/01_felhom_oneshot` reads `felhom_next` from a GRUB
|
||||
env block on the ESP (`EFI/felhom/oneshot.env`), clears and saves it BEFORE the menu, and — only if it names an
|
||||
installed kernel — boots that kernel's one-shot entry. `/etc/grub.d/42_felhom_oneshot` gives each installed kernel an
|
||||
entry `felhom-oneshot-<ver>`: the normal entry plus `softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10`
|
||||
(option C; sorted after `10_linux`, so never entry 0 and never the default). A panic on the one-shot restarts the box in
|
||||
10 s into the default — the old kernel. Without a vfat ESP both print nothing.
|
||||
|
||||
**The default** is the kernel the box RUNS, pinned in `/etc/default/grub.d/zz-felhom-kernel-default.cfg` before the
|
||||
install and proved from grub.cfg after it (an install would otherwise make the newest kernel the default — R-836's
|
||||
defect). Only a healthy one-shot boot moves it.
|
||||
|
||||
**The root wrapper** (`felhom-os-apply`, layer `kernel`, lane slow, an appliance, authority = a signed `os_kernel_step`
|
||||
or the root-owned ring-0 mark): `apply` STAGES (installs the set — the series meta-package, at most one new kernel
|
||||
image, the boot helper and firmware — pins the default, writes the flag; never reboots); `kernel-reboot` reboots a
|
||||
staged step; `kernel-boot` says what became of it after a boot; `kernel-good` moves the default; `kernel-revert`
|
||||
reboots ONCE into the old kernel; `kernel-cancel`; `kernel-status`. Refusals R20 (the box cannot do a one-shot: not
|
||||
UEFI, no vfat ESP, a separate /boot, GRUB without fat/loadenv, the generators missing, a hand pin), R21 (the crash
|
||||
guard tripped or saw an unclean boot within its window), R22 (the step's phase does not allow it; never two steps
|
||||
within 20 h), R23 (not exactly one newer kernel, or not the one signed / told). State `/var/lib/felhom-kernel/state.json`.
|
||||
|
||||
**The night slot** (agent): the kernel step ENDS the night leg — after the host step (and the Proxmox step when one ran)
|
||||
ended healthy, after the whole-guest backup (the leg runs only after it), under the same heavy-op gate, trigger `night`
|
||||
only, at most one per night. Ring 0 stages the pending kernel and reboots; ring 1 reboots only a kernel a signed
|
||||
`os_kernel_step` staged by day (the job never reboots). The hub hears `staged` before the reboot.
|
||||
|
||||
**"Boot good" and the self-revert** (agent, at every start): on the new kernel the agent judges the boot for **20
|
||||
minutes** — the host health rule (§8.2: the Proxmox daemons and the agent active, the customer guest running and its
|
||||
own rule passing, the tunnel running) AND the box reached the hub (the `judging` report itself). Healthy → the new
|
||||
kernel becomes the default (`applied`). Not healthy by the deadline → `health_failed`, then ONE self-revert into the old
|
||||
kernel (`self_reverted` after it boots; a revert that comes back on the new kernel is `revert_failed` and never
|
||||
retried). The wait, measured 2026-10-07 by a plain reboot (`audits/kernel-lane-2026-10-07/B/`): every container healthy
|
||||
68 s after the reboot on demo-felhom (the hub reached at 63 s; Tailscale back at 53 s — the spike's > 6 min did not
|
||||
recur) and 272 s on demo-hp (the hub at 189 s). A box that never comes back raises the hub's existing `host_stale`
|
||||
(45 min, `alerting.stale_threshold`) and `host_down` (90 min) to the operator.
|
||||
|
||||
**The crash guard** (§5.9): the step's reboot and the self-revert are orderly (the clean-stop marker), so a step adds at
|
||||
most ONE unclean boot (a panic before userspace adds none: the planned reboot's marker is still there); with R21 the
|
||||
step starts only with none in the window — the box cannot reach the 3rd unclean boot that leaves it off. Pinned by
|
||||
`KernelStepCannotLeaveTheBoxOff`.
|
||||
|
||||
**The household mail** (hub): a box is DUE when ring 0 has a pending kernel (its host report's `proxmox-kernel-X.Y`
|
||||
upgrade) or a ring-1 box runs a kernel a signed job staged, and no step for that kernel has ended (`applied`,
|
||||
`fell_back`, `health_failed`, `self_reverted`, `revert_failed`, or a refusal R20/R23/R3/R12 — the operator decides; never
|
||||
retried by itself). Its household gets ONE mail (`mail.kernel.*`, its language, informal; to the registered address)
|
||||
between 09:00 and 20:00 Budapest time, at most once per 20 h and three times per kernel. The box's `os_update.kernel`
|
||||
block says `tonight` only within 24 h of a mail the mail service ACCEPTED — **no mail, no step**. Operator events
|
||||
`os_kernel_step` and `os_kernel_notice`; the household's timeline gets the usual "security fixes installed" line after
|
||||
`applied`.
|
||||
|
||||
**Approval** (hub): "Approve kernel set" on the System page appears when every ring-0 box's newest ended kernel step is
|
||||
`applied` for the same kernel, after a NIGHT stage. The approved set is the series meta-package and the signed image. An
|
||||
approval nudges no box: ring 1 takes it only through a signed `os_kernel_step` (stage), then its own told night.
|
||||
|
||||
**The System page** shows per box: the running kernel, the next boot, the default (amber while a one-shot flag names
|
||||
another kernel), and the kernel step (the newest result, the kernel due, whether the household was told for tonight).
|
||||
|
||||
**Not measured:** option C itself — the Proxmox kernel ships no lockup test module (`CONFIG_TEST_LOCKUP` is not set on
|
||||
7.0.14-20), so a soft lockup turning into a panic is recorded as unmeasured (no tool was built). A true dead freeze
|
||||
needs a person (accepted, decision 172). Evidence and proofs: `audits/kernel-lane-2026-10-07/`.
|
||||
|
||||
## 6. Risks and edge cases
|
||||
|
||||
| # | What can go wrong | What the design does |
|
||||
@@ -580,7 +652,8 @@ Each step returns to the operator for go or no-go.
|
||||
after enrolment (16:24/16:25 UTC: 49 guest + 106 host packages, 110 s + 44 s, healthy). An installer pass would cost
|
||||
~45 s and run BEFORE any whole-guest backup exists (no undo) — not built.
|
||||
6. **Slow lane: host kernel and Proxmox packages, with the reboot.** **Split 2026-10-07 (decisions 163–164):** the Proxmox
|
||||
USERSPACE packages are §5.10 (BUILT, unreleased, no reboot); the kernel lane is R-836 (a spike with reboots first).
|
||||
USERSPACE packages are §5.10 (BUILT, proven on demo-felhom); the kernel lane is §5.11 (**BUILT 2026-10-07**, agent
|
||||
v0.152.0 + hub v0.143.0, decision 172; the spike with reboots came first, `audits/kernel-spike-2026-10-07/`).
|
||||
7. **Later:** the Proxmox major upgrade (PVE 9 → 10), drilled on ring 0 first.
|
||||
|
||||
---
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
# kernel lane red-proof 2026-10-07T13:08:35Z — each mutation must turn its test red (FAILED); the clean tree is green
|
||||
clean: OK
|
||||
no default pin before the install -> test_installing_a_kernel_does_not_change_the_grub_default: FAILED (failures=1)
|
||||
no boot-setup check -> test_a_box_without_the_esp_flag_is_refused: FAILED (failures=1)
|
||||
reboot without the flag check -> test_a_reboot_without_the_flag_is_refused: FAILED (failures=1)
|
||||
no signed-set image check -> test_a_kernel_outside_the_approved_set_is_refused: FAILED (failures=1) (re-run after the test was sharpened: the first run stayed green because a second image was ALSO refused by the one-kernel rule)
|
||||
no expect_kver check -> test_a_kernel_outside_the_approved_set_is_refused: FAILED (failures=1)
|
||||
snippet sets the default before clearing the flag -> test_the_snippet_clears_the_flag_on_use: FAILED (errors=1)
|
||||
no crash-guard check -> test_the_crash_guard_must_be_armed_and_quiet: FAILED (failures=1)
|
||||
self-revert used twice -> test_one_self_revert_then_never_again: FAILED (failures=1)
|
||||
guard trips one unclean boot early (LIMIT-2) -> KernelStepCannotLeaveTheBoxOff.test_planned_reboot_one_crash_one_self_revert: FAILED (failures=1)
|
||||
go: kernelDue ignores Tonight -> TestKernel_NoMailNoStep: --- FAIL: TestKernel_NoMailNoStep (0.00s) FAIL
|
||||
go: KernelVerdict ignores the hub -> TestKernelVerdict: --- FAIL: TestKernelVerdict (0.00s) FAIL
|
||||
go: no self-revert call -> TestKernelAfterBoot_UnhealthyRevertsOnceAfterTheWait: --- FAIL: TestKernelAfterBoot_UnhealthyRevertsOnceAfterTheWait (0.00s) FAIL
|
||||
hub: tonight without a mail -> TestKernel_DueButNotToldIsNotTonight: --- FAIL: TestKernel_DueButNotToldIsNotTonight (0.04s)
|
||||
hub: a failed mail still recorded -> TestKernel_NoMailNoStep: --- FAIL: TestKernel_NoMailNoStep (0.04s)
|
||||
hub: approval without a night stage -> TestKernel_ApproveNeedsEveryRing0BoxHealthyAfterANightStep: --- FAIL: TestKernel_ApproveNeedsEveryRing0BoxHealthyAfterANightStep (0.03s)
|
||||
hub: kernel button without its readiness gate (.Fingerprint and Waiting dropped) -> TestSystemPage_KernelButtonOnlyWhenReady: --- FAIL: TestSystemPage_KernelButtonOnlyWhenReady (0.04s)
|
||||
@@ -0,0 +1,36 @@
|
||||
2026-10-07T15:11:39+02:00 demo-felhom systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
|
||||
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.785+02:00 level=INFO msg="felhom-agent daemon starting" version=0.151.0 host_id=demo-felhom-8363b5 hub_url=https://hub.felhom.eu interval_s=900
|
||||
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.795+02:00 level=INFO msg="daemon: operator signers pinned" count=2
|
||||
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="local-api leaf LOADED" fingerprint_sha256=c4b96bf0f97efcd462b11d33bd39a28133b4b3648535ee5b497c005eadf3e51e cert=/var/lib/felhom-agent/local-api.crt
|
||||
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="backup tier armed" target=felhom-backup cadence=24h0m0s keep_last=3 wait_timeout=30m0s prune_pbs_allowed=false primary=true
|
||||
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
|
||||
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="lanresolver: enabled" host_ip=192.168.0.162 upstreams="[1.1.1.1 8.8.8.8]" interval_s=300
|
||||
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="wgtunnel: enabled" interval_s=60 state_dir=/var/lib/felhom-agent
|
||||
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="poke: agent-plane sync listener enabled" port=51822
|
||||
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="felhomsshd (OOB): enabled" interval_s=60 state_dir=/var/lib/felhom-agent
|
||||
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)"
|
||||
2026-10-07T15:11:40+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:40.530+02:00 level=INFO msg="capabilities self-check" ok=68 total=68 degraded=0 inactive=0
|
||||
2026-10-07T15:11:40+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:40.530+02:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
|
||||
2026-10-07T15:11:40+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:40.530+02:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
|
||||
2026-10-07T15:11:40+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:40.531+02:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
|
||||
2026-10-07T15:11:40+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:40.530+02:00 level=INFO msg="fast-tick armed: 30s out-of-band cadence while desired-state is unapplied" interval=30s
|
||||
2026-10-07T15:11:40+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:40.534+02:00 level=INFO msg="poke: listening for hub sync-pokes (WG-confined, contentless)" addr=10.77.0.2:51822
|
||||
2026-10-07T15:11:43+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:43.004+02:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
|
||||
2026-10-07T15:11:43+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:43.004+02:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
|
||||
2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.915+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.933+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.933+02:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s guests_dir=/var/lib/felhom-agent/guests
|
||||
2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.933+02:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3
|
||||
2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.933+02:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
|
||||
2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.934+02:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
|
||||
2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.934+02:00 level=INFO msg="local-api server listening" addr=169.254.253.1:8443
|
||||
2026-10-07T15:11:45+02:00 demo-felhom felhom-priv-apply[1711]: felhom-priv-apply: SAME sshd-config /etc/felhom-sshd/sshd_config
|
||||
2026-10-07T15:11:46+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:46.084+02:00 level=INFO msg="felhomsshd: config applied" port=8822 action=reload
|
||||
2026-10-07T15:11:46+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:46.142+02:00 level=INFO msg="felhomsshd belt: set synced" set=ssh_port elements=8822
|
||||
2026-10-07T15:11:49+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:49.890+02:00 level=INFO msg="guestnet: guest network unhealthy but not acting" vmid=9201 reason="agent started less than 3m0s ago" detail="no IPv4 address on eth0"
|
||||
2026-10-07T15:11:50+02:00 demo-felhom felhom-agent[1198]: 2026/10/07 15:11:50 http: TLS handshake error from 169.254.253.2:55264: EOF
|
||||
2026-10-07T15:11:50+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:50.475+02:00 level=WARN msg="lanresolver: reconcile failed" vmid=9201 err="discover guest 9201 domain: pct exec cat controller.yaml: : exit status 137"
|
||||
2026-10-07T15:11:51+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:51.814+02:00 level=INFO msg="desired: updated from hub" generation=18 guests=0
|
||||
2026-10-07T15:11:51+02:00 demo-felhom felhom-priv-apply[3439]: felhom-priv-apply: SAME sshd-key /etc/felhom-sshd/authorized_keys/felhom-op
|
||||
2026-10-07T15:11:51+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:51.868+02:00 level=INFO msg="felhomsshd: operator authorized_keys updated" user=felhom-op present=true
|
||||
2026-10-07T15:11:51+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:51.930+02:00 level=INFO msg="felhomsshd belt: set synced" set=operator_ips elements=10.77.0.250
|
||||
@@ -0,0 +1,11 @@
|
||||
old boot b7c1573e-e8d3-4921-9a98-4ec43dcbad70
|
||||
reboot sent 2026-10-07T13:10:48Z
|
||||
+53s LAN ssh back
|
||||
+53s guest running
|
||||
+53s containers running=5/5 starting-or-unhealthy=4
|
||||
+53s tunnel container running
|
||||
+53s tailnet ssh back
|
||||
+68s agent reached the hub (2026-10-07T15:11:51+02:00)
|
||||
+68s containers running=5/5 starting-or-unhealthy=0
|
||||
+68s ALL containers running and healthy
|
||||
SUMMARY lan=53 hub=68 guest=53 apps=68 tunnel=53 tailnet=53
|
||||
@@ -0,0 +1,138 @@
|
||||
2026-10-07T15:11:34+02:00 demo-felhom systemd[1]: Starting felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit)...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom felhom-crash-guard[532]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10
|
||||
2026-10-07T15:11:34+02:00 demo-felhom felhom-crash-guard[653]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10
|
||||
2026-10-07T15:11:34+02:00 demo-felhom systemd[1]: Finished felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit).
|
||||
2026-10-07T15:11:34+02:00 demo-felhom systemd[1]: Starting tailscaled.service - Tailscale node agent...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: TPM: successfully read all properties
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Program starting: v1.102.2-t6cac91817-g6ff0ddc72, Go 1.26.5: []string{"/usr/sbin/tailscaled", "--state=/var/lib/tailscale/tailscaled.state", "--socket=/run/tailscale/tailscaled.sock", "--port=41641"}
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: LogID: 94a2aaa0b2c8fac06a42677d5acc9c23a59b45535ba49413ff0e63882f48728e
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: logpolicy: using $STATE_DIRECTORY, "/var/lib/tailscale"
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: [rc=unknown ret=direct]
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: using "direct" mode
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: using *dns.directManager
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: inotify: NewDirWatcher: context canceled
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: logtail: dial "log.tailscale.com:443" failed: dial tcp: lookup log.tailscale.com on 192.168.0.1:53: dial udp 192.168.0.1:53: connect: network is unreachable (in 2ms), trying bootstrap...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp6.tailscale.com", "68.183.90.120") for "log.tailscale.com" ...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp6.tailscale.com", "68.183.90.120") for "log.tailscale.com" error: Get "https://derp6.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp 68.183.90.120:443: connect: network is unreachable
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp4e.tailscale.com", "2a03:b0c0:3:d0::29:9001") for "log.tailscale.com" ...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp4e.tailscale.com", "2a03:b0c0:3:d0::29:9001") for "log.tailscale.com" error: Get "https://derp4e.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp [2a03:b0c0:3:d0::29:9001]:443: connect: network is unreachable
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp4c.tailscale.com", "134.122.77.138") for "log.tailscale.com" ...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp4c.tailscale.com", "134.122.77.138") for "log.tailscale.com" error: Get "https://derp4c.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp 134.122.77.138:443: connect: network is unreachable
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp7.tailscale.com", "2401:c080:1000:467f:5400:2ff:feee:22aa") for "log.tailscale.com" ...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp7.tailscale.com", "2401:c080:1000:467f:5400:2ff:feee:22aa") for "log.tailscale.com" error: Get "https://derp7.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp [2401:c080:1000:467f:5400:2ff:feee:22aa]:443: connect: network is unreachable
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp4e.tailscale.com", "134.122.74.153") for "log.tailscale.com" ...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp4e.tailscale.com", "134.122.74.153") for "log.tailscale.com" error: Get "https://derp4e.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp 134.122.74.153:443: connect: network is unreachable
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp6.tailscale.com", "2400:6180:100:d0::982:d001") for "log.tailscale.com" ...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp6.tailscale.com", "2400:6180:100:d0::982:d001") for "log.tailscale.com" error: Get "https://derp6.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp [2400:6180:100:d0::982:d001]:443: connect: network is unreachable
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp7.tailscale.com", "167.179.89.145") for "log.tailscale.com" ...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp7.tailscale.com", "167.179.89.145") for "log.tailscale.com" error: Get "https://derp7.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp 167.179.89.145:443: connect: network is unreachable
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp12b.tailscale.com", "2001:19f0:5c01:48a:5400:3ff:fe8d:cb5f") for "log.tailscale.com" ...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp12b.tailscale.com", "2001:19f0:5c01:48a:5400:3ff:fe8d:cb5f") for "log.tailscale.com" error: Get "https://derp12b.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp [2001:19f0:5c01:48a:5400:3ff:fe8d:cb5f]:443: connect: network is unreachable
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp8c.tailscale.com", "206.189.16.32") for "log.tailscale.com" ...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp8c.tailscale.com", "206.189.16.32") for "log.tailscale.com" error: Get "https://derp8c.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp 206.189.16.32:443: connect: network is unreachable
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp4d.tailscale.com", "2a03:b0c0:3:d0::1501:b001") for "log.tailscale.com" ...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp4d.tailscale.com", "2a03:b0c0:3:d0::1501:b001") for "log.tailscale.com" error: Get "https://derp4d.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp [2a03:b0c0:3:d0::1501:b001]:443: connect: network is unreachable
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp1e.tailscale.com", "64.225.56.166") for "log.tailscale.com" ...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp1e.tailscale.com", "64.225.56.166") for "log.tailscale.com" error: Get "https://derp1e.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp 64.225.56.166:443: connect: network is unreachable
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp12.tailscale.com", "2001:19f0:5c01:289:5400:3ff:fe8d:cb5e") for "log.tailscale.com" ...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp12.tailscale.com", "2001:19f0:5c01:289:5400:3ff:fe8d:cb5e") for "log.tailscale.com" error: Get "https://derp12.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp [2001:19f0:5c01:289:5400:3ff:fe8d:cb5e]:443: connect: network is unreachable
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: logtail: upload: log upload of 365 bytes compressed failed: Post "https://log.tailscale.com/c/tailnode.log.tailscale.io/a8874038d193b4737b146408485f2db2abd40834fbfa94f59c39122bb7f51754": failed to resolve "log.tailscale.com": no DNS fallback candidates remain for "log.tailscale.com"
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: router: enumerating tailscale0 addresses for cleanup failed: failed to look up link "tailscale0": Link not found
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: wgengine.NewUserspaceEngine(tun "tailscale0") ...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom systemd[1]: Started tailscaled.service - Tailscale node agent.
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: [rc=unknown ret=direct]
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: using "direct" mode
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: using *dns.directManager
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: link state: interfaces.State{defaultRoute= ifs={} v4=false v6=false}
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: router: portUpdate(port=41641, network=udp6)
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: magicsock: disco key = d:f75f0468399c429d
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: router: using firewall mode pref
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: magicsock: SetNetworkUp(false)
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Creating WireGuard device...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Bringing WireGuard device up...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: router: default choosing iptables
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Bringing router up...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: external route: up
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: router: netfilter running in iptables mode v6 = true, v6filter = true, v6nat = true
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: router: portUpdate(port=41641, network=udp4)
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Clearing router settings...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Starting network monitor...
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Engine created.
|
||||
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: LinkChange: all links down; pausing: old: interfaces.State{defaultRoute= ifs={} v4=false v6=false} new: interfaces.State{defaultRoute= ifs={} v4=false v6=false} diff: numInterfaces: 3->4; numInterfaceIPs: 3->4; if tailscale0: added; ips tailscale0: added [fe80::fbbf:d4ae:bf83:b0ed/64]
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: pm: using backend prefs for "profile-10be": Prefs{ra=false dns=false want=true routes=[] statefulFiltering=false nf=on host="felhom-pve" update=on Persist{o=, n=[CE5tx] u="nagyfenyvesi.viktor@gmail.com" ak=-}}
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: envknob: PORT="41641"
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: logpolicy: using $STATE_DIRECTORY, "/var/lib/tailscale"
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: linkChange: in state NoState; PAC or proxyConfig changed; updating routes
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: got LocalBackend in 35ms
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: Start
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: offline auto-update: starting update checks
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: ipnext: "conn25": skipping extension
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: ipnext: active extensions: captiveportal, acme, portlist, posture, clientupdate, relayserver, routecheck, serviceclientprefs, taildrop
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: load netmap from cache: netmap cache is not available
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: control: setPaused(true)
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: control: authRoutine: awaiting unpause
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: control: mapRoutine: awaiting unpause
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: Backend: logs: be:94a2aaa0b2c8fac06a42677d5acc9c23a59b45535ba49413ff0e63882f48728e fe:
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: control: updateRoutine: awaiting unpause
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: control: client.Login(0)
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: health(warnable=warming-up): error: Tailscale is starting. Please wait.
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=192.168.0.162/0, dst=192.168.0.162/32, gw=, outif=5, table=255
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=192.168.0.162/0, dst=192.168.0.0/24, gw=, outif=5, table=254
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=192.168.0.162/0, dst=192.168.0.255/32, gw=, outif=5, table=255
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=, dst=, gw=192.168.0.1, outif=5, table=254
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=10.77.0.2/0, dst=10.77.0.2/32, gw=, outif=7, table=255
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=, dst=10.77.0.250/32, gw=, outif=7, table=254
|
||||
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=, dst=10.77.0.1/32, gw=, outif=7, table=254
|
||||
2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: control: setPaused(false)
|
||||
2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: LinkChange: major, rebinding: old: interfaces.State{defaultRoute= ifs={} v4=false v6=false} new: interfaces.State{defaultRoute=vmbr0 ifs={vmbr0:[192.168.0.162/24 llu6] wg-felhom:[10.77.0.2/32]} v4=true v6=false} diff: HaveV4: false->true; DefaultRoute: ""->"vmbr0"; numInterfaces: 4->7; numInterfaceIPs: 4->7; if enp1s0 flags: broadcast|multicast->up|broadcast|multicast|running; if vmbr0: added; if vmbr9: added; if wg-felhom: added; ips vmbr0: added [192.168.0.162/24 fe80::6a1d:efff:fe5d:a664/64]; ips vmbr9: added [169.254.253.1/30 fe80::c0fc:cff:feca:d18c/64]; ips wg-felhom: added [10.77.0.2/32] rebind-reason=[default-if-changed,ips-changed,protocols-changed]
|
||||
2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: magicsock: SetNetworkUp(true)
|
||||
2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: control: doLogin(regen=false, hasUrl=false)
|
||||
2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: router: portUpdate(port=41641, network=udp6)
|
||||
2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: Rebind; defIf="vmbr0", ips=[192.168.0.162/24 fe80::6a1d:efff:fe5d:a664/64]
|
||||
2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: magicsock: 0 active derp conns
|
||||
2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: router: portUpdate(port=41641, network=udp4)
|
||||
2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: monitor: gateway and self IP changed: gw=192.168.0.1 self=192.168.0.162
|
||||
2026-10-07T15:11:40+02:00 demo-felhom tailscaled[711]: health(warnable=warming-up): ok
|
||||
2026-10-07T15:11:40+02:00 demo-felhom systemd[1]: Starting pve-guests.service - PVE guests...
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: control: control server key from https://controlplane.tailscale.com: ts2021=[fSeS+], legacy=[nlFWp]
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: control: RegisterReq: onode= node=[CE5tx] fup=false nks=false
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: control: RegisterReq: got response; nodeKeyExpired=false, machineAuthorized=true; authURL=false
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: control: netmap: got new dial plan from control
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: new contact: control-netmap usec=6317945 cached=false
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: active login: nagyfenyvesi.viktor@gmail.com
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: netmap: suggested exit node: no preferred DERP, try again later
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: offline auto-update: stopping update checks
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: Switching ipn state NoState -> Starting (WantRunning=true, nm=true)
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: SetPrivateKey called (init)
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: wgengine: Reconfig: configuring router
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: router: enabling connmark-based rp_filter workaround
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: wgengine: Reconfig: user dialer
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: tsdial: bart table size: 5
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: wgengine: Reconfig: configuring DNS
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: dns: Set: {DefaultResolvers:[] Routes:{} SearchDomains:[] Hosts:0}
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: dns: Resolvercfg: {Routes:{} Hosts:0 LocalDomains:[]}
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: dns: OScfg: {}
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: peerapi: serving on http://100.70.170.35:36209
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: peerapi: serving on http://[fd7a:115c:a1e0::5236:aa24]:61570
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: portmapper: UPnP discovery response from 192.168.0.254, but gateway IP is 192.168.0.1
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: portmapper: UPnP meta changed: [{Location:http://192.168.0.1:1900/rootDesc.xml Server:TP-Link/TP-LINK UPnP/1.1 MiniUPnPd/1.8 USN:uuid:e4c797e0-82ce-4f84-b56d-085d9d77c8cd::urn:schemas-upnp-org:device:InternetGatewayDevice:1} {Location:http://192.168.0.254:1900/igd.xml Server:ipos/7.0 UPnP/1.0 Archer_C5/2.0 USN:uuid:060b7353-fca6-4070-85f4-1fbfb9add62c::urn:schemas-upnp-org:device:InternetGatewayDevice:1}]
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: home DERP changing from derp-0 [0ms] to derp-4 [39ms] (forced=false)
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: home is now derp-4 (fra)
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: adding connection to derp-4 for home-keep-alive
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: 1 active derp conns: derp-4=cr0s,wr0s
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: derphttp.Client.Connect: connecting to derp-4 (fra)
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: updating netmap in disk cache
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: endpoints changed: 37.191.56.193:41641 (portmap), 10.77.0.2:41641 (local), 192.168.0.162:41641 (local)
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: Switching ipn state Starting -> Running (WantRunning=true, nm=true)
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: control: NetInfo: NetInfo{varies=false ipv6=false ipv6os=true udp=true icmpv4=false derp=#4 portmap=active-UM link="" firewallmode="ipt-default"}
|
||||
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: derp-4 connected; connGen=1
|
||||
2026-10-07T15:11:41+02:00 demo-felhom pve-guests[1375]: <root@pam> starting task UPID:demo-felhom:000005D5:00000504:6AC6450D:startall::root@pam:
|
||||
2026-10-07T15:11:41+02:00 demo-felhom pvesh[1375]: Starting CT 9201
|
||||
2026-10-07T15:11:41+02:00 demo-felhom pve-guests[1493]: <root@pam> starting task UPID:demo-felhom:000005D6:00000505:6AC6450D:vzstart:9201:root@pam:
|
||||
2026-10-07T15:11:41+02:00 demo-felhom pve-guests[1494]: starting CT 9201: UPID:demo-felhom:000005D6:00000505:6AC6450D:vzstart:9201:root@pam:
|
||||
2026-10-07T15:11:45+02:00 demo-felhom tailscaled[711]: magicsock: new contact: peer=[G+z+N] usec=10136339 cached=false via=derp
|
||||
2026-10-07T15:11:45+02:00 demo-felhom tailscaled[711]: magicsock: disco: node [G+z+N] d:ac1b0afc01e49a40 now using 192.168.0.180:35032 mtu=1360 tx=3f73e074118a
|
||||
2026-10-07T15:11:45+02:00 demo-felhom tailscaled[711]: magicsock: new contact: peer=[G+z+N] usec=10137837 cached=false via=direct
|
||||
2026-10-07T15:11:46+02:00 demo-felhom pve-guests[1375]: <root@pam> end task UPID:demo-felhom:000005D5:00000504:6AC6450D:startall::root@pam: OK
|
||||
2026-10-07T15:11:46+02:00 demo-felhom systemd[1]: Finished pve-guests.service - PVE guests.
|
||||
2026-10-07T15:11:51+02:00 demo-felhom tailscaled[711]: magicsock: disco: node [G+z+N] d:ac1b0afc01e49a40 now using 192.168.0.180:35032 mtu=1360 tx=2b416b182a12
|
||||
@@ -0,0 +1,50 @@
|
||||
2026-10-07T15:16:05+02:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
|
||||
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.853+02:00 level=INFO msg="felhom-agent daemon starting" version=0.151.0 host_id=demo-hp-bb76ea hub_url=https://hub.felhom.eu interval_s=900
|
||||
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.864+02:00 level=INFO msg="daemon: operator signers pinned" count=2
|
||||
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="local-api leaf LOADED" fingerprint_sha256=34122ac2ace991685ad9adaaf882702581ae931837203dbeb25f0d9e493a837d cert=/var/lib/felhom-agent/local-api.crt
|
||||
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=1 wait_timeout=30m0s prune_pbs_allowed=false primary=true
|
||||
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
|
||||
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="lanresolver: enabled" host_ip=192.168.0.104 upstreams="[1.1.1.1 8.8.8.8]" interval_s=300
|
||||
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="wgtunnel: enabled" interval_s=60 state_dir=/var/lib/felhom-agent
|
||||
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="poke: agent-plane sync listener enabled" port=51822
|
||||
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="felhomsshd (OOB): enabled" interval_s=60 state_dir=/var/lib/felhom-agent
|
||||
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.872+02:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)"
|
||||
2026-10-07T15:16:06+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:06.808+02:00 level=INFO msg="capabilities self-check" ok=68 total=68 degraded=0 inactive=0
|
||||
2026-10-07T15:16:06+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:06.809+02:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
|
||||
2026-10-07T15:16:06+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:06.809+02:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
|
||||
2026-10-07T15:16:06+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:06.809+02:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
|
||||
2026-10-07T15:16:06+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:06.809+02:00 level=INFO msg="poke: listening for hub sync-pokes (WG-confined, contentless)" addr=10.77.0.3:51822
|
||||
2026-10-07T15:16:06+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:06.809+02:00 level=INFO msg="fast-tick armed: 30s out-of-band cadence while desired-state is unapplied" interval=30s
|
||||
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.595+02:00 level=INFO msg="guest-attach: drive bound under shared parent (normalized to one bind, live)" vmid=9201 where=/mnt/hdd_1 stable=/mnt/felhom-drives/hdd_1 prior_binds=0
|
||||
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.595+02:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
|
||||
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.595+02:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
|
||||
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.595+02:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
|
||||
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.613+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.616+02:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3
|
||||
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.616+02:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s guests_dir=/var/lib/felhom-agent/guests
|
||||
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.616+02:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
|
||||
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.617+02:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
|
||||
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.617+02:00 level=INFO msg="local-api server listening" addr=169.254.253.1:8443
|
||||
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.619+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2026-10-07T15:16:07+02:00 demo-hp felhom-priv-apply[1599]: felhom-priv-apply: SAME sshd-config /etc/felhom-sshd/sshd_config
|
||||
2026-10-07T15:16:08+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:08.339+02:00 level=INFO msg="felhomsshd: config applied" port=8822 action=reload
|
||||
2026-10-07T15:16:08+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:08.395+02:00 level=INFO msg="felhomsshd belt: set synced" set=ssh_port elements=8822
|
||||
2026-10-07T15:16:08+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:08.930+02:00 level=INFO msg="desired: updated from hub" generation=9 guests=0
|
||||
2026-10-07T15:16:08+02:00 demo-hp felhom-priv-apply[1911]: felhom-priv-apply: SAME sshd-key /etc/felhom-sshd/authorized_keys/felhom-op
|
||||
2026-10-07T15:16:09+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:09.008+02:00 level=INFO msg="felhomsshd: operator authorized_keys updated" user=felhom-op present=true
|
||||
2026-10-07T15:16:09+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:09.083+02:00 level=INFO msg="felhomsshd belt: set synced" set=operator_ips elements=10.77.0.250
|
||||
2026-10-07T15:16:09+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:09.581+02:00 level=INFO msg="lanresolver: guest IP not yet leased — skipping (will retry)" vmid=9201
|
||||
2026-10-07T15:16:10+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:10.562+02:00 level=INFO msg="lanresolver: guest IP not yet leased — skipping (will retry)" vmid=9202
|
||||
2026-10-07T15:16:23+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:23.620+02:00 level=WARN msg="local-api: storage view unavailable for class hints" err="storage: NodeStorage: proxmox: GET /nodes/demo-hp/storage: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/storage\": context canceled"
|
||||
2026-10-07T15:16:28+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:28.656+02:00 level=INFO msg="guest-attach: drive bound under shared parent (normalized to one bind, live)" vmid=9201 where=/mnt/hdd_1 stable=/mnt/felhom-drives/hdd_1 prior_binds=1
|
||||
2026-10-07T15:16:28+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:28.657+02:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
|
||||
2026-10-07T15:16:37+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:37.744+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2026-10-07T15:16:39+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:39.272+02:00 level=INFO msg="controller-supervisor: slow counter restored from disk" vmid=9201 restarts_24h=0 slow_crashloop_since=2026-09-17T09:22:29Z
|
||||
2026-10-07T15:16:48+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:48.483+02:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
|
||||
2026-10-07T15:16:55+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:55.099+02:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
|
||||
2026-10-07T15:17:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:17:07.712+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2026-10-07T15:17:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:17:07.725+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2026-10-07T15:17:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:17:07.737+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2026-10-07T15:17:08+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:17:08.416+02:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
|
||||
2026-10-07T15:17:28+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:17:28.462+02:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
|
||||
2026-10-07T15:17:37+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:17:37.717+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
@@ -0,0 +1,21 @@
|
||||
old boot 8ea2cdff-d166-4af6-9937-156f4ee659a5
|
||||
reboot sent 2026-10-07T13:13:06Z
|
||||
+176s LAN ssh back
|
||||
+176s containers running=0/0 starting-or-unhealthy=0
|
||||
+189s agent reached the hub (2026-10-07T15:16:08+02:00)
|
||||
+189s guest running
|
||||
+189s containers running=18/21 starting-or-unhealthy=17
|
||||
+189s tunnel container running
|
||||
+204s containers running=20/21 starting-or-unhealthy=11
|
||||
+211s containers running=20/21 starting-or-unhealthy=7
|
||||
+217s containers running=21/21 starting-or-unhealthy=6
|
||||
+224s containers running=21/21 starting-or-unhealthy=6
|
||||
+231s containers running=21/21 starting-or-unhealthy=6
|
||||
+238s containers running=21/21 starting-or-unhealthy=6
|
||||
+245s containers running=21/21 starting-or-unhealthy=6
|
||||
+252s containers running=21/21 starting-or-unhealthy=6
|
||||
+258s containers running=21/21 starting-or-unhealthy=2
|
||||
+265s containers running=21/21 starting-or-unhealthy=1
|
||||
+272s containers running=21/21 starting-or-unhealthy=0
|
||||
+272s ALL containers running and healthy
|
||||
SUMMARY lan=176 hub=189 guest=189 apps=272 tunnel=189 tailnet=n/a
|
||||
@@ -0,0 +1,18 @@
|
||||
2026-10-07T15:16:00+02:00 demo-hp systemd[1]: Starting felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit)...
|
||||
2026-10-07T15:16:00+02:00 demo-hp felhom-crash-guard[628]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10
|
||||
2026-10-07T15:16:00+02:00 demo-hp felhom-crash-guard[773]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10
|
||||
2026-10-07T15:16:00+02:00 demo-hp systemd[1]: Finished felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit).
|
||||
2026-10-07T15:16:06+02:00 demo-hp systemd[1]: Starting pve-guests.service - PVE guests...
|
||||
2026-10-07T15:16:08+02:00 demo-hp pve-guests[1493]: <root@pam> starting task UPID:demo-hp:000006CD:000006D3:6AC64618:startall::root@pam:
|
||||
2026-10-07T15:16:08+02:00 demo-hp pvesh[1493]: Starting VM 341
|
||||
2026-10-07T15:16:08+02:00 demo-hp pve-guests[1741]: <root@pam> starting task UPID:demo-hp:000006D4:000006D5:6AC64618:qmstart:341:root@pam:
|
||||
2026-10-07T15:16:08+02:00 demo-hp pve-guests[1748]: start VM 341: UPID:demo-hp:000006D4:000006D5:6AC64618:qmstart:341:root@pam:
|
||||
2026-10-07T15:16:09+02:00 demo-hp pve-guests[1748]: VM 341 started with PID 1891.
|
||||
2026-10-07T15:16:11+02:00 demo-hp pvesh[1493]: Starting CT 9201
|
||||
2026-10-07T15:16:11+02:00 demo-hp pve-guests[1741]: <root@pam> starting task UPID:demo-hp:000007A3:00000804:6AC6461B:vzstart:9201:root@pam:
|
||||
2026-10-07T15:16:11+02:00 demo-hp pve-guests[1955]: starting CT 9201: UPID:demo-hp:000007A3:00000804:6AC6461B:vzstart:9201:root@pam:
|
||||
2026-10-07T15:16:11+02:00 demo-hp pvesh[1493]: Starting CT 9202
|
||||
2026-10-07T15:16:11+02:00 demo-hp pve-guests[1741]: <root@pam> starting task UPID:demo-hp:000007A5:00000808:6AC6461B:vzstart:9202:root@pam:
|
||||
2026-10-07T15:16:11+02:00 demo-hp pve-guests[1957]: starting CT 9202: UPID:demo-hp:000007A5:00000808:6AC6461B:vzstart:9202:root@pam:
|
||||
2026-10-07T15:16:16+02:00 demo-hp pve-guests[1493]: <root@pam> end task UPID:demo-hp:000006CD:000006D3:6AC64618:startall::root@pam: OK
|
||||
2026-10-07T15:16:16+02:00 demo-hp systemd[1]: Finished pve-guests.service - PVE guests.
|
||||
@@ -0,0 +1,4 @@
|
||||
== agent_update 0.152.0 → tester-1, 2026-10-07T13:21:33Z
|
||||
signed: op=agent_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=31589d832a25a458ce3878c0fcbd3ae3 expires=2026-10-07T14:06:33Z
|
||||
wrote envelope to /dev/null
|
||||
uploaded signed op to the hub jobs queue
|
||||
Reference in New Issue
Block a user