releases 2b and 3 are out (2026-11-15): report, register (R-1380, R-1395, R-1050 closed; R-1397 filed), status, golden 0.353.0 record, evidence
gates / gates (push) Successful in 2m9s

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-15 12:14:54 +01:00
1 parent f561c85bbc
commit 07bf778699
70 files changed
+2416 -13

No files matched your search

@@ -0,0 +1,29 @@
# Releases 2b and 3 — evidence (2026-11-15)
Report: `REPORT-release-2b-3-2026-11-15.md` (repo root). Golden and fresh install:
`documentation/tests/golden-0.353.0-2026-11-15/`. The ids of the deleted boxes whose rows the sweep removed are NOT
here: `/mnt/5_hdd/felhom.eu/evidence/release-2b-3-2026-11-15/sweep-preview-ids.txt` (off git; an id can be a name).
| File | What it shows |
|---|---|
| `part0/redproof-agent-cutwrite.txt` | the cut-write fix of the agent's config: 9 mutations, each seen red; 3 redone and one hollow test removed, said in the file |
| `part0/sweep-preview-counts.txt`, `sweep-preview.sh` | on a copy of the hub's own 08:00 snapshot: the first sweep deletes 26 of 504 `os_reports` rows, 13 owners, 0 in the five other tables; the control (a row to delete whose owner exists) is 0; the copy is deleted |
| `part1/hub-0182-start-log.txt` | hub 0.182.0's start: „Purged 26 row(s) … map[os_reports:26]”, the daily ledger's first lines, the lost-mail scan |
| `part1/hub-system-page-0950.txt` | the System page on 0.182.0 with the two new cells |
| `part2/t1-before.txt`, `t1-after-agent.txt`, `t1-moved.txt` | Tester 1: pinned → agent 0.175.0 (the wrapper refuses the unknown mode; the false trust problem) → root files → „MOVED … cleared AFTER a working call with the copy” |
| `part2/hp-moved.txt`, `n100-moved.txt` | demo-hp and the N100 with 0.175.1: no trust problem in the window, then MOVED |
| `part2/hp-cert-round1.txt`, `hp-cert-round2.txt` | two forced new Proxmox certificates on demo-hp; after each: the same agent process, 0 trust lines, 0 ERROR lines, 14 and 40 calls by the agent's token answered 200 in Proxmox's own access log |
| `part2/hp-ca-wrong.txt` | the CA copy replaced by another CA and the agent restarted: the right root is written first; after the next start the file is identical to the saved one |
| `part2/redproof-agent-root-step-too-old.txt` | the fix of the false trust alarm and its narrowing after the security review: 7 mutations seen red, 3 redone |
| `part2/t1-clock-A-*.txt` | the clock 150 s back with the OLD controller: one app recreated at the step back and again at the step forward; the kernel's boot id unchanged |
| `part2/system-1107.txt` | the „Security profiles” cell filled for the three boxes |
| `part3/boxwalk-0353.txt`, `watch-boxes.txt`, `watch-hub.txt` | the three boxes on controller 0.353.0 (image `61cf2a8b…`), 0 ERROR lines; 25 minutes after the global floor |
| `part3/https-only.txt`, `hsts-headers.txt` | the https-only check green; 26 of 26 public names answer exactly `max-age=31536000` |
| `part3/hub-webcerts.txt` | from a copy of the live hub database (shredded after): the three boxes' reports carry `web_certs`; the ledger's and the end-date baseline's marks |
| `part3/t1-clock-B-*.txt` | the same clock step with the NEW controller and agent: the same container ids before and after, in both directions |
| `part3/t1-after-reboot.txt` | one real restart of Tester 1: a new boot id, the boot pass recreates the drive-backed app |
| `part3/per-box-floors-cleared.txt`, `hub-system-page-after.txt` | the per-box floors cleared; „No per-customer floors”; global floor 0.353.0, vouched agent 0.175.1 |
The hub's two mails of the day (both to the operator's address only, read in the test mailbox): `proxmox_trust_problem`
for tester-1 at 10:04 CET (false, fixed), `box_clock_wrong` for tester-1 at 11:05 CET („150 seconds behind”). The
second clock alarm at 11:20 was suppressed by the one-hour rule (the hub's log).
@@ -0,0 +1,13 @@
snapshot: /data/snapshots/hub-20261115T070000Z.db; hub pod clock: 2026-11-15T07:21:25Z; DooPlex: 2026-11-15T07:21:25Z
copy: 674902016 bytes
integrity: ok
freshness control: newest host report 2026-11-15 06:51:43
hosts: 4 customer_configs: 5
os_host_settings would delete 0 of 2 rows, 0 owner(s)
os_reports would delete 26 of 504 rows, 13 owner(s)
os_kernel_notices would delete 0 of 18 rows, 0 owner(s)
offsite_keys would delete 0 of 5 rows, 0 owner(s)
offsite_windows would delete 0 of 23 rows, 0 owner(s)
offsite_abandon_requests would delete 0 of 1 rows, 0 owner(s)
TOTAL rows: 26; deleted boxes named: 13; deleted households named: 0 (ids saved off git, not printed)
copy deleted: yes; ids file: /mnt/5_hdd/felhom.eu/evidence/release-2b-3-2026-11-15/sweep-preview-ids.txt (13 lines); rc=0
@@ -0,0 +1,50 @@
#!/bin/bash
# Part 0.2: what the hub's first sweep (PurgeOrphanRows, D73) will delete. Reads a COPY of the hub's newest own
# snapshot (a VACUUM INTO file: whole, no WAL needed, no old free-space bytes). The copy is deleted at the end.
# The ids go OFF git (an id can be a name).
set -u
EV=/mnt/5_hdd/felhom.eu/evidence/release-2b-3-2026-11-15
mkdir -p "$EV"; chmod 700 "$EV"
SNAP=$(sudo kubectl -n felhom-system exec deploy/hub -- sh -c 'ls -1 /data/snapshots/hub-*.db | tail -1')
echo "snapshot: $SNAP; hub pod clock: $(sudo kubectl -n felhom-system exec deploy/hub -- date -u +%FT%TZ); DooPlex: $(date -u +%FT%TZ)"
COPY="$EV/hubdb-copy.db"
sudo kubectl -n felhom-system exec deploy/hub -- cat "$SNAP" > "$COPY"
chmod 600 "$COPY"
echo "copy: $(stat -c %s "$COPY") bytes"
python3 -I - "$COPY" "$EV/sweep-preview-ids.txt" <<'PY'
import sqlite3, sys
db = sqlite3.connect("file:%s?mode=ro" % sys.argv[1], uri=True)
print("integrity:", db.execute("PRAGMA integrity_check").fetchone()[0])
cols = [r[1] for r in db.execute("PRAGMA table_info(host_reports)")]
tcol = "received_at" if "received_at" in cols else cols[-1]
print("freshness control: newest host report", db.execute("SELECT MAX(%s) FROM host_reports" % tcol).fetchone()[0])
print("hosts:", db.execute("SELECT COUNT(*) FROM hosts").fetchone()[0], " customer_configs:", db.execute("SELECT COUNT(*) FROM customer_configs").fetchone()[0])
tables = [
("os_host_settings", "host_id", "host_id NOT IN (SELECT host_id FROM hosts)"),
("os_reports", "host_id", "host_id NOT IN (SELECT host_id FROM hosts)"),
("os_kernel_notices", "host_id", "host_id NOT IN (SELECT host_id FROM hosts)"),
("offsite_keys", "customer_id", "customer_id NOT IN (SELECT customer_id FROM customer_configs) AND customer_id NOT IN (SELECT customer_id FROM hosts)"),
("offsite_windows", "customer_id", "customer_id NOT IN (SELECT customer_id FROM customer_configs) AND customer_id NOT IN (SELECT customer_id FROM hosts)"),
("offsite_abandon_requests", "customer_id", "customer_id NOT IN (SELECT customer_id FROM customer_configs) AND customer_id NOT IN (SELECT customer_id FROM hosts)"),
]
total = 0; boxes = set(); houses = set()
with open(sys.argv[2], "w") as out:
for t, col, where in tables:
all_n = db.execute("SELECT COUNT(*) FROM %s" % t).fetchone()[0]
rows = db.execute("SELECT %s, COUNT(*) FROM %s WHERE %s GROUP BY 1 ORDER BY 1" % (col, t, where)).fetchall()
n = sum(r[1] for r in rows); total += n
(boxes if col == "host_id" else houses).update(r[0] for r in rows)
print(" %-26s would delete %5d of %6d rows, %d owner(s)" % (t, n, all_n, len(rows)))
for owner, c in rows:
out.write("%s\t%s\t%s\t%d\n" % (t, col, owner, c))
# the control: no row to delete may name an owner that exists
if col == "host_id":
bad = db.execute("SELECT COUNT(*) FROM %s WHERE %s AND host_id IN (SELECT host_id FROM hosts)" % (t, where)).fetchone()[0]
else:
bad = db.execute("SELECT COUNT(*) FROM %s WHERE %s AND (customer_id IN (SELECT customer_id FROM customer_configs) OR customer_id IN (SELECT customer_id FROM hosts))" % (t, where)).fetchone()[0]
assert bad == 0, (t, bad)
print("TOTAL rows: %d; deleted boxes named: %d; deleted households named: %d (ids saved off git, not printed)" % (total, len(boxes), len(houses)))
PY
rc=$?
shred -u "$COPY" 2>/dev/null || rm -f "$COPY"
echo "copy deleted: $([ -e "$COPY" ] && echo NO || echo yes); ids file: $EV/sweep-preview-ids.txt ($(wc -l < "$EV/sweep-preview-ids.txt") lines); rc=$rc"
@@ -0,0 +1,43 @@
2026/11/15 09:48:59 [INFO] felhom-hub 0.182.0 starting
2026/11/15 09:48:59 [INFO] off-site secrets sealed at rest (0 legacy plaintext row(s) sealed now)
2026/11/15 09:48:59 [INFO] console passwords sealed at rest (0 legacy plaintext row(s) sealed now)
2026/11/15 09:48:59 [INFO] box secrets sealed at rest (0 legacy plaintext value(s) sealed now)
2026/11/15 09:48:59 [INFO] config tokens sealed at rest (0 legacy plaintext token(s) sealed now)
2026/11/15 09:48:59 [INFO] Database opened at /data/hub.db
2026/11/15 09:48:59 [INFO] Default controller-version floor: 0.120.0
2026/11/15 09:48:59 [INFO] Template fetcher started (every 1h)
2026/11/15 09:49:00 [INFO] Asset seed: all 263 files up-to-date
2026/11/15 09:49:00 [INFO] Asset manifest built: 288 files
2026/11/15 09:49:00 [INFO] App-email relay enabled (limit 30/min/customer, From domains [felhom.eu])
2026/11/15 09:49:00 [INFO] Gitea artifact browser enabled (Day-0 version dropdowns) via http://gitea.gitea-system.svc.cluster.local:3000
2026/11/15 09:49:00 [INFO] Offsite provisioning enabled (pool_box=611714, location=fsn1)
2026/11/15 09:49:00 [INFO] Offsite pool-box checker initialized: box=611714 fill warn=80% crit=90%, oversub warn=2.00x, unreachable after 3 consecutive failed reads, refresh 15m0s
2026/11/15 09:49:00 [INFO] Root files: bundles are compared by their files (R-1392)
2026/11/15 09:49:00 [INFO] osupdates: approval rule = every ring-0 box healthy for 24h0m0s and 1 night run(s)
2026/11/15 09:49:00 [INFO] osupdates: the Docker engine set is approved only by the operator, after 2 healthy ring-0 night(s)
2026/11/15 09:49:00 [INFO] osupdates: alarms = no OS leg 168h0m0s, reboot needed 336h0m0s, ring 0 stalled 168h0m0s, not covered 336h0m0s
2026/11/15 09:49:00 [INFO] Off-site key registrar enabled; daily key check at 07:10 Budapest
2026/11/15 09:49:00 [INFO] access log: /data/access-log, one file per UTC day, kept 90 days
2026/11/15 09:49:00 [INFO] Registry version checker started (every 6h)
2026/11/15 09:49:00 [INFO] WG peer-sync enabled (endpoint 167.233.158.164:22, user felhom-peersync)
2026/11/15 09:49:00 [INFO] PBS DR tenantsync enabled (endpoint 167.233.158.164:22, user felhom-peersync; WG-registration auto-provision hook armed)
2026/11/15 09:49:00 [INFO] PBS-DR box checker initialized: fill warn=80% crit=90%, unreachable after 3 consecutive failed reads, refresh 15m0s
2026/11/15 09:49:00 [INFO] agent-plane poke enabled (endpoint 167.233.158.164:22, user felhom-peersync; web + api admin seams armed)
2026/11/15 09:49:00 [INFO] PBS-DR self-heal reconciler started (interval 5m)
2026/11/15 09:49:00 [INFO] offsite credential self-heal reconciler started (interval 5m, debounce 2 reports)
2026/11/15 09:49:00 [INFO] daily ledger: offsite-key-audit is watched from now (no earlier record)
2026/11/15 09:49:00 [INFO] offsite-key-audit: next run at 2026-11-16 07:10 CET (in 21h20m59s)
2026/11/15 09:49:01 [INFO] Pruned 86 old report rows
2026/11/15 09:49:01 [INFO] Pruned 86 old app telemetry rows
2026/11/15 09:49:04 [WARN] slow statement: 3091ms (over 1s): DELETE FROM app_log_issues WHERE last_seen < ?
2026/11/15 09:49:04 [INFO] Pruned 34 stale app issues
2026/11/15 09:49:05 [INFO] Purged 26 row(s) of boxes and households that no longer exist: map[os_reports:26]
2026/11/15 09:49:05 [INFO] prune: next run at 2026-11-16 04:30 CET (in 18h40m55s)
2026/11/15 09:49:05 [INFO] Staleness checker initialized: 3 ok, 0 stale, 2 down (node_stale after 45m0s, node_down after 1h30m0s)
2026/11/15 09:49:05 [INFO] Host staleness checker initialized: 3 ok, 1 already alarmed (seeded silently, R-1001), 0 stale, 0 down (left unseeded → first Check emits; host_stale after 45m0s, host_down after 1h30m0s)
2026/11/15 09:49:05 [INFO] Host capability checker initialized: 4 ok, 0 degraded, 0 with a refused whole-guest backup (degraded/refused left unseeded → first Check emits)
2026/11/15 09:49:05 [INFO] Host leaf checker initialized: 4 host fingerprint(s) seeded
2026/11/15 09:49:05 [INFO] Host disk checker initialized: warn=90% crit=95%, 4 ok seeded, 0 already-breached left unseeded (first Check emits)
2026/11/15 09:49:05 [INFO] Storage fill checker initialized: warn=90% crit=95%, 9 ok seeded, 0 already-breached left unseeded, 4 root-backed excluded
2026/11/15 09:49:05 [INFO] Host mgmt-plane checker initialized: 0 host heal-state(s) seeded
2026/11/15 09:49:05 [INFO] Host OOB checker initialized: 0 host(s) seeded degraded
@@ -0,0 +1,256 @@
System — Felhom Hub
Felhom Hub
Dashboard
Customers
Apps
Hosts
System
Offsite
Configuration
System — versions and OS updates
Needs attention
One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3).
[hover: worth a look] look
Tester4-46836f Tester4
[hover: cannot do a one-shot boot: the box does not boot UEFI; /boot/efi is not a mounted vfat ESP (not mounted)] kernel step: — · the host needs a restart since 2026-11-14 · [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] root files behind the vouched agent's · [hover: 1 minor release behind — sign an agent_update for this box] agent behind: 0.173.0 → 0.174.0 (since 2026-11-14) · [hover: the box could not read these values (agent older than v0.142.0, or the guest is down) — never a guess] could not read: default kernel
Waiting for you
Update sets that only you approve: the kernel, the Docker engine and the Proxmox packages. Guest and host Debian updates approve themselves after 24 h and one night on the [hover: Ring 0 = the demo boxes. They install every new fix first; the other boxes (ring 1) install only what is approved.] ring-0 boxes.
Docker engine 29.9.3-1
Ran on every ring-0 box — demo-felhom-8363b5: 2 healthy night(s), demo-hp-bb76ea: 2 healthy night(s). The memory-kill check passed.
first seen 2 days ago (2026-11-13 04:49 UTC). Approving installs nothing by itself: a ring-1 box takes it only through a signed operator job.
Approve Docker set
Proxmox packages 9.2.27 — still being tested: demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set
Boxes
Click a box to see every value it reports, and its ring and update switches. "unknown": the box could not read the value — never a guess.
Box Ring Health Controller · agent OS updates Kernel [hover: The newest OS update run] Last night
Tester4-46836f Tester4
[hover: Ring 1: a normal box — it installs only approved releases] 1
[hover: worth a look] look
0.352.1 [hover: 1 minor release behind — sign an agent_update for this box] agent 0.173.0 → 0.174.0 (since 2026-11-14)
[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind
7.0.2-6-pve
[hover: 22 h ago · applied · 93 s — last successful leg: 22 h ago] ok · 22 h ago
Ring and updates
Ring ring 1
→ demo
OS updates updates ON
switch off
Tunnel [hover: no cloudflared container in the guest (agent state: absent)] none set up
Controller 0.352.1
Agent [hover: 1 minor release behind — sign an agent_update for this box] 0.173.0 → 0.174.0 (since 2026-11-14)
Last OS leg [hover: last successful leg: 22 h ago] 22 h ago · applied · 93 s
Docker engine
Docker 29.9.2
containerd 2.4.2-3~debian.13~trixie
live-restore on
Docker release —
Host
Proxmox 9.2.2
Kernel (running) 7.0.2-6-pve
Kernel (next boot) 7.0.2-6-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: cannot do a one-shot boot: the box does not boot UEFI; /boot/efi is not a mounted vfat ESP (not mounted)] —
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] —
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: the box reports no security-profile refresh (an agent older than release 3, a BYO host, or no refresh yet)] —
Debian 13.7
Felhom release os-host-20261111-034654
Pending 100
Not covered 0
Held none
Reboot needed since 2026-11-14
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] 0.173.0
Guest
Guest Debian 13.7
Felhom release os-guest-20261111-034654
Pending 6
Restart needed 7
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 22 h ago · 78.4 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] not yet
demo-felhom-8363b5 Demo Ügyfél
[hover: Ring 0: a demo box — it installs every new fix first] 0
[hover: nothing amber or red] fine
0.352.1 [hover: current (vouched 0.174.0)] agent 0.174.0
[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind
7.0.14-28-pve
[hover: 3 h ago · nothing · 23 s — last successful leg: 3 h ago] ok · 3 h ago
Ring and updates
Ring ring 0
→ normal
OS updates updates ON
switch off
Tunnel running
Controller 0.352.1
Agent [hover: current (vouched 0.174.0)] 0.174.0
Last OS leg [hover: last successful leg: 3 h ago] 3 h ago · nothing · 23 s
Docker engine
Docker 29.9.3
containerd 2.4.2-4~debian.13~trixie
live-restore on
Docker release ring0-20261115T045634Z
Host
Proxmox 9.2.27
Kernel (running) 7.0.14-28-pve
Kernel (next boot) 7.0.14-28-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 1m14s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 7 days ago · due 7.0.14-29-pve — household told 49 min ago: TONIGHT · phase good
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: removed 6 old kernel package(s): proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed; kept 7.0.14-26-pve, 7.0.14-28-pve; 6.3 GiB gained; 0 left-over path(s)] cleaned 3 h ago
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: the box reports no security-profile refresh (an agent older than release 3, a BYO host, or no refresh yet)] —
Debian 13.7
Felhom release ring0-20261115T045634Z
Pending 29
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.174.0
Guest
Guest Debian 13.7
Felhom release ring0-20261115T045634Z
Pending 0
Restart needed 1
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 3 days ago · 272.7 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 92.7 Mbit/s · 4 days ago
demo-hp-bb76ea Demo HP
[hover: Ring 0: a demo box — it installs every new fix first] 0
[hover: nothing amber or red] fine
0.352.1 [hover: current (vouched 0.174.0)] agent 0.174.0
[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind
7.0.14-28-pve
[hover: 4 h ago · nothing · 30 s — last successful leg: 4 h ago] ok · 4 h ago
Ring and updates
Ring ring 0
→ normal
OS updates updates ON
switch off
Tunnel running
Controller 0.352.1
Agent [hover: current (vouched 0.174.0)] 0.174.0
Last OS leg [hover: last successful leg: 4 h ago] 4 h ago · nothing · 30 s
Docker engine
Docker 29.9.3
containerd 2.4.2-4~debian.13~trixie
live-restore on
Docker release ring0-20261115T040248Z
Host
Proxmox 9.2.27
Kernel (running) 7.0.14-28-pve
Kernel (next boot) 7.0.14-28-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 2m39s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 5 days ago · due 7.0.14-29-pve — household told 49 min ago: TONIGHT · phase good
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] —
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: the box reports no security-profile refresh (an agent older than release 3, a BYO host, or no refresh yet)] —
Debian 13.7
Felhom release ring0-20261115T040248Z
Pending 30
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.174.0
Guest
Guest Debian 13.7
Felhom release ring0-20261115T040248Z
Pending 0
Restart needed 1
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 3 days ago · 70.5 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 94.3 Mbit/s · 4 days ago
tester-1-d70be4 Tester 1
[hover: Ring 1: a normal box — it installs only approved releases] 1
[hover: nothing amber or red] fine
0.352.1 [hover: current (vouched 0.174.0)] agent 0.174.0
[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind
7.0.14-22-pve
[hover: 3 h ago · nothing · 16 s — last successful leg: 3 h ago] ok · 3 h ago
Ring and updates
Ring ring 1
→ demo
OS updates updates ON
switch off
Tunnel running
Controller 0.352.1
Agent [hover: current (vouched 0.174.0)] 0.174.0
Last OS leg [hover: last successful leg: 3 h ago] 3 h ago · nothing · 16 s
Docker engine
Docker 29.8.2
containerd 2.3.6-1~debian.13~trixie
live-restore on
Docker release —
Host
Proxmox 9.2.2
Kernel (running) 7.0.14-22-pve
Kernel (next boot) 7.0.14-22-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-22-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: ring 1: no kernel staged by a signed os_kernel_step healthy 1m17s after the agent started; the new kernel is the default] 7.0.14-22-pve applied 38 days ago · phase good
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] —
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: the box reports no security-profile refresh (an agent older than release 3, a BYO host, or no refresh yet)] —
Debian 13.7
Felhom release os-host-20261111-034654
Pending 101
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.174.0
Guest
Guest Debian 13.7
Felhom release os-guest-20261111-034654
Pending 6
Restart needed 0
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 3 days ago · 90.4 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 93.0 Mbit/s · 4 days ago
Details
Release ids, cancelled approvals, what ring 0 runs, the version floors, and every box's crash guard and root files.
Approved releases
The newest approved set of each layer. A [hover: Approved while a TEST wait override was active. It is cancelled when the hub starts without the override (`11` §5.3.1).] TEST approval is amber.
guest : os-guest-20261111-034654
272 packages · 2026-11-11 03:46 UTC · by auto
host : os-host-20261115-045718
607 packages · 2026-11-15 04:57 UTC · by auto
docker : os-docker-20261026-190351
6 packages · 2026-10-26 19:03 UTC · by operator
pve : os-pve-20261114-075712
121 packages · 2026-11-14 07:57 UTC · by operator
kernel : os-kernel-20261114-075716
2 packages · 2026-11-14 07:57 UTC · by operator
Approve now (guest + host)
An urgent fix only — normally the hub approves after 24 h and one night.
What ring 0 runs now
guest :
272 packages, first seen 2026-11-10 01:49 UTC
approved as os-guest-20261111-034654
host :
607 packages, first seen 2026-11-14 04:52 UTC
approved as os-host-20261115-045718
docker :
6 packages, first seen 2026-11-13 04:49 UTC
ready — see "Waiting for you"
pve :
121 packages, first seen 2026-11-15 04:58 UTC
demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set
kernel :
2 packages, first seen 2026-11-10 01:57 UTC
approved as os-kernel-20261114-075716
Version floors
Global controller floor: 0.352.1 · vouched agent: 0.174.0
[hover: A floor is the lowest controller version a box must run; the hub moves a box below it up.] What is a floor? · [hover: The agent version the operator checked and signed off for the fleet. Agents update only by a per-box signed job.] What is a vouched agent?
No per-customer floors: every box follows the global floor.
Crash guard and root files
Box ¦ kernel.panic ¦ Oops ¦ Crash restarts 24 h ¦ Crash guard ¦ [hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files ¦
Tester4-46836f ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] 0.173.0 ¦
demo-felhom-8363b5 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.174.0 ¦
demo-hp-bb76ea ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.174.0 ¦
tester-1-d70be4 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.174.0 ¦
Felhom Hub 0.182.0
@@ -0,0 +1,15 @@
# READS ONLY, on a box: what the agent did since a time ($1 = 'YYYY-MM-DD HH:MM:SS').
# The control is Proxmox's OWN access log (another channel than the agent's journal): calls by the agent's token
# that Proxmox answered, after the time.
set -u
T0="$1"; HM=$(date -d "$T0" +%H:%M:%S); D=$(date -d "$T0" +%d/%m/%Y)
echo "#### $(hostname) $(date +%F\ %T): since $T0"
echo "agent unit active since $(systemctl show felhom-agent -p ActiveEnterTimestamp --value), pid $(systemctl show felhom-agent -p MainPID --value), restarts $(systemctl show felhom-agent -p NRestarts --value)"
J=$(journalctl -u felhom-agent --since "$T0" --no-pager -o short-iso 2>&1 | grep -v "sudo\[")
echo "agent journal lines since: $(echo "$J" | wc -l)"
echo "trust / certificate lines: $(echo "$J" | grep -ciE "pvetrust|x509|certificate|tls: ")"; echo "$J" | grep -iE "pvetrust|x509|certificate|tls: " | cut -c1-360 | tail -8
echo "ERROR lines: $(echo "$J" | grep -c "level=ERROR")"; echo "$J" | grep "level=ERROR" | cut -c1-300 | tail -5
echo "Proxmox access log, the agent's token, after $HM:"
awk -v d="$D" -v hm="$HM" '$3 ~ /felhom-agent@pve/ { split($4,a,":"); day=substr(a[1],2); t=a[2]":"a[3]":"a[4]; if (day==d && t>=hm) { n++; c[$9]++; if (!first) first=$0; last=$0 } } END { printf " calls: %d; by status:", n; for (k in c) printf " %s=%d", k, c[k]; print ""; print " first: " substr(first,1,170); print " last: " substr(last,1,170) }' /var/log/pveproxy/access.log
echo "served now: $(echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256 2>/dev/null | cut -d= -f2)"
echo "#### end"
@@ -0,0 +1,23 @@
# Part 2 step 4, demo-hp ONLY: make the agent's CA copy wrong (another, self-made CA in its place), restart the agent,
# and watch. The copy is saved first and compared at the end. Changes: /etc/felhom-agent/pve-root-ca.pem (one file),
# one restart of felhom-agent. Nothing under /etc/pve is touched.
set -u
[ "$(hostname)" = demo-hp ] || { echo "REFUSED: this is not demo-hp"; exit 2; }
busy=$(ps -eo comm | grep -cE "^(vzdump|apt-get|dpkg|pct|proxmox-backup-)")
[ "$busy" = 0 ] || { echo "REFUSED: something is in flight"; exit 2; }
C=/etc/felhom-agent/pve-root-ca.pem; SAVE=/root/pve-root-ca.pem.saved-part2
W=$(mktemp -d)
cp -p "$C" "$SAVE"
echo "#### $(date +%F\ %T) the copy before: sha256 $(sha256sum < "$C" | cut -c1-16), $(stat -c '%A %U:%G' "$C"); saved at $SAVE"
openssl req -x509 -newkey rsa:2048 -nodes -days 3650 -subj "/CN=CC wrong CA for a test 2026-11-15" -addext "basicConstraints=critical,CA:TRUE" -keyout "$W/k.pem" -out "$W/wrong.pem" 2>/dev/null
cat "$W/wrong.pem" > "$C"
rm -rf "$W"
echo "the copy staged wrong: sha256 $(sha256sum < "$C" | cut -c1-16), subject $(openssl x509 -in "$C" -noout -subject)"
T0=$(date '+%F %T')
systemctl restart felhom-agent; echo "agent restart rc=$? at $T0"
for i in $(seq 1 18); do sleep 5; [ "$(sha256sum < "$C")" = "$(sha256sum < "$SAVE")" ] && break; done
echo "the copy $([ "$(sha256sum < "$C")" = "$(sha256sum < "$SAVE")" ] && echo "is the saved one again, by itself, after about $((i*5)) s" || echo "is STILL WRONG after 90 s")"
sleep 20
journalctl -u felhom-agent --since "$T0" --no-pager -o short-iso | grep -v "sudo\[" | grep -iE "pvetrust|x509|certificate|level=ERROR" | cut -c1-380 | tail -10
echo "T0=$T0"
echo "#### end"
@@ -0,0 +1,19 @@
# Part 2 step 3, on demo-hp ONLY: force a new Proxmox certificate the way the spike did (certpin-spike-2026-11-11),
# then watch the agent. Changes on the box: /etc/pve/local/pve-ssl.pem (+ key) is made new; pveproxy is reloaded.
set -u
[ "$(hostname)" = demo-hp ] || { echo "REFUSED: this is not demo-hp"; exit 2; }
busy=$(ps -eo comm | grep -cE "^(vzdump|apt-get|dpkg|pct|proxmox-backup-)" )
[ "$busy" = 0 ] || { echo "REFUSED: something is in flight ($busy)"; ps -eo comm | grep -E "^(vzdump|apt-get|dpkg|pct|proxmox-backup-)"; exit 2; }
served() { echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256 2>/dev/null | cut -d= -f2; }
echo "#### $(date +%F\ %T) round ${1:-1}: agent $(felhom-agent --version | awk '{print $2}'), unit active since $(systemctl show felhom-agent -p ActiveEnterTimestamp --value), pid $(systemctl show felhom-agent -p MainPID --value)"
echo "root CA before: $(openssl x509 -in /etc/pve/pve-root-ca.pem -noout -fingerprint -sha256 | cut -d= -f2)"
echo "served before: $(served)"
T0=$(date '+%F %T')
pvecm updatecerts --force 2>&1 | tail -3
echo "file after updatecerts: $(openssl x509 -in /etc/pve/local/pve-ssl.pem -noout -fingerprint -sha256 | cut -d= -f2)"
systemctl reload pveproxy; echo "pveproxy reload rc=$?"
for i in $(seq 1 12); do sleep 5; s=$(served); [ "$s" = "$(openssl x509 -in /etc/pve/local/pve-ssl.pem -noout -fingerprint -sha256 | cut -d= -f2)" ] && break; done
echo "served after: $s (after about $((i*5)) s)"
echo "root CA after: $(openssl x509 -in /etc/pve/pve-root-ca.pem -noout -fingerprint -sha256 | cut -d= -f2)"
echo "T0=$T0"
echo "#### end round"
@@ -0,0 +1,31 @@
# Tester 1 ONLY. clock.sh snap|back|right
# snap — READ: the clock, the time service, every container's id and creation time in guest 9201.
# back — stop the time service and set the clock 150 s BACK (the smaller harm: putting it right is then a
# step FORWARD, the direction a time service steps a late clock; nothing gets a stamp from the future
# while the clock is wrong, and no certificate or signed job ends early).
# right — start the time service and step the clock right.
set -u
grep -q '"tester-1' /etc/felhom-agent/agent.json 2>/dev/null || grep -q 'tester-1' /etc/felhom-agent/agent.json || { echo "REFUSED: this box is not Tester 1"; exit 2; }
snap() {
echo "## $(hostname) clock $(date '+%F %T.%N %Z' | cut -c1-27) epoch $(date +%s.%N | cut -c1-14); chrony $(systemctl is-active chrony); $(chronyc tracking 2>/dev/null | grep -E 'System time|Leap' | tr -s ' ' | tr '\n' ';')"
echo "## kernel boot_id $(cat /proc/sys/kernel/random/boot_id); btime $(awk '/^btime/ {print $2}' /proc/stat); uptime $(cut -d' ' -f1 /proc/uptime) s"
echo "## containers in 9201 (name id created):"
pct exec 9201 -- docker ps -a --format '{{.Names}} {{.ID}} {{.CreatedAt}} [{{.Status}}]' | sort | sed 's/^/ /'
echo "## count $(pct exec 9201 -- docker ps -q | wc -l) running"
}
case "${1:-}" in
snap) snap ;;
back)
busy=$(ps -eo comm | grep -cE "^(vzdump|apt-get|dpkg|proxmox-backup-)")
[ "$busy" = 0 ] || { echo "REFUSED: something is in flight"; exit 2; }
snap
systemctl stop chrony; echo "chrony stopped rc=$?"
date -s "@$(( $(date +%s) - 150 ))" >/dev/null; echo "clock set 150 s back rc=$? → $(date '+%F %T')"
;;
right)
systemctl start chrony; echo "chrony started rc=$?"
sleep 8; chronyc makestep; sleep 6
echo "clock now $(date '+%F %T'); $(chronyc tracking | grep -E 'System time|Leap' | tr -s ' ' | tr '\n' ';')"
;;
*) echo "usage: snap|back|right"; exit 2 ;;
esac
@@ -0,0 +1,10 @@
#### 2026-11-15 10:38:01 the copy before: sha256 eb0ce31aaa8d94ee, -rw-r--r-- root:root; saved at /root/pve-root-ca.pem.saved-part2
the copy staged wrong: sha256 47ea4b8f5ea6a13b, subject subject=CN=CC wrong CA for a test 2026-11-15
agent restart rc=0 at 2026-11-15 10:38:02
the copy is STILL WRONG after 90 s
2026-11-15T10:38:02+01:00 demo-hp felhom-agent[2251809]: time=2026-11-15T10:38:02.288+01:00 level=WARN msg="pvetrust: the CA copy was refreshed and proven — the running client uses it now" why=start mode=ca outcome=written
T0=2026-11-15 10:38:02
#### end
after a second start: certs 1; identical to the saved copy: yes; mode -rw-r--r-- root:root
2026-11-15T10:40:06+01:00 demo-hp felhom-agent[2259683]: time=2026-11-15T10:40:06.827+01:00 level=WARN msg="pvetrust: the CA copy was refreshed and proven — the running client uses it now" why=start mode=ca outcome=written
2026-11-15T10:40:06+01:00 demo-hp felhom-os-apply[2259713]: os-apply: PVE CA COPY the previous root is dropped from /etc/felhom-agent/pve-root-ca.pem (Proxmox serves a certificate of the new root)
@@ -0,0 +1,23 @@
#### 2026-11-15 10:35:05 round 1: agent 0.175.1, unit active since Sun 2026-11-15 10:18:18 CET, pid 2183005
root CA before: E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7
served before: 1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74
(re)generate node files
generate new node certificate
merge authorized SSH keys
file after updatecerts: AA:76:56:BD:A7:1C:3D:5D:1D:A5:2C:8F:5D:51:3F:F8:E3:B8:14:52:4E:90:95:9C:D8:E8:4E:78:0B:8C:15:45
pveproxy reload rc=0
served after: AA:76:56:BD:A7:1C:3D:5D:1D:A5:2C:8F:5D:51:3F:F8:E3:B8:14:52:4E:90:95:9C:D8:E8:4E:78:0B:8C:15:45 (after about 10 s)
root CA after: E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7
T0=2026-11-15 10:35:05
#### end round
#### demo-hp 2026-11-15 10:35:36: since 2026-11-15 10:35:16
agent unit active since Sun 2026-11-15 10:18:18 CET, pid 2183005, restarts 0
agent journal lines since: 4
trust / certificate lines: 0
ERROR lines: 0
Proxmox access log, the agent's token, after 10:35:16:
calls: 14; by status: 200=14
first: ::ffff:127.0.0.1 - felhom-agent@pve!agent [15/11/2026:10:35:20 +0100] "GET /api2/json/storage HTTP/1.1" 200 918
last: ::ffff:127.0.0.1 - felhom-agent@pve!agent [15/11/2026:10:35:33 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config HTTP/1.1" 200 673
served now: AA:76:56:BD:A7:1C:3D:5D:1D:A5:2C:8F:5D:51:3F:F8:E3:B8:14:52:4E:90:95:9C:D8:E8:4E:78:0B:8C:15:45
#### end
@@ -0,0 +1,37 @@
#### 2026-11-15 10:35:43 round 2: agent 0.175.1, unit active since Sun 2026-11-15 10:18:18 CET, pid 2183005
root CA before: E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7
served before: AA:76:56:BD:A7:1C:3D:5D:1D:A5:2C:8F:5D:51:3F:F8:E3:B8:14:52:4E:90:95:9C:D8:E8:4E:78:0B:8C:15:45
(re)generate node files
generate new node certificate
merge authorized SSH keys
file after updatecerts: EF:06:C8:01:9E:B2:34:82:97:DE:11:DA:8B:57:8E:56:0C:11:91:1F:17:C0:64:98:C9:1A:1F:C5:47:33:CE:E1
pveproxy reload rc=0
served after: EF:06:C8:01:9E:B2:34:82:97:DE:11:DA:8B:57:8E:56:0C:11:91:1F:17:C0:64:98:C9:1A:1F:C5:47:33:CE:E1 (after about 10 s)
root CA after: E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7
T0=2026-11-15 10:35:43
#### end round
#### demo-hp 2026-11-15 10:37:10: since 2026-11-14 20:35:44
agent unit active since Sun 2026-11-15 10:18:18 CET, pid 2183005, restarts 0
agent journal lines since: 6374
trust / certificate lines: 2
2026-11-15T10:18:18+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:18:18.654+01:00 level=INFO msg="pvetrust: the root files are older than this agent (no CA-copy step yet) — the pinned certificate stays; the next root-files update moves the box" why=start mode=fingerprint err="pve-ca-copy refused: {\"code\": \"R11\", \"reason\": \"unknown mode 'pv
2026-11-15T10:33:34+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:34.354+01:00 level=WARN msg="pvetrust: MOVED — the agent trusts Proxmox by the box's own root CA now; the pinned fingerprint was cleared AFTER a working call with the copy" why="config bundle installed" mode=fingerprint ca_file=/etc/felhom-agent/pve-root-ca.pem ca_not_after=2036
ERROR lines: 0
Proxmox access log, the agent's token, after 20:35:44:
calls: 0; by status:
first:
last:
served now: EF:06:C8:01:9E:B2:34:82:97:DE:11:DA:8B:57:8E:56:0C:11:91:1F:17:C0:64:98:C9:1A:1F:C5:47:33:CE:E1
#### end
## read again with the right time (the first read's time was computed wrong: 'date -d ... + 15 seconds')
#### demo-hp 2026-11-15 10:37:18: since 2026-11-15 10:35:58
agent unit active since Sun 2026-11-15 10:18:18 CET, pid 2183005, restarts 0
agent journal lines since: 8
trust / certificate lines: 0
ERROR lines: 0
Proxmox access log, the agent's token, after 10:35:58:
calls: 40; by status: 200=40
first: ::ffff:127.0.0.1 - felhom-agent@pve!agent [15/11/2026:10:36:00 +0100] "GET /api2/json/storage HTTP/1.1" 200 918
last: ::ffff:127.0.0.1 - felhom-agent@pve!agent [15/11/2026:10:37:03 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config HTTP/1.1" 200 673
served now: EF:06:C8:01:9E:B2:34:82:97:DE:11:DA:8B:57:8E:56:0C:11:91:1F:17:C0:64:98:C9:1A:1F:C5:47:33:CE:E1
#### end
@@ -0,0 +1,31 @@
#### demo-hp 2026-11-15 10:34:56 CET agent 0.175.1 uptime-of-unit Sun 2026-11-15 10:18:18 CET
config proxmox.tls: ca_file='/etc/felhom-agent/pve-root-ca.pem' fingerprint=EMPTY insecure_skip_verify=False
-rw-r--r-- root:root 2094 bytes 2026-11-15 10:33:34.311028892 +0100 /etc/felhom-agent/pve-root-ca.pem
drwxr-xr-x root:root 4096 bytes 2026-11-15 10:33:34.317623237 +0100 /etc/felhom-agent
-rw------- felhom-agent:felhom-agent 2455 bytes 2026-11-15 10:33:34.34 /var/lib/felhom-agent/agent.json.pre-pve-trust
(absent) /var/lib/felhom-agent/agent.json.write-in-flight
CA copy /etc/felhom-agent/pve-root-ca.pem: 1 certificate(s); first: subject=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA notAfter=Aug 17 15:44:42 2036 GMT sha256 Fingerprint=E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7
same as /etc/pve/pve-root-ca.pem: yes
agent user can read it: yes; others can read: r
served certificate (127.0.0.1:8006): sha256 Fingerprint=1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74 notAfter=Nov 9 13:59:54 2028 GMT
/etc/pve/local/pve-ssl.pem: sha256 Fingerprint=1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74 notBefore=Nov 10 13:59:54 2026 GMT
## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min)
2026-11-15T10:18:15+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T10:18:15.625+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=90d84d793f66c18d op=agent_update key_id=felhom-op-1 nonce=9e1c24967ec755838aeda82ce2520f14
2026-11-15T10:18:15+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T10:18:15.872+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.175.1 version=0.175.1
2026-11-15T10:18:16+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T10:18:16.400+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.175.1 report="{\"agent_update\": {\"sha256\": \"38b1e2350de90a7940c16f8cd2a21d83fcd535410d1017a7df0fee14f0904612\", \"version\": \"0.175.1\", \"wrapper\": \"Running timer as unit: run-p2182880-i23144936.timer\\nWill run service as u
2026-11-15T10:18:16+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T10:18:16.401+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=90d84d793f66c18d op=agent_update
2026-11-15T10:18:18+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:18:18.654+01:00 level=INFO msg="pvetrust: the root files are older than this agent (no CA-copy step yet) — the pinned certificate stays; the next root-files update moves the box" why=start mode=fingerprint err="pve-ca-copy refused: {\"code\": \"R11\", \"reason\": \"unknown mode 'pve-ca-copy'\"}"
2026-11-15T10:18:20+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:18:20.815+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.175.1 prev=0.174.0 dwell=1m0s
2026-11-15T10:19:20+01:00 demo-hp felhom-selfupdate-guarded[2187989]: committed (pending cleared, .prev retained)
2026-11-15T10:19:20+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:19:20.843+01:00 level=WARN msg="selfupdate: update committed" version=0.175.1 wrapper=""
2026-11-15T10:19:20+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:19:20.962+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.175.1 removed=1
2026-11-15T10:19:20+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:19:20.962+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=1 kept=felhom-agent-0.175.1
2026-11-15T10:33:31+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:31.703+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=9b72aa335a322ba6 op=agent_config_update key_id=felhom-op-1 nonce=c93c80251a155095f98d648bf9b16f51
2026-11-15T10:33:31+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:31.813+01:00 level=INFO msg="osupdate: config bundle downloaded; handing it to the root wrapper" op=agent_config_update agent_version=0.175.1 sha256=07ebf86362ed9d44 duration_ms=110
2026-11-15T10:33:33+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:33.036+01:00 level=WARN msg="osupdate: config bundle INSTALLED" op=agent_config_update agent_version=0.175.1 bundle="{\"agent_version\": \"0.175.1\", \"authority\": \"signed\", \"kept\": [\"/etc/felhom/crash-guard.conf\"], \"prev_dir\": \"/var/lib/felhom-os-apply/bundle-prev/20261115T093332Z-before-0.175.1\", \"removed\": [], \"same\": 30, \"self_ch
2026-11-15T10:33:34+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:34.180+01:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=77 total=77 degraded=""
2026-11-15T10:33:34+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:34.180+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=9b72aa335a322ba6 op=agent_config_update
2026-11-15T10:33:34+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:34.293+01:00 level=INFO msg="osupdate: profile refresh not needed" mode=profile-refresh why="config bundle installed" kernel=7.0.14-28-pve reason="the profiles were refreshed on this kernel (7.0.14-28-pve) at 2026-11-14T11:33:18Z"
2026-11-15T10:33:34+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:34.354+01:00 level=WARN msg="pvetrust: MOVED — the agent trusts Proxmox by the box's own root CA now; the pinned fingerprint was cleared AFTER a working call with the copy" why="config bundle installed" mode=fingerprint ca_file=/etc/felhom-agent/pve-root-ca.pem ca_not_after=2036-08-17T15:44:42Z
## one control: the agent's newest successful host report and newest Proxmox call error
#### end
@@ -0,0 +1,45 @@
== MUTATION: older root files are named as a trust problem again
RED on the mutant, GREEN on the real file
--- FAIL: TestRootStepOlderThanTheAgent_IsNotATrustProblem (0.00s)
pvetrust_test.go:507: a pinned box with older root files reports a trust problem: &{Mode:fingerprint Problem:ca_copy_unavailable Detail:pve-ca-copy refused: {"code": "R11", "reason": "unknown mode 'pve-ca-copy'"} CANotAfter: CADay
== MUTATION: an UNPINNED box with no copy step is silent too
RED on the mutant, GREEN on the real file
--- FAIL: TestRootStepOlderThanTheAgent_IsNotATrustProblem (0.00s)
pvetrust_test.go:518: an unpinned box with no copy step must name the problem: &{Mode:system Problem: Detail: CANotAfter: CADaysLeft:<nil> MovedAt: CheckedAt:2026-11-15T09:05:53Z}
== MUTATION: every refusal is silent
*** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0)
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/pvetrust [build failed]
3 mutations, 1 not a red-proof
== MUTATION: every refusal is silent (redone: the first mutant did not compile)
RED on the mutant, GREEN on the real file
--- FAIL: TestRootStepRefuses_NothingChanges (0.00s)
pvetrust_test.go:374: status = &{Mode:fingerprint Problem: Detail: CANotAfter: CADaysLeft:<nil> MovedAt: CheckedAt:2026-11-15T09:06:04Z}
--- FAIL: TestRootStepOlderThanTheAgent_IsNotATrustProblem (0.00s)
pvetrust_test.go:526: a real refusal lost its name: &{Mode:fingerprint Problem: Detail: CANotAfter: CADaysLeft:<nil> MovedAt: CheckedAt:2026-11-15T09:06:04Z}
1 mutations, 0 not a red-proof
## after the commit's security review: the quiet branch never hides a failed pin
== MUTATION: a run started by a failed call takes the quiet branch
RED on the mutant, GREEN on the real file
--- FAIL: TestRootStepOlderThanTheAgent_NeverHidesAFailedPin (0.00s)
pvetrust_test.go:543: a run started by a failed pin reports no problem: &{Mode:fingerprint Problem: Detail: CANotAfter: CADaysLeft:<nil> MovedAt: CheckedAt:2026-11-15T09:07:36Z}
== MUTATION: the timer clears a recorded pin failure
*** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0)
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/pvetrust [build failed]
internal/pvetrust/pvetrust.go:251:3: declared and not used: recorded
== MUTATION: any refusal that says unknown mode counts, without the code
*** NOT A RED-PROOF *** (mutant rc=0 built=True real rc=0)
3 mutations, 2 not a red-proof
== MUTATION: the timer clears a recorded pin failure (redone: the first mutant did not compile)
RED on the mutant, GREEN on the real file
--- FAIL: TestRootStepOlderThanTheAgent_NeverHidesAFailedPin (0.00s)
pvetrust_test.go:564: the timer cleared a recorded pin failure: &{Mode:fingerprint Problem: Detail: CANotAfter: CADaysLeft:<nil> MovedAt: CheckedAt:2026-11-15T09:07:52Z}
== MUTATION: any refusal that says unknown mode counts, without the code (redone: the test had no such case; one was added)
RED on the mutant, GREEN on the real file
--- FAIL: TestRootStepOlderThanTheAgent_IsNotATrustProblem (0.00s)
pvetrust_test.go:536: a refusal with another code was read as older root files: &{Mode:fingerprint Problem: Detail: CANotAfter: CADaysLeft:<nil> MovedAt: CheckedAt:2026-11-15T09:07:54Z}
2 mutations, 0 not a red-proof
@@ -0,0 +1,265 @@
System — Felhom Hub
Felhom Hub
Dashboard
Customers
Apps
Hosts
System
Offsite
Configuration
System — versions and OS updates
Needs attention
One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3).
[hover: worth a look] look
Tester4-46836f Tester4
[hover: cannot do a one-shot boot: the box does not boot UEFI; /boot/efi is not a mounted vfat ESP (not mounted)] kernel step: — · the host needs a restart since 2026-11-14 · [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] root files behind the vouched agent's · [hover: 1 minor release behind — sign an agent_update for this box] agent behind: 0.173.0 → 0.174.0 (since 2026-11-14) · [hover: the box could not read these values (agent older than v0.142.0, or the guest is down) — never a guess] could not read: default kernel
[hover: worth a look] look
demo-felhom-8363b5 Demo Ügyfél
[hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update] root files behind the vouched agent's
[hover: worth a look] look
demo-hp-bb76ea Demo HP
[hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-15 09:51 UTC)] root files behind the vouched agent's
[hover: worth a look] look
tester-1-d70be4 Tester 1
[hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-15 09:51 UTC)] root files behind the vouched agent's
Waiting for you
Update sets that only you approve: the kernel, the Docker engine and the Proxmox packages. Guest and host Debian updates approve themselves after 24 h and one night on the [hover: Ring 0 = the demo boxes. They install every new fix first; the other boxes (ring 1) install only what is approved.] ring-0 boxes.
Docker engine 29.9.3-1
Ran on every ring-0 box — demo-felhom-8363b5: 2 healthy night(s), demo-hp-bb76ea: 2 healthy night(s). The memory-kill check passed.
first seen 2 days ago (2026-11-13 04:49 UTC). Approving installs nothing by itself: a ring-1 box takes it only through a signed operator job.
Approve Docker set
Proxmox packages 9.2.27 — still being tested: demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set
Boxes
Click a box to see every value it reports, and its ring and update switches. "unknown": the box could not read the value — never a guess.
Box Ring Health Controller · agent OS updates Kernel [hover: The newest OS update run] Last night
Tester4-46836f Tester4
[hover: Ring 1: a normal box — it installs only approved releases] 1
[hover: worth a look] look
0.352.1 [hover: 1 minor release behind — sign an agent_update for this box] agent 0.173.0 → 0.174.0 (since 2026-11-14)
[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind
7.0.2-6-pve
[hover: 23 h ago · applied · 93 s — last successful leg: 23 h ago] ok · 23 h ago
Ring and updates
Ring ring 1
→ demo
OS updates updates ON
switch off
Tunnel [hover: no cloudflared container in the guest (agent state: absent)] none set up
Controller 0.352.1
Agent [hover: 1 minor release behind — sign an agent_update for this box] 0.173.0 → 0.174.0 (since 2026-11-14)
Last OS leg [hover: last successful leg: 23 h ago] 23 h ago · applied · 93 s
Docker engine
Docker 29.9.2
containerd 2.4.2-3~debian.13~trixie
live-restore on
Docker release —
Host
Proxmox 9.2.2
Kernel (running) 7.0.2-6-pve
Kernel (next boot) 7.0.2-6-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: cannot do a one-shot boot: the box does not boot UEFI; /boot/efi is not a mounted vfat ESP (not mounted)] —
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] —
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: the box reports no security-profile refresh (an agent older than release 3, a BYO host, or no refresh yet)] —
Debian 13.7
Felhom release os-host-20261111-034654
Pending 100
Not covered 0
Held none
Reboot needed since 2026-11-14
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] 0.173.0
Guest
Guest Debian 13.7
Felhom release os-guest-20261111-034654
Pending 6
Restart needed 7
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 23 h ago · 78.4 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] not yet
demo-felhom-8363b5 Demo Ügyfél
[hover: Ring 0: a demo box — it installs every new fix first] 0
[hover: worth a look] look
0.352.1 [hover: current (vouched 0.174.0)] agent 0.175.1
[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind
7.0.14-28-pve
[hover: 5 h ago · nothing · 23 s — last successful leg: 5 h ago] ok · 5 h ago
Ring and updates
Ring ring 0
→ normal
OS updates updates ON
switch off
Tunnel running
Controller 0.352.1
Agent [hover: current (vouched 0.174.0)] 0.175.1
Last OS leg [hover: last successful leg: 5 h ago] 5 h ago · nothing · 23 s
Docker engine
Docker 29.9.3
containerd 2.4.2-4~debian.13~trixie
live-restore on
Docker release ring0-20261115T045634Z
Host
Proxmox 9.2.27
Kernel (running) 7.0.14-28-pve
Kernel (next boot) 7.0.14-28-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 1m14s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 7 days ago · due 7.0.14-29-pve — household told 2 h ago: TONIGHT · phase good
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: removed 6 old kernel package(s): proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed; kept 7.0.14-26-pve, 7.0.14-28-pve; 6.3 GiB gained; 0 left-over path(s)] cleaned 5 h ago
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: not_needed] 7.0.14-28-pve · 22 h ago
Debian 13.7
Felhom release ring0-20261115T045634Z
Pending 29
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update] 0.175.1
Guest
Guest Debian 13.7
Felhom release ring0-20261115T045634Z
Pending 0
Restart needed 1
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 4 days ago · 272.7 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 92.7 Mbit/s · 5 days ago
demo-hp-bb76ea Demo HP
[hover: Ring 0: a demo box — it installs every new fix first] 0
[hover: worth a look] look
0.352.1 [hover: current (vouched 0.174.0)] agent 0.175.1
[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind
7.0.14-28-pve
[hover: 6 h ago · nothing · 30 s — last successful leg: 6 h ago] ok · 6 h ago
Ring and updates
Ring ring 0
→ normal
OS updates updates ON
switch off
Tunnel running
Controller 0.352.1
Agent [hover: current (vouched 0.174.0)] 0.175.1
Last OS leg [hover: last successful leg: 6 h ago] 6 h ago · nothing · 30 s
Docker engine
Docker 29.9.3
containerd 2.4.2-4~debian.13~trixie
live-restore on
Docker release ring0-20261115T040248Z
Host
Proxmox 9.2.27
Kernel (running) 7.0.14-28-pve
Kernel (next boot) 7.0.14-28-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 2m39s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 5 days ago · due 7.0.14-29-pve — household told 2 h ago: TONIGHT · phase good
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] —
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: not_needed] 7.0.14-28-pve · 22 h ago
Debian 13.7
Felhom release ring0-20261115T040248Z
Pending 30
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-15 09:51 UTC)] 0.175.1
Guest
Guest Debian 13.7
Felhom release ring0-20261115T040248Z
Pending 0
Restart needed 1
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 4 days ago · 70.5 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 94.3 Mbit/s · 4 days ago
tester-1-d70be4 Tester 1
[hover: Ring 1: a normal box — it installs only approved releases] 1
[hover: worth a look] look
0.352.1 [hover: current (vouched 0.174.0)] agent 0.175.1
[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind
7.0.14-22-pve
[hover: 5 h ago · nothing · 16 s — last successful leg: 5 h ago] ok · 5 h ago
Ring and updates
Ring ring 1
→ demo
OS updates updates ON
switch off
Tunnel running
Controller 0.352.1
Agent [hover: current (vouched 0.174.0)] 0.175.1
Last OS leg [hover: last successful leg: 5 h ago] 5 h ago · nothing · 16 s
Docker engine
Docker 29.8.2
containerd 2.3.6-1~debian.13~trixie
live-restore on
Docker release —
Host
Proxmox 9.2.2
Kernel (running) 7.0.14-22-pve
Kernel (next boot) 7.0.14-22-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-22-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: ring 1: no kernel staged by a signed os_kernel_step healthy 1m17s after the agent started; the new kernel is the default] 7.0.14-22-pve applied 38 days ago · phase good
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] —
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: not_needed] 7.0.14-22-pve · 22 h ago
Debian 13.7
Felhom release os-host-20261111-034654
Pending 101
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-15 09:51 UTC)] 0.175.1
Guest
Guest Debian 13.7
Felhom release os-guest-20261111-034654
Pending 6
Restart needed 0
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 4 days ago · 90.4 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 93.0 Mbit/s · 4 days ago
Details
Release ids, cancelled approvals, what ring 0 runs, the version floors, and every box's crash guard and root files.
Approved releases
The newest approved set of each layer. A [hover: Approved while a TEST wait override was active. It is cancelled when the hub starts without the override (`11` §5.3.1).] TEST approval is amber.
guest : os-guest-20261111-034654
272 packages · 2026-11-11 03:46 UTC · by auto
host : os-host-20261115-045718
607 packages · 2026-11-15 04:57 UTC · by auto
docker : os-docker-20261026-190351
6 packages · 2026-10-26 19:03 UTC · by operator
pve : os-pve-20261114-075712
121 packages · 2026-11-14 07:57 UTC · by operator
kernel : os-kernel-20261114-075716
2 packages · 2026-11-14 07:57 UTC · by operator
Approve now (guest + host)
An urgent fix only — normally the hub approves after 24 h and one night.
What ring 0 runs now
guest :
272 packages, first seen 2026-11-10 01:49 UTC
approved as os-guest-20261111-034654
host :
607 packages, first seen 2026-11-14 04:52 UTC
approved as os-host-20261115-045718
docker :
6 packages, first seen 2026-11-13 04:49 UTC
ready — see "Waiting for you"
pve :
121 packages, first seen 2026-11-15 04:58 UTC
demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set
kernel :
2 packages, first seen 2026-11-10 01:57 UTC
approved as os-kernel-20261114-075716
Version floors
Global controller floor: 0.352.1 · vouched agent: 0.174.0
[hover: A floor is the lowest controller version a box must run; the hub moves a box below it up.] What is a floor? · [hover: The agent version the operator checked and signed off for the fleet. Agents update only by a per-box signed job.] What is a vouched agent?
No per-customer floors: every box follows the global floor.
Crash guard and root files
Box ¦ kernel.panic ¦ Oops ¦ Crash restarts 24 h ¦ Crash guard ¦ [hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files ¦
Tester4-46836f ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] 0.173.0 ¦
demo-felhom-8363b5 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update] 0.175.1 ¦
demo-hp-bb76ea ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-15 09:51 UTC)] 0.175.1 ¦
tester-1-d70be4 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-15 09:51 UTC)] 0.175.1 ¦
Felhom Hub 0.182.0
@@ -0,0 +1,17 @@
#### felhom 2026-11-15 10:04:46 CET agent 0.175.0 uptime-of-unit Sun 2026-11-15 10:04:33 CET
config proxmox.tls: ca_file='' fingerprint=set (BD:EE:9F:C4…) insecure_skip_verify=False
(absent) /etc/felhom-agent/pve-root-ca.pem
drwxr-xr-x root:root 4096 bytes 2026-10-04 21:40:41.128703040 +0200 /etc/felhom-agent
(absent) /var/lib/felhom-agent/agent.json.pre-pve-trust
(absent) /var/lib/felhom-agent/agent.json.write-in-flight
served certificate (127.0.0.1:8006): sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notAfter=Oct 2 19:38:05 2028 GMT
/etc/pve/local/pve-ssl.pem: sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notBefore=Oct 3 19:38:05 2026 GMT
## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min)
2026-11-15T10:04:30+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:30.371+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=b5c45465f233eb81 op=agent_update key_id=felhom-op-1 nonce=d5fd87f441408954947c67b4b44ab6b2
2026-11-15T10:04:30+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:30.619+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.175.0 version=0.175.0
2026-11-15T10:04:31+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:31.020+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.175.0 report="{\"agent_update\": {\"sha256\": \"d1a30ed51f6663d96369ad3e88c221194af92773addf4fbb48852ee58be4b19a\", \"version\": \"0.175.0\", \"wrapper\": \"Running timer as unit: run-p1546906-i1549255.timer\\nWill run service as unit:
2026-11-15T10:04:31+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:31.020+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=b5c45465f233eb81 op=agent_update
2026-11-15T10:04:33+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:04:33.269+01:00 level=WARN msg="pvetrust: the root step gave no CA copy — the trust stays as it is" why=start mode=fingerprint err="pve-ca-copy refused: {\"code\": \"R11\", \"reason\": \"unknown mode 'pve-ca-copy'\"}"
2026-11-15T10:04:34+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:04:34.922+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.175.0 prev=0.174.0 dwell=1m0s
## one control: the agent's newest successful host report and newest Proxmox call error
#### end
@@ -0,0 +1,21 @@
#### felhom 2026-11-15 10:09:02 CET agent 0.175.0 uptime-of-unit Sun 2026-11-15 10:04:33 CET
config proxmox.tls: ca_file='' fingerprint=set (BD:EE:9F:C4…) insecure_skip_verify=False
(absent) /etc/felhom-agent/pve-root-ca.pem
drwxr-xr-x root:root 4096 bytes 2026-10-04 21:40:41.128703040 +0200 /etc/felhom-agent
(absent) /var/lib/felhom-agent/agent.json.pre-pve-trust
(absent) /var/lib/felhom-agent/agent.json.write-in-flight
served certificate (127.0.0.1:8006): sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notAfter=Oct 2 19:38:05 2028 GMT
/etc/pve/local/pve-ssl.pem: sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notBefore=Oct 3 19:38:05 2026 GMT
## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min)
2026-11-15T10:04:30+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:30.371+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=b5c45465f233eb81 op=agent_update key_id=felhom-op-1 nonce=d5fd87f441408954947c67b4b44ab6b2
2026-11-15T10:04:30+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:30.619+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.175.0 version=0.175.0
2026-11-15T10:04:31+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:31.020+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.175.0 report="{\"agent_update\": {\"sha256\": \"d1a30ed51f6663d96369ad3e88c221194af92773addf4fbb48852ee58be4b19a\", \"version\": \"0.175.0\", \"wrapper\": \"Running timer as unit: run-p1546906-i1549255.timer\\nWill run service as unit:
2026-11-15T10:04:31+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:31.020+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=b5c45465f233eb81 op=agent_update
2026-11-15T10:04:33+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:04:33.269+01:00 level=WARN msg="pvetrust: the root step gave no CA copy — the trust stays as it is" why=start mode=fingerprint err="pve-ca-copy refused: {\"code\": \"R11\", \"reason\": \"unknown mode 'pve-ca-copy'\"}"
2026-11-15T10:04:34+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:04:34.922+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.175.0 prev=0.174.0 dwell=1m0s
2026-11-15T10:05:34+01:00 felhom felhom-selfupdate-guarded[1549546]: committed (pending cleared, .prev retained)
2026-11-15T10:05:34+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:05:34.955+01:00 level=WARN msg="selfupdate: update committed" version=0.175.0 wrapper=""
2026-11-15T10:05:35+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:05:35.150+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.175.0 removed=1
2026-11-15T10:05:35+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:05:35.150+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=1 kept=felhom-agent-0.175.0
## one control: the agent's newest successful host report and newest Proxmox call error
#### end
@@ -0,0 +1,11 @@
#### felhom 2026-11-15 09:51:08 CET agent 0.174.0 uptime-of-unit Sat 2026-11-14 15:19:22 CET
config proxmox.tls: ca_file='' fingerprint=set (BD:EE:9F:C4…) insecure_skip_verify=False
(absent) /etc/felhom-agent/pve-root-ca.pem
drwxr-xr-x root:root 4096 bytes 2026-10-04 21:40:41.128703040 +0200 /etc/felhom-agent
(absent) /var/lib/felhom-agent/agent.json.pre-pve-trust
(absent) /var/lib/felhom-agent/agent.json.write-in-flight
served certificate (127.0.0.1:8006): sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notAfter=Oct 2 19:38:05 2028 GMT
/etc/pve/local/pve-ssl.pem: sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notBefore=Oct 3 19:38:05 2026 GMT
## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min)
## one control: the agent's newest successful host report and newest Proxmox call error
#### end
@@ -0,0 +1,14 @@
## felhom clock 2026-11-15 10:49:52.0408764 epoch 1794736192.042; chrony inactive;
## kernel boot_id 7a721957-7e95-476c-be40-52d0dafc7284; btime 1794665801; uptime 70390.82 s
## containers in 9201 (name id created):
bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 5 hours (healthy)]
bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 5 hours (healthy)]
cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 20 hours (healthy)]
felhom-controller 2a7970a2c8a7 2026-11-14 14:19:33 +0000 UTC [Up 20 hours (healthy)]
filebrowser 95f38571abee 2026-11-14 14:19:57 +0000 UTC [Up 19 hours (healthy)]
paperless-postgres c7807ae06022 2026-11-15 09:48:39 +0000 UTC [Up About a minute (healthy)]
paperless-redis 56d085b17130 2026-11-15 09:48:39 +0000 UTC [Up About a minute (healthy)]
paperless-webserver 71aee1583019 2026-11-15 09:48:39 +0000 UTC [Up About a minute (healthy)]
privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 5 hours (healthy)]
traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 20 hours]
## count 10 running
@@ -0,0 +1,14 @@
## felhom clock 2026-11-15 11:06:49.5647785 epoch 1794737209.566; chrony active; System time : 0.001194180 seconds slow of NTP time;Leap status : Normal;
## kernel boot_id 7a721957-7e95-476c-be40-52d0dafc7284; btime 1794665951; uptime 71258.03 s
## containers in 9201 (name id created):
bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 5 hours (healthy)]
bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 5 hours (healthy)]
cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 20 hours (healthy)]
felhom-controller 2a7970a2c8a7 2026-11-14 14:19:33 +0000 UTC [Up 20 hours (healthy)]
filebrowser 95f38571abee 2026-11-14 14:19:57 +0000 UTC [Up 20 hours (healthy)]
paperless-postgres 31b2490aa677 2026-11-15 10:05:37 +0000 UTC [Up About a minute (healthy)]
paperless-redis df1c44607a9b 2026-11-15 10:05:37 +0000 UTC [Up About a minute (healthy)]
paperless-webserver 1f2a0d7e4f7b 2026-11-15 10:05:37 +0000 UTC [Up About a minute (healthy)]
privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 5 hours (healthy)]
traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 20 hours]
## count 10 running
@@ -0,0 +1,16 @@
## felhom clock 2026-11-15 10:50:27.2531193 epoch 1794736227.254; chrony active; System time : 0.000242970 seconds slow of NTP time;Leap status : Normal;
## kernel boot_id 7a721957-7e95-476c-be40-52d0dafc7284; btime 1794665951; uptime 70275.75 s
## containers in 9201 (name id created):
bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 5 hours (healthy)]
bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 5 hours (healthy)]
cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 20 hours (healthy)]
felhom-controller 2a7970a2c8a7 2026-11-14 14:19:33 +0000 UTC [Up 20 hours (healthy)]
filebrowser 95f38571abee 2026-11-14 14:19:57 +0000 UTC [Up 20 hours (healthy)]
paperless-postgres e0d945977ebc 2026-11-15 04:50:09 +0000 UTC [Up 5 hours (healthy)]
paperless-redis 6188dea5adbc 2026-11-15 04:50:09 +0000 UTC [Up 5 hours (healthy)]
paperless-webserver cc93a8574d66 2026-11-15 04:50:10 +0000 UTC [Up 5 hours (healthy)]
privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 5 hours (healthy)]
traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 20 hours]
## count 10 running
chrony stopped rc=0
clock set 150 s back rc=0 → 2026-11-15 10:47:59
@@ -0,0 +1,4 @@
chrony started rc=0
200 OK
clock now 2026-11-15 11:05:36; System time : 0.000000000 seconds slow of NTP time;Leap status : Normal;
Tester 1 minus DooPlex: +0.073 s
@@ -0,0 +1,31 @@
#### felhom 2026-11-15 10:20:19 CET agent 0.175.0 uptime-of-unit Sun 2026-11-15 10:04:33 CET
config proxmox.tls: ca_file='/etc/felhom-agent/pve-root-ca.pem' fingerprint=EMPTY insecure_skip_verify=False
-rw-r--r-- root:root 2094 bytes 2026-11-15 10:19:47.714398283 +0100 /etc/felhom-agent/pve-root-ca.pem
drwxr-xr-x root:root 4096 bytes 2026-11-15 10:19:47.716626910 +0100 /etc/felhom-agent
-rw------- felhom-agent:felhom-agent 2451 bytes 2026-11-15 10:19:47.73 /var/lib/felhom-agent/agent.json.pre-pve-trust
(absent) /var/lib/felhom-agent/agent.json.write-in-flight
CA copy /etc/felhom-agent/pve-root-ca.pem: 1 certificate(s); first: subject=CN=Proxmox Virtual Environment, OU=f9c69a56-050d-43dd-87c8-d596f6a79d19, O=PVE Cluster Manager CA notAfter=Sep 30 19:38:05 2036 GMT sha256 Fingerprint=4E:46:8D:BC:45:A2:2C:CD:AB:EF:82:C0:A4:B1:4D:6C:5C:DA:5A:7F:6F:D7:3B:E9:B7:50:CA:ED:C4:0C:80:CF
same as /etc/pve/pve-root-ca.pem: yes
agent user can read it: yes; others can read: r
served certificate (127.0.0.1:8006): sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notAfter=Oct 2 19:38:05 2028 GMT
/etc/pve/local/pve-ssl.pem: sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notBefore=Oct 3 19:38:05 2026 GMT
## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min)
2026-11-15T10:04:30+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:30.371+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=b5c45465f233eb81 op=agent_update key_id=felhom-op-1 nonce=d5fd87f441408954947c67b4b44ab6b2
2026-11-15T10:04:30+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:30.619+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.175.0 version=0.175.0
2026-11-15T10:04:31+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:31.020+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.175.0 report="{\"agent_update\": {\"sha256\": \"d1a30ed51f6663d96369ad3e88c221194af92773addf4fbb48852ee58be4b19a\", \"version\": \"0.175.0\", \"wrapper\": \"Running timer as unit: run-p1546906-i1549255.timer\\nWill run service as unit:
2026-11-15T10:04:31+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:31.020+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=b5c45465f233eb81 op=agent_update
2026-11-15T10:04:33+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:04:33.269+01:00 level=WARN msg="pvetrust: the root step gave no CA copy — the trust stays as it is" why=start mode=fingerprint err="pve-ca-copy refused: {\"code\": \"R11\", \"reason\": \"unknown mode 'pve-ca-copy'\"}"
2026-11-15T10:04:34+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:04:34.922+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.175.0 prev=0.174.0 dwell=1m0s
2026-11-15T10:05:34+01:00 felhom felhom-selfupdate-guarded[1549546]: committed (pending cleared, .prev retained)
2026-11-15T10:05:34+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:05:34.955+01:00 level=WARN msg="selfupdate: update committed" version=0.175.0 wrapper=""
2026-11-15T10:05:35+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:05:35.150+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.175.0 removed=1
2026-11-15T10:05:35+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:05:35.150+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=1 kept=felhom-agent-0.175.0
2026-11-15T10:19:45+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:45.500+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=0c91a3e29ac9ba27 op=agent_config_update key_id=felhom-op-1 nonce=cb391943ab870fd924c8b90ef98b2065
2026-11-15T10:19:45+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:45.543+01:00 level=INFO msg="osupdate: config bundle downloaded; handing it to the root wrapper" op=agent_config_update agent_version=0.175.0 sha256=3b4a8b78155fc438 duration_ms=42
2026-11-15T10:19:46+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:46.553+01:00 level=WARN msg="osupdate: config bundle INSTALLED" op=agent_config_update agent_version=0.175.0 bundle="{\"agent_version\": \"0.175.0\", \"authority\": \"signed\", \"kept\": [\"/etc/felhom/crash-guard.conf\"], \"prev_dir\": \"/var/lib/felhom-os-apply/bundle-prev/20261115T091945Z-before-0.175.0\", \"removed\": [], \"same\": 30, \"self_che
2026-11-15T10:19:47+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:47.590+01:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=77 total=77 degraded=""
2026-11-15T10:19:47+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:47.590+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=0c91a3e29ac9ba27 op=agent_config_update
2026-11-15T10:19:47+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:47.685+01:00 level=INFO msg="osupdate: profile refresh not needed" mode=profile-refresh why="config bundle installed" kernel=7.0.14-22-pve reason="the profiles were refreshed on this kernel (7.0.14-22-pve) at 2026-11-14T11:30:11Z"
2026-11-15T10:19:47+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:47.744+01:00 level=WARN msg="pvetrust: MOVED — the agent trusts Proxmox by the box's own root CA now; the pinned fingerprint was cleared AFTER a working call with the copy" why="config bundle installed" mode=fingerprint ca_file=/etc/felhom-agent/pve-root-ca.pem ca_not_after=2036-09-30T19:38:05Z
## one control: the agent's newest successful host report and newest Proxmox call error
#### end
@@ -0,0 +1,23 @@
# Part 2 — READS ONLY: the agent's version and its trust in Proxmox, on one box. Never prints a token or a key.
set -u
echo "#### $(hostname) $(date +%F\ %T\ %Z) agent $(felhom-agent --version 2>&1 | awk '{print $2}') uptime-of-unit $(systemctl show felhom-agent -p ActiveEnterTimestamp --value)"
python3 - <<'PY'
import json
d = json.load(open("/etc/felhom-agent/agent.json"))
t = d.get("proxmox", {}).get("tls", {})
fp = t.get("fingerprint", "")
print("config proxmox.tls: ca_file=%r fingerprint=%s insecure_skip_verify=%r" % (t.get("ca_file", ""), ("set (%s…)" % fp[:11]) if fp else "EMPTY", t.get("insecure_skip_verify")))
PY
for f in /etc/felhom-agent/pve-root-ca.pem /etc/felhom-agent /var/lib/felhom-agent/agent.json.pre-pve-trust /var/lib/felhom-agent/agent.json.write-in-flight; do
[ -e "$f" ] && echo " $(stat -c '%A %U:%G %s bytes %y' "$f" | cut -c1-70) $f" || echo " (absent) $f"; done
CA=$(python3 -c 'import json;print(json.load(open("/etc/felhom-agent/agent.json")).get("proxmox",{}).get("tls",{}).get("ca_file",""))')
for c in "$CA" /etc/felhom-agent/pve-root-ca.pem; do
[ -n "$c" ] && [ -e "$c" ] && { echo "CA copy $c: $(grep -c 'BEGIN CERTIFICATE' "$c") certificate(s); first: $(openssl x509 -in "$c" -noout -subject -enddate -fingerprint -sha256 2>/dev/null | tr '\n' ' ' | cut -c1-260)"; echo " same as /etc/pve/pve-root-ca.pem: $(cmp -s <(openssl x509 -in "$c") <(openssl x509 -in /etc/pve/pve-root-ca.pem) && echo yes || echo NO)"; echo " agent user can read it: $(sudo -n -u felhom-agent test -r "$c" && echo yes || echo no); others can read: $(stat -c %A "$c" | cut -c8)"; break; }
done
echo "served certificate (127.0.0.1:8006): $(echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256 -enddate 2>/dev/null | tr '\n' ' ')"
echo "/etc/pve/local/pve-ssl.pem: $(openssl x509 -in /etc/pve/local/pve-ssl.pem -noout -fingerprint -sha256 -startdate 2>/dev/null | tr '\n' ' ')"
echo "## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min)"
journalctl -u felhom-agent --since "-40min" --no-pager -o short-iso 2>&1 | grep -iE "pvetrust|trust|selfupdate|config bundle|signedjobs|capability|x509|tls:|certificate|config: " | grep -v "sudo\[" | cut -c1-430 | tail -${1:-25}
echo "## one control: the agent's newest successful host report and newest Proxmox call error"
journalctl -u felhom-agent --since "-20min" --no-pager -o short-iso 2>&1 | grep -iE "host report|report sent|hub: report|heartbeat" | tail -2 | cut -c1-260
echo "#### end"
@@ -0,0 +1,10 @@
#!/bin/bash
# wait-agent.sh SSH_TARGET VERSION [MINUTES] — wait until the box's agent says VERSION (a read each 30 s).
T=$1; V=$2; M=${3:-22}
end=$(( $(date +%s) + M*60 ))
while [ "$(date +%s)" -lt "$end" ]; do
cur=$(timeout 15 ssh -o BatchMode=yes -o ConnectTimeout=8 "$T" 'felhom-agent --version' 2>/dev/null | awk '{print $2}')
if [ "$cur" = "$V" ]; then echo "$(date +%T) $T agent $cur"; exit 0; fi
sleep 30
done
echo "$(date +%T) $T TIMEOUT, agent is ${cur:-unreadable}"; exit 1
@@ -0,0 +1,10 @@
#!/bin/bash
# wait-bundle.sh SSH_TARGET VERSION [MINUTES] — wait until the box's journal says the config bundle VERSION is installed.
T=$1; V=$2; M=${3:-22}
end=$(( $(date +%s) + M*60 ))
while [ "$(date +%s)" -lt "$end" ]; do
n=$(timeout 20 ssh -o BatchMode=yes -o ConnectTimeout=8 "$T" "journalctl -u felhom-agent --since '-60min' --no-pager | grep 'config bundle INSTALLED' | grep -c 'agent_version=$V'" 2>/dev/null)
if [ "${n:-0}" -ge 1 ]; then echo "$(date +%T) $T bundle $V installed"; exit 0; fi
sleep 30
done
echo "$(date +%T) $T TIMEOUT"; exit 1
@@ -0,0 +1,10 @@
#!/bin/bash
# wait-hublog.sh 'REGEX' SINCE(e.g. 5m) [MINUTES] — wait for a hub log line (a read each 20 s), print the matches.
R=$1; SINCE=$2; M=${3:-20}
end=$(( $(date +%s) + M*60 ))
while [ "$(date +%s)" -lt "$end" ]; do
out=$(sudo kubectl -n felhom-system logs deploy/hub --since="$SINCE" 2>/dev/null | grep -iE "$R")
if [ -n "$out" ]; then echo "$(date +%T) found:"; echo "$out" | cut -c1-320; exit 0; fi
sleep 20
done
echo "$(date +%T) TIMEOUT: no line for $R"; exit 1
@@ -0,0 +1,10 @@
#!/bin/bash
# wait-moved.sh SSH_TARGET [MINUTES] — wait until the box's agent config holds a ca_file (a read each 30 s).
T=$1; M=${2:-25}
end=$(( $(date +%s) + M*60 ))
while [ "$(date +%s)" -lt "$end" ]; do
cur=$(timeout 15 ssh -o BatchMode=yes -o ConnectTimeout=8 "$T" "python3 -c 'import json;t=json.load(open(\"/etc/felhom-agent/agent.json\")).get(\"proxmox\",{}).get(\"tls\",{});print(\"ca=\"+t.get(\"ca_file\",\"\")+\" fp=\"+(\"set\" if t.get(\"fingerprint\") else \"empty\"))'" 2>/dev/null)
case "$cur" in ca=/*) echo "$(date +%T) $T $cur"; exit 0;; esac
sleep 30
done
echo "$(date +%T) $T TIMEOUT: ${cur:-unreadable}"; exit 1
@@ -0,0 +1,33 @@
### root@192.168.0.154 2026-11-15T10:09:03Z
containers=10 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 43 seconds (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.353.0
image digest: sha256:4be0410d3bd5e0db2e98ab2a6f386a73784b72f62ea259821e8b13fb2cc51a6c repo: gitea.dooplex.hu/admin/felhom-controller@sha256:61cf2a8bfb82e02694bf85cca774bce2cbb9f947d395f91af7640aa5a5d944f0
started=2026-11-15T10:08:21.35521743Z
go in the running binary: 2 go1.27.4
errors=0 warn=0 info=132 panics=0
2026/11/15 10:08:21 [INFO] felhom-controller 0.353.0 starting (customer: tester-1, domain: enkicsifelhom.hu)
2026/11/15 10:08:22 [INFO] [selfupdate] Post-update startup: update successful (0.352.1 → 0.353.0)
### demo-hp 2026-11-15T10:09:05Z
containers=26 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 42 seconds (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.353.0
image digest: sha256:4be0410d3bd5e0db2e98ab2a6f386a73784b72f62ea259821e8b13fb2cc51a6c repo: gitea.dooplex.hu/admin/felhom-controller@sha256:61cf2a8bfb82e02694bf85cca774bce2cbb9f947d395f91af7640aa5a5d944f0
started=2026-11-15T10:08:23.282656164Z
go in the running binary: 2 go1.27.4
errors=0 warn=3 info=140 panics=0
2026/11/15 10:08:24 main.go:335: [INFO] felhom-controller 0.353.0 starting (customer: demo-hp, domain: enkisfelhom.hu)
2026/11/15 10:08:24 deploy.go:1470: [WARN] [stacks] Stack plex: new field PLEX_CLAIM (type=text) requires manual configuration
2026/11/15 10:08:24 deploy.go:1470: [WARN] [stacks] Stack calibre-web: new field ADMIN_PASSWORD (type=password) requires manual configuration
2026/11/15 10:08:24 deploy.go:1470: [WARN] [stacks] Stack bookstack: new field ADMIN_PASSWORD (type=password) requires manual configuration
2026/11/15 10:08:24 updater.go:855: [INFO] [selfupdate] Post-update startup: update successful (0.352.1 → 0.353.0)
### felhom-pve 2026-11-15T10:09:06Z
containers=9 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 47 seconds (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.353.0
image digest: sha256:4be0410d3bd5e0db2e98ab2a6f386a73784b72f62ea259821e8b13fb2cc51a6c repo: gitea.dooplex.hu/admin/felhom-controller@sha256:61cf2a8bfb82e02694bf85cca774bce2cbb9f947d395f91af7640aa5a5d944f0
started=2026-11-15T10:08:20.225850218Z
go in the running binary: 2 go1.27.4
errors=0 warn=0 info=146 panics=0
2026/11/15 10:08:21 [INFO] felhom-controller 0.353.0 starting (customer: demo-felhom, domain: demo-felhom.eu)
2026/11/15 10:08:21 [INFO] [selfupdate] Post-update startup: update successful (0.352.1 → 0.353.0)
@@ -0,0 +1,31 @@
# Tester 1 ONLY. clock.sh snap|back|right
# snap — READ: the clock, the time service, every container's id and creation time in guest 9201.
# back — stop the time service and set the clock 150 s BACK (the smaller harm: putting it right is then a
# step FORWARD, the direction a time service steps a late clock; nothing gets a stamp from the future
# while the clock is wrong, and no certificate or signed job ends early).
# right — start the time service and step the clock right.
set -u
grep -q '"tester-1' /etc/felhom-agent/agent.json 2>/dev/null || grep -q 'tester-1' /etc/felhom-agent/agent.json || { echo "REFUSED: this box is not Tester 1"; exit 2; }
snap() {
echo "## $(hostname) clock $(date '+%F %T.%N %Z' | cut -c1-27) epoch $(date +%s.%N | cut -c1-14); chrony $(systemctl is-active chrony); $(chronyc tracking 2>/dev/null | grep -E 'System time|Leap' | tr -s ' ' | tr '\n' ';')"
echo "## kernel boot_id $(cat /proc/sys/kernel/random/boot_id); btime $(awk '/^btime/ {print $2}' /proc/stat); uptime $(cut -d' ' -f1 /proc/uptime) s"
echo "## containers in 9201 (name id created):"
pct exec 9201 -- docker ps -a --format '{{.Names}} {{.ID}} {{.CreatedAt}} [{{.Status}}]' | sort | sed 's/^/ /'
echo "## count $(pct exec 9201 -- docker ps -q | wc -l) running"
}
case "${1:-}" in
snap) snap ;;
back)
busy=$(ps -eo comm | grep -cE "^(vzdump|apt-get|dpkg|proxmox-backup-)")
[ "$busy" = 0 ] || { echo "REFUSED: something is in flight"; exit 2; }
snap
systemctl stop chrony; echo "chrony stopped rc=$?"
date -s "@$(( $(date +%s) - 150 ))" >/dev/null; echo "clock set 150 s back rc=$? → $(date '+%F %T')"
;;
right)
systemctl start chrony; echo "chrony started rc=$?"
sleep 8; chronyc makestep; sleep 6
echo "clock now $(date '+%F %T'); $(chronyc tracking | grep -E 'System time|Leap' | tr -s ' ' | tr '\n' ';')"
;;
*) echo "usage: snap|back|right"; exit 2 ;;
esac
@@ -0,0 +1,2 @@
names read: 26; header exactly 'max-age=31536000': 26
control (a name that is not a box): felhom.eu -> []
@@ -0,0 +1,13 @@
#!/bin/bash
# hsts.sh HTTPS-ONLY-OUTPUT — read the Strict-Transport-Security header of every public name the https-only check
# walked (a GET of each name's https address; reads only). One line per name; a tally at the end.
names=$(grep -oE "https://[a-z0-9.-]+" "$1" | grep -v "no-such-name" | sort -u)
n=0; ok=0
for u in $names; do
h=$(curl -s -o /dev/null -D - --max-time 15 "$u/" | tr -d '\r' | grep -i '^strict-transport-security:' | head -1)
n=$((n+1))
v=$(echo "$h" | cut -d: -f2- | sed 's/^ *//')
if [ "$v" = "max-age=31536000" ]; then ok=$((ok+1)); else echo " NOT one year: $u -> [${v:-no header}]"; fi
done
echo "names read: $n; header exactly 'max-age=31536000': $ok"
echo "control (a name that is not a box): felhom.eu -> [$(curl -s -o /dev/null -D - --max-time 15 https://felhom.eu/ | tr -d '\r' | grep -i '^strict-transport-security:' | cut -d: -f2- | sed 's/^ *//')]"
@@ -0,0 +1,34 @@
control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
== tester-1 (enkicsifelhom.hu): 6 public name(s)
OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
== demo-hp (enkisfelhom.hu): 15 public name(s)
OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
OK http://time.enkisfelhom.hu/r1033-check?q=1 -> 301 https://time.enkisfelhom.hu/r1033-check?q=1
OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
== demo-felhom (demo-felhom.eu): 8 public name(s)
OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
== every public name redirects http -> https
@@ -0,0 +1,257 @@
System — Felhom Hub
Felhom Hub
Dashboard
Customers
Apps
Hosts
System
Offsite
Configuration
System — versions and OS updates
Needs attention
One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3).
[hover: worth a look] look
Tester4-2f0c25 Tester4
partial: no running customer guest · the host needs a restart since 2026-11-15 · crash guard not installed · [hover: the box could not read these values (agent older than v0.142.0, or the guest is down) — never a guess] could not read: running kernel, next-boot kernel, default kernel, host Debian, held packages, kernel.panic, crash restarts, guest Debian, Docker engine, containerd, Docker live-restore
Waiting for you
Update sets that only you approve: the kernel, the Docker engine and the Proxmox packages. Guest and host Debian updates approve themselves after 24 h and one night on the [hover: Ring 0 = the demo boxes. They install every new fix first; the other boxes (ring 1) install only what is approved.] ring-0 boxes.
Docker engine 29.9.3-1
Ran on every ring-0 box — demo-felhom-8363b5: 2 healthy night(s), demo-hp-bb76ea: 2 healthy night(s). The memory-kill check passed.
first seen 2 days ago (2026-11-13 04:49 UTC). Approving installs nothing by itself: a ring-1 box takes it only through a signed operator job.
Approve Docker set
Proxmox packages 9.2.27 — still being tested: demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set
Boxes
Click a box to see every value it reports, and its ring and update switches. "unknown": the box could not read the value — never a guess.
Box Ring Health Controller · agent OS updates Kernel [hover: The newest OS update run] Last night
Tester4-2f0c25 Tester4
[hover: Ring 1: a normal box — it installs only approved releases] 1
[hover: worth a look] look
0.353.0 [hover: current (vouched 0.175.1)] agent 0.175.1
[hover: runs the newest approved guest and host releases] on · up to date
unknown
[hover: 1 min ago · applied · 89 s — last successful leg: 3 min ago] ok · 1 min ago
partial: no running customer guest
Ring and updates
Ring ring 1
→ demo
OS updates updates ON
switch off
Tunnel [hover: no cloudflared container in the guest (agent state: absent)] none set up
Controller 0.353.0
Agent [hover: current (vouched 0.175.1)] 0.175.1
Last OS leg [hover: last successful leg: 3 min ago] 1 min ago · applied · 89 s
Docker engine
Docker [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
containerd [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
live-restore [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
Docker release —
Host
Proxmox 9.2.2
Kernel (running) [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
Kernel (next boot) [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: the box reports no kernel lane (agent older than v0.152.0)] —
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] —
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: refreshed] 7.0.2-6-pve · 1 rewritten · 13 min ago
Debian [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
Felhom release os-host-20261115-045718
Pending 99
Not covered 0
Held [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
Reboot needed since 2026-11-15
kernel.panic [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
Oops no
Crash restarts 24 h [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
Crash guard not installed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.175.1
Guest
Guest Debian [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
Felhom release os-guest-20261111-034654
Pending 6
Restart needed 7
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 8 min ago · 78.1 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] not yet
demo-felhom-8363b5 Demo Ügyfél
[hover: Ring 0: a demo box — it installs every new fix first] 0
[hover: nothing amber or red] fine
0.353.0 [hover: current (vouched 0.175.1)] agent 0.175.1
[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind
7.0.14-28-pve
[hover: 6 h ago · nothing · 23 s — last successful leg: 6 h ago] ok · 6 h ago
Ring and updates
Ring ring 0
→ normal
OS updates updates ON
switch off
Tunnel running
Controller 0.353.0
Agent [hover: current (vouched 0.175.1)] 0.175.1
Last OS leg [hover: last successful leg: 6 h ago] 6 h ago · nothing · 23 s
Docker engine
Docker 29.9.3
containerd 2.4.2-4~debian.13~trixie
live-restore on
Docker release ring0-20261115T045634Z
Host
Proxmox 9.2.27
Kernel (running) 7.0.14-28-pve
Kernel (next boot) 7.0.14-28-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 1m14s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 7 days ago · due 7.0.14-29-pve — household told 2 h ago: TONIGHT · phase good
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: removed 6 old kernel package(s): proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed; kept 7.0.14-26-pve, 7.0.14-28-pve; 6.3 GiB gained; 0 left-over path(s)] cleaned 6 h ago
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: not_needed] 7.0.14-28-pve · 23 h ago
Debian 13.7
Felhom release ring0-20261115T045634Z
Pending 29
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.175.1
Guest
Guest Debian 13.7
Felhom release ring0-20261115T045634Z
Pending 0
Restart needed 1
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 4 days ago · 272.7 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 92.7 Mbit/s · 5 days ago
demo-hp-bb76ea Demo HP
[hover: Ring 0: a demo box — it installs every new fix first] 0
[hover: nothing amber or red] fine
0.353.0 [hover: current (vouched 0.175.1)] agent 0.175.1
[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind
7.0.14-28-pve
[hover: 6 h ago · nothing · 30 s — last successful leg: 6 h ago] ok · 6 h ago
Ring and updates
Ring ring 0
→ normal
OS updates updates ON
switch off
Tunnel running
Controller 0.353.0
Agent [hover: current (vouched 0.175.1)] 0.175.1
Last OS leg [hover: last successful leg: 6 h ago] 6 h ago · nothing · 30 s
Docker engine
Docker 29.9.3
containerd 2.4.2-4~debian.13~trixie
live-restore on
Docker release ring0-20261115T040248Z
Host
Proxmox 9.2.27
Kernel (running) 7.0.14-28-pve
Kernel (next boot) 7.0.14-28-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 2m39s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 5 days ago · due 7.0.14-29-pve — household told 2 h ago: TONIGHT · phase good
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] —
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: not_needed] 7.0.14-28-pve · 23 h ago
Debian 13.7
Felhom release ring0-20261115T040248Z
Pending 30
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.175.1
Guest
Guest Debian 13.7
Felhom release ring0-20261115T040248Z
Pending 0
Restart needed 1
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 4 days ago · 70.5 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 94.3 Mbit/s · 5 days ago
tester-1-d70be4 Tester 1
[hover: Ring 1: a normal box — it installs only approved releases] 1
[hover: nothing amber or red] fine
0.353.0 [hover: current (vouched 0.175.1)] agent 0.175.1
[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind
7.0.14-22-pve
[hover: 6 h ago · nothing · 16 s — last successful leg: 6 h ago] ok · 6 h ago
Ring and updates
Ring ring 1
→ demo
OS updates updates ON
switch off
Tunnel running
Controller 0.353.0
Agent [hover: current (vouched 0.175.1)] 0.175.1
Last OS leg [hover: last successful leg: 6 h ago] 6 h ago · nothing · 16 s
Docker engine
Docker 29.8.2
containerd 2.3.6-1~debian.13~trixie
live-restore on
Docker release —
Host
Proxmox 9.2.2
Kernel (running) 7.0.14-22-pve
Kernel (next boot) 7.0.14-22-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-22-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: ring 1: no kernel staged by a signed os_kernel_step healthy 1m17s after the agent started; the new kernel is the default] 7.0.14-22-pve applied 38 days ago · phase good
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] —
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: not_needed] 7.0.14-22-pve · 23 h ago
Debian 13.7
Felhom release os-host-20261111-034654
Pending 101
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.175.1
Guest
Guest Debian 13.7
Felhom release os-guest-20261111-034654
Pending 6
Restart needed 0
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 4 days ago · 90.4 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 93.0 Mbit/s · 4 days ago
Details
Release ids, cancelled approvals, what ring 0 runs, the version floors, and every box's crash guard and root files.
Approved releases
The newest approved set of each layer. A [hover: Approved while a TEST wait override was active. It is cancelled when the hub starts without the override (`11` §5.3.1).] TEST approval is amber.
guest : os-guest-20261111-034654
272 packages · 2026-11-11 03:46 UTC · by auto
host : os-host-20261115-045718
607 packages · 2026-11-15 04:57 UTC · by auto
docker : os-docker-20261026-190351
6 packages · 2026-10-26 19:03 UTC · by operator
pve : os-pve-20261114-075712
121 packages · 2026-11-14 07:57 UTC · by operator
kernel : os-kernel-20261114-075716
2 packages · 2026-11-14 07:57 UTC · by operator
Approve now (guest + host)
An urgent fix only — normally the hub approves after 24 h and one night.
What ring 0 runs now
guest :
272 packages, first seen 2026-11-10 01:49 UTC
approved as os-guest-20261111-034654
host :
607 packages, first seen 2026-11-14 04:52 UTC
approved as os-host-20261115-045718
docker :
6 packages, first seen 2026-11-13 04:49 UTC
ready — see "Waiting for you"
pve :
121 packages, first seen 2026-11-15 04:58 UTC
demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set
kernel :
2 packages, first seen 2026-11-10 01:57 UTC
approved as os-kernel-20261114-075716
Version floors
Global controller floor: 0.353.0 · vouched agent: 0.175.1
[hover: A floor is the lowest controller version a box must run; the hub moves a box below it up.] What is a floor? · [hover: The agent version the operator checked and signed off for the fleet. Agents update only by a per-box signed job.] What is a vouched agent?
No per-customer floors: every box follows the global floor.
Crash guard and root files
Box ¦ kernel.panic ¦ Oops ¦ Crash restarts 24 h ¦ Crash guard ¦ [hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files ¦
Tester4-2f0c25 ¦ [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown ¦ no ¦ [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown ¦ not installed ¦ 0.175.1 ¦
demo-felhom-8363b5 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.175.1 ¦
demo-hp-bb76ea ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.175.1 ¦
tester-1-d70be4 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.175.1 ¦
Felhom Hub 0.182.0
@@ -0,0 +1,34 @@
#!/bin/bash
# Read from the LIVE hub database (the three files in one tar, as the note says), then shred the copy:
# does the newest report of each test household carry web_certs? Also the daily ledger and the floors.
set -u
W=/mnt/5_hdd/felhom.eu/evidence/release-2b-3-2026-11-15/dbcopy; mkdir -p "$W"; chmod 700 "$W"
sudo kubectl -n felhom-system exec deploy/hub -- sh -c 'cd /data && tar cf - hub.db hub.db-wal hub.db-shm' > "$W/db.tar"
( cd "$W" && tar xf db.tar )
python3 -I - "$W/hub.db" <<'PY'
import sqlite3, sys, json
db = sqlite3.connect(sys.argv[1])
tabs = [r[0] for r in db.execute("SELECT name FROM sqlite_master WHERE type='table'")]
rt = "reports" if "reports" in tabs else [t for t in tabs if "report" in t][0]
cols = [r[1] for r in db.execute("PRAGMA table_info(%s)" % rt)]
print("report table:", rt, "columns:", ",".join(cols)[:200])
tcol = [c for c in cols if c in ("received_at", "reported_at", "created_at")][0]
jcol = [c for c in cols if c in ("data", "report", "payload", "body", "json", "report_json")]
print("freshness control: newest report", db.execute("SELECT MAX(%s) FROM %s" % (tcol, rt)).fetchone()[0])
for cust in ("tester-1", "demo-hp", "demo-felhom"):
row = db.execute("SELECT %s, %s FROM %s WHERE customer_id=? ORDER BY %s DESC LIMIT 1" % (tcol, jcol[0], rt, tcol), (cust,)).fetchone()
d = json.loads(row[1])
wc = d.get("web_certs")
def find(o):
if isinstance(o, dict):
if "web_certs" in o: return o["web_certs"]
for v in o.values():
r = find(v)
if r is not None: return r
return None
wc = wc if wc is not None else find(d)
print(cust, "report at", row[0], "controller", d.get("controller_version") or d.get("version"), "web_certs:", json.dumps(wc)[:300])
for k, v in db.execute("SELECT key, substr(value,1,120) FROM hub_settings WHERE key LIKE 'daily_ledger%' OR key LIKE 'os_alarm:%' OR key LIKE 'min_controller%' ORDER BY key"):
print("setting", k, "=", v)
PY
find "$W" -type f -exec shred -u {} \; ; rmdir "$W"; echo "copy shredded: $([ -e "$W" ] && echo NO || echo yes)"
@@ -0,0 +1,20 @@
report table: reports columns: id,customer_id,received_at,report_json,health_status,cpu_percent,memory_percent,container_total,container_running,backup_last_snapshot,controller_version,controller_url,language
freshness control: newest report 2026-11-15 10:08:31
tester-1 report at 2026-11-15 10:08:28 controller 0.353.0 web_certs: [{"domain": "*.enkicsifelhom.hu", "not_after": "2027-01-02T18:44:06Z", "issuer": "YR1"}]
demo-hp report at 2026-11-15 10:08:31 controller 0.353.0 web_certs: [{"domain": "*.enkisfelhom.hu", "not_after": "2027-01-19T01:46:00Z", "issuer": "YR2"}]
demo-felhom report at 2026-11-15 10:08:27 controller 0.353.0 web_certs: [{"domain": "*.demo-felhom.eu", "not_after": "2026-12-31T18:52:10Z", "issuer": "YR1"}]
setting min_controller_version = 0.352.1
setting min_controller_version_declared_min_agent = 0.352.1=0.158.1
setting os_alarm:agent:Tester-2-be8404 = 2026-10-19T10:30:41Z
setting os_alarm:bundle:demo-felhom-8363b5 =
setting os_alarm:bundle:demo-hp-bb76ea =
setting os_alarm:bundle:tester-1-d70be4 =
setting os_alarm:daily_last:deadline-check = 2026-11-15T04:00:00Z
setting os_alarm:daily_last:offsite-key-audit = 2026-11-15T06:10:00Z
setting os_alarm:end_date:baseline = 2026-11-15T08:50:09Z
setting os_alarm:offsite_escrow_pending:Tester-2 = 2026-10-18T16:14:28Z
setting os_alarm:operator_mail_lost_scan = 2026-11-15T08:48:59Z
setting os_alarm:reboot:Tester-2-be8404 = 2026-10-18T16:28:55Z
setting os_alarm:stale:Tester-2-be8404 = 2026-10-18T17:28:55Z
setting os_alarm:stale:demo-hp-bb76ea =
copy shredded: yes
@@ -0,0 +1,9 @@
== /customers/tester-1/floor
HTTP/1.1 303 See Other
Location: /customers/tester-1?flash=floor_set
== /customers/demo-hp/floor
HTTP/1.1 303 See Other
Location: /customers/demo-hp?flash=floor_set
== /customers/demo-felhom/floor
HTTP/1.1 303 See Other
Location: /customers/demo-felhom?flash=floor_set
@@ -0,0 +1,19 @@
## felhom clock 2026-11-15 11:26:38.0075053 epoch 1794738398.009; chrony active; System time : 0.002849487 seconds slow of NTP time;Leap status : Normal;
## kernel boot_id 0581d703-3fae-47e4-8598-ee462aa3d516; btime 1794738245; uptime 152.48 s
## containers in 9201 (name id created):
bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 2 minutes (healthy)]
bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 2 minutes (healthy)]
cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 2 minutes (healthy)]
felhom-controller 9f210e5a321f 2026-11-15 10:24:27 +0000 UTC [Up 2 minutes (healthy)]
filebrowser c3b5388d9c1e 2026-11-15 10:24:52 +0000 UTC [Up About a minute (healthy)]
paperless-postgres 913e7c4090c1 2026-11-15 10:24:41 +0000 UTC [Up About a minute (healthy)]
paperless-redis 5024d8448930 2026-11-15 10:24:41 +0000 UTC [Up About a minute (healthy)]
paperless-webserver 3e7d16433e63 2026-11-15 10:24:41 +0000 UTC [Up About a minute (healthy)]
privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 2 minutes (healthy)]
traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 2 minutes]
## count 10 running
2026/11/15 10:24:30 [INFO] [gate] boot 0581d703-3fae-47e4-8598-ee462aa3d516-1535: waiting (≤2m0s) for live drive bind(s) [/mnt/felhom-drives/adatlemez] before recreating drive-backed apps
2026/11/15 10:24:40 [INFO] [gate] boot 0581d703-3fae-47e4-8598-ee462aa3d516-1535: live bind confirmed — recreating drive-backed app paperless-ngx (state=degraded) onto /mnt/felhom-drives/adatlemez
2026/11/15 10:24:52 [INFO] [gate] boot 0581d703-3fae-47e4-8598-ee462aa3d516-1535: re-syncing FileBrowser mounts against the live binds
felhom-agent 0.175.1
0
@@ -0,0 +1,14 @@
## felhom clock 2026-11-15 11:10:47.4544935 epoch 1794737447.456; chrony inactive;
## kernel boot_id 7a721957-7e95-476c-be40-52d0dafc7284; btime 1794665801; uptime 71646.26 s
## containers in 9201 (name id created):
bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 5 hours (healthy)]
bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 5 hours (healthy)]
cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 20 hours (healthy)]
felhom-controller 09343ef712fa 2026-11-15 10:08:21 +0000 UTC [Up 2 minutes (healthy)]
filebrowser 95f38571abee 2026-11-14 14:19:57 +0000 UTC [Up 20 hours (healthy)]
paperless-postgres 31b2490aa677 2026-11-15 10:05:37 +0000 UTC [Up 5 minutes (healthy)]
paperless-redis df1c44607a9b 2026-11-15 10:05:37 +0000 UTC [Up 5 minutes (healthy)]
paperless-webserver 1f2a0d7e4f7b 2026-11-15 10:05:37 +0000 UTC [Up 5 minutes (healthy)]
privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 5 hours (healthy)]
traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 20 hours]
## count 10 running
@@ -0,0 +1,14 @@
## felhom clock 2026-11-15 11:21:49.3394920 epoch 1794738109.341; chrony active; System time : 0.001603887 seconds slow of NTP time;Leap status : Normal;
## kernel boot_id 7a721957-7e95-476c-be40-52d0dafc7284; btime 1794665951; uptime 72157.81 s
## containers in 9201 (name id created):
bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 6 hours (healthy)]
bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 6 hours (healthy)]
cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 20 hours (healthy)]
felhom-controller 09343ef712fa 2026-11-15 10:08:21 +0000 UTC [Up 13 minutes (healthy)]
filebrowser 95f38571abee 2026-11-14 14:19:57 +0000 UTC [Up 20 hours (healthy)]
paperless-postgres 31b2490aa677 2026-11-15 10:05:37 +0000 UTC [Up 16 minutes (healthy)]
paperless-redis df1c44607a9b 2026-11-15 10:05:37 +0000 UTC [Up 16 minutes (healthy)]
paperless-webserver 1f2a0d7e4f7b 2026-11-15 10:05:37 +0000 UTC [Up 16 minutes (healthy)]
privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 6 hours (healthy)]
traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 20 hours]
## count 10 running
@@ -0,0 +1,16 @@
## felhom clock 2026-11-15 11:11:17.4153213 epoch 1794737477.417; chrony active; System time : 0.000710471 seconds slow of NTP time;Leap status : Normal;
## kernel boot_id 7a721957-7e95-476c-be40-52d0dafc7284; btime 1794665951; uptime 71525.88 s
## containers in 9201 (name id created):
bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 5 hours (healthy)]
bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 5 hours (healthy)]
cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 20 hours (healthy)]
felhom-controller 09343ef712fa 2026-11-15 10:08:21 +0000 UTC [Up 2 minutes (healthy)]
filebrowser 95f38571abee 2026-11-14 14:19:57 +0000 UTC [Up 20 hours (healthy)]
paperless-postgres 31b2490aa677 2026-11-15 10:05:37 +0000 UTC [Up 5 minutes (healthy)]
paperless-redis df1c44607a9b 2026-11-15 10:05:37 +0000 UTC [Up 5 minutes (healthy)]
paperless-webserver 1f2a0d7e4f7b 2026-11-15 10:05:37 +0000 UTC [Up 5 minutes (healthy)]
privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 5 hours (healthy)]
traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 20 hours]
## count 10 running
chrony stopped rc=0
clock set 150 s back rc=0 → 2026-11-15 11:08:49
@@ -0,0 +1,4 @@
chrony started rc=0
200 OK
clock now 2026-11-15 11:20:27; System time : 0.000000000 seconds fast of NTP time;Leap status : Normal;
Tester 1 minus DooPlex: +0.071 s
@@ -0,0 +1,15 @@
#!/bin/bash
# wait-ctl.sh VERSION [MINUTES] — wait until guest 9201 on the three test boxes runs controller VERSION (reads).
V=$1; M=${2:-25}
end=$(( $(date +%s) + M*60 ))
while [ "$(date +%s)" -lt "$end" ]; do
ok=0; line=""
for h in root@192.168.0.154 demo-hp felhom-pve; do
img=$(timeout 20 ssh -o BatchMode=yes -o ConnectTimeout=8 "$h" "pct exec 9201 -- docker ps --filter name=^felhom-controller$ --format '{{.Image}} {{.Status}}'" 2>/dev/null)
line="$line | $h: ${img##*:}"
case "$img" in *":$V "*healthy*) ok=$((ok+1));; esac
done
if [ "$ok" = 3 ]; then echo "$(date +%T) all three$line"; exit 0; fi
sleep 30
done
echo "$(date +%T) TIMEOUT$line"; exit 1
@@ -0,0 +1,16 @@
### root@192.168.0.154 2026-11-15T10:51:31Z
containers=10 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 27 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.353.0
errors=0 warn=1 info=635 panics=0
### demo-hp 2026-11-15T10:51:33Z
containers=26 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 43 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.353.0
errors=0 warn=3 info=907 panics=0
### felhom-pve 2026-11-15T10:51:35Z
containers=9 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 43 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.353.0
errors=0 warn=0 info=856 panics=0
9202: gitea.dooplex.hu/admin/felhom-controller:0.353.0-rc.1 Up About an hour (healthy)
@@ -0,0 +1,5 @@
2026/11/15 11:26:53 [INFO] Global controller-version floor set to "0.353.0" (declared MinAgent "0.158.1")
2026/11/15 11:26:53 [INFO] access token "cc-releases-2": POST /configuration/global-floor → 303
2026/11/15 11:46:10 [WARN] slow request: POST /api/v1/host-enroll 201 11241ms (over 5s)
2026/11/15 11:46:32 [WARN] escrow CARRIED for Tester4: the sealed package of deleted host Tester4-46836f is now the current package of Tester4-2f0c25 (repo-key hash 5501e4bf2cda…); the box will ask for the recovery cod
2026/11/15 11:48:03 [INFO] managed floor SERVED for Tester4: floor 0.353.0, agent requirement "0.158.1" from declared (golden 0.352.1)