releases 2b and 3 are out (2026-11-15): report, register (R-1380, R-1395, R-1050 closed; R-1397 filed), status, golden 0.353.0 record, evidence
gates / gates (push) Successful in 2m9s

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-15 12:14:54 +01:00
1 parent f561c85bbc
commit 07bf778699
70 files changed
+2416 -13

No files matched your search

+14
View File
@@ -16,6 +16,20 @@
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
> **2026-11-15 (attended release run) — releases 2b and 3 are RELEASED; the note below is history.**
> Report: `REPORT-release-2b-3-2026-11-15.md`. Hub 0.182.0 (deployed 09:48 CET on the operator's yes), agent 0.175.1
> (vouched 11:07:35; 0.175.0 exists and is NOT vouched), controller 0.353.0 (global floor 10:26:53 UTC, MinAgent
> 0.158.1), golden 0.353.0 (vouched 11:52:54), installer 1.37.0 (tag `installer-v1.37.0`, published). Proven on
> boxes: the move to the CA copy on Tester 1, demo-hp and the N100; two forced certificate renewals on demo-hp with the
> agent in control; the clock alarm raised and cleared; no app recreate at a clock step with the new controller AND
> agent; a real restart still read as a new boot. **Two things a session must know:** (1) `agent.json` is written only
> through `config.ReplaceInPlace` (journal `agent.json.write-in-flight` in `/var/lib/felhom-agent`; the daemon repairs
> a cut write before it reads the config; a selftest never repairs; a test package sets `config.CutWriteDir`).
> (2) An agent that is newer than the box's root files gets the wrapper's R11 „unknown mode”: on a pinned box that is
> NOT a trust problem, but only when no call failed (`pvetrust.rootStepTooOld`, `isTrustCause`). A wrong CA copy
> heals at the agent's start, and a blind box sends no report — so the trust alarm cannot be staged from a wrong
> copy (R-1396 gap 2 stays open). Rows closed: R-1380, R-1395, R-1050; opened: R-1397.
> **2026-11-14 night → 11-15 morning (unattended) — releases 2b and 3 are BUILT IN SOURCE; nothing is released.**
> Report: `REPORT-night-build-2026-11-14.md`. Heads: `felhom.eu` `d2683b28`, `felhom-agent` `54ac2c2`,
> `felhom-controller` `ee3b05e`; the installer is not tagged. Live versions did not move (hub 0.181.0, agent 0.174.0
+120
View File
@@ -0,0 +1,120 @@
# Release run 2026-11-15: releases 2b and 3 — hub, agent, controller, golden, installer
Source: `REPORT-night-build-2026-11-14.md` §2. Evidence: `documentation/audits/release-2b-3-2026-11-15/`,
`documentation/tests/golden-0.353.0-2026-11-15/`. Three stops were asked and answered (hub deploy: yes; agent vouch:
yes; golden vouch and installer tag: yes, yes). One more question was asked: to delete the old test host before the
fresh install (yes).
## 1. Claims in the prompt that turned out wrong
1. **„Make the CA copy wrong … read the operator alarm on the hub.”** No alarm can come from that. A wrong copy is
repaired by the agent at its start (seen on demo-hp). And a box whose trust does not work sends no report, so the
hub cannot name the cause (the open gap, R-1396 part 2). The alarm that names a cause WAS seen, from a real event
on Tester 1 (below).
2. **„The CA copy … readable by the agent only as designed.”** The design is mode 0644, owner root: a CA certificate
is public (`03-host-agent.md` §12a). Read so on all three boxes and on the fresh box.
3. **„You will get one test mail.”** There were two. The second was a false trust alarm from Tester 1 (below).
4. **„Scratch 9202 … the controller rc”, then the floor moves every box.** The scratch guest does not report to the
hub, so no floor moves it. It was put on 0.353.0 by hand.
The three claims the reviewer asked to try:
- „The hub's sweep deletes only rows whose owner is gone” — **true**. Read in source; on a copy of the database a
control query found 0 rows to delete whose owner exists. The live sweep deleted the same 26 rows as the preview.
- „A host clock can be staged wrong on Tester 1 and put right with no lasting effect” — **true, with two notes.**
With the old controller each step recreated one app (expected). Log lines of those minutes carry a time 150 s
behind. After each run the offset to DooPlex was +0.07 s.
- „The untagged installer can be used for one fresh install” — **true, with two hand steps**: the file was copied
in, and one line was added to name the not-yet-vouched golden.
## 2. Six lines
1. **The hub is live** (0.182.0, 09:48 CET). Its first sweep deleted 26 old update reports of 13 boxes that were
deleted earlier, and no household row.
2. **The three boxes moved to the CA copy: Tester 1 yes, demo-hp yes, the N100 yes** (the N100 by itself).
3. **The agent kept control through two new certificates on demo-hp**: the same process, no restart, no alarm.
4. **The clock alarm was seen and cleared; with the new controller no app was recreated.** A real restart of
Tester 1 is still read as a new boot.
5. **Every box is on controller 0.353.0** with the one-year header (26 of 26 public names).
6. **The golden is vouched: yes. The installer is tagged and published: yes.**
## 3. What was found and fixed on the way
| What | Where it was found | Fix |
|---|---|---|
| A power cut in the middle of the agent's config write left an empty file; the agent then does not start, and the hub cannot repair that box | Part 0, reading the write path | The old file goes into a journal first; the agent repairs the config at start. Tests cut the write at four steps; 9 mutations seen red (3 redone, said in the evidence). **Not proven with a real power cut** |
| Between the agent update and the root-files update a box said „trust problem”, and the hub mailed the operator. False: the pin worked | Tester 1, 10:04, with agent 0.175.0 | Agent 0.175.1: older root files are not a trust problem on a pinned box. The security review of that fix found a hole (the quiet path could hide a failed pin); narrowed, 7 mutations seen red (3 redone). Seen on demo-hp and the N100: no alarm |
Agent 0.175.0 was released, reached Tester 1 only, and is **not** vouched. Two agent releases in one session is one
more than the rule says; the second replaced the first before the vouch.
## 4. Versions, digests, CI by commit
| Part | Version | Digest | Built from | CI run (by commit) | Live since (CET) |
|---|---|---|---|---|---|
| Hub | 0.182.0 | image `sha256:4f8bf649ef85db56a727e77327f5ba7ebcceb739a453c81c5423517882ae4681` | `35c5b7ce` | 2673 success; the manifest commit `3fdfab09` 2674 success, read before the deploy | 09:48:59 |
| Agent | 0.175.1 | binary sha256 `38b1e2350de90a7940c16f8cd2a21d83fcd535410d1017a7df0fee14f0904612`; bundle `07ebf86362ed9d44776416bd5c2c4b6b629e44d34595a8120fd8fc3f4d9277ff` | `88d5d56` | 2678 success | vouched 11:07:35 |
| (Agent, not vouched) | 0.175.0 | binary `d1a30ed5…4b19a`; bundle `3b4a8b78…0d95` | `db01ae0` | 2671 success | Tester 1 only, 10:04–10:34 |
| Controller | 0.353.0, MinAgent 0.158.1 (not moved: the boot key is optional) | image `sha256:61cf2a8bfb82e02694bf85cca774bce2cbb9f947d395f91af7640aa5a5d944f0` | `ae07ed4` | 2681 success | per-box floors 11:08; global floor 11:26:53 |
| Golden | 0.353.0 | sha256 `86b5ba20a15ebd511db8828f51d38fb984f68efa23c4d4e89ae33aa195f19971` | controller 0.353.0 | — | vouched 11:52:54 |
| Installer | 1.37.0, tag `installer-v1.37.0` | file sha256 `93a021db17a98d1e5a31e1497481b41294319aa79ffecdb7038c697339f26e65` | `6174944f` | 2683 success; the pointer commit `f561c85b` 2685 success | published 12:02; read back from the public name with the same sha256 |
Does the controller wait for the agent's vouch? No: it works with the old agent. The clock-step fix needs both.
## 5. The register
250 rows before; **248 after**; opened 1 (one Proxmox call answers 500 during a fresh install and the agent logs no
error, R-1397); closed 3 (R-1380, R-1395, R-1050). Narrowed with what is live and what is tests only: R-1384, R-1383,
R-1375, R-1376, R-1379, R-1396.
`unproven.py --summary`: 32 walked, 15 partial, 4 built, 4 missing, 23 not walked — no number moved. The capability
map: no row moved (no row's status word changes by today's proofs; the candidates are the rows on trust in Proxmox
and on alarms — to read at the next map pass).
## 6. Machines changed and how each was left
| Machine | What changed | Left as |
|---|---|---|
| DooPlex | hub 0.182.0 (manifest, hub only); image builds; the bake in the drill VM; the installer publish | drill VM reverted before use and powered off; both copies of the hub database shredded after the read |
| Tester 1 | agent 0.175.0 then 0.175.1 and their root files; controller 0.353.0; the clock set 150 s back twice and put right; one restart | healthy, 10 containers, clock +0.07 s, time service active; one app was recreated three times (twice by the old controller's rule, once by the real restart) |
| demo-hp | agent 0.175.1 and root files; controller 0.353.0; two new Proxmox certificates (the node's certificate is new — by design of the proof); the CA copy made wrong and repaired, identical to the saved copy; two agent restarts; the scratch guest 9202 on rc.1 then 0.353.0 | healthy, 26 containers; VM 9386 and the installer ISO removed |
| The N100 | nothing by hand. It took agent 0.175.1, its root files and controller 0.353.0 by the product's own path | healthy, 9 containers, moved to the CA copy at 10:36:58 |
| ep0, the off-site provider | nothing | — |
Teardown, three layers: machine — VM 9386 destroyed with its disks; host — the ISO and the saved copy removed from
demo-hp; hub — **host `Tester4-2f0c25` and the appliance `ab0b8310-…` are left for the operator** (only he can
discard). The floors per box are cleared.
## 7. Proven by tests only
- The repair of a cut config write (no real power cut).
- A refused signed job told to the operator; a guest-power give-up told; a lost operator mail told; a skipped daily
check told.
- The off-site account path of D71 (no provider call was made).
- A warning 14 days before an end date (no date is that near).
- The clean-up detail sent by the NEW agent (no box removed a kernel after the update; the N100's cell shows the
night's removal, sent by the old agent).
## 8. Observations
- The false trust alarm at a box's update — FIXED in agent 0.175.1 (above).
- One Proxmox call answered 500 during the fresh install, with no ERROR line in the agent — FILED: R-1397.
- The hub's own alarm marks of a box deleted before the release are not swept (4 marks of one box) — on R-1376,
dated line.
- The scratch guest stayed on rc.1 under the global floor — NOT-A-FINDING: it does not report to the hub (R-1083).
- „Household cleared its mail address — deleted” in the hub's log at each controller start of Tester 1 —
NOT-A-FINDING: the same line is in the evidence of 2026-11-09 and 11-10; Tester 1 has no household address.
- The hub suppressed the second clock mail inside one hour — NOT-A-FINDING: the one-hour rule for one type.
- After a wrong CA copy, the right root is written first and the wrong one stays behind it until the next run —
NOT-A-FINDING: the design of the changeover (`03` §12a item 6); the next start dropped it.
- The fresh host's enrolment took 11 s at the hub („slow request”) — NOT-A-FINDING: the known first-enrolment cost
(key and tunnel setup); no failure.
- My own instruments were wrong three times and each was read again: a wait that printed nothing, a time computed
with `date -d … + 15 seconds`, a log search with the wrong word („orphan” for a line that says „Purged”).
## 9. Documents read and changed
Read: `03-host-agent.md` §12a; `05-hub-architecture.md` §14, §16.6; `08-alarm-ladder.md`; `11-os-updates.md`
§5.11–5.12; `09` §3 decisions 247, 249, 250, 303; `runbooks/publish-train-rules.md`; `RUNBOOK-manual-build.md` §4;
`audits/certpin-spike-2026-11-11/README.md`; the golden record of 2026-11-14.
Changed: `03-host-agent.md` §12a item 7 (the cut write). Instruction files changed: none.
System poster: no fact on the poster changed; no edit, no refresh asked.
+2 -2
View File
@@ -2,7 +2,7 @@
**Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).**
**Updated 2026-11-15 (morning).** The N100 removed 6 of its 8 old kernels by itself at night and is healthy: 1 of the 2 nights D82 asks for; demo-hp had nothing to remove (0 of 2). Releases 2b and 3 are built and tested in source; nothing is released (`REPORT-night-build-2026-11-14.md`).
**Updated 2026-11-15 (midday).** Releases 2b and 3 are out: hub 0.182.0, agent 0.175.1 (vouched), controller 0.353.0 on every box, golden 0.353.0 (vouched), installer 1.37.0 (published). A box now trusts Proxmox through its own root certificate, and the hub tells you about a wrong box clock and other silent problems (`REPORT-release-2b-3-2026-11-15.md`).
## What works
- **The Plex attack did not reach Felhom's secrets** (R-1241).
@@ -42,7 +42,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
| D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — |
## Waiting on you — do
**2026-11-15 morning.** 1. Read `REPORT-night-build-2026-11-14.md`, „Ready to release”: hub, agent, controller and installer wait for your yes. If not: nothing reaches a box. 2. Say „HSTS now” (R-1050). 3. Hosts page: delete host `Tester4-46836f` (else it alarms). 4. The report has a privacy sentence as a proposal; you publish it or not. **2026-11-11.** 2. Hosts page: discard `3fb5fed3…`. 3. Rotate login `kisfenyo`'s Gitea token into the store as `GITEA_GIT_TOKEN` (R-925); the push after `sec put` is in R-1390. 4. Home DNS: remove the `felhom.eu` override (R-1389). 5. Read the dates only you can read (R-1381). 6. Search Console read (R-1035, 11-18).
**2026-11-15 midday.** 1. Hosts page: discard host `Tester4-2f0c25` and appliance `ab0b8310…` (CC's test install; else it alarms). 2. Once: read each off-site sub-account's label against the hub's list (R-1375). 3. The privacy sentence in the night report: publish or not. **2026-11-11.** 2. Hosts page: discard `3fb5fed3…`. 3. Rotate login `kisfenyo`'s Gitea token into the store as `GITEA_GIT_TOKEN` (R-925); the push after `sec put` is in R-1390. 4. Home DNS: remove the `felhom.eu` override (R-1389). 5. Read the dates only you can read (R-1381). 6. Search Console read (R-1035, 11-18).
**Still needed? (yes/no)** — answer per line.
1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). (since 10-16)
@@ -0,0 +1,29 @@
# Releases 2b and 3 — evidence (2026-11-15)
Report: `REPORT-release-2b-3-2026-11-15.md` (repo root). Golden and fresh install:
`documentation/tests/golden-0.353.0-2026-11-15/`. The ids of the deleted boxes whose rows the sweep removed are NOT
here: `/mnt/5_hdd/felhom.eu/evidence/release-2b-3-2026-11-15/sweep-preview-ids.txt` (off git; an id can be a name).
| File | What it shows |
|---|---|
| `part0/redproof-agent-cutwrite.txt` | the cut-write fix of the agent's config: 9 mutations, each seen red; 3 redone and one hollow test removed, said in the file |
| `part0/sweep-preview-counts.txt`, `sweep-preview.sh` | on a copy of the hub's own 08:00 snapshot: the first sweep deletes 26 of 504 `os_reports` rows, 13 owners, 0 in the five other tables; the control (a row to delete whose owner exists) is 0; the copy is deleted |
| `part1/hub-0182-start-log.txt` | hub 0.182.0's start: „Purged 26 row(s) … map[os_reports:26]”, the daily ledger's first lines, the lost-mail scan |
| `part1/hub-system-page-0950.txt` | the System page on 0.182.0 with the two new cells |
| `part2/t1-before.txt`, `t1-after-agent.txt`, `t1-moved.txt` | Tester 1: pinned → agent 0.175.0 (the wrapper refuses the unknown mode; the false trust problem) → root files → „MOVED … cleared AFTER a working call with the copy” |
| `part2/hp-moved.txt`, `n100-moved.txt` | demo-hp and the N100 with 0.175.1: no trust problem in the window, then MOVED |
| `part2/hp-cert-round1.txt`, `hp-cert-round2.txt` | two forced new Proxmox certificates on demo-hp; after each: the same agent process, 0 trust lines, 0 ERROR lines, 14 and 40 calls by the agent's token answered 200 in Proxmox's own access log |
| `part2/hp-ca-wrong.txt` | the CA copy replaced by another CA and the agent restarted: the right root is written first; after the next start the file is identical to the saved one |
| `part2/redproof-agent-root-step-too-old.txt` | the fix of the false trust alarm and its narrowing after the security review: 7 mutations seen red, 3 redone |
| `part2/t1-clock-A-*.txt` | the clock 150 s back with the OLD controller: one app recreated at the step back and again at the step forward; the kernel's boot id unchanged |
| `part2/system-1107.txt` | the „Security profiles” cell filled for the three boxes |
| `part3/boxwalk-0353.txt`, `watch-boxes.txt`, `watch-hub.txt` | the three boxes on controller 0.353.0 (image `61cf2a8b…`), 0 ERROR lines; 25 minutes after the global floor |
| `part3/https-only.txt`, `hsts-headers.txt` | the https-only check green; 26 of 26 public names answer exactly `max-age=31536000` |
| `part3/hub-webcerts.txt` | from a copy of the live hub database (shredded after): the three boxes' reports carry `web_certs`; the ledger's and the end-date baseline's marks |
| `part3/t1-clock-B-*.txt` | the same clock step with the NEW controller and agent: the same container ids before and after, in both directions |
| `part3/t1-after-reboot.txt` | one real restart of Tester 1: a new boot id, the boot pass recreates the drive-backed app |
| `part3/per-box-floors-cleared.txt`, `hub-system-page-after.txt` | the per-box floors cleared; „No per-customer floors”; global floor 0.353.0, vouched agent 0.175.1 |
The hub's two mails of the day (both to the operator's address only, read in the test mailbox): `proxmox_trust_problem`
for tester-1 at 10:04 CET (false, fixed), `box_clock_wrong` for tester-1 at 11:05 CET („150 seconds behind”). The
second clock alarm at 11:20 was suppressed by the one-hour rule (the hub's log).
@@ -0,0 +1,13 @@
snapshot: /data/snapshots/hub-20261115T070000Z.db; hub pod clock: 2026-11-15T07:21:25Z; DooPlex: 2026-11-15T07:21:25Z
copy: 674902016 bytes
integrity: ok
freshness control: newest host report 2026-11-15 06:51:43
hosts: 4 customer_configs: 5
os_host_settings would delete 0 of 2 rows, 0 owner(s)
os_reports would delete 26 of 504 rows, 13 owner(s)
os_kernel_notices would delete 0 of 18 rows, 0 owner(s)
offsite_keys would delete 0 of 5 rows, 0 owner(s)
offsite_windows would delete 0 of 23 rows, 0 owner(s)
offsite_abandon_requests would delete 0 of 1 rows, 0 owner(s)
TOTAL rows: 26; deleted boxes named: 13; deleted households named: 0 (ids saved off git, not printed)
copy deleted: yes; ids file: /mnt/5_hdd/felhom.eu/evidence/release-2b-3-2026-11-15/sweep-preview-ids.txt (13 lines); rc=0
@@ -0,0 +1,50 @@
#!/bin/bash
# Part 0.2: what the hub's first sweep (PurgeOrphanRows, D73) will delete. Reads a COPY of the hub's newest own
# snapshot (a VACUUM INTO file: whole, no WAL needed, no old free-space bytes). The copy is deleted at the end.
# The ids go OFF git (an id can be a name).
set -u
EV=/mnt/5_hdd/felhom.eu/evidence/release-2b-3-2026-11-15
mkdir -p "$EV"; chmod 700 "$EV"
SNAP=$(sudo kubectl -n felhom-system exec deploy/hub -- sh -c 'ls -1 /data/snapshots/hub-*.db | tail -1')
echo "snapshot: $SNAP; hub pod clock: $(sudo kubectl -n felhom-system exec deploy/hub -- date -u +%FT%TZ); DooPlex: $(date -u +%FT%TZ)"
COPY="$EV/hubdb-copy.db"
sudo kubectl -n felhom-system exec deploy/hub -- cat "$SNAP" > "$COPY"
chmod 600 "$COPY"
echo "copy: $(stat -c %s "$COPY") bytes"
python3 -I - "$COPY" "$EV/sweep-preview-ids.txt" <<'PY'
import sqlite3, sys
db = sqlite3.connect("file:%s?mode=ro" % sys.argv[1], uri=True)
print("integrity:", db.execute("PRAGMA integrity_check").fetchone()[0])
cols = [r[1] for r in db.execute("PRAGMA table_info(host_reports)")]
tcol = "received_at" if "received_at" in cols else cols[-1]
print("freshness control: newest host report", db.execute("SELECT MAX(%s) FROM host_reports" % tcol).fetchone()[0])
print("hosts:", db.execute("SELECT COUNT(*) FROM hosts").fetchone()[0], " customer_configs:", db.execute("SELECT COUNT(*) FROM customer_configs").fetchone()[0])
tables = [
("os_host_settings", "host_id", "host_id NOT IN (SELECT host_id FROM hosts)"),
("os_reports", "host_id", "host_id NOT IN (SELECT host_id FROM hosts)"),
("os_kernel_notices", "host_id", "host_id NOT IN (SELECT host_id FROM hosts)"),
("offsite_keys", "customer_id", "customer_id NOT IN (SELECT customer_id FROM customer_configs) AND customer_id NOT IN (SELECT customer_id FROM hosts)"),
("offsite_windows", "customer_id", "customer_id NOT IN (SELECT customer_id FROM customer_configs) AND customer_id NOT IN (SELECT customer_id FROM hosts)"),
("offsite_abandon_requests", "customer_id", "customer_id NOT IN (SELECT customer_id FROM customer_configs) AND customer_id NOT IN (SELECT customer_id FROM hosts)"),
]
total = 0; boxes = set(); houses = set()
with open(sys.argv[2], "w") as out:
for t, col, where in tables:
all_n = db.execute("SELECT COUNT(*) FROM %s" % t).fetchone()[0]
rows = db.execute("SELECT %s, COUNT(*) FROM %s WHERE %s GROUP BY 1 ORDER BY 1" % (col, t, where)).fetchall()
n = sum(r[1] for r in rows); total += n
(boxes if col == "host_id" else houses).update(r[0] for r in rows)
print(" %-26s would delete %5d of %6d rows, %d owner(s)" % (t, n, all_n, len(rows)))
for owner, c in rows:
out.write("%s\t%s\t%s\t%d\n" % (t, col, owner, c))
# the control: no row to delete may name an owner that exists
if col == "host_id":
bad = db.execute("SELECT COUNT(*) FROM %s WHERE %s AND host_id IN (SELECT host_id FROM hosts)" % (t, where)).fetchone()[0]
else:
bad = db.execute("SELECT COUNT(*) FROM %s WHERE %s AND (customer_id IN (SELECT customer_id FROM customer_configs) OR customer_id IN (SELECT customer_id FROM hosts))" % (t, where)).fetchone()[0]
assert bad == 0, (t, bad)
print("TOTAL rows: %d; deleted boxes named: %d; deleted households named: %d (ids saved off git, not printed)" % (total, len(boxes), len(houses)))
PY
rc=$?
shred -u "$COPY" 2>/dev/null || rm -f "$COPY"
echo "copy deleted: $([ -e "$COPY" ] && echo NO || echo yes); ids file: $EV/sweep-preview-ids.txt ($(wc -l < "$EV/sweep-preview-ids.txt") lines); rc=$rc"
@@ -0,0 +1,43 @@
2026/11/15 09:48:59 [INFO] felhom-hub 0.182.0 starting
2026/11/15 09:48:59 [INFO] off-site secrets sealed at rest (0 legacy plaintext row(s) sealed now)
2026/11/15 09:48:59 [INFO] console passwords sealed at rest (0 legacy plaintext row(s) sealed now)
2026/11/15 09:48:59 [INFO] box secrets sealed at rest (0 legacy plaintext value(s) sealed now)
2026/11/15 09:48:59 [INFO] config tokens sealed at rest (0 legacy plaintext token(s) sealed now)
2026/11/15 09:48:59 [INFO] Database opened at /data/hub.db
2026/11/15 09:48:59 [INFO] Default controller-version floor: 0.120.0
2026/11/15 09:48:59 [INFO] Template fetcher started (every 1h)
2026/11/15 09:49:00 [INFO] Asset seed: all 263 files up-to-date
2026/11/15 09:49:00 [INFO] Asset manifest built: 288 files
2026/11/15 09:49:00 [INFO] App-email relay enabled (limit 30/min/customer, From domains [felhom.eu])
2026/11/15 09:49:00 [INFO] Gitea artifact browser enabled (Day-0 version dropdowns) via http://gitea.gitea-system.svc.cluster.local:3000
2026/11/15 09:49:00 [INFO] Offsite provisioning enabled (pool_box=611714, location=fsn1)
2026/11/15 09:49:00 [INFO] Offsite pool-box checker initialized: box=611714 fill warn=80% crit=90%, oversub warn=2.00x, unreachable after 3 consecutive failed reads, refresh 15m0s
2026/11/15 09:49:00 [INFO] Root files: bundles are compared by their files (R-1392)
2026/11/15 09:49:00 [INFO] osupdates: approval rule = every ring-0 box healthy for 24h0m0s and 1 night run(s)
2026/11/15 09:49:00 [INFO] osupdates: the Docker engine set is approved only by the operator, after 2 healthy ring-0 night(s)
2026/11/15 09:49:00 [INFO] osupdates: alarms = no OS leg 168h0m0s, reboot needed 336h0m0s, ring 0 stalled 168h0m0s, not covered 336h0m0s
2026/11/15 09:49:00 [INFO] Off-site key registrar enabled; daily key check at 07:10 Budapest
2026/11/15 09:49:00 [INFO] access log: /data/access-log, one file per UTC day, kept 90 days
2026/11/15 09:49:00 [INFO] Registry version checker started (every 6h)
2026/11/15 09:49:00 [INFO] WG peer-sync enabled (endpoint 167.233.158.164:22, user felhom-peersync)
2026/11/15 09:49:00 [INFO] PBS DR tenantsync enabled (endpoint 167.233.158.164:22, user felhom-peersync; WG-registration auto-provision hook armed)
2026/11/15 09:49:00 [INFO] PBS-DR box checker initialized: fill warn=80% crit=90%, unreachable after 3 consecutive failed reads, refresh 15m0s
2026/11/15 09:49:00 [INFO] agent-plane poke enabled (endpoint 167.233.158.164:22, user felhom-peersync; web + api admin seams armed)
2026/11/15 09:49:00 [INFO] PBS-DR self-heal reconciler started (interval 5m)
2026/11/15 09:49:00 [INFO] offsite credential self-heal reconciler started (interval 5m, debounce 2 reports)
2026/11/15 09:49:00 [INFO] daily ledger: offsite-key-audit is watched from now (no earlier record)
2026/11/15 09:49:00 [INFO] offsite-key-audit: next run at 2026-11-16 07:10 CET (in 21h20m59s)
2026/11/15 09:49:01 [INFO] Pruned 86 old report rows
2026/11/15 09:49:01 [INFO] Pruned 86 old app telemetry rows
2026/11/15 09:49:04 [WARN] slow statement: 3091ms (over 1s): DELETE FROM app_log_issues WHERE last_seen < ?
2026/11/15 09:49:04 [INFO] Pruned 34 stale app issues
2026/11/15 09:49:05 [INFO] Purged 26 row(s) of boxes and households that no longer exist: map[os_reports:26]
2026/11/15 09:49:05 [INFO] prune: next run at 2026-11-16 04:30 CET (in 18h40m55s)
2026/11/15 09:49:05 [INFO] Staleness checker initialized: 3 ok, 0 stale, 2 down (node_stale after 45m0s, node_down after 1h30m0s)
2026/11/15 09:49:05 [INFO] Host staleness checker initialized: 3 ok, 1 already alarmed (seeded silently, R-1001), 0 stale, 0 down (left unseeded → first Check emits; host_stale after 45m0s, host_down after 1h30m0s)
2026/11/15 09:49:05 [INFO] Host capability checker initialized: 4 ok, 0 degraded, 0 with a refused whole-guest backup (degraded/refused left unseeded → first Check emits)
2026/11/15 09:49:05 [INFO] Host leaf checker initialized: 4 host fingerprint(s) seeded
2026/11/15 09:49:05 [INFO] Host disk checker initialized: warn=90% crit=95%, 4 ok seeded, 0 already-breached left unseeded (first Check emits)
2026/11/15 09:49:05 [INFO] Storage fill checker initialized: warn=90% crit=95%, 9 ok seeded, 0 already-breached left unseeded, 4 root-backed excluded
2026/11/15 09:49:05 [INFO] Host mgmt-plane checker initialized: 0 host heal-state(s) seeded
2026/11/15 09:49:05 [INFO] Host OOB checker initialized: 0 host(s) seeded degraded
@@ -0,0 +1,256 @@
System — Felhom Hub
Felhom Hub
Dashboard
Customers
Apps
Hosts
System
Offsite
Configuration
System — versions and OS updates
Needs attention
One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3).
[hover: worth a look] look
Tester4-46836f Tester4
[hover: cannot do a one-shot boot: the box does not boot UEFI; /boot/efi is not a mounted vfat ESP (not mounted)] kernel step: — · the host needs a restart since 2026-11-14 · [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] root files behind the vouched agent's · [hover: 1 minor release behind — sign an agent_update for this box] agent behind: 0.173.0 → 0.174.0 (since 2026-11-14) · [hover: the box could not read these values (agent older than v0.142.0, or the guest is down) — never a guess] could not read: default kernel
Waiting for you
Update sets that only you approve: the kernel, the Docker engine and the Proxmox packages. Guest and host Debian updates approve themselves after 24 h and one night on the [hover: Ring 0 = the demo boxes. They install every new fix first; the other boxes (ring 1) install only what is approved.] ring-0 boxes.
Docker engine 29.9.3-1
Ran on every ring-0 box — demo-felhom-8363b5: 2 healthy night(s), demo-hp-bb76ea: 2 healthy night(s). The memory-kill check passed.
first seen 2 days ago (2026-11-13 04:49 UTC). Approving installs nothing by itself: a ring-1 box takes it only through a signed operator job.
Approve Docker set
Proxmox packages 9.2.27 — still being tested: demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set
Boxes
Click a box to see every value it reports, and its ring and update switches. "unknown": the box could not read the value — never a guess.
Box Ring Health Controller · agent OS updates Kernel [hover: The newest OS update run] Last night
Tester4-46836f Tester4
[hover: Ring 1: a normal box — it installs only approved releases] 1
[hover: worth a look] look
0.352.1 [hover: 1 minor release behind — sign an agent_update for this box] agent 0.173.0 → 0.174.0 (since 2026-11-14)
[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind
7.0.2-6-pve
[hover: 22 h ago · applied · 93 s — last successful leg: 22 h ago] ok · 22 h ago
Ring and updates
Ring ring 1
→ demo
OS updates updates ON
switch off
Tunnel [hover: no cloudflared container in the guest (agent state: absent)] none set up
Controller 0.352.1
Agent [hover: 1 minor release behind — sign an agent_update for this box] 0.173.0 → 0.174.0 (since 2026-11-14)
Last OS leg [hover: last successful leg: 22 h ago] 22 h ago · applied · 93 s
Docker engine
Docker 29.9.2
containerd 2.4.2-3~debian.13~trixie
live-restore on
Docker release —
Host
Proxmox 9.2.2
Kernel (running) 7.0.2-6-pve
Kernel (next boot) 7.0.2-6-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: cannot do a one-shot boot: the box does not boot UEFI; /boot/efi is not a mounted vfat ESP (not mounted)] —
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] —
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: the box reports no security-profile refresh (an agent older than release 3, a BYO host, or no refresh yet)] —
Debian 13.7
Felhom release os-host-20261111-034654
Pending 100
Not covered 0
Held none
Reboot needed since 2026-11-14
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] 0.173.0
Guest
Guest Debian 13.7
Felhom release os-guest-20261111-034654
Pending 6
Restart needed 7
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 22 h ago · 78.4 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] not yet
demo-felhom-8363b5 Demo Ügyfél
[hover: Ring 0: a demo box — it installs every new fix first] 0
[hover: nothing amber or red] fine
0.352.1 [hover: current (vouched 0.174.0)] agent 0.174.0
[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind
7.0.14-28-pve
[hover: 3 h ago · nothing · 23 s — last successful leg: 3 h ago] ok · 3 h ago
Ring and updates
Ring ring 0
→ normal
OS updates updates ON
switch off
Tunnel running
Controller 0.352.1
Agent [hover: current (vouched 0.174.0)] 0.174.0
Last OS leg [hover: last successful leg: 3 h ago] 3 h ago · nothing · 23 s
Docker engine
Docker 29.9.3
containerd 2.4.2-4~debian.13~trixie
live-restore on
Docker release ring0-20261115T045634Z
Host
Proxmox 9.2.27
Kernel (running) 7.0.14-28-pve
Kernel (next boot) 7.0.14-28-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 1m14s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 7 days ago · due 7.0.14-29-pve — household told 49 min ago: TONIGHT · phase good
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: removed 6 old kernel package(s): proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed; kept 7.0.14-26-pve, 7.0.14-28-pve; 6.3 GiB gained; 0 left-over path(s)] cleaned 3 h ago
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: the box reports no security-profile refresh (an agent older than release 3, a BYO host, or no refresh yet)] —
Debian 13.7
Felhom release ring0-20261115T045634Z
Pending 29
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.174.0
Guest
Guest Debian 13.7
Felhom release ring0-20261115T045634Z
Pending 0
Restart needed 1
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 3 days ago · 272.7 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 92.7 Mbit/s · 4 days ago
demo-hp-bb76ea Demo HP
[hover: Ring 0: a demo box — it installs every new fix first] 0
[hover: nothing amber or red] fine
0.352.1 [hover: current (vouched 0.174.0)] agent 0.174.0
[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind
7.0.14-28-pve
[hover: 4 h ago · nothing · 30 s — last successful leg: 4 h ago] ok · 4 h ago
Ring and updates
Ring ring 0
→ normal
OS updates updates ON
switch off
Tunnel running
Controller 0.352.1
Agent [hover: current (vouched 0.174.0)] 0.174.0
Last OS leg [hover: last successful leg: 4 h ago] 4 h ago · nothing · 30 s
Docker engine
Docker 29.9.3
containerd 2.4.2-4~debian.13~trixie
live-restore on
Docker release ring0-20261115T040248Z
Host
Proxmox 9.2.27
Kernel (running) 7.0.14-28-pve
Kernel (next boot) 7.0.14-28-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 2m39s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 5 days ago · due 7.0.14-29-pve — household told 49 min ago: TONIGHT · phase good
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] —
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: the box reports no security-profile refresh (an agent older than release 3, a BYO host, or no refresh yet)] —
Debian 13.7
Felhom release ring0-20261115T040248Z
Pending 30
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.174.0
Guest
Guest Debian 13.7
Felhom release ring0-20261115T040248Z
Pending 0
Restart needed 1
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 3 days ago · 70.5 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 94.3 Mbit/s · 4 days ago
tester-1-d70be4 Tester 1
[hover: Ring 1: a normal box — it installs only approved releases] 1
[hover: nothing amber or red] fine
0.352.1 [hover: current (vouched 0.174.0)] agent 0.174.0
[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind
7.0.14-22-pve
[hover: 3 h ago · nothing · 16 s — last successful leg: 3 h ago] ok · 3 h ago
Ring and updates
Ring ring 1
→ demo
OS updates updates ON
switch off
Tunnel running
Controller 0.352.1
Agent [hover: current (vouched 0.174.0)] 0.174.0
Last OS leg [hover: last successful leg: 3 h ago] 3 h ago · nothing · 16 s
Docker engine
Docker 29.8.2
containerd 2.3.6-1~debian.13~trixie
live-restore on
Docker release —
Host
Proxmox 9.2.2
Kernel (running) 7.0.14-22-pve
Kernel (next boot) 7.0.14-22-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-22-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: ring 1: no kernel staged by a signed os_kernel_step healthy 1m17s after the agent started; the new kernel is the default] 7.0.14-22-pve applied 38 days ago · phase good
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] —
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: the box reports no security-profile refresh (an agent older than release 3, a BYO host, or no refresh yet)] —
Debian 13.7
Felhom release os-host-20261111-034654
Pending 101
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.174.0
Guest
Guest Debian 13.7
Felhom release os-guest-20261111-034654
Pending 6
Restart needed 0
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 3 days ago · 90.4 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 93.0 Mbit/s · 4 days ago
Details
Release ids, cancelled approvals, what ring 0 runs, the version floors, and every box's crash guard and root files.
Approved releases
The newest approved set of each layer. A [hover: Approved while a TEST wait override was active. It is cancelled when the hub starts without the override (`11` §5.3.1).] TEST approval is amber.
guest : os-guest-20261111-034654
272 packages · 2026-11-11 03:46 UTC · by auto
host : os-host-20261115-045718
607 packages · 2026-11-15 04:57 UTC · by auto
docker : os-docker-20261026-190351
6 packages · 2026-10-26 19:03 UTC · by operator
pve : os-pve-20261114-075712
121 packages · 2026-11-14 07:57 UTC · by operator
kernel : os-kernel-20261114-075716
2 packages · 2026-11-14 07:57 UTC · by operator
Approve now (guest + host)
An urgent fix only — normally the hub approves after 24 h and one night.
What ring 0 runs now
guest :
272 packages, first seen 2026-11-10 01:49 UTC
approved as os-guest-20261111-034654
host :
607 packages, first seen 2026-11-14 04:52 UTC
approved as os-host-20261115-045718
docker :
6 packages, first seen 2026-11-13 04:49 UTC
ready — see "Waiting for you"
pve :
121 packages, first seen 2026-11-15 04:58 UTC
demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set
kernel :
2 packages, first seen 2026-11-10 01:57 UTC
approved as os-kernel-20261114-075716
Version floors
Global controller floor: 0.352.1 · vouched agent: 0.174.0
[hover: A floor is the lowest controller version a box must run; the hub moves a box below it up.] What is a floor? · [hover: The agent version the operator checked and signed off for the fleet. Agents update only by a per-box signed job.] What is a vouched agent?
No per-customer floors: every box follows the global floor.
Crash guard and root files
Box ¦ kernel.panic ¦ Oops ¦ Crash restarts 24 h ¦ Crash guard ¦ [hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files ¦
Tester4-46836f ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] 0.173.0 ¦
demo-felhom-8363b5 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.174.0 ¦
demo-hp-bb76ea ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.174.0 ¦
tester-1-d70be4 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.174.0 ¦
Felhom Hub 0.182.0
@@ -0,0 +1,15 @@
# READS ONLY, on a box: what the agent did since a time ($1 = 'YYYY-MM-DD HH:MM:SS').
# The control is Proxmox's OWN access log (another channel than the agent's journal): calls by the agent's token
# that Proxmox answered, after the time.
set -u
T0="$1"; HM=$(date -d "$T0" +%H:%M:%S); D=$(date -d "$T0" +%d/%m/%Y)
echo "#### $(hostname) $(date +%F\ %T): since $T0"
echo "agent unit active since $(systemctl show felhom-agent -p ActiveEnterTimestamp --value), pid $(systemctl show felhom-agent -p MainPID --value), restarts $(systemctl show felhom-agent -p NRestarts --value)"
J=$(journalctl -u felhom-agent --since "$T0" --no-pager -o short-iso 2>&1 | grep -v "sudo\[")
echo "agent journal lines since: $(echo "$J" | wc -l)"
echo "trust / certificate lines: $(echo "$J" | grep -ciE "pvetrust|x509|certificate|tls: ")"; echo "$J" | grep -iE "pvetrust|x509|certificate|tls: " | cut -c1-360 | tail -8
echo "ERROR lines: $(echo "$J" | grep -c "level=ERROR")"; echo "$J" | grep "level=ERROR" | cut -c1-300 | tail -5
echo "Proxmox access log, the agent's token, after $HM:"
awk -v d="$D" -v hm="$HM" '$3 ~ /felhom-agent@pve/ { split($4,a,":"); day=substr(a[1],2); t=a[2]":"a[3]":"a[4]; if (day==d && t>=hm) { n++; c[$9]++; if (!first) first=$0; last=$0 } } END { printf " calls: %d; by status:", n; for (k in c) printf " %s=%d", k, c[k]; print ""; print " first: " substr(first,1,170); print " last: " substr(last,1,170) }' /var/log/pveproxy/access.log
echo "served now: $(echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256 2>/dev/null | cut -d= -f2)"
echo "#### end"
@@ -0,0 +1,23 @@
# Part 2 step 4, demo-hp ONLY: make the agent's CA copy wrong (another, self-made CA in its place), restart the agent,
# and watch. The copy is saved first and compared at the end. Changes: /etc/felhom-agent/pve-root-ca.pem (one file),
# one restart of felhom-agent. Nothing under /etc/pve is touched.
set -u
[ "$(hostname)" = demo-hp ] || { echo "REFUSED: this is not demo-hp"; exit 2; }
busy=$(ps -eo comm | grep -cE "^(vzdump|apt-get|dpkg|pct|proxmox-backup-)")
[ "$busy" = 0 ] || { echo "REFUSED: something is in flight"; exit 2; }
C=/etc/felhom-agent/pve-root-ca.pem; SAVE=/root/pve-root-ca.pem.saved-part2
W=$(mktemp -d)
cp -p "$C" "$SAVE"
echo "#### $(date +%F\ %T) the copy before: sha256 $(sha256sum < "$C" | cut -c1-16), $(stat -c '%A %U:%G' "$C"); saved at $SAVE"
openssl req -x509 -newkey rsa:2048 -nodes -days 3650 -subj "/CN=CC wrong CA for a test 2026-11-15" -addext "basicConstraints=critical,CA:TRUE" -keyout "$W/k.pem" -out "$W/wrong.pem" 2>/dev/null
cat "$W/wrong.pem" > "$C"
rm -rf "$W"
echo "the copy staged wrong: sha256 $(sha256sum < "$C" | cut -c1-16), subject $(openssl x509 -in "$C" -noout -subject)"
T0=$(date '+%F %T')
systemctl restart felhom-agent; echo "agent restart rc=$? at $T0"
for i in $(seq 1 18); do sleep 5; [ "$(sha256sum < "$C")" = "$(sha256sum < "$SAVE")" ] && break; done
echo "the copy $([ "$(sha256sum < "$C")" = "$(sha256sum < "$SAVE")" ] && echo "is the saved one again, by itself, after about $((i*5)) s" || echo "is STILL WRONG after 90 s")"
sleep 20
journalctl -u felhom-agent --since "$T0" --no-pager -o short-iso | grep -v "sudo\[" | grep -iE "pvetrust|x509|certificate|level=ERROR" | cut -c1-380 | tail -10
echo "T0=$T0"
echo "#### end"
@@ -0,0 +1,19 @@
# Part 2 step 3, on demo-hp ONLY: force a new Proxmox certificate the way the spike did (certpin-spike-2026-11-11),
# then watch the agent. Changes on the box: /etc/pve/local/pve-ssl.pem (+ key) is made new; pveproxy is reloaded.
set -u
[ "$(hostname)" = demo-hp ] || { echo "REFUSED: this is not demo-hp"; exit 2; }
busy=$(ps -eo comm | grep -cE "^(vzdump|apt-get|dpkg|pct|proxmox-backup-)" )
[ "$busy" = 0 ] || { echo "REFUSED: something is in flight ($busy)"; ps -eo comm | grep -E "^(vzdump|apt-get|dpkg|pct|proxmox-backup-)"; exit 2; }
served() { echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256 2>/dev/null | cut -d= -f2; }
echo "#### $(date +%F\ %T) round ${1:-1}: agent $(felhom-agent --version | awk '{print $2}'), unit active since $(systemctl show felhom-agent -p ActiveEnterTimestamp --value), pid $(systemctl show felhom-agent -p MainPID --value)"
echo "root CA before: $(openssl x509 -in /etc/pve/pve-root-ca.pem -noout -fingerprint -sha256 | cut -d= -f2)"
echo "served before: $(served)"
T0=$(date '+%F %T')
pvecm updatecerts --force 2>&1 | tail -3
echo "file after updatecerts: $(openssl x509 -in /etc/pve/local/pve-ssl.pem -noout -fingerprint -sha256 | cut -d= -f2)"
systemctl reload pveproxy; echo "pveproxy reload rc=$?"
for i in $(seq 1 12); do sleep 5; s=$(served); [ "$s" = "$(openssl x509 -in /etc/pve/local/pve-ssl.pem -noout -fingerprint -sha256 | cut -d= -f2)" ] && break; done
echo "served after: $s (after about $((i*5)) s)"
echo "root CA after: $(openssl x509 -in /etc/pve/pve-root-ca.pem -noout -fingerprint -sha256 | cut -d= -f2)"
echo "T0=$T0"
echo "#### end round"
@@ -0,0 +1,31 @@
# Tester 1 ONLY. clock.sh snap|back|right
# snap — READ: the clock, the time service, every container's id and creation time in guest 9201.
# back — stop the time service and set the clock 150 s BACK (the smaller harm: putting it right is then a
# step FORWARD, the direction a time service steps a late clock; nothing gets a stamp from the future
# while the clock is wrong, and no certificate or signed job ends early).
# right — start the time service and step the clock right.
set -u
grep -q '"tester-1' /etc/felhom-agent/agent.json 2>/dev/null || grep -q 'tester-1' /etc/felhom-agent/agent.json || { echo "REFUSED: this box is not Tester 1"; exit 2; }
snap() {
echo "## $(hostname) clock $(date '+%F %T.%N %Z' | cut -c1-27) epoch $(date +%s.%N | cut -c1-14); chrony $(systemctl is-active chrony); $(chronyc tracking 2>/dev/null | grep -E 'System time|Leap' | tr -s ' ' | tr '\n' ';')"
echo "## kernel boot_id $(cat /proc/sys/kernel/random/boot_id); btime $(awk '/^btime/ {print $2}' /proc/stat); uptime $(cut -d' ' -f1 /proc/uptime) s"
echo "## containers in 9201 (name id created):"
pct exec 9201 -- docker ps -a --format '{{.Names}} {{.ID}} {{.CreatedAt}} [{{.Status}}]' | sort | sed 's/^/ /'
echo "## count $(pct exec 9201 -- docker ps -q | wc -l) running"
}
case "${1:-}" in
snap) snap ;;
back)
busy=$(ps -eo comm | grep -cE "^(vzdump|apt-get|dpkg|proxmox-backup-)")
[ "$busy" = 0 ] || { echo "REFUSED: something is in flight"; exit 2; }
snap
systemctl stop chrony; echo "chrony stopped rc=$?"
date -s "@$(( $(date +%s) - 150 ))" >/dev/null; echo "clock set 150 s back rc=$? → $(date '+%F %T')"
;;
right)
systemctl start chrony; echo "chrony started rc=$?"
sleep 8; chronyc makestep; sleep 6
echo "clock now $(date '+%F %T'); $(chronyc tracking | grep -E 'System time|Leap' | tr -s ' ' | tr '\n' ';')"
;;
*) echo "usage: snap|back|right"; exit 2 ;;
esac
@@ -0,0 +1,10 @@
#### 2026-11-15 10:38:01 the copy before: sha256 eb0ce31aaa8d94ee, -rw-r--r-- root:root; saved at /root/pve-root-ca.pem.saved-part2
the copy staged wrong: sha256 47ea4b8f5ea6a13b, subject subject=CN=CC wrong CA for a test 2026-11-15
agent restart rc=0 at 2026-11-15 10:38:02
the copy is STILL WRONG after 90 s
2026-11-15T10:38:02+01:00 demo-hp felhom-agent[2251809]: time=2026-11-15T10:38:02.288+01:00 level=WARN msg="pvetrust: the CA copy was refreshed and proven — the running client uses it now" why=start mode=ca outcome=written
T0=2026-11-15 10:38:02
#### end
after a second start: certs 1; identical to the saved copy: yes; mode -rw-r--r-- root:root
2026-11-15T10:40:06+01:00 demo-hp felhom-agent[2259683]: time=2026-11-15T10:40:06.827+01:00 level=WARN msg="pvetrust: the CA copy was refreshed and proven — the running client uses it now" why=start mode=ca outcome=written
2026-11-15T10:40:06+01:00 demo-hp felhom-os-apply[2259713]: os-apply: PVE CA COPY the previous root is dropped from /etc/felhom-agent/pve-root-ca.pem (Proxmox serves a certificate of the new root)
@@ -0,0 +1,23 @@
#### 2026-11-15 10:35:05 round 1: agent 0.175.1, unit active since Sun 2026-11-15 10:18:18 CET, pid 2183005
root CA before: E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7
served before: 1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74
(re)generate node files
generate new node certificate
merge authorized SSH keys
file after updatecerts: AA:76:56:BD:A7:1C:3D:5D:1D:A5:2C:8F:5D:51:3F:F8:E3:B8:14:52:4E:90:95:9C:D8:E8:4E:78:0B:8C:15:45
pveproxy reload rc=0
served after: AA:76:56:BD:A7:1C:3D:5D:1D:A5:2C:8F:5D:51:3F:F8:E3:B8:14:52:4E:90:95:9C:D8:E8:4E:78:0B:8C:15:45 (after about 10 s)
root CA after: E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7
T0=2026-11-15 10:35:05
#### end round
#### demo-hp 2026-11-15 10:35:36: since 2026-11-15 10:35:16
agent unit active since Sun 2026-11-15 10:18:18 CET, pid 2183005, restarts 0
agent journal lines since: 4
trust / certificate lines: 0
ERROR lines: 0
Proxmox access log, the agent's token, after 10:35:16:
calls: 14; by status: 200=14
first: ::ffff:127.0.0.1 - felhom-agent@pve!agent [15/11/2026:10:35:20 +0100] "GET /api2/json/storage HTTP/1.1" 200 918
last: ::ffff:127.0.0.1 - felhom-agent@pve!agent [15/11/2026:10:35:33 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config HTTP/1.1" 200 673
served now: AA:76:56:BD:A7:1C:3D:5D:1D:A5:2C:8F:5D:51:3F:F8:E3:B8:14:52:4E:90:95:9C:D8:E8:4E:78:0B:8C:15:45
#### end
@@ -0,0 +1,37 @@
#### 2026-11-15 10:35:43 round 2: agent 0.175.1, unit active since Sun 2026-11-15 10:18:18 CET, pid 2183005
root CA before: E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7
served before: AA:76:56:BD:A7:1C:3D:5D:1D:A5:2C:8F:5D:51:3F:F8:E3:B8:14:52:4E:90:95:9C:D8:E8:4E:78:0B:8C:15:45
(re)generate node files
generate new node certificate
merge authorized SSH keys
file after updatecerts: EF:06:C8:01:9E:B2:34:82:97:DE:11:DA:8B:57:8E:56:0C:11:91:1F:17:C0:64:98:C9:1A:1F:C5:47:33:CE:E1
pveproxy reload rc=0
served after: EF:06:C8:01:9E:B2:34:82:97:DE:11:DA:8B:57:8E:56:0C:11:91:1F:17:C0:64:98:C9:1A:1F:C5:47:33:CE:E1 (after about 10 s)
root CA after: E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7
T0=2026-11-15 10:35:43
#### end round
#### demo-hp 2026-11-15 10:37:10: since 2026-11-14 20:35:44
agent unit active since Sun 2026-11-15 10:18:18 CET, pid 2183005, restarts 0
agent journal lines since: 6374
trust / certificate lines: 2
2026-11-15T10:18:18+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:18:18.654+01:00 level=INFO msg="pvetrust: the root files are older than this agent (no CA-copy step yet) — the pinned certificate stays; the next root-files update moves the box" why=start mode=fingerprint err="pve-ca-copy refused: {\"code\": \"R11\", \"reason\": \"unknown mode 'pv
2026-11-15T10:33:34+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:34.354+01:00 level=WARN msg="pvetrust: MOVED — the agent trusts Proxmox by the box's own root CA now; the pinned fingerprint was cleared AFTER a working call with the copy" why="config bundle installed" mode=fingerprint ca_file=/etc/felhom-agent/pve-root-ca.pem ca_not_after=2036
ERROR lines: 0
Proxmox access log, the agent's token, after 20:35:44:
calls: 0; by status:
first:
last:
served now: EF:06:C8:01:9E:B2:34:82:97:DE:11:DA:8B:57:8E:56:0C:11:91:1F:17:C0:64:98:C9:1A:1F:C5:47:33:CE:E1
#### end
## read again with the right time (the first read's time was computed wrong: 'date -d ... + 15 seconds')
#### demo-hp 2026-11-15 10:37:18: since 2026-11-15 10:35:58
agent unit active since Sun 2026-11-15 10:18:18 CET, pid 2183005, restarts 0
agent journal lines since: 8
trust / certificate lines: 0
ERROR lines: 0
Proxmox access log, the agent's token, after 10:35:58:
calls: 40; by status: 200=40
first: ::ffff:127.0.0.1 - felhom-agent@pve!agent [15/11/2026:10:36:00 +0100] "GET /api2/json/storage HTTP/1.1" 200 918
last: ::ffff:127.0.0.1 - felhom-agent@pve!agent [15/11/2026:10:37:03 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config HTTP/1.1" 200 673
served now: EF:06:C8:01:9E:B2:34:82:97:DE:11:DA:8B:57:8E:56:0C:11:91:1F:17:C0:64:98:C9:1A:1F:C5:47:33:CE:E1
#### end
@@ -0,0 +1,31 @@
#### demo-hp 2026-11-15 10:34:56 CET agent 0.175.1 uptime-of-unit Sun 2026-11-15 10:18:18 CET
config proxmox.tls: ca_file='/etc/felhom-agent/pve-root-ca.pem' fingerprint=EMPTY insecure_skip_verify=False
-rw-r--r-- root:root 2094 bytes 2026-11-15 10:33:34.311028892 +0100 /etc/felhom-agent/pve-root-ca.pem
drwxr-xr-x root:root 4096 bytes 2026-11-15 10:33:34.317623237 +0100 /etc/felhom-agent
-rw------- felhom-agent:felhom-agent 2455 bytes 2026-11-15 10:33:34.34 /var/lib/felhom-agent/agent.json.pre-pve-trust
(absent) /var/lib/felhom-agent/agent.json.write-in-flight
CA copy /etc/felhom-agent/pve-root-ca.pem: 1 certificate(s); first: subject=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA notAfter=Aug 17 15:44:42 2036 GMT sha256 Fingerprint=E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7
same as /etc/pve/pve-root-ca.pem: yes
agent user can read it: yes; others can read: r
served certificate (127.0.0.1:8006): sha256 Fingerprint=1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74 notAfter=Nov 9 13:59:54 2028 GMT
/etc/pve/local/pve-ssl.pem: sha256 Fingerprint=1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74 notBefore=Nov 10 13:59:54 2026 GMT
## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min)
2026-11-15T10:18:15+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T10:18:15.625+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=90d84d793f66c18d op=agent_update key_id=felhom-op-1 nonce=9e1c24967ec755838aeda82ce2520f14
2026-11-15T10:18:15+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T10:18:15.872+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.175.1 version=0.175.1
2026-11-15T10:18:16+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T10:18:16.400+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.175.1 report="{\"agent_update\": {\"sha256\": \"38b1e2350de90a7940c16f8cd2a21d83fcd535410d1017a7df0fee14f0904612\", \"version\": \"0.175.1\", \"wrapper\": \"Running timer as unit: run-p2182880-i23144936.timer\\nWill run service as u
2026-11-15T10:18:16+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T10:18:16.401+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=90d84d793f66c18d op=agent_update
2026-11-15T10:18:18+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:18:18.654+01:00 level=INFO msg="pvetrust: the root files are older than this agent (no CA-copy step yet) — the pinned certificate stays; the next root-files update moves the box" why=start mode=fingerprint err="pve-ca-copy refused: {\"code\": \"R11\", \"reason\": \"unknown mode 'pve-ca-copy'\"}"
2026-11-15T10:18:20+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:18:20.815+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.175.1 prev=0.174.0 dwell=1m0s
2026-11-15T10:19:20+01:00 demo-hp felhom-selfupdate-guarded[2187989]: committed (pending cleared, .prev retained)
2026-11-15T10:19:20+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:19:20.843+01:00 level=WARN msg="selfupdate: update committed" version=0.175.1 wrapper=""
2026-11-15T10:19:20+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:19:20.962+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.175.1 removed=1
2026-11-15T10:19:20+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:19:20.962+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=1 kept=felhom-agent-0.175.1
2026-11-15T10:33:31+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:31.703+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=9b72aa335a322ba6 op=agent_config_update key_id=felhom-op-1 nonce=c93c80251a155095f98d648bf9b16f51
2026-11-15T10:33:31+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:31.813+01:00 level=INFO msg="osupdate: config bundle downloaded; handing it to the root wrapper" op=agent_config_update agent_version=0.175.1 sha256=07ebf86362ed9d44 duration_ms=110
2026-11-15T10:33:33+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:33.036+01:00 level=WARN msg="osupdate: config bundle INSTALLED" op=agent_config_update agent_version=0.175.1 bundle="{\"agent_version\": \"0.175.1\", \"authority\": \"signed\", \"kept\": [\"/etc/felhom/crash-guard.conf\"], \"prev_dir\": \"/var/lib/felhom-os-apply/bundle-prev/20261115T093332Z-before-0.175.1\", \"removed\": [], \"same\": 30, \"self_ch
2026-11-15T10:33:34+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:34.180+01:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=77 total=77 degraded=""
2026-11-15T10:33:34+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:34.180+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=9b72aa335a322ba6 op=agent_config_update
2026-11-15T10:33:34+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:34.293+01:00 level=INFO msg="osupdate: profile refresh not needed" mode=profile-refresh why="config bundle installed" kernel=7.0.14-28-pve reason="the profiles were refreshed on this kernel (7.0.14-28-pve) at 2026-11-14T11:33:18Z"
2026-11-15T10:33:34+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:34.354+01:00 level=WARN msg="pvetrust: MOVED — the agent trusts Proxmox by the box's own root CA now; the pinned fingerprint was cleared AFTER a working call with the copy" why="config bundle installed" mode=fingerprint ca_file=/etc/felhom-agent/pve-root-ca.pem ca_not_after=2036-08-17T15:44:42Z
## one control: the agent's newest successful host report and newest Proxmox call error
#### end
@@ -0,0 +1,45 @@
== MUTATION: older root files are named as a trust problem again
RED on the mutant, GREEN on the real file
--- FAIL: TestRootStepOlderThanTheAgent_IsNotATrustProblem (0.00s)
pvetrust_test.go:507: a pinned box with older root files reports a trust problem: &{Mode:fingerprint Problem:ca_copy_unavailable Detail:pve-ca-copy refused: {"code": "R11", "reason": "unknown mode 'pve-ca-copy'"} CANotAfter: CADay
== MUTATION: an UNPINNED box with no copy step is silent too
RED on the mutant, GREEN on the real file
--- FAIL: TestRootStepOlderThanTheAgent_IsNotATrustProblem (0.00s)
pvetrust_test.go:518: an unpinned box with no copy step must name the problem: &{Mode:system Problem: Detail: CANotAfter: CADaysLeft:<nil> MovedAt: CheckedAt:2026-11-15T09:05:53Z}
== MUTATION: every refusal is silent
*** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0)
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/pvetrust [build failed]
3 mutations, 1 not a red-proof
== MUTATION: every refusal is silent (redone: the first mutant did not compile)
RED on the mutant, GREEN on the real file
--- FAIL: TestRootStepRefuses_NothingChanges (0.00s)
pvetrust_test.go:374: status = &{Mode:fingerprint Problem: Detail: CANotAfter: CADaysLeft:<nil> MovedAt: CheckedAt:2026-11-15T09:06:04Z}
--- FAIL: TestRootStepOlderThanTheAgent_IsNotATrustProblem (0.00s)
pvetrust_test.go:526: a real refusal lost its name: &{Mode:fingerprint Problem: Detail: CANotAfter: CADaysLeft:<nil> MovedAt: CheckedAt:2026-11-15T09:06:04Z}
1 mutations, 0 not a red-proof
## after the commit's security review: the quiet branch never hides a failed pin
== MUTATION: a run started by a failed call takes the quiet branch
RED on the mutant, GREEN on the real file
--- FAIL: TestRootStepOlderThanTheAgent_NeverHidesAFailedPin (0.00s)
pvetrust_test.go:543: a run started by a failed pin reports no problem: &{Mode:fingerprint Problem: Detail: CANotAfter: CADaysLeft:<nil> MovedAt: CheckedAt:2026-11-15T09:07:36Z}
== MUTATION: the timer clears a recorded pin failure
*** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0)
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/pvetrust [build failed]
internal/pvetrust/pvetrust.go:251:3: declared and not used: recorded
== MUTATION: any refusal that says unknown mode counts, without the code
*** NOT A RED-PROOF *** (mutant rc=0 built=True real rc=0)
3 mutations, 2 not a red-proof
== MUTATION: the timer clears a recorded pin failure (redone: the first mutant did not compile)
RED on the mutant, GREEN on the real file
--- FAIL: TestRootStepOlderThanTheAgent_NeverHidesAFailedPin (0.00s)
pvetrust_test.go:564: the timer cleared a recorded pin failure: &{Mode:fingerprint Problem: Detail: CANotAfter: CADaysLeft:<nil> MovedAt: CheckedAt:2026-11-15T09:07:52Z}
== MUTATION: any refusal that says unknown mode counts, without the code (redone: the test had no such case; one was added)
RED on the mutant, GREEN on the real file
--- FAIL: TestRootStepOlderThanTheAgent_IsNotATrustProblem (0.00s)
pvetrust_test.go:536: a refusal with another code was read as older root files: &{Mode:fingerprint Problem: Detail: CANotAfter: CADaysLeft:<nil> MovedAt: CheckedAt:2026-11-15T09:07:54Z}
2 mutations, 0 not a red-proof
@@ -0,0 +1,265 @@
System — Felhom Hub
Felhom Hub
Dashboard
Customers
Apps
Hosts
System
Offsite
Configuration
System — versions and OS updates
Needs attention
One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3).
[hover: worth a look] look
Tester4-46836f Tester4
[hover: cannot do a one-shot boot: the box does not boot UEFI; /boot/efi is not a mounted vfat ESP (not mounted)] kernel step: — · the host needs a restart since 2026-11-14 · [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] root files behind the vouched agent's · [hover: 1 minor release behind — sign an agent_update for this box] agent behind: 0.173.0 → 0.174.0 (since 2026-11-14) · [hover: the box could not read these values (agent older than v0.142.0, or the guest is down) — never a guess] could not read: default kernel
[hover: worth a look] look
demo-felhom-8363b5 Demo Ügyfél
[hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update] root files behind the vouched agent's
[hover: worth a look] look
demo-hp-bb76ea Demo HP
[hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-15 09:51 UTC)] root files behind the vouched agent's
[hover: worth a look] look
tester-1-d70be4 Tester 1
[hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-15 09:51 UTC)] root files behind the vouched agent's
Waiting for you
Update sets that only you approve: the kernel, the Docker engine and the Proxmox packages. Guest and host Debian updates approve themselves after 24 h and one night on the [hover: Ring 0 = the demo boxes. They install every new fix first; the other boxes (ring 1) install only what is approved.] ring-0 boxes.
Docker engine 29.9.3-1
Ran on every ring-0 box — demo-felhom-8363b5: 2 healthy night(s), demo-hp-bb76ea: 2 healthy night(s). The memory-kill check passed.
first seen 2 days ago (2026-11-13 04:49 UTC). Approving installs nothing by itself: a ring-1 box takes it only through a signed operator job.
Approve Docker set
Proxmox packages 9.2.27 — still being tested: demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set
Boxes
Click a box to see every value it reports, and its ring and update switches. "unknown": the box could not read the value — never a guess.
Box Ring Health Controller · agent OS updates Kernel [hover: The newest OS update run] Last night
Tester4-46836f Tester4
[hover: Ring 1: a normal box — it installs only approved releases] 1
[hover: worth a look] look
0.352.1 [hover: 1 minor release behind — sign an agent_update for this box] agent 0.173.0 → 0.174.0 (since 2026-11-14)
[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind
7.0.2-6-pve
[hover: 23 h ago · applied · 93 s — last successful leg: 23 h ago] ok · 23 h ago
Ring and updates
Ring ring 1
→ demo
OS updates updates ON
switch off
Tunnel [hover: no cloudflared container in the guest (agent state: absent)] none set up
Controller 0.352.1
Agent [hover: 1 minor release behind — sign an agent_update for this box] 0.173.0 → 0.174.0 (since 2026-11-14)
Last OS leg [hover: last successful leg: 23 h ago] 23 h ago · applied · 93 s
Docker engine
Docker 29.9.2
containerd 2.4.2-3~debian.13~trixie
live-restore on
Docker release —
Host
Proxmox 9.2.2
Kernel (running) 7.0.2-6-pve
Kernel (next boot) 7.0.2-6-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: cannot do a one-shot boot: the box does not boot UEFI; /boot/efi is not a mounted vfat ESP (not mounted)] —
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] —
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: the box reports no security-profile refresh (an agent older than release 3, a BYO host, or no refresh yet)] —
Debian 13.7
Felhom release os-host-20261111-034654
Pending 100
Not covered 0
Held none
Reboot needed since 2026-11-14
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] 0.173.0
Guest
Guest Debian 13.7
Felhom release os-guest-20261111-034654
Pending 6
Restart needed 7
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 23 h ago · 78.4 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] not yet
demo-felhom-8363b5 Demo Ügyfél
[hover: Ring 0: a demo box — it installs every new fix first] 0
[hover: worth a look] look
0.352.1 [hover: current (vouched 0.174.0)] agent 0.175.1
[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind
7.0.14-28-pve
[hover: 5 h ago · nothing · 23 s — last successful leg: 5 h ago] ok · 5 h ago
Ring and updates
Ring ring 0
→ normal
OS updates updates ON
switch off
Tunnel running
Controller 0.352.1
Agent [hover: current (vouched 0.174.0)] 0.175.1
Last OS leg [hover: last successful leg: 5 h ago] 5 h ago · nothing · 23 s
Docker engine
Docker 29.9.3
containerd 2.4.2-4~debian.13~trixie
live-restore on
Docker release ring0-20261115T045634Z
Host
Proxmox 9.2.27
Kernel (running) 7.0.14-28-pve
Kernel (next boot) 7.0.14-28-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 1m14s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 7 days ago · due 7.0.14-29-pve — household told 2 h ago: TONIGHT · phase good
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: removed 6 old kernel package(s): proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed; kept 7.0.14-26-pve, 7.0.14-28-pve; 6.3 GiB gained; 0 left-over path(s)] cleaned 5 h ago
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: not_needed] 7.0.14-28-pve · 22 h ago
Debian 13.7
Felhom release ring0-20261115T045634Z
Pending 29
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update] 0.175.1
Guest
Guest Debian 13.7
Felhom release ring0-20261115T045634Z
Pending 0
Restart needed 1
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 4 days ago · 272.7 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 92.7 Mbit/s · 5 days ago
demo-hp-bb76ea Demo HP
[hover: Ring 0: a demo box — it installs every new fix first] 0
[hover: worth a look] look
0.352.1 [hover: current (vouched 0.174.0)] agent 0.175.1
[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind
7.0.14-28-pve
[hover: 6 h ago · nothing · 30 s — last successful leg: 6 h ago] ok · 6 h ago
Ring and updates
Ring ring 0
→ normal
OS updates updates ON
switch off
Tunnel running
Controller 0.352.1
Agent [hover: current (vouched 0.174.0)] 0.175.1
Last OS leg [hover: last successful leg: 6 h ago] 6 h ago · nothing · 30 s
Docker engine
Docker 29.9.3
containerd 2.4.2-4~debian.13~trixie
live-restore on
Docker release ring0-20261115T040248Z
Host
Proxmox 9.2.27
Kernel (running) 7.0.14-28-pve
Kernel (next boot) 7.0.14-28-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 2m39s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 5 days ago · due 7.0.14-29-pve — household told 2 h ago: TONIGHT · phase good
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] —
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: not_needed] 7.0.14-28-pve · 22 h ago
Debian 13.7
Felhom release ring0-20261115T040248Z
Pending 30
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-15 09:51 UTC)] 0.175.1
Guest
Guest Debian 13.7
Felhom release ring0-20261115T040248Z
Pending 0
Restart needed 1
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 4 days ago · 70.5 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 94.3 Mbit/s · 4 days ago
tester-1-d70be4 Tester 1
[hover: Ring 1: a normal box — it installs only approved releases] 1
[hover: worth a look] look
0.352.1 [hover: current (vouched 0.174.0)] agent 0.175.1
[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind
7.0.14-22-pve
[hover: 5 h ago · nothing · 16 s — last successful leg: 5 h ago] ok · 5 h ago
Ring and updates
Ring ring 1
→ demo
OS updates updates ON
switch off
Tunnel running
Controller 0.352.1
Agent [hover: current (vouched 0.174.0)] 0.175.1
Last OS leg [hover: last successful leg: 5 h ago] 5 h ago · nothing · 16 s
Docker engine
Docker 29.8.2
containerd 2.3.6-1~debian.13~trixie
live-restore on
Docker release —
Host
Proxmox 9.2.2
Kernel (running) 7.0.14-22-pve
Kernel (next boot) 7.0.14-22-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-22-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: ring 1: no kernel staged by a signed os_kernel_step healthy 1m17s after the agent started; the new kernel is the default] 7.0.14-22-pve applied 38 days ago · phase good
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] —
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: not_needed] 7.0.14-22-pve · 22 h ago
Debian 13.7
Felhom release os-host-20261111-034654
Pending 101
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-15 09:51 UTC)] 0.175.1
Guest
Guest Debian 13.7
Felhom release os-guest-20261111-034654
Pending 6
Restart needed 0
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 4 days ago · 90.4 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 93.0 Mbit/s · 4 days ago
Details
Release ids, cancelled approvals, what ring 0 runs, the version floors, and every box's crash guard and root files.
Approved releases
The newest approved set of each layer. A [hover: Approved while a TEST wait override was active. It is cancelled when the hub starts without the override (`11` §5.3.1).] TEST approval is amber.
guest : os-guest-20261111-034654
272 packages · 2026-11-11 03:46 UTC · by auto
host : os-host-20261115-045718
607 packages · 2026-11-15 04:57 UTC · by auto
docker : os-docker-20261026-190351
6 packages · 2026-10-26 19:03 UTC · by operator
pve : os-pve-20261114-075712
121 packages · 2026-11-14 07:57 UTC · by operator
kernel : os-kernel-20261114-075716
2 packages · 2026-11-14 07:57 UTC · by operator
Approve now (guest + host)
An urgent fix only — normally the hub approves after 24 h and one night.
What ring 0 runs now
guest :
272 packages, first seen 2026-11-10 01:49 UTC
approved as os-guest-20261111-034654
host :
607 packages, first seen 2026-11-14 04:52 UTC
approved as os-host-20261115-045718
docker :
6 packages, first seen 2026-11-13 04:49 UTC
ready — see "Waiting for you"
pve :
121 packages, first seen 2026-11-15 04:58 UTC
demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set
kernel :
2 packages, first seen 2026-11-10 01:57 UTC
approved as os-kernel-20261114-075716
Version floors
Global controller floor: 0.352.1 · vouched agent: 0.174.0
[hover: A floor is the lowest controller version a box must run; the hub moves a box below it up.] What is a floor? · [hover: The agent version the operator checked and signed off for the fleet. Agents update only by a per-box signed job.] What is a vouched agent?
No per-customer floors: every box follows the global floor.
Crash guard and root files
Box ¦ kernel.panic ¦ Oops ¦ Crash restarts 24 h ¦ Crash guard ¦ [hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files ¦
Tester4-46836f ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] 0.173.0 ¦
demo-felhom-8363b5 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update] 0.175.1 ¦
demo-hp-bb76ea ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-15 09:51 UTC)] 0.175.1 ¦
tester-1-d70be4 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-15 09:51 UTC)] 0.175.1 ¦
Felhom Hub 0.182.0
@@ -0,0 +1,17 @@
#### felhom 2026-11-15 10:04:46 CET agent 0.175.0 uptime-of-unit Sun 2026-11-15 10:04:33 CET
config proxmox.tls: ca_file='' fingerprint=set (BD:EE:9F:C4…) insecure_skip_verify=False
(absent) /etc/felhom-agent/pve-root-ca.pem
drwxr-xr-x root:root 4096 bytes 2026-10-04 21:40:41.128703040 +0200 /etc/felhom-agent
(absent) /var/lib/felhom-agent/agent.json.pre-pve-trust
(absent) /var/lib/felhom-agent/agent.json.write-in-flight
served certificate (127.0.0.1:8006): sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notAfter=Oct 2 19:38:05 2028 GMT
/etc/pve/local/pve-ssl.pem: sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notBefore=Oct 3 19:38:05 2026 GMT
## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min)
2026-11-15T10:04:30+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:30.371+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=b5c45465f233eb81 op=agent_update key_id=felhom-op-1 nonce=d5fd87f441408954947c67b4b44ab6b2
2026-11-15T10:04:30+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:30.619+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.175.0 version=0.175.0
2026-11-15T10:04:31+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:31.020+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.175.0 report="{\"agent_update\": {\"sha256\": \"d1a30ed51f6663d96369ad3e88c221194af92773addf4fbb48852ee58be4b19a\", \"version\": \"0.175.0\", \"wrapper\": \"Running timer as unit: run-p1546906-i1549255.timer\\nWill run service as unit:
2026-11-15T10:04:31+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:31.020+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=b5c45465f233eb81 op=agent_update
2026-11-15T10:04:33+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:04:33.269+01:00 level=WARN msg="pvetrust: the root step gave no CA copy — the trust stays as it is" why=start mode=fingerprint err="pve-ca-copy refused: {\"code\": \"R11\", \"reason\": \"unknown mode 'pve-ca-copy'\"}"
2026-11-15T10:04:34+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:04:34.922+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.175.0 prev=0.174.0 dwell=1m0s
## one control: the agent's newest successful host report and newest Proxmox call error
#### end
@@ -0,0 +1,21 @@
#### felhom 2026-11-15 10:09:02 CET agent 0.175.0 uptime-of-unit Sun 2026-11-15 10:04:33 CET
config proxmox.tls: ca_file='' fingerprint=set (BD:EE:9F:C4…) insecure_skip_verify=False
(absent) /etc/felhom-agent/pve-root-ca.pem
drwxr-xr-x root:root 4096 bytes 2026-10-04 21:40:41.128703040 +0200 /etc/felhom-agent
(absent) /var/lib/felhom-agent/agent.json.pre-pve-trust
(absent) /var/lib/felhom-agent/agent.json.write-in-flight
served certificate (127.0.0.1:8006): sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notAfter=Oct 2 19:38:05 2028 GMT
/etc/pve/local/pve-ssl.pem: sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notBefore=Oct 3 19:38:05 2026 GMT
## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min)
2026-11-15T10:04:30+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:30.371+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=b5c45465f233eb81 op=agent_update key_id=felhom-op-1 nonce=d5fd87f441408954947c67b4b44ab6b2
2026-11-15T10:04:30+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:30.619+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.175.0 version=0.175.0
2026-11-15T10:04:31+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:31.020+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.175.0 report="{\"agent_update\": {\"sha256\": \"d1a30ed51f6663d96369ad3e88c221194af92773addf4fbb48852ee58be4b19a\", \"version\": \"0.175.0\", \"wrapper\": \"Running timer as unit: run-p1546906-i1549255.timer\\nWill run service as unit:
2026-11-15T10:04:31+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:31.020+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=b5c45465f233eb81 op=agent_update
2026-11-15T10:04:33+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:04:33.269+01:00 level=WARN msg="pvetrust: the root step gave no CA copy — the trust stays as it is" why=start mode=fingerprint err="pve-ca-copy refused: {\"code\": \"R11\", \"reason\": \"unknown mode 'pve-ca-copy'\"}"
2026-11-15T10:04:34+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:04:34.922+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.175.0 prev=0.174.0 dwell=1m0s
2026-11-15T10:05:34+01:00 felhom felhom-selfupdate-guarded[1549546]: committed (pending cleared, .prev retained)
2026-11-15T10:05:34+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:05:34.955+01:00 level=WARN msg="selfupdate: update committed" version=0.175.0 wrapper=""
2026-11-15T10:05:35+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:05:35.150+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.175.0 removed=1
2026-11-15T10:05:35+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:05:35.150+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=1 kept=felhom-agent-0.175.0
## one control: the agent's newest successful host report and newest Proxmox call error
#### end
@@ -0,0 +1,11 @@
#### felhom 2026-11-15 09:51:08 CET agent 0.174.0 uptime-of-unit Sat 2026-11-14 15:19:22 CET
config proxmox.tls: ca_file='' fingerprint=set (BD:EE:9F:C4…) insecure_skip_verify=False
(absent) /etc/felhom-agent/pve-root-ca.pem
drwxr-xr-x root:root 4096 bytes 2026-10-04 21:40:41.128703040 +0200 /etc/felhom-agent
(absent) /var/lib/felhom-agent/agent.json.pre-pve-trust
(absent) /var/lib/felhom-agent/agent.json.write-in-flight
served certificate (127.0.0.1:8006): sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notAfter=Oct 2 19:38:05 2028 GMT
/etc/pve/local/pve-ssl.pem: sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notBefore=Oct 3 19:38:05 2026 GMT
## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min)
## one control: the agent's newest successful host report and newest Proxmox call error
#### end
@@ -0,0 +1,14 @@
## felhom clock 2026-11-15 10:49:52.0408764 epoch 1794736192.042; chrony inactive;
## kernel boot_id 7a721957-7e95-476c-be40-52d0dafc7284; btime 1794665801; uptime 70390.82 s
## containers in 9201 (name id created):
bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 5 hours (healthy)]
bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 5 hours (healthy)]
cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 20 hours (healthy)]
felhom-controller 2a7970a2c8a7 2026-11-14 14:19:33 +0000 UTC [Up 20 hours (healthy)]
filebrowser 95f38571abee 2026-11-14 14:19:57 +0000 UTC [Up 19 hours (healthy)]
paperless-postgres c7807ae06022 2026-11-15 09:48:39 +0000 UTC [Up About a minute (healthy)]
paperless-redis 56d085b17130 2026-11-15 09:48:39 +0000 UTC [Up About a minute (healthy)]
paperless-webserver 71aee1583019 2026-11-15 09:48:39 +0000 UTC [Up About a minute (healthy)]
privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 5 hours (healthy)]
traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 20 hours]
## count 10 running
@@ -0,0 +1,14 @@
## felhom clock 2026-11-15 11:06:49.5647785 epoch 1794737209.566; chrony active; System time : 0.001194180 seconds slow of NTP time;Leap status : Normal;
## kernel boot_id 7a721957-7e95-476c-be40-52d0dafc7284; btime 1794665951; uptime 71258.03 s
## containers in 9201 (name id created):
bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 5 hours (healthy)]
bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 5 hours (healthy)]
cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 20 hours (healthy)]
felhom-controller 2a7970a2c8a7 2026-11-14 14:19:33 +0000 UTC [Up 20 hours (healthy)]
filebrowser 95f38571abee 2026-11-14 14:19:57 +0000 UTC [Up 20 hours (healthy)]
paperless-postgres 31b2490aa677 2026-11-15 10:05:37 +0000 UTC [Up About a minute (healthy)]
paperless-redis df1c44607a9b 2026-11-15 10:05:37 +0000 UTC [Up About a minute (healthy)]
paperless-webserver 1f2a0d7e4f7b 2026-11-15 10:05:37 +0000 UTC [Up About a minute (healthy)]
privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 5 hours (healthy)]
traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 20 hours]
## count 10 running
@@ -0,0 +1,16 @@
## felhom clock 2026-11-15 10:50:27.2531193 epoch 1794736227.254; chrony active; System time : 0.000242970 seconds slow of NTP time;Leap status : Normal;
## kernel boot_id 7a721957-7e95-476c-be40-52d0dafc7284; btime 1794665951; uptime 70275.75 s
## containers in 9201 (name id created):
bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 5 hours (healthy)]
bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 5 hours (healthy)]
cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 20 hours (healthy)]
felhom-controller 2a7970a2c8a7 2026-11-14 14:19:33 +0000 UTC [Up 20 hours (healthy)]
filebrowser 95f38571abee 2026-11-14 14:19:57 +0000 UTC [Up 20 hours (healthy)]
paperless-postgres e0d945977ebc 2026-11-15 04:50:09 +0000 UTC [Up 5 hours (healthy)]
paperless-redis 6188dea5adbc 2026-11-15 04:50:09 +0000 UTC [Up 5 hours (healthy)]
paperless-webserver cc93a8574d66 2026-11-15 04:50:10 +0000 UTC [Up 5 hours (healthy)]
privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 5 hours (healthy)]
traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 20 hours]
## count 10 running
chrony stopped rc=0
clock set 150 s back rc=0 → 2026-11-15 10:47:59
@@ -0,0 +1,4 @@
chrony started rc=0
200 OK
clock now 2026-11-15 11:05:36; System time : 0.000000000 seconds slow of NTP time;Leap status : Normal;
Tester 1 minus DooPlex: +0.073 s
@@ -0,0 +1,31 @@
#### felhom 2026-11-15 10:20:19 CET agent 0.175.0 uptime-of-unit Sun 2026-11-15 10:04:33 CET
config proxmox.tls: ca_file='/etc/felhom-agent/pve-root-ca.pem' fingerprint=EMPTY insecure_skip_verify=False
-rw-r--r-- root:root 2094 bytes 2026-11-15 10:19:47.714398283 +0100 /etc/felhom-agent/pve-root-ca.pem
drwxr-xr-x root:root 4096 bytes 2026-11-15 10:19:47.716626910 +0100 /etc/felhom-agent
-rw------- felhom-agent:felhom-agent 2451 bytes 2026-11-15 10:19:47.73 /var/lib/felhom-agent/agent.json.pre-pve-trust
(absent) /var/lib/felhom-agent/agent.json.write-in-flight
CA copy /etc/felhom-agent/pve-root-ca.pem: 1 certificate(s); first: subject=CN=Proxmox Virtual Environment, OU=f9c69a56-050d-43dd-87c8-d596f6a79d19, O=PVE Cluster Manager CA notAfter=Sep 30 19:38:05 2036 GMT sha256 Fingerprint=4E:46:8D:BC:45:A2:2C:CD:AB:EF:82:C0:A4:B1:4D:6C:5C:DA:5A:7F:6F:D7:3B:E9:B7:50:CA:ED:C4:0C:80:CF
same as /etc/pve/pve-root-ca.pem: yes
agent user can read it: yes; others can read: r
served certificate (127.0.0.1:8006): sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notAfter=Oct 2 19:38:05 2028 GMT
/etc/pve/local/pve-ssl.pem: sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notBefore=Oct 3 19:38:05 2026 GMT
## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min)
2026-11-15T10:04:30+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:30.371+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=b5c45465f233eb81 op=agent_update key_id=felhom-op-1 nonce=d5fd87f441408954947c67b4b44ab6b2
2026-11-15T10:04:30+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:30.619+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.175.0 version=0.175.0
2026-11-15T10:04:31+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:31.020+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.175.0 report="{\"agent_update\": {\"sha256\": \"d1a30ed51f6663d96369ad3e88c221194af92773addf4fbb48852ee58be4b19a\", \"version\": \"0.175.0\", \"wrapper\": \"Running timer as unit: run-p1546906-i1549255.timer\\nWill run service as unit:
2026-11-15T10:04:31+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:31.020+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=b5c45465f233eb81 op=agent_update
2026-11-15T10:04:33+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:04:33.269+01:00 level=WARN msg="pvetrust: the root step gave no CA copy — the trust stays as it is" why=start mode=fingerprint err="pve-ca-copy refused: {\"code\": \"R11\", \"reason\": \"unknown mode 'pve-ca-copy'\"}"
2026-11-15T10:04:34+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:04:34.922+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.175.0 prev=0.174.0 dwell=1m0s
2026-11-15T10:05:34+01:00 felhom felhom-selfupdate-guarded[1549546]: committed (pending cleared, .prev retained)
2026-11-15T10:05:34+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:05:34.955+01:00 level=WARN msg="selfupdate: update committed" version=0.175.0 wrapper=""
2026-11-15T10:05:35+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:05:35.150+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.175.0 removed=1
2026-11-15T10:05:35+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:05:35.150+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=1 kept=felhom-agent-0.175.0
2026-11-15T10:19:45+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:45.500+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=0c91a3e29ac9ba27 op=agent_config_update key_id=felhom-op-1 nonce=cb391943ab870fd924c8b90ef98b2065
2026-11-15T10:19:45+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:45.543+01:00 level=INFO msg="osupdate: config bundle downloaded; handing it to the root wrapper" op=agent_config_update agent_version=0.175.0 sha256=3b4a8b78155fc438 duration_ms=42
2026-11-15T10:19:46+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:46.553+01:00 level=WARN msg="osupdate: config bundle INSTALLED" op=agent_config_update agent_version=0.175.0 bundle="{\"agent_version\": \"0.175.0\", \"authority\": \"signed\", \"kept\": [\"/etc/felhom/crash-guard.conf\"], \"prev_dir\": \"/var/lib/felhom-os-apply/bundle-prev/20261115T091945Z-before-0.175.0\", \"removed\": [], \"same\": 30, \"self_che
2026-11-15T10:19:47+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:47.590+01:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=77 total=77 degraded=""
2026-11-15T10:19:47+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:47.590+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=0c91a3e29ac9ba27 op=agent_config_update
2026-11-15T10:19:47+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:47.685+01:00 level=INFO msg="osupdate: profile refresh not needed" mode=profile-refresh why="config bundle installed" kernel=7.0.14-22-pve reason="the profiles were refreshed on this kernel (7.0.14-22-pve) at 2026-11-14T11:30:11Z"
2026-11-15T10:19:47+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:47.744+01:00 level=WARN msg="pvetrust: MOVED — the agent trusts Proxmox by the box's own root CA now; the pinned fingerprint was cleared AFTER a working call with the copy" why="config bundle installed" mode=fingerprint ca_file=/etc/felhom-agent/pve-root-ca.pem ca_not_after=2036-09-30T19:38:05Z
## one control: the agent's newest successful host report and newest Proxmox call error
#### end
@@ -0,0 +1,23 @@
# Part 2 — READS ONLY: the agent's version and its trust in Proxmox, on one box. Never prints a token or a key.
set -u
echo "#### $(hostname) $(date +%F\ %T\ %Z) agent $(felhom-agent --version 2>&1 | awk '{print $2}') uptime-of-unit $(systemctl show felhom-agent -p ActiveEnterTimestamp --value)"
python3 - <<'PY'
import json
d = json.load(open("/etc/felhom-agent/agent.json"))
t = d.get("proxmox", {}).get("tls", {})
fp = t.get("fingerprint", "")
print("config proxmox.tls: ca_file=%r fingerprint=%s insecure_skip_verify=%r" % (t.get("ca_file", ""), ("set (%s…)" % fp[:11]) if fp else "EMPTY", t.get("insecure_skip_verify")))
PY
for f in /etc/felhom-agent/pve-root-ca.pem /etc/felhom-agent /var/lib/felhom-agent/agent.json.pre-pve-trust /var/lib/felhom-agent/agent.json.write-in-flight; do
[ -e "$f" ] && echo " $(stat -c '%A %U:%G %s bytes %y' "$f" | cut -c1-70) $f" || echo " (absent) $f"; done
CA=$(python3 -c 'import json;print(json.load(open("/etc/felhom-agent/agent.json")).get("proxmox",{}).get("tls",{}).get("ca_file",""))')
for c in "$CA" /etc/felhom-agent/pve-root-ca.pem; do
[ -n "$c" ] && [ -e "$c" ] && { echo "CA copy $c: $(grep -c 'BEGIN CERTIFICATE' "$c") certificate(s); first: $(openssl x509 -in "$c" -noout -subject -enddate -fingerprint -sha256 2>/dev/null | tr '\n' ' ' | cut -c1-260)"; echo " same as /etc/pve/pve-root-ca.pem: $(cmp -s <(openssl x509 -in "$c") <(openssl x509 -in /etc/pve/pve-root-ca.pem) && echo yes || echo NO)"; echo " agent user can read it: $(sudo -n -u felhom-agent test -r "$c" && echo yes || echo no); others can read: $(stat -c %A "$c" | cut -c8)"; break; }
done
echo "served certificate (127.0.0.1:8006): $(echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256 -enddate 2>/dev/null | tr '\n' ' ')"
echo "/etc/pve/local/pve-ssl.pem: $(openssl x509 -in /etc/pve/local/pve-ssl.pem -noout -fingerprint -sha256 -startdate 2>/dev/null | tr '\n' ' ')"
echo "## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min)"
journalctl -u felhom-agent --since "-40min" --no-pager -o short-iso 2>&1 | grep -iE "pvetrust|trust|selfupdate|config bundle|signedjobs|capability|x509|tls:|certificate|config: " | grep -v "sudo\[" | cut -c1-430 | tail -${1:-25}
echo "## one control: the agent's newest successful host report and newest Proxmox call error"
journalctl -u felhom-agent --since "-20min" --no-pager -o short-iso 2>&1 | grep -iE "host report|report sent|hub: report|heartbeat" | tail -2 | cut -c1-260
echo "#### end"
@@ -0,0 +1,10 @@
#!/bin/bash
# wait-agent.sh SSH_TARGET VERSION [MINUTES] — wait until the box's agent says VERSION (a read each 30 s).
T=$1; V=$2; M=${3:-22}
end=$(( $(date +%s) + M*60 ))
while [ "$(date +%s)" -lt "$end" ]; do
cur=$(timeout 15 ssh -o BatchMode=yes -o ConnectTimeout=8 "$T" 'felhom-agent --version' 2>/dev/null | awk '{print $2}')
if [ "$cur" = "$V" ]; then echo "$(date +%T) $T agent $cur"; exit 0; fi
sleep 30
done
echo "$(date +%T) $T TIMEOUT, agent is ${cur:-unreadable}"; exit 1
@@ -0,0 +1,10 @@
#!/bin/bash
# wait-bundle.sh SSH_TARGET VERSION [MINUTES] — wait until the box's journal says the config bundle VERSION is installed.
T=$1; V=$2; M=${3:-22}
end=$(( $(date +%s) + M*60 ))
while [ "$(date +%s)" -lt "$end" ]; do
n=$(timeout 20 ssh -o BatchMode=yes -o ConnectTimeout=8 "$T" "journalctl -u felhom-agent --since '-60min' --no-pager | grep 'config bundle INSTALLED' | grep -c 'agent_version=$V'" 2>/dev/null)
if [ "${n:-0}" -ge 1 ]; then echo "$(date +%T) $T bundle $V installed"; exit 0; fi
sleep 30
done
echo "$(date +%T) $T TIMEOUT"; exit 1
@@ -0,0 +1,10 @@
#!/bin/bash
# wait-hublog.sh 'REGEX' SINCE(e.g. 5m) [MINUTES] — wait for a hub log line (a read each 20 s), print the matches.
R=$1; SINCE=$2; M=${3:-20}
end=$(( $(date +%s) + M*60 ))
while [ "$(date +%s)" -lt "$end" ]; do
out=$(sudo kubectl -n felhom-system logs deploy/hub --since="$SINCE" 2>/dev/null | grep -iE "$R")
if [ -n "$out" ]; then echo "$(date +%T) found:"; echo "$out" | cut -c1-320; exit 0; fi
sleep 20
done
echo "$(date +%T) TIMEOUT: no line for $R"; exit 1
@@ -0,0 +1,10 @@
#!/bin/bash
# wait-moved.sh SSH_TARGET [MINUTES] — wait until the box's agent config holds a ca_file (a read each 30 s).
T=$1; M=${2:-25}
end=$(( $(date +%s) + M*60 ))
while [ "$(date +%s)" -lt "$end" ]; do
cur=$(timeout 15 ssh -o BatchMode=yes -o ConnectTimeout=8 "$T" "python3 -c 'import json;t=json.load(open(\"/etc/felhom-agent/agent.json\")).get(\"proxmox\",{}).get(\"tls\",{});print(\"ca=\"+t.get(\"ca_file\",\"\")+\" fp=\"+(\"set\" if t.get(\"fingerprint\") else \"empty\"))'" 2>/dev/null)
case "$cur" in ca=/*) echo "$(date +%T) $T $cur"; exit 0;; esac
sleep 30
done
echo "$(date +%T) $T TIMEOUT: ${cur:-unreadable}"; exit 1
@@ -0,0 +1,33 @@
### root@192.168.0.154 2026-11-15T10:09:03Z
containers=10 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 43 seconds (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.353.0
image digest: sha256:4be0410d3bd5e0db2e98ab2a6f386a73784b72f62ea259821e8b13fb2cc51a6c repo: gitea.dooplex.hu/admin/felhom-controller@sha256:61cf2a8bfb82e02694bf85cca774bce2cbb9f947d395f91af7640aa5a5d944f0
started=2026-11-15T10:08:21.35521743Z
go in the running binary: 2 go1.27.4
errors=0 warn=0 info=132 panics=0
2026/11/15 10:08:21 [INFO] felhom-controller 0.353.0 starting (customer: tester-1, domain: enkicsifelhom.hu)
2026/11/15 10:08:22 [INFO] [selfupdate] Post-update startup: update successful (0.352.1 → 0.353.0)
### demo-hp 2026-11-15T10:09:05Z
containers=26 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 42 seconds (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.353.0
image digest: sha256:4be0410d3bd5e0db2e98ab2a6f386a73784b72f62ea259821e8b13fb2cc51a6c repo: gitea.dooplex.hu/admin/felhom-controller@sha256:61cf2a8bfb82e02694bf85cca774bce2cbb9f947d395f91af7640aa5a5d944f0
started=2026-11-15T10:08:23.282656164Z
go in the running binary: 2 go1.27.4
errors=0 warn=3 info=140 panics=0
2026/11/15 10:08:24 main.go:335: [INFO] felhom-controller 0.353.0 starting (customer: demo-hp, domain: enkisfelhom.hu)
2026/11/15 10:08:24 deploy.go:1470: [WARN] [stacks] Stack plex: new field PLEX_CLAIM (type=text) requires manual configuration
2026/11/15 10:08:24 deploy.go:1470: [WARN] [stacks] Stack calibre-web: new field ADMIN_PASSWORD (type=password) requires manual configuration
2026/11/15 10:08:24 deploy.go:1470: [WARN] [stacks] Stack bookstack: new field ADMIN_PASSWORD (type=password) requires manual configuration
2026/11/15 10:08:24 updater.go:855: [INFO] [selfupdate] Post-update startup: update successful (0.352.1 → 0.353.0)
### felhom-pve 2026-11-15T10:09:06Z
containers=9 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 47 seconds (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.353.0
image digest: sha256:4be0410d3bd5e0db2e98ab2a6f386a73784b72f62ea259821e8b13fb2cc51a6c repo: gitea.dooplex.hu/admin/felhom-controller@sha256:61cf2a8bfb82e02694bf85cca774bce2cbb9f947d395f91af7640aa5a5d944f0
started=2026-11-15T10:08:20.225850218Z
go in the running binary: 2 go1.27.4
errors=0 warn=0 info=146 panics=0
2026/11/15 10:08:21 [INFO] felhom-controller 0.353.0 starting (customer: demo-felhom, domain: demo-felhom.eu)
2026/11/15 10:08:21 [INFO] [selfupdate] Post-update startup: update successful (0.352.1 → 0.353.0)
@@ -0,0 +1,31 @@
# Tester 1 ONLY. clock.sh snap|back|right
# snap — READ: the clock, the time service, every container's id and creation time in guest 9201.
# back — stop the time service and set the clock 150 s BACK (the smaller harm: putting it right is then a
# step FORWARD, the direction a time service steps a late clock; nothing gets a stamp from the future
# while the clock is wrong, and no certificate or signed job ends early).
# right — start the time service and step the clock right.
set -u
grep -q '"tester-1' /etc/felhom-agent/agent.json 2>/dev/null || grep -q 'tester-1' /etc/felhom-agent/agent.json || { echo "REFUSED: this box is not Tester 1"; exit 2; }
snap() {
echo "## $(hostname) clock $(date '+%F %T.%N %Z' | cut -c1-27) epoch $(date +%s.%N | cut -c1-14); chrony $(systemctl is-active chrony); $(chronyc tracking 2>/dev/null | grep -E 'System time|Leap' | tr -s ' ' | tr '\n' ';')"
echo "## kernel boot_id $(cat /proc/sys/kernel/random/boot_id); btime $(awk '/^btime/ {print $2}' /proc/stat); uptime $(cut -d' ' -f1 /proc/uptime) s"
echo "## containers in 9201 (name id created):"
pct exec 9201 -- docker ps -a --format '{{.Names}} {{.ID}} {{.CreatedAt}} [{{.Status}}]' | sort | sed 's/^/ /'
echo "## count $(pct exec 9201 -- docker ps -q | wc -l) running"
}
case "${1:-}" in
snap) snap ;;
back)
busy=$(ps -eo comm | grep -cE "^(vzdump|apt-get|dpkg|proxmox-backup-)")
[ "$busy" = 0 ] || { echo "REFUSED: something is in flight"; exit 2; }
snap
systemctl stop chrony; echo "chrony stopped rc=$?"
date -s "@$(( $(date +%s) - 150 ))" >/dev/null; echo "clock set 150 s back rc=$? → $(date '+%F %T')"
;;
right)
systemctl start chrony; echo "chrony started rc=$?"
sleep 8; chronyc makestep; sleep 6
echo "clock now $(date '+%F %T'); $(chronyc tracking | grep -E 'System time|Leap' | tr -s ' ' | tr '\n' ';')"
;;
*) echo "usage: snap|back|right"; exit 2 ;;
esac
@@ -0,0 +1,2 @@
names read: 26; header exactly 'max-age=31536000': 26
control (a name that is not a box): felhom.eu -> []
@@ -0,0 +1,13 @@
#!/bin/bash
# hsts.sh HTTPS-ONLY-OUTPUT — read the Strict-Transport-Security header of every public name the https-only check
# walked (a GET of each name's https address; reads only). One line per name; a tally at the end.
names=$(grep -oE "https://[a-z0-9.-]+" "$1" | grep -v "no-such-name" | sort -u)
n=0; ok=0
for u in $names; do
h=$(curl -s -o /dev/null -D - --max-time 15 "$u/" | tr -d '\r' | grep -i '^strict-transport-security:' | head -1)
n=$((n+1))
v=$(echo "$h" | cut -d: -f2- | sed 's/^ *//')
if [ "$v" = "max-age=31536000" ]; then ok=$((ok+1)); else echo " NOT one year: $u -> [${v:-no header}]"; fi
done
echo "names read: $n; header exactly 'max-age=31536000': $ok"
echo "control (a name that is not a box): felhom.eu -> [$(curl -s -o /dev/null -D - --max-time 15 https://felhom.eu/ | tr -d '\r' | grep -i '^strict-transport-security:' | cut -d: -f2- | sed 's/^ *//')]"
@@ -0,0 +1,34 @@
control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
== tester-1 (enkicsifelhom.hu): 6 public name(s)
OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
== demo-hp (enkisfelhom.hu): 15 public name(s)
OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
OK http://time.enkisfelhom.hu/r1033-check?q=1 -> 301 https://time.enkisfelhom.hu/r1033-check?q=1
OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
== demo-felhom (demo-felhom.eu): 8 public name(s)
OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
== every public name redirects http -> https
@@ -0,0 +1,257 @@
System — Felhom Hub
Felhom Hub
Dashboard
Customers
Apps
Hosts
System
Offsite
Configuration
System — versions and OS updates
Needs attention
One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3).
[hover: worth a look] look
Tester4-2f0c25 Tester4
partial: no running customer guest · the host needs a restart since 2026-11-15 · crash guard not installed · [hover: the box could not read these values (agent older than v0.142.0, or the guest is down) — never a guess] could not read: running kernel, next-boot kernel, default kernel, host Debian, held packages, kernel.panic, crash restarts, guest Debian, Docker engine, containerd, Docker live-restore
Waiting for you
Update sets that only you approve: the kernel, the Docker engine and the Proxmox packages. Guest and host Debian updates approve themselves after 24 h and one night on the [hover: Ring 0 = the demo boxes. They install every new fix first; the other boxes (ring 1) install only what is approved.] ring-0 boxes.
Docker engine 29.9.3-1
Ran on every ring-0 box — demo-felhom-8363b5: 2 healthy night(s), demo-hp-bb76ea: 2 healthy night(s). The memory-kill check passed.
first seen 2 days ago (2026-11-13 04:49 UTC). Approving installs nothing by itself: a ring-1 box takes it only through a signed operator job.
Approve Docker set
Proxmox packages 9.2.27 — still being tested: demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set
Boxes
Click a box to see every value it reports, and its ring and update switches. "unknown": the box could not read the value — never a guess.
Box Ring Health Controller · agent OS updates Kernel [hover: The newest OS update run] Last night
Tester4-2f0c25 Tester4
[hover: Ring 1: a normal box — it installs only approved releases] 1
[hover: worth a look] look
0.353.0 [hover: current (vouched 0.175.1)] agent 0.175.1
[hover: runs the newest approved guest and host releases] on · up to date
unknown
[hover: 1 min ago · applied · 89 s — last successful leg: 3 min ago] ok · 1 min ago
partial: no running customer guest
Ring and updates
Ring ring 1
→ demo
OS updates updates ON
switch off
Tunnel [hover: no cloudflared container in the guest (agent state: absent)] none set up
Controller 0.353.0
Agent [hover: current (vouched 0.175.1)] 0.175.1
Last OS leg [hover: last successful leg: 3 min ago] 1 min ago · applied · 89 s
Docker engine
Docker [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
containerd [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
live-restore [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
Docker release —
Host
Proxmox 9.2.2
Kernel (running) [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
Kernel (next boot) [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: the box reports no kernel lane (agent older than v0.152.0)] —
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] —
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: refreshed] 7.0.2-6-pve · 1 rewritten · 13 min ago
Debian [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
Felhom release os-host-20261115-045718
Pending 99
Not covered 0
Held [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
Reboot needed since 2026-11-15
kernel.panic [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
Oops no
Crash restarts 24 h [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
Crash guard not installed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.175.1
Guest
Guest Debian [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
Felhom release os-guest-20261111-034654
Pending 6
Restart needed 7
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 8 min ago · 78.1 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] not yet
demo-felhom-8363b5 Demo Ügyfél
[hover: Ring 0: a demo box — it installs every new fix first] 0
[hover: nothing amber or red] fine
0.353.0 [hover: current (vouched 0.175.1)] agent 0.175.1
[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind
7.0.14-28-pve
[hover: 6 h ago · nothing · 23 s — last successful leg: 6 h ago] ok · 6 h ago
Ring and updates
Ring ring 0
→ normal
OS updates updates ON
switch off
Tunnel running
Controller 0.353.0
Agent [hover: current (vouched 0.175.1)] 0.175.1
Last OS leg [hover: last successful leg: 6 h ago] 6 h ago · nothing · 23 s
Docker engine
Docker 29.9.3
containerd 2.4.2-4~debian.13~trixie
live-restore on
Docker release ring0-20261115T045634Z
Host
Proxmox 9.2.27
Kernel (running) 7.0.14-28-pve
Kernel (next boot) 7.0.14-28-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 1m14s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 7 days ago · due 7.0.14-29-pve — household told 2 h ago: TONIGHT · phase good
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: removed 6 old kernel package(s): proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed; kept 7.0.14-26-pve, 7.0.14-28-pve; 6.3 GiB gained; 0 left-over path(s)] cleaned 6 h ago
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: not_needed] 7.0.14-28-pve · 23 h ago
Debian 13.7
Felhom release ring0-20261115T045634Z
Pending 29
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.175.1
Guest
Guest Debian 13.7
Felhom release ring0-20261115T045634Z
Pending 0
Restart needed 1
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 4 days ago · 272.7 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 92.7 Mbit/s · 5 days ago
demo-hp-bb76ea Demo HP
[hover: Ring 0: a demo box — it installs every new fix first] 0
[hover: nothing amber or red] fine
0.353.0 [hover: current (vouched 0.175.1)] agent 0.175.1
[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind
7.0.14-28-pve
[hover: 6 h ago · nothing · 30 s — last successful leg: 6 h ago] ok · 6 h ago
Ring and updates
Ring ring 0
→ normal
OS updates updates ON
switch off
Tunnel running
Controller 0.353.0
Agent [hover: current (vouched 0.175.1)] 0.175.1
Last OS leg [hover: last successful leg: 6 h ago] 6 h ago · nothing · 30 s
Docker engine
Docker 29.9.3
containerd 2.4.2-4~debian.13~trixie
live-restore on
Docker release ring0-20261115T040248Z
Host
Proxmox 9.2.27
Kernel (running) 7.0.14-28-pve
Kernel (next boot) 7.0.14-28-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 2m39s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 5 days ago · due 7.0.14-29-pve — household told 2 h ago: TONIGHT · phase good
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] —
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: not_needed] 7.0.14-28-pve · 23 h ago
Debian 13.7
Felhom release ring0-20261115T040248Z
Pending 30
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.175.1
Guest
Guest Debian 13.7
Felhom release ring0-20261115T040248Z
Pending 0
Restart needed 1
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 4 days ago · 70.5 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 94.3 Mbit/s · 5 days ago
tester-1-d70be4 Tester 1
[hover: Ring 1: a normal box — it installs only approved releases] 1
[hover: nothing amber or red] fine
0.353.0 [hover: current (vouched 0.175.1)] agent 0.175.1
[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind
7.0.14-22-pve
[hover: 6 h ago · nothing · 16 s — last successful leg: 6 h ago] ok · 6 h ago
Ring and updates
Ring ring 1
→ demo
OS updates updates ON
switch off
Tunnel running
Controller 0.353.0
Agent [hover: current (vouched 0.175.1)] 0.175.1
Last OS leg [hover: last successful leg: 6 h ago] 6 h ago · nothing · 16 s
Docker engine
Docker 29.8.2
containerd 2.3.6-1~debian.13~trixie
live-restore on
Docker release —
Host
Proxmox 9.2.2
Kernel (running) 7.0.14-22-pve
Kernel (next boot) 7.0.14-22-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-22-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: ring 1: no kernel staged by a signed os_kernel_step healthy 1m17s after the agent started; the new kernel is the default] 7.0.14-22-pve applied 38 days ago · phase good
[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] —
[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: not_needed] 7.0.14-22-pve · 23 h ago
Debian 13.7
Felhom release os-host-20261111-034654
Pending 101
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.175.1
Guest
Guest Debian 13.7
Felhom release os-guest-20261111-034654
Pending 6
Restart needed 0
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 4 days ago · 90.4 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 93.0 Mbit/s · 4 days ago
Details
Release ids, cancelled approvals, what ring 0 runs, the version floors, and every box's crash guard and root files.
Approved releases
The newest approved set of each layer. A [hover: Approved while a TEST wait override was active. It is cancelled when the hub starts without the override (`11` §5.3.1).] TEST approval is amber.
guest : os-guest-20261111-034654
272 packages · 2026-11-11 03:46 UTC · by auto
host : os-host-20261115-045718
607 packages · 2026-11-15 04:57 UTC · by auto
docker : os-docker-20261026-190351
6 packages · 2026-10-26 19:03 UTC · by operator
pve : os-pve-20261114-075712
121 packages · 2026-11-14 07:57 UTC · by operator
kernel : os-kernel-20261114-075716
2 packages · 2026-11-14 07:57 UTC · by operator
Approve now (guest + host)
An urgent fix only — normally the hub approves after 24 h and one night.
What ring 0 runs now
guest :
272 packages, first seen 2026-11-10 01:49 UTC
approved as os-guest-20261111-034654
host :
607 packages, first seen 2026-11-14 04:52 UTC
approved as os-host-20261115-045718
docker :
6 packages, first seen 2026-11-13 04:49 UTC
ready — see "Waiting for you"
pve :
121 packages, first seen 2026-11-15 04:58 UTC
demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set
kernel :
2 packages, first seen 2026-11-10 01:57 UTC
approved as os-kernel-20261114-075716
Version floors
Global controller floor: 0.353.0 · vouched agent: 0.175.1
[hover: A floor is the lowest controller version a box must run; the hub moves a box below it up.] What is a floor? · [hover: The agent version the operator checked and signed off for the fleet. Agents update only by a per-box signed job.] What is a vouched agent?
No per-customer floors: every box follows the global floor.
Crash guard and root files
Box ¦ kernel.panic ¦ Oops ¦ Crash restarts 24 h ¦ Crash guard ¦ [hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files ¦
Tester4-2f0c25 ¦ [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown ¦ no ¦ [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown ¦ not installed ¦ 0.175.1 ¦
demo-felhom-8363b5 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.175.1 ¦
demo-hp-bb76ea ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.175.1 ¦
tester-1-d70be4 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.175.1 ¦
Felhom Hub 0.182.0
@@ -0,0 +1,34 @@
#!/bin/bash
# Read from the LIVE hub database (the three files in one tar, as the note says), then shred the copy:
# does the newest report of each test household carry web_certs? Also the daily ledger and the floors.
set -u
W=/mnt/5_hdd/felhom.eu/evidence/release-2b-3-2026-11-15/dbcopy; mkdir -p "$W"; chmod 700 "$W"
sudo kubectl -n felhom-system exec deploy/hub -- sh -c 'cd /data && tar cf - hub.db hub.db-wal hub.db-shm' > "$W/db.tar"
( cd "$W" && tar xf db.tar )
python3 -I - "$W/hub.db" <<'PY'
import sqlite3, sys, json
db = sqlite3.connect(sys.argv[1])
tabs = [r[0] for r in db.execute("SELECT name FROM sqlite_master WHERE type='table'")]
rt = "reports" if "reports" in tabs else [t for t in tabs if "report" in t][0]
cols = [r[1] for r in db.execute("PRAGMA table_info(%s)" % rt)]
print("report table:", rt, "columns:", ",".join(cols)[:200])
tcol = [c for c in cols if c in ("received_at", "reported_at", "created_at")][0]
jcol = [c for c in cols if c in ("data", "report", "payload", "body", "json", "report_json")]
print("freshness control: newest report", db.execute("SELECT MAX(%s) FROM %s" % (tcol, rt)).fetchone()[0])
for cust in ("tester-1", "demo-hp", "demo-felhom"):
row = db.execute("SELECT %s, %s FROM %s WHERE customer_id=? ORDER BY %s DESC LIMIT 1" % (tcol, jcol[0], rt, tcol), (cust,)).fetchone()
d = json.loads(row[1])
wc = d.get("web_certs")
def find(o):
if isinstance(o, dict):
if "web_certs" in o: return o["web_certs"]
for v in o.values():
r = find(v)
if r is not None: return r
return None
wc = wc if wc is not None else find(d)
print(cust, "report at", row[0], "controller", d.get("controller_version") or d.get("version"), "web_certs:", json.dumps(wc)[:300])
for k, v in db.execute("SELECT key, substr(value,1,120) FROM hub_settings WHERE key LIKE 'daily_ledger%' OR key LIKE 'os_alarm:%' OR key LIKE 'min_controller%' ORDER BY key"):
print("setting", k, "=", v)
PY
find "$W" -type f -exec shred -u {} \; ; rmdir "$W"; echo "copy shredded: $([ -e "$W" ] && echo NO || echo yes)"
@@ -0,0 +1,20 @@
report table: reports columns: id,customer_id,received_at,report_json,health_status,cpu_percent,memory_percent,container_total,container_running,backup_last_snapshot,controller_version,controller_url,language
freshness control: newest report 2026-11-15 10:08:31
tester-1 report at 2026-11-15 10:08:28 controller 0.353.0 web_certs: [{"domain": "*.enkicsifelhom.hu", "not_after": "2027-01-02T18:44:06Z", "issuer": "YR1"}]
demo-hp report at 2026-11-15 10:08:31 controller 0.353.0 web_certs: [{"domain": "*.enkisfelhom.hu", "not_after": "2027-01-19T01:46:00Z", "issuer": "YR2"}]
demo-felhom report at 2026-11-15 10:08:27 controller 0.353.0 web_certs: [{"domain": "*.demo-felhom.eu", "not_after": "2026-12-31T18:52:10Z", "issuer": "YR1"}]
setting min_controller_version = 0.352.1
setting min_controller_version_declared_min_agent = 0.352.1=0.158.1
setting os_alarm:agent:Tester-2-be8404 = 2026-10-19T10:30:41Z
setting os_alarm:bundle:demo-felhom-8363b5 =
setting os_alarm:bundle:demo-hp-bb76ea =
setting os_alarm:bundle:tester-1-d70be4 =
setting os_alarm:daily_last:deadline-check = 2026-11-15T04:00:00Z
setting os_alarm:daily_last:offsite-key-audit = 2026-11-15T06:10:00Z
setting os_alarm:end_date:baseline = 2026-11-15T08:50:09Z
setting os_alarm:offsite_escrow_pending:Tester-2 = 2026-10-18T16:14:28Z
setting os_alarm:operator_mail_lost_scan = 2026-11-15T08:48:59Z
setting os_alarm:reboot:Tester-2-be8404 = 2026-10-18T16:28:55Z
setting os_alarm:stale:Tester-2-be8404 = 2026-10-18T17:28:55Z
setting os_alarm:stale:demo-hp-bb76ea =
copy shredded: yes
@@ -0,0 +1,9 @@
== /customers/tester-1/floor
HTTP/1.1 303 See Other
Location: /customers/tester-1?flash=floor_set
== /customers/demo-hp/floor
HTTP/1.1 303 See Other
Location: /customers/demo-hp?flash=floor_set
== /customers/demo-felhom/floor
HTTP/1.1 303 See Other
Location: /customers/demo-felhom?flash=floor_set
@@ -0,0 +1,19 @@
## felhom clock 2026-11-15 11:26:38.0075053 epoch 1794738398.009; chrony active; System time : 0.002849487 seconds slow of NTP time;Leap status : Normal;
## kernel boot_id 0581d703-3fae-47e4-8598-ee462aa3d516; btime 1794738245; uptime 152.48 s
## containers in 9201 (name id created):
bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 2 minutes (healthy)]
bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 2 minutes (healthy)]
cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 2 minutes (healthy)]
felhom-controller 9f210e5a321f 2026-11-15 10:24:27 +0000 UTC [Up 2 minutes (healthy)]
filebrowser c3b5388d9c1e 2026-11-15 10:24:52 +0000 UTC [Up About a minute (healthy)]
paperless-postgres 913e7c4090c1 2026-11-15 10:24:41 +0000 UTC [Up About a minute (healthy)]
paperless-redis 5024d8448930 2026-11-15 10:24:41 +0000 UTC [Up About a minute (healthy)]
paperless-webserver 3e7d16433e63 2026-11-15 10:24:41 +0000 UTC [Up About a minute (healthy)]
privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 2 minutes (healthy)]
traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 2 minutes]
## count 10 running
2026/11/15 10:24:30 [INFO] [gate] boot 0581d703-3fae-47e4-8598-ee462aa3d516-1535: waiting (≤2m0s) for live drive bind(s) [/mnt/felhom-drives/adatlemez] before recreating drive-backed apps
2026/11/15 10:24:40 [INFO] [gate] boot 0581d703-3fae-47e4-8598-ee462aa3d516-1535: live bind confirmed — recreating drive-backed app paperless-ngx (state=degraded) onto /mnt/felhom-drives/adatlemez
2026/11/15 10:24:52 [INFO] [gate] boot 0581d703-3fae-47e4-8598-ee462aa3d516-1535: re-syncing FileBrowser mounts against the live binds
felhom-agent 0.175.1
0
@@ -0,0 +1,14 @@
## felhom clock 2026-11-15 11:10:47.4544935 epoch 1794737447.456; chrony inactive;
## kernel boot_id 7a721957-7e95-476c-be40-52d0dafc7284; btime 1794665801; uptime 71646.26 s
## containers in 9201 (name id created):
bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 5 hours (healthy)]
bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 5 hours (healthy)]
cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 20 hours (healthy)]
felhom-controller 09343ef712fa 2026-11-15 10:08:21 +0000 UTC [Up 2 minutes (healthy)]
filebrowser 95f38571abee 2026-11-14 14:19:57 +0000 UTC [Up 20 hours (healthy)]
paperless-postgres 31b2490aa677 2026-11-15 10:05:37 +0000 UTC [Up 5 minutes (healthy)]
paperless-redis df1c44607a9b 2026-11-15 10:05:37 +0000 UTC [Up 5 minutes (healthy)]
paperless-webserver 1f2a0d7e4f7b 2026-11-15 10:05:37 +0000 UTC [Up 5 minutes (healthy)]
privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 5 hours (healthy)]
traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 20 hours]
## count 10 running
@@ -0,0 +1,14 @@
## felhom clock 2026-11-15 11:21:49.3394920 epoch 1794738109.341; chrony active; System time : 0.001603887 seconds slow of NTP time;Leap status : Normal;
## kernel boot_id 7a721957-7e95-476c-be40-52d0dafc7284; btime 1794665951; uptime 72157.81 s
## containers in 9201 (name id created):
bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 6 hours (healthy)]
bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 6 hours (healthy)]
cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 20 hours (healthy)]
felhom-controller 09343ef712fa 2026-11-15 10:08:21 +0000 UTC [Up 13 minutes (healthy)]
filebrowser 95f38571abee 2026-11-14 14:19:57 +0000 UTC [Up 20 hours (healthy)]
paperless-postgres 31b2490aa677 2026-11-15 10:05:37 +0000 UTC [Up 16 minutes (healthy)]
paperless-redis df1c44607a9b 2026-11-15 10:05:37 +0000 UTC [Up 16 minutes (healthy)]
paperless-webserver 1f2a0d7e4f7b 2026-11-15 10:05:37 +0000 UTC [Up 16 minutes (healthy)]
privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 6 hours (healthy)]
traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 20 hours]
## count 10 running
@@ -0,0 +1,16 @@
## felhom clock 2026-11-15 11:11:17.4153213 epoch 1794737477.417; chrony active; System time : 0.000710471 seconds slow of NTP time;Leap status : Normal;
## kernel boot_id 7a721957-7e95-476c-be40-52d0dafc7284; btime 1794665951; uptime 71525.88 s
## containers in 9201 (name id created):
bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 5 hours (healthy)]
bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 5 hours (healthy)]
cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 20 hours (healthy)]
felhom-controller 09343ef712fa 2026-11-15 10:08:21 +0000 UTC [Up 2 minutes (healthy)]
filebrowser 95f38571abee 2026-11-14 14:19:57 +0000 UTC [Up 20 hours (healthy)]
paperless-postgres 31b2490aa677 2026-11-15 10:05:37 +0000 UTC [Up 5 minutes (healthy)]
paperless-redis df1c44607a9b 2026-11-15 10:05:37 +0000 UTC [Up 5 minutes (healthy)]
paperless-webserver 1f2a0d7e4f7b 2026-11-15 10:05:37 +0000 UTC [Up 5 minutes (healthy)]
privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 5 hours (healthy)]
traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 20 hours]
## count 10 running
chrony stopped rc=0
clock set 150 s back rc=0 → 2026-11-15 11:08:49
@@ -0,0 +1,4 @@
chrony started rc=0
200 OK
clock now 2026-11-15 11:20:27; System time : 0.000000000 seconds fast of NTP time;Leap status : Normal;
Tester 1 minus DooPlex: +0.071 s
@@ -0,0 +1,15 @@
#!/bin/bash
# wait-ctl.sh VERSION [MINUTES] — wait until guest 9201 on the three test boxes runs controller VERSION (reads).
V=$1; M=${2:-25}
end=$(( $(date +%s) + M*60 ))
while [ "$(date +%s)" -lt "$end" ]; do
ok=0; line=""
for h in root@192.168.0.154 demo-hp felhom-pve; do
img=$(timeout 20 ssh -o BatchMode=yes -o ConnectTimeout=8 "$h" "pct exec 9201 -- docker ps --filter name=^felhom-controller$ --format '{{.Image}} {{.Status}}'" 2>/dev/null)
line="$line | $h: ${img##*:}"
case "$img" in *":$V "*healthy*) ok=$((ok+1));; esac
done
if [ "$ok" = 3 ]; then echo "$(date +%T) all three$line"; exit 0; fi
sleep 30
done
echo "$(date +%T) TIMEOUT$line"; exit 1
@@ -0,0 +1,16 @@
### root@192.168.0.154 2026-11-15T10:51:31Z
containers=10 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 27 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.353.0
errors=0 warn=1 info=635 panics=0
### demo-hp 2026-11-15T10:51:33Z
containers=26 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 43 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.353.0
errors=0 warn=3 info=907 panics=0
### felhom-pve 2026-11-15T10:51:35Z
containers=9 unhealthy=0 restarting=0 exited=[]
gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 43 minutes (healthy)
image file: gitea.dooplex.hu/admin/felhom-controller:0.353.0
errors=0 warn=0 info=856 panics=0
9202: gitea.dooplex.hu/admin/felhom-controller:0.353.0-rc.1 Up About an hour (healthy)
@@ -0,0 +1,5 @@
2026/11/15 11:26:53 [INFO] Global controller-version floor set to "0.353.0" (declared MinAgent "0.158.1")
2026/11/15 11:26:53 [INFO] access token "cc-releases-2": POST /configuration/global-floor → 303
2026/11/15 11:46:10 [WARN] slow request: POST /api/v1/host-enroll 201 11241ms (over 5s)
2026/11/15 11:46:32 [WARN] escrow CARRIED for Tester4: the sealed package of deleted host Tester4-46836f is now the current package of Tester4-2f0c25 (repo-key hash 5501e4bf2cda…); the box will ask for the recovery cod
2026/11/15 11:48:03 [INFO] managed floor SERVED for Tester4: floor 0.353.0, agent requirement "0.158.1" from declared (golden 0.352.1)
+12
View File
@@ -26,6 +26,18 @@
---
## 2026-11-15 — releases 2b and 3: R-1380, R-1395, R-1050 closed (full text of the rows: `git show f561c85b:documentation/backlog/OPEN-ITEMS.md`)
| Row | What | Closed | Evidence |
|---|---|---|---|
| **R-1380** | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** (P3) | CLOSED 2026-11-15 — agent 0.175.1 (vouched 11:07:35 CET), installer 1.37.0 (tagged and published), golden 0.353.0 (vouched). Tester 1, demo-hp and the N100 moved from the pinned certificate to the copy of their own root CA: each proved a working call with the copy first and cleared the fingerprint last. On demo-hp two forced new Proxmox certificates changed nothing for the agent: the same process, no restart, 14 and 40 calls answered 200 in Proxmox's own log, no alarm. A wrong CA copy was repaired at the agent's start. A fresh install ended with the copy and no fingerprint. A cut write of the agent's config is repaired at start (tests with a cut at four steps; NOT proven with a real power cut). What stays open is R-1396 gap 2 (a fully blind box) | `audits/release-2b-3-2026-11-15/` `part0/`, `part2/`; `documentation/tests/golden-0.353.0-2026-11-15/` |
| **R-1395** | **A step of the host's clock makes the controller read a new boot: it then recreates every drive-backed app, once.** (P4) | CLOSED 2026-11-15 — controller 0.353.0 (global floor 10:26:53 UTC) with agent 0.175.1. On Tester 1 the same 150 s clock step was staged twice: with the old controller it recreated one drive-backed app (three containers) at the step back and again at the step forward; with the new controller and agent it recreated nothing in either direction (the same container ids before and after). One real restart of Tester 1 after that was read as a new boot and the app was recreated, so the gate still works. On a box whose agent is older than 0.175.0 a clock step still recreates once | `audits/release-2b-3-2026-11-15/` `part2/t1-clock-A-*.txt`, `part3/t1-clock-B-*.txt`, `part3/t1-after-reboot.txt` |
| **R-1050** | **Raise HSTS to one year on or after 2026-11-15, if no https trouble was reported.** (P4) | CLOSED 2026-11-15 — controller 0.353.0, on the operator's word in the release prompt (decision 249). Every public name of the three test boxes answers `strict-transport-security: max-age=31536000` and nothing more (26 names of 26; no includeSubDomains, no preload); the https-only check is green. The night's search found no https trouble since 2026-10-15. Not tested: a browser | `audits/release-2b-3-2026-11-15/` `part3/hsts-headers.txt`, `part3/https-only.txt` |
**Reasoning kept (R-1380).** A set fingerprint wins over a CA file, so the move clears the fingerprint LAST and only after a working call with the copy. `agent.json` is written only through `config.ReplaceInPlace` (a journal of the old bytes first; the daemon repairs a cut write before it reads the config). Root files that are older than the agent are not a trust problem on a pinned box — but that quiet path is never taken after a failed call.
**Reasoning kept (R-1395).** The boot token is the agent's; the controller only compares. A boot is the kernel's boot id, never a time computed from the wall clock.
## 2026-11-14 — release 2a: R-1388, R-1301, R-1391 closed (full text of the rows: `git show a2173830:documentation/backlog/OPEN-ITEMS.md`)
| Row | What | Closed | Evidence |
+7 -10
View File
@@ -268,7 +268,6 @@ stopping line that lies.
| **R-779** | Security & access | P4 | **[P3-LOW] Part A's "two outside addresses seen as two" is proven through the simulated tunnel only; on the REAL tunnel the second outside address (ep0, one request allowed) was refused by Cloudflare's edge with 403 and never reached the box.** Measured 2026-10-01 19:51 UTC (`audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt`): no log line on demo-hp; demo-hp's box has no geo restriction in its settings, so a Cloudflare ZONE rule (country or bot, not read) refused a German datacenter address. DooPlex's own address on the real tunnel was seen as itself. **Needs:** one sign-in from a second Hungarian address (the operator's phone off wifi) while DooPlex is locked out — 2 minutes; and say which Cloudflare rule refused ep0. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (a phone), CC reads the logs** **Re-ranked 2026-10-03: P3→P4: a proof gap on the real tunnel; operator-only follow-up.** | — | — | CC + operator |
| **R-904** | Security & access | P4 | **Cloudflare can read every household's app traffic; replacing it with our own relay is a later item.** Facts (reviewer discussion 2026-10-08; `01`): app traffic and the dashboard reach the box through the Cloudflare Tunnel (`01` §5 trust table, rows end-user ↔ apps and customer ↔ controller UI; §7), and the tunnel's public end is Cloudflare's edge, where TLS ends — so Cloudflare can technically read that traffic (the FAQ says so since 2026-10-08, R-900; „TLS ends at the edge" is not written in `01` — add it there). What Cloudflare gives today, free: inbound reach with no router setup, the CGNAT answer (`01` §4, §7); certificates (`01` §7, the free tier covers one level below a zone); the geo-WAF the hub enforces (`01` §5 last row, §7); flood protection (not in `01`). The alternative named: our own EU relay over WireGuard with TLS passthrough by SNI, certificates on the box, the geo-block on the relay. Its costs: one more machine the operator keeps up, and a single point of reach for every box; weaker flood protection; about a week of work after a spike. Operator ruling 2026-10-08 09:07 (`09` §3 decision 184): a later item. | **DEFERRED** — **2026-10-10: the asked sentence is in `01` §7** („TLS ends at Cloudflare's edge …”); the spike stays deferred. **DEFERRED — after the first customers (operator ruling 2026-10-08 09:07)** | — | A spike after the first customers (the relay's reach, cost and flood behaviour, measured) | operator |
| **R-1040** | Security & access | P4 | **Until the household makes Sonarr's or Radarr's first admin, any container on the box can read the app's API key.** Measured on the bench 2026-10-14 with the R-1030 seed in place: `AuthenticationRequired=Enabled`, `AuthenticationMethod=None` — `/initialize.json` from another container answers 200 with `apiKey`. Outsiders are blocked by the box's setup gate in that window; a compromised other app is not. | **OPEN** — filed 2026-10-14 | — | Seed the API key at install (`SONARR__AUTH__APIKEY`, measured to work) so the box knows it, and close the window by telling the household in the first steps; or seed a Forms user | CC |
| **R-1050** | Security & access | P4 | **Raise HSTS to one year on or after 2026-11-15, if no https trouble was reported.** `09` §3 decision 249 (D2 = A, operator 2026-10-15): HSTS stays `max-age=86400` now; after a month with no https trouble it becomes one year (`31536000`); never preload, never `includeSubDomains`. The value is `HSTSMaxAgeSeconds` in `felhom-controller/controller/internal/infra/infra.go`, pinned by `TestRenderForwardedHeaders_HTTPSOnly`. „Trouble” = any report of a name that will not open, or a box answering http after the floor. | NARROWED — one year is in controller `main`, not released. **2026-11-14 (night build):** searched for https trouble since 2026-10-15: the open and closed registers (ASCII fragments; controls: a common word found, a nonsense word not found), the operator's mailbox (words for „does not open”, „not secure”, certificate, HSTS, the browser's error codes: one thread, a tester's install questions, no https trouble), the hub's log since its start today (0 lines). No name that does not open, no box answering http after the floor. Not searched: the hub's log before today's deploy (not kept), Tester 2 (off, R-1104). The release must be built on or after 2026-11-15. WATCHING — **2026-11-14: NOT in controller 0.352.1.** That release was built on 2026-11-14, one day before the ruled date, and the operator's words „one day early is OK” were not given. Check date moved to 2026-11-21 (one week: the next day a controller release is likely; a one-day move would only turn the nightly CI red with nothing to release). The search for https trouble since 2026-10-15 is NOT done yet. Earlier: **WATCHING** — filed 2026-10-15 | the date | On or after 2026-11-15: read the register and the operator's mail for https trouble since 2026-10-15; none → a controller release with one year (test updated), bake → vouch → floor | CC |
| **R-1054** | Security & access | P3 | **The hub's operator/global API key is still one bearer that can do everything on `/api/v1`, including reading every box's break-glass root password (`GET /admin/hosts/{id}/recovery-credential`).** Hub v0.152.0 gave the web surface scoped tokens (`05` §17) but tokens are refused on `/api/v1` by design. Who uses the global key today is not listed in one place. Fix: list its consumers; move each operator use to a scoped token on the web side (or a token class on the API), and keep the global key only where a box path needs it — or retire it. | OPEN — filed 2026-10-15 by the token build | — | List the global key's consumers (pod env, scripts, runbooks) | CC |
| **R-1194** | Security & access | P3 | **On a fresh box the operator's break-glass SSH door was broken from the first minute: `oob_degraded` „felhom-sshd config invalid (sshd -t fails)".** Rehearsal 2026-10-20, Tester4-816e21, mail at 12:03 UTC (one minute after the host record was created): details `active:false, config_invalid:true, felhom_sshd_port:0, operator_key_configured:false, operator_key_reported:true, reachable:false`. Possibly the shape of a customer with no operator key set (Tester4 is a test customer) — then the alarm is a false one on every such box; or a real config defect. Not diagnosed (no shell). Checked the same day: the only `oob_degraded` in the operator mailbox in 30 days, and no recovery mail followed; the report says port 0. Two candidate mechanisms in the agent (`internal/felhomsshd/health.go:63-66`): `sshd -t -f <conf>` run by the NON-root agent fails on unreadable host keys, or a config rendered before the desired state arrived (port 0). **CAUSE FOUND AND FIXED 2026-10-21 night** (`audits/night-2026-10-21/P5-checks/`): reproduced on a fresh box from the public ISO — `oob_degraded` mailed 20:34 CEST while felhom-sshd was applied 0.4 s after the check and is healthy (port 8822 listening, `sshd -t` passes as the agent's user through its sudo grant — so the non-root theory is FALSE; without sudo it fails on the host key, which the agent never does). The first host report runs before the felhom-sshd loop's first pass and reads a config never rendered. Fix: agent `0da792e` — no OOB stanza until the first pass has run (the hub does not evaluate a missing one; a failed pass counts). `TestR1194_NoOOBStanzaBeforeTheFirstPass` red-proved. | WATCHING — released in agent v0.162.0 (2026-10-22, operator yes; on the 3 test boxes); the false alarm comes only on a FRESH box's first report | the next fresh install with agent ≥ 0.162.0 | At the next fresh install read the hub's events for `oob_degraded` in the first hour (none expected); then close | CC |
| **R-1237** | Security & access | P3 | **The nightly CC-secrets sweep found stored one-time codes in a session record dated 2026-10-23: 14 names in one file (bind tokens and setup codes of test boxes — NIGHT_2026_10_23_*, FIXES_2026_10_21_*, NIGHT_2026_10_21_*, REHEARSAL_2026_10_20_* — and R_DEMO_FELHOM in a file of 2026-10-20).** Mail „[DooPlex CC secrets] 2 secret name(s) found in old session records", 2026-10-23 01:57 UTC, read in the day session (names only). One file holding all of them suggests one command printed many stored values at once (a listing or a dump) in the night session. All but R_DEMO_FELHOM are one-time install codes of test boxes; NIGHT_2026_10_23_* die when Tester4-16fdef is deleted (the operator's do list). | **OPEN** — 2026-10-27 (operator yes): the dead names were REMOVED from the store with `sec rm` (46, one commit each; `audits/r1299-window-button-2026-10-27/sec-rm.txt`, names only): every one-time code this row names is gone; R_DEMO_FELHOM stays (unsure). Consequence: the nightly sweep no longer watches for the removed values — they open nothing (their boxes are destroyed). The session transcripts themselves are unchanged — that question stays here. **Earlier:** OPEN | — | CC: find the file and the command by the sweep's own report (names only), add a hook shape for that command if it is not one; R_DEMO_FELHOM (demo-felhom's recovery code) — the operator decides whether to rotate it | CC |
@@ -292,14 +291,12 @@ stopping line that lies.
| **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC |
| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. **2026-11-11:** the End-dates list leans on this row (`documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB21); the newest golden passed its 7-day limit on 2026-11-10 and no waiver is in force, so the gate refuses code pushes to `felhom.eu`. **2026-11-11: golden 0.352.0 baked and published; one fresh box installed from it (`documentation/tests/golden-0.352.0-2026-11-11/README.md`); the gate is green (`python3 scripts/golden_currency_gate.py`, rc 0). The operator vouched it on 2026-11-11 at 14:48:11 CET.** | WATCHING — **2026-11-14: golden 0.352.1 baked (sha256 `4f411cbe…346d`), one fresh install proven on a nested VM on demo-hp, vouched by the operator 10:25:43 UTC (`documentation/tests/golden-0.352.1-2026-11-14/`). The gate judges the bake's age: the next bake is due by 2026-11-21.** Earlier: **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement **2026-11-03: the Sunday 2026-11-01 bake was missed (it was in no brief): golden 0.344.0 is 8 days old, the floor 0.349.0; a CODE push to felhom.eu is refused until a bake and the vouch or a dated waiver. The night's close-out was documents only, which the gate exempts; no waiver written. First on the morning list.** | — | — | CC |
| **R-1358** | Box system & updates | P3 | **cloudflared 2026.10.4 misses two dependency fixes (VULN-183456 chi, 2026.10.5; VULN-184686 go-jose, 2026.10.6; the newest is 2026.10.7).** No public text on reach. Not pinned tonight: the tunnel is a box's only door, and 9202 has none to prove it on. | OPEN — filed 2026-11-03 | — | raise `CloudflaredImage` on the next controller release with a tunnel proof on Tester 1 | CC |
| **R-1380** | Box system & updates | P3 | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** Found 2026-11-11 (the End-dates list, class T2, `documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB02-3 to DB02-7). MEASURED: the agent trusts one exact Proxmox certificate (a pin), and the certificate on the three test boxes ends about two years after install — the N100's about 2028-07-16, then demo-hp and Tester 1 within three months. Proxmox makes a new certificate by itself about 14 days before the old one ends (so about 2028-07-02 on the N100). No code path gives the agent the new fingerprint, no alarm names the cause, no runbook holds the hand step, and nobody has tried it. Every box installed in the same season fails in the same fortnight. P3 only because the first date is in 2028; it is certain, not a risk. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 297). Not built.** **2026-11-11 (afternoon): tried on demo-hp (`documentation/audits/certpin-spike-2026-11-11/README.md`).** A forced new certificate (`pvecm updatecerts --force`, then a reload of the web service; Proxmox's own renewal does that reload by itself) blinds the agent at once: `TLS pin mismatch`, no reconcile, no host report to the hub, 502 to the controller; guests and apps stay up. One operator mail names a symptom (`agent_channel_unknown`), none names the cause; the hub's Hosts page still read ONLINE eleven minutes after the last report (when it turns offline was not measured). The hand repair works and is written down (`documentation/runbooks/RUNBOOK-agent-proxmox-pin-repair.md`, 2.1 s from restart to a working call). **A CA file in place of the fingerprint works with no agent code:** with `proxmox.tls.ca_file` on a readable copy of the box's root CA and the fingerprint empty, the agent went through a second new certificate with no hand step; a wrong CA file was refused. Two facts for the fix: the agent's user cannot read Proxmox's certificate files, and a set fingerprint wins over the CA file. The design with both ways is `03-host-agent.md` §12a; CC's pick differs from the ruled shape, so it is decision D80. | READY — **2026-11-14 night, after an independent review (agent `ec7e699`, `54ac2c2`):** one lock for every writer of `agent.json`; the trust run is serialised and ends after 90 s; the root step keeps the previous root only during a changeover and compares exactly. **One low item stays open:** the move writes `agent.json` in place and nothing restores `agent.json.pre-pve-trust` by itself after a cut write (`03` §12a says so) — to weigh at the box proof. Earlier: READY — **2026-11-14 night: BUILT, not released** (felhom-agent `386a0c4` + `a9ab47e`; the installer on felhom.eu `main`, not tagged). The root step keeps the copy; the agent moves a pinned box in the order copy → proof → config, clears the fingerprint last, and a pinned box whose certificate changed heals at the first failed call; the installer gives a new box the copy and no fingerprint; the cause of a trust failure is named. `03` §12a is `[FACT — built, not released]`. **LEFT: the proof on a box** (demo-hp, a forced new certificate, twice; then the release with both signed jobs and the installer tag). What the hub does not tell yet: R-1396. Earlier: READY — **2026-11-14 09:55 — RULED (D80, `09` §3 decision 303): the root CA copy.** Built in release 2 (the host release). OPEN — filed 2026-11-11 (end-dates list) | — | the operator rules D80 (CA file, or follow the fingerprint); then the design's open parts in `03-host-agent.md` §12a (the root step that refreshes the copy; an alarm that names the cause), then build, then prove on a scratch host with a forced renewal | CC |
| **R-1382** | Box system & updates | P3 | **A box that was off past an end date: nobody knows if it comes back by itself (5 answers).** Found 2026-11-11 (the End-dates list, class T4, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; read in source and documents, never tried). After months without power: the web certificate has ended (DB01-6: does Traefik renew at the first start, and what does the household see until then), Debian's package keys may be too old to verify an update (DB16-6), the tunnel client may be below what Cloudflare accepts (DB19-6), a 30-day restore copy is past its date (DB24-6), and the 14-day countdown after a stopped off-site copy ran out while the box was off (DB04-6). Tester 2 has been off since 2026-10-04 and is the natural test. | OPEN — filed 2026-11-11 (end-dates list) | — | When Tester 2 comes back (or with a scratch host and a moved clock on Tier 0, as a task): record what renews by itself and what needs a hand; put the hand steps in a runbook. | CC |
| **R-1384** | Box system & updates | P3 | **A verdict or an „already ran” check trusts a clock that can be wrong (30 answers).** Found 2026-11-11 (the Clock list, class W2, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED: the N100's host is 32.6 s behind and its time service does not run (R-1301); the other two boxes are in step. Nothing checks a box's clock except one hub alarm for a tunnel handshake stamped in the future. The hub takes the box's own time for backup age, off-site age and proof age (CH02-6, CH07-6, CH07-8, CH08-6); a box with a wrong clock refuses every signed job (CA09-6); a stamp from the future blocks the OS-update leg, the trim and the line test until the clock passes it (CA02-6, CA02-8, CA11-6, CA11-8, CA12-6); the crash guard reads the clock at early boot (CA04-6, CA04-8); a hub clock jump moves its own daily jobs (CH03-6, CH04-6, CH06-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6). The others: CA05-6, CA05-8, CA06-6, CA07-6, CA10-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6. Known parts stay in their rows: R-1189, R-1206, R-1260. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 300). Not built.** **2026-11-11 (afternoon): why the N100's clock is wrong is known (R-1301, `documentation/audits/n100-clock-2026-11-11/README.md`).** Numbers from source: a host clock more than 120 s behind refuses every signed job (`felhom-agent/internal/authz/verifier.go:21`, `:153-162`); at about 1.25 s a day the N100 gets there in about 70 days (mid-January 2027, an estimate from one measurement). | NARROWED — the hub's check is built, not released. **2026-11-14 (night build):** the agent stamps its clock at the moment of sending (`sent_at`) and the hub alarms the operator when it is more than one minute from the hub's clock, in either direction, and says so when it is back (`08-alarm-ladder.md`, `box_clock_wrong`). A report that waited on the box raises nothing (tested). Not released, not seen on a box. LEFT after the release: the 30 answers themselves still trust the clock; the alarm only makes a wrong clock known. OPEN — **2026-11-14: the one box whose clock was wrong (the N100, 32.6 s) is right again (R-1301 closed); this row's check is still not built.** Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): the hub compares the time in each report with its own, alarms the operator above one minute, and does not take a time from the future as „fresh”. | CC |
| **R-1384** | Box system & updates | P3 | **A verdict or an „already ran” check trusts a clock that can be wrong (30 answers).** Found 2026-11-11 (the Clock list, class W2, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED: the N100's host is 32.6 s behind and its time service does not run (R-1301); the other two boxes are in step. Nothing checks a box's clock except one hub alarm for a tunnel handshake stamped in the future. The hub takes the box's own time for backup age, off-site age and proof age (CH02-6, CH07-6, CH07-8, CH08-6); a box with a wrong clock refuses every signed job (CA09-6); a stamp from the future blocks the OS-update leg, the trim and the line test until the clock passes it (CA02-6, CA02-8, CA11-6, CA11-8, CA12-6); the crash guard reads the clock at early boot (CA04-6, CA04-8); a hub clock jump moves its own daily jobs (CH03-6, CH04-6, CH06-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6). The others: CA05-6, CA05-8, CA06-6, CA07-6, CA10-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6. Known parts stay in their rows: R-1189, R-1206, R-1260. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 300). Not built.** **2026-11-11 (afternoon): why the N100's clock is wrong is known (R-1301, `documentation/audits/n100-clock-2026-11-11/README.md`).** Numbers from source: a host clock more than 120 s behind refuses every signed job (`felhom-agent/internal/authz/verifier.go:21`, `:153-162`); at about 1.25 s a day the N100 gets there in about 70 days (mid-January 2027, an estimate from one measurement). | NARROWED — **2026-11-15: LIVE and seen on a box** (hub 0.182.0, agent 0.175.1). Tester 1's clock was set 150 s back twice: the hub said „150 s behind”, one mail went to the operator only, a second alarm inside the hour sent no second mail, and the hub said „in step again” after the clock was put right (`audits/release-2b-3-2026-11-15/` `part2/`, `part3/`). LEFT, as before: the 30 answers themselves still trust the clock. Earlier: NARROWED — the hub's check is built, not released. **2026-11-14 (night build):** the agent stamps its clock at the moment of sending (`sent_at`) and the hub alarms the operator when it is more than one minute from the hub's clock, in either direction, and says so when it is back (`08-alarm-ladder.md`, `box_clock_wrong`). A report that waited on the box raises nothing (tested). Not released, not seen on a box. LEFT after the release: the 30 answers themselves still trust the clock; the alarm only makes a wrong clock known. OPEN — **2026-11-14: the one box whose clock was wrong (the N100, 32.6 s) is right again (R-1301 closed); this row's check is still not built.** Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): the hub compares the time in each report with its own, alarms the operator above one minute, and does not take a time from the future as „fresh”. | CC |
| **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC |
| **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC |
| **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC |
| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-27: agent v0.167.0 RELEASED; in the NORMAL signed delivery the pass now runs right after the commit on all three boxes („running the bundle policy pass after the self-update commit (R-1285)” → `BUNDLE POLICY DONE install_mode=appliance removed=0`; `audits/release-2026-10-27/agent-0167-tester1-demofelhom.txt`). Not live: a BYO box coming FROM 0.164.0 (none in the fleet). **Earlier:** **VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:** **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
| **R-1395** | Box system & updates | P4 | **A step of the host's clock makes the controller read a new boot: it then recreates every drive-backed app, once.** Found 2026-11-14 on the N100 (`audits/release-2a-2026-11-14/D/13b-n100-container-near-the-step.txt`): the time service came back and stepped the clock by +32.7 s; 9 seconds later the controller logged „[gate] boot 1794112550-…: recreating drive-backed app” for radarr, sonarr and qbittorrent (each down for 1 to 5 s; all healthy after). Cause, read on the box: the controller's boot identity carries `/proc/stat` `btime`, and the kernel computes `btime` from the wall clock, so it moved by the same 32 s (the crash guard's `last_boot_at` reads 04:35:18Z, `btime` now 04:35:50Z). The crash guard was NOT fooled: it reads the kernel's `boot_id`. A step this large needs a clock that was wrong for weeks, so a healthy box does not meet it; a fresh box with a wrong hardware clock can, at its first time sync. | NARROWED — built in the agent and the controller, not released. **2026-11-14 (night build):** the boot token is the AGENT's (`guest_boot_id` in its `/disks` answer), not the controller's; the controller only compares it. The agent now also sends `guest_boot_key` (the kernel's `boot_id` in place of the boot time) and the controller's boot pass prefers it; the move to the key recreates nothing; an older agent or controller behaves as before. Tests with a moved boot time on both sides (`audits/night-build-2026-11-14/C/`). Needs BOTH releases to take effect. Not proven on a box (a clock step cannot be staged without changing a box). OPEN — filed 2026-11-14 | — | make the controller's boot identity the kernel's `boot_id` (as the crash guard's is), with a test that a moved `btime` is not a new boot | CC |
## Monitoring & notifications — 29 rows (P2 3, P3 18, P4 8)
@@ -331,9 +328,10 @@ stopping line that lies.
| **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC |
| **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) |
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | NARROWED — the two ruled warnings are built, not released. **2026-11-14 (night build):** the hub warns the operator 14 days before an operator token ends and 14 days before a box's web certificate ends (the box reads the date from its own store and sends it in its report) — `08-alarm-ladder.md` §12.1. The other sixteen are a design table there (§12.2): one is small and waits for a yes (the hub's own five public names), one is a design question (a household's Cloudflare token), the rest stay the operator's list. Not seen on a box or a deployed hub. OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC |
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | NARROWED — **2026-11-14 (night build, second part):** the two things release 2a did not tell are built, not released: the System page has a cell for the kernel clean-up (removed, kept, space gained) and one for the security-profile refresh (the kernel, how many profiles changed, any failed service it restarted) — `11` §5.11.1, §5.12. On the way a display defect was fixed: a clean-up hid a kernel that fell back the night before in the „Kernel step” cell. NARROWED — the four worst cases are built, not released. **2026-11-14 (night build):** the operator is told when a hub daily check's slot passed while the hub was down, when an operator mail was never sent, when a box did not take a signed job in two hours, when a box refused a signed job (with the cause), and when the guest-power watchdog gave up (`05` §16.6, `08-alarm-ladder.md`). Tests only; not deployed, not seen on a box. LEFT: the other 15 answers of the class; a household mail is still never retried (R-1361); the kernel clean-up and profile-refresh detail (next part of release 3). OPEN — **2026-11-14 (release 2a), two more things the hub does not tell:** (1) a kernel clean-up that removed kernels shows only as „cleaned <time ago>” in the System page's „Kernel step” cell — what was removed, kept and gained is stored and shown nowhere; (2) the security-profile refresh sends nothing to the hub at all, also when it restarted a failed service (it did on the N100). Both for release 3. Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
| **R-1396** | Monitoring & notifications | P3 | **The agent now names WHY its trust in Proxmox is wrong, and the hub does not tell the operator yet.** Found 2026-11-14 night (release 2b, built). The agent's host report carries a `proxmox_trust` stanza (`mode`, `problem`, `detail`, the CA copy's end date; `03-host-agent.md` §12a). Two gaps: (1) the hub does not read the stanza — a box whose CA copy does not verify, ends within 30 days, or whose move failed is said only in the box's journal; (2) a box that is fully blind (neither the pinned certificate nor the CA copy works) sends NO host report, because the report starts with a call to Proxmox — that case reaches the operator only as „host stale” after 45 minutes, with no cause. With release 2b a pinned box heals itself at the first failed call, so (2) needs both ways to fail at once. | NARROWED — **2026-11-14 night: gap (1) is BUILT, not deployed** — the hub reads `proxmox_trust` and raises one operator-only `proxmox_trust_problem` per (box, cause), and `proxmox_trust_ok` at the recovery (hub `main` after v0.181.0; the wire-contract gate asked for the receiver in the same push as the sender). **LEFT: gap (2), the fully blind box.** Earlier: OPEN — filed 2026-11-14 | — | hub: an operator-only event from `proxmox_trust.problem` (raise once per cause, clear when it is empty), and the mode on the System page; agent: send a report without the Proxmox part when the failure is one of trust (a design choice: what the hub does with a report that has no guests) | CC |
| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | NARROWED — **2026-11-15: the two warnings are LIVE** (hub 0.182.0, controller 0.353.0). The three test boxes' reports carry their web certificate's end date and issuer (read from a copy of the live database: 2026-12-31, 2027-01-02, 2027-01-19); the hub's end-date check set its baseline at the start. **Not seen:** no real certificate or token 14 days from its end, so no warning was seen. The other sixteen stay a design table; D85 is not ruled. Earlier: NARROWED — the two ruled warnings are built, not released. **2026-11-14 (night build):** the hub warns the operator 14 days before an operator token ends and 14 days before a box's web certificate ends (the box reads the date from its own store and sends it in its report) — `08-alarm-ladder.md` §12.1. The other sixteen are a design table there (§12.2): one is small and waits for a yes (the hub's own five public names), one is a design question (a household's Cloudflare token), the rest stay the operator's list. Not seen on a box or a deployed hub. OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC |
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | NARROWED — **2026-11-15: LIVE** (hub 0.182.0, agent 0.175.1 vouched). Seen on the running hub: the daily-check ledger has its first records, the lost-mail scan ran at the start („none”), the „Security profiles” cell is filled for the three test boxes, the „Kernel clean-up” cell shows the N100's night. **Tests only, not seen on a box:** a refused signed job told, a guest-power give-up told, a lost operator mail told, a skipped daily check told. Earlier: NARROWED — **2026-11-14 (night build, second part):** the two things release 2a did not tell are built, not released: the System page has a cell for the kernel clean-up (removed, kept, space gained) and one for the security-profile refresh (the kernel, how many profiles changed, any failed service it restarted) — `11` §5.11.1, §5.12. On the way a display defect was fixed: a clean-up hid a kernel that fell back the night before in the „Kernel step” cell. NARROWED — the four worst cases are built, not released. **2026-11-14 (night build):** the operator is told when a hub daily check's slot passed while the hub was down, when an operator mail was never sent, when a box did not take a signed job in two hours, when a box refused a signed job (with the cause), and when the guest-power watchdog gave up (`05` §16.6, `08-alarm-ladder.md`). Tests only; not deployed, not seen on a box. LEFT: the other 15 answers of the class; a household mail is still never retried (R-1361); the kernel clean-up and profile-refresh detail (next part of release 3). OPEN — **2026-11-14 (release 2a), two more things the hub does not tell:** (1) a kernel clean-up that removed kernels shows only as „cleaned <time ago>” in the System page's „Kernel step” cell — what was removed, kept and gained is stored and shown nowhere; (2) the security-profile refresh sends nothing to the hub at all, also when it restarted a failed service (it did on the N100). Both for release 3. Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
| **R-1396** | Monitoring & notifications | P3 | **The agent now names WHY its trust in Proxmox is wrong, and the hub does not tell the operator yet.** Found 2026-11-14 night (release 2b, built). The agent's host report carries a `proxmox_trust` stanza (`mode`, `problem`, `detail`, the CA copy's end date; `03-host-agent.md` §12a). Two gaps: (1) the hub does not read the stanza — a box whose CA copy does not verify, ends within 30 days, or whose move failed is said only in the box's journal; (2) a box that is fully blind (neither the pinned certificate nor the CA copy works) sends NO host report, because the report starts with a call to Proxmox — that case reaches the operator only as „host stale” after 45 minutes, with no cause. With release 2b a pinned box heals itself at the first failed call, so (2) needs both ways to fail at once. | NARROWED — **2026-11-15: gap (1) is LIVE and was seen from a real event** (hub 0.182.0): Tester 1 reported `ca_copy_unavailable` for the minutes between its agent update and its root-files update; the hub mailed the operator only, named the cause, and said „no problem any more” at the next report. That alarm was FALSE (the pin worked) and agent 0.175.1 no longer raises it. **LEFT: gap (2), the fully blind box** — a box whose trust does not work sends no report, so a staged wrong CA copy can raise no alarm (it also repairs itself at the agent's start, seen on demo-hp). Earlier: NARROWED — **2026-11-14 night: gap (1) is BUILT, not deployed** — the hub reads `proxmox_trust` and raises one operator-only `proxmox_trust_problem` per (box, cause), and `proxmox_trust_ok` at the recovery (hub `main` after v0.181.0; the wire-contract gate asked for the receiver in the same push as the sender). **LEFT: gap (2), the fully blind box.** Earlier: OPEN — filed 2026-11-14 | — | hub: an operator-only event from `proxmox_trust.problem` (raise once per cause, clear when it is empty), and the mode on the System page; agent: send a report without the Proxmox part when the failure is one of trust (a design choice: what the hub does with a report that has no guests) | CC |
| **R-1397** | Monitoring & notifications | P4 | **One call of the agent to Proxmox is answered 500 during a fresh install, and the agent logs no ERROR line.** Seen 2026-11-15 on the fresh test box (golden 0.353.0, agent 0.175.1): Proxmox's own access log holds `PUT /api2/json/pools/felhom` → 500 at day-0, between 102 calls answered 200; the install ended „SUCCESS”, the guest is in the pool's box and one app installed; the agent's journal has 0 ERROR lines. Not read: which call it is in the agent, whether it is retried, and whether the same line was there on earlier installs (`documentation/tests/golden-0.353.0-2026-11-15/fresh-install/`). | OPEN — filed 2026-11-15 | — | read the pool call in the agent's day-0 path: is the 500 an expected „already a member” answer, and is it logged | CC |
| **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC |
## Hub & operator — 18 rows (P2 1, P3 5, P4 12)
@@ -354,8 +352,8 @@ stopping line that lies.
| **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CH23-6). | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC |
| **R-1345** | Hub & operator | P4 | **An access token's scopes cannot be changed after it is made — widening or narrowing one means a new token (and a new secret).** The operator, 2026-10-29: to give CC `hosts:read` + `hosts:write` for the live proofs he had to make `CC_HOSTS` (now in `sec`) beside `cc-releases-2`. Wanted: edit a token's scopes in the session (an audit line per change), so a scope can also be taken away after a session without retiring the secret. | OPEN | — | design + build the scope edit (session only), red-first test that an unticked scope is refused at once | CC |
| **R-1353** | Hub & operator | P4 | **Statements still over 1 s, named by the new slow-statement line (hub 0.178.0) in its first hour:** a background deadline/restore-test check reading days of full host-report JSON (`GetHostReportsSince`, `store.go`; 1.5–9.2 s cold, from `monitor/deadline.go` and `monitor/restoretest.go` — no page waits on it); the start-up prune of `events` and `app_log_issues` (1.1 s and 1.8 s; no index on `created_at` / `last_seen`); the customer page's newest-report read (`GetCustomer`, 1.3 s cold in the pod probe). With the 1 GiB limit most of this should come from the cache once warm. Evidence: the hub log 2026-10-30 09:22–10:05 UTC; `audits/release-2026-10-30/p2-r1346/`. | OPEN **2026-11-02 read (`audits/security-2026-11-02/part4/hub-slow-lines-2026-11-01-02.txt`; the log covers only the 31 h since DooPlex restarted on 2026-11-01 16:11 CET, the lines from 10-30 are gone with the old pod):** 35 lines, 15 statements. What repeats: `VACUUM INTO` 5× (avg 110 s, the 6-hourly snapshot, expected); `GetHostReportsSince` 9× (avg 7.9 s, max 26.6 s); `GetCustomer`'s newest report 3× (1.1 s). Once, at the cold start: the prune of `events` (10.4 s) and `app_log_issues` (21 s, again 04:30), the OS-release and host lists (2–22 s). No fix before the full week (11-06). **2026-11-10 read (the hub's start after a deploy, `audits/release-space-fixes-2026-11-10/hub/10-watch.txt`):** 11 slow lines in the first 2 minutes of the new pod, the longest 19.1 s (`GetHostReportsSince`), then the OS-report reads (3–4 s) and the two prunes (2.7 s, 4.2 s); none in the 28 minutes after. The 13:02 UTC snapshot's `VACUUM INTO` took 127.8 s. The same statements as the 11-02 read; no new one. | — | read a week of slow-statement lines; index or narrow what still repeats (the history read needs only a few fields, not the whole JSON) | CC |
| **R-1375** | Hub & operator | P2 | **The outside part of three endings is done by hand or not at all — the worst: an off-site account can stay at the provider with no record in the hub.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH06, EH01, EA21 and the ending "a tester ends"; read in source and documents; no account was called). (1) **Unticking the off-site tier** of a household takes nothing down and leaves no reminder (`hub/internal/web/configs.go:1737-1743`). A later RESET takes the account down only when the tier is ticked (`hub/internal/web/customer_reset.go:203-245`), so the sub-account and its data stay, are paid for, and the hub holds no record that they exist. (2) **Switching the second off-site tier off** takes nothing down: storage entry, token and copies stay; the source calls the teardown 'a deliberate future op' (agent `internal/pbsdr/manager.go:274`). (3) **A tester ends**: no handler and no runbook; done by hand once (2026-09-25); no list says what to collect. **Members (answers marked N):** EH06-4, EH01-4, EA21-4, and 'a tester ends' in the table's head. Related: R-810 (a household leaves), R-32. | NARROWED — part 1 (D71) is built, not released. **2026-11-14 (night build):** the hub keeps the record of an off-site account after the tier is unticked; RESET and delete take the account down when a record exists; an untick alone takes nothing down (pinned by a test); the config form says that the account still exists (`05` §14.1). The runbook „a household or a tester leaves” is written (D74, `runbooks/household-leaves.md`). **For the operator, once, after the release:** at the provider's page read each sub-account's label `felhom-customer=<id>`; for a household whose tier is unticked in the hub, tick it again and save — the account's record comes back. LEFT: part (2) has no fault (the second tier keeps its record, read in source and its test); part (3) is the runbook; a household mail at leaving stays R-810. OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D71 = yes, D74 = yes; `09` §3 decisions 291, 294).** Not built. | — | Hub (D71): keep the account record after an untick; RESET and delete take the account down, ticked or not. Docs (D74): the runbook 'a household or a tester leaves', from the five code doors, in a later documents-only session. Parts (2) and (3) of this row have no ruling of their own. | CC |
| **R-1376** | Hub & operator | P3 | **Records of a thing that is gone stay in six places.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH00, EH02, EH21, EA00, EA11, EA27; read in source, `documentation/audits/endings-list-2026-11-10/evidence/hub-delete-statements.txt`; no hub database was read). (1) **Hub:** thirteen tables have no delete statement anywhere. Deleting a box leaves its update settings, its update reports and its kernel notices; deleting a household leaves its off-site key rows, off-site window rows and abandon requests. Most are small; the update reports of each box grow with time. (2) **Host uninstall** leaves the mount units and host mounts of household drives (said when R-179 closed; no open row carried it). (3) **A drive wiped by a signed job** keeps a record and a mount unit that name the old filesystem (agent `internal/signedjobs/wipe.go:61-110`). (4) **The token of a destroyed guest** stays valid until a new guest takes its number: the token store has no remove (agent `internal/localapi/tokenstore.go:127-210`). **Members (answers marked N):** EH00-4, EH02-4, EH21-4, EA00-4, EA11-4, EA27-4. | NARROWED — part 1 (the hub, D73) is built, not released. **2026-11-14 (night build):** deleting a box removes its update settings, update reports, kernel notices and the hub's alarm marks that name it; deleting a household removes its off-site key, window and abandon rows; the operator's action log of a deleted household goes one year after the deletion with the other audit rows (decided by CC unattended — the operator may reverse); the delete journal stays; a daily sweep removes what earlier deletes left (`05` §14.2). The privacy sentence is proposed in the report, not published. LEFT: the agent part (a destroyed guest's token), parts (2) and (3), and an age limit for a LIVE box's update reports (needs a design: the kernel lane reads old reports). OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D73 = yes, with one exception; `09` §3 decision 293).** Not built. | — | Hub (D73): add the tables with no delete to the two purges and to the daily clean-up; the delete journal (`host_deletions`, `customer_resets`) stays on purpose, and the privacy text says so. Agent: remove a guest's token when the guest is destroyed. Parts (2) and (3) have no ruling of their own. | CC |
| **R-1375** | Hub & operator | P2 | **The outside part of three endings is done by hand or not at all — the worst: an off-site account can stay at the provider with no record in the hub.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH06, EH01, EA21 and the ending "a tester ends"; read in source and documents; no account was called). (1) **Unticking the off-site tier** of a household takes nothing down and leaves no reminder (`hub/internal/web/configs.go:1737-1743`). A later RESET takes the account down only when the tier is ticked (`hub/internal/web/customer_reset.go:203-245`), so the sub-account and its data stay, are paid for, and the hub holds no record that they exist. (2) **Switching the second off-site tier off** takes nothing down: storage entry, token and copies stay; the source calls the teardown 'a deliberate future op' (agent `internal/pbsdr/manager.go:274`). (3) **A tester ends**: no handler and no runbook; done by hand once (2026-09-25); no list says what to collect. **Members (answers marked N):** EH06-4, EH01-4, EA21-4, and 'a tester ends' in the table's head. Related: R-810 (a household leaves), R-32. | NARROWED — **2026-11-15: part 1 is LIVE** (hub 0.182.0). The record of an off-site account now outlives an untick. **Tests only:** the provider path (no call to the provider was made today; no household was unticked). The operator's one-time label check at the provider is still to do. Earlier: NARROWED — part 1 (D71) is built, not released. **2026-11-14 (night build):** the hub keeps the record of an off-site account after the tier is unticked; RESET and delete take the account down when a record exists; an untick alone takes nothing down (pinned by a test); the config form says that the account still exists (`05` §14.1). The runbook „a household or a tester leaves” is written (D74, `runbooks/household-leaves.md`). **For the operator, once, after the release:** at the provider's page read each sub-account's label `felhom-customer=<id>`; for a household whose tier is unticked in the hub, tick it again and save — the account's record comes back. LEFT: part (2) has no fault (the second tier keeps its record, read in source and its test); part (3) is the runbook; a household mail at leaving stays R-810. OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D71 = yes, D74 = yes; `09` §3 decisions 291, 294).** Not built. | — | Hub (D71): keep the account record after an untick; RESET and delete take the account down, ticked or not. Docs (D74): the runbook 'a household or a tester leaves', from the five code doors, in a later documents-only session. Parts (2) and (3) of this row have no ruling of their own. | CC |
| **R-1376** | Hub & operator | P3 | **Records of a thing that is gone stay in six places.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH00, EH02, EH21, EA00, EA11, EA27; read in source, `documentation/audits/endings-list-2026-11-10/evidence/hub-delete-statements.txt`; no hub database was read). (1) **Hub:** thirteen tables have no delete statement anywhere. Deleting a box leaves its update settings, its update reports and its kernel notices; deleting a household leaves its off-site key rows, off-site window rows and abandon requests. Most are small; the update reports of each box grow with time. (2) **Host uninstall** leaves the mount units and host mounts of household drives (said when R-179 closed; no open row carried it). (3) **A drive wiped by a signed job** keeps a record and a mount unit that name the old filesystem (agent `internal/signedjobs/wipe.go:61-110`). (4) **The token of a destroyed guest** stays valid until a new guest takes its number: the token store has no remove (agent `internal/localapi/tokenstore.go:127-210`). **Members (answers marked N):** EH00-4, EH02-4, EH21-4, EA00-4, EA11-4, EA27-4. | NARROWED — **2026-11-15: part 1 is LIVE** (hub 0.182.0). The first sweep at the start deleted 26 rows (old update reports of 13 boxes deleted earlier), the same count as the preview on a copy; the ids are kept off git. A box delete by the operator at 11:29 ran with the new code. **One thing the sweep does not cover:** the hub's own alarm marks of a box deleted BEFORE the release (4 marks of one such box were read today) — a delete now removes them, the sweep of old ones does not. Small; to add to the sweep. Earlier: NARROWED — part 1 (the hub, D73) is built, not released. **2026-11-14 (night build):** deleting a box removes its update settings, update reports, kernel notices and the hub's alarm marks that name it; deleting a household removes its off-site key, window and abandon rows; the operator's action log of a deleted household goes one year after the deletion with the other audit rows (decided by CC unattended — the operator may reverse); the delete journal stays; a daily sweep removes what earlier deletes left (`05` §14.2). The privacy sentence is proposed in the report, not published. LEFT: the agent part (a destroyed guest's token), parts (2) and (3), and an age limit for a LIVE box's update reports (needs a design: the kernel lane reads old reports). OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D73 = yes, with one exception; `09` §3 decision 293).** Not built. | — | Hub (D73): add the tables with no delete to the two purges and to the daily clean-up; the delete journal (`host_deletions`, `customer_resets`) stays on purpose, and the privacy text says so. Agent: remove a guest's token when the guest is destroyed. Parts (2) and (3) have no ruling of their own. | CC |
| **R-1386** | Hub & operator | P3 | **Timed jobs with no time limit and no measured duration (25 answers).** Found 2026-11-11 (the Clock list, class W4, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source; the hub was not measured, by ruling). The hub's 60-second sweep runs its checks one after the other with no limit for each: one slow check delays all the others, the liveness check too (CH19-3, CH19-8, CH01-3, CH08-3, CH08-8, CH10-3, CH13-8, CH18-3, CH06-8, CH02-3); the 07:10 audit has 10 minutes, measured with three boxes only (CH11-3, CH11-8). On the box: the interval jobs have no limit of their own (CC11-3, CC12-3, CC13-3, CC14-3, CC15-3, CC16-3, CC20-3, CA06-3, CA10-3, CA10-8, CA13-8, CA15-3, CA19-3). Every measured duration is from a small test store. | OPEN — filed 2026-11-11 (clock list) | — | Hub: log each sweep step's duration and alarm when one sweep takes longer than its interval; then read a week of numbers and set limits. Box: the same for the controller's interval jobs. | CC |
| **R-1393** | Hub & operator | P4 | **For about 8 minutes after a hub restart its database statements take 1 to 32 seconds; the System page took 12.7 s and a box's enrolment 11.4 s.** Measured 2026-11-14 at the 0.181.0 deploy (`audits/release-1-dates-2026-11-14/hub/05-watch.txt`): 24 `slow statement` lines from 09:58 to 10:06 UTC, none in the 20 minutes after. The database file is 678 MB; the slow ones read `host_reports`, `os_reports` and `events` (whole report JSON per row). One 32-second statement ran while CC copied the database out of the pod for a read (CC's own load). No report was lost that CC could see; not measured: whether a box's report timed out in that window. | OPEN — filed 2026-11-14 | — | measure a restart with no copy running; then decide: an index or a shorter keep on `host_reports`/`os_reports`, or accept | CC |
@@ -423,7 +421,6 @@ stopping line that lies.
| item | due (UTC) | what to measure |
|---|---|---|
| R-1293 | 2027-03-29 | the spring clock change: the hub's 02:00 snapshot and the 02:30 push on 2027-03-28 (02:00–03:00 does not exist) — which file each push took; detail in the R-1293 row |
| R-1050 | 2026-11-21 | HSTS one year in the next controller release if no https trouble since 2026-10-15; detail in the R-1050 row |
| R-1035 | 2026-11-18 | „felhő tárhely” + „saját felhő tárhely” impressions and average position, and the new page's own impressions/clicks/position, 4 weeks after publishing; operator supplies the read; detail in the R-1035 row |
| R-528 | 2026-11-28 | no further failed oom_check after the one on 2026-11-11 (demo-felhom), no unexplained app_oom; detail in the R-528 row |
| R-1035 | 2026-12-09 | the same three numbers 8 weeks after publishing, compared with the 4-week read; a null at 4 weeks is not a verdict, both reads are needed; detail in the R-1035 row |
+4
View File
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
## Moved from STATUS on 2026-11-15 (midday)
**Updated 2026-11-15 (morning).** The N100 removed 6 of its 8 old kernels by itself at night and is healthy: 1 of the 2 nights D82 asks for; demo-hp had nothing to remove (0 of 2). Releases 2b and 3 are built and tested in source; nothing is released (`REPORT-night-build-2026-11-14.md`).
## Moved from STATUS on 2026-11-15 (morning)
**Updated 2026-11-14 (afternoon).** Release 2a is out: agent 0.174.0, vouched. A ring-0 box now removes its old kernels at night, and a box reloads its security profiles after a kernel change. The N100 keeps time again; the product repaired it, with no hand step.
+1
View File
@@ -56,3 +56,4 @@ what happened, who noticed, and the register row it became. Prose here is not a
- **2026-11-10 — every place that holds data was measured (read-only; the space list).** Host system disk 38.09 of 100.86 GB (38 %), 4.6 GiB more than on 10-15: the box holds 8 kernels (8.15 GB) and none was ever removed → R-1368. The agent's self-update folder holds 48 old binaries (724 MB) and there are 11 old agent copies beside the binary (155 MB) → R-1368. The host journal sits at its cap (4.28 GB, 74 files, oldest entry 10-04) → R-1367. The three whole-box copies of 2026-07-28 still lie on the system disk (17.59 GB) → R-1087; four template archives (2.56 GB) → R-1368. On the USB drive, `backups/offsite-restore` holds 1.65 GB of two apps that are not installed any more → R-1372. Nothing was deleted or changed. Table: `audits/space-list-2026-11-10/TABLE.md`. (CC)
- **2026-11-14 — the clock is right again, repaired by the product.** When the box took agent 0.174.0 and its root files (11:36 and 11:51 UTC), the new „profile refresh" found the old security profile of the time service, loaded the right one and started the service: the clock stepped forward 32.7 s (offset to DooPlex −32.55 s → +0.18 s; still so 2.5 hours later). Nine seconds after the step the controller read a new boot and recreated Radarr, Sonarr and qBittorrent once (each down for a few seconds; all healthy). A dry run says tonight's night removes 6 of the 8 kernels. Register: R-1301 and R-1388 closed, R-1395 (the controller and the clock step), R-1368 (the kernels — read the night of 11-15). Evidence: `audits/release-2a-2026-11-14/D/` (`09`, `13`, `13b`, `14`). (CC)
- **2026-11-15 — the box removed its old kernels by itself, the first time.** At 05:58 CET, at the end of the night's update run (the Proxmox step applied 12 packages first), the clean-up purged 6 of 8 kernel packages in 35 s and kept 7.0.14-26 and the running 7.0.14-28: 6.81 GB gained, 65.1 GB free after, 0 left-over paths. The default kernel and the pinned file did not change; 0 failed units, 9 of 9 containers healthy, the clock +0.06 s to DooPlex; the hub's „Kernel step” cell reads „cleaned”. Nothing was done by hand. Register: R-1368 (D82: 1 of 2 nights). Evidence: `audits/night-build-2026-11-14/E/`. (CC, read-only)
- **2026-11-15 — the box took releases 2b and 3 by itself.** Agent 0.175.1 at 10:21 and its root files at 10:36 CET (signed jobs), controller 0.353.0 at 11:08 (the floor). At 10:36:58 the agent moved from the pinned Proxmox certificate to the copy of the box's own root CA, after a working call with the copy; no alarm, no hand step. The controller moved its boot record to the kernel's boot key and recreated no app. 9 of 9 containers healthy after. Register: R-1380 and R-1395 closed. Evidence: `audits/release-2b-3-2026-11-15/part2/n100-moved.txt`, `part3/boxwalk-0353.txt`. (CC, reads only)
@@ -0,0 +1,6 @@
reverted to virgin
ssh after about 20 s: pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
update successful
template: debian-13-standard_13.6-1_amd64.tar.zst
download of 'http://download.proxmox.com/images/system/debian-13-standard_13.6-1_amd64.tar.zst' to '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst' finished
build-golden.sh: 645b3b659cbadebe v3.0.0
@@ -0,0 +1,3 @@
registry file downloaded with no login: 640207443 bytes, sha256 86b5ba20a15ebd511db8828f51d38fb984f68efa23c4d4e89ae33aa195f19971 (= the bake)
registry login files in the archive: 0 of 41384 entries
token in the committed copy of bake.log: 0 (control with the token appended: 1)
@@ -0,0 +1,7 @@
## in the drill VM 2026-11-15T10:34:55Z
token in bake.log: 0 (control with the token appended: 1)
pass markers: docker OK=1 including-mount-point=2 excluding=0 FATAL=0 not-set=0 upload=1
purging CT 9100 from related configurations..
pct destroy rc=0
ls: cannot access '/root/.gitea-token': No such file or directory
ls: cannot access '/root/bake.log': No such file or directory
@@ -0,0 +1,4 @@
2026/11/15 11:07:35 [INFO] Artifact manifest set: agent=0.175.1 golden=0.352.1 min_agent="0.158.1" wrapper_sha=false bundle_sha="07ebf86362ed9d44776416bd5c2c4b6b629e44d34595a8120fd8fc3f4d9277ff"
2026/11/15 11:07:35 [INFO] artifact vouch timing (R-1346): registry probes 91ms, sha lookups 19ms, bundle sha 10ms, fleet check + save 256ms, total 379ms
2026/11/15 11:52:54 [INFO] Artifact manifest set: agent=0.175.1 golden=0.353.0 min_agent="0.158.1" wrapper_sha=false bundle_sha="07ebf86362ed9d44776416bd5c2c4b6b629e44d34595a8120fd8fc3f4d9277ff"
2026/11/15 11:52:54 [INFO] artifact vouch timing (R-1346): registry probes 154ms, sha lookups 23ms, bundle sha 12ms, fleet check + save 106ms, total 296ms
@@ -0,0 +1,62 @@
# Golden 0.353.0 — bake, publish, one fresh install, vouch — 2026-11-15 (releases 2b and 3, Part 4)
Why: controller 0.353.0 became the global floor today, and the installer changed (a new box trusts Proxmox by a copy
of its own root CA). One fresh install proves the golden and the installer together.
Procedure: `runbooks/RUNBOOK-manual-build.md` §4, the same recipe as `../golden-0.352.1-2026-11-14/`.
| | Previous (`../golden-0.352.1-2026-11-14/`) | This bake |
|---|---|---|
| `build-golden.sh` | v3.2.0 (sha256 `645b3b65…`) | the same file |
| Controller | `felhom-controller:0.352.1` | **`felhom-controller:0.353.0`** (MinAgent **0.158.1**) |
| Docker engine | the six approved pairs (29.9.2) | the same |
| Template | `debian-13-standard_13.6-1_amd64.tar.zst` | the same |
| Token | from `sec`, file → file | the same; 0 in the unit's properties |
## Pass markers (`bake.log`)
```
[golden] build-golden.sh v3.2.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.353.0
docker OK (overlay2; data-root /var/lib/docker)
[golden] upload OK (HTTP 201)
GOLDEN_VERSION=0.353.0
GOLDEN_SHA256=86b5ba20a15ebd511db8828f51d38fb984f68efa23c4d4e89ae33aa195f19971
```
`including mount point` 2, `excluding` 0, `FATAL` 0, `GOLDEN_DOCKER_PKGS not set` 0 (`03-teardown.txt`). The token is
not in `bake.log`: 0, with a control of 1, in the VM and in the committed copy. Round trip: the registry's file,
downloaded with no login, has the bake's sha256; 640,207,443 bytes; no registry login file in the archive
(`02-round-trip.txt`). Bake 10:28:40Z to before 10:34:46Z. The build guest was destroyed and the drill VM powered off.
**Vouched by the operator 2026-11-15 11:52:54 CET** (`04-vouch-readback.txt`: agent 0.175.1, golden 0.353.0, min
agent 0.158.1, the bundle of the release).
**Golden gate:** see the release report; it was run after this folder was committed.
## The fresh install (`fresh-install/`)
**The path, exactly.** The hub offered only the vouched golden (0.352.1) at that time, and the new installer was not
tagged. So the box ran **the untagged `scripts/felhom-host-install.sh` of felhom.eu commit `6174944f`**
(`SCRIPT_VERSION` 1.37.0, sha256 `93a021db…f26e65` — the file the tag `installer-v1.37.0` names since) with **one
added line** after line 479, which names golden 0.353.0 and its sha256 (`installer-the-one-line.diff.txt`). It was
copied in from DooPlex and run by hand as root in the direct shape (`--customer-id Tester4 --mode appliance --hub-url
https://hub.felhom.eu --passphrase-file …`), not through the bind page. All other steps are the product's own.
| Step | Result | Proof |
|---|---|---|
| VM 9386 `golden1115-box` on demo-hp, disks on `nvme-scratch`, Proxmox from the Felhom ISO 1.30.0 (sha256 checked against the published file) | Install pressed 10:41:12Z; SSH answered before 10:44:58Z | `install/` |
| CC's SSH key on the host before the installer ran | done | — |
| Time service on the fresh host | `chrony` active, 0.0000003 s off | — |
| Installer 1.37.0, started 10:45:22Z | „Day-0 provision SUCCESS — vmid=9201 host_id=Tester4-2f0c25” before 10:48:22Z | `box-state-and-install-log.txt` |
| **Trust in Proxmox** | the installer's line: „the box's own root CA (… a handshake to 127.0.0.1:8006 verified with it alone) — no pinned certificate”; the agent's config holds `ca_file` and an EMPTY fingerprint; the copy equals `/etc/pve/pve-root-ca.pem` | `trust-state.txt` |
| **The agent controls Proxmox** | Proxmox's own access log: 103 calls by the agent's token, 102 answered 200, one 500 (`PUT /pools/felhom`, filed as R-1397) | `trust-state.txt` |
| Golden on the box | sha256 `86b5ba20…9971` = the bake and the registry | read on the box |
| Agent / controller | 0.175.1 / 0.353.0 from the golden; 0 ERROR lines in both | `box-state-and-install-log.txt` |
| Hub | host `Tester4-2f0c25`, reports arrive; the floor is served (0.353.0) | the hub's log |
| Setup-code mail | arrived 10:46:00Z, to the test household's address | the test mailbox |
| Claim page | `POST /claim` → 302 `/` at 10:48:52Z | — |
| One app: BentoPDF, through the page's own deploy request | 202 at 10:48:59Z; `running` at 10:51:09Z; `pdf.tester4.test` answers 200; a name with no app answers 404 | `app.txt` |
## Teardown (`fresh-install/teardown.txt`)
VM 9386 destroyed with its disks; the installer ISO removed from demo-hp; the throwaway root password and the session
cookie shredded on DooPlex. **Left for the operator (only he can):** hub host `Tester4-2f0c25`; the appliance with
smbios uuid `ab0b8310-d5f3-4042-9e7d-0a0db2135433`, if it is on the Hosts page.
@@ -0,0 +1,340 @@
[golden] build-golden.sh v3.2.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.353.0
[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) …
Logical volume "vm-9100-disk-0" created.
Logical volume pve/vm-9100-disk-0 changed.
Creating filesystem with 8388608 4k blocks and 2097152 inodes
Filesystem UUID: 7979a8e1-e6ac-40e4-9747-e4d8383e43f3
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624
Logical volume "vm-9100-disk-1" created.
Logical volume pve/vm-9100-disk-1 changed.
Creating filesystem with 6291456 4k blocks and 1572864 inodes
Filesystem UUID: a2585570-4065-4377-b18e-66cb89a4c914
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst'
Total bytes read: 553512960 (528MiB, 131MiB/s)
Detected container architecture: amd64
Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ...
done: SHA256:IXRGusWh5539gqd6Vw2ceApNbd6+6h58AaV2BpSEsyU root@felhom-golden
Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ...
done: SHA256:hoSHcUoMdZuk2thmtJ+60PgwoDAsQDSgLM7E4Erungc root@felhom-golden
Creating SSH host key 'ssh_host_rsa_key' - this may take some time ...
done: SHA256:LaF1qibflJm6F9Qfd6QmhIm/PEYYcov6EOfH2Q7oFHY root@felhom-golden
[golden] starting + installing Docker (official repo, trixie channel) …
[golden] Docker engine set PINNED to the approved release: containerd.io=2.4.2-3~debian.13~trixie docker-buildx-plugin=0.38.1-1~debian.13~trixie docker-ce=5:29.9.2-1~debian.13~trixie docker-ce-cli=5:29.9.2-1~debian.13~trixie docker-ce-rootless-extras=5:29.9.2-1~debian.13~trixie docker-compose-plugin=5.6.1-1~debian.13~trixie
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = (unset),
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = (unset),
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
installed: containerd.io 2.4.2-3~debian.13~trixie
installed: docker-buildx-plugin 0.38.1-1~debian.13~trixie
installed: docker-ce 5:29.9.2-1~debian.13~trixie
installed: docker-ce-cli 5:29.9.2-1~debian.13~trixie
installed: docker-ce-rootless-extras 5:29.9.2-1~debian.13~trixie
installed: docker-compose-plugin 5.6.1-1~debian.13~trixie
[golden] no approved guest release given - the template versions stay; first-night count vs an approved release: n/a
[golden] pending Debian upgrades in the baked guest (what a FUTURE approval may bring): 63
[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …
[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds …
[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
4f55086f7dd0: Pulling fs layer
4f55086f7dd0: Verifying Checksum
4f55086f7dd0: Download complete
4f55086f7dd0: Pull complete
Digest: sha256:cf1b20a0c89a666b6950fecda7ccbc1b1a86520e7c53f908f3a21e79d0d60458
Status: Downloaded newer image for hello-world:latest
docker OK (overlay2; data-root /var/lib/docker)
live-restore: on
/var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4
/mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4
both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576
[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.353.0 (no registry cred at deploy) …
WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
Configure a credential helper to remove this warning. See
https://docs.docker.com/go/credential-store/
0.353.0: Pulling from admin/felhom-controller
ecc510c1e359: Pulling fs layer
c3f83389ce1b: Pulling fs layer
e81c1fce9826: Pulling fs layer
63a1406b1961: Pulling fs layer
d747c9b6922d: Pulling fs layer
e00f70f14abe: Pulling fs layer
341f4e3af64b: Pulling fs layer
63a1406b1961: Waiting
d747c9b6922d: Waiting
e00f70f14abe: Waiting
341f4e3af64b: Waiting
e81c1fce9826: Verifying Checksum
e81c1fce9826: Download complete
c3f83389ce1b: Verifying Checksum
c3f83389ce1b: Download complete
d747c9b6922d: Verifying Checksum
d747c9b6922d: Download complete
63a1406b1961: Verifying Checksum
63a1406b1961: Download complete
e00f70f14abe: Verifying Checksum
e00f70f14abe: Download complete
341f4e3af64b: Verifying Checksum
341f4e3af64b: Download complete
ecc510c1e359: Download complete
ecc510c1e359: Pull complete
c3f83389ce1b: Pull complete
e81c1fce9826: Pull complete
63a1406b1961: Pull complete
d747c9b6922d: Pull complete
e00f70f14abe: Pull complete
341f4e3af64b: Pull complete
Digest: sha256:61cf2a8bfb82e02694bf85cca774bce2cbb9f947d395f91af7640aa5a5d944f0
Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.353.0
gitea.dooplex.hu/admin/felhom-controller:0.353.0
[golden] asking the controller which infra images it manages …
[golden] baking infra images (4): traefik:v3.7.17 cloudflare/cloudflared:2026.10.4 gtstef/filebrowser:1.5.8-stable gitea.dooplex.hu/admin/felhom-samba:1.2.0 …
v3.7.17: Pulling from library/traefik
881fc932939e: Pulling fs layer
c901ba6ff9e0: Pulling fs layer
4f376259b112: Pulling fs layer
7705a4c1b3f8: Pulling fs layer
7705a4c1b3f8: Waiting
c901ba6ff9e0: Download complete
881fc932939e: Verifying Checksum
881fc932939e: Download complete
7705a4c1b3f8: Verifying Checksum
7705a4c1b3f8: Download complete
4f376259b112: Verifying Checksum
4f376259b112: Download complete
881fc932939e: Pull complete
c901ba6ff9e0: Pull complete
4f376259b112: Pull complete
7705a4c1b3f8: Pull complete
Digest: sha256:f4bb143d1135a579b630ed491e17ec50f6f7c9c6a30585ff3d385471d2a86f5a
Status: Downloaded newer image for traefik:v3.7.17
docker.io/library/traefik:v3.7.17
2026.10.4: Pulling from cloudflare/cloudflared
2cc7ee286bf3: Pulling fs layer
c172f21841df: Pulling fs layer
218cf840d0d9: Pulling fs layer
f6069939f718: Pulling fs layer
d6b1b89eccac: Pulling fs layer
2780920e5dbf: Pulling fs layer
7c12895b777b: Pulling fs layer
3214acf345c0: Pulling fs layer
52630fc75a18: Pulling fs layer
dd64bf2dd177: Pulling fs layer
b839dfae01f6: Pulling fs layer
ebddc55facdc: Pulling fs layer
a0921f9d8de9: Pulling fs layer
b96fe2995f90: Pulling fs layer
810e2b405a08: Pulling fs layer
bd8962e29291: Pulling fs layer
cac2ae0193cb: Pulling fs layer
eb8da9d09a5e: Pulling fs layer
dd64bf2dd177: Waiting
b839dfae01f6: Waiting
ebddc55facdc: Waiting
a0921f9d8de9: Waiting
b96fe2995f90: Waiting
810e2b405a08: Waiting
bd8962e29291: Waiting
cac2ae0193cb: Waiting
eb8da9d09a5e: Waiting
2780920e5dbf: Waiting
7c12895b777b: Waiting
3214acf345c0: Waiting
52630fc75a18: Waiting
f6069939f718: Waiting
d6b1b89eccac: Waiting
2cc7ee286bf3: Verifying Checksum
c172f21841df: Verifying Checksum
c172f21841df: Download complete
218cf840d0d9: Verifying Checksum
218cf840d0d9: Download complete
f6069939f718: Verifying Checksum
f6069939f718: Download complete
2cc7ee286bf3: Pull complete
d6b1b89eccac: Verifying Checksum
d6b1b89eccac: Download complete
2780920e5dbf: Verifying Checksum
2780920e5dbf: Download complete
7c12895b777b: Verifying Checksum
7c12895b777b: Download complete
3214acf345c0: Download complete
52630fc75a18: Verifying Checksum
52630fc75a18: Download complete
dd64bf2dd177: Verifying Checksum
dd64bf2dd177: Download complete
b839dfae01f6: Verifying Checksum
b839dfae01f6: Download complete
ebddc55facdc: Verifying Checksum
ebddc55facdc: Download complete
c172f21841df: Pull complete
a0921f9d8de9: Download complete
b96fe2995f90: Verifying Checksum
b96fe2995f90: Download complete
810e2b405a08: Verifying Checksum
810e2b405a08: Download complete
bd8962e29291: Verifying Checksum
bd8962e29291: Download complete
cac2ae0193cb: Verifying Checksum
cac2ae0193cb: Download complete
eb8da9d09a5e: Verifying Checksum
eb8da9d09a5e: Download complete
218cf840d0d9: Pull complete
f6069939f718: Pull complete
d6b1b89eccac: Pull complete
2780920e5dbf: Pull complete
7c12895b777b: Pull complete
3214acf345c0: Pull complete
52630fc75a18: Pull complete
dd64bf2dd177: Pull complete
b839dfae01f6: Pull complete
ebddc55facdc: Pull complete
a0921f9d8de9: Pull complete
b96fe2995f90: Pull complete
810e2b405a08: Pull complete
bd8962e29291: Pull complete
cac2ae0193cb: Pull complete
eb8da9d09a5e: Pull complete
Digest: sha256:c6c7ae1d5dfdcb094ad344ed68ccffc26635c9c6be80965664c33baf064b0515
Status: Downloaded newer image for cloudflare/cloudflared:2026.10.4
docker.io/cloudflare/cloudflared:2026.10.4
1.5.8-stable: Pulling from gtstef/filebrowser
e2de96513ba9: Pulling fs layer
93c39e448959: Pulling fs layer
b7597036c324: Pulling fs layer
c6591d351da9: Pulling fs layer
4f4fb700ef54: Pulling fs layer
74aec9809d97: Pulling fs layer
9f7b5accdf20: Pulling fs layer
19114688b75e: Pulling fs layer
2067d5df1f91: Pulling fs layer
5669e416ca05: Pulling fs layer
08c322a8a1d4: Pulling fs layer
c6591d351da9: Waiting
4f4fb700ef54: Waiting
74aec9809d97: Waiting
9f7b5accdf20: Waiting
19114688b75e: Waiting
2067d5df1f91: Waiting
5669e416ca05: Waiting
08c322a8a1d4: Waiting
e2de96513ba9: Verifying Checksum
e2de96513ba9: Download complete
c6591d351da9: Verifying Checksum
c6591d351da9: Download complete
4f4fb700ef54: Verifying Checksum
4f4fb700ef54: Download complete
b7597036c324: Verifying Checksum
b7597036c324: Download complete
e2de96513ba9: Pull complete
9f7b5accdf20: Verifying Checksum
9f7b5accdf20: Download complete
93c39e448959: Verifying Checksum
93c39e448959: Download complete
19114688b75e: Verifying Checksum
19114688b75e: Download complete
74aec9809d97: Verifying Checksum
74aec9809d97: Download complete
2067d5df1f91: Verifying Checksum
2067d5df1f91: Download complete
5669e416ca05: Verifying Checksum
5669e416ca05: Download complete
08c322a8a1d4: Verifying Checksum
08c322a8a1d4: Download complete
93c39e448959: Pull complete
b7597036c324: Pull complete
c6591d351da9: Pull complete
4f4fb700ef54: Pull complete
74aec9809d97: Pull complete
9f7b5accdf20: Pull complete
19114688b75e: Pull complete
2067d5df1f91: Pull complete
5669e416ca05: Pull complete
08c322a8a1d4: Pull complete
Digest: sha256:68455d4953bad8e984e1bc5d112ed1e9c918a0a04f104c13c9ece50790308894
Status: Downloaded newer image for gtstef/filebrowser:1.5.8-stable
docker.io/gtstef/filebrowser:1.5.8-stable
1.2.0: Pulling from admin/felhom-samba
e2de96513ba9: Already exists
0ddbc9f3a6b6: Pulling fs layer
ce57a3f93416: Pulling fs layer
fb94eeec2fe1: Pulling fs layer
fb94eeec2fe1: Verifying Checksum
fb94eeec2fe1: Download complete
ce57a3f93416: Verifying Checksum
ce57a3f93416: Download complete
0ddbc9f3a6b6: Verifying Checksum
0ddbc9f3a6b6: Download complete
0ddbc9f3a6b6: Pull complete
ce57a3f93416: Pull complete
fb94eeec2fe1: Pull complete
Digest: sha256:da96f25033827fbb77d0ee398e86fe5b744e2f29be9de898994533340c97acf9
Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.2.0
gitea.dooplex.hu/admin/felhom-samba:1.2.0
[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) …
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'.
[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) …
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'.
[golden] baking the first-boot SSH host-key regeneration unit (F3) …
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'.
[golden] identity-clean + minimize …
[golden] stop + archive …
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/var/lib/felhom') in backup
INFO: archive file size: 610MB
INFO: Finished Backup of VM 9100 (00:00:30)
[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_11_15-11_33_13.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive)
[golden] publishing golden (640207443 bytes, sha256 86b5ba20a15ebd51…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.353.0/golden.tar.zst
[golden] pre-delete existing: HTTP 404 (404/204 expected)
[golden] upload OK (HTTP 201)
GOLDEN_VERSION=0.353.0
GOLDEN_SHA256=86b5ba20a15ebd511db8828f51d38fb984f68efa23c4d4e89ae33aa195f19971
[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.353.0 / 86b5ba20a15ebd511db8828f51d38fb984f68efa23c4d4e89ae33aa195f19971
[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge)
@@ -0,0 +1,23 @@
session csrf_len 64
controller version on the box: {"ok":true,"message":"felhom-controller is healthy"}
2026-11-15T10:48:59Z
{"ok":true,"message":"Telepítés elindítva – az állapot a kártyán követhető"}
HTTP 202
10:48:59 {'deployed': True, 'state': 'not_deployed', 'subdomain': None}
10:49:09 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:49:19 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:49:29 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:49:39 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:49:49 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:49:59 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:50:09 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:50:19 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:50:29 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:50:39 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:50:49 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:50:59 {'deployed': True, 'state': 'deploying', 'subdomain': None}
10:51:09 {'deployed': True, 'state': 'running', 'subdomain': None}
bentopdf name: pdf.tester4.test
route: 200 at 2026-11-15T10:51:22Z
negative control (no such app name): 404
@@ -0,0 +1,160 @@
felhom-agent 0.175.1
ghcr.io/alam00000/bentopdf:v2.8.10 Up 23 seconds (healthy)
gtstef/filebrowser:1.5.8-stable Up 3 minutes (healthy)
gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 3 minutes (healthy)
traefik:v3.7.17 Up 3 minutes
2026/11/15 10:47:57 [INFO] felhom-controller 0.353.0 starting (customer: Tester4, domain: tester4.test)
errors: 0
agent journal ERROR lines: 0
[INFO] auto-sized guest RAM: 5847 MiB (host 9943 MiB; clamp(host-4096, min 4096, max host-2048), ceiling host-1024)
[INFO] auto-sized guest cores: 5 (host 6 cores; host-1, min 2)
[INFO] felhom-host-install v1.37.0 — mode=appliance customer=Tester4 vmid=9201
[STEP] 1/8 pre-flight
[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
[INFO] node: goldenbox (auto)
[INFO] agent config: /etc/felhom-agent/agent.json
[INFO] agent: not installed yet — will be fetched + installed in step 5/8
[INFO] local-lvm free: ~19 GiB
[WARN] local-lvm free ~19 GiB is below the recommended 120 GiB — the install continues; proceed only if you sized the grows deliberately
[INFO] free RAM: ~8351 MiB
[INFO] existing guests on this host: 0 (pct+qm)
[INFO] archive storage 'local' present
[INFO] acl storage 'felhom-pbs' not present yet — expected: the PBS-DR tier creates it; the grant is pre-positioned deliberately
[INFO] dnsmasq: not present before Felhom — recorded; uninstall will remove it again if we install it
[INFO] hub reachable (https://hub.felhom.eu)
[OK] customer 'Tester4' exists + passphrase valid
[INFO] golden: none local — will fetch + verify from Gitea in step 7/8
[OK] pre-flight passed
[STEP] 2/8 Proxmox API token
[OK] token minted (secret captured, not logged)
[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm felhom-pbs])
[SKIP] old broad role FelhomAgent already absent
[STEP] 3/8 compute volume grows
[INFO] auto-computed from a 19 GiB thin pool (16 §7: 50/100/130 GiB)
[INFO] grows: rootfs +0G (->32G), data +26G (->50G, ONE volume)
[STEP] 4/8 host enrollment (POST /host-enroll)
[OK] host MINTED (first enroll)
[INFO] host_id: Tester4-2f0c25 (api_key captured, not logged)
[STEP] 4b/8 break-glass credential (root@pam console password → hub vault)
[INFO] recovery password vaulted at the hub (stored) — now setting it on root@pam
[OK] root@pam password vaulted, set, and the PVE login takes it (retrieve via the operator /admin path; never logged here)
[WARN] NOTE: the root@pam password just CHANGED — the old one now fails at the PVE web GUI (:8006).
[WARN] Retrieve the new one at hub → host page (vaulted recovery credential).
[STEP] 5/8 agent install (fetch + verify + install)
[INFO] disabled pve-enterprise.sources (Enabled: no)
[INFO] disabled ceph.sources (Enabled: no)
[INFO] added pve-no-subscription.sources (suite=trixie)
[OK] apt repos aligned to no-subscription (changed; apt-get update OK)
[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
[INFO] manifest: agent v0.175.1 (sha 38b1e2350de90a79…), golden v0.353.0
[INFO] fetching agent binary v0.175.1 from Gitea …
[OK] verified sha256 38b1e2350de90a79… matches the hub manifest
[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.175.1)
[INFO] installing the 'sudo' package (required for the non-root agent model) …
[OK] sudo installed (Sudo version 1.9.16p2)
[INFO] installing the 'age' package (escrow ceremony identity-wrap dependency) …
[OK] age installed (1.2.1)
[OK] created service user felhom-agent
[OK] added felhom-agent to systemd-journal (unprivileged journal read for NAS verify)
[OK] verified sha256 07ebf86362ed9d44… matches the hub manifest
os-apply: BUNDLE START agent=0.175.1 sha=07ebf86362ed9d44 authority=installer files=32 write=31 same=1 kept=0 skipped=0
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mkfs-guarded (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-selfupdate-guarded (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-pbs-apply (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-backup-target-apply (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-vzdump-partial-clean (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-thin-check (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-internal-place (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-crash-guard (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-priv-apply (new)
os-apply: BUNDLE WROTE /var/lib/vz/snippets/felhom-guest-hook.sh (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-shared-parent.sh (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-shared-parent.service (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard.service (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.service (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.timer (new)
os-apply: BUNDLE WROTE /etc/felhom/crash-guard.conf (new)
os-apply: BUNDLE WROTE /etc/grub.d/01_felhom_oneshot (new)
os-apply: BUNDLE WROTE /etc/grub.d/42_felhom_oneshot (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent-rollback.service (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf (new)
os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mgmt-watchdog (new)
os-apply: BUNDLE WROTE /etc/tmpfiles.d/felhom-privsep.conf (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.service (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.timer (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-sshd.service (new)
os-apply: BUNDLE WROTE /etc/felhom-oob.nft (new)
os-apply: BUNDLE WROTE /etc/systemd/system/felhom-oob-nft.service (new)
os-apply: BUNDLE WROTE /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf (new)
os-apply: BUNDLE WROTE /etc/sudoers.d/felhom-op (new)
os-apply: BUNDLE WROTE /etc/sudoers.d/felhom-agent (new)
os-apply: BUNDLE DONE agent=0.175.1 written=31 same=1 self-check=ok signers-created=False
[OK] installed every root-owned file from config bundle v0.175.1 (sha 07ebf86362ed9d44…; checked, previous copies kept)
[OK] OOB felhom-sshd instance + static belt from the bundle (agent renders config + fills sets once oob.enabled)
[STEP] 5b/8 internal data place (Belső tárhely)
[INFO] no internal place: only -79 GiB free for it (pool 19, reserve 16, promised 0, planned 82; needs 20)
[STEP] 6/8 agent config + service
[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive.
[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard.
[INFO] Proxmox trust: the box's own root CA (/etc/felhom-agent/pve-root-ca.pem; a handshake to 127.0.0.1:8006 verified with it alone) — no pinned certificate
[INFO] creating island bridge vmbr9 (portless, 169.254.253.1/30)
[OK] vmbr9 up: 169.254.253.1/30
[INFO] R-50 island ON: local_api=169.254.253.1:8443 (vmbr9); guest net1=169.254.253.2/30; lan_resolver.host_ip=192.168.0.128
[INFO] node=goldenbox local_api=169.254.253.1:8443 tls_ca=/etc/felhom-agent/pve-root-ca.pem tls_fp=
[OK] wrote /etc/felhom-agent/agent.json (0600 felhom-agent)
[OK] agent --selftest (read-only) passed
[OK] felhom-agent service active (non-root felhom-agent reads the config OK)
[OK] wrote /etc/felhom/os-trust.json + operator-signers (felhom-op-1) — the root-owned slow-lane anchors
[STEP] 7/8 golden archive
[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged)
[INFO] fetching golden v0.353.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_11_15-11_46_33.tar.zst
[OK] verified sha256 86b5ba20a15ebd51… matches the hub manifest
[OK] golden imported + verified: local:backup/vzdump-lxc-9100-2026_11_15-11_46_33.tar.zst
[STEP] 8/8 provision guest 9201
[SKIP] pool felhom already exists
=== felhom-agent 0.175.1 selftest=provision (vmid=9201 customer=Tester4 hostname=tester4) ===
--- front half: bring-up (provision) local:backup/vzdump-lxc-9100-2026_11_15-11_46_33.tar.zst → vmid 9201 ---
time=2026-11-15T11:47:36.724+01:00 level=INFO msg="bring-up: pool membership re-asserted" vmid=9201 pool=felhom
[OK] front half: vmid 9201 up (boot+running) in 54s; MAC=BC:24:11:A3:8A:3E
--- back half: mint per-guest token + populate bootstrap config mount ---
time=2026-11-15T11:47:41.612+01:00 level=INFO msg="provision: back-half complete" vmid=9201 mount=mp9 guest_path=/etc/felhom-bootstrap endpoint=169.254.253.1:8443
[OK] back half: bootstrap mount mp9 → /etc/felhom-bootstrap on vmid 9201 (host dir /var/lib/felhom-agent/guests/9201/bootstrap)
local-api endpoint 169.254.253.1:8443 · leaf fp a090c1586500cc687b10bad077b625d33eafbcdb2311303e6efb684bf771e487 · token: minted (not printed)
=== selftest=provision OK — guest 9201 provisioned + bootstrap-mounted (KEPT) ===
next: reboot the guest → the golden's baked controller-bootstrap unit deploys the controller,
which PULLS its controller.yaml from the hub (retrieval passphrase) and merges in this local_api.
[OK] provision completed
[INFO] rebooting guest 9201 so the baked controller-bootstrap unit picks up the mount
[STEP] verify
[OK] pct status: running
[OK] onboot: 1
mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=50G
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
rootfs: local-lvm:vm-9201-disk-0,size=32G
[OK] pool: guest 9201 is a member of felhom
[OK] acl: FelhomAgentBase@/ present (user+token)
[OK] acl: FelhomAgentGuest@/pool/felhom present (user+token)
[OK] acl: FelhomAgentStore@/storage/local present (user+token)
[OK] acl: FelhomAgentStore@/storage/local-lvm present (user+token)
[OK] acl: FelhomAgentStore@/storage/felhom-pbs present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990000 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990001 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990002 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990003 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990004 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990005 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990006 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990007 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990008 present (user+token)
[OK] acl: FelhomAgentGuest@/vms/990009 present (user+token)
[OK] authz signers: 2 (operator-signed self-update armed)
[OK] controller: Up 20 seconds (healthy) (after ~0s)
[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.353.0
[WARN] cloudflared not visible yet
[INFO] (confirm in the hub UI that host Tester4-2f0c25 reports guest 9201)
[OK] Day-0 provision SUCCESS — vmid=9201 host_id=Tester4-2f0c25 customer=Tester4 golden=local:backup/vzdump-lxc-9100-2026_11_15-11_46_33.tar.zst
[INFO] root@pam was rotated + vaulted at step 4b — retrieve at hub → host page (the old GUI password no longer works).
@@ -0,0 +1 @@
Install pressed 2026-11-15T10:41:12Z
Binary file not shown.

After

Width:  |  Height:  |  Size: 127 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 111 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 124 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 136 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 112 KiB

@@ -0,0 +1,9 @@
installer: the UNTAGGED scripts/felhom-host-install.sh of felhom.eu commit 6174944f (SCRIPT_VERSION 1.37.0), copied in from DooPlex
93a021db17a98d1e5a31e1497481b41294319aa79ffecdb7038c697339f26e65 /root/hi-orig.sh
197:SCRIPT_VERSION="1.37.0" # the SINGLE version source (F-1): -h and the run banners follow it.
line: 479
479a480
> ART_GOLDEN_VER=0.353.0; ART_GOLDEN_SHA=86b5ba20a15ebd511db8828f51d38fb984f68efa23c4d4e89ae33aa195f19971
syntax rc=0
2026-11-15T10:45:22Z
Running as unit: install1115.service; invocation ID: 8e1d7feeaa404e579f82f6f720010201
@@ -0,0 +1,8 @@
## teardown 2026-11-15T10:59:21Z
purging VM 9386 from related configurations..
Configuration file 'nodes/demo-hp/qemu-server/9386.conf' does not exist
ls: cannot access '/mnt/hdd_1/images/9386': No such file or directory
0
341 night1004-tester1 running 8192 200.00 1950
the installer ISO removed from demo-hp (it was not there before this run)
left for the operator: hub host Tester4-2f0c25; appliance with smbios uuid ab0b8310-d5f3-4042-9e7d-0a0db2135433 (if it is on the Hosts page)
@@ -0,0 +1,15 @@
#### goldenbox 2026-11-15 11:48:30 CET agent 0.175.1 uptime-of-unit Sun 2026-11-15 11:46:29 CET
config proxmox.tls: ca_file='/etc/felhom-agent/pve-root-ca.pem' fingerprint=EMPTY insecure_skip_verify=False
-rw-r--r-- root:root 2094 bytes 2026-11-15 11:46:28.427650314 +0100 /etc/felhom-agent/pve-root-ca.pem
drwxr-xr-x root:root 4096 bytes 2026-11-15 11:46:28.931455184 +0100 /etc/felhom-agent
(absent) /var/lib/felhom-agent/agent.json.pre-pve-trust
(absent) /var/lib/felhom-agent/agent.json.write-in-flight
CA copy /etc/felhom-agent/pve-root-ca.pem: 1 certificate(s); first: subject=CN=Proxmox Virtual Environment, OU=98ce4c95-ac67-4896-9bcd-9de6357f45b5, O=PVE Cluster Manager CA notAfter=Nov 11 10:44:09 2036 GMT sha256 Fingerprint=55:84:62:1D:FE:C2:BB:F8:38:2A:35:EA:9A:53:0A:3A:6D:FF:37:4E:E3:63:68:6D:FD:FA:9E:0D:B6:60:A3:DB
same as /etc/pve/pve-root-ca.pem: yes
agent user can read it: yes; others can read: r
served certificate (127.0.0.1:8006): sha256 Fingerprint=0A:5E:F8:24:99:27:DC:CD:06:2D:52:D9:AD:59:00:AC:B3:7A:94:51:7B:CC:3D:8C:7C:69:DA:48:7A:EB:B3:65 notAfter=Nov 13 10:44:09 2028 GMT
/etc/pve/local/pve-ssl.pem: sha256 Fingerprint=0A:5E:F8:24:99:27:DC:CD:06:2D:52:D9:AD:59:00:AC:B3:7A:94:51:7B:CC:3D:8C:7C:69:DA:48:7A:EB:B3:65 notBefore=Nov 14 10:44:09 2026 GMT
## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min)
2026-11-15T11:46:31+01:00 goldenbox felhom-agent[2912]: time=2026-11-15T11:46:31.044+01:00 level=INFO msg="shared-parent: boot-persistence unit enabled (files from the config bundle)"
## one control: the agent's newest successful host report and newest Proxmox call error
#### end
+1 -1
View File
@@ -1,4 +1,4 @@
## v0.182.0 (2026-11-15) — release 3 „the hub tells you”: eight silent problems reach the operator; the kernel clean-up and the profile refresh are shown; an off-site account's record outlives an untick (D71); a deleted box or household leaves no rows (D73) (image `sha256:4f8bf649ef85db56a727e77327f5ba7ebcceb739a453c81c5423517882ae4681` built from `35c5b7ce`, CI run 2673)
## v0.182.0 (2026-11-15) — release 3 „the hub tells you”: eight silent problems reach the operator; the kernel clean-up and the profile refresh are shown; an off-site account's record outlives an untick (D71); a deleted box or household leaves no rows (D73) (image `sha256:4f8bf649ef85db56a727e77327f5ba7ebcceb739a453c81c5423517882ae4681` built from `35c5b7ce`, CI run 2673) — DEPLOYED 2026-11-15 09:48 CET on the operator's yes (manifest `3fdfab09`, CI run 2674); the first sweep deleted 26 rows
**Read before the deploy:** the daily prune now runs `store.PurgeOrphanRows` — at its first run it deletes the
left-over rows of boxes and households that were deleted earlier (counted before the deploy on a copy of the hub's