diff --git a/CONTEXT.md b/CONTEXT.md index 0fd7cef1..1b325293 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -16,6 +16,20 @@ > and holds nothing of its own; this file does hold its own content, namely the standing rulings below. +> **2026-11-15 (attended release run) — releases 2b and 3 are RELEASED; the note below is history.** +> Report: `REPORT-release-2b-3-2026-11-15.md`. Hub 0.182.0 (deployed 09:48 CET on the operator's yes), agent 0.175.1 +> (vouched 11:07:35; 0.175.0 exists and is NOT vouched), controller 0.353.0 (global floor 10:26:53 UTC, MinAgent +> 0.158.1), golden 0.353.0 (vouched 11:52:54), installer 1.37.0 (tag `installer-v1.37.0`, published). Proven on +> boxes: the move to the CA copy on Tester 1, demo-hp and the N100; two forced certificate renewals on demo-hp with the +> agent in control; the clock alarm raised and cleared; no app recreate at a clock step with the new controller AND +> agent; a real restart still read as a new boot. **Two things a session must know:** (1) `agent.json` is written only +> through `config.ReplaceInPlace` (journal `agent.json.write-in-flight` in `/var/lib/felhom-agent`; the daemon repairs +> a cut write before it reads the config; a selftest never repairs; a test package sets `config.CutWriteDir`). +> (2) An agent that is newer than the box's root files gets the wrapper's R11 „unknown mode”: on a pinned box that is +> NOT a trust problem, but only when no call failed (`pvetrust.rootStepTooOld`, `isTrustCause`). A wrong CA copy +> heals at the agent's start, and a blind box sends no report — so the trust alarm cannot be staged from a wrong +> copy (R-1396 gap 2 stays open). Rows closed: R-1380, R-1395, R-1050; opened: R-1397. + > **2026-11-14 night → 11-15 morning (unattended) — releases 2b and 3 are BUILT IN SOURCE; nothing is released.** > Report: `REPORT-night-build-2026-11-14.md`. Heads: `felhom.eu` `d2683b28`, `felhom-agent` `54ac2c2`, > `felhom-controller` `ee3b05e`; the installer is not tagged. Live versions did not move (hub 0.181.0, agent 0.174.0 diff --git a/REPORT-release-2b-3-2026-11-15.md b/REPORT-release-2b-3-2026-11-15.md new file mode 100644 index 00000000..257400a4 --- /dev/null +++ b/REPORT-release-2b-3-2026-11-15.md @@ -0,0 +1,120 @@ +# Release run 2026-11-15: releases 2b and 3 — hub, agent, controller, golden, installer + +Source: `REPORT-night-build-2026-11-14.md` §2. Evidence: `documentation/audits/release-2b-3-2026-11-15/`, +`documentation/tests/golden-0.353.0-2026-11-15/`. Three stops were asked and answered (hub deploy: yes; agent vouch: +yes; golden vouch and installer tag: yes, yes). One more question was asked: to delete the old test host before the +fresh install (yes). + +## 1. Claims in the prompt that turned out wrong + +1. **„Make the CA copy wrong … read the operator alarm on the hub.”** No alarm can come from that. A wrong copy is + repaired by the agent at its start (seen on demo-hp). And a box whose trust does not work sends no report, so the + hub cannot name the cause (the open gap, R-1396 part 2). The alarm that names a cause WAS seen, from a real event + on Tester 1 (below). +2. **„The CA copy … readable by the agent only as designed.”** The design is mode 0644, owner root: a CA certificate + is public (`03-host-agent.md` §12a). Read so on all three boxes and on the fresh box. +3. **„You will get one test mail.”** There were two. The second was a false trust alarm from Tester 1 (below). +4. **„Scratch 9202 … the controller rc”, then the floor moves every box.** The scratch guest does not report to the + hub, so no floor moves it. It was put on 0.353.0 by hand. + +The three claims the reviewer asked to try: + +- „The hub's sweep deletes only rows whose owner is gone” — **true**. Read in source; on a copy of the database a + control query found 0 rows to delete whose owner exists. The live sweep deleted the same 26 rows as the preview. +- „A host clock can be staged wrong on Tester 1 and put right with no lasting effect” — **true, with two notes.** + With the old controller each step recreated one app (expected). Log lines of those minutes carry a time 150 s + behind. After each run the offset to DooPlex was +0.07 s. +- „The untagged installer can be used for one fresh install” — **true, with two hand steps**: the file was copied + in, and one line was added to name the not-yet-vouched golden. + +## 2. Six lines + +1. **The hub is live** (0.182.0, 09:48 CET). Its first sweep deleted 26 old update reports of 13 boxes that were + deleted earlier, and no household row. +2. **The three boxes moved to the CA copy: Tester 1 yes, demo-hp yes, the N100 yes** (the N100 by itself). +3. **The agent kept control through two new certificates on demo-hp**: the same process, no restart, no alarm. +4. **The clock alarm was seen and cleared; with the new controller no app was recreated.** A real restart of + Tester 1 is still read as a new boot. +5. **Every box is on controller 0.353.0** with the one-year header (26 of 26 public names). +6. **The golden is vouched: yes. The installer is tagged and published: yes.** + +## 3. What was found and fixed on the way + +| What | Where it was found | Fix | +|---|---|---| +| A power cut in the middle of the agent's config write left an empty file; the agent then does not start, and the hub cannot repair that box | Part 0, reading the write path | The old file goes into a journal first; the agent repairs the config at start. Tests cut the write at four steps; 9 mutations seen red (3 redone, said in the evidence). **Not proven with a real power cut** | +| Between the agent update and the root-files update a box said „trust problem”, and the hub mailed the operator. False: the pin worked | Tester 1, 10:04, with agent 0.175.0 | Agent 0.175.1: older root files are not a trust problem on a pinned box. The security review of that fix found a hole (the quiet path could hide a failed pin); narrowed, 7 mutations seen red (3 redone). Seen on demo-hp and the N100: no alarm | + +Agent 0.175.0 was released, reached Tester 1 only, and is **not** vouched. Two agent releases in one session is one +more than the rule says; the second replaced the first before the vouch. + +## 4. Versions, digests, CI by commit + +| Part | Version | Digest | Built from | CI run (by commit) | Live since (CET) | +|---|---|---|---|---|---| +| Hub | 0.182.0 | image `sha256:4f8bf649ef85db56a727e77327f5ba7ebcceb739a453c81c5423517882ae4681` | `35c5b7ce` | 2673 success; the manifest commit `3fdfab09` 2674 success, read before the deploy | 09:48:59 | +| Agent | 0.175.1 | binary sha256 `38b1e2350de90a7940c16f8cd2a21d83fcd535410d1017a7df0fee14f0904612`; bundle `07ebf86362ed9d44776416bd5c2c4b6b629e44d34595a8120fd8fc3f4d9277ff` | `88d5d56` | 2678 success | vouched 11:07:35 | +| (Agent, not vouched) | 0.175.0 | binary `d1a30ed5…4b19a`; bundle `3b4a8b78…0d95` | `db01ae0` | 2671 success | Tester 1 only, 10:04–10:34 | +| Controller | 0.353.0, MinAgent 0.158.1 (not moved: the boot key is optional) | image `sha256:61cf2a8bfb82e02694bf85cca774bce2cbb9f947d395f91af7640aa5a5d944f0` | `ae07ed4` | 2681 success | per-box floors 11:08; global floor 11:26:53 | +| Golden | 0.353.0 | sha256 `86b5ba20a15ebd511db8828f51d38fb984f68efa23c4d4e89ae33aa195f19971` | controller 0.353.0 | — | vouched 11:52:54 | +| Installer | 1.37.0, tag `installer-v1.37.0` | file sha256 `93a021db17a98d1e5a31e1497481b41294319aa79ffecdb7038c697339f26e65` | `6174944f` | 2683 success; the pointer commit `f561c85b` 2685 success | published 12:02; read back from the public name with the same sha256 | + +Does the controller wait for the agent's vouch? No: it works with the old agent. The clock-step fix needs both. + +## 5. The register + +250 rows before; **248 after**; opened 1 (one Proxmox call answers 500 during a fresh install and the agent logs no +error, R-1397); closed 3 (R-1380, R-1395, R-1050). Narrowed with what is live and what is tests only: R-1384, R-1383, +R-1375, R-1376, R-1379, R-1396. +`unproven.py --summary`: 32 walked, 15 partial, 4 built, 4 missing, 23 not walked — no number moved. The capability +map: no row moved (no row's status word changes by today's proofs; the candidates are the rows on trust in Proxmox +and on alarms — to read at the next map pass). + +## 6. Machines changed and how each was left + +| Machine | What changed | Left as | +|---|---|---| +| DooPlex | hub 0.182.0 (manifest, hub only); image builds; the bake in the drill VM; the installer publish | drill VM reverted before use and powered off; both copies of the hub database shredded after the read | +| Tester 1 | agent 0.175.0 then 0.175.1 and their root files; controller 0.353.0; the clock set 150 s back twice and put right; one restart | healthy, 10 containers, clock +0.07 s, time service active; one app was recreated three times (twice by the old controller's rule, once by the real restart) | +| demo-hp | agent 0.175.1 and root files; controller 0.353.0; two new Proxmox certificates (the node's certificate is new — by design of the proof); the CA copy made wrong and repaired, identical to the saved copy; two agent restarts; the scratch guest 9202 on rc.1 then 0.353.0 | healthy, 26 containers; VM 9386 and the installer ISO removed | +| The N100 | nothing by hand. It took agent 0.175.1, its root files and controller 0.353.0 by the product's own path | healthy, 9 containers, moved to the CA copy at 10:36:58 | +| ep0, the off-site provider | nothing | — | + +Teardown, three layers: machine — VM 9386 destroyed with its disks; host — the ISO and the saved copy removed from +demo-hp; hub — **host `Tester4-2f0c25` and the appliance `ab0b8310-…` are left for the operator** (only he can +discard). The floors per box are cleared. + +## 7. Proven by tests only + +- The repair of a cut config write (no real power cut). +- A refused signed job told to the operator; a guest-power give-up told; a lost operator mail told; a skipped daily + check told. +- The off-site account path of D71 (no provider call was made). +- A warning 14 days before an end date (no date is that near). +- The clean-up detail sent by the NEW agent (no box removed a kernel after the update; the N100's cell shows the + night's removal, sent by the old agent). + +## 8. Observations + +- The false trust alarm at a box's update — FIXED in agent 0.175.1 (above). +- One Proxmox call answered 500 during the fresh install, with no ERROR line in the agent — FILED: R-1397. +- The hub's own alarm marks of a box deleted before the release are not swept (4 marks of one box) — on R-1376, + dated line. +- The scratch guest stayed on rc.1 under the global floor — NOT-A-FINDING: it does not report to the hub (R-1083). +- „Household cleared its mail address — deleted” in the hub's log at each controller start of Tester 1 — + NOT-A-FINDING: the same line is in the evidence of 2026-11-09 and 11-10; Tester 1 has no household address. +- The hub suppressed the second clock mail inside one hour — NOT-A-FINDING: the one-hour rule for one type. +- After a wrong CA copy, the right root is written first and the wrong one stays behind it until the next run — + NOT-A-FINDING: the design of the changeover (`03` §12a item 6); the next start dropped it. +- The fresh host's enrolment took 11 s at the hub („slow request”) — NOT-A-FINDING: the known first-enrolment cost + (key and tunnel setup); no failure. +- My own instruments were wrong three times and each was read again: a wait that printed nothing, a time computed + with `date -d … + 15 seconds`, a log search with the wrong word („orphan” for a line that says „Purged”). + +## 9. Documents read and changed + +Read: `03-host-agent.md` §12a; `05-hub-architecture.md` §14, §16.6; `08-alarm-ladder.md`; `11-os-updates.md` +§5.11–5.12; `09` §3 decisions 247, 249, 250, 303; `runbooks/publish-train-rules.md`; `RUNBOOK-manual-build.md` §4; +`audits/certpin-spike-2026-11-11/README.md`; the golden record of 2026-11-14. +Changed: `03-host-agent.md` §12a item 7 (the cut write). Instruction files changed: none. +System poster: no fact on the poster changed; no edit, no refresh asked. diff --git a/STATUS.md b/STATUS.md index 64795538..e7680f87 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-11-15 (morning).** The N100 removed 6 of its 8 old kernels by itself at night and is healthy: 1 of the 2 nights D82 asks for; demo-hp had nothing to remove (0 of 2). Releases 2b and 3 are built and tested in source; nothing is released (`REPORT-night-build-2026-11-14.md`). +**Updated 2026-11-15 (midday).** Releases 2b and 3 are out: hub 0.182.0, agent 0.175.1 (vouched), controller 0.353.0 on every box, golden 0.353.0 (vouched), installer 1.37.0 (published). A box now trusts Proxmox through its own root certificate, and the hub tells you about a wrong box clock and other silent problems (`REPORT-release-2b-3-2026-11-15.md`). ## What works - **The Plex attack did not reach Felhom's secrets** (R-1241). @@ -42,7 +42,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — | ## Waiting on you — do -**2026-11-15 morning.** 1. Read `REPORT-night-build-2026-11-14.md`, „Ready to release”: hub, agent, controller and installer wait for your yes. If not: nothing reaches a box. 2. Say „HSTS now” (R-1050). 3. Hosts page: delete host `Tester4-46836f` (else it alarms). 4. The report has a privacy sentence as a proposal; you publish it or not. **2026-11-11.** 2. Hosts page: discard `3fb5fed3…`. 3. Rotate login `kisfenyo`'s Gitea token into the store as `GITEA_GIT_TOKEN` (R-925); the push after `sec put` is in R-1390. 4. Home DNS: remove the `felhom.eu` override (R-1389). 5. Read the dates only you can read (R-1381). 6. Search Console read (R-1035, 11-18). +**2026-11-15 midday.** 1. Hosts page: discard host `Tester4-2f0c25` and appliance `ab0b8310…` (CC's test install; else it alarms). 2. Once: read each off-site sub-account's label against the hub's list (R-1375). 3. The privacy sentence in the night report: publish or not. **2026-11-11.** 2. Hosts page: discard `3fb5fed3…`. 3. Rotate login `kisfenyo`'s Gitea token into the store as `GITEA_GIT_TOKEN` (R-925); the push after `sec put` is in R-1390. 4. Home DNS: remove the `felhom.eu` override (R-1389). 5. Read the dates only you can read (R-1381). 6. Search Console read (R-1035, 11-18). **Still needed? (yes/no)** — answer per line. 1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). (since 10-16) diff --git a/documentation/audits/release-2b-3-2026-11-15/README.md b/documentation/audits/release-2b-3-2026-11-15/README.md new file mode 100644 index 00000000..236493b2 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/README.md @@ -0,0 +1,29 @@ +# Releases 2b and 3 — evidence (2026-11-15) + +Report: `REPORT-release-2b-3-2026-11-15.md` (repo root). Golden and fresh install: +`documentation/tests/golden-0.353.0-2026-11-15/`. The ids of the deleted boxes whose rows the sweep removed are NOT +here: `/mnt/5_hdd/felhom.eu/evidence/release-2b-3-2026-11-15/sweep-preview-ids.txt` (off git; an id can be a name). + +| File | What it shows | +|---|---| +| `part0/redproof-agent-cutwrite.txt` | the cut-write fix of the agent's config: 9 mutations, each seen red; 3 redone and one hollow test removed, said in the file | +| `part0/sweep-preview-counts.txt`, `sweep-preview.sh` | on a copy of the hub's own 08:00 snapshot: the first sweep deletes 26 of 504 `os_reports` rows, 13 owners, 0 in the five other tables; the control (a row to delete whose owner exists) is 0; the copy is deleted | +| `part1/hub-0182-start-log.txt` | hub 0.182.0's start: „Purged 26 row(s) … map[os_reports:26]”, the daily ledger's first lines, the lost-mail scan | +| `part1/hub-system-page-0950.txt` | the System page on 0.182.0 with the two new cells | +| `part2/t1-before.txt`, `t1-after-agent.txt`, `t1-moved.txt` | Tester 1: pinned → agent 0.175.0 (the wrapper refuses the unknown mode; the false trust problem) → root files → „MOVED … cleared AFTER a working call with the copy” | +| `part2/hp-moved.txt`, `n100-moved.txt` | demo-hp and the N100 with 0.175.1: no trust problem in the window, then MOVED | +| `part2/hp-cert-round1.txt`, `hp-cert-round2.txt` | two forced new Proxmox certificates on demo-hp; after each: the same agent process, 0 trust lines, 0 ERROR lines, 14 and 40 calls by the agent's token answered 200 in Proxmox's own access log | +| `part2/hp-ca-wrong.txt` | the CA copy replaced by another CA and the agent restarted: the right root is written first; after the next start the file is identical to the saved one | +| `part2/redproof-agent-root-step-too-old.txt` | the fix of the false trust alarm and its narrowing after the security review: 7 mutations seen red, 3 redone | +| `part2/t1-clock-A-*.txt` | the clock 150 s back with the OLD controller: one app recreated at the step back and again at the step forward; the kernel's boot id unchanged | +| `part2/system-1107.txt` | the „Security profiles” cell filled for the three boxes | +| `part3/boxwalk-0353.txt`, `watch-boxes.txt`, `watch-hub.txt` | the three boxes on controller 0.353.0 (image `61cf2a8b…`), 0 ERROR lines; 25 minutes after the global floor | +| `part3/https-only.txt`, `hsts-headers.txt` | the https-only check green; 26 of 26 public names answer exactly `max-age=31536000` | +| `part3/hub-webcerts.txt` | from a copy of the live hub database (shredded after): the three boxes' reports carry `web_certs`; the ledger's and the end-date baseline's marks | +| `part3/t1-clock-B-*.txt` | the same clock step with the NEW controller and agent: the same container ids before and after, in both directions | +| `part3/t1-after-reboot.txt` | one real restart of Tester 1: a new boot id, the boot pass recreates the drive-backed app | +| `part3/per-box-floors-cleared.txt`, `hub-system-page-after.txt` | the per-box floors cleared; „No per-customer floors”; global floor 0.353.0, vouched agent 0.175.1 | + +The hub's two mails of the day (both to the operator's address only, read in the test mailbox): `proxmox_trust_problem` +for tester-1 at 10:04 CET (false, fixed), `box_clock_wrong` for tester-1 at 11:05 CET („150 seconds behind”). The +second clock alarm at 11:20 was suppressed by the one-hour rule (the hub's log). diff --git a/documentation/audits/release-2b-3-2026-11-15/part0/sweep-preview-counts.txt b/documentation/audits/release-2b-3-2026-11-15/part0/sweep-preview-counts.txt new file mode 100644 index 00000000..f85ec5f0 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part0/sweep-preview-counts.txt @@ -0,0 +1,13 @@ +snapshot: /data/snapshots/hub-20261115T070000Z.db; hub pod clock: 2026-11-15T07:21:25Z; DooPlex: 2026-11-15T07:21:25Z +copy: 674902016 bytes +integrity: ok +freshness control: newest host report 2026-11-15 06:51:43 +hosts: 4 customer_configs: 5 + os_host_settings would delete 0 of 2 rows, 0 owner(s) + os_reports would delete 26 of 504 rows, 13 owner(s) + os_kernel_notices would delete 0 of 18 rows, 0 owner(s) + offsite_keys would delete 0 of 5 rows, 0 owner(s) + offsite_windows would delete 0 of 23 rows, 0 owner(s) + offsite_abandon_requests would delete 0 of 1 rows, 0 owner(s) +TOTAL rows: 26; deleted boxes named: 13; deleted households named: 0 (ids saved off git, not printed) +copy deleted: yes; ids file: /mnt/5_hdd/felhom.eu/evidence/release-2b-3-2026-11-15/sweep-preview-ids.txt (13 lines); rc=0 diff --git a/documentation/audits/release-2b-3-2026-11-15/part0/sweep-preview.sh b/documentation/audits/release-2b-3-2026-11-15/part0/sweep-preview.sh new file mode 100644 index 00000000..e6b2009c --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part0/sweep-preview.sh @@ -0,0 +1,50 @@ +#!/bin/bash +# Part 0.2: what the hub's first sweep (PurgeOrphanRows, D73) will delete. Reads a COPY of the hub's newest own +# snapshot (a VACUUM INTO file: whole, no WAL needed, no old free-space bytes). The copy is deleted at the end. +# The ids go OFF git (an id can be a name). +set -u +EV=/mnt/5_hdd/felhom.eu/evidence/release-2b-3-2026-11-15 +mkdir -p "$EV"; chmod 700 "$EV" +SNAP=$(sudo kubectl -n felhom-system exec deploy/hub -- sh -c 'ls -1 /data/snapshots/hub-*.db | tail -1') +echo "snapshot: $SNAP; hub pod clock: $(sudo kubectl -n felhom-system exec deploy/hub -- date -u +%FT%TZ); DooPlex: $(date -u +%FT%TZ)" +COPY="$EV/hubdb-copy.db" +sudo kubectl -n felhom-system exec deploy/hub -- cat "$SNAP" > "$COPY" +chmod 600 "$COPY" +echo "copy: $(stat -c %s "$COPY") bytes" +python3 -I - "$COPY" "$EV/sweep-preview-ids.txt" <<'PY' +import sqlite3, sys +db = sqlite3.connect("file:%s?mode=ro" % sys.argv[1], uri=True) +print("integrity:", db.execute("PRAGMA integrity_check").fetchone()[0]) +cols = [r[1] for r in db.execute("PRAGMA table_info(host_reports)")] +tcol = "received_at" if "received_at" in cols else cols[-1] +print("freshness control: newest host report", db.execute("SELECT MAX(%s) FROM host_reports" % tcol).fetchone()[0]) +print("hosts:", db.execute("SELECT COUNT(*) FROM hosts").fetchone()[0], " customer_configs:", db.execute("SELECT COUNT(*) FROM customer_configs").fetchone()[0]) +tables = [ + ("os_host_settings", "host_id", "host_id NOT IN (SELECT host_id FROM hosts)"), + ("os_reports", "host_id", "host_id NOT IN (SELECT host_id FROM hosts)"), + ("os_kernel_notices", "host_id", "host_id NOT IN (SELECT host_id FROM hosts)"), + ("offsite_keys", "customer_id", "customer_id NOT IN (SELECT customer_id FROM customer_configs) AND customer_id NOT IN (SELECT customer_id FROM hosts)"), + ("offsite_windows", "customer_id", "customer_id NOT IN (SELECT customer_id FROM customer_configs) AND customer_id NOT IN (SELECT customer_id FROM hosts)"), + ("offsite_abandon_requests", "customer_id", "customer_id NOT IN (SELECT customer_id FROM customer_configs) AND customer_id NOT IN (SELECT customer_id FROM hosts)"), +] +total = 0; boxes = set(); houses = set() +with open(sys.argv[2], "w") as out: + for t, col, where in tables: + all_n = db.execute("SELECT COUNT(*) FROM %s" % t).fetchone()[0] + rows = db.execute("SELECT %s, COUNT(*) FROM %s WHERE %s GROUP BY 1 ORDER BY 1" % (col, t, where)).fetchall() + n = sum(r[1] for r in rows); total += n + (boxes if col == "host_id" else houses).update(r[0] for r in rows) + print(" %-26s would delete %5d of %6d rows, %d owner(s)" % (t, n, all_n, len(rows))) + for owner, c in rows: + out.write("%s\t%s\t%s\t%d\n" % (t, col, owner, c)) + # the control: no row to delete may name an owner that exists + if col == "host_id": + bad = db.execute("SELECT COUNT(*) FROM %s WHERE %s AND host_id IN (SELECT host_id FROM hosts)" % (t, where)).fetchone()[0] + else: + bad = db.execute("SELECT COUNT(*) FROM %s WHERE %s AND (customer_id IN (SELECT customer_id FROM customer_configs) OR customer_id IN (SELECT customer_id FROM hosts))" % (t, where)).fetchone()[0] + assert bad == 0, (t, bad) +print("TOTAL rows: %d; deleted boxes named: %d; deleted households named: %d (ids saved off git, not printed)" % (total, len(boxes), len(houses))) +PY +rc=$? +shred -u "$COPY" 2>/dev/null || rm -f "$COPY" +echo "copy deleted: $([ -e "$COPY" ] && echo NO || echo yes); ids file: $EV/sweep-preview-ids.txt ($(wc -l < "$EV/sweep-preview-ids.txt") lines); rc=$rc" diff --git a/documentation/audits/release-2b-3-2026-11-15/part1/hub-0182-start-log.txt b/documentation/audits/release-2b-3-2026-11-15/part1/hub-0182-start-log.txt new file mode 100644 index 00000000..76802907 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part1/hub-0182-start-log.txt @@ -0,0 +1,43 @@ +2026/11/15 09:48:59 [INFO] felhom-hub 0.182.0 starting +2026/11/15 09:48:59 [INFO] off-site secrets sealed at rest (0 legacy plaintext row(s) sealed now) +2026/11/15 09:48:59 [INFO] console passwords sealed at rest (0 legacy plaintext row(s) sealed now) +2026/11/15 09:48:59 [INFO] box secrets sealed at rest (0 legacy plaintext value(s) sealed now) +2026/11/15 09:48:59 [INFO] config tokens sealed at rest (0 legacy plaintext token(s) sealed now) +2026/11/15 09:48:59 [INFO] Database opened at /data/hub.db +2026/11/15 09:48:59 [INFO] Default controller-version floor: 0.120.0 +2026/11/15 09:48:59 [INFO] Template fetcher started (every 1h) +2026/11/15 09:49:00 [INFO] Asset seed: all 263 files up-to-date +2026/11/15 09:49:00 [INFO] Asset manifest built: 288 files +2026/11/15 09:49:00 [INFO] App-email relay enabled (limit 30/min/customer, From domains [felhom.eu]) +2026/11/15 09:49:00 [INFO] Gitea artifact browser enabled (Day-0 version dropdowns) via http://gitea.gitea-system.svc.cluster.local:3000 +2026/11/15 09:49:00 [INFO] Offsite provisioning enabled (pool_box=611714, location=fsn1) +2026/11/15 09:49:00 [INFO] Offsite pool-box checker initialized: box=611714 fill warn=80% crit=90%, oversub warn=2.00x, unreachable after 3 consecutive failed reads, refresh 15m0s +2026/11/15 09:49:00 [INFO] Root files: bundles are compared by their files (R-1392) +2026/11/15 09:49:00 [INFO] osupdates: approval rule = every ring-0 box healthy for 24h0m0s and 1 night run(s) +2026/11/15 09:49:00 [INFO] osupdates: the Docker engine set is approved only by the operator, after 2 healthy ring-0 night(s) +2026/11/15 09:49:00 [INFO] osupdates: alarms = no OS leg 168h0m0s, reboot needed 336h0m0s, ring 0 stalled 168h0m0s, not covered 336h0m0s +2026/11/15 09:49:00 [INFO] Off-site key registrar enabled; daily key check at 07:10 Budapest +2026/11/15 09:49:00 [INFO] access log: /data/access-log, one file per UTC day, kept 90 days +2026/11/15 09:49:00 [INFO] Registry version checker started (every 6h) +2026/11/15 09:49:00 [INFO] WG peer-sync enabled (endpoint 167.233.158.164:22, user felhom-peersync) +2026/11/15 09:49:00 [INFO] PBS DR tenantsync enabled (endpoint 167.233.158.164:22, user felhom-peersync; WG-registration auto-provision hook armed) +2026/11/15 09:49:00 [INFO] PBS-DR box checker initialized: fill warn=80% crit=90%, unreachable after 3 consecutive failed reads, refresh 15m0s +2026/11/15 09:49:00 [INFO] agent-plane poke enabled (endpoint 167.233.158.164:22, user felhom-peersync; web + api admin seams armed) +2026/11/15 09:49:00 [INFO] PBS-DR self-heal reconciler started (interval 5m) +2026/11/15 09:49:00 [INFO] offsite credential self-heal reconciler started (interval 5m, debounce 2 reports) +2026/11/15 09:49:00 [INFO] daily ledger: offsite-key-audit is watched from now (no earlier record) +2026/11/15 09:49:00 [INFO] offsite-key-audit: next run at 2026-11-16 07:10 CET (in 21h20m59s) +2026/11/15 09:49:01 [INFO] Pruned 86 old report rows +2026/11/15 09:49:01 [INFO] Pruned 86 old app telemetry rows +2026/11/15 09:49:04 [WARN] slow statement: 3091ms (over 1s): DELETE FROM app_log_issues WHERE last_seen < ? +2026/11/15 09:49:04 [INFO] Pruned 34 stale app issues +2026/11/15 09:49:05 [INFO] Purged 26 row(s) of boxes and households that no longer exist: map[os_reports:26] +2026/11/15 09:49:05 [INFO] prune: next run at 2026-11-16 04:30 CET (in 18h40m55s) +2026/11/15 09:49:05 [INFO] Staleness checker initialized: 3 ok, 0 stale, 2 down (node_stale after 45m0s, node_down after 1h30m0s) +2026/11/15 09:49:05 [INFO] Host staleness checker initialized: 3 ok, 1 already alarmed (seeded silently, R-1001), 0 stale, 0 down (left unseeded → first Check emits; host_stale after 45m0s, host_down after 1h30m0s) +2026/11/15 09:49:05 [INFO] Host capability checker initialized: 4 ok, 0 degraded, 0 with a refused whole-guest backup (degraded/refused left unseeded → first Check emits) +2026/11/15 09:49:05 [INFO] Host leaf checker initialized: 4 host fingerprint(s) seeded +2026/11/15 09:49:05 [INFO] Host disk checker initialized: warn=90% crit=95%, 4 ok seeded, 0 already-breached left unseeded (first Check emits) +2026/11/15 09:49:05 [INFO] Storage fill checker initialized: warn=90% crit=95%, 9 ok seeded, 0 already-breached left unseeded, 4 root-backed excluded +2026/11/15 09:49:05 [INFO] Host mgmt-plane checker initialized: 0 host heal-state(s) seeded +2026/11/15 09:49:05 [INFO] Host OOB checker initialized: 0 host(s) seeded degraded diff --git a/documentation/audits/release-2b-3-2026-11-15/part1/hub-system-page-0950.txt b/documentation/audits/release-2b-3-2026-11-15/part1/hub-system-page-0950.txt new file mode 100644 index 00000000..d28b67db --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part1/hub-system-page-0950.txt @@ -0,0 +1,256 @@ +System — Felhom Hub +Felhom Hub +Dashboard +Customers +Apps +Hosts +System +Offsite +Configuration +System — versions and OS updates +Needs attention +One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3). +[hover: worth a look] look +Tester4-46836f Tester4 +[hover: cannot do a one-shot boot: the box does not boot UEFI; /boot/efi is not a mounted vfat ESP (not mounted)] kernel step: — · the host needs a restart since 2026-11-14 · [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] root files behind the vouched agent's · [hover: 1 minor release behind — sign an agent_update for this box] agent behind: 0.173.0 → 0.174.0 (since 2026-11-14) · [hover: the box could not read these values (agent older than v0.142.0, or the guest is down) — never a guess] could not read: default kernel +Waiting for you +Update sets that only you approve: the kernel, the Docker engine and the Proxmox packages. Guest and host Debian updates approve themselves after 24 h and one night on the [hover: Ring 0 = the demo boxes. They install every new fix first; the other boxes (ring 1) install only what is approved.] ring-0 boxes. +Docker engine 29.9.3-1 +Ran on every ring-0 box — demo-felhom-8363b5: 2 healthy night(s), demo-hp-bb76ea: 2 healthy night(s). The memory-kill check passed. +first seen 2 days ago (2026-11-13 04:49 UTC). Approving installs nothing by itself: a ring-1 box takes it only through a signed operator job. +Approve Docker set +Proxmox packages 9.2.27 — still being tested: demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set +Boxes +Click a box to see every value it reports, and its ring and update switches. "unknown": the box could not read the value — never a guess. +Box Ring Health Controller · agent OS updates Kernel [hover: The newest OS update run] Last night +Tester4-46836f Tester4 +[hover: Ring 1: a normal box — it installs only approved releases] 1 +[hover: worth a look] look +0.352.1 [hover: 1 minor release behind — sign an agent_update for this box] agent 0.173.0 → 0.174.0 (since 2026-11-14) +[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind +7.0.2-6-pve +[hover: 22 h ago · applied · 93 s — last successful leg: 22 h ago] ok · 22 h ago +Ring and updates +Ring ring 1 +→ demo +OS updates updates ON +switch off +Tunnel [hover: no cloudflared container in the guest (agent state: absent)] none set up +Controller 0.352.1 +Agent [hover: 1 minor release behind — sign an agent_update for this box] 0.173.0 → 0.174.0 (since 2026-11-14) +Last OS leg [hover: last successful leg: 22 h ago] 22 h ago · applied · 93 s +Docker engine +Docker 29.9.2 +containerd 2.4.2-3~debian.13~trixie +live-restore on +Docker release — +Host +Proxmox 9.2.2 +Kernel (running) 7.0.2-6-pve +Kernel (next boot) 7.0.2-6-pve +[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown +[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: cannot do a one-shot boot: the box does not boot UEFI; /boot/efi is not a mounted vfat ESP (not mounted)] — +[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] — +[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: the box reports no security-profile refresh (an agent older than release 3, a BYO host, or no refresh yet)] — +Debian 13.7 +Felhom release os-host-20261111-034654 +Pending 100 +Not covered 0 +Held none +Reboot needed since 2026-11-14 +kernel.panic 10 s +Oops no +Crash restarts 24 h 0 +Crash guard armed +[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] 0.173.0 +Guest +Guest Debian 13.7 +Felhom release os-guest-20261111-034654 +Pending 6 +Restart needed 7 +[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 22 h ago · 78.4 GiB +[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] not yet +demo-felhom-8363b5 Demo Ügyfél +[hover: Ring 0: a demo box — it installs every new fix first] 0 +[hover: nothing amber or red] fine +0.352.1 [hover: current (vouched 0.174.0)] agent 0.174.0 +[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind +7.0.14-28-pve +[hover: 3 h ago · nothing · 23 s — last successful leg: 3 h ago] ok · 3 h ago +Ring and updates +Ring ring 0 +→ normal +OS updates updates ON +switch off +Tunnel running +Controller 0.352.1 +Agent [hover: current (vouched 0.174.0)] 0.174.0 +Last OS leg [hover: last successful leg: 3 h ago] 3 h ago · nothing · 23 s +Docker engine +Docker 29.9.3 +containerd 2.4.2-4~debian.13~trixie +live-restore on +Docker release ring0-20261115T045634Z +Host +Proxmox 9.2.27 +Kernel (running) 7.0.14-28-pve +Kernel (next boot) 7.0.14-28-pve +[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve +[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 1m14s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 7 days ago · due 7.0.14-29-pve — household told 49 min ago: TONIGHT · phase good +[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: removed 6 old kernel package(s): proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed; kept 7.0.14-26-pve, 7.0.14-28-pve; 6.3 GiB gained; 0 left-over path(s)] cleaned 3 h ago +[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: the box reports no security-profile refresh (an agent older than release 3, a BYO host, or no refresh yet)] — +Debian 13.7 +Felhom release ring0-20261115T045634Z +Pending 29 +Not covered 0 +Held none +Reboot needed no +kernel.panic 10 s +Oops no +Crash restarts 24 h 0 +Crash guard armed +[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.174.0 +Guest +Guest Debian 13.7 +Felhom release ring0-20261115T045634Z +Pending 0 +Restart needed 1 +[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 3 days ago · 272.7 GiB +[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 92.7 Mbit/s · 4 days ago +demo-hp-bb76ea Demo HP +[hover: Ring 0: a demo box — it installs every new fix first] 0 +[hover: nothing amber or red] fine +0.352.1 [hover: current (vouched 0.174.0)] agent 0.174.0 +[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind +7.0.14-28-pve +[hover: 4 h ago · nothing · 30 s — last successful leg: 4 h ago] ok · 4 h ago +Ring and updates +Ring ring 0 +→ normal +OS updates updates ON +switch off +Tunnel running +Controller 0.352.1 +Agent [hover: current (vouched 0.174.0)] 0.174.0 +Last OS leg [hover: last successful leg: 4 h ago] 4 h ago · nothing · 30 s +Docker engine +Docker 29.9.3 +containerd 2.4.2-4~debian.13~trixie +live-restore on +Docker release ring0-20261115T040248Z +Host +Proxmox 9.2.27 +Kernel (running) 7.0.14-28-pve +Kernel (next boot) 7.0.14-28-pve +[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve +[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 2m39s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 5 days ago · due 7.0.14-29-pve — household told 49 min ago: TONIGHT · phase good +[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] — +[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: the box reports no security-profile refresh (an agent older than release 3, a BYO host, or no refresh yet)] — +Debian 13.7 +Felhom release ring0-20261115T040248Z +Pending 30 +Not covered 0 +Held none +Reboot needed no +kernel.panic 10 s +Oops no +Crash restarts 24 h 0 +Crash guard armed +[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.174.0 +Guest +Guest Debian 13.7 +Felhom release ring0-20261115T040248Z +Pending 0 +Restart needed 1 +[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 3 days ago · 70.5 GiB +[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 94.3 Mbit/s · 4 days ago +tester-1-d70be4 Tester 1 +[hover: Ring 1: a normal box — it installs only approved releases] 1 +[hover: nothing amber or red] fine +0.352.1 [hover: current (vouched 0.174.0)] agent 0.174.0 +[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind +7.0.14-22-pve +[hover: 3 h ago · nothing · 16 s — last successful leg: 3 h ago] ok · 3 h ago +Ring and updates +Ring ring 1 +→ demo +OS updates updates ON +switch off +Tunnel running +Controller 0.352.1 +Agent [hover: current (vouched 0.174.0)] 0.174.0 +Last OS leg [hover: last successful leg: 3 h ago] 3 h ago · nothing · 16 s +Docker engine +Docker 29.8.2 +containerd 2.3.6-1~debian.13~trixie +live-restore on +Docker release — +Host +Proxmox 9.2.2 +Kernel (running) 7.0.14-22-pve +Kernel (next boot) 7.0.14-22-pve +[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-22-pve +[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: ring 1: no kernel staged by a signed os_kernel_step healthy 1m17s after the agent started; the new kernel is the default] 7.0.14-22-pve applied 38 days ago · phase good +[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] — +[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: the box reports no security-profile refresh (an agent older than release 3, a BYO host, or no refresh yet)] — +Debian 13.7 +Felhom release os-host-20261111-034654 +Pending 101 +Not covered 0 +Held none +Reboot needed no +kernel.panic 10 s +Oops no +Crash restarts 24 h 0 +Crash guard armed +[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.174.0 +Guest +Guest Debian 13.7 +Felhom release os-guest-20261111-034654 +Pending 6 +Restart needed 0 +[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 3 days ago · 90.4 GiB +[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 93.0 Mbit/s · 4 days ago +Details +Release ids, cancelled approvals, what ring 0 runs, the version floors, and every box's crash guard and root files. +Approved releases +The newest approved set of each layer. A [hover: Approved while a TEST wait override was active. It is cancelled when the hub starts without the override (`11` §5.3.1).] TEST approval is amber. +guest : os-guest-20261111-034654 +272 packages · 2026-11-11 03:46 UTC · by auto +host : os-host-20261115-045718 +607 packages · 2026-11-15 04:57 UTC · by auto +docker : os-docker-20261026-190351 +6 packages · 2026-10-26 19:03 UTC · by operator +pve : os-pve-20261114-075712 +121 packages · 2026-11-14 07:57 UTC · by operator +kernel : os-kernel-20261114-075716 +2 packages · 2026-11-14 07:57 UTC · by operator +Approve now (guest + host) +An urgent fix only — normally the hub approves after 24 h and one night. +What ring 0 runs now +guest : +272 packages, first seen 2026-11-10 01:49 UTC +approved as os-guest-20261111-034654 +host : +607 packages, first seen 2026-11-14 04:52 UTC +approved as os-host-20261115-045718 +docker : +6 packages, first seen 2026-11-13 04:49 UTC +ready — see "Waiting for you" +pve : +121 packages, first seen 2026-11-15 04:58 UTC +demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set +kernel : +2 packages, first seen 2026-11-10 01:57 UTC +approved as os-kernel-20261114-075716 +Version floors +Global controller floor: 0.352.1 · vouched agent: 0.174.0 +[hover: A floor is the lowest controller version a box must run; the hub moves a box below it up.] What is a floor? · [hover: The agent version the operator checked and signed off for the fleet. Agents update only by a per-box signed job.] What is a vouched agent? +No per-customer floors: every box follows the global floor. +Crash guard and root files +Box ¦ kernel.panic ¦ Oops ¦ Crash restarts 24 h ¦ Crash guard ¦ [hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files ¦ +Tester4-46836f ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] 0.173.0 ¦ +demo-felhom-8363b5 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.174.0 ¦ +demo-hp-bb76ea ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.174.0 ¦ +tester-1-d70be4 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.174.0 ¦ +Felhom Hub 0.182.0 diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/after-cert.sh b/documentation/audits/release-2b-3-2026-11-15/part2/after-cert.sh new file mode 100644 index 00000000..29942baf --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/after-cert.sh @@ -0,0 +1,15 @@ +# READS ONLY, on a box: what the agent did since a time ($1 = 'YYYY-MM-DD HH:MM:SS'). +# The control is Proxmox's OWN access log (another channel than the agent's journal): calls by the agent's token +# that Proxmox answered, after the time. +set -u +T0="$1"; HM=$(date -d "$T0" +%H:%M:%S); D=$(date -d "$T0" +%d/%m/%Y) +echo "#### $(hostname) $(date +%F\ %T): since $T0" +echo "agent unit active since $(systemctl show felhom-agent -p ActiveEnterTimestamp --value), pid $(systemctl show felhom-agent -p MainPID --value), restarts $(systemctl show felhom-agent -p NRestarts --value)" +J=$(journalctl -u felhom-agent --since "$T0" --no-pager -o short-iso 2>&1 | grep -v "sudo\[") +echo "agent journal lines since: $(echo "$J" | wc -l)" +echo "trust / certificate lines: $(echo "$J" | grep -ciE "pvetrust|x509|certificate|tls: ")"; echo "$J" | grep -iE "pvetrust|x509|certificate|tls: " | cut -c1-360 | tail -8 +echo "ERROR lines: $(echo "$J" | grep -c "level=ERROR")"; echo "$J" | grep "level=ERROR" | cut -c1-300 | tail -5 +echo "Proxmox access log, the agent's token, after $HM:" +awk -v d="$D" -v hm="$HM" '$3 ~ /felhom-agent@pve/ { split($4,a,":"); day=substr(a[1],2); t=a[2]":"a[3]":"a[4]; if (day==d && t>=hm) { n++; c[$9]++; if (!first) first=$0; last=$0 } } END { printf " calls: %d; by status:", n; for (k in c) printf " %s=%d", k, c[k]; print ""; print " first: " substr(first,1,170); print " last: " substr(last,1,170) }' /var/log/pveproxy/access.log +echo "served now: $(echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256 2>/dev/null | cut -d= -f2)" +echo "#### end" diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/ca-wrong.sh b/documentation/audits/release-2b-3-2026-11-15/part2/ca-wrong.sh new file mode 100644 index 00000000..8c7178b4 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/ca-wrong.sh @@ -0,0 +1,23 @@ +# Part 2 step 4, demo-hp ONLY: make the agent's CA copy wrong (another, self-made CA in its place), restart the agent, +# and watch. The copy is saved first and compared at the end. Changes: /etc/felhom-agent/pve-root-ca.pem (one file), +# one restart of felhom-agent. Nothing under /etc/pve is touched. +set -u +[ "$(hostname)" = demo-hp ] || { echo "REFUSED: this is not demo-hp"; exit 2; } +busy=$(ps -eo comm | grep -cE "^(vzdump|apt-get|dpkg|pct|proxmox-backup-)") +[ "$busy" = 0 ] || { echo "REFUSED: something is in flight"; exit 2; } +C=/etc/felhom-agent/pve-root-ca.pem; SAVE=/root/pve-root-ca.pem.saved-part2 +W=$(mktemp -d) +cp -p "$C" "$SAVE" +echo "#### $(date +%F\ %T) the copy before: sha256 $(sha256sum < "$C" | cut -c1-16), $(stat -c '%A %U:%G' "$C"); saved at $SAVE" +openssl req -x509 -newkey rsa:2048 -nodes -days 3650 -subj "/CN=CC wrong CA for a test 2026-11-15" -addext "basicConstraints=critical,CA:TRUE" -keyout "$W/k.pem" -out "$W/wrong.pem" 2>/dev/null +cat "$W/wrong.pem" > "$C" +rm -rf "$W" +echo "the copy staged wrong: sha256 $(sha256sum < "$C" | cut -c1-16), subject $(openssl x509 -in "$C" -noout -subject)" +T0=$(date '+%F %T') +systemctl restart felhom-agent; echo "agent restart rc=$? at $T0" +for i in $(seq 1 18); do sleep 5; [ "$(sha256sum < "$C")" = "$(sha256sum < "$SAVE")" ] && break; done +echo "the copy $([ "$(sha256sum < "$C")" = "$(sha256sum < "$SAVE")" ] && echo "is the saved one again, by itself, after about $((i*5)) s" || echo "is STILL WRONG after 90 s")" +sleep 20 +journalctl -u felhom-agent --since "$T0" --no-pager -o short-iso | grep -v "sudo\[" | grep -iE "pvetrust|x509|certificate|level=ERROR" | cut -c1-380 | tail -10 +echo "T0=$T0" +echo "#### end" diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/cert-renew.sh b/documentation/audits/release-2b-3-2026-11-15/part2/cert-renew.sh new file mode 100644 index 00000000..f28efd8c --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/cert-renew.sh @@ -0,0 +1,19 @@ +# Part 2 step 3, on demo-hp ONLY: force a new Proxmox certificate the way the spike did (certpin-spike-2026-11-11), +# then watch the agent. Changes on the box: /etc/pve/local/pve-ssl.pem (+ key) is made new; pveproxy is reloaded. +set -u +[ "$(hostname)" = demo-hp ] || { echo "REFUSED: this is not demo-hp"; exit 2; } +busy=$(ps -eo comm | grep -cE "^(vzdump|apt-get|dpkg|pct|proxmox-backup-)" ) +[ "$busy" = 0 ] || { echo "REFUSED: something is in flight ($busy)"; ps -eo comm | grep -E "^(vzdump|apt-get|dpkg|pct|proxmox-backup-)"; exit 2; } +served() { echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256 2>/dev/null | cut -d= -f2; } +echo "#### $(date +%F\ %T) round ${1:-1}: agent $(felhom-agent --version | awk '{print $2}'), unit active since $(systemctl show felhom-agent -p ActiveEnterTimestamp --value), pid $(systemctl show felhom-agent -p MainPID --value)" +echo "root CA before: $(openssl x509 -in /etc/pve/pve-root-ca.pem -noout -fingerprint -sha256 | cut -d= -f2)" +echo "served before: $(served)" +T0=$(date '+%F %T') +pvecm updatecerts --force 2>&1 | tail -3 +echo "file after updatecerts: $(openssl x509 -in /etc/pve/local/pve-ssl.pem -noout -fingerprint -sha256 | cut -d= -f2)" +systemctl reload pveproxy; echo "pveproxy reload rc=$?" +for i in $(seq 1 12); do sleep 5; s=$(served); [ "$s" = "$(openssl x509 -in /etc/pve/local/pve-ssl.pem -noout -fingerprint -sha256 | cut -d= -f2)" ] && break; done +echo "served after: $s (after about $((i*5)) s)" +echo "root CA after: $(openssl x509 -in /etc/pve/pve-root-ca.pem -noout -fingerprint -sha256 | cut -d= -f2)" +echo "T0=$T0" +echo "#### end round" diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/clock.sh b/documentation/audits/release-2b-3-2026-11-15/part2/clock.sh new file mode 100644 index 00000000..f08a5928 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/clock.sh @@ -0,0 +1,31 @@ +# Tester 1 ONLY. clock.sh snap|back|right +# snap — READ: the clock, the time service, every container's id and creation time in guest 9201. +# back — stop the time service and set the clock 150 s BACK (the smaller harm: putting it right is then a +# step FORWARD, the direction a time service steps a late clock; nothing gets a stamp from the future +# while the clock is wrong, and no certificate or signed job ends early). +# right — start the time service and step the clock right. +set -u +grep -q '"tester-1' /etc/felhom-agent/agent.json 2>/dev/null || grep -q 'tester-1' /etc/felhom-agent/agent.json || { echo "REFUSED: this box is not Tester 1"; exit 2; } +snap() { + echo "## $(hostname) clock $(date '+%F %T.%N %Z' | cut -c1-27) epoch $(date +%s.%N | cut -c1-14); chrony $(systemctl is-active chrony); $(chronyc tracking 2>/dev/null | grep -E 'System time|Leap' | tr -s ' ' | tr '\n' ';')" + echo "## kernel boot_id $(cat /proc/sys/kernel/random/boot_id); btime $(awk '/^btime/ {print $2}' /proc/stat); uptime $(cut -d' ' -f1 /proc/uptime) s" + echo "## containers in 9201 (name id created):" + pct exec 9201 -- docker ps -a --format '{{.Names}} {{.ID}} {{.CreatedAt}} [{{.Status}}]' | sort | sed 's/^/ /' + echo "## count $(pct exec 9201 -- docker ps -q | wc -l) running" +} +case "${1:-}" in + snap) snap ;; + back) + busy=$(ps -eo comm | grep -cE "^(vzdump|apt-get|dpkg|proxmox-backup-)") + [ "$busy" = 0 ] || { echo "REFUSED: something is in flight"; exit 2; } + snap + systemctl stop chrony; echo "chrony stopped rc=$?" + date -s "@$(( $(date +%s) - 150 ))" >/dev/null; echo "clock set 150 s back rc=$? → $(date '+%F %T')" + ;; + right) + systemctl start chrony; echo "chrony started rc=$?" + sleep 8; chronyc makestep; sleep 6 + echo "clock now $(date '+%F %T'); $(chronyc tracking | grep -E 'System time|Leap' | tr -s ' ' | tr '\n' ';')" + ;; + *) echo "usage: snap|back|right"; exit 2 ;; +esac diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/hp-ca-wrong.txt b/documentation/audits/release-2b-3-2026-11-15/part2/hp-ca-wrong.txt new file mode 100644 index 00000000..6a4a241d --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/hp-ca-wrong.txt @@ -0,0 +1,10 @@ +#### 2026-11-15 10:38:01 the copy before: sha256 eb0ce31aaa8d94ee, -rw-r--r-- root:root; saved at /root/pve-root-ca.pem.saved-part2 +the copy staged wrong: sha256 47ea4b8f5ea6a13b, subject subject=CN=CC wrong CA for a test 2026-11-15 +agent restart rc=0 at 2026-11-15 10:38:02 +the copy is STILL WRONG after 90 s +2026-11-15T10:38:02+01:00 demo-hp felhom-agent[2251809]: time=2026-11-15T10:38:02.288+01:00 level=WARN msg="pvetrust: the CA copy was refreshed and proven — the running client uses it now" why=start mode=ca outcome=written +T0=2026-11-15 10:38:02 +#### end +after a second start: certs 1; identical to the saved copy: yes; mode -rw-r--r-- root:root +2026-11-15T10:40:06+01:00 demo-hp felhom-agent[2259683]: time=2026-11-15T10:40:06.827+01:00 level=WARN msg="pvetrust: the CA copy was refreshed and proven — the running client uses it now" why=start mode=ca outcome=written +2026-11-15T10:40:06+01:00 demo-hp felhom-os-apply[2259713]: os-apply: PVE CA COPY the previous root is dropped from /etc/felhom-agent/pve-root-ca.pem (Proxmox serves a certificate of the new root) diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/hp-cert-round1.txt b/documentation/audits/release-2b-3-2026-11-15/part2/hp-cert-round1.txt new file mode 100644 index 00000000..efbdbab5 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/hp-cert-round1.txt @@ -0,0 +1,23 @@ +#### 2026-11-15 10:35:05 round 1: agent 0.175.1, unit active since Sun 2026-11-15 10:18:18 CET, pid 2183005 +root CA before: E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7 +served before: 1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74 +(re)generate node files +generate new node certificate +merge authorized SSH keys +file after updatecerts: AA:76:56:BD:A7:1C:3D:5D:1D:A5:2C:8F:5D:51:3F:F8:E3:B8:14:52:4E:90:95:9C:D8:E8:4E:78:0B:8C:15:45 +pveproxy reload rc=0 +served after: AA:76:56:BD:A7:1C:3D:5D:1D:A5:2C:8F:5D:51:3F:F8:E3:B8:14:52:4E:90:95:9C:D8:E8:4E:78:0B:8C:15:45 (after about 10 s) +root CA after: E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7 +T0=2026-11-15 10:35:05 +#### end round +#### demo-hp 2026-11-15 10:35:36: since 2026-11-15 10:35:16 +agent unit active since Sun 2026-11-15 10:18:18 CET, pid 2183005, restarts 0 +agent journal lines since: 4 +trust / certificate lines: 0 +ERROR lines: 0 +Proxmox access log, the agent's token, after 10:35:16: + calls: 14; by status: 200=14 + first: ::ffff:127.0.0.1 - felhom-agent@pve!agent [15/11/2026:10:35:20 +0100] "GET /api2/json/storage HTTP/1.1" 200 918 + last: ::ffff:127.0.0.1 - felhom-agent@pve!agent [15/11/2026:10:35:33 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config HTTP/1.1" 200 673 +served now: AA:76:56:BD:A7:1C:3D:5D:1D:A5:2C:8F:5D:51:3F:F8:E3:B8:14:52:4E:90:95:9C:D8:E8:4E:78:0B:8C:15:45 +#### end diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/hp-cert-round2.txt b/documentation/audits/release-2b-3-2026-11-15/part2/hp-cert-round2.txt new file mode 100644 index 00000000..6d6d9b7b --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/hp-cert-round2.txt @@ -0,0 +1,37 @@ +#### 2026-11-15 10:35:43 round 2: agent 0.175.1, unit active since Sun 2026-11-15 10:18:18 CET, pid 2183005 +root CA before: E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7 +served before: AA:76:56:BD:A7:1C:3D:5D:1D:A5:2C:8F:5D:51:3F:F8:E3:B8:14:52:4E:90:95:9C:D8:E8:4E:78:0B:8C:15:45 +(re)generate node files +generate new node certificate +merge authorized SSH keys +file after updatecerts: EF:06:C8:01:9E:B2:34:82:97:DE:11:DA:8B:57:8E:56:0C:11:91:1F:17:C0:64:98:C9:1A:1F:C5:47:33:CE:E1 +pveproxy reload rc=0 +served after: EF:06:C8:01:9E:B2:34:82:97:DE:11:DA:8B:57:8E:56:0C:11:91:1F:17:C0:64:98:C9:1A:1F:C5:47:33:CE:E1 (after about 10 s) +root CA after: E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7 +T0=2026-11-15 10:35:43 +#### end round +#### demo-hp 2026-11-15 10:37:10: since 2026-11-14 20:35:44 +agent unit active since Sun 2026-11-15 10:18:18 CET, pid 2183005, restarts 0 +agent journal lines since: 6374 +trust / certificate lines: 2 +2026-11-15T10:18:18+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:18:18.654+01:00 level=INFO msg="pvetrust: the root files are older than this agent (no CA-copy step yet) — the pinned certificate stays; the next root-files update moves the box" why=start mode=fingerprint err="pve-ca-copy refused: {\"code\": \"R11\", \"reason\": \"unknown mode 'pv +2026-11-15T10:33:34+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:34.354+01:00 level=WARN msg="pvetrust: MOVED — the agent trusts Proxmox by the box's own root CA now; the pinned fingerprint was cleared AFTER a working call with the copy" why="config bundle installed" mode=fingerprint ca_file=/etc/felhom-agent/pve-root-ca.pem ca_not_after=2036 +ERROR lines: 0 +Proxmox access log, the agent's token, after 20:35:44: + calls: 0; by status: + first: + last: +served now: EF:06:C8:01:9E:B2:34:82:97:DE:11:DA:8B:57:8E:56:0C:11:91:1F:17:C0:64:98:C9:1A:1F:C5:47:33:CE:E1 +#### end +## read again with the right time (the first read's time was computed wrong: 'date -d ... + 15 seconds') +#### demo-hp 2026-11-15 10:37:18: since 2026-11-15 10:35:58 +agent unit active since Sun 2026-11-15 10:18:18 CET, pid 2183005, restarts 0 +agent journal lines since: 8 +trust / certificate lines: 0 +ERROR lines: 0 +Proxmox access log, the agent's token, after 10:35:58: + calls: 40; by status: 200=40 + first: ::ffff:127.0.0.1 - felhom-agent@pve!agent [15/11/2026:10:36:00 +0100] "GET /api2/json/storage HTTP/1.1" 200 918 + last: ::ffff:127.0.0.1 - felhom-agent@pve!agent [15/11/2026:10:37:03 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config HTTP/1.1" 200 673 +served now: EF:06:C8:01:9E:B2:34:82:97:DE:11:DA:8B:57:8E:56:0C:11:91:1F:17:C0:64:98:C9:1A:1F:C5:47:33:CE:E1 +#### end diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/hp-moved.txt b/documentation/audits/release-2b-3-2026-11-15/part2/hp-moved.txt new file mode 100644 index 00000000..42574f85 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/hp-moved.txt @@ -0,0 +1,31 @@ +#### demo-hp 2026-11-15 10:34:56 CET agent 0.175.1 uptime-of-unit Sun 2026-11-15 10:18:18 CET +config proxmox.tls: ca_file='/etc/felhom-agent/pve-root-ca.pem' fingerprint=EMPTY insecure_skip_verify=False + -rw-r--r-- root:root 2094 bytes 2026-11-15 10:33:34.311028892 +0100 /etc/felhom-agent/pve-root-ca.pem + drwxr-xr-x root:root 4096 bytes 2026-11-15 10:33:34.317623237 +0100 /etc/felhom-agent + -rw------- felhom-agent:felhom-agent 2455 bytes 2026-11-15 10:33:34.34 /var/lib/felhom-agent/agent.json.pre-pve-trust + (absent) /var/lib/felhom-agent/agent.json.write-in-flight +CA copy /etc/felhom-agent/pve-root-ca.pem: 1 certificate(s); first: subject=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA notAfter=Aug 17 15:44:42 2036 GMT sha256 Fingerprint=E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7 + same as /etc/pve/pve-root-ca.pem: yes + agent user can read it: yes; others can read: r +served certificate (127.0.0.1:8006): sha256 Fingerprint=1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74 notAfter=Nov 9 13:59:54 2028 GMT +/etc/pve/local/pve-ssl.pem: sha256 Fingerprint=1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74 notBefore=Nov 10 13:59:54 2026 GMT +## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min) +2026-11-15T10:18:15+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T10:18:15.625+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=90d84d793f66c18d op=agent_update key_id=felhom-op-1 nonce=9e1c24967ec755838aeda82ce2520f14 +2026-11-15T10:18:15+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T10:18:15.872+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.175.1 version=0.175.1 +2026-11-15T10:18:16+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T10:18:16.400+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.175.1 report="{\"agent_update\": {\"sha256\": \"38b1e2350de90a7940c16f8cd2a21d83fcd535410d1017a7df0fee14f0904612\", \"version\": \"0.175.1\", \"wrapper\": \"Running timer as unit: run-p2182880-i23144936.timer\\nWill run service as u +2026-11-15T10:18:16+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T10:18:16.401+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=90d84d793f66c18d op=agent_update +2026-11-15T10:18:18+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:18:18.654+01:00 level=INFO msg="pvetrust: the root files are older than this agent (no CA-copy step yet) — the pinned certificate stays; the next root-files update moves the box" why=start mode=fingerprint err="pve-ca-copy refused: {\"code\": \"R11\", \"reason\": \"unknown mode 'pve-ca-copy'\"}" +2026-11-15T10:18:20+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:18:20.815+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.175.1 prev=0.174.0 dwell=1m0s +2026-11-15T10:19:20+01:00 demo-hp felhom-selfupdate-guarded[2187989]: committed (pending cleared, .prev retained) +2026-11-15T10:19:20+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:19:20.843+01:00 level=WARN msg="selfupdate: update committed" version=0.175.1 wrapper="" +2026-11-15T10:19:20+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:19:20.962+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.175.1 removed=1 +2026-11-15T10:19:20+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:19:20.962+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=1 kept=felhom-agent-0.175.1 +2026-11-15T10:33:31+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:31.703+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=9b72aa335a322ba6 op=agent_config_update key_id=felhom-op-1 nonce=c93c80251a155095f98d648bf9b16f51 +2026-11-15T10:33:31+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:31.813+01:00 level=INFO msg="osupdate: config bundle downloaded; handing it to the root wrapper" op=agent_config_update agent_version=0.175.1 sha256=07ebf86362ed9d44 duration_ms=110 +2026-11-15T10:33:33+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:33.036+01:00 level=WARN msg="osupdate: config bundle INSTALLED" op=agent_config_update agent_version=0.175.1 bundle="{\"agent_version\": \"0.175.1\", \"authority\": \"signed\", \"kept\": [\"/etc/felhom/crash-guard.conf\"], \"prev_dir\": \"/var/lib/felhom-os-apply/bundle-prev/20261115T093332Z-before-0.175.1\", \"removed\": [], \"same\": 30, \"self_ch +2026-11-15T10:33:34+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:34.180+01:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=77 total=77 degraded="" +2026-11-15T10:33:34+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:34.180+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=9b72aa335a322ba6 op=agent_config_update +2026-11-15T10:33:34+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:34.293+01:00 level=INFO msg="osupdate: profile refresh not needed" mode=profile-refresh why="config bundle installed" kernel=7.0.14-28-pve reason="the profiles were refreshed on this kernel (7.0.14-28-pve) at 2026-11-14T11:33:18Z" +2026-11-15T10:33:34+01:00 demo-hp felhom-agent[2183005]: time=2026-11-15T10:33:34.354+01:00 level=WARN msg="pvetrust: MOVED — the agent trusts Proxmox by the box's own root CA now; the pinned fingerprint was cleared AFTER a working call with the copy" why="config bundle installed" mode=fingerprint ca_file=/etc/felhom-agent/pve-root-ca.pem ca_not_after=2036-08-17T15:44:42Z +## one control: the agent's newest successful host report and newest Proxmox call error +#### end diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/redproof-agent-root-step-too-old.txt b/documentation/audits/release-2b-3-2026-11-15/part2/redproof-agent-root-step-too-old.txt new file mode 100644 index 00000000..c4c2ce33 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/redproof-agent-root-step-too-old.txt @@ -0,0 +1,45 @@ +== MUTATION: older root files are named as a trust problem again + RED on the mutant, GREEN on the real file + --- FAIL: TestRootStepOlderThanTheAgent_IsNotATrustProblem (0.00s) + pvetrust_test.go:507: a pinned box with older root files reports a trust problem: &{Mode:fingerprint Problem:ca_copy_unavailable Detail:pve-ca-copy refused: {"code": "R11", "reason": "unknown mode 'pve-ca-copy'"} CANotAfter: CADay +== MUTATION: an UNPINNED box with no copy step is silent too + RED on the mutant, GREEN on the real file + --- FAIL: TestRootStepOlderThanTheAgent_IsNotATrustProblem (0.00s) + pvetrust_test.go:518: an unpinned box with no copy step must name the problem: &{Mode:system Problem: Detail: CANotAfter: CADaysLeft: MovedAt: CheckedAt:2026-11-15T09:05:53Z} +== MUTATION: every refusal is silent + *** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0) + FAIL gitea.dooplex.hu/admin/felhom-agent/internal/pvetrust [build failed] + +3 mutations, 1 not a red-proof +== MUTATION: every refusal is silent (redone: the first mutant did not compile) + RED on the mutant, GREEN on the real file + --- FAIL: TestRootStepRefuses_NothingChanges (0.00s) + pvetrust_test.go:374: status = &{Mode:fingerprint Problem: Detail: CANotAfter: CADaysLeft: MovedAt: CheckedAt:2026-11-15T09:06:04Z} + --- FAIL: TestRootStepOlderThanTheAgent_IsNotATrustProblem (0.00s) + pvetrust_test.go:526: a real refusal lost its name: &{Mode:fingerprint Problem: Detail: CANotAfter: CADaysLeft: MovedAt: CheckedAt:2026-11-15T09:06:04Z} + +1 mutations, 0 not a red-proof + +## after the commit's security review: the quiet branch never hides a failed pin +== MUTATION: a run started by a failed call takes the quiet branch + RED on the mutant, GREEN on the real file + --- FAIL: TestRootStepOlderThanTheAgent_NeverHidesAFailedPin (0.00s) + pvetrust_test.go:543: a run started by a failed pin reports no problem: &{Mode:fingerprint Problem: Detail: CANotAfter: CADaysLeft: MovedAt: CheckedAt:2026-11-15T09:07:36Z} +== MUTATION: the timer clears a recorded pin failure + *** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0) + FAIL gitea.dooplex.hu/admin/felhom-agent/internal/pvetrust [build failed] + internal/pvetrust/pvetrust.go:251:3: declared and not used: recorded +== MUTATION: any refusal that says unknown mode counts, without the code + *** NOT A RED-PROOF *** (mutant rc=0 built=True real rc=0) + +3 mutations, 2 not a red-proof +== MUTATION: the timer clears a recorded pin failure (redone: the first mutant did not compile) + RED on the mutant, GREEN on the real file + --- FAIL: TestRootStepOlderThanTheAgent_NeverHidesAFailedPin (0.00s) + pvetrust_test.go:564: the timer cleared a recorded pin failure: &{Mode:fingerprint Problem: Detail: CANotAfter: CADaysLeft: MovedAt: CheckedAt:2026-11-15T09:07:52Z} +== MUTATION: any refusal that says unknown mode counts, without the code (redone: the test had no such case; one was added) + RED on the mutant, GREEN on the real file + --- FAIL: TestRootStepOlderThanTheAgent_IsNotATrustProblem (0.00s) + pvetrust_test.go:536: a refusal with another code was read as older root files: &{Mode:fingerprint Problem: Detail: CANotAfter: CADaysLeft: MovedAt: CheckedAt:2026-11-15T09:07:54Z} + +2 mutations, 0 not a red-proof diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/system-1107.txt b/documentation/audits/release-2b-3-2026-11-15/part2/system-1107.txt new file mode 100644 index 00000000..61b02c99 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/system-1107.txt @@ -0,0 +1,265 @@ +System — Felhom Hub +Felhom Hub +Dashboard +Customers +Apps +Hosts +System +Offsite +Configuration +System — versions and OS updates +Needs attention +One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3). +[hover: worth a look] look +Tester4-46836f Tester4 +[hover: cannot do a one-shot boot: the box does not boot UEFI; /boot/efi is not a mounted vfat ESP (not mounted)] kernel step: — · the host needs a restart since 2026-11-14 · [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] root files behind the vouched agent's · [hover: 1 minor release behind — sign an agent_update for this box] agent behind: 0.173.0 → 0.174.0 (since 2026-11-14) · [hover: the box could not read these values (agent older than v0.142.0, or the guest is down) — never a guess] could not read: default kernel +[hover: worth a look] look +demo-felhom-8363b5 Demo Ügyfél +[hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update] root files behind the vouched agent's +[hover: worth a look] look +demo-hp-bb76ea Demo HP +[hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-15 09:51 UTC)] root files behind the vouched agent's +[hover: worth a look] look +tester-1-d70be4 Tester 1 +[hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-15 09:51 UTC)] root files behind the vouched agent's +Waiting for you +Update sets that only you approve: the kernel, the Docker engine and the Proxmox packages. Guest and host Debian updates approve themselves after 24 h and one night on the [hover: Ring 0 = the demo boxes. They install every new fix first; the other boxes (ring 1) install only what is approved.] ring-0 boxes. +Docker engine 29.9.3-1 +Ran on every ring-0 box — demo-felhom-8363b5: 2 healthy night(s), demo-hp-bb76ea: 2 healthy night(s). The memory-kill check passed. +first seen 2 days ago (2026-11-13 04:49 UTC). Approving installs nothing by itself: a ring-1 box takes it only through a signed operator job. +Approve Docker set +Proxmox packages 9.2.27 — still being tested: demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set +Boxes +Click a box to see every value it reports, and its ring and update switches. "unknown": the box could not read the value — never a guess. +Box Ring Health Controller · agent OS updates Kernel [hover: The newest OS update run] Last night +Tester4-46836f Tester4 +[hover: Ring 1: a normal box — it installs only approved releases] 1 +[hover: worth a look] look +0.352.1 [hover: 1 minor release behind — sign an agent_update for this box] agent 0.173.0 → 0.174.0 (since 2026-11-14) +[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind +7.0.2-6-pve +[hover: 23 h ago · applied · 93 s — last successful leg: 23 h ago] ok · 23 h ago +Ring and updates +Ring ring 1 +→ demo +OS updates updates ON +switch off +Tunnel [hover: no cloudflared container in the guest (agent state: absent)] none set up +Controller 0.352.1 +Agent [hover: 1 minor release behind — sign an agent_update for this box] 0.173.0 → 0.174.0 (since 2026-11-14) +Last OS leg [hover: last successful leg: 23 h ago] 23 h ago · applied · 93 s +Docker engine +Docker 29.9.2 +containerd 2.4.2-3~debian.13~trixie +live-restore on +Docker release — +Host +Proxmox 9.2.2 +Kernel (running) 7.0.2-6-pve +Kernel (next boot) 7.0.2-6-pve +[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown +[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: cannot do a one-shot boot: the box does not boot UEFI; /boot/efi is not a mounted vfat ESP (not mounted)] — +[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] — +[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: the box reports no security-profile refresh (an agent older than release 3, a BYO host, or no refresh yet)] — +Debian 13.7 +Felhom release os-host-20261111-034654 +Pending 100 +Not covered 0 +Held none +Reboot needed since 2026-11-14 +kernel.panic 10 s +Oops no +Crash restarts 24 h 0 +Crash guard armed +[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] 0.173.0 +Guest +Guest Debian 13.7 +Felhom release os-guest-20261111-034654 +Pending 6 +Restart needed 7 +[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 23 h ago · 78.4 GiB +[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] not yet +demo-felhom-8363b5 Demo Ügyfél +[hover: Ring 0: a demo box — it installs every new fix first] 0 +[hover: worth a look] look +0.352.1 [hover: current (vouched 0.174.0)] agent 0.175.1 +[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind +7.0.14-28-pve +[hover: 5 h ago · nothing · 23 s — last successful leg: 5 h ago] ok · 5 h ago +Ring and updates +Ring ring 0 +→ normal +OS updates updates ON +switch off +Tunnel running +Controller 0.352.1 +Agent [hover: current (vouched 0.174.0)] 0.175.1 +Last OS leg [hover: last successful leg: 5 h ago] 5 h ago · nothing · 23 s +Docker engine +Docker 29.9.3 +containerd 2.4.2-4~debian.13~trixie +live-restore on +Docker release ring0-20261115T045634Z +Host +Proxmox 9.2.27 +Kernel (running) 7.0.14-28-pve +Kernel (next boot) 7.0.14-28-pve +[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve +[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 1m14s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 7 days ago · due 7.0.14-29-pve — household told 2 h ago: TONIGHT · phase good +[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: removed 6 old kernel package(s): proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed; kept 7.0.14-26-pve, 7.0.14-28-pve; 6.3 GiB gained; 0 left-over path(s)] cleaned 5 h ago +[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: not_needed] 7.0.14-28-pve · 22 h ago +Debian 13.7 +Felhom release ring0-20261115T045634Z +Pending 29 +Not covered 0 +Held none +Reboot needed no +kernel.panic 10 s +Oops no +Crash restarts 24 h 0 +Crash guard armed +[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update] 0.175.1 +Guest +Guest Debian 13.7 +Felhom release ring0-20261115T045634Z +Pending 0 +Restart needed 1 +[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 4 days ago · 272.7 GiB +[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 92.7 Mbit/s · 5 days ago +demo-hp-bb76ea Demo HP +[hover: Ring 0: a demo box — it installs every new fix first] 0 +[hover: worth a look] look +0.352.1 [hover: current (vouched 0.174.0)] agent 0.175.1 +[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind +7.0.14-28-pve +[hover: 6 h ago · nothing · 30 s — last successful leg: 6 h ago] ok · 6 h ago +Ring and updates +Ring ring 0 +→ normal +OS updates updates ON +switch off +Tunnel running +Controller 0.352.1 +Agent [hover: current (vouched 0.174.0)] 0.175.1 +Last OS leg [hover: last successful leg: 6 h ago] 6 h ago · nothing · 30 s +Docker engine +Docker 29.9.3 +containerd 2.4.2-4~debian.13~trixie +live-restore on +Docker release ring0-20261115T040248Z +Host +Proxmox 9.2.27 +Kernel (running) 7.0.14-28-pve +Kernel (next boot) 7.0.14-28-pve +[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve +[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 2m39s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 5 days ago · due 7.0.14-29-pve — household told 2 h ago: TONIGHT · phase good +[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] — +[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: not_needed] 7.0.14-28-pve · 22 h ago +Debian 13.7 +Felhom release ring0-20261115T040248Z +Pending 30 +Not covered 0 +Held none +Reboot needed no +kernel.panic 10 s +Oops no +Crash restarts 24 h 0 +Crash guard armed +[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-15 09:51 UTC)] 0.175.1 +Guest +Guest Debian 13.7 +Felhom release ring0-20261115T040248Z +Pending 0 +Restart needed 1 +[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 4 days ago · 70.5 GiB +[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 94.3 Mbit/s · 4 days ago +tester-1-d70be4 Tester 1 +[hover: Ring 1: a normal box — it installs only approved releases] 1 +[hover: worth a look] look +0.352.1 [hover: current (vouched 0.174.0)] agent 0.175.1 +[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind +7.0.14-22-pve +[hover: 5 h ago · nothing · 16 s — last successful leg: 5 h ago] ok · 5 h ago +Ring and updates +Ring ring 1 +→ demo +OS updates updates ON +switch off +Tunnel running +Controller 0.352.1 +Agent [hover: current (vouched 0.174.0)] 0.175.1 +Last OS leg [hover: last successful leg: 5 h ago] 5 h ago · nothing · 16 s +Docker engine +Docker 29.8.2 +containerd 2.3.6-1~debian.13~trixie +live-restore on +Docker release — +Host +Proxmox 9.2.2 +Kernel (running) 7.0.14-22-pve +Kernel (next boot) 7.0.14-22-pve +[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-22-pve +[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: ring 1: no kernel staged by a signed os_kernel_step healthy 1m17s after the agent started; the new kernel is the default] 7.0.14-22-pve applied 38 days ago · phase good +[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] — +[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: not_needed] 7.0.14-22-pve · 22 h ago +Debian 13.7 +Felhom release os-host-20261111-034654 +Pending 101 +Not covered 0 +Held none +Reboot needed no +kernel.panic 10 s +Oops no +Crash restarts 24 h 0 +Crash guard armed +[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-15 09:51 UTC)] 0.175.1 +Guest +Guest Debian 13.7 +Felhom release os-guest-20261111-034654 +Pending 6 +Restart needed 0 +[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 4 days ago · 90.4 GiB +[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 93.0 Mbit/s · 4 days ago +Details +Release ids, cancelled approvals, what ring 0 runs, the version floors, and every box's crash guard and root files. +Approved releases +The newest approved set of each layer. A [hover: Approved while a TEST wait override was active. It is cancelled when the hub starts without the override (`11` §5.3.1).] TEST approval is amber. +guest : os-guest-20261111-034654 +272 packages · 2026-11-11 03:46 UTC · by auto +host : os-host-20261115-045718 +607 packages · 2026-11-15 04:57 UTC · by auto +docker : os-docker-20261026-190351 +6 packages · 2026-10-26 19:03 UTC · by operator +pve : os-pve-20261114-075712 +121 packages · 2026-11-14 07:57 UTC · by operator +kernel : os-kernel-20261114-075716 +2 packages · 2026-11-14 07:57 UTC · by operator +Approve now (guest + host) +An urgent fix only — normally the hub approves after 24 h and one night. +What ring 0 runs now +guest : +272 packages, first seen 2026-11-10 01:49 UTC +approved as os-guest-20261111-034654 +host : +607 packages, first seen 2026-11-14 04:52 UTC +approved as os-host-20261115-045718 +docker : +6 packages, first seen 2026-11-13 04:49 UTC +ready — see "Waiting for you" +pve : +121 packages, first seen 2026-11-15 04:58 UTC +demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set +kernel : +2 packages, first seen 2026-11-10 01:57 UTC +approved as os-kernel-20261114-075716 +Version floors +Global controller floor: 0.352.1 · vouched agent: 0.174.0 +[hover: A floor is the lowest controller version a box must run; the hub moves a box below it up.] What is a floor? · [hover: The agent version the operator checked and signed off for the fleet. Agents update only by a per-box signed job.] What is a vouched agent? +No per-customer floors: every box follows the global floor. +Crash guard and root files +Box ¦ kernel.panic ¦ Oops ¦ Crash restarts 24 h ¦ Crash guard ¦ [hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files ¦ +Tester4-46836f ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] 0.173.0 ¦ +demo-felhom-8363b5 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update] 0.175.1 ¦ +demo-hp-bb76ea ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-15 09:51 UTC)] 0.175.1 ¦ +tester-1-d70be4 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-15 09:51 UTC)] 0.175.1 ¦ +Felhom Hub 0.182.0 diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/t1-after-agent.txt b/documentation/audits/release-2b-3-2026-11-15/part2/t1-after-agent.txt new file mode 100644 index 00000000..1f694fb7 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/t1-after-agent.txt @@ -0,0 +1,17 @@ +#### felhom 2026-11-15 10:04:46 CET agent 0.175.0 uptime-of-unit Sun 2026-11-15 10:04:33 CET +config proxmox.tls: ca_file='' fingerprint=set (BD:EE:9F:C4…) insecure_skip_verify=False + (absent) /etc/felhom-agent/pve-root-ca.pem + drwxr-xr-x root:root 4096 bytes 2026-10-04 21:40:41.128703040 +0200 /etc/felhom-agent + (absent) /var/lib/felhom-agent/agent.json.pre-pve-trust + (absent) /var/lib/felhom-agent/agent.json.write-in-flight +served certificate (127.0.0.1:8006): sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notAfter=Oct 2 19:38:05 2028 GMT +/etc/pve/local/pve-ssl.pem: sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notBefore=Oct 3 19:38:05 2026 GMT +## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min) +2026-11-15T10:04:30+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:30.371+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=b5c45465f233eb81 op=agent_update key_id=felhom-op-1 nonce=d5fd87f441408954947c67b4b44ab6b2 +2026-11-15T10:04:30+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:30.619+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.175.0 version=0.175.0 +2026-11-15T10:04:31+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:31.020+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.175.0 report="{\"agent_update\": {\"sha256\": \"d1a30ed51f6663d96369ad3e88c221194af92773addf4fbb48852ee58be4b19a\", \"version\": \"0.175.0\", \"wrapper\": \"Running timer as unit: run-p1546906-i1549255.timer\\nWill run service as unit: +2026-11-15T10:04:31+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:31.020+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=b5c45465f233eb81 op=agent_update +2026-11-15T10:04:33+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:04:33.269+01:00 level=WARN msg="pvetrust: the root step gave no CA copy — the trust stays as it is" why=start mode=fingerprint err="pve-ca-copy refused: {\"code\": \"R11\", \"reason\": \"unknown mode 'pve-ca-copy'\"}" +2026-11-15T10:04:34+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:04:34.922+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.175.0 prev=0.174.0 dwell=1m0s +## one control: the agent's newest successful host report and newest Proxmox call error +#### end diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/t1-after-bundle.txt b/documentation/audits/release-2b-3-2026-11-15/part2/t1-after-bundle.txt new file mode 100644 index 00000000..666923c0 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/t1-after-bundle.txt @@ -0,0 +1,21 @@ +#### felhom 2026-11-15 10:09:02 CET agent 0.175.0 uptime-of-unit Sun 2026-11-15 10:04:33 CET +config proxmox.tls: ca_file='' fingerprint=set (BD:EE:9F:C4…) insecure_skip_verify=False + (absent) /etc/felhom-agent/pve-root-ca.pem + drwxr-xr-x root:root 4096 bytes 2026-10-04 21:40:41.128703040 +0200 /etc/felhom-agent + (absent) /var/lib/felhom-agent/agent.json.pre-pve-trust + (absent) /var/lib/felhom-agent/agent.json.write-in-flight +served certificate (127.0.0.1:8006): sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notAfter=Oct 2 19:38:05 2028 GMT +/etc/pve/local/pve-ssl.pem: sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notBefore=Oct 3 19:38:05 2026 GMT +## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min) +2026-11-15T10:04:30+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:30.371+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=b5c45465f233eb81 op=agent_update key_id=felhom-op-1 nonce=d5fd87f441408954947c67b4b44ab6b2 +2026-11-15T10:04:30+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:30.619+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.175.0 version=0.175.0 +2026-11-15T10:04:31+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:31.020+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.175.0 report="{\"agent_update\": {\"sha256\": \"d1a30ed51f6663d96369ad3e88c221194af92773addf4fbb48852ee58be4b19a\", \"version\": \"0.175.0\", \"wrapper\": \"Running timer as unit: run-p1546906-i1549255.timer\\nWill run service as unit: +2026-11-15T10:04:31+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:31.020+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=b5c45465f233eb81 op=agent_update +2026-11-15T10:04:33+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:04:33.269+01:00 level=WARN msg="pvetrust: the root step gave no CA copy — the trust stays as it is" why=start mode=fingerprint err="pve-ca-copy refused: {\"code\": \"R11\", \"reason\": \"unknown mode 'pve-ca-copy'\"}" +2026-11-15T10:04:34+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:04:34.922+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.175.0 prev=0.174.0 dwell=1m0s +2026-11-15T10:05:34+01:00 felhom felhom-selfupdate-guarded[1549546]: committed (pending cleared, .prev retained) +2026-11-15T10:05:34+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:05:34.955+01:00 level=WARN msg="selfupdate: update committed" version=0.175.0 wrapper="" +2026-11-15T10:05:35+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:05:35.150+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.175.0 removed=1 +2026-11-15T10:05:35+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:05:35.150+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=1 kept=felhom-agent-0.175.0 +## one control: the agent's newest successful host report and newest Proxmox call error +#### end diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/t1-before.txt b/documentation/audits/release-2b-3-2026-11-15/part2/t1-before.txt new file mode 100644 index 00000000..ab455da9 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/t1-before.txt @@ -0,0 +1,11 @@ +#### felhom 2026-11-15 09:51:08 CET agent 0.174.0 uptime-of-unit Sat 2026-11-14 15:19:22 CET +config proxmox.tls: ca_file='' fingerprint=set (BD:EE:9F:C4…) insecure_skip_verify=False + (absent) /etc/felhom-agent/pve-root-ca.pem + drwxr-xr-x root:root 4096 bytes 2026-10-04 21:40:41.128703040 +0200 /etc/felhom-agent + (absent) /var/lib/felhom-agent/agent.json.pre-pve-trust + (absent) /var/lib/felhom-agent/agent.json.write-in-flight +served certificate (127.0.0.1:8006): sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notAfter=Oct 2 19:38:05 2028 GMT +/etc/pve/local/pve-ssl.pem: sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notBefore=Oct 3 19:38:05 2026 GMT +## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min) +## one control: the agent's newest successful host report and newest Proxmox call error +#### end diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/t1-clock-A-2min.txt b/documentation/audits/release-2b-3-2026-11-15/part2/t1-clock-A-2min.txt new file mode 100644 index 00000000..097e7c7b --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/t1-clock-A-2min.txt @@ -0,0 +1,14 @@ +## felhom clock 2026-11-15 10:49:52.0408764 epoch 1794736192.042; chrony inactive; +## kernel boot_id 7a721957-7e95-476c-be40-52d0dafc7284; btime 1794665801; uptime 70390.82 s +## containers in 9201 (name id created): + bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 5 hours (healthy)] + bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 5 hours (healthy)] + cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 20 hours (healthy)] + felhom-controller 2a7970a2c8a7 2026-11-14 14:19:33 +0000 UTC [Up 20 hours (healthy)] + filebrowser 95f38571abee 2026-11-14 14:19:57 +0000 UTC [Up 19 hours (healthy)] + paperless-postgres c7807ae06022 2026-11-15 09:48:39 +0000 UTC [Up About a minute (healthy)] + paperless-redis 56d085b17130 2026-11-15 09:48:39 +0000 UTC [Up About a minute (healthy)] + paperless-webserver 71aee1583019 2026-11-15 09:48:39 +0000 UTC [Up About a minute (healthy)] + privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 5 hours (healthy)] + traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 20 hours] +## count 10 running diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/t1-clock-A-after-right.txt b/documentation/audits/release-2b-3-2026-11-15/part2/t1-clock-A-after-right.txt new file mode 100644 index 00000000..22fd95f7 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/t1-clock-A-after-right.txt @@ -0,0 +1,14 @@ +## felhom clock 2026-11-15 11:06:49.5647785 epoch 1794737209.566; chrony active; System time : 0.001194180 seconds slow of NTP time;Leap status : Normal; +## kernel boot_id 7a721957-7e95-476c-be40-52d0dafc7284; btime 1794665951; uptime 71258.03 s +## containers in 9201 (name id created): + bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 5 hours (healthy)] + bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 5 hours (healthy)] + cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 20 hours (healthy)] + felhom-controller 2a7970a2c8a7 2026-11-14 14:19:33 +0000 UTC [Up 20 hours (healthy)] + filebrowser 95f38571abee 2026-11-14 14:19:57 +0000 UTC [Up 20 hours (healthy)] + paperless-postgres 31b2490aa677 2026-11-15 10:05:37 +0000 UTC [Up About a minute (healthy)] + paperless-redis df1c44607a9b 2026-11-15 10:05:37 +0000 UTC [Up About a minute (healthy)] + paperless-webserver 1f2a0d7e4f7b 2026-11-15 10:05:37 +0000 UTC [Up About a minute (healthy)] + privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 5 hours (healthy)] + traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 20 hours] +## count 10 running diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/t1-clock-A-back.txt b/documentation/audits/release-2b-3-2026-11-15/part2/t1-clock-A-back.txt new file mode 100644 index 00000000..411964eb --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/t1-clock-A-back.txt @@ -0,0 +1,16 @@ +## felhom clock 2026-11-15 10:50:27.2531193 epoch 1794736227.254; chrony active; System time : 0.000242970 seconds slow of NTP time;Leap status : Normal; +## kernel boot_id 7a721957-7e95-476c-be40-52d0dafc7284; btime 1794665951; uptime 70275.75 s +## containers in 9201 (name id created): + bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 5 hours (healthy)] + bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 5 hours (healthy)] + cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 20 hours (healthy)] + felhom-controller 2a7970a2c8a7 2026-11-14 14:19:33 +0000 UTC [Up 20 hours (healthy)] + filebrowser 95f38571abee 2026-11-14 14:19:57 +0000 UTC [Up 20 hours (healthy)] + paperless-postgres e0d945977ebc 2026-11-15 04:50:09 +0000 UTC [Up 5 hours (healthy)] + paperless-redis 6188dea5adbc 2026-11-15 04:50:09 +0000 UTC [Up 5 hours (healthy)] + paperless-webserver cc93a8574d66 2026-11-15 04:50:10 +0000 UTC [Up 5 hours (healthy)] + privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 5 hours (healthy)] + traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 20 hours] +## count 10 running +chrony stopped rc=0 +clock set 150 s back rc=0 → 2026-11-15 10:47:59 diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/t1-clock-A-right.txt b/documentation/audits/release-2b-3-2026-11-15/part2/t1-clock-A-right.txt new file mode 100644 index 00000000..2fac9c8a --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/t1-clock-A-right.txt @@ -0,0 +1,4 @@ +chrony started rc=0 +200 OK +clock now 2026-11-15 11:05:36; System time : 0.000000000 seconds slow of NTP time;Leap status : Normal; +Tester 1 minus DooPlex: +0.073 s diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/t1-moved.txt b/documentation/audits/release-2b-3-2026-11-15/part2/t1-moved.txt new file mode 100644 index 00000000..45818568 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/t1-moved.txt @@ -0,0 +1,31 @@ +#### felhom 2026-11-15 10:20:19 CET agent 0.175.0 uptime-of-unit Sun 2026-11-15 10:04:33 CET +config proxmox.tls: ca_file='/etc/felhom-agent/pve-root-ca.pem' fingerprint=EMPTY insecure_skip_verify=False + -rw-r--r-- root:root 2094 bytes 2026-11-15 10:19:47.714398283 +0100 /etc/felhom-agent/pve-root-ca.pem + drwxr-xr-x root:root 4096 bytes 2026-11-15 10:19:47.716626910 +0100 /etc/felhom-agent + -rw------- felhom-agent:felhom-agent 2451 bytes 2026-11-15 10:19:47.73 /var/lib/felhom-agent/agent.json.pre-pve-trust + (absent) /var/lib/felhom-agent/agent.json.write-in-flight +CA copy /etc/felhom-agent/pve-root-ca.pem: 1 certificate(s); first: subject=CN=Proxmox Virtual Environment, OU=f9c69a56-050d-43dd-87c8-d596f6a79d19, O=PVE Cluster Manager CA notAfter=Sep 30 19:38:05 2036 GMT sha256 Fingerprint=4E:46:8D:BC:45:A2:2C:CD:AB:EF:82:C0:A4:B1:4D:6C:5C:DA:5A:7F:6F:D7:3B:E9:B7:50:CA:ED:C4:0C:80:CF + same as /etc/pve/pve-root-ca.pem: yes + agent user can read it: yes; others can read: r +served certificate (127.0.0.1:8006): sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notAfter=Oct 2 19:38:05 2028 GMT +/etc/pve/local/pve-ssl.pem: sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B notBefore=Oct 3 19:38:05 2026 GMT +## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min) +2026-11-15T10:04:30+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:30.371+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=b5c45465f233eb81 op=agent_update key_id=felhom-op-1 nonce=d5fd87f441408954947c67b4b44ab6b2 +2026-11-15T10:04:30+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:30.619+01:00 level=WARN msg="agent_update: handing the signed job to the root wrapper (felhom-os-apply agent_update)" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.175.0 version=0.175.0 +2026-11-15T10:04:31+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:31.020+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.175.0 report="{\"agent_update\": {\"sha256\": \"d1a30ed51f6663d96369ad3e88c221194af92773addf4fbb48852ee58be4b19a\", \"version\": \"0.175.0\", \"wrapper\": \"Running timer as unit: run-p1546906-i1549255.timer\\nWill run service as unit: +2026-11-15T10:04:31+01:00 felhom felhom-agent[1148]: time=2026-11-15T10:04:31.020+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=b5c45465f233eb81 op=agent_update +2026-11-15T10:04:33+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:04:33.269+01:00 level=WARN msg="pvetrust: the root step gave no CA copy — the trust stays as it is" why=start mode=fingerprint err="pve-ca-copy refused: {\"code\": \"R11\", \"reason\": \"unknown mode 'pve-ca-copy'\"}" +2026-11-15T10:04:34+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:04:34.922+01:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.175.0 prev=0.174.0 dwell=1m0s +2026-11-15T10:05:34+01:00 felhom felhom-selfupdate-guarded[1549546]: committed (pending cleared, .prev retained) +2026-11-15T10:05:34+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:05:34.955+01:00 level=WARN msg="selfupdate: update committed" version=0.175.0 wrapper="" +2026-11-15T10:05:35+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:05:35.150+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.175.0 removed=1 +2026-11-15T10:05:35+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:05:35.150+01:00 level=INFO msg="selfupdate: old staged binaries removed after the commit" count=1 kept=felhom-agent-0.175.0 +2026-11-15T10:19:45+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:45.500+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=0c91a3e29ac9ba27 op=agent_config_update key_id=felhom-op-1 nonce=cb391943ab870fd924c8b90ef98b2065 +2026-11-15T10:19:45+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:45.543+01:00 level=INFO msg="osupdate: config bundle downloaded; handing it to the root wrapper" op=agent_config_update agent_version=0.175.0 sha256=3b4a8b78155fc438 duration_ms=42 +2026-11-15T10:19:46+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:46.553+01:00 level=WARN msg="osupdate: config bundle INSTALLED" op=agent_config_update agent_version=0.175.0 bundle="{\"agent_version\": \"0.175.0\", \"authority\": \"signed\", \"kept\": [\"/etc/felhom/crash-guard.conf\"], \"prev_dir\": \"/var/lib/felhom-os-apply/bundle-prev/20261115T091945Z-before-0.175.0\", \"removed\": [], \"same\": 30, \"self_che +2026-11-15T10:19:47+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:47.590+01:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=77 total=77 degraded="" +2026-11-15T10:19:47+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:47.590+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=0c91a3e29ac9ba27 op=agent_config_update +2026-11-15T10:19:47+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:47.685+01:00 level=INFO msg="osupdate: profile refresh not needed" mode=profile-refresh why="config bundle installed" kernel=7.0.14-22-pve reason="the profiles were refreshed on this kernel (7.0.14-22-pve) at 2026-11-14T11:30:11Z" +2026-11-15T10:19:47+01:00 felhom felhom-agent[1546925]: time=2026-11-15T10:19:47.744+01:00 level=WARN msg="pvetrust: MOVED — the agent trusts Proxmox by the box's own root CA now; the pinned fingerprint was cleared AFTER a working call with the copy" why="config bundle installed" mode=fingerprint ca_file=/etc/felhom-agent/pve-root-ca.pem ca_not_after=2036-09-30T19:38:05Z +## one control: the agent's newest successful host report and newest Proxmox call error +#### end diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/trust-read.sh b/documentation/audits/release-2b-3-2026-11-15/part2/trust-read.sh new file mode 100644 index 00000000..d49156ff --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/trust-read.sh @@ -0,0 +1,23 @@ +# Part 2 — READS ONLY: the agent's version and its trust in Proxmox, on one box. Never prints a token or a key. +set -u +echo "#### $(hostname) $(date +%F\ %T\ %Z) agent $(felhom-agent --version 2>&1 | awk '{print $2}') uptime-of-unit $(systemctl show felhom-agent -p ActiveEnterTimestamp --value)" +python3 - <<'PY' +import json +d = json.load(open("/etc/felhom-agent/agent.json")) +t = d.get("proxmox", {}).get("tls", {}) +fp = t.get("fingerprint", "") +print("config proxmox.tls: ca_file=%r fingerprint=%s insecure_skip_verify=%r" % (t.get("ca_file", ""), ("set (%s…)" % fp[:11]) if fp else "EMPTY", t.get("insecure_skip_verify"))) +PY +for f in /etc/felhom-agent/pve-root-ca.pem /etc/felhom-agent /var/lib/felhom-agent/agent.json.pre-pve-trust /var/lib/felhom-agent/agent.json.write-in-flight; do + [ -e "$f" ] && echo " $(stat -c '%A %U:%G %s bytes %y' "$f" | cut -c1-70) $f" || echo " (absent) $f"; done +CA=$(python3 -c 'import json;print(json.load(open("/etc/felhom-agent/agent.json")).get("proxmox",{}).get("tls",{}).get("ca_file",""))') +for c in "$CA" /etc/felhom-agent/pve-root-ca.pem; do + [ -n "$c" ] && [ -e "$c" ] && { echo "CA copy $c: $(grep -c 'BEGIN CERTIFICATE' "$c") certificate(s); first: $(openssl x509 -in "$c" -noout -subject -enddate -fingerprint -sha256 2>/dev/null | tr '\n' ' ' | cut -c1-260)"; echo " same as /etc/pve/pve-root-ca.pem: $(cmp -s <(openssl x509 -in "$c") <(openssl x509 -in /etc/pve/pve-root-ca.pem) && echo yes || echo NO)"; echo " agent user can read it: $(sudo -n -u felhom-agent test -r "$c" && echo yes || echo no); others can read: $(stat -c %A "$c" | cut -c8)"; break; } +done +echo "served certificate (127.0.0.1:8006): $(echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256 -enddate 2>/dev/null | tr '\n' ' ')" +echo "/etc/pve/local/pve-ssl.pem: $(openssl x509 -in /etc/pve/local/pve-ssl.pem -noout -fingerprint -sha256 -startdate 2>/dev/null | tr '\n' ' ')" +echo "## agent journal: trust, self-update, bundle, and Proxmox call errors (last 40 min)" +journalctl -u felhom-agent --since "-40min" --no-pager -o short-iso 2>&1 | grep -iE "pvetrust|trust|selfupdate|config bundle|signedjobs|capability|x509|tls:|certificate|config: " | grep -v "sudo\[" | cut -c1-430 | tail -${1:-25} +echo "## one control: the agent's newest successful host report and newest Proxmox call error" +journalctl -u felhom-agent --since "-20min" --no-pager -o short-iso 2>&1 | grep -iE "host report|report sent|hub: report|heartbeat" | tail -2 | cut -c1-260 +echo "#### end" diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/wait-agent.sh b/documentation/audits/release-2b-3-2026-11-15/part2/wait-agent.sh new file mode 100644 index 00000000..a3e59178 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/wait-agent.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# wait-agent.sh SSH_TARGET VERSION [MINUTES] — wait until the box's agent says VERSION (a read each 30 s). +T=$1; V=$2; M=${3:-22} +end=$(( $(date +%s) + M*60 )) +while [ "$(date +%s)" -lt "$end" ]; do + cur=$(timeout 15 ssh -o BatchMode=yes -o ConnectTimeout=8 "$T" 'felhom-agent --version' 2>/dev/null | awk '{print $2}') + if [ "$cur" = "$V" ]; then echo "$(date +%T) $T agent $cur"; exit 0; fi + sleep 30 +done +echo "$(date +%T) $T TIMEOUT, agent is ${cur:-unreadable}"; exit 1 diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/wait-bundle.sh b/documentation/audits/release-2b-3-2026-11-15/part2/wait-bundle.sh new file mode 100644 index 00000000..b4dae9cb --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/wait-bundle.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# wait-bundle.sh SSH_TARGET VERSION [MINUTES] — wait until the box's journal says the config bundle VERSION is installed. +T=$1; V=$2; M=${3:-22} +end=$(( $(date +%s) + M*60 )) +while [ "$(date +%s)" -lt "$end" ]; do + n=$(timeout 20 ssh -o BatchMode=yes -o ConnectTimeout=8 "$T" "journalctl -u felhom-agent --since '-60min' --no-pager | grep 'config bundle INSTALLED' | grep -c 'agent_version=$V'" 2>/dev/null) + if [ "${n:-0}" -ge 1 ]; then echo "$(date +%T) $T bundle $V installed"; exit 0; fi + sleep 30 +done +echo "$(date +%T) $T TIMEOUT"; exit 1 diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/wait-hublog.sh b/documentation/audits/release-2b-3-2026-11-15/part2/wait-hublog.sh new file mode 100644 index 00000000..6e9f63cd --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/wait-hublog.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# wait-hublog.sh 'REGEX' SINCE(e.g. 5m) [MINUTES] — wait for a hub log line (a read each 20 s), print the matches. +R=$1; SINCE=$2; M=${3:-20} +end=$(( $(date +%s) + M*60 )) +while [ "$(date +%s)" -lt "$end" ]; do + out=$(sudo kubectl -n felhom-system logs deploy/hub --since="$SINCE" 2>/dev/null | grep -iE "$R") + if [ -n "$out" ]; then echo "$(date +%T) found:"; echo "$out" | cut -c1-320; exit 0; fi + sleep 20 +done +echo "$(date +%T) TIMEOUT: no line for $R"; exit 1 diff --git a/documentation/audits/release-2b-3-2026-11-15/part2/wait-moved.sh b/documentation/audits/release-2b-3-2026-11-15/part2/wait-moved.sh new file mode 100644 index 00000000..66adda6b --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part2/wait-moved.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# wait-moved.sh SSH_TARGET [MINUTES] — wait until the box's agent config holds a ca_file (a read each 30 s). +T=$1; M=${2:-25} +end=$(( $(date +%s) + M*60 )) +while [ "$(date +%s)" -lt "$end" ]; do + cur=$(timeout 15 ssh -o BatchMode=yes -o ConnectTimeout=8 "$T" "python3 -c 'import json;t=json.load(open(\"/etc/felhom-agent/agent.json\")).get(\"proxmox\",{}).get(\"tls\",{});print(\"ca=\"+t.get(\"ca_file\",\"\")+\" fp=\"+(\"set\" if t.get(\"fingerprint\") else \"empty\"))'" 2>/dev/null) + case "$cur" in ca=/*) echo "$(date +%T) $T $cur"; exit 0;; esac + sleep 30 +done +echo "$(date +%T) $T TIMEOUT: ${cur:-unreadable}"; exit 1 diff --git a/documentation/audits/release-2b-3-2026-11-15/part3/boxwalk-0353.txt b/documentation/audits/release-2b-3-2026-11-15/part3/boxwalk-0353.txt new file mode 100644 index 00000000..95e9343e --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part3/boxwalk-0353.txt @@ -0,0 +1,33 @@ +### root@192.168.0.154 2026-11-15T10:09:03Z +containers=10 unhealthy=0 restarting=0 exited=[] +gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 43 seconds (healthy) +image file: gitea.dooplex.hu/admin/felhom-controller:0.353.0 +image digest: sha256:4be0410d3bd5e0db2e98ab2a6f386a73784b72f62ea259821e8b13fb2cc51a6c repo: gitea.dooplex.hu/admin/felhom-controller@sha256:61cf2a8bfb82e02694bf85cca774bce2cbb9f947d395f91af7640aa5a5d944f0 +started=2026-11-15T10:08:21.35521743Z +go in the running binary: 2 go1.27.4 +errors=0 warn=0 info=132 panics=0 +2026/11/15 10:08:21 [INFO] felhom-controller 0.353.0 starting (customer: tester-1, domain: enkicsifelhom.hu) +2026/11/15 10:08:22 [INFO] [selfupdate] Post-update startup: update successful (0.352.1 → 0.353.0) +### demo-hp 2026-11-15T10:09:05Z +containers=26 unhealthy=0 restarting=0 exited=[] +gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 42 seconds (healthy) +image file: gitea.dooplex.hu/admin/felhom-controller:0.353.0 +image digest: sha256:4be0410d3bd5e0db2e98ab2a6f386a73784b72f62ea259821e8b13fb2cc51a6c repo: gitea.dooplex.hu/admin/felhom-controller@sha256:61cf2a8bfb82e02694bf85cca774bce2cbb9f947d395f91af7640aa5a5d944f0 +started=2026-11-15T10:08:23.282656164Z +go in the running binary: 2 go1.27.4 +errors=0 warn=3 info=140 panics=0 +2026/11/15 10:08:24 main.go:335: [INFO] felhom-controller 0.353.0 starting (customer: demo-hp, domain: enkisfelhom.hu) +2026/11/15 10:08:24 deploy.go:1470: [WARN] [stacks] Stack plex: new field PLEX_CLAIM (type=text) requires manual configuration +2026/11/15 10:08:24 deploy.go:1470: [WARN] [stacks] Stack calibre-web: new field ADMIN_PASSWORD (type=password) requires manual configuration +2026/11/15 10:08:24 deploy.go:1470: [WARN] [stacks] Stack bookstack: new field ADMIN_PASSWORD (type=password) requires manual configuration +2026/11/15 10:08:24 updater.go:855: [INFO] [selfupdate] Post-update startup: update successful (0.352.1 → 0.353.0) +### felhom-pve 2026-11-15T10:09:06Z +containers=9 unhealthy=0 restarting=0 exited=[] +gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 47 seconds (healthy) +image file: gitea.dooplex.hu/admin/felhom-controller:0.353.0 +image digest: sha256:4be0410d3bd5e0db2e98ab2a6f386a73784b72f62ea259821e8b13fb2cc51a6c repo: gitea.dooplex.hu/admin/felhom-controller@sha256:61cf2a8bfb82e02694bf85cca774bce2cbb9f947d395f91af7640aa5a5d944f0 +started=2026-11-15T10:08:20.225850218Z +go in the running binary: 2 go1.27.4 +errors=0 warn=0 info=146 panics=0 +2026/11/15 10:08:21 [INFO] felhom-controller 0.353.0 starting (customer: demo-felhom, domain: demo-felhom.eu) +2026/11/15 10:08:21 [INFO] [selfupdate] Post-update startup: update successful (0.352.1 → 0.353.0) diff --git a/documentation/audits/release-2b-3-2026-11-15/part3/clock.sh b/documentation/audits/release-2b-3-2026-11-15/part3/clock.sh new file mode 100644 index 00000000..f08a5928 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part3/clock.sh @@ -0,0 +1,31 @@ +# Tester 1 ONLY. clock.sh snap|back|right +# snap — READ: the clock, the time service, every container's id and creation time in guest 9201. +# back — stop the time service and set the clock 150 s BACK (the smaller harm: putting it right is then a +# step FORWARD, the direction a time service steps a late clock; nothing gets a stamp from the future +# while the clock is wrong, and no certificate or signed job ends early). +# right — start the time service and step the clock right. +set -u +grep -q '"tester-1' /etc/felhom-agent/agent.json 2>/dev/null || grep -q 'tester-1' /etc/felhom-agent/agent.json || { echo "REFUSED: this box is not Tester 1"; exit 2; } +snap() { + echo "## $(hostname) clock $(date '+%F %T.%N %Z' | cut -c1-27) epoch $(date +%s.%N | cut -c1-14); chrony $(systemctl is-active chrony); $(chronyc tracking 2>/dev/null | grep -E 'System time|Leap' | tr -s ' ' | tr '\n' ';')" + echo "## kernel boot_id $(cat /proc/sys/kernel/random/boot_id); btime $(awk '/^btime/ {print $2}' /proc/stat); uptime $(cut -d' ' -f1 /proc/uptime) s" + echo "## containers in 9201 (name id created):" + pct exec 9201 -- docker ps -a --format '{{.Names}} {{.ID}} {{.CreatedAt}} [{{.Status}}]' | sort | sed 's/^/ /' + echo "## count $(pct exec 9201 -- docker ps -q | wc -l) running" +} +case "${1:-}" in + snap) snap ;; + back) + busy=$(ps -eo comm | grep -cE "^(vzdump|apt-get|dpkg|proxmox-backup-)") + [ "$busy" = 0 ] || { echo "REFUSED: something is in flight"; exit 2; } + snap + systemctl stop chrony; echo "chrony stopped rc=$?" + date -s "@$(( $(date +%s) - 150 ))" >/dev/null; echo "clock set 150 s back rc=$? → $(date '+%F %T')" + ;; + right) + systemctl start chrony; echo "chrony started rc=$?" + sleep 8; chronyc makestep; sleep 6 + echo "clock now $(date '+%F %T'); $(chronyc tracking | grep -E 'System time|Leap' | tr -s ' ' | tr '\n' ';')" + ;; + *) echo "usage: snap|back|right"; exit 2 ;; +esac diff --git a/documentation/audits/release-2b-3-2026-11-15/part3/hsts-headers.txt b/documentation/audits/release-2b-3-2026-11-15/part3/hsts-headers.txt new file mode 100644 index 00000000..6ed70cc1 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part3/hsts-headers.txt @@ -0,0 +1,2 @@ +names read: 26; header exactly 'max-age=31536000': 26 +control (a name that is not a box): felhom.eu -> [] diff --git a/documentation/audits/release-2b-3-2026-11-15/part3/hsts.sh b/documentation/audits/release-2b-3-2026-11-15/part3/hsts.sh new file mode 100644 index 00000000..17ae58ba --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part3/hsts.sh @@ -0,0 +1,13 @@ +#!/bin/bash +# hsts.sh HTTPS-ONLY-OUTPUT — read the Strict-Transport-Security header of every public name the https-only check +# walked (a GET of each name's https address; reads only). One line per name; a tally at the end. +names=$(grep -oE "https://[a-z0-9.-]+" "$1" | grep -v "no-such-name" | sort -u) +n=0; ok=0 +for u in $names; do + h=$(curl -s -o /dev/null -D - --max-time 15 "$u/" | tr -d '\r' | grep -i '^strict-transport-security:' | head -1) + n=$((n+1)) + v=$(echo "$h" | cut -d: -f2- | sed 's/^ *//') + if [ "$v" = "max-age=31536000" ]; then ok=$((ok+1)); else echo " NOT one year: $u -> [${v:-no header}]"; fi +done +echo "names read: $n; header exactly 'max-age=31536000': $ok" +echo "control (a name that is not a box): felhom.eu -> [$(curl -s -o /dev/null -D - --max-time 15 https://felhom.eu/ | tr -d '\r' | grep -i '^strict-transport-security:' | cut -d: -f2- | sed 's/^ *//')]" diff --git a/documentation/audits/release-2b-3-2026-11-15/part3/https-only.txt b/documentation/audits/release-2b-3-2026-11-15/part3/https-only.txt new file mode 100644 index 00000000..dcd31fd2 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part3/https-only.txt @@ -0,0 +1,34 @@ +control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -) +== tester-1 (enkicsifelhom.hu): 6 public name(s) + OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1 + OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1 + OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 + OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1 + OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1 + OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1 +== demo-hp (enkisfelhom.hu): 15 public name(s) + OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1 + OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1 + OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1 + OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1 + OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1 + OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1 + OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1 + OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 + OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1 + OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1 + OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1 + OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1 + OK http://time.enkisfelhom.hu/r1033-check?q=1 -> 301 https://time.enkisfelhom.hu/r1033-check?q=1 + OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1 + OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1 +== demo-felhom (demo-felhom.eu): 8 public name(s) + OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1 + OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1 + OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1 + OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 + OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1 + OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1 + OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1 + OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1 +== every public name redirects http -> https diff --git a/documentation/audits/release-2b-3-2026-11-15/part3/hub-system-page-after.txt b/documentation/audits/release-2b-3-2026-11-15/part3/hub-system-page-after.txt new file mode 100644 index 00000000..ab624d00 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part3/hub-system-page-after.txt @@ -0,0 +1,257 @@ +System — Felhom Hub +Felhom Hub +Dashboard +Customers +Apps +Hosts +System +Offsite +Configuration +System — versions and OS updates +Needs attention +One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3). +[hover: worth a look] look +Tester4-2f0c25 Tester4 +partial: no running customer guest · the host needs a restart since 2026-11-15 · crash guard not installed · [hover: the box could not read these values (agent older than v0.142.0, or the guest is down) — never a guess] could not read: running kernel, next-boot kernel, default kernel, host Debian, held packages, kernel.panic, crash restarts, guest Debian, Docker engine, containerd, Docker live-restore +Waiting for you +Update sets that only you approve: the kernel, the Docker engine and the Proxmox packages. Guest and host Debian updates approve themselves after 24 h and one night on the [hover: Ring 0 = the demo boxes. They install every new fix first; the other boxes (ring 1) install only what is approved.] ring-0 boxes. +Docker engine 29.9.3-1 +Ran on every ring-0 box — demo-felhom-8363b5: 2 healthy night(s), demo-hp-bb76ea: 2 healthy night(s). The memory-kill check passed. +first seen 2 days ago (2026-11-13 04:49 UTC). Approving installs nothing by itself: a ring-1 box takes it only through a signed operator job. +Approve Docker set +Proxmox packages 9.2.27 — still being tested: demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set +Boxes +Click a box to see every value it reports, and its ring and update switches. "unknown": the box could not read the value — never a guess. +Box Ring Health Controller · agent OS updates Kernel [hover: The newest OS update run] Last night +Tester4-2f0c25 Tester4 +[hover: Ring 1: a normal box — it installs only approved releases] 1 +[hover: worth a look] look +0.353.0 [hover: current (vouched 0.175.1)] agent 0.175.1 +[hover: runs the newest approved guest and host releases] on · up to date +unknown +[hover: 1 min ago · applied · 89 s — last successful leg: 3 min ago] ok · 1 min ago +partial: no running customer guest +Ring and updates +Ring ring 1 +→ demo +OS updates updates ON +switch off +Tunnel [hover: no cloudflared container in the guest (agent state: absent)] none set up +Controller 0.353.0 +Agent [hover: current (vouched 0.175.1)] 0.175.1 +Last OS leg [hover: last successful leg: 3 min ago] 1 min ago · applied · 89 s +Docker engine +Docker [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown +containerd [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown +live-restore [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown +Docker release — +Host +Proxmox 9.2.2 +Kernel (running) [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown +Kernel (next boot) [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown +[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown +[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: the box reports no kernel lane (agent older than v0.152.0)] — +[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] — +[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: refreshed] 7.0.2-6-pve · 1 rewritten · 13 min ago +Debian [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown +Felhom release os-host-20261115-045718 +Pending 99 +Not covered 0 +Held [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown +Reboot needed since 2026-11-15 +kernel.panic [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown +Oops no +Crash restarts 24 h [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown +Crash guard not installed +[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.175.1 +Guest +Guest Debian [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown +Felhom release os-guest-20261111-034654 +Pending 6 +Restart needed 7 +[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 8 min ago · 78.1 GiB +[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] not yet +demo-felhom-8363b5 Demo Ügyfél +[hover: Ring 0: a demo box — it installs every new fix first] 0 +[hover: nothing amber or red] fine +0.353.0 [hover: current (vouched 0.175.1)] agent 0.175.1 +[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind +7.0.14-28-pve +[hover: 6 h ago · nothing · 23 s — last successful leg: 6 h ago] ok · 6 h ago +Ring and updates +Ring ring 0 +→ normal +OS updates updates ON +switch off +Tunnel running +Controller 0.353.0 +Agent [hover: current (vouched 0.175.1)] 0.175.1 +Last OS leg [hover: last successful leg: 6 h ago] 6 h ago · nothing · 23 s +Docker engine +Docker 29.9.3 +containerd 2.4.2-4~debian.13~trixie +live-restore on +Docker release ring0-20261115T045634Z +Host +Proxmox 9.2.27 +Kernel (running) 7.0.14-28-pve +Kernel (next boot) 7.0.14-28-pve +[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve +[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 1m14s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 7 days ago · due 7.0.14-29-pve — household told 2 h ago: TONIGHT · phase good +[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: removed 6 old kernel package(s): proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed; kept 7.0.14-26-pve, 7.0.14-28-pve; 6.3 GiB gained; 0 left-over path(s)] cleaned 6 h ago +[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: not_needed] 7.0.14-28-pve · 23 h ago +Debian 13.7 +Felhom release ring0-20261115T045634Z +Pending 29 +Not covered 0 +Held none +Reboot needed no +kernel.panic 10 s +Oops no +Crash restarts 24 h 0 +Crash guard armed +[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.175.1 +Guest +Guest Debian 13.7 +Felhom release ring0-20261115T045634Z +Pending 0 +Restart needed 1 +[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 4 days ago · 272.7 GiB +[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 92.7 Mbit/s · 5 days ago +demo-hp-bb76ea Demo HP +[hover: Ring 0: a demo box — it installs every new fix first] 0 +[hover: nothing amber or red] fine +0.353.0 [hover: current (vouched 0.175.1)] agent 0.175.1 +[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind +7.0.14-28-pve +[hover: 6 h ago · nothing · 30 s — last successful leg: 6 h ago] ok · 6 h ago +Ring and updates +Ring ring 0 +→ normal +OS updates updates ON +switch off +Tunnel running +Controller 0.353.0 +Agent [hover: current (vouched 0.175.1)] 0.175.1 +Last OS leg [hover: last successful leg: 6 h ago] 6 h ago · nothing · 30 s +Docker engine +Docker 29.9.3 +containerd 2.4.2-4~debian.13~trixie +live-restore on +Docker release ring0-20261115T040248Z +Host +Proxmox 9.2.27 +Kernel (running) 7.0.14-28-pve +Kernel (next boot) 7.0.14-28-pve +[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve +[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 2m39s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 5 days ago · due 7.0.14-29-pve — household told 2 h ago: TONIGHT · phase good +[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] — +[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: not_needed] 7.0.14-28-pve · 23 h ago +Debian 13.7 +Felhom release ring0-20261115T040248Z +Pending 30 +Not covered 0 +Held none +Reboot needed no +kernel.panic 10 s +Oops no +Crash restarts 24 h 0 +Crash guard armed +[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.175.1 +Guest +Guest Debian 13.7 +Felhom release ring0-20261115T040248Z +Pending 0 +Restart needed 1 +[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 4 days ago · 70.5 GiB +[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 94.3 Mbit/s · 5 days ago +tester-1-d70be4 Tester 1 +[hover: Ring 1: a normal box — it installs only approved releases] 1 +[hover: nothing amber or red] fine +0.353.0 [hover: current (vouched 0.175.1)] agent 0.175.1 +[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind +7.0.14-22-pve +[hover: 6 h ago · nothing · 16 s — last successful leg: 6 h ago] ok · 6 h ago +Ring and updates +Ring ring 1 +→ demo +OS updates updates ON +switch off +Tunnel running +Controller 0.353.0 +Agent [hover: current (vouched 0.175.1)] 0.175.1 +Last OS leg [hover: last successful leg: 6 h ago] 6 h ago · nothing · 16 s +Docker engine +Docker 29.8.2 +containerd 2.3.6-1~debian.13~trixie +live-restore on +Docker release — +Host +Proxmox 9.2.2 +Kernel (running) 7.0.14-22-pve +Kernel (next boot) 7.0.14-22-pve +[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-22-pve +[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: ring 1: no kernel staged by a signed os_kernel_step healthy 1m17s after the agent started; the new kernel is the default] 7.0.14-22-pve applied 38 days ago · phase good +[hover: The newest clean-up of old kernels (R-1368): how many it removed and kept, and the space gained. A box reports one only when it removed kernels, was refused, or failed.] Kernel clean-up [hover: no clean-up of old kernels reported (a box reports one only when it removed kernels, was refused, or failed)] — +[hover: The box's last refresh of its compiled security (AppArmor) profiles (R-1388): the kernel it ran on, how many changed, and a failed service it restarted.] Security profiles [hover: last ask: not_needed] 7.0.14-22-pve · 23 h ago +Debian 13.7 +Felhom release os-host-20261111-034654 +Pending 101 +Not covered 0 +Held none +Reboot needed no +kernel.panic 10 s +Oops no +Crash restarts 24 h 0 +Crash guard armed +[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.175.1 +Guest +Guest Debian 13.7 +Felhom release os-guest-20261111-034654 +Pending 6 +Restart needed 0 +[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 4 days ago · 90.4 GiB +[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 93.0 Mbit/s · 4 days ago +Details +Release ids, cancelled approvals, what ring 0 runs, the version floors, and every box's crash guard and root files. +Approved releases +The newest approved set of each layer. A [hover: Approved while a TEST wait override was active. It is cancelled when the hub starts without the override (`11` §5.3.1).] TEST approval is amber. +guest : os-guest-20261111-034654 +272 packages · 2026-11-11 03:46 UTC · by auto +host : os-host-20261115-045718 +607 packages · 2026-11-15 04:57 UTC · by auto +docker : os-docker-20261026-190351 +6 packages · 2026-10-26 19:03 UTC · by operator +pve : os-pve-20261114-075712 +121 packages · 2026-11-14 07:57 UTC · by operator +kernel : os-kernel-20261114-075716 +2 packages · 2026-11-14 07:57 UTC · by operator +Approve now (guest + host) +An urgent fix only — normally the hub approves after 24 h and one night. +What ring 0 runs now +guest : +272 packages, first seen 2026-11-10 01:49 UTC +approved as os-guest-20261111-034654 +host : +607 packages, first seen 2026-11-14 04:52 UTC +approved as os-host-20261115-045718 +docker : +6 packages, first seen 2026-11-13 04:49 UTC +ready — see "Waiting for you" +pve : +121 packages, first seen 2026-11-15 04:58 UTC +demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set +kernel : +2 packages, first seen 2026-11-10 01:57 UTC +approved as os-kernel-20261114-075716 +Version floors +Global controller floor: 0.353.0 · vouched agent: 0.175.1 +[hover: A floor is the lowest controller version a box must run; the hub moves a box below it up.] What is a floor? · [hover: The agent version the operator checked and signed off for the fleet. Agents update only by a per-box signed job.] What is a vouched agent? +No per-customer floors: every box follows the global floor. +Crash guard and root files +Box ¦ kernel.panic ¦ Oops ¦ Crash restarts 24 h ¦ Crash guard ¦ [hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files ¦ +Tester4-2f0c25 ¦ [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown ¦ no ¦ [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown ¦ not installed ¦ 0.175.1 ¦ +demo-felhom-8363b5 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.175.1 ¦ +demo-hp-bb76ea ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.175.1 ¦ +tester-1-d70be4 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.175.1 ¦ +Felhom Hub 0.182.0 diff --git a/documentation/audits/release-2b-3-2026-11-15/part3/hub-webcerts.sh b/documentation/audits/release-2b-3-2026-11-15/part3/hub-webcerts.sh new file mode 100644 index 00000000..d976a898 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part3/hub-webcerts.sh @@ -0,0 +1,34 @@ +#!/bin/bash +# Read from the LIVE hub database (the three files in one tar, as the note says), then shred the copy: +# does the newest report of each test household carry web_certs? Also the daily ledger and the floors. +set -u +W=/mnt/5_hdd/felhom.eu/evidence/release-2b-3-2026-11-15/dbcopy; mkdir -p "$W"; chmod 700 "$W" +sudo kubectl -n felhom-system exec deploy/hub -- sh -c 'cd /data && tar cf - hub.db hub.db-wal hub.db-shm' > "$W/db.tar" +( cd "$W" && tar xf db.tar ) +python3 -I - "$W/hub.db" <<'PY' +import sqlite3, sys, json +db = sqlite3.connect(sys.argv[1]) +tabs = [r[0] for r in db.execute("SELECT name FROM sqlite_master WHERE type='table'")] +rt = "reports" if "reports" in tabs else [t for t in tabs if "report" in t][0] +cols = [r[1] for r in db.execute("PRAGMA table_info(%s)" % rt)] +print("report table:", rt, "columns:", ",".join(cols)[:200]) +tcol = [c for c in cols if c in ("received_at", "reported_at", "created_at")][0] +jcol = [c for c in cols if c in ("data", "report", "payload", "body", "json", "report_json")] +print("freshness control: newest report", db.execute("SELECT MAX(%s) FROM %s" % (tcol, rt)).fetchone()[0]) +for cust in ("tester-1", "demo-hp", "demo-felhom"): + row = db.execute("SELECT %s, %s FROM %s WHERE customer_id=? ORDER BY %s DESC LIMIT 1" % (tcol, jcol[0], rt, tcol), (cust,)).fetchone() + d = json.loads(row[1]) + wc = d.get("web_certs") + def find(o): + if isinstance(o, dict): + if "web_certs" in o: return o["web_certs"] + for v in o.values(): + r = find(v) + if r is not None: return r + return None + wc = wc if wc is not None else find(d) + print(cust, "report at", row[0], "controller", d.get("controller_version") or d.get("version"), "web_certs:", json.dumps(wc)[:300]) +for k, v in db.execute("SELECT key, substr(value,1,120) FROM hub_settings WHERE key LIKE 'daily_ledger%' OR key LIKE 'os_alarm:%' OR key LIKE 'min_controller%' ORDER BY key"): + print("setting", k, "=", v) +PY +find "$W" -type f -exec shred -u {} \; ; rmdir "$W"; echo "copy shredded: $([ -e "$W" ] && echo NO || echo yes)" diff --git a/documentation/audits/release-2b-3-2026-11-15/part3/hub-webcerts.txt b/documentation/audits/release-2b-3-2026-11-15/part3/hub-webcerts.txt new file mode 100644 index 00000000..473db4c2 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part3/hub-webcerts.txt @@ -0,0 +1,20 @@ +report table: reports columns: id,customer_id,received_at,report_json,health_status,cpu_percent,memory_percent,container_total,container_running,backup_last_snapshot,controller_version,controller_url,language +freshness control: newest report 2026-11-15 10:08:31 +tester-1 report at 2026-11-15 10:08:28 controller 0.353.0 web_certs: [{"domain": "*.enkicsifelhom.hu", "not_after": "2027-01-02T18:44:06Z", "issuer": "YR1"}] +demo-hp report at 2026-11-15 10:08:31 controller 0.353.0 web_certs: [{"domain": "*.enkisfelhom.hu", "not_after": "2027-01-19T01:46:00Z", "issuer": "YR2"}] +demo-felhom report at 2026-11-15 10:08:27 controller 0.353.0 web_certs: [{"domain": "*.demo-felhom.eu", "not_after": "2026-12-31T18:52:10Z", "issuer": "YR1"}] +setting min_controller_version = 0.352.1 +setting min_controller_version_declared_min_agent = 0.352.1=0.158.1 +setting os_alarm:agent:Tester-2-be8404 = 2026-10-19T10:30:41Z +setting os_alarm:bundle:demo-felhom-8363b5 = +setting os_alarm:bundle:demo-hp-bb76ea = +setting os_alarm:bundle:tester-1-d70be4 = +setting os_alarm:daily_last:deadline-check = 2026-11-15T04:00:00Z +setting os_alarm:daily_last:offsite-key-audit = 2026-11-15T06:10:00Z +setting os_alarm:end_date:baseline = 2026-11-15T08:50:09Z +setting os_alarm:offsite_escrow_pending:Tester-2 = 2026-10-18T16:14:28Z +setting os_alarm:operator_mail_lost_scan = 2026-11-15T08:48:59Z +setting os_alarm:reboot:Tester-2-be8404 = 2026-10-18T16:28:55Z +setting os_alarm:stale:Tester-2-be8404 = 2026-10-18T17:28:55Z +setting os_alarm:stale:demo-hp-bb76ea = +copy shredded: yes diff --git a/documentation/audits/release-2b-3-2026-11-15/part3/per-box-floors-cleared.txt b/documentation/audits/release-2b-3-2026-11-15/part3/per-box-floors-cleared.txt new file mode 100644 index 00000000..b9820aa2 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part3/per-box-floors-cleared.txt @@ -0,0 +1,9 @@ +== /customers/tester-1/floor +HTTP/1.1 303 See Other +Location: /customers/tester-1?flash=floor_set +== /customers/demo-hp/floor +HTTP/1.1 303 See Other +Location: /customers/demo-hp?flash=floor_set +== /customers/demo-felhom/floor +HTTP/1.1 303 See Other +Location: /customers/demo-felhom?flash=floor_set diff --git a/documentation/audits/release-2b-3-2026-11-15/part3/t1-after-reboot.txt b/documentation/audits/release-2b-3-2026-11-15/part3/t1-after-reboot.txt new file mode 100644 index 00000000..644c5815 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part3/t1-after-reboot.txt @@ -0,0 +1,19 @@ +## felhom clock 2026-11-15 11:26:38.0075053 epoch 1794738398.009; chrony active; System time : 0.002849487 seconds slow of NTP time;Leap status : Normal; +## kernel boot_id 0581d703-3fae-47e4-8598-ee462aa3d516; btime 1794738245; uptime 152.48 s +## containers in 9201 (name id created): + bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 2 minutes (healthy)] + bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 2 minutes (healthy)] + cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 2 minutes (healthy)] + felhom-controller 9f210e5a321f 2026-11-15 10:24:27 +0000 UTC [Up 2 minutes (healthy)] + filebrowser c3b5388d9c1e 2026-11-15 10:24:52 +0000 UTC [Up About a minute (healthy)] + paperless-postgres 913e7c4090c1 2026-11-15 10:24:41 +0000 UTC [Up About a minute (healthy)] + paperless-redis 5024d8448930 2026-11-15 10:24:41 +0000 UTC [Up About a minute (healthy)] + paperless-webserver 3e7d16433e63 2026-11-15 10:24:41 +0000 UTC [Up About a minute (healthy)] + privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 2 minutes (healthy)] + traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 2 minutes] +## count 10 running +2026/11/15 10:24:30 [INFO] [gate] boot 0581d703-3fae-47e4-8598-ee462aa3d516-1535: waiting (≤2m0s) for live drive bind(s) [/mnt/felhom-drives/adatlemez] before recreating drive-backed apps +2026/11/15 10:24:40 [INFO] [gate] boot 0581d703-3fae-47e4-8598-ee462aa3d516-1535: live bind confirmed — recreating drive-backed app paperless-ngx (state=degraded) onto /mnt/felhom-drives/adatlemez +2026/11/15 10:24:52 [INFO] [gate] boot 0581d703-3fae-47e4-8598-ee462aa3d516-1535: re-syncing FileBrowser mounts against the live binds +felhom-agent 0.175.1 +0 diff --git a/documentation/audits/release-2b-3-2026-11-15/part3/t1-clock-B-2min.txt b/documentation/audits/release-2b-3-2026-11-15/part3/t1-clock-B-2min.txt new file mode 100644 index 00000000..f2466bf4 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part3/t1-clock-B-2min.txt @@ -0,0 +1,14 @@ +## felhom clock 2026-11-15 11:10:47.4544935 epoch 1794737447.456; chrony inactive; +## kernel boot_id 7a721957-7e95-476c-be40-52d0dafc7284; btime 1794665801; uptime 71646.26 s +## containers in 9201 (name id created): + bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 5 hours (healthy)] + bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 5 hours (healthy)] + cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 20 hours (healthy)] + felhom-controller 09343ef712fa 2026-11-15 10:08:21 +0000 UTC [Up 2 minutes (healthy)] + filebrowser 95f38571abee 2026-11-14 14:19:57 +0000 UTC [Up 20 hours (healthy)] + paperless-postgres 31b2490aa677 2026-11-15 10:05:37 +0000 UTC [Up 5 minutes (healthy)] + paperless-redis df1c44607a9b 2026-11-15 10:05:37 +0000 UTC [Up 5 minutes (healthy)] + paperless-webserver 1f2a0d7e4f7b 2026-11-15 10:05:37 +0000 UTC [Up 5 minutes (healthy)] + privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 5 hours (healthy)] + traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 20 hours] +## count 10 running diff --git a/documentation/audits/release-2b-3-2026-11-15/part3/t1-clock-B-after-right.txt b/documentation/audits/release-2b-3-2026-11-15/part3/t1-clock-B-after-right.txt new file mode 100644 index 00000000..0f2c55e7 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part3/t1-clock-B-after-right.txt @@ -0,0 +1,14 @@ +## felhom clock 2026-11-15 11:21:49.3394920 epoch 1794738109.341; chrony active; System time : 0.001603887 seconds slow of NTP time;Leap status : Normal; +## kernel boot_id 7a721957-7e95-476c-be40-52d0dafc7284; btime 1794665951; uptime 72157.81 s +## containers in 9201 (name id created): + bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 6 hours (healthy)] + bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 6 hours (healthy)] + cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 20 hours (healthy)] + felhom-controller 09343ef712fa 2026-11-15 10:08:21 +0000 UTC [Up 13 minutes (healthy)] + filebrowser 95f38571abee 2026-11-14 14:19:57 +0000 UTC [Up 20 hours (healthy)] + paperless-postgres 31b2490aa677 2026-11-15 10:05:37 +0000 UTC [Up 16 minutes (healthy)] + paperless-redis df1c44607a9b 2026-11-15 10:05:37 +0000 UTC [Up 16 minutes (healthy)] + paperless-webserver 1f2a0d7e4f7b 2026-11-15 10:05:37 +0000 UTC [Up 16 minutes (healthy)] + privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 6 hours (healthy)] + traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 20 hours] +## count 10 running diff --git a/documentation/audits/release-2b-3-2026-11-15/part3/t1-clock-B-back.txt b/documentation/audits/release-2b-3-2026-11-15/part3/t1-clock-B-back.txt new file mode 100644 index 00000000..bcecb4fc --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part3/t1-clock-B-back.txt @@ -0,0 +1,16 @@ +## felhom clock 2026-11-15 11:11:17.4153213 epoch 1794737477.417; chrony active; System time : 0.000710471 seconds slow of NTP time;Leap status : Normal; +## kernel boot_id 7a721957-7e95-476c-be40-52d0dafc7284; btime 1794665951; uptime 71525.88 s +## containers in 9201 (name id created): + bookstack cc420862d9db 2026-11-15 04:50:03 +0000 UTC [Up 5 hours (healthy)] + bookstack-db e8df089ab4ed 2026-11-15 04:50:02 +0000 UTC [Up 5 hours (healthy)] + cloudflared 45d7a4eaa914 2026-10-27 15:41:50 +0000 UTC [Up 20 hours (healthy)] + felhom-controller 09343ef712fa 2026-11-15 10:08:21 +0000 UTC [Up 2 minutes (healthy)] + filebrowser 95f38571abee 2026-11-14 14:19:57 +0000 UTC [Up 20 hours (healthy)] + paperless-postgres 31b2490aa677 2026-11-15 10:05:37 +0000 UTC [Up 5 minutes (healthy)] + paperless-redis df1c44607a9b 2026-11-15 10:05:37 +0000 UTC [Up 5 minutes (healthy)] + paperless-webserver 1f2a0d7e4f7b 2026-11-15 10:05:37 +0000 UTC [Up 5 minutes (healthy)] + privatebin b44d977950cb 2026-11-15 04:50:21 +0000 UTC [Up 5 hours (healthy)] + traefik ea1383c240f1 2026-11-03 07:04:43 +0000 UTC [Up 20 hours] +## count 10 running +chrony stopped rc=0 +clock set 150 s back rc=0 → 2026-11-15 11:08:49 diff --git a/documentation/audits/release-2b-3-2026-11-15/part3/t1-clock-B-right.txt b/documentation/audits/release-2b-3-2026-11-15/part3/t1-clock-B-right.txt new file mode 100644 index 00000000..91f428b0 --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part3/t1-clock-B-right.txt @@ -0,0 +1,4 @@ +chrony started rc=0 +200 OK +clock now 2026-11-15 11:20:27; System time : 0.000000000 seconds fast of NTP time;Leap status : Normal; +Tester 1 minus DooPlex: +0.071 s diff --git a/documentation/audits/release-2b-3-2026-11-15/part3/wait-ctl.sh b/documentation/audits/release-2b-3-2026-11-15/part3/wait-ctl.sh new file mode 100644 index 00000000..1b4e1a1c --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part3/wait-ctl.sh @@ -0,0 +1,15 @@ +#!/bin/bash +# wait-ctl.sh VERSION [MINUTES] — wait until guest 9201 on the three test boxes runs controller VERSION (reads). +V=$1; M=${2:-25} +end=$(( $(date +%s) + M*60 )) +while [ "$(date +%s)" -lt "$end" ]; do + ok=0; line="" + for h in root@192.168.0.154 demo-hp felhom-pve; do + img=$(timeout 20 ssh -o BatchMode=yes -o ConnectTimeout=8 "$h" "pct exec 9201 -- docker ps --filter name=^felhom-controller$ --format '{{.Image}} {{.Status}}'" 2>/dev/null) + line="$line | $h: ${img##*:}" + case "$img" in *":$V "*healthy*) ok=$((ok+1));; esac + done + if [ "$ok" = 3 ]; then echo "$(date +%T) all three$line"; exit 0; fi + sleep 30 +done +echo "$(date +%T) TIMEOUT$line"; exit 1 diff --git a/documentation/audits/release-2b-3-2026-11-15/part3/watch-boxes.txt b/documentation/audits/release-2b-3-2026-11-15/part3/watch-boxes.txt new file mode 100644 index 00000000..a61d2baf --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part3/watch-boxes.txt @@ -0,0 +1,16 @@ +### root@192.168.0.154 2026-11-15T10:51:31Z +containers=10 unhealthy=0 restarting=0 exited=[] +gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 27 minutes (healthy) +image file: gitea.dooplex.hu/admin/felhom-controller:0.353.0 +errors=0 warn=1 info=635 panics=0 +### demo-hp 2026-11-15T10:51:33Z +containers=26 unhealthy=0 restarting=0 exited=[] +gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 43 minutes (healthy) +image file: gitea.dooplex.hu/admin/felhom-controller:0.353.0 +errors=0 warn=3 info=907 panics=0 +### felhom-pve 2026-11-15T10:51:35Z +containers=9 unhealthy=0 restarting=0 exited=[] +gitea.dooplex.hu/admin/felhom-controller:0.353.0 Up 43 minutes (healthy) +image file: gitea.dooplex.hu/admin/felhom-controller:0.353.0 +errors=0 warn=0 info=856 panics=0 +9202: gitea.dooplex.hu/admin/felhom-controller:0.353.0-rc.1 Up About an hour (healthy) diff --git a/documentation/audits/release-2b-3-2026-11-15/part3/watch-hub.txt b/documentation/audits/release-2b-3-2026-11-15/part3/watch-hub.txt new file mode 100644 index 00000000..ca1f276b --- /dev/null +++ b/documentation/audits/release-2b-3-2026-11-15/part3/watch-hub.txt @@ -0,0 +1,5 @@ +2026/11/15 11:26:53 [INFO] Global controller-version floor set to "0.353.0" (declared MinAgent "0.158.1") +2026/11/15 11:26:53 [INFO] access token "cc-releases-2": POST /configuration/global-floor → 303 +2026/11/15 11:46:10 [WARN] slow request: POST /api/v1/host-enroll 201 11241ms (over 5s) +2026/11/15 11:46:32 [WARN] escrow CARRIED for Tester4: the sealed package of deleted host Tester4-46836f is now the current package of Tester4-2f0c25 (repo-key hash 5501e4bf2cda…); the box will ask for the recovery cod +2026/11/15 11:48:03 [INFO] managed floor SERVED for Tester4: floor 0.353.0, agent requirement "0.158.1" from declared (golden 0.352.1) diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index 45bf7f35..a518071d 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -26,6 +26,18 @@ --- +## 2026-11-15 — releases 2b and 3: R-1380, R-1395, R-1050 closed (full text of the rows: `git show f561c85b:documentation/backlog/OPEN-ITEMS.md`) + +| Row | What | Closed | Evidence | +|---|---|---|---| +| **R-1380** | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** (P3) | CLOSED 2026-11-15 — agent 0.175.1 (vouched 11:07:35 CET), installer 1.37.0 (tagged and published), golden 0.353.0 (vouched). Tester 1, demo-hp and the N100 moved from the pinned certificate to the copy of their own root CA: each proved a working call with the copy first and cleared the fingerprint last. On demo-hp two forced new Proxmox certificates changed nothing for the agent: the same process, no restart, 14 and 40 calls answered 200 in Proxmox's own log, no alarm. A wrong CA copy was repaired at the agent's start. A fresh install ended with the copy and no fingerprint. A cut write of the agent's config is repaired at start (tests with a cut at four steps; NOT proven with a real power cut). What stays open is R-1396 gap 2 (a fully blind box) | `audits/release-2b-3-2026-11-15/` `part0/`, `part2/`; `documentation/tests/golden-0.353.0-2026-11-15/` | +| **R-1395** | **A step of the host's clock makes the controller read a new boot: it then recreates every drive-backed app, once.** (P4) | CLOSED 2026-11-15 — controller 0.353.0 (global floor 10:26:53 UTC) with agent 0.175.1. On Tester 1 the same 150 s clock step was staged twice: with the old controller it recreated one drive-backed app (three containers) at the step back and again at the step forward; with the new controller and agent it recreated nothing in either direction (the same container ids before and after). One real restart of Tester 1 after that was read as a new boot and the app was recreated, so the gate still works. On a box whose agent is older than 0.175.0 a clock step still recreates once | `audits/release-2b-3-2026-11-15/` `part2/t1-clock-A-*.txt`, `part3/t1-clock-B-*.txt`, `part3/t1-after-reboot.txt` | +| **R-1050** | **Raise HSTS to one year on or after 2026-11-15, if no https trouble was reported.** (P4) | CLOSED 2026-11-15 — controller 0.353.0, on the operator's word in the release prompt (decision 249). Every public name of the three test boxes answers `strict-transport-security: max-age=31536000` and nothing more (26 names of 26; no includeSubDomains, no preload); the https-only check is green. The night's search found no https trouble since 2026-10-15. Not tested: a browser | `audits/release-2b-3-2026-11-15/` `part3/hsts-headers.txt`, `part3/https-only.txt` | + +**Reasoning kept (R-1380).** A set fingerprint wins over a CA file, so the move clears the fingerprint LAST and only after a working call with the copy. `agent.json` is written only through `config.ReplaceInPlace` (a journal of the old bytes first; the daemon repairs a cut write before it reads the config). Root files that are older than the agent are not a trust problem on a pinned box — but that quiet path is never taken after a failed call. + +**Reasoning kept (R-1395).** The boot token is the agent's; the controller only compares. A boot is the kernel's boot id, never a time computed from the wall clock. + ## 2026-11-14 — release 2a: R-1388, R-1301, R-1391 closed (full text of the rows: `git show a2173830:documentation/backlog/OPEN-ITEMS.md`) | Row | What | Closed | Evidence | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index a4c33bdb..3f4f91a6 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -268,7 +268,6 @@ stopping line that lies. | **R-779** | Security & access | P4 | **[P3-LOW] Part A's "two outside addresses seen as two" is proven through the simulated tunnel only; on the REAL tunnel the second outside address (ep0, one request allowed) was refused by Cloudflare's edge with 403 and never reached the box.** Measured 2026-10-01 19:51 UTC (`audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt`): no log line on demo-hp; demo-hp's box has no geo restriction in its settings, so a Cloudflare ZONE rule (country or bot, not read) refused a German datacenter address. DooPlex's own address on the real tunnel was seen as itself. **Needs:** one sign-in from a second Hungarian address (the operator's phone off wifi) while DooPlex is locked out — 2 minutes; and say which Cloudflare rule refused ep0. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (a phone), CC reads the logs** **Re-ranked 2026-10-03: P3→P4: a proof gap on the real tunnel; operator-only follow-up.** | — | — | CC + operator | | **R-904** | Security & access | P4 | **Cloudflare can read every household's app traffic; replacing it with our own relay is a later item.** Facts (reviewer discussion 2026-10-08; `01`): app traffic and the dashboard reach the box through the Cloudflare Tunnel (`01` §5 trust table, rows end-user ↔ apps and customer ↔ controller UI; §7), and the tunnel's public end is Cloudflare's edge, where TLS ends — so Cloudflare can technically read that traffic (the FAQ says so since 2026-10-08, R-900; „TLS ends at the edge" is not written in `01` — add it there). What Cloudflare gives today, free: inbound reach with no router setup, the CGNAT answer (`01` §4, §7); certificates (`01` §7, the free tier covers one level below a zone); the geo-WAF the hub enforces (`01` §5 last row, §7); flood protection (not in `01`). The alternative named: our own EU relay over WireGuard with TLS passthrough by SNI, certificates on the box, the geo-block on the relay. Its costs: one more machine the operator keeps up, and a single point of reach for every box; weaker flood protection; about a week of work after a spike. Operator ruling 2026-10-08 09:07 (`09` §3 decision 184): a later item. | **DEFERRED** — **2026-10-10: the asked sentence is in `01` §7** („TLS ends at Cloudflare's edge …”); the spike stays deferred. **DEFERRED — after the first customers (operator ruling 2026-10-08 09:07)** | — | A spike after the first customers (the relay's reach, cost and flood behaviour, measured) | operator | | **R-1040** | Security & access | P4 | **Until the household makes Sonarr's or Radarr's first admin, any container on the box can read the app's API key.** Measured on the bench 2026-10-14 with the R-1030 seed in place: `AuthenticationRequired=Enabled`, `AuthenticationMethod=None` — `/initialize.json` from another container answers 200 with `apiKey`. Outsiders are blocked by the box's setup gate in that window; a compromised other app is not. | **OPEN** — filed 2026-10-14 | — | Seed the API key at install (`SONARR__AUTH__APIKEY`, measured to work) so the box knows it, and close the window by telling the household in the first steps; or seed a Forms user | CC | -| **R-1050** | Security & access | P4 | **Raise HSTS to one year on or after 2026-11-15, if no https trouble was reported.** `09` §3 decision 249 (D2 = A, operator 2026-10-15): HSTS stays `max-age=86400` now; after a month with no https trouble it becomes one year (`31536000`); never preload, never `includeSubDomains`. The value is `HSTSMaxAgeSeconds` in `felhom-controller/controller/internal/infra/infra.go`, pinned by `TestRenderForwardedHeaders_HTTPSOnly`. „Trouble” = any report of a name that will not open, or a box answering http after the floor. | NARROWED — one year is in controller `main`, not released. **2026-11-14 (night build):** searched for https trouble since 2026-10-15: the open and closed registers (ASCII fragments; controls: a common word found, a nonsense word not found), the operator's mailbox (words for „does not open”, „not secure”, certificate, HSTS, the browser's error codes: one thread, a tester's install questions, no https trouble), the hub's log since its start today (0 lines). No name that does not open, no box answering http after the floor. Not searched: the hub's log before today's deploy (not kept), Tester 2 (off, R-1104). The release must be built on or after 2026-11-15. WATCHING — **2026-11-14: NOT in controller 0.352.1.** That release was built on 2026-11-14, one day before the ruled date, and the operator's words „one day early is OK” were not given. Check date moved to 2026-11-21 (one week: the next day a controller release is likely; a one-day move would only turn the nightly CI red with nothing to release). The search for https trouble since 2026-10-15 is NOT done yet. Earlier: **WATCHING** — filed 2026-10-15 | the date | On or after 2026-11-15: read the register and the operator's mail for https trouble since 2026-10-15; none → a controller release with one year (test updated), bake → vouch → floor | CC | | **R-1054** | Security & access | P3 | **The hub's operator/global API key is still one bearer that can do everything on `/api/v1`, including reading every box's break-glass root password (`GET /admin/hosts/{id}/recovery-credential`).** Hub v0.152.0 gave the web surface scoped tokens (`05` §17) but tokens are refused on `/api/v1` by design. Who uses the global key today is not listed in one place. Fix: list its consumers; move each operator use to a scoped token on the web side (or a token class on the API), and keep the global key only where a box path needs it — or retire it. | OPEN — filed 2026-10-15 by the token build | — | List the global key's consumers (pod env, scripts, runbooks) | CC | | **R-1194** | Security & access | P3 | **On a fresh box the operator's break-glass SSH door was broken from the first minute: `oob_degraded` „felhom-sshd config invalid (sshd -t fails)".** Rehearsal 2026-10-20, Tester4-816e21, mail at 12:03 UTC (one minute after the host record was created): details `active:false, config_invalid:true, felhom_sshd_port:0, operator_key_configured:false, operator_key_reported:true, reachable:false`. Possibly the shape of a customer with no operator key set (Tester4 is a test customer) — then the alarm is a false one on every such box; or a real config defect. Not diagnosed (no shell). Checked the same day: the only `oob_degraded` in the operator mailbox in 30 days, and no recovery mail followed; the report says port 0. Two candidate mechanisms in the agent (`internal/felhomsshd/health.go:63-66`): `sshd -t -f ` run by the NON-root agent fails on unreadable host keys, or a config rendered before the desired state arrived (port 0). **CAUSE FOUND AND FIXED 2026-10-21 night** (`audits/night-2026-10-21/P5-checks/`): reproduced on a fresh box from the public ISO — `oob_degraded` mailed 20:34 CEST while felhom-sshd was applied 0.4 s after the check and is healthy (port 8822 listening, `sshd -t` passes as the agent's user through its sudo grant — so the non-root theory is FALSE; without sudo it fails on the host key, which the agent never does). The first host report runs before the felhom-sshd loop's first pass and reads a config never rendered. Fix: agent `0da792e` — no OOB stanza until the first pass has run (the hub does not evaluate a missing one; a failed pass counts). `TestR1194_NoOOBStanzaBeforeTheFirstPass` red-proved. | WATCHING — released in agent v0.162.0 (2026-10-22, operator yes; on the 3 test boxes); the false alarm comes only on a FRESH box's first report | the next fresh install with agent ≥ 0.162.0 | At the next fresh install read the hub's events for `oob_degraded` in the first hour (none expected); then close | CC | | **R-1237** | Security & access | P3 | **The nightly CC-secrets sweep found stored one-time codes in a session record dated 2026-10-23: 14 names in one file (bind tokens and setup codes of test boxes — NIGHT_2026_10_23_*, FIXES_2026_10_21_*, NIGHT_2026_10_21_*, REHEARSAL_2026_10_20_* — and R_DEMO_FELHOM in a file of 2026-10-20).** Mail „[DooPlex CC secrets] 2 secret name(s) found in old session records", 2026-10-23 01:57 UTC, read in the day session (names only). One file holding all of them suggests one command printed many stored values at once (a listing or a dump) in the night session. All but R_DEMO_FELHOM are one-time install codes of test boxes; NIGHT_2026_10_23_* die when Tester4-16fdef is deleted (the operator's do list). | **OPEN** — 2026-10-27 (operator yes): the dead names were REMOVED from the store with `sec rm` (46, one commit each; `audits/r1299-window-button-2026-10-27/sec-rm.txt`, names only): every one-time code this row names is gone; R_DEMO_FELHOM stays (unsure). Consequence: the nightly sweep no longer watches for the removed values — they open nothing (their boxes are destroyed). The session transcripts themselves are unchanged — that question stays here. **Earlier:** OPEN | — | CC: find the file and the command by the sweep's own report (names only), add a hook shape for that command if it is not one; R_DEMO_FELHOM (demo-felhom's recovery code) — the operator decides whether to rotate it | CC | @@ -292,14 +291,12 @@ stopping line that lies. | **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC | | **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. **2026-11-11:** the End-dates list leans on this row (`documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB21); the newest golden passed its 7-day limit on 2026-11-10 and no waiver is in force, so the gate refuses code pushes to `felhom.eu`. **2026-11-11: golden 0.352.0 baked and published; one fresh box installed from it (`documentation/tests/golden-0.352.0-2026-11-11/README.md`); the gate is green (`python3 scripts/golden_currency_gate.py`, rc 0). The operator vouched it on 2026-11-11 at 14:48:11 CET.** | WATCHING — **2026-11-14: golden 0.352.1 baked (sha256 `4f411cbe…346d`), one fresh install proven on a nested VM on demo-hp, vouched by the operator 10:25:43 UTC (`documentation/tests/golden-0.352.1-2026-11-14/`). The gate judges the bake's age: the next bake is due by 2026-11-21.** Earlier: **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement **2026-11-03: the Sunday 2026-11-01 bake was missed (it was in no brief): golden 0.344.0 is 8 days old, the floor 0.349.0; a CODE push to felhom.eu is refused until a bake and the vouch or a dated waiver. The night's close-out was documents only, which the gate exempts; no waiver written. First on the morning list.** | — | — | CC | | **R-1358** | Box system & updates | P3 | **cloudflared 2026.10.4 misses two dependency fixes (VULN-183456 chi, 2026.10.5; VULN-184686 go-jose, 2026.10.6; the newest is 2026.10.7).** No public text on reach. Not pinned tonight: the tunnel is a box's only door, and 9202 has none to prove it on. | OPEN — filed 2026-11-03 | — | raise `CloudflaredImage` on the next controller release with a tunnel proof on Tester 1 | CC | -| **R-1380** | Box system & updates | P3 | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** Found 2026-11-11 (the End-dates list, class T2, `documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB02-3 to DB02-7). MEASURED: the agent trusts one exact Proxmox certificate (a pin), and the certificate on the three test boxes ends about two years after install — the N100's about 2028-07-16, then demo-hp and Tester 1 within three months. Proxmox makes a new certificate by itself about 14 days before the old one ends (so about 2028-07-02 on the N100). No code path gives the agent the new fingerprint, no alarm names the cause, no runbook holds the hand step, and nobody has tried it. Every box installed in the same season fails in the same fortnight. P3 only because the first date is in 2028; it is certain, not a risk. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 297). Not built.** **2026-11-11 (afternoon): tried on demo-hp (`documentation/audits/certpin-spike-2026-11-11/README.md`).** A forced new certificate (`pvecm updatecerts --force`, then a reload of the web service; Proxmox's own renewal does that reload by itself) blinds the agent at once: `TLS pin mismatch`, no reconcile, no host report to the hub, 502 to the controller; guests and apps stay up. One operator mail names a symptom (`agent_channel_unknown`), none names the cause; the hub's Hosts page still read ONLINE eleven minutes after the last report (when it turns offline was not measured). The hand repair works and is written down (`documentation/runbooks/RUNBOOK-agent-proxmox-pin-repair.md`, 2.1 s from restart to a working call). **A CA file in place of the fingerprint works with no agent code:** with `proxmox.tls.ca_file` on a readable copy of the box's root CA and the fingerprint empty, the agent went through a second new certificate with no hand step; a wrong CA file was refused. Two facts for the fix: the agent's user cannot read Proxmox's certificate files, and a set fingerprint wins over the CA file. The design with both ways is `03-host-agent.md` §12a; CC's pick differs from the ruled shape, so it is decision D80. | READY — **2026-11-14 night, after an independent review (agent `ec7e699`, `54ac2c2`):** one lock for every writer of `agent.json`; the trust run is serialised and ends after 90 s; the root step keeps the previous root only during a changeover and compares exactly. **One low item stays open:** the move writes `agent.json` in place and nothing restores `agent.json.pre-pve-trust` by itself after a cut write (`03` §12a says so) — to weigh at the box proof. Earlier: READY — **2026-11-14 night: BUILT, not released** (felhom-agent `386a0c4` + `a9ab47e`; the installer on felhom.eu `main`, not tagged). The root step keeps the copy; the agent moves a pinned box in the order copy → proof → config, clears the fingerprint last, and a pinned box whose certificate changed heals at the first failed call; the installer gives a new box the copy and no fingerprint; the cause of a trust failure is named. `03` §12a is `[FACT — built, not released]`. **LEFT: the proof on a box** (demo-hp, a forced new certificate, twice; then the release with both signed jobs and the installer tag). What the hub does not tell yet: R-1396. Earlier: READY — **2026-11-14 09:55 — RULED (D80, `09` §3 decision 303): the root CA copy.** Built in release 2 (the host release). OPEN — filed 2026-11-11 (end-dates list) | — | the operator rules D80 (CA file, or follow the fingerprint); then the design's open parts in `03-host-agent.md` §12a (the root step that refreshes the copy; an alarm that names the cause), then build, then prove on a scratch host with a forced renewal | CC | | **R-1382** | Box system & updates | P3 | **A box that was off past an end date: nobody knows if it comes back by itself (5 answers).** Found 2026-11-11 (the End-dates list, class T4, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; read in source and documents, never tried). After months without power: the web certificate has ended (DB01-6: does Traefik renew at the first start, and what does the household see until then), Debian's package keys may be too old to verify an update (DB16-6), the tunnel client may be below what Cloudflare accepts (DB19-6), a 30-day restore copy is past its date (DB24-6), and the 14-day countdown after a stopped off-site copy ran out while the box was off (DB04-6). Tester 2 has been off since 2026-10-04 and is the natural test. | OPEN — filed 2026-11-11 (end-dates list) | — | When Tester 2 comes back (or with a scratch host and a moved clock on Tier 0, as a task): record what renews by itself and what needs a hand; put the hand steps in a runbook. | CC | -| **R-1384** | Box system & updates | P3 | **A verdict or an „already ran” check trusts a clock that can be wrong (30 answers).** Found 2026-11-11 (the Clock list, class W2, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED: the N100's host is 32.6 s behind and its time service does not run (R-1301); the other two boxes are in step. Nothing checks a box's clock except one hub alarm for a tunnel handshake stamped in the future. The hub takes the box's own time for backup age, off-site age and proof age (CH02-6, CH07-6, CH07-8, CH08-6); a box with a wrong clock refuses every signed job (CA09-6); a stamp from the future blocks the OS-update leg, the trim and the line test until the clock passes it (CA02-6, CA02-8, CA11-6, CA11-8, CA12-6); the crash guard reads the clock at early boot (CA04-6, CA04-8); a hub clock jump moves its own daily jobs (CH03-6, CH04-6, CH06-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6). The others: CA05-6, CA05-8, CA06-6, CA07-6, CA10-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6. Known parts stay in their rows: R-1189, R-1206, R-1260. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 300). Not built.** **2026-11-11 (afternoon): why the N100's clock is wrong is known (R-1301, `documentation/audits/n100-clock-2026-11-11/README.md`).** Numbers from source: a host clock more than 120 s behind refuses every signed job (`felhom-agent/internal/authz/verifier.go:21`, `:153-162`); at about 1.25 s a day the N100 gets there in about 70 days (mid-January 2027, an estimate from one measurement). | NARROWED — the hub's check is built, not released. **2026-11-14 (night build):** the agent stamps its clock at the moment of sending (`sent_at`) and the hub alarms the operator when it is more than one minute from the hub's clock, in either direction, and says so when it is back (`08-alarm-ladder.md`, `box_clock_wrong`). A report that waited on the box raises nothing (tested). Not released, not seen on a box. LEFT after the release: the 30 answers themselves still trust the clock; the alarm only makes a wrong clock known. OPEN — **2026-11-14: the one box whose clock was wrong (the N100, 32.6 s) is right again (R-1301 closed); this row's check is still not built.** Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): the hub compares the time in each report with its own, alarms the operator above one minute, and does not take a time from the future as „fresh”. | CC | +| **R-1384** | Box system & updates | P3 | **A verdict or an „already ran” check trusts a clock that can be wrong (30 answers).** Found 2026-11-11 (the Clock list, class W2, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED: the N100's host is 32.6 s behind and its time service does not run (R-1301); the other two boxes are in step. Nothing checks a box's clock except one hub alarm for a tunnel handshake stamped in the future. The hub takes the box's own time for backup age, off-site age and proof age (CH02-6, CH07-6, CH07-8, CH08-6); a box with a wrong clock refuses every signed job (CA09-6); a stamp from the future blocks the OS-update leg, the trim and the line test until the clock passes it (CA02-6, CA02-8, CA11-6, CA11-8, CA12-6); the crash guard reads the clock at early boot (CA04-6, CA04-8); a hub clock jump moves its own daily jobs (CH03-6, CH04-6, CH06-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6). The others: CA05-6, CA05-8, CA06-6, CA07-6, CA10-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6. Known parts stay in their rows: R-1189, R-1206, R-1260. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 300). Not built.** **2026-11-11 (afternoon): why the N100's clock is wrong is known (R-1301, `documentation/audits/n100-clock-2026-11-11/README.md`).** Numbers from source: a host clock more than 120 s behind refuses every signed job (`felhom-agent/internal/authz/verifier.go:21`, `:153-162`); at about 1.25 s a day the N100 gets there in about 70 days (mid-January 2027, an estimate from one measurement). | NARROWED — **2026-11-15: LIVE and seen on a box** (hub 0.182.0, agent 0.175.1). Tester 1's clock was set 150 s back twice: the hub said „150 s behind”, one mail went to the operator only, a second alarm inside the hour sent no second mail, and the hub said „in step again” after the clock was put right (`audits/release-2b-3-2026-11-15/` `part2/`, `part3/`). LEFT, as before: the 30 answers themselves still trust the clock. Earlier: NARROWED — the hub's check is built, not released. **2026-11-14 (night build):** the agent stamps its clock at the moment of sending (`sent_at`) and the hub alarms the operator when it is more than one minute from the hub's clock, in either direction, and says so when it is back (`08-alarm-ladder.md`, `box_clock_wrong`). A report that waited on the box raises nothing (tested). Not released, not seen on a box. LEFT after the release: the 30 answers themselves still trust the clock; the alarm only makes a wrong clock known. OPEN — **2026-11-14: the one box whose clock was wrong (the N100, 32.6 s) is right again (R-1301 closed); this row's check is still not built.** Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): the hub compares the time in each report with its own, alarms the operator above one minute, and does not take a time from the future as „fresh”. | CC | | **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC | | **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC | | **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC | | **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-27: agent v0.167.0 RELEASED; in the NORMAL signed delivery the pass now runs right after the commit on all three boxes („running the bundle policy pass after the self-update commit (R-1285)” → `BUNDLE POLICY DONE install_mode=appliance removed=0`; `audits/release-2026-10-27/agent-0167-tester1-demofelhom.txt`). Not live: a BYO box coming FROM 0.164.0 (none in the fleet). **Earlier:** **VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:** **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC | -| **R-1395** | Box system & updates | P4 | **A step of the host's clock makes the controller read a new boot: it then recreates every drive-backed app, once.** Found 2026-11-14 on the N100 (`audits/release-2a-2026-11-14/D/13b-n100-container-near-the-step.txt`): the time service came back and stepped the clock by +32.7 s; 9 seconds later the controller logged „[gate] boot 1794112550-…: recreating drive-backed app” for radarr, sonarr and qbittorrent (each down for 1 to 5 s; all healthy after). Cause, read on the box: the controller's boot identity carries `/proc/stat` `btime`, and the kernel computes `btime` from the wall clock, so it moved by the same 32 s (the crash guard's `last_boot_at` reads 04:35:18Z, `btime` now 04:35:50Z). The crash guard was NOT fooled: it reads the kernel's `boot_id`. A step this large needs a clock that was wrong for weeks, so a healthy box does not meet it; a fresh box with a wrong hardware clock can, at its first time sync. | NARROWED — built in the agent and the controller, not released. **2026-11-14 (night build):** the boot token is the AGENT's (`guest_boot_id` in its `/disks` answer), not the controller's; the controller only compares it. The agent now also sends `guest_boot_key` (the kernel's `boot_id` in place of the boot time) and the controller's boot pass prefers it; the move to the key recreates nothing; an older agent or controller behaves as before. Tests with a moved boot time on both sides (`audits/night-build-2026-11-14/C/`). Needs BOTH releases to take effect. Not proven on a box (a clock step cannot be staged without changing a box). OPEN — filed 2026-11-14 | — | make the controller's boot identity the kernel's `boot_id` (as the crash guard's is), with a test that a moved `btime` is not a new boot | CC | ## Monitoring & notifications — 29 rows (P2 3, P3 18, P4 8) @@ -331,9 +328,10 @@ stopping line that lies. | **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC | | **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) | | **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC | -| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | NARROWED — the two ruled warnings are built, not released. **2026-11-14 (night build):** the hub warns the operator 14 days before an operator token ends and 14 days before a box's web certificate ends (the box reads the date from its own store and sends it in its report) — `08-alarm-ladder.md` §12.1. The other sixteen are a design table there (§12.2): one is small and waits for a yes (the hub's own five public names), one is a design question (a household's Cloudflare token), the rest stay the operator's list. Not seen on a box or a deployed hub. OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC | -| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | NARROWED — **2026-11-14 (night build, second part):** the two things release 2a did not tell are built, not released: the System page has a cell for the kernel clean-up (removed, kept, space gained) and one for the security-profile refresh (the kernel, how many profiles changed, any failed service it restarted) — `11` §5.11.1, §5.12. On the way a display defect was fixed: a clean-up hid a kernel that fell back the night before in the „Kernel step” cell. NARROWED — the four worst cases are built, not released. **2026-11-14 (night build):** the operator is told when a hub daily check's slot passed while the hub was down, when an operator mail was never sent, when a box did not take a signed job in two hours, when a box refused a signed job (with the cause), and when the guest-power watchdog gave up (`05` §16.6, `08-alarm-ladder.md`). Tests only; not deployed, not seen on a box. LEFT: the other 15 answers of the class; a household mail is still never retried (R-1361); the kernel clean-up and profile-refresh detail (next part of release 3). OPEN — **2026-11-14 (release 2a), two more things the hub does not tell:** (1) a kernel clean-up that removed kernels shows only as „cleaned