lists: the Clock list (70 jobs) and the End-dates list (39 things), 9 register rows, 4 decisions
gates / gates (push) Successful in 2m2s

Read-only review of every job on a timer and of everything that ends on a date,
with measurements from the three test boxes. Nothing ends in the next 14 days.
Register 236 -> 245 (R-1379..R-1387), pointer lines on twelve rows, D76-D79 in
STATUS.md. Part C: felhom-agent's clone address holds no token now; the other
clones wait for the operator's yes. R-1035's 4-week check moved to 2026-11-18
(operator read needed).

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-11 11:54:42 +01:00
1 parent 43e244ad8f
commit 978666dde3
81 files changed
+26738 -20

No files matched your search

@@ -0,0 +1,117 @@
# The Clock list — the matrix
Made by `tools/make-matrix.py` from `TABLE.md`. Do not edit it by hand.
`A` answered · `D` decided, not built · `K` known gap (has a row) · `N` nobody asked. Columns are the eight
questions: 1 what runs, owner, what starts it · 2 when, how often, order · 3 how long, limit, 10x data · 4 if it does not run · 5 late, twice, meets another job · 6 clock wrong or jumps · 7 who is told, when · 8 how do we know.
| Job | Named | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | N |
|---|---|---|---|---|---|---|---|---|---|---|
| CA01 Whole-guest backup: due-check and run · CUSTOMER | mentioned | A | A | A | A | A | K | A | K | 0 |
| CA02 OS-update night leg: guest, host, Docker, Proxmox · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | **N** | 2 |
| CA03 Kernel lane: night reboot, 20-minute boot judge, one self-revert · CUSTOMER | mentioned | A | A | A | A | A | K | A | A | 0 |
| CA04 Crash guard: early-boot unit and the hourly re-arm timer · CUSTOMER | mentioned | A | A | A | A | **N** | **N** | A | **N** | 3 |
| CA05 Scheduled restore-test evaluation · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | **N** | 2 |
| CA06 Controller supervisor: restarts a dead or silent controller · CUSTOMER | mentioned | A | A | **N** | A | A | **N** | A | A | 2 |
| CA07 Guest-power watchdog: starts a guest that must run · CUSTOMER | mentioned | A | A | A | A | A | **N** | **N** | **N** | 3 |
| CA08 Hub report and heartbeat loop · CUSTOMER | mentioned | A | A | A | A | A | A | A | A | 0 |
| CA09 Signed jobs: the validity window against the host clock · OPERATOR | mentioned | A | A | A | **N** | A | **N** | **N** | A | 3 |
| CA10 Storage watchdog and the 20-second mount re-assert · CUSTOMER | mentioned | A | A | **N** | A | A | **N** | A | **N** | 3 |
| CA11 Weekly guest disk trim · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | **N** | 2 |
| CA12 Weekly upload line test · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | A | 1 |
| CA13 PBS datastore verify loop · CUSTOMER | mentioned | A | A | A | A | **N** | A | **N** | **N** | 3 |
| CA14 Janitor: leftover scratch guests and stale backup locks · CUSTOMER | mentioned | A | A | A | A | A | A | A | A | 0 |
| CA15 Guest network watchdog: DHCP client heal · CUSTOMER | mentioned | A | A | **N** | A | A | A | A | A | 1 |
| CA16 Host networking self-heal · CUSTOMER | mentioned | A | A | A | A | A | A | A | A | 0 |
| CA17 Management-plane watchdog timer: keeps SSH reachable · OPERATOR | mentioned | A | A | A | A | A | A | A | A | 0 |
| CA18 Agent service: restart, start limit, rollback · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | **N** | 2 |
| CA19 Off-site tunnel loop and the clock-ahead re-key · CUSTOMER | mentioned | A | A | **N** | A | A | A | A | A | 1 |
| CA20 Whole-guest backup cap: rolling windows · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | A | 1 |
| CA21 Unsent OS-update report sender · OPERATOR | unnamed | A | A | A | A | A | A | A | A | 0 |
| CC01 Night dump leg: databases and volumes · CUSTOMER | mentioned | A | A | A | A | A | A | A | A | 0 |
| CC02 Second-drive copy leg · CUSTOMER | mentioned | A | A | A | A | **N** | A | A | A | 1 |
| CC03 Off-site push leg, with the clean-up of old snapshots · CUSTOMER | mentioned | A | A | A | A | K | A | A | A | 0 |
| CC04 Missed-night catch-up and the resume watch · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | A | 1 |
| CC05 Missed-backup banner (26 hours) · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | A | 1 |
| CC06 Whole-guest backup driver: stop apps, ask the agent, start apps · CUSTOMER | mentioned | A | A | A | A | A | K | A | A | 0 |
| CC07 Automatic app-update leg · CUSTOMER | mentioned | A | A | A | A | A | K | **N** | A | 1 |
| CC08 Off-site integrity check · CUSTOMER | mentioned | A | A | K | A | A | A | A | A | 0 |
| CC09 Off-site proof restore · CUSTOMER | mentioned | A | A | A | A | **N** | A | A | A | 1 |
| CC10 Hub report: every 15 minutes, on change, and the hanging wait · CUSTOMER | mentioned | A | A | A | A | A | A | A | A | 0 |
| CC11 Controller self-update: check, floor, nightly auto · CUSTOMER | mentioned | A | A | **N** | A | **N** | A | A | A | 2 |
| CC12 Catalog sync: app templates from git · CUSTOMER | mentioned | A | A | **N** | **N** | A | A | A | A | 2 |
| CC13 System health check and base-stack self-heal · CUSTOMER | mentioned | A | A | **N** | A | A | A | A | A | 1 |
| CC14 Dead-app scan · CUSTOMER | mentioned | A | A | **N** | A | A | **N** | A | A | 2 |
| CC15 Disk-fill watch · CUSTOMER | mentioned | A | A | **N** | A | A | A | A | A | 1 |
| CC16 Drive health check, hourly · CUSTOMER | mentioned | A | A | **N** | A | A | A | A | A | 1 |
| CC17 Agent channel health, every 60 seconds · CUSTOMER | mentioned | A | A | A | A | A | A | A | A | 0 |
| CC18 Held-app report to the hub · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | A | 1 |
| CC19 Abandoned off-site store sweep · CUSTOMER | mentioned | A | A | A | A | **N** | A | **N** | A | 2 |
| CC20 Restore-copy sweep and conversion-copy release · CUSTOMER | mentioned | A | A | **N** | **N** | A | A | A | A | 2 |
| CC21 Upload and CPU watch, torrent limit rule, line-test hold · CUSTOMER | mentioned | A | A | A | A | **N** | A | A | A | 1 |
| CC22 The scheduler itself: daily slots and interval jobs · CUSTOMER | mentioned | A | A | A | A | A | K | A | A | 0 |
| CH01 Box liveness: 'your server cannot be reached' · CUSTOMER | mentioned | A | A | **N** | A | A | A | A | A | 1 |
| CH02 Missed-backup deadline check (05:00) · CUSTOMER | mentioned | A | A | **N** | **N** | A | **N** | A | **N** | 4 |
| CH03 Mail cooldowns and the weekly outage mail · CUSTOMER | mentioned | A | A | A | A | A | **N** | **N** | A | 2 |
| CH04 Retry of a failed operator mail · OPERATOR | mentioned | A | A | A | **N** | A | **N** | **N** | **N** | 4 |
| CH05 Kernel mail: 'your box restarts tonight' · CUSTOMER | mentioned | A | A | A | A | A | A | A | **N** | 1 |
| CH06 OS-update approval tick and the hourly OS alarms · OPERATOR | mentioned | A | A | A | A | A | **N** | A | **N** | 2 |
| CH07 Off-site copy too old, and escrow never finished · CUSTOMER | mentioned | A | A | A | **N** | A | **N** | A | **N** | 3 |
| CH08 Restore-test proof too old · OPERATOR | mentioned | A | A | **N** | A | A | **N** | A | **N** | 3 |
| CH09 Off-site clean-up window (20 minutes, weekly) · CUSTOMER | mentioned | A | A | A | **N** | A | **N** | A | A | 2 |
| CH10 Delayed deletion of a set-aside off-site copy · CUSTOMER | mentioned | A | A | **N** | A | A | A | A | A | 1 |
| CH11 Off-site key audit (07:10) · OPERATOR | mentioned | A | A | **N** | **N** | A | A | A | **N** | 3 |
| CH12 Off-site credential stuck, and its self-heal · OPERATOR | mentioned | A | A | A | A | A | A | A | A | 0 |
| CH13 PBS whole-box credential self-heal · OPERATOR | mentioned | A | A | A | A | A | A | A | **N** | 1 |
| CH14 WireGuard peer list push to ep0 · OPERATOR | mentioned | A | A | A | **N** | A | A | A | A | 1 |
| CH15 Box clock ran ahead: the tunnel alarm · OPERATOR | mentioned | A | A | A | A | A | A | A | A | 0 |
| CH16 The daily scheduler and the hour change · OPERATOR | mentioned | A | A | A | **N** | A | **N** | A | A | 2 |
| CH17 Hub database snapshot (four slots) · OPERATOR | mentioned | A | A | A | A | A | **N** | A | A | 1 |
| CH18 Daily prune of old hub rows (04:30) · OPERATOR | mentioned | A | A | **N** | A | A | **N** | A | A | 2 |
| CH19 The other checks on the 60-second sweep · OPERATOR | mentioned | A | A | **N** | A | A | A | A | **N** | 2 |
| CH20 Reminder about a held app · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | A | 1 |
| CH21 'A golden is published but not vouched' · OPERATOR | mentioned | A | A | A | A | A | A | A | A | 0 |
| CH22 Shared storage boxes: fill and reachability · OPERATOR | mentioned | A | A | A | A | A | A | A | A | 0 |
| CH23 DooPlex: push of the hub database snapshot off-site · OPERATOR | mentioned | A | A | A | A | A | K | **N** | A | 1 |
| CH24 DooPlex: Sunday restore tests · OPERATOR | mentioned | A | A | A | A | **N** | **N** | A | A | 2 |
| CH25 DooPlex: nightly off-site push and the release copy · OPERATOR | mentioned | A | A | **N** | A | A | A | A | A | 1 |
| CH26 DooPlex and ep0: PBS copy, prune, verify, clean-up · OPERATOR | mentioned | A | A | A | A | **N** | **N** | **N** | A | 3 |
| CH27 Nightly CI: the gates and the security watch · OPERATOR | mentioned | A | A | A | **N** | A | A | A | **N** | 2 |
## Totals
- Jobs: **70** in the table · named **0** · mentioned only **69** · named by no architecture document **1**.
- Answers: **560** · A **457** · D **0** · K **9** · N **94**.
- Jobs with no N: **18** of 70.
| Question | A | D | K | N |
|---|---|---|---|---|
| 1 what runs, owner, what starts it | 70 | 0 | 0 | 0 |
| 2 when, how often, order | 70 | 0 | 0 | 0 |
| 3 how long, limit, 10x data | 50 | 0 | 1 | 19 |
| 4 if it does not run | 59 | 0 | 0 | 11 |
| 5 late, twice, meets another job | 60 | 0 | 1 | 9 |
| 6 clock wrong or jumps | 36 | 0 | 6 | 28 |
| 7 who is told, when | 61 | 0 | 0 | 9 |
| 8 how do we know | 51 | 0 | 1 | 18 |
## The N answers, by class row
| Class | Row | Count | Members (id-question) |
|---|---|---|---|
| W1 | R-1383 | 23 | CA07-7, CA07-8, CA09-4, CA09-7, CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH02-4, CH02-8, CH03-7, CH04-4, CH04-7, CH04-8, CH07-4, CH09-4, CH11-4, CH14-4, CH16-4, CH26-7, CH27-4, CH27-8 |
| W2 | R-1384 | 30 | CA02-6, CA02-8, CA04-6, CA04-8, CA05-6, CA05-8, CA06-6, CA07-6, CA09-6, CA10-6, CA11-6, CA11-8, CA12-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6, CH02-6, CH03-6, CH04-6, CH06-6, CH07-6, CH07-8, CH08-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6 |
| W3 | R-1385 | 12 | CA04-5, CA13-5, CC02-5, CC09-5, CC11-5, CC19-5, CC21-5, CH05-8, CH24-5, CH25-3, CH26-5, CH26-6 |
| W4 | R-1386 | 25 | CA06-3, CA10-3, CA10-8, CA13-8, CA15-3, CA19-3, CC11-3, CC12-3, CC13-3, CC14-3, CC15-3, CC16-3, CC20-3, CH01-3, CH02-3, CH06-8, CH08-3, CH08-8, CH10-3, CH11-3, CH11-8, CH13-8, CH18-3, CH19-3, CH19-8 |
| W5 | R-1387 | 4 | CA18-6, CA18-8, CH17-6, CH23-7 |
## The K answers, by existing row
| Row | Count | Members |
|---|---|---|
| R-401 | 1 | CC08-3 |
| R-1186 | 1 | CC03-5 |
| R-1189 | 2 | CA01-6, CA01-8 |
| R-1260 | 3 | CC06-6, CC07-6, CC22-6 |
| R-1293 | 1 | CH23-6 |
| R-1301 | 1 | CA03-6 |
@@ -0,0 +1,95 @@
# The Clock list — 2026-11-11
The third list of the gap review. One block for each job that a clock starts, with eight questions each. Built from
the source of four repositories and from read-only measurements on the three test boxes (host and guest). **No
document lists every timer** — the claim in the brief is true (search S19, 0 hits). DooPlex, ep0 and the hub's
database are `NOT MEASURED`, by ruling; their jobs are in the list from source and documents only.
Files: `TABLE.md` (the answers), `MATRIX.md` (the marks), `evidence/`, `tools/`.
## Five lines
1. **Jobs found: 77** helper blocks. 70 are in the table (22 in the guest, 21 on the host, 27 on the hub, DooPlex and
ep0). 7 are tiny, 2 are end dates (all named in the table's head). No `TEST-ONLY` job was found.
2. **Jobs no architecture document names: 1** of 70 by the search (the sender of unsent OS-update reports). The other
69 are mentioned somewhere; no document puts them in one list or one order.
3. **Marks (560 answers):** `A` 457 · `D` 0 · `K` 9 · `N` 94.
4. **The worst `N`:** when the guest-power watchdog gives up, it writes one line in the host's journal and tells nobody
(CA07). The household's box is then down, and the only outside sign is the hub's "cannot be reached" check.
Next worst: a box with a wrong clock refuses every signed job, so it cannot take an agent update (CA09).
5. **The operator decides** the two questions at the end of this page.
## The clock itself, per box (MEASURED 2026-11-11)
| Box | Host clock against DooPlex | Who keeps the time | Zone: host · guest |
|---|---|---|---|
| demo-hp | +0.07 s | the host's time service, in step | Europe/Budapest · UTC |
| Tester 1 | +0.07 s | the host's time service, in step | Europe/Budapest · UTC |
| N100 | **32.6 s behind** | **nobody: the time service is not running** (R-1301) | Europe/Budapest · UTC |
The guest has no clock of its own; it reads the host's. The controller's night slots are in Budapest time. At a start
with a wrong time the crash guard reads the clock before any time service runs (CA04). Who notices a wrong clock: the
hub alarms only when a tunnel handshake is stamped in the future (CH15). A clock that is behind: nobody.
## One night on a box (backup time 02:30; local time; durations from the night of 2026-11-11)
| Start | Job | demo-hp · Tester 1 · N100 | Limit | Waits for |
|---|---|---|---|---|
| 02:30 | Dump of databases and volumes (CC01) | 2 min 33 s · 40 s · 18 s | 5 min for each dump; none for the leg | nothing |
| 03:30 | Copy to the second drive (CC02) | 39 s · 15 s · 11 s | none | the clock only (+60 min) |
| 04:15 | Off-site push and clean-up (CC03) | 3 min 09 s · 59 s · 38 s | none | the clock (+105 min); skipped if a backup holds the flag |
| after it | App updates (CC07) | not seen this night | backup time + 5 h | the off-site push |
| 04:30 | Whole-guest gate opens; controller auto-update, if on (CC11) | — | — | no backup, no app update in flight |
| 05:10 | Sweep of an abandoned off-site store (CC19) | ran | 1 min | nothing |
| 05:30 | Off-site proof restore (CC09) | 5 s · 5 s · 3 s | — | the off-site flag |
| 06:00 | Off-site integrity check (CC08) | under 1 s | — | the off-site flag |
| 02:30 to 05:37 | Whole-guest backup, by the host (CA01) | 11–13 min · 3 min · under 1 min | the agent's cap | the gate; then the OS-update leg (CA02) follows it |
**Every duration is from a small test store.** Nobody knows these numbers for a full household disk.
**Where two jobs can meet:** (1) on Tester 1 and the N100 the whole-guest backup started at about 05:30, the minute of
the proof restore — measured, and nothing orders them (CC09-5). (2) On demo-hp one whole-guest backup started at
02:30:38, with the dump leg. (3) The auto-update's default time is the minute the gate opens (CC11-5). (4) A long
second-drive copy makes the off-site push skip the night with no message (R-1186). (5) On the N100 the whole-guest
backup starts 5 minutes later each day (05:18 to 05:37 in four days).
## The `N` answers, by class (one register row each)
| Class | Row | What | Count | Main members |
|---|---|---|---|---|
| W1 | R-1383 | A job does not run, gives up, or loses its state at a restart, and nobody is told | 23 | guest-power give-up (CA07) · refused signed job (CA09) · hub 05:00 check and 07:10 audit skipped when the hub is down (CH02, CH11, CH16) · operator mail retry lost at a restart, household mail never retried (CH04, CH03) · peer list push to ep0 (CH14) · app-update leg (CC07) · catalog sync (CC12) · PBS verify failure (CA13) |
| W2 | R-1384 | A verdict or an "already ran" check trusts a clock that can be wrong | 30 | the hub trusts the box's time for backup age, off-site age, proof age (CH02, CH07, CH08) · signed jobs (CA09) · stamps from the future block the OS-update leg, the trim, the line test (CA02, CA11, CA12) · crash guard at early boot (CA04) · hub clock jumps (CH03, CH04, CH09, CH16) |
| W3 | R-1385 | Two jobs can meet with no lock and no order | 12 | proof restore and whole-guest backup (CC09) · auto-update and the gate (CC11) · PBS verify beside a backup (CA13) · the timers on DooPlex and ep0 (CH24, CH25, CH26) |
| W4 | R-1386 | No time limit and no measured duration | 25 | the hub's 60-second sweep: one slow check delays all, the liveness check too (CH19, CH01, CH08) · the 07:10 audit's 10 minutes at fleet size (CH11) · the interval jobs on the box (CC12 to CC16, CC20, CA06, CA10) |
| W5 | R-1387 | A document and the source disagree on a time | 4 | the agent unit: `Restart=always` and a watchdog in the design, `on-failure` and no watchdog in the unit (CA18) · 13 h or 26 h for the hub-database alarm (CH23) · old "24 h / 02:30" text (CH17) |
Known gaps (`K`, 9 cells): R-1260 the hour-change night on a box (3) · R-1189 a host clock ahead during a backup (2) ·
R-1186 the silent off-site skip · R-1293 the hub copy on the autumn hour-change day · R-1301 · R-401.
## Not made up, or no limit, by ruling (each is an `A`)
Only the three backup legs are made up after a missed night; the proof restore, the integrity check, the app updates
and the abandon sweep wait for the next night · a daily slot that is more than 60 minutes late is skipped, so nothing
heavy runs at noon · no OS-update leg on a night with no good backup · the hub's snapshot and prune are made up, its
other daily jobs are not · household mails about an outage: one each 7 days · days of 23 and 25 hours on the hub (D24).
## Decisions for the operator (also in `STATUS.md`)
1. **Shall a job that gives up or is skipped tell you?** Yes (CC's pick): one hub alarm for the four worst cases — the
guest-power watchdog gave up, a hub daily check was skipped, an operator mail was lost, a signed job was refused.
Small changes in the agent and the hub. No: the logs stay the only trace; the cost is that you learn of a dead guest
from the household. If nothing happens: the same as "No".
2. **Shall the hub check each box's clock?** Yes (CC's pick): the hub compares the time in each report with its own and
alarms above one minute of error; it then stops trusting a backup time from the future. No: R-1301 (repair the
N100's time service) stays the only guard; the cost is that a box with a wrong clock can look healthy and cannot
take an agent update. If nothing happens: the N100 stays 32 s behind and nobody is told when it gets worse.
## Method verdict (third list)
- **It worked for the map, and the measurement found the meetings.** Three read-only helpers gave 77 blocks. The 05:30
meeting of two jobs and the 5-minutes-a-day drift on the N100 are in no helper answer; they came from the boxes.
- **Read by eye: 138 of 560 cells** hold the main session's words (every `K` and `N`, and 35 `A`). For 70 of them the
pointer was opened by eye; for 68 the helper's pointer stands and a tool opened it. The main session read every
helper answer to questions 4 to 7 (280 cells); it did not open each of their pointers. A tool opened all 896.
- **The weak points:** the hub's jobs and the DooPlex and ep0 timers are from source only. The controller's log held
one night. The "named" column is a machine count with wide search words; it says "mentioned", not "explained".
- **The ninth line:** 6 jobs have a hand step in a runbook; 4 have a hand step that no runbook holds (CC03, CA07, CA09,
CH02); 1 in part (CA18). The other 59 have no hand step.
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,468 @@
# guest-clock demo-felhom taken 2026-11-11T10:04:22Z from DooPlex
=== clock ===
utc 2026-11-11T10:03:50.305180545Z
local 2026-11-11T10:03:50 UTC +0000
/usr/share/zoneinfo/Etc/UTC
Local time: Wed 2026-11-11 10:03:50 UTC
Universal time: Wed 2026-11-11 10:03:50 UTC
RTC time: n/a
Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: no
NTP service: inactive
RTC in local TZ: no
chrony: inactive
systemd-timesyncd: inactive
=== boots ===
2026-11-08 04:35:32
=== timers ===
NEXT LEFT LAST PASSED UNIT ACTIVATES
Wed 2026-11-11 12:30:43 UTC 2h 26min Wed 2026-11-11 01:28:16 UTC 8h ago apt-daily.timer apt-daily.service
Thu 2026-11-12 00:00:00 UTC 13h Wed 2026-11-11 00:00:14 UTC 10h ago dpkg-db-backup.timer dpkg-db-backup.service
Thu 2026-11-12 00:17:27 UTC 14h Wed 2026-11-11 01:00:14 UTC 9h ago logrotate.timer logrotate.service
Thu 2026-11-12 04:52:34 UTC 18h Wed 2026-11-11 04:52:34 UTC 5h 11min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Thu 2026-11-12 06:50:13 UTC 20h Wed 2026-11-11 06:43:16 UTC 3h 20min ago apt-daily-upgrade.timer apt-daily-upgrade.service
Thu 2026-11-12 11:29:47 UTC 1 day 1h Wed 2026-11-11 09:21:14 UTC 42min ago man-db.timer man-db.service
Sun 2026-11-15 03:10:29 UTC 3 days Sun 2026-11-08 03:10:56 UTC - e2scrub_all.timer e2scrub_all.service
- - - - fstrim.timer fstrim.service
8 timers listed.
=== timer units (all) ===
--- apt-daily-upgrade.timer [enabled/active]
OnCalendar=*-*-* 6:00
RandomizedDelaySec=60m
Persistent=true
Unit=apt-daily-upgrade.service NextElapseUSecRealtime=Thu 2026-11-12 06:50:13 UTC LastTriggerUSec=Wed 2026-11-11 06:43:16 UTC
--- apt-daily.timer [enabled/active]
OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h
Persistent=true
Unit=apt-daily.service NextElapseUSecRealtime=Wed 2026-11-11 12:30:43 UTC LastTriggerUSec=Wed 2026-11-11 01:28:16 UTC
--- apt-listchanges.timer [disabled/inactive]
OnCalendar=hourly
Unit=apt-listchanges.service NextElapseUSecRealtime= LastTriggerUSec=
--- dpkg-db-backup.timer [enabled/active]
OnCalendar=daily
Persistent=true
Unit=dpkg-db-backup.service NextElapseUSecRealtime=Thu 2026-11-12 00:00:00 UTC LastTriggerUSec=Wed 2026-11-11 00:00:14 UTC
--- e2scrub_all.timer [enabled/active]
OnCalendar=Sun *-*-* 03:10:00
RandomizedDelaySec=60
Persistent=true
Unit=e2scrub_all.service NextElapseUSecRealtime=Sun 2026-11-15 03:10:29 UTC LastTriggerUSec=Sun 2026-11-08 03:10:56 UTC
--- fstrim.timer [enabled/inactive]
OnCalendar=weekly
AccuracySec=1h
Persistent=true
RandomizedDelaySec=100min
Unit=fstrim.service NextElapseUSecRealtime= LastTriggerUSec=
--- logrotate.timer [enabled/active]
OnCalendar=daily
RandomizedDelaySec=1h
Persistent=true
Unit=logrotate.service NextElapseUSecRealtime=Thu 2026-11-12 00:17:27 UTC LastTriggerUSec=Wed 2026-11-11 01:00:14 UTC
--- man-db.timer [enabled/active]
OnCalendar=daily
RandomizedDelaySec=12h
Persistent=true
Unit=man-db.service NextElapseUSecRealtime=Thu 2026-11-12 11:29:47 UTC LastTriggerUSec=Wed 2026-11-11 09:21:14 UTC
--- systemd-tmpfiles-clean.timer [static/active]
OnBootSec=15min
OnUnitActiveSec=1d
Unit=systemd-tmpfiles-clean.service NextElapseUSecRealtime= LastTriggerUSec=Wed 2026-11-11 04:52:34 UTC
=== felhom units ===
felhom-controller-bootstrap.path loaded active running
felhom-controller-bootstrap.service loaded active exited
felhom-regen-hostkeys.service loaded inactive dead
--- felhom-controller-bootstrap.service
Type=oneshot Restart=no RestartUSec=100ms TimeoutStartUSec=infinity NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:16:28 UTC
--- felhom-regen-hostkeys.service
Type=oneshot Restart=no RestartUSec=100ms TimeoutStartUSec=infinity NRestarts=0 ActiveEnterTimestamp=
=== cron ===
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
44 * * * * root cd / && run-parts --report /etc/cron.hourly
42 6 * * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; }
43 6 * * 7 root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.weekly; }
6 6 1 * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.monthly; }
--- /etc/cron.d/e2scrub_all
25 3 * * 0 root test -e /run/systemd/system || SERVICE_MODE=1 /usr/libexec/e2fsprogs/e2scrub_all_cron
21 3 * * * root test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r
--- cron.hourly:
--- cron.daily: apt-compat dpkg logrotate man-db
--- cron.weekly: man-db
--- cron.monthly:
--- user crontabs:
=== at ===
no at
=== docker daemon timers ===
{
"features": {
"containerd-snapshotter": false
},
"live-restore": true,
"log-driver": "json-file",
"log-opts": {
"max-file": "3",
"max-size": "10m"
}
}
=== containers ===
sonarr lscr.io/linuxserver/sonarr:4.0.20 Up 5 hours (healthy) 5 hours ago
radarr lscr.io/linuxserver/radarr:6.4.4 Up 5 hours (healthy) 5 hours ago
qbittorrent qbittorrentofficial/qbittorrent-nox:5.2.4-1 Up 5 hours (healthy) 5 hours ago
prowlarr lscr.io/linuxserver/prowlarr:version-2.6.5.5623 Up 5 hours (healthy) 5 hours ago
opengist ghcr.io/thomiceli/opengist:1.15 Up 5 hours (healthy) 5 hours ago
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 19 hours (healthy) 19 hours ago
traefik traefik:v3.7.17 Up 3 days 8 days ago
filebrowser gtstef/filebrowser:1.5.8-stable Up 3 days (healthy) 13 days ago
cloudflared cloudflare/cloudflared:2026.10.4 Up 3 days (healthy) 2 weeks ago
=== restart policies + healthcheck intervals ===
/sonarr restart=unless-stopped health=30s started=2026-11-11T04:37:03.420231296Z
/radarr restart=unless-stopped health=30s started=2026-11-11T04:37:02.695192531Z
/qbittorrent restart=unless-stopped health=30s started=2026-11-11T04:37:02.078094125Z
/prowlarr restart=unless-stopped health=30s started=2026-11-11T04:37:01.619681768Z
/opengist restart=unless-stopped health=30s started=2026-11-11T04:37:01.184181029Z
/felhom-controller restart=unless-stopped health=30s started=2026-11-10T15:16:28.700768259Z
/traefik restart=unless-stopped health=none started=2026-11-08T04:35:38.624971549Z
/filebrowser restart=unless-stopped health=30s started=2026-11-08T04:35:38.716750959Z
/cloudflared restart=unless-stopped health=30s started=2026-11-08T04:35:38.687709246Z
=== controller TZ ===
TZ=unset
in-container 2026-11-11T10:03:51 UTC +0000
=== controller log: first line, line count per hour ===
2026-11-10T15:16:29.674880422Z 2026/11/10 15:16:29 [INFO] local-api: channel up (agent 169.254.253.1:8443) — guest 9201, 3 mount(s) visible
856 2026-11-10T15 1037 2026-11-10T16 1020 2026-11-10T17 1018 2026-11-10T18 1030 2026-11-10T19 1037 2026-11-10T20 1040 2026-11-10T21 1019 2026-11-10T22 1020 2026-11-10T23 1035 2026-11-11T00 1115 2026-11-11T01 1052 2026-11-11T02 1151 2026-11-11T03 1080 2026-11-11T04 1028 2026-11-11T05 1021 2026-11-11T06 1029 2026-11-11T07 1031 2026-11-11T08 1041 2026-11-11T09 76 2026-11-11T10
=== controller log: schedule words (tail 320) ===
2026-11-11T09:19:30.032745986Z 2026/11/11 09:19:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:19:31.047893767Z 2026/11/11 09:19:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.015s)
2026-11-11T09:20:29.940744236Z 2026/11/11 09:20:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:20:29.940769896Z 2026/11/11 09:20:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:20:29.940774583Z 2026/11/11 09:20:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:20:30.009309170Z 2026/11/11 09:20:30 [INFO] [scheduler] Job stack-scan completed (took 71ms)
2026-11-11T09:20:30.034500437Z 2026/11/11 09:20:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:20:31.059835845Z 2026/11/11 09:20:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.026s)
2026-11-11T09:21:29.939509198Z 2026/11/11 09:21:29 [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T09:21:29.939540940Z 2026/11/11 09:21:29 [INFO] [scheduler] Running job: backup-cache
2026-11-11T09:21:29.941504600Z 2026/11/11 09:21:29 [INFO] [scheduler] Running job: system-health
2026-11-11T09:21:29.941534001Z 2026/11/11 09:21:29 [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T09:21:29.941537787Z 2026/11/11 09:21:29 [INFO] [scheduler] Job offsite-credential-retry completed (took 1ms)
2026-11-11T09:21:29.942211529Z 2026/11/11 09:21:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:21:29.942228817Z 2026/11/11 09:21:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:21:29.945081927Z 2026/11/11 09:21:29 [INFO] [backup] Found 0 DB dump files across drives
2026-11-11T09:21:30.034951149Z 2026/11/11 09:21:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:21:30.102663774Z 2026/11/11 09:21:30 [INFO] [scheduler] Job backup-cache completed (took 164ms)
2026-11-11T09:21:30.204935092Z 2026/11/11 09:21:30 [INFO] [scheduler] Job docker-socket-users completed (took 266ms)
2026-11-11T09:21:30.250439318Z 2026/11/11 09:21:30 [INFO] [scheduler] Job system-health completed (took 311ms)
2026-11-11T09:21:31.346105744Z 2026/11/11 09:21:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.312s)
2026-11-11T09:22:29.939276939Z 2026/11/11 09:22:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:22:29.939301660Z 2026/11/11 09:22:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:22:29.939305512Z 2026/11/11 09:22:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:22:30.012521266Z 2026/11/11 09:22:30 [INFO] [scheduler] Job stack-scan completed (took 74ms)
2026-11-11T09:22:30.033434608Z 2026/11/11 09:22:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:22:31.084002257Z 2026/11/11 09:22:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.051s)
2026-11-11T09:23:29.939326665Z 2026/11/11 09:23:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:23:29.940987297Z 2026/11/11 09:23:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:23:30.033579597Z 2026/11/11 09:23:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:23:31.107122786Z 2026/11/11 09:23:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.074s)
2026-11-11T09:24:29.939215794Z 2026/11/11 09:24:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:24:29.939454810Z 2026/11/11 09:24:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:24:29.939470466Z 2026/11/11 09:24:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:24:30.017668786Z 2026/11/11 09:24:30 [INFO] [scheduler] Job stack-scan completed (took 78ms)
2026-11-11T09:24:30.033748377Z 2026/11/11 09:24:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:24:31.162157165Z 2026/11/11 09:24:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.129s)
2026-11-11T09:25:29.939161798Z 2026/11/11 09:25:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:25:29.939409395Z 2026/11/11 09:25:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:25:30.032864595Z 2026/11/11 09:25:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:25:31.183233301Z 2026/11/11 09:25:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.15s)
2026-11-11T09:26:29.940607254Z 2026/11/11 09:26:29 [INFO] [scheduler] Running job: fill-watch-interval
2026-11-11T09:26:29.940638975Z 2026/11/11 09:26:29 [INFO] [scheduler] Job fill-watch-interval completed (took 0s)
2026-11-11T09:26:29.940642119Z 2026/11/11 09:26:29 [INFO] [scheduler] Running job: backup-cache
2026-11-11T09:26:29.940763975Z 2026/11/11 09:26:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:26:29.943610557Z 2026/11/11 09:26:29 [INFO] [backup] Found 0 DB dump files across drives
2026-11-11T09:26:29.944082566Z 2026/11/11 09:26:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:26:29.944349967Z 2026/11/11 09:26:29 [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T09:26:29.944376480Z 2026/11/11 09:26:29 [INFO] [scheduler] Running job: system-health
2026-11-11T09:26:29.944381766Z 2026/11/11 09:26:29 [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T09:26:29.944585395Z 2026/11/11 09:26:29 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T09:26:29.944593819Z 2026/11/11 09:26:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:26:30.038204950Z 2026/11/11 09:26:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:26:30.071885967Z 2026/11/11 09:26:30 [INFO] [scheduler] Job stack-scan completed (took 131ms)
2026-11-11T09:26:30.165842166Z 2026/11/11 09:26:30 [INFO] [scheduler] Job backup-cache completed (took 227ms)
2026-11-11T09:26:30.171985503Z 2026/11/11 09:26:30 [INFO] [scheduler] Job docker-socket-users completed (took 227ms)
2026-11-11T09:26:30.225845525Z 2026/11/11 09:26:30 [INFO] [scheduler] Job system-health completed (took 281ms)
2026-11-11T09:26:31.294720387Z 2026/11/11 09:26:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.261s)
2026-11-11T09:27:29.939192432Z 2026/11/11 09:27:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:27:29.939217192Z 2026/11/11 09:27:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:27:30.033904651Z 2026/11/11 09:27:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:27:31.163828722Z 2026/11/11 09:27:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.13s)
2026-11-11T09:28:29.940472096Z 2026/11/11 09:28:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:28:29.940531284Z 2026/11/11 09:28:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:28:29.940538458Z 2026/11/11 09:28:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:28:30.026628913Z 2026/11/11 09:28:30 [INFO] [scheduler] Job stack-scan completed (took 87ms)
2026-11-11T09:28:30.033679277Z 2026/11/11 09:28:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:28:31.207054624Z 2026/11/11 09:28:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.173s)
2026-11-11T09:29:29.941472926Z 2026/11/11 09:29:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:29:29.941671085Z 2026/11/11 09:29:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:29:30.033599628Z 2026/11/11 09:29:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:29:31.149169007Z 2026/11/11 09:29:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.116s)
2026-11-11T09:30:29.938847460Z 2026/11/11 09:30:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:30:29.939465542Z 2026/11/11 09:30:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:30:29.939478537Z 2026/11/11 09:30:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:30:30.015935960Z 2026/11/11 09:30:30 [INFO] [scheduler] Job stack-scan completed (took 76ms)
2026-11-11T09:30:30.034569582Z 2026/11/11 09:30:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:30:31.113682420Z 2026/11/11 09:30:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.079s)
2026-11-11T09:31:29.941516773Z 2026/11/11 09:31:29 [INFO] [scheduler] Running job: hub-report
2026-11-11T09:31:29.941548062Z 2026/11/11 09:31:29 [INFO] [scheduler] Running job: backup-cache
2026-11-11T09:31:29.941551119Z 2026/11/11 09:31:29 [INFO] [backup] Found 0 DB dump files across drives
2026-11-11T09:31:29.944189386Z 2026/11/11 09:31:29 [INFO] [scheduler] Running job: system-health
2026-11-11T09:31:29.945026613Z 2026/11/11 09:31:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:31:29.945082046Z 2026/11/11 09:31:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:31:29.945202121Z 2026/11/11 09:31:29 [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T09:31:29.945292919Z 2026/11/11 09:31:29 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T09:31:29.945298582Z 2026/11/11 09:31:29 [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T09:31:30.035711429Z 2026/11/11 09:31:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:31:30.069460589Z 2026/11/11 09:31:30 [INFO] [scheduler] Job backup-cache completed (took 130ms)
2026-11-11T09:31:30.251500638Z 2026/11/11 09:31:30 [INFO] [scheduler] Job docker-socket-users completed (took 306ms)
2026-11-11T09:31:30.306432329Z 2026/11/11 09:31:30 [INFO] [sync] Catalog sync complete
2026-11-11T09:31:30.344821157Z 2026/11/11 09:31:30 [INFO] [scheduler] Job system-health completed (took 398ms)
2026-11-11T09:31:31.652261371Z 2026/11/11 09:31:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.616s)
2026-11-11T09:31:32.061972037Z 2026/11/11 09:31:32 [INFO] [scheduler] Job hub-report completed (took 2.123s)
2026-11-11T09:32:29.941413423Z 2026/11/11 09:32:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:32:29.941441173Z 2026/11/11 09:32:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:32:29.941445383Z 2026/11/11 09:32:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:32:30.021707700Z 2026/11/11 09:32:30 [INFO] [scheduler] Job stack-scan completed (took 81ms)
2026-11-11T09:32:30.033382812Z 2026/11/11 09:32:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:32:31.109247136Z 2026/11/11 09:32:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.076s)
2026-11-11T09:33:29.941508561Z 2026/11/11 09:33:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:33:29.941535555Z 2026/11/11 09:33:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:33:30.033710844Z 2026/11/11 09:33:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:33:31.048626808Z 2026/11/11 09:33:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.015s)
2026-11-11T09:34:29.938741935Z 2026/11/11 09:34:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:34:29.938857831Z 2026/11/11 09:34:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:34:29.938911235Z 2026/11/11 09:34:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:34:30.014463050Z 2026/11/11 09:34:30 [INFO] [scheduler] Job stack-scan completed (took 75ms)
2026-11-11T09:34:30.033598262Z 2026/11/11 09:34:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:34:31.149557118Z 2026/11/11 09:34:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.116s)
2026-11-11T09:35:29.938834372Z 2026/11/11 09:35:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:35:29.939201779Z 2026/11/11 09:35:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:35:30.033416828Z 2026/11/11 09:35:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:35:31.169062289Z 2026/11/11 09:35:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.136s)
2026-11-11T09:36:29.940612711Z 2026/11/11 09:36:29 [INFO] [scheduler] Running job: fill-watch-interval
2026-11-11T09:36:29.940649615Z 2026/11/11 09:36:29 [INFO] [scheduler] Job fill-watch-interval completed (took 0s)
2026-11-11T09:36:29.940652254Z 2026/11/11 09:36:29 [INFO] [scheduler] Running job: backup-cache
2026-11-11T09:36:29.940665261Z 2026/11/11 09:36:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:36:29.940668410Z 2026/11/11 09:36:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:36:29.940670532Z 2026/11/11 09:36:29 [INFO] [backup] Found 0 DB dump files across drives
2026-11-11T09:36:29.940672508Z 2026/11/11 09:36:29 [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T09:36:29.940761371Z 2026/11/11 09:36:29 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T09:36:29.941875324Z 2026/11/11 09:36:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:36:29.941894114Z 2026/11/11 09:36:29 [INFO] [scheduler] Running job: system-health
2026-11-11T09:36:29.942060419Z 2026/11/11 09:36:29 [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T09:36:30.033243949Z 2026/11/11 09:36:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:36:30.103883556Z 2026/11/11 09:36:30 [INFO] [scheduler] Job stack-scan completed (took 155ms)
2026-11-11T09:36:30.173816222Z 2026/11/11 09:36:30 [INFO] [scheduler] Job docker-socket-users completed (took 232ms)
2026-11-11T09:36:30.196014446Z 2026/11/11 09:36:30 [INFO] [scheduler] Job backup-cache completed (took 257ms)
2026-11-11T09:36:30.221705078Z 2026/11/11 09:36:30 [INFO] [scheduler] Job system-health completed (took 280ms)
2026-11-11T09:36:31.326292598Z 2026/11/11 09:36:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.293s)
2026-11-11T09:37:29.939062916Z 2026/11/11 09:37:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:37:29.939385804Z 2026/11/11 09:37:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:37:30.033685229Z 2026/11/11 09:37:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:37:31.155442754Z 2026/11/11 09:37:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.122s)
2026-11-11T09:38:29.939118414Z 2026/11/11 09:38:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:38:29.942508551Z 2026/11/11 09:38:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:38:29.942531863Z 2026/11/11 09:38:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:38:30.026780673Z 2026/11/11 09:38:30 [INFO] [scheduler] Job stack-scan completed (took 87ms)
2026-11-11T09:38:30.034520506Z 2026/11/11 09:38:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:38:31.105873817Z 2026/11/11 09:38:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.072s)
2026-11-11T09:39:29.939864710Z 2026/11/11 09:39:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:39:29.939895666Z 2026/11/11 09:39:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:39:30.032814539Z 2026/11/11 09:39:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:39:31.131566834Z 2026/11/11 09:39:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.099s)
2026-11-11T09:40:29.938961079Z 2026/11/11 09:40:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:40:29.938995582Z 2026/11/11 09:40:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:40:29.938999507Z 2026/11/11 09:40:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:40:30.016668894Z 2026/11/11 09:40:30 [INFO] [scheduler] Job stack-scan completed (took 78ms)
2026-11-11T09:40:30.035402779Z 2026/11/11 09:40:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:40:31.149009267Z 2026/11/11 09:40:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.115s)
2026-11-11T09:41:29.940693099Z 2026/11/11 09:41:29 [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T09:41:29.940737815Z 2026/11/11 09:41:29 [INFO] [scheduler] Running job: backup-cache
2026-11-11T09:41:29.940742520Z 2026/11/11 09:41:29 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T09:41:29.940745604Z 2026/11/11 09:41:29 [INFO] [backup] Found 0 DB dump files across drives
2026-11-11T09:41:29.943573777Z 2026/11/11 09:41:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:41:29.943719078Z 2026/11/11 09:41:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:41:29.943832388Z 2026/11/11 09:41:29 [INFO] [scheduler] Running job: system-health
2026-11-11T09:41:29.944494176Z 2026/11/11 09:41:29 [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T09:41:30.033933971Z 2026/11/11 09:41:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:41:30.081286147Z 2026/11/11 09:41:30 [INFO] [scheduler] Job backup-cache completed (took 142ms)
2026-11-11T09:41:30.156219314Z 2026/11/11 09:41:30 [INFO] [scheduler] Job docker-socket-users completed (took 211ms)
2026-11-11T09:41:30.192790386Z 2026/11/11 09:41:30 [INFO] [scheduler] Job system-health completed (took 249ms)
2026-11-11T09:41:31.400481678Z 2026/11/11 09:41:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.365s)
2026-11-11T09:42:29.939869431Z 2026/11/11 09:42:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:42:29.940027682Z 2026/11/11 09:42:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:42:29.940225672Z 2026/11/11 09:42:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:42:30.017638053Z 2026/11/11 09:42:30 [INFO] [scheduler] Job stack-scan completed (took 77ms)
2026-11-11T09:42:30.032884786Z 2026/11/11 09:42:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:42:31.210835039Z 2026/11/11 09:42:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.178s)
2026-11-11T09:43:29.942441119Z 2026/11/11 09:43:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:43:29.942469213Z 2026/11/11 09:43:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:43:30.033142287Z 2026/11/11 09:43:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:43:31.197181516Z 2026/11/11 09:43:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.164s)
2026-11-11T09:44:29.939367673Z 2026/11/11 09:44:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:44:29.939389720Z 2026/11/11 09:44:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:44:29.939393976Z 2026/11/11 09:44:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:44:30.008640544Z 2026/11/11 09:44:30 [INFO] [scheduler] Job stack-scan completed (took 69ms)
2026-11-11T09:44:30.033536834Z 2026/11/11 09:44:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:44:31.097227054Z 2026/11/11 09:44:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.064s)
2026-11-11T09:45:29.939259992Z 2026/11/11 09:45:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:45:29.939297495Z 2026/11/11 09:45:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:45:30.032994407Z 2026/11/11 09:45:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:45:31.070440102Z 2026/11/11 09:45:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.037s)
2026-11-11T09:46:29.939450882Z 2026/11/11 09:46:29 [INFO] [scheduler] Running job: fill-watch-interval
2026-11-11T09:46:29.939497086Z 2026/11/11 09:46:29 [INFO] [scheduler] Job fill-watch-interval completed (took 0s)
2026-11-11T09:46:29.939499258Z 2026/11/11 09:46:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:46:29.939772208Z 2026/11/11 09:46:29 [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T09:46:29.939784524Z 2026/11/11 09:46:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:46:29.939786985Z 2026/11/11 09:46:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:46:29.942545161Z 2026/11/11 09:46:29 [INFO] [scheduler] Running job: hub-report
2026-11-11T09:46:29.944289379Z 2026/11/11 09:46:29 [INFO] [scheduler] Running job: system-health
2026-11-11T09:46:29.944316340Z 2026/11/11 09:46:29 [INFO] [scheduler] Running job: backup-cache
2026-11-11T09:46:29.949012043Z 2026/11/11 09:46:29 [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T09:46:29.949056892Z 2026/11/11 09:46:29 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T09:46:30.038628812Z 2026/11/11 09:46:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:46:30.146613911Z 2026/11/11 09:46:30 [INFO] [scheduler] Job stack-scan completed (took 203ms)
2026-11-11T09:46:30.150177820Z 2026/11/11 09:46:30 [INFO] [backup] Found 0 DB dump files across drives
2026-11-11T09:46:30.263896201Z 2026/11/11 09:46:30 [INFO] [scheduler] Job docker-socket-users completed (took 324ms)
2026-11-11T09:46:30.314429495Z 2026/11/11 09:46:30 [INFO] [scheduler] Job system-health completed (took 371ms)
2026-11-11T09:46:30.377003979Z 2026/11/11 09:46:30 [INFO] [sync] Catalog sync complete
2026-11-11T09:46:31.723468696Z 2026/11/11 09:46:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.685s)
2026-11-11T09:46:31.761256256Z 2026/11/11 09:46:31 [INFO] [scheduler] Job backup-cache completed (took 1.818s)
2026-11-11T09:46:32.132913551Z 2026/11/11 09:46:32 [INFO] [scheduler] Job hub-report completed (took 2.19s)
2026-11-11T09:47:29.939089006Z 2026/11/11 09:47:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:47:29.940757607Z 2026/11/11 09:47:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:47:30.033145375Z 2026/11/11 09:47:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:47:31.116187602Z 2026/11/11 09:47:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.083s)
2026-11-11T09:48:29.940607224Z 2026/11/11 09:48:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:48:29.944727640Z 2026/11/11 09:48:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:48:29.944753009Z 2026/11/11 09:48:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:48:30.024438338Z 2026/11/11 09:48:30 [INFO] [scheduler] Job stack-scan completed (took 83ms)
2026-11-11T09:48:30.034601093Z 2026/11/11 09:48:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:48:31.082565125Z 2026/11/11 09:48:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.049s)
2026-11-11T09:49:29.939378064Z 2026/11/11 09:49:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:49:29.941989144Z 2026/11/11 09:49:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:49:30.033433512Z 2026/11/11 09:49:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:49:31.125298440Z 2026/11/11 09:49:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.092s)
2026-11-11T09:50:29.939798950Z 2026/11/11 09:50:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:50:29.942571272Z 2026/11/11 09:50:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:50:29.942708176Z 2026/11/11 09:50:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:50:30.033187088Z 2026/11/11 09:50:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:50:30.037636389Z 2026/11/11 09:50:30 [INFO] [scheduler] Job stack-scan completed (took 95ms)
2026-11-11T09:50:31.096643566Z 2026/11/11 09:50:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.063s)
2026-11-11T09:51:29.940226248Z 2026/11/11 09:51:29 [INFO] [scheduler] Running job: backup-cache
2026-11-11T09:51:29.940267126Z 2026/11/11 09:51:29 [INFO] [backup] Found 0 DB dump files across drives
2026-11-11T09:51:29.940596892Z 2026/11/11 09:51:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:51:29.940617671Z 2026/11/11 09:51:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:51:29.943677817Z 2026/11/11 09:51:29 [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T09:51:29.944251922Z 2026/11/11 09:51:29 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T09:51:29.944407613Z 2026/11/11 09:51:29 [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T09:51:29.944675022Z 2026/11/11 09:51:29 [INFO] [scheduler] Running job: system-health
2026-11-11T09:51:30.033353310Z 2026/11/11 09:51:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:51:30.044089267Z 2026/11/11 09:51:30 [INFO] [scheduler] Job backup-cache completed (took 106ms)
2026-11-11T09:51:30.159339236Z 2026/11/11 09:51:30 [INFO] [scheduler] Job docker-socket-users completed (took 215ms)
2026-11-11T09:51:30.192138415Z 2026/11/11 09:51:30 [INFO] [scheduler] Job system-health completed (took 247ms)
2026-11-11T09:51:31.337185333Z 2026/11/11 09:51:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.304s)
2026-11-11T09:52:29.939605526Z 2026/11/11 09:52:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:52:29.939628545Z 2026/11/11 09:52:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:52:29.939632737Z 2026/11/11 09:52:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:52:30.013892871Z 2026/11/11 09:52:30 [INFO] [scheduler] Job stack-scan completed (took 74ms)
2026-11-11T09:52:30.038448202Z 2026/11/11 09:52:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:52:31.062480482Z 2026/11/11 09:52:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.027s)
2026-11-11T09:53:29.939189707Z 2026/11/11 09:53:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:53:29.939231431Z 2026/11/11 09:53:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:53:30.033878382Z 2026/11/11 09:53:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:53:31.128207147Z 2026/11/11 09:53:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.095s)
2026-11-11T09:54:29.941475063Z 2026/11/11 09:54:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:54:29.941500202Z 2026/11/11 09:54:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:54:29.941504121Z 2026/11/11 09:54:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:54:30.015818171Z 2026/11/11 09:54:30 [INFO] [scheduler] Job stack-scan completed (took 77ms)
2026-11-11T09:54:30.033142254Z 2026/11/11 09:54:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:54:31.055222046Z 2026/11/11 09:54:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.022s)
2026-11-11T09:55:29.939173324Z 2026/11/11 09:55:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:55:29.939204929Z 2026/11/11 09:55:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:55:30.033725359Z 2026/11/11 09:55:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:55:31.088110813Z 2026/11/11 09:55:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.054s)
2026-11-11T09:56:29.940873666Z 2026/11/11 09:56:29 [INFO] [scheduler] Running job: backup-cache
2026-11-11T09:56:29.940901521Z 2026/11/11 09:56:29 [INFO] [scheduler] Running job: fill-watch-interval
2026-11-11T09:56:29.940908808Z 2026/11/11 09:56:29 [INFO] [scheduler] Job fill-watch-interval completed (took 0s)
2026-11-11T09:56:29.940911388Z 2026/11/11 09:56:29 [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T09:56:29.940913857Z 2026/11/11 09:56:29 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T09:56:29.940916417Z 2026/11/11 09:56:29 [INFO] [scheduler] Running job: system-health
2026-11-11T09:56:29.941686823Z 2026/11/11 09:56:29 [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T09:56:29.943096654Z 2026/11/11 09:56:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:56:29.955313406Z 2026/11/11 09:56:29 [INFO] [backup] Found 0 DB dump files across drives
2026-11-11T09:56:29.955489900Z 2026/11/11 09:56:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:56:29.955608054Z 2026/11/11 09:56:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:56:30.033368984Z 2026/11/11 09:56:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:56:30.071586073Z 2026/11/11 09:56:30 [INFO] [scheduler] Job stack-scan completed (took 128ms)
2026-11-11T09:56:30.160507282Z 2026/11/11 09:56:30 [INFO] [scheduler] Job docker-socket-users completed (took 218ms)
2026-11-11T09:56:30.181769446Z 2026/11/11 09:56:30 [INFO] [scheduler] Job backup-cache completed (took 243ms)
2026-11-11T09:56:30.246207320Z 2026/11/11 09:56:30 [INFO] [scheduler] Job system-health completed (took 306ms)
2026-11-11T09:56:31.376228125Z 2026/11/11 09:56:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.343s)
2026-11-11T09:57:29.938865122Z 2026/11/11 09:57:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:57:29.938888865Z 2026/11/11 09:57:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:57:30.033473166Z 2026/11/11 09:57:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:57:31.115547788Z 2026/11/11 09:57:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.082s)
2026-11-11T09:58:29.941618096Z 2026/11/11 09:58:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:58:29.941647476Z 2026/11/11 09:58:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:58:29.941652112Z 2026/11/11 09:58:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:58:30.017199930Z 2026/11/11 09:58:30 [INFO] [scheduler] Job stack-scan completed (took 78ms)
2026-11-11T09:58:30.034666906Z 2026/11/11 09:58:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:58:31.145412165Z 2026/11/11 09:58:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.11s)
2026-11-11T09:59:29.942485597Z 2026/11/11 09:59:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:59:29.942509543Z 2026/11/11 09:59:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:59:30.032918207Z 2026/11/11 09:59:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:59:31.187962617Z 2026/11/11 09:59:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.155s)
2026-11-11T10:00:29.939789317Z 2026/11/11 10:00:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T10:00:29.940257245Z 2026/11/11 10:00:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T10:00:29.940279501Z 2026/11/11 10:00:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T10:00:30.016476119Z 2026/11/11 10:00:30 [INFO] [scheduler] Job stack-scan completed (took 75ms)
2026-11-11T10:00:30.034280832Z 2026/11/11 10:00:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T10:00:31.133303785Z 2026/11/11 10:00:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.099s)
2026-11-11T10:01:29.938757701Z 2026/11/11 10:01:29 [INFO] [scheduler] Running job: backup-cache
2026-11-11T10:01:29.942223659Z 2026/11/11 10:01:29 [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T10:01:29.942330274Z 2026/11/11 10:01:29 [INFO] [scheduler] Running job: hub-report
2026-11-11T10:01:29.944029296Z 2026/11/11 10:01:29 [INFO] [backup] Found 0 DB dump files across drives
2026-11-11T10:01:29.945116583Z 2026/11/11 10:01:29 [INFO] [scheduler] Running job: system-health
2026-11-11T10:01:29.949333886Z 2026/11/11 10:01:29 [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T10:01:29.949408711Z 2026/11/11 10:01:29 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T10:01:29.949444935Z 2026/11/11 10:01:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T10:01:29.949471164Z 2026/11/11 10:01:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T10:01:30.035400222Z 2026/11/11 10:01:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T10:01:30.258333370Z 2026/11/11 10:01:30 [INFO] [scheduler] Job docker-socket-users completed (took 316ms)
2026-11-11T10:01:30.273972608Z 2026/11/11 10:01:30 [INFO] [sync] Catalog sync complete
2026-11-11T10:01:30.315878773Z 2026/11/11 10:01:30 [INFO] [scheduler] Job system-health completed (took 371ms)
2026-11-11T10:01:31.590833808Z 2026/11/11 10:01:31 [INFO] [scheduler] Job backup-cache completed (took 1.649s)
2026-11-11T10:01:31.729413745Z 2026/11/11 10:01:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.694s)
2026-11-11T10:01:31.963664882Z 2026/11/11 10:01:31 [INFO] [scheduler] Job hub-report completed (took 2.021s)
2026-11-11T10:02:29.939771665Z 2026/11/11 10:02:29 [INFO] [scheduler] Running job: stack-scan
2026-11-11T10:02:29.939857115Z 2026/11/11 10:02:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T10:02:29.939863878Z 2026/11/11 10:02:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T10:02:30.013085616Z 2026/11/11 10:02:30 [INFO] [scheduler] Job stack-scan completed (took 73ms)
2026-11-11T10:02:30.033880349Z 2026/11/11 10:02:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T10:02:31.093182115Z 2026/11/11 10:02:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.06s)
2026-11-11T10:03:29.939938323Z 2026/11/11 10:03:29 [INFO] [scheduler] Running job: hold-report
2026-11-11T10:03:29.939961541Z 2026/11/11 10:03:29 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T10:03:30.033226002Z 2026/11/11 10:03:30 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T10:03:31.150124665Z 2026/11/11 10:03:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.117s)
RC=0
@@ -0,0 +1,228 @@
# guest-night demo-felhom taken 2026-11-11T10:09:28Z
log first line: 2026-11-10T15:16:29.674880422Z
log last line: 2026-11-11T10:08:49.976032577Z
=== scheduler jobs (UTC): runs, first start, last start, last three "took" ===
agent-channel-health runs=1132 first=2026-11-10T15:17:30 last=2026-11-11T10:08:30 [completed (took 1.297s)] [completed (took 1.041s)] [completed (took 1.06s)]
backup-cache runs=226 first=2026-11-10T15:21:29 last=2026-11-11T10:06:29 [completed (took 243ms)] [completed (took 1.649s)] [completed (took 220ms)]
conversion-copy-release runs=18 first=2026-11-10T16:16:29 last=2026-11-11T09:16:29 [completed (took 0s)] [completed (took 0s)] [completed (took 288ms)]
db-dump runs=1 first=2026-11-11T01:30:00 last=2026-11-11T01:30:00 [completed (took 18.336s)]
disk-health-check runs=18 first=2026-11-10T16:16:30 last=2026-11-11T09:16:30 [completed (took 1.965s)] [completed (took 3.058s)] [completed (took 2.176s)]
docker-socket-users runs=226 first=2026-11-10T15:21:29 last=2026-11-11T10:06:29 [completed (took 218ms)] [completed (took 316ms)] [completed (took 188ms)]
fill-watch runs=1 first=2026-11-11T02:30:00 last=2026-11-11T02:30:00 [completed (took 0s)]
fill-watch-interval runs=113 first=2026-11-10T15:26:29 last=2026-11-11T10:06:29 [completed (took 0s)] [completed (took 0s)] [completed (took 10ms)]
geo-verify runs=3 first=2026-11-10T21:16:29 last=2026-11-11T09:16:29 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)]
hold-report runs=1132 first=2026-11-10T15:17:29 last=2026-11-11T10:08:29 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)]
hub-report runs=75 first=2026-11-10T15:31:29 last=2026-11-11T10:01:29 [completed (took 2.123s)] [completed (took 2.19s)] [completed (took 2.021s)]
metrics-prune runs=1 first=2026-11-11T03:00:00 last=2026-11-11T03:00:00 [completed (took 59ms)]
offbox-backup runs=1 first=2026-11-11T03:15:00 last=2026-11-11T03:15:00 [completed (took 38.095s)]
offsite-abandon-sweep runs=1 first=2026-11-11T04:10:00 last=2026-11-11T04:10:00 [completed (took 0s)]
offsite-credential-retry runs=226 first=2026-11-10T15:21:29 last=2026-11-11T10:06:29 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)]
offsite-integrity runs=1 first=2026-11-11T05:00:00 last=2026-11-11T05:00:00 [completed (took 0s)]
offsite-proof runs=1 first=2026-11-11T04:30:00 last=2026-11-11T04:30:00 [completed (took 3.044s)]
restore-copy-sweep runs=3 first=2026-11-10T21:16:29 last=2026-11-11T09:16:29 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)]
selfupdate-check runs=3 first=2026-11-10T21:16:29 last=2026-11-11T09:16:29 [completed (took 49ms)] [completed (took 62ms)] [completed (took 72ms)]
stack-scan runs=566 first=2026-11-10T15:18:29 last=2026-11-11T10:08:29 [completed (took 78ms)] [completed (took 156ms)] [completed (took 68ms)]
system-health runs=226 first=2026-11-10T15:21:29 last=2026-11-11T10:06:29 [completed (took 306ms)] [completed (took 371ms)] [completed (took 278ms)]
tier2-backup runs=1 first=2026-11-11T02:30:00 last=2026-11-11T02:30:00 [completed (took 11.412s)]
=== jobs that ran fewer than 30 times: every start and end (UTC) ===
2026/11/10 16:16:29 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/10 16:16:30 [INFO] [scheduler] Running job: disk-health-check
2026/11/10 16:16:30 [INFO] [scheduler] Job conversion-copy-release completed (took 230ms)
2026/11/10 16:16:31 [INFO] [scheduler] Job disk-health-check completed (took 1.869s)
2026/11/10 17:16:29 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/10 17:16:30 [INFO] [scheduler] Running job: disk-health-check
2026/11/10 17:16:30 [INFO] [scheduler] Job conversion-copy-release completed (took 257ms)
2026/11/10 17:16:31 [INFO] [scheduler] Job disk-health-check completed (took 1.78s)
2026/11/10 18:16:29 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/10 18:16:30 [INFO] [scheduler] Running job: disk-health-check
2026/11/10 18:16:30 [INFO] [scheduler] Job conversion-copy-release completed (took 249ms)
2026/11/10 18:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.132s)
2026/11/10 19:16:29 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/10 19:16:29 [INFO] [scheduler] Job conversion-copy-release completed (took 1ms)
2026/11/10 19:16:30 [INFO] [scheduler] Running job: disk-health-check
2026/11/10 19:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.138s)
2026/11/10 20:16:29 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/10 20:16:29 [INFO] [scheduler] Job conversion-copy-release completed (took 1ms)
2026/11/10 20:16:30 [INFO] [scheduler] Running job: disk-health-check
2026/11/10 20:16:31 [INFO] [scheduler] Job disk-health-check completed (took 1.813s)
2026/11/10 21:16:29 [INFO] [scheduler] Running job: selfupdate-check
2026/11/10 21:16:29 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/10 21:16:29 [INFO] [scheduler] Running job: restore-copy-sweep
2026/11/10 21:16:29 [INFO] [scheduler] Job restore-copy-sweep completed (took 0s)
2026/11/10 21:16:29 [INFO] [scheduler] Running job: geo-verify
2026/11/10 21:16:29 [INFO] [scheduler] Job geo-verify completed (took 0s)
2026/11/10 21:16:29 [INFO] [scheduler] Job selfupdate-check completed (took 49ms)
2026/11/10 21:16:30 [INFO] [scheduler] Running job: disk-health-check
2026/11/10 21:16:30 [INFO] [scheduler] Job conversion-copy-release completed (took 260ms)
2026/11/10 21:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.038s)
2026/11/10 22:16:29 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/10 22:16:30 [INFO] [scheduler] Running job: disk-health-check
2026/11/10 22:16:30 [INFO] [scheduler] Job conversion-copy-release completed (took 180ms)
2026/11/10 22:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.013s)
2026/11/10 23:16:29 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/10 23:16:29 [INFO] [scheduler] Job conversion-copy-release completed (took 29ms)
2026/11/10 23:16:30 [INFO] [scheduler] Running job: disk-health-check
2026/11/10 23:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.292s)
2026/11/11 00:16:29 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 00:16:29 [INFO] [scheduler] Job conversion-copy-release completed (took 0s)
2026/11/11 00:16:30 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 00:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.29s)
2026/11/11 01:16:29 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 01:16:30 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 01:16:30 [INFO] [scheduler] Job conversion-copy-release completed (took 188ms)
2026/11/11 01:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.276s)
2026/11/11 01:30:00 [INFO] [scheduler] Running job: db-dump
2026/11/11 01:30:18 [INFO] [scheduler] Job db-dump completed (took 18.336s)
2026/11/11 02:16:29 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 02:16:30 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 02:16:30 [INFO] [scheduler] Job conversion-copy-release completed (took 93ms)
2026/11/11 02:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.166s)
2026/11/11 02:30:00 [INFO] [scheduler] Running job: tier2-backup
2026/11/11 02:30:00 [INFO] [scheduler] Running job: fill-watch
2026/11/11 02:30:00 [INFO] [scheduler] Job fill-watch completed (took 0s)
2026/11/11 02:30:11 [INFO] [scheduler] Job tier2-backup completed (took 11.412s)
2026/11/11 03:00:00 [INFO] [scheduler] Running job: metrics-prune
2026/11/11 03:00:00 [INFO] [scheduler] Job metrics-prune completed (took 59ms)
2026/11/11 03:15:00 [INFO] [scheduler] Running job: offbox-backup
2026/11/11 03:15:38 [INFO] [scheduler] Job offbox-backup completed (took 38.095s)
2026/11/11 03:16:29 [INFO] [scheduler] Running job: selfupdate-check
2026/11/11 03:16:29 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 03:16:29 [INFO] [scheduler] Running job: restore-copy-sweep
2026/11/11 03:16:29 [INFO] [scheduler] Job restore-copy-sweep completed (took 0s)
2026/11/11 03:16:29 [INFO] [scheduler] Job conversion-copy-release completed (took 0s)
2026/11/11 03:16:29 [INFO] [scheduler] Running job: geo-verify
2026/11/11 03:16:29 [INFO] [scheduler] Job geo-verify completed (took 0s)
2026/11/11 03:16:30 [INFO] [scheduler] Job selfupdate-check completed (took 62ms)
2026/11/11 03:16:30 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 03:16:31 [INFO] [scheduler] Job disk-health-check completed (took 1.751s)
2026/11/11 04:10:00 [INFO] [scheduler] Running job: offsite-abandon-sweep
2026/11/11 04:10:00 [INFO] [scheduler] Job offsite-abandon-sweep completed (took 0s)
2026/11/11 04:16:29 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 04:16:30 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 04:16:30 [INFO] [scheduler] Job conversion-copy-release completed (took 316ms)
2026/11/11 04:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.328s)
2026/11/11 04:30:00 [INFO] [scheduler] Running job: offsite-proof
2026/11/11 04:30:03 [INFO] [scheduler] Job offsite-proof completed (took 3.044s)
2026/11/11 05:00:00 [INFO] [scheduler] Running job: offsite-integrity
2026/11/11 05:00:00 [INFO] [scheduler] Job offsite-integrity completed (took 0s)
2026/11/11 05:16:29 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 05:16:29 [INFO] [scheduler] Job conversion-copy-release completed (took 1ms)
2026/11/11 05:16:30 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 05:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.099s)
2026/11/11 06:16:29 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 06:16:29 [INFO] [scheduler] Job conversion-copy-release completed (took 0s)
2026/11/11 06:16:30 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 06:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.355s)
2026/11/11 07:16:29 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 07:16:29 [INFO] [scheduler] Job conversion-copy-release completed (took 0s)
2026/11/11 07:16:30 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 07:16:32 [INFO] [scheduler] Job disk-health-check completed (took 1.965s)
2026/11/11 08:16:29 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 08:16:29 [INFO] [scheduler] Job conversion-copy-release completed (took 0s)
2026/11/11 08:16:30 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 08:16:33 [INFO] [scheduler] Job disk-health-check completed (took 3.058s)
2026/11/11 09:16:29 [INFO] [scheduler] Running job: selfupdate-check
2026/11/11 09:16:29 [INFO] [scheduler] Running job: restore-copy-sweep
2026/11/11 09:16:29 [INFO] [scheduler] Job restore-copy-sweep completed (took 0s)
2026/11/11 09:16:29 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 09:16:29 [INFO] [scheduler] Running job: geo-verify
2026/11/11 09:16:29 [INFO] [scheduler] Job geo-verify completed (took 0s)
2026/11/11 09:16:30 [INFO] [scheduler] Job selfupdate-check completed (took 72ms)
2026/11/11 09:16:30 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 09:16:30 [INFO] [scheduler] Job conversion-copy-release completed (took 288ms)
2026/11/11 09:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.176s)
=== night lines 19:00-08:00 UTC, INFO/WARN/ERROR, backup words (first 2 per message shape) ===
2026/11/10 19:01:29 [INFO] [backup] Found 0 DB dump files across drives
2026/11/10 19:01:30 [INFO] [backup] Discovered 0 databases
2026/11/10 19:01:30 [INFO] [backup] Discovered app data: 5 apps
2026/11/10 19:01:30 [INFO] [monitor] Health check: status=ok
2026/11/10 19:01:31 [INFO] [backup] Backup status cache refreshed
2026/11/10 19:01:31 [INFO] [monitor] Health check: status=ok
2026/11/10 19:06:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 19:06:30 [INFO] [backup] Found 0 DB dump files across drives
2026/11/10 19:06:30 [INFO] [backup] Discovered 0 databases
2026/11/10 19:06:30 [INFO] [backup] Discovered app data: 5 apps
2026/11/10 19:06:30 [INFO] [backup] Backup status cache refreshed
2026/11/10 19:07:29 [INFO] [deadapp] check alive: 460 scans since boot, 5 deployed app(s) evaluated, 0 currently down
2026/11/10 19:16:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 19:16:32 [INFO] [web] disk-health check complete: 2 disk(s) evaluated, 0 alert(s)
2026/11/10 19:17:29 [INFO] [deadapp] check alive: 480 scans since boot, 5 deployed app(s) evaluated, 0 currently down
2026/11/10 20:16:31 [INFO] [web] disk-health check complete: 2 disk(s) evaluated, 0 alert(s)
2026/11/11 01:30:00 [INFO] [backup] Starting database dump run
2026/11/11 01:30:00 [INFO] [backup] No database containers found
2026/11/11 01:30:00 [INFO] [backup] Stopping opengist for safe volume dump
2026/11/11 01:30:00 [INFO] [backup] Volume dump: opengist/opengist_opengist_data → 182.5 KB
2026/11/11 01:30:00 [INFO] [backup] Restarting opengist after volume dump
2026/11/11 01:30:00 [INFO] [backup] Stopping prowlarr for safe volume dump
2026/11/11 01:30:04 [WARN] [backup] prowlarr: template volume entry ignored — volume "prowlarr_config": class "important" — only "excluded" is meaningful for a named volume
2026/11/11 01:30:04 [INFO] [backup] Volume dump: prowlarr/prowlarr_prowlarr_config → 6.5 MB
2026/11/11 01:30:04 [INFO] [backup] Restarting prowlarr after volume dump
2026/11/11 01:30:04 [INFO] [backup] Stopping radarr for safe volume dump
2026/11/11 01:30:11 [INFO] [backup] Volume dump: radarr/radarr_radarr_config → 7.0 MB
2026/11/11 01:30:11 [INFO] [backup] Restarting radarr after volume dump
2026/11/11 01:30:11 [INFO] [backup] Stopping sonarr for safe volume dump
2026/11/11 01:30:14 [INFO] [busymark] busy: a backup, a restore or an off-site run is running — an operator reboot of the box is refused until it ends
2026/11/11 01:30:16 [INFO] [backup] Volume dump: sonarr/sonarr_sonarr_config → 25.7 MB
2026/11/11 01:30:16 [INFO] [backup] Restarting sonarr after volume dump
2026/11/11 01:30:16 [INFO] [backup] Stopping qbittorrent for safe volume dump
2026/11/11 01:30:17 [WARN] [backup] qbittorrent: template volume entry ignored — volume "qbittorrent_config": class "important" — only "excluded" is meaningful for a named volume
2026/11/11 01:30:17 [INFO] [backup] Volume dump: qbittorrent/qbittorrent_qbittorrent_config → 9.9 MB
2026/11/11 01:30:17 [INFO] [backup] Restarting qbittorrent after volume dump
2026/11/11 01:30:18 [INFO] [backup] App-data backup completed: 0 databases (0 B total), 5 volume dump(s) (18.202s)
2026/11/11 01:30:18 [INFO] [backup] Recovery unit captured for opengist → /mnt/sys_drive/felhom-data/backups/primary/opengist (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/11/11 01:30:18 [INFO] [backup] Recovery unit captured for prowlarr → /mnt/sys_drive/felhom-data/backups/primary/prowlarr (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/11/11 01:30:18 [INFO] [backup] Recovery unit captured for radarr → /mnt/felhom-drives/hdd_1/backups/primary/radarr (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/11/11 01:30:18 [INFO] [backup] Recovery unit captured for sonarr → /mnt/felhom-drives/hdd_1/backups/primary/sonarr (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/11/11 01:30:18 [INFO] [backup] Recovery unit captured for qbittorrent → /mnt/felhom-drives/hdd_1/backups/primary/qbittorrent (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1)
2026/11/11 01:30:18 [INFO] Event pushed: db_dump_completed (info) [hu-only] — Adatbázis mentés elkészült
2026/11/11 01:30:30 [INFO] [busymark] idle again (was: a backup, a restore or an off-site run is running) — an operator reboot of the box is no longer refused for it
2026/11/11 02:30:02 [INFO] [backup] Tier 2 copied opengist → /mnt/felhom-drives/hdd_1/backups/secondary/opengist (191.0 KB, 0 leg(s), 0s)
2026/11/11 02:30:05 [INFO] [backup] Tier 2 copied prowlarr → /mnt/felhom-drives/hdd_1/backups/secondary/prowlarr (6.5 MB, 0 leg(s), 1s)
2026/11/11 02:30:07 [INFO] [backup] Tier 2 copied radarr → /mnt/sys_drive/felhom-data/backups/secondary/radarr (7.0 MB, 0 leg(s), 0s) [SSD: state-only]
2026/11/11 02:30:09 [INFO] [backup] Tier 2 copied sonarr → /mnt/sys_drive/felhom-data/backups/secondary/sonarr (25.7 MB, 0 leg(s), 0s) [SSD: state-only]
2026/11/11 02:30:11 [INFO] [backup] Tier 2 copied qbittorrent → /mnt/sys_drive/felhom-data/backups/secondary/qbittorrent (9.9 MB, 0 leg(s), 0s) [SSD: state-only]
2026/11/11 02:30:11 [INFO] [backup] Tier 2 run complete: 5 app(s) processed (incl. volume-only — F6)
2026/11/11 03:00:00 [INFO] Pruned 8640 old metric rows
2026/11/11 03:15:00 [INFO] [offbox] backup run started (5 app(s) toggled)
2026/11/11 03:15:00 [INFO] [backup] Starting database dump run
2026/11/11 03:15:00 [INFO] [backup] No database containers found
2026/11/11 03:15:00 [INFO] [backup] Stopping opengist for safe volume dump
2026/11/11 03:15:00 [INFO] [backup] Volume dump: opengist/opengist_opengist_data → 182.5 KB
2026/11/11 03:15:00 [INFO] [backup] Restarting opengist after volume dump
2026/11/11 03:15:00 [INFO] [backup] Stopping prowlarr for safe volume dump
2026/11/11 03:15:04 [WARN] [backup] prowlarr: template volume entry ignored — volume "prowlarr_config": class "important" — only "excluded" is meaningful for a named volume
2026/11/11 03:15:04 [INFO] [backup] Volume dump: prowlarr/prowlarr_prowlarr_config → 6.5 MB
2026/11/11 03:15:04 [INFO] [backup] Restarting prowlarr after volume dump
2026/11/11 03:15:04 [INFO] [backup] Stopping radarr for safe volume dump
2026/11/11 03:15:11 [INFO] [backup] Volume dump: radarr/radarr_radarr_config → 7.0 MB
2026/11/11 03:15:11 [INFO] [backup] Restarting radarr after volume dump
2026/11/11 03:15:11 [INFO] [backup] Stopping sonarr for safe volume dump
2026/11/11 03:15:14 [INFO] [busymark] busy: a backup, a restore or an off-site run is running — an operator reboot of the box is refused until it ends
2026/11/11 03:15:15 [INFO] [backup] Volume dump: sonarr/sonarr_sonarr_config → 23.5 MB
2026/11/11 03:15:15 [INFO] [backup] Restarting sonarr after volume dump
2026/11/11 03:15:16 [INFO] [backup] Stopping qbittorrent for safe volume dump
2026/11/11 03:15:16 [WARN] [backup] qbittorrent: template volume entry ignored — volume "qbittorrent_config": class "important" — only "excluded" is meaningful for a named volume
2026/11/11 03:15:17 [INFO] [backup] Volume dump: qbittorrent/qbittorrent_qbittorrent_config → 9.9 MB
2026/11/11 03:15:17 [INFO] [backup] Restarting qbittorrent after volume dump
2026/11/11 03:15:18 [INFO] [backup] App-data backup completed: 0 databases (0 B total), 5 volume dump(s) (18.031s)
2026/11/11 03:15:18 [INFO] [backup] Recovery unit captured for opengist → /mnt/sys_drive/felhom-data/backups/primary/opengist (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/11/11 03:15:18 [INFO] [backup] Recovery unit captured for prowlarr → /mnt/sys_drive/felhom-data/backups/primary/prowlarr (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/11/11 03:15:18 [INFO] [backup] Recovery unit captured for radarr → /mnt/felhom-drives/hdd_1/backups/primary/radarr (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/11/11 03:15:18 [INFO] [backup] Recovery unit captured for sonarr → /mnt/felhom-drives/hdd_1/backups/primary/sonarr (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/11/11 03:15:18 [INFO] [backup] Recovery unit captured for qbittorrent → /mnt/felhom-drives/hdd_1/backups/primary/qbittorrent (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1)
2026/11/11 03:15:18 [INFO] [offbox] pre-push dump leg completed in 18.172s — snapshot pair is coherent
2026/11/11 03:15:21 [WARN] [offbox] prowlarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable)
2026/11/11 03:15:23 [INFO] [offbox] backed up radarr (/mnt/felhom-drives/hdd_1/backups/primary/radarr, 0 mandatory path(s))
2026/11/11 03:15:26 [INFO] [offbox] backed up sonarr (/mnt/felhom-drives/hdd_1/backups/primary/sonarr, 0 mandatory path(s))
2026/11/11 03:15:28 [INFO] [offbox] backed up opengist (/mnt/sys_drive/felhom-data/backups/primary/opengist, 0 mandatory path(s))
2026/11/11 03:15:28 [WARN] [offbox] prowlarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable)
2026/11/11 03:15:30 [INFO] [offbox] backed up prowlarr (/mnt/sys_drive/felhom-data/backups/primary/prowlarr, 0 mandatory path(s))
2026/11/11 03:15:32 [INFO] [offbox] backed up qbittorrent (/mnt/felhom-drives/hdd_1/backups/primary/qbittorrent, 0 mandatory path(s))
2026/11/11 03:15:33 [INFO] [offbox] retention skipped (after-run): no clean-up window now (not due (last window 2026-11-09T03:16:07Z)) — nothing deleted (decision 68)
2026/11/11 03:15:38 [INFO] [offbox] backup OK: 5 app(s) backed up, 67 snapshot(s), 35s
2026/11/11 03:15:38 [INFO] [update-leg] started (after-offsite): window 02:30, no step starts at or after 07:30
2026/11/11 03:15:38 [INFO] [update-leg] update leg (after-offsite): done=0 undone=0 held=0 failed=0 skipped=0 in 0s [skipped: ]
RC=0
@@ -0,0 +1,665 @@
# host-clock demo-felhom taken 2026-11-11T10:03:57Z from DooPlex
=== in-flight ===
checked
=== clock ===
utc 2026-11-11T10:03:24.716263814Z
local 2026-11-11T11:03:24 CET +0100
/usr/share/zoneinfo/Europe/Budapest
Timezone=Europe/Budapest
LocalRTC=no
CanNTP=yes
NTP=no
NTPSynchronized=no
TimeUSec=Wed 2026-11-11 11:03:24 CET
RTCTimeUSec=Wed 2026-11-11 11:03:21 CET
Local time: Wed 2026-11-11 11:03:24 CET
Universal time: Wed 2026-11-11 10:03:24 UTC
RTC time: Wed 2026-11-11 10:03:21
Time zone: Europe/Budapest (CET, +0100)
System clock synchronized: no
NTP service: inactive
RTC in local TZ: no
=== who keeps the time ===
chrony: failed
enabled: enabled
chronyd: failed
enabled: alias
systemd-timesyncd: inactive
enabled: not-found
ntp: inactive
enabled: not-found
ntpsec: inactive
enabled: not-found
openntpd: inactive
enabled: not-found
506 Cannot talk to daemon
506 Cannot talk to daemon
Failed to query server: The name org.freedesktop.timesync1 was not provided by any .service files
pool 2.debian.pool.ntp.org iburst
sourcedir /run/chrony-dhcp
sourcedir /etc/chrony/sources.d
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/chrony.drift
ntsdumpdir /var/lib/chrony
logdir /var/log/chrony
maxupdateskew 100.0
rtcsync
makestep 1 3
leapseclist /usr/share/zoneinfo/leap-seconds.list
confdir /etc/chrony/conf.d
/etc/chrony/conf.d:
README
/etc/chrony/sources.d:
README
=== start with no time ===
ls: cannot access '/var/lib/systemd/timesync/clock': No such file or directory
/var/lib/chrony/:
total 4
-rw-r--r-- 1 _chrony _chrony 42 2026-10-08 04:39:08.514818246 +0200 chrony.drift
2026-11-11 11:03:21.936805+01:00
1794391402
1794391404
=== boots ===
2026-11-08 05:35:18
-5 46081f567f384ab88591d4e604555a48 Sun 2026-10-04 21:43:33 CEST Sun 2026-10-04 21:43:46 CEST
-4 e82176ae7cf644518507468fc609eaa9 Wed 2026-10-14 15:43:07 CEST Fri 2026-10-16 05:09:06 CEST
-3 ddd3d7305e244b2a9eafb5cf229476f1 Fri 2026-10-16 05:09:23 CEST Wed 2026-10-21 05:23:34 CEST
-2 56db71938b304a57bdedf30f89860e1f Wed 2026-10-21 05:23:48 CEST Thu 2026-10-29 04:47:58 CET
-1 4f6d8b388e224b8ca8ed45e06a2f96b9 Thu 2026-10-29 04:48:11 CET Sun 2026-11-08 05:35:13 CET
0 72899d2625244e069e8e8099775d1e51 Sun 2026-11-08 05:35:21 CET Wed 2026-11-11 11:03:24 CET
=== timers ===
NEXT LEFT LAST PASSED UNIT ACTIVATES
Wed 2026-11-11 11:03:50 CET 25s Wed 2026-11-11 11:02:50 CET 34s ago felhom-mgmt-watchdog.timer felhom-mgmt-watchdog.service
Wed 2026-11-11 11:56:50 CET 53min Wed 2026-11-11 10:56:50 CET 6min ago felhom-crash-guard-check.timer felhom-crash-guard-check.service
Thu 2026-11-12 00:00:00 CET 12h Wed 2026-11-11 00:00:30 CET 11h ago dpkg-db-backup.timer dpkg-db-backup.service
Thu 2026-11-12 00:36:19 CET 13h Wed 2026-11-11 00:40:45 CET 10h ago logrotate.timer logrotate.service
Thu 2026-11-12 00:37:37 CET 13h Wed 2026-11-11 10:46:55 CET 16min ago apt-daily.timer apt-daily.service
Thu 2026-11-12 05:41:31 CET 18h Wed 2026-11-11 01:11:21 CET 9h ago pve-daily-update.timer pve-daily-update.service
Thu 2026-11-12 05:50:49 CET 18h Wed 2026-11-11 05:50:49 CET 5h 12min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Thu 2026-11-12 06:54:07 CET 19h Wed 2026-11-11 06:40:15 CET 4h 23min ago apt-daily-upgrade.timer apt-daily-upgrade.service
Thu 2026-11-12 07:50:39 CET 20h Wed 2026-11-11 10:09:10 CET 54min ago man-db.timer man-db.service
Sun 2026-11-15 03:10:21 CET 3 days Sun 2026-11-08 03:10:44 CET - xfs_scrub_all.timer xfs_scrub_all.service
Sun 2026-11-15 03:10:52 CET 3 days Sun 2026-11-08 03:10:44 CET - e2scrub_all.timer e2scrub_all.service
Mon 2026-11-16 01:20:47 CET 4 days Mon 2026-11-09 01:32:30 CET 2 days ago fstrim.timer fstrim.service
12 timers listed.
=== timer units (all) ===
--- apt-daily-upgrade.timer [enabled/active]
OnCalendar=*-*-* 6:00
RandomizedDelaySec=60m
Persistent=true
Unit=apt-daily-upgrade.service NextElapseUSecRealtime=Thu 2026-11-12 06:54:07 CET LastTriggerUSec=Wed 2026-11-11 06:40:15 CET
--- apt-daily.timer [enabled/active]
OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h
Persistent=true
Unit=apt-daily.service NextElapseUSecRealtime=Thu 2026-11-12 00:37:37 CET LastTriggerUSec=Wed 2026-11-11 10:46:55 CET
--- apt-listchanges.timer [disabled/inactive]
OnCalendar=hourly
Unit=apt-listchanges.service NextElapseUSecRealtime= LastTriggerUSec=
--- chrony-dnssrv@.timer [disabled/]
OnActiveSec=0
OnUnitInactiveSec=1h
--- dpkg-db-backup.timer [enabled/active]
OnCalendar=daily
Persistent=true
Unit=dpkg-db-backup.service NextElapseUSecRealtime=Thu 2026-11-12 00:00:00 CET LastTriggerUSec=Wed 2026-11-11 00:00:30 CET
--- e2scrub_all.timer [enabled/active]
OnCalendar=Sun *-*-* 03:10:00
RandomizedDelaySec=60
Persistent=true
Unit=e2scrub_all.service NextElapseUSecRealtime=Sun 2026-11-15 03:10:52 CET LastTriggerUSec=Sun 2026-11-08 03:10:44 CET
--- felhom-crash-guard-check.timer [enabled/active]
OnBootSec=15min
OnUnitActiveSec=1h
Unit=felhom-crash-guard-check.service NextElapseUSecRealtime= LastTriggerUSec=Wed 2026-11-11 10:56:50 CET
--- felhom-mgmt-watchdog.timer [enabled/active]
OnBootSec=30s
OnUnitActiveSec=60s
AccuracySec=5s
Persistent=true
Unit=felhom-mgmt-watchdog.service
Unit=felhom-mgmt-watchdog.service NextElapseUSecRealtime= LastTriggerUSec=Wed 2026-11-11 11:02:50 CET
--- fstrim.timer [enabled/active]
OnCalendar=weekly
AccuracySec=1h
Persistent=true
RandomizedDelaySec=100min
Unit=fstrim.service NextElapseUSecRealtime=Mon 2026-11-16 01:20:47 CET LastTriggerUSec=Mon 2026-11-09 01:32:30 CET
--- logrotate.timer [enabled/active]
OnCalendar=daily
RandomizedDelaySec=1h
Persistent=true
Unit=logrotate.service NextElapseUSecRealtime=Thu 2026-11-12 00:36:19 CET LastTriggerUSec=Wed 2026-11-11 00:40:45 CET
--- man-db.timer [enabled/active]
OnCalendar=daily
RandomizedDelaySec=12h
Persistent=true
Unit=man-db.service NextElapseUSecRealtime=Thu 2026-11-12 07:50:39 CET LastTriggerUSec=Wed 2026-11-11 10:09:10 CET
--- proxmox-firewall.timer [disabled/inactive]
OnBootSec=1s
OnUnitInactiveSec=5s
Unit=proxmox-firewall.service
Unit=proxmox-firewall.service NextElapseUSecRealtime= LastTriggerUSec=
--- pve-daily-update.timer [enabled/active]
OnCalendar=*-*-* 1:00
RandomizedDelaySec=5h
Persistent=true
Unit=pve-daily-update.service NextElapseUSecRealtime=Thu 2026-11-12 05:41:31 CET LastTriggerUSec=Wed 2026-11-11 01:11:21 CET
--- systemd-tmpfiles-clean.timer [static/active]
OnBootSec=15min
OnUnitActiveSec=1d
Unit=systemd-tmpfiles-clean.service NextElapseUSecRealtime= LastTriggerUSec=Wed 2026-11-11 05:50:49 CET
--- xfs_scrub_all.timer [enabled/active]
OnCalendar=Sun *-*-* 03:10:00
RandomizedDelaySec=60
Persistent=true
Unit=xfs_scrub_all.service NextElapseUSecRealtime=Sun 2026-11-15 03:10:21 CET LastTriggerUSec=Sun 2026-11-08 03:10:44 CET
--- zfs-scrub-monthly@.timer [disabled/]
OnCalendar=monthly
Persistent=true
RandomizedDelaySec=1h
Unit=zfs-scrub@%i.service
--- zfs-scrub-weekly@.timer [disabled/]
OnCalendar=weekly
Persistent=true
RandomizedDelaySec=1h
Unit=zfs-scrub@%i.service
--- zfs-trim-monthly@.timer [disabled/]
OnCalendar=monthly
Persistent=true
RandomizedDelaySec=1h
Unit=zfs-trim@%i.service
--- zfs-trim-weekly@.timer [disabled/]
OnCalendar=weekly
Persistent=true
RandomizedDelaySec=1h
Unit=zfs-trim@%i.service
=== service limits of felhom + backup timers ===
--- apt-daily-upgrade.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 06:40:15 CET
ExecMainExitTimestamp=Wed 2026-11-11 06:40:15 CET
ExecMainStatus=0
ExecStart={ path=/usr/lib/apt/apt.systemd.daily ; argv[]=/usr/lib/apt/apt.systemd.daily install ; ignore_errors=no ; start_time=[Wed 2026-11-11 06:40:15 CET] ; stop_time=[Wed 2026-11-11 06:40:15 CET] ; pid=809079 ; code=exited ; status=0 }
--- apt-daily.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 10:46:55 CET
ExecMainExitTimestamp=Wed 2026-11-11 10:46:55 CET
ExecMainStatus=0
ExecStart={ path=/usr/lib/apt/apt.systemd.daily ; argv[]=/usr/lib/apt/apt.systemd.daily update ; ignore_errors=no ; start_time=[Wed 2026-11-11 10:46:55 CET] ; stop_time=[Wed 2026-11-11 10:46:55 CET] ; pid=1088520 ; code=exited ; status=0 }
--- apt-listchanges.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=
ExecMainExitTimestamp=
ExecMainStatus=0
ExecStart={ path=python3 ; argv[]=python3 -m apt_listchanges.populate_database --profile apt ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
--- dpkg-db-backup.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 00:00:30 CET
ExecMainExitTimestamp=Wed 2026-11-11 00:00:30 CET
ExecMainStatus=0
ExecStart={ path=/usr/libexec/dpkg/dpkg-db-backup ; argv[]=/usr/libexec/dpkg/dpkg-db-backup ; ignore_errors=no ; start_time=[Wed 2026-11-11 00:00:30 CET] ; stop_time=[Wed 2026-11-11 00:00:30 CET] ; pid=345759 ; code=exited ; status=0 }
--- e2scrub_all.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=
ExecMainExitTimestamp=
ExecMainStatus=0
ExecStart={ path=/sbin/e2scrub_all ; argv[]=/sbin/e2scrub_all ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
--- felhom-crash-guard-check.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 10:56:50 CET
ExecMainExitTimestamp=Wed 2026-11-11 10:56:50 CET
ExecMainStatus=0
ExecStart={ path=/usr/local/sbin/felhom-crash-guard ; argv[]=/usr/local/sbin/felhom-crash-guard check ; ignore_errors=no ; start_time=[Wed 2026-11-11 10:56:50 CET] ; stop_time=[Wed 2026-11-11 10:56:50 CET] ; pid=1099750 ; code=exited ; status=0 }
--- felhom-mgmt-watchdog.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 11:02:50 CET
ExecMainExitTimestamp=Wed 2026-11-11 11:02:50 CET
ExecMainStatus=0
ExecStart={ path=/usr/local/sbin/felhom-mgmt-watchdog ; argv[]=/usr/local/sbin/felhom-mgmt-watchdog ; ignore_errors=no ; start_time=[Wed 2026-11-11 11:02:50 CET] ; stop_time=[Wed 2026-11-11 11:02:50 CET] ; pid=1106395 ; code=exited ; status=0 }
--- fstrim.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Mon 2026-11-09 01:32:30 CET
ExecMainExitTimestamp=Mon 2026-11-09 01:32:46 CET
ExecMainStatus=0
ExecStart={ path=/sbin/fstrim ; argv[]=/sbin/fstrim --listed-in /etc/fstab:/proc/self/mountinfo --verbose --quiet-unsupported ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
--- logrotate.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 00:40:45 CET
ExecMainExitTimestamp=Wed 2026-11-11 00:40:47 CET
ExecMainStatus=0
ExecStart={ path=/usr/sbin/logrotate ; argv[]=/usr/sbin/logrotate /etc/logrotate.conf ; ignore_errors=no ; start_time=[Wed 2026-11-11 00:40:45 CET] ; stop_time=[Wed 2026-11-11 00:40:47 CET] ; pid=391150 ; code=exited ; status=0 }
--- man-db.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 10:09:10 CET
ExecMainExitTimestamp=Wed 2026-11-11 10:09:10 CET
ExecMainStatus=0
ExecStart={ path=/usr/bin/install ; argv[]=/usr/bin/install -d -o man -g man -m 0755 /var/cache/man ; ignore_errors=no ; start_time=[Wed 2026-11-11 10:09:10 CET] ; stop_time=[Wed 2026-11-11 10:09:10 CET] ; pid=1044483 ; code=exited ; status=0 }
ExecStart={ path=/usr/bin/mandb ; argv[]=/usr/bin/mandb --quiet ; ignore_errors=no ; start_time=[Wed 2026-11-11 10:09:10 CET] ; stop_time=[Wed 2026-11-11 10:09:10 CET] ; pid=1044490 ; code=exited ; status=0 }
--- pve-daily-update.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 01:11:21 CET
ExecMainExitTimestamp=Wed 2026-11-11 01:11:25 CET
ExecMainStatus=0
ExecStart={ path=/usr/bin/pveupdate ; argv[]=/usr/bin/pveupdate ; ignore_errors=no ; start_time=[Wed 2026-11-11 01:11:21 CET] ; stop_time=[Wed 2026-11-11 01:11:25 CET] ; pid=425598 ; code=exited ; status=0 }
=== felhom units that are not timers ===
felhom-agent-rollback.service loaded inactive dead
felhom-agent.service loaded active running
felhom-crash-guard-check.service loaded inactive dead
felhom-crash-guard.service loaded active exited
felhom-mgmt-watchdog.service loaded inactive dead
felhom-oob-nft.service loaded active exited
felhom-shared-parent.service loaded active exited
felhom-sshd.service loaded active running
felhom-crash-guard-check.timer loaded active waiting
felhom-mgmt-watchdog.timer loaded active waiting
--- felhom-agent-rollback.service
Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=infinity NRestarts=0 ActiveEnterTimestamp=
--- felhom-agent.service
Restart=on-failure RestartUSec=5s TimeoutStartUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:50:48 CET
--- felhom-bootstrap.service
Restart=on-failure RestartUSec=30s TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=infinity NRestarts=0 ActiveEnterTimestamp=
--- felhom-crash-guard-check.service
Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=
--- felhom-crash-guard.service
Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Sun 2026-11-08 05:35:23 CET
--- felhom-mgmt-watchdog.service
Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=
--- felhom-oob-nft.service
Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Sun 2026-11-08 05:35:23 CET
--- felhom-shared-parent.service
Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Sun 2026-11-08 05:35:23 CET
--- felhom-sshd.service
Restart=on-failure RestartUSec=100ms TimeoutStartUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Sun 2026-11-08 05:35:24 CET
=== cron ===
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; }
47 6 * * 7 root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.weekly; }
52 6 1 * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.monthly; }
--- /etc/cron.d/e2scrub_all
30 3 * * 0 root test -e /run/systemd/system || SERVICE_MODE=1 /usr/libexec/e2fsprogs/e2scrub_all_cron
10 3 * * * root test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r
--- /etc/cron.d/vzdump
PATH="/usr/sbin:/usr/bin:/sbin:/bin"
--- /etc/cron.d/zfsutils-linux
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
24 0 1-7 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/trim ]; then /usr/lib/zfs-linux/trim; fi
24 0 8-14 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs-linux/scrub; fi
--- cron.hourly:
--- cron.daily: apt-compat dpkg logrotate man-db
--- cron.weekly: man-db
--- cron.monthly:
--- user crontabs:
=== at ===
no at
=== pve jobs ===
--- vzdump.cron
PATH="/usr/sbin:/usr/bin:/sbin:/bin"
--- vzdump.conf
=== pve tasks (start end type id status), last 14 days ===
awk: line 2: function strtonum never defined
awk: line 2: function strtonum never defined
=== runs of timer services, last 10 days (start/finish lines) ===
--- apt-daily-upgrade.service
2026-11-09T06:30:32+01:00 demo-felhom systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities...
2026-11-09T06:30:32+01:00 demo-felhom systemd[1]: apt-daily-upgrade.service: Deactivated successfully.
2026-11-09T06:30:32+01:00 demo-felhom systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities.
2026-11-10T06:03:00+01:00 demo-felhom systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities...
2026-11-10T06:03:00+01:00 demo-felhom systemd[1]: apt-daily-upgrade.service: Deactivated successfully.
2026-11-10T06:03:00+01:00 demo-felhom systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities.
2026-11-11T06:40:15+01:00 demo-felhom systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities...
2026-11-11T06:40:15+01:00 demo-felhom systemd[1]: apt-daily-upgrade.service: Deactivated successfully.
2026-11-11T06:40:15+01:00 demo-felhom systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities.
--- apt-daily.service
2026-11-10T13:12:32+01:00 demo-felhom systemd[1]: Starting apt-daily.service - Daily apt download activities...
2026-11-10T13:12:32+01:00 demo-felhom systemd[1]: apt-daily.service: Deactivated successfully.
2026-11-10T13:12:32+01:00 demo-felhom systemd[1]: Finished apt-daily.service - Daily apt download activities.
2026-11-11T01:34:49+01:00 demo-felhom systemd[1]: Starting apt-daily.service - Daily apt download activities...
2026-11-11T01:34:50+01:00 demo-felhom systemd[1]: apt-daily.service: Deactivated successfully.
2026-11-11T01:34:50+01:00 demo-felhom systemd[1]: Finished apt-daily.service - Daily apt download activities.
2026-11-11T10:46:55+01:00 demo-felhom systemd[1]: Starting apt-daily.service - Daily apt download activities...
2026-11-11T10:46:55+01:00 demo-felhom systemd[1]: apt-daily.service: Deactivated successfully.
2026-11-11T10:46:55+01:00 demo-felhom systemd[1]: Finished apt-daily.service - Daily apt download activities.
--- felhom-crash-guard-check.service
2026-11-11T08:56:50+01:00 demo-felhom systemd[1]: Starting felhom-crash-guard-check.service - Felhom crash guard re-arm check...
2026-11-11T08:56:50+01:00 demo-felhom systemd[1]: felhom-crash-guard-check.service: Deactivated successfully.
2026-11-11T08:56:50+01:00 demo-felhom systemd[1]: Finished felhom-crash-guard-check.service - Felhom crash guard re-arm check.
2026-11-11T09:56:50+01:00 demo-felhom systemd[1]: Starting felhom-crash-guard-check.service - Felhom crash guard re-arm check...
2026-11-11T09:56:50+01:00 demo-felhom systemd[1]: felhom-crash-guard-check.service: Deactivated successfully.
2026-11-11T09:56:50+01:00 demo-felhom systemd[1]: Finished felhom-crash-guard-check.service - Felhom crash guard re-arm check.
2026-11-11T10:56:50+01:00 demo-felhom systemd[1]: Starting felhom-crash-guard-check.service - Felhom crash guard re-arm check...
2026-11-11T10:56:50+01:00 demo-felhom systemd[1]: felhom-crash-guard-check.service: Deactivated successfully.
2026-11-11T10:56:50+01:00 demo-felhom systemd[1]: Finished felhom-crash-guard-check.service - Felhom crash guard re-arm check.
--- felhom-mgmt-watchdog.service
2026-11-11T11:00:49+01:00 demo-felhom systemd[1]: Starting felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)...
2026-11-11T11:00:49+01:00 demo-felhom systemd[1]: felhom-mgmt-watchdog.service: Deactivated successfully.
2026-11-11T11:00:49+01:00 demo-felhom systemd[1]: Finished felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent).
2026-11-11T11:01:50+01:00 demo-felhom systemd[1]: Starting felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)...
2026-11-11T11:01:50+01:00 demo-felhom systemd[1]: felhom-mgmt-watchdog.service: Deactivated successfully.
2026-11-11T11:01:50+01:00 demo-felhom systemd[1]: Finished felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent).
2026-11-11T11:02:50+01:00 demo-felhom systemd[1]: Starting felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)...
2026-11-11T11:02:50+01:00 demo-felhom systemd[1]: felhom-mgmt-watchdog.service: Deactivated successfully.
2026-11-11T11:02:50+01:00 demo-felhom systemd[1]: Finished felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent).
--- pve-daily-update.service
2026-11-09T02:07:09+01:00 demo-felhom systemd[1]: pve-daily-update.service: Consumed 3.774s CPU time, 234.8M memory peak.
2026-11-10T01:23:25+01:00 demo-felhom systemd[1]: Starting pve-daily-update.service - Daily PVE download activities...
2026-11-10T01:23:29+01:00 demo-felhom systemd[1]: pve-daily-update.service: Deactivated successfully.
2026-11-10T01:23:29+01:00 demo-felhom systemd[1]: Finished pve-daily-update.service - Daily PVE download activities.
2026-11-10T01:23:29+01:00 demo-felhom systemd[1]: pve-daily-update.service: Consumed 3.725s CPU time, 234.2M memory peak.
2026-11-11T01:11:21+01:00 demo-felhom systemd[1]: Starting pve-daily-update.service - Daily PVE download activities...
2026-11-11T01:11:25+01:00 demo-felhom systemd[1]: pve-daily-update.service: Deactivated successfully.
2026-11-11T01:11:25+01:00 demo-felhom systemd[1]: Finished pve-daily-update.service - Daily PVE download activities.
2026-11-11T01:11:25+01:00 demo-felhom systemd[1]: pve-daily-update.service: Consumed 3.757s CPU time, 234.8M memory peak.
=== agent log: schedule words, last 3 days (tail 260) ===
2026-11-10T00:20:42+01:00 demo-felhom sudo[2915765]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T00:35:42+01:00 demo-felhom sudo[2932690]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T00:50:42+01:00 demo-felhom sudo[2949587]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T01:05:42+01:00 demo-felhom sudo[2966528]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T01:20:42+01:00 demo-felhom sudo[2983412]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T01:35:42+01:00 demo-felhom sudo[3000347]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T01:50:42+01:00 demo-felhom sudo[3017217]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T02:05:42+01:00 demo-felhom sudo[3034102]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T02:20:42+01:00 demo-felhom sudo[3051036]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T02:35:42+01:00 demo-felhom sudo[3070754]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T02:50:42+01:00 demo-felhom sudo[3087822]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T03:05:42+01:00 demo-felhom sudo[3104852]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T03:20:42+01:00 demo-felhom sudo[3121847]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T03:35:42+01:00 demo-felhom sudo[3139471]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T03:50:42+01:00 demo-felhom sudo[3156417]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T04:05:42+01:00 demo-felhom sudo[3173412]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T04:20:42+01:00 demo-felhom sudo[3193437]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T04:35:42+01:00 demo-felhom sudo[3210510]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T04:50:42+01:00 demo-felhom sudo[3227558]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T05:05:42+01:00 demo-felhom sudo[3244549]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T05:20:42+01:00 demo-felhom sudo[3261530]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T05:34:26+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:34:26.031+01:00 level=INFO msg="backup: space preflight passed" vmid=9201 target=felhom-backup last_archive_bytes=1528966739 need_bytes=2984950247 avail_bytes=924950589440
2026-11-10T05:34:27+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:34:27.065+01:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=felhom-backup job=backup-9201-1794285265667942744
2026-11-10T05:35:12+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:35:12.208+01:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-backup archive=felhom-backup:backup/vzdump-lxc-9201-2026_11_10-05_34_26.tar.zst size_bytes=1299653068 uncove
2026-11-10T05:35:12+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:35:12.208+01:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=felhom-backup job=backup-9201-1794285265667942744 archive=felhom-backup:backup/vzdump-lxc-9201-2026_11
2026-11-10T05:35:33+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:35:33.864+01:00 level=INFO msg="janitor: stale-lock sweep deferred — a heavy operation is in flight" busy=backup:felhom-backup
2026-11-10T05:35:40+01:00 demo-felhom sudo[3280846]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T05:35:42+01:00 demo-felhom sudo[3281005]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T05:35:52+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:35:52.839+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-11-10T05:36:42+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:36:42.210+01:00 level=INFO msg="osupdate: START" run=20261110T043642Z layer=guest vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261110T043642Z
2026-11-10T05:36:56+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:36:56.012+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261110T043642Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
2026-11-10T05:36:56+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:36:56.012+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-10T05:36:56+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:36:56.012+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
2026-11-10T05:36:56+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:36:56.012+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
2026-11-10T05:36:56+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:36:56.013+01:00 level=INFO msg="osupdate: DONE" run=20261110T043642Z layer=guest vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered
2026-11-10T05:36:56+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:36:56.023+01:00 level=INFO msg="osupdate: START" run=20261110T043642Z layer=host vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261110T043642Z
2026-11-10T05:37:06+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:06.868+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261110T043642Z layer=host lane=fast mode=apply select=pending-fast packages=0"
2026-11-10T05:37:06+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:06.868+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-10T05:37:06+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:06.868+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
2026-11-10T05:37:06+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:06.868+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
2026-11-10T05:37:07+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:07.688+01:00 level=INFO msg="osupdate: DONE" run=20261110T043642Z layer=host vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=14 not_covered
2026-11-10T05:37:09+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:09.362+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE already on"
2026-11-10T05:37:09+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:09.362+01:00 level=INFO msg="osupdate: live-restore" vmid=9201 result="{\"result\": \"already on\"}"
2026-11-10T05:37:09+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:09.362+01:00 level=INFO msg="osupdate: START" run=20261110T043642Z layer=docker vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261110T043642Z
2026-11-10T05:37:31+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:31.993+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261110T043642Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authori
2026-11-10T05:37:31+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:31.993+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-10T05:37:31+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:31.993+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
2026-11-10T05:37:31+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:31.993+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
2026-11-10T05:37:31+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:31.993+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: OOM-CHECK result=pass oom_killed=True oom_event=True exit=137 image=gitea.dooplex.hu/admin/felhom-controller:0.351.0
2026-11-10T05:37:31+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:31.993+01:00 level=INFO msg="osupdate: DONE" run=20261110T043642Z layer=docker vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covere
2026-11-10T05:37:32+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:32.001+01:00 level=INFO msg="osupdate: pve step holds the /etc/pve write gate — the agent's own writes wait until it ends" run=20261110T043642Z layer=pve vmid=9201 trigger=night
2026-11-10T05:37:32+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:32.001+01:00 level=INFO msg="osupdate: START" run=20261110T043642Z layer=pve vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261110T043642Z
2026-11-10T05:37:43+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:43.960+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261110T043642Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=rin
2026-11-10T05:37:43+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:43.960+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-10T05:37:43+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:43.960+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
2026-11-10T05:37:43+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:43.960+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
2026-11-10T05:37:44+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:44.758+01:00 level=INFO msg="osupdate: DONE" run=20261110T043642Z layer=pve vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=1 not_covered=0
2026-11-10T05:37:44+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:44.779+01:00 level=INFO msg="osupdate: pve step released the /etc/pve write gate" run=20261110T043642Z layer=pve vmid=9201 trigger=night
2026-11-10T05:37:44+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:44.779+01:00 level=INFO msg="osupdate: kernel step skipped — the hub names no kernel step for this box" run=20261110T043642Z layer=kernel vmid=9201 trigger=night ring=0
2026-11-10T05:39:27+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:39:27.093+01:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=felhom-pbs job=backup-9201-felhom-pbs-1794285565692471312
2026-11-10T05:40:12+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:40:12.531+01:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-11-10T04:39:26Z size_bytes=3502516951 uncovered_volumes=2
2026-11-10T05:40:12+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:40:12.531+01:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=felhom-pbs job=backup-9201-felhom-pbs-1794285565692471312 archive=felhom-pbs:backup/ct/9201/2026-11-10
2026-11-10T05:50:42+01:00 demo-felhom sudo[3302512]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T06:05:42+01:00 demo-felhom sudo[3319580]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T06:06:34+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T06:06:34.950+01:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=felhom-backup archive=felhom-backup:backup/vzdump-lxc-9201-
2026-11-10T06:06:35+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T06:06:35.818+01:00 level=INFO msg="restore-test: space preflight passed" storage=local-lvm required_bytes=10566459392 avail_bytes=363190239731
2026-11-10T06:06:35+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T06:06:35.841+01:00 level=INFO msg="restore-test: full-fidelity restore params derived from the archive config" scratch=990000 params=4
2026-11-10T06:07:34+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T06:07:34.438+01:00 level=INFO msg="restore-test: scratch guest torn down" vmid=990000
2026-11-10T06:07:34+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T06:07:34.439+01:00 level=INFO msg="backup: scheduled restore-test passed" archive=felhom-backup:backup/vzdump-lxc-9201-2026_11_09-05_31_03.tar.zst duration_s=59.485928852
2026-11-10T06:20:42+01:00 demo-felhom sudo[3337978]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T06:35:42+01:00 demo-felhom sudo[3354915]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T06:50:42+01:00 demo-felhom sudo[3371871]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T07:05:42+01:00 demo-felhom sudo[3388825]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T07:20:42+01:00 demo-felhom sudo[3405797]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T07:35:42+01:00 demo-felhom sudo[3422762]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T07:50:42+01:00 demo-felhom sudo[3439766]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T08:05:42+01:00 demo-felhom sudo[3456667]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T08:20:42+01:00 demo-felhom sudo[3473575]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T08:35:42+01:00 demo-felhom sudo[3491405]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T08:50:42+01:00 demo-felhom sudo[3509858]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T09:05:42+01:00 demo-felhom sudo[3526966]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T09:20:42+01:00 demo-felhom sudo[3543898]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T09:35:42+01:00 demo-felhom sudo[3560731]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T09:50:42+01:00 demo-felhom sudo[3577606]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T10:05:42+01:00 demo-felhom sudo[3594532]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T10:20:42+01:00 demo-felhom sudo[3611505]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T10:35:42+01:00 demo-felhom sudo[3628405]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T10:50:42+01:00 demo-felhom sudo[3645823]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T11:05:42+01:00 demo-felhom sudo[3662674]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T11:20:42+01:00 demo-felhom sudo[3679581]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T11:35:40+01:00 demo-felhom sudo[3696377]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T11:35:42+01:00 demo-felhom sudo[3696468]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T11:35:54+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T11:35:54.826+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=3
2026-11-10T11:50:42+01:00 demo-felhom sudo[3713367]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T12:05:42+01:00 demo-felhom sudo[3730217]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T12:06:35+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T12:06:35.294+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-03T04:35:12Z) is already proven; felhom-ba
2026-11-10T12:20:42+01:00 demo-felhom sudo[3747183]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T12:35:42+01:00 demo-felhom sudo[3764104]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T12:50:42+01:00 demo-felhom sudo[3781025]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T13:05:42+01:00 demo-felhom sudo[3797895]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T13:20:42+01:00 demo-felhom sudo[3814879]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T13:35:42+01:00 demo-felhom sudo[3831759]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T13:50:42+01:00 demo-felhom sudo[3848706]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T14:05:42+01:00 demo-felhom sudo[3865605]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T14:20:42+01:00 demo-felhom sudo[3882602]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T14:35:42+01:00 demo-felhom sudo[3899445]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T14:50:42+01:00 demo-felhom sudo[3916354]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:05:42+01:00 demo-felhom sudo[3933239]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:20:42+01:00 demo-felhom sudo[3950147]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:35:42+01:00 demo-felhom sudo[3967102]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:50:42+01:00 demo-felhom sudo[3984749]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:50:46+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T15:50:46.327+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.173.0 report="{\"agent_update\": {\"sha256\": \"752b2727fb604
2026-11-10T15:50:48+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T15:50:48.430+01:00 level=INFO msg="pbs: verify loop shutting down" reason="context canceled"
2026-11-10T15:50:48+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T15:50:48.430+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled"
2026-11-10T15:50:48+01:00 demo-felhom systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)...
2026-11-10T15:50:48+01:00 demo-felhom systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T15:50:48+01:00 demo-felhom systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T15:50:48+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:48.463+01:00 level=INFO msg="backup tier armed" target=felhom-backup cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true
2026-11-10T15:50:48+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:48.463+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
2026-11-10T15:50:48+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:48.464+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)"
2026-11-10T15:50:48+01:00 demo-felhom sudo[3985075]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T145048Z-kernel-kernel-boot.json
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.046+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.046+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:50:49+01:00 demo-felhom sudo[3985235]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup
2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; nev
2026-11-10T15:50:50+01:00 demo-felhom sudo[3985377]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:51:01+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:01.241+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=3
2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.032+01:00 level=INFO msg="osupdate: running the bundle policy pass after the self-update commit (R-1285)" agent_version=0.173.0
2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.121+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE POLICY DONE install_mode=appliance removed=0"
2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.122+01:00 level=INFO msg="osupdate: bundle policy pass done" bundle_policy="{\"install_mode\": \"appliance\", \"removed\": []}"
2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.128+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.173.0 removed=48
2026-11-10T16:05:54+01:00 demo-felhom sudo[4002880]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T16:20:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T16:20:49.047+01:00 level=INFO msg="backup: restore-test first evaluation after start (R-874)" after=30m0s
2026-11-10T16:20:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T16:20:49.736+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-03T04:35:12Z) is already proven; felhom
2026-11-10T16:20:54+01:00 demo-felhom sudo[4020715]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T16:28:44+01:00 demo-felhom sudo[4029648]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T16:35:54+01:00 demo-felhom sudo[4038486]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T16:50:54+01:00 demo-felhom sudo[4055454]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T17:05:54+01:00 demo-felhom sudo[4072670]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T17:20:54+01:00 demo-felhom sudo[4089639]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T17:35:54+01:00 demo-felhom sudo[4106493]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T17:50:54+01:00 demo-felhom sudo[4123369]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T18:05:54+01:00 demo-felhom sudo[4140267]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T18:20:54+01:00 demo-felhom sudo[4157202]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T18:35:54+01:00 demo-felhom sudo[4174095]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T18:50:54+01:00 demo-felhom sudo[4190953]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T19:05:54+01:00 demo-felhom sudo[14085]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T19:20:54+01:00 demo-felhom sudo[30996]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T19:35:54+01:00 demo-felhom sudo[47836]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T19:50:54+01:00 demo-felhom sudo[64771]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T20:05:54+01:00 demo-felhom sudo[81620]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T20:20:54+01:00 demo-felhom sudo[98549]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T20:35:54+01:00 demo-felhom sudo[115453]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T20:50:54+01:00 demo-felhom sudo[132275]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T21:05:54+01:00 demo-felhom sudo[149156]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T21:20:54+01:00 demo-felhom sudo[166106]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T21:35:54+01:00 demo-felhom sudo[183009]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T21:50:54+01:00 demo-felhom sudo[199912]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T21:51:01+01:00 demo-felhom sudo[200244]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T21:51:15+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T21:51:15.415+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=3
2026-11-10T22:05:54+01:00 demo-felhom sudo[216811]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T22:20:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T22:20:50.434+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-03T04:35:12Z) is already proven; felhom
2026-11-10T22:20:54+01:00 demo-felhom sudo[233785]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T22:35:54+01:00 demo-felhom sudo[250624]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T22:50:54+01:00 demo-felhom sudo[267511]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T23:05:54+01:00 demo-felhom sudo[284411]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T23:20:54+01:00 demo-felhom sudo[301342]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T23:35:54+01:00 demo-felhom sudo[318204]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T23:50:54+01:00 demo-felhom sudo[335082]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T00:05:54+01:00 demo-felhom sudo[351965]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T00:20:54+01:00 demo-felhom sudo[368862]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T00:35:54+01:00 demo-felhom sudo[385722]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T00:50:54+01:00 demo-felhom sudo[402645]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T01:05:54+01:00 demo-felhom sudo[419499]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T01:20:54+01:00 demo-felhom sudo[436416]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T01:35:54+01:00 demo-felhom sudo[453352]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T01:50:54+01:00 demo-felhom sudo[470292]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T02:05:54+01:00 demo-felhom sudo[487150]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T02:20:54+01:00 demo-felhom sudo[504080]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T02:35:54+01:00 demo-felhom sudo[524004]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T02:50:54+01:00 demo-felhom sudo[541023]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T03:05:54+01:00 demo-felhom sudo[558023]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T03:20:54+01:00 demo-felhom sudo[575005]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T03:35:54+01:00 demo-felhom sudo[592593]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T03:50:54+01:00 demo-felhom sudo[609616]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T03:51:01+01:00 demo-felhom sudo[610005]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T03:51:15+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T03:51:15.426+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=3
2026-11-11T04:05:54+01:00 demo-felhom sudo[626547]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T04:20:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T04:20:50.416+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-03T04:35:12Z) is already proven; felhom
2026-11-11T04:20:54+01:00 demo-felhom sudo[646613]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T04:35:54+01:00 demo-felhom sudo[663687]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T04:50:54+01:00 demo-felhom sudo[680674]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T05:05:54+01:00 demo-felhom sudo[697658]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T05:20:54+01:00 demo-felhom sudo[714656]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T05:35:54+01:00 demo-felhom sudo[731604]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T05:36:51+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:36:51.383+01:00 level=INFO msg="backup: space preflight passed" vmid=9201 target=felhom-backup last_archive_bytes=1299653068 need_bytes=2698308159 avail_bytes=925178314752
2026-11-11T05:36:52+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:36:52.425+01:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=felhom-backup job=backup-9201-1794371810991152516
2026-11-11T05:37:37+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:37:37.562+01:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-backup archive=felhom-backup:backup/vzdump-lxc-9201-2026_11_11-05_36_51.tar.zst size_bytes=1300036887 unc
2026-11-11T05:37:37+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:37:37.562+01:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=felhom-backup job=backup-9201-1794371810991152516 archive=felhom-backup:backup/vzdump-lxc-9201-2026
2026-11-11T05:39:07+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:07.567+01:00 level=INFO msg="osupdate: START" run=20261111T043907Z layer=guest vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261111T043907Z
2026-11-11T05:39:21+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:21.549+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261111T043907Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
2026-11-11T05:39:21+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:21.549+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-11T05:39:21+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:21.549+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
2026-11-11T05:39:21+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:21.549+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
2026-11-11T05:39:21+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:21.549+01:00 level=INFO msg="osupdate: DONE" run=20261111T043907Z layer=guest vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_cove
2026-11-11T05:39:21+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:21.617+01:00 level=INFO msg="osupdate: START" run=20261111T043907Z layer=host vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261111T043907Z
2026-11-11T05:39:32+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:32.620+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261111T043907Z layer=host lane=fast mode=apply select=pending-fast packages=0"
2026-11-11T05:39:32+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:32.620+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-11T05:39:32+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:32.620+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
2026-11-11T05:39:32+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:32.620+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
2026-11-11T05:39:33+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:33.444+01:00 level=INFO msg="osupdate: DONE" run=20261111T043907Z layer=host vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=15 not_cove
2026-11-11T05:39:35+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:35.096+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE already on"
2026-11-11T05:39:35+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:35.096+01:00 level=INFO msg="osupdate: live-restore" vmid=9201 result="{\"result\": \"already on\"}"
2026-11-11T05:39:35+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:35.096+01:00 level=INFO msg="osupdate: START" run=20261111T043907Z layer=docker vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261111T043907Z
2026-11-11T05:39:57+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:57.910+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261111T043907Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 auth
2026-11-11T05:39:57+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:57.910+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-11T05:39:57+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:57.910+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
2026-11-11T05:39:57+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:57.910+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
2026-11-11T05:39:57+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:57.910+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: OOM-CHECK result=fail oom_killed=False oom_event=False exit=137 image=gitea.dooplex.hu/admin/felhom-controller:0.
2026-11-11T05:39:57+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:57.910+01:00 level=INFO msg="osupdate: DONE" run=20261111T043907Z layer=docker vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_cov
2026-11-11T05:39:57+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:57.917+01:00 level=INFO msg="osupdate: pve step holds the /etc/pve write gate — the agent's own writes wait until it ends" run=20261111T043907Z layer=pve vmid=9201 trigger=nig
2026-11-11T05:39:57+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:57.917+01:00 level=INFO msg="osupdate: START" run=20261111T043907Z layer=pve vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261111T043907Z
2026-11-11T05:40:09+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:40:09.469+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261111T043907Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=
2026-11-11T05:40:09+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:40:09.469+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-11T05:40:09+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:40:09.469+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
2026-11-11T05:40:09+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:40:09.469+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
2026-11-11T05:40:10+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:40:10.295+01:00 level=INFO msg="osupdate: DONE" run=20261111T043907Z layer=pve vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=1 not_covere
2026-11-11T05:40:10+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:40:10.301+01:00 level=INFO msg="osupdate: pve step released the /etc/pve write gate" run=20261111T043907Z layer=pve vmid=9201 trigger=night
2026-11-11T05:40:10+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:40:10.301+01:00 level=INFO msg="osupdate: kernel step skipped — the hub names no kernel step for this box" run=20261111T043907Z layer=kernel vmid=9201 trigger=night ring=0
2026-11-11T05:50:55+01:00 demo-felhom sudo[753077]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T06:05:54+01:00 demo-felhom sudo[770116]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T06:20:54+01:00 demo-felhom sudo[787124]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T06:35:54+01:00 demo-felhom sudo[804022]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T06:50:54+01:00 demo-felhom sudo[821020]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T07:05:54+01:00 demo-felhom sudo[837952]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T07:20:54+01:00 demo-felhom sudo[854940]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T07:35:54+01:00 demo-felhom sudo[871826]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T07:50:54+01:00 demo-felhom sudo[888788]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T08:05:54+01:00 demo-felhom sudo[905669]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T08:20:54+01:00 demo-felhom sudo[922546]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T08:35:54+01:00 demo-felhom sudo[939426]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T08:50:54+01:00 demo-felhom sudo[956204]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T09:05:54+01:00 demo-felhom sudo[973033]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T09:20:54+01:00 demo-felhom sudo[989961]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T09:35:54+01:00 demo-felhom sudo[1006771]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T09:50:54+01:00 demo-felhom sudo[1023597]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T09:51:01+01:00 demo-felhom sudo[1024025]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T09:51:11+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T09:51:11.409+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-11-11T10:05:54+01:00 demo-felhom sudo[1040510]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T10:20:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T10:20:50.077+01:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-11-10T
2026-11-11T10:20:51+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T10:20:51.359+01:00 level=INFO msg="restore-test: space preflight passed" storage=local-lvm required_bytes=9571729458 avail_bytes=363152785854
2026-11-11T10:20:51+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T10:20:51.385+01:00 level=INFO msg="restore-test: full-fidelity restore params derived from the archive config" scratch=990000 params=4
2026-11-11T10:20:54+01:00 demo-felhom sudo[1057518]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T10:22:20+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T10:22:20.013+01:00 level=INFO msg="restore-test: scratch guest torn down" vmid=990000
2026-11-11T10:22:20+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T10:22:20.014+01:00 level=INFO msg="backup: scheduled restore-test passed" archive=felhom-pbs:backup/ct/9201/2026-11-10T04:39:26Z duration_s=89.933700579
2026-11-11T10:35:54+01:00 demo-felhom sudo[1075697]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T10:50:54+01:00 demo-felhom sudo[1092663]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
=== agent log: line count per hour, last 26 h ===
12376 2026-11-10T09 13173 2026-11-10T10 13176 2026-11-10T11 13191 2026-11-10T12 13181 2026-11-10T13 13166 2026-11-10T14 13300 2026-11-10T15 13246 2026-11-10T16 13200 2026-11-10T17 13190 2026-11-10T18 13190 2026-11-10T19 13181 2026-11-10T20 13196 2026-11-10T21 13184 2026-11-10T22 13156 2026-11-10T23 13189 2026-11-11T00 13192 2026-11-11T01 13168 2026-11-11T02 13284 2026-11-11T03 13175 2026-11-11T04 13292 2026-11-11T05 13183 2026-11-11T06 13186 2026-11-11T07 13195 2026-11-11T08 13199 2026-11-11T09 13207 2026-11-11T10 826 2026-11-11T11
=== agent version ===
felhom-agent 0.173.0
RC=0
@@ -0,0 +1,89 @@
# host-night demo-felhom taken 2026-11-11T10:09:27Z
=== pve tasks, last 14 days (local time) ===
2026-10-29 04:41:48 -> 04:42:35 0.8 min vzdump 9201 OK
2026-10-29 04:46:00 -> 04:47:39 1.6 min stopall OK
2026-10-29 04:46:00 -> 04:47:39 1.6 min vzshutdown 9201 OK
2026-10-29 04:48:21 -> 04:48:25 0.1 min startall OK
2026-10-29 04:48:21 -> 04:48:25 0.1 min vzstart 9201 OK
2026-10-29 05:18:22 -> 05:19:08 0.8 min vzrestore 990000 OK
2026-10-29 05:19:13 -> 05:19:16 0.1 min vzstart 990000 OK
2026-10-29 05:19:19 -> 05:19:21 0.0 min vzdestroy 990000 OK
2026-10-29 06:54:18 -> 06:54:21 0.1 min aptupdate OK
2026-10-30 03:37:42 -> 03:37:46 0.1 min aptupdate OK
2026-10-30 04:46:29 -> 04:47:15 0.8 min vzdump 9201 OK
2026-10-30 05:34:15 -> 05:34:58 0.7 min vzrestore 990000 OK
2026-10-30 05:35:01 -> 05:35:03 0.0 min vzstart 990000 OK
2026-10-30 05:35:07 -> 05:35:09 0.0 min vzdestroy 990000 OK
2026-10-31 02:52:28 -> 02:52:31 0.1 min aptupdate OK
2026-10-31 04:49:27 -> 04:49:48 0.3 min vzrestore 990000 OK
2026-10-31 04:49:49 -> 04:49:51 0.0 min vzstart 990000 OK
2026-10-31 04:49:55 -> 04:49:57 0.0 min vzdestroy 990000 OK
2026-10-31 04:51:30 -> 04:52:20 0.8 min vzdump 9201 OK
2026-11-01 04:55:28 -> 04:55:31 0.1 min aptupdate OK
2026-11-01 04:56:28 -> 04:57:14 0.8 min vzdump 9201 OK
2026-11-01 10:49:27 -> 10:49:48 0.3 min vzrestore 990000 OK
2026-11-01 10:49:49 -> 10:49:51 0.0 min vzstart 990000 OK
2026-11-01 10:49:55 -> 10:49:57 0.0 min vzdestroy 990000 OK
2026-11-02 05:01:29 -> 05:02:15 0.8 min vzdump 9201 OK
2026-11-02 05:47:18 -> 05:47:21 0.1 min aptupdate OK
2026-11-02 10:49:27 -> 10:49:48 0.3 min vzrestore 990000 OK
2026-11-02 10:49:49 -> 10:49:51 0.0 min vzstart 990000 OK
2026-11-02 10:49:55 -> 10:49:57 0.0 min vzdestroy 990000 OK
2026-11-03 05:06:28 -> 05:07:15 0.8 min vzdump 9201 OK
2026-11-03 05:10:20 -> 05:10:30 0.2 min aptupdate OK
2026-11-03 05:35:12 -> 05:36:00 0.8 min vzdump 9201 OK
2026-11-03 09:19:40 -> 09:20:21 0.7 min vzrestore 990000 OK
2026-11-03 09:20:26 -> 09:20:29 0.1 min vzstart 990000 OK
2026-11-03 09:20:32 -> 09:20:35 0.1 min vzdestroy 990000 OK
2026-11-04 03:20:45 -> 03:20:48 0.1 min aptupdate OK
2026-11-04 05:09:34 -> 05:10:23 0.8 min vzdump 9201 OK
2026-11-04 09:20:40 -> 09:22:05 1.4 min vzrestore 990000 OK
2026-11-04 09:22:07 -> 09:22:09 0.0 min vzstart 990000 OK
2026-11-04 09:22:13 -> 09:22:15 0.0 min vzdestroy 990000 OK
2026-11-04 15:20:40 -> 15:21:20 0.7 min vzrestore 990000 OK
2026-11-04 15:21:21 -> 15:21:24 0.1 min vzstart 990000 OK
2026-11-04 15:21:27 -> 15:21:30 0.1 min vzdestroy 990000 OK
2026-11-05 02:09:12 -> 02:09:16 0.1 min aptupdate OK
2026-11-05 05:13:22 -> 05:14:18 0.9 min vzdump 9201 OK
2026-11-05 09:20:40 -> 09:21:22 0.7 min vzrestore 990000 OK
2026-11-05 09:21:26 -> 09:21:29 0.1 min vzstart 990000 OK
2026-11-05 09:21:32 -> 09:21:35 0.1 min vzdestroy 990000 OK
2026-11-06 04:33:41 -> 04:33:45 0.1 min aptupdate OK
2026-11-06 05:18:21 -> 05:19:11 0.8 min vzdump 9201 OK
2026-11-06 09:20:40 -> 09:21:23 0.7 min vzrestore 990000 OK
2026-11-06 09:21:26 -> 09:21:29 0.1 min vzstart 990000 OK
2026-11-06 09:21:32 -> 09:21:41 0.1 min vzdestroy 990000 OK
2026-11-07 05:10:41 -> 05:10:45 0.1 min aptupdate OK
2026-11-07 05:23:21 -> 05:24:10 0.8 min vzdump 9201 OK
2026-11-07 09:20:40 -> 09:21:22 0.7 min vzrestore 990000 OK
2026-11-07 09:21:26 -> 09:21:29 0.1 min vzstart 990000 OK
2026-11-07 09:21:32 -> 09:21:35 0.1 min vzdestroy 990000 OK
2026-11-08 01:08:23 -> 01:08:27 0.1 min aptupdate OK
2026-11-08 05:28:21 -> 05:29:10 0.8 min vzdump 9201 OK
2026-11-08 05:33:27 -> 05:35:06 1.6 min stopall OK
2026-11-08 05:33:27 -> 05:35:06 1.6 min vzshutdown 9201 OK
2026-11-08 05:35:30 -> 05:35:34 0.1 min startall OK
2026-11-08 05:35:30 -> 05:35:34 0.1 min vzstart 9201 OK
2026-11-08 06:05:31 -> 06:06:19 0.8 min vzrestore 990000 OK
2026-11-08 06:06:22 -> 06:06:25 0.1 min vzstart 990000 OK
2026-11-08 06:06:28 -> 06:06:30 0.0 min vzdestroy 990000 OK
2026-11-09 02:07:06 -> 02:07:09 0.1 min aptupdate OK
2026-11-09 05:31:03 -> 05:31:52 0.8 min vzdump 9201 OK
2026-11-09 06:06:36 -> 06:07:20 0.7 min vzrestore 990000 OK
2026-11-09 06:07:22 -> 06:07:24 0.0 min vzstart 990000 OK
2026-11-09 06:07:28 -> 06:07:30 0.0 min vzdestroy 990000 OK
2026-11-10 01:23:26 -> 01:23:29 0.1 min aptupdate OK
2026-11-10 05:34:26 -> 05:35:08 0.7 min vzdump 9201 OK
2026-11-10 05:39:26 -> 05:40:08 0.7 min vzdump 9201 OK
2026-11-10 06:06:36 -> 06:07:21 0.8 min vzrestore 990000 OK
2026-11-10 06:07:22 -> 06:07:24 0.0 min vzstart 990000 OK
2026-11-10 06:07:28 -> 06:07:30 0.0 min vzdestroy 990000 OK
2026-11-11 01:11:22 -> 01:11:25 0.1 min aptupdate OK
2026-11-11 05:36:51 -> 05:37:33 0.7 min vzdump 9201 OK
2026-11-11 10:20:51 -> 10:22:06 1.2 min vzrestore 990000 OK
2026-11-11 10:22:07 -> 10:22:10 0.1 min vzstart 990000 OK
2026-11-11 10:22:13 -> 10:22:16 0.1 min vzdestroy 990000 OK
=== agent: night words, 20:00 yesterday to 09:00 today (first 3 lines per message) ===
--- counts
RC=0
@@ -0,0 +1,6 @@
# offset demo-felhom: box clock minus DooPlex clock (midpoint of the call), seconds
offset=-32.638 rtt=0.135
offset=-32.641 rtt=0.128
offset=-32.639 rtt=0.132
offset=-32.638 rtt=0.133
offset=-32.642 rtt=0.127
@@ -0,0 +1,502 @@
# guest-clock demo-hp taken 2026-11-11T10:04:23Z from DooPlex
=== clock ===
utc 2026-11-11T10:04:24.512980194Z
local 2026-11-11T10:04:24 UTC +0000
/usr/share/zoneinfo/Etc/UTC
Local time: Wed 2026-11-11 10:04:24 UTC
Universal time: Wed 2026-11-11 10:04:24 UTC
RTC time: n/a
Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
NTP service: inactive
RTC in local TZ: no
chrony: inactive
systemd-timesyncd: inactive
=== boots ===
2026-11-10 14:43:12
=== timers ===
NEXT LEFT LAST PASSED UNIT ACTIVATES
Wed 2026-11-11 13:52:01 UTC 3h 47min Tue 2026-11-10 23:40:53 UTC 10h ago apt-daily.timer apt-daily.service
Wed 2026-11-11 14:58:53 UTC 4h 54min Tue 2026-11-10 14:58:53 UTC 19h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Thu 2026-11-12 00:00:00 UTC 13h Wed 2026-11-11 00:00:01 UTC 10h ago dpkg-db-backup.timer dpkg-db-backup.service
Thu 2026-11-12 00:04:10 UTC 13h Wed 2026-11-11 00:57:03 UTC 9h ago logrotate.timer logrotate.service
Thu 2026-11-12 00:52:48 UTC 14h Wed 2026-11-11 05:52:53 UTC 4h 11min ago man-db.timer man-db.service
Thu 2026-11-12 06:26:47 UTC 20h Wed 2026-11-11 06:45:13 UTC 3h 19min ago apt-daily-upgrade.timer apt-daily-upgrade.service
Sun 2026-11-15 03:10:58 UTC 3 days Sun 2026-11-08 03:10:50 UTC - e2scrub_all.timer e2scrub_all.service
- - - - fstrim.timer fstrim.service
8 timers listed.
=== timer units (all) ===
--- apt-daily-upgrade.timer [enabled/active]
OnCalendar=*-*-* 6:00
RandomizedDelaySec=60m
Persistent=true
Unit=apt-daily-upgrade.service NextElapseUSecRealtime=Thu 2026-11-12 06:26:47 UTC LastTriggerUSec=Wed 2026-11-11 06:45:13 UTC
--- apt-daily.timer [enabled/active]
OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h
Persistent=true
Unit=apt-daily.service NextElapseUSecRealtime=Wed 2026-11-11 13:52:01 UTC LastTriggerUSec=Tue 2026-11-10 23:40:53 UTC
--- apt-listchanges.timer [disabled/inactive]
OnCalendar=hourly
Unit=apt-listchanges.service NextElapseUSecRealtime= LastTriggerUSec=
--- dpkg-db-backup.timer [enabled/active]
OnCalendar=daily
Persistent=true
Unit=dpkg-db-backup.service NextElapseUSecRealtime=Thu 2026-11-12 00:00:00 UTC LastTriggerUSec=Wed 2026-11-11 00:00:01 UTC
--- e2scrub_all.timer [enabled/active]
OnCalendar=Sun *-*-* 03:10:00
RandomizedDelaySec=60
Persistent=true
Unit=e2scrub_all.service NextElapseUSecRealtime=Sun 2026-11-15 03:10:58 UTC LastTriggerUSec=Sun 2026-11-08 03:10:50 UTC
--- fstrim.timer [enabled/inactive]
OnCalendar=weekly
AccuracySec=1h
Persistent=true
RandomizedDelaySec=100min
Unit=fstrim.service NextElapseUSecRealtime= LastTriggerUSec=
--- logrotate.timer [enabled/active]
OnCalendar=daily
RandomizedDelaySec=1h
Persistent=true
Unit=logrotate.service NextElapseUSecRealtime=Thu 2026-11-12 00:04:10 UTC LastTriggerUSec=Wed 2026-11-11 00:57:03 UTC
--- man-db.timer [enabled/active]
OnCalendar=daily
RandomizedDelaySec=12h
Persistent=true
Unit=man-db.service NextElapseUSecRealtime=Thu 2026-11-12 00:52:48 UTC LastTriggerUSec=Wed 2026-11-11 05:52:53 UTC
--- systemd-tmpfiles-clean.timer [static/active]
OnBootSec=15min
OnUnitActiveSec=1d
Unit=systemd-tmpfiles-clean.service NextElapseUSecRealtime= LastTriggerUSec=Tue 2026-11-10 14:58:53 UTC
=== felhom units ===
felhom-controller-bootstrap.path loaded active running
felhom-controller-bootstrap.service loaded active exited
felhom-regen-hostkeys.service loaded inactive dead
--- felhom-controller-bootstrap.service
Type=oneshot Restart=no RestartUSec=100ms TimeoutStartUSec=infinity NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:17:04 UTC
--- felhom-regen-hostkeys.service
Type=oneshot Restart=no RestartUSec=100ms TimeoutStartUSec=infinity NRestarts=0 ActiveEnterTimestamp=
=== cron ===
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
27 * * * * root cd / && run-parts --report /etc/cron.hourly
28 6 * * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; }
45 6 * * 7 root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.weekly; }
42 6 1 * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.monthly; }
--- /etc/cron.d/e2scrub_all
57 3 * * 0 root test -e /run/systemd/system || SERVICE_MODE=1 /usr/libexec/e2fsprogs/e2scrub_all_cron
27 3 * * * root test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r
--- cron.hourly:
--- cron.daily: apt-compat dpkg logrotate man-db
--- cron.weekly: man-db
--- cron.monthly:
--- user crontabs:
=== at ===
no at
=== docker daemon timers ===
{
"features": {
"containerd-snapshotter": false
},
"live-restore": true,
"log-driver": "json-file",
"log-opts": {
"max-file": "3",
"max-size": "10m"
}
}
=== containers ===
romm rommapp/romm:5.4.1 Up 7 hours (healthy) 7 hours ago
romm-db mariadb:11.8 Up 7 hours (healthy) 7 hours ago
romm-redis redis:7-alpine Up 7 hours (healthy) 7 hours ago
privatebin privatebin/pdo:2.1.0 Up 7 hours (healthy) 7 hours ago
plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0 Up 7 hours (healthy) 7 hours ago
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:3.4.1 Up 7 hours (healthy) 7 hours ago
paperless-redis redis:7-alpine Up 7 hours (healthy) 7 hours ago
paperless-postgres postgres:18-alpine Up 7 hours (healthy) 7 hours ago
opengist ghcr.io/thomiceli/opengist:1.15 Up 7 hours (healthy) 7 hours ago
kimai kimai/kimai2:2.67.0 Up 7 hours (healthy) 7 hours ago
kimai-db mariadb:11.8 Up 7 hours (healthy) 7 hours ago
home-assistant ghcr.io/home-assistant/home-assistant:2026.10.4 Up 7 hours (healthy) 7 hours ago
docmost docmost/docmost:0.97.0 Up 7 hours (healthy) 7 hours ago
docmost-postgres postgres:18-alpine Up 7 hours (healthy) 7 hours ago
docmost-redis redis:7-alpine Up 7 hours (healthy) 7 hours ago
calibre-web crocodilestick/calibre-web-automated:v4.0.8 Up 7 hours (healthy) 7 hours ago
bookstack lscr.io/linuxserver/bookstack:26.09.2 Up 7 hours (healthy) 7 hours ago
bookstack-db mariadb:12.3 Up 7 hours (healthy) 7 hours ago
bentopdf ghcr.io/alam00000/bentopdf:v2.8.10 Up 7 hours (healthy) 7 hours ago
adventurelog ghcr.io/seanmorley15/adventurelog-backend:v0.13.1 Up 7 hours (healthy) 7 hours ago
adventurelog-postgres postgis/postgis:16-3.5-alpine Up 7 hours (healthy) 7 hours ago
adventurelog-frontend ghcr.io/seanmorley15/adventurelog-frontend:v0.13.1 Up 7 hours (healthy) 7 hours ago
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 19 hours (healthy) 19 hours ago
traefik traefik:v3.7.17 Up 19 hours 8 days ago
cloudflared cloudflare/cloudflared:2026.10.4 Up 19 hours (healthy) 2 weeks ago
filebrowser gtstef/filebrowser:1.5.8-stable Up 19 hours (healthy) 2 weeks ago
=== restart policies + healthcheck intervals ===
/romm restart=unless-stopped health=30s started=2026-11-11T03:34:23.246874464Z
/romm-db restart=unless-stopped health=10s started=2026-11-11T03:34:11.866673198Z
/romm-redis restart=unless-stopped health=10s started=2026-11-11T03:34:12.322525014Z
/privatebin restart=unless-stopped health=30s started=2026-11-11T03:34:10.436496587Z
/plex restart=unless-stopped health=30s started=2026-11-11T03:34:09.416236529Z
/paperless-webserver restart=unless-stopped health=30s started=2026-11-11T03:34:06.356684002Z
/paperless-redis restart=unless-stopped health=10s started=2026-11-11T03:33:55.572315165Z
/paperless-postgres restart=unless-stopped health=10s started=2026-11-11T03:33:55.30857347Z
/opengist restart=unless-stopped health=30s started=2026-11-11T03:33:54.299740686Z
/kimai restart=unless-stopped health=30s started=2026-11-11T03:33:52.511754977Z
/kimai-db restart=unless-stopped health=10s started=2026-11-11T03:33:46.665886435Z
/home-assistant restart=unless-stopped health=30s started=2026-11-11T03:33:45.366565279Z
/docmost restart=unless-stopped health=30s started=2026-11-11T03:33:38.285678301Z
/docmost-postgres restart=unless-stopped health=10s started=2026-11-11T03:33:27.248437109Z
/docmost-redis restart=unless-stopped health=10s started=2026-11-11T03:33:27.525319794Z
/calibre-web restart=unless-stopped health=30s started=2026-11-11T03:33:26.194198782Z
/bookstack restart=unless-stopped health=30s started=2026-11-11T03:33:25.391109868Z
/bookstack-db restart=unless-stopped health=10s started=2026-11-11T03:33:19.649504107Z
/bentopdf restart=unless-stopped health=30s started=2026-11-11T03:33:18.71207642Z
/adventurelog restart=unless-stopped health=30s started=2026-11-11T03:33:17.943139983Z
/adventurelog-postgres restart=unless-stopped health=10s started=2026-11-11T03:33:12.219942766Z
/adventurelog-frontend restart=unless-stopped health=30s started=2026-11-11T03:33:11.981285328Z
/felhom-controller restart=unless-stopped health=30s started=2026-11-10T15:17:03.918469562Z
/traefik restart=unless-stopped health=none started=2026-11-10T14:43:21.982539486Z
/cloudflared restart=unless-stopped health=30s started=2026-11-10T14:43:21.872203889Z
/filebrowser restart=unless-stopped health=30s started=2026-11-10T14:43:32.41426216Z
=== controller TZ ===
TZ=unset
in-container 2026-11-11T10:04:27 UTC +0000
=== controller log: first line, line count per hour ===
2026-11-10T23:15:05.505129813Z 2026/11/10 23:15:05 manager.go:687: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml
9125 2026-11-10T23 12234 2026-11-11T00 12689 2026-11-11T01 12383 2026-11-11T02 13030 2026-11-11T03 12262 2026-11-11T04 12254 2026-11-11T05 12243 2026-11-11T06 12243 2026-11-11T07 12231 2026-11-11T08 12250 2026-11-11T09 1057 2026-11-11T10
=== controller log: schedule words (tail 320) ===
2026-11-11T09:57:05.391680485Z 2026/11/11 09:57:05 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
2026-11-11T09:57:05.391684032Z 2026/11/11 09:57:05 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T09:57:05.391687198Z 2026/11/11 09:57:05 scheduler.go:450: [INFO] [scheduler] Running job: hold-report
2026-11-11T09:57:05.391702807Z 2026/11/11 09:57:05 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting
2026-11-11T09:57:05.391706013Z 2026/11/11 09:57:05 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:57:05.398983593Z 2026/11/11 09:57:05 scheduler.go:450: [INFO] [scheduler] Running job: backup-cache
2026-11-11T09:57:05.399062742Z 2026/11/11 09:57:05 scheduler.go:85: [DEBUG] [scheduler] job backup-cache: execution starting
2026-11-11T09:57:05.399096566Z 2026/11/11 09:57:05 scheduler.go:450: [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T09:57:05.399128506Z 2026/11/11 09:57:05 scheduler.go:85: [DEBUG] [scheduler] job offsite-credential-retry: execution starting
2026-11-11T09:57:05.399139557Z 2026/11/11 09:57:05 scheduler.go:467: [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T09:57:05.496882413Z 2026/11/11 09:57:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:57:05.497054418Z 2026/11/11 09:57:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting
2026-11-11T09:57:05.695301873Z 2026/11/11 09:57:05 scheduler.go:467: [INFO] [scheduler] Job docker-socket-users completed (took 305ms)
2026-11-11T09:57:05.711141758Z 2026/11/11 09:57:05 scheduler.go:467: [INFO] [scheduler] Job stack-scan completed (took 320ms)
2026-11-11T09:57:05.716073599Z 2026/11/11 09:57:05 backup.go:1252: [INFO] [backup] Found 21 DB dump files across drives
2026-11-11T09:57:05.726924596Z 2026/11/11 09:57:05 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers
2026-11-11T09:57:05.773888562Z 2026/11/11 09:57:05 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: e8a4549a77a0 romm romm rommapp/romm:5.4.1
2026-11-11T09:57:05.773952964Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.4.1, not a database)
2026-11-11T09:57:05.773956290Z 2026/11/11 09:57:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=0c0a552f1d66)
2026-11-11T09:57:05.786675764Z 2026/11/11 09:57:05 scheduler.go:467: [INFO] [scheduler] Job system-health completed (took 396ms)
2026-11-11T09:57:05.813792380Z 2026/11/11 09:57:05 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm
2026-11-11T09:57:05.813834600Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database)
2026-11-11T09:57:05.813858254Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.1.0, not a database)
2026-11-11T09:57:05.813862582Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container plex (image=plexinc/pms-docker:1.43.5.11029-6bf8af3f0, not a database)
2026-11-11T09:57:05.813881508Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:3.4.1, not a database)
2026-11-11T09:57:05.813886347Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
2026-11-11T09:57:05.813889844Z 2026/11/11 09:57:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=d5dc3d71319d)
2026-11-11T09:57:05.813893120Z 2026/11/11 09:57:05 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
2026-11-11T09:57:05.849834747Z 2026/11/11 09:57:05 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
2026-11-11T09:57:05.849868540Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container opengist (image=ghcr.io/thomiceli/opengist:1.15, not a database)
2026-11-11T09:57:05.849872859Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container kimai (image=kimai/kimai2:2.67.0, not a database)
2026-11-11T09:57:05.849876555Z 2026/11/11 09:57:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: kimai-db (id=170178b88458)
2026-11-11T09:57:05.885779890Z 2026/11/11 09:57:05 dbdump.go:198: [DEBUG] DiscoverDatabases: kimai-db → stack=kimai, dbUser=root, dbName=kimai
2026-11-11T09:57:05.885810207Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container home-assistant (image=ghcr.io/home-assistant/home-assistant:2026.10.4, not a database)
2026-11-11T09:57:05.885814836Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost (image=docmost/docmost:0.97.0, not a database)
2026-11-11T09:57:05.885818413Z 2026/11/11 09:57:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: docmost-postgres (id=ed402849ac4c)
2026-11-11T09:57:05.916260477Z 2026/11/11 09:57:05 dbdump.go:198: [DEBUG] DiscoverDatabases: docmost-postgres → stack=docmost, dbUser=docmost, dbName=docmost
2026-11-11T09:57:05.916299420Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost-redis (image=redis:7-alpine, not a database)
2026-11-11T09:57:05.916303839Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container calibre-web (image=crocodilestick/calibre-web-automated:v4.0.8, not a database)
2026-11-11T09:57:05.916307465Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container bookstack (image=lscr.io/linuxserver/bookstack:26.09.2, not a database)
2026-11-11T09:57:05.916310651Z 2026/11/11 09:57:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: bookstack-db (id=bd002743cdb7)
2026-11-11T09:57:05.947648136Z 2026/11/11 09:57:05 dbdump.go:198: [DEBUG] DiscoverDatabases: bookstack-db → stack=bookstack, dbUser=root, dbName=bookstack
2026-11-11T09:57:05.947696166Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container bentopdf (image=ghcr.io/alam00000/bentopdf:v2.8.10, not a database)
2026-11-11T09:57:05.947701426Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog (image=ghcr.io/seanmorley15/adventurelog-backend:v0.13.1, not a database)
2026-11-11T09:57:05.947705344Z 2026/11/11 09:57:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: adventurelog-postgres (id=39ccc9825d38)
2026-11-11T09:57:05.979127858Z 2026/11/11 09:57:05 dbdump.go:198: [DEBUG] DiscoverDatabases: adventurelog-postgres → stack=adventurelog, dbUser=adventurelog, dbName=adventurelog
2026-11-11T09:57:05.979858929Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog-frontend (image=ghcr.io/seanmorley15/adventurelog-frontend:v0.13.1, not a database)
2026-11-11T09:57:05.979889687Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.352.0, not a database)
2026-11-11T09:57:05.979893784Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.17, not a database)
2026-11-11T09:57:05.979897291Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container cloudflared (image=cloudflare/cloudflared:2026.10.4, not a database)
2026-11-11T09:57:05.979900938Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.8-stable, not a database)
2026-11-11T09:57:05.979904274Z 2026/11/11 09:57:05 dbdump.go:205: [DEBUG] DiscoverDatabases: found 6 database(s), skipped 20 non-DB container(s)
2026-11-11T09:57:05.979907420Z 2026/11/11 09:57:05 dbdump.go:208: [INFO] [backup] Discovered 6 databases
2026-11-11T09:57:06.072467391Z 2026/11/11 09:57:06 scheduler.go:467: [INFO] [scheduler] Job backup-cache completed (took 674ms)
2026-11-11T09:57:06.156094586Z 2026/11/11 09:57:06 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 659ms)
2026-11-11T09:57:15.391174599Z 2026/11/11 09:57:15 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T09:57:15.391220426Z 2026/11/11 09:57:15 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T09:57:20.390884897Z 2026/11/11 09:57:20 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T09:57:25.389356846Z 2026/11/11 09:57:25 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T09:57:25.389402032Z 2026/11/11 09:57:25 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T09:57:35.391423638Z 2026/11/11 09:57:35 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T09:57:35.391463113Z 2026/11/11 09:57:35 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting
2026-11-11T09:57:35.391467260Z 2026/11/11 09:57:35 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting
2026-11-11T09:57:35.391478602Z 2026/11/11 09:57:35 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T09:57:35.391904236Z 2026/11/11 09:57:35 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
2026-11-11T09:57:35.391935715Z 2026/11/11 09:57:35 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T09:57:45.389771608Z 2026/11/11 09:57:45 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T09:57:45.389882147Z 2026/11/11 09:57:45 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T09:57:50.390771070Z 2026/11/11 09:57:50 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T09:57:55.390753201Z 2026/11/11 09:57:55 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T09:57:55.390880702Z 2026/11/11 09:57:55 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T09:58:05.389539549Z 2026/11/11 09:58:05 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T09:58:05.389570227Z 2026/11/11 09:58:05 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T09:58:05.389575026Z 2026/11/11 09:58:05 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
2026-11-11T09:58:05.389579184Z 2026/11/11 09:58:05 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting
2026-11-11T09:58:05.389583131Z 2026/11/11 09:58:05 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting
2026-11-11T09:58:05.390410503Z 2026/11/11 09:58:05 scheduler.go:450: [INFO] [scheduler] Running job: hold-report
2026-11-11T09:58:05.391331873Z 2026/11/11 09:58:05 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting
2026-11-11T09:58:05.391365216Z 2026/11/11 09:58:05 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:58:05.391479281Z 2026/11/11 09:58:05 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T09:58:05.496565621Z 2026/11/11 09:58:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:58:05.496603543Z 2026/11/11 09:58:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting
2026-11-11T09:58:06.147290381Z 2026/11/11 09:58:06 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 651ms)
2026-11-11T09:58:15.389718636Z 2026/11/11 09:58:15 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T09:58:15.389814016Z 2026/11/11 09:58:15 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T09:58:20.390767652Z 2026/11/11 09:58:20 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T09:58:25.389232498Z 2026/11/11 09:58:25 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T09:58:25.389349288Z 2026/11/11 09:58:25 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T09:58:35.390771513Z 2026/11/11 09:58:35 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting
2026-11-11T09:58:35.390809485Z 2026/11/11 09:58:35 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
2026-11-11T09:58:35.390818452Z 2026/11/11 09:58:35 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T09:58:35.391003812Z 2026/11/11 09:58:35 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting
2026-11-11T09:58:35.391099092Z 2026/11/11 09:58:35 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T09:58:35.391235469Z 2026/11/11 09:58:35 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T09:58:45.390879506Z 2026/11/11 09:58:45 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T09:58:45.390956802Z 2026/11/11 09:58:45 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T09:58:50.390733225Z 2026/11/11 09:58:50 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T09:58:55.391093460Z 2026/11/11 09:58:55 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T09:58:55.391251869Z 2026/11/11 09:58:55 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T09:59:05.389079767Z 2026/11/11 09:59:05 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T09:59:05.389125574Z 2026/11/11 09:59:05 scheduler.go:450: [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:59:05.389130163Z 2026/11/11 09:59:05 scheduler.go:85: [DEBUG] [scheduler] job stack-scan: execution starting
2026-11-11T09:59:05.389183233Z 2026/11/11 09:59:05 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T09:59:05.389349446Z 2026/11/11 09:59:05 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting
2026-11-11T09:59:05.389357441Z 2026/11/11 09:59:05 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T09:59:05.389361539Z 2026/11/11 09:59:05 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
2026-11-11T09:59:05.389365156Z 2026/11/11 09:59:05 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting
2026-11-11T09:59:05.389434237Z 2026/11/11 09:59:05 scheduler.go:450: [INFO] [scheduler] Running job: hold-report
2026-11-11T09:59:05.389763308Z 2026/11/11 09:59:05 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting
2026-11-11T09:59:05.389795940Z 2026/11/11 09:59:05 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:59:05.496793027Z 2026/11/11 09:59:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:59:05.496820689Z 2026/11/11 09:59:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting
2026-11-11T09:59:05.546081752Z 2026/11/11 09:59:05 scheduler.go:467: [INFO] [scheduler] Job stack-scan completed (took 157ms)
2026-11-11T09:59:06.138583409Z 2026/11/11 09:59:06 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 642ms)
2026-11-11T09:59:15.389722199Z 2026/11/11 09:59:15 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T09:59:15.389844350Z 2026/11/11 09:59:15 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T09:59:20.390897764Z 2026/11/11 09:59:20 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T09:59:25.389905394Z 2026/11/11 09:59:25 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T09:59:25.389944187Z 2026/11/11 09:59:25 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T09:59:35.390723549Z 2026/11/11 09:59:35 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting
2026-11-11T09:59:35.390762122Z 2026/11/11 09:59:35 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
2026-11-11T09:59:35.390767712Z 2026/11/11 09:59:35 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T09:59:35.390772101Z 2026/11/11 09:59:35 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T09:59:35.390781188Z 2026/11/11 09:59:35 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T09:59:35.390889132Z 2026/11/11 09:59:35 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting
2026-11-11T09:59:45.390790355Z 2026/11/11 09:59:45 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T09:59:45.390905241Z 2026/11/11 09:59:45 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T09:59:50.389748662Z 2026/11/11 09:59:50 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T09:59:55.390894642Z 2026/11/11 09:59:55 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T09:59:55.390960737Z 2026/11/11 09:59:55 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:00:05.389436996Z 2026/11/11 10:00:05 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
2026-11-11T10:00:05.389475098Z 2026/11/11 10:00:05 scheduler.go:450: [INFO] [scheduler] Running job: hold-report
2026-11-11T10:00:05.389480338Z 2026/11/11 10:00:05 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting
2026-11-11T10:00:05.389484315Z 2026/11/11 10:00:05 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting
2026-11-11T10:00:05.390130095Z 2026/11/11 10:00:05 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:00:05.390159310Z 2026/11/11 10:00:05 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:00:05.390164259Z 2026/11/11 10:00:05 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting
2026-11-11T10:00:05.390167836Z 2026/11/11 10:00:05 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T10:00:05.390171252Z 2026/11/11 10:00:05 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T10:00:05.496014350Z 2026/11/11 10:00:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T10:00:05.496056049Z 2026/11/11 10:00:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting
2026-11-11T10:00:06.214357347Z 2026/11/11 10:00:06 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 718ms)
2026-11-11T10:00:15.390703141Z 2026/11/11 10:00:15 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:00:15.390812899Z 2026/11/11 10:00:15 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:00:20.390761640Z 2026/11/11 10:00:20 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T10:00:25.389255824Z 2026/11/11 10:00:25 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:00:25.389400588Z 2026/11/11 10:00:25 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:00:35.389066542Z 2026/11/11 10:00:35 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:00:35.389110936Z 2026/11/11 10:00:35 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T10:00:35.389329970Z 2026/11/11 10:00:35 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
2026-11-11T10:00:35.389343415Z 2026/11/11 10:00:35 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting
2026-11-11T10:00:35.389350098Z 2026/11/11 10:00:35 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:00:35.389355869Z 2026/11/11 10:00:35 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting
2026-11-11T10:00:45.390716182Z 2026/11/11 10:00:45 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:00:45.390752881Z 2026/11/11 10:00:45 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:00:50.390752909Z 2026/11/11 10:00:50 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T10:00:55.390768987Z 2026/11/11 10:00:55 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:00:55.390997839Z 2026/11/11 10:00:55 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:01:05.389781849Z 2026/11/11 10:01:05 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting
2026-11-11T10:01:05.389815813Z 2026/11/11 10:01:05 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T10:01:05.389819861Z 2026/11/11 10:01:05 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
2026-11-11T10:01:05.389823578Z 2026/11/11 10:01:05 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:01:05.389826784Z 2026/11/11 10:01:05 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting
2026-11-11T10:01:05.389829920Z 2026/11/11 10:01:05 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:01:05.389960667Z 2026/11/11 10:01:05 scheduler.go:450: [INFO] [scheduler] Running job: stack-scan
2026-11-11T10:01:05.389971487Z 2026/11/11 10:01:05 scheduler.go:85: [DEBUG] [scheduler] job stack-scan: execution starting
2026-11-11T10:01:05.389975475Z 2026/11/11 10:01:05 scheduler.go:450: [INFO] [scheduler] Running job: hold-report
2026-11-11T10:01:05.389979071Z 2026/11/11 10:01:05 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting
2026-11-11T10:01:05.390019447Z 2026/11/11 10:01:05 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T10:01:05.496095034Z 2026/11/11 10:01:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T10:01:05.496126183Z 2026/11/11 10:01:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting
2026-11-11T10:01:05.562234764Z 2026/11/11 10:01:05 scheduler.go:467: [INFO] [scheduler] Job stack-scan completed (took 172ms)
2026-11-11T10:01:06.108225686Z 2026/11/11 10:01:06 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 612ms)
2026-11-11T10:01:15.389714517Z 2026/11/11 10:01:15 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:01:15.389788557Z 2026/11/11 10:01:15 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:01:20.390818659Z 2026/11/11 10:01:20 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T10:01:25.388861282Z 2026/11/11 10:01:25 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:01:25.389373799Z 2026/11/11 10:01:25 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:01:35.390747057Z 2026/11/11 10:01:35 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T10:01:35.390788105Z 2026/11/11 10:01:35 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:01:35.390793244Z 2026/11/11 10:01:35 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
2026-11-11T10:01:35.390802171Z 2026/11/11 10:01:35 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting
2026-11-11T10:01:35.390953747Z 2026/11/11 10:01:35 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:01:35.390996538Z 2026/11/11 10:01:35 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting
2026-11-11T10:01:45.391055024Z 2026/11/11 10:01:45 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:01:45.391763742Z 2026/11/11 10:01:45 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:01:50.390812373Z 2026/11/11 10:01:50 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T10:01:55.390786933Z 2026/11/11 10:01:55 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:01:55.390949149Z 2026/11/11 10:01:55 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:02:05.387578911Z 2026/11/11 10:02:05 scheduler.go:450: [INFO] [scheduler] Running job: hub-report
2026-11-11T10:02:05.387924694Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job hub-report: execution starting
2026-11-11T10:02:05.389380553Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting
2026-11-11T10:02:05.389536748Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting
2026-11-11T10:02:05.391369959Z 2026/11/11 10:02:05 scheduler.go:450: [INFO] [scheduler] Running job: hold-report
2026-11-11T10:02:05.391513340Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting
2026-11-11T10:02:05.391674855Z 2026/11/11 10:02:05 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T10:02:05.391811272Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T10:02:05.392586205Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
2026-11-11T10:02:05.392611794Z 2026/11/11 10:02:05 scheduler.go:450: [INFO] [scheduler] Running job: system-health
2026-11-11T10:02:05.392616312Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job system-health: execution starting
2026-11-11T10:02:05.392864891Z 2026/11/11 10:02:05 scheduler.go:450: [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T10:02:05.393071522Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-users: execution starting
2026-11-11T10:02:05.393246031Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:02:05.394939408Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:02:05.399781049Z 2026/11/11 10:02:05 scheduler.go:450: [INFO] [scheduler] Running job: backup-cache
2026-11-11T10:02:05.399804212Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job backup-cache: execution starting
2026-11-11T10:02:05.402003384Z 2026/11/11 10:02:05 scheduler.go:450: [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T10:02:05.402038621Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job offsite-credential-retry: execution starting
2026-11-11T10:02:05.402055192Z 2026/11/11 10:02:05 scheduler.go:467: [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T10:02:05.495851567Z 2026/11/11 10:02:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T10:02:05.495974188Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting
2026-11-11T10:02:05.504097565Z 2026/11/11 10:02:05 backup.go:1252: [INFO] [backup] Found 21 DB dump files across drives
2026-11-11T10:02:05.507146601Z 2026/11/11 10:02:05 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers
2026-11-11T10:02:05.592428120Z 2026/11/11 10:02:05 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: e8a4549a77a0 romm romm rommapp/romm:5.4.1
2026-11-11T10:02:05.592503783Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.4.1, not a database)
2026-11-11T10:02:05.592506899Z 2026/11/11 10:02:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=0c0a552f1d66)
2026-11-11T10:02:05.680886095Z 2026/11/11 10:02:05 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm
2026-11-11T10:02:05.680889582Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database)
2026-11-11T10:02:05.680892848Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.1.0, not a database)
2026-11-11T10:02:05.680895914Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container plex (image=plexinc/pms-docker:1.43.5.11029-6bf8af3f0, not a database)
2026-11-11T10:02:05.680899320Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:3.4.1, not a database)
2026-11-11T10:02:05.680902637Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
2026-11-11T10:02:05.680905753Z 2026/11/11 10:02:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=d5dc3d71319d)
2026-11-11T10:02:05.680908568Z 2026/11/11 10:02:05 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
2026-11-11T10:02:05.683188011Z 2026/11/11 10:02:05 sync.go:299: [INFO] [sync] Catalog sync complete
2026-11-11T10:02:05.742313528Z 2026/11/11 10:02:05 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
2026-11-11T10:02:05.742663499Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container opengist (image=ghcr.io/thomiceli/opengist:1.15, not a database)
2026-11-11T10:02:05.742729233Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container kimai (image=kimai/kimai2:2.67.0, not a database)
2026-11-11T10:02:05.742735595Z 2026/11/11 10:02:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: kimai-db (id=170178b88458)
2026-11-11T10:02:05.807651023Z 2026/11/11 10:02:05 scheduler.go:467: [INFO] [scheduler] Job docker-socket-users completed (took 414ms)
2026-11-11T10:02:05.807686861Z 2026/11/11 10:02:05 dbdump.go:198: [DEBUG] DiscoverDatabases: kimai-db → stack=kimai, dbUser=root, dbName=kimai
2026-11-11T10:02:05.807752876Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container home-assistant (image=ghcr.io/home-assistant/home-assistant:2026.10.4, not a database)
2026-11-11T10:02:05.807763816Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost (image=docmost/docmost:0.97.0, not a database)
2026-11-11T10:02:05.807856421Z 2026/11/11 10:02:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: docmost-postgres (id=ed402849ac4c)
2026-11-11T10:02:05.851076910Z 2026/11/11 10:02:05 dbdump.go:198: [DEBUG] DiscoverDatabases: docmost-postgres → stack=docmost, dbUser=docmost, dbName=docmost
2026-11-11T10:02:05.851146251Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost-redis (image=redis:7-alpine, not a database)
2026-11-11T10:02:05.851152713Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container calibre-web (image=crocodilestick/calibre-web-automated:v4.0.8, not a database)
2026-11-11T10:02:05.851156781Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container bookstack (image=lscr.io/linuxserver/bookstack:26.09.2, not a database)
2026-11-11T10:02:05.851160528Z 2026/11/11 10:02:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: bookstack-db (id=bd002743cdb7)
2026-11-11T10:02:05.887691086Z 2026/11/11 10:02:05 dbdump.go:198: [DEBUG] DiscoverDatabases: bookstack-db → stack=bookstack, dbUser=root, dbName=bookstack
2026-11-11T10:02:05.887975193Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container bentopdf (image=ghcr.io/alam00000/bentopdf:v2.8.10, not a database)
2026-11-11T10:02:05.888128943Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog (image=ghcr.io/seanmorley15/adventurelog-backend:v0.13.1, not a database)
2026-11-11T10:02:05.888276010Z 2026/11/11 10:02:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: adventurelog-postgres (id=39ccc9825d38)
2026-11-11T10:02:05.903919495Z 2026/11/11 10:02:05 scheduler.go:467: [INFO] [scheduler] Job system-health completed (took 511ms)
2026-11-11T10:02:05.921175844Z 2026/11/11 10:02:05 dbdump.go:198: [DEBUG] DiscoverDatabases: adventurelog-postgres → stack=adventurelog, dbUser=adventurelog, dbName=adventurelog
2026-11-11T10:02:05.921210189Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog-frontend (image=ghcr.io/seanmorley15/adventurelog-frontend:v0.13.1, not a database)
2026-11-11T10:02:05.921214067Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.352.0, not a database)
2026-11-11T10:02:05.921217403Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.17, not a database)
2026-11-11T10:02:05.921220609Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container cloudflared (image=cloudflare/cloudflared:2026.10.4, not a database)
2026-11-11T10:02:05.921232732Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.8-stable, not a database)
2026-11-11T10:02:05.921236709Z 2026/11/11 10:02:05 dbdump.go:205: [DEBUG] DiscoverDatabases: found 6 database(s), skipped 20 non-DB container(s)
2026-11-11T10:02:05.921291703Z 2026/11/11 10:02:05 dbdump.go:208: [INFO] [backup] Discovered 6 databases
2026-11-11T10:02:06.186619994Z 2026/11/11 10:02:06 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 690ms)
2026-11-11T10:02:06.368610565Z 2026/11/11 10:02:06 scheduler.go:467: [INFO] [scheduler] Job backup-cache completed (took 970ms)
2026-11-11T10:02:07.513372767Z 2026/11/11 10:02:07 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action
2026-11-11T10:02:07.513398967Z 2026/11/11 10:02:07 scheduler.go:467: [INFO] [scheduler] Job hub-report completed (took 2.126s)
2026-11-11T10:02:15.391097059Z 2026/11/11 10:02:15 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:02:15.391173023Z 2026/11/11 10:02:15 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:02:20.389796882Z 2026/11/11 10:02:20 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T10:02:25.390664969Z 2026/11/11 10:02:25 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:02:25.390703060Z 2026/11/11 10:02:25 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:02:35.390676456Z 2026/11/11 10:02:35 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting
2026-11-11T10:02:35.390722753Z 2026/11/11 10:02:35 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
2026-11-11T10:02:35.390731730Z 2026/11/11 10:02:35 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:02:35.390751377Z 2026/11/11 10:02:35 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting
2026-11-11T10:02:35.390811912Z 2026/11/11 10:02:35 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:02:35.391461909Z 2026/11/11 10:02:35 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T10:02:45.390729942Z 2026/11/11 10:02:45 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:02:45.390764447Z 2026/11/11 10:02:45 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:02:50.390855407Z 2026/11/11 10:02:50 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T10:02:55.389055897Z 2026/11/11 10:02:55 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:02:55.389331066Z 2026/11/11 10:02:55 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:03:05.389612583Z 2026/11/11 10:03:05 scheduler.go:450: [INFO] [scheduler] Running job: hold-report
2026-11-11T10:03:05.389648330Z 2026/11/11 10:03:05 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting
2026-11-11T10:03:05.389652959Z 2026/11/11 10:03:05 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T10:03:05.389656737Z 2026/11/11 10:03:05 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
2026-11-11T10:03:05.389660403Z 2026/11/11 10:03:05 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:03:05.389663830Z 2026/11/11 10:03:05 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T10:03:05.389670893Z 2026/11/11 10:03:05 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting
2026-11-11T10:03:05.389683487Z 2026/11/11 10:03:05 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting
2026-11-11T10:03:05.390470593Z 2026/11/11 10:03:05 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:03:05.390552227Z 2026/11/11 10:03:05 scheduler.go:450: [INFO] [scheduler] Running job: stack-scan
2026-11-11T10:03:05.390556856Z 2026/11/11 10:03:05 scheduler.go:85: [DEBUG] [scheduler] job stack-scan: execution starting
2026-11-11T10:03:05.495928234Z 2026/11/11 10:03:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T10:03:05.495933243Z 2026/11/11 10:03:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting
2026-11-11T10:03:05.568710434Z 2026/11/11 10:03:05 scheduler.go:467: [INFO] [scheduler] Job stack-scan completed (took 178ms)
2026-11-11T10:03:06.174079520Z 2026/11/11 10:03:06 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 678ms)
2026-11-11T10:03:15.390644266Z 2026/11/11 10:03:15 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:03:15.390681356Z 2026/11/11 10:03:15 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:03:20.390760133Z 2026/11/11 10:03:20 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T10:03:25.389607284Z 2026/11/11 10:03:25 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:03:25.389662548Z 2026/11/11 10:03:25 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:03:35.390668091Z 2026/11/11 10:03:35 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:03:35.390744706Z 2026/11/11 10:03:35 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
2026-11-11T10:03:35.390780864Z 2026/11/11 10:03:35 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting
2026-11-11T10:03:35.390785132Z 2026/11/11 10:03:35 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting
2026-11-11T10:03:35.391076372Z 2026/11/11 10:03:35 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:03:35.391086271Z 2026/11/11 10:03:35 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T10:03:45.389773899Z 2026/11/11 10:03:45 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:03:45.389882043Z 2026/11/11 10:03:45 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:03:50.390737836Z 2026/11/11 10:03:50 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T10:03:55.389956328Z 2026/11/11 10:03:55 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:03:55.390106231Z 2026/11/11 10:03:55 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:04:05.389588870Z 2026/11/11 10:04:05 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting
2026-11-11T10:04:05.389624548Z 2026/11/11 10:04:05 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting
2026-11-11T10:04:05.389777767Z 2026/11/11 10:04:05 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:04:05.389786073Z 2026/11/11 10:04:05 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
2026-11-11T10:04:05.389829194Z 2026/11/11 10:04:05 scheduler.go:450: [INFO] [scheduler] Running job: hold-report
2026-11-11T10:04:05.389835827Z 2026/11/11 10:04:05 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting
2026-11-11T10:04:05.389840044Z 2026/11/11 10:04:05 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T10:04:05.389844232Z 2026/11/11 10:04:05 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:04:05.389848080Z 2026/11/11 10:04:05 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T10:04:05.496334663Z 2026/11/11 10:04:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T10:04:05.496394095Z 2026/11/11 10:04:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting
2026-11-11T10:04:06.139729280Z 2026/11/11 10:04:06 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 643ms)
2026-11-11T10:04:15.389890014Z 2026/11/11 10:04:15 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:04:15.389939438Z 2026/11/11 10:04:15 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
2026-11-11T10:04:20.390815980Z 2026/11/11 10:04:20 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
2026-11-11T10:04:25.389069540Z 2026/11/11 10:04:25 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
2026-11-11T10:04:25.389738794Z 2026/11/11 10:04:25 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
RC=0
@@ -0,0 +1,196 @@
# guest-night demo-hp taken 2026-11-11T10:09:24Z
log first line: 2026-11-10T23:15:05.505129813Z
log last line: 2026-11-11T10:09:20.389434141Z
=== scheduler jobs (UTC): runs, first start, last start, last three "took" ===
agent-channel-health runs=654 first=2026-11-10T23:16:05 last=2026-11-11T10:09:05 [completed (took 664ms)] [completed (took 647ms)] [completed (took 669ms)]
backup-cache runs=131 first=2026-11-10T23:17:05 last=2026-11-11T10:07:05 [completed (took 674ms)] [completed (took 970ms)] [completed (took 655ms)]
conversion-copy-release runs=11 first=2026-11-10T23:17:05 last=2026-11-11T09:17:05 [completed (took 77ms)] [completed (took 342ms)] [completed (took 266ms)]
db-dump runs=1 first=2026-11-11T01:30:00 last=2026-11-11T01:30:00 [completed (took 2m33.216s)]
disk-health-check runs=11 first=2026-11-10T23:17:05 last=2026-11-11T09:17:05 [completed (took 962ms)] [completed (took 907ms)] [completed (took 876ms)]
docker-socket-users runs=131 first=2026-11-10T23:17:05 last=2026-11-11T10:07:05 [completed (took 305ms)] [completed (took 414ms)] [completed (took 337ms)]
fill-watch runs=1 first=2026-11-11T02:30:00 last=2026-11-11T02:30:00 [completed (took 0s)]
fill-watch-interval runs=66 first=2026-11-10T23:17:05 last=2026-11-11T10:07:05 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)]
geo-verify runs=2 first=2026-11-11T03:17:05 last=2026-11-11T09:17:05 [completed (took 0s)] [completed (took 0s)]
hold-report runs=654 first=2026-11-10T23:16:05 last=2026-11-11T10:09:05 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)]
hub-report runs=44 first=2026-11-10T23:17:05 last=2026-11-11T10:02:05 [completed (took 2.129s)] [completed (took 2.298s)] [completed (took 2.126s)]
metrics-prune runs=1 first=2026-11-11T03:00:00 last=2026-11-11T03:00:00 [completed (took 419ms)]
offbox-backup runs=1 first=2026-11-11T03:15:00 last=2026-11-11T03:15:00 [completed (took 3m8.568s)]
offsite-abandon-sweep runs=1 first=2026-11-11T04:10:00 last=2026-11-11T04:10:00 [completed (took 0s)]
offsite-credential-retry runs=131 first=2026-11-10T23:17:05 last=2026-11-11T10:07:05 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)]
offsite-integrity runs=1 first=2026-11-11T05:00:00 last=2026-11-11T05:00:00 [completed (took 0s)]
offsite-proof runs=1 first=2026-11-11T04:30:00 last=2026-11-11T04:30:00 [completed (took 5.282s)]
restore-copy-sweep runs=2 first=2026-11-11T03:17:05 last=2026-11-11T09:17:05 [completed (took 0s)] [completed (took 0s)]
selfupdate-check runs=2 first=2026-11-11T03:17:05 last=2026-11-11T09:17:05 [completed (took 139ms)] [completed (took 134ms)]
stack-scan runs=327 first=2026-11-10T23:17:05 last=2026-11-11T10:09:05 [completed (took 136ms)] [completed (took 274ms)] [completed (took 158ms)]
system-health runs=131 first=2026-11-10T23:17:05 last=2026-11-11T10:07:05 [completed (took 396ms)] [completed (took 511ms)] [completed (took 415ms)]
tier2-backup runs=1 first=2026-11-11T02:30:00 last=2026-11-11T02:30:00 [completed (took 38.939s)]
=== jobs that ran fewer than 30 times: every start and end (UTC) ===
2026/11/10 23:17:05 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/10 23:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/10 23:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 106ms)
2026/11/10 23:17:06 [INFO] [scheduler] Job disk-health-check completed (took 833ms)
2026/11/11 00:17:05 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 00:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 00:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 366ms)
2026/11/11 00:17:06 [INFO] [scheduler] Job disk-health-check completed (took 868ms)
2026/11/11 01:17:05 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 01:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 01:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 396ms)
2026/11/11 01:17:06 [INFO] [scheduler] Job disk-health-check completed (took 847ms)
2026/11/11 01:30:00 [INFO] [scheduler] Running job: db-dump
2026/11/11 01:32:33 [INFO] [scheduler] Job db-dump completed (took 2m33.216s)
2026/11/11 02:17:05 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 02:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 84ms)
2026/11/11 02:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 02:17:06 [INFO] [scheduler] Job disk-health-check completed (took 911ms)
2026/11/11 02:30:00 [INFO] [scheduler] Running job: tier2-backup
2026/11/11 02:30:00 [INFO] [scheduler] Running job: fill-watch
2026/11/11 02:30:00 [INFO] [scheduler] Job fill-watch completed (took 0s)
2026/11/11 02:30:38 [INFO] [scheduler] Job tier2-backup completed (took 38.939s)
2026/11/11 03:00:00 [INFO] [scheduler] Running job: metrics-prune
2026/11/11 03:00:00 [INFO] [scheduler] Job metrics-prune completed (took 419ms)
2026/11/11 03:15:00 [INFO] [scheduler] Running job: offbox-backup
2026/11/11 03:17:05 [INFO] [scheduler] Running job: geo-verify
2026/11/11 03:17:05 [INFO] [scheduler] Job geo-verify completed (took 0s)
2026/11/11 03:17:05 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 03:17:05 [INFO] [scheduler] Running job: selfupdate-check
2026/11/11 03:17:05 [INFO] [scheduler] Running job: restore-copy-sweep
2026/11/11 03:17:05 [INFO] [scheduler] Job restore-copy-sweep completed (took 0s)
2026/11/11 03:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 03:17:05 [INFO] [scheduler] Job selfupdate-check completed (took 139ms)
2026/11/11 03:17:06 [INFO] [scheduler] Job conversion-copy-release completed (took 834ms)
2026/11/11 03:17:06 [INFO] [scheduler] Job disk-health-check completed (took 1.069s)
2026/11/11 03:18:08 [INFO] [scheduler] Job offbox-backup completed (took 3m8.568s)
2026/11/11 04:10:00 [INFO] [scheduler] Running job: offsite-abandon-sweep
2026/11/11 04:10:00 [INFO] [scheduler] Job offsite-abandon-sweep completed (took 0s)
2026/11/11 04:17:05 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 04:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 04:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 285ms)
2026/11/11 04:17:06 [INFO] [scheduler] Job disk-health-check completed (took 839ms)
2026/11/11 04:30:00 [INFO] [scheduler] Running job: offsite-proof
2026/11/11 04:30:05 [INFO] [scheduler] Job offsite-proof completed (took 5.282s)
2026/11/11 05:00:00 [INFO] [scheduler] Running job: offsite-integrity
2026/11/11 05:00:00 [INFO] [scheduler] Job offsite-integrity completed (took 0s)
2026/11/11 05:17:05 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 05:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 62ms)
2026/11/11 05:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 05:17:06 [INFO] [scheduler] Job disk-health-check completed (took 816ms)
2026/11/11 06:17:05 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 06:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 06:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 323ms)
2026/11/11 06:17:06 [INFO] [scheduler] Job disk-health-check completed (took 853ms)
2026/11/11 07:17:05 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 07:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 77ms)
2026/11/11 07:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 07:17:06 [INFO] [scheduler] Job disk-health-check completed (took 962ms)
2026/11/11 08:17:05 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 08:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 08:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 342ms)
2026/11/11 08:17:06 [INFO] [scheduler] Job disk-health-check completed (took 907ms)
2026/11/11 09:17:05 [INFO] [scheduler] Running job: selfupdate-check
2026/11/11 09:17:05 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 09:17:05 [INFO] [scheduler] Running job: geo-verify
2026/11/11 09:17:05 [INFO] [scheduler] Job geo-verify completed (took 0s)
2026/11/11 09:17:05 [INFO] [scheduler] Running job: restore-copy-sweep
2026/11/11 09:17:05 [INFO] [scheduler] Job restore-copy-sweep completed (took 0s)
2026/11/11 09:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 09:17:05 [INFO] [scheduler] Job selfupdate-check completed (took 134ms)
2026/11/11 09:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 266ms)
2026/11/11 09:17:06 [INFO] [scheduler] Job disk-health-check completed (took 876ms)
=== night lines 19:00-08:00 UTC, INFO/WARN/ERROR, backup words (first 2 per message shape) ===
2026/11/10 23:17:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 23:17:05 backup.go:1252: [INFO] [backup] Found 21 DB dump files across drives
2026/11/10 23:17:05 healthcheck.go:176: [INFO] [monitor] Health check: status=ok
2026/11/10 23:17:06 dbdump.go:208: [INFO] [backup] Discovered 6 databases
2026/11/10 23:17:06 [INFO] [backup] Discovered app data: 12 apps
2026/11/10 23:17:06 disk_health.go:277: [INFO] [web] disk-health check complete: 2 disk(s) evaluated, 0 alert(s)
2026/11/10 23:17:06 backup.go:1323: [INFO] [backup] Backup status cache refreshed
2026/11/10 23:17:06 healthcheck.go:176: [INFO] [monitor] Health check: status=ok
2026/11/10 23:18:05 main.go:2430: [INFO] [deadapp] check alive: 960 scans since boot, 12 deployed app(s) evaluated, 0 currently down
2026/11/10 23:22:05 backup.go:1252: [INFO] [backup] Found 21 DB dump files across drives
2026/11/10 23:22:05 dbdump.go:208: [INFO] [backup] Discovered 6 databases
2026/11/10 23:22:05 [INFO] [backup] Discovered app data: 12 apps
2026/11/10 23:22:05 backup.go:1323: [INFO] [backup] Backup status cache refreshed
2026/11/10 23:27:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 23:28:05 main.go:2430: [INFO] [deadapp] check alive: 980 scans since boot, 12 deployed app(s) evaluated, 0 currently down
2026/11/11 00:17:06 disk_health.go:277: [INFO] [web] disk-health check complete: 2 disk(s) evaluated, 0 alert(s)
2026/11/11 01:30:00 backup.go:601: [INFO] [backup] Starting database dump run
2026/11/11 01:30:00 backup.go:640: [INFO] [backup] Discovered 6 database(s): romm-db(mariadb), paperless-postgres(postgres), kimai-db(mariadb), docmost-postgres(postgres), bookstack-db(mariadb), adventurelog-postgres(postgres)
2026/11/11 01:30:00 dbdump.go:429: [INFO] [backup] DB dump: romm-db → romm-mariadb.sql (136.3 KB, 459ms, 40 tables)
2026/11/11 01:30:01 dbdump.go:429: [INFO] [backup] DB dump: paperless-postgres → paperless-ngx-postgres.sql (733.5 KB, 575ms, 75 tables)
2026/11/11 01:30:01 dbdump.go:429: [INFO] [backup] DB dump: kimai-db → kimai-mariadb.sql (51.3 KB, 346ms, 35 tables)
2026/11/11 01:30:02 dbdump.go:429: [INFO] [backup] DB dump: docmost-postgres → docmost-postgres.sql (164.7 KB, 563ms, 52 tables)
2026/11/11 01:30:02 dbdump.go:429: [INFO] [backup] DB dump: bookstack-db → bookstack-mariadb.sql (57.8 KB, 377ms, 42 tables)
2026/11/11 01:30:03 dbdump.go:429: [INFO] [backup] DB dump: adventurelog-postgres → adventurelog-postgres.sql (12.5 MB, 922ms, 51 tables)
2026/11/11 01:30:03 backup.go:1044: [INFO] [backup] Stopping adventurelog for safe volume dump
2026/11/11 01:30:05 busymark.go:81: [INFO] [busymark] busy: a backup, a restore or an off-site run is running — an operator reboot of the box is refused until it ends
2026/11/11 01:30:08 backup.go:941: [INFO] [backup] Volume dump: adventurelog/adventurelog_adventurelog_media → 90.1 MB
2026/11/11 01:30:14 backup.go:941: [INFO] [backup] Volume dump: adventurelog/adventurelog_adventurelog_postgres_data → 273.4 MB
2026/11/11 01:30:14 backup.go:1060: [INFO] [backup] Restarting adventurelog after volume dump
2026/11/11 01:30:20 backup.go:1044: [INFO] [backup] Stopping bookstack for safe volume dump
2026/11/11 01:30:26 backup.go:941: [INFO] [backup] Volume dump: bookstack/bookstack_bookstack_config → 6.4 MB
2026/11/11 01:30:28 backup.go:941: [INFO] [backup] Volume dump: bookstack/bookstack_bookstack_db_data → 153.6 MB
2026/11/11 01:30:28 backup.go:1060: [INFO] [backup] Restarting bookstack after volume dump
2026/11/11 01:30:35 backup.go:1044: [INFO] [backup] Stopping calibre-web for safe volume dump
2026/11/11 01:30:41 backup.go:941: [INFO] [backup] Volume dump: calibre-web/calibre-web_calibre_web_config → 921.5 KB
2026/11/11 01:30:41 backup.go:1060: [INFO] [backup] Restarting calibre-web after volume dump
2026/11/11 01:30:42 backup.go:1044: [INFO] [backup] Stopping docmost for safe volume dump
2026/11/11 01:30:46 backup.go:941: [INFO] [backup] Volume dump: docmost/docmost_docmost_storage → 2.5 KB
2026/11/11 01:30:49 backup.go:941: [INFO] [backup] Volume dump: docmost/docmost_docmost_postgres_data → 66.8 MB
2026/11/11 01:30:50 backup.go:1060: [INFO] [backup] Restarting docmost after volume dump
2026/11/11 01:31:02 backup.go:1044: [INFO] [backup] Stopping home-assistant for safe volume dump
2026/11/11 01:31:13 backup.go:941: [INFO] [backup] Volume dump: home-assistant/home-assistant_homeassistant_config → 17.8 MB
2026/11/11 01:31:13 backup.go:1060: [INFO] [backup] Restarting home-assistant after volume dump
2026/11/11 01:31:13 backup.go:1044: [INFO] [backup] Stopping kimai for safe volume dump
2026/11/11 01:31:16 backup.go:941: [INFO] [backup] Volume dump: kimai/kimai_kimai_var → 18.3 MB
2026/11/11 01:31:20 backup.go:941: [INFO] [backup] Volume dump: kimai/kimai_kimai_db_data → 154.5 MB
2026/11/11 01:31:20 backup.go:1060: [INFO] [backup] Restarting kimai after volume dump
2026/11/11 01:31:26 backup.go:1044: [INFO] [backup] Stopping opengist for safe volume dump
2026/11/11 01:31:28 backup.go:941: [INFO] [backup] Volume dump: opengist/opengist_opengist_data → 181.5 KB
2026/11/11 01:31:28 backup.go:1060: [INFO] [backup] Restarting opengist after volume dump
2026/11/11 01:31:28 backup.go:1044: [INFO] [backup] Stopping paperless-ngx for safe volume dump
2026/11/11 01:31:38 backup.go:941: [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_data → 21.0 MB
2026/11/11 01:31:40 backup.go:941: [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_postgres_data → 70.2 MB
2026/11/11 01:31:41 backup.go:1060: [INFO] [backup] Restarting paperless-ngx after volume dump
2026/11/11 01:31:52 backup.go:1044: [INFO] [backup] Stopping plex for safe volume dump
2026/11/11 01:32:03 backup.go:941: [INFO] [backup] Volume dump: plex/plex_plex_config → 9.9 MB
2026/11/11 01:32:03 backup.go:941: [INFO] [backup] Volume dump: plex/plex_plex_transcode → 3.0 KB
2026/11/11 01:32:03 backup.go:1060: [INFO] [backup] Restarting plex after volume dump
2026/11/11 01:32:04 backup.go:1044: [INFO] [backup] Stopping privatebin for safe volume dump
2026/11/11 01:32:04 backup.go:941: [INFO] [backup] Volume dump: privatebin/privatebin_privatebin_data → 2.0 MB
2026/11/11 01:32:04 backup.go:1060: [INFO] [backup] Restarting privatebin after volume dump
2026/11/11 01:32:07 recovery_unit.go:297: [INFO] [backup] Recovery unit captured for adventurelog → /mnt/sys_drive/felhom-data/backups/primary/adventurelog (images=3, secrets-referenced=2, data_keys=1, portable-carried=2/2, with
2026/11/11 01:32:07 backup.go:1044: [INFO] [backup] Stopping romm for safe volume dump
2026/11/11 01:32:07 recovery_unit.go:297: [INFO] [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1
2026/11/11 01:32:17 backup.go:941: [INFO] [backup] Volume dump: romm/romm_romm_redis_data → 61.6 MB
2026/11/11 01:32:17 backup.go:941: [INFO] [backup] Volume dump: romm/romm_romm_config → 2.5 KB
2026/11/11 01:32:20 backup.go:1060: [INFO] [backup] Restarting romm after volume dump
2026/11/11 01:32:32 backup.go:743: [INFO] [backup] App-data backup completed: 6 databases (13.6 MB total), 11 volume dump(s) (2m32.713s)
2026/11/11 01:32:33 notifier.go:357: [INFO] Event pushed: db_dump_completed (info) [hu-only] — Adatbázis mentés elkészült
2026/11/11 01:32:35 busymark.go:81: [INFO] [busymark] idle again (was: a backup, a restore or an off-site run is running) — an operator reboot of the box is no longer refused for it
2026/11/11 02:30:08 tier2.go:525: [INFO] [backup] Tier 2 copied adventurelog → /mnt/felhom-drives/hdd_1/backups/secondary/adventurelog (404.1 MB, 0 leg(s), 7s)
2026/11/11 02:30:10 tier2.go:525: [INFO] [backup] Tier 2 copied bentopdf → /mnt/felhom-drives/hdd_1/backups/secondary/bentopdf (12.8 KB, 0 leg(s), 0s)
2026/11/11 02:30:14 tier2.go:525: [INFO] [backup] Tier 2 copied bookstack → /mnt/felhom-drives/hdd_1/backups/secondary/bookstack (160.3 MB, 0 leg(s), 3s)
2026/11/11 02:30:16 tier2.go:525: [INFO] [backup] Tier 2 copied calibre-web → /mnt/sys_drive/felhom-data/backups/secondary/calibre-web (5.6 MB, 1 leg(s), 1s) [SSD: state-only]
2026/11/11 02:30:20 tier2.go:525: [INFO] [backup] Tier 2 copied docmost → /mnt/felhom-drives/hdd_1/backups/secondary/docmost (120.1 MB, 0 leg(s), 2s)
2026/11/11 02:30:21 tier2.go:525: [INFO] [backup] Tier 2 copied home-assistant → /mnt/felhom-drives/hdd_1/backups/secondary/home-assistant (17.8 MB, 0 leg(s), 0s)
2026/11/11 02:30:26 tier2.go:525: [INFO] [backup] Tier 2 copied kimai → /mnt/felhom-drives/hdd_1/backups/secondary/kimai (173.0 MB, 0 leg(s), 4s)
2026/11/11 02:30:28 tier2.go:525: [INFO] [backup] Tier 2 copied opengist → /mnt/felhom-drives/hdd_1/backups/secondary/opengist (189.7 KB, 0 leg(s), 0s)
2026/11/11 02:30:31 tier2.go:525: [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sys_drive/felhom-data/backups/secondary/paperless-ngx (155.1 MB, 1 leg(s), 2s) [SSD: state-only]
2026/11/11 02:30:33 tier2.go:525: [INFO] [backup] Tier 2 copied plex → /mnt/sys_drive/felhom-data/backups/secondary/plex (9.9 MB, 0 leg(s), 0s) [SSD: state-only]
2026/11/11 02:30:35 tier2.go:525: [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/hdd_1/backups/secondary/privatebin (2.0 MB, 0 leg(s), 0s)
2026/11/11 02:30:38 tier2.go:525: [INFO] [backup] Tier 2 copied romm → /mnt/sys_drive/felhom-data/backups/secondary/romm (217.2 MB, 0 leg(s), 2s) [SSD: state-only]
2026/11/11 02:30:38 tier2.go:587: [INFO] [backup] Tier 2 run complete: 12 app(s) processed (incl. volume-only — F6)
2026/11/11 03:00:00 main.go:1759: [INFO] Pruned 31667 old metric rows
2026/11/11 03:15:00 offbox.go:1094: [INFO] [offbox] backup run started (15 app(s) toggled)
2026/11/11 03:15:00 backup.go:601: [INFO] [backup] Starting database dump run
2026/11/11 03:15:00 backup.go:640: [INFO] [backup] Discovered 6 database(s): romm-db(mariadb), paperless-postgres(postgres), kimai-db(mariadb), docmost-postgres(postgres), bookstack-db(mariadb), adventurelog-postgres(postgres)
2026/11/11 03:15:00 dbdump.go:429: [INFO] [backup] DB dump: romm-db → romm-mariadb.sql (136.3 KB, 360ms, 40 tables)
2026/11/11 03:15:00 dbdump.go:429: [INFO] [backup] DB dump: paperless-postgres → paperless-ngx-postgres.sql (736.3 KB, 356ms, 75 tables)
2026/11/11 03:15:01 dbdump.go:429: [INFO] [backup] DB dump: kimai-db → kimai-mariadb.sql (51.3 KB, 266ms, 35 tables)
awk: write failure (Broken pipe)
awk: close failed on file "/dev/stdout" (Broken pipe)
RC=0
@@ -0,0 +1,682 @@
# host-clock demo-hp taken 2026-11-11T10:03:57Z from DooPlex
=== in-flight ===
checked
=== clock ===
utc 2026-11-11T10:03:57.824658172Z
local 2026-11-11T11:03:57 CET +0100
/usr/share/zoneinfo/Europe/Budapest
Timezone=Europe/Budapest
LocalRTC=no
CanNTP=yes
NTP=yes
NTPSynchronized=yes
TimeUSec=Wed 2026-11-11 11:03:57 CET
RTCTimeUSec=Wed 2026-11-11 11:03:58 CET
Local time: Wed 2026-11-11 11:03:57 CET
Universal time: Wed 2026-11-11 10:03:57 UTC
RTC time: Wed 2026-11-11 10:03:58
Time zone: Europe/Budapest (CET, +0100)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
=== who keeps the time ===
chrony: active
enabled: enabled
chronyd: active
enabled: alias
systemd-timesyncd: inactive
enabled: not-found
ntp: inactive
enabled: not-found
ntpsec: inactive
enabled: not-found
openntpd: inactive
enabled: not-found
Reference ID : 25DDD118 (37.221.209.24)
Stratum : 3
Ref time (UTC) : Wed Nov 11 09:59:51 2026
System time : 0.000200578 seconds fast of NTP time
Last offset : +0.000003393 seconds
RMS offset : 0.000097092 seconds
Frequency : 12.934 ppm slow
Residual freq : +0.000 ppm
Skew : 0.017 ppm
Root delay : 0.016422609 seconds
Root dispersion : 0.002579133 seconds
Update interval : 1036.9 seconds
Leap status : Normal
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^- 62.112.195.56 2 10 377 301 -2821us[-2817us] +/- 33ms
^* 37.221.209.24 2 10 377 246 -3262us[-3259us] +/- 14ms
^+ 81.0.124.253 3 10 377 76 -2267us[-2267us] +/- 25ms
^+ 195.111.92.55 2 10 377 741 -2832us[-2828us] +/- 52ms
Failed to query server: The name org.freedesktop.timesync1 was not provided by any .service files
pool 2.debian.pool.ntp.org iburst
sourcedir /run/chrony-dhcp
sourcedir /etc/chrony/sources.d
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/chrony.drift
ntsdumpdir /var/lib/chrony
logdir /var/log/chrony
maxupdateskew 100.0
rtcsync
makestep 1 3
leapseclist /usr/share/zoneinfo/leap-seconds.list
confdir /etc/chrony/conf.d
/etc/chrony/conf.d:
README
/etc/chrony/sources.d:
README
=== start with no time ===
ls: cannot access '/var/lib/systemd/timesync/clock': No such file or directory
/var/lib/chrony/:
total 4
-rw-r--r-- 1 _chrony _chrony 42 2026-11-11 10:59:52.023776089 +0100 chrony.drift
2026-11-11 11:03:58.342660+01:00
1794391439
1794391438
=== boots ===
2026-11-10 15:42:49
-5 182af91122a941ec97ac7ee247a4d470 Sun 2026-10-04 21:42:08 CEST Sun 2026-10-04 21:43:33 CEST
-4 bb68aee4240244718d8a7f39d999a3a8 Wed 2026-11-04 21:06:10 CET Tue 2026-11-10 02:53:28 CET
-3 b6e0edea2b5647dfa36f5bb3825adc3a Tue 2026-11-10 02:53:57 CET Tue 2026-11-10 13:56:35 CET
-2 d1167d9431bf459abf2a569e8997a569 Tue 2026-11-10 13:57:04 CET Tue 2026-11-10 15:32:31 CET
-1 e2cc9e3e60f14bde8949f01d7ba35ccf Tue 2026-11-10 15:32:59 CET Tue 2026-11-10 15:42:26 CET
0 bed106f79e804eb1ace314a26da7f049 Tue 2026-11-10 15:42:54 CET Wed 2026-11-11 11:03:57 CET
=== timers ===
NEXT LEFT LAST PASSED UNIT ACTIVATES
Wed 2026-11-11 11:04:28 CET 29s Wed 2026-11-11 11:03:28 CET 30s ago felhom-mgmt-watchdog.timer felhom-mgmt-watchdog.service
Wed 2026-11-11 11:59:19 CET 55min Wed 2026-11-11 10:59:19 CET 4min 39s ago felhom-crash-guard-check.timer felhom-crash-guard-check.service
Wed 2026-11-11 15:29:36 CET 4h 25min Tue 2026-11-10 23:19:19 CET 11h ago apt-daily.timer apt-daily.service
Wed 2026-11-11 15:57:49 CET 4h 53min Tue 2026-11-10 15:57:49 CET 19h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Thu 2026-11-12 00:00:00 CET 12h Wed 2026-11-11 00:00:19 CET 11h ago dpkg-db-backup.timer dpkg-db-backup.service
Thu 2026-11-12 00:11:48 CET 13h Wed 2026-11-11 00:16:19 CET 10h ago logrotate.timer logrotate.service
Thu 2026-11-12 01:46:13 CET 14h Wed 2026-11-11 02:22:58 CET 8h ago pve-daily-update.timer pve-daily-update.service
Thu 2026-11-12 06:13:04 CET 19h Wed 2026-11-11 06:44:19 CET 4h 19min ago apt-daily-upgrade.timer apt-daily-upgrade.service
Thu 2026-11-12 07:37:24 CET 20h Wed 2026-11-11 02:10:19 CET 8h ago man-db.timer man-db.service
Sun 2026-11-15 03:10:25 CET 3 days Sun 2026-11-08 03:10:49 CET - xfs_scrub_all.timer xfs_scrub_all.service
Sun 2026-11-15 03:10:43 CET 3 days Sun 2026-11-08 03:10:49 CET - e2scrub_all.timer e2scrub_all.service
Mon 2026-11-16 00:02:01 CET 4 days Mon 2026-11-09 01:25:39 CET - fstrim.timer fstrim.service
12 timers listed.
=== timer units (all) ===
--- apt-daily-upgrade.timer [enabled/active]
OnCalendar=*-*-* 6:00
RandomizedDelaySec=60m
Persistent=true
Unit=apt-daily-upgrade.service NextElapseUSecRealtime=Thu 2026-11-12 06:13:04 CET LastTriggerUSec=Wed 2026-11-11 06:44:19 CET
--- apt-daily.timer [enabled/active]
OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h
Persistent=true
Unit=apt-daily.service NextElapseUSecRealtime=Wed 2026-11-11 15:29:36 CET LastTriggerUSec=Tue 2026-11-10 23:19:19 CET
--- apt-listchanges.timer [disabled/inactive]
OnCalendar=hourly
Unit=apt-listchanges.service NextElapseUSecRealtime= LastTriggerUSec=
--- chrony-dnssrv@.timer [disabled/]
OnActiveSec=0
OnUnitInactiveSec=1h
--- dpkg-db-backup.timer [enabled/active]
OnCalendar=daily
Persistent=true
Unit=dpkg-db-backup.service NextElapseUSecRealtime=Thu 2026-11-12 00:00:00 CET LastTriggerUSec=Wed 2026-11-11 00:00:19 CET
--- e2scrub_all.timer [enabled/active]
OnCalendar=Sun *-*-* 03:10:00
RandomizedDelaySec=60
Persistent=true
Unit=e2scrub_all.service NextElapseUSecRealtime=Sun 2026-11-15 03:10:43 CET LastTriggerUSec=Sun 2026-11-08 03:10:49 CET
--- felhom-crash-guard-check.timer [enabled/active]
OnBootSec=15min
OnUnitActiveSec=1h
Unit=felhom-crash-guard-check.service NextElapseUSecRealtime= LastTriggerUSec=Wed 2026-11-11 10:59:19 CET
--- felhom-mgmt-watchdog.timer [enabled/active]
OnBootSec=30s
OnUnitActiveSec=60s
AccuracySec=5s
Persistent=true
Unit=felhom-mgmt-watchdog.service
Unit=felhom-mgmt-watchdog.service NextElapseUSecRealtime= LastTriggerUSec=Wed 2026-11-11 11:03:28 CET
--- fstrim.timer [enabled/active]
OnCalendar=weekly
AccuracySec=1h
Persistent=true
RandomizedDelaySec=100min
Unit=fstrim.service NextElapseUSecRealtime=Mon 2026-11-16 00:02:01 CET LastTriggerUSec=Mon 2026-11-09 01:25:39 CET
--- logrotate.timer [enabled/active]
OnCalendar=daily
RandomizedDelaySec=1h
Persistent=true
Unit=logrotate.service NextElapseUSecRealtime=Thu 2026-11-12 00:11:48 CET LastTriggerUSec=Wed 2026-11-11 00:16:19 CET
--- man-db.timer [enabled/active]
OnCalendar=daily
RandomizedDelaySec=12h
Persistent=true
Unit=man-db.service NextElapseUSecRealtime=Thu 2026-11-12 07:37:24 CET LastTriggerUSec=Wed 2026-11-11 02:10:19 CET
--- proxmox-firewall.timer [disabled/inactive]
OnBootSec=1s
OnUnitInactiveSec=5s
Unit=proxmox-firewall.service
Unit=proxmox-firewall.service NextElapseUSecRealtime= LastTriggerUSec=
--- pve-daily-update.timer [enabled/active]
OnCalendar=*-*-* 1:00
RandomizedDelaySec=5h
Persistent=true
Unit=pve-daily-update.service NextElapseUSecRealtime=Thu 2026-11-12 01:46:13 CET LastTriggerUSec=Wed 2026-11-11 02:22:58 CET
--- systemd-tmpfiles-clean.timer [static/active]
OnBootSec=15min
OnUnitActiveSec=1d
Unit=systemd-tmpfiles-clean.service NextElapseUSecRealtime= LastTriggerUSec=Tue 2026-11-10 15:57:49 CET
--- xfs_scrub_all.timer [enabled/active]
OnCalendar=Sun *-*-* 03:10:00
RandomizedDelaySec=60
Persistent=true
Unit=xfs_scrub_all.service NextElapseUSecRealtime=Sun 2026-11-15 03:10:25 CET LastTriggerUSec=Sun 2026-11-08 03:10:49 CET
--- zfs-scrub-monthly@.timer [disabled/]
OnCalendar=monthly
Persistent=true
RandomizedDelaySec=1h
Unit=zfs-scrub@%i.service
--- zfs-scrub-weekly@.timer [disabled/]
OnCalendar=weekly
Persistent=true
RandomizedDelaySec=1h
Unit=zfs-scrub@%i.service
--- zfs-trim-monthly@.timer [disabled/]
OnCalendar=monthly
Persistent=true
RandomizedDelaySec=1h
Unit=zfs-trim@%i.service
--- zfs-trim-weekly@.timer [disabled/]
OnCalendar=weekly
Persistent=true
RandomizedDelaySec=1h
Unit=zfs-trim@%i.service
=== service limits of felhom + backup timers ===
--- apt-daily-upgrade.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 06:44:19 CET
ExecMainExitTimestamp=Wed 2026-11-11 06:44:19 CET
ExecMainStatus=0
ExecStart={ path=/usr/lib/apt/apt.systemd.daily ; argv[]=/usr/lib/apt/apt.systemd.daily install ; ignore_errors=no ; start_time=[Wed 2026-11-11 06:44:19 CET] ; stop_time=[Wed 2026-11-11 06:44:19 CET] ; pid=3061802 ; code=exited ; status=0 }
--- apt-daily.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Tue 2026-11-10 23:19:19 CET
ExecMainExitTimestamp=Tue 2026-11-10 23:19:19 CET
ExecMainStatus=0
ExecStart={ path=/usr/lib/apt/apt.systemd.daily ; argv[]=/usr/lib/apt/apt.systemd.daily update ; ignore_errors=no ; start_time=[Tue 2026-11-10 23:19:19 CET] ; stop_time=[Tue 2026-11-10 23:19:19 CET] ; pid=1543951 ; code=exited ; status=0 }
--- apt-listchanges.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=
ExecMainExitTimestamp=
ExecMainStatus=0
ExecStart={ path=python3 ; argv[]=python3 -m apt_listchanges.populate_database --profile apt ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
--- dpkg-db-backup.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 00:00:19 CET
ExecMainExitTimestamp=Wed 2026-11-11 00:00:19 CET
ExecMainStatus=0
ExecStart={ path=/usr/libexec/dpkg/dpkg-db-backup ; argv[]=/usr/libexec/dpkg/dpkg-db-backup ; ignore_errors=no ; start_time=[Wed 2026-11-11 00:00:19 CET] ; stop_time=[Wed 2026-11-11 00:00:19 CET] ; pid=1679989 ; code=exited ; status=0 }
--- e2scrub_all.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=
ExecMainExitTimestamp=
ExecMainStatus=0
ExecStart={ path=/sbin/e2scrub_all ; argv[]=/sbin/e2scrub_all ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
--- felhom-crash-guard-check.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 10:59:19 CET
ExecMainExitTimestamp=Wed 2026-11-11 10:59:19 CET
ExecMainStatus=0
ExecStart={ path=/usr/local/sbin/felhom-crash-guard ; argv[]=/usr/local/sbin/felhom-crash-guard check ; ignore_errors=no ; start_time=[Wed 2026-11-11 10:59:19 CET] ; stop_time=[Wed 2026-11-11 10:59:19 CET] ; pid=3911370 ; code=exited ; status=0 }
--- felhom-mgmt-watchdog.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 11:03:28 CET
ExecMainExitTimestamp=Wed 2026-11-11 11:03:28 CET
ExecMainStatus=0
ExecStart={ path=/usr/local/sbin/felhom-mgmt-watchdog ; argv[]=/usr/local/sbin/felhom-mgmt-watchdog ; ignore_errors=no ; start_time=[Wed 2026-11-11 11:03:28 CET] ; stop_time=[Wed 2026-11-11 11:03:28 CET] ; pid=3925600 ; code=exited ; status=0 }
--- fstrim.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=
ExecMainExitTimestamp=
ExecMainStatus=0
ExecStart={ path=/sbin/fstrim ; argv[]=/sbin/fstrim --listed-in /etc/fstab:/proc/self/mountinfo --verbose --quiet-unsupported ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
--- logrotate.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 00:16:19 CET
ExecMainExitTimestamp=Wed 2026-11-11 00:16:21 CET
ExecMainStatus=0
ExecStart={ path=/usr/sbin/logrotate ; argv[]=/usr/sbin/logrotate /etc/logrotate.conf ; ignore_errors=no ; start_time=[Wed 2026-11-11 00:16:19 CET] ; stop_time=[Wed 2026-11-11 00:16:21 CET] ; pid=1733122 ; code=exited ; status=0 }
--- man-db.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 02:10:19 CET
ExecMainExitTimestamp=Wed 2026-11-11 02:10:19 CET
ExecMainStatus=0
ExecStart={ path=/usr/bin/install ; argv[]=/usr/bin/install -d -o man -g man -m 0755 /var/cache/man ; ignore_errors=no ; start_time=[Wed 2026-11-11 02:10:19 CET] ; stop_time=[Wed 2026-11-11 02:10:19 CET] ; pid=2113150 ; code=exited ; status=0 }
ExecStart={ path=/usr/bin/mandb ; argv[]=/usr/bin/mandb --quiet ; ignore_errors=no ; start_time=[Wed 2026-11-11 02:10:19 CET] ; stop_time=[Wed 2026-11-11 02:10:19 CET] ; pid=2113157 ; code=exited ; status=0 }
--- pve-daily-update.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 02:22:58 CET
ExecMainExitTimestamp=Wed 2026-11-11 02:23:04 CET
ExecMainStatus=0
ExecStart={ path=/usr/bin/pveupdate ; argv[]=/usr/bin/pveupdate ; ignore_errors=no ; start_time=[Wed 2026-11-11 02:22:58 CET] ; stop_time=[Wed 2026-11-11 02:23:04 CET] ; pid=2155738 ; code=exited ; status=0 }
=== felhom units that are not timers ===
felhom-agent-rollback.service loaded inactive dead
felhom-agent.service loaded active running
felhom-crash-guard-check.service loaded inactive dead
felhom-crash-guard.service loaded active exited
felhom-mgmt-watchdog.service loaded inactive dead
felhom-oob-nft.service loaded active exited
felhom-shared-parent.service loaded active exited
felhom-sshd.service loaded active running
felhom-crash-guard-check.timer loaded active waiting
felhom-mgmt-watchdog.timer loaded active waiting
--- felhom-agent-rollback.service
Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=infinity NRestarts=0 ActiveEnterTimestamp=
--- felhom-agent.service
Restart=on-failure RestartUSec=5s TimeoutStartUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:58:17 CET
--- felhom-bootstrap.service
Restart=on-failure RestartUSec=30s TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=infinity NRestarts=0 ActiveEnterTimestamp=
--- felhom-crash-guard-check.service
Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=
--- felhom-crash-guard.service
Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:42:59 CET
--- felhom-mgmt-watchdog.service
Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=
--- felhom-oob-nft.service
Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:42:59 CET
--- felhom-shared-parent.service
Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:42:59 CET
--- felhom-sshd.service
Restart=on-failure RestartUSec=100ms TimeoutStartUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:43:00 CET
=== cron ===
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; }
47 6 * * 7 root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.weekly; }
52 6 1 * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.monthly; }
--- /etc/cron.d/e2scrub_all
30 3 * * 0 root test -e /run/systemd/system || SERVICE_MODE=1 /usr/libexec/e2fsprogs/e2scrub_all_cron
10 3 * * * root test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r
--- /etc/cron.d/vzdump
PATH="/usr/sbin:/usr/bin:/sbin:/bin"
--- /etc/cron.d/zfsutils-linux
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
24 0 1-7 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/trim ]; then /usr/lib/zfs-linux/trim; fi
24 0 8-14 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs-linux/scrub; fi
--- cron.hourly:
--- cron.daily: apt-compat dpkg logrotate man-db
--- cron.weekly: man-db
--- cron.monthly:
--- user crontabs:
=== at ===
no at
=== pve jobs ===
--- vzdump.cron
PATH="/usr/sbin:/usr/bin:/sbin:/bin"
--- vzdump.conf
=== pve tasks (start end type id status), last 14 days ===
awk: line 2: function strtonum never defined
awk: line 2: function strtonum never defined
=== runs of timer services, last 10 days (start/finish lines) ===
--- apt-daily-upgrade.service
2026-11-09T06:46:01+01:00 demo-hp systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities...
2026-11-09T06:46:01+01:00 demo-hp systemd[1]: apt-daily-upgrade.service: Deactivated successfully.
2026-11-09T06:46:01+01:00 demo-hp systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities.
2026-11-10T06:55:09+01:00 demo-hp systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities...
2026-11-10T06:55:10+01:00 demo-hp systemd[1]: apt-daily-upgrade.service: Deactivated successfully.
2026-11-10T06:55:10+01:00 demo-hp systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities.
2026-11-11T06:44:19+01:00 demo-hp systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities...
2026-11-11T06:44:19+01:00 demo-hp systemd[1]: apt-daily-upgrade.service: Deactivated successfully.
2026-11-11T06:44:19+01:00 demo-hp systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities.
--- apt-daily.service
2026-11-09T07:55:02+01:00 demo-hp systemd[1]: Starting apt-daily.service - Daily apt download activities...
2026-11-09T07:55:03+01:00 demo-hp systemd[1]: apt-daily.service: Deactivated successfully.
2026-11-09T07:55:03+01:00 demo-hp systemd[1]: Finished apt-daily.service - Daily apt download activities.
2026-11-10T04:06:22+01:00 demo-hp systemd[1]: Starting apt-daily.service - Daily apt download activities...
2026-11-10T04:06:23+01:00 demo-hp systemd[1]: apt-daily.service: Deactivated successfully.
2026-11-10T04:06:23+01:00 demo-hp systemd[1]: Finished apt-daily.service - Daily apt download activities.
2026-11-10T23:19:19+01:00 demo-hp systemd[1]: Starting apt-daily.service - Daily apt download activities...
2026-11-10T23:19:19+01:00 demo-hp systemd[1]: apt-daily.service: Deactivated successfully.
2026-11-10T23:19:19+01:00 demo-hp systemd[1]: Finished apt-daily.service - Daily apt download activities.
--- felhom-crash-guard-check.service
2026-11-11T08:59:19+01:00 demo-hp systemd[1]: Starting felhom-crash-guard-check.service - Felhom crash guard re-arm check...
2026-11-11T08:59:19+01:00 demo-hp systemd[1]: felhom-crash-guard-check.service: Deactivated successfully.
2026-11-11T08:59:19+01:00 demo-hp systemd[1]: Finished felhom-crash-guard-check.service - Felhom crash guard re-arm check.
2026-11-11T09:59:19+01:00 demo-hp systemd[1]: Starting felhom-crash-guard-check.service - Felhom crash guard re-arm check...
2026-11-11T09:59:19+01:00 demo-hp systemd[1]: felhom-crash-guard-check.service: Deactivated successfully.
2026-11-11T09:59:19+01:00 demo-hp systemd[1]: Finished felhom-crash-guard-check.service - Felhom crash guard re-arm check.
2026-11-11T10:59:19+01:00 demo-hp systemd[1]: Starting felhom-crash-guard-check.service - Felhom crash guard re-arm check...
2026-11-11T10:59:19+01:00 demo-hp systemd[1]: felhom-crash-guard-check.service: Deactivated successfully.
2026-11-11T10:59:19+01:00 demo-hp systemd[1]: Finished felhom-crash-guard-check.service - Felhom crash guard re-arm check.
--- felhom-mgmt-watchdog.service
2026-11-11T11:01:20+01:00 demo-hp systemd[1]: Starting felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)...
2026-11-11T11:01:20+01:00 demo-hp systemd[1]: felhom-mgmt-watchdog.service: Deactivated successfully.
2026-11-11T11:01:20+01:00 demo-hp systemd[1]: Finished felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent).
2026-11-11T11:02:24+01:00 demo-hp systemd[1]: Starting felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)...
2026-11-11T11:02:24+01:00 demo-hp systemd[1]: felhom-mgmt-watchdog.service: Deactivated successfully.
2026-11-11T11:02:24+01:00 demo-hp systemd[1]: Finished felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent).
2026-11-11T11:03:28+01:00 demo-hp systemd[1]: Starting felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)...
2026-11-11T11:03:28+01:00 demo-hp systemd[1]: felhom-mgmt-watchdog.service: Deactivated successfully.
2026-11-11T11:03:28+01:00 demo-hp systemd[1]: Finished felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent).
--- pve-daily-update.service
2026-11-09T03:29:48+01:00 demo-hp systemd[1]: pve-daily-update.service: Consumed 5.005s CPU time, 235.1M memory peak.
2026-11-10T03:05:09+01:00 demo-hp systemd[1]: Starting pve-daily-update.service - Daily PVE download activities...
2026-11-10T03:05:15+01:00 demo-hp systemd[1]: pve-daily-update.service: Deactivated successfully.
2026-11-10T03:05:15+01:00 demo-hp systemd[1]: Finished pve-daily-update.service - Daily PVE download activities.
2026-11-10T03:05:15+01:00 demo-hp systemd[1]: pve-daily-update.service: Consumed 5.281s CPU time, 295.5M memory peak.
2026-11-11T02:22:58+01:00 demo-hp systemd[1]: Starting pve-daily-update.service - Daily PVE download activities...
2026-11-11T02:23:04+01:00 demo-hp systemd[1]: pve-daily-update.service: Deactivated successfully.
2026-11-11T02:23:04+01:00 demo-hp systemd[1]: Finished pve-daily-update.service - Daily PVE download activities.
2026-11-11T02:23:04+01:00 demo-hp systemd[1]: pve-daily-update.service: Consumed 5.171s CPU time, 293.5M memory peak.
=== agent log: schedule words, last 3 days (tail 260) ===
2026-11-10T03:24:11+01:00 demo-hp sudo[112883]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T03:39:11+01:00 demo-hp sudo[165130]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T03:54:12+01:00 demo-hp sudo[215104]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T04:09:11+01:00 demo-hp sudo[265287]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T04:24:11+01:00 demo-hp sudo[326129]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T04:39:11+01:00 demo-hp sudo[376165]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T04:54:11+01:00 demo-hp sudo[426145]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T05:09:11+01:00 demo-hp sudo[476293]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T05:24:11+01:00 demo-hp sudo[526258]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T05:39:11+01:00 demo-hp sudo[576428]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T05:54:11+01:00 demo-hp sudo[626334]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T06:09:11+01:00 demo-hp sudo[676490]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T06:24:11+01:00 demo-hp sudo[726421]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T06:39:12+01:00 demo-hp sudo[776549]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T06:54:11+01:00 demo-hp sudo[826483]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T07:09:11+01:00 demo-hp sudo[876732]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T07:24:11+01:00 demo-hp sudo[926734]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T07:39:11+01:00 demo-hp sudo[976930]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T07:54:11+01:00 demo-hp sudo[1027027]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T08:09:11+01:00 demo-hp sudo[1077199]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T08:24:11+01:00 demo-hp sudo[1127154]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T08:39:11+01:00 demo-hp sudo[1180619]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T08:54:12+01:00 demo-hp sudo[1231276]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T08:55:21+01:00 demo-hp sudo[1236551]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T08:56:33+01:00 demo-hp felhom-agent[1264]: time=2026-11-10T08:56:33.425+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-11-10T09:09:11+01:00 demo-hp sudo[1281634]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T09:24:10+01:00 demo-hp felhom-agent[1264]: time=2026-11-10T09:24:10.329+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-05T20:50:35Z) is already proven; local: no set
2026-11-10T09:24:11+01:00 demo-hp sudo[1331688]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T09:39:11+01:00 demo-hp sudo[1381939]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T09:54:11+01:00 demo-hp sudo[1431884]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T10:09:11+01:00 demo-hp sudo[1481961]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T10:24:11+01:00 demo-hp sudo[1532001]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T10:39:11+01:00 demo-hp sudo[1582028]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T10:54:11+01:00 demo-hp sudo[1632919]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T11:09:11+01:00 demo-hp sudo[1683016]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T11:24:11+01:00 demo-hp sudo[1733023]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T11:39:11+01:00 demo-hp sudo[1783077]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T11:54:12+01:00 demo-hp sudo[1833125]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T12:09:11+01:00 demo-hp sudo[1883187]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T12:24:12+01:00 demo-hp sudo[1933288]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T12:39:11+01:00 demo-hp sudo[1983322]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T12:54:11+01:00 demo-hp sudo[2033570]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T13:09:12+01:00 demo-hp sudo[2095848]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T13:24:11+01:00 demo-hp sudo[2159490]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T13:39:11+01:00 demo-hp sudo[2216453]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T13:54:11+01:00 demo-hp sudo[2275938]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T13:54:26+01:00 demo-hp systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)...
2026-11-10T13:54:26+01:00 demo-hp felhom-agent[1264]: time=2026-11-10T13:54:26.844+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled"
2026-11-10T13:54:26+01:00 demo-hp systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T13:57:14+01:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T13:57:15+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:57:15.046+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=1 prune="keep-daily=1" wait_timeout=30m0s prune_pbs_allowed=false primary=true
2026-11-10T13:57:15+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:57:15.046+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
2026-11-10T13:57:15+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:57:15.047+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)"
2026-11-10T13:57:15+01:00 demo-hp sudo[1275]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T125715Z-kernel-kernel-boot.json
2026-11-10T13:57:16+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:57:16.045+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T13:57:16+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:57:16.045+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T13:57:16+01:00 demo-hp sudo[1446]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T13:57:16+01:00 demo-hp sudo[1507]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T13:57:16+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:57:16.911+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gue
2026-11-10T13:57:16+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:57:16.911+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T13:57:16+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:57:16.912+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or rest
2026-11-10T13:57:16+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:57:16.912+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never besi
2026-11-10T13:58:28+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:58:28.317+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-11-10T14:12:19+01:00 demo-hp sudo[62389]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T14:27:16+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T14:27:16.046+01:00 level=INFO msg="backup: restore-test first evaluation after start (R-874)" after=30m0s
2026-11-10T14:27:16+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T14:27:16.383+01:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=local volid=local:backup/felhom-golden-0.236.0.tar.zst size_bytes=654115664 rea
2026-11-10T14:27:16+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T14:27:16.735+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-05T20:50:35Z) is already proven; local: no set
2026-11-10T14:27:19+01:00 demo-hp sudo[112482]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T14:42:19+01:00 demo-hp sudo[162646]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T14:57:19+01:00 demo-hp sudo[212644]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:12:19+01:00 demo-hp sudo[262858]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:27:19+01:00 demo-hp sudo[312873]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:30:24+01:00 demo-hp systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)...
2026-11-10T15:30:24+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T15:30:24.913+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled"
2026-11-10T15:30:24+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T15:30:24.913+01:00 level=INFO msg="pbs: verify loop shutting down" reason="context canceled"
2026-11-10T15:30:25+01:00 demo-hp systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T15:33:09+01:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T15:33:09+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:33:09.997+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=1 prune="keep-daily=1" wait_timeout=30m0s prune_pbs_allowed=false primary=true
2026-11-10T15:33:09+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:33:09.997+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
2026-11-10T15:33:09+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:33:09.997+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)"
2026-11-10T15:33:10+01:00 demo-hp sudo[1318]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T143309Z-kernel-kernel-boot.json
2026-11-10T15:33:10+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:33:10.983+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:33:10+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:33:10.983+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:33:11+01:00 demo-hp sudo[1496]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:33:11+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:33:11.889+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gue
2026-11-10T15:33:11+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:33:11.889+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:33:11+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:33:11.889+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or rest
2026-11-10T15:33:11+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:33:11.890+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never besi
2026-11-10T15:33:12+01:00 demo-hp sudo[1660]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:34:21+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:34:21.282+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-11-10T15:40:09+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:40:09.290+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled"
2026-11-10T15:40:09+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:40:09.290+01:00 level=INFO msg="pbs: verify loop shutting down" reason="context canceled"
2026-11-10T15:40:09+01:00 demo-hp systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)...
2026-11-10T15:40:09+01:00 demo-hp systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T15:43:04+01:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T15:43:05+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:05.011+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=1 prune="keep-daily=1" wait_timeout=30m0s prune_pbs_allowed=false primary=true
2026-11-10T15:43:05+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:05.011+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
2026-11-10T15:43:05+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:05.011+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)"
2026-11-10T15:43:05+01:00 demo-hp sudo[1323]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T144304Z-kernel-kernel-boot.json
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.000+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.000+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:43:06+01:00 demo-hp sudo[1499]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:43:06+01:00 demo-hp sudo[1563]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.858+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gue
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.858+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.859+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or rest
2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.859+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never besi
2026-11-10T15:44:16+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:44:16.241+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-11-10T15:58:09+01:00 demo-hp sudo[65997]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:58:15+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:58:15.231+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.173.0 report="{\"agent_update\": {\"sha256\": \"752b2727fb6043417
2026-11-10T15:58:17+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:58:17.289+01:00 level=INFO msg="pbs: verify loop shutting down" reason="context canceled"
2026-11-10T15:58:17+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:58:17.289+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled"
2026-11-10T15:58:17+01:00 demo-hp systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)...
2026-11-10T15:58:17+01:00 demo-hp systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T15:58:17+01:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T15:58:17+01:00 demo-hp sudo[66874]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T145817Z-kernel-kernel-boot.json
2026-11-10T15:58:17+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:17.356+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=1 prune="keep-daily=1" wait_timeout=30m0s prune_pbs_allowed=false primary=true
2026-11-10T15:58:17+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:17.356+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
2026-11-10T15:58:17+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:17.356+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)"
2026-11-10T15:58:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:18.424+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:58:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:18.424+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:58:18+01:00 demo-hp sudo[67051]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:58:19+01:00 demo-hp sudo[67167]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.388+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gu
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.388+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.389+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or res
2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.389+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never bes
2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.427+01:00 level=INFO msg="osupdate: running the bundle policy pass after the self-update commit (R-1285)" agent_version=0.173.0
2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.589+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE POLICY DONE install_mode=appliance removed=0"
2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.589+01:00 level=INFO msg="osupdate: bundle policy pass done" bundle_policy="{\"install_mode\": \"appliance\", \"removed\": []}"
2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.592+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.173.0 removed=49
2026-11-10T15:59:26+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:26.622+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-11-10T16:13:24+01:00 demo-hp sudo[120555]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T16:28:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T16:28:18.425+01:00 level=INFO msg="backup: restore-test first evaluation after start (R-874)" after=30m0s
2026-11-10T16:28:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T16:28:18.768+01:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=local volid=local:backup/felhom-golden-0.236.0.tar.zst size_bytes=654115664 re
2026-11-10T16:28:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T16:28:19.097+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-05T20:50:35Z) is already proven; local: no se
2026-11-10T16:28:24+01:00 demo-hp sudo[172820]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T16:29:17+01:00 demo-hp sudo[176191]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T16:43:24+01:00 demo-hp sudo[224892]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T16:58:24+01:00 demo-hp sudo[274799]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T17:13:24+01:00 demo-hp sudo[325353]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T17:28:24+01:00 demo-hp sudo[375316]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T17:43:24+01:00 demo-hp sudo[425306]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T17:58:24+01:00 demo-hp sudo[475151]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T18:13:24+01:00 demo-hp sudo[525197]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T18:28:24+01:00 demo-hp sudo[575041]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T18:43:24+01:00 demo-hp sudo[625011]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T18:58:24+01:00 demo-hp sudo[674867]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T19:13:24+01:00 demo-hp sudo[724869]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T19:28:24+01:00 demo-hp sudo[774788]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T19:43:24+01:00 demo-hp sudo[824794]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T19:58:24+01:00 demo-hp sudo[874679]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T20:13:24+01:00 demo-hp sudo[924671]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T20:28:24+01:00 demo-hp sudo[974638]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T20:43:24+01:00 demo-hp sudo[1024545]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T20:58:24+01:00 demo-hp sudo[1074504]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T21:13:24+01:00 demo-hp sudo[1124372]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T21:28:24+01:00 demo-hp sudo[1174370]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T21:43:24+01:00 demo-hp sudo[1224315]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T21:58:24+01:00 demo-hp sudo[1274176]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T21:59:26+01:00 demo-hp sudo[1277687]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T22:00:40+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T22:00:40.811+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-11-10T22:13:24+01:00 demo-hp sudo[1324125]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T22:28:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T22:28:19.944+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-05T20:50:35Z) is already proven; local: no se
2026-11-10T22:28:24+01:00 demo-hp sudo[1374071]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T22:43:24+01:00 demo-hp sudo[1424003]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T22:58:24+01:00 demo-hp sudo[1474011]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T23:13:24+01:00 demo-hp sudo[1523951]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T23:28:24+01:00 demo-hp sudo[1574019]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T23:43:24+01:00 demo-hp sudo[1623864]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T23:58:24+01:00 demo-hp sudo[1673826]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T00:13:24+01:00 demo-hp sudo[1723921]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T00:28:24+01:00 demo-hp sudo[1773909]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T00:43:24+01:00 demo-hp sudo[1823847]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T00:58:24+01:00 demo-hp sudo[1873734]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T01:13:24+01:00 demo-hp sudo[1923814]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T01:28:24+01:00 demo-hp sudo[1973839]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T01:43:24+01:00 demo-hp sudo[2023859]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T01:58:24+01:00 demo-hp sudo[2073734]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T02:13:24+01:00 demo-hp sudo[2123673]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T02:28:24+01:00 demo-hp sudo[2173583]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T02:43:24+01:00 demo-hp sudo[2235197]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T02:58:24+01:00 demo-hp sudo[2285057]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T03:13:24+01:00 demo-hp sudo[2335092]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T03:28:24+01:00 demo-hp sudo[2385056]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T03:43:24+01:00 demo-hp sudo[2437365]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T03:58:24+01:00 demo-hp sudo[2487208]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T03:59:26+01:00 demo-hp sudo[2490710]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T04:00:40+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:00:40.820+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-11-11T04:13:24+01:00 demo-hp sudo[2537305]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T04:28:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:28:19.579+01:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=local archive=local:backup/vzdump-lxc-9201-2026_11_10-02_35_34
2026-11-11T04:28:20+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:28:20.131+01:00 level=INFO msg="restore-test: space preflight passed" storage=nvme-scratch required_bytes=41914830848 avail_bytes=633007927296
2026-11-11T04:28:20+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:28:20.232+01:00 level=INFO msg="restore-test: full-fidelity restore params derived from the archive config" scratch=990000 params=4
2026-11-11T04:28:24+01:00 demo-hp sudo[2598186]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T04:31:39+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:31:39.144+01:00 level=INFO msg="restore-test: scratch guest torn down" vmid=990000
2026-11-11T04:31:39+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:31:39.145+01:00 level=INFO msg="backup: scheduled restore-test passed" archive=local:backup/vzdump-lxc-9201-2026_11_10-02_35_34.tar.zst duration_s=199.562184354
2026-11-11T04:33:02+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:33:02.092+01:00 level=INFO msg="backup: space preflight passed" vmid=9201 target=local last_archive_bytes=9095951125 need_bytes=12443680730 avail_bytes=22617755648
2026-11-11T04:33:03+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:33:03.153+01:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=local job=backup-9201-1794367981574952245
2026-11-11T04:38:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:38:19.390+01:00 level=INFO msg="janitor: stale-lock sweep deferred — a heavy operation is in flight" busy=backup:local
2026-11-11T04:43:24+01:00 demo-hp sudo[2655608]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T04:44:09+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:44:09.948+01:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_11_11-04_33_02.tar.zst size_bytes=9096384826 uncovered_volumes=2
2026-11-11T04:44:09+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:44:09.949+01:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1794367981574952245 archive=local:backup/vzdump-lxc-9201-2026_11_11-04_33_02.tar.zs
2026-11-11T04:45:39+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:45:39.956+01:00 level=INFO msg="osupdate: START" run=20261111T034539Z layer=guest vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261111T034539Z
2026-11-11T04:45:58+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:45:58.939+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261111T034539Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
2026-11-11T04:45:58+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:45:58.939+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-11T04:45:58+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:45:58.939+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
2026-11-11T04:45:58+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:45:58.939+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
2026-11-11T04:45:58+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:45:58.940+01:00 level=INFO msg="osupdate: DONE" run=20261111T034539Z layer=guest vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0
2026-11-11T04:45:58+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:45:58.982+01:00 level=INFO msg="osupdate: START" run=20261111T034539Z layer=host vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261111T034539Z
2026-11-11T04:46:15+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:15.451+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261111T034539Z layer=host lane=fast mode=apply select=pending-fast packages=0"
2026-11-11T04:46:15+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:15.451+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-11T04:46:15+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:15.451+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
2026-11-11T04:46:15+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:15.451+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
2026-11-11T04:46:16+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:16.538+01:00 level=INFO msg="osupdate: DONE" run=20261111T034539Z layer=host vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=16 not_covered=16
2026-11-11T04:46:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:18.657+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE already on"
2026-11-11T04:46:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:18.657+01:00 level=INFO msg="osupdate: live-restore" vmid=9201 result="{\"result\": \"already on\"}"
2026-11-11T04:46:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:18.657+01:00 level=INFO msg="osupdate: START" run=20261111T034539Z layer=docker vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261111T034539Z
2026-11-11T04:46:48+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:48.868+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261111T034539Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=
2026-11-11T04:46:48+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:48.868+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-11T04:46:48+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:48.868+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
2026-11-11T04:46:48+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:48.868+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
2026-11-11T04:46:48+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:48.868+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: OOM-CHECK result=pass oom_killed=True oom_event=True exit=137 image=gitea.dooplex.hu/admin/felhom-controller:0.352.0 â€
2026-11-11T04:46:48+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:48.869+01:00 level=INFO msg="osupdate: DONE" run=20261111T034539Z layer=docker vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0
2026-11-11T04:46:48+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:48.883+01:00 level=INFO msg="osupdate: pve step holds the /etc/pve write gate — the agent's own writes wait until it ends" run=20261111T034539Z layer=pve vmid=9201 trigger=night
2026-11-11T04:46:48+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:48.883+01:00 level=INFO msg="osupdate: START" run=20261111T034539Z layer=pve vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261111T034539Z
2026-11-11T04:47:06+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:47:06.472+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261111T034539Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0"
2026-11-11T04:47:06+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:47:06.472+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-11T04:47:06+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:47:06.472+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
2026-11-11T04:47:06+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:47:06.472+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
2026-11-11T04:47:07+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:47:07.887+01:00 level=INFO msg="osupdate: DONE" run=20261111T034539Z layer=pve vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=2 not_covered=0 re
2026-11-11T04:47:07+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:47:07.892+01:00 level=INFO msg="osupdate: pve step released the /etc/pve write gate" run=20261111T034539Z layer=pve vmid=9201 trigger=night
2026-11-11T04:47:07+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:47:07.892+01:00 level=INFO msg="osupdate: kernel step skipped — the hub names no kernel step for this box" run=20261111T034539Z layer=kernel vmid=9201 trigger=night ring=0
2026-11-11T04:58:24+01:00 demo-hp sudo[2708953]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T05:13:24+01:00 demo-hp sudo[2758966]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T05:28:24+01:00 demo-hp sudo[2808864]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T05:43:24+01:00 demo-hp sudo[2858958]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T05:58:24+01:00 demo-hp sudo[2908875]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T06:13:24+01:00 demo-hp sudo[2958776]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T06:28:24+01:00 demo-hp sudo[3008761]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T06:43:24+01:00 demo-hp sudo[3058666]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T06:58:24+01:00 demo-hp sudo[3108643]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T07:13:24+01:00 demo-hp sudo[3158591]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T07:28:24+01:00 demo-hp sudo[3208701]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T07:43:24+01:00 demo-hp sudo[3258653]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T07:58:24+01:00 demo-hp sudo[3308680]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T08:13:24+01:00 demo-hp sudo[3358593]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T08:28:24+01:00 demo-hp sudo[3408639]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T08:43:24+01:00 demo-hp sudo[3458520]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T08:58:24+01:00 demo-hp sudo[3508327]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T09:13:24+01:00 demo-hp sudo[3558359]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T09:28:24+01:00 demo-hp sudo[3608281]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T09:43:24+01:00 demo-hp sudo[3658187]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T09:58:24+01:00 demo-hp sudo[3708187]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T09:59:26+01:00 demo-hp sudo[3711739]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T10:00:40+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T10:00:40.832+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-11-11T10:13:24+01:00 demo-hp sudo[3758186]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T10:28:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T10:28:19.885+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-05T20:50:35Z) is already proven; local: no se
2026-11-11T10:28:24+01:00 demo-hp sudo[3808072]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T10:43:24+01:00 demo-hp sudo[3858055]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T10:58:24+01:00 demo-hp sudo[3908221]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
=== agent log: line count per hour, last 26 h ===
12339 2026-11-10T09 13253 2026-11-10T10 13266 2026-11-10T11 13262 2026-11-10T12 12658 2026-11-10T13 13271 2026-11-10T14 12173 2026-11-10T15 13301 2026-11-10T16 13278 2026-11-10T17 13268 2026-11-10T18 13244 2026-11-10T19 13256 2026-11-10T20 13270 2026-11-10T21 13266 2026-11-10T22 13246 2026-11-10T23 13249 2026-11-11T00 13261 2026-11-11T01 13267 2026-11-11T02 13507 2026-11-11T03 13345 2026-11-11T04 13267 2026-11-11T05 13264 2026-11-11T06 13261 2026-11-11T07 13261 2026-11-11T08 13252 2026-11-11T09 13262 2026-11-11T10 996 2026-11-11T11
=== agent version ===
felhom-agent 0.173.0
RC=0
@@ -0,0 +1,156 @@
# host-night demo-hp taken 2026-11-11T10:09:23Z
=== pve tasks, last 14 days (local time) ===
2026-11-03 03:52:35 -> 03:52:35 0.0 min push_file 9202 OK
2026-11-03 03:52:43 -> 03:52:43 0.0 min push_file 9202 OK
2026-11-03 03:52:51 -> 03:52:51 0.0 min push_file 9202 OK
2026-11-03 03:53:00 -> 03:53:00 0.0 min push_file 9202 OK
2026-11-03 03:53:08 -> 03:53:08 0.0 min push_file 9202 OK
2026-11-03 03:53:32 -> 03:53:32 0.0 min push_file 9202 OK
2026-11-03 03:53:35 -> 03:53:35 0.0 min push_file 9202 OK
2026-11-03 03:53:38 -> 03:53:38 0.0 min push_file 9202 OK
2026-11-03 03:53:41 -> 03:53:41 0.0 min push_file 9202 OK
2026-11-03 03:54:23 -> 03:54:23 0.0 min push_file 9202 OK
2026-11-03 03:54:27 -> 03:54:27 0.0 min push_file 9202 OK
2026-11-03 03:54:31 -> 03:54:31 0.0 min push_file 9202 OK
2026-11-03 03:54:32 -> 03:54:32 0.0 min push_file 9202 OK
2026-11-03 03:55:11 -> 03:55:11 0.0 min push_file 9202 OK
2026-11-03 03:55:15 -> 03:55:15 0.0 min push_file 9202 OK
2026-11-03 03:55:16 -> 03:55:16 0.0 min push_file 9202 OK
2026-11-03 03:56:51 -> 03:56:51 0.0 min push_file 9202 OK
2026-11-03 03:56:55 -> 03:56:55 0.0 min push_file 9202 OK
2026-11-03 03:56:56 -> 03:56:56 0.0 min push_file 9202 OK
2026-11-03 03:57:04 -> 03:57:04 0.0 min push_file 9202 OK
2026-11-03 03:57:13 -> 03:57:13 0.0 min push_file 9202 OK
2026-11-03 03:57:21 -> 03:57:21 0.0 min push_file 9202 OK
2026-11-03 03:57:30 -> 03:57:30 0.0 min push_file 9202 OK
2026-11-03 03:57:38 -> 03:57:38 0.0 min push_file 9202 OK
2026-11-03 03:57:47 -> 03:57:47 0.0 min push_file 9202 OK
2026-11-03 03:57:55 -> 03:57:55 0.0 min push_file 9202 OK
2026-11-03 03:58:04 -> 03:58:04 0.0 min push_file 9202 OK
2026-11-03 03:58:12 -> 03:58:12 0.0 min push_file 9202 OK
2026-11-03 03:58:21 -> 03:58:21 0.0 min push_file 9202 OK
2026-11-03 03:58:29 -> 03:58:29 0.0 min push_file 9202 OK
2026-11-03 03:58:38 -> 03:58:38 0.0 min push_file 9202 OK
2026-11-03 03:58:46 -> 03:58:46 0.0 min push_file 9202 OK
2026-11-03 03:58:54 -> 03:58:54 0.0 min push_file 9202 OK
2026-11-03 03:59:02 -> 03:59:02 0.0 min push_file 9202 OK
2026-11-03 03:59:11 -> 03:59:11 0.0 min push_file 9202 OK
2026-11-03 03:59:19 -> 03:59:19 0.0 min push_file 9202 OK
2026-11-03 03:59:27 -> 03:59:27 0.0 min push_file 9202 OK
2026-11-03 03:59:35 -> 03:59:35 0.0 min push_file 9202 OK
2026-11-03 03:59:43 -> 03:59:43 0.0 min push_file 9202 OK
2026-11-03 03:59:51 -> 03:59:51 0.0 min push_file 9202 OK
2026-11-03 03:59:59 -> 04:00:00 0.0 min push_file 9202 OK
2026-11-03 04:00:08 -> 04:00:08 0.0 min push_file 9202 OK
2026-11-03 04:00:16 -> 04:00:16 0.0 min push_file 9202 OK
2026-11-03 04:00:24 -> 04:00:24 0.0 min push_file 9202 OK
2026-11-03 04:00:33 -> 04:00:34 0.0 min push_file 9202 OK
2026-11-03 04:00:42 -> 04:00:42 0.0 min push_file 9202 OK
2026-11-03 04:00:50 -> 04:00:50 0.0 min push_file 9202 OK
2026-11-03 04:00:58 -> 04:00:58 0.0 min push_file 9202 OK
2026-11-03 04:01:06 -> 04:01:06 0.0 min push_file 9202 OK
2026-11-03 04:01:14 -> 04:01:14 0.0 min push_file 9202 OK
2026-11-03 04:01:23 -> 04:01:23 0.0 min push_file 9202 OK
2026-11-03 04:01:31 -> 04:01:31 0.0 min push_file 9202 OK
2026-11-03 04:01:39 -> 04:01:39 0.0 min push_file 9202 OK
2026-11-03 04:01:47 -> 04:01:47 0.0 min push_file 9202 OK
2026-11-03 04:01:56 -> 04:01:56 0.0 min push_file 9202 OK
2026-11-03 04:02:04 -> 04:02:04 0.0 min push_file 9202 OK
2026-11-03 04:02:12 -> 04:02:12 0.0 min push_file 9202 OK
2026-11-03 04:02:20 -> 04:02:20 0.0 min push_file 9202 OK
2026-11-03 04:02:28 -> 04:02:28 0.0 min push_file 9202 OK
2026-11-03 04:02:37 -> 04:02:37 0.0 min push_file 9202 OK
2026-11-03 04:02:45 -> 04:02:45 0.0 min push_file 9202 OK
2026-11-03 04:02:53 -> 04:02:53 0.0 min push_file 9202 OK
2026-11-03 04:03:02 -> 04:03:02 0.0 min push_file 9202 OK
2026-11-03 04:03:25 -> 04:03:25 0.0 min push_file 9202 OK
2026-11-03 04:03:29 -> 04:03:29 0.0 min push_file 9202 OK
2026-11-03 04:03:32 -> 04:03:32 0.0 min push_file 9202 OK
2026-11-03 04:03:35 -> 04:03:35 0.0 min push_file 9202 OK
2026-11-03 04:04:17 -> 04:04:18 0.0 min push_file 9202 OK
2026-11-03 04:04:22 -> 04:04:22 0.0 min push_file 9202 OK
2026-11-03 04:04:26 -> 04:04:26 0.0 min push_file 9202 OK
2026-11-03 04:04:27 -> 04:04:27 0.0 min push_file 9202 OK
2026-11-03 04:04:53 -> 04:04:53 0.0 min push_file 9202 OK
2026-11-03 04:05:09 -> 04:05:09 0.0 min push_file 9202 OK
2026-11-03 04:07:29 -> 04:07:29 0.0 min push_file 9202 OK
2026-11-03 04:07:33 -> 04:07:37 0.1 min vzshutdown 9401 OK
2026-11-03 05:50:30 -> 05:50:35 0.1 min aptupdate OK
2026-11-03 08:22:29 -> 08:22:29 0.0 min qmcreate 9383 OK
2026-11-03 08:22:32 -> 08:22:33 0.0 min qmstart 9383 OK
2026-11-03 10:14:45 -> 10:14:46 0.0 min qmstop 9383 OK
2026-11-03 10:14:47 -> 10:14:47 0.0 min qmdestroy 9383 OK
2026-11-04 02:09:47 -> 02:09:51 0.1 min aptupdate OK
2026-11-05 01:18:47 -> 01:18:52 0.1 min aptupdate OK
2026-11-05 21:50:35 -> 21:57:39 7.1 min vzdump 9201 OK
2026-11-06 03:38:47 -> 03:38:51 0.1 min aptupdate OK
2026-11-07 05:01:47 -> 05:01:51 0.1 min aptupdate OK
2026-11-07 22:39:02 -> 22:39:02 0.0 min imgdel 9100@local OK
2026-11-07 23:42:03 -> 23:55:58 13.9 min vzrestore 990000 OK
2026-11-07 23:56:01 -> 23:56:05 0.1 min vzstart 990000 OK
2026-11-07 23:56:07 -> 23:56:10 0.1 min vzdestroy 990000 OK
2026-11-08 01:22:04 -> 01:22:08 0.1 min aptupdate OK
2026-11-08 02:30:38 -> 02:43:34 12.9 min vzdump 9201 OK
2026-11-09 03:29:44 -> 03:29:48 0.1 min aptupdate OK
2026-11-09 05:56:15 -> 05:59:17 3.0 min vzrestore 990000 OK
2026-11-09 05:59:21 -> 05:59:25 0.1 min vzstart 990000 OK
2026-11-09 05:59:28 -> 05:59:30 0.0 min vzdestroy 990000 OK
2026-11-09 09:51:30 -> 09:51:30 0.0 min imgdel local OK
2026-11-10 02:35:34 -> 02:46:27 10.9 min vzdump 9201 OK
2026-11-10 02:51:00 -> 02:52:45 1.8 min vzshutdown 9201 OK
2026-11-10 02:51:00 -> 02:52:45 1.8 min vzshutdown 9202 OK
2026-11-10 02:51:00 -> 02:53:09 2.1 min qmshutdown 341 OK
2026-11-10 02:51:00 -> 02:53:09 2.1 min stopall OK
2026-11-10 02:54:10 -> 02:54:13 0.1 min qmstart 341 OK
2026-11-10 02:54:10 -> 02:54:18 0.1 min startall OK
2026-11-10 02:54:13 -> 02:54:18 0.1 min vzstart 9201 OK
2026-11-10 02:54:13 -> 02:54:18 0.1 min vzstart 9202 OK
2026-11-10 03:05:11 -> 03:05:15 0.1 min aptupdate OK
2026-11-10 12:56:44 -> 12:56:44 0.0 min push_file 9202 OK
2026-11-10 12:56:51 -> 12:56:51 0.0 min push_file 9202 OK
2026-11-10 12:56:52 -> 12:56:52 0.0 min push_file 9202 OK
2026-11-10 12:56:57 -> 12:56:57 0.0 min push_file 9202 OK
2026-11-10 12:57:00 -> 12:57:00 0.0 min push_file 9202 OK
2026-11-10 12:57:14 -> 12:57:15 0.0 min push_file 9202 OK
2026-11-10 12:57:18 -> 12:57:18 0.0 min push_file 9202 OK
2026-11-10 13:24:35 -> 13:24:35 0.0 min push_file 9202 OK
2026-11-10 13:24:36 -> 13:24:36 0.0 min push_file 9202 OK
2026-11-10 13:26:55 -> 13:26:55 0.0 min push_file 9202 OK
2026-11-10 13:31:27 -> 13:31:27 0.0 min push_file 9202 OK
2026-11-10 13:31:30 -> 13:31:30 0.0 min push_file 9202 OK
2026-11-10 13:31:44 -> 13:31:44 0.0 min push_file 9202 OK
2026-11-10 13:31:48 -> 13:31:48 0.0 min push_file 9202 OK
2026-11-10 13:31:53 -> 13:31:53 0.0 min push_file 9202 OK
2026-11-10 13:54:31 -> 13:56:16 1.8 min vzshutdown 9201 OK
2026-11-10 13:54:31 -> 13:56:16 1.8 min vzshutdown 9202 OK
2026-11-10 13:54:31 -> 13:56:26 1.9 min qmshutdown 341 OK
2026-11-10 13:54:31 -> 13:56:26 1.9 min stopall OK
2026-11-10 13:57:17 -> 13:57:20 0.1 min qmstart 341 OK
2026-11-10 13:57:17 -> 13:57:25 0.1 min startall OK
2026-11-10 13:57:20 -> 13:57:25 0.1 min vzstart 9201 OK
2026-11-10 13:57:20 -> 13:57:25 0.1 min vzstart 9202 OK
2026-11-10 15:30:29 -> 15:32:12 1.7 min vzshutdown 9201 OK
2026-11-10 15:30:29 -> 15:32:12 1.7 min vzshutdown 9202 OK
2026-11-10 15:30:29 -> 15:32:22 1.9 min qmshutdown 341 OK
2026-11-10 15:30:29 -> 15:32:23 1.9 min stopall OK
2026-11-10 15:33:12 -> 15:33:15 0.1 min qmstart 341 OK
2026-11-10 15:33:12 -> 15:33:21 0.1 min startall OK
2026-11-10 15:33:15 -> 15:33:21 0.1 min vzstart 9201 OK
2026-11-10 15:33:15 -> 15:33:21 0.1 min vzstart 9202 OK
2026-11-10 15:40:13 -> 15:41:57 1.7 min vzshutdown 9201 OK
2026-11-10 15:40:13 -> 15:41:57 1.7 min vzshutdown 9202 OK
2026-11-10 15:40:13 -> 15:42:17 2.1 min qmshutdown 341 OK
2026-11-10 15:40:13 -> 15:42:17 2.1 min stopall OK
2026-11-10 15:43:07 -> 15:43:10 0.1 min qmstart 341 OK
2026-11-10 15:43:07 -> 15:43:16 0.1 min startall OK
2026-11-10 15:43:10 -> 15:43:16 0.1 min vzstart 9201 OK
2026-11-10 15:43:10 -> 15:43:16 0.1 min vzstart 9202 OK
2026-11-11 02:23:00 -> 02:23:04 0.1 min aptupdate OK
2026-11-11 04:28:20 -> 04:31:22 3.0 min vzrestore 990000 OK
2026-11-11 04:31:26 -> 04:31:30 0.1 min vzstart 990000 OK
2026-11-11 04:31:33 -> 04:31:35 0.0 min vzdestroy 990000 OK
2026-11-11 04:33:02 -> 04:44:08 11.1 min vzdump 9201 OK
=== agent: night words, 20:00 yesterday to 09:00 today (first 3 lines per message) ===
--- counts
RC=0
@@ -0,0 +1,6 @@
# offset demo-hp: box clock minus DooPlex clock (midpoint of the call), seconds
offset=0.067 rtt=0.142
offset=0.073 rtt=0.155
offset=0.071 rtt=0.152
offset=0.074 rtt=0.157
offset=0.069 rtt=0.148
@@ -0,0 +1,465 @@
# guest-clock tester-1 taken 2026-11-11T10:04:20Z from DooPlex
=== clock ===
utc 2026-11-11T10:04:21.821299550Z
local 2026-11-11T10:04:21 UTC +0000
/usr/share/zoneinfo/Etc/UTC
Local time: Wed 2026-11-11 10:04:22 UTC
Universal time: Wed 2026-11-11 10:04:22 UTC
RTC time: n/a
Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
NTP service: inactive
RTC in local TZ: no
chrony: inactive
systemd-timesyncd: inactive
=== boots ===
2026-11-10 14:43:51
=== timers ===
NEXT LEFT LAST PASSED UNIT ACTIVATES
Wed 2026-11-11 14:59:32 UTC 4h 55min Tue 2026-11-10 14:59:32 UTC 19h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Wed 2026-11-11 15:04:07 UTC 4h 59min Tue 2026-11-10 18:44:20 UTC 15h ago apt-daily.timer apt-daily.service
Thu 2026-11-12 00:00:00 UTC 13h Wed 2026-11-11 00:00:20 UTC 10h ago dpkg-db-backup.timer dpkg-db-backup.service
Thu 2026-11-12 00:33:59 UTC 14h Wed 2026-11-11 00:01:20 UTC 10h ago logrotate.timer logrotate.service
Thu 2026-11-12 05:02:46 UTC 18h Wed 2026-11-11 02:25:42 UTC 7h ago man-db.timer man-db.service
Thu 2026-11-12 06:02:16 UTC 19h Wed 2026-11-11 06:46:20 UTC 3h 18min ago apt-daily-upgrade.timer apt-daily-upgrade.service
Sun 2026-11-15 03:10:52 UTC 3 days Sun 2026-11-08 03:10:38 UTC - e2scrub_all.timer e2scrub_all.service
- - - - fstrim.timer fstrim.service
8 timers listed.
=== timer units (all) ===
--- apt-daily-upgrade.timer [enabled/active]
OnCalendar=*-*-* 6:00
RandomizedDelaySec=60m
Persistent=true
Unit=apt-daily-upgrade.service NextElapseUSecRealtime=Thu 2026-11-12 06:02:16 UTC LastTriggerUSec=Wed 2026-11-11 06:46:20 UTC
--- apt-daily.timer [enabled/active]
OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h
Persistent=true
Unit=apt-daily.service NextElapseUSecRealtime=Wed 2026-11-11 15:04:07 UTC LastTriggerUSec=Tue 2026-11-10 18:44:20 UTC
--- apt-listchanges.timer [disabled/inactive]
OnCalendar=hourly
Unit=apt-listchanges.service NextElapseUSecRealtime= LastTriggerUSec=
--- dpkg-db-backup.timer [enabled/active]
OnCalendar=daily
Persistent=true
Unit=dpkg-db-backup.service NextElapseUSecRealtime=Thu 2026-11-12 00:00:00 UTC LastTriggerUSec=Wed 2026-11-11 00:00:20 UTC
--- e2scrub_all.timer [enabled/active]
OnCalendar=Sun *-*-* 03:10:00
RandomizedDelaySec=60
Persistent=true
Unit=e2scrub_all.service NextElapseUSecRealtime=Sun 2026-11-15 03:10:52 UTC LastTriggerUSec=Sun 2026-11-08 03:10:38 UTC
--- fstrim.timer [enabled/inactive]
OnCalendar=weekly
AccuracySec=1h
Persistent=true
RandomizedDelaySec=100min
Unit=fstrim.service NextElapseUSecRealtime= LastTriggerUSec=
--- logrotate.timer [enabled/active]
OnCalendar=daily
RandomizedDelaySec=1h
Persistent=true
Unit=logrotate.service NextElapseUSecRealtime=Thu 2026-11-12 00:33:59 UTC LastTriggerUSec=Wed 2026-11-11 00:01:20 UTC
--- man-db.timer [enabled/active]
OnCalendar=daily
RandomizedDelaySec=12h
Persistent=true
Unit=man-db.service NextElapseUSecRealtime=Thu 2026-11-12 05:02:46 UTC LastTriggerUSec=Wed 2026-11-11 02:25:42 UTC
--- systemd-tmpfiles-clean.timer [static/active]
OnBootSec=15min
OnUnitActiveSec=1d
Unit=systemd-tmpfiles-clean.service NextElapseUSecRealtime= LastTriggerUSec=Tue 2026-11-10 14:59:32 UTC
=== felhom units ===
felhom-controller-bootstrap.path loaded active running
felhom-controller-bootstrap.service loaded active exited
felhom-regen-hostkeys.service loaded inactive dead
--- felhom-controller-bootstrap.service
Type=oneshot Restart=no RestartUSec=100ms TimeoutStartUSec=infinity NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:17:03 UTC
--- felhom-regen-hostkeys.service
Type=oneshot Restart=no RestartUSec=100ms TimeoutStartUSec=infinity NRestarts=0 ActiveEnterTimestamp=
=== cron ===
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
25 * * * * root cd / && run-parts --report /etc/cron.hourly
5 6 * * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; }
8 6 * * 7 root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.weekly; }
58 6 1 * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.monthly; }
--- /etc/cron.d/e2scrub_all
35 3 * * 0 root test -e /run/systemd/system || SERVICE_MODE=1 /usr/libexec/e2fsprogs/e2scrub_all_cron
38 3 * * * root test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r
--- cron.hourly:
--- cron.daily: apt-compat dpkg logrotate man-db
--- cron.weekly: man-db
--- cron.monthly:
--- user crontabs:
=== at ===
no at
=== docker daemon timers ===
{
"features": { "containerd-snapshotter": false },
"live-restore": true,
"log-driver": "json-file",
"log-opts": { "max-size": "10m", "max-file": "3" }
}
=== containers ===
privatebin privatebin/pdo:2.1.0 Up 6 hours (healthy) 6 hours ago
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:3.4.1 Up 6 hours (healthy) 6 hours ago
paperless-redis redis:7-alpine Up 6 hours (healthy) 6 hours ago
paperless-postgres postgres:18-alpine Up 6 hours (healthy) 6 hours ago
bookstack lscr.io/linuxserver/bookstack:26.09.2 Up 6 hours (healthy) 6 hours ago
bookstack-db mariadb:12.3 Up 6 hours (healthy) 6 hours ago
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 19 hours (healthy) 19 hours ago
traefik traefik:v3.7.17 Up 19 hours 8 days ago
filebrowser gtstef/filebrowser:1.5.8-stable Up 19 hours (healthy) 12 days ago
cloudflared cloudflare/cloudflared:2026.10.4 Up 19 hours (healthy) 2 weeks ago
=== restart policies + healthcheck intervals ===
/privatebin restart=unless-stopped health=30s started=2026-11-11T04:32:50.895902437Z
/paperless-webserver restart=unless-stopped health=30s started=2026-11-11T04:32:49.930525952Z
/paperless-redis restart=unless-stopped health=10s started=2026-11-11T04:32:39.478872694Z
/paperless-postgres restart=unless-stopped health=10s started=2026-11-11T04:32:39.708807897Z
/bookstack restart=unless-stopped health=30s started=2026-11-11T04:32:38.567990532Z
/bookstack-db restart=unless-stopped health=10s started=2026-11-11T04:32:32.829563948Z
/felhom-controller restart=always health=30s started=2026-11-10T15:17:03.705107516Z
/traefik restart=unless-stopped health=none started=2026-11-10T14:44:04.469852075Z
/filebrowser restart=unless-stopped health=30s started=2026-11-10T14:44:13.671000663Z
/cloudflared restart=unless-stopped health=30s started=2026-11-10T14:44:04.702351924Z
=== controller TZ ===
TZ=unset
in-container 2026-11-11T10:04:23 UTC +0000
=== controller log: first line, line count per hour ===
2026-11-10T15:17:04.380509770Z 2026/11/10 15:17:04 [INFO] local-api: channel up (agent 169.254.253.1:8443) — guest 9201, 3 mount(s) visible
881 2026-11-10T15 1093 2026-11-10T16 1088 2026-11-10T17 1090 2026-11-10T18 1096 2026-11-10T19 1096 2026-11-10T20 1095 2026-11-10T21 1095 2026-11-10T22 1101 2026-11-10T23 1102 2026-11-11T00 1165 2026-11-11T01 1148 2026-11-11T02 1221 2026-11-11T03 1149 2026-11-11T04 1104 2026-11-11T05 1100 2026-11-11T06 1105 2026-11-11T07 1111 2026-11-11T08 1113 2026-11-11T09 89 2026-11-11T10
=== controller log: schedule words (tail 320) ===
2026-11-11T09:20:05.160676453Z 2026/11/11 09:20:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:20:05.683405736Z 2026/11/11 09:20:05 [INFO] [scheduler] Job agent-channel-health completed (took 523ms)
2026-11-11T09:21:04.906157881Z 2026/11/11 09:21:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:21:04.906190012Z 2026/11/11 09:21:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:21:04.906196204Z 2026/11/11 09:21:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:21:05.056970262Z 2026/11/11 09:21:05 [INFO] [scheduler] Job stack-scan completed (took 151ms)
2026-11-11T09:21:05.161044883Z 2026/11/11 09:21:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:21:05.664432943Z 2026/11/11 09:21:05 [INFO] [scheduler] Job agent-channel-health completed (took 503ms)
2026-11-11T09:22:04.905778525Z 2026/11/11 09:22:04 [INFO] [scheduler] Running job: backup-cache
2026-11-11T09:22:04.906641884Z 2026/11/11 09:22:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:22:04.906667213Z 2026/11/11 09:22:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:22:04.906671090Z 2026/11/11 09:22:04 [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T09:22:04.906674587Z 2026/11/11 09:22:04 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T09:22:04.906677953Z 2026/11/11 09:22:04 [INFO] [scheduler] Running job: system-health
2026-11-11T09:22:04.906693452Z 2026/11/11 09:22:04 [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T09:22:04.957577420Z 2026/11/11 09:22:04 [INFO] [backup] Found 4 DB dump files across drives
2026-11-11T09:22:05.139314206Z 2026/11/11 09:22:05 [INFO] [scheduler] Job docker-socket-users completed (took 233ms)
2026-11-11T09:22:05.140202182Z 2026/11/11 09:22:05 [INFO] [scheduler] Job backup-cache completed (took 234ms)
2026-11-11T09:22:05.161068167Z 2026/11/11 09:22:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:22:05.252001057Z 2026/11/11 09:22:05 [INFO] [scheduler] Job system-health completed (took 346ms)
2026-11-11T09:22:05.729358951Z 2026/11/11 09:22:05 [INFO] [scheduler] Job agent-channel-health completed (took 568ms)
2026-11-11T09:23:04.906690828Z 2026/11/11 09:23:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:23:04.907120039Z 2026/11/11 09:23:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:23:04.907251316Z 2026/11/11 09:23:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:23:05.063022179Z 2026/11/11 09:23:05 [INFO] [scheduler] Job stack-scan completed (took 156ms)
2026-11-11T09:23:05.160568685Z 2026/11/11 09:23:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:23:05.722110117Z 2026/11/11 09:23:05 [INFO] [scheduler] Job agent-channel-health completed (took 562ms)
2026-11-11T09:24:04.907045613Z 2026/11/11 09:24:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:24:04.907066863Z 2026/11/11 09:24:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:24:05.161051942Z 2026/11/11 09:24:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:24:05.667477165Z 2026/11/11 09:24:05 [INFO] [scheduler] Job agent-channel-health completed (took 506ms)
2026-11-11T09:25:04.906335291Z 2026/11/11 09:25:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:25:04.906871874Z 2026/11/11 09:25:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:25:04.906889197Z 2026/11/11 09:25:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:25:05.065177442Z 2026/11/11 09:25:05 [INFO] [scheduler] Job stack-scan completed (took 159ms)
2026-11-11T09:25:05.160304921Z 2026/11/11 09:25:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:25:05.674373114Z 2026/11/11 09:25:05 [INFO] [scheduler] Job agent-channel-health completed (took 514ms)
2026-11-11T09:26:04.906927150Z 2026/11/11 09:26:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:26:04.907411494Z 2026/11/11 09:26:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:26:05.161348483Z 2026/11/11 09:26:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:26:05.657398417Z 2026/11/11 09:26:05 [INFO] [scheduler] Job agent-channel-health completed (took 496ms)
2026-11-11T09:27:04.909971068Z 2026/11/11 09:27:04 [INFO] [scheduler] Running job: fill-watch-interval
2026-11-11T09:27:04.910033204Z 2026/11/11 09:27:04 [INFO] [scheduler] Job fill-watch-interval completed (took 0s)
2026-11-11T09:27:04.910037132Z 2026/11/11 09:27:04 [INFO] [scheduler] Running job: backup-cache
2026-11-11T09:27:04.910041590Z 2026/11/11 09:27:04 [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T09:27:04.910045058Z 2026/11/11 09:27:04 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T09:27:04.910048514Z 2026/11/11 09:27:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:27:04.910051610Z 2026/11/11 09:27:04 [INFO] [scheduler] Running job: system-health
2026-11-11T09:27:04.910757212Z 2026/11/11 09:27:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:27:04.912246804Z 2026/11/11 09:27:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:27:04.912272412Z 2026/11/11 09:27:04 [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T09:27:04.976000830Z 2026/11/11 09:27:04 [INFO] [backup] Found 4 DB dump files across drives
2026-11-11T09:27:05.151495802Z 2026/11/11 09:27:05 [INFO] [scheduler] Job docker-socket-users completed (took 239ms)
2026-11-11T09:27:05.152952152Z 2026/11/11 09:27:05 [INFO] [scheduler] Job stack-scan completed (took 244ms)
2026-11-11T09:27:05.160243608Z 2026/11/11 09:27:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:27:05.285384311Z 2026/11/11 09:27:05 [INFO] [scheduler] Job system-health completed (took 376ms)
2026-11-11T09:27:05.335228296Z 2026/11/11 09:27:05 [INFO] [scheduler] Job backup-cache completed (took 429ms)
2026-11-11T09:27:05.772479480Z 2026/11/11 09:27:05 [INFO] [scheduler] Job agent-channel-health completed (took 612ms)
2026-11-11T09:28:04.905755596Z 2026/11/11 09:28:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:28:04.905798426Z 2026/11/11 09:28:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:28:05.161118225Z 2026/11/11 09:28:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:28:05.667198788Z 2026/11/11 09:28:05 [INFO] [scheduler] Job agent-channel-health completed (took 506ms)
2026-11-11T09:29:04.907039643Z 2026/11/11 09:29:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:29:04.907070982Z 2026/11/11 09:29:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:29:04.907190909Z 2026/11/11 09:29:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:29:05.060136598Z 2026/11/11 09:29:05 [INFO] [scheduler] Job stack-scan completed (took 154ms)
2026-11-11T09:29:05.161026467Z 2026/11/11 09:29:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:29:05.669569076Z 2026/11/11 09:29:05 [INFO] [scheduler] Job agent-channel-health completed (took 509ms)
2026-11-11T09:30:04.907128947Z 2026/11/11 09:30:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:30:04.907156159Z 2026/11/11 09:30:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:30:05.160948675Z 2026/11/11 09:30:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:30:05.674362853Z 2026/11/11 09:30:05 [INFO] [scheduler] Job agent-channel-health completed (took 513ms)
2026-11-11T09:31:04.906170340Z 2026/11/11 09:31:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:31:04.906216067Z 2026/11/11 09:31:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:31:04.907849150Z 2026/11/11 09:31:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:31:05.055301047Z 2026/11/11 09:31:05 [INFO] [scheduler] Job stack-scan completed (took 148ms)
2026-11-11T09:31:05.161215142Z 2026/11/11 09:31:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:31:05.668219827Z 2026/11/11 09:31:05 [INFO] [scheduler] Job agent-channel-health completed (took 507ms)
2026-11-11T09:32:04.906098122Z 2026/11/11 09:32:04 [INFO] [scheduler] Running job: hub-report
2026-11-11T09:32:04.906159077Z 2026/11/11 09:32:04 [INFO] [scheduler] Running job: backup-cache
2026-11-11T09:32:04.906791701Z 2026/11/11 09:32:04 [INFO] [scheduler] Running job: system-health
2026-11-11T09:32:04.908869394Z 2026/11/11 09:32:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:32:04.908896174Z 2026/11/11 09:32:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:32:04.909802516Z 2026/11/11 09:32:04 [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T09:32:04.909865736Z 2026/11/11 09:32:04 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T09:32:04.909870334Z 2026/11/11 09:32:04 [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T09:32:04.961770841Z 2026/11/11 09:32:04 [INFO] [backup] Found 4 DB dump files across drives
2026-11-11T09:32:05.160909201Z 2026/11/11 09:32:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:32:05.236015390Z 2026/11/11 09:32:05 [INFO] [scheduler] Job docker-socket-users completed (took 327ms)
2026-11-11T09:32:05.389071178Z 2026/11/11 09:32:05 [INFO] [scheduler] Job system-health completed (took 481ms)
2026-11-11T09:32:05.674110667Z 2026/11/11 09:32:05 [INFO] [scheduler] Job backup-cache completed (took 768ms)
2026-11-11T09:32:05.829814684Z 2026/11/11 09:32:05 [INFO] [scheduler] Job agent-channel-health completed (took 667ms)
2026-11-11T09:32:06.010013867Z 2026/11/11 09:32:06 [INFO] [sync] Catalog sync complete
2026-11-11T09:32:06.560976906Z 2026/11/11 09:32:06 [INFO] [scheduler] Job hub-report completed (took 1.655s)
2026-11-11T09:33:04.907112851Z 2026/11/11 09:33:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:33:04.907267472Z 2026/11/11 09:33:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:33:04.907276790Z 2026/11/11 09:33:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:33:05.059922042Z 2026/11/11 09:33:05 [INFO] [scheduler] Job stack-scan completed (took 153ms)
2026-11-11T09:33:05.161104323Z 2026/11/11 09:33:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:33:05.669271854Z 2026/11/11 09:33:05 [INFO] [scheduler] Job agent-channel-health completed (took 508ms)
2026-11-11T09:34:04.906833303Z 2026/11/11 09:34:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:34:04.906854333Z 2026/11/11 09:34:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:34:05.160818043Z 2026/11/11 09:34:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:34:05.678849137Z 2026/11/11 09:34:05 [INFO] [scheduler] Job agent-channel-health completed (took 518ms)
2026-11-11T09:35:04.906633371Z 2026/11/11 09:35:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:35:04.907741413Z 2026/11/11 09:35:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:35:04.907932474Z 2026/11/11 09:35:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:35:05.054765892Z 2026/11/11 09:35:05 [INFO] [scheduler] Job stack-scan completed (took 147ms)
2026-11-11T09:35:05.160883591Z 2026/11/11 09:35:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:35:05.655921946Z 2026/11/11 09:35:05 [INFO] [scheduler] Job agent-channel-health completed (took 495ms)
2026-11-11T09:36:04.905976348Z 2026/11/11 09:36:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:36:04.906399446Z 2026/11/11 09:36:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:36:05.160918645Z 2026/11/11 09:36:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:36:05.670465732Z 2026/11/11 09:36:05 [INFO] [scheduler] Job agent-channel-health completed (took 510ms)
2026-11-11T09:37:04.908816714Z 2026/11/11 09:37:04 [INFO] [scheduler] Running job: backup-cache
2026-11-11T09:37:04.908842753Z 2026/11/11 09:37:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:37:04.908846751Z 2026/11/11 09:37:04 [INFO] [scheduler] Running job: fill-watch-interval
2026-11-11T09:37:04.908853283Z 2026/11/11 09:37:04 [INFO] [scheduler] Job fill-watch-interval completed (took 0s)
2026-11-11T09:37:04.908856278Z 2026/11/11 09:37:04 [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T09:37:04.908859134Z 2026/11/11 09:37:04 [INFO] [scheduler] Running job: system-health
2026-11-11T09:37:04.908861879Z 2026/11/11 09:37:04 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T09:37:04.908864625Z 2026/11/11 09:37:04 [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T09:37:04.908867419Z 2026/11/11 09:37:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:37:04.908870135Z 2026/11/11 09:37:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:37:05.092148252Z 2026/11/11 09:37:05 [INFO] [scheduler] Job stack-scan completed (took 185ms)
2026-11-11T09:37:05.119825034Z 2026/11/11 09:37:05 [INFO] [scheduler] Job docker-socket-users completed (took 214ms)
2026-11-11T09:37:05.147265501Z 2026/11/11 09:37:05 [INFO] [backup] Found 4 DB dump files across drives
2026-11-11T09:37:05.160883592Z 2026/11/11 09:37:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:37:05.272417437Z 2026/11/11 09:37:05 [INFO] [scheduler] Job system-health completed (took 366ms)
2026-11-11T09:37:05.318963655Z 2026/11/11 09:37:05 [INFO] [scheduler] Job backup-cache completed (took 413ms)
2026-11-11T09:37:05.787306309Z 2026/11/11 09:37:05 [INFO] [scheduler] Job agent-channel-health completed (took 627ms)
2026-11-11T09:38:04.914499105Z 2026/11/11 09:38:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:38:04.914531786Z 2026/11/11 09:38:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:38:05.160319781Z 2026/11/11 09:38:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:38:05.664415855Z 2026/11/11 09:38:05 [INFO] [scheduler] Job agent-channel-health completed (took 504ms)
2026-11-11T09:39:04.905994526Z 2026/11/11 09:39:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:39:04.906975628Z 2026/11/11 09:39:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:39:04.907017007Z 2026/11/11 09:39:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:39:05.009613137Z 2026/11/11 09:39:05 [INFO] [scheduler] Job stack-scan completed (took 104ms)
2026-11-11T09:39:05.160344104Z 2026/11/11 09:39:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:39:05.667819880Z 2026/11/11 09:39:05 [INFO] [scheduler] Job agent-channel-health completed (took 507ms)
2026-11-11T09:40:04.908299813Z 2026/11/11 09:40:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:40:04.908326914Z 2026/11/11 09:40:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:40:05.160622744Z 2026/11/11 09:40:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:40:05.646534628Z 2026/11/11 09:40:05 [INFO] [scheduler] Job agent-channel-health completed (took 486ms)
2026-11-11T09:41:04.908980390Z 2026/11/11 09:41:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:41:04.909024773Z 2026/11/11 09:41:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:41:04.909032287Z 2026/11/11 09:41:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:41:05.059955628Z 2026/11/11 09:41:05 [INFO] [scheduler] Job stack-scan completed (took 151ms)
2026-11-11T09:41:05.160896905Z 2026/11/11 09:41:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:41:05.673532120Z 2026/11/11 09:41:05 [INFO] [scheduler] Job agent-channel-health completed (took 513ms)
2026-11-11T09:42:04.909896065Z 2026/11/11 09:42:04 [INFO] [scheduler] Running job: backup-cache
2026-11-11T09:42:04.909933365Z 2026/11/11 09:42:04 [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T09:42:04.909939897Z 2026/11/11 09:42:04 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T09:42:04.909969123Z 2026/11/11 09:42:04 [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T09:42:04.909976496Z 2026/11/11 09:42:04 [INFO] [scheduler] Running job: system-health
2026-11-11T09:42:04.909996925Z 2026/11/11 09:42:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:42:04.910002816Z 2026/11/11 09:42:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:42:04.979062770Z 2026/11/11 09:42:04 [INFO] [backup] Found 4 DB dump files across drives
2026-11-11T09:42:05.146209070Z 2026/11/11 09:42:05 [INFO] [scheduler] Job docker-socket-users completed (took 238ms)
2026-11-11T09:42:05.160235922Z 2026/11/11 09:42:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:42:05.168840697Z 2026/11/11 09:42:05 [INFO] [scheduler] Job backup-cache completed (took 263ms)
2026-11-11T09:42:05.255446688Z 2026/11/11 09:42:05 [INFO] [scheduler] Job system-health completed (took 346ms)
2026-11-11T09:42:05.725984005Z 2026/11/11 09:42:05 [INFO] [scheduler] Job agent-channel-health completed (took 565ms)
2026-11-11T09:43:04.906280103Z 2026/11/11 09:43:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:43:04.907075686Z 2026/11/11 09:43:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:43:04.907105071Z 2026/11/11 09:43:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:43:05.069338426Z 2026/11/11 09:43:05 [INFO] [scheduler] Job stack-scan completed (took 162ms)
2026-11-11T09:43:05.160939298Z 2026/11/11 09:43:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:43:05.730904454Z 2026/11/11 09:43:05 [INFO] [scheduler] Job agent-channel-health completed (took 570ms)
2026-11-11T09:44:04.906796713Z 2026/11/11 09:44:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:44:04.908982730Z 2026/11/11 09:44:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:44:05.161369203Z 2026/11/11 09:44:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:44:05.660224386Z 2026/11/11 09:44:05 [INFO] [scheduler] Job agent-channel-health completed (took 499ms)
2026-11-11T09:45:04.906259715Z 2026/11/11 09:45:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:45:04.906294671Z 2026/11/11 09:45:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:45:04.909803566Z 2026/11/11 09:45:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:45:05.072780594Z 2026/11/11 09:45:05 [INFO] [scheduler] Job stack-scan completed (took 166ms)
2026-11-11T09:45:05.161751480Z 2026/11/11 09:45:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:45:05.720180919Z 2026/11/11 09:45:05 [INFO] [scheduler] Job agent-channel-health completed (took 559ms)
2026-11-11T09:46:04.906118553Z 2026/11/11 09:46:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:46:04.906319432Z 2026/11/11 09:46:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:46:05.160560375Z 2026/11/11 09:46:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:46:05.657381989Z 2026/11/11 09:46:05 [INFO] [scheduler] Job agent-channel-health completed (took 497ms)
2026-11-11T09:47:04.905860034Z 2026/11/11 09:47:04 [INFO] [scheduler] Running job: backup-cache
2026-11-11T09:47:04.907815135Z 2026/11/11 09:47:04 [INFO] [backup] Found 4 DB dump files across drives
2026-11-11T09:47:04.917507445Z 2026/11/11 09:47:04 [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T09:47:04.917539665Z 2026/11/11 09:47:04 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T09:47:04.917543462Z 2026/11/11 09:47:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:47:04.917562007Z 2026/11/11 09:47:04 [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T09:47:04.922234609Z 2026/11/11 09:47:04 [INFO] [scheduler] Running job: system-health
2026-11-11T09:47:04.926517074Z 2026/11/11 09:47:04 [INFO] [scheduler] Running job: fill-watch-interval
2026-11-11T09:47:04.929753685Z 2026/11/11 09:47:04 [INFO] [scheduler] Job fill-watch-interval completed (took 0s)
2026-11-11T09:47:04.934823557Z 2026/11/11 09:47:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:47:04.935067388Z 2026/11/11 09:47:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:47:04.935197834Z 2026/11/11 09:47:04 [INFO] [scheduler] Running job: hub-report
2026-11-11T09:47:05.161685018Z 2026/11/11 09:47:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:47:05.286530074Z 2026/11/11 09:47:05 [INFO] [scheduler] Job docker-socket-users completed (took 367ms)
2026-11-11T09:47:05.293348628Z 2026/11/11 09:47:05 [INFO] [scheduler] Job stack-scan completed (took 376ms)
2026-11-11T09:47:05.324248857Z 2026/11/11 09:47:05 [INFO] [sync] Catalog sync complete
2026-11-11T09:47:05.414292479Z 2026/11/11 09:47:05 [INFO] [scheduler] Job system-health completed (took 491ms)
2026-11-11T09:47:05.782668478Z 2026/11/11 09:47:05 [INFO] [scheduler] Job agent-channel-health completed (took 621ms)
2026-11-11T09:47:05.948318272Z 2026/11/11 09:47:05 [INFO] [scheduler] Job backup-cache completed (took 1.043s)
2026-11-11T09:47:06.674302522Z 2026/11/11 09:47:06 [INFO] [scheduler] Job hub-report completed (took 1.739s)
2026-11-11T09:48:04.906768544Z 2026/11/11 09:48:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:48:04.906796998Z 2026/11/11 09:48:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:48:05.160288997Z 2026/11/11 09:48:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:48:05.646123176Z 2026/11/11 09:48:05 [INFO] [scheduler] Job agent-channel-health completed (took 486ms)
2026-11-11T09:49:04.907042614Z 2026/11/11 09:49:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:49:04.908892617Z 2026/11/11 09:49:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:49:04.908914979Z 2026/11/11 09:49:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:49:05.049545998Z 2026/11/11 09:49:05 [INFO] [scheduler] Job stack-scan completed (took 142ms)
2026-11-11T09:49:05.161155636Z 2026/11/11 09:49:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:49:05.674796784Z 2026/11/11 09:49:05 [INFO] [scheduler] Job agent-channel-health completed (took 514ms)
2026-11-11T09:50:04.906998145Z 2026/11/11 09:50:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:50:04.907051226Z 2026/11/11 09:50:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:50:05.161225283Z 2026/11/11 09:50:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:50:05.673788877Z 2026/11/11 09:50:05 [INFO] [scheduler] Job agent-channel-health completed (took 513ms)
2026-11-11T09:51:04.906823229Z 2026/11/11 09:51:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:51:04.906868464Z 2026/11/11 09:51:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:51:04.906874867Z 2026/11/11 09:51:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:51:05.023425543Z 2026/11/11 09:51:05 [INFO] [scheduler] Job stack-scan completed (took 117ms)
2026-11-11T09:51:05.160588625Z 2026/11/11 09:51:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:51:05.650758119Z 2026/11/11 09:51:05 [INFO] [scheduler] Job agent-channel-health completed (took 490ms)
2026-11-11T09:52:04.906081037Z 2026/11/11 09:52:04 [INFO] [scheduler] Running job: backup-cache
2026-11-11T09:52:04.906372236Z 2026/11/11 09:52:04 [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T09:52:04.906387466Z 2026/11/11 09:52:04 [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T09:52:04.906512150Z 2026/11/11 09:52:04 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T09:52:04.906527740Z 2026/11/11 09:52:04 [INFO] [scheduler] Running job: system-health
2026-11-11T09:52:04.908160293Z 2026/11/11 09:52:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:52:04.908185821Z 2026/11/11 09:52:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:52:04.958138891Z 2026/11/11 09:52:04 [INFO] [backup] Found 4 DB dump files across drives
2026-11-11T09:52:05.128735957Z 2026/11/11 09:52:05 [INFO] [scheduler] Job backup-cache completed (took 222ms)
2026-11-11T09:52:05.158951424Z 2026/11/11 09:52:05 [INFO] [scheduler] Job docker-socket-users completed (took 253ms)
2026-11-11T09:52:05.161798499Z 2026/11/11 09:52:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:52:05.248963214Z 2026/11/11 09:52:05 [INFO] [scheduler] Job system-health completed (took 342ms)
2026-11-11T09:52:05.703048703Z 2026/11/11 09:52:05 [INFO] [scheduler] Job agent-channel-health completed (took 542ms)
2026-11-11T09:53:04.906946847Z 2026/11/11 09:53:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:53:04.906971574Z 2026/11/11 09:53:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:53:04.906984368Z 2026/11/11 09:53:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:53:05.070103264Z 2026/11/11 09:53:05 [INFO] [scheduler] Job stack-scan completed (took 164ms)
2026-11-11T09:53:05.161115956Z 2026/11/11 09:53:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:53:05.662560556Z 2026/11/11 09:53:05 [INFO] [scheduler] Job agent-channel-health completed (took 501ms)
2026-11-11T09:54:04.906582146Z 2026/11/11 09:54:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:54:04.906901760Z 2026/11/11 09:54:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:54:05.161094072Z 2026/11/11 09:54:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:54:05.655618268Z 2026/11/11 09:54:05 [INFO] [scheduler] Job agent-channel-health completed (took 495ms)
2026-11-11T09:55:04.908006862Z 2026/11/11 09:55:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:55:04.908089568Z 2026/11/11 09:55:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:55:04.908099978Z 2026/11/11 09:55:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:55:05.076915529Z 2026/11/11 09:55:05 [INFO] [scheduler] Job stack-scan completed (took 171ms)
2026-11-11T09:55:05.161220797Z 2026/11/11 09:55:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:55:05.676659488Z 2026/11/11 09:55:05 [INFO] [scheduler] Job agent-channel-health completed (took 515ms)
2026-11-11T09:56:04.906822968Z 2026/11/11 09:56:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:56:04.906848386Z 2026/11/11 09:56:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:56:05.160840460Z 2026/11/11 09:56:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:56:05.647325257Z 2026/11/11 09:56:05 [INFO] [scheduler] Job agent-channel-health completed (took 487ms)
2026-11-11T09:57:04.906004970Z 2026/11/11 09:57:04 [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T09:57:04.906045016Z 2026/11/11 09:57:04 [INFO] [scheduler] Running job: system-health
2026-11-11T09:57:04.906050737Z 2026/11/11 09:57:04 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T09:57:04.906055866Z 2026/11/11 09:57:04 [INFO] [scheduler] Running job: fill-watch-interval
2026-11-11T09:57:04.906469476Z 2026/11/11 09:57:04 [INFO] [scheduler] Running job: backup-cache
2026-11-11T09:57:04.906490947Z 2026/11/11 09:57:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:57:04.906788419Z 2026/11/11 09:57:04 [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T09:57:04.907257544Z 2026/11/11 09:57:04 [INFO] [scheduler] Job fill-watch-interval completed (took 1ms)
2026-11-11T09:57:04.907261371Z 2026/11/11 09:57:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:57:04.907264868Z 2026/11/11 09:57:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:57:05.087667456Z 2026/11/11 09:57:05 [INFO] [scheduler] Job stack-scan completed (took 179ms)
2026-11-11T09:57:05.087754741Z 2026/11/11 09:57:05 [INFO] [backup] Found 4 DB dump files across drives
2026-11-11T09:57:05.099198416Z 2026/11/11 09:57:05 [INFO] [scheduler] Job docker-socket-users completed (took 193ms)
2026-11-11T09:57:05.160555308Z 2026/11/11 09:57:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:57:05.240108604Z 2026/11/11 09:57:05 [INFO] [scheduler] Job system-health completed (took 335ms)
2026-11-11T09:57:05.259868099Z 2026/11/11 09:57:05 [INFO] [scheduler] Job backup-cache completed (took 353ms)
2026-11-11T09:57:05.723041564Z 2026/11/11 09:57:05 [INFO] [scheduler] Job agent-channel-health completed (took 563ms)
2026-11-11T09:58:04.906409622Z 2026/11/11 09:58:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:58:04.906432204Z 2026/11/11 09:58:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:58:05.161208860Z 2026/11/11 09:58:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:58:05.669828277Z 2026/11/11 09:58:05 [INFO] [scheduler] Job agent-channel-health completed (took 509ms)
2026-11-11T09:59:04.906890675Z 2026/11/11 09:59:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T09:59:04.908525281Z 2026/11/11 09:59:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T09:59:04.908548084Z 2026/11/11 09:59:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T09:59:05.044093351Z 2026/11/11 09:59:05 [INFO] [scheduler] Job stack-scan completed (took 137ms)
2026-11-11T09:59:05.161227580Z 2026/11/11 09:59:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T09:59:05.665779027Z 2026/11/11 09:59:05 [INFO] [scheduler] Job agent-channel-health completed (took 505ms)
2026-11-11T10:00:04.906894655Z 2026/11/11 10:00:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T10:00:04.907004353Z 2026/11/11 10:00:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T10:00:05.160977310Z 2026/11/11 10:00:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T10:00:05.672821867Z 2026/11/11 10:00:05 [INFO] [scheduler] Job agent-channel-health completed (took 512ms)
2026-11-11T10:01:04.906496615Z 2026/11/11 10:01:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T10:01:04.906855773Z 2026/11/11 10:01:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T10:01:04.906872295Z 2026/11/11 10:01:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T10:01:05.039623186Z 2026/11/11 10:01:05 [INFO] [scheduler] Job stack-scan completed (took 133ms)
2026-11-11T10:01:05.160440789Z 2026/11/11 10:01:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T10:01:05.640544930Z 2026/11/11 10:01:05 [INFO] [scheduler] Job agent-channel-health completed (took 480ms)
2026-11-11T10:02:04.907854323Z 2026/11/11 10:02:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T10:02:04.907947148Z 2026/11/11 10:02:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T10:02:04.907955415Z 2026/11/11 10:02:04 [INFO] [scheduler] Running job: backup-cache
2026-11-11T10:02:04.907959652Z 2026/11/11 10:02:04 [INFO] [scheduler] Running job: offsite-credential-retry
2026-11-11T10:02:04.907963460Z 2026/11/11 10:02:04 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026-11-11T10:02:04.907967127Z 2026/11/11 10:02:04 [INFO] [scheduler] Running job: hub-report
2026-11-11T10:02:04.907973949Z 2026/11/11 10:02:04 [INFO] [scheduler] Running job: docker-socket-users
2026-11-11T10:02:04.908108814Z 2026/11/11 10:02:04 [INFO] [scheduler] Running job: system-health
2026-11-11T10:02:04.975842475Z 2026/11/11 10:02:04 [INFO] [backup] Found 4 DB dump files across drives
2026-11-11T10:02:04.976149264Z 2026/11/11 10:02:04 [INFO] [sync] Catalog sync complete
2026-11-11T10:02:05.160870204Z 2026/11/11 10:02:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T10:02:05.206269368Z 2026/11/11 10:02:05 [INFO] [scheduler] Job docker-socket-users completed (took 298ms)
2026-11-11T10:02:05.289480299Z 2026/11/11 10:02:05 [INFO] [scheduler] Job system-health completed (took 382ms)
2026-11-11T10:02:05.774210864Z 2026/11/11 10:02:05 [INFO] [scheduler] Job backup-cache completed (took 866ms)
2026-11-11T10:02:05.823947156Z 2026/11/11 10:02:05 [INFO] [scheduler] Job agent-channel-health completed (took 663ms)
2026-11-11T10:02:06.515959397Z 2026/11/11 10:02:06 [INFO] [scheduler] Job hub-report completed (took 1.608s)
2026-11-11T10:03:04.908587389Z 2026/11/11 10:03:04 [INFO] [scheduler] Running job: stack-scan
2026-11-11T10:03:04.908621484Z 2026/11/11 10:03:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T10:03:04.908678831Z 2026/11/11 10:03:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T10:03:05.045884214Z 2026/11/11 10:03:05 [INFO] [scheduler] Job stack-scan completed (took 138ms)
2026-11-11T10:03:05.160800565Z 2026/11/11 10:03:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T10:03:05.663391911Z 2026/11/11 10:03:05 [INFO] [scheduler] Job agent-channel-health completed (took 503ms)
2026-11-11T10:04:04.909163911Z 2026/11/11 10:04:04 [INFO] [scheduler] Running job: hold-report
2026-11-11T10:04:04.909233302Z 2026/11/11 10:04:04 [INFO] [scheduler] Job hold-report completed (took 0s)
2026-11-11T10:04:05.160865681Z 2026/11/11 10:04:05 [INFO] [scheduler] Running job: agent-channel-health
2026-11-11T10:04:05.701032100Z 2026/11/11 10:04:05 [INFO] [scheduler] Job agent-channel-health completed (took 541ms)
RC=0
@@ -0,0 +1,227 @@
# guest-night tester-1 taken 2026-11-11T10:09:26Z
log first line: 2026-11-10T15:17:04.380509770Z
log last line: 2026-11-11T10:09:24.943219455Z
=== scheduler jobs (UTC): runs, first start, last start, last three "took" ===
agent-channel-health runs=1132 first=2026-11-10T15:18:05 last=2026-11-11T10:09:05 [completed (took 595ms)] [completed (took 510ms)] [completed (took 506ms)]
backup-cache runs=226 first=2026-11-10T15:22:04 last=2026-11-11T10:07:04 [completed (took 353ms)] [completed (took 866ms)] [completed (took 421ms)]
conversion-copy-release runs=18 first=2026-11-10T16:17:04 last=2026-11-11T09:17:04 [completed (took 234ms)] [completed (took 291ms)] [completed (took 62ms)]
db-dump runs=1 first=2026-11-11T01:30:00 last=2026-11-11T01:30:00 [completed (took 40.147s)]
disk-health-check runs=18 first=2026-11-10T16:17:05 last=2026-11-11T09:17:05 [completed (took 1.388s)] [completed (took 826ms)] [completed (took 1.142s)]
docker-socket-users runs=226 first=2026-11-10T15:22:04 last=2026-11-11T10:07:04 [completed (took 193ms)] [completed (took 298ms)] [completed (took 215ms)]
fill-watch runs=1 first=2026-11-11T02:30:00 last=2026-11-11T02:30:00 [completed (took 0s)]
fill-watch-interval runs=113 first=2026-11-10T15:27:04 last=2026-11-11T10:07:04 [completed (took 0s)] [completed (took 1ms)] [completed (took 2ms)]
geo-verify runs=3 first=2026-11-10T21:17:04 last=2026-11-11T09:17:04 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)]
hold-report runs=1132 first=2026-11-10T15:18:04 last=2026-11-11T10:09:04 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)]
hub-report runs=75 first=2026-11-10T15:32:04 last=2026-11-11T10:02:04 [completed (took 1.655s)] [completed (took 1.739s)] [completed (took 1.608s)]
metrics-prune runs=1 first=2026-11-11T03:00:00 last=2026-11-11T03:00:00 [completed (took 125ms)]
offbox-backup runs=1 first=2026-11-11T03:15:00 last=2026-11-11T03:15:00 [completed (took 58.568s)]
offsite-abandon-sweep runs=1 first=2026-11-11T04:10:00 last=2026-11-11T04:10:00 [completed (took 0s)]
offsite-credential-retry runs=226 first=2026-11-10T15:22:04 last=2026-11-11T10:07:04 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)]
offsite-integrity runs=1 first=2026-11-11T05:00:00 last=2026-11-11T05:00:00 [completed (took 0s)]
offsite-proof runs=1 first=2026-11-11T04:30:00 last=2026-11-11T04:30:00 [completed (took 5.06s)]
restore-copy-sweep runs=3 first=2026-11-10T21:17:04 last=2026-11-11T09:17:04 [completed (took 5ms)] [completed (took 0s)] [completed (took 10ms)]
selfupdate-check runs=3 first=2026-11-10T21:17:04 last=2026-11-11T09:17:04 [completed (took 49ms)] [completed (took 119ms)] [completed (took 60ms)]
stack-scan runs=566 first=2026-11-10T15:19:04 last=2026-11-11T10:09:04 [completed (took 221ms)] [completed (took 216ms)] [completed (took 138ms)]
system-health runs=226 first=2026-11-10T15:22:04 last=2026-11-11T10:07:04 [completed (took 335ms)] [completed (took 382ms)] [completed (took 335ms)]
tier2-backup runs=1 first=2026-11-11T02:30:00 last=2026-11-11T02:30:00 [completed (took 15.055s)]
=== jobs that ran fewer than 30 times: every start and end (UTC) ===
2026/11/10 16:17:04 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/10 16:17:04 [INFO] [scheduler] Job conversion-copy-release completed (took 1ms)
2026/11/10 16:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/10 16:17:05 [INFO] [scheduler] Job disk-health-check completed (took 763ms)
2026/11/10 17:17:04 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/10 17:17:04 [INFO] [scheduler] Job conversion-copy-release completed (took 28ms)
2026/11/10 17:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/10 17:17:06 [INFO] [scheduler] Job disk-health-check completed (took 888ms)
2026/11/10 18:17:04 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/10 18:17:04 [INFO] [scheduler] Job conversion-copy-release completed (took 62ms)
2026/11/10 18:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/10 18:17:06 [INFO] [scheduler] Job disk-health-check completed (took 1.2s)
2026/11/10 19:17:04 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/10 19:17:04 [INFO] [scheduler] Job conversion-copy-release completed (took 1ms)
2026/11/10 19:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/10 19:17:06 [INFO] [scheduler] Job disk-health-check completed (took 903ms)
2026/11/10 20:17:04 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/10 20:17:04 [INFO] [scheduler] Job conversion-copy-release completed (took 34ms)
2026/11/10 20:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/10 20:17:06 [INFO] [scheduler] Job disk-health-check completed (took 1.147s)
2026/11/10 21:17:04 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/10 21:17:04 [INFO] [scheduler] Job conversion-copy-release completed (took 1ms)
2026/11/10 21:17:04 [INFO] [scheduler] Running job: selfupdate-check
2026/11/10 21:17:04 [INFO] [scheduler] Running job: restore-copy-sweep
2026/11/10 21:17:04 [INFO] [scheduler] Job restore-copy-sweep completed (took 5ms)
2026/11/10 21:17:04 [INFO] [scheduler] Running job: geo-verify
2026/11/10 21:17:04 [INFO] [scheduler] Job geo-verify completed (took 0s)
2026/11/10 21:17:04 [INFO] [scheduler] Job selfupdate-check completed (took 49ms)
2026/11/10 21:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/10 21:17:06 [INFO] [scheduler] Job disk-health-check completed (took 1.208s)
2026/11/10 22:17:04 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/10 22:17:04 [INFO] [scheduler] Job conversion-copy-release completed (took 1ms)
2026/11/10 22:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/10 22:17:05 [INFO] [scheduler] Job disk-health-check completed (took 814ms)
2026/11/10 23:17:04 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/10 23:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/10 23:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 264ms)
2026/11/10 23:17:05 [INFO] [scheduler] Job disk-health-check completed (took 774ms)
2026/11/11 00:17:04 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 00:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 249ms)
2026/11/11 00:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 00:17:06 [INFO] [scheduler] Job disk-health-check completed (took 866ms)
2026/11/11 01:17:04 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 01:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 01:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 321ms)
2026/11/11 01:17:05 [INFO] [scheduler] Job disk-health-check completed (took 787ms)
2026/11/11 01:30:00 [INFO] [scheduler] Running job: db-dump
2026/11/11 01:30:40 [INFO] [scheduler] Job db-dump completed (took 40.147s)
2026/11/11 02:17:04 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 02:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 221ms)
2026/11/11 02:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 02:17:06 [INFO] [scheduler] Job disk-health-check completed (took 1.182s)
2026/11/11 02:30:00 [INFO] [scheduler] Running job: tier2-backup
2026/11/11 02:30:00 [INFO] [scheduler] Running job: fill-watch
2026/11/11 02:30:00 [INFO] [scheduler] Job fill-watch completed (took 0s)
2026/11/11 02:30:15 [INFO] [scheduler] Job tier2-backup completed (took 15.055s)
2026/11/11 03:00:00 [INFO] [scheduler] Running job: metrics-prune
2026/11/11 03:00:00 [INFO] [scheduler] Job metrics-prune completed (took 125ms)
2026/11/11 03:15:00 [INFO] [scheduler] Running job: offbox-backup
2026/11/11 03:15:58 [INFO] [scheduler] Job offbox-backup completed (took 58.568s)
2026/11/11 03:17:04 [INFO] [scheduler] Running job: selfupdate-check
2026/11/11 03:17:04 [INFO] [scheduler] Running job: restore-copy-sweep
2026/11/11 03:17:04 [INFO] [scheduler] Job restore-copy-sweep completed (took 0s)
2026/11/11 03:17:04 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 03:17:04 [INFO] [scheduler] Running job: geo-verify
2026/11/11 03:17:04 [INFO] [scheduler] Job geo-verify completed (took 0s)
2026/11/11 03:17:05 [INFO] [scheduler] Job selfupdate-check completed (took 119ms)
2026/11/11 03:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 03:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 454ms)
2026/11/11 03:17:06 [INFO] [scheduler] Job disk-health-check completed (took 1.115s)
2026/11/11 04:10:00 [INFO] [scheduler] Running job: offsite-abandon-sweep
2026/11/11 04:10:00 [INFO] [scheduler] Job offsite-abandon-sweep completed (took 0s)
2026/11/11 04:17:04 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 04:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 04:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 278ms)
2026/11/11 04:17:06 [INFO] [scheduler] Job disk-health-check completed (took 848ms)
2026/11/11 04:30:00 [INFO] [scheduler] Running job: offsite-proof
2026/11/11 04:30:05 [INFO] [scheduler] Job offsite-proof completed (took 5.06s)
2026/11/11 05:00:00 [INFO] [scheduler] Running job: offsite-integrity
2026/11/11 05:00:00 [INFO] [scheduler] Job offsite-integrity completed (took 0s)
2026/11/11 05:17:04 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 05:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 05:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 304ms)
2026/11/11 05:17:05 [INFO] [scheduler] Job disk-health-check completed (took 806ms)
2026/11/11 06:17:04 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 06:17:04 [INFO] [scheduler] Job conversion-copy-release completed (took 52ms)
2026/11/11 06:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 06:17:06 [INFO] [scheduler] Job disk-health-check completed (took 827ms)
2026/11/11 07:17:04 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 07:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 234ms)
2026/11/11 07:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 07:17:06 [INFO] [scheduler] Job disk-health-check completed (took 1.388s)
2026/11/11 08:17:04 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 08:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 08:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 291ms)
2026/11/11 08:17:05 [INFO] [scheduler] Job disk-health-check completed (took 826ms)
2026/11/11 09:17:04 [INFO] [scheduler] Running job: selfupdate-check
2026/11/11 09:17:04 [INFO] [scheduler] Running job: conversion-copy-release
2026/11/11 09:17:04 [INFO] [scheduler] Running job: geo-verify
2026/11/11 09:17:04 [INFO] [scheduler] Job geo-verify completed (took 0s)
2026/11/11 09:17:04 [INFO] [scheduler] Running job: restore-copy-sweep
2026/11/11 09:17:04 [INFO] [scheduler] Job restore-copy-sweep completed (took 10ms)
2026/11/11 09:17:04 [INFO] [scheduler] Job selfupdate-check completed (took 60ms)
2026/11/11 09:17:04 [INFO] [scheduler] Job conversion-copy-release completed (took 62ms)
2026/11/11 09:17:05 [INFO] [scheduler] Running job: disk-health-check
2026/11/11 09:17:06 [INFO] [scheduler] Job disk-health-check completed (took 1.142s)
=== night lines 19:00-08:00 UTC, INFO/WARN/ERROR, backup words (first 2 per message shape) ===
2026/11/10 19:02:04 [INFO] [backup] Found 4 DB dump files across drives
2026/11/10 19:02:05 [INFO] [backup] Discovered 2 databases
2026/11/10 19:02:05 [INFO] [backup] Discovered app data: 3 apps
2026/11/10 19:02:05 [INFO] [monitor] Health check: status=ok
2026/11/10 19:02:05 [INFO] [backup] Backup status cache refreshed
2026/11/10 19:02:06 [INFO] [monitor] Health check: status=ok
2026/11/10 19:07:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 19:07:04 [INFO] [backup] Found 4 DB dump files across drives
2026/11/10 19:07:05 [INFO] [backup] Discovered 2 databases
2026/11/10 19:07:05 [INFO] [backup] Discovered app data: 3 apps
2026/11/10 19:07:05 [INFO] [backup] Backup status cache refreshed
2026/11/10 19:08:05 [INFO] [deadapp] check alive: 460 scans since boot, 3 deployed app(s) evaluated, 0 currently down
2026/11/10 19:17:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/11/10 19:17:06 [INFO] [web] disk-health check complete: 0 disk(s) evaluated, 0 alert(s)
2026/11/10 19:18:05 [INFO] [deadapp] check alive: 480 scans since boot, 3 deployed app(s) evaluated, 0 currently down
2026/11/10 20:17:06 [INFO] [web] disk-health check complete: 0 disk(s) evaluated, 0 alert(s)
2026/11/11 01:30:00 [INFO] [backup] Starting database dump run
2026/11/11 01:30:00 [INFO] [backup] Discovered 2 database(s): paperless-postgres(postgres), bookstack-db(mariadb)
2026/11/11 01:30:00 [INFO] [backup] DB dump: paperless-postgres → paperless-ngx-postgres.sql (557.9 KB, 572ms, 75 tables)
2026/11/11 01:30:01 [INFO] [backup] DB dump: bookstack-db → bookstack-mariadb.sql (57.9 KB, 426ms, 41 tables)
2026/11/11 01:30:01 [INFO] [backup] Stopping bookstack for safe volume dump
2026/11/11 01:30:04 [INFO] [busymark] busy: a backup, a restore or an off-site run is running — an operator reboot of the box is refused until it ends
2026/11/11 01:30:06 [INFO] [backup] Volume dump: bookstack/bookstack_bookstack_config → 3.3 MB
2026/11/11 01:30:07 [INFO] [backup] Volume dump: bookstack/bookstack_bookstack_db_data → 153.4 MB
2026/11/11 01:30:07 [INFO] [backup] Restarting bookstack after volume dump
2026/11/11 01:30:13 [INFO] [backup] Stopping paperless-ngx for safe volume dump
2026/11/11 01:30:23 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_data → 17.4 MB
2026/11/11 01:30:25 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_postgres_data → 70.4 MB
2026/11/11 01:30:25 [INFO] [backup] Restarting paperless-ngx after volume dump
2026/11/11 01:30:37 [INFO] [backup] Stopping privatebin for safe volume dump
2026/11/11 01:30:39 [INFO] [backup] Volume dump: privatebin/privatebin_privatebin_data → 8.5 KB
2026/11/11 01:30:39 [INFO] [backup] Restarting privatebin after volume dump
2026/11/11 01:30:39 [INFO] [backup] App-data backup completed: 2 databases (615.8 KB total), 3 volume dump(s) (39.942s)
2026/11/11 01:30:39 [INFO] [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1)
2026/11/11 01:30:39 [INFO] [backup] Recovery unit captured for paperless-ngx → /mnt/felhom-drives/adatlemez/backups/primary/paperless-ngx (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1)
2026/11/11 01:30:39 [INFO] [backup] Recovery unit captured for privatebin → /mnt/sys_drive/felhom-data/backups/primary/privatebin (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/11/11 01:30:40 [INFO] Event pushed: db_dump_completed (info) [hu-only] — Adatbázis mentés elkészült
2026/11/11 01:30:49 [INFO] [busymark] idle again (was: a backup, a restore or an off-site run is running) — an operator reboot of the box is no longer refused for it
2026/11/11 02:30:04 [INFO] [backup] Tier 2 copied bookstack → /mnt/felhom-drives/adatlemez/backups/secondary/bookstack (156.8 MB, 0 leg(s), 3s)
2026/11/11 02:30:10 [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sys_drive/felhom-data/backups/secondary/paperless-ngx (96.1 MB, 1 leg(s), 5s) [SSD: state-only]
2026/11/11 02:30:15 [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/adatlemez/backups/secondary/privatebin (22.7 KB, 0 leg(s), 3s)
2026/11/11 02:30:15 [INFO] [backup] Tier 2 run complete: 3 app(s) processed (incl. volume-only — F6)
2026/11/11 03:00:00 [INFO] Pruned 15840 old metric rows
2026/11/11 03:15:00 [INFO] [offbox] backup run started (9 app(s) toggled)
2026/11/11 03:15:00 [INFO] [backup] Starting database dump run
2026/11/11 03:15:00 [INFO] [backup] Discovered 2 database(s): paperless-postgres(postgres), bookstack-db(mariadb)
2026/11/11 03:15:00 [INFO] [backup] DB dump: paperless-postgres → paperless-ngx-postgres.sql (560.7 KB, 457ms, 75 tables)
2026/11/11 03:15:00 [INFO] [backup] DB dump: bookstack-db → bookstack-mariadb.sql (57.9 KB, 317ms, 41 tables)
2026/11/11 03:15:00 [INFO] [backup] Stopping bookstack for safe volume dump
2026/11/11 03:15:04 [INFO] [busymark] busy: a backup, a restore or an off-site run is running — an operator reboot of the box is refused until it ends
2026/11/11 03:15:06 [INFO] [backup] Restarting bookstack after volume dump
2026/11/11 03:15:12 [INFO] [backup] Stopping paperless-ngx for safe volume dump
2026/11/11 03:15:24 [INFO] [backup] Restarting paperless-ngx after volume dump
2026/11/11 03:15:35 [INFO] [backup] Stopping privatebin for safe volume dump
2026/11/11 03:15:36 [INFO] [backup] Volume dump: privatebin/privatebin_privatebin_data → 8.5 KB
2026/11/11 03:15:36 [INFO] [backup] Restarting privatebin after volume dump
2026/11/11 03:15:37 [INFO] [backup] App-data backup completed: 2 databases (618.6 KB total), 3 volume dump(s) (37.935s)
2026/11/11 03:15:37 [INFO] [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1)
2026/11/11 03:15:37 [INFO] [backup] Recovery unit captured for paperless-ngx → /mnt/felhom-drives/adatlemez/backups/primary/paperless-ngx (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1)
2026/11/11 03:15:37 [INFO] [backup] Recovery unit captured for privatebin → /mnt/sys_drive/felhom-data/backups/primary/privatebin (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/11/11 03:15:38 [INFO] [offbox] pre-push dump leg completed in 38.077s — snapshot pair is coherent
2026/11/11 03:15:41 [WARN] [offbox] prowlarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable)
2026/11/11 03:15:41 [WARN] [offbox] sonarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable)
2026/11/11 03:15:41 [WARN] [offbox] transmission: not deployed — offsite push is unit-only (mandatory userdata not resolvable)
2026/11/11 03:15:41 [WARN] [offbox] jellyfin: not deployed — offsite push is unit-only (mandatory userdata not resolvable)
2026/11/11 03:15:41 [WARN] [offbox] plex: not deployed — offsite push is unit-only (mandatory userdata not resolvable)
2026/11/11 03:15:41 [WARN] [offbox] radarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable)
2026/11/11 03:15:45 [INFO] [offbox] backed up bookstack (/mnt/sys_drive/felhom-data/backups/primary/bookstack, 0 mandatory path(s))
2026/11/11 03:15:45 [WARN] [offbox] prowlarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable)
2026/11/11 03:15:47 [INFO] [offbox] backed up prowlarr (/mnt/sys_drive/felhom-data/backups/primary/prowlarr, 0 mandatory path(s))
2026/11/11 03:15:47 [WARN] [offbox] sonarr: no recovery unit found on any connected drive — skipping
2026/11/11 03:15:47 [WARN] [offbox] transmission: no recovery unit found on any connected drive — skipping
2026/11/11 03:15:47 [WARN] [offbox] jellyfin: no recovery unit found on any connected drive — skipping
2026/11/11 03:15:50 [INFO] [offbox] backed up paperless-ngx (/mnt/felhom-drives/adatlemez/backups/primary/paperless-ngx, 1 mandatory path(s))
2026/11/11 03:15:50 [WARN] [offbox] plex: no recovery unit found on any connected drive — skipping
2026/11/11 03:15:52 [INFO] [offbox] backed up privatebin (/mnt/sys_drive/felhom-data/backups/primary/privatebin, 0 mandatory path(s))
2026/11/11 03:15:52 [WARN] [offbox] radarr: no recovery unit found on any connected drive — skipping
2026/11/11 03:15:54 [INFO] [offbox] retention skipped (after-run): no clean-up window now (not due (last window 2026-11-09T03:15:56Z)) — nothing deleted (decision 68)
2026/11/11 03:15:58 [INFO] [offbox] backup OK: 4 app(s) backed up, 5 skipped (no unit), 71 snapshot(s), 55s
2026/11/11 03:15:58 [INFO] [update-leg] started (after-offsite): window 02:30, no step starts at or after 07:30
2026/11/11 03:15:58 [INFO] [update-leg] update leg (after-offsite): done=0 undone=0 held=0 failed=0 skipped=0 in 0s [skipped: ]
2026/11/11 03:16:04 [INFO] [busymark] idle again (was: a backup, a restore or an off-site run is running) — an operator reboot of the box is no longer refused for it
2026/11/11 04:30:05 [INFO] [offbox] proof: bookstack PASSED on snapshot d4dfbe69 in 3.287s — the backup holds what this app should have
2026/11/11 04:32:09 [INFO] [quiesce] backup due on 1 tier(s) — quiescing 3 stack(s): [bookstack paperless-ngx privatebin]
2026/11/11 04:32:22 [INFO] [quiesce] tier local: backup job backup-9201-1794371542405626888 started — polling
2026/11/11 04:32:32 [INFO] [quiesce] tier local: job backup-9201-1794371542405626888 snapshotted — resuming app early (8B.2); the upload continues with the apps running
2026/11/11 04:32:32 [INFO] [quiesce] unquiescing (snapshotted (early resume)): restarting 3 stack(s)
2026/11/11 04:35:11 [INFO] [quiesce] tier local: backup job backup-9201-1794371542405626888 done
2026/11/11 05:00:00 [INFO] [offbox] integrity: not due (last successful check 48h0m0s ago, at 2026-11-09 05:00) — nothing was run
RC=0
@@ -0,0 +1,682 @@
# host-clock tester-1 taken 2026-11-11T10:03:57Z from DooPlex
=== in-flight ===
checked
=== clock ===
utc 2026-11-11T10:03:57.568914293Z
local 2026-11-11T11:03:57 CET +0100
/usr/share/zoneinfo/Europe/Budapest
Timezone=Europe/Budapest
LocalRTC=no
CanNTP=yes
NTP=yes
NTPSynchronized=yes
TimeUSec=Wed 2026-11-11 11:03:57 CET
RTCTimeUSec=Wed 2026-11-11 11:03:57 CET
Local time: Wed 2026-11-11 11:03:57 CET
Universal time: Wed 2026-11-11 10:03:57 UTC
RTC time: Wed 2026-11-11 10:03:57
Time zone: Europe/Budapest (CET, +0100)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
=== who keeps the time ===
chrony: active
enabled: enabled
chronyd: active
enabled: alias
systemd-timesyncd: inactive
enabled: not-found
ntp: inactive
enabled: not-found
ntpsec: inactive
enabled: not-found
openntpd: inactive
enabled: not-found
Reference ID : C2255A28 (194.37.90.40)
Stratum : 2
Ref time (UTC) : Wed Nov 11 10:00:19 2026
System time : 0.000132173 seconds slow of NTP time
Last offset : -0.000024902 seconds
RMS offset : 0.000064157 seconds
Frequency : 12.931 ppm slow
Residual freq : +0.000 ppm
Skew : 0.013 ppm
Root delay : 0.014427286 seconds
Root dispersion : 0.000787278 seconds
Update interval : 1044.3 seconds
Leap status : Normal
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^+ 37.221.209.24 2 10 377 315 -1626us[-1651us] +/- 15ms
^+ 193.41.47.85 2 10 377 66 -1695us[-1695us] +/- 8973us
^+ 193.227.197.2 3 10 377 699 -4821us[-4846us] +/- 19ms
^* 194.37.90.40 1 10 377 218 -3822us[-3847us] +/- 13ms
Failed to query server: The name org.freedesktop.timesync1 was not provided by any .service files
pool 2.debian.pool.ntp.org iburst
sourcedir /run/chrony-dhcp
sourcedir /etc/chrony/sources.d
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/chrony.drift
ntsdumpdir /var/lib/chrony
logdir /var/log/chrony
maxupdateskew 100.0
rtcsync
makestep 1 3
leapseclist /usr/share/zoneinfo/leap-seconds.list
confdir /etc/chrony/conf.d
/etc/chrony/conf.d:
README
/etc/chrony/sources.d:
README
=== start with no time ===
ls: cannot access '/var/lib/systemd/timesync/clock': No such file or directory
/var/lib/chrony/:
total 4
-rw-r--r-- 1 _chrony _chrony 42 2026-11-11 11:00:19.119554513 +0100 chrony.drift
2026-11-11 11:03:57.838270+01:00
1794391438
1794391437
=== boots ===
2026-11-10 15:43:22
-5 c06cfce08e2d470b9e32d4f7e2c5f94b Sat 2026-10-24 19:44:06 CEST Thu 2026-10-29 05:09:31 CET
-4 f8210297fa3f492eb5d6a1cc5844dc6b Thu 2026-10-29 05:10:44 CET Tue 2026-11-10 02:53:06 CET
-3 bf6ac6018a8e4819beed5a725e3967f2 Tue 2026-11-10 02:54:30 CET Tue 2026-11-10 13:56:23 CET
-2 a506eb94c8f54a17ba4918d0cf981676 Tue 2026-11-10 13:57:37 CET Tue 2026-11-10 15:32:20 CET
-1 9d8a5ed3f79b4d85ae816a1d308f379e Tue 2026-11-10 15:33:34 CET Tue 2026-11-10 15:42:14 CET
0 6407d71b1d894748865549c6de8cf61b Tue 2026-11-10 15:43:29 CET Wed 2026-11-11 11:04:02 CET
=== timers ===
NEXT LEFT LAST PASSED UNIT ACTIVATES
Wed 2026-11-11 11:04:57 CET 54s Wed 2026-11-11 11:03:57 CET 5s ago felhom-mgmt-watchdog.timer felhom-mgmt-watchdog.service
Wed 2026-11-11 11:59:43 CET 55min Wed 2026-11-11 10:59:43 CET 4min 19s ago felhom-crash-guard-check.timer felhom-crash-guard-check.service
Wed 2026-11-11 15:58:30 CET 4h 54min Tue 2026-11-10 15:58:30 CET 19h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Thu 2026-11-12 00:00:00 CET 12h Wed 2026-11-11 00:00:02 CET 11h ago dpkg-db-backup.timer dpkg-db-backup.service
Thu 2026-11-12 00:22:35 CET 13h Wed 2026-11-11 00:19:20 CET 10h ago logrotate.timer logrotate.service
Thu 2026-11-12 00:54:30 CET 13h Wed 2026-11-11 10:38:42 CET 25min ago apt-daily.timer apt-daily.service
Thu 2026-11-12 02:57:50 CET 15h Wed 2026-11-11 08:20:42 CET 2h 43min ago man-db.timer man-db.service
Thu 2026-11-12 03:01:31 CET 15h Wed 2026-11-11 01:02:20 CET 10h ago pve-daily-update.timer pve-daily-update.service
Thu 2026-11-12 06:07:23 CET 19h Wed 2026-11-11 06:57:02 CET 4h 7min ago apt-daily-upgrade.timer apt-daily-upgrade.service
Sun 2026-11-15 03:10:10 CET 3 days Sun 2026-11-08 03:10:39 CET - e2scrub_all.timer e2scrub_all.service
Sun 2026-11-15 03:10:22 CET 3 days Sun 2026-11-08 03:11:23 CET - xfs_scrub_all.timer xfs_scrub_all.service
Mon 2026-11-16 00:54:35 CET 4 days Mon 2026-11-09 01:13:20 CET - fstrim.timer fstrim.service
12 timers listed.
=== timer units (all) ===
--- apt-daily-upgrade.timer [enabled/active]
OnCalendar=*-*-* 6:00
RandomizedDelaySec=60m
Persistent=true
Unit=apt-daily-upgrade.service NextElapseUSecRealtime=Thu 2026-11-12 06:07:23 CET LastTriggerUSec=Wed 2026-11-11 06:57:02 CET
--- apt-daily.timer [enabled/active]
OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h
Persistent=true
Unit=apt-daily.service NextElapseUSecRealtime=Thu 2026-11-12 00:54:30 CET LastTriggerUSec=Wed 2026-11-11 10:38:42 CET
--- apt-listchanges.timer [disabled/inactive]
OnCalendar=hourly
Unit=apt-listchanges.service NextElapseUSecRealtime= LastTriggerUSec=
--- chrony-dnssrv@.timer [disabled/]
OnActiveSec=0
OnUnitInactiveSec=1h
--- dpkg-db-backup.timer [enabled/active]
OnCalendar=daily
Persistent=true
Unit=dpkg-db-backup.service NextElapseUSecRealtime=Thu 2026-11-12 00:00:00 CET LastTriggerUSec=Wed 2026-11-11 00:00:02 CET
--- e2scrub_all.timer [enabled/active]
OnCalendar=Sun *-*-* 03:10:00
RandomizedDelaySec=60
Persistent=true
Unit=e2scrub_all.service NextElapseUSecRealtime=Sun 2026-11-15 03:10:10 CET LastTriggerUSec=Sun 2026-11-08 03:10:39 CET
--- felhom-crash-guard-check.timer [enabled/active]
OnBootSec=15min
OnUnitActiveSec=1h
Unit=felhom-crash-guard-check.service NextElapseUSecRealtime= LastTriggerUSec=Wed 2026-11-11 10:59:43 CET
--- felhom-mgmt-watchdog.timer [enabled/active]
OnBootSec=30s
OnUnitActiveSec=60s
AccuracySec=5s
Persistent=true
Unit=felhom-mgmt-watchdog.service
Unit=felhom-mgmt-watchdog.service NextElapseUSecRealtime= LastTriggerUSec=Wed 2026-11-11 11:03:57 CET
--- fstrim.timer [enabled/active]
OnCalendar=weekly
AccuracySec=1h
Persistent=true
RandomizedDelaySec=100min
Unit=fstrim.service NextElapseUSecRealtime=Mon 2026-11-16 00:54:35 CET LastTriggerUSec=Mon 2026-11-09 01:13:20 CET
--- logrotate.timer [enabled/active]
OnCalendar=daily
RandomizedDelaySec=1h
Persistent=true
Unit=logrotate.service NextElapseUSecRealtime=Thu 2026-11-12 00:22:35 CET LastTriggerUSec=Wed 2026-11-11 00:19:20 CET
--- man-db.timer [enabled/active]
OnCalendar=daily
RandomizedDelaySec=12h
Persistent=true
Unit=man-db.service NextElapseUSecRealtime=Thu 2026-11-12 02:57:50 CET LastTriggerUSec=Wed 2026-11-11 08:20:42 CET
--- proxmox-firewall.timer [disabled/inactive]
OnBootSec=1s
OnUnitInactiveSec=5s
Unit=proxmox-firewall.service
Unit=proxmox-firewall.service NextElapseUSecRealtime= LastTriggerUSec=
--- pve-daily-update.timer [enabled/active]
OnCalendar=*-*-* 1:00
RandomizedDelaySec=5h
Persistent=true
Unit=pve-daily-update.service NextElapseUSecRealtime=Thu 2026-11-12 03:01:31 CET LastTriggerUSec=Wed 2026-11-11 01:02:20 CET
--- systemd-tmpfiles-clean.timer [static/active]
OnBootSec=15min
OnUnitActiveSec=1d
Unit=systemd-tmpfiles-clean.service NextElapseUSecRealtime= LastTriggerUSec=Tue 2026-11-10 15:58:30 CET
--- xfs_scrub_all.timer [enabled/active]
OnCalendar=Sun *-*-* 03:10:00
RandomizedDelaySec=60
Persistent=true
Unit=xfs_scrub_all.service NextElapseUSecRealtime=Sun 2026-11-15 03:10:22 CET LastTriggerUSec=Sun 2026-11-08 03:11:23 CET
--- zfs-scrub-monthly@.timer [disabled/]
OnCalendar=monthly
Persistent=true
RandomizedDelaySec=1h
Unit=zfs-scrub@%i.service
--- zfs-scrub-weekly@.timer [disabled/]
OnCalendar=weekly
Persistent=true
RandomizedDelaySec=1h
Unit=zfs-scrub@%i.service
--- zfs-trim-monthly@.timer [disabled/]
OnCalendar=monthly
Persistent=true
RandomizedDelaySec=1h
Unit=zfs-trim@%i.service
--- zfs-trim-weekly@.timer [disabled/]
OnCalendar=weekly
Persistent=true
RandomizedDelaySec=1h
Unit=zfs-trim@%i.service
=== service limits of felhom + backup timers ===
--- apt-daily-upgrade.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 06:57:02 CET
ExecMainExitTimestamp=Wed 2026-11-11 06:57:02 CET
ExecMainStatus=0
ExecStart={ path=/usr/lib/apt/apt.systemd.daily ; argv[]=/usr/lib/apt/apt.systemd.daily install ; ignore_errors=no ; start_time=[Wed 2026-11-11 06:57:02 CET] ; stop_time=[Wed 2026-11-11 06:57:02 CET] ; pid=1262190 ; code=exited ; status=0 }
--- apt-daily.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 10:38:42 CET
ExecMainExitTimestamp=Wed 2026-11-11 10:38:42 CET
ExecMainStatus=0
ExecStart={ path=/usr/lib/apt/apt.systemd.daily ; argv[]=/usr/lib/apt/apt.systemd.daily update ; ignore_errors=no ; start_time=[Wed 2026-11-11 10:38:42 CET] ; stop_time=[Wed 2026-11-11 10:38:42 CET] ; pid=1563704 ; code=exited ; status=0 }
--- apt-listchanges.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=
ExecMainExitTimestamp=
ExecMainStatus=0
ExecStart={ path=python3 ; argv[]=python3 -m apt_listchanges.populate_database --profile apt ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
--- dpkg-db-backup.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 00:00:02 CET
ExecMainExitTimestamp=Wed 2026-11-11 00:00:02 CET
ExecMainStatus=0
ExecStart={ path=/usr/libexec/dpkg/dpkg-db-backup ; argv[]=/usr/libexec/dpkg/dpkg-db-backup ; ignore_errors=no ; start_time=[Wed 2026-11-11 00:00:02 CET] ; stop_time=[Wed 2026-11-11 00:00:02 CET] ; pid=686273 ; code=exited ; status=0 }
--- e2scrub_all.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=
ExecMainExitTimestamp=
ExecMainStatus=0
ExecStart={ path=/sbin/e2scrub_all ; argv[]=/sbin/e2scrub_all ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
--- felhom-crash-guard-check.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 10:59:43 CET
ExecMainExitTimestamp=Wed 2026-11-11 10:59:43 CET
ExecMainStatus=0
ExecStart={ path=/usr/local/sbin/felhom-crash-guard ; argv[]=/usr/local/sbin/felhom-crash-guard check ; ignore_errors=no ; start_time=[Wed 2026-11-11 10:59:43 CET] ; stop_time=[Wed 2026-11-11 10:59:43 CET] ; pid=1593022 ; code=exited ; status=0 }
--- felhom-mgmt-watchdog.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 11:03:57 CET
ExecMainExitTimestamp=Wed 2026-11-11 11:03:57 CET
ExecMainStatus=0
ExecStart={ path=/usr/local/sbin/felhom-mgmt-watchdog ; argv[]=/usr/local/sbin/felhom-mgmt-watchdog ; ignore_errors=no ; start_time=[Wed 2026-11-11 11:03:57 CET] ; stop_time=[Wed 2026-11-11 11:03:57 CET] ; pid=1599130 ; code=exited ; status=0 }
--- fstrim.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=
ExecMainExitTimestamp=
ExecMainStatus=0
ExecStart={ path=/sbin/fstrim ; argv[]=/sbin/fstrim --listed-in /etc/fstab:/proc/self/mountinfo --verbose --quiet-unsupported ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
--- logrotate.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 00:19:20 CET
ExecMainExitTimestamp=Wed 2026-11-11 00:19:23 CET
ExecMainStatus=0
ExecStart={ path=/usr/sbin/logrotate ; argv[]=/usr/sbin/logrotate /etc/logrotate.conf ; ignore_errors=no ; start_time=[Wed 2026-11-11 00:19:20 CET] ; stop_time=[Wed 2026-11-11 00:19:23 CET] ; pid=712662 ; code=exited ; status=0 }
--- man-db.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 08:20:42 CET
ExecMainExitTimestamp=Wed 2026-11-11 08:20:42 CET
ExecMainStatus=0
ExecStart={ path=/usr/bin/install ; argv[]=/usr/bin/install -d -o man -g man -m 0755 /var/cache/man ; ignore_errors=no ; start_time=[Wed 2026-11-11 08:20:42 CET] ; stop_time=[Wed 2026-11-11 08:20:42 CET] ; pid=1376314 ; code=exited ; status=0 }
ExecStart={ path=/usr/bin/mandb ; argv[]=/usr/bin/mandb --quiet ; ignore_errors=no ; start_time=[Wed 2026-11-11 08:20:42 CET] ; stop_time=[Wed 2026-11-11 08:20:42 CET] ; pid=1376319 ; code=exited ; status=0 }
--- pve-daily-update.service
Type=oneshot
TimeoutStartUSec=infinity
RuntimeMaxUSec=infinity
ExecMainStartTimestamp=Wed 2026-11-11 01:02:20 CET
ExecMainExitTimestamp=Wed 2026-11-11 01:02:26 CET
ExecMainStatus=0
ExecStart={ path=/usr/bin/pveupdate ; argv[]=/usr/bin/pveupdate ; ignore_errors=no ; start_time=[Wed 2026-11-11 01:02:20 CET] ; stop_time=[Wed 2026-11-11 01:02:26 CET] ; pid=771302 ; code=exited ; status=0 }
=== felhom units that are not timers ===
felhom-agent-rollback.service loaded inactive dead
felhom-agent.service loaded active running
felhom-crash-guard-check.service loaded inactive dead
felhom-crash-guard.service loaded active exited
felhom-mgmt-watchdog.service loaded inactive dead
felhom-oob-nft.service loaded active exited
felhom-shared-parent.service loaded active exited
felhom-sshd.service loaded active running
felhom-crash-guard-check.timer loaded active waiting
felhom-mgmt-watchdog.timer loaded active waiting
--- felhom-agent-rollback.service
Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=
--- felhom-agent.service
Restart=on-failure RestartUSec=5s TimeoutStartUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 16:14:40 CET
--- felhom-bootstrap.service
Restart=on-failure RestartUSec=30s TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=infinity NRestarts=0 ActiveEnterTimestamp=
--- felhom-crash-guard-check.service
Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=
--- felhom-crash-guard.service
Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:43:32 CET
--- felhom-mgmt-watchdog.service
Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=
--- felhom-oob-nft.service
Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:43:32 CET
--- felhom-shared-parent.service
Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:43:32 CET
--- felhom-sshd.service
Restart=on-failure RestartUSec=100ms TimeoutStartUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:43:34 CET
=== cron ===
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; }
47 6 * * 7 root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.weekly; }
52 6 1 * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.monthly; }
--- /etc/cron.d/e2scrub_all
30 3 * * 0 root test -e /run/systemd/system || SERVICE_MODE=1 /usr/libexec/e2fsprogs/e2scrub_all_cron
10 3 * * * root test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r
--- /etc/cron.d/vzdump
PATH="/usr/sbin:/usr/bin:/sbin:/bin"
--- /etc/cron.d/zfsutils-linux
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
24 0 1-7 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/trim ]; then /usr/lib/zfs-linux/trim; fi
24 0 8-14 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs-linux/scrub; fi
--- cron.hourly:
--- cron.daily: apt-compat dpkg logrotate man-db
--- cron.weekly: man-db
--- cron.monthly:
--- user crontabs:
=== at ===
no at
=== pve jobs ===
--- vzdump.cron
PATH="/usr/sbin:/usr/bin:/sbin:/bin"
--- vzdump.conf
=== pve tasks (start end type id status), last 14 days ===
awk: line 2: function strtonum never defined
awk: line 2: function strtonum never defined
=== runs of timer services, last 10 days (start/finish lines) ===
--- apt-daily-upgrade.service
2026-11-09T06:36:19+01:00 felhom systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities...
2026-11-09T06:36:20+01:00 felhom systemd[1]: apt-daily-upgrade.service: Deactivated successfully.
2026-11-09T06:36:20+01:00 felhom systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities.
2026-11-10T06:06:30+01:00 felhom systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities...
2026-11-10T06:06:30+01:00 felhom systemd[1]: apt-daily-upgrade.service: Deactivated successfully.
2026-11-10T06:06:30+01:00 felhom systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities.
2026-11-11T06:57:02+01:00 felhom systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities...
2026-11-11T06:57:02+01:00 felhom systemd[1]: apt-daily-upgrade.service: Deactivated successfully.
2026-11-11T06:57:02+01:00 felhom systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities.
--- apt-daily.service
2026-11-10T07:10:39+01:00 felhom systemd[1]: Starting apt-daily.service - Daily apt download activities...
2026-11-10T07:10:39+01:00 felhom systemd[1]: apt-daily.service: Deactivated successfully.
2026-11-10T07:10:39+01:00 felhom systemd[1]: Finished apt-daily.service - Daily apt download activities.
2026-11-11T03:38:42+01:00 felhom systemd[1]: Starting apt-daily.service - Daily apt download activities...
2026-11-11T03:38:42+01:00 felhom systemd[1]: apt-daily.service: Deactivated successfully.
2026-11-11T03:38:42+01:00 felhom systemd[1]: Finished apt-daily.service - Daily apt download activities.
2026-11-11T10:38:42+01:00 felhom systemd[1]: Starting apt-daily.service - Daily apt download activities...
2026-11-11T10:38:42+01:00 felhom systemd[1]: apt-daily.service: Deactivated successfully.
2026-11-11T10:38:42+01:00 felhom systemd[1]: Finished apt-daily.service - Daily apt download activities.
--- felhom-crash-guard-check.service
2026-11-11T08:59:42+01:00 felhom systemd[1]: Starting felhom-crash-guard-check.service - Felhom crash guard re-arm check...
2026-11-11T08:59:43+01:00 felhom systemd[1]: felhom-crash-guard-check.service: Deactivated successfully.
2026-11-11T08:59:43+01:00 felhom systemd[1]: Finished felhom-crash-guard-check.service - Felhom crash guard re-arm check.
2026-11-11T09:59:43+01:00 felhom systemd[1]: Starting felhom-crash-guard-check.service - Felhom crash guard re-arm check...
2026-11-11T09:59:43+01:00 felhom systemd[1]: felhom-crash-guard-check.service: Deactivated successfully.
2026-11-11T09:59:43+01:00 felhom systemd[1]: Finished felhom-crash-guard-check.service - Felhom crash guard re-arm check.
2026-11-11T10:59:43+01:00 felhom systemd[1]: Starting felhom-crash-guard-check.service - Felhom crash guard re-arm check...
2026-11-11T10:59:43+01:00 felhom systemd[1]: felhom-crash-guard-check.service: Deactivated successfully.
2026-11-11T10:59:43+01:00 felhom systemd[1]: Finished felhom-crash-guard-check.service - Felhom crash guard re-arm check.
--- felhom-mgmt-watchdog.service
2026-11-11T11:01:47+01:00 felhom systemd[1]: Starting felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)...
2026-11-11T11:01:47+01:00 felhom systemd[1]: felhom-mgmt-watchdog.service: Deactivated successfully.
2026-11-11T11:01:47+01:00 felhom systemd[1]: Finished felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent).
2026-11-11T11:02:52+01:00 felhom systemd[1]: Starting felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)...
2026-11-11T11:02:52+01:00 felhom systemd[1]: felhom-mgmt-watchdog.service: Deactivated successfully.
2026-11-11T11:02:52+01:00 felhom systemd[1]: Finished felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent).
2026-11-11T11:03:57+01:00 felhom systemd[1]: Starting felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)...
2026-11-11T11:03:57+01:00 felhom systemd[1]: felhom-mgmt-watchdog.service: Deactivated successfully.
2026-11-11T11:03:57+01:00 felhom systemd[1]: Finished felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent).
--- pve-daily-update.service
2026-11-09T02:36:45+01:00 felhom systemd[1]: pve-daily-update.service: Consumed 5.122s CPU time, 232.9M memory peak.
2026-11-10T03:05:00+01:00 felhom systemd[1]: Starting pve-daily-update.service - Daily PVE download activities...
2026-11-10T03:05:06+01:00 felhom systemd[1]: pve-daily-update.service: Deactivated successfully.
2026-11-10T03:05:06+01:00 felhom systemd[1]: Finished pve-daily-update.service - Daily PVE download activities.
2026-11-10T03:05:06+01:00 felhom systemd[1]: pve-daily-update.service: Consumed 5.162s CPU time, 294M memory peak.
2026-11-11T01:02:20+01:00 felhom systemd[1]: Starting pve-daily-update.service - Daily PVE download activities...
2026-11-11T01:02:26+01:00 felhom systemd[1]: pve-daily-update.service: Deactivated successfully.
2026-11-11T01:02:26+01:00 felhom systemd[1]: Finished pve-daily-update.service - Daily PVE download activities.
2026-11-11T01:02:26+01:00 felhom systemd[1]: pve-daily-update.service: Consumed 5.193s CPU time, 292.5M memory peak.
=== agent log: schedule words, last 3 days (tail 260) ===
2026-11-10T10:54:50+01:00 felhom sudo[667389]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T11:09:50+01:00 felhom sudo[687736]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T11:24:50+01:00 felhom sudo[708124]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T11:39:50+01:00 felhom sudo[728450]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T11:54:50+01:00 felhom sudo[748825]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T12:09:50+01:00 felhom sudo[769204]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T12:24:50+01:00 felhom sudo[789601]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T12:39:50+01:00 felhom sudo[809946]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T12:54:50+01:00 felhom sudo[830357]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T13:09:50+01:00 felhom sudo[850689]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T13:24:50+01:00 felhom sudo[870998]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T13:39:50+01:00 felhom sudo[891351]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T13:54:31+01:00 felhom systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)...
2026-11-10T13:54:31+01:00 felhom felhom-agent[1160]: time=2026-11-10T13:54:31.299+01:00 level=INFO msg="pbs: verify loop shutting down" reason="context canceled"
2026-11-10T13:54:31+01:00 felhom felhom-agent[1160]: time=2026-11-10T13:54:31.299+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled"
2026-11-10T13:54:31+01:00 felhom systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T13:57:49+01:00 felhom systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T13:57:49+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:57:49.796+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true
2026-11-10T13:57:49+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:57:49.796+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
2026-11-10T13:57:49+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:57:49.798+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)"
2026-11-10T13:57:49+01:00 felhom sudo[1173]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T125749Z-kernel-kernel-boot.json
2026-11-10T13:57:51+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:57:51.810+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T13:57:51+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:57:51.818+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T13:57:51+01:00 felhom sudo[1338]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T13:57:52+01:00 felhom sudo[1381]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T13:57:53+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:57:53.108+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T13:57:53+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:57:53.109+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gues
2026-11-10T13:57:53+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:57:53.109+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or resto
2026-11-10T13:57:53+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:57:53.113+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never besid
2026-11-10T13:58:10+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:58:10.187+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-11-10T14:12:56+01:00 felhom sudo[24807]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T14:27:51+01:00 felhom felhom-agent[1159]: time=2026-11-10T14:27:51.819+01:00 level=INFO msg="backup: restore-test first evaluation after start (R-874)" after=30m0s
2026-11-10T14:27:52+01:00 felhom felhom-agent[1159]: time=2026-11-10T14:27:52.173+01:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=local volid=local:backup/vzdump-lxc-9100-2026_10_04-21_40_45.tar.zst size_bytes=
2026-11-10T14:27:52+01:00 felhom felhom-agent[1159]: time=2026-11-10T14:27:52.548+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-08T20:14:53Z) is already proven; local: newest
2026-11-10T14:27:56+01:00 felhom sudo[45165]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T14:42:56+01:00 felhom sudo[65482]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T14:57:56+01:00 felhom sudo[85853]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:12:56+01:00 felhom sudo[106277]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:27:56+01:00 felhom sudo[126638]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:30:29+01:00 felhom felhom-agent[1159]: time=2026-11-10T15:30:29.267+01:00 level=INFO msg="pbs: verify loop shutting down" reason="context canceled"
2026-11-10T15:30:29+01:00 felhom felhom-agent[1159]: time=2026-11-10T15:30:29.267+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled"
2026-11-10T15:30:29+01:00 felhom systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)...
2026-11-10T15:30:29+01:00 felhom systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T15:33:47+01:00 felhom systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T15:33:47+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:33:47.796+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true
2026-11-10T15:33:47+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:33:47.796+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
2026-11-10T15:33:47+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:33:47.798+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)"
2026-11-10T15:33:47+01:00 felhom sudo[1171]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T143347Z-kernel-kernel-boot.json
2026-11-10T15:33:50+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:33:50.028+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:33:50+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:33:50.031+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:33:50+01:00 felhom sudo[1337]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:33:51+01:00 felhom sudo[1391]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:33:51+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:33:51.499+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:33:51+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:33:51.500+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gues
2026-11-10T15:33:51+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:33:51.503+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or resto
2026-11-10T15:33:51+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:33:51.505+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never besid
2026-11-10T15:34:08+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:34:08.409+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-11-10T15:40:13+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:40:13.600+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled"
2026-11-10T15:40:13+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:40:13.604+01:00 level=INFO msg="pbs: verify loop shutting down" reason="context canceled"
2026-11-10T15:40:13+01:00 felhom systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)...
2026-11-10T15:40:13+01:00 felhom systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T15:43:42+01:00 felhom systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T15:43:43+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:43.037+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true
2026-11-10T15:43:43+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:43.037+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
2026-11-10T15:43:43+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:43.039+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)"
2026-11-10T15:43:43+01:00 felhom sudo[1170]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T144343Z-kernel-kernel-boot.json
2026-11-10T15:43:45+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:45.438+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:43:45+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:45.441+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:43:45+01:00 felhom sudo[1334]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:43:46+01:00 felhom sudo[1381]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.727+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gues
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.728+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.728+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or resto
2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.732+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never besid
2026-11-10T15:44:05+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:44:05.780+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-11-10T15:59:09+01:00 felhom sudo[27166]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:59:15+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:59:15.660+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.173.0 report="{\"agent_update\": {\"sha256\": \"752b2727fb6043417a
2026-11-10T15:59:17+01:00 felhom systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)...
2026-11-10T15:59:17+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:59:17.726+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled"
2026-11-10T15:59:17+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:59:17.726+01:00 level=INFO msg="pbs: verify loop shutting down" reason="context canceled"
2026-11-10T15:59:17+01:00 felhom systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T15:59:17+01:00 felhom systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T15:59:17+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:17.769+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true
2026-11-10T15:59:17+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:17.769+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
2026-11-10T15:59:17+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:17.770+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)"
2026-11-10T15:59:17+01:00 felhom sudo[28278]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T145917Z-kernel-kernel-boot.json
2026-11-10T15:59:18+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:18.689+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T15:59:18+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:18.689+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T15:59:18+01:00 felhom sudo[28456]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:59:19+01:00 felhom sudo[28533]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.338+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gue
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.338+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.339+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or rest
2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.339+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never besi
2026-11-10T15:59:36+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:36.894+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.391+01:00 level=INFO msg="osupdate: running the bundle policy pass after the self-update commit (R-1285)" agent_version=0.173.0
2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.593+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE POLICY DONE install_mode=appliance removed=0"
2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.593+01:00 level=INFO msg="osupdate: bundle policy pass done" bundle_policy="{\"install_mode\": \"appliance\", \"removed\": []}"
2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.607+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.173.0 removed=33
2026-11-10T16:14:25+01:00 felhom sudo[50673]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T16:14:31+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:14:31.582+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.172.1 report="{\"agent_update\": {\"sha256\": \"505a0594a2fb117f3
2026-11-10T16:14:33+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:14:33.688+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled"
2026-11-10T16:14:33+01:00 felhom systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)...
2026-11-10T16:14:33+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:14:33.689+01:00 level=INFO msg="pbs: verify loop shutting down" reason="context canceled"
2026-11-10T16:14:33+01:00 felhom systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T16:14:33+01:00 felhom systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T16:14:33+01:00 felhom sudo[51129]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T151433Z-kernel-kernel-boot.json
2026-11-10T16:14:33+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:33.737+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true
2026-11-10T16:14:33+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:33.737+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
2026-11-10T16:14:33+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:33.737+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)"
2026-11-10T16:14:34+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:34.663+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T16:14:34+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:34.663+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T16:14:34+01:00 felhom sudo[51328]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T16:14:35+01:00 felhom sudo[51387]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.319+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gue
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.319+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.320+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or rest
2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.321+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never besi
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: runtime.schedule()
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/osupdate.(*Leg).SendUnsentLoop(0x1e342d01db80, {0xe5d130, 0x1e342d05aac0}, 0x45d964b800, 0x1e342d112060)
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/osupdate/unsent.go:76 +0xd2 fp=0x1e342cda3fa8 sp=0x1e342cda3f38 pc=0x86b292
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbs.(*Client).WaitVerify(0x1e342d05bdc0, {0xe5d130, 0x1e342d05aac0}, {0x1e342cf63800, 0x77}, 0x77359400, 0x1a3185c5000)
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbs/client.go:192 +0x228 fp=0x1e342cf0fa00 sp=0x1e342cf0f8e8 pc=0x86eec8
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbs.(*VerifyLoop).verifyOne(0x1e342d132320, {0xe5d130, 0x1e342d05aac0}, {{0x1e342cedb5d0, 0xe}, 0x1e342d05bdc0, {0x1e342cedb5a0, 0xa}})
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbs/verify.go:107 +0x1c5 fp=0x1e342cf0fe18 sp=0x1e342cf0fa00 pc=0x872405
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbs.(*VerifyLoop).tick(0x1e342d132320, {0xe5d130, 0x1e342d05aac0})
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbs/verify.go:98 +0xf5 fp=0x1e342cf0fea0 sp=0x1e342cf0fe18 pc=0x8721f5
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbs.(*VerifyLoop).Run(0x1e342d132320, {0xe5d130, 0x1e342d05aac0})
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbs/verify.go:71 +0x114 fp=0x1e342cf0ffa0 sp=0x1e342cf0fea0 pc=0x871eb4
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/backup.(*Scheduler).Run(0x1e342d168580, {0xe5d130, 0x1e342d05aac0})
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/backup/schedule.go:158 +0x21f fp=0x1e342cedffa0 sp=0x1e342cedfdf0 pc=0x7b143f
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbsdr.(*Loop).Run(0x1e342d149640, {0xe5d130, 0x1e342d05aac0})
2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbsdr/loop.go:68 +0x157 fp=0x1e342cda0fa0 sp=0x1e342cda0ef0 pc=0x8735b7
2026-11-10T16:14:40+01:00 felhom systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T16:14:40+01:00 felhom systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-10T16:14:40+01:00 felhom sudo[51951]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T151440Z-kernel-kernel-boot.json
2026-11-10T16:14:40+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:40.589+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true
2026-11-10T16:14:40+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:40.589+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
2026-11-10T16:14:40+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:40.589+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)"
2026-11-10T16:14:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:41.489+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-11-10T16:14:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:41.489+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-11-10T16:14:41+01:00 felhom sudo[52108]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T16:14:42+01:00 felhom sudo[52161]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.172+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gue
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.172+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.173+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or rest
2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.173+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never besi
2026-11-10T16:14:45+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:45.707+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-11-10T16:29:17+01:00 felhom sudo[72801]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T16:29:47+01:00 felhom sudo[73744]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T16:44:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:44:41.490+01:00 level=INFO msg="backup: restore-test first evaluation after start (R-874)" after=30m0s
2026-11-10T16:44:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:44:41.820+01:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=local volid=local:backup/vzdump-lxc-9100-2026_10_04-21_40_45.tar.zst size_bytes
2026-11-10T16:44:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:44:42.189+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-08T20:14:53Z) is already proven; local: newest
2026-11-10T16:44:47+01:00 felhom sudo[94752]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T16:59:47+01:00 felhom sudo[115085]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T17:14:47+01:00 felhom sudo[135881]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T17:29:47+01:00 felhom sudo[156253]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T17:44:47+01:00 felhom sudo[176638]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T17:59:47+01:00 felhom sudo[196955]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T18:14:47+01:00 felhom sudo[217335]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T18:29:47+01:00 felhom sudo[237706]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T18:44:47+01:00 felhom sudo[258098]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T18:59:47+01:00 felhom sudo[278436]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T19:14:47+01:00 felhom sudo[298822]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T19:29:47+01:00 felhom sudo[319218]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T19:44:47+01:00 felhom sudo[339601]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T19:59:47+01:00 felhom sudo[360021]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T20:14:47+01:00 felhom sudo[380361]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T20:29:47+01:00 felhom sudo[400764]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T20:44:47+01:00 felhom sudo[421042]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T20:59:47+01:00 felhom sudo[441405]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T21:14:47+01:00 felhom sudo[461752]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T21:29:47+01:00 felhom sudo[482106]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T21:44:47+01:00 felhom sudo[502460]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T21:59:47+01:00 felhom sudo[522716]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T22:14:45+01:00 felhom sudo[543063]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T22:14:47+01:00 felhom sudo[543124]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T22:15:03+01:00 felhom felhom-agent[51939]: time=2026-11-10T22:15:03.892+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-11-10T22:29:47+01:00 felhom sudo[563495]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T22:44:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T22:44:42.974+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-08T20:14:53Z) is already proven; local: newest
2026-11-10T22:44:47+01:00 felhom sudo[583859]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T22:59:47+01:00 felhom sudo[604236]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T23:14:47+01:00 felhom sudo[624634]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T23:29:47+01:00 felhom sudo[645036]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T23:44:47+01:00 felhom sudo[665416]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-10T23:59:47+01:00 felhom sudo[685773]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T00:14:47+01:00 felhom sudo[706194]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T00:29:47+01:00 felhom sudo[726604]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T00:44:47+01:00 felhom sudo[746951]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T00:59:47+01:00 felhom sudo[767304]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T01:14:47+01:00 felhom sudo[787701]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T01:29:47+01:00 felhom sudo[808115]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T01:44:47+01:00 felhom sudo[828434]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T01:59:47+01:00 felhom sudo[848773]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T02:14:47+01:00 felhom sudo[869104]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T02:29:47+01:00 felhom sudo[889486]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T02:44:47+01:00 felhom sudo[912524]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T02:59:47+01:00 felhom sudo[932907]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T03:14:47+01:00 felhom sudo[953298]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T03:29:47+01:00 felhom sudo[973698]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T03:44:47+01:00 felhom sudo[994598]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T03:59:47+01:00 felhom sudo[1014973]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T04:14:45+01:00 felhom sudo[1035243]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T04:14:47+01:00 felhom sudo[1035366]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T04:15:05+01:00 felhom felhom-agent[51939]: time=2026-11-11T04:15:05.895+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-11-11T04:29:47+01:00 felhom sudo[1058436]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T04:44:42+01:00 felhom felhom-agent[51939]: time=2026-11-11T04:44:42.969+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-08T20:14:53Z) is already proven; local: newest
2026-11-11T04:44:47+01:00 felhom sudo[1078775]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T04:59:47+01:00 felhom sudo[1099130]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T05:14:47+01:00 felhom sudo[1119503]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T05:29:47+01:00 felhom sudo[1139900]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T05:32:22+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:32:22.829+01:00 level=INFO msg="backup: space preflight passed" vmid=9201 target=local last_archive_bytes=2839981414 need_bytes=4623718591 avail_bytes=38510018560
2026-11-11T05:32:23+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:32:23.882+01:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=local job=backup-9201-1794371542405626888
2026-11-11T05:34:42+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:34:42.173+01:00 level=INFO msg="janitor: stale-lock sweep deferred — a heavy operation is in flight" busy=backup:local
2026-11-11T05:35:09+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:35:09.450+01:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_11_11-05_32_22.tar.zst size_bytes=2841262471 uncovered_volumes=2
2026-11-11T05:35:09+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:35:09.450+01:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1794371542405626888 archive=local:backup/vzdump-lxc-9201-2026_11_11-05_32_22.tar.zst
2026-11-11T05:36:39+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:36:39.454+01:00 level=INFO msg="osupdate: START" run=20261111T043639Z layer=guest vmid=9201 ring=1 trigger=night enabled=true release=os-guest-20261111-034654
2026-11-11T05:37:10+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:10.254+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=os-guest-20261111-034654 layer=guest:9201 lane=fast mode=apply select=listed packages=272"
2026-11-11T05:37:10+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:10.254+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-11T05:37:10+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:10.254+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=2 already=270 not-installed=0 from-snapshot=0"
2026-11-11T05:37:10+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:10.254+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=4.1 upgraded=2 restart-needed=- reboot-needed=no"
2026-11-11T05:37:10+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:10.255+01:00 level=INFO msg="osupdate: DONE" run=20261111T043639Z layer=guest vmid=9201 ring=1 trigger=night outcome=applied healthy=true reason="" upgraded=2 pending=6 not_covered=6 r
2026-11-11T05:37:10+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:10.334+01:00 level=INFO msg="osupdate: START" run=20261111T043639Z layer=host vmid=9201 ring=1 trigger=night enabled=true release=os-host-20261111-034654
2026-11-11T05:37:32+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:32.686+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=os-host-20261111-034654 layer=host lane=fast mode=apply select=listed packages=607"
2026-11-11T05:37:32+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:32.686+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-11T05:37:32+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:32.686+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=2 already=603 not-installed=2 from-snapshot=0"
2026-11-11T05:37:32+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:32.686+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=3.9 upgraded=2 restart-needed=- reboot-needed=no"
2026-11-11T05:37:33+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:33.710+01:00 level=INFO msg="osupdate: DONE" run=20261111T043639Z layer=host vmid=9201 ring=1 trigger=night outcome=applied healthy=true reason="" upgraded=2 pending=100 not_covered=10
2026-11-11T05:37:33+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:33.729+01:00 level=INFO msg="osupdate: docker step skipped — ring 1 takes an engine set only inside a signed operator job (`11` §5.8)" run=20261111T043639Z vmid=9201 trigger=night r
2026-11-11T05:37:33+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:33.729+01:00 level=INFO msg="osupdate: pve step skipped — ring 1 takes a Proxmox set only inside a signed operator job (`11` §5.10)" run=20261111T043639Z vmid=9201 trigger=night rin
2026-11-11T05:37:33+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:33.729+01:00 level=INFO msg="osupdate: kernel step skipped — the hub names no kernel step for this box" run=20261111T043639Z layer=kernel vmid=9201 trigger=night ring=1
2026-11-11T05:44:47+01:00 felhom sudo[1164292]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T05:59:47+01:00 felhom sudo[1184673]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T06:14:47+01:00 felhom sudo[1205082]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T06:29:47+01:00 felhom sudo[1225487]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T06:44:47+01:00 felhom sudo[1245747]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T06:59:47+01:00 felhom sudo[1266207]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T07:14:47+01:00 felhom sudo[1286559]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T07:29:47+01:00 felhom sudo[1306966]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T07:44:47+01:00 felhom sudo[1327307]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T07:59:47+01:00 felhom sudo[1347776]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T08:14:47+01:00 felhom sudo[1368158]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T08:29:47+01:00 felhom sudo[1388479]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T08:44:47+01:00 felhom sudo[1408823]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T08:59:47+01:00 felhom sudo[1429228]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T09:14:47+01:00 felhom sudo[1449547]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T09:29:47+01:00 felhom sudo[1469898]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T09:44:47+01:00 felhom sudo[1490278]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T09:59:47+01:00 felhom sudo[1510612]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T10:14:45+01:00 felhom sudo[1530915]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T10:14:47+01:00 felhom sudo[1530998]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T10:15:05+01:00 felhom felhom-agent[51939]: time=2026-11-11T10:15:05.894+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-11-11T10:29:47+01:00 felhom sudo[1551423]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T10:44:42+01:00 felhom felhom-agent[51939]: time=2026-11-11T10:44:42.639+01:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=local archive=local:backup/vzdump-lxc-9201-2026_11_10-05_30_28.
2026-11-11T10:44:43+01:00 felhom felhom-agent[51939]: time=2026-11-11T10:44:43.578+01:00 level=INFO msg="restore-test: space preflight passed" storage=local-lvm required_bytes=17155162112 avail_bytes=109126369674
2026-11-11T10:44:43+01:00 felhom felhom-agent[51939]: time=2026-11-11T10:44:43.624+01:00 level=INFO msg="restore-test: full-fidelity restore params derived from the archive config" scratch=990000 params=4
2026-11-11T10:44:47+01:00 felhom sudo[1571971]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-11T10:45:42+01:00 felhom felhom-agent[51939]: time=2026-11-11T10:45:42.474+01:00 level=INFO msg="restore-test: scratch guest torn down" vmid=990000
2026-11-11T10:45:42+01:00 felhom felhom-agent[51939]: time=2026-11-11T10:45:42.475+01:00 level=INFO msg="backup: scheduled restore-test passed" archive=local:backup/vzdump-lxc-9201-2026_11_10-05_30_28.tar.zst duration_s=59.829316696
2026-11-11T10:59:47+01:00 felhom sudo[1593180]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
=== agent log: line count per hour, last 26 h ===
10116 2026-11-10T09 10861 2026-11-10T10 10862 2026-11-10T11 10865 2026-11-10T12 10301 2026-11-10T13 10865 2026-11-10T14 9888 2026-11-10T15 11980 2026-11-10T16 10869 2026-11-10T17 10862 2026-11-10T18 10862 2026-11-10T19 10850 2026-11-10T20 10858 2026-11-10T21 10845 2026-11-10T22 10846 2026-11-10T23 10861 2026-11-11T00 10858 2026-11-11T01 10863 2026-11-11T02 10903 2026-11-11T03 10870 2026-11-11T04 10921 2026-11-11T05 10861 2026-11-11T06 10861 2026-11-11T07 10861 2026-11-11T08 10861 2026-11-11T09 10880 2026-11-11T10 784 2026-11-11T11
=== agent version ===
felhom-agent 0.173.0
RC=0
@@ -0,0 +1,156 @@
# host-night tester-1 taken 2026-11-11T10:09:26Z
=== pve tasks, last 14 days (local time) ===
2026-11-03 02:01:41 -> 02:01:41 0.0 min push_file 9201 OK
2026-11-03 02:03:15 -> 02:03:15 0.0 min push_file 9201 OK
2026-11-03 02:03:33 -> 02:03:33 0.0 min push_file 9201 OK
2026-11-03 02:03:52 -> 02:03:52 0.0 min push_file 9201 OK
2026-11-03 02:04:11 -> 02:04:11 0.0 min push_file 9201 OK
2026-11-03 02:04:30 -> 02:04:30 0.0 min push_file 9201 OK
2026-11-03 02:04:49 -> 02:04:49 0.0 min push_file 9201 OK
2026-11-03 02:05:08 -> 02:05:08 0.0 min push_file 9201 OK
2026-11-03 02:05:27 -> 02:05:27 0.0 min push_file 9201 OK
2026-11-03 02:05:46 -> 02:05:46 0.0 min push_file 9201 OK
2026-11-03 02:06:06 -> 02:06:06 0.0 min push_file 9201 OK
2026-11-03 02:06:25 -> 02:06:25 0.0 min push_file 9201 OK
2026-11-03 02:06:45 -> 02:06:45 0.0 min push_file 9201 OK
2026-11-03 02:07:05 -> 02:07:05 0.0 min push_file 9201 OK
2026-11-03 02:07:25 -> 02:07:25 0.0 min push_file 9201 OK
2026-11-03 02:07:45 -> 02:07:45 0.0 min push_file 9201 OK
2026-11-03 02:08:05 -> 02:08:05 0.0 min push_file 9201 OK
2026-11-03 02:08:24 -> 02:08:24 0.0 min push_file 9201 OK
2026-11-03 02:08:43 -> 02:08:43 0.0 min push_file 9201 OK
2026-11-03 02:09:02 -> 02:09:02 0.0 min push_file 9201 OK
2026-11-03 02:09:22 -> 02:09:22 0.0 min push_file 9201 OK
2026-11-03 02:09:41 -> 02:09:41 0.0 min push_file 9201 OK
2026-11-03 02:10:01 -> 02:10:01 0.0 min push_file 9201 OK
2026-11-03 02:10:20 -> 02:10:20 0.0 min push_file 9201 OK
2026-11-03 02:10:39 -> 02:10:40 0.0 min push_file 9201 OK
2026-11-03 02:10:59 -> 02:10:59 0.0 min push_file 9201 OK
2026-11-03 02:11:18 -> 02:11:18 0.0 min push_file 9201 OK
2026-11-03 02:11:37 -> 02:11:37 0.0 min push_file 9201 OK
2026-11-03 02:11:56 -> 02:11:56 0.0 min push_file 9201 OK
2026-11-03 02:12:15 -> 02:12:15 0.0 min push_file 9201 OK
2026-11-03 02:12:34 -> 02:12:34 0.0 min push_file 9201 OK
2026-11-03 02:12:53 -> 02:12:53 0.0 min push_file 9201 OK
2026-11-03 02:13:12 -> 02:13:12 0.0 min push_file 9201 OK
2026-11-03 02:13:31 -> 02:13:31 0.0 min push_file 9201 OK
2026-11-03 02:13:50 -> 02:13:50 0.0 min push_file 9201 OK
2026-11-03 02:14:09 -> 02:14:09 0.0 min push_file 9201 OK
2026-11-03 02:14:28 -> 02:14:28 0.0 min push_file 9201 OK
2026-11-03 02:14:47 -> 02:14:47 0.0 min push_file 9201 OK
2026-11-03 02:15:06 -> 02:15:06 0.0 min push_file 9201 OK
2026-11-03 02:15:25 -> 02:15:25 0.0 min push_file 9201 OK
2026-11-03 02:15:44 -> 02:15:44 0.0 min push_file 9201 OK
2026-11-03 02:16:03 -> 02:16:03 0.0 min push_file 9201 OK
2026-11-03 02:16:22 -> 02:16:22 0.0 min push_file 9201 OK
2026-11-03 02:16:42 -> 02:16:42 0.0 min push_file 9201 OK
2026-11-03 02:17:01 -> 02:17:01 0.0 min push_file 9201 OK
2026-11-03 02:17:22 -> 02:17:22 0.0 min push_file 9201 OK
2026-11-03 02:17:41 -> 02:17:41 0.0 min push_file 9201 OK
2026-11-03 02:18:01 -> 02:18:01 0.0 min push_file 9201 OK
2026-11-03 02:18:20 -> 02:18:20 0.0 min push_file 9201 OK
2026-11-03 02:18:40 -> 02:18:40 0.0 min push_file 9201 OK
2026-11-03 02:18:58 -> 02:18:58 0.0 min push_file 9201 OK
2026-11-03 02:19:17 -> 02:19:17 0.0 min push_file 9201 OK
2026-11-03 02:19:24 -> 02:19:24 0.0 min push_file 9201 OK
2026-11-03 02:19:41 -> 02:19:41 0.0 min push_file 9201 OK
2026-11-03 02:20:18 -> 02:20:18 0.0 min push_file 9201 OK
2026-11-03 02:22:39 -> 02:22:39 0.0 min push_file 9201 OK
2026-11-03 02:32:43 -> 02:32:43 0.0 min push_file 9201 OK
2026-11-03 02:33:06 -> 02:33:06 0.0 min push_file 9201 OK
2026-11-03 02:33:25 -> 02:33:25 0.0 min push_file 9201 OK
2026-11-03 02:33:44 -> 02:33:44 0.0 min push_file 9201 OK
2026-11-03 02:34:02 -> 02:34:02 0.0 min push_file 9201 OK
2026-11-03 02:39:06 -> 02:39:06 0.0 min push_file 9201 OK
2026-11-03 02:43:03 -> 02:43:03 0.0 min push_file 9201 OK
2026-11-03 02:43:58 -> 02:43:58 0.0 min push_file 9201 OK
2026-11-03 02:46:39 -> 02:46:39 0.0 min push_file 9201 OK
2026-11-03 02:56:23 -> 02:56:23 0.0 min push_file 9201 OK
2026-11-03 02:56:43 -> 02:56:43 0.0 min push_file 9201 OK
2026-11-03 02:56:55 -> 02:57:00 0.1 min aptupdate OK
2026-11-03 02:57:02 -> 02:57:02 0.0 min push_file 9201 OK
2026-11-03 02:57:20 -> 02:57:20 0.0 min push_file 9201 OK
2026-11-03 02:57:39 -> 02:57:39 0.0 min push_file 9201 OK
2026-11-03 03:02:43 -> 03:02:43 0.0 min push_file 9201 OK
2026-11-03 05:01:41 -> 05:04:29 2.8 min vzdump 9201 OK
2026-11-03 08:15:47 -> 08:15:47 0.0 min push_file 9201 OK
2026-11-03 08:16:14 -> 08:16:14 0.0 min push_file 9201 OK
2026-11-03 08:16:17 -> 08:16:17 0.0 min push_file 9201 OK
2026-11-03 08:16:20 -> 08:16:20 0.0 min push_file 9201 OK
2026-11-03 08:16:23 -> 08:16:23 0.0 min push_file 9201 OK
2026-11-03 08:16:26 -> 08:16:26 0.0 min push_file 9201 OK
2026-11-03 08:17:33 -> 08:17:33 0.0 min push_file 9201 OK
2026-11-03 08:17:47 -> 08:17:47 0.0 min push_file 9201 OK
2026-11-03 08:17:50 -> 08:17:50 0.0 min push_file 9201 OK
2026-11-03 08:17:53 -> 08:17:53 0.0 min push_file 9201 OK
2026-11-03 08:17:56 -> 08:17:56 0.0 min push_file 9201 OK
2026-11-03 08:17:59 -> 08:17:59 0.0 min push_file 9201 OK
2026-11-03 08:18:02 -> 08:18:02 0.0 min push_file 9201 OK
2026-11-03 09:07:03 -> 09:07:03 0.0 min push_file 9201 OK
2026-11-03 09:27:35 -> 09:28:38 1.1 min vzrestore 990000 OK
2026-11-03 09:28:41 -> 09:28:45 0.1 min vzstart 990000 OK
2026-11-03 09:28:48 -> 09:28:50 0.0 min vzdestroy 990000 OK
2026-11-04 03:37:26 -> 03:37:30 0.1 min aptupdate OK
2026-11-04 05:04:55 -> 05:08:18 3.4 min vzdump 9201 OK
2026-11-04 09:28:55 -> 09:30:14 1.3 min vzrestore 990000 OK
2026-11-04 09:30:17 -> 09:30:20 0.1 min vzstart 990000 OK
2026-11-04 09:30:23 -> 09:30:26 0.1 min vzdestroy 990000 OK
2026-11-05 02:14:40 -> 02:14:45 0.1 min aptupdate OK
2026-11-05 05:09:55 -> 05:13:24 3.5 min vzdump 9201 OK
2026-11-05 09:28:55 -> 09:30:22 1.4 min vzrestore 990000 OK
2026-11-05 09:30:27 -> 09:30:30 0.1 min vzstart 990000 OK
2026-11-05 09:30:33 -> 09:30:36 0.1 min vzdestroy 990000 OK
2026-11-06 03:30:07 -> 03:30:13 0.1 min aptupdate OK
2026-11-06 05:14:55 -> 05:18:34 3.6 min vzdump 9201 OK
2026-11-06 09:28:55 -> 09:30:31 1.6 min vzrestore 990000 OK
2026-11-06 09:30:32 -> 09:30:35 0.1 min vzstart 990000 OK
2026-11-06 09:30:38 -> 09:30:41 0.1 min vzdestroy 990000 OK
2026-11-07 03:56:36 -> 03:56:42 0.1 min aptupdate OK
2026-11-07 05:19:54 -> 05:23:27 3.5 min vzdump 9201 OK
2026-11-07 09:28:55 -> 09:30:27 1.5 min vzrestore 990000 OK
2026-11-07 09:30:32 -> 09:30:36 0.1 min vzstart 990000 OK
2026-11-07 09:30:38 -> 09:30:41 0.1 min vzdestroy 990000 OK
2026-11-08 03:01:24 -> 03:01:29 0.1 min aptupdate OK
2026-11-08 05:24:54 -> 05:28:25 3.5 min vzdump 9201 OK
2026-11-08 05:42:51 -> 05:44:12 1.4 min vzrestore 990000 OK
2026-11-08 05:44:17 -> 05:44:20 0.1 min vzstart 990000 OK
2026-11-08 05:44:23 -> 05:44:26 0.1 min vzdestroy 990000 OK
2026-11-08 21:14:53 -> 21:17:13 2.3 min vzdump 9201 OK
2026-11-09 02:36:40 -> 02:36:45 0.1 min aptupdate OK
2026-11-09 05:29:55 -> 05:33:45 3.8 min vzdump 9201 OK
2026-11-09 05:42:51 -> 05:44:30 1.6 min vzrestore 990000 OK
2026-11-09 05:44:32 -> 05:44:36 0.1 min vzstart 990000 OK
2026-11-09 05:44:38 -> 05:44:41 0.1 min vzdestroy 990000 OK
2026-11-09 23:42:51 -> 23:46:31 3.7 min vzrestore 990000 OK
2026-11-09 23:46:33 -> 23:46:37 0.1 min vzstart 990000 OK
2026-11-09 23:46:39 -> 23:46:42 0.1 min vzdestroy 990000 OK
2026-11-10 02:51:05 -> 02:52:46 1.7 min stopall OK
2026-11-10 02:51:05 -> 02:52:46 1.7 min vzshutdown 9201 OK
2026-11-10 02:54:48 -> 02:54:58 0.2 min startall OK
2026-11-10 02:54:48 -> 02:54:58 0.2 min vzstart 9201 OK
2026-11-10 03:05:01 -> 03:05:06 0.1 min aptupdate OK
2026-11-10 05:30:28 -> 05:33:15 2.8 min vzdump 9201 OK
2026-11-10 09:24:47 -> 09:25:45 1.0 min vzrestore 990000 OK
2026-11-10 09:25:49 -> 09:25:52 0.1 min vzstart 990000 OK
2026-11-10 09:25:55 -> 09:25:58 0.1 min vzdestroy 990000 OK
2026-11-10 13:54:35 -> 13:56:15 1.7 min stopall OK
2026-11-10 13:54:35 -> 13:56:15 1.7 min vzshutdown 9201 OK
2026-11-10 13:57:55 -> 13:58:03 0.1 min startall OK
2026-11-10 13:57:55 -> 13:58:03 0.1 min vzstart 9201 OK
2026-11-10 15:30:33 -> 15:32:12 1.6 min stopall OK
2026-11-10 15:30:33 -> 15:32:12 1.6 min vzshutdown 9201 OK
2026-11-10 15:33:53 -> 15:34:01 0.1 min startall OK
2026-11-10 15:33:53 -> 15:34:01 0.1 min vzstart 9201 OK
2026-11-10 15:40:17 -> 15:41:56 1.6 min stopall OK
2026-11-10 15:40:17 -> 15:41:56 1.6 min vzshutdown 9201 OK
2026-11-10 15:43:48 -> 15:43:56 0.1 min startall OK
2026-11-10 15:43:48 -> 15:43:56 0.1 min vzstart 9201 OK
2026-11-11 01:02:22 -> 01:02:26 0.1 min aptupdate OK
2026-11-11 05:32:22 -> 05:35:09 2.8 min vzdump 9201 OK
2026-11-11 10:44:43 -> 10:45:27 0.7 min vzrestore 990000 OK
2026-11-11 10:45:30 -> 10:45:33 0.1 min vzstart 990000 OK
2026-11-11 10:45:36 -> 10:45:39 0.1 min vzdestroy 990000 OK
=== agent: night words, 20:00 yesterday to 09:00 today (first 3 lines per message) ===
--- counts
RC=0
@@ -0,0 +1,6 @@
# offset tester-1: box clock minus DooPlex clock (midpoint of the call), seconds
offset=0.074 rtt=0.154
offset=0.072 rtt=0.154
offset=0.071 rtt=0.152
offset=0.073 rtt=0.153
offset=0.072 rtt=0.152
@@ -0,0 +1,256 @@
# Clock list: host agent and host installer (helper result, candidates only)
### 1. Whole-guest backup: due-check and run
1. The in-guest controller asks `handleBackupDue` `AG/internal/localapi/server.go:1134-1217` and starts `handleBackup` `AG/internal/localapi/server.go:878-1015`; the host has no timer for it; the agent runs the vzdump. CUSTOMER.
2. Due when the newest archive's age reaches the cadence, `age >= tier.Cadence` `AG/internal/localapi/server.go:1204-1213`; default `BackupCadence` 24 h `AG/internal/config/config.go:576-580`; the night window is the controller's `AR/07-backup-architecture.md:551`; the agent compares ages only, no time zone.
3. Task wait `defaultPrimaryTierWaitTimeout` 30 min and `defaultExtraTierWaitTimeout` 12 h `AG/internal/config/config.go:464-465`; outer bound is the tier wait plus 15 min `AG/internal/localapi/server.go:971-975`; the copy grows with the guest's size.
4. A tier not backed up stays due at every later ask; the controller has a 48 h safety valve `AR/07-backup-architecture.md:551-552`; the hub judges 72 h without a whole-guest backup `AR/07-backup-architecture.md:579-581`.
5. Same tier in flight answers 202 with the same job `AG/internal/localapi/server.go:913-917`; another tier answers 409 `AG/internal/localapi/server.go:919-925`; the host-wide gate is `TryAcquire` `AG/internal/backup/inflight.go:40-59`, in memory.
6. The clock is wall time, `now` = `time.Now().UTC()` `AG/internal/localapi/server.go:509`; candidate: a clock set back makes the age small, so the tier reads not due until real time catches up; unreadable storage uses the saved success `AG/internal/localapi/server.go:1176-1182`.
7. The hub raises `expected_backup_missed` to the operator `AR/07-backup-architecture.md:579-582`; a flood refused by the cap raises an operator event, rationale `AG/internal/localapi/backup_cap.go:12-35`.
8. TEST `TestBackupDue_R894_RestartThenUnreadableStorage_OldSavedCopyIsDue` `AG/internal/localapi/r894_saved_backup_test.go:86`; DOC `AR/07-backup-architecture.md:420-429`; clock jump: NOBODY KNOWS (grep `^func Test.*(Clock|Future|Skew|Jump|DST)` finds none for the due-check).
### 2. OS-update night leg (guest, host, Docker, Proxmox)
1. The agent runs it after a successful primary, non-manual whole-guest backup: `SetAfterPrimaryBackup` `AG/cmd/felhom-agent/main.go:1390-1398`, trigger at `AG/internal/localapi/server.go:1008-1011`; `Run` `AG/internal/osupdate/leg.go:538-586`. CUSTOMER.
2. 90 s after the backup ends `AG/cmd/felhom-agent/main.go:1395`; at most once per `MinGap`, default 20 h `AG/internal/osupdate/leg.go:656-667`; order guest, host, Docker, Proxmox, kernel `AG/internal/osupdate/leg.go:538-586`; no clock slot of its own `AR/11-os-updates.md:378-387`.
3. Wrapper limits: host command 600 s and guest command 1800 s `AG/configs/felhom-os-apply:279-284`; health wait default 5 min `AG/internal/osupdate/leg.go:810-813`; time grows with the number of packages; measured 23 to 70 s `AR/07-backup-architecture.md:449-450`.
4. A failed or missed backup means no OS leg that night `AR/07-backup-architecture.md:447`; no catch-up found; the hub shows how far behind a box is `AR/11-os-updates.md:406-408`.
5. It runs holding the heavy-op gate taken by the backup `AR/07-backup-architecture.md:444-446`; one pass at a time by flock `tryLockPass` `AG/internal/osupdate/unsent.go:35-52`; a second night run inside 20 h is skipped `AG/internal/osupdate/leg.go:663`.
6. The 20 h gap compares `l.now().Sub(last)` with a time read from a file `AG/internal/osupdate/leg.go:661-662`; candidate: a stamp written while the clock was ahead keeps the leg skipped until real time passes that stamp plus 20 h.
7. The result is posted to the hub by `finish` `AG/internal/osupdate/leg.go:922-939`; operator gets a hub event per update and failure, the household one timeline line `AR/11-os-updates.md:406-423`.
8. TEST `TestOncePerNight` `AG/internal/osupdate/leg_test.go:367` and `TestAfterPrimaryBackup` `AG/internal/localapi/afterbackup_test.go:17`; DOC `AR/11-os-updates.md:366-387`; a stamp from the future: NOBODY KNOWS.
### 3. Kernel lane: night reboot, 20-minute boot judge, one self-revert
1. Agent: `runKernel` ends the night leg `AG/internal/osupdate/kernel.go:119-178`; at every agent start `KernelAfterBoot` judges the boot `AG/cmd/felhom-agent/main.go:1018`. CUSTOMER (the box restarts at night).
2. Only on trigger night and only when the hub's block says `Tonight` `AG/internal/osupdate/kernel.go:101-115`; the household is mailed the day before, 09:00 to 20:00 Budapest time, by the hub `AR/11-os-updates.md:580-585`.
3. Judge wait `DefaultKernelJudgeWait` 20 min `AG/internal/osupdate/kernel.go:37`, polled every 30 s `AG/internal/osupdate/kernel.go:251`; measured healthy at 68 s and 272 s `AR/11-os-updates.md:570-572`.
4. No mail accepted means no step `AR/11-os-updates.md:584-585`; a step is never retried by itself `AR/11-os-updates.md:581-583`; a box that never comes back raises `host_stale` at 45 min and `host_down` at 90 min `AR/11-os-updates.md:572-573`.
5. At most one per night under the same heavy-op gate `AR/11-os-updates.md:560-562`; never two steps within 20 h (wrapper refusal R22) `AR/11-os-updates.md:557-558`; one self-revert, never retried `AG/internal/osupdate/kernel.go:300-312`.
6. The judge deadline is in-process from `now` = `time.Now()` `AG/internal/osupdate/leg.go:291-296` (monotonic kept); a clock step during the 20 min does not move it. Host starting with a wrong clock: NOT FOUND (grep `timedatectl|chrony|ntp` in `E/scripts/felhom-host-install.sh:1` finds nothing).
7. The hub hears staged before the reboot `AG/internal/osupdate/kernel.go:165`; operator events `os_kernel_step` and `os_kernel_notice` `AR/11-os-updates.md:585-586`; a frozen new kernel needs a person, accepted `AR/11-os-updates.md:534-535`.
8. TEST `TestKernelAfterBoot_UnhealthyRevertsOnceAfterTheWait` `AG/internal/osupdate/kernel_test.go:200`, `TestKernel_NoMailNoStep` `AG/internal/osupdate/kernel_test.go:79`; DOC `AR/11-os-updates.md:530-601`; a real freeze: NOBODY KNOWS `AR/11-os-updates.md:599-601`.
### 4. Crash guard: early-boot unit plus hourly re-arm timer
1. Root systemd units installed by the host installer `E/scripts/felhom-host-install.sh:2944-2963`: boot unit `AG/configs/felhom-crash-guard.service:15-16` and timer `AG/configs/felhom-crash-guard-check.timer:5-6`. CUSTOMER (decides if a crashed box restarts).
2. Timer as written: `OnBootSec=15min`, `OnUnitActiveSec=1h` `AG/configs/felhom-crash-guard-check.timer:5-6`; no OnCalendar, no RandomizedDelaySec, no Persistent; boot unit runs before sysinit `AG/configs/felhom-crash-guard.service:8`.
3. Oneshot `AG/configs/felhom-crash-guard-check.service:6-7` with no time limit written; history is cut to 7 days `AG/configs/felhom-crash-guard:148`, so nothing grows.
4. A missed hourly check only delays the re-arm; a crash before the boot unit runs leaves the box off, the safe side `AR/11-os-updates.md:484-485`; the System page shows the guard `AR/11-os-updates.md:487-488`.
5. Limits `LIMIT=3`, `WINDOW_MINUTES=60`, `REARM_HOURS=24` `AG/configs/crash-guard.conf:3-8`; the 2nd unclean boot in the window trips it `AG/configs/felhom-crash-guard:155`; no lock between boot and check found (candidate, grep `flock|lock` not run on it: NOT FOUND).
6. Wall clock: `now` is `time.time()` `AG/configs/felhom-crash-guard:43-44` and the window test is `now - t <= window` `AG/configs/felhom-crash-guard:124`, read at early boot; candidate: a clock that is wrong at boot can move unclean boots in or out of the 60 min window.
7. Each unclean boot is an operator mail `host_crash_restart` and a household line; the trip is alarm `host_crash_guard_tripped` `AR/11-os-updates.md:487-488`; the trip mail came 15 min after the boot `AR/11-os-updates.md:489-490`.
8. TEST `test_third_crash_in_an_hour_leaves_the_box_off` `AG/configs/test_felhom_crash_guard.py:75` and `test_rearms_after_24h_of_normal_running` `AG/configs/test_felhom_crash_guard.py:102`; DOC `AR/11-os-updates.md:473-490`; wrong clock at boot: NOBODY KNOWS.
### 5. Scheduled restore-test evaluation
1. Agent goroutine `Run` `AG/internal/backup/schedule.go:148-177`, wired at `AG/cmd/felhom-agent/main.go:1968-2018`; restores an archive into a scratch guest on the host. CUSTOMER (proof the backup opens).
2. First look `DefaultFirstEval` 30 min after agent start `AG/internal/backup/schedule.go:76`, then every `defaultRestoreTestEvalInterval` 6 h `AG/internal/config/config.go:636`; an archive counts after `defaultRestoreTestSettle` 24 h `AG/internal/config/config.go:641`; not windowed, any hour `AR/11-os-updates.md:385-386`.
3. Restore task wait: `RestoreTestPBSRestoreTimeout` 120 min for the PBS tier `AG/internal/config/config.go:584-588`, else the 10 min default `AG/cmd/felhom-agent/main.go:1917-1919`; boot wait `DefaultBootTimeout` 2 min `AG/internal/reconcile/restoretest.go:28`; grows with guest size.
4. The ticker restarts with the agent, so a missed look comes 30 min after the next start `AR/07-backup-architecture.md:583-584`; the rule is per archive, so a tier stays due until proven `AR/07-backup-architecture.md:179-180`; hub proof-age window 7 to 12 days `AR/07-backup-architecture.md:215-217`.
5. Takes the heavy-op gate with `TryAcquire("restore-test")` `AG/internal/backup/schedule.go:228`; busy means deferred to the next 6 h look `AG/internal/backup/schedule.go:230`; proof credit only on a pass `AG/internal/backup/schedule.go:248-255`.
6. `now` is `time.Now().UTC()`, wall clock `AG/internal/backup/schedule.go:119`, used for `settleCutoff` `AG/internal/backup/restoretest_due.go:96-101`; candidate: a clock set back makes no archive look settled; the ticker itself is monotonic.
7. A failed test is an ERROR log `AG/internal/backup/schedule.go:262` and a record the hub turns into `restore_test_failed` for the operator, as the janitor comment says `AG/cmd/felhom-agent/janitor.go:42-44`.
8. TEST `TestR874_FirstEvaluationAfterStart` `AG/internal/backup/r874_first_eval_test.go:17`, `TestDue_DailyTierIsProvedDailyOnItsOwnArchive` `AG/internal/backup/restoretest_due_test.go:139`; DOC `AR/07-backup-architecture.md:174-200`; clock set back: NOBODY KNOWS.
### 6. Controller supervisor (restarts a dead or silent controller)
1. Agent goroutine `WatchControllers` `AG/internal/localapi/controllersupervisor.go:143-163`, started at `AG/cmd/felhom-agent/main.go:1714`. CUSTOMER (the household's dashboard comes back).
2. Every `controllerSupervisorInterval` 30 s `AG/internal/localapi/controllersupervisor.go:54`; restart on the 2nd not-running sweep `AG/internal/localapi/controllersupervisor.go:56`; a running but unhealthy one is alarmed at 6 sweeps and restarted at 10 `AG/internal/localapi/controllersupervisor.go:92-100`.
3. No per-sweep time limit seen in the lines read (NOT FOUND, only `AG/internal/localapi/controllersupervisor.go:1-200` was opened); nothing grows with data; a liveness line every 20 sweeps `AG/internal/localapi/controllersupervisor.go:64`.
4. If the agent is down nothing restarts the controller: measured 33 min of 502 before this existed `AG/internal/localapi/controllersupervisor.go:20-25`; the first sweep is 30 s after agent start (ticker only) `AG/internal/localapi/controllersupervisor.go:153-161`.
5. Brake: `controllerCrashloopMax` 3 in `controllerCrashloopWindow` 15 min, then `controllerCrashloopPause` 30 min `AG/internal/localapi/controllersupervisor.go:59-61`; not during a swap, a parked or locked guest, or a vzdump `AG/internal/localapi/controllersupervisor.go:34-42`.
6. The fast record is in memory, so an agent restart forgets a pause `AG/internal/localapi/controllersupervisor.go:103-105`; the 24 h counter is persisted `AG/internal/localapi/controllersupervisor.go:115-119`; wall or monotonic for the windows: NOT FOUND (the tick body was not opened).
7. Hub mints `controller_restarted_by_agent` (info) and `controller_crashloop` (error, operator only) from the host report `AG/internal/localapi/controllersupervisor.go:44-49`; a warning at the 5th restart in 24 h `AG/internal/localapi/controllersupervisor.go:82-83`.
8. TEST `TestControllerSupervisor_KilledControllerIsRestarted` `AG/internal/localapi/controllersupervisor_test.go:101`, `TestSupervisor_UnresponsiveIsAlarmedThenRestarted` `AG/internal/localapi/supervisor_unresponsive_test.go:32`; DOC `AR/03-host-agent.md:186-192`.
### 7. Guest-power watchdog (starts a guest that should be running)
1. Agent goroutine `WatchGuestPower` `AG/internal/localapi/guestpower.go:89-105`, started at `AG/cmd/felhom-agent/main.go:1710`. CUSTOMER (the whole box's apps are off while the guest is stopped).
2. Every `guestPowerInterval` 60 s `AG/internal/localapi/guestpower.go:52`; acts only on a stopped, unlocked guest with onboot set `AG/internal/localapi/guestpower.go:167-180`; first sweep one interval after start (ticker only) `AG/internal/localapi/guestpower.go:95-102`.
3. At most `guestPowerMaxAttempts` 3 starts `AG/internal/localapi/guestpower.go:56` with waits of 1, 2 and 4 min, `guestPowerBackoff` `AG/internal/localapi/guestpower.go:75`; about 7 min in all `AG/internal/localapi/guestpower.go:71-74`.
4. Before it existed a guest stayed down 9 min 47 s until a person started it `AG/internal/localapi/guestpower.go:14-18`; with the agent down nobody starts it (the platform's onboot acts only at host boot `AG/internal/localapi/guestpower.go:42-43`).
5. Never starts under a running vzdump and fails safe on doubt `AG/internal/localapi/guestpower.go:182-194`; leaves a locked guest to the stale-lock path `AG/internal/localapi/guestpower.go:176-180`; skips a guest the operator is rebooting `AG/internal/localapi/guestpower.go:157-159`.
6. Attempt state is in memory on purpose, so an agent restart tries again at once `AG/internal/localapi/guestpower.go:77-80`; wall or monotonic for `nextAt`: NOT FOUND (`guestPowerDue` `AG/internal/localapi/guestpower.go:226` was not opened).
7. Candidate: the give-up is an ERROR journal line only `AG/internal/localapi/guestpower.go:205-206`; a hub event for it is NOT FOUND (grep `hub\.|Report|report` in `AG/internal/localapi/guestpower.go:1` finds no sender).
8. TEST `TestGuestPower_RetryIsBoundedThenEscalates` `AG/internal/localapi/guestpower_test.go:162`, `TestGuestPower_InFlightBackupIsNotDisturbed` `AG/internal/localapi/guestpower_test.go:119`; who hears the give-up: NOBODY KNOWS.
### 8. Hub report and heartbeat loop (with reconcile, fast tick and poke)
1. Agent goroutine `Run` `AG/internal/hub/loop.go:107-138`, built at `AG/cmd/felhom-agent/main.go:958`; it carries every status the hub alarms on. OPERATOR and CUSTOMER (alarms depend on it). Merged here: the reconcile engine, fast tick and poke.
2. `PollSeconds` default 900 `AG/internal/config/config.go:797`; first report at once, then a ticker `AG/internal/hub/loop.go:107-138`; the hub may change it, clamped by `MinPollSeconds` 60 and `MaxPollSeconds` 3600 `AG/internal/hub/loop.go:11-12`; fast tick `DefaultInterval` 30 s `AG/internal/fasttick/fasttick.go:24`.
3. Per-request limit `TimeoutSeconds` default 30 `AG/internal/config/config.go:798`; the system facts in it are read at most every 10 min `AG/cmd/felhom-agent/main.go:4149`; the report grows with guests and drives.
4. A failed cycle is logged and the interval kept `AG/internal/hub/loop.go:162-164`; no backlog, the next report replaces it; the hub raises `host_stale` at 45 min and `host_down` at 90 min `AR/11-os-updates.md:572-573`.
5. Extra reports come by the trigger channel `AG/internal/hub/loop.go:125-135`; a poke is debounced by `DebounceWindow` 2 s `AG/internal/poke/poke.go:40`; the reconcile engine runs at once and then on the start-time interval `AG/internal/reconcile/engine.go:332-344`, `AG/cmd/felhom-agent/main.go:1525`.
6. The ticker is monotonic; candidate: the reconcile engine keeps the interval read at start `AG/cmd/felhom-agent/main.go:969` and does not follow the hub's override `AG/internal/hub/loop.go:188-196`. A suspended host: NOT FOUND for the agent (the controller has a rule `AR/07-backup-architecture.md:555-558`).
7. The operator, by the hub's staleness alarms `AR/11-os-updates.md:572-573`; the household hears cannot-be-reached at most once per 7 days `AR/07-backup-architecture.md:582-583`.
8. TEST `TestLoop_RunImmediateAndResilientAfterError` `AG/internal/hub/loop_test.go:140`, `TestClampInterval` `AG/internal/hub/loop_test.go:81`; DOC `AR/03-host-agent.md:276-279`.
### 9. Signed jobs: validity window against the host clock (agent updates ride it)
1. Agent `OnEnvelope` runs pending signed jobs when a heartbeat says there are some `AG/internal/signedjobs/runner.go:93-102`; an agent update arrives only this way. OPERATOR (and CUSTOMER: fixes reach the box this way).
2. Checked at every hub cycle (item 8); a job is valid from issue minus `DefaultClockSkew` 2 min `AG/internal/authz/verifier.go:21` to its expiry, and a window over `MaxWindow` 24 h is refused `AG/internal/authz/verifier.go:26`.
3. The self-update download has a 5 min client limit `AG/internal/selfupdate/executor.go:73`; the new binary must run `DwellSeconds` 60 s before it commits `AG/internal/config/config.go:97-99`.
4. A job not picked up before its expiry is refused as expired `AG/internal/authz/verifier.go:161-162`; a commit that fails is retried at the next start `AG/internal/selfupdate/commit.go:122-125`; who notices an expired job: NOT FOUND in the agent (hub side).
5. One run at a time, `RunOnce` single-flight `AG/internal/signedjobs/runner.go:107-115`; a replay is refused by the nonce store `AG/internal/authz/verifier.go:166`.
6. Wall clock: candidate, a host clock that is behind refuses every job as not yet valid `AG/internal/authz/verifier.go:155-156`, one that is ahead refuses them as expired `AG/internal/authz/verifier.go:161-162`; so a box with a wrong clock cannot be updated by signed job.
7. NOT FOUND in the agent source read (grep `not_yet|NotYet|expired` shown only in `AG/internal/authz/verify_test.go:140-155`); the result path to the hub was not opened.
8. TEST `TestVerify_RejectsPerStage` `AG/internal/authz/verify_test.go:82`, `TestRunner_ExpiredRejected` `AG/internal/signedjobs/runner_test.go:168`, `TestManager_ShutdownBeforeDwellLeavesPending` `AG/internal/selfupdate/commit_test.go:102`.
### 10. Storage watchdog and the 20-second mount and bind re-assert
1. Agent: the watchdog `Run` `AG/internal/storage/watchdog.go:196-215` wired at `AG/cmd/felhom-agent/main.go:1142`, and a 20 s ticker that re-asserts mounts and guest binds `AG/cmd/felhom-agent/main.go:1613-1631`. CUSTOMER (a dropped USB drive).
2. `DefaultWatchdogInterval` 8 s and `DefaultWatchdogDebounce` 30 s `AG/internal/storage/watchdog.go:15-16`; setting `WatchdogIntervalSeconds`, 0 means the default `AG/internal/config/config.go:770-778`; the re-assert is 20 s, below the controller's 30 s tick `AG/cmd/felhom-agent/main.go:1612-1614`.
3. No per-tick limit found in the lines read (NOT FOUND, grep `WithTimeout` not run on the watchdog); the work grows with the number of enrolled drives.
4. A missed tick is made up by the next one; with the agent down a dropped drive is seen only at the next 15 min report, the reason the watchdog exists `AR/03-host-agent.md:461`.
5. A flapping drive backs off: `backoffWindow` `AG/internal/storage/watchdog.go:183-192`, alert after 4 failed cycles `AG/internal/storage/watchdog.go:77-79`; the retry test is `now.Sub(last)` `AG/internal/storage/watchdog.go:290`.
6. `now` is `time.Now().UTC()`, wall clock `AG/internal/storage/watchdog.go:160`; candidate: a clock set back stretches the back-off, set forward ends it early; the tickers are monotonic.
7. A drop requests an immediate hub report through the trigger channel `AG/internal/hub/loop.go:125-135`; the hub then decides who is mailed (hub side, not read).
8. TEST `TestWatchdog_DebounceCoalescesFlaps` `AG/internal/storage/watchdog_test.go:112`, `TestWatchdog_Reconcile_FlappingBacksOffAndCaps` `AG/internal/storage/watchdog_test.go:460`; the 20 s ticker: NOBODY KNOWS (no test found for the goroutine at `AG/cmd/felhom-agent/main.go:1613`).
### 11. Weekly guest disk trim
1. Agent goroutine `Run` `AG/internal/fstrim/fstrim.go:186-199`, started at `AG/cmd/felhom-agent/main.go:1741`; runs `pct fstrim` per owned running guest. CUSTOMER (a full thin pool turns every guest read-only `AG/internal/fstrim/fstrim.go:3-5`).
2. Due from Wednesday 10:00 host-local, starts only 10:00 to 20:59 `AG/internal/fstrim/fstrim.go:42-46`; looks 5 min after start then hourly, `FirstTickDelay` and `TickInterval` `AG/internal/fstrim/fstrim.go:48-50`; zone is `time.Local` `AG/internal/fstrim/fstrim.go:110`.
3. `PerGuestTimeout` 30 min `AG/internal/fstrim/fstrim.go:52`; measured 24.4 s for 84 GiB `AG/internal/fstrim/fstrim.go:6`; grows with freed blocks.
4. A box off on Wednesday catches up at its next eligible hour `AG/internal/fstrim/fstrim.go:10-11`; a failure is retried at most `MaxAttemptsPerWeek` 3 times `AG/internal/fstrim/fstrim.go:46`; the result rides the host report `AG/internal/fstrim/fstrim.go:326-330`.
5. Holds the heavy-op gate for the whole pass; busy means deferred to the next hour `AG/internal/fstrim/fstrim.go:247-251`; the persisted record stops a second trim after a restart `AG/internal/fstrim/fstrim.go:17`.
6. Wall clock in host-local time `AG/internal/fstrim/fstrim.go:204`; candidate: an attempt stamped while the clock was ahead is not before the anchor, so `due` stays false until real time passes it `AG/internal/fstrim/fstrim.go:179-182`. No zone set by the installer: NOT FOUND (grep `timedatectl|set-timezone`).
7. A failed trim is a WARN journal line `AG/internal/fstrim/fstrim.go:318-319` and a field on the host report; who is mailed: NOT FOUND in the agent (hub side).
8. TEST `TestPassNeverRunsInTheNight` `AG/internal/fstrim/fstrim_test.go:161`, `TestPassDefersToAHeavyOperationAndRetriesNextHour` `AG/internal/fstrim/fstrim_test.go:175`, `TestWeekAnchorIsTheLastWednesdayTen` `AG/internal/fstrim/fstrim_test.go:104`; DST and clock jump: NOBODY KNOWS.
### 12. Weekly upload line test
1. Agent goroutine `Run` `AG/internal/linetest/linetest.go:256-268`, started at `AG/cmd/felhom-agent/main.go:1762`; a 10 s iperf3 upload over the tunnel. CUSTOMER (sets the torrent app's upload limit `AG/internal/linetest/linetest.go:3-4`).
2. Due from Tuesday 10:00 host-local, starts 10:00 to 15:59 `AG/internal/linetest/linetest.go:47-49`; each host from its own minute, `SlotMinute` `AG/internal/linetest/linetest.go:148-152`; ticks every minute after a 5 min delay `AG/internal/linetest/linetest.go:60-61`; zone `time.Local` `AG/internal/linetest/linetest.go:131`.
3. Three runs of 10 s, each bounded by `RunTimeout` 25 s, 30 s apart `AG/internal/linetest/linetest.go:53-57`; nothing grows with data.
4. A box that was off catches up on a later day inside the window `AG/internal/linetest/linetest.go:10-11`; the hub page turns amber when the last success is over 15 days old `AR/11-os-updates.md:409-411`.
5. Needs 10 quiet minutes and the heavy-op gate `AG/internal/linetest/linetest.go:316-325`; server busy is retried at the next tick, at most `MaxTriesPerDay` 6 `AG/internal/linetest/linetest.go:58`; no tunnel means no test `AG/internal/linetest/linetest.go:297-300`.
6. Wall clock, host-local `AG/internal/linetest/linetest.go:281`; candidate: `Due` is false while `LastOKAt` is not before the anchor, so a success stamped from the future blocks tests until then `AG/internal/linetest/linetest.go:211-213`; a sample gap over 3 min is not quiet `AG/internal/linetest/linetest.go:244-245`.
7. The operator, by the hub System page cell judged on the last success, not the attempt `AR/11-os-updates.md:409-411`; nobody is mailed as far as the agent source shows (NOT FOUND, hub side).
8. TEST `TestDueWeeklyFromTuesday` `AG/internal/linetest/linetest_test.go:206`, `TestNeverTestsInTheNightOrBeforeTuesday` `AG/internal/linetest/linetest_test.go:363`, `TestSlotDelaysTheFirstStartOnTheDueDayOnly` `AG/internal/linetest/linetest_test.go:189`.
### 13. PBS datastore verify loop
1. Agent goroutine `Run` `AG/internal/pbs/verify.go:64-83`, cadence wired at `AG/cmd/felhom-agent/main.go:1188`; asks the PBS server to verify the box's snapshots. CUSTOMER (is the off-site copy readable).
2. `DefaultVerifyCadence` 6 h `AG/internal/pbs/verify.go:13`; setting through `PBSVerifyCadence` `AG/internal/config/config.go:710-719`; first pass at once at agent start, then the ticker `AG/internal/pbs/verify.go:64-83`; no time zone.
3. One verify is polled every 2 s for at most 30 min `AG/internal/pbs/verify.go:107`; grows with the datastore's size.
4. No catch-up state: every agent start verifies again at once `AG/internal/pbs/verify.go:64-83`; a negative cadence switches it off, test `TestVerifyLoop_DisabledByNegativeCadence` `AG/internal/pbs/verify_test.go:87`.
5. Candidate: it is not behind the heavy-op gate, the lanresolver comment calls it "not journaled/gated" `AG/internal/lanresolver/loop.go:17`; so it can run beside a backup; NOT FOUND for a lock (grep `TryAcquire` not run on `AG/internal/pbs/verify.go:1`).
6. Ticker only, monotonic; no wall-clock decision found in `AG/internal/pbs/verify.go:64-83`.
7. Failed-verify snapshots are an ERROR journal line `AG/internal/pbs/verify.go:127` and recorded state for the report; who is mailed: NOT FOUND in the agent (hub side).
8. TEST `TestVerifyLoop_FailedVerifyRecorded` `AG/internal/pbs/verify_test.go:52`; a verify longer than 30 min: NOBODY KNOWS.
### 14. Janitor: leftover scratch guests and stale backup locks
1. Agent goroutine `runJanitor` `AG/cmd/felhom-agent/janitor.go:69-81`, started at `AG/cmd/felhom-agent/main.go:1725`. CUSTOMER (a stale lock blocked whole-box backups for five hours `AG/cmd/felhom-agent/janitor.go:14-17`).
2. Every `janitorInterval` 10 min `AG/cmd/felhom-agent/janitor.go:18`; ticker only, so the first pass is 10 min after agent start `AG/cmd/felhom-agent/janitor.go:71-78`; no time zone.
3. No time limit on a pass found in `AG/cmd/felhom-agent/janitor.go:34-66`; work grows with the number of leftover scratch guests.
4. A missed pass is retried 10 min later; a scratch guest that cannot be removed after the retry limit is reported as a failed restore-test record `AG/cmd/felhom-agent/janitor.go:41-54`.
5. The stale-lock sweep runs only holding the heavy-op gate, else it is skipped this pass `AG/cmd/felhom-agent/janitor.go:57-64`; rule stated at `AG/cmd/felhom-agent/janitor.go:30-33`.
6. Ticker only, monotonic; a clock step must not kill a live task: `TestVzdumpInFlight_ClockStepCannotKillALiveTask` `AG/internal/localapi/stalelock_inflight_test.go:97`.
7. The operator, through the hub's `restore_test_failed`, once per distinct archive `AG/cmd/felhom-agent/janitor.go:42-44`.
8. TEST `TestJanitor_StaleLockSweepWaitsForTheHeavyGate` `AG/cmd/felhom-agent/janitor_test.go:40`, `TestJanitor_GaveUpIsReportedAsAFailure` `AG/cmd/felhom-agent/janitor_test.go:22`.
### 15. Guest network watchdog (DHCP client heal)
1. Agent goroutine `Watch` `AG/internal/guestnet/watchdog.go:105-121`, built at `AG/cmd/felhom-agent/main.go:1686`. CUSTOMER (a guest without network).
2. `DefaultInterval` 60 s `AG/internal/guestnet/watchdog.go:23`; immediate first sweep `AG/internal/guestnet/watchdog.go:111`; settings interval 60, `MinHealIntervalSeconds` 600, `MaxHealsPerHour` 3 `AG/internal/config/config.go:176-196`.
3. No per-probe limit seen in `AG/internal/guestnet/watchdog.go:1-200` (NOT FOUND); nothing grows with data.
4. A missed sweep is made up a minute later; beyond 3 heals an hour it only reports `AG/internal/guestnet/watchdog.go:26-28`.
5. Two consecutive bad probes before a heal, `requiredBadProbes` `AG/internal/guestnet/watchdog.go:32-33`; 10 min between heals, `DefaultMinHealInterval` `AG/internal/guestnet/watchdog.go:25`; boot-race guard `DefaultSettle` 3 min `AG/internal/guestnet/watchdog.go:29-31`.
6. `now` is plain `time.Now` `AG/internal/guestnet/watchdog.go:80`, so in-process gaps are monotonic; heal history is in memory `AG/internal/guestnet/watchdog.go:56-62`.
7. A report per guest with `HealsLastHour` and `CheckedAt` `AG/internal/guestnet/watchdog.go:166-170`; who is mailed: NOT FOUND in the agent (hub side).
8. TEST `TestWatchdog_DampingCeilings` `AG/internal/guestnet/watchdog_test.go:360`, `TestWatchdog_MinimumIntervalBetweenHeals` `AG/internal/guestnet/watchdog_test.go:384`, `TestWatchdog_BootRacesObserveOnly` `AG/internal/guestnet/watchdog_test.go:416`.
### 16. Host networking self-heal
1. Agent goroutine `Watch` `AG/internal/selfheal/selfheal.go:67-82`, interval set at `AG/cmd/felhom-agent/main.go:1648`; starts `networking.service` on an appliance only. CUSTOMER.
2. One pass at agent start, then every 60 s `AG/internal/selfheal/selfheal.go:67-82`, `AG/cmd/felhom-agent/main.go:1648`; no setting found for the interval (hard-coded).
3. A remedy is 3 start attempts with waits of 10, 30 and 60 s, `networkingBackoffs` `AG/internal/selfheal/selfheal.go:114-116`.
4. A missed pass is made up a minute later; on a BYO host the remedy is skipped and only a WARN is written `AG/internal/selfheal/selfheal.go:54-58`.
5. No lock seen; the pass is sequential inside one goroutine `AG/internal/selfheal/selfheal.go:46-64`.
6. Ticker only, monotonic; no wall-clock decision in `AG/internal/selfheal/selfheal.go:67-82`.
7. A give-up is an ERROR journal line `AG/internal/selfheal/selfheal.go:61`; with networking down the hub cannot be told, so the hub's stale alarm is the only signal `AR/11-os-updates.md:572-573`.
8. TEST `TestNetworkingHeal_GivesUpAfterAllAttempts` `AG/internal/selfheal/selfheal_test.go:109`, `TestManager_ByoNeverRemediates` `AG/internal/selfheal/selfheal_test.go:32`.
### 17. Management-plane watchdog timer (keeps SSH reachable)
1. Root systemd timer and oneshot installed by the host installer `E/scripts/felhom-host-install.sh:3067-3076`; script `AG/configs/felhom-mgmt-watchdog.sh:26-68`; works without the agent `AG/configs/felhom-mgmt-watchdog.service:3-7`. OPERATOR.
2. As written: `OnBootSec=30s`, `OnUnitActiveSec=60s`, `AccuracySec=5s`, `Persistent=true` `AG/configs/felhom-mgmt-watchdog.timer:11-14`; no OnCalendar, no RandomizedDelaySec.
3. No time limit written in the unit `AG/configs/felhom-mgmt-watchdog.service:16-18`; the script is a few stat and systemctl calls `AG/configs/felhom-mgmt-watchdog.sh:36-61`.
4. A missed tick is retried by the next one `AG/configs/felhom-mgmt-watchdog.sh:23-24`; the heal budget is one tick `AG/configs/felhom-mgmt-watchdog.timer:3-6`.
5. Idempotent: a healthy directory is never touched `AG/configs/felhom-mgmt-watchdog.sh:13-15`; candidate: `Persistent=true` only acts on OnCalendar timers in systemd, and this timer has none `AG/configs/felhom-mgmt-watchdog.timer:11-14`.
6. Monotonic timer settings only `AG/configs/felhom-mgmt-watchdog.timer:11-12`; the heal marker is a UTC stamp from `date -u` `AG/configs/felhom-mgmt-watchdog.sh:65`.
7. The marker lets the agent report the heal and the hub raise a warning `AG/configs/felhom-mgmt-watchdog.sh:18-19`, `AG/configs/felhom-mgmt-watchdog.sh:63`.
8. SOURCE `AG/configs/felhom-mgmt-watchdog.sh:1-68`; a test for the script: NOT FOUND (a listing of the agent configs folder shows no `test_felhom_mgmt_watchdog` file); DOC comment `AG/configs/felhom-mgmt-watchdog.timer:3-6`.
### 18. Agent service restart, start limit and rollback
1. systemd restarts the agent: `Restart=on-failure`, `RestartSec=5s` `AG/configs/felhom-agent.service:41-42`; drop-in limits `AG/configs/felhom-agent-limits.conf:17-19`. OPERATOR (every loop in this list dies with the agent).
2. `StartLimitIntervalSec=120`, `StartLimitBurst=4`, `OnFailure=felhom-agent-rollback.service` `AG/configs/felhom-agent-limits.conf:17-19`; a crash-looping binary reaches failed about 20 s after the first crash `AG/configs/felhom-agent-limits.conf:5-8`.
3. No WatchdogSec in the unit (grep `WatchdogSec` in `AG/configs/felhom-agent.service:1` finds nothing); a new binary commits after a 60 s dwell `AG/internal/selfupdate/commit.go:40-42`.
4. After the start limit the unit stays failed and nothing on the host restarts it (candidate); the hub's `host_stale` at 45 min is the signal `AR/11-os-updates.md:572-573`.
5. The rollback unit fires on every crash and is guarded by a pending marker `AG/configs/felhom-agent-limits.conf:14-15`; a shutdown before the dwell leaves the update pending `AG/internal/selfupdate/commit.go:112-116`.
6. Candidate: the design text says `Restart=always` and a systemd watchdog that kills a hung agent `AR/03-host-agent.md:72`; the unit has `Restart=on-failure` and no watchdog `AG/configs/felhom-agent.service:41-42`, so a hung (not crashed) agent is not restarted.
7. The operator, by the hub's stale and down alarms only `AR/11-os-updates.md:572-573`; a restore-test crash loop that never evaluates is covered by `TestR874_CrashLoopNeverEvaluates` `AG/internal/backup/r874_first_eval_test.go:44`.
8. SOURCE `AG/configs/felhom-agent-limits.conf:5-15` (measured values quoted there); TEST `TestManager_ShutdownBeforeDwellLeavesPending` `AG/internal/selfupdate/commit_test.go:102`; a hung agent: NOBODY KNOWS.
### 19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)
1. Agent goroutines of one shape: tunnel `Run` `AG/internal/wgtunnel/loop.go:62-78`, PBS-DR `AG/internal/pbsdr/loop.go:62-77`, OOB sshd `AG/internal/felhomsshd/loop.go:67-80`, LAN resolver `AG/internal/lanresolver/loop.go:35-52`. CUSTOMER (off-site backup path) and OPERATOR. Four loops merged.
2. Tunnel, PBS-DR and OOB every 60 s `AG/internal/config/config.go:132-133`, `AG/cmd/felhom-agent/main.go:1504`, `AG/internal/config/config.go:74-75`; resolver every 5 min `AG/internal/lanresolver/loop.go:27-29`; each applies at once at start and on a desired-state nudge.
3. No per-pass limit read (NOT FOUND, manager bodies not opened); nothing grows with data.
4. A missed pass is made up at the next tick; a handshake older than `StaleAfterSeconds` 180 re-resolves the endpoint `AG/internal/config/config.go:124-127`, `AG/internal/config/config.go:138-140`.
5. A handshake stamped in the future for `rekeyAfterDefault` 15 min makes a new key, at most once per `rekeyMinGap` 1 h `AG/internal/wgtunnel/rekey.go:36-38`; the hour bound survives a restart, the 15 min count does not `AR/03-host-agent.md:376-378`.
6. A box whose clock once ran ahead is locked out of the off-site endpoint until real time passes that moment `AR/03-host-agent.md:142`; healed by the re-key `AR/03-host-agent.md:144-150`; known gap: a reboot after the clock was corrected forgets the future stamp `AR/03-host-agent.md:155-156`.
7. The heartbeat carries `handshake_future_s` and the hub alarms on it `AR/03-host-agent.md:142`.
8. TEST `TestD30_AFutureHandshakeThatStaysReKeys` `AG/internal/wgtunnel/rekey_test.go:35`, `TestD30_ALastReKeyFromTheFutureDoesNotBlock` `AG/internal/wgtunnel/rekey_test.go:213`, `TestR1204_AFutureHandshakeIsReported` `AG/internal/wgtunnel/manager_test.go:704`; measured live `AR/03-host-agent.md:151-155`.
### 20. Whole-guest backup cap (rolling windows)
1. Agent local API check `capRefuses` before a backup starts `AG/internal/localapi/server.go:929`; rule `backupCadenceCap` `AG/internal/localapi/backup_cap.go:41-53`. CUSTOMER (protects older copies from a flood of new ones).
2. Tier of 24 h or less: 3 successes per rolling 24 h; longer tier: 1 per the larger of cadence minus 24 h and half the cadence `AG/internal/localapi/backup_cap.go:12-35`; a refusal holds for `backupCapRefusalHold` 24 h `AG/internal/localapi/backup_cap.go:38`.
3. A lookup, no duration of its own; the success list is bounded by the window `AG/internal/localapi/backup_cap.go:41-53`.
4. Does not apply to a missed run: failures do not count, `TestBackupCap_FailuresDoNotCount` `AG/internal/localapi/backup_cap_test.go:266`.
5. The scheduled daily run is never refused, `TestBackupCap_TheScheduledDayIsNeverRefused` `AG/internal/localapi/backup_cap_test.go:220`; the count survives a restart, `TestBackupCap_SurvivesARestart` `AG/internal/localapi/backup_cap_test.go:248`.
6. Wall clock from the server's `now` `AG/internal/localapi/server.go:509`; candidate: successes stamped while the clock was ahead stay inside the rolling window longer and could refuse real runs; NOBODY KNOWS.
7. The operator, by event `whole_guest_backup_refused` named in `AG/internal/localapi/backup_cap.go:12-35`; test `TestBackupCap_RefusalRaisesTheOperatorSignal` `AG/internal/localapi/backup_cap_test.go:285`.
8. TEST `TestBackupCap_LocalFloodRunsAtMostThreeADay` `AG/internal/localapi/backup_cap_test.go:198`, `TestBackupCap_OffsiteFloodKeepsAPreCompromiseCopy` `AG/internal/localapi/backup_cap_test.go:151`.
### 21. Operator restart and reboot limits
1. Agent local API rules for the operator's hub buttons: `opRebootEvery` `AG/internal/localapi/opreboot.go:46`, `opBrakeRefusal` `AG/internal/localapi/opactions.go:42`. OPERATOR (the household sees a notice line).
2. At most one operator reboot per guest per hour `AG/internal/localapi/opreboot.go:45-46`; the household's dashboard shows the line for `opRebootNoticeFor` 24 h `AG/internal/localapi/opreboot.go:47-48`; a brake refusal holds 30 min `AG/internal/localapi/opactions.go:42`.
3. `opRebootTimeout` 5 min bounds `pct reboot` `AG/internal/localapi/opreboot.go:49-50`; the health wait after a restart is `defaultOpHealthWait` 3 min, polled every 2 s `AG/internal/localapi/opactions.go:46-49`.
4. Not a scheduled job: nothing runs if the operator does not press; a result is forgotten after `opForgetAfter` 24 h `AG/internal/localapi/opactions.go:44`.
5. Refused during a swap, a heavy operation or a busy controller `AG/internal/localapi/opreboot.go:23-31`; the heavy-op gate is held until the guest is up `AG/internal/localapi/opreboot.go:32-34`; the busy file counts as fresh for `controllerBusyFresh` 90 s `AG/internal/localapi/opreboot.go:59-61`.
6. The reboot record is persisted with a wall-clock time `AG/internal/localapi/opreboot.go:64-70`; candidate: a clock set back after a reboot extends the one-hour refusal; NOBODY KNOWS.
7. The operator reads the result of the press; the household reads the line on its dashboard `AG/internal/localapi/opreboot.go:36-39`.
8. TEST stated as "Pinned by opreboot_test.go" `AG/internal/localapi/opreboot.go:39` (the test names were not opened); DOC `AR/03-host-agent.md:236-239`.
### 22. Unsent OS-update report sender
1. Agent goroutine `SendUnsentLoop` `AG/internal/osupdate/unsent.go:71-82`, started at `AG/cmd/felhom-agent/main.go:1009`; posts OS-update results the hub has not received. OPERATOR.
2. Every 5 min `AG/cmd/felhom-agent/main.go:1009`, by `time.After` `AG/internal/osupdate/unsent.go:79`; one send at start, `SendUnsent` `AG/internal/osupdate/unsent.go:56-64`.
3. A report post has a 1 min limit in `finish` `AG/internal/osupdate/leg.go:922-939`; the kept copies grow only while the hub is away.
4. Hub away keeps the copy for a later pass, `TestR868_HubAwayKeepsTheCopy` `AG/internal/osupdate/unsent_test.go:87`.
5. The sender stays off while a pass holds the flock, `tryLockPass` `AG/internal/osupdate/unsent.go:35-52`; test `TestR868_RunningPassKeepsTheSenderOff` `AG/internal/osupdate/unsent_test.go:110`.
6. `time.After` is monotonic `AG/internal/osupdate/unsent.go:79`; no wall-clock decision found in `AG/internal/osupdate/unsent.go:56-82`.
7. The operator sees the result late, at most 5 min after the hub is back; no alarm of its own found (NOT FOUND, grep `Warn|Error` not run on `AG/internal/osupdate/unsent.go:1`).
8. TEST `TestR868_ACopyWrittenAfterTheStartIsSentByTheLoop` `AG/internal/osupdate/unsent_test.go:127`, `TestR868_KilledPassIsReportedAtStart` `AG/internal/osupdate/unsent_test.go:20`.
### 23. Local-API certificate lifetime (a ten-year deadline)
1. The agent makes a self-signed leaf once, `EnsureLeaf` `AG/internal/localapi/cert.go:38-52`; every guest's controller pins its fingerprint `AG/internal/localapi/cert.go:21-24`. CUSTOMER (controller to agent calls, the backup trigger included).
2. `certValidity` is 10 years `AG/internal/localapi/cert.go:25`; start is backdated 1 h for clock skew `AG/internal/localapi/cert.go:65`; no renewal job: rotation is an operator action `AG/internal/localapi/cert.go:23-24`.
3. Not a running job; nothing grows.
4. No timer renews it; what happens at expiry and who notices: NOT FOUND (grep `NotAfter|expir` in `AG/internal/localapi/cert.go:1` shows only the creation at `AG/internal/localapi/cert.go:70`).
5. A second generation would break every issued pin; it is logged loudly `AG/internal/localapi/cert.go:34-37`.
6. Candidate: a leaf made while the host clock was far ahead has a start date in the future; whether the controller checks dates or only the pin is NOT FOUND here (controller side not read).
7. NOT FOUND: no expiry warning in the agent source read.
8. SOURCE `AG/internal/localapi/cert.go:21-25`; expiry behaviour: NOBODY KNOWS.
## Tiny
- Guest swap check runs once, 1 min after agent start, with a 2 min limit `AG/cmd/felhom-agent/main.go:1021-1034`.
- System facts for the host report are read at most every 10 min `AG/cmd/felhom-agent/main.go:4149`.
- A backup-store grant is re-granted at most once per hour per tier, `storeGrantRepairMinInterval` `AG/cmd/felhom-agent/main.go:513`.
- Escrow ceremony: claim lives 10 min, confirm 30 min, `escrowClaimTTL` and `escrowConfirmTTL` `AG/internal/localapi/escrow_ceremony.go:158-166`.
- A network change must verify within 95 s or it is rolled back, `netVerifyDeadline` `AG/internal/localapi/netverifyjob.go:51`.
- Host power-off is refused before 10 min of uptime, `MinUptime` `AG/internal/localapi/hostpoweroff.go:76`.
- Controller swap verify: 90 s, polled every 3 s `AG/internal/localapi/controllerswap.go:86-87`.
- Guest hook phases are bounded at 30 s each `AG/cmd/felhom-agent/main.go:88-98`.
- Proxmox task wait polls with a doubling, capped interval, `WaitTask` `AG/internal/proxmox/task.go:110-141`.
- Thin-pool check reads twice with a 3 s pause `AG/internal/thincheck/thincheck.go:183`.
- Poke listener re-binds every 5 s, `rebindInterval` `AG/internal/poke/poke.go:44`.
- Backup snapshot watch ticks every 1 s during a vzdump, `snapshotWatchInterval` `AG/internal/backup/runner.go:172`.
- Proxmox package step waits up to 2 min for tasks to drain, `pveDrainWait` `AG/internal/osupdate/leg.go:589`.
- Restore-test scratch teardown has a 2 min limit detached from the caller `AG/internal/reconcile/restoretest.go:485`.
- Local whole-guest retention default is 3 daily copies, `defaultLocalBackupKeepLast` `AG/internal/config/config.go:541`.
## Not found
- Any cron entry or `OnCalendar` timer on the host: grep `OnCalendar|cron` over the agent configs folder timers, services and shell files finds none; the only two timers are items 4 and 17.
- Any time-zone or time-sync step in the installer: grep `timedatectl|chrony|timesyncd|set-timezone|ntp` in `E/scripts/felhom-host-install.sh:1` finds nothing; items 11 and 12 use the host's local zone as found.
- A systemd watchdog for the agent: grep `WatchdogSec` over the agent configs folder finds nothing, against `AR/03-host-agent.md:72`.
- A rule in the agent for a suspended host (laptop lid): not searched beyond the controller's rule `AR/07-backup-architecture.md:555-558`; grep `^func Test.*(Clock|Future|Skew|Jump|DST|Monotonic)` in the agent repository finds only the tunnel re-key and stale-lock tests.
- Other files under the felhom.eu scripts folder that install a host timer: grep `timer|cron` named only `due_checks_gate.py`, `hostinstall-mode-harness.sh` and `test_hostinstall.py`, which are gates and tests and were not opened.
- SMART reads have no clock of their own: they ride each host report `AG/internal/storage/observe.go:184-193`.
@@ -0,0 +1,269 @@
# Clock list: in-guest controller (helper result, candidates only)
### 1. Night dump leg (databases and volumes) at W
1. `dbDumpLeg` `CT/cmd/controller/main.go:1336-1344` runs `RunDBDumps` `CT/internal/backup/backup.go:564`; owner: controller; started by the daily job "db-dump" `CT/cmd/controller/main.go:1345`. CUSTOMER.
2. Daily at W. W = household setting, else `DBDumpSchedule`, else `DefaultWindow` "02:30" `CT/internal/backupwindow/backupwindow.go:12`, decided in `EffectiveWindow` `CT/internal/backupwindow/backupwindow.go:86-94`. First leg. Zone: Europe/Budapest, hard-coded, `getBudapestLocation` `CT/internal/scheduler/scheduler.go:17-27`.
3. One database dump: 5 min limit `CT/internal/appbackup/dbdump.go:291`. Whole leg limit: NOT FOUND (grep `WithTimeout|WithDeadline` in backup.go gave only `CT/internal/backup/backup.go:983` and `CT/internal/backup/backup.go:1268`). Grows with number and size of databases and volumes.
4. Box off at W: `Missed` `CT/internal/nightchain/nightchain.go:197-218` sees it from the ledger and the catch-up runs it (block 4). The hub judges a missing dump at its 05:00 deadline `AR/07-backup-architecture.md:579-584`.
5. `withLeg` takes the leg lock `CT/cmd/controller/main.go:1321-1335`; `acquireRunning` refuses a second backup `CT/internal/backup/backup.go:1105-1113`; a job still running is skipped `executeJob` `CT/internal/scheduler/scheduler.go:354-364`. A running migration skips the leg and returns no error `CT/internal/backup/backup.go:565-568`.
6. Slot from the wall clock, waited on a monotonic timer `runDailyJob` `CT/internal/scheduler/scheduler.go:300-352`; fired over 60 min late = skipped `DailyLateLimit` `CT/internal/scheduler/scheduler.go:67`. See block 24 for the hour change.
7. Failure: `NotifyDBDumpFailed` `CT/cmd/controller/main.go:1339`. A made-up miss: one timeline line, never mailed `AR/07-backup-architecture.md:562-564`. A miss not made up: banner after 26 h `StaleAfter` `CT/internal/nightchain/banner.go:23`.
8. TEST `TestCatchUp_OffAtWThenOn_OneCatchUpAfter15Min` `CT/internal/nightchain/nightchain_test.go:92`; TEST `TestDaily_LateFireIsSkipped` `CT/internal/scheduler/scheduler_test.go:87`.
### 2. Second-drive copy leg at W+60 min
1. `RunAllTier2` `CT/internal/backup/tier2.go:534` copies each app's data to the second drive; owner: controller; started by the daily job "tier2-backup" `CT/cmd/controller/main.go:1442`. CUSTOMER.
2. Daily at W+60 min, `tier2OffsetMin` `CT/internal/backupwindow/backupwindow.go:17`; no setting of its own. It does not wait for the dump leg: "NOTHING waits on the leg before it" `AR/09-update-architecture.md:2751-2757`. Zone as block 1.
3. One copy: 60 min, `tier2CopyTimeout` `CT/internal/backup/tier2_copy_lifecycle.go:40`, applied at `CT/internal/backup/tier2.go:828`. No limit for the whole leg found (same grep as block 1). Grows with app count and data size.
4. Catch-up as block 4 (the leg is in the ledger through `withLeg` `CT/cmd/controller/main.go:1438-1441`). The leg function always returns nil `CT/cmd/controller/main.go:1438-1441`, so the scheduler never sees a failure.
5. Leg lock `CT/cmd/controller/main.go:1321-1335`. Skipped while a migration runs `CT/internal/backup/tier2.go:538-541`. `acquireRunning` in tier2.go: NOT FOUND (grep `acquireRunning\(\)|TryLock` in tier2.go, 0 hits). A stray copy container is retried every minute `tier2RetryEvery` `CT/internal/backup/tier2_copy_lifecycle.go:172`.
6. As block 1: wall-clock slot, monotonic wait, 60 min late limit `CT/internal/scheduler/scheduler.go:67`.
7. Who is told of a failed copy: NOT FOUND in the wiring (grep `Notify[A-Za-z]+\(` in main.go lines 1300-1460 shows only the dump and the off-site notices, `CT/cmd/controller/main.go:1339` and `CT/cmd/controller/main.go:1446-1452`). Candidate, the answer may sit inside tier2.go.
8. SOURCE `CT/internal/backup/tier2.go:534`; catch-up part TEST `TestCatchUp_PowerCutMidChain_FinishesTheRest` `CT/internal/nightchain/nightchain_test.go:145`.
### 3. Off-site push leg at W+105 min (with clean-up of old snapshots)
1. `RunOffboxBackup` `CT/internal/backup/offbox.go:1018` pushes dumps to the remote store, then runs retention; owner: controller; started by the daily job "offbox-backup" `CT/cmd/controller/main.go:1641-1643`. CUSTOMER.
2. Daily at W+105 min, `offboxOffsetMin` `CT/internal/backupwindow/backupwindow.go:18`. Retention keeps 7 daily, `keepDaily` `CT/internal/backup/offbox_window.go:47`. Zone as block 1.
3. Per app 2 h, `offboxBackupTimeout` `CT/internal/backup/offbox.go:39`, applied at `CT/internal/backup/offbox.go:1564`; SSH connect 10 s `CT/internal/backup/offbox.go:37`. Grows with data changed and line speed. One real night: 3 min 28 s `AR/09-update-architecture.md:2751-2757`.
4. Catch-up as block 4. No escrowed key, orphaned store or migration: skipped with an INFO log line and nil `CT/internal/backup/offbox.go:1040-1053`.
5. Another backup running: "skipped — another backup is running", returns nil, "the next scheduled run retries" `CT/internal/backup/offbox.go:1054-1057`. So a long second-drive leg can cost the night's off-site push (candidate).
6. As block 1. The clean-up window has a one-hour clock tolerance `offsiteGuardSkew` `CT/internal/backup/offbox_window.go:42`.
7. Failure: `NotifyOffboxBackupFailed` with a cleaned message `CT/cmd/controller/main.go:1446-1452`. The hub's daily deadline is the independent signal `CT/cmd/controller/main.go:1524-1528`.
8. SOURCE `CT/internal/backup/offbox.go:1040-1057`; DOC `AR/07-backup-architecture.md:455-457`.
### 4. Missed-night catch-up and resume watch
1. `CatchUp.Evaluate` `CT/internal/nightchain/nightchain.go:281-299` runs the backup legs a box missed; owner: controller; started at controller start `CT/cmd/controller/main.go:1676` and by the resume watch `CT/cmd/controller/main.go:1677`. CUSTOMER.
2. 15 min after the trigger, `DefaultDelay` `CT/internal/nightchain/nightchain.go:62`; a leg due within 30 min is left alone, `leaveToNormal` `CT/internal/nightchain/nightchain.go:64`. Values decided by CC `AR/07-backup-architecture.md:574-577`. No setting.
3. As long as the legs it runs. It waits up to 2 h for a whole-guest backup, `quiesceWaitMax` `CT/internal/nightchain/nightchain.go:66`, then drops with a WARN `CT/internal/nightchain/nightchain.go:319`.
4. If the catch-up itself does not run: banner after 26 h (block 5) and the hub's longer lines, 48 h without a dump `AR/07-backup-architecture.md:579-584`. Two missed nights give one catch-up, not two.
5. One at a time: pending/running guard `CT/internal/nightchain/nightchain.go:288-292`; leg lock `CT/cmd/controller/main.go:1321-1335`. It never runs the update leg `CT/cmd/controller/main.go:1660-1664`.
6. `ResumeWatch` compares wall clock and monotonic clock each minute; a gap over 5 min triggers, `d > gap` `CT/internal/nightchain/nightchain.go:375`, gap set at `CT/cmd/controller/main.go:1681`. A forward clock correction over 5 min looks the same as a resume (candidate). "not measured live" `AR/07-backup-architecture.md:551-559`.
7. Household: one timeline line, never mailed `AR/07-backup-architecture.md:562-564`, event sent at `CT/internal/nightchain/nightchain.go:357-359`. Dropped catch-up: log line only `CT/internal/nightchain/nightchain.go:319`.
8. TEST `TestCatchUp_TwoMissedNights_One` `CT/internal/nightchain/nightchain_test.go:126`; TEST `TestResumeWatch_SeesASuspend` `CT/internal/nightchain/nightchain_test.go:217`; TEST `TestR871_CatchUpWiring` `CT/cmd/controller/r871_catchup_wiring_test.go:16`.
### 5. Missed-backup banner (26 hours)
1. `ComputeBanner` `CT/internal/nightchain/banner.go:56-88` shows the household a banner when no night backup ran; owner: controller; computed when the page is drawn (no timer of its own). CUSTOMER.
2. Shown when the last backup is older than 26 h, `StaleAfter` `CT/internal/nightchain/banner.go:23`. After 5 missed days it suggests another hour, `suggestMinDays` `CT/internal/nightchain/banner.go:26`, `Suggest` `CT/internal/nightchain/banner.go:95-113`.
3. Instant; reads the ledger only `CT/internal/nightchain/banner.go:56-88`.
4. Nobody opens the page: nobody sees it. The hub's deadline check is the other channel `AR/07-backup-architecture.md:579-584`.
5. Not a job; no lock needed. Dismissed, it comes back after a new miss, design at `AR/07-backup-architecture.md:566-572`.
6. Age is wall-clock arithmetic `CT/internal/nightchain/banner.go:56-88`; a clock set back hides the banner for as long as the step (candidate). A test for a clock set back: NOT FOUND (grep `^func Test.*(Back|Clock)` in banner_test.go gave only the dismiss test).
7. The household only, on the page; no mail `AR/07-backup-architecture.md:566-572`.
8. TEST `TestBanner_ShownWithReasonAndSuggestion` `CT/internal/nightchain/banner_test.go:29`; TEST `TestBanner_DismissedThenBackAfterANewMiss` `CT/internal/nightchain/banner_test.go:59`.
### 6. Whole-guest backup driver (stop apps, ask the host agent, start apps)
1. `Loop.Run` `CT/internal/quiesce/quiesce.go:191-206` stops the apps and asks the agent for the whole-guest backup; owner: controller drives, agent does the backup; started by `startQuiesceLoop` `CT/cmd/controller/main.go:4008`. CUSTOMER.
2. Polls every 5 min `CT/internal/quiesce/quiesce.go:149-151`; runs once per 24 h cadence `CT/internal/quiesce/quiesce.go:161-163`, only inside W+2h to W+6h, `gateOpenOffsetMin` `CT/internal/quiesce/quiesce.go:760`, `gateSpanMin` `CT/internal/quiesce/quiesce.go:761`. Zone `budapestLocation` `CT/internal/quiesce/quiesce.go:747-757`.
3. Apps stay down at most 30 min, `MaxQuiesce` default `CT/cmd/controller/main.go:3990`, setting `max_quiesce_duration` `CT/internal/config/config.go:132`. Manual run limit = that + 5 min `CT/internal/quiesce/quiesce.go:430`. Grows with guest disk size.
4. Outside the window it still runs when the last one is older than cadence + 24 h (the 48 h valve) `CT/internal/quiesce/quiesce.go:801-818`. Apps left stopped by a crash are started by `Recover` `CT/internal/quiesce/quiesce.go:177-188`, called at `CT/cmd/controller/main.go:4007`.
5. Single-flight `TryLock` `CT/internal/quiesce/quiesce.go:213-216`; waits for a catch-up `CT/internal/quiesce/quiesce.go:250-253`; waits for the update leg `CT/internal/quiesce/quiesce.go:271-274`, never past W+5h30 `legStepGraceMin` `CT/internal/quiesce/quiesce.go:935`.
6. Gate is minute-of-day arithmetic on the wall clock `CT/internal/quiesce/quiesce.go:801-818`; the poll is a monotonic ticker `CT/internal/quiesce/quiesce.go:191-206`. Hour-change night: NOT FOUND a test (grep `DST|Budapest|SpringForward|Daylight` in `^func Test` lines, 0 hits).
7. First failure notifies once `noteTierFailure` `CT/internal/quiesce/quiesce.go:283-298`; retry back-off 15 min to 4 h `CT/internal/quiesce/breaker.go:47`, `CT/internal/quiesce/breaker.go:54`; busy agent: retry 15 min, alarm after 3 h `CT/internal/quiesce/contention.go:53`, `CT/internal/quiesce/contention.go:64`.
8. TEST `TestScheduledRunAllowed` `CT/internal/quiesce/quiesce_window_test.go:23`; TEST `TestRunOnce_ValveRunsOutsideWindow` `CT/internal/quiesce/quiesce_window_test.go:94`; TEST `TestBreaker_BackoffSchedule` `CT/internal/quiesce/breaker_test.go:218`.
### 7. Automatic app-update leg (after the off-site push)
1. `RunUpdateLeg` `CT/internal/stacks/unattended.go:230` presses one update step per app; owner: controller; started by `chainUpdateLeg` when the off-site half ends, on every path `CT/cmd/controller/main.go:1641-1643`. CUSTOMER.
2. Each night after the off-site leg; a box with no off-site tier runs it at W+105 min `AR/07-backup-architecture.md:459-465`. One step per app per night, switch default ON `AR/09-update-architecture.md:2806-2817`.
3. No new step at or after W+5h: `LegDeadline` `CT/internal/stacks/unattended.go:212-216`, `UpdateLegStopOffsetMin` `CT/internal/backupwindow/backupwindow.go:27`; check at `CT/internal/stacks/unattended.go:374-375`. Grows with the number of apps that have an update.
4. Missed night: no catch-up by design `CT/cmd/controller/main.go:1660-1664`; the rest "waits for the next night" `CT/internal/stacks/unattended.go:374-375`. Controller restarted mid-leg: the rest of the night is lost (R-686) `AR/09-update-architecture.md:2815-2817`.
5. A passing refusal (busy, updating) is retried once after 2 min, `RetryWait` `CT/internal/stacks/unattended.go:167-171`, rule at `AR/09-update-architecture.md:2882-2883`. The whole-guest gate waits for the leg `CT/internal/quiesce/quiesce.go:271-274`.
6. Deadline is a wall-clock time in `time.Local` when no location is given `CT/internal/stacks/unattended.go:212-216`, while the scheduler uses Europe/Budapest `CT/internal/scheduler/scheduler.go:17-27` (candidate: two sources of zone). A late daily fire over 60 min is skipped, so no noon updates `AR/07-backup-architecture.md:551-559`.
7. NOT FOUND in this read for who is told when the leg does not run (grep `update-leg` gave the log line `CT/internal/stacks/unattended.go:374-375` only).
8. TEST `TestChainUpdateLeg_EveryPath` `CT/cmd/controller/update_leg_wiring_test.go:19`; TEST `TestLegDeadlineAndGateShareW5h` `CT/internal/quiesce/d20_update_leg_test.go:78`.
### 8. Off-site integrity check at 06:00
1. Daily job "offsite-integrity" `CT/cmd/controller/main.go:1724` checks the remote store's data; owner: controller. CUSTOMER.
2. Fires daily at 06:00 (fixed, no setting for the hour) but works only when due: last good check older than 7 days, `defaultIntegrityMaxAgeDays` `CT/internal/backup/offbox_integrity.go:81`, `IntegrityDue` `CT/internal/backup/offbox_integrity.go:221`. Zone as block 1.
3. Limit 30 min, `integrityCheckTimeout` `CT/internal/backup/offbox_integrity.go:63`; a notice when slower than 5 min `CT/internal/backup/offbox_integrity.go:78`. Grows with store size and line speed.
4. Box off at 06:00: caught up at the next 06:00 by due-ness `CT/internal/backup/offbox_integrity.go:212-220`; also run after a catch-up `CT/cmd/controller/main.go:1665-1668`.
5. Skipped when "a backup or restore is already running" `CT/internal/backup/offbox_integrity.go:298`; then it waits for the next 06:00 (source comment `CT/cmd/controller/main.go:1710-1723`).
6. Due-ness compares wall-clock stamps `CT/internal/backup/offbox_integrity.go:231`; a never-checked or unreadable stamp counts as due `CT/internal/backup/offbox_integrity.go:223-229`. Slot behaviour as block 24.
7. NOT FOUND in this read (the job body `CT/cmd/controller/main.go:1724-1729` was read; the notify path inside offbox_integrity.go was not).
8. SOURCE `CT/internal/backup/offbox_integrity.go:212-232`; TEST `TestR1189d_TheChecksRunAfterTheCatchUpLegs` `CT/internal/nightchain/r1189d_test.go:13`.
### 9. Off-site proof restore at 05:30
1. Daily job "offsite-proof" `CT/cmd/controller/main.go:1746` restores one due snapshot to prove it opens, `ProveOffboxUnit` `CT/internal/backup/offbox_proof.go:140`; owner: controller. CUSTOMER.
2. Daily at 05:30, fixed, no setting `CT/cmd/controller/main.go:1746`. One app per night in rotation (comment `CT/internal/backup/offbox_proof.go:265`). Zone as block 1.
3. Limit 10 min, `proofRestoreTimeout` `CT/internal/backup/offbox_proof.go:80`. Grows with the size of the one unit.
4. Box on for the backup legs but off at 05:30: "that day's proof is not made up" `CT/cmd/controller/main.go:1660-1664`. Made up only together with a missed backup leg `CT/cmd/controller/main.go:1665-1668`.
5. Skipped when "a backup or restore is already running" `CT/internal/backup/offbox_proof.go:149`. With W = 02:30 the whole-guest gate is open from 04:30 `AR/09-update-architecture.md:2751-2757`, so 05:30 can fall inside it (candidate).
6. As block 24.
7. NOT FOUND in this read (grep `Notify` near `CT/cmd/controller/main.go:1746-1749`, 0 hits there).
8. SOURCE `CT/cmd/controller/main.go:1660-1664`; TEST `TestC5_TheWholeGuestBackupDoesNotWaitForTheChecks` `CT/internal/nightchain/r1189d_test.go:51`.
### 10. Hub report every 15 minutes (plus push on change and hanging wait)
1. Job "hub-report" `CT/cmd/controller/main.go:1835` sends the box state to the hub; owner: controller. The hub's answer carries the version floor and config `CT/cmd/controller/main.go:1233-1290`. CUSTOMER and OPERATOR.
2. Every `PushInterval`, default "15m" `CT/internal/config/config.go:416`, fallback `CT/cmd/controller/main.go:1831-1834`. Also 5 s after start `CT/cmd/controller/main.go:1932-1936`, and on change with a 15 s floor, `triggerMinInterval` `CT/internal/report/trigger.go:30`. No time zone.
3. HTTP limit 30 s `CT/internal/report/pusher.go:85`; 3 tries, 5 s apart `CT/internal/report/pusher.go:114-117`. The wait connection holds up to 300 s `CT/internal/report/waiter.go:35`.
4. A failed push is not queued: "next cycle reconciles" `CT/internal/report/trigger.go:98`. The hub notices silence (hub side, not read here).
5. A report still running is skipped `CT/internal/scheduler/scheduler.go:354-364`. Bursts are merged after a 2 s quiet window `CT/internal/report/trigger.go:27`.
6. Monotonic ticker `CT/internal/scheduler/scheduler.go:281-298`; a wrong wall clock does not change the rhythm, only the stamps inside the report.
7. Waiter back-off 5 s to 5 min `CT/internal/report/waiter.go:36-37`. Who is told on the box that pushes fail: NOT FOUND (only `LastAttempt` is stored `CT/internal/report/pusher.go:109-111`).
8. TEST `TestTrigger_BurstCoalescesTrailingEdge` `CT/internal/report/trigger_test.go:114`; TEST `TestWaiter_FiresOnceOnGenChange` `CT/internal/report/waiter_test.go:60`.
### 11. Controller self-update: check, floor, nightly auto
1. `CheckForUpdate` `CT/internal/selfupdate/updater.go:153` looks for a newer controller image; `MaybeAutoUpdate` `CT/internal/selfupdate/updater.go:612` moves a box below the floor. Owner: controller, the agent swaps the image. CUSTOMER and OPERATOR.
2. Check every `CheckInterval`, default "6h" `CT/internal/config/config.go:417`, and 25 s after start `CT/cmd/controller/main.go:1981-1989`. Floor: after every hub report `CT/cmd/controller/main.go:1249-1252`. Auto: daily at `AutoUpdateTime`, default "04:30" `CT/internal/config/config.go:419`, only if `AutoUpdate` `CT/internal/config/config.go:278`.
3. Registry call limit 15 s `CT/internal/selfupdate/updater.go:245`. Limit of the swap itself: NOT FOUND in this read (grep `time\.(Second|Minute|Hour)` in updater.go gave one hit).
4. No catch-up needed for the floor: it is tried again on each report until done, but one attempt per floor value `CT/internal/selfupdate/updater.go:656-661`. A failed attempt at a floor is not repeated `CT/internal/selfupdate/updater.go:604-606`.
5. `updateRunning` flag `CT/internal/selfupdate/updater.go:520-524`; refused while a backup runs `CT/internal/selfupdate/updater.go:533-536` or an app update is in flight `CT/internal/selfupdate/updater.go:540-544`. Default 04:30 equals gate-open time at W = 02:30 (candidate).
6. Check: monotonic ticker. Auto: daily slot as block 24. Default for `AutoUpdate`: NOT FOUND in the defaults block (grep `AutoUpdate ` in config.go: only the field `CT/internal/config/config.go:278`).
7. Auto-update refused: WARN log only `CT/cmd/controller/main.go:1879-1881`. A version the floor cannot compare: WARN once `CT/internal/selfupdate/updater.go:639`.
8. TEST `TestMaybeAutoUpdate_NoFlap` `CT/internal/selfupdate/floor_test.go:93`; TEST `TestMaybeAutoUpdate_BelowFloor_UpdatesToFloor` `CT/internal/selfupdate/floor_test.go:11`.
### 12. Catalog sync (app templates from git)
1. `Syncer.Start` `CT/internal/sync/sync.go:125` pulls the app catalog; owner: controller; started at controller start. CUSTOMER.
2. Once at start `CT/internal/sync/sync.go:140-143`, then every `SyncInterval`, default "15m" `CT/internal/config/config.go:399`; a bad value falls back to 15 min `CT/internal/sync/sync.go:131-135`. No time zone.
3. Time limit for one sync: NOT FOUND in the lines read (`CT/internal/sync/sync.go:124-188`; `doSync` body not read).
4. Missed sync: the next tick does it. Empty repo address: sync is off with one WARN `CT/internal/sync/sync.go:126-129`. Who notices a long failure: NOT FOUND.
5. Manual sync is refused while one runs or within 30 s of the last `CT/internal/sync/sync.go:175-185`. The periodic tick calls `doSync` directly without that check `CT/internal/sync/sync.go:153-155` (candidate: read `doSync` before judging).
6. Monotonic ticker `CT/internal/sync/sync.go:146`; own ticker, not the shared scheduler, so it has no "still running" skip from `CT/internal/scheduler/scheduler.go:354-364`.
7. Log line per sync only `CT/internal/sync/sync.go:155`.
8. SOURCE `CT/internal/sync/sync.go:124-159`.
### 13. System health check and base-stack self-heal
1. Job "system-health" `CT/cmd/controller/main.go:981-1006` checks disk, memory and base services, and puts the base stack back; owner: controller. CUSTOMER.
2. Every `SystemHealthInterval`, default "5m" `CT/internal/config/config.go:409`, fallback `CT/cmd/controller/main.go:977-980`; env override `CT/internal/config/config.go:453`. First run one interval after start `CT/internal/scheduler/scheduler.go:281-298`.
3. Time limit: NOT FOUND in the job body `CT/cmd/controller/main.go:981-1006`.
4. No catch-up; next tick. A controller that is down checks nothing; the hub's missing reports are the outside sign (block 10).
5. Skipped while the previous run is still going `CT/internal/scheduler/scheduler.go:354-364`.
6. Monotonic ticker; not sensitive to wall clock.
7. `NotifyHealthChange` on a change of state `CT/cmd/controller/main.go:1004`.
8. SOURCE `CT/cmd/controller/main.go:981-1006`.
### 14. Dead-app scan every 30 seconds
1. Job "deadapp-check" `CT/cmd/controller/main.go:1086-1092` finds apps that should run and do not; owner: controller. CUSTOMER.
2. Every 30 s; silent for 90 s after start, about 15 min after a crash boot `CT/cmd/controller/main.go:1087-1088`.
3. Time limit: NOT FOUND (job body `CT/cmd/controller/main.go:1086-1143` has no context limit). Grows with app count.
4. Next tick; no catch-up needed.
5. Skipped if still running `CT/internal/scheduler/scheduler.go:354-364`. It reads the whole-guest loop state so a planned stop is not an alarm `CT/cmd/controller/main.go:1090`.
6. Boot grace uses `time.Now()` `CT/cmd/controller/main.go:1087`; a clock stepped back after start would lengthen the silence (candidate, no test found).
7. Alert on the page via `SetDeadAppAlerts` `CT/cmd/controller/main.go:1091`.
8. SOURCE `CT/cmd/controller/main.go:1086-1092`.
### 15. Disk-fill watch
1. `fillWatcher.Check` watches how full the drives are; owner: controller; three starters `CT/cmd/controller/main.go:1804-1805`. CUSTOMER.
2. Every 10 min, `fillWatchInterval` `CT/cmd/controller/main.go:2690`; daily at 03:30 `CT/cmd/controller/main.go:1804`; once 90 s after start, `fillWatchStartupDelay` `CT/cmd/controller/main.go:2685`. Cadence stated at `CT/cmd/controller/main.go:1772`.
3. Time limit: NOT FOUND (the callbacks ignore the context `CT/cmd/controller/main.go:1804-1805`).
4. Next 10-minute tick.
5. The 03:30 slot and the interval job are two jobs with two names `CT/cmd/controller/main.go:1804-1805`; a lock between them: NOT FOUND in the wiring (the fillwatch package was not read).
6. Interval is monotonic; the 03:30 slot as block 24.
7. NOT FOUND in this read (fillwatch package not read).
8. TEST `TestFillWatchRunsOnAnInterval` `CT/cmd/controller/r363_fillwatch_interval_test.go:17`.
### 16. Drive health check, hourly
1. Job "disk-health-check" `CT/cmd/controller/main.go:2226` asks the agent for drive health; owner: controller reads, agent measures. CUSTOMER.
2. Every 1 h; reason for hourly, not 6 h: `CT/cmd/controller/main.go:2219-2224`. Only when the agent channel is configured `CT/cmd/controller/main.go:2225`.
3. One fetch measured about 0.8 s `CT/cmd/controller/main.go:2219-2224`. Limit: NOT FOUND.
4. Next hour; first run one hour after start `CT/internal/scheduler/scheduler.go:281-298`.
5. Skipped if still running `CT/internal/scheduler/scheduler.go:354-364`.
6. Monotonic ticker.
7. NOT FOUND in this read (`RunDiskHealthCheck` body not read).
8. SOURCE `CT/cmd/controller/main.go:2219-2226`.
### 17. Agent channel health, every 60 seconds
1. Job "agent-channel-health" `CT/cmd/controller/main.go:2289` probes the controller-to-agent channel; owner: controller. OPERATOR.
2. Every 60 s. A passing fault needs 2 misses in a row, `debounceThreshold` `CT/internal/channelhealth/checker.go:29`.
3. Probe limit 15 s, `probeTimeout` `CT/internal/channelhealth/checker.go:32`.
4. Next minute.
5. Skipped if still running `CT/internal/scheduler/scheduler.go:354-364`. An alert marker file survives a restart `CT/cmd/controller/main.go:2288`.
6. Monotonic ticker.
7. Sink = notifier and page alert `CT/cmd/controller/main.go:2286`.
8. TEST `TestDebounce_TransientBlipSuppressed` `CT/internal/channelhealth/checker_test.go:119`.
### 18. Held-app report to the hub
1. `Reporter.Tick` `CT/internal/holdreport/holdreport.go:89` tells the hub which apps are held; owner: controller; job "hold-report" `CT/cmd/controller/main.go:1081-1084`. OPERATOR.
2. Looks every minute; each hold is sent again once a day, `ResendEvery` `CT/internal/holdreport/holdreport.go:36`, and after each start `CT/cmd/controller/main.go:1071-1073`.
3. Instant; reads the hold store.
4. Next minute.
5. Skipped if still running `CT/internal/scheduler/scheduler.go:354-364`.
6. `Tick` gets `time.Now()` `CT/cmd/controller/main.go:1082`; the daily resend is wall-clock arithmetic (candidate: a clock set back delays the resend).
7. The hub, by event `CT/cmd/controller/main.go:1074`.
8. TEST `TestD64_TheHoldReporterTellsTheHubAfterTheClear` `CT/cmd/controller/holdclear_test.go:333`.
### 19. Abandoned off-site store sweep at 05:10
1. Daily job "offsite-abandon-sweep" `CT/cmd/controller/main.go:1693` runs `AbandonSweep` `CT/internal/backup/offbox_abandon.go:214`; owner: controller. CUSTOMER.
2. Daily at 05:10, fixed. The due date is counted in UTC days `CT/internal/backup/offbox_abandon.go:366`, the slot in Europe/Budapest `CT/internal/scheduler/scheduler.go:17-27`.
3. NOT FOUND (body of `AbandonSweep` not read).
4. Box off at 05:10: next day; no catch-up (it is not a ledger leg, see `CT/cmd/controller/main.go:1660-1664`).
5. NOT FOUND (grep `acquireRunning\(\)|TryLock` in offbox_abandon.go, 0 hits).
6. As block 24.
7. NOT FOUND in this read.
8. SOURCE `CT/cmd/controller/main.go:1693`.
### 20. Restore-copy sweep and conversion-copy release
1. `SweepUnusedRestoreCopies` `CT/internal/backup/restore_copy_sweep.go:36` deletes old kept copies; job "restore-copy-sweep" `CT/cmd/controller/main.go:1414`; job "conversion-copy-release" `CT/cmd/controller/main.go:1407`. CUSTOMER.
2. Sweep every 6 h; release every 1 h. Age line: the wiring comment says 14 days `CT/cmd/controller/main.go:1412`, the constant says 30 days, `restoreCopyUnusedAge` `CT/internal/backup/restore_copy_sweep.go:29` (candidate: comment and constant differ).
3. NOT FOUND.
4. Next tick; first run 6 h after start `CT/internal/scheduler/scheduler.go:281-298`, so a box on for under 6 h a day never sweeps (candidate).
5. Skipped if still running `CT/internal/scheduler/scheduler.go:354-364`.
6. Age is wall-clock arithmetic on `now` `CT/internal/backup/restore_copy_sweep.go:36`.
7. NOT FOUND.
8. SOURCE `CT/internal/backup/restore_copy_sweep.go:29`.
### 21. Session clean-up (dashboard sign-ins)
1. `cleanupSessions` `CT/internal/web/auth.go:369` removes expired sign-in sessions; owner: controller web server. CUSTOMER.
2. Every 15 min `CT/internal/web/auth.go:370`. A session lives 7 days, `sessionMaxAge` `CT/internal/web/auth.go:30`. Family sessions 30 days, `SessionLife` `CT/internal/family/family.go:42`; share cookie 30 days `CT/internal/web/share.go:31`.
3. Instant; grows with session count.
4. Not run: expired sessions stay in the file. Whether an expired session is refused at use: NOT FOUND in the lines read (`CT/internal/web/auth.go:355-395`).
5. `sessionsMu` lock `CT/internal/web/auth.go:377`.
6. Expiry compares the wall clock, `now.After(sess.expiresAt)` `CT/internal/web/auth.go:381`; a clock set forward signs everyone out, set back keeps sessions longer (candidate).
7. Log line only `CT/internal/web/auth.go:392`.
8. SOURCE `CT/internal/web/auth.go:369-393`.
### 22. Upload and CPU watch, torrent limit rule, line-test hold
1. `startUpwatch` `CT/cmd/controller/upwatch_wiring.go:136` runs the minute watcher and a 15-minute rule that holds the torrent app for the line test; owner: controller. CUSTOMER.
2. Watcher every 1 min `CT/cmd/controller/upwatch_wiring.go:139`; rule every 15 min `CT/cmd/controller/upwatch_wiring.go:141`. Baseline: 7 days of data, cached 24 h `CT/cmd/controller/upwatch_wiring.go:124-127`.
3. Hold call limit 30 s, `holdTimeout` `CT/internal/upwatch/linehold.go:79`; drain wait at most 90 s `CT/internal/upwatch/linehold.go:216`.
4. A limit left at the test value by a stopped controller is put back at start, `RecoverHold` `CT/cmd/controller/upwatch_wiring.go:137`.
5. Own tickers; no scheduler skip. Lock between the two loops: NOT FOUND in the lines read.
6. Monotonic tickers `CT/internal/upwatch/watch.go:151`.
7. NOT FOUND in this read (`emit` `CT/internal/upwatch/watch.go:382` not followed).
8. SOURCE `CT/cmd/controller/upwatch_wiring.go:134-152`.
### 23. Old image clean-up after start
1. `RunImageRetentionOnceUntilDone` `CT/internal/stacks/image_retention.go:399` deletes app images no app uses; owner: controller; started about 3 min after start `CT/cmd/controller/main.go:2048-2069`. CUSTOMER.
2. Once per start, retried every 2 min up to 30 times `CT/cmd/controller/main.go:2056`. Also after an update or a remove (test names below).
3. Up to about 60 min of retries (30 x 2 min) `CT/cmd/controller/main.go:2056`.
4. A box that never restarts and never updates runs no sweep (candidate).
5. No pass while an update runs: TEST `TestImageRetention_NoPassWhileAnUpdateRuns` `CT/internal/stacks/image_retention_test.go:238`.
6. Monotonic waits.
7. Log only; an unreadable keep set deletes nothing `CT/internal/stacks/image_retention.go:221`.
8. TEST `TestImageRetention_KeepsRunningAndPreviousDeletesOlder` `CT/internal/stacks/image_retention_test.go:111`.
### 24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)
1. `Daily` `CT/internal/scheduler/scheduler.go:131-157` and `Every` `CT/internal/scheduler/scheduler.go:105-127` carry every named job; owner: controller. CUSTOMER.
2. Daily slots are HH:MM in Europe/Budapest, hard-coded; if the zone data cannot load it silently uses UTC `CT/internal/scheduler/scheduler.go:17-27`. Interval jobs do not run at start: first run after one interval `CT/internal/scheduler/scheduler.go:281-298`.
3. No time limit of its own; shutdown waits 30 s `CT/internal/scheduler/scheduler.go:264`. A panic in a job is caught `CT/internal/scheduler/scheduler.go:438-447`.
4. A daily job has no catch-up in the scheduler; a fire over 60 min late is skipped `CT/internal/scheduler/scheduler.go:344-347`. Only the three backup legs are made up (block 4).
5. One run per job at a time: "still running" skip `CT/internal/scheduler/scheduler.go:354-364`. No rule between different jobs here.
6. Next slot by calendar day, not +24 h `CT/internal/scheduler/scheduler.go:499-519`. Spring night: 02:30 does not exist; autumn night: 02:30 comes twice. Test for either: NOT FOUND (grep `DST|Budapest|SpringForward|Daylight` in `^func Test`, 0 hits). A clock set back while waiting: the timer still fires on monotonic time.
7. A skipped late fire is a log line only `CT/internal/scheduler/scheduler.go:344-347`.
8. TEST `TestDaily_LateFireIsSkipped` `CT/internal/scheduler/scheduler_test.go:87`; TEST `TestUpdateDaily_GoroutineConsumesReschedule` `CT/internal/scheduler/scheduler_test.go:62`.
## Tiny
- Status refresh every 10 s `CT/cmd/controller/main.go:950`.
- Stack scan every 2 min `CT/cmd/controller/main.go:953`.
- Docker socket watch every 15 s and socket users every 5 min `CT/cmd/controller/main.go:960-962`.
- Health probes every 10 s `CT/cmd/controller/main.go:971`.
- After-unclean-boot repair, every 30 s until done, own `TryLock` `CT/cmd/controller/main.go:1064-1065`.
- Debug ring spill every 30 s `CT/cmd/controller/main.go:1148-1150`.
- Off-site credential retry every 5 min `CT/cmd/controller/main.go:1375`.
- Backup cache refresh every 5 min `CT/cmd/controller/main.go:1398-1402`.
- Metrics sample every 60 s `CT/internal/metrics/collector.go:53`.
- Metrics prune daily 04:00, keeps 30 days `CT/cmd/controller/main.go:1754`.
- Asset sync daily at `SyncSchedule`, default "05:00" `CT/internal/config/config.go:425`.
- Geo-restriction verify every 6 h `CT/cmd/controller/main.go:2102`.
- Setup-gate loop every 20 s `CT/cmd/controller/main.go:2045`.
- Busy marker written every 15 s `CT/internal/busymark/busymark.go:29`.
- Startup report: 5 s after start `CT/cmd/controller/main.go:1932-1936`.
- Stray second-drive copy container retry every minute `CT/internal/backup/tier2_copy_lifecycle.go:172`.
- App wait after install: 10 min `CT/cmd/controller/main.go:2194`.
- Family sign-in form lives 1 h `CT/internal/web/family_gate.go:56`.
- Stale pointers inside the design text: the legs are cited at old main.go lines `AR/07-backup-architecture.md:455-457` (candidate, hygiene).
## Not found
- Log rotation: NOT FOUND in controller Go code. Grep `rotat|lumberjack|max-size|MaxSizeMB` found only the config fields `CT/internal/config/config.go:291-292` and their defaults `CT/internal/config/config.go:422-423`; no reader of them.
- Share-link expiry sweep: NOT FOUND. Grep `expire|expiry|ExpiresAt` in `internal/web/share*.go` gave no sweep; only the 30-day cookie `CT/internal/web/share.go:31`.
- A test for the hour-change nights or a named time zone: NOT FOUND (grep `DST|Budapest|SpringForward|Daylight` over `^func Test` in cmd and internal, 0 hits).
- Who is told when the second-drive copy, the proof, the integrity check, the fill watch or the drive health check fails: not read to the end; see line 7 of blocks 2, 8, 9, 15, 16.
- Not read at all: `CT/internal/bootrecon/bootrecon.go:147`, `CT/internal/bootstrap/waiting.go:95`, `CT/internal/offsiteapply/offsiteapply.go:137`, `CT/internal/web/operator_reboot_notice.go:78`, `CT/internal/web/handler_export_upload.go:204`, `CT/internal/web/media_events.go:97`, `CT/internal/notify/notifier.go:324`.
@@ -0,0 +1,333 @@
# Clock list: hub jobs and timers outside product code (helper result, candidates only)
### 1. Box liveness: "your server cannot be reached"
1. CUSTOMER. The hub's 60 s sweep runs `stalenessChecker.Check` and `hostStaleness.Check` `E/hub/cmd/hub/main.go:935-936`; they raise `host_stale`, `host_down`, `host_recovered` `HUB/monitor/host_staleness.go:207-221`. Owner: hub pod. Merged here: the controller-report checker and the host-report checker.
2. Every 60 s `E/hub/cmd/hub/main.go:927`. Stale after `stale_threshold`, default `"30m"` `E/hub/cmd/hub/main.go:1027`, deployed 45 m `E/manifests/hub.yaml:88`; down after 2x `HUB/monitor/host_staleness.go:39`. No time zone: ages are durations `HUB/monitor/host_staleness.go:131`.
3. No time limit; the checkers run one after another in one goroutine `E/hub/cmd/hub/main.go:935-962`. One store read of all hosts per pass `HUB/monitor/host_staleness.go:113`, then a loop per host; grows linearly with boxes. Measured duration: NOT FOUND (grep `time.Since(start)` in `E/hub/cmd/hub/main.go:927-965`).
4. Pod down: nothing is judged. At start a box already silent is left unseeded and the first pass alarms once `HUB/monitor/host_staleness.go:59-60`, unless that alarm was already raised `HUB/monitor/host_staleness.go:62-68`. The controller checker seeds stale/down silently `HUB/monitor/staleness.go:86-91` (candidate: differs from the host checker).
5. One pod, `replicas: 1` `E/manifests/hub.yaml:122`; state under a mutex `HUB/monitor/host_staleness.go:119-120`. Events fire on transitions only `HUB/monitor/host_staleness.go:170-172`. A household power-off holds alarms `HUB/monitor/host_staleness.go:134-136` for `DeliberateOffHold` `HUB/store/deliberateoff.go:18`.
6. Judged on the hub's receive time, not the box clock: `last_report_at = datetime('now')` `HUB/store/store.go:3916`. A wrong box clock cannot hide silence. A hub clock jump moves every age at once `HUB/monitor/host_staleness.go:131`; DST has no effect (durations).
7. Operator: every edge; liveness events bypass the 1 h quiet hour, 5 min dedupe `HUB/notify/dispatcher.go:609-612`. Household: at most once per 7 days, `householdLivenessQuiet` `HUB/notify/dispatcher.go:1018`. If the hub itself is down nobody is told by this job.
8. TEST `TestHostStaleness_R1001_RestartDoesNotReMail` `HUB/monitor/r1001_restart_test.go:56`; TEST `TestR873_HouseholdHearsAnOutageAtMostWeekly` `HUB/notify/r873_liveness_weekly_test.go:17`; DOC `AR/05-hub-architecture.md:88-118`.
### 2. Missed-backup deadline check (05:00)
1. CUSTOMER. `scheduleDaily(ctx, "deadline-check", "05:00", ...)` `E/hub/cmd/hub/main.go:986-988` runs `CheckBackupDeadlines` `HUB/monitor/deadline.go:409`; it raises `expected_backup_missed` `HUB/monitor/deadline.go:539-544`. Owner: hub pod.
2. Once a day at 05:00 Budapest wall-clock; the slot is a literal in `E/hub/cmd/hub/main.go:986`, no setting. Zone chosen in `scheduleDaily` `E/hub/cmd/hub/main.go:1072`. Limits: `backupStaleAfter` 26 h `HUB/monitor/deadline.go:29`, off-site `offsiteBackupStaleAfter` 8 d `HUB/monitor/deadline_tiers.go:28`.
3. No time limit; runs inline in the scheduler goroutine `E/hub/cmd/hub/main.go:1090`. Loops over every customer and reads up to 7 days of evidence, `backupEvidenceLookback` `HUB/monitor/deadline.go:36`; grows with boxes times reports. Only a summary line is logged `HUB/monitor/deadline.go:569-570`; duration NOT FOUND (grep `duration|took`).
4. Pod down at 05:00: the day's check is skipped, no catch-up; `nextDailyRun` moves to the next calendar day `E/hub/cmd/hub/main.go:1095-1112` and `last` lives in memory `E/hub/cmd/hub/main.go:1069-1093`. Nobody notices a skipped check (candidate).
5. Cannot run twice for one slot: the `last` guard in `nextDailyRun` `E/hub/cmd/hub/main.go:1095-1112`. No lock against the 60 s sweep; they are separate goroutines `E/hub/cmd/hub/main.go:927` and `E/hub/cmd/hub/main.go:986`. A down box is judged on longer lines, 48 h / 72 h `HUB/monitor/deadline.go:340-343`.
6. Freshness is the box's own backup timestamp against the hub's now `HUB/monitor/deadline.go:243`, parsed by `parseBackupTime` `HUB/monitor/deadline.go:290-299`. A box clock that runs ahead would look fresh (candidate; guard NOT FOUND, grep `future|skew`). "Since midnight" is Budapest midnight `HUB/monitor/deadline.go:417-419`.
7. Household and operator through the dispatcher on `expected_backup_missed` `HUB/monitor/deadline.go:539-544`; at the 05:00 run only. A box that never bound is "unknown", not alarmed `HUB/monitor/deadline.go:486-494`; disabled / deliberately-off are skipped `HUB/monitor/deadline.go:432-442`.
8. TEST `TestR872_DownEveryNightRaisesTheMissedAlarms` `HUB/monitor/r872_down_box_test.go:57`; TEST `TestR1183_TheSameSlotNeverRunsTwice` `E/hub/cmd/hub/r1183_dst_schedule_test.go:82`. A skipped day: NOBODY KNOWS.
### 3. Mail cooldowns and the weekly outage mail
1. CUSTOMER. Not a job but a clock rule inside every mail: `processCustomer` `HUB/notify/dispatcher.go:1020-1098` and `processOperator` `HUB/notify/dispatcher.go:626-696` decide whether an alarm is mailed now or swallowed. Owner: hub dispatcher.
2. Operator: 1 h, liveness 5 min `HUB/notify/dispatcher.go:609-612`, fill `storageFillCooldown` 6 h `HUB/notify/dispatcher.go:573`. Household: default 6 h `HUB/notify/dispatcher.go:1065-1069`; outage mails weekly `HUB/notify/dispatcher.go:1054-1062`. A new box's first hour is quiet, `firstHourOfABox` `HUB/notify/dispatcher.go:1281`.
3. Per event, no loop. HTTP client limit 10 s `HUB/notify/dispatcher.go:86`. The cooldown maps grow with customers times event types; an entry is removed only on a mail hold `HUB/notify/dispatcher.go:684` or by `forgetCustomerCooldown` `HUB/notify/dispatcher.go:1180-1183`; a prune by age: NOT FOUND (grep `delete(d.`).
4. Pod restart: the cooldown maps are in memory, so quiet periods restart from zero; only the 7-day household rule is read from the persisted notification log `HUB/notify/dispatcher.go:1054-1062`. Design text: `AR/08-alarm-ladder.md:435-439`.
5. The operator cooldown is armed before the send `HUB/notify/dispatcher.go:674`, so two alarms close together collapse to one; a suppression is recorded `HUB/notify/dispatcher.go:665-672`. A MAIL-HOLD drops the mail `HUB/notify/dispatcher.go:680-687`.
6. Cooldowns use the hub clock only. A hub clock jump back would stretch a quiet period, forward would end it early; handling: NOT FOUND (grep `monotonic` in the dispatcher file).
7. A swallowed operator alarm leaves a "suppressed" row `HUB/notify/dispatcher.go:665-672`. A failed household mail is logged and NOT retried `HUB/notify/dispatcher.go:1091-1094` (candidate: only operator mails are retried, see block 4).
8. TEST `TestOperatorCooldown_NodeLivenessBypassesQuietHour` `HUB/notify/node_liveness_cooldown_test.go:17`; TEST `TestR873_HouseholdHearsAnOutageAtMostWeekly` `HUB/notify/r873_liveness_weekly_test.go:17`; DOC `AR/08-alarm-ladder.md:165-168`.
### 4. Retry of a failed operator mail
1. OPERATOR. `retryOperatorEmail` `HUB/notify/dispatcher.go:99-104` re-sends an operator mail whose send failed; started by the failure branch of `processOperator` `HUB/notify/dispatcher.go:688-692`.
2. After 1, 5 and 15 minutes: `operatorRetryDelays` `HUB/notify/dispatcher.go:76`, armed with `afterFn = time.AfterFunc` `HUB/notify/dispatcher.go:92`. No setting. Durations, no time zone.
3. Each try is bounded by the 10 s client timeout `HUB/notify/dispatcher.go:86`. One timer per failed mail; grows only with failures.
4. Pod restart inside the 21 minutes: the pending timers are in memory `HUB/notify/dispatcher.go:92` and are lost; a catch-up at start: NOT FOUND (grep `retryOperatorEmail` under hub/cmd). After the third failure: an ERROR log line only `HUB/notify/dispatcher.go:101`.
5. The cooldown for that alarm was armed before the first send `HUB/notify/dispatcher.go:674`, so a repeat alarm in the same hour does not mail either; the retry is the only path. Explained in `AR/08-alarm-ladder.md:358-363`.
6. Timers are Go durations; DST has no effect. A hub wall-clock jump: NOBODY KNOWS (not tested).
7. Nobody is mailed when the retries give up; the trace is the ERROR line `HUB/notify/dispatcher.go:101` and `failed` rows in the notification log `AR/08-alarm-ladder.md:362-363`.
8. TEST `TestOperatorMailRetry_FailedOperatorMailIsRetried` `HUB/notify/operator_mail_retry_test.go:17`. Restart during the retry window: NOBODY KNOWS.
### 5. Kernel "your box restarts tonight" mail
1. CUSTOMER. `KernelNotify` `HUB/osupdates/kernel.go:191` mails each due household the day before a kernel step; called from `Evaluate` `HUB/osupdates/service.go:436`, which the 60 s OS ticker calls `E/hub/cmd/hub/main.go:564`.
2. Every 60 s, but acts only 09:00-20:00 Budapest: `KernelNoticeFromHour`/`KernelNoticeToHour` `HUB/osupdates/kernel.go:46-47`, checked at `HUB/osupdates/kernel.go:193-196`. Valid 24 h, gap 20 h, at most 3 mails `HUB/osupdates/kernel.go:48-50`. Zone from `budapestLoc` `HUB/osupdates/kernel.go:73-78`. Constants, no setting.
3. No time limit. Loops all hosts every minute with store reads per host `HUB/osupdates/kernel.go:203-208`; grows linearly with boxes, 1440 passes a day.
4. Pod down all day 09-20: no mail, and without a mail there is no step: `KernelBlockFor` sets `Tonight` only for a notice younger than 24 h `HUB/osupdates/kernel.go:179-185`. It mails on the next daytime tick; the notices are persisted `HUB/osupdates/kernel.go:236`.
5. Twice is blocked by the stored notice and the 20 h gap `HUB/osupdates/kernel.go:208-211`; a restart cannot double-mail because the row is in the store. A mail sent but not recorded means no step tonight `HUB/osupdates/kernel.go:236-239`.
6. Hub clock decides the daytime window. A mail sent at 19:59 is valid until 19:59 next day `HUB/osupdates/kernel.go:182`, so it covers one night. DST: the hour test uses Budapest local time `HUB/osupdates/kernel.go:193-194`; the 20 h / 24 h are durations. A deliberately-off box gets no mail `HUB/osupdates/kernel.go:201`.
7. Mail failure: operator event `os_kernel_notice` warning, once per box, kernel and day `HUB/osupdates/kernel.go:226-233`. After 3 mails with no step: operator warning, once `HUB/osupdates/kernel.go:212-219`.
8. TEST `TestKernel_AtMostThreeMails` `HUB/osupdates/kernel_test.go:192`; TEST `TestKernel_DayBeforeMailMakesTonight` `HUB/osupdates/kernel_test.go:79`; TEST `TestKernel_NoMailNoStep` `HUB/osupdates/kernel_test.go:115`. A mail sent late in the day vs the box's night slot: NOBODY KNOWS.
### 6. OS update approval tick and hourly OS alarms
1. OPERATOR (the outcome reaches households as updates). 60 s ticker `E/hub/cmd/hub/main.go:554-586`: `osSvc.Evaluate()` `E/hub/cmd/hub/main.go:564` approves a set when its rule holds; `osSvc.Alarms()` `E/hub/cmd/hub/main.go:577` raises the fleet alarms.
2. Evaluate every 60 s; Alarms when `time.Since(lastAlarms) >= time.Hour` `E/hub/cmd/hub/main.go:574`. Rule default `ApproveAfter: 24 * time.Hour, NightsRequired: 1` `E/hub/cmd/hub/main.go:476`, env `OS_APPROVE_AFTER` `E/hub/cmd/hub/main.go:484-491`. Alarm limits 7 d / 14 d, env table `E/hub/cmd/hub/main.go:536-541`.
3. No time limit. The alarm pass logs its own duration `E/hub/cmd/hub/main.go:581`. Evaluate walks every layer `HUB/osupdates/service.go:414-422`; grows with boxes times layers, every minute.
4. Pod restart: `lastAlarms` is a local variable `E/hub/cmd/hub/main.go:558`, zero at start, so the first tick runs the alarms at once. Re-mail is prevented by the stored raise time, 7 days `HUB/osupdates/service.go:1102`.
5. One goroutine, so Evaluate and Alarms never overlap `E/hub/cmd/hub/main.go:559-585`. `raise` clears the mark when the condition ends so the next occurrence is announced `HUB/osupdates/service.go:1094-1108`.
6. Ages use the hub clock against stored times `HUB/osupdates/service.go:1102`. A box's claimed reboot time is allowed 1 minute of skew, `rebootClaimSkew` `HUB/osupdates/service.go:970`. Larger box-clock error: NOBODY KNOWS.
7. Operator, by event through the dispatcher: `os_update_stale`, `os_ring0_stalled` and others, listed in `AR/08-alarm-ladder.md:375-383`. Repeats weekly while it holds `HUB/osupdates/service.go:1092-1093`.
8. TEST `TestAlarm_Ring0Stalled` `HUB/osupdates/alarms_test.go:226`; DOC `AR/08-alarm-ladder.md:365`. Cost of the per-minute pass at 100 boxes: NOBODY KNOWS.
### 7. Off-site copy too old, and escrow never finished
1. CUSTOMER. `offsiteChecker.Check` `E/hub/cmd/hub/main.go:943` on the 60 s sweep; `isStale` `HUB/monitor/offsite.go:166` and `checkEscrowPending` `HUB/monitor/offsite_escrow_pending.go:66`. Owner: hub.
2. Every 60 s. Stale after `defaultOffsiteStaleAfter` 48 h `HUB/monitor/offsite.go:54`, counted from the last SUCCESS `HUB/monitor/offsite.go:203-207`. Escrow pending after `escrowPendingAfter` 7 d `HUB/monitor/offsite_escrow_pending.go:37`, re-raised each 7 d `HUB/monitor/offsite_escrow_pending.go:110`.
3. No time limit. Reads every customer's latest report each minute `HUB/monitor/offsite.go:443-459`; grows linearly with boxes.
4. Pod down: not judged. The edge states are in-memory maps (the ones cleared at `HUB/monitor/offsite.go:463-468`); whether a still-stale tier is mailed again after a restart: NOBODY KNOWS (restart seeding NOT FOUND, grep `R-1001|alreadyReported` in that file).
5. One mutex per pass `HUB/monitor/offsite.go:448-449`; a customer listed twice is handled once `HUB/monitor/offsite.go:453-458`. A never-ran tier is owned by the delivery checker, "one state, one owner" `HUB/monitor/offsite.go:160-162`.
6. `LastSuccess` is the BOX's timestamp compared with the hub's now `HUB/monitor/offsite.go:203-207`. A box clock ahead hides staleness (candidate); an unparseable time counts as stale `HUB/monitor/offsite.go:204-206`. Snapshot-drop tolerates `windowSlack` 2 h `HUB/monitor/offsite.go:296`.
7. Dispatcher mail on the stale edge `HUB/monitor/offsite.go:636`; `offsite_escrow_pending` is operator-only, see `AR/08-alarm-ladder.md:383`.
8. TEST `TestIsStale_PersistentlyFailingTierGoesStale` `HUB/monitor/offsite_r100_test.go:55`; TEST `TestIsStale_NewbornBoxDoesNotAlarm` `HUB/monitor/offsite_r100_test.go:74`. Box clock ahead: NOBODY KNOWS.
### 8. Restore-test proof too old
1. CUSTOMER (protects the promise that a backup restores; mail goes by event). `restoreTestChecker.Check` `E/hub/cmd/hub/main.go:944` on the 60 s sweep; `checkStaleness` `HUB/monitor/restoretest.go:242`.
2. Every 60 s. Window per tier: 4 archive generations `HUB/monitor/restoretest.go:62`, never under `restoreProvenWindowFloor` 7 d `HUB/monitor/restoretest.go:68`, never over `restoreProvenWindowCap` 12 d `HUB/monitor/restoretest.go:74`. Constants, no setting.
3. No time limit. Each pass reads 14 days of host reports per customer, `restoreWindowRead` `HUB/monitor/restoretest.go:79` at `HUB/monitor/restoretest.go:252`, every minute. This is the heaviest sweep read found; grows with boxes times report rate (candidate).
4. Pod restart: the first pass does not re-send a stale signal already in the events table `HUB/monitor/restoretest.go:281-283`, `staleAlreadyReported` `HUB/monitor/restoretest.go:300-312`.
5. Edge-triggered per customer and tier under a mutex `HUB/monitor/restoretest.go:270-278`. A deliberately-off box is skipped `HUB/monitor/restoretest.go:192-194`.
6. A new box is "unknown" until its window has passed since first contact `HUB/monitor/restoretest.go:326-342`; first contact is hub receive time `HUB/store/store.go:3994-4005`. Proof times come from box reports; box clock error: NOBODY KNOWS.
7. Event `restore_test_stale` warning through the dispatcher `HUB/monitor/restoretest.go:284-285`; a failed run is a separate event, `checkFailure` `HUB/monitor/restoretest.go:205`.
8. TEST `TestRestoreTestStale_R1001_RestartDoesNotReMail` `HUB/monitor/r1001_restart_test.go:147`. Cost at 100 boxes: NOBODY KNOWS.
### 9. Off-site clean-up window (20 minutes, weekly)
1. CUSTOMER DATA. The box asks; `OpenWindowFor` `HUB/offsitekeys/service.go:334` adds a deleting key line for 20 minutes; the hub's 60 s sweep `SweepExpiredWindows` `HUB/offsitekeys/service.go:413` closes a window left open. Sweep loop: `E/hub/cmd/hub/main.go:636-650`.
2. Granted when the last window is at least `windowCadence` 6 d 12 h old `HUB/offsitekeys/service.go:264`, test at `HUB/offsitekeys/service.go:337`; length `windowLength` 20 min `HUB/offsitekeys/service.go:263`. Sweep every 60 s. Constants; the weekly switch is a store setting `HUB/offsitekeys/service.go:338`.
3. Each sweep has a 2-minute context `E/hub/cmd/hub/main.go:644`. Work is per expired window, one remote call each `HUB/offsitekeys/service.go:419-422`; grows with boxes that crash mid-prune.
4. Pod down when a window expires: the deleting key line stays on the storage account until the hub is back; the sweep then closes it `HUB/offsitekeys/service.go:419-431`. Nobody is told during the gap (candidate). A failed close is retried next sweep `HUB/offsitekeys/service.go:424-426`.
5. One window per cadence via the ledger row `HUB/offsitekeys/service.go:364`. The daily key audit knows an open window `HUB/offsitekeys/service.go:212-213`. No lock against the abandon sweep; both run in the same goroutine `E/hub/cmd/hub/main.go:636-650`.
6. Cadence and expiry use the hub clock, `s.now()` `HUB/offsitekeys/service.go:337`. The box's clock does not decide. Hub clock jump back: a window could stay open longer: NOBODY KNOWS.
7. Operator event `offsite_window_failed` warning when the hub had to close it `HUB/offsitekeys/service.go:430-431`; `offsite_window_drop` error when too much went `HUB/offsitekeys/service.go:259`.
8. TEST `TestWindow_LeftOpenIsClosedByTheSweep` `HUB/offsitekeys/service_test.go:99`; TEST `TestWindow_GrantOpenCloseAndDropAlarm` `HUB/offsitekeys/service_test.go:37`; DOC `AR/07-backup-architecture.md:256`.
### 10. Delayed deletion of a set-aside off-site copy
1. CUSTOMER DATA. `SweepAbandons` `HUB/offsitekeys/service.go:529` deletes a household's set-aside copy once its request is due; called from the 60 s sweep `E/hub/cmd/hub/main.go:636-650`.
2. Due `DefaultAbandonDelay` 7 days after the request `HUB/offsitekeys/service.go:448`; override `OFFSITE_ABANDON_DELAY` `E/hub/cmd/hub/main.go:598-605`. Checked every 60 s. Hub clock.
3. 2-minute context shared with the window sweep `E/hub/cmd/hub/main.go:644`. One remote delete per due request `HUB/offsitekeys/service.go:536-538`; duration of a large delete: NOBODY KNOWS.
4. Pod down at the due time: deleted on the first sweep after start; the due list is in the store `HUB/offsitekeys/service.go:530`. A failed delete is marked and retried every sweep `HUB/offsitekeys/service.go:540-547`.
5. A request is deleted once, then marked `HUB/offsitekeys/service.go:549`. A cancel before the due time wins: design text `HUB/offsitekeys/service.go:438-441`.
6. Hub clock only. With an env override set too short by mistake the week of warning shrinks (candidate; value read at `E/hub/cmd/hub/main.go:598-605`).
7. Operator events: requested, cancelled, deleted, failed `HUB/offsitekeys/service.go:444-447`; failure mails once per request `HUB/offsitekeys/service.go:543-546`.
8. TEST `TestAbandon_DelayCancelAndScope` `HUB/offsitekeys/service_test.go:134`.
### 11. Off-site key audit (07:10)
1. OPERATOR (guards household copies against deletion). `scheduleDaily(ctx, "offsite-key-audit", "07:10", ...)` `E/hub/cmd/hub/main.go:651-655` runs `AuditAll` `HUB/offsitekeys/service.go:195`.
2. Daily 07:10 Budapest, literal slot `E/hub/cmd/hub/main.go:651`. Zone from `scheduleDaily` `E/hub/cmd/hub/main.go:1072`.
3. 10-minute limit for the whole fleet `E/hub/cmd/hub/main.go:652`. One remote read per provisioned customer, in sequence `HUB/offsitekeys/service.go:202-215`; at 100 boxes the 10 minutes may not cover all (candidate).
4. Pod down at 07:10: skipped that day, no catch-up `E/hub/cmd/hub/main.go:1095-1112`. Nobody notices.
5. Runs once per day by the `last` guard `E/hub/cmd/hub/main.go:1095-1112`. Meets an open clean-up window: the audit is told the window is open `HUB/offsitekeys/service.go:212-213`.
6. Wall-clock slot, DST-safe by `nextDailyRun` `E/hub/cmd/hub/main.go:1095-1112`.
7. Operator: `offsite_key_unlocked` error, or `offsite_key_audit_failed` when a file cannot be read `HUB/offsitekeys/service.go:192-194`.
8. TEST `TestWindow_PrependAuditClose` `HUB/offsitekeys/offsitekeys_test.go:158`; DOC `AR/05-hub-architecture.md:642`. The 10-minute limit at fleet size: NOBODY KNOWS.
### 12. Off-site credential stuck and its self-heal
1. CUSTOMER. Two jobs, merged: `offsiteDeliveryChecker.Check` `E/hub/cmd/hub/main.go:945` (60 s sweep) and the `offsiteheal` reconciler `HUB/offsiteheal/reconciler.go:165-177`, started at `E/hub/cmd/hub/main.go:846-854`.
2. Stuck after `stuckAfter` 1 h `HUB/monitor/offsite_delivery.go:38`; cooldowns 24 h `HUB/monitor/offsite_delivery.go:40-41`. Heal sweep every `tickIntervalDefault` 5 min `HUB/offsiteheal/reconciler.go:74`, acts after 2 distinct reports `HUB/offsiteheal/reconciler.go:70`.
3. Heal call limit 3 minutes `HUB/monitor/offsite_delivery.go:275`. Both list every customer config per pass `HUB/monitor/offsite_delivery.go:72` and `HUB/offsiteheal/reconciler.go:183`; linear in boxes.
4. Pod down: nothing; on return the next tick re-reads state. The cooldowns are "durable via the events table" `HUB/monitor/offsite_delivery.go:39`.
5. Two healers could meet: the checker re-reads the secret row immediately before acting `HUB/monitor/offsite_delivery.go:28-31`, and a box declaration is owned by offsiteheal `HUB/monitor/offsite_delivery.go:97-99`.
6. Age counted from the hub-held `status.Since` `HUB/monitor/offsite_delivery.go:93`. Debounce is counted in reports, not ticks `HUB/offsiteheal/reconciler.go:61-69`.
7. Operator events `offsite_delivery_stuck`, `offsite_credential_restaged` `HUB/monitor/offsite_delivery.go:34-35`; `offsite_selfheal_restaged` / `_reissued` `HUB/offsiteheal/reconciler.go:57-58`.
8. TEST `TestDeliveryChecker_StuckEvent_WarningOncePer24h` `HUB/monitor/offsite_delivery_test.go:121`; SOURCE `HUB/offsiteheal/reconciler.go:61-74`.
### 13. PBS whole-box credential self-heal
1. CUSTOMER. `pbsdrheal` reconciler `HUB/pbsdrheal/reconciler.go:153-165`, started at `E/hub/cmd/hub/main.go:829-837`; repairs a box whose off-site PBS credential did not land.
2. Every 5 minutes `HUB/pbsdrheal/reconciler.go:130`, or on a trigger `HUB/pbsdrheal/reconciler.go:160`; acts after 2 reports `HUB/pbsdrheal/reconciler.go:131`. A staged secret unconsumed for `unconsumedGrace` 15 min is flagged `HUB/pbsdrheal/reconciler.go:63`.
3. No time limit on a pass, `reconcileOnce(ctx)` `HUB/pbsdrheal/reconciler.go:163`. Reads all heal states `HUB/pbsdrheal/reconciler.go:170`; linear in boxes.
4. Pod down: no heal; the debounce maps are in memory `HUB/pbsdrheal/reconciler.go:134-135`, so after a restart a box waits two more reports.
5. One goroutine; a read error does nothing this tick `HUB/pbsdrheal/reconciler.go:171-174`.
6. Grace is hub time. Box clock: not used here.
7. Operator audit events `pbsdr_selfheal_*` and `pbsdr_unconsumed_secret` `HUB/pbsdrheal/reconciler.go:50-54`.
8. SOURCE `HUB/pbsdrheal/reconciler.go:57-63`. A pass that hangs on the remote: NOBODY KNOWS.
### 14. WireGuard peer list push to ep0
1. OPERATOR (keeps every box's tunnel allowed). `wgReconciler.Run` `E/hub/cmd/hub/main.go:752`, loop `HUB/wgsync/reconciler.go:90-106`; pushes the FULL peer list over SSH.
2. Every 5 minutes `HUB/wgsync/reconciler.go:41` and on every change via `Trigger` `HUB/wgsync/reconciler.go:48-53`. No setting.
3. 60 s limit per push `HUB/wgsync/reconciler.go:100`. The payload grows with peers `HUB/wgsync/reconciler.go:71-73`.
4. Pod down: no push; ep0 keeps the last list. A failed push is logged and retried next tick `HUB/wgsync/reconciler.go:101-103`. An event or mail on repeated failure: NOT FOUND (grep `SaveEvent|onEvent` in that file) (candidate).
5. One loop; a pending trigger covers later ones `HUB/wgsync/reconciler.go:46-47`. Full list each time erases drift `HUB/wgsync/reconciler.go:20-23`.
6. No clock in the payload.
7. Log line only `HUB/wgsync/reconciler.go:102`.
8. TEST `TestReconciler_FailureRetriedOnTick` `HUB/wgsync/reconciler_test.go:119`; TEST `TestReconciler_TriggerPushesFullList` `HUB/wgsync/reconciler_test.go:82`.
### 15. Box clock ran ahead: tunnel alarm
1. OPERATOR (the household's whole-box copies stop). `checkWGTunnel` `HUB/api/host_tunnel.go:61`, called on every host report `HUB/api/handler.go:1010`; not a timer, it runs when the box reports.
2. Fires when `handshake_future_s` > 0 `HUB/api/host_tunnel.go:41` or the last handshake is older than `wgTunnelStaleAfterS` 6 h `HUB/api/host_tunnel.go:23`. Constant.
3. One store read per report `HUB/api/host_tunnel.go:66`.
4. If the box stops reporting, this check never runs; block 1 covers that. An older agent that sends neither field is not judged `HUB/api/host_tunnel.go:62-64`.
5. Edge-triggered from the newest stored tunnel event `HUB/api/host_tunnel.go:71-83`, so a restart does not re-mail.
6. This IS the clock-ahead handling: the future stamp is measured by the box's own clock `HUB/api/host_tunnel.go:31`; repair is a new key after 15 minutes on the agent side `HUB/api/host_tunnel.go:14-17`. Known gaps are listed in `AR/05-hub-architecture.md:489-490`.
7. Operator only, `wg_tunnel_stale` warning; `wg_tunnel_ok` is stored, not mailed `HUB/api/host_tunnel.go:12-14`.
8. TEST `TestD30_TheAlarmClearsWhenTheNewKeyShakesHands` `HUB/api/host_tunnel_test.go:83`; DOC `AR/05-hub-architecture.md:482-490`.
### 16. The daily scheduler and the clock change
1. OPERATOR (mechanism under blocks 2, 11, 17, 18). `scheduleDaily` `E/hub/cmd/hub/main.go:1069-1093` with `nextDailyRun` `E/hub/cmd/hub/main.go:1095-1112`.
2. Budapest wall-clock, `LoadLocation` `E/hub/cmd/hub/main.go:1072`; fallback fixed CET `E/hub/cmd/hub/main.go:1074` (candidate: in summer the fallback is one hour off). Pod `TZ` is `Europe/Budapest` `E/manifests/hub.yaml:140-141`.
3. Sleeps with `time.After(delay)` `E/hub/cmd/hub/main.go:1088`; the job runs inline `E/hub/cmd/hub/main.go:1090`, so a long job delays only its own next slot.
4. No catch-up for a slot missed while the pod was down; exceptions are the snapshot (block 17) and prune (block 18). A bad time string becomes 00:00 silently, `parseHM` `E/hub/cmd/hub/main.go:1115-1121` (candidate).
5. Next run is the calendar day after and the slot that ran last never runs again `AR/05-hub-architecture.md:642-645`.
6. 25-hour and 23-hour days handled, D24 `AR/05-hub-architecture.md:640-646`. A hub clock jump while sleeping: the timer keeps its old duration: NOBODY KNOWS.
7. Nobody; start-up logs only.
8. TEST `TestR1183_DailyJobRunsOnceOnTheDayTheClocksGoBack` `E/hub/cmd/hub/r1183_dst_schedule_test.go:31`; TEST `TestR1183_DailyJobRunsOnceOnTheDayTheClocksGoForward` `E/hub/cmd/hub/r1183_dst_schedule_test.go:56`.
### 17. Hub database snapshot (four slots)
1. OPERATOR (the hub holds every household's escrow and settings). `runSnapshot` scheduled over `dbsnap.Slots` `E/hub/cmd/hub/main.go:967-983`; `Make` `HUB/dbsnap/dbsnap.go:72-117`.
2. 02:00, 08:00, 14:00, 20:00 Budapest: `Slots` `HUB/dbsnap/dbsnap.go:32`. Keeps `Keep = 2` `HUB/dbsnap/dbsnap.go:27`. File name carries UTC `HUB/dbsnap/dbsnap.go:89`.
3. 10-minute limit `E/hub/cmd/hub/main.go:972`. Time grows with database size; written to a tmp file then renamed `HUB/dbsnap/dbsnap.go:91-104`; duration is logged `HUB/dbsnap/dbsnap.go:113-114`.
4. The one job WITH catch-up: at start, if the newest snapshot is older than `CatchUpAge` 6 h, one is made `E/hub/cmd/hub/main.go:978-980`, `NeedsCatchUp` `HUB/dbsnap/dbsnap.go:148-159`.
5. A running flag refuses a second run with `ErrBusy` `HUB/dbsnap/dbsnap.go:73-80`. The DooPlex push timer runs 30 min later and waits for the tmp file (block 26).
6. DST-safe slots (block 16). The doc still says "older than 24 h" and "02:30" in its original text `AR/05-hub-architecture.md:618-620` (candidate: stale text).
7. Failure is a log line; the outside alarm is the push timer's (block 26).
8. TEST `TestD34_SlotsAreEverySixHours` `HUB/dbsnap/dbsnap_test.go:214`; TEST `TestR173_MainSchedulesTheDBSnapshot` `E/hub/cmd/hub/r173_wiring_test.go:14`.
### 18. Daily prune of old hub rows (04:30)
1. OPERATOR. `pruneAll` `E/hub/cmd/hub/main.go:1123-1150`, at start and by `scheduleDaily(ctx, "prune", ...)` `E/hub/cmd/hub/main.go:860-865`.
2. Daily at `prune_schedule`, default `"04:30"` `E/hub/cmd/hub/main.go:1026`, deployed `E/manifests/hub.yaml:83`; keeps `max_days` 90 `E/manifests/hub.yaml:82`. Also app telemetry 90 d `E/hub/cmd/hub/main.go:1134`, stale issues 30 d `E/hub/cmd/hub/main.go:1139`, deleted-customer audit 1 year `E/hub/cmd/hub/main.go:1145`.
3. No time limit. Deletes grow with reports per box per day; duration NOT FOUND (grep `time.Since` in `E/hub/cmd/hub/main.go:1123-1150`).
4. Runs at every start, so a missed slot is caught up by the next restart; with `MaxDays` 0 it never runs `E/hub/cmd/hub/main.go:860-865`.
5. May meet the 05:00 deadline check or a snapshot only if it runs over 30 min; no lock found.
6. `Prune` builds its cutoff from local time without `.UTC()` `HUB/store/store.go:1607` while the column is UTC `HUB/store/store.go:129`; `PruneEvents` uses UTC `HUB/store/store.go:2873` (candidate: 1-2 h skew on a 90-day cutoff, harmless in size).
7. Log only.
8. SOURCE `E/hub/cmd/hub/main.go:1123-1150`. A test for the prune: NOT FOUND (grep `func Test.*Prune` in hub/internal/store).
### 19. The other checks on the 60-second sweep
1. CUSTOMER and OPERATOR mixed. Merged: host capability, host leaf, host disk, storage fill, management plane, OOB, controller supervisor `E/hub/cmd/hub/main.go:937-946`. All run in the one sweep goroutine `E/hub/cmd/hub/main.go:927-965`.
2. Every 60 s in the fixed order of `E/hub/cmd/hub/main.go:935-962`. A backup-cadence refusal repeats every `refusalRepeatEvery` 24 h `HUB/monitor/host_capability.go:72`.
3. No per-checker limit: a slow checker delays all the others, including liveness (block 1), because they run in sequence `E/hub/cmd/hub/main.go:935-962` (candidate). Each reads the whole fleet per pass.
4. Pod down: nothing. After start, band states are rebuilt from the latest reports.
5. One goroutine, so no overlap; a Go ticker drops ticks when a pass overruns.
6. They judge the latest report's content, not time, except the 24 h repeat `HUB/monitor/host_capability.go:172`.
7. Dispatcher mails; storage fill has its own 6 h grain `AR/08-alarm-ladder.md:305`.
8. SOURCE `E/hub/cmd/hub/main.go:927-965`. Duration of one full sweep: NOBODY KNOWS.
### 20. Reminder about a held app
1. OPERATOR (D53: no household mail). `heldApps.Check` `E/hub/cmd/hub/main.go:947`; `HeldAppsChecker.Check` `HUB/monitor/held_apps.go:102`.
2. One pass per `heldAppCheckEvery` 15 min `HUB/monitor/held_apps.go:36`. First mail after a grace of 6 h / 72 h / 24 h by reason `HUB/monitor/held_apps.go:40-48`; then every `heldAppRepeat` 7 d `HUB/monitor/held_apps.go:34`.
3. One list of all held apps `HUB/monitor/held_apps.go:114`.
4. Pod down: the reminder comes on the first pass after start; the reminded-at time is stored BEFORE sending `HUB/monitor/held_apps.go:141-145`, so a restart cannot mail twice.
5. A row the box has not confirmed for `heldAppStale` 50 h is not reminded `HUB/monitor/held_apps.go:125-129`.
6. The age counts from the hold's own time, which is the BOX's clock, else the hub's first sight `HUB/monitor/held_apps.go:130-134` and `HUB/monitor/held_apps.go:22`. Box clock behind: reminder early; ahead: late (candidate).
7. Operator, `app_held_reminder` warning `HUB/monitor/held_apps.go:157-163`.
8. TEST `TestD53_OneReminderAfterTheGraceThenWeekly` `HUB/monitor/held_apps_test.go:34`; TEST `TestD53_AStaleRowIsNotReminded` `HUB/monitor/held_apps_test.go:99`; DOC `AR/08-alarm-ladder.md:655`.
### 21. "A golden is published but not vouched"
1. OPERATOR. `goldenVouch` check on the sweep `E/hub/cmd/hub/main.go:948-950`; `GoldenVouchChecker.Check` `HUB/monitor/golden_vouch.go:54`.
2. One Gitea listing per `goldenVouchCheckEvery` 15 min; alarm after `goldenVouchGrace` 2 h `HUB/monitor/golden_vouch.go:42-43`.
3. Listing limit `goldenVouchListDeadline` 20 s `HUB/monitor/golden_vouch.go:44`; it runs inside the sweep, so a slow Gitea holds the sweep for up to 20 s `HUB/monitor/golden_vouch.go:62-64`.
4. Pod restart: the gap clock and the alerted mark are in memory `HUB/monitor/golden_vouch.go:34-37`, so the 2 h starts again and the alarm is mailed again ("one alarm per unvouched version per process" `HUB/monitor/golden_vouch.go:23`).
5. Gitea unreachable or empty list: no verdict, never an alarm `HUB/monitor/golden_vouch.go:65-67`.
6. Hub clock only.
7. Operator event `golden_not_vouched` `HUB/monitor/golden_vouch.go:95-96`.
8. TEST `TestGoldenVouchChecker` `HUB/monitor/golden_vouch_test.go:15`. Re-mail after a restart: NOBODY KNOWS.
### 22. Shared storage boxes: fill and reachability
1. OPERATOR (every household's off-site space). `offsiteBox` and `pbsdrBox` checks `E/hub/cmd/hub/main.go:951-956`.
2. On the 60 s sweep, but one provider read per `boxFetchInterval` 15 min `HUB/monitor/offsite_box.go:72`, test at `HUB/monitor/offsite_box.go:132` and `HUB/monitor/pbsdr_box.go:130`. Warn 80 %, critical 90 % `HUB/monitor/offsite_box.go:67-68`.
3. Provider client limit 30 s `HUB/hetznerapi/hetznerapi.go:143-145`. Does not grow with boxes.
4. Pod down: no reading. Unreachable is declared after consecutive failed 15-minute windows `HUB/monitor/pbsdr_box.go:80-82`.
5. Throttled by `lastFetch` in memory; first pass after a restart fetches at once `HUB/monitor/offsite_box.go:132`.
6. Hub clock only.
7. Operator events for fill, oversubscription, unreachable and recovered `HUB/monitor/offsite_box.go:284-339`.
8. SOURCE `HUB/monitor/offsite_box.go:70-72`.
### 23. Purges: log bundles, access log, sessions
1. OPERATOR. Merged: `PurgeExpiredLogBundles` on the sweep `E/hub/cmd/hub/main.go:958-962`; access-log `prune` `HUB/accesslog/accesslog.go:192`; `CleanupSessions` `HUB/web/server.go:199-217`.
2. Bundles: every 60 s, TTL `logBundleTTL` 72 h `HUB/store/logbundle.go:29`. Access log: on the first line of each UTC day `HUB/accesslog/accesslog.go:166-184`, keeps `DefaultKeep` 90 d `HUB/accesslog/accesslog.go:44`. Sessions: every 15 min `HUB/web/server.go:200`, life 7 d `HUB/web/server.go:949`.
3. Small. Access-log prune lists one directory `HUB/accesslog/accesslog.go:196`.
4. Pod restart: sessions are in memory `HUB/web/server.go:207-213`, so every operator is logged out. Access-log prune runs when the day file is first opened `HUB/accesslog/accesslog.go:182-184`, so it also runs after a restart.
5. Access-log prune only removes exact `access-YYYY-MM-DD.log` names `HUB/accesslog/accesslog.go:191`.
6. Access log days are UTC `HUB/accesslog/accesslog.go:166`, unlike the Budapest daily jobs; documented `AR/05-hub-architecture.md:634-636`.
7. Nobody.
8. TEST `TestAccessLog_Keeps90DaysThenDeletes` `HUB/accesslog/accesslog_test.go:135`.
### 24. Links and tokens that expire by themselves
1. CUSTOMER (bind link) and OPERATOR (tokens). No sweep: each is checked when used. Bind link `selfBindTTL` 7 d `HUB/web/selfbind_mint.go:20`; access token expiry `HUB/web/tokens.go:95-96`; DR recovery `defaultRecoveryTTL` 30 min, max 4 h `HUB/api/dr.go:21-22`.
2. Bind link: 7 days from mint `HUB/web/selfbind_mint.go:77`; a fresh link is mailed at most once per `bindResendEvery` 1 h `HUB/web/selfbind.go:382`. Token life is chosen in days at creation `HUB/web/tokens.go:180`. Operator actions: `OperatorActionTTL` 24 h `HUB/store/opactions.go:62`.
3. Nothing runs; expired rows stay in the tables. A bind token row is deleted only when a new one is minted for that customer `HUB/store/selfbind.go:86`; a purge by age: NOT FOUND (grep `DELETE FROM` in the store's selfbind and api_tokens files).
4. Not applicable: no job. An expired link reads as expired and falls back to operator bind `HUB/web/selfbind_mint.go:18-19`.
5. Bind expiry is tested twice, in Go `HUB/store/selfbind.go:39` and in SQL `HUB/store/selfbind.go:177`.
6. Hub clock. SQL side uses `datetime('now')` UTC `HUB/store/selfbind.go:177`.
7. Nobody is warned BEFORE a token or link expires; an expired token logs a WARN on use `HUB/web/tokens.go:96` (candidate: a token used by automation expires silently until the next call).
8. SOURCE `HUB/store/selfbind.go:38-46`. Warning before expiry: NOT FOUND (grep `expires in|about to expire|expiring` in web/tokens.go and monitor).
### 25. Version check and catalog template fetch
1. OPERATOR. `VersionChecker.Run` `HUB/web/version.go:43-55` and `TemplateFetcher.Run` `HUB/web/templatefetcher.go:41-53`, started at `E/hub/cmd/hub/main.go:711-722` and `E/hub/cmd/hub/main.go:327-335`.
2. Once at start, then every `check_interval` 6 h `E/manifests/hub.yaml:104` and `template_interval` 1 h `E/manifests/hub.yaml:105`.
3. One HTTP call each.
4. Pod down: nothing; both fetch at once on start `HUB/web/version.go:44` and `HUB/web/templatefetcher.go:42`.
5. One goroutine each.
6. Interval tickers; no wall clock.
7. A failure is a WARN line and a stored `lastError` `HUB/web/version.go:62-65`; nobody is mailed.
8. SOURCE `HUB/web/version.go:41-55`.
### 26. DooPlex: push of the hub database snapshot off-site
1. NOT MEASURED. OPERATOR. systemd timer `felhom-hub-db-backup.timer` on DooPlex, `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10`, runs the script `E/scripts/hub-db-backup/felhom-hub-db-backup:19`; installed by hand per `RB/RUNBOOK-hub-db-offsite-backup.md:107-108`.
2. 02:30, 08:30, 14:30, 20:30 `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10`, `RandomizedDelaySec=2min` `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:12`. Host local time (no zone in the unit). DooPlex's zone: NOT FOUND (grep `timedatectl|Europe/Budapest` in documentation/operations and the three runbooks).
3. `TimeoutStartSec=45min` `E/scripts/hub-db-backup/felhom-hub-db-backup.service:13`. Waits up to `WAIT_S` 1200 s for the hub's tmp file `E/scripts/hub-db-backup/felhom-hub-db-backup:20`, loop `E/scripts/hub-db-backup/felhom-hub-db-backup:35-40`. Grows with database size.
4. `Persistent=true` `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:11`: a slot missed while DooPlex was off runs at boot. It refuses a snapshot older than `MAX_AGE_H` 7 `E/scripts/hub-db-backup/felhom-hub-db-backup:19`, line `E/scripts/hub-db-backup/felhom-hub-db-backup:50`.
5. No flock in the script (grep `flock|lock` returned nothing); systemd does not start a oneshot unit twice. A catch-up run right after boot can meet a hub that has not made its catch-up snapshot yet: history of a false failure in `E/scripts/hub-db-backup/felhom-hub-db-backup:15-18`.
6. Hub slots are Budapest (pod TZ), timer slots are host local: the 30-minute offset holds only if both zones match; pinned in text by `E/scripts/hub-db-backup/test_hub_db_backup.py:309-316`. DST night: NOBODY KNOWS.
7. Failure: `OnFailure=felhom-backup-failmail@%n.service` `E/scripts/hub-db-backup/felhom-hub-db-backup.service:6`, mail `E/scripts/dooplex-offsite/felhom-backup-failmail:2`. Silence: `HubDBBackupStale` at 13 h per `RB/RUNBOOK-hub-db-offsite-backup.md:100`, but the YAML there shows 26 h `RB/RUNBOOK-hub-db-offsite-backup.md:153` (candidate).
8. TEST `E/scripts/hub-db-backup/test_hub_db_backup.py:309-316` (run by hand, `RB/RUNBOOK-hub-db-offsite-backup.md:107-108`); the live alert rule is in homelab-manifests, not in this clone: NOBODY KNOWS.
### 27. DooPlex: Sunday restore tests
1. NOT MEASURED. OPERATOR. Two timers, merged: `felhom-hub-db-restore-test.timer` `E/scripts/hub-db-backup/felhom-hub-db-restore-test.timer:6` and `felhom-dooplex-offsite-restore-test.timer` `E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test.timer:6`.
2. Sunday 04:30 and Sunday 05:30, host local time; `Persistent=true` `E/scripts/hub-db-backup/felhom-hub-db-restore-test.timer:7`.
3. Limits 45 min `E/scripts/hub-db-backup/felhom-hub-db-restore-test.service:13` and 90 min `E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test.service:13`. Grow with database and dump size.
4. Missed Sunday runs at next boot (Persistent). Each refuses a copy older than `MAX_AGE_H` 50 `E/scripts/hub-db-backup/felhom-hub-db-restore-test:13` and `E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test:14`.
5. Sunday 04:30 lies between the hub's 02:00 snapshot push and the 05:00 PBS pull; a lock between them: NOT FOUND (grep `flock` in scripts/hub-db-backup and scripts/dooplex-offsite).
6. Last Sunday of October has 02:00-03:00 twice; systemd's handling for 04:30 is outside the repo: NOBODY KNOWS.
7. Staleness alarm `HubDBRestoreTestStale` after 8 days `RB/RUNBOOK-hub-db-offsite-backup.md:158`.
8. DOC `RB/RUNBOOK-hub-db-offsite-backup.md:152-161`. Whether the rule is deployed: NOBODY KNOWS (homelab-manifests is not in the clone tree).
### 28. DooPlex: nightly off-site push and release copy
1. NOT MEASURED. OPERATOR. Merged: `felhom-dooplex-offsite.timer` `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:6` and `felhom-release-copy.timer` `E/scripts/release-copy/felhom-release-copy.timer:7`.
2. 00:20 and 01:40 host local time; the order is stated in a comment `E/scripts/release-copy/felhom-release-copy.timer:6`. Random delay 2 min and 5 min `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:8`, `E/scripts/release-copy/felhom-release-copy.timer:9`.
3. Limits 90 min `E/scripts/dooplex-offsite/felhom-dooplex-offsite.service:13` and 2 h `E/scripts/release-copy/felhom-release-copy.service:12`. A 90-minute run from 00:20 ends 01:50, after the 01:40 start (candidate: can overlap; no lock found).
4. Both `Persistent=true` `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:7`, `E/scripts/release-copy/felhom-release-copy.timer:8`. The push refuses a dump older than `DUMP_MAX_AGE_H` 7 `E/scripts/dooplex-offsite/felhom-dooplex-offsite:23`, message `E/scripts/dooplex-offsite/felhom-dooplex-offsite:55`.
5. The push depends on a dump made by another job, not in this clone; it only checks the dump's age `E/scripts/dooplex-offsite/felhom-dooplex-offsite:55`. One retry `E/scripts/dooplex-offsite/felhom-dooplex-offsite:66`.
6. 00:20 and 01:40 are outside the 02:00-03:00 change hour.
7. `OnFailure` mail on both `E/scripts/dooplex-offsite/felhom-dooplex-offsite.service:6`, `E/scripts/release-copy/felhom-release-copy.service:6`; a metric file is written `E/scripts/dooplex-offsite/felhom-dooplex-offsite:143-153`.
8. SOURCE only. The release-copy timer is enabled by hand `E/scripts/release-copy/install.sh:4`; whether it is enabled: NOBODY KNOWS.
### 29. DooPlex and ep0: PBS copy, prune, verify, clean-up
1. NOT MEASURED. OPERATOR (holds the second copy of every household's whole-box backup). PBS jobs described in `RB/ep0-datastore-copy.md:15-18`, plus `felhom-ep0-copy-gc.timer` `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:5`.
2. ep0 prune 03:30, pull to DooPlex 05:00 `RB/ep0-datastore-copy.md:15`; DooPlex prune 07:30 `RB/ep0-datastore-copy.md:17`; copy clean-up 08:00 `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:5`; verify Sat 06:30, GC Sun 08:30 `RB/ep0-datastore-copy.md:16-18`.
3. The PBS jobs live in PBS's own config, not in the repo; their duration and limits: NOBODY KNOWS. Grow with every box's backup size.
4. The gc timer is `Persistent=true` `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:6`. PBS jobs: catch-up behaviour not in repo.
5. Order is by clock only: prune 07:30 then clean-up 08:00 `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:2`. A pull that runs past 07:30 meets the prune: NOBODY KNOWS.
6. Two machines, two clocks (ep0 and DooPlex): "ep0 prune 03:30 before the 05:00 pull" holds only while both agree. Their zones: NOT FOUND (grep `timedatectl|Europe/Budapest` in that runbook).
7. PBS notification target per `RB/ep0-datastore-copy.md:19`. The gc unit has no `OnFailure` and no time limit (`OnFailure|TimeoutStartSec`: NOT FOUND by grep in `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.service:1-9`) (candidate).
8. DOC `RB/ep0-datastore-copy.md:15-19`.
### 30. Nightly CI: gates and the security watch
1. NOT MEASURED. OPERATOR. Gitea CI crons in felhom.eu: `gates` `E/.gitea/workflows/gates.yml:26-27` and `security-watch` `E/.gitea/workflows/security-watch.yml:21-22`. The other three repos have no scheduled job (grep `schedule:|cron:` in their `.gitea/workflows/` returned nothing).
2. 03:20 UTC and 03:40 UTC every night; the reason for the order is in `E/.gitea/workflows/gates.yml:21-23`. CI cron is UTC per `AR/08-alarm-ladder.md:710`, so it moves one hour against the Budapest night jobs at each clock change.
3. Limits: 5 min `E/.gitea/workflows/gates.yml:53`, 20 min `E/.gitea/workflows/security-watch.yml:32`. The watch grows with the number of pinned programs.
4. Missed night: no catch-up found. The watch's weekly summary self-heals: it is sent when the last one is 8+ days old `AR/08-alarm-ladder.md:717`. A runner that drops the job silently: NOBODY KNOWS.
5. Both run on `felhom-gates` runners `E/.gitea/workflows/security-watch.yml:31`; the watch commits its memory to main with 3 attempts `E/.gitea/workflows/security-watch.yml:89-101`. A push race with a human commit is retried, then fails.
6. UTC cron, no DST. Reminder ages are computed by the script from its memory file `AR/08-alarm-ladder.md:716-720`.
7. Failure mail to admin: "[felhom CI] security watch FAILED" `E/.gitea/workflows/security-watch.yml:104-119`; gates has its own alarm step `E/.gitea/workflows/gates.yml:245`. Whether the alarm step runs for a reaped job is recorded as unknown `E/.gitea/workflows/gates.yml:50-52`.
8. DOC `AR/08-alarm-ladder.md:708-722`. A night with no run at all: NOBODY KNOWS.
## Tiny
- Deliberate power-off hold: alarms held 14 days, `DeliberateOffHold` `HUB/store/deliberateoff.go:18`.
- A future-dated power-off request from a box is replaced by hub time `HUB/api/deliberate_off.go:30`.
- Long-poll wait: `waitMaxHold` 240 s, heartbeat 25 s `HUB/api/wait.go:17`.
- Presence: `PresenceWaitCadence` 243 s, grace 90 s `HUB/intent/presence.go:21`.
- Hetzner provisioning poll: every 3 s, at most 3 min `HUB/hetznerapi/hetznerapi.go:143-145`.
- Off-site scan back-off 2,4,8,16,30,30,30 s `HUB/offsite/offsite.go:112`.
- PBS self-check pause 7 s `HUB/web/pbsdr.go:571`.
- Host delete allowed after 360 s off `HUB/web/hosts.go:770`.
- Host operator reboot action TTL 30 min `HUB/store/opactions.go:47`.
- System page: line test stale after 15 days `HUB/web/system.go:174`.
- HTTP server timeouts `E/hub/cmd/hub/main.go:705-707`; shutdown grace 15 s `E/hub/cmd/hub/main.go:1002`.
- MAIL-HOLD has no expiry timer; it is released by hand `HUB/mailhold/mailhold.go:74`.
- No timed digest in the hub: the per-run digest comes from the box `HUB/notify/dispatcher.go:243`.
- NOT MEASURED: weekly golden bake is a manual Sunday-night step, not a timer `RB/nightly-rotation.md:15`.
- NOT MEASURED: `systemd-run --unit=golden-bake` is a manual start `RB/RUNBOOK-manual-build.md:180`.
- NOT MEASURED: ep0's hub-DB prune 03:45 `RB/RUNBOOK-hub-db-offsite-backup.md:101`.
- NOT MEASURED: DooPlex's own `dooplex-backup.timer` 03:19 and Longhorn 04:00 / Sunday 05:00 are named only in `RB/RUNBOOK-hub-db-offsite-backup.md:12-14`.
- NOT MEASURED: box-side `felhom-mgmt-watchdog.timer` is enabled by the installer `E/scripts/felhom-host-install.sh:3076` (belongs to the agent/installer helper).
## Not found
- /mnt/5_hdd/felhom.eu/git/homelab-manifests does not exist in the clone tree (tried: ls of /mnt/5_hdd/felhom.eu/git/, find -maxdepth 2 -name 'homelab*'). So no k8s CronJob, Prometheus rule or Renovate schedule could be read from it.
- k8s CronJob in felhom.eu/manifests/: NOT FOUND (grep `kind: CronJob|schedule:` matched only `E/manifests/hub.yaml:83`, the hub's own prune time).
- Cloudflare periodic sync in the hub: NOT FOUND (grep `NewTicker|time.After|time.Sleep` in hub/internal/cloudflare showed only request timeouts).
- A hub-side security watch: NOT FOUND in hub code; it is the CI job of block 30.
- A timed mail digest in the hub: NOT FOUND (grep -il `digest` in hub/internal: only the box's per-run digest).
- A sweep that deletes expired bind links, tokens or claim codes: NOT FOUND (grep `expires|Expir|TTL` in hub/internal/claim, web/tokens.go, store/selfbind.go shows read-time checks only).
- DooPlex / ep0 host time zone for `OnCalendar`: NOT FOUND (grep `timedatectl|Europe/Budapest` in documentation/operations/*.md, RUNBOOK-hub-db-offsite-backup.md, ep0-datastore-copy.md, total-loss-of-dooplex.md).
- cron/crontab/`at ` installs by repo scripts on DooPlex or ep0: NOT FOUND (grep `crontab|/etc/cron|OnCalendar|systemd-run` in felhom.eu/scripts found only the systemd units listed in blocks 26-29 and the box installer).
- Scheduled CI in felhom-controller, felhom-agent, app-catalog-felhom.eu: NOT FOUND (grep `schedule:|cron:` in each `.gitea/workflows/`).
- Measured duration of any hub job: NOT FOUND in repository files; only the snapshot and the OS alarm pass log their own duration.
@@ -0,0 +1,29 @@
searches run 2026-11-11 10:33:51 UTC
C+ | positive control (a row that exists) | /R-1301/ | ar files=0 | open: R-1301 | closed: | roadmap=0
C+2 | positive control (a word in all four) | /backup/ | ar files=31 | open: R-1013 R-1025 R-1035 R-1062 R-1064 R-1078 R-1083 R-1087 R-1171 R-1172 R-1173 R-1184 R-1185 R-1186 R-1188 R-1189 R-1190 R-1198 R-1200 R-1206 R-1211 R-1212 R-1214 R-1221 R-1222 R-1229 R-1231 R-1235 R-1241 R-1242 R-1243 R-1254 R-1259 R-1260 R-1263 R-1268 R-1274 R-1293 R-1294 R-1296 R-1301 R-1312 R-1328 R-1362 R-1366 R-1368 R-1370 R-1371 R-1372 R-1373 R-1374 R-164 R-213 R-232 R-264 R-314 R-32 R-336 R-340 R-388 R-401 R-409 R-433 R-516 R-540 R-541 R-548 R-683 R-691 R-724 R-759 R-775 R-78 R-816 R-832 R-862 R-878 R-895 R-905 R-91 R-923 R-924 R-933 R-984 R-989 | closed: R-1006 R-1008 R-1012 R-1019 R-102 R-103 R-1048 R-1057 R-1059 R-1063 R-1072 R-1075 R-1079 R-1084 R-1088 R-1089 R-109 R-1097 R-110 R-1105 R-1110 R-1117 R-1119 R-1134 R-1138 R-1144 R-1149 R-1150 R-1157 R-116 R-1169 R-1175 R-1177 R-1179 R-1181 R-1195 R-1219 R-1226 R-1234 R-1252 R-1269 R-1278 R-1283 R-1287 R-1288 R-1299 R-133 R-158 R-161 R-165 R-167 R-173 R-175 R-183 R-185 R-191 R-202 R-215 R-218 R-225 R-231 R-237 R-240 R-245 R-258 R-263 R-275 R-279 R-281 R-289 R-296 R-298 R-337 R-348 R-351 R-361 R-363 R-366 R-371 R-375 R-389 R-398 R-400 R-412 R-430 R-431 R-436 R-442 R-448 R-474 R-475 R-481 R-485 R-486 R-487 R-488 R-49 R-499 R-500 R-517 R-519 R-526 R-534 R-537 R-538 R-546 R-550 R-553 R-556 R-557 R-562 R-563 R-578 R-597 R-598 R-626 R-634 R-637 R-640 R-641 R-643 R-645 R-648 R-650 R-660 R-661 R-666 R-670 R-671 R-673 R-679 R-682 R-684 R-685 R-689 R-690 R-694 R-696 R-698 R-705 R-706 R-721 R-752 R-762 R-765 R-773 R-815 R-837 R-842 R-86 R-871 R-872 R-88a R-88b R-893 R-897 R-899 R-906 R-910 R-921 R-926 R-97 R-973 R-985 R-988 | roadmap=8
C- | negative control | /zzqxclocklistzz/ | ar files=0 | open: | closed: | roadmap=0
S01 | a daily hub job missed while the hub is down | /(missed|skipped).{0,60}(slot|check|audit).{0,80}(pod|restart|hub (is|was) down)|catch.?up.{0,80}(hub job|scheduleDaily|daily job)/ | ar files=1 | open: | closed: | roadmap=0
S02 | a failed household mail is not sent again | /(customer|household).{0,30}(mail|e-mail).{0,80}(retry|retried|not sent again)/ | ar files=1 | open: | closed: | roadmap=0
S03 | the operator mail retry is lost at a hub restart | /retry.{0,100}(restart|in memory|lost)|GAVE UP/ | ar files=1 | open: R-1215 | closed: R-1163 | roadmap=0
S04 | the guest-power watchdog gives up and tells nobody | /guest.power.{0,160}(give|gave|giving).up|GIVING UP/ | ar files=1 | open: | closed: | roadmap=0
S05 | the peer list push to ep0 fails again and again | /wgsync.{0,120}(fail|alarm|event)|peer list.{0,80}(fail|stale|alarm)/ | ar files=0 | open: | closed: | roadmap=0
S06 | the app-update leg did not run: who is told | /update.leg.{0,120}(did not run|missed|skipped|lost).{0,100}(told|mail|alarm|event|nobody)/ | ar files=0 | open: | closed: | roadmap=0
S07 | a verdict trusts the box's clock (fresh backup, off-site age, proof age) | /(box|host).{0,12}clock.{0,120}(ahead|future|wrong).{0,160}(fresh|stale|latest|backup|proof)|timestamp.{0,60}(in the future|ahead)/ | ar files=0 | open: R-1189 R-1206 | closed: | roadmap=0
S08 | an 'already ran' stamp written by a clock that was ahead | /(stamp|already ran|last run|LastOKAt).{0,100}(clock|future|ahead)/ | ar files=3 | open: | closed: | roadmap=0
S09 | a wrong clock refuses every signed job | /(signed|job).{0,80}(not.yet.valid|expired).{0,120}clock|clock.{0,120}(signed job|issued_at|ErrNotYetValid)/ | ar files=0 | open: | closed: | roadmap=0
S10 | the hub's own clock jumps | /hub.{0,20}clock.{0,80}(jump|step|wrong|drift)/ | ar files=0 | open: | closed: | roadmap=0
S11 | the off-site push is skipped because another backup runs | /off-?site.{0,100}skipped.{0,100}(another|running)|another backup is running/ | ar files=0 | open: R-1186 | closed: | roadmap=0
S12 | the 05:30 proof restore can fall inside the whole-guest backup | /proof.{0,120}(05:30|gate|whole-guest|vzdump|meets)/ | ar files=4 | open: R-132 | closed: R-1142 R-418 R-425 R-454 R-463 R-466 R-470 R-477 R-478 R-559 R-564 R-637 R-643 R-709 R-885 | roadmap=0
S13 | the PBS verify runs beside a backup | /verify.{0,80}(heavy-op|gate|beside|during a backup)/ | ar files=2 | open: | closed: | roadmap=0
S14 | two DooPlex timers overlap, or two machines' clocks order the jobs | /(timer|slot).{0,80}(overlap|meet|same time)|holds only (if|while) both/ | ar files=0 | open: | closed: | roadmap=0
S15 | an interval job with no time limit | /(time limit|timeout|deadline).{0,80}(sweep|interval job|health check|scan)|slow checker|no per-check/ | ar files=1 | open: | closed: R-1344 | roadmap=0
S16 | how long the hub's sweep takes with many boxes | /(100|hundred|1000) (boxes|customers|households)|sweep.{0,80}(duration|takes|seconds).{0,60}(boxes|fleet)/ | ar files=0 | open: | closed: | roadmap=0
S17 | the agent unit: Restart=always and a watchdog | /Restart=always|WatchdogSec|systemd watchdog/ | ar files=2 | open: | closed: | roadmap=0
S18 | hub database staleness alarm: 13 h or 26 h | /HubDBBackupStale/ | ar files=0 | open: | closed: | roadmap=0
S19 | a list of every timer | /every timer|list of (all )?(timers|scheduled jobs)|all (timers|scheduled jobs)|job (table|inventory)|timer inventory/ | ar files=0 | open: | closed: | roadmap=0
S20 | the hour change | /DST|clock change|daylight|spring forward|2027-03-28/ | ar files=12 | open: R-1189 R-1259 R-1260 R-1293 R-914 | closed: R-1169 R-246 R-356 R-386 R-521 R-660 R-921 | roadmap=0
S21 | who keeps the time on a box | /chrony|timesyncd|NTP/ | ar files=12 | open: R-1301 | closed: R-220 R-444 | roadmap=0
S22 | the crash guard reads the clock at early boot | /crash.guard.{0,160}(clock|RTC|time is wrong)/ | ar files=0 | open: | closed: | roadmap=0
S23 | the catalog sync fails for a long time | /(catalog|template).sync.{0,100}(fail|stale|days|alarm)/ | ar files=0 | open: | closed: R-607 | roadmap=0
S24 | the 05:30 proof restore is not made up | /proof.{0,120}(not made up|catch.?up|missed)/ | ar files=0 | open: | closed: R-873 R-875 | roadmap=0
S25 | the restore-copy sweep on a box that is on for under six hours | /(sweep|every 6 h).{0,120}(first run|on for under|short uptime)/ | ar files=0 | open: | closed: | roadmap=0
@@ -0,0 +1,78 @@
#!/usr/bin/env python3
"""Print the text behind every pointer in a table file, so a person can read it.
Read-only by construction: it opens files for reading and prints. It runs on DooPlex against the
git clones only; it never touches a box.
Usage: check-pointers.py [FILE ...] (default: ../TABLE.md)
check-pointers.py --quiet [FILE] (only the verdict lines; rc 1 when a pointer is dead)
A pointer is a backticked token `PREFIX/path[:line[-line]]` with one of the prefixes below.
A pointer is DEAD when the file is missing or the line range lies beyond the end of the file.
The script cannot judge whether the line says what the table claims — that is the reader's job.
"""
import os
import re
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
AUDIT = os.path.dirname(HERE)
REPO = os.path.abspath(os.path.join(AUDIT, "..", "..", "..")) # felhom.eu
WORKSPACE = os.path.dirname(REPO)
PREFIX = {
"AG": os.path.join(WORKSPACE, "felhom-agent"),
"CT": os.path.join(WORKSPACE, "felhom-controller", "controller"),
"CAT": os.path.join(WORKSPACE, "app-catalog-felhom.eu"),
"HUB": os.path.join(REPO, "hub", "internal"),
"E": REPO,
"AR": os.path.join(REPO, "documentation", "architecture"),
"RB": os.path.join(REPO, "documentation", "runbooks"),
"EV": os.path.join(AUDIT, "evidence"),
}
TOKEN = re.compile(r"`((?:%s)/[^`:\s]+)(?::(\d+)(?:-(\d+))?)?`" % "|".join(PREFIX))
def main() -> int:
args = [a for a in sys.argv[1:] if a != "--quiet"]
quiet = "--quiet" in sys.argv[1:]
files = args or [os.path.join(AUDIT, "TABLE.md")]
seen, dead, total = set(), 0, 0
for table in files:
with open(table, encoding="utf-8") as fh:
text = fh.read()
for m in TOKEN.finditer(text):
key = m.group(0)
if key in seen:
continue
seen.add(key)
total += 1
pre, rest = m.group(1).split("/", 1)
path = os.path.join(PREFIX[pre], rest)
first = int(m.group(2)) if m.group(2) else None
last = int(m.group(3)) if m.group(3) else first
if not os.path.exists(path):
dead += 1
print(f"DEAD {key} (no such file: {path})")
continue
if first is None:
if not quiet:
print(f"OK {key} (file exists)")
continue
with open(path, encoding="utf-8", errors="replace") as fh:
lines = fh.readlines()
if last > len(lines):
dead += 1
print(f"DEAD {key} (file has {len(lines)} lines)")
continue
if quiet:
continue
print(f"OK {key}")
for n in range(first, min(last, first + 11) + 1):
print(f" {n:>5} {lines[n - 1].rstrip()[:200]}")
print(f"pointers: {total} dead: {dead}")
return 1 if dead else 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,21 @@
#!/bin/bash
# Clock list 2026-11-11 — guest 9201. READ-ONLY: date, systemctl list/show/cat, cron files, atq, docker ps/logs (read).
set -u
M() { sed -E 's/((token|password|passwd|secret|apikey|api_key|authorization)[=: ]+)[^ ,;"]+/\1***/Ig'; }
h() { echo; echo "=== $* ==="; }
h "clock"; date -u +'utc %FT%T.%NZ'; date +'local %FT%T %Z %z'; cat /etc/timezone 2>/dev/null; readlink /etc/localtime; timedatectl 2>&1 | head -8
for u in chrony systemd-timesyncd; do printf '%s: ' $u; systemctl is-active $u 2>/dev/null; done
h "boots"; uptime -s
h "timers"; systemctl list-timers --all --no-pager
h "timer units (all)"; for t in $(systemctl list-unit-files --type=timer --no-legend --no-pager | awk '{print $1}'); do echo "--- $t [$(systemctl is-enabled $t 2>/dev/null)/$(systemctl is-active $t 2>/dev/null)]"; systemctl cat $t 2>/dev/null | /usr/bin/grep -E '^(On|Persistent|RandomizedDelay|AccuracySec|Unit)'; systemctl show $t -p LastTriggerUSec -p NextElapseUSecRealtime -p Unit 2>/dev/null | tr '\n' ' '; echo; done
h "felhom units"; systemctl list-units --all --no-pager --no-legend 'felhom*' | awk '{print $1,$2,$3,$4}'
for s in $(systemctl list-unit-files --no-legend --no-pager 'felhom*.service' | awk '{print $1}'); do echo "--- $s"; systemctl show $s -p Restart -p RestartUSec -p TimeoutStartUSec -p Type -p ActiveEnterTimestamp -p NRestarts 2>/dev/null | tr '\n' ' '; echo; done
h "cron"; /usr/bin/grep -v -E '^\s*(#|$)' /etc/crontab 2>/dev/null | M; for f in /etc/cron.d/*; do [ -f "$f" ] && { echo "--- $f"; /usr/bin/grep -v -E '^\s*(#|$)' "$f" | M; }; done; for d in hourly daily weekly monthly; do echo "--- cron.$d: $(ls /etc/cron.$d 2>/dev/null | tr '\n' ' ')"; done
echo "--- user crontabs: $(ls /var/spool/cron/crontabs 2>/dev/null | tr '\n' ' ')"
h "at"; command -v atq >/dev/null && atq 2>&1 || echo "no at"
h "docker daemon timers"; /usr/bin/grep -v -E '^\s*$' /etc/docker/daemon.json 2>/dev/null | M
h "containers"; docker ps -a --format '{{.Names}}\t{{.Image}}\t{{.Status}}\t{{.RunningFor}}'
h "restart policies + healthcheck intervals"; for c in $(docker ps -q); do docker inspect --format '{{.Name}} restart={{.HostConfig.RestartPolicy.Name}} health={{if .Config.Healthcheck}}{{.Config.Healthcheck.Interval}}{{else}}none{{end}} started={{.State.StartedAt}}' $c; done
h "controller TZ"; docker exec felhom-controller sh -c 'printf "TZ=%s\n" "${TZ:-unset}"' 2>&1; docker exec felhom-controller date +'in-container %FT%T %Z %z' 2>&1
h "controller log: first line, line count per hour"; docker logs --timestamps felhom-controller 2>&1 | head -1 | cut -c1-200 | M; docker logs --timestamps --since 30h felhom-controller 2>&1 | cut -c1-13 | sort | uniq -c | tr '\n' ' '
h "controller log: schedule words (tail 320)"; docker logs --timestamps --since 72h felhom-controller 2>&1 | /usr/bin/grep -i -E 'schedul|night|backup (start|finish|complet|done|fail)|dump|restic|snapshot|prune|offbox|off-site|offsite|second.?drive|window|slot|missed|catch.?up|restore.?test|report (sent|fail)|heartbeat|sync(ed|ing)? |update check|self.?update|clock|skew|deadline|timed out|timeout|cron' | /usr/bin/grep -v -i -E 'GET /|POST /' | cut -c1-240 | M | tail -320
@@ -0,0 +1,14 @@
#!/bin/bash
# guest-night.sh — read-only: per scheduler job, count, first/last start, the last three durations; and night lines.
L=$(mktemp); docker logs --timestamps felhom-controller > "$L" 2>&1
printf 'log first line: %s\nlog last line: %s\n' "$(head -1 "$L" | cut -c1-30)" "$(tail -1 "$L" | cut -c1-30)"
printf '\n=== scheduler jobs (UTC): runs, first start, last start, last three "took" ===\n'
grep -E '\[scheduler\] (Running job: |Job .* (completed|failed|skipped))' "$L" | awk '
/ Running job: / { j=$NF; n[j]++; if(!(j in f)) f[j]=substr($1,1,19); l[j]=substr($1,1,19); next }
/ Job / { for(i=1;i<=NF;i++) if($i=="Job") {j=$(i+1); break}; s=$0; sub(/^.*\[scheduler\] Job [^ ]+ /,"",s); d[j]=d3[j]; d3[j]=d2[j] " | " s; d2[j]=s; last[j]=(last[j]==""?s:last2[j]); h[j]=h[j] "#" s; }
END { for (j in n) { k=split(h[j],p,"#"); o=""; for(i=(k>3?k-2:2);i<=k;i++) o=o " [" p[i] "]"; printf "%-34s runs=%-5d first=%s last=%s%s\n", j, n[j], f[j], l[j], o } }' | sort
printf '\n=== jobs that ran fewer than 30 times: every start and end (UTC) ===\n'
grep -E '\[scheduler\] (Running job: |Job .* (completed|failed|skipped))' "$L" | awk '{ if ($0 ~ / Running job: /) j=$NF; else { for(i=1;i<=NF;i++) if($i=="Job") {j=$(i+1); break} } c[j]++; line[NR]=$0; jj[NR]=j } END { for(i=1;i<=NR;i++) if (c[jj[i]]<60) { s=line[i]; sub(/^[^ ]+ /,"",s); sub(/scheduler.go:[0-9]+: /,"",s); print substr(s,1,170) } }' | tail -120
printf '\n=== night lines 19:00-08:00 UTC, INFO/WARN/ERROR, backup words (first 2 per message shape) ===\n'
grep -E 'T(19|2[0-3]|0[0-7]):' "$L" | grep -E '\[(INFO|WARN|ERROR)\]' | grep -v '\[scheduler\]' | grep -i -E 'backup|restic|offbox|offsite|restore|prune|dump|verify|snapshot|retention|forget|check' | awk '{ k=$0; sub(/^[^ ]+ [^ ]+ [^ ]+ /,"",k); gsub(/[0-9]+/,"N",k); k=substr(k,1,48); c[k]++; if (c[k]<=2) { s=$0; sub(/^[^ ]+ /,"",s); print substr(s,1,230) } }' | head -90
rm -f "$L"
@@ -0,0 +1,30 @@
#!/bin/bash
# Clock list 2026-11-11 — host. READ-ONLY by construction: date, timedatectl (show), chronyc (read), systemctl list/show/cat,
# ls and cat of cron files, atq, journalctl (read), the PVE task index (read). It starts no job and writes no file.
set -u
M() { sed -E 's/((token|password|passwd|secret|apikey|api_key|authorization)[=: ]+)[^ ,;"]+/\1***/Ig'; }
h() { echo; echo "=== $* ==="; }
h "in-flight"; pgrep -x vzdump >/dev/null && echo "vzdump RUNNING"; pgrep -x apt-get >/dev/null && echo "apt RUNNING"; pgrep -f 'pct restore' >/dev/null && echo "restore RUNNING"; echo "checked"
h "clock"; date -u +'utc %FT%T.%NZ'; date +'local %FT%T %Z %z'; cat /etc/timezone 2>/dev/null; readlink /etc/localtime
timedatectl show 2>/dev/null; timedatectl 2>/dev/null
h "who keeps the time"; for u in chrony chronyd systemd-timesyncd ntp ntpsec openntpd; do printf '%s: ' $u; systemctl is-active $u 2>/dev/null; printf ' enabled: '; systemctl is-enabled $u 2>/dev/null; done
command -v chronyc >/dev/null && { chronyc -n tracking 2>&1; chronyc -n sources 2>&1; }
timedatectl timesync-status 2>&1 | head -12
/usr/bin/grep -v -E '^\s*(#|$)' /etc/chrony/chrony.conf 2>/dev/null; ls /etc/chrony/conf.d /etc/chrony/sources.d 2>/dev/null
/usr/bin/grep -v -E '^\s*(#|$)' /etc/systemd/timesyncd.conf 2>/dev/null
h "start with no time"; ls -l --time-style=full-iso /var/lib/systemd/timesync/clock /var/lib/chrony/ 2>&1 | head; dpkg -l fake-hwclock 2>/dev/null | tail -1; cat /etc/adjtime 2>/dev/null; hwclock --show --utc 2>&1 | head -2; cat /sys/class/rtc/rtc0/since_epoch 2>/dev/null; date -u +%s
h "boots"; uptime -s; journalctl --list-boots --no-pager 2>/dev/null | tail -6
h "timers"; systemctl list-timers --all --no-pager
h "timer units (all)"; for t in $(systemctl list-unit-files --type=timer --no-legend --no-pager | awk '{print $1}'); do echo "--- $t [$(systemctl is-enabled $t 2>/dev/null)/$(systemctl is-active $t 2>/dev/null)]"; systemctl cat $t 2>/dev/null | /usr/bin/grep -E '^(On|Persistent|RandomizedDelay|AccuracySec|Unit|FixedRandomDelay|WakeSystem)' ; systemctl show $t -p LastTriggerUSec -p NextElapseUSecRealtime -p Unit 2>/dev/null | tr '\n' ' '; echo; done
h "service limits of felhom + backup timers"; for t in $(systemctl list-unit-files --type=timer --no-legend --no-pager | awk '{print $1}' | /usr/bin/grep -i -E 'felhom|pve|pbs|apt|unattended|fstrim|logrotate|man-db|e2scrub|dpkg|smart'); do s=$(systemctl show $t -p Unit --value); echo "--- $s"; systemctl show "$s" -p TimeoutStartUSec -p RuntimeMaxUSec -p ExecStart -p Type -p ExecMainStartTimestamp -p ExecMainExitTimestamp -p ExecMainStatus 2>/dev/null | cut -c1-300 | M; done
h "felhom units that are not timers"; systemctl list-units --all --no-pager --no-legend 'felhom*' | awk '{print $1,$2,$3,$4}'
for s in $(systemctl list-unit-files --no-legend --no-pager 'felhom*.service' | awk '{print $1}'); do echo "--- $s"; systemctl show $s -p Restart -p RestartUSec -p WatchdogUSec -p TimeoutStartUSec -p RuntimeMaxUSec -p ActiveEnterTimestamp -p NRestarts 2>/dev/null | tr '\n' ' '; echo; done
h "cron"; /usr/bin/grep -v -E '^\s*(#|$)' /etc/crontab 2>/dev/null | M; for f in /etc/cron.d/*; do echo "--- $f"; /usr/bin/grep -v -E '^\s*(#|$)' "$f" | M; done; for d in hourly daily weekly monthly; do echo "--- cron.$d: $(ls /etc/cron.$d 2>/dev/null | tr '\n' ' ')"; done
echo "--- user crontabs: $(ls /var/spool/cron/crontabs 2>/dev/null | tr '\n' ' ')"; for f in /var/spool/cron/crontabs/*; do [ -f "$f" ] && { echo "--- $f"; /usr/bin/grep -v -E '^\s*(#|$)' "$f" | M; }; done
h "at"; command -v atq >/dev/null && atq 2>&1 || echo "no at"
h "pve jobs"; cat /etc/pve/jobs.cfg 2>/dev/null | M; echo "--- vzdump.cron"; /usr/bin/grep -v -E '^\s*(#|$)' /etc/pve/vzdump.cron 2>/dev/null; echo "--- vzdump.conf"; /usr/bin/grep -v -E '^\s*(#|$)' /etc/vzdump.conf 2>/dev/null
h "pve tasks (start end type id status), last 14 days"; cat /var/log/pve/tasks/index.1 /var/log/pve/tasks/index 2>/dev/null | awk -v now=$(date +%s) '{split($1,a,":"); st=strtonum("0x" a[5]); en=strtonum("0x" $2); if (now-st < 14*86400) printf "%s %s dur=%ds %s %s %s\n", strftime("%FT%T%z",st), strftime("%T",en), en-st, a[6], a[7], $3}' | /usr/bin/grep -v -E ' (vncproxy|vncshell|spiceproxy) ' | tail -150
h "runs of timer services, last 10 days (start/finish lines)"; for t in $(systemctl list-unit-files --type=timer --no-legend --no-pager | awk '{print $1}' | /usr/bin/grep -i -E 'felhom|pve-daily|apt-daily|pbs'); do s=$(systemctl show $t -p Unit --value); echo "--- $s"; journalctl -u "$s" --since '-10 days' -o short-iso --no-pager -q 2>/dev/null | /usr/bin/grep -E 'Starting|Started|Finished|Failed|Deactivated|Consumed' | tail -9 | cut -c1-220 | M; done
h "agent log: schedule words, last 3 days (tail 260)"; journalctl -u felhom-agent --since '-3 days' -o short-iso --no-pager -q 2>/dev/null | /usr/bin/grep -i -E 'night|vzdump|backup|restore.?test|osupdate|os-update|kernel|slot|window|schedul|prune|pbs|report (sent|fail)|clock|skew|drift|missed|catch.?up|deadline|timeout' | cut -c1-260 | M | tail -260
h "agent log: line count per hour, last 26 h"; journalctl -u felhom-agent --since '-26 hours' -o short-iso --no-pager -q 2>/dev/null | cut -c1-13 | sort | uniq -c | tr '\n' ' '
h "agent version"; /usr/local/bin/felhom-agent --version 2>&1 | head -1
@@ -0,0 +1,10 @@
#!/bin/bash
# host-night.sh — read-only: PVE task index (start, end, minutes, type, id, status) for the last 14 days; agent night lines.
printf '=== pve tasks, last 14 days (local time) ===\n'
cat /var/log/pve/tasks/index.1 /var/log/pve/tasks/index 2>/dev/null | perl -ne '
next unless /^UPID:[^:]+:[0-9A-Fa-f]+:[0-9A-Fa-f]+:([0-9A-Fa-f]+):([^:]+):([^:]*):[^:]+: ([0-9A-Fa-f]+) (.*)$/;
my ($s,$t,$id,$e,$st)=(hex($1),$2,$3,hex($4),$5); next if $s < time-14*86400;
my @a=localtime($s); my @b=localtime($e);
printf "%04d-%02d-%02d %02d:%02d:%02d -> %02d:%02d:%02d %6.1f min %-12s %-8s %s\n",$a[5]+1900,$a[4]+1,$a[3],$a[2],$a[1],$a[0],$b[2],$b[1],$b[0],($e-$s)/60,$t,$id,substr($st,0,60);' | sort | grep -v -E ' (vncproxy|vncshell|spiceproxy) ' | tail -150
printf '\n=== agent: night words, 20:00 yesterday to 09:00 today (first 3 lines per message) ===\n'
journalctl -u felhom-agent --since "yesterday 20:00" --until "today 09:00" --no-pager -o short-iso 2>/dev/null | grep -E 'level=(INFO|WARN|ERROR)' | grep -i -E 'backup|vzdump|verify|restore-test|prune|update|osupdate|escrow|gc|sweep|retention|offsite|window|sched' | perl -ne 'my ($m)=/msg="([^"]{0,70})/; $m//="?"; $c{$m}++; if($c{$m}<=3){ s/^(\S+) \S+ \S+ time=\S+ /$1 /; print substr($_,0,260),"\n" } END{ print "--- counts\n"; printf "%6d %s\n",$c{$_},$_ for sort keys %c }'
@@ -0,0 +1,9 @@
"""listconf.py — the Clock list's own words for make-table.py and make-matrix.py. Data only."""
HELPERS = ("clock-controller", "clock-agent", "clock-hub")
KINDS = ["Night", "Interval", "Watchdog", "Hub daily", "Hub sweep", "DooPlex and ep0"]
WORD, WORDS = "job", "Jobs"
TITLE = "The Clock list"
ID_RE = r"C[CAH]\d\d"
QUESTIONS = ["what runs, owner, what starts it", "when, how often, order", "how long, limit, 10x data", "if it does not run",
"late, twice, meets another job", "clock wrong or jumps", "who is told, when", "how do we know"]
CLASS_ROW = {"W1": "R-1383", "W2": "R-1384", "W3": "R-1385", "W4": "R-1386", "W5": "R-1387"}
@@ -0,0 +1,131 @@
#!/usr/bin/env python3
"""Build MATRIX.md from TABLE.md, and check the table's shape.
The same tool as `audits/endings-list-2026-11-10/tools/make-matrix.py`, made general: the list's own words are in `listconf.py`.
Read-only on everything but MATRIX.md in this folder. It runs on DooPlex against the git clone only.
Usage: make-matrix.py write ../MATRIX.md, print the totals
make-matrix.py --check write nothing; rc 1 when the table breaks a rule
Rules checked (a broken rule is printed and makes rc 1):
- a heading is `### <ID> — <name> · <seed kind> · NAMED|MENTIONED|UNNAMED`
- each block has exactly eight answer rows, numbered 1..8, each with one mark A, D, K or N
- an A row carries a backticked pointer; a K row an `R-<n>`; an N row one class from listconf.py and a search S<n>
"""
import os
import re
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import listconf as C
HERE = os.path.dirname(os.path.abspath(__file__))
AUDIT = os.path.dirname(HERE)
HEAD = re.compile(r"^### (" + C.ID_RE + r") — (.+?) · (?:" + "|".join(C.KINDS) + r") · (NAMED|MENTIONED|UNNAMED)\s*$")
ROW = re.compile(r"^\| ([1-8]) \| (.*) \| ([ADKN]) \| (.*) \|\s*$")
QUESTIONS = C.QUESTIONS
CLASS_ROW = C.CLASS_ROW
CLS = re.compile(r"\b(?:" + "|".join(C.CLASS_ROW) + r")\b")
def parse(path):
places, errors, cur = [], [], None
with open(path, encoding="utf-8") as fh:
for ln, line in enumerate(fh, 1):
if line.startswith("## TEST-ONLY") or line.startswith("## DOC-ONLY"):
cur = None
continue
if line.startswith("### "):
m = HEAD.match(line)
if not m:
errors.append(f"line {ln}: heading has the wrong shape: {line.strip()}")
cur = None
continue
cur = {"id": m.group(1), "name": m.group(2), "named": m.group(3), "marks": {}, "n": {}, "k": {}}
places.append(cur)
continue
m = ROW.match(line)
if not m or cur is None:
continue
q, mark, ptr = int(m.group(1)), m.group(3), m.group(4)
if q in cur["marks"]:
errors.append(f"{cur['id']}: question {q} is answered twice")
cur["marks"][q] = mark
if mark == "A" and "`" not in ptr:
errors.append(f"{cur['id']}-{q}: A with no pointer")
if mark == "K":
rows = re.findall(r"R-\d+", ptr)
if not rows:
errors.append(f"{cur['id']}-{q}: K with no row")
cur["k"][q] = rows
if mark == "N":
cls = CLS.findall(ptr)
if len(cls) != 1:
errors.append(f"{cur['id']}-{q}: N needs exactly one class, found {cls}")
elif not re.search(r"\bS\d+\b", ptr) and "NOBODY KNOWS" not in m.group(2):
errors.append(f"{cur['id']}-{q}: N with no search id")
cur["n"][q] = cls[0] if cls else "?"
for p in places:
if sorted(p["marks"]) != list(range(1, 9)):
errors.append(f"{p['id']}: has answers {sorted(p['marks'])}, wants 1..8")
return places, errors
def build(places):
out = ["# " + C.TITLE + " — the matrix", "",
"Made by `tools/make-matrix.py` from `TABLE.md`. Do not edit it by hand.", "",
"`A` answered · `D` decided, not built · `K` known gap (has a row) · `N` nobody asked. Columns are the eight",
"questions: " + " · ".join(f"{i} {q}" for i, q in enumerate(QUESTIONS, 1)) + ".",
"", "| " + C.WORD.capitalize() + " | Named | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | N |", "|---|---|---|---|---|---|---|---|---|---|---|"]
tot = {m: 0 for m in "ADKN"}
perq = {q: {m: 0 for m in "ADKN"} for q in range(1, 9)}
named = {"NAMED": 0, "MENTIONED": 0, "UNNAMED": 0}
members, krows = {c: [] for c in CLASS_ROW}, {}
for p in places:
named[p["named"]] += 1
cells = []
for q in range(1, 9):
mk = p["marks"].get(q, "?")
cells.append(f"**{mk}**" if mk == "N" else mk)
if mk in tot:
tot[mk] += 1
perq[q][mk] += 1
for q, c in p["n"].items():
members.setdefault(c, []).append(f"{p['id']}-{q}")
for q, rows in p["k"].items():
for r in rows:
krows.setdefault(r, []).append(f"{p['id']}-{q}")
out.append(f"| {p['id']} {p['name']} | {p['named'].lower()} | " + " | ".join(cells) + f" | {len(p['n'])} |")
n = len(places)
out += ["", "## Totals", "",
f"- {C.WORDS}: **{n}** in the table · named **{named['NAMED']}** · mentioned only **{named['MENTIONED']}** · "
f"named by no architecture document **{named['UNNAMED']}**.",
f"- Answers: **{n * 8}** · A **{tot['A']}** · D **{tot['D']}** · K **{tot['K']}** · N **{tot['N']}**.",
f"- {C.WORDS} with no N: **{sum(1 for p in places if not p['n'])}** of {n}.",
"", "| Question | A | D | K | N |", "|---|---|---|---|---|"]
for q in range(1, 9):
out.append(f"| {q} {QUESTIONS[q - 1]} | {perq[q]['A']} | {perq[q]['D']} | {perq[q]['K']} | {perq[q]['N']} |")
out += ["", "## The N answers, by class row", "", "| Class | Row | Count | Members (id-question) |", "|---|---|---|---|"]
for c in sorted(members):
out.append(f"| {c} | {CLASS_ROW.get(c, '?')} | {len(members[c])} | {', '.join(members[c])} |")
out += ["", "## The K answers, by existing row", "", "| Row | Count | Members |", "|---|---|---|"]
for r in sorted(krows, key=lambda x: int(x[2:])):
out.append(f"| {r} | {len(krows[r])} | {', '.join(krows[r])} |")
return "\n".join(out) + "\n", tot, named, members
def main() -> int:
places, errors = parse(os.path.join(AUDIT, "TABLE.md"))
for e in errors:
print("BROKEN " + e)
text, tot, named, members = build(places)
if "--check" not in sys.argv[1:]:
with open(os.path.join(AUDIT, "MATRIX.md"), "w", encoding="utf-8") as fh:
fh.write(text)
print(f"places {len(places)} named {named} marks {tot} "
+ " ".join(f"{c}={len(m)}" for c, m in sorted(members.items())) + f" broken {len(errors)}")
return 1 if errors else 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,126 @@
#!/usr/bin/env python3
"""make-table.py — build TABLE.md from the helpers' answers (evidence/helpers/resolved-*.json), marks.py and listconf.py.
The same tool as `audits/endings-list-2026-11-10/tools/make-table.py`, made general: what differs between two lists
(the heading words, the helper names, the kinds, the questions) is in `listconf.py` beside it.
A cell with an entry in marks.py carries the MAIN SESSION's words. Every other cell carries the helper's words and the
mark A; its pointers were opened by resolve-helper.py (the file and the line exist; the name the helper gave stands at
the line). A dead pointer is dropped here. Read-only but for TABLE.md.
"""
import json, os, re, subprocess, sys
import marks, listconf as C
HERE = os.path.dirname(os.path.abspath(__file__)); AUDIT = os.path.dirname(HERE)
REPO = os.path.abspath(os.path.join(HERE, "..", "..", "..", "..")); WS = os.path.dirname(REPO)
PREFIX = {"AG": os.path.join(WS, "felhom-agent"), "CT": os.path.join(WS, "felhom-controller", "controller"),
"CAT": os.path.join(WS, "app-catalog-felhom.eu"), "HUB": os.path.join(REPO, "hub", "internal"), "E": REPO,
"AR": os.path.join(REPO, "documentation", "architecture"), "RB": os.path.join(REPO, "documentation", "runbooks"),
"EV": os.path.join(AUDIT, "evidence")}
TOK = re.compile(r"`((?:AG|CT|CAT|HUB|E|AR|RB|EV)/[^`:\s]+)(?::(\d+)(?:-(\d+))?)?`")
R = {h: json.load(open(os.path.join(AUDIT, "evidence", "helpers", f"resolved-{h}.json")))["endings"] for h in C.HELPERS}
dropped = []
def live(tok):
m = TOK.fullmatch(tok)
pre, rest = m.group(1).split("/", 1)
p = os.path.join(PREFIX[pre], rest)
if not os.path.exists(p):
return False
if m.group(2):
n = sum(1 for _ in open(p, encoding="utf-8", errors="replace"))
return int(m.group(3) or m.group(2)) <= n
return True
def split(ans, eid, q):
"""helper answer -> (plain text, [live pointers])"""
ptrs = []
for m in TOK.finditer(ans):
t = m.group(0)
if live(t):
if t not in ptrs:
ptrs.append(t)
else:
dropped.append(f"{eid}-{q} {t}")
t = TOK.sub("", ans)
t = re.sub(r"`~?:\d+(?:-\d+)?`", "", t)
t = re.sub(r"\(\s*(?:called|creator|see)?\s*[,;]?\s*\)", "", t)
t = re.sub(r"\s+([,.;:)])", r"\1", t)
t = re.sub(r"\(\s+", "(", t)
t = re.sub(r"([,;:])(?:\s*[,;:])+", r"\1", t)
t = re.sub(r"\s{2,}", " ", t).replace("|", "/").strip(" ;,·")
t = re.sub(r"\(\)", "", t)
if len(t) > 380:
cut = max(t.rfind(". ", 0, 380), t.rfind("; ", 0, 380))
t = t[:cut + 1 if cut > 150 else 377].rstrip(" ;") + " …"
return t, ptrs
def named(m):
kind, why = m["named"]
if kind != "grep":
return kind, f"architecture: {why}"
r = subprocess.run(["/usr/bin/grep", "-rEic", "--include=*.md", why, PREFIX["AR"]], capture_output=True, text=True)
files = [l for l in r.stdout.split("\n") if l and not l.endswith(":0")]
lines = sum(int(l.rsplit(":", 1)[1]) for l in files)
return ("MENTIONED" if lines else "UNNAMED"), f"architecture folder: {lines} lines in {len(files)} files match `/{why}/` (a machine count, not a reading)"
def main():
out = [open(os.path.join(HERE, "table-head.md"), encoding="utf-8").read().rstrip(), ""]
kindmap, bad, body = {}, [], []
for eid, m in sorted(marks.M.items()):
e = R[m["src"][0]][m["src"][1]]
st, why = named(m)
worst, rows = "A", []
for q in range(1, 9):
if q in m["o"]:
mark, text, ptr = m["o"][q]
if ptr is None or ptr.startswith("+"): # "+ rest" = the helper's pointers, then rest
_, ps = split(e["answers"].get(str(q), ""), eid, q)
ptr = (" ".join(ps[:4]) + (ptr[1:] if ptr else "")).strip(" ·")
else:
mark = "A"
text, ps = split(e["answers"].get(str(q), ""), eid, q)
ptr = " ".join(ps[:4])
if q == 8 and eid in marks.MEASURED:
mt, mp = marks.MEASURED[eid]
text = mt + " · " + text
ptr = (mp + " " + ptr).strip()
if mark == "A" and "`" not in ptr: # borrow the code path from question 1, then 2
for alt in (1, 2, 3):
_, ps = split(e["answers"].get(str(alt), ""), eid, alt)
if ps:
ptr = " ".join(ps[:2]) + f" (the code path of question {alt})"
break
if mark == "A" and "`" not in ptr:
bad.append(f"{eid}-{q}: A with no pointer")
if not text:
bad.append(f"{eid}-{q}: empty answer")
rows.append(f"| {q} | {text} | {mark} | {ptr} |")
worst = max(worst, mark, key="ADKN".index)
kindmap.setdefault(m["seed"], []).append((eid, st, worst))
body += [f"### {eid} — {m['name']} · {m['seed']} · {st}", "",
f"{why} · helper section: {m['src'][0]} „{e['title'].strip()}”", "",
"| # | Answer | Mark | Pointer |", "|---|---|---|---|"] + rows + ["", f"**Hand step in a runbook?** {m['rb']}", ""]
out += [f"## The kinds, and where each {C.WORD} is", "", f"| Kind | {C.WORDS} (worst mark) | Named · mentioned · named nowhere |", "|---|---|---|"]
for kind in C.KINDS:
v = kindmap.get(kind, [])
c = {s: sum(1 for x in v if x[1] == s) for s in ("NAMED", "MENTIONED", "UNNAMED")}
out.append(f"| {kind} | " + ", ".join(f"{x[0]} ({x[2]})" for x in v) + f" | {c['NAMED']} · {c['MENTIONED']} · {c['UNNAMED']} |")
for k in kindmap:
if k not in C.KINDS:
bad.append(f"kind {k} is not in listconf.KINDS")
out += ["", "## Not in the table", "",
"**Tiny (named once, not tabled):** " + " · ".join(marks.TINY) + ".", "",
"**Left out for another reason:**", ""] + [f"- {a} — {b}." for a, b in marks.LEFT_OUT]
out += ["", "---", ""] + body
open(os.path.join(AUDIT, "TABLE.md"), "w", encoding="utf-8").write("\n".join(out).rstrip() + "\n")
print(f"{C.WORDS} {len(marks.M)} · dead pointers dropped {len(dropped)} · broken {len(bad)}")
for b in bad + dropped:
print(" " + b)
return 1 if bad else 0
sys.exit(main())
@@ -0,0 +1,290 @@
"""marks.py — the main session's rulings for the Clock list (2026-11-11). Data only; make-table.py reads it.
One entry for each job that is in the table:
src (helper, index) the helper section the default answers come from (evidence/helpers/resolved-<helper>.json)
name the heading
seed the kind (Night, Interval, Watchdog, Hub daily, Hub sweep, DooPlex and ep0)
named ("grep", REGEX): 0 lines in the architecture folder = UNNAMED, else MENTIONED
o {question: (mark, answer, pointers)} — the MAIN SESSION's words.
pointers "+ rest" = the helper's pointers for that question (opened by the tool), then rest.
A pointer written out here was opened by eye.
A question with no entry here takes the helper's answer and the mark A.
rb the ninth line: is the hand step written in a runbook?
"""
NM = "NOT MEASURED (DooPlex and ep0 are outside this run). "
def N(cls, s, text, p="+"):
return ("N", text, f"{p} · class {cls} · S{s:02d}")
def K(row, text, p="+"):
return ("K", text, f"{p} · R-{row}")
def A(text, p="+"):
return ("A", text, p)
NOLIMIT = "No time limit in the source, and no duration was measured. Nobody knows how long it takes on a big box."
NH = "no hand step"
_M = [
# ---------------------------------------------------------------- controller (in the guest)
("CC01", "clock-controller", 0, "Night dump leg: databases and volumes · CUSTOMER", "Night", r"db-dump|dump leg",
{3: A("MEASURED, the night of 2026-11-11: 2 min 33 s on demo-hp, 40 s on Tester 1, 18 s on the N100. One database dump has a 5-minute limit. The whole leg has no limit. It grows with the number and size of the apps; all three stores are small.",
"`EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt` `CT/internal/appbackup/dbdump.go:291`")}, NH),
("CC02", "clock-controller", 1, "Second-drive copy leg · CUSTOMER", "Night", r"tier.?2|second.drive|cross.drive",
{5: N("W3", 14, "It waits for the dump leg by a start time only (60 minutes later). It is skipped while a migration runs. The file holds no lock of its own. With a backup time between 02:00 and 02:59 on the spring hour-change night the two legs start at the same instant; that case is row R-1260."),
7: A("The household and the hub are told of a failed copy: the copy's result goes to the notifier.", "`CT/cmd/controller/main.go:1427`")}, NH),
("CC03", "clock-controller", 2, "Off-site push leg, with the clean-up of old snapshots · CUSTOMER", "Night", r"offbox|off.site push|offsite push",
{5: K(1186, "When another backup holds the flag, the push is skipped with an INFO line and the leg is marked as ended. Only the hub's 48-hour check notices. A long second-drive leg can cost the night's off-site copy.",
"`CT/internal/backup/offbox.go:1054-1057`")},
"NO — no runbook says how to see or repair a skipped off-site night"),
("CC04", "clock-controller", 3, "Missed-night catch-up and the resume watch · CUSTOMER", "Night", r"catch.?up|nightchain|missed night",
{6: N("W2", 8, "The watch reads a gap of more than 5 minutes in the wall clock as 'the box was asleep'. A clock that is corrected forward by more than 5 minutes looks the same. The design text says this was not measured live.")}, NH),
("CC05", "clock-controller", 4, "Missed-backup banner (26 hours) · CUSTOMER", "Night", r"26 h|missed.backup banner|banner",
{6: N("W2", 8, "The age is wall-clock arithmetic. A clock that is set back hides the banner for as long as the step. No test sets the clock back.")}, NH),
("CC06", "clock-controller", 5, "Whole-guest backup driver: stop apps, ask the agent, start apps · CUSTOMER", "Night", r"quiesce|whole.guest|vzdump",
{6: K(1260, "The gate counts wall-clock minutes; the poll is a monotonic ticker. On the hour-change night the two disagree. No test covers that night.")}, NH),
("CC07", "clock-controller", 6, "Automatic app-update leg · CUSTOMER", "Night", r"update.leg|unattended",
{6: K(1260, "The leg's deadline is the backup time plus 5 hours of real time, and the scheduler's slots are in Budapest time. On the hour-change night a step can start after the backup gate stopped waiting. A daily fire that is more than 60 minutes late is skipped, so no update runs at noon."),
7: N("W1", 6, "Nobody. When the leg does not run, the only trace is one log line on the box. No event goes to the hub.", "`CT/internal/stacks/unattended.go:374-375`")}, NH),
("CC08", "clock-controller", 7, "Off-site integrity check · CUSTOMER", "Night", r"integrity",
{3: K(401, "MEASURED 2026-11-11: under 1 second on all three boxes (small stores). The check reads the whole store; the time on a large store rests on one measurement.", "`EV/box/demo-hp-guest-night.txt`"),
7: A("A failed check goes to the notifier: the household and the hub are told.", "`CT/cmd/controller/main.go:4144-4186`")}, NH),
("CC09", "clock-controller", 8, "Off-site proof restore · CUSTOMER", "Night", r"proof restore|offsite.proof|proof",
{5: N("W3", 12, "MEASURED 2026-11-11: on Tester 1 and on the N100 the whole-guest backup started at about 05:30 local time, the same minute as this job. This job is not behind the whole-guest gate. It takes the same flag as the off-site push; a meeting there is a silent skip (R-1186).",
"`EV/box/tester-1-host-night.txt` `EV/box/demo-felhom-host-night.txt` `EV/box/tester-1-guest-night.txt`"),
7: A("An empty or failed proof goes to the notifier: the household and the hub are told.", "`CT/cmd/controller/main.go:4144-4186`")}, NH),
("CC10", "clock-controller", 9, "Hub report: every 15 minutes, on change, and the hanging wait · CUSTOMER", "Interval", r"15.min|hub report|pusher",
{7: A("Nobody on the box: the box stores the time of the last attempt only. The outside sign is the hub's liveness check (CH01).", "`CT/internal/report/pusher.go:109-111` `HUB/monitor/host_staleness.go:59-60`")}, NH),
("CC11", "clock-controller", 10, "Controller self-update: check, floor, nightly auto · CUSTOMER", "Interval", r"self.?update",
{3: N("W4", 15, "The registry call has a 15-second limit. The swap of the container has no limit in the file, and no duration was measured in this run."),
5: N("W3", 12, "The auto-update waits while a backup or an app update runs. Its default time, 04:30, is the minute at which the whole-guest gate opens for a 02:30 backup time. Nothing orders the two.", "`CT/internal/config/config.go:419` `CT/internal/selfupdate/updater.go:533-544`"),
6: A("The check uses a monotonic ticker. The nightly auto-update is a daily slot (see CC22). The switch is off unless the config turns it on; the default time is 04:30.", "`CT/internal/config/config.go:278-279` `CT/internal/config/config.go:419`")}, NH),
("CC12", "clock-controller", 11, "Catalog sync: app templates from git · CUSTOMER", "Interval", r"catalog sync|git.sync|sync",
{3: N("W4", 15, "Each git command has its own limit. One whole sync has none, and no duration was measured.", "`CT/internal/sync/sync.go:724` `CT/internal/sync/sync.go:215-216`"),
4: N("W1", 23, "A missed sync is done at the next tick. A sync that fails for days writes an ERROR line on the box at each tick. Nobody is told.", "`CT/internal/sync/sync.go:236` `CT/internal/sync/sync.go:247`"),
5: A("Two syncs cannot run at once: `doSync` takes the syncer's lock first. The 30-second guard is for the button only.", "`CT/internal/sync/sync.go:215-216` `CT/internal/sync/sync.go:153-155`")}, NH),
("CC13", "clock-controller", 12, "System health check and base-stack self-heal · CUSTOMER", "Interval", r"health check|self.heal",
{3: N("W4", 15, NOLIMIT)}, NH),
("CC14", "clock-controller", 13, "Dead-app scan · CUSTOMER", "Interval", r"stack.scan|dead.app|crash.?loop",
{3: N("W4", 15, NOLIMIT + " It grows with the number of apps."),
6: N("W2", 8, "The quiet time after a start uses the wall clock. A clock that steps back after the start makes the quiet time longer. No test.")}, NH),
("CC15", "clock-controller", 14, "Disk-fill watch · CUSTOMER", "Interval", r"fill.?watch|disk.fill",
{3: N("W4", 15, NOLIMIT),
5: A("The night slot and the interval job are two jobs with two names. The watcher holds one lock, so the two cannot change its state at the same time.", "`CT/internal/fillwatch/fillwatch.go:162` `CT/cmd/controller/main.go:1804-1805`"),
7: A("The household: the watcher pushes a customer event when a disk fills.", "`CT/internal/fillwatch/fillwatch.go:80-153`")}, NH),
("CC16", "clock-controller", 15, "Drive health check, hourly · CUSTOMER", "Interval", r"SMART|disk.health|drive health",
{3: N("W4", 15, "One fetch was measured at about 0.8 s (a comment in the source). No time limit in the source."),
7: A("The household and the hub: the check emits `disk_health_degraded`.", "`CT/internal/web/disk_health.go:175-280`")}, NH),
("CC17", "clock-controller", 16, "Agent channel health, every 60 seconds · CUSTOMER", "Interval", r"agent channel|channel health", {}, NH),
("CC18", "clock-controller", 17, "Held-app report to the hub · CUSTOMER", "Interval", r"held.app|held app",
{6: N("W2", 8, "The daily resend is wall-clock arithmetic. A clock that is set back delays the resend by the size of the step.")}, NH),
("CC19", "clock-controller", 18, "Abandoned off-site store sweep · CUSTOMER", "Night", r"abandon",
{3: A("One sweep has a 1-minute limit.", "`CT/internal/backup/offbox_abandon.go:180`"),
5: N("W3", 11, "The sweep takes no lock. MEASURED: on the three boxes it ran 55 minutes after the off-site push ended, so they did not meet on these small stores.", "`CT/internal/backup/offbox_abandon.go:180` `EV/box/demo-hp-guest-night.txt`"),
7: N("W1", 6, "Nobody. A sweep that fails writes a WARN or ERROR line on the box only.", "`CT/internal/backup/offbox_abandon.go:180`")}, NH),
("CC20", "clock-controller", 19, "Restore-copy sweep and conversion-copy release · CUSTOMER", "Interval", r"restore.copy|restore copy",
{3: N("W4", 15, NOLIMIT),
4: N("W1", 25, "The first run is 6 hours after the controller starts. A box that is on for under 6 hours a day never runs it; old restore copies stay. Nobody is told."),
7: A("The household is told when a restore copy is deleted.", "`CT/internal/notify/notifier.go:1601`")}, NH),
("CC21", "clock-controller", 21, "Upload and CPU watch, torrent limit rule, line-test hold · CUSTOMER", "Interval", r"upwatch|unexplained|upload",
{5: N("W3", 14, "Two loops with their own tickers. The file holds no lock between them."),
7: A("The household and the hub: the watch emits `app_unexplained_upload` and `app_unexplained_cpu`.", "`CT/internal/upwatch/watch.go:309-382`")}, NH),
("CC22", "clock-controller", 23, "The scheduler itself: daily slots and interval jobs · CUSTOMER", "Night", r"scheduler|daily slot",
{6: K(1260, "Slots are in Budapest time. On the spring night 02:30 does not exist; on the autumn night it comes twice. No test covers either night. A clock set back while a timer waits does not matter: the timer runs on monotonic time.", "`CT/internal/scheduler/scheduler.go:499-519`"),
8: A("SOURCE. MEASURED 2026-11-11: the night's order and the start times on three boxes (see the README's night table).", "`CT/internal/scheduler/scheduler.go:344-347` `EV/box/demo-hp-guest-night.txt`")}, NH),
# ---------------------------------------------------------------- agent (on the host)
("CA01", "clock-agent", 0, "Whole-guest backup: due-check and run · CUSTOMER", "Night", r"vzdump|whole.guest|due.check",
{6: K(1189, "The age is wall-clock time. A host clock that was ahead during one backup gives a negative age afterwards; the tier reads 'not due' until real time passes it."),
8: K(1189, "TEST and DOC for the due-check. MEASURED 2026-11-11: demo-hp 11 to 13 minutes, Tester 1 about 3 minutes, the N100 under 1 minute; on the N100 the start time moved 5 minutes later each day. A clock jump has no test.", "`EV/box/demo-hp-host-night.txt` `EV/box/tester-1-host-night.txt` `EV/box/demo-felhom-host-night.txt`")}, NH),
("CA02", "clock-agent", 1, "OS-update night leg: guest, host, Docker, Proxmox · CUSTOMER", "Night", r"os.?update|OS update",
{6: N("W2", 8, "The 20-hour gap compares the clock with a time read from a file. A stamp that was written while the clock was ahead keeps the leg skipped until real time passes that stamp plus 20 hours.", "`AG/internal/osupdate/leg.go:661-663`"),
8: N("W2", 8, "TEST and DOC for the normal path. A stamp from the future: NOBODY KNOWS.")}, NH),
("CA03", "clock-agent", 2, "Kernel lane: night reboot, 20-minute boot judge, one self-revert · CUSTOMER", "Night", r"kernel",
{6: K(1301, "The 20-minute judge runs on monotonic time; a clock step does not move it. Nothing in the installer sets up time keeping, and a host can start with a wrong clock: the N100's host clock is 32 s behind today."),
8: A("TEST and DOC. The design text itself says that a real freeze was not measured.", "`AR/11-os-updates.md:599-601` `AG/internal/osupdate/kernel_test.go:79`")},
"YES — `RB/crash-guard.md`"),
("CA04", "clock-agent", 3, "Crash guard: early-boot unit and the hourly re-arm timer · CUSTOMER", "Watchdog", r"crash.guard|crash guard",
{5: N("W3", 14, "The second unclean boot inside 60 minutes trips it; it arms again after 24 hours. The boot unit and the hourly check share one state file and take no lock."),
6: N("W2", 22, "The script reads the wall clock at early boot, before any time service runs. A clock that is wrong at boot can move unclean boots into or out of the 60-minute window."),
8: N("W2", 22, "TEST and DOC for the windows. A wrong clock at boot: NOBODY KNOWS.")},
"YES — `RB/crash-guard.md`"),
("CA05", "clock-agent", 4, "Scheduled restore-test evaluation · CUSTOMER", "Night", r"restore.test",
{6: N("W2", 8, "The 'settled' cut-off uses the wall clock. A clock that is set back makes no archive look settled, so no test starts. The ticker itself is monotonic."),
8: N("W2", 8, "TEST and DOC. MEASURED 2026-11-11: restore tests took 1 to 14 minutes on the three boxes. A clock set back: NOBODY KNOWS.", "`EV/box/demo-hp-host-night.txt`")}, NH),
("CA06", "clock-agent", 5, "Controller supervisor: restarts a dead or silent controller · CUSTOMER", "Watchdog", r"supervisor",
{3: N("W4", 15, "No time limit on one sweep in the source, and no duration was measured. Nothing grows with data."),
6: N("W2", 8, "The windows (3 restarts in 15 minutes, the 24-hour counter) use the wall clock in UTC. Nobody tested a clock jump.", "`AG/internal/localapi/controllersupervisor.go:550-552` `AG/internal/localapi/controllersupervisor.go:308`")}, NH),
("CA07", "clock-agent", 6, "Guest-power watchdog: starts a guest that must run · CUSTOMER", "Watchdog", r"guest.power|guest power",
{6: N("W2", 8, "The next try time is kept in memory from the wall clock. A clock set back delays the next try; an agent restart tries again at once.", "`AG/internal/localapi/guestpower.go:234` `AG/internal/localapi/guestpower.go:253`"),
7: N("W1", 4, "When the watchdog gives up, it writes one ERROR line in the host's journal. No event goes to the hub. The household's box is down and the outside sign is only the hub's liveness check.", "`AG/internal/localapi/guestpower.go:200-206`"),
8: N("W1", 4, "TEST for the start and the back-off. Who hears the give-up: NOBODY KNOWS.")},
"NO — no runbook says what to do after the watchdog gave up"),
("CA08", "clock-agent", 7, "Hub report and heartbeat loop · CUSTOMER", "Interval", r"heartbeat|poll interval|report loop",
{6: A("The ticker is monotonic. The report loop follows the hub's interval override, clamped to a range. The reconcile engine keeps the interval that it read at the start.", "`AG/internal/hub/loop.go:188-196` `AG/cmd/felhom-agent/main.go:969`")}, NH),
("CA09", "clock-agent", 8, "Signed jobs: the validity window against the host clock · OPERATOR", "Interval", r"signed job|signed op|not.before|valid",
{4: N("W1", 9, "A job that the agent refuses as expired is not run. The agent's log line is the only trace on the box; no path that tells the hub or the operator 'refused: expired' was found."),
6: N("W2", 9, "The window is checked against the host's wall clock. A host clock that is behind refuses every job as 'not yet valid'; one that is ahead refuses them as 'expired'. So a box with a wrong clock cannot take an agent update.", "`AG/internal/authz/verifier.go:153-162`"),
7: N("W1", 9, "The runner logs a finished or failed job on the host. For a job refused on its dates, no message to the hub was found.", "`AG/internal/signedjobs/runner.go:176-190`")},
"NO — no runbook names a wrong host clock as a cause of refused signed jobs"),
("CA10", "clock-agent", 9, "Storage watchdog and the 20-second mount re-assert · CUSTOMER", "Watchdog", r"storage watchdog|re.?assert|ReassertGuestBinds",
{3: N("W4", 15, "No time limit on one tick in the source, and no duration was measured. The work grows with the number of drives."),
6: N("W2", 8, "The back-off uses the wall clock in UTC. A clock set back stretches the back-off; a clock set forward ends it early. The tickers are monotonic."),
8: N("W4", 15, "TEST for the reconcile and the back-off. The 20-second ticker has no test: NOBODY KNOWS.")}, NH),
("CA11", "clock-agent", 10, "Weekly guest disk trim · CUSTOMER", "Interval", r"fstrim|trim",
{6: N("W2", 8, "The 'done this week' check uses host-local wall time. An attempt that was stamped while the clock was ahead keeps the trim 'not due' until real time passes it.", "`AG/internal/fstrim/fstrim.go:179-182`"),
7: A("The operator, on the hub's System page (one cell). A failed trim is a WARN line on the host. Nobody is mailed.", "`HUB/web/system.go:45` `HUB/web/system.go:206` `AG/internal/fstrim/fstrim.go:318-319`"),
8: N("W2", 20, "TEST for the week anchor. The hour change and a clock jump: NOBODY KNOWS.")}, NH),
("CA12", "clock-agent", 11, "Weekly upload line test · CUSTOMER", "Interval", r"line.?test|linetest",
{6: N("W2", 8, "The check uses host-local wall time. A success that was stamped from the future blocks the tests until that time.", "`AG/internal/linetest/linetest.go:211-213`"),
7: A("The operator, on the hub's System page; the cell is judged on the last success, not on the attempt. Nobody is mailed.", "`AR/11-os-updates.md:409-411`")}, NH),
("CA13", "clock-agent", 12, "PBS datastore verify loop · CUSTOMER", "Interval", r"verify loop|PBS verify|verify",
{5: N("W3", 13, "The verify is not behind the heavy-work gate; a comment in the source says so. It can run beside a whole-guest backup."),
7: N("W1", 5, "A snapshot that fails the verify is an ERROR line on the host and a field in the report. No hub event or mail for it was found."),
8: N("W4", 15, "TEST for the record of a failed verify. A verify that takes longer than its 30-minute cadence: NOBODY KNOWS.")}, NH),
("CA14", "clock-agent", 13, "Janitor: leftover scratch guests and stale backup locks · CUSTOMER", "Interval", r"janitor|stale lock", {}, NH),
("CA15", "clock-agent", 14, "Guest network watchdog: DHCP client heal · CUSTOMER", "Watchdog", r"guest.?net|DHCP",
{3: N("W4", 15, "No time limit on one probe in the source, and no duration was measured."),
7: A("The operator, on the hub's Hosts page: the report carries the heal count for each guest. Nobody is mailed.", "`HUB/web/hosts.go:298-397` `AG/internal/guestnet/watchdog.go:166-170`")}, NH),
("CA16", "clock-agent", 15, "Host networking self-heal · CUSTOMER", "Watchdog", r"self.?heal", {}, NH),
("CA17", "clock-agent", 16, "Management-plane watchdog timer: keeps SSH reachable · OPERATOR", "Watchdog", r"mgmt.watchdog|management.plane",
{5: A("The script changes nothing when the directory is healthy. MEASURED: the timer is active on the three hosts and fires each 60 seconds. `Persistent=true` in the unit has no effect, because the timer has no calendar line.", "`AG/configs/felhom-mgmt-watchdog.sh:13-15` `AG/configs/felhom-mgmt-watchdog.timer:11-14` `EV/box/demo-hp-host-clock.txt`"),
8: A("SOURCE only. The script has no test file.", "`AG/configs/felhom-mgmt-watchdog.sh:1-68` `AG/configs/felhom-mgmt-watchdog.timer:3-6`")}, NH),
("CA18", "clock-agent", 17, "Agent service: restart, start limit, rollback · CUSTOMER", "Watchdog", r"Restart=|start limit|rollback",
{4: A("After the start limit the unit stays failed. Nothing on the host starts it again. The outside sign is the hub's 'host stale' alarm at 45 minutes.", "`AG/configs/felhom-agent.service:41-42` `AR/11-os-updates.md:572-573`"),
6: N("W5", 17, "The design text says `Restart=always` and a systemd watchdog that kills a hung agent. The unit has `Restart=on-failure` and no watchdog.", "`AR/03-host-agent.md:72` `AG/configs/felhom-agent.service:41-42`"),
8: N("W5", 17, "MEASURED values and a TEST for the rollback. A hung agent (alive, not working): NOBODY KNOWS; no watchdog restarts it.", "`AG/configs/felhom-agent.service:41-42`")},
"PARTLY — `RB/break-glass.md` names `reset-failed`; no runbook names the start limit of the agent"),
("CA19", "clock-agent", 18, "Off-site tunnel loop and the clock-ahead re-key · CUSTOMER", "Interval", r"wgtunnel|re.?key|tunnel",
{3: N("W4", 15, "No time limit on one pass in the source, and no duration was measured. Nothing grows with data.")}, NH),
("CA20", "clock-agent", 19, "Whole-guest backup cap: rolling windows · CUSTOMER", "Night", r"backup cap|rolling window|cap",
{6: N("W2", 8, "The windows use the wall clock. Successes that were stamped while the clock was ahead stay inside the window longer and can refuse real runs. NOBODY KNOWS.")}, NH),
("CA21", "clock-agent", 21, "Unsent OS-update report sender · OPERATOR", "Interval", r"unsent",
{7: A("The operator sees the result late: at most 5 minutes after the hub is back. The sender has no alarm of its own.")}, NH),
# ---------------------------------------------------------------- hub (source only)
("CH01", "clock-hub", 0, "Box liveness: 'your server cannot be reached' · CUSTOMER", "Hub sweep", r"host_stale|liveness|staleness", {}, NH),
("CH02", "clock-hub", 1, "Missed-backup deadline check (05:00) · CUSTOMER", "Hub daily", r"deadline check|backup.deadline|05:00",
{4: N("W1", 1, "When the hub is down at 05:00, that day's check is skipped. The 'last run' value lives in memory and the next slot is the next day. Nobody notices. The next day's check uses longer limits, so a missed backup is seen one day late.", "`E/hub/cmd/hub/main.go:1069-1112`"),
6: N("W2", 7, "The check compares the BOX's backup time with the hub's clock. A box whose clock runs ahead looks fresh. No guard in the file. (R-1206 is the same cause seen on the box's own pages.)", "`HUB/monitor/deadline.go:243`"),
8: N("W1", 1, "TEST for a down box and for 'the same slot never runs twice'. A skipped day: NOBODY KNOWS.")},
"NO — no runbook names a hub that was down at 05:00"),
("CH03", "clock-hub", 2, "Mail cooldowns and the weekly outage mail · CUSTOMER", "Hub sweep", r"cooldown",
{3: A("Each mail call has a 10-second limit. The cooldown lists are in memory; an entry is removed when its alarm clears.", "`HUB/notify/dispatcher.go:86` `HUB/notify/dispatcher.go:684` `HUB/notify/dispatcher.go:1182`"),
6: N("W2", 10, "Cooldowns use the hub's wall clock only. A jump back stretches a quiet period; a jump forward ends it early. Nothing handles it."),
7: N("W1", 2, "A swallowed operator alarm leaves a 'suppressed' row. A household mail that fails is logged and is not sent again; only operator mails are retried.", "`HUB/notify/dispatcher.go:1091-1094`")}, NH),
("CH04", "clock-hub", 3, "Retry of a failed operator mail · OPERATOR", "Hub sweep", r"retry",
{4: N("W1", 3, "The retry timers are in memory. A hub restart inside the 21 minutes loses them, and nothing sends the mail later. After the third failure the hub writes one ERROR line.", "`HUB/notify/dispatcher.go:90-102`"),
6: N("W2", 10, "The timers are durations; the hour change has no effect. A jump of the hub's wall clock: NOBODY KNOWS."),
7: N("W1", 3, "Nobody is mailed when the retries give up. The trace is the ERROR line and the 'failed' rows in the notification log.", "`HUB/notify/dispatcher.go:90-102` `AR/08-alarm-ladder.md:363`"),
8: N("W1", 3, "TEST for the retry. A restart during the retry window: NOBODY KNOWS.")}, NH),
("CH05", "clock-hub", 4, "Kernel mail: 'your box restarts tonight' · CUSTOMER", "Hub sweep", r"kernel",
{8: N("W3", 14, "TESTS for 'no mail, no step'. A mail that goes out late in the day against the box's own night slot: NOBODY KNOWS.")}, NH),
("CH06", "clock-hub", 5, "OS-update approval tick and the hourly OS alarms · OPERATOR", "Hub sweep", r"approval|ring",
{6: N("W2", 7, "A box's claimed reboot time is allowed 1 minute of error. A larger error of the box's clock: NOBODY KNOWS. (The N100's host is 32 s behind today.)"),
8: N("W4", 16, "TESTS and DOC. The cost of the pass each minute with 100 boxes: NOBODY KNOWS.")}, NH),
("CH07", "clock-hub", 6, "Off-site copy too old, and escrow never finished · CUSTOMER", "Hub sweep", r"offsite_stale|off.site.{0,20}stale|48 h",
{4: N("W1", 3, "With the hub down nothing is judged. After a start the state lists in memory are empty. If a tier that is still too old is mailed again after a restart: NOBODY KNOWS."),
6: N("W2", 7, "The last-success time is the BOX's time, compared with the hub's clock. A box clock that is ahead hides an old copy. A time that cannot be read counts as too old.", "`HUB/monitor/offsite.go:203-207`"),
8: N("W2", 7, "TESTS for the limits and for a newborn box. A box clock that is ahead: NOBODY KNOWS.")}, NH),
("CH08", "clock-hub", 7, "Restore-test proof too old · OPERATOR", "Hub sweep", r"restore.test",
{6: N("W2", 7, "First contact is the hub's receive time. The proof times come from the box's reports. An error of the box's clock: NOBODY KNOWS."),
8: N("W4", 16, "TEST that a restart does not mail again. The cost with 100 boxes (14 days of reports per customer, each minute): NOBODY KNOWS.")}, NH),
("CH09", "clock-hub", 8, "Off-site clean-up window (20 minutes, weekly) · CUSTOMER", "Hub sweep", r"clean.?up window|prune window|deleting key",
{4: N("W1", 1, "If the hub stops while a window is open, the key line that can delete stays on the storage account until the hub is back; the sweep then closes it. Nobody is told during the gap."),
6: N("W2", 10, "The window runs on the hub's clock; the box's clock does not decide. A hub clock that jumps back can keep a window open longer: NOBODY KNOWS.")}, NH),
("CH10", "clock-hub", 9, "Delayed deletion of a set-aside off-site copy · CUSTOMER", "Hub sweep", r"set.aside|delayed delet|superseded",
{3: N("W4", 15, "One remote delete for each due request, in the same sweep as the window. How long a large delete takes: NOBODY KNOWS."),
6: A("The hub's clock only. The length of the wait is one setting, read at the start.", "`E/hub/cmd/hub/main.go:598-605`")}, NH),
("CH11", "clock-hub", 10, "Off-site key audit (07:10) · OPERATOR", "Hub daily", r"key audit|07:10",
{3: N("W4", 16, "The audit has a 10-minute limit and reads one customer after the other from the storage account. With 100 boxes the 10 minutes may not be enough; nobody measured it."),
4: N("W1", 1, "When the hub is down at 07:10, that day's audit is skipped. No catch-up. Nobody notices.", "`E/hub/cmd/hub/main.go:1069-1112`"),
8: N("W4", 16, "TEST and DOC. The 10-minute limit at fleet size: NOBODY KNOWS.")}, NH),
("CH12", "clock-hub", 11, "Off-site credential stuck, and its self-heal · OPERATOR", "Hub sweep", r"offsiteheal|credential.{0,30}(stuck|heal)",
{8: A("SOURCE and DOC only. No test name for the timing was found.")}, NH),
("CH13", "clock-hub", 12, "PBS whole-box credential self-heal · OPERATOR", "Hub sweep", r"pbsdrheal|PBS.{0,30}heal",
{8: N("W4", 15, "SOURCE only. A pass that hangs on the remote side: NOBODY KNOWS; the pass has no time limit.")}, NH),
("CH14", "clock-hub", 13, "WireGuard peer list push to ep0 · OPERATOR", "Hub sweep", r"wgsync|peer list",
{4: N("W1", 5, "With the hub down ep0 keeps the last list. A push that fails is logged and tried again at the next tick. Nobody is mailed when it fails again and again; a new box then cannot reach the off-site store.", "`HUB/wgsync/reconciler.go:100-102`")}, NH),
("CH15", "clock-hub", 14, "Box clock ran ahead: the tunnel alarm · OPERATOR", "Hub sweep", r"wg_tunnel|stamped in the future|clock.ahead", {}, NH),
("CH16", "clock-hub", 15, "The daily scheduler and the hour change · OPERATOR", "Hub daily", r"scheduleDaily|nextDailyRun|D24",
{6: N("W2", 10, "Days of 23 and 25 hours are handled (D24, with tests). While the scheduler sleeps, a jump of the hub's clock does not move the timer: NOBODY KNOWS what then runs when.")}, NH),
("CH17", "clock-hub", 16, "Hub database snapshot (four slots) · OPERATOR", "Hub daily", r"snapshot",
{6: N("W5", 18, "The slots are safe on the hour-change night. The design text still says 'older than 24 h' and '02:30' in its first paragraph; the source has four slots.", "`AR/05-hub-architecture.md:618-620`")},
"YES — `RB/RUNBOOK-hub-db-offsite-backup.md`"),
("CH18", "clock-hub", 17, "Daily prune of old hub rows (04:30) · OPERATOR", "Hub daily", r"prune",
{3: N("W4", 15, "No time limit and no duration in the log. The deletes grow with the reports of each box each day."),
6: N("W2", 10, "One cut-off is made from the pod's local time and compared with a UTC column; the other uses UTC. The error is 1 or 2 hours on 90 days."),
8: A("SOURCE only. No test for the cut-off's time zone was found.")}, NH),
("CH19", "clock-hub", 18, "The other checks on the 60-second sweep · OPERATOR", "Hub sweep", r"60.second|sweep",
{3: N("W4", 15, "All checkers run one after the other in one loop, with no limit for each. A slow checker delays all the others, the liveness check too. Each reads the whole fleet at each pass.", "`E/hub/cmd/hub/main.go:935-962`"),
8: N("W4", 16, "SOURCE only. How long one whole sweep takes: NOBODY KNOWS.")}, NH),
("CH20", "clock-hub", 19, "Reminder about a held app · CUSTOMER", "Hub sweep", r"held.app|held app",
{6: N("W2", 7, "The reminder counts from a time that the box reports. A box clock that is behind makes the reminder early; one that is ahead makes it late.")}, NH),
("CH21", "clock-hub", 20, "'A golden is published but not vouched' · OPERATOR", "Hub sweep", r"vouch", {}, NH),
("CH22", "clock-hub", 21, "Shared storage boxes: fill and reachability · OPERATOR", "Hub sweep", r"storage box|pool.box|oversub", {}, NH),
("CH23", "clock-hub", 25, "DooPlex: push of the hub database snapshot off-site · OPERATOR", "DooPlex and ep0", r"hub.db|database snapshot",
{6: K(1293, NM + "The timer's zone and the hub's zone must match for the 30-minute offset. On the autumn hour-change day the 02:00 copy never reaches ep0 and the 20:00 copy is pushed twice."),
7: N("W5", 18, "The runbook says in one place that the alarm fires after 13 hours, and the alarm rule printed in the same runbook says 26 hours.", "`RB/RUNBOOK-hub-db-offsite-backup.md:100` `RB/RUNBOOK-hub-db-offsite-backup.md:152-153`"),
8: A(NM + "A test file for the script exists and is run by hand. The live alarm rule is in another repository and on DooPlex; this run did not read it.", "`RB/RUNBOOK-hub-db-offsite-backup.md:107-108` `E/scripts/hub-db-backup/test_hub_db_backup.py:309-316`")},
"YES — `RB/RUNBOOK-hub-db-offsite-backup.md`"),
("CH24", "clock-hub", 26, "DooPlex: Sunday restore tests · OPERATOR", "DooPlex and ep0", r"restore.test",
{5: N("W3", 14, NM + "Sunday 04:30 is between the 02:00 snapshot push and the 05:00 PBS pull. The script takes no lock against them."),
6: N("W2", 20, NM + "On the last Sunday of October the hour 02:00 to 03:00 comes twice. What the timer does at 04:30 that day: NOBODY KNOWS."),
8: A(NM + "DOC only. This run did not read if the alarm rule is deployed.", "`RB/RUNBOOK-hub-db-offsite-backup.md:152-161`")},
"YES — `RB/RUNBOOK-hub-db-offsite-backup.md`"),
("CH25", "clock-hub", 27, "DooPlex: nightly off-site push and the release copy · OPERATOR", "DooPlex and ep0", r"release.copy|release copy",
{3: N("W3", 14, NM + "The units have limits of 90 minutes and 2 hours. A 90-minute run from 00:20 ends at 01:50, after the 01:40 start of the next job. No lock between them."),
8: A(NM + "SOURCE only. The release-copy timer is switched on by hand; this run did not read if it is on.", "`E/scripts/release-copy/install.sh:4`")}, NH),
("CH26", "clock-hub", 28, "DooPlex and ep0: PBS copy, prune, verify, clean-up · OPERATOR", "DooPlex and ep0", r"ep0|PBS.{0,20}(prune|sync|pull)",
{3: A(NM + "The PBS jobs are in PBS's own config, not in the repository. They grow with the backup size of every box.", "`RB/ep0-datastore-copy.md:19`"),
5: N("W3", 14, NM + "The order is by the clock only: prune 07:30, clean-up 08:00. A pull that runs past 07:30 meets the prune: NOBODY KNOWS what then happens."),
6: N("W3", 14, NM + "Two machines, two clocks. 'The ep0 prune at 03:30 is before the 05:00 pull' is true only while both clocks and both zones agree. No document gives their zones."),
7: N("W1", 1, NM + "PBS has its own notification target. The clean-up unit has no failure hook and no time limit; a clean-up that fails tells nobody.", "`E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.service:9` `RB/ep0-datastore-copy.md:19`")},
"YES — `RB/ep0-datastore-copy.md`"),
("CH27", "clock-hub", 29, "Nightly CI: the gates and the security watch · OPERATOR", "DooPlex and ep0", r"nightly|security.watch|cron",
{4: N("W1", 1, NM + "A missed night is not made up. The watch's weekly summary repairs itself: it is sent when the last one is 8 or more days old. A runner that drops the job tells nobody."),
8: N("W1", 1, NM + "DOC. A night with no run at all: NOBODY KNOWS who notices.", "`AR/08-alarm-ladder.md:708-722`")}, NH),
]
M = {i: dict(src=(h, n), name=name, seed=seed, named=("grep", rx), o=o, rb=rb) for i, h, n, name, seed, rx, o, rb in _M}
# ------------------------------------------------------------------ second pass
# Helper answers that still said "not found" or "candidate" after the first build: the main session's words in their place.
M["CC20"]["o"][2] = A("The sweep runs each 6 hours; the release each hour. A restore copy that nobody used goes after 30 days (the constant, D62). A comment in the wiring still says 14 days; the comment is stale.", "`CT/cmd/controller/main.go:1412`")
M["CH01"]["o"][3] = N("W4", 16, "No time limit. The checkers run one after the other in one loop: one read of all hosts at each pass, then a loop over the hosts. The time grows with the number of boxes. Nobody measured one pass.", "`E/hub/cmd/hub/main.go:935-962` `HUB/monitor/host_staleness.go:113`")
M["CH01"]["o"][4] = A("With the hub down nothing is judged. At the start, a box that is already silent gets one alarm at the first pass, unless that alarm was raised before. The controller's checker starts silent for such a box.", "`HUB/monitor/host_staleness.go:59-68` `HUB/monitor/staleness.go:86-91`")
M["CH02"]["o"][3] = N("W4", 16, "No time limit; the check runs inside the scheduler's own loop. It reads up to 7 days of evidence for every customer. The time grows with boxes times reports. Nobody measured it.", "`E/hub/cmd/hub/main.go:1090`")
M["CH08"]["o"][3] = N("W4", 16, "No time limit. Each pass reads 14 days of host reports for each customer, each minute. It is the heaviest read on the sweep. Nobody measured it.", "`HUB/monitor/restoretest.go:79` `HUB/monitor/restoretest.go:252`")
M["CH16"]["o"][2] = A("Budapest wall-clock time. If the zone cannot be loaded, the fallback is fixed CET, which is one hour off in summer. The pod's zone is set to Europe/Budapest.", "`E/hub/cmd/hub/main.go:1072`")
M["CH16"]["o"][4] = N("W1", 1, "A slot that is missed while the hub is down is not made up; the snapshot and the prune are the two exceptions. A time setting that cannot be read becomes 00:00 with no message.", "`E/hub/cmd/hub/main.go:1069-1112`")
M["CH23"]["o"][2] = A(NM + "02:30, 08:30, 14:30 and 20:30 with up to 2 minutes of random delay, in the host's local time (the unit names no zone).")
NIGHT = "`EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt`"
MEASURED = {
"CC01": ("MEASURED 2026-11-11, one night, small stores: 2 min 33 s · 40 s · 18 s (demo-hp · Tester 1 · N100)", NIGHT),
"CC02": ("MEASURED 2026-11-11: 39 s · 15 s · 11 s", NIGHT),
"CC03": ("MEASURED 2026-11-11: 3 min 09 s · 59 s · 38 s", NIGHT),
"CC08": ("MEASURED 2026-11-11: under 1 s on each box", NIGHT),
"CC09": ("MEASURED 2026-11-11: 5 s · 5 s · 3 s", NIGHT),
"CC19": ("MEASURED 2026-11-11: the slot ran on each box", NIGHT),
"CA04": ("MEASURED 2026-11-11: the hourly timer is active on the three hosts", "`EV/box/demo-hp-host-clock.txt` `EV/box/tester-1-host-clock.txt` `EV/box/demo-felhom-host-clock.txt`"),
}
TINY = [
"session clean-up of dashboard sign-ins on the box (a new sign-in repairs it)",
"old image clean-up after a controller start (log only; deletes nothing when it cannot read its keep list)",
"operator restart and reboot limits on the agent (one hour; an operator action)",
"the hub's purges of log bundles, access log and sessions (nobody is told, nothing depends on them)",
"the hub's version check and catalog template fetch (a WARN line and a stored error; a page shows it)",
"the guest swap check, once after an agent start (2-minute limit)",
"the 14-day hold of alarms after a deliberate power-off (a window, not a job)",
]
LEFT_OUT = [
("The agent's local-API certificate (10 years)", "it is an end date: the End-dates list, DB03"),
("Links and tokens that end by themselves", "they are end dates: the End-dates list, DH22 and the by-ruling list there"),
("The 14-day countdown after a household stops the off-site copy", "the End-dates list, DB04; its daily sweep is CC19 here"),
("Debian's and Proxmox's own timers on a host (apt-daily, pve-daily-update, logrotate, fstrim.timer, man-db)", "they are the OS's jobs, not ours; MEASURED present on the three hosts, `evidence/box/*-host-clock.txt`"),
("TEST-ONLY jobs", "none was found: every job in the table runs on a household's box or for the operator"),
]
@@ -0,0 +1,155 @@
#!/usr/bin/env python3
"""resolve-helper.py HELPER.md [--json OUT] — open every pointer in a helper's answer BEFORE it may enter the table.
A helper (a read-only subagent) writes pointers in three shapes: a full one (`CT/internal/stacks/delete.go:624`), a
bare file name (`delete.go:756`) and a short one (`:1025`, "same file as the last one"). This tool
1. resolves each to a full `PREFIX/path:line[-line]`,
2. says DEAD when the file or the line does not exist, UNRESOLVED when a bare name fits no file or more than one,
3. checks the NAME the helper put in front of the pointer (a backticked identifier within 90 characters, with no other
pointer between): it must stand within 3 lines of the cited range. IDENT-OK / IDENT-MISS / NO-IDENT.
It does not judge the claim. A claim that carries a mark in TABLE.md is read by the main session (check-pointers.py
prints the lines). Read-only: it opens files in the git clones on DooPlex and prints.
"""
import json, os, re, sys
HERE = os.path.dirname(os.path.abspath(__file__))
REPO = os.path.abspath(os.path.join(HERE, "..", "..", "..", ".."))
WS = os.path.dirname(REPO)
PREFIX = {"AG": os.path.join(WS, "felhom-agent"), "CT": os.path.join(WS, "felhom-controller", "controller"),
"CAT": os.path.join(WS, "app-catalog-felhom.eu"), "HUB": os.path.join(REPO, "hub", "internal"), "E": REPO,
"AR": os.path.join(REPO, "documentation", "architecture"), "RB": os.path.join(REPO, "documentation", "runbooks")}
RANGES = r"~?:?(\d+(?:-\d+)?(?:\s?,\s?:?\d+(?:-\d+)?)*)"
PTR = re.compile(r"(?P<full>(?:%s)/[\w./@+-]+\.\w+)(?::(?P<fr>\d+(?:-\d+)?(?:,\s?:?\d+(?:-\d+)?)*))?"
r"|(?<![\w/.-])(?P<bare>[\w.-]+\.(?:go|sh|py|md|yml|yaml|html|json|service|timer))(?::(?P<br>\d+(?:-\d+)?(?:,\s?:?\d+(?:-\d+)?)*))"
r"|(?<![\w`/])~?:(?P<sr>\d+(?:-\d+)?)(?![\d:])" % "|".join(PREFIX))
IDENT = re.compile(r"`([^`]{2,80})`")
_cache, _walk = {}, {}
def lines_of(path):
if path not in _cache:
try:
_cache[path] = open(path, encoding="utf-8", errors="replace").read().split("\n")
except OSError:
_cache[path] = None
return _cache[path]
def find_base(pre, base):
if pre not in _walk:
idx = {}
for d, dn, fn in os.walk(PREFIX[pre]):
dn[:] = [x for x in dn if x not in (".git", "node_modules", "vendor", "evidence")]
for f in fn:
idx.setdefault(f, []).append(os.path.relpath(os.path.join(d, f), PREFIX[pre]))
_walk[pre] = idx
return _walk[pre].get(base, [])
def check(full, rng, text, pos):
pre, rest = full.split("/", 1)
ls = lines_of(os.path.join(PREFIX[pre], rest))
if ls is None:
return "DEAD", "NO-IDENT"
if not rng:
return "OK", "NO-IDENT"
a, _, b = rng.partition("-")
a, b = int(a), int(b or a)
if b > len(ls):
return "DEAD", "NO-IDENT"
back = text[max(0, pos - 90):pos]
cut = max(back.rfind(":"), back.rfind(";"), back.rfind(",")) # a pointer or a clause break in between: do not reach over it
ids = [m for m in IDENT.finditer(back) if m.start() > cut - 1 or cut < 0]
ids = ids or list(IDENT.finditer(back))[-1:]
if not ids:
return "OK", "NO-IDENT"
name = re.sub(r"[()…]|\.\.\.", "", ids[-1].group(1)).strip().split(" ")[0].split("(")[0]
name = name.split(".")[-1] if re.fullmatch(r"[\w.]+", name) else name
win = "\n".join(ls[max(0, a - 4):b + 3])
return "OK", ("IDENT-OK" if name and name in win else f"IDENT-MISS({name})")
def resolve(text, known, default_pre):
"""Returns (text with every pointer rewritten as a backticked full token, [records])."""
out, recs, last, i = [], [], None, 0
for m in PTR.finditer(text):
out.append(text[i:m.start()])
i = m.end()
if m.group("full"):
full, rngs = m.group("full"), m.group("fr")
elif m.group("bare"):
base, rngs = m.group("bare"), m.group("br")
full = known.get(base)
if not full:
c = find_base(default_pre, base)
full = f"{default_pre}/{c[0]}" if len(c) == 1 else None
if not full:
recs.append({"ptr": m.group(0), "file": "UNRESOLVED", "ident": "NO-IDENT"})
out.append(m.group(0))
continue
else:
full, rngs = last, m.group("sr")
if not full:
out.append(m.group(0))
continue
last = full
known.setdefault(os.path.basename(full), full)
toks = []
for r in re.split(r"\s?,\s?:?", rngs) if rngs else [None]:
st, idn = check(full, r, text, m.start())
tok = f"{full}:{r}" if r else full
recs.append({"ptr": tok, "file": st, "ident": idn})
toks.append(f"`{tok}`")
out.append(" ".join(toks))
out.append(text[i:])
return "".join(out), recs
def main():
src = sys.argv[1]
text = open(src, encoding="utf-8").read()
default_pre = "CT" if "controller" in src else "AG" if "agent" in src else "HUB" if "hub" in src else "AR"
known = {}
for m in re.finditer(r"((?:%s)/[\w./@+-]+\.\w+)" % "|".join(PREFIX), text): # first pass: every full path the helper named
known.setdefault(os.path.basename(m.group(1)), m.group(1))
endings, cur, allrecs, lastq = [], None, [], None
for line in text.split("\n"):
h = re.match(r"^### (.+)$", line)
if h:
cur = {"title": h.group(1).strip(), "answers": {}, "extra": []}
endings.append(cur)
lastq = None
continue
if cur is None:
continue
a = re.match(r"^([1-8])\. (.*)$", line)
body = a.group(2) if a else line
new, recs = resolve(re.sub(r"`((?:%s)/[^`]+)`" % "|".join(PREFIX), r"\1", body), known, default_pre)
for r in recs:
r["ending"] = cur["title"]
r["q"] = a.group(1) if a else (lastq or "-")
allrecs += recs
if a:
cur["answers"][a.group(1)] = new
lastq = a.group(1)
elif line.strip() and lastq and not line.startswith(("#", "---")):
cur["answers"][lastq] += " · " + new.strip().lstrip("-* ") # a continuation line belongs to the answer above it
elif line.strip():
cur["extra"].append(new)
n = len(allrecs)
dead = [r for r in allrecs if r["file"] != "OK"]
miss = [r for r in allrecs if r["ident"].startswith("IDENT-MISS")]
ok = sum(1 for r in allrecs if r["ident"] == "IDENT-OK")
print(f"{os.path.basename(src)}: endings {len(endings)} · pointers {n} · dead/unresolved {len(dead)} · "
f"name found at the line {ok} · name NOT at the line {len(miss)} · no name to check {n - ok - len(miss) - len(dead)}")
for r in dead:
print(f" {r['file']:<10} {r['ptr']} [{r['ending'][:50]} q{r['q']}]")
for r in miss:
print(f" {r['ident']:<28} {r['ptr']} [{r['ending'][:50]} q{r['q']}]")
if "--json" in sys.argv:
json.dump({"endings": endings, "pointers": allrecs}, open(sys.argv[sys.argv.index("--json") + 1], "w"), indent=1)
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,26 @@
#!/bin/bash
# run-box.sh KIND NAME SSHTARGET — run tools/host-KIND.sh on the host and tools/guest-KIND.sh in guest 9201, read-only.
# Output: box/NAME-host-KIND.txt and box/NAME-guest-KIND.txt beside this folder.
set -u
D=$(cd "$(dirname "$0")/.." && pwd)
k=$1; n=$2; t=$3
SSH="ssh -o BatchMode=yes -o ConnectTimeout=10"
{
printf '# host-%s %s taken %s from DooPlex\n' "$k" "$n" "$(date -u +%FT%TZ)"
timeout 240 $SSH "$t" 'bash -s' < "$D/tools/host-$k.sh"
printf 'RC=%s\n' "$?"
} > "$D/box/$n-host-$k.txt" 2>&1
{
printf '# guest-%s %s taken %s from DooPlex\n' "$k" "$n" "$(date -u +%FT%TZ)"
timeout 240 $SSH "$t" 'pct exec 9201 -- bash -s' < "$D/tools/guest-$k.sh"
printf 'RC=%s\n' "$?"
} > "$D/box/$n-guest-$k.txt" 2>&1
if [ "$k" = clock ]; then
{
printf '# offset %s: box clock minus DooPlex clock (midpoint of the call), seconds\n' "$n"
for i in 1 2 3 4 5; do
a=$(date +%s.%N); b=$($SSH "$t" 'date +%s.%N'); c=$(date +%s.%N)
awk -v a="$a" -v b="$b" -v c="$c" 'BEGIN{printf "offset=%.3f rtt=%.3f\n", b-(a+c)/2, c-a}'
done
} > "$D/box/$n-offset.txt" 2>&1
fi
@@ -0,0 +1,7 @@
#!/bin/bash
set -u
D=$(cd "$(dirname "$0")/.." && pwd); n=$1; t=$2
SSH="ssh -o BatchMode=yes -o ConnectTimeout=10"
{ printf '# host-night %s taken %s\n' "$n" "$(date -u +%FT%TZ)"; timeout 200 $SSH "$t" 'bash -s' < "$D/tools/host-night.sh"; printf 'RC=%s\n' "$?"; } > "$D/box/$n-host-night.txt" 2>&1
{ printf '# guest-night %s taken %s\n' "$n" "$(date -u +%FT%TZ)"; timeout 200 $SSH "$t" 'pct exec 9201 -- bash -s' < "$D/tools/guest-night.sh"; printf 'RC=%s\n' "$?"; } > "$D/box/$n-guest-night.txt" 2>&1
tail -1 "$D/box/$n-host-night.txt" "$D/box/$n-guest-night.txt"
@@ -0,0 +1,43 @@
#!/bin/bash
# searches.sh — the searches an `N` in TABLE.md must survive. Read-only; prints to stdout (saved as evidence/searches.txt).
# The same tool as audits/end-dates-list-2026-11-11/tools/searches.sh with the Clock list's searches.
# Each search: an ASCII fragment (no accented letter), case-insensitive, over (ar) documentation/architecture,
# (op) backlog/OPEN-ITEMS.md, (cl) backlog/CLOSED-ITEMS.md, (rm) backlog/ROADMAP.md. For register files it prints the row ids
# that match, so a reader can open them. Controls: C+ must hit, C- must not.
E="$(cd "$(dirname "$0")/../../../.." && pwd)"
B="$E/documentation/backlog"; AR="$E/documentation/architecture"
G=/usr/bin/grep
ids() { $G -i -E -- "$1" "$2" 2>/dev/null | $G -o -E '^\| \*\*R-[0-9a-z]+\*\*' | tr -d '|* ' | sort -u | tr '\n' ' '; }
s() { # id, what, pattern
local n_ar; n_ar=$($G -r -i -l -E -- "$3" "$AR" 2>/dev/null | wc -l)
printf '%s | %s | /%s/ | ar files=%s | open: %s| closed: %s| roadmap=%s\n' "$1" "$2" "$3" "$n_ar" "$(ids "$3" "$B/OPEN-ITEMS.md")" "$(ids "$3" "$B/CLOSED-ITEMS.md")" "$($G -c -i -E -- "$3" "$B/ROADMAP.md")"
}
date -u +'searches run %Y-%m-%d %H:%M:%S UTC'
s C+ "positive control (a row that exists)" "R-1301"
s C+2 "positive control (a word in all four)" "backup"
s C- "negative control" "zzqxclocklistzz"
s S01 "a daily hub job missed while the hub is down" "(missed|skipped).{0,60}(slot|check|audit).{0,80}(pod|restart|hub (is|was) down)|catch.?up.{0,80}(hub job|scheduleDaily|daily job)"
s S02 "a failed household mail is not sent again" "(customer|household).{0,30}(mail|e-mail).{0,80}(retry|retried|not sent again)"
s S03 "the operator mail retry is lost at a hub restart" "retry.{0,100}(restart|in memory|lost)|GAVE UP"
s S04 "the guest-power watchdog gives up and tells nobody" "guest.power.{0,160}(give|gave|giving).up|GIVING UP"
s S05 "the peer list push to ep0 fails again and again" "wgsync.{0,120}(fail|alarm|event)|peer list.{0,80}(fail|stale|alarm)"
s S06 "the app-update leg did not run: who is told" "update.leg.{0,120}(did not run|missed|skipped|lost).{0,100}(told|mail|alarm|event|nobody)"
s S07 "a verdict trusts the box's clock (fresh backup, off-site age, proof age)" "(box|host).{0,12}clock.{0,120}(ahead|future|wrong).{0,160}(fresh|stale|latest|backup|proof)|timestamp.{0,60}(in the future|ahead)"
s S08 "an 'already ran' stamp written by a clock that was ahead" "(stamp|already ran|last run|LastOKAt).{0,100}(clock|future|ahead)"
s S09 "a wrong clock refuses every signed job" "(signed|job).{0,80}(not.yet.valid|expired).{0,120}clock|clock.{0,120}(signed job|issued_at|ErrNotYetValid)"
s S10 "the hub's own clock jumps" "hub.{0,20}clock.{0,80}(jump|step|wrong|drift)"
s S11 "the off-site push is skipped because another backup runs" "off-?site.{0,100}skipped.{0,100}(another|running)|another backup is running"
s S12 "the 05:30 proof restore can fall inside the whole-guest backup" "proof.{0,120}(05:30|gate|whole-guest|vzdump|meets)"
s S13 "the PBS verify runs beside a backup" "verify.{0,80}(heavy-op|gate|beside|during a backup)"
s S14 "two DooPlex timers overlap, or two machines' clocks order the jobs" "(timer|slot).{0,80}(overlap|meet|same time)|holds only (if|while) both"
s S15 "an interval job with no time limit" "(time limit|timeout|deadline).{0,80}(sweep|interval job|health check|scan)|slow checker|no per-check"
s S16 "how long the hub's sweep takes with many boxes" "(100|hundred|1000) (boxes|customers|households)|sweep.{0,80}(duration|takes|seconds).{0,60}(boxes|fleet)"
s S17 "the agent unit: Restart=always and a watchdog" "Restart=always|WatchdogSec|systemd watchdog"
s S18 "hub database staleness alarm: 13 h or 26 h" "HubDBBackupStale"
s S19 "a list of every timer" "every timer|list of (all )?(timers|scheduled jobs)|all (timers|scheduled jobs)|job (table|inventory)|timer inventory"
s S20 "the hour change" "DST|clock change|daylight|spring forward|2027-03-28"
s S21 "who keeps the time on a box" "chrony|timesyncd|NTP"
s S22 "the crash guard reads the clock at early boot" "crash.guard.{0,160}(clock|RTC|time is wrong)"
s S23 "the catalog sync fails for a long time" "(catalog|template).sync.{0,100}(fail|stale|days|alarm)"
s S24 "the 05:30 proof restore is not made up" "proof.{0,120}(not made up|catch.?up|missed)"
s S25 "the restore-copy sweep on a box that is on for under six hours" "(sweep|every 6 h).{0,120}(first run|on for under|short uptime)"
@@ -0,0 +1,41 @@
# The Clock list — the table
Built 2026-11-11 from the source of the four repositories and from read-only measurements on the three test boxes
(Tester 1, demo-hp, the N100; host and guest). One block for each job that a clock starts: the night steps on a box,
the interval jobs, the watchdogs, the hub's daily jobs and its 60-second sweep, and the timers on DooPlex and ep0.
`tools/make-table.py` writes this file from the helpers' answers (`evidence/helpers/`) and from `tools/marks.py`; do
not edit it by hand.
**NOT MEASURED, by the operator's ruling:** DooPlex and ep0, and the hub's database. What this table says about the
hub's jobs and about the timers on those two machines comes from the repository's own source, scripts and documents.
**The eight questions:** 1 what runs, who owns it, what starts it (with `CUSTOMER`, `TEST-ONLY` or `OPERATOR`) ·
2 when, how often, in what order · 3 how long it takes (measured where a box shows it), its time limit, and 10 or 100
times the data · 4 what happens if it does not run · 5 it runs late, or twice, or meets another job (the lock is
named) · 6 the clock is wrong or jumps · 7 who is told, and when · 8 how do we know (`MEASURED`, a test, `SOURCE`, a
document, or nobody knows). A ninth line, outside the marks: is the hand step written in a runbook?
**Every measured duration comes from a small test store** (three boxes, a few apps, one night of controller logs).
No number in this table says how long a job takes on a full household disk.
**Marks:** `A` answered, with a pointer · `D` decided, not built · `K` known gap, with its row · `N` nobody asked — it
survived the searches in `EV/searches.txt` (S01…S25; controls in the same file). Every `N` names its class row:
**W1** = R-1383 a job does not run, gives up, or loses its state at a restart, and nobody is told · **W2** = R-1384 a
verdict or an "already ran" check trusts a clock that can be wrong · **W3** = R-1385 two jobs can meet with no lock and
no order · **W4** = R-1386 no time limit and no measured duration · **W5** = R-1387 a document and the source disagree
on a time.
**Who read what.** A cell whose mark is `K` or `N`, and every `A` that `tools/marks.py` carries, holds the main
session's words. Where its pointer was opened by eye, the pointer is written out in `tools/marks.py`; where
`tools/marks.py` shows `+`, the pointer is the helper's and `tools/resolve-helper.py` opened it (the file and the line
exist, and the name the helper gave stands at that line). Every other cell holds a helper's words with the mark `A`.
The main session read every helper answer to questions 4 to 7; it did not open each of their pointers. Treat those
cells as a map, not as proof.
**Pointers:** `AG/` = `felhom-agent/` · `CT/` = `felhom-controller/controller/` · `CAT/` = `app-catalog-felhom.eu/` ·
`HUB/` = `felhom.eu/hub/internal/` · `E/` = `felhom.eu/` · `AR/` = `felhom.eu/documentation/architecture/` ·
`RB/` = `felhom.eu/documentation/runbooks/` · `EV/` = this folder's `evidence/`. `tools/check-pointers.py` prints the
text behind each one.
**Named status** (does an architecture document name this job?): the status comes from a search of the architecture
folder: 0 lines = `UNNAMED`, else `MENTIONED`. It is a machine count, not a reading.
@@ -0,0 +1,82 @@
# The End-dates list — the matrix
Made by `tools/make-matrix.py` from `TABLE.md`. Do not edit it by hand.
`A` answered · `D` decided, not built · `K` known gap (has a row) · `N` nobody asked. Columns are the eight
questions: 1 what, owner, where · 2 when it ends · 3 who renews, how early · 4 what breaks · 5 who is warned before · 6 a box off past the date · 7 if renewal fails · 8 how do we know.
| Thing | Named | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | N |
|---|---|---|---|---|---|---|---|---|---|---|
| DB01 Web certificate of the household's address | unnamed | A | A | A | A | **N** | **N** | **N** | A | 3 |
| DB02 Proxmox's own certificate, pinned by the agent | unnamed | A | A | **N** | **N** | **N** | **N** | **N** | A | 5 |
| DB03 The agent's local-API certificate, pinned in the guest | named | A | A | A | A | **N** | A | A | A | 1 |
| DB04 Off-site copy after the household stops it: the 14-day countdown | mentioned | A | A | A | A | A | **N** | A | A | 1 |
| DB05 Off-site history: how far back a household can go | mentioned | A | A | A | A | A | A | K | A | 0 |
| DB06 Off-site credentials: SSH key, host key, store password, PBS pin | mentioned | A | A | A | A | A | A | A | A | 0 |
| DB08 Claim code (72 hours) and the one-use gate token | mentioned | A | A | A | A | A | A | A | A | 0 |
| DB09 Share link | mentioned | A | A | A | A | A | A | A | A | 0 |
| DB11 Signed operator jobs: the validity window | named | A | A | A | A | A | A | A | A | 0 |
| DB12 Operator signing keys on each box | named | A | A | D | A | A | A | K | A | 0 |
| DB13 The agent's Proxmox access token | mentioned | A | A | A | A | A | A | A | A | 0 |
| DB16 Package signing keys: Docker in the guest, Proxmox and Debian on the host | mentioned | A | A | A | A | **N** | **N** | A | A | 2 |
| DB17 Debian 13 and Proxmox 9: the series the product is tied to | mentioned | A | **N** | **N** | A | **N** | A | A | A | 3 |
| DB18 Catalog app images and their recorded digests | mentioned | A | A | A | A | **N** | A | A | A | 1 |
| DB19 Images the controller itself starts (Traefik, tunnel client, file browser, Samba) | mentioned | A | **N** | A | A | **N** | **N** | A | A | 3 |
| DB20 Published agent versions: the newest five stay | named | A | A | A | A | A | A | A | A | 0 |
| DB21 The golden image's age (7 days) and the waiver (14 days) | named | A | A | A | A | A | A | K | A | 0 |
| DB24 Copies kept before a restore: deleted after 30 unused days | mentioned | A | A | A | A | **N** | **N** | A | A | 2 |
| DH01 Connect link in the household's e-mail (7 days) | mentioned | A | A | A | A | A | A | A | A | 0 |
| DH05 'Switched off on purpose' hold (14 days) | mentioned | A | A | A | A | A | A | A | A | 0 |
| DH08 Storage Box sub-account password, and the hub's seal key | mentioned | A | A | D | A | A | A | A | A | 0 |
| DH09 PBS off-site token on ep0 | named | A | A | A | A | A | A | A | A | 0 |
| DH10 WireGuard keys of the box's tunnel to the off-site endpoint | named | A | A | A | A | A | A | A | A | 0 |
| DH14 Recovery mode for a host (30 minutes to 4 hours) | mentioned | A | A | A | A | A | A | A | A | 0 |
| DH15 Break-glass password of a host | mentioned | A | A | A | A | A | A | A | A | 0 |
| DH19 The installer ISO | mentioned | A | A | **N** | A | A | A | A | A | 1 |
| DH20 The certificate that the hub's address serves | mentioned | A | A | A | A | **N** | A | A | A | 1 |
| DH22 Operator access tokens, among them CC's hub token | named | A | A | A | A | **N** | A | A | A | 1 |
| DH23 A household's own Cloudflare token | mentioned | A | **N** | A | A | **N** | A | A | A | 2 |
| DH24 The mail service key (all household and operator mail) | mentioned | A | **N** | A | A | **N** | A | A | A | 2 |
| DH25 The hub's global bearer key | named | A | A | A | A | A | A | A | A | 0 |
| DH26 Pinned tool versions in builds and CI | mentioned | A | A | A | A | A | A | A | A | 0 |
| DH27 Domains and registrars | mentioned | A | **N** | A | A | **N** | A | A | A | 2 |
| DH28 Cloudflare account, the site publish token, the ISO store keys | mentioned | A | **N** | A | A | **N** | A | A | A | 2 |
| DH29 Hetzner account: the off-site endpoint and the storage boxes | mentioned | A | **N** | A | A | **N** | A | A | A | 2 |
| DH30 Tailscale node keys (the operator's remote way in) | mentioned | A | A | A | A | **N** | A | A | A | 1 |
| DH31 Gitea and registry credentials | mentioned | A | **N** | A | A | **N** | A | A | A | 2 |
| DH32 CC's secret store: the tokens inside | mentioned | A | **N** | A | A | **N** | A | A | A | 2 |
| DH33 SSH keys between the hub, DooPlex and the endpoint | mentioned | A | A | A | A | A | A | A | A | 0 |
## Totals
- Things: **39** in the table · named **9** · mentioned only **28** · named by no architecture document **2**.
- Answers: **312** · A **268** · D **2** · K **3** · N **39**.
- Things with no N: **19** of 39.
| Question | A | D | K | N |
|---|---|---|---|---|
| 1 what, owner, where | 39 | 0 | 0 | 0 |
| 2 when it ends | 30 | 0 | 0 | 9 |
| 3 who renews, how early | 34 | 2 | 0 | 3 |
| 4 what breaks | 38 | 0 | 0 | 1 |
| 5 who is warned before | 21 | 0 | 0 | 18 |
| 6 a box off past the date | 33 | 0 | 0 | 6 |
| 7 if renewal fails | 34 | 0 | 3 | 2 |
| 8 how do we know | 39 | 0 | 0 | 0 |
## The N answers, by class row
| Class | Row | Count | Members (id-question) |
|---|---|---|---|
| T1 | R-1379 | 18 | DB01-5, DB01-7, DB03-5, DB16-5, DB17-5, DB18-5, DB19-5, DB24-5, DH20-5, DH22-5, DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH30-5, DH31-5, DH32-5 |
| T2 | R-1380 | 5 | DB02-3, DB02-4, DB02-5, DB02-6, DB02-7 |
| T3 | R-1381 | 11 | DB17-2, DB17-3, DB19-2, DH19-3, DH23-2, DH24-2, DH27-2, DH28-2, DH29-2, DH31-2, DH32-2 |
| T4 | R-1382 | 5 | DB01-6, DB04-6, DB16-6, DB19-6, DB24-6 |
## The K answers, by existing row
| Row | Count | Members |
|---|---|---|
| R-468 | 1 | DB21-7 |
| R-924 | 1 | DB12-7 |
| R-1294 | 1 | DB05-7 |
@@ -0,0 +1,97 @@
# The End-dates list — 2026-11-11
The fourth list of the gap review. One block for each thing that stops working, or stops being safe, on a date or after
an age, with eight questions each. Built from the source of four repositories, from read-only measurements on the three
test boxes, and from what any visitor can read with no login. **No document lists every end date** — the claim in the
brief is true (search S18, 0 hits). DooPlex, ep0 and every account are `NOT MEASURED`, by ruling.
Files: `TABLE.md` (the answers), `MATRIX.md` (the marks), `evidence/`, `tools/`.
## Five lines
1. **Things found: 58** helper blocks. 39 are in the table. The others are tiny (8), or the same thing seen from the
other side, or not an end date (all are named in the table's head).
2. **Things no architecture document names: 2** of 39 by the search; 28 more are only mentioned. No document gives a
date for any of them except one Tailscale key.
3. **Marks (312 answers):** `A` 268 · `D` 2 · `K` 3 · `N` 39.
4. **The worst `N`:** each box's agent trusts one exact Proxmox certificate. Proxmox makes a new one by itself about 14
days before the old one ends. The agent does not follow. From that day the agent cannot start, stop, back up or
restore a guest. First box: the N100, about **2028-07-02**; then demo-hp and Tester 1 within three months.
Every box installed in the same season fails in the same fortnight (DB02).
5. **The operator decides** the two questions at the end of this page, and does the two "do" items.
## Ends in the next 90 days (to 2027-02-09), soonest first
| Ends | What | Who renews | Renewal proven? |
|---|---|---|---|
| **2026-11-10 (past)** | The golden image's 7-day age limit | You: bake and vouch (Sunday) | No. The gate fails today; code pushes to `felhom.eu` are refused |
| each 7 days | Debian's package lists (`Valid-Until`) | Debian's servers, at each fetch | Yes, measured rolling on three boxes |
| 2026-12-23 | Certificate of the download address (`files.felhom.eu`) | DooPlex, by itself | Not measured |
| 2026-12-27 | Certificate of the ISO address (`iso.felhom.eu`) | Cloudflare, by itself | Not measured |
| 2026-12-31 | Web certificate of the N100's household address | The box (Traefik), by itself | On demo-hp yes; on this box not yet |
| 2027-01-02 | Web certificate of Tester 1's household address | The box, by itself | As above |
| 2027-01-02 | Cloudflare's edge certificate for the N100's name | Cloudflare, by itself | Not measured |
| 2027-01-03 | Certificate of the Gitea address | DooPlex, by itself | Not measured |
| 2027-01-12 | Certificate of the hub's address | DooPlex, by itself | One renewal seen from outside (issued 2026-10-14) |
| 2027-01-17 | Tailscale key of demo-hp (date from the document) | **You**, in the Tailscale console | No. Tailscale is not running on demo-hp today |
| 2027-01-19 | Web certificate of demo-hp's household address | The box, by itself | Yes: issued again 2026-10-21 |
| 2027-01-19, -20 | Cloudflare's edge certificates for the two other household names | Cloudflare, by itself | Not measured |
| 2027-01-28 | Certificate of the website | Cloudflare, by itself | Not measured |
| not known | CC's hub token (90 days from its making) | **You**: make a new one, put it in the store | No. The date is on the hub's token page only |
Not in this table: two Debian keys that end 2027-02-03 are in the "removed keys" file; nothing uses them.
## Only you can read these dates
| What | Where you read the date |
|---|---|
| The domains: `felhom.eu`, `demo-felhom.eu`, `enkisfelhom.hu`, `enkicsifelhom.hu`, `dooplex.hu` | Each registrar's account page (this run could not read the public records) |
| Cloudflare: the account's plan, the site publish token, the ISO store keys | Cloudflare dashboard: the API tokens page, and the R2 tokens page |
| Hetzner: the payment state, the API token, the storage boxes | Hetzner console: the security page (tokens) and the invoices |
| The mail service key and plan | The mail service dashboard: the API keys page |
| Tailscale keys of the nodes | Tailscale console: the Machines page |
| CC's hub token, and every other operator token | The hub: the access tokens page |
| The GitHub advisory token of the nightly security watch | GitHub: Settings, Developer settings, tokens |
| Gitea tokens (the scoped token, CC's token) | Gitea: Settings, Applications |
| A tester household's own Cloudflare token | That household's Cloudflare dashboard |
| The day security fixes stop for Debian 13 and for Proxmox 9 | The public release pages of Debian and Proxmox |
## The `N` answers, by class (one register row each)
| Class | Row | What | Members |
|---|---|---|---|
| T1 | R-1379 | An end exists and nobody is warned before it | web certificate (DB01-5, -7) · hub address certificate (DH20-5) · operator tokens (DH22-5) · Tailscale key (DH30-5) · package keys (DB16-5) · restore copy (DB24-5) · OS support (DB17-5) · app images (DB18-5, DB19-5) · mail key, domains, accounts, tokens (DH23/24/27/28/29/31/32-5) · local-API certificate (DB03-5) |
| T2 | R-1380 | Another party renews and our pin does not follow | Proxmox's certificate and the agent's pin (DB02-3 to -7) |
| T3 | R-1381 | No document says when it ends, or where to read the date | OS support (DB17-2, -3) · tunnel client (DB19-2) · installer ISO (DH19-3) · the outside accounts and tokens (DH23/24/27/28/29/31/32-2) |
| T4 | R-1382 | A box that was off past the date: nobody knows if it comes back | web certificate (DB01-6) · package keys (DB16-6) · old tunnel client (DB19-6) · restore copy (DB24-6) · the 14-day countdown (DB04-6) |
## No end date, or a fixed window, by ruling (each is an `A`)
The share link (no end) · operator signing keys (no end; a change is designed, DB12) · the agent's Proxmox token (no
end, measured) · the local-API certificate (10 years) · WireGuard keys (no end) · the break-glass password (paper: each
year) · the off-site history window (7 daily, 4 weekly, monthly) · the 14-day countdown after a household stops the
off-site copy · restore copies, 30 days (D62) · the newest five agent versions (D50) · the golden's 7 days and the
waiver's 14 (decision 250) · the 72-hour claim code · the 7-day connect link · signed jobs, 30 minutes to 24 hours.
## Decisions for the operator (also in `STATUS.md`)
1. **Shall the agent follow Proxmox's new certificate by itself?** Yes (CC's pick): the agent reads the certificate file
on its own host and takes the new fingerprint; one agent change, long before 2028. No: a hand step in a runbook and a
date in your calendar for each box; the cost is that it depends on memory, on every box, for ever. If nothing
happens: in July 2028 the first box loses its backups and guest control, and no alarm says why.
2. **Shall the product warn before an end date?** Yes (CC's pick): the box reads its web certificate's date and the hub
reads its tokens' dates; an alarm goes to you 14 days before. No: you keep the dates in a calendar by hand; the cost
is that the list above is the only guard. If nothing happens: a renewal that fails is silent until the day.
**Do, not decide:** (a) bake and vouch a golden, or write a dated waiver; (b) read the dates of the table "Only you can
read these dates" once, and write them in one place.
## Method verdict (fourth list)
- **It worked, and the measurement found the worst gap.** Two read-only helpers gave 58 raw things. The pin gap (DB02)
is in no helper answer as a date: it came from reading the certificate's date on the boxes and then Proxmox's own
renewal rule.
- **Read by eye: 85 of 312 cells** (every `D`, `K` and `N`, and the 41 `A` cells that `tools/marks.py` carries). The main session
read all helper answers to questions 4 to 7, and it did not open each of their pointers; a tool opened all 259.
- **The weak point:** for an outside account a list from documents can only say "no document says". Ten lines of this
list wait for a date that only the operator can read.
- **The ninth line:** 11 things have a hand step that no runbook holds; 2 more have it in part.
@@ -0,0 +1,724 @@
# The End-dates list — the table
Built 2026-11-11 from the source of the four repositories, from read-only measurements on the three test boxes
(Tester 1, demo-hp, the N100; host and guest) and from what any visitor can read with no login. One block for each
thing that stops working, or stops being safe, on a date or after an age. `tools/make-table.py` writes this file from
the helpers' answers (`evidence/helpers/`) and from `tools/marks.py`; do not edit it by hand.
**NOT MEASURED, by the operator's ruling of 2026-11-10:** DooPlex and ep0, and every account (Cloudflare, Hetzner, the
mail service, the registrars). What this table says about them comes from the repository's own scripts and documents.
**The eight questions:** 1 what it is, who owns it, where it lives · 2 when it ends (a real date for each box where one
was measured, else the rule) · 3 who renews it, and how long before · 4 what breaks, and for whom · 5 who is warned
before the end · 6 a box that was off past the end date: does it come back by itself? · 7 what happens if the renewal
fails · 8 how do we know (`MEASURED`, a test, `SOURCE`, a document, or nobody knows).
A ninth line, outside the marks: is the hand step written in a runbook?
**Marks:** `A` answered, with a pointer · `D` decided, not built · `K` known gap, with its row · `N` nobody asked — it
survived the searches in `EV/searches.txt` (S01…S23; controls in the same file). Every `N` names its class row:
**T1** = R-1379 an end exists and nobody is warned before it · **T2** = R-1380 another party renews and our pin does
not follow · **T3** = R-1381 no document says when it ends, or where to read the date · **T4** = R-1382 a box that was
off past the date: nobody knows if it comes back.
**Who read what.** A cell whose mark is `D`, `K` or `N`, and every `A` that `tools/marks.py` carries, holds the main
session's words: the pointer was opened and read. Every other cell holds a helper's words with the mark `A`:
`tools/resolve-helper.py` opened each of its pointers (the file and the line exist, and the name the helper gave stands
at that line). The main session read every helper answer to questions 4 to 7; it did not open each of their pointers.
Treat those cells as a map, not as proof.
**Pointers:** `AG/` = `felhom-agent/` · `CT/` = `felhom-controller/controller/` · `CAT/` = `app-catalog-felhom.eu/` ·
`HUB/` = `felhom.eu/hub/internal/` · `E/` = `felhom.eu/` · `AR/` = `felhom.eu/documentation/architecture/` ·
`RB/` = `felhom.eu/documentation/runbooks/` · `EV/` = this folder's `evidence/`. `tools/check-pointers.py` prints the
text behind each one.
**Named status** (does an architecture document name this end date?): `NAMED` and `MENTIONED` with a pointer were read
in the documents. Where the line says "a machine count", the status comes from a search of the architecture folder:
0 lines = `UNNAMED`, else `MENTIONED`.
## The kinds, and where each thing is
| Kind | Things (worst mark) | Named · mentioned · named nowhere |
|---|---|---|
| Certificate | DB01 (N), DB02 (N), DB03 (N), DH20 (N) | 1 · 1 · 2 |
| Key | DB06 (A), DB12 (K), DB13 (A), DB16 (N), DH08 (D), DH09 (A), DH10 (A), DH15 (A), DH22 (N), DH23 (N), DH25 (A), DH30 (N), DH31 (N), DH32 (N), DH33 (A) | 5 · 10 · 0 |
| Window | DB04 (N), DB05 (K), DB08 (A), DB09 (A), DB11 (A), DB24 (N), DH01 (A), DH05 (A), DH14 (A) | 1 · 8 · 0 |
| Version | DB18 (N), DB19 (N), DB20 (A), DB21 (K), DH19 (N), DH26 (A) | 2 · 4 · 0 |
| Account | DH24 (N), DH27 (N), DH28 (N), DH29 (N) | 0 · 4 · 0 |
| OS | DB17 (N) | 0 · 1 · 0 |
## Not in the table
**Tiny (named once, not tabled):** login sessions on the box and on the hub (hours to 7 days; a new login repairs it) · the one-use codes of the recovery ceremony (minutes) · the guest's token for the agent and the hub key (no date) · pinned build tools of the controller (a build fails loudly) · the hub's claim code and pairing code (no date; three resets a day) · operator actions in the queue (24 hours; a restart order 30 minutes) · the vouch alarm (2 hours) · the Debian template that the golden is built from (a bake fails loudly; the operator changes the default).
**Left out for another reason:**
- The recovery passphrase — it stops on an event (a new one is made), not on a date.
- Kept escrow packages — they have no end date by ruling; the 14-day countdown that removes one is DB04.
- The off-site clean-up window (20 minutes, 6.5 days) — it is a timer, so it is on the Clock list.
- The hub's view of signed jobs, signing keys, the golden's age, the agent versions, and OS support — the same things as DB11, DB12, DB21, DB20 and DB17.
- WireGuard re-key on the box — the same thing as DH10.
---
### DB01 — Web certificate of the household's address · Certificate · UNNAMED
architecture folder: 0 lines in 0 files match `/acme\.json|Let.s Encrypt|letsencrypt/` (a machine count, not a reading) · helper section: dates-box „1. Web certificate of the household's address (Let's Encrypt through Traefik)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The public TLS certificate for the dashboard and apps. Product-owned Traefik asks Let's Encrypt with the household's e-mail `ACMEEmail`; stored in `acme.json`. | A | `CT/internal/stacks/infra.go:130` `CT/internal/infra/templates/traefik.yml.tmpl:51` |
| 2 | MEASURED 2026-11-11: the N100's certificate ends 2026-12-31, Tester 1's 2027-01-02, demo-hp's 2027-01-19. The life (90 days) is the issuer's rule, not ours. | A | `EV/box/demo-felhom-guest-dates.txt` `EV/box/tester-1-guest-dates.txt` `EV/box/demo-hp-guest-dates.txt` |
| 3 | Traefik renews by itself, by its own default lead time. MEASURED on demo-hp: it got a new certificate on 2026-10-21, 90 days before the end date 2027-01-19. | A | `CT/internal/infra/templates/traefik.yml.tmpl:53` `CT/internal/infra/templates/traefik.yml.tmpl:59` |
| 4 | Browsers show a certificate error on every app address of the household. With no e-mail set Traefik serves a self-signed certificate, see comment at `TraefikData`. | A | `CT/internal/infra/infra.go:111-118` |
| 5 | Nobody. No part of the product reads the certificate's end date. If the renewal fails (the household's Cloudflare token ends, or the name check fails), the first sign is a browser warning on the end date. | N | `CT/internal/infra/infra.go:157-159` · class T1 · S01 |
| 6 | NOBODY KNOWS. A box that was off past the end date starts with a dead certificate. Traefik must get a new one at the start. Nobody has tested this. | N | `CT/internal/infra/infra.go:157-159` · class T4 · S11 |
| 7 | Traefik tries again by itself. The old certificate works until its end date. Nothing in the product reads the result, so a renewal that fails for weeks is silent. | N | `CT/internal/infra/templates/traefik.yml.tmpl:51` · class T1 · S01 |
| 8 | MEASURED: the three end dates, and one renewal by the product on demo-hp (the certificate was issued again on 2026-10-21) · SOURCE for the setup; NOBODY KNOWS for renewal timing, warning and the switched-off case. | A | `EV/box/demo-hp-guest-dates.txt` `CT/internal/infra/templates/traefik.yml.tmpl:48-59` |
**Hand step in a runbook?** NO — no runbook says what to do when a household's certificate does not renew
### DB02 — Proxmox's own certificate, pinned by the agent · Certificate · UNNAMED
architecture folder: 0 lines in 0 files match `/pve-ssl|tls\.fingerprint|TLS pin/` (a machine count, not a reading) · helper section: dates-box „2. Proxmox web certificate pinned by the agent”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The agent talks to Proxmox on the host and trusts ONE certificate by fingerprint. The installer reads the served leaf once, `fp=`; the agent checks it in `TLSConfig.build`. Product-owned. | A | `E/scripts/felhom-host-install.sh:3182-3185` `AG/internal/proxmox/tls.go:39-59` |
| 2 | MEASURED 2026-11-11: the certificate ends 2028-07-16 on the N100, 2028-08-19 on demo-hp, 2028-10-02 on Tester 1. On all three the agent's pin equals this certificate. The pin itself has no date. | A | `EV/box/pin.txt` `AG/internal/proxmox/tls.go:40-58` |
| 3 | Proxmox renews it by itself, about 14 days before the end (its daily update job). No agent code reads the new fingerprint. The installer writes the pin once. | N | `EV/box/pveupdate-excerpt.txt` `AG/internal/proxmox/tls.go:40-58` `E/scripts/felhom-host-install.sh:3182-3185` · class T2 · S02 |
| 4 | After Proxmox makes the new certificate, every call from the agent to Proxmox fails with 'TLS pin mismatch'. That is: guest start and stop, whole-guest backups, restore tests, disk work. First box: the N100, about 2028-07-02. The other two follow within three months. | N | `AG/internal/proxmox/tls.go:55` `EV/box/pin.txt` · class T2 · S02 |
| 5 | Nobody before the day. No code compares the pin with the certificate's end date. | N | `AG/internal/proxmox/tls.go:40-58` · class T2 · S02 |
| 6 | No. Proxmox also renews a certificate that has ended, at its next daily job. The box comes back with a new certificate and the old pin. | N | `EV/box/pveupdate-excerpt.txt` · class T2 · S02 |
| 7 | Here the renewal that works is the failure. The repair is by hand: write the new fingerprint into the agent's config and restart the agent. No code does it. | N | `AG/internal/proxmox/tls.go:40-58` · class T2 · S02 |
| 8 | MEASURED: the three end dates, pin equals certificate on all three, and Proxmox's own renewal rule read on demo-hp · TEST `TestTLSConfig_Build` for the pin check; NOBODY KNOWS for the renewal case. | A | `EV/box/pin.txt` `EV/box/pveupdate-excerpt.txt` `AG/internal/proxmox/tls_test.go:28` |
**Hand step in a runbook?** NO — a search of the runbooks for 'tls.fingerprint', 're-pin' and 'pin mismatch' finds nothing
### DB03 — The agent's local-API certificate, pinned in the guest · Certificate · NAMED
architecture: `AR/03-host-agent.md:420-423` · helper section: dates-box „3. Agent local-API certificate and its pin in the guest”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The self-signed leaf the agent serves to the in-guest controller, made by `generateLeaf`. The controller pins its fingerprint in `New`. Product-owned, on the host. | A | `AG/internal/localapi/cert.go:56` `CT/internal/agentapi/client.go:105` |
| 2 | `certValidity = 10 * 365 * 24 * time.Hour`, set as `NotAfter`. So a box installed in 2026 has a leaf that ends in 2036. | A | `AG/internal/localapi/cert.go:25` `AG/internal/localapi/cert.go:70` |
| 3 | Nobody: `EnsureLeaf` loads the existing pair and never looks at its date. The comment says rotation is an operator action for a later slice. | A | `AG/internal/localapi/cert.go:38-52` `AG/internal/localapi/cert.go:21-24` |
| 4 | At the end date nothing breaks: the controller compares the fingerprint only, not the date. A NEW certificate is what breaks the link. | A | `CT/internal/agentapi/client.go:118-128` `CT/internal/agentapi/client.go:125` |
| 5 | Nobody. The certificate lives 10 years and nothing reads its end date. The first one ends about 2036. | N | `AG/internal/localapi/cert.go:25` · class T1 · S01 |
| 6 | Yes, by itself: the leaf is generated once and kept, and the reinstall path keeps it with `--preserve-state-from`. | A | `AR/03-host-agent.md:420-423` `E/scripts/felhom-host-install.sh:2749-2786` |
| 7 | No renewal exists, so no failure state. A reinstall that makes a new leaf needs `--allow-new-leaf` and then the guest's pin is stale. | A | `E/scripts/felhom-host-install.sh:2749-2786` |
| 8 | TEST `TestEnsureLeaf_StableFingerprintAcrossReload` and `TestClient_WrongPinFailsClosed`; the 10-year end itself: SOURCE only. | A | `AG/internal/localapi/cert_test.go:12` `CT/internal/agentapi/client_test.go:55` |
**Hand step in a runbook?** no hand step
### DB04 — Off-site copy after the household stops it: the 14-day countdown · Window · MENTIONED
architecture folder: 21 lines in 9 files match `/14.day countdown|abandon/` (a machine count, not a reading) · helper section: dates-box „4. Off-site copy after the household stops it: 14-day countdown”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | When the household abandons the off-site backup, the copies are kept for a grace period and then deleted. `startAbandonCountdown`. The household starts it; the product runs it. | A | `CT/internal/backup/offbox_abandon.go:68-70` |
| 2 | `abandonGraceDays = 14`. | A | `CT/internal/backup/offbox_abandon.go:31` |
| 3 | The household can stop or extend it; extending never shortens, test `TestD1_ExtendNeverShortensTheCountdown`. | A | `CT/internal/backup/offbox_abandon_levers_d1_test.go:20` |
| 4 | The off-site copies are gone for good; the household loses its only copy outside the house. The deleting side is the hub (see the hub helper); test name `TestD1_ExtendRefusedWhileTheDeletionIsTheHubs`. | A | `CT/internal/backup/offbox_abandon_levers_d1_test.go:44` |
| 5 | The household, at 5, 3 and 1 days left: `AbandonRemindAtDays`. | A | `CT/internal/backup/offbox_abandon.go:40` |
| 6 | NOBODY KNOWS for a box that is off during the whole countdown: the reminders at 5, 3 and 1 days come from the box. Nobody traced if the old store is removed with no reminder shown. | N | `CT/internal/backup/offbox_abandon.go:31-40` · class T4 · S23 |
| 7 | A failed hub cancel is reported, not hidden: `TestD1_StopReportsFailureWhenTheHubCancelFails`. | A | `CT/internal/backup/offbox_abandon_levers_d1_test.go:69` |
| 8 | TEST `TestR241_ScenarioE_AbandonStartsAReversibleCountdown`. | A | `CT/internal/backup/offbox_abandon_r241_test.go:48` |
**Hand step in a runbook?** no hand step
### DB05 — Off-site history: how far back a household can go · Window · MENTIONED
architecture folder: 13 lines in 5 files match `/keepDaily|keep-daily|7 daily/` (a machine count, not a reading) · helper section: dates-box „5. Off-site backup history: how far back a household can go”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The restic off-site store keeps a thinning history. Policy in `retentionPolicy`. Product rule, household's data. | A | `CT/internal/backup/offbox_window.go:81-82` |
| 2 | `keepDaily = 7`, `keepWeekly = 4`, `keepMonthly = 6`. Older than about six months: gone. | A | `CT/internal/backup/offbox_window.go:47` `CT/internal/backup/offbox_window.go:48` `CT/internal/backup/offbox_window.go:49` |
| 3 | Nobody renews; each run forgets by the same rule, `retentionPolicy`. | A | `CT/internal/backup/offbox_window.go:81-82` |
| 4 | A file deleted or damaged more than six monthly points ago cannot be brought back from off-site. On-host copies are fewer: `KeepLast`, default 3 per comment. | A | `AG/internal/config/config.go:436` `AG/internal/config/config.go:409` |
| 5 | Nobody, and by ruling: the window is the rule. A point leaves it with no warning. | A | `CT/internal/backup/offbox_window.go:81-82` (the code path of question 1) |
| 6 | Yes. A box that was off simply has no points for that time; the next run applies the same rule. PBS copies are not pruned by the box by default, comment. | A | `AG/internal/backup/runner.go:51-58` |
| 7 | A clean-up that the guard refuses leaves more history than the rule says, not less. | K | R-1294 |
| 8 | SOURCE. | A | `CT/internal/backup/offbox_window.go:47-49` |
**Hand step in a runbook?** no hand step
### DB06 — Off-site credentials: SSH key, host key, store password, PBS pin · Key · MENTIONED
architecture folder: 37 lines in 9 files match `/host key|known_hosts|sub-account/` (a machine count, not a reading) · helper section: dates-box „6. Off-site credentials: SSH key, host key, store password, PBS pin”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The box reaches its off-site store with an SSH key and a pinned host key, `StrictHostKeyChecking=yes`, and a password file `RESTIC_PASSWORD_FILE`. PBS is pinned in `pinnedTLS`. | A | `CT/internal/backup/offbox.go:345` `CT/internal/backup/offbox.go:898` `AG/internal/pbs/pin.go:17-30` |
| 2 | NO END DATE in box code: no expiry field on the key, the host key or the password (grep `expir` in found none). The PBS pin ignores the date, `VerifyPeerCertificate`. The end comes from outside: the provider changes its host key or the hub ends the account. | A | `CT/internal/backup/offbox.go` `AG/internal/pbs/pin.go:24` |
| 3 | Nobody on a schedule. A new PBS descriptor is checked before use: `probeFP`. | A | `AG/internal/pbsdr/manager.go:375-379` |
| 4 | If the provider's host key changes, every off-site run fails at the SSH step, `StrictHostKeyChecking=yes`. The household has no off-site copy from that night on. | A | `CT/internal/backup/offbox.go:895` |
| 5 | Nobody, because there is no date. A failed night run is the sign. | A | `CT/internal/backup/offbox.go:345` `CT/internal/backup/offbox.go:898` (the code path of question 1) |
| 6 | Yes if nothing changed outside. A guest that was built again gets a new store password and cannot open the old store; the 14-day countdown (DB04) is the way out. | A | `CT/internal/backup/offbox.go:910` |
| 7 | PBS: state `verify_failed` and a retry at the next tick, `probeFP`. Restic: the run fails and the next night tries again with the same pins. | A | `AG/internal/pbsdr/manager.go:375-379` |
| 8 | SOURCE; provider-side dates: NOBODY KNOWS from the box. | A | `CT/internal/backup/offbox.go:345` |
**Hand step in a runbook?** yes — `RB/offsite-endpoint.md`
### DB08 — Claim code (72 hours) and the one-use gate token · Window · MENTIONED
architecture folder: 12 lines in 8 files match `/claim code|claimCodeTTL|72 h/` (a machine count, not a reading) · helper section: dates-box „8. Claim code and the one-use gate token”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The one-time code a household uses to set its own password, and the short token that carries a pass between addresses. Product-made. `claimCodeTTL`; `gateTokenLife`. | A | `CT/internal/web/claim.go:30` `CT/internal/web/setup_gate.go:47` |
| 2 | `claimCodeTTL = 72 * time.Hour`; `gateTokenLife = 60 * time.Second`, also stated in. | A | `CT/internal/web/claim.go:30` `CT/internal/web/setup_gate.go:47` `AR/01-topology-and-trust.md:192` |
| 3 | The operator sends a new code (hub side, see the hub helper). Nothing on the box mints one by itself. | A | `CT/internal/web/claim.go:30` `CT/internal/web/setup_gate.go:47` (the code path of question 1) |
| 4 | A household that opens the mail after three days cannot set its password until a new code arrives. Checked against the controller's clock `s.claimNow`. | A | `CT/internal/web/claim.go:333` |
| 5 | Nobody before the end. The household sees the dead code at the form, and the operator sends a new one. | A | `CT/internal/web/claim.go:30` `CT/internal/web/setup_gate.go:47` (the code path of question 1) |
| 6 | A box off for more than 72 hours comes back with a dead code; first failing step is the claim form,. | A | `CT/internal/web/claim.go:333` |
| 7 | Wrong tries lock the form for a time: `claimLockoutWindow`. | A | `CT/internal/web/claim.go:33` |
| 8 | TEST `TestClaimSubmit_ExpiredCodeRefused`. | A | `CT/internal/web/claim_test.go:189` |
**Hand step in a runbook?** no hand step
### DB09 — Share link · Window · MENTIONED
architecture folder: 5 lines in 4 files match `//s/<token>|share token|LauncherShareToken/` (a machine count, not a reading) · helper section: dates-box „9. Share link `/s/<token>`”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The link a household gives to guests to see its app tiles. Household-owned, stored as `LauncherShareToken`. | A | `CT/internal/settings/settings.go:45` |
| 2 | NO END DATE: the token has no expiry field; it ends only when the household rotates or disables it, `SetLauncherShareToken`. | A | `CT/internal/settings/settings.go:1111` |
| 3 | The household, by hand. No schedule. | A | `CT/internal/settings/settings.go:45` (the code path of question 1) |
| 4 | Nothing ends. A link that was given away years ago still opens the page, until the household changes it. | A | `CT/internal/web/share.go:31` |
| 5 | Nobody, because there is no date. The link ends only when the household changes it. | A | `CT/internal/settings/settings.go:45` (the code path of question 1) |
| 6 | Yes, unchanged. | A | `CT/internal/settings/settings.go:45` (the code path of question 1) |
| 7 | No renewal to fail. | A | `CT/internal/settings/settings.go:45` (the code path of question 1) |
| 8 | TEST `TestShareGuest_RotateAndDisable`. | A | `CT/internal/web/share_test.go:256` |
**Hand step in a runbook?** no hand step
### DB11 — Signed operator jobs: the validity window · Window · NAMED
architecture: `AR/04-control-plane-authorization.md:139-150` · helper section: dates-box „11. Signed operator jobs: the validity window”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | An order signed by the operator (update the agent, apply OS packages, destructive acts). Operator-owned; it carries a nonce and an expiry,. Checked by `Verify` and by the root wrapper. | A | `AR/03-host-agent.md:268` `AG/internal/authz/verifier.go:104` |
| 2 | At most `MaxWindow = 24 * time.Hour`; the signer's default is 30 minutes, `-ttl`. Wrapper: `MAX_WINDOW_S = 24 * 3600` AG/configs/felhom-os-apply:194. Design:. | A | `AG/internal/authz/verifier.go:26` `AG/cmd/felhom-opsign/main.go:57` `AR/04-control-plane-authorization.md:78-84` |
| 3 | The operator signs a new one. There is no renewal of a signed job. | A | `AR/03-host-agent.md:268` `AG/internal/authz/verifier.go:104` (the code path of question 1) |
| 4 | The job is refused and cleared; the executor is never called,. The box stays on its old agent or old packages. `ErrExpired`. | A | `AG/internal/signedjobs/runner.go:163-170` `AG/internal/authz/verifier.go:161-162` |
| 5 | Nobody before. A job that ends with no result is what the operator sees. A new job is signed. | A | `AR/04-control-plane-authorization.md:184` |
| 6 | A box off for longer than the window never runs that job; it is checked against the BOX's clock, `time.Now.UTC`, and in the wrapper `if now > exp or now < iss - 120` AG/configs/felhom-os-apply:676. A wrong box clock decides. | A | `AG/internal/authz/verifier.go:90` |
| 7 | The refused job is logged as a security event and removed, no retry. Used nonces are kept until their end, comment. | A | `AG/internal/signedjobs/runner.go:163-170` `AG/internal/authz/noncestore.go:47-50` |
| 8 | TEST `test_expired_signed_job_is_refused`, `test_a_window_longer_than_24h_is_refused`, `TestVerify_RejectsPerStage`. | A | `AG/configs/test_felhom_os_apply.py:833` `AG/configs/test_felhom_os_apply.py:836` `AG/internal/authz/verify_test.go:82` |
**Hand step in a runbook?** yes — the build skill's agent section
### DB12 — Operator signing keys on each box · Key · NAMED
architecture: `AR/04-control-plane-authorization.md:139-150` · helper section: dates-box „12. Operator signing keys on each box”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The two public keys every box trusts for signed jobs: `OPERATOR_KEY_OPERATIONAL_ID` and `OPERATOR_KEY_RECOVERY_ID`. Written to `TRUST_SIGNERS` AG/configs/felhom-os-apply:124 by `_write_slow_lane_trust`. Operator-owned. | A | `E/scripts/felhom-host-install.sh:212` `E/scripts/felhom-host-install.sh:214` `E/scripts/felhom-host-install.sh:3424` |
| 2 | NO END DATE: the keys carry no expiry; trust is by key material,. The key id is advisory,. | A | `AG/internal/authz/verifier.go:122-126` `AG/internal/authz/verifier.go:170-175` |
| 3 | The keys have no end date. A change of key is designed as a signed order of its own. Only the name of that order exists in code. | D | `AR/04-control-plane-authorization.md:139-150` |
| 4 | If the operator loses both private keys, no box can be updated or repaired by a signed job again; households keep running on what they have. Design text. | A | `AR/04-control-plane-authorization.md:139-150` |
| 5 | Nobody; there is no date to warn about. | A | `E/scripts/felhom-host-install.sh:212` `E/scripts/felhom-host-install.sh:214` (the code path of question 1) |
| 6 | Yes, the keys are in the installer and on disk,. | A | `E/scripts/felhom-host-install.sh:3452-3453` |
| 7 | If the keys are lost, no box accepts a signed job again without a visit. The keys exist on DooPlex only. | K | R-924 |
| 8 | DOC; for a real rotation: NOBODY KNOWS. | A | `AR/04-control-plane-authorization.md:139-150` |
**Hand step in a runbook?** no — the design exists, a procedure does not
### DB13 — The agent's Proxmox access token · Key · MENTIONED
architecture folder: 33 lines in 12 files match `/pveum|API token|privsep/` (a machine count, not a reading) · helper section: dates-box „13. Proxmox API token of the agent”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The token the agent uses for every Proxmox call. Made by the installer in `step_token`. Product-owned, on the host. | A | `E/scripts/felhom-host-install.sh:2405` |
| 2 | MEASURED 2026-11-11 on the three boxes: the user and its token have no end date (expire 0). | A | `EV/box/pin.txt` `E/scripts/felhom-host-install.sh:2448` |
| 3 | Nobody on a schedule. A re-run of the installer reuses it or makes a new one,. | A | `E/scripts/felhom-host-install.sh:2424-2445` |
| 4 | Nothing ends by date. If someone sets an expiry by hand in Proxmox, every agent call fails that day. | A | `E/scripts/felhom-host-install.sh:2405` (the code path of question 1) |
| 5 | Nobody, because there is no date. | A | `EV/box/pin.txt` |
| 6 | Yes. | A | `E/scripts/felhom-host-install.sh:2405` (the code path of question 1) |
| 7 | No renewal to fail. | A | `E/scripts/felhom-host-install.sh:2405` (the code path of question 1) |
| 8 | MEASURED: expire 0 on three boxes · SOURCE. | A | `EV/box/pin.txt` `E/scripts/felhom-host-install.sh:2448` |
**Hand step in a runbook?** no hand step
### DB16 — Package signing keys: Docker in the guest, Proxmox and Debian on the host · Key · MENTIONED
architecture: `AR/11-os-updates.md:657` · helper section: dates-box „16. Package signing keys: Docker in the guest, Proxmox and Debian on the host”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The keys apt uses to trust packages. Docker's key is fetched once at golden bake into `/etc/apt/keyrings/docker.asc`. Proxmox's keyring path default is in `align_apt_repos`. Owned by Docker, Proxmox, Debian. | A | `AG/configs/build-golden.sh:129` `E/scripts/felhom-host-install.sh:2616` |
| 2 | MEASURED 2026-11-11 on the three boxes: Docker's key never ends. The Debian keys in use end 2029-01-15 at the soonest. The Proxmox key for this series ends 2034-11-10. Two keys that end 2027-02-03 and 2027-04-12 are in the 'removed keys' file; nothing uses them. | A | `EV/box/keyrings.txt` |
| 3 | Debian and Proxmox keys arrive as packages through the normal update. Docker's key has no package: a change is "a slow-lane act for a person",. | A | `AR/11-os-updates.md:657` |
| 4 | The OS update is refused with code R7 when `apt-get update` fails, AG/configs/felhom-os-apply:1125-1127. The box keeps running but stops getting Docker security updates. Household unaware; operator sees the report. | A | `AG/configs/build-golden.sh:129` `E/scripts/felhom-host-install.sh:2616` (the code path of question 1) |
| 5 | Nobody before the end. After it the box reports a failed update. The first date is 2029-01-15. | N | `AR/11-os-updates.md:657` `EV/box/keyrings.txt` · class T1 · S10 |
| 6 | NOBODY KNOWS. A box that was off for years has old keys. The new keys come as a package, and the package list is checked with the old keys. | N | `AR/11-os-updates.md:657` · class T4 · S11 |
| 7 | The update fails cleanly and nothing is half-installed, the next signed job tries again and fails the same way until a person replaces the key. | A | `AR/11-os-updates.md:657` |
| 8 | MEASURED: every key's end date on three hosts and three guests · DOC; TEST for the refusal `test_R7_not_downloadable_and_no_snapshot`; key dates: NOBODY KNOWS. | A | `EV/box/keyrings.txt` `AR/11-os-updates.md:657` `AG/configs/test_felhom_os_apply.py:488` |
**Hand step in a runbook?** NO — the document says a key change is 'a slow-lane act for a person'; no procedure is written
### DB17 — Debian 13 and Proxmox 9: the series the product is tied to · OS · MENTIONED
architecture folder: 9 lines in 2 files match `/trixie|Debian 13|Proxmox (VE )?9/` (a machine count, not a reading) · helper section: dates-box „17. Debian 13 (trixie) and Proxmox 9: the series the product is tied to”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The OS series under everything. Written into source: suite default `trixie`, Docker repo `trixie stable`, base images CT/Dockerfile:52. Owned by Debian and Proxmox. | A | `E/scripts/felhom-host-install.sh:2615` `AG/configs/build-golden.sh:130` |
| 2 | No document gives the day when security fixes stop for Debian 13 or for Proxmox 9. MEASURED: the three boxes run Debian 13.7 and Proxmox 9.2. | N | `EV/box/demo-hp-host-dates.txt` `AR/11-os-updates.md:1` · class T3 · S09 |
| 3 | No plan with a date. The move to the next series is not designed. | N | `AR/11-os-updates.md:1` · class T3 · S09 |
| 4 | After the series ends: no security updates for host and guest; a new install on a newer Proxmox is refused in byo mode and only warned in appliance mode,. | A | `E/scripts/felhom-host-install.sh:2146-2152` |
| 5 | Nobody. The dates are public; nothing in the product or in a document reads them. | N | `AR/11-os-updates.md:1` · class T1 · S09 |
| 6 | Yes while the archives still serve trixie. Pinned OS updates fall back to snapshot.debian.org in `add_snapshot_sources` AG/configs/felhom-os-apply:1451-1462; if that source stops, the update is refused. | A | `E/scripts/felhom-host-install.sh:2615` `AG/configs/build-golden.sh:130` (the code path of question 1) |
| 7 | Refused update, box unchanged, AG/configs/felhom-os-apply:1125-1127. | A | `E/scripts/felhom-host-install.sh:2615` `AG/configs/build-golden.sh:130` (the code path of question 1) |
| 8 | SOURCE; the dates: NOBODY KNOWS. | A | `E/scripts/felhom-host-install.sh:2146-2152` |
**Hand step in a runbook?** NO
### DB18 — Catalog app images and their recorded digests · Version · MENTIONED
architecture: `AR/09-update-architecture.md:391` · helper section: dates-box „18. Catalog app images and their recorded digests”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | 65 app templates under CAT/templates/; 84 different image references; registries docker.io 47, ghcr.io 27, lscr.io 8, gitea.dooplex.hu 1, registry.gitlab.com 1. The update ladders record 149 different digests, e.g.. Owned by each app's publisher. | A | `CAT/templates/actualbudget/.felhom.yml:196` |
| 2 | NO END DATE in source. The end is the day a registry stops serving a tag or a digest. Moving tags are banned: `BANNED_TAGS`. | A | `CAT/scripts/check-image-pins.py:28` |
| 3 | The product, when it tests a new version and records its digest. reports how far behind a pin is and gates nothing. | A | `CAT/scripts/check-currency.py:22` |
| 4 | The box renders `tag@sha256` in `RenderWithLadderDigests`; a digest the registry no longer serves makes the pull fail on every box,. A household cannot install or update that app; a running app keeps running. | A | `CT/internal/stacks/digest.go:94` `AR/09-update-architecture.md:391` |
| 5 | Nobody. No check asks the registries if every recorded image is still served. The document speaks of the day a version is recorded, not of the days after. | N | `AR/09-update-architecture.md:391` `CAT/scripts/check-currency.py:22` · class T1 · S21 |
| 6 | Running apps come back (images are local). The first failing step is the next install, update or restore that must pull,. | A | `CT/internal/stacks/digest.go:10-23` |
| 7 | The pull fails and the app stays on its current image; no automatic repair. Retry happens only when the household presses again. | A | `CAT/templates/actualbudget/.felhom.yml:196` (the code path of question 1) |
| 8 | DOC; SOURCE; which digests are still served today: NOBODY KNOWS. | A | `AR/09-update-architecture.md:391` `CT/internal/stacks/digest.go:94` |
**Hand step in a runbook?** no hand step
### DB19 — Images the controller itself starts (Traefik, tunnel client, file browser, Samba) · Version · MENTIONED
architecture folder: 41 lines in 15 files match `/cloudflared|TraefikImage|traefik:v/` (a machine count, not a reading) · helper section: dates-box „19. Images the controller itself starts (Traefik, tunnel, file browser, Samba)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Four fixed images in the controller binary: `TraefikImage`, `CloudflaredImage`, `FileBrowserImage`, `SambaImage`. Product-owned pins of other people's images. | A | `CT/internal/infra/infra.go:33` `CT/internal/infra/infra.go:34` `CT/internal/infra/infra.go:35` `CT/internal/infra/infra.go:50` |
| 2 | No document says for how long Cloudflare accepts an old tunnel client. MEASURED on the N100: tunnel client 2026.10.3, Traefik v3.7.16. | N | `EV/box/demo-felhom-guest-dates.txt` · class T3 · S12 |
| 3 | The product, by a controller release that names newer tags. | A | `CT/internal/infra/infra.go:33` `CT/internal/infra/infra.go:34` (the code path of question 1) |
| 4 | If Cloudflare stops accepting an old tunnel client, the household's address stops answering from outside while the box is healthy. If a registry drops a tag, a box that is built again cannot start that part. | A | `CT/internal/infra/infra.go:33` `CT/internal/infra/infra.go:34` (the code path of question 1) |
| 5 | Nobody. | N | `EV/box/demo-felhom-guest-dates.txt` · class T1 · S12 |
| 6 | NOBODY KNOWS if the tunnel of a box that was off for a year still connects with its old client. Without the tunnel the box gets no update from outside. | N | `EV/box/demo-felhom-guest-dates.txt` · class T4 · S12 |
| 7 | Not traced. | A | `CT/internal/infra/infra.go:33` `CT/internal/infra/infra.go:34` (the code path of question 1) |
| 8 | SOURCE; NOBODY KNOWS for the tunnel client's age limit. | A | `CT/internal/infra/infra.go:33-50` |
**Hand step in a runbook?** no hand step
### DB20 — Published agent versions: the newest five stay · Version · NAMED
architecture: `AR/09-update-architecture.md` §3, D50 · helper section: dates-box „20. Published agent versions: only the newest five stay downloadable”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The registry of agent binaries. Rule file `generic_versions_kept`. Operator-owned; he prunes by hand,. | A | `AG/scripts/retention-policy.json:23` `AG/scripts/retention-policy.json:7-10` |
| 2 | Newest 5 versions plus two pins: `hub_min_agent` and `day0_agent`. Decision D50,. | A | `AG/scripts/retention-policy.json:30` `AG/scripts/retention-policy.json:31` `AR/09-update-architecture.md:1370-1375` |
| 3 | The session that records a vouch or a new floor updates the pins in the same commit,. | A | `AG/scripts/retention-policy.json:12-15` |
| 4 | A box told to install a version that was pruned cannot download it. A pin that names a missing version is a CI failure,. Deleting a git tag also ends the ability to install that version,. | A | `AG/scripts/retention-policy.json:14-15` `AG/scripts/retention-policy.json:20-21` |
| 5 | The operator, through CI:. | A | `AG/scripts/retention-policy.json:25` |
| 6 | A box off for long is on an old agent; it updates only by a fresh signed job that names a version still kept. First failing step otherwise: the download in the wrapper, `SELFUPDATE_OP` AG/configs/felhom-os-apply:180. | A | `AG/scripts/retention-policy.json:23` `AG/scripts/retention-policy.json:7-10` (the code path of question 1) |
| 7 | Not traced on a failed download. | A | `AG/scripts/retention-policy.json:23` `AG/scripts/retention-policy.json:7-10` (the code path of question 1) |
| 8 | SOURCE; DOC. | A | `AG/scripts/retention-policy.json:23-31` `AR/09-update-architecture.md:1370-1375` |
**Hand step in a runbook?** yes — the build skill's agent section
### DB21 — The golden image's age (7 days) and the waiver (14 days) · Version · NAMED
architecture: `AR/09-update-architecture.md:1010-1018` · helper section: dates-box „21. Golden image age and the waiver file”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The ready-made guest image a new box is installed from, baked weekly,. Operator vouches. Gate:. | A | `AR/09-update-architecture.md:1010-1018` `E/scripts/golden_currency_gate.py:169` |
| 2 | MEASURED 2026-11-11: the newest golden is 8 days old (baked 2026-11-03). The limit is 7. No waiver file exists. The gate fails today. | A | `E/scripts/golden_currency_gate.py:169` `EV/box/golden-gate.txt` |
| 3 | The operator, weekly (Sunday night per). | A | `AR/09-update-architecture.md:1010-1018` |
| 4 | The gate goes red for the product team; no household is touched. A new install gets an older image and must update more on first start. | A | `AR/09-update-architecture.md:1010-1018` `E/scripts/golden_currency_gate.py:169` (the code path of question 1) |
| 5 | The operator, by the gate itself; no earlier warning found. | A | `AR/09-update-architecture.md:1010-1018` `E/scripts/golden_currency_gate.py:169` (the code path of question 1) |
| 6 | Not about a box. | A | `AR/09-update-architecture.md:1010-1018` `E/scripts/golden_currency_gate.py:169` (the code path of question 1) |
| 7 | A push of code to this repository is refused until a golden is baked or a dated waiver is written. A push of documents passes with a notice. | K | R-468 |
| 8 | MEASURED: the gate's own output today · SOURCE; tests exist in (names not matched to this rule). | A | `EV/box/golden-gate.txt` `E/scripts/golden_currency_gate.py:162-169` `E/scripts/test_golden_currency_gate.py:1` |
**Hand step in a runbook?** yes — `RB/RUNBOOK-manual-build.md`
### DB24 — Copies kept before a restore: deleted after 30 unused days · Window · MENTIONED
architecture folder: 18 lines in 4 files match `/restore cop|restore-undo|D62/` (a machine count, not a reading) · helper section: dates-box „24. Kept copies made before a restore: deleted after 30 unused days”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The copy of an app's data the box keeps when the household restores an older state. Household's data, product rule D62. `restoreCopyUnusedAge`. | A | `CT/internal/backup/restore_copy_sweep.go:29` |
| 2 | `restoreCopyUnusedAge = 30 * 24 * time.Hour`. | A | `CT/internal/backup/restore_copy_sweep.go:29` |
| 3 | Nobody; using the copy is what keeps it. | A | `CT/internal/backup/restore_copy_sweep.go:29` (the code path of question 1) |
| 4 | The household loses the way back to the state from before the restore. | A | `CT/internal/backup/restore_copy_sweep.go:29` (the code path of question 1) |
| 5 | Nobody. The household is not told before the copy goes. The household can delete it sooner by hand. | N | `CT/internal/backup/restore_copy_sweep.go:18-29` · class T1 · S05 |
| 6 | A box that was off for more than 30 days deletes the copy at its first sweep after the start: the age is counted from the folder's time stamp by the wall clock. Nobody asked if the off time must count. | N | `CT/internal/backup/restore_copy_sweep.go:21-29` · class T4 · S11 |
| 7 | Not traced. | A | `CT/internal/backup/restore_copy_sweep.go:29` (the code path of question 1) |
| 8 | SOURCE. · Mail relay's in-memory self-signed leaf, 10 years, remade at each start:. · Language cookie, 365 days:. · HSTS max-age one day: `HSTSMaxAgeSeconds`. · A backup counts as fresh enough for an update for 24 hours: `DefaultUpdateBackupMaxAge`. · Night-backup banner turns stale after 26 hours: `StaleAfter`. … | A | `CT/internal/backup/restore_copy_sweep.go:29` `CT/internal/mailrelay/server.go:386` `CT/internal/web/i18n_web.go:478` `CT/internal/infra/infra.go:108` |
**Hand step in a runbook?** no hand step
### DH01 — Connect link in the household's e-mail (7 days) · Window · MENTIONED
architecture folder: 32 lines in 7 files match `/self-bind|selfbind|/bind//` (a machine count, not a reading) · helper section: dates-hub „1. Self-bind (connect) link in the household's e-mail”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | A one-use link that lets the household tie a new box to its account. Household uses it, product mints it. Stored hashed in the hub table `selfbind_tokens`; minted by `selfBindTTL` code. | A | `HUB/store/store.go:796-806` `HUB/web/selfbind_mint.go:20` |
| 2 | 7 days after minting: `selfBindTTL`, passed to the store at; `Expired`. Also locks at the 5th wrong answer, `SelfBindMaxAttempts`. | A | `HUB/web/selfbind_mint.go:20` `HUB/web/selfbind_mint.go:77` `HUB/store/selfbind.go:39` `HUB/store/selfbind.go:21` |
| 3 | The household itself: the expired page has a button, `handleBindResend`, at most one new link an hour, `bindResendEvery`. No renewal before the end. The hub also mints at customer creation and RESET. | A | `HUB/web/selfbind.go:384` `HUB/web/selfbind.go:382` `HUB/web/selfbind_mint.go:97` |
| 4 | The household cannot bind the box from the mail; unknown and expired links show the same "expired" page. The box itself keeps waiting; nothing on it is lost. | A | `HUB/web/selfbind.go:206` |
| 5 | Nobody before, by design: the page of a dead link has a button that sends a new one. | A | `HUB/web/selfbind_mint.go:1-110` `HUB/web/selfbind_mint.go:93` |
| 6 | A box first switched on after day 7: the link fails at the first step (opening it); the household presses the resend button. If an old host is still linked no link is sent and the operator gets `eventBindOldHostLinked`, logged at. | A | `HUB/web/selfbind.go:440` `HUB/web/selfbind.go:413` |
| 7 | If the mail send fails at mint the token is deleted, so no dead link is left. No automatic retry; the household presses again after an hour. | A | `HUB/web/selfbind_mint.go:82` |
| 8 | TEST `TestR719_AnExpiredLinkOffersAndSendsAFreshOne`; SOURCE; DOC. | A | `HUB/web/selfbind_resend_test.go:21` `HUB/web/selfbind_mint.go:20` `AR/05-hub-architecture.md:359` |
**Hand step in a runbook?** no hand step
### DH05 — 'Switched off on purpose' hold (14 days) · Window · MENTIONED
architecture folder: 10 lines in 4 files match `/deliberate.off|switched off on purpose/` (a machine count, not a reading) · helper section: dates-hub „5. "Switched off on purpose" hold (14 days of silence allowed)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | When a household switches the box off through the product, the hub stays quiet about the silence. Household act, hub rule: `DeliberateOffHold`. | A | `HUB/store/deliberateoff.go:18` |
| 2 | 14 days after the switch-off: `DeliberateOffHold`, judged by `DeliberateOffHeld`. | A | `HUB/store/deliberateoff.go:18` `HUB/store/deliberateoff.go:21` |
| 3 | Nobody renews. The hold ends by itself when the box reports again or at day 14. | A | `HUB/store/deliberateoff.go:18` (the code path of question 1) |
| 4 | After day 14 the silence and missed-backup alarms start again for that household; the operator is told. | A | `HUB/store/deliberateoff.go:18` (the code path of question 1) |
| 5 | The operator gets an event at switch-off time naming the moment, message built at. No warning before day 14 was found (grep `DeliberateOffHold` in hub). | A | `HUB/api/deliberate_off.go:42` |
| 6 | Yes, it comes back by itself: a box switched on at any time reports and the hold clears. TEST `TestDeliberateOff_BackOnIsQuiet`. | A | `HUB/monitor/deliberate_off_test.go:138` |
| 7 | Not applicable: nothing is renewed. | A | `HUB/store/deliberateoff.go:18` (the code path of question 1) |
| 8 | TEST `TestDeliberateOff_AlarmsAfterFourteenDays` and `TestDeliberateOff_NoSilenceAlarmDuringTheHold`. | A | `HUB/monitor/deliberate_off_test.go:112` `HUB/monitor/deliberate_off_test.go:90` |
**Hand step in a runbook?** no hand step
### DH08 — Storage Box sub-account password, and the hub's seal key · Key · MENTIONED
architecture folder: 27 lines in 8 files match `/seal key|sub-account/` (a machine count, not a reading) · helper section: dates-hub „8. Storage Box sub-account password and the hub's seal key”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Each household's off-site store account at the provider, minted by the hub, and the hub key that seals stored secrets (`OFFSITE_SECRET_KEY`, named in). Operator and product. | A | `E/manifests/hub.yaml:167-171` |
| 2 | NO END DATE in code: no lifetime on the sub-account password. A clean reinstall's re-issue resets it, DOC; `ResetBoxPassword`. | A | `AR/05-hub-architecture.md:274-279` `HUB/hetznerapi/hetznerapi.go:297` |
| 3 | NOT MEASURED (DooPlex, ep0 and the accounts are outside this run). The password changes when the operator presses Re-issue. A change of the seal key is 'not built'. | D | `RB/secrets.md:159` |
| 4 | If the seal key is lost or changed, the old sealed rows cannot be opened, DOC: operator loses stored off-site secrets and break-glass passwords until re-issued per household. | A | `RB/secrets.md:159` |
| 5 | Nobody, because there is no date. | A | `HUB/hetznerapi/hetznerapi.go:1-320` |
| 6 | A box off for a long time comes back by itself as long as its account was not re-issued; it uses its SSH key, the password is used only by the hub, DOC. | A | `AR/05-hub-architecture.md:274-279` |
| 7 | No renewal on a date exists, so none can fail on a date. | A | `E/manifests/hub.yaml:167-171` (the code path of question 1) |
| 8 | DOC and; SOURCE. No test of a lifetime, because there is none. | A | `RB/secrets.md:159` `AR/05-hub-architecture.md:274-279` `HUB/hetznerapi/hetznerapi.go:297` |
**Hand step in a runbook?** partly — `RB/secrets.md:159` says the seal key change is not built
### DH09 — PBS off-site token on ep0 · Key · NAMED
architecture: `AR/05-hub-architecture.md:379-387` · helper section: dates-hub „9. PBS off-site token on ep0 and its descriptor on the hub”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The per-household token on the rented endpoint for the whole-guest off-site copy, and the hub record that must agree with it. Product mints, operator can re-issue. DOC. | A | `AR/05-hub-architecture.md:373-387` |
| 2 | NO END DATE stated: the lifecycle table has only events (provision, re-key, deprovision). One age rule: a staged secret not consumed in 15 minutes is wrong, `unconsumedGrace`. | A | `AR/05-hub-architecture.md:379-387` `HUB/pbsdrheal/reconciler.go:63` |
| 3 | The hub re-keys on re-issue (delete and recreate). No scheduled renewal. | A | `AR/05-hub-architecture.md:382` |
| 4 | A re-key the box did not pick up leaves the box "serving a revoked credential": the household has no whole-guest off-site copy while both tiers look green, comment at. | A | `HUB/pbsdrheal/reconciler.go:55-63` |
| 5 | Operator only, after the fact: `pbsdr_provision_failed` on a final failure, DOC. | A | `AR/05-hub-architecture.md:400-402` |
| 6 | Host delete keeps the token; a box off and back uses the same token. After a re-issue while it was off, it must consume the new secret on its next report. | A | `AR/05-hub-architecture.md:384` |
| 7 | A failed self-check rolls back and is repeated 3 times, 7 s apart, then raises the operator event, DOC. | A | `AR/05-hub-architecture.md:400-402` |
| 8 | DOC; SOURCE. Lifetime of the token on the endpoint itself: NOBODY KNOWS from these sources (ep0 not read). | A | `AR/05-hub-architecture.md:373-403` `HUB/pbsdrheal/reconciler.go:63` |
**Hand step in a runbook?** yes — `RB/offsite-endpoint.md`
### DH10 — WireGuard keys of the box's tunnel to the off-site endpoint · Key · NAMED
architecture: `AR/06-offsite-connectivity.md:200-205` · helper section: dates-hub „10. WireGuard peers (the box's tunnel to the off-site endpoint)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | One peer per box on the endpoint, registered through the hub; the private key is in the household's sealed bundle. Product. DOC. | A | `AR/06-offsite-connectivity.md:172-182` |
| 2 | NO END DATE: the hub store only removes a peer on request, `RemoveWGPeer`. Scheduled key rotation is an open point, DOC. | A | `HUB/store/wg.go:345` `AR/06-offsite-connectivity.md:368` |
| 3 | Nobody on a schedule. The agent makes a new key itself only after a handshake stamped in the future for 15 minutes, DOC. | A | `AR/06-offsite-connectivity.md:205-207` |
| 4 | Revocation: the box is off the off-site network at once, DOC; the household's off-site copies stop. | A | `AR/06-offsite-connectivity.md:180-182` |
| 5 | Nobody, because there is no date. | A | `HUB/store/wg.go:1-360` |
| 6 | Yes, a box off for months keeps its peer. A revoked box never re-registers on its own, DOC. | A | `AR/06-offsite-connectivity.md:200-205` |
| 7 | A failed fetch is never read as a teardown, DOC. Re-add is an operator call. | A | `AR/06-offsite-connectivity.md:207-208` |
| 8 | DOC; SOURCE. Box-side tests belong to the box helper. | A | `AR/06-offsite-connectivity.md:200-208` `HUB/store/wg.go:345` |
**Hand step in a runbook?** yes — `RB/offsite-endpoint.md`
### DH14 — Recovery mode for a host (30 minutes to 4 hours) · Window · MENTIONED
architecture folder: 6 lines in 4 files match `/recovery mode|defaultRecoveryTTL/` (a machine count, not a reading) · helper section: dates-hub „14. Recovery mode for a host (the door a rebuilt box uses)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | A switch the operator sets on a host so a rebuilt box may re-enrol and fetch its restore data. Operator, global key only, `handleSetRecoveryMode`. | A | `HUB/api/dr.go:34` |
| 2 | 30 minutes by default, `defaultRecoveryTTL`; at most 4 hours, `maxRecoveryTTL`; clamp at. | A | `HUB/api/dr.go:21` `HUB/api/dr.go:22` `HUB/api/dr.go:52` |
| 3 | The operator sets it again. No lead time. | A | `HUB/api/dr.go:34` (the code path of question 1) |
| 4 | A restore that takes longer than the window: the box's re-enrol call is refused and the household's recovery stops until the operator re-opens it. | A | `HUB/api/dr.go:34` (the code path of question 1) |
| 5 | Nobody before. The operator who opened the window is present; he opens it again. | A | `HUB/api/dr.go:34` (the code path of question 1) |
| 6 | No: a box that starts its recovery after the window fails at the re-enrol step. TEST `TestReEnroll_GatedOnRecoveryMode`. | A | `HUB/api/dr_test.go:14` |
| 7 | Nothing is left half-done by the expiry itself; the operator opens a new window. | A | `HUB/api/dr.go:34` (the code path of question 1) |
| 8 | TEST `TestRestoreDirective_GatedAndExpires`; SOURCE. | A | `HUB/api/dr_test.go:70` `HUB/api/dr.go:21-22` |
**Hand step in a runbook?** yes — the recovery runbooks
### DH15 — Break-glass password of a host · Key · MENTIONED
architecture folder: 42 lines in 13 files match `/break-glass|host_recovery/` (a machine count, not a reading) · helper section: dates-hub „15. Break-glass credential (the host's emergency root password)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | A per-host emergency password, sealed in the hub, shown only to a logged-in operator. Operator. `SaveHostRecoveryCredential`; reveal. | A | `HUB/store/host_recovery.go:28` `HUB/web/hosts.go:983` |
| 2 | NO END DATE: last write wins, day-0 sets it and a rotate re-sets it. Paper rule: re-print the sheet "when a key rotates, and yearly". | A | `HUB/store/host_recovery.go:20` `RB/break-glass-sheet.md:75` |
| 3 | The operator, yearly by the paper rule; no code reminds him. | A | `HUB/store/host_recovery.go:28` `HUB/web/hosts.go:983` (the code path of question 1) |
| 4 | If the sealed copy cannot be opened (seal key, block 8) or the paper is stale, the operator has no way into a host whose agent is dead. | A | `HUB/store/host_recovery.go:28` `HUB/web/hosts.go:983` (the code path of question 1) |
| 5 | Nobody before. Each reveal writes the event `recovery_credential_revealed`. | A | `HUB/web/hosts.go:1010` |
| 6 | Yes, the password does not age on the box. | A | `HUB/store/host_recovery.go:28` `HUB/web/hosts.go:983` (the code path of question 1) |
| 7 | No renewal on a date exists, so none can fail on a date. | A | `HUB/store/host_recovery.go:20-43` |
| 8 | SOURCE; session-only route; DOC. A yearly re-print has no record: NOBODY KNOWS. | A | `HUB/store/host_recovery.go:20-28` `HUB/web/routes.go:130` `RB/break-glass-sheet.md:75` |
**Hand step in a runbook?** yes — `RB/break-glass-sheet.md:75` (print the sheet again each year)
### DH19 — The installer ISO · Version · MENTIONED
architecture folder: 3 lines in 3 files match `/ISO_VERSION|installer ISO|iso\.felhom\.eu/` (a machine count, not a reading) · helper section: dates-hub „19. The installer ISO (Proxmox base, installer version, first boot)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The USB image a household or volunteer installs from. Product; built by, served from Cloudflare R2, DOC. | A | `E/scripts/iso/build-felhom-iso.sh:51` `RB/total-loss-of-dooplex.md:125` |
| 2 | NO END DATE in code. `ISO_VERSION` is 1.30.0. The Proxmox ISO and its checksum are build ARGUMENTS, not pinned: the example names 9.2-1. | A | `E/scripts/iso/build-felhom-iso.sh:51` `E/scripts/iso/build-felhom-iso.sh:194` `E/scripts/iso/README.md:40` |
| 3 | No document says how old a public ISO may be, or when to build a new one. The ISO carries a Proxmox base and a first-boot program that age. | N | `E/scripts/iso/build-felhom-iso.sh:51` `RB/iso-release-gate.md:1` · class T3 · S22 |
| 4 | An old ISO installs an old Proxmox base; the install script itself is fetched fresh at first boot (comment block at), so the hub decides the agent and golden. | A | `E/scripts/iso/build-felhom-iso.sh:44-52` |
| 5 | Nobody. The build warns only if golden and floor are not given: unenforced branch. | A | `E/scripts/iso/build-felhom-iso.sh:79` |
| 6 | A too-old ISO on first boot: registers, pairs, fetches the current install script; first failing step would be that fetch or the manifest. No "ISO too old" check found (grep `too old/min.*version` in bootstrap). | A | `E/scripts/felhom-host-install.sh:472` |
| 7 | A failed install is retried every 30 s; a failed script fetch is retried. | A | `E/scripts/iso/felhom-bootstrap.sh:430` `E/scripts/iso/felhom-bootstrap.sh:396` |
| 8 | SOURCE (checksum of the given ISO is verified). How old an ISO still installs cleanly: NOBODY KNOWS. Build image is `FROM debian:trixie`, not pinned. | A | `E/scripts/iso/build-felhom-iso.sh:222` `E/scripts/iso/Dockerfile.assistant:10` |
**Hand step in a runbook?** yes for a build — `RB/iso-release-gate.md`; NO for 'when'
### DH20 — The certificate that the hub's address serves · Certificate · MENTIONED
architecture folder: 1 lines in 1 files match `/cert-manager|letsencrypt-prod/` (a machine count, not a reading) · helper section: dates-hub „20. The certificate the hub's address serves”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The TLS certificate for the hub's public name, used by every box and the operator. Operator's cluster issues it: annotation, stored under the secret name at. | A | `E/manifests/hub.yaml:342` `E/manifests/hub.yaml:360` |
| 2 | Read from outside 2026-11-11, with no login: the certificate of the hub's address ends 2027-01-12. The download address ends 2026-12-23, the ISO address 2026-12-27, the Gitea address 2027-01-03, the website 2027-01-28. | A | `EV/box/public-reads.txt` |
| 3 | NOT MEASURED on DooPlex. cert-manager on DooPlex renews by itself. Read from outside: the certificate served today was issued on 2026-10-14, so a renewal worked. | A | `E/manifests/hub.yaml:342` `EV/box/public-reads.txt` |
| 4 | If it lapses: boxes that pin or verify the hub cannot report or fetch config; households get no alarms and no updates; the operator page shows a browser warning. | A | `E/manifests/hub.yaml:342` `E/manifests/hub.yaml:360` (the code path of question 1) |
| 5 | NOT MEASURED (DooPlex, ep0 and the accounts are outside this run). In the Felhom repositories nothing warns before this date. If it ends, every box fails to report, and the hub then sees all boxes as silent. | N | `E/manifests/hub.yaml:342` `EV/box/public-reads.txt` · class T1 · S01 |
| 6 | Yes. The agent trusts the public issuers, not one certificate, so a renewed certificate is accepted. | A | `AG/internal/hub/client.go:41-51` |
| 7 | The old certificate serves until its end date. Nothing in the Felhom repositories warns (question 5). | A | `E/manifests/hub.yaml:342` |
| 8 | READ FROM OUTSIDE, no login · SOURCE and only. Dates: NOBODY KNOWS from files; needs a measurement. | A | `EV/box/public-reads.txt` `E/manifests/hub.yaml:342` `E/manifests/hub.yaml:360` |
**Hand step in a runbook?** NO
### DH22 — Operator access tokens, among them CC's hub token · Key · NAMED
architecture: `AR/05-hub-architecture.md` §17 · helper section: dates-hub „22. Operator access tokens, among them CC's hub token”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Named, scoped tokens for scripts, shown once, stored hashed. Operator creates them in a browser session; CC holds one, with `overview:read` and `floor:write`, DOC. | A | `AR/09-update-architecture.md:1022` |
| 2 | 90 days by default, `tokenDefaultDays`; at most 365, `tokenMaxDays`; "never" is not offered. End computed at. | A | `HUB/web/tokens.go:29` `HUB/web/tokens.go:30` `HUB/web/tokens.go:254` `HUB/web/tokens.go:180` |
| 3 | The operator makes a new token and puts it in CC's store. There is no renew; no lead time is set anywhere. | A | `AR/09-update-architecture.md:1022` (the code path of question 1) |
| 4 | CC's floor raise and overview reads get 401; a controller release cannot be carried to boxes by CC until the operator acts. Households wait for the update. | A | `HUB/web/tokens.go:95` |
| 5 | Nobody. A token lives 90 days by default and 365 at most. The token page shows the date; no monitor reads it. On the end date the release step that sets a floor answers 401. | N | `HUB/web/tokens.go:29-30` `HUB/web/tokens.go:180` · class T1 · S03 |
| 6 | Not a box matter. | A | `AR/09-update-architecture.md:1022` (the code path of question 1) |
| 7 | Not applicable; an expired token stays listed as expired. Column `expires_at`. | A | `HUB/store/api_tokens.go:52` |
| 8 | TEST `TestToken_NoTokenUnknownExpiredRevoked401`; DOC. The end DATE of CC's token is in no document (grep `HUB_TOKEN` in AR, RB, operations, STATUS): NOBODY KNOWS without the hub page. | A | `HUB/web/tokens_test.go:310` `AR/05-hub-architecture.md:655-669` |
**Hand step in a runbook?** partly — the build skill says '401 = token expired'; no step says to look at the date before
### DH23 — A household's own Cloudflare token · Key · MENTIONED
architecture folder: 9 lines in 4 files match `/Cloudflare token|CF_DNS_API_TOKEN/` (a machine count, not a reading) · helper section: dates-hub „23. A household's own Cloudflare token (hub side)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | A household that brings its own domain gives a Cloudflare API token; the hub checks what it can reach before saving. Household owns it.. | A | `HUB/cloudflare/reach.go:14` |
| 2 | The hub and the box do not know the end date that the household set at Cloudflare. No document tells the household to set none. | N | `HUB/cloudflare/reach.go:14` · class T3 · S15 |
| 3 | The household, at Cloudflare; then the operator saves the new token in the hub. | A | `HUB/cloudflare/reach.go:14` (the code path of question 1) |
| 4 | If it lapses, the household's remote address stops being managed; candidate: first sign is a failed DNS or tunnel call on the box. | A | `HUB/cloudflare/reach.go:14` (the code path of question 1) |
| 5 | Nobody. When the token ends, the web certificate stops renewing (see DB01), with no sign until the certificate's own end. | N | `CT/internal/infra/infra.go:157-159` · class T1 · S15 |
| 6 | The running apps and the tunnel do not use this token. The certificate renewal does. | A | `CT/internal/infra/infra.go:157-159` |
| 7 | The renewal is the household's hand step at Cloudflare. If it is not done, the web certificate stops renewing (DB01). | A | `HUB/cloudflare/reach.go:14` (the code path of question 1) |
| 8 | SOURCE; the rest NOBODY KNOWS from hub files. | A | `HUB/cloudflare/reach.go:14` |
**Hand step in a runbook?** NO
### DH24 — The mail service key (all household and operator mail) · Account · MENTIONED
architecture: `AR/_recovery-inventory-2026-07-28.md` · helper section: dates-hub „24. The mail service key (all household and operator mail)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The key the hub uses to send every mail: alarms, bind links, claim codes. Operator's account at the mail service; named in, value not opened. | A | `E/manifests/hub.yaml:148-152` |
| 2 | NOT MEASURED (DooPlex, ep0 and the accounts are outside this run). No document gives an end date for the key or for the account. | N | `RB/secrets.md:33` · class T3 · S08 |
| 3 | The operator re-mints at the service dashboard, DOC. | A | `RB/total-loss-of-dooplex.md:47` |
| 4 | All mail stops: no alarm reaches the operator or a household, no bind link, no claim code. Everyone. | A | `E/manifests/hub.yaml:148-152` (the code path of question 1) |
| 5 | Nobody. No scheduled check proves that mail still goes out. If the key ends, no alarm mail can say so. | N | `RB/secrets.md:33` · class T1 · S08 |
| 6 | Not a box matter; boxes send through the hub. | A | `E/manifests/hub.yaml:148-152` (the code path of question 1) |
| 7 | If the key ends and no new key is made, no mail goes out, alarm mail included (question 5). | A | `HUB/web/selfbind_mint.go:82` |
| 8 | DOC and. Whether the key or the plan has an end date: NOBODY KNOWS from files. | A | `RB/secrets.md:33` `RB/total-loss-of-dooplex.md:47` |
**Hand step in a runbook?** yes for a new key — `RB/secrets.md:33`
### DH25 — The hub's global bearer key · Key · NAMED
architecture: `AR/13-threat-model.md:70` · helper section: dates-hub „25. The hub's global bearer key”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The operator-tier key for the hub API (recovery mode, peer re-add). Operator. Named in; inventory row. | A | `E/manifests/hub.yaml:157-161` `AR/_recovery-inventory-2026-07-28.md:501` |
| 2 | NO END DATE. Rotation is a supervised runbook. | A | `RB/secrets.md:124` |
| 3 | The operator, with an ordered procedure; no cadence stated. | A | `RB/secrets.md:124` |
| 4 | A rotation done out of order locks out whatever still holds the old key; DOC calls the order load-bearing. | A | `RB/secrets.md:124` |
| 5 | Nobody; no date. | A | `E/manifests/hub.yaml:157-161` `AR/_recovery-inventory-2026-07-28.md:501` (the code path of question 1) |
| 6 | No date drives it. Boxes use their own keys since 2026-10-12. What a box does that holds only the old global key after a change was not traced in this run. | A | `AR/13-threat-model.md:70` |
| 7 | No renewal on a date exists, so none can fail on a date. | A | `E/manifests/hub.yaml:157-161` `AR/_recovery-inventory-2026-07-28.md:501` (the code path of question 1) |
| 8 | DOC only. Two documents disagree on whether it was rotated: "rotated" at, "when last rotated" unknown at. Candidate for a doc fix. | A | `AR/13-threat-model.md:70` `AR/_recovery-inventory-2026-07-28.md:1030` |
**Hand step in a runbook?** yes — `RB/secrets.md:124`
### DH26 — Pinned tool versions in builds and CI · Version · MENTIONED
architecture folder: 8 lines in 2 files match `/security.watch|pinned by digest/` (a machine count, not a reading) · helper section: dates-hub „26. Pinned tool versions in builds and CI”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Versions of the tools that build and check the product. Operator and CC. Hub build image E/hub/Dockerfile:3, run image E/hub/Dockerfile:24, site publisher. | A | `E/scripts/publish-site-cloudflare.sh:28` |
| 2 | NO END DATE in code. Hub images are pinned by digest E/hub/Dockerfile:3; the ISO helper image is the moving tag at; the mailer image is `latest` at. | A | `E/scripts/iso/Dockerfile.assistant:10` `E/manifests/contact-mailer.yaml:54` |
| 3 | Whoever bumps the pin. The nightly security watch runs at 03:40 UTC; nightly gates at 03:20. | A | `E/.gitea/workflows/security-watch.yml:22` `E/.gitea/workflows/gates.yml:27` |
| 4 | A pinned digest removed upstream breaks the hub build for the operator; a moving tag changes the ISO build without a commit. | A | `E/scripts/publish-site-cloudflare.sh:28` (the code path of question 1) |
| 5 | The security watch treats a release line with no fixed version as end of life, it reports to the operator. | A | `E/scripts/security_watch.py:292` |
| 6 | Not a box matter. | A | `E/scripts/publish-site-cloudflare.sh:28` (the code path of question 1) |
| 7 | A CI job that does not run leaves no result. Whether a missed night run is seen is a question of the Clock list. | A | `E/.gitea/workflows/gates.yml:27` |
| 8 | SOURCE E/hub/Dockerfile:3 and. Support end dates of Go, Alpine, wrangler: no document states one. | A | `E/scripts/security_watch.py:292` |
**Hand step in a runbook?** no hand step
### DH27 — Domains and registrars · Account · MENTIONED
architecture folder: 4 lines in 3 files match `/registrar|domain renew/` (a machine count, not a reading) · helper section: dates-hub „27. Domains, registrars and the home dynamic-DNS name”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | NOT MEASURED. The product's two domains and the dynamic name the hub's address points to. Operator. Listed as row S13; the hub name is a CNAME to the dynamic name. | A | `RB/total-loss-of-dooplex.md:46` `RB/total-loss-of-dooplex.md:73` |
| 2 | NOT MEASURED (DooPlex, ep0 and the accounts are outside this run). No document gives a registrar or a renewal date for any of the product's names. This run could not read the public records: no whois tool here, and the public lookups for .eu and .hu gave no answer. | N | `EV/box/public-reads.txt` · class T3 · S06 |
| 3 | The operator, at the registrar. | A | `RB/total-loss-of-dooplex.md:46` `RB/total-loss-of-dooplex.md:73` (the code path of question 1) |
| 4 | If a domain or the dynamic name lapses, every box loses the hub (reports, alarms, updates, bind links) and the site and installer disappear. Every household. | A | `RB/total-loss-of-dooplex.md:46` `RB/total-loss-of-dooplex.md:73` (the code path of question 1) |
| 5 | Nobody that a document names. A name that ends takes down the hub address, the website, the mail and every household address under it. | N | `EV/box/public-reads.txt` · class T1 · S06 |
| 6 | A box comes back by itself once the name resolves again; while it does not, the first failing step is the hub lookup. | A | `RB/total-loss-of-dooplex.md:46` `RB/total-loss-of-dooplex.md:73` (the code path of question 1) |
| 7 | The renewal is the operator's hand step at the outside account. If it is not done, the thing ends (question 4). | A | `RB/total-loss-of-dooplex.md:46` `RB/total-loss-of-dooplex.md:73` (the code path of question 1) |
| 8 | DOC. WHERE THE OPERATOR READS THE DATE: NOT FOUND. NOBODY KNOWS from files. | A | `RB/total-loss-of-dooplex.md:46` |
**Hand step in a runbook?** NO
### DH28 — Cloudflare account, the site publish token, the ISO store keys · Account · MENTIONED
architecture folder: 3 lines in 3 files match `/CF_PAGES_TOKEN|R2 /` (a machine count, not a reading) · helper section: dates-hub „28. Cloudflare account, the site publish token and the ISO store keys”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | NOT MEASURED. The account that serves DNS for both domains, the site, the installer script and the ISO store. Operator. Row S11; site and ISO there since 2026-10-30. | A | `RB/total-loss-of-dooplex.md:44` `RB/total-loss-of-dooplex.md:125` |
| 2 | NOT MEASURED (DooPlex, ep0 and the accounts are outside this run). No document gives an end date for the account, for the publish token or for the store keys. | N | `E/scripts/publish-site-cloudflare.sh:12` · class T3 · S07 |
| 3 | The operator at the Cloudflare dashboard. | A | `RB/total-loss-of-dooplex.md:44` `RB/total-loss-of-dooplex.md:125` (the code path of question 1) |
| 4 | Token lapse: CC cannot publish the site or installer script; the old one stays served. Account trouble: DNS, site, ISO downloads and households' tunnels, per row S11. | A | `RB/total-loss-of-dooplex.md:44` |
| 5 | Nobody. | N | `E/scripts/publish-site-cloudflare.sh:12` · class T1 · S07 |
| 6 | A new box needs the installer script from the site at first boot; it retries until it is served. | A | `E/scripts/iso/felhom-bootstrap.sh:396` |
| 7 | A failed publish leaves the previous site live (candidate; not stated in the script header). | A | `E/scripts/publish-site-cloudflare.sh:12` |
| 8 | DOC. WHERE THE OPERATOR READS THE DATE: NOT FOUND. | A | `RB/total-loss-of-dooplex.md:44` |
**Hand step in a runbook?** NO
### DH29 — Hetzner account: the off-site endpoint and the storage boxes · Account · MENTIONED
architecture: `AR/_recovery-inventory-2026-07-28.md:1019` · helper section: dates-hub „29. Hetzner account, the rented endpoint and the storage boxes”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | NOT MEASURED. The paid account behind ep0 (tunnel endpoint and off-site store) and the Storage Boxes. Operator. Row S8; API token named in. | A | `RB/total-loss-of-dooplex.md:41` `E/manifests/hub.yaml:222-226` |
| 2 | NOT MEASURED (DooPlex, ep0 and the accounts are outside this run). No document gives a payment day, a contract end or the end of the API token. The recovery inventory asks the question and no document answers it. | N | `AR/_recovery-inventory-2026-07-28.md:1019` · class T3 · S07 |
| 3 | The operator; paid monthly by the provider's own cycle, not stated in any document read. | A | `RB/total-loss-of-dooplex.md:41` `E/manifests/hub.yaml:222-226` (the code path of question 1) |
| 4 | If the account is closed or unpaid: every household's off-site copies and the PBS tunnel stop; the hub cannot mint or reset sub-accounts. | A | `RB/total-loss-of-dooplex.md:41` `E/manifests/hub.yaml:222-226` (the code path of question 1) |
| 5 | Nobody before. After the end, the hub's 'off-site copy is old' alarm fires at 8 days. | N | `HUB/monitor/deadline_tiers.go:1` · class T1 · S07 |
| 6 | A box comes back by itself when the endpoint is back; first failing step is the off-site push. | A | `RB/total-loss-of-dooplex.md:41` `E/manifests/hub.yaml:222-226` (the code path of question 1) |
| 7 | The renewal is the operator's hand step at the outside account. If it is not done, the thing ends (question 4). | A | `RB/total-loss-of-dooplex.md:41` `E/manifests/hub.yaml:222-226` (the code path of question 1) |
| 8 | DOC. WHERE THE OPERATOR READS THE DATE: NOT FOUND. | A | `RB/total-loss-of-dooplex.md:41` |
**Hand step in a runbook?** NO
### DH30 — Tailscale node keys (the operator's remote way in) · Key · MENTIONED
architecture folder: 4 lines in 2 files match `/tailscale|tailnet/` (a machine count, not a reading) · helper section: dates-hub „30. Tailscale node keys (operator's remote way in)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | NOT MEASURED. The operator's private network to DooPlex and the two demo boxes. Operator. Table. | A | `E/documentation/operations/tailscale.md:11-13` |
| 2 | DOCUMENT: demo-hp's key ends 2027-01-17; the other two nodes have no end. MEASURED 2026-11-11: the Tailscale service is not running on demo-hp and Tester 1; it runs on the N100. So the date could not be read on the box. | A | `E/documentation/operations/tailscale.md:41-43` `EV/box/pin.txt` |
| 3 | The operator, in the admin console (Machines, Disable key expiry). No lead time set. | A | `E/documentation/operations/tailscale.md:43` |
| 4 | That node drops off the private network; the operator loses remote reach to it. No household is on it. | A | `E/documentation/operations/tailscale.md:91` |
| 5 | Nobody. The date is on the Machines page of the Tailscale console. | N | `E/documentation/operations/tailscale.md:41-43` · class T1 · S04 |
| 6 | The box keeps running; only the operator's remote path is gone until someone re-authenticates on site or over the LAN. | A | `E/documentation/operations/tailscale.md:11-13` (the code path of question 1) |
| 7 | The renewal is the operator's hand step at the outside account. If it is not done, the thing ends (question 4). | A | `E/documentation/operations/tailscale.md:11-13` (the code path of question 1) |
| 8 | DOC. Candidate: the section at says "both" nodes while the table has three. | A | `E/documentation/operations/tailscale.md:41` `E/documentation/operations/tailscale.md:89-92` |
**Hand step in a runbook?** yes — `E/documentation/operations/tailscale.md:41-43`
### DH31 — Gitea and registry credentials · Key · MENTIONED
architecture folder: 2 lines in 1 files match `/gitea.{0,40}(token|credential)/` (a machine count, not a reading) · helper section: dates-hub „31. Gitea and registry credentials”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | NOT MEASURED. The login the cluster and the operator use to pull and push images and code. Operator. Row S15; the pull secret is a scoped token. | A | `RB/total-loss-of-dooplex.md:48` `RB/secrets.md:231` |
| 2 | NOT MEASURED (DooPlex, ep0 and the accounts are outside this run). No document gives an end date for the scoped token or for CC's Gitea token. | N | `RB/secrets.md:1` · class T3 · S13 |
| 3 | The operator, in Gitea. | A | `RB/total-loss-of-dooplex.md:48` `RB/secrets.md:231` (the code path of question 1) |
| 4 | If the pull token lapses: the hub pod cannot be re-pulled on a restart, and boxes pulling the controller image through the same registry would fail (box helper's item). | A | `RB/total-loss-of-dooplex.md:48` `RB/secrets.md:231` (the code path of question 1) |
| 5 | Nobody. | N | `RB/secrets.md:1` · class T1 · S13 |
| 6 | Not a box matter: a box pulls its images with no login. | A | `RB/secrets.md:1` |
| 7 | The renewal is the operator's hand step at the outside account. If it is not done, the thing ends (question 4). | A | `RB/total-loss-of-dooplex.md:48` `RB/secrets.md:231` (the code path of question 1) |
| 8 | DOC and. WHERE THE OPERATOR READS THE DATE: NOT FOUND. | A | `RB/secrets.md:231` `RB/total-loss-of-dooplex.md:48` |
**Hand step in a runbook?** yes for a new token — `RB/secrets.md`
### DH32 — CC's secret store: the tokens inside · Key · MENTIONED
architecture folder: 3 lines in 2 files match `/cc-secrets|sec run/` (a machine count, not a reading) · helper section: dates-hub „32. CC's secret store: its key and the tokens inside (GitHub advisory token, hub token)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | NOT MEASURED. The store CC uses through `sec`, and the key that opens it. Operator owns; row S19; design in /mnt/5_hdd/felhom.eu/git/cc-secrets/ARCHITECTURE.md (lines 47-48 give the key location). | A | `RB/total-loss-of-dooplex.md:52` |
| 2 | The store holds a value and no end date. A token with an end at its issuer (the GitHub advisory token, the hub token) ends with no record here. | N | `E/scripts/security_watch.py:1` · class T3 · S14 |
| 3 | The operator stores a new value; only he may delete one. | A | `RB/total-loss-of-dooplex.md:52` (the code path of question 1) |
| 4 | An expired GitHub token: the nightly security watch loses its source. An expired hub token: block 22. | A | `E/.gitea/workflows/security-watch.yml:22` |
| 5 | Nobody. The nightly security watch and the release step fail on the day. | N | `E/scripts/security_watch.py:1` · class T1 · S14 |
| 6 | Not a box matter. | A | `RB/total-loss-of-dooplex.md:52` (the code path of question 1) |
| 7 | The renewal is the operator's hand step at the outside account. If it is not done, the thing ends (question 4). | A | `RB/total-loss-of-dooplex.md:52` (the code path of question 1) |
| 8 | DOC. WHERE THE OPERATOR READS THE DATES: NOT FOUND. NOBODY KNOWS from files. | A | `RB/total-loss-of-dooplex.md:52` |
**Hand step in a runbook?** NO
### DH33 — SSH keys between the hub, DooPlex and the endpoint · Key · MENTIONED
architecture folder: 5 lines in 4 files match `/ssh key|id_ed25519/` (a machine count, not a reading) · helper section: dates-hub „33. SSH keys between the hub, DooPlex and the endpoint”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | NOT MEASURED. Keys the hub uses to reach the endpoint (three key files named at,) and the root key, row S9. Operator. | A | `E/manifests/hub.yaml:191` `E/manifests/hub.yaml:202` `E/manifests/hub.yaml:209` `RB/total-loss-of-dooplex.md:42` |
| 2 | NO END DATE in any document. The only rule: rotate on an endpoint rebuild, DOC. | A | `RB/offsite-endpoint.md:313` |
| 3 | The operator, on rebuild. | A | `E/manifests/hub.yaml:191` `E/manifests/hub.yaml:202` (the code path of question 1) |
| 4 | A lost or changed key: the hub cannot add tunnel peers, mint off-site tokens or poke the endpoint; new households get no off-site tier. | A | `E/manifests/hub.yaml:191` `E/manifests/hub.yaml:202` (the code path of question 1) |
| 5 | Nobody; no date. | A | `E/manifests/hub.yaml:191` `E/manifests/hub.yaml:202` (the code path of question 1) |
| 6 | Existing boxes keep their own tunnel and token; only new provisioning fails. | A | `E/manifests/hub.yaml:191` `E/manifests/hub.yaml:202` (the code path of question 1) |
| 7 | No renewal on a date exists, so none can fail on a date. | A | `E/manifests/hub.yaml:191` `E/manifests/hub.yaml:202` (the code path of question 1) |
| 8 | DOC; row S9 says one authorized key's owner is "not recorded". NOBODY KNOWS who holds it. | A | `RB/offsite-endpoint.md:313` `RB/total-loss-of-dooplex.md:42` |
**Hand step in a runbook?** yes — `RB/offsite-endpoint.md:313`
@@ -0,0 +1,124 @@
# guest-dates demo-felhom taken 2026-11-11T10:06:04Z from DooPlex
=== now ===
2026-11-11T10:05:32Z
=== layout ===
stacks
actualbudget adventurelog audiobookshelf bentopdf bookstack calcom calibre-web claper cloudflared code-server crafty-controller dawarich docmost emby filebrowser ghost gitea glance gokapi grafana gramps-web grimmory grocy home-assistant homebox homepage immich jellyfin karakeep kimai komga lubelogger mealie metube n8n navidrome nextcloud nzbget onlyoffice opengist outline paperless-ngx papra plant-it plex privatebin prowlarr qbittorrent radarr radicale rallly recipe-importer romm sabnzbd seerr sonarr sparkyfitness tandoor termix traefik transmission uptime-kuma vaultwarden vikunja wanderer wger wishlist zipline
felhom-controller-data
=== certificate files (system paths and infra stacks only) ===
=== acme stores (certificate fields only) ===
--- /opt/docker/stacks/traefik/acme.json mtime=2026-10-02T19:50:44Z
letsencrypt *.demo-felhom.eu | subject=CN=*.demo-felhom.eu issuer=C=US, O=Let's Encrypt, CN=YR1 notBefore=Oct 2 18:52:11 2026 GMT notAfter=Dec 31 18:52:10 2026 GMT sha256 Fingerprint=7F:2C:1A:DB:15:29:2E:EC:B4:0F:84:B3:20:A2:59:88:29:33:4A:39:CD:1D:F5:B8:7F:2E:BC:98:FC:6C:96:95
=== served certificates (TLS handshake to own listeners) ===
*:22 -> no TLS
0.0.0.0:443 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 04:37:09 2026 GMT notAfter=Nov 11 04:37:09 2027 GMT sha256 Fingerprint=B2:11:42:FE:A7:ED:57:0B:A7:B1:70:7A:09:A1:C3:21:44:D2:8D:7A:A3:B9:D7:90:97:89:E7:B3:64:EE:61:AC
0.0.0.0:80 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 04:37:09 2026 GMT notAfter=Nov 11 04:37:09 2027 GMT sha256 Fingerprint=B2:11:42:FE:A7:ED:57:0B:A7:B1:70:7A:09:A1:C3:21:44:D2:8D:7A:A3:B9:D7:90:97:89:E7:B3:64:EE:61:AC
127.0.0.1:25 -> no TLS
[::1]:25 -> no TLS
[::]:443 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 04:37:09 2026 GMT notAfter=Nov 11 04:37:09 2027 GMT sha256 Fingerprint=B2:11:42:FE:A7:ED:57:0B:A7:B1:70:7A:09:A1:C3:21:44:D2:8D:7A:A3:B9:D7:90:97:89:E7:B3:64:EE:61:AC
[::]:80 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 04:37:09 2026 GMT notAfter=Nov 11 04:37:09 2027 GMT sha256 Fingerprint=B2:11:42:FE:A7:ED:57:0B:A7:B1:70:7A:09:A1:C3:21:44:D2:8D:7A:A3:B9:D7:90:97:89:E7:B3:64:EE:61:AC
=== os ===
Debian GNU/Linux 13 (trixie) / trixie
13.7
=== docker ===
client=29.9.3 server=29.9.3
containerd.io 2.4.2-3~debian.13~trixie
docker-ce 5:29.9.3-1~debian.13~trixie
docker-compose-plugin 5.7.0-1~debian.13~trixie
gitea.dooplex.hu/admin/felhom-controller:0.352.0
=== restic and tools inside the controller ===
restic 0.19.1 compiled with go1.26.4 on linux/amd64
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
NAME="Debian GNU/Linux"
VERSION_ID="13"
=== images: repo tag digest created ===
alpine:latest sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b 2026-06-16 00:01:29 +0000 UTC
cloudflare/cloudflared:2026.10.3 sha256:faa8e0a1475f137c910be6025f43c8e4092fdd35461b31e19fd1390657a1ff39 2026-10-21 13:26:38 +0000 UTC
cloudflare/cloudflared:2026.10.4 sha256:c6c7ae1d5dfdcb094ad344ed68ccffc26635c9c6be80965664c33baf064b0515 2026-10-26 11:27:24 +0000 UTC
gitea.dooplex.hu/admin/felhom-controller:0.351.0 sha256:1d90563c443022770b2db90a7730f1ea5aea45cc5b0fa64fec9823d60d42d97f 2026-11-09 12:14:14 +0000 UTC
gitea.dooplex.hu/admin/felhom-controller:0.352.0 sha256:3c785dd396b091f19d4c777db9d7a206e75ab018b6238b5d887e1a4c6cfd3b29 2026-11-10 15:16:20 +0000 UTC
gitea.dooplex.hu/admin/felhom-samba:1.1.0 sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10 2026-07-20 11:36:16 +0000 UTC
gtstef/filebrowser:1.5.6-stable sha256:7c5d7ac8ffda31294d278063cf9d2e04303b39e6dce1f4c691342240ca7703b8 2026-09-04 21:33:58 +0000 UTC
gtstef/filebrowser:1.5.8-stable sha256:68455d4953bad8e984e1bc5d112ed1e9c918a0a04f104c13c9ece50790308894 2026-10-04 21:15:40 +0000 UTC
hello-world:latest sha256:c3cbe1cc1aa588a64951ac6286e0df7b27fe2e6324b1001c619bb358770c0178 2026-03-23 21:33:59 +0000 UTC
traefik:v3.7.16 sha256:1336e85aaed293bd63d005086e0dd013b04078b26aaeb1c18b26a2f928af7e65 2026-10-21 22:55:46 +0000 UTC
traefik:v3.7.17 sha256:68ae5fe5b64ca166704b746a9e8afcd385e36b3f569a5691401bf652d57b3748 2026-10-30 20:17:26 +0000 UTC
=== apt sources ===
--- /etc/apt/sources.list.d/debian.sources
Types: deb
URIs: http://security.debian.org
Suites: trixie-security
Components: contrib main
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
Types: deb
URIs: http://deb.debian.org/debian
Suites: trixie trixie-updates
Components: contrib main
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
--- /etc/apt/sources.list.d/docker.list
deb [signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian trixie stable
=== apt keyring files ===
3817 2026-08-03 docker.asc
11861 2025-04-09 debian-archive-bookworm-automatic.asc
11873 2025-04-09 debian-archive-bookworm-security-automatic.asc
461 2025-04-09 debian-archive-bookworm-stable.asc
11861 2025-04-09 debian-archive-bullseye-automatic.asc
11873 2025-04-09 debian-archive-bullseye-security-automatic.asc
3403 2025-04-09 debian-archive-bullseye-stable.asc
11861 2025-04-09 debian-archive-trixie-automatic.asc
11873 2025-04-09 debian-archive-trixie-security-automatic.asc
1384 2025-04-09 debian-archive-trixie-stable.asc
37 2025-04-09 debian-archive-bookworm-automatic.gpg
8700 2025-04-09 debian-archive-bookworm-automatic.pgp
46 2025-04-09 debian-archive-bookworm-security-automatic.gpg
8709 2025-04-09 debian-archive-bookworm-security-automatic.pgp
34 2025-04-09 debian-archive-bookworm-stable.gpg
280 2025-04-09 debian-archive-bookworm-stable.pgp
37 2025-04-09 debian-archive-bullseye-automatic.gpg
8700 2025-04-09 debian-archive-bullseye-automatic.pgp
46 2025-04-09 debian-archive-bullseye-security-automatic.gpg
8709 2025-04-09 debian-archive-bullseye-security-automatic.pgp
34 2025-04-09 debian-archive-bullseye-stable.gpg
2453 2025-04-09 debian-archive-bullseye-stable.pgp
26 2025-04-09 debian-archive-keyring.gpg
55918 2025-04-09 debian-archive-keyring.pgp
31 2025-04-09 debian-archive-removed-keys.gpg
72636 2025-04-09 debian-archive-removed-keys.pgp
35 2025-04-09 debian-archive-trixie-automatic.gpg
8698 2025-04-09 debian-archive-trixie-automatic.pgp
44 2025-04-09 debian-archive-trixie-security-automatic.gpg
8707 2025-04-09 debian-archive-trixie-security-automatic.pgp
32 2025-04-09 debian-archive-trixie-stable.gpg
962 2025-04-09 debian-archive-trixie-stable.pgp
=== apt release files: Valid-Until ===
deb.debian.org_debian_dists_trixie-updates_InRelease | Date: Wed, 11 Nov 2026 02:14:24 UTC | Valid-Until: Wed, 18 Nov 2026 02:14:24 UTC
deb.debian.org_debian_dists_trixie_InRelease | Date: Sat, 12 Sep 2026 07:55:41 UTC | Valid-Until:
download.docker.com_linux_debian_dists_trixie_InRelease | Date: Fri, 06 Nov 2026 03:19:25 +0000 | Valid-Until:
security.debian.org_dists_trixie-security_InRelease | Date: Wed, 11 Nov 2026 00:23:32 UTC | Valid-Until: Wed, 18 Nov 2026 00:23:32 UTC
=== controller data folder (top level: names and dates only) ===
--- felhom-controller-data
total
-rw------- 2993 2026-09-01 08:32 controller.yaml
drwxr-xr-x 4096 2026-11-11 10:05 data
RC=0
@@ -0,0 +1,235 @@
# host-dates demo-felhom taken 2026-11-11T10:05:43Z from DooPlex
=== now ===
2026-11-11T10:05:10Z
=== certificate files ===
/etc/pve/local/pve-ssl.pem [certs=1 mtime=2026-07-18] subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-felhom.local issuer=CN=Proxmox Virtual Environment, OU=5bd8f1aa-d679-4c28-ba2a-d2d16cf957c3, O=PVE Cluster Manager CA notBefore=Jul 17 16:17:07 2026 GMT notAfter=Jul 16 16:17:07 2028 GMT sha256 Fingerprint=60:8F:4C:50:C3:8E:4F:F7:49:01:0A:70:6D:40:13:AE:18:C4:43:60:A8:5B:94:F4:6D:28:23:78:C6:E2:F1:78
/etc/pve/nodes/demo-felhom/pve-ssl.pem [certs=1 mtime=2026-07-18] subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-felhom.local issuer=CN=Proxmox Virtual Environment, OU=5bd8f1aa-d679-4c28-ba2a-d2d16cf957c3, O=PVE Cluster Manager CA notBefore=Jul 17 16:17:07 2026 GMT notAfter=Jul 16 16:17:07 2028 GMT sha256 Fingerprint=60:8F:4C:50:C3:8E:4F:F7:49:01:0A:70:6D:40:13:AE:18:C4:43:60:A8:5B:94:F4:6D:28:23:78:C6:E2:F1:78
/etc/pve/pve-root-ca.pem [certs=1 mtime=2026-07-18] subject=CN=Proxmox Virtual Environment, OU=5bd8f1aa-d679-4c28-ba2a-d2d16cf957c3, O=PVE Cluster Manager CA issuer=CN=Proxmox Virtual Environment, OU=5bd8f1aa-d679-4c28-ba2a-d2d16cf957c3, O=PVE Cluster Manager CA notBefore=Jul 17 16:17:07 2026 GMT notAfter=Jul 14 16:17:07 2036 GMT sha256 Fingerprint=53:17:92:A0:1E:4B:20:FA:7D:1F:26:AA:FE:C9:DC:9F:BD:37:66:32:12:0F:36:F7:61:06:EA:04:49:3E:E4:C7
/var/lib/felhom-agent/local-api.crt [certs=1 mtime=2026-08-03] subject=CN=felhom-agent local-api issuer=CN=felhom-agent local-api notBefore=Aug 3 06:15:30 2026 GMT notAfter=Jul 31 06:15:30 2036 GMT sha256 Fingerprint=C4:B9:6B:F0:F9:7E:FC:D4:62:B1:1D:33:BD:39:A2:81:33:B4:B3:64:85:35:EE:5B:49:7C:00:5E:AD:F3:E5:1E
=== system CA store ===
148
ca-certificates 20250419+deb13u1
=== served certificates (TLS handshake to own listeners) ===
0.0.0.0:111 -> no TLS
0.0.0.0:22 -> no TLS
0.0.0.0:8822 -> no TLS
100.70.170.35%tailscale0:36209 -> no TLS
[::]:111 -> no TLS
[::1]:25 -> no TLS
127.0.0.1:25 -> no TLS
127.0.0.1:53 -> no TLS
127.0.0.1:85 -> no TLS
169.254.253.1:8443 -> subject=CN=felhom-agent local-api issuer=CN=felhom-agent local-api notBefore=Aug 3 06:15:30 2026 GMT notAfter=Jul 31 06:15:30 2036 GMT sha256 Fingerprint=C4:B9:6B:F0:F9:7E:FC:D4:62:B1:1D:33:BD:39:A2:81:33:B4:B3:64:85:35:EE:5B:49:7C:00:5E:AD:F3:E5:1E
192.168.0.162:53 -> no TLS
[::]:22 -> no TLS
*:3128 -> no TLS
*:8006 -> subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-felhom.local issuer=CN=Proxmox Virtual Environment, OU=5bd8f1aa-d679-4c28-ba2a-d2d16cf957c3, O=PVE Cluster Manager CA notBefore=Jul 17 16:17:07 2026 GMT notAfter=Jul 16 16:17:07 2028 GMT sha256 Fingerprint=60:8F:4C:50:C3:8E:4F:F7:49:01:0A:70:6D:40:13:AE:18:C4:43:60:A8:5B:94:F4:6D:28:23:78:C6:E2:F1:78
[::]:8822 -> no TLS
[fd7a:115c:a1e0::5236:aa24]%tailscale0:61570 -> no TLS
=== listeners (port, process) ===
0.0.0.0:111 users:(("rpcbind",),("systemd",))
0.0.0.0:22 users:(("sshd",))
0.0.0.0:8822 users:(("sshd",))
100.70.170.35%tailscale0:36209 users:(("tailscaled",))
[::]:111 users:(("rpcbind",),("systemd",))
[::1]:25 users:(("smtpd",),("master",))
127.0.0.1:25 users:(("smtpd",),("master",))
127.0.0.1:53 users:(("dnsmasq",))
127.0.0.1:85 users:(("pvedaemon
169.254.253.1:8443 users:(("felhom-agent",))
192.168.0.162:53 users:(("dnsmasq",))
[::]:22 users:(("sshd",))
*:3128 users:(("spiceproxy
*:8006 users:(("pveproxy
[::]:8822 users:(("sshd",))
[fd7a:115c:a1e0::5236:aa24]%tailscale0:61570 users:(("tailscaled",))
=== pve users and tokens (expire: 0 = never) ===
user felhom-agent@pve enable 1 expire 0 tokens []
user root@pam enable 1 expire 0 tokens []
=== acme / subscription ===
message: There is no subscription key
status: notfound
=== os ===
Debian GNU/Linux 13 (trixie) / trixie
13.7
pve-manager/9.2.26/41a0e2e09b9e6f9d (running kernel: 7.0.14-28-pve)
7.0.14-28-pve
Manually selected kernels:
None.
Automatically selected kernels:
7.0.14-26-pve
7.0.14-28-pve
proxmox-kernel-7.0.14-20-pve-signed 7.0.14-20
proxmox-kernel-7.0.14-22-pve-signed 7.0.14-22
proxmox-kernel-7.0.14-23-pve-signed 7.0.14-23
proxmox-kernel-7.0.14-24-pve-signed 7.0.14-24
proxmox-kernel-7.0.14-25-pve-signed 7.0.14-25
proxmox-kernel-7.0.14-26-pve-signed 7.0.14-26
proxmox-kernel-7.0.14-28-pve-signed 7.0.14-28
proxmox-kernel-7.0.2-6-pve-signed 7.0.2-6
=== apt sources ===
--- /etc/apt/sources.list.d/ceph.sources
Types: deb
URIs: https://enterprise.proxmox.com/debian/ceph-squid
Suites: trixie
Components: enterprise
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
Enabled: no
--- /etc/apt/sources.list.d/debian.sources
Types: deb
URIs: http://deb.debian.org/debian/
Suites: trixie trixie-updates
Components: main contrib non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
Types: deb
URIs: http://security.debian.org/debian-security/
Suites: trixie-security
Components: main contrib non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
--- /etc/apt/sources.list.d/pve-enterprise.sources
Types: deb
URIs: https://enterprise.proxmox.com/debian/pve
Suites: trixie
Components: pve-enterprise
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
Enabled: no
--- /etc/apt/sources.list.d/pve-no-subscription.sources
Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
--- /etc/apt/sources.list.d/tailscale.list
deb [signed-by=/usr/share/keyrings/tailscale-archive-keyring.gpg] https://pkgs.tailscale.com/stable/debian trixie main
--- auth files (names only): /etc/apt/auth.conf.d:
=== apt keyring files ===
11861 2025-04-10 debian-archive-bookworm-automatic.asc
11873 2025-04-10 debian-archive-bookworm-security-automatic.asc
461 2025-04-10 debian-archive-bookworm-stable.asc
11861 2025-04-10 debian-archive-bullseye-automatic.asc
11873 2025-04-10 debian-archive-bullseye-security-automatic.asc
3403 2025-04-10 debian-archive-bullseye-stable.asc
11861 2025-04-10 debian-archive-trixie-automatic.asc
11873 2025-04-10 debian-archive-trixie-security-automatic.asc
1384 2025-04-10 debian-archive-trixie-stable.asc
1187 2022-11-27 proxmox-release-bookworm.gpg
1185 2024-11-10 proxmox-release-trixie.gpg
37 2025-04-10 debian-archive-bookworm-automatic.gpg
8700 2025-04-10 debian-archive-bookworm-automatic.pgp
46 2025-04-10 debian-archive-bookworm-security-automatic.gpg
8709 2025-04-10 debian-archive-bookworm-security-automatic.pgp
34 2025-04-10 debian-archive-bookworm-stable.gpg
280 2025-04-10 debian-archive-bookworm-stable.pgp
37 2025-04-10 debian-archive-bullseye-automatic.gpg
8700 2025-04-10 debian-archive-bullseye-automatic.pgp
46 2025-04-10 debian-archive-bullseye-security-automatic.gpg
8709 2025-04-10 debian-archive-bullseye-security-automatic.pgp
34 2025-04-10 debian-archive-bullseye-stable.gpg
2453 2025-04-10 debian-archive-bullseye-stable.pgp
26 2025-04-10 debian-archive-keyring.gpg
55918 2025-04-10 debian-archive-keyring.pgp
31 2025-04-10 debian-archive-removed-keys.gpg
72636 2025-04-10 debian-archive-removed-keys.pgp
35 2025-04-10 debian-archive-trixie-automatic.gpg
8698 2025-04-10 debian-archive-trixie-automatic.pgp
44 2025-04-10 debian-archive-trixie-security-automatic.gpg
8707 2025-04-10 debian-archive-trixie-security-automatic.pgp
32 2025-04-10 debian-archive-trixie-stable.gpg
962 2025-04-10 debian-archive-trixie-stable.pgp
2372 2025-07-09 proxmox-archive-keyring.gpg
113 2022-11-27 proxmox-offline-signing-key.pub
1187 2022-11-27 proxmox-release-bookworm.gpg
1185 2024-11-10 proxmox-release-trixie.gpg
2288 2023-01-23 tailscale-archive-keyring.gpg
=== apt release files: Valid-Until ===
deb.debian.org_debian_dists_trixie_InRelease | Date: Sat, 12 Sep 2026 07:55:41 UTC | Valid-Until:
deb.debian.org_debian_dists_trixie-updates_InRelease | Date: Wed, 11 Nov 2026 02:14:24 UTC | Valid-Until: Wed, 18 Nov 2026 02:14:24 UTC
download.proxmox.com_debian_pve_dists_trixie_InRelease | Date: Wed, 11 Nov 2026 01:41:17 +0000 | Valid-Until:
pkgs.tailscale.com_stable_debian_dists_trixie_InRelease | Date: Wed, 11 Nov 2026 00:00:32 UTC | Valid-Until:
security.debian.org_debian-security_dists_trixie-security_InRelease | Date: Wed, 11 Nov 2026 00:23:32 UTC | Valid-Until: Wed, 18 Nov 2026 00:23:32 UTC
=== templates and golden ===
/var/lib/vz 129954319 2026-07-14 debian-13-standard_13.6-1_amd64.tar.zst
=== agent files (names and dates only) ===
/etc/felhom:
total
-rw-r--r-- root 7 2026-07-18 18:17 appliance-pairing-code
-rw-r--r-- root 3897 2026-11-07 23:20 config-bundle.json
-rw-r--r-- root 456 2026-10-04 16:12 crash-guard.conf
-rw-r--r-- root 131 2026-10-04 16:12 operator-signers
-rw-r--r-- root 65 2026-10-04 18:34 os-trust.json
/etc/felhom-agent:
total
-rw------- felhom-agent 2467 2026-09-24 21:36 agent.json
-rw------- root 2467 2026-08-04 09:32 agent.json.bak-r191
-rw------- felhom-agent 2465 2026-07-27 11:31 agent.json.campaign8-before
-rw------- root 2465 2026-07-28 17:36 agent.json.campaign9-before
-rw------- root 2465 2026-07-27 11:31 agent.json.campaign9-prev
-rw------- felhom-agent 2514 2026-09-24 15:33 agent.json.night-0925-off
-rw------- felhom-agent 2465 2026-07-28 18:10 agent.json.pre-e-target-move
-rw------- felhom-agent 2464 2026-07-27 08:08 agent.json.pre-prunegate.bak
-rw------- root 2467 2026-08-08 20:04 agent.json.pre-r672
/var/lib/felhom-agent:
total
-rw------- felhom-agent 631 2026-11-11 05:37 backup-success-state.json
-rw------- felhom-agent 290 2026-11-10 16:16 controller-swap-9201.json
-rw------- felhom-agent 61 2026-08-03 09:15 drive-intents.json
drwx------ felhom-agent 4096 2026-10-03 17:03 escrow-stage
drwx------ felhom-agent 4096 2026-08-03 09:15 felhom-sshd
-rw------- felhom-agent 67 2026-10-14 15:29 guest-binds.json
-rw------- felhom-agent 243 2026-11-11 10:55 guest-disk-trim.json
drwx------ felhom-agent 4096 2026-08-03 09:17 guests
-rw------- felhom-agent 65682 2026-11-11 10:22 journal.log
-rw------- felhom-agent 305 2026-11-10 10:22 line-test.json
-rw-r--r-- felhom-agent 591 2026-08-03 09:15 local-api.crt
-rw------- felhom-agent 227 2026-08-03 09:15 local-api.key
-rw------- felhom-agent 164 2026-08-09 09:51 local-tokens.log
-rw------- felhom-agent 5508 2026-11-10 15:50 nonces.log
drwx------ felhom-agent 4096 2026-11-11 10:50 os
drwx------ felhom-agent 4096 2026-08-03 09:15 pbsdr
-rw------- felhom-agent 77 2026-10-13 20:52 poweroff.json
-rw------- felhom-agent 383 2026-11-11 10:22 restore-test-state.json
drwxr-x--- felhom-agent 4096 2026-11-10 15:51 selfupdate
-rw------- felhom-agent 40 2026-11-03 08:49 store-grant-seen.json
drwx------ felhom-agent 4096 2026-08-03 09:15 wg
=== agent ===
felhom-agent 0.173.0
=== wireguard (interface and latest handshake time only) ===
wg-felhom peer f3d1ZI… 2026-11-11T10:05:30Z
=== tailscale ===
self key expiry: None online: True
=== ssh host keys / authorized keys (counts only) ===
3
/root/.ssh/authorized_keys lines=3
=== smart: power-on hours and wear (no serials) ===
sda: 9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 5155;
sdb: 9 Power_On_Hours 0x0032 085 085 000 Old_age Always - 6093;
RC=0
@@ -0,0 +1,137 @@
# guest-dates demo-hp taken 2026-11-11T10:06:05Z from DooPlex
=== now ===
2026-11-11T10:06:06Z
=== layout ===
stacks
actualbudget adventurelog audiobookshelf bentopdf bookstack calcom calibre-web claper cloudflared code-server crafty-controller dawarich docmost emby filebrowser ghost gitea glance gokapi grafana gramps-web grimmory grocy home-assistant homebox homepage immich jellyfin karakeep kimai komga lubelogger mealie metube n8n navidrome nextcloud nzbget onlyoffice opengist outline paperless-ngx papra plant-it plex privatebin prowlarr qbittorrent radarr radicale rallly recipe-importer romm sabnzbd samba seerr sonarr sparkyfitness tandoor termix traefik transmission uptime-kuma vaultwarden vikunja wanderer wger wishlist zipline
felhom-controller-data
=== certificate files (system paths and infra stacks only) ===
=== acme stores (certificate fields only) ===
--- /opt/docker/stacks/traefik/acme.json mtime=2026-10-21T02:44:33Z
letsencrypt *.enkisfelhom.hu | subject=CN=*.enkisfelhom.hu issuer=C=US, O=Let's Encrypt, CN=YR2 notBefore=Oct 21 01:46:01 2026 GMT notAfter=Jan 19 01:46:00 2027 GMT sha256 Fingerprint=A1:C4:17:B6:34:2C:DC:FD:B2:A0:75:AA:22:EA:C1:E0:3B:CE:AA:8D:A9:CF:79:86:F2:A9:B5:55:F2:91:00:5C
=== served certificates (TLS handshake to own listeners) ===
*:22 -> no TLS
0.0.0.0:443 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 03:35:36 2026 GMT notAfter=Nov 11 03:35:36 2027 GMT sha256 Fingerprint=06:13:E1:5D:F0:97:57:20:45:3A:94:9D:4B:F3:A9:B4:13:91:07:89:72:E9:DD:9F:AA:9F:5F:9B:96:8D:D2:2D
0.0.0.0:80 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 03:35:36 2026 GMT notAfter=Nov 11 03:35:36 2027 GMT sha256 Fingerprint=06:13:E1:5D:F0:97:57:20:45:3A:94:9D:4B:F3:A9:B4:13:91:07:89:72:E9:DD:9F:AA:9F:5F:9B:96:8D:D2:2D
127.0.0.1:25 -> no TLS
[::1]:25 -> no TLS
[::]:443 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 03:35:36 2026 GMT notAfter=Nov 11 03:35:36 2027 GMT sha256 Fingerprint=06:13:E1:5D:F0:97:57:20:45:3A:94:9D:4B:F3:A9:B4:13:91:07:89:72:E9:DD:9F:AA:9F:5F:9B:96:8D:D2:2D
[::]:80 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 03:35:36 2026 GMT notAfter=Nov 11 03:35:36 2027 GMT sha256 Fingerprint=06:13:E1:5D:F0:97:57:20:45:3A:94:9D:4B:F3:A9:B4:13:91:07:89:72:E9:DD:9F:AA:9F:5F:9B:96:8D:D2:2D
=== os ===
Debian GNU/Linux 13 (trixie) / trixie
13.7
=== docker ===
client=29.9.3 server=29.9.3
containerd.io 2.4.2-3~debian.13~trixie
docker-ce 5:29.9.3-1~debian.13~trixie
docker-compose-plugin 5.7.0-1~debian.13~trixie
gitea.dooplex.hu/admin/felhom-controller:0.352.0
=== restic and tools inside the controller ===
restic 0.19.1 compiled with go1.26.4 on linux/amd64
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
NAME="Debian GNU/Linux"
VERSION_ID="13"
=== images: repo tag digest created ===
alpine:latest sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b 2026-06-16 00:01:29 +0000 UTC
cloudflare/cloudflared:2026.10.3 sha256:faa8e0a1475f137c910be6025f43c8e4092fdd35461b31e19fd1390657a1ff39 2026-10-21 13:26:38 +0000 UTC
cloudflare/cloudflared:2026.10.4 sha256:c6c7ae1d5dfdcb094ad344ed68ccffc26635c9c6be80965664c33baf064b0515 2026-10-26 11:27:24 +0000 UTC
crocodilestick/calibre-web-automated:v4.0.6 sha256:c31a738b6d5ec6982c050063dd3f063b6943eb1051fc81144789f840d9093a8d 2026-02-04 23:42:57 +0000 UTC
docmost/docmost:0.96.0 sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a 2026-09-08 15:49:25 +0000 UTC
ghcr.io/alam00000/bentopdf:v2.8.6 sha256:eaeea1e447205a79cb61d7efdc6966f37311dc1bc9c36a3a5c897bf79107c2c3 2026-06-28 06:30:01 +0000 UTC
ghcr.io/paperless-ngx/paperless-ngx:2.20.15 sha256:6c86cad803970ea782683a8e80e7403444c5bf3cf70de63b4d3c8e87500db92f 2026-04-27 02:45:05 +0000 UTC
ghcr.io/thomiceli/opengist:1.15 sha256:7edc91273ee7d10a17406da8a08c803158baaff984c39b789c1313473d068f21 2026-08-30 02:07:48 +0000 UTC
gitea.dooplex.hu/admin/felhom-controller:0.351.0 sha256:1d90563c443022770b2db90a7730f1ea5aea45cc5b0fa64fec9823d60d42d97f 2026-11-09 12:14:14 +0000 UTC
gitea.dooplex.hu/admin/felhom-controller:0.352.0 sha256:3c785dd396b091f19d4c777db9d7a206e75ab018b6238b5d887e1a4c6cfd3b29 2026-11-10 15:16:20 +0000 UTC
gitea.dooplex.hu/admin/felhom-samba:1.2.0 sha256:da96f25033827fbb77d0ee398e86fe5b744e2f29be9de898994533340c97acf9 2026-10-16 18:25:34 +0000 UTC
gtstef/filebrowser:1.5.6-stable sha256:7c5d7ac8ffda31294d278063cf9d2e04303b39e6dce1f4c691342240ca7703b8 2026-09-04 21:33:58 +0000 UTC
gtstef/filebrowser:1.5.8-stable sha256:68455d4953bad8e984e1bc5d112ed1e9c918a0a04f104c13c9ece50790308894 2026-10-04 21:15:40 +0000 UTC
mariadb:11.8 sha256:79d59758afc91b89b120b0a8904d637f5a3b3e1c4900f29b740d6d46c72fef68 2026-09-16 03:26:24 +0000 UTC
mariadb:12.3 sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 2026-09-16 03:26:24 +0000 UTC
plexinc/pms-docker:1.41.4.9463-630c9f557 sha256:b3a376a04fa1912af17a61755089ad2324502fe1b2f8bbf042eab31afe36fc25 2025-02-07 10:23:03 +0000 UTC
plexinc/pms-docker:1.43.5.11029-6bf8af3f0 sha256:01bdf0ad6d53198da4331540c1e73ab65db7a72e961fff97ecb0847d46055c5f 2026-10-13 07:57:12 +0000 UTC
postgis/postgis:16-3.5-alpine sha256:47e961a569fd52ff31f0fe205ed91eeab17d9f5fff6722e6d7ea6b588748b293 2026-08-31 11:36:26 +0000 UTC
privatebin/pdo:2.0.6 sha256:4c141b2326f8b353598ce9ce7507a9cfecf2dad5c60a39fea903d430e296d8f5 2026-08-08 06:24:54 +0000 UTC
redis:7-alpine sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 2026-09-17 21:32:46 +0000 UTC
traefik:v3.7.16 sha256:1336e85aaed293bd63d005086e0dd013b04078b26aaeb1c18b26a2f928af7e65 2026-10-21 22:55:46 +0000 UTC
traefik:v3.7.17 sha256:68ae5fe5b64ca166704b746a9e8afcd385e36b3f569a5691401bf652d57b3748 2026-10-30 20:17:26 +0000 UTC
=== apt sources ===
--- /etc/apt/sources.list.d/debian.sources
Types: deb
URIs: http://security.debian.org
Suites: trixie-security
Components: contrib main
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
Types: deb
URIs: http://deb.debian.org/debian
Suites: trixie trixie-updates
Components: contrib main
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
--- /etc/apt/sources.list.d/docker.list
deb [signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian trixie stable
=== apt keyring files ===
3817 2026-08-18 docker.asc
11861 2025-04-09 debian-archive-bookworm-automatic.asc
11873 2025-04-09 debian-archive-bookworm-security-automatic.asc
461 2025-04-09 debian-archive-bookworm-stable.asc
11861 2025-04-09 debian-archive-bullseye-automatic.asc
11873 2025-04-09 debian-archive-bullseye-security-automatic.asc
3403 2025-04-09 debian-archive-bullseye-stable.asc
11861 2025-04-09 debian-archive-trixie-automatic.asc
11873 2025-04-09 debian-archive-trixie-security-automatic.asc
1384 2025-04-09 debian-archive-trixie-stable.asc
37 2025-04-09 debian-archive-bookworm-automatic.gpg
8700 2025-04-09 debian-archive-bookworm-automatic.pgp
46 2025-04-09 debian-archive-bookworm-security-automatic.gpg
8709 2025-04-09 debian-archive-bookworm-security-automatic.pgp
34 2025-04-09 debian-archive-bookworm-stable.gpg
280 2025-04-09 debian-archive-bookworm-stable.pgp
37 2025-04-09 debian-archive-bullseye-automatic.gpg
8700 2025-04-09 debian-archive-bullseye-automatic.pgp
46 2025-04-09 debian-archive-bullseye-security-automatic.gpg
8709 2025-04-09 debian-archive-bullseye-security-automatic.pgp
34 2025-04-09 debian-archive-bullseye-stable.gpg
2453 2025-04-09 debian-archive-bullseye-stable.pgp
26 2025-04-09 debian-archive-keyring.gpg
55918 2025-04-09 debian-archive-keyring.pgp
31 2025-04-09 debian-archive-removed-keys.gpg
72636 2025-04-09 debian-archive-removed-keys.pgp
35 2025-04-09 debian-archive-trixie-automatic.gpg
8698 2025-04-09 debian-archive-trixie-automatic.pgp
44 2025-04-09 debian-archive-trixie-security-automatic.gpg
8707 2025-04-09 debian-archive-trixie-security-automatic.pgp
32 2025-04-09 debian-archive-trixie-stable.gpg
962 2025-04-09 debian-archive-trixie-stable.pgp
=== apt release files: Valid-Until ===
deb.debian.org_debian_dists_trixie-updates_InRelease | Date: Wed, 11 Nov 2026 02:14:24 UTC | Valid-Until: Wed, 18 Nov 2026 02:14:24 UTC
deb.debian.org_debian_dists_trixie_InRelease | Date: Sat, 12 Sep 2026 07:55:41 UTC | Valid-Until:
download.docker.com_linux_debian_dists_trixie_InRelease | Date: Fri, 06 Nov 2026 03:19:25 +0000 | Valid-Until:
security.debian.org_dists_trixie-security_InRelease | Date: Wed, 11 Nov 2026 00:23:32 UTC | Valid-Until: Wed, 18 Nov 2026 00:23:32 UTC
=== controller data folder (top level: names and dates only) ===
--- felhom-controller-data
total
-rw------- 2989 2026-08-31 08:32 controller.yaml
-rw------- 2989 2026-08-31 08:32 controller.yaml.pre-gate-day
-rw------- 2989 2026-08-31 08:32 controller.yaml.pre-undofleet
drwxr-xr-x 4096 2026-11-11 10:06 data
RC=0
@@ -0,0 +1,218 @@
# host-dates demo-hp taken 2026-11-11T10:05:43Z from DooPlex
=== now ===
2026-11-11T10:05:43Z
=== certificate files ===
/etc/pve/local/pve-ssl.pem [certs=1 mtime=2026-08-21] subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA notBefore=Aug 20 15:44:42 2026 GMT notAfter=Aug 19 15:44:42 2028 GMT sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB
/etc/pve/nodes/demo-hp/pve-ssl.pem [certs=1 mtime=2026-08-21] subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA notBefore=Aug 20 15:44:42 2026 GMT notAfter=Aug 19 15:44:42 2028 GMT sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB
/etc/pve/pve-root-ca.pem [certs=1 mtime=2026-08-21] subject=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA notBefore=Aug 20 15:44:42 2026 GMT notAfter=Aug 17 15:44:42 2036 GMT sha256 Fingerprint=E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7
/var/lib/felhom-agent/local-api.crt [certs=1 mtime=2026-08-21] subject=CN=felhom-agent local-api issuer=CN=felhom-agent local-api notBefore=Aug 21 14:59:12 2026 GMT notAfter=Aug 18 14:59:12 2036 GMT sha256 Fingerprint=34:12:2A:C2:AC:E9:91:68:5A:D9:AD:AA:F8:82:70:25:81:AE:93:18:37:20:3D:BE:B2:5F:0D:9E:49:3A:83:7D
=== system CA store ===
148
ca-certificates 20250419+deb13u1
=== served certificates (TLS handshake to own listeners) ===
0.0.0.0:111 -> no TLS
0.0.0.0:22 -> no TLS
0.0.0.0:8822 -> no TLS
[::]:111 -> no TLS
[::1]:25 -> no TLS
127.0.0.1:25 -> no TLS
127.0.0.1:53 -> no TLS
127.0.0.1:85 -> no TLS
169.254.253.1:8443 -> subject=CN=felhom-agent local-api issuer=CN=felhom-agent local-api notBefore=Aug 21 14:59:12 2026 GMT notAfter=Aug 18 14:59:12 2036 GMT sha256 Fingerprint=34:12:2A:C2:AC:E9:91:68:5A:D9:AD:AA:F8:82:70:25:81:AE:93:18:37:20:3D:BE:B2:5F:0D:9E:49:3A:83:7D
192.168.0.104:53 -> no TLS
[::]:22 -> no TLS
*:3128 -> no TLS
*:8006 -> subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA notBefore=Aug 20 15:44:42 2026 GMT notAfter=Aug 19 15:44:42 2028 GMT sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB
[::]:8822 -> no TLS
=== listeners (port, process) ===
0.0.0.0:111 users:(("rpcbind",),("systemd",))
0.0.0.0:22 users:(("sshd",))
0.0.0.0:8822 users:(("sshd",))
[::]:111 users:(("rpcbind",),("systemd",))
[::1]:25 users:(("smtpd",),("master",))
127.0.0.1:25 users:(("smtpd",),("master",))
127.0.0.1:53 users:(("dnsmasq",))
127.0.0.1:85 users:(("pvedaemon
169.254.253.1:8443 users:(("felhom-agent",))
192.168.0.104:53 users:(("dnsmasq",))
[::]:22 users:(("sshd",))
*:3128 users:(("spiceproxy
*:8006 users:(("pveproxy
[::]:8822 users:(("sshd",))
=== pve users and tokens (expire: 0 = never) ===
user felhom-agent@pve enable 1 expire 0 tokens []
user root@pam enable 1 expire 0 tokens []
=== acme / subscription ===
message: There is no subscription key
status: notfound
=== os ===
Debian GNU/Linux 13 (trixie) / trixie
13.7
pve-manager/9.2.26/41a0e2e09b9e6f9d (running kernel: 7.0.14-28-pve)
7.0.14-28-pve
Manually selected kernels:
None.
Automatically selected kernels:
7.0.14-26-pve
7.0.14-28-pve
proxmox-kernel-7.0.14-26-pve-signed 7.0.14-26
proxmox-kernel-7.0.14-28-pve-signed 7.0.14-28
=== apt sources ===
--- /etc/apt/sources.list.d/ceph.sources
Types: deb
URIs: https://enterprise.proxmox.com/debian/ceph-squid
Suites: trixie
Components: enterprise
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
Enabled: no
--- /etc/apt/sources.list.d/debian.sources
Types: deb
URIs: http://deb.debian.org/debian/
Suites: trixie trixie-updates
Components: main contrib non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
Types: deb
URIs: http://security.debian.org/debian-security/
Suites: trixie-security
Components: main contrib non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
--- /etc/apt/sources.list.d/pve-enterprise.sources
Types: deb
URIs: https://enterprise.proxmox.com/debian/pve
Suites: trixie
Components: pve-enterprise
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
Enabled: no
--- /etc/apt/sources.list.d/pve-no-subscription.sources
Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
--- auth files (names only): /etc/apt/auth.conf.d:
=== apt keyring files ===
11861 2025-04-10 debian-archive-bookworm-automatic.asc
11873 2025-04-10 debian-archive-bookworm-security-automatic.asc
461 2025-04-10 debian-archive-bookworm-stable.asc
11861 2025-04-10 debian-archive-bullseye-automatic.asc
11873 2025-04-10 debian-archive-bullseye-security-automatic.asc
3403 2025-04-10 debian-archive-bullseye-stable.asc
11861 2025-04-10 debian-archive-trixie-automatic.asc
11873 2025-04-10 debian-archive-trixie-security-automatic.asc
1384 2025-04-10 debian-archive-trixie-stable.asc
1187 2022-11-27 proxmox-release-bookworm.gpg
1185 2024-11-10 proxmox-release-trixie.gpg
37 2025-04-10 debian-archive-bookworm-automatic.gpg
8700 2025-04-10 debian-archive-bookworm-automatic.pgp
46 2025-04-10 debian-archive-bookworm-security-automatic.gpg
8709 2025-04-10 debian-archive-bookworm-security-automatic.pgp
34 2025-04-10 debian-archive-bookworm-stable.gpg
280 2025-04-10 debian-archive-bookworm-stable.pgp
37 2025-04-10 debian-archive-bullseye-automatic.gpg
8700 2025-04-10 debian-archive-bullseye-automatic.pgp
46 2025-04-10 debian-archive-bullseye-security-automatic.gpg
8709 2025-04-10 debian-archive-bullseye-security-automatic.pgp
34 2025-04-10 debian-archive-bullseye-stable.gpg
2453 2025-04-10 debian-archive-bullseye-stable.pgp
26 2025-04-10 debian-archive-keyring.gpg
55918 2025-04-10 debian-archive-keyring.pgp
31 2025-04-10 debian-archive-removed-keys.gpg
72636 2025-04-10 debian-archive-removed-keys.pgp
35 2025-04-10 debian-archive-trixie-automatic.gpg
8698 2025-04-10 debian-archive-trixie-automatic.pgp
44 2025-04-10 debian-archive-trixie-security-automatic.gpg
8707 2025-04-10 debian-archive-trixie-security-automatic.pgp
32 2025-04-10 debian-archive-trixie-stable.gpg
962 2025-04-10 debian-archive-trixie-stable.pgp
2372 2025-07-09 proxmox-archive-keyring.gpg
113 2022-11-27 proxmox-offline-signing-key.pub
1187 2022-11-27 proxmox-release-bookworm.gpg
1185 2024-11-10 proxmox-release-trixie.gpg
=== apt release files: Valid-Until ===
deb.debian.org_debian_dists_trixie_InRelease | Date: Sat, 12 Sep 2026 07:55:41 UTC | Valid-Until:
deb.debian.org_debian_dists_trixie-updates_InRelease | Date: Wed, 11 Nov 2026 02:14:24 UTC | Valid-Until: Wed, 18 Nov 2026 02:14:24 UTC
download.proxmox.com_debian_pve_dists_trixie_InRelease | Date: Wed, 11 Nov 2026 01:41:17 +0000 | Valid-Until:
security.debian.org_debian-security_dists_trixie-security_InRelease | Date: Wed, 11 Nov 2026 00:23:32 UTC | Valid-Until: Wed, 18 Nov 2026 00:23:32 UTC
=== templates and golden ===
/var/lib/vz 129954319 2026-07-14 debian-13-standard_13.6-1_amd64.tar.zst
=== agent files (names and dates only) ===
/etc/felhom:
total
-rw-r--r-- root 7 2026-08-21 17:44 appliance-pairing-code
-rw-r--r-- root 3897 2026-11-07 23:27 config-bundle.json
-rw-r--r-- root 456 2026-10-04 16:08 crash-guard.conf
-rw-r--r-- root 131 2026-10-04 16:08 operator-signers
-rw-r--r-- root 61 2026-10-04 18:34 os-trust.json
/etc/felhom-agent:
total
-rw------- felhom-agent 2454 2026-09-28 16:25 agent.json
-rw------- felhom-agent 2498 2026-09-24 15:33 agent.json.night-0925-off
-rw------- felhom-agent 2451 2026-09-24 21:56 agent.json.pre-a4-retention
-rw------- felhom-agent 2451 2026-09-24 21:57 agent.json.pre-d44
-rw------- root 2451 2026-08-21 18:09 agent.json.pre-r672
/var/lib/felhom-agent:
total
-rw------- felhom-agent 593 2026-11-11 04:44 backup-success-state.json
-rw------- felhom-agent 290 2026-11-10 16:17 controller-swap-9201.json
-rw------- felhom-agent 126 2026-10-17 16:23 drive-intents.json
drwx------ felhom-agent 4096 2026-10-19 03:48 escrow-stage
drwx------ felhom-agent 4096 2026-08-21 17:59 felhom-sshd
-rw------- felhom-agent 301 2026-10-19 03:49 format-job.json
-rw------- felhom-agent 67 2026-10-17 17:08 guest-binds.json
-rw------- felhom-agent 241 2026-11-11 10:03 guest-disk-trim.json
drwx------ felhom-agent 4096 2026-09-13 22:16 guests
-rw------- felhom-agent 36314 2026-11-11 04:31 journal.log
-rw------- felhom-agent 305 2026-11-10 11:36 line-test.json
-rw-r--r-- felhom-agent 591 2026-08-21 17:59 local-api.crt
-rw------- felhom-agent 227 2026-08-21 17:59 local-api.key
-rw------- felhom-agent 82 2026-08-21 18:00 local-tokens.log
-rw------- felhom-agent 6120 2026-11-10 15:58 nonces.log
drwx------ felhom-agent 4096 2026-11-11 10:58 os
drwx------ felhom-agent 4096 2026-08-21 18:09 pbsdr
-rw------- felhom-agent 367 2026-11-11 04:31 restore-test-state.json
drwxr-x--- felhom-agent 4096 2026-11-10 15:59 selfupdate
-rw------- felhom-agent 32 2026-11-03 08:56 store-grant-seen.json
drwx------ felhom-agent 4096 2026-10-17 16:11 units
drwx------ felhom-agent 4096 2026-08-21 17:59 wg
=== agent ===
felhom-agent 0.173.0
=== wireguard (interface and latest handshake time only) ===
wg-felhom peer f3d1ZI… 2026-11-11T10:05:07Z
=== tailscale ===
no tailscale
=== ssh host keys / authorized keys (counts only) ===
3
/root/.ssh/authorized_keys lines=2
=== smart: power-on hours and wear (no serials) ===
sda: 9 Power_On_Hours 0x0032 100 100 --- Old_age Always - 32293;177 Wear_Leveling_Count 0x0032 100 100 --- Old_age Always - 33;
nvme0n1: Available Spare: 100%;Percentage Used: 8%;Power On Hours: 30,963;
RC=0
@@ -0,0 +1,10 @@
# python3 scripts/golden_currency_gate.py, run on DooPlex 2026-11-11T10:27:34Z (reads the clone only)
newest released controller : 0.352.0 (## v0.352.0 (2026-11-10) — R-1371: one fill warning per filesystem — RELEASED (image `sha2)
newest golden baked : 0.350.0 (documentation/tests/golden-0.350.0-2026-11-03 [sha 082e2e040de5…])
golden age : 8 day(s) (baked 2026-11-03; the limit is 7 — `09` §3 decision 250)
GOLDEN CURRENCY GATE FAILED: controller v0.352.0 is released and the newest golden (0.350.0) is 8 day(s) old — more than 7 (decision 250: the golden is baked weekly; 2 release(s) behind).
A machine installed right now would receive v0.350.0 — the release is written, tested and pushed, and NOT delivered.
Fix: bake the weekly golden at the floor's version per documentation/runbooks/RUNBOOK-manual-build.md §4.1 (runbooks/nightly-rotation.md, Sunday) and put „vouch golden X" on the operator's morning list — the vouch is the operator's (a THREE-field change: golden_version + agent_version + min_agent).
If the cadence ruling covers this release, issue a DATED waiver at documentation/tests/golden-waiver.yml (RUNBOOK-manual-build.md §4.2) — never a bypass.
RC=1
@@ -0,0 +1,798 @@
=== demo-hp host: keyring files 26
etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | uid Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21
etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | uid Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>
etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | uid Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11
etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | uid Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>
etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | uid Debian Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22
etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | uid Debian Stable Release Key (13/trixie) <debian-release@lists.debian.org>
etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg | pub 1140AF8F639E0C39 algo=1 bits=4096 | created 2022-11-27 | expires 2032-11-24
etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg | uid Proxmox Bookworm Release Key <proxmox-release@proxmox.com>
etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg | pub A7BCD1420BFE778E algo=1 bits=4096 | created 2024-11-10 | expires 2034-11-10
etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg | uid Proxmox Trixie Release Key <proxmox-release@proxmox.com>
usr/share/keyrings/debian-archive-bookworm-automatic.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-automatic.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bookworm-automatic.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-stable.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21
usr/share/keyrings/debian-archive-bookworm-stable.pgp | uid Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-bullseye-automatic.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-automatic.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bullseye-automatic.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-stable.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11
usr/share/keyrings/debian-archive-bullseye-stable.pgp | uid Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (13/trixie) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FFA8EF91DB114E0 algo=1 bits=1024 | created 2004-01-15 | expires 2005-01-27
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2004) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub F1D53D8C4F368D5D algo=17 bits=1024 | created 2005-01-31 | expires 2006-01-31
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2005) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub E415B2B4B5F5BBED algo=17 bits=1024 | created 2005-04-24 | expires never
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian AMD64 Archive Key <debian-amd64@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub B7A50B4134FC6FE5 algo=16 bits=2048 | created 2005-04-24 | expires never
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 010908312D230C5F algo=17 bits=1024 | created 2006-01-03 | expires 2007-02-07
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2006) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub A70DAF536070D3A1 algo=17 bits=1024 | created 2006-11-20 | expires 2009-07-01
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (4.0/etch) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub B5D0C804ADB11277 algo=17 bits=1024 | created 2006-09-17 | expires never
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Etch Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub EC61E0B0BBE55AB3 algo=17 bits=1024 | created 2007-03-31 | expires 2010-03-30
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (4.0/etch)
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 0A3B614236CA98F3 algo=16 bits=2048 | created 2007-03-31 | expires 2010-03-30
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9AA38DCD55BE302B algo=1 bits=4096 | created 2009-01-27 | expires 2012-12-31
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (5.0/lenny) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4D270D06F42584E6 algo=17 bits=1024 | created 2008-04-06 | expires 2012-05-15
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Lenny Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub DFD993306D849617 algo=1 bits=2048 | created 2009-01-24 | expires 2013-01-23
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (5.0/lenny)
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 64481591B98321F9 algo=1 bits=4096 | created 2010-08-07 | expires 2017-08-05
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Squeeze Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub AED4B06F473041FA algo=1 bits=4096 | created 2010-08-27 | expires 2018-03-05
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (6.0/squeeze) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 8B48AD6246925553 algo=1 bits=4096 | created 2012-04-27 | expires 2020-04-25
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (7.0/wheezy) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FB2A1C265FFB764 algo=1 bits=4096 | created 2012-05-08 | expires 2019-05-07
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Wheezy Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub CBF8D6FD518E17E1 algo=1 bits=4096 | created 2013-08-17 | expires 2021-08-15
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Jessie Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 7638D0442B90D010 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9D6D8F6BC857C906 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub EF0F382A1A7B6500 algo=1 bits=4096 | created 2017-05-20 | expires 2025-05-18
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (9/stretch) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub E0B11894F66AEC98 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 04EE7237B7D453EC algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | pub EDA0D2388AE22BA9 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub AA8E81B4331F7F50 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | pub DCC9EFBF77E11517 algo=1 bits=4096 | created 2019-02-05 | expires 2027-02-03
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (10/buster) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub DC30D7C23CBBABEE algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 648ACFD622F3D138 algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4DFAB270CAA96DFA algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 112695A0E562B32A algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-trixie-automatic.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-automatic.pgp | uid Debian Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-trixie-automatic.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-stable.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22
usr/share/keyrings/debian-archive-trixie-stable.pgp | uid Debian Stable Release Key (13/trixie) <debian-release@lists.debian.org>
usr/share/keyrings/proxmox-archive-keyring.gpg | pub 1140AF8F639E0C39 algo=1 bits=4096 | created 2022-11-27 | expires 2032-11-24
usr/share/keyrings/proxmox-archive-keyring.gpg | uid Proxmox Bookworm Release Key <proxmox-release@proxmox.com>
usr/share/keyrings/proxmox-archive-keyring.gpg | pub A7BCD1420BFE778E algo=1 bits=4096 | created 2024-11-10 | expires 2034-11-10
usr/share/keyrings/proxmox-archive-keyring.gpg | uid Proxmox Trixie Release Key <proxmox-release@proxmox.com>
usr/share/keyrings/proxmox-release-bookworm.gpg | pub 1140AF8F639E0C39 algo=1 bits=4096 | created 2022-11-27 | expires 2032-11-24
usr/share/keyrings/proxmox-release-bookworm.gpg | uid Proxmox Bookworm Release Key <proxmox-release@proxmox.com>
usr/share/keyrings/proxmox-release-trixie.gpg | pub A7BCD1420BFE778E algo=1 bits=4096 | created 2024-11-10 | expires 2034-11-10
usr/share/keyrings/proxmox-release-trixie.gpg | uid Proxmox Trixie Release Key <proxmox-release@proxmox.com>
=== demo-hp guest: keyring files 21
etc/apt/keyrings/docker.asc | pub 8D81803C0EBFCD88 algo=1 bits=4096 | created 2017-02-22 | expires never
etc/apt/keyrings/docker.asc | uid Docker Release (CE deb) <docker@docker.com>
etc/apt/keyrings/docker.asc | sub 7EA0A9C3F273FCD8 algo=1 bits=4096 | created 2017-02-22 | expires never
etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | uid Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21
etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | uid Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>
etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | uid Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11
etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | uid Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>
etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | uid Debian Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22
etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | uid Debian Stable Release Key (13/trixie) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-bookworm-automatic.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-automatic.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bookworm-automatic.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-stable.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21
usr/share/keyrings/debian-archive-bookworm-stable.pgp | uid Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-bullseye-automatic.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-automatic.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bullseye-automatic.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-stable.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11
usr/share/keyrings/debian-archive-bullseye-stable.pgp | uid Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (13/trixie) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FFA8EF91DB114E0 algo=1 bits=1024 | created 2004-01-15 | expires 2005-01-27
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2004) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub F1D53D8C4F368D5D algo=17 bits=1024 | created 2005-01-31 | expires 2006-01-31
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2005) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub E415B2B4B5F5BBED algo=17 bits=1024 | created 2005-04-24 | expires never
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian AMD64 Archive Key <debian-amd64@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub B7A50B4134FC6FE5 algo=16 bits=2048 | created 2005-04-24 | expires never
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 010908312D230C5F algo=17 bits=1024 | created 2006-01-03 | expires 2007-02-07
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2006) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub A70DAF536070D3A1 algo=17 bits=1024 | created 2006-11-20 | expires 2009-07-01
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (4.0/etch) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub B5D0C804ADB11277 algo=17 bits=1024 | created 2006-09-17 | expires never
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Etch Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub EC61E0B0BBE55AB3 algo=17 bits=1024 | created 2007-03-31 | expires 2010-03-30
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (4.0/etch)
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 0A3B614236CA98F3 algo=16 bits=2048 | created 2007-03-31 | expires 2010-03-30
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9AA38DCD55BE302B algo=1 bits=4096 | created 2009-01-27 | expires 2012-12-31
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (5.0/lenny) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4D270D06F42584E6 algo=17 bits=1024 | created 2008-04-06 | expires 2012-05-15
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Lenny Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub DFD993306D849617 algo=1 bits=2048 | created 2009-01-24 | expires 2013-01-23
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (5.0/lenny)
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 64481591B98321F9 algo=1 bits=4096 | created 2010-08-07 | expires 2017-08-05
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Squeeze Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub AED4B06F473041FA algo=1 bits=4096 | created 2010-08-27 | expires 2018-03-05
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (6.0/squeeze) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 8B48AD6246925553 algo=1 bits=4096 | created 2012-04-27 | expires 2020-04-25
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (7.0/wheezy) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FB2A1C265FFB764 algo=1 bits=4096 | created 2012-05-08 | expires 2019-05-07
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Wheezy Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub CBF8D6FD518E17E1 algo=1 bits=4096 | created 2013-08-17 | expires 2021-08-15
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Jessie Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 7638D0442B90D010 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9D6D8F6BC857C906 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub EF0F382A1A7B6500 algo=1 bits=4096 | created 2017-05-20 | expires 2025-05-18
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (9/stretch) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub E0B11894F66AEC98 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 04EE7237B7D453EC algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | pub EDA0D2388AE22BA9 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub AA8E81B4331F7F50 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | pub DCC9EFBF77E11517 algo=1 bits=4096 | created 2019-02-05 | expires 2027-02-03
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (10/buster) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub DC30D7C23CBBABEE algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 648ACFD622F3D138 algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4DFAB270CAA96DFA algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 112695A0E562B32A algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-trixie-automatic.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-automatic.pgp | uid Debian Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-trixie-automatic.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-stable.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22
usr/share/keyrings/debian-archive-trixie-stable.pgp | uid Debian Stable Release Key (13/trixie) <debian-release@lists.debian.org>
=== tester-1 host: keyring files 26
etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | uid Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21
etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | uid Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>
etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | uid Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11
etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | uid Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>
etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | uid Debian Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22
etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | uid Debian Stable Release Key (13/trixie) <debian-release@lists.debian.org>
etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg | pub 1140AF8F639E0C39 algo=1 bits=4096 | created 2022-11-27 | expires 2032-11-24
etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg | uid Proxmox Bookworm Release Key <proxmox-release@proxmox.com>
etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg | pub A7BCD1420BFE778E algo=1 bits=4096 | created 2024-11-10 | expires 2034-11-10
etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg | uid Proxmox Trixie Release Key <proxmox-release@proxmox.com>
usr/share/keyrings/debian-archive-bookworm-automatic.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-automatic.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bookworm-automatic.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-stable.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21
usr/share/keyrings/debian-archive-bookworm-stable.pgp | uid Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-bullseye-automatic.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-automatic.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bullseye-automatic.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-stable.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11
usr/share/keyrings/debian-archive-bullseye-stable.pgp | uid Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (13/trixie) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FFA8EF91DB114E0 algo=1 bits=1024 | created 2004-01-15 | expires 2005-01-27
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2004) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub F1D53D8C4F368D5D algo=17 bits=1024 | created 2005-01-31 | expires 2006-01-31
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2005) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub E415B2B4B5F5BBED algo=17 bits=1024 | created 2005-04-24 | expires never
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian AMD64 Archive Key <debian-amd64@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub B7A50B4134FC6FE5 algo=16 bits=2048 | created 2005-04-24 | expires never
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 010908312D230C5F algo=17 bits=1024 | created 2006-01-03 | expires 2007-02-07
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2006) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub A70DAF536070D3A1 algo=17 bits=1024 | created 2006-11-20 | expires 2009-07-01
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (4.0/etch) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub B5D0C804ADB11277 algo=17 bits=1024 | created 2006-09-17 | expires never
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Etch Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub EC61E0B0BBE55AB3 algo=17 bits=1024 | created 2007-03-31 | expires 2010-03-30
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (4.0/etch)
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 0A3B614236CA98F3 algo=16 bits=2048 | created 2007-03-31 | expires 2010-03-30
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9AA38DCD55BE302B algo=1 bits=4096 | created 2009-01-27 | expires 2012-12-31
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (5.0/lenny) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4D270D06F42584E6 algo=17 bits=1024 | created 2008-04-06 | expires 2012-05-15
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Lenny Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub DFD993306D849617 algo=1 bits=2048 | created 2009-01-24 | expires 2013-01-23
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (5.0/lenny)
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 64481591B98321F9 algo=1 bits=4096 | created 2010-08-07 | expires 2017-08-05
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Squeeze Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub AED4B06F473041FA algo=1 bits=4096 | created 2010-08-27 | expires 2018-03-05
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (6.0/squeeze) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 8B48AD6246925553 algo=1 bits=4096 | created 2012-04-27 | expires 2020-04-25
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (7.0/wheezy) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FB2A1C265FFB764 algo=1 bits=4096 | created 2012-05-08 | expires 2019-05-07
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Wheezy Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub CBF8D6FD518E17E1 algo=1 bits=4096 | created 2013-08-17 | expires 2021-08-15
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Jessie Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 7638D0442B90D010 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9D6D8F6BC857C906 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub EF0F382A1A7B6500 algo=1 bits=4096 | created 2017-05-20 | expires 2025-05-18
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (9/stretch) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub E0B11894F66AEC98 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 04EE7237B7D453EC algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | pub EDA0D2388AE22BA9 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub AA8E81B4331F7F50 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | pub DCC9EFBF77E11517 algo=1 bits=4096 | created 2019-02-05 | expires 2027-02-03
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (10/buster) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub DC30D7C23CBBABEE algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 648ACFD622F3D138 algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4DFAB270CAA96DFA algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 112695A0E562B32A algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-trixie-automatic.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-automatic.pgp | uid Debian Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-trixie-automatic.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-stable.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22
usr/share/keyrings/debian-archive-trixie-stable.pgp | uid Debian Stable Release Key (13/trixie) <debian-release@lists.debian.org>
usr/share/keyrings/proxmox-archive-keyring.gpg | pub 1140AF8F639E0C39 algo=1 bits=4096 | created 2022-11-27 | expires 2032-11-24
usr/share/keyrings/proxmox-archive-keyring.gpg | uid Proxmox Bookworm Release Key <proxmox-release@proxmox.com>
usr/share/keyrings/proxmox-archive-keyring.gpg | pub A7BCD1420BFE778E algo=1 bits=4096 | created 2024-11-10 | expires 2034-11-10
usr/share/keyrings/proxmox-archive-keyring.gpg | uid Proxmox Trixie Release Key <proxmox-release@proxmox.com>
usr/share/keyrings/proxmox-release-bookworm.gpg | pub 1140AF8F639E0C39 algo=1 bits=4096 | created 2022-11-27 | expires 2032-11-24
usr/share/keyrings/proxmox-release-bookworm.gpg | uid Proxmox Bookworm Release Key <proxmox-release@proxmox.com>
usr/share/keyrings/proxmox-release-trixie.gpg | pub A7BCD1420BFE778E algo=1 bits=4096 | created 2024-11-10 | expires 2034-11-10
usr/share/keyrings/proxmox-release-trixie.gpg | uid Proxmox Trixie Release Key <proxmox-release@proxmox.com>
=== tester-1 guest: keyring files 21
etc/apt/keyrings/docker.asc | pub 8D81803C0EBFCD88 algo=1 bits=4096 | created 2017-02-22 | expires never
etc/apt/keyrings/docker.asc | uid Docker Release (CE deb) <docker@docker.com>
etc/apt/keyrings/docker.asc | sub 7EA0A9C3F273FCD8 algo=1 bits=4096 | created 2017-02-22 | expires never
etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | uid Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21
etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | uid Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>
etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | uid Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11
etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | uid Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>
etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | uid Debian Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22
etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | uid Debian Stable Release Key (13/trixie) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-bookworm-automatic.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-automatic.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bookworm-automatic.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-stable.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21
usr/share/keyrings/debian-archive-bookworm-stable.pgp | uid Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-bullseye-automatic.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-automatic.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bullseye-automatic.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-stable.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11
usr/share/keyrings/debian-archive-bullseye-stable.pgp | uid Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (13/trixie) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FFA8EF91DB114E0 algo=1 bits=1024 | created 2004-01-15 | expires 2005-01-27
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2004) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub F1D53D8C4F368D5D algo=17 bits=1024 | created 2005-01-31 | expires 2006-01-31
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2005) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub E415B2B4B5F5BBED algo=17 bits=1024 | created 2005-04-24 | expires never
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian AMD64 Archive Key <debian-amd64@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub B7A50B4134FC6FE5 algo=16 bits=2048 | created 2005-04-24 | expires never
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 010908312D230C5F algo=17 bits=1024 | created 2006-01-03 | expires 2007-02-07
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2006) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub A70DAF536070D3A1 algo=17 bits=1024 | created 2006-11-20 | expires 2009-07-01
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (4.0/etch) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub B5D0C804ADB11277 algo=17 bits=1024 | created 2006-09-17 | expires never
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Etch Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub EC61E0B0BBE55AB3 algo=17 bits=1024 | created 2007-03-31 | expires 2010-03-30
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (4.0/etch)
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 0A3B614236CA98F3 algo=16 bits=2048 | created 2007-03-31 | expires 2010-03-30
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9AA38DCD55BE302B algo=1 bits=4096 | created 2009-01-27 | expires 2012-12-31
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (5.0/lenny) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4D270D06F42584E6 algo=17 bits=1024 | created 2008-04-06 | expires 2012-05-15
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Lenny Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub DFD993306D849617 algo=1 bits=2048 | created 2009-01-24 | expires 2013-01-23
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (5.0/lenny)
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 64481591B98321F9 algo=1 bits=4096 | created 2010-08-07 | expires 2017-08-05
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Squeeze Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub AED4B06F473041FA algo=1 bits=4096 | created 2010-08-27 | expires 2018-03-05
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (6.0/squeeze) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 8B48AD6246925553 algo=1 bits=4096 | created 2012-04-27 | expires 2020-04-25
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (7.0/wheezy) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FB2A1C265FFB764 algo=1 bits=4096 | created 2012-05-08 | expires 2019-05-07
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Wheezy Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub CBF8D6FD518E17E1 algo=1 bits=4096 | created 2013-08-17 | expires 2021-08-15
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Jessie Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 7638D0442B90D010 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9D6D8F6BC857C906 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub EF0F382A1A7B6500 algo=1 bits=4096 | created 2017-05-20 | expires 2025-05-18
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (9/stretch) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub E0B11894F66AEC98 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 04EE7237B7D453EC algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | pub EDA0D2388AE22BA9 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub AA8E81B4331F7F50 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | pub DCC9EFBF77E11517 algo=1 bits=4096 | created 2019-02-05 | expires 2027-02-03
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (10/buster) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub DC30D7C23CBBABEE algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 648ACFD622F3D138 algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4DFAB270CAA96DFA algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 112695A0E562B32A algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-trixie-automatic.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-automatic.pgp | uid Debian Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-trixie-automatic.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-stable.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22
usr/share/keyrings/debian-archive-trixie-stable.pgp | uid Debian Stable Release Key (13/trixie) <debian-release@lists.debian.org>
=== demo-felhom host: keyring files 27
etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | uid Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21
etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | uid Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>
etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | uid Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11
etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | uid Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>
etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | uid Debian Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22
etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | uid Debian Stable Release Key (13/trixie) <debian-release@lists.debian.org>
etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg | pub 1140AF8F639E0C39 algo=1 bits=4096 | created 2022-11-27 | expires 2032-11-24
etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg | uid Proxmox Bookworm Release Key <proxmox-release@proxmox.com>
etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg | pub A7BCD1420BFE778E algo=1 bits=4096 | created 2024-11-10 | expires 2034-11-10
etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg | uid Proxmox Trixie Release Key <proxmox-release@proxmox.com>
usr/share/keyrings/debian-archive-bookworm-automatic.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-automatic.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bookworm-automatic.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-stable.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21
usr/share/keyrings/debian-archive-bookworm-stable.pgp | uid Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-bullseye-automatic.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-automatic.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bullseye-automatic.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-stable.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11
usr/share/keyrings/debian-archive-bullseye-stable.pgp | uid Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (13/trixie) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FFA8EF91DB114E0 algo=1 bits=1024 | created 2004-01-15 | expires 2005-01-27
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2004) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub F1D53D8C4F368D5D algo=17 bits=1024 | created 2005-01-31 | expires 2006-01-31
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2005) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub E415B2B4B5F5BBED algo=17 bits=1024 | created 2005-04-24 | expires never
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian AMD64 Archive Key <debian-amd64@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub B7A50B4134FC6FE5 algo=16 bits=2048 | created 2005-04-24 | expires never
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 010908312D230C5F algo=17 bits=1024 | created 2006-01-03 | expires 2007-02-07
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2006) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub A70DAF536070D3A1 algo=17 bits=1024 | created 2006-11-20 | expires 2009-07-01
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (4.0/etch) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub B5D0C804ADB11277 algo=17 bits=1024 | created 2006-09-17 | expires never
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Etch Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub EC61E0B0BBE55AB3 algo=17 bits=1024 | created 2007-03-31 | expires 2010-03-30
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (4.0/etch)
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 0A3B614236CA98F3 algo=16 bits=2048 | created 2007-03-31 | expires 2010-03-30
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9AA38DCD55BE302B algo=1 bits=4096 | created 2009-01-27 | expires 2012-12-31
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (5.0/lenny) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4D270D06F42584E6 algo=17 bits=1024 | created 2008-04-06 | expires 2012-05-15
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Lenny Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub DFD993306D849617 algo=1 bits=2048 | created 2009-01-24 | expires 2013-01-23
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (5.0/lenny)
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 64481591B98321F9 algo=1 bits=4096 | created 2010-08-07 | expires 2017-08-05
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Squeeze Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub AED4B06F473041FA algo=1 bits=4096 | created 2010-08-27 | expires 2018-03-05
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (6.0/squeeze) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 8B48AD6246925553 algo=1 bits=4096 | created 2012-04-27 | expires 2020-04-25
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (7.0/wheezy) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FB2A1C265FFB764 algo=1 bits=4096 | created 2012-05-08 | expires 2019-05-07
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Wheezy Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub CBF8D6FD518E17E1 algo=1 bits=4096 | created 2013-08-17 | expires 2021-08-15
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Jessie Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 7638D0442B90D010 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9D6D8F6BC857C906 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub EF0F382A1A7B6500 algo=1 bits=4096 | created 2017-05-20 | expires 2025-05-18
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (9/stretch) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub E0B11894F66AEC98 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 04EE7237B7D453EC algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | pub EDA0D2388AE22BA9 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub AA8E81B4331F7F50 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | pub DCC9EFBF77E11517 algo=1 bits=4096 | created 2019-02-05 | expires 2027-02-03
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (10/buster) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub DC30D7C23CBBABEE algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 648ACFD622F3D138 algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4DFAB270CAA96DFA algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 112695A0E562B32A algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-trixie-automatic.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-automatic.pgp | uid Debian Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-trixie-automatic.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-stable.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22
usr/share/keyrings/debian-archive-trixie-stable.pgp | uid Debian Stable Release Key (13/trixie) <debian-release@lists.debian.org>
usr/share/keyrings/proxmox-archive-keyring.gpg | pub 1140AF8F639E0C39 algo=1 bits=4096 | created 2022-11-27 | expires 2032-11-24
usr/share/keyrings/proxmox-archive-keyring.gpg | uid Proxmox Bookworm Release Key <proxmox-release@proxmox.com>
usr/share/keyrings/proxmox-archive-keyring.gpg | pub A7BCD1420BFE778E algo=1 bits=4096 | created 2024-11-10 | expires 2034-11-10
usr/share/keyrings/proxmox-archive-keyring.gpg | uid Proxmox Trixie Release Key <proxmox-release@proxmox.com>
usr/share/keyrings/proxmox-release-bookworm.gpg | pub 1140AF8F639E0C39 algo=1 bits=4096 | created 2022-11-27 | expires 2032-11-24
usr/share/keyrings/proxmox-release-bookworm.gpg | uid Proxmox Bookworm Release Key <proxmox-release@proxmox.com>
usr/share/keyrings/proxmox-release-trixie.gpg | pub A7BCD1420BFE778E algo=1 bits=4096 | created 2024-11-10 | expires 2034-11-10
usr/share/keyrings/proxmox-release-trixie.gpg | uid Proxmox Trixie Release Key <proxmox-release@proxmox.com>
usr/share/keyrings/tailscale-archive-keyring.gpg | pub 458CA832957F5868 algo=1 bits=4096 | created 2020-02-25 | expires never
usr/share/keyrings/tailscale-archive-keyring.gpg | uid Tailscale Inc. (Package repository signing key) <info@tailscale.com>
usr/share/keyrings/tailscale-archive-keyring.gpg | sub B1547A3DDAAF03C6 algo=1 bits=4096 | created 2020-02-25 | expires never
=== demo-felhom guest: keyring files 21
etc/apt/keyrings/docker.asc | pub 8D81803C0EBFCD88 algo=1 bits=4096 | created 2017-02-22 | expires never
etc/apt/keyrings/docker.asc | uid Docker Release (CE deb) <docker@docker.com>
etc/apt/keyrings/docker.asc | sub 7EA0A9C3F273FCD8 algo=1 bits=4096 | created 2017-02-22 | expires never
etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | uid Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21
etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | uid Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>
etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | uid Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11
etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | uid Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>
etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | uid Debian Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22
etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | uid Debian Stable Release Key (13/trixie) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-bookworm-automatic.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-automatic.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bookworm-automatic.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-bookworm-stable.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21
usr/share/keyrings/debian-archive-bookworm-stable.pgp | uid Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-bullseye-automatic.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-automatic.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bullseye-automatic.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-bullseye-stable.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11
usr/share/keyrings/debian-archive-bullseye-stable.pgp | uid Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15
usr/share/keyrings/debian-archive-keyring.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19
usr/share/keyrings/debian-archive-keyring.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-keyring.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-keyring.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22
usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (13/trixie) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FFA8EF91DB114E0 algo=1 bits=1024 | created 2004-01-15 | expires 2005-01-27
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2004) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub F1D53D8C4F368D5D algo=17 bits=1024 | created 2005-01-31 | expires 2006-01-31
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2005) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub E415B2B4B5F5BBED algo=17 bits=1024 | created 2005-04-24 | expires never
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian AMD64 Archive Key <debian-amd64@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub B7A50B4134FC6FE5 algo=16 bits=2048 | created 2005-04-24 | expires never
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 010908312D230C5F algo=17 bits=1024 | created 2006-01-03 | expires 2007-02-07
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2006) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub A70DAF536070D3A1 algo=17 bits=1024 | created 2006-11-20 | expires 2009-07-01
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (4.0/etch) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub B5D0C804ADB11277 algo=17 bits=1024 | created 2006-09-17 | expires never
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Etch Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub EC61E0B0BBE55AB3 algo=17 bits=1024 | created 2007-03-31 | expires 2010-03-30
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (4.0/etch)
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 0A3B614236CA98F3 algo=16 bits=2048 | created 2007-03-31 | expires 2010-03-30
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9AA38DCD55BE302B algo=1 bits=4096 | created 2009-01-27 | expires 2012-12-31
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (5.0/lenny) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4D270D06F42584E6 algo=17 bits=1024 | created 2008-04-06 | expires 2012-05-15
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Lenny Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub DFD993306D849617 algo=1 bits=2048 | created 2009-01-24 | expires 2013-01-23
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (5.0/lenny)
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 64481591B98321F9 algo=1 bits=4096 | created 2010-08-07 | expires 2017-08-05
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Squeeze Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub AED4B06F473041FA algo=1 bits=4096 | created 2010-08-27 | expires 2018-03-05
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (6.0/squeeze) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 8B48AD6246925553 algo=1 bits=4096 | created 2012-04-27 | expires 2020-04-25
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (7.0/wheezy) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FB2A1C265FFB764 algo=1 bits=4096 | created 2012-05-08 | expires 2019-05-07
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Wheezy Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub CBF8D6FD518E17E1 algo=1 bits=4096 | created 2013-08-17 | expires 2021-08-15
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Jessie Stable Release Key <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 7638D0442B90D010 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9D6D8F6BC857C906 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub EF0F382A1A7B6500 algo=1 bits=4096 | created 2017-05-20 | expires 2025-05-18
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (9/stretch) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub E0B11894F66AEC98 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 04EE7237B7D453EC algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | pub EDA0D2388AE22BA9 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub AA8E81B4331F7F50 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20
usr/share/keyrings/debian-archive-removed-keys.pgp | pub DCC9EFBF77E11517 algo=1 bits=4096 | created 2019-02-05 | expires 2027-02-03
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (10/buster) <debian-release@lists.debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | pub DC30D7C23CBBABEE algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 648ACFD622F3D138 algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4DFAB270CAA96DFA algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-removed-keys.pgp | sub 112695A0E562B32A algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12
usr/share/keyrings/debian-archive-trixie-automatic.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-automatic.pgp | uid Debian Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-trixie-automatic.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) <ftpmaster@debian.org>
usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28
usr/share/keyrings/debian-archive-trixie-stable.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22
usr/share/keyrings/debian-archive-trixie-stable.pgp | uid Debian Stable Release Key (13/trixie) <debian-release@lists.debian.org>
@@ -0,0 +1,16 @@
taken 2026-11-11T10:17:10Z from DooPlex
== demo-hp
pve-ssl notAfter=Aug 19 15:44:42 2028 GMT
proxmox.tls.fingerprint set equals pve-ssl leaf: True
[{"expire":0,"privsep":1,"tokenid":"agent"}]
inactive
== root@192.168.0.154
pve-ssl notAfter=Oct 2 19:38:05 2028 GMT
proxmox.tls.fingerprint set equals pve-ssl leaf: True
[{"expire":0,"privsep":1,"tokenid":"agent"}]
inactive
== felhom-pve
pve-ssl notAfter=Jul 16 16:17:07 2028 GMT
proxmox.tls.fingerprint set equals pve-ssl leaf: True
[{"expire":0,"privsep":1,"tokenid":"agent"}]
active
@@ -0,0 +1,29 @@
taken 2026-11-11T10:06:59Z from DooPlex
=== certificates served by public names ===
felhom.eu [resolves here to 188.114.96.3] -> subject=CN=felhom.eu issuer=C=US, O=Google Trust Services, CN=WE1 notBefore=Oct 30 09:04:46 2026 GMT notAfter=Jan 28 10:04:43 2027 GMT sha256 Fingerprint=90:5A:85:D7:9E:45:7B:62:CF:13:C3:88:59:3B:E7:DF:D1:45:72:20:80:33:A6:D2:5F:92:42:12:46:77:8F:FD
www.felhom.eu [resolves here to 188.114.97.3] -> subject=CN=www.felhom.eu issuer=C=US, O=Google Trust Services, CN=WE1 notBefore=Oct 30 09:04:46 2026 GMT notAfter=Jan 28 10:04:45 2027 GMT sha256 Fingerprint=8F:A4:81:B4:E6:4C:33:E4:26:33:BC:8D:73:EC:37:12:D4:43:DC:3D:A0:72:AC:12:9C:C2:14:1F:5F:41:F1:53
hub.felhom.eu [resolves here to 37.191.56.193] -> subject=CN=hub.felhom.eu issuer=C=US, O=Let's Encrypt, CN=YR2 notBefore=Oct 14 07:36:58 2026 GMT notAfter=Jan 12 07:36:57 2027 GMT sha256 Fingerprint=05:C2:F0:F2:56:D4:5D:33:0B:A1:95:69:1D:5A:7E:E3:2E:76:37:35:2C:E7:AF:5C:72:0B:3A:06:DC:45:32:53
iso.felhom.eu [resolves here to 188.114.97.3] -> subject=CN=b7d41af6.sni.cloudflaressl.com issuer=C=US, O=Google Trust Services, CN=WE1 notBefore=Sep 28 08:12:03 2026 GMT notAfter=Dec 27 09:11:59 2026 GMT sha256 Fingerprint=8C:D9:5A:7A:55:BC:F3:2A:D8:1B:52:2D:A4:C0:20:0A:B9:6A:D7:9B:62:D1:AE:CF:2A:7E:D4:93:45:1A:49:1A
files.felhom.eu [resolves here to 37.191.56.193] -> subject=CN=files.felhom.eu issuer=C=US, O=Let's Encrypt, CN=YR1 notBefore=Sep 24 04:36:30 2026 GMT notAfter=Dec 23 04:36:29 2026 GMT sha256 Fingerprint=04:5A:A5:37:38:FB:BE:61:BE:9B:5C:3B:32:78:D2:90:B0:28:D0:11:B8:E6:D1:D2:E2:68:21:9D:0F:D6:24:EA
gitea.dooplex.hu [resolves here to 37.191.56.193] -> subject=CN=gitea.dooplex.hu issuer=C=US, O=Let's Encrypt, CN=YR2 notBefore=Oct 5 03:52:51 2026 GMT notAfter=Jan 3 03:52:50 2027 GMT sha256 Fingerprint=00:55:A8:AB:39:49:D6:51:7F:8B:8C:37:95:6F:6B:2B:83:8B:3A:06:44:96:EE:42:1E:9C:31:B9:10:CB:7B:B0
felhom.enkisfelhom.hu [resolves here to 188.114.96.3] -> subject=CN=enkisfelhom.hu issuer=C=US, O=Let's Encrypt, CN=YE2 notBefore=Oct 22 02:42:30 2026 GMT notAfter=Jan 20 02:42:29 2027 GMT sha256 Fingerprint=69:DC:B3:59:83:C6:96:2A:2A:5E:D0:75:D5:34:F1:11:CF:6B:F1:3E:90:24:32:D7:80:E2:6D:83:18:92:91:96
felhom.enkicsifelhom.hu [resolves here to 104.21.22.132] -> subject=CN=enkicsifelhom.hu issuer=C=US, O=Let's Encrypt, CN=YE2 notBefore=Oct 21 13:19:02 2026 GMT notAfter=Jan 19 13:19:01 2027 GMT sha256 Fingerprint=92:6F:C1:DD:25:6C:0A:66:ED:1B:F3:23:5A:10:58:D5:66:6C:6D:C4:B3:1B:58:C4:1F:1D:B8:28:97:5D:E6:AB
felhom.demo-felhom.eu [resolves here to 188.114.97.3] -> subject=CN=demo-felhom.eu issuer=C=US, O=Google Trust Services, CN=WE1 notBefore=Oct 4 06:07:02 2026 GMT notAfter=Jan 2 07:05:44 2027 GMT sha256 Fingerprint=12:78:83:48:BC:B7:9E:7C:06:13:19:25:24:2B:6E:40:1D:24:CF:F7:59:0F:2D:BB:22:D2:9E:AA:D7:C4:F3:A5
=== public registry records (domains) ===
--- felhom.eu
no whois tool on DooPlex
rdap: no JSON answer
--- demo-felhom.eu
no whois tool on DooPlex
rdap: no JSON answer
--- enkisfelhom.hu
no whois tool on DooPlex
rdap: no JSON answer
--- enkicsifelhom.hu
no whois tool on DooPlex
rdap: no JSON answer
--- dooplex.hu
no whois tool on DooPlex
rdap: no JSON answer
@@ -0,0 +1,73 @@
# /usr/bin/pveupdate lines 100-170 on demo-hp, taken 2026-11-11T10:24:12Z (read-only)
);
}
}
};
syslog('err', "Renewing ACME certificate failed: $@") if $@;
eval { PVE::Cluster::Setup::renew_pveca_cert() };
syslog('err', "Renewing cluster CA certificate failed: $@") if $@;
eval {
my $certpath = PVE::CertHelpers::default_cert_path_prefix($nodename) . ".pem";
my $capath = "/etc/pve/pve-root-ca.pem";
my $check_issued_by_ca = sub {
# get CA info
my $cainfo = PVE::Certificate::get_certificate_info($capath);
# get cert and check issuer and chain metadata
my $certinfo = PVE::Certificate::get_certificate_info($certpath);
if ($certinfo->{issuer} ne $cainfo->{subject}) {
die "SSL certificate ($certpath) is not issued by root CA ($capath)!\n";
}
# check if cert is really signed by the ca
# TODO: replace by low level ssleay interface if version 1.86 is available
my $cmd = [
'/usr/bin/openssl', 'verify', '-no_check_time', '-CAfile', $capath, '--', $certpath,
];
PVE::Tools::run_command($cmd);
};
my $renew = sub {
my ($msg, $replaced_ca) = @_;
# one from a replaced cluster CA fails these checks, but is still ours to replace
$check_issued_by_ca->() if !$replaced_ca;
print "PVE certificate $msg\n";
# create new certificate
my $ip = PVE::Cluster::remote_node_ip($nodename);
PVE::Cluster::Setup::gen_pve_ssl_cert(1, $nodename, $ip);
my $expiry = localtime(PVE::Certificate::get_expiration_as_epoch($certpath));
print "Renewed PVE certificate is valid until $expiry\n";
print "Restarting pveproxy after renewing certificate\n";
PVE::Tools::run_command(['systemctl', 'reload-or-restart', 'pveproxy']);
};
my $status = eval { PVE::Cluster::Setup::pve_ssl_cert_status($nodename) };
syslog('err', "Checking SSL certificate failed: $@") if $@;
if ($status && $status->{'issued-by'} eq 'replaced-ca') {
$renew->("was issued by a replaced cluster CA, renewing...", 1);
} elsif (PVE::Certificate::check_expiry($certpath)) {
# already expired
$renew->("expired, renewing...");
} elsif (PVE::Certificate::check_expiry($certpath, time() + 14 * 24 * 60 * 60)) {
# expires in next 2 weeks
$renew->("expires soon, renewing...");
} elsif (!PVE::Certificate::check_expiry($certpath, time() + 2 * 365 * 24 * 60 * 60)) {
# expires in more than 2 years
$renew->(
"expires in more than 2 years, renewing to reduce certificate life-span for client compatibility..."
);
}
};
syslog('err', "Checking/Renewing SSL certificate failed: $@") if $@;
sub cleanup_tasks {
my $taskdir = "/var/log/pve/tasks";
RC=0
@@ -0,0 +1,126 @@
# guest-dates tester-1 taken 2026-11-11T10:06:05Z from DooPlex
=== now ===
2026-11-11T10:06:06Z
=== layout ===
stacks
actualbudget adventurelog audiobookshelf bentopdf bookstack calcom calibre-web claper cloudflared code-server crafty-controller dawarich docmost emby filebrowser ghost gitea glance gokapi grafana gramps-web grimmory grocy home-assistant homebox homepage immich jellyfin karakeep kimai komga lubelogger mealie metube n8n navidrome nextcloud nzbget onlyoffice opengist outline paperless-ngx papra plant-it plex privatebin prowlarr qbittorrent radarr radicale rallly recipe-importer romm sabnzbd seerr sonarr sparkyfitness tandoor termix traefik transmission uptime-kuma vaultwarden vikunja wanderer wger wishlist zipline
felhom-controller-data
=== certificate files (system paths and infra stacks only) ===
=== acme stores (certificate fields only) ===
--- /opt/docker/stacks/traefik/acme.json mtime=2026-10-14T21:15:22Z
letsencrypt *.enkicsifelhom.hu | subject=CN=*.enkicsifelhom.hu issuer=C=US, O=Let's Encrypt, CN=YR1 notBefore=Oct 4 18:44:07 2026 GMT notAfter=Jan 2 18:44:06 2027 GMT sha256 Fingerprint=41:37:66:A9:A3:80:76:6D:3D:E1:64:AD:D2:CC:98:98:47:05:92:22:1D:00:F1:6F:E3:82:9C:1A:0D:E6:30:FD
=== served certificates (TLS handshake to own listeners) ===
*:22 -> no TLS
0.0.0.0:443 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 04:33:46 2026 GMT notAfter=Nov 11 04:33:46 2027 GMT sha256 Fingerprint=68:F4:15:12:FB:45:7A:81:06:62:56:3E:B5:DA:77:AD:0E:9C:B8:B1:9E:21:CF:79:C0:D9:DB:82:2A:9A:16:84
0.0.0.0:80 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 04:33:46 2026 GMT notAfter=Nov 11 04:33:46 2027 GMT sha256 Fingerprint=68:F4:15:12:FB:45:7A:81:06:62:56:3E:B5:DA:77:AD:0E:9C:B8:B1:9E:21:CF:79:C0:D9:DB:82:2A:9A:16:84
127.0.0.1:25 -> no TLS
[::1]:25 -> no TLS
[::]:443 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 04:33:46 2026 GMT notAfter=Nov 11 04:33:46 2027 GMT sha256 Fingerprint=68:F4:15:12:FB:45:7A:81:06:62:56:3E:B5:DA:77:AD:0E:9C:B8:B1:9E:21:CF:79:C0:D9:DB:82:2A:9A:16:84
[::]:80 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 04:33:46 2026 GMT notAfter=Nov 11 04:33:46 2027 GMT sha256 Fingerprint=68:F4:15:12:FB:45:7A:81:06:62:56:3E:B5:DA:77:AD:0E:9C:B8:B1:9E:21:CF:79:C0:D9:DB:82:2A:9A:16:84
=== os ===
Debian GNU/Linux 13 (trixie) / trixie
13.7
=== docker ===
client=29.8.2 server=29.8.2
containerd.io 2.3.6-1~debian.13~trixie
docker-ce 5:29.8.2-1~debian.13~trixie
docker-compose-plugin 5.6.0-1~debian.13~trixie
gitea.dooplex.hu/admin/felhom-controller:0.352.0
=== restic and tools inside the controller ===
restic 0.19.1 compiled with go1.26.4 on linux/amd64
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
NAME="Debian GNU/Linux"
VERSION_ID="13"
=== images: repo tag digest created ===
alpine:latest sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6 2026-09-17 20:37:20 +0000 UTC
cloudflare/cloudflared:2026.10.3 sha256:faa8e0a1475f137c910be6025f43c8e4092fdd35461b31e19fd1390657a1ff39 2026-10-21 13:26:38 +0000 UTC
cloudflare/cloudflared:2026.10.4 sha256:c6c7ae1d5dfdcb094ad344ed68ccffc26635c9c6be80965664c33baf064b0515 2026-10-26 11:27:24 +0000 UTC
gitea.dooplex.hu/admin/felhom-controller:0.351.0 sha256:1d90563c443022770b2db90a7730f1ea5aea45cc5b0fa64fec9823d60d42d97f 2026-11-09 12:14:14 +0000 UTC
gitea.dooplex.hu/admin/felhom-controller:0.352.0 sha256:3c785dd396b091f19d4c777db9d7a206e75ab018b6238b5d887e1a4c6cfd3b29 2026-11-10 15:16:20 +0000 UTC
gitea.dooplex.hu/admin/felhom-samba:1.1.0 sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10 2026-07-20 11:36:16 +0000 UTC
gtstef/filebrowser:1.5.6-stable sha256:7c5d7ac8ffda31294d278063cf9d2e04303b39e6dce1f4c691342240ca7703b8 2026-09-04 21:33:58 +0000 UTC
gtstef/filebrowser:1.5.8-stable sha256:68455d4953bad8e984e1bc5d112ed1e9c918a0a04f104c13c9ece50790308894 2026-10-04 21:15:40 +0000 UTC
hello-world:latest sha256:5e23090353324d887c48ad5e5c56d294eab81588df9605b07d1afe895f9cc8f8 2026-03-23 21:33:59 +0000 UTC
traefik/whoami:v1.10.3 sha256:43a68d10b9dfcfc3ffbfe4dd42100dc9aeaf29b3a5636c856337a5940f1b4f1c 2024-08-22 08:07:12 +0000 UTC
traefik:v3.7.16 sha256:1336e85aaed293bd63d005086e0dd013b04078b26aaeb1c18b26a2f928af7e65 2026-10-21 22:55:46 +0000 UTC
traefik:v3.7.17 sha256:68ae5fe5b64ca166704b746a9e8afcd385e36b3f569a5691401bf652d57b3748 2026-10-30 20:17:26 +0000 UTC
=== apt sources ===
--- /etc/apt/sources.list.d/debian.sources
Types: deb
URIs: http://security.debian.org
Suites: trixie-security
Components: contrib main
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
Types: deb
URIs: http://deb.debian.org/debian
Suites: trixie trixie-updates
Components: contrib main
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
--- /etc/apt/sources.list.d/docker.list
deb [signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian trixie stable
=== apt keyring files ===
3817 2026-10-04 docker.asc
11861 2025-04-09 debian-archive-bookworm-automatic.asc
11873 2025-04-09 debian-archive-bookworm-security-automatic.asc
461 2025-04-09 debian-archive-bookworm-stable.asc
11861 2025-04-09 debian-archive-bullseye-automatic.asc
11873 2025-04-09 debian-archive-bullseye-security-automatic.asc
3403 2025-04-09 debian-archive-bullseye-stable.asc
11861 2025-04-09 debian-archive-trixie-automatic.asc
11873 2025-04-09 debian-archive-trixie-security-automatic.asc
1384 2025-04-09 debian-archive-trixie-stable.asc
37 2025-04-09 debian-archive-bookworm-automatic.gpg
8700 2025-04-09 debian-archive-bookworm-automatic.pgp
46 2025-04-09 debian-archive-bookworm-security-automatic.gpg
8709 2025-04-09 debian-archive-bookworm-security-automatic.pgp
34 2025-04-09 debian-archive-bookworm-stable.gpg
280 2025-04-09 debian-archive-bookworm-stable.pgp
37 2025-04-09 debian-archive-bullseye-automatic.gpg
8700 2025-04-09 debian-archive-bullseye-automatic.pgp
46 2025-04-09 debian-archive-bullseye-security-automatic.gpg
8709 2025-04-09 debian-archive-bullseye-security-automatic.pgp
34 2025-04-09 debian-archive-bullseye-stable.gpg
2453 2025-04-09 debian-archive-bullseye-stable.pgp
26 2025-04-09 debian-archive-keyring.gpg
55918 2025-04-09 debian-archive-keyring.pgp
31 2025-04-09 debian-archive-removed-keys.gpg
72636 2025-04-09 debian-archive-removed-keys.pgp
35 2025-04-09 debian-archive-trixie-automatic.gpg
8698 2025-04-09 debian-archive-trixie-automatic.pgp
44 2025-04-09 debian-archive-trixie-security-automatic.gpg
8707 2025-04-09 debian-archive-trixie-security-automatic.pgp
32 2025-04-09 debian-archive-trixie-stable.gpg
962 2025-04-09 debian-archive-trixie-stable.pgp
=== apt release files: Valid-Until ===
deb.debian.org_debian_dists_trixie-updates_InRelease | Date: Wed, 11 Nov 2026 02:14:24 UTC | Valid-Until: Wed, 18 Nov 2026 02:14:24 UTC
deb.debian.org_debian_dists_trixie_InRelease | Date: Sat, 12 Sep 2026 07:55:41 UTC | Valid-Until:
download.docker.com_linux_debian_dists_trixie_InRelease | Date: Fri, 06 Nov 2026 03:19:25 +0000 | Valid-Until:
security.debian.org_dists_trixie-security_InRelease | Date: Wed, 11 Nov 2026 00:23:32 UTC | Valid-Until: Wed, 18 Nov 2026 00:23:32 UTC
=== controller data folder (top level: names and dates only) ===
--- felhom-controller-data
total
-rw------- 3044 2026-10-04 19:41 controller.yaml
-rw------- 3044 2026-10-04 19:41 controller.yaml.pre-media-2026-10-14
drwxr-xr-x 4096 2026-11-11 10:06 data
RC=0
@@ -0,0 +1,214 @@
# host-dates tester-1 taken 2026-11-11T10:05:43Z from DooPlex
=== now ===
2026-11-11T10:05:43Z
=== certificate files ===
/etc/pve/local/pve-ssl.pem [certs=1 mtime=2026-10-04] subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=felhom.enkicsifelhom.hu issuer=CN=Proxmox Virtual Environment, OU=f9c69a56-050d-43dd-87c8-d596f6a79d19, O=PVE Cluster Manager CA notBefore=Oct 3 19:38:05 2026 GMT notAfter=Oct 2 19:38:05 2028 GMT sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B
/etc/pve/nodes/felhom/pve-ssl.pem [certs=1 mtime=2026-10-04] subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=felhom.enkicsifelhom.hu issuer=CN=Proxmox Virtual Environment, OU=f9c69a56-050d-43dd-87c8-d596f6a79d19, O=PVE Cluster Manager CA notBefore=Oct 3 19:38:05 2026 GMT notAfter=Oct 2 19:38:05 2028 GMT sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B
/etc/pve/pve-root-ca.pem [certs=1 mtime=2026-10-04] subject=CN=Proxmox Virtual Environment, OU=f9c69a56-050d-43dd-87c8-d596f6a79d19, O=PVE Cluster Manager CA issuer=CN=Proxmox Virtual Environment, OU=f9c69a56-050d-43dd-87c8-d596f6a79d19, O=PVE Cluster Manager CA notBefore=Oct 3 19:38:05 2026 GMT notAfter=Sep 30 19:38:05 2036 GMT sha256 Fingerprint=4E:46:8D:BC:45:A2:2C:CD:AB:EF:82:C0:A4:B1:4D:6C:5C:DA:5A:7F:6F:D7:3B:E9:B7:50:CA:ED:C4:0C:80:CF
/var/lib/felhom-agent/local-api.crt [certs=1 mtime=2026-10-04] subject=CN=felhom-agent local-api issuer=CN=felhom-agent local-api notBefore=Oct 4 18:40:41 2026 GMT notAfter=Oct 1 18:40:41 2036 GMT sha256 Fingerprint=93:5A:5D:8A:FA:36:BE:87:4D:4E:FE:7D:83:8A:27:3E:89:7C:83:CD:32:00:CE:47:93:79:47:46:FF:63:DA:C6
=== system CA store ===
148
ca-certificates 20250419+deb13u1
=== served certificates (TLS handshake to own listeners) ===
0.0.0.0:111 -> no TLS
0.0.0.0:22 -> no TLS
0.0.0.0:8822 -> no TLS
[::]:111 -> no TLS
[::1]:25 -> no TLS
127.0.0.1:25 -> no TLS
127.0.0.1:53 -> no TLS
127.0.0.1:85 -> no TLS
169.254.253.1:8443 -> subject=CN=felhom-agent local-api issuer=CN=felhom-agent local-api notBefore=Oct 4 18:40:41 2026 GMT notAfter=Oct 1 18:40:41 2036 GMT sha256 Fingerprint=93:5A:5D:8A:FA:36:BE:87:4D:4E:FE:7D:83:8A:27:3E:89:7C:83:CD:32:00:CE:47:93:79:47:46:FF:63:DA:C6
192.168.0.154:53 -> no TLS
[::]:22 -> no TLS
*:3128 -> no TLS
*:8006 -> subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=felhom.enkicsifelhom.hu issuer=CN=Proxmox Virtual Environment, OU=f9c69a56-050d-43dd-87c8-d596f6a79d19, O=PVE Cluster Manager CA notBefore=Oct 3 19:38:05 2026 GMT notAfter=Oct 2 19:38:05 2028 GMT sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B
[::]:8822 -> no TLS
=== listeners (port, process) ===
0.0.0.0:111 users:(("rpcbind",),("systemd",))
0.0.0.0:22 users:(("sshd",))
0.0.0.0:8822 users:(("sshd",))
[::]:111 users:(("rpcbind",),("systemd",))
[::1]:25 users:(("smtpd",),("master",))
127.0.0.1:25 users:(("smtpd",),("master",))
127.0.0.1:53 users:(("dnsmasq",))
127.0.0.1:85 users:(("pvedaemon
169.254.253.1:8443 users:(("felhom-agent",))
192.168.0.154:53 users:(("dnsmasq",))
[::]:22 users:(("sshd",))
*:3128 users:(("spiceproxy
*:8006 users:(("pveproxy
[::]:8822 users:(("sshd",))
=== pve users and tokens (expire: 0 = never) ===
user felhom-agent@pve enable 1 expire 0 tokens []
user root@pam enable 1 expire 0 tokens []
=== acme / subscription ===
message: There is no subscription key
status: notfound
=== os ===
Debian GNU/Linux 13 (trixie) / trixie
13.7
pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.14-22-pve)
7.0.14-22-pve
Manually selected kernels:
None.
Automatically selected kernels:
7.0.14-20-pve
7.0.14-22-pve
proxmox-kernel-7.0.14-20-pve-signed 7.0.14-20
proxmox-kernel-7.0.14-22-pve-signed 7.0.14-22
proxmox-kernel-7.0.2-6-pve-signed 7.0.2-6
=== apt sources ===
--- /etc/apt/sources.list.d/ceph.sources
Types: deb
URIs: https://enterprise.proxmox.com/debian/ceph-squid
Suites: trixie
Components: enterprise
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
Enabled: no
--- /etc/apt/sources.list.d/debian.sources
Types: deb
URIs: http://deb.debian.org/debian/
Suites: trixie trixie-updates
Components: main contrib non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
Types: deb
URIs: http://security.debian.org/debian-security/
Suites: trixie-security
Components: main contrib non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
--- /etc/apt/sources.list.d/pve-enterprise.sources
Types: deb
URIs: https://enterprise.proxmox.com/debian/pve
Suites: trixie
Components: pve-enterprise
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
Enabled: no
--- /etc/apt/sources.list.d/pve-no-subscription.sources
Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
--- auth files (names only): /etc/apt/auth.conf.d:
=== apt keyring files ===
11861 2025-04-10 debian-archive-bookworm-automatic.asc
11873 2025-04-10 debian-archive-bookworm-security-automatic.asc
461 2025-04-10 debian-archive-bookworm-stable.asc
11861 2025-04-10 debian-archive-bullseye-automatic.asc
11873 2025-04-10 debian-archive-bullseye-security-automatic.asc
3403 2025-04-10 debian-archive-bullseye-stable.asc
11861 2025-04-10 debian-archive-trixie-automatic.asc
11873 2025-04-10 debian-archive-trixie-security-automatic.asc
1384 2025-04-10 debian-archive-trixie-stable.asc
1187 2022-11-27 proxmox-release-bookworm.gpg
1185 2024-11-10 proxmox-release-trixie.gpg
37 2025-04-10 debian-archive-bookworm-automatic.gpg
8700 2025-04-10 debian-archive-bookworm-automatic.pgp
46 2025-04-10 debian-archive-bookworm-security-automatic.gpg
8709 2025-04-10 debian-archive-bookworm-security-automatic.pgp
34 2025-04-10 debian-archive-bookworm-stable.gpg
280 2025-04-10 debian-archive-bookworm-stable.pgp
37 2025-04-10 debian-archive-bullseye-automatic.gpg
8700 2025-04-10 debian-archive-bullseye-automatic.pgp
46 2025-04-10 debian-archive-bullseye-security-automatic.gpg
8709 2025-04-10 debian-archive-bullseye-security-automatic.pgp
34 2025-04-10 debian-archive-bullseye-stable.gpg
2453 2025-04-10 debian-archive-bullseye-stable.pgp
26 2025-04-10 debian-archive-keyring.gpg
55918 2025-04-10 debian-archive-keyring.pgp
31 2025-04-10 debian-archive-removed-keys.gpg
72636 2025-04-10 debian-archive-removed-keys.pgp
35 2025-04-10 debian-archive-trixie-automatic.gpg
8698 2025-04-10 debian-archive-trixie-automatic.pgp
44 2025-04-10 debian-archive-trixie-security-automatic.gpg
8707 2025-04-10 debian-archive-trixie-security-automatic.pgp
32 2025-04-10 debian-archive-trixie-stable.gpg
962 2025-04-10 debian-archive-trixie-stable.pgp
2372 2025-07-09 proxmox-archive-keyring.gpg
113 2022-11-27 proxmox-offline-signing-key.pub
1187 2022-11-27 proxmox-release-bookworm.gpg
1185 2024-11-10 proxmox-release-trixie.gpg
=== apt release files: Valid-Until ===
deb.debian.org_debian_dists_trixie_InRelease | Date: Sat, 12 Sep 2026 07:55:41 UTC | Valid-Until:
deb.debian.org_debian_dists_trixie-updates_InRelease | Date: Wed, 11 Nov 2026 02:14:24 UTC | Valid-Until: Wed, 18 Nov 2026 02:14:24 UTC
download.proxmox.com_debian_pve_dists_trixie_InRelease | Date: Wed, 11 Nov 2026 01:41:17 +0000 | Valid-Until:
security.debian.org_debian-security_dists_trixie-security_InRelease | Date: Wed, 11 Nov 2026 00:23:32 UTC | Valid-Until: Wed, 18 Nov 2026 00:23:32 UTC
=== templates and golden ===
=== agent files (names and dates only) ===
/etc/felhom:
total
-rw-r--r-- root 7 2026-10-04 21:38 appliance-pairing-code
-rw-r--r-- root 3897 2026-11-07 23:28 config-bundle.json
-rw-r--r-- root 456 2026-10-04 21:40 crash-guard.conf
-rw-r--r-- root 131 2026-10-04 21:40 operator-signers
-rw-r--r-- root 63 2026-10-04 21:40 os-trust.json
/etc/felhom-agent:
total
-rw------- felhom-agent 2451 2026-10-04 21:55 agent.json
/var/lib/felhom-agent:
total
-rw------- felhom-agent 593 2026-11-11 05:35 backup-success-state.json
-rw------- felhom-agent 290 2026-11-10 16:17 controller-swap-9201.json
-rw------- felhom-agent 321 2026-10-17 10:57 drive-intents.json
drwx------ felhom-agent 4096 2026-10-29 10:34 escrow-stage
drwx------ felhom-agent 4096 2026-10-04 21:40 felhom-sshd
-rw------- felhom-agent 308 2026-10-15 10:04 format-job.json
-rw------- felhom-agent 67 2026-10-17 10:57 guest-binds.json
-rw------- felhom-agent 242 2026-11-11 10:19 guest-disk-trim.json
drwx------ felhom-agent 4096 2026-10-04 21:41 guests
-rw------- felhom-agent 24760 2026-11-11 10:45 journal.log
-rw------- felhom-agent 304 2026-11-10 13:35 line-test.json
-rw-r--r-- felhom-agent 591 2026-10-04 21:40 local-api.crt
-rw------- felhom-agent 227 2026-10-04 21:40 local-api.key
-rw------- felhom-agent 82 2026-10-04 21:41 local-tokens.log
-rw------- felhom-agent 4556 2026-11-10 16:14 nonces.log
drwx------ felhom-agent 4096 2026-11-11 10:59 os
drwx------ felhom-agent 4096 2026-10-04 21:55 pbsdr
-rw------- felhom-agent 367 2026-11-11 10:45 restore-test-state.json
drwxr-x--- felhom-agent 4096 2026-11-10 16:14 selfupdate
-rw------- felhom-agent 32 2026-11-03 08:57 store-grant-seen.json
drwx------ felhom-agent 4096 2026-10-15 10:04 units
drwx------ felhom-agent 4096 2026-10-04 21:40 wg
=== agent ===
felhom-agent 0.173.0
=== wireguard (interface and latest handshake time only) ===
wg-felhom peer f3d1ZI… 2026-11-11T10:04:06Z
=== tailscale ===
no tailscale
=== ssh host keys / authorized keys (counts only) ===
3
/root/.ssh/authorized_keys lines=3
=== smart: power-on hours and wear (no serials) ===
sda:
sdb:
RC=0
@@ -0,0 +1,265 @@
# End dates on the box: agent, controller, catalog, installer (helper result, candidates only)
### 1. Web certificate of the household's address (Let's Encrypt through Traefik)
1. The public TLS certificate for the dashboard and apps. Product-owned Traefik asks Let's Encrypt with the household's e-mail `ACMEEmail` `CT/internal/stacks/infra.go:130`; stored in `acme.json` `CT/internal/infra/templates/traefik.yml.tmpl:51`.
2. The lifetime is the issuer's, not ours: no validity constant in our source. Resolver `letsencrypt` `CT/internal/infra/templates/traefik.yml.tmpl:48-59`. Lifetime value: NOT FOUND (grep `NotAfter|renew|90 ?days` in `CT/internal/infra`).
3. Traefik renews by itself; we configure only the challenge: `dnsChallenge` with a Cloudflare token `CT/internal/infra/templates/traefik.yml.tmpl:53`, else `httpChallenge` `CT/internal/infra/templates/traefik.yml.tmpl:59`. How long before the end: NOT FOUND in our source (Traefik's own default).
4. Browsers show a certificate error on every app address of the household. With no e-mail set Traefik serves a self-signed certificate, see comment at `TraefikData` `CT/internal/infra/infra.go:111-118`.
5. NOT FOUND: no expiry check or warning in the controller (grep `cert.*expir|expir.*cert|NotAfter|acme\.json` in `CT/internal` hit only `CT/internal/stacks/infra.go:160-168` and the mail relay). The issuer mails `ACMEEmail`, which is the household.
6. NOBODY KNOWS from our source: renewal on start is Traefik's behaviour. The file survives because it is created once, mode 0600, `CT/internal/stacks/infra.go:160-168`. The DNS route also needs the Cloudflare token `CFAPIToken` `CT/internal/infra/infra.go:116-118` to be still valid.
7. NOT FOUND in our source: retry is inside Traefik. The old certificate stays in `acme.json` `CT/internal/infra/templates/traefik.yml.tmpl:51` until it ends. Nothing in the controller reads the result (same grep as line 5).
8. SOURCE `CT/internal/infra/templates/traefik.yml.tmpl:48-59` for the setup; NOBODY KNOWS for renewal timing, warning and the switched-off case.
### 2. Proxmox web certificate pinned by the agent
1. The agent talks to Proxmox on the host and trusts ONE certificate by fingerprint. The installer reads the served leaf once, `fp=` `E/scripts/felhom-host-install.sh:3182-3185`; the agent checks it in `TLSConfig.build` `AG/internal/proxmox/tls.go:39-59`. Product-owned.
2. The pin has NO END DATE: the date of the certificate is never looked at, `VerifyPeerCertificate` `AG/internal/proxmox/tls.go:49`. The end is the day Proxmox serves a different leaf. Proxmox's own certificate lifetime: NOT FOUND (grep `pve-ssl|updatecerts|pveproxy cert` in `AG/`, `E/scripts`, `AR/`, `RB/`).
3. NOT FOUND: no code re-reads the fingerprint (grep `TLS pin mismatch|tls\.fingerprint|\.Fingerprint =` in `AG/internal`). The pin is loaded from config `AG/internal/config/config.go:917` and used at start `AG/cmd/felhom-agent/main.go:340`.
4. Candidate: if Proxmox replaces its certificate, every agent call to Proxmox fails with the mismatch error `AG/internal/proxmox/tls.go:55`: no backup, restore, guest start or resize through the agent. Household and operator both.
5. NOT FOUND: no warning before (same greps as line 2 and 3). After the fact the calls fail; which alarm fires was not traced.
6. NOBODY KNOWS: depends on whether Proxmox regenerates its certificate at boot, which is not in our source. If it does, the first failing step is the first Proxmox API call after start, `AG/internal/proxmox/tls.go:55`.
7. The agent keeps the old pin and keeps failing closed, `AG/internal/proxmox/tls.go:55`. Repair path found: none in code. An empty pin falls back to system trust, warning at `E/scripts/felhom-host-install.sh:3185`.
8. TEST `TestTLSConfig_Build` `AG/internal/proxmox/tls_test.go:28` for the pin check; NOBODY KNOWS for the renewal case.
### 3. Agent local-API certificate and its pin in the guest
1. The self-signed leaf the agent serves to the in-guest controller, made by `generateLeaf` `AG/internal/localapi/cert.go:56`. The controller pins its fingerprint in `New` `CT/internal/agentapi/client.go:105`. Product-owned, on the host.
2. `certValidity = 10 * 365 * 24 * time.Hour` `AG/internal/localapi/cert.go:25`, set as `NotAfter` `AG/internal/localapi/cert.go:70`. So a box installed in 2026 has a leaf that ends in 2036.
3. Nobody: `EnsureLeaf` loads the existing pair and never looks at its date `AG/internal/localapi/cert.go:38-52`. The comment says rotation is an operator action for a later slice `AG/internal/localapi/cert.go:21-24`.
4. Candidate: at the end date nothing breaks in the controller, because it sets `InsecureSkipVerify` and compares only the fingerprint `CT/internal/agentapi/client.go:118-128`. What breaks things is a NEW leaf: mismatch `CT/internal/agentapi/client.go:125`.
5. Nobody. NOT FOUND: no expiry reading of this leaf (grep `NotAfter` in `AG/internal` hit only `AG/internal/localapi/cert.go:70`).
6. Yes, by itself: the leaf is generated once and kept, `AR/03-host-agent.md:420-423`, and the reinstall path keeps it with `--preserve-state-from` `E/scripts/felhom-host-install.sh:2749-2786`.
7. No renewal exists, so no failure state. A reinstall that makes a new leaf needs `--allow-new-leaf` `E/scripts/felhom-host-install.sh:2749-2786` and then the guest's pin is stale.
8. TEST `TestEnsureLeaf_StableFingerprintAcrossReload` `AG/internal/localapi/cert_test.go:12` and `TestClient_WrongPinFailsClosed` `CT/internal/agentapi/client_test.go:55`; the 10-year end itself: SOURCE only.
### 4. Off-site copy after the household stops it: 14-day countdown
1. When the household abandons the off-site backup, the copies are kept for a grace period and then deleted. `startAbandonCountdown` `CT/internal/backup/offbox_abandon.go:68-70`. The household starts it; the product runs it.
2. `abandonGraceDays = 14` `CT/internal/backup/offbox_abandon.go:31`.
3. The household can stop or extend it; extending never shortens, test `TestD1_ExtendNeverShortensTheCountdown` `CT/internal/backup/offbox_abandon_levers_d1_test.go:20`.
4. The off-site copies are gone for good; the household loses its only copy outside the house. The deleting side is the hub (see the hub helper); test name `TestD1_ExtendRefusedWhileTheDeletionIsTheHubs` `CT/internal/backup/offbox_abandon_levers_d1_test.go:44`.
5. The household, at 5, 3 and 1 days left: `AbandonRemindAtDays` `CT/internal/backup/offbox_abandon.go:40`.
6. NOT FOUND for a box that is off during the whole countdown: whether the hub deletes without the box's reminders was not traced on the box side (grep `abandonNow|AbandonRemindAtDays` in `CT/internal/backup`).
7. A failed hub cancel is reported, not hidden: `TestD1_StopReportsFailureWhenTheHubCancelFails` `CT/internal/backup/offbox_abandon_levers_d1_test.go:69`.
8. TEST `TestR241_ScenarioE_AbandonStartsAReversibleCountdown` `CT/internal/backup/offbox_abandon_r241_test.go:48`.
### 5. Off-site backup history: how far back a household can go
1. The restic off-site store keeps a thinning history. Policy in `retentionPolicy` `CT/internal/backup/offbox_window.go:81-82`. Product rule, household's data.
2. `keepDaily = 7` `CT/internal/backup/offbox_window.go:47`, `keepWeekly = 4` `CT/internal/backup/offbox_window.go:48`, `keepMonthly = 6` `CT/internal/backup/offbox_window.go:49`. Older than about six months: gone.
3. Nobody renews; each run forgets by the same rule, `retentionPolicy` `CT/internal/backup/offbox_window.go:81-82`.
4. A file deleted or damaged more than six monthly points ago cannot be brought back from off-site. On-host copies are fewer: `KeepLast` `AG/internal/config/config.go:436`, default 3 per comment `AG/internal/config/config.go:409`.
5. NOT FOUND: no warning that a point is about to leave the window (grep `keepMonthly|about to|will be removed` in `CT/internal/backup`).
6. Yes. A box that was off simply has no points for that time; the next run applies the same rule. PBS copies are not pruned by the box by default, comment `AG/internal/backup/runner.go:51-58`.
7. NOT FOUND for a failed forget (grep `forget` failure handling not traced). The weekly content check has its own age `defaultIntegrityMaxAgeDays = 7` `CT/internal/backup/offbox_integrity.go:81`.
8. SOURCE `CT/internal/backup/offbox_window.go:47-49`.
### 6. Off-site credentials: SSH key, host key, store password, PBS pin
1. The box reaches its off-site store with an SSH key and a pinned host key, `StrictHostKeyChecking=yes` `CT/internal/backup/offbox.go:345`, and a password file `RESTIC_PASSWORD_FILE` `CT/internal/backup/offbox.go:898`. PBS is pinned in `pinnedTLS` `AG/internal/pbs/pin.go:17-30`.
2. NO END DATE in box code: no expiry field on the key, the host key or the password (grep `expir` in `CT/internal/backup/offbox.go` found none). The PBS pin ignores the date, `VerifyPeerCertificate` `AG/internal/pbs/pin.go:24`. The end comes from outside: the provider changes its host key or the hub ends the account.
3. Nobody on a schedule. A new PBS descriptor is checked before use: `probeFP` `AG/internal/pbsdr/manager.go:375-379`.
4. If the provider's host key changes, every off-site run fails at the SSH step, `StrictHostKeyChecking=yes` `CT/internal/backup/offbox.go:895`. The household has no off-site copy from that night on.
5. NOT FOUND before the end. After it, the failed run is the signal (alarm path not traced here).
6. Yes if nothing changed outside. A rebuilt guest gets a fresh store password and cannot open the old store: NOT FOUND in this pass (grep `RESTIC_PASSWORD_FILE` `CT/internal/backup/offbox.go:910` shows use only).
7. PBS: state `verify_failed` and a retry at the next tick, `probeFP` `AG/internal/pbsdr/manager.go:375-379`. Restic: the run fails and the next night tries again with the same pins.
8. SOURCE `CT/internal/backup/offbox.go:345`; provider-side dates: NOBODY KNOWS from the box.
### 7. Sign-in sessions and cookies (dashboard, setup gate, family, share) — merged
1. Merged: dashboard session, setup-gate pass, family session, share-guest cookie. All product-made, in the household's browser and the controller's store, `loadSessions` `CT/internal/web/session_store.go:73`.
2. `sessionMaxAge = 7 * 24 * time.Hour` `CT/internal/web/auth.go:30`; `gateCookieLife = 7 * 24 * time.Hour` `CT/internal/web/setup_gate.go:46`; `SessionLife = 30 * 24 * time.Hour` `CT/internal/family/family.go:42`; `shareCookieMaxAge = 30 * 24 * time.Hour` `CT/internal/web/share.go:31`.
3. The family session renews itself at half-life `CT/internal/family/family.go:506-514`. The dashboard session has a fixed end set at sign-in `CT/internal/web/auth.go:308`.
4. The person is asked to sign in again. Nothing is lost.
5. Nobody; no warning is needed.
6. Yes. Expired rows are dropped on load `CT/internal/web/session_store.go:105` and by `cleanupSessions` `CT/internal/web/auth.go:369`; the person signs in again.
7. No renewal to fail.
8. TEST `TestR35_ExpiredRowNotLoaded` `CT/internal/web/session_store_test.go:144` and `TestSetupGate_ARestartKeepsTheHouseholdsPass` `CT/internal/web/setup_gate_test.go:182`.
### 8. Claim code and the one-use gate token
1. The one-time code a household uses to set its own password, and the short token that carries a pass between addresses. Product-made. `claimCodeTTL` `CT/internal/web/claim.go:30`; `gateTokenLife` `CT/internal/web/setup_gate.go:47`.
2. `claimCodeTTL = 72 * time.Hour` `CT/internal/web/claim.go:30`; `gateTokenLife = 60 * time.Second` `CT/internal/web/setup_gate.go:47`, also stated in `AR/01-topology-and-trust.md:192`.
3. The operator sends a new code (hub side, see the hub helper). Nothing on the box mints one by itself.
4. A household that opens the mail after three days cannot set its password until a new code arrives. Checked against the controller's clock `s.claimNow()` `CT/internal/web/claim.go:333`.
5. NOT FOUND: no reminder before the 72 hours end (grep `claimCodeTTL` in `CT/internal/web`).
6. A box off for more than 72 hours comes back with a dead code; first failing step is the claim form, `CT/internal/web/claim.go:333`.
7. Wrong tries lock the form for a time: `claimLockoutWindow` `CT/internal/web/claim.go:33`.
8. TEST `TestClaimSubmit_ExpiredCodeRefused` `CT/internal/web/claim_test.go:189`.
### 9. Share link `/s/<token>`
1. The link a household gives to guests to see its app tiles. Household-owned, stored as `LauncherShareToken` `CT/internal/settings/settings.go:45`.
2. NO END DATE: the token has no expiry field; it ends only when the household rotates or disables it, `SetLauncherShareToken` `CT/internal/settings/settings.go:1111`.
3. The household, by hand. No schedule.
4. Nothing ends. Candidate for the list: a link given away years ago still opens the page. The guest's cookie lasts `shareCookieMaxAge` `CT/internal/web/share.go:31`.
5. Nobody; NOT FOUND any reminder of an old link (grep `LauncherShareToken` in `CT/internal`).
6. Yes, unchanged.
7. No renewal to fail.
8. TEST `TestShareGuest_RotateAndDisable` `CT/internal/web/share_test.go:256`.
### 10. Recovery-code ceremony windows
1. The steps where the household confirms it wrote down its recovery code. Product-made, held by the agent. `escrowConfirmTTL` `AG/internal/localapi/escrow_ceremony.go:158`.
2. `escrowConfirmTTL = 30 * time.Minute` `AG/internal/localapi/escrow_ceremony.go:158`; `escrowClaimTTL = 10 * time.Minute` `AG/internal/localapi/escrow_ceremony.go:166`; the page allows `escrowCeremonyGraceWindow` `CT/internal/web/handlers.go:1169`.
3. The household starts the ceremony again.
4. A household that walks away mid-step must start over. The written-down code itself: NO END DATE found (grep `recovery.*expir|expir.*recovery` in `AG/internal`, `CT/internal` hit only the unclaimed-ceremony refusal `AG/internal/localapi/escrow_ceremony.go:506`).
5. Nobody.
6. Yes; an unfinished ceremony is simply started again.
7. A ceremony that timed out answers "no longer available" (HTTP 410) `AG/internal/localapi/escrow_ceremony.go:506`; the comment says an abandoned one leaves the household's written-down code as it was, `AG/internal/localapi/escrow_ceremony.go:38`. No automatic retry found.
8. SOURCE `AG/internal/localapi/escrow_ceremony.go:158-166`.
### 11. Signed operator jobs: the validity window
1. An order signed by the operator (update the agent, apply OS packages, destructive acts). Operator-owned; it carries a nonce and an expiry, `AR/03-host-agent.md:268`. Checked by `Verify` `AG/internal/authz/verifier.go:104` and by the root wrapper.
2. At most `MaxWindow = 24 * time.Hour` `AG/internal/authz/verifier.go:26`; the signer's default is 30 minutes, `-ttl` `AG/cmd/felhom-opsign/main.go:57`. Wrapper: `MAX_WINDOW_S = 24 * 3600` `AG/configs/felhom-os-apply:194`. Design: `AR/04-control-plane-authorization.md:78-84`.
3. The operator signs a new one. There is no renewal of a signed job.
4. The job is refused and cleared; the executor is never called, `AG/internal/signedjobs/runner.go:163-170`. The box stays on its old agent or old packages. `ErrExpired` `AG/internal/authz/verifier.go:161-162`.
5. NOT FOUND a warning before a job ends. The design says an order unanswered after 24 hours is expired, `AR/04-control-plane-authorization.md:184`.
6. A box off for longer than the window never runs that job; it is checked against the BOX's clock, `time.Now().UTC()` `AG/internal/authz/verifier.go:90`, and in the wrapper `if now > exp or now < iss - 120` `AG/configs/felhom-os-apply:676`. A wrong box clock decides.
7. The refused job is logged as a security event and removed, `AG/internal/signedjobs/runner.go:163-170`; no retry. Used nonces are kept until their end, comment `AG/internal/authz/noncestore.go:47-50`.
8. TEST `test_expired_signed_job_is_refused` `AG/configs/test_felhom_os_apply.py:833`, `test_a_window_longer_than_24h_is_refused` `AG/configs/test_felhom_os_apply.py:836`, `TestVerify_RejectsPerStage` `AG/internal/authz/verify_test.go:82`.
### 12. Operator signing keys on each box
1. The two public keys every box trusts for signed jobs: `OPERATOR_KEY_OPERATIONAL_ID` `E/scripts/felhom-host-install.sh:212` and `OPERATOR_KEY_RECOVERY_ID` `E/scripts/felhom-host-install.sh:214`. Written to `TRUST_SIGNERS` `AG/configs/felhom-os-apply:124` by `_write_slow_lane_trust` `E/scripts/felhom-host-install.sh:3424`. Operator-owned.
2. NO END DATE: the keys carry no expiry; trust is by key material, `AG/internal/authz/verifier.go:122-126`. The key id is advisory, `AG/internal/authz/verifier.go:170-175`.
3. Design only: every re-pin is itself a signed order, `AR/04-control-plane-authorization.md:139-150`. In code only the class name exists, `ClassKeyRotation` `AG/internal/reconcile/classify.go:41`.
4. If the operator loses both private keys, no box can be updated or repaired by a signed job again; households keep running on what they have. Design text `AR/04-control-plane-authorization.md:139-150`.
5. Nobody; there is no date to warn about.
6. Yes, the keys are in the installer and on disk, `E/scripts/felhom-host-install.sh:3452-3453`.
7. NOT FOUND: an executor for key rotation (grep `OpRepin|repin_key|key_rotation|OpKeyRotat|rotate_operator` in `AG/` hit only `AG/internal/reconcile/classify.go:41`). Candidate: rotation is designed, not built.
8. DOC `AR/04-control-plane-authorization.md:139-150`; for a real rotation: NOBODY KNOWS.
### 13. Proxmox API token of the agent
1. The token the agent uses for every Proxmox call. Made by the installer in `step_token` `E/scripts/felhom-host-install.sh:2405`. Product-owned, on the host.
2. NO END DATE: minted with `pveum user token add ... --privsep 1` and no expire option, `E/scripts/felhom-host-install.sh:2448`.
3. Nobody on a schedule. A re-run of the installer reuses it or makes a new one, `E/scripts/felhom-host-install.sh:2424-2445`.
4. Nothing ends by date. If someone sets an expiry by hand in Proxmox, every agent call fails that day.
5. Nobody; NOT FOUND any check of the token's expiry (grep `expire` in `E/scripts/felhom-host-install.sh` near `step_token` found none).
6. Yes.
7. No renewal to fail.
8. SOURCE `E/scripts/felhom-host-install.sh:2448`.
### 14. Guest token for the agent and the box's hub key — merged
1. Merged: the per-guest token the controller shows to the agent, kept as a hash in `TokenStore` `AG/internal/localapi/tokenstore.go:24-33`, and the agent's hub key used in `NewClient` `AG/internal/hub/client.go:60`. Product-owned.
2. NO END DATE for both: no expiry field in `TokenStore` `AG/internal/localapi/tokenstore.go:24-33`; none seen in `NewClient` `AG/internal/hub/client.go:36`.
3. Nobody on a schedule; a new guest token replaces the old one for that guest (last write wins), `AG/internal/localapi/tokenstore.go:24-33`.
4. Nothing ends by date. The hub connection also depends on public root certificates of the host's system store, `AG/internal/hub/client.go:41`, which age with the OS.
5. Nobody.
6. Yes; first step that can fail after years off is the TLS handshake to the hub if the host's root store is too old, `AG/internal/hub/client.go:41`. Candidate, not tested.
7. No renewal to fail.
8. SOURCE `AG/internal/localapi/tokenstore.go:24-33`; old root store: NOBODY KNOWS.
### 15. WireGuard key of the operator tunnel
1. The box's key for the tunnel the operator uses to reach it. Product-owned, on the host. `rotateKey` `AG/internal/wgtunnel/rekey.go:110`.
2. NO END DATE and no periodic rotation: a new key is made only when the handshake time sits in the future for `rekeyAfterDefault = 15 * time.Minute` `AG/internal/wgtunnel/rekey.go:36`. Reason in the comment `AG/internal/wgtunnel/rekey.go:16-30`.
3. The agent by itself, at most once per `rekeyMinGap = time.Hour` `AG/internal/wgtunnel/rekey.go:38`; the old key is kept as `prevKeyName` `AG/internal/wgtunnel/rekey.go:39`.
4. Without the re-key, a box whose clock once ran ahead has a tunnel that never connects: the operator cannot reach it. The household notices nothing.
5. Nobody before; it is a repair, not a date.
6. Yes. This is exactly the case the re-key is for (a clock that jumped), `AG/internal/wgtunnel/rekey.go:16-30`.
7. A failed registration is retried by the existing path: `TestD30_AFailedRegistrationIsRetriedByTheExistingPath` `AG/internal/wgtunnel/rekey_test.go:119`.
8. TEST `TestD30_AFutureHandshakeThatStaysReKeys` `AG/internal/wgtunnel/rekey_test.go:35`, `TestD30_TheHourBoundSurvivesARestart` `AG/internal/wgtunnel/rekey_test.go:182`.
### 16. Package signing keys: Docker in the guest, Proxmox and Debian on the host
1. The keys apt uses to trust packages. Docker's key is fetched once at golden bake into `/etc/apt/keyrings/docker.asc` `AG/configs/build-golden.sh:129`. Proxmox's keyring path default is in `align_apt_repos` `E/scripts/felhom-host-install.sh:2616`. Owned by Docker, Proxmox, Debian.
2. NOT FOUND in our source: the keys' own end dates (grep `expire|valid-until|keyring` in `AG/configs/build-golden.sh` and `E/scripts/felhom-host-install.sh`). The end is set by the publisher.
3. Debian and Proxmox keys arrive as packages through the normal update. Docker's key has no package: a change is "a slow-lane act for a person", `AR/11-os-updates.md:657`.
4. The OS update is refused with code R7 when `apt-get update` fails, `AG/configs/felhom-os-apply:1125-1127`. The box keeps running but stops getting Docker security updates. Household unaware; operator sees the report.
5. Nobody before the end. After it the box reports the failed update, `AR/11-os-updates.md:657`.
6. A box off for years comes back with old keys; the first failing step is `apt-get update` in the wrapper, `AG/configs/felhom-os-apply:1125-1127`. Whether the old key still verifies: NOBODY KNOWS.
7. The update fails cleanly and nothing is half-installed, `AR/11-os-updates.md:657`; the next signed job tries again and fails the same way until a person replaces the key.
8. DOC `AR/11-os-updates.md:657`; TEST for the refusal `test_R7_not_downloadable_and_no_snapshot` `AG/configs/test_felhom_os_apply.py:488`; key dates: NOBODY KNOWS.
### 17. Debian 13 (trixie) and Proxmox 9: the series the product is tied to
1. The OS series under everything. Written into source: suite default `trixie` `E/scripts/felhom-host-install.sh:2615`, Docker repo `trixie stable` `AG/configs/build-golden.sh:130`, base images `CT/Dockerfile:52`. Owned by Debian and Proxmox.
2. NOT FOUND in source: the end-of-support dates of trixie and Proxmox 9 (grep `end of life|EOL|end-of-life|bookworm|forky` in `AR/11-os-updates.md` and `E/scripts`). The step to Proxmox 10 is named as later work, `AR/11-os-updates.md:725`.
3. The operator and the product; no date-driven plan found in the documents read.
4. After the series ends: no security updates for host and guest; a new install on a newer Proxmox is refused in byo mode and only warned in appliance mode, `E/scripts/felhom-host-install.sh:2146-2152`.
5. NOT FOUND (same grep as line 2).
6. Yes while the archives still serve trixie. Pinned OS updates fall back to snapshot.debian.org in `add_snapshot_sources` `AG/configs/felhom-os-apply:1451-1462`; if that source stops, the update is refused.
7. Refused update, box unchanged, `AG/configs/felhom-os-apply:1125-1127`.
8. SOURCE `E/scripts/felhom-host-install.sh:2146-2152`; the dates: NOBODY KNOWS.
### 18. Catalog app images and their recorded digests
1. 65 app templates under `CAT/templates/`; 84 different image references; registries docker.io 47, ghcr.io 27, lscr.io 8, gitea.dooplex.hu 1, registry.gitlab.com 1. The update ladders record 149 different digests, e.g. `CAT/templates/actualbudget/.felhom.yml:196`. Owned by each app's publisher.
2. NO END DATE in source. The end is the day a registry stops serving a tag or a digest. Moving tags are banned: `BANNED_TAGS` `CAT/scripts/check-image-pins.py:28`.
3. The product, when it tests a new version and records its digest. `CAT/scripts/check-currency.py:22` reports how far behind a pin is and gates nothing.
4. The box renders `tag@sha256` in `RenderWithLadderDigests` `CT/internal/stacks/digest.go:94`; a digest the registry no longer serves makes the pull fail on every box, `AR/09-update-architecture.md:391`. A household cannot install or update that app; a running app keeps running.
5. NOT FOUND: a check that every recorded digest is still served (grep `manifest inspect|still served|HEAD` in `CAT/scripts`). `CAT/scripts/check-currency.py:22` reads tags, not old digests.
6. Running apps come back (images are local). The first failing step is the next install, update or restore that must pull, `CT/internal/stacks/digest.go:10-23`.
7. The pull fails and the app stays on its current image; no automatic repair. Retry happens only when the household presses again.
8. DOC `AR/09-update-architecture.md:391`; SOURCE `CT/internal/stacks/digest.go:94`; which digests are still served today: NOBODY KNOWS.
### 19. Images the controller itself starts (Traefik, tunnel, file browser, Samba)
1. Four fixed images in the controller binary: `TraefikImage` `CT/internal/infra/infra.go:33`, `CloudflaredImage` `CT/internal/infra/infra.go:34`, `FileBrowserImage` `CT/internal/infra/infra.go:35`, `SambaImage` `CT/internal/infra/infra.go:50`. Product-owned pins of other people's images.
2. NO END DATE in source; tags only, no digest. Cloudflare's support window for an old cloudflared: NOT FOUND (grep `one year|1 year|12 months|support.*cloudflared` in `AR/01-topology-and-trust.md`).
3. The product, by a controller release that names newer tags.
4. Candidate: if Cloudflare stops accepting an old tunnel client, the household's address stops answering from outside though the box is healthy. If a registry drops a tag, a rebuilt box cannot start that part.
5. NOT FOUND (same grep as line 2).
6. A box off for a long time comes back with the old images it already holds; whether the tunnel still connects with an old client: NOBODY KNOWS.
7. Not traced.
8. SOURCE `CT/internal/infra/infra.go:33-50`; NOBODY KNOWS for the tunnel client's age limit.
### 20. Published agent versions: only the newest five stay downloadable
1. The registry of agent binaries. Rule file `generic_versions_kept` `AG/scripts/retention-policy.json:23`. Operator-owned; he prunes by hand, `AG/scripts/retention-policy.json:7-10`.
2. Newest 5 versions plus two pins: `hub_min_agent` `AG/scripts/retention-policy.json:30` and `day0_agent` `AG/scripts/retention-policy.json:31`. Decision D50, `AR/09-update-architecture.md:1370-1375`.
3. The session that records a vouch or a new floor updates the pins in the same commit, `AG/scripts/retention-policy.json:12-15`.
4. A box told to install a version that was pruned cannot download it. A pin that names a missing version is a CI failure, `AG/scripts/retention-policy.json:14-15`. Deleting a git tag also ends the ability to install that version, `AG/scripts/retention-policy.json:20-21`.
5. The operator, through CI: `AG/scripts/retention-policy.json:25`.
6. A box off for long is on an old agent; it updates only by a fresh signed job that names a version still kept. First failing step otherwise: the download in the wrapper, `SELFUPDATE_OP` `AG/configs/felhom-os-apply:180`.
7. Not traced on a failed download.
8. SOURCE `AG/scripts/retention-policy.json:23-31`; DOC `AR/09-update-architecture.md:1370-1375`.
### 21. Golden image age and the waiver file
1. The ready-made guest image a new box is installed from, baked weekly, `AR/09-update-architecture.md:1010-1018`. Operator vouches. Gate: `E/scripts/golden_currency_gate.py:169`.
2. `GOLDEN_MAX_AGE_DAYS = 7` `E/scripts/golden_currency_gate.py:169`. A waiver may span at most `WAIVER_MAX_DAYS = 14` `E/scripts/golden_currency_gate.py:167`, checked in `read_waiver` `E/scripts/golden_currency_gate.py:329-333`; path `WAIVER_PATH` `E/scripts/golden_currency_gate.py:162-163`.
3. The operator, weekly (Sunday night per `AR/09-update-architecture.md:1010-1018`).
4. The gate goes red for the product team; no household is touched. A new install gets an older image and must update more on first start.
5. The operator, by the gate itself; no earlier warning found.
6. Not about a box.
7. A waiver that has run out is no longer honoured, `E/scripts/golden_currency_gate.py:329-333`; no waiver file exists in the clones today (searched by file name `golden-waiver`).
8. SOURCE `E/scripts/golden_currency_gate.py:162-169`; tests exist in `E/scripts/test_golden_currency_gate.py:1` (names not matched to this rule).
### 22. Debian container template the golden is built from
1. The upstream Debian LXC template: `TEMPLATE` default `debian-13-standard_13.1-2` `AG/configs/build-golden.sh:67`, used in `pct create` `AG/configs/build-golden.sh:101`. Owned by Proxmox's template mirror.
2. NO END DATE in source; the end is the day the mirror drops this exact file name.
3. The operator, by changing the default at a bake.
4. The weekly bake fails; no household is touched until a fresh golden is needed.
5. NOT FOUND (grep `pveam|TEMPLATE` in `AG/configs/build-golden.sh` shows only lines 23, 67, 79, 101: the script does not download or check the template).
6. Not about a box.
7. The bake stops at `pct create` `AG/configs/build-golden.sh:101`.
8. SOURCE `AG/configs/build-golden.sh:67`.
### 23. Build-time pins of the controller image
1. What the controller image is built from: Go image `CT/Dockerfile:11`, Debian base `CT/Dockerfile:52`, restic `RESTIC_VERSION` `CT/Dockerfile:55` with checksums `CT/Dockerfile:56-57` and a vendored release key `CT/Dockerfile:58`, Docker's apt key fetched at build `CT/Dockerfile:110`. Product-owned.
2. NO END DATE in source. The base images are pinned by digest, so the end is the day the registry stops serving that digest. The vendored restic key's own expiry: NOT FOUND (not opened; key files are not read by this helper).
3. The product, at a release.
4. A new controller release cannot be built; boxes keep the image they have. A version number is pushed once, probe at `CT/build.sh:113-125`.
5. Nobody before; the build fails.
6. Not about a box.
7. The build fails with nothing pushed.
8. SOURCE `CT/Dockerfile:52-58`.
### 24. Kept copies made before a restore: deleted after 30 unused days
1. The copy of an app's data the box keeps when the household restores an older state. Household's data, product rule D62. `restoreCopyUnusedAge` `CT/internal/backup/restore_copy_sweep.go:29`.
2. `restoreCopyUnusedAge = 30 * 24 * time.Hour` `CT/internal/backup/restore_copy_sweep.go:29`.
3. Nobody; using the copy is what keeps it.
4. The household loses the way back to the state from before the restore.
5. NOT FOUND a warning before the sweep (grep `restoreCopyUnusedAge` in `CT/internal`).
6. Candidate: a box off for more than 30 days deletes the copy at the first sweep after start, since age is counted by the wall clock, `CT/internal/backup/restore_copy_sweep.go:29`. A clock tolerance exists elsewhere: `KeptClockSkew` `CT/internal/backup/kept_load.go:217`.
7. Not traced.
8. SOURCE `CT/internal/backup/restore_copy_sweep.go:29`.
## Tiny
- Mail relay's in-memory self-signed leaf, 10 years, remade at each start: `CT/internal/mailrelay/server.go:386`.
- Language cookie, 365 days: `CT/internal/web/i18n_web.go:478`.
- HSTS max-age one day: `HSTSMaxAgeSeconds` `CT/internal/infra/infra.go:108`.
- A backup counts as fresh enough for an update for 24 hours: `DefaultUpdateBackupMaxAge` `CT/internal/config/config.go:70`.
- Night-backup banner turns stale after 26 hours: `StaleAfter` `CT/internal/nightchain/banner.go:23`.
- Operator action brake 30 minutes and forget after 24 hours: `opBrakeRefusal` `AG/internal/localapi/opactions.go:42`.
- Backup-cap refusal held 24 hours: `backupCapRefusalHold` `AG/internal/localapi/backup_cap.go:38`.
- Reboot notice shown 24 hours: `opRebootNoticeFor` `AG/internal/localapi/opreboot.go:48`.
- Controller floor is kept in memory only: `floor` `CT/internal/selfupdate/updater.go:71`.
- Catalog sync uses a git user and token from config, no end date in code: `CT/internal/sync/sync.go:399`.
- Operator SSH key file comes from the hub's desired state, no end date seen: `AuthKeysUserPath` `AG/internal/felhomsshd/manager.go:20`.
- Kernel pin file for host updates: `KERNEL_PIN_FILE` `AG/configs/felhom-os-apply:151`.
## Not found
- Proxmox's own certificate renewal and any re-pin after it: grep `pve-ssl|pveproxy cert|updatecerts|certificate.*(renew|expire)` in `AG/`, `E/scripts`, `AR/`, `RB/`.
- A warning before the web certificate ends: grep `cert.*expir|expir.*cert|NotAfter|acme\.json` in `CT/internal`.
- Cloudflare origin certificates: no use found; the box uses ACME only (grep `origin.?cert|origin_ca` in `CT/internal`, `AG/internal`).
- An executor for operator-key rotation: grep `OpRepin|repin_key|key_rotation|OpKeyRotat|rotate_operator` in `AG/`.
- End dates of the Docker, Debian and Proxmox apt keys, and of trixie and Proxmox 9: grep `expire|valid-until|EOL|end of life` in `AG/configs`, `E/scripts`, `AR/11-os-updates.md`.
- The PBS server certificate's validity: the pin ignores the date, `AG/internal/pbs/pin.go:24`.
- Retained (superseded) escrow keys' lifetime on the box side: grep `retain.*key|superseded` in `AG/internal`, `CT/internal` shows the agent recognising an earlier package, `AG/internal/localapi/escrow_recover.go:122-129`, but no age rule; see the hub helper.
- Expiry of the Cloudflare DNS token used for the web certificate: grep `CFAPIToken` in `CT/internal` shows use only.
@@ -0,0 +1,379 @@
# End-dates list: hub, operator tooling and document-only things (helper result, candidates only)
### 1. Self-bind (connect) link in the household's e-mail
1. A one-use link that lets the household tie a new box to its account. Household uses it, product mints it. Stored hashed in the hub table `selfbind_tokens` `HUB/store/store.go:796-806`; minted by `selfBindTTL` code `HUB/web/selfbind_mint.go:20`.
2. 7 days after minting: `selfBindTTL` `HUB/web/selfbind_mint.go:20`, passed to the store at `HUB/web/selfbind_mint.go:77`; `Expired` `HUB/store/selfbind.go:39`. Also locks at the 5th wrong answer, `SelfBindMaxAttempts` `HUB/store/selfbind.go:21`.
3. The household itself: the expired page has a button, `handleBindResend` `HUB/web/selfbind.go:384`, at most one new link an hour, `bindResendEvery` `HUB/web/selfbind.go:382`. No renewal before the end. The hub also mints at customer creation and RESET `HUB/web/selfbind_mint.go:97`.
4. The household cannot bind the box from the mail; unknown and expired links show the same "expired" page `HUB/web/selfbind.go:206`. The box itself keeps waiting; nothing on it is lost.
5. Nobody is warned before the end. NOT FOUND: grep `expir|remind` in `HUB/web/selfbind_mint.go:1-110` shows no reminder. The mint log line says "valid 7 days" `HUB/web/selfbind_mint.go:93`.
6. A box first switched on after day 7: the link fails at the first step (opening it); the household presses the resend button. If an old host is still linked no link is sent and the operator gets `eventBindOldHostLinked` `HUB/web/selfbind.go:440`, logged at `HUB/web/selfbind.go:413`.
7. If the mail send fails at mint the token is deleted, so no dead link is left `HUB/web/selfbind_mint.go:82`. No automatic retry; the household presses again after an hour.
8. TEST `TestR719_AnExpiredLinkOffersAndSendsAFreshOne` `HUB/web/selfbind_resend_test.go:21`; SOURCE `HUB/web/selfbind_mint.go:20`; DOC `AR/05-hub-architecture.md:359`.
### 2. Customer claim code (the household's first password step)
1. A code the hub mails so the household can set its own box password. Household uses it; the hub stores a hash and a generation. `RotateClaimCode` `HUB/store/store.go:2084`.
2. NO END DATE: `RotateClaimCode` `HUB/store/store.go:2084` bumps a generation and writes no expiry. It ends when a newer code replaces it. Limit by count, not age: `maxResetPerDay` = 3 `HUB/claim/engine.go:30`.
3. The household asks for a reset; the hub mints a new code. No lead time, because there is no end.
4. After 3 resets in a day the 4th is refused, `maxResetPerDay` `HUB/claim/engine.go:30`, counted by `BumpResetCount` `HUB/store/store.go:2124`. The box side has its own 5-attempt, 15-minute lock, named at `HUB/claim/engine.go:27`.
5. Nobody; there is no end to warn about.
6. Comes back by itself: an old code still works until it is rotated. SOURCE `HUB/store/store.go:2084`.
7. NOT FOUND for a failed code mail: searched grep `expire|ttl|valid` in `HUB/claim/engine.go:1-60`; only the daily count limit is there.
8. SOURCE `HUB/claim/engine.go:30` and `HUB/store/store.go:2084`. No test for "a code never ages" was found (grep `Expire` in the claim package): NOBODY KNOWS whether that is intended.
### 3. Retrieval passphrase: the moment it stops giving keys
1. The household's retrieval passphrase opens the box's keys at the hub only for a while. Household owns it; rule lives in `passphraseKeysClosed` `HUB/api/passphrase_scope.go:46`.
2. Ends on an EVENT, not a date: once the box has a key of its own and uses it, the passphrase gives no keys. Rule text `HUB/api/passphrase_scope.go:22`; needs controller `boxConfigMinController` 0.334.0 `HUB/api/passphrase_scope.go:40`.
3. Not renewed. To open it again the old host is deleted at the hub first (operator act), then a reinstall mints anew. DOC `AR/05-hub-architecture.md:270-272`.
4. A reinstall that reuses the passphrase without the old host deleted is refused; the household's new box gets no keys. Event `eventPassphraseKeysRefused` `HUB/api/passphrase_scope.go:42`.
5. Nobody before; the operator sees the event after a refusal, `eventPassphraseKeysRefused` `HUB/api/passphrase_scope.go:42`.
6. A running box is not touched: it uses its own key. A wiped box coming back fails at the key fetch step until the operator deletes the old host.
7. The refusal changes nothing; the same request works after the host delete. No retry by the hub.
8. TEST `TestR1159_PassphraseGivesKeysOnlyUntilTheBoxUsesItsOwn` `HUB/api/r1159_passphrase_scope_test.go:110`.
### 4. Pairing code and appliance registration (box console code)
1. The 6-character code a new box shows on its screen, and the hub row behind it. Product mints, household types it. `PairingCode` `HUB/store/appliance.go:35`; box side `E/scripts/iso/felhom-bootstrap.sh:50`.
2. NO END DATE: `ApplianceByPairingCode` `HUB/store/appliance.go:155` has no age check. After 7 days without a poll the row only gets a "stale" badge, `applianceStaleAfter` `HUB/web/appliances.go:21`.
3. Nobody; the code stays the same across re-registration `HUB/store/appliance.go:35`.
4. Nothing breaks by age. An operator who discards the row makes the box log "appliance token not recognized" and keep trying `E/scripts/iso/felhom-bootstrap.sh:589`.
5. Operator only, by the stale badge on the appliances page `HUB/web/appliances.go:78`. No mail.
6. Comes back by itself: the box polls every `POLL_INTERVAL` 30 s `E/scripts/iso/felhom-bootstrap.sh:54` and retries the network gate every `GATE_RETRY_INTERVAL` 60 s `E/scripts/iso/felhom-bootstrap.sh:180`, with no give-up found.
7. A failed registration is retried by the unit `E/scripts/iso/felhom-bootstrap.sh:495`.
8. SOURCE `HUB/store/appliance.go:155` and `HUB/web/appliances.go:21`. No test of "never expires" found (grep `Stale|expire` in appliance files).
### 5. "Switched off on purpose" hold (14 days of silence allowed)
1. When a household switches the box off through the product, the hub stays quiet about the silence. Household act, hub rule: `DeliberateOffHold` `HUB/store/deliberateoff.go:18`.
2. 14 days after the switch-off: `DeliberateOffHold` `HUB/store/deliberateoff.go:18`, judged by `DeliberateOffHeld` `HUB/store/deliberateoff.go:21`.
3. Nobody renews. The hold ends by itself when the box reports again or at day 14.
4. After day 14 the silence and missed-backup alarms start again for that household; the operator is told.
5. The operator gets an event at switch-off time naming the moment, message built at `HUB/api/deliberate_off.go:42`. No warning before day 14 was found (grep `DeliberateOffHold` in hub).
6. Yes, it comes back by itself: a box switched on at any time reports and the hold clears. TEST `TestDeliberateOff_BackOnIsQuiet` `HUB/monitor/deliberate_off_test.go:138`.
7. Not applicable: nothing is renewed.
8. TEST `TestDeliberateOff_AlarmsAfterFourteenDays` `HUB/monitor/deliberate_off_test.go:112` and `TestDeliberateOff_NoSilenceAlarmDuringTheHold` `HUB/monitor/deliberate_off_test.go:90`.
### 6. Retained old escrow packages and the 7-day abandon of set-aside off-site data
1. Old sealed key packages the hub keeps after a re-key, and off-site data set aside from a previous install. Household's data, hub keeps it. Table `host_escrow_superseded` `HUB/store/store.go:440`.
2. Packages: NO END DATE, the schema comment says no pruning `HUB/store/store.go:417`. Set-aside data: deleted 7 days after the household asks to abandon it, `DefaultAbandonDelay` `HUB/offsitekeys/service.go:448`.
3. Nobody renews. The household can cancel inside the 7 days; request path `RequestAbandon` `HUB/offsitekeys/service.go:473`.
4. After the abandon runs, the old off-site data and its retained packages are gone for that household, `PurgeAbandonedEscrowForCustomer` `HUB/store/store.go:4360`. A customer delete also purges them `HUB/store/store.go:1985`.
5. NOT FOUND: no reminder before day 7 (grep `DefaultAbandonDelay|remind` in `HUB/offsitekeys/service.go:440-560`). A failure raises the event at `HUB/offsitekeys/service.go:540`.
6. Yes for the box: the sweep runs on the hub, `SweepAbandons` `HUB/offsitekeys/service.go:529`, and needs no box. A box that was off simply finds the old data gone.
7. A failed delete is marked and tried again on the next sweep, with one event `HUB/offsitekeys/service.go:540`.
8. TEST `TestAbandon_DelayCancelAndScope` `HUB/offsitekeys/service_test.go:134`; SOURCE `HUB/store/store.go:417`.
### 7. Off-site clean-up window (the short time a box may delete off-site)
1. A time-boxed permission the hub opens so a box can prune its own off-site store. Product. `windowLength` `HUB/offsitekeys/service.go:263`.
2. 20 minutes per window, `windowLength` `HUB/offsitekeys/service.go:263`; a new one is allowed about weekly, `windowCadence` 6 days 12 hours `HUB/offsitekeys/service.go:264`.
3. The hub closes it; the box asks for the next one on its own night run. No human.
4. If the prune does not finish in 20 minutes it waits a week; the off-site store grows meanwhile. Operator and, in time, the household's off-site space.
5. NOT FOUND for a warning before close (grep `windowLength` in hub). The operator gets an event only when a window cannot be closed, logged at `HUB/offsitekeys/service.go:425`.
6. Yes: a box that was off asks again on its next run. First failing step if the cadence has not passed: the grant is refused.
7. A window past its time that cannot be closed is retried at the next sweep, `SweepExpiredWindows` `HUB/offsitekeys/service.go:413`, log `HUB/offsitekeys/service.go:425`. State left: delete right still open.
8. TEST `TestWindow_LeftOpenIsClosedByTheSweep` `HUB/offsitekeys/service_test.go:99` and `TestWindow_GrantOpenCloseAndDropAlarm` `HUB/offsitekeys/service_test.go:37`.
### 8. Storage Box sub-account password and the hub's seal key
1. Each household's off-site store account at the provider, minted by the hub, and the hub key that seals stored secrets (`OFFSITE_SECRET_KEY`, named in `E/manifests/hub.yaml:167-171`). Operator and product.
2. NO END DATE in code: no lifetime on the sub-account password. A clean reinstall's re-issue resets it, DOC `AR/05-hub-architecture.md:274-279`; `ResetBoxPassword` `HUB/hetznerapi/hetznerapi.go:297`.
3. Operator, by pressing Re-issue. Seal key rotation is "not built", DOC `RB/secrets.md:159`.
4. If the seal key is lost or changed, the old sealed rows cannot be opened, DOC `RB/secrets.md:159`: operator loses stored off-site secrets and break-glass passwords until re-issued per household.
5. Nobody. NOT FOUND: grep `expir|rotat` in `HUB/hetznerapi/hetznerapi.go:1-320` finds no age rule or warning.
6. A box off for a long time comes back by itself as long as its account was not re-issued; it uses its SSH key, the password is used only by the hub, DOC `AR/05-hub-architecture.md:274-279`.
7. NOT FOUND for a half-done reset (searched `ResetBoxPassword` callers only by name). Candidate question for the main session.
8. DOC `RB/secrets.md:159` and `AR/05-hub-architecture.md:274-279`; SOURCE `HUB/hetznerapi/hetznerapi.go:297`. No test of a lifetime, because there is none.
### 9. PBS off-site token on ep0 and its descriptor on the hub
1. The per-household token on the rented endpoint for the whole-guest off-site copy, and the hub record that must agree with it. Product mints, operator can re-issue. DOC `AR/05-hub-architecture.md:373-387`.
2. NO END DATE stated: the lifecycle table has only events (provision, re-key, deprovision) `AR/05-hub-architecture.md:379-387`. One age rule: a staged secret not consumed in 15 minutes is wrong, `unconsumedGrace` `HUB/pbsdrheal/reconciler.go:63`.
3. The hub re-keys on re-issue (delete and recreate) `AR/05-hub-architecture.md:382`. No scheduled renewal.
4. A re-key the box did not pick up leaves the box "serving a revoked credential": the household has no whole-guest off-site copy while both tiers look green, comment at `HUB/pbsdrheal/reconciler.go:55-63`.
5. Operator only, after the fact: `pbsdr_provision_failed` on a final failure, DOC `AR/05-hub-architecture.md:400-402`.
6. Host delete keeps the token `AR/05-hub-architecture.md:384`; a box off and back uses the same token. After a re-issue while it was off, it must consume the new secret on its next report.
7. A failed self-check rolls back and is repeated 3 times, 7 s apart, then raises the operator event, DOC `AR/05-hub-architecture.md:400-402`.
8. DOC `AR/05-hub-architecture.md:373-403`; SOURCE `HUB/pbsdrheal/reconciler.go:63`. Lifetime of the token on the endpoint itself: NOBODY KNOWS from these sources (ep0 not read).
### 10. WireGuard peers (the box's tunnel to the off-site endpoint)
1. One peer per box on the endpoint, registered through the hub; the private key is in the household's sealed bundle. Product. DOC `AR/06-offsite-connectivity.md:172-182`.
2. NO END DATE: the hub store only removes a peer on request, `RemoveWGPeer` `HUB/store/wg.go:345`. Scheduled key rotation is an open point, DOC `AR/06-offsite-connectivity.md:368`.
3. Nobody on a schedule. The agent makes a new key itself only after a handshake stamped in the future for 15 minutes, DOC `AR/06-offsite-connectivity.md:205-207`.
4. Revocation: the box is off the off-site network at once, DOC `AR/06-offsite-connectivity.md:180-182`; the household's off-site copies stop.
5. Nobody before. NOT FOUND: grep `expire|rotat|stale|handshake` in `HUB/store/wg.go:1-360` finds no age rule.
6. Yes, a box off for months keeps its peer. A revoked box never re-registers on its own, DOC `AR/06-offsite-connectivity.md:200-205`.
7. A failed fetch is never read as a teardown, DOC `AR/06-offsite-connectivity.md:207-208`. Re-add is an operator call.
8. DOC `AR/06-offsite-connectivity.md:200-208`; SOURCE `HUB/store/wg.go:345`. Box-side tests belong to the box helper.
### 11. Signed jobs (agent update, config bundle, OS step)
1. An operator-signed order the hub queues for one box. Operator signs with `felhom-opsign`; hub stores an opaque blob, `EnqueueSignedJob` `HUB/store/store.go:3764`; the agent checks it.
2. 30 minutes by default: flag `ttl` `AG/cmd/felhom-opsign/main.go:57`, end set at `AG/cmd/felhom-opsign/main.go:130`. Never more than 24 hours: `MaxWindow` `AG/internal/authz/verifier.go:26`, refused at signing `AG/cmd/felhom-opsign/main.go:126`.
3. The operator (or CC, see block 12) signs a new job. Nothing renews by itself.
4. A box that does not poll inside the window does not get the update or the root-file change; the operator must sign again for that box.
5. Nobody before. The agent logs the rejection and clears the job `AG/internal/signedjobs/runner.go:163`. The hub has no expiry of its own on the queue (grep `expire` near `HUB/store/store.go:3764`).
6. No. A box off past the end polls, gets the job, and fails at the time check `AG/internal/authz/verifier.go:161`. Clock skew allowed: `DefaultClockSkew` 2 minutes `AG/internal/authz/verifier.go:21`.
7. A rejected job is cleared; a job that passes the check but fails to run is also cleared and must be re-signed, default branch at `AG/internal/signedjobs/runner.go:186`. No retry.
8. TEST `TestVerify_RejectsPerStage` `AG/internal/authz/verify_test.go:82`; DOC `AR/04-control-plane-authorization.md:78`.
### 12. Operational signing key and cold recovery key
1. The two keys every agent trusts for destructive orders. Operator owns them; both live on DooPlex today, DOC `AR/04-control-plane-authorization.md:111` and `AR/04-control-plane-authorization.md:96`.
2. NO END DATE on either key in any source. The only dated rule: "Revisit on the first sale", in §3.1 starting `AR/04-control-plane-authorization.md:111`.
3. The operator. Planned rotation is a key-signs-key order `AR/04-control-plane-authorization.md:143`; rotating a signer is described as not built `AR/04-control-plane-authorization.md:133`.
4. Both keys lost: no update, no config bundle, no OS step for any box until someone visits each box, DOC `AR/04-control-plane-authorization.md:148`. Every household.
5. Nobody; there is no date. The loss case is in the recovery runbook row S6 `RB/total-loss-of-dooplex.md:39`.
6. Yes. A box off for any time still trusts the same pinned keys. If the key was rotated while it was off: NOT FOUND how it learns the new key (searched §4 `AR/04-control-plane-authorization.md:137-149`).
7. NOT FOUND: no source says what state a half-done re-pin leaves (searched §4 `AR/04-control-plane-authorization.md:137-149`).
8. DOC `AR/04-control-plane-authorization.md:137-149`. Rotation never run on a real box as far as these files say: NOBODY KNOWS.
### 13. Operator actions waiting for a box (24 hours; reboot 30 minutes)
1. Buttons the operator presses in the hub that a box must answer (restart, reboot and so on). Operator. `OperatorActionTTL` `HUB/store/opactions.go:62`.
2. 24 hours unanswered: `OperatorActionTTL` `HUB/store/opactions.go:62`. A host reboot order: 30 minutes, `HostOperatorActionRebootTTL` `HUB/store/opactions.go:47`.
3. The operator presses again. No renewal.
4. The action is marked expired with "did not answer within a day" `HUB/store/opactions.go:222` and `HUB/store/opactions.go:296`; nothing happens on the box.
5. Nobody before; the operator sees the expired state on the hub page.
6. Safe by design: a box back after the end does not run the stale order. The operator presses again.
7. Not applicable; an expired action is a final state.
8. DOC `AR/04-control-plane-authorization.md:184` names `TestOpActions_ClosedList` and `TestOperatorActions_ClosedList`; SOURCE `HUB/store/opactions.go:62`.
### 14. Recovery mode for a host (the door a rebuilt box uses)
1. A switch the operator sets on a host so a rebuilt box may re-enrol and fetch its restore data. Operator, global key only, `handleSetRecoveryMode` `HUB/api/dr.go:34`.
2. 30 minutes by default, `defaultRecoveryTTL` `HUB/api/dr.go:21`; at most 4 hours, `maxRecoveryTTL` `HUB/api/dr.go:22`; clamp at `HUB/api/dr.go:52`.
3. The operator sets it again. No lead time.
4. A restore that takes longer than the window: the box's re-enrol call is refused and the household's recovery stops until the operator re-opens it.
5. Nobody. NOT FOUND: grep `defaultRecoveryTTL|maxRecoveryTTL` shows no warning before the end.
6. No: a box that starts its recovery after the window fails at the re-enrol step. TEST `TestReEnroll_GatedOnRecoveryMode` `HUB/api/dr_test.go:14`.
7. Nothing is left half-done by the expiry itself; the operator opens a new window.
8. TEST `TestRestoreDirective_GatedAndExpires` `HUB/api/dr_test.go:70`; SOURCE `HUB/api/dr.go:21-22`.
### 15. Break-glass credential (the host's emergency root password)
1. A per-host emergency password, sealed in the hub, shown only to a logged-in operator. Operator. `SaveHostRecoveryCredential` `HUB/store/host_recovery.go:28`; reveal `HUB/web/hosts.go:983`.
2. NO END DATE: last write wins, day-0 sets it and a rotate re-sets it `HUB/store/host_recovery.go:20`. Paper rule: re-print the sheet "when a key rotates, and yearly" `RB/break-glass-sheet.md:75`.
3. The operator, yearly by the paper rule; no code reminds him.
4. If the sealed copy cannot be opened (seal key, block 8) or the paper is stale, the operator has no way into a host whose agent is dead.
5. Nobody before. Each reveal writes the event `recovery_credential_revealed` `HUB/web/hosts.go:1010`.
6. Yes, the password does not age on the box.
7. NOT FOUND what a half-done rotate leaves (hub updated, box not, or the reverse); searched `HUB/store/host_recovery.go:20-43` only.
8. SOURCE `HUB/store/host_recovery.go:20-28`; session-only route `HUB/web/routes.go:130`; DOC `RB/break-glass-sheet.md:75`. A yearly re-print has no record: NOBODY KNOWS.
### 16. Vouched artifact manifest and the "golden not vouched" alarm
1. The hub's list of what a fresh install may fetch (agent, golden, checksums), vouched by the operator. Read by `resolve_artifacts` `E/scripts/felhom-host-install.sh:467`.
2. NO END DATE on a vouch. An age rule on the gap: a published golden not vouched after 2 hours raises an alarm, `goldenVouchGrace` `HUB/monitor/golden_vouch.go:42`, checked every 15 minutes `HUB/monitor/golden_vouch.go:43`.
3. The operator vouches; CC never does, DOC `AR/09-update-architecture.md:1022`.
4. An old vouch means a fresh install gets the older controller, message at `HUB/monitor/golden_vouch.go:92`. If the manifest is not served the install dies `E/scripts/felhom-host-install.sh:472`.
5. The operator, 2 hours after the publish, by the event built at `HUB/monitor/golden_vouch.go:92`. If Gitea cannot be read there is no verdict `HUB/monitor/golden_vouch.go:65`.
6. A new box after a long time installs what is vouched and then updates to the floor; a vouched version already pruned from the registry fails at the download (see block 18).
7. An artifact with no expected checksum is refused `E/scripts/felhom-host-install.sh:531`; the bootstrap unit retries the install every 30 s `E/scripts/iso/felhom-bootstrap.sh:430`.
8. TEST `TestGoldenVouchChecker` `HUB/monitor/golden_vouch_test.go:15`; SOURCE `E/scripts/felhom-host-install.sh:467-472`.
### 17. The golden's age rule and its waiver file
1. A gate that fails a push when the newest release is ahead of the newest baked golden and that golden is old. Operator and CC. `GOLDEN_MAX_AGE_DAYS` `E/scripts/golden_currency_gate.py:169`.
2. Golden: 7 days from the date in its evidence folder name, `GOLDEN_MAX_AGE_DAYS` `E/scripts/golden_currency_gate.py:169`. Waiver: its own `expires` line, at most 14 days after issue, `WAIVER_MAX_DAYS` `E/scripts/golden_currency_gate.py:167`.
3. Operator bakes and vouches weekly, Sunday night, DOC `AR/09-update-architecture.md:997`. A waiver is renewed by editing two dates, DOC `RB/RUNBOOK-manual-build.md:291`.
4. The gate goes red and blocks pushes for the operator and CC; households are not touched. Expired check at `E/scripts/golden_currency_gate.py:354`.
5. The gate prints the age on every run `E/scripts/golden_currency_gate.py:422`. No warning before day 7 or before the waiver's end was found.
6. Not a box matter.
7. A waiver that is malformed or too long is refused out loud (exit 2), `read_waiver` `E/scripts/golden_currency_gate.py:287`. Candidate: the waiver file `WAIVER_PATH` `E/scripts/golden_currency_gate.py:162` is absent in this clone, and the newest evidence folder under documentation/tests is dated 2026-11-03, 8 days before today.
8. SOURCE `E/scripts/golden_currency_gate.py:162-169`; DOC `AR/09-update-architecture.md:1362` and `RB/RUNBOOK-manual-build.md:279`. Test names in the gate's test file NOT FOUND (grep `def test` returned nothing).
### 18. Which agent versions stay downloadable (D50)
1. The rule for which published agent packages are kept in the registry. Operator prunes by hand; CI checks. File `AG/scripts/retention-policy.json:23-32`.
2. By count, not date: newest 5 (`generic_versions_kept` `AG/scripts/retention-policy.json:23`) plus two pins, `hub_min_agent` and `day0_agent` `AG/scripts/retention-policy.json:30-31`. Recorded 2026-11-10 `AG/scripts/retention-policy.json:27`.
3. The session that records a vouch or a new floor updates the pins in the same commit, rule text `AG/scripts/retention-policy.json:12-15`.
4. A box on a version outside the set cannot fetch its own binary again, DOC `AR/09-update-architecture.md:1370-1373`. A deleted git tag ends the ability to install that version `AG/scripts/retention-policy.json:20-21`.
5. CI only: `AG/scripts/check-published-versions.py:102` reads the file. No warning to anyone when a box's version is about to fall out of the newest 5 was found.
6. A box off while 5 newer versions ship comes back and updates forward by a signed job; what it does if it needs its OLD binary: fails at the download, per `AR/09-update-architecture.md:1372`.
7. A pin naming a version with no tag is a CI failure, never a skip `AG/scripts/retention-policy.json:14-15`.
8. DOC `AR/09-update-architecture.md:1370-1376`; SOURCE `AG/scripts/retention-policy.json:23-32`. The pins are hand copies of hub state `AG/scripts/retention-policy.json:12`: whether they match today, NOBODY KNOWS from files.
### 19. The installer ISO (Proxmox base, installer version, first boot)
1. The USB image a household or volunteer installs from. Product; built by `E/scripts/iso/build-felhom-iso.sh:51`, served from Cloudflare R2, DOC `RB/total-loss-of-dooplex.md:125`.
2. NO END DATE in code. `ISO_VERSION` is 1.30.0 `E/scripts/iso/build-felhom-iso.sh:51`. The Proxmox ISO and its checksum are build ARGUMENTS, not pinned: `E/scripts/iso/build-felhom-iso.sh:194`; the example names 9.2-1 `E/scripts/iso/README.md:40`.
3. The operator rebuilds. No cadence found (grep `days|stale|too old` in `RB/iso-release-gate.md:1-230` returned nothing on age).
4. An old ISO installs an old Proxmox base; the install script itself is fetched fresh at first boot (comment block at `E/scripts/iso/build-felhom-iso.sh:44-52`), so the hub decides the agent and golden.
5. Nobody. The build warns only if golden and floor are not given: unenforced branch `E/scripts/iso/build-felhom-iso.sh:79`.
6. A too-old ISO on first boot: registers, pairs, fetches the current install script; first failing step would be that fetch or the manifest `E/scripts/felhom-host-install.sh:472`. No "ISO too old" check found (grep `too old|min.*version` in bootstrap).
7. A failed install is retried every 30 s `E/scripts/iso/felhom-bootstrap.sh:430`; a failed script fetch is retried `E/scripts/iso/felhom-bootstrap.sh:396`.
8. SOURCE `E/scripts/iso/build-felhom-iso.sh:222` (checksum of the given ISO is verified). How old an ISO still installs cleanly: NOBODY KNOWS. Build image is `FROM debian:trixie`, not pinned `E/scripts/iso/Dockerfile.assistant:10`.
### 20. The certificate the hub's address serves
1. The TLS certificate for the hub's public name, used by every box and the operator. Operator's cluster issues it: annotation `E/manifests/hub.yaml:342`, stored under the secret name at `E/manifests/hub.yaml:360`.
2. NOT FOUND in these sources: the issuer `letsencrypt-prod` is only named `E/manifests/hub.yaml:342`; its definition and renewal timing are in the DooPlex cluster repository, outside this task. Searched grep `ClusterIssuer|renewBefore|duration` in E/manifests.
3. cert-manager on DooPlex, by the annotation `E/manifests/hub.yaml:342`. Lead time NOT FOUND (same search).
4. If it lapses: boxes that pin or verify the hub cannot report or fetch config; households get no alarms and no updates; the operator page shows a browser warning.
5. NOT FOUND: grep `cert|expir` in `HUB/monitor/golden_vouch.go:1-100` and across HUB/monitor finds no certificate-age alarm.
6. A box off past a renewal: depends on what it pins (box helper's item). From hub files: NOT FOUND.
7. NOT FOUND. The renewal needs the challenge path reachable; the ingress is limited to two countries in the block starting `E/manifests/hub.yaml:342`. Candidate question: does that block the issuer's check.
8. SOURCE `E/manifests/hub.yaml:342` and `E/manifests/hub.yaml:360` only. Dates: NOBODY KNOWS from files; needs a measurement.
### 21. Operator login session on the hub
1. The operator's browser session cookie. Operator. `SessionCookieName` `HUB/web/server.go:930`; sessions are held in memory.
2. 7 days from login: session end `HUB/web/server.go:949`, cookie `MaxAge` `HUB/web/server.go:964`. Also ends at any hub restart, because the map is in memory (cleanup `CleanupSessions` `HUB/web/server.go:199`).
3. The operator logs in again. No sliding renewal found: the check only compares the fixed end `HUB/web/server.go:901`.
4. Session-only buttons (delete, reset, reveal, tokens) stop until he logs in; route rule example `HUB/web/routes.go:130`. Scripts with a token are not touched.
5. Nobody; the next click shows the login page.
6. Not a box matter.
7. Not applicable. Basic auth with the password still works as a second door `HUB/web/server.go:909`.
8. TEST `TestR136_LoginSetsHostPrefixedCookie` `HUB/web/r136_host_cookie_test.go:16` (cookie name); the 7 days are SOURCE only `HUB/web/server.go:949`.
### 22. Operator access tokens, among them CC's hub token
1. Named, scoped tokens for scripts, shown once, stored hashed. Operator creates them in a browser session; CC holds one, with `overview:read` and `floor:write`, DOC `AR/09-update-architecture.md:1022`.
2. 90 days by default, `tokenDefaultDays` `HUB/web/tokens.go:29`; at most 365, `tokenMaxDays` `HUB/web/tokens.go:30`; "never" is not offered `HUB/web/tokens.go:254`. End computed at `HUB/web/tokens.go:180`.
3. The operator makes a new token and puts it in CC's store. There is no renew; no lead time is set anywhere.
4. CC's floor raise and overview reads get 401 `HUB/web/tokens.go:95`; a controller release cannot be carried to boxes by CC until the operator acts. Households wait for the update.
5. Nobody is warned. NOT FOUND: grep `token.*(expiring|days left)|token_expir` in the hub finds only comments. The page shows the date and the state "expired" `HUB/web/tokens.go:245`.
6. Not a box matter.
7. Not applicable; an expired token stays listed as expired. Column `expires_at` `HUB/store/api_tokens.go:52`.
8. TEST `TestToken_NoTokenUnknownExpiredRevoked401` `HUB/web/tokens_test.go:310`; DOC `AR/05-hub-architecture.md:655-669`. The end DATE of CC's token is in no document (grep `HUB_TOKEN` in AR, RB, operations, STATUS): NOBODY KNOWS without the hub page.
### 23. A household's own Cloudflare token (hub side)
1. A household that brings its own domain gives a Cloudflare API token; the hub checks what it can reach before saving. Household owns it. `HUB/cloudflare/reach.go:14`.
2. NO END DATE known to the hub: the check at `HUB/cloudflare/reach.go:14` is about zones, not age. Whatever end the household set at Cloudflare is not read (grep `expire|rotat` in the cloudflare package: nothing).
3. The household, at Cloudflare; then the operator saves the new token in the hub.
4. If it lapses, the household's remote address stops being managed; candidate: first sign is a failed DNS or tunnel call on the box.
5. Nobody. NOT FOUND: same grep, no expiry field stored.
6. Box helper's item; from hub files NOT FOUND.
7. NOT FOUND.
8. SOURCE `HUB/cloudflare/reach.go:14`; the rest NOBODY KNOWS from hub files.
### 24. The mail service key (all household and operator mail)
1. The key the hub uses to send every mail: alarms, bind links, claim codes. Operator's account at the mail service; named in `E/manifests/hub.yaml:148-152`, value not opened.
2. NO END DATE stated in any document read: `RB/secrets.md:33` describes create and rotate, with no date; inventory row `AR/_recovery-inventory-2026-07-28.md:502`.
3. The operator re-mints at the service dashboard, DOC `RB/total-loss-of-dooplex.md:47`.
4. All mail stops: no alarm reaches the operator or a household, no bind link, no claim code. Everyone.
5. NOT FOUND: no check that mail still goes out on a schedule was found in the documents read (grep `expir|renew` in RB/secrets.md and STATUS.md).
6. Not a box matter; boxes send through the hub.
7. A failed bind mail deletes its token `HUB/web/selfbind_mint.go:82`; for other mail kinds NOT FOUND in this pass.
8. DOC `RB/secrets.md:33` and `RB/total-loss-of-dooplex.md:47`. Whether the key or the plan has an end date: NOBODY KNOWS from files.
### 25. The hub's global bearer key
1. The operator-tier key for the hub API (recovery mode, peer re-add). Operator. Named in `E/manifests/hub.yaml:157-161`; inventory row `AR/_recovery-inventory-2026-07-28.md:501`.
2. NO END DATE. Rotation is a supervised runbook `RB/secrets.md:124`.
3. The operator, with an ordered procedure `RB/secrets.md:124`; no cadence stated.
4. A rotation done out of order locks out whatever still holds the old key; DOC `RB/secrets.md:124` calls the order load-bearing.
5. Nobody; no date.
6. Boxes use per-box keys bound to their box since 2026-10-12, DOC `AR/13-threat-model.md:70`; a box still holding only the old global key after a rotation: NOT FOUND what it does.
7. NOT FOUND beyond the runbook order.
8. DOC only. Two documents disagree on whether it was rotated: "rotated" at `AR/13-threat-model.md:70`, "when last rotated" unknown at `AR/_recovery-inventory-2026-07-28.md:1030`. Candidate for a doc fix.
### 26. Pinned tool versions in builds and CI
1. Versions of the tools that build and check the product. Operator and CC. Hub build image `E/hub/Dockerfile:3`, run image `E/hub/Dockerfile:24`, site publisher `E/scripts/publish-site-cloudflare.sh:28`.
2. NO END DATE in code. Hub images are pinned by digest `E/hub/Dockerfile:3`; the ISO helper image is the moving tag at `E/scripts/iso/Dockerfile.assistant:10`; the mailer image is `latest` at `E/manifests/contact-mailer.yaml:54`.
3. Whoever bumps the pin. The nightly security watch runs at 03:40 UTC `E/.gitea/workflows/security-watch.yml:22`; nightly gates at 03:20 `E/.gitea/workflows/gates.yml:27`.
4. A pinned digest removed upstream breaks the hub build for the operator; a moving tag changes the ISO build without a commit.
5. The security watch treats a release line with no fixed version as end of life, `E/scripts/security_watch.py:292`; it reports to the operator.
6. Not a box matter.
7. A CI job that does not run leaves no result; whether a missed nightly is noticed: NOT FOUND in the workflow file `E/.gitea/workflows/gates.yml:27`.
8. SOURCE `E/hub/Dockerfile:3` and `E/scripts/security_watch.py:292`. Support end dates of Go, Alpine, wrangler: no document states one.
### 27. Domains, registrars and the home dynamic-DNS name
1. NOT MEASURED. The product's two domains and the dynamic name the hub's address points to. Operator. Listed as row S13 `RB/total-loss-of-dooplex.md:46`; the hub name is a CNAME to the dynamic name `RB/total-loss-of-dooplex.md:73`.
2. NOT FOUND: no document gives a registrar name or a renewal date. Searched grep `expir|renew|valid until|until 20` in documentation/operations, RB, AR and STATUS.md.
3. The operator. NOT FOUND how long before.
4. If a domain or the dynamic name lapses, every box loses the hub (reports, alarms, updates, bind links) and the site and installer disappear. Every household.
5. NOT FOUND. Presumably the registrar's own mails to the operator's mailbox, row S12 `RB/total-loss-of-dooplex.md:45`; no document says so.
6. A box comes back by itself once the name resolves again; while it does not, the first failing step is the hub lookup.
7. NOT FOUND.
8. DOC `RB/total-loss-of-dooplex.md:46`. WHERE THE OPERATOR READS THE DATE: NOT FOUND. NOBODY KNOWS from files.
### 28. Cloudflare account, the site publish token and the ISO store keys
1. NOT MEASURED. The account that serves DNS for both domains, the site, the installer script and the ISO store. Operator. Row S11 `RB/total-loss-of-dooplex.md:44`; site and ISO there since 2026-10-30 `RB/total-loss-of-dooplex.md:125`.
2. NOT FOUND: no end date for the account, the token `CF_PAGES_TOKEN` (named at `E/scripts/publish-site-cloudflare.sh:12`) or any R2 key. Searched grep `R2_ACCESS|R2 key|expir` in AR, RB, operations.
3. The operator at the Cloudflare dashboard.
4. Token lapse: CC cannot publish the site or installer script; the old one stays served. Account trouble: DNS, site, ISO downloads and households' tunnels, per row S11 `RB/total-loss-of-dooplex.md:44`.
5. NOT FOUND.
6. A new box needs the installer script from the site at first boot `E/scripts/iso/felhom-bootstrap.sh:396`; it retries until it is served.
7. A failed publish leaves the previous site live (candidate; not stated in the script header `E/scripts/publish-site-cloudflare.sh:12`).
8. DOC `RB/total-loss-of-dooplex.md:44`. WHERE THE OPERATOR READS THE DATE: NOT FOUND.
### 29. Hetzner account, the rented endpoint and the storage boxes
1. NOT MEASURED. The paid account behind ep0 (tunnel endpoint and off-site store) and the Storage Boxes. Operator. Row S8 `RB/total-loss-of-dooplex.md:41`; API token named in `E/manifests/hub.yaml:222-226`.
2. NOT FOUND: no document gives a billing date, a contract end or an API token end. Searched grep `expir|renew|invoice|billing` in RB, AR, operations.
3. The operator; paid monthly by the provider's own cycle, not stated in any document read.
4. If the account is closed or unpaid: every household's off-site copies and the PBS tunnel stop; the hub cannot mint or reset sub-accounts.
5. NOT FOUND. Hub-side sign would be the off-site stale alarm after 8 days, `offsiteBackupStaleAfter` `HUB/monitor/deadline_tiers.go:28`.
6. A box comes back by itself when the endpoint is back; first failing step is the off-site push.
7. NOT FOUND.
8. DOC `RB/total-loss-of-dooplex.md:41`. WHERE THE OPERATOR READS THE DATE: NOT FOUND.
### 30. Tailscale node keys (operator's remote way in)
1. NOT MEASURED. The operator's private network to DooPlex and the two demo boxes. Operator. Table `E/documentation/operations/tailscale.md:11-13`.
2. Document date: the HP demo box's key expires 2027-01-17, written as OPEN at `E/documentation/operations/tailscale.md:41`. The other two have expiry disabled `E/documentation/operations/tailscale.md:11-12`.
3. The operator, in the admin console (Machines, Disable key expiry) `E/documentation/operations/tailscale.md:43`. No lead time set.
4. That node drops off the private network `E/documentation/operations/tailscale.md:91`; the operator loses remote reach to it. No household is on it.
5. NOT FOUND: no reminder is named. WHERE THE OPERATOR READS THE DATE: the Machines page, "Expiry disabled" badge `E/documentation/operations/tailscale.md:92`.
6. The box keeps running; only the operator's remote path is gone until someone re-authenticates on site or over the LAN.
7. NOT FOUND.
8. DOC `E/documentation/operations/tailscale.md:41`. Candidate: the section at `E/documentation/operations/tailscale.md:89-92` says "both" nodes while the table has three.
### 31. Gitea and registry credentials
1. NOT MEASURED. The login the cluster and the operator use to pull and push images and code. Operator. Row S15 `RB/total-loss-of-dooplex.md:48`; the pull secret is a scoped token `RB/secrets.md:231`.
2. NOT FOUND: no end date for the scoped token or for CC's own Gitea token. Searched grep `expir` in RB/secrets.md and cc-secrets ARCHITECTURE.md.
3. The operator, in Gitea.
4. If the pull token lapses: the hub pod cannot be re-pulled on a restart, and boxes pulling the controller image through the same registry would fail (box helper's item).
5. NOT FOUND.
6. NOT FOUND from these files.
7. NOT FOUND.
8. DOC `RB/secrets.md:231` and `RB/total-loss-of-dooplex.md:48`. WHERE THE OPERATOR READS THE DATE: NOT FOUND.
### 32. CC's secret store: its key and the tokens inside (GitHub advisory token, hub token)
1. NOT MEASURED. The store CC uses through `sec`, and the key that opens it. Operator owns; row S19 `RB/total-loss-of-dooplex.md:52`; design in /mnt/5_hdd/felhom.eu/git/cc-secrets/ARCHITECTURE.md (lines 47-48 give the key location).
2. NOT FOUND: the design document states no expiry for any stored secret (grep `expir|rotat|days|lifetime`). The hub token has one by code, block 22. A GitHub advisory token is named only in CC's memory index, in no document (grep `GH_ADVISORY_TOKEN` in AR, RB, operations, STATUS).
3. The operator stores a new value; only he may delete one.
4. An expired GitHub token: the nightly security watch `E/.gitea/workflows/security-watch.yml:22` loses its source. An expired hub token: block 22.
5. NOT FOUND.
6. Not a box matter.
7. NOT FOUND.
8. DOC `RB/total-loss-of-dooplex.md:52`. WHERE THE OPERATOR READS THE DATES: NOT FOUND. NOBODY KNOWS from files.
### 33. SSH keys between the hub, DooPlex and the endpoint
1. NOT MEASURED. Keys the hub uses to reach the endpoint (three key files named at `E/manifests/hub.yaml:191`, `E/manifests/hub.yaml:202`, `E/manifests/hub.yaml:209`) and the root key, row S9 `RB/total-loss-of-dooplex.md:42`. Operator.
2. NO END DATE in any document. The only rule: rotate on an endpoint rebuild, DOC `RB/offsite-endpoint.md:313`.
3. The operator, on rebuild.
4. A lost or changed key: the hub cannot add tunnel peers, mint off-site tokens or poke the endpoint; new households get no off-site tier.
5. Nobody; no date.
6. Existing boxes keep their own tunnel and token; only new provisioning fails.
7. NOT FOUND.
8. DOC `RB/offsite-endpoint.md:313`; row S9 says one authorized key's owner is "not recorded" `RB/total-loss-of-dooplex.md:42`. NOBODY KNOWS who holds it.
### 34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)
1. NOT MEASURED. The base systems under every box and the endpoint. Product. Host is Proxmox VE 9.2 on Debian 13, DOC `AR/11-os-updates.md:103`.
2. NOT FOUND: no document states a support end date. Searched grep `EOL|end of life|support ends|until 20` in AR, RB, operations. The major upgrade 9 to 10 is "later" `AR/11-os-updates.md:91` and `AR/11-os-updates.md:725`.
3. The operator, by a drilled major upgrade first on a test box `AR/11-os-updates.md:725`. No date.
4. After the upstream end, boxes stop getting security fixes; every household, silently.
5. NOT FOUND. The security watch flags a runtime line with no fixed version as end of life `E/scripts/security_watch.py:292`; whether Debian or Proxmox lines are in its list was not checked.
6. A box off across a major release: NOT FOUND what its updater does first.
7. NOT FOUND.
8. DOC `AR/11-os-updates.md:103`. The paid repository is a later money decision `AR/11-os-updates.md:662`. WHERE THE OPERATOR READS THE DATE: NOT FOUND.
## Tiny
- Diagnostic log bundles kept 72 hours: `logBundleTTL` `HUB/store/logbundle.go:29`.
- Hub access log kept 90 days: `DefaultKeep` `HUB/accesslog/accesslog.go:44`.
- Deleted-customer audit rows kept 365 days: `DeletedCustomerAuditKeep` `HUB/store/deleted_customer_audit.go:23`.
- App telemetry pruned after 90 days and stale issues after 30: `E/hub/cmd/hub/main.go:1134-1139`.
- Hub prune runs daily at 04:30: `prune_schedule` `E/manifests/hub.yaml:83`.
- Kernel-reboot notice valid 24 hours: `KernelNoticeValid` `HUB/osupdates/kernel.go:48`.
- Local backup counted stale after 26 hours: `backupStaleAfter` `HUB/monitor/deadline.go:29`.
- Off-site backup counted stale after 8 days: `offsiteBackupStaleAfter` `HUB/monitor/deadline_tiers.go:28`.
- Restore test counted proven for 7 to 12 days: `HUB/monitor/restoretest.go:68-74`.
- Household quiet-box mail after 7 days: `householdLivenessQuiet` `HUB/notify/dispatcher.go:1018`.
- Line-test result stale after 15 days: `lineStaleAfter` `HUB/web/system.go:174`.
- Hub database snapshot catch-up age 6 hours: `CatchUpAge` `HUB/dbsnap/dbsnap.go:36`.
- Escrow still pending after 7 days raises an operator alarm: `escrowPendingAfter` `HUB/monitor/offsite_escrow_pending.go:37`.
- Escrow ceremony: an unclaimed recovery code is void after 10 minutes: `RB/RUNBOOK-escrow-ceremony.md:46`.
- Appliance row badged stale after 7 days without a poll: `applianceStaleAfter` `HUB/web/appliances.go:21`.
- Waiver example in the runbook carries dates from September 2026: `RB/RUNBOOK-manual-build.md:278-279`.
- Hub image tag in the manifest: `E/manifests/hub.yaml:135`.
- Same issuer annotation on the website ingress: `E/manifests/webpage.yaml:141`.
## Not found
- The definition of the certificate issuer, certificate lifetime and renew-before: searched `ClusterIssuer|renewBefore|duration` in E/manifests; it lives in the DooPlex cluster repository, outside this task.
- Any warning before an operator access token ends: searched `token.*(expiring|expires soon|about to expire|days left)|token_expir` in the hub.
- The end date of CC's hub token in a document: searched `HUB_TOKEN` in AR, RB, documentation/operations and STATUS.md.
- An expiry on the claim code or on the pairing code: searched `expire|ttl|valid` in the claim engine and the appliance store.
- A hub-side expiry on queued signed jobs: searched `expire` around `EnqueueSignedJob`.
- A pinned Proxmox ISO version or checksum in the build script: they are arguments; searched `sha256|pve-iso|9\.[0-9]-[0-9]` in the ISO script and the release gate runbook.
- An "ISO too old" check at first boot: searched `too old|min.*version|ISO_VERSION` in the bootstrap script.
- The waiver file documentation/tests/golden-waiver.yml in this clone (it exists only in sibling worktrees, which are not sources).
- Test names in the golden gate's test file: `def test` returned nothing.
- Registrar names, domain renewal dates, the dynamic-DNS account's renewal: searched `expir|renew|valid until|until 20|No-IP|noip`.
- Billing or contract dates for Hetzner, Cloudflare, the mail service, Tailscale: searched `expir|renew|invoice|billing`.
- Expiry of Gitea tokens, registry logins, R2 keys, the GitHub advisory token, SSH keys: searched `expir|GH_ADVISORY_TOKEN|R2_ACCESS|R2 key` in AR, RB, operations, STATUS.md and the cc-secrets design document.
- Support end dates for Debian 13, Proxmox VE 9 or PBS: searched `EOL|end of life|support ends`.
- Anything dated in STATUS.md: searched `expir|renew|HUB_TOKEN|waiver|trixie|2027|2028|2029|2030`.
- WireGuard peer expiry or rotation in the hub: searched `expire|rotat|stale|handshake` in the hub's wg files.
- Merged: self-bind mint and resend into block 1; escrow retention and abandon into 6; sub-account password and seal key into 8; vouch and manifest into 16; golden age and waiver into 17; CC store, hub token date and GitHub token into 32.
@@ -0,0 +1,27 @@
searches run 2026-11-11 10:25:25 UTC
C+ | positive control (a row that exists) | /R-1301/ | ar files=0 | open: R-1301 | closed: | roadmap=0
C+2 | positive control (a word in all four) | /certificate/ | ar files=3 | open: R-1202 R-904 | closed: R-1090 R-1300 R-510 | roadmap=0
C- | negative control | /zzqxenddatezz/ | ar files=0 | open: | closed: | roadmap=0
S01 | web certificate: warning before it ends | /(certificate|cert).{0,80}(expir|renew|runs out|days left)|acme\.json|NotAfter/ | ar files=0 | open: | closed: | roadmap=0
S02 | Proxmox renews its certificate, the agent's pin does not follow | /pve-ssl|pveproxy.{0,60}cert|(tls|certificate|cert) pin.{0,120}(renew|expir|rotat|re-pin|repin)|fingerprint.{0,80}(renew|expir|stale)/ | ar files=0 | open: | closed: | roadmap=0
S03 | operator access token ends, nobody warned | /(access|api|hub|operator).token.{0,120}(expir|90 days|days left|renew)|HUB_TOKEN.{0,80}(expir|ends|renew)|token_expir/ | ar files=0 | open: | closed: | roadmap=0
S04 | Tailscale key expiry | /tailscale.{0,120}(expir|key)|key expiry/ | ar files=1 | open: | closed: | roadmap=0
S05 | restore copy swept after 30 days, no warning | /restore cop.{0,100}(30 days|sweep|unused)|restoreCopyUnusedAge/ | ar files=1 | open: | closed: | roadmap=0
S06 | domain renewal / registrar | /registrar|domain.{0,60}(renew|expir|lapse)|whois/ | ar files=3 | open: | closed: R-1158 R-820 R-826 | roadmap=0
S07 | paid accounts: billing, contract end | /(hetzner|cloudflare|resend|mail service).{0,100}(billing|invoice|unpaid|contract|account (is )?closed|payment)|credit card/ | ar files=1 | open: | closed: | roadmap=0
S08 | mail service key end / check that mail still goes | /(resend|mail).{0,40}(key|token).{0,80}(expir|rotat|ends)|canary mail|test mail.{0,40}(daily|weekly)/ | ar files=0 | open: | closed: | roadmap=0
S09 | OS support end (Debian 13, Proxmox 9) | /EOL|end of life|end-of-life|support (ends|end date)|trixie.{0,80}(until|ends|support)/ | ar files=5 | open: | closed: R-1203 R-921 | roadmap=0
S10 | apt signing keys end | /(signing|apt|archive|repo) key.{0,100}(expir|ends|rotat)|keyring.{0,80}(expir|old)|Valid-Until/ | ar files=1 | open: | closed: | roadmap=0
S11 | a box off for a long time comes back | /(off|unplugged|powered off|switched off|in a drawer).{0,60}(months|a year|years)|long.{0,10}(off|absence)|cold box/ | ar files=3 | open: | closed: | roadmap=0
S12 | old tunnel client / infra image stops being accepted | /cloudflared.{0,100}(old|version|support|deprecat)|tunnel client.{0,60}(old|support)/ | ar files=0 | open: | closed: | roadmap=0
S13 | Gitea / registry token end | /(gitea|registry).{0,60}(token|credential|password).{0,80}(expir|ends|rotat)/ | ar files=1 | open: R-925 | closed: R-616 | roadmap=0
S14 | GitHub advisory token / CC store token end | /GH_ADVISORY_TOKEN|advisory token|github.{0,40}token.{0,60}(expir|ends)/ | ar files=0 | open: | closed: | roadmap=0
S15 | household's own Cloudflare token end | /(cloudflare|cf).{0,20}token.{0,100}(expir|ends|ttl|revok)/ | ar files=0 | open: | closed: | roadmap=0
S16 | operator signing key rotation | /key.rotation|re-pin|rotat.{0,40}(operational|signing|cold) key/ | ar files=5 | open: R-924 | closed: | roadmap=0
S17 | seal key rotation | /seal.{0,10}key.{0,100}(rotat|lost|change)/ | ar files=0 | open: | closed: | roadmap=0
S18 | a list of what ends when | /end dates|expiry (list|calendar|table)|renewal calendar|what ends/ | ar files=0 | open: | closed: | roadmap=0
S19 | golden age: warning before day 7 | /golden.{0,80}(7 days|too old|age).{0,80}(warn|remind)|waiver.{0,60}(expir|ends)/ | ar files=0 | open: R-468 | closed: | roadmap=0
S20 | served certificate of the hub address (cert-manager) | /cert-manager|letsencrypt-prod|ClusterIssuer/ | ar files=1 | open: | closed: | roadmap=0
S21 | a recorded image digest is no longer served | /digest.{0,100}(still served|no longer|gone|deleted|removed from)|registry.{0,60}(drops|deletes|garbage)/ | ar files=1 | open: | closed: | roadmap=0
S22 | the installer ISO grows old | /ISO.{0,80}(too old|stale|age|rebuild every|rebuilt every)/ | ar files=5 | open: R-1214 R-504 R-548 | closed: R-1176 R-472 R-531 R-556 R-588 R-725 | roadmap=0
S23 | a box off during the 14-day off-site countdown | /(abandon|countdown).{0,120}(box is off|switched off|offline|unreachable)/ | ar files=0 | open: | closed: | roadmap=0
@@ -0,0 +1,78 @@
#!/usr/bin/env python3
"""Print the text behind every pointer in a table file, so a person can read it.
Read-only by construction: it opens files for reading and prints. It runs on DooPlex against the
git clones only; it never touches a box.
Usage: check-pointers.py [FILE ...] (default: ../TABLE.md)
check-pointers.py --quiet [FILE] (only the verdict lines; rc 1 when a pointer is dead)
A pointer is a backticked token `PREFIX/path[:line[-line]]` with one of the prefixes below.
A pointer is DEAD when the file is missing or the line range lies beyond the end of the file.
The script cannot judge whether the line says what the table claims — that is the reader's job.
"""
import os
import re
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
AUDIT = os.path.dirname(HERE)
REPO = os.path.abspath(os.path.join(AUDIT, "..", "..", "..")) # felhom.eu
WORKSPACE = os.path.dirname(REPO)
PREFIX = {
"AG": os.path.join(WORKSPACE, "felhom-agent"),
"CT": os.path.join(WORKSPACE, "felhom-controller", "controller"),
"CAT": os.path.join(WORKSPACE, "app-catalog-felhom.eu"),
"HUB": os.path.join(REPO, "hub", "internal"),
"E": REPO,
"AR": os.path.join(REPO, "documentation", "architecture"),
"RB": os.path.join(REPO, "documentation", "runbooks"),
"EV": os.path.join(AUDIT, "evidence"),
}
TOKEN = re.compile(r"`((?:%s)/[^`:\s]+)(?::(\d+)(?:-(\d+))?)?`" % "|".join(PREFIX))
def main() -> int:
args = [a for a in sys.argv[1:] if a != "--quiet"]
quiet = "--quiet" in sys.argv[1:]
files = args or [os.path.join(AUDIT, "TABLE.md")]
seen, dead, total = set(), 0, 0
for table in files:
with open(table, encoding="utf-8") as fh:
text = fh.read()
for m in TOKEN.finditer(text):
key = m.group(0)
if key in seen:
continue
seen.add(key)
total += 1
pre, rest = m.group(1).split("/", 1)
path = os.path.join(PREFIX[pre], rest)
first = int(m.group(2)) if m.group(2) else None
last = int(m.group(3)) if m.group(3) else first
if not os.path.exists(path):
dead += 1
print(f"DEAD {key} (no such file: {path})")
continue
if first is None:
if not quiet:
print(f"OK {key} (file exists)")
continue
with open(path, encoding="utf-8", errors="replace") as fh:
lines = fh.readlines()
if last > len(lines):
dead += 1
print(f"DEAD {key} (file has {len(lines)} lines)")
continue
if quiet:
continue
print(f"OK {key}")
for n in range(first, min(last, first + 11) + 1):
print(f" {n:>5} {lines[n - 1].rstrip()[:200]}")
print(f"pointers: {total} dead: {dead}")
return 1 if dead else 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,30 @@
#!/bin/bash
# End-dates list 2026-11-11 — guest 9201. READ-ONLY. Prints for a certificate only subject, issuer, dates, fingerprint.
# It never prints a key or a token value. acme.json holds private keys: only its certificate fields are decoded.
set -u
h() { echo; echo "=== $* ==="; }
U() { sed -E 's#(https?://)[^/@ ]+@#\1***@#g'; }
cert() { openssl x509 -noout -subject -issuer -startdate -enddate -fingerprint -sha256 2>/dev/null | tr '\n' ' '; echo; }
h "now"; date -u +%FT%TZ
h "layout"; ls /opt/docker 2>/dev/null | tr '\n' ' '; echo; ls /opt/docker/stacks 2>/dev/null | tr '\n' ' '; echo; docker volume ls --format '{{.Name}}' | /usr/bin/grep -i -E 'felhom|traefik|cloudflare|controller' | tr '\n' ' '; echo
h "certificate files (system paths and infra stacks only)"; for f in $( { /usr/bin/grep -rlIs 'BEGIN CERTIFICATE' /etc/felhom* /etc/docker /usr/local/share/ca-certificates /etc/ssl/private 2>/dev/null; find /opt/docker -maxdepth 4 \( -name '*.pem' -o -name '*.crt' -o -name '*.cert' \) -size -200k 2>/dev/null; for v in $(docker volume ls --format '{{.Name}}' | /usr/bin/grep -i -E 'felhom|traefik|controller'); do find /var/lib/docker/volumes/$v/_data -maxdepth 3 \( -name '*.pem' -o -name '*.crt' -o -name '*.cert' \) -size -200k 2>/dev/null; done; } | sort -u); do /usr/bin/grep -qs 'BEGIN CERTIFICATE' "$f" || continue; printf '%s [mtime=%s] ' "$f" "$(date -u -r "$f" +%F)"; cert < "$f"; done
h "acme stores (certificate fields only)"; for f in $(find /opt/docker /var/lib/docker/volumes -maxdepth 5 -name 'acme*.json' 2>/dev/null); do echo "--- $f mtime=$(date -u -r "$f" +%FT%TZ)"; python3 - "$f" <<'P'
import json,sys,base64,subprocess
try: d=json.load(open(sys.argv[1]))
except Exception as e: print("unreadable:", type(e).__name__); sys.exit(0)
for res,v in (d.items() if isinstance(d,dict) else []):
for c in (v or {}).get("Certificates") or []:
pem=base64.b64decode(c.get("certificate",""))
o=subprocess.run(["openssl","x509","-noout","-subject","-issuer","-startdate","-enddate","-fingerprint","-sha256"],input=pem,capture_output=True)
print(res, (c.get("domain") or {}).get("main"), "|", o.stdout.decode().replace("\n"," "))
P
done
h "served certificates (TLS handshake to own listeners)"; ss -H -tln | awk '{print $4}' | sort -u | while read a; do p=${a##*:}; ip=${a%:*}; ip=${ip#[}; ip=${ip%]}; case "$ip" in '*'|'0.0.0.0'|'::') ip=127.0.0.1;; esac; out=$(echo | timeout 4 openssl s_client -connect "$ip:$p" 2>/dev/null | cert); [ -n "${out// /}" ] && echo "$a -> $out" || echo "$a -> no TLS"; done
h "os"; . /etc/os-release; echo "$PRETTY_NAME / $VERSION_CODENAME"; cat /etc/debian_version
h "docker"; docker version --format 'client={{.Client.Version}} server={{.Server.Version}}' 2>&1; dpkg -l docker-ce containerd.io docker-compose-plugin 2>/dev/null | awk '/^ii/{print $2,$3}'; cat /etc/felhom-controller-image 2>/dev/null
h "restic and tools inside the controller"; docker exec felhom-controller restic version 2>&1 | head -1; docker exec felhom-controller sh -c 'cat /etc/os-release | head -3; command -v rclone age docker 2>/dev/null' 2>&1 | head -8
h "images: repo tag digest created"; docker images --digests --format '{{.Repository}}:{{.Tag}} {{.Digest}} {{.CreatedAt}}' | cut -c1-200 | sort
h "apt sources"; for f in /etc/apt/sources.list /etc/apt/sources.list.d/*; do [ -f "$f" ] && { echo "--- $f"; /usr/bin/grep -v -E '^\s*(#|$)' "$f" | U; }; done
h "apt keyring files"; ls -l --time-style=long-iso /etc/apt/trusted.gpg /etc/apt/trusted.gpg.d/ /etc/apt/keyrings/ /usr/share/keyrings/ 2>/dev/null | awk '{print $5,$6,$8}'
h "apt release files: Valid-Until"; for f in /var/lib/apt/lists/*Release; do [ -f "$f" ] && printf '%s | Date: %s | Valid-Until: %s\n' "$(basename "$f")" "$(/usr/bin/grep -m1 '^Date:' "$f" | cut -c7-)" "$(/usr/bin/grep -m1 '^Valid-Until:' "$f" | cut -c14-)"; done
h "controller data folder (top level: names and dates only)"; for v in $(docker volume ls --format '{{.Name}}' | /usr/bin/grep -i -E 'controller'); do echo "--- $v"; ls -l --time-style=long-iso /var/lib/docker/volumes/$v/_data 2>/dev/null | awk '{print $1,$5,$6,$7,$8}' | head -60; done
@@ -0,0 +1,31 @@
#!/bin/bash
# End-dates list 2026-11-11 — host. READ-ONLY: openssl x509 -noout (public fields only), a TLS handshake to the box's
# own listening ports, pveum/pvenode/pvesubscription reads, ls, cat of apt source lists, dpkg -l. It prints NO key,
# NO token value and NO certificate body: for a certificate only subject, issuer, the two dates and the fingerprint.
set -u
h() { echo; echo "=== $* ==="; }
U() { sed -E 's#(https?://)[^/@ ]+@#\1***@#g'; }
cert() { openssl x509 -noout -subject -issuer -startdate -enddate -fingerprint -sha256 2>/dev/null | tr '\n' ' '; echo; }
h "now"; date -u +%FT%TZ
h "certificate files"; for f in $(/usr/bin/grep -rlIs 'BEGIN CERTIFICATE' /etc/pve/local /etc/pve/pve-root-ca.pem /etc/pve/nodes /etc/felhom /etc/felhom-agent /var/lib/felhom-agent /var/lib/felhom-install /etc/ssl/private /usr/local/share/ca-certificates /etc/proxmox-backup 2>/dev/null | sort -u); do n=$(/usr/bin/grep -c 'BEGIN CERTIFICATE' "$f"); printf '%s [certs=%s mtime=%s] ' "$f" "$n" "$(date -u -r "$f" +%F)"; cert < "$f"; done
h "system CA store"; ls /etc/ssl/certs/*.pem 2>/dev/null | wc -l; dpkg -l ca-certificates 2>/dev/null | tail -1 | awk '{print $2,$3}'
h "served certificates (TLS handshake to own listeners)"; ss -H -tln | awk '{print $4}' | sort -u | while read a; do p=${a##*:}; ip=${a%:*}; ip=${ip#[}; ip=${ip%]}; case "$ip" in '*'|'0.0.0.0'|'::') ip=127.0.0.1;; esac; out=$(echo | timeout 4 openssl s_client -connect "$ip:$p" 2>/dev/null | cert); [ -n "${out// /}" ] && echo "$a -> $out" || echo "$a -> no TLS"; done
h "listeners (port, process)"; ss -H -tlnp | awk '{print $4, $6}' | sed -E 's/pid=[0-9]+,fd=[0-9]+//g' | sort -u
h "pve users and tokens (expire: 0 = never)"; pveum user list --output-format json 2>/dev/null | python3 -c '
import json,sys,time
for u in json.load(sys.stdin):
print("user", u.get("userid"), "enable", u.get("enable"), "expire", u.get("expire"), "tokens", [(t.get("tokenid"), "expire", t.get("expire")) for t in (u.get("tokens") or [])])'
h "acme / subscription"; pvenode acme account list 2>&1 | head -5; pvenode config get 2>&1 | head -10; pvesubscription get 2>&1 | /usr/bin/grep -E 'status|message|nextduedate' | head
h "os"; . /etc/os-release; echo "$PRETTY_NAME / $VERSION_CODENAME"; cat /etc/debian_version; pveversion 2>&1; uname -r; proxmox-boot-tool kernel list 2>&1 | head -12; dpkg -l 'proxmox-kernel-*-signed' 'pve-kernel-*' 2>/dev/null | awk '/^ii/{print $2,$3}' | head -12
h "apt sources"; for f in /etc/apt/sources.list /etc/apt/sources.list.d/*; do [ -f "$f" ] && { echo "--- $f"; /usr/bin/grep -v -E '^\s*(#|$)' "$f" | U; }; done; echo "--- auth files (names only): $(ls /etc/apt/auth.conf /etc/apt/auth.conf.d 2>/dev/null | tr '\n' ' ')"
h "apt keyring files"; ls -l --time-style=long-iso /etc/apt/trusted.gpg /etc/apt/trusted.gpg.d/ /etc/apt/keyrings/ /usr/share/keyrings/ 2>/dev/null | awk '{print $5,$6,$8}'
h "apt release files: Valid-Until"; for f in /var/lib/apt/lists/*Release; do [ -f "$f" ] && printf '%s | Date: %s | Valid-Until: %s\n' "$(basename "$f")" "$(/usr/bin/grep -m1 '^Date:' "$f" | cut -c7-)" "$(/usr/bin/grep -m1 '^Valid-Until:' "$f" | cut -c14-)"; done
h "templates and golden"; for d in $(awk '/^dir:/{f=1;next} /^[a-z]+:/{f=0} f&&$1=="path"{print $2}' /etc/pve/storage.cfg 2>/dev/null) /var/lib/vz; do ls -l --time-style=long-iso "$d/template/cache" 2>/dev/null | awk -v d="$d" 'NR>1{print d, $5,$6,$8}'; done | sort -u
h "agent files (names and dates only)"; ls -l --time-style=long-iso /etc/felhom-agent /etc/felhom /var/lib/felhom-agent 2>/dev/null | awk '{print $1,$3,$5,$6,$7,$8}'
h "agent"; /usr/local/bin/felhom-agent --version 2>&1 | head -1
h "wireguard (interface and latest handshake time only)"; wg show all latest-handshakes 2>/dev/null | awk '{print $1, "peer", substr($2,1,6) "…", strftime("%FT%TZ",$3,1)}'
h "tailscale"; command -v tailscale >/dev/null && tailscale status --json 2>/dev/null | python3 -c '
import json,sys
d=json.load(sys.stdin); s=d.get("Self",{}); print("self key expiry:", s.get("KeyExpiry"), "online:", s.get("Online"))' || echo "no tailscale"
h "ssh host keys / authorized keys (counts only)"; ls /etc/ssh/ssh_host_*_key.pub 2>/dev/null | wc -l; for f in /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys; do [ -f "$f" ] && echo "$f lines=$(/usr/bin/grep -c -v -E '^\s*(#|$)' "$f")"; done
h "smart: power-on hours and wear (no serials)"; for d in $(lsblk -dno NAME,TYPE | awk '$2=="disk"{print $1}'); do printf '%s: ' $d; smartctl -A /dev/$d 2>/dev/null | /usr/bin/grep -i -E 'Power_On_Hours|Percentage Used|Power On Hours|Wear_Leveling|Media_Wearout|Available Spare:' | awk '{$1=$1;print}' | tr '\n' ';'; echo; done
@@ -0,0 +1,17 @@
# host-pin.py — read-only: does the agent's pinned Proxmox fingerprint equal the certificate Proxmox serves?
# Prints the certificate's end date and yes/no. It prints no value from agent.json.
import json, subprocess, re
c = json.load(open("/etc/felhom-agent/agent.json"))
def find(o, p=""):
if isinstance(o, dict):
for k, v in o.items():
if "ingerprint" in k and isinstance(v, str):
yield p + k, v
else:
yield from find(v, p + k + ".")
fp = subprocess.run(["openssl", "x509", "-in", "/etc/pve/local/pve-ssl.pem", "-noout", "-fingerprint", "-sha256", "-enddate"], capture_output=True, text=True).stdout
leaf = re.sub(r"[^0-9a-f]", "", fp.split("\n")[0].split("=")[1].lower())
print("pve-ssl", fp.strip().split("\n")[1])
for k, v in find(c):
n = re.sub(r"[^0-9a-f]", "", v.lower())
print(k, "set" if n else "EMPTY", "equals pve-ssl leaf:", n == leaf)
@@ -0,0 +1,17 @@
#!/bin/bash
# keyrings.sh NAME SSHTARGET — copy the PUBLIC apt keyring files of host and guest to DooPlex and print each key's
# id, creation date and end date. Public keys only. Read-only on the box (tar to stdout).
set -u
D=$(cd "$(dirname "$0")/.." && pwd); n=$1; t=$2; SSH="ssh -o BatchMode=yes -o ConnectTimeout=10"
export GNUPGHOME="$D/gnupg"; mkdir -p "$GNUPGHOME"; chmod 700 "$GNUPGHOME"
for where in host guest; do
w="$D/keyrings/$n/$where"; mkdir -p "$w"
cmd='tar -C / -cf - --ignore-failed-read etc/apt/trusted.gpg etc/apt/trusted.gpg.d etc/apt/keyrings usr/share/keyrings 2>/dev/null'
if [ $where = host ]; then $SSH "$t" "$cmd" | tar -C "$w" -xf - 2>/dev/null; else $SSH "$t" "pct exec 9201 -- sh -c '$cmd'" | tar -C "$w" -xf - 2>/dev/null; fi
printf '=== %s %s: keyring files %s\n' "$n" "$where" "$(find "$w" -type f | wc -l)"
find "$w" -type f | sort | while read f; do
gpg --batch --no-options --show-keys --with-colons "$f" 2>/dev/null | awk -F: -v f="${f#$w/}" '
$1=="pub"||$1=="sub"{ printf "%s | %s %s algo=%s bits=%s | created %s | expires %s\n", f, $1, $5, $4, $3, strftime("%F",$6,1), ($7==""?"never":strftime("%F",$7,1)) }
$1=="uid"{ printf "%s | uid %s\n", f, $10 }'
done
done
@@ -0,0 +1,9 @@
"""listconf.py — the End-dates list's own words for make-table.py and make-matrix.py. Data only."""
HELPERS = ("dates-box", "dates-hub")
KINDS = ["Certificate", "Key", "Window", "Version", "Account", "OS"]
WORD, WORDS = "thing", "Things"
TITLE = "The End-dates list"
ID_RE = r"D[BH]\d\d"
QUESTIONS = ["what, owner, where", "when it ends", "who renews, how early", "what breaks", "who is warned before",
"a box off past the date", "if renewal fails", "how do we know"]
CLASS_ROW = {"T1": "R-1379", "T2": "R-1380", "T3": "R-1381", "T4": "R-1382"}
@@ -0,0 +1,131 @@
#!/usr/bin/env python3
"""Build MATRIX.md from TABLE.md, and check the table's shape.
The same tool as `audits/endings-list-2026-11-10/tools/make-matrix.py`, made general: the list's own words are in `listconf.py`.
Read-only on everything but MATRIX.md in this folder. It runs on DooPlex against the git clone only.
Usage: make-matrix.py write ../MATRIX.md, print the totals
make-matrix.py --check write nothing; rc 1 when the table breaks a rule
Rules checked (a broken rule is printed and makes rc 1):
- a heading is `### <ID> — <name> · <seed kind> · NAMED|MENTIONED|UNNAMED`
- each block has exactly eight answer rows, numbered 1..8, each with one mark A, D, K or N
- an A row carries a backticked pointer; a K row an `R-<n>`; an N row one class from listconf.py and a search S<n>
"""
import os
import re
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import listconf as C
HERE = os.path.dirname(os.path.abspath(__file__))
AUDIT = os.path.dirname(HERE)
HEAD = re.compile(r"^### (" + C.ID_RE + r") — (.+?) · (?:" + "|".join(C.KINDS) + r") · (NAMED|MENTIONED|UNNAMED)\s*$")
ROW = re.compile(r"^\| ([1-8]) \| (.*) \| ([ADKN]) \| (.*) \|\s*$")
QUESTIONS = C.QUESTIONS
CLASS_ROW = C.CLASS_ROW
CLS = re.compile(r"\b(?:" + "|".join(C.CLASS_ROW) + r")\b")
def parse(path):
places, errors, cur = [], [], None
with open(path, encoding="utf-8") as fh:
for ln, line in enumerate(fh, 1):
if line.startswith("## TEST-ONLY") or line.startswith("## DOC-ONLY"):
cur = None
continue
if line.startswith("### "):
m = HEAD.match(line)
if not m:
errors.append(f"line {ln}: heading has the wrong shape: {line.strip()}")
cur = None
continue
cur = {"id": m.group(1), "name": m.group(2), "named": m.group(3), "marks": {}, "n": {}, "k": {}}
places.append(cur)
continue
m = ROW.match(line)
if not m or cur is None:
continue
q, mark, ptr = int(m.group(1)), m.group(3), m.group(4)
if q in cur["marks"]:
errors.append(f"{cur['id']}: question {q} is answered twice")
cur["marks"][q] = mark
if mark == "A" and "`" not in ptr:
errors.append(f"{cur['id']}-{q}: A with no pointer")
if mark == "K":
rows = re.findall(r"R-\d+", ptr)
if not rows:
errors.append(f"{cur['id']}-{q}: K with no row")
cur["k"][q] = rows
if mark == "N":
cls = CLS.findall(ptr)
if len(cls) != 1:
errors.append(f"{cur['id']}-{q}: N needs exactly one class, found {cls}")
elif not re.search(r"\bS\d+\b", ptr) and "NOBODY KNOWS" not in m.group(2):
errors.append(f"{cur['id']}-{q}: N with no search id")
cur["n"][q] = cls[0] if cls else "?"
for p in places:
if sorted(p["marks"]) != list(range(1, 9)):
errors.append(f"{p['id']}: has answers {sorted(p['marks'])}, wants 1..8")
return places, errors
def build(places):
out = ["# " + C.TITLE + " — the matrix", "",
"Made by `tools/make-matrix.py` from `TABLE.md`. Do not edit it by hand.", "",
"`A` answered · `D` decided, not built · `K` known gap (has a row) · `N` nobody asked. Columns are the eight",
"questions: " + " · ".join(f"{i} {q}" for i, q in enumerate(QUESTIONS, 1)) + ".",
"", "| " + C.WORD.capitalize() + " | Named | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | N |", "|---|---|---|---|---|---|---|---|---|---|---|"]
tot = {m: 0 for m in "ADKN"}
perq = {q: {m: 0 for m in "ADKN"} for q in range(1, 9)}
named = {"NAMED": 0, "MENTIONED": 0, "UNNAMED": 0}
members, krows = {c: [] for c in CLASS_ROW}, {}
for p in places:
named[p["named"]] += 1
cells = []
for q in range(1, 9):
mk = p["marks"].get(q, "?")
cells.append(f"**{mk}**" if mk == "N" else mk)
if mk in tot:
tot[mk] += 1
perq[q][mk] += 1
for q, c in p["n"].items():
members.setdefault(c, []).append(f"{p['id']}-{q}")
for q, rows in p["k"].items():
for r in rows:
krows.setdefault(r, []).append(f"{p['id']}-{q}")
out.append(f"| {p['id']} {p['name']} | {p['named'].lower()} | " + " | ".join(cells) + f" | {len(p['n'])} |")
n = len(places)
out += ["", "## Totals", "",
f"- {C.WORDS}: **{n}** in the table · named **{named['NAMED']}** · mentioned only **{named['MENTIONED']}** · "
f"named by no architecture document **{named['UNNAMED']}**.",
f"- Answers: **{n * 8}** · A **{tot['A']}** · D **{tot['D']}** · K **{tot['K']}** · N **{tot['N']}**.",
f"- {C.WORDS} with no N: **{sum(1 for p in places if not p['n'])}** of {n}.",
"", "| Question | A | D | K | N |", "|---|---|---|---|---|"]
for q in range(1, 9):
out.append(f"| {q} {QUESTIONS[q - 1]} | {perq[q]['A']} | {perq[q]['D']} | {perq[q]['K']} | {perq[q]['N']} |")
out += ["", "## The N answers, by class row", "", "| Class | Row | Count | Members (id-question) |", "|---|---|---|---|"]
for c in sorted(members):
out.append(f"| {c} | {CLASS_ROW.get(c, '?')} | {len(members[c])} | {', '.join(members[c])} |")
out += ["", "## The K answers, by existing row", "", "| Row | Count | Members |", "|---|---|---|"]
for r in sorted(krows, key=lambda x: int(x[2:])):
out.append(f"| {r} | {len(krows[r])} | {', '.join(krows[r])} |")
return "\n".join(out) + "\n", tot, named, members
def main() -> int:
places, errors = parse(os.path.join(AUDIT, "TABLE.md"))
for e in errors:
print("BROKEN " + e)
text, tot, named, members = build(places)
if "--check" not in sys.argv[1:]:
with open(os.path.join(AUDIT, "MATRIX.md"), "w", encoding="utf-8") as fh:
fh.write(text)
print(f"places {len(places)} named {named} marks {tot} "
+ " ".join(f"{c}={len(m)}" for c, m in sorted(members.items())) + f" broken {len(errors)}")
return 1 if errors else 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,126 @@
#!/usr/bin/env python3
"""make-table.py — build TABLE.md from the helpers' answers (evidence/helpers/resolved-*.json), marks.py and listconf.py.
The same tool as `audits/endings-list-2026-11-10/tools/make-table.py`, made general: what differs between two lists
(the heading words, the helper names, the kinds, the questions) is in `listconf.py` beside it.
A cell with an entry in marks.py carries the MAIN SESSION's words. Every other cell carries the helper's words and the
mark A; its pointers were opened by resolve-helper.py (the file and the line exist; the name the helper gave stands at
the line). A dead pointer is dropped here. Read-only but for TABLE.md.
"""
import json, os, re, subprocess, sys
import marks, listconf as C
HERE = os.path.dirname(os.path.abspath(__file__)); AUDIT = os.path.dirname(HERE)
REPO = os.path.abspath(os.path.join(HERE, "..", "..", "..", "..")); WS = os.path.dirname(REPO)
PREFIX = {"AG": os.path.join(WS, "felhom-agent"), "CT": os.path.join(WS, "felhom-controller", "controller"),
"CAT": os.path.join(WS, "app-catalog-felhom.eu"), "HUB": os.path.join(REPO, "hub", "internal"), "E": REPO,
"AR": os.path.join(REPO, "documentation", "architecture"), "RB": os.path.join(REPO, "documentation", "runbooks"),
"EV": os.path.join(AUDIT, "evidence")}
TOK = re.compile(r"`((?:AG|CT|CAT|HUB|E|AR|RB|EV)/[^`:\s]+)(?::(\d+)(?:-(\d+))?)?`")
R = {h: json.load(open(os.path.join(AUDIT, "evidence", "helpers", f"resolved-{h}.json")))["endings"] for h in C.HELPERS}
dropped = []
def live(tok):
m = TOK.fullmatch(tok)
pre, rest = m.group(1).split("/", 1)
p = os.path.join(PREFIX[pre], rest)
if not os.path.exists(p):
return False
if m.group(2):
n = sum(1 for _ in open(p, encoding="utf-8", errors="replace"))
return int(m.group(3) or m.group(2)) <= n
return True
def split(ans, eid, q):
"""helper answer -> (plain text, [live pointers])"""
ptrs = []
for m in TOK.finditer(ans):
t = m.group(0)
if live(t):
if t not in ptrs:
ptrs.append(t)
else:
dropped.append(f"{eid}-{q} {t}")
t = TOK.sub("", ans)
t = re.sub(r"`~?:\d+(?:-\d+)?`", "", t)
t = re.sub(r"\(\s*(?:called|creator|see)?\s*[,;]?\s*\)", "", t)
t = re.sub(r"\s+([,.;:)])", r"\1", t)
t = re.sub(r"\(\s+", "(", t)
t = re.sub(r"([,;:])(?:\s*[,;:])+", r"\1", t)
t = re.sub(r"\s{2,}", " ", t).replace("|", "/").strip(" ;,·")
t = re.sub(r"\(\)", "", t)
if len(t) > 380:
cut = max(t.rfind(". ", 0, 380), t.rfind("; ", 0, 380))
t = t[:cut + 1 if cut > 150 else 377].rstrip(" ;") + " …"
return t, ptrs
def named(m):
kind, why = m["named"]
if kind != "grep":
return kind, f"architecture: {why}"
r = subprocess.run(["/usr/bin/grep", "-rEic", "--include=*.md", why, PREFIX["AR"]], capture_output=True, text=True)
files = [l for l in r.stdout.split("\n") if l and not l.endswith(":0")]
lines = sum(int(l.rsplit(":", 1)[1]) for l in files)
return ("MENTIONED" if lines else "UNNAMED"), f"architecture folder: {lines} lines in {len(files)} files match `/{why}/` (a machine count, not a reading)"
def main():
out = [open(os.path.join(HERE, "table-head.md"), encoding="utf-8").read().rstrip(), ""]
kindmap, bad, body = {}, [], []
for eid, m in sorted(marks.M.items()):
e = R[m["src"][0]][m["src"][1]]
st, why = named(m)
worst, rows = "A", []
for q in range(1, 9):
if q in m["o"]:
mark, text, ptr = m["o"][q]
if ptr is None:
_, ps = split(e["answers"].get(str(q), ""), eid, q)
ptr = " ".join(ps[:4])
else:
mark = "A"
text, ps = split(e["answers"].get(str(q), ""), eid, q)
ptr = " ".join(ps[:4])
if q == 8 and eid in marks.MEASURED:
mt, mp = marks.MEASURED[eid]
text = mt + " · " + text
ptr = (mp + " " + ptr).strip()
if mark == "A" and "`" not in ptr: # borrow the code path from question 1, then 2
for alt in (1, 2, 3):
_, ps = split(e["answers"].get(str(alt), ""), eid, alt)
if ps:
ptr = " ".join(ps[:2]) + f" (the code path of question {alt})"
break
if mark == "A" and "`" not in ptr:
bad.append(f"{eid}-{q}: A with no pointer")
if not text:
bad.append(f"{eid}-{q}: empty answer")
rows.append(f"| {q} | {text} | {mark} | {ptr} |")
worst = max(worst, mark, key="ADKN".index)
kindmap.setdefault(m["seed"], []).append((eid, st, worst))
body += [f"### {eid} — {m['name']} · {m['seed']} · {st}", "",
f"{why} · helper section: {m['src'][0]} „{e['title'].strip()}”", "",
"| # | Answer | Mark | Pointer |", "|---|---|---|---|"] + rows + ["", f"**Hand step in a runbook?** {m['rb']}", ""]
out += [f"## The kinds, and where each {C.WORD} is", "", f"| Kind | {C.WORDS} (worst mark) | Named · mentioned · named nowhere |", "|---|---|---|"]
for kind in C.KINDS:
v = kindmap.get(kind, [])
c = {s: sum(1 for x in v if x[1] == s) for s in ("NAMED", "MENTIONED", "UNNAMED")}
out.append(f"| {kind} | " + ", ".join(f"{x[0]} ({x[2]})" for x in v) + f" | {c['NAMED']} · {c['MENTIONED']} · {c['UNNAMED']} |")
for k in kindmap:
if k not in C.KINDS:
bad.append(f"kind {k} is not in listconf.KINDS")
out += ["", "## Not in the table", "",
"**Tiny (named once, not tabled):** " + " · ".join(marks.TINY) + ".", "",
"**Left out for another reason:**", ""] + [f"- {a} — {b}." for a, b in marks.LEFT_OUT]
out += ["", "---", ""] + body
open(os.path.join(AUDIT, "TABLE.md"), "w", encoding="utf-8").write("\n".join(out).rstrip() + "\n")
print(f"{C.WORDS} {len(marks.M)} · dead pointers dropped {len(dropped)} · broken {len(bad)}")
for b in bad + dropped:
print(" " + b)
return 1 if bad else 0
sys.exit(main())
@@ -0,0 +1,266 @@
"""marks.py — the main session's rulings for the End-dates list (2026-11-11). Data only; make-table.py reads it.
One entry for each thing that is in the table:
src (helper, index) the helper section the default answers come from (evidence/helpers/resolved-<helper>.json)
name the heading
seed the kind (Certificate, Key, Window, Version, Account, OS)
named a fixed status with its reason, or ("grep", REGEX): 0 lines in the architecture folder = UNNAMED, else MENTIONED
o {question: (mark, answer, pointers)} — an answer the MAIN SESSION wrote after it read the pointer by eye.
pointers None = keep the helper's pointers for that question.
A question with no entry here takes the helper's answer and the mark A.
rb the ninth line: is the hand step written in a runbook?
"""
NM = "NOT MEASURED (DooPlex, ep0 and the accounts are outside this run). "
M = {
# ------------------------------------------------------------------ on a box
"DB01": dict(src=("dates-box", 0), name="Web certificate of the household's address", seed="Certificate",
named=("grep", r"acme\.json|Let.s Encrypt|letsencrypt"),
o={2: ("A", "MEASURED 2026-11-11: the N100's certificate ends 2026-12-31, Tester 1's 2027-01-02, demo-hp's 2027-01-19. The life (90 days) is the issuer's rule, not ours.",
"`EV/box/demo-felhom-guest-dates.txt` `EV/box/tester-1-guest-dates.txt` `EV/box/demo-hp-guest-dates.txt`"),
5: ("N", "Nobody. No part of the product reads the certificate's end date. If the renewal fails (the household's Cloudflare token ends, or the name check fails), the first sign is a browser warning on the end date.",
"`CT/internal/infra/infra.go:157-159` · class T1 · S01"),
6: ("N", "NOBODY KNOWS. A box that was off past the end date starts with a dead certificate. Traefik must get a new one at the start. Nobody has tested this.",
"`CT/internal/infra/infra.go:157-159` · class T4 · S11")},
rb="NO — no runbook says what to do when a household's certificate does not renew"),
"DB02": dict(src=("dates-box", 1), name="Proxmox's own certificate, pinned by the agent", seed="Certificate",
named=("grep", r"pve-ssl|tls\.fingerprint|TLS pin"),
o={2: ("A", "MEASURED 2026-11-11: the certificate ends 2028-07-16 on the N100, 2028-08-19 on demo-hp, 2028-10-02 on Tester 1. On all three the agent's pin equals this certificate. The pin itself has no date.",
"`EV/box/pin.txt` `AG/internal/proxmox/tls.go:40-58`"),
3: ("N", "Proxmox renews it by itself, about 14 days before the end (its daily update job). No agent code reads the new fingerprint. The installer writes the pin once.",
"`EV/box/pveupdate-excerpt.txt` `AG/internal/proxmox/tls.go:40-58` `E/scripts/felhom-host-install.sh:3182-3185` · class T2 · S02"),
4: ("N", "After Proxmox makes the new certificate, every call from the agent to Proxmox fails with 'TLS pin mismatch'. That is: guest start and stop, whole-guest backups, restore tests, disk work. First box: the N100, about 2028-07-02. The other two follow within three months.",
"`AG/internal/proxmox/tls.go:55` `EV/box/pin.txt` · class T2 · S02"),
5: ("N", "Nobody before the day. No code compares the pin with the certificate's end date.", "`AG/internal/proxmox/tls.go:40-58` · class T2 · S02"),
6: ("N", "No. Proxmox also renews a certificate that has ended, at its next daily job. The box comes back with a new certificate and the old pin.",
"`EV/box/pveupdate-excerpt.txt` · class T2 · S02"),
7: ("N", "Here the renewal that works is the failure. The repair is by hand: write the new fingerprint into the agent's config and restart the agent. No code does it.",
"`AG/internal/proxmox/tls.go:40-58` · class T2 · S02")},
rb="NO — a search of the runbooks for 'tls.fingerprint', 're-pin' and 'pin mismatch' finds nothing"),
"DB03": dict(src=("dates-box", 2), name="The agent's local-API certificate, pinned in the guest", seed="Certificate",
named=("NAMED", "`AR/03-host-agent.md:420-423`"),
o={5: ("N", "Nobody. The certificate lives 10 years and nothing reads its end date. The first one ends about 2036.", "`AG/internal/localapi/cert.go:25` · class T1 · S01")},
rb="no hand step"),
"DB04": dict(src=("dates-box", 3), name="Off-site copy after the household stops it: the 14-day countdown", seed="Window",
named=("grep", r"14.day countdown|abandon"),
o={6: ("N", "NOBODY KNOWS for a box that is off during the whole countdown: the reminders at 5, 3 and 1 days come from the box. Nobody traced if the old store is removed with no reminder shown.",
"`CT/internal/backup/offbox_abandon.go:31-40` · class T4 · S23")},
rb="no hand step"),
"DB05": dict(src=("dates-box", 4), name="Off-site history: how far back a household can go", seed="Window",
named=("grep", r"keepDaily|keep-daily|7 daily"),
o={5: ("A", "Nobody, and by ruling: the window is the rule. A point leaves it with no warning.", None)},
rb="no hand step"),
"DB06": dict(src=("dates-box", 5), name="Off-site credentials: SSH key, host key, store password, PBS pin", seed="Key",
named=("grep", r"host key|known_hosts|sub-account"),
o={5: ("A", "Nobody, because there is no date. A failed night run is the sign.", None)},
rb="yes — `RB/offsite-endpoint.md`"),
"DB08": dict(src=("dates-box", 7), name="Claim code (72 hours) and the one-use gate token", seed="Window",
named=("grep", r"claim code|claimCodeTTL|72 h"),
o={5: ("A", "Nobody before the end. The household sees the dead code at the form, and the operator sends a new one.", None)},
rb="no hand step"),
"DB09": dict(src=("dates-box", 8), name="Share link", seed="Window", named=("grep", r"/s/<token>|share token|LauncherShareToken"),
o={5: ("A", "Nobody, because there is no date. The link ends only when the household changes it.", None)},
rb="no hand step"),
"DB11": dict(src=("dates-box", 10), name="Signed operator jobs: the validity window", seed="Window",
named=("NAMED", "`AR/04-control-plane-authorization.md:139-150`"),
o={5: ("A", "Nobody before. A job that ends with no result is what the operator sees. A new job is signed.", None)},
rb="yes — the build skill's agent section"),
"DB12": dict(src=("dates-box", 11), name="Operator signing keys on each box", seed="Key",
named=("NAMED", "`AR/04-control-plane-authorization.md:139-150`"),
o={3: ("D", "The keys have no end date. A change of key is designed as a signed order of its own. Only the name of that order exists in code.",
"`AR/04-control-plane-authorization.md:139-150`"),
7: ("K", "If the keys are lost, no box accepts a signed job again without a visit. The keys exist on DooPlex only.", "R-924")},
rb="no — the design exists, a procedure does not"),
"DB13": dict(src=("dates-box", 12), name="The agent's Proxmox access token", seed="Key",
named=("grep", r"pveum|API token|privsep"),
o={2: ("A", "MEASURED 2026-11-11 on the three boxes: the user and its token have no end date (expire 0).", "`EV/box/pin.txt` `E/scripts/felhom-host-install.sh:2448`"),
5: ("A", "Nobody, because there is no date.", "`EV/box/pin.txt`")},
rb="no hand step"),
"DB16": dict(src=("dates-box", 15), name="Package signing keys: Docker in the guest, Proxmox and Debian on the host", seed="Key",
named=("MENTIONED", "`AR/11-os-updates.md:657`"),
o={2: ("A", "MEASURED 2026-11-11 on the three boxes: Docker's key never ends. The Debian keys in use end 2029-01-15 at the soonest. The Proxmox key for this series ends 2034-11-10. Two keys that end 2027-02-03 and 2027-04-12 are in the 'removed keys' file; nothing uses them.",
"`EV/box/keyrings.txt`"),
5: ("N", "Nobody before the end. After it the box reports a failed update. The first date is 2029-01-15.", "`AR/11-os-updates.md:657` `EV/box/keyrings.txt` · class T1 · S10"),
6: ("N", "NOBODY KNOWS. A box that was off for years has old keys. The new keys come as a package, and the package list is checked with the old keys.",
"`AR/11-os-updates.md:657` · class T4 · S11")},
rb="NO — the document says a key change is 'a slow-lane act for a person'; no procedure is written"),
"DB17": dict(src=("dates-box", 16), name="Debian 13 and Proxmox 9: the series the product is tied to", seed="OS",
named=("grep", r"trixie|Debian 13|Proxmox (VE )?9"),
o={2: ("N", "No document gives the day when security fixes stop for Debian 13 or for Proxmox 9. MEASURED: the three boxes run Debian 13.7 and Proxmox 9.2.",
"`EV/box/demo-hp-host-dates.txt` `AR/11-os-updates.md:1` · class T3 · S09"),
3: ("N", "No plan with a date. The move to the next series is not designed.", "`AR/11-os-updates.md:1` · class T3 · S09"),
5: ("N", "Nobody. The dates are public; nothing in the product or in a document reads them.", "`AR/11-os-updates.md:1` · class T1 · S09")},
rb="NO"),
"DB18": dict(src=("dates-box", 17), name="Catalog app images and their recorded digests", seed="Version",
named=("MENTIONED", "`AR/09-update-architecture.md:391`"),
o={5: ("N", "Nobody. No check asks the registries if every recorded image is still served. The document speaks of the day a version is recorded, not of the days after.",
"`AR/09-update-architecture.md:391` `CAT/scripts/check-currency.py:22` · class T1 · S21")},
rb="no hand step"),
"DB19": dict(src=("dates-box", 18), name="Images the controller itself starts (Traefik, tunnel client, file browser, Samba)", seed="Version",
named=("grep", r"cloudflared|TraefikImage|traefik:v"),
o={2: ("N", "No document says for how long Cloudflare accepts an old tunnel client. MEASURED on the N100: tunnel client 2026.10.3, Traefik v3.7.16.",
"`EV/box/demo-felhom-guest-dates.txt` · class T3 · S12"),
5: ("N", "Nobody.", "`EV/box/demo-felhom-guest-dates.txt` · class T1 · S12"),
6: ("N", "NOBODY KNOWS if the tunnel of a box that was off for a year still connects with its old client. Without the tunnel the box gets no update from outside.",
"`EV/box/demo-felhom-guest-dates.txt` · class T4 · S12")},
rb="no hand step"),
"DB20": dict(src=("dates-box", 19), name="Published agent versions: the newest five stay", seed="Version",
named=("NAMED", "`AR/09-update-architecture.md` §3, D50"), o={}, rb="yes — the build skill's agent section"),
"DB21": dict(src=("dates-box", 20), name="The golden image's age (7 days) and the waiver (14 days)", seed="Version",
named=("NAMED", "`AR/09-update-architecture.md:1010-1018`"),
o={2: ("A", "MEASURED 2026-11-11: the newest golden is 8 days old (baked 2026-11-03). The limit is 7. No waiver file exists. The gate fails today.",
"`E/scripts/golden_currency_gate.py:169` `EV/box/golden-gate.txt`"),
7: ("K", "A push of code to this repository is refused until a golden is baked or a dated waiver is written. A push of documents passes with a notice.", "R-468")},
rb="yes — `RB/RUNBOOK-manual-build.md`"),
"DB24": dict(src=("dates-box", 23), name="Copies kept before a restore: deleted after 30 unused days", seed="Window",
named=("grep", r"restore cop|restore-undo|D62"),
o={5: ("N", "Nobody. The household is not told before the copy goes. The household can delete it sooner by hand.",
"`CT/internal/backup/restore_copy_sweep.go:18-29` · class T1 · S05"),
6: ("N", "A box that was off for more than 30 days deletes the copy at its first sweep after the start: the age is counted from the folder's time stamp by the wall clock. Nobody asked if the off time must count.",
"`CT/internal/backup/restore_copy_sweep.go:21-29` · class T4 · S11")},
rb="no hand step"),
# ------------------------------------------------------------------ hub, operator, outside
"DH01": dict(src=("dates-hub", 0), name="Connect link in the household's e-mail (7 days)", seed="Window",
named=("grep", r"self-bind|selfbind|/bind/"),
o={5: ("A", "Nobody before, by design: the page of a dead link has a button that sends a new one.", None)},
rb="no hand step"),
"DH05": dict(src=("dates-hub", 4), name="'Switched off on purpose' hold (14 days)", seed="Window",
named=("grep", r"deliberate.off|switched off on purpose"), o={}, rb="no hand step"),
"DH08": dict(src=("dates-hub", 7), name="Storage Box sub-account password, and the hub's seal key", seed="Key",
named=("grep", r"seal key|sub-account"),
o={3: ("D", NM + "The password changes when the operator presses Re-issue. A change of the seal key is 'not built'.", "`RB/secrets.md:159`"),
5: ("A", "Nobody, because there is no date.", None)},
rb="partly — `RB/secrets.md:159` says the seal key change is not built"),
"DH09": dict(src=("dates-hub", 8), name="PBS off-site token on ep0", seed="Key", named=("NAMED", "`AR/05-hub-architecture.md:379-387`"), o={},
rb="yes — `RB/offsite-endpoint.md`"),
"DH10": dict(src=("dates-hub", 9), name="WireGuard keys of the box's tunnel to the off-site endpoint", seed="Key",
named=("NAMED", "`AR/06-offsite-connectivity.md:200-205`"),
o={5: ("A", "Nobody, because there is no date.", None)}, rb="yes — `RB/offsite-endpoint.md`"),
"DH14": dict(src=("dates-hub", 13), name="Recovery mode for a host (30 minutes to 4 hours)", seed="Window",
named=("grep", r"recovery mode|defaultRecoveryTTL"),
o={5: ("A", "Nobody before. The operator who opened the window is present; he opens it again.", None)},
rb="yes — the recovery runbooks"),
"DH15": dict(src=("dates-hub", 14), name="Break-glass password of a host", seed="Key", named=("grep", r"break-glass|host_recovery"), o={},
rb="yes — `RB/break-glass-sheet.md:75` (print the sheet again each year)"),
"DH19": dict(src=("dates-hub", 18), name="The installer ISO", seed="Version", named=("grep", r"ISO_VERSION|installer ISO|iso\.felhom\.eu"),
o={3: ("N", "No document says how old a public ISO may be, or when to build a new one. The ISO carries a Proxmox base and a first-boot program that age.",
"`E/scripts/iso/build-felhom-iso.sh:51` `RB/iso-release-gate.md:1` · class T3 · S22")},
rb="yes for a build — `RB/iso-release-gate.md`; NO for 'when'"),
"DH20": dict(src=("dates-hub", 19), name="The certificate that the hub's address serves", seed="Certificate",
named=("grep", r"cert-manager|letsencrypt-prod"),
o={2: ("A", "Read from outside 2026-11-11, with no login: the certificate of the hub's address ends 2027-01-12. The download address ends 2026-12-23, the ISO address 2026-12-27, the Gitea address 2027-01-03, the website 2027-01-28.",
"`EV/box/public-reads.txt`"),
5: ("N", NM + "In the Felhom repositories nothing warns before this date. If it ends, every box fails to report, and the hub then sees all boxes as silent.",
"`E/manifests/hub.yaml:342` `EV/box/public-reads.txt` · class T1 · S01")},
rb="NO"),
"DH22": dict(src=("dates-hub", 21), name="Operator access tokens, among them CC's hub token", seed="Key",
named=("NAMED", "`AR/05-hub-architecture.md` §17"),
o={5: ("N", "Nobody. A token lives 90 days by default and 365 at most. The token page shows the date; no monitor reads it. On the end date the release step that sets a floor answers 401.",
"`HUB/web/tokens.go:29-30` `HUB/web/tokens.go:180` · class T1 · S03")},
rb="partly — the build skill says '401 = token expired'; no step says to look at the date before"),
"DH23": dict(src=("dates-hub", 22), name="A household's own Cloudflare token", seed="Key", named=("grep", r"Cloudflare token|CF_DNS_API_TOKEN"),
o={2: ("N", "The hub and the box do not know the end date that the household set at Cloudflare. No document tells the household to set none.",
"`HUB/cloudflare/reach.go:14` · class T3 · S15"),
5: ("N", "Nobody. When the token ends, the web certificate stops renewing (see DB01), with no sign until the certificate's own end.",
"`CT/internal/infra/infra.go:157-159` · class T1 · S15"),
6: ("A", "The running apps and the tunnel do not use this token. The certificate renewal does.", "`CT/internal/infra/infra.go:157-159`")},
rb="NO"),
"DH24": dict(src=("dates-hub", 23), name="The mail service key (all household and operator mail)", seed="Account",
named=("MENTIONED", "`AR/_recovery-inventory-2026-07-28.md`"),
o={2: ("N", NM + "No document gives an end date for the key or for the account.", "`RB/secrets.md:33` · class T3 · S08"),
5: ("N", "Nobody. No scheduled check proves that mail still goes out. If the key ends, no alarm mail can say so.", "`RB/secrets.md:33` · class T1 · S08")},
rb="yes for a new key — `RB/secrets.md:33`"),
"DH25": dict(src=("dates-hub", 24), name="The hub's global bearer key", seed="Key", named=("NAMED", "`AR/13-threat-model.md:70`"), o={},
rb="yes — `RB/secrets.md:124`"),
"DH26": dict(src=("dates-hub", 25), name="Pinned tool versions in builds and CI", seed="Version", named=("grep", r"security.watch|pinned by digest"), o={},
rb="no hand step"),
"DH27": dict(src=("dates-hub", 26), name="Domains and registrars", seed="Account", named=("grep", r"registrar|domain renew"),
o={2: ("N", NM + "No document gives a registrar or a renewal date for any of the product's names. This run could not read the public records: no whois tool here, and the public lookups for .eu and .hu gave no answer.",
"`EV/box/public-reads.txt` · class T3 · S06"),
5: ("N", "Nobody that a document names. A name that ends takes down the hub address, the website, the mail and every household address under it.",
"`EV/box/public-reads.txt` · class T1 · S06")},
rb="NO"),
"DH28": dict(src=("dates-hub", 27), name="Cloudflare account, the site publish token, the ISO store keys", seed="Account",
named=("grep", r"CF_PAGES_TOKEN|R2 "),
o={2: ("N", NM + "No document gives an end date for the account, for the publish token or for the store keys.", "`E/scripts/publish-site-cloudflare.sh:12` · class T3 · S07"),
5: ("N", "Nobody.", "`E/scripts/publish-site-cloudflare.sh:12` · class T1 · S07")},
rb="NO"),
"DH29": dict(src=("dates-hub", 28), name="Hetzner account: the off-site endpoint and the storage boxes", seed="Account",
named=("MENTIONED", "`AR/_recovery-inventory-2026-07-28.md:1019`"),
o={2: ("N", NM + "No document gives a payment day, a contract end or the end of the API token. The recovery inventory asks the question and no document answers it.",
"`AR/_recovery-inventory-2026-07-28.md:1019` · class T3 · S07"),
5: ("N", "Nobody before. After the end, the hub's 'off-site copy is old' alarm fires at 8 days.", "`HUB/monitor/deadline_tiers.go:1` · class T1 · S07")},
rb="NO"),
"DH30": dict(src=("dates-hub", 29), name="Tailscale node keys (the operator's remote way in)", seed="Key",
named=("grep", r"tailscale|tailnet"),
o={2: ("A", "DOCUMENT: demo-hp's key ends 2027-01-17; the other two nodes have no end. MEASURED 2026-11-11: the Tailscale service is not running on demo-hp and Tester 1; it runs on the N100. So the date could not be read on the box.",
"`E/documentation/operations/tailscale.md:41-43` `EV/box/pin.txt`"),
5: ("N", "Nobody. The date is on the Machines page of the Tailscale console.", "`E/documentation/operations/tailscale.md:41-43` · class T1 · S04")},
rb="yes — `E/documentation/operations/tailscale.md:41-43`"),
"DH31": dict(src=("dates-hub", 30), name="Gitea and registry credentials", seed="Key", named=("grep", r"gitea.{0,40}(token|credential)"),
o={2: ("N", NM + "No document gives an end date for the scoped token or for CC's Gitea token.", "`RB/secrets.md:1` · class T3 · S13"),
5: ("N", "Nobody.", "`RB/secrets.md:1` · class T1 · S13"),
6: ("A", "Not a box matter: a box pulls its images with no login.", "`RB/secrets.md:1`")},
rb="yes for a new token — `RB/secrets.md`"),
"DH32": dict(src=("dates-hub", 31), name="CC's secret store: the tokens inside", seed="Key", named=("grep", r"cc-secrets|sec run"),
o={2: ("N", "The store holds a value and no end date. A token with an end at its issuer (the GitHub advisory token, the hub token) ends with no record here.",
"`E/scripts/security_watch.py:1` · class T3 · S14"),
5: ("N", "Nobody. The nightly security watch and the release step fail on the day.", "`E/scripts/security_watch.py:1` · class T1 · S14")},
rb="NO"),
"DH33": dict(src=("dates-hub", 32), name="SSH keys between the hub, DooPlex and the endpoint", seed="Key",
named=("grep", r"ssh key|id_ed25519"), o={}, rb="yes — `RB/offsite-endpoint.md:313`"),
}
MEASURED = {
"DB01": ("MEASURED: the three end dates, and one renewal by the product on demo-hp (the certificate was issued again on 2026-10-21)", "`EV/box/demo-hp-guest-dates.txt`"),
"DB02": ("MEASURED: the three end dates, pin equals certificate on all three, and Proxmox's own renewal rule read on demo-hp", "`EV/box/pin.txt` `EV/box/pveupdate-excerpt.txt`"),
"DB13": ("MEASURED: expire 0 on three boxes", "`EV/box/pin.txt`"),
"DB16": ("MEASURED: every key's end date on three hosts and three guests", "`EV/box/keyrings.txt`"),
"DB21": ("MEASURED: the gate's own output today", "`EV/box/golden-gate.txt`"),
"DH20": ("READ FROM OUTSIDE, no login", "`EV/box/public-reads.txt`"),
}
TINY = [
"login sessions on the box and on the hub (hours to 7 days; a new login repairs it)",
"the one-use codes of the recovery ceremony (minutes)",
"the guest's token for the agent and the hub key (no date)",
"pinned build tools of the controller (a build fails loudly)",
"the hub's claim code and pairing code (no date; three resets a day)",
"operator actions in the queue (24 hours; a restart order 30 minutes)",
"the vouch alarm (2 hours)",
"the Debian template that the golden is built from (a bake fails loudly; the operator changes the default)",
]
LEFT_OUT = [
("The recovery passphrase", "it stops on an event (a new one is made), not on a date"),
("Kept escrow packages", "they have no end date by ruling; the 14-day countdown that removes one is DB04"),
("The off-site clean-up window (20 minutes, 6.5 days)", "it is a timer, so it is on the Clock list"),
("The hub's view of signed jobs, signing keys, the golden's age, the agent versions, and OS support", "the same things as DB11, DB12, DB21, DB20 and DB17"),
("WireGuard re-key on the box", "the same thing as DH10"),
]
# ------------------------------------------------------------------ second pass
# Helper answers to questions 3 to 7 that said "not found" or "candidate": the main session's words in their place.
NODATE = "No renewal on a date exists, so none can fail on a date."
HAND = "The renewal is the operator's hand step at the outside account. If it is not done, the thing ends (question 4)."
for _id, _q, _cell in [
("DB01", 3, ("A", "Traefik renews by itself, by its own default lead time. MEASURED on demo-hp: it got a new certificate on 2026-10-21, 90 days before the end date 2027-01-19.", None)),
("DB01", 7, ("N", "Traefik tries again by itself. The old certificate works until its end date. Nothing in the product reads the result, so a renewal that fails for weeks is silent.",
"`CT/internal/infra/templates/traefik.yml.tmpl:51` · class T1 · S01")),
("DB03", 4, ("A", "At the end date nothing breaks: the controller compares the fingerprint only, not the date. A NEW certificate is what breaks the link.", None)),
("DB05", 7, ("K", "A clean-up that the guard refuses leaves more history than the rule says, not less.", "R-1294")),
("DB06", 6, ("A", "Yes if nothing changed outside. A guest that was built again gets a new store password and cannot open the old store; the 14-day countdown (DB04) is the way out.", None)),
("DB09", 4, ("A", "Nothing ends. A link that was given away years ago still opens the page, until the household changes it.", None)),
("DB19", 4, ("A", "If Cloudflare stops accepting an old tunnel client, the household's address stops answering from outside while the box is healthy. If a registry drops a tag, a box that is built again cannot start that part.", None)),
("DH08", 7, ("A", NODATE, None)), ("DH15", 7, ("A", NODATE, None)), ("DH25", 7, ("A", NODATE, None)), ("DH33", 7, ("A", NODATE, None)),
("DH23", 7, ("A", "The renewal is the household's hand step at Cloudflare. If it is not done, the web certificate stops renewing (DB01).", None)),
("DH27", 7, ("A", HAND, None)), ("DH29", 7, ("A", HAND, None)), ("DH30", 7, ("A", HAND, None)), ("DH31", 7, ("A", HAND, None)), ("DH32", 7, ("A", HAND, None)),
("DH27", 3, ("A", "The operator, at the registrar.", None)),
("DH24", 7, ("A", "If the key ends and no new key is made, no mail goes out, alarm mail included (question 5).", None)),
("DH25", 6, ("A", "No date drives it. Boxes use their own keys since 2026-10-12. What a box does that holds only the old global key after a change was not traced in this run.", None)),
("DH26", 7, ("A", "A CI job that does not run leaves no result. Whether a missed night run is seen is a question of the Clock list.", None)),
("DH20", 3, ("A", "NOT MEASURED on DooPlex. cert-manager on DooPlex renews by itself. Read from outside: the certificate served today was issued on 2026-10-14, so a renewal worked.",
"`E/manifests/hub.yaml:342` `EV/box/public-reads.txt`")),
("DH20", 6, ("A", "Yes. The agent trusts the public issuers, not one certificate, so a renewed certificate is accepted.", "`AG/internal/hub/client.go:41-51`")),
("DH20", 7, ("A", "The old certificate serves until its end date. Nothing in the Felhom repositories warns (question 5).", "`E/manifests/hub.yaml:342`")),
]:
M[_id]["o"][_q] = _cell
@@ -0,0 +1,29 @@
#!/bin/bash
# public-reads.sh — what any visitor can read with no login: the TLS certificate a public name serves, and the public
# registry record of a domain. Read-only; no account call. For a certificate: subject, issuer, two dates, fingerprint.
set -u
printf 'taken %s from DooPlex\n' "$(date -u +%FT%TZ)"
printf '\n=== certificates served by public names ===\n'
for n in "$@"; do
case "$n" in */*) continue;; esac
ip=$(getent ahostsv4 "$n" | awk 'NR==1{print $1}')
out=$(printf '' | timeout 8 openssl s_client -connect "$n:443" -servername "$n" 2>/dev/null | openssl x509 -noout -subject -issuer -startdate -enddate -fingerprint -sha256 2>/dev/null | tr '\n' ' ')
printf '%s [resolves here to %s] -> %s\n' "$n" "${ip:-none}" "${out:-NO CERTIFICATE READ}"
done
printf '\n=== public registry records (domains) ===\n'
for d in "$@"; do
case "$d" in */*) d=${d#domain/};; *) continue;; esac
printf -- '--- %s\n' "$d"
if command -v whois >/dev/null; then
timeout 15 whois "$d" 2>&1 | /usr/bin/grep -i -E 'expir|renew|paid-till|registered|created|record created|registrar:|status|^domain' | /usr/bin/grep -v -i -E 'terms|abuse|notice' | head -12
else
printf 'no whois tool on DooPlex\n'
fi
tld=${d##*.}
case "$tld" in eu) u="https://rdap.eu/domain/$d";; hu) u="https://rdap.nic.hu/domain/$d";; *) u="https://rdap.org/domain/$d";; esac
timeout 15 curl -s -L -m 12 -H 'Accept: application/rdap+json' "$u" | python3 -c '
import json,sys
try: d=json.load(sys.stdin)
except Exception as e: print("rdap: no JSON answer"); sys.exit(0)
print("rdap status:", d.get("status"), "| events:", [(e.get("eventAction"), e.get("eventDate")) for e in d.get("events",[])] or "none", "| error:", d.get("errorCode"))'
done
@@ -0,0 +1,155 @@
#!/usr/bin/env python3
"""resolve-helper.py HELPER.md [--json OUT] — open every pointer in a helper's answer BEFORE it may enter the table.
A helper (a read-only subagent) writes pointers in three shapes: a full one (`CT/internal/stacks/delete.go:624`), a
bare file name (`delete.go:756`) and a short one (`:1025`, "same file as the last one"). This tool
1. resolves each to a full `PREFIX/path:line[-line]`,
2. says DEAD when the file or the line does not exist, UNRESOLVED when a bare name fits no file or more than one,
3. checks the NAME the helper put in front of the pointer (a backticked identifier within 90 characters, with no other
pointer between): it must stand within 3 lines of the cited range. IDENT-OK / IDENT-MISS / NO-IDENT.
It does not judge the claim. A claim that carries a mark in TABLE.md is read by the main session (check-pointers.py
prints the lines). Read-only: it opens files in the git clones on DooPlex and prints.
"""
import json, os, re, sys
HERE = os.path.dirname(os.path.abspath(__file__))
REPO = os.path.abspath(os.path.join(HERE, "..", "..", "..", ".."))
WS = os.path.dirname(REPO)
PREFIX = {"AG": os.path.join(WS, "felhom-agent"), "CT": os.path.join(WS, "felhom-controller", "controller"),
"CAT": os.path.join(WS, "app-catalog-felhom.eu"), "HUB": os.path.join(REPO, "hub", "internal"), "E": REPO,
"AR": os.path.join(REPO, "documentation", "architecture"), "RB": os.path.join(REPO, "documentation", "runbooks")}
RANGES = r"~?:?(\d+(?:-\d+)?(?:\s?,\s?:?\d+(?:-\d+)?)*)"
PTR = re.compile(r"(?P<full>(?:%s)/[\w./@+-]+\.\w+)(?::(?P<fr>\d+(?:-\d+)?(?:,\s?:?\d+(?:-\d+)?)*))?"
r"|(?<![\w/.-])(?P<bare>[\w.-]+\.(?:go|sh|py|md|yml|yaml|html|json|service|timer))(?::(?P<br>\d+(?:-\d+)?(?:,\s?:?\d+(?:-\d+)?)*))"
r"|(?<![\w`/])~?:(?P<sr>\d+(?:-\d+)?)(?![\d:])" % "|".join(PREFIX))
IDENT = re.compile(r"`([^`]{2,80})`")
_cache, _walk = {}, {}
def lines_of(path):
if path not in _cache:
try:
_cache[path] = open(path, encoding="utf-8", errors="replace").read().split("\n")
except OSError:
_cache[path] = None
return _cache[path]
def find_base(pre, base):
if pre not in _walk:
idx = {}
for d, dn, fn in os.walk(PREFIX[pre]):
dn[:] = [x for x in dn if x not in (".git", "node_modules", "vendor", "evidence")]
for f in fn:
idx.setdefault(f, []).append(os.path.relpath(os.path.join(d, f), PREFIX[pre]))
_walk[pre] = idx
return _walk[pre].get(base, [])
def check(full, rng, text, pos):
pre, rest = full.split("/", 1)
ls = lines_of(os.path.join(PREFIX[pre], rest))
if ls is None:
return "DEAD", "NO-IDENT"
if not rng:
return "OK", "NO-IDENT"
a, _, b = rng.partition("-")
a, b = int(a), int(b or a)
if b > len(ls):
return "DEAD", "NO-IDENT"
back = text[max(0, pos - 90):pos]
cut = max(back.rfind(":"), back.rfind(";"), back.rfind(",")) # a pointer or a clause break in between: do not reach over it
ids = [m for m in IDENT.finditer(back) if m.start() > cut - 1 or cut < 0]
ids = ids or list(IDENT.finditer(back))[-1:]
if not ids:
return "OK", "NO-IDENT"
name = re.sub(r"[()…]|\.\.\.", "", ids[-1].group(1)).strip().split(" ")[0].split("(")[0]
name = name.split(".")[-1] if re.fullmatch(r"[\w.]+", name) else name
win = "\n".join(ls[max(0, a - 4):b + 3])
return "OK", ("IDENT-OK" if name and name in win else f"IDENT-MISS({name})")
def resolve(text, known, default_pre):
"""Returns (text with every pointer rewritten as a backticked full token, [records])."""
out, recs, last, i = [], [], None, 0
for m in PTR.finditer(text):
out.append(text[i:m.start()])
i = m.end()
if m.group("full"):
full, rngs = m.group("full"), m.group("fr")
elif m.group("bare"):
base, rngs = m.group("bare"), m.group("br")
full = known.get(base)
if not full:
c = find_base(default_pre, base)
full = f"{default_pre}/{c[0]}" if len(c) == 1 else None
if not full:
recs.append({"ptr": m.group(0), "file": "UNRESOLVED", "ident": "NO-IDENT"})
out.append(m.group(0))
continue
else:
full, rngs = last, m.group("sr")
if not full:
out.append(m.group(0))
continue
last = full
known.setdefault(os.path.basename(full), full)
toks = []
for r in re.split(r"\s?,\s?:?", rngs) if rngs else [None]:
st, idn = check(full, r, text, m.start())
tok = f"{full}:{r}" if r else full
recs.append({"ptr": tok, "file": st, "ident": idn})
toks.append(f"`{tok}`")
out.append(" ".join(toks))
out.append(text[i:])
return "".join(out), recs
def main():
src = sys.argv[1]
text = open(src, encoding="utf-8").read()
default_pre = "CT" if "controller" in src else "AG" if "agent" in src else "HUB" if "hub" in src else "AR"
known = {}
for m in re.finditer(r"((?:%s)/[\w./@+-]+\.\w+)" % "|".join(PREFIX), text): # first pass: every full path the helper named
known.setdefault(os.path.basename(m.group(1)), m.group(1))
endings, cur, allrecs, lastq = [], None, [], None
for line in text.split("\n"):
h = re.match(r"^### (.+)$", line)
if h:
cur = {"title": h.group(1).strip(), "answers": {}, "extra": []}
endings.append(cur)
lastq = None
continue
if cur is None:
continue
a = re.match(r"^([1-8])\. (.*)$", line)
body = a.group(2) if a else line
new, recs = resolve(re.sub(r"`((?:%s)/[^`]+)`" % "|".join(PREFIX), r"\1", body), known, default_pre)
for r in recs:
r["ending"] = cur["title"]
r["q"] = a.group(1) if a else (lastq or "-")
allrecs += recs
if a:
cur["answers"][a.group(1)] = new
lastq = a.group(1)
elif line.strip() and lastq and not line.startswith(("#", "---")):
cur["answers"][lastq] += " · " + new.strip().lstrip("-* ") # a continuation line belongs to the answer above it
elif line.strip():
cur["extra"].append(new)
n = len(allrecs)
dead = [r for r in allrecs if r["file"] != "OK"]
miss = [r for r in allrecs if r["ident"].startswith("IDENT-MISS")]
ok = sum(1 for r in allrecs if r["ident"] == "IDENT-OK")
print(f"{os.path.basename(src)}: endings {len(endings)} · pointers {n} · dead/unresolved {len(dead)} · "
f"name found at the line {ok} · name NOT at the line {len(miss)} · no name to check {n - ok - len(miss) - len(dead)}")
for r in dead:
print(f" {r['file']:<10} {r['ptr']} [{r['ending'][:50]} q{r['q']}]")
for r in miss:
print(f" {r['ident']:<28} {r['ptr']} [{r['ending'][:50]} q{r['q']}]")
if "--json" in sys.argv:
json.dump({"endings": endings, "pointers": allrecs}, open(sys.argv[sys.argv.index("--json") + 1], "w"), indent=1)
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,26 @@
#!/bin/bash
# run-box.sh KIND NAME SSHTARGET — run tools/host-KIND.sh on the host and tools/guest-KIND.sh in guest 9201, read-only.
# Output: box/NAME-host-KIND.txt and box/NAME-guest-KIND.txt beside this folder.
set -u
D=$(cd "$(dirname "$0")/.." && pwd)
k=$1; n=$2; t=$3
SSH="ssh -o BatchMode=yes -o ConnectTimeout=10"
{
printf '# host-%s %s taken %s from DooPlex\n' "$k" "$n" "$(date -u +%FT%TZ)"
timeout 240 $SSH "$t" 'bash -s' < "$D/tools/host-$k.sh"
printf 'RC=%s\n' "$?"
} > "$D/box/$n-host-$k.txt" 2>&1
{
printf '# guest-%s %s taken %s from DooPlex\n' "$k" "$n" "$(date -u +%FT%TZ)"
timeout 240 $SSH "$t" 'pct exec 9201 -- bash -s' < "$D/tools/guest-$k.sh"
printf 'RC=%s\n' "$?"
} > "$D/box/$n-guest-$k.txt" 2>&1
if [ "$k" = clock ]; then
{
printf '# offset %s: box clock minus DooPlex clock (midpoint of the call), seconds\n' "$n"
for i in 1 2 3 4 5; do
a=$(date +%s.%N); b=$($SSH "$t" 'date +%s.%N'); c=$(date +%s.%N)
awk -v a="$a" -v b="$b" -v c="$c" 'BEGIN{printf "offset=%.3f rtt=%.3f\n", b-(a+c)/2, c-a}'
done
} > "$D/box/$n-offset.txt" 2>&1
fi
@@ -0,0 +1,9 @@
#!/bin/bash
# run-pin.sh — run host-pin.py on the three test hosts, read-only. Output: box/pin.txt
D=$(cd "$(dirname "$0")/.." && pwd)
{ printf 'taken %s from DooPlex\n' "$(date -u +%FT%TZ)"
for t in demo-hp root@192.168.0.154 felhom-pve; do
printf '== %s\n' "$t"
ssh -o BatchMode=yes -o ConnectTimeout=10 "$t" 'python3 -' < "$D/tools/host-pin.py"
ssh -o BatchMode=yes -o ConnectTimeout=10 "$t" 'pveum user token list felhom-agent@pve --output-format json; systemctl is-active tailscaled'
done; } > "$D/box/pin.txt" 2>&1
@@ -0,0 +1,41 @@
#!/bin/bash
# searches.sh — the searches an `N` in TABLE.md must survive. Read-only; prints to stdout (saved as evidence/searches.txt).
# The same tool as audits/endings-list-2026-11-10/tools/searches.sh with this list's searches.
# Each search: an ASCII fragment (no accented letter), case-insensitive, over (ar) documentation/architecture,
# (op) backlog/OPEN-ITEMS.md, (cl) backlog/CLOSED-ITEMS.md, (rm) backlog/ROADMAP.md. For register files it prints the row ids
# that match, so a reader can open them. Controls: C+ must hit, C- must not.
E="$(cd "$(dirname "$0")/../../../.." && pwd)"
B="$E/documentation/backlog"; AR="$E/documentation/architecture"
G=/usr/bin/grep
ids() { $G -i -E -- "$1" "$2" 2>/dev/null | $G -o -E '^\| \*\*R-[0-9a-z]+\*\*' | tr -d '|* ' | sort -u | tr '\n' ' '; }
s() { # id, what, pattern
local n_ar; n_ar=$($G -r -i -l -E -- "$3" "$AR" 2>/dev/null | wc -l)
printf '%s | %s | /%s/ | ar files=%s | open: %s| closed: %s| roadmap=%s\n' "$1" "$2" "$3" "$n_ar" "$(ids "$3" "$B/OPEN-ITEMS.md")" "$(ids "$3" "$B/CLOSED-ITEMS.md")" "$($G -c -i -E -- "$3" "$B/ROADMAP.md")"
}
date -u +'searches run %Y-%m-%d %H:%M:%S UTC'
s C+ "positive control (a row that exists)" "R-1301"
s C+2 "positive control (a word in all four)" "certificate"
s C- "negative control" "zzqxenddatezz"
s S01 "web certificate: warning before it ends" "(certificate|cert).{0,80}(expir|renew|runs out|days left)|acme\.json|NotAfter"
s S02 "Proxmox renews its certificate, the agent's pin does not follow" "pve-ssl|pveproxy.{0,60}cert|(tls|certificate|cert) pin.{0,120}(renew|expir|rotat|re-pin|repin)|fingerprint.{0,80}(renew|expir|stale)"
s S03 "operator access token ends, nobody warned" "(access|api|hub|operator).token.{0,120}(expir|90 days|days left|renew)|HUB_TOKEN.{0,80}(expir|ends|renew)|token_expir"
s S04 "Tailscale key expiry" "tailscale.{0,120}(expir|key)|key expiry"
s S05 "restore copy swept after 30 days, no warning" "restore cop.{0,100}(30 days|sweep|unused)|restoreCopyUnusedAge"
s S06 "domain renewal / registrar" "registrar|domain.{0,60}(renew|expir|lapse)|whois"
s S07 "paid accounts: billing, contract end" "(hetzner|cloudflare|resend|mail service).{0,100}(billing|invoice|unpaid|contract|account (is )?closed|payment)|credit card"
s S08 "mail service key end / check that mail still goes" "(resend|mail).{0,40}(key|token).{0,80}(expir|rotat|ends)|canary mail|test mail.{0,40}(daily|weekly)"
s S09 "OS support end (Debian 13, Proxmox 9)" "EOL|end of life|end-of-life|support (ends|end date)|trixie.{0,80}(until|ends|support)"
s S10 "apt signing keys end" "(signing|apt|archive|repo) key.{0,100}(expir|ends|rotat)|keyring.{0,80}(expir|old)|Valid-Until"
s S11 "a box off for a long time comes back" "(off|unplugged|powered off|switched off|in a drawer).{0,60}(months|a year|years)|long.{0,10}(off|absence)|cold box"
s S12 "old tunnel client / infra image stops being accepted" "cloudflared.{0,100}(old|version|support|deprecat)|tunnel client.{0,60}(old|support)"
s S13 "Gitea / registry token end" "(gitea|registry).{0,60}(token|credential|password).{0,80}(expir|ends|rotat)"
s S14 "GitHub advisory token / CC store token end" "GH_ADVISORY_TOKEN|advisory token|github.{0,40}token.{0,60}(expir|ends)"
s S15 "household's own Cloudflare token end" "(cloudflare|cf).{0,20}token.{0,100}(expir|ends|ttl|revok)"
s S16 "operator signing key rotation" "key.rotation|re-pin|rotat.{0,40}(operational|signing|cold) key"
s S17 "seal key rotation" "seal.{0,10}key.{0,100}(rotat|lost|change)"
s S18 "a list of what ends when" "end dates|expiry (list|calendar|table)|renewal calendar|what ends"
s S19 "golden age: warning before day 7" "golden.{0,80}(7 days|too old|age).{0,80}(warn|remind)|waiver.{0,60}(expir|ends)"
s S20 "served certificate of the hub address (cert-manager)" "cert-manager|letsencrypt-prod|ClusterIssuer"
s S21 "a recorded image digest is no longer served" "digest.{0,100}(still served|no longer|gone|deleted|removed from)|registry.{0,60}(drops|deletes|garbage)"
s S22 "the installer ISO grows old" "ISO.{0,80}(too old|stale|age|rebuild every|rebuilt every)"
s S23 "a box off during the 14-day off-site countdown" "(abandon|countdown).{0,120}(box is off|switched off|offline|unreachable)"
@@ -0,0 +1,36 @@
# The End-dates list — the table
Built 2026-11-11 from the source of the four repositories, from read-only measurements on the three test boxes
(Tester 1, demo-hp, the N100; host and guest) and from what any visitor can read with no login. One block for each
thing that stops working, or stops being safe, on a date or after an age. `tools/make-table.py` writes this file from
the helpers' answers (`evidence/helpers/`) and from `tools/marks.py`; do not edit it by hand.
**NOT MEASURED, by the operator's ruling of 2026-11-10:** DooPlex and ep0, and every account (Cloudflare, Hetzner, the
mail service, the registrars). What this table says about them comes from the repository's own scripts and documents.
**The eight questions:** 1 what it is, who owns it, where it lives · 2 when it ends (a real date for each box where one
was measured, else the rule) · 3 who renews it, and how long before · 4 what breaks, and for whom · 5 who is warned
before the end · 6 a box that was off past the end date: does it come back by itself? · 7 what happens if the renewal
fails · 8 how do we know (`MEASURED`, a test, `SOURCE`, a document, or nobody knows).
A ninth line, outside the marks: is the hand step written in a runbook?
**Marks:** `A` answered, with a pointer · `D` decided, not built · `K` known gap, with its row · `N` nobody asked — it
survived the searches in `EV/searches.txt` (S01…S23; controls in the same file). Every `N` names its class row:
**T1** = R-1379 an end exists and nobody is warned before it · **T2** = R-1380 another party renews and our pin does
not follow · **T3** = R-1381 no document says when it ends, or where to read the date · **T4** = R-1382 a box that was
off past the date: nobody knows if it comes back.
**Who read what.** A cell whose mark is `D`, `K` or `N`, and every `A` that `tools/marks.py` carries, holds the main
session's words: the pointer was opened and read. Every other cell holds a helper's words with the mark `A`:
`tools/resolve-helper.py` opened each of its pointers (the file and the line exist, and the name the helper gave stands
at that line). The main session read every helper answer to questions 4 to 7; it did not open each of their pointers.
Treat those cells as a map, not as proof.
**Pointers:** `AG/` = `felhom-agent/` · `CT/` = `felhom-controller/controller/` · `CAT/` = `app-catalog-felhom.eu/` ·
`HUB/` = `felhom.eu/hub/internal/` · `E/` = `felhom.eu/` · `AR/` = `felhom.eu/documentation/architecture/` ·
`RB/` = `felhom.eu/documentation/runbooks/` · `EV/` = this folder's `evidence/`. `tools/check-pointers.py` prints the
text behind each one.
**Named status** (does an architecture document name this end date?): `NAMED` and `MENTIONED` with a pointer were read
in the documents. Where the line says "a machine count", the status comes from a search of the architecture folder:
0 lines = `UNNAMED`, else `MENTIONED`.
@@ -0,0 +1,12 @@
#!/bin/bash
# fix-clone.sh CLONE LOGIN — take the credential out of the address; point git at the store helper. Prints no address.
set -u
r=$1; login=$2
u=$(git -C "$r" config --get remote.origin.url | sed -E 's#^([a-z]+://)[^/@]*@#\1#')
case "$u" in *@*) printf 'still holds an @, stop\n'; exit 1;; esac
git -C "$r" config --local remote.origin.url "$u" || exit 1
git -C "$r" config --local --unset-all credential.helper 2>/dev/null
git -C "$r" config --local credential.helper "" || exit 1
git -C "$r" config --local --add credential.helper /mnt/5_hdd/felhom.eu/git/felhom.eu/scripts/git-credential-sec.sh || exit 1
git -C "$r" config --local credential.username "$login" || exit 1
printf '%s: fixed. address now: %s\n' "$r" "$u"
@@ -0,0 +1,30 @@
taken 2026-11-11T10:46:07Z
=== clones: remote address holds a credential? ===
build/felhom-agent url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
clones-plugmeter-2026-10-13/app-catalog-felhom.eu url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
clones-plugmeter-2026-10-13/felhom-agent url=yes extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
clones-plugmeter-2026-10-13/felhom-controller url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
clones-plugmeter-2026-10-13/felhom.eu url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
drill/app-catalog-drill url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
git/app-catalog-felhom.eu url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
git/app-catalog-felhom.eu/output/.git-repo url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
git/cc-secrets url=yes extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
git/felhom-agent url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
git/felhom-controller url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
git/felhom.eu url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
git/felhom.eu-media url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
git/felhom.eu-sizing-push url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
git/felhom.eu-wt-sizing url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
wt/secreview-clone url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
=== global and system git settings ===
--global: url-with-credential=no extraHeader-lines=0 insteadOf-with-credential=no
--global: credential helper = store
--system: url-with-credential=no extraHeader-lines=0 insteadOf-with-credential=no
--system: credential helper =
clear-text credential file /home/kisfenyo/.git-credentials: EXISTS, 3 line(s), mode 600, hosts: gitea.dooplex.hu github.com
clear-text credential file /home/kisfenyo/.config/git/credentials: none
clear-text credential file /home/kisfenyo/.netrc: none
env: GIT_ASKPASS= SSH_ASKPASS= GIT_CONFIG_COUNT=unset
=== scripts and skills that write a credential into an address (file:line, the line is NOT printed) ===
@@ -0,0 +1,30 @@
taken 2026-11-11T10:10:20Z
=== clones: remote address holds a credential? ===
build/felhom-agent url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
clones-plugmeter-2026-10-13/app-catalog-felhom.eu url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
clones-plugmeter-2026-10-13/felhom-agent url=yes extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
clones-plugmeter-2026-10-13/felhom-controller url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
clones-plugmeter-2026-10-13/felhom.eu url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
drill/app-catalog-drill url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
git/app-catalog-felhom.eu url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
git/app-catalog-felhom.eu/output/.git-repo url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
git/cc-secrets url=yes extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
git/felhom-agent url=yes extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
git/felhom-controller url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
git/felhom.eu url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
git/felhom.eu-media url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
git/felhom.eu-sizing-push url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
git/felhom.eu-wt-sizing url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
wt/secreview-clone url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu
=== global and system git settings ===
--global: url-with-credential=no extraHeader-lines=0 insteadOf-with-credential=no
--global: credential helper = store
--system: url-with-credential=no extraHeader-lines=0 insteadOf-with-credential=no
--system: credential helper =
clear-text credential file /home/kisfenyo/.git-credentials: EXISTS, 3 line(s), mode 600, hosts: gitea.dooplex.hu github.com
clear-text credential file /home/kisfenyo/.config/git/credentials: none
clear-text credential file /home/kisfenyo/.netrc: none
env: GIT_ASKPASS= SSH_ASKPASS= GIT_CONFIG_COUNT=unset
=== scripts and skills that write a credential into an address (file:line, the line is NOT printed) ===
@@ -0,0 +1,31 @@
#!/bin/bash
# measure.sh — does any git address or setting hold a credential? Prints names and yes/no only, never an address.
set -u
has() { /usr/bin/grep -q -E '://[^/@[:space:]]+@' ; }
yn() { if has; then printf yes; else printf no; fi; }
printf 'taken %s\n=== clones: remote address holds a credential? ===\n' "$(date -u +%FT%TZ)"
for g in $(find /mnt/5_hdd/felhom.eu /home/kisfenyo/build /home/kisfenyo/maintenance /home/kisfenyo/.cache/felhom* /tmp/felhom* -maxdepth 5 -name .git 2>/dev/null | sort); do
r=${g%/.git}
[ -e "$g" ] || continue
u=$(git -C "$r" config --get-regexp '^remote\..*\.(url|pushurl)$' 2>/dev/null | yn)
x=$(git -C "$r" config --local --get-regexp 'extraheader' 2>/dev/null | /usr/bin/grep -c . )
i=$(git -C "$r" config --local --get-regexp 'insteadof' 2>/dev/null | yn)
sm=no; [ -f "$r/.gitmodules" ] && sm=$(yn < "$r/.gitmodules")
host=$(git -C "$r" config --get remote.origin.url 2>/dev/null | sed -E 's#^[a-z]+://([^/@]*@)?([^/:]+).*#\2#; s#^[^@]*@([^:]+):.*#\1#' | cut -c1-40)
printf '%-62s url=%-3s extraHeader=%s insteadOf=%s gitmodules=%s host=%s\n' "${r#/mnt/5_hdd/felhom.eu/}" "$u" "$x" "$i" "$sm" "$host"
done
printf '\n=== global and system git settings ===\n'
for sc in --global --system; do
printf '%s: url-with-credential=%s extraHeader-lines=%s insteadOf-with-credential=%s\n' "$sc" \
"$(git config $sc --get-regexp 'url' 2>/dev/null | yn)" \
"$(git config $sc --get-regexp -i 'extraheader' 2>/dev/null | /usr/bin/grep -c .)" \
"$(git config $sc --get-regexp -i 'insteadof' 2>/dev/null | yn)"
printf '%s: credential helper = %s\n' "$sc" "$(git config $sc --get-all credential.helper 2>/dev/null | tr '\n' ' ')"
git config $sc --name-only --get-regexp -i 'credential|askpass' 2>/dev/null | sed "s/^/$sc key: /"
done
for f in /home/kisfenyo/.git-credentials /home/kisfenyo/.config/git/credentials /home/kisfenyo/.netrc; do
if [ -f "$f" ]; then printf 'clear-text credential file %s: EXISTS, %s line(s), mode %s, hosts: %s\n' "$f" "$(wc -l < "$f")" "$(stat -c %a "$f")" "$(sed -E 's#^[a-z]+://[^@]*@##; s#^machine +([^ ]+).*#\1#; s#/.*##' "$f" | sort -u | tr '\n' ' ')"; else printf 'clear-text credential file %s: none\n' "$f"; fi
done
printf 'env: GIT_ASKPASS=%s SSH_ASKPASS=%s GIT_CONFIG_COUNT=%s\n' "${GIT_ASKPASS:+set}" "${SSH_ASKPASS:+set}" "${GIT_CONFIG_COUNT:-unset}"
printf '\n=== scripts and skills that write a credential into an address (file:line, the line is NOT printed) ===\n'
/usr/bin/grep -rn -I -E 'https?://[^/"[:space:]]*(\$\{?[A-Za-z_]+\}?|%s)[^/"[:space:]]*@' /mnt/5_hdd/felhom.eu/git/*/scripts /mnt/5_hdd/felhom.eu/git/felhom.eu/skills /mnt/5_hdd/felhom.eu/git/*/.gitea /mnt/5_hdd/felhom.eu/git/felhom-controller/controller/build.sh 2>/dev/null | /usr/bin/grep -i -E 'git|clone|gitea' | cut -d: -f1,2 | head -40