From 978666dde327aa76d9d1c9b0c80446e1af2a7cd1 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Wed, 11 Nov 2026 11:54:42 +0100 Subject: [PATCH] lists: the Clock list (70 jobs) and the End-dates list (39 things), 9 register rows, 4 decisions Read-only review of every job on a timer and of everything that ends on a date, with measurements from the three test boxes. Nothing ends in the next 14 days. Register 236 -> 245 (R-1379..R-1387), pointer lines on twelve rows, D76-D79 in STATUS.md. Part C: felhom-agent's clone address holds no token now; the other clones wait for the operator's yes. R-1035's 4-week check moved to 2026-11-18 (operator read needed). Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS --- REPORT-clock-end-dates-2026-11-11.md | 113 + STATUS.md | 7 +- .../audits/clock-list-2026-11-11/MATRIX.md | 117 + .../audits/clock-list-2026-11-11/README.md | 95 + .../audits/clock-list-2026-11-11/TABLE.md | 1256 ++++++ .../evidence/box/demo-felhom-guest-clock.txt | 468 +++ .../evidence/box/demo-felhom-guest-night.txt | 228 + .../evidence/box/demo-felhom-host-clock.txt | 665 +++ .../evidence/box/demo-felhom-host-night.txt | 89 + .../evidence/box/demo-felhom-offset.txt | 6 + .../evidence/box/demo-hp-guest-clock.txt | 502 +++ .../evidence/box/demo-hp-guest-night.txt | 196 + .../evidence/box/demo-hp-host-clock.txt | 682 +++ .../evidence/box/demo-hp-host-night.txt | 156 + .../evidence/box/demo-hp-offset.txt | 6 + .../evidence/box/tester-1-guest-clock.txt | 465 +++ .../evidence/box/tester-1-guest-night.txt | 227 + .../evidence/box/tester-1-host-clock.txt | 682 +++ .../evidence/box/tester-1-host-night.txt | 156 + .../evidence/box/tester-1-offset.txt | 6 + .../evidence/helpers/helper-clock-agent.md | 256 ++ .../helpers/helper-clock-controller.md | 269 ++ .../evidence/helpers/helper-clock-hub.md | 333 ++ .../helpers/resolved-clock-agent.json | 3124 ++++++++++++++ .../helpers/resolved-clock-controller.json | 2690 ++++++++++++ .../evidence/helpers/resolved-clock-hub.json | 3684 +++++++++++++++++ .../evidence/searches.txt | 29 + .../tools/check-pointers.py | 78 + .../tools/guest-clock.sh | 21 + .../tools/guest-night.sh | 14 + .../clock-list-2026-11-11/tools/host-clock.sh | 30 + .../clock-list-2026-11-11/tools/host-night.sh | 10 + .../clock-list-2026-11-11/tools/listconf.py | 9 + .../tools/make-matrix.py | 131 + .../clock-list-2026-11-11/tools/make-table.py | 126 + .../clock-list-2026-11-11/tools/marks.py | 290 ++ .../tools/resolve-helper.py | 155 + .../clock-list-2026-11-11/tools/run-box.sh | 26 + .../clock-list-2026-11-11/tools/run-night.sh | 7 + .../clock-list-2026-11-11/tools/searches.sh | 43 + .../clock-list-2026-11-11/tools/table-head.md | 41 + .../end-dates-list-2026-11-11/MATRIX.md | 82 + .../end-dates-list-2026-11-11/README.md | 97 + .../audits/end-dates-list-2026-11-11/TABLE.md | 724 ++++ .../evidence/box/demo-felhom-guest-dates.txt | 124 + .../evidence/box/demo-felhom-host-dates.txt | 235 ++ .../evidence/box/demo-hp-guest-dates.txt | 137 + .../evidence/box/demo-hp-host-dates.txt | 218 + .../evidence/box/golden-gate.txt | 10 + .../evidence/box/keyrings.txt | 798 ++++ .../evidence/box/pin.txt | 16 + .../evidence/box/public-reads.txt | 29 + .../evidence/box/pveupdate-excerpt.txt | 73 + .../evidence/box/tester-1-guest-dates.txt | 126 + .../evidence/box/tester-1-host-dates.txt | 214 + .../evidence/helpers/helper-dates-box.md | 265 ++ .../evidence/helpers/helper-dates-hub.md | 379 ++ .../evidence/helpers/resolved-dates-box.json | 1864 +++++++++ .../evidence/helpers/resolved-dates-hub.json | 2697 ++++++++++++ .../evidence/searches.txt | 27 + .../tools/check-pointers.py | 78 + .../tools/guest-dates.sh | 30 + .../tools/host-dates.sh | 31 + .../tools/host-pin.py | 17 + .../tools/keyrings.sh | 17 + .../tools/listconf.py | 9 + .../tools/make-matrix.py | 131 + .../tools/make-table.py | 126 + .../end-dates-list-2026-11-11/tools/marks.py | 266 ++ .../tools/public-reads.sh | 29 + .../tools/resolve-helper.py | 155 + .../tools/run-box.sh | 26 + .../tools/run-pin.sh | 9 + .../tools/searches.sh | 41 + .../tools/table-head.md | 36 + .../fix-clone.sh | 12 + .../measure-after.txt | 30 + .../measure-before.txt | 30 + .../no-token-in-address-2026-11-11/measure.sh | 31 + documentation/backlog/OPEN-ITEMS.md | 47 +- documentation/backlog/STATUS-HISTORY.md | 4 + 81 files changed, 26738 insertions(+), 20 deletions(-) create mode 100644 REPORT-clock-end-dates-2026-11-11.md create mode 100644 documentation/audits/clock-list-2026-11-11/MATRIX.md create mode 100644 documentation/audits/clock-list-2026-11-11/README.md create mode 100644 documentation/audits/clock-list-2026-11-11/TABLE.md create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-guest-clock.txt create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-guest-night.txt create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-host-clock.txt create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-host-night.txt create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-offset.txt create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-guest-clock.txt create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-guest-night.txt create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-host-clock.txt create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-host-night.txt create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-offset.txt create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-guest-clock.txt create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-guest-night.txt create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-host-clock.txt create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-host-night.txt create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-offset.txt create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/helpers/helper-clock-agent.md create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/helpers/helper-clock-controller.md create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/helpers/helper-clock-hub.md create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/helpers/resolved-clock-agent.json create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/helpers/resolved-clock-controller.json create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/helpers/resolved-clock-hub.json create mode 100644 documentation/audits/clock-list-2026-11-11/evidence/searches.txt create mode 100644 documentation/audits/clock-list-2026-11-11/tools/check-pointers.py create mode 100644 documentation/audits/clock-list-2026-11-11/tools/guest-clock.sh create mode 100644 documentation/audits/clock-list-2026-11-11/tools/guest-night.sh create mode 100644 documentation/audits/clock-list-2026-11-11/tools/host-clock.sh create mode 100644 documentation/audits/clock-list-2026-11-11/tools/host-night.sh create mode 100644 documentation/audits/clock-list-2026-11-11/tools/listconf.py create mode 100644 documentation/audits/clock-list-2026-11-11/tools/make-matrix.py create mode 100644 documentation/audits/clock-list-2026-11-11/tools/make-table.py create mode 100644 documentation/audits/clock-list-2026-11-11/tools/marks.py create mode 100644 documentation/audits/clock-list-2026-11-11/tools/resolve-helper.py create mode 100644 documentation/audits/clock-list-2026-11-11/tools/run-box.sh create mode 100644 documentation/audits/clock-list-2026-11-11/tools/run-night.sh create mode 100644 documentation/audits/clock-list-2026-11-11/tools/searches.sh create mode 100644 documentation/audits/clock-list-2026-11-11/tools/table-head.md create mode 100644 documentation/audits/end-dates-list-2026-11-11/MATRIX.md create mode 100644 documentation/audits/end-dates-list-2026-11-11/README.md create mode 100644 documentation/audits/end-dates-list-2026-11-11/TABLE.md create mode 100644 documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-felhom-guest-dates.txt create mode 100644 documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-felhom-host-dates.txt create mode 100644 documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-hp-guest-dates.txt create mode 100644 documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-hp-host-dates.txt create mode 100644 documentation/audits/end-dates-list-2026-11-11/evidence/box/golden-gate.txt create mode 100644 documentation/audits/end-dates-list-2026-11-11/evidence/box/keyrings.txt create mode 100644 documentation/audits/end-dates-list-2026-11-11/evidence/box/pin.txt create mode 100644 documentation/audits/end-dates-list-2026-11-11/evidence/box/public-reads.txt create mode 100644 documentation/audits/end-dates-list-2026-11-11/evidence/box/pveupdate-excerpt.txt create mode 100644 documentation/audits/end-dates-list-2026-11-11/evidence/box/tester-1-guest-dates.txt create mode 100644 documentation/audits/end-dates-list-2026-11-11/evidence/box/tester-1-host-dates.txt create mode 100644 documentation/audits/end-dates-list-2026-11-11/evidence/helpers/helper-dates-box.md create mode 100644 documentation/audits/end-dates-list-2026-11-11/evidence/helpers/helper-dates-hub.md create mode 100644 documentation/audits/end-dates-list-2026-11-11/evidence/helpers/resolved-dates-box.json create mode 100644 documentation/audits/end-dates-list-2026-11-11/evidence/helpers/resolved-dates-hub.json create mode 100644 documentation/audits/end-dates-list-2026-11-11/evidence/searches.txt create mode 100644 documentation/audits/end-dates-list-2026-11-11/tools/check-pointers.py create mode 100644 documentation/audits/end-dates-list-2026-11-11/tools/guest-dates.sh create mode 100644 documentation/audits/end-dates-list-2026-11-11/tools/host-dates.sh create mode 100644 documentation/audits/end-dates-list-2026-11-11/tools/host-pin.py create mode 100644 documentation/audits/end-dates-list-2026-11-11/tools/keyrings.sh create mode 100644 documentation/audits/end-dates-list-2026-11-11/tools/listconf.py create mode 100644 documentation/audits/end-dates-list-2026-11-11/tools/make-matrix.py create mode 100644 documentation/audits/end-dates-list-2026-11-11/tools/make-table.py create mode 100644 documentation/audits/end-dates-list-2026-11-11/tools/marks.py create mode 100644 documentation/audits/end-dates-list-2026-11-11/tools/public-reads.sh create mode 100644 documentation/audits/end-dates-list-2026-11-11/tools/resolve-helper.py create mode 100644 documentation/audits/end-dates-list-2026-11-11/tools/run-box.sh create mode 100644 documentation/audits/end-dates-list-2026-11-11/tools/run-pin.sh create mode 100644 documentation/audits/end-dates-list-2026-11-11/tools/searches.sh create mode 100644 documentation/audits/end-dates-list-2026-11-11/tools/table-head.md create mode 100644 documentation/audits/no-token-in-address-2026-11-11/fix-clone.sh create mode 100644 documentation/audits/no-token-in-address-2026-11-11/measure-after.txt create mode 100644 documentation/audits/no-token-in-address-2026-11-11/measure-before.txt create mode 100644 documentation/audits/no-token-in-address-2026-11-11/measure.sh diff --git a/REPORT-clock-end-dates-2026-11-11.md b/REPORT-clock-end-dates-2026-11-11.md new file mode 100644 index 00000000..8f9e09d3 --- /dev/null +++ b/REPORT-clock-end-dates-2026-11-11.md @@ -0,0 +1,113 @@ +# Report — the Clock list, the End-dates list, no token in a repository address (2026-11-11) + +Session start 2026-11-11T09:57Z. Read-only on the three test boxes. No release, no build, no deploy, no household mail. + +## 1. Claims in the prompt that turned out wrong + +- **"No architecture document lists every timer or every end date" — true.** Search S19 (Clock) and S18 (End dates) + found no such list. 69 of 70 jobs are mentioned somewhere; no document puts them in one order. +- **The plan file `claude/PLAN-architecture-gap-review-2026-11-10.md` does not exist.** The lists follow the shape of + the two lists of 2026-11-10 in `documentation/audits/`. +- **"The same happened before with another clone (R-925)"** — true: three clones held a login and a token in the + address (§3, Part C), not one. +- Not in the prompt, and it shaped the run: **the golden gate is red** (the newest golden is 8 days old, the limit is 7, + no waiver). A push that holds code is refused. This run pushed documents only. + +## 2. Parts + +| Part | State | Why | +|---|---|---| +| B — the two operator tables | done | in `documentation/audits/end-dates-list-2026-11-11/README.md` | +| B — the End-dates list | done | 39 things, 312 answers | +| A — the Clock list | done | 70 jobs, 560 answers | +| C — proof on felhom-agent | done | fetch, pull, push, hook, release dry run, positive control | +| C — the other clones, the scripts, the `CLAUDE.md` line | **not done** | no answer to the STOP question in 30 minutes; the brief says: change nothing else | +| C — the helper script in the repository | **changed** | `scripts/git-credential-sec.sh` is on DooPlex's disk, not pushed: the golden gate refuses a code push | + +## 3. The lists + +**End dates** (`documentation/audits/end-dates-list-2026-11-11/`): found 58, in the table 39 · named by no document 2, +only mentioned 28 · marks `A` 268, `D` 2, `K` 3, `N` 39 · worst `N`: the agent's pin on Proxmox's certificate; Proxmox +renews by itself and the agent does not follow (first box about 2028-07-02) · two decisions (D76, D77). +Method: it worked; the measurement found the worst gap, no helper did. + +**Clock** (`documentation/audits/clock-list-2026-11-11/`): found 77, in the table 70 · named by no document 1 · marks +`A` 457, `D` 0, `K` 9, `N` 94 · worst `N`: the guest-power watchdog gives up and tells nobody · two decisions (D78, +D79). Method: it worked for the map; the two meetings of jobs came from the boxes, not from a helper. + +**Ends in the next 90 days** — the full table is in the End-dates README. Short form: + +| Ends | What | Renewal proven? | +|---|---|---| +| 2026-11-10 (past) | the golden's 7-day age limit | no — you bake or write a waiver | +| 2026-12-23 to 2027-01-28 | eleven web certificates (the hub, Gitea, the download and ISO addresses, the website, the three household names and their Cloudflare edges) | demo-hp's box: yes (renewed 2026-10-21); the hub's: seen once; the others: not measured | +| 2027-01-17 | demo-hp's Tailscale key (date from a document) | no — yours; Tailscale does not run there today | +| not known | CC's hub token (90 days) | no — the date is on the hub's token page | + +**Nothing ends within 14 days.** The one date already past is the golden's. + +**Part C.** Clones with a credential in the address (names only): `git/felhom-agent`, `git/cc-secrets`, +`clones-plugmeter-2026-10-13/felhom-agent`. 13 other clones: no. No `extraHeader`, no `insteadOf`, no script and no +skill writes a credential into an address. **The fix:** the address holds no login; the clone's own credential helper +is `scripts/git-credential-sec.sh`, which answers from the environment of `sec run GITEA_GIT_TOKEN -- git …`. +**The proof on felhom-agent:** fetch, pull, the push of commit `91a2924` through the pre-push hook (all gates OK, +`PUSH RC=0`), a dry run of the release script's two git steps, then `git remote -v` and `git config --get-regexp` +with no secret in them. **The operator's answer: none by 10:50 UTC**, so the two other clones are as they were. +**Token to rotate:** the Gitea token of login **`kisfenyo`** (printed on 2026-11-10). Put the new one in the store +only, as `GITEA_GIT_TOKEN`. The `cc-secrets` clone holds login `admin`'s token in its address; it was not printed in +this run. The clear-text file `~/.git-credentials` (3 lines) and the global `store` helper still serve the other +clones; this run did not touch them. + +## 4. Machines, and what changed + +Read: demo-hp, Tester 1, the N100 — host and guest each. Public reads: the TLS certificates of public names. Not +read: DooPlex's workloads, ep0, the hub's database, any account. + +**No box was changed**, with one thing to name: the script that read the night's times wrote one `mktemp` file in +each guest's `/tmp` and removed it. No job was started, no setting written. + +Changed on DooPlex, file by file: `git/felhom-agent/.git/config` (address without login; `credential.helper`; +`credential.username`) · new file `git/felhom.eu/scripts/git-credential-sec.sh` (untracked) · one new entry +`GITEA_GIT_TOKEN` in the `sec` store (cc-secrets commit `8bdcf3a`; a copy of the token the clone already used, no +token made, rotated or deleted) · the files of this commit. + +Deviations: (1) one probe `git push --dry-run --no-verify` on felhom-agent; nothing was pushed, and the real push ran +the hook. (2) felhom-agent's `REPORT.md` was not overwritten for its documents-only commit. (3) Questions 4 to 7: the +main session read every helper answer by eye, and it opened the pointers by eye only for a part (below). + +## 5. Commits, CI, register + +- felhom-agent `91a2924ad0d5e8e5601c2a0c261be390826a465e` — CI run 2624: Success (read from the public Actions page). +- felhom.eu — this commit; its CI result is in the chat note, by hash. +- cc-secrets `8bdcf3a` (the store entry). +- The 4-week search check of R-1035 came due today and stopped the push gate. Only the operator can read it, so its + date moved to 2026-11-18, with the reason in the row (the gate's own rule allows this). +- **Register: before 236, after 245, opened 9 (R-1379 to R-1387), closed 0.** Dated pointer lines on R-401, R-468, + R-924, R-925, R-1186, R-1189, R-1206, R-1260, R-1293, R-1294, R-1301, R-1361; a moved-date line on R-1035. + +## 6. Documents read; cells by eye and by tool + +Architecture: `01-topology-and-trust.md`, `03-host-agent.md`, `04-control-plane-authorization.md`, +`05-hub-architecture.md`, `06-offsite-connectivity.md`, `07-backup-architecture.md`, `08-alarm-ladder.md`, +`09-update-architecture.md`, `11-os-updates.md`, `13-threat-model.md`, `_recovery-inventory-2026-07-28.md`; and +`cc-secrets/ARCHITECTURE.md` for Part C. + +| List | Cells | Main session's own words (read by eye) | Pointers opened by eye | Pointers opened by tool | +|---|---|---|---|---| +| End dates | 312 | 85 (every `D`, `K`, `N`; 41 `A`) | part | all 259, dead 0 | +| Clock | 560 | 138 (every `K`, `N`; 35 `A`) | 70 | all 896, dead 0 (one dead helper pointer dropped) | + +Document edits in place: none. Instruction-file edits: none. Text about a protected machine that looks stale and was +not edited: the hub-database copy alarm, 13 h or 26 h (in R-1387). The system poster: no fact in its facts file +changed; nothing to do. + +## 7. Observations outside the scope + +- The agent unit and its design document disagree (`Restart`, watchdog) — FILED: R-1387. +- A comment in the controller says 14 days where the code keeps 30 — FILED: R-1387. +- Stale line cites in `07-backup-architecture.md:455-457` — FILED: R-1387. +- `tailscale.md` says "both nodes"; Tailscale does not run on demo-hp and Tester 1 today — FILED: R-1387. +- Threat model and recovery inventory differ on the hub bearer's rotation — FILED: R-1387. +- The golden gate is red — FILED: pointer line on R-468; a "do" in `STATUS.md`. +- The N100's whole-guest backup starts 5 minutes later each day — FILED: R-1385. +- `MEMORY.md` is at its size limit — NOT-A-FINDING: known; the gate warns only. diff --git a/STATUS.md b/STATUS.md index cb505c56..cbe339c1 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-11-10 (evening).** Three releases are live: the agent that cleans its own update folder (you vouched it), the controller that sends one warning for one full disk (every box takes it by itself), and the hub that sends the host-disk alarm to you only. You ruled D71–D75 at 15:19 and none is built; the smallest system disk is 200 GB, and what the installer does below 200 GB (refuse, or warn) is open (R-1370). The report of the run is `REPORT-release-space-fixes-2026-11-10.md`. +**Updated 2026-11-11 (midday).** Two read-only lists are ready: every job on a timer (70) and everything that ends on a date (39); nothing ends in the next 14 days (`documentation/audits/end-dates-list-2026-11-11/README.md`). The golden passed its 7-day limit on 2026-11-10, so code pushes to `felhom.eu` are refused. Four decisions wait (D76–D79). ## What works - **The Plex attack did not reach Felhom's secrets** (R-1241). @@ -32,6 +32,10 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked | |---|---|---|---|---|---|---| +| D76 | R-1380 | Shall the agent follow Proxmox's new certificate by itself? | **Yes** (one agent change, long before 2028; no box then depends on a calendar) | a hand step in a runbook and a calendar date for each box, for ever | in July 2028 the N100's agent loses guest control and backups, with no alarm that says why; the other boxes follow | — | +| D77 | R-1379 | Shall the product warn you 14 days before an end date (a box's web certificate, a hub token)? | **Yes** (a renewal that fails is silent today) | you keep the dates in a calendar by hand | the list of 2026-11-11 is the only guard | — | +| D78 | R-1383 | Shall a timed job that gives up or is skipped tell you (four cases: the guest-power watchdog, a skipped hub daily check, a lost operator mail, a refused signed job)? | **Yes** (small changes in the agent and the hub) | the logs stay the only trace | you learn of a dead guest from the household | — | +| D79 | R-1384 | Shall the hub check each box's clock and alarm above one minute of error? | **Yes** (a box with a wrong clock can look healthy and cannot take an agent update) | R-1301 (repair the N100's time service) stays the only guard | the N100 stays 32 s behind, and nobody is told when it gets worse | — | | D66 | R-1365 | Should a security-watch source that stays down mail again every 7 days, as an open notice does? Today it mails once, then only the Monday summary lists it. | **Yes** (a real outage then cannot be forgotten; the seven pins that can never be read no longer count, so it stays quiet when all is well) | no: one line in a long Monday mail stays the only reminder | a source that stays down mails once; the Monday list still shows it | — | | D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — | | D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — | @@ -39,6 +43,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — | ## Waiting on you — do +**2026-11-11.** 1. Bake and vouch a golden, or write a waiver (R-468). If not: no code push to `felhom.eu`. 2. Read the dates of „Only you can read these dates” (`documentation/audits/end-dates-list-2026-11-11/README.md`, R-1381). If not: a domain or a token can end with no warning. 3. Rotate the Gitea token of login `kisfenyo` into the store as `GITEA_GIT_TOKEN` (R-925); it was printed on 11-10. 4. Yes or no: may CC fix the two other clone addresses (`cc-secrets`, the plug-meter clone)? Only `felhom-agent` is fixed. 5. Search Console read (R-1035, now 11-18). **Still needed? (yes/no)** — answer per line. 1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). (since 10-16) diff --git a/documentation/audits/clock-list-2026-11-11/MATRIX.md b/documentation/audits/clock-list-2026-11-11/MATRIX.md new file mode 100644 index 00000000..12533ff5 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/MATRIX.md @@ -0,0 +1,117 @@ +# The Clock list — the matrix + +Made by `tools/make-matrix.py` from `TABLE.md`. Do not edit it by hand. + +`A` answered · `D` decided, not built · `K` known gap (has a row) · `N` nobody asked. Columns are the eight +questions: 1 what runs, owner, what starts it · 2 when, how often, order · 3 how long, limit, 10x data · 4 if it does not run · 5 late, twice, meets another job · 6 clock wrong or jumps · 7 who is told, when · 8 how do we know. + +| Job | Named | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | N | +|---|---|---|---|---|---|---|---|---|---|---| +| CA01 Whole-guest backup: due-check and run · CUSTOMER | mentioned | A | A | A | A | A | K | A | K | 0 | +| CA02 OS-update night leg: guest, host, Docker, Proxmox · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | **N** | 2 | +| CA03 Kernel lane: night reboot, 20-minute boot judge, one self-revert · CUSTOMER | mentioned | A | A | A | A | A | K | A | A | 0 | +| CA04 Crash guard: early-boot unit and the hourly re-arm timer · CUSTOMER | mentioned | A | A | A | A | **N** | **N** | A | **N** | 3 | +| CA05 Scheduled restore-test evaluation · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | **N** | 2 | +| CA06 Controller supervisor: restarts a dead or silent controller · CUSTOMER | mentioned | A | A | **N** | A | A | **N** | A | A | 2 | +| CA07 Guest-power watchdog: starts a guest that must run · CUSTOMER | mentioned | A | A | A | A | A | **N** | **N** | **N** | 3 | +| CA08 Hub report and heartbeat loop · CUSTOMER | mentioned | A | A | A | A | A | A | A | A | 0 | +| CA09 Signed jobs: the validity window against the host clock · OPERATOR | mentioned | A | A | A | **N** | A | **N** | **N** | A | 3 | +| CA10 Storage watchdog and the 20-second mount re-assert · CUSTOMER | mentioned | A | A | **N** | A | A | **N** | A | **N** | 3 | +| CA11 Weekly guest disk trim · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | **N** | 2 | +| CA12 Weekly upload line test · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | A | 1 | +| CA13 PBS datastore verify loop · CUSTOMER | mentioned | A | A | A | A | **N** | A | **N** | **N** | 3 | +| CA14 Janitor: leftover scratch guests and stale backup locks · CUSTOMER | mentioned | A | A | A | A | A | A | A | A | 0 | +| CA15 Guest network watchdog: DHCP client heal · CUSTOMER | mentioned | A | A | **N** | A | A | A | A | A | 1 | +| CA16 Host networking self-heal · CUSTOMER | mentioned | A | A | A | A | A | A | A | A | 0 | +| CA17 Management-plane watchdog timer: keeps SSH reachable · OPERATOR | mentioned | A | A | A | A | A | A | A | A | 0 | +| CA18 Agent service: restart, start limit, rollback · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | **N** | 2 | +| CA19 Off-site tunnel loop and the clock-ahead re-key · CUSTOMER | mentioned | A | A | **N** | A | A | A | A | A | 1 | +| CA20 Whole-guest backup cap: rolling windows · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | A | 1 | +| CA21 Unsent OS-update report sender · OPERATOR | unnamed | A | A | A | A | A | A | A | A | 0 | +| CC01 Night dump leg: databases and volumes · CUSTOMER | mentioned | A | A | A | A | A | A | A | A | 0 | +| CC02 Second-drive copy leg · CUSTOMER | mentioned | A | A | A | A | **N** | A | A | A | 1 | +| CC03 Off-site push leg, with the clean-up of old snapshots · CUSTOMER | mentioned | A | A | A | A | K | A | A | A | 0 | +| CC04 Missed-night catch-up and the resume watch · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | A | 1 | +| CC05 Missed-backup banner (26 hours) · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | A | 1 | +| CC06 Whole-guest backup driver: stop apps, ask the agent, start apps · CUSTOMER | mentioned | A | A | A | A | A | K | A | A | 0 | +| CC07 Automatic app-update leg · CUSTOMER | mentioned | A | A | A | A | A | K | **N** | A | 1 | +| CC08 Off-site integrity check · CUSTOMER | mentioned | A | A | K | A | A | A | A | A | 0 | +| CC09 Off-site proof restore · CUSTOMER | mentioned | A | A | A | A | **N** | A | A | A | 1 | +| CC10 Hub report: every 15 minutes, on change, and the hanging wait · CUSTOMER | mentioned | A | A | A | A | A | A | A | A | 0 | +| CC11 Controller self-update: check, floor, nightly auto · CUSTOMER | mentioned | A | A | **N** | A | **N** | A | A | A | 2 | +| CC12 Catalog sync: app templates from git · CUSTOMER | mentioned | A | A | **N** | **N** | A | A | A | A | 2 | +| CC13 System health check and base-stack self-heal · CUSTOMER | mentioned | A | A | **N** | A | A | A | A | A | 1 | +| CC14 Dead-app scan · CUSTOMER | mentioned | A | A | **N** | A | A | **N** | A | A | 2 | +| CC15 Disk-fill watch · CUSTOMER | mentioned | A | A | **N** | A | A | A | A | A | 1 | +| CC16 Drive health check, hourly · CUSTOMER | mentioned | A | A | **N** | A | A | A | A | A | 1 | +| CC17 Agent channel health, every 60 seconds · CUSTOMER | mentioned | A | A | A | A | A | A | A | A | 0 | +| CC18 Held-app report to the hub · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | A | 1 | +| CC19 Abandoned off-site store sweep · CUSTOMER | mentioned | A | A | A | A | **N** | A | **N** | A | 2 | +| CC20 Restore-copy sweep and conversion-copy release · CUSTOMER | mentioned | A | A | **N** | **N** | A | A | A | A | 2 | +| CC21 Upload and CPU watch, torrent limit rule, line-test hold · CUSTOMER | mentioned | A | A | A | A | **N** | A | A | A | 1 | +| CC22 The scheduler itself: daily slots and interval jobs · CUSTOMER | mentioned | A | A | A | A | A | K | A | A | 0 | +| CH01 Box liveness: 'your server cannot be reached' · CUSTOMER | mentioned | A | A | **N** | A | A | A | A | A | 1 | +| CH02 Missed-backup deadline check (05:00) · CUSTOMER | mentioned | A | A | **N** | **N** | A | **N** | A | **N** | 4 | +| CH03 Mail cooldowns and the weekly outage mail · CUSTOMER | mentioned | A | A | A | A | A | **N** | **N** | A | 2 | +| CH04 Retry of a failed operator mail · OPERATOR | mentioned | A | A | A | **N** | A | **N** | **N** | **N** | 4 | +| CH05 Kernel mail: 'your box restarts tonight' · CUSTOMER | mentioned | A | A | A | A | A | A | A | **N** | 1 | +| CH06 OS-update approval tick and the hourly OS alarms · OPERATOR | mentioned | A | A | A | A | A | **N** | A | **N** | 2 | +| CH07 Off-site copy too old, and escrow never finished · CUSTOMER | mentioned | A | A | A | **N** | A | **N** | A | **N** | 3 | +| CH08 Restore-test proof too old · OPERATOR | mentioned | A | A | **N** | A | A | **N** | A | **N** | 3 | +| CH09 Off-site clean-up window (20 minutes, weekly) · CUSTOMER | mentioned | A | A | A | **N** | A | **N** | A | A | 2 | +| CH10 Delayed deletion of a set-aside off-site copy · CUSTOMER | mentioned | A | A | **N** | A | A | A | A | A | 1 | +| CH11 Off-site key audit (07:10) · OPERATOR | mentioned | A | A | **N** | **N** | A | A | A | **N** | 3 | +| CH12 Off-site credential stuck, and its self-heal · OPERATOR | mentioned | A | A | A | A | A | A | A | A | 0 | +| CH13 PBS whole-box credential self-heal · OPERATOR | mentioned | A | A | A | A | A | A | A | **N** | 1 | +| CH14 WireGuard peer list push to ep0 · OPERATOR | mentioned | A | A | A | **N** | A | A | A | A | 1 | +| CH15 Box clock ran ahead: the tunnel alarm · OPERATOR | mentioned | A | A | A | A | A | A | A | A | 0 | +| CH16 The daily scheduler and the hour change · OPERATOR | mentioned | A | A | A | **N** | A | **N** | A | A | 2 | +| CH17 Hub database snapshot (four slots) · OPERATOR | mentioned | A | A | A | A | A | **N** | A | A | 1 | +| CH18 Daily prune of old hub rows (04:30) · OPERATOR | mentioned | A | A | **N** | A | A | **N** | A | A | 2 | +| CH19 The other checks on the 60-second sweep · OPERATOR | mentioned | A | A | **N** | A | A | A | A | **N** | 2 | +| CH20 Reminder about a held app · CUSTOMER | mentioned | A | A | A | A | A | **N** | A | A | 1 | +| CH21 'A golden is published but not vouched' · OPERATOR | mentioned | A | A | A | A | A | A | A | A | 0 | +| CH22 Shared storage boxes: fill and reachability · OPERATOR | mentioned | A | A | A | A | A | A | A | A | 0 | +| CH23 DooPlex: push of the hub database snapshot off-site · OPERATOR | mentioned | A | A | A | A | A | K | **N** | A | 1 | +| CH24 DooPlex: Sunday restore tests · OPERATOR | mentioned | A | A | A | A | **N** | **N** | A | A | 2 | +| CH25 DooPlex: nightly off-site push and the release copy · OPERATOR | mentioned | A | A | **N** | A | A | A | A | A | 1 | +| CH26 DooPlex and ep0: PBS copy, prune, verify, clean-up · OPERATOR | mentioned | A | A | A | A | **N** | **N** | **N** | A | 3 | +| CH27 Nightly CI: the gates and the security watch · OPERATOR | mentioned | A | A | A | **N** | A | A | A | **N** | 2 | + +## Totals + +- Jobs: **70** in the table · named **0** · mentioned only **69** · named by no architecture document **1**. +- Answers: **560** · A **457** · D **0** · K **9** · N **94**. +- Jobs with no N: **18** of 70. + +| Question | A | D | K | N | +|---|---|---|---|---| +| 1 what runs, owner, what starts it | 70 | 0 | 0 | 0 | +| 2 when, how often, order | 70 | 0 | 0 | 0 | +| 3 how long, limit, 10x data | 50 | 0 | 1 | 19 | +| 4 if it does not run | 59 | 0 | 0 | 11 | +| 5 late, twice, meets another job | 60 | 0 | 1 | 9 | +| 6 clock wrong or jumps | 36 | 0 | 6 | 28 | +| 7 who is told, when | 61 | 0 | 0 | 9 | +| 8 how do we know | 51 | 0 | 1 | 18 | + +## The N answers, by class row + +| Class | Row | Count | Members (id-question) | +|---|---|---|---| +| W1 | R-1383 | 23 | CA07-7, CA07-8, CA09-4, CA09-7, CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH02-4, CH02-8, CH03-7, CH04-4, CH04-7, CH04-8, CH07-4, CH09-4, CH11-4, CH14-4, CH16-4, CH26-7, CH27-4, CH27-8 | +| W2 | R-1384 | 30 | CA02-6, CA02-8, CA04-6, CA04-8, CA05-6, CA05-8, CA06-6, CA07-6, CA09-6, CA10-6, CA11-6, CA11-8, CA12-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6, CH02-6, CH03-6, CH04-6, CH06-6, CH07-6, CH07-8, CH08-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6 | +| W3 | R-1385 | 12 | CA04-5, CA13-5, CC02-5, CC09-5, CC11-5, CC19-5, CC21-5, CH05-8, CH24-5, CH25-3, CH26-5, CH26-6 | +| W4 | R-1386 | 25 | CA06-3, CA10-3, CA10-8, CA13-8, CA15-3, CA19-3, CC11-3, CC12-3, CC13-3, CC14-3, CC15-3, CC16-3, CC20-3, CH01-3, CH02-3, CH06-8, CH08-3, CH08-8, CH10-3, CH11-3, CH11-8, CH13-8, CH18-3, CH19-3, CH19-8 | +| W5 | R-1387 | 4 | CA18-6, CA18-8, CH17-6, CH23-7 | + +## The K answers, by existing row + +| Row | Count | Members | +|---|---|---| +| R-401 | 1 | CC08-3 | +| R-1186 | 1 | CC03-5 | +| R-1189 | 2 | CA01-6, CA01-8 | +| R-1260 | 3 | CC06-6, CC07-6, CC22-6 | +| R-1293 | 1 | CH23-6 | +| R-1301 | 1 | CA03-6 | diff --git a/documentation/audits/clock-list-2026-11-11/README.md b/documentation/audits/clock-list-2026-11-11/README.md new file mode 100644 index 00000000..5fe956e0 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/README.md @@ -0,0 +1,95 @@ +# The Clock list — 2026-11-11 + +The third list of the gap review. One block for each job that a clock starts, with eight questions each. Built from +the source of four repositories and from read-only measurements on the three test boxes (host and guest). **No +document lists every timer** — the claim in the brief is true (search S19, 0 hits). DooPlex, ep0 and the hub's +database are `NOT MEASURED`, by ruling; their jobs are in the list from source and documents only. +Files: `TABLE.md` (the answers), `MATRIX.md` (the marks), `evidence/`, `tools/`. + +## Five lines + +1. **Jobs found: 77** helper blocks. 70 are in the table (22 in the guest, 21 on the host, 27 on the hub, DooPlex and + ep0). 7 are tiny, 2 are end dates (all named in the table's head). No `TEST-ONLY` job was found. +2. **Jobs no architecture document names: 1** of 70 by the search (the sender of unsent OS-update reports). The other + 69 are mentioned somewhere; no document puts them in one list or one order. +3. **Marks (560 answers):** `A` 457 · `D` 0 · `K` 9 · `N` 94. +4. **The worst `N`:** when the guest-power watchdog gives up, it writes one line in the host's journal and tells nobody + (CA07). The household's box is then down, and the only outside sign is the hub's "cannot be reached" check. + Next worst: a box with a wrong clock refuses every signed job, so it cannot take an agent update (CA09). +5. **The operator decides** the two questions at the end of this page. + +## The clock itself, per box (MEASURED 2026-11-11) + +| Box | Host clock against DooPlex | Who keeps the time | Zone: host · guest | +|---|---|---|---| +| demo-hp | +0.07 s | the host's time service, in step | Europe/Budapest · UTC | +| Tester 1 | +0.07 s | the host's time service, in step | Europe/Budapest · UTC | +| N100 | **32.6 s behind** | **nobody: the time service is not running** (R-1301) | Europe/Budapest · UTC | + +The guest has no clock of its own; it reads the host's. The controller's night slots are in Budapest time. At a start +with a wrong time the crash guard reads the clock before any time service runs (CA04). Who notices a wrong clock: the +hub alarms only when a tunnel handshake is stamped in the future (CH15). A clock that is behind: nobody. + +## One night on a box (backup time 02:30; local time; durations from the night of 2026-11-11) + +| Start | Job | demo-hp · Tester 1 · N100 | Limit | Waits for | +|---|---|---|---|---| +| 02:30 | Dump of databases and volumes (CC01) | 2 min 33 s · 40 s · 18 s | 5 min for each dump; none for the leg | nothing | +| 03:30 | Copy to the second drive (CC02) | 39 s · 15 s · 11 s | none | the clock only (+60 min) | +| 04:15 | Off-site push and clean-up (CC03) | 3 min 09 s · 59 s · 38 s | none | the clock (+105 min); skipped if a backup holds the flag | +| after it | App updates (CC07) | not seen this night | backup time + 5 h | the off-site push | +| 04:30 | Whole-guest gate opens; controller auto-update, if on (CC11) | — | — | no backup, no app update in flight | +| 05:10 | Sweep of an abandoned off-site store (CC19) | ran | 1 min | nothing | +| 05:30 | Off-site proof restore (CC09) | 5 s · 5 s · 3 s | — | the off-site flag | +| 06:00 | Off-site integrity check (CC08) | under 1 s | — | the off-site flag | +| 02:30 to 05:37 | Whole-guest backup, by the host (CA01) | 11–13 min · 3 min · under 1 min | the agent's cap | the gate; then the OS-update leg (CA02) follows it | + +**Every duration is from a small test store.** Nobody knows these numbers for a full household disk. +**Where two jobs can meet:** (1) on Tester 1 and the N100 the whole-guest backup started at about 05:30, the minute of +the proof restore — measured, and nothing orders them (CC09-5). (2) On demo-hp one whole-guest backup started at +02:30:38, with the dump leg. (3) The auto-update's default time is the minute the gate opens (CC11-5). (4) A long +second-drive copy makes the off-site push skip the night with no message (R-1186). (5) On the N100 the whole-guest +backup starts 5 minutes later each day (05:18 to 05:37 in four days). + +## The `N` answers, by class (one register row each) + +| Class | Row | What | Count | Main members | +|---|---|---|---|---| +| W1 | R-1383 | A job does not run, gives up, or loses its state at a restart, and nobody is told | 23 | guest-power give-up (CA07) · refused signed job (CA09) · hub 05:00 check and 07:10 audit skipped when the hub is down (CH02, CH11, CH16) · operator mail retry lost at a restart, household mail never retried (CH04, CH03) · peer list push to ep0 (CH14) · app-update leg (CC07) · catalog sync (CC12) · PBS verify failure (CA13) | +| W2 | R-1384 | A verdict or an "already ran" check trusts a clock that can be wrong | 30 | the hub trusts the box's time for backup age, off-site age, proof age (CH02, CH07, CH08) · signed jobs (CA09) · stamps from the future block the OS-update leg, the trim, the line test (CA02, CA11, CA12) · crash guard at early boot (CA04) · hub clock jumps (CH03, CH04, CH09, CH16) | +| W3 | R-1385 | Two jobs can meet with no lock and no order | 12 | proof restore and whole-guest backup (CC09) · auto-update and the gate (CC11) · PBS verify beside a backup (CA13) · the timers on DooPlex and ep0 (CH24, CH25, CH26) | +| W4 | R-1386 | No time limit and no measured duration | 25 | the hub's 60-second sweep: one slow check delays all, the liveness check too (CH19, CH01, CH08) · the 07:10 audit's 10 minutes at fleet size (CH11) · the interval jobs on the box (CC12 to CC16, CC20, CA06, CA10) | +| W5 | R-1387 | A document and the source disagree on a time | 4 | the agent unit: `Restart=always` and a watchdog in the design, `on-failure` and no watchdog in the unit (CA18) · 13 h or 26 h for the hub-database alarm (CH23) · old "24 h / 02:30" text (CH17) | + +Known gaps (`K`, 9 cells): R-1260 the hour-change night on a box (3) · R-1189 a host clock ahead during a backup (2) · +R-1186 the silent off-site skip · R-1293 the hub copy on the autumn hour-change day · R-1301 · R-401. + +## Not made up, or no limit, by ruling (each is an `A`) + +Only the three backup legs are made up after a missed night; the proof restore, the integrity check, the app updates +and the abandon sweep wait for the next night · a daily slot that is more than 60 minutes late is skipped, so nothing +heavy runs at noon · no OS-update leg on a night with no good backup · the hub's snapshot and prune are made up, its +other daily jobs are not · household mails about an outage: one each 7 days · days of 23 and 25 hours on the hub (D24). + +## Decisions for the operator (also in `STATUS.md`) + +1. **Shall a job that gives up or is skipped tell you?** Yes (CC's pick): one hub alarm for the four worst cases — the + guest-power watchdog gave up, a hub daily check was skipped, an operator mail was lost, a signed job was refused. + Small changes in the agent and the hub. No: the logs stay the only trace; the cost is that you learn of a dead guest + from the household. If nothing happens: the same as "No". +2. **Shall the hub check each box's clock?** Yes (CC's pick): the hub compares the time in each report with its own and + alarms above one minute of error; it then stops trusting a backup time from the future. No: R-1301 (repair the + N100's time service) stays the only guard; the cost is that a box with a wrong clock can look healthy and cannot + take an agent update. If nothing happens: the N100 stays 32 s behind and nobody is told when it gets worse. + +## Method verdict (third list) + +- **It worked for the map, and the measurement found the meetings.** Three read-only helpers gave 77 blocks. The 05:30 + meeting of two jobs and the 5-minutes-a-day drift on the N100 are in no helper answer; they came from the boxes. +- **Read by eye: 138 of 560 cells** hold the main session's words (every `K` and `N`, and 35 `A`). For 70 of them the + pointer was opened by eye; for 68 the helper's pointer stands and a tool opened it. The main session read every + helper answer to questions 4 to 7 (280 cells); it did not open each of their pointers. A tool opened all 896. +- **The weak points:** the hub's jobs and the DooPlex and ep0 timers are from source only. The controller's log held + one night. The "named" column is a machine count with wide search words; it says "mentioned", not "explained". +- **The ninth line:** 6 jobs have a hand step in a runbook; 4 have a hand step that no runbook holds (CC03, CA07, CA09, + CH02); 1 in part (CA18). The other 59 have no hand step. diff --git a/documentation/audits/clock-list-2026-11-11/TABLE.md b/documentation/audits/clock-list-2026-11-11/TABLE.md new file mode 100644 index 00000000..60a47346 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/TABLE.md @@ -0,0 +1,1256 @@ +# The Clock list — the table + +Built 2026-11-11 from the source of the four repositories and from read-only measurements on the three test boxes +(Tester 1, demo-hp, the N100; host and guest). One block for each job that a clock starts: the night steps on a box, +the interval jobs, the watchdogs, the hub's daily jobs and its 60-second sweep, and the timers on DooPlex and ep0. +`tools/make-table.py` writes this file from the helpers' answers (`evidence/helpers/`) and from `tools/marks.py`; do +not edit it by hand. + +**NOT MEASURED, by the operator's ruling:** DooPlex and ep0, and the hub's database. What this table says about the +hub's jobs and about the timers on those two machines comes from the repository's own source, scripts and documents. + +**The eight questions:** 1 what runs, who owns it, what starts it (with `CUSTOMER`, `TEST-ONLY` or `OPERATOR`) · +2 when, how often, in what order · 3 how long it takes (measured where a box shows it), its time limit, and 10 or 100 +times the data · 4 what happens if it does not run · 5 it runs late, or twice, or meets another job (the lock is +named) · 6 the clock is wrong or jumps · 7 who is told, and when · 8 how do we know (`MEASURED`, a test, `SOURCE`, a +document, or nobody knows). A ninth line, outside the marks: is the hand step written in a runbook? + +**Every measured duration comes from a small test store** (three boxes, a few apps, one night of controller logs). +No number in this table says how long a job takes on a full household disk. + +**Marks:** `A` answered, with a pointer · `D` decided, not built · `K` known gap, with its row · `N` nobody asked — it +survived the searches in `EV/searches.txt` (S01…S25; controls in the same file). Every `N` names its class row: +**W1** = R-1383 a job does not run, gives up, or loses its state at a restart, and nobody is told · **W2** = R-1384 a +verdict or an "already ran" check trusts a clock that can be wrong · **W3** = R-1385 two jobs can meet with no lock and +no order · **W4** = R-1386 no time limit and no measured duration · **W5** = R-1387 a document and the source disagree +on a time. + +**Who read what.** A cell whose mark is `K` or `N`, and every `A` that `tools/marks.py` carries, holds the main +session's words. Where its pointer was opened by eye, the pointer is written out in `tools/marks.py`; where +`tools/marks.py` shows `+`, the pointer is the helper's and `tools/resolve-helper.py` opened it (the file and the line +exist, and the name the helper gave stands at that line). Every other cell holds a helper's words with the mark `A`. +The main session read every helper answer to questions 4 to 7; it did not open each of their pointers. Treat those +cells as a map, not as proof. + +**Pointers:** `AG/` = `felhom-agent/` · `CT/` = `felhom-controller/controller/` · `CAT/` = `app-catalog-felhom.eu/` · +`HUB/` = `felhom.eu/hub/internal/` · `E/` = `felhom.eu/` · `AR/` = `felhom.eu/documentation/architecture/` · +`RB/` = `felhom.eu/documentation/runbooks/` · `EV/` = this folder's `evidence/`. `tools/check-pointers.py` prints the +text behind each one. + +**Named status** (does an architecture document name this job?): the status comes from a search of the architecture +folder: 0 lines = `UNNAMED`, else `MENTIONED`. It is a machine count, not a reading. + +## The kinds, and where each job is + +| Kind | Jobs (worst mark) | Named · mentioned · named nowhere | +|---|---|---| +| Night | CA01 (K), CA02 (N), CA03 (K), CA05 (N), CA20 (N), CC01 (A), CC02 (N), CC03 (K), CC04 (N), CC05 (N), CC06 (K), CC07 (N), CC08 (K), CC09 (N), CC19 (N), CC22 (K) | 0 · 16 · 0 | +| Interval | CA08 (A), CA09 (N), CA11 (N), CA12 (N), CA13 (N), CA14 (A), CA19 (N), CA21 (A), CC10 (A), CC11 (N), CC12 (N), CC13 (N), CC14 (N), CC15 (N), CC16 (N), CC17 (A), CC18 (N), CC20 (N), CC21 (N) | 0 · 18 · 1 | +| Watchdog | CA04 (N), CA06 (N), CA07 (N), CA10 (N), CA15 (N), CA16 (A), CA17 (A), CA18 (N) | 0 · 8 · 0 | +| Hub daily | CH02 (N), CH11 (N), CH16 (N), CH17 (N), CH18 (N) | 0 · 5 · 0 | +| Hub sweep | CH01 (N), CH03 (N), CH04 (N), CH05 (N), CH06 (N), CH07 (N), CH08 (N), CH09 (N), CH10 (N), CH12 (A), CH13 (N), CH14 (N), CH15 (A), CH19 (N), CH20 (N), CH21 (A), CH22 (A) | 0 · 17 · 0 | +| DooPlex and ep0 | CH23 (N), CH24 (N), CH25 (N), CH26 (N), CH27 (N) | 0 · 5 · 0 | + +## Not in the table + +**Tiny (named once, not tabled):** session clean-up of dashboard sign-ins on the box (a new sign-in repairs it) · old image clean-up after a controller start (log only; deletes nothing when it cannot read its keep list) · operator restart and reboot limits on the agent (one hour; an operator action) · the hub's purges of log bundles, access log and sessions (nobody is told, nothing depends on them) · the hub's version check and catalog template fetch (a WARN line and a stored error; a page shows it) · the guest swap check, once after an agent start (2-minute limit) · the 14-day hold of alarms after a deliberate power-off (a window, not a job). + +**Left out for another reason:** + +- The agent's local-API certificate (10 years) — it is an end date: the End-dates list, DB03. +- Links and tokens that end by themselves — they are end dates: the End-dates list, DH22 and the by-ruling list there. +- The 14-day countdown after a household stops the off-site copy — the End-dates list, DB04; its daily sweep is CC19 here. +- Debian's and Proxmox's own timers on a host (apt-daily, pve-daily-update, logrotate, fstrim.timer, man-db) — they are the OS's jobs, not ours; MEASURED present on the three hosts, `evidence/box/*-host-clock.txt`. +- TEST-ONLY jobs — none was found: every job in the table runs on a household's box or for the operator. + +--- + +### CA01 — Whole-guest backup: due-check and run · CUSTOMER · Night · MENTIONED + +architecture folder: 210 lines in 20 files match `/vzdump|whole.guest|due.check/` (a machine count, not a reading) · helper section: clock-agent „1. Whole-guest backup: due-check and run” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The in-guest controller asks `handleBackupDue` and starts `handleBackup`; the host has no timer for it; the agent runs the vzdump. CUSTOMER. | A | `AG/internal/localapi/server.go:1134-1217` `AG/internal/localapi/server.go:878-1015` | +| 2 | Due when the newest archive's age reaches the cadence, `age >= tier.Cadence`; default `BackupCadence` 24 h; the night window is the controller's; the agent compares ages only, no time zone. | A | `AG/internal/localapi/server.go:1204-1213` `AG/internal/config/config.go:576-580` `AR/07-backup-architecture.md:551` | +| 3 | Task wait `defaultPrimaryTierWaitTimeout` 30 min and `defaultExtraTierWaitTimeout` 12 h; outer bound is the tier wait plus 15 min; the copy grows with the guest's size. | A | `AG/internal/config/config.go:464-465` `AG/internal/localapi/server.go:971-975` | +| 4 | A tier not backed up stays due at every later ask; the controller has a 48 h safety valve; the hub judges 72 h without a whole-guest backup. | A | `AR/07-backup-architecture.md:551-552` `AR/07-backup-architecture.md:579-581` | +| 5 | Same tier in flight answers 202 with the same job; another tier answers 409; the host-wide gate is `TryAcquire`, in memory. | A | `AG/internal/localapi/server.go:913-917` `AG/internal/localapi/server.go:919-925` `AG/internal/backup/inflight.go:40-59` | +| 6 | The age is wall-clock time. A host clock that was ahead during one backup gives a negative age afterwards; the tier reads 'not due' until real time passes it. | K | `AG/internal/localapi/server.go:509` `AG/internal/localapi/server.go:1176-1182` · R-1189 | +| 7 | The hub raises `expected_backup_missed` to the operator; a flood refused by the cap raises an operator event, rationale. | A | `AR/07-backup-architecture.md:579-582` `AG/internal/localapi/backup_cap.go:12-35` | +| 8 | TEST and DOC for the due-check. MEASURED 2026-11-11: demo-hp 11 to 13 minutes, Tester 1 about 3 minutes, the N100 under 1 minute; on the N100 the start time moved 5 minutes later each day. A clock jump has no test. | K | `EV/box/demo-hp-host-night.txt` `EV/box/tester-1-host-night.txt` `EV/box/demo-felhom-host-night.txt` · R-1189 | + +**Hand step in a runbook?** no hand step + +### CA02 — OS-update night leg: guest, host, Docker, Proxmox · CUSTOMER · Night · MENTIONED + +architecture folder: 51 lines in 10 files match `/os.?update|OS update/` (a machine count, not a reading) · helper section: clock-agent „2. OS-update night leg (guest, host, Docker, Proxmox)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The agent runs it after a successful primary, non-manual whole-guest backup: `SetAfterPrimaryBackup`, trigger at; `Run`. CUSTOMER. | A | `AG/cmd/felhom-agent/main.go:1390-1398` `AG/internal/localapi/server.go:1008-1011` `AG/internal/osupdate/leg.go:538-586` | +| 2 | 90 s after the backup ends; at most once per `MinGap`, default 20 h; order guest, host, Docker, Proxmox, kernel; no clock slot of its own. | A | `AG/cmd/felhom-agent/main.go:1395` `AG/internal/osupdate/leg.go:656-667` `AG/internal/osupdate/leg.go:538-586` `AR/11-os-updates.md:378-387` | +| 3 | Wrapper limits: host command 600 s and guest command 1800 s AG/configs/felhom-os-apply:279-284; health wait default 5 min; time grows with the number of packages; measured 23 to 70 s. | A | `AG/internal/osupdate/leg.go:810-813` `AR/07-backup-architecture.md:449-450` | +| 4 | A failed or missed backup means no OS leg that night; no catch-up found; the hub shows how far behind a box is. | A | `AR/07-backup-architecture.md:447` `AR/11-os-updates.md:406-408` | +| 5 | It runs holding the heavy-op gate taken by the backup; one pass at a time by flock `tryLockPass`; a second night run inside 20 h is skipped. | A | `AR/07-backup-architecture.md:444-446` `AG/internal/osupdate/unsent.go:35-52` `AG/internal/osupdate/leg.go:663` | +| 6 | The 20-hour gap compares the clock with a time read from a file. A stamp that was written while the clock was ahead keeps the leg skipped until real time passes that stamp plus 20 hours. | N | `AG/internal/osupdate/leg.go:661-663` · class W2 · S08 | +| 7 | The result is posted to the hub by `finish`; operator gets a hub event per update and failure, the household one timeline line. | A | `AG/internal/osupdate/leg.go:922-939` `AR/11-os-updates.md:406-423` | +| 8 | TEST and DOC for the normal path. A stamp from the future: NOBODY KNOWS. | N | `AG/internal/osupdate/leg_test.go:367` `AG/internal/localapi/afterbackup_test.go:17` `AR/11-os-updates.md:366-387` · class W2 · S08 | + +**Hand step in a runbook?** no hand step + +### CA03 — Kernel lane: night reboot, 20-minute boot judge, one self-revert · CUSTOMER · Night · MENTIONED + +architecture folder: 165 lines in 14 files match `/kernel/` (a machine count, not a reading) · helper section: clock-agent „3. Kernel lane: night reboot, 20-minute boot judge, one self-revert” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Agent: `runKernel` ends the night leg; at every agent start `KernelAfterBoot` judges the boot. CUSTOMER (the box restarts at night). | A | `AG/internal/osupdate/kernel.go:119-178` `AG/cmd/felhom-agent/main.go:1018` | +| 2 | Only on trigger night and only when the hub's block says `Tonight`; the household is mailed the day before, 09:00 to 20:00 Budapest time, by the hub. | A | `AG/internal/osupdate/kernel.go:101-115` `AR/11-os-updates.md:580-585` | +| 3 | Judge wait `DefaultKernelJudgeWait` 20 min, polled every 30 s; measured healthy at 68 s and 272 s. | A | `AG/internal/osupdate/kernel.go:37` `AG/internal/osupdate/kernel.go:251` `AR/11-os-updates.md:570-572` | +| 4 | No mail accepted means no step; a step is never retried by itself; a box that never comes back raises `host_stale` at 45 min and `host_down` at 90 min. | A | `AR/11-os-updates.md:584-585` `AR/11-os-updates.md:581-583` `AR/11-os-updates.md:572-573` | +| 5 | At most one per night under the same heavy-op gate; never two steps within 20 h (wrapper refusal R22); one self-revert, never retried. | A | `AR/11-os-updates.md:560-562` `AR/11-os-updates.md:557-558` `AG/internal/osupdate/kernel.go:300-312` | +| 6 | The 20-minute judge runs on monotonic time; a clock step does not move it. Nothing in the installer sets up time keeping, and a host can start with a wrong clock: the N100's host clock is 32 s behind today. | K | `AG/internal/osupdate/leg.go:291-296` `E/scripts/felhom-host-install.sh:1` · R-1301 | +| 7 | The hub hears staged before the reboot; operator events `os_kernel_step` and `os_kernel_notice`; a frozen new kernel needs a person, accepted. | A | `AG/internal/osupdate/kernel.go:165` `AR/11-os-updates.md:585-586` `AR/11-os-updates.md:534-535` | +| 8 | TEST and DOC. The design text itself says that a real freeze was not measured. | A | `AR/11-os-updates.md:599-601` `AG/internal/osupdate/kernel_test.go:79` | + +**Hand step in a runbook?** YES — `RB/crash-guard.md` + +### CA04 — Crash guard: early-boot unit and the hourly re-arm timer · CUSTOMER · Watchdog · MENTIONED + +architecture folder: 23 lines in 7 files match `/crash.guard|crash guard/` (a machine count, not a reading) · helper section: clock-agent „4. Crash guard: early-boot unit plus hourly re-arm timer” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Root systemd units installed by the host installer: boot unit and timer. CUSTOMER (decides if a crashed box restarts). | A | `E/scripts/felhom-host-install.sh:2944-2963` `AG/configs/felhom-crash-guard.service:15-16` `AG/configs/felhom-crash-guard-check.timer:5-6` | +| 2 | Timer as written: `OnBootSec=15min`, `OnUnitActiveSec=1h`; no OnCalendar, no RandomizedDelaySec, no Persistent; boot unit runs before sysinit. | A | `AG/configs/felhom-crash-guard-check.timer:5-6` `AG/configs/felhom-crash-guard.service:8` | +| 3 | Oneshot with no time limit written; history is cut to 7 days AG/configs/felhom-crash-guard:148, so nothing grows. | A | `AG/configs/felhom-crash-guard-check.service:6-7` | +| 4 | A missed hourly check only delays the re-arm; a crash before the boot unit runs leaves the box off, the safe side; the System page shows the guard. | A | `AR/11-os-updates.md:484-485` `AR/11-os-updates.md:487-488` | +| 5 | The second unclean boot inside 60 minutes trips it; it arms again after 24 hours. The boot unit and the hourly check share one state file and take no lock. | N | `AG/configs/crash-guard.conf:3-8` · class W3 · S14 | +| 6 | The script reads the wall clock at early boot, before any time service runs. A clock that is wrong at boot can move unclean boots into or out of the 60-minute window. | N | class W2 · S22 | +| 7 | Each unclean boot is an operator mail `host_crash_restart` and a household line; the trip is alarm `host_crash_guard_tripped`; the trip mail came 15 min after the boot. | A | `AR/11-os-updates.md:487-488` `AR/11-os-updates.md:489-490` | +| 8 | MEASURED 2026-11-11: the hourly timer is active on the three hosts · TEST and DOC for the windows. A wrong clock at boot: NOBODY KNOWS. | N | `EV/box/demo-hp-host-clock.txt` `EV/box/tester-1-host-clock.txt` `EV/box/demo-felhom-host-clock.txt` `AG/configs/test_felhom_crash_guard.py:75` `AG/configs/test_felhom_crash_guard.py:102` `AR/11-os-updates.md:473-490` · class W2 · S22 | + +**Hand step in a runbook?** YES — `RB/crash-guard.md` + +### CA05 — Scheduled restore-test evaluation · CUSTOMER · Night · MENTIONED + +architecture folder: 121 lines in 19 files match `/restore.test/` (a machine count, not a reading) · helper section: clock-agent „5. Scheduled restore-test evaluation” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Agent goroutine `Run`, wired at; restores an archive into a scratch guest on the host. CUSTOMER (proof the backup opens). | A | `AG/internal/backup/schedule.go:148-177` `AG/cmd/felhom-agent/main.go:1968-2018` | +| 2 | First look `DefaultFirstEval` 30 min after agent start, then every `defaultRestoreTestEvalInterval` 6 h; an archive counts after `defaultRestoreTestSettle` 24 h; not windowed, any hour. | A | `AG/internal/backup/schedule.go:76` `AG/internal/config/config.go:636` `AG/internal/config/config.go:641` `AR/11-os-updates.md:385-386` | +| 3 | Restore task wait: `RestoreTestPBSRestoreTimeout` 120 min for the PBS tier, else the 10 min default; boot wait `DefaultBootTimeout` 2 min; grows with guest size. | A | `AG/internal/config/config.go:584-588` `AG/cmd/felhom-agent/main.go:1917-1919` `AG/internal/reconcile/restoretest.go:28` | +| 4 | The ticker restarts with the agent, so a missed look comes 30 min after the next start; the rule is per archive, so a tier stays due until proven; hub proof-age window 7 to 12 days. | A | `AR/07-backup-architecture.md:583-584` `AR/07-backup-architecture.md:179-180` `AR/07-backup-architecture.md:215-217` | +| 5 | Takes the heavy-op gate with `TryAcquire("restore-test")`; busy means deferred to the next 6 h look; proof credit only on a pass. | A | `AG/internal/backup/schedule.go:228` `AG/internal/backup/schedule.go:230` `AG/internal/backup/schedule.go:248-255` | +| 6 | The 'settled' cut-off uses the wall clock. A clock that is set back makes no archive look settled, so no test starts. The ticker itself is monotonic. | N | `AG/internal/backup/schedule.go:119` `AG/internal/backup/restoretest_due.go:96-101` · class W2 · S08 | +| 7 | A failed test is an ERROR log and a record the hub turns into `restore_test_failed` for the operator, as the janitor comment says. | A | `AG/internal/backup/schedule.go:262` `AG/cmd/felhom-agent/janitor.go:42-44` | +| 8 | TEST and DOC. MEASURED 2026-11-11: restore tests took 1 to 14 minutes on the three boxes. A clock set back: NOBODY KNOWS. | N | `EV/box/demo-hp-host-night.txt` · class W2 · S08 | + +**Hand step in a runbook?** no hand step + +### CA06 — Controller supervisor: restarts a dead or silent controller · CUSTOMER · Watchdog · MENTIONED + +architecture folder: 12 lines in 5 files match `/supervisor/` (a machine count, not a reading) · helper section: clock-agent „6. Controller supervisor (restarts a dead or silent controller)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Agent goroutine `WatchControllers`, started at. CUSTOMER (the household's dashboard comes back). | A | `AG/internal/localapi/controllersupervisor.go:143-163` `AG/cmd/felhom-agent/main.go:1714` | +| 2 | Every `controllerSupervisorInterval` 30 s; restart on the 2nd not-running sweep; a running but unhealthy one is alarmed at 6 sweeps and restarted at 10. | A | `AG/internal/localapi/controllersupervisor.go:54` `AG/internal/localapi/controllersupervisor.go:56` `AG/internal/localapi/controllersupervisor.go:92-100` | +| 3 | No time limit on one sweep in the source, and no duration was measured. Nothing grows with data. | N | `AG/internal/localapi/controllersupervisor.go:1-200` `AG/internal/localapi/controllersupervisor.go:64` · class W4 · S15 | +| 4 | If the agent is down nothing restarts the controller: measured 33 min of 502 before this existed; the first sweep is 30 s after agent start (ticker only). | A | `AG/internal/localapi/controllersupervisor.go:20-25` `AG/internal/localapi/controllersupervisor.go:153-161` | +| 5 | Brake: `controllerCrashloopMax` 3 in `controllerCrashloopWindow` 15 min, then `controllerCrashloopPause` 30 min; not during a swap, a parked or locked guest, or a vzdump. | A | `AG/internal/localapi/controllersupervisor.go:59-61` `AG/internal/localapi/controllersupervisor.go:34-42` | +| 6 | The windows (3 restarts in 15 minutes, the 24-hour counter) use the wall clock in UTC. Nobody tested a clock jump. | N | `AG/internal/localapi/controllersupervisor.go:550-552` `AG/internal/localapi/controllersupervisor.go:308` · class W2 · S08 | +| 7 | Hub mints `controller_restarted_by_agent` (info) and `controller_crashloop` (error, operator only) from the host report; a warning at the 5th restart in 24 h. | A | `AG/internal/localapi/controllersupervisor.go:44-49` `AG/internal/localapi/controllersupervisor.go:82-83` | +| 8 | TEST `TestControllerSupervisor_KilledControllerIsRestarted`, `TestSupervisor_UnresponsiveIsAlarmedThenRestarted`; DOC. | A | `AG/internal/localapi/controllersupervisor_test.go:101` `AG/internal/localapi/supervisor_unresponsive_test.go:32` `AR/03-host-agent.md:186-192` | + +**Hand step in a runbook?** no hand step + +### CA07 — Guest-power watchdog: starts a guest that must run · CUSTOMER · Watchdog · MENTIONED + +architecture folder: 8 lines in 4 files match `/guest.power|guest power/` (a machine count, not a reading) · helper section: clock-agent „7. Guest-power watchdog (starts a guest that should be running)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Agent goroutine `WatchGuestPower`, started at. CUSTOMER (the whole box's apps are off while the guest is stopped). | A | `AG/internal/localapi/guestpower.go:89-105` `AG/cmd/felhom-agent/main.go:1710` | +| 2 | Every `guestPowerInterval` 60 s; acts only on a stopped, unlocked guest with onboot set; first sweep one interval after start (ticker only). | A | `AG/internal/localapi/guestpower.go:52` `AG/internal/localapi/guestpower.go:167-180` `AG/internal/localapi/guestpower.go:95-102` | +| 3 | At most `guestPowerMaxAttempts` 3 starts with waits of 1, 2 and 4 min, `guestPowerBackoff`; about 7 min in all. | A | `AG/internal/localapi/guestpower.go:56` `AG/internal/localapi/guestpower.go:75` `AG/internal/localapi/guestpower.go:71-74` | +| 4 | Before it existed a guest stayed down 9 min 47 s until a person started it; with the agent down nobody starts it (the platform's onboot acts only at host boot). | A | `AG/internal/localapi/guestpower.go:14-18` `AG/internal/localapi/guestpower.go:42-43` | +| 5 | Never starts under a running vzdump and fails safe on doubt; leaves a locked guest to the stale-lock path; skips a guest the operator is rebooting. | A | `AG/internal/localapi/guestpower.go:182-194` `AG/internal/localapi/guestpower.go:176-180` `AG/internal/localapi/guestpower.go:157-159` | +| 6 | The next try time is kept in memory from the wall clock. A clock set back delays the next try; an agent restart tries again at once. | N | `AG/internal/localapi/guestpower.go:234` `AG/internal/localapi/guestpower.go:253` · class W2 · S08 | +| 7 | When the watchdog gives up, it writes one ERROR line in the host's journal. No event goes to the hub. The household's box is down and the outside sign is only the hub's liveness check. | N | `AG/internal/localapi/guestpower.go:200-206` · class W1 · S04 | +| 8 | TEST for the start and the back-off. Who hears the give-up: NOBODY KNOWS. | N | `AG/internal/localapi/guestpower_test.go:162` `AG/internal/localapi/guestpower_test.go:119` · class W1 · S04 | + +**Hand step in a runbook?** NO — no runbook says what to do after the watchdog gave up + +### CA08 — Hub report and heartbeat loop · CUSTOMER · Interval · MENTIONED + +architecture folder: 23 lines in 10 files match `/heartbeat|poll interval|report loop/` (a machine count, not a reading) · helper section: clock-agent „8. Hub report and heartbeat loop (with reconcile, fast tick and poke)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Agent goroutine `Run`, built at; it carries every status the hub alarms on. OPERATOR and CUSTOMER (alarms depend on it). Merged here: the reconcile engine, fast tick and poke. | A | `AG/internal/hub/loop.go:107-138` `AG/cmd/felhom-agent/main.go:958` | +| 2 | `PollSeconds` default 900; first report at once, then a ticker; the hub may change it, clamped by `MinPollSeconds` 60 and `MaxPollSeconds` 3600; fast tick `DefaultInterval` 30 s. | A | `AG/internal/config/config.go:797` `AG/internal/hub/loop.go:107-138` `AG/internal/hub/loop.go:11-12` `AG/internal/fasttick/fasttick.go:24` | +| 3 | Per-request limit `TimeoutSeconds` default 30; the system facts in it are read at most every 10 min; the report grows with guests and drives. | A | `AG/internal/config/config.go:798` `AG/cmd/felhom-agent/main.go:4149` | +| 4 | A failed cycle is logged and the interval kept; no backlog, the next report replaces it; the hub raises `host_stale` at 45 min and `host_down` at 90 min. | A | `AG/internal/hub/loop.go:162-164` `AR/11-os-updates.md:572-573` | +| 5 | Extra reports come by the trigger channel; a poke is debounced by `DebounceWindow` 2 s; the reconcile engine runs at once and then on the start-time interval,. | A | `AG/internal/hub/loop.go:125-135` `AG/internal/poke/poke.go:40` `AG/internal/reconcile/engine.go:332-344` `AG/cmd/felhom-agent/main.go:1525` | +| 6 | The ticker is monotonic. The report loop follows the hub's interval override, clamped to a range. The reconcile engine keeps the interval that it read at the start. | A | `AG/internal/hub/loop.go:188-196` `AG/cmd/felhom-agent/main.go:969` | +| 7 | The operator, by the hub's staleness alarms; the household hears cannot-be-reached at most once per 7 days. | A | `AR/11-os-updates.md:572-573` `AR/07-backup-architecture.md:582-583` | +| 8 | TEST `TestLoop_RunImmediateAndResilientAfterError`, `TestClampInterval`; DOC. | A | `AG/internal/hub/loop_test.go:140` `AG/internal/hub/loop_test.go:81` `AR/03-host-agent.md:276-279` | + +**Hand step in a runbook?** no hand step + +### CA09 — Signed jobs: the validity window against the host clock · OPERATOR · Interval · MENTIONED + +architecture folder: 155 lines in 27 files match `/signed job|signed op|not.before|valid/` (a machine count, not a reading) · helper section: clock-agent „9. Signed jobs: validity window against the host clock (agent updates ride it)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Agent `OnEnvelope` runs pending signed jobs when a heartbeat says there are some; an agent update arrives only this way. OPERATOR (and CUSTOMER: fixes reach the box this way). | A | `AG/internal/signedjobs/runner.go:93-102` | +| 2 | Checked at every hub cycle (item 8); a job is valid from issue minus `DefaultClockSkew` 2 min to its expiry, and a window over `MaxWindow` 24 h is refused. | A | `AG/internal/authz/verifier.go:21` `AG/internal/authz/verifier.go:26` | +| 3 | The self-update download has a 5 min client limit; the new binary must run `DwellSeconds` 60 s before it commits. | A | `AG/internal/selfupdate/executor.go:73` `AG/internal/config/config.go:97-99` | +| 4 | A job that the agent refuses as expired is not run. The agent's log line is the only trace on the box; no path that tells the hub or the operator 'refused: expired' was found. | N | `AG/internal/authz/verifier.go:161-162` `AG/internal/selfupdate/commit.go:122-125` · class W1 · S09 | +| 5 | One run at a time, `RunOnce` single-flight; a replay is refused by the nonce store. | A | `AG/internal/signedjobs/runner.go:107-115` `AG/internal/authz/verifier.go:166` | +| 6 | The window is checked against the host's wall clock. A host clock that is behind refuses every job as 'not yet valid'; one that is ahead refuses them as 'expired'. So a box with a wrong clock cannot take an agent update. | N | `AG/internal/authz/verifier.go:153-162` · class W2 · S09 | +| 7 | The runner logs a finished or failed job on the host. For a job refused on its dates, no message to the hub was found. | N | `AG/internal/signedjobs/runner.go:176-190` · class W1 · S09 | +| 8 | TEST `TestVerify_RejectsPerStage`, `TestRunner_ExpiredRejected`, `TestManager_ShutdownBeforeDwellLeavesPending`. | A | `AG/internal/authz/verify_test.go:82` `AG/internal/signedjobs/runner_test.go:168` `AG/internal/selfupdate/commit_test.go:102` | + +**Hand step in a runbook?** NO — no runbook names a wrong host clock as a cause of refused signed jobs + +### CA10 — Storage watchdog and the 20-second mount re-assert · CUSTOMER · Watchdog · MENTIONED + +architecture folder: 2 lines in 2 files match `/storage watchdog|re.?assert|ReassertGuestBinds/` (a machine count, not a reading) · helper section: clock-agent „10. Storage watchdog and the 20-second mount and bind re-assert” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Agent: the watchdog `Run` wired at, and a 20 s ticker that re-asserts mounts and guest binds. CUSTOMER (a dropped USB drive). | A | `AG/internal/storage/watchdog.go:196-215` `AG/cmd/felhom-agent/main.go:1142` `AG/cmd/felhom-agent/main.go:1613-1631` | +| 2 | `DefaultWatchdogInterval` 8 s and `DefaultWatchdogDebounce` 30 s; setting `WatchdogIntervalSeconds`, 0 means the default; the re-assert is 20 s, below the controller's 30 s tick. | A | `AG/internal/storage/watchdog.go:15-16` `AG/internal/config/config.go:770-778` `AG/cmd/felhom-agent/main.go:1612-1614` | +| 3 | No time limit on one tick in the source, and no duration was measured. The work grows with the number of drives. | N | class W4 · S15 | +| 4 | A missed tick is made up by the next one; with the agent down a dropped drive is seen only at the next 15 min report, the reason the watchdog exists. | A | `AR/03-host-agent.md:461` | +| 5 | A flapping drive backs off: `backoffWindow`, alert after 4 failed cycles; the retry test is `now.Sub(last)`. | A | `AG/internal/storage/watchdog.go:183-192` `AG/internal/storage/watchdog.go:77-79` `AG/internal/storage/watchdog.go:290` | +| 6 | The back-off uses the wall clock in UTC. A clock set back stretches the back-off; a clock set forward ends it early. The tickers are monotonic. | N | `AG/internal/storage/watchdog.go:160` · class W2 · S08 | +| 7 | A drop requests an immediate hub report through the trigger channel; the hub then decides who is mailed (hub side, not read). | A | `AG/internal/hub/loop.go:125-135` | +| 8 | TEST for the reconcile and the back-off. The 20-second ticker has no test: NOBODY KNOWS. | N | `AG/internal/storage/watchdog_test.go:112` `AG/internal/storage/watchdog_test.go:460` `AG/cmd/felhom-agent/main.go:1613` · class W4 · S15 | + +**Hand step in a runbook?** no hand step + +### CA11 — Weekly guest disk trim · CUSTOMER · Interval · MENTIONED + +architecture folder: 9 lines in 5 files match `/fstrim|trim/` (a machine count, not a reading) · helper section: clock-agent „11. Weekly guest disk trim” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Agent goroutine `Run`, started at; runs `pct fstrim` per owned running guest. CUSTOMER (a full thin pool turns every guest read-only). | A | `AG/internal/fstrim/fstrim.go:186-199` `AG/cmd/felhom-agent/main.go:1741` `AG/internal/fstrim/fstrim.go:3-5` | +| 2 | Due from Wednesday 10:00 host-local, starts only 10:00 to 20:59; looks 5 min after start then hourly, `FirstTickDelay` and `TickInterval`; zone is `time.Local`. | A | `AG/internal/fstrim/fstrim.go:42-46` `AG/internal/fstrim/fstrim.go:48-50` `AG/internal/fstrim/fstrim.go:110` | +| 3 | `PerGuestTimeout` 30 min; measured 24.4 s for 84 GiB; grows with freed blocks. | A | `AG/internal/fstrim/fstrim.go:52` `AG/internal/fstrim/fstrim.go:6` | +| 4 | A box off on Wednesday catches up at its next eligible hour; a failure is retried at most `MaxAttemptsPerWeek` 3 times; the result rides the host report. | A | `AG/internal/fstrim/fstrim.go:10-11` `AG/internal/fstrim/fstrim.go:46` `AG/internal/fstrim/fstrim.go:326-330` | +| 5 | Holds the heavy-op gate for the whole pass; busy means deferred to the next hour; the persisted record stops a second trim after a restart. | A | `AG/internal/fstrim/fstrim.go:247-251` `AG/internal/fstrim/fstrim.go:17` | +| 6 | The 'done this week' check uses host-local wall time. An attempt that was stamped while the clock was ahead keeps the trim 'not due' until real time passes it. | N | `AG/internal/fstrim/fstrim.go:179-182` · class W2 · S08 | +| 7 | The operator, on the hub's System page (one cell). A failed trim is a WARN line on the host. Nobody is mailed. | A | `HUB/web/system.go:45` `HUB/web/system.go:206` `AG/internal/fstrim/fstrim.go:318-319` | +| 8 | TEST for the week anchor. The hour change and a clock jump: NOBODY KNOWS. | N | `AG/internal/fstrim/fstrim_test.go:161` `AG/internal/fstrim/fstrim_test.go:175` `AG/internal/fstrim/fstrim_test.go:104` · class W2 · S20 | + +**Hand step in a runbook?** no hand step + +### CA12 — Weekly upload line test · CUSTOMER · Interval · MENTIONED + +architecture folder: 28 lines in 10 files match `/line.?test|linetest/` (a machine count, not a reading) · helper section: clock-agent „12. Weekly upload line test” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Agent goroutine `Run`, started at; a 10 s iperf3 upload over the tunnel. CUSTOMER (sets the torrent app's upload limit). | A | `AG/internal/linetest/linetest.go:256-268` `AG/cmd/felhom-agent/main.go:1762` `AG/internal/linetest/linetest.go:3-4` | +| 2 | Due from Tuesday 10:00 host-local, starts 10:00 to 15:59; each host from its own minute, `SlotMinute`; ticks every minute after a 5 min delay; zone `time.Local`. | A | `AG/internal/linetest/linetest.go:47-49` `AG/internal/linetest/linetest.go:148-152` `AG/internal/linetest/linetest.go:60-61` `AG/internal/linetest/linetest.go:131` | +| 3 | Three runs of 10 s, each bounded by `RunTimeout` 25 s, 30 s apart; nothing grows with data. | A | `AG/internal/linetest/linetest.go:53-57` | +| 4 | A box that was off catches up on a later day inside the window; the hub page turns amber when the last success is over 15 days old. | A | `AG/internal/linetest/linetest.go:10-11` `AR/11-os-updates.md:409-411` | +| 5 | Needs 10 quiet minutes and the heavy-op gate; server busy is retried at the next tick, at most `MaxTriesPerDay` 6; no tunnel means no test. | A | `AG/internal/linetest/linetest.go:316-325` `AG/internal/linetest/linetest.go:58` `AG/internal/linetest/linetest.go:297-300` | +| 6 | The check uses host-local wall time. A success that was stamped from the future blocks the tests until that time. | N | `AG/internal/linetest/linetest.go:211-213` · class W2 · S08 | +| 7 | The operator, on the hub's System page; the cell is judged on the last success, not on the attempt. Nobody is mailed. | A | `AR/11-os-updates.md:409-411` | +| 8 | TEST `TestDueWeeklyFromTuesday`, `TestNeverTestsInTheNightOrBeforeTuesday`, `TestSlotDelaysTheFirstStartOnTheDueDayOnly`. | A | `AG/internal/linetest/linetest_test.go:206` `AG/internal/linetest/linetest_test.go:363` `AG/internal/linetest/linetest_test.go:189` | + +**Hand step in a runbook?** no hand step + +### CA13 — PBS datastore verify loop · CUSTOMER · Interval · MENTIONED + +architecture folder: 64 lines in 17 files match `/verify loop|PBS verify|verify/` (a machine count, not a reading) · helper section: clock-agent „13. PBS datastore verify loop” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Agent goroutine `Run`, cadence wired at; asks the PBS server to verify the box's snapshots. CUSTOMER (is the off-site copy readable). | A | `AG/internal/pbs/verify.go:64-83` `AG/cmd/felhom-agent/main.go:1188` | +| 2 | `DefaultVerifyCadence` 6 h; setting through `PBSVerifyCadence`; first pass at once at agent start, then the ticker; no time zone. | A | `AG/internal/pbs/verify.go:13` `AG/internal/config/config.go:710-719` `AG/internal/pbs/verify.go:64-83` | +| 3 | One verify is polled every 2 s for at most 30 min; grows with the datastore's size. | A | `AG/internal/pbs/verify.go:107` | +| 4 | No catch-up state: every agent start verifies again at once; a negative cadence switches it off, test `TestVerifyLoop_DisabledByNegativeCadence`. | A | `AG/internal/pbs/verify.go:64-83` `AG/internal/pbs/verify_test.go:87` | +| 5 | The verify is not behind the heavy-work gate; a comment in the source says so. It can run beside a whole-guest backup. | N | `AG/internal/lanresolver/loop.go:17` `AG/internal/pbs/verify.go:1` · class W3 · S13 | +| 6 | Ticker only, monotonic; no wall-clock decision found in. | A | `AG/internal/pbs/verify.go:64-83` | +| 7 | A snapshot that fails the verify is an ERROR line on the host and a field in the report. No hub event or mail for it was found. | N | `AG/internal/pbs/verify.go:127` · class W1 · S05 | +| 8 | TEST for the record of a failed verify. A verify that takes longer than its 30-minute cadence: NOBODY KNOWS. | N | `AG/internal/pbs/verify_test.go:52` · class W4 · S15 | + +**Hand step in a runbook?** no hand step + +### CA14 — Janitor: leftover scratch guests and stale backup locks · CUSTOMER · Interval · MENTIONED + +architecture folder: 2 lines in 2 files match `/janitor|stale lock/` (a machine count, not a reading) · helper section: clock-agent „14. Janitor: leftover scratch guests and stale backup locks” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Agent goroutine `runJanitor`, started at. CUSTOMER (a stale lock blocked whole-box backups for five hours). | A | `AG/cmd/felhom-agent/janitor.go:69-81` `AG/cmd/felhom-agent/main.go:1725` `AG/cmd/felhom-agent/janitor.go:14-17` | +| 2 | Every `janitorInterval` 10 min; ticker only, so the first pass is 10 min after agent start; no time zone. | A | `AG/cmd/felhom-agent/janitor.go:18` `AG/cmd/felhom-agent/janitor.go:71-78` | +| 3 | No time limit on a pass found in; work grows with the number of leftover scratch guests. | A | `AG/cmd/felhom-agent/janitor.go:34-66` | +| 4 | A missed pass is retried 10 min later; a scratch guest that cannot be removed after the retry limit is reported as a failed restore-test record. | A | `AG/cmd/felhom-agent/janitor.go:41-54` | +| 5 | The stale-lock sweep runs only holding the heavy-op gate, else it is skipped this pass; rule stated at. | A | `AG/cmd/felhom-agent/janitor.go:57-64` `AG/cmd/felhom-agent/janitor.go:30-33` | +| 6 | Ticker only, monotonic; a clock step must not kill a live task: `TestVzdumpInFlight_ClockStepCannotKillALiveTask`. | A | `AG/internal/localapi/stalelock_inflight_test.go:97` | +| 7 | The operator, through the hub's `restore_test_failed`, once per distinct archive. | A | `AG/cmd/felhom-agent/janitor.go:42-44` | +| 8 | TEST `TestJanitor_StaleLockSweepWaitsForTheHeavyGate`, `TestJanitor_GaveUpIsReportedAsAFailure`. | A | `AG/cmd/felhom-agent/janitor_test.go:40` `AG/cmd/felhom-agent/janitor_test.go:22` | + +**Hand step in a runbook?** no hand step + +### CA15 — Guest network watchdog: DHCP client heal · CUSTOMER · Watchdog · MENTIONED + +architecture folder: 17 lines in 5 files match `/guest.?net|DHCP/` (a machine count, not a reading) · helper section: clock-agent „15. Guest network watchdog (DHCP client heal)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Agent goroutine `Watch`, built at. CUSTOMER (a guest without network). | A | `AG/internal/guestnet/watchdog.go:105-121` `AG/cmd/felhom-agent/main.go:1686` | +| 2 | `DefaultInterval` 60 s; immediate first sweep; settings interval 60, `MinHealIntervalSeconds` 600, `MaxHealsPerHour` 3. | A | `AG/internal/guestnet/watchdog.go:23` `AG/internal/guestnet/watchdog.go:111` `AG/internal/config/config.go:176-196` | +| 3 | No time limit on one probe in the source, and no duration was measured. | N | `AG/internal/guestnet/watchdog.go:1-200` · class W4 · S15 | +| 4 | A missed sweep is made up a minute later; beyond 3 heals an hour it only reports. | A | `AG/internal/guestnet/watchdog.go:26-28` | +| 5 | Two consecutive bad probes before a heal, `requiredBadProbes`; 10 min between heals, `DefaultMinHealInterval`; boot-race guard `DefaultSettle` 3 min. | A | `AG/internal/guestnet/watchdog.go:32-33` `AG/internal/guestnet/watchdog.go:25` `AG/internal/guestnet/watchdog.go:29-31` | +| 6 | `now` is plain `time.Now`, so in-process gaps are monotonic; heal history is in memory. | A | `AG/internal/guestnet/watchdog.go:80` `AG/internal/guestnet/watchdog.go:56-62` | +| 7 | The operator, on the hub's Hosts page: the report carries the heal count for each guest. Nobody is mailed. | A | `HUB/web/hosts.go:298-397` `AG/internal/guestnet/watchdog.go:166-170` | +| 8 | TEST `TestWatchdog_DampingCeilings`, `TestWatchdog_MinimumIntervalBetweenHeals`, `TestWatchdog_BootRacesObserveOnly`. | A | `AG/internal/guestnet/watchdog_test.go:360` `AG/internal/guestnet/watchdog_test.go:384` `AG/internal/guestnet/watchdog_test.go:416` | + +**Hand step in a runbook?** no hand step + +### CA16 — Host networking self-heal · CUSTOMER · Watchdog · MENTIONED + +architecture folder: 39 lines in 12 files match `/self.?heal/` (a machine count, not a reading) · helper section: clock-agent „16. Host networking self-heal” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Agent goroutine `Watch`, interval set at; starts `networking.service` on an appliance only. CUSTOMER. | A | `AG/internal/selfheal/selfheal.go:67-82` `AG/cmd/felhom-agent/main.go:1648` | +| 2 | One pass at agent start, then every 60 s, no setting found for the interval (hard-coded). | A | `AG/internal/selfheal/selfheal.go:67-82` `AG/cmd/felhom-agent/main.go:1648` | +| 3 | A remedy is 3 start attempts with waits of 10, 30 and 60 s, `networkingBackoffs`. | A | `AG/internal/selfheal/selfheal.go:114-116` | +| 4 | A missed pass is made up a minute later; on a BYO host the remedy is skipped and only a WARN is written. | A | `AG/internal/selfheal/selfheal.go:54-58` | +| 5 | No lock seen; the pass is sequential inside one goroutine. | A | `AG/internal/selfheal/selfheal.go:46-64` | +| 6 | Ticker only, monotonic; no wall-clock decision in. | A | `AG/internal/selfheal/selfheal.go:67-82` | +| 7 | A give-up is an ERROR journal line; with networking down the hub cannot be told, so the hub's stale alarm is the only signal. | A | `AG/internal/selfheal/selfheal.go:61` `AR/11-os-updates.md:572-573` | +| 8 | TEST `TestNetworkingHeal_GivesUpAfterAllAttempts`, `TestManager_ByoNeverRemediates`. | A | `AG/internal/selfheal/selfheal_test.go:109` `AG/internal/selfheal/selfheal_test.go:32` | + +**Hand step in a runbook?** no hand step + +### CA17 — Management-plane watchdog timer: keeps SSH reachable · OPERATOR · Watchdog · MENTIONED + +architecture folder: 5 lines in 4 files match `/mgmt.watchdog|management.plane/` (a machine count, not a reading) · helper section: clock-agent „17. Management-plane watchdog timer (keeps SSH reachable)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Root systemd timer and oneshot installed by the host installer; script; works without the agent. OPERATOR. | A | `E/scripts/felhom-host-install.sh:3067-3076` `AG/configs/felhom-mgmt-watchdog.sh:26-68` `AG/configs/felhom-mgmt-watchdog.service:3-7` | +| 2 | As written: `OnBootSec=30s`, `OnUnitActiveSec=60s`, `AccuracySec=5s`, `Persistent=true`; no OnCalendar, no RandomizedDelaySec. | A | `AG/configs/felhom-mgmt-watchdog.timer:11-14` | +| 3 | No time limit written in the unit; the script is a few stat and systemctl calls. | A | `AG/configs/felhom-mgmt-watchdog.service:16-18` `AG/configs/felhom-mgmt-watchdog.sh:36-61` | +| 4 | A missed tick is retried by the next one; the heal budget is one tick. | A | `AG/configs/felhom-mgmt-watchdog.sh:23-24` `AG/configs/felhom-mgmt-watchdog.timer:3-6` | +| 5 | The script changes nothing when the directory is healthy. MEASURED: the timer is active on the three hosts and fires each 60 seconds. `Persistent=true` in the unit has no effect, because the timer has no calendar line. | A | `AG/configs/felhom-mgmt-watchdog.sh:13-15` `AG/configs/felhom-mgmt-watchdog.timer:11-14` `EV/box/demo-hp-host-clock.txt` | +| 6 | Monotonic timer settings only; the heal marker is a UTC stamp from `date -u`. | A | `AG/configs/felhom-mgmt-watchdog.timer:11-12` `AG/configs/felhom-mgmt-watchdog.sh:65` | +| 7 | The marker lets the agent report the heal and the hub raise a warning,. | A | `AG/configs/felhom-mgmt-watchdog.sh:18-19` `AG/configs/felhom-mgmt-watchdog.sh:63` | +| 8 | SOURCE only. The script has no test file. | A | `AG/configs/felhom-mgmt-watchdog.sh:1-68` `AG/configs/felhom-mgmt-watchdog.timer:3-6` | + +**Hand step in a runbook?** no hand step + +### CA18 — Agent service: restart, start limit, rollback · CUSTOMER · Watchdog · MENTIONED + +architecture folder: 72 lines in 15 files match `/Restart=|start limit|rollback/` (a machine count, not a reading) · helper section: clock-agent „18. Agent service restart, start limit and rollback” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | systemd restarts the agent: `Restart=on-failure`, `RestartSec=5s`; drop-in limits. OPERATOR (every loop in this list dies with the agent). | A | `AG/configs/felhom-agent.service:41-42` `AG/configs/felhom-agent-limits.conf:17-19` | +| 2 | `StartLimitIntervalSec=120`, `StartLimitBurst=4`, `OnFailure=felhom-agent-rollback.service`; a crash-looping binary reaches failed about 20 s after the first crash. | A | `AG/configs/felhom-agent-limits.conf:17-19` `AG/configs/felhom-agent-limits.conf:5-8` | +| 3 | No WatchdogSec in the unit (grep `WatchdogSec` in finds nothing); a new binary commits after a 60 s dwell. | A | `AG/configs/felhom-agent.service:1` `AG/internal/selfupdate/commit.go:40-42` | +| 4 | After the start limit the unit stays failed. Nothing on the host starts it again. The outside sign is the hub's 'host stale' alarm at 45 minutes. | A | `AG/configs/felhom-agent.service:41-42` `AR/11-os-updates.md:572-573` | +| 5 | The rollback unit fires on every crash and is guarded by a pending marker; a shutdown before the dwell leaves the update pending. | A | `AG/configs/felhom-agent-limits.conf:14-15` `AG/internal/selfupdate/commit.go:112-116` | +| 6 | The design text says `Restart=always` and a systemd watchdog that kills a hung agent. The unit has `Restart=on-failure` and no watchdog. | N | `AR/03-host-agent.md:72` `AG/configs/felhom-agent.service:41-42` · class W5 · S17 | +| 7 | The operator, by the hub's stale and down alarms only; a restore-test crash loop that never evaluates is covered by `TestR874_CrashLoopNeverEvaluates`. | A | `AR/11-os-updates.md:572-573` `AG/internal/backup/r874_first_eval_test.go:44` | +| 8 | MEASURED values and a TEST for the rollback. A hung agent (alive, not working): NOBODY KNOWS; no watchdog restarts it. | N | `AG/configs/felhom-agent.service:41-42` · class W5 · S17 | + +**Hand step in a runbook?** PARTLY — `RB/break-glass.md` names `reset-failed`; no runbook names the start limit of the agent + +### CA19 — Off-site tunnel loop and the clock-ahead re-key · CUSTOMER · Interval · MENTIONED + +architecture folder: 236 lines in 22 files match `/wgtunnel|re.?key|tunnel/` (a machine count, not a reading) · helper section: clock-agent „19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Agent goroutines of one shape: tunnel `Run`, PBS-DR, OOB sshd, LAN resolver. CUSTOMER (off-site backup path) and OPERATOR. Four loops merged. | A | `AG/internal/wgtunnel/loop.go:62-78` `AG/internal/pbsdr/loop.go:62-77` `AG/internal/felhomsshd/loop.go:67-80` `AG/internal/lanresolver/loop.go:35-52` | +| 2 | Tunnel, PBS-DR and OOB every 60 s, resolver every 5 min; each applies at once at start and on a desired-state nudge. | A | `AG/internal/config/config.go:132-133` `AG/cmd/felhom-agent/main.go:1504` `AG/internal/config/config.go:74-75` `AG/internal/lanresolver/loop.go:27-29` | +| 3 | No time limit on one pass in the source, and no duration was measured. Nothing grows with data. | N | class W4 · S15 | +| 4 | A missed pass is made up at the next tick; a handshake older than `StaleAfterSeconds` 180 re-resolves the endpoint,. | A | `AG/internal/config/config.go:124-127` `AG/internal/config/config.go:138-140` | +| 5 | A handshake stamped in the future for `rekeyAfterDefault` 15 min makes a new key, at most once per `rekeyMinGap` 1 h; the hour bound survives a restart, the 15 min count does not. | A | `AG/internal/wgtunnel/rekey.go:36-38` `AR/03-host-agent.md:376-378` | +| 6 | A box whose clock once ran ahead is locked out of the off-site endpoint until real time passes that moment; healed by the re-key; known gap: a reboot after the clock was corrected forgets the future stamp. | A | `AR/03-host-agent.md:142` `AR/03-host-agent.md:144-150` `AR/03-host-agent.md:155-156` | +| 7 | The heartbeat carries `handshake_future_s` and the hub alarms on it. | A | `AR/03-host-agent.md:142` | +| 8 | TEST `TestD30_AFutureHandshakeThatStaysReKeys`, `TestD30_ALastReKeyFromTheFutureDoesNotBlock`, `TestR1204_AFutureHandshakeIsReported`; measured live. | A | `AG/internal/wgtunnel/rekey_test.go:35` `AG/internal/wgtunnel/rekey_test.go:213` `AG/internal/wgtunnel/manager_test.go:704` `AR/03-host-agent.md:151-155` | + +**Hand step in a runbook?** no hand step + +### CA20 — Whole-guest backup cap: rolling windows · CUSTOMER · Night · MENTIONED + +architecture folder: 577 lines in 27 files match `/backup cap|rolling window|cap/` (a machine count, not a reading) · helper section: clock-agent „20. Whole-guest backup cap (rolling windows)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Agent local API check `capRefuses` before a backup starts; rule `backupCadenceCap`. CUSTOMER (protects older copies from a flood of new ones). | A | `AG/internal/localapi/server.go:929` `AG/internal/localapi/backup_cap.go:41-53` | +| 2 | Tier of 24 h or less: 3 successes per rolling 24 h; longer tier: 1 per the larger of cadence minus 24 h and half the cadence; a refusal holds for `backupCapRefusalHold` 24 h. | A | `AG/internal/localapi/backup_cap.go:12-35` `AG/internal/localapi/backup_cap.go:38` | +| 3 | A lookup, no duration of its own; the success list is bounded by the window. | A | `AG/internal/localapi/backup_cap.go:41-53` | +| 4 | Does not apply to a missed run: failures do not count, `TestBackupCap_FailuresDoNotCount`. | A | `AG/internal/localapi/backup_cap_test.go:266` | +| 5 | The scheduled daily run is never refused, `TestBackupCap_TheScheduledDayIsNeverRefused`; the count survives a restart, `TestBackupCap_SurvivesARestart`. | A | `AG/internal/localapi/backup_cap_test.go:220` `AG/internal/localapi/backup_cap_test.go:248` | +| 6 | The windows use the wall clock. Successes that were stamped while the clock was ahead stay inside the window longer and can refuse real runs. NOBODY KNOWS. | N | `AG/internal/localapi/server.go:509` · class W2 · S08 | +| 7 | The operator, by event `whole_guest_backup_refused` named in; test `TestBackupCap_RefusalRaisesTheOperatorSignal`. | A | `AG/internal/localapi/backup_cap.go:12-35` `AG/internal/localapi/backup_cap_test.go:285` | +| 8 | TEST `TestBackupCap_LocalFloodRunsAtMostThreeADay`, `TestBackupCap_OffsiteFloodKeepsAPreCompromiseCopy`. | A | `AG/internal/localapi/backup_cap_test.go:198` `AG/internal/localapi/backup_cap_test.go:151` | + +**Hand step in a runbook?** no hand step + +### CA21 — Unsent OS-update report sender · OPERATOR · Interval · UNNAMED + +architecture folder: 0 lines in 0 files match `/unsent/` (a machine count, not a reading) · helper section: clock-agent „22. Unsent OS-update report sender” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Agent goroutine `SendUnsentLoop`, started at; posts OS-update results the hub has not received. OPERATOR. | A | `AG/internal/osupdate/unsent.go:71-82` `AG/cmd/felhom-agent/main.go:1009` | +| 2 | Every 5 min, by `time.After`; one send at start, `SendUnsent`. | A | `AG/cmd/felhom-agent/main.go:1009` `AG/internal/osupdate/unsent.go:79` `AG/internal/osupdate/unsent.go:56-64` | +| 3 | A report post has a 1 min limit in `finish`; the kept copies grow only while the hub is away. | A | `AG/internal/osupdate/leg.go:922-939` | +| 4 | Hub away keeps the copy for a later pass, `TestR868_HubAwayKeepsTheCopy`. | A | `AG/internal/osupdate/unsent_test.go:87` | +| 5 | The sender stays off while a pass holds the flock, `tryLockPass`; test `TestR868_RunningPassKeepsTheSenderOff`. | A | `AG/internal/osupdate/unsent.go:35-52` `AG/internal/osupdate/unsent_test.go:110` | +| 6 | `time.After` is monotonic; no wall-clock decision found in. | A | `AG/internal/osupdate/unsent.go:79` `AG/internal/osupdate/unsent.go:56-82` | +| 7 | The operator sees the result late: at most 5 minutes after the hub is back. The sender has no alarm of its own. | A | `AG/internal/osupdate/unsent.go:1` | +| 8 | TEST `TestR868_ACopyWrittenAfterTheStartIsSentByTheLoop`, `TestR868_KilledPassIsReportedAtStart`. | A | `AG/internal/osupdate/unsent_test.go:127` `AG/internal/osupdate/unsent_test.go:20` | + +**Hand step in a runbook?** no hand step + +### CC01 — Night dump leg: databases and volumes · CUSTOMER · Night · MENTIONED + +architecture folder: 17 lines in 5 files match `/db-dump|dump leg/` (a machine count, not a reading) · helper section: clock-controller „1. Night dump leg (databases and volumes) at W” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | `dbDumpLeg` runs `RunDBDumps`; owner: controller; started by the daily job "db-dump". CUSTOMER. | A | `CT/cmd/controller/main.go:1336-1344` `CT/internal/backup/backup.go:564` `CT/cmd/controller/main.go:1345` | +| 2 | Daily at W. W = household setting, else `DBDumpSchedule`, else `DefaultWindow` "02:30", decided in `EffectiveWindow`. First leg. Zone: Europe/Budapest, hard-coded, `getBudapestLocation`. | A | `CT/internal/backupwindow/backupwindow.go:12` `CT/internal/backupwindow/backupwindow.go:86-94` `CT/internal/scheduler/scheduler.go:17-27` | +| 3 | MEASURED, the night of 2026-11-11: 2 min 33 s on demo-hp, 40 s on Tester 1, 18 s on the N100. One database dump has a 5-minute limit. The whole leg has no limit. It grows with the number and size of the apps; all three stores are small. | A | `EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt` `CT/internal/appbackup/dbdump.go:291` | +| 4 | Box off at W: `Missed` sees it from the ledger and the catch-up runs it (block 4). The hub judges a missing dump at its 05:00 deadline. | A | `CT/internal/nightchain/nightchain.go:197-218` `AR/07-backup-architecture.md:579-584` | +| 5 | `withLeg` takes the leg lock; `acquireRunning` refuses a second backup; a job still running is skipped `executeJob`. A running migration skips the leg and returns no error. | A | `CT/cmd/controller/main.go:1321-1335` `CT/internal/backup/backup.go:1105-1113` `CT/internal/scheduler/scheduler.go:354-364` `CT/internal/backup/backup.go:565-568` | +| 6 | Slot from the wall clock, waited on a monotonic timer `runDailyJob`; fired over 60 min late = skipped `DailyLateLimit`. See block 24 for the hour change. | A | `CT/internal/scheduler/scheduler.go:300-352` `CT/internal/scheduler/scheduler.go:67` | +| 7 | Failure: `NotifyDBDumpFailed`. A made-up miss: one timeline line, never mailed. A miss not made up: banner after 26 h `StaleAfter`. | A | `CT/cmd/controller/main.go:1339` `AR/07-backup-architecture.md:562-564` `CT/internal/nightchain/banner.go:23` | +| 8 | MEASURED 2026-11-11, one night, small stores: 2 min 33 s · 40 s · 18 s (demo-hp · Tester 1 · N100) · TEST `TestCatchUp_OffAtWThenOn_OneCatchUpAfter15Min`; TEST `TestDaily_LateFireIsSkipped`. | A | `EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt` `CT/internal/nightchain/nightchain_test.go:92` `CT/internal/scheduler/scheduler_test.go:87` | + +**Hand step in a runbook?** no hand step + +### CC02 — Second-drive copy leg · CUSTOMER · Night · MENTIONED + +architecture folder: 238 lines in 20 files match `/tier.?2|second.drive|cross.drive/` (a machine count, not a reading) · helper section: clock-controller „2. Second-drive copy leg at W+60 min” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | `RunAllTier2` copies each app's data to the second drive; owner: controller; started by the daily job "tier2-backup". CUSTOMER. | A | `CT/internal/backup/tier2.go:534` `CT/cmd/controller/main.go:1442` | +| 2 | Daily at W+60 min, `tier2OffsetMin`; no setting of its own. It does not wait for the dump leg: "NOTHING waits on the leg before it". Zone as block 1. | A | `CT/internal/backupwindow/backupwindow.go:17` `AR/09-update-architecture.md:2751-2757` | +| 3 | One copy: 60 min, `tier2CopyTimeout`, applied at. No limit for the whole leg found (same grep as block 1). Grows with app count and data size. | A | `CT/internal/backup/tier2_copy_lifecycle.go:40` `CT/internal/backup/tier2.go:828` | +| 4 | Catch-up as block 4 (the leg is in the ledger through `withLeg`). The leg function always returns nil, so the scheduler never sees a failure. | A | `CT/cmd/controller/main.go:1438-1441` | +| 5 | It waits for the dump leg by a start time only (60 minutes later). It is skipped while a migration runs. The file holds no lock of its own. With a backup time between 02:00 and 02:59 on the spring hour-change night the two legs start at the same instant; that case is row R-1260. | N | `CT/cmd/controller/main.go:1321-1335` `CT/internal/backup/tier2.go:538-541` `CT/internal/backup/tier2_copy_lifecycle.go:172` · class W3 · S14 | +| 6 | As block 1: wall-clock slot, monotonic wait, 60 min late limit. | A | `CT/internal/scheduler/scheduler.go:67` | +| 7 | The household and the hub are told of a failed copy: the copy's result goes to the notifier. | A | `CT/cmd/controller/main.go:1427` | +| 8 | MEASURED 2026-11-11: 39 s · 15 s · 11 s · SOURCE; catch-up part TEST `TestCatchUp_PowerCutMidChain_FinishesTheRest`. | A | `EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt` `CT/internal/backup/tier2.go:534` `CT/internal/nightchain/nightchain_test.go:145` | + +**Hand step in a runbook?** no hand step + +### CC03 — Off-site push leg, with the clean-up of old snapshots · CUSTOMER · Night · MENTIONED + +architecture folder: 82 lines in 13 files match `/offbox|off.site push|offsite push/` (a machine count, not a reading) · helper section: clock-controller „3. Off-site push leg at W+105 min (with clean-up of old snapshots)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | `RunOffboxBackup` pushes dumps to the remote store, then runs retention; owner: controller; started by the daily job "offbox-backup". CUSTOMER. | A | `CT/internal/backup/offbox.go:1018` `CT/cmd/controller/main.go:1641-1643` | +| 2 | Daily at W+105 min, `offboxOffsetMin`. Retention keeps 7 daily, `keepDaily`. Zone as block 1. | A | `CT/internal/backupwindow/backupwindow.go:18` `CT/internal/backup/offbox_window.go:47` | +| 3 | Per app 2 h, `offboxBackupTimeout`, applied at; SSH connect 10 s. Grows with data changed and line speed. One real night: 3 min 28 s. | A | `CT/internal/backup/offbox.go:39` `CT/internal/backup/offbox.go:1564` `CT/internal/backup/offbox.go:37` `AR/09-update-architecture.md:2751-2757` | +| 4 | Catch-up as block 4. No escrowed key, orphaned store or migration: skipped with an INFO log line and nil. | A | `CT/internal/backup/offbox.go:1040-1053` | +| 5 | When another backup holds the flag, the push is skipped with an INFO line and the leg is marked as ended. Only the hub's 48-hour check notices. A long second-drive leg can cost the night's off-site copy. | K | `CT/internal/backup/offbox.go:1054-1057` · R-1186 | +| 6 | As block 1. The clean-up window has a one-hour clock tolerance `offsiteGuardSkew`. | A | `CT/internal/backup/offbox_window.go:42` | +| 7 | Failure: `NotifyOffboxBackupFailed` with a cleaned message. The hub's daily deadline is the independent signal. | A | `CT/cmd/controller/main.go:1446-1452` `CT/cmd/controller/main.go:1524-1528` | +| 8 | MEASURED 2026-11-11: 3 min 09 s · 59 s · 38 s · SOURCE; DOC. | A | `EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt` `CT/internal/backup/offbox.go:1040-1057` `AR/07-backup-architecture.md:455-457` | + +**Hand step in a runbook?** NO — no runbook says how to see or repair a skipped off-site night + +### CC04 — Missed-night catch-up and the resume watch · CUSTOMER · Night · MENTIONED + +architecture folder: 43 lines in 12 files match `/catch.?up|nightchain|missed night/` (a machine count, not a reading) · helper section: clock-controller „4. Missed-night catch-up and resume watch” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | `CatchUp.Evaluate` runs the backup legs a box missed; owner: controller; started at controller start and by the resume watch. CUSTOMER. | A | `CT/internal/nightchain/nightchain.go:281-299` `CT/cmd/controller/main.go:1676` `CT/cmd/controller/main.go:1677` | +| 2 | 15 min after the trigger, `DefaultDelay`; a leg due within 30 min is left alone, `leaveToNormal`. Values decided by CC. No setting. | A | `CT/internal/nightchain/nightchain.go:62` `CT/internal/nightchain/nightchain.go:64` `AR/07-backup-architecture.md:574-577` | +| 3 | As long as the legs it runs. It waits up to 2 h for a whole-guest backup, `quiesceWaitMax`, then drops with a WARN. | A | `CT/internal/nightchain/nightchain.go:66` `CT/internal/nightchain/nightchain.go:319` | +| 4 | If the catch-up itself does not run: banner after 26 h (block 5) and the hub's longer lines, 48 h without a dump. Two missed nights give one catch-up, not two. | A | `AR/07-backup-architecture.md:579-584` | +| 5 | One at a time: pending/running guard; leg lock. It never runs the update leg. | A | `CT/internal/nightchain/nightchain.go:288-292` `CT/cmd/controller/main.go:1321-1335` `CT/cmd/controller/main.go:1660-1664` | +| 6 | The watch reads a gap of more than 5 minutes in the wall clock as 'the box was asleep'. A clock that is corrected forward by more than 5 minutes looks the same. The design text says this was not measured live. | N | `CT/internal/nightchain/nightchain.go:375` `CT/cmd/controller/main.go:1681` `AR/07-backup-architecture.md:551-559` · class W2 · S08 | +| 7 | Household: one timeline line, never mailed, event sent at. Dropped catch-up: log line only. | A | `AR/07-backup-architecture.md:562-564` `CT/internal/nightchain/nightchain.go:357-359` `CT/internal/nightchain/nightchain.go:319` | +| 8 | TEST `TestCatchUp_TwoMissedNights_One`; TEST `TestResumeWatch_SeesASuspend`; TEST `TestR871_CatchUpWiring`. | A | `CT/internal/nightchain/nightchain_test.go:126` `CT/internal/nightchain/nightchain_test.go:217` `CT/cmd/controller/r871_catchup_wiring_test.go:16` | + +**Hand step in a runbook?** no hand step + +### CC05 — Missed-backup banner (26 hours) · CUSTOMER · Night · MENTIONED + +architecture folder: 48 lines in 14 files match `/26 h|missed.backup banner|banner/` (a machine count, not a reading) · helper section: clock-controller „5. Missed-backup banner (26 hours)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | `ComputeBanner` shows the household a banner when no night backup ran; owner: controller; computed when the page is drawn (no timer of its own). CUSTOMER. | A | `CT/internal/nightchain/banner.go:56-88` | +| 2 | Shown when the last backup is older than 26 h, `StaleAfter`. After 5 missed days it suggests another hour, `suggestMinDays`, `Suggest`. | A | `CT/internal/nightchain/banner.go:23` `CT/internal/nightchain/banner.go:26` `CT/internal/nightchain/banner.go:95-113` | +| 3 | Instant; reads the ledger only. | A | `CT/internal/nightchain/banner.go:56-88` | +| 4 | Nobody opens the page: nobody sees it. The hub's deadline check is the other channel. | A | `AR/07-backup-architecture.md:579-584` | +| 5 | Not a job; no lock needed. Dismissed, it comes back after a new miss, design at. | A | `AR/07-backup-architecture.md:566-572` | +| 6 | The age is wall-clock arithmetic. A clock that is set back hides the banner for as long as the step. No test sets the clock back. | N | `CT/internal/nightchain/banner.go:56-88` · class W2 · S08 | +| 7 | The household only, on the page; no mail. | A | `AR/07-backup-architecture.md:566-572` | +| 8 | TEST `TestBanner_ShownWithReasonAndSuggestion`; TEST `TestBanner_DismissedThenBackAfterANewMiss`. | A | `CT/internal/nightchain/banner_test.go:29` `CT/internal/nightchain/banner_test.go:59` | + +**Hand step in a runbook?** no hand step + +### CC06 — Whole-guest backup driver: stop apps, ask the agent, start apps · CUSTOMER · Night · MENTIONED + +architecture folder: 264 lines in 21 files match `/quiesce|whole.guest|vzdump/` (a machine count, not a reading) · helper section: clock-controller „6. Whole-guest backup driver (stop apps, ask the host agent, start apps)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | `Loop.Run` stops the apps and asks the agent for the whole-guest backup; owner: controller drives, agent does the backup; started by `startQuiesceLoop`. CUSTOMER. | A | `CT/internal/quiesce/quiesce.go:191-206` `CT/cmd/controller/main.go:4008` | +| 2 | Polls every 5 min; runs once per 24 h cadence, only inside W+2h to W+6h, `gateOpenOffsetMin`, `gateSpanMin`. Zone `budapestLocation`. | A | `CT/internal/quiesce/quiesce.go:149-151` `CT/internal/quiesce/quiesce.go:161-163` `CT/internal/quiesce/quiesce.go:760` `CT/internal/quiesce/quiesce.go:761` | +| 3 | Apps stay down at most 30 min, `MaxQuiesce` default, setting `max_quiesce_duration`. Manual run limit = that + 5 min. Grows with guest disk size. | A | `CT/cmd/controller/main.go:3990` `CT/internal/config/config.go:132` `CT/internal/quiesce/quiesce.go:430` | +| 4 | Outside the window it still runs when the last one is older than cadence + 24 h (the 48 h valve). Apps left stopped by a crash are started by `Recover`, called at. | A | `CT/internal/quiesce/quiesce.go:801-818` `CT/internal/quiesce/quiesce.go:177-188` `CT/cmd/controller/main.go:4007` | +| 5 | Single-flight `TryLock`; waits for a catch-up; waits for the update leg, never past W+5h30 `legStepGraceMin`. | A | `CT/internal/quiesce/quiesce.go:213-216` `CT/internal/quiesce/quiesce.go:250-253` `CT/internal/quiesce/quiesce.go:271-274` `CT/internal/quiesce/quiesce.go:935` | +| 6 | The gate counts wall-clock minutes; the poll is a monotonic ticker. On the hour-change night the two disagree. No test covers that night. | K | `CT/internal/quiesce/quiesce.go:801-818` `CT/internal/quiesce/quiesce.go:191-206` · R-1260 | +| 7 | First failure notifies once `noteTierFailure`; retry back-off 15 min to 4 h, busy agent: retry 15 min, alarm after 3 h,. | A | `CT/internal/quiesce/quiesce.go:283-298` `CT/internal/quiesce/breaker.go:47` `CT/internal/quiesce/breaker.go:54` `CT/internal/quiesce/contention.go:53` | +| 8 | TEST `TestScheduledRunAllowed`; TEST `TestRunOnce_ValveRunsOutsideWindow`; TEST `TestBreaker_BackoffSchedule`. | A | `CT/internal/quiesce/quiesce_window_test.go:23` `CT/internal/quiesce/quiesce_window_test.go:94` `CT/internal/quiesce/breaker_test.go:218` | + +**Hand step in a runbook?** no hand step + +### CC07 — Automatic app-update leg · CUSTOMER · Night · MENTIONED + +architecture folder: 136 lines in 17 files match `/update.leg|unattended/` (a machine count, not a reading) · helper section: clock-controller „7. Automatic app-update leg (after the off-site push)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | `RunUpdateLeg` presses one update step per app; owner: controller; started by `chainUpdateLeg` when the off-site half ends, on every path. CUSTOMER. | A | `CT/internal/stacks/unattended.go:230` `CT/cmd/controller/main.go:1641-1643` | +| 2 | Each night after the off-site leg; a box with no off-site tier runs it at W+105 min. One step per app per night, switch default ON. | A | `AR/07-backup-architecture.md:459-465` `AR/09-update-architecture.md:2806-2817` | +| 3 | No new step at or after W+5h: `LegDeadline`, `UpdateLegStopOffsetMin`; check at. Grows with the number of apps that have an update. | A | `CT/internal/stacks/unattended.go:212-216` `CT/internal/backupwindow/backupwindow.go:27` `CT/internal/stacks/unattended.go:374-375` | +| 4 | Missed night: no catch-up by design; the rest "waits for the next night". Controller restarted mid-leg: the rest of the night is lost (R-686). | A | `CT/cmd/controller/main.go:1660-1664` `CT/internal/stacks/unattended.go:374-375` `AR/09-update-architecture.md:2815-2817` | +| 5 | A passing refusal (busy, updating) is retried once after 2 min, `RetryWait`, rule at. The whole-guest gate waits for the leg. | A | `CT/internal/stacks/unattended.go:167-171` `AR/09-update-architecture.md:2882-2883` `CT/internal/quiesce/quiesce.go:271-274` | +| 6 | The leg's deadline is the backup time plus 5 hours of real time, and the scheduler's slots are in Budapest time. On the hour-change night a step can start after the backup gate stopped waiting. A daily fire that is more than 60 minutes late is skipped, so no update runs at noon. | K | `CT/internal/stacks/unattended.go:212-216` `CT/internal/scheduler/scheduler.go:17-27` `AR/07-backup-architecture.md:551-559` · R-1260 | +| 7 | Nobody. When the leg does not run, the only trace is one log line on the box. No event goes to the hub. | N | `CT/internal/stacks/unattended.go:374-375` · class W1 · S06 | +| 8 | TEST `TestChainUpdateLeg_EveryPath`; TEST `TestLegDeadlineAndGateShareW5h`. | A | `CT/cmd/controller/update_leg_wiring_test.go:19` `CT/internal/quiesce/d20_update_leg_test.go:78` | + +**Hand step in a runbook?** no hand step + +### CC08 — Off-site integrity check · CUSTOMER · Night · MENTIONED + +architecture folder: 23 lines in 9 files match `/integrity/` (a machine count, not a reading) · helper section: clock-controller „8. Off-site integrity check at 06:00” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Daily job "offsite-integrity" checks the remote store's data; owner: controller. CUSTOMER. | A | `CT/cmd/controller/main.go:1724` | +| 2 | Fires daily at 06:00 (fixed, no setting for the hour) but works only when due: last good check older than 7 days, `defaultIntegrityMaxAgeDays`, `IntegrityDue`. Zone as block 1. | A | `CT/internal/backup/offbox_integrity.go:81` `CT/internal/backup/offbox_integrity.go:221` | +| 3 | MEASURED 2026-11-11: under 1 second on all three boxes (small stores). The check reads the whole store; the time on a large store rests on one measurement. | K | `EV/box/demo-hp-guest-night.txt` · R-401 | +| 4 | Box off at 06:00: caught up at the next 06:00 by due-ness; also run after a catch-up. | A | `CT/internal/backup/offbox_integrity.go:212-220` `CT/cmd/controller/main.go:1665-1668` | +| 5 | Skipped when "a backup or restore is already running"; then it waits for the next 06:00 (source comment). | A | `CT/internal/backup/offbox_integrity.go:298` `CT/cmd/controller/main.go:1710-1723` | +| 6 | Due-ness compares wall-clock stamps; a never-checked or unreadable stamp counts as due. Slot behaviour as block 24. | A | `CT/internal/backup/offbox_integrity.go:231` `CT/internal/backup/offbox_integrity.go:223-229` | +| 7 | A failed check goes to the notifier: the household and the hub are told. | A | `CT/cmd/controller/main.go:4144-4186` | +| 8 | MEASURED 2026-11-11: under 1 s on each box · SOURCE; TEST `TestR1189d_TheChecksRunAfterTheCatchUpLegs`. | A | `EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt` `CT/internal/backup/offbox_integrity.go:212-232` `CT/internal/nightchain/r1189d_test.go:13` | + +**Hand step in a runbook?** no hand step + +### CC09 — Off-site proof restore · CUSTOMER · Night · MENTIONED + +architecture folder: 136 lines in 21 files match `/proof restore|offsite.proof|proof/` (a machine count, not a reading) · helper section: clock-controller „9. Off-site proof restore at 05:30” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Daily job "offsite-proof" restores one due snapshot to prove it opens, `ProveOffboxUnit`; owner: controller. CUSTOMER. | A | `CT/cmd/controller/main.go:1746` `CT/internal/backup/offbox_proof.go:140` | +| 2 | Daily at 05:30, fixed, no setting. One app per night in rotation (comment). Zone as block 1. | A | `CT/cmd/controller/main.go:1746` `CT/internal/backup/offbox_proof.go:265` | +| 3 | Limit 10 min, `proofRestoreTimeout`. Grows with the size of the one unit. | A | `CT/internal/backup/offbox_proof.go:80` | +| 4 | Box on for the backup legs but off at 05:30: "that day's proof is not made up". Made up only together with a missed backup leg. | A | `CT/cmd/controller/main.go:1660-1664` `CT/cmd/controller/main.go:1665-1668` | +| 5 | MEASURED 2026-11-11: on Tester 1 and on the N100 the whole-guest backup started at about 05:30 local time, the same minute as this job. This job is not behind the whole-guest gate. It takes the same flag as the off-site push; a meeting there is a silent skip (R-1186). | N | `EV/box/tester-1-host-night.txt` `EV/box/demo-felhom-host-night.txt` `EV/box/tester-1-guest-night.txt` · class W3 · S12 | +| 6 | As block 24. | A | `CT/cmd/controller/main.go:1746` `CT/internal/backup/offbox_proof.go:140` (the code path of question 1) | +| 7 | An empty or failed proof goes to the notifier: the household and the hub are told. | A | `CT/cmd/controller/main.go:4144-4186` | +| 8 | MEASURED 2026-11-11: 5 s · 5 s · 3 s · SOURCE; TEST `TestC5_TheWholeGuestBackupDoesNotWaitForTheChecks`. | A | `EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt` `CT/cmd/controller/main.go:1660-1664` `CT/internal/nightchain/r1189d_test.go:51` | + +**Hand step in a runbook?** no hand step + +### CC10 — Hub report: every 15 minutes, on change, and the hanging wait · CUSTOMER · Interval · MENTIONED + +architecture folder: 91 lines in 22 files match `/15.min|hub report|pusher/` (a machine count, not a reading) · helper section: clock-controller „10. Hub report every 15 minutes (plus push on change and hanging wait)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Job "hub-report" sends the box state to the hub; owner: controller. The hub's answer carries the version floor and config. CUSTOMER and OPERATOR. | A | `CT/cmd/controller/main.go:1835` `CT/cmd/controller/main.go:1233-1290` | +| 2 | Every `PushInterval`, default "15m", fallback. Also 5 s after start, and on change with a 15 s floor, `triggerMinInterval`. No time zone. | A | `CT/internal/config/config.go:416` `CT/cmd/controller/main.go:1831-1834` `CT/cmd/controller/main.go:1932-1936` `CT/internal/report/trigger.go:30` | +| 3 | HTTP limit 30 s; 3 tries, 5 s apart. The wait connection holds up to 300 s. | A | `CT/internal/report/pusher.go:85` `CT/internal/report/pusher.go:114-117` `CT/internal/report/waiter.go:35` | +| 4 | A failed push is not queued: "next cycle reconciles". The hub notices silence (hub side, not read here). | A | `CT/internal/report/trigger.go:98` | +| 5 | A report still running is skipped. Bursts are merged after a 2 s quiet window. | A | `CT/internal/scheduler/scheduler.go:354-364` `CT/internal/report/trigger.go:27` | +| 6 | Monotonic ticker; a wrong wall clock does not change the rhythm, only the stamps inside the report. | A | `CT/internal/scheduler/scheduler.go:281-298` | +| 7 | Nobody on the box: the box stores the time of the last attempt only. The outside sign is the hub's liveness check (CH01). | A | `CT/internal/report/pusher.go:109-111` `HUB/monitor/host_staleness.go:59-60` | +| 8 | TEST `TestTrigger_BurstCoalescesTrailingEdge`; TEST `TestWaiter_FiresOnceOnGenChange`. | A | `CT/internal/report/trigger_test.go:114` `CT/internal/report/waiter_test.go:60` | + +**Hand step in a runbook?** no hand step + +### CC11 — Controller self-update: check, floor, nightly auto · CUSTOMER · Interval · MENTIONED + +architecture folder: 65 lines in 13 files match `/self.?update/` (a machine count, not a reading) · helper section: clock-controller „11. Controller self-update: check, floor, nightly auto” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | `CheckForUpdate` looks for a newer controller image; `MaybeAutoUpdate` moves a box below the floor. Owner: controller, the agent swaps the image. CUSTOMER and OPERATOR. | A | `CT/internal/selfupdate/updater.go:153` `CT/internal/selfupdate/updater.go:612` | +| 2 | Check every `CheckInterval`, default "6h", and 25 s after start. Floor: after every hub report. Auto: daily at `AutoUpdateTime`, default "04:30", only if `AutoUpdate`. | A | `CT/internal/config/config.go:417` `CT/cmd/controller/main.go:1981-1989` `CT/cmd/controller/main.go:1249-1252` `CT/internal/config/config.go:419` | +| 3 | The registry call has a 15-second limit. The swap of the container has no limit in the file, and no duration was measured in this run. | N | `CT/internal/selfupdate/updater.go:245` · class W4 · S15 | +| 4 | No catch-up needed for the floor: it is tried again on each report until done, but one attempt per floor value. A failed attempt at a floor is not repeated. | A | `CT/internal/selfupdate/updater.go:656-661` `CT/internal/selfupdate/updater.go:604-606` | +| 5 | The auto-update waits while a backup or an app update runs. Its default time, 04:30, is the minute at which the whole-guest gate opens for a 02:30 backup time. Nothing orders the two. | N | `CT/internal/config/config.go:419` `CT/internal/selfupdate/updater.go:533-544` · class W3 · S12 | +| 6 | The check uses a monotonic ticker. The nightly auto-update is a daily slot (see CC22). The switch is off unless the config turns it on; the default time is 04:30. | A | `CT/internal/config/config.go:278-279` `CT/internal/config/config.go:419` | +| 7 | Auto-update refused: WARN log only. A version the floor cannot compare: WARN once. | A | `CT/cmd/controller/main.go:1879-1881` `CT/internal/selfupdate/updater.go:639` | +| 8 | TEST `TestMaybeAutoUpdate_NoFlap`; TEST `TestMaybeAutoUpdate_BelowFloor_UpdatesToFloor`. | A | `CT/internal/selfupdate/floor_test.go:93` `CT/internal/selfupdate/floor_test.go:11` | + +**Hand step in a runbook?** no hand step + +### CC12 — Catalog sync: app templates from git · CUSTOMER · Interval · MENTIONED + +architecture folder: 147 lines in 18 files match `/catalog sync|git.sync|sync/` (a machine count, not a reading) · helper section: clock-controller „12. Catalog sync (app templates from git)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | `Syncer.Start` pulls the app catalog; owner: controller; started at controller start. CUSTOMER. | A | `CT/internal/sync/sync.go:125` | +| 2 | Once at start, then every `SyncInterval`, default "15m"; a bad value falls back to 15 min. No time zone. | A | `CT/internal/sync/sync.go:140-143` `CT/internal/config/config.go:399` `CT/internal/sync/sync.go:131-135` | +| 3 | Each git command has its own limit. One whole sync has none, and no duration was measured. | N | `CT/internal/sync/sync.go:724` `CT/internal/sync/sync.go:215-216` · class W4 · S15 | +| 4 | A missed sync is done at the next tick. A sync that fails for days writes an ERROR line on the box at each tick. Nobody is told. | N | `CT/internal/sync/sync.go:236` `CT/internal/sync/sync.go:247` · class W1 · S23 | +| 5 | Two syncs cannot run at once: `doSync` takes the syncer's lock first. The 30-second guard is for the button only. | A | `CT/internal/sync/sync.go:215-216` `CT/internal/sync/sync.go:153-155` | +| 6 | Monotonic ticker; own ticker, not the shared scheduler, so it has no "still running" skip from. | A | `CT/internal/sync/sync.go:146` `CT/internal/scheduler/scheduler.go:354-364` | +| 7 | Log line per sync only. | A | `CT/internal/sync/sync.go:155` | +| 8 | SOURCE. | A | `CT/internal/sync/sync.go:124-159` | + +**Hand step in a runbook?** no hand step + +### CC13 — System health check and base-stack self-heal · CUSTOMER · Interval · MENTIONED + +architecture folder: 57 lines in 16 files match `/health check|self.heal/` (a machine count, not a reading) · helper section: clock-controller „13. System health check and base-stack self-heal” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Job "system-health" checks disk, memory and base services, and puts the base stack back; owner: controller. CUSTOMER. | A | `CT/cmd/controller/main.go:981-1006` | +| 2 | Every `SystemHealthInterval`, default "5m", fallback; env override. First run one interval after start. | A | `CT/internal/config/config.go:409` `CT/cmd/controller/main.go:977-980` `CT/internal/config/config.go:453` `CT/internal/scheduler/scheduler.go:281-298` | +| 3 | No time limit in the source, and no duration was measured. Nobody knows how long it takes on a big box. | N | `CT/cmd/controller/main.go:981-1006` · class W4 · S15 | +| 4 | No catch-up; next tick. A controller that is down checks nothing; the hub's missing reports are the outside sign (block 10). | A | `CT/cmd/controller/main.go:981-1006` (the code path of question 1) | +| 5 | Skipped while the previous run is still going. | A | `CT/internal/scheduler/scheduler.go:354-364` | +| 6 | Monotonic ticker; not sensitive to wall clock. | A | `CT/cmd/controller/main.go:981-1006` (the code path of question 1) | +| 7 | `NotifyHealthChange` on a change of state. | A | `CT/cmd/controller/main.go:1004` | +| 8 | SOURCE. | A | `CT/cmd/controller/main.go:981-1006` | + +**Hand step in a runbook?** no hand step + +### CC14 — Dead-app scan · CUSTOMER · Interval · MENTIONED + +architecture folder: 62 lines in 13 files match `/stack.scan|dead.app|crash.?loop/` (a machine count, not a reading) · helper section: clock-controller „14. Dead-app scan every 30 seconds” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Job "deadapp-check" finds apps that should run and do not; owner: controller. CUSTOMER. | A | `CT/cmd/controller/main.go:1086-1092` | +| 2 | Every 30 s; silent for 90 s after start, about 15 min after a crash boot. | A | `CT/cmd/controller/main.go:1087-1088` | +| 3 | No time limit in the source, and no duration was measured. Nobody knows how long it takes on a big box. It grows with the number of apps. | N | `CT/cmd/controller/main.go:1086-1143` · class W4 · S15 | +| 4 | Next tick; no catch-up needed. | A | `CT/cmd/controller/main.go:1086-1092` (the code path of question 1) | +| 5 | Skipped if still running. It reads the whole-guest loop state so a planned stop is not an alarm. | A | `CT/internal/scheduler/scheduler.go:354-364` `CT/cmd/controller/main.go:1090` | +| 6 | The quiet time after a start uses the wall clock. A clock that steps back after the start makes the quiet time longer. No test. | N | `CT/cmd/controller/main.go:1087` · class W2 · S08 | +| 7 | Alert on the page via `SetDeadAppAlerts`. | A | `CT/cmd/controller/main.go:1091` | +| 8 | SOURCE. | A | `CT/cmd/controller/main.go:1086-1092` | + +**Hand step in a runbook?** no hand step + +### CC15 — Disk-fill watch · CUSTOMER · Interval · MENTIONED + +architecture folder: 4 lines in 3 files match `/fill.?watch|disk.fill/` (a machine count, not a reading) · helper section: clock-controller „15. Disk-fill watch” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | `fillWatcher.Check` watches how full the drives are; owner: controller; three starters. CUSTOMER. | A | `CT/cmd/controller/main.go:1804-1805` | +| 2 | Every 10 min, `fillWatchInterval`; daily at 03:30; once 90 s after start, `fillWatchStartupDelay`. Cadence stated at. | A | `CT/cmd/controller/main.go:2690` `CT/cmd/controller/main.go:1804` `CT/cmd/controller/main.go:2685` `CT/cmd/controller/main.go:1772` | +| 3 | No time limit in the source, and no duration was measured. Nobody knows how long it takes on a big box. | N | `CT/cmd/controller/main.go:1804-1805` · class W4 · S15 | +| 4 | Next 10-minute tick. | A | `CT/cmd/controller/main.go:1804-1805` (the code path of question 1) | +| 5 | The night slot and the interval job are two jobs with two names. The watcher holds one lock, so the two cannot change its state at the same time. | A | `CT/internal/fillwatch/fillwatch.go:162` `CT/cmd/controller/main.go:1804-1805` | +| 6 | Interval is monotonic; the 03:30 slot as block 24. | A | `CT/cmd/controller/main.go:1804-1805` (the code path of question 1) | +| 7 | The household: the watcher pushes a customer event when a disk fills. | A | `CT/internal/fillwatch/fillwatch.go:80-153` | +| 8 | TEST `TestFillWatchRunsOnAnInterval`. | A | `CT/cmd/controller/r363_fillwatch_interval_test.go:17` | + +**Hand step in a runbook?** no hand step + +### CC16 — Drive health check, hourly · CUSTOMER · Interval · MENTIONED + +architecture folder: 15 lines in 6 files match `/SMART|disk.health|drive health/` (a machine count, not a reading) · helper section: clock-controller „16. Drive health check, hourly” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Job "disk-health-check" asks the agent for drive health; owner: controller reads, agent measures. CUSTOMER. | A | `CT/cmd/controller/main.go:2226` | +| 2 | Every 1 h; reason for hourly, not 6 h:. Only when the agent channel is configured. | A | `CT/cmd/controller/main.go:2219-2224` `CT/cmd/controller/main.go:2225` | +| 3 | One fetch was measured at about 0.8 s (a comment in the source). No time limit in the source. | N | `CT/cmd/controller/main.go:2219-2224` · class W4 · S15 | +| 4 | Next hour; first run one hour after start. | A | `CT/internal/scheduler/scheduler.go:281-298` | +| 5 | Skipped if still running. | A | `CT/internal/scheduler/scheduler.go:354-364` | +| 6 | Monotonic ticker. | A | `CT/cmd/controller/main.go:2226` (the code path of question 1) | +| 7 | The household and the hub: the check emits `disk_health_degraded`. | A | `CT/internal/web/disk_health.go:175-280` | +| 8 | SOURCE. | A | `CT/cmd/controller/main.go:2219-2226` | + +**Hand step in a runbook?** no hand step + +### CC17 — Agent channel health, every 60 seconds · CUSTOMER · Interval · MENTIONED + +architecture folder: 2 lines in 2 files match `/agent channel|channel health/` (a machine count, not a reading) · helper section: clock-controller „17. Agent channel health, every 60 seconds” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Job "agent-channel-health" probes the controller-to-agent channel; owner: controller. OPERATOR. | A | `CT/cmd/controller/main.go:2289` | +| 2 | Every 60 s. A passing fault needs 2 misses in a row, `debounceThreshold`. | A | `CT/internal/channelhealth/checker.go:29` | +| 3 | Probe limit 15 s, `probeTimeout`. | A | `CT/internal/channelhealth/checker.go:32` | +| 4 | Next minute. | A | `CT/cmd/controller/main.go:2289` (the code path of question 1) | +| 5 | Skipped if still running. An alert marker file survives a restart. | A | `CT/internal/scheduler/scheduler.go:354-364` `CT/cmd/controller/main.go:2288` | +| 6 | Monotonic ticker. | A | `CT/cmd/controller/main.go:2289` (the code path of question 1) | +| 7 | Sink = notifier and page alert. | A | `CT/cmd/controller/main.go:2286` | +| 8 | TEST `TestDebounce_TransientBlipSuppressed`. | A | `CT/internal/channelhealth/checker_test.go:119` | + +**Hand step in a runbook?** no hand step + +### CC18 — Held-app report to the hub · CUSTOMER · Interval · MENTIONED + +architecture folder: 32 lines in 7 files match `/held.app|held app/` (a machine count, not a reading) · helper section: clock-controller „18. Held-app report to the hub” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | `Reporter.Tick` tells the hub which apps are held; owner: controller; job "hold-report". OPERATOR. | A | `CT/internal/holdreport/holdreport.go:89` `CT/cmd/controller/main.go:1081-1084` | +| 2 | Looks every minute; each hold is sent again once a day, `ResendEvery`, and after each start. | A | `CT/internal/holdreport/holdreport.go:36` `CT/cmd/controller/main.go:1071-1073` | +| 3 | Instant; reads the hold store. | A | `CT/internal/holdreport/holdreport.go:89` `CT/cmd/controller/main.go:1081-1084` (the code path of question 1) | +| 4 | Next minute. | A | `CT/internal/holdreport/holdreport.go:89` `CT/cmd/controller/main.go:1081-1084` (the code path of question 1) | +| 5 | Skipped if still running. | A | `CT/internal/scheduler/scheduler.go:354-364` | +| 6 | The daily resend is wall-clock arithmetic. A clock that is set back delays the resend by the size of the step. | N | `CT/cmd/controller/main.go:1082` · class W2 · S08 | +| 7 | The hub, by event. | A | `CT/cmd/controller/main.go:1074` | +| 8 | TEST `TestD64_TheHoldReporterTellsTheHubAfterTheClear`. | A | `CT/cmd/controller/holdclear_test.go:333` | + +**Hand step in a runbook?** no hand step + +### CC19 — Abandoned off-site store sweep · CUSTOMER · Night · MENTIONED + +architecture folder: 21 lines in 9 files match `/abandon/` (a machine count, not a reading) · helper section: clock-controller „19. Abandoned off-site store sweep at 05:10” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Daily job "offsite-abandon-sweep" runs `AbandonSweep`; owner: controller. CUSTOMER. | A | `CT/cmd/controller/main.go:1693` `CT/internal/backup/offbox_abandon.go:214` | +| 2 | Daily at 05:10, fixed. The due date is counted in UTC days, the slot in Europe/Budapest. | A | `CT/internal/backup/offbox_abandon.go:366` `CT/internal/scheduler/scheduler.go:17-27` | +| 3 | One sweep has a 1-minute limit. | A | `CT/internal/backup/offbox_abandon.go:180` | +| 4 | Box off at 05:10: next day; no catch-up (it is not a ledger leg, see). | A | `CT/cmd/controller/main.go:1660-1664` | +| 5 | The sweep takes no lock. MEASURED: on the three boxes it ran 55 minutes after the off-site push ended, so they did not meet on these small stores. | N | `CT/internal/backup/offbox_abandon.go:180` `EV/box/demo-hp-guest-night.txt` · class W3 · S11 | +| 6 | As block 24. | A | `CT/cmd/controller/main.go:1693` `CT/internal/backup/offbox_abandon.go:214` (the code path of question 1) | +| 7 | Nobody. A sweep that fails writes a WARN or ERROR line on the box only. | N | `CT/internal/backup/offbox_abandon.go:180` · class W1 · S06 | +| 8 | MEASURED 2026-11-11: the slot ran on each box · SOURCE. | A | `EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt` `CT/cmd/controller/main.go:1693` | + +**Hand step in a runbook?** no hand step + +### CC20 — Restore-copy sweep and conversion-copy release · CUSTOMER · Interval · MENTIONED + +architecture folder: 14 lines in 3 files match `/restore.copy|restore copy/` (a machine count, not a reading) · helper section: clock-controller „20. Restore-copy sweep and conversion-copy release” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | `SweepUnusedRestoreCopies` deletes old kept copies; job "restore-copy-sweep"; job "conversion-copy-release". CUSTOMER. | A | `CT/internal/backup/restore_copy_sweep.go:36` `CT/cmd/controller/main.go:1414` `CT/cmd/controller/main.go:1407` | +| 2 | The sweep runs each 6 hours; the release each hour. A restore copy that nobody used goes after 30 days (the constant, D62). A comment in the wiring still says 14 days; the comment is stale. | A | `CT/cmd/controller/main.go:1412` | +| 3 | No time limit in the source, and no duration was measured. Nobody knows how long it takes on a big box. | N | class W4 · S15 | +| 4 | The first run is 6 hours after the controller starts. A box that is on for under 6 hours a day never runs it; old restore copies stay. Nobody is told. | N | `CT/internal/scheduler/scheduler.go:281-298` · class W1 · S25 | +| 5 | Skipped if still running. | A | `CT/internal/scheduler/scheduler.go:354-364` | +| 6 | Age is wall-clock arithmetic on `now`. | A | `CT/internal/backup/restore_copy_sweep.go:36` | +| 7 | The household is told when a restore copy is deleted. | A | `CT/internal/notify/notifier.go:1601` | +| 8 | SOURCE. | A | `CT/internal/backup/restore_copy_sweep.go:29` | + +**Hand step in a runbook?** no hand step + +### CC21 — Upload and CPU watch, torrent limit rule, line-test hold · CUSTOMER · Interval · MENTIONED + +architecture folder: 70 lines in 16 files match `/upwatch|unexplained|upload/` (a machine count, not a reading) · helper section: clock-controller „22. Upload and CPU watch, torrent limit rule, line-test hold” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | `startUpwatch` runs the minute watcher and a 15-minute rule that holds the torrent app for the line test; owner: controller. CUSTOMER. | A | `CT/cmd/controller/upwatch_wiring.go:136` | +| 2 | Watcher every 1 min; rule every 15 min. Baseline: 7 days of data, cached 24 h. | A | `CT/cmd/controller/upwatch_wiring.go:139` `CT/cmd/controller/upwatch_wiring.go:141` `CT/cmd/controller/upwatch_wiring.go:124-127` | +| 3 | Hold call limit 30 s, `holdTimeout`; drain wait at most 90 s. | A | `CT/internal/upwatch/linehold.go:79` `CT/internal/upwatch/linehold.go:216` | +| 4 | A limit left at the test value by a stopped controller is put back at start, `RecoverHold`. | A | `CT/cmd/controller/upwatch_wiring.go:137` | +| 5 | Two loops with their own tickers. The file holds no lock between them. | N | class W3 · S14 | +| 6 | Monotonic tickers. | A | `CT/internal/upwatch/watch.go:151` | +| 7 | The household and the hub: the watch emits `app_unexplained_upload` and `app_unexplained_cpu`. | A | `CT/internal/upwatch/watch.go:309-382` | +| 8 | SOURCE. | A | `CT/cmd/controller/upwatch_wiring.go:134-152` | + +**Hand step in a runbook?** no hand step + +### CC22 — The scheduler itself: daily slots and interval jobs · CUSTOMER · Night · MENTIONED + +architecture folder: 24 lines in 10 files match `/scheduler|daily slot/` (a machine count, not a reading) · helper section: clock-controller „24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | `Daily` and `Every` carry every named job; owner: controller. CUSTOMER. | A | `CT/internal/scheduler/scheduler.go:131-157` `CT/internal/scheduler/scheduler.go:105-127` | +| 2 | Daily slots are HH:MM in Europe/Budapest, hard-coded; if the zone data cannot load it silently uses UTC. Interval jobs do not run at start: first run after one interval. | A | `CT/internal/scheduler/scheduler.go:17-27` `CT/internal/scheduler/scheduler.go:281-298` | +| 3 | No time limit of its own; shutdown waits 30 s. A panic in a job is caught. | A | `CT/internal/scheduler/scheduler.go:264` `CT/internal/scheduler/scheduler.go:438-447` | +| 4 | A daily job has no catch-up in the scheduler; a fire over 60 min late is skipped. Only the three backup legs are made up (block 4). | A | `CT/internal/scheduler/scheduler.go:344-347` | +| 5 | One run per job at a time: "still running" skip. No rule between different jobs here. | A | `CT/internal/scheduler/scheduler.go:354-364` | +| 6 | Slots are in Budapest time. On the spring night 02:30 does not exist; on the autumn night it comes twice. No test covers either night. A clock set back while a timer waits does not matter: the timer runs on monotonic time. | K | `CT/internal/scheduler/scheduler.go:499-519` · R-1260 | +| 7 | A skipped late fire is a log line only. | A | `CT/internal/scheduler/scheduler.go:344-347` | +| 8 | SOURCE. MEASURED 2026-11-11: the night's order and the start times on three boxes (see the README's night table). | A | `CT/internal/scheduler/scheduler.go:344-347` `EV/box/demo-hp-guest-night.txt` | + +**Hand step in a runbook?** no hand step + +### CH01 — Box liveness: 'your server cannot be reached' · CUSTOMER · Hub sweep · MENTIONED + +architecture folder: 50 lines in 16 files match `/host_stale|liveness|staleness/` (a machine count, not a reading) · helper section: clock-hub „1. Box liveness: "your server cannot be reached"” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | CUSTOMER. The hub's 60 s sweep runs `stalenessChecker.Check` and `hostStaleness.Check`; they raise `host_stale`, `host_down`, `host_recovered`. Owner: hub pod. Merged here: the controller-report checker and the host-report checker. | A | `E/hub/cmd/hub/main.go:935-936` `HUB/monitor/host_staleness.go:207-221` | +| 2 | Every 60 s. Stale after `stale_threshold`, default `"30m"`, deployed 45 m; down after 2x. No time zone: ages are durations. | A | `E/hub/cmd/hub/main.go:927` `E/hub/cmd/hub/main.go:1027` `E/manifests/hub.yaml:88` `HUB/monitor/host_staleness.go:39` | +| 3 | No time limit. The checkers run one after the other in one loop: one read of all hosts at each pass, then a loop over the hosts. The time grows with the number of boxes. Nobody measured one pass. | N | `E/hub/cmd/hub/main.go:935-962` `HUB/monitor/host_staleness.go:113` · class W4 · S16 | +| 4 | With the hub down nothing is judged. At the start, a box that is already silent gets one alarm at the first pass, unless that alarm was raised before. The controller's checker starts silent for such a box. | A | `HUB/monitor/host_staleness.go:59-68` `HUB/monitor/staleness.go:86-91` | +| 5 | One pod, `replicas: 1`, `Recreate`; state under a mutex. Events fire on transitions only. A household power-off holds alarms for `DeliberateOffHold`. | A | `E/manifests/hub.yaml:122` `E/manifests/hub.yaml:123-124` `HUB/monitor/host_staleness.go:119-120` `HUB/monitor/host_staleness.go:170-172` | +| 6 | Judged on the hub's receive time, not the box clock: `last_report_at = datetime('now')`. A wrong box clock cannot hide silence. A hub clock jump moves every age at once; DST has no effect (durations). | A | `HUB/store/store.go:3916` `HUB/monitor/host_staleness.go:131` | +| 7 | Operator: every edge; liveness events bypass the 1 h quiet hour, 5 min dedupe. Household: at most once per 7 days, `householdLivenessQuiet`. If the hub itself is down nobody is told by this job. | A | `HUB/notify/dispatcher.go:609-612` `HUB/notify/dispatcher.go:1018` | +| 8 | TEST `TestHostStaleness_R1001_RestartDoesNotReMail`; TEST `TestR873_HouseholdHearsAnOutageAtMostWeekly`; DOC. | A | `HUB/monitor/r1001_restart_test.go:56` `HUB/notify/r873_liveness_weekly_test.go:17` `AR/05-hub-architecture.md:88-118` | + +**Hand step in a runbook?** no hand step + +### CH02 — Missed-backup deadline check (05:00) · CUSTOMER · Hub daily · MENTIONED + +architecture folder: 28 lines in 11 files match `/deadline check|backup.deadline|05:00/` (a machine count, not a reading) · helper section: clock-hub „2. Missed-backup deadline check (05:00)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | CUSTOMER. `scheduleDaily(ctx, "deadline-check", "05:00",...)` runs `CheckBackupDeadlines`; it raises `expected_backup_missed`. Owner: hub pod. | A | `E/hub/cmd/hub/main.go:986-988` `HUB/monitor/deadline.go:409` `HUB/monitor/deadline.go:539-544` | +| 2 | Once a day at 05:00 Budapest wall-clock; the slot is a literal in, no setting. Zone chosen in `scheduleDaily`. Limits: `backupStaleAfter` 26 h, off-site `offsiteBackupStaleAfter` 8 d. | A | `E/hub/cmd/hub/main.go:986` `E/hub/cmd/hub/main.go:1072` `HUB/monitor/deadline.go:29` `HUB/monitor/deadline_tiers.go:28` | +| 3 | No time limit; the check runs inside the scheduler's own loop. It reads up to 7 days of evidence for every customer. The time grows with boxes times reports. Nobody measured it. | N | `E/hub/cmd/hub/main.go:1090` · class W4 · S16 | +| 4 | When the hub is down at 05:00, that day's check is skipped. The 'last run' value lives in memory and the next slot is the next day. Nobody notices. The next day's check uses longer limits, so a missed backup is seen one day late. | N | `E/hub/cmd/hub/main.go:1069-1112` · class W1 · S01 | +| 5 | Cannot run twice for one slot: the `last` guard in `nextDailyRun`. No lock against the 60 s sweep; they are separate goroutines and. A down box is judged on longer lines, 48 h / 72 h. | A | `E/hub/cmd/hub/main.go:1095-1112` `E/hub/cmd/hub/main.go:927` `E/hub/cmd/hub/main.go:986` `HUB/monitor/deadline.go:340-343` | +| 6 | The check compares the BOX's backup time with the hub's clock. A box whose clock runs ahead looks fresh. No guard in the file. (R-1206 is the same cause seen on the box's own pages.) | N | `HUB/monitor/deadline.go:243` · class W2 · S07 | +| 7 | Household and operator through the dispatcher on `expected_backup_missed`; at the 05:00 run only. A box that never bound is "unknown", not alarmed; disabled / deliberately-off are skipped. | A | `HUB/monitor/deadline.go:539-544` `HUB/monitor/deadline.go:486-494` `HUB/monitor/deadline.go:432-442` | +| 8 | TEST for a down box and for 'the same slot never runs twice'. A skipped day: NOBODY KNOWS. | N | `HUB/monitor/r872_down_box_test.go:57` `E/hub/cmd/hub/r1183_dst_schedule_test.go:82` · class W1 · S01 | + +**Hand step in a runbook?** NO — no runbook names a hub that was down at 05:00 + +### CH03 — Mail cooldowns and the weekly outage mail · CUSTOMER · Hub sweep · MENTIONED + +architecture folder: 38 lines in 8 files match `/cooldown/` (a machine count, not a reading) · helper section: clock-hub „3. Mail cooldowns and the weekly outage mail” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | CUSTOMER. Not a job but a clock rule inside every mail: `processCustomer` and `processOperator` decide whether an alarm is mailed now or swallowed. Owner: hub dispatcher. | A | `HUB/notify/dispatcher.go:1020-1098` `HUB/notify/dispatcher.go:626-696` | +| 2 | Operator: `operatorCooldown` 1 h, liveness 5 min, storage fill `storageFillCooldown` 6 h. Household: default 6 h; outage mails once per 7 days. First hour of a new box is suppressed, `firstHourOfABox`. | A | `HUB/notify/dispatcher.go:609-612` `HUB/notify/dispatcher.go:573` `HUB/notify/dispatcher.go:1065-1069` `HUB/notify/dispatcher.go:1054-1062` | +| 3 | Each mail call has a 10-second limit. The cooldown lists are in memory; an entry is removed when its alarm clears. | A | `HUB/notify/dispatcher.go:86` `HUB/notify/dispatcher.go:684` `HUB/notify/dispatcher.go:1182` | +| 4 | Pod restart: the cooldown maps are in memory, so quiet periods restart from zero; only the 7-day household rule is read from the persisted notification log. Design text:. | A | `HUB/notify/dispatcher.go:1054-1062` `AR/08-alarm-ladder.md:435-439` | +| 5 | The operator cooldown is armed before the send, so two alarms close together collapse to one; a suppression is recorded. A MAIL-HOLD drops the mail. | A | `HUB/notify/dispatcher.go:674` `HUB/notify/dispatcher.go:665-672` `HUB/notify/dispatcher.go:680-687` | +| 6 | Cooldowns use the hub's wall clock only. A jump back stretches a quiet period; a jump forward ends it early. Nothing handles it. | N | `HUB/notify/dispatcher.go:1` · class W2 · S10 | +| 7 | A swallowed operator alarm leaves a 'suppressed' row. A household mail that fails is logged and is not sent again; only operator mails are retried. | N | `HUB/notify/dispatcher.go:1091-1094` · class W1 · S02 | +| 8 | TEST `TestOperatorCooldown_NodeLivenessBypassesQuietHour`; TEST `TestR873_HouseholdHearsAnOutageAtMostWeekly`; DOC. | A | `HUB/notify/node_liveness_cooldown_test.go:17` `HUB/notify/r873_liveness_weekly_test.go:17` `AR/08-alarm-ladder.md:165-168` | + +**Hand step in a runbook?** no hand step + +### CH04 — Retry of a failed operator mail · OPERATOR · Hub sweep · MENTIONED + +architecture folder: 27 lines in 10 files match `/retry/` (a machine count, not a reading) · helper section: clock-hub „4. Retry of a failed operator mail” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | OPERATOR. `retryOperatorEmail` re-sends an operator mail whose send failed; started by the failure branch of `processOperator`. | A | `HUB/notify/dispatcher.go:99-104` `HUB/notify/dispatcher.go:688-692` | +| 2 | After 1, 5 and 15 minutes: `operatorRetryDelays`, armed with `afterFn = time.AfterFunc`. No setting. Durations, no time zone. | A | `HUB/notify/dispatcher.go:76` `HUB/notify/dispatcher.go:92` | +| 3 | Each try is bounded by the 10 s client timeout. One timer per failed mail; grows only with failures. | A | `HUB/notify/dispatcher.go:86` | +| 4 | The retry timers are in memory. A hub restart inside the 21 minutes loses them, and nothing sends the mail later. After the third failure the hub writes one ERROR line. | N | `HUB/notify/dispatcher.go:90-102` · class W1 · S03 | +| 5 | The cooldown for that alarm was armed before the first send, so a repeat alarm in the same hour does not mail either; the retry is the only path. Explained in. | A | `HUB/notify/dispatcher.go:674` `AR/08-alarm-ladder.md:358-363` | +| 6 | The timers are durations; the hour change has no effect. A jump of the hub's wall clock: NOBODY KNOWS. | N | class W2 · S10 | +| 7 | Nobody is mailed when the retries give up. The trace is the ERROR line and the 'failed' rows in the notification log. | N | `HUB/notify/dispatcher.go:90-102` `AR/08-alarm-ladder.md:363` · class W1 · S03 | +| 8 | TEST for the retry. A restart during the retry window: NOBODY KNOWS. | N | `HUB/notify/operator_mail_retry_test.go:17` · class W1 · S03 | + +**Hand step in a runbook?** no hand step + +### CH05 — Kernel mail: 'your box restarts tonight' · CUSTOMER · Hub sweep · MENTIONED + +architecture folder: 165 lines in 14 files match `/kernel/` (a machine count, not a reading) · helper section: clock-hub „5. Kernel "your box restarts tonight" mail” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | CUSTOMER. `KernelNotify` mails each due household the day before a kernel step; called from `Evaluate`, which the 60 s OS ticker calls. | A | `HUB/osupdates/kernel.go:191` `HUB/osupdates/service.go:436` `E/hub/cmd/hub/main.go:564` | +| 2 | Every 60 s, but acts only 09:00-20:00 Budapest: `KernelNoticeFromHour`/`KernelNoticeToHour`, checked at. Valid 24 h, gap 20 h, at most 3 mails. Zone from `budapestLoc`. Constants, no setting. | A | `HUB/osupdates/kernel.go:46-47` `HUB/osupdates/kernel.go:193-196` `HUB/osupdates/kernel.go:48-50` `HUB/osupdates/kernel.go:73-78` | +| 3 | No time limit. Loops all hosts every minute with store reads per host; grows linearly with boxes, 1440 passes a day. | A | `HUB/osupdates/kernel.go:203-208` | +| 4 | Pod down all day 09-20: no mail, and without a mail there is no step: `KernelBlockFor` sets `Tonight` only for a notice younger than 24 h. It mails on the next daytime tick; the notices are persisted. | A | `HUB/osupdates/kernel.go:179-185` `HUB/osupdates/kernel.go:236` | +| 5 | Twice is blocked by the stored notice and the 20 h gap; a restart cannot double-mail because the row is in the store. A mail sent but not recorded means no step tonight. | A | `HUB/osupdates/kernel.go:208-211` `HUB/osupdates/kernel.go:236-239` | +| 6 | Hub clock decides the daytime window. A mail sent at 19:59 is valid until 19:59 next day, so it covers one night. DST: the hour test uses Budapest local time; the 20 h / 24 h are durations. A deliberately-off box gets no mail. | A | `HUB/osupdates/kernel.go:182` `HUB/osupdates/kernel.go:193-194` `HUB/osupdates/kernel.go:201` | +| 7 | Mail failure: operator event `os_kernel_notice` warning, once per box, kernel and day. After 3 mails with no step: operator warning, once. | A | `HUB/osupdates/kernel.go:226-233` `HUB/osupdates/kernel.go:212-219` | +| 8 | TESTS for 'no mail, no step'. A mail that goes out late in the day against the box's own night slot: NOBODY KNOWS. | N | `HUB/osupdates/kernel_test.go:192` `HUB/osupdates/kernel_test.go:79` `HUB/osupdates/kernel_test.go:115` · class W3 · S14 | + +**Hand step in a runbook?** no hand step + +### CH06 — OS-update approval tick and the hourly OS alarms · OPERATOR · Hub sweep · MENTIONED + +architecture folder: 468 lines in 28 files match `/approval|ring/` (a machine count, not a reading) · helper section: clock-hub „6. OS update approval tick and hourly OS alarms” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | OPERATOR (the outcome reaches households as updates). 60 s ticker: `osSvc.Evaluate` approves a set when its rule holds; `osSvc.Alarms` raises the fleet alarms. | A | `E/hub/cmd/hub/main.go:554-586` `E/hub/cmd/hub/main.go:564` `E/hub/cmd/hub/main.go:577` | +| 2 | Evaluate every 60 s; Alarms when `time.Since(lastAlarms) >= time.Hour`. Rule default `ApproveAfter: 24 * time.Hour, NightsRequired: 1`, env `OS_APPROVE_AFTER`. Alarm limits 7 d / 14 d, env table. | A | `E/hub/cmd/hub/main.go:574` `E/hub/cmd/hub/main.go:476` `E/hub/cmd/hub/main.go:484-491` `E/hub/cmd/hub/main.go:536-541` | +| 3 | No time limit. The alarm pass logs its own duration. Evaluate walks every layer; grows with boxes times layers, every minute. | A | `E/hub/cmd/hub/main.go:581` `HUB/osupdates/service.go:414-422` | +| 4 | Pod restart: `lastAlarms` is a local variable, zero at start, so the first tick runs the alarms at once. Re-mail is prevented by the stored raise time, 7 days. | A | `E/hub/cmd/hub/main.go:558` `HUB/osupdates/service.go:1102` | +| 5 | One goroutine, so Evaluate and Alarms never overlap. `raise` clears the mark when the condition ends so the next occurrence is announced. | A | `E/hub/cmd/hub/main.go:559-585` `HUB/osupdates/service.go:1094-1108` | +| 6 | A box's claimed reboot time is allowed 1 minute of error. A larger error of the box's clock: NOBODY KNOWS. (The N100's host is 32 s behind today.) | N | `HUB/osupdates/service.go:1102` `HUB/osupdates/service.go:970` · class W2 · S07 | +| 7 | Operator, by event through the dispatcher: `os_update_stale`, `os_ring0_stalled` and others, listed in. Repeats weekly while it holds. | A | `AR/08-alarm-ladder.md:375-383` `HUB/osupdates/service.go:1092-1093` | +| 8 | TESTS and DOC. The cost of the pass each minute with 100 boxes: NOBODY KNOWS. | N | `HUB/osupdates/alarms_test.go:226` `AR/08-alarm-ladder.md:365` · class W4 · S16 | + +**Hand step in a runbook?** no hand step + +### CH07 — Off-site copy too old, and escrow never finished · CUSTOMER · Hub sweep · MENTIONED + +architecture folder: 12 lines in 6 files match `/offsite_stale|off.site.{0,20}stale|48 h/` (a machine count, not a reading) · helper section: clock-hub „7. Off-site copy too old, and escrow never finished” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | CUSTOMER. `offsiteChecker.Check` on the 60 s sweep; `isStale` and `checkEscrowPending`. Owner: hub. | A | `E/hub/cmd/hub/main.go:943` `HUB/monitor/offsite.go:166` `HUB/monitor/offsite_escrow_pending.go:66` | +| 2 | Every 60 s. Stale after `defaultOffsiteStaleAfter` 48 h, counted from the last SUCCESS. Escrow pending after `escrowPendingAfter` 7 d, re-raised each 7 d. | A | `HUB/monitor/offsite.go:54` `HUB/monitor/offsite.go:203-207` `HUB/monitor/offsite_escrow_pending.go:37` `HUB/monitor/offsite_escrow_pending.go:110` | +| 3 | No time limit. Reads every customer's latest report each minute; grows linearly with boxes. | A | `HUB/monitor/offsite.go:443-459` | +| 4 | With the hub down nothing is judged. After a start the state lists in memory are empty. If a tier that is still too old is mailed again after a restart: NOBODY KNOWS. | N | `HUB/monitor/offsite.go:463-468` `HUB/monitor/offsite.go:1` · class W1 · S03 | +| 5 | One mutex per pass; a customer listed twice is handled once. A never-ran tier is owned by the delivery checker, "one state, one owner". | A | `HUB/monitor/offsite.go:448-449` `HUB/monitor/offsite.go:453-458` `HUB/monitor/offsite.go:160-162` | +| 6 | The last-success time is the BOX's time, compared with the hub's clock. A box clock that is ahead hides an old copy. A time that cannot be read counts as too old. | N | `HUB/monitor/offsite.go:203-207` · class W2 · S07 | +| 7 | Dispatcher mail on the stale edge; `offsite_escrow_pending` is operator-only, see. | A | `HUB/monitor/offsite.go:636` `AR/08-alarm-ladder.md:383` | +| 8 | TESTS for the limits and for a newborn box. A box clock that is ahead: NOBODY KNOWS. | N | `HUB/monitor/offsite_r100_test.go:55` `HUB/monitor/offsite_r100_test.go:74` · class W2 · S07 | + +**Hand step in a runbook?** no hand step + +### CH08 — Restore-test proof too old · OPERATOR · Hub sweep · MENTIONED + +architecture folder: 121 lines in 19 files match `/restore.test/` (a machine count, not a reading) · helper section: clock-hub „8. Restore-test proof too old” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | CUSTOMER (protects the promise that a backup restores; mail goes by event). `restoreTestChecker.Check` on the 60 s sweep; `checkStaleness`. | A | `E/hub/cmd/hub/main.go:944` `HUB/monitor/restoretest.go:242` | +| 2 | Every 60 s. Window per tier: 4 archive generations, never under `restoreProvenWindowFloor` 7 d, never over `restoreProvenWindowCap` 12 d. Constants, no setting. | A | `HUB/monitor/restoretest.go:62` `HUB/monitor/restoretest.go:68` `HUB/monitor/restoretest.go:74` | +| 3 | No time limit. Each pass reads 14 days of host reports for each customer, each minute. It is the heaviest read on the sweep. Nobody measured it. | N | `HUB/monitor/restoretest.go:79` `HUB/monitor/restoretest.go:252` · class W4 · S16 | +| 4 | Pod restart: the first pass does not re-send a stale signal already in the events table, `staleAlreadyReported`. | A | `HUB/monitor/restoretest.go:281-283` `HUB/monitor/restoretest.go:300-312` | +| 5 | Edge-triggered per customer and tier under a mutex. A deliberately-off box is skipped. | A | `HUB/monitor/restoretest.go:270-278` `HUB/monitor/restoretest.go:192-194` | +| 6 | First contact is the hub's receive time. The proof times come from the box's reports. An error of the box's clock: NOBODY KNOWS. | N | `HUB/monitor/restoretest.go:326-342` `HUB/store/store.go:3994-4005` · class W2 · S07 | +| 7 | Event `restore_test_stale` warning through the dispatcher; a failed run is a separate event, `checkFailure`. | A | `HUB/monitor/restoretest.go:284-285` `HUB/monitor/restoretest.go:205` | +| 8 | TEST that a restart does not mail again. The cost with 100 boxes (14 days of reports per customer, each minute): NOBODY KNOWS. | N | `HUB/monitor/r1001_restart_test.go:147` · class W4 · S16 | + +**Hand step in a runbook?** no hand step + +### CH09 — Off-site clean-up window (20 minutes, weekly) · CUSTOMER · Hub sweep · MENTIONED + +architecture folder: 11 lines in 5 files match `/clean.?up window|prune window|deleting key/` (a machine count, not a reading) · helper section: clock-hub „9. Off-site clean-up window (20 minutes, weekly)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | CUSTOMER DATA. The box asks; `OpenWindowFor` adds a deleting key line for 20 minutes; the hub's 60 s sweep `SweepExpiredWindows` closes a window left open. Sweep loop:. | A | `HUB/offsitekeys/service.go:334` `HUB/offsitekeys/service.go:413` `E/hub/cmd/hub/main.go:636-650` | +| 2 | Granted when the last window is at least `windowCadence` 6 d 12 h old, test at; length `windowLength` 20 min. Sweep every 60 s. Constants; the weekly switch is a store setting. | A | `HUB/offsitekeys/service.go:264` `HUB/offsitekeys/service.go:337` `HUB/offsitekeys/service.go:263` `HUB/offsitekeys/service.go:338` | +| 3 | Each sweep has a 2-minute context. Work is per expired window, one remote call each; grows with boxes that crash mid-prune. | A | `E/hub/cmd/hub/main.go:644` `HUB/offsitekeys/service.go:419-422` | +| 4 | If the hub stops while a window is open, the key line that can delete stays on the storage account until the hub is back; the sweep then closes it. Nobody is told during the gap. | N | `HUB/offsitekeys/service.go:419-431` `HUB/offsitekeys/service.go:424-426` · class W1 · S01 | +| 5 | One window per cadence via the ledger row. The daily key audit knows an open window. No lock against the abandon sweep; both run in the same goroutine. | A | `HUB/offsitekeys/service.go:364` `HUB/offsitekeys/service.go:212-213` `E/hub/cmd/hub/main.go:636-650` | +| 6 | The window runs on the hub's clock; the box's clock does not decide. A hub clock that jumps back can keep a window open longer: NOBODY KNOWS. | N | `HUB/offsitekeys/service.go:337` · class W2 · S10 | +| 7 | Operator event `offsite_window_failed` warning when the hub had to close it; `offsite_window_drop` error when too much went. | A | `HUB/offsitekeys/service.go:430-431` `HUB/offsitekeys/service.go:259` | +| 8 | TEST `TestWindow_LeftOpenIsClosedByTheSweep`; TEST `TestWindow_GrantOpenCloseAndDropAlarm`; DOC. | A | `HUB/offsitekeys/service_test.go:99` `HUB/offsitekeys/service_test.go:37` `AR/07-backup-architecture.md:256` | + +**Hand step in a runbook?** no hand step + +### CH10 — Delayed deletion of a set-aside off-site copy · CUSTOMER · Hub sweep · MENTIONED + +architecture folder: 51 lines in 16 files match `/set.aside|delayed delet|superseded/` (a machine count, not a reading) · helper section: clock-hub „10. Delayed deletion of a set-aside off-site copy” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | CUSTOMER DATA. `SweepAbandons` deletes a household's set-aside copy once its request is due; called from the 60 s sweep. | A | `HUB/offsitekeys/service.go:529` `E/hub/cmd/hub/main.go:636-650` | +| 2 | Due `DefaultAbandonDelay` 7 days after the request; override `OFFSITE_ABANDON_DELAY`. Checked every 60 s. Hub clock. | A | `HUB/offsitekeys/service.go:448` `E/hub/cmd/hub/main.go:598-605` | +| 3 | One remote delete for each due request, in the same sweep as the window. How long a large delete takes: NOBODY KNOWS. | N | `E/hub/cmd/hub/main.go:644` `HUB/offsitekeys/service.go:536-538` · class W4 · S15 | +| 4 | Pod down at the due time: deleted on the first sweep after start; the due list is in the store. A failed delete is marked and retried every sweep. | A | `HUB/offsitekeys/service.go:530` `HUB/offsitekeys/service.go:540-547` | +| 5 | A request is deleted once, then marked. A cancel before the due time wins: design text. | A | `HUB/offsitekeys/service.go:549` `HUB/offsitekeys/service.go:438-441` | +| 6 | The hub's clock only. The length of the wait is one setting, read at the start. | A | `E/hub/cmd/hub/main.go:598-605` | +| 7 | Operator events: requested, cancelled, deleted, failed; failure mails once per request. | A | `HUB/offsitekeys/service.go:444-447` `HUB/offsitekeys/service.go:543-546` | +| 8 | TEST `TestAbandon_DelayCancelAndScope`. | A | `HUB/offsitekeys/service_test.go:134` | + +**Hand step in a runbook?** no hand step + +### CH11 — Off-site key audit (07:10) · OPERATOR · Hub daily · MENTIONED + +architecture folder: 2 lines in 2 files match `/key audit|07:10/` (a machine count, not a reading) · helper section: clock-hub „11. Off-site key audit (07:10)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | OPERATOR (guards household copies against deletion). `scheduleDaily(ctx, "offsite-key-audit", "07:10",...)` runs `AuditAll`. | A | `E/hub/cmd/hub/main.go:651-655` `HUB/offsitekeys/service.go:195` | +| 2 | Daily 07:10 Budapest, literal slot. Zone from `scheduleDaily`. | A | `E/hub/cmd/hub/main.go:651` `E/hub/cmd/hub/main.go:1072` | +| 3 | The audit has a 10-minute limit and reads one customer after the other from the storage account. With 100 boxes the 10 minutes may not be enough; nobody measured it. | N | `E/hub/cmd/hub/main.go:652` `HUB/offsitekeys/service.go:202-215` · class W4 · S16 | +| 4 | When the hub is down at 07:10, that day's audit is skipped. No catch-up. Nobody notices. | N | `E/hub/cmd/hub/main.go:1069-1112` · class W1 · S01 | +| 5 | Runs once per day by the `last` guard. Meets an open clean-up window: the audit is told the window is open. | A | `E/hub/cmd/hub/main.go:1095-1112` `HUB/offsitekeys/service.go:212-213` | +| 6 | Wall-clock slot, DST-safe by `nextDailyRun`. | A | `E/hub/cmd/hub/main.go:1095-1112` | +| 7 | Operator: `offsite_key_unlocked` error, or `offsite_key_audit_failed` when a file cannot be read. | A | `HUB/offsitekeys/service.go:192-194` | +| 8 | TEST and DOC. The 10-minute limit at fleet size: NOBODY KNOWS. | N | `HUB/offsitekeys/offsitekeys_test.go:158` `AR/05-hub-architecture.md:642` · class W4 · S16 | + +**Hand step in a runbook?** no hand step + +### CH12 — Off-site credential stuck, and its self-heal · OPERATOR · Hub sweep · MENTIONED + +architecture folder: 4 lines in 3 files match `/offsiteheal|credential.{0,30}(stuck|heal)/` (a machine count, not a reading) · helper section: clock-hub „12. Off-site credential stuck and its self-heal” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | CUSTOMER. Two jobs, merged: `offsiteDeliveryChecker.Check` (60 s sweep) and the `offsiteheal` reconciler, started at. | A | `E/hub/cmd/hub/main.go:945` `HUB/offsiteheal/reconciler.go:165-177` `E/hub/cmd/hub/main.go:846-854` | +| 2 | Stuck after `stuckAfter` 1 h; cooldowns 24 h. Heal sweep every `tickIntervalDefault` 5 min, acts after 2 distinct reports. | A | `HUB/monitor/offsite_delivery.go:38` `HUB/monitor/offsite_delivery.go:40-41` `HUB/offsiteheal/reconciler.go:74` `HUB/offsiteheal/reconciler.go:70` | +| 3 | Heal call limit 3 minutes. Both list every customer config per pass and; linear in boxes. | A | `HUB/monitor/offsite_delivery.go:275` `HUB/monitor/offsite_delivery.go:72` `HUB/offsiteheal/reconciler.go:183` | +| 4 | Pod down: nothing; on return the next tick re-reads state. The cooldowns are "durable via the events table". | A | `HUB/monitor/offsite_delivery.go:39` | +| 5 | Two healers could meet: the checker re-reads the secret row immediately before acting, and a box declaration is owned by offsiteheal. | A | `HUB/monitor/offsite_delivery.go:28-31` `HUB/monitor/offsite_delivery.go:97-99` | +| 6 | Age counted from the hub-held `status.Since`. Debounce is counted in reports, not ticks. | A | `HUB/monitor/offsite_delivery.go:93` `HUB/offsiteheal/reconciler.go:61-69` | +| 7 | Operator events `offsite_delivery_stuck`, `offsite_credential_restaged`; `offsite_selfheal_restaged` / `_reissued`. | A | `HUB/monitor/offsite_delivery.go:34-35` `HUB/offsiteheal/reconciler.go:57-58` | +| 8 | SOURCE and DOC only. No test name for the timing was found. | A | `HUB/offsiteheal/reconciler.go:61-74` `AR/06-offsite-connectivity.md:386` `HUB/monitor/offsite_delivery.go:1` | + +**Hand step in a runbook?** no hand step + +### CH13 — PBS whole-box credential self-heal · OPERATOR · Hub sweep · MENTIONED + +architecture folder: 9 lines in 5 files match `/pbsdrheal|PBS.{0,30}heal/` (a machine count, not a reading) · helper section: clock-hub „13. PBS whole-box credential self-heal” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | CUSTOMER. `pbsdrheal` reconciler, started at; repairs a box whose off-site PBS credential did not land. | A | `HUB/pbsdrheal/reconciler.go:153-165` `E/hub/cmd/hub/main.go:829-837` | +| 2 | Every 5 minutes, or on a trigger; acts after 2 reports. A staged secret unconsumed for `unconsumedGrace` 15 min is flagged. | A | `HUB/pbsdrheal/reconciler.go:130` `HUB/pbsdrheal/reconciler.go:160` `HUB/pbsdrheal/reconciler.go:131` `HUB/pbsdrheal/reconciler.go:63` | +| 3 | No time limit on a pass, `reconcileOnce(ctx)`. Reads all heal states; linear in boxes. | A | `HUB/pbsdrheal/reconciler.go:163` `HUB/pbsdrheal/reconciler.go:170` | +| 4 | Pod down: no heal; the debounce maps are in memory, so after a restart a box waits two more reports. | A | `HUB/pbsdrheal/reconciler.go:134-135` | +| 5 | One goroutine; a read error does nothing this tick. | A | `HUB/pbsdrheal/reconciler.go:171-174` | +| 6 | Grace is hub time. Box clock: not used here. | A | `HUB/pbsdrheal/reconciler.go:153-165` `E/hub/cmd/hub/main.go:829-837` (the code path of question 1) | +| 7 | Operator audit events `pbsdr_selfheal_*` and `pbsdr_unconsumed_secret`. | A | `HUB/pbsdrheal/reconciler.go:50-54` | +| 8 | SOURCE only. A pass that hangs on the remote side: NOBODY KNOWS; the pass has no time limit. | N | `HUB/pbsdrheal/reconciler.go:57-63` · class W4 · S15 | + +**Hand step in a runbook?** no hand step + +### CH14 — WireGuard peer list push to ep0 · OPERATOR · Hub sweep · MENTIONED + +architecture folder: 8 lines in 3 files match `/wgsync|peer list/` (a machine count, not a reading) · helper section: clock-hub „14. WireGuard peer list push to ep0” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | OPERATOR (keeps every box's tunnel allowed). `wgReconciler.Run`, loop; pushes the FULL peer list over SSH. | A | `E/hub/cmd/hub/main.go:752` `HUB/wgsync/reconciler.go:90-106` | +| 2 | Every 5 minutes and on every change via `Trigger`. No setting. | A | `HUB/wgsync/reconciler.go:41` `HUB/wgsync/reconciler.go:48-53` | +| 3 | 60 s limit per push. The payload grows with peers. | A | `HUB/wgsync/reconciler.go:100` `HUB/wgsync/reconciler.go:71-73` | +| 4 | With the hub down ep0 keeps the last list. A push that fails is logged and tried again at the next tick. Nobody is mailed when it fails again and again; a new box then cannot reach the off-site store. | N | `HUB/wgsync/reconciler.go:100-102` · class W1 · S05 | +| 5 | One loop; a pending trigger covers later ones. Full list each time erases drift. | A | `HUB/wgsync/reconciler.go:46-47` `HUB/wgsync/reconciler.go:20-23` | +| 6 | No clock in the payload. | A | `E/hub/cmd/hub/main.go:752` `HUB/wgsync/reconciler.go:90-106` (the code path of question 1) | +| 7 | Log line only. | A | `HUB/wgsync/reconciler.go:102` | +| 8 | TEST `TestReconciler_FailureRetriedOnTick`; TEST `TestReconciler_TriggerPushesFullList`. | A | `HUB/wgsync/reconciler_test.go:119` `HUB/wgsync/reconciler_test.go:82` | + +**Hand step in a runbook?** no hand step + +### CH15 — Box clock ran ahead: the tunnel alarm · OPERATOR · Hub sweep · MENTIONED + +architecture folder: 13 lines in 4 files match `/wg_tunnel|stamped in the future|clock.ahead/` (a machine count, not a reading) · helper section: clock-hub „15. Box clock ran ahead: tunnel alarm” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | OPERATOR (the household's whole-box copies stop). `checkWGTunnel`, called on every host report; not a timer, it runs when the box reports. | A | `HUB/api/host_tunnel.go:61` `HUB/api/handler.go:1010` | +| 2 | Fires when `handshake_future_s` > 0 or the last handshake is older than `wgTunnelStaleAfterS` 6 h. Constant. | A | `HUB/api/host_tunnel.go:41` `HUB/api/host_tunnel.go:23` | +| 3 | One store read per report. | A | `HUB/api/host_tunnel.go:66` | +| 4 | If the box stops reporting, this check never runs; block 1 covers that. An older agent that sends neither field is not judged. | A | `HUB/api/host_tunnel.go:62-64` | +| 5 | Edge-triggered from the newest stored tunnel event, so a restart does not re-mail. | A | `HUB/api/host_tunnel.go:71-83` | +| 6 | This IS the clock-ahead handling: the future stamp is measured by the box's own clock; repair is a new key after 15 minutes on the agent side. Known gaps are listed in. | A | `HUB/api/host_tunnel.go:31` `HUB/api/host_tunnel.go:14-17` `AR/05-hub-architecture.md:489-490` | +| 7 | Operator only, `wg_tunnel_stale` warning; `wg_tunnel_ok` is stored, not mailed. | A | `HUB/api/host_tunnel.go:12-14` | +| 8 | TEST `TestD30_TheAlarmClearsWhenTheNewKeyShakesHands`; DOC. | A | `HUB/api/host_tunnel_test.go:83` `AR/05-hub-architecture.md:482-490` | + +**Hand step in a runbook?** no hand step + +### CH16 — The daily scheduler and the hour change · OPERATOR · Hub daily · MENTIONED + +architecture folder: 2 lines in 1 files match `/scheduleDaily|nextDailyRun|D24/` (a machine count, not a reading) · helper section: clock-hub „16. The daily scheduler and the clock change” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | OPERATOR (mechanism under blocks 2, 11, 17, 18). `scheduleDaily` with `nextDailyRun`. | A | `E/hub/cmd/hub/main.go:1069-1093` `E/hub/cmd/hub/main.go:1095-1112` | +| 2 | Budapest wall-clock time. If the zone cannot be loaded, the fallback is fixed CET, which is one hour off in summer. The pod's zone is set to Europe/Budapest. | A | `E/hub/cmd/hub/main.go:1072` | +| 3 | Sleeps with `time.After(delay)`; the job runs inline, so a long job delays only its own next slot. | A | `E/hub/cmd/hub/main.go:1088` `E/hub/cmd/hub/main.go:1090` | +| 4 | A slot that is missed while the hub is down is not made up; the snapshot and the prune are the two exceptions. A time setting that cannot be read becomes 00:00 with no message. | N | `E/hub/cmd/hub/main.go:1069-1112` · class W1 · S01 | +| 5 | Next run is the calendar day after and the slot that ran last never runs again. | A | `AR/05-hub-architecture.md:642-645` | +| 6 | Days of 23 and 25 hours are handled (D24, with tests). While the scheduler sleeps, a jump of the hub's clock does not move the timer: NOBODY KNOWS what then runs when. | N | `AR/05-hub-architecture.md:640-646` · class W2 · S10 | +| 7 | Nobody; start-up logs only. | A | `E/hub/cmd/hub/main.go:1069-1093` `E/hub/cmd/hub/main.go:1095-1112` (the code path of question 1) | +| 8 | TEST `TestR1183_DailyJobRunsOnceOnTheDayTheClocksGoBack`; TEST `TestR1183_DailyJobRunsOnceOnTheDayTheClocksGoForward`. | A | `E/hub/cmd/hub/r1183_dst_schedule_test.go:31` `E/hub/cmd/hub/r1183_dst_schedule_test.go:56` | + +**Hand step in a runbook?** no hand step + +### CH17 — Hub database snapshot (four slots) · OPERATOR · Hub daily · MENTIONED + +architecture folder: 236 lines in 18 files match `/snapshot/` (a machine count, not a reading) · helper section: clock-hub „17. Hub database snapshot (four slots)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | OPERATOR (the hub holds every household's escrow and settings). `runSnapshot` scheduled over `dbsnap.Slots`; `Make`. | A | `E/hub/cmd/hub/main.go:967-983` `HUB/dbsnap/dbsnap.go:72-117` | +| 2 | 02:00, 08:00, 14:00, 20:00 Budapest: `Slots`. Keeps `Keep = 2`. File name carries UTC. | A | `HUB/dbsnap/dbsnap.go:32` `HUB/dbsnap/dbsnap.go:27` `HUB/dbsnap/dbsnap.go:89` | +| 3 | 10-minute limit. Time grows with database size; written to a tmp file then renamed; duration is logged. | A | `E/hub/cmd/hub/main.go:972` `HUB/dbsnap/dbsnap.go:91-104` `HUB/dbsnap/dbsnap.go:113-114` | +| 4 | The one job WITH catch-up: at start, if the newest snapshot is older than `CatchUpAge` 6 h, one is made, `NeedsCatchUp`. | A | `E/hub/cmd/hub/main.go:978-980` `HUB/dbsnap/dbsnap.go:148-159` | +| 5 | A running flag refuses a second run with `ErrBusy`. The DooPlex push timer runs 30 min later and waits for the tmp file (block 26). | A | `HUB/dbsnap/dbsnap.go:73-80` | +| 6 | The slots are safe on the hour-change night. The design text still says 'older than 24 h' and '02:30' in its first paragraph; the source has four slots. | N | `AR/05-hub-architecture.md:618-620` · class W5 · S18 | +| 7 | Failure is a log line; the outside alarm is the push timer's (block 26). | A | `E/hub/cmd/hub/main.go:967-983` `HUB/dbsnap/dbsnap.go:72-117` (the code path of question 1) | +| 8 | TEST `TestD34_SlotsAreEverySixHours`; TEST `TestR173_MainSchedulesTheDBSnapshot`. | A | `HUB/dbsnap/dbsnap_test.go:214` `E/hub/cmd/hub/r173_wiring_test.go:14` | + +**Hand step in a runbook?** YES — `RB/RUNBOOK-hub-db-offsite-backup.md` + +### CH18 — Daily prune of old hub rows (04:30) · OPERATOR · Hub daily · MENTIONED + +architecture folder: 54 lines in 12 files match `/prune/` (a machine count, not a reading) · helper section: clock-hub „18. Daily prune of old hub rows (04:30)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | OPERATOR. `pruneAll`, at start and by `scheduleDaily(ctx, "prune",...)`. | A | `E/hub/cmd/hub/main.go:1123-1150` `E/hub/cmd/hub/main.go:860-865` | +| 2 | Daily at `prune_schedule`, default `"04:30"`, deployed; keeps `max_days` 90. Also app telemetry 90 d, stale issues 30 d, deleted-customer audit 1 year. | A | `E/hub/cmd/hub/main.go:1026` `E/manifests/hub.yaml:83` `E/manifests/hub.yaml:82` `E/hub/cmd/hub/main.go:1134` | +| 3 | No time limit and no duration in the log. The deletes grow with the reports of each box each day. | N | `E/hub/cmd/hub/main.go:1123-1150` · class W4 · S15 | +| 4 | Runs at every start, so a missed slot is caught up by the next restart; with `MaxDays` 0 it never runs. | A | `E/hub/cmd/hub/main.go:860-865` | +| 5 | May meet the 05:00 deadline check or a snapshot only if it runs over 30 min; no lock found. | A | `E/hub/cmd/hub/main.go:1123-1150` `E/hub/cmd/hub/main.go:860-865` (the code path of question 1) | +| 6 | One cut-off is made from the pod's local time and compared with a UTC column; the other uses UTC. The error is 1 or 2 hours on 90 days. | N | `HUB/store/store.go:1607` `HUB/store/store.go:129` `HUB/store/store.go:2873` · class W2 · S10 | +| 7 | Log only. | A | `E/hub/cmd/hub/main.go:1123-1150` `E/hub/cmd/hub/main.go:860-865` (the code path of question 1) | +| 8 | SOURCE only. No test for the cut-off's time zone was found. | A | `E/hub/cmd/hub/main.go:1123-1150` `HUB/store/store.go:1606` | + +**Hand step in a runbook?** no hand step + +### CH19 — The other checks on the 60-second sweep · OPERATOR · Hub sweep · MENTIONED + +architecture folder: 59 lines in 16 files match `/60.second|sweep/` (a machine count, not a reading) · helper section: clock-hub „19. The other checks on the 60-second sweep” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | CUSTOMER and OPERATOR mixed. Merged: host capability, host leaf, host disk, storage fill, management plane, OOB, controller supervisor. All run in the one sweep goroutine. | A | `E/hub/cmd/hub/main.go:937-946` `E/hub/cmd/hub/main.go:927-965` | +| 2 | Every 60 s in the fixed order of. A backup-cadence refusal repeats every `refusalRepeatEvery` 24 h. | A | `E/hub/cmd/hub/main.go:935-962` `HUB/monitor/host_capability.go:72` | +| 3 | All checkers run one after the other in one loop, with no limit for each. A slow checker delays all the others, the liveness check too. Each reads the whole fleet at each pass. | N | `E/hub/cmd/hub/main.go:935-962` · class W4 · S15 | +| 4 | Pod down: nothing. After start, band states are rebuilt from the latest reports. | A | `E/hub/cmd/hub/main.go:937-946` `E/hub/cmd/hub/main.go:927-965` (the code path of question 1) | +| 5 | One goroutine, so no overlap; a Go ticker drops ticks when a pass overruns. | A | `E/hub/cmd/hub/main.go:937-946` `E/hub/cmd/hub/main.go:927-965` (the code path of question 1) | +| 6 | They judge the latest report's content, not time, except the 24 h repeat. | A | `HUB/monitor/host_capability.go:172` | +| 7 | Dispatcher mails; storage fill has its own 6 h grain. | A | `AR/08-alarm-ladder.md:305` | +| 8 | SOURCE only. How long one whole sweep takes: NOBODY KNOWS. | N | `E/hub/cmd/hub/main.go:927-965` · class W4 · S16 | + +**Hand step in a runbook?** no hand step + +### CH20 — Reminder about a held app · CUSTOMER · Hub sweep · MENTIONED + +architecture folder: 32 lines in 7 files match `/held.app|held app/` (a machine count, not a reading) · helper section: clock-hub „20. Reminder about a held app” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | OPERATOR (D53: no household mail). `heldApps.Check`; `HeldAppsChecker.Check`. | A | `E/hub/cmd/hub/main.go:947` `HUB/monitor/held_apps.go:102` | +| 2 | One pass per `heldAppCheckEvery` 15 min. First mail after a grace of 6 h / 72 h / 24 h by reason; then every `heldAppRepeat` 7 d. | A | `HUB/monitor/held_apps.go:36` `HUB/monitor/held_apps.go:40-48` `HUB/monitor/held_apps.go:34` | +| 3 | One list of all held apps. | A | `HUB/monitor/held_apps.go:114` | +| 4 | Pod down: the reminder comes on the first pass after start; the reminded-at time is stored BEFORE sending, so a restart cannot mail twice. | A | `HUB/monitor/held_apps.go:141-145` | +| 5 | A row the box has not confirmed for `heldAppStale` 50 h is not reminded. | A | `HUB/monitor/held_apps.go:125-129` | +| 6 | The reminder counts from a time that the box reports. A box clock that is behind makes the reminder early; one that is ahead makes it late. | N | `HUB/monitor/held_apps.go:130-134` `HUB/monitor/held_apps.go:22` · class W2 · S07 | +| 7 | Operator, `app_held_reminder` warning. | A | `HUB/monitor/held_apps.go:157-163` | +| 8 | TEST file named in source; DOC. | A | `HUB/monitor/held_apps.go:31` `AR/08-alarm-ladder.md:655` | + +**Hand step in a runbook?** no hand step + +### CH21 — 'A golden is published but not vouched' · OPERATOR · Hub sweep · MENTIONED + +architecture folder: 57 lines in 17 files match `/vouch/` (a machine count, not a reading) · helper section: clock-hub „21. "A golden is published but not vouched"” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | OPERATOR. `goldenVouch` check on the sweep; `GoldenVouchChecker.Check`. | A | `E/hub/cmd/hub/main.go:948-950` `HUB/monitor/golden_vouch.go:54` | +| 2 | One Gitea listing per `goldenVouchCheckEvery` 15 min; alarm after `goldenVouchGrace` 2 h. | A | `HUB/monitor/golden_vouch.go:42-43` | +| 3 | Listing limit `goldenVouchListDeadline` 20 s; it runs inside the sweep, so a slow Gitea holds the sweep for up to 20 s. | A | `HUB/monitor/golden_vouch.go:44` `HUB/monitor/golden_vouch.go:62-64` | +| 4 | Pod restart: the gap clock and the alerted mark are in memory, so the 2 h starts again and the alarm is mailed again ("one alarm per unvouched version per process"). | A | `HUB/monitor/golden_vouch.go:34-37` `HUB/monitor/golden_vouch.go:23` | +| 5 | Gitea unreachable or empty list: no verdict, never an alarm. | A | `HUB/monitor/golden_vouch.go:65-67` | +| 6 | Hub clock only. | A | `E/hub/cmd/hub/main.go:948-950` `HUB/monitor/golden_vouch.go:54` (the code path of question 1) | +| 7 | Operator event `golden_not_vouched`. | A | `HUB/monitor/golden_vouch.go:95-96` | +| 8 | TEST `TestGoldenVouchChecker` named at. | A | `HUB/monitor/golden_vouch.go:24` | + +**Hand step in a runbook?** no hand step + +### CH22 — Shared storage boxes: fill and reachability · OPERATOR · Hub sweep · MENTIONED + +architecture folder: 37 lines in 9 files match `/storage box|pool.box|oversub/` (a machine count, not a reading) · helper section: clock-hub „22. Shared storage boxes: fill and reachability” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | OPERATOR (every household's off-site space). `offsiteBox` and `pbsdrBox` checks. | A | `E/hub/cmd/hub/main.go:951-956` | +| 2 | On the 60 s sweep, but one provider read per `boxFetchInterval` 15 min, test at and. Warn 80 %, critical 90 %. | A | `HUB/monitor/offsite_box.go:72` `HUB/monitor/offsite_box.go:132` `HUB/monitor/pbsdr_box.go:130` `HUB/monitor/offsite_box.go:67-68` | +| 3 | Provider client limit 30 s. Does not grow with boxes. | A | `HUB/hetznerapi/hetznerapi.go:143-145` | +| 4 | Pod down: no reading. Unreachable is declared after consecutive failed 15-minute windows. | A | `HUB/monitor/pbsdr_box.go:80-82` | +| 5 | Throttled by `lastFetch` in memory; first pass after a restart fetches at once. | A | `HUB/monitor/offsite_box.go:132` | +| 6 | Hub clock only. | A | `E/hub/cmd/hub/main.go:951-956` (the code path of question 1) | +| 7 | Operator events for fill, oversubscription, unreachable and recovered. | A | `HUB/monitor/offsite_box.go:266-330` | +| 8 | SOURCE. | A | `HUB/monitor/offsite_box.go:70-72` | + +**Hand step in a runbook?** no hand step + +### CH23 — DooPlex: push of the hub database snapshot off-site · OPERATOR · DooPlex and ep0 · MENTIONED + +architecture folder: 33 lines in 8 files match `/hub.db|database snapshot/` (a machine count, not a reading) · helper section: clock-hub „26. DooPlex: push of the hub database snapshot off-site” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | NOT MEASURED. OPERATOR. systemd timer `felhom-hub-db-backup.timer` on DooPlex, runs the script E/scripts/hub-db-backup/felhom-hub-db-backup:19; installed by hand per. | A | `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10` `RB/RUNBOOK-hub-db-offsite-backup.md:107-108` | +| 2 | NOT MEASURED (DooPlex and ep0 are outside this run). 02:30, 08:30, 14:30 and 20:30 with up to 2 minutes of random delay, in the host's local time (the unit names no zone). | A | `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10` `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:12` | +| 3 | `TimeoutStartSec=45min`. Waits up to `WAIT_S` 1200 s for the hub's tmp file E/scripts/hub-db-backup/felhom-hub-db-backup:20, loop E/scripts/hub-db-backup/felhom-hub-db-backup:35-40. Grows with database size. | A | `E/scripts/hub-db-backup/felhom-hub-db-backup.service:13` | +| 4 | `Persistent=true`: a slot missed while DooPlex was off runs at boot. It refuses a snapshot older than `MAX_AGE_H` 7 E/scripts/hub-db-backup/felhom-hub-db-backup:19, line E/scripts/hub-db-backup/felhom-hub-db-backup:50. | A | `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:11` | +| 5 | No flock in the script (grep `flock/lock` returned nothing); systemd does not start a oneshot unit twice. A catch-up run right after boot can meet a hub that has not made its catch-up snapshot yet: history of a false failure in E/scripts/hub-db-backup/felhom-hub-db-backup:15-18. | A | `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10` `RB/RUNBOOK-hub-db-offsite-backup.md:107-108` (the code path of question 1) | +| 6 | NOT MEASURED (DooPlex and ep0 are outside this run). The timer's zone and the hub's zone must match for the 30-minute offset. On the autumn hour-change day the 02:00 copy never reaches ep0 and the 20:00 copy is pushed twice. | K | `E/scripts/hub-db-backup/test_hub_db_backup.py:309-316` · R-1293 | +| 7 | The runbook says in one place that the alarm fires after 13 hours, and the alarm rule printed in the same runbook says 26 hours. | N | `RB/RUNBOOK-hub-db-offsite-backup.md:100` `RB/RUNBOOK-hub-db-offsite-backup.md:152-153` · class W5 · S18 | +| 8 | NOT MEASURED (DooPlex and ep0 are outside this run). A test file for the script exists and is run by hand. The live alarm rule is in another repository and on DooPlex; this run did not read it. | A | `RB/RUNBOOK-hub-db-offsite-backup.md:107-108` `E/scripts/hub-db-backup/test_hub_db_backup.py:309-316` | + +**Hand step in a runbook?** YES — `RB/RUNBOOK-hub-db-offsite-backup.md` + +### CH24 — DooPlex: Sunday restore tests · OPERATOR · DooPlex and ep0 · MENTIONED + +architecture folder: 121 lines in 19 files match `/restore.test/` (a machine count, not a reading) · helper section: clock-hub „27. DooPlex: Sunday restore tests” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | NOT MEASURED. OPERATOR. Two timers, merged: `felhom-hub-db-restore-test.timer` and `felhom-dooplex-offsite-restore-test.timer`. | A | `E/scripts/hub-db-backup/felhom-hub-db-restore-test.timer:6` `E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test.timer:6` | +| 2 | Sunday 04:30 and Sunday 05:30, host local time; `Persistent=true`. | A | `E/scripts/hub-db-backup/felhom-hub-db-restore-test.timer:7` | +| 3 | Limits 45 min and 90 min. Grow with database and dump size. | A | `E/scripts/hub-db-backup/felhom-hub-db-restore-test.service:13` `E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test.service:13` | +| 4 | Missed Sunday runs at next boot (Persistent). Each refuses a copy older than `MAX_AGE_H` 50 E/scripts/hub-db-backup/felhom-hub-db-restore-test:13 and E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test:14. | A | `E/scripts/hub-db-backup/felhom-hub-db-restore-test.timer:6` `E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test.timer:6` (the code path of question 1) | +| 5 | NOT MEASURED (DooPlex and ep0 are outside this run). Sunday 04:30 is between the 02:00 snapshot push and the 05:00 PBS pull. The script takes no lock against them. | N | class W3 · S14 | +| 6 | NOT MEASURED (DooPlex and ep0 are outside this run). On the last Sunday of October the hour 02:00 to 03:00 comes twice. What the timer does at 04:30 that day: NOBODY KNOWS. | N | class W2 · S20 | +| 7 | Staleness alarm `HubDBRestoreTestStale` after 8 days. | A | `RB/RUNBOOK-hub-db-offsite-backup.md:158` | +| 8 | NOT MEASURED (DooPlex and ep0 are outside this run). DOC only. This run did not read if the alarm rule is deployed. | A | `RB/RUNBOOK-hub-db-offsite-backup.md:152-161` | + +**Hand step in a runbook?** YES — `RB/RUNBOOK-hub-db-offsite-backup.md` + +### CH25 — DooPlex: nightly off-site push and the release copy · OPERATOR · DooPlex and ep0 · MENTIONED + +architecture folder: 7 lines in 2 files match `/release.copy|release copy/` (a machine count, not a reading) · helper section: clock-hub „28. DooPlex: nightly off-site push and release copy” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | NOT MEASURED. OPERATOR. Merged: `felhom-dooplex-offsite.timer` and `felhom-release-copy.timer`. | A | `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:6` `E/scripts/release-copy/felhom-release-copy.timer:7` | +| 2 | 00:20 and 01:40 host local time; the order is stated in a comment. Random delay 2 min and 5 min,. | A | `E/scripts/release-copy/felhom-release-copy.timer:6` `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:8` `E/scripts/release-copy/felhom-release-copy.timer:9` | +| 3 | NOT MEASURED (DooPlex and ep0 are outside this run). The units have limits of 90 minutes and 2 hours. A 90-minute run from 00:20 ends at 01:50, after the 01:40 start of the next job. No lock between them. | N | `E/scripts/dooplex-offsite/felhom-dooplex-offsite.service:13` `E/scripts/release-copy/felhom-release-copy.service:12` · class W3 · S14 | +| 4 | Both `Persistent=true`,. The push refuses a dump older than `DUMP_MAX_AGE_H` 7 E/scripts/dooplex-offsite/felhom-dooplex-offsite:23, message E/scripts/dooplex-offsite/felhom-dooplex-offsite:55. | A | `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:7` `E/scripts/release-copy/felhom-release-copy.timer:8` | +| 5 | The push depends on a dump CronJob that is not in this clone (named at E/scripts/dooplex-offsite/felhom-dooplex-offsite:55). One retry E/scripts/dooplex-offsite/felhom-dooplex-offsite:66. | A | `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:6` `E/scripts/release-copy/felhom-release-copy.timer:7` (the code path of question 1) | +| 6 | 00:20 and 01:40 are outside the 02:00-03:00 change hour. | A | `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:6` `E/scripts/release-copy/felhom-release-copy.timer:7` (the code path of question 1) | +| 7 | `OnFailure` mail on both, a metric file is written E/scripts/dooplex-offsite/felhom-dooplex-offsite:143-153. | A | `E/scripts/dooplex-offsite/felhom-dooplex-offsite.service:6` `E/scripts/release-copy/felhom-release-copy.service:6` | +| 8 | NOT MEASURED (DooPlex and ep0 are outside this run). SOURCE only. The release-copy timer is switched on by hand; this run did not read if it is on. | A | `E/scripts/release-copy/install.sh:4` | + +**Hand step in a runbook?** no hand step + +### CH26 — DooPlex and ep0: PBS copy, prune, verify, clean-up · OPERATOR · DooPlex and ep0 · MENTIONED + +architecture folder: 124 lines in 17 files match `/ep0|PBS.{0,20}(prune|sync|pull)/` (a machine count, not a reading) · helper section: clock-hub „29. DooPlex and ep0: PBS copy, prune, verify, clean-up” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | NOT MEASURED. OPERATOR (holds the second copy of every household's whole-box backup). PBS jobs described in, plus `felhom-ep0-copy-gc.timer`. | A | `RB/ep0-datastore-copy.md:15-18` `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:5` | +| 2 | ep0 prune 03:30, pull to DooPlex 05:00; DooPlex prune 07:30; copy clean-up 08:00; verify Saturday 06:30; GC Sunday 08:30; hub-DB prune on ep0 03:45. | A | `RB/ep0-datastore-copy.md:15` `RB/ep0-datastore-copy.md:17` `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:5` `RB/ep0-datastore-copy.md:16` | +| 3 | NOT MEASURED (DooPlex and ep0 are outside this run). The PBS jobs are in PBS's own config, not in the repository. They grow with the backup size of every box. | A | `RB/ep0-datastore-copy.md:19` | +| 4 | The gc timer is `Persistent=true`. PBS jobs: catch-up behaviour not in repo. | A | `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:6` | +| 5 | NOT MEASURED (DooPlex and ep0 are outside this run). The order is by the clock only: prune 07:30, clean-up 08:00. A pull that runs past 07:30 meets the prune: NOBODY KNOWS what then happens. | N | `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:2` · class W3 · S14 | +| 6 | NOT MEASURED (DooPlex and ep0 are outside this run). Two machines, two clocks. 'The ep0 prune at 03:30 is before the 05:00 pull' is true only while both clocks and both zones agree. No document gives their zones. | N | `RB/ep0-datastore-copy.md:1` · class W3 · S14 | +| 7 | NOT MEASURED (DooPlex and ep0 are outside this run). PBS has its own notification target. The clean-up unit has no failure hook and no time limit; a clean-up that fails tells nobody. | N | `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.service:9` `RB/ep0-datastore-copy.md:19` · class W1 · S01 | +| 8 | DOC. | A | `RB/ep0-datastore-copy.md:15-19` | + +**Hand step in a runbook?** YES — `RB/ep0-datastore-copy.md` + +### CH27 — Nightly CI: the gates and the security watch · OPERATOR · DooPlex and ep0 · MENTIONED + +architecture folder: 85 lines in 19 files match `/nightly|security.watch|cron/` (a machine count, not a reading) · helper section: clock-hub „30. Nightly CI: gates and the security watch” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | NOT MEASURED. OPERATOR. Gitea CI crons in felhom.eu: `gates` and `security-watch`. The other three repos have no scheduled job (grep `schedule:/cron:` in their `.gitea/workflows/` returned nothing). | A | `E/.gitea/workflows/gates.yml:26-27` `E/.gitea/workflows/security-watch.yml:21-22` | +| 2 | 03:20 UTC and 03:40 UTC every night; the reason for the order is in. CI cron is UTC per, so it moves one hour against the Budapest night jobs at each clock change. | A | `E/.gitea/workflows/gates.yml:21-23` `AR/08-alarm-ladder.md:710` | +| 3 | Limits: 5 min min. The watch grows with the number of pinned programs. | A | `E/.gitea/workflows/gates.yml:53` `E/.gitea/workflows/gates.yml:20` `E/.gitea/workflows/security-watch.yml:33` | +| 4 | NOT MEASURED (DooPlex and ep0 are outside this run). A missed night is not made up. The watch's weekly summary repairs itself: it is sent when the last one is 8 or more days old. A runner that drops the job tells nobody. | N | `AR/08-alarm-ladder.md:717` · class W1 · S01 | +| 5 | Both run on `felhom-gates` runners; the watch commits its memory to main with 3 attempts. A push race with a human commit is retried, then fails. | A | `E/.gitea/workflows/security-watch.yml:32` `E/.gitea/workflows/security-watch.yml:95-106` | +| 6 | UTC cron, no DST. Reminder ages are computed by the script from its memory file. | A | `AR/08-alarm-ladder.md:716-720` | +| 7 | Failure mail to admin: "[felhom CI] security watch FAILED"; gates has its own alarm step. Whether the alarm step runs for a reaped job is recorded as unknown. | A | `E/.gitea/workflows/gates.yml:245` `E/.gitea/workflows/gates.yml:50-52` | +| 8 | NOT MEASURED (DooPlex and ep0 are outside this run). DOC. A night with no run at all: NOBODY KNOWS who notices. | N | `AR/08-alarm-ladder.md:708-722` · class W1 · S01 | + +**Hand step in a runbook?** no hand step diff --git a/documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-guest-clock.txt b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-guest-clock.txt new file mode 100644 index 00000000..abf1b0b9 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-guest-clock.txt @@ -0,0 +1,468 @@ +# guest-clock demo-felhom taken 2026-11-11T10:04:22Z from DooPlex + +=== clock === +utc 2026-11-11T10:03:50.305180545Z +local 2026-11-11T10:03:50 UTC +0000 +/usr/share/zoneinfo/Etc/UTC + Local time: Wed 2026-11-11 10:03:50 UTC + Universal time: Wed 2026-11-11 10:03:50 UTC + RTC time: n/a + Time zone: Etc/UTC (UTC, +0000) +System clock synchronized: no + NTP service: inactive + RTC in local TZ: no +chrony: inactive +systemd-timesyncd: inactive + +=== boots === +2026-11-08 04:35:32 + +=== timers === +NEXT LEFT LAST PASSED UNIT ACTIVATES +Wed 2026-11-11 12:30:43 UTC 2h 26min Wed 2026-11-11 01:28:16 UTC 8h ago apt-daily.timer apt-daily.service +Thu 2026-11-12 00:00:00 UTC 13h Wed 2026-11-11 00:00:14 UTC 10h ago dpkg-db-backup.timer dpkg-db-backup.service +Thu 2026-11-12 00:17:27 UTC 14h Wed 2026-11-11 01:00:14 UTC 9h ago logrotate.timer logrotate.service +Thu 2026-11-12 04:52:34 UTC 18h Wed 2026-11-11 04:52:34 UTC 5h 11min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service +Thu 2026-11-12 06:50:13 UTC 20h Wed 2026-11-11 06:43:16 UTC 3h 20min ago apt-daily-upgrade.timer apt-daily-upgrade.service +Thu 2026-11-12 11:29:47 UTC 1 day 1h Wed 2026-11-11 09:21:14 UTC 42min ago man-db.timer man-db.service +Sun 2026-11-15 03:10:29 UTC 3 days Sun 2026-11-08 03:10:56 UTC - e2scrub_all.timer e2scrub_all.service +- - - - fstrim.timer fstrim.service + +8 timers listed. + +=== timer units (all) === +--- apt-daily-upgrade.timer [enabled/active] +OnCalendar=*-*-* 6:00 +RandomizedDelaySec=60m +Persistent=true +Unit=apt-daily-upgrade.service NextElapseUSecRealtime=Thu 2026-11-12 06:50:13 UTC LastTriggerUSec=Wed 2026-11-11 06:43:16 UTC +--- apt-daily.timer [enabled/active] +OnCalendar=*-*-* 6,18:00 +RandomizedDelaySec=12h +Persistent=true +Unit=apt-daily.service NextElapseUSecRealtime=Wed 2026-11-11 12:30:43 UTC LastTriggerUSec=Wed 2026-11-11 01:28:16 UTC +--- apt-listchanges.timer [disabled/inactive] +OnCalendar=hourly +Unit=apt-listchanges.service NextElapseUSecRealtime= LastTriggerUSec= +--- dpkg-db-backup.timer [enabled/active] +OnCalendar=daily +Persistent=true +Unit=dpkg-db-backup.service NextElapseUSecRealtime=Thu 2026-11-12 00:00:00 UTC LastTriggerUSec=Wed 2026-11-11 00:00:14 UTC +--- e2scrub_all.timer [enabled/active] +OnCalendar=Sun *-*-* 03:10:00 +RandomizedDelaySec=60 +Persistent=true +Unit=e2scrub_all.service NextElapseUSecRealtime=Sun 2026-11-15 03:10:29 UTC LastTriggerUSec=Sun 2026-11-08 03:10:56 UTC +--- fstrim.timer [enabled/inactive] +OnCalendar=weekly +AccuracySec=1h +Persistent=true +RandomizedDelaySec=100min +Unit=fstrim.service NextElapseUSecRealtime= LastTriggerUSec= +--- logrotate.timer [enabled/active] +OnCalendar=daily +RandomizedDelaySec=1h +Persistent=true +Unit=logrotate.service NextElapseUSecRealtime=Thu 2026-11-12 00:17:27 UTC LastTriggerUSec=Wed 2026-11-11 01:00:14 UTC +--- man-db.timer [enabled/active] +OnCalendar=daily +RandomizedDelaySec=12h +Persistent=true +Unit=man-db.service NextElapseUSecRealtime=Thu 2026-11-12 11:29:47 UTC LastTriggerUSec=Wed 2026-11-11 09:21:14 UTC +--- systemd-tmpfiles-clean.timer [static/active] +OnBootSec=15min +OnUnitActiveSec=1d +Unit=systemd-tmpfiles-clean.service NextElapseUSecRealtime= LastTriggerUSec=Wed 2026-11-11 04:52:34 UTC + +=== felhom units === +felhom-controller-bootstrap.path loaded active running +felhom-controller-bootstrap.service loaded active exited +felhom-regen-hostkeys.service loaded inactive dead +--- felhom-controller-bootstrap.service +Type=oneshot Restart=no RestartUSec=100ms TimeoutStartUSec=infinity NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:16:28 UTC +--- felhom-regen-hostkeys.service +Type=oneshot Restart=no RestartUSec=100ms TimeoutStartUSec=infinity NRestarts=0 ActiveEnterTimestamp= + +=== cron === +SHELL=/bin/sh +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +44 * * * * root cd / && run-parts --report /etc/cron.hourly +42 6 * * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; } +43 6 * * 7 root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.weekly; } +6 6 1 * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.monthly; } +--- /etc/cron.d/e2scrub_all +25 3 * * 0 root test -e /run/systemd/system || SERVICE_MODE=1 /usr/libexec/e2fsprogs/e2scrub_all_cron +21 3 * * * root test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r +--- cron.hourly: +--- cron.daily: apt-compat dpkg logrotate man-db +--- cron.weekly: man-db +--- cron.monthly: +--- user crontabs: + +=== at === +no at + +=== docker daemon timers === +{ + "features": { + "containerd-snapshotter": false + }, + "live-restore": true, + "log-driver": "json-file", + "log-opts": { + "max-file": "3", + "max-size": "10m" + } +} + +=== containers === +sonarr lscr.io/linuxserver/sonarr:4.0.20 Up 5 hours (healthy) 5 hours ago +radarr lscr.io/linuxserver/radarr:6.4.4 Up 5 hours (healthy) 5 hours ago +qbittorrent qbittorrentofficial/qbittorrent-nox:5.2.4-1 Up 5 hours (healthy) 5 hours ago +prowlarr lscr.io/linuxserver/prowlarr:version-2.6.5.5623 Up 5 hours (healthy) 5 hours ago +opengist ghcr.io/thomiceli/opengist:1.15 Up 5 hours (healthy) 5 hours ago +felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 19 hours (healthy) 19 hours ago +traefik traefik:v3.7.17 Up 3 days 8 days ago +filebrowser gtstef/filebrowser:1.5.8-stable Up 3 days (healthy) 13 days ago +cloudflared cloudflare/cloudflared:2026.10.4 Up 3 days (healthy) 2 weeks ago + +=== restart policies + healthcheck intervals === +/sonarr restart=unless-stopped health=30s started=2026-11-11T04:37:03.420231296Z +/radarr restart=unless-stopped health=30s started=2026-11-11T04:37:02.695192531Z +/qbittorrent restart=unless-stopped health=30s started=2026-11-11T04:37:02.078094125Z +/prowlarr restart=unless-stopped health=30s started=2026-11-11T04:37:01.619681768Z +/opengist restart=unless-stopped health=30s started=2026-11-11T04:37:01.184181029Z +/felhom-controller restart=unless-stopped health=30s started=2026-11-10T15:16:28.700768259Z +/traefik restart=unless-stopped health=none started=2026-11-08T04:35:38.624971549Z +/filebrowser restart=unless-stopped health=30s started=2026-11-08T04:35:38.716750959Z +/cloudflared restart=unless-stopped health=30s started=2026-11-08T04:35:38.687709246Z + +=== controller TZ === +TZ=unset +in-container 2026-11-11T10:03:51 UTC +0000 + +=== controller log: first line, line count per hour === +2026-11-10T15:16:29.674880422Z 2026/11/10 15:16:29 [INFO] local-api: channel up (agent 169.254.253.1:8443) — guest 9201, 3 mount(s) visible + 856 2026-11-10T15 1037 2026-11-10T16 1020 2026-11-10T17 1018 2026-11-10T18 1030 2026-11-10T19 1037 2026-11-10T20 1040 2026-11-10T21 1019 2026-11-10T22 1020 2026-11-10T23 1035 2026-11-11T00 1115 2026-11-11T01 1052 2026-11-11T02 1151 2026-11-11T03 1080 2026-11-11T04 1028 2026-11-11T05 1021 2026-11-11T06 1029 2026-11-11T07 1031 2026-11-11T08 1041 2026-11-11T09 76 2026-11-11T10 +=== controller log: schedule words (tail 320) === +2026-11-11T09:19:30.032745986Z 2026/11/11 09:19:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:19:31.047893767Z 2026/11/11 09:19:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.015s) +2026-11-11T09:20:29.940744236Z 2026/11/11 09:20:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:20:29.940769896Z 2026/11/11 09:20:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:20:29.940774583Z 2026/11/11 09:20:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:20:30.009309170Z 2026/11/11 09:20:30 [INFO] [scheduler] Job stack-scan completed (took 71ms) +2026-11-11T09:20:30.034500437Z 2026/11/11 09:20:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:20:31.059835845Z 2026/11/11 09:20:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.026s) +2026-11-11T09:21:29.939509198Z 2026/11/11 09:21:29 [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T09:21:29.939540940Z 2026/11/11 09:21:29 [INFO] [scheduler] Running job: backup-cache +2026-11-11T09:21:29.941504600Z 2026/11/11 09:21:29 [INFO] [scheduler] Running job: system-health +2026-11-11T09:21:29.941534001Z 2026/11/11 09:21:29 [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T09:21:29.941537787Z 2026/11/11 09:21:29 [INFO] [scheduler] Job offsite-credential-retry completed (took 1ms) +2026-11-11T09:21:29.942211529Z 2026/11/11 09:21:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:21:29.942228817Z 2026/11/11 09:21:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:21:29.945081927Z 2026/11/11 09:21:29 [INFO] [backup] Found 0 DB dump files across drives +2026-11-11T09:21:30.034951149Z 2026/11/11 09:21:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:21:30.102663774Z 2026/11/11 09:21:30 [INFO] [scheduler] Job backup-cache completed (took 164ms) +2026-11-11T09:21:30.204935092Z 2026/11/11 09:21:30 [INFO] [scheduler] Job docker-socket-users completed (took 266ms) +2026-11-11T09:21:30.250439318Z 2026/11/11 09:21:30 [INFO] [scheduler] Job system-health completed (took 311ms) +2026-11-11T09:21:31.346105744Z 2026/11/11 09:21:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.312s) +2026-11-11T09:22:29.939276939Z 2026/11/11 09:22:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:22:29.939301660Z 2026/11/11 09:22:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:22:29.939305512Z 2026/11/11 09:22:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:22:30.012521266Z 2026/11/11 09:22:30 [INFO] [scheduler] Job stack-scan completed (took 74ms) +2026-11-11T09:22:30.033434608Z 2026/11/11 09:22:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:22:31.084002257Z 2026/11/11 09:22:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.051s) +2026-11-11T09:23:29.939326665Z 2026/11/11 09:23:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:23:29.940987297Z 2026/11/11 09:23:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:23:30.033579597Z 2026/11/11 09:23:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:23:31.107122786Z 2026/11/11 09:23:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.074s) +2026-11-11T09:24:29.939215794Z 2026/11/11 09:24:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:24:29.939454810Z 2026/11/11 09:24:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:24:29.939470466Z 2026/11/11 09:24:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:24:30.017668786Z 2026/11/11 09:24:30 [INFO] [scheduler] Job stack-scan completed (took 78ms) +2026-11-11T09:24:30.033748377Z 2026/11/11 09:24:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:24:31.162157165Z 2026/11/11 09:24:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.129s) +2026-11-11T09:25:29.939161798Z 2026/11/11 09:25:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:25:29.939409395Z 2026/11/11 09:25:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:25:30.032864595Z 2026/11/11 09:25:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:25:31.183233301Z 2026/11/11 09:25:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.15s) +2026-11-11T09:26:29.940607254Z 2026/11/11 09:26:29 [INFO] [scheduler] Running job: fill-watch-interval +2026-11-11T09:26:29.940638975Z 2026/11/11 09:26:29 [INFO] [scheduler] Job fill-watch-interval completed (took 0s) +2026-11-11T09:26:29.940642119Z 2026/11/11 09:26:29 [INFO] [scheduler] Running job: backup-cache +2026-11-11T09:26:29.940763975Z 2026/11/11 09:26:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:26:29.943610557Z 2026/11/11 09:26:29 [INFO] [backup] Found 0 DB dump files across drives +2026-11-11T09:26:29.944082566Z 2026/11/11 09:26:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:26:29.944349967Z 2026/11/11 09:26:29 [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T09:26:29.944376480Z 2026/11/11 09:26:29 [INFO] [scheduler] Running job: system-health +2026-11-11T09:26:29.944381766Z 2026/11/11 09:26:29 [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T09:26:29.944585395Z 2026/11/11 09:26:29 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T09:26:29.944593819Z 2026/11/11 09:26:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:26:30.038204950Z 2026/11/11 09:26:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:26:30.071885967Z 2026/11/11 09:26:30 [INFO] [scheduler] Job stack-scan completed (took 131ms) +2026-11-11T09:26:30.165842166Z 2026/11/11 09:26:30 [INFO] [scheduler] Job backup-cache completed (took 227ms) +2026-11-11T09:26:30.171985503Z 2026/11/11 09:26:30 [INFO] [scheduler] Job docker-socket-users completed (took 227ms) +2026-11-11T09:26:30.225845525Z 2026/11/11 09:26:30 [INFO] [scheduler] Job system-health completed (took 281ms) +2026-11-11T09:26:31.294720387Z 2026/11/11 09:26:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.261s) +2026-11-11T09:27:29.939192432Z 2026/11/11 09:27:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:27:29.939217192Z 2026/11/11 09:27:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:27:30.033904651Z 2026/11/11 09:27:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:27:31.163828722Z 2026/11/11 09:27:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.13s) +2026-11-11T09:28:29.940472096Z 2026/11/11 09:28:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:28:29.940531284Z 2026/11/11 09:28:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:28:29.940538458Z 2026/11/11 09:28:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:28:30.026628913Z 2026/11/11 09:28:30 [INFO] [scheduler] Job stack-scan completed (took 87ms) +2026-11-11T09:28:30.033679277Z 2026/11/11 09:28:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:28:31.207054624Z 2026/11/11 09:28:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.173s) +2026-11-11T09:29:29.941472926Z 2026/11/11 09:29:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:29:29.941671085Z 2026/11/11 09:29:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:29:30.033599628Z 2026/11/11 09:29:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:29:31.149169007Z 2026/11/11 09:29:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.116s) +2026-11-11T09:30:29.938847460Z 2026/11/11 09:30:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:30:29.939465542Z 2026/11/11 09:30:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:30:29.939478537Z 2026/11/11 09:30:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:30:30.015935960Z 2026/11/11 09:30:30 [INFO] [scheduler] Job stack-scan completed (took 76ms) +2026-11-11T09:30:30.034569582Z 2026/11/11 09:30:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:30:31.113682420Z 2026/11/11 09:30:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.079s) +2026-11-11T09:31:29.941516773Z 2026/11/11 09:31:29 [INFO] [scheduler] Running job: hub-report +2026-11-11T09:31:29.941548062Z 2026/11/11 09:31:29 [INFO] [scheduler] Running job: backup-cache +2026-11-11T09:31:29.941551119Z 2026/11/11 09:31:29 [INFO] [backup] Found 0 DB dump files across drives +2026-11-11T09:31:29.944189386Z 2026/11/11 09:31:29 [INFO] [scheduler] Running job: system-health +2026-11-11T09:31:29.945026613Z 2026/11/11 09:31:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:31:29.945082046Z 2026/11/11 09:31:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:31:29.945202121Z 2026/11/11 09:31:29 [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T09:31:29.945292919Z 2026/11/11 09:31:29 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T09:31:29.945298582Z 2026/11/11 09:31:29 [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T09:31:30.035711429Z 2026/11/11 09:31:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:31:30.069460589Z 2026/11/11 09:31:30 [INFO] [scheduler] Job backup-cache completed (took 130ms) +2026-11-11T09:31:30.251500638Z 2026/11/11 09:31:30 [INFO] [scheduler] Job docker-socket-users completed (took 306ms) +2026-11-11T09:31:30.306432329Z 2026/11/11 09:31:30 [INFO] [sync] Catalog sync complete +2026-11-11T09:31:30.344821157Z 2026/11/11 09:31:30 [INFO] [scheduler] Job system-health completed (took 398ms) +2026-11-11T09:31:31.652261371Z 2026/11/11 09:31:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.616s) +2026-11-11T09:31:32.061972037Z 2026/11/11 09:31:32 [INFO] [scheduler] Job hub-report completed (took 2.123s) +2026-11-11T09:32:29.941413423Z 2026/11/11 09:32:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:32:29.941441173Z 2026/11/11 09:32:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:32:29.941445383Z 2026/11/11 09:32:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:32:30.021707700Z 2026/11/11 09:32:30 [INFO] [scheduler] Job stack-scan completed (took 81ms) +2026-11-11T09:32:30.033382812Z 2026/11/11 09:32:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:32:31.109247136Z 2026/11/11 09:32:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.076s) +2026-11-11T09:33:29.941508561Z 2026/11/11 09:33:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:33:29.941535555Z 2026/11/11 09:33:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:33:30.033710844Z 2026/11/11 09:33:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:33:31.048626808Z 2026/11/11 09:33:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.015s) +2026-11-11T09:34:29.938741935Z 2026/11/11 09:34:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:34:29.938857831Z 2026/11/11 09:34:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:34:29.938911235Z 2026/11/11 09:34:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:34:30.014463050Z 2026/11/11 09:34:30 [INFO] [scheduler] Job stack-scan completed (took 75ms) +2026-11-11T09:34:30.033598262Z 2026/11/11 09:34:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:34:31.149557118Z 2026/11/11 09:34:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.116s) +2026-11-11T09:35:29.938834372Z 2026/11/11 09:35:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:35:29.939201779Z 2026/11/11 09:35:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:35:30.033416828Z 2026/11/11 09:35:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:35:31.169062289Z 2026/11/11 09:35:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.136s) +2026-11-11T09:36:29.940612711Z 2026/11/11 09:36:29 [INFO] [scheduler] Running job: fill-watch-interval +2026-11-11T09:36:29.940649615Z 2026/11/11 09:36:29 [INFO] [scheduler] Job fill-watch-interval completed (took 0s) +2026-11-11T09:36:29.940652254Z 2026/11/11 09:36:29 [INFO] [scheduler] Running job: backup-cache +2026-11-11T09:36:29.940665261Z 2026/11/11 09:36:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:36:29.940668410Z 2026/11/11 09:36:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:36:29.940670532Z 2026/11/11 09:36:29 [INFO] [backup] Found 0 DB dump files across drives +2026-11-11T09:36:29.940672508Z 2026/11/11 09:36:29 [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T09:36:29.940761371Z 2026/11/11 09:36:29 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T09:36:29.941875324Z 2026/11/11 09:36:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:36:29.941894114Z 2026/11/11 09:36:29 [INFO] [scheduler] Running job: system-health +2026-11-11T09:36:29.942060419Z 2026/11/11 09:36:29 [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T09:36:30.033243949Z 2026/11/11 09:36:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:36:30.103883556Z 2026/11/11 09:36:30 [INFO] [scheduler] Job stack-scan completed (took 155ms) +2026-11-11T09:36:30.173816222Z 2026/11/11 09:36:30 [INFO] [scheduler] Job docker-socket-users completed (took 232ms) +2026-11-11T09:36:30.196014446Z 2026/11/11 09:36:30 [INFO] [scheduler] Job backup-cache completed (took 257ms) +2026-11-11T09:36:30.221705078Z 2026/11/11 09:36:30 [INFO] [scheduler] Job system-health completed (took 280ms) +2026-11-11T09:36:31.326292598Z 2026/11/11 09:36:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.293s) +2026-11-11T09:37:29.939062916Z 2026/11/11 09:37:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:37:29.939385804Z 2026/11/11 09:37:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:37:30.033685229Z 2026/11/11 09:37:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:37:31.155442754Z 2026/11/11 09:37:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.122s) +2026-11-11T09:38:29.939118414Z 2026/11/11 09:38:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:38:29.942508551Z 2026/11/11 09:38:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:38:29.942531863Z 2026/11/11 09:38:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:38:30.026780673Z 2026/11/11 09:38:30 [INFO] [scheduler] Job stack-scan completed (took 87ms) +2026-11-11T09:38:30.034520506Z 2026/11/11 09:38:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:38:31.105873817Z 2026/11/11 09:38:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.072s) +2026-11-11T09:39:29.939864710Z 2026/11/11 09:39:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:39:29.939895666Z 2026/11/11 09:39:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:39:30.032814539Z 2026/11/11 09:39:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:39:31.131566834Z 2026/11/11 09:39:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.099s) +2026-11-11T09:40:29.938961079Z 2026/11/11 09:40:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:40:29.938995582Z 2026/11/11 09:40:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:40:29.938999507Z 2026/11/11 09:40:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:40:30.016668894Z 2026/11/11 09:40:30 [INFO] [scheduler] Job stack-scan completed (took 78ms) +2026-11-11T09:40:30.035402779Z 2026/11/11 09:40:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:40:31.149009267Z 2026/11/11 09:40:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.115s) +2026-11-11T09:41:29.940693099Z 2026/11/11 09:41:29 [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T09:41:29.940737815Z 2026/11/11 09:41:29 [INFO] [scheduler] Running job: backup-cache +2026-11-11T09:41:29.940742520Z 2026/11/11 09:41:29 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T09:41:29.940745604Z 2026/11/11 09:41:29 [INFO] [backup] Found 0 DB dump files across drives +2026-11-11T09:41:29.943573777Z 2026/11/11 09:41:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:41:29.943719078Z 2026/11/11 09:41:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:41:29.943832388Z 2026/11/11 09:41:29 [INFO] [scheduler] Running job: system-health +2026-11-11T09:41:29.944494176Z 2026/11/11 09:41:29 [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T09:41:30.033933971Z 2026/11/11 09:41:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:41:30.081286147Z 2026/11/11 09:41:30 [INFO] [scheduler] Job backup-cache completed (took 142ms) +2026-11-11T09:41:30.156219314Z 2026/11/11 09:41:30 [INFO] [scheduler] Job docker-socket-users completed (took 211ms) +2026-11-11T09:41:30.192790386Z 2026/11/11 09:41:30 [INFO] [scheduler] Job system-health completed (took 249ms) +2026-11-11T09:41:31.400481678Z 2026/11/11 09:41:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.365s) +2026-11-11T09:42:29.939869431Z 2026/11/11 09:42:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:42:29.940027682Z 2026/11/11 09:42:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:42:29.940225672Z 2026/11/11 09:42:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:42:30.017638053Z 2026/11/11 09:42:30 [INFO] [scheduler] Job stack-scan completed (took 77ms) +2026-11-11T09:42:30.032884786Z 2026/11/11 09:42:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:42:31.210835039Z 2026/11/11 09:42:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.178s) +2026-11-11T09:43:29.942441119Z 2026/11/11 09:43:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:43:29.942469213Z 2026/11/11 09:43:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:43:30.033142287Z 2026/11/11 09:43:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:43:31.197181516Z 2026/11/11 09:43:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.164s) +2026-11-11T09:44:29.939367673Z 2026/11/11 09:44:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:44:29.939389720Z 2026/11/11 09:44:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:44:29.939393976Z 2026/11/11 09:44:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:44:30.008640544Z 2026/11/11 09:44:30 [INFO] [scheduler] Job stack-scan completed (took 69ms) +2026-11-11T09:44:30.033536834Z 2026/11/11 09:44:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:44:31.097227054Z 2026/11/11 09:44:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.064s) +2026-11-11T09:45:29.939259992Z 2026/11/11 09:45:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:45:29.939297495Z 2026/11/11 09:45:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:45:30.032994407Z 2026/11/11 09:45:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:45:31.070440102Z 2026/11/11 09:45:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.037s) +2026-11-11T09:46:29.939450882Z 2026/11/11 09:46:29 [INFO] [scheduler] Running job: fill-watch-interval +2026-11-11T09:46:29.939497086Z 2026/11/11 09:46:29 [INFO] [scheduler] Job fill-watch-interval completed (took 0s) +2026-11-11T09:46:29.939499258Z 2026/11/11 09:46:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:46:29.939772208Z 2026/11/11 09:46:29 [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T09:46:29.939784524Z 2026/11/11 09:46:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:46:29.939786985Z 2026/11/11 09:46:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:46:29.942545161Z 2026/11/11 09:46:29 [INFO] [scheduler] Running job: hub-report +2026-11-11T09:46:29.944289379Z 2026/11/11 09:46:29 [INFO] [scheduler] Running job: system-health +2026-11-11T09:46:29.944316340Z 2026/11/11 09:46:29 [INFO] [scheduler] Running job: backup-cache +2026-11-11T09:46:29.949012043Z 2026/11/11 09:46:29 [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T09:46:29.949056892Z 2026/11/11 09:46:29 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T09:46:30.038628812Z 2026/11/11 09:46:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:46:30.146613911Z 2026/11/11 09:46:30 [INFO] [scheduler] Job stack-scan completed (took 203ms) +2026-11-11T09:46:30.150177820Z 2026/11/11 09:46:30 [INFO] [backup] Found 0 DB dump files across drives +2026-11-11T09:46:30.263896201Z 2026/11/11 09:46:30 [INFO] [scheduler] Job docker-socket-users completed (took 324ms) +2026-11-11T09:46:30.314429495Z 2026/11/11 09:46:30 [INFO] [scheduler] Job system-health completed (took 371ms) +2026-11-11T09:46:30.377003979Z 2026/11/11 09:46:30 [INFO] [sync] Catalog sync complete +2026-11-11T09:46:31.723468696Z 2026/11/11 09:46:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.685s) +2026-11-11T09:46:31.761256256Z 2026/11/11 09:46:31 [INFO] [scheduler] Job backup-cache completed (took 1.818s) +2026-11-11T09:46:32.132913551Z 2026/11/11 09:46:32 [INFO] [scheduler] Job hub-report completed (took 2.19s) +2026-11-11T09:47:29.939089006Z 2026/11/11 09:47:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:47:29.940757607Z 2026/11/11 09:47:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:47:30.033145375Z 2026/11/11 09:47:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:47:31.116187602Z 2026/11/11 09:47:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.083s) +2026-11-11T09:48:29.940607224Z 2026/11/11 09:48:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:48:29.944727640Z 2026/11/11 09:48:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:48:29.944753009Z 2026/11/11 09:48:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:48:30.024438338Z 2026/11/11 09:48:30 [INFO] [scheduler] Job stack-scan completed (took 83ms) +2026-11-11T09:48:30.034601093Z 2026/11/11 09:48:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:48:31.082565125Z 2026/11/11 09:48:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.049s) +2026-11-11T09:49:29.939378064Z 2026/11/11 09:49:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:49:29.941989144Z 2026/11/11 09:49:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:49:30.033433512Z 2026/11/11 09:49:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:49:31.125298440Z 2026/11/11 09:49:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.092s) +2026-11-11T09:50:29.939798950Z 2026/11/11 09:50:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:50:29.942571272Z 2026/11/11 09:50:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:50:29.942708176Z 2026/11/11 09:50:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:50:30.033187088Z 2026/11/11 09:50:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:50:30.037636389Z 2026/11/11 09:50:30 [INFO] [scheduler] Job stack-scan completed (took 95ms) +2026-11-11T09:50:31.096643566Z 2026/11/11 09:50:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.063s) +2026-11-11T09:51:29.940226248Z 2026/11/11 09:51:29 [INFO] [scheduler] Running job: backup-cache +2026-11-11T09:51:29.940267126Z 2026/11/11 09:51:29 [INFO] [backup] Found 0 DB dump files across drives +2026-11-11T09:51:29.940596892Z 2026/11/11 09:51:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:51:29.940617671Z 2026/11/11 09:51:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:51:29.943677817Z 2026/11/11 09:51:29 [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T09:51:29.944251922Z 2026/11/11 09:51:29 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T09:51:29.944407613Z 2026/11/11 09:51:29 [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T09:51:29.944675022Z 2026/11/11 09:51:29 [INFO] [scheduler] Running job: system-health +2026-11-11T09:51:30.033353310Z 2026/11/11 09:51:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:51:30.044089267Z 2026/11/11 09:51:30 [INFO] [scheduler] Job backup-cache completed (took 106ms) +2026-11-11T09:51:30.159339236Z 2026/11/11 09:51:30 [INFO] [scheduler] Job docker-socket-users completed (took 215ms) +2026-11-11T09:51:30.192138415Z 2026/11/11 09:51:30 [INFO] [scheduler] Job system-health completed (took 247ms) +2026-11-11T09:51:31.337185333Z 2026/11/11 09:51:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.304s) +2026-11-11T09:52:29.939605526Z 2026/11/11 09:52:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:52:29.939628545Z 2026/11/11 09:52:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:52:29.939632737Z 2026/11/11 09:52:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:52:30.013892871Z 2026/11/11 09:52:30 [INFO] [scheduler] Job stack-scan completed (took 74ms) +2026-11-11T09:52:30.038448202Z 2026/11/11 09:52:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:52:31.062480482Z 2026/11/11 09:52:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.027s) +2026-11-11T09:53:29.939189707Z 2026/11/11 09:53:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:53:29.939231431Z 2026/11/11 09:53:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:53:30.033878382Z 2026/11/11 09:53:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:53:31.128207147Z 2026/11/11 09:53:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.095s) +2026-11-11T09:54:29.941475063Z 2026/11/11 09:54:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:54:29.941500202Z 2026/11/11 09:54:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:54:29.941504121Z 2026/11/11 09:54:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:54:30.015818171Z 2026/11/11 09:54:30 [INFO] [scheduler] Job stack-scan completed (took 77ms) +2026-11-11T09:54:30.033142254Z 2026/11/11 09:54:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:54:31.055222046Z 2026/11/11 09:54:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.022s) +2026-11-11T09:55:29.939173324Z 2026/11/11 09:55:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:55:29.939204929Z 2026/11/11 09:55:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:55:30.033725359Z 2026/11/11 09:55:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:55:31.088110813Z 2026/11/11 09:55:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.054s) +2026-11-11T09:56:29.940873666Z 2026/11/11 09:56:29 [INFO] [scheduler] Running job: backup-cache +2026-11-11T09:56:29.940901521Z 2026/11/11 09:56:29 [INFO] [scheduler] Running job: fill-watch-interval +2026-11-11T09:56:29.940908808Z 2026/11/11 09:56:29 [INFO] [scheduler] Job fill-watch-interval completed (took 0s) +2026-11-11T09:56:29.940911388Z 2026/11/11 09:56:29 [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T09:56:29.940913857Z 2026/11/11 09:56:29 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T09:56:29.940916417Z 2026/11/11 09:56:29 [INFO] [scheduler] Running job: system-health +2026-11-11T09:56:29.941686823Z 2026/11/11 09:56:29 [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T09:56:29.943096654Z 2026/11/11 09:56:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:56:29.955313406Z 2026/11/11 09:56:29 [INFO] [backup] Found 0 DB dump files across drives +2026-11-11T09:56:29.955489900Z 2026/11/11 09:56:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:56:29.955608054Z 2026/11/11 09:56:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:56:30.033368984Z 2026/11/11 09:56:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:56:30.071586073Z 2026/11/11 09:56:30 [INFO] [scheduler] Job stack-scan completed (took 128ms) +2026-11-11T09:56:30.160507282Z 2026/11/11 09:56:30 [INFO] [scheduler] Job docker-socket-users completed (took 218ms) +2026-11-11T09:56:30.181769446Z 2026/11/11 09:56:30 [INFO] [scheduler] Job backup-cache completed (took 243ms) +2026-11-11T09:56:30.246207320Z 2026/11/11 09:56:30 [INFO] [scheduler] Job system-health completed (took 306ms) +2026-11-11T09:56:31.376228125Z 2026/11/11 09:56:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.343s) +2026-11-11T09:57:29.938865122Z 2026/11/11 09:57:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:57:29.938888865Z 2026/11/11 09:57:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:57:30.033473166Z 2026/11/11 09:57:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:57:31.115547788Z 2026/11/11 09:57:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.082s) +2026-11-11T09:58:29.941618096Z 2026/11/11 09:58:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:58:29.941647476Z 2026/11/11 09:58:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:58:29.941652112Z 2026/11/11 09:58:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:58:30.017199930Z 2026/11/11 09:58:30 [INFO] [scheduler] Job stack-scan completed (took 78ms) +2026-11-11T09:58:30.034666906Z 2026/11/11 09:58:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:58:31.145412165Z 2026/11/11 09:58:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.11s) +2026-11-11T09:59:29.942485597Z 2026/11/11 09:59:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:59:29.942509543Z 2026/11/11 09:59:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:59:30.032918207Z 2026/11/11 09:59:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:59:31.187962617Z 2026/11/11 09:59:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.155s) +2026-11-11T10:00:29.939789317Z 2026/11/11 10:00:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T10:00:29.940257245Z 2026/11/11 10:00:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T10:00:29.940279501Z 2026/11/11 10:00:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T10:00:30.016476119Z 2026/11/11 10:00:30 [INFO] [scheduler] Job stack-scan completed (took 75ms) +2026-11-11T10:00:30.034280832Z 2026/11/11 10:00:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T10:00:31.133303785Z 2026/11/11 10:00:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.099s) +2026-11-11T10:01:29.938757701Z 2026/11/11 10:01:29 [INFO] [scheduler] Running job: backup-cache +2026-11-11T10:01:29.942223659Z 2026/11/11 10:01:29 [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T10:01:29.942330274Z 2026/11/11 10:01:29 [INFO] [scheduler] Running job: hub-report +2026-11-11T10:01:29.944029296Z 2026/11/11 10:01:29 [INFO] [backup] Found 0 DB dump files across drives +2026-11-11T10:01:29.945116583Z 2026/11/11 10:01:29 [INFO] [scheduler] Running job: system-health +2026-11-11T10:01:29.949333886Z 2026/11/11 10:01:29 [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T10:01:29.949408711Z 2026/11/11 10:01:29 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T10:01:29.949444935Z 2026/11/11 10:01:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T10:01:29.949471164Z 2026/11/11 10:01:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T10:01:30.035400222Z 2026/11/11 10:01:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T10:01:30.258333370Z 2026/11/11 10:01:30 [INFO] [scheduler] Job docker-socket-users completed (took 316ms) +2026-11-11T10:01:30.273972608Z 2026/11/11 10:01:30 [INFO] [sync] Catalog sync complete +2026-11-11T10:01:30.315878773Z 2026/11/11 10:01:30 [INFO] [scheduler] Job system-health completed (took 371ms) +2026-11-11T10:01:31.590833808Z 2026/11/11 10:01:31 [INFO] [scheduler] Job backup-cache completed (took 1.649s) +2026-11-11T10:01:31.729413745Z 2026/11/11 10:01:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.694s) +2026-11-11T10:01:31.963664882Z 2026/11/11 10:01:31 [INFO] [scheduler] Job hub-report completed (took 2.021s) +2026-11-11T10:02:29.939771665Z 2026/11/11 10:02:29 [INFO] [scheduler] Running job: stack-scan +2026-11-11T10:02:29.939857115Z 2026/11/11 10:02:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T10:02:29.939863878Z 2026/11/11 10:02:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T10:02:30.013085616Z 2026/11/11 10:02:30 [INFO] [scheduler] Job stack-scan completed (took 73ms) +2026-11-11T10:02:30.033880349Z 2026/11/11 10:02:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T10:02:31.093182115Z 2026/11/11 10:02:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.06s) +2026-11-11T10:03:29.939938323Z 2026/11/11 10:03:29 [INFO] [scheduler] Running job: hold-report +2026-11-11T10:03:29.939961541Z 2026/11/11 10:03:29 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T10:03:30.033226002Z 2026/11/11 10:03:30 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T10:03:31.150124665Z 2026/11/11 10:03:31 [INFO] [scheduler] Job agent-channel-health completed (took 1.117s) +RC=0 diff --git a/documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-guest-night.txt b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-guest-night.txt new file mode 100644 index 00000000..a9b68793 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-guest-night.txt @@ -0,0 +1,228 @@ +# guest-night demo-felhom taken 2026-11-11T10:09:28Z +log first line: 2026-11-10T15:16:29.674880422Z +log last line: 2026-11-11T10:08:49.976032577Z + +=== scheduler jobs (UTC): runs, first start, last start, last three "took" === +agent-channel-health runs=1132 first=2026-11-10T15:17:30 last=2026-11-11T10:08:30 [completed (took 1.297s)] [completed (took 1.041s)] [completed (took 1.06s)] +backup-cache runs=226 first=2026-11-10T15:21:29 last=2026-11-11T10:06:29 [completed (took 243ms)] [completed (took 1.649s)] [completed (took 220ms)] +conversion-copy-release runs=18 first=2026-11-10T16:16:29 last=2026-11-11T09:16:29 [completed (took 0s)] [completed (took 0s)] [completed (took 288ms)] +db-dump runs=1 first=2026-11-11T01:30:00 last=2026-11-11T01:30:00 [completed (took 18.336s)] +disk-health-check runs=18 first=2026-11-10T16:16:30 last=2026-11-11T09:16:30 [completed (took 1.965s)] [completed (took 3.058s)] [completed (took 2.176s)] +docker-socket-users runs=226 first=2026-11-10T15:21:29 last=2026-11-11T10:06:29 [completed (took 218ms)] [completed (took 316ms)] [completed (took 188ms)] +fill-watch runs=1 first=2026-11-11T02:30:00 last=2026-11-11T02:30:00 [completed (took 0s)] +fill-watch-interval runs=113 first=2026-11-10T15:26:29 last=2026-11-11T10:06:29 [completed (took 0s)] [completed (took 0s)] [completed (took 10ms)] +geo-verify runs=3 first=2026-11-10T21:16:29 last=2026-11-11T09:16:29 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)] +hold-report runs=1132 first=2026-11-10T15:17:29 last=2026-11-11T10:08:29 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)] +hub-report runs=75 first=2026-11-10T15:31:29 last=2026-11-11T10:01:29 [completed (took 2.123s)] [completed (took 2.19s)] [completed (took 2.021s)] +metrics-prune runs=1 first=2026-11-11T03:00:00 last=2026-11-11T03:00:00 [completed (took 59ms)] +offbox-backup runs=1 first=2026-11-11T03:15:00 last=2026-11-11T03:15:00 [completed (took 38.095s)] +offsite-abandon-sweep runs=1 first=2026-11-11T04:10:00 last=2026-11-11T04:10:00 [completed (took 0s)] +offsite-credential-retry runs=226 first=2026-11-10T15:21:29 last=2026-11-11T10:06:29 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)] +offsite-integrity runs=1 first=2026-11-11T05:00:00 last=2026-11-11T05:00:00 [completed (took 0s)] +offsite-proof runs=1 first=2026-11-11T04:30:00 last=2026-11-11T04:30:00 [completed (took 3.044s)] +restore-copy-sweep runs=3 first=2026-11-10T21:16:29 last=2026-11-11T09:16:29 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)] +selfupdate-check runs=3 first=2026-11-10T21:16:29 last=2026-11-11T09:16:29 [completed (took 49ms)] [completed (took 62ms)] [completed (took 72ms)] +stack-scan runs=566 first=2026-11-10T15:18:29 last=2026-11-11T10:08:29 [completed (took 78ms)] [completed (took 156ms)] [completed (took 68ms)] +system-health runs=226 first=2026-11-10T15:21:29 last=2026-11-11T10:06:29 [completed (took 306ms)] [completed (took 371ms)] [completed (took 278ms)] +tier2-backup runs=1 first=2026-11-11T02:30:00 last=2026-11-11T02:30:00 [completed (took 11.412s)] + +=== jobs that ran fewer than 30 times: every start and end (UTC) === +2026/11/10 16:16:29 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/10 16:16:30 [INFO] [scheduler] Running job: disk-health-check +2026/11/10 16:16:30 [INFO] [scheduler] Job conversion-copy-release completed (took 230ms) +2026/11/10 16:16:31 [INFO] [scheduler] Job disk-health-check completed (took 1.869s) +2026/11/10 17:16:29 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/10 17:16:30 [INFO] [scheduler] Running job: disk-health-check +2026/11/10 17:16:30 [INFO] [scheduler] Job conversion-copy-release completed (took 257ms) +2026/11/10 17:16:31 [INFO] [scheduler] Job disk-health-check completed (took 1.78s) +2026/11/10 18:16:29 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/10 18:16:30 [INFO] [scheduler] Running job: disk-health-check +2026/11/10 18:16:30 [INFO] [scheduler] Job conversion-copy-release completed (took 249ms) +2026/11/10 18:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.132s) +2026/11/10 19:16:29 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/10 19:16:29 [INFO] [scheduler] Job conversion-copy-release completed (took 1ms) +2026/11/10 19:16:30 [INFO] [scheduler] Running job: disk-health-check +2026/11/10 19:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.138s) +2026/11/10 20:16:29 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/10 20:16:29 [INFO] [scheduler] Job conversion-copy-release completed (took 1ms) +2026/11/10 20:16:30 [INFO] [scheduler] Running job: disk-health-check +2026/11/10 20:16:31 [INFO] [scheduler] Job disk-health-check completed (took 1.813s) +2026/11/10 21:16:29 [INFO] [scheduler] Running job: selfupdate-check +2026/11/10 21:16:29 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/10 21:16:29 [INFO] [scheduler] Running job: restore-copy-sweep +2026/11/10 21:16:29 [INFO] [scheduler] Job restore-copy-sweep completed (took 0s) +2026/11/10 21:16:29 [INFO] [scheduler] Running job: geo-verify +2026/11/10 21:16:29 [INFO] [scheduler] Job geo-verify completed (took 0s) +2026/11/10 21:16:29 [INFO] [scheduler] Job selfupdate-check completed (took 49ms) +2026/11/10 21:16:30 [INFO] [scheduler] Running job: disk-health-check +2026/11/10 21:16:30 [INFO] [scheduler] Job conversion-copy-release completed (took 260ms) +2026/11/10 21:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.038s) +2026/11/10 22:16:29 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/10 22:16:30 [INFO] [scheduler] Running job: disk-health-check +2026/11/10 22:16:30 [INFO] [scheduler] Job conversion-copy-release completed (took 180ms) +2026/11/10 22:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.013s) +2026/11/10 23:16:29 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/10 23:16:29 [INFO] [scheduler] Job conversion-copy-release completed (took 29ms) +2026/11/10 23:16:30 [INFO] [scheduler] Running job: disk-health-check +2026/11/10 23:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.292s) +2026/11/11 00:16:29 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 00:16:29 [INFO] [scheduler] Job conversion-copy-release completed (took 0s) +2026/11/11 00:16:30 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 00:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.29s) +2026/11/11 01:16:29 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 01:16:30 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 01:16:30 [INFO] [scheduler] Job conversion-copy-release completed (took 188ms) +2026/11/11 01:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.276s) +2026/11/11 01:30:00 [INFO] [scheduler] Running job: db-dump +2026/11/11 01:30:18 [INFO] [scheduler] Job db-dump completed (took 18.336s) +2026/11/11 02:16:29 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 02:16:30 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 02:16:30 [INFO] [scheduler] Job conversion-copy-release completed (took 93ms) +2026/11/11 02:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.166s) +2026/11/11 02:30:00 [INFO] [scheduler] Running job: tier2-backup +2026/11/11 02:30:00 [INFO] [scheduler] Running job: fill-watch +2026/11/11 02:30:00 [INFO] [scheduler] Job fill-watch completed (took 0s) +2026/11/11 02:30:11 [INFO] [scheduler] Job tier2-backup completed (took 11.412s) +2026/11/11 03:00:00 [INFO] [scheduler] Running job: metrics-prune +2026/11/11 03:00:00 [INFO] [scheduler] Job metrics-prune completed (took 59ms) +2026/11/11 03:15:00 [INFO] [scheduler] Running job: offbox-backup +2026/11/11 03:15:38 [INFO] [scheduler] Job offbox-backup completed (took 38.095s) +2026/11/11 03:16:29 [INFO] [scheduler] Running job: selfupdate-check +2026/11/11 03:16:29 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 03:16:29 [INFO] [scheduler] Running job: restore-copy-sweep +2026/11/11 03:16:29 [INFO] [scheduler] Job restore-copy-sweep completed (took 0s) +2026/11/11 03:16:29 [INFO] [scheduler] Job conversion-copy-release completed (took 0s) +2026/11/11 03:16:29 [INFO] [scheduler] Running job: geo-verify +2026/11/11 03:16:29 [INFO] [scheduler] Job geo-verify completed (took 0s) +2026/11/11 03:16:30 [INFO] [scheduler] Job selfupdate-check completed (took 62ms) +2026/11/11 03:16:30 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 03:16:31 [INFO] [scheduler] Job disk-health-check completed (took 1.751s) +2026/11/11 04:10:00 [INFO] [scheduler] Running job: offsite-abandon-sweep +2026/11/11 04:10:00 [INFO] [scheduler] Job offsite-abandon-sweep completed (took 0s) +2026/11/11 04:16:29 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 04:16:30 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 04:16:30 [INFO] [scheduler] Job conversion-copy-release completed (took 316ms) +2026/11/11 04:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.328s) +2026/11/11 04:30:00 [INFO] [scheduler] Running job: offsite-proof +2026/11/11 04:30:03 [INFO] [scheduler] Job offsite-proof completed (took 3.044s) +2026/11/11 05:00:00 [INFO] [scheduler] Running job: offsite-integrity +2026/11/11 05:00:00 [INFO] [scheduler] Job offsite-integrity completed (took 0s) +2026/11/11 05:16:29 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 05:16:29 [INFO] [scheduler] Job conversion-copy-release completed (took 1ms) +2026/11/11 05:16:30 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 05:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.099s) +2026/11/11 06:16:29 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 06:16:29 [INFO] [scheduler] Job conversion-copy-release completed (took 0s) +2026/11/11 06:16:30 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 06:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.355s) +2026/11/11 07:16:29 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 07:16:29 [INFO] [scheduler] Job conversion-copy-release completed (took 0s) +2026/11/11 07:16:30 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 07:16:32 [INFO] [scheduler] Job disk-health-check completed (took 1.965s) +2026/11/11 08:16:29 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 08:16:29 [INFO] [scheduler] Job conversion-copy-release completed (took 0s) +2026/11/11 08:16:30 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 08:16:33 [INFO] [scheduler] Job disk-health-check completed (took 3.058s) +2026/11/11 09:16:29 [INFO] [scheduler] Running job: selfupdate-check +2026/11/11 09:16:29 [INFO] [scheduler] Running job: restore-copy-sweep +2026/11/11 09:16:29 [INFO] [scheduler] Job restore-copy-sweep completed (took 0s) +2026/11/11 09:16:29 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 09:16:29 [INFO] [scheduler] Running job: geo-verify +2026/11/11 09:16:29 [INFO] [scheduler] Job geo-verify completed (took 0s) +2026/11/11 09:16:30 [INFO] [scheduler] Job selfupdate-check completed (took 72ms) +2026/11/11 09:16:30 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 09:16:30 [INFO] [scheduler] Job conversion-copy-release completed (took 288ms) +2026/11/11 09:16:32 [INFO] [scheduler] Job disk-health-check completed (took 2.176s) + +=== night lines 19:00-08:00 UTC, INFO/WARN/ERROR, backup words (first 2 per message shape) === +2026/11/10 19:01:29 [INFO] [backup] Found 0 DB dump files across drives +2026/11/10 19:01:30 [INFO] [backup] Discovered 0 databases +2026/11/10 19:01:30 [INFO] [backup] Discovered app data: 5 apps +2026/11/10 19:01:30 [INFO] [monitor] Health check: status=ok +2026/11/10 19:01:31 [INFO] [backup] Backup status cache refreshed +2026/11/10 19:01:31 [INFO] [monitor] Health check: status=ok +2026/11/10 19:06:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok +2026/11/10 19:06:30 [INFO] [backup] Found 0 DB dump files across drives +2026/11/10 19:06:30 [INFO] [backup] Discovered 0 databases +2026/11/10 19:06:30 [INFO] [backup] Discovered app data: 5 apps +2026/11/10 19:06:30 [INFO] [backup] Backup status cache refreshed +2026/11/10 19:07:29 [INFO] [deadapp] check alive: 460 scans since boot, 5 deployed app(s) evaluated, 0 currently down +2026/11/10 19:16:29 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok +2026/11/10 19:16:32 [INFO] [web] disk-health check complete: 2 disk(s) evaluated, 0 alert(s) +2026/11/10 19:17:29 [INFO] [deadapp] check alive: 480 scans since boot, 5 deployed app(s) evaluated, 0 currently down +2026/11/10 20:16:31 [INFO] [web] disk-health check complete: 2 disk(s) evaluated, 0 alert(s) +2026/11/11 01:30:00 [INFO] [backup] Starting database dump run +2026/11/11 01:30:00 [INFO] [backup] No database containers found +2026/11/11 01:30:00 [INFO] [backup] Stopping opengist for safe volume dump +2026/11/11 01:30:00 [INFO] [backup] Volume dump: opengist/opengist_opengist_data → 182.5 KB +2026/11/11 01:30:00 [INFO] [backup] Restarting opengist after volume dump +2026/11/11 01:30:00 [INFO] [backup] Stopping prowlarr for safe volume dump +2026/11/11 01:30:04 [WARN] [backup] prowlarr: template volume entry ignored — volume "prowlarr_config": class "important" — only "excluded" is meaningful for a named volume +2026/11/11 01:30:04 [INFO] [backup] Volume dump: prowlarr/prowlarr_prowlarr_config → 6.5 MB +2026/11/11 01:30:04 [INFO] [backup] Restarting prowlarr after volume dump +2026/11/11 01:30:04 [INFO] [backup] Stopping radarr for safe volume dump +2026/11/11 01:30:11 [INFO] [backup] Volume dump: radarr/radarr_radarr_config → 7.0 MB +2026/11/11 01:30:11 [INFO] [backup] Restarting radarr after volume dump +2026/11/11 01:30:11 [INFO] [backup] Stopping sonarr for safe volume dump +2026/11/11 01:30:14 [INFO] [busymark] busy: a backup, a restore or an off-site run is running — an operator reboot of the box is refused until it ends +2026/11/11 01:30:16 [INFO] [backup] Volume dump: sonarr/sonarr_sonarr_config → 25.7 MB +2026/11/11 01:30:16 [INFO] [backup] Restarting sonarr after volume dump +2026/11/11 01:30:16 [INFO] [backup] Stopping qbittorrent for safe volume dump +2026/11/11 01:30:17 [WARN] [backup] qbittorrent: template volume entry ignored — volume "qbittorrent_config": class "important" — only "excluded" is meaningful for a named volume +2026/11/11 01:30:17 [INFO] [backup] Volume dump: qbittorrent/qbittorrent_qbittorrent_config → 9.9 MB +2026/11/11 01:30:17 [INFO] [backup] Restarting qbittorrent after volume dump +2026/11/11 01:30:18 [INFO] [backup] App-data backup completed: 0 databases (0 B total), 5 volume dump(s) (18.202s) +2026/11/11 01:30:18 [INFO] [backup] Recovery unit captured for opengist → /mnt/sys_drive/felhom-data/backups/primary/opengist (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/11/11 01:30:18 [INFO] [backup] Recovery unit captured for prowlarr → /mnt/sys_drive/felhom-data/backups/primary/prowlarr (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/11/11 01:30:18 [INFO] [backup] Recovery unit captured for radarr → /mnt/felhom-drives/hdd_1/backups/primary/radarr (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/11/11 01:30:18 [INFO] [backup] Recovery unit captured for sonarr → /mnt/felhom-drives/hdd_1/backups/primary/sonarr (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/11/11 01:30:18 [INFO] [backup] Recovery unit captured for qbittorrent → /mnt/felhom-drives/hdd_1/backups/primary/qbittorrent (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1) +2026/11/11 01:30:18 [INFO] Event pushed: db_dump_completed (info) [hu-only] — Adatbázis mentés elkészült +2026/11/11 01:30:30 [INFO] [busymark] idle again (was: a backup, a restore or an off-site run is running) — an operator reboot of the box is no longer refused for it +2026/11/11 02:30:02 [INFO] [backup] Tier 2 copied opengist → /mnt/felhom-drives/hdd_1/backups/secondary/opengist (191.0 KB, 0 leg(s), 0s) +2026/11/11 02:30:05 [INFO] [backup] Tier 2 copied prowlarr → /mnt/felhom-drives/hdd_1/backups/secondary/prowlarr (6.5 MB, 0 leg(s), 1s) +2026/11/11 02:30:07 [INFO] [backup] Tier 2 copied radarr → /mnt/sys_drive/felhom-data/backups/secondary/radarr (7.0 MB, 0 leg(s), 0s) [SSD: state-only] +2026/11/11 02:30:09 [INFO] [backup] Tier 2 copied sonarr → /mnt/sys_drive/felhom-data/backups/secondary/sonarr (25.7 MB, 0 leg(s), 0s) [SSD: state-only] +2026/11/11 02:30:11 [INFO] [backup] Tier 2 copied qbittorrent → /mnt/sys_drive/felhom-data/backups/secondary/qbittorrent (9.9 MB, 0 leg(s), 0s) [SSD: state-only] +2026/11/11 02:30:11 [INFO] [backup] Tier 2 run complete: 5 app(s) processed (incl. volume-only — F6) +2026/11/11 03:00:00 [INFO] Pruned 8640 old metric rows +2026/11/11 03:15:00 [INFO] [offbox] backup run started (5 app(s) toggled) +2026/11/11 03:15:00 [INFO] [backup] Starting database dump run +2026/11/11 03:15:00 [INFO] [backup] No database containers found +2026/11/11 03:15:00 [INFO] [backup] Stopping opengist for safe volume dump +2026/11/11 03:15:00 [INFO] [backup] Volume dump: opengist/opengist_opengist_data → 182.5 KB +2026/11/11 03:15:00 [INFO] [backup] Restarting opengist after volume dump +2026/11/11 03:15:00 [INFO] [backup] Stopping prowlarr for safe volume dump +2026/11/11 03:15:04 [WARN] [backup] prowlarr: template volume entry ignored — volume "prowlarr_config": class "important" — only "excluded" is meaningful for a named volume +2026/11/11 03:15:04 [INFO] [backup] Volume dump: prowlarr/prowlarr_prowlarr_config → 6.5 MB +2026/11/11 03:15:04 [INFO] [backup] Restarting prowlarr after volume dump +2026/11/11 03:15:04 [INFO] [backup] Stopping radarr for safe volume dump +2026/11/11 03:15:11 [INFO] [backup] Volume dump: radarr/radarr_radarr_config → 7.0 MB +2026/11/11 03:15:11 [INFO] [backup] Restarting radarr after volume dump +2026/11/11 03:15:11 [INFO] [backup] Stopping sonarr for safe volume dump +2026/11/11 03:15:14 [INFO] [busymark] busy: a backup, a restore or an off-site run is running — an operator reboot of the box is refused until it ends +2026/11/11 03:15:15 [INFO] [backup] Volume dump: sonarr/sonarr_sonarr_config → 23.5 MB +2026/11/11 03:15:15 [INFO] [backup] Restarting sonarr after volume dump +2026/11/11 03:15:16 [INFO] [backup] Stopping qbittorrent for safe volume dump +2026/11/11 03:15:16 [WARN] [backup] qbittorrent: template volume entry ignored — volume "qbittorrent_config": class "important" — only "excluded" is meaningful for a named volume +2026/11/11 03:15:17 [INFO] [backup] Volume dump: qbittorrent/qbittorrent_qbittorrent_config → 9.9 MB +2026/11/11 03:15:17 [INFO] [backup] Restarting qbittorrent after volume dump +2026/11/11 03:15:18 [INFO] [backup] App-data backup completed: 0 databases (0 B total), 5 volume dump(s) (18.031s) +2026/11/11 03:15:18 [INFO] [backup] Recovery unit captured for opengist → /mnt/sys_drive/felhom-data/backups/primary/opengist (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/11/11 03:15:18 [INFO] [backup] Recovery unit captured for prowlarr → /mnt/sys_drive/felhom-data/backups/primary/prowlarr (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/11/11 03:15:18 [INFO] [backup] Recovery unit captured for radarr → /mnt/felhom-drives/hdd_1/backups/primary/radarr (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/11/11 03:15:18 [INFO] [backup] Recovery unit captured for sonarr → /mnt/felhom-drives/hdd_1/backups/primary/sonarr (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/11/11 03:15:18 [INFO] [backup] Recovery unit captured for qbittorrent → /mnt/felhom-drives/hdd_1/backups/primary/qbittorrent (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1) +2026/11/11 03:15:18 [INFO] [offbox] pre-push dump leg completed in 18.172s — snapshot pair is coherent +2026/11/11 03:15:21 [WARN] [offbox] prowlarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/11/11 03:15:23 [INFO] [offbox] backed up radarr (/mnt/felhom-drives/hdd_1/backups/primary/radarr, 0 mandatory path(s)) +2026/11/11 03:15:26 [INFO] [offbox] backed up sonarr (/mnt/felhom-drives/hdd_1/backups/primary/sonarr, 0 mandatory path(s)) +2026/11/11 03:15:28 [INFO] [offbox] backed up opengist (/mnt/sys_drive/felhom-data/backups/primary/opengist, 0 mandatory path(s)) +2026/11/11 03:15:28 [WARN] [offbox] prowlarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/11/11 03:15:30 [INFO] [offbox] backed up prowlarr (/mnt/sys_drive/felhom-data/backups/primary/prowlarr, 0 mandatory path(s)) +2026/11/11 03:15:32 [INFO] [offbox] backed up qbittorrent (/mnt/felhom-drives/hdd_1/backups/primary/qbittorrent, 0 mandatory path(s)) +2026/11/11 03:15:33 [INFO] [offbox] retention skipped (after-run): no clean-up window now (not due (last window 2026-11-09T03:16:07Z)) — nothing deleted (decision 68) +2026/11/11 03:15:38 [INFO] [offbox] backup OK: 5 app(s) backed up, 67 snapshot(s), 35s +2026/11/11 03:15:38 [INFO] [update-leg] started (after-offsite): window 02:30, no step starts at or after 07:30 +2026/11/11 03:15:38 [INFO] [update-leg] update leg (after-offsite): done=0 undone=0 held=0 failed=0 skipped=0 in 0s [skipped: ] +RC=0 diff --git a/documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-host-clock.txt b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-host-clock.txt new file mode 100644 index 00000000..09864b59 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-host-clock.txt @@ -0,0 +1,665 @@ +# host-clock demo-felhom taken 2026-11-11T10:03:57Z from DooPlex + +=== in-flight === +checked + +=== clock === +utc 2026-11-11T10:03:24.716263814Z +local 2026-11-11T11:03:24 CET +0100 +/usr/share/zoneinfo/Europe/Budapest +Timezone=Europe/Budapest +LocalRTC=no +CanNTP=yes +NTP=no +NTPSynchronized=no +TimeUSec=Wed 2026-11-11 11:03:24 CET +RTCTimeUSec=Wed 2026-11-11 11:03:21 CET + Local time: Wed 2026-11-11 11:03:24 CET + Universal time: Wed 2026-11-11 10:03:24 UTC + RTC time: Wed 2026-11-11 10:03:21 + Time zone: Europe/Budapest (CET, +0100) +System clock synchronized: no + NTP service: inactive + RTC in local TZ: no + +=== who keeps the time === +chrony: failed + enabled: enabled +chronyd: failed + enabled: alias +systemd-timesyncd: inactive + enabled: not-found +ntp: inactive + enabled: not-found +ntpsec: inactive + enabled: not-found +openntpd: inactive + enabled: not-found +506 Cannot talk to daemon +506 Cannot talk to daemon +Failed to query server: The name org.freedesktop.timesync1 was not provided by any .service files +pool 2.debian.pool.ntp.org iburst +sourcedir /run/chrony-dhcp +sourcedir /etc/chrony/sources.d +keyfile /etc/chrony/chrony.keys +driftfile /var/lib/chrony/chrony.drift +ntsdumpdir /var/lib/chrony +logdir /var/log/chrony +maxupdateskew 100.0 +rtcsync +makestep 1 3 +leapseclist /usr/share/zoneinfo/leap-seconds.list +confdir /etc/chrony/conf.d +/etc/chrony/conf.d: +README + +/etc/chrony/sources.d: +README + +=== start with no time === +ls: cannot access '/var/lib/systemd/timesync/clock': No such file or directory +/var/lib/chrony/: +total 4 +-rw-r--r-- 1 _chrony _chrony 42 2026-10-08 04:39:08.514818246 +0200 chrony.drift +2026-11-11 11:03:21.936805+01:00 +1794391402 +1794391404 + +=== boots === +2026-11-08 05:35:18 + -5 46081f567f384ab88591d4e604555a48 Sun 2026-10-04 21:43:33 CEST Sun 2026-10-04 21:43:46 CEST + -4 e82176ae7cf644518507468fc609eaa9 Wed 2026-10-14 15:43:07 CEST Fri 2026-10-16 05:09:06 CEST + -3 ddd3d7305e244b2a9eafb5cf229476f1 Fri 2026-10-16 05:09:23 CEST Wed 2026-10-21 05:23:34 CEST + -2 56db71938b304a57bdedf30f89860e1f Wed 2026-10-21 05:23:48 CEST Thu 2026-10-29 04:47:58 CET + -1 4f6d8b388e224b8ca8ed45e06a2f96b9 Thu 2026-10-29 04:48:11 CET Sun 2026-11-08 05:35:13 CET + 0 72899d2625244e069e8e8099775d1e51 Sun 2026-11-08 05:35:21 CET Wed 2026-11-11 11:03:24 CET + +=== timers === +NEXT LEFT LAST PASSED UNIT ACTIVATES +Wed 2026-11-11 11:03:50 CET 25s Wed 2026-11-11 11:02:50 CET 34s ago felhom-mgmt-watchdog.timer felhom-mgmt-watchdog.service +Wed 2026-11-11 11:56:50 CET 53min Wed 2026-11-11 10:56:50 CET 6min ago felhom-crash-guard-check.timer felhom-crash-guard-check.service +Thu 2026-11-12 00:00:00 CET 12h Wed 2026-11-11 00:00:30 CET 11h ago dpkg-db-backup.timer dpkg-db-backup.service +Thu 2026-11-12 00:36:19 CET 13h Wed 2026-11-11 00:40:45 CET 10h ago logrotate.timer logrotate.service +Thu 2026-11-12 00:37:37 CET 13h Wed 2026-11-11 10:46:55 CET 16min ago apt-daily.timer apt-daily.service +Thu 2026-11-12 05:41:31 CET 18h Wed 2026-11-11 01:11:21 CET 9h ago pve-daily-update.timer pve-daily-update.service +Thu 2026-11-12 05:50:49 CET 18h Wed 2026-11-11 05:50:49 CET 5h 12min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service +Thu 2026-11-12 06:54:07 CET 19h Wed 2026-11-11 06:40:15 CET 4h 23min ago apt-daily-upgrade.timer apt-daily-upgrade.service +Thu 2026-11-12 07:50:39 CET 20h Wed 2026-11-11 10:09:10 CET 54min ago man-db.timer man-db.service +Sun 2026-11-15 03:10:21 CET 3 days Sun 2026-11-08 03:10:44 CET - xfs_scrub_all.timer xfs_scrub_all.service +Sun 2026-11-15 03:10:52 CET 3 days Sun 2026-11-08 03:10:44 CET - e2scrub_all.timer e2scrub_all.service +Mon 2026-11-16 01:20:47 CET 4 days Mon 2026-11-09 01:32:30 CET 2 days ago fstrim.timer fstrim.service + +12 timers listed. + +=== timer units (all) === +--- apt-daily-upgrade.timer [enabled/active] +OnCalendar=*-*-* 6:00 +RandomizedDelaySec=60m +Persistent=true +Unit=apt-daily-upgrade.service NextElapseUSecRealtime=Thu 2026-11-12 06:54:07 CET LastTriggerUSec=Wed 2026-11-11 06:40:15 CET +--- apt-daily.timer [enabled/active] +OnCalendar=*-*-* 6,18:00 +RandomizedDelaySec=12h +Persistent=true +Unit=apt-daily.service NextElapseUSecRealtime=Thu 2026-11-12 00:37:37 CET LastTriggerUSec=Wed 2026-11-11 10:46:55 CET +--- apt-listchanges.timer [disabled/inactive] +OnCalendar=hourly +Unit=apt-listchanges.service NextElapseUSecRealtime= LastTriggerUSec= +--- chrony-dnssrv@.timer [disabled/] +OnActiveSec=0 +OnUnitInactiveSec=1h + +--- dpkg-db-backup.timer [enabled/active] +OnCalendar=daily +Persistent=true +Unit=dpkg-db-backup.service NextElapseUSecRealtime=Thu 2026-11-12 00:00:00 CET LastTriggerUSec=Wed 2026-11-11 00:00:30 CET +--- e2scrub_all.timer [enabled/active] +OnCalendar=Sun *-*-* 03:10:00 +RandomizedDelaySec=60 +Persistent=true +Unit=e2scrub_all.service NextElapseUSecRealtime=Sun 2026-11-15 03:10:52 CET LastTriggerUSec=Sun 2026-11-08 03:10:44 CET +--- felhom-crash-guard-check.timer [enabled/active] +OnBootSec=15min +OnUnitActiveSec=1h +Unit=felhom-crash-guard-check.service NextElapseUSecRealtime= LastTriggerUSec=Wed 2026-11-11 10:56:50 CET +--- felhom-mgmt-watchdog.timer [enabled/active] +OnBootSec=30s +OnUnitActiveSec=60s +AccuracySec=5s +Persistent=true +Unit=felhom-mgmt-watchdog.service +Unit=felhom-mgmt-watchdog.service NextElapseUSecRealtime= LastTriggerUSec=Wed 2026-11-11 11:02:50 CET +--- fstrim.timer [enabled/active] +OnCalendar=weekly +AccuracySec=1h +Persistent=true +RandomizedDelaySec=100min +Unit=fstrim.service NextElapseUSecRealtime=Mon 2026-11-16 01:20:47 CET LastTriggerUSec=Mon 2026-11-09 01:32:30 CET +--- logrotate.timer [enabled/active] +OnCalendar=daily +RandomizedDelaySec=1h +Persistent=true +Unit=logrotate.service NextElapseUSecRealtime=Thu 2026-11-12 00:36:19 CET LastTriggerUSec=Wed 2026-11-11 00:40:45 CET +--- man-db.timer [enabled/active] +OnCalendar=daily +RandomizedDelaySec=12h +Persistent=true +Unit=man-db.service NextElapseUSecRealtime=Thu 2026-11-12 07:50:39 CET LastTriggerUSec=Wed 2026-11-11 10:09:10 CET +--- proxmox-firewall.timer [disabled/inactive] +OnBootSec=1s +OnUnitInactiveSec=5s +Unit=proxmox-firewall.service +Unit=proxmox-firewall.service NextElapseUSecRealtime= LastTriggerUSec= +--- pve-daily-update.timer [enabled/active] +OnCalendar=*-*-* 1:00 +RandomizedDelaySec=5h +Persistent=true +Unit=pve-daily-update.service NextElapseUSecRealtime=Thu 2026-11-12 05:41:31 CET LastTriggerUSec=Wed 2026-11-11 01:11:21 CET +--- systemd-tmpfiles-clean.timer [static/active] +OnBootSec=15min +OnUnitActiveSec=1d +Unit=systemd-tmpfiles-clean.service NextElapseUSecRealtime= LastTriggerUSec=Wed 2026-11-11 05:50:49 CET +--- xfs_scrub_all.timer [enabled/active] +OnCalendar=Sun *-*-* 03:10:00 +RandomizedDelaySec=60 +Persistent=true +Unit=xfs_scrub_all.service NextElapseUSecRealtime=Sun 2026-11-15 03:10:21 CET LastTriggerUSec=Sun 2026-11-08 03:10:44 CET +--- zfs-scrub-monthly@.timer [disabled/] +OnCalendar=monthly +Persistent=true +RandomizedDelaySec=1h +Unit=zfs-scrub@%i.service + +--- zfs-scrub-weekly@.timer [disabled/] +OnCalendar=weekly +Persistent=true +RandomizedDelaySec=1h +Unit=zfs-scrub@%i.service + +--- zfs-trim-monthly@.timer [disabled/] +OnCalendar=monthly +Persistent=true +RandomizedDelaySec=1h +Unit=zfs-trim@%i.service + +--- zfs-trim-weekly@.timer [disabled/] +OnCalendar=weekly +Persistent=true +RandomizedDelaySec=1h +Unit=zfs-trim@%i.service + + +=== service limits of felhom + backup timers === +--- apt-daily-upgrade.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 06:40:15 CET +ExecMainExitTimestamp=Wed 2026-11-11 06:40:15 CET +ExecMainStatus=0 +ExecStart={ path=/usr/lib/apt/apt.systemd.daily ; argv[]=/usr/lib/apt/apt.systemd.daily install ; ignore_errors=no ; start_time=[Wed 2026-11-11 06:40:15 CET] ; stop_time=[Wed 2026-11-11 06:40:15 CET] ; pid=809079 ; code=exited ; status=0 } +--- apt-daily.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 10:46:55 CET +ExecMainExitTimestamp=Wed 2026-11-11 10:46:55 CET +ExecMainStatus=0 +ExecStart={ path=/usr/lib/apt/apt.systemd.daily ; argv[]=/usr/lib/apt/apt.systemd.daily update ; ignore_errors=no ; start_time=[Wed 2026-11-11 10:46:55 CET] ; stop_time=[Wed 2026-11-11 10:46:55 CET] ; pid=1088520 ; code=exited ; status=0 } +--- apt-listchanges.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp= +ExecMainExitTimestamp= +ExecMainStatus=0 +ExecStart={ path=python3 ; argv[]=python3 -m apt_listchanges.populate_database --profile apt ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } +--- dpkg-db-backup.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 00:00:30 CET +ExecMainExitTimestamp=Wed 2026-11-11 00:00:30 CET +ExecMainStatus=0 +ExecStart={ path=/usr/libexec/dpkg/dpkg-db-backup ; argv[]=/usr/libexec/dpkg/dpkg-db-backup ; ignore_errors=no ; start_time=[Wed 2026-11-11 00:00:30 CET] ; stop_time=[Wed 2026-11-11 00:00:30 CET] ; pid=345759 ; code=exited ; status=0 } +--- e2scrub_all.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp= +ExecMainExitTimestamp= +ExecMainStatus=0 +ExecStart={ path=/sbin/e2scrub_all ; argv[]=/sbin/e2scrub_all ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } +--- felhom-crash-guard-check.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 10:56:50 CET +ExecMainExitTimestamp=Wed 2026-11-11 10:56:50 CET +ExecMainStatus=0 +ExecStart={ path=/usr/local/sbin/felhom-crash-guard ; argv[]=/usr/local/sbin/felhom-crash-guard check ; ignore_errors=no ; start_time=[Wed 2026-11-11 10:56:50 CET] ; stop_time=[Wed 2026-11-11 10:56:50 CET] ; pid=1099750 ; code=exited ; status=0 } +--- felhom-mgmt-watchdog.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 11:02:50 CET +ExecMainExitTimestamp=Wed 2026-11-11 11:02:50 CET +ExecMainStatus=0 +ExecStart={ path=/usr/local/sbin/felhom-mgmt-watchdog ; argv[]=/usr/local/sbin/felhom-mgmt-watchdog ; ignore_errors=no ; start_time=[Wed 2026-11-11 11:02:50 CET] ; stop_time=[Wed 2026-11-11 11:02:50 CET] ; pid=1106395 ; code=exited ; status=0 } +--- fstrim.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Mon 2026-11-09 01:32:30 CET +ExecMainExitTimestamp=Mon 2026-11-09 01:32:46 CET +ExecMainStatus=0 +ExecStart={ path=/sbin/fstrim ; argv[]=/sbin/fstrim --listed-in /etc/fstab:/proc/self/mountinfo --verbose --quiet-unsupported ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } +--- logrotate.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 00:40:45 CET +ExecMainExitTimestamp=Wed 2026-11-11 00:40:47 CET +ExecMainStatus=0 +ExecStart={ path=/usr/sbin/logrotate ; argv[]=/usr/sbin/logrotate /etc/logrotate.conf ; ignore_errors=no ; start_time=[Wed 2026-11-11 00:40:45 CET] ; stop_time=[Wed 2026-11-11 00:40:47 CET] ; pid=391150 ; code=exited ; status=0 } +--- man-db.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 10:09:10 CET +ExecMainExitTimestamp=Wed 2026-11-11 10:09:10 CET +ExecMainStatus=0 +ExecStart={ path=/usr/bin/install ; argv[]=/usr/bin/install -d -o man -g man -m 0755 /var/cache/man ; ignore_errors=no ; start_time=[Wed 2026-11-11 10:09:10 CET] ; stop_time=[Wed 2026-11-11 10:09:10 CET] ; pid=1044483 ; code=exited ; status=0 } +ExecStart={ path=/usr/bin/mandb ; argv[]=/usr/bin/mandb --quiet ; ignore_errors=no ; start_time=[Wed 2026-11-11 10:09:10 CET] ; stop_time=[Wed 2026-11-11 10:09:10 CET] ; pid=1044490 ; code=exited ; status=0 } +--- pve-daily-update.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 01:11:21 CET +ExecMainExitTimestamp=Wed 2026-11-11 01:11:25 CET +ExecMainStatus=0 +ExecStart={ path=/usr/bin/pveupdate ; argv[]=/usr/bin/pveupdate ; ignore_errors=no ; start_time=[Wed 2026-11-11 01:11:21 CET] ; stop_time=[Wed 2026-11-11 01:11:25 CET] ; pid=425598 ; code=exited ; status=0 } + +=== felhom units that are not timers === +felhom-agent-rollback.service loaded inactive dead +felhom-agent.service loaded active running +felhom-crash-guard-check.service loaded inactive dead +felhom-crash-guard.service loaded active exited +felhom-mgmt-watchdog.service loaded inactive dead +felhom-oob-nft.service loaded active exited +felhom-shared-parent.service loaded active exited +felhom-sshd.service loaded active running +felhom-crash-guard-check.timer loaded active waiting +felhom-mgmt-watchdog.timer loaded active waiting +--- felhom-agent-rollback.service +Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=infinity NRestarts=0 ActiveEnterTimestamp= +--- felhom-agent.service +Restart=on-failure RestartUSec=5s TimeoutStartUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:50:48 CET +--- felhom-bootstrap.service +Restart=on-failure RestartUSec=30s TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=infinity NRestarts=0 ActiveEnterTimestamp= +--- felhom-crash-guard-check.service +Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp= +--- felhom-crash-guard.service +Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Sun 2026-11-08 05:35:23 CET +--- felhom-mgmt-watchdog.service +Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp= +--- felhom-oob-nft.service +Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Sun 2026-11-08 05:35:23 CET +--- felhom-shared-parent.service +Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Sun 2026-11-08 05:35:23 CET +--- felhom-sshd.service +Restart=on-failure RestartUSec=100ms TimeoutStartUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Sun 2026-11-08 05:35:24 CET + +=== cron === +SHELL=/bin/sh +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +17 * * * * root cd / && run-parts --report /etc/cron.hourly +25 6 * * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; } +47 6 * * 7 root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.weekly; } +52 6 1 * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.monthly; } +--- /etc/cron.d/e2scrub_all +30 3 * * 0 root test -e /run/systemd/system || SERVICE_MODE=1 /usr/libexec/e2fsprogs/e2scrub_all_cron +10 3 * * * root test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r +--- /etc/cron.d/vzdump +PATH="/usr/sbin:/usr/bin:/sbin:/bin" +--- /etc/cron.d/zfsutils-linux +PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin +24 0 1-7 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/trim ]; then /usr/lib/zfs-linux/trim; fi +24 0 8-14 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs-linux/scrub; fi +--- cron.hourly: +--- cron.daily: apt-compat dpkg logrotate man-db +--- cron.weekly: man-db +--- cron.monthly: +--- user crontabs: + +=== at === +no at + +=== pve jobs === +--- vzdump.cron +PATH="/usr/sbin:/usr/bin:/sbin:/bin" +--- vzdump.conf + +=== pve tasks (start end type id status), last 14 days === +awk: line 2: function strtonum never defined +awk: line 2: function strtonum never defined + +=== runs of timer services, last 10 days (start/finish lines) === +--- apt-daily-upgrade.service +2026-11-09T06:30:32+01:00 demo-felhom systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities... +2026-11-09T06:30:32+01:00 demo-felhom systemd[1]: apt-daily-upgrade.service: Deactivated successfully. +2026-11-09T06:30:32+01:00 demo-felhom systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities. +2026-11-10T06:03:00+01:00 demo-felhom systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities... +2026-11-10T06:03:00+01:00 demo-felhom systemd[1]: apt-daily-upgrade.service: Deactivated successfully. +2026-11-10T06:03:00+01:00 demo-felhom systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities. +2026-11-11T06:40:15+01:00 demo-felhom systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities... +2026-11-11T06:40:15+01:00 demo-felhom systemd[1]: apt-daily-upgrade.service: Deactivated successfully. +2026-11-11T06:40:15+01:00 demo-felhom systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities. +--- apt-daily.service +2026-11-10T13:12:32+01:00 demo-felhom systemd[1]: Starting apt-daily.service - Daily apt download activities... +2026-11-10T13:12:32+01:00 demo-felhom systemd[1]: apt-daily.service: Deactivated successfully. +2026-11-10T13:12:32+01:00 demo-felhom systemd[1]: Finished apt-daily.service - Daily apt download activities. +2026-11-11T01:34:49+01:00 demo-felhom systemd[1]: Starting apt-daily.service - Daily apt download activities... +2026-11-11T01:34:50+01:00 demo-felhom systemd[1]: apt-daily.service: Deactivated successfully. +2026-11-11T01:34:50+01:00 demo-felhom systemd[1]: Finished apt-daily.service - Daily apt download activities. +2026-11-11T10:46:55+01:00 demo-felhom systemd[1]: Starting apt-daily.service - Daily apt download activities... +2026-11-11T10:46:55+01:00 demo-felhom systemd[1]: apt-daily.service: Deactivated successfully. +2026-11-11T10:46:55+01:00 demo-felhom systemd[1]: Finished apt-daily.service - Daily apt download activities. +--- felhom-crash-guard-check.service +2026-11-11T08:56:50+01:00 demo-felhom systemd[1]: Starting felhom-crash-guard-check.service - Felhom crash guard re-arm check... +2026-11-11T08:56:50+01:00 demo-felhom systemd[1]: felhom-crash-guard-check.service: Deactivated successfully. +2026-11-11T08:56:50+01:00 demo-felhom systemd[1]: Finished felhom-crash-guard-check.service - Felhom crash guard re-arm check. +2026-11-11T09:56:50+01:00 demo-felhom systemd[1]: Starting felhom-crash-guard-check.service - Felhom crash guard re-arm check... +2026-11-11T09:56:50+01:00 demo-felhom systemd[1]: felhom-crash-guard-check.service: Deactivated successfully. +2026-11-11T09:56:50+01:00 demo-felhom systemd[1]: Finished felhom-crash-guard-check.service - Felhom crash guard re-arm check. +2026-11-11T10:56:50+01:00 demo-felhom systemd[1]: Starting felhom-crash-guard-check.service - Felhom crash guard re-arm check... +2026-11-11T10:56:50+01:00 demo-felhom systemd[1]: felhom-crash-guard-check.service: Deactivated successfully. +2026-11-11T10:56:50+01:00 demo-felhom systemd[1]: Finished felhom-crash-guard-check.service - Felhom crash guard re-arm check. +--- felhom-mgmt-watchdog.service +2026-11-11T11:00:49+01:00 demo-felhom systemd[1]: Starting felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)... +2026-11-11T11:00:49+01:00 demo-felhom systemd[1]: felhom-mgmt-watchdog.service: Deactivated successfully. +2026-11-11T11:00:49+01:00 demo-felhom systemd[1]: Finished felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent). +2026-11-11T11:01:50+01:00 demo-felhom systemd[1]: Starting felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)... +2026-11-11T11:01:50+01:00 demo-felhom systemd[1]: felhom-mgmt-watchdog.service: Deactivated successfully. +2026-11-11T11:01:50+01:00 demo-felhom systemd[1]: Finished felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent). +2026-11-11T11:02:50+01:00 demo-felhom systemd[1]: Starting felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)... +2026-11-11T11:02:50+01:00 demo-felhom systemd[1]: felhom-mgmt-watchdog.service: Deactivated successfully. +2026-11-11T11:02:50+01:00 demo-felhom systemd[1]: Finished felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent). +--- pve-daily-update.service +2026-11-09T02:07:09+01:00 demo-felhom systemd[1]: pve-daily-update.service: Consumed 3.774s CPU time, 234.8M memory peak. +2026-11-10T01:23:25+01:00 demo-felhom systemd[1]: Starting pve-daily-update.service - Daily PVE download activities... +2026-11-10T01:23:29+01:00 demo-felhom systemd[1]: pve-daily-update.service: Deactivated successfully. +2026-11-10T01:23:29+01:00 demo-felhom systemd[1]: Finished pve-daily-update.service - Daily PVE download activities. +2026-11-10T01:23:29+01:00 demo-felhom systemd[1]: pve-daily-update.service: Consumed 3.725s CPU time, 234.2M memory peak. +2026-11-11T01:11:21+01:00 demo-felhom systemd[1]: Starting pve-daily-update.service - Daily PVE download activities... +2026-11-11T01:11:25+01:00 demo-felhom systemd[1]: pve-daily-update.service: Deactivated successfully. +2026-11-11T01:11:25+01:00 demo-felhom systemd[1]: Finished pve-daily-update.service - Daily PVE download activities. +2026-11-11T01:11:25+01:00 demo-felhom systemd[1]: pve-daily-update.service: Consumed 3.757s CPU time, 234.8M memory peak. + +=== agent log: schedule words, last 3 days (tail 260) === +2026-11-10T00:20:42+01:00 demo-felhom sudo[2915765]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T00:35:42+01:00 demo-felhom sudo[2932690]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T00:50:42+01:00 demo-felhom sudo[2949587]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T01:05:42+01:00 demo-felhom sudo[2966528]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T01:20:42+01:00 demo-felhom sudo[2983412]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T01:35:42+01:00 demo-felhom sudo[3000347]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T01:50:42+01:00 demo-felhom sudo[3017217]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T02:05:42+01:00 demo-felhom sudo[3034102]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T02:20:42+01:00 demo-felhom sudo[3051036]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T02:35:42+01:00 demo-felhom sudo[3070754]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T02:50:42+01:00 demo-felhom sudo[3087822]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T03:05:42+01:00 demo-felhom sudo[3104852]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T03:20:42+01:00 demo-felhom sudo[3121847]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T03:35:42+01:00 demo-felhom sudo[3139471]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T03:50:42+01:00 demo-felhom sudo[3156417]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T04:05:42+01:00 demo-felhom sudo[3173412]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T04:20:42+01:00 demo-felhom sudo[3193437]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T04:35:42+01:00 demo-felhom sudo[3210510]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T04:50:42+01:00 demo-felhom sudo[3227558]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T05:05:42+01:00 demo-felhom sudo[3244549]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T05:20:42+01:00 demo-felhom sudo[3261530]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T05:34:26+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:34:26.031+01:00 level=INFO msg="backup: space preflight passed" vmid=9201 target=felhom-backup last_archive_bytes=1528966739 need_bytes=2984950247 avail_bytes=924950589440 +2026-11-10T05:34:27+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:34:27.065+01:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=felhom-backup job=backup-9201-1794285265667942744 +2026-11-10T05:35:12+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:35:12.208+01:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-backup archive=felhom-backup:backup/vzdump-lxc-9201-2026_11_10-05_34_26.tar.zst size_bytes=1299653068 uncove +2026-11-10T05:35:12+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:35:12.208+01:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=felhom-backup job=backup-9201-1794285265667942744 archive=felhom-backup:backup/vzdump-lxc-9201-2026_11 +2026-11-10T05:35:33+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:35:33.864+01:00 level=INFO msg="janitor: stale-lock sweep deferred — a heavy operation is in flight" busy=backup:felhom-backup +2026-11-10T05:35:40+01:00 demo-felhom sudo[3280846]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T05:35:42+01:00 demo-felhom sudo[3281005]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T05:35:52+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:35:52.839+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-11-10T05:36:42+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:36:42.210+01:00 level=INFO msg="osupdate: START" run=20261110T043642Z layer=guest vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261110T043642Z +2026-11-10T05:36:56+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:36:56.012+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261110T043642Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0" +2026-11-10T05:36:56+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:36:56.012+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-10T05:36:56+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:36:56.012+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-11-10T05:36:56+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:36:56.012+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-11-10T05:36:56+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:36:56.013+01:00 level=INFO msg="osupdate: DONE" run=20261110T043642Z layer=guest vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered +2026-11-10T05:36:56+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:36:56.023+01:00 level=INFO msg="osupdate: START" run=20261110T043642Z layer=host vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261110T043642Z +2026-11-10T05:37:06+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:06.868+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261110T043642Z layer=host lane=fast mode=apply select=pending-fast packages=0" +2026-11-10T05:37:06+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:06.868+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-10T05:37:06+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:06.868+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-11-10T05:37:06+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:06.868+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-11-10T05:37:07+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:07.688+01:00 level=INFO msg="osupdate: DONE" run=20261110T043642Z layer=host vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=14 not_covered +2026-11-10T05:37:09+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:09.362+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE already on" +2026-11-10T05:37:09+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:09.362+01:00 level=INFO msg="osupdate: live-restore" vmid=9201 result="{\"result\": \"already on\"}" +2026-11-10T05:37:09+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:09.362+01:00 level=INFO msg="osupdate: START" run=20261110T043642Z layer=docker vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261110T043642Z +2026-11-10T05:37:31+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:31.993+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261110T043642Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authori +2026-11-10T05:37:31+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:31.993+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-10T05:37:31+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:31.993+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-11-10T05:37:31+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:31.993+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-11-10T05:37:31+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:31.993+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: OOM-CHECK result=pass oom_killed=True oom_event=True exit=137 image=gitea.dooplex.hu/admin/felhom-controller:0.351.0 +2026-11-10T05:37:31+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:31.993+01:00 level=INFO msg="osupdate: DONE" run=20261110T043642Z layer=docker vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covere +2026-11-10T05:37:32+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:32.001+01:00 level=INFO msg="osupdate: pve step holds the /etc/pve write gate — the agent's own writes wait until it ends" run=20261110T043642Z layer=pve vmid=9201 trigger=night +2026-11-10T05:37:32+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:32.001+01:00 level=INFO msg="osupdate: START" run=20261110T043642Z layer=pve vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261110T043642Z +2026-11-10T05:37:43+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:43.960+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261110T043642Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=rin +2026-11-10T05:37:43+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:43.960+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-10T05:37:43+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:43.960+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-11-10T05:37:43+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:43.960+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-11-10T05:37:44+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:44.758+01:00 level=INFO msg="osupdate: DONE" run=20261110T043642Z layer=pve vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=1 not_covered=0 +2026-11-10T05:37:44+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:44.779+01:00 level=INFO msg="osupdate: pve step released the /etc/pve write gate" run=20261110T043642Z layer=pve vmid=9201 trigger=night +2026-11-10T05:37:44+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:37:44.779+01:00 level=INFO msg="osupdate: kernel step skipped — the hub names no kernel step for this box" run=20261110T043642Z layer=kernel vmid=9201 trigger=night ring=0 +2026-11-10T05:39:27+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:39:27.093+01:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=felhom-pbs job=backup-9201-felhom-pbs-1794285565692471312 +2026-11-10T05:40:12+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:40:12.531+01:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-11-10T04:39:26Z size_bytes=3502516951 uncovered_volumes=2 +2026-11-10T05:40:12+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T05:40:12.531+01:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=felhom-pbs job=backup-9201-felhom-pbs-1794285565692471312 archive=felhom-pbs:backup/ct/9201/2026-11-10 +2026-11-10T05:50:42+01:00 demo-felhom sudo[3302512]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T06:05:42+01:00 demo-felhom sudo[3319580]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T06:06:34+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T06:06:34.950+01:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=felhom-backup archive=felhom-backup:backup/vzdump-lxc-9201- +2026-11-10T06:06:35+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T06:06:35.818+01:00 level=INFO msg="restore-test: space preflight passed" storage=local-lvm required_bytes=10566459392 avail_bytes=363190239731 +2026-11-10T06:06:35+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T06:06:35.841+01:00 level=INFO msg="restore-test: full-fidelity restore params derived from the archive config" scratch=990000 params=4 +2026-11-10T06:07:34+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T06:07:34.438+01:00 level=INFO msg="restore-test: scratch guest torn down" vmid=990000 +2026-11-10T06:07:34+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T06:07:34.439+01:00 level=INFO msg="backup: scheduled restore-test passed" archive=felhom-backup:backup/vzdump-lxc-9201-2026_11_09-05_31_03.tar.zst duration_s=59.485928852 +2026-11-10T06:20:42+01:00 demo-felhom sudo[3337978]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T06:35:42+01:00 demo-felhom sudo[3354915]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T06:50:42+01:00 demo-felhom sudo[3371871]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T07:05:42+01:00 demo-felhom sudo[3388825]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T07:20:42+01:00 demo-felhom sudo[3405797]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T07:35:42+01:00 demo-felhom sudo[3422762]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T07:50:42+01:00 demo-felhom sudo[3439766]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T08:05:42+01:00 demo-felhom sudo[3456667]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T08:20:42+01:00 demo-felhom sudo[3473575]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T08:35:42+01:00 demo-felhom sudo[3491405]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T08:50:42+01:00 demo-felhom sudo[3509858]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T09:05:42+01:00 demo-felhom sudo[3526966]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T09:20:42+01:00 demo-felhom sudo[3543898]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T09:35:42+01:00 demo-felhom sudo[3560731]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T09:50:42+01:00 demo-felhom sudo[3577606]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T10:05:42+01:00 demo-felhom sudo[3594532]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T10:20:42+01:00 demo-felhom sudo[3611505]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T10:35:42+01:00 demo-felhom sudo[3628405]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T10:50:42+01:00 demo-felhom sudo[3645823]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T11:05:42+01:00 demo-felhom sudo[3662674]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T11:20:42+01:00 demo-felhom sudo[3679581]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T11:35:40+01:00 demo-felhom sudo[3696377]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T11:35:42+01:00 demo-felhom sudo[3696468]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T11:35:54+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T11:35:54.826+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=3 +2026-11-10T11:50:42+01:00 demo-felhom sudo[3713367]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T12:05:42+01:00 demo-felhom sudo[3730217]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T12:06:35+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T12:06:35.294+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-03T04:35:12Z) is already proven; felhom-ba +2026-11-10T12:20:42+01:00 demo-felhom sudo[3747183]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T12:35:42+01:00 demo-felhom sudo[3764104]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T12:50:42+01:00 demo-felhom sudo[3781025]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T13:05:42+01:00 demo-felhom sudo[3797895]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T13:20:42+01:00 demo-felhom sudo[3814879]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T13:35:42+01:00 demo-felhom sudo[3831759]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T13:50:42+01:00 demo-felhom sudo[3848706]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T14:05:42+01:00 demo-felhom sudo[3865605]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T14:20:42+01:00 demo-felhom sudo[3882602]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T14:35:42+01:00 demo-felhom sudo[3899445]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T14:50:42+01:00 demo-felhom sudo[3916354]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:05:42+01:00 demo-felhom sudo[3933239]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:20:42+01:00 demo-felhom sudo[3950147]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:35:42+01:00 demo-felhom sudo[3967102]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:50:42+01:00 demo-felhom sudo[3984749]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:50:46+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T15:50:46.327+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.173.0 report="{\"agent_update\": {\"sha256\": \"752b2727fb604 +2026-11-10T15:50:48+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T15:50:48.430+01:00 level=INFO msg="pbs: verify loop shutting down" reason="context canceled" +2026-11-10T15:50:48+01:00 demo-felhom felhom-agent[1206]: time=2026-11-10T15:50:48.430+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled" +2026-11-10T15:50:48+01:00 demo-felhom systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)... +2026-11-10T15:50:48+01:00 demo-felhom systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T15:50:48+01:00 demo-felhom systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T15:50:48+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:48.463+01:00 level=INFO msg="backup tier armed" target=felhom-backup cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true +2026-11-10T15:50:48+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:48.463+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false +2026-11-10T15:50:48+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:48.464+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)" +2026-11-10T15:50:48+01:00 demo-felhom sudo[3985075]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T145048Z-kernel-kernel-boot.json +2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.046+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s +2026-11-10T15:50:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:49.046+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s +2026-11-10T15:50:49+01:00 demo-felhom sudo[3985235]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0 +2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s +2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup +2026-11-10T15:50:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:50:50.001+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; nev +2026-11-10T15:50:50+01:00 demo-felhom sudo[3985377]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:51:01+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:01.241+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=3 +2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.032+01:00 level=INFO msg="osupdate: running the bundle policy pass after the self-update commit (R-1285)" agent_version=0.173.0 +2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.121+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE POLICY DONE install_mode=appliance removed=0" +2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.122+01:00 level=INFO msg="osupdate: bundle policy pass done" bundle_policy="{\"install_mode\": \"appliance\", \"removed\": []}" +2026-11-10T15:51:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T15:51:50.128+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.173.0 removed=48 +2026-11-10T16:05:54+01:00 demo-felhom sudo[4002880]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T16:20:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T16:20:49.047+01:00 level=INFO msg="backup: restore-test first evaluation after start (R-874)" after=30m0s +2026-11-10T16:20:49+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T16:20:49.736+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-03T04:35:12Z) is already proven; felhom +2026-11-10T16:20:54+01:00 demo-felhom sudo[4020715]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T16:28:44+01:00 demo-felhom sudo[4029648]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T16:35:54+01:00 demo-felhom sudo[4038486]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T16:50:54+01:00 demo-felhom sudo[4055454]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T17:05:54+01:00 demo-felhom sudo[4072670]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T17:20:54+01:00 demo-felhom sudo[4089639]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T17:35:54+01:00 demo-felhom sudo[4106493]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T17:50:54+01:00 demo-felhom sudo[4123369]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T18:05:54+01:00 demo-felhom sudo[4140267]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T18:20:54+01:00 demo-felhom sudo[4157202]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T18:35:54+01:00 demo-felhom sudo[4174095]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T18:50:54+01:00 demo-felhom sudo[4190953]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T19:05:54+01:00 demo-felhom sudo[14085]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T19:20:54+01:00 demo-felhom sudo[30996]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T19:35:54+01:00 demo-felhom sudo[47836]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T19:50:54+01:00 demo-felhom sudo[64771]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T20:05:54+01:00 demo-felhom sudo[81620]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T20:20:54+01:00 demo-felhom sudo[98549]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T20:35:54+01:00 demo-felhom sudo[115453]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T20:50:54+01:00 demo-felhom sudo[132275]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T21:05:54+01:00 demo-felhom sudo[149156]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T21:20:54+01:00 demo-felhom sudo[166106]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T21:35:54+01:00 demo-felhom sudo[183009]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T21:50:54+01:00 demo-felhom sudo[199912]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T21:51:01+01:00 demo-felhom sudo[200244]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T21:51:15+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T21:51:15.415+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=3 +2026-11-10T22:05:54+01:00 demo-felhom sudo[216811]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T22:20:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-10T22:20:50.434+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-03T04:35:12Z) is already proven; felhom +2026-11-10T22:20:54+01:00 demo-felhom sudo[233785]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T22:35:54+01:00 demo-felhom sudo[250624]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T22:50:54+01:00 demo-felhom sudo[267511]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T23:05:54+01:00 demo-felhom sudo[284411]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T23:20:54+01:00 demo-felhom sudo[301342]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T23:35:54+01:00 demo-felhom sudo[318204]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T23:50:54+01:00 demo-felhom sudo[335082]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T00:05:54+01:00 demo-felhom sudo[351965]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T00:20:54+01:00 demo-felhom sudo[368862]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T00:35:54+01:00 demo-felhom sudo[385722]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T00:50:54+01:00 demo-felhom sudo[402645]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T01:05:54+01:00 demo-felhom sudo[419499]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T01:20:54+01:00 demo-felhom sudo[436416]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T01:35:54+01:00 demo-felhom sudo[453352]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T01:50:54+01:00 demo-felhom sudo[470292]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T02:05:54+01:00 demo-felhom sudo[487150]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T02:20:54+01:00 demo-felhom sudo[504080]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T02:35:54+01:00 demo-felhom sudo[524004]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T02:50:54+01:00 demo-felhom sudo[541023]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T03:05:54+01:00 demo-felhom sudo[558023]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T03:20:54+01:00 demo-felhom sudo[575005]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T03:35:54+01:00 demo-felhom sudo[592593]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T03:50:54+01:00 demo-felhom sudo[609616]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T03:51:01+01:00 demo-felhom sudo[610005]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T03:51:15+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T03:51:15.426+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=3 +2026-11-11T04:05:54+01:00 demo-felhom sudo[626547]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T04:20:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T04:20:50.416+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-03T04:35:12Z) is already proven; felhom +2026-11-11T04:20:54+01:00 demo-felhom sudo[646613]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T04:35:54+01:00 demo-felhom sudo[663687]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T04:50:54+01:00 demo-felhom sudo[680674]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T05:05:54+01:00 demo-felhom sudo[697658]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T05:20:54+01:00 demo-felhom sudo[714656]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T05:35:54+01:00 demo-felhom sudo[731604]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T05:36:51+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:36:51.383+01:00 level=INFO msg="backup: space preflight passed" vmid=9201 target=felhom-backup last_archive_bytes=1299653068 need_bytes=2698308159 avail_bytes=925178314752 +2026-11-11T05:36:52+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:36:52.425+01:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=felhom-backup job=backup-9201-1794371810991152516 +2026-11-11T05:37:37+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:37:37.562+01:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-backup archive=felhom-backup:backup/vzdump-lxc-9201-2026_11_11-05_36_51.tar.zst size_bytes=1300036887 unc +2026-11-11T05:37:37+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:37:37.562+01:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=felhom-backup job=backup-9201-1794371810991152516 archive=felhom-backup:backup/vzdump-lxc-9201-2026 +2026-11-11T05:39:07+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:07.567+01:00 level=INFO msg="osupdate: START" run=20261111T043907Z layer=guest vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261111T043907Z +2026-11-11T05:39:21+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:21.549+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261111T043907Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0" +2026-11-11T05:39:21+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:21.549+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-11T05:39:21+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:21.549+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-11-11T05:39:21+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:21.549+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-11-11T05:39:21+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:21.549+01:00 level=INFO msg="osupdate: DONE" run=20261111T043907Z layer=guest vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_cove +2026-11-11T05:39:21+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:21.617+01:00 level=INFO msg="osupdate: START" run=20261111T043907Z layer=host vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261111T043907Z +2026-11-11T05:39:32+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:32.620+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261111T043907Z layer=host lane=fast mode=apply select=pending-fast packages=0" +2026-11-11T05:39:32+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:32.620+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-11T05:39:32+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:32.620+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-11-11T05:39:32+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:32.620+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-11-11T05:39:33+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:33.444+01:00 level=INFO msg="osupdate: DONE" run=20261111T043907Z layer=host vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=15 not_cove +2026-11-11T05:39:35+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:35.096+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE already on" +2026-11-11T05:39:35+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:35.096+01:00 level=INFO msg="osupdate: live-restore" vmid=9201 result="{\"result\": \"already on\"}" +2026-11-11T05:39:35+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:35.096+01:00 level=INFO msg="osupdate: START" run=20261111T043907Z layer=docker vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261111T043907Z +2026-11-11T05:39:57+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:57.910+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261111T043907Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 auth +2026-11-11T05:39:57+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:57.910+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-11T05:39:57+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:57.910+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-11-11T05:39:57+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:57.910+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-11-11T05:39:57+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:57.910+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: OOM-CHECK result=fail oom_killed=False oom_event=False exit=137 image=gitea.dooplex.hu/admin/felhom-controller:0. +2026-11-11T05:39:57+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:57.910+01:00 level=INFO msg="osupdate: DONE" run=20261111T043907Z layer=docker vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_cov +2026-11-11T05:39:57+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:57.917+01:00 level=INFO msg="osupdate: pve step holds the /etc/pve write gate — the agent's own writes wait until it ends" run=20261111T043907Z layer=pve vmid=9201 trigger=nig +2026-11-11T05:39:57+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:39:57.917+01:00 level=INFO msg="osupdate: START" run=20261111T043907Z layer=pve vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261111T043907Z +2026-11-11T05:40:09+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:40:09.469+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261111T043907Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority= +2026-11-11T05:40:09+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:40:09.469+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-11T05:40:09+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:40:09.469+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-11-11T05:40:09+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:40:09.469+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-11-11T05:40:10+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:40:10.295+01:00 level=INFO msg="osupdate: DONE" run=20261111T043907Z layer=pve vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=1 not_covere +2026-11-11T05:40:10+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:40:10.301+01:00 level=INFO msg="osupdate: pve step released the /etc/pve write gate" run=20261111T043907Z layer=pve vmid=9201 trigger=night +2026-11-11T05:40:10+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T05:40:10.301+01:00 level=INFO msg="osupdate: kernel step skipped — the hub names no kernel step for this box" run=20261111T043907Z layer=kernel vmid=9201 trigger=night ring=0 +2026-11-11T05:50:55+01:00 demo-felhom sudo[753077]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T06:05:54+01:00 demo-felhom sudo[770116]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T06:20:54+01:00 demo-felhom sudo[787124]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T06:35:54+01:00 demo-felhom sudo[804022]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T06:50:54+01:00 demo-felhom sudo[821020]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T07:05:54+01:00 demo-felhom sudo[837952]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T07:20:54+01:00 demo-felhom sudo[854940]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T07:35:54+01:00 demo-felhom sudo[871826]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T07:50:54+01:00 demo-felhom sudo[888788]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T08:05:54+01:00 demo-felhom sudo[905669]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T08:20:54+01:00 demo-felhom sudo[922546]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T08:35:54+01:00 demo-felhom sudo[939426]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T08:50:54+01:00 demo-felhom sudo[956204]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T09:05:54+01:00 demo-felhom sudo[973033]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T09:20:54+01:00 demo-felhom sudo[989961]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T09:35:54+01:00 demo-felhom sudo[1006771]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T09:50:54+01:00 demo-felhom sudo[1023597]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T09:51:01+01:00 demo-felhom sudo[1024025]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T09:51:11+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T09:51:11.409+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-11-11T10:05:54+01:00 demo-felhom sudo[1040510]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T10:20:50+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T10:20:50.077+01:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-11-10T +2026-11-11T10:20:51+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T10:20:51.359+01:00 level=INFO msg="restore-test: space preflight passed" storage=local-lvm required_bytes=9571729458 avail_bytes=363152785854 +2026-11-11T10:20:51+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T10:20:51.385+01:00 level=INFO msg="restore-test: full-fidelity restore params derived from the archive config" scratch=990000 params=4 +2026-11-11T10:20:54+01:00 demo-felhom sudo[1057518]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T10:22:20+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T10:22:20.013+01:00 level=INFO msg="restore-test: scratch guest torn down" vmid=990000 +2026-11-11T10:22:20+01:00 demo-felhom felhom-agent[3985063]: time=2026-11-11T10:22:20.014+01:00 level=INFO msg="backup: scheduled restore-test passed" archive=felhom-pbs:backup/ct/9201/2026-11-10T04:39:26Z duration_s=89.933700579 +2026-11-11T10:35:54+01:00 demo-felhom sudo[1075697]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T10:50:54+01:00 demo-felhom sudo[1092663]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage + +=== agent log: line count per hour, last 26 h === + 12376 2026-11-10T09 13173 2026-11-10T10 13176 2026-11-10T11 13191 2026-11-10T12 13181 2026-11-10T13 13166 2026-11-10T14 13300 2026-11-10T15 13246 2026-11-10T16 13200 2026-11-10T17 13190 2026-11-10T18 13190 2026-11-10T19 13181 2026-11-10T20 13196 2026-11-10T21 13184 2026-11-10T22 13156 2026-11-10T23 13189 2026-11-11T00 13192 2026-11-11T01 13168 2026-11-11T02 13284 2026-11-11T03 13175 2026-11-11T04 13292 2026-11-11T05 13183 2026-11-11T06 13186 2026-11-11T07 13195 2026-11-11T08 13199 2026-11-11T09 13207 2026-11-11T10 826 2026-11-11T11 +=== agent version === +felhom-agent 0.173.0 +RC=0 diff --git a/documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-host-night.txt b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-host-night.txt new file mode 100644 index 00000000..efc7e05c --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-host-night.txt @@ -0,0 +1,89 @@ +# host-night demo-felhom taken 2026-11-11T10:09:27Z +=== pve tasks, last 14 days (local time) === +2026-10-29 04:41:48 -> 04:42:35 0.8 min vzdump 9201 OK +2026-10-29 04:46:00 -> 04:47:39 1.6 min stopall OK +2026-10-29 04:46:00 -> 04:47:39 1.6 min vzshutdown 9201 OK +2026-10-29 04:48:21 -> 04:48:25 0.1 min startall OK +2026-10-29 04:48:21 -> 04:48:25 0.1 min vzstart 9201 OK +2026-10-29 05:18:22 -> 05:19:08 0.8 min vzrestore 990000 OK +2026-10-29 05:19:13 -> 05:19:16 0.1 min vzstart 990000 OK +2026-10-29 05:19:19 -> 05:19:21 0.0 min vzdestroy 990000 OK +2026-10-29 06:54:18 -> 06:54:21 0.1 min aptupdate OK +2026-10-30 03:37:42 -> 03:37:46 0.1 min aptupdate OK +2026-10-30 04:46:29 -> 04:47:15 0.8 min vzdump 9201 OK +2026-10-30 05:34:15 -> 05:34:58 0.7 min vzrestore 990000 OK +2026-10-30 05:35:01 -> 05:35:03 0.0 min vzstart 990000 OK +2026-10-30 05:35:07 -> 05:35:09 0.0 min vzdestroy 990000 OK +2026-10-31 02:52:28 -> 02:52:31 0.1 min aptupdate OK +2026-10-31 04:49:27 -> 04:49:48 0.3 min vzrestore 990000 OK +2026-10-31 04:49:49 -> 04:49:51 0.0 min vzstart 990000 OK +2026-10-31 04:49:55 -> 04:49:57 0.0 min vzdestroy 990000 OK +2026-10-31 04:51:30 -> 04:52:20 0.8 min vzdump 9201 OK +2026-11-01 04:55:28 -> 04:55:31 0.1 min aptupdate OK +2026-11-01 04:56:28 -> 04:57:14 0.8 min vzdump 9201 OK +2026-11-01 10:49:27 -> 10:49:48 0.3 min vzrestore 990000 OK +2026-11-01 10:49:49 -> 10:49:51 0.0 min vzstart 990000 OK +2026-11-01 10:49:55 -> 10:49:57 0.0 min vzdestroy 990000 OK +2026-11-02 05:01:29 -> 05:02:15 0.8 min vzdump 9201 OK +2026-11-02 05:47:18 -> 05:47:21 0.1 min aptupdate OK +2026-11-02 10:49:27 -> 10:49:48 0.3 min vzrestore 990000 OK +2026-11-02 10:49:49 -> 10:49:51 0.0 min vzstart 990000 OK +2026-11-02 10:49:55 -> 10:49:57 0.0 min vzdestroy 990000 OK +2026-11-03 05:06:28 -> 05:07:15 0.8 min vzdump 9201 OK +2026-11-03 05:10:20 -> 05:10:30 0.2 min aptupdate OK +2026-11-03 05:35:12 -> 05:36:00 0.8 min vzdump 9201 OK +2026-11-03 09:19:40 -> 09:20:21 0.7 min vzrestore 990000 OK +2026-11-03 09:20:26 -> 09:20:29 0.1 min vzstart 990000 OK +2026-11-03 09:20:32 -> 09:20:35 0.1 min vzdestroy 990000 OK +2026-11-04 03:20:45 -> 03:20:48 0.1 min aptupdate OK +2026-11-04 05:09:34 -> 05:10:23 0.8 min vzdump 9201 OK +2026-11-04 09:20:40 -> 09:22:05 1.4 min vzrestore 990000 OK +2026-11-04 09:22:07 -> 09:22:09 0.0 min vzstart 990000 OK +2026-11-04 09:22:13 -> 09:22:15 0.0 min vzdestroy 990000 OK +2026-11-04 15:20:40 -> 15:21:20 0.7 min vzrestore 990000 OK +2026-11-04 15:21:21 -> 15:21:24 0.1 min vzstart 990000 OK +2026-11-04 15:21:27 -> 15:21:30 0.1 min vzdestroy 990000 OK +2026-11-05 02:09:12 -> 02:09:16 0.1 min aptupdate OK +2026-11-05 05:13:22 -> 05:14:18 0.9 min vzdump 9201 OK +2026-11-05 09:20:40 -> 09:21:22 0.7 min vzrestore 990000 OK +2026-11-05 09:21:26 -> 09:21:29 0.1 min vzstart 990000 OK +2026-11-05 09:21:32 -> 09:21:35 0.1 min vzdestroy 990000 OK +2026-11-06 04:33:41 -> 04:33:45 0.1 min aptupdate OK +2026-11-06 05:18:21 -> 05:19:11 0.8 min vzdump 9201 OK +2026-11-06 09:20:40 -> 09:21:23 0.7 min vzrestore 990000 OK +2026-11-06 09:21:26 -> 09:21:29 0.1 min vzstart 990000 OK +2026-11-06 09:21:32 -> 09:21:41 0.1 min vzdestroy 990000 OK +2026-11-07 05:10:41 -> 05:10:45 0.1 min aptupdate OK +2026-11-07 05:23:21 -> 05:24:10 0.8 min vzdump 9201 OK +2026-11-07 09:20:40 -> 09:21:22 0.7 min vzrestore 990000 OK +2026-11-07 09:21:26 -> 09:21:29 0.1 min vzstart 990000 OK +2026-11-07 09:21:32 -> 09:21:35 0.1 min vzdestroy 990000 OK +2026-11-08 01:08:23 -> 01:08:27 0.1 min aptupdate OK +2026-11-08 05:28:21 -> 05:29:10 0.8 min vzdump 9201 OK +2026-11-08 05:33:27 -> 05:35:06 1.6 min stopall OK +2026-11-08 05:33:27 -> 05:35:06 1.6 min vzshutdown 9201 OK +2026-11-08 05:35:30 -> 05:35:34 0.1 min startall OK +2026-11-08 05:35:30 -> 05:35:34 0.1 min vzstart 9201 OK +2026-11-08 06:05:31 -> 06:06:19 0.8 min vzrestore 990000 OK +2026-11-08 06:06:22 -> 06:06:25 0.1 min vzstart 990000 OK +2026-11-08 06:06:28 -> 06:06:30 0.0 min vzdestroy 990000 OK +2026-11-09 02:07:06 -> 02:07:09 0.1 min aptupdate OK +2026-11-09 05:31:03 -> 05:31:52 0.8 min vzdump 9201 OK +2026-11-09 06:06:36 -> 06:07:20 0.7 min vzrestore 990000 OK +2026-11-09 06:07:22 -> 06:07:24 0.0 min vzstart 990000 OK +2026-11-09 06:07:28 -> 06:07:30 0.0 min vzdestroy 990000 OK +2026-11-10 01:23:26 -> 01:23:29 0.1 min aptupdate OK +2026-11-10 05:34:26 -> 05:35:08 0.7 min vzdump 9201 OK +2026-11-10 05:39:26 -> 05:40:08 0.7 min vzdump 9201 OK +2026-11-10 06:06:36 -> 06:07:21 0.8 min vzrestore 990000 OK +2026-11-10 06:07:22 -> 06:07:24 0.0 min vzstart 990000 OK +2026-11-10 06:07:28 -> 06:07:30 0.0 min vzdestroy 990000 OK +2026-11-11 01:11:22 -> 01:11:25 0.1 min aptupdate OK +2026-11-11 05:36:51 -> 05:37:33 0.7 min vzdump 9201 OK +2026-11-11 10:20:51 -> 10:22:06 1.2 min vzrestore 990000 OK +2026-11-11 10:22:07 -> 10:22:10 0.1 min vzstart 990000 OK +2026-11-11 10:22:13 -> 10:22:16 0.1 min vzdestroy 990000 OK + +=== agent: night words, 20:00 yesterday to 09:00 today (first 3 lines per message) === +--- counts +RC=0 diff --git a/documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-offset.txt b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-offset.txt new file mode 100644 index 00000000..b17d8509 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-felhom-offset.txt @@ -0,0 +1,6 @@ +# offset demo-felhom: box clock minus DooPlex clock (midpoint of the call), seconds +offset=-32.638 rtt=0.135 +offset=-32.641 rtt=0.128 +offset=-32.639 rtt=0.132 +offset=-32.638 rtt=0.133 +offset=-32.642 rtt=0.127 diff --git a/documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-guest-clock.txt b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-guest-clock.txt new file mode 100644 index 00000000..4da6c70a --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-guest-clock.txt @@ -0,0 +1,502 @@ +# guest-clock demo-hp taken 2026-11-11T10:04:23Z from DooPlex + +=== clock === +utc 2026-11-11T10:04:24.512980194Z +local 2026-11-11T10:04:24 UTC +0000 +/usr/share/zoneinfo/Etc/UTC + Local time: Wed 2026-11-11 10:04:24 UTC + Universal time: Wed 2026-11-11 10:04:24 UTC + RTC time: n/a + Time zone: Etc/UTC (UTC, +0000) +System clock synchronized: yes + NTP service: inactive + RTC in local TZ: no +chrony: inactive +systemd-timesyncd: inactive + +=== boots === +2026-11-10 14:43:12 + +=== timers === +NEXT LEFT LAST PASSED UNIT ACTIVATES +Wed 2026-11-11 13:52:01 UTC 3h 47min Tue 2026-11-10 23:40:53 UTC 10h ago apt-daily.timer apt-daily.service +Wed 2026-11-11 14:58:53 UTC 4h 54min Tue 2026-11-10 14:58:53 UTC 19h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service +Thu 2026-11-12 00:00:00 UTC 13h Wed 2026-11-11 00:00:01 UTC 10h ago dpkg-db-backup.timer dpkg-db-backup.service +Thu 2026-11-12 00:04:10 UTC 13h Wed 2026-11-11 00:57:03 UTC 9h ago logrotate.timer logrotate.service +Thu 2026-11-12 00:52:48 UTC 14h Wed 2026-11-11 05:52:53 UTC 4h 11min ago man-db.timer man-db.service +Thu 2026-11-12 06:26:47 UTC 20h Wed 2026-11-11 06:45:13 UTC 3h 19min ago apt-daily-upgrade.timer apt-daily-upgrade.service +Sun 2026-11-15 03:10:58 UTC 3 days Sun 2026-11-08 03:10:50 UTC - e2scrub_all.timer e2scrub_all.service +- - - - fstrim.timer fstrim.service + +8 timers listed. + +=== timer units (all) === +--- apt-daily-upgrade.timer [enabled/active] +OnCalendar=*-*-* 6:00 +RandomizedDelaySec=60m +Persistent=true +Unit=apt-daily-upgrade.service NextElapseUSecRealtime=Thu 2026-11-12 06:26:47 UTC LastTriggerUSec=Wed 2026-11-11 06:45:13 UTC +--- apt-daily.timer [enabled/active] +OnCalendar=*-*-* 6,18:00 +RandomizedDelaySec=12h +Persistent=true +Unit=apt-daily.service NextElapseUSecRealtime=Wed 2026-11-11 13:52:01 UTC LastTriggerUSec=Tue 2026-11-10 23:40:53 UTC +--- apt-listchanges.timer [disabled/inactive] +OnCalendar=hourly +Unit=apt-listchanges.service NextElapseUSecRealtime= LastTriggerUSec= +--- dpkg-db-backup.timer [enabled/active] +OnCalendar=daily +Persistent=true +Unit=dpkg-db-backup.service NextElapseUSecRealtime=Thu 2026-11-12 00:00:00 UTC LastTriggerUSec=Wed 2026-11-11 00:00:01 UTC +--- e2scrub_all.timer [enabled/active] +OnCalendar=Sun *-*-* 03:10:00 +RandomizedDelaySec=60 +Persistent=true +Unit=e2scrub_all.service NextElapseUSecRealtime=Sun 2026-11-15 03:10:58 UTC LastTriggerUSec=Sun 2026-11-08 03:10:50 UTC +--- fstrim.timer [enabled/inactive] +OnCalendar=weekly +AccuracySec=1h +Persistent=true +RandomizedDelaySec=100min +Unit=fstrim.service NextElapseUSecRealtime= LastTriggerUSec= +--- logrotate.timer [enabled/active] +OnCalendar=daily +RandomizedDelaySec=1h +Persistent=true +Unit=logrotate.service NextElapseUSecRealtime=Thu 2026-11-12 00:04:10 UTC LastTriggerUSec=Wed 2026-11-11 00:57:03 UTC +--- man-db.timer [enabled/active] +OnCalendar=daily +RandomizedDelaySec=12h +Persistent=true +Unit=man-db.service NextElapseUSecRealtime=Thu 2026-11-12 00:52:48 UTC LastTriggerUSec=Wed 2026-11-11 05:52:53 UTC +--- systemd-tmpfiles-clean.timer [static/active] +OnBootSec=15min +OnUnitActiveSec=1d +Unit=systemd-tmpfiles-clean.service NextElapseUSecRealtime= LastTriggerUSec=Tue 2026-11-10 14:58:53 UTC + +=== felhom units === +felhom-controller-bootstrap.path loaded active running +felhom-controller-bootstrap.service loaded active exited +felhom-regen-hostkeys.service loaded inactive dead +--- felhom-controller-bootstrap.service +Type=oneshot Restart=no RestartUSec=100ms TimeoutStartUSec=infinity NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:17:04 UTC +--- felhom-regen-hostkeys.service +Type=oneshot Restart=no RestartUSec=100ms TimeoutStartUSec=infinity NRestarts=0 ActiveEnterTimestamp= + +=== cron === +SHELL=/bin/sh +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +27 * * * * root cd / && run-parts --report /etc/cron.hourly +28 6 * * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; } +45 6 * * 7 root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.weekly; } +42 6 1 * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.monthly; } +--- /etc/cron.d/e2scrub_all +57 3 * * 0 root test -e /run/systemd/system || SERVICE_MODE=1 /usr/libexec/e2fsprogs/e2scrub_all_cron +27 3 * * * root test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r +--- cron.hourly: +--- cron.daily: apt-compat dpkg logrotate man-db +--- cron.weekly: man-db +--- cron.monthly: +--- user crontabs: + +=== at === +no at + +=== docker daemon timers === +{ + "features": { + "containerd-snapshotter": false + }, + "live-restore": true, + "log-driver": "json-file", + "log-opts": { + "max-file": "3", + "max-size": "10m" + } +} + +=== containers === +romm rommapp/romm:5.4.1 Up 7 hours (healthy) 7 hours ago +romm-db mariadb:11.8 Up 7 hours (healthy) 7 hours ago +romm-redis redis:7-alpine Up 7 hours (healthy) 7 hours ago +privatebin privatebin/pdo:2.1.0 Up 7 hours (healthy) 7 hours ago +plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0 Up 7 hours (healthy) 7 hours ago +paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:3.4.1 Up 7 hours (healthy) 7 hours ago +paperless-redis redis:7-alpine Up 7 hours (healthy) 7 hours ago +paperless-postgres postgres:18-alpine Up 7 hours (healthy) 7 hours ago +opengist ghcr.io/thomiceli/opengist:1.15 Up 7 hours (healthy) 7 hours ago +kimai kimai/kimai2:2.67.0 Up 7 hours (healthy) 7 hours ago +kimai-db mariadb:11.8 Up 7 hours (healthy) 7 hours ago +home-assistant ghcr.io/home-assistant/home-assistant:2026.10.4 Up 7 hours (healthy) 7 hours ago +docmost docmost/docmost:0.97.0 Up 7 hours (healthy) 7 hours ago +docmost-postgres postgres:18-alpine Up 7 hours (healthy) 7 hours ago +docmost-redis redis:7-alpine Up 7 hours (healthy) 7 hours ago +calibre-web crocodilestick/calibre-web-automated:v4.0.8 Up 7 hours (healthy) 7 hours ago +bookstack lscr.io/linuxserver/bookstack:26.09.2 Up 7 hours (healthy) 7 hours ago +bookstack-db mariadb:12.3 Up 7 hours (healthy) 7 hours ago +bentopdf ghcr.io/alam00000/bentopdf:v2.8.10 Up 7 hours (healthy) 7 hours ago +adventurelog ghcr.io/seanmorley15/adventurelog-backend:v0.13.1 Up 7 hours (healthy) 7 hours ago +adventurelog-postgres postgis/postgis:16-3.5-alpine Up 7 hours (healthy) 7 hours ago +adventurelog-frontend ghcr.io/seanmorley15/adventurelog-frontend:v0.13.1 Up 7 hours (healthy) 7 hours ago +felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 19 hours (healthy) 19 hours ago +traefik traefik:v3.7.17 Up 19 hours 8 days ago +cloudflared cloudflare/cloudflared:2026.10.4 Up 19 hours (healthy) 2 weeks ago +filebrowser gtstef/filebrowser:1.5.8-stable Up 19 hours (healthy) 2 weeks ago + +=== restart policies + healthcheck intervals === +/romm restart=unless-stopped health=30s started=2026-11-11T03:34:23.246874464Z +/romm-db restart=unless-stopped health=10s started=2026-11-11T03:34:11.866673198Z +/romm-redis restart=unless-stopped health=10s started=2026-11-11T03:34:12.322525014Z +/privatebin restart=unless-stopped health=30s started=2026-11-11T03:34:10.436496587Z +/plex restart=unless-stopped health=30s started=2026-11-11T03:34:09.416236529Z +/paperless-webserver restart=unless-stopped health=30s started=2026-11-11T03:34:06.356684002Z +/paperless-redis restart=unless-stopped health=10s started=2026-11-11T03:33:55.572315165Z +/paperless-postgres restart=unless-stopped health=10s started=2026-11-11T03:33:55.30857347Z +/opengist restart=unless-stopped health=30s started=2026-11-11T03:33:54.299740686Z +/kimai restart=unless-stopped health=30s started=2026-11-11T03:33:52.511754977Z +/kimai-db restart=unless-stopped health=10s started=2026-11-11T03:33:46.665886435Z +/home-assistant restart=unless-stopped health=30s started=2026-11-11T03:33:45.366565279Z +/docmost restart=unless-stopped health=30s started=2026-11-11T03:33:38.285678301Z +/docmost-postgres restart=unless-stopped health=10s started=2026-11-11T03:33:27.248437109Z +/docmost-redis restart=unless-stopped health=10s started=2026-11-11T03:33:27.525319794Z +/calibre-web restart=unless-stopped health=30s started=2026-11-11T03:33:26.194198782Z +/bookstack restart=unless-stopped health=30s started=2026-11-11T03:33:25.391109868Z +/bookstack-db restart=unless-stopped health=10s started=2026-11-11T03:33:19.649504107Z +/bentopdf restart=unless-stopped health=30s started=2026-11-11T03:33:18.71207642Z +/adventurelog restart=unless-stopped health=30s started=2026-11-11T03:33:17.943139983Z +/adventurelog-postgres restart=unless-stopped health=10s started=2026-11-11T03:33:12.219942766Z +/adventurelog-frontend restart=unless-stopped health=30s started=2026-11-11T03:33:11.981285328Z +/felhom-controller restart=unless-stopped health=30s started=2026-11-10T15:17:03.918469562Z +/traefik restart=unless-stopped health=none started=2026-11-10T14:43:21.982539486Z +/cloudflared restart=unless-stopped health=30s started=2026-11-10T14:43:21.872203889Z +/filebrowser restart=unless-stopped health=30s started=2026-11-10T14:43:32.41426216Z + +=== controller TZ === +TZ=unset +in-container 2026-11-11T10:04:27 UTC +0000 + +=== controller log: first line, line count per hour === +2026-11-10T23:15:05.505129813Z 2026/11/10 23:15:05 manager.go:687: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml + 9125 2026-11-10T23 12234 2026-11-11T00 12689 2026-11-11T01 12383 2026-11-11T02 13030 2026-11-11T03 12262 2026-11-11T04 12254 2026-11-11T05 12243 2026-11-11T06 12243 2026-11-11T07 12231 2026-11-11T08 12250 2026-11-11T09 1057 2026-11-11T10 +=== controller log: schedule words (tail 320) === +2026-11-11T09:57:05.391680485Z 2026/11/11 09:57:05 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026-11-11T09:57:05.391684032Z 2026/11/11 09:57:05 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T09:57:05.391687198Z 2026/11/11 09:57:05 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026-11-11T09:57:05.391702807Z 2026/11/11 09:57:05 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026-11-11T09:57:05.391706013Z 2026/11/11 09:57:05 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:57:05.398983593Z 2026/11/11 09:57:05 scheduler.go:450: [INFO] [scheduler] Running job: backup-cache +2026-11-11T09:57:05.399062742Z 2026/11/11 09:57:05 scheduler.go:85: [DEBUG] [scheduler] job backup-cache: execution starting +2026-11-11T09:57:05.399096566Z 2026/11/11 09:57:05 scheduler.go:450: [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T09:57:05.399128506Z 2026/11/11 09:57:05 scheduler.go:85: [DEBUG] [scheduler] job offsite-credential-retry: execution starting +2026-11-11T09:57:05.399139557Z 2026/11/11 09:57:05 scheduler.go:467: [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T09:57:05.496882413Z 2026/11/11 09:57:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:57:05.497054418Z 2026/11/11 09:57:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting +2026-11-11T09:57:05.695301873Z 2026/11/11 09:57:05 scheduler.go:467: [INFO] [scheduler] Job docker-socket-users completed (took 305ms) +2026-11-11T09:57:05.711141758Z 2026/11/11 09:57:05 scheduler.go:467: [INFO] [scheduler] Job stack-scan completed (took 320ms) +2026-11-11T09:57:05.716073599Z 2026/11/11 09:57:05 backup.go:1252: [INFO] [backup] Found 21 DB dump files across drives +2026-11-11T09:57:05.726924596Z 2026/11/11 09:57:05 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026-11-11T09:57:05.773888562Z 2026/11/11 09:57:05 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: e8a4549a77a0 romm romm rommapp/romm:5.4.1 +2026-11-11T09:57:05.773952964Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.4.1, not a database) +2026-11-11T09:57:05.773956290Z 2026/11/11 09:57:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=0c0a552f1d66) +2026-11-11T09:57:05.786675764Z 2026/11/11 09:57:05 scheduler.go:467: [INFO] [scheduler] Job system-health completed (took 396ms) +2026-11-11T09:57:05.813792380Z 2026/11/11 09:57:05 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026-11-11T09:57:05.813834600Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026-11-11T09:57:05.813858254Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.1.0, not a database) +2026-11-11T09:57:05.813862582Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container plex (image=plexinc/pms-docker:1.43.5.11029-6bf8af3f0, not a database) +2026-11-11T09:57:05.813881508Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:3.4.1, not a database) +2026-11-11T09:57:05.813886347Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026-11-11T09:57:05.813889844Z 2026/11/11 09:57:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=d5dc3d71319d) +2026-11-11T09:57:05.813893120Z 2026/11/11 09:57:05 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026-11-11T09:57:05.849834747Z 2026/11/11 09:57:05 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026-11-11T09:57:05.849868540Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container opengist (image=ghcr.io/thomiceli/opengist:1.15, not a database) +2026-11-11T09:57:05.849872859Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container kimai (image=kimai/kimai2:2.67.0, not a database) +2026-11-11T09:57:05.849876555Z 2026/11/11 09:57:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: kimai-db (id=170178b88458) +2026-11-11T09:57:05.885779890Z 2026/11/11 09:57:05 dbdump.go:198: [DEBUG] DiscoverDatabases: kimai-db → stack=kimai, dbUser=root, dbName=kimai +2026-11-11T09:57:05.885810207Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container home-assistant (image=ghcr.io/home-assistant/home-assistant:2026.10.4, not a database) +2026-11-11T09:57:05.885814836Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost (image=docmost/docmost:0.97.0, not a database) +2026-11-11T09:57:05.885818413Z 2026/11/11 09:57:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: docmost-postgres (id=ed402849ac4c) +2026-11-11T09:57:05.916260477Z 2026/11/11 09:57:05 dbdump.go:198: [DEBUG] DiscoverDatabases: docmost-postgres → stack=docmost, dbUser=docmost, dbName=docmost +2026-11-11T09:57:05.916299420Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost-redis (image=redis:7-alpine, not a database) +2026-11-11T09:57:05.916303839Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container calibre-web (image=crocodilestick/calibre-web-automated:v4.0.8, not a database) +2026-11-11T09:57:05.916307465Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container bookstack (image=lscr.io/linuxserver/bookstack:26.09.2, not a database) +2026-11-11T09:57:05.916310651Z 2026/11/11 09:57:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: bookstack-db (id=bd002743cdb7) +2026-11-11T09:57:05.947648136Z 2026/11/11 09:57:05 dbdump.go:198: [DEBUG] DiscoverDatabases: bookstack-db → stack=bookstack, dbUser=root, dbName=bookstack +2026-11-11T09:57:05.947696166Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container bentopdf (image=ghcr.io/alam00000/bentopdf:v2.8.10, not a database) +2026-11-11T09:57:05.947701426Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog (image=ghcr.io/seanmorley15/adventurelog-backend:v0.13.1, not a database) +2026-11-11T09:57:05.947705344Z 2026/11/11 09:57:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: adventurelog-postgres (id=39ccc9825d38) +2026-11-11T09:57:05.979127858Z 2026/11/11 09:57:05 dbdump.go:198: [DEBUG] DiscoverDatabases: adventurelog-postgres → stack=adventurelog, dbUser=adventurelog, dbName=adventurelog +2026-11-11T09:57:05.979858929Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog-frontend (image=ghcr.io/seanmorley15/adventurelog-frontend:v0.13.1, not a database) +2026-11-11T09:57:05.979889687Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.352.0, not a database) +2026-11-11T09:57:05.979893784Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.17, not a database) +2026-11-11T09:57:05.979897291Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container cloudflared (image=cloudflare/cloudflared:2026.10.4, not a database) +2026-11-11T09:57:05.979900938Z 2026/11/11 09:57:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.8-stable, not a database) +2026-11-11T09:57:05.979904274Z 2026/11/11 09:57:05 dbdump.go:205: [DEBUG] DiscoverDatabases: found 6 database(s), skipped 20 non-DB container(s) +2026-11-11T09:57:05.979907420Z 2026/11/11 09:57:05 dbdump.go:208: [INFO] [backup] Discovered 6 databases +2026-11-11T09:57:06.072467391Z 2026/11/11 09:57:06 scheduler.go:467: [INFO] [scheduler] Job backup-cache completed (took 674ms) +2026-11-11T09:57:06.156094586Z 2026/11/11 09:57:06 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 659ms) +2026-11-11T09:57:15.391174599Z 2026/11/11 09:57:15 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T09:57:15.391220426Z 2026/11/11 09:57:15 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T09:57:20.390884897Z 2026/11/11 09:57:20 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T09:57:25.389356846Z 2026/11/11 09:57:25 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T09:57:25.389402032Z 2026/11/11 09:57:25 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T09:57:35.391423638Z 2026/11/11 09:57:35 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T09:57:35.391463113Z 2026/11/11 09:57:35 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026-11-11T09:57:35.391467260Z 2026/11/11 09:57:35 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026-11-11T09:57:35.391478602Z 2026/11/11 09:57:35 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T09:57:35.391904236Z 2026/11/11 09:57:35 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026-11-11T09:57:35.391935715Z 2026/11/11 09:57:35 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T09:57:45.389771608Z 2026/11/11 09:57:45 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T09:57:45.389882147Z 2026/11/11 09:57:45 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T09:57:50.390771070Z 2026/11/11 09:57:50 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T09:57:55.390753201Z 2026/11/11 09:57:55 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T09:57:55.390880702Z 2026/11/11 09:57:55 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T09:58:05.389539549Z 2026/11/11 09:58:05 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T09:58:05.389570227Z 2026/11/11 09:58:05 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T09:58:05.389575026Z 2026/11/11 09:58:05 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026-11-11T09:58:05.389579184Z 2026/11/11 09:58:05 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026-11-11T09:58:05.389583131Z 2026/11/11 09:58:05 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026-11-11T09:58:05.390410503Z 2026/11/11 09:58:05 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026-11-11T09:58:05.391331873Z 2026/11/11 09:58:05 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026-11-11T09:58:05.391365216Z 2026/11/11 09:58:05 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:58:05.391479281Z 2026/11/11 09:58:05 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T09:58:05.496565621Z 2026/11/11 09:58:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:58:05.496603543Z 2026/11/11 09:58:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting +2026-11-11T09:58:06.147290381Z 2026/11/11 09:58:06 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 651ms) +2026-11-11T09:58:15.389718636Z 2026/11/11 09:58:15 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T09:58:15.389814016Z 2026/11/11 09:58:15 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T09:58:20.390767652Z 2026/11/11 09:58:20 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T09:58:25.389232498Z 2026/11/11 09:58:25 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T09:58:25.389349288Z 2026/11/11 09:58:25 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T09:58:35.390771513Z 2026/11/11 09:58:35 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026-11-11T09:58:35.390809485Z 2026/11/11 09:58:35 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026-11-11T09:58:35.390818452Z 2026/11/11 09:58:35 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T09:58:35.391003812Z 2026/11/11 09:58:35 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026-11-11T09:58:35.391099092Z 2026/11/11 09:58:35 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T09:58:35.391235469Z 2026/11/11 09:58:35 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T09:58:45.390879506Z 2026/11/11 09:58:45 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T09:58:45.390956802Z 2026/11/11 09:58:45 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T09:58:50.390733225Z 2026/11/11 09:58:50 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T09:58:55.391093460Z 2026/11/11 09:58:55 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T09:58:55.391251869Z 2026/11/11 09:58:55 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T09:59:05.389079767Z 2026/11/11 09:59:05 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T09:59:05.389125574Z 2026/11/11 09:59:05 scheduler.go:450: [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:59:05.389130163Z 2026/11/11 09:59:05 scheduler.go:85: [DEBUG] [scheduler] job stack-scan: execution starting +2026-11-11T09:59:05.389183233Z 2026/11/11 09:59:05 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T09:59:05.389349446Z 2026/11/11 09:59:05 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026-11-11T09:59:05.389357441Z 2026/11/11 09:59:05 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T09:59:05.389361539Z 2026/11/11 09:59:05 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026-11-11T09:59:05.389365156Z 2026/11/11 09:59:05 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026-11-11T09:59:05.389434237Z 2026/11/11 09:59:05 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026-11-11T09:59:05.389763308Z 2026/11/11 09:59:05 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026-11-11T09:59:05.389795940Z 2026/11/11 09:59:05 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:59:05.496793027Z 2026/11/11 09:59:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:59:05.496820689Z 2026/11/11 09:59:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting +2026-11-11T09:59:05.546081752Z 2026/11/11 09:59:05 scheduler.go:467: [INFO] [scheduler] Job stack-scan completed (took 157ms) +2026-11-11T09:59:06.138583409Z 2026/11/11 09:59:06 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 642ms) +2026-11-11T09:59:15.389722199Z 2026/11/11 09:59:15 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T09:59:15.389844350Z 2026/11/11 09:59:15 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T09:59:20.390897764Z 2026/11/11 09:59:20 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T09:59:25.389905394Z 2026/11/11 09:59:25 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T09:59:25.389944187Z 2026/11/11 09:59:25 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T09:59:35.390723549Z 2026/11/11 09:59:35 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026-11-11T09:59:35.390762122Z 2026/11/11 09:59:35 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026-11-11T09:59:35.390767712Z 2026/11/11 09:59:35 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T09:59:35.390772101Z 2026/11/11 09:59:35 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T09:59:35.390781188Z 2026/11/11 09:59:35 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T09:59:35.390889132Z 2026/11/11 09:59:35 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026-11-11T09:59:45.390790355Z 2026/11/11 09:59:45 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T09:59:45.390905241Z 2026/11/11 09:59:45 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T09:59:50.389748662Z 2026/11/11 09:59:50 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T09:59:55.390894642Z 2026/11/11 09:59:55 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T09:59:55.390960737Z 2026/11/11 09:59:55 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:00:05.389436996Z 2026/11/11 10:00:05 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026-11-11T10:00:05.389475098Z 2026/11/11 10:00:05 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026-11-11T10:00:05.389480338Z 2026/11/11 10:00:05 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026-11-11T10:00:05.389484315Z 2026/11/11 10:00:05 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026-11-11T10:00:05.390130095Z 2026/11/11 10:00:05 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:00:05.390159310Z 2026/11/11 10:00:05 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:00:05.390164259Z 2026/11/11 10:00:05 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026-11-11T10:00:05.390167836Z 2026/11/11 10:00:05 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T10:00:05.390171252Z 2026/11/11 10:00:05 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T10:00:05.496014350Z 2026/11/11 10:00:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T10:00:05.496056049Z 2026/11/11 10:00:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting +2026-11-11T10:00:06.214357347Z 2026/11/11 10:00:06 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 718ms) +2026-11-11T10:00:15.390703141Z 2026/11/11 10:00:15 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:00:15.390812899Z 2026/11/11 10:00:15 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:00:20.390761640Z 2026/11/11 10:00:20 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T10:00:25.389255824Z 2026/11/11 10:00:25 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:00:25.389400588Z 2026/11/11 10:00:25 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:00:35.389066542Z 2026/11/11 10:00:35 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:00:35.389110936Z 2026/11/11 10:00:35 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T10:00:35.389329970Z 2026/11/11 10:00:35 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026-11-11T10:00:35.389343415Z 2026/11/11 10:00:35 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026-11-11T10:00:35.389350098Z 2026/11/11 10:00:35 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:00:35.389355869Z 2026/11/11 10:00:35 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026-11-11T10:00:45.390716182Z 2026/11/11 10:00:45 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:00:45.390752881Z 2026/11/11 10:00:45 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:00:50.390752909Z 2026/11/11 10:00:50 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T10:00:55.390768987Z 2026/11/11 10:00:55 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:00:55.390997839Z 2026/11/11 10:00:55 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:01:05.389781849Z 2026/11/11 10:01:05 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026-11-11T10:01:05.389815813Z 2026/11/11 10:01:05 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T10:01:05.389819861Z 2026/11/11 10:01:05 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026-11-11T10:01:05.389823578Z 2026/11/11 10:01:05 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:01:05.389826784Z 2026/11/11 10:01:05 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026-11-11T10:01:05.389829920Z 2026/11/11 10:01:05 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:01:05.389960667Z 2026/11/11 10:01:05 scheduler.go:450: [INFO] [scheduler] Running job: stack-scan +2026-11-11T10:01:05.389971487Z 2026/11/11 10:01:05 scheduler.go:85: [DEBUG] [scheduler] job stack-scan: execution starting +2026-11-11T10:01:05.389975475Z 2026/11/11 10:01:05 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026-11-11T10:01:05.389979071Z 2026/11/11 10:01:05 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026-11-11T10:01:05.390019447Z 2026/11/11 10:01:05 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T10:01:05.496095034Z 2026/11/11 10:01:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T10:01:05.496126183Z 2026/11/11 10:01:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting +2026-11-11T10:01:05.562234764Z 2026/11/11 10:01:05 scheduler.go:467: [INFO] [scheduler] Job stack-scan completed (took 172ms) +2026-11-11T10:01:06.108225686Z 2026/11/11 10:01:06 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 612ms) +2026-11-11T10:01:15.389714517Z 2026/11/11 10:01:15 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:01:15.389788557Z 2026/11/11 10:01:15 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:01:20.390818659Z 2026/11/11 10:01:20 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T10:01:25.388861282Z 2026/11/11 10:01:25 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:01:25.389373799Z 2026/11/11 10:01:25 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:01:35.390747057Z 2026/11/11 10:01:35 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T10:01:35.390788105Z 2026/11/11 10:01:35 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:01:35.390793244Z 2026/11/11 10:01:35 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026-11-11T10:01:35.390802171Z 2026/11/11 10:01:35 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026-11-11T10:01:35.390953747Z 2026/11/11 10:01:35 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:01:35.390996538Z 2026/11/11 10:01:35 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026-11-11T10:01:45.391055024Z 2026/11/11 10:01:45 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:01:45.391763742Z 2026/11/11 10:01:45 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:01:50.390812373Z 2026/11/11 10:01:50 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T10:01:55.390786933Z 2026/11/11 10:01:55 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:01:55.390949149Z 2026/11/11 10:01:55 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:02:05.387578911Z 2026/11/11 10:02:05 scheduler.go:450: [INFO] [scheduler] Running job: hub-report +2026-11-11T10:02:05.387924694Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job hub-report: execution starting +2026-11-11T10:02:05.389380553Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026-11-11T10:02:05.389536748Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026-11-11T10:02:05.391369959Z 2026/11/11 10:02:05 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026-11-11T10:02:05.391513340Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026-11-11T10:02:05.391674855Z 2026/11/11 10:02:05 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T10:02:05.391811272Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T10:02:05.392586205Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026-11-11T10:02:05.392611794Z 2026/11/11 10:02:05 scheduler.go:450: [INFO] [scheduler] Running job: system-health +2026-11-11T10:02:05.392616312Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job system-health: execution starting +2026-11-11T10:02:05.392864891Z 2026/11/11 10:02:05 scheduler.go:450: [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T10:02:05.393071522Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-users: execution starting +2026-11-11T10:02:05.393246031Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:02:05.394939408Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:02:05.399781049Z 2026/11/11 10:02:05 scheduler.go:450: [INFO] [scheduler] Running job: backup-cache +2026-11-11T10:02:05.399804212Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job backup-cache: execution starting +2026-11-11T10:02:05.402003384Z 2026/11/11 10:02:05 scheduler.go:450: [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T10:02:05.402038621Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job offsite-credential-retry: execution starting +2026-11-11T10:02:05.402055192Z 2026/11/11 10:02:05 scheduler.go:467: [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T10:02:05.495851567Z 2026/11/11 10:02:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T10:02:05.495974188Z 2026/11/11 10:02:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting +2026-11-11T10:02:05.504097565Z 2026/11/11 10:02:05 backup.go:1252: [INFO] [backup] Found 21 DB dump files across drives +2026-11-11T10:02:05.507146601Z 2026/11/11 10:02:05 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026-11-11T10:02:05.592428120Z 2026/11/11 10:02:05 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: e8a4549a77a0 romm romm rommapp/romm:5.4.1 +2026-11-11T10:02:05.592503783Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.4.1, not a database) +2026-11-11T10:02:05.592506899Z 2026/11/11 10:02:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=0c0a552f1d66) +2026-11-11T10:02:05.680886095Z 2026/11/11 10:02:05 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026-11-11T10:02:05.680889582Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026-11-11T10:02:05.680892848Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.1.0, not a database) +2026-11-11T10:02:05.680895914Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container plex (image=plexinc/pms-docker:1.43.5.11029-6bf8af3f0, not a database) +2026-11-11T10:02:05.680899320Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:3.4.1, not a database) +2026-11-11T10:02:05.680902637Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026-11-11T10:02:05.680905753Z 2026/11/11 10:02:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=d5dc3d71319d) +2026-11-11T10:02:05.680908568Z 2026/11/11 10:02:05 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026-11-11T10:02:05.683188011Z 2026/11/11 10:02:05 sync.go:299: [INFO] [sync] Catalog sync complete +2026-11-11T10:02:05.742313528Z 2026/11/11 10:02:05 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026-11-11T10:02:05.742663499Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container opengist (image=ghcr.io/thomiceli/opengist:1.15, not a database) +2026-11-11T10:02:05.742729233Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container kimai (image=kimai/kimai2:2.67.0, not a database) +2026-11-11T10:02:05.742735595Z 2026/11/11 10:02:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: kimai-db (id=170178b88458) +2026-11-11T10:02:05.807651023Z 2026/11/11 10:02:05 scheduler.go:467: [INFO] [scheduler] Job docker-socket-users completed (took 414ms) +2026-11-11T10:02:05.807686861Z 2026/11/11 10:02:05 dbdump.go:198: [DEBUG] DiscoverDatabases: kimai-db → stack=kimai, dbUser=root, dbName=kimai +2026-11-11T10:02:05.807752876Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container home-assistant (image=ghcr.io/home-assistant/home-assistant:2026.10.4, not a database) +2026-11-11T10:02:05.807763816Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost (image=docmost/docmost:0.97.0, not a database) +2026-11-11T10:02:05.807856421Z 2026/11/11 10:02:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: docmost-postgres (id=ed402849ac4c) +2026-11-11T10:02:05.851076910Z 2026/11/11 10:02:05 dbdump.go:198: [DEBUG] DiscoverDatabases: docmost-postgres → stack=docmost, dbUser=docmost, dbName=docmost +2026-11-11T10:02:05.851146251Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost-redis (image=redis:7-alpine, not a database) +2026-11-11T10:02:05.851152713Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container calibre-web (image=crocodilestick/calibre-web-automated:v4.0.8, not a database) +2026-11-11T10:02:05.851156781Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container bookstack (image=lscr.io/linuxserver/bookstack:26.09.2, not a database) +2026-11-11T10:02:05.851160528Z 2026/11/11 10:02:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: bookstack-db (id=bd002743cdb7) +2026-11-11T10:02:05.887691086Z 2026/11/11 10:02:05 dbdump.go:198: [DEBUG] DiscoverDatabases: bookstack-db → stack=bookstack, dbUser=root, dbName=bookstack +2026-11-11T10:02:05.887975193Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container bentopdf (image=ghcr.io/alam00000/bentopdf:v2.8.10, not a database) +2026-11-11T10:02:05.888128943Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog (image=ghcr.io/seanmorley15/adventurelog-backend:v0.13.1, not a database) +2026-11-11T10:02:05.888276010Z 2026/11/11 10:02:05 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: adventurelog-postgres (id=39ccc9825d38) +2026-11-11T10:02:05.903919495Z 2026/11/11 10:02:05 scheduler.go:467: [INFO] [scheduler] Job system-health completed (took 511ms) +2026-11-11T10:02:05.921175844Z 2026/11/11 10:02:05 dbdump.go:198: [DEBUG] DiscoverDatabases: adventurelog-postgres → stack=adventurelog, dbUser=adventurelog, dbName=adventurelog +2026-11-11T10:02:05.921210189Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog-frontend (image=ghcr.io/seanmorley15/adventurelog-frontend:v0.13.1, not a database) +2026-11-11T10:02:05.921214067Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.352.0, not a database) +2026-11-11T10:02:05.921217403Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.17, not a database) +2026-11-11T10:02:05.921220609Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container cloudflared (image=cloudflare/cloudflared:2026.10.4, not a database) +2026-11-11T10:02:05.921232732Z 2026/11/11 10:02:05 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.8-stable, not a database) +2026-11-11T10:02:05.921236709Z 2026/11/11 10:02:05 dbdump.go:205: [DEBUG] DiscoverDatabases: found 6 database(s), skipped 20 non-DB container(s) +2026-11-11T10:02:05.921291703Z 2026/11/11 10:02:05 dbdump.go:208: [INFO] [backup] Discovered 6 databases +2026-11-11T10:02:06.186619994Z 2026/11/11 10:02:06 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 690ms) +2026-11-11T10:02:06.368610565Z 2026/11/11 10:02:06 scheduler.go:467: [INFO] [scheduler] Job backup-cache completed (took 970ms) +2026-11-11T10:02:07.513372767Z 2026/11/11 10:02:07 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.352.0 >= floor 0.352.0 — no action +2026-11-11T10:02:07.513398967Z 2026/11/11 10:02:07 scheduler.go:467: [INFO] [scheduler] Job hub-report completed (took 2.126s) +2026-11-11T10:02:15.391097059Z 2026/11/11 10:02:15 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:02:15.391173023Z 2026/11/11 10:02:15 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:02:20.389796882Z 2026/11/11 10:02:20 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T10:02:25.390664969Z 2026/11/11 10:02:25 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:02:25.390703060Z 2026/11/11 10:02:25 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:02:35.390676456Z 2026/11/11 10:02:35 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026-11-11T10:02:35.390722753Z 2026/11/11 10:02:35 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026-11-11T10:02:35.390731730Z 2026/11/11 10:02:35 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:02:35.390751377Z 2026/11/11 10:02:35 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026-11-11T10:02:35.390811912Z 2026/11/11 10:02:35 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:02:35.391461909Z 2026/11/11 10:02:35 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T10:02:45.390729942Z 2026/11/11 10:02:45 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:02:45.390764447Z 2026/11/11 10:02:45 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:02:50.390855407Z 2026/11/11 10:02:50 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T10:02:55.389055897Z 2026/11/11 10:02:55 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:02:55.389331066Z 2026/11/11 10:02:55 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:03:05.389612583Z 2026/11/11 10:03:05 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026-11-11T10:03:05.389648330Z 2026/11/11 10:03:05 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026-11-11T10:03:05.389652959Z 2026/11/11 10:03:05 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T10:03:05.389656737Z 2026/11/11 10:03:05 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026-11-11T10:03:05.389660403Z 2026/11/11 10:03:05 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:03:05.389663830Z 2026/11/11 10:03:05 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T10:03:05.389670893Z 2026/11/11 10:03:05 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026-11-11T10:03:05.389683487Z 2026/11/11 10:03:05 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026-11-11T10:03:05.390470593Z 2026/11/11 10:03:05 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:03:05.390552227Z 2026/11/11 10:03:05 scheduler.go:450: [INFO] [scheduler] Running job: stack-scan +2026-11-11T10:03:05.390556856Z 2026/11/11 10:03:05 scheduler.go:85: [DEBUG] [scheduler] job stack-scan: execution starting +2026-11-11T10:03:05.495928234Z 2026/11/11 10:03:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T10:03:05.495933243Z 2026/11/11 10:03:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting +2026-11-11T10:03:05.568710434Z 2026/11/11 10:03:05 scheduler.go:467: [INFO] [scheduler] Job stack-scan completed (took 178ms) +2026-11-11T10:03:06.174079520Z 2026/11/11 10:03:06 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 678ms) +2026-11-11T10:03:15.390644266Z 2026/11/11 10:03:15 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:03:15.390681356Z 2026/11/11 10:03:15 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:03:20.390760133Z 2026/11/11 10:03:20 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T10:03:25.389607284Z 2026/11/11 10:03:25 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:03:25.389662548Z 2026/11/11 10:03:25 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:03:35.390668091Z 2026/11/11 10:03:35 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:03:35.390744706Z 2026/11/11 10:03:35 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026-11-11T10:03:35.390780864Z 2026/11/11 10:03:35 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026-11-11T10:03:35.390785132Z 2026/11/11 10:03:35 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026-11-11T10:03:35.391076372Z 2026/11/11 10:03:35 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:03:35.391086271Z 2026/11/11 10:03:35 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T10:03:45.389773899Z 2026/11/11 10:03:45 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:03:45.389882043Z 2026/11/11 10:03:45 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:03:50.390737836Z 2026/11/11 10:03:50 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T10:03:55.389956328Z 2026/11/11 10:03:55 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:03:55.390106231Z 2026/11/11 10:03:55 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:04:05.389588870Z 2026/11/11 10:04:05 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026-11-11T10:04:05.389624548Z 2026/11/11 10:04:05 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026-11-11T10:04:05.389777767Z 2026/11/11 10:04:05 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:04:05.389786073Z 2026/11/11 10:04:05 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026-11-11T10:04:05.389829194Z 2026/11/11 10:04:05 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026-11-11T10:04:05.389835827Z 2026/11/11 10:04:05 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026-11-11T10:04:05.389840044Z 2026/11/11 10:04:05 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T10:04:05.389844232Z 2026/11/11 10:04:05 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:04:05.389848080Z 2026/11/11 10:04:05 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T10:04:05.496334663Z 2026/11/11 10:04:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T10:04:05.496394095Z 2026/11/11 10:04:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting +2026-11-11T10:04:06.139729280Z 2026/11/11 10:04:06 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 643ms) +2026-11-11T10:04:15.389890014Z 2026/11/11 10:04:15 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:04:15.389939438Z 2026/11/11 10:04:15 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026-11-11T10:04:20.390815980Z 2026/11/11 10:04:20 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026-11-11T10:04:25.389069540Z 2026/11/11 10:04:25 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026-11-11T10:04:25.389738794Z 2026/11/11 10:04:25 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +RC=0 diff --git a/documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-guest-night.txt b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-guest-night.txt new file mode 100644 index 00000000..b3fbba43 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-guest-night.txt @@ -0,0 +1,196 @@ +# guest-night demo-hp taken 2026-11-11T10:09:24Z +log first line: 2026-11-10T23:15:05.505129813Z +log last line: 2026-11-11T10:09:20.389434141Z + +=== scheduler jobs (UTC): runs, first start, last start, last three "took" === +agent-channel-health runs=654 first=2026-11-10T23:16:05 last=2026-11-11T10:09:05 [completed (took 664ms)] [completed (took 647ms)] [completed (took 669ms)] +backup-cache runs=131 first=2026-11-10T23:17:05 last=2026-11-11T10:07:05 [completed (took 674ms)] [completed (took 970ms)] [completed (took 655ms)] +conversion-copy-release runs=11 first=2026-11-10T23:17:05 last=2026-11-11T09:17:05 [completed (took 77ms)] [completed (took 342ms)] [completed (took 266ms)] +db-dump runs=1 first=2026-11-11T01:30:00 last=2026-11-11T01:30:00 [completed (took 2m33.216s)] +disk-health-check runs=11 first=2026-11-10T23:17:05 last=2026-11-11T09:17:05 [completed (took 962ms)] [completed (took 907ms)] [completed (took 876ms)] +docker-socket-users runs=131 first=2026-11-10T23:17:05 last=2026-11-11T10:07:05 [completed (took 305ms)] [completed (took 414ms)] [completed (took 337ms)] +fill-watch runs=1 first=2026-11-11T02:30:00 last=2026-11-11T02:30:00 [completed (took 0s)] +fill-watch-interval runs=66 first=2026-11-10T23:17:05 last=2026-11-11T10:07:05 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)] +geo-verify runs=2 first=2026-11-11T03:17:05 last=2026-11-11T09:17:05 [completed (took 0s)] [completed (took 0s)] +hold-report runs=654 first=2026-11-10T23:16:05 last=2026-11-11T10:09:05 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)] +hub-report runs=44 first=2026-11-10T23:17:05 last=2026-11-11T10:02:05 [completed (took 2.129s)] [completed (took 2.298s)] [completed (took 2.126s)] +metrics-prune runs=1 first=2026-11-11T03:00:00 last=2026-11-11T03:00:00 [completed (took 419ms)] +offbox-backup runs=1 first=2026-11-11T03:15:00 last=2026-11-11T03:15:00 [completed (took 3m8.568s)] +offsite-abandon-sweep runs=1 first=2026-11-11T04:10:00 last=2026-11-11T04:10:00 [completed (took 0s)] +offsite-credential-retry runs=131 first=2026-11-10T23:17:05 last=2026-11-11T10:07:05 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)] +offsite-integrity runs=1 first=2026-11-11T05:00:00 last=2026-11-11T05:00:00 [completed (took 0s)] +offsite-proof runs=1 first=2026-11-11T04:30:00 last=2026-11-11T04:30:00 [completed (took 5.282s)] +restore-copy-sweep runs=2 first=2026-11-11T03:17:05 last=2026-11-11T09:17:05 [completed (took 0s)] [completed (took 0s)] +selfupdate-check runs=2 first=2026-11-11T03:17:05 last=2026-11-11T09:17:05 [completed (took 139ms)] [completed (took 134ms)] +stack-scan runs=327 first=2026-11-10T23:17:05 last=2026-11-11T10:09:05 [completed (took 136ms)] [completed (took 274ms)] [completed (took 158ms)] +system-health runs=131 first=2026-11-10T23:17:05 last=2026-11-11T10:07:05 [completed (took 396ms)] [completed (took 511ms)] [completed (took 415ms)] +tier2-backup runs=1 first=2026-11-11T02:30:00 last=2026-11-11T02:30:00 [completed (took 38.939s)] + +=== jobs that ran fewer than 30 times: every start and end (UTC) === +2026/11/10 23:17:05 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/10 23:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/10 23:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 106ms) +2026/11/10 23:17:06 [INFO] [scheduler] Job disk-health-check completed (took 833ms) +2026/11/11 00:17:05 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 00:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 00:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 366ms) +2026/11/11 00:17:06 [INFO] [scheduler] Job disk-health-check completed (took 868ms) +2026/11/11 01:17:05 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 01:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 01:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 396ms) +2026/11/11 01:17:06 [INFO] [scheduler] Job disk-health-check completed (took 847ms) +2026/11/11 01:30:00 [INFO] [scheduler] Running job: db-dump +2026/11/11 01:32:33 [INFO] [scheduler] Job db-dump completed (took 2m33.216s) +2026/11/11 02:17:05 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 02:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 84ms) +2026/11/11 02:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 02:17:06 [INFO] [scheduler] Job disk-health-check completed (took 911ms) +2026/11/11 02:30:00 [INFO] [scheduler] Running job: tier2-backup +2026/11/11 02:30:00 [INFO] [scheduler] Running job: fill-watch +2026/11/11 02:30:00 [INFO] [scheduler] Job fill-watch completed (took 0s) +2026/11/11 02:30:38 [INFO] [scheduler] Job tier2-backup completed (took 38.939s) +2026/11/11 03:00:00 [INFO] [scheduler] Running job: metrics-prune +2026/11/11 03:00:00 [INFO] [scheduler] Job metrics-prune completed (took 419ms) +2026/11/11 03:15:00 [INFO] [scheduler] Running job: offbox-backup +2026/11/11 03:17:05 [INFO] [scheduler] Running job: geo-verify +2026/11/11 03:17:05 [INFO] [scheduler] Job geo-verify completed (took 0s) +2026/11/11 03:17:05 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 03:17:05 [INFO] [scheduler] Running job: selfupdate-check +2026/11/11 03:17:05 [INFO] [scheduler] Running job: restore-copy-sweep +2026/11/11 03:17:05 [INFO] [scheduler] Job restore-copy-sweep completed (took 0s) +2026/11/11 03:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 03:17:05 [INFO] [scheduler] Job selfupdate-check completed (took 139ms) +2026/11/11 03:17:06 [INFO] [scheduler] Job conversion-copy-release completed (took 834ms) +2026/11/11 03:17:06 [INFO] [scheduler] Job disk-health-check completed (took 1.069s) +2026/11/11 03:18:08 [INFO] [scheduler] Job offbox-backup completed (took 3m8.568s) +2026/11/11 04:10:00 [INFO] [scheduler] Running job: offsite-abandon-sweep +2026/11/11 04:10:00 [INFO] [scheduler] Job offsite-abandon-sweep completed (took 0s) +2026/11/11 04:17:05 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 04:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 04:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 285ms) +2026/11/11 04:17:06 [INFO] [scheduler] Job disk-health-check completed (took 839ms) +2026/11/11 04:30:00 [INFO] [scheduler] Running job: offsite-proof +2026/11/11 04:30:05 [INFO] [scheduler] Job offsite-proof completed (took 5.282s) +2026/11/11 05:00:00 [INFO] [scheduler] Running job: offsite-integrity +2026/11/11 05:00:00 [INFO] [scheduler] Job offsite-integrity completed (took 0s) +2026/11/11 05:17:05 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 05:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 62ms) +2026/11/11 05:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 05:17:06 [INFO] [scheduler] Job disk-health-check completed (took 816ms) +2026/11/11 06:17:05 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 06:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 06:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 323ms) +2026/11/11 06:17:06 [INFO] [scheduler] Job disk-health-check completed (took 853ms) +2026/11/11 07:17:05 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 07:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 77ms) +2026/11/11 07:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 07:17:06 [INFO] [scheduler] Job disk-health-check completed (took 962ms) +2026/11/11 08:17:05 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 08:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 08:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 342ms) +2026/11/11 08:17:06 [INFO] [scheduler] Job disk-health-check completed (took 907ms) +2026/11/11 09:17:05 [INFO] [scheduler] Running job: selfupdate-check +2026/11/11 09:17:05 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 09:17:05 [INFO] [scheduler] Running job: geo-verify +2026/11/11 09:17:05 [INFO] [scheduler] Job geo-verify completed (took 0s) +2026/11/11 09:17:05 [INFO] [scheduler] Running job: restore-copy-sweep +2026/11/11 09:17:05 [INFO] [scheduler] Job restore-copy-sweep completed (took 0s) +2026/11/11 09:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 09:17:05 [INFO] [scheduler] Job selfupdate-check completed (took 134ms) +2026/11/11 09:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 266ms) +2026/11/11 09:17:06 [INFO] [scheduler] Job disk-health-check completed (took 876ms) + +=== night lines 19:00-08:00 UTC, INFO/WARN/ERROR, backup words (first 2 per message shape) === +2026/11/10 23:17:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok +2026/11/10 23:17:05 backup.go:1252: [INFO] [backup] Found 21 DB dump files across drives +2026/11/10 23:17:05 healthcheck.go:176: [INFO] [monitor] Health check: status=ok +2026/11/10 23:17:06 dbdump.go:208: [INFO] [backup] Discovered 6 databases +2026/11/10 23:17:06 [INFO] [backup] Discovered app data: 12 apps +2026/11/10 23:17:06 disk_health.go:277: [INFO] [web] disk-health check complete: 2 disk(s) evaluated, 0 alert(s) +2026/11/10 23:17:06 backup.go:1323: [INFO] [backup] Backup status cache refreshed +2026/11/10 23:17:06 healthcheck.go:176: [INFO] [monitor] Health check: status=ok +2026/11/10 23:18:05 main.go:2430: [INFO] [deadapp] check alive: 960 scans since boot, 12 deployed app(s) evaluated, 0 currently down +2026/11/10 23:22:05 backup.go:1252: [INFO] [backup] Found 21 DB dump files across drives +2026/11/10 23:22:05 dbdump.go:208: [INFO] [backup] Discovered 6 databases +2026/11/10 23:22:05 [INFO] [backup] Discovered app data: 12 apps +2026/11/10 23:22:05 backup.go:1323: [INFO] [backup] Backup status cache refreshed +2026/11/10 23:27:05 fillwatch.go:284: [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok +2026/11/10 23:28:05 main.go:2430: [INFO] [deadapp] check alive: 980 scans since boot, 12 deployed app(s) evaluated, 0 currently down +2026/11/11 00:17:06 disk_health.go:277: [INFO] [web] disk-health check complete: 2 disk(s) evaluated, 0 alert(s) +2026/11/11 01:30:00 backup.go:601: [INFO] [backup] Starting database dump run +2026/11/11 01:30:00 backup.go:640: [INFO] [backup] Discovered 6 database(s): romm-db(mariadb), paperless-postgres(postgres), kimai-db(mariadb), docmost-postgres(postgres), bookstack-db(mariadb), adventurelog-postgres(postgres) +2026/11/11 01:30:00 dbdump.go:429: [INFO] [backup] DB dump: romm-db → romm-mariadb.sql (136.3 KB, 459ms, 40 tables) +2026/11/11 01:30:01 dbdump.go:429: [INFO] [backup] DB dump: paperless-postgres → paperless-ngx-postgres.sql (733.5 KB, 575ms, 75 tables) +2026/11/11 01:30:01 dbdump.go:429: [INFO] [backup] DB dump: kimai-db → kimai-mariadb.sql (51.3 KB, 346ms, 35 tables) +2026/11/11 01:30:02 dbdump.go:429: [INFO] [backup] DB dump: docmost-postgres → docmost-postgres.sql (164.7 KB, 563ms, 52 tables) +2026/11/11 01:30:02 dbdump.go:429: [INFO] [backup] DB dump: bookstack-db → bookstack-mariadb.sql (57.8 KB, 377ms, 42 tables) +2026/11/11 01:30:03 dbdump.go:429: [INFO] [backup] DB dump: adventurelog-postgres → adventurelog-postgres.sql (12.5 MB, 922ms, 51 tables) +2026/11/11 01:30:03 backup.go:1044: [INFO] [backup] Stopping adventurelog for safe volume dump +2026/11/11 01:30:05 busymark.go:81: [INFO] [busymark] busy: a backup, a restore or an off-site run is running — an operator reboot of the box is refused until it ends +2026/11/11 01:30:08 backup.go:941: [INFO] [backup] Volume dump: adventurelog/adventurelog_adventurelog_media → 90.1 MB +2026/11/11 01:30:14 backup.go:941: [INFO] [backup] Volume dump: adventurelog/adventurelog_adventurelog_postgres_data → 273.4 MB +2026/11/11 01:30:14 backup.go:1060: [INFO] [backup] Restarting adventurelog after volume dump +2026/11/11 01:30:20 backup.go:1044: [INFO] [backup] Stopping bookstack for safe volume dump +2026/11/11 01:30:26 backup.go:941: [INFO] [backup] Volume dump: bookstack/bookstack_bookstack_config → 6.4 MB +2026/11/11 01:30:28 backup.go:941: [INFO] [backup] Volume dump: bookstack/bookstack_bookstack_db_data → 153.6 MB +2026/11/11 01:30:28 backup.go:1060: [INFO] [backup] Restarting bookstack after volume dump +2026/11/11 01:30:35 backup.go:1044: [INFO] [backup] Stopping calibre-web for safe volume dump +2026/11/11 01:30:41 backup.go:941: [INFO] [backup] Volume dump: calibre-web/calibre-web_calibre_web_config → 921.5 KB +2026/11/11 01:30:41 backup.go:1060: [INFO] [backup] Restarting calibre-web after volume dump +2026/11/11 01:30:42 backup.go:1044: [INFO] [backup] Stopping docmost for safe volume dump +2026/11/11 01:30:46 backup.go:941: [INFO] [backup] Volume dump: docmost/docmost_docmost_storage → 2.5 KB +2026/11/11 01:30:49 backup.go:941: [INFO] [backup] Volume dump: docmost/docmost_docmost_postgres_data → 66.8 MB +2026/11/11 01:30:50 backup.go:1060: [INFO] [backup] Restarting docmost after volume dump +2026/11/11 01:31:02 backup.go:1044: [INFO] [backup] Stopping home-assistant for safe volume dump +2026/11/11 01:31:13 backup.go:941: [INFO] [backup] Volume dump: home-assistant/home-assistant_homeassistant_config → 17.8 MB +2026/11/11 01:31:13 backup.go:1060: [INFO] [backup] Restarting home-assistant after volume dump +2026/11/11 01:31:13 backup.go:1044: [INFO] [backup] Stopping kimai for safe volume dump +2026/11/11 01:31:16 backup.go:941: [INFO] [backup] Volume dump: kimai/kimai_kimai_var → 18.3 MB +2026/11/11 01:31:20 backup.go:941: [INFO] [backup] Volume dump: kimai/kimai_kimai_db_data → 154.5 MB +2026/11/11 01:31:20 backup.go:1060: [INFO] [backup] Restarting kimai after volume dump +2026/11/11 01:31:26 backup.go:1044: [INFO] [backup] Stopping opengist for safe volume dump +2026/11/11 01:31:28 backup.go:941: [INFO] [backup] Volume dump: opengist/opengist_opengist_data → 181.5 KB +2026/11/11 01:31:28 backup.go:1060: [INFO] [backup] Restarting opengist after volume dump +2026/11/11 01:31:28 backup.go:1044: [INFO] [backup] Stopping paperless-ngx for safe volume dump +2026/11/11 01:31:38 backup.go:941: [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_data → 21.0 MB +2026/11/11 01:31:40 backup.go:941: [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_postgres_data → 70.2 MB +2026/11/11 01:31:41 backup.go:1060: [INFO] [backup] Restarting paperless-ngx after volume dump +2026/11/11 01:31:52 backup.go:1044: [INFO] [backup] Stopping plex for safe volume dump +2026/11/11 01:32:03 backup.go:941: [INFO] [backup] Volume dump: plex/plex_plex_config → 9.9 MB +2026/11/11 01:32:03 backup.go:941: [INFO] [backup] Volume dump: plex/plex_plex_transcode → 3.0 KB +2026/11/11 01:32:03 backup.go:1060: [INFO] [backup] Restarting plex after volume dump +2026/11/11 01:32:04 backup.go:1044: [INFO] [backup] Stopping privatebin for safe volume dump +2026/11/11 01:32:04 backup.go:941: [INFO] [backup] Volume dump: privatebin/privatebin_privatebin_data → 2.0 MB +2026/11/11 01:32:04 backup.go:1060: [INFO] [backup] Restarting privatebin after volume dump +2026/11/11 01:32:07 recovery_unit.go:297: [INFO] [backup] Recovery unit captured for adventurelog → /mnt/sys_drive/felhom-data/backups/primary/adventurelog (images=3, secrets-referenced=2, data_keys=1, portable-carried=2/2, with +2026/11/11 01:32:07 backup.go:1044: [INFO] [backup] Stopping romm for safe volume dump +2026/11/11 01:32:07 recovery_unit.go:297: [INFO] [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1 +2026/11/11 01:32:17 backup.go:941: [INFO] [backup] Volume dump: romm/romm_romm_redis_data → 61.6 MB +2026/11/11 01:32:17 backup.go:941: [INFO] [backup] Volume dump: romm/romm_romm_config → 2.5 KB +2026/11/11 01:32:20 backup.go:1060: [INFO] [backup] Restarting romm after volume dump +2026/11/11 01:32:32 backup.go:743: [INFO] [backup] App-data backup completed: 6 databases (13.6 MB total), 11 volume dump(s) (2m32.713s) +2026/11/11 01:32:33 notifier.go:357: [INFO] Event pushed: db_dump_completed (info) [hu-only] — Adatbázis mentés elkészült +2026/11/11 01:32:35 busymark.go:81: [INFO] [busymark] idle again (was: a backup, a restore or an off-site run is running) — an operator reboot of the box is no longer refused for it +2026/11/11 02:30:08 tier2.go:525: [INFO] [backup] Tier 2 copied adventurelog → /mnt/felhom-drives/hdd_1/backups/secondary/adventurelog (404.1 MB, 0 leg(s), 7s) +2026/11/11 02:30:10 tier2.go:525: [INFO] [backup] Tier 2 copied bentopdf → /mnt/felhom-drives/hdd_1/backups/secondary/bentopdf (12.8 KB, 0 leg(s), 0s) +2026/11/11 02:30:14 tier2.go:525: [INFO] [backup] Tier 2 copied bookstack → /mnt/felhom-drives/hdd_1/backups/secondary/bookstack (160.3 MB, 0 leg(s), 3s) +2026/11/11 02:30:16 tier2.go:525: [INFO] [backup] Tier 2 copied calibre-web → /mnt/sys_drive/felhom-data/backups/secondary/calibre-web (5.6 MB, 1 leg(s), 1s) [SSD: state-only] +2026/11/11 02:30:20 tier2.go:525: [INFO] [backup] Tier 2 copied docmost → /mnt/felhom-drives/hdd_1/backups/secondary/docmost (120.1 MB, 0 leg(s), 2s) +2026/11/11 02:30:21 tier2.go:525: [INFO] [backup] Tier 2 copied home-assistant → /mnt/felhom-drives/hdd_1/backups/secondary/home-assistant (17.8 MB, 0 leg(s), 0s) +2026/11/11 02:30:26 tier2.go:525: [INFO] [backup] Tier 2 copied kimai → /mnt/felhom-drives/hdd_1/backups/secondary/kimai (173.0 MB, 0 leg(s), 4s) +2026/11/11 02:30:28 tier2.go:525: [INFO] [backup] Tier 2 copied opengist → /mnt/felhom-drives/hdd_1/backups/secondary/opengist (189.7 KB, 0 leg(s), 0s) +2026/11/11 02:30:31 tier2.go:525: [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sys_drive/felhom-data/backups/secondary/paperless-ngx (155.1 MB, 1 leg(s), 2s) [SSD: state-only] +2026/11/11 02:30:33 tier2.go:525: [INFO] [backup] Tier 2 copied plex → /mnt/sys_drive/felhom-data/backups/secondary/plex (9.9 MB, 0 leg(s), 0s) [SSD: state-only] +2026/11/11 02:30:35 tier2.go:525: [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/hdd_1/backups/secondary/privatebin (2.0 MB, 0 leg(s), 0s) +2026/11/11 02:30:38 tier2.go:525: [INFO] [backup] Tier 2 copied romm → /mnt/sys_drive/felhom-data/backups/secondary/romm (217.2 MB, 0 leg(s), 2s) [SSD: state-only] +2026/11/11 02:30:38 tier2.go:587: [INFO] [backup] Tier 2 run complete: 12 app(s) processed (incl. volume-only — F6) +2026/11/11 03:00:00 main.go:1759: [INFO] Pruned 31667 old metric rows +2026/11/11 03:15:00 offbox.go:1094: [INFO] [offbox] backup run started (15 app(s) toggled) +2026/11/11 03:15:00 backup.go:601: [INFO] [backup] Starting database dump run +2026/11/11 03:15:00 backup.go:640: [INFO] [backup] Discovered 6 database(s): romm-db(mariadb), paperless-postgres(postgres), kimai-db(mariadb), docmost-postgres(postgres), bookstack-db(mariadb), adventurelog-postgres(postgres) +2026/11/11 03:15:00 dbdump.go:429: [INFO] [backup] DB dump: romm-db → romm-mariadb.sql (136.3 KB, 360ms, 40 tables) +2026/11/11 03:15:00 dbdump.go:429: [INFO] [backup] DB dump: paperless-postgres → paperless-ngx-postgres.sql (736.3 KB, 356ms, 75 tables) +2026/11/11 03:15:01 dbdump.go:429: [INFO] [backup] DB dump: kimai-db → kimai-mariadb.sql (51.3 KB, 266ms, 35 tables) +awk: write failure (Broken pipe) +awk: close failed on file "/dev/stdout" (Broken pipe) +RC=0 diff --git a/documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-host-clock.txt b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-host-clock.txt new file mode 100644 index 00000000..7a6cae76 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-host-clock.txt @@ -0,0 +1,682 @@ +# host-clock demo-hp taken 2026-11-11T10:03:57Z from DooPlex + +=== in-flight === +checked + +=== clock === +utc 2026-11-11T10:03:57.824658172Z +local 2026-11-11T11:03:57 CET +0100 +/usr/share/zoneinfo/Europe/Budapest +Timezone=Europe/Budapest +LocalRTC=no +CanNTP=yes +NTP=yes +NTPSynchronized=yes +TimeUSec=Wed 2026-11-11 11:03:57 CET +RTCTimeUSec=Wed 2026-11-11 11:03:58 CET + Local time: Wed 2026-11-11 11:03:57 CET + Universal time: Wed 2026-11-11 10:03:57 UTC + RTC time: Wed 2026-11-11 10:03:58 + Time zone: Europe/Budapest (CET, +0100) +System clock synchronized: yes + NTP service: active + RTC in local TZ: no + +=== who keeps the time === +chrony: active + enabled: enabled +chronyd: active + enabled: alias +systemd-timesyncd: inactive + enabled: not-found +ntp: inactive + enabled: not-found +ntpsec: inactive + enabled: not-found +openntpd: inactive + enabled: not-found +Reference ID : 25DDD118 (37.221.209.24) +Stratum : 3 +Ref time (UTC) : Wed Nov 11 09:59:51 2026 +System time : 0.000200578 seconds fast of NTP time +Last offset : +0.000003393 seconds +RMS offset : 0.000097092 seconds +Frequency : 12.934 ppm slow +Residual freq : +0.000 ppm +Skew : 0.017 ppm +Root delay : 0.016422609 seconds +Root dispersion : 0.002579133 seconds +Update interval : 1036.9 seconds +Leap status : Normal +MS Name/IP address Stratum Poll Reach LastRx Last sample +=============================================================================== +^- 62.112.195.56 2 10 377 301 -2821us[-2817us] +/- 33ms +^* 37.221.209.24 2 10 377 246 -3262us[-3259us] +/- 14ms +^+ 81.0.124.253 3 10 377 76 -2267us[-2267us] +/- 25ms +^+ 195.111.92.55 2 10 377 741 -2832us[-2828us] +/- 52ms +Failed to query server: The name org.freedesktop.timesync1 was not provided by any .service files +pool 2.debian.pool.ntp.org iburst +sourcedir /run/chrony-dhcp +sourcedir /etc/chrony/sources.d +keyfile /etc/chrony/chrony.keys +driftfile /var/lib/chrony/chrony.drift +ntsdumpdir /var/lib/chrony +logdir /var/log/chrony +maxupdateskew 100.0 +rtcsync +makestep 1 3 +leapseclist /usr/share/zoneinfo/leap-seconds.list +confdir /etc/chrony/conf.d +/etc/chrony/conf.d: +README + +/etc/chrony/sources.d: +README + +=== start with no time === +ls: cannot access '/var/lib/systemd/timesync/clock': No such file or directory +/var/lib/chrony/: +total 4 +-rw-r--r-- 1 _chrony _chrony 42 2026-11-11 10:59:52.023776089 +0100 chrony.drift +2026-11-11 11:03:58.342660+01:00 +1794391439 +1794391438 + +=== boots === +2026-11-10 15:42:49 + -5 182af91122a941ec97ac7ee247a4d470 Sun 2026-10-04 21:42:08 CEST Sun 2026-10-04 21:43:33 CEST + -4 bb68aee4240244718d8a7f39d999a3a8 Wed 2026-11-04 21:06:10 CET Tue 2026-11-10 02:53:28 CET + -3 b6e0edea2b5647dfa36f5bb3825adc3a Tue 2026-11-10 02:53:57 CET Tue 2026-11-10 13:56:35 CET + -2 d1167d9431bf459abf2a569e8997a569 Tue 2026-11-10 13:57:04 CET Tue 2026-11-10 15:32:31 CET + -1 e2cc9e3e60f14bde8949f01d7ba35ccf Tue 2026-11-10 15:32:59 CET Tue 2026-11-10 15:42:26 CET + 0 bed106f79e804eb1ace314a26da7f049 Tue 2026-11-10 15:42:54 CET Wed 2026-11-11 11:03:57 CET + +=== timers === +NEXT LEFT LAST PASSED UNIT ACTIVATES +Wed 2026-11-11 11:04:28 CET 29s Wed 2026-11-11 11:03:28 CET 30s ago felhom-mgmt-watchdog.timer felhom-mgmt-watchdog.service +Wed 2026-11-11 11:59:19 CET 55min Wed 2026-11-11 10:59:19 CET 4min 39s ago felhom-crash-guard-check.timer felhom-crash-guard-check.service +Wed 2026-11-11 15:29:36 CET 4h 25min Tue 2026-11-10 23:19:19 CET 11h ago apt-daily.timer apt-daily.service +Wed 2026-11-11 15:57:49 CET 4h 53min Tue 2026-11-10 15:57:49 CET 19h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service +Thu 2026-11-12 00:00:00 CET 12h Wed 2026-11-11 00:00:19 CET 11h ago dpkg-db-backup.timer dpkg-db-backup.service +Thu 2026-11-12 00:11:48 CET 13h Wed 2026-11-11 00:16:19 CET 10h ago logrotate.timer logrotate.service +Thu 2026-11-12 01:46:13 CET 14h Wed 2026-11-11 02:22:58 CET 8h ago pve-daily-update.timer pve-daily-update.service +Thu 2026-11-12 06:13:04 CET 19h Wed 2026-11-11 06:44:19 CET 4h 19min ago apt-daily-upgrade.timer apt-daily-upgrade.service +Thu 2026-11-12 07:37:24 CET 20h Wed 2026-11-11 02:10:19 CET 8h ago man-db.timer man-db.service +Sun 2026-11-15 03:10:25 CET 3 days Sun 2026-11-08 03:10:49 CET - xfs_scrub_all.timer xfs_scrub_all.service +Sun 2026-11-15 03:10:43 CET 3 days Sun 2026-11-08 03:10:49 CET - e2scrub_all.timer e2scrub_all.service +Mon 2026-11-16 00:02:01 CET 4 days Mon 2026-11-09 01:25:39 CET - fstrim.timer fstrim.service + +12 timers listed. + +=== timer units (all) === +--- apt-daily-upgrade.timer [enabled/active] +OnCalendar=*-*-* 6:00 +RandomizedDelaySec=60m +Persistent=true +Unit=apt-daily-upgrade.service NextElapseUSecRealtime=Thu 2026-11-12 06:13:04 CET LastTriggerUSec=Wed 2026-11-11 06:44:19 CET +--- apt-daily.timer [enabled/active] +OnCalendar=*-*-* 6,18:00 +RandomizedDelaySec=12h +Persistent=true +Unit=apt-daily.service NextElapseUSecRealtime=Wed 2026-11-11 15:29:36 CET LastTriggerUSec=Tue 2026-11-10 23:19:19 CET +--- apt-listchanges.timer [disabled/inactive] +OnCalendar=hourly +Unit=apt-listchanges.service NextElapseUSecRealtime= LastTriggerUSec= +--- chrony-dnssrv@.timer [disabled/] +OnActiveSec=0 +OnUnitInactiveSec=1h + +--- dpkg-db-backup.timer [enabled/active] +OnCalendar=daily +Persistent=true +Unit=dpkg-db-backup.service NextElapseUSecRealtime=Thu 2026-11-12 00:00:00 CET LastTriggerUSec=Wed 2026-11-11 00:00:19 CET +--- e2scrub_all.timer [enabled/active] +OnCalendar=Sun *-*-* 03:10:00 +RandomizedDelaySec=60 +Persistent=true +Unit=e2scrub_all.service NextElapseUSecRealtime=Sun 2026-11-15 03:10:43 CET LastTriggerUSec=Sun 2026-11-08 03:10:49 CET +--- felhom-crash-guard-check.timer [enabled/active] +OnBootSec=15min +OnUnitActiveSec=1h +Unit=felhom-crash-guard-check.service NextElapseUSecRealtime= LastTriggerUSec=Wed 2026-11-11 10:59:19 CET +--- felhom-mgmt-watchdog.timer [enabled/active] +OnBootSec=30s +OnUnitActiveSec=60s +AccuracySec=5s +Persistent=true +Unit=felhom-mgmt-watchdog.service +Unit=felhom-mgmt-watchdog.service NextElapseUSecRealtime= LastTriggerUSec=Wed 2026-11-11 11:03:28 CET +--- fstrim.timer [enabled/active] +OnCalendar=weekly +AccuracySec=1h +Persistent=true +RandomizedDelaySec=100min +Unit=fstrim.service NextElapseUSecRealtime=Mon 2026-11-16 00:02:01 CET LastTriggerUSec=Mon 2026-11-09 01:25:39 CET +--- logrotate.timer [enabled/active] +OnCalendar=daily +RandomizedDelaySec=1h +Persistent=true +Unit=logrotate.service NextElapseUSecRealtime=Thu 2026-11-12 00:11:48 CET LastTriggerUSec=Wed 2026-11-11 00:16:19 CET +--- man-db.timer [enabled/active] +OnCalendar=daily +RandomizedDelaySec=12h +Persistent=true +Unit=man-db.service NextElapseUSecRealtime=Thu 2026-11-12 07:37:24 CET LastTriggerUSec=Wed 2026-11-11 02:10:19 CET +--- proxmox-firewall.timer [disabled/inactive] +OnBootSec=1s +OnUnitInactiveSec=5s +Unit=proxmox-firewall.service +Unit=proxmox-firewall.service NextElapseUSecRealtime= LastTriggerUSec= +--- pve-daily-update.timer [enabled/active] +OnCalendar=*-*-* 1:00 +RandomizedDelaySec=5h +Persistent=true +Unit=pve-daily-update.service NextElapseUSecRealtime=Thu 2026-11-12 01:46:13 CET LastTriggerUSec=Wed 2026-11-11 02:22:58 CET +--- systemd-tmpfiles-clean.timer [static/active] +OnBootSec=15min +OnUnitActiveSec=1d +Unit=systemd-tmpfiles-clean.service NextElapseUSecRealtime= LastTriggerUSec=Tue 2026-11-10 15:57:49 CET +--- xfs_scrub_all.timer [enabled/active] +OnCalendar=Sun *-*-* 03:10:00 +RandomizedDelaySec=60 +Persistent=true +Unit=xfs_scrub_all.service NextElapseUSecRealtime=Sun 2026-11-15 03:10:25 CET LastTriggerUSec=Sun 2026-11-08 03:10:49 CET +--- zfs-scrub-monthly@.timer [disabled/] +OnCalendar=monthly +Persistent=true +RandomizedDelaySec=1h +Unit=zfs-scrub@%i.service + +--- zfs-scrub-weekly@.timer [disabled/] +OnCalendar=weekly +Persistent=true +RandomizedDelaySec=1h +Unit=zfs-scrub@%i.service + +--- zfs-trim-monthly@.timer [disabled/] +OnCalendar=monthly +Persistent=true +RandomizedDelaySec=1h +Unit=zfs-trim@%i.service + +--- zfs-trim-weekly@.timer [disabled/] +OnCalendar=weekly +Persistent=true +RandomizedDelaySec=1h +Unit=zfs-trim@%i.service + + +=== service limits of felhom + backup timers === +--- apt-daily-upgrade.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 06:44:19 CET +ExecMainExitTimestamp=Wed 2026-11-11 06:44:19 CET +ExecMainStatus=0 +ExecStart={ path=/usr/lib/apt/apt.systemd.daily ; argv[]=/usr/lib/apt/apt.systemd.daily install ; ignore_errors=no ; start_time=[Wed 2026-11-11 06:44:19 CET] ; stop_time=[Wed 2026-11-11 06:44:19 CET] ; pid=3061802 ; code=exited ; status=0 } +--- apt-daily.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Tue 2026-11-10 23:19:19 CET +ExecMainExitTimestamp=Tue 2026-11-10 23:19:19 CET +ExecMainStatus=0 +ExecStart={ path=/usr/lib/apt/apt.systemd.daily ; argv[]=/usr/lib/apt/apt.systemd.daily update ; ignore_errors=no ; start_time=[Tue 2026-11-10 23:19:19 CET] ; stop_time=[Tue 2026-11-10 23:19:19 CET] ; pid=1543951 ; code=exited ; status=0 } +--- apt-listchanges.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp= +ExecMainExitTimestamp= +ExecMainStatus=0 +ExecStart={ path=python3 ; argv[]=python3 -m apt_listchanges.populate_database --profile apt ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } +--- dpkg-db-backup.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 00:00:19 CET +ExecMainExitTimestamp=Wed 2026-11-11 00:00:19 CET +ExecMainStatus=0 +ExecStart={ path=/usr/libexec/dpkg/dpkg-db-backup ; argv[]=/usr/libexec/dpkg/dpkg-db-backup ; ignore_errors=no ; start_time=[Wed 2026-11-11 00:00:19 CET] ; stop_time=[Wed 2026-11-11 00:00:19 CET] ; pid=1679989 ; code=exited ; status=0 } +--- e2scrub_all.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp= +ExecMainExitTimestamp= +ExecMainStatus=0 +ExecStart={ path=/sbin/e2scrub_all ; argv[]=/sbin/e2scrub_all ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } +--- felhom-crash-guard-check.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 10:59:19 CET +ExecMainExitTimestamp=Wed 2026-11-11 10:59:19 CET +ExecMainStatus=0 +ExecStart={ path=/usr/local/sbin/felhom-crash-guard ; argv[]=/usr/local/sbin/felhom-crash-guard check ; ignore_errors=no ; start_time=[Wed 2026-11-11 10:59:19 CET] ; stop_time=[Wed 2026-11-11 10:59:19 CET] ; pid=3911370 ; code=exited ; status=0 } +--- felhom-mgmt-watchdog.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 11:03:28 CET +ExecMainExitTimestamp=Wed 2026-11-11 11:03:28 CET +ExecMainStatus=0 +ExecStart={ path=/usr/local/sbin/felhom-mgmt-watchdog ; argv[]=/usr/local/sbin/felhom-mgmt-watchdog ; ignore_errors=no ; start_time=[Wed 2026-11-11 11:03:28 CET] ; stop_time=[Wed 2026-11-11 11:03:28 CET] ; pid=3925600 ; code=exited ; status=0 } +--- fstrim.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp= +ExecMainExitTimestamp= +ExecMainStatus=0 +ExecStart={ path=/sbin/fstrim ; argv[]=/sbin/fstrim --listed-in /etc/fstab:/proc/self/mountinfo --verbose --quiet-unsupported ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } +--- logrotate.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 00:16:19 CET +ExecMainExitTimestamp=Wed 2026-11-11 00:16:21 CET +ExecMainStatus=0 +ExecStart={ path=/usr/sbin/logrotate ; argv[]=/usr/sbin/logrotate /etc/logrotate.conf ; ignore_errors=no ; start_time=[Wed 2026-11-11 00:16:19 CET] ; stop_time=[Wed 2026-11-11 00:16:21 CET] ; pid=1733122 ; code=exited ; status=0 } +--- man-db.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 02:10:19 CET +ExecMainExitTimestamp=Wed 2026-11-11 02:10:19 CET +ExecMainStatus=0 +ExecStart={ path=/usr/bin/install ; argv[]=/usr/bin/install -d -o man -g man -m 0755 /var/cache/man ; ignore_errors=no ; start_time=[Wed 2026-11-11 02:10:19 CET] ; stop_time=[Wed 2026-11-11 02:10:19 CET] ; pid=2113150 ; code=exited ; status=0 } +ExecStart={ path=/usr/bin/mandb ; argv[]=/usr/bin/mandb --quiet ; ignore_errors=no ; start_time=[Wed 2026-11-11 02:10:19 CET] ; stop_time=[Wed 2026-11-11 02:10:19 CET] ; pid=2113157 ; code=exited ; status=0 } +--- pve-daily-update.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 02:22:58 CET +ExecMainExitTimestamp=Wed 2026-11-11 02:23:04 CET +ExecMainStatus=0 +ExecStart={ path=/usr/bin/pveupdate ; argv[]=/usr/bin/pveupdate ; ignore_errors=no ; start_time=[Wed 2026-11-11 02:22:58 CET] ; stop_time=[Wed 2026-11-11 02:23:04 CET] ; pid=2155738 ; code=exited ; status=0 } + +=== felhom units that are not timers === +felhom-agent-rollback.service loaded inactive dead +felhom-agent.service loaded active running +felhom-crash-guard-check.service loaded inactive dead +felhom-crash-guard.service loaded active exited +felhom-mgmt-watchdog.service loaded inactive dead +felhom-oob-nft.service loaded active exited +felhom-shared-parent.service loaded active exited +felhom-sshd.service loaded active running +felhom-crash-guard-check.timer loaded active waiting +felhom-mgmt-watchdog.timer loaded active waiting +--- felhom-agent-rollback.service +Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=infinity NRestarts=0 ActiveEnterTimestamp= +--- felhom-agent.service +Restart=on-failure RestartUSec=5s TimeoutStartUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:58:17 CET +--- felhom-bootstrap.service +Restart=on-failure RestartUSec=30s TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=infinity NRestarts=0 ActiveEnterTimestamp= +--- felhom-crash-guard-check.service +Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp= +--- felhom-crash-guard.service +Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:42:59 CET +--- felhom-mgmt-watchdog.service +Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp= +--- felhom-oob-nft.service +Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:42:59 CET +--- felhom-shared-parent.service +Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:42:59 CET +--- felhom-sshd.service +Restart=on-failure RestartUSec=100ms TimeoutStartUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:43:00 CET + +=== cron === +SHELL=/bin/sh +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +17 * * * * root cd / && run-parts --report /etc/cron.hourly +25 6 * * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; } +47 6 * * 7 root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.weekly; } +52 6 1 * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.monthly; } +--- /etc/cron.d/e2scrub_all +30 3 * * 0 root test -e /run/systemd/system || SERVICE_MODE=1 /usr/libexec/e2fsprogs/e2scrub_all_cron +10 3 * * * root test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r +--- /etc/cron.d/vzdump +PATH="/usr/sbin:/usr/bin:/sbin:/bin" +--- /etc/cron.d/zfsutils-linux +PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin +24 0 1-7 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/trim ]; then /usr/lib/zfs-linux/trim; fi +24 0 8-14 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs-linux/scrub; fi +--- cron.hourly: +--- cron.daily: apt-compat dpkg logrotate man-db +--- cron.weekly: man-db +--- cron.monthly: +--- user crontabs: + +=== at === +no at + +=== pve jobs === +--- vzdump.cron +PATH="/usr/sbin:/usr/bin:/sbin:/bin" +--- vzdump.conf + +=== pve tasks (start end type id status), last 14 days === +awk: line 2: function strtonum never defined +awk: line 2: function strtonum never defined + +=== runs of timer services, last 10 days (start/finish lines) === +--- apt-daily-upgrade.service +2026-11-09T06:46:01+01:00 demo-hp systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities... +2026-11-09T06:46:01+01:00 demo-hp systemd[1]: apt-daily-upgrade.service: Deactivated successfully. +2026-11-09T06:46:01+01:00 demo-hp systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities. +2026-11-10T06:55:09+01:00 demo-hp systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities... +2026-11-10T06:55:10+01:00 demo-hp systemd[1]: apt-daily-upgrade.service: Deactivated successfully. +2026-11-10T06:55:10+01:00 demo-hp systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities. +2026-11-11T06:44:19+01:00 demo-hp systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities... +2026-11-11T06:44:19+01:00 demo-hp systemd[1]: apt-daily-upgrade.service: Deactivated successfully. +2026-11-11T06:44:19+01:00 demo-hp systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities. +--- apt-daily.service +2026-11-09T07:55:02+01:00 demo-hp systemd[1]: Starting apt-daily.service - Daily apt download activities... +2026-11-09T07:55:03+01:00 demo-hp systemd[1]: apt-daily.service: Deactivated successfully. +2026-11-09T07:55:03+01:00 demo-hp systemd[1]: Finished apt-daily.service - Daily apt download activities. +2026-11-10T04:06:22+01:00 demo-hp systemd[1]: Starting apt-daily.service - Daily apt download activities... +2026-11-10T04:06:23+01:00 demo-hp systemd[1]: apt-daily.service: Deactivated successfully. +2026-11-10T04:06:23+01:00 demo-hp systemd[1]: Finished apt-daily.service - Daily apt download activities. +2026-11-10T23:19:19+01:00 demo-hp systemd[1]: Starting apt-daily.service - Daily apt download activities... +2026-11-10T23:19:19+01:00 demo-hp systemd[1]: apt-daily.service: Deactivated successfully. +2026-11-10T23:19:19+01:00 demo-hp systemd[1]: Finished apt-daily.service - Daily apt download activities. +--- felhom-crash-guard-check.service +2026-11-11T08:59:19+01:00 demo-hp systemd[1]: Starting felhom-crash-guard-check.service - Felhom crash guard re-arm check... +2026-11-11T08:59:19+01:00 demo-hp systemd[1]: felhom-crash-guard-check.service: Deactivated successfully. +2026-11-11T08:59:19+01:00 demo-hp systemd[1]: Finished felhom-crash-guard-check.service - Felhom crash guard re-arm check. +2026-11-11T09:59:19+01:00 demo-hp systemd[1]: Starting felhom-crash-guard-check.service - Felhom crash guard re-arm check... +2026-11-11T09:59:19+01:00 demo-hp systemd[1]: felhom-crash-guard-check.service: Deactivated successfully. +2026-11-11T09:59:19+01:00 demo-hp systemd[1]: Finished felhom-crash-guard-check.service - Felhom crash guard re-arm check. +2026-11-11T10:59:19+01:00 demo-hp systemd[1]: Starting felhom-crash-guard-check.service - Felhom crash guard re-arm check... +2026-11-11T10:59:19+01:00 demo-hp systemd[1]: felhom-crash-guard-check.service: Deactivated successfully. +2026-11-11T10:59:19+01:00 demo-hp systemd[1]: Finished felhom-crash-guard-check.service - Felhom crash guard re-arm check. +--- felhom-mgmt-watchdog.service +2026-11-11T11:01:20+01:00 demo-hp systemd[1]: Starting felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)... +2026-11-11T11:01:20+01:00 demo-hp systemd[1]: felhom-mgmt-watchdog.service: Deactivated successfully. +2026-11-11T11:01:20+01:00 demo-hp systemd[1]: Finished felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent). +2026-11-11T11:02:24+01:00 demo-hp systemd[1]: Starting felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)... +2026-11-11T11:02:24+01:00 demo-hp systemd[1]: felhom-mgmt-watchdog.service: Deactivated successfully. +2026-11-11T11:02:24+01:00 demo-hp systemd[1]: Finished felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent). +2026-11-11T11:03:28+01:00 demo-hp systemd[1]: Starting felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)... +2026-11-11T11:03:28+01:00 demo-hp systemd[1]: felhom-mgmt-watchdog.service: Deactivated successfully. +2026-11-11T11:03:28+01:00 demo-hp systemd[1]: Finished felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent). +--- pve-daily-update.service +2026-11-09T03:29:48+01:00 demo-hp systemd[1]: pve-daily-update.service: Consumed 5.005s CPU time, 235.1M memory peak. +2026-11-10T03:05:09+01:00 demo-hp systemd[1]: Starting pve-daily-update.service - Daily PVE download activities... +2026-11-10T03:05:15+01:00 demo-hp systemd[1]: pve-daily-update.service: Deactivated successfully. +2026-11-10T03:05:15+01:00 demo-hp systemd[1]: Finished pve-daily-update.service - Daily PVE download activities. +2026-11-10T03:05:15+01:00 demo-hp systemd[1]: pve-daily-update.service: Consumed 5.281s CPU time, 295.5M memory peak. +2026-11-11T02:22:58+01:00 demo-hp systemd[1]: Starting pve-daily-update.service - Daily PVE download activities... +2026-11-11T02:23:04+01:00 demo-hp systemd[1]: pve-daily-update.service: Deactivated successfully. +2026-11-11T02:23:04+01:00 demo-hp systemd[1]: Finished pve-daily-update.service - Daily PVE download activities. +2026-11-11T02:23:04+01:00 demo-hp systemd[1]: pve-daily-update.service: Consumed 5.171s CPU time, 293.5M memory peak. + +=== agent log: schedule words, last 3 days (tail 260) === +2026-11-10T03:24:11+01:00 demo-hp sudo[112883]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T03:39:11+01:00 demo-hp sudo[165130]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T03:54:12+01:00 demo-hp sudo[215104]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T04:09:11+01:00 demo-hp sudo[265287]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T04:24:11+01:00 demo-hp sudo[326129]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T04:39:11+01:00 demo-hp sudo[376165]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T04:54:11+01:00 demo-hp sudo[426145]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T05:09:11+01:00 demo-hp sudo[476293]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T05:24:11+01:00 demo-hp sudo[526258]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T05:39:11+01:00 demo-hp sudo[576428]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T05:54:11+01:00 demo-hp sudo[626334]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T06:09:11+01:00 demo-hp sudo[676490]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T06:24:11+01:00 demo-hp sudo[726421]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T06:39:12+01:00 demo-hp sudo[776549]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T06:54:11+01:00 demo-hp sudo[826483]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T07:09:11+01:00 demo-hp sudo[876732]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T07:24:11+01:00 demo-hp sudo[926734]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T07:39:11+01:00 demo-hp sudo[976930]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T07:54:11+01:00 demo-hp sudo[1027027]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T08:09:11+01:00 demo-hp sudo[1077199]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T08:24:11+01:00 demo-hp sudo[1127154]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T08:39:11+01:00 demo-hp sudo[1180619]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T08:54:12+01:00 demo-hp sudo[1231276]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T08:55:21+01:00 demo-hp sudo[1236551]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T08:56:33+01:00 demo-hp felhom-agent[1264]: time=2026-11-10T08:56:33.425+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-11-10T09:09:11+01:00 demo-hp sudo[1281634]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T09:24:10+01:00 demo-hp felhom-agent[1264]: time=2026-11-10T09:24:10.329+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-05T20:50:35Z) is already proven; local: no set +2026-11-10T09:24:11+01:00 demo-hp sudo[1331688]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T09:39:11+01:00 demo-hp sudo[1381939]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T09:54:11+01:00 demo-hp sudo[1431884]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T10:09:11+01:00 demo-hp sudo[1481961]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T10:24:11+01:00 demo-hp sudo[1532001]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T10:39:11+01:00 demo-hp sudo[1582028]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T10:54:11+01:00 demo-hp sudo[1632919]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T11:09:11+01:00 demo-hp sudo[1683016]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T11:24:11+01:00 demo-hp sudo[1733023]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T11:39:11+01:00 demo-hp sudo[1783077]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T11:54:12+01:00 demo-hp sudo[1833125]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T12:09:11+01:00 demo-hp sudo[1883187]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T12:24:12+01:00 demo-hp sudo[1933288]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T12:39:11+01:00 demo-hp sudo[1983322]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T12:54:11+01:00 demo-hp sudo[2033570]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T13:09:12+01:00 demo-hp sudo[2095848]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T13:24:11+01:00 demo-hp sudo[2159490]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T13:39:11+01:00 demo-hp sudo[2216453]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T13:54:11+01:00 demo-hp sudo[2275938]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T13:54:26+01:00 demo-hp systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)... +2026-11-10T13:54:26+01:00 demo-hp felhom-agent[1264]: time=2026-11-10T13:54:26.844+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled" +2026-11-10T13:54:26+01:00 demo-hp systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T13:57:14+01:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T13:57:15+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:57:15.046+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=1 prune="keep-daily=1" wait_timeout=30m0s prune_pbs_allowed=false primary=true +2026-11-10T13:57:15+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:57:15.046+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false +2026-11-10T13:57:15+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:57:15.047+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)" +2026-11-10T13:57:15+01:00 demo-hp sudo[1275]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T125715Z-kernel-kernel-boot.json +2026-11-10T13:57:16+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:57:16.045+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s +2026-11-10T13:57:16+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:57:16.045+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s +2026-11-10T13:57:16+01:00 demo-hp sudo[1446]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T13:57:16+01:00 demo-hp sudo[1507]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T13:57:16+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:57:16.911+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gue +2026-11-10T13:57:16+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:57:16.911+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s +2026-11-10T13:57:16+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:57:16.912+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or rest +2026-11-10T13:57:16+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:57:16.912+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never besi +2026-11-10T13:58:28+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:58:28.317+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-11-10T14:12:19+01:00 demo-hp sudo[62389]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T14:27:16+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T14:27:16.046+01:00 level=INFO msg="backup: restore-test first evaluation after start (R-874)" after=30m0s +2026-11-10T14:27:16+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T14:27:16.383+01:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=local volid=local:backup/felhom-golden-0.236.0.tar.zst size_bytes=654115664 rea +2026-11-10T14:27:16+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T14:27:16.735+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-05T20:50:35Z) is already proven; local: no set +2026-11-10T14:27:19+01:00 demo-hp sudo[112482]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T14:42:19+01:00 demo-hp sudo[162646]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T14:57:19+01:00 demo-hp sudo[212644]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:12:19+01:00 demo-hp sudo[262858]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:27:19+01:00 demo-hp sudo[312873]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:30:24+01:00 demo-hp systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)... +2026-11-10T15:30:24+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T15:30:24.913+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled" +2026-11-10T15:30:24+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T15:30:24.913+01:00 level=INFO msg="pbs: verify loop shutting down" reason="context canceled" +2026-11-10T15:30:25+01:00 demo-hp systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T15:33:09+01:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T15:33:09+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:33:09.997+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=1 prune="keep-daily=1" wait_timeout=30m0s prune_pbs_allowed=false primary=true +2026-11-10T15:33:09+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:33:09.997+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false +2026-11-10T15:33:09+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:33:09.997+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)" +2026-11-10T15:33:10+01:00 demo-hp sudo[1318]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T143309Z-kernel-kernel-boot.json +2026-11-10T15:33:10+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:33:10.983+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s +2026-11-10T15:33:10+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:33:10.983+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s +2026-11-10T15:33:11+01:00 demo-hp sudo[1496]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:33:11+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:33:11.889+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gue +2026-11-10T15:33:11+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:33:11.889+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s +2026-11-10T15:33:11+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:33:11.889+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or rest +2026-11-10T15:33:11+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:33:11.890+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never besi +2026-11-10T15:33:12+01:00 demo-hp sudo[1660]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:34:21+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:34:21.282+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-11-10T15:40:09+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:40:09.290+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled" +2026-11-10T15:40:09+01:00 demo-hp felhom-agent[1302]: time=2026-11-10T15:40:09.290+01:00 level=INFO msg="pbs: verify loop shutting down" reason="context canceled" +2026-11-10T15:40:09+01:00 demo-hp systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)... +2026-11-10T15:40:09+01:00 demo-hp systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T15:43:04+01:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T15:43:05+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:05.011+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=1 prune="keep-daily=1" wait_timeout=30m0s prune_pbs_allowed=false primary=true +2026-11-10T15:43:05+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:05.011+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false +2026-11-10T15:43:05+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:05.011+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)" +2026-11-10T15:43:05+01:00 demo-hp sudo[1323]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T144304Z-kernel-kernel-boot.json +2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.000+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s +2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.000+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s +2026-11-10T15:43:06+01:00 demo-hp sudo[1499]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:43:06+01:00 demo-hp sudo[1563]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.858+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gue +2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.858+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s +2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.859+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or rest +2026-11-10T15:43:06+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:43:06.859+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never besi +2026-11-10T15:44:16+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:44:16.241+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-11-10T15:58:09+01:00 demo-hp sudo[65997]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:58:15+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:58:15.231+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.173.0 report="{\"agent_update\": {\"sha256\": \"752b2727fb6043417 +2026-11-10T15:58:17+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:58:17.289+01:00 level=INFO msg="pbs: verify loop shutting down" reason="context canceled" +2026-11-10T15:58:17+01:00 demo-hp felhom-agent[1306]: time=2026-11-10T15:58:17.289+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled" +2026-11-10T15:58:17+01:00 demo-hp systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)... +2026-11-10T15:58:17+01:00 demo-hp systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T15:58:17+01:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T15:58:17+01:00 demo-hp sudo[66874]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T145817Z-kernel-kernel-boot.json +2026-11-10T15:58:17+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:17.356+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=1 prune="keep-daily=1" wait_timeout=30m0s prune_pbs_allowed=false primary=true +2026-11-10T15:58:17+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:17.356+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false +2026-11-10T15:58:17+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:17.356+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)" +2026-11-10T15:58:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:18.424+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s +2026-11-10T15:58:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:18.424+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s +2026-11-10T15:58:18+01:00 demo-hp sudo[67051]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:58:19+01:00 demo-hp sudo[67167]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.388+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gu +2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.388+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s +2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.389+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or res +2026-11-10T15:58:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:58:19.389+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never bes +2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.427+01:00 level=INFO msg="osupdate: running the bundle policy pass after the self-update commit (R-1285)" agent_version=0.173.0 +2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.589+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE POLICY DONE install_mode=appliance removed=0" +2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.589+01:00 level=INFO msg="osupdate: bundle policy pass done" bundle_policy="{\"install_mode\": \"appliance\", \"removed\": []}" +2026-11-10T15:59:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:19.592+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.173.0 removed=49 +2026-11-10T15:59:26+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T15:59:26.622+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-11-10T16:13:24+01:00 demo-hp sudo[120555]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T16:28:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T16:28:18.425+01:00 level=INFO msg="backup: restore-test first evaluation after start (R-874)" after=30m0s +2026-11-10T16:28:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T16:28:18.768+01:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=local volid=local:backup/felhom-golden-0.236.0.tar.zst size_bytes=654115664 re +2026-11-10T16:28:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T16:28:19.097+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-05T20:50:35Z) is already proven; local: no se +2026-11-10T16:28:24+01:00 demo-hp sudo[172820]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T16:29:17+01:00 demo-hp sudo[176191]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T16:43:24+01:00 demo-hp sudo[224892]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T16:58:24+01:00 demo-hp sudo[274799]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T17:13:24+01:00 demo-hp sudo[325353]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T17:28:24+01:00 demo-hp sudo[375316]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T17:43:24+01:00 demo-hp sudo[425306]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T17:58:24+01:00 demo-hp sudo[475151]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T18:13:24+01:00 demo-hp sudo[525197]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T18:28:24+01:00 demo-hp sudo[575041]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T18:43:24+01:00 demo-hp sudo[625011]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T18:58:24+01:00 demo-hp sudo[674867]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T19:13:24+01:00 demo-hp sudo[724869]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T19:28:24+01:00 demo-hp sudo[774788]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T19:43:24+01:00 demo-hp sudo[824794]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T19:58:24+01:00 demo-hp sudo[874679]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T20:13:24+01:00 demo-hp sudo[924671]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T20:28:24+01:00 demo-hp sudo[974638]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T20:43:24+01:00 demo-hp sudo[1024545]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T20:58:24+01:00 demo-hp sudo[1074504]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T21:13:24+01:00 demo-hp sudo[1124372]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T21:28:24+01:00 demo-hp sudo[1174370]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T21:43:24+01:00 demo-hp sudo[1224315]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T21:58:24+01:00 demo-hp sudo[1274176]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T21:59:26+01:00 demo-hp sudo[1277687]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T22:00:40+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T22:00:40.811+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-11-10T22:13:24+01:00 demo-hp sudo[1324125]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T22:28:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-10T22:28:19.944+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-05T20:50:35Z) is already proven; local: no se +2026-11-10T22:28:24+01:00 demo-hp sudo[1374071]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T22:43:24+01:00 demo-hp sudo[1424003]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T22:58:24+01:00 demo-hp sudo[1474011]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T23:13:24+01:00 demo-hp sudo[1523951]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T23:28:24+01:00 demo-hp sudo[1574019]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T23:43:24+01:00 demo-hp sudo[1623864]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T23:58:24+01:00 demo-hp sudo[1673826]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T00:13:24+01:00 demo-hp sudo[1723921]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T00:28:24+01:00 demo-hp sudo[1773909]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T00:43:24+01:00 demo-hp sudo[1823847]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T00:58:24+01:00 demo-hp sudo[1873734]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T01:13:24+01:00 demo-hp sudo[1923814]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T01:28:24+01:00 demo-hp sudo[1973839]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T01:43:24+01:00 demo-hp sudo[2023859]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T01:58:24+01:00 demo-hp sudo[2073734]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T02:13:24+01:00 demo-hp sudo[2123673]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T02:28:24+01:00 demo-hp sudo[2173583]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T02:43:24+01:00 demo-hp sudo[2235197]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T02:58:24+01:00 demo-hp sudo[2285057]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T03:13:24+01:00 demo-hp sudo[2335092]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T03:28:24+01:00 demo-hp sudo[2385056]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T03:43:24+01:00 demo-hp sudo[2437365]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T03:58:24+01:00 demo-hp sudo[2487208]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T03:59:26+01:00 demo-hp sudo[2490710]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T04:00:40+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:00:40.820+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-11-11T04:13:24+01:00 demo-hp sudo[2537305]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T04:28:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:28:19.579+01:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=local archive=local:backup/vzdump-lxc-9201-2026_11_10-02_35_34 +2026-11-11T04:28:20+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:28:20.131+01:00 level=INFO msg="restore-test: space preflight passed" storage=nvme-scratch required_bytes=41914830848 avail_bytes=633007927296 +2026-11-11T04:28:20+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:28:20.232+01:00 level=INFO msg="restore-test: full-fidelity restore params derived from the archive config" scratch=990000 params=4 +2026-11-11T04:28:24+01:00 demo-hp sudo[2598186]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T04:31:39+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:31:39.144+01:00 level=INFO msg="restore-test: scratch guest torn down" vmid=990000 +2026-11-11T04:31:39+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:31:39.145+01:00 level=INFO msg="backup: scheduled restore-test passed" archive=local:backup/vzdump-lxc-9201-2026_11_10-02_35_34.tar.zst duration_s=199.562184354 +2026-11-11T04:33:02+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:33:02.092+01:00 level=INFO msg="backup: space preflight passed" vmid=9201 target=local last_archive_bytes=9095951125 need_bytes=12443680730 avail_bytes=22617755648 +2026-11-11T04:33:03+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:33:03.153+01:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=local job=backup-9201-1794367981574952245 +2026-11-11T04:38:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:38:19.390+01:00 level=INFO msg="janitor: stale-lock sweep deferred — a heavy operation is in flight" busy=backup:local +2026-11-11T04:43:24+01:00 demo-hp sudo[2655608]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T04:44:09+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:44:09.948+01:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_11_11-04_33_02.tar.zst size_bytes=9096384826 uncovered_volumes=2 +2026-11-11T04:44:09+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:44:09.949+01:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1794367981574952245 archive=local:backup/vzdump-lxc-9201-2026_11_11-04_33_02.tar.zs +2026-11-11T04:45:39+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:45:39.956+01:00 level=INFO msg="osupdate: START" run=20261111T034539Z layer=guest vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261111T034539Z +2026-11-11T04:45:58+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:45:58.939+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261111T034539Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0" +2026-11-11T04:45:58+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:45:58.939+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-11T04:45:58+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:45:58.939+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-11-11T04:45:58+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:45:58.939+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-11-11T04:45:58+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:45:58.940+01:00 level=INFO msg="osupdate: DONE" run=20261111T034539Z layer=guest vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 +2026-11-11T04:45:58+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:45:58.982+01:00 level=INFO msg="osupdate: START" run=20261111T034539Z layer=host vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261111T034539Z +2026-11-11T04:46:15+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:15.451+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261111T034539Z layer=host lane=fast mode=apply select=pending-fast packages=0" +2026-11-11T04:46:15+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:15.451+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-11T04:46:15+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:15.451+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-11-11T04:46:15+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:15.451+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-11-11T04:46:16+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:16.538+01:00 level=INFO msg="osupdate: DONE" run=20261111T034539Z layer=host vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=16 not_covered=16 +2026-11-11T04:46:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:18.657+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE already on" +2026-11-11T04:46:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:18.657+01:00 level=INFO msg="osupdate: live-restore" vmid=9201 result="{\"result\": \"already on\"}" +2026-11-11T04:46:18+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:18.657+01:00 level=INFO msg="osupdate: START" run=20261111T034539Z layer=docker vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261111T034539Z +2026-11-11T04:46:48+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:48.868+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261111T034539Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority= +2026-11-11T04:46:48+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:48.868+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-11T04:46:48+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:48.868+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-11-11T04:46:48+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:48.868+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-11-11T04:46:48+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:48.868+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: OOM-CHECK result=pass oom_killed=True oom_event=True exit=137 image=gitea.dooplex.hu/admin/felhom-controller:0.352.0 †+2026-11-11T04:46:48+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:48.869+01:00 level=INFO msg="osupdate: DONE" run=20261111T034539Z layer=docker vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 +2026-11-11T04:46:48+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:48.883+01:00 level=INFO msg="osupdate: pve step holds the /etc/pve write gate — the agent's own writes wait until it ends" run=20261111T034539Z layer=pve vmid=9201 trigger=night +2026-11-11T04:46:48+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:46:48.883+01:00 level=INFO msg="osupdate: START" run=20261111T034539Z layer=pve vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261111T034539Z +2026-11-11T04:47:06+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:47:06.472+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261111T034539Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0" +2026-11-11T04:47:06+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:47:06.472+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-11T04:47:06+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:47:06.472+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-11-11T04:47:06+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:47:06.472+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-11-11T04:47:07+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:47:07.887+01:00 level=INFO msg="osupdate: DONE" run=20261111T034539Z layer=pve vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=2 not_covered=0 re +2026-11-11T04:47:07+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:47:07.892+01:00 level=INFO msg="osupdate: pve step released the /etc/pve write gate" run=20261111T034539Z layer=pve vmid=9201 trigger=night +2026-11-11T04:47:07+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T04:47:07.892+01:00 level=INFO msg="osupdate: kernel step skipped — the hub names no kernel step for this box" run=20261111T034539Z layer=kernel vmid=9201 trigger=night ring=0 +2026-11-11T04:58:24+01:00 demo-hp sudo[2708953]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T05:13:24+01:00 demo-hp sudo[2758966]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T05:28:24+01:00 demo-hp sudo[2808864]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T05:43:24+01:00 demo-hp sudo[2858958]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T05:58:24+01:00 demo-hp sudo[2908875]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T06:13:24+01:00 demo-hp sudo[2958776]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T06:28:24+01:00 demo-hp sudo[3008761]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T06:43:24+01:00 demo-hp sudo[3058666]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T06:58:24+01:00 demo-hp sudo[3108643]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T07:13:24+01:00 demo-hp sudo[3158591]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T07:28:24+01:00 demo-hp sudo[3208701]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T07:43:24+01:00 demo-hp sudo[3258653]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T07:58:24+01:00 demo-hp sudo[3308680]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T08:13:24+01:00 demo-hp sudo[3358593]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T08:28:24+01:00 demo-hp sudo[3408639]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T08:43:24+01:00 demo-hp sudo[3458520]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T08:58:24+01:00 demo-hp sudo[3508327]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T09:13:24+01:00 demo-hp sudo[3558359]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T09:28:24+01:00 demo-hp sudo[3608281]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T09:43:24+01:00 demo-hp sudo[3658187]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T09:58:24+01:00 demo-hp sudo[3708187]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T09:59:26+01:00 demo-hp sudo[3711739]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T10:00:40+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T10:00:40.832+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-11-11T10:13:24+01:00 demo-hp sudo[3758186]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T10:28:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T10:28:19.885+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-05T20:50:35Z) is already proven; local: no se +2026-11-11T10:28:24+01:00 demo-hp sudo[3808072]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T10:43:24+01:00 demo-hp sudo[3858055]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T10:58:24+01:00 demo-hp sudo[3908221]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage + +=== agent log: line count per hour, last 26 h === + 12339 2026-11-10T09 13253 2026-11-10T10 13266 2026-11-10T11 13262 2026-11-10T12 12658 2026-11-10T13 13271 2026-11-10T14 12173 2026-11-10T15 13301 2026-11-10T16 13278 2026-11-10T17 13268 2026-11-10T18 13244 2026-11-10T19 13256 2026-11-10T20 13270 2026-11-10T21 13266 2026-11-10T22 13246 2026-11-10T23 13249 2026-11-11T00 13261 2026-11-11T01 13267 2026-11-11T02 13507 2026-11-11T03 13345 2026-11-11T04 13267 2026-11-11T05 13264 2026-11-11T06 13261 2026-11-11T07 13261 2026-11-11T08 13252 2026-11-11T09 13262 2026-11-11T10 996 2026-11-11T11 +=== agent version === +felhom-agent 0.173.0 +RC=0 diff --git a/documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-host-night.txt b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-host-night.txt new file mode 100644 index 00000000..34c0d14c --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-host-night.txt @@ -0,0 +1,156 @@ +# host-night demo-hp taken 2026-11-11T10:09:23Z +=== pve tasks, last 14 days (local time) === +2026-11-03 03:52:35 -> 03:52:35 0.0 min push_file 9202 OK +2026-11-03 03:52:43 -> 03:52:43 0.0 min push_file 9202 OK +2026-11-03 03:52:51 -> 03:52:51 0.0 min push_file 9202 OK +2026-11-03 03:53:00 -> 03:53:00 0.0 min push_file 9202 OK +2026-11-03 03:53:08 -> 03:53:08 0.0 min push_file 9202 OK +2026-11-03 03:53:32 -> 03:53:32 0.0 min push_file 9202 OK +2026-11-03 03:53:35 -> 03:53:35 0.0 min push_file 9202 OK +2026-11-03 03:53:38 -> 03:53:38 0.0 min push_file 9202 OK +2026-11-03 03:53:41 -> 03:53:41 0.0 min push_file 9202 OK +2026-11-03 03:54:23 -> 03:54:23 0.0 min push_file 9202 OK +2026-11-03 03:54:27 -> 03:54:27 0.0 min push_file 9202 OK +2026-11-03 03:54:31 -> 03:54:31 0.0 min push_file 9202 OK +2026-11-03 03:54:32 -> 03:54:32 0.0 min push_file 9202 OK +2026-11-03 03:55:11 -> 03:55:11 0.0 min push_file 9202 OK +2026-11-03 03:55:15 -> 03:55:15 0.0 min push_file 9202 OK +2026-11-03 03:55:16 -> 03:55:16 0.0 min push_file 9202 OK +2026-11-03 03:56:51 -> 03:56:51 0.0 min push_file 9202 OK +2026-11-03 03:56:55 -> 03:56:55 0.0 min push_file 9202 OK +2026-11-03 03:56:56 -> 03:56:56 0.0 min push_file 9202 OK +2026-11-03 03:57:04 -> 03:57:04 0.0 min push_file 9202 OK +2026-11-03 03:57:13 -> 03:57:13 0.0 min push_file 9202 OK +2026-11-03 03:57:21 -> 03:57:21 0.0 min push_file 9202 OK +2026-11-03 03:57:30 -> 03:57:30 0.0 min push_file 9202 OK +2026-11-03 03:57:38 -> 03:57:38 0.0 min push_file 9202 OK +2026-11-03 03:57:47 -> 03:57:47 0.0 min push_file 9202 OK +2026-11-03 03:57:55 -> 03:57:55 0.0 min push_file 9202 OK +2026-11-03 03:58:04 -> 03:58:04 0.0 min push_file 9202 OK +2026-11-03 03:58:12 -> 03:58:12 0.0 min push_file 9202 OK +2026-11-03 03:58:21 -> 03:58:21 0.0 min push_file 9202 OK +2026-11-03 03:58:29 -> 03:58:29 0.0 min push_file 9202 OK +2026-11-03 03:58:38 -> 03:58:38 0.0 min push_file 9202 OK +2026-11-03 03:58:46 -> 03:58:46 0.0 min push_file 9202 OK +2026-11-03 03:58:54 -> 03:58:54 0.0 min push_file 9202 OK +2026-11-03 03:59:02 -> 03:59:02 0.0 min push_file 9202 OK +2026-11-03 03:59:11 -> 03:59:11 0.0 min push_file 9202 OK +2026-11-03 03:59:19 -> 03:59:19 0.0 min push_file 9202 OK +2026-11-03 03:59:27 -> 03:59:27 0.0 min push_file 9202 OK +2026-11-03 03:59:35 -> 03:59:35 0.0 min push_file 9202 OK +2026-11-03 03:59:43 -> 03:59:43 0.0 min push_file 9202 OK +2026-11-03 03:59:51 -> 03:59:51 0.0 min push_file 9202 OK +2026-11-03 03:59:59 -> 04:00:00 0.0 min push_file 9202 OK +2026-11-03 04:00:08 -> 04:00:08 0.0 min push_file 9202 OK +2026-11-03 04:00:16 -> 04:00:16 0.0 min push_file 9202 OK +2026-11-03 04:00:24 -> 04:00:24 0.0 min push_file 9202 OK +2026-11-03 04:00:33 -> 04:00:34 0.0 min push_file 9202 OK +2026-11-03 04:00:42 -> 04:00:42 0.0 min push_file 9202 OK +2026-11-03 04:00:50 -> 04:00:50 0.0 min push_file 9202 OK +2026-11-03 04:00:58 -> 04:00:58 0.0 min push_file 9202 OK +2026-11-03 04:01:06 -> 04:01:06 0.0 min push_file 9202 OK +2026-11-03 04:01:14 -> 04:01:14 0.0 min push_file 9202 OK +2026-11-03 04:01:23 -> 04:01:23 0.0 min push_file 9202 OK +2026-11-03 04:01:31 -> 04:01:31 0.0 min push_file 9202 OK +2026-11-03 04:01:39 -> 04:01:39 0.0 min push_file 9202 OK +2026-11-03 04:01:47 -> 04:01:47 0.0 min push_file 9202 OK +2026-11-03 04:01:56 -> 04:01:56 0.0 min push_file 9202 OK +2026-11-03 04:02:04 -> 04:02:04 0.0 min push_file 9202 OK +2026-11-03 04:02:12 -> 04:02:12 0.0 min push_file 9202 OK +2026-11-03 04:02:20 -> 04:02:20 0.0 min push_file 9202 OK +2026-11-03 04:02:28 -> 04:02:28 0.0 min push_file 9202 OK +2026-11-03 04:02:37 -> 04:02:37 0.0 min push_file 9202 OK +2026-11-03 04:02:45 -> 04:02:45 0.0 min push_file 9202 OK +2026-11-03 04:02:53 -> 04:02:53 0.0 min push_file 9202 OK +2026-11-03 04:03:02 -> 04:03:02 0.0 min push_file 9202 OK +2026-11-03 04:03:25 -> 04:03:25 0.0 min push_file 9202 OK +2026-11-03 04:03:29 -> 04:03:29 0.0 min push_file 9202 OK +2026-11-03 04:03:32 -> 04:03:32 0.0 min push_file 9202 OK +2026-11-03 04:03:35 -> 04:03:35 0.0 min push_file 9202 OK +2026-11-03 04:04:17 -> 04:04:18 0.0 min push_file 9202 OK +2026-11-03 04:04:22 -> 04:04:22 0.0 min push_file 9202 OK +2026-11-03 04:04:26 -> 04:04:26 0.0 min push_file 9202 OK +2026-11-03 04:04:27 -> 04:04:27 0.0 min push_file 9202 OK +2026-11-03 04:04:53 -> 04:04:53 0.0 min push_file 9202 OK +2026-11-03 04:05:09 -> 04:05:09 0.0 min push_file 9202 OK +2026-11-03 04:07:29 -> 04:07:29 0.0 min push_file 9202 OK +2026-11-03 04:07:33 -> 04:07:37 0.1 min vzshutdown 9401 OK +2026-11-03 05:50:30 -> 05:50:35 0.1 min aptupdate OK +2026-11-03 08:22:29 -> 08:22:29 0.0 min qmcreate 9383 OK +2026-11-03 08:22:32 -> 08:22:33 0.0 min qmstart 9383 OK +2026-11-03 10:14:45 -> 10:14:46 0.0 min qmstop 9383 OK +2026-11-03 10:14:47 -> 10:14:47 0.0 min qmdestroy 9383 OK +2026-11-04 02:09:47 -> 02:09:51 0.1 min aptupdate OK +2026-11-05 01:18:47 -> 01:18:52 0.1 min aptupdate OK +2026-11-05 21:50:35 -> 21:57:39 7.1 min vzdump 9201 OK +2026-11-06 03:38:47 -> 03:38:51 0.1 min aptupdate OK +2026-11-07 05:01:47 -> 05:01:51 0.1 min aptupdate OK +2026-11-07 22:39:02 -> 22:39:02 0.0 min imgdel 9100@local OK +2026-11-07 23:42:03 -> 23:55:58 13.9 min vzrestore 990000 OK +2026-11-07 23:56:01 -> 23:56:05 0.1 min vzstart 990000 OK +2026-11-07 23:56:07 -> 23:56:10 0.1 min vzdestroy 990000 OK +2026-11-08 01:22:04 -> 01:22:08 0.1 min aptupdate OK +2026-11-08 02:30:38 -> 02:43:34 12.9 min vzdump 9201 OK +2026-11-09 03:29:44 -> 03:29:48 0.1 min aptupdate OK +2026-11-09 05:56:15 -> 05:59:17 3.0 min vzrestore 990000 OK +2026-11-09 05:59:21 -> 05:59:25 0.1 min vzstart 990000 OK +2026-11-09 05:59:28 -> 05:59:30 0.0 min vzdestroy 990000 OK +2026-11-09 09:51:30 -> 09:51:30 0.0 min imgdel local OK +2026-11-10 02:35:34 -> 02:46:27 10.9 min vzdump 9201 OK +2026-11-10 02:51:00 -> 02:52:45 1.8 min vzshutdown 9201 OK +2026-11-10 02:51:00 -> 02:52:45 1.8 min vzshutdown 9202 OK +2026-11-10 02:51:00 -> 02:53:09 2.1 min qmshutdown 341 OK +2026-11-10 02:51:00 -> 02:53:09 2.1 min stopall OK +2026-11-10 02:54:10 -> 02:54:13 0.1 min qmstart 341 OK +2026-11-10 02:54:10 -> 02:54:18 0.1 min startall OK +2026-11-10 02:54:13 -> 02:54:18 0.1 min vzstart 9201 OK +2026-11-10 02:54:13 -> 02:54:18 0.1 min vzstart 9202 OK +2026-11-10 03:05:11 -> 03:05:15 0.1 min aptupdate OK +2026-11-10 12:56:44 -> 12:56:44 0.0 min push_file 9202 OK +2026-11-10 12:56:51 -> 12:56:51 0.0 min push_file 9202 OK +2026-11-10 12:56:52 -> 12:56:52 0.0 min push_file 9202 OK +2026-11-10 12:56:57 -> 12:56:57 0.0 min push_file 9202 OK +2026-11-10 12:57:00 -> 12:57:00 0.0 min push_file 9202 OK +2026-11-10 12:57:14 -> 12:57:15 0.0 min push_file 9202 OK +2026-11-10 12:57:18 -> 12:57:18 0.0 min push_file 9202 OK +2026-11-10 13:24:35 -> 13:24:35 0.0 min push_file 9202 OK +2026-11-10 13:24:36 -> 13:24:36 0.0 min push_file 9202 OK +2026-11-10 13:26:55 -> 13:26:55 0.0 min push_file 9202 OK +2026-11-10 13:31:27 -> 13:31:27 0.0 min push_file 9202 OK +2026-11-10 13:31:30 -> 13:31:30 0.0 min push_file 9202 OK +2026-11-10 13:31:44 -> 13:31:44 0.0 min push_file 9202 OK +2026-11-10 13:31:48 -> 13:31:48 0.0 min push_file 9202 OK +2026-11-10 13:31:53 -> 13:31:53 0.0 min push_file 9202 OK +2026-11-10 13:54:31 -> 13:56:16 1.8 min vzshutdown 9201 OK +2026-11-10 13:54:31 -> 13:56:16 1.8 min vzshutdown 9202 OK +2026-11-10 13:54:31 -> 13:56:26 1.9 min qmshutdown 341 OK +2026-11-10 13:54:31 -> 13:56:26 1.9 min stopall OK +2026-11-10 13:57:17 -> 13:57:20 0.1 min qmstart 341 OK +2026-11-10 13:57:17 -> 13:57:25 0.1 min startall OK +2026-11-10 13:57:20 -> 13:57:25 0.1 min vzstart 9201 OK +2026-11-10 13:57:20 -> 13:57:25 0.1 min vzstart 9202 OK +2026-11-10 15:30:29 -> 15:32:12 1.7 min vzshutdown 9201 OK +2026-11-10 15:30:29 -> 15:32:12 1.7 min vzshutdown 9202 OK +2026-11-10 15:30:29 -> 15:32:22 1.9 min qmshutdown 341 OK +2026-11-10 15:30:29 -> 15:32:23 1.9 min stopall OK +2026-11-10 15:33:12 -> 15:33:15 0.1 min qmstart 341 OK +2026-11-10 15:33:12 -> 15:33:21 0.1 min startall OK +2026-11-10 15:33:15 -> 15:33:21 0.1 min vzstart 9201 OK +2026-11-10 15:33:15 -> 15:33:21 0.1 min vzstart 9202 OK +2026-11-10 15:40:13 -> 15:41:57 1.7 min vzshutdown 9201 OK +2026-11-10 15:40:13 -> 15:41:57 1.7 min vzshutdown 9202 OK +2026-11-10 15:40:13 -> 15:42:17 2.1 min qmshutdown 341 OK +2026-11-10 15:40:13 -> 15:42:17 2.1 min stopall OK +2026-11-10 15:43:07 -> 15:43:10 0.1 min qmstart 341 OK +2026-11-10 15:43:07 -> 15:43:16 0.1 min startall OK +2026-11-10 15:43:10 -> 15:43:16 0.1 min vzstart 9201 OK +2026-11-10 15:43:10 -> 15:43:16 0.1 min vzstart 9202 OK +2026-11-11 02:23:00 -> 02:23:04 0.1 min aptupdate OK +2026-11-11 04:28:20 -> 04:31:22 3.0 min vzrestore 990000 OK +2026-11-11 04:31:26 -> 04:31:30 0.1 min vzstart 990000 OK +2026-11-11 04:31:33 -> 04:31:35 0.0 min vzdestroy 990000 OK +2026-11-11 04:33:02 -> 04:44:08 11.1 min vzdump 9201 OK + +=== agent: night words, 20:00 yesterday to 09:00 today (first 3 lines per message) === +--- counts +RC=0 diff --git a/documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-offset.txt b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-offset.txt new file mode 100644 index 00000000..3e90aadb --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/box/demo-hp-offset.txt @@ -0,0 +1,6 @@ +# offset demo-hp: box clock minus DooPlex clock (midpoint of the call), seconds +offset=0.067 rtt=0.142 +offset=0.073 rtt=0.155 +offset=0.071 rtt=0.152 +offset=0.074 rtt=0.157 +offset=0.069 rtt=0.148 diff --git a/documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-guest-clock.txt b/documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-guest-clock.txt new file mode 100644 index 00000000..17de3b75 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-guest-clock.txt @@ -0,0 +1,465 @@ +# guest-clock tester-1 taken 2026-11-11T10:04:20Z from DooPlex + +=== clock === +utc 2026-11-11T10:04:21.821299550Z +local 2026-11-11T10:04:21 UTC +0000 +/usr/share/zoneinfo/Etc/UTC + Local time: Wed 2026-11-11 10:04:22 UTC + Universal time: Wed 2026-11-11 10:04:22 UTC + RTC time: n/a + Time zone: Etc/UTC (UTC, +0000) +System clock synchronized: yes + NTP service: inactive + RTC in local TZ: no +chrony: inactive +systemd-timesyncd: inactive + +=== boots === +2026-11-10 14:43:51 + +=== timers === +NEXT LEFT LAST PASSED UNIT ACTIVATES +Wed 2026-11-11 14:59:32 UTC 4h 55min Tue 2026-11-10 14:59:32 UTC 19h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service +Wed 2026-11-11 15:04:07 UTC 4h 59min Tue 2026-11-10 18:44:20 UTC 15h ago apt-daily.timer apt-daily.service +Thu 2026-11-12 00:00:00 UTC 13h Wed 2026-11-11 00:00:20 UTC 10h ago dpkg-db-backup.timer dpkg-db-backup.service +Thu 2026-11-12 00:33:59 UTC 14h Wed 2026-11-11 00:01:20 UTC 10h ago logrotate.timer logrotate.service +Thu 2026-11-12 05:02:46 UTC 18h Wed 2026-11-11 02:25:42 UTC 7h ago man-db.timer man-db.service +Thu 2026-11-12 06:02:16 UTC 19h Wed 2026-11-11 06:46:20 UTC 3h 18min ago apt-daily-upgrade.timer apt-daily-upgrade.service +Sun 2026-11-15 03:10:52 UTC 3 days Sun 2026-11-08 03:10:38 UTC - e2scrub_all.timer e2scrub_all.service +- - - - fstrim.timer fstrim.service + +8 timers listed. + +=== timer units (all) === +--- apt-daily-upgrade.timer [enabled/active] +OnCalendar=*-*-* 6:00 +RandomizedDelaySec=60m +Persistent=true +Unit=apt-daily-upgrade.service NextElapseUSecRealtime=Thu 2026-11-12 06:02:16 UTC LastTriggerUSec=Wed 2026-11-11 06:46:20 UTC +--- apt-daily.timer [enabled/active] +OnCalendar=*-*-* 6,18:00 +RandomizedDelaySec=12h +Persistent=true +Unit=apt-daily.service NextElapseUSecRealtime=Wed 2026-11-11 15:04:07 UTC LastTriggerUSec=Tue 2026-11-10 18:44:20 UTC +--- apt-listchanges.timer [disabled/inactive] +OnCalendar=hourly +Unit=apt-listchanges.service NextElapseUSecRealtime= LastTriggerUSec= +--- dpkg-db-backup.timer [enabled/active] +OnCalendar=daily +Persistent=true +Unit=dpkg-db-backup.service NextElapseUSecRealtime=Thu 2026-11-12 00:00:00 UTC LastTriggerUSec=Wed 2026-11-11 00:00:20 UTC +--- e2scrub_all.timer [enabled/active] +OnCalendar=Sun *-*-* 03:10:00 +RandomizedDelaySec=60 +Persistent=true +Unit=e2scrub_all.service NextElapseUSecRealtime=Sun 2026-11-15 03:10:52 UTC LastTriggerUSec=Sun 2026-11-08 03:10:38 UTC +--- fstrim.timer [enabled/inactive] +OnCalendar=weekly +AccuracySec=1h +Persistent=true +RandomizedDelaySec=100min +Unit=fstrim.service NextElapseUSecRealtime= LastTriggerUSec= +--- logrotate.timer [enabled/active] +OnCalendar=daily +RandomizedDelaySec=1h +Persistent=true +Unit=logrotate.service NextElapseUSecRealtime=Thu 2026-11-12 00:33:59 UTC LastTriggerUSec=Wed 2026-11-11 00:01:20 UTC +--- man-db.timer [enabled/active] +OnCalendar=daily +RandomizedDelaySec=12h +Persistent=true +Unit=man-db.service NextElapseUSecRealtime=Thu 2026-11-12 05:02:46 UTC LastTriggerUSec=Wed 2026-11-11 02:25:42 UTC +--- systemd-tmpfiles-clean.timer [static/active] +OnBootSec=15min +OnUnitActiveSec=1d +Unit=systemd-tmpfiles-clean.service NextElapseUSecRealtime= LastTriggerUSec=Tue 2026-11-10 14:59:32 UTC + +=== felhom units === +felhom-controller-bootstrap.path loaded active running +felhom-controller-bootstrap.service loaded active exited +felhom-regen-hostkeys.service loaded inactive dead +--- felhom-controller-bootstrap.service +Type=oneshot Restart=no RestartUSec=100ms TimeoutStartUSec=infinity NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:17:03 UTC +--- felhom-regen-hostkeys.service +Type=oneshot Restart=no RestartUSec=100ms TimeoutStartUSec=infinity NRestarts=0 ActiveEnterTimestamp= + +=== cron === +SHELL=/bin/sh +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +25 * * * * root cd / && run-parts --report /etc/cron.hourly +5 6 * * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; } +8 6 * * 7 root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.weekly; } +58 6 1 * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.monthly; } +--- /etc/cron.d/e2scrub_all +35 3 * * 0 root test -e /run/systemd/system || SERVICE_MODE=1 /usr/libexec/e2fsprogs/e2scrub_all_cron +38 3 * * * root test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r +--- cron.hourly: +--- cron.daily: apt-compat dpkg logrotate man-db +--- cron.weekly: man-db +--- cron.monthly: +--- user crontabs: + +=== at === +no at + +=== docker daemon timers === +{ + "features": { "containerd-snapshotter": false }, + "live-restore": true, + "log-driver": "json-file", + "log-opts": { "max-size": "10m", "max-file": "3" } +} + +=== containers === +privatebin privatebin/pdo:2.1.0 Up 6 hours (healthy) 6 hours ago +paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:3.4.1 Up 6 hours (healthy) 6 hours ago +paperless-redis redis:7-alpine Up 6 hours (healthy) 6 hours ago +paperless-postgres postgres:18-alpine Up 6 hours (healthy) 6 hours ago +bookstack lscr.io/linuxserver/bookstack:26.09.2 Up 6 hours (healthy) 6 hours ago +bookstack-db mariadb:12.3 Up 6 hours (healthy) 6 hours ago +felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.352.0 Up 19 hours (healthy) 19 hours ago +traefik traefik:v3.7.17 Up 19 hours 8 days ago +filebrowser gtstef/filebrowser:1.5.8-stable Up 19 hours (healthy) 12 days ago +cloudflared cloudflare/cloudflared:2026.10.4 Up 19 hours (healthy) 2 weeks ago + +=== restart policies + healthcheck intervals === +/privatebin restart=unless-stopped health=30s started=2026-11-11T04:32:50.895902437Z +/paperless-webserver restart=unless-stopped health=30s started=2026-11-11T04:32:49.930525952Z +/paperless-redis restart=unless-stopped health=10s started=2026-11-11T04:32:39.478872694Z +/paperless-postgres restart=unless-stopped health=10s started=2026-11-11T04:32:39.708807897Z +/bookstack restart=unless-stopped health=30s started=2026-11-11T04:32:38.567990532Z +/bookstack-db restart=unless-stopped health=10s started=2026-11-11T04:32:32.829563948Z +/felhom-controller restart=always health=30s started=2026-11-10T15:17:03.705107516Z +/traefik restart=unless-stopped health=none started=2026-11-10T14:44:04.469852075Z +/filebrowser restart=unless-stopped health=30s started=2026-11-10T14:44:13.671000663Z +/cloudflared restart=unless-stopped health=30s started=2026-11-10T14:44:04.702351924Z + +=== controller TZ === +TZ=unset +in-container 2026-11-11T10:04:23 UTC +0000 + +=== controller log: first line, line count per hour === +2026-11-10T15:17:04.380509770Z 2026/11/10 15:17:04 [INFO] local-api: channel up (agent 169.254.253.1:8443) — guest 9201, 3 mount(s) visible + 881 2026-11-10T15 1093 2026-11-10T16 1088 2026-11-10T17 1090 2026-11-10T18 1096 2026-11-10T19 1096 2026-11-10T20 1095 2026-11-10T21 1095 2026-11-10T22 1101 2026-11-10T23 1102 2026-11-11T00 1165 2026-11-11T01 1148 2026-11-11T02 1221 2026-11-11T03 1149 2026-11-11T04 1104 2026-11-11T05 1100 2026-11-11T06 1105 2026-11-11T07 1111 2026-11-11T08 1113 2026-11-11T09 89 2026-11-11T10 +=== controller log: schedule words (tail 320) === +2026-11-11T09:20:05.160676453Z 2026/11/11 09:20:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:20:05.683405736Z 2026/11/11 09:20:05 [INFO] [scheduler] Job agent-channel-health completed (took 523ms) +2026-11-11T09:21:04.906157881Z 2026/11/11 09:21:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:21:04.906190012Z 2026/11/11 09:21:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:21:04.906196204Z 2026/11/11 09:21:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:21:05.056970262Z 2026/11/11 09:21:05 [INFO] [scheduler] Job stack-scan completed (took 151ms) +2026-11-11T09:21:05.161044883Z 2026/11/11 09:21:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:21:05.664432943Z 2026/11/11 09:21:05 [INFO] [scheduler] Job agent-channel-health completed (took 503ms) +2026-11-11T09:22:04.905778525Z 2026/11/11 09:22:04 [INFO] [scheduler] Running job: backup-cache +2026-11-11T09:22:04.906641884Z 2026/11/11 09:22:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:22:04.906667213Z 2026/11/11 09:22:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:22:04.906671090Z 2026/11/11 09:22:04 [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T09:22:04.906674587Z 2026/11/11 09:22:04 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T09:22:04.906677953Z 2026/11/11 09:22:04 [INFO] [scheduler] Running job: system-health +2026-11-11T09:22:04.906693452Z 2026/11/11 09:22:04 [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T09:22:04.957577420Z 2026/11/11 09:22:04 [INFO] [backup] Found 4 DB dump files across drives +2026-11-11T09:22:05.139314206Z 2026/11/11 09:22:05 [INFO] [scheduler] Job docker-socket-users completed (took 233ms) +2026-11-11T09:22:05.140202182Z 2026/11/11 09:22:05 [INFO] [scheduler] Job backup-cache completed (took 234ms) +2026-11-11T09:22:05.161068167Z 2026/11/11 09:22:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:22:05.252001057Z 2026/11/11 09:22:05 [INFO] [scheduler] Job system-health completed (took 346ms) +2026-11-11T09:22:05.729358951Z 2026/11/11 09:22:05 [INFO] [scheduler] Job agent-channel-health completed (took 568ms) +2026-11-11T09:23:04.906690828Z 2026/11/11 09:23:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:23:04.907120039Z 2026/11/11 09:23:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:23:04.907251316Z 2026/11/11 09:23:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:23:05.063022179Z 2026/11/11 09:23:05 [INFO] [scheduler] Job stack-scan completed (took 156ms) +2026-11-11T09:23:05.160568685Z 2026/11/11 09:23:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:23:05.722110117Z 2026/11/11 09:23:05 [INFO] [scheduler] Job agent-channel-health completed (took 562ms) +2026-11-11T09:24:04.907045613Z 2026/11/11 09:24:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:24:04.907066863Z 2026/11/11 09:24:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:24:05.161051942Z 2026/11/11 09:24:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:24:05.667477165Z 2026/11/11 09:24:05 [INFO] [scheduler] Job agent-channel-health completed (took 506ms) +2026-11-11T09:25:04.906335291Z 2026/11/11 09:25:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:25:04.906871874Z 2026/11/11 09:25:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:25:04.906889197Z 2026/11/11 09:25:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:25:05.065177442Z 2026/11/11 09:25:05 [INFO] [scheduler] Job stack-scan completed (took 159ms) +2026-11-11T09:25:05.160304921Z 2026/11/11 09:25:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:25:05.674373114Z 2026/11/11 09:25:05 [INFO] [scheduler] Job agent-channel-health completed (took 514ms) +2026-11-11T09:26:04.906927150Z 2026/11/11 09:26:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:26:04.907411494Z 2026/11/11 09:26:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:26:05.161348483Z 2026/11/11 09:26:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:26:05.657398417Z 2026/11/11 09:26:05 [INFO] [scheduler] Job agent-channel-health completed (took 496ms) +2026-11-11T09:27:04.909971068Z 2026/11/11 09:27:04 [INFO] [scheduler] Running job: fill-watch-interval +2026-11-11T09:27:04.910033204Z 2026/11/11 09:27:04 [INFO] [scheduler] Job fill-watch-interval completed (took 0s) +2026-11-11T09:27:04.910037132Z 2026/11/11 09:27:04 [INFO] [scheduler] Running job: backup-cache +2026-11-11T09:27:04.910041590Z 2026/11/11 09:27:04 [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T09:27:04.910045058Z 2026/11/11 09:27:04 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T09:27:04.910048514Z 2026/11/11 09:27:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:27:04.910051610Z 2026/11/11 09:27:04 [INFO] [scheduler] Running job: system-health +2026-11-11T09:27:04.910757212Z 2026/11/11 09:27:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:27:04.912246804Z 2026/11/11 09:27:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:27:04.912272412Z 2026/11/11 09:27:04 [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T09:27:04.976000830Z 2026/11/11 09:27:04 [INFO] [backup] Found 4 DB dump files across drives +2026-11-11T09:27:05.151495802Z 2026/11/11 09:27:05 [INFO] [scheduler] Job docker-socket-users completed (took 239ms) +2026-11-11T09:27:05.152952152Z 2026/11/11 09:27:05 [INFO] [scheduler] Job stack-scan completed (took 244ms) +2026-11-11T09:27:05.160243608Z 2026/11/11 09:27:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:27:05.285384311Z 2026/11/11 09:27:05 [INFO] [scheduler] Job system-health completed (took 376ms) +2026-11-11T09:27:05.335228296Z 2026/11/11 09:27:05 [INFO] [scheduler] Job backup-cache completed (took 429ms) +2026-11-11T09:27:05.772479480Z 2026/11/11 09:27:05 [INFO] [scheduler] Job agent-channel-health completed (took 612ms) +2026-11-11T09:28:04.905755596Z 2026/11/11 09:28:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:28:04.905798426Z 2026/11/11 09:28:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:28:05.161118225Z 2026/11/11 09:28:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:28:05.667198788Z 2026/11/11 09:28:05 [INFO] [scheduler] Job agent-channel-health completed (took 506ms) +2026-11-11T09:29:04.907039643Z 2026/11/11 09:29:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:29:04.907070982Z 2026/11/11 09:29:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:29:04.907190909Z 2026/11/11 09:29:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:29:05.060136598Z 2026/11/11 09:29:05 [INFO] [scheduler] Job stack-scan completed (took 154ms) +2026-11-11T09:29:05.161026467Z 2026/11/11 09:29:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:29:05.669569076Z 2026/11/11 09:29:05 [INFO] [scheduler] Job agent-channel-health completed (took 509ms) +2026-11-11T09:30:04.907128947Z 2026/11/11 09:30:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:30:04.907156159Z 2026/11/11 09:30:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:30:05.160948675Z 2026/11/11 09:30:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:30:05.674362853Z 2026/11/11 09:30:05 [INFO] [scheduler] Job agent-channel-health completed (took 513ms) +2026-11-11T09:31:04.906170340Z 2026/11/11 09:31:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:31:04.906216067Z 2026/11/11 09:31:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:31:04.907849150Z 2026/11/11 09:31:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:31:05.055301047Z 2026/11/11 09:31:05 [INFO] [scheduler] Job stack-scan completed (took 148ms) +2026-11-11T09:31:05.161215142Z 2026/11/11 09:31:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:31:05.668219827Z 2026/11/11 09:31:05 [INFO] [scheduler] Job agent-channel-health completed (took 507ms) +2026-11-11T09:32:04.906098122Z 2026/11/11 09:32:04 [INFO] [scheduler] Running job: hub-report +2026-11-11T09:32:04.906159077Z 2026/11/11 09:32:04 [INFO] [scheduler] Running job: backup-cache +2026-11-11T09:32:04.906791701Z 2026/11/11 09:32:04 [INFO] [scheduler] Running job: system-health +2026-11-11T09:32:04.908869394Z 2026/11/11 09:32:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:32:04.908896174Z 2026/11/11 09:32:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:32:04.909802516Z 2026/11/11 09:32:04 [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T09:32:04.909865736Z 2026/11/11 09:32:04 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T09:32:04.909870334Z 2026/11/11 09:32:04 [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T09:32:04.961770841Z 2026/11/11 09:32:04 [INFO] [backup] Found 4 DB dump files across drives +2026-11-11T09:32:05.160909201Z 2026/11/11 09:32:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:32:05.236015390Z 2026/11/11 09:32:05 [INFO] [scheduler] Job docker-socket-users completed (took 327ms) +2026-11-11T09:32:05.389071178Z 2026/11/11 09:32:05 [INFO] [scheduler] Job system-health completed (took 481ms) +2026-11-11T09:32:05.674110667Z 2026/11/11 09:32:05 [INFO] [scheduler] Job backup-cache completed (took 768ms) +2026-11-11T09:32:05.829814684Z 2026/11/11 09:32:05 [INFO] [scheduler] Job agent-channel-health completed (took 667ms) +2026-11-11T09:32:06.010013867Z 2026/11/11 09:32:06 [INFO] [sync] Catalog sync complete +2026-11-11T09:32:06.560976906Z 2026/11/11 09:32:06 [INFO] [scheduler] Job hub-report completed (took 1.655s) +2026-11-11T09:33:04.907112851Z 2026/11/11 09:33:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:33:04.907267472Z 2026/11/11 09:33:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:33:04.907276790Z 2026/11/11 09:33:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:33:05.059922042Z 2026/11/11 09:33:05 [INFO] [scheduler] Job stack-scan completed (took 153ms) +2026-11-11T09:33:05.161104323Z 2026/11/11 09:33:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:33:05.669271854Z 2026/11/11 09:33:05 [INFO] [scheduler] Job agent-channel-health completed (took 508ms) +2026-11-11T09:34:04.906833303Z 2026/11/11 09:34:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:34:04.906854333Z 2026/11/11 09:34:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:34:05.160818043Z 2026/11/11 09:34:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:34:05.678849137Z 2026/11/11 09:34:05 [INFO] [scheduler] Job agent-channel-health completed (took 518ms) +2026-11-11T09:35:04.906633371Z 2026/11/11 09:35:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:35:04.907741413Z 2026/11/11 09:35:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:35:04.907932474Z 2026/11/11 09:35:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:35:05.054765892Z 2026/11/11 09:35:05 [INFO] [scheduler] Job stack-scan completed (took 147ms) +2026-11-11T09:35:05.160883591Z 2026/11/11 09:35:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:35:05.655921946Z 2026/11/11 09:35:05 [INFO] [scheduler] Job agent-channel-health completed (took 495ms) +2026-11-11T09:36:04.905976348Z 2026/11/11 09:36:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:36:04.906399446Z 2026/11/11 09:36:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:36:05.160918645Z 2026/11/11 09:36:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:36:05.670465732Z 2026/11/11 09:36:05 [INFO] [scheduler] Job agent-channel-health completed (took 510ms) +2026-11-11T09:37:04.908816714Z 2026/11/11 09:37:04 [INFO] [scheduler] Running job: backup-cache +2026-11-11T09:37:04.908842753Z 2026/11/11 09:37:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:37:04.908846751Z 2026/11/11 09:37:04 [INFO] [scheduler] Running job: fill-watch-interval +2026-11-11T09:37:04.908853283Z 2026/11/11 09:37:04 [INFO] [scheduler] Job fill-watch-interval completed (took 0s) +2026-11-11T09:37:04.908856278Z 2026/11/11 09:37:04 [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T09:37:04.908859134Z 2026/11/11 09:37:04 [INFO] [scheduler] Running job: system-health +2026-11-11T09:37:04.908861879Z 2026/11/11 09:37:04 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T09:37:04.908864625Z 2026/11/11 09:37:04 [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T09:37:04.908867419Z 2026/11/11 09:37:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:37:04.908870135Z 2026/11/11 09:37:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:37:05.092148252Z 2026/11/11 09:37:05 [INFO] [scheduler] Job stack-scan completed (took 185ms) +2026-11-11T09:37:05.119825034Z 2026/11/11 09:37:05 [INFO] [scheduler] Job docker-socket-users completed (took 214ms) +2026-11-11T09:37:05.147265501Z 2026/11/11 09:37:05 [INFO] [backup] Found 4 DB dump files across drives +2026-11-11T09:37:05.160883592Z 2026/11/11 09:37:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:37:05.272417437Z 2026/11/11 09:37:05 [INFO] [scheduler] Job system-health completed (took 366ms) +2026-11-11T09:37:05.318963655Z 2026/11/11 09:37:05 [INFO] [scheduler] Job backup-cache completed (took 413ms) +2026-11-11T09:37:05.787306309Z 2026/11/11 09:37:05 [INFO] [scheduler] Job agent-channel-health completed (took 627ms) +2026-11-11T09:38:04.914499105Z 2026/11/11 09:38:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:38:04.914531786Z 2026/11/11 09:38:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:38:05.160319781Z 2026/11/11 09:38:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:38:05.664415855Z 2026/11/11 09:38:05 [INFO] [scheduler] Job agent-channel-health completed (took 504ms) +2026-11-11T09:39:04.905994526Z 2026/11/11 09:39:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:39:04.906975628Z 2026/11/11 09:39:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:39:04.907017007Z 2026/11/11 09:39:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:39:05.009613137Z 2026/11/11 09:39:05 [INFO] [scheduler] Job stack-scan completed (took 104ms) +2026-11-11T09:39:05.160344104Z 2026/11/11 09:39:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:39:05.667819880Z 2026/11/11 09:39:05 [INFO] [scheduler] Job agent-channel-health completed (took 507ms) +2026-11-11T09:40:04.908299813Z 2026/11/11 09:40:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:40:04.908326914Z 2026/11/11 09:40:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:40:05.160622744Z 2026/11/11 09:40:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:40:05.646534628Z 2026/11/11 09:40:05 [INFO] [scheduler] Job agent-channel-health completed (took 486ms) +2026-11-11T09:41:04.908980390Z 2026/11/11 09:41:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:41:04.909024773Z 2026/11/11 09:41:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:41:04.909032287Z 2026/11/11 09:41:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:41:05.059955628Z 2026/11/11 09:41:05 [INFO] [scheduler] Job stack-scan completed (took 151ms) +2026-11-11T09:41:05.160896905Z 2026/11/11 09:41:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:41:05.673532120Z 2026/11/11 09:41:05 [INFO] [scheduler] Job agent-channel-health completed (took 513ms) +2026-11-11T09:42:04.909896065Z 2026/11/11 09:42:04 [INFO] [scheduler] Running job: backup-cache +2026-11-11T09:42:04.909933365Z 2026/11/11 09:42:04 [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T09:42:04.909939897Z 2026/11/11 09:42:04 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T09:42:04.909969123Z 2026/11/11 09:42:04 [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T09:42:04.909976496Z 2026/11/11 09:42:04 [INFO] [scheduler] Running job: system-health +2026-11-11T09:42:04.909996925Z 2026/11/11 09:42:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:42:04.910002816Z 2026/11/11 09:42:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:42:04.979062770Z 2026/11/11 09:42:04 [INFO] [backup] Found 4 DB dump files across drives +2026-11-11T09:42:05.146209070Z 2026/11/11 09:42:05 [INFO] [scheduler] Job docker-socket-users completed (took 238ms) +2026-11-11T09:42:05.160235922Z 2026/11/11 09:42:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:42:05.168840697Z 2026/11/11 09:42:05 [INFO] [scheduler] Job backup-cache completed (took 263ms) +2026-11-11T09:42:05.255446688Z 2026/11/11 09:42:05 [INFO] [scheduler] Job system-health completed (took 346ms) +2026-11-11T09:42:05.725984005Z 2026/11/11 09:42:05 [INFO] [scheduler] Job agent-channel-health completed (took 565ms) +2026-11-11T09:43:04.906280103Z 2026/11/11 09:43:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:43:04.907075686Z 2026/11/11 09:43:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:43:04.907105071Z 2026/11/11 09:43:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:43:05.069338426Z 2026/11/11 09:43:05 [INFO] [scheduler] Job stack-scan completed (took 162ms) +2026-11-11T09:43:05.160939298Z 2026/11/11 09:43:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:43:05.730904454Z 2026/11/11 09:43:05 [INFO] [scheduler] Job agent-channel-health completed (took 570ms) +2026-11-11T09:44:04.906796713Z 2026/11/11 09:44:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:44:04.908982730Z 2026/11/11 09:44:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:44:05.161369203Z 2026/11/11 09:44:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:44:05.660224386Z 2026/11/11 09:44:05 [INFO] [scheduler] Job agent-channel-health completed (took 499ms) +2026-11-11T09:45:04.906259715Z 2026/11/11 09:45:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:45:04.906294671Z 2026/11/11 09:45:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:45:04.909803566Z 2026/11/11 09:45:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:45:05.072780594Z 2026/11/11 09:45:05 [INFO] [scheduler] Job stack-scan completed (took 166ms) +2026-11-11T09:45:05.161751480Z 2026/11/11 09:45:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:45:05.720180919Z 2026/11/11 09:45:05 [INFO] [scheduler] Job agent-channel-health completed (took 559ms) +2026-11-11T09:46:04.906118553Z 2026/11/11 09:46:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:46:04.906319432Z 2026/11/11 09:46:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:46:05.160560375Z 2026/11/11 09:46:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:46:05.657381989Z 2026/11/11 09:46:05 [INFO] [scheduler] Job agent-channel-health completed (took 497ms) +2026-11-11T09:47:04.905860034Z 2026/11/11 09:47:04 [INFO] [scheduler] Running job: backup-cache +2026-11-11T09:47:04.907815135Z 2026/11/11 09:47:04 [INFO] [backup] Found 4 DB dump files across drives +2026-11-11T09:47:04.917507445Z 2026/11/11 09:47:04 [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T09:47:04.917539665Z 2026/11/11 09:47:04 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T09:47:04.917543462Z 2026/11/11 09:47:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:47:04.917562007Z 2026/11/11 09:47:04 [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T09:47:04.922234609Z 2026/11/11 09:47:04 [INFO] [scheduler] Running job: system-health +2026-11-11T09:47:04.926517074Z 2026/11/11 09:47:04 [INFO] [scheduler] Running job: fill-watch-interval +2026-11-11T09:47:04.929753685Z 2026/11/11 09:47:04 [INFO] [scheduler] Job fill-watch-interval completed (took 0s) +2026-11-11T09:47:04.934823557Z 2026/11/11 09:47:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:47:04.935067388Z 2026/11/11 09:47:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:47:04.935197834Z 2026/11/11 09:47:04 [INFO] [scheduler] Running job: hub-report +2026-11-11T09:47:05.161685018Z 2026/11/11 09:47:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:47:05.286530074Z 2026/11/11 09:47:05 [INFO] [scheduler] Job docker-socket-users completed (took 367ms) +2026-11-11T09:47:05.293348628Z 2026/11/11 09:47:05 [INFO] [scheduler] Job stack-scan completed (took 376ms) +2026-11-11T09:47:05.324248857Z 2026/11/11 09:47:05 [INFO] [sync] Catalog sync complete +2026-11-11T09:47:05.414292479Z 2026/11/11 09:47:05 [INFO] [scheduler] Job system-health completed (took 491ms) +2026-11-11T09:47:05.782668478Z 2026/11/11 09:47:05 [INFO] [scheduler] Job agent-channel-health completed (took 621ms) +2026-11-11T09:47:05.948318272Z 2026/11/11 09:47:05 [INFO] [scheduler] Job backup-cache completed (took 1.043s) +2026-11-11T09:47:06.674302522Z 2026/11/11 09:47:06 [INFO] [scheduler] Job hub-report completed (took 1.739s) +2026-11-11T09:48:04.906768544Z 2026/11/11 09:48:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:48:04.906796998Z 2026/11/11 09:48:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:48:05.160288997Z 2026/11/11 09:48:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:48:05.646123176Z 2026/11/11 09:48:05 [INFO] [scheduler] Job agent-channel-health completed (took 486ms) +2026-11-11T09:49:04.907042614Z 2026/11/11 09:49:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:49:04.908892617Z 2026/11/11 09:49:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:49:04.908914979Z 2026/11/11 09:49:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:49:05.049545998Z 2026/11/11 09:49:05 [INFO] [scheduler] Job stack-scan completed (took 142ms) +2026-11-11T09:49:05.161155636Z 2026/11/11 09:49:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:49:05.674796784Z 2026/11/11 09:49:05 [INFO] [scheduler] Job agent-channel-health completed (took 514ms) +2026-11-11T09:50:04.906998145Z 2026/11/11 09:50:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:50:04.907051226Z 2026/11/11 09:50:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:50:05.161225283Z 2026/11/11 09:50:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:50:05.673788877Z 2026/11/11 09:50:05 [INFO] [scheduler] Job agent-channel-health completed (took 513ms) +2026-11-11T09:51:04.906823229Z 2026/11/11 09:51:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:51:04.906868464Z 2026/11/11 09:51:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:51:04.906874867Z 2026/11/11 09:51:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:51:05.023425543Z 2026/11/11 09:51:05 [INFO] [scheduler] Job stack-scan completed (took 117ms) +2026-11-11T09:51:05.160588625Z 2026/11/11 09:51:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:51:05.650758119Z 2026/11/11 09:51:05 [INFO] [scheduler] Job agent-channel-health completed (took 490ms) +2026-11-11T09:52:04.906081037Z 2026/11/11 09:52:04 [INFO] [scheduler] Running job: backup-cache +2026-11-11T09:52:04.906372236Z 2026/11/11 09:52:04 [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T09:52:04.906387466Z 2026/11/11 09:52:04 [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T09:52:04.906512150Z 2026/11/11 09:52:04 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T09:52:04.906527740Z 2026/11/11 09:52:04 [INFO] [scheduler] Running job: system-health +2026-11-11T09:52:04.908160293Z 2026/11/11 09:52:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:52:04.908185821Z 2026/11/11 09:52:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:52:04.958138891Z 2026/11/11 09:52:04 [INFO] [backup] Found 4 DB dump files across drives +2026-11-11T09:52:05.128735957Z 2026/11/11 09:52:05 [INFO] [scheduler] Job backup-cache completed (took 222ms) +2026-11-11T09:52:05.158951424Z 2026/11/11 09:52:05 [INFO] [scheduler] Job docker-socket-users completed (took 253ms) +2026-11-11T09:52:05.161798499Z 2026/11/11 09:52:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:52:05.248963214Z 2026/11/11 09:52:05 [INFO] [scheduler] Job system-health completed (took 342ms) +2026-11-11T09:52:05.703048703Z 2026/11/11 09:52:05 [INFO] [scheduler] Job agent-channel-health completed (took 542ms) +2026-11-11T09:53:04.906946847Z 2026/11/11 09:53:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:53:04.906971574Z 2026/11/11 09:53:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:53:04.906984368Z 2026/11/11 09:53:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:53:05.070103264Z 2026/11/11 09:53:05 [INFO] [scheduler] Job stack-scan completed (took 164ms) +2026-11-11T09:53:05.161115956Z 2026/11/11 09:53:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:53:05.662560556Z 2026/11/11 09:53:05 [INFO] [scheduler] Job agent-channel-health completed (took 501ms) +2026-11-11T09:54:04.906582146Z 2026/11/11 09:54:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:54:04.906901760Z 2026/11/11 09:54:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:54:05.161094072Z 2026/11/11 09:54:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:54:05.655618268Z 2026/11/11 09:54:05 [INFO] [scheduler] Job agent-channel-health completed (took 495ms) +2026-11-11T09:55:04.908006862Z 2026/11/11 09:55:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:55:04.908089568Z 2026/11/11 09:55:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:55:04.908099978Z 2026/11/11 09:55:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:55:05.076915529Z 2026/11/11 09:55:05 [INFO] [scheduler] Job stack-scan completed (took 171ms) +2026-11-11T09:55:05.161220797Z 2026/11/11 09:55:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:55:05.676659488Z 2026/11/11 09:55:05 [INFO] [scheduler] Job agent-channel-health completed (took 515ms) +2026-11-11T09:56:04.906822968Z 2026/11/11 09:56:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:56:04.906848386Z 2026/11/11 09:56:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:56:05.160840460Z 2026/11/11 09:56:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:56:05.647325257Z 2026/11/11 09:56:05 [INFO] [scheduler] Job agent-channel-health completed (took 487ms) +2026-11-11T09:57:04.906004970Z 2026/11/11 09:57:04 [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T09:57:04.906045016Z 2026/11/11 09:57:04 [INFO] [scheduler] Running job: system-health +2026-11-11T09:57:04.906050737Z 2026/11/11 09:57:04 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T09:57:04.906055866Z 2026/11/11 09:57:04 [INFO] [scheduler] Running job: fill-watch-interval +2026-11-11T09:57:04.906469476Z 2026/11/11 09:57:04 [INFO] [scheduler] Running job: backup-cache +2026-11-11T09:57:04.906490947Z 2026/11/11 09:57:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:57:04.906788419Z 2026/11/11 09:57:04 [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T09:57:04.907257544Z 2026/11/11 09:57:04 [INFO] [scheduler] Job fill-watch-interval completed (took 1ms) +2026-11-11T09:57:04.907261371Z 2026/11/11 09:57:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:57:04.907264868Z 2026/11/11 09:57:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:57:05.087667456Z 2026/11/11 09:57:05 [INFO] [scheduler] Job stack-scan completed (took 179ms) +2026-11-11T09:57:05.087754741Z 2026/11/11 09:57:05 [INFO] [backup] Found 4 DB dump files across drives +2026-11-11T09:57:05.099198416Z 2026/11/11 09:57:05 [INFO] [scheduler] Job docker-socket-users completed (took 193ms) +2026-11-11T09:57:05.160555308Z 2026/11/11 09:57:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:57:05.240108604Z 2026/11/11 09:57:05 [INFO] [scheduler] Job system-health completed (took 335ms) +2026-11-11T09:57:05.259868099Z 2026/11/11 09:57:05 [INFO] [scheduler] Job backup-cache completed (took 353ms) +2026-11-11T09:57:05.723041564Z 2026/11/11 09:57:05 [INFO] [scheduler] Job agent-channel-health completed (took 563ms) +2026-11-11T09:58:04.906409622Z 2026/11/11 09:58:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:58:04.906432204Z 2026/11/11 09:58:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:58:05.161208860Z 2026/11/11 09:58:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:58:05.669828277Z 2026/11/11 09:58:05 [INFO] [scheduler] Job agent-channel-health completed (took 509ms) +2026-11-11T09:59:04.906890675Z 2026/11/11 09:59:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T09:59:04.908525281Z 2026/11/11 09:59:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T09:59:04.908548084Z 2026/11/11 09:59:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T09:59:05.044093351Z 2026/11/11 09:59:05 [INFO] [scheduler] Job stack-scan completed (took 137ms) +2026-11-11T09:59:05.161227580Z 2026/11/11 09:59:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T09:59:05.665779027Z 2026/11/11 09:59:05 [INFO] [scheduler] Job agent-channel-health completed (took 505ms) +2026-11-11T10:00:04.906894655Z 2026/11/11 10:00:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T10:00:04.907004353Z 2026/11/11 10:00:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T10:00:05.160977310Z 2026/11/11 10:00:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T10:00:05.672821867Z 2026/11/11 10:00:05 [INFO] [scheduler] Job agent-channel-health completed (took 512ms) +2026-11-11T10:01:04.906496615Z 2026/11/11 10:01:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T10:01:04.906855773Z 2026/11/11 10:01:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T10:01:04.906872295Z 2026/11/11 10:01:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T10:01:05.039623186Z 2026/11/11 10:01:05 [INFO] [scheduler] Job stack-scan completed (took 133ms) +2026-11-11T10:01:05.160440789Z 2026/11/11 10:01:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T10:01:05.640544930Z 2026/11/11 10:01:05 [INFO] [scheduler] Job agent-channel-health completed (took 480ms) +2026-11-11T10:02:04.907854323Z 2026/11/11 10:02:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T10:02:04.907947148Z 2026/11/11 10:02:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T10:02:04.907955415Z 2026/11/11 10:02:04 [INFO] [scheduler] Running job: backup-cache +2026-11-11T10:02:04.907959652Z 2026/11/11 10:02:04 [INFO] [scheduler] Running job: offsite-credential-retry +2026-11-11T10:02:04.907963460Z 2026/11/11 10:02:04 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026-11-11T10:02:04.907967127Z 2026/11/11 10:02:04 [INFO] [scheduler] Running job: hub-report +2026-11-11T10:02:04.907973949Z 2026/11/11 10:02:04 [INFO] [scheduler] Running job: docker-socket-users +2026-11-11T10:02:04.908108814Z 2026/11/11 10:02:04 [INFO] [scheduler] Running job: system-health +2026-11-11T10:02:04.975842475Z 2026/11/11 10:02:04 [INFO] [backup] Found 4 DB dump files across drives +2026-11-11T10:02:04.976149264Z 2026/11/11 10:02:04 [INFO] [sync] Catalog sync complete +2026-11-11T10:02:05.160870204Z 2026/11/11 10:02:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T10:02:05.206269368Z 2026/11/11 10:02:05 [INFO] [scheduler] Job docker-socket-users completed (took 298ms) +2026-11-11T10:02:05.289480299Z 2026/11/11 10:02:05 [INFO] [scheduler] Job system-health completed (took 382ms) +2026-11-11T10:02:05.774210864Z 2026/11/11 10:02:05 [INFO] [scheduler] Job backup-cache completed (took 866ms) +2026-11-11T10:02:05.823947156Z 2026/11/11 10:02:05 [INFO] [scheduler] Job agent-channel-health completed (took 663ms) +2026-11-11T10:02:06.515959397Z 2026/11/11 10:02:06 [INFO] [scheduler] Job hub-report completed (took 1.608s) +2026-11-11T10:03:04.908587389Z 2026/11/11 10:03:04 [INFO] [scheduler] Running job: stack-scan +2026-11-11T10:03:04.908621484Z 2026/11/11 10:03:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T10:03:04.908678831Z 2026/11/11 10:03:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T10:03:05.045884214Z 2026/11/11 10:03:05 [INFO] [scheduler] Job stack-scan completed (took 138ms) +2026-11-11T10:03:05.160800565Z 2026/11/11 10:03:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T10:03:05.663391911Z 2026/11/11 10:03:05 [INFO] [scheduler] Job agent-channel-health completed (took 503ms) +2026-11-11T10:04:04.909163911Z 2026/11/11 10:04:04 [INFO] [scheduler] Running job: hold-report +2026-11-11T10:04:04.909233302Z 2026/11/11 10:04:04 [INFO] [scheduler] Job hold-report completed (took 0s) +2026-11-11T10:04:05.160865681Z 2026/11/11 10:04:05 [INFO] [scheduler] Running job: agent-channel-health +2026-11-11T10:04:05.701032100Z 2026/11/11 10:04:05 [INFO] [scheduler] Job agent-channel-health completed (took 541ms) +RC=0 diff --git a/documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-guest-night.txt b/documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-guest-night.txt new file mode 100644 index 00000000..7dc9b56b --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-guest-night.txt @@ -0,0 +1,227 @@ +# guest-night tester-1 taken 2026-11-11T10:09:26Z +log first line: 2026-11-10T15:17:04.380509770Z +log last line: 2026-11-11T10:09:24.943219455Z + +=== scheduler jobs (UTC): runs, first start, last start, last three "took" === +agent-channel-health runs=1132 first=2026-11-10T15:18:05 last=2026-11-11T10:09:05 [completed (took 595ms)] [completed (took 510ms)] [completed (took 506ms)] +backup-cache runs=226 first=2026-11-10T15:22:04 last=2026-11-11T10:07:04 [completed (took 353ms)] [completed (took 866ms)] [completed (took 421ms)] +conversion-copy-release runs=18 first=2026-11-10T16:17:04 last=2026-11-11T09:17:04 [completed (took 234ms)] [completed (took 291ms)] [completed (took 62ms)] +db-dump runs=1 first=2026-11-11T01:30:00 last=2026-11-11T01:30:00 [completed (took 40.147s)] +disk-health-check runs=18 first=2026-11-10T16:17:05 last=2026-11-11T09:17:05 [completed (took 1.388s)] [completed (took 826ms)] [completed (took 1.142s)] +docker-socket-users runs=226 first=2026-11-10T15:22:04 last=2026-11-11T10:07:04 [completed (took 193ms)] [completed (took 298ms)] [completed (took 215ms)] +fill-watch runs=1 first=2026-11-11T02:30:00 last=2026-11-11T02:30:00 [completed (took 0s)] +fill-watch-interval runs=113 first=2026-11-10T15:27:04 last=2026-11-11T10:07:04 [completed (took 0s)] [completed (took 1ms)] [completed (took 2ms)] +geo-verify runs=3 first=2026-11-10T21:17:04 last=2026-11-11T09:17:04 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)] +hold-report runs=1132 first=2026-11-10T15:18:04 last=2026-11-11T10:09:04 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)] +hub-report runs=75 first=2026-11-10T15:32:04 last=2026-11-11T10:02:04 [completed (took 1.655s)] [completed (took 1.739s)] [completed (took 1.608s)] +metrics-prune runs=1 first=2026-11-11T03:00:00 last=2026-11-11T03:00:00 [completed (took 125ms)] +offbox-backup runs=1 first=2026-11-11T03:15:00 last=2026-11-11T03:15:00 [completed (took 58.568s)] +offsite-abandon-sweep runs=1 first=2026-11-11T04:10:00 last=2026-11-11T04:10:00 [completed (took 0s)] +offsite-credential-retry runs=226 first=2026-11-10T15:22:04 last=2026-11-11T10:07:04 [completed (took 0s)] [completed (took 0s)] [completed (took 0s)] +offsite-integrity runs=1 first=2026-11-11T05:00:00 last=2026-11-11T05:00:00 [completed (took 0s)] +offsite-proof runs=1 first=2026-11-11T04:30:00 last=2026-11-11T04:30:00 [completed (took 5.06s)] +restore-copy-sweep runs=3 first=2026-11-10T21:17:04 last=2026-11-11T09:17:04 [completed (took 5ms)] [completed (took 0s)] [completed (took 10ms)] +selfupdate-check runs=3 first=2026-11-10T21:17:04 last=2026-11-11T09:17:04 [completed (took 49ms)] [completed (took 119ms)] [completed (took 60ms)] +stack-scan runs=566 first=2026-11-10T15:19:04 last=2026-11-11T10:09:04 [completed (took 221ms)] [completed (took 216ms)] [completed (took 138ms)] +system-health runs=226 first=2026-11-10T15:22:04 last=2026-11-11T10:07:04 [completed (took 335ms)] [completed (took 382ms)] [completed (took 335ms)] +tier2-backup runs=1 first=2026-11-11T02:30:00 last=2026-11-11T02:30:00 [completed (took 15.055s)] + +=== jobs that ran fewer than 30 times: every start and end (UTC) === +2026/11/10 16:17:04 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/10 16:17:04 [INFO] [scheduler] Job conversion-copy-release completed (took 1ms) +2026/11/10 16:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/10 16:17:05 [INFO] [scheduler] Job disk-health-check completed (took 763ms) +2026/11/10 17:17:04 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/10 17:17:04 [INFO] [scheduler] Job conversion-copy-release completed (took 28ms) +2026/11/10 17:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/10 17:17:06 [INFO] [scheduler] Job disk-health-check completed (took 888ms) +2026/11/10 18:17:04 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/10 18:17:04 [INFO] [scheduler] Job conversion-copy-release completed (took 62ms) +2026/11/10 18:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/10 18:17:06 [INFO] [scheduler] Job disk-health-check completed (took 1.2s) +2026/11/10 19:17:04 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/10 19:17:04 [INFO] [scheduler] Job conversion-copy-release completed (took 1ms) +2026/11/10 19:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/10 19:17:06 [INFO] [scheduler] Job disk-health-check completed (took 903ms) +2026/11/10 20:17:04 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/10 20:17:04 [INFO] [scheduler] Job conversion-copy-release completed (took 34ms) +2026/11/10 20:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/10 20:17:06 [INFO] [scheduler] Job disk-health-check completed (took 1.147s) +2026/11/10 21:17:04 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/10 21:17:04 [INFO] [scheduler] Job conversion-copy-release completed (took 1ms) +2026/11/10 21:17:04 [INFO] [scheduler] Running job: selfupdate-check +2026/11/10 21:17:04 [INFO] [scheduler] Running job: restore-copy-sweep +2026/11/10 21:17:04 [INFO] [scheduler] Job restore-copy-sweep completed (took 5ms) +2026/11/10 21:17:04 [INFO] [scheduler] Running job: geo-verify +2026/11/10 21:17:04 [INFO] [scheduler] Job geo-verify completed (took 0s) +2026/11/10 21:17:04 [INFO] [scheduler] Job selfupdate-check completed (took 49ms) +2026/11/10 21:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/10 21:17:06 [INFO] [scheduler] Job disk-health-check completed (took 1.208s) +2026/11/10 22:17:04 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/10 22:17:04 [INFO] [scheduler] Job conversion-copy-release completed (took 1ms) +2026/11/10 22:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/10 22:17:05 [INFO] [scheduler] Job disk-health-check completed (took 814ms) +2026/11/10 23:17:04 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/10 23:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/10 23:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 264ms) +2026/11/10 23:17:05 [INFO] [scheduler] Job disk-health-check completed (took 774ms) +2026/11/11 00:17:04 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 00:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 249ms) +2026/11/11 00:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 00:17:06 [INFO] [scheduler] Job disk-health-check completed (took 866ms) +2026/11/11 01:17:04 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 01:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 01:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 321ms) +2026/11/11 01:17:05 [INFO] [scheduler] Job disk-health-check completed (took 787ms) +2026/11/11 01:30:00 [INFO] [scheduler] Running job: db-dump +2026/11/11 01:30:40 [INFO] [scheduler] Job db-dump completed (took 40.147s) +2026/11/11 02:17:04 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 02:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 221ms) +2026/11/11 02:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 02:17:06 [INFO] [scheduler] Job disk-health-check completed (took 1.182s) +2026/11/11 02:30:00 [INFO] [scheduler] Running job: tier2-backup +2026/11/11 02:30:00 [INFO] [scheduler] Running job: fill-watch +2026/11/11 02:30:00 [INFO] [scheduler] Job fill-watch completed (took 0s) +2026/11/11 02:30:15 [INFO] [scheduler] Job tier2-backup completed (took 15.055s) +2026/11/11 03:00:00 [INFO] [scheduler] Running job: metrics-prune +2026/11/11 03:00:00 [INFO] [scheduler] Job metrics-prune completed (took 125ms) +2026/11/11 03:15:00 [INFO] [scheduler] Running job: offbox-backup +2026/11/11 03:15:58 [INFO] [scheduler] Job offbox-backup completed (took 58.568s) +2026/11/11 03:17:04 [INFO] [scheduler] Running job: selfupdate-check +2026/11/11 03:17:04 [INFO] [scheduler] Running job: restore-copy-sweep +2026/11/11 03:17:04 [INFO] [scheduler] Job restore-copy-sweep completed (took 0s) +2026/11/11 03:17:04 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 03:17:04 [INFO] [scheduler] Running job: geo-verify +2026/11/11 03:17:04 [INFO] [scheduler] Job geo-verify completed (took 0s) +2026/11/11 03:17:05 [INFO] [scheduler] Job selfupdate-check completed (took 119ms) +2026/11/11 03:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 03:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 454ms) +2026/11/11 03:17:06 [INFO] [scheduler] Job disk-health-check completed (took 1.115s) +2026/11/11 04:10:00 [INFO] [scheduler] Running job: offsite-abandon-sweep +2026/11/11 04:10:00 [INFO] [scheduler] Job offsite-abandon-sweep completed (took 0s) +2026/11/11 04:17:04 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 04:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 04:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 278ms) +2026/11/11 04:17:06 [INFO] [scheduler] Job disk-health-check completed (took 848ms) +2026/11/11 04:30:00 [INFO] [scheduler] Running job: offsite-proof +2026/11/11 04:30:05 [INFO] [scheduler] Job offsite-proof completed (took 5.06s) +2026/11/11 05:00:00 [INFO] [scheduler] Running job: offsite-integrity +2026/11/11 05:00:00 [INFO] [scheduler] Job offsite-integrity completed (took 0s) +2026/11/11 05:17:04 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 05:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 05:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 304ms) +2026/11/11 05:17:05 [INFO] [scheduler] Job disk-health-check completed (took 806ms) +2026/11/11 06:17:04 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 06:17:04 [INFO] [scheduler] Job conversion-copy-release completed (took 52ms) +2026/11/11 06:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 06:17:06 [INFO] [scheduler] Job disk-health-check completed (took 827ms) +2026/11/11 07:17:04 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 07:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 234ms) +2026/11/11 07:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 07:17:06 [INFO] [scheduler] Job disk-health-check completed (took 1.388s) +2026/11/11 08:17:04 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 08:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 08:17:05 [INFO] [scheduler] Job conversion-copy-release completed (took 291ms) +2026/11/11 08:17:05 [INFO] [scheduler] Job disk-health-check completed (took 826ms) +2026/11/11 09:17:04 [INFO] [scheduler] Running job: selfupdate-check +2026/11/11 09:17:04 [INFO] [scheduler] Running job: conversion-copy-release +2026/11/11 09:17:04 [INFO] [scheduler] Running job: geo-verify +2026/11/11 09:17:04 [INFO] [scheduler] Job geo-verify completed (took 0s) +2026/11/11 09:17:04 [INFO] [scheduler] Running job: restore-copy-sweep +2026/11/11 09:17:04 [INFO] [scheduler] Job restore-copy-sweep completed (took 10ms) +2026/11/11 09:17:04 [INFO] [scheduler] Job selfupdate-check completed (took 60ms) +2026/11/11 09:17:04 [INFO] [scheduler] Job conversion-copy-release completed (took 62ms) +2026/11/11 09:17:05 [INFO] [scheduler] Running job: disk-health-check +2026/11/11 09:17:06 [INFO] [scheduler] Job disk-health-check completed (took 1.142s) + +=== night lines 19:00-08:00 UTC, INFO/WARN/ERROR, backup words (first 2 per message shape) === +2026/11/10 19:02:04 [INFO] [backup] Found 4 DB dump files across drives +2026/11/10 19:02:05 [INFO] [backup] Discovered 2 databases +2026/11/10 19:02:05 [INFO] [backup] Discovered app data: 3 apps +2026/11/10 19:02:05 [INFO] [monitor] Health check: status=ok +2026/11/10 19:02:05 [INFO] [backup] Backup status cache refreshed +2026/11/10 19:02:06 [INFO] [monitor] Health check: status=ok +2026/11/10 19:07:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok +2026/11/10 19:07:04 [INFO] [backup] Found 4 DB dump files across drives +2026/11/10 19:07:05 [INFO] [backup] Discovered 2 databases +2026/11/10 19:07:05 [INFO] [backup] Discovered app data: 3 apps +2026/11/10 19:07:05 [INFO] [backup] Backup status cache refreshed +2026/11/10 19:08:05 [INFO] [deadapp] check alive: 460 scans since boot, 3 deployed app(s) evaluated, 0 currently down +2026/11/10 19:17:04 [INFO] [fillwatch] checked 2 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok +2026/11/10 19:17:06 [INFO] [web] disk-health check complete: 0 disk(s) evaluated, 0 alert(s) +2026/11/10 19:18:05 [INFO] [deadapp] check alive: 480 scans since boot, 3 deployed app(s) evaluated, 0 currently down +2026/11/10 20:17:06 [INFO] [web] disk-health check complete: 0 disk(s) evaluated, 0 alert(s) +2026/11/11 01:30:00 [INFO] [backup] Starting database dump run +2026/11/11 01:30:00 [INFO] [backup] Discovered 2 database(s): paperless-postgres(postgres), bookstack-db(mariadb) +2026/11/11 01:30:00 [INFO] [backup] DB dump: paperless-postgres → paperless-ngx-postgres.sql (557.9 KB, 572ms, 75 tables) +2026/11/11 01:30:01 [INFO] [backup] DB dump: bookstack-db → bookstack-mariadb.sql (57.9 KB, 426ms, 41 tables) +2026/11/11 01:30:01 [INFO] [backup] Stopping bookstack for safe volume dump +2026/11/11 01:30:04 [INFO] [busymark] busy: a backup, a restore or an off-site run is running — an operator reboot of the box is refused until it ends +2026/11/11 01:30:06 [INFO] [backup] Volume dump: bookstack/bookstack_bookstack_config → 3.3 MB +2026/11/11 01:30:07 [INFO] [backup] Volume dump: bookstack/bookstack_bookstack_db_data → 153.4 MB +2026/11/11 01:30:07 [INFO] [backup] Restarting bookstack after volume dump +2026/11/11 01:30:13 [INFO] [backup] Stopping paperless-ngx for safe volume dump +2026/11/11 01:30:23 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_data → 17.4 MB +2026/11/11 01:30:25 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_postgres_data → 70.4 MB +2026/11/11 01:30:25 [INFO] [backup] Restarting paperless-ngx after volume dump +2026/11/11 01:30:37 [INFO] [backup] Stopping privatebin for safe volume dump +2026/11/11 01:30:39 [INFO] [backup] Volume dump: privatebin/privatebin_privatebin_data → 8.5 KB +2026/11/11 01:30:39 [INFO] [backup] Restarting privatebin after volume dump +2026/11/11 01:30:39 [INFO] [backup] App-data backup completed: 2 databases (615.8 KB total), 3 volume dump(s) (39.942s) +2026/11/11 01:30:39 [INFO] [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1) +2026/11/11 01:30:39 [INFO] [backup] Recovery unit captured for paperless-ngx → /mnt/felhom-drives/adatlemez/backups/primary/paperless-ngx (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026/11/11 01:30:39 [INFO] [backup] Recovery unit captured for privatebin → /mnt/sys_drive/felhom-data/backups/primary/privatebin (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/11/11 01:30:40 [INFO] Event pushed: db_dump_completed (info) [hu-only] — Adatbázis mentés elkészült +2026/11/11 01:30:49 [INFO] [busymark] idle again (was: a backup, a restore or an off-site run is running) — an operator reboot of the box is no longer refused for it +2026/11/11 02:30:04 [INFO] [backup] Tier 2 copied bookstack → /mnt/felhom-drives/adatlemez/backups/secondary/bookstack (156.8 MB, 0 leg(s), 3s) +2026/11/11 02:30:10 [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sys_drive/felhom-data/backups/secondary/paperless-ngx (96.1 MB, 1 leg(s), 5s) [SSD: state-only] +2026/11/11 02:30:15 [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/adatlemez/backups/secondary/privatebin (22.7 KB, 0 leg(s), 3s) +2026/11/11 02:30:15 [INFO] [backup] Tier 2 run complete: 3 app(s) processed (incl. volume-only — F6) +2026/11/11 03:00:00 [INFO] Pruned 15840 old metric rows +2026/11/11 03:15:00 [INFO] [offbox] backup run started (9 app(s) toggled) +2026/11/11 03:15:00 [INFO] [backup] Starting database dump run +2026/11/11 03:15:00 [INFO] [backup] Discovered 2 database(s): paperless-postgres(postgres), bookstack-db(mariadb) +2026/11/11 03:15:00 [INFO] [backup] DB dump: paperless-postgres → paperless-ngx-postgres.sql (560.7 KB, 457ms, 75 tables) +2026/11/11 03:15:00 [INFO] [backup] DB dump: bookstack-db → bookstack-mariadb.sql (57.9 KB, 317ms, 41 tables) +2026/11/11 03:15:00 [INFO] [backup] Stopping bookstack for safe volume dump +2026/11/11 03:15:04 [INFO] [busymark] busy: a backup, a restore or an off-site run is running — an operator reboot of the box is refused until it ends +2026/11/11 03:15:06 [INFO] [backup] Restarting bookstack after volume dump +2026/11/11 03:15:12 [INFO] [backup] Stopping paperless-ngx for safe volume dump +2026/11/11 03:15:24 [INFO] [backup] Restarting paperless-ngx after volume dump +2026/11/11 03:15:35 [INFO] [backup] Stopping privatebin for safe volume dump +2026/11/11 03:15:36 [INFO] [backup] Volume dump: privatebin/privatebin_privatebin_data → 8.5 KB +2026/11/11 03:15:36 [INFO] [backup] Restarting privatebin after volume dump +2026/11/11 03:15:37 [INFO] [backup] App-data backup completed: 2 databases (618.6 KB total), 3 volume dump(s) (37.935s) +2026/11/11 03:15:37 [INFO] [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1) +2026/11/11 03:15:37 [INFO] [backup] Recovery unit captured for paperless-ngx → /mnt/felhom-drives/adatlemez/backups/primary/paperless-ngx (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026/11/11 03:15:37 [INFO] [backup] Recovery unit captured for privatebin → /mnt/sys_drive/felhom-data/backups/primary/privatebin (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/11/11 03:15:38 [INFO] [offbox] pre-push dump leg completed in 38.077s — snapshot pair is coherent +2026/11/11 03:15:41 [WARN] [offbox] prowlarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/11/11 03:15:41 [WARN] [offbox] sonarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/11/11 03:15:41 [WARN] [offbox] transmission: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/11/11 03:15:41 [WARN] [offbox] jellyfin: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/11/11 03:15:41 [WARN] [offbox] plex: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/11/11 03:15:41 [WARN] [offbox] radarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/11/11 03:15:45 [INFO] [offbox] backed up bookstack (/mnt/sys_drive/felhom-data/backups/primary/bookstack, 0 mandatory path(s)) +2026/11/11 03:15:45 [WARN] [offbox] prowlarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/11/11 03:15:47 [INFO] [offbox] backed up prowlarr (/mnt/sys_drive/felhom-data/backups/primary/prowlarr, 0 mandatory path(s)) +2026/11/11 03:15:47 [WARN] [offbox] sonarr: no recovery unit found on any connected drive — skipping +2026/11/11 03:15:47 [WARN] [offbox] transmission: no recovery unit found on any connected drive — skipping +2026/11/11 03:15:47 [WARN] [offbox] jellyfin: no recovery unit found on any connected drive — skipping +2026/11/11 03:15:50 [INFO] [offbox] backed up paperless-ngx (/mnt/felhom-drives/adatlemez/backups/primary/paperless-ngx, 1 mandatory path(s)) +2026/11/11 03:15:50 [WARN] [offbox] plex: no recovery unit found on any connected drive — skipping +2026/11/11 03:15:52 [INFO] [offbox] backed up privatebin (/mnt/sys_drive/felhom-data/backups/primary/privatebin, 0 mandatory path(s)) +2026/11/11 03:15:52 [WARN] [offbox] radarr: no recovery unit found on any connected drive — skipping +2026/11/11 03:15:54 [INFO] [offbox] retention skipped (after-run): no clean-up window now (not due (last window 2026-11-09T03:15:56Z)) — nothing deleted (decision 68) +2026/11/11 03:15:58 [INFO] [offbox] backup OK: 4 app(s) backed up, 5 skipped (no unit), 71 snapshot(s), 55s +2026/11/11 03:15:58 [INFO] [update-leg] started (after-offsite): window 02:30, no step starts at or after 07:30 +2026/11/11 03:15:58 [INFO] [update-leg] update leg (after-offsite): done=0 undone=0 held=0 failed=0 skipped=0 in 0s [skipped: ] +2026/11/11 03:16:04 [INFO] [busymark] idle again (was: a backup, a restore or an off-site run is running) — an operator reboot of the box is no longer refused for it +2026/11/11 04:30:05 [INFO] [offbox] proof: bookstack PASSED on snapshot d4dfbe69 in 3.287s — the backup holds what this app should have +2026/11/11 04:32:09 [INFO] [quiesce] backup due on 1 tier(s) — quiescing 3 stack(s): [bookstack paperless-ngx privatebin] +2026/11/11 04:32:22 [INFO] [quiesce] tier local: backup job backup-9201-1794371542405626888 started — polling +2026/11/11 04:32:32 [INFO] [quiesce] tier local: job backup-9201-1794371542405626888 snapshotted — resuming app early (8B.2); the upload continues with the apps running +2026/11/11 04:32:32 [INFO] [quiesce] unquiescing (snapshotted (early resume)): restarting 3 stack(s) +2026/11/11 04:35:11 [INFO] [quiesce] tier local: backup job backup-9201-1794371542405626888 done +2026/11/11 05:00:00 [INFO] [offbox] integrity: not due (last successful check 48h0m0s ago, at 2026-11-09 05:00) — nothing was run +RC=0 diff --git a/documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-host-clock.txt b/documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-host-clock.txt new file mode 100644 index 00000000..8fb5400d --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-host-clock.txt @@ -0,0 +1,682 @@ +# host-clock tester-1 taken 2026-11-11T10:03:57Z from DooPlex + +=== in-flight === +checked + +=== clock === +utc 2026-11-11T10:03:57.568914293Z +local 2026-11-11T11:03:57 CET +0100 +/usr/share/zoneinfo/Europe/Budapest +Timezone=Europe/Budapest +LocalRTC=no +CanNTP=yes +NTP=yes +NTPSynchronized=yes +TimeUSec=Wed 2026-11-11 11:03:57 CET +RTCTimeUSec=Wed 2026-11-11 11:03:57 CET + Local time: Wed 2026-11-11 11:03:57 CET + Universal time: Wed 2026-11-11 10:03:57 UTC + RTC time: Wed 2026-11-11 10:03:57 + Time zone: Europe/Budapest (CET, +0100) +System clock synchronized: yes + NTP service: active + RTC in local TZ: no + +=== who keeps the time === +chrony: active + enabled: enabled +chronyd: active + enabled: alias +systemd-timesyncd: inactive + enabled: not-found +ntp: inactive + enabled: not-found +ntpsec: inactive + enabled: not-found +openntpd: inactive + enabled: not-found +Reference ID : C2255A28 (194.37.90.40) +Stratum : 2 +Ref time (UTC) : Wed Nov 11 10:00:19 2026 +System time : 0.000132173 seconds slow of NTP time +Last offset : -0.000024902 seconds +RMS offset : 0.000064157 seconds +Frequency : 12.931 ppm slow +Residual freq : +0.000 ppm +Skew : 0.013 ppm +Root delay : 0.014427286 seconds +Root dispersion : 0.000787278 seconds +Update interval : 1044.3 seconds +Leap status : Normal +MS Name/IP address Stratum Poll Reach LastRx Last sample +=============================================================================== +^+ 37.221.209.24 2 10 377 315 -1626us[-1651us] +/- 15ms +^+ 193.41.47.85 2 10 377 66 -1695us[-1695us] +/- 8973us +^+ 193.227.197.2 3 10 377 699 -4821us[-4846us] +/- 19ms +^* 194.37.90.40 1 10 377 218 -3822us[-3847us] +/- 13ms +Failed to query server: The name org.freedesktop.timesync1 was not provided by any .service files +pool 2.debian.pool.ntp.org iburst +sourcedir /run/chrony-dhcp +sourcedir /etc/chrony/sources.d +keyfile /etc/chrony/chrony.keys +driftfile /var/lib/chrony/chrony.drift +ntsdumpdir /var/lib/chrony +logdir /var/log/chrony +maxupdateskew 100.0 +rtcsync +makestep 1 3 +leapseclist /usr/share/zoneinfo/leap-seconds.list +confdir /etc/chrony/conf.d +/etc/chrony/conf.d: +README + +/etc/chrony/sources.d: +README + +=== start with no time === +ls: cannot access '/var/lib/systemd/timesync/clock': No such file or directory +/var/lib/chrony/: +total 4 +-rw-r--r-- 1 _chrony _chrony 42 2026-11-11 11:00:19.119554513 +0100 chrony.drift +2026-11-11 11:03:57.838270+01:00 +1794391438 +1794391437 + +=== boots === +2026-11-10 15:43:22 + -5 c06cfce08e2d470b9e32d4f7e2c5f94b Sat 2026-10-24 19:44:06 CEST Thu 2026-10-29 05:09:31 CET + -4 f8210297fa3f492eb5d6a1cc5844dc6b Thu 2026-10-29 05:10:44 CET Tue 2026-11-10 02:53:06 CET + -3 bf6ac6018a8e4819beed5a725e3967f2 Tue 2026-11-10 02:54:30 CET Tue 2026-11-10 13:56:23 CET + -2 a506eb94c8f54a17ba4918d0cf981676 Tue 2026-11-10 13:57:37 CET Tue 2026-11-10 15:32:20 CET + -1 9d8a5ed3f79b4d85ae816a1d308f379e Tue 2026-11-10 15:33:34 CET Tue 2026-11-10 15:42:14 CET + 0 6407d71b1d894748865549c6de8cf61b Tue 2026-11-10 15:43:29 CET Wed 2026-11-11 11:04:02 CET + +=== timers === +NEXT LEFT LAST PASSED UNIT ACTIVATES +Wed 2026-11-11 11:04:57 CET 54s Wed 2026-11-11 11:03:57 CET 5s ago felhom-mgmt-watchdog.timer felhom-mgmt-watchdog.service +Wed 2026-11-11 11:59:43 CET 55min Wed 2026-11-11 10:59:43 CET 4min 19s ago felhom-crash-guard-check.timer felhom-crash-guard-check.service +Wed 2026-11-11 15:58:30 CET 4h 54min Tue 2026-11-10 15:58:30 CET 19h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service +Thu 2026-11-12 00:00:00 CET 12h Wed 2026-11-11 00:00:02 CET 11h ago dpkg-db-backup.timer dpkg-db-backup.service +Thu 2026-11-12 00:22:35 CET 13h Wed 2026-11-11 00:19:20 CET 10h ago logrotate.timer logrotate.service +Thu 2026-11-12 00:54:30 CET 13h Wed 2026-11-11 10:38:42 CET 25min ago apt-daily.timer apt-daily.service +Thu 2026-11-12 02:57:50 CET 15h Wed 2026-11-11 08:20:42 CET 2h 43min ago man-db.timer man-db.service +Thu 2026-11-12 03:01:31 CET 15h Wed 2026-11-11 01:02:20 CET 10h ago pve-daily-update.timer pve-daily-update.service +Thu 2026-11-12 06:07:23 CET 19h Wed 2026-11-11 06:57:02 CET 4h 7min ago apt-daily-upgrade.timer apt-daily-upgrade.service +Sun 2026-11-15 03:10:10 CET 3 days Sun 2026-11-08 03:10:39 CET - e2scrub_all.timer e2scrub_all.service +Sun 2026-11-15 03:10:22 CET 3 days Sun 2026-11-08 03:11:23 CET - xfs_scrub_all.timer xfs_scrub_all.service +Mon 2026-11-16 00:54:35 CET 4 days Mon 2026-11-09 01:13:20 CET - fstrim.timer fstrim.service + +12 timers listed. + +=== timer units (all) === +--- apt-daily-upgrade.timer [enabled/active] +OnCalendar=*-*-* 6:00 +RandomizedDelaySec=60m +Persistent=true +Unit=apt-daily-upgrade.service NextElapseUSecRealtime=Thu 2026-11-12 06:07:23 CET LastTriggerUSec=Wed 2026-11-11 06:57:02 CET +--- apt-daily.timer [enabled/active] +OnCalendar=*-*-* 6,18:00 +RandomizedDelaySec=12h +Persistent=true +Unit=apt-daily.service NextElapseUSecRealtime=Thu 2026-11-12 00:54:30 CET LastTriggerUSec=Wed 2026-11-11 10:38:42 CET +--- apt-listchanges.timer [disabled/inactive] +OnCalendar=hourly +Unit=apt-listchanges.service NextElapseUSecRealtime= LastTriggerUSec= +--- chrony-dnssrv@.timer [disabled/] +OnActiveSec=0 +OnUnitInactiveSec=1h + +--- dpkg-db-backup.timer [enabled/active] +OnCalendar=daily +Persistent=true +Unit=dpkg-db-backup.service NextElapseUSecRealtime=Thu 2026-11-12 00:00:00 CET LastTriggerUSec=Wed 2026-11-11 00:00:02 CET +--- e2scrub_all.timer [enabled/active] +OnCalendar=Sun *-*-* 03:10:00 +RandomizedDelaySec=60 +Persistent=true +Unit=e2scrub_all.service NextElapseUSecRealtime=Sun 2026-11-15 03:10:10 CET LastTriggerUSec=Sun 2026-11-08 03:10:39 CET +--- felhom-crash-guard-check.timer [enabled/active] +OnBootSec=15min +OnUnitActiveSec=1h +Unit=felhom-crash-guard-check.service NextElapseUSecRealtime= LastTriggerUSec=Wed 2026-11-11 10:59:43 CET +--- felhom-mgmt-watchdog.timer [enabled/active] +OnBootSec=30s +OnUnitActiveSec=60s +AccuracySec=5s +Persistent=true +Unit=felhom-mgmt-watchdog.service +Unit=felhom-mgmt-watchdog.service NextElapseUSecRealtime= LastTriggerUSec=Wed 2026-11-11 11:03:57 CET +--- fstrim.timer [enabled/active] +OnCalendar=weekly +AccuracySec=1h +Persistent=true +RandomizedDelaySec=100min +Unit=fstrim.service NextElapseUSecRealtime=Mon 2026-11-16 00:54:35 CET LastTriggerUSec=Mon 2026-11-09 01:13:20 CET +--- logrotate.timer [enabled/active] +OnCalendar=daily +RandomizedDelaySec=1h +Persistent=true +Unit=logrotate.service NextElapseUSecRealtime=Thu 2026-11-12 00:22:35 CET LastTriggerUSec=Wed 2026-11-11 00:19:20 CET +--- man-db.timer [enabled/active] +OnCalendar=daily +RandomizedDelaySec=12h +Persistent=true +Unit=man-db.service NextElapseUSecRealtime=Thu 2026-11-12 02:57:50 CET LastTriggerUSec=Wed 2026-11-11 08:20:42 CET +--- proxmox-firewall.timer [disabled/inactive] +OnBootSec=1s +OnUnitInactiveSec=5s +Unit=proxmox-firewall.service +Unit=proxmox-firewall.service NextElapseUSecRealtime= LastTriggerUSec= +--- pve-daily-update.timer [enabled/active] +OnCalendar=*-*-* 1:00 +RandomizedDelaySec=5h +Persistent=true +Unit=pve-daily-update.service NextElapseUSecRealtime=Thu 2026-11-12 03:01:31 CET LastTriggerUSec=Wed 2026-11-11 01:02:20 CET +--- systemd-tmpfiles-clean.timer [static/active] +OnBootSec=15min +OnUnitActiveSec=1d +Unit=systemd-tmpfiles-clean.service NextElapseUSecRealtime= LastTriggerUSec=Tue 2026-11-10 15:58:30 CET +--- xfs_scrub_all.timer [enabled/active] +OnCalendar=Sun *-*-* 03:10:00 +RandomizedDelaySec=60 +Persistent=true +Unit=xfs_scrub_all.service NextElapseUSecRealtime=Sun 2026-11-15 03:10:22 CET LastTriggerUSec=Sun 2026-11-08 03:11:23 CET +--- zfs-scrub-monthly@.timer [disabled/] +OnCalendar=monthly +Persistent=true +RandomizedDelaySec=1h +Unit=zfs-scrub@%i.service + +--- zfs-scrub-weekly@.timer [disabled/] +OnCalendar=weekly +Persistent=true +RandomizedDelaySec=1h +Unit=zfs-scrub@%i.service + +--- zfs-trim-monthly@.timer [disabled/] +OnCalendar=monthly +Persistent=true +RandomizedDelaySec=1h +Unit=zfs-trim@%i.service + +--- zfs-trim-weekly@.timer [disabled/] +OnCalendar=weekly +Persistent=true +RandomizedDelaySec=1h +Unit=zfs-trim@%i.service + + +=== service limits of felhom + backup timers === +--- apt-daily-upgrade.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 06:57:02 CET +ExecMainExitTimestamp=Wed 2026-11-11 06:57:02 CET +ExecMainStatus=0 +ExecStart={ path=/usr/lib/apt/apt.systemd.daily ; argv[]=/usr/lib/apt/apt.systemd.daily install ; ignore_errors=no ; start_time=[Wed 2026-11-11 06:57:02 CET] ; stop_time=[Wed 2026-11-11 06:57:02 CET] ; pid=1262190 ; code=exited ; status=0 } +--- apt-daily.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 10:38:42 CET +ExecMainExitTimestamp=Wed 2026-11-11 10:38:42 CET +ExecMainStatus=0 +ExecStart={ path=/usr/lib/apt/apt.systemd.daily ; argv[]=/usr/lib/apt/apt.systemd.daily update ; ignore_errors=no ; start_time=[Wed 2026-11-11 10:38:42 CET] ; stop_time=[Wed 2026-11-11 10:38:42 CET] ; pid=1563704 ; code=exited ; status=0 } +--- apt-listchanges.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp= +ExecMainExitTimestamp= +ExecMainStatus=0 +ExecStart={ path=python3 ; argv[]=python3 -m apt_listchanges.populate_database --profile apt ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } +--- dpkg-db-backup.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 00:00:02 CET +ExecMainExitTimestamp=Wed 2026-11-11 00:00:02 CET +ExecMainStatus=0 +ExecStart={ path=/usr/libexec/dpkg/dpkg-db-backup ; argv[]=/usr/libexec/dpkg/dpkg-db-backup ; ignore_errors=no ; start_time=[Wed 2026-11-11 00:00:02 CET] ; stop_time=[Wed 2026-11-11 00:00:02 CET] ; pid=686273 ; code=exited ; status=0 } +--- e2scrub_all.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp= +ExecMainExitTimestamp= +ExecMainStatus=0 +ExecStart={ path=/sbin/e2scrub_all ; argv[]=/sbin/e2scrub_all ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } +--- felhom-crash-guard-check.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 10:59:43 CET +ExecMainExitTimestamp=Wed 2026-11-11 10:59:43 CET +ExecMainStatus=0 +ExecStart={ path=/usr/local/sbin/felhom-crash-guard ; argv[]=/usr/local/sbin/felhom-crash-guard check ; ignore_errors=no ; start_time=[Wed 2026-11-11 10:59:43 CET] ; stop_time=[Wed 2026-11-11 10:59:43 CET] ; pid=1593022 ; code=exited ; status=0 } +--- felhom-mgmt-watchdog.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 11:03:57 CET +ExecMainExitTimestamp=Wed 2026-11-11 11:03:57 CET +ExecMainStatus=0 +ExecStart={ path=/usr/local/sbin/felhom-mgmt-watchdog ; argv[]=/usr/local/sbin/felhom-mgmt-watchdog ; ignore_errors=no ; start_time=[Wed 2026-11-11 11:03:57 CET] ; stop_time=[Wed 2026-11-11 11:03:57 CET] ; pid=1599130 ; code=exited ; status=0 } +--- fstrim.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp= +ExecMainExitTimestamp= +ExecMainStatus=0 +ExecStart={ path=/sbin/fstrim ; argv[]=/sbin/fstrim --listed-in /etc/fstab:/proc/self/mountinfo --verbose --quiet-unsupported ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } +--- logrotate.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 00:19:20 CET +ExecMainExitTimestamp=Wed 2026-11-11 00:19:23 CET +ExecMainStatus=0 +ExecStart={ path=/usr/sbin/logrotate ; argv[]=/usr/sbin/logrotate /etc/logrotate.conf ; ignore_errors=no ; start_time=[Wed 2026-11-11 00:19:20 CET] ; stop_time=[Wed 2026-11-11 00:19:23 CET] ; pid=712662 ; code=exited ; status=0 } +--- man-db.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 08:20:42 CET +ExecMainExitTimestamp=Wed 2026-11-11 08:20:42 CET +ExecMainStatus=0 +ExecStart={ path=/usr/bin/install ; argv[]=/usr/bin/install -d -o man -g man -m 0755 /var/cache/man ; ignore_errors=no ; start_time=[Wed 2026-11-11 08:20:42 CET] ; stop_time=[Wed 2026-11-11 08:20:42 CET] ; pid=1376314 ; code=exited ; status=0 } +ExecStart={ path=/usr/bin/mandb ; argv[]=/usr/bin/mandb --quiet ; ignore_errors=no ; start_time=[Wed 2026-11-11 08:20:42 CET] ; stop_time=[Wed 2026-11-11 08:20:42 CET] ; pid=1376319 ; code=exited ; status=0 } +--- pve-daily-update.service +Type=oneshot +TimeoutStartUSec=infinity +RuntimeMaxUSec=infinity +ExecMainStartTimestamp=Wed 2026-11-11 01:02:20 CET +ExecMainExitTimestamp=Wed 2026-11-11 01:02:26 CET +ExecMainStatus=0 +ExecStart={ path=/usr/bin/pveupdate ; argv[]=/usr/bin/pveupdate ; ignore_errors=no ; start_time=[Wed 2026-11-11 01:02:20 CET] ; stop_time=[Wed 2026-11-11 01:02:26 CET] ; pid=771302 ; code=exited ; status=0 } + +=== felhom units that are not timers === +felhom-agent-rollback.service loaded inactive dead +felhom-agent.service loaded active running +felhom-crash-guard-check.service loaded inactive dead +felhom-crash-guard.service loaded active exited +felhom-mgmt-watchdog.service loaded inactive dead +felhom-oob-nft.service loaded active exited +felhom-shared-parent.service loaded active exited +felhom-sshd.service loaded active running +felhom-crash-guard-check.timer loaded active waiting +felhom-mgmt-watchdog.timer loaded active waiting +--- felhom-agent-rollback.service +Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp= +--- felhom-agent.service +Restart=on-failure RestartUSec=5s TimeoutStartUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 16:14:40 CET +--- felhom-bootstrap.service +Restart=on-failure RestartUSec=30s TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=infinity NRestarts=0 ActiveEnterTimestamp= +--- felhom-crash-guard-check.service +Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp= +--- felhom-crash-guard.service +Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:43:32 CET +--- felhom-mgmt-watchdog.service +Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp= +--- felhom-oob-nft.service +Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:43:32 CET +--- felhom-shared-parent.service +Restart=no RestartUSec=100ms TimeoutStartUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:43:32 CET +--- felhom-sshd.service +Restart=on-failure RestartUSec=100ms TimeoutStartUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 NRestarts=0 ActiveEnterTimestamp=Tue 2026-11-10 15:43:34 CET + +=== cron === +SHELL=/bin/sh +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +17 * * * * root cd / && run-parts --report /etc/cron.hourly +25 6 * * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; } +47 6 * * 7 root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.weekly; } +52 6 1 * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.monthly; } +--- /etc/cron.d/e2scrub_all +30 3 * * 0 root test -e /run/systemd/system || SERVICE_MODE=1 /usr/libexec/e2fsprogs/e2scrub_all_cron +10 3 * * * root test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r +--- /etc/cron.d/vzdump +PATH="/usr/sbin:/usr/bin:/sbin:/bin" +--- /etc/cron.d/zfsutils-linux +PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin +24 0 1-7 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/trim ]; then /usr/lib/zfs-linux/trim; fi +24 0 8-14 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs-linux/scrub; fi +--- cron.hourly: +--- cron.daily: apt-compat dpkg logrotate man-db +--- cron.weekly: man-db +--- cron.monthly: +--- user crontabs: + +=== at === +no at + +=== pve jobs === +--- vzdump.cron +PATH="/usr/sbin:/usr/bin:/sbin:/bin" +--- vzdump.conf + +=== pve tasks (start end type id status), last 14 days === +awk: line 2: function strtonum never defined +awk: line 2: function strtonum never defined + +=== runs of timer services, last 10 days (start/finish lines) === +--- apt-daily-upgrade.service +2026-11-09T06:36:19+01:00 felhom systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities... +2026-11-09T06:36:20+01:00 felhom systemd[1]: apt-daily-upgrade.service: Deactivated successfully. +2026-11-09T06:36:20+01:00 felhom systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities. +2026-11-10T06:06:30+01:00 felhom systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities... +2026-11-10T06:06:30+01:00 felhom systemd[1]: apt-daily-upgrade.service: Deactivated successfully. +2026-11-10T06:06:30+01:00 felhom systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities. +2026-11-11T06:57:02+01:00 felhom systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities... +2026-11-11T06:57:02+01:00 felhom systemd[1]: apt-daily-upgrade.service: Deactivated successfully. +2026-11-11T06:57:02+01:00 felhom systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities. +--- apt-daily.service +2026-11-10T07:10:39+01:00 felhom systemd[1]: Starting apt-daily.service - Daily apt download activities... +2026-11-10T07:10:39+01:00 felhom systemd[1]: apt-daily.service: Deactivated successfully. +2026-11-10T07:10:39+01:00 felhom systemd[1]: Finished apt-daily.service - Daily apt download activities. +2026-11-11T03:38:42+01:00 felhom systemd[1]: Starting apt-daily.service - Daily apt download activities... +2026-11-11T03:38:42+01:00 felhom systemd[1]: apt-daily.service: Deactivated successfully. +2026-11-11T03:38:42+01:00 felhom systemd[1]: Finished apt-daily.service - Daily apt download activities. +2026-11-11T10:38:42+01:00 felhom systemd[1]: Starting apt-daily.service - Daily apt download activities... +2026-11-11T10:38:42+01:00 felhom systemd[1]: apt-daily.service: Deactivated successfully. +2026-11-11T10:38:42+01:00 felhom systemd[1]: Finished apt-daily.service - Daily apt download activities. +--- felhom-crash-guard-check.service +2026-11-11T08:59:42+01:00 felhom systemd[1]: Starting felhom-crash-guard-check.service - Felhom crash guard re-arm check... +2026-11-11T08:59:43+01:00 felhom systemd[1]: felhom-crash-guard-check.service: Deactivated successfully. +2026-11-11T08:59:43+01:00 felhom systemd[1]: Finished felhom-crash-guard-check.service - Felhom crash guard re-arm check. +2026-11-11T09:59:43+01:00 felhom systemd[1]: Starting felhom-crash-guard-check.service - Felhom crash guard re-arm check... +2026-11-11T09:59:43+01:00 felhom systemd[1]: felhom-crash-guard-check.service: Deactivated successfully. +2026-11-11T09:59:43+01:00 felhom systemd[1]: Finished felhom-crash-guard-check.service - Felhom crash guard re-arm check. +2026-11-11T10:59:43+01:00 felhom systemd[1]: Starting felhom-crash-guard-check.service - Felhom crash guard re-arm check... +2026-11-11T10:59:43+01:00 felhom systemd[1]: felhom-crash-guard-check.service: Deactivated successfully. +2026-11-11T10:59:43+01:00 felhom systemd[1]: Finished felhom-crash-guard-check.service - Felhom crash guard re-arm check. +--- felhom-mgmt-watchdog.service +2026-11-11T11:01:47+01:00 felhom systemd[1]: Starting felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)... +2026-11-11T11:01:47+01:00 felhom systemd[1]: felhom-mgmt-watchdog.service: Deactivated successfully. +2026-11-11T11:01:47+01:00 felhom systemd[1]: Finished felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent). +2026-11-11T11:02:52+01:00 felhom systemd[1]: Starting felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)... +2026-11-11T11:02:52+01:00 felhom systemd[1]: felhom-mgmt-watchdog.service: Deactivated successfully. +2026-11-11T11:02:52+01:00 felhom systemd[1]: Finished felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent). +2026-11-11T11:03:57+01:00 felhom systemd[1]: Starting felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)... +2026-11-11T11:03:57+01:00 felhom systemd[1]: felhom-mgmt-watchdog.service: Deactivated successfully. +2026-11-11T11:03:57+01:00 felhom systemd[1]: Finished felhom-mgmt-watchdog.service - Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent). +--- pve-daily-update.service +2026-11-09T02:36:45+01:00 felhom systemd[1]: pve-daily-update.service: Consumed 5.122s CPU time, 232.9M memory peak. +2026-11-10T03:05:00+01:00 felhom systemd[1]: Starting pve-daily-update.service - Daily PVE download activities... +2026-11-10T03:05:06+01:00 felhom systemd[1]: pve-daily-update.service: Deactivated successfully. +2026-11-10T03:05:06+01:00 felhom systemd[1]: Finished pve-daily-update.service - Daily PVE download activities. +2026-11-10T03:05:06+01:00 felhom systemd[1]: pve-daily-update.service: Consumed 5.162s CPU time, 294M memory peak. +2026-11-11T01:02:20+01:00 felhom systemd[1]: Starting pve-daily-update.service - Daily PVE download activities... +2026-11-11T01:02:26+01:00 felhom systemd[1]: pve-daily-update.service: Deactivated successfully. +2026-11-11T01:02:26+01:00 felhom systemd[1]: Finished pve-daily-update.service - Daily PVE download activities. +2026-11-11T01:02:26+01:00 felhom systemd[1]: pve-daily-update.service: Consumed 5.193s CPU time, 292.5M memory peak. + +=== agent log: schedule words, last 3 days (tail 260) === +2026-11-10T10:54:50+01:00 felhom sudo[667389]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T11:09:50+01:00 felhom sudo[687736]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T11:24:50+01:00 felhom sudo[708124]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T11:39:50+01:00 felhom sudo[728450]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T11:54:50+01:00 felhom sudo[748825]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T12:09:50+01:00 felhom sudo[769204]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T12:24:50+01:00 felhom sudo[789601]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T12:39:50+01:00 felhom sudo[809946]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T12:54:50+01:00 felhom sudo[830357]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T13:09:50+01:00 felhom sudo[850689]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T13:24:50+01:00 felhom sudo[870998]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T13:39:50+01:00 felhom sudo[891351]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T13:54:31+01:00 felhom systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)... +2026-11-10T13:54:31+01:00 felhom felhom-agent[1160]: time=2026-11-10T13:54:31.299+01:00 level=INFO msg="pbs: verify loop shutting down" reason="context canceled" +2026-11-10T13:54:31+01:00 felhom felhom-agent[1160]: time=2026-11-10T13:54:31.299+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled" +2026-11-10T13:54:31+01:00 felhom systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T13:57:49+01:00 felhom systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T13:57:49+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:57:49.796+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true +2026-11-10T13:57:49+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:57:49.796+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false +2026-11-10T13:57:49+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:57:49.798+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)" +2026-11-10T13:57:49+01:00 felhom sudo[1173]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T125749Z-kernel-kernel-boot.json +2026-11-10T13:57:51+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:57:51.810+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s +2026-11-10T13:57:51+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:57:51.818+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s +2026-11-10T13:57:51+01:00 felhom sudo[1338]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T13:57:52+01:00 felhom sudo[1381]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T13:57:53+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:57:53.108+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s +2026-11-10T13:57:53+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:57:53.109+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gues +2026-11-10T13:57:53+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:57:53.109+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or resto +2026-11-10T13:57:53+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:57:53.113+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never besid +2026-11-10T13:58:10+01:00 felhom felhom-agent[1159]: time=2026-11-10T13:58:10.187+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-11-10T14:12:56+01:00 felhom sudo[24807]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T14:27:51+01:00 felhom felhom-agent[1159]: time=2026-11-10T14:27:51.819+01:00 level=INFO msg="backup: restore-test first evaluation after start (R-874)" after=30m0s +2026-11-10T14:27:52+01:00 felhom felhom-agent[1159]: time=2026-11-10T14:27:52.173+01:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=local volid=local:backup/vzdump-lxc-9100-2026_10_04-21_40_45.tar.zst size_bytes= +2026-11-10T14:27:52+01:00 felhom felhom-agent[1159]: time=2026-11-10T14:27:52.548+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-08T20:14:53Z) is already proven; local: newest +2026-11-10T14:27:56+01:00 felhom sudo[45165]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T14:42:56+01:00 felhom sudo[65482]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T14:57:56+01:00 felhom sudo[85853]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:12:56+01:00 felhom sudo[106277]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:27:56+01:00 felhom sudo[126638]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:30:29+01:00 felhom felhom-agent[1159]: time=2026-11-10T15:30:29.267+01:00 level=INFO msg="pbs: verify loop shutting down" reason="context canceled" +2026-11-10T15:30:29+01:00 felhom felhom-agent[1159]: time=2026-11-10T15:30:29.267+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled" +2026-11-10T15:30:29+01:00 felhom systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)... +2026-11-10T15:30:29+01:00 felhom systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T15:33:47+01:00 felhom systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T15:33:47+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:33:47.796+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true +2026-11-10T15:33:47+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:33:47.796+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false +2026-11-10T15:33:47+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:33:47.798+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)" +2026-11-10T15:33:47+01:00 felhom sudo[1171]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T143347Z-kernel-kernel-boot.json +2026-11-10T15:33:50+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:33:50.028+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s +2026-11-10T15:33:50+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:33:50.031+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s +2026-11-10T15:33:50+01:00 felhom sudo[1337]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:33:51+01:00 felhom sudo[1391]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:33:51+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:33:51.499+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s +2026-11-10T15:33:51+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:33:51.500+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gues +2026-11-10T15:33:51+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:33:51.503+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or resto +2026-11-10T15:33:51+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:33:51.505+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never besid +2026-11-10T15:34:08+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:34:08.409+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-11-10T15:40:13+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:40:13.600+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled" +2026-11-10T15:40:13+01:00 felhom felhom-agent[1157]: time=2026-11-10T15:40:13.604+01:00 level=INFO msg="pbs: verify loop shutting down" reason="context canceled" +2026-11-10T15:40:13+01:00 felhom systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)... +2026-11-10T15:40:13+01:00 felhom systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T15:43:42+01:00 felhom systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T15:43:43+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:43.037+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true +2026-11-10T15:43:43+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:43.037+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false +2026-11-10T15:43:43+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:43.039+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)" +2026-11-10T15:43:43+01:00 felhom sudo[1170]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T144343Z-kernel-kernel-boot.json +2026-11-10T15:43:45+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:45.438+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s +2026-11-10T15:43:45+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:45.441+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s +2026-11-10T15:43:45+01:00 felhom sudo[1334]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:43:46+01:00 felhom sudo[1381]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.727+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gues +2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.728+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s +2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.728+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or resto +2026-11-10T15:43:46+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:43:46.732+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never besid +2026-11-10T15:44:05+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:44:05.780+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-11-10T15:59:09+01:00 felhom sudo[27166]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:59:15+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:59:15.660+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.173.0 report="{\"agent_update\": {\"sha256\": \"752b2727fb6043417a +2026-11-10T15:59:17+01:00 felhom systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)... +2026-11-10T15:59:17+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:59:17.726+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled" +2026-11-10T15:59:17+01:00 felhom felhom-agent[1158]: time=2026-11-10T15:59:17.726+01:00 level=INFO msg="pbs: verify loop shutting down" reason="context canceled" +2026-11-10T15:59:17+01:00 felhom systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T15:59:17+01:00 felhom systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T15:59:17+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:17.769+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true +2026-11-10T15:59:17+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:17.769+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false +2026-11-10T15:59:17+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:17.770+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)" +2026-11-10T15:59:17+01:00 felhom sudo[28278]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T145917Z-kernel-kernel-boot.json +2026-11-10T15:59:18+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:18.689+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s +2026-11-10T15:59:18+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:18.689+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s +2026-11-10T15:59:18+01:00 felhom sudo[28456]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:59:19+01:00 felhom sudo[28533]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.338+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gue +2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.338+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s +2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.339+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or rest +2026-11-10T15:59:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:19.339+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never besi +2026-11-10T15:59:36+01:00 felhom felhom-agent[28267]: time=2026-11-10T15:59:36.894+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.391+01:00 level=INFO msg="osupdate: running the bundle policy pass after the self-update commit (R-1285)" agent_version=0.173.0 +2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.593+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE POLICY DONE install_mode=appliance removed=0" +2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.593+01:00 level=INFO msg="osupdate: bundle policy pass done" bundle_policy="{\"install_mode\": \"appliance\", \"removed\": []}" +2026-11-10T16:00:19+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:00:19.607+01:00 level=INFO msg="selfupdate prune: the self-update folder holds what it needs" kept=felhom-agent-0.173.0 removed=33 +2026-11-10T16:14:25+01:00 felhom sudo[50673]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T16:14:31+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:14:31.582+01:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.172.1 report="{\"agent_update\": {\"sha256\": \"505a0594a2fb117f3 +2026-11-10T16:14:33+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:14:33.688+01:00 level=INFO msg="backup: restore-test scheduler shutting down" reason="context canceled" +2026-11-10T16:14:33+01:00 felhom systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)... +2026-11-10T16:14:33+01:00 felhom felhom-agent[28267]: time=2026-11-10T16:14:33.689+01:00 level=INFO msg="pbs: verify loop shutting down" reason="context canceled" +2026-11-10T16:14:33+01:00 felhom systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T16:14:33+01:00 felhom systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T16:14:33+01:00 felhom sudo[51129]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T151433Z-kernel-kernel-boot.json +2026-11-10T16:14:33+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:33.737+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true +2026-11-10T16:14:33+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:33.737+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false +2026-11-10T16:14:33+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:33.737+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)" +2026-11-10T16:14:34+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:34.663+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s +2026-11-10T16:14:34+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:34.663+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s +2026-11-10T16:14:34+01:00 felhom sudo[51328]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T16:14:35+01:00 felhom sudo[51387]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.319+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gue +2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.319+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s +2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.320+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or rest +2026-11-10T16:14:35+01:00 felhom felhom-agent[51118]: time=2026-11-10T16:14:35.321+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never besi +2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: runtime.schedule() +2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/osupdate.(*Leg).SendUnsentLoop(0x1e342d01db80, {0xe5d130, 0x1e342d05aac0}, 0x45d964b800, 0x1e342d112060) +2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/osupdate/unsent.go:76 +0xd2 fp=0x1e342cda3fa8 sp=0x1e342cda3f38 pc=0x86b292 +2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbs.(*Client).WaitVerify(0x1e342d05bdc0, {0xe5d130, 0x1e342d05aac0}, {0x1e342cf63800, 0x77}, 0x77359400, 0x1a3185c5000) +2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbs/client.go:192 +0x228 fp=0x1e342cf0fa00 sp=0x1e342cf0f8e8 pc=0x86eec8 +2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbs.(*VerifyLoop).verifyOne(0x1e342d132320, {0xe5d130, 0x1e342d05aac0}, {{0x1e342cedb5d0, 0xe}, 0x1e342d05bdc0, {0x1e342cedb5a0, 0xa}}) +2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbs/verify.go:107 +0x1c5 fp=0x1e342cf0fe18 sp=0x1e342cf0fa00 pc=0x872405 +2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbs.(*VerifyLoop).tick(0x1e342d132320, {0xe5d130, 0x1e342d05aac0}) +2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbs/verify.go:98 +0xf5 fp=0x1e342cf0fea0 sp=0x1e342cf0fe18 pc=0x8721f5 +2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbs.(*VerifyLoop).Run(0x1e342d132320, {0xe5d130, 0x1e342d05aac0}) +2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbs/verify.go:71 +0x114 fp=0x1e342cf0ffa0 sp=0x1e342cf0fea0 pc=0x871eb4 +2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/backup.(*Scheduler).Run(0x1e342d168580, {0xe5d130, 0x1e342d05aac0}) +2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/backup/schedule.go:158 +0x21f fp=0x1e342cedffa0 sp=0x1e342cedfdf0 pc=0x7b143f +2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbsdr.(*Loop).Run(0x1e342d149640, {0xe5d130, 0x1e342d05aac0}) +2026-11-10T16:14:40+01:00 felhom felhom-agent[51118]: gitea.dooplex.hu/admin/felhom-agent/internal/pbsdr/loop.go:68 +0x157 fp=0x1e342cda0fa0 sp=0x1e342cda0ef0 pc=0x8735b7 +2026-11-10T16:14:40+01:00 felhom systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T16:14:40+01:00 felhom systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-10T16:14:40+01:00 felhom sudo[51951]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T151440Z-kernel-kernel-boot.json +2026-11-10T16:14:40+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:40.589+01:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true +2026-11-10T16:14:40+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:40.589+01:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false +2026-11-10T16:14:40+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:40.589+01:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)" +2026-11-10T16:14:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:41.489+01:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s +2026-11-10T16:14:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:41.489+01:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s +2026-11-10T16:14:41+01:00 felhom sudo[52108]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T16:14:42+01:00 felhom sudo[52161]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.172+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s gue +2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.172+01:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s +2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.173+01:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or rest +2026-11-10T16:14:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:42.173+01:00 level=INFO msg="linetest: weekly upload line test starting" schedule="weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never besi +2026-11-10T16:14:45+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:14:45.707+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-11-10T16:29:17+01:00 felhom sudo[72801]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T16:29:47+01:00 felhom sudo[73744]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T16:44:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:44:41.490+01:00 level=INFO msg="backup: restore-test first evaluation after start (R-874)" after=30m0s +2026-11-10T16:44:41+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:44:41.820+01:00 level=INFO msg="backup: restore-test skips an entry that is not a backup of a guest" target=local volid=local:backup/vzdump-lxc-9100-2026_10_04-21_40_45.tar.zst size_bytes +2026-11-10T16:44:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T16:44:42.189+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-08T20:14:53Z) is already proven; local: newest +2026-11-10T16:44:47+01:00 felhom sudo[94752]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T16:59:47+01:00 felhom sudo[115085]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T17:14:47+01:00 felhom sudo[135881]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T17:29:47+01:00 felhom sudo[156253]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T17:44:47+01:00 felhom sudo[176638]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T17:59:47+01:00 felhom sudo[196955]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T18:14:47+01:00 felhom sudo[217335]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T18:29:47+01:00 felhom sudo[237706]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T18:44:47+01:00 felhom sudo[258098]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T18:59:47+01:00 felhom sudo[278436]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T19:14:47+01:00 felhom sudo[298822]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T19:29:47+01:00 felhom sudo[319218]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T19:44:47+01:00 felhom sudo[339601]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T19:59:47+01:00 felhom sudo[360021]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T20:14:47+01:00 felhom sudo[380361]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T20:29:47+01:00 felhom sudo[400764]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T20:44:47+01:00 felhom sudo[421042]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T20:59:47+01:00 felhom sudo[441405]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T21:14:47+01:00 felhom sudo[461752]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T21:29:47+01:00 felhom sudo[482106]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T21:44:47+01:00 felhom sudo[502460]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T21:59:47+01:00 felhom sudo[522716]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T22:14:45+01:00 felhom sudo[543063]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T22:14:47+01:00 felhom sudo[543124]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T22:15:03+01:00 felhom felhom-agent[51939]: time=2026-11-10T22:15:03.892+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-11-10T22:29:47+01:00 felhom sudo[563495]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T22:44:42+01:00 felhom felhom-agent[51939]: time=2026-11-10T22:44:42.974+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-08T20:14:53Z) is already proven; local: newest +2026-11-10T22:44:47+01:00 felhom sudo[583859]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T22:59:47+01:00 felhom sudo[604236]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T23:14:47+01:00 felhom sudo[624634]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T23:29:47+01:00 felhom sudo[645036]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T23:44:47+01:00 felhom sudo[665416]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-10T23:59:47+01:00 felhom sudo[685773]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T00:14:47+01:00 felhom sudo[706194]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T00:29:47+01:00 felhom sudo[726604]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T00:44:47+01:00 felhom sudo[746951]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T00:59:47+01:00 felhom sudo[767304]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T01:14:47+01:00 felhom sudo[787701]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T01:29:47+01:00 felhom sudo[808115]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T01:44:47+01:00 felhom sudo[828434]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T01:59:47+01:00 felhom sudo[848773]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T02:14:47+01:00 felhom sudo[869104]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T02:29:47+01:00 felhom sudo[889486]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T02:44:47+01:00 felhom sudo[912524]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T02:59:47+01:00 felhom sudo[932907]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T03:14:47+01:00 felhom sudo[953298]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T03:29:47+01:00 felhom sudo[973698]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T03:44:47+01:00 felhom sudo[994598]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T03:59:47+01:00 felhom sudo[1014973]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T04:14:45+01:00 felhom sudo[1035243]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T04:14:47+01:00 felhom sudo[1035366]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T04:15:05+01:00 felhom felhom-agent[51939]: time=2026-11-11T04:15:05.895+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-11-11T04:29:47+01:00 felhom sudo[1058436]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T04:44:42+01:00 felhom felhom-agent[51939]: time=2026-11-11T04:44:42.969+01:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-11-08T20:14:53Z) is already proven; local: newest +2026-11-11T04:44:47+01:00 felhom sudo[1078775]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T04:59:47+01:00 felhom sudo[1099130]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T05:14:47+01:00 felhom sudo[1119503]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T05:29:47+01:00 felhom sudo[1139900]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T05:32:22+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:32:22.829+01:00 level=INFO msg="backup: space preflight passed" vmid=9201 target=local last_archive_bytes=2839981414 need_bytes=4623718591 avail_bytes=38510018560 +2026-11-11T05:32:23+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:32:23.882+01:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=local job=backup-9201-1794371542405626888 +2026-11-11T05:34:42+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:34:42.173+01:00 level=INFO msg="janitor: stale-lock sweep deferred — a heavy operation is in flight" busy=backup:local +2026-11-11T05:35:09+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:35:09.450+01:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_11_11-05_32_22.tar.zst size_bytes=2841262471 uncovered_volumes=2 +2026-11-11T05:35:09+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:35:09.450+01:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1794371542405626888 archive=local:backup/vzdump-lxc-9201-2026_11_11-05_32_22.tar.zst +2026-11-11T05:36:39+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:36:39.454+01:00 level=INFO msg="osupdate: START" run=20261111T043639Z layer=guest vmid=9201 ring=1 trigger=night enabled=true release=os-guest-20261111-034654 +2026-11-11T05:37:10+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:10.254+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=os-guest-20261111-034654 layer=guest:9201 lane=fast mode=apply select=listed packages=272" +2026-11-11T05:37:10+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:10.254+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-11T05:37:10+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:10.254+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=2 already=270 not-installed=0 from-snapshot=0" +2026-11-11T05:37:10+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:10.254+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=4.1 upgraded=2 restart-needed=- reboot-needed=no" +2026-11-11T05:37:10+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:10.255+01:00 level=INFO msg="osupdate: DONE" run=20261111T043639Z layer=guest vmid=9201 ring=1 trigger=night outcome=applied healthy=true reason="" upgraded=2 pending=6 not_covered=6 r +2026-11-11T05:37:10+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:10.334+01:00 level=INFO msg="osupdate: START" run=20261111T043639Z layer=host vmid=9201 ring=1 trigger=night enabled=true release=os-host-20261111-034654 +2026-11-11T05:37:32+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:32.686+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=os-host-20261111-034654 layer=host lane=fast mode=apply select=listed packages=607" +2026-11-11T05:37:32+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:32.686+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-11T05:37:32+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:32.686+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=2 already=603 not-installed=2 from-snapshot=0" +2026-11-11T05:37:32+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:32.686+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=3.9 upgraded=2 restart-needed=- reboot-needed=no" +2026-11-11T05:37:33+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:33.710+01:00 level=INFO msg="osupdate: DONE" run=20261111T043639Z layer=host vmid=9201 ring=1 trigger=night outcome=applied healthy=true reason="" upgraded=2 pending=100 not_covered=10 +2026-11-11T05:37:33+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:33.729+01:00 level=INFO msg="osupdate: docker step skipped — ring 1 takes an engine set only inside a signed operator job (`11` §5.8)" run=20261111T043639Z vmid=9201 trigger=night r +2026-11-11T05:37:33+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:33.729+01:00 level=INFO msg="osupdate: pve step skipped — ring 1 takes a Proxmox set only inside a signed operator job (`11` §5.10)" run=20261111T043639Z vmid=9201 trigger=night rin +2026-11-11T05:37:33+01:00 felhom felhom-agent[51939]: time=2026-11-11T05:37:33.729+01:00 level=INFO msg="osupdate: kernel step skipped — the hub names no kernel step for this box" run=20261111T043639Z layer=kernel vmid=9201 trigger=night ring=1 +2026-11-11T05:44:47+01:00 felhom sudo[1164292]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T05:59:47+01:00 felhom sudo[1184673]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T06:14:47+01:00 felhom sudo[1205082]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T06:29:47+01:00 felhom sudo[1225487]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T06:44:47+01:00 felhom sudo[1245747]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T06:59:47+01:00 felhom sudo[1266207]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T07:14:47+01:00 felhom sudo[1286559]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T07:29:47+01:00 felhom sudo[1306966]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T07:44:47+01:00 felhom sudo[1327307]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T07:59:47+01:00 felhom sudo[1347776]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T08:14:47+01:00 felhom sudo[1368158]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T08:29:47+01:00 felhom sudo[1388479]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T08:44:47+01:00 felhom sudo[1408823]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T08:59:47+01:00 felhom sudo[1429228]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T09:14:47+01:00 felhom sudo[1449547]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T09:29:47+01:00 felhom sudo[1469898]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T09:44:47+01:00 felhom sudo[1490278]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T09:59:47+01:00 felhom sudo[1510612]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T10:14:45+01:00 felhom sudo[1530915]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T10:14:47+01:00 felhom sudo[1530998]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T10:15:05+01:00 felhom felhom-agent[51939]: time=2026-11-11T10:15:05.894+01:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-11-11T10:29:47+01:00 felhom sudo[1551423]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T10:44:42+01:00 felhom felhom-agent[51939]: time=2026-11-11T10:44:42.639+01:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=local archive=local:backup/vzdump-lxc-9201-2026_11_10-05_30_28. +2026-11-11T10:44:43+01:00 felhom felhom-agent[51939]: time=2026-11-11T10:44:43.578+01:00 level=INFO msg="restore-test: space preflight passed" storage=local-lvm required_bytes=17155162112 avail_bytes=109126369674 +2026-11-11T10:44:43+01:00 felhom felhom-agent[51939]: time=2026-11-11T10:44:43.624+01:00 level=INFO msg="restore-test: full-fidelity restore params derived from the archive config" scratch=990000 params=4 +2026-11-11T10:44:47+01:00 felhom sudo[1571971]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +2026-11-11T10:45:42+01:00 felhom felhom-agent[51939]: time=2026-11-11T10:45:42.474+01:00 level=INFO msg="restore-test: scratch guest torn down" vmid=990000 +2026-11-11T10:45:42+01:00 felhom felhom-agent[51939]: time=2026-11-11T10:45:42.475+01:00 level=INFO msg="backup: scheduled restore-test passed" archive=local:backup/vzdump-lxc-9201-2026_11_10-05_30_28.tar.zst duration_s=59.829316696 +2026-11-11T10:59:47+01:00 felhom sudo[1593180]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage + +=== agent log: line count per hour, last 26 h === + 10116 2026-11-10T09 10861 2026-11-10T10 10862 2026-11-10T11 10865 2026-11-10T12 10301 2026-11-10T13 10865 2026-11-10T14 9888 2026-11-10T15 11980 2026-11-10T16 10869 2026-11-10T17 10862 2026-11-10T18 10862 2026-11-10T19 10850 2026-11-10T20 10858 2026-11-10T21 10845 2026-11-10T22 10846 2026-11-10T23 10861 2026-11-11T00 10858 2026-11-11T01 10863 2026-11-11T02 10903 2026-11-11T03 10870 2026-11-11T04 10921 2026-11-11T05 10861 2026-11-11T06 10861 2026-11-11T07 10861 2026-11-11T08 10861 2026-11-11T09 10880 2026-11-11T10 784 2026-11-11T11 +=== agent version === +felhom-agent 0.173.0 +RC=0 diff --git a/documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-host-night.txt b/documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-host-night.txt new file mode 100644 index 00000000..c760a051 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-host-night.txt @@ -0,0 +1,156 @@ +# host-night tester-1 taken 2026-11-11T10:09:26Z +=== pve tasks, last 14 days (local time) === +2026-11-03 02:01:41 -> 02:01:41 0.0 min push_file 9201 OK +2026-11-03 02:03:15 -> 02:03:15 0.0 min push_file 9201 OK +2026-11-03 02:03:33 -> 02:03:33 0.0 min push_file 9201 OK +2026-11-03 02:03:52 -> 02:03:52 0.0 min push_file 9201 OK +2026-11-03 02:04:11 -> 02:04:11 0.0 min push_file 9201 OK +2026-11-03 02:04:30 -> 02:04:30 0.0 min push_file 9201 OK +2026-11-03 02:04:49 -> 02:04:49 0.0 min push_file 9201 OK +2026-11-03 02:05:08 -> 02:05:08 0.0 min push_file 9201 OK +2026-11-03 02:05:27 -> 02:05:27 0.0 min push_file 9201 OK +2026-11-03 02:05:46 -> 02:05:46 0.0 min push_file 9201 OK +2026-11-03 02:06:06 -> 02:06:06 0.0 min push_file 9201 OK +2026-11-03 02:06:25 -> 02:06:25 0.0 min push_file 9201 OK +2026-11-03 02:06:45 -> 02:06:45 0.0 min push_file 9201 OK +2026-11-03 02:07:05 -> 02:07:05 0.0 min push_file 9201 OK +2026-11-03 02:07:25 -> 02:07:25 0.0 min push_file 9201 OK +2026-11-03 02:07:45 -> 02:07:45 0.0 min push_file 9201 OK +2026-11-03 02:08:05 -> 02:08:05 0.0 min push_file 9201 OK +2026-11-03 02:08:24 -> 02:08:24 0.0 min push_file 9201 OK +2026-11-03 02:08:43 -> 02:08:43 0.0 min push_file 9201 OK +2026-11-03 02:09:02 -> 02:09:02 0.0 min push_file 9201 OK +2026-11-03 02:09:22 -> 02:09:22 0.0 min push_file 9201 OK +2026-11-03 02:09:41 -> 02:09:41 0.0 min push_file 9201 OK +2026-11-03 02:10:01 -> 02:10:01 0.0 min push_file 9201 OK +2026-11-03 02:10:20 -> 02:10:20 0.0 min push_file 9201 OK +2026-11-03 02:10:39 -> 02:10:40 0.0 min push_file 9201 OK +2026-11-03 02:10:59 -> 02:10:59 0.0 min push_file 9201 OK +2026-11-03 02:11:18 -> 02:11:18 0.0 min push_file 9201 OK +2026-11-03 02:11:37 -> 02:11:37 0.0 min push_file 9201 OK +2026-11-03 02:11:56 -> 02:11:56 0.0 min push_file 9201 OK +2026-11-03 02:12:15 -> 02:12:15 0.0 min push_file 9201 OK +2026-11-03 02:12:34 -> 02:12:34 0.0 min push_file 9201 OK +2026-11-03 02:12:53 -> 02:12:53 0.0 min push_file 9201 OK +2026-11-03 02:13:12 -> 02:13:12 0.0 min push_file 9201 OK +2026-11-03 02:13:31 -> 02:13:31 0.0 min push_file 9201 OK +2026-11-03 02:13:50 -> 02:13:50 0.0 min push_file 9201 OK +2026-11-03 02:14:09 -> 02:14:09 0.0 min push_file 9201 OK +2026-11-03 02:14:28 -> 02:14:28 0.0 min push_file 9201 OK +2026-11-03 02:14:47 -> 02:14:47 0.0 min push_file 9201 OK +2026-11-03 02:15:06 -> 02:15:06 0.0 min push_file 9201 OK +2026-11-03 02:15:25 -> 02:15:25 0.0 min push_file 9201 OK +2026-11-03 02:15:44 -> 02:15:44 0.0 min push_file 9201 OK +2026-11-03 02:16:03 -> 02:16:03 0.0 min push_file 9201 OK +2026-11-03 02:16:22 -> 02:16:22 0.0 min push_file 9201 OK +2026-11-03 02:16:42 -> 02:16:42 0.0 min push_file 9201 OK +2026-11-03 02:17:01 -> 02:17:01 0.0 min push_file 9201 OK +2026-11-03 02:17:22 -> 02:17:22 0.0 min push_file 9201 OK +2026-11-03 02:17:41 -> 02:17:41 0.0 min push_file 9201 OK +2026-11-03 02:18:01 -> 02:18:01 0.0 min push_file 9201 OK +2026-11-03 02:18:20 -> 02:18:20 0.0 min push_file 9201 OK +2026-11-03 02:18:40 -> 02:18:40 0.0 min push_file 9201 OK +2026-11-03 02:18:58 -> 02:18:58 0.0 min push_file 9201 OK +2026-11-03 02:19:17 -> 02:19:17 0.0 min push_file 9201 OK +2026-11-03 02:19:24 -> 02:19:24 0.0 min push_file 9201 OK +2026-11-03 02:19:41 -> 02:19:41 0.0 min push_file 9201 OK +2026-11-03 02:20:18 -> 02:20:18 0.0 min push_file 9201 OK +2026-11-03 02:22:39 -> 02:22:39 0.0 min push_file 9201 OK +2026-11-03 02:32:43 -> 02:32:43 0.0 min push_file 9201 OK +2026-11-03 02:33:06 -> 02:33:06 0.0 min push_file 9201 OK +2026-11-03 02:33:25 -> 02:33:25 0.0 min push_file 9201 OK +2026-11-03 02:33:44 -> 02:33:44 0.0 min push_file 9201 OK +2026-11-03 02:34:02 -> 02:34:02 0.0 min push_file 9201 OK +2026-11-03 02:39:06 -> 02:39:06 0.0 min push_file 9201 OK +2026-11-03 02:43:03 -> 02:43:03 0.0 min push_file 9201 OK +2026-11-03 02:43:58 -> 02:43:58 0.0 min push_file 9201 OK +2026-11-03 02:46:39 -> 02:46:39 0.0 min push_file 9201 OK +2026-11-03 02:56:23 -> 02:56:23 0.0 min push_file 9201 OK +2026-11-03 02:56:43 -> 02:56:43 0.0 min push_file 9201 OK +2026-11-03 02:56:55 -> 02:57:00 0.1 min aptupdate OK +2026-11-03 02:57:02 -> 02:57:02 0.0 min push_file 9201 OK +2026-11-03 02:57:20 -> 02:57:20 0.0 min push_file 9201 OK +2026-11-03 02:57:39 -> 02:57:39 0.0 min push_file 9201 OK +2026-11-03 03:02:43 -> 03:02:43 0.0 min push_file 9201 OK +2026-11-03 05:01:41 -> 05:04:29 2.8 min vzdump 9201 OK +2026-11-03 08:15:47 -> 08:15:47 0.0 min push_file 9201 OK +2026-11-03 08:16:14 -> 08:16:14 0.0 min push_file 9201 OK +2026-11-03 08:16:17 -> 08:16:17 0.0 min push_file 9201 OK +2026-11-03 08:16:20 -> 08:16:20 0.0 min push_file 9201 OK +2026-11-03 08:16:23 -> 08:16:23 0.0 min push_file 9201 OK +2026-11-03 08:16:26 -> 08:16:26 0.0 min push_file 9201 OK +2026-11-03 08:17:33 -> 08:17:33 0.0 min push_file 9201 OK +2026-11-03 08:17:47 -> 08:17:47 0.0 min push_file 9201 OK +2026-11-03 08:17:50 -> 08:17:50 0.0 min push_file 9201 OK +2026-11-03 08:17:53 -> 08:17:53 0.0 min push_file 9201 OK +2026-11-03 08:17:56 -> 08:17:56 0.0 min push_file 9201 OK +2026-11-03 08:17:59 -> 08:17:59 0.0 min push_file 9201 OK +2026-11-03 08:18:02 -> 08:18:02 0.0 min push_file 9201 OK +2026-11-03 09:07:03 -> 09:07:03 0.0 min push_file 9201 OK +2026-11-03 09:27:35 -> 09:28:38 1.1 min vzrestore 990000 OK +2026-11-03 09:28:41 -> 09:28:45 0.1 min vzstart 990000 OK +2026-11-03 09:28:48 -> 09:28:50 0.0 min vzdestroy 990000 OK +2026-11-04 03:37:26 -> 03:37:30 0.1 min aptupdate OK +2026-11-04 05:04:55 -> 05:08:18 3.4 min vzdump 9201 OK +2026-11-04 09:28:55 -> 09:30:14 1.3 min vzrestore 990000 OK +2026-11-04 09:30:17 -> 09:30:20 0.1 min vzstart 990000 OK +2026-11-04 09:30:23 -> 09:30:26 0.1 min vzdestroy 990000 OK +2026-11-05 02:14:40 -> 02:14:45 0.1 min aptupdate OK +2026-11-05 05:09:55 -> 05:13:24 3.5 min vzdump 9201 OK +2026-11-05 09:28:55 -> 09:30:22 1.4 min vzrestore 990000 OK +2026-11-05 09:30:27 -> 09:30:30 0.1 min vzstart 990000 OK +2026-11-05 09:30:33 -> 09:30:36 0.1 min vzdestroy 990000 OK +2026-11-06 03:30:07 -> 03:30:13 0.1 min aptupdate OK +2026-11-06 05:14:55 -> 05:18:34 3.6 min vzdump 9201 OK +2026-11-06 09:28:55 -> 09:30:31 1.6 min vzrestore 990000 OK +2026-11-06 09:30:32 -> 09:30:35 0.1 min vzstart 990000 OK +2026-11-06 09:30:38 -> 09:30:41 0.1 min vzdestroy 990000 OK +2026-11-07 03:56:36 -> 03:56:42 0.1 min aptupdate OK +2026-11-07 05:19:54 -> 05:23:27 3.5 min vzdump 9201 OK +2026-11-07 09:28:55 -> 09:30:27 1.5 min vzrestore 990000 OK +2026-11-07 09:30:32 -> 09:30:36 0.1 min vzstart 990000 OK +2026-11-07 09:30:38 -> 09:30:41 0.1 min vzdestroy 990000 OK +2026-11-08 03:01:24 -> 03:01:29 0.1 min aptupdate OK +2026-11-08 05:24:54 -> 05:28:25 3.5 min vzdump 9201 OK +2026-11-08 05:42:51 -> 05:44:12 1.4 min vzrestore 990000 OK +2026-11-08 05:44:17 -> 05:44:20 0.1 min vzstart 990000 OK +2026-11-08 05:44:23 -> 05:44:26 0.1 min vzdestroy 990000 OK +2026-11-08 21:14:53 -> 21:17:13 2.3 min vzdump 9201 OK +2026-11-09 02:36:40 -> 02:36:45 0.1 min aptupdate OK +2026-11-09 05:29:55 -> 05:33:45 3.8 min vzdump 9201 OK +2026-11-09 05:42:51 -> 05:44:30 1.6 min vzrestore 990000 OK +2026-11-09 05:44:32 -> 05:44:36 0.1 min vzstart 990000 OK +2026-11-09 05:44:38 -> 05:44:41 0.1 min vzdestroy 990000 OK +2026-11-09 23:42:51 -> 23:46:31 3.7 min vzrestore 990000 OK +2026-11-09 23:46:33 -> 23:46:37 0.1 min vzstart 990000 OK +2026-11-09 23:46:39 -> 23:46:42 0.1 min vzdestroy 990000 OK +2026-11-10 02:51:05 -> 02:52:46 1.7 min stopall OK +2026-11-10 02:51:05 -> 02:52:46 1.7 min vzshutdown 9201 OK +2026-11-10 02:54:48 -> 02:54:58 0.2 min startall OK +2026-11-10 02:54:48 -> 02:54:58 0.2 min vzstart 9201 OK +2026-11-10 03:05:01 -> 03:05:06 0.1 min aptupdate OK +2026-11-10 05:30:28 -> 05:33:15 2.8 min vzdump 9201 OK +2026-11-10 09:24:47 -> 09:25:45 1.0 min vzrestore 990000 OK +2026-11-10 09:25:49 -> 09:25:52 0.1 min vzstart 990000 OK +2026-11-10 09:25:55 -> 09:25:58 0.1 min vzdestroy 990000 OK +2026-11-10 13:54:35 -> 13:56:15 1.7 min stopall OK +2026-11-10 13:54:35 -> 13:56:15 1.7 min vzshutdown 9201 OK +2026-11-10 13:57:55 -> 13:58:03 0.1 min startall OK +2026-11-10 13:57:55 -> 13:58:03 0.1 min vzstart 9201 OK +2026-11-10 15:30:33 -> 15:32:12 1.6 min stopall OK +2026-11-10 15:30:33 -> 15:32:12 1.6 min vzshutdown 9201 OK +2026-11-10 15:33:53 -> 15:34:01 0.1 min startall OK +2026-11-10 15:33:53 -> 15:34:01 0.1 min vzstart 9201 OK +2026-11-10 15:40:17 -> 15:41:56 1.6 min stopall OK +2026-11-10 15:40:17 -> 15:41:56 1.6 min vzshutdown 9201 OK +2026-11-10 15:43:48 -> 15:43:56 0.1 min startall OK +2026-11-10 15:43:48 -> 15:43:56 0.1 min vzstart 9201 OK +2026-11-11 01:02:22 -> 01:02:26 0.1 min aptupdate OK +2026-11-11 05:32:22 -> 05:35:09 2.8 min vzdump 9201 OK +2026-11-11 10:44:43 -> 10:45:27 0.7 min vzrestore 990000 OK +2026-11-11 10:45:30 -> 10:45:33 0.1 min vzstart 990000 OK +2026-11-11 10:45:36 -> 10:45:39 0.1 min vzdestroy 990000 OK + +=== agent: night words, 20:00 yesterday to 09:00 today (first 3 lines per message) === +--- counts +RC=0 diff --git a/documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-offset.txt b/documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-offset.txt new file mode 100644 index 00000000..4b7ad6a6 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/box/tester-1-offset.txt @@ -0,0 +1,6 @@ +# offset tester-1: box clock minus DooPlex clock (midpoint of the call), seconds +offset=0.074 rtt=0.154 +offset=0.072 rtt=0.154 +offset=0.071 rtt=0.152 +offset=0.073 rtt=0.153 +offset=0.072 rtt=0.152 diff --git a/documentation/audits/clock-list-2026-11-11/evidence/helpers/helper-clock-agent.md b/documentation/audits/clock-list-2026-11-11/evidence/helpers/helper-clock-agent.md new file mode 100644 index 00000000..841e242e --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/helpers/helper-clock-agent.md @@ -0,0 +1,256 @@ +# Clock list: host agent and host installer (helper result, candidates only) + +### 1. Whole-guest backup: due-check and run +1. The in-guest controller asks `handleBackupDue` `AG/internal/localapi/server.go:1134-1217` and starts `handleBackup` `AG/internal/localapi/server.go:878-1015`; the host has no timer for it; the agent runs the vzdump. CUSTOMER. +2. Due when the newest archive's age reaches the cadence, `age >= tier.Cadence` `AG/internal/localapi/server.go:1204-1213`; default `BackupCadence` 24 h `AG/internal/config/config.go:576-580`; the night window is the controller's `AR/07-backup-architecture.md:551`; the agent compares ages only, no time zone. +3. Task wait `defaultPrimaryTierWaitTimeout` 30 min and `defaultExtraTierWaitTimeout` 12 h `AG/internal/config/config.go:464-465`; outer bound is the tier wait plus 15 min `AG/internal/localapi/server.go:971-975`; the copy grows with the guest's size. +4. A tier not backed up stays due at every later ask; the controller has a 48 h safety valve `AR/07-backup-architecture.md:551-552`; the hub judges 72 h without a whole-guest backup `AR/07-backup-architecture.md:579-581`. +5. Same tier in flight answers 202 with the same job `AG/internal/localapi/server.go:913-917`; another tier answers 409 `AG/internal/localapi/server.go:919-925`; the host-wide gate is `TryAcquire` `AG/internal/backup/inflight.go:40-59`, in memory. +6. The clock is wall time, `now` = `time.Now().UTC()` `AG/internal/localapi/server.go:509`; candidate: a clock set back makes the age small, so the tier reads not due until real time catches up; unreadable storage uses the saved success `AG/internal/localapi/server.go:1176-1182`. +7. The hub raises `expected_backup_missed` to the operator `AR/07-backup-architecture.md:579-582`; a flood refused by the cap raises an operator event, rationale `AG/internal/localapi/backup_cap.go:12-35`. +8. TEST `TestBackupDue_R894_RestartThenUnreadableStorage_OldSavedCopyIsDue` `AG/internal/localapi/r894_saved_backup_test.go:86`; DOC `AR/07-backup-architecture.md:420-429`; clock jump: NOBODY KNOWS (grep `^func Test.*(Clock|Future|Skew|Jump|DST)` finds none for the due-check). + +### 2. OS-update night leg (guest, host, Docker, Proxmox) +1. The agent runs it after a successful primary, non-manual whole-guest backup: `SetAfterPrimaryBackup` `AG/cmd/felhom-agent/main.go:1390-1398`, trigger at `AG/internal/localapi/server.go:1008-1011`; `Run` `AG/internal/osupdate/leg.go:538-586`. CUSTOMER. +2. 90 s after the backup ends `AG/cmd/felhom-agent/main.go:1395`; at most once per `MinGap`, default 20 h `AG/internal/osupdate/leg.go:656-667`; order guest, host, Docker, Proxmox, kernel `AG/internal/osupdate/leg.go:538-586`; no clock slot of its own `AR/11-os-updates.md:378-387`. +3. Wrapper limits: host command 600 s and guest command 1800 s `AG/configs/felhom-os-apply:279-284`; health wait default 5 min `AG/internal/osupdate/leg.go:810-813`; time grows with the number of packages; measured 23 to 70 s `AR/07-backup-architecture.md:449-450`. +4. A failed or missed backup means no OS leg that night `AR/07-backup-architecture.md:447`; no catch-up found; the hub shows how far behind a box is `AR/11-os-updates.md:406-408`. +5. It runs holding the heavy-op gate taken by the backup `AR/07-backup-architecture.md:444-446`; one pass at a time by flock `tryLockPass` `AG/internal/osupdate/unsent.go:35-52`; a second night run inside 20 h is skipped `AG/internal/osupdate/leg.go:663`. +6. The 20 h gap compares `l.now().Sub(last)` with a time read from a file `AG/internal/osupdate/leg.go:661-662`; candidate: a stamp written while the clock was ahead keeps the leg skipped until real time passes that stamp plus 20 h. +7. The result is posted to the hub by `finish` `AG/internal/osupdate/leg.go:922-939`; operator gets a hub event per update and failure, the household one timeline line `AR/11-os-updates.md:406-423`. +8. TEST `TestOncePerNight` `AG/internal/osupdate/leg_test.go:367` and `TestAfterPrimaryBackup` `AG/internal/localapi/afterbackup_test.go:17`; DOC `AR/11-os-updates.md:366-387`; a stamp from the future: NOBODY KNOWS. + +### 3. Kernel lane: night reboot, 20-minute boot judge, one self-revert +1. Agent: `runKernel` ends the night leg `AG/internal/osupdate/kernel.go:119-178`; at every agent start `KernelAfterBoot` judges the boot `AG/cmd/felhom-agent/main.go:1018`. CUSTOMER (the box restarts at night). +2. Only on trigger night and only when the hub's block says `Tonight` `AG/internal/osupdate/kernel.go:101-115`; the household is mailed the day before, 09:00 to 20:00 Budapest time, by the hub `AR/11-os-updates.md:580-585`. +3. Judge wait `DefaultKernelJudgeWait` 20 min `AG/internal/osupdate/kernel.go:37`, polled every 30 s `AG/internal/osupdate/kernel.go:251`; measured healthy at 68 s and 272 s `AR/11-os-updates.md:570-572`. +4. No mail accepted means no step `AR/11-os-updates.md:584-585`; a step is never retried by itself `AR/11-os-updates.md:581-583`; a box that never comes back raises `host_stale` at 45 min and `host_down` at 90 min `AR/11-os-updates.md:572-573`. +5. At most one per night under the same heavy-op gate `AR/11-os-updates.md:560-562`; never two steps within 20 h (wrapper refusal R22) `AR/11-os-updates.md:557-558`; one self-revert, never retried `AG/internal/osupdate/kernel.go:300-312`. +6. The judge deadline is in-process from `now` = `time.Now()` `AG/internal/osupdate/leg.go:291-296` (monotonic kept); a clock step during the 20 min does not move it. Host starting with a wrong clock: NOT FOUND (grep `timedatectl|chrony|ntp` in `E/scripts/felhom-host-install.sh:1` finds nothing). +7. The hub hears staged before the reboot `AG/internal/osupdate/kernel.go:165`; operator events `os_kernel_step` and `os_kernel_notice` `AR/11-os-updates.md:585-586`; a frozen new kernel needs a person, accepted `AR/11-os-updates.md:534-535`. +8. TEST `TestKernelAfterBoot_UnhealthyRevertsOnceAfterTheWait` `AG/internal/osupdate/kernel_test.go:200`, `TestKernel_NoMailNoStep` `AG/internal/osupdate/kernel_test.go:79`; DOC `AR/11-os-updates.md:530-601`; a real freeze: NOBODY KNOWS `AR/11-os-updates.md:599-601`. + +### 4. Crash guard: early-boot unit plus hourly re-arm timer +1. Root systemd units installed by the host installer `E/scripts/felhom-host-install.sh:2944-2963`: boot unit `AG/configs/felhom-crash-guard.service:15-16` and timer `AG/configs/felhom-crash-guard-check.timer:5-6`. CUSTOMER (decides if a crashed box restarts). +2. Timer as written: `OnBootSec=15min`, `OnUnitActiveSec=1h` `AG/configs/felhom-crash-guard-check.timer:5-6`; no OnCalendar, no RandomizedDelaySec, no Persistent; boot unit runs before sysinit `AG/configs/felhom-crash-guard.service:8`. +3. Oneshot `AG/configs/felhom-crash-guard-check.service:6-7` with no time limit written; history is cut to 7 days `AG/configs/felhom-crash-guard:148`, so nothing grows. +4. A missed hourly check only delays the re-arm; a crash before the boot unit runs leaves the box off, the safe side `AR/11-os-updates.md:484-485`; the System page shows the guard `AR/11-os-updates.md:487-488`. +5. Limits `LIMIT=3`, `WINDOW_MINUTES=60`, `REARM_HOURS=24` `AG/configs/crash-guard.conf:3-8`; the 2nd unclean boot in the window trips it `AG/configs/felhom-crash-guard:155`; no lock between boot and check found (candidate, grep `flock|lock` not run on it: NOT FOUND). +6. Wall clock: `now` is `time.time()` `AG/configs/felhom-crash-guard:43-44` and the window test is `now - t <= window` `AG/configs/felhom-crash-guard:124`, read at early boot; candidate: a clock that is wrong at boot can move unclean boots in or out of the 60 min window. +7. Each unclean boot is an operator mail `host_crash_restart` and a household line; the trip is alarm `host_crash_guard_tripped` `AR/11-os-updates.md:487-488`; the trip mail came 15 min after the boot `AR/11-os-updates.md:489-490`. +8. TEST `test_third_crash_in_an_hour_leaves_the_box_off` `AG/configs/test_felhom_crash_guard.py:75` and `test_rearms_after_24h_of_normal_running` `AG/configs/test_felhom_crash_guard.py:102`; DOC `AR/11-os-updates.md:473-490`; wrong clock at boot: NOBODY KNOWS. + +### 5. Scheduled restore-test evaluation +1. Agent goroutine `Run` `AG/internal/backup/schedule.go:148-177`, wired at `AG/cmd/felhom-agent/main.go:1968-2018`; restores an archive into a scratch guest on the host. CUSTOMER (proof the backup opens). +2. First look `DefaultFirstEval` 30 min after agent start `AG/internal/backup/schedule.go:76`, then every `defaultRestoreTestEvalInterval` 6 h `AG/internal/config/config.go:636`; an archive counts after `defaultRestoreTestSettle` 24 h `AG/internal/config/config.go:641`; not windowed, any hour `AR/11-os-updates.md:385-386`. +3. Restore task wait: `RestoreTestPBSRestoreTimeout` 120 min for the PBS tier `AG/internal/config/config.go:584-588`, else the 10 min default `AG/cmd/felhom-agent/main.go:1917-1919`; boot wait `DefaultBootTimeout` 2 min `AG/internal/reconcile/restoretest.go:28`; grows with guest size. +4. The ticker restarts with the agent, so a missed look comes 30 min after the next start `AR/07-backup-architecture.md:583-584`; the rule is per archive, so a tier stays due until proven `AR/07-backup-architecture.md:179-180`; hub proof-age window 7 to 12 days `AR/07-backup-architecture.md:215-217`. +5. Takes the heavy-op gate with `TryAcquire("restore-test")` `AG/internal/backup/schedule.go:228`; busy means deferred to the next 6 h look `AG/internal/backup/schedule.go:230`; proof credit only on a pass `AG/internal/backup/schedule.go:248-255`. +6. `now` is `time.Now().UTC()`, wall clock `AG/internal/backup/schedule.go:119`, used for `settleCutoff` `AG/internal/backup/restoretest_due.go:96-101`; candidate: a clock set back makes no archive look settled; the ticker itself is monotonic. +7. A failed test is an ERROR log `AG/internal/backup/schedule.go:262` and a record the hub turns into `restore_test_failed` for the operator, as the janitor comment says `AG/cmd/felhom-agent/janitor.go:42-44`. +8. TEST `TestR874_FirstEvaluationAfterStart` `AG/internal/backup/r874_first_eval_test.go:17`, `TestDue_DailyTierIsProvedDailyOnItsOwnArchive` `AG/internal/backup/restoretest_due_test.go:139`; DOC `AR/07-backup-architecture.md:174-200`; clock set back: NOBODY KNOWS. + +### 6. Controller supervisor (restarts a dead or silent controller) +1. Agent goroutine `WatchControllers` `AG/internal/localapi/controllersupervisor.go:143-163`, started at `AG/cmd/felhom-agent/main.go:1714`. CUSTOMER (the household's dashboard comes back). +2. Every `controllerSupervisorInterval` 30 s `AG/internal/localapi/controllersupervisor.go:54`; restart on the 2nd not-running sweep `AG/internal/localapi/controllersupervisor.go:56`; a running but unhealthy one is alarmed at 6 sweeps and restarted at 10 `AG/internal/localapi/controllersupervisor.go:92-100`. +3. No per-sweep time limit seen in the lines read (NOT FOUND, only `AG/internal/localapi/controllersupervisor.go:1-200` was opened); nothing grows with data; a liveness line every 20 sweeps `AG/internal/localapi/controllersupervisor.go:64`. +4. If the agent is down nothing restarts the controller: measured 33 min of 502 before this existed `AG/internal/localapi/controllersupervisor.go:20-25`; the first sweep is 30 s after agent start (ticker only) `AG/internal/localapi/controllersupervisor.go:153-161`. +5. Brake: `controllerCrashloopMax` 3 in `controllerCrashloopWindow` 15 min, then `controllerCrashloopPause` 30 min `AG/internal/localapi/controllersupervisor.go:59-61`; not during a swap, a parked or locked guest, or a vzdump `AG/internal/localapi/controllersupervisor.go:34-42`. +6. The fast record is in memory, so an agent restart forgets a pause `AG/internal/localapi/controllersupervisor.go:103-105`; the 24 h counter is persisted `AG/internal/localapi/controllersupervisor.go:115-119`; wall or monotonic for the windows: NOT FOUND (the tick body was not opened). +7. Hub mints `controller_restarted_by_agent` (info) and `controller_crashloop` (error, operator only) from the host report `AG/internal/localapi/controllersupervisor.go:44-49`; a warning at the 5th restart in 24 h `AG/internal/localapi/controllersupervisor.go:82-83`. +8. TEST `TestControllerSupervisor_KilledControllerIsRestarted` `AG/internal/localapi/controllersupervisor_test.go:101`, `TestSupervisor_UnresponsiveIsAlarmedThenRestarted` `AG/internal/localapi/supervisor_unresponsive_test.go:32`; DOC `AR/03-host-agent.md:186-192`. + +### 7. Guest-power watchdog (starts a guest that should be running) +1. Agent goroutine `WatchGuestPower` `AG/internal/localapi/guestpower.go:89-105`, started at `AG/cmd/felhom-agent/main.go:1710`. CUSTOMER (the whole box's apps are off while the guest is stopped). +2. Every `guestPowerInterval` 60 s `AG/internal/localapi/guestpower.go:52`; acts only on a stopped, unlocked guest with onboot set `AG/internal/localapi/guestpower.go:167-180`; first sweep one interval after start (ticker only) `AG/internal/localapi/guestpower.go:95-102`. +3. At most `guestPowerMaxAttempts` 3 starts `AG/internal/localapi/guestpower.go:56` with waits of 1, 2 and 4 min, `guestPowerBackoff` `AG/internal/localapi/guestpower.go:75`; about 7 min in all `AG/internal/localapi/guestpower.go:71-74`. +4. Before it existed a guest stayed down 9 min 47 s until a person started it `AG/internal/localapi/guestpower.go:14-18`; with the agent down nobody starts it (the platform's onboot acts only at host boot `AG/internal/localapi/guestpower.go:42-43`). +5. Never starts under a running vzdump and fails safe on doubt `AG/internal/localapi/guestpower.go:182-194`; leaves a locked guest to the stale-lock path `AG/internal/localapi/guestpower.go:176-180`; skips a guest the operator is rebooting `AG/internal/localapi/guestpower.go:157-159`. +6. Attempt state is in memory on purpose, so an agent restart tries again at once `AG/internal/localapi/guestpower.go:77-80`; wall or monotonic for `nextAt`: NOT FOUND (`guestPowerDue` `AG/internal/localapi/guestpower.go:226` was not opened). +7. Candidate: the give-up is an ERROR journal line only `AG/internal/localapi/guestpower.go:205-206`; a hub event for it is NOT FOUND (grep `hub\.|Report|report` in `AG/internal/localapi/guestpower.go:1` finds no sender). +8. TEST `TestGuestPower_RetryIsBoundedThenEscalates` `AG/internal/localapi/guestpower_test.go:162`, `TestGuestPower_InFlightBackupIsNotDisturbed` `AG/internal/localapi/guestpower_test.go:119`; who hears the give-up: NOBODY KNOWS. + +### 8. Hub report and heartbeat loop (with reconcile, fast tick and poke) +1. Agent goroutine `Run` `AG/internal/hub/loop.go:107-138`, built at `AG/cmd/felhom-agent/main.go:958`; it carries every status the hub alarms on. OPERATOR and CUSTOMER (alarms depend on it). Merged here: the reconcile engine, fast tick and poke. +2. `PollSeconds` default 900 `AG/internal/config/config.go:797`; first report at once, then a ticker `AG/internal/hub/loop.go:107-138`; the hub may change it, clamped by `MinPollSeconds` 60 and `MaxPollSeconds` 3600 `AG/internal/hub/loop.go:11-12`; fast tick `DefaultInterval` 30 s `AG/internal/fasttick/fasttick.go:24`. +3. Per-request limit `TimeoutSeconds` default 30 `AG/internal/config/config.go:798`; the system facts in it are read at most every 10 min `AG/cmd/felhom-agent/main.go:4149`; the report grows with guests and drives. +4. A failed cycle is logged and the interval kept `AG/internal/hub/loop.go:162-164`; no backlog, the next report replaces it; the hub raises `host_stale` at 45 min and `host_down` at 90 min `AR/11-os-updates.md:572-573`. +5. Extra reports come by the trigger channel `AG/internal/hub/loop.go:125-135`; a poke is debounced by `DebounceWindow` 2 s `AG/internal/poke/poke.go:40`; the reconcile engine runs at once and then on the start-time interval `AG/internal/reconcile/engine.go:332-344`, `AG/cmd/felhom-agent/main.go:1525`. +6. The ticker is monotonic; candidate: the reconcile engine keeps the interval read at start `AG/cmd/felhom-agent/main.go:969` and does not follow the hub's override `AG/internal/hub/loop.go:188-196`. A suspended host: NOT FOUND for the agent (the controller has a rule `AR/07-backup-architecture.md:555-558`). +7. The operator, by the hub's staleness alarms `AR/11-os-updates.md:572-573`; the household hears cannot-be-reached at most once per 7 days `AR/07-backup-architecture.md:582-583`. +8. TEST `TestLoop_RunImmediateAndResilientAfterError` `AG/internal/hub/loop_test.go:140`, `TestClampInterval` `AG/internal/hub/loop_test.go:81`; DOC `AR/03-host-agent.md:276-279`. + +### 9. Signed jobs: validity window against the host clock (agent updates ride it) +1. Agent `OnEnvelope` runs pending signed jobs when a heartbeat says there are some `AG/internal/signedjobs/runner.go:93-102`; an agent update arrives only this way. OPERATOR (and CUSTOMER: fixes reach the box this way). +2. Checked at every hub cycle (item 8); a job is valid from issue minus `DefaultClockSkew` 2 min `AG/internal/authz/verifier.go:21` to its expiry, and a window over `MaxWindow` 24 h is refused `AG/internal/authz/verifier.go:26`. +3. The self-update download has a 5 min client limit `AG/internal/selfupdate/executor.go:73`; the new binary must run `DwellSeconds` 60 s before it commits `AG/internal/config/config.go:97-99`. +4. A job not picked up before its expiry is refused as expired `AG/internal/authz/verifier.go:161-162`; a commit that fails is retried at the next start `AG/internal/selfupdate/commit.go:122-125`; who notices an expired job: NOT FOUND in the agent (hub side). +5. One run at a time, `RunOnce` single-flight `AG/internal/signedjobs/runner.go:107-115`; a replay is refused by the nonce store `AG/internal/authz/verifier.go:166`. +6. Wall clock: candidate, a host clock that is behind refuses every job as not yet valid `AG/internal/authz/verifier.go:155-156`, one that is ahead refuses them as expired `AG/internal/authz/verifier.go:161-162`; so a box with a wrong clock cannot be updated by signed job. +7. NOT FOUND in the agent source read (grep `not_yet|NotYet|expired` shown only in `AG/internal/authz/verify_test.go:140-155`); the result path to the hub was not opened. +8. TEST `TestVerify_RejectsPerStage` `AG/internal/authz/verify_test.go:82`, `TestRunner_ExpiredRejected` `AG/internal/signedjobs/runner_test.go:168`, `TestManager_ShutdownBeforeDwellLeavesPending` `AG/internal/selfupdate/commit_test.go:102`. + +### 10. Storage watchdog and the 20-second mount and bind re-assert +1. Agent: the watchdog `Run` `AG/internal/storage/watchdog.go:196-215` wired at `AG/cmd/felhom-agent/main.go:1142`, and a 20 s ticker that re-asserts mounts and guest binds `AG/cmd/felhom-agent/main.go:1613-1631`. CUSTOMER (a dropped USB drive). +2. `DefaultWatchdogInterval` 8 s and `DefaultWatchdogDebounce` 30 s `AG/internal/storage/watchdog.go:15-16`; setting `WatchdogIntervalSeconds`, 0 means the default `AG/internal/config/config.go:770-778`; the re-assert is 20 s, below the controller's 30 s tick `AG/cmd/felhom-agent/main.go:1612-1614`. +3. No per-tick limit found in the lines read (NOT FOUND, grep `WithTimeout` not run on the watchdog); the work grows with the number of enrolled drives. +4. A missed tick is made up by the next one; with the agent down a dropped drive is seen only at the next 15 min report, the reason the watchdog exists `AR/03-host-agent.md:461`. +5. A flapping drive backs off: `backoffWindow` `AG/internal/storage/watchdog.go:183-192`, alert after 4 failed cycles `AG/internal/storage/watchdog.go:77-79`; the retry test is `now.Sub(last)` `AG/internal/storage/watchdog.go:290`. +6. `now` is `time.Now().UTC()`, wall clock `AG/internal/storage/watchdog.go:160`; candidate: a clock set back stretches the back-off, set forward ends it early; the tickers are monotonic. +7. A drop requests an immediate hub report through the trigger channel `AG/internal/hub/loop.go:125-135`; the hub then decides who is mailed (hub side, not read). +8. TEST `TestWatchdog_DebounceCoalescesFlaps` `AG/internal/storage/watchdog_test.go:112`, `TestWatchdog_Reconcile_FlappingBacksOffAndCaps` `AG/internal/storage/watchdog_test.go:460`; the 20 s ticker: NOBODY KNOWS (no test found for the goroutine at `AG/cmd/felhom-agent/main.go:1613`). + +### 11. Weekly guest disk trim +1. Agent goroutine `Run` `AG/internal/fstrim/fstrim.go:186-199`, started at `AG/cmd/felhom-agent/main.go:1741`; runs `pct fstrim` per owned running guest. CUSTOMER (a full thin pool turns every guest read-only `AG/internal/fstrim/fstrim.go:3-5`). +2. Due from Wednesday 10:00 host-local, starts only 10:00 to 20:59 `AG/internal/fstrim/fstrim.go:42-46`; looks 5 min after start then hourly, `FirstTickDelay` and `TickInterval` `AG/internal/fstrim/fstrim.go:48-50`; zone is `time.Local` `AG/internal/fstrim/fstrim.go:110`. +3. `PerGuestTimeout` 30 min `AG/internal/fstrim/fstrim.go:52`; measured 24.4 s for 84 GiB `AG/internal/fstrim/fstrim.go:6`; grows with freed blocks. +4. A box off on Wednesday catches up at its next eligible hour `AG/internal/fstrim/fstrim.go:10-11`; a failure is retried at most `MaxAttemptsPerWeek` 3 times `AG/internal/fstrim/fstrim.go:46`; the result rides the host report `AG/internal/fstrim/fstrim.go:326-330`. +5. Holds the heavy-op gate for the whole pass; busy means deferred to the next hour `AG/internal/fstrim/fstrim.go:247-251`; the persisted record stops a second trim after a restart `AG/internal/fstrim/fstrim.go:17`. +6. Wall clock in host-local time `AG/internal/fstrim/fstrim.go:204`; candidate: an attempt stamped while the clock was ahead is not before the anchor, so `due` stays false until real time passes it `AG/internal/fstrim/fstrim.go:179-182`. No zone set by the installer: NOT FOUND (grep `timedatectl|set-timezone`). +7. A failed trim is a WARN journal line `AG/internal/fstrim/fstrim.go:318-319` and a field on the host report; who is mailed: NOT FOUND in the agent (hub side). +8. TEST `TestPassNeverRunsInTheNight` `AG/internal/fstrim/fstrim_test.go:161`, `TestPassDefersToAHeavyOperationAndRetriesNextHour` `AG/internal/fstrim/fstrim_test.go:175`, `TestWeekAnchorIsTheLastWednesdayTen` `AG/internal/fstrim/fstrim_test.go:104`; DST and clock jump: NOBODY KNOWS. + +### 12. Weekly upload line test +1. Agent goroutine `Run` `AG/internal/linetest/linetest.go:256-268`, started at `AG/cmd/felhom-agent/main.go:1762`; a 10 s iperf3 upload over the tunnel. CUSTOMER (sets the torrent app's upload limit `AG/internal/linetest/linetest.go:3-4`). +2. Due from Tuesday 10:00 host-local, starts 10:00 to 15:59 `AG/internal/linetest/linetest.go:47-49`; each host from its own minute, `SlotMinute` `AG/internal/linetest/linetest.go:148-152`; ticks every minute after a 5 min delay `AG/internal/linetest/linetest.go:60-61`; zone `time.Local` `AG/internal/linetest/linetest.go:131`. +3. Three runs of 10 s, each bounded by `RunTimeout` 25 s, 30 s apart `AG/internal/linetest/linetest.go:53-57`; nothing grows with data. +4. A box that was off catches up on a later day inside the window `AG/internal/linetest/linetest.go:10-11`; the hub page turns amber when the last success is over 15 days old `AR/11-os-updates.md:409-411`. +5. Needs 10 quiet minutes and the heavy-op gate `AG/internal/linetest/linetest.go:316-325`; server busy is retried at the next tick, at most `MaxTriesPerDay` 6 `AG/internal/linetest/linetest.go:58`; no tunnel means no test `AG/internal/linetest/linetest.go:297-300`. +6. Wall clock, host-local `AG/internal/linetest/linetest.go:281`; candidate: `Due` is false while `LastOKAt` is not before the anchor, so a success stamped from the future blocks tests until then `AG/internal/linetest/linetest.go:211-213`; a sample gap over 3 min is not quiet `AG/internal/linetest/linetest.go:244-245`. +7. The operator, by the hub System page cell judged on the last success, not the attempt `AR/11-os-updates.md:409-411`; nobody is mailed as far as the agent source shows (NOT FOUND, hub side). +8. TEST `TestDueWeeklyFromTuesday` `AG/internal/linetest/linetest_test.go:206`, `TestNeverTestsInTheNightOrBeforeTuesday` `AG/internal/linetest/linetest_test.go:363`, `TestSlotDelaysTheFirstStartOnTheDueDayOnly` `AG/internal/linetest/linetest_test.go:189`. + +### 13. PBS datastore verify loop +1. Agent goroutine `Run` `AG/internal/pbs/verify.go:64-83`, cadence wired at `AG/cmd/felhom-agent/main.go:1188`; asks the PBS server to verify the box's snapshots. CUSTOMER (is the off-site copy readable). +2. `DefaultVerifyCadence` 6 h `AG/internal/pbs/verify.go:13`; setting through `PBSVerifyCadence` `AG/internal/config/config.go:710-719`; first pass at once at agent start, then the ticker `AG/internal/pbs/verify.go:64-83`; no time zone. +3. One verify is polled every 2 s for at most 30 min `AG/internal/pbs/verify.go:107`; grows with the datastore's size. +4. No catch-up state: every agent start verifies again at once `AG/internal/pbs/verify.go:64-83`; a negative cadence switches it off, test `TestVerifyLoop_DisabledByNegativeCadence` `AG/internal/pbs/verify_test.go:87`. +5. Candidate: it is not behind the heavy-op gate, the lanresolver comment calls it "not journaled/gated" `AG/internal/lanresolver/loop.go:17`; so it can run beside a backup; NOT FOUND for a lock (grep `TryAcquire` not run on `AG/internal/pbs/verify.go:1`). +6. Ticker only, monotonic; no wall-clock decision found in `AG/internal/pbs/verify.go:64-83`. +7. Failed-verify snapshots are an ERROR journal line `AG/internal/pbs/verify.go:127` and recorded state for the report; who is mailed: NOT FOUND in the agent (hub side). +8. TEST `TestVerifyLoop_FailedVerifyRecorded` `AG/internal/pbs/verify_test.go:52`; a verify longer than 30 min: NOBODY KNOWS. + +### 14. Janitor: leftover scratch guests and stale backup locks +1. Agent goroutine `runJanitor` `AG/cmd/felhom-agent/janitor.go:69-81`, started at `AG/cmd/felhom-agent/main.go:1725`. CUSTOMER (a stale lock blocked whole-box backups for five hours `AG/cmd/felhom-agent/janitor.go:14-17`). +2. Every `janitorInterval` 10 min `AG/cmd/felhom-agent/janitor.go:18`; ticker only, so the first pass is 10 min after agent start `AG/cmd/felhom-agent/janitor.go:71-78`; no time zone. +3. No time limit on a pass found in `AG/cmd/felhom-agent/janitor.go:34-66`; work grows with the number of leftover scratch guests. +4. A missed pass is retried 10 min later; a scratch guest that cannot be removed after the retry limit is reported as a failed restore-test record `AG/cmd/felhom-agent/janitor.go:41-54`. +5. The stale-lock sweep runs only holding the heavy-op gate, else it is skipped this pass `AG/cmd/felhom-agent/janitor.go:57-64`; rule stated at `AG/cmd/felhom-agent/janitor.go:30-33`. +6. Ticker only, monotonic; a clock step must not kill a live task: `TestVzdumpInFlight_ClockStepCannotKillALiveTask` `AG/internal/localapi/stalelock_inflight_test.go:97`. +7. The operator, through the hub's `restore_test_failed`, once per distinct archive `AG/cmd/felhom-agent/janitor.go:42-44`. +8. TEST `TestJanitor_StaleLockSweepWaitsForTheHeavyGate` `AG/cmd/felhom-agent/janitor_test.go:40`, `TestJanitor_GaveUpIsReportedAsAFailure` `AG/cmd/felhom-agent/janitor_test.go:22`. + +### 15. Guest network watchdog (DHCP client heal) +1. Agent goroutine `Watch` `AG/internal/guestnet/watchdog.go:105-121`, built at `AG/cmd/felhom-agent/main.go:1686`. CUSTOMER (a guest without network). +2. `DefaultInterval` 60 s `AG/internal/guestnet/watchdog.go:23`; immediate first sweep `AG/internal/guestnet/watchdog.go:111`; settings interval 60, `MinHealIntervalSeconds` 600, `MaxHealsPerHour` 3 `AG/internal/config/config.go:176-196`. +3. No per-probe limit seen in `AG/internal/guestnet/watchdog.go:1-200` (NOT FOUND); nothing grows with data. +4. A missed sweep is made up a minute later; beyond 3 heals an hour it only reports `AG/internal/guestnet/watchdog.go:26-28`. +5. Two consecutive bad probes before a heal, `requiredBadProbes` `AG/internal/guestnet/watchdog.go:32-33`; 10 min between heals, `DefaultMinHealInterval` `AG/internal/guestnet/watchdog.go:25`; boot-race guard `DefaultSettle` 3 min `AG/internal/guestnet/watchdog.go:29-31`. +6. `now` is plain `time.Now` `AG/internal/guestnet/watchdog.go:80`, so in-process gaps are monotonic; heal history is in memory `AG/internal/guestnet/watchdog.go:56-62`. +7. A report per guest with `HealsLastHour` and `CheckedAt` `AG/internal/guestnet/watchdog.go:166-170`; who is mailed: NOT FOUND in the agent (hub side). +8. TEST `TestWatchdog_DampingCeilings` `AG/internal/guestnet/watchdog_test.go:360`, `TestWatchdog_MinimumIntervalBetweenHeals` `AG/internal/guestnet/watchdog_test.go:384`, `TestWatchdog_BootRacesObserveOnly` `AG/internal/guestnet/watchdog_test.go:416`. + +### 16. Host networking self-heal +1. Agent goroutine `Watch` `AG/internal/selfheal/selfheal.go:67-82`, interval set at `AG/cmd/felhom-agent/main.go:1648`; starts `networking.service` on an appliance only. CUSTOMER. +2. One pass at agent start, then every 60 s `AG/internal/selfheal/selfheal.go:67-82`, `AG/cmd/felhom-agent/main.go:1648`; no setting found for the interval (hard-coded). +3. A remedy is 3 start attempts with waits of 10, 30 and 60 s, `networkingBackoffs` `AG/internal/selfheal/selfheal.go:114-116`. +4. A missed pass is made up a minute later; on a BYO host the remedy is skipped and only a WARN is written `AG/internal/selfheal/selfheal.go:54-58`. +5. No lock seen; the pass is sequential inside one goroutine `AG/internal/selfheal/selfheal.go:46-64`. +6. Ticker only, monotonic; no wall-clock decision in `AG/internal/selfheal/selfheal.go:67-82`. +7. A give-up is an ERROR journal line `AG/internal/selfheal/selfheal.go:61`; with networking down the hub cannot be told, so the hub's stale alarm is the only signal `AR/11-os-updates.md:572-573`. +8. TEST `TestNetworkingHeal_GivesUpAfterAllAttempts` `AG/internal/selfheal/selfheal_test.go:109`, `TestManager_ByoNeverRemediates` `AG/internal/selfheal/selfheal_test.go:32`. + +### 17. Management-plane watchdog timer (keeps SSH reachable) +1. Root systemd timer and oneshot installed by the host installer `E/scripts/felhom-host-install.sh:3067-3076`; script `AG/configs/felhom-mgmt-watchdog.sh:26-68`; works without the agent `AG/configs/felhom-mgmt-watchdog.service:3-7`. OPERATOR. +2. As written: `OnBootSec=30s`, `OnUnitActiveSec=60s`, `AccuracySec=5s`, `Persistent=true` `AG/configs/felhom-mgmt-watchdog.timer:11-14`; no OnCalendar, no RandomizedDelaySec. +3. No time limit written in the unit `AG/configs/felhom-mgmt-watchdog.service:16-18`; the script is a few stat and systemctl calls `AG/configs/felhom-mgmt-watchdog.sh:36-61`. +4. A missed tick is retried by the next one `AG/configs/felhom-mgmt-watchdog.sh:23-24`; the heal budget is one tick `AG/configs/felhom-mgmt-watchdog.timer:3-6`. +5. Idempotent: a healthy directory is never touched `AG/configs/felhom-mgmt-watchdog.sh:13-15`; candidate: `Persistent=true` only acts on OnCalendar timers in systemd, and this timer has none `AG/configs/felhom-mgmt-watchdog.timer:11-14`. +6. Monotonic timer settings only `AG/configs/felhom-mgmt-watchdog.timer:11-12`; the heal marker is a UTC stamp from `date -u` `AG/configs/felhom-mgmt-watchdog.sh:65`. +7. The marker lets the agent report the heal and the hub raise a warning `AG/configs/felhom-mgmt-watchdog.sh:18-19`, `AG/configs/felhom-mgmt-watchdog.sh:63`. +8. SOURCE `AG/configs/felhom-mgmt-watchdog.sh:1-68`; a test for the script: NOT FOUND (a listing of the agent configs folder shows no `test_felhom_mgmt_watchdog` file); DOC comment `AG/configs/felhom-mgmt-watchdog.timer:3-6`. + +### 18. Agent service restart, start limit and rollback +1. systemd restarts the agent: `Restart=on-failure`, `RestartSec=5s` `AG/configs/felhom-agent.service:41-42`; drop-in limits `AG/configs/felhom-agent-limits.conf:17-19`. OPERATOR (every loop in this list dies with the agent). +2. `StartLimitIntervalSec=120`, `StartLimitBurst=4`, `OnFailure=felhom-agent-rollback.service` `AG/configs/felhom-agent-limits.conf:17-19`; a crash-looping binary reaches failed about 20 s after the first crash `AG/configs/felhom-agent-limits.conf:5-8`. +3. No WatchdogSec in the unit (grep `WatchdogSec` in `AG/configs/felhom-agent.service:1` finds nothing); a new binary commits after a 60 s dwell `AG/internal/selfupdate/commit.go:40-42`. +4. After the start limit the unit stays failed and nothing on the host restarts it (candidate); the hub's `host_stale` at 45 min is the signal `AR/11-os-updates.md:572-573`. +5. The rollback unit fires on every crash and is guarded by a pending marker `AG/configs/felhom-agent-limits.conf:14-15`; a shutdown before the dwell leaves the update pending `AG/internal/selfupdate/commit.go:112-116`. +6. Candidate: the design text says `Restart=always` and a systemd watchdog that kills a hung agent `AR/03-host-agent.md:72`; the unit has `Restart=on-failure` and no watchdog `AG/configs/felhom-agent.service:41-42`, so a hung (not crashed) agent is not restarted. +7. The operator, by the hub's stale and down alarms only `AR/11-os-updates.md:572-573`; a restore-test crash loop that never evaluates is covered by `TestR874_CrashLoopNeverEvaluates` `AG/internal/backup/r874_first_eval_test.go:44`. +8. SOURCE `AG/configs/felhom-agent-limits.conf:5-15` (measured values quoted there); TEST `TestManager_ShutdownBeforeDwellLeavesPending` `AG/internal/selfupdate/commit_test.go:102`; a hung agent: NOBODY KNOWS. + +### 19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops) +1. Agent goroutines of one shape: tunnel `Run` `AG/internal/wgtunnel/loop.go:62-78`, PBS-DR `AG/internal/pbsdr/loop.go:62-77`, OOB sshd `AG/internal/felhomsshd/loop.go:67-80`, LAN resolver `AG/internal/lanresolver/loop.go:35-52`. CUSTOMER (off-site backup path) and OPERATOR. Four loops merged. +2. Tunnel, PBS-DR and OOB every 60 s `AG/internal/config/config.go:132-133`, `AG/cmd/felhom-agent/main.go:1504`, `AG/internal/config/config.go:74-75`; resolver every 5 min `AG/internal/lanresolver/loop.go:27-29`; each applies at once at start and on a desired-state nudge. +3. No per-pass limit read (NOT FOUND, manager bodies not opened); nothing grows with data. +4. A missed pass is made up at the next tick; a handshake older than `StaleAfterSeconds` 180 re-resolves the endpoint `AG/internal/config/config.go:124-127`, `AG/internal/config/config.go:138-140`. +5. A handshake stamped in the future for `rekeyAfterDefault` 15 min makes a new key, at most once per `rekeyMinGap` 1 h `AG/internal/wgtunnel/rekey.go:36-38`; the hour bound survives a restart, the 15 min count does not `AR/03-host-agent.md:376-378`. +6. A box whose clock once ran ahead is locked out of the off-site endpoint until real time passes that moment `AR/03-host-agent.md:142`; healed by the re-key `AR/03-host-agent.md:144-150`; known gap: a reboot after the clock was corrected forgets the future stamp `AR/03-host-agent.md:155-156`. +7. The heartbeat carries `handshake_future_s` and the hub alarms on it `AR/03-host-agent.md:142`. +8. TEST `TestD30_AFutureHandshakeThatStaysReKeys` `AG/internal/wgtunnel/rekey_test.go:35`, `TestD30_ALastReKeyFromTheFutureDoesNotBlock` `AG/internal/wgtunnel/rekey_test.go:213`, `TestR1204_AFutureHandshakeIsReported` `AG/internal/wgtunnel/manager_test.go:704`; measured live `AR/03-host-agent.md:151-155`. + +### 20. Whole-guest backup cap (rolling windows) +1. Agent local API check `capRefuses` before a backup starts `AG/internal/localapi/server.go:929`; rule `backupCadenceCap` `AG/internal/localapi/backup_cap.go:41-53`. CUSTOMER (protects older copies from a flood of new ones). +2. Tier of 24 h or less: 3 successes per rolling 24 h; longer tier: 1 per the larger of cadence minus 24 h and half the cadence `AG/internal/localapi/backup_cap.go:12-35`; a refusal holds for `backupCapRefusalHold` 24 h `AG/internal/localapi/backup_cap.go:38`. +3. A lookup, no duration of its own; the success list is bounded by the window `AG/internal/localapi/backup_cap.go:41-53`. +4. Does not apply to a missed run: failures do not count, `TestBackupCap_FailuresDoNotCount` `AG/internal/localapi/backup_cap_test.go:266`. +5. The scheduled daily run is never refused, `TestBackupCap_TheScheduledDayIsNeverRefused` `AG/internal/localapi/backup_cap_test.go:220`; the count survives a restart, `TestBackupCap_SurvivesARestart` `AG/internal/localapi/backup_cap_test.go:248`. +6. Wall clock from the server's `now` `AG/internal/localapi/server.go:509`; candidate: successes stamped while the clock was ahead stay inside the rolling window longer and could refuse real runs; NOBODY KNOWS. +7. The operator, by event `whole_guest_backup_refused` named in `AG/internal/localapi/backup_cap.go:12-35`; test `TestBackupCap_RefusalRaisesTheOperatorSignal` `AG/internal/localapi/backup_cap_test.go:285`. +8. TEST `TestBackupCap_LocalFloodRunsAtMostThreeADay` `AG/internal/localapi/backup_cap_test.go:198`, `TestBackupCap_OffsiteFloodKeepsAPreCompromiseCopy` `AG/internal/localapi/backup_cap_test.go:151`. + +### 21. Operator restart and reboot limits +1. Agent local API rules for the operator's hub buttons: `opRebootEvery` `AG/internal/localapi/opreboot.go:46`, `opBrakeRefusal` `AG/internal/localapi/opactions.go:42`. OPERATOR (the household sees a notice line). +2. At most one operator reboot per guest per hour `AG/internal/localapi/opreboot.go:45-46`; the household's dashboard shows the line for `opRebootNoticeFor` 24 h `AG/internal/localapi/opreboot.go:47-48`; a brake refusal holds 30 min `AG/internal/localapi/opactions.go:42`. +3. `opRebootTimeout` 5 min bounds `pct reboot` `AG/internal/localapi/opreboot.go:49-50`; the health wait after a restart is `defaultOpHealthWait` 3 min, polled every 2 s `AG/internal/localapi/opactions.go:46-49`. +4. Not a scheduled job: nothing runs if the operator does not press; a result is forgotten after `opForgetAfter` 24 h `AG/internal/localapi/opactions.go:44`. +5. Refused during a swap, a heavy operation or a busy controller `AG/internal/localapi/opreboot.go:23-31`; the heavy-op gate is held until the guest is up `AG/internal/localapi/opreboot.go:32-34`; the busy file counts as fresh for `controllerBusyFresh` 90 s `AG/internal/localapi/opreboot.go:59-61`. +6. The reboot record is persisted with a wall-clock time `AG/internal/localapi/opreboot.go:64-70`; candidate: a clock set back after a reboot extends the one-hour refusal; NOBODY KNOWS. +7. The operator reads the result of the press; the household reads the line on its dashboard `AG/internal/localapi/opreboot.go:36-39`. +8. TEST stated as "Pinned by opreboot_test.go" `AG/internal/localapi/opreboot.go:39` (the test names were not opened); DOC `AR/03-host-agent.md:236-239`. + +### 22. Unsent OS-update report sender +1. Agent goroutine `SendUnsentLoop` `AG/internal/osupdate/unsent.go:71-82`, started at `AG/cmd/felhom-agent/main.go:1009`; posts OS-update results the hub has not received. OPERATOR. +2. Every 5 min `AG/cmd/felhom-agent/main.go:1009`, by `time.After` `AG/internal/osupdate/unsent.go:79`; one send at start, `SendUnsent` `AG/internal/osupdate/unsent.go:56-64`. +3. A report post has a 1 min limit in `finish` `AG/internal/osupdate/leg.go:922-939`; the kept copies grow only while the hub is away. +4. Hub away keeps the copy for a later pass, `TestR868_HubAwayKeepsTheCopy` `AG/internal/osupdate/unsent_test.go:87`. +5. The sender stays off while a pass holds the flock, `tryLockPass` `AG/internal/osupdate/unsent.go:35-52`; test `TestR868_RunningPassKeepsTheSenderOff` `AG/internal/osupdate/unsent_test.go:110`. +6. `time.After` is monotonic `AG/internal/osupdate/unsent.go:79`; no wall-clock decision found in `AG/internal/osupdate/unsent.go:56-82`. +7. The operator sees the result late, at most 5 min after the hub is back; no alarm of its own found (NOT FOUND, grep `Warn|Error` not run on `AG/internal/osupdate/unsent.go:1`). +8. TEST `TestR868_ACopyWrittenAfterTheStartIsSentByTheLoop` `AG/internal/osupdate/unsent_test.go:127`, `TestR868_KilledPassIsReportedAtStart` `AG/internal/osupdate/unsent_test.go:20`. + +### 23. Local-API certificate lifetime (a ten-year deadline) +1. The agent makes a self-signed leaf once, `EnsureLeaf` `AG/internal/localapi/cert.go:38-52`; every guest's controller pins its fingerprint `AG/internal/localapi/cert.go:21-24`. CUSTOMER (controller to agent calls, the backup trigger included). +2. `certValidity` is 10 years `AG/internal/localapi/cert.go:25`; start is backdated 1 h for clock skew `AG/internal/localapi/cert.go:65`; no renewal job: rotation is an operator action `AG/internal/localapi/cert.go:23-24`. +3. Not a running job; nothing grows. +4. No timer renews it; what happens at expiry and who notices: NOT FOUND (grep `NotAfter|expir` in `AG/internal/localapi/cert.go:1` shows only the creation at `AG/internal/localapi/cert.go:70`). +5. A second generation would break every issued pin; it is logged loudly `AG/internal/localapi/cert.go:34-37`. +6. Candidate: a leaf made while the host clock was far ahead has a start date in the future; whether the controller checks dates or only the pin is NOT FOUND here (controller side not read). +7. NOT FOUND: no expiry warning in the agent source read. +8. SOURCE `AG/internal/localapi/cert.go:21-25`; expiry behaviour: NOBODY KNOWS. + +## Tiny +- Guest swap check runs once, 1 min after agent start, with a 2 min limit `AG/cmd/felhom-agent/main.go:1021-1034`. +- System facts for the host report are read at most every 10 min `AG/cmd/felhom-agent/main.go:4149`. +- A backup-store grant is re-granted at most once per hour per tier, `storeGrantRepairMinInterval` `AG/cmd/felhom-agent/main.go:513`. +- Escrow ceremony: claim lives 10 min, confirm 30 min, `escrowClaimTTL` and `escrowConfirmTTL` `AG/internal/localapi/escrow_ceremony.go:158-166`. +- A network change must verify within 95 s or it is rolled back, `netVerifyDeadline` `AG/internal/localapi/netverifyjob.go:51`. +- Host power-off is refused before 10 min of uptime, `MinUptime` `AG/internal/localapi/hostpoweroff.go:76`. +- Controller swap verify: 90 s, polled every 3 s `AG/internal/localapi/controllerswap.go:86-87`. +- Guest hook phases are bounded at 30 s each `AG/cmd/felhom-agent/main.go:88-98`. +- Proxmox task wait polls with a doubling, capped interval, `WaitTask` `AG/internal/proxmox/task.go:110-141`. +- Thin-pool check reads twice with a 3 s pause `AG/internal/thincheck/thincheck.go:183`. +- Poke listener re-binds every 5 s, `rebindInterval` `AG/internal/poke/poke.go:44`. +- Backup snapshot watch ticks every 1 s during a vzdump, `snapshotWatchInterval` `AG/internal/backup/runner.go:172`. +- Proxmox package step waits up to 2 min for tasks to drain, `pveDrainWait` `AG/internal/osupdate/leg.go:589`. +- Restore-test scratch teardown has a 2 min limit detached from the caller `AG/internal/reconcile/restoretest.go:485`. +- Local whole-guest retention default is 3 daily copies, `defaultLocalBackupKeepLast` `AG/internal/config/config.go:541`. + +## Not found +- Any cron entry or `OnCalendar` timer on the host: grep `OnCalendar|cron` over the agent configs folder timers, services and shell files finds none; the only two timers are items 4 and 17. +- Any time-zone or time-sync step in the installer: grep `timedatectl|chrony|timesyncd|set-timezone|ntp` in `E/scripts/felhom-host-install.sh:1` finds nothing; items 11 and 12 use the host's local zone as found. +- A systemd watchdog for the agent: grep `WatchdogSec` over the agent configs folder finds nothing, against `AR/03-host-agent.md:72`. +- A rule in the agent for a suspended host (laptop lid): not searched beyond the controller's rule `AR/07-backup-architecture.md:555-558`; grep `^func Test.*(Clock|Future|Skew|Jump|DST|Monotonic)` in the agent repository finds only the tunnel re-key and stale-lock tests. +- Other files under the felhom.eu scripts folder that install a host timer: grep `timer|cron` named only `due_checks_gate.py`, `hostinstall-mode-harness.sh` and `test_hostinstall.py`, which are gates and tests and were not opened. +- SMART reads have no clock of their own: they ride each host report `AG/internal/storage/observe.go:184-193`. diff --git a/documentation/audits/clock-list-2026-11-11/evidence/helpers/helper-clock-controller.md b/documentation/audits/clock-list-2026-11-11/evidence/helpers/helper-clock-controller.md new file mode 100644 index 00000000..c1b95945 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/helpers/helper-clock-controller.md @@ -0,0 +1,269 @@ +# Clock list: in-guest controller (helper result, candidates only) + +### 1. Night dump leg (databases and volumes) at W +1. `dbDumpLeg` `CT/cmd/controller/main.go:1336-1344` runs `RunDBDumps` `CT/internal/backup/backup.go:564`; owner: controller; started by the daily job "db-dump" `CT/cmd/controller/main.go:1345`. CUSTOMER. +2. Daily at W. W = household setting, else `DBDumpSchedule`, else `DefaultWindow` "02:30" `CT/internal/backupwindow/backupwindow.go:12`, decided in `EffectiveWindow` `CT/internal/backupwindow/backupwindow.go:86-94`. First leg. Zone: Europe/Budapest, hard-coded, `getBudapestLocation` `CT/internal/scheduler/scheduler.go:17-27`. +3. One database dump: 5 min limit `CT/internal/appbackup/dbdump.go:291`. Whole leg limit: NOT FOUND (grep `WithTimeout|WithDeadline` in backup.go gave only `CT/internal/backup/backup.go:983` and `CT/internal/backup/backup.go:1268`). Grows with number and size of databases and volumes. +4. Box off at W: `Missed` `CT/internal/nightchain/nightchain.go:197-218` sees it from the ledger and the catch-up runs it (block 4). The hub judges a missing dump at its 05:00 deadline `AR/07-backup-architecture.md:579-584`. +5. `withLeg` takes the leg lock `CT/cmd/controller/main.go:1321-1335`; `acquireRunning` refuses a second backup `CT/internal/backup/backup.go:1105-1113`; a job still running is skipped `executeJob` `CT/internal/scheduler/scheduler.go:354-364`. A running migration skips the leg and returns no error `CT/internal/backup/backup.go:565-568`. +6. Slot from the wall clock, waited on a monotonic timer `runDailyJob` `CT/internal/scheduler/scheduler.go:300-352`; fired over 60 min late = skipped `DailyLateLimit` `CT/internal/scheduler/scheduler.go:67`. See block 24 for the hour change. +7. Failure: `NotifyDBDumpFailed` `CT/cmd/controller/main.go:1339`. A made-up miss: one timeline line, never mailed `AR/07-backup-architecture.md:562-564`. A miss not made up: banner after 26 h `StaleAfter` `CT/internal/nightchain/banner.go:23`. +8. TEST `TestCatchUp_OffAtWThenOn_OneCatchUpAfter15Min` `CT/internal/nightchain/nightchain_test.go:92`; TEST `TestDaily_LateFireIsSkipped` `CT/internal/scheduler/scheduler_test.go:87`. + +### 2. Second-drive copy leg at W+60 min +1. `RunAllTier2` `CT/internal/backup/tier2.go:534` copies each app's data to the second drive; owner: controller; started by the daily job "tier2-backup" `CT/cmd/controller/main.go:1442`. CUSTOMER. +2. Daily at W+60 min, `tier2OffsetMin` `CT/internal/backupwindow/backupwindow.go:17`; no setting of its own. It does not wait for the dump leg: "NOTHING waits on the leg before it" `AR/09-update-architecture.md:2751-2757`. Zone as block 1. +3. One copy: 60 min, `tier2CopyTimeout` `CT/internal/backup/tier2_copy_lifecycle.go:40`, applied at `CT/internal/backup/tier2.go:828`. No limit for the whole leg found (same grep as block 1). Grows with app count and data size. +4. Catch-up as block 4 (the leg is in the ledger through `withLeg` `CT/cmd/controller/main.go:1438-1441`). The leg function always returns nil `CT/cmd/controller/main.go:1438-1441`, so the scheduler never sees a failure. +5. Leg lock `CT/cmd/controller/main.go:1321-1335`. Skipped while a migration runs `CT/internal/backup/tier2.go:538-541`. `acquireRunning` in tier2.go: NOT FOUND (grep `acquireRunning\(\)|TryLock` in tier2.go, 0 hits). A stray copy container is retried every minute `tier2RetryEvery` `CT/internal/backup/tier2_copy_lifecycle.go:172`. +6. As block 1: wall-clock slot, monotonic wait, 60 min late limit `CT/internal/scheduler/scheduler.go:67`. +7. Who is told of a failed copy: NOT FOUND in the wiring (grep `Notify[A-Za-z]+\(` in main.go lines 1300-1460 shows only the dump and the off-site notices, `CT/cmd/controller/main.go:1339` and `CT/cmd/controller/main.go:1446-1452`). Candidate, the answer may sit inside tier2.go. +8. SOURCE `CT/internal/backup/tier2.go:534`; catch-up part TEST `TestCatchUp_PowerCutMidChain_FinishesTheRest` `CT/internal/nightchain/nightchain_test.go:145`. + +### 3. Off-site push leg at W+105 min (with clean-up of old snapshots) +1. `RunOffboxBackup` `CT/internal/backup/offbox.go:1018` pushes dumps to the remote store, then runs retention; owner: controller; started by the daily job "offbox-backup" `CT/cmd/controller/main.go:1641-1643`. CUSTOMER. +2. Daily at W+105 min, `offboxOffsetMin` `CT/internal/backupwindow/backupwindow.go:18`. Retention keeps 7 daily, `keepDaily` `CT/internal/backup/offbox_window.go:47`. Zone as block 1. +3. Per app 2 h, `offboxBackupTimeout` `CT/internal/backup/offbox.go:39`, applied at `CT/internal/backup/offbox.go:1564`; SSH connect 10 s `CT/internal/backup/offbox.go:37`. Grows with data changed and line speed. One real night: 3 min 28 s `AR/09-update-architecture.md:2751-2757`. +4. Catch-up as block 4. No escrowed key, orphaned store or migration: skipped with an INFO log line and nil `CT/internal/backup/offbox.go:1040-1053`. +5. Another backup running: "skipped — another backup is running", returns nil, "the next scheduled run retries" `CT/internal/backup/offbox.go:1054-1057`. So a long second-drive leg can cost the night's off-site push (candidate). +6. As block 1. The clean-up window has a one-hour clock tolerance `offsiteGuardSkew` `CT/internal/backup/offbox_window.go:42`. +7. Failure: `NotifyOffboxBackupFailed` with a cleaned message `CT/cmd/controller/main.go:1446-1452`. The hub's daily deadline is the independent signal `CT/cmd/controller/main.go:1524-1528`. +8. SOURCE `CT/internal/backup/offbox.go:1040-1057`; DOC `AR/07-backup-architecture.md:455-457`. + +### 4. Missed-night catch-up and resume watch +1. `CatchUp.Evaluate` `CT/internal/nightchain/nightchain.go:281-299` runs the backup legs a box missed; owner: controller; started at controller start `CT/cmd/controller/main.go:1676` and by the resume watch `CT/cmd/controller/main.go:1677`. CUSTOMER. +2. 15 min after the trigger, `DefaultDelay` `CT/internal/nightchain/nightchain.go:62`; a leg due within 30 min is left alone, `leaveToNormal` `CT/internal/nightchain/nightchain.go:64`. Values decided by CC `AR/07-backup-architecture.md:574-577`. No setting. +3. As long as the legs it runs. It waits up to 2 h for a whole-guest backup, `quiesceWaitMax` `CT/internal/nightchain/nightchain.go:66`, then drops with a WARN `CT/internal/nightchain/nightchain.go:319`. +4. If the catch-up itself does not run: banner after 26 h (block 5) and the hub's longer lines, 48 h without a dump `AR/07-backup-architecture.md:579-584`. Two missed nights give one catch-up, not two. +5. One at a time: pending/running guard `CT/internal/nightchain/nightchain.go:288-292`; leg lock `CT/cmd/controller/main.go:1321-1335`. It never runs the update leg `CT/cmd/controller/main.go:1660-1664`. +6. `ResumeWatch` compares wall clock and monotonic clock each minute; a gap over 5 min triggers, `d > gap` `CT/internal/nightchain/nightchain.go:375`, gap set at `CT/cmd/controller/main.go:1681`. A forward clock correction over 5 min looks the same as a resume (candidate). "not measured live" `AR/07-backup-architecture.md:551-559`. +7. Household: one timeline line, never mailed `AR/07-backup-architecture.md:562-564`, event sent at `CT/internal/nightchain/nightchain.go:357-359`. Dropped catch-up: log line only `CT/internal/nightchain/nightchain.go:319`. +8. TEST `TestCatchUp_TwoMissedNights_One` `CT/internal/nightchain/nightchain_test.go:126`; TEST `TestResumeWatch_SeesASuspend` `CT/internal/nightchain/nightchain_test.go:217`; TEST `TestR871_CatchUpWiring` `CT/cmd/controller/r871_catchup_wiring_test.go:16`. + +### 5. Missed-backup banner (26 hours) +1. `ComputeBanner` `CT/internal/nightchain/banner.go:56-88` shows the household a banner when no night backup ran; owner: controller; computed when the page is drawn (no timer of its own). CUSTOMER. +2. Shown when the last backup is older than 26 h, `StaleAfter` `CT/internal/nightchain/banner.go:23`. After 5 missed days it suggests another hour, `suggestMinDays` `CT/internal/nightchain/banner.go:26`, `Suggest` `CT/internal/nightchain/banner.go:95-113`. +3. Instant; reads the ledger only `CT/internal/nightchain/banner.go:56-88`. +4. Nobody opens the page: nobody sees it. The hub's deadline check is the other channel `AR/07-backup-architecture.md:579-584`. +5. Not a job; no lock needed. Dismissed, it comes back after a new miss, design at `AR/07-backup-architecture.md:566-572`. +6. Age is wall-clock arithmetic `CT/internal/nightchain/banner.go:56-88`; a clock set back hides the banner for as long as the step (candidate). A test for a clock set back: NOT FOUND (grep `^func Test.*(Back|Clock)` in banner_test.go gave only the dismiss test). +7. The household only, on the page; no mail `AR/07-backup-architecture.md:566-572`. +8. TEST `TestBanner_ShownWithReasonAndSuggestion` `CT/internal/nightchain/banner_test.go:29`; TEST `TestBanner_DismissedThenBackAfterANewMiss` `CT/internal/nightchain/banner_test.go:59`. + +### 6. Whole-guest backup driver (stop apps, ask the host agent, start apps) +1. `Loop.Run` `CT/internal/quiesce/quiesce.go:191-206` stops the apps and asks the agent for the whole-guest backup; owner: controller drives, agent does the backup; started by `startQuiesceLoop` `CT/cmd/controller/main.go:4008`. CUSTOMER. +2. Polls every 5 min `CT/internal/quiesce/quiesce.go:149-151`; runs once per 24 h cadence `CT/internal/quiesce/quiesce.go:161-163`, only inside W+2h to W+6h, `gateOpenOffsetMin` `CT/internal/quiesce/quiesce.go:760`, `gateSpanMin` `CT/internal/quiesce/quiesce.go:761`. Zone `budapestLocation` `CT/internal/quiesce/quiesce.go:747-757`. +3. Apps stay down at most 30 min, `MaxQuiesce` default `CT/cmd/controller/main.go:3990`, setting `max_quiesce_duration` `CT/internal/config/config.go:132`. Manual run limit = that + 5 min `CT/internal/quiesce/quiesce.go:430`. Grows with guest disk size. +4. Outside the window it still runs when the last one is older than cadence + 24 h (the 48 h valve) `CT/internal/quiesce/quiesce.go:801-818`. Apps left stopped by a crash are started by `Recover` `CT/internal/quiesce/quiesce.go:177-188`, called at `CT/cmd/controller/main.go:4007`. +5. Single-flight `TryLock` `CT/internal/quiesce/quiesce.go:213-216`; waits for a catch-up `CT/internal/quiesce/quiesce.go:250-253`; waits for the update leg `CT/internal/quiesce/quiesce.go:271-274`, never past W+5h30 `legStepGraceMin` `CT/internal/quiesce/quiesce.go:935`. +6. Gate is minute-of-day arithmetic on the wall clock `CT/internal/quiesce/quiesce.go:801-818`; the poll is a monotonic ticker `CT/internal/quiesce/quiesce.go:191-206`. Hour-change night: NOT FOUND a test (grep `DST|Budapest|SpringForward|Daylight` in `^func Test` lines, 0 hits). +7. First failure notifies once `noteTierFailure` `CT/internal/quiesce/quiesce.go:283-298`; retry back-off 15 min to 4 h `CT/internal/quiesce/breaker.go:47`, `CT/internal/quiesce/breaker.go:54`; busy agent: retry 15 min, alarm after 3 h `CT/internal/quiesce/contention.go:53`, `CT/internal/quiesce/contention.go:64`. +8. TEST `TestScheduledRunAllowed` `CT/internal/quiesce/quiesce_window_test.go:23`; TEST `TestRunOnce_ValveRunsOutsideWindow` `CT/internal/quiesce/quiesce_window_test.go:94`; TEST `TestBreaker_BackoffSchedule` `CT/internal/quiesce/breaker_test.go:218`. + +### 7. Automatic app-update leg (after the off-site push) +1. `RunUpdateLeg` `CT/internal/stacks/unattended.go:230` presses one update step per app; owner: controller; started by `chainUpdateLeg` when the off-site half ends, on every path `CT/cmd/controller/main.go:1641-1643`. CUSTOMER. +2. Each night after the off-site leg; a box with no off-site tier runs it at W+105 min `AR/07-backup-architecture.md:459-465`. One step per app per night, switch default ON `AR/09-update-architecture.md:2806-2817`. +3. No new step at or after W+5h: `LegDeadline` `CT/internal/stacks/unattended.go:212-216`, `UpdateLegStopOffsetMin` `CT/internal/backupwindow/backupwindow.go:27`; check at `CT/internal/stacks/unattended.go:374-375`. Grows with the number of apps that have an update. +4. Missed night: no catch-up by design `CT/cmd/controller/main.go:1660-1664`; the rest "waits for the next night" `CT/internal/stacks/unattended.go:374-375`. Controller restarted mid-leg: the rest of the night is lost (R-686) `AR/09-update-architecture.md:2815-2817`. +5. A passing refusal (busy, updating) is retried once after 2 min, `RetryWait` `CT/internal/stacks/unattended.go:167-171`, rule at `AR/09-update-architecture.md:2882-2883`. The whole-guest gate waits for the leg `CT/internal/quiesce/quiesce.go:271-274`. +6. Deadline is a wall-clock time in `time.Local` when no location is given `CT/internal/stacks/unattended.go:212-216`, while the scheduler uses Europe/Budapest `CT/internal/scheduler/scheduler.go:17-27` (candidate: two sources of zone). A late daily fire over 60 min is skipped, so no noon updates `AR/07-backup-architecture.md:551-559`. +7. NOT FOUND in this read for who is told when the leg does not run (grep `update-leg` gave the log line `CT/internal/stacks/unattended.go:374-375` only). +8. TEST `TestChainUpdateLeg_EveryPath` `CT/cmd/controller/update_leg_wiring_test.go:19`; TEST `TestLegDeadlineAndGateShareW5h` `CT/internal/quiesce/d20_update_leg_test.go:78`. + +### 8. Off-site integrity check at 06:00 +1. Daily job "offsite-integrity" `CT/cmd/controller/main.go:1724` checks the remote store's data; owner: controller. CUSTOMER. +2. Fires daily at 06:00 (fixed, no setting for the hour) but works only when due: last good check older than 7 days, `defaultIntegrityMaxAgeDays` `CT/internal/backup/offbox_integrity.go:81`, `IntegrityDue` `CT/internal/backup/offbox_integrity.go:221`. Zone as block 1. +3. Limit 30 min, `integrityCheckTimeout` `CT/internal/backup/offbox_integrity.go:63`; a notice when slower than 5 min `CT/internal/backup/offbox_integrity.go:78`. Grows with store size and line speed. +4. Box off at 06:00: caught up at the next 06:00 by due-ness `CT/internal/backup/offbox_integrity.go:212-220`; also run after a catch-up `CT/cmd/controller/main.go:1665-1668`. +5. Skipped when "a backup or restore is already running" `CT/internal/backup/offbox_integrity.go:298`; then it waits for the next 06:00 (source comment `CT/cmd/controller/main.go:1710-1723`). +6. Due-ness compares wall-clock stamps `CT/internal/backup/offbox_integrity.go:231`; a never-checked or unreadable stamp counts as due `CT/internal/backup/offbox_integrity.go:223-229`. Slot behaviour as block 24. +7. NOT FOUND in this read (the job body `CT/cmd/controller/main.go:1724-1729` was read; the notify path inside offbox_integrity.go was not). +8. SOURCE `CT/internal/backup/offbox_integrity.go:212-232`; TEST `TestR1189d_TheChecksRunAfterTheCatchUpLegs` `CT/internal/nightchain/r1189d_test.go:13`. + +### 9. Off-site proof restore at 05:30 +1. Daily job "offsite-proof" `CT/cmd/controller/main.go:1746` restores one due snapshot to prove it opens, `ProveOffboxUnit` `CT/internal/backup/offbox_proof.go:140`; owner: controller. CUSTOMER. +2. Daily at 05:30, fixed, no setting `CT/cmd/controller/main.go:1746`. One app per night in rotation (comment `CT/internal/backup/offbox_proof.go:265`). Zone as block 1. +3. Limit 10 min, `proofRestoreTimeout` `CT/internal/backup/offbox_proof.go:80`. Grows with the size of the one unit. +4. Box on for the backup legs but off at 05:30: "that day's proof is not made up" `CT/cmd/controller/main.go:1660-1664`. Made up only together with a missed backup leg `CT/cmd/controller/main.go:1665-1668`. +5. Skipped when "a backup or restore is already running" `CT/internal/backup/offbox_proof.go:149`. With W = 02:30 the whole-guest gate is open from 04:30 `AR/09-update-architecture.md:2751-2757`, so 05:30 can fall inside it (candidate). +6. As block 24. +7. NOT FOUND in this read (grep `Notify` near `CT/cmd/controller/main.go:1746-1749`, 0 hits there). +8. SOURCE `CT/cmd/controller/main.go:1660-1664`; TEST `TestC5_TheWholeGuestBackupDoesNotWaitForTheChecks` `CT/internal/nightchain/r1189d_test.go:51`. + +### 10. Hub report every 15 minutes (plus push on change and hanging wait) +1. Job "hub-report" `CT/cmd/controller/main.go:1835` sends the box state to the hub; owner: controller. The hub's answer carries the version floor and config `CT/cmd/controller/main.go:1233-1290`. CUSTOMER and OPERATOR. +2. Every `PushInterval`, default "15m" `CT/internal/config/config.go:416`, fallback `CT/cmd/controller/main.go:1831-1834`. Also 5 s after start `CT/cmd/controller/main.go:1932-1936`, and on change with a 15 s floor, `triggerMinInterval` `CT/internal/report/trigger.go:30`. No time zone. +3. HTTP limit 30 s `CT/internal/report/pusher.go:85`; 3 tries, 5 s apart `CT/internal/report/pusher.go:114-117`. The wait connection holds up to 300 s `CT/internal/report/waiter.go:35`. +4. A failed push is not queued: "next cycle reconciles" `CT/internal/report/trigger.go:98`. The hub notices silence (hub side, not read here). +5. A report still running is skipped `CT/internal/scheduler/scheduler.go:354-364`. Bursts are merged after a 2 s quiet window `CT/internal/report/trigger.go:27`. +6. Monotonic ticker `CT/internal/scheduler/scheduler.go:281-298`; a wrong wall clock does not change the rhythm, only the stamps inside the report. +7. Waiter back-off 5 s to 5 min `CT/internal/report/waiter.go:36-37`. Who is told on the box that pushes fail: NOT FOUND (only `LastAttempt` is stored `CT/internal/report/pusher.go:109-111`). +8. TEST `TestTrigger_BurstCoalescesTrailingEdge` `CT/internal/report/trigger_test.go:114`; TEST `TestWaiter_FiresOnceOnGenChange` `CT/internal/report/waiter_test.go:60`. + +### 11. Controller self-update: check, floor, nightly auto +1. `CheckForUpdate` `CT/internal/selfupdate/updater.go:153` looks for a newer controller image; `MaybeAutoUpdate` `CT/internal/selfupdate/updater.go:612` moves a box below the floor. Owner: controller, the agent swaps the image. CUSTOMER and OPERATOR. +2. Check every `CheckInterval`, default "6h" `CT/internal/config/config.go:417`, and 25 s after start `CT/cmd/controller/main.go:1981-1989`. Floor: after every hub report `CT/cmd/controller/main.go:1249-1252`. Auto: daily at `AutoUpdateTime`, default "04:30" `CT/internal/config/config.go:419`, only if `AutoUpdate` `CT/internal/config/config.go:278`. +3. Registry call limit 15 s `CT/internal/selfupdate/updater.go:245`. Limit of the swap itself: NOT FOUND in this read (grep `time\.(Second|Minute|Hour)` in updater.go gave one hit). +4. No catch-up needed for the floor: it is tried again on each report until done, but one attempt per floor value `CT/internal/selfupdate/updater.go:656-661`. A failed attempt at a floor is not repeated `CT/internal/selfupdate/updater.go:604-606`. +5. `updateRunning` flag `CT/internal/selfupdate/updater.go:520-524`; refused while a backup runs `CT/internal/selfupdate/updater.go:533-536` or an app update is in flight `CT/internal/selfupdate/updater.go:540-544`. Default 04:30 equals gate-open time at W = 02:30 (candidate). +6. Check: monotonic ticker. Auto: daily slot as block 24. Default for `AutoUpdate`: NOT FOUND in the defaults block (grep `AutoUpdate ` in config.go: only the field `CT/internal/config/config.go:278`). +7. Auto-update refused: WARN log only `CT/cmd/controller/main.go:1879-1881`. A version the floor cannot compare: WARN once `CT/internal/selfupdate/updater.go:639`. +8. TEST `TestMaybeAutoUpdate_NoFlap` `CT/internal/selfupdate/floor_test.go:93`; TEST `TestMaybeAutoUpdate_BelowFloor_UpdatesToFloor` `CT/internal/selfupdate/floor_test.go:11`. + +### 12. Catalog sync (app templates from git) +1. `Syncer.Start` `CT/internal/sync/sync.go:125` pulls the app catalog; owner: controller; started at controller start. CUSTOMER. +2. Once at start `CT/internal/sync/sync.go:140-143`, then every `SyncInterval`, default "15m" `CT/internal/config/config.go:399`; a bad value falls back to 15 min `CT/internal/sync/sync.go:131-135`. No time zone. +3. Time limit for one sync: NOT FOUND in the lines read (`CT/internal/sync/sync.go:124-188`; `doSync` body not read). +4. Missed sync: the next tick does it. Empty repo address: sync is off with one WARN `CT/internal/sync/sync.go:126-129`. Who notices a long failure: NOT FOUND. +5. Manual sync is refused while one runs or within 30 s of the last `CT/internal/sync/sync.go:175-185`. The periodic tick calls `doSync` directly without that check `CT/internal/sync/sync.go:153-155` (candidate: read `doSync` before judging). +6. Monotonic ticker `CT/internal/sync/sync.go:146`; own ticker, not the shared scheduler, so it has no "still running" skip from `CT/internal/scheduler/scheduler.go:354-364`. +7. Log line per sync only `CT/internal/sync/sync.go:155`. +8. SOURCE `CT/internal/sync/sync.go:124-159`. + +### 13. System health check and base-stack self-heal +1. Job "system-health" `CT/cmd/controller/main.go:981-1006` checks disk, memory and base services, and puts the base stack back; owner: controller. CUSTOMER. +2. Every `SystemHealthInterval`, default "5m" `CT/internal/config/config.go:409`, fallback `CT/cmd/controller/main.go:977-980`; env override `CT/internal/config/config.go:453`. First run one interval after start `CT/internal/scheduler/scheduler.go:281-298`. +3. Time limit: NOT FOUND in the job body `CT/cmd/controller/main.go:981-1006`. +4. No catch-up; next tick. A controller that is down checks nothing; the hub's missing reports are the outside sign (block 10). +5. Skipped while the previous run is still going `CT/internal/scheduler/scheduler.go:354-364`. +6. Monotonic ticker; not sensitive to wall clock. +7. `NotifyHealthChange` on a change of state `CT/cmd/controller/main.go:1004`. +8. SOURCE `CT/cmd/controller/main.go:981-1006`. + +### 14. Dead-app scan every 30 seconds +1. Job "deadapp-check" `CT/cmd/controller/main.go:1086-1092` finds apps that should run and do not; owner: controller. CUSTOMER. +2. Every 30 s; silent for 90 s after start, about 15 min after a crash boot `CT/cmd/controller/main.go:1087-1088`. +3. Time limit: NOT FOUND (job body `CT/cmd/controller/main.go:1086-1143` has no context limit). Grows with app count. +4. Next tick; no catch-up needed. +5. Skipped if still running `CT/internal/scheduler/scheduler.go:354-364`. It reads the whole-guest loop state so a planned stop is not an alarm `CT/cmd/controller/main.go:1090`. +6. Boot grace uses `time.Now()` `CT/cmd/controller/main.go:1087`; a clock stepped back after start would lengthen the silence (candidate, no test found). +7. Alert on the page via `SetDeadAppAlerts` `CT/cmd/controller/main.go:1091`. +8. SOURCE `CT/cmd/controller/main.go:1086-1092`. + +### 15. Disk-fill watch +1. `fillWatcher.Check` watches how full the drives are; owner: controller; three starters `CT/cmd/controller/main.go:1804-1805`. CUSTOMER. +2. Every 10 min, `fillWatchInterval` `CT/cmd/controller/main.go:2690`; daily at 03:30 `CT/cmd/controller/main.go:1804`; once 90 s after start, `fillWatchStartupDelay` `CT/cmd/controller/main.go:2685`. Cadence stated at `CT/cmd/controller/main.go:1772`. +3. Time limit: NOT FOUND (the callbacks ignore the context `CT/cmd/controller/main.go:1804-1805`). +4. Next 10-minute tick. +5. The 03:30 slot and the interval job are two jobs with two names `CT/cmd/controller/main.go:1804-1805`; a lock between them: NOT FOUND in the wiring (the fillwatch package was not read). +6. Interval is monotonic; the 03:30 slot as block 24. +7. NOT FOUND in this read (fillwatch package not read). +8. TEST `TestFillWatchRunsOnAnInterval` `CT/cmd/controller/r363_fillwatch_interval_test.go:17`. + +### 16. Drive health check, hourly +1. Job "disk-health-check" `CT/cmd/controller/main.go:2226` asks the agent for drive health; owner: controller reads, agent measures. CUSTOMER. +2. Every 1 h; reason for hourly, not 6 h: `CT/cmd/controller/main.go:2219-2224`. Only when the agent channel is configured `CT/cmd/controller/main.go:2225`. +3. One fetch measured about 0.8 s `CT/cmd/controller/main.go:2219-2224`. Limit: NOT FOUND. +4. Next hour; first run one hour after start `CT/internal/scheduler/scheduler.go:281-298`. +5. Skipped if still running `CT/internal/scheduler/scheduler.go:354-364`. +6. Monotonic ticker. +7. NOT FOUND in this read (`RunDiskHealthCheck` body not read). +8. SOURCE `CT/cmd/controller/main.go:2219-2226`. + +### 17. Agent channel health, every 60 seconds +1. Job "agent-channel-health" `CT/cmd/controller/main.go:2289` probes the controller-to-agent channel; owner: controller. OPERATOR. +2. Every 60 s. A passing fault needs 2 misses in a row, `debounceThreshold` `CT/internal/channelhealth/checker.go:29`. +3. Probe limit 15 s, `probeTimeout` `CT/internal/channelhealth/checker.go:32`. +4. Next minute. +5. Skipped if still running `CT/internal/scheduler/scheduler.go:354-364`. An alert marker file survives a restart `CT/cmd/controller/main.go:2288`. +6. Monotonic ticker. +7. Sink = notifier and page alert `CT/cmd/controller/main.go:2286`. +8. TEST `TestDebounce_TransientBlipSuppressed` `CT/internal/channelhealth/checker_test.go:119`. + +### 18. Held-app report to the hub +1. `Reporter.Tick` `CT/internal/holdreport/holdreport.go:89` tells the hub which apps are held; owner: controller; job "hold-report" `CT/cmd/controller/main.go:1081-1084`. OPERATOR. +2. Looks every minute; each hold is sent again once a day, `ResendEvery` `CT/internal/holdreport/holdreport.go:36`, and after each start `CT/cmd/controller/main.go:1071-1073`. +3. Instant; reads the hold store. +4. Next minute. +5. Skipped if still running `CT/internal/scheduler/scheduler.go:354-364`. +6. `Tick` gets `time.Now()` `CT/cmd/controller/main.go:1082`; the daily resend is wall-clock arithmetic (candidate: a clock set back delays the resend). +7. The hub, by event `CT/cmd/controller/main.go:1074`. +8. TEST `TestD64_TheHoldReporterTellsTheHubAfterTheClear` `CT/cmd/controller/holdclear_test.go:333`. + +### 19. Abandoned off-site store sweep at 05:10 +1. Daily job "offsite-abandon-sweep" `CT/cmd/controller/main.go:1693` runs `AbandonSweep` `CT/internal/backup/offbox_abandon.go:214`; owner: controller. CUSTOMER. +2. Daily at 05:10, fixed. The due date is counted in UTC days `CT/internal/backup/offbox_abandon.go:366`, the slot in Europe/Budapest `CT/internal/scheduler/scheduler.go:17-27`. +3. NOT FOUND (body of `AbandonSweep` not read). +4. Box off at 05:10: next day; no catch-up (it is not a ledger leg, see `CT/cmd/controller/main.go:1660-1664`). +5. NOT FOUND (grep `acquireRunning\(\)|TryLock` in offbox_abandon.go, 0 hits). +6. As block 24. +7. NOT FOUND in this read. +8. SOURCE `CT/cmd/controller/main.go:1693`. + +### 20. Restore-copy sweep and conversion-copy release +1. `SweepUnusedRestoreCopies` `CT/internal/backup/restore_copy_sweep.go:36` deletes old kept copies; job "restore-copy-sweep" `CT/cmd/controller/main.go:1414`; job "conversion-copy-release" `CT/cmd/controller/main.go:1407`. CUSTOMER. +2. Sweep every 6 h; release every 1 h. Age line: the wiring comment says 14 days `CT/cmd/controller/main.go:1412`, the constant says 30 days, `restoreCopyUnusedAge` `CT/internal/backup/restore_copy_sweep.go:29` (candidate: comment and constant differ). +3. NOT FOUND. +4. Next tick; first run 6 h after start `CT/internal/scheduler/scheduler.go:281-298`, so a box on for under 6 h a day never sweeps (candidate). +5. Skipped if still running `CT/internal/scheduler/scheduler.go:354-364`. +6. Age is wall-clock arithmetic on `now` `CT/internal/backup/restore_copy_sweep.go:36`. +7. NOT FOUND. +8. SOURCE `CT/internal/backup/restore_copy_sweep.go:29`. + +### 21. Session clean-up (dashboard sign-ins) +1. `cleanupSessions` `CT/internal/web/auth.go:369` removes expired sign-in sessions; owner: controller web server. CUSTOMER. +2. Every 15 min `CT/internal/web/auth.go:370`. A session lives 7 days, `sessionMaxAge` `CT/internal/web/auth.go:30`. Family sessions 30 days, `SessionLife` `CT/internal/family/family.go:42`; share cookie 30 days `CT/internal/web/share.go:31`. +3. Instant; grows with session count. +4. Not run: expired sessions stay in the file. Whether an expired session is refused at use: NOT FOUND in the lines read (`CT/internal/web/auth.go:355-395`). +5. `sessionsMu` lock `CT/internal/web/auth.go:377`. +6. Expiry compares the wall clock, `now.After(sess.expiresAt)` `CT/internal/web/auth.go:381`; a clock set forward signs everyone out, set back keeps sessions longer (candidate). +7. Log line only `CT/internal/web/auth.go:392`. +8. SOURCE `CT/internal/web/auth.go:369-393`. + +### 22. Upload and CPU watch, torrent limit rule, line-test hold +1. `startUpwatch` `CT/cmd/controller/upwatch_wiring.go:136` runs the minute watcher and a 15-minute rule that holds the torrent app for the line test; owner: controller. CUSTOMER. +2. Watcher every 1 min `CT/cmd/controller/upwatch_wiring.go:139`; rule every 15 min `CT/cmd/controller/upwatch_wiring.go:141`. Baseline: 7 days of data, cached 24 h `CT/cmd/controller/upwatch_wiring.go:124-127`. +3. Hold call limit 30 s, `holdTimeout` `CT/internal/upwatch/linehold.go:79`; drain wait at most 90 s `CT/internal/upwatch/linehold.go:216`. +4. A limit left at the test value by a stopped controller is put back at start, `RecoverHold` `CT/cmd/controller/upwatch_wiring.go:137`. +5. Own tickers; no scheduler skip. Lock between the two loops: NOT FOUND in the lines read. +6. Monotonic tickers `CT/internal/upwatch/watch.go:151`. +7. NOT FOUND in this read (`emit` `CT/internal/upwatch/watch.go:382` not followed). +8. SOURCE `CT/cmd/controller/upwatch_wiring.go:134-152`. + +### 23. Old image clean-up after start +1. `RunImageRetentionOnceUntilDone` `CT/internal/stacks/image_retention.go:399` deletes app images no app uses; owner: controller; started about 3 min after start `CT/cmd/controller/main.go:2048-2069`. CUSTOMER. +2. Once per start, retried every 2 min up to 30 times `CT/cmd/controller/main.go:2056`. Also after an update or a remove (test names below). +3. Up to about 60 min of retries (30 x 2 min) `CT/cmd/controller/main.go:2056`. +4. A box that never restarts and never updates runs no sweep (candidate). +5. No pass while an update runs: TEST `TestImageRetention_NoPassWhileAnUpdateRuns` `CT/internal/stacks/image_retention_test.go:238`. +6. Monotonic waits. +7. Log only; an unreadable keep set deletes nothing `CT/internal/stacks/image_retention.go:221`. +8. TEST `TestImageRetention_KeepsRunningAndPreviousDeletesOlder` `CT/internal/stacks/image_retention_test.go:111`. + +### 24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above) +1. `Daily` `CT/internal/scheduler/scheduler.go:131-157` and `Every` `CT/internal/scheduler/scheduler.go:105-127` carry every named job; owner: controller. CUSTOMER. +2. Daily slots are HH:MM in Europe/Budapest, hard-coded; if the zone data cannot load it silently uses UTC `CT/internal/scheduler/scheduler.go:17-27`. Interval jobs do not run at start: first run after one interval `CT/internal/scheduler/scheduler.go:281-298`. +3. No time limit of its own; shutdown waits 30 s `CT/internal/scheduler/scheduler.go:264`. A panic in a job is caught `CT/internal/scheduler/scheduler.go:438-447`. +4. A daily job has no catch-up in the scheduler; a fire over 60 min late is skipped `CT/internal/scheduler/scheduler.go:344-347`. Only the three backup legs are made up (block 4). +5. One run per job at a time: "still running" skip `CT/internal/scheduler/scheduler.go:354-364`. No rule between different jobs here. +6. Next slot by calendar day, not +24 h `CT/internal/scheduler/scheduler.go:499-519`. Spring night: 02:30 does not exist; autumn night: 02:30 comes twice. Test for either: NOT FOUND (grep `DST|Budapest|SpringForward|Daylight` in `^func Test`, 0 hits). A clock set back while waiting: the timer still fires on monotonic time. +7. A skipped late fire is a log line only `CT/internal/scheduler/scheduler.go:344-347`. +8. TEST `TestDaily_LateFireIsSkipped` `CT/internal/scheduler/scheduler_test.go:87`; TEST `TestUpdateDaily_GoroutineConsumesReschedule` `CT/internal/scheduler/scheduler_test.go:62`. + +## Tiny +- Status refresh every 10 s `CT/cmd/controller/main.go:950`. +- Stack scan every 2 min `CT/cmd/controller/main.go:953`. +- Docker socket watch every 15 s and socket users every 5 min `CT/cmd/controller/main.go:960-962`. +- Health probes every 10 s `CT/cmd/controller/main.go:971`. +- After-unclean-boot repair, every 30 s until done, own `TryLock` `CT/cmd/controller/main.go:1064-1065`. +- Debug ring spill every 30 s `CT/cmd/controller/main.go:1148-1150`. +- Off-site credential retry every 5 min `CT/cmd/controller/main.go:1375`. +- Backup cache refresh every 5 min `CT/cmd/controller/main.go:1398-1402`. +- Metrics sample every 60 s `CT/internal/metrics/collector.go:53`. +- Metrics prune daily 04:00, keeps 30 days `CT/cmd/controller/main.go:1754`. +- Asset sync daily at `SyncSchedule`, default "05:00" `CT/internal/config/config.go:425`. +- Geo-restriction verify every 6 h `CT/cmd/controller/main.go:2102`. +- Setup-gate loop every 20 s `CT/cmd/controller/main.go:2045`. +- Busy marker written every 15 s `CT/internal/busymark/busymark.go:29`. +- Startup report: 5 s after start `CT/cmd/controller/main.go:1932-1936`. +- Stray second-drive copy container retry every minute `CT/internal/backup/tier2_copy_lifecycle.go:172`. +- App wait after install: 10 min `CT/cmd/controller/main.go:2194`. +- Family sign-in form lives 1 h `CT/internal/web/family_gate.go:56`. +- Stale pointers inside the design text: the legs are cited at old main.go lines `AR/07-backup-architecture.md:455-457` (candidate, hygiene). + +## Not found +- Log rotation: NOT FOUND in controller Go code. Grep `rotat|lumberjack|max-size|MaxSizeMB` found only the config fields `CT/internal/config/config.go:291-292` and their defaults `CT/internal/config/config.go:422-423`; no reader of them. +- Share-link expiry sweep: NOT FOUND. Grep `expire|expiry|ExpiresAt` in `internal/web/share*.go` gave no sweep; only the 30-day cookie `CT/internal/web/share.go:31`. +- A test for the hour-change nights or a named time zone: NOT FOUND (grep `DST|Budapest|SpringForward|Daylight` over `^func Test` in cmd and internal, 0 hits). +- Who is told when the second-drive copy, the proof, the integrity check, the fill watch or the drive health check fails: not read to the end; see line 7 of blocks 2, 8, 9, 15, 16. +- Not read at all: `CT/internal/bootrecon/bootrecon.go:147`, `CT/internal/bootstrap/waiting.go:95`, `CT/internal/offsiteapply/offsiteapply.go:137`, `CT/internal/web/operator_reboot_notice.go:78`, `CT/internal/web/handler_export_upload.go:204`, `CT/internal/web/media_events.go:97`, `CT/internal/notify/notifier.go:324`. diff --git a/documentation/audits/clock-list-2026-11-11/evidence/helpers/helper-clock-hub.md b/documentation/audits/clock-list-2026-11-11/evidence/helpers/helper-clock-hub.md new file mode 100644 index 00000000..0032f070 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/helpers/helper-clock-hub.md @@ -0,0 +1,333 @@ +# Clock list: hub jobs and timers outside product code (helper result, candidates only) + +### 1. Box liveness: "your server cannot be reached" +1. CUSTOMER. The hub's 60 s sweep runs `stalenessChecker.Check` and `hostStaleness.Check` `E/hub/cmd/hub/main.go:935-936`; they raise `host_stale`, `host_down`, `host_recovered` `HUB/monitor/host_staleness.go:207-221`. Owner: hub pod. Merged here: the controller-report checker and the host-report checker. +2. Every 60 s `E/hub/cmd/hub/main.go:927`. Stale after `stale_threshold`, default `"30m"` `E/hub/cmd/hub/main.go:1027`, deployed 45 m `E/manifests/hub.yaml:88`; down after 2x `HUB/monitor/host_staleness.go:39`. No time zone: ages are durations `HUB/monitor/host_staleness.go:131`. +3. No time limit; the checkers run one after another in one goroutine `E/hub/cmd/hub/main.go:935-962`. One store read of all hosts per pass `HUB/monitor/host_staleness.go:113`, then a loop per host; grows linearly with boxes. Measured duration: NOT FOUND (grep `time.Since(start)` in `E/hub/cmd/hub/main.go:927-965`). +4. Pod down: nothing is judged. At start a box already silent is left unseeded and the first pass alarms once `HUB/monitor/host_staleness.go:59-60`, unless that alarm was already raised `HUB/monitor/host_staleness.go:62-68`. The controller checker seeds stale/down silently `HUB/monitor/staleness.go:86-91` (candidate: differs from the host checker). +5. One pod, `replicas: 1` `E/manifests/hub.yaml:122`; state under a mutex `HUB/monitor/host_staleness.go:119-120`. Events fire on transitions only `HUB/monitor/host_staleness.go:170-172`. A household power-off holds alarms `HUB/monitor/host_staleness.go:134-136` for `DeliberateOffHold` `HUB/store/deliberateoff.go:18`. +6. Judged on the hub's receive time, not the box clock: `last_report_at = datetime('now')` `HUB/store/store.go:3916`. A wrong box clock cannot hide silence. A hub clock jump moves every age at once `HUB/monitor/host_staleness.go:131`; DST has no effect (durations). +7. Operator: every edge; liveness events bypass the 1 h quiet hour, 5 min dedupe `HUB/notify/dispatcher.go:609-612`. Household: at most once per 7 days, `householdLivenessQuiet` `HUB/notify/dispatcher.go:1018`. If the hub itself is down nobody is told by this job. +8. TEST `TestHostStaleness_R1001_RestartDoesNotReMail` `HUB/monitor/r1001_restart_test.go:56`; TEST `TestR873_HouseholdHearsAnOutageAtMostWeekly` `HUB/notify/r873_liveness_weekly_test.go:17`; DOC `AR/05-hub-architecture.md:88-118`. + +### 2. Missed-backup deadline check (05:00) +1. CUSTOMER. `scheduleDaily(ctx, "deadline-check", "05:00", ...)` `E/hub/cmd/hub/main.go:986-988` runs `CheckBackupDeadlines` `HUB/monitor/deadline.go:409`; it raises `expected_backup_missed` `HUB/monitor/deadline.go:539-544`. Owner: hub pod. +2. Once a day at 05:00 Budapest wall-clock; the slot is a literal in `E/hub/cmd/hub/main.go:986`, no setting. Zone chosen in `scheduleDaily` `E/hub/cmd/hub/main.go:1072`. Limits: `backupStaleAfter` 26 h `HUB/monitor/deadline.go:29`, off-site `offsiteBackupStaleAfter` 8 d `HUB/monitor/deadline_tiers.go:28`. +3. No time limit; runs inline in the scheduler goroutine `E/hub/cmd/hub/main.go:1090`. Loops over every customer and reads up to 7 days of evidence, `backupEvidenceLookback` `HUB/monitor/deadline.go:36`; grows with boxes times reports. Only a summary line is logged `HUB/monitor/deadline.go:569-570`; duration NOT FOUND (grep `duration|took`). +4. Pod down at 05:00: the day's check is skipped, no catch-up; `nextDailyRun` moves to the next calendar day `E/hub/cmd/hub/main.go:1095-1112` and `last` lives in memory `E/hub/cmd/hub/main.go:1069-1093`. Nobody notices a skipped check (candidate). +5. Cannot run twice for one slot: the `last` guard in `nextDailyRun` `E/hub/cmd/hub/main.go:1095-1112`. No lock against the 60 s sweep; they are separate goroutines `E/hub/cmd/hub/main.go:927` and `E/hub/cmd/hub/main.go:986`. A down box is judged on longer lines, 48 h / 72 h `HUB/monitor/deadline.go:340-343`. +6. Freshness is the box's own backup timestamp against the hub's now `HUB/monitor/deadline.go:243`, parsed by `parseBackupTime` `HUB/monitor/deadline.go:290-299`. A box clock that runs ahead would look fresh (candidate; guard NOT FOUND, grep `future|skew`). "Since midnight" is Budapest midnight `HUB/monitor/deadline.go:417-419`. +7. Household and operator through the dispatcher on `expected_backup_missed` `HUB/monitor/deadline.go:539-544`; at the 05:00 run only. A box that never bound is "unknown", not alarmed `HUB/monitor/deadline.go:486-494`; disabled / deliberately-off are skipped `HUB/monitor/deadline.go:432-442`. +8. TEST `TestR872_DownEveryNightRaisesTheMissedAlarms` `HUB/monitor/r872_down_box_test.go:57`; TEST `TestR1183_TheSameSlotNeverRunsTwice` `E/hub/cmd/hub/r1183_dst_schedule_test.go:82`. A skipped day: NOBODY KNOWS. + +### 3. Mail cooldowns and the weekly outage mail +1. CUSTOMER. Not a job but a clock rule inside every mail: `processCustomer` `HUB/notify/dispatcher.go:1020-1098` and `processOperator` `HUB/notify/dispatcher.go:626-696` decide whether an alarm is mailed now or swallowed. Owner: hub dispatcher. +2. Operator: 1 h, liveness 5 min `HUB/notify/dispatcher.go:609-612`, fill `storageFillCooldown` 6 h `HUB/notify/dispatcher.go:573`. Household: default 6 h `HUB/notify/dispatcher.go:1065-1069`; outage mails weekly `HUB/notify/dispatcher.go:1054-1062`. A new box's first hour is quiet, `firstHourOfABox` `HUB/notify/dispatcher.go:1281`. +3. Per event, no loop. HTTP client limit 10 s `HUB/notify/dispatcher.go:86`. The cooldown maps grow with customers times event types; an entry is removed only on a mail hold `HUB/notify/dispatcher.go:684` or by `forgetCustomerCooldown` `HUB/notify/dispatcher.go:1180-1183`; a prune by age: NOT FOUND (grep `delete(d.`). +4. Pod restart: the cooldown maps are in memory, so quiet periods restart from zero; only the 7-day household rule is read from the persisted notification log `HUB/notify/dispatcher.go:1054-1062`. Design text: `AR/08-alarm-ladder.md:435-439`. +5. The operator cooldown is armed before the send `HUB/notify/dispatcher.go:674`, so two alarms close together collapse to one; a suppression is recorded `HUB/notify/dispatcher.go:665-672`. A MAIL-HOLD drops the mail `HUB/notify/dispatcher.go:680-687`. +6. Cooldowns use the hub clock only. A hub clock jump back would stretch a quiet period, forward would end it early; handling: NOT FOUND (grep `monotonic` in the dispatcher file). +7. A swallowed operator alarm leaves a "suppressed" row `HUB/notify/dispatcher.go:665-672`. A failed household mail is logged and NOT retried `HUB/notify/dispatcher.go:1091-1094` (candidate: only operator mails are retried, see block 4). +8. TEST `TestOperatorCooldown_NodeLivenessBypassesQuietHour` `HUB/notify/node_liveness_cooldown_test.go:17`; TEST `TestR873_HouseholdHearsAnOutageAtMostWeekly` `HUB/notify/r873_liveness_weekly_test.go:17`; DOC `AR/08-alarm-ladder.md:165-168`. + +### 4. Retry of a failed operator mail +1. OPERATOR. `retryOperatorEmail` `HUB/notify/dispatcher.go:99-104` re-sends an operator mail whose send failed; started by the failure branch of `processOperator` `HUB/notify/dispatcher.go:688-692`. +2. After 1, 5 and 15 minutes: `operatorRetryDelays` `HUB/notify/dispatcher.go:76`, armed with `afterFn = time.AfterFunc` `HUB/notify/dispatcher.go:92`. No setting. Durations, no time zone. +3. Each try is bounded by the 10 s client timeout `HUB/notify/dispatcher.go:86`. One timer per failed mail; grows only with failures. +4. Pod restart inside the 21 minutes: the pending timers are in memory `HUB/notify/dispatcher.go:92` and are lost; a catch-up at start: NOT FOUND (grep `retryOperatorEmail` under hub/cmd). After the third failure: an ERROR log line only `HUB/notify/dispatcher.go:101`. +5. The cooldown for that alarm was armed before the first send `HUB/notify/dispatcher.go:674`, so a repeat alarm in the same hour does not mail either; the retry is the only path. Explained in `AR/08-alarm-ladder.md:358-363`. +6. Timers are Go durations; DST has no effect. A hub wall-clock jump: NOBODY KNOWS (not tested). +7. Nobody is mailed when the retries give up; the trace is the ERROR line `HUB/notify/dispatcher.go:101` and `failed` rows in the notification log `AR/08-alarm-ladder.md:362-363`. +8. TEST `TestOperatorMailRetry_FailedOperatorMailIsRetried` `HUB/notify/operator_mail_retry_test.go:17`. Restart during the retry window: NOBODY KNOWS. + +### 5. Kernel "your box restarts tonight" mail +1. CUSTOMER. `KernelNotify` `HUB/osupdates/kernel.go:191` mails each due household the day before a kernel step; called from `Evaluate` `HUB/osupdates/service.go:436`, which the 60 s OS ticker calls `E/hub/cmd/hub/main.go:564`. +2. Every 60 s, but acts only 09:00-20:00 Budapest: `KernelNoticeFromHour`/`KernelNoticeToHour` `HUB/osupdates/kernel.go:46-47`, checked at `HUB/osupdates/kernel.go:193-196`. Valid 24 h, gap 20 h, at most 3 mails `HUB/osupdates/kernel.go:48-50`. Zone from `budapestLoc` `HUB/osupdates/kernel.go:73-78`. Constants, no setting. +3. No time limit. Loops all hosts every minute with store reads per host `HUB/osupdates/kernel.go:203-208`; grows linearly with boxes, 1440 passes a day. +4. Pod down all day 09-20: no mail, and without a mail there is no step: `KernelBlockFor` sets `Tonight` only for a notice younger than 24 h `HUB/osupdates/kernel.go:179-185`. It mails on the next daytime tick; the notices are persisted `HUB/osupdates/kernel.go:236`. +5. Twice is blocked by the stored notice and the 20 h gap `HUB/osupdates/kernel.go:208-211`; a restart cannot double-mail because the row is in the store. A mail sent but not recorded means no step tonight `HUB/osupdates/kernel.go:236-239`. +6. Hub clock decides the daytime window. A mail sent at 19:59 is valid until 19:59 next day `HUB/osupdates/kernel.go:182`, so it covers one night. DST: the hour test uses Budapest local time `HUB/osupdates/kernel.go:193-194`; the 20 h / 24 h are durations. A deliberately-off box gets no mail `HUB/osupdates/kernel.go:201`. +7. Mail failure: operator event `os_kernel_notice` warning, once per box, kernel and day `HUB/osupdates/kernel.go:226-233`. After 3 mails with no step: operator warning, once `HUB/osupdates/kernel.go:212-219`. +8. TEST `TestKernel_AtMostThreeMails` `HUB/osupdates/kernel_test.go:192`; TEST `TestKernel_DayBeforeMailMakesTonight` `HUB/osupdates/kernel_test.go:79`; TEST `TestKernel_NoMailNoStep` `HUB/osupdates/kernel_test.go:115`. A mail sent late in the day vs the box's night slot: NOBODY KNOWS. + +### 6. OS update approval tick and hourly OS alarms +1. OPERATOR (the outcome reaches households as updates). 60 s ticker `E/hub/cmd/hub/main.go:554-586`: `osSvc.Evaluate()` `E/hub/cmd/hub/main.go:564` approves a set when its rule holds; `osSvc.Alarms()` `E/hub/cmd/hub/main.go:577` raises the fleet alarms. +2. Evaluate every 60 s; Alarms when `time.Since(lastAlarms) >= time.Hour` `E/hub/cmd/hub/main.go:574`. Rule default `ApproveAfter: 24 * time.Hour, NightsRequired: 1` `E/hub/cmd/hub/main.go:476`, env `OS_APPROVE_AFTER` `E/hub/cmd/hub/main.go:484-491`. Alarm limits 7 d / 14 d, env table `E/hub/cmd/hub/main.go:536-541`. +3. No time limit. The alarm pass logs its own duration `E/hub/cmd/hub/main.go:581`. Evaluate walks every layer `HUB/osupdates/service.go:414-422`; grows with boxes times layers, every minute. +4. Pod restart: `lastAlarms` is a local variable `E/hub/cmd/hub/main.go:558`, zero at start, so the first tick runs the alarms at once. Re-mail is prevented by the stored raise time, 7 days `HUB/osupdates/service.go:1102`. +5. One goroutine, so Evaluate and Alarms never overlap `E/hub/cmd/hub/main.go:559-585`. `raise` clears the mark when the condition ends so the next occurrence is announced `HUB/osupdates/service.go:1094-1108`. +6. Ages use the hub clock against stored times `HUB/osupdates/service.go:1102`. A box's claimed reboot time is allowed 1 minute of skew, `rebootClaimSkew` `HUB/osupdates/service.go:970`. Larger box-clock error: NOBODY KNOWS. +7. Operator, by event through the dispatcher: `os_update_stale`, `os_ring0_stalled` and others, listed in `AR/08-alarm-ladder.md:375-383`. Repeats weekly while it holds `HUB/osupdates/service.go:1092-1093`. +8. TEST `TestAlarm_Ring0Stalled` `HUB/osupdates/alarms_test.go:226`; DOC `AR/08-alarm-ladder.md:365`. Cost of the per-minute pass at 100 boxes: NOBODY KNOWS. + +### 7. Off-site copy too old, and escrow never finished +1. CUSTOMER. `offsiteChecker.Check` `E/hub/cmd/hub/main.go:943` on the 60 s sweep; `isStale` `HUB/monitor/offsite.go:166` and `checkEscrowPending` `HUB/monitor/offsite_escrow_pending.go:66`. Owner: hub. +2. Every 60 s. Stale after `defaultOffsiteStaleAfter` 48 h `HUB/monitor/offsite.go:54`, counted from the last SUCCESS `HUB/monitor/offsite.go:203-207`. Escrow pending after `escrowPendingAfter` 7 d `HUB/monitor/offsite_escrow_pending.go:37`, re-raised each 7 d `HUB/monitor/offsite_escrow_pending.go:110`. +3. No time limit. Reads every customer's latest report each minute `HUB/monitor/offsite.go:443-459`; grows linearly with boxes. +4. Pod down: not judged. The edge states are in-memory maps (the ones cleared at `HUB/monitor/offsite.go:463-468`); whether a still-stale tier is mailed again after a restart: NOBODY KNOWS (restart seeding NOT FOUND, grep `R-1001|alreadyReported` in that file). +5. One mutex per pass `HUB/monitor/offsite.go:448-449`; a customer listed twice is handled once `HUB/monitor/offsite.go:453-458`. A never-ran tier is owned by the delivery checker, "one state, one owner" `HUB/monitor/offsite.go:160-162`. +6. `LastSuccess` is the BOX's timestamp compared with the hub's now `HUB/monitor/offsite.go:203-207`. A box clock ahead hides staleness (candidate); an unparseable time counts as stale `HUB/monitor/offsite.go:204-206`. Snapshot-drop tolerates `windowSlack` 2 h `HUB/monitor/offsite.go:296`. +7. Dispatcher mail on the stale edge `HUB/monitor/offsite.go:636`; `offsite_escrow_pending` is operator-only, see `AR/08-alarm-ladder.md:383`. +8. TEST `TestIsStale_PersistentlyFailingTierGoesStale` `HUB/monitor/offsite_r100_test.go:55`; TEST `TestIsStale_NewbornBoxDoesNotAlarm` `HUB/monitor/offsite_r100_test.go:74`. Box clock ahead: NOBODY KNOWS. + +### 8. Restore-test proof too old +1. CUSTOMER (protects the promise that a backup restores; mail goes by event). `restoreTestChecker.Check` `E/hub/cmd/hub/main.go:944` on the 60 s sweep; `checkStaleness` `HUB/monitor/restoretest.go:242`. +2. Every 60 s. Window per tier: 4 archive generations `HUB/monitor/restoretest.go:62`, never under `restoreProvenWindowFloor` 7 d `HUB/monitor/restoretest.go:68`, never over `restoreProvenWindowCap` 12 d `HUB/monitor/restoretest.go:74`. Constants, no setting. +3. No time limit. Each pass reads 14 days of host reports per customer, `restoreWindowRead` `HUB/monitor/restoretest.go:79` at `HUB/monitor/restoretest.go:252`, every minute. This is the heaviest sweep read found; grows with boxes times report rate (candidate). +4. Pod restart: the first pass does not re-send a stale signal already in the events table `HUB/monitor/restoretest.go:281-283`, `staleAlreadyReported` `HUB/monitor/restoretest.go:300-312`. +5. Edge-triggered per customer and tier under a mutex `HUB/monitor/restoretest.go:270-278`. A deliberately-off box is skipped `HUB/monitor/restoretest.go:192-194`. +6. A new box is "unknown" until its window has passed since first contact `HUB/monitor/restoretest.go:326-342`; first contact is hub receive time `HUB/store/store.go:3994-4005`. Proof times come from box reports; box clock error: NOBODY KNOWS. +7. Event `restore_test_stale` warning through the dispatcher `HUB/monitor/restoretest.go:284-285`; a failed run is a separate event, `checkFailure` `HUB/monitor/restoretest.go:205`. +8. TEST `TestRestoreTestStale_R1001_RestartDoesNotReMail` `HUB/monitor/r1001_restart_test.go:147`. Cost at 100 boxes: NOBODY KNOWS. + +### 9. Off-site clean-up window (20 minutes, weekly) +1. CUSTOMER DATA. The box asks; `OpenWindowFor` `HUB/offsitekeys/service.go:334` adds a deleting key line for 20 minutes; the hub's 60 s sweep `SweepExpiredWindows` `HUB/offsitekeys/service.go:413` closes a window left open. Sweep loop: `E/hub/cmd/hub/main.go:636-650`. +2. Granted when the last window is at least `windowCadence` 6 d 12 h old `HUB/offsitekeys/service.go:264`, test at `HUB/offsitekeys/service.go:337`; length `windowLength` 20 min `HUB/offsitekeys/service.go:263`. Sweep every 60 s. Constants; the weekly switch is a store setting `HUB/offsitekeys/service.go:338`. +3. Each sweep has a 2-minute context `E/hub/cmd/hub/main.go:644`. Work is per expired window, one remote call each `HUB/offsitekeys/service.go:419-422`; grows with boxes that crash mid-prune. +4. Pod down when a window expires: the deleting key line stays on the storage account until the hub is back; the sweep then closes it `HUB/offsitekeys/service.go:419-431`. Nobody is told during the gap (candidate). A failed close is retried next sweep `HUB/offsitekeys/service.go:424-426`. +5. One window per cadence via the ledger row `HUB/offsitekeys/service.go:364`. The daily key audit knows an open window `HUB/offsitekeys/service.go:212-213`. No lock against the abandon sweep; both run in the same goroutine `E/hub/cmd/hub/main.go:636-650`. +6. Cadence and expiry use the hub clock, `s.now()` `HUB/offsitekeys/service.go:337`. The box's clock does not decide. Hub clock jump back: a window could stay open longer: NOBODY KNOWS. +7. Operator event `offsite_window_failed` warning when the hub had to close it `HUB/offsitekeys/service.go:430-431`; `offsite_window_drop` error when too much went `HUB/offsitekeys/service.go:259`. +8. TEST `TestWindow_LeftOpenIsClosedByTheSweep` `HUB/offsitekeys/service_test.go:99`; TEST `TestWindow_GrantOpenCloseAndDropAlarm` `HUB/offsitekeys/service_test.go:37`; DOC `AR/07-backup-architecture.md:256`. + +### 10. Delayed deletion of a set-aside off-site copy +1. CUSTOMER DATA. `SweepAbandons` `HUB/offsitekeys/service.go:529` deletes a household's set-aside copy once its request is due; called from the 60 s sweep `E/hub/cmd/hub/main.go:636-650`. +2. Due `DefaultAbandonDelay` 7 days after the request `HUB/offsitekeys/service.go:448`; override `OFFSITE_ABANDON_DELAY` `E/hub/cmd/hub/main.go:598-605`. Checked every 60 s. Hub clock. +3. 2-minute context shared with the window sweep `E/hub/cmd/hub/main.go:644`. One remote delete per due request `HUB/offsitekeys/service.go:536-538`; duration of a large delete: NOBODY KNOWS. +4. Pod down at the due time: deleted on the first sweep after start; the due list is in the store `HUB/offsitekeys/service.go:530`. A failed delete is marked and retried every sweep `HUB/offsitekeys/service.go:540-547`. +5. A request is deleted once, then marked `HUB/offsitekeys/service.go:549`. A cancel before the due time wins: design text `HUB/offsitekeys/service.go:438-441`. +6. Hub clock only. With an env override set too short by mistake the week of warning shrinks (candidate; value read at `E/hub/cmd/hub/main.go:598-605`). +7. Operator events: requested, cancelled, deleted, failed `HUB/offsitekeys/service.go:444-447`; failure mails once per request `HUB/offsitekeys/service.go:543-546`. +8. TEST `TestAbandon_DelayCancelAndScope` `HUB/offsitekeys/service_test.go:134`. + +### 11. Off-site key audit (07:10) +1. OPERATOR (guards household copies against deletion). `scheduleDaily(ctx, "offsite-key-audit", "07:10", ...)` `E/hub/cmd/hub/main.go:651-655` runs `AuditAll` `HUB/offsitekeys/service.go:195`. +2. Daily 07:10 Budapest, literal slot `E/hub/cmd/hub/main.go:651`. Zone from `scheduleDaily` `E/hub/cmd/hub/main.go:1072`. +3. 10-minute limit for the whole fleet `E/hub/cmd/hub/main.go:652`. One remote read per provisioned customer, in sequence `HUB/offsitekeys/service.go:202-215`; at 100 boxes the 10 minutes may not cover all (candidate). +4. Pod down at 07:10: skipped that day, no catch-up `E/hub/cmd/hub/main.go:1095-1112`. Nobody notices. +5. Runs once per day by the `last` guard `E/hub/cmd/hub/main.go:1095-1112`. Meets an open clean-up window: the audit is told the window is open `HUB/offsitekeys/service.go:212-213`. +6. Wall-clock slot, DST-safe by `nextDailyRun` `E/hub/cmd/hub/main.go:1095-1112`. +7. Operator: `offsite_key_unlocked` error, or `offsite_key_audit_failed` when a file cannot be read `HUB/offsitekeys/service.go:192-194`. +8. TEST `TestWindow_PrependAuditClose` `HUB/offsitekeys/offsitekeys_test.go:158`; DOC `AR/05-hub-architecture.md:642`. The 10-minute limit at fleet size: NOBODY KNOWS. + +### 12. Off-site credential stuck and its self-heal +1. CUSTOMER. Two jobs, merged: `offsiteDeliveryChecker.Check` `E/hub/cmd/hub/main.go:945` (60 s sweep) and the `offsiteheal` reconciler `HUB/offsiteheal/reconciler.go:165-177`, started at `E/hub/cmd/hub/main.go:846-854`. +2. Stuck after `stuckAfter` 1 h `HUB/monitor/offsite_delivery.go:38`; cooldowns 24 h `HUB/monitor/offsite_delivery.go:40-41`. Heal sweep every `tickIntervalDefault` 5 min `HUB/offsiteheal/reconciler.go:74`, acts after 2 distinct reports `HUB/offsiteheal/reconciler.go:70`. +3. Heal call limit 3 minutes `HUB/monitor/offsite_delivery.go:275`. Both list every customer config per pass `HUB/monitor/offsite_delivery.go:72` and `HUB/offsiteheal/reconciler.go:183`; linear in boxes. +4. Pod down: nothing; on return the next tick re-reads state. The cooldowns are "durable via the events table" `HUB/monitor/offsite_delivery.go:39`. +5. Two healers could meet: the checker re-reads the secret row immediately before acting `HUB/monitor/offsite_delivery.go:28-31`, and a box declaration is owned by offsiteheal `HUB/monitor/offsite_delivery.go:97-99`. +6. Age counted from the hub-held `status.Since` `HUB/monitor/offsite_delivery.go:93`. Debounce is counted in reports, not ticks `HUB/offsiteheal/reconciler.go:61-69`. +7. Operator events `offsite_delivery_stuck`, `offsite_credential_restaged` `HUB/monitor/offsite_delivery.go:34-35`; `offsite_selfheal_restaged` / `_reissued` `HUB/offsiteheal/reconciler.go:57-58`. +8. TEST `TestDeliveryChecker_StuckEvent_WarningOncePer24h` `HUB/monitor/offsite_delivery_test.go:121`; SOURCE `HUB/offsiteheal/reconciler.go:61-74`. + +### 13. PBS whole-box credential self-heal +1. CUSTOMER. `pbsdrheal` reconciler `HUB/pbsdrheal/reconciler.go:153-165`, started at `E/hub/cmd/hub/main.go:829-837`; repairs a box whose off-site PBS credential did not land. +2. Every 5 minutes `HUB/pbsdrheal/reconciler.go:130`, or on a trigger `HUB/pbsdrheal/reconciler.go:160`; acts after 2 reports `HUB/pbsdrheal/reconciler.go:131`. A staged secret unconsumed for `unconsumedGrace` 15 min is flagged `HUB/pbsdrheal/reconciler.go:63`. +3. No time limit on a pass, `reconcileOnce(ctx)` `HUB/pbsdrheal/reconciler.go:163`. Reads all heal states `HUB/pbsdrheal/reconciler.go:170`; linear in boxes. +4. Pod down: no heal; the debounce maps are in memory `HUB/pbsdrheal/reconciler.go:134-135`, so after a restart a box waits two more reports. +5. One goroutine; a read error does nothing this tick `HUB/pbsdrheal/reconciler.go:171-174`. +6. Grace is hub time. Box clock: not used here. +7. Operator audit events `pbsdr_selfheal_*` and `pbsdr_unconsumed_secret` `HUB/pbsdrheal/reconciler.go:50-54`. +8. SOURCE `HUB/pbsdrheal/reconciler.go:57-63`. A pass that hangs on the remote: NOBODY KNOWS. + +### 14. WireGuard peer list push to ep0 +1. OPERATOR (keeps every box's tunnel allowed). `wgReconciler.Run` `E/hub/cmd/hub/main.go:752`, loop `HUB/wgsync/reconciler.go:90-106`; pushes the FULL peer list over SSH. +2. Every 5 minutes `HUB/wgsync/reconciler.go:41` and on every change via `Trigger` `HUB/wgsync/reconciler.go:48-53`. No setting. +3. 60 s limit per push `HUB/wgsync/reconciler.go:100`. The payload grows with peers `HUB/wgsync/reconciler.go:71-73`. +4. Pod down: no push; ep0 keeps the last list. A failed push is logged and retried next tick `HUB/wgsync/reconciler.go:101-103`. An event or mail on repeated failure: NOT FOUND (grep `SaveEvent|onEvent` in that file) (candidate). +5. One loop; a pending trigger covers later ones `HUB/wgsync/reconciler.go:46-47`. Full list each time erases drift `HUB/wgsync/reconciler.go:20-23`. +6. No clock in the payload. +7. Log line only `HUB/wgsync/reconciler.go:102`. +8. TEST `TestReconciler_FailureRetriedOnTick` `HUB/wgsync/reconciler_test.go:119`; TEST `TestReconciler_TriggerPushesFullList` `HUB/wgsync/reconciler_test.go:82`. + +### 15. Box clock ran ahead: tunnel alarm +1. OPERATOR (the household's whole-box copies stop). `checkWGTunnel` `HUB/api/host_tunnel.go:61`, called on every host report `HUB/api/handler.go:1010`; not a timer, it runs when the box reports. +2. Fires when `handshake_future_s` > 0 `HUB/api/host_tunnel.go:41` or the last handshake is older than `wgTunnelStaleAfterS` 6 h `HUB/api/host_tunnel.go:23`. Constant. +3. One store read per report `HUB/api/host_tunnel.go:66`. +4. If the box stops reporting, this check never runs; block 1 covers that. An older agent that sends neither field is not judged `HUB/api/host_tunnel.go:62-64`. +5. Edge-triggered from the newest stored tunnel event `HUB/api/host_tunnel.go:71-83`, so a restart does not re-mail. +6. This IS the clock-ahead handling: the future stamp is measured by the box's own clock `HUB/api/host_tunnel.go:31`; repair is a new key after 15 minutes on the agent side `HUB/api/host_tunnel.go:14-17`. Known gaps are listed in `AR/05-hub-architecture.md:489-490`. +7. Operator only, `wg_tunnel_stale` warning; `wg_tunnel_ok` is stored, not mailed `HUB/api/host_tunnel.go:12-14`. +8. TEST `TestD30_TheAlarmClearsWhenTheNewKeyShakesHands` `HUB/api/host_tunnel_test.go:83`; DOC `AR/05-hub-architecture.md:482-490`. + +### 16. The daily scheduler and the clock change +1. OPERATOR (mechanism under blocks 2, 11, 17, 18). `scheduleDaily` `E/hub/cmd/hub/main.go:1069-1093` with `nextDailyRun` `E/hub/cmd/hub/main.go:1095-1112`. +2. Budapest wall-clock, `LoadLocation` `E/hub/cmd/hub/main.go:1072`; fallback fixed CET `E/hub/cmd/hub/main.go:1074` (candidate: in summer the fallback is one hour off). Pod `TZ` is `Europe/Budapest` `E/manifests/hub.yaml:140-141`. +3. Sleeps with `time.After(delay)` `E/hub/cmd/hub/main.go:1088`; the job runs inline `E/hub/cmd/hub/main.go:1090`, so a long job delays only its own next slot. +4. No catch-up for a slot missed while the pod was down; exceptions are the snapshot (block 17) and prune (block 18). A bad time string becomes 00:00 silently, `parseHM` `E/hub/cmd/hub/main.go:1115-1121` (candidate). +5. Next run is the calendar day after and the slot that ran last never runs again `AR/05-hub-architecture.md:642-645`. +6. 25-hour and 23-hour days handled, D24 `AR/05-hub-architecture.md:640-646`. A hub clock jump while sleeping: the timer keeps its old duration: NOBODY KNOWS. +7. Nobody; start-up logs only. +8. TEST `TestR1183_DailyJobRunsOnceOnTheDayTheClocksGoBack` `E/hub/cmd/hub/r1183_dst_schedule_test.go:31`; TEST `TestR1183_DailyJobRunsOnceOnTheDayTheClocksGoForward` `E/hub/cmd/hub/r1183_dst_schedule_test.go:56`. + +### 17. Hub database snapshot (four slots) +1. OPERATOR (the hub holds every household's escrow and settings). `runSnapshot` scheduled over `dbsnap.Slots` `E/hub/cmd/hub/main.go:967-983`; `Make` `HUB/dbsnap/dbsnap.go:72-117`. +2. 02:00, 08:00, 14:00, 20:00 Budapest: `Slots` `HUB/dbsnap/dbsnap.go:32`. Keeps `Keep = 2` `HUB/dbsnap/dbsnap.go:27`. File name carries UTC `HUB/dbsnap/dbsnap.go:89`. +3. 10-minute limit `E/hub/cmd/hub/main.go:972`. Time grows with database size; written to a tmp file then renamed `HUB/dbsnap/dbsnap.go:91-104`; duration is logged `HUB/dbsnap/dbsnap.go:113-114`. +4. The one job WITH catch-up: at start, if the newest snapshot is older than `CatchUpAge` 6 h, one is made `E/hub/cmd/hub/main.go:978-980`, `NeedsCatchUp` `HUB/dbsnap/dbsnap.go:148-159`. +5. A running flag refuses a second run with `ErrBusy` `HUB/dbsnap/dbsnap.go:73-80`. The DooPlex push timer runs 30 min later and waits for the tmp file (block 26). +6. DST-safe slots (block 16). The doc still says "older than 24 h" and "02:30" in its original text `AR/05-hub-architecture.md:618-620` (candidate: stale text). +7. Failure is a log line; the outside alarm is the push timer's (block 26). +8. TEST `TestD34_SlotsAreEverySixHours` `HUB/dbsnap/dbsnap_test.go:214`; TEST `TestR173_MainSchedulesTheDBSnapshot` `E/hub/cmd/hub/r173_wiring_test.go:14`. + +### 18. Daily prune of old hub rows (04:30) +1. OPERATOR. `pruneAll` `E/hub/cmd/hub/main.go:1123-1150`, at start and by `scheduleDaily(ctx, "prune", ...)` `E/hub/cmd/hub/main.go:860-865`. +2. Daily at `prune_schedule`, default `"04:30"` `E/hub/cmd/hub/main.go:1026`, deployed `E/manifests/hub.yaml:83`; keeps `max_days` 90 `E/manifests/hub.yaml:82`. Also app telemetry 90 d `E/hub/cmd/hub/main.go:1134`, stale issues 30 d `E/hub/cmd/hub/main.go:1139`, deleted-customer audit 1 year `E/hub/cmd/hub/main.go:1145`. +3. No time limit. Deletes grow with reports per box per day; duration NOT FOUND (grep `time.Since` in `E/hub/cmd/hub/main.go:1123-1150`). +4. Runs at every start, so a missed slot is caught up by the next restart; with `MaxDays` 0 it never runs `E/hub/cmd/hub/main.go:860-865`. +5. May meet the 05:00 deadline check or a snapshot only if it runs over 30 min; no lock found. +6. `Prune` builds its cutoff from local time without `.UTC()` `HUB/store/store.go:1607` while the column is UTC `HUB/store/store.go:129`; `PruneEvents` uses UTC `HUB/store/store.go:2873` (candidate: 1-2 h skew on a 90-day cutoff, harmless in size). +7. Log only. +8. SOURCE `E/hub/cmd/hub/main.go:1123-1150`. A test for the prune: NOT FOUND (grep `func Test.*Prune` in hub/internal/store). + +### 19. The other checks on the 60-second sweep +1. CUSTOMER and OPERATOR mixed. Merged: host capability, host leaf, host disk, storage fill, management plane, OOB, controller supervisor `E/hub/cmd/hub/main.go:937-946`. All run in the one sweep goroutine `E/hub/cmd/hub/main.go:927-965`. +2. Every 60 s in the fixed order of `E/hub/cmd/hub/main.go:935-962`. A backup-cadence refusal repeats every `refusalRepeatEvery` 24 h `HUB/monitor/host_capability.go:72`. +3. No per-checker limit: a slow checker delays all the others, including liveness (block 1), because they run in sequence `E/hub/cmd/hub/main.go:935-962` (candidate). Each reads the whole fleet per pass. +4. Pod down: nothing. After start, band states are rebuilt from the latest reports. +5. One goroutine, so no overlap; a Go ticker drops ticks when a pass overruns. +6. They judge the latest report's content, not time, except the 24 h repeat `HUB/monitor/host_capability.go:172`. +7. Dispatcher mails; storage fill has its own 6 h grain `AR/08-alarm-ladder.md:305`. +8. SOURCE `E/hub/cmd/hub/main.go:927-965`. Duration of one full sweep: NOBODY KNOWS. + +### 20. Reminder about a held app +1. OPERATOR (D53: no household mail). `heldApps.Check` `E/hub/cmd/hub/main.go:947`; `HeldAppsChecker.Check` `HUB/monitor/held_apps.go:102`. +2. One pass per `heldAppCheckEvery` 15 min `HUB/monitor/held_apps.go:36`. First mail after a grace of 6 h / 72 h / 24 h by reason `HUB/monitor/held_apps.go:40-48`; then every `heldAppRepeat` 7 d `HUB/monitor/held_apps.go:34`. +3. One list of all held apps `HUB/monitor/held_apps.go:114`. +4. Pod down: the reminder comes on the first pass after start; the reminded-at time is stored BEFORE sending `HUB/monitor/held_apps.go:141-145`, so a restart cannot mail twice. +5. A row the box has not confirmed for `heldAppStale` 50 h is not reminded `HUB/monitor/held_apps.go:125-129`. +6. The age counts from the hold's own time, which is the BOX's clock, else the hub's first sight `HUB/monitor/held_apps.go:130-134` and `HUB/monitor/held_apps.go:22`. Box clock behind: reminder early; ahead: late (candidate). +7. Operator, `app_held_reminder` warning `HUB/monitor/held_apps.go:157-163`. +8. TEST `TestD53_OneReminderAfterTheGraceThenWeekly` `HUB/monitor/held_apps_test.go:34`; TEST `TestD53_AStaleRowIsNotReminded` `HUB/monitor/held_apps_test.go:99`; DOC `AR/08-alarm-ladder.md:655`. + +### 21. "A golden is published but not vouched" +1. OPERATOR. `goldenVouch` check on the sweep `E/hub/cmd/hub/main.go:948-950`; `GoldenVouchChecker.Check` `HUB/monitor/golden_vouch.go:54`. +2. One Gitea listing per `goldenVouchCheckEvery` 15 min; alarm after `goldenVouchGrace` 2 h `HUB/monitor/golden_vouch.go:42-43`. +3. Listing limit `goldenVouchListDeadline` 20 s `HUB/monitor/golden_vouch.go:44`; it runs inside the sweep, so a slow Gitea holds the sweep for up to 20 s `HUB/monitor/golden_vouch.go:62-64`. +4. Pod restart: the gap clock and the alerted mark are in memory `HUB/monitor/golden_vouch.go:34-37`, so the 2 h starts again and the alarm is mailed again ("one alarm per unvouched version per process" `HUB/monitor/golden_vouch.go:23`). +5. Gitea unreachable or empty list: no verdict, never an alarm `HUB/monitor/golden_vouch.go:65-67`. +6. Hub clock only. +7. Operator event `golden_not_vouched` `HUB/monitor/golden_vouch.go:95-96`. +8. TEST `TestGoldenVouchChecker` `HUB/monitor/golden_vouch_test.go:15`. Re-mail after a restart: NOBODY KNOWS. + +### 22. Shared storage boxes: fill and reachability +1. OPERATOR (every household's off-site space). `offsiteBox` and `pbsdrBox` checks `E/hub/cmd/hub/main.go:951-956`. +2. On the 60 s sweep, but one provider read per `boxFetchInterval` 15 min `HUB/monitor/offsite_box.go:72`, test at `HUB/monitor/offsite_box.go:132` and `HUB/monitor/pbsdr_box.go:130`. Warn 80 %, critical 90 % `HUB/monitor/offsite_box.go:67-68`. +3. Provider client limit 30 s `HUB/hetznerapi/hetznerapi.go:143-145`. Does not grow with boxes. +4. Pod down: no reading. Unreachable is declared after consecutive failed 15-minute windows `HUB/monitor/pbsdr_box.go:80-82`. +5. Throttled by `lastFetch` in memory; first pass after a restart fetches at once `HUB/monitor/offsite_box.go:132`. +6. Hub clock only. +7. Operator events for fill, oversubscription, unreachable and recovered `HUB/monitor/offsite_box.go:284-339`. +8. SOURCE `HUB/monitor/offsite_box.go:70-72`. + +### 23. Purges: log bundles, access log, sessions +1. OPERATOR. Merged: `PurgeExpiredLogBundles` on the sweep `E/hub/cmd/hub/main.go:958-962`; access-log `prune` `HUB/accesslog/accesslog.go:192`; `CleanupSessions` `HUB/web/server.go:199-217`. +2. Bundles: every 60 s, TTL `logBundleTTL` 72 h `HUB/store/logbundle.go:29`. Access log: on the first line of each UTC day `HUB/accesslog/accesslog.go:166-184`, keeps `DefaultKeep` 90 d `HUB/accesslog/accesslog.go:44`. Sessions: every 15 min `HUB/web/server.go:200`, life 7 d `HUB/web/server.go:949`. +3. Small. Access-log prune lists one directory `HUB/accesslog/accesslog.go:196`. +4. Pod restart: sessions are in memory `HUB/web/server.go:207-213`, so every operator is logged out. Access-log prune runs when the day file is first opened `HUB/accesslog/accesslog.go:182-184`, so it also runs after a restart. +5. Access-log prune only removes exact `access-YYYY-MM-DD.log` names `HUB/accesslog/accesslog.go:191`. +6. Access log days are UTC `HUB/accesslog/accesslog.go:166`, unlike the Budapest daily jobs; documented `AR/05-hub-architecture.md:634-636`. +7. Nobody. +8. TEST `TestAccessLog_Keeps90DaysThenDeletes` `HUB/accesslog/accesslog_test.go:135`. + +### 24. Links and tokens that expire by themselves +1. CUSTOMER (bind link) and OPERATOR (tokens). No sweep: each is checked when used. Bind link `selfBindTTL` 7 d `HUB/web/selfbind_mint.go:20`; access token expiry `HUB/web/tokens.go:95-96`; DR recovery `defaultRecoveryTTL` 30 min, max 4 h `HUB/api/dr.go:21-22`. +2. Bind link: 7 days from mint `HUB/web/selfbind_mint.go:77`; a fresh link is mailed at most once per `bindResendEvery` 1 h `HUB/web/selfbind.go:382`. Token life is chosen in days at creation `HUB/web/tokens.go:180`. Operator actions: `OperatorActionTTL` 24 h `HUB/store/opactions.go:62`. +3. Nothing runs; expired rows stay in the tables. A bind token row is deleted only when a new one is minted for that customer `HUB/store/selfbind.go:86`; a purge by age: NOT FOUND (grep `DELETE FROM` in the store's selfbind and api_tokens files). +4. Not applicable: no job. An expired link reads as expired and falls back to operator bind `HUB/web/selfbind_mint.go:18-19`. +5. Bind expiry is tested twice, in Go `HUB/store/selfbind.go:39` and in SQL `HUB/store/selfbind.go:177`. +6. Hub clock. SQL side uses `datetime('now')` UTC `HUB/store/selfbind.go:177`. +7. Nobody is warned BEFORE a token or link expires; an expired token logs a WARN on use `HUB/web/tokens.go:96` (candidate: a token used by automation expires silently until the next call). +8. SOURCE `HUB/store/selfbind.go:38-46`. Warning before expiry: NOT FOUND (grep `expires in|about to expire|expiring` in web/tokens.go and monitor). + +### 25. Version check and catalog template fetch +1. OPERATOR. `VersionChecker.Run` `HUB/web/version.go:43-55` and `TemplateFetcher.Run` `HUB/web/templatefetcher.go:41-53`, started at `E/hub/cmd/hub/main.go:711-722` and `E/hub/cmd/hub/main.go:327-335`. +2. Once at start, then every `check_interval` 6 h `E/manifests/hub.yaml:104` and `template_interval` 1 h `E/manifests/hub.yaml:105`. +3. One HTTP call each. +4. Pod down: nothing; both fetch at once on start `HUB/web/version.go:44` and `HUB/web/templatefetcher.go:42`. +5. One goroutine each. +6. Interval tickers; no wall clock. +7. A failure is a WARN line and a stored `lastError` `HUB/web/version.go:62-65`; nobody is mailed. +8. SOURCE `HUB/web/version.go:41-55`. + +### 26. DooPlex: push of the hub database snapshot off-site +1. NOT MEASURED. OPERATOR. systemd timer `felhom-hub-db-backup.timer` on DooPlex, `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10`, runs the script `E/scripts/hub-db-backup/felhom-hub-db-backup:19`; installed by hand per `RB/RUNBOOK-hub-db-offsite-backup.md:107-108`. +2. 02:30, 08:30, 14:30, 20:30 `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10`, `RandomizedDelaySec=2min` `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:12`. Host local time (no zone in the unit). DooPlex's zone: NOT FOUND (grep `timedatectl|Europe/Budapest` in documentation/operations and the three runbooks). +3. `TimeoutStartSec=45min` `E/scripts/hub-db-backup/felhom-hub-db-backup.service:13`. Waits up to `WAIT_S` 1200 s for the hub's tmp file `E/scripts/hub-db-backup/felhom-hub-db-backup:20`, loop `E/scripts/hub-db-backup/felhom-hub-db-backup:35-40`. Grows with database size. +4. `Persistent=true` `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:11`: a slot missed while DooPlex was off runs at boot. It refuses a snapshot older than `MAX_AGE_H` 7 `E/scripts/hub-db-backup/felhom-hub-db-backup:19`, line `E/scripts/hub-db-backup/felhom-hub-db-backup:50`. +5. No flock in the script (grep `flock|lock` returned nothing); systemd does not start a oneshot unit twice. A catch-up run right after boot can meet a hub that has not made its catch-up snapshot yet: history of a false failure in `E/scripts/hub-db-backup/felhom-hub-db-backup:15-18`. +6. Hub slots are Budapest (pod TZ), timer slots are host local: the 30-minute offset holds only if both zones match; pinned in text by `E/scripts/hub-db-backup/test_hub_db_backup.py:309-316`. DST night: NOBODY KNOWS. +7. Failure: `OnFailure=felhom-backup-failmail@%n.service` `E/scripts/hub-db-backup/felhom-hub-db-backup.service:6`, mail `E/scripts/dooplex-offsite/felhom-backup-failmail:2`. Silence: `HubDBBackupStale` at 13 h per `RB/RUNBOOK-hub-db-offsite-backup.md:100`, but the YAML there shows 26 h `RB/RUNBOOK-hub-db-offsite-backup.md:153` (candidate). +8. TEST `E/scripts/hub-db-backup/test_hub_db_backup.py:309-316` (run by hand, `RB/RUNBOOK-hub-db-offsite-backup.md:107-108`); the live alert rule is in homelab-manifests, not in this clone: NOBODY KNOWS. + +### 27. DooPlex: Sunday restore tests +1. NOT MEASURED. OPERATOR. Two timers, merged: `felhom-hub-db-restore-test.timer` `E/scripts/hub-db-backup/felhom-hub-db-restore-test.timer:6` and `felhom-dooplex-offsite-restore-test.timer` `E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test.timer:6`. +2. Sunday 04:30 and Sunday 05:30, host local time; `Persistent=true` `E/scripts/hub-db-backup/felhom-hub-db-restore-test.timer:7`. +3. Limits 45 min `E/scripts/hub-db-backup/felhom-hub-db-restore-test.service:13` and 90 min `E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test.service:13`. Grow with database and dump size. +4. Missed Sunday runs at next boot (Persistent). Each refuses a copy older than `MAX_AGE_H` 50 `E/scripts/hub-db-backup/felhom-hub-db-restore-test:13` and `E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test:14`. +5. Sunday 04:30 lies between the hub's 02:00 snapshot push and the 05:00 PBS pull; a lock between them: NOT FOUND (grep `flock` in scripts/hub-db-backup and scripts/dooplex-offsite). +6. Last Sunday of October has 02:00-03:00 twice; systemd's handling for 04:30 is outside the repo: NOBODY KNOWS. +7. Staleness alarm `HubDBRestoreTestStale` after 8 days `RB/RUNBOOK-hub-db-offsite-backup.md:158`. +8. DOC `RB/RUNBOOK-hub-db-offsite-backup.md:152-161`. Whether the rule is deployed: NOBODY KNOWS (homelab-manifests is not in the clone tree). + +### 28. DooPlex: nightly off-site push and release copy +1. NOT MEASURED. OPERATOR. Merged: `felhom-dooplex-offsite.timer` `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:6` and `felhom-release-copy.timer` `E/scripts/release-copy/felhom-release-copy.timer:7`. +2. 00:20 and 01:40 host local time; the order is stated in a comment `E/scripts/release-copy/felhom-release-copy.timer:6`. Random delay 2 min and 5 min `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:8`, `E/scripts/release-copy/felhom-release-copy.timer:9`. +3. Limits 90 min `E/scripts/dooplex-offsite/felhom-dooplex-offsite.service:13` and 2 h `E/scripts/release-copy/felhom-release-copy.service:12`. A 90-minute run from 00:20 ends 01:50, after the 01:40 start (candidate: can overlap; no lock found). +4. Both `Persistent=true` `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:7`, `E/scripts/release-copy/felhom-release-copy.timer:8`. The push refuses a dump older than `DUMP_MAX_AGE_H` 7 `E/scripts/dooplex-offsite/felhom-dooplex-offsite:23`, message `E/scripts/dooplex-offsite/felhom-dooplex-offsite:55`. +5. The push depends on a dump made by another job, not in this clone; it only checks the dump's age `E/scripts/dooplex-offsite/felhom-dooplex-offsite:55`. One retry `E/scripts/dooplex-offsite/felhom-dooplex-offsite:66`. +6. 00:20 and 01:40 are outside the 02:00-03:00 change hour. +7. `OnFailure` mail on both `E/scripts/dooplex-offsite/felhom-dooplex-offsite.service:6`, `E/scripts/release-copy/felhom-release-copy.service:6`; a metric file is written `E/scripts/dooplex-offsite/felhom-dooplex-offsite:143-153`. +8. SOURCE only. The release-copy timer is enabled by hand `E/scripts/release-copy/install.sh:4`; whether it is enabled: NOBODY KNOWS. + +### 29. DooPlex and ep0: PBS copy, prune, verify, clean-up +1. NOT MEASURED. OPERATOR (holds the second copy of every household's whole-box backup). PBS jobs described in `RB/ep0-datastore-copy.md:15-18`, plus `felhom-ep0-copy-gc.timer` `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:5`. +2. ep0 prune 03:30, pull to DooPlex 05:00 `RB/ep0-datastore-copy.md:15`; DooPlex prune 07:30 `RB/ep0-datastore-copy.md:17`; copy clean-up 08:00 `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:5`; verify Sat 06:30, GC Sun 08:30 `RB/ep0-datastore-copy.md:16-18`. +3. The PBS jobs live in PBS's own config, not in the repo; their duration and limits: NOBODY KNOWS. Grow with every box's backup size. +4. The gc timer is `Persistent=true` `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:6`. PBS jobs: catch-up behaviour not in repo. +5. Order is by clock only: prune 07:30 then clean-up 08:00 `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:2`. A pull that runs past 07:30 meets the prune: NOBODY KNOWS. +6. Two machines, two clocks (ep0 and DooPlex): "ep0 prune 03:30 before the 05:00 pull" holds only while both agree. Their zones: NOT FOUND (grep `timedatectl|Europe/Budapest` in that runbook). +7. PBS notification target per `RB/ep0-datastore-copy.md:19`. The gc unit has no `OnFailure` and no time limit (`OnFailure|TimeoutStartSec`: NOT FOUND by grep in `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.service:1-9`) (candidate). +8. DOC `RB/ep0-datastore-copy.md:15-19`. + +### 30. Nightly CI: gates and the security watch +1. NOT MEASURED. OPERATOR. Gitea CI crons in felhom.eu: `gates` `E/.gitea/workflows/gates.yml:26-27` and `security-watch` `E/.gitea/workflows/security-watch.yml:21-22`. The other three repos have no scheduled job (grep `schedule:|cron:` in their `.gitea/workflows/` returned nothing). +2. 03:20 UTC and 03:40 UTC every night; the reason for the order is in `E/.gitea/workflows/gates.yml:21-23`. CI cron is UTC per `AR/08-alarm-ladder.md:710`, so it moves one hour against the Budapest night jobs at each clock change. +3. Limits: 5 min `E/.gitea/workflows/gates.yml:53`, 20 min `E/.gitea/workflows/security-watch.yml:32`. The watch grows with the number of pinned programs. +4. Missed night: no catch-up found. The watch's weekly summary self-heals: it is sent when the last one is 8+ days old `AR/08-alarm-ladder.md:717`. A runner that drops the job silently: NOBODY KNOWS. +5. Both run on `felhom-gates` runners `E/.gitea/workflows/security-watch.yml:31`; the watch commits its memory to main with 3 attempts `E/.gitea/workflows/security-watch.yml:89-101`. A push race with a human commit is retried, then fails. +6. UTC cron, no DST. Reminder ages are computed by the script from its memory file `AR/08-alarm-ladder.md:716-720`. +7. Failure mail to admin: "[felhom CI] security watch FAILED" `E/.gitea/workflows/security-watch.yml:104-119`; gates has its own alarm step `E/.gitea/workflows/gates.yml:245`. Whether the alarm step runs for a reaped job is recorded as unknown `E/.gitea/workflows/gates.yml:50-52`. +8. DOC `AR/08-alarm-ladder.md:708-722`. A night with no run at all: NOBODY KNOWS. + +## Tiny +- Deliberate power-off hold: alarms held 14 days, `DeliberateOffHold` `HUB/store/deliberateoff.go:18`. +- A future-dated power-off request from a box is replaced by hub time `HUB/api/deliberate_off.go:30`. +- Long-poll wait: `waitMaxHold` 240 s, heartbeat 25 s `HUB/api/wait.go:17`. +- Presence: `PresenceWaitCadence` 243 s, grace 90 s `HUB/intent/presence.go:21`. +- Hetzner provisioning poll: every 3 s, at most 3 min `HUB/hetznerapi/hetznerapi.go:143-145`. +- Off-site scan back-off 2,4,8,16,30,30,30 s `HUB/offsite/offsite.go:112`. +- PBS self-check pause 7 s `HUB/web/pbsdr.go:571`. +- Host delete allowed after 360 s off `HUB/web/hosts.go:770`. +- Host operator reboot action TTL 30 min `HUB/store/opactions.go:47`. +- System page: line test stale after 15 days `HUB/web/system.go:174`. +- HTTP server timeouts `E/hub/cmd/hub/main.go:705-707`; shutdown grace 15 s `E/hub/cmd/hub/main.go:1002`. +- MAIL-HOLD has no expiry timer; it is released by hand `HUB/mailhold/mailhold.go:74`. +- No timed digest in the hub: the per-run digest comes from the box `HUB/notify/dispatcher.go:243`. +- NOT MEASURED: weekly golden bake is a manual Sunday-night step, not a timer `RB/nightly-rotation.md:15`. +- NOT MEASURED: `systemd-run --unit=golden-bake` is a manual start `RB/RUNBOOK-manual-build.md:180`. +- NOT MEASURED: ep0's hub-DB prune 03:45 `RB/RUNBOOK-hub-db-offsite-backup.md:101`. +- NOT MEASURED: DooPlex's own `dooplex-backup.timer` 03:19 and Longhorn 04:00 / Sunday 05:00 are named only in `RB/RUNBOOK-hub-db-offsite-backup.md:12-14`. +- NOT MEASURED: box-side `felhom-mgmt-watchdog.timer` is enabled by the installer `E/scripts/felhom-host-install.sh:3076` (belongs to the agent/installer helper). + +## Not found +- /mnt/5_hdd/felhom.eu/git/homelab-manifests does not exist in the clone tree (tried: ls of /mnt/5_hdd/felhom.eu/git/, find -maxdepth 2 -name 'homelab*'). So no k8s CronJob, Prometheus rule or Renovate schedule could be read from it. +- k8s CronJob in felhom.eu/manifests/: NOT FOUND (grep `kind: CronJob|schedule:` matched only `E/manifests/hub.yaml:83`, the hub's own prune time). +- Cloudflare periodic sync in the hub: NOT FOUND (grep `NewTicker|time.After|time.Sleep` in hub/internal/cloudflare showed only request timeouts). +- A hub-side security watch: NOT FOUND in hub code; it is the CI job of block 30. +- A timed mail digest in the hub: NOT FOUND (grep -il `digest` in hub/internal: only the box's per-run digest). +- A sweep that deletes expired bind links, tokens or claim codes: NOT FOUND (grep `expires|Expir|TTL` in hub/internal/claim, web/tokens.go, store/selfbind.go shows read-time checks only). +- DooPlex / ep0 host time zone for `OnCalendar`: NOT FOUND (grep `timedatectl|Europe/Budapest` in documentation/operations/*.md, RUNBOOK-hub-db-offsite-backup.md, ep0-datastore-copy.md, total-loss-of-dooplex.md). +- cron/crontab/`at ` installs by repo scripts on DooPlex or ep0: NOT FOUND (grep `crontab|/etc/cron|OnCalendar|systemd-run` in felhom.eu/scripts found only the systemd units listed in blocks 26-29 and the box installer). +- Scheduled CI in felhom-controller, felhom-agent, app-catalog-felhom.eu: NOT FOUND (grep `schedule:|cron:` in each `.gitea/workflows/`). +- Measured duration of any hub job: NOT FOUND in repository files; only the snapshot and the OS alarm pass log their own duration. diff --git a/documentation/audits/clock-list-2026-11-11/evidence/helpers/resolved-clock-agent.json b/documentation/audits/clock-list-2026-11-11/evidence/helpers/resolved-clock-agent.json new file mode 100644 index 00000000..8a6fa127 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/helpers/resolved-clock-agent.json @@ -0,0 +1,3124 @@ +{ + "endings": [ + { + "title": "1. Whole-guest backup: due-check and run", + "answers": { + "1": "The in-guest controller asks `handleBackupDue` `AG/internal/localapi/server.go:1134-1217` and starts `handleBackup` `AG/internal/localapi/server.go:878-1015`; the host has no timer for it; the agent runs the vzdump. CUSTOMER.", + "2": "Due when the newest archive's age reaches the cadence, `age >= tier.Cadence` `AG/internal/localapi/server.go:1204-1213`; default `BackupCadence` 24 h `AG/internal/config/config.go:576-580`; the night window is the controller's `AR/07-backup-architecture.md:551`; the agent compares ages only, no time zone.", + "3": "Task wait `defaultPrimaryTierWaitTimeout` 30 min and `defaultExtraTierWaitTimeout` 12 h `AG/internal/config/config.go:464-465`; outer bound is the tier wait plus 15 min `AG/internal/localapi/server.go:971-975`; the copy grows with the guest's size.", + "4": "A tier not backed up stays due at every later ask; the controller has a 48 h safety valve `AR/07-backup-architecture.md:551-552`; the hub judges 72 h without a whole-guest backup `AR/07-backup-architecture.md:579-581`.", + "5": "Same tier in flight answers 202 with the same job `AG/internal/localapi/server.go:913-917`; another tier answers 409 `AG/internal/localapi/server.go:919-925`; the host-wide gate is `TryAcquire` `AG/internal/backup/inflight.go:40-59`, in memory.", + "6": "The clock is wall time, `now` = `time.Now().UTC()` `AG/internal/localapi/server.go:509`; candidate: a clock set back makes the age small, so the tier reads not due until real time catches up; unreadable storage uses the saved success `AG/internal/localapi/server.go:1176-1182`.", + "7": "The hub raises `expected_backup_missed` to the operator `AR/07-backup-architecture.md:579-582`; a flood refused by the cap raises an operator event, rationale `AG/internal/localapi/backup_cap.go:12-35`.", + "8": "TEST `TestBackupDue_R894_RestartThenUnreadableStorage_OldSavedCopyIsDue` `AG/internal/localapi/r894_saved_backup_test.go:86`; DOC `AR/07-backup-architecture.md:420-429`; clock jump: NOBODY KNOWS (grep `^func Test.*(Clock|Future|Skew|Jump|DST)` finds none for the due-check)." + }, + "extra": [] + }, + { + "title": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "answers": { + "1": "The agent runs it after a successful primary, non-manual whole-guest backup: `SetAfterPrimaryBackup` `AG/cmd/felhom-agent/main.go:1390-1398`, trigger at `AG/internal/localapi/server.go:1008-1011`; `Run` `AG/internal/osupdate/leg.go:538-586`. CUSTOMER.", + "2": "90 s after the backup ends `AG/cmd/felhom-agent/main.go:1395`; at most once per `MinGap`, default 20 h `AG/internal/osupdate/leg.go:656-667`; order guest, host, Docker, Proxmox, kernel `AG/internal/osupdate/leg.go:538-586`; no clock slot of its own `AR/11-os-updates.md:378-387`.", + "3": "Wrapper limits: host command 600 s and guest command 1800 s AG/configs/felhom-os-apply:279-284; health wait default 5 min `AG/internal/osupdate/leg.go:810-813`; time grows with the number of packages; measured 23 to 70 s `AR/07-backup-architecture.md:449-450`.", + "4": "A failed or missed backup means no OS leg that night `AR/07-backup-architecture.md:447`; no catch-up found; the hub shows how far behind a box is `AR/11-os-updates.md:406-408`.", + "5": "It runs holding the heavy-op gate taken by the backup `AR/07-backup-architecture.md:444-446`; one pass at a time by flock `tryLockPass` `AG/internal/osupdate/unsent.go:35-52`; a second night run inside 20 h is skipped `AG/internal/osupdate/leg.go:663`.", + "6": "The 20 h gap compares `l.now().Sub(last)` with a time read from a file `AG/internal/osupdate/leg.go:661-662`; candidate: a stamp written while the clock was ahead keeps the leg skipped until real time passes that stamp plus 20 h.", + "7": "The result is posted to the hub by `finish` `AG/internal/osupdate/leg.go:922-939`; operator gets a hub event per update and failure, the household one timeline line `AR/11-os-updates.md:406-423`.", + "8": "TEST `TestOncePerNight` `AG/internal/osupdate/leg_test.go:367` and `TestAfterPrimaryBackup` `AG/internal/localapi/afterbackup_test.go:17`; DOC `AR/11-os-updates.md:366-387`; a stamp from the future: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "answers": { + "1": "Agent: `runKernel` ends the night leg `AG/internal/osupdate/kernel.go:119-178`; at every agent start `KernelAfterBoot` judges the boot `AG/cmd/felhom-agent/main.go:1018`. CUSTOMER (the box restarts at night).", + "2": "Only on trigger night and only when the hub's block says `Tonight` `AG/internal/osupdate/kernel.go:101-115`; the household is mailed the day before, 09:00 to 20:00 Budapest time, by the hub `AR/11-os-updates.md:580-585`.", + "3": "Judge wait `DefaultKernelJudgeWait` 20 min `AG/internal/osupdate/kernel.go:37`, polled every 30 s `AG/internal/osupdate/kernel.go:251`; measured healthy at 68 s and 272 s `AR/11-os-updates.md:570-572`.", + "4": "No mail accepted means no step `AR/11-os-updates.md:584-585`; a step is never retried by itself `AR/11-os-updates.md:581-583`; a box that never comes back raises `host_stale` at 45 min and `host_down` at 90 min `AR/11-os-updates.md:572-573`.", + "5": "At most one per night under the same heavy-op gate `AR/11-os-updates.md:560-562`; never two steps within 20 h (wrapper refusal R22) `AR/11-os-updates.md:557-558`; one self-revert, never retried `AG/internal/osupdate/kernel.go:300-312`.", + "6": "The judge deadline is in-process from `now` = `time.Now()` `AG/internal/osupdate/leg.go:291-296` (monotonic kept); a clock step during the 20 min does not move it. Host starting with a wrong clock: NOT FOUND (grep `timedatectl|chrony|ntp` in `E/scripts/felhom-host-install.sh:1` finds nothing).", + "7": "The hub hears staged before the reboot `AG/internal/osupdate/kernel.go:165`; operator events `os_kernel_step` and `os_kernel_notice` `AR/11-os-updates.md:585-586`; a frozen new kernel needs a person, accepted `AR/11-os-updates.md:534-535`.", + "8": "TEST `TestKernelAfterBoot_UnhealthyRevertsOnceAfterTheWait` `AG/internal/osupdate/kernel_test.go:200`, `TestKernel_NoMailNoStep` `AG/internal/osupdate/kernel_test.go:79`; DOC `AR/11-os-updates.md:530-601`; a real freeze: NOBODY KNOWS `AR/11-os-updates.md:599-601`." + }, + "extra": [] + }, + { + "title": "4. Crash guard: early-boot unit plus hourly re-arm timer", + "answers": { + "1": "Root systemd units installed by the host installer `E/scripts/felhom-host-install.sh:2944-2963`: boot unit `AG/configs/felhom-crash-guard.service:15-16` and timer `AG/configs/felhom-crash-guard-check.timer:5-6`. CUSTOMER (decides if a crashed box restarts).", + "2": "Timer as written: `OnBootSec=15min`, `OnUnitActiveSec=1h` `AG/configs/felhom-crash-guard-check.timer:5-6`; no OnCalendar, no RandomizedDelaySec, no Persistent; boot unit runs before sysinit `AG/configs/felhom-crash-guard.service:8`.", + "3": "Oneshot `AG/configs/felhom-crash-guard-check.service:6-7` with no time limit written; history is cut to 7 days AG/configs/felhom-crash-guard:148, so nothing grows.", + "4": "A missed hourly check only delays the re-arm; a crash before the boot unit runs leaves the box off, the safe side `AR/11-os-updates.md:484-485`; the System page shows the guard `AR/11-os-updates.md:487-488`.", + "5": "Limits `LIMIT=3`, `WINDOW_MINUTES=60`, `REARM_HOURS=24` `AG/configs/crash-guard.conf:3-8`; the 2nd unclean boot in the window trips it AG/configs/felhom-crash-guard:155; no lock between boot and check found (candidate, grep `flock|lock` not run on it: NOT FOUND).", + "6": "Wall clock: `now` is `time.time()` AG/configs/felhom-crash-guard:43-44 and the window test is `now - t <= window` AG/configs/felhom-crash-guard:124, read at early boot; candidate: a clock that is wrong at boot can move unclean boots in or out of the 60 min window.", + "7": "Each unclean boot is an operator mail `host_crash_restart` and a household line; the trip is alarm `host_crash_guard_tripped` `AR/11-os-updates.md:487-488`; the trip mail came 15 min after the boot `AR/11-os-updates.md:489-490`.", + "8": "TEST `test_third_crash_in_an_hour_leaves_the_box_off` `AG/configs/test_felhom_crash_guard.py:75` and `test_rearms_after_24h_of_normal_running` `AG/configs/test_felhom_crash_guard.py:102`; DOC `AR/11-os-updates.md:473-490`; wrong clock at boot: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "5. Scheduled restore-test evaluation", + "answers": { + "1": "Agent goroutine `Run` `AG/internal/backup/schedule.go:148-177`, wired at `AG/cmd/felhom-agent/main.go:1968-2018`; restores an archive into a scratch guest on the host. CUSTOMER (proof the backup opens).", + "2": "First look `DefaultFirstEval` 30 min after agent start `AG/internal/backup/schedule.go:76`, then every `defaultRestoreTestEvalInterval` 6 h `AG/internal/config/config.go:636`; an archive counts after `defaultRestoreTestSettle` 24 h `AG/internal/config/config.go:641`; not windowed, any hour `AR/11-os-updates.md:385-386`.", + "3": "Restore task wait: `RestoreTestPBSRestoreTimeout` 120 min for the PBS tier `AG/internal/config/config.go:584-588`, else the 10 min default `AG/cmd/felhom-agent/main.go:1917-1919`; boot wait `DefaultBootTimeout` 2 min `AG/internal/reconcile/restoretest.go:28`; grows with guest size.", + "4": "The ticker restarts with the agent, so a missed look comes 30 min after the next start `AR/07-backup-architecture.md:583-584`; the rule is per archive, so a tier stays due until proven `AR/07-backup-architecture.md:179-180`; hub proof-age window 7 to 12 days `AR/07-backup-architecture.md:215-217`.", + "5": "Takes the heavy-op gate with `TryAcquire(\"restore-test\")` `AG/internal/backup/schedule.go:228`; busy means deferred to the next 6 h look `AG/internal/backup/schedule.go:230`; proof credit only on a pass `AG/internal/backup/schedule.go:248-255`.", + "6": "`now` is `time.Now().UTC()`, wall clock `AG/internal/backup/schedule.go:119`, used for `settleCutoff` `AG/internal/backup/restoretest_due.go:96-101`; candidate: a clock set back makes no archive look settled; the ticker itself is monotonic.", + "7": "A failed test is an ERROR log `AG/internal/backup/schedule.go:262` and a record the hub turns into `restore_test_failed` for the operator, as the janitor comment says `AG/cmd/felhom-agent/janitor.go:42-44`.", + "8": "TEST `TestR874_FirstEvaluationAfterStart` `AG/internal/backup/r874_first_eval_test.go:17`, `TestDue_DailyTierIsProvedDailyOnItsOwnArchive` `AG/internal/backup/restoretest_due_test.go:139`; DOC `AR/07-backup-architecture.md:174-200`; clock set back: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "6. Controller supervisor (restarts a dead or silent controller)", + "answers": { + "1": "Agent goroutine `WatchControllers` `AG/internal/localapi/controllersupervisor.go:143-163`, started at `AG/cmd/felhom-agent/main.go:1714`. CUSTOMER (the household's dashboard comes back).", + "2": "Every `controllerSupervisorInterval` 30 s `AG/internal/localapi/controllersupervisor.go:54`; restart on the 2nd not-running sweep `AG/internal/localapi/controllersupervisor.go:56`; a running but unhealthy one is alarmed at 6 sweeps and restarted at 10 `AG/internal/localapi/controllersupervisor.go:92-100`.", + "3": "No per-sweep time limit seen in the lines read (NOT FOUND, only `AG/internal/localapi/controllersupervisor.go:1-200` was opened); nothing grows with data; a liveness line every 20 sweeps `AG/internal/localapi/controllersupervisor.go:64`.", + "4": "If the agent is down nothing restarts the controller: measured 33 min of 502 before this existed `AG/internal/localapi/controllersupervisor.go:20-25`; the first sweep is 30 s after agent start (ticker only) `AG/internal/localapi/controllersupervisor.go:153-161`.", + "5": "Brake: `controllerCrashloopMax` 3 in `controllerCrashloopWindow` 15 min, then `controllerCrashloopPause` 30 min `AG/internal/localapi/controllersupervisor.go:59-61`; not during a swap, a parked or locked guest, or a vzdump `AG/internal/localapi/controllersupervisor.go:34-42`.", + "6": "The fast record is in memory, so an agent restart forgets a pause `AG/internal/localapi/controllersupervisor.go:103-105`; the 24 h counter is persisted `AG/internal/localapi/controllersupervisor.go:115-119`; wall or monotonic for the windows: NOT FOUND (the tick body was not opened).", + "7": "Hub mints `controller_restarted_by_agent` (info) and `controller_crashloop` (error, operator only) from the host report `AG/internal/localapi/controllersupervisor.go:44-49`; a warning at the 5th restart in 24 h `AG/internal/localapi/controllersupervisor.go:82-83`.", + "8": "TEST `TestControllerSupervisor_KilledControllerIsRestarted` `AG/internal/localapi/controllersupervisor_test.go:101`, `TestSupervisor_UnresponsiveIsAlarmedThenRestarted` `AG/internal/localapi/supervisor_unresponsive_test.go:32`; DOC `AR/03-host-agent.md:186-192`." + }, + "extra": [] + }, + { + "title": "7. Guest-power watchdog (starts a guest that should be running)", + "answers": { + "1": "Agent goroutine `WatchGuestPower` `AG/internal/localapi/guestpower.go:89-105`, started at `AG/cmd/felhom-agent/main.go:1710`. CUSTOMER (the whole box's apps are off while the guest is stopped).", + "2": "Every `guestPowerInterval` 60 s `AG/internal/localapi/guestpower.go:52`; acts only on a stopped, unlocked guest with onboot set `AG/internal/localapi/guestpower.go:167-180`; first sweep one interval after start (ticker only) `AG/internal/localapi/guestpower.go:95-102`.", + "3": "At most `guestPowerMaxAttempts` 3 starts `AG/internal/localapi/guestpower.go:56` with waits of 1, 2 and 4 min, `guestPowerBackoff` `AG/internal/localapi/guestpower.go:75`; about 7 min in all `AG/internal/localapi/guestpower.go:71-74`.", + "4": "Before it existed a guest stayed down 9 min 47 s until a person started it `AG/internal/localapi/guestpower.go:14-18`; with the agent down nobody starts it (the platform's onboot acts only at host boot `AG/internal/localapi/guestpower.go:42-43`).", + "5": "Never starts under a running vzdump and fails safe on doubt `AG/internal/localapi/guestpower.go:182-194`; leaves a locked guest to the stale-lock path `AG/internal/localapi/guestpower.go:176-180`; skips a guest the operator is rebooting `AG/internal/localapi/guestpower.go:157-159`.", + "6": "Attempt state is in memory on purpose, so an agent restart tries again at once `AG/internal/localapi/guestpower.go:77-80`; wall or monotonic for `nextAt`: NOT FOUND (`guestPowerDue` `AG/internal/localapi/guestpower.go:226` was not opened).", + "7": "Candidate: the give-up is an ERROR journal line only `AG/internal/localapi/guestpower.go:205-206`; a hub event for it is NOT FOUND (grep `hub\\.|Report|report` in `AG/internal/localapi/guestpower.go:1` finds no sender).", + "8": "TEST `TestGuestPower_RetryIsBoundedThenEscalates` `AG/internal/localapi/guestpower_test.go:162`, `TestGuestPower_InFlightBackupIsNotDisturbed` `AG/internal/localapi/guestpower_test.go:119`; who hears the give-up: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "answers": { + "1": "Agent goroutine `Run` `AG/internal/hub/loop.go:107-138`, built at `AG/cmd/felhom-agent/main.go:958`; it carries every status the hub alarms on. OPERATOR and CUSTOMER (alarms depend on it). Merged here: the reconcile engine, fast tick and poke.", + "2": "`PollSeconds` default 900 `AG/internal/config/config.go:797`; first report at once, then a ticker `AG/internal/hub/loop.go:107-138`; the hub may change it, clamped by `MinPollSeconds` 60 and `MaxPollSeconds` 3600 `AG/internal/hub/loop.go:11-12`; fast tick `DefaultInterval` 30 s `AG/internal/fasttick/fasttick.go:24`.", + "3": "Per-request limit `TimeoutSeconds` default 30 `AG/internal/config/config.go:798`; the system facts in it are read at most every 10 min `AG/cmd/felhom-agent/main.go:4149`; the report grows with guests and drives.", + "4": "A failed cycle is logged and the interval kept `AG/internal/hub/loop.go:162-164`; no backlog, the next report replaces it; the hub raises `host_stale` at 45 min and `host_down` at 90 min `AR/11-os-updates.md:572-573`.", + "5": "Extra reports come by the trigger channel `AG/internal/hub/loop.go:125-135`; a poke is debounced by `DebounceWindow` 2 s `AG/internal/poke/poke.go:40`; the reconcile engine runs at once and then on the start-time interval `AG/internal/reconcile/engine.go:332-344`, `AG/cmd/felhom-agent/main.go:1525`.", + "6": "The ticker is monotonic; candidate: the reconcile engine keeps the interval read at start `AG/cmd/felhom-agent/main.go:969` and does not follow the hub's override `AG/internal/hub/loop.go:188-196`. A suspended host: NOT FOUND for the agent (the controller has a rule `AR/07-backup-architecture.md:555-558`).", + "7": "The operator, by the hub's staleness alarms `AR/11-os-updates.md:572-573`; the household hears cannot-be-reached at most once per 7 days `AR/07-backup-architecture.md:582-583`.", + "8": "TEST `TestLoop_RunImmediateAndResilientAfterError` `AG/internal/hub/loop_test.go:140`, `TestClampInterval` `AG/internal/hub/loop_test.go:81`; DOC `AR/03-host-agent.md:276-279`." + }, + "extra": [] + }, + { + "title": "9. Signed jobs: validity window against the host clock (agent updates ride it)", + "answers": { + "1": "Agent `OnEnvelope` runs pending signed jobs when a heartbeat says there are some `AG/internal/signedjobs/runner.go:93-102`; an agent update arrives only this way. OPERATOR (and CUSTOMER: fixes reach the box this way).", + "2": "Checked at every hub cycle (item 8); a job is valid from issue minus `DefaultClockSkew` 2 min `AG/internal/authz/verifier.go:21` to its expiry, and a window over `MaxWindow` 24 h is refused `AG/internal/authz/verifier.go:26`.", + "3": "The self-update download has a 5 min client limit `AG/internal/selfupdate/executor.go:73`; the new binary must run `DwellSeconds` 60 s before it commits `AG/internal/config/config.go:97-99`.", + "4": "A job not picked up before its expiry is refused as expired `AG/internal/authz/verifier.go:161-162`; a commit that fails is retried at the next start `AG/internal/selfupdate/commit.go:122-125`; who notices an expired job: NOT FOUND in the agent (hub side).", + "5": "One run at a time, `RunOnce` single-flight `AG/internal/signedjobs/runner.go:107-115`; a replay is refused by the nonce store `AG/internal/authz/verifier.go:166`.", + "6": "Wall clock: candidate, a host clock that is behind refuses every job as not yet valid `AG/internal/authz/verifier.go:155-156`, one that is ahead refuses them as expired `AG/internal/authz/verifier.go:161-162`; so a box with a wrong clock cannot be updated by signed job.", + "7": "NOT FOUND in the agent source read (grep `not_yet|NotYet|expired` shown only in `AG/internal/authz/verify_test.go:140-155`); the result path to the hub was not opened.", + "8": "TEST `TestVerify_RejectsPerStage` `AG/internal/authz/verify_test.go:82`, `TestRunner_ExpiredRejected` `AG/internal/signedjobs/runner_test.go:168`, `TestManager_ShutdownBeforeDwellLeavesPending` `AG/internal/selfupdate/commit_test.go:102`." + }, + "extra": [] + }, + { + "title": "10. Storage watchdog and the 20-second mount and bind re-assert", + "answers": { + "1": "Agent: the watchdog `Run` `AG/internal/storage/watchdog.go:196-215` wired at `AG/cmd/felhom-agent/main.go:1142`, and a 20 s ticker that re-asserts mounts and guest binds `AG/cmd/felhom-agent/main.go:1613-1631`. CUSTOMER (a dropped USB drive).", + "2": "`DefaultWatchdogInterval` 8 s and `DefaultWatchdogDebounce` 30 s `AG/internal/storage/watchdog.go:15-16`; setting `WatchdogIntervalSeconds`, 0 means the default `AG/internal/config/config.go:770-778`; the re-assert is 20 s, below the controller's 30 s tick `AG/cmd/felhom-agent/main.go:1612-1614`.", + "3": "No per-tick limit found in the lines read (NOT FOUND, grep `WithTimeout` not run on the watchdog); the work grows with the number of enrolled drives.", + "4": "A missed tick is made up by the next one; with the agent down a dropped drive is seen only at the next 15 min report, the reason the watchdog exists `AR/03-host-agent.md:461`.", + "5": "A flapping drive backs off: `backoffWindow` `AG/internal/storage/watchdog.go:183-192`, alert after 4 failed cycles `AG/internal/storage/watchdog.go:77-79`; the retry test is `now.Sub(last)` `AG/internal/storage/watchdog.go:290`.", + "6": "`now` is `time.Now().UTC()`, wall clock `AG/internal/storage/watchdog.go:160`; candidate: a clock set back stretches the back-off, set forward ends it early; the tickers are monotonic.", + "7": "A drop requests an immediate hub report through the trigger channel `AG/internal/hub/loop.go:125-135`; the hub then decides who is mailed (hub side, not read).", + "8": "TEST `TestWatchdog_DebounceCoalescesFlaps` `AG/internal/storage/watchdog_test.go:112`, `TestWatchdog_Reconcile_FlappingBacksOffAndCaps` `AG/internal/storage/watchdog_test.go:460`; the 20 s ticker: NOBODY KNOWS (no test found for the goroutine at `AG/cmd/felhom-agent/main.go:1613`)." + }, + "extra": [] + }, + { + "title": "11. Weekly guest disk trim", + "answers": { + "1": "Agent goroutine `Run` `AG/internal/fstrim/fstrim.go:186-199`, started at `AG/cmd/felhom-agent/main.go:1741`; runs `pct fstrim` per owned running guest. CUSTOMER (a full thin pool turns every guest read-only `AG/internal/fstrim/fstrim.go:3-5`).", + "2": "Due from Wednesday 10:00 host-local, starts only 10:00 to 20:59 `AG/internal/fstrim/fstrim.go:42-46`; looks 5 min after start then hourly, `FirstTickDelay` and `TickInterval` `AG/internal/fstrim/fstrim.go:48-50`; zone is `time.Local` `AG/internal/fstrim/fstrim.go:110`.", + "3": "`PerGuestTimeout` 30 min `AG/internal/fstrim/fstrim.go:52`; measured 24.4 s for 84 GiB `AG/internal/fstrim/fstrim.go:6`; grows with freed blocks.", + "4": "A box off on Wednesday catches up at its next eligible hour `AG/internal/fstrim/fstrim.go:10-11`; a failure is retried at most `MaxAttemptsPerWeek` 3 times `AG/internal/fstrim/fstrim.go:46`; the result rides the host report `AG/internal/fstrim/fstrim.go:326-330`.", + "5": "Holds the heavy-op gate for the whole pass; busy means deferred to the next hour `AG/internal/fstrim/fstrim.go:247-251`; the persisted record stops a second trim after a restart `AG/internal/fstrim/fstrim.go:17`.", + "6": "Wall clock in host-local time `AG/internal/fstrim/fstrim.go:204`; candidate: an attempt stamped while the clock was ahead is not before the anchor, so `due` stays false until real time passes it `AG/internal/fstrim/fstrim.go:179-182`. No zone set by the installer: NOT FOUND (grep `timedatectl|set-timezone`).", + "7": "A failed trim is a WARN journal line `AG/internal/fstrim/fstrim.go:318-319` and a field on the host report; who is mailed: NOT FOUND in the agent (hub side).", + "8": "TEST `TestPassNeverRunsInTheNight` `AG/internal/fstrim/fstrim_test.go:161`, `TestPassDefersToAHeavyOperationAndRetriesNextHour` `AG/internal/fstrim/fstrim_test.go:175`, `TestWeekAnchorIsTheLastWednesdayTen` `AG/internal/fstrim/fstrim_test.go:104`; DST and clock jump: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "12. Weekly upload line test", + "answers": { + "1": "Agent goroutine `Run` `AG/internal/linetest/linetest.go:256-268`, started at `AG/cmd/felhom-agent/main.go:1762`; a 10 s iperf3 upload over the tunnel. CUSTOMER (sets the torrent app's upload limit `AG/internal/linetest/linetest.go:3-4`).", + "2": "Due from Tuesday 10:00 host-local, starts 10:00 to 15:59 `AG/internal/linetest/linetest.go:47-49`; each host from its own minute, `SlotMinute` `AG/internal/linetest/linetest.go:148-152`; ticks every minute after a 5 min delay `AG/internal/linetest/linetest.go:60-61`; zone `time.Local` `AG/internal/linetest/linetest.go:131`.", + "3": "Three runs of 10 s, each bounded by `RunTimeout` 25 s, 30 s apart `AG/internal/linetest/linetest.go:53-57`; nothing grows with data.", + "4": "A box that was off catches up on a later day inside the window `AG/internal/linetest/linetest.go:10-11`; the hub page turns amber when the last success is over 15 days old `AR/11-os-updates.md:409-411`.", + "5": "Needs 10 quiet minutes and the heavy-op gate `AG/internal/linetest/linetest.go:316-325`; server busy is retried at the next tick, at most `MaxTriesPerDay` 6 `AG/internal/linetest/linetest.go:58`; no tunnel means no test `AG/internal/linetest/linetest.go:297-300`.", + "6": "Wall clock, host-local `AG/internal/linetest/linetest.go:281`; candidate: `Due` is false while `LastOKAt` is not before the anchor, so a success stamped from the future blocks tests until then `AG/internal/linetest/linetest.go:211-213`; a sample gap over 3 min is not quiet `AG/internal/linetest/linetest.go:244-245`.", + "7": "The operator, by the hub System page cell judged on the last success, not the attempt `AR/11-os-updates.md:409-411`; nobody is mailed as far as the agent source shows (NOT FOUND, hub side).", + "8": "TEST `TestDueWeeklyFromTuesday` `AG/internal/linetest/linetest_test.go:206`, `TestNeverTestsInTheNightOrBeforeTuesday` `AG/internal/linetest/linetest_test.go:363`, `TestSlotDelaysTheFirstStartOnTheDueDayOnly` `AG/internal/linetest/linetest_test.go:189`." + }, + "extra": [] + }, + { + "title": "13. PBS datastore verify loop", + "answers": { + "1": "Agent goroutine `Run` `AG/internal/pbs/verify.go:64-83`, cadence wired at `AG/cmd/felhom-agent/main.go:1188`; asks the PBS server to verify the box's snapshots. CUSTOMER (is the off-site copy readable).", + "2": "`DefaultVerifyCadence` 6 h `AG/internal/pbs/verify.go:13`; setting through `PBSVerifyCadence` `AG/internal/config/config.go:710-719`; first pass at once at agent start, then the ticker `AG/internal/pbs/verify.go:64-83`; no time zone.", + "3": "One verify is polled every 2 s for at most 30 min `AG/internal/pbs/verify.go:107`; grows with the datastore's size.", + "4": "No catch-up state: every agent start verifies again at once `AG/internal/pbs/verify.go:64-83`; a negative cadence switches it off, test `TestVerifyLoop_DisabledByNegativeCadence` `AG/internal/pbs/verify_test.go:87`.", + "5": "Candidate: it is not behind the heavy-op gate, the lanresolver comment calls it \"not journaled/gated\" `AG/internal/lanresolver/loop.go:17`; so it can run beside a backup; NOT FOUND for a lock (grep `TryAcquire` not run on `AG/internal/pbs/verify.go:1`).", + "6": "Ticker only, monotonic; no wall-clock decision found in `AG/internal/pbs/verify.go:64-83`.", + "7": "Failed-verify snapshots are an ERROR journal line `AG/internal/pbs/verify.go:127` and recorded state for the report; who is mailed: NOT FOUND in the agent (hub side).", + "8": "TEST `TestVerifyLoop_FailedVerifyRecorded` `AG/internal/pbs/verify_test.go:52`; a verify longer than 30 min: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "14. Janitor: leftover scratch guests and stale backup locks", + "answers": { + "1": "Agent goroutine `runJanitor` `AG/cmd/felhom-agent/janitor.go:69-81`, started at `AG/cmd/felhom-agent/main.go:1725`. CUSTOMER (a stale lock blocked whole-box backups for five hours `AG/cmd/felhom-agent/janitor.go:14-17`).", + "2": "Every `janitorInterval` 10 min `AG/cmd/felhom-agent/janitor.go:18`; ticker only, so the first pass is 10 min after agent start `AG/cmd/felhom-agent/janitor.go:71-78`; no time zone.", + "3": "No time limit on a pass found in `AG/cmd/felhom-agent/janitor.go:34-66`; work grows with the number of leftover scratch guests.", + "4": "A missed pass is retried 10 min later; a scratch guest that cannot be removed after the retry limit is reported as a failed restore-test record `AG/cmd/felhom-agent/janitor.go:41-54`.", + "5": "The stale-lock sweep runs only holding the heavy-op gate, else it is skipped this pass `AG/cmd/felhom-agent/janitor.go:57-64`; rule stated at `AG/cmd/felhom-agent/janitor.go:30-33`.", + "6": "Ticker only, monotonic; a clock step must not kill a live task: `TestVzdumpInFlight_ClockStepCannotKillALiveTask` `AG/internal/localapi/stalelock_inflight_test.go:97`.", + "7": "The operator, through the hub's `restore_test_failed`, once per distinct archive `AG/cmd/felhom-agent/janitor.go:42-44`.", + "8": "TEST `TestJanitor_StaleLockSweepWaitsForTheHeavyGate` `AG/cmd/felhom-agent/janitor_test.go:40`, `TestJanitor_GaveUpIsReportedAsAFailure` `AG/cmd/felhom-agent/janitor_test.go:22`." + }, + "extra": [] + }, + { + "title": "15. Guest network watchdog (DHCP client heal)", + "answers": { + "1": "Agent goroutine `Watch` `AG/internal/guestnet/watchdog.go:105-121`, built at `AG/cmd/felhom-agent/main.go:1686`. CUSTOMER (a guest without network).", + "2": "`DefaultInterval` 60 s `AG/internal/guestnet/watchdog.go:23`; immediate first sweep `AG/internal/guestnet/watchdog.go:111`; settings interval 60, `MinHealIntervalSeconds` 600, `MaxHealsPerHour` 3 `AG/internal/config/config.go:176-196`.", + "3": "No per-probe limit seen in `AG/internal/guestnet/watchdog.go:1-200` (NOT FOUND); nothing grows with data.", + "4": "A missed sweep is made up a minute later; beyond 3 heals an hour it only reports `AG/internal/guestnet/watchdog.go:26-28`.", + "5": "Two consecutive bad probes before a heal, `requiredBadProbes` `AG/internal/guestnet/watchdog.go:32-33`; 10 min between heals, `DefaultMinHealInterval` `AG/internal/guestnet/watchdog.go:25`; boot-race guard `DefaultSettle` 3 min `AG/internal/guestnet/watchdog.go:29-31`.", + "6": "`now` is plain `time.Now` `AG/internal/guestnet/watchdog.go:80`, so in-process gaps are monotonic; heal history is in memory `AG/internal/guestnet/watchdog.go:56-62`.", + "7": "A report per guest with `HealsLastHour` and `CheckedAt` `AG/internal/guestnet/watchdog.go:166-170`; who is mailed: NOT FOUND in the agent (hub side).", + "8": "TEST `TestWatchdog_DampingCeilings` `AG/internal/guestnet/watchdog_test.go:360`, `TestWatchdog_MinimumIntervalBetweenHeals` `AG/internal/guestnet/watchdog_test.go:384`, `TestWatchdog_BootRacesObserveOnly` `AG/internal/guestnet/watchdog_test.go:416`." + }, + "extra": [] + }, + { + "title": "16. Host networking self-heal", + "answers": { + "1": "Agent goroutine `Watch` `AG/internal/selfheal/selfheal.go:67-82`, interval set at `AG/cmd/felhom-agent/main.go:1648`; starts `networking.service` on an appliance only. CUSTOMER.", + "2": "One pass at agent start, then every 60 s `AG/internal/selfheal/selfheal.go:67-82`, `AG/cmd/felhom-agent/main.go:1648`; no setting found for the interval (hard-coded).", + "3": "A remedy is 3 start attempts with waits of 10, 30 and 60 s, `networkingBackoffs` `AG/internal/selfheal/selfheal.go:114-116`.", + "4": "A missed pass is made up a minute later; on a BYO host the remedy is skipped and only a WARN is written `AG/internal/selfheal/selfheal.go:54-58`.", + "5": "No lock seen; the pass is sequential inside one goroutine `AG/internal/selfheal/selfheal.go:46-64`.", + "6": "Ticker only, monotonic; no wall-clock decision in `AG/internal/selfheal/selfheal.go:67-82`.", + "7": "A give-up is an ERROR journal line `AG/internal/selfheal/selfheal.go:61`; with networking down the hub cannot be told, so the hub's stale alarm is the only signal `AR/11-os-updates.md:572-573`.", + "8": "TEST `TestNetworkingHeal_GivesUpAfterAllAttempts` `AG/internal/selfheal/selfheal_test.go:109`, `TestManager_ByoNeverRemediates` `AG/internal/selfheal/selfheal_test.go:32`." + }, + "extra": [] + }, + { + "title": "17. Management-plane watchdog timer (keeps SSH reachable)", + "answers": { + "1": "Root systemd timer and oneshot installed by the host installer `E/scripts/felhom-host-install.sh:3067-3076`; script `AG/configs/felhom-mgmt-watchdog.sh:26-68`; works without the agent `AG/configs/felhom-mgmt-watchdog.service:3-7`. OPERATOR.", + "2": "As written: `OnBootSec=30s`, `OnUnitActiveSec=60s`, `AccuracySec=5s`, `Persistent=true` `AG/configs/felhom-mgmt-watchdog.timer:11-14`; no OnCalendar, no RandomizedDelaySec.", + "3": "No time limit written in the unit `AG/configs/felhom-mgmt-watchdog.service:16-18`; the script is a few stat and systemctl calls `AG/configs/felhom-mgmt-watchdog.sh:36-61`.", + "4": "A missed tick is retried by the next one `AG/configs/felhom-mgmt-watchdog.sh:23-24`; the heal budget is one tick `AG/configs/felhom-mgmt-watchdog.timer:3-6`.", + "5": "Idempotent: a healthy directory is never touched `AG/configs/felhom-mgmt-watchdog.sh:13-15`; candidate: `Persistent=true` only acts on OnCalendar timers in systemd, and this timer has none `AG/configs/felhom-mgmt-watchdog.timer:11-14`.", + "6": "Monotonic timer settings only `AG/configs/felhom-mgmt-watchdog.timer:11-12`; the heal marker is a UTC stamp from `date -u` `AG/configs/felhom-mgmt-watchdog.sh:65`.", + "7": "The marker lets the agent report the heal and the hub raise a warning `AG/configs/felhom-mgmt-watchdog.sh:18-19`, `AG/configs/felhom-mgmt-watchdog.sh:63`.", + "8": "SOURCE `AG/configs/felhom-mgmt-watchdog.sh:1-68`; a test for the script: NOT FOUND (a listing of the agent configs folder shows no `test_felhom_mgmt_watchdog` file); DOC comment `AG/configs/felhom-mgmt-watchdog.timer:3-6`." + }, + "extra": [] + }, + { + "title": "18. Agent service restart, start limit and rollback", + "answers": { + "1": "systemd restarts the agent: `Restart=on-failure`, `RestartSec=5s` `AG/configs/felhom-agent.service:41-42`; drop-in limits `AG/configs/felhom-agent-limits.conf:17-19`. OPERATOR (every loop in this list dies with the agent).", + "2": "`StartLimitIntervalSec=120`, `StartLimitBurst=4`, `OnFailure=felhom-agent-rollback.service` `AG/configs/felhom-agent-limits.conf:17-19`; a crash-looping binary reaches failed about 20 s after the first crash `AG/configs/felhom-agent-limits.conf:5-8`.", + "3": "No WatchdogSec in the unit (grep `WatchdogSec` in `AG/configs/felhom-agent.service:1` finds nothing); a new binary commits after a 60 s dwell `AG/internal/selfupdate/commit.go:40-42`.", + "4": "After the start limit the unit stays failed and nothing on the host restarts it (candidate); the hub's `host_stale` at 45 min is the signal `AR/11-os-updates.md:572-573`.", + "5": "The rollback unit fires on every crash and is guarded by a pending marker `AG/configs/felhom-agent-limits.conf:14-15`; a shutdown before the dwell leaves the update pending `AG/internal/selfupdate/commit.go:112-116`.", + "6": "Candidate: the design text says `Restart=always` and a systemd watchdog that kills a hung agent `AR/03-host-agent.md:72`; the unit has `Restart=on-failure` and no watchdog `AG/configs/felhom-agent.service:41-42`, so a hung (not crashed) agent is not restarted.", + "7": "The operator, by the hub's stale and down alarms only `AR/11-os-updates.md:572-573`; a restore-test crash loop that never evaluates is covered by `TestR874_CrashLoopNeverEvaluates` `AG/internal/backup/r874_first_eval_test.go:44`.", + "8": "SOURCE `AG/configs/felhom-agent-limits.conf:5-15` (measured values quoted there); TEST `TestManager_ShutdownBeforeDwellLeavesPending` `AG/internal/selfupdate/commit_test.go:102`; a hung agent: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "answers": { + "1": "Agent goroutines of one shape: tunnel `Run` `AG/internal/wgtunnel/loop.go:62-78`, PBS-DR `AG/internal/pbsdr/loop.go:62-77`, OOB sshd `AG/internal/felhomsshd/loop.go:67-80`, LAN resolver `AG/internal/lanresolver/loop.go:35-52`. CUSTOMER (off-site backup path) and OPERATOR. Four loops merged.", + "2": "Tunnel, PBS-DR and OOB every 60 s `AG/internal/config/config.go:132-133`, `AG/cmd/felhom-agent/main.go:1504`, `AG/internal/config/config.go:74-75`; resolver every 5 min `AG/internal/lanresolver/loop.go:27-29`; each applies at once at start and on a desired-state nudge.", + "3": "No per-pass limit read (NOT FOUND, manager bodies not opened); nothing grows with data.", + "4": "A missed pass is made up at the next tick; a handshake older than `StaleAfterSeconds` 180 re-resolves the endpoint `AG/internal/config/config.go:124-127`, `AG/internal/config/config.go:138-140`.", + "5": "A handshake stamped in the future for `rekeyAfterDefault` 15 min makes a new key, at most once per `rekeyMinGap` 1 h `AG/internal/wgtunnel/rekey.go:36-38`; the hour bound survives a restart, the 15 min count does not `AR/03-host-agent.md:376-378`.", + "6": "A box whose clock once ran ahead is locked out of the off-site endpoint until real time passes that moment `AR/03-host-agent.md:142`; healed by the re-key `AR/03-host-agent.md:144-150`; known gap: a reboot after the clock was corrected forgets the future stamp `AR/03-host-agent.md:155-156`.", + "7": "The heartbeat carries `handshake_future_s` and the hub alarms on it `AR/03-host-agent.md:142`.", + "8": "TEST `TestD30_AFutureHandshakeThatStaysReKeys` `AG/internal/wgtunnel/rekey_test.go:35`, `TestD30_ALastReKeyFromTheFutureDoesNotBlock` `AG/internal/wgtunnel/rekey_test.go:213`, `TestR1204_AFutureHandshakeIsReported` `AG/internal/wgtunnel/manager_test.go:704`; measured live `AR/03-host-agent.md:151-155`." + }, + "extra": [] + }, + { + "title": "20. Whole-guest backup cap (rolling windows)", + "answers": { + "1": "Agent local API check `capRefuses` before a backup starts `AG/internal/localapi/server.go:929`; rule `backupCadenceCap` `AG/internal/localapi/backup_cap.go:41-53`. CUSTOMER (protects older copies from a flood of new ones).", + "2": "Tier of 24 h or less: 3 successes per rolling 24 h; longer tier: 1 per the larger of cadence minus 24 h and half the cadence `AG/internal/localapi/backup_cap.go:12-35`; a refusal holds for `backupCapRefusalHold` 24 h `AG/internal/localapi/backup_cap.go:38`.", + "3": "A lookup, no duration of its own; the success list is bounded by the window `AG/internal/localapi/backup_cap.go:41-53`.", + "4": "Does not apply to a missed run: failures do not count, `TestBackupCap_FailuresDoNotCount` `AG/internal/localapi/backup_cap_test.go:266`.", + "5": "The scheduled daily run is never refused, `TestBackupCap_TheScheduledDayIsNeverRefused` `AG/internal/localapi/backup_cap_test.go:220`; the count survives a restart, `TestBackupCap_SurvivesARestart` `AG/internal/localapi/backup_cap_test.go:248`.", + "6": "Wall clock from the server's `now` `AG/internal/localapi/server.go:509`; candidate: successes stamped while the clock was ahead stay inside the rolling window longer and could refuse real runs; NOBODY KNOWS.", + "7": "The operator, by event `whole_guest_backup_refused` named in `AG/internal/localapi/backup_cap.go:12-35`; test `TestBackupCap_RefusalRaisesTheOperatorSignal` `AG/internal/localapi/backup_cap_test.go:285`.", + "8": "TEST `TestBackupCap_LocalFloodRunsAtMostThreeADay` `AG/internal/localapi/backup_cap_test.go:198`, `TestBackupCap_OffsiteFloodKeepsAPreCompromiseCopy` `AG/internal/localapi/backup_cap_test.go:151`." + }, + "extra": [] + }, + { + "title": "21. Operator restart and reboot limits", + "answers": { + "1": "Agent local API rules for the operator's hub buttons: `opRebootEvery` `AG/internal/localapi/opreboot.go:46`, `opBrakeRefusal` `AG/internal/localapi/opactions.go:42`. OPERATOR (the household sees a notice line).", + "2": "At most one operator reboot per guest per hour `AG/internal/localapi/opreboot.go:45-46`; the household's dashboard shows the line for `opRebootNoticeFor` 24 h `AG/internal/localapi/opreboot.go:47-48`; a brake refusal holds 30 min `AG/internal/localapi/opactions.go:42`.", + "3": "`opRebootTimeout` 5 min bounds `pct reboot` `AG/internal/localapi/opreboot.go:49-50`; the health wait after a restart is `defaultOpHealthWait` 3 min, polled every 2 s `AG/internal/localapi/opactions.go:46-49`.", + "4": "Not a scheduled job: nothing runs if the operator does not press; a result is forgotten after `opForgetAfter` 24 h `AG/internal/localapi/opactions.go:44`.", + "5": "Refused during a swap, a heavy operation or a busy controller `AG/internal/localapi/opreboot.go:23-31`; the heavy-op gate is held until the guest is up `AG/internal/localapi/opreboot.go:32-34`; the busy file counts as fresh for `controllerBusyFresh` 90 s `AG/internal/localapi/opreboot.go:59-61`.", + "6": "The reboot record is persisted with a wall-clock time `AG/internal/localapi/opreboot.go:64-70`; candidate: a clock set back after a reboot extends the one-hour refusal; NOBODY KNOWS.", + "7": "The operator reads the result of the press; the household reads the line on its dashboard `AG/internal/localapi/opreboot.go:36-39`.", + "8": "TEST stated as \"Pinned by opreboot_test.go\" `AG/internal/localapi/opreboot.go:39` (the test names were not opened); DOC `AR/03-host-agent.md:236-239`." + }, + "extra": [] + }, + { + "title": "22. Unsent OS-update report sender", + "answers": { + "1": "Agent goroutine `SendUnsentLoop` `AG/internal/osupdate/unsent.go:71-82`, started at `AG/cmd/felhom-agent/main.go:1009`; posts OS-update results the hub has not received. OPERATOR.", + "2": "Every 5 min `AG/cmd/felhom-agent/main.go:1009`, by `time.After` `AG/internal/osupdate/unsent.go:79`; one send at start, `SendUnsent` `AG/internal/osupdate/unsent.go:56-64`.", + "3": "A report post has a 1 min limit in `finish` `AG/internal/osupdate/leg.go:922-939`; the kept copies grow only while the hub is away.", + "4": "Hub away keeps the copy for a later pass, `TestR868_HubAwayKeepsTheCopy` `AG/internal/osupdate/unsent_test.go:87`.", + "5": "The sender stays off while a pass holds the flock, `tryLockPass` `AG/internal/osupdate/unsent.go:35-52`; test `TestR868_RunningPassKeepsTheSenderOff` `AG/internal/osupdate/unsent_test.go:110`.", + "6": "`time.After` is monotonic `AG/internal/osupdate/unsent.go:79`; no wall-clock decision found in `AG/internal/osupdate/unsent.go:56-82`.", + "7": "The operator sees the result late, at most 5 min after the hub is back; no alarm of its own found (NOT FOUND, grep `Warn|Error` not run on `AG/internal/osupdate/unsent.go:1`).", + "8": "TEST `TestR868_ACopyWrittenAfterTheStartIsSentByTheLoop` `AG/internal/osupdate/unsent_test.go:127`, `TestR868_KilledPassIsReportedAtStart` `AG/internal/osupdate/unsent_test.go:20`." + }, + "extra": [] + }, + { + "title": "23. Local-API certificate lifetime (a ten-year deadline)", + "answers": { + "1": "The agent makes a self-signed leaf once, `EnsureLeaf` `AG/internal/localapi/cert.go:38-52`; every guest's controller pins its fingerprint `AG/internal/localapi/cert.go:21-24`. CUSTOMER (controller to agent calls, the backup trigger included).", + "2": "`certValidity` is 10 years `AG/internal/localapi/cert.go:25`; start is backdated 1 h for clock skew `AG/internal/localapi/cert.go:65`; no renewal job: rotation is an operator action `AG/internal/localapi/cert.go:23-24`.", + "3": "Not a running job; nothing grows.", + "4": "No timer renews it; what happens at expiry and who notices: NOT FOUND (grep `NotAfter|expir` in `AG/internal/localapi/cert.go:1` shows only the creation at `AG/internal/localapi/cert.go:70`).", + "5": "A second generation would break every issued pin; it is logged loudly `AG/internal/localapi/cert.go:34-37`.", + "6": "Candidate: a leaf made while the host clock was far ahead has a start date in the future; whether the controller checks dates or only the pin is NOT FOUND here (controller side not read).", + "7": "NOT FOUND: no expiry warning in the agent source read.", + "8": "SOURCE `AG/internal/localapi/cert.go:21-25`; expiry behaviour: NOBODY KNOWS. \u00b7 Guest swap check runs once, 1 min after agent start, with a 2 min limit `AG/cmd/felhom-agent/main.go:1021-1034`. \u00b7 System facts for the host report are read at most every 10 min `AG/cmd/felhom-agent/main.go:4149`. \u00b7 A backup-store grant is re-granted at most once per hour per tier, `storeGrantRepairMinInterval` `AG/cmd/felhom-agent/main.go:513`. \u00b7 Escrow ceremony: claim lives 10 min, confirm 30 min, `escrowClaimTTL` and `escrowConfirmTTL` `AG/internal/localapi/escrow_ceremony.go:158-166`. \u00b7 A network change must verify within 95 s or it is rolled back, `netVerifyDeadline` `AG/internal/localapi/netverifyjob.go:51`. \u00b7 Host power-off is refused before 10 min of uptime, `MinUptime` `AG/internal/localapi/hostpoweroff.go:76`. \u00b7 Controller swap verify: 90 s, polled every 3 s `AG/internal/localapi/controllerswap.go:86-87`. \u00b7 Guest hook phases are bounded at 30 s each `AG/cmd/felhom-agent/main.go:88-98`. \u00b7 Proxmox task wait polls with a doubling, capped interval, `WaitTask` `AG/internal/proxmox/task.go:110-141`. \u00b7 Thin-pool check reads twice with a 3 s pause `AG/internal/thincheck/thincheck.go:183`. \u00b7 Poke listener re-binds every 5 s, `rebindInterval` `AG/internal/poke/poke.go:44`. \u00b7 Backup snapshot watch ticks every 1 s during a vzdump, `snapshotWatchInterval` `AG/internal/backup/runner.go:172`. \u00b7 Proxmox package step waits up to 2 min for tasks to drain, `pveDrainWait` `AG/internal/osupdate/leg.go:589`. \u00b7 Restore-test scratch teardown has a 2 min limit detached from the caller `AG/internal/reconcile/restoretest.go:485`. \u00b7 Local whole-guest retention default is 3 daily copies, `defaultLocalBackupKeepLast` `AG/internal/config/config.go:541`. \u00b7 Any cron entry or `OnCalendar` timer on the host: grep `OnCalendar|cron` over the agent configs folder timers, services and shell files finds none; the only two timers are items 4 and 17. \u00b7 Any time-zone or time-sync step in the installer: grep `timedatectl|chrony|timesyncd|set-timezone|ntp` in `E/scripts/felhom-host-install.sh:1` finds nothing; items 11 and 12 use the host's local zone as found. \u00b7 A systemd watchdog for the agent: grep `WatchdogSec` over the agent configs folder finds nothing, against `AR/03-host-agent.md:72`. \u00b7 A rule in the agent for a suspended host (laptop lid): not searched beyond the controller's rule `AR/07-backup-architecture.md:555-558`; grep `^func Test.*(Clock|Future|Skew|Jump|DST|Monotonic)` in the agent repository finds only the tunnel re-key and stale-lock tests. \u00b7 Other files under the felhom.eu scripts folder that install a host timer: grep `timer|cron` named only `due_checks_gate.py`, `hostinstall-mode-harness.sh` and `test_hostinstall.py`, which are gates and tests and were not opened. \u00b7 SMART reads have no clock of their own: they ride each host report `AG/internal/storage/observe.go:184-193`." + }, + "extra": [ + "## Tiny", + "## Not found" + ] + } + ], + "pointers": [ + { + "ptr": "AG/internal/localapi/server.go:1134-1217", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Whole-guest backup: due-check and run", + "q": "1" + }, + { + "ptr": "AG/internal/localapi/server.go:878-1015", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Whole-guest backup: due-check and run", + "q": "1" + }, + { + "ptr": "AG/internal/localapi/server.go:1204-1213", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Whole-guest backup: due-check and run", + "q": "2" + }, + { + "ptr": "AG/internal/config/config.go:576-580", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/localapi/server.go:1204-1213;)", + "ending": "1. Whole-guest backup: due-check and run", + "q": "2" + }, + { + "ptr": "AR/07-backup-architecture.md:551", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Whole-guest backup: due-check and run", + "q": "2" + }, + { + "ptr": "AG/internal/config/config.go:464-465", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Whole-guest backup: due-check and run", + "q": "3" + }, + { + "ptr": "AG/internal/localapi/server.go:971-975", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Whole-guest backup: due-check and run", + "q": "3" + }, + { + "ptr": "AR/07-backup-architecture.md:551-552", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Whole-guest backup: due-check and run", + "q": "4" + }, + { + "ptr": "AR/07-backup-architecture.md:579-581", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Whole-guest backup: due-check and run", + "q": "4" + }, + { + "ptr": "AG/internal/localapi/server.go:913-917", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Whole-guest backup: due-check and run", + "q": "5" + }, + { + "ptr": "AG/internal/localapi/server.go:919-925", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Whole-guest backup: due-check and run", + "q": "5" + }, + { + "ptr": "AG/internal/backup/inflight.go:40-59", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Whole-guest backup: due-check and run", + "q": "5" + }, + { + "ptr": "AG/internal/localapi/server.go:509", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Whole-guest backup: due-check and run", + "q": "6" + }, + { + "ptr": "AG/internal/localapi/server.go:1176-1182", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Whole-guest backup: due-check and run", + "q": "6" + }, + { + "ptr": "AR/07-backup-architecture.md:579-582", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Whole-guest backup: due-check and run", + "q": "7" + }, + { + "ptr": "AG/internal/localapi/backup_cap.go:12-35", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Whole-guest backup: due-check and run", + "q": "7" + }, + { + "ptr": "AG/internal/localapi/r894_saved_backup_test.go:86", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Whole-guest backup: due-check and run", + "q": "8" + }, + { + "ptr": "AR/07-backup-architecture.md:420-429", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Whole-guest backup: due-check and run", + "q": "8" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1390-1398", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "1" + }, + { + "ptr": "AG/internal/localapi/server.go:1008-1011", + "file": "OK", + "ident": "IDENT-MISS(SetAfterPrimaryBackup)", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "1" + }, + { + "ptr": "AG/internal/osupdate/leg.go:538-586", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1395", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "2" + }, + { + "ptr": "AG/internal/osupdate/leg.go:656-667", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "2" + }, + { + "ptr": "AG/internal/osupdate/leg.go:538-586", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "2" + }, + { + "ptr": "AR/11-os-updates.md:378-387", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "2" + }, + { + "ptr": "AG/internal/osupdate/leg.go:810-813", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "3" + }, + { + "ptr": "AR/07-backup-architecture.md:449-450", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "3" + }, + { + "ptr": "AR/07-backup-architecture.md:447", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "4" + }, + { + "ptr": "AR/11-os-updates.md:406-408", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "4" + }, + { + "ptr": "AR/07-backup-architecture.md:444-446", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "5" + }, + { + "ptr": "AG/internal/osupdate/unsent.go:35-52", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "5" + }, + { + "ptr": "AG/internal/osupdate/leg.go:663", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "5" + }, + { + "ptr": "AG/internal/osupdate/leg.go:661-662", + "file": "OK", + "ident": "IDENT-MISS(Sublast)", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "6" + }, + { + "ptr": "AG/internal/osupdate/leg.go:922-939", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "7" + }, + { + "ptr": "AR/11-os-updates.md:406-423", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "7" + }, + { + "ptr": "AG/internal/osupdate/leg_test.go:367", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/afterbackup_test.go:17", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "8" + }, + { + "ptr": "AR/11-os-updates.md:366-387", + "file": "OK", + "ident": "IDENT-MISS(TestAfterPrimaryBackup)", + "ending": "2. OS-update night leg (guest, host, Docker, Proxmox)", + "q": "8" + }, + { + "ptr": "AG/internal/osupdate/kernel.go:119-178", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1018", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "1" + }, + { + "ptr": "AG/internal/osupdate/kernel.go:101-115", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "2" + }, + { + "ptr": "AR/11-os-updates.md:580-585", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "2" + }, + { + "ptr": "AG/internal/osupdate/kernel.go:37", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "3" + }, + { + "ptr": "AG/internal/osupdate/kernel.go:251", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "3" + }, + { + "ptr": "AR/11-os-updates.md:570-572", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "3" + }, + { + "ptr": "AR/11-os-updates.md:584-585", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "4" + }, + { + "ptr": "AR/11-os-updates.md:581-583", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "4" + }, + { + "ptr": "AR/11-os-updates.md:572-573", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "4" + }, + { + "ptr": "AR/11-os-updates.md:560-562", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "5" + }, + { + "ptr": "AR/11-os-updates.md:557-558", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "5" + }, + { + "ptr": "AG/internal/osupdate/kernel.go:300-312", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "5" + }, + { + "ptr": "AG/internal/osupdate/leg.go:291-296", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "6" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:1", + "file": "OK", + "ident": "IDENT-MISS(timedatectl|chrony|ntp)", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "6" + }, + { + "ptr": "AG/internal/osupdate/kernel.go:165", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "7" + }, + { + "ptr": "AR/11-os-updates.md:585-586", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "7" + }, + { + "ptr": "AR/11-os-updates.md:534-535", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "7" + }, + { + "ptr": "AG/internal/osupdate/kernel_test.go:200", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "8" + }, + { + "ptr": "AG/internal/osupdate/kernel_test.go:79", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/osupdate/kernel_test.go:200,)", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "8" + }, + { + "ptr": "AR/11-os-updates.md:530-601", + "file": "OK", + "ident": "IDENT-MISS(TestKernel_NoMailNoStep)", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "8" + }, + { + "ptr": "AR/11-os-updates.md:599-601", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Kernel lane: night reboot, 20-minute boot judge, one self-revert", + "q": "8" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:2944-2963", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Crash guard: early-boot unit plus hourly re-arm timer", + "q": "1" + }, + { + "ptr": "AG/configs/felhom-crash-guard.service:15-16", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Crash guard: early-boot unit plus hourly re-arm timer", + "q": "1" + }, + { + "ptr": "AG/configs/felhom-crash-guard-check.timer:5-6", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Crash guard: early-boot unit plus hourly re-arm timer", + "q": "1" + }, + { + "ptr": "AG/configs/felhom-crash-guard-check.timer:5-6", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Crash guard: early-boot unit plus hourly re-arm timer", + "q": "2" + }, + { + "ptr": "AG/configs/felhom-crash-guard.service:8", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Crash guard: early-boot unit plus hourly re-arm timer", + "q": "2" + }, + { + "ptr": "AG/configs/felhom-crash-guard-check.service:6-7", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Crash guard: early-boot unit plus hourly re-arm timer", + "q": "3" + }, + { + "ptr": "AR/11-os-updates.md:484-485", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Crash guard: early-boot unit plus hourly re-arm timer", + "q": "4" + }, + { + "ptr": "AR/11-os-updates.md:487-488", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Crash guard: early-boot unit plus hourly re-arm timer", + "q": "4" + }, + { + "ptr": "AG/configs/crash-guard.conf:3-8", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Crash guard: early-boot unit plus hourly re-arm timer", + "q": "5" + }, + { + "ptr": "AR/11-os-updates.md:487-488", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Crash guard: early-boot unit plus hourly re-arm timer", + "q": "7" + }, + { + "ptr": "AR/11-os-updates.md:489-490", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Crash guard: early-boot unit plus hourly re-arm timer", + "q": "7" + }, + { + "ptr": "AG/configs/test_felhom_crash_guard.py:75", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Crash guard: early-boot unit plus hourly re-arm timer", + "q": "8" + }, + { + "ptr": "AG/configs/test_felhom_crash_guard.py:102", + "file": "OK", + "ident": "IDENT-MISS(AG/configs/test_felhom_crash_guard.py:75)", + "ending": "4. Crash guard: early-boot unit plus hourly re-arm timer", + "q": "8" + }, + { + "ptr": "AR/11-os-updates.md:473-490", + "file": "OK", + "ident": "IDENT-MISS(test_rearms_after_24h_of_normal_running)", + "ending": "4. Crash guard: early-boot unit plus hourly re-arm timer", + "q": "8" + }, + { + "ptr": "AG/internal/backup/schedule.go:148-177", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Scheduled restore-test evaluation", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1968-2018", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Scheduled restore-test evaluation", + "q": "1" + }, + { + "ptr": "AG/internal/backup/schedule.go:76", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Scheduled restore-test evaluation", + "q": "2" + }, + { + "ptr": "AG/internal/config/config.go:636", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Scheduled restore-test evaluation", + "q": "2" + }, + { + "ptr": "AG/internal/config/config.go:641", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Scheduled restore-test evaluation", + "q": "2" + }, + { + "ptr": "AR/11-os-updates.md:385-386", + "file": "OK", + "ident": "IDENT-MISS(defaultRestoreTestSettle)", + "ending": "5. Scheduled restore-test evaluation", + "q": "2" + }, + { + "ptr": "AG/internal/config/config.go:584-588", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Scheduled restore-test evaluation", + "q": "3" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1917-1919", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Scheduled restore-test evaluation", + "q": "3" + }, + { + "ptr": "AG/internal/reconcile/restoretest.go:28", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Scheduled restore-test evaluation", + "q": "3" + }, + { + "ptr": "AR/07-backup-architecture.md:583-584", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Scheduled restore-test evaluation", + "q": "4" + }, + { + "ptr": "AR/07-backup-architecture.md:179-180", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Scheduled restore-test evaluation", + "q": "4" + }, + { + "ptr": "AR/07-backup-architecture.md:215-217", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Scheduled restore-test evaluation", + "q": "4" + }, + { + "ptr": "AG/internal/backup/schedule.go:228", + "file": "OK", + "ident": "IDENT-MISS(TryAcquire\"restore-test\")", + "ending": "5. Scheduled restore-test evaluation", + "q": "5" + }, + { + "ptr": "AG/internal/backup/schedule.go:230", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Scheduled restore-test evaluation", + "q": "5" + }, + { + "ptr": "AG/internal/backup/schedule.go:248-255", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Scheduled restore-test evaluation", + "q": "5" + }, + { + "ptr": "AG/internal/backup/schedule.go:119", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Scheduled restore-test evaluation", + "q": "6" + }, + { + "ptr": "AG/internal/backup/restoretest_due.go:96-101", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Scheduled restore-test evaluation", + "q": "6" + }, + { + "ptr": "AG/internal/backup/schedule.go:262", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Scheduled restore-test evaluation", + "q": "7" + }, + { + "ptr": "AG/cmd/felhom-agent/janitor.go:42-44", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Scheduled restore-test evaluation", + "q": "7" + }, + { + "ptr": "AG/internal/backup/r874_first_eval_test.go:17", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Scheduled restore-test evaluation", + "q": "8" + }, + { + "ptr": "AG/internal/backup/restoretest_due_test.go:139", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Scheduled restore-test evaluation", + "q": "8" + }, + { + "ptr": "AR/07-backup-architecture.md:174-200", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Scheduled restore-test evaluation", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/controllersupervisor.go:143-163", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Controller supervisor (restarts a dead or silent controller)", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1714", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Controller supervisor (restarts a dead or silent controller)", + "q": "1" + }, + { + "ptr": "AG/internal/localapi/controllersupervisor.go:54", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Controller supervisor (restarts a dead or silent controller)", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/controllersupervisor.go:56", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Controller supervisor (restarts a dead or silent controller)", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/controllersupervisor.go:92-100", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Controller supervisor (restarts a dead or silent controller)", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/controllersupervisor.go:1-200", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Controller supervisor (restarts a dead or silent controller)", + "q": "3" + }, + { + "ptr": "AG/internal/localapi/controllersupervisor.go:64", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Controller supervisor (restarts a dead or silent controller)", + "q": "3" + }, + { + "ptr": "AG/internal/localapi/controllersupervisor.go:20-25", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Controller supervisor (restarts a dead or silent controller)", + "q": "4" + }, + { + "ptr": "AG/internal/localapi/controllersupervisor.go:153-161", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Controller supervisor (restarts a dead or silent controller)", + "q": "4" + }, + { + "ptr": "AG/internal/localapi/controllersupervisor.go:59-61", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Controller supervisor (restarts a dead or silent controller)", + "q": "5" + }, + { + "ptr": "AG/internal/localapi/controllersupervisor.go:34-42", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Controller supervisor (restarts a dead or silent controller)", + "q": "5" + }, + { + "ptr": "AG/internal/localapi/controllersupervisor.go:103-105", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Controller supervisor (restarts a dead or silent controller)", + "q": "6" + }, + { + "ptr": "AG/internal/localapi/controllersupervisor.go:115-119", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Controller supervisor (restarts a dead or silent controller)", + "q": "6" + }, + { + "ptr": "AG/internal/localapi/controllersupervisor.go:44-49", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Controller supervisor (restarts a dead or silent controller)", + "q": "7" + }, + { + "ptr": "AG/internal/localapi/controllersupervisor.go:82-83", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Controller supervisor (restarts a dead or silent controller)", + "q": "7" + }, + { + "ptr": "AG/internal/localapi/controllersupervisor_test.go:101", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Controller supervisor (restarts a dead or silent controller)", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/supervisor_unresponsive_test.go:32", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Controller supervisor (restarts a dead or silent controller)", + "q": "8" + }, + { + "ptr": "AR/03-host-agent.md:186-192", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Controller supervisor (restarts a dead or silent controller)", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/guestpower.go:89-105", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1710", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "1" + }, + { + "ptr": "AG/internal/localapi/guestpower.go:52", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/guestpower.go:167-180", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/guestpower.go:95-102", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/guestpower.go:56", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "3" + }, + { + "ptr": "AG/internal/localapi/guestpower.go:75", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "3" + }, + { + "ptr": "AG/internal/localapi/guestpower.go:71-74", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "3" + }, + { + "ptr": "AG/internal/localapi/guestpower.go:14-18", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "4" + }, + { + "ptr": "AG/internal/localapi/guestpower.go:42-43", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "4" + }, + { + "ptr": "AG/internal/localapi/guestpower.go:182-194", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "5" + }, + { + "ptr": "AG/internal/localapi/guestpower.go:176-180", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "5" + }, + { + "ptr": "AG/internal/localapi/guestpower.go:157-159", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "5" + }, + { + "ptr": "AG/internal/localapi/guestpower.go:77-80", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "6" + }, + { + "ptr": "AG/internal/localapi/guestpower.go:226", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "6" + }, + { + "ptr": "AG/internal/localapi/guestpower.go:205-206", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "7" + }, + { + "ptr": "AG/internal/localapi/guestpower.go:1", + "file": "OK", + "ident": "IDENT-MISS(hub\\.|Report|report)", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "7" + }, + { + "ptr": "AG/internal/localapi/guestpower_test.go:162", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/guestpower_test.go:119", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Guest-power watchdog (starts a guest that should be running)", + "q": "8" + }, + { + "ptr": "AG/internal/hub/loop.go:107-138", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:958", + "file": "OK", + "ident": "IDENT-MISS(Run)", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "1" + }, + { + "ptr": "AG/internal/config/config.go:797", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "2" + }, + { + "ptr": "AG/internal/hub/loop.go:107-138", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "2" + }, + { + "ptr": "AG/internal/hub/loop.go:11-12", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "2" + }, + { + "ptr": "AG/internal/fasttick/fasttick.go:24", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "2" + }, + { + "ptr": "AG/internal/config/config.go:798", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "3" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:4149", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "3" + }, + { + "ptr": "AG/internal/hub/loop.go:162-164", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "4" + }, + { + "ptr": "AR/11-os-updates.md:572-573", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "4" + }, + { + "ptr": "AG/internal/hub/loop.go:125-135", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "5" + }, + { + "ptr": "AG/internal/poke/poke.go:40", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "5" + }, + { + "ptr": "AG/internal/reconcile/engine.go:332-344", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "5" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1525", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "5" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:969", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "6" + }, + { + "ptr": "AG/internal/hub/loop.go:188-196", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "6" + }, + { + "ptr": "AR/07-backup-architecture.md:555-558", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "6" + }, + { + "ptr": "AR/11-os-updates.md:572-573", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "7" + }, + { + "ptr": "AR/07-backup-architecture.md:582-583", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "7" + }, + { + "ptr": "AG/internal/hub/loop_test.go:140", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "8" + }, + { + "ptr": "AG/internal/hub/loop_test.go:81", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/hub/loop_test.go:140,)", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "8" + }, + { + "ptr": "AR/03-host-agent.md:276-279", + "file": "OK", + "ident": "IDENT-MISS(TestClampInterval)", + "ending": "8. Hub report and heartbeat loop (with reconcile, fast tick and poke)", + "q": "8" + }, + { + "ptr": "AG/internal/signedjobs/runner.go:93-102", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Signed jobs: validity window against the host clock (agent updates ride it)", + "q": "1" + }, + { + "ptr": "AG/internal/authz/verifier.go:21", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Signed jobs: validity window against the host clock (agent updates ride it)", + "q": "2" + }, + { + "ptr": "AG/internal/authz/verifier.go:26", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Signed jobs: validity window against the host clock (agent updates ride it)", + "q": "2" + }, + { + "ptr": "AG/internal/selfupdate/executor.go:73", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Signed jobs: validity window against the host clock (agent updates ride it)", + "q": "3" + }, + { + "ptr": "AG/internal/config/config.go:97-99", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Signed jobs: validity window against the host clock (agent updates ride it)", + "q": "3" + }, + { + "ptr": "AG/internal/authz/verifier.go:161-162", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Signed jobs: validity window against the host clock (agent updates ride it)", + "q": "4" + }, + { + "ptr": "AG/internal/selfupdate/commit.go:122-125", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Signed jobs: validity window against the host clock (agent updates ride it)", + "q": "4" + }, + { + "ptr": "AG/internal/signedjobs/runner.go:107-115", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Signed jobs: validity window against the host clock (agent updates ride it)", + "q": "5" + }, + { + "ptr": "AG/internal/authz/verifier.go:166", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Signed jobs: validity window against the host clock (agent updates ride it)", + "q": "5" + }, + { + "ptr": "AG/internal/authz/verifier.go:155-156", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Signed jobs: validity window against the host clock (agent updates ride it)", + "q": "6" + }, + { + "ptr": "AG/internal/authz/verifier.go:161-162", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Signed jobs: validity window against the host clock (agent updates ride it)", + "q": "6" + }, + { + "ptr": "AG/internal/authz/verify_test.go:140-155", + "file": "OK", + "ident": "IDENT-MISS(not_yet|NotYet|expired)", + "ending": "9. Signed jobs: validity window against the host clock (agent updates ride it)", + "q": "7" + }, + { + "ptr": "AG/internal/authz/verify_test.go:82", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Signed jobs: validity window against the host clock (agent updates ride it)", + "q": "8" + }, + { + "ptr": "AG/internal/signedjobs/runner_test.go:168", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/authz/verify_test.go:82,)", + "ending": "9. Signed jobs: validity window against the host clock (agent updates ride it)", + "q": "8" + }, + { + "ptr": "AG/internal/selfupdate/commit_test.go:102", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Signed jobs: validity window against the host clock (agent updates ride it)", + "q": "8" + }, + { + "ptr": "AG/internal/storage/watchdog.go:196-215", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Storage watchdog and the 20-second mount and bind re-assert", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1142", + "file": "OK", + "ident": "IDENT-MISS(Run)", + "ending": "10. Storage watchdog and the 20-second mount and bind re-assert", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1613-1631", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Storage watchdog and the 20-second mount and bind re-assert", + "q": "1" + }, + { + "ptr": "AG/internal/storage/watchdog.go:15-16", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Storage watchdog and the 20-second mount and bind re-assert", + "q": "2" + }, + { + "ptr": "AG/internal/config/config.go:770-778", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Storage watchdog and the 20-second mount and bind re-assert", + "q": "2" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1612-1614", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Storage watchdog and the 20-second mount and bind re-assert", + "q": "2" + }, + { + "ptr": "AR/03-host-agent.md:461", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Storage watchdog and the 20-second mount and bind re-assert", + "q": "4" + }, + { + "ptr": "AG/internal/storage/watchdog.go:183-192", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Storage watchdog and the 20-second mount and bind re-assert", + "q": "5" + }, + { + "ptr": "AG/internal/storage/watchdog.go:77-79", + "file": "OK", + "ident": "IDENT-MISS(backoffWindow)", + "ending": "10. Storage watchdog and the 20-second mount and bind re-assert", + "q": "5" + }, + { + "ptr": "AG/internal/storage/watchdog.go:290", + "file": "OK", + "ident": "IDENT-MISS(Sublast)", + "ending": "10. Storage watchdog and the 20-second mount and bind re-assert", + "q": "5" + }, + { + "ptr": "AG/internal/storage/watchdog.go:160", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Storage watchdog and the 20-second mount and bind re-assert", + "q": "6" + }, + { + "ptr": "AG/internal/hub/loop.go:125-135", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Storage watchdog and the 20-second mount and bind re-assert", + "q": "7" + }, + { + "ptr": "AG/internal/storage/watchdog_test.go:112", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Storage watchdog and the 20-second mount and bind re-assert", + "q": "8" + }, + { + "ptr": "AG/internal/storage/watchdog_test.go:460", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Storage watchdog and the 20-second mount and bind re-assert", + "q": "8" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1613", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Storage watchdog and the 20-second mount and bind re-assert", + "q": "8" + }, + { + "ptr": "AG/internal/fstrim/fstrim.go:186-199", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Weekly guest disk trim", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1741", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Weekly guest disk trim", + "q": "1" + }, + { + "ptr": "AG/internal/fstrim/fstrim.go:3-5", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Weekly guest disk trim", + "q": "1" + }, + { + "ptr": "AG/internal/fstrim/fstrim.go:42-46", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Weekly guest disk trim", + "q": "2" + }, + { + "ptr": "AG/internal/fstrim/fstrim.go:48-50", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Weekly guest disk trim", + "q": "2" + }, + { + "ptr": "AG/internal/fstrim/fstrim.go:110", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/fstrim/fstrim.go:48-50;)", + "ending": "11. Weekly guest disk trim", + "q": "2" + }, + { + "ptr": "AG/internal/fstrim/fstrim.go:52", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Weekly guest disk trim", + "q": "3" + }, + { + "ptr": "AG/internal/fstrim/fstrim.go:6", + "file": "OK", + "ident": "IDENT-MISS(PerGuestTimeout)", + "ending": "11. Weekly guest disk trim", + "q": "3" + }, + { + "ptr": "AG/internal/fstrim/fstrim.go:10-11", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Weekly guest disk trim", + "q": "4" + }, + { + "ptr": "AG/internal/fstrim/fstrim.go:46", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Weekly guest disk trim", + "q": "4" + }, + { + "ptr": "AG/internal/fstrim/fstrim.go:326-330", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Weekly guest disk trim", + "q": "4" + }, + { + "ptr": "AG/internal/fstrim/fstrim.go:247-251", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Weekly guest disk trim", + "q": "5" + }, + { + "ptr": "AG/internal/fstrim/fstrim.go:17", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Weekly guest disk trim", + "q": "5" + }, + { + "ptr": "AG/internal/fstrim/fstrim.go:204", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Weekly guest disk trim", + "q": "6" + }, + { + "ptr": "AG/internal/fstrim/fstrim.go:179-182", + "file": "OK", + "ident": "IDENT-MISS(due)", + "ending": "11. Weekly guest disk trim", + "q": "6" + }, + { + "ptr": "AG/internal/fstrim/fstrim.go:318-319", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Weekly guest disk trim", + "q": "7" + }, + { + "ptr": "AG/internal/fstrim/fstrim_test.go:161", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Weekly guest disk trim", + "q": "8" + }, + { + "ptr": "AG/internal/fstrim/fstrim_test.go:175", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Weekly guest disk trim", + "q": "8" + }, + { + "ptr": "AG/internal/fstrim/fstrim_test.go:104", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/fstrim/fstrim_test.go:175,)", + "ending": "11. Weekly guest disk trim", + "q": "8" + }, + { + "ptr": "AG/internal/linetest/linetest.go:256-268", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Weekly upload line test", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1762", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Weekly upload line test", + "q": "1" + }, + { + "ptr": "AG/internal/linetest/linetest.go:3-4", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Weekly upload line test", + "q": "1" + }, + { + "ptr": "AG/internal/linetest/linetest.go:47-49", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Weekly upload line test", + "q": "2" + }, + { + "ptr": "AG/internal/linetest/linetest.go:148-152", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Weekly upload line test", + "q": "2" + }, + { + "ptr": "AG/internal/linetest/linetest.go:60-61", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Weekly upload line test", + "q": "2" + }, + { + "ptr": "AG/internal/linetest/linetest.go:131", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Weekly upload line test", + "q": "2" + }, + { + "ptr": "AG/internal/linetest/linetest.go:53-57", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Weekly upload line test", + "q": "3" + }, + { + "ptr": "AG/internal/linetest/linetest.go:10-11", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Weekly upload line test", + "q": "4" + }, + { + "ptr": "AR/11-os-updates.md:409-411", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Weekly upload line test", + "q": "4" + }, + { + "ptr": "AG/internal/linetest/linetest.go:316-325", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Weekly upload line test", + "q": "5" + }, + { + "ptr": "AG/internal/linetest/linetest.go:58", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Weekly upload line test", + "q": "5" + }, + { + "ptr": "AG/internal/linetest/linetest.go:297-300", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Weekly upload line test", + "q": "5" + }, + { + "ptr": "AG/internal/linetest/linetest.go:281", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Weekly upload line test", + "q": "6" + }, + { + "ptr": "AG/internal/linetest/linetest.go:211-213", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Weekly upload line test", + "q": "6" + }, + { + "ptr": "AG/internal/linetest/linetest.go:244-245", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Weekly upload line test", + "q": "6" + }, + { + "ptr": "AR/11-os-updates.md:409-411", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Weekly upload line test", + "q": "7" + }, + { + "ptr": "AG/internal/linetest/linetest_test.go:206", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Weekly upload line test", + "q": "8" + }, + { + "ptr": "AG/internal/linetest/linetest_test.go:363", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/linetest/linetest_test.go:206,)", + "ending": "12. Weekly upload line test", + "q": "8" + }, + { + "ptr": "AG/internal/linetest/linetest_test.go:189", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/linetest/linetest_test.go:363,)", + "ending": "12. Weekly upload line test", + "q": "8" + }, + { + "ptr": "AG/internal/pbs/verify.go:64-83", + "file": "OK", + "ident": "IDENT-OK", + "ending": "13. PBS datastore verify loop", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1188", + "file": "OK", + "ident": "IDENT-MISS(Run)", + "ending": "13. PBS datastore verify loop", + "q": "1" + }, + { + "ptr": "AG/internal/pbs/verify.go:13", + "file": "OK", + "ident": "IDENT-OK", + "ending": "13. PBS datastore verify loop", + "q": "2" + }, + { + "ptr": "AG/internal/config/config.go:710-719", + "file": "OK", + "ident": "IDENT-OK", + "ending": "13. PBS datastore verify loop", + "q": "2" + }, + { + "ptr": "AG/internal/pbs/verify.go:64-83", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. PBS datastore verify loop", + "q": "2" + }, + { + "ptr": "AG/internal/pbs/verify.go:107", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. PBS datastore verify loop", + "q": "3" + }, + { + "ptr": "AG/internal/pbs/verify.go:64-83", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. PBS datastore verify loop", + "q": "4" + }, + { + "ptr": "AG/internal/pbs/verify_test.go:87", + "file": "OK", + "ident": "IDENT-OK", + "ending": "13. PBS datastore verify loop", + "q": "4" + }, + { + "ptr": "AG/internal/lanresolver/loop.go:17", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. PBS datastore verify loop", + "q": "5" + }, + { + "ptr": "AG/internal/pbs/verify.go:1", + "file": "OK", + "ident": "IDENT-MISS(TryAcquire)", + "ending": "13. PBS datastore verify loop", + "q": "5" + }, + { + "ptr": "AG/internal/pbs/verify.go:64-83", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. PBS datastore verify loop", + "q": "6" + }, + { + "ptr": "AG/internal/pbs/verify.go:127", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. PBS datastore verify loop", + "q": "7" + }, + { + "ptr": "AG/internal/pbs/verify_test.go:52", + "file": "OK", + "ident": "IDENT-OK", + "ending": "13. PBS datastore verify loop", + "q": "8" + }, + { + "ptr": "AG/cmd/felhom-agent/janitor.go:69-81", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. Janitor: leftover scratch guests and stale backup locks", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1725", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. Janitor: leftover scratch guests and stale backup locks", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-agent/janitor.go:14-17", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. Janitor: leftover scratch guests and stale backup locks", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-agent/janitor.go:18", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. Janitor: leftover scratch guests and stale backup locks", + "q": "2" + }, + { + "ptr": "AG/cmd/felhom-agent/janitor.go:71-78", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. Janitor: leftover scratch guests and stale backup locks", + "q": "2" + }, + { + "ptr": "AG/cmd/felhom-agent/janitor.go:34-66", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. Janitor: leftover scratch guests and stale backup locks", + "q": "3" + }, + { + "ptr": "AG/cmd/felhom-agent/janitor.go:41-54", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. Janitor: leftover scratch guests and stale backup locks", + "q": "4" + }, + { + "ptr": "AG/cmd/felhom-agent/janitor.go:57-64", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. Janitor: leftover scratch guests and stale backup locks", + "q": "5" + }, + { + "ptr": "AG/cmd/felhom-agent/janitor.go:30-33", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. Janitor: leftover scratch guests and stale backup locks", + "q": "5" + }, + { + "ptr": "AG/internal/localapi/stalelock_inflight_test.go:97", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. Janitor: leftover scratch guests and stale backup locks", + "q": "6" + }, + { + "ptr": "AG/cmd/felhom-agent/janitor.go:42-44", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. Janitor: leftover scratch guests and stale backup locks", + "q": "7" + }, + { + "ptr": "AG/cmd/felhom-agent/janitor_test.go:40", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. Janitor: leftover scratch guests and stale backup locks", + "q": "8" + }, + { + "ptr": "AG/cmd/felhom-agent/janitor_test.go:22", + "file": "OK", + "ident": "IDENT-MISS(AG/cmd/felhom-agent/janitor_test.go:40,)", + "ending": "14. Janitor: leftover scratch guests and stale backup locks", + "q": "8" + }, + { + "ptr": "AG/internal/guestnet/watchdog.go:105-121", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Guest network watchdog (DHCP client heal)", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1686", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Guest network watchdog (DHCP client heal)", + "q": "1" + }, + { + "ptr": "AG/internal/guestnet/watchdog.go:23", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Guest network watchdog (DHCP client heal)", + "q": "2" + }, + { + "ptr": "AG/internal/guestnet/watchdog.go:111", + "file": "OK", + "ident": "IDENT-MISS(DefaultInterval)", + "ending": "15. Guest network watchdog (DHCP client heal)", + "q": "2" + }, + { + "ptr": "AG/internal/config/config.go:176-196", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Guest network watchdog (DHCP client heal)", + "q": "2" + }, + { + "ptr": "AG/internal/guestnet/watchdog.go:1-200", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. Guest network watchdog (DHCP client heal)", + "q": "3" + }, + { + "ptr": "AG/internal/guestnet/watchdog.go:26-28", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. Guest network watchdog (DHCP client heal)", + "q": "4" + }, + { + "ptr": "AG/internal/guestnet/watchdog.go:32-33", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Guest network watchdog (DHCP client heal)", + "q": "5" + }, + { + "ptr": "AG/internal/guestnet/watchdog.go:25", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/guestnet/watchdog.go:32-33;)", + "ending": "15. Guest network watchdog (DHCP client heal)", + "q": "5" + }, + { + "ptr": "AG/internal/guestnet/watchdog.go:29-31", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/guestnet/watchdog.go:25;)", + "ending": "15. Guest network watchdog (DHCP client heal)", + "q": "5" + }, + { + "ptr": "AG/internal/guestnet/watchdog.go:80", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Guest network watchdog (DHCP client heal)", + "q": "6" + }, + { + "ptr": "AG/internal/guestnet/watchdog.go:56-62", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. Guest network watchdog (DHCP client heal)", + "q": "6" + }, + { + "ptr": "AG/internal/guestnet/watchdog.go:166-170", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Guest network watchdog (DHCP client heal)", + "q": "7" + }, + { + "ptr": "AG/internal/guestnet/watchdog_test.go:360", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Guest network watchdog (DHCP client heal)", + "q": "8" + }, + { + "ptr": "AG/internal/guestnet/watchdog_test.go:384", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/guestnet/watchdog_test.go:360,)", + "ending": "15. Guest network watchdog (DHCP client heal)", + "q": "8" + }, + { + "ptr": "AG/internal/guestnet/watchdog_test.go:416", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/guestnet/watchdog_test.go:384,)", + "ending": "15. Guest network watchdog (DHCP client heal)", + "q": "8" + }, + { + "ptr": "AG/internal/selfheal/selfheal.go:67-82", + "file": "OK", + "ident": "IDENT-OK", + "ending": "16. Host networking self-heal", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1648", + "file": "OK", + "ident": "IDENT-MISS(Watch)", + "ending": "16. Host networking self-heal", + "q": "1" + }, + { + "ptr": "AG/internal/selfheal/selfheal.go:67-82", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Host networking self-heal", + "q": "2" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1648", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Host networking self-heal", + "q": "2" + }, + { + "ptr": "AG/internal/selfheal/selfheal.go:114-116", + "file": "OK", + "ident": "IDENT-OK", + "ending": "16. Host networking self-heal", + "q": "3" + }, + { + "ptr": "AG/internal/selfheal/selfheal.go:54-58", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Host networking self-heal", + "q": "4" + }, + { + "ptr": "AG/internal/selfheal/selfheal.go:46-64", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Host networking self-heal", + "q": "5" + }, + { + "ptr": "AG/internal/selfheal/selfheal.go:67-82", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Host networking self-heal", + "q": "6" + }, + { + "ptr": "AG/internal/selfheal/selfheal.go:61", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Host networking self-heal", + "q": "7" + }, + { + "ptr": "AR/11-os-updates.md:572-573", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Host networking self-heal", + "q": "7" + }, + { + "ptr": "AG/internal/selfheal/selfheal_test.go:109", + "file": "OK", + "ident": "IDENT-OK", + "ending": "16. Host networking self-heal", + "q": "8" + }, + { + "ptr": "AG/internal/selfheal/selfheal_test.go:32", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/selfheal/selfheal_test.go:109,)", + "ending": "16. Host networking self-heal", + "q": "8" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:3067-3076", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Management-plane watchdog timer (keeps SSH reachable)", + "q": "1" + }, + { + "ptr": "AG/configs/felhom-mgmt-watchdog.sh:26-68", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Management-plane watchdog timer (keeps SSH reachable)", + "q": "1" + }, + { + "ptr": "AG/configs/felhom-mgmt-watchdog.service:3-7", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Management-plane watchdog timer (keeps SSH reachable)", + "q": "1" + }, + { + "ptr": "AG/configs/felhom-mgmt-watchdog.timer:11-14", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. Management-plane watchdog timer (keeps SSH reachable)", + "q": "2" + }, + { + "ptr": "AG/configs/felhom-mgmt-watchdog.service:16-18", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Management-plane watchdog timer (keeps SSH reachable)", + "q": "3" + }, + { + "ptr": "AG/configs/felhom-mgmt-watchdog.sh:36-61", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Management-plane watchdog timer (keeps SSH reachable)", + "q": "3" + }, + { + "ptr": "AG/configs/felhom-mgmt-watchdog.sh:23-24", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Management-plane watchdog timer (keeps SSH reachable)", + "q": "4" + }, + { + "ptr": "AG/configs/felhom-mgmt-watchdog.timer:3-6", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Management-plane watchdog timer (keeps SSH reachable)", + "q": "4" + }, + { + "ptr": "AG/configs/felhom-mgmt-watchdog.sh:13-15", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Management-plane watchdog timer (keeps SSH reachable)", + "q": "5" + }, + { + "ptr": "AG/configs/felhom-mgmt-watchdog.timer:11-14", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. Management-plane watchdog timer (keeps SSH reachable)", + "q": "5" + }, + { + "ptr": "AG/configs/felhom-mgmt-watchdog.timer:11-12", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Management-plane watchdog timer (keeps SSH reachable)", + "q": "6" + }, + { + "ptr": "AG/configs/felhom-mgmt-watchdog.sh:65", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. Management-plane watchdog timer (keeps SSH reachable)", + "q": "6" + }, + { + "ptr": "AG/configs/felhom-mgmt-watchdog.sh:18-19", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Management-plane watchdog timer (keeps SSH reachable)", + "q": "7" + }, + { + "ptr": "AG/configs/felhom-mgmt-watchdog.sh:63", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Management-plane watchdog timer (keeps SSH reachable)", + "q": "7" + }, + { + "ptr": "AG/configs/felhom-mgmt-watchdog.sh:1-68", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Management-plane watchdog timer (keeps SSH reachable)", + "q": "8" + }, + { + "ptr": "AG/configs/felhom-mgmt-watchdog.timer:3-6", + "file": "OK", + "ident": "IDENT-MISS(test_felhom_mgmt_watchdog)", + "ending": "17. Management-plane watchdog timer (keeps SSH reachable)", + "q": "8" + }, + { + "ptr": "AG/configs/felhom-agent.service:41-42", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Agent service restart, start limit and rollback", + "q": "1" + }, + { + "ptr": "AG/configs/felhom-agent-limits.conf:17-19", + "file": "OK", + "ident": "IDENT-MISS(,)", + "ending": "18. Agent service restart, start limit and rollback", + "q": "1" + }, + { + "ptr": "AG/configs/felhom-agent-limits.conf:17-19", + "file": "OK", + "ident": "IDENT-MISS(,)", + "ending": "18. Agent service restart, start limit and rollback", + "q": "2" + }, + { + "ptr": "AG/configs/felhom-agent-limits.conf:5-8", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Agent service restart, start limit and rollback", + "q": "2" + }, + { + "ptr": "AG/configs/felhom-agent.service:1", + "file": "OK", + "ident": "IDENT-MISS(WatchdogSec)", + "ending": "18. Agent service restart, start limit and rollback", + "q": "3" + }, + { + "ptr": "AG/internal/selfupdate/commit.go:40-42", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Agent service restart, start limit and rollback", + "q": "3" + }, + { + "ptr": "AR/11-os-updates.md:572-573", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Agent service restart, start limit and rollback", + "q": "4" + }, + { + "ptr": "AG/configs/felhom-agent-limits.conf:14-15", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Agent service restart, start limit and rollback", + "q": "5" + }, + { + "ptr": "AG/internal/selfupdate/commit.go:112-116", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Agent service restart, start limit and rollback", + "q": "5" + }, + { + "ptr": "AR/03-host-agent.md:72", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Agent service restart, start limit and rollback", + "q": "6" + }, + { + "ptr": "AG/configs/felhom-agent.service:41-42", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Agent service restart, start limit and rollback", + "q": "6" + }, + { + "ptr": "AR/11-os-updates.md:572-573", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Agent service restart, start limit and rollback", + "q": "7" + }, + { + "ptr": "AG/internal/backup/r874_first_eval_test.go:44", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Agent service restart, start limit and rollback", + "q": "7" + }, + { + "ptr": "AG/configs/felhom-agent-limits.conf:5-15", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Agent service restart, start limit and rollback", + "q": "8" + }, + { + "ptr": "AG/internal/selfupdate/commit_test.go:102", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Agent service restart, start limit and rollback", + "q": "8" + }, + { + "ptr": "AG/internal/wgtunnel/loop.go:62-78", + "file": "OK", + "ident": "IDENT-OK", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "1" + }, + { + "ptr": "AG/internal/pbsdr/loop.go:62-77", + "file": "OK", + "ident": "IDENT-OK", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "1" + }, + { + "ptr": "AG/internal/felhomsshd/loop.go:67-80", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "1" + }, + { + "ptr": "AG/internal/lanresolver/loop.go:35-52", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "1" + }, + { + "ptr": "AG/internal/config/config.go:132-133", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "2" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1504", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "2" + }, + { + "ptr": "AG/internal/config/config.go:74-75", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "2" + }, + { + "ptr": "AG/internal/lanresolver/loop.go:27-29", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "2" + }, + { + "ptr": "AG/internal/config/config.go:124-127", + "file": "OK", + "ident": "IDENT-OK", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "4" + }, + { + "ptr": "AG/internal/config/config.go:138-140", + "file": "OK", + "ident": "IDENT-OK", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "4" + }, + { + "ptr": "AG/internal/wgtunnel/rekey.go:36-38", + "file": "OK", + "ident": "IDENT-OK", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "5" + }, + { + "ptr": "AR/03-host-agent.md:376-378", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "5" + }, + { + "ptr": "AR/03-host-agent.md:142", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "6" + }, + { + "ptr": "AR/03-host-agent.md:144-150", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "6" + }, + { + "ptr": "AR/03-host-agent.md:155-156", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "6" + }, + { + "ptr": "AR/03-host-agent.md:142", + "file": "OK", + "ident": "IDENT-OK", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "7" + }, + { + "ptr": "AG/internal/wgtunnel/rekey_test.go:35", + "file": "OK", + "ident": "IDENT-OK", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "8" + }, + { + "ptr": "AG/internal/wgtunnel/rekey_test.go:213", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/wgtunnel/rekey_test.go:35,)", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "8" + }, + { + "ptr": "AG/internal/wgtunnel/manager_test.go:704", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/wgtunnel/rekey_test.go:213,)", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "8" + }, + { + "ptr": "AR/03-host-agent.md:151-155", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/server.go:929", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Whole-guest backup cap (rolling windows)", + "q": "1" + }, + { + "ptr": "AG/internal/localapi/backup_cap.go:41-53", + "file": "OK", + "ident": "IDENT-MISS(before)", + "ending": "20. Whole-guest backup cap (rolling windows)", + "q": "1" + }, + { + "ptr": "AG/internal/localapi/backup_cap.go:12-35", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Whole-guest backup cap (rolling windows)", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/backup_cap.go:38", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Whole-guest backup cap (rolling windows)", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/backup_cap.go:41-53", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Whole-guest backup cap (rolling windows)", + "q": "3" + }, + { + "ptr": "AG/internal/localapi/backup_cap_test.go:266", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Whole-guest backup cap (rolling windows)", + "q": "4" + }, + { + "ptr": "AG/internal/localapi/backup_cap_test.go:220", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Whole-guest backup cap (rolling windows)", + "q": "5" + }, + { + "ptr": "AG/internal/localapi/backup_cap_test.go:248", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Whole-guest backup cap (rolling windows)", + "q": "5" + }, + { + "ptr": "AG/internal/localapi/server.go:509", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Whole-guest backup cap (rolling windows)", + "q": "6" + }, + { + "ptr": "AG/internal/localapi/backup_cap.go:12-35", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Whole-guest backup cap (rolling windows)", + "q": "7" + }, + { + "ptr": "AG/internal/localapi/backup_cap_test.go:285", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Whole-guest backup cap (rolling windows)", + "q": "7" + }, + { + "ptr": "AG/internal/localapi/backup_cap_test.go:198", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Whole-guest backup cap (rolling windows)", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/backup_cap_test.go:151", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Whole-guest backup cap (rolling windows)", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/opreboot.go:46", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. Operator restart and reboot limits", + "q": "1" + }, + { + "ptr": "AG/internal/localapi/opactions.go:42", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. Operator restart and reboot limits", + "q": "1" + }, + { + "ptr": "AG/internal/localapi/opreboot.go:45-46", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Operator restart and reboot limits", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/opreboot.go:47-48", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. Operator restart and reboot limits", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/opactions.go:42", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Operator restart and reboot limits", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/opreboot.go:49-50", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. Operator restart and reboot limits", + "q": "3" + }, + { + "ptr": "AG/internal/localapi/opactions.go:46-49", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. Operator restart and reboot limits", + "q": "3" + }, + { + "ptr": "AG/internal/localapi/opactions.go:44", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. Operator restart and reboot limits", + "q": "4" + }, + { + "ptr": "AG/internal/localapi/opreboot.go:23-31", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Operator restart and reboot limits", + "q": "5" + }, + { + "ptr": "AG/internal/localapi/opreboot.go:32-34", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Operator restart and reboot limits", + "q": "5" + }, + { + "ptr": "AG/internal/localapi/opreboot.go:59-61", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. Operator restart and reboot limits", + "q": "5" + }, + { + "ptr": "AG/internal/localapi/opreboot.go:64-70", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Operator restart and reboot limits", + "q": "6" + }, + { + "ptr": "AG/internal/localapi/opreboot.go:36-39", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Operator restart and reboot limits", + "q": "7" + }, + { + "ptr": "AG/internal/localapi/opreboot.go:39", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Operator restart and reboot limits", + "q": "8" + }, + { + "ptr": "AR/03-host-agent.md:236-239", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Operator restart and reboot limits", + "q": "8" + }, + { + "ptr": "AG/internal/osupdate/unsent.go:71-82", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Unsent OS-update report sender", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1009", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Unsent OS-update report sender", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1009", + "file": "OK", + "ident": "NO-IDENT", + "ending": "22. Unsent OS-update report sender", + "q": "2" + }, + { + "ptr": "AG/internal/osupdate/unsent.go:79", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Unsent OS-update report sender", + "q": "2" + }, + { + "ptr": "AG/internal/osupdate/unsent.go:56-64", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Unsent OS-update report sender", + "q": "2" + }, + { + "ptr": "AG/internal/osupdate/leg.go:922-939", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Unsent OS-update report sender", + "q": "3" + }, + { + "ptr": "AG/internal/osupdate/unsent_test.go:87", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Unsent OS-update report sender", + "q": "4" + }, + { + "ptr": "AG/internal/osupdate/unsent.go:35-52", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Unsent OS-update report sender", + "q": "5" + }, + { + "ptr": "AG/internal/osupdate/unsent_test.go:110", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/osupdate/unsent.go:35-52;)", + "ending": "22. Unsent OS-update report sender", + "q": "5" + }, + { + "ptr": "AG/internal/osupdate/unsent.go:79", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Unsent OS-update report sender", + "q": "6" + }, + { + "ptr": "AG/internal/osupdate/unsent.go:56-82", + "file": "OK", + "ident": "NO-IDENT", + "ending": "22. Unsent OS-update report sender", + "q": "6" + }, + { + "ptr": "AG/internal/osupdate/unsent.go:1", + "file": "OK", + "ident": "IDENT-MISS(Warn|Error)", + "ending": "22. Unsent OS-update report sender", + "q": "7" + }, + { + "ptr": "AG/internal/osupdate/unsent_test.go:127", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Unsent OS-update report sender", + "q": "8" + }, + { + "ptr": "AG/internal/osupdate/unsent_test.go:20", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/osupdate/unsent_test.go:127,)", + "ending": "22. Unsent OS-update report sender", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/cert.go:38-52", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "1" + }, + { + "ptr": "AG/internal/localapi/cert.go:21-24", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "1" + }, + { + "ptr": "AG/internal/localapi/cert.go:25", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/cert.go:65", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/cert.go:23-24", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/cert.go:1", + "file": "OK", + "ident": "IDENT-MISS(NotAfter|expir)", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "4" + }, + { + "ptr": "AG/internal/localapi/cert.go:70", + "file": "OK", + "ident": "IDENT-MISS(NotAfter|expir)", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "4" + }, + { + "ptr": "AG/internal/localapi/cert.go:34-37", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "5" + }, + { + "ptr": "AG/internal/localapi/cert.go:21-25", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:1021-1034", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:4149", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:513", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/escrow_ceremony.go:158-166", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/netverifyjob.go:51", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/hostpoweroff.go:76", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/controllerswap.go:86-87", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:88-98", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "AG/internal/proxmox/task.go:110-141", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "AG/internal/thincheck/thincheck.go:183", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "AG/internal/poke/poke.go:44", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "AG/internal/backup/runner.go:172", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "AG/internal/osupdate/leg.go:589", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "AG/internal/reconcile/restoretest.go:485", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "AG/internal/config/config.go:541", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:1", + "file": "OK", + "ident": "IDENT-MISS(timedatectl|chrony|timesyncd|set-timezone|ntp)", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "AR/03-host-agent.md:72", + "file": "OK", + "ident": "IDENT-MISS(WatchdogSec)", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "AR/07-backup-architecture.md:555-558", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + }, + { + "ptr": "AG/internal/storage/observe.go:184-193", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Local-API certificate lifetime (a ten-year deadline)", + "q": "8" + } + ] +} \ No newline at end of file diff --git a/documentation/audits/clock-list-2026-11-11/evidence/helpers/resolved-clock-controller.json b/documentation/audits/clock-list-2026-11-11/evidence/helpers/resolved-clock-controller.json new file mode 100644 index 00000000..14470f8a --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/helpers/resolved-clock-controller.json @@ -0,0 +1,2690 @@ +{ + "endings": [ + { + "title": "1. Night dump leg (databases and volumes) at W", + "answers": { + "1": "`dbDumpLeg` `CT/cmd/controller/main.go:1336-1344` runs `RunDBDumps` `CT/internal/backup/backup.go:564`; owner: controller; started by the daily job \"db-dump\" `CT/cmd/controller/main.go:1345`. CUSTOMER.", + "2": "Daily at W. W = household setting, else `DBDumpSchedule`, else `DefaultWindow` \"02:30\" `CT/internal/backupwindow/backupwindow.go:12`, decided in `EffectiveWindow` `CT/internal/backupwindow/backupwindow.go:86-94`. First leg. Zone: Europe/Budapest, hard-coded, `getBudapestLocation` `CT/internal/scheduler/scheduler.go:17-27`.", + "3": "One database dump: 5 min limit `CT/internal/appbackup/dbdump.go:291`. Whole leg limit: NOT FOUND (grep `WithTimeout|WithDeadline` in backup.go gave only `CT/internal/backup/backup.go:983` and `CT/internal/backup/backup.go:1268`). Grows with number and size of databases and volumes.", + "4": "Box off at W: `Missed` `CT/internal/nightchain/nightchain.go:197-218` sees it from the ledger and the catch-up runs it (block 4). The hub judges a missing dump at its 05:00 deadline `AR/07-backup-architecture.md:579-584`.", + "5": "`withLeg` takes the leg lock `CT/cmd/controller/main.go:1321-1335`; `acquireRunning` refuses a second backup `CT/internal/backup/backup.go:1105-1113`; a job still running is skipped `executeJob` `CT/internal/scheduler/scheduler.go:354-364`. A running migration skips the leg and returns no error `CT/internal/backup/backup.go:565-568`.", + "6": "Slot from the wall clock, waited on a monotonic timer `runDailyJob` `CT/internal/scheduler/scheduler.go:300-352`; fired over 60 min late = skipped `DailyLateLimit` `CT/internal/scheduler/scheduler.go:67`. See block 24 for the hour change.", + "7": "Failure: `NotifyDBDumpFailed` `CT/cmd/controller/main.go:1339`. A made-up miss: one timeline line, never mailed `AR/07-backup-architecture.md:562-564`. A miss not made up: banner after 26 h `StaleAfter` `CT/internal/nightchain/banner.go:23`.", + "8": "TEST `TestCatchUp_OffAtWThenOn_OneCatchUpAfter15Min` `CT/internal/nightchain/nightchain_test.go:92`; TEST `TestDaily_LateFireIsSkipped` `CT/internal/scheduler/scheduler_test.go:87`." + }, + "extra": [] + }, + { + "title": "2. Second-drive copy leg at W+60 min", + "answers": { + "1": "`RunAllTier2` `CT/internal/backup/tier2.go:534` copies each app's data to the second drive; owner: controller; started by the daily job \"tier2-backup\" `CT/cmd/controller/main.go:1442`. CUSTOMER.", + "2": "Daily at W+60 min, `tier2OffsetMin` `CT/internal/backupwindow/backupwindow.go:17`; no setting of its own. It does not wait for the dump leg: \"NOTHING waits on the leg before it\" `AR/09-update-architecture.md:2751-2757`. Zone as block 1.", + "3": "One copy: 60 min, `tier2CopyTimeout` `CT/internal/backup/tier2_copy_lifecycle.go:40`, applied at `CT/internal/backup/tier2.go:828`. No limit for the whole leg found (same grep as block 1). Grows with app count and data size.", + "4": "Catch-up as block 4 (the leg is in the ledger through `withLeg` `CT/cmd/controller/main.go:1438-1441`). The leg function always returns nil `CT/cmd/controller/main.go:1438-1441`, so the scheduler never sees a failure.", + "5": "Leg lock `CT/cmd/controller/main.go:1321-1335`. Skipped while a migration runs `CT/internal/backup/tier2.go:538-541`. `acquireRunning` in tier2.go: NOT FOUND (grep `acquireRunning\\(\\)|TryLock` in tier2.go, 0 hits). A stray copy container is retried every minute `tier2RetryEvery` `CT/internal/backup/tier2_copy_lifecycle.go:172`.", + "6": "As block 1: wall-clock slot, monotonic wait, 60 min late limit `CT/internal/scheduler/scheduler.go:67`.", + "7": "Who is told of a failed copy: NOT FOUND in the wiring (grep `Notify[A-Za-z]+\\(` in main.go lines 1300-1460 shows only the dump and the off-site notices, `CT/cmd/controller/main.go:1339` and `CT/cmd/controller/main.go:1446-1452`). Candidate, the answer may sit inside tier2.go.", + "8": "SOURCE `CT/internal/backup/tier2.go:534`; catch-up part TEST `TestCatchUp_PowerCutMidChain_FinishesTheRest` `CT/internal/nightchain/nightchain_test.go:145`." + }, + "extra": [] + }, + { + "title": "3. Off-site push leg at W+105 min (with clean-up of old snapshots)", + "answers": { + "1": "`RunOffboxBackup` `CT/internal/backup/offbox.go:1018` pushes dumps to the remote store, then runs retention; owner: controller; started by the daily job \"offbox-backup\" `CT/cmd/controller/main.go:1641-1643`. CUSTOMER.", + "2": "Daily at W+105 min, `offboxOffsetMin` `CT/internal/backupwindow/backupwindow.go:18`. Retention keeps 7 daily, `keepDaily` `CT/internal/backup/offbox_window.go:47`. Zone as block 1.", + "3": "Per app 2 h, `offboxBackupTimeout` `CT/internal/backup/offbox.go:39`, applied at `CT/internal/backup/offbox.go:1564`; SSH connect 10 s `CT/internal/backup/offbox.go:37`. Grows with data changed and line speed. One real night: 3 min 28 s `AR/09-update-architecture.md:2751-2757`.", + "4": "Catch-up as block 4. No escrowed key, orphaned store or migration: skipped with an INFO log line and nil `CT/internal/backup/offbox.go:1040-1053`.", + "5": "Another backup running: \"skipped \u2014 another backup is running\", returns nil, \"the next scheduled run retries\" `CT/internal/backup/offbox.go:1054-1057`. So a long second-drive leg can cost the night's off-site push (candidate).", + "6": "As block 1. The clean-up window has a one-hour clock tolerance `offsiteGuardSkew` `CT/internal/backup/offbox_window.go:42`.", + "7": "Failure: `NotifyOffboxBackupFailed` with a cleaned message `CT/cmd/controller/main.go:1446-1452`. The hub's daily deadline is the independent signal `CT/cmd/controller/main.go:1524-1528`.", + "8": "SOURCE `CT/internal/backup/offbox.go:1040-1057`; DOC `AR/07-backup-architecture.md:455-457`." + }, + "extra": [] + }, + { + "title": "4. Missed-night catch-up and resume watch", + "answers": { + "1": "`CatchUp.Evaluate` `CT/internal/nightchain/nightchain.go:281-299` runs the backup legs a box missed; owner: controller; started at controller start `CT/cmd/controller/main.go:1676` and by the resume watch `CT/cmd/controller/main.go:1677`. CUSTOMER.", + "2": "15 min after the trigger, `DefaultDelay` `CT/internal/nightchain/nightchain.go:62`; a leg due within 30 min is left alone, `leaveToNormal` `CT/internal/nightchain/nightchain.go:64`. Values decided by CC `AR/07-backup-architecture.md:574-577`. No setting.", + "3": "As long as the legs it runs. It waits up to 2 h for a whole-guest backup, `quiesceWaitMax` `CT/internal/nightchain/nightchain.go:66`, then drops with a WARN `CT/internal/nightchain/nightchain.go:319`.", + "4": "If the catch-up itself does not run: banner after 26 h (block 5) and the hub's longer lines, 48 h without a dump `AR/07-backup-architecture.md:579-584`. Two missed nights give one catch-up, not two.", + "5": "One at a time: pending/running guard `CT/internal/nightchain/nightchain.go:288-292`; leg lock `CT/cmd/controller/main.go:1321-1335`. It never runs the update leg `CT/cmd/controller/main.go:1660-1664`.", + "6": "`ResumeWatch` compares wall clock and monotonic clock each minute; a gap over 5 min triggers, `d > gap` `CT/internal/nightchain/nightchain.go:375`, gap set at `CT/cmd/controller/main.go:1681`. A forward clock correction over 5 min looks the same as a resume (candidate). \"not measured live\" `AR/07-backup-architecture.md:551-559`.", + "7": "Household: one timeline line, never mailed `AR/07-backup-architecture.md:562-564`, event sent at `CT/internal/nightchain/nightchain.go:357-359`. Dropped catch-up: log line only `CT/internal/nightchain/nightchain.go:319`.", + "8": "TEST `TestCatchUp_TwoMissedNights_One` `CT/internal/nightchain/nightchain_test.go:126`; TEST `TestResumeWatch_SeesASuspend` `CT/internal/nightchain/nightchain_test.go:217`; TEST `TestR871_CatchUpWiring` `CT/cmd/controller/r871_catchup_wiring_test.go:16`." + }, + "extra": [] + }, + { + "title": "5. Missed-backup banner (26 hours)", + "answers": { + "1": "`ComputeBanner` `CT/internal/nightchain/banner.go:56-88` shows the household a banner when no night backup ran; owner: controller; computed when the page is drawn (no timer of its own). CUSTOMER.", + "2": "Shown when the last backup is older than 26 h, `StaleAfter` `CT/internal/nightchain/banner.go:23`. After 5 missed days it suggests another hour, `suggestMinDays` `CT/internal/nightchain/banner.go:26`, `Suggest` `CT/internal/nightchain/banner.go:95-113`.", + "3": "Instant; reads the ledger only `CT/internal/nightchain/banner.go:56-88`.", + "4": "Nobody opens the page: nobody sees it. The hub's deadline check is the other channel `AR/07-backup-architecture.md:579-584`.", + "5": "Not a job; no lock needed. Dismissed, it comes back after a new miss, design at `AR/07-backup-architecture.md:566-572`.", + "6": "Age is wall-clock arithmetic `CT/internal/nightchain/banner.go:56-88`; a clock set back hides the banner for as long as the step (candidate). A test for a clock set back: NOT FOUND (grep `^func Test.*(Back|Clock)` in banner_test.go gave only the dismiss test).", + "7": "The household only, on the page; no mail `AR/07-backup-architecture.md:566-572`.", + "8": "TEST `TestBanner_ShownWithReasonAndSuggestion` `CT/internal/nightchain/banner_test.go:29`; TEST `TestBanner_DismissedThenBackAfterANewMiss` `CT/internal/nightchain/banner_test.go:59`." + }, + "extra": [] + }, + { + "title": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "answers": { + "1": "`Loop.Run` `CT/internal/quiesce/quiesce.go:191-206` stops the apps and asks the agent for the whole-guest backup; owner: controller drives, agent does the backup; started by `startQuiesceLoop` `CT/cmd/controller/main.go:4008`. CUSTOMER.", + "2": "Polls every 5 min `CT/internal/quiesce/quiesce.go:149-151`; runs once per 24 h cadence `CT/internal/quiesce/quiesce.go:161-163`, only inside W+2h to W+6h, `gateOpenOffsetMin` `CT/internal/quiesce/quiesce.go:760`, `gateSpanMin` `CT/internal/quiesce/quiesce.go:761`. Zone `budapestLocation` `CT/internal/quiesce/quiesce.go:747-757`.", + "3": "Apps stay down at most 30 min, `MaxQuiesce` default `CT/cmd/controller/main.go:3990`, setting `max_quiesce_duration` `CT/internal/config/config.go:132`. Manual run limit = that + 5 min `CT/internal/quiesce/quiesce.go:430`. Grows with guest disk size.", + "4": "Outside the window it still runs when the last one is older than cadence + 24 h (the 48 h valve) `CT/internal/quiesce/quiesce.go:801-818`. Apps left stopped by a crash are started by `Recover` `CT/internal/quiesce/quiesce.go:177-188`, called at `CT/cmd/controller/main.go:4007`.", + "5": "Single-flight `TryLock` `CT/internal/quiesce/quiesce.go:213-216`; waits for a catch-up `CT/internal/quiesce/quiesce.go:250-253`; waits for the update leg `CT/internal/quiesce/quiesce.go:271-274`, never past W+5h30 `legStepGraceMin` `CT/internal/quiesce/quiesce.go:935`.", + "6": "Gate is minute-of-day arithmetic on the wall clock `CT/internal/quiesce/quiesce.go:801-818`; the poll is a monotonic ticker `CT/internal/quiesce/quiesce.go:191-206`. Hour-change night: NOT FOUND a test (grep `DST|Budapest|SpringForward|Daylight` in `^func Test` lines, 0 hits).", + "7": "First failure notifies once `noteTierFailure` `CT/internal/quiesce/quiesce.go:283-298`; retry back-off 15 min to 4 h `CT/internal/quiesce/breaker.go:47`, `CT/internal/quiesce/breaker.go:54`; busy agent: retry 15 min, alarm after 3 h `CT/internal/quiesce/contention.go:53`, `CT/internal/quiesce/contention.go:64`.", + "8": "TEST `TestScheduledRunAllowed` `CT/internal/quiesce/quiesce_window_test.go:23`; TEST `TestRunOnce_ValveRunsOutsideWindow` `CT/internal/quiesce/quiesce_window_test.go:94`; TEST `TestBreaker_BackoffSchedule` `CT/internal/quiesce/breaker_test.go:218`." + }, + "extra": [] + }, + { + "title": "7. Automatic app-update leg (after the off-site push)", + "answers": { + "1": "`RunUpdateLeg` `CT/internal/stacks/unattended.go:230` presses one update step per app; owner: controller; started by `chainUpdateLeg` when the off-site half ends, on every path `CT/cmd/controller/main.go:1641-1643`. CUSTOMER.", + "2": "Each night after the off-site leg; a box with no off-site tier runs it at W+105 min `AR/07-backup-architecture.md:459-465`. One step per app per night, switch default ON `AR/09-update-architecture.md:2806-2817`.", + "3": "No new step at or after W+5h: `LegDeadline` `CT/internal/stacks/unattended.go:212-216`, `UpdateLegStopOffsetMin` `CT/internal/backupwindow/backupwindow.go:27`; check at `CT/internal/stacks/unattended.go:374-375`. Grows with the number of apps that have an update.", + "4": "Missed night: no catch-up by design `CT/cmd/controller/main.go:1660-1664`; the rest \"waits for the next night\" `CT/internal/stacks/unattended.go:374-375`. Controller restarted mid-leg: the rest of the night is lost (R-686) `AR/09-update-architecture.md:2815-2817`.", + "5": "A passing refusal (busy, updating) is retried once after 2 min, `RetryWait` `CT/internal/stacks/unattended.go:167-171`, rule at `AR/09-update-architecture.md:2882-2883`. The whole-guest gate waits for the leg `CT/internal/quiesce/quiesce.go:271-274`.", + "6": "Deadline is a wall-clock time in `time.Local` when no location is given `CT/internal/stacks/unattended.go:212-216`, while the scheduler uses Europe/Budapest `CT/internal/scheduler/scheduler.go:17-27` (candidate: two sources of zone). A late daily fire over 60 min is skipped, so no noon updates `AR/07-backup-architecture.md:551-559`.", + "7": "NOT FOUND in this read for who is told when the leg does not run (grep `update-leg` gave the log line `CT/internal/stacks/unattended.go:374-375` only).", + "8": "TEST `TestChainUpdateLeg_EveryPath` `CT/cmd/controller/update_leg_wiring_test.go:19`; TEST `TestLegDeadlineAndGateShareW5h` `CT/internal/quiesce/d20_update_leg_test.go:78`." + }, + "extra": [] + }, + { + "title": "8. Off-site integrity check at 06:00", + "answers": { + "1": "Daily job \"offsite-integrity\" `CT/cmd/controller/main.go:1724` checks the remote store's data; owner: controller. CUSTOMER.", + "2": "Fires daily at 06:00 (fixed, no setting for the hour) but works only when due: last good check older than 7 days, `defaultIntegrityMaxAgeDays` `CT/internal/backup/offbox_integrity.go:81`, `IntegrityDue` `CT/internal/backup/offbox_integrity.go:221`. Zone as block 1.", + "3": "Limit 30 min, `integrityCheckTimeout` `CT/internal/backup/offbox_integrity.go:63`; a notice when slower than 5 min `CT/internal/backup/offbox_integrity.go:78`. Grows with store size and line speed.", + "4": "Box off at 06:00: caught up at the next 06:00 by due-ness `CT/internal/backup/offbox_integrity.go:212-220`; also run after a catch-up `CT/cmd/controller/main.go:1665-1668`.", + "5": "Skipped when \"a backup or restore is already running\" `CT/internal/backup/offbox_integrity.go:298`; then it waits for the next 06:00 (source comment `CT/cmd/controller/main.go:1710-1723`).", + "6": "Due-ness compares wall-clock stamps `CT/internal/backup/offbox_integrity.go:231`; a never-checked or unreadable stamp counts as due `CT/internal/backup/offbox_integrity.go:223-229`. Slot behaviour as block 24.", + "7": "NOT FOUND in this read (the job body `CT/cmd/controller/main.go:1724-1729` was read; the notify path inside offbox_integrity.go was not).", + "8": "SOURCE `CT/internal/backup/offbox_integrity.go:212-232`; TEST `TestR1189d_TheChecksRunAfterTheCatchUpLegs` `CT/internal/nightchain/r1189d_test.go:13`." + }, + "extra": [] + }, + { + "title": "9. Off-site proof restore at 05:30", + "answers": { + "1": "Daily job \"offsite-proof\" `CT/cmd/controller/main.go:1746` restores one due snapshot to prove it opens, `ProveOffboxUnit` `CT/internal/backup/offbox_proof.go:140`; owner: controller. CUSTOMER.", + "2": "Daily at 05:30, fixed, no setting `CT/cmd/controller/main.go:1746`. One app per night in rotation (comment `CT/internal/backup/offbox_proof.go:265`). Zone as block 1.", + "3": "Limit 10 min, `proofRestoreTimeout` `CT/internal/backup/offbox_proof.go:80`. Grows with the size of the one unit.", + "4": "Box on for the backup legs but off at 05:30: \"that day's proof is not made up\" `CT/cmd/controller/main.go:1660-1664`. Made up only together with a missed backup leg `CT/cmd/controller/main.go:1665-1668`.", + "5": "Skipped when \"a backup or restore is already running\" `CT/internal/backup/offbox_proof.go:149`. With W = 02:30 the whole-guest gate is open from 04:30 `AR/09-update-architecture.md:2751-2757`, so 05:30 can fall inside it (candidate).", + "6": "As block 24.", + "7": "NOT FOUND in this read (grep `Notify` near `CT/cmd/controller/main.go:1746-1749` `CT/cmd/controller/main.go:0` hits there).", + "8": "SOURCE `CT/cmd/controller/main.go:1660-1664`; TEST `TestC5_TheWholeGuestBackupDoesNotWaitForTheChecks` `CT/internal/nightchain/r1189d_test.go:51`." + }, + "extra": [] + }, + { + "title": "10. Hub report every 15 minutes (plus push on change and hanging wait)", + "answers": { + "1": "Job \"hub-report\" `CT/cmd/controller/main.go:1835` sends the box state to the hub; owner: controller. The hub's answer carries the version floor and config `CT/cmd/controller/main.go:1233-1290`. CUSTOMER and OPERATOR.", + "2": "Every `PushInterval`, default \"15m\" `CT/internal/config/config.go:416`, fallback `CT/cmd/controller/main.go:1831-1834`. Also 5 s after start `CT/cmd/controller/main.go:1932-1936`, and on change with a 15 s floor, `triggerMinInterval` `CT/internal/report/trigger.go:30`. No time zone.", + "3": "HTTP limit 30 s `CT/internal/report/pusher.go:85`; 3 tries, 5 s apart `CT/internal/report/pusher.go:114-117`. The wait connection holds up to 300 s `CT/internal/report/waiter.go:35`.", + "4": "A failed push is not queued: \"next cycle reconciles\" `CT/internal/report/trigger.go:98`. The hub notices silence (hub side, not read here).", + "5": "A report still running is skipped `CT/internal/scheduler/scheduler.go:354-364`. Bursts are merged after a 2 s quiet window `CT/internal/report/trigger.go:27`.", + "6": "Monotonic ticker `CT/internal/scheduler/scheduler.go:281-298`; a wrong wall clock does not change the rhythm, only the stamps inside the report.", + "7": "Waiter back-off 5 s to 5 min `CT/internal/report/waiter.go:36-37`. Who is told on the box that pushes fail: NOT FOUND (only `LastAttempt` is stored `CT/internal/report/pusher.go:109-111`).", + "8": "TEST `TestTrigger_BurstCoalescesTrailingEdge` `CT/internal/report/trigger_test.go:114`; TEST `TestWaiter_FiresOnceOnGenChange` `CT/internal/report/waiter_test.go:60`." + }, + "extra": [] + }, + { + "title": "11. Controller self-update: check, floor, nightly auto", + "answers": { + "1": "`CheckForUpdate` `CT/internal/selfupdate/updater.go:153` looks for a newer controller image; `MaybeAutoUpdate` `CT/internal/selfupdate/updater.go:612` moves a box below the floor. Owner: controller, the agent swaps the image. CUSTOMER and OPERATOR.", + "2": "Check every `CheckInterval`, default \"6h\" `CT/internal/config/config.go:417`, and 25 s after start `CT/cmd/controller/main.go:1981-1989`. Floor: after every hub report `CT/cmd/controller/main.go:1249-1252`. Auto: daily at `AutoUpdateTime`, default \"04:30\" `CT/internal/config/config.go:419`, only if `AutoUpdate` `CT/internal/config/config.go:278`.", + "3": "Registry call limit 15 s `CT/internal/selfupdate/updater.go:245`. Limit of the swap itself: NOT FOUND in this read (grep `time\\.(Second|Minute|Hour)` in updater.go gave one hit).", + "4": "No catch-up needed for the floor: it is tried again on each report until done, but one attempt per floor value `CT/internal/selfupdate/updater.go:656-661`. A failed attempt at a floor is not repeated `CT/internal/selfupdate/updater.go:604-606`.", + "5": "`updateRunning` flag `CT/internal/selfupdate/updater.go:520-524`; refused while a backup runs `CT/internal/selfupdate/updater.go:533-536` or an app update is in flight `CT/internal/selfupdate/updater.go:540-544`. Default 04:30 equals gate-open time at W = 02:30 (candidate).", + "6": "Check: monotonic ticker. Auto: daily slot as block 24. Default for `AutoUpdate`: NOT FOUND in the defaults block (grep `AutoUpdate ` in config.go: only the field `CT/internal/config/config.go:278`).", + "7": "Auto-update refused: WARN log only `CT/cmd/controller/main.go:1879-1881`. A version the floor cannot compare: WARN once `CT/internal/selfupdate/updater.go:639`.", + "8": "TEST `TestMaybeAutoUpdate_NoFlap` `CT/internal/selfupdate/floor_test.go:93`; TEST `TestMaybeAutoUpdate_BelowFloor_UpdatesToFloor` `CT/internal/selfupdate/floor_test.go:11`." + }, + "extra": [] + }, + { + "title": "12. Catalog sync (app templates from git)", + "answers": { + "1": "`Syncer.Start` `CT/internal/sync/sync.go:125` pulls the app catalog; owner: controller; started at controller start. CUSTOMER.", + "2": "Once at start `CT/internal/sync/sync.go:140-143`, then every `SyncInterval`, default \"15m\" `CT/internal/config/config.go:399`; a bad value falls back to 15 min `CT/internal/sync/sync.go:131-135`. No time zone.", + "3": "Time limit for one sync: NOT FOUND in the lines read (`CT/internal/sync/sync.go:124-188`; `doSync` body not read).", + "4": "Missed sync: the next tick does it. Empty repo address: sync is off with one WARN `CT/internal/sync/sync.go:126-129`. Who notices a long failure: NOT FOUND.", + "5": "Manual sync is refused while one runs or within 30 s of the last `CT/internal/sync/sync.go:175-185`. The periodic tick calls `doSync` directly without that check `CT/internal/sync/sync.go:153-155` (candidate: read `doSync` before judging).", + "6": "Monotonic ticker `CT/internal/sync/sync.go:146`; own ticker, not the shared scheduler, so it has no \"still running\" skip from `CT/internal/scheduler/scheduler.go:354-364`.", + "7": "Log line per sync only `CT/internal/sync/sync.go:155`.", + "8": "SOURCE `CT/internal/sync/sync.go:124-159`." + }, + "extra": [] + }, + { + "title": "13. System health check and base-stack self-heal", + "answers": { + "1": "Job \"system-health\" `CT/cmd/controller/main.go:981-1006` checks disk, memory and base services, and puts the base stack back; owner: controller. CUSTOMER.", + "2": "Every `SystemHealthInterval`, default \"5m\" `CT/internal/config/config.go:409`, fallback `CT/cmd/controller/main.go:977-980`; env override `CT/internal/config/config.go:453`. First run one interval after start `CT/internal/scheduler/scheduler.go:281-298`.", + "3": "Time limit: NOT FOUND in the job body `CT/cmd/controller/main.go:981-1006`.", + "4": "No catch-up; next tick. A controller that is down checks nothing; the hub's missing reports are the outside sign (block 10).", + "5": "Skipped while the previous run is still going `CT/internal/scheduler/scheduler.go:354-364`.", + "6": "Monotonic ticker; not sensitive to wall clock.", + "7": "`NotifyHealthChange` on a change of state `CT/cmd/controller/main.go:1004`.", + "8": "SOURCE `CT/cmd/controller/main.go:981-1006`." + }, + "extra": [] + }, + { + "title": "14. Dead-app scan every 30 seconds", + "answers": { + "1": "Job \"deadapp-check\" `CT/cmd/controller/main.go:1086-1092` finds apps that should run and do not; owner: controller. CUSTOMER.", + "2": "Every 30 s; silent for 90 s after start, about 15 min after a crash boot `CT/cmd/controller/main.go:1087-1088`.", + "3": "Time limit: NOT FOUND (job body `CT/cmd/controller/main.go:1086-1143` has no context limit). Grows with app count.", + "4": "Next tick; no catch-up needed.", + "5": "Skipped if still running `CT/internal/scheduler/scheduler.go:354-364`. It reads the whole-guest loop state so a planned stop is not an alarm `CT/cmd/controller/main.go:1090`.", + "6": "Boot grace uses `time.Now()` `CT/cmd/controller/main.go:1087`; a clock stepped back after start would lengthen the silence (candidate, no test found).", + "7": "Alert on the page via `SetDeadAppAlerts` `CT/cmd/controller/main.go:1091`.", + "8": "SOURCE `CT/cmd/controller/main.go:1086-1092`." + }, + "extra": [] + }, + { + "title": "15. Disk-fill watch", + "answers": { + "1": "`fillWatcher.Check` watches how full the drives are; owner: controller; three starters `CT/cmd/controller/main.go:1804-1805`. CUSTOMER.", + "2": "Every 10 min, `fillWatchInterval` `CT/cmd/controller/main.go:2690`; daily at 03:30 `CT/cmd/controller/main.go:1804`; once 90 s after start, `fillWatchStartupDelay` `CT/cmd/controller/main.go:2685`. Cadence stated at `CT/cmd/controller/main.go:1772`.", + "3": "Time limit: NOT FOUND (the callbacks ignore the context `CT/cmd/controller/main.go:1804-1805`).", + "4": "Next 10-minute tick.", + "5": "The 03:30 slot and the interval job are two jobs with two names `CT/cmd/controller/main.go:1804-1805`; a lock between them: NOT FOUND in the wiring (the fillwatch package was not read).", + "6": "Interval is monotonic; the 03:30 slot as block 24.", + "7": "NOT FOUND in this read (fillwatch package not read).", + "8": "TEST `TestFillWatchRunsOnAnInterval` `CT/cmd/controller/r363_fillwatch_interval_test.go:17`." + }, + "extra": [] + }, + { + "title": "16. Drive health check, hourly", + "answers": { + "1": "Job \"disk-health-check\" `CT/cmd/controller/main.go:2226` asks the agent for drive health; owner: controller reads, agent measures. CUSTOMER.", + "2": "Every 1 h; reason for hourly, not 6 h: `CT/cmd/controller/main.go:2219-2224`. Only when the agent channel is configured `CT/cmd/controller/main.go:2225`.", + "3": "One fetch measured about 0.8 s `CT/cmd/controller/main.go:2219-2224`. Limit: NOT FOUND.", + "4": "Next hour; first run one hour after start `CT/internal/scheduler/scheduler.go:281-298`.", + "5": "Skipped if still running `CT/internal/scheduler/scheduler.go:354-364`.", + "6": "Monotonic ticker.", + "7": "NOT FOUND in this read (`RunDiskHealthCheck` body not read).", + "8": "SOURCE `CT/cmd/controller/main.go:2219-2226`." + }, + "extra": [] + }, + { + "title": "17. Agent channel health, every 60 seconds", + "answers": { + "1": "Job \"agent-channel-health\" `CT/cmd/controller/main.go:2289` probes the controller-to-agent channel; owner: controller. OPERATOR.", + "2": "Every 60 s. A passing fault needs 2 misses in a row, `debounceThreshold` `CT/internal/channelhealth/checker.go:29`.", + "3": "Probe limit 15 s, `probeTimeout` `CT/internal/channelhealth/checker.go:32`.", + "4": "Next minute.", + "5": "Skipped if still running `CT/internal/scheduler/scheduler.go:354-364`. An alert marker file survives a restart `CT/cmd/controller/main.go:2288`.", + "6": "Monotonic ticker.", + "7": "Sink = notifier and page alert `CT/cmd/controller/main.go:2286`.", + "8": "TEST `TestDebounce_TransientBlipSuppressed` `CT/internal/channelhealth/checker_test.go:119`." + }, + "extra": [] + }, + { + "title": "18. Held-app report to the hub", + "answers": { + "1": "`Reporter.Tick` `CT/internal/holdreport/holdreport.go:89` tells the hub which apps are held; owner: controller; job \"hold-report\" `CT/cmd/controller/main.go:1081-1084`. OPERATOR.", + "2": "Looks every minute; each hold is sent again once a day, `ResendEvery` `CT/internal/holdreport/holdreport.go:36`, and after each start `CT/cmd/controller/main.go:1071-1073`.", + "3": "Instant; reads the hold store.", + "4": "Next minute.", + "5": "Skipped if still running `CT/internal/scheduler/scheduler.go:354-364`.", + "6": "`Tick` gets `time.Now()` `CT/cmd/controller/main.go:1082`; the daily resend is wall-clock arithmetic (candidate: a clock set back delays the resend).", + "7": "The hub, by event `CT/cmd/controller/main.go:1074`.", + "8": "TEST `TestD64_TheHoldReporterTellsTheHubAfterTheClear` `CT/cmd/controller/holdclear_test.go:333`." + }, + "extra": [] + }, + { + "title": "19. Abandoned off-site store sweep at 05:10", + "answers": { + "1": "Daily job \"offsite-abandon-sweep\" `CT/cmd/controller/main.go:1693` runs `AbandonSweep` `CT/internal/backup/offbox_abandon.go:214`; owner: controller. CUSTOMER.", + "2": "Daily at 05:10, fixed. The due date is counted in UTC days `CT/internal/backup/offbox_abandon.go:366`, the slot in Europe/Budapest `CT/internal/scheduler/scheduler.go:17-27`.", + "3": "NOT FOUND (body of `AbandonSweep` not read).", + "4": "Box off at 05:10: next day; no catch-up (it is not a ledger leg, see `CT/cmd/controller/main.go:1660-1664`).", + "5": "NOT FOUND (grep `acquireRunning\\(\\)|TryLock` in offbox_abandon.go, 0 hits).", + "6": "As block 24.", + "7": "NOT FOUND in this read.", + "8": "SOURCE `CT/cmd/controller/main.go:1693`." + }, + "extra": [] + }, + { + "title": "20. Restore-copy sweep and conversion-copy release", + "answers": { + "1": "`SweepUnusedRestoreCopies` `CT/internal/backup/restore_copy_sweep.go:36` deletes old kept copies; job \"restore-copy-sweep\" `CT/cmd/controller/main.go:1414`; job \"conversion-copy-release\" `CT/cmd/controller/main.go:1407`. CUSTOMER.", + "2": "Sweep every 6 h; release every 1 h. Age line: the wiring comment says 14 days `CT/cmd/controller/main.go:1412`, the constant says 30 days, `restoreCopyUnusedAge` `CT/internal/backup/restore_copy_sweep.go:29` (candidate: comment and constant differ).", + "3": "NOT FOUND.", + "4": "Next tick; first run 6 h after start `CT/internal/scheduler/scheduler.go:281-298`, so a box on for under 6 h a day never sweeps (candidate).", + "5": "Skipped if still running `CT/internal/scheduler/scheduler.go:354-364`.", + "6": "Age is wall-clock arithmetic on `now` `CT/internal/backup/restore_copy_sweep.go:36`.", + "7": "NOT FOUND.", + "8": "SOURCE `CT/internal/backup/restore_copy_sweep.go:29`." + }, + "extra": [] + }, + { + "title": "21. Session clean-up (dashboard sign-ins)", + "answers": { + "1": "`cleanupSessions` `CT/internal/web/auth.go:369` removes expired sign-in sessions; owner: controller web server. CUSTOMER.", + "2": "Every 15 min `CT/internal/web/auth.go:370`. A session lives 7 days, `sessionMaxAge` `CT/internal/web/auth.go:30`. Family sessions 30 days, `SessionLife` `CT/internal/family/family.go:42`; share cookie 30 days `CT/internal/web/share.go:31`.", + "3": "Instant; grows with session count.", + "4": "Not run: expired sessions stay in the file. Whether an expired session is refused at use: NOT FOUND in the lines read (`CT/internal/web/auth.go:355-395`).", + "5": "`sessionsMu` lock `CT/internal/web/auth.go:377`.", + "6": "Expiry compares the wall clock, `now.After(sess.expiresAt)` `CT/internal/web/auth.go:381`; a clock set forward signs everyone out, set back keeps sessions longer (candidate).", + "7": "Log line only `CT/internal/web/auth.go:392`.", + "8": "SOURCE `CT/internal/web/auth.go:369-393`." + }, + "extra": [] + }, + { + "title": "22. Upload and CPU watch, torrent limit rule, line-test hold", + "answers": { + "1": "`startUpwatch` `CT/cmd/controller/upwatch_wiring.go:136` runs the minute watcher and a 15-minute rule that holds the torrent app for the line test; owner: controller. CUSTOMER.", + "2": "Watcher every 1 min `CT/cmd/controller/upwatch_wiring.go:139`; rule every 15 min `CT/cmd/controller/upwatch_wiring.go:141`. Baseline: 7 days of data, cached 24 h `CT/cmd/controller/upwatch_wiring.go:124-127`.", + "3": "Hold call limit 30 s, `holdTimeout` `CT/internal/upwatch/linehold.go:79`; drain wait at most 90 s `CT/internal/upwatch/linehold.go:216`.", + "4": "A limit left at the test value by a stopped controller is put back at start, `RecoverHold` `CT/cmd/controller/upwatch_wiring.go:137`.", + "5": "Own tickers; no scheduler skip. Lock between the two loops: NOT FOUND in the lines read.", + "6": "Monotonic tickers `CT/internal/upwatch/watch.go:151`.", + "7": "NOT FOUND in this read (`emit` `CT/internal/upwatch/watch.go:382` not followed).", + "8": "SOURCE `CT/cmd/controller/upwatch_wiring.go:134-152`." + }, + "extra": [] + }, + { + "title": "23. Old image clean-up after start", + "answers": { + "1": "`RunImageRetentionOnceUntilDone` `CT/internal/stacks/image_retention.go:399` deletes app images no app uses; owner: controller; started about 3 min after start `CT/cmd/controller/main.go:2048-2069`. CUSTOMER.", + "2": "Once per start, retried every 2 min up to 30 times `CT/cmd/controller/main.go:2056`. Also after an update or a remove (test names below).", + "3": "Up to about 60 min of retries (30 x 2 min) `CT/cmd/controller/main.go:2056`.", + "4": "A box that never restarts and never updates runs no sweep (candidate).", + "5": "No pass while an update runs: TEST `TestImageRetention_NoPassWhileAnUpdateRuns` `CT/internal/stacks/image_retention_test.go:238`.", + "6": "Monotonic waits.", + "7": "Log only; an unreadable keep set deletes nothing `CT/internal/stacks/image_retention.go:221`.", + "8": "TEST `TestImageRetention_KeepsRunningAndPreviousDeletesOlder` `CT/internal/stacks/image_retention_test.go:111`." + }, + "extra": [] + }, + { + "title": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "answers": { + "1": "`Daily` `CT/internal/scheduler/scheduler.go:131-157` and `Every` `CT/internal/scheduler/scheduler.go:105-127` carry every named job; owner: controller. CUSTOMER.", + "2": "Daily slots are HH:MM in Europe/Budapest, hard-coded; if the zone data cannot load it silently uses UTC `CT/internal/scheduler/scheduler.go:17-27`. Interval jobs do not run at start: first run after one interval `CT/internal/scheduler/scheduler.go:281-298`.", + "3": "No time limit of its own; shutdown waits 30 s `CT/internal/scheduler/scheduler.go:264`. A panic in a job is caught `CT/internal/scheduler/scheduler.go:438-447`.", + "4": "A daily job has no catch-up in the scheduler; a fire over 60 min late is skipped `CT/internal/scheduler/scheduler.go:344-347`. Only the three backup legs are made up (block 4).", + "5": "One run per job at a time: \"still running\" skip `CT/internal/scheduler/scheduler.go:354-364`. No rule between different jobs here.", + "6": "Next slot by calendar day, not +24 h `CT/internal/scheduler/scheduler.go:499-519`. Spring night: 02:30 does not exist; autumn night: 02:30 comes twice. Test for either: NOT FOUND (grep `DST|Budapest|SpringForward|Daylight` in `^func Test`, 0 hits). A clock set back while waiting: the timer still fires on monotonic time.", + "7": "A skipped late fire is a log line only `CT/internal/scheduler/scheduler.go:344-347`.", + "8": "TEST `TestDaily_LateFireIsSkipped` `CT/internal/scheduler/scheduler_test.go:87`; TEST `TestUpdateDaily_GoroutineConsumesReschedule` `CT/internal/scheduler/scheduler_test.go:62`. \u00b7 Status refresh every 10 s `CT/cmd/controller/main.go:950`. \u00b7 Stack scan every 2 min `CT/cmd/controller/main.go:953`. \u00b7 Docker socket watch every 15 s and socket users every 5 min `CT/cmd/controller/main.go:960-962`. \u00b7 Health probes every 10 s `CT/cmd/controller/main.go:971`. \u00b7 After-unclean-boot repair, every 30 s until done, own `TryLock` `CT/cmd/controller/main.go:1064-1065`. \u00b7 Debug ring spill every 30 s `CT/cmd/controller/main.go:1148-1150`. \u00b7 Off-site credential retry every 5 min `CT/cmd/controller/main.go:1375`. \u00b7 Backup cache refresh every 5 min `CT/cmd/controller/main.go:1398-1402`. \u00b7 Metrics sample every 60 s `CT/internal/metrics/collector.go:53`. \u00b7 Metrics prune daily 04:00, keeps 30 days `CT/cmd/controller/main.go:1754`. \u00b7 Asset sync daily at `SyncSchedule`, default \"05:00\" `CT/internal/config/config.go:425`. \u00b7 Geo-restriction verify every 6 h `CT/cmd/controller/main.go:2102`. \u00b7 Setup-gate loop every 20 s `CT/cmd/controller/main.go:2045`. \u00b7 Busy marker written every 15 s `CT/internal/busymark/busymark.go:29`. \u00b7 Startup report: 5 s after start `CT/cmd/controller/main.go:1932-1936`. \u00b7 Stray second-drive copy container retry every minute `CT/internal/backup/tier2_copy_lifecycle.go:172`. \u00b7 App wait after install: 10 min `CT/cmd/controller/main.go:2194`. \u00b7 Family sign-in form lives 1 h `CT/internal/web/family_gate.go:56`. \u00b7 Stale pointers inside the design text: the legs are cited at old main.go lines `AR/07-backup-architecture.md:455-457` (candidate, hygiene). \u00b7 Log rotation: NOT FOUND in controller Go code. Grep `rotat|lumberjack|max-size|MaxSizeMB` found only the config fields `CT/internal/config/config.go:291-292` and their defaults `CT/internal/config/config.go:422-423`; no reader of them. \u00b7 Share-link expiry sweep: NOT FOUND. Grep `expire|expiry|ExpiresAt` in `internal/web/share*.go` gave no sweep; only the 30-day cookie `CT/internal/web/share.go:31`. \u00b7 A test for the hour-change nights or a named time zone: NOT FOUND (grep `DST|Budapest|SpringForward|Daylight` over `^func Test` in cmd and internal, 0 hits). \u00b7 Who is told when the second-drive copy, the proof, the integrity check, the fill watch or the drive health check fails: not read to the end; see line 7 of blocks 2, 8, 9, 15, 16. \u00b7 Not read at all: `CT/internal/bootrecon/bootrecon.go:147`, `CT/internal/bootstrap/waiting.go:95`, `CT/internal/offsiteapply/offsiteapply.go:137`, `CT/internal/web/operator_reboot_notice.go:78`, `CT/internal/web/handler_export_upload.go:204`, `CT/internal/web/media_events.go:97`, `CT/internal/notify/notifier.go:324`." + }, + "extra": [ + "## Tiny", + "## Not found" + ] + } + ], + "pointers": [ + { + "ptr": "CT/cmd/controller/main.go:1336-1344", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "1" + }, + { + "ptr": "CT/internal/backup/backup.go:564", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/main.go:1345", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "1" + }, + { + "ptr": "CT/internal/backupwindow/backupwindow.go:12", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "2" + }, + { + "ptr": "CT/internal/backupwindow/backupwindow.go:86-94", + "file": "OK", + "ident": "IDENT-MISS(\"02:30\")", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "2" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:17-27", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "2" + }, + { + "ptr": "CT/internal/appbackup/dbdump.go:291", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "3" + }, + { + "ptr": "CT/internal/backup/backup.go:983", + "file": "OK", + "ident": "IDENT-MISS(WithTimeout|WithDeadline)", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "3" + }, + { + "ptr": "CT/internal/backup/backup.go:1268", + "file": "OK", + "ident": "IDENT-MISS(WithTimeout|WithDeadline)", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "3" + }, + { + "ptr": "CT/internal/nightchain/nightchain.go:197-218", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "4" + }, + { + "ptr": "AR/07-backup-architecture.md:579-584", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "4" + }, + { + "ptr": "CT/cmd/controller/main.go:1321-1335", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "5" + }, + { + "ptr": "CT/internal/backup/backup.go:1105-1113", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "5" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:354-364", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "5" + }, + { + "ptr": "CT/internal/backup/backup.go:565-568", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "5" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:300-352", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "6" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:67", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "6" + }, + { + "ptr": "CT/cmd/controller/main.go:1339", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "7" + }, + { + "ptr": "AR/07-backup-architecture.md:562-564", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "7" + }, + { + "ptr": "CT/internal/nightchain/banner.go:23", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "7" + }, + { + "ptr": "CT/internal/nightchain/nightchain_test.go:92", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "8" + }, + { + "ptr": "CT/internal/scheduler/scheduler_test.go:87", + "file": "OK", + "ident": "IDENT-MISS(CT/internal/nightchain/nightchain_test.go:92;)", + "ending": "1. Night dump leg (databases and volumes) at W", + "q": "8" + }, + { + "ptr": "CT/internal/backup/tier2.go:534", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Second-drive copy leg at W+60 min", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/main.go:1442", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Second-drive copy leg at W+60 min", + "q": "1" + }, + { + "ptr": "CT/internal/backupwindow/backupwindow.go:17", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Second-drive copy leg at W+60 min", + "q": "2" + }, + { + "ptr": "AR/09-update-architecture.md:2751-2757", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Second-drive copy leg at W+60 min", + "q": "2" + }, + { + "ptr": "CT/internal/backup/tier2_copy_lifecycle.go:40", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Second-drive copy leg at W+60 min", + "q": "3" + }, + { + "ptr": "CT/internal/backup/tier2.go:828", + "file": "OK", + "ident": "IDENT-MISS(tier2CopyTimeout)", + "ending": "2. Second-drive copy leg at W+60 min", + "q": "3" + }, + { + "ptr": "CT/cmd/controller/main.go:1438-1441", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Second-drive copy leg at W+60 min", + "q": "4" + }, + { + "ptr": "CT/cmd/controller/main.go:1438-1441", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Second-drive copy leg at W+60 min", + "q": "4" + }, + { + "ptr": "CT/cmd/controller/main.go:1321-1335", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Second-drive copy leg at W+60 min", + "q": "5" + }, + { + "ptr": "CT/internal/backup/tier2.go:538-541", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Second-drive copy leg at W+60 min", + "q": "5" + }, + { + "ptr": "CT/internal/backup/tier2_copy_lifecycle.go:172", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Second-drive copy leg at W+60 min", + "q": "5" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:67", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Second-drive copy leg at W+60 min", + "q": "6" + }, + { + "ptr": "CT/cmd/controller/main.go:1339", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Second-drive copy leg at W+60 min", + "q": "7" + }, + { + "ptr": "CT/cmd/controller/main.go:1446-1452", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Second-drive copy leg at W+60 min", + "q": "7" + }, + { + "ptr": "CT/internal/backup/tier2.go:534", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Second-drive copy leg at W+60 min", + "q": "8" + }, + { + "ptr": "CT/internal/nightchain/nightchain_test.go:145", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Second-drive copy leg at W+60 min", + "q": "8" + }, + { + "ptr": "CT/internal/backup/offbox.go:1018", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Off-site push leg at W+105 min (with clean-up of old snapshots)", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/main.go:1641-1643", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Off-site push leg at W+105 min (with clean-up of old snapshots)", + "q": "1" + }, + { + "ptr": "CT/internal/backupwindow/backupwindow.go:18", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Off-site push leg at W+105 min (with clean-up of old snapshots)", + "q": "2" + }, + { + "ptr": "CT/internal/backup/offbox_window.go:47", + "file": "OK", + "ident": "IDENT-MISS(CT/internal/backupwindow/backupwindow.go:18.)", + "ending": "3. Off-site push leg at W+105 min (with clean-up of old snapshots)", + "q": "2" + }, + { + "ptr": "CT/internal/backup/offbox.go:39", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Off-site push leg at W+105 min (with clean-up of old snapshots)", + "q": "3" + }, + { + "ptr": "CT/internal/backup/offbox.go:1564", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Off-site push leg at W+105 min (with clean-up of old snapshots)", + "q": "3" + }, + { + "ptr": "CT/internal/backup/offbox.go:37", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Off-site push leg at W+105 min (with clean-up of old snapshots)", + "q": "3" + }, + { + "ptr": "AR/09-update-architecture.md:2751-2757", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Off-site push leg at W+105 min (with clean-up of old snapshots)", + "q": "3" + }, + { + "ptr": "CT/internal/backup/offbox.go:1040-1053", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Off-site push leg at W+105 min (with clean-up of old snapshots)", + "q": "4" + }, + { + "ptr": "CT/internal/backup/offbox.go:1054-1057", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Off-site push leg at W+105 min (with clean-up of old snapshots)", + "q": "5" + }, + { + "ptr": "CT/internal/backup/offbox_window.go:42", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Off-site push leg at W+105 min (with clean-up of old snapshots)", + "q": "6" + }, + { + "ptr": "CT/cmd/controller/main.go:1446-1452", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Off-site push leg at W+105 min (with clean-up of old snapshots)", + "q": "7" + }, + { + "ptr": "CT/cmd/controller/main.go:1524-1528", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Off-site push leg at W+105 min (with clean-up of old snapshots)", + "q": "7" + }, + { + "ptr": "CT/internal/backup/offbox.go:1040-1057", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Off-site push leg at W+105 min (with clean-up of old snapshots)", + "q": "8" + }, + { + "ptr": "AR/07-backup-architecture.md:455-457", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Off-site push leg at W+105 min (with clean-up of old snapshots)", + "q": "8" + }, + { + "ptr": "CT/internal/nightchain/nightchain.go:281-299", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Missed-night catch-up and resume watch", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/main.go:1676", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Missed-night catch-up and resume watch", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/main.go:1677", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Missed-night catch-up and resume watch", + "q": "1" + }, + { + "ptr": "CT/internal/nightchain/nightchain.go:62", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Missed-night catch-up and resume watch", + "q": "2" + }, + { + "ptr": "CT/internal/nightchain/nightchain.go:64", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Missed-night catch-up and resume watch", + "q": "2" + }, + { + "ptr": "AR/07-backup-architecture.md:574-577", + "file": "OK", + "ident": "IDENT-MISS(leaveToNormal)", + "ending": "4. Missed-night catch-up and resume watch", + "q": "2" + }, + { + "ptr": "CT/internal/nightchain/nightchain.go:66", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Missed-night catch-up and resume watch", + "q": "3" + }, + { + "ptr": "CT/internal/nightchain/nightchain.go:319", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Missed-night catch-up and resume watch", + "q": "3" + }, + { + "ptr": "AR/07-backup-architecture.md:579-584", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Missed-night catch-up and resume watch", + "q": "4" + }, + { + "ptr": "CT/internal/nightchain/nightchain.go:288-292", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Missed-night catch-up and resume watch", + "q": "5" + }, + { + "ptr": "CT/cmd/controller/main.go:1321-1335", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Missed-night catch-up and resume watch", + "q": "5" + }, + { + "ptr": "CT/cmd/controller/main.go:1660-1664", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Missed-night catch-up and resume watch", + "q": "5" + }, + { + "ptr": "CT/internal/nightchain/nightchain.go:375", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Missed-night catch-up and resume watch", + "q": "6" + }, + { + "ptr": "CT/cmd/controller/main.go:1681", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Missed-night catch-up and resume watch", + "q": "6" + }, + { + "ptr": "AR/07-backup-architecture.md:551-559", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Missed-night catch-up and resume watch", + "q": "6" + }, + { + "ptr": "AR/07-backup-architecture.md:562-564", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Missed-night catch-up and resume watch", + "q": "7" + }, + { + "ptr": "CT/internal/nightchain/nightchain.go:357-359", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Missed-night catch-up and resume watch", + "q": "7" + }, + { + "ptr": "CT/internal/nightchain/nightchain.go:319", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Missed-night catch-up and resume watch", + "q": "7" + }, + { + "ptr": "CT/internal/nightchain/nightchain_test.go:126", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Missed-night catch-up and resume watch", + "q": "8" + }, + { + "ptr": "CT/internal/nightchain/nightchain_test.go:217", + "file": "OK", + "ident": "IDENT-MISS(CT/internal/nightchain/nightchain_test.go:126;)", + "ending": "4. Missed-night catch-up and resume watch", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/r871_catchup_wiring_test.go:16", + "file": "OK", + "ident": "IDENT-MISS(CT/internal/nightchain/nightchain_test.go:217;)", + "ending": "4. Missed-night catch-up and resume watch", + "q": "8" + }, + { + "ptr": "CT/internal/nightchain/banner.go:56-88", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Missed-backup banner (26 hours)", + "q": "1" + }, + { + "ptr": "CT/internal/nightchain/banner.go:23", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Missed-backup banner (26 hours)", + "q": "2" + }, + { + "ptr": "CT/internal/nightchain/banner.go:26", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Missed-backup banner (26 hours)", + "q": "2" + }, + { + "ptr": "CT/internal/nightchain/banner.go:95-113", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Missed-backup banner (26 hours)", + "q": "2" + }, + { + "ptr": "CT/internal/nightchain/banner.go:56-88", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Missed-backup banner (26 hours)", + "q": "3" + }, + { + "ptr": "AR/07-backup-architecture.md:579-584", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Missed-backup banner (26 hours)", + "q": "4" + }, + { + "ptr": "AR/07-backup-architecture.md:566-572", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Missed-backup banner (26 hours)", + "q": "5" + }, + { + "ptr": "CT/internal/nightchain/banner.go:56-88", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Missed-backup banner (26 hours)", + "q": "6" + }, + { + "ptr": "AR/07-backup-architecture.md:566-572", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Missed-backup banner (26 hours)", + "q": "7" + }, + { + "ptr": "CT/internal/nightchain/banner_test.go:29", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Missed-backup banner (26 hours)", + "q": "8" + }, + { + "ptr": "CT/internal/nightchain/banner_test.go:59", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Missed-backup banner (26 hours)", + "q": "8" + }, + { + "ptr": "CT/internal/quiesce/quiesce.go:191-206", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/main.go:4008", + "file": "OK", + "ident": "IDENT-MISS(startQuiesceLoop)", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "1" + }, + { + "ptr": "CT/internal/quiesce/quiesce.go:149-151", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "2" + }, + { + "ptr": "CT/internal/quiesce/quiesce.go:161-163", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "2" + }, + { + "ptr": "CT/internal/quiesce/quiesce.go:760", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "2" + }, + { + "ptr": "CT/internal/quiesce/quiesce.go:761", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "2" + }, + { + "ptr": "CT/internal/quiesce/quiesce.go:747-757", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/main.go:3990", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "3" + }, + { + "ptr": "CT/internal/config/config.go:132", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "3" + }, + { + "ptr": "CT/internal/quiesce/quiesce.go:430", + "file": "OK", + "ident": "IDENT-MISS(max_quiesce_duration)", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "3" + }, + { + "ptr": "CT/internal/quiesce/quiesce.go:801-818", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "4" + }, + { + "ptr": "CT/internal/quiesce/quiesce.go:177-188", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "4" + }, + { + "ptr": "CT/cmd/controller/main.go:4007", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "4" + }, + { + "ptr": "CT/internal/quiesce/quiesce.go:213-216", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "5" + }, + { + "ptr": "CT/internal/quiesce/quiesce.go:250-253", + "file": "OK", + "ident": "IDENT-MISS(TryLock)", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "5" + }, + { + "ptr": "CT/internal/quiesce/quiesce.go:271-274", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "5" + }, + { + "ptr": "CT/internal/quiesce/quiesce.go:935", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "5" + }, + { + "ptr": "CT/internal/quiesce/quiesce.go:801-818", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "6" + }, + { + "ptr": "CT/internal/quiesce/quiesce.go:191-206", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "6" + }, + { + "ptr": "CT/internal/quiesce/quiesce.go:283-298", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "7" + }, + { + "ptr": "CT/internal/quiesce/breaker.go:47", + "file": "OK", + "ident": "IDENT-MISS(noteTierFailure)", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "7" + }, + { + "ptr": "CT/internal/quiesce/breaker.go:54", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "7" + }, + { + "ptr": "CT/internal/quiesce/contention.go:53", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "7" + }, + { + "ptr": "CT/internal/quiesce/contention.go:64", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "7" + }, + { + "ptr": "CT/internal/quiesce/quiesce_window_test.go:23", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "8" + }, + { + "ptr": "CT/internal/quiesce/quiesce_window_test.go:94", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "8" + }, + { + "ptr": "CT/internal/quiesce/breaker_test.go:218", + "file": "OK", + "ident": "IDENT-MISS(CT/internal/quiesce/quiesce_window_test.go:94;)", + "ending": "6. Whole-guest backup driver (stop apps, ask the host agent, start apps)", + "q": "8" + }, + { + "ptr": "CT/internal/stacks/unattended.go:230", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/main.go:1641-1643", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "1" + }, + { + "ptr": "AR/07-backup-architecture.md:459-465", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "2" + }, + { + "ptr": "AR/09-update-architecture.md:2806-2817", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "2" + }, + { + "ptr": "CT/internal/stacks/unattended.go:212-216", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "3" + }, + { + "ptr": "CT/internal/backupwindow/backupwindow.go:27", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "3" + }, + { + "ptr": "CT/internal/stacks/unattended.go:374-375", + "file": "OK", + "ident": "IDENT-MISS(UpdateLegStopOffsetMin)", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "3" + }, + { + "ptr": "CT/cmd/controller/main.go:1660-1664", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "4" + }, + { + "ptr": "CT/internal/stacks/unattended.go:374-375", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "4" + }, + { + "ptr": "AR/09-update-architecture.md:2815-2817", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "4" + }, + { + "ptr": "CT/internal/stacks/unattended.go:167-171", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "5" + }, + { + "ptr": "AR/09-update-architecture.md:2882-2883", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "5" + }, + { + "ptr": "CT/internal/quiesce/quiesce.go:271-274", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "5" + }, + { + "ptr": "CT/internal/stacks/unattended.go:212-216", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "6" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:17-27", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "6" + }, + { + "ptr": "AR/07-backup-architecture.md:551-559", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "6" + }, + { + "ptr": "CT/internal/stacks/unattended.go:374-375", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "7" + }, + { + "ptr": "CT/cmd/controller/update_leg_wiring_test.go:19", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "8" + }, + { + "ptr": "CT/internal/quiesce/d20_update_leg_test.go:78", + "file": "OK", + "ident": "IDENT-MISS(CT/cmd/controller/update_leg_wiring_test.go:19;)", + "ending": "7. Automatic app-update leg (after the off-site push)", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:1724", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Off-site integrity check at 06:00", + "q": "1" + }, + { + "ptr": "CT/internal/backup/offbox_integrity.go:81", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Off-site integrity check at 06:00", + "q": "2" + }, + { + "ptr": "CT/internal/backup/offbox_integrity.go:221", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Off-site integrity check at 06:00", + "q": "2" + }, + { + "ptr": "CT/internal/backup/offbox_integrity.go:63", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Off-site integrity check at 06:00", + "q": "3" + }, + { + "ptr": "CT/internal/backup/offbox_integrity.go:78", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Off-site integrity check at 06:00", + "q": "3" + }, + { + "ptr": "CT/internal/backup/offbox_integrity.go:212-220", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Off-site integrity check at 06:00", + "q": "4" + }, + { + "ptr": "CT/cmd/controller/main.go:1665-1668", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Off-site integrity check at 06:00", + "q": "4" + }, + { + "ptr": "CT/internal/backup/offbox_integrity.go:298", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Off-site integrity check at 06:00", + "q": "5" + }, + { + "ptr": "CT/cmd/controller/main.go:1710-1723", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Off-site integrity check at 06:00", + "q": "5" + }, + { + "ptr": "CT/internal/backup/offbox_integrity.go:231", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Off-site integrity check at 06:00", + "q": "6" + }, + { + "ptr": "CT/internal/backup/offbox_integrity.go:223-229", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Off-site integrity check at 06:00", + "q": "6" + }, + { + "ptr": "CT/cmd/controller/main.go:1724-1729", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Off-site integrity check at 06:00", + "q": "7" + }, + { + "ptr": "CT/internal/backup/offbox_integrity.go:212-232", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Off-site integrity check at 06:00", + "q": "8" + }, + { + "ptr": "CT/internal/nightchain/r1189d_test.go:13", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Off-site integrity check at 06:00", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:1746", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Off-site proof restore at 05:30", + "q": "1" + }, + { + "ptr": "CT/internal/backup/offbox_proof.go:140", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Off-site proof restore at 05:30", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/main.go:1746", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Off-site proof restore at 05:30", + "q": "2" + }, + { + "ptr": "CT/internal/backup/offbox_proof.go:265", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Off-site proof restore at 05:30", + "q": "2" + }, + { + "ptr": "CT/internal/backup/offbox_proof.go:80", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Off-site proof restore at 05:30", + "q": "3" + }, + { + "ptr": "CT/cmd/controller/main.go:1660-1664", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Off-site proof restore at 05:30", + "q": "4" + }, + { + "ptr": "CT/cmd/controller/main.go:1665-1668", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Off-site proof restore at 05:30", + "q": "4" + }, + { + "ptr": "CT/internal/backup/offbox_proof.go:149", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Off-site proof restore at 05:30", + "q": "5" + }, + { + "ptr": "AR/09-update-architecture.md:2751-2757", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Off-site proof restore at 05:30", + "q": "5" + }, + { + "ptr": "CT/cmd/controller/main.go:1746-1749", + "file": "OK", + "ident": "IDENT-MISS(Notify)", + "ending": "9. Off-site proof restore at 05:30", + "q": "7" + }, + { + "ptr": "CT/cmd/controller/main.go:0", + "file": "OK", + "ident": "IDENT-MISS(Notify)", + "ending": "9. Off-site proof restore at 05:30", + "q": "7" + }, + { + "ptr": "CT/cmd/controller/main.go:1660-1664", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Off-site proof restore at 05:30", + "q": "8" + }, + { + "ptr": "CT/internal/nightchain/r1189d_test.go:51", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Off-site proof restore at 05:30", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:1835", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Hub report every 15 minutes (plus push on change and hanging wait)", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/main.go:1233-1290", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Hub report every 15 minutes (plus push on change and hanging wait)", + "q": "1" + }, + { + "ptr": "CT/internal/config/config.go:416", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Hub report every 15 minutes (plus push on change and hanging wait)", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/main.go:1831-1834", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Hub report every 15 minutes (plus push on change and hanging wait)", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/main.go:1932-1936", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Hub report every 15 minutes (plus push on change and hanging wait)", + "q": "2" + }, + { + "ptr": "CT/internal/report/trigger.go:30", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Hub report every 15 minutes (plus push on change and hanging wait)", + "q": "2" + }, + { + "ptr": "CT/internal/report/pusher.go:85", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Hub report every 15 minutes (plus push on change and hanging wait)", + "q": "3" + }, + { + "ptr": "CT/internal/report/pusher.go:114-117", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Hub report every 15 minutes (plus push on change and hanging wait)", + "q": "3" + }, + { + "ptr": "CT/internal/report/waiter.go:35", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Hub report every 15 minutes (plus push on change and hanging wait)", + "q": "3" + }, + { + "ptr": "CT/internal/report/trigger.go:98", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Hub report every 15 minutes (plus push on change and hanging wait)", + "q": "4" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:354-364", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Hub report every 15 minutes (plus push on change and hanging wait)", + "q": "5" + }, + { + "ptr": "CT/internal/report/trigger.go:27", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Hub report every 15 minutes (plus push on change and hanging wait)", + "q": "5" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:281-298", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Hub report every 15 minutes (plus push on change and hanging wait)", + "q": "6" + }, + { + "ptr": "CT/internal/report/waiter.go:36-37", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Hub report every 15 minutes (plus push on change and hanging wait)", + "q": "7" + }, + { + "ptr": "CT/internal/report/pusher.go:109-111", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Hub report every 15 minutes (plus push on change and hanging wait)", + "q": "7" + }, + { + "ptr": "CT/internal/report/trigger_test.go:114", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Hub report every 15 minutes (plus push on change and hanging wait)", + "q": "8" + }, + { + "ptr": "CT/internal/report/waiter_test.go:60", + "file": "OK", + "ident": "IDENT-MISS(CT/internal/report/trigger_test.go:114;)", + "ending": "10. Hub report every 15 minutes (plus push on change and hanging wait)", + "q": "8" + }, + { + "ptr": "CT/internal/selfupdate/updater.go:153", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Controller self-update: check, floor, nightly auto", + "q": "1" + }, + { + "ptr": "CT/internal/selfupdate/updater.go:612", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Controller self-update: check, floor, nightly auto", + "q": "1" + }, + { + "ptr": "CT/internal/config/config.go:417", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Controller self-update: check, floor, nightly auto", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/main.go:1981-1989", + "file": "OK", + "ident": "IDENT-MISS(CheckInterval)", + "ending": "11. Controller self-update: check, floor, nightly auto", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/main.go:1249-1252", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Controller self-update: check, floor, nightly auto", + "q": "2" + }, + { + "ptr": "CT/internal/config/config.go:419", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Controller self-update: check, floor, nightly auto", + "q": "2" + }, + { + "ptr": "CT/internal/config/config.go:278", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Controller self-update: check, floor, nightly auto", + "q": "2" + }, + { + "ptr": "CT/internal/selfupdate/updater.go:245", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Controller self-update: check, floor, nightly auto", + "q": "3" + }, + { + "ptr": "CT/internal/selfupdate/updater.go:656-661", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Controller self-update: check, floor, nightly auto", + "q": "4" + }, + { + "ptr": "CT/internal/selfupdate/updater.go:604-606", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Controller self-update: check, floor, nightly auto", + "q": "4" + }, + { + "ptr": "CT/internal/selfupdate/updater.go:520-524", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Controller self-update: check, floor, nightly auto", + "q": "5" + }, + { + "ptr": "CT/internal/selfupdate/updater.go:533-536", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Controller self-update: check, floor, nightly auto", + "q": "5" + }, + { + "ptr": "CT/internal/selfupdate/updater.go:540-544", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Controller self-update: check, floor, nightly auto", + "q": "5" + }, + { + "ptr": "CT/internal/config/config.go:278", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Controller self-update: check, floor, nightly auto", + "q": "6" + }, + { + "ptr": "CT/cmd/controller/main.go:1879-1881", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Controller self-update: check, floor, nightly auto", + "q": "7" + }, + { + "ptr": "CT/internal/selfupdate/updater.go:639", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Controller self-update: check, floor, nightly auto", + "q": "7" + }, + { + "ptr": "CT/internal/selfupdate/floor_test.go:93", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Controller self-update: check, floor, nightly auto", + "q": "8" + }, + { + "ptr": "CT/internal/selfupdate/floor_test.go:11", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Controller self-update: check, floor, nightly auto", + "q": "8" + }, + { + "ptr": "CT/internal/sync/sync.go:125", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Catalog sync (app templates from git)", + "q": "1" + }, + { + "ptr": "CT/internal/sync/sync.go:140-143", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Catalog sync (app templates from git)", + "q": "2" + }, + { + "ptr": "CT/internal/config/config.go:399", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Catalog sync (app templates from git)", + "q": "2" + }, + { + "ptr": "CT/internal/sync/sync.go:131-135", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Catalog sync (app templates from git)", + "q": "2" + }, + { + "ptr": "CT/internal/sync/sync.go:124-188", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Catalog sync (app templates from git)", + "q": "3" + }, + { + "ptr": "CT/internal/sync/sync.go:126-129", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Catalog sync (app templates from git)", + "q": "4" + }, + { + "ptr": "CT/internal/sync/sync.go:175-185", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Catalog sync (app templates from git)", + "q": "5" + }, + { + "ptr": "CT/internal/sync/sync.go:153-155", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Catalog sync (app templates from git)", + "q": "5" + }, + { + "ptr": "CT/internal/sync/sync.go:146", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Catalog sync (app templates from git)", + "q": "6" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:354-364", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Catalog sync (app templates from git)", + "q": "6" + }, + { + "ptr": "CT/internal/sync/sync.go:155", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Catalog sync (app templates from git)", + "q": "7" + }, + { + "ptr": "CT/internal/sync/sync.go:124-159", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Catalog sync (app templates from git)", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:981-1006", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. System health check and base-stack self-heal", + "q": "1" + }, + { + "ptr": "CT/internal/config/config.go:409", + "file": "OK", + "ident": "IDENT-OK", + "ending": "13. System health check and base-stack self-heal", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/main.go:977-980", + "file": "OK", + "ident": "IDENT-OK", + "ending": "13. System health check and base-stack self-heal", + "q": "2" + }, + { + "ptr": "CT/internal/config/config.go:453", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. System health check and base-stack self-heal", + "q": "2" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:281-298", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. System health check and base-stack self-heal", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/main.go:981-1006", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. System health check and base-stack self-heal", + "q": "3" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:354-364", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. System health check and base-stack self-heal", + "q": "5" + }, + { + "ptr": "CT/cmd/controller/main.go:1004", + "file": "OK", + "ident": "IDENT-OK", + "ending": "13. System health check and base-stack self-heal", + "q": "7" + }, + { + "ptr": "CT/cmd/controller/main.go:981-1006", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. System health check and base-stack self-heal", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:1086-1092", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. Dead-app scan every 30 seconds", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/main.go:1087-1088", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. Dead-app scan every 30 seconds", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/main.go:1086-1143", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. Dead-app scan every 30 seconds", + "q": "3" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:354-364", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. Dead-app scan every 30 seconds", + "q": "5" + }, + { + "ptr": "CT/cmd/controller/main.go:1090", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. Dead-app scan every 30 seconds", + "q": "5" + }, + { + "ptr": "CT/cmd/controller/main.go:1087", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. Dead-app scan every 30 seconds", + "q": "6" + }, + { + "ptr": "CT/cmd/controller/main.go:1091", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. Dead-app scan every 30 seconds", + "q": "7" + }, + { + "ptr": "CT/cmd/controller/main.go:1086-1092", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. Dead-app scan every 30 seconds", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:1804-1805", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Disk-fill watch", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/main.go:2690", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Disk-fill watch", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/main.go:1804", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Disk-fill watch", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/main.go:2685", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Disk-fill watch", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/main.go:1772", + "file": "OK", + "ident": "IDENT-MISS(fillWatchStartupDelay)", + "ending": "15. Disk-fill watch", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/main.go:1804-1805", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. Disk-fill watch", + "q": "3" + }, + { + "ptr": "CT/cmd/controller/main.go:1804-1805", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. Disk-fill watch", + "q": "5" + }, + { + "ptr": "CT/cmd/controller/r363_fillwatch_interval_test.go:17", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Disk-fill watch", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:2226", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Drive health check, hourly", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/main.go:2219-2224", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Drive health check, hourly", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/main.go:2225", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Drive health check, hourly", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/main.go:2219-2224", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Drive health check, hourly", + "q": "3" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:281-298", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Drive health check, hourly", + "q": "4" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:354-364", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Drive health check, hourly", + "q": "5" + }, + { + "ptr": "CT/cmd/controller/main.go:2219-2226", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Drive health check, hourly", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:2289", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Agent channel health, every 60 seconds", + "q": "1" + }, + { + "ptr": "CT/internal/channelhealth/checker.go:29", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. Agent channel health, every 60 seconds", + "q": "2" + }, + { + "ptr": "CT/internal/channelhealth/checker.go:32", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. Agent channel health, every 60 seconds", + "q": "3" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:354-364", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Agent channel health, every 60 seconds", + "q": "5" + }, + { + "ptr": "CT/cmd/controller/main.go:2288", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Agent channel health, every 60 seconds", + "q": "5" + }, + { + "ptr": "CT/cmd/controller/main.go:2286", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Agent channel health, every 60 seconds", + "q": "7" + }, + { + "ptr": "CT/internal/channelhealth/checker_test.go:119", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. Agent channel health, every 60 seconds", + "q": "8" + }, + { + "ptr": "CT/internal/holdreport/holdreport.go:89", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Held-app report to the hub", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/main.go:1081-1084", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Held-app report to the hub", + "q": "1" + }, + { + "ptr": "CT/internal/holdreport/holdreport.go:36", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Held-app report to the hub", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/main.go:1071-1073", + "file": "OK", + "ident": "IDENT-MISS(ResendEvery)", + "ending": "18. Held-app report to the hub", + "q": "2" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:354-364", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Held-app report to the hub", + "q": "5" + }, + { + "ptr": "CT/cmd/controller/main.go:1082", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Held-app report to the hub", + "q": "6" + }, + { + "ptr": "CT/cmd/controller/main.go:1074", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Held-app report to the hub", + "q": "7" + }, + { + "ptr": "CT/cmd/controller/holdclear_test.go:333", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Held-app report to the hub", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:1693", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. Abandoned off-site store sweep at 05:10", + "q": "1" + }, + { + "ptr": "CT/internal/backup/offbox_abandon.go:214", + "file": "OK", + "ident": "IDENT-OK", + "ending": "19. Abandoned off-site store sweep at 05:10", + "q": "1" + }, + { + "ptr": "CT/internal/backup/offbox_abandon.go:366", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. Abandoned off-site store sweep at 05:10", + "q": "2" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:17-27", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. Abandoned off-site store sweep at 05:10", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/main.go:1660-1664", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. Abandoned off-site store sweep at 05:10", + "q": "4" + }, + { + "ptr": "CT/cmd/controller/main.go:1693", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. Abandoned off-site store sweep at 05:10", + "q": "8" + }, + { + "ptr": "CT/internal/backup/restore_copy_sweep.go:36", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Restore-copy sweep and conversion-copy release", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/main.go:1414", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Restore-copy sweep and conversion-copy release", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/main.go:1407", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Restore-copy sweep and conversion-copy release", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/main.go:1412", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Restore-copy sweep and conversion-copy release", + "q": "2" + }, + { + "ptr": "CT/internal/backup/restore_copy_sweep.go:29", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Restore-copy sweep and conversion-copy release", + "q": "2" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:281-298", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Restore-copy sweep and conversion-copy release", + "q": "4" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:354-364", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Restore-copy sweep and conversion-copy release", + "q": "5" + }, + { + "ptr": "CT/internal/backup/restore_copy_sweep.go:36", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Restore-copy sweep and conversion-copy release", + "q": "6" + }, + { + "ptr": "CT/internal/backup/restore_copy_sweep.go:29", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Restore-copy sweep and conversion-copy release", + "q": "8" + }, + { + "ptr": "CT/internal/web/auth.go:369", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. Session clean-up (dashboard sign-ins)", + "q": "1" + }, + { + "ptr": "CT/internal/web/auth.go:370", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Session clean-up (dashboard sign-ins)", + "q": "2" + }, + { + "ptr": "CT/internal/web/auth.go:30", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. Session clean-up (dashboard sign-ins)", + "q": "2" + }, + { + "ptr": "CT/internal/family/family.go:42", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. Session clean-up (dashboard sign-ins)", + "q": "2" + }, + { + "ptr": "CT/internal/web/share.go:31", + "file": "OK", + "ident": "IDENT-MISS(SessionLife)", + "ending": "21. Session clean-up (dashboard sign-ins)", + "q": "2" + }, + { + "ptr": "CT/internal/web/auth.go:355-395", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Session clean-up (dashboard sign-ins)", + "q": "4" + }, + { + "ptr": "CT/internal/web/auth.go:377", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. Session clean-up (dashboard sign-ins)", + "q": "5" + }, + { + "ptr": "CT/internal/web/auth.go:381", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. Session clean-up (dashboard sign-ins)", + "q": "6" + }, + { + "ptr": "CT/internal/web/auth.go:392", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Session clean-up (dashboard sign-ins)", + "q": "7" + }, + { + "ptr": "CT/internal/web/auth.go:369-393", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Session clean-up (dashboard sign-ins)", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/upwatch_wiring.go:136", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Upload and CPU watch, torrent limit rule, line-test hold", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/upwatch_wiring.go:139", + "file": "OK", + "ident": "NO-IDENT", + "ending": "22. Upload and CPU watch, torrent limit rule, line-test hold", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/upwatch_wiring.go:141", + "file": "OK", + "ident": "NO-IDENT", + "ending": "22. Upload and CPU watch, torrent limit rule, line-test hold", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/upwatch_wiring.go:124-127", + "file": "OK", + "ident": "NO-IDENT", + "ending": "22. Upload and CPU watch, torrent limit rule, line-test hold", + "q": "2" + }, + { + "ptr": "CT/internal/upwatch/linehold.go:79", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Upload and CPU watch, torrent limit rule, line-test hold", + "q": "3" + }, + { + "ptr": "CT/internal/upwatch/linehold.go:216", + "file": "OK", + "ident": "IDENT-MISS(holdTimeout)", + "ending": "22. Upload and CPU watch, torrent limit rule, line-test hold", + "q": "3" + }, + { + "ptr": "CT/cmd/controller/upwatch_wiring.go:137", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Upload and CPU watch, torrent limit rule, line-test hold", + "q": "4" + }, + { + "ptr": "CT/internal/upwatch/watch.go:151", + "file": "OK", + "ident": "NO-IDENT", + "ending": "22. Upload and CPU watch, torrent limit rule, line-test hold", + "q": "6" + }, + { + "ptr": "CT/internal/upwatch/watch.go:382", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Upload and CPU watch, torrent limit rule, line-test hold", + "q": "7" + }, + { + "ptr": "CT/cmd/controller/upwatch_wiring.go:134-152", + "file": "OK", + "ident": "NO-IDENT", + "ending": "22. Upload and CPU watch, torrent limit rule, line-test hold", + "q": "8" + }, + { + "ptr": "CT/internal/stacks/image_retention.go:399", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Old image clean-up after start", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/main.go:2048-2069", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Old image clean-up after start", + "q": "1" + }, + { + "ptr": "CT/cmd/controller/main.go:2056", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Old image clean-up after start", + "q": "2" + }, + { + "ptr": "CT/cmd/controller/main.go:2056", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Old image clean-up after start", + "q": "3" + }, + { + "ptr": "CT/internal/stacks/image_retention_test.go:238", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Old image clean-up after start", + "q": "5" + }, + { + "ptr": "CT/internal/stacks/image_retention.go:221", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Old image clean-up after start", + "q": "7" + }, + { + "ptr": "CT/internal/stacks/image_retention_test.go:111", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Old image clean-up after start", + "q": "8" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:131-157", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "1" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:105-127", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "1" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:17-27", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "2" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:281-298", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "2" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:264", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "3" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:438-447", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "3" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:344-347", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "4" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:354-364", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "5" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:499-519", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "6" + }, + { + "ptr": "CT/internal/scheduler/scheduler.go:344-347", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "7" + }, + { + "ptr": "CT/internal/scheduler/scheduler_test.go:87", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/internal/scheduler/scheduler_test.go:62", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:950", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:953", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:960-962", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:971", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:1064-1065", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:1148-1150", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:1375", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:1398-1402", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/internal/metrics/collector.go:53", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:1754", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/internal/config/config.go:425", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:2102", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:2045", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/internal/busymark/busymark.go:29", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:1932-1936", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/internal/backup/tier2_copy_lifecycle.go:172", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/cmd/controller/main.go:2194", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/internal/web/family_gate.go:56", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "AR/07-backup-architecture.md:455-457", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/internal/config/config.go:291-292", + "file": "OK", + "ident": "IDENT-MISS(rotat|lumberjack|max-size|MaxSizeMB)", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/internal/config/config.go:422-423", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/internal/web/share.go:31", + "file": "OK", + "ident": "IDENT-MISS(in)", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/internal/bootrecon/bootrecon.go:147", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/internal/bootstrap/waiting.go:95", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/internal/offsiteapply/offsiteapply.go:137", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/internal/web/operator_reboot_notice.go:78", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/internal/web/handler_export_upload.go:204", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/internal/web/media_events.go:97", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + }, + { + "ptr": "CT/internal/notify/notifier.go:324", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)", + "q": "8" + } + ] +} \ No newline at end of file diff --git a/documentation/audits/clock-list-2026-11-11/evidence/helpers/resolved-clock-hub.json b/documentation/audits/clock-list-2026-11-11/evidence/helpers/resolved-clock-hub.json new file mode 100644 index 00000000..c7b1dce9 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/helpers/resolved-clock-hub.json @@ -0,0 +1,3684 @@ +{ + "endings": [ + { + "title": "1. Box liveness: \"your server cannot be reached\"", + "answers": { + "1": "CUSTOMER. The hub's 60 s sweep runs `stalenessChecker.Check` and `hostStaleness.Check` `E/hub/cmd/hub/main.go:935-936`; they raise `host_stale`, `host_down`, `host_recovered` `HUB/monitor/host_staleness.go:207-221`. Owner: hub pod. Merged here: the controller-report checker and the host-report checker.", + "2": "Every 60 s `E/hub/cmd/hub/main.go:927`. Stale after `stale_threshold`, default `\"30m\"` `E/hub/cmd/hub/main.go:1027`, deployed 45 m `E/manifests/hub.yaml:88`; down after 2x `HUB/monitor/host_staleness.go:39`. No time zone: ages are durations `HUB/monitor/host_staleness.go:131`.", + "3": "No time limit; the checkers run one after another in one goroutine `E/hub/cmd/hub/main.go:935-962`. One store read of all hosts per pass `HUB/monitor/host_staleness.go:113`, then a loop per host; grows linearly with boxes. Measured duration: NOT FOUND (grep `time.Since(start)` in `E/hub/cmd/hub/main.go:927-965`).", + "4": "Pod down: nothing is judged. At start a box already silent is left unseeded and the first pass alarms once `HUB/monitor/host_staleness.go:59-60`, unless that alarm was already raised `HUB/monitor/host_staleness.go:62-68`. The controller checker seeds stale/down silently `HUB/monitor/staleness.go:86-91` (candidate: differs from the host checker).", + "5": "One pod, `replicas: 1` `E/manifests/hub.yaml:122`, `Recreate` `E/manifests/hub.yaml:123-124`; state under a mutex `HUB/monitor/host_staleness.go:119-120`. Events fire on transitions only `HUB/monitor/host_staleness.go:170-172`. A household power-off holds alarms `HUB/monitor/host_staleness.go:134-136` for `DeliberateOffHold` `HUB/store/deliberateoff.go:18`.", + "6": "Judged on the hub's receive time, not the box clock: `last_report_at = datetime('now')` `HUB/store/store.go:3916`. A wrong box clock cannot hide silence. A hub clock jump moves every age at once `HUB/monitor/host_staleness.go:131`; DST has no effect (durations).", + "7": "Operator: every edge; liveness events bypass the 1 h quiet hour, 5 min dedupe `HUB/notify/dispatcher.go:609-612`. Household: at most once per 7 days, `householdLivenessQuiet` `HUB/notify/dispatcher.go:1018`. If the hub itself is down nobody is told by this job.", + "8": "TEST `TestHostStaleness_R1001_RestartDoesNotReMail` `HUB/monitor/r1001_restart_test.go:56`; TEST `TestR873_HouseholdHearsAnOutageAtMostWeekly` `HUB/notify/r873_liveness_weekly_test.go:17`; DOC `AR/05-hub-architecture.md:88-118`." + }, + "extra": [] + }, + { + "title": "2. Missed-backup deadline check (05:00)", + "answers": { + "1": "CUSTOMER. `scheduleDaily(ctx, \"deadline-check\", \"05:00\", ...)` `E/hub/cmd/hub/main.go:986-988` runs `CheckBackupDeadlines` `HUB/monitor/deadline.go:409`; it raises `expected_backup_missed` `HUB/monitor/deadline.go:539-544`. Owner: hub pod.", + "2": "Once a day at 05:00 Budapest wall-clock; the slot is a literal in `E/hub/cmd/hub/main.go:986`, no setting. Zone chosen in `scheduleDaily` `E/hub/cmd/hub/main.go:1072`. Limits: `backupStaleAfter` 26 h `HUB/monitor/deadline.go:29`, off-site `offsiteBackupStaleAfter` 8 d `HUB/monitor/deadline_tiers.go:28`.", + "3": "No time limit; runs inline in the scheduler goroutine `E/hub/cmd/hub/main.go:1090`. Loops over every customer and reads up to 7 days of evidence, `backupEvidenceLookback` `HUB/monitor/deadline.go:36`; grows with boxes times reports. Only a summary line is logged `HUB/monitor/deadline.go:569-570`; duration NOT FOUND (grep `duration|took` in that file).", + "4": "Pod down at 05:00: the day's check is skipped, no catch-up; `nextDailyRun` moves to the next calendar day `E/hub/cmd/hub/main.go:1095-1112` and `last` lives in memory `E/hub/cmd/hub/main.go:1069-1093`. Nobody notices a skipped check (candidate).", + "5": "Cannot run twice for one slot: the `last` guard in `nextDailyRun` `E/hub/cmd/hub/main.go:1095-1112`. No lock against the 60 s sweep; they are separate goroutines `E/hub/cmd/hub/main.go:927` and `E/hub/cmd/hub/main.go:986`. A down box is judged on longer lines, 48 h / 72 h `HUB/monitor/deadline.go:340-343`.", + "6": "Freshness is the box's own backup timestamp against the hub's now `HUB/monitor/deadline.go:243`, parsed by `parseBackupTime` `HUB/monitor/deadline.go:290-299`. A box whose clock runs ahead would look fresh (candidate; no guard found, grep `future|skew` in `HUB/monitor/deadline.go:1` whole file). \"Since midnight\" is Budapest midnight `HUB/monitor/deadline.go:417-419`.", + "7": "Household and operator through the dispatcher on `expected_backup_missed` `HUB/monitor/deadline.go:539-544`; at the 05:00 run only. A box that never bound is \"unknown\", not alarmed `HUB/monitor/deadline.go:486-494`; disabled / deliberately-off are skipped `HUB/monitor/deadline.go:432-442`.", + "8": "TEST `TestR872_DownEveryNightRaisesTheMissedAlarms` `HUB/monitor/r872_down_box_test.go:57`; TEST `TestR1183_TheSameSlotNeverRunsTwice` `E/hub/cmd/hub/r1183_dst_schedule_test.go:82`. A skipped day: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "3. Mail cooldowns and the weekly outage mail", + "answers": { + "1": "CUSTOMER. Not a job but a clock rule inside every mail: `processCustomer` `HUB/notify/dispatcher.go:1020-1098` and `processOperator` `HUB/notify/dispatcher.go:626-696` decide whether an alarm is mailed now or swallowed. Owner: hub dispatcher.", + "2": "Operator: `operatorCooldown` 1 h, liveness 5 min `HUB/notify/dispatcher.go:609-612`, storage fill `storageFillCooldown` 6 h `HUB/notify/dispatcher.go:573`. Household: default 6 h `HUB/notify/dispatcher.go:1065-1069`; outage mails once per 7 days `HUB/notify/dispatcher.go:1054-1062`. First hour of a new box is suppressed, `firstHourOfABox` `HUB/notify/dispatcher.go:1281`.", + "3": "Per event, no loop. HTTP client limit 10 s `HUB/notify/dispatcher.go:86`. The cooldown maps grow with customers times event types; never pruned: NOT FOUND (grep `delete(d.` in `HUB/notify/dispatcher.go:1` whole file not run to completion; candidate to check).", + "4": "Pod restart: the cooldown maps are in memory, so quiet periods restart from zero; only the 7-day household rule is read from the persisted notification log `HUB/notify/dispatcher.go:1054-1062`. Design text: `AR/08-alarm-ladder.md:435-439`.", + "5": "The operator cooldown is armed before the send `HUB/notify/dispatcher.go:674`, so two alarms close together collapse to one; a suppression is recorded `HUB/notify/dispatcher.go:665-672`. A MAIL-HOLD drops the mail `HUB/notify/dispatcher.go:680-687`.", + "6": "Cooldowns use the hub clock only. A hub clock jump back would stretch a quiet period, forward would end it early; nothing in source handles it: NOT FOUND (grep `monotonic|clock` in `HUB/notify/dispatcher.go:1` whole file).", + "7": "A swallowed operator alarm leaves a \"suppressed\" row `HUB/notify/dispatcher.go:665-672`. A failed household mail is logged and NOT retried `HUB/notify/dispatcher.go:1091-1094` (candidate: only operator mails are retried, see block 4).", + "8": "TEST `TestOperatorCooldown_NodeLivenessBypassesQuietHour` `HUB/notify/node_liveness_cooldown_test.go:17`; TEST `TestR873_HouseholdHearsAnOutageAtMostWeekly` `HUB/notify/r873_liveness_weekly_test.go:17`; DOC `AR/08-alarm-ladder.md:165-168`." + }, + "extra": [] + }, + { + "title": "4. Retry of a failed operator mail", + "answers": { + "1": "OPERATOR. `retryOperatorEmail` `HUB/notify/dispatcher.go:99-104` re-sends an operator mail whose send failed; started by the failure branch of `processOperator` `HUB/notify/dispatcher.go:688-692`.", + "2": "After 1, 5 and 15 minutes: `operatorRetryDelays` `HUB/notify/dispatcher.go:76`, armed with `afterFn = time.AfterFunc` `HUB/notify/dispatcher.go:92`. No setting. Durations, no time zone.", + "3": "Each try is bounded by the 10 s client timeout `HUB/notify/dispatcher.go:86`. One timer per failed mail; grows only with failures.", + "4": "Pod restart inside the 21 minutes: the pending timers are in memory `HUB/notify/dispatcher.go:92` and are lost; no catch-up was found (NOT FOUND: grep `retry` in `E/hub/cmd/hub/main.go:1` whole file). After the third failure: an ERROR log line only `HUB/notify/dispatcher.go:101`.", + "5": "The cooldown for that alarm was armed before the first send `HUB/notify/dispatcher.go:674`, so a repeat alarm in the same hour does not mail either; the retry is the only path. Explained in `AR/08-alarm-ladder.md:358-363`.", + "6": "Timers are Go durations; DST has no effect. A hub wall-clock jump: NOBODY KNOWS (not tested).", + "7": "Nobody is mailed when the retries give up; the trace is the ERROR line `HUB/notify/dispatcher.go:101` and `failed` rows in the notification log `AR/08-alarm-ladder.md:362-363`.", + "8": "TEST `TestOperatorMailRetry_FailedOperatorMailIsRetried` `HUB/notify/operator_mail_retry_test.go:17`. Restart during the retry window: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "5. Kernel \"your box restarts tonight\" mail", + "answers": { + "1": "CUSTOMER. `KernelNotify` `HUB/osupdates/kernel.go:191` mails each due household the day before a kernel step; called from `Evaluate` `HUB/osupdates/service.go:436`, which the 60 s OS ticker calls `E/hub/cmd/hub/main.go:564`.", + "2": "Every 60 s, but acts only 09:00-20:00 Budapest: `KernelNoticeFromHour`/`KernelNoticeToHour` `HUB/osupdates/kernel.go:46-47`, checked at `HUB/osupdates/kernel.go:193-196`. Valid 24 h, gap 20 h, at most 3 mails `HUB/osupdates/kernel.go:48-50`. Zone from `budapestLoc` `HUB/osupdates/kernel.go:73-78`. Constants, no setting.", + "3": "No time limit. Loops all hosts every minute with store reads per host `HUB/osupdates/kernel.go:203-208`; grows linearly with boxes, 1440 passes a day.", + "4": "Pod down all day 09-20: no mail, and without a mail there is no step: `KernelBlockFor` sets `Tonight` only for a notice younger than 24 h `HUB/osupdates/kernel.go:179-185`. It mails on the next daytime tick; the notices are persisted `HUB/osupdates/kernel.go:236`.", + "5": "Twice is blocked by the stored notice and the 20 h gap `HUB/osupdates/kernel.go:208-211`; a restart cannot double-mail because the row is in the store. A mail sent but not recorded means no step tonight `HUB/osupdates/kernel.go:236-239`.", + "6": "Hub clock decides the daytime window. A mail sent at 19:59 is valid until 19:59 next day `HUB/osupdates/kernel.go:182`, so it covers one night. DST: the hour test uses Budapest local time `HUB/osupdates/kernel.go:193-194`; the 20 h / 24 h are durations. A deliberately-off box gets no mail `HUB/osupdates/kernel.go:201`.", + "7": "Mail failure: operator event `os_kernel_notice` warning, once per box, kernel and day `HUB/osupdates/kernel.go:226-233`. After 3 mails with no step: operator warning, once `HUB/osupdates/kernel.go:212-219`.", + "8": "TEST `TestKernel_AtMostThreeMails` `HUB/osupdates/kernel_test.go:192`; TEST `TestKernel_DayBeforeMailMakesTonight` `HUB/osupdates/kernel_test.go:79`; TEST `TestKernel_NoMailNoStep` `HUB/osupdates/kernel_test.go:115`. A mail sent late in the day vs the box's night slot: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "6. OS update approval tick and hourly OS alarms", + "answers": { + "1": "OPERATOR (the outcome reaches households as updates). 60 s ticker `E/hub/cmd/hub/main.go:554-586`: `osSvc.Evaluate()` `E/hub/cmd/hub/main.go:564` approves a set when its rule holds; `osSvc.Alarms()` `E/hub/cmd/hub/main.go:577` raises the fleet alarms.", + "2": "Evaluate every 60 s; Alarms when `time.Since(lastAlarms) >= time.Hour` `E/hub/cmd/hub/main.go:574`. Rule default `ApproveAfter: 24 * time.Hour, NightsRequired: 1` `E/hub/cmd/hub/main.go:476`, env `OS_APPROVE_AFTER` `E/hub/cmd/hub/main.go:484-491`. Alarm limits 7 d / 14 d, env table `E/hub/cmd/hub/main.go:536-541`.", + "3": "No time limit. The alarm pass logs its own duration `E/hub/cmd/hub/main.go:581`. Evaluate walks every layer `HUB/osupdates/service.go:414-422`; grows with boxes times layers, every minute.", + "4": "Pod restart: `lastAlarms` is a local variable `E/hub/cmd/hub/main.go:558`, zero at start, so the first tick runs the alarms at once. Re-mail is prevented by the stored raise time, 7 days `HUB/osupdates/service.go:1102`.", + "5": "One goroutine, so Evaluate and Alarms never overlap `E/hub/cmd/hub/main.go:559-585`. `raise` clears the mark when the condition ends so the next occurrence is announced `HUB/osupdates/service.go:1094-1108`.", + "6": "Ages use the hub clock against stored times `HUB/osupdates/service.go:1102`. A box's claimed reboot time is allowed 1 minute of skew, `rebootClaimSkew` `HUB/osupdates/service.go:970`. Larger box-clock error: NOBODY KNOWS.", + "7": "Operator, by event through the dispatcher: `os_update_stale`, `os_ring0_stalled` and others, listed in `AR/08-alarm-ladder.md:375-383`. Repeats weekly while it holds `HUB/osupdates/service.go:1092-1093`.", + "8": "TEST `TestAlarm_Ring0Stalled` `HUB/osupdates/alarms_test.go:226`; DOC `AR/08-alarm-ladder.md:365`. Cost of the per-minute pass at 100 boxes: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "7. Off-site copy too old, and escrow never finished", + "answers": { + "1": "CUSTOMER. `offsiteChecker.Check` `E/hub/cmd/hub/main.go:943` on the 60 s sweep; `isStale` `HUB/monitor/offsite.go:166` and `checkEscrowPending` `HUB/monitor/offsite_escrow_pending.go:66`. Owner: hub.", + "2": "Every 60 s. Stale after `defaultOffsiteStaleAfter` 48 h `HUB/monitor/offsite.go:54`, counted from the last SUCCESS `HUB/monitor/offsite.go:203-207`. Escrow pending after `escrowPendingAfter` 7 d `HUB/monitor/offsite_escrow_pending.go:37`, re-raised each 7 d `HUB/monitor/offsite_escrow_pending.go:110`.", + "3": "No time limit. Reads every customer's latest report each minute `HUB/monitor/offsite.go:443-459`; grows linearly with boxes.", + "4": "Pod down: not judged. After start the in-memory state maps are empty `HUB/monitor/offsite.go:463-468`; whether a still-stale tier is mailed again after a restart: NOBODY KNOWS (no R-1001 seeding found here; grep `R-1001` in `HUB/monitor/offsite.go:1` whole file returned nothing).", + "5": "One mutex per pass `HUB/monitor/offsite.go:448-449`; a customer listed twice is handled once `HUB/monitor/offsite.go:453-458`. A never-ran tier is owned by the delivery checker, \"one state, one owner\" `HUB/monitor/offsite.go:160-162`.", + "6": "`LastSuccess` is the BOX's timestamp compared with the hub's now `HUB/monitor/offsite.go:203-207`. A box clock ahead hides staleness (candidate); an unparseable time counts as stale `HUB/monitor/offsite.go:204-206`. Snapshot-drop tolerates `windowSlack` 2 h `HUB/monitor/offsite.go:296`.", + "7": "Dispatcher mail on the stale edge `HUB/monitor/offsite.go:636`; `offsite_escrow_pending` is operator-only, see `AR/08-alarm-ladder.md:383`.", + "8": "TEST `TestIsStale_PersistentlyFailingTierGoesStale` `HUB/monitor/offsite_r100_test.go:55`; TEST `TestIsStale_NewbornBoxDoesNotAlarm` `HUB/monitor/offsite_r100_test.go:74`. Box clock ahead: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "8. Restore-test proof too old", + "answers": { + "1": "CUSTOMER (protects the promise that a backup restores; mail goes by event). `restoreTestChecker.Check` `E/hub/cmd/hub/main.go:944` on the 60 s sweep; `checkStaleness` `HUB/monitor/restoretest.go:242`.", + "2": "Every 60 s. Window per tier: 4 archive generations `HUB/monitor/restoretest.go:62`, never under `restoreProvenWindowFloor` 7 d `HUB/monitor/restoretest.go:68`, never over `restoreProvenWindowCap` 12 d `HUB/monitor/restoretest.go:74`. Constants, no setting.", + "3": "No time limit. Each pass reads 14 days of host reports per customer, `restoreWindowRead` `HUB/monitor/restoretest.go:79` at `HUB/monitor/restoretest.go:252`, every minute. This is the heaviest sweep read found; grows with boxes times report rate (candidate).", + "4": "Pod restart: the first pass does not re-send a stale signal already in the events table `HUB/monitor/restoretest.go:281-283`, `staleAlreadyReported` `HUB/monitor/restoretest.go:300-312`.", + "5": "Edge-triggered per customer and tier under a mutex `HUB/monitor/restoretest.go:270-278`. A deliberately-off box is skipped `HUB/monitor/restoretest.go:192-194`.", + "6": "A new box is \"unknown\" until its window has passed since first contact `HUB/monitor/restoretest.go:326-342`; first contact is hub receive time `HUB/store/store.go:3994-4005`. Proof times come from box reports; box clock error: NOBODY KNOWS.", + "7": "Event `restore_test_stale` warning through the dispatcher `HUB/monitor/restoretest.go:284-285`; a failed run is a separate event, `checkFailure` `HUB/monitor/restoretest.go:205`.", + "8": "TEST `TestRestoreTestStale_R1001_RestartDoesNotReMail` `HUB/monitor/r1001_restart_test.go:147`. Cost at 100 boxes: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "9. Off-site clean-up window (20 minutes, weekly)", + "answers": { + "1": "CUSTOMER DATA. The box asks; `OpenWindowFor` `HUB/offsitekeys/service.go:334` adds a deleting key line for 20 minutes; the hub's 60 s sweep `SweepExpiredWindows` `HUB/offsitekeys/service.go:413` closes a window left open. Sweep loop: `E/hub/cmd/hub/main.go:636-650`.", + "2": "Granted when the last window is at least `windowCadence` 6 d 12 h old `HUB/offsitekeys/service.go:264`, test at `HUB/offsitekeys/service.go:337`; length `windowLength` 20 min `HUB/offsitekeys/service.go:263`. Sweep every 60 s. Constants; the weekly switch is a store setting `HUB/offsitekeys/service.go:338`.", + "3": "Each sweep has a 2-minute context `E/hub/cmd/hub/main.go:644`. Work is per expired window, one remote call each `HUB/offsitekeys/service.go:419-422`; grows with boxes that crash mid-prune.", + "4": "Pod down when a window expires: the deleting key line stays on the storage account until the hub is back; the sweep then closes it `HUB/offsitekeys/service.go:419-431`. Nobody is told during the gap (candidate). A failed close is retried next sweep `HUB/offsitekeys/service.go:424-426`.", + "5": "One window per cadence via the ledger row `HUB/offsitekeys/service.go:364`. The daily key audit knows an open window `HUB/offsitekeys/service.go:212-213`. No lock against the abandon sweep; both run in the same goroutine `E/hub/cmd/hub/main.go:636-650`.", + "6": "Cadence and expiry use the hub clock, `s.now()` `HUB/offsitekeys/service.go:337`. The box's clock does not decide. Hub clock jump back: a window could stay open longer: NOBODY KNOWS.", + "7": "Operator event `offsite_window_failed` warning when the hub had to close it `HUB/offsitekeys/service.go:430-431`; `offsite_window_drop` error when too much went `HUB/offsitekeys/service.go:259`.", + "8": "TEST `TestWindow_LeftOpenIsClosedByTheSweep` `HUB/offsitekeys/service_test.go:99`; TEST `TestWindow_GrantOpenCloseAndDropAlarm` `HUB/offsitekeys/service_test.go:37`; DOC `AR/07-backup-architecture.md:256`." + }, + "extra": [] + }, + { + "title": "10. Delayed deletion of a set-aside off-site copy", + "answers": { + "1": "CUSTOMER DATA. `SweepAbandons` `HUB/offsitekeys/service.go:529` deletes a household's set-aside copy once its request is due; called from the 60 s sweep `E/hub/cmd/hub/main.go:636-650`.", + "2": "Due `DefaultAbandonDelay` 7 days after the request `HUB/offsitekeys/service.go:448`; override `OFFSITE_ABANDON_DELAY` `E/hub/cmd/hub/main.go:598-605`. Checked every 60 s. Hub clock.", + "3": "2-minute context shared with the window sweep `E/hub/cmd/hub/main.go:644`. One remote delete per due request `HUB/offsitekeys/service.go:536-538`; duration of a large delete: NOBODY KNOWS.", + "4": "Pod down at the due time: deleted on the first sweep after start; the due list is in the store `HUB/offsitekeys/service.go:530`. A failed delete is marked and retried every sweep `HUB/offsitekeys/service.go:540-547`.", + "5": "A request is deleted once, then marked `HUB/offsitekeys/service.go:549`. A cancel before the due time wins: design text `HUB/offsitekeys/service.go:438-441`.", + "6": "Hub clock only. With an env override set too short by mistake the week of warning shrinks (candidate; value read at `E/hub/cmd/hub/main.go:598-605`).", + "7": "Operator events: requested, cancelled, deleted, failed `HUB/offsitekeys/service.go:444-447`; failure mails once per request `HUB/offsitekeys/service.go:543-546`.", + "8": "TEST `TestAbandon_DelayCancelAndScope` `HUB/offsitekeys/service_test.go:134`." + }, + "extra": [] + }, + { + "title": "11. Off-site key audit (07:10)", + "answers": { + "1": "OPERATOR (guards household copies against deletion). `scheduleDaily(ctx, \"offsite-key-audit\", \"07:10\", ...)` `E/hub/cmd/hub/main.go:651-655` runs `AuditAll` `HUB/offsitekeys/service.go:195`.", + "2": "Daily 07:10 Budapest, literal slot `E/hub/cmd/hub/main.go:651`. Zone from `scheduleDaily` `E/hub/cmd/hub/main.go:1072`.", + "3": "10-minute limit for the whole fleet `E/hub/cmd/hub/main.go:652`. One remote read per provisioned customer, in sequence `HUB/offsitekeys/service.go:202-215`; at 100 boxes the 10 minutes may not cover all (candidate).", + "4": "Pod down at 07:10: skipped that day, no catch-up `E/hub/cmd/hub/main.go:1095-1112`. Nobody notices.", + "5": "Runs once per day by the `last` guard `E/hub/cmd/hub/main.go:1095-1112`. Meets an open clean-up window: the audit is told the window is open `HUB/offsitekeys/service.go:212-213`.", + "6": "Wall-clock slot, DST-safe by `nextDailyRun` `E/hub/cmd/hub/main.go:1095-1112`.", + "7": "Operator: `offsite_key_unlocked` error, or `offsite_key_audit_failed` when a file cannot be read `HUB/offsitekeys/service.go:192-194`.", + "8": "TEST `TestWindow_PrependAuditClose` `HUB/offsitekeys/offsitekeys_test.go:158`; DOC `AR/05-hub-architecture.md:642`. The 10-minute limit at fleet size: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "12. Off-site credential stuck and its self-heal", + "answers": { + "1": "CUSTOMER. Two jobs, merged: `offsiteDeliveryChecker.Check` `E/hub/cmd/hub/main.go:945` (60 s sweep) and the `offsiteheal` reconciler `HUB/offsiteheal/reconciler.go:165-177`, started at `E/hub/cmd/hub/main.go:846-854`.", + "2": "Stuck after `stuckAfter` 1 h `HUB/monitor/offsite_delivery.go:38`; cooldowns 24 h `HUB/monitor/offsite_delivery.go:40-41`. Heal sweep every `tickIntervalDefault` 5 min `HUB/offsiteheal/reconciler.go:74`, acts after 2 distinct reports `HUB/offsiteheal/reconciler.go:70`.", + "3": "Heal call limit 3 minutes `HUB/monitor/offsite_delivery.go:275`. Both list every customer config per pass `HUB/monitor/offsite_delivery.go:72` and `HUB/offsiteheal/reconciler.go:183`; linear in boxes.", + "4": "Pod down: nothing; on return the next tick re-reads state. The cooldowns are \"durable via the events table\" `HUB/monitor/offsite_delivery.go:39`.", + "5": "Two healers could meet: the checker re-reads the secret row immediately before acting `HUB/monitor/offsite_delivery.go:28-31`, and a box declaration is owned by offsiteheal `HUB/monitor/offsite_delivery.go:97-99`.", + "6": "Age counted from the hub-held `status.Since` `HUB/monitor/offsite_delivery.go:93`. Debounce is counted in reports, not ticks `HUB/offsiteheal/reconciler.go:61-69`.", + "7": "Operator events `offsite_delivery_stuck`, `offsite_credential_restaged` `HUB/monitor/offsite_delivery.go:34-35`; `offsite_selfheal_restaged` / `_reissued` `HUB/offsiteheal/reconciler.go:57-58`.", + "8": "SOURCE `HUB/offsiteheal/reconciler.go:61-74`; DOC `AR/06-offsite-connectivity.md:386`. Test names for the timing: NOT FOUND (grep `func Test.*Cooldown` in `HUB/monitor/offsite_delivery.go:1` directory not run)." + }, + "extra": [] + }, + { + "title": "13. PBS whole-box credential self-heal", + "answers": { + "1": "CUSTOMER. `pbsdrheal` reconciler `HUB/pbsdrheal/reconciler.go:153-165`, started at `E/hub/cmd/hub/main.go:829-837`; repairs a box whose off-site PBS credential did not land.", + "2": "Every 5 minutes `HUB/pbsdrheal/reconciler.go:130`, or on a trigger `HUB/pbsdrheal/reconciler.go:160`; acts after 2 reports `HUB/pbsdrheal/reconciler.go:131`. A staged secret unconsumed for `unconsumedGrace` 15 min is flagged `HUB/pbsdrheal/reconciler.go:63`.", + "3": "No time limit on a pass, `reconcileOnce(ctx)` `HUB/pbsdrheal/reconciler.go:163`. Reads all heal states `HUB/pbsdrheal/reconciler.go:170`; linear in boxes.", + "4": "Pod down: no heal; the debounce maps are in memory `HUB/pbsdrheal/reconciler.go:134-135`, so after a restart a box waits two more reports.", + "5": "One goroutine; a read error does nothing this tick `HUB/pbsdrheal/reconciler.go:171-174`.", + "6": "Grace is hub time. Box clock: not used here.", + "7": "Operator audit events `pbsdr_selfheal_*` and `pbsdr_unconsumed_secret` `HUB/pbsdrheal/reconciler.go:50-54`.", + "8": "SOURCE `HUB/pbsdrheal/reconciler.go:57-63`. A pass that hangs on the remote: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "14. WireGuard peer list push to ep0", + "answers": { + "1": "OPERATOR (keeps every box's tunnel allowed). `wgReconciler.Run` `E/hub/cmd/hub/main.go:752`, loop `HUB/wgsync/reconciler.go:90-106`; pushes the FULL peer list over SSH.", + "2": "Every 5 minutes `HUB/wgsync/reconciler.go:41` and on every change via `Trigger` `HUB/wgsync/reconciler.go:48-53`. No setting.", + "3": "60 s limit per push `HUB/wgsync/reconciler.go:100`. The payload grows with peers `HUB/wgsync/reconciler.go:71-73`.", + "4": "Pod down: no push; ep0 keeps the last list. A failed push is logged and retried next tick `HUB/wgsync/reconciler.go:101-103`. Nobody is mailed on repeated failure (candidate): NOT FOUND (grep `SaveEvent` in `HUB/wgsync/reconciler.go:1` whole file).", + "5": "One loop; a pending trigger covers later ones `HUB/wgsync/reconciler.go:46-47`. Full list each time erases drift `HUB/wgsync/reconciler.go:20-23`.", + "6": "No clock in the payload.", + "7": "Log line only `HUB/wgsync/reconciler.go:102`.", + "8": "TEST `TestReconciler_FailureRetriedOnTick` `HUB/wgsync/reconciler_test.go:119`; TEST `TestReconciler_TriggerPushesFullList` `HUB/wgsync/reconciler_test.go:82`." + }, + "extra": [] + }, + { + "title": "15. Box clock ran ahead: tunnel alarm", + "answers": { + "1": "OPERATOR (the household's whole-box copies stop). `checkWGTunnel` `HUB/api/host_tunnel.go:61`, called on every host report `HUB/api/handler.go:1010`; not a timer, it runs when the box reports.", + "2": "Fires when `handshake_future_s` > 0 `HUB/api/host_tunnel.go:41` or the last handshake is older than `wgTunnelStaleAfterS` 6 h `HUB/api/host_tunnel.go:23`. Constant.", + "3": "One store read per report `HUB/api/host_tunnel.go:66`.", + "4": "If the box stops reporting, this check never runs; block 1 covers that. An older agent that sends neither field is not judged `HUB/api/host_tunnel.go:62-64`.", + "5": "Edge-triggered from the newest stored tunnel event `HUB/api/host_tunnel.go:71-83`, so a restart does not re-mail.", + "6": "This IS the clock-ahead handling: the future stamp is measured by the box's own clock `HUB/api/host_tunnel.go:31`; repair is a new key after 15 minutes on the agent side `HUB/api/host_tunnel.go:14-17`. Known gaps are listed in `AR/05-hub-architecture.md:489-490`.", + "7": "Operator only, `wg_tunnel_stale` warning; `wg_tunnel_ok` is stored, not mailed `HUB/api/host_tunnel.go:12-14`.", + "8": "TEST `TestD30_TheAlarmClearsWhenTheNewKeyShakesHands` `HUB/api/host_tunnel_test.go:83`; DOC `AR/05-hub-architecture.md:482-490`." + }, + "extra": [] + }, + { + "title": "16. The daily scheduler and the clock change", + "answers": { + "1": "OPERATOR (mechanism under blocks 2, 11, 17, 18). `scheduleDaily` `E/hub/cmd/hub/main.go:1069-1093` with `nextDailyRun` `E/hub/cmd/hub/main.go:1095-1112`.", + "2": "Budapest wall-clock, `LoadLocation` `E/hub/cmd/hub/main.go:1072`; fallback fixed CET `E/hub/cmd/hub/main.go:1074` (candidate: in summer the fallback is one hour off). Pod `TZ` is `Europe/Budapest` `E/manifests/hub.yaml:140-141`.", + "3": "Sleeps with `time.After(delay)` `E/hub/cmd/hub/main.go:1088`; the job runs inline `E/hub/cmd/hub/main.go:1090`, so a long job delays only its own next slot.", + "4": "No catch-up for a slot missed while the pod was down; exceptions are the snapshot (block 17) and prune (block 18). A bad time string becomes 00:00 silently, `parseHM` `E/hub/cmd/hub/main.go:1115-1121` (candidate).", + "5": "Next run is the calendar day after and the slot that ran last never runs again `AR/05-hub-architecture.md:642-645`.", + "6": "25-hour and 23-hour days handled, D24 `AR/05-hub-architecture.md:640-646`. A hub clock jump while sleeping: the timer keeps its old duration: NOBODY KNOWS.", + "7": "Nobody; start-up logs only.", + "8": "TEST `TestR1183_DailyJobRunsOnceOnTheDayTheClocksGoBack` `E/hub/cmd/hub/r1183_dst_schedule_test.go:31`; TEST `TestR1183_DailyJobRunsOnceOnTheDayTheClocksGoForward` `E/hub/cmd/hub/r1183_dst_schedule_test.go:56`." + }, + "extra": [] + }, + { + "title": "17. Hub database snapshot (four slots)", + "answers": { + "1": "OPERATOR (the hub holds every household's escrow and settings). `runSnapshot` scheduled over `dbsnap.Slots` `E/hub/cmd/hub/main.go:967-983`; `Make` `HUB/dbsnap/dbsnap.go:72-117`.", + "2": "02:00, 08:00, 14:00, 20:00 Budapest: `Slots` `HUB/dbsnap/dbsnap.go:32`. Keeps `Keep = 2` `HUB/dbsnap/dbsnap.go:27`. File name carries UTC `HUB/dbsnap/dbsnap.go:89`.", + "3": "10-minute limit `E/hub/cmd/hub/main.go:972`. Time grows with database size; written to a tmp file then renamed `HUB/dbsnap/dbsnap.go:91-104`; duration is logged `HUB/dbsnap/dbsnap.go:113-114`.", + "4": "The one job WITH catch-up: at start, if the newest snapshot is older than `CatchUpAge` 6 h, one is made `E/hub/cmd/hub/main.go:978-980`, `NeedsCatchUp` `HUB/dbsnap/dbsnap.go:148-159`.", + "5": "A running flag refuses a second run with `ErrBusy` `HUB/dbsnap/dbsnap.go:73-80`. The DooPlex push timer runs 30 min later and waits for the tmp file (block 26).", + "6": "DST-safe slots (block 16). The doc still says \"older than 24 h\" and \"02:30\" in its original text `AR/05-hub-architecture.md:618-620` (candidate: stale text).", + "7": "Failure is a log line; the outside alarm is the push timer's (block 26).", + "8": "TEST `TestD34_SlotsAreEverySixHours` `HUB/dbsnap/dbsnap_test.go:214`; TEST `TestR173_MainSchedulesTheDBSnapshot` `E/hub/cmd/hub/r173_wiring_test.go:14`." + }, + "extra": [] + }, + { + "title": "18. Daily prune of old hub rows (04:30)", + "answers": { + "1": "OPERATOR. `pruneAll` `E/hub/cmd/hub/main.go:1123-1150`, at start and by `scheduleDaily(ctx, \"prune\", ...)` `E/hub/cmd/hub/main.go:860-865`.", + "2": "Daily at `prune_schedule`, default `\"04:30\"` `E/hub/cmd/hub/main.go:1026`, deployed `E/manifests/hub.yaml:83`; keeps `max_days` 90 `E/manifests/hub.yaml:82`. Also app telemetry 90 d `E/hub/cmd/hub/main.go:1134`, stale issues 30 d `E/hub/cmd/hub/main.go:1139`, deleted-customer audit 1 year `E/hub/cmd/hub/main.go:1145`.", + "3": "No time limit. Deletes grow with reports per box per day; duration NOT FOUND (grep `time.Since` in `E/hub/cmd/hub/main.go:1123-1150`).", + "4": "Runs at every start, so a missed slot is caught up by the next restart; with `MaxDays` 0 it never runs `E/hub/cmd/hub/main.go:860-865`.", + "5": "May meet the 05:00 deadline check or a snapshot only if it runs over 30 min; no lock found.", + "6": "`Prune` builds its cutoff from local time without `.UTC()` `HUB/store/store.go:1607` while the column is UTC `HUB/store/store.go:129`; `PruneEvents` uses UTC `HUB/store/store.go:2873` (candidate: 1-2 h skew on a 90-day cutoff, harmless in size).", + "7": "Log only.", + "8": "SOURCE `E/hub/cmd/hub/main.go:1123-1150`. Test for the cutoff zone: NOT FOUND (grep `func Test.*Prune` not run across `HUB/store/store.go:1606` directory)." + }, + "extra": [] + }, + { + "title": "19. The other checks on the 60-second sweep", + "answers": { + "1": "CUSTOMER and OPERATOR mixed. Merged: host capability, host leaf, host disk, storage fill, management plane, OOB, controller supervisor `E/hub/cmd/hub/main.go:937-946`. All run in the one sweep goroutine `E/hub/cmd/hub/main.go:927-965`.", + "2": "Every 60 s in the fixed order of `E/hub/cmd/hub/main.go:935-962`. A backup-cadence refusal repeats every `refusalRepeatEvery` 24 h `HUB/monitor/host_capability.go:72`.", + "3": "No per-checker limit: a slow checker delays all the others, including liveness (block 1), because they run in sequence `E/hub/cmd/hub/main.go:935-962` (candidate). Each reads the whole fleet per pass.", + "4": "Pod down: nothing. After start, band states are rebuilt from the latest reports.", + "5": "One goroutine, so no overlap; a Go ticker drops ticks when a pass overruns.", + "6": "They judge the latest report's content, not time, except the 24 h repeat `HUB/monitor/host_capability.go:172`.", + "7": "Dispatcher mails; storage fill has its own 6 h grain `AR/08-alarm-ladder.md:305`.", + "8": "SOURCE `E/hub/cmd/hub/main.go:927-965`. Duration of one full sweep: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "20. Reminder about a held app", + "answers": { + "1": "OPERATOR (D53: no household mail). `heldApps.Check` `E/hub/cmd/hub/main.go:947`; `HeldAppsChecker.Check` `HUB/monitor/held_apps.go:102`.", + "2": "One pass per `heldAppCheckEvery` 15 min `HUB/monitor/held_apps.go:36`. First mail after a grace of 6 h / 72 h / 24 h by reason `HUB/monitor/held_apps.go:40-48`; then every `heldAppRepeat` 7 d `HUB/monitor/held_apps.go:34`.", + "3": "One list of all held apps `HUB/monitor/held_apps.go:114`.", + "4": "Pod down: the reminder comes on the first pass after start; the reminded-at time is stored BEFORE sending `HUB/monitor/held_apps.go:141-145`, so a restart cannot mail twice.", + "5": "A row the box has not confirmed for `heldAppStale` 50 h is not reminded `HUB/monitor/held_apps.go:125-129`.", + "6": "The age counts from the hold's own time, which is the BOX's clock, else the hub's first sight `HUB/monitor/held_apps.go:130-134` and `HUB/monitor/held_apps.go:22`. Box clock behind: reminder early; ahead: late (candidate).", + "7": "Operator, `app_held_reminder` warning `HUB/monitor/held_apps.go:157-163`.", + "8": "TEST file named in source `HUB/monitor/held_apps.go:31`; DOC `AR/08-alarm-ladder.md:655`." + }, + "extra": [] + }, + { + "title": "21. \"A golden is published but not vouched\"", + "answers": { + "1": "OPERATOR. `goldenVouch` check on the sweep `E/hub/cmd/hub/main.go:948-950`; `GoldenVouchChecker.Check` `HUB/monitor/golden_vouch.go:54`.", + "2": "One Gitea listing per `goldenVouchCheckEvery` 15 min; alarm after `goldenVouchGrace` 2 h `HUB/monitor/golden_vouch.go:42-43`.", + "3": "Listing limit `goldenVouchListDeadline` 20 s `HUB/monitor/golden_vouch.go:44`; it runs inside the sweep, so a slow Gitea holds the sweep for up to 20 s `HUB/monitor/golden_vouch.go:62-64`.", + "4": "Pod restart: the gap clock and the alerted mark are in memory `HUB/monitor/golden_vouch.go:34-37`, so the 2 h starts again and the alarm is mailed again (\"one alarm per unvouched version per process\" `HUB/monitor/golden_vouch.go:23`).", + "5": "Gitea unreachable or empty list: no verdict, never an alarm `HUB/monitor/golden_vouch.go:65-67`.", + "6": "Hub clock only.", + "7": "Operator event `golden_not_vouched` `HUB/monitor/golden_vouch.go:95-96`.", + "8": "TEST `TestGoldenVouchChecker` named at `HUB/monitor/golden_vouch.go:24`." + }, + "extra": [] + }, + { + "title": "22. Shared storage boxes: fill and reachability", + "answers": { + "1": "OPERATOR (every household's off-site space). `offsiteBox` and `pbsdrBox` checks `E/hub/cmd/hub/main.go:951-956`.", + "2": "On the 60 s sweep, but one provider read per `boxFetchInterval` 15 min `HUB/monitor/offsite_box.go:72`, test at `HUB/monitor/offsite_box.go:132` and `HUB/monitor/pbsdr_box.go:130`. Warn 80 %, critical 90 % `HUB/monitor/offsite_box.go:67-68`.", + "3": "Provider client limit 30 s `HUB/hetznerapi/hetznerapi.go:143-145`. Does not grow with boxes.", + "4": "Pod down: no reading. Unreachable is declared after consecutive failed 15-minute windows `HUB/monitor/pbsdr_box.go:80-82`.", + "5": "Throttled by `lastFetch` in memory; first pass after a restart fetches at once `HUB/monitor/offsite_box.go:132`.", + "6": "Hub clock only.", + "7": "Operator events for fill, oversubscription, unreachable and recovered `HUB/monitor/offsite_box.go:266-330`.", + "8": "SOURCE `HUB/monitor/offsite_box.go:70-72`." + }, + "extra": [] + }, + { + "title": "23. Purges: log bundles, access log, sessions", + "answers": { + "1": "OPERATOR. Merged: `PurgeExpiredLogBundles` on the sweep `E/hub/cmd/hub/main.go:958-962`; access-log `prune` `HUB/accesslog/accesslog.go:192`; `CleanupSessions` `HUB/web/server.go:199-217`.", + "2": "Bundles: every 60 s, TTL `logBundleTTL` 72 h `HUB/store/logbundle.go:29`. Access log: on the first line of each UTC day `HUB/accesslog/accesslog.go:166-184`, keeps `DefaultKeep` 90 d `HUB/accesslog/accesslog.go:44`. Sessions: every 15 min `HUB/web/server.go:200`, life 7 d `HUB/web/server.go:949`.", + "3": "Small. Access-log prune lists one directory `HUB/accesslog/accesslog.go:196`.", + "4": "Pod restart: sessions are in memory `HUB/web/server.go:207-213`, so every operator is logged out. Access-log prune runs when the day file is first opened `HUB/accesslog/accesslog.go:182-184`, so it also runs after a restart.", + "5": "Access-log prune only removes exact `access-YYYY-MM-DD.log` names `HUB/accesslog/accesslog.go:191`.", + "6": "Access log days are UTC `HUB/accesslog/accesslog.go:166`, unlike the Budapest daily jobs; documented `AR/05-hub-architecture.md:634-636`.", + "7": "Nobody.", + "8": "TEST `TestAccessLog_Keeps90DaysThenDeletes` `HUB/accesslog/accesslog_test.go:135`." + }, + "extra": [] + }, + { + "title": "24. Links and tokens that expire by themselves", + "answers": { + "1": "CUSTOMER (bind link) and OPERATOR (tokens). No sweep: each is checked when used. Bind link `selfBindTTL` 7 d `HUB/web/selfbind_mint.go:20`; access token expiry `HUB/web/tokens.go:95-96`; DR recovery `defaultRecoveryTTL` 30 min, max 4 h `HUB/api/dr.go:21-22`.", + "2": "Bind link: 7 days from mint `HUB/web/selfbind_mint.go:77`; a fresh link is mailed at most once per `bindResendEvery` 1 h `HUB/web/selfbind.go:382`. Token life is chosen in days at creation `HUB/web/tokens.go:180`. Operator actions: `OperatorActionTTL` 24 h `HUB/store/opactions.go:62`.", + "3": "Nothing runs; expired rows stay in the tables. A purge of expired bind tokens: NOT FOUND (grep `DELETE FROM selfbind_tokens` not run; grep `selfBindTTL|Expired` in `HUB/web/selfbind.go:206` area shows read-time checks only).", + "4": "Not applicable: no job. An expired link reads as expired and falls back to operator bind `HUB/web/selfbind_mint.go:18-19`.", + "5": "Bind expiry is tested twice, in Go `HUB/store/selfbind.go:39` and in SQL `HUB/store/selfbind.go:177`.", + "6": "Hub clock. SQL side uses `datetime('now')` UTC `HUB/store/selfbind.go:177`.", + "7": "Nobody is warned BEFORE a token or link expires; an expired token logs a WARN on use `HUB/web/tokens.go:96` (candidate: a token used by automation expires silently until the next call).", + "8": "SOURCE `HUB/store/selfbind.go:38-46`. Warning before expiry: NOT FOUND (grep `expires in|about to expire` not matched in `HUB/web/tokens.go:1` whole file)." + }, + "extra": [] + }, + { + "title": "25. Version check and catalog template fetch", + "answers": { + "1": "OPERATOR. `VersionChecker.Run` `HUB/web/version.go:43-55` and `TemplateFetcher.Run` `HUB/web/templatefetcher.go:41-53`, started at `E/hub/cmd/hub/main.go:711-722` and `E/hub/cmd/hub/main.go:327-335`.", + "2": "Once at start, then every `check_interval` 6 h `E/manifests/hub.yaml:104` and `template_interval` 1 h `E/manifests/hub.yaml:105`.", + "3": "One HTTP call each.", + "4": "Pod down: nothing; both fetch at once on start `HUB/web/version.go:44` and `HUB/web/templatefetcher.go:42`.", + "5": "One goroutine each.", + "6": "Interval tickers; no wall clock.", + "7": "A failure is a WARN line and a stored `lastError` `HUB/web/version.go:62-65`; nobody is mailed.", + "8": "SOURCE `HUB/web/version.go:41-55`." + }, + "extra": [] + }, + { + "title": "26. DooPlex: push of the hub database snapshot off-site", + "answers": { + "1": "NOT MEASURED. OPERATOR. systemd timer `felhom-hub-db-backup.timer` on DooPlex, `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10`, runs the script E/scripts/hub-db-backup/felhom-hub-db-backup:19; installed by hand per `RB/RUNBOOK-hub-db-offsite-backup.md:107-108`.", + "2": "02:30, 08:30, 14:30, 20:30 `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10`, `RandomizedDelaySec=2min` `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:12`. Host local time (no zone in the unit). DooPlex's zone: NOT FOUND (grep `timedatectl|Europe/Budapest` in documentation/operations and the three runbooks).", + "3": "`TimeoutStartSec=45min` `E/scripts/hub-db-backup/felhom-hub-db-backup.service:13`. Waits up to `WAIT_S` 1200 s for the hub's tmp file E/scripts/hub-db-backup/felhom-hub-db-backup:20, loop E/scripts/hub-db-backup/felhom-hub-db-backup:35-40. Grows with database size.", + "4": "`Persistent=true` `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:11`: a slot missed while DooPlex was off runs at boot. It refuses a snapshot older than `MAX_AGE_H` 7 E/scripts/hub-db-backup/felhom-hub-db-backup:19, line E/scripts/hub-db-backup/felhom-hub-db-backup:50.", + "5": "No flock in the script (grep `flock|lock` returned nothing); systemd does not start a oneshot unit twice. A catch-up run right after boot can meet a hub that has not made its catch-up snapshot yet: history of a false failure in E/scripts/hub-db-backup/felhom-hub-db-backup:15-18.", + "6": "Hub slots are Budapest (pod TZ), timer slots are host local: the 30-minute offset holds only if both zones match; pinned in text by `E/scripts/hub-db-backup/test_hub_db_backup.py:309-316`. DST night: NOBODY KNOWS.", + "7": "Failure: `OnFailure=felhom-backup-failmail@%n.service` `E/scripts/hub-db-backup/felhom-hub-db-backup.service:6`, mail to admin E/scripts/dooplex-offsite/felhom-backup-failmail:2. Silence: Prometheus `HubDBBackupStale` at 13 h per `RB/RUNBOOK-hub-db-offsite-backup.md:100`, but the alert YAML in the same runbook shows 26 h `RB/RUNBOOK-hub-db-offsite-backup.md:153` (candidate).", + "8": "TEST `E/scripts/hub-db-backup/test_hub_db_backup.py:309-316` (run by hand, `RB/RUNBOOK-hub-db-offsite-backup.md:107-108`); the live alert rule is in homelab-manifests, not in this clone: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "27. DooPlex: Sunday restore tests", + "answers": { + "1": "NOT MEASURED. OPERATOR. Two timers, merged: `felhom-hub-db-restore-test.timer` `E/scripts/hub-db-backup/felhom-hub-db-restore-test.timer:6` and `felhom-dooplex-offsite-restore-test.timer` `E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test.timer:6`.", + "2": "Sunday 04:30 and Sunday 05:30, host local time; `Persistent=true` `E/scripts/hub-db-backup/felhom-hub-db-restore-test.timer:7`.", + "3": "Limits 45 min `E/scripts/hub-db-backup/felhom-hub-db-restore-test.service:13` and 90 min `E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test.service:13`. Grow with database and dump size.", + "4": "Missed Sunday runs at next boot (Persistent). Each refuses a copy older than `MAX_AGE_H` 50 E/scripts/hub-db-backup/felhom-hub-db-restore-test:13 and E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test:14.", + "5": "Sunday 04:30 lies between the hub's 02:00 snapshot push and the 05:00 PBS pull; no lock between them found (NOT FOUND: grep `flock` in E/scripts/hub-db-backup/felhom-hub-db-restore-test:1 whole file).", + "6": "Last Sunday of October has 02:00-03:00 twice; systemd's handling for 04:30 is outside the repo: NOBODY KNOWS.", + "7": "Staleness alarm `HubDBRestoreTestStale` after 8 days `RB/RUNBOOK-hub-db-offsite-backup.md:158`.", + "8": "DOC `RB/RUNBOOK-hub-db-offsite-backup.md:152-161`. Whether the rule is deployed: NOBODY KNOWS (homelab-manifests is not in the clone tree)." + }, + "extra": [] + }, + { + "title": "28. DooPlex: nightly off-site push and release copy", + "answers": { + "1": "NOT MEASURED. OPERATOR. Merged: `felhom-dooplex-offsite.timer` `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:6` and `felhom-release-copy.timer` `E/scripts/release-copy/felhom-release-copy.timer:7`.", + "2": "00:20 and 01:40 host local time; the order is stated in a comment `E/scripts/release-copy/felhom-release-copy.timer:6`. Random delay 2 min and 5 min `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:8`, `E/scripts/release-copy/felhom-release-copy.timer:9`.", + "3": "Limits 90 min `E/scripts/dooplex-offsite/felhom-dooplex-offsite.service:13` and 2 h `E/scripts/release-copy/felhom-release-copy.service:12`. A 90-minute run from 00:20 ends 01:50, after the 01:40 start (candidate: can overlap; no lock found).", + "4": "Both `Persistent=true` `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:7`, `E/scripts/release-copy/felhom-release-copy.timer:8`. The push refuses a dump older than `DUMP_MAX_AGE_H` 7 E/scripts/dooplex-offsite/felhom-dooplex-offsite:23, message E/scripts/dooplex-offsite/felhom-dooplex-offsite:55.", + "5": "The push depends on a dump CronJob that is not in this clone (named at E/scripts/dooplex-offsite/felhom-dooplex-offsite:55). One retry E/scripts/dooplex-offsite/felhom-dooplex-offsite:66.", + "6": "00:20 and 01:40 are outside the 02:00-03:00 change hour.", + "7": "`OnFailure` mail on both `E/scripts/dooplex-offsite/felhom-dooplex-offsite.service:6`, `E/scripts/release-copy/felhom-release-copy.service:6`; a metric file is written E/scripts/dooplex-offsite/felhom-dooplex-offsite:143-153.", + "8": "SOURCE only. The release-copy timer is enabled by hand `E/scripts/release-copy/install.sh:4`; whether it is enabled: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "29. DooPlex and ep0: PBS copy, prune, verify, clean-up", + "answers": { + "1": "NOT MEASURED. OPERATOR (holds the second copy of every household's whole-box backup). PBS jobs described in `RB/ep0-datastore-copy.md:15-18`, plus `felhom-ep0-copy-gc.timer` `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:5`.", + "2": "ep0 prune 03:30, pull to DooPlex 05:00 `RB/ep0-datastore-copy.md:15`; DooPlex prune 07:30 `RB/ep0-datastore-copy.md:17`; copy clean-up 08:00 `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:5`; verify Saturday 06:30 `RB/ep0-datastore-copy.md:16`; GC Sunday 08:30 `RB/ep0-datastore-copy.md:18`; hub-DB prune on ep0 03:45 `RB/RUNBOOK-hub-db-offsite-backup.md:101`.", + "3": "The PBS jobs live in PBS's own config, not in the repo; their duration and limits: NOBODY KNOWS. Grow with every box's backup size.", + "4": "The gc timer is `Persistent=true` `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:6`. PBS jobs: catch-up behaviour not in repo.", + "5": "Order is by clock only: prune 07:30 then clean-up 08:00 `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:2`. A pull that runs past 07:30 meets the prune: NOBODY KNOWS.", + "6": "Two machines, two clocks (ep0 and DooPlex): \"ep0 prune 03:30 before the 05:00 pull\" holds only while both agree. Their zones: NOT FOUND (grep `timedatectl|Europe/Budapest` in `RB/ep0-datastore-copy.md:1` whole file).", + "7": "PBS notification target per `RB/ep0-datastore-copy.md:19`. The gc unit has no `OnFailure` and no time limit (grep `OnFailure|TimeoutStartSec` in `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.service:9` whole file returned nothing) (candidate).", + "8": "DOC `RB/ep0-datastore-copy.md:15-19`." + }, + "extra": [] + }, + { + "title": "30. Nightly CI: gates and the security watch", + "answers": { + "1": "NOT MEASURED. OPERATOR. Gitea CI crons in felhom.eu: `gates` `E/.gitea/workflows/gates.yml:26-27` and `security-watch` `E/.gitea/workflows/security-watch.yml:21-22`. The other three repos have no scheduled job (grep `schedule:|cron:` in their `.gitea/workflows/` returned nothing).", + "2": "03:20 UTC and 03:40 UTC every night; the reason for the order is in `E/.gitea/workflows/gates.yml:21-23`. CI cron is UTC per `AR/08-alarm-ladder.md:710`, so it moves one hour against the Budapest night jobs at each clock change.", + "3": "Limits: 5 min `E/.gitea/workflows/gates.yml:53` `E/.gitea/workflows/gates.yml:20` min `E/.gitea/workflows/security-watch.yml:33`. The watch grows with the number of pinned programs.", + "4": "Missed night: no catch-up found. The watch's weekly summary self-heals: it is sent when the last one is 8+ days old `AR/08-alarm-ladder.md:717`. A runner that drops the job silently: NOBODY KNOWS.", + "5": "Both run on `felhom-gates` runners `E/.gitea/workflows/security-watch.yml:32`; the watch commits its memory to main with 3 attempts `E/.gitea/workflows/security-watch.yml:95-106`. A push race with a human commit is retried, then fails.", + "6": "UTC cron, no DST. Reminder ages are computed by the script from its memory file `AR/08-alarm-ladder.md:716-720`.", + "7": "Failure mail to admin: \"[felhom CI] security watch FAILED\" `E/.gitea/workflows/security-watch.yml:108-131`; gates has its own alarm step `E/.gitea/workflows/gates.yml:245`. Whether the alarm step runs for a reaped job is recorded as unknown `E/.gitea/workflows/gates.yml:50-52`.", + "8": "DOC `AR/08-alarm-ladder.md:708-722`. A night with no run at all: NOBODY KNOWS. \u00b7 Deliberate power-off hold: alarms held 14 days, `DeliberateOffHold` `HUB/store/deliberateoff.go:18`. \u00b7 A future-dated power-off request from a box is replaced by hub time `HUB/api/deliberate_off.go:30`. \u00b7 Long-poll wait: `waitMaxHold` 240 s, heartbeat 25 s `HUB/api/wait.go:17`. \u00b7 Presence: `PresenceWaitCadence` 243 s, grace 90 s `HUB/intent/presence.go:21`. \u00b7 Hetzner provisioning poll: every 3 s, at most 3 min `HUB/hetznerapi/hetznerapi.go:143-145`. \u00b7 Off-site scan back-off 2,4,8,16,30,30,30 s `HUB/offsite/offsite.go:112`. \u00b7 PBS self-check pause 7 s `HUB/web/pbsdr.go:571`. \u00b7 Host delete allowed after 360 s off `HUB/web/hosts.go:770`. \u00b7 Host operator reboot action TTL 30 min `HUB/store/opactions.go:47`. \u00b7 System page: line test stale after 15 days `HUB/web/system.go:174`. \u00b7 HTTP server timeouts `E/hub/cmd/hub/main.go:705-707`; shutdown grace 15 s `E/hub/cmd/hub/main.go:1002`. \u00b7 MAIL-HOLD has no expiry timer; it is released by hand `HUB/mailhold/mailhold.go:74`. \u00b7 No timed digest in the hub: the per-run digest comes from the box `HUB/notify/dispatcher.go:243`. \u00b7 NOT MEASURED: weekly golden bake is a manual Sunday-night step, not a timer `RB/nightly-rotation.md:15`. \u00b7 NOT MEASURED: `systemd-run --unit=golden-bake` is a manual start `RB/RUNBOOK-manual-build.md:180`. \u00b7 NOT MEASURED: DooPlex's own `dooplex-backup.timer` 03:19 and Longhorn 04:00 / Sunday 05:00 are named only in `RB/RUNBOOK-hub-db-offsite-backup.md:12-14`. \u00b7 NOT MEASURED: box-side `felhom-mgmt-watchdog.timer` is enabled by the installer `E/scripts/felhom-host-install.sh:3076` (belongs to the agent/installer helper). \u00b7 /mnt/5_hdd/felhom.eu/git/homelab-manifests does not exist in the clone tree (tried: ls of /mnt/5_hdd/felhom.eu/git/, find -maxdepth 2 -name 'homelab*'). So no k8s CronJob, Prometheus rule or Renovate schedule could be read from it. \u00b7 k8s CronJob in felhom.eu/manifests/: NOT FOUND (grep `kind: CronJob|schedule:` matched only `E/manifests/hub.yaml:83`, the hub's own prune time). \u00b7 Cloudflare periodic sync in the hub: NOT FOUND (grep `NewTicker|time.After|time.Sleep` in hub/internal/cloudflare showed only request timeouts). \u00b7 A hub-side security watch: NOT FOUND in hub code; it is the CI job of block 30. \u00b7 A timed mail digest in the hub: NOT FOUND (grep -il `digest` in hub/internal: only the box's per-run digest). \u00b7 A sweep that deletes expired bind links, tokens or claim codes: NOT FOUND (grep `expires|Expir|TTL` in hub/internal/claim, web/tokens.go, store/selfbind.go shows read-time checks only). \u00b7 DooPlex / ep0 host time zone for `OnCalendar`: NOT FOUND (grep `timedatectl|Europe/Budapest` in documentation/operations/*.md, RUNBOOK-hub-db-offsite-backup.md, ep0-datastore-copy.md, total-loss-of-dooplex.md). \u00b7 cron/crontab/`at ` installs by repo scripts on DooPlex or ep0: NOT FOUND (grep `crontab|/etc/cron|OnCalendar|systemd-run` in felhom.eu/scripts found only the systemd units listed in blocks 26-29 and the box installer). \u00b7 Scheduled CI in felhom-controller, felhom-agent, app-catalog-felhom.eu: NOT FOUND (grep `schedule:|cron:` in each `.gitea/workflows/`). \u00b7 Measured duration of any hub job: NOT FOUND in repository files; only the snapshot and the OS alarm pass log their own duration." + }, + "extra": [ + "## Tiny", + "## Not found" + ] + } + ], + "pointers": [ + { + "ptr": "E/hub/cmd/hub/main.go:935-936", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "1" + }, + { + "ptr": "HUB/monitor/host_staleness.go:207-221", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:927", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1027", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "2" + }, + { + "ptr": "E/manifests/hub.yaml:88", + "file": "OK", + "ident": "IDENT-MISS(\"30m\")", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "2" + }, + { + "ptr": "HUB/monitor/host_staleness.go:39", + "file": "OK", + "ident": "IDENT-MISS(\"30m\")", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "2" + }, + { + "ptr": "HUB/monitor/host_staleness.go:131", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:935-962", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "3" + }, + { + "ptr": "HUB/monitor/host_staleness.go:113", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "3" + }, + { + "ptr": "E/hub/cmd/hub/main.go:927-965", + "file": "OK", + "ident": "IDENT-MISS(Sincestart)", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "3" + }, + { + "ptr": "HUB/monitor/host_staleness.go:59-60", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "4" + }, + { + "ptr": "HUB/monitor/host_staleness.go:62-68", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "4" + }, + { + "ptr": "HUB/monitor/staleness.go:86-91", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "4" + }, + { + "ptr": "E/manifests/hub.yaml:122", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "5" + }, + { + "ptr": "E/manifests/hub.yaml:123-124", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "5" + }, + { + "ptr": "HUB/monitor/host_staleness.go:119-120", + "file": "OK", + "ident": "IDENT-MISS(E/manifests/hub.yaml:122,)", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "5" + }, + { + "ptr": "HUB/monitor/host_staleness.go:170-172", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "5" + }, + { + "ptr": "HUB/monitor/host_staleness.go:134-136", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "5" + }, + { + "ptr": "HUB/store/deliberateoff.go:18", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "5" + }, + { + "ptr": "HUB/store/store.go:3916", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "6" + }, + { + "ptr": "HUB/monitor/host_staleness.go:131", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "6" + }, + { + "ptr": "HUB/notify/dispatcher.go:609-612", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "7" + }, + { + "ptr": "HUB/notify/dispatcher.go:1018", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "7" + }, + { + "ptr": "HUB/monitor/r1001_restart_test.go:56", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "8" + }, + { + "ptr": "HUB/notify/r873_liveness_weekly_test.go:17", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "8" + }, + { + "ptr": "AR/05-hub-architecture.md:88-118", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Box liveness: \"your server cannot be reached\"", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/main.go:986-988", + "file": "OK", + "ident": "IDENT-MISS(scheduleDailyctx,)", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "1" + }, + { + "ptr": "HUB/monitor/deadline.go:409", + "file": "OK", + "ident": "IDENT-MISS(E/hub/cmd/hub/main.go:986-988)", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "1" + }, + { + "ptr": "HUB/monitor/deadline.go:539-544", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:986", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1072", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "2" + }, + { + "ptr": "HUB/monitor/deadline.go:29", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "2" + }, + { + "ptr": "HUB/monitor/deadline_tiers.go:28", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1090", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "3" + }, + { + "ptr": "HUB/monitor/deadline.go:36", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "3" + }, + { + "ptr": "HUB/monitor/deadline.go:569-570", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "3" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1095-1112", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "4" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1069-1093", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "4" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1095-1112", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "5" + }, + { + "ptr": "E/hub/cmd/hub/main.go:927", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "5" + }, + { + "ptr": "E/hub/cmd/hub/main.go:986", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "5" + }, + { + "ptr": "HUB/monitor/deadline.go:340-343", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "5" + }, + { + "ptr": "HUB/monitor/deadline.go:243", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "6" + }, + { + "ptr": "HUB/monitor/deadline.go:290-299", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "6" + }, + { + "ptr": "HUB/monitor/deadline.go:1", + "file": "OK", + "ident": "IDENT-MISS(future|skew)", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "6" + }, + { + "ptr": "HUB/monitor/deadline.go:417-419", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "6" + }, + { + "ptr": "HUB/monitor/deadline.go:539-544", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "7" + }, + { + "ptr": "HUB/monitor/deadline.go:486-494", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "7" + }, + { + "ptr": "HUB/monitor/deadline.go:432-442", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "7" + }, + { + "ptr": "HUB/monitor/r872_down_box_test.go:57", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/r1183_dst_schedule_test.go:82", + "file": "OK", + "ident": "IDENT-MISS(HUB/monitor/r872_down_box_test.go:57;)", + "ending": "2. Missed-backup deadline check (05:00)", + "q": "8" + }, + { + "ptr": "HUB/notify/dispatcher.go:1020-1098", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "1" + }, + { + "ptr": "HUB/notify/dispatcher.go:626-696", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "1" + }, + { + "ptr": "HUB/notify/dispatcher.go:609-612", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "2" + }, + { + "ptr": "HUB/notify/dispatcher.go:573", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "2" + }, + { + "ptr": "HUB/notify/dispatcher.go:1065-1069", + "file": "OK", + "ident": "IDENT-MISS(storageFillCooldown)", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "2" + }, + { + "ptr": "HUB/notify/dispatcher.go:1054-1062", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "2" + }, + { + "ptr": "HUB/notify/dispatcher.go:1281", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "2" + }, + { + "ptr": "HUB/notify/dispatcher.go:86", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "3" + }, + { + "ptr": "HUB/notify/dispatcher.go:1", + "file": "OK", + "ident": "IDENT-MISS()", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "3" + }, + { + "ptr": "HUB/notify/dispatcher.go:1054-1062", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "4" + }, + { + "ptr": "AR/08-alarm-ladder.md:435-439", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "4" + }, + { + "ptr": "HUB/notify/dispatcher.go:674", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "5" + }, + { + "ptr": "HUB/notify/dispatcher.go:665-672", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "5" + }, + { + "ptr": "HUB/notify/dispatcher.go:680-687", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "5" + }, + { + "ptr": "HUB/notify/dispatcher.go:1", + "file": "OK", + "ident": "IDENT-MISS(monotonic|clock)", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "6" + }, + { + "ptr": "HUB/notify/dispatcher.go:665-672", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "7" + }, + { + "ptr": "HUB/notify/dispatcher.go:1091-1094", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "7" + }, + { + "ptr": "HUB/notify/node_liveness_cooldown_test.go:17", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "8" + }, + { + "ptr": "HUB/notify/r873_liveness_weekly_test.go:17", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "8" + }, + { + "ptr": "AR/08-alarm-ladder.md:165-168", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Mail cooldowns and the weekly outage mail", + "q": "8" + }, + { + "ptr": "HUB/notify/dispatcher.go:99-104", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Retry of a failed operator mail", + "q": "1" + }, + { + "ptr": "HUB/notify/dispatcher.go:688-692", + "file": "OK", + "ident": "IDENT-MISS(processOperator)", + "ending": "4. Retry of a failed operator mail", + "q": "1" + }, + { + "ptr": "HUB/notify/dispatcher.go:76", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Retry of a failed operator mail", + "q": "2" + }, + { + "ptr": "HUB/notify/dispatcher.go:92", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Retry of a failed operator mail", + "q": "2" + }, + { + "ptr": "HUB/notify/dispatcher.go:86", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Retry of a failed operator mail", + "q": "3" + }, + { + "ptr": "HUB/notify/dispatcher.go:92", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Retry of a failed operator mail", + "q": "4" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1", + "file": "OK", + "ident": "IDENT-MISS(retry)", + "ending": "4. Retry of a failed operator mail", + "q": "4" + }, + { + "ptr": "HUB/notify/dispatcher.go:101", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Retry of a failed operator mail", + "q": "4" + }, + { + "ptr": "HUB/notify/dispatcher.go:674", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Retry of a failed operator mail", + "q": "5" + }, + { + "ptr": "AR/08-alarm-ladder.md:358-363", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Retry of a failed operator mail", + "q": "5" + }, + { + "ptr": "HUB/notify/dispatcher.go:101", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Retry of a failed operator mail", + "q": "7" + }, + { + "ptr": "AR/08-alarm-ladder.md:362-363", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Retry of a failed operator mail", + "q": "7" + }, + { + "ptr": "HUB/notify/operator_mail_retry_test.go:17", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Retry of a failed operator mail", + "q": "8" + }, + { + "ptr": "HUB/osupdates/kernel.go:191", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "1" + }, + { + "ptr": "HUB/osupdates/service.go:436", + "file": "OK", + "ident": "IDENT-MISS(Evaluate)", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:564", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "1" + }, + { + "ptr": "HUB/osupdates/kernel.go:46-47", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "2" + }, + { + "ptr": "HUB/osupdates/kernel.go:193-196", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "2" + }, + { + "ptr": "HUB/osupdates/kernel.go:48-50", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "2" + }, + { + "ptr": "HUB/osupdates/kernel.go:73-78", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "2" + }, + { + "ptr": "HUB/osupdates/kernel.go:203-208", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "3" + }, + { + "ptr": "HUB/osupdates/kernel.go:179-185", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "4" + }, + { + "ptr": "HUB/osupdates/kernel.go:236", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "4" + }, + { + "ptr": "HUB/osupdates/kernel.go:208-211", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "5" + }, + { + "ptr": "HUB/osupdates/kernel.go:236-239", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "5" + }, + { + "ptr": "HUB/osupdates/kernel.go:182", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "6" + }, + { + "ptr": "HUB/osupdates/kernel.go:193-194", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "6" + }, + { + "ptr": "HUB/osupdates/kernel.go:201", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "6" + }, + { + "ptr": "HUB/osupdates/kernel.go:226-233", + "file": "OK", + "ident": "IDENT-MISS(os_kernel_notice)", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "7" + }, + { + "ptr": "HUB/osupdates/kernel.go:212-219", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "7" + }, + { + "ptr": "HUB/osupdates/kernel_test.go:192", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "8" + }, + { + "ptr": "HUB/osupdates/kernel_test.go:79", + "file": "OK", + "ident": "IDENT-MISS(HUB/osupdates/kernel_test.go:192;)", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "8" + }, + { + "ptr": "HUB/osupdates/kernel_test.go:115", + "file": "OK", + "ident": "IDENT-MISS(HUB/osupdates/kernel_test.go:79;)", + "ending": "5. Kernel \"your box restarts tonight\" mail", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/main.go:554-586", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:564", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:577", + "file": "OK", + "ident": "IDENT-MISS(E/hub/cmd/hub/main.go:564)", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:574", + "file": "OK", + "ident": "IDENT-MISS(SincelastAlarms)", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:476", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:484-491", + "file": "OK", + "ident": "IDENT-MISS(E/hub/cmd/hub/main.go:476,)", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:536-541", + "file": "OK", + "ident": "IDENT-MISS(OS_APPROVE_AFTER)", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:581", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "3" + }, + { + "ptr": "HUB/osupdates/service.go:414-422", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "3" + }, + { + "ptr": "E/hub/cmd/hub/main.go:558", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "4" + }, + { + "ptr": "HUB/osupdates/service.go:1102", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "4" + }, + { + "ptr": "E/hub/cmd/hub/main.go:559-585", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "5" + }, + { + "ptr": "HUB/osupdates/service.go:1094-1108", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "5" + }, + { + "ptr": "HUB/osupdates/service.go:1102", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "6" + }, + { + "ptr": "HUB/osupdates/service.go:970", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "6" + }, + { + "ptr": "AR/08-alarm-ladder.md:375-383", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "7" + }, + { + "ptr": "HUB/osupdates/service.go:1092-1093", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "7" + }, + { + "ptr": "HUB/osupdates/alarms_test.go:226", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "8" + }, + { + "ptr": "AR/08-alarm-ladder.md:365", + "file": "OK", + "ident": "IDENT-MISS(TestAlarm_Ring0Stalled)", + "ending": "6. OS update approval tick and hourly OS alarms", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/main.go:943", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "1" + }, + { + "ptr": "HUB/monitor/offsite.go:166", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "1" + }, + { + "ptr": "HUB/monitor/offsite_escrow_pending.go:66", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "1" + }, + { + "ptr": "HUB/monitor/offsite.go:54", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "2" + }, + { + "ptr": "HUB/monitor/offsite.go:203-207", + "file": "OK", + "ident": "IDENT-MISS(defaultOffsiteStaleAfter)", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "2" + }, + { + "ptr": "HUB/monitor/offsite_escrow_pending.go:37", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "2" + }, + { + "ptr": "HUB/monitor/offsite_escrow_pending.go:110", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "2" + }, + { + "ptr": "HUB/monitor/offsite.go:443-459", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "3" + }, + { + "ptr": "HUB/monitor/offsite.go:463-468", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "4" + }, + { + "ptr": "HUB/monitor/offsite.go:1", + "file": "OK", + "ident": "IDENT-MISS(R-1001)", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "4" + }, + { + "ptr": "HUB/monitor/offsite.go:448-449", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "5" + }, + { + "ptr": "HUB/monitor/offsite.go:453-458", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "5" + }, + { + "ptr": "HUB/monitor/offsite.go:160-162", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "5" + }, + { + "ptr": "HUB/monitor/offsite.go:203-207", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "6" + }, + { + "ptr": "HUB/monitor/offsite.go:204-206", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "6" + }, + { + "ptr": "HUB/monitor/offsite.go:296", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "6" + }, + { + "ptr": "HUB/monitor/offsite.go:636", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "7" + }, + { + "ptr": "AR/08-alarm-ladder.md:383", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "7" + }, + { + "ptr": "HUB/monitor/offsite_r100_test.go:55", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "8" + }, + { + "ptr": "HUB/monitor/offsite_r100_test.go:74", + "file": "OK", + "ident": "IDENT-MISS(HUB/monitor/offsite_r100_test.go:55;)", + "ending": "7. Off-site copy too old, and escrow never finished", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/main.go:944", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Restore-test proof too old", + "q": "1" + }, + { + "ptr": "HUB/monitor/restoretest.go:242", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Restore-test proof too old", + "q": "1" + }, + { + "ptr": "HUB/monitor/restoretest.go:62", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Restore-test proof too old", + "q": "2" + }, + { + "ptr": "HUB/monitor/restoretest.go:68", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Restore-test proof too old", + "q": "2" + }, + { + "ptr": "HUB/monitor/restoretest.go:74", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Restore-test proof too old", + "q": "2" + }, + { + "ptr": "HUB/monitor/restoretest.go:79", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Restore-test proof too old", + "q": "3" + }, + { + "ptr": "HUB/monitor/restoretest.go:252", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Restore-test proof too old", + "q": "3" + }, + { + "ptr": "HUB/monitor/restoretest.go:281-283", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Restore-test proof too old", + "q": "4" + }, + { + "ptr": "HUB/monitor/restoretest.go:300-312", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Restore-test proof too old", + "q": "4" + }, + { + "ptr": "HUB/monitor/restoretest.go:270-278", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Restore-test proof too old", + "q": "5" + }, + { + "ptr": "HUB/monitor/restoretest.go:192-194", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Restore-test proof too old", + "q": "5" + }, + { + "ptr": "HUB/monitor/restoretest.go:326-342", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Restore-test proof too old", + "q": "6" + }, + { + "ptr": "HUB/store/store.go:3994-4005", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Restore-test proof too old", + "q": "6" + }, + { + "ptr": "HUB/monitor/restoretest.go:284-285", + "file": "OK", + "ident": "IDENT-MISS(restore_test_stale)", + "ending": "8. Restore-test proof too old", + "q": "7" + }, + { + "ptr": "HUB/monitor/restoretest.go:205", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Restore-test proof too old", + "q": "7" + }, + { + "ptr": "HUB/monitor/r1001_restart_test.go:147", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Restore-test proof too old", + "q": "8" + }, + { + "ptr": "HUB/offsitekeys/service.go:334", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "1" + }, + { + "ptr": "HUB/offsitekeys/service.go:413", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:636-650", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "1" + }, + { + "ptr": "HUB/offsitekeys/service.go:264", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "2" + }, + { + "ptr": "HUB/offsitekeys/service.go:337", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "2" + }, + { + "ptr": "HUB/offsitekeys/service.go:263", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "2" + }, + { + "ptr": "HUB/offsitekeys/service.go:338", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:644", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "3" + }, + { + "ptr": "HUB/offsitekeys/service.go:419-422", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "3" + }, + { + "ptr": "HUB/offsitekeys/service.go:419-431", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "4" + }, + { + "ptr": "HUB/offsitekeys/service.go:424-426", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "4" + }, + { + "ptr": "HUB/offsitekeys/service.go:364", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "5" + }, + { + "ptr": "HUB/offsitekeys/service.go:212-213", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "5" + }, + { + "ptr": "E/hub/cmd/hub/main.go:636-650", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "5" + }, + { + "ptr": "HUB/offsitekeys/service.go:337", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "6" + }, + { + "ptr": "HUB/offsitekeys/service.go:430-431", + "file": "OK", + "ident": "IDENT-MISS(offsite_window_failed)", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "7" + }, + { + "ptr": "HUB/offsitekeys/service.go:259", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "7" + }, + { + "ptr": "HUB/offsitekeys/service_test.go:99", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "8" + }, + { + "ptr": "HUB/offsitekeys/service_test.go:37", + "file": "OK", + "ident": "IDENT-MISS(HUB/offsitekeys/service_test.go:99;)", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "8" + }, + { + "ptr": "AR/07-backup-architecture.md:256", + "file": "OK", + "ident": "IDENT-MISS(TestWindow_GrantOpenCloseAndDropAlarm)", + "ending": "9. Off-site clean-up window (20 minutes, weekly)", + "q": "8" + }, + { + "ptr": "HUB/offsitekeys/service.go:529", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Delayed deletion of a set-aside off-site copy", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:636-650", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Delayed deletion of a set-aside off-site copy", + "q": "1" + }, + { + "ptr": "HUB/offsitekeys/service.go:448", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Delayed deletion of a set-aside off-site copy", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:598-605", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Delayed deletion of a set-aside off-site copy", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:644", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Delayed deletion of a set-aside off-site copy", + "q": "3" + }, + { + "ptr": "HUB/offsitekeys/service.go:536-538", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Delayed deletion of a set-aside off-site copy", + "q": "3" + }, + { + "ptr": "HUB/offsitekeys/service.go:530", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Delayed deletion of a set-aside off-site copy", + "q": "4" + }, + { + "ptr": "HUB/offsitekeys/service.go:540-547", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Delayed deletion of a set-aside off-site copy", + "q": "4" + }, + { + "ptr": "HUB/offsitekeys/service.go:549", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Delayed deletion of a set-aside off-site copy", + "q": "5" + }, + { + "ptr": "HUB/offsitekeys/service.go:438-441", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Delayed deletion of a set-aside off-site copy", + "q": "5" + }, + { + "ptr": "E/hub/cmd/hub/main.go:598-605", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Delayed deletion of a set-aside off-site copy", + "q": "6" + }, + { + "ptr": "HUB/offsitekeys/service.go:444-447", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Delayed deletion of a set-aside off-site copy", + "q": "7" + }, + { + "ptr": "HUB/offsitekeys/service.go:543-546", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Delayed deletion of a set-aside off-site copy", + "q": "7" + }, + { + "ptr": "HUB/offsitekeys/service_test.go:134", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Delayed deletion of a set-aside off-site copy", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/main.go:651-655", + "file": "OK", + "ident": "IDENT-MISS(scheduleDailyctx,)", + "ending": "11. Off-site key audit (07:10)", + "q": "1" + }, + { + "ptr": "HUB/offsitekeys/service.go:195", + "file": "OK", + "ident": "IDENT-MISS(E/hub/cmd/hub/main.go:651-655)", + "ending": "11. Off-site key audit (07:10)", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:651", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Off-site key audit (07:10)", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1072", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Off-site key audit (07:10)", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:652", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Off-site key audit (07:10)", + "q": "3" + }, + { + "ptr": "HUB/offsitekeys/service.go:202-215", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Off-site key audit (07:10)", + "q": "3" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1095-1112", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Off-site key audit (07:10)", + "q": "4" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1095-1112", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Off-site key audit (07:10)", + "q": "5" + }, + { + "ptr": "HUB/offsitekeys/service.go:212-213", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Off-site key audit (07:10)", + "q": "5" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1095-1112", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Off-site key audit (07:10)", + "q": "6" + }, + { + "ptr": "HUB/offsitekeys/service.go:192-194", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Off-site key audit (07:10)", + "q": "7" + }, + { + "ptr": "HUB/offsitekeys/offsitekeys_test.go:158", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Off-site key audit (07:10)", + "q": "8" + }, + { + "ptr": "AR/05-hub-architecture.md:642", + "file": "OK", + "ident": "IDENT-MISS(TestWindow_PrependAuditClose)", + "ending": "11. Off-site key audit (07:10)", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/main.go:945", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "1" + }, + { + "ptr": "HUB/offsiteheal/reconciler.go:165-177", + "file": "OK", + "ident": "IDENT-MISS(E/hub/cmd/hub/main.go:945)", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:846-854", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "1" + }, + { + "ptr": "HUB/monitor/offsite_delivery.go:38", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "2" + }, + { + "ptr": "HUB/monitor/offsite_delivery.go:40-41", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "2" + }, + { + "ptr": "HUB/offsiteheal/reconciler.go:74", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "2" + }, + { + "ptr": "HUB/offsiteheal/reconciler.go:70", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "2" + }, + { + "ptr": "HUB/monitor/offsite_delivery.go:275", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "3" + }, + { + "ptr": "HUB/monitor/offsite_delivery.go:72", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "3" + }, + { + "ptr": "HUB/offsiteheal/reconciler.go:183", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "3" + }, + { + "ptr": "HUB/monitor/offsite_delivery.go:39", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "4" + }, + { + "ptr": "HUB/monitor/offsite_delivery.go:28-31", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "5" + }, + { + "ptr": "HUB/monitor/offsite_delivery.go:97-99", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "5" + }, + { + "ptr": "HUB/monitor/offsite_delivery.go:93", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "6" + }, + { + "ptr": "HUB/offsiteheal/reconciler.go:61-69", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "6" + }, + { + "ptr": "HUB/monitor/offsite_delivery.go:34-35", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "7" + }, + { + "ptr": "HUB/offsiteheal/reconciler.go:57-58", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "7" + }, + { + "ptr": "HUB/offsiteheal/reconciler.go:61-74", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "8" + }, + { + "ptr": "AR/06-offsite-connectivity.md:386", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "8" + }, + { + "ptr": "HUB/monitor/offsite_delivery.go:1", + "file": "OK", + "ident": "IDENT-MISS(func)", + "ending": "12. Off-site credential stuck and its self-heal", + "q": "8" + }, + { + "ptr": "HUB/pbsdrheal/reconciler.go:153-165", + "file": "OK", + "ident": "IDENT-MISS(pbsdrheal)", + "ending": "13. PBS whole-box credential self-heal", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:829-837", + "file": "OK", + "ident": "IDENT-OK", + "ending": "13. PBS whole-box credential self-heal", + "q": "1" + }, + { + "ptr": "HUB/pbsdrheal/reconciler.go:130", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. PBS whole-box credential self-heal", + "q": "2" + }, + { + "ptr": "HUB/pbsdrheal/reconciler.go:160", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. PBS whole-box credential self-heal", + "q": "2" + }, + { + "ptr": "HUB/pbsdrheal/reconciler.go:131", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. PBS whole-box credential self-heal", + "q": "2" + }, + { + "ptr": "HUB/pbsdrheal/reconciler.go:63", + "file": "OK", + "ident": "IDENT-OK", + "ending": "13. PBS whole-box credential self-heal", + "q": "2" + }, + { + "ptr": "HUB/pbsdrheal/reconciler.go:163", + "file": "OK", + "ident": "IDENT-MISS(reconcileOncectx)", + "ending": "13. PBS whole-box credential self-heal", + "q": "3" + }, + { + "ptr": "HUB/pbsdrheal/reconciler.go:170", + "file": "OK", + "ident": "IDENT-MISS(reconcileOncectx)", + "ending": "13. PBS whole-box credential self-heal", + "q": "3" + }, + { + "ptr": "HUB/pbsdrheal/reconciler.go:134-135", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. PBS whole-box credential self-heal", + "q": "4" + }, + { + "ptr": "HUB/pbsdrheal/reconciler.go:171-174", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. PBS whole-box credential self-heal", + "q": "5" + }, + { + "ptr": "HUB/pbsdrheal/reconciler.go:50-54", + "file": "OK", + "ident": "IDENT-OK", + "ending": "13. PBS whole-box credential self-heal", + "q": "7" + }, + { + "ptr": "HUB/pbsdrheal/reconciler.go:57-63", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. PBS whole-box credential self-heal", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/main.go:752", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. WireGuard peer list push to ep0", + "q": "1" + }, + { + "ptr": "HUB/wgsync/reconciler.go:90-106", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. WireGuard peer list push to ep0", + "q": "1" + }, + { + "ptr": "HUB/wgsync/reconciler.go:41", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. WireGuard peer list push to ep0", + "q": "2" + }, + { + "ptr": "HUB/wgsync/reconciler.go:48-53", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. WireGuard peer list push to ep0", + "q": "2" + }, + { + "ptr": "HUB/wgsync/reconciler.go:100", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. WireGuard peer list push to ep0", + "q": "3" + }, + { + "ptr": "HUB/wgsync/reconciler.go:71-73", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. WireGuard peer list push to ep0", + "q": "3" + }, + { + "ptr": "HUB/wgsync/reconciler.go:101-103", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. WireGuard peer list push to ep0", + "q": "4" + }, + { + "ptr": "HUB/wgsync/reconciler.go:1", + "file": "OK", + "ident": "IDENT-MISS(SaveEvent)", + "ending": "14. WireGuard peer list push to ep0", + "q": "4" + }, + { + "ptr": "HUB/wgsync/reconciler.go:46-47", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. WireGuard peer list push to ep0", + "q": "5" + }, + { + "ptr": "HUB/wgsync/reconciler.go:20-23", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. WireGuard peer list push to ep0", + "q": "5" + }, + { + "ptr": "HUB/wgsync/reconciler.go:102", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. WireGuard peer list push to ep0", + "q": "7" + }, + { + "ptr": "HUB/wgsync/reconciler_test.go:119", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. WireGuard peer list push to ep0", + "q": "8" + }, + { + "ptr": "HUB/wgsync/reconciler_test.go:82", + "file": "OK", + "ident": "IDENT-MISS(HUB/wgsync/reconciler_test.go:119;)", + "ending": "14. WireGuard peer list push to ep0", + "q": "8" + }, + { + "ptr": "HUB/api/host_tunnel.go:61", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Box clock ran ahead: tunnel alarm", + "q": "1" + }, + { + "ptr": "HUB/api/handler.go:1010", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Box clock ran ahead: tunnel alarm", + "q": "1" + }, + { + "ptr": "HUB/api/host_tunnel.go:41", + "file": "OK", + "ident": "IDENT-MISS(handshake_future_s)", + "ending": "15. Box clock ran ahead: tunnel alarm", + "q": "2" + }, + { + "ptr": "HUB/api/host_tunnel.go:23", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Box clock ran ahead: tunnel alarm", + "q": "2" + }, + { + "ptr": "HUB/api/host_tunnel.go:66", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. Box clock ran ahead: tunnel alarm", + "q": "3" + }, + { + "ptr": "HUB/api/host_tunnel.go:62-64", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. Box clock ran ahead: tunnel alarm", + "q": "4" + }, + { + "ptr": "HUB/api/host_tunnel.go:71-83", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. Box clock ran ahead: tunnel alarm", + "q": "5" + }, + { + "ptr": "HUB/api/host_tunnel.go:31", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. Box clock ran ahead: tunnel alarm", + "q": "6" + }, + { + "ptr": "HUB/api/host_tunnel.go:14-17", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. Box clock ran ahead: tunnel alarm", + "q": "6" + }, + { + "ptr": "AR/05-hub-architecture.md:489-490", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. Box clock ran ahead: tunnel alarm", + "q": "6" + }, + { + "ptr": "HUB/api/host_tunnel.go:12-14", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Box clock ran ahead: tunnel alarm", + "q": "7" + }, + { + "ptr": "HUB/api/host_tunnel_test.go:83", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Box clock ran ahead: tunnel alarm", + "q": "8" + }, + { + "ptr": "AR/05-hub-architecture.md:482-490", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Box clock ran ahead: tunnel alarm", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1069-1093", + "file": "OK", + "ident": "IDENT-OK", + "ending": "16. The daily scheduler and the clock change", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1095-1112", + "file": "OK", + "ident": "IDENT-OK", + "ending": "16. The daily scheduler and the clock change", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1072", + "file": "OK", + "ident": "IDENT-OK", + "ending": "16. The daily scheduler and the clock change", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1074", + "file": "OK", + "ident": "IDENT-OK", + "ending": "16. The daily scheduler and the clock change", + "q": "2" + }, + { + "ptr": "E/manifests/hub.yaml:140-141", + "file": "OK", + "ident": "IDENT-OK", + "ending": "16. The daily scheduler and the clock change", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1088", + "file": "OK", + "ident": "IDENT-MISS(Afterdelay)", + "ending": "16. The daily scheduler and the clock change", + "q": "3" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1090", + "file": "OK", + "ident": "IDENT-MISS(Afterdelay)", + "ending": "16. The daily scheduler and the clock change", + "q": "3" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1115-1121", + "file": "OK", + "ident": "IDENT-OK", + "ending": "16. The daily scheduler and the clock change", + "q": "4" + }, + { + "ptr": "AR/05-hub-architecture.md:642-645", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. The daily scheduler and the clock change", + "q": "5" + }, + { + "ptr": "AR/05-hub-architecture.md:640-646", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. The daily scheduler and the clock change", + "q": "6" + }, + { + "ptr": "E/hub/cmd/hub/r1183_dst_schedule_test.go:31", + "file": "OK", + "ident": "IDENT-OK", + "ending": "16. The daily scheduler and the clock change", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/r1183_dst_schedule_test.go:56", + "file": "OK", + "ident": "IDENT-OK", + "ending": "16. The daily scheduler and the clock change", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/main.go:967-983", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. Hub database snapshot (four slots)", + "q": "1" + }, + { + "ptr": "HUB/dbsnap/dbsnap.go:72-117", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. Hub database snapshot (four slots)", + "q": "1" + }, + { + "ptr": "HUB/dbsnap/dbsnap.go:32", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. Hub database snapshot (four slots)", + "q": "2" + }, + { + "ptr": "HUB/dbsnap/dbsnap.go:27", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. Hub database snapshot (four slots)", + "q": "2" + }, + { + "ptr": "HUB/dbsnap/dbsnap.go:89", + "file": "OK", + "ident": "IDENT-MISS(Keep)", + "ending": "17. Hub database snapshot (four slots)", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:972", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Hub database snapshot (four slots)", + "q": "3" + }, + { + "ptr": "HUB/dbsnap/dbsnap.go:91-104", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Hub database snapshot (four slots)", + "q": "3" + }, + { + "ptr": "HUB/dbsnap/dbsnap.go:113-114", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Hub database snapshot (four slots)", + "q": "3" + }, + { + "ptr": "E/hub/cmd/hub/main.go:978-980", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. Hub database snapshot (four slots)", + "q": "4" + }, + { + "ptr": "HUB/dbsnap/dbsnap.go:148-159", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. Hub database snapshot (four slots)", + "q": "4" + }, + { + "ptr": "HUB/dbsnap/dbsnap.go:73-80", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. Hub database snapshot (four slots)", + "q": "5" + }, + { + "ptr": "AR/05-hub-architecture.md:618-620", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Hub database snapshot (four slots)", + "q": "6" + }, + { + "ptr": "HUB/dbsnap/dbsnap_test.go:214", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. Hub database snapshot (four slots)", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/r173_wiring_test.go:14", + "file": "OK", + "ident": "IDENT-MISS(HUB/dbsnap/dbsnap_test.go:214;)", + "ending": "17. Hub database snapshot (four slots)", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1123-1150", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Daily prune of old hub rows (04:30)", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:860-865", + "file": "OK", + "ident": "IDENT-MISS(E/hub/cmd/hub/main.go:1123-1150,)", + "ending": "18. Daily prune of old hub rows (04:30)", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1026", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Daily prune of old hub rows (04:30)", + "q": "2" + }, + { + "ptr": "E/manifests/hub.yaml:83", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Daily prune of old hub rows (04:30)", + "q": "2" + }, + { + "ptr": "E/manifests/hub.yaml:82", + "file": "OK", + "ident": "IDENT-MISS(E/hub/cmd/hub/main.go:1026,)", + "ending": "18. Daily prune of old hub rows (04:30)", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1134", + "file": "OK", + "ident": "IDENT-MISS(max_days)", + "ending": "18. Daily prune of old hub rows (04:30)", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1139", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Daily prune of old hub rows (04:30)", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1145", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Daily prune of old hub rows (04:30)", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1123-1150", + "file": "OK", + "ident": "IDENT-MISS(Since)", + "ending": "18. Daily prune of old hub rows (04:30)", + "q": "3" + }, + { + "ptr": "E/hub/cmd/hub/main.go:860-865", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Daily prune of old hub rows (04:30)", + "q": "4" + }, + { + "ptr": "HUB/store/store.go:1607", + "file": "OK", + "ident": "IDENT-MISS(UTC)", + "ending": "18. Daily prune of old hub rows (04:30)", + "q": "6" + }, + { + "ptr": "HUB/store/store.go:129", + "file": "OK", + "ident": "IDENT-MISS(UTC)", + "ending": "18. Daily prune of old hub rows (04:30)", + "q": "6" + }, + { + "ptr": "HUB/store/store.go:2873", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Daily prune of old hub rows (04:30)", + "q": "6" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1123-1150", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Daily prune of old hub rows (04:30)", + "q": "8" + }, + { + "ptr": "HUB/store/store.go:1606", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Daily prune of old hub rows (04:30)", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/main.go:937-946", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. The other checks on the 60-second sweep", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:927-965", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. The other checks on the 60-second sweep", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:935-962", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. The other checks on the 60-second sweep", + "q": "2" + }, + { + "ptr": "HUB/monitor/host_capability.go:72", + "file": "OK", + "ident": "IDENT-OK", + "ending": "19. The other checks on the 60-second sweep", + "q": "2" + }, + { + "ptr": "E/hub/cmd/hub/main.go:935-962", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. The other checks on the 60-second sweep", + "q": "3" + }, + { + "ptr": "HUB/monitor/host_capability.go:172", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. The other checks on the 60-second sweep", + "q": "6" + }, + { + "ptr": "AR/08-alarm-ladder.md:305", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. The other checks on the 60-second sweep", + "q": "7" + }, + { + "ptr": "E/hub/cmd/hub/main.go:927-965", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. The other checks on the 60-second sweep", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/main.go:947", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Reminder about a held app", + "q": "1" + }, + { + "ptr": "HUB/monitor/held_apps.go:102", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Reminder about a held app", + "q": "1" + }, + { + "ptr": "HUB/monitor/held_apps.go:36", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Reminder about a held app", + "q": "2" + }, + { + "ptr": "HUB/monitor/held_apps.go:40-48", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Reminder about a held app", + "q": "2" + }, + { + "ptr": "HUB/monitor/held_apps.go:34", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Reminder about a held app", + "q": "2" + }, + { + "ptr": "HUB/monitor/held_apps.go:114", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Reminder about a held app", + "q": "3" + }, + { + "ptr": "HUB/monitor/held_apps.go:141-145", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Reminder about a held app", + "q": "4" + }, + { + "ptr": "HUB/monitor/held_apps.go:125-129", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Reminder about a held app", + "q": "5" + }, + { + "ptr": "HUB/monitor/held_apps.go:130-134", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Reminder about a held app", + "q": "6" + }, + { + "ptr": "HUB/monitor/held_apps.go:22", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Reminder about a held app", + "q": "6" + }, + { + "ptr": "HUB/monitor/held_apps.go:157-163", + "file": "OK", + "ident": "IDENT-MISS(app_held_reminder)", + "ending": "20. Reminder about a held app", + "q": "7" + }, + { + "ptr": "HUB/monitor/held_apps.go:31", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Reminder about a held app", + "q": "8" + }, + { + "ptr": "AR/08-alarm-ladder.md:655", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Reminder about a held app", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/main.go:948-950", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. \"A golden is published but not vouched\"", + "q": "1" + }, + { + "ptr": "HUB/monitor/golden_vouch.go:54", + "file": "OK", + "ident": "IDENT-MISS(check)", + "ending": "21. \"A golden is published but not vouched\"", + "q": "1" + }, + { + "ptr": "HUB/monitor/golden_vouch.go:42-43", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. \"A golden is published but not vouched\"", + "q": "2" + }, + { + "ptr": "HUB/monitor/golden_vouch.go:44", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. \"A golden is published but not vouched\"", + "q": "3" + }, + { + "ptr": "HUB/monitor/golden_vouch.go:62-64", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. \"A golden is published but not vouched\"", + "q": "3" + }, + { + "ptr": "HUB/monitor/golden_vouch.go:34-37", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. \"A golden is published but not vouched\"", + "q": "4" + }, + { + "ptr": "HUB/monitor/golden_vouch.go:23", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. \"A golden is published but not vouched\"", + "q": "4" + }, + { + "ptr": "HUB/monitor/golden_vouch.go:65-67", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. \"A golden is published but not vouched\"", + "q": "5" + }, + { + "ptr": "HUB/monitor/golden_vouch.go:95-96", + "file": "OK", + "ident": "IDENT-MISS(golden_not_vouched)", + "ending": "21. \"A golden is published but not vouched\"", + "q": "7" + }, + { + "ptr": "HUB/monitor/golden_vouch.go:24", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. \"A golden is published but not vouched\"", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/main.go:951-956", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Shared storage boxes: fill and reachability", + "q": "1" + }, + { + "ptr": "HUB/monitor/offsite_box.go:72", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Shared storage boxes: fill and reachability", + "q": "2" + }, + { + "ptr": "HUB/monitor/offsite_box.go:132", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Shared storage boxes: fill and reachability", + "q": "2" + }, + { + "ptr": "HUB/monitor/pbsdr_box.go:130", + "file": "OK", + "ident": "NO-IDENT", + "ending": "22. Shared storage boxes: fill and reachability", + "q": "2" + }, + { + "ptr": "HUB/monitor/offsite_box.go:67-68", + "file": "OK", + "ident": "NO-IDENT", + "ending": "22. Shared storage boxes: fill and reachability", + "q": "2" + }, + { + "ptr": "HUB/hetznerapi/hetznerapi.go:143-145", + "file": "OK", + "ident": "NO-IDENT", + "ending": "22. Shared storage boxes: fill and reachability", + "q": "3" + }, + { + "ptr": "HUB/monitor/pbsdr_box.go:80-82", + "file": "OK", + "ident": "NO-IDENT", + "ending": "22. Shared storage boxes: fill and reachability", + "q": "4" + }, + { + "ptr": "HUB/monitor/offsite_box.go:132", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Shared storage boxes: fill and reachability", + "q": "5" + }, + { + "ptr": "HUB/monitor/offsite_box.go:266-330", + "file": "OK", + "ident": "NO-IDENT", + "ending": "22. Shared storage boxes: fill and reachability", + "q": "7" + }, + { + "ptr": "HUB/monitor/offsite_box.go:70-72", + "file": "OK", + "ident": "NO-IDENT", + "ending": "22. Shared storage boxes: fill and reachability", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/main.go:958-962", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Purges: log bundles, access log, sessions", + "q": "1" + }, + { + "ptr": "HUB/accesslog/accesslog.go:192", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Purges: log bundles, access log, sessions", + "q": "1" + }, + { + "ptr": "HUB/web/server.go:199-217", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Purges: log bundles, access log, sessions", + "q": "1" + }, + { + "ptr": "HUB/store/logbundle.go:29", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Purges: log bundles, access log, sessions", + "q": "2" + }, + { + "ptr": "HUB/accesslog/accesslog.go:166-184", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Purges: log bundles, access log, sessions", + "q": "2" + }, + { + "ptr": "HUB/accesslog/accesslog.go:44", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Purges: log bundles, access log, sessions", + "q": "2" + }, + { + "ptr": "HUB/web/server.go:200", + "file": "OK", + "ident": "IDENT-MISS(DefaultKeep)", + "ending": "23. Purges: log bundles, access log, sessions", + "q": "2" + }, + { + "ptr": "HUB/web/server.go:949", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Purges: log bundles, access log, sessions", + "q": "2" + }, + { + "ptr": "HUB/accesslog/accesslog.go:196", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Purges: log bundles, access log, sessions", + "q": "3" + }, + { + "ptr": "HUB/web/server.go:207-213", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Purges: log bundles, access log, sessions", + "q": "4" + }, + { + "ptr": "HUB/accesslog/accesslog.go:182-184", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Purges: log bundles, access log, sessions", + "q": "4" + }, + { + "ptr": "HUB/accesslog/accesslog.go:191", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Purges: log bundles, access log, sessions", + "q": "5" + }, + { + "ptr": "HUB/accesslog/accesslog.go:166", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Purges: log bundles, access log, sessions", + "q": "6" + }, + { + "ptr": "AR/05-hub-architecture.md:634-636", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Purges: log bundles, access log, sessions", + "q": "6" + }, + { + "ptr": "HUB/accesslog/accesslog_test.go:135", + "file": "OK", + "ident": "IDENT-OK", + "ending": "23. Purges: log bundles, access log, sessions", + "q": "8" + }, + { + "ptr": "HUB/web/selfbind_mint.go:20", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. Links and tokens that expire by themselves", + "q": "1" + }, + { + "ptr": "HUB/web/tokens.go:95-96", + "file": "OK", + "ident": "IDENT-MISS(selfBindTTL)", + "ending": "24. Links and tokens that expire by themselves", + "q": "1" + }, + { + "ptr": "HUB/api/dr.go:21-22", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. Links and tokens that expire by themselves", + "q": "1" + }, + { + "ptr": "HUB/web/selfbind_mint.go:77", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. Links and tokens that expire by themselves", + "q": "2" + }, + { + "ptr": "HUB/web/selfbind.go:382", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. Links and tokens that expire by themselves", + "q": "2" + }, + { + "ptr": "HUB/web/tokens.go:180", + "file": "OK", + "ident": "IDENT-MISS(bindResendEvery)", + "ending": "24. Links and tokens that expire by themselves", + "q": "2" + }, + { + "ptr": "HUB/store/opactions.go:62", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. Links and tokens that expire by themselves", + "q": "2" + }, + { + "ptr": "HUB/web/selfbind.go:206", + "file": "OK", + "ident": "IDENT-MISS(selfBindTTL|Expired)", + "ending": "24. Links and tokens that expire by themselves", + "q": "3" + }, + { + "ptr": "HUB/web/selfbind_mint.go:18-19", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. Links and tokens that expire by themselves", + "q": "4" + }, + { + "ptr": "HUB/store/selfbind.go:39", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. Links and tokens that expire by themselves", + "q": "5" + }, + { + "ptr": "HUB/store/selfbind.go:177", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. Links and tokens that expire by themselves", + "q": "5" + }, + { + "ptr": "HUB/store/selfbind.go:177", + "file": "OK", + "ident": "IDENT-MISS(datetime'now')", + "ending": "24. Links and tokens that expire by themselves", + "q": "6" + }, + { + "ptr": "HUB/web/tokens.go:96", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. Links and tokens that expire by themselves", + "q": "7" + }, + { + "ptr": "HUB/store/selfbind.go:38-46", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. Links and tokens that expire by themselves", + "q": "8" + }, + { + "ptr": "HUB/web/tokens.go:1", + "file": "OK", + "ident": "IDENT-MISS(expires)", + "ending": "24. Links and tokens that expire by themselves", + "q": "8" + }, + { + "ptr": "HUB/web/version.go:43-55", + "file": "OK", + "ident": "IDENT-OK", + "ending": "25. Version check and catalog template fetch", + "q": "1" + }, + { + "ptr": "HUB/web/templatefetcher.go:41-53", + "file": "OK", + "ident": "IDENT-OK", + "ending": "25. Version check and catalog template fetch", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:711-722", + "file": "OK", + "ident": "IDENT-OK", + "ending": "25. Version check and catalog template fetch", + "q": "1" + }, + { + "ptr": "E/hub/cmd/hub/main.go:327-335", + "file": "OK", + "ident": "NO-IDENT", + "ending": "25. Version check and catalog template fetch", + "q": "1" + }, + { + "ptr": "E/manifests/hub.yaml:104", + "file": "OK", + "ident": "IDENT-OK", + "ending": "25. Version check and catalog template fetch", + "q": "2" + }, + { + "ptr": "E/manifests/hub.yaml:105", + "file": "OK", + "ident": "IDENT-OK", + "ending": "25. Version check and catalog template fetch", + "q": "2" + }, + { + "ptr": "HUB/web/version.go:44", + "file": "OK", + "ident": "NO-IDENT", + "ending": "25. Version check and catalog template fetch", + "q": "4" + }, + { + "ptr": "HUB/web/templatefetcher.go:42", + "file": "OK", + "ident": "NO-IDENT", + "ending": "25. Version check and catalog template fetch", + "q": "4" + }, + { + "ptr": "HUB/web/version.go:62-65", + "file": "OK", + "ident": "IDENT-OK", + "ending": "25. Version check and catalog template fetch", + "q": "7" + }, + { + "ptr": "HUB/web/version.go:41-55", + "file": "OK", + "ident": "NO-IDENT", + "ending": "25. Version check and catalog template fetch", + "q": "8" + }, + { + "ptr": "E/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10", + "file": "OK", + "ident": "IDENT-MISS(felhom-hub-db-backup.timer)", + "ending": "26. DooPlex: push of the hub database snapshot off-site", + "q": "1" + }, + { + "ptr": "RB/RUNBOOK-hub-db-offsite-backup.md:107-108", + "file": "OK", + "ident": "NO-IDENT", + "ending": "26. DooPlex: push of the hub database snapshot off-site", + "q": "1" + }, + { + "ptr": "E/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10", + "file": "OK", + "ident": "NO-IDENT", + "ending": "26. DooPlex: push of the hub database snapshot off-site", + "q": "2" + }, + { + "ptr": "E/scripts/hub-db-backup/felhom-hub-db-backup.timer:12", + "file": "OK", + "ident": "IDENT-OK", + "ending": "26. DooPlex: push of the hub database snapshot off-site", + "q": "2" + }, + { + "ptr": "E/scripts/hub-db-backup/felhom-hub-db-backup.service:13", + "file": "OK", + "ident": "IDENT-OK", + "ending": "26. DooPlex: push of the hub database snapshot off-site", + "q": "3" + }, + { + "ptr": "E/scripts/hub-db-backup/felhom-hub-db-backup.timer:11", + "file": "OK", + "ident": "IDENT-OK", + "ending": "26. DooPlex: push of the hub database snapshot off-site", + "q": "4" + }, + { + "ptr": "E/scripts/hub-db-backup/test_hub_db_backup.py:309-316", + "file": "OK", + "ident": "NO-IDENT", + "ending": "26. DooPlex: push of the hub database snapshot off-site", + "q": "6" + }, + { + "ptr": "E/scripts/hub-db-backup/felhom-hub-db-backup.service:6", + "file": "OK", + "ident": "IDENT-OK", + "ending": "26. DooPlex: push of the hub database snapshot off-site", + "q": "7" + }, + { + "ptr": "RB/RUNBOOK-hub-db-offsite-backup.md:100", + "file": "OK", + "ident": "IDENT-OK", + "ending": "26. DooPlex: push of the hub database snapshot off-site", + "q": "7" + }, + { + "ptr": "RB/RUNBOOK-hub-db-offsite-backup.md:153", + "file": "OK", + "ident": "NO-IDENT", + "ending": "26. DooPlex: push of the hub database snapshot off-site", + "q": "7" + }, + { + "ptr": "E/scripts/hub-db-backup/test_hub_db_backup.py:309-316", + "file": "OK", + "ident": "NO-IDENT", + "ending": "26. DooPlex: push of the hub database snapshot off-site", + "q": "8" + }, + { + "ptr": "RB/RUNBOOK-hub-db-offsite-backup.md:107-108", + "file": "OK", + "ident": "NO-IDENT", + "ending": "26. DooPlex: push of the hub database snapshot off-site", + "q": "8" + }, + { + "ptr": "E/scripts/hub-db-backup/felhom-hub-db-restore-test.timer:6", + "file": "OK", + "ident": "IDENT-MISS(felhom-hub-db-restore-test.timer)", + "ending": "27. DooPlex: Sunday restore tests", + "q": "1" + }, + { + "ptr": "E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test.timer:6", + "file": "OK", + "ident": "IDENT-MISS(felhom-dooplex-offsite-restore-test.timer)", + "ending": "27. DooPlex: Sunday restore tests", + "q": "1" + }, + { + "ptr": "E/scripts/hub-db-backup/felhom-hub-db-restore-test.timer:7", + "file": "OK", + "ident": "IDENT-OK", + "ending": "27. DooPlex: Sunday restore tests", + "q": "2" + }, + { + "ptr": "E/scripts/hub-db-backup/felhom-hub-db-restore-test.service:13", + "file": "OK", + "ident": "NO-IDENT", + "ending": "27. DooPlex: Sunday restore tests", + "q": "3" + }, + { + "ptr": "E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test.service:13", + "file": "OK", + "ident": "NO-IDENT", + "ending": "27. DooPlex: Sunday restore tests", + "q": "3" + }, + { + "ptr": "RB/RUNBOOK-hub-db-offsite-backup.md:158", + "file": "OK", + "ident": "IDENT-OK", + "ending": "27. DooPlex: Sunday restore tests", + "q": "7" + }, + { + "ptr": "RB/RUNBOOK-hub-db-offsite-backup.md:152-161", + "file": "OK", + "ident": "NO-IDENT", + "ending": "27. DooPlex: Sunday restore tests", + "q": "8" + }, + { + "ptr": "E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:6", + "file": "OK", + "ident": "IDENT-MISS(felhom-dooplex-offsite.timer)", + "ending": "28. DooPlex: nightly off-site push and release copy", + "q": "1" + }, + { + "ptr": "E/scripts/release-copy/felhom-release-copy.timer:7", + "file": "OK", + "ident": "IDENT-MISS(felhom-release-copy.timer)", + "ending": "28. DooPlex: nightly off-site push and release copy", + "q": "1" + }, + { + "ptr": "E/scripts/release-copy/felhom-release-copy.timer:6", + "file": "OK", + "ident": "NO-IDENT", + "ending": "28. DooPlex: nightly off-site push and release copy", + "q": "2" + }, + { + "ptr": "E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:8", + "file": "OK", + "ident": "NO-IDENT", + "ending": "28. DooPlex: nightly off-site push and release copy", + "q": "2" + }, + { + "ptr": "E/scripts/release-copy/felhom-release-copy.timer:9", + "file": "OK", + "ident": "NO-IDENT", + "ending": "28. DooPlex: nightly off-site push and release copy", + "q": "2" + }, + { + "ptr": "E/scripts/dooplex-offsite/felhom-dooplex-offsite.service:13", + "file": "OK", + "ident": "NO-IDENT", + "ending": "28. DooPlex: nightly off-site push and release copy", + "q": "3" + }, + { + "ptr": "E/scripts/release-copy/felhom-release-copy.service:12", + "file": "OK", + "ident": "NO-IDENT", + "ending": "28. DooPlex: nightly off-site push and release copy", + "q": "3" + }, + { + "ptr": "E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:7", + "file": "OK", + "ident": "IDENT-OK", + "ending": "28. DooPlex: nightly off-site push and release copy", + "q": "4" + }, + { + "ptr": "E/scripts/release-copy/felhom-release-copy.timer:8", + "file": "OK", + "ident": "IDENT-OK", + "ending": "28. DooPlex: nightly off-site push and release copy", + "q": "4" + }, + { + "ptr": "E/scripts/dooplex-offsite/felhom-dooplex-offsite.service:6", + "file": "OK", + "ident": "IDENT-OK", + "ending": "28. DooPlex: nightly off-site push and release copy", + "q": "7" + }, + { + "ptr": "E/scripts/release-copy/felhom-release-copy.service:6", + "file": "OK", + "ident": "IDENT-OK", + "ending": "28. DooPlex: nightly off-site push and release copy", + "q": "7" + }, + { + "ptr": "E/scripts/release-copy/install.sh:4", + "file": "OK", + "ident": "NO-IDENT", + "ending": "28. DooPlex: nightly off-site push and release copy", + "q": "8" + }, + { + "ptr": "RB/ep0-datastore-copy.md:15-18", + "file": "OK", + "ident": "NO-IDENT", + "ending": "29. DooPlex and ep0: PBS copy, prune, verify, clean-up", + "q": "1" + }, + { + "ptr": "E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:5", + "file": "OK", + "ident": "IDENT-MISS(felhom-ep0-copy-gc.timer)", + "ending": "29. DooPlex and ep0: PBS copy, prune, verify, clean-up", + "q": "1" + }, + { + "ptr": "RB/ep0-datastore-copy.md:15", + "file": "OK", + "ident": "NO-IDENT", + "ending": "29. DooPlex and ep0: PBS copy, prune, verify, clean-up", + "q": "2" + }, + { + "ptr": "RB/ep0-datastore-copy.md:17", + "file": "OK", + "ident": "NO-IDENT", + "ending": "29. DooPlex and ep0: PBS copy, prune, verify, clean-up", + "q": "2" + }, + { + "ptr": "E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:5", + "file": "OK", + "ident": "NO-IDENT", + "ending": "29. DooPlex and ep0: PBS copy, prune, verify, clean-up", + "q": "2" + }, + { + "ptr": "RB/ep0-datastore-copy.md:16", + "file": "OK", + "ident": "NO-IDENT", + "ending": "29. DooPlex and ep0: PBS copy, prune, verify, clean-up", + "q": "2" + }, + { + "ptr": "RB/ep0-datastore-copy.md:18", + "file": "OK", + "ident": "NO-IDENT", + "ending": "29. DooPlex and ep0: PBS copy, prune, verify, clean-up", + "q": "2" + }, + { + "ptr": "RB/RUNBOOK-hub-db-offsite-backup.md:101", + "file": "OK", + "ident": "NO-IDENT", + "ending": "29. DooPlex and ep0: PBS copy, prune, verify, clean-up", + "q": "2" + }, + { + "ptr": "E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:6", + "file": "OK", + "ident": "IDENT-OK", + "ending": "29. DooPlex and ep0: PBS copy, prune, verify, clean-up", + "q": "4" + }, + { + "ptr": "E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:2", + "file": "OK", + "ident": "NO-IDENT", + "ending": "29. DooPlex and ep0: PBS copy, prune, verify, clean-up", + "q": "5" + }, + { + "ptr": "RB/ep0-datastore-copy.md:1", + "file": "OK", + "ident": "IDENT-MISS(timedatectl|Europe/Budapest)", + "ending": "29. DooPlex and ep0: PBS copy, prune, verify, clean-up", + "q": "6" + }, + { + "ptr": "RB/ep0-datastore-copy.md:19", + "file": "OK", + "ident": "NO-IDENT", + "ending": "29. DooPlex and ep0: PBS copy, prune, verify, clean-up", + "q": "7" + }, + { + "ptr": "E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.service:9", + "file": "OK", + "ident": "IDENT-MISS(OnFailure|TimeoutStartSec)", + "ending": "29. DooPlex and ep0: PBS copy, prune, verify, clean-up", + "q": "7" + }, + { + "ptr": "RB/ep0-datastore-copy.md:15-19", + "file": "OK", + "ident": "NO-IDENT", + "ending": "29. DooPlex and ep0: PBS copy, prune, verify, clean-up", + "q": "8" + }, + { + "ptr": "E/.gitea/workflows/gates.yml:26-27", + "file": "OK", + "ident": "IDENT-MISS(gates)", + "ending": "30. Nightly CI: gates and the security watch", + "q": "1" + }, + { + "ptr": "E/.gitea/workflows/security-watch.yml:21-22", + "file": "OK", + "ident": "IDENT-OK", + "ending": "30. Nightly CI: gates and the security watch", + "q": "1" + }, + { + "ptr": "E/.gitea/workflows/gates.yml:21-23", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "2" + }, + { + "ptr": "AR/08-alarm-ladder.md:710", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "2" + }, + { + "ptr": "E/.gitea/workflows/gates.yml:53", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "3" + }, + { + "ptr": "E/.gitea/workflows/gates.yml:20", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "3" + }, + { + "ptr": "E/.gitea/workflows/security-watch.yml:33", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "3" + }, + { + "ptr": "AR/08-alarm-ladder.md:717", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "4" + }, + { + "ptr": "E/.gitea/workflows/security-watch.yml:32", + "file": "OK", + "ident": "IDENT-OK", + "ending": "30. Nightly CI: gates and the security watch", + "q": "5" + }, + { + "ptr": "E/.gitea/workflows/security-watch.yml:95-106", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "5" + }, + { + "ptr": "AR/08-alarm-ladder.md:716-720", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "6" + }, + { + "ptr": "E/.gitea/workflows/security-watch.yml:108-131", + "file": "DEAD", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "7" + }, + { + "ptr": "E/.gitea/workflows/gates.yml:245", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "7" + }, + { + "ptr": "E/.gitea/workflows/gates.yml:50-52", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "7" + }, + { + "ptr": "AR/08-alarm-ladder.md:708-722", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "HUB/store/deliberateoff.go:18", + "file": "OK", + "ident": "IDENT-OK", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "HUB/api/deliberate_off.go:30", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "HUB/api/wait.go:17", + "file": "OK", + "ident": "IDENT-OK", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "HUB/intent/presence.go:21", + "file": "OK", + "ident": "IDENT-OK", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "HUB/hetznerapi/hetznerapi.go:143-145", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "HUB/offsite/offsite.go:112", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "HUB/web/pbsdr.go:571", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "HUB/web/hosts.go:770", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "HUB/store/opactions.go:47", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "HUB/web/system.go:174", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/main.go:705-707", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1002", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "HUB/mailhold/mailhold.go:74", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "HUB/notify/dispatcher.go:243", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "RB/nightly-rotation.md:15", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "RB/RUNBOOK-manual-build.md:180", + "file": "OK", + "ident": "IDENT-OK", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "RB/RUNBOOK-hub-db-offsite-backup.md:12-14", + "file": "OK", + "ident": "IDENT-OK", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:3076", + "file": "OK", + "ident": "IDENT-OK", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + }, + { + "ptr": "E/manifests/hub.yaml:83", + "file": "OK", + "ident": "IDENT-MISS(kind:)", + "ending": "30. Nightly CI: gates and the security watch", + "q": "8" + } + ] +} \ No newline at end of file diff --git a/documentation/audits/clock-list-2026-11-11/evidence/searches.txt b/documentation/audits/clock-list-2026-11-11/evidence/searches.txt new file mode 100644 index 00000000..63f94b8e --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/evidence/searches.txt @@ -0,0 +1,29 @@ +searches run 2026-11-11 10:33:51 UTC +C+ | positive control (a row that exists) | /R-1301/ | ar files=0 | open: R-1301 | closed: | roadmap=0 +C+2 | positive control (a word in all four) | /backup/ | ar files=31 | open: R-1013 R-1025 R-1035 R-1062 R-1064 R-1078 R-1083 R-1087 R-1171 R-1172 R-1173 R-1184 R-1185 R-1186 R-1188 R-1189 R-1190 R-1198 R-1200 R-1206 R-1211 R-1212 R-1214 R-1221 R-1222 R-1229 R-1231 R-1235 R-1241 R-1242 R-1243 R-1254 R-1259 R-1260 R-1263 R-1268 R-1274 R-1293 R-1294 R-1296 R-1301 R-1312 R-1328 R-1362 R-1366 R-1368 R-1370 R-1371 R-1372 R-1373 R-1374 R-164 R-213 R-232 R-264 R-314 R-32 R-336 R-340 R-388 R-401 R-409 R-433 R-516 R-540 R-541 R-548 R-683 R-691 R-724 R-759 R-775 R-78 R-816 R-832 R-862 R-878 R-895 R-905 R-91 R-923 R-924 R-933 R-984 R-989 | closed: R-1006 R-1008 R-1012 R-1019 R-102 R-103 R-1048 R-1057 R-1059 R-1063 R-1072 R-1075 R-1079 R-1084 R-1088 R-1089 R-109 R-1097 R-110 R-1105 R-1110 R-1117 R-1119 R-1134 R-1138 R-1144 R-1149 R-1150 R-1157 R-116 R-1169 R-1175 R-1177 R-1179 R-1181 R-1195 R-1219 R-1226 R-1234 R-1252 R-1269 R-1278 R-1283 R-1287 R-1288 R-1299 R-133 R-158 R-161 R-165 R-167 R-173 R-175 R-183 R-185 R-191 R-202 R-215 R-218 R-225 R-231 R-237 R-240 R-245 R-258 R-263 R-275 R-279 R-281 R-289 R-296 R-298 R-337 R-348 R-351 R-361 R-363 R-366 R-371 R-375 R-389 R-398 R-400 R-412 R-430 R-431 R-436 R-442 R-448 R-474 R-475 R-481 R-485 R-486 R-487 R-488 R-49 R-499 R-500 R-517 R-519 R-526 R-534 R-537 R-538 R-546 R-550 R-553 R-556 R-557 R-562 R-563 R-578 R-597 R-598 R-626 R-634 R-637 R-640 R-641 R-643 R-645 R-648 R-650 R-660 R-661 R-666 R-670 R-671 R-673 R-679 R-682 R-684 R-685 R-689 R-690 R-694 R-696 R-698 R-705 R-706 R-721 R-752 R-762 R-765 R-773 R-815 R-837 R-842 R-86 R-871 R-872 R-88a R-88b R-893 R-897 R-899 R-906 R-910 R-921 R-926 R-97 R-973 R-985 R-988 | roadmap=8 +C- | negative control | /zzqxclocklistzz/ | ar files=0 | open: | closed: | roadmap=0 +S01 | a daily hub job missed while the hub is down | /(missed|skipped).{0,60}(slot|check|audit).{0,80}(pod|restart|hub (is|was) down)|catch.?up.{0,80}(hub job|scheduleDaily|daily job)/ | ar files=1 | open: | closed: | roadmap=0 +S02 | a failed household mail is not sent again | /(customer|household).{0,30}(mail|e-mail).{0,80}(retry|retried|not sent again)/ | ar files=1 | open: | closed: | roadmap=0 +S03 | the operator mail retry is lost at a hub restart | /retry.{0,100}(restart|in memory|lost)|GAVE UP/ | ar files=1 | open: R-1215 | closed: R-1163 | roadmap=0 +S04 | the guest-power watchdog gives up and tells nobody | /guest.power.{0,160}(give|gave|giving).up|GIVING UP/ | ar files=1 | open: | closed: | roadmap=0 +S05 | the peer list push to ep0 fails again and again | /wgsync.{0,120}(fail|alarm|event)|peer list.{0,80}(fail|stale|alarm)/ | ar files=0 | open: | closed: | roadmap=0 +S06 | the app-update leg did not run: who is told | /update.leg.{0,120}(did not run|missed|skipped|lost).{0,100}(told|mail|alarm|event|nobody)/ | ar files=0 | open: | closed: | roadmap=0 +S07 | a verdict trusts the box's clock (fresh backup, off-site age, proof age) | /(box|host).{0,12}clock.{0,120}(ahead|future|wrong).{0,160}(fresh|stale|latest|backup|proof)|timestamp.{0,60}(in the future|ahead)/ | ar files=0 | open: R-1189 R-1206 | closed: | roadmap=0 +S08 | an 'already ran' stamp written by a clock that was ahead | /(stamp|already ran|last run|LastOKAt).{0,100}(clock|future|ahead)/ | ar files=3 | open: | closed: | roadmap=0 +S09 | a wrong clock refuses every signed job | /(signed|job).{0,80}(not.yet.valid|expired).{0,120}clock|clock.{0,120}(signed job|issued_at|ErrNotYetValid)/ | ar files=0 | open: | closed: | roadmap=0 +S10 | the hub's own clock jumps | /hub.{0,20}clock.{0,80}(jump|step|wrong|drift)/ | ar files=0 | open: | closed: | roadmap=0 +S11 | the off-site push is skipped because another backup runs | /off-?site.{0,100}skipped.{0,100}(another|running)|another backup is running/ | ar files=0 | open: R-1186 | closed: | roadmap=0 +S12 | the 05:30 proof restore can fall inside the whole-guest backup | /proof.{0,120}(05:30|gate|whole-guest|vzdump|meets)/ | ar files=4 | open: R-132 | closed: R-1142 R-418 R-425 R-454 R-463 R-466 R-470 R-477 R-478 R-559 R-564 R-637 R-643 R-709 R-885 | roadmap=0 +S13 | the PBS verify runs beside a backup | /verify.{0,80}(heavy-op|gate|beside|during a backup)/ | ar files=2 | open: | closed: | roadmap=0 +S14 | two DooPlex timers overlap, or two machines' clocks order the jobs | /(timer|slot).{0,80}(overlap|meet|same time)|holds only (if|while) both/ | ar files=0 | open: | closed: | roadmap=0 +S15 | an interval job with no time limit | /(time limit|timeout|deadline).{0,80}(sweep|interval job|health check|scan)|slow checker|no per-check/ | ar files=1 | open: | closed: R-1344 | roadmap=0 +S16 | how long the hub's sweep takes with many boxes | /(100|hundred|1000) (boxes|customers|households)|sweep.{0,80}(duration|takes|seconds).{0,60}(boxes|fleet)/ | ar files=0 | open: | closed: | roadmap=0 +S17 | the agent unit: Restart=always and a watchdog | /Restart=always|WatchdogSec|systemd watchdog/ | ar files=2 | open: | closed: | roadmap=0 +S18 | hub database staleness alarm: 13 h or 26 h | /HubDBBackupStale/ | ar files=0 | open: | closed: | roadmap=0 +S19 | a list of every timer | /every timer|list of (all )?(timers|scheduled jobs)|all (timers|scheduled jobs)|job (table|inventory)|timer inventory/ | ar files=0 | open: | closed: | roadmap=0 +S20 | the hour change | /DST|clock change|daylight|spring forward|2027-03-28/ | ar files=12 | open: R-1189 R-1259 R-1260 R-1293 R-914 | closed: R-1169 R-246 R-356 R-386 R-521 R-660 R-921 | roadmap=0 +S21 | who keeps the time on a box | /chrony|timesyncd|NTP/ | ar files=12 | open: R-1301 | closed: R-220 R-444 | roadmap=0 +S22 | the crash guard reads the clock at early boot | /crash.guard.{0,160}(clock|RTC|time is wrong)/ | ar files=0 | open: | closed: | roadmap=0 +S23 | the catalog sync fails for a long time | /(catalog|template).sync.{0,100}(fail|stale|days|alarm)/ | ar files=0 | open: | closed: R-607 | roadmap=0 +S24 | the 05:30 proof restore is not made up | /proof.{0,120}(not made up|catch.?up|missed)/ | ar files=0 | open: | closed: R-873 R-875 | roadmap=0 +S25 | the restore-copy sweep on a box that is on for under six hours | /(sweep|every 6 h).{0,120}(first run|on for under|short uptime)/ | ar files=0 | open: | closed: | roadmap=0 diff --git a/documentation/audits/clock-list-2026-11-11/tools/check-pointers.py b/documentation/audits/clock-list-2026-11-11/tools/check-pointers.py new file mode 100644 index 00000000..480f3fb7 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/tools/check-pointers.py @@ -0,0 +1,78 @@ +#!/usr/bin/env python3 +"""Print the text behind every pointer in a table file, so a person can read it. + +Read-only by construction: it opens files for reading and prints. It runs on DooPlex against the +git clones only; it never touches a box. + +Usage: check-pointers.py [FILE ...] (default: ../TABLE.md) + check-pointers.py --quiet [FILE] (only the verdict lines; rc 1 when a pointer is dead) + +A pointer is a backticked token `PREFIX/path[:line[-line]]` with one of the prefixes below. +A pointer is DEAD when the file is missing or the line range lies beyond the end of the file. +The script cannot judge whether the line says what the table claims — that is the reader's job. +""" +import os +import re +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +AUDIT = os.path.dirname(HERE) +REPO = os.path.abspath(os.path.join(AUDIT, "..", "..", "..")) # felhom.eu +WORKSPACE = os.path.dirname(REPO) + +PREFIX = { + "AG": os.path.join(WORKSPACE, "felhom-agent"), + "CT": os.path.join(WORKSPACE, "felhom-controller", "controller"), + "CAT": os.path.join(WORKSPACE, "app-catalog-felhom.eu"), + "HUB": os.path.join(REPO, "hub", "internal"), + "E": REPO, + "AR": os.path.join(REPO, "documentation", "architecture"), + "RB": os.path.join(REPO, "documentation", "runbooks"), + "EV": os.path.join(AUDIT, "evidence"), +} +TOKEN = re.compile(r"`((?:%s)/[^`:\s]+)(?::(\d+)(?:-(\d+))?)?`" % "|".join(PREFIX)) + + +def main() -> int: + args = [a for a in sys.argv[1:] if a != "--quiet"] + quiet = "--quiet" in sys.argv[1:] + files = args or [os.path.join(AUDIT, "TABLE.md")] + seen, dead, total = set(), 0, 0 + for table in files: + with open(table, encoding="utf-8") as fh: + text = fh.read() + for m in TOKEN.finditer(text): + key = m.group(0) + if key in seen: + continue + seen.add(key) + total += 1 + pre, rest = m.group(1).split("/", 1) + path = os.path.join(PREFIX[pre], rest) + first = int(m.group(2)) if m.group(2) else None + last = int(m.group(3)) if m.group(3) else first + if not os.path.exists(path): + dead += 1 + print(f"DEAD {key} (no such file: {path})") + continue + if first is None: + if not quiet: + print(f"OK {key} (file exists)") + continue + with open(path, encoding="utf-8", errors="replace") as fh: + lines = fh.readlines() + if last > len(lines): + dead += 1 + print(f"DEAD {key} (file has {len(lines)} lines)") + continue + if quiet: + continue + print(f"OK {key}") + for n in range(first, min(last, first + 11) + 1): + print(f" {n:>5} {lines[n - 1].rstrip()[:200]}") + print(f"pointers: {total} dead: {dead}") + return 1 if dead else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/documentation/audits/clock-list-2026-11-11/tools/guest-clock.sh b/documentation/audits/clock-list-2026-11-11/tools/guest-clock.sh new file mode 100644 index 00000000..5e843cc7 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/tools/guest-clock.sh @@ -0,0 +1,21 @@ +#!/bin/bash +# Clock list 2026-11-11 — guest 9201. READ-ONLY: date, systemctl list/show/cat, cron files, atq, docker ps/logs (read). +set -u +M() { sed -E 's/((token|password|passwd|secret|apikey|api_key|authorization)[=: ]+)[^ ,;"]+/\1***/Ig'; } +h() { echo; echo "=== $* ==="; } +h "clock"; date -u +'utc %FT%T.%NZ'; date +'local %FT%T %Z %z'; cat /etc/timezone 2>/dev/null; readlink /etc/localtime; timedatectl 2>&1 | head -8 +for u in chrony systemd-timesyncd; do printf '%s: ' $u; systemctl is-active $u 2>/dev/null; done +h "boots"; uptime -s +h "timers"; systemctl list-timers --all --no-pager +h "timer units (all)"; for t in $(systemctl list-unit-files --type=timer --no-legend --no-pager | awk '{print $1}'); do echo "--- $t [$(systemctl is-enabled $t 2>/dev/null)/$(systemctl is-active $t 2>/dev/null)]"; systemctl cat $t 2>/dev/null | /usr/bin/grep -E '^(On|Persistent|RandomizedDelay|AccuracySec|Unit)'; systemctl show $t -p LastTriggerUSec -p NextElapseUSecRealtime -p Unit 2>/dev/null | tr '\n' ' '; echo; done +h "felhom units"; systemctl list-units --all --no-pager --no-legend 'felhom*' | awk '{print $1,$2,$3,$4}' +for s in $(systemctl list-unit-files --no-legend --no-pager 'felhom*.service' | awk '{print $1}'); do echo "--- $s"; systemctl show $s -p Restart -p RestartUSec -p TimeoutStartUSec -p Type -p ActiveEnterTimestamp -p NRestarts 2>/dev/null | tr '\n' ' '; echo; done +h "cron"; /usr/bin/grep -v -E '^\s*(#|$)' /etc/crontab 2>/dev/null | M; for f in /etc/cron.d/*; do [ -f "$f" ] && { echo "--- $f"; /usr/bin/grep -v -E '^\s*(#|$)' "$f" | M; }; done; for d in hourly daily weekly monthly; do echo "--- cron.$d: $(ls /etc/cron.$d 2>/dev/null | tr '\n' ' ')"; done +echo "--- user crontabs: $(ls /var/spool/cron/crontabs 2>/dev/null | tr '\n' ' ')" +h "at"; command -v atq >/dev/null && atq 2>&1 || echo "no at" +h "docker daemon timers"; /usr/bin/grep -v -E '^\s*$' /etc/docker/daemon.json 2>/dev/null | M +h "containers"; docker ps -a --format '{{.Names}}\t{{.Image}}\t{{.Status}}\t{{.RunningFor}}' +h "restart policies + healthcheck intervals"; for c in $(docker ps -q); do docker inspect --format '{{.Name}} restart={{.HostConfig.RestartPolicy.Name}} health={{if .Config.Healthcheck}}{{.Config.Healthcheck.Interval}}{{else}}none{{end}} started={{.State.StartedAt}}' $c; done +h "controller TZ"; docker exec felhom-controller sh -c 'printf "TZ=%s\n" "${TZ:-unset}"' 2>&1; docker exec felhom-controller date +'in-container %FT%T %Z %z' 2>&1 +h "controller log: first line, line count per hour"; docker logs --timestamps felhom-controller 2>&1 | head -1 | cut -c1-200 | M; docker logs --timestamps --since 30h felhom-controller 2>&1 | cut -c1-13 | sort | uniq -c | tr '\n' ' ' +h "controller log: schedule words (tail 320)"; docker logs --timestamps --since 72h felhom-controller 2>&1 | /usr/bin/grep -i -E 'schedul|night|backup (start|finish|complet|done|fail)|dump|restic|snapshot|prune|offbox|off-site|offsite|second.?drive|window|slot|missed|catch.?up|restore.?test|report (sent|fail)|heartbeat|sync(ed|ing)? |update check|self.?update|clock|skew|deadline|timed out|timeout|cron' | /usr/bin/grep -v -i -E 'GET /|POST /' | cut -c1-240 | M | tail -320 diff --git a/documentation/audits/clock-list-2026-11-11/tools/guest-night.sh b/documentation/audits/clock-list-2026-11-11/tools/guest-night.sh new file mode 100644 index 00000000..fe1e97e5 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/tools/guest-night.sh @@ -0,0 +1,14 @@ +#!/bin/bash +# guest-night.sh — read-only: per scheduler job, count, first/last start, the last three durations; and night lines. +L=$(mktemp); docker logs --timestamps felhom-controller > "$L" 2>&1 +printf 'log first line: %s\nlog last line: %s\n' "$(head -1 "$L" | cut -c1-30)" "$(tail -1 "$L" | cut -c1-30)" +printf '\n=== scheduler jobs (UTC): runs, first start, last start, last three "took" ===\n' +grep -E '\[scheduler\] (Running job: |Job .* (completed|failed|skipped))' "$L" | awk ' + / Running job: / { j=$NF; n[j]++; if(!(j in f)) f[j]=substr($1,1,19); l[j]=substr($1,1,19); next } + / Job / { for(i=1;i<=NF;i++) if($i=="Job") {j=$(i+1); break}; s=$0; sub(/^.*\[scheduler\] Job [^ ]+ /,"",s); d[j]=d3[j]; d3[j]=d2[j] " | " s; d2[j]=s; last[j]=(last[j]==""?s:last2[j]); h[j]=h[j] "#" s; } + END { for (j in n) { k=split(h[j],p,"#"); o=""; for(i=(k>3?k-2:2);i<=k;i++) o=o " [" p[i] "]"; printf "%-34s runs=%-5d first=%s last=%s%s\n", j, n[j], f[j], l[j], o } }' | sort +printf '\n=== jobs that ran fewer than 30 times: every start and end (UTC) ===\n' +grep -E '\[scheduler\] (Running job: |Job .* (completed|failed|skipped))' "$L" | awk '{ if ($0 ~ / Running job: /) j=$NF; else { for(i=1;i<=NF;i++) if($i=="Job") {j=$(i+1); break} } c[j]++; line[NR]=$0; jj[NR]=j } END { for(i=1;i<=NR;i++) if (c[jj[i]]<60) { s=line[i]; sub(/^[^ ]+ /,"",s); sub(/scheduler.go:[0-9]+: /,"",s); print substr(s,1,170) } }' | tail -120 +printf '\n=== night lines 19:00-08:00 UTC, INFO/WARN/ERROR, backup words (first 2 per message shape) ===\n' +grep -E 'T(19|2[0-3]|0[0-7]):' "$L" | grep -E '\[(INFO|WARN|ERROR)\]' | grep -v '\[scheduler\]' | grep -i -E 'backup|restic|offbox|offsite|restore|prune|dump|verify|snapshot|retention|forget|check' | awk '{ k=$0; sub(/^[^ ]+ [^ ]+ [^ ]+ /,"",k); gsub(/[0-9]+/,"N",k); k=substr(k,1,48); c[k]++; if (c[k]<=2) { s=$0; sub(/^[^ ]+ /,"",s); print substr(s,1,230) } }' | head -90 +rm -f "$L" diff --git a/documentation/audits/clock-list-2026-11-11/tools/host-clock.sh b/documentation/audits/clock-list-2026-11-11/tools/host-clock.sh new file mode 100644 index 00000000..56dcd534 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/tools/host-clock.sh @@ -0,0 +1,30 @@ +#!/bin/bash +# Clock list 2026-11-11 — host. READ-ONLY by construction: date, timedatectl (show), chronyc (read), systemctl list/show/cat, +# ls and cat of cron files, atq, journalctl (read), the PVE task index (read). It starts no job and writes no file. +set -u +M() { sed -E 's/((token|password|passwd|secret|apikey|api_key|authorization)[=: ]+)[^ ,;"]+/\1***/Ig'; } +h() { echo; echo "=== $* ==="; } +h "in-flight"; pgrep -x vzdump >/dev/null && echo "vzdump RUNNING"; pgrep -x apt-get >/dev/null && echo "apt RUNNING"; pgrep -f 'pct restore' >/dev/null && echo "restore RUNNING"; echo "checked" +h "clock"; date -u +'utc %FT%T.%NZ'; date +'local %FT%T %Z %z'; cat /etc/timezone 2>/dev/null; readlink /etc/localtime +timedatectl show 2>/dev/null; timedatectl 2>/dev/null +h "who keeps the time"; for u in chrony chronyd systemd-timesyncd ntp ntpsec openntpd; do printf '%s: ' $u; systemctl is-active $u 2>/dev/null; printf ' enabled: '; systemctl is-enabled $u 2>/dev/null; done +command -v chronyc >/dev/null && { chronyc -n tracking 2>&1; chronyc -n sources 2>&1; } +timedatectl timesync-status 2>&1 | head -12 +/usr/bin/grep -v -E '^\s*(#|$)' /etc/chrony/chrony.conf 2>/dev/null; ls /etc/chrony/conf.d /etc/chrony/sources.d 2>/dev/null +/usr/bin/grep -v -E '^\s*(#|$)' /etc/systemd/timesyncd.conf 2>/dev/null +h "start with no time"; ls -l --time-style=full-iso /var/lib/systemd/timesync/clock /var/lib/chrony/ 2>&1 | head; dpkg -l fake-hwclock 2>/dev/null | tail -1; cat /etc/adjtime 2>/dev/null; hwclock --show --utc 2>&1 | head -2; cat /sys/class/rtc/rtc0/since_epoch 2>/dev/null; date -u +%s +h "boots"; uptime -s; journalctl --list-boots --no-pager 2>/dev/null | tail -6 +h "timers"; systemctl list-timers --all --no-pager +h "timer units (all)"; for t in $(systemctl list-unit-files --type=timer --no-legend --no-pager | awk '{print $1}'); do echo "--- $t [$(systemctl is-enabled $t 2>/dev/null)/$(systemctl is-active $t 2>/dev/null)]"; systemctl cat $t 2>/dev/null | /usr/bin/grep -E '^(On|Persistent|RandomizedDelay|AccuracySec|Unit|FixedRandomDelay|WakeSystem)' ; systemctl show $t -p LastTriggerUSec -p NextElapseUSecRealtime -p Unit 2>/dev/null | tr '\n' ' '; echo; done +h "service limits of felhom + backup timers"; for t in $(systemctl list-unit-files --type=timer --no-legend --no-pager | awk '{print $1}' | /usr/bin/grep -i -E 'felhom|pve|pbs|apt|unattended|fstrim|logrotate|man-db|e2scrub|dpkg|smart'); do s=$(systemctl show $t -p Unit --value); echo "--- $s"; systemctl show "$s" -p TimeoutStartUSec -p RuntimeMaxUSec -p ExecStart -p Type -p ExecMainStartTimestamp -p ExecMainExitTimestamp -p ExecMainStatus 2>/dev/null | cut -c1-300 | M; done +h "felhom units that are not timers"; systemctl list-units --all --no-pager --no-legend 'felhom*' | awk '{print $1,$2,$3,$4}' +for s in $(systemctl list-unit-files --no-legend --no-pager 'felhom*.service' | awk '{print $1}'); do echo "--- $s"; systemctl show $s -p Restart -p RestartUSec -p WatchdogUSec -p TimeoutStartUSec -p RuntimeMaxUSec -p ActiveEnterTimestamp -p NRestarts 2>/dev/null | tr '\n' ' '; echo; done +h "cron"; /usr/bin/grep -v -E '^\s*(#|$)' /etc/crontab 2>/dev/null | M; for f in /etc/cron.d/*; do echo "--- $f"; /usr/bin/grep -v -E '^\s*(#|$)' "$f" | M; done; for d in hourly daily weekly monthly; do echo "--- cron.$d: $(ls /etc/cron.$d 2>/dev/null | tr '\n' ' ')"; done +echo "--- user crontabs: $(ls /var/spool/cron/crontabs 2>/dev/null | tr '\n' ' ')"; for f in /var/spool/cron/crontabs/*; do [ -f "$f" ] && { echo "--- $f"; /usr/bin/grep -v -E '^\s*(#|$)' "$f" | M; }; done +h "at"; command -v atq >/dev/null && atq 2>&1 || echo "no at" +h "pve jobs"; cat /etc/pve/jobs.cfg 2>/dev/null | M; echo "--- vzdump.cron"; /usr/bin/grep -v -E '^\s*(#|$)' /etc/pve/vzdump.cron 2>/dev/null; echo "--- vzdump.conf"; /usr/bin/grep -v -E '^\s*(#|$)' /etc/vzdump.conf 2>/dev/null +h "pve tasks (start end type id status), last 14 days"; cat /var/log/pve/tasks/index.1 /var/log/pve/tasks/index 2>/dev/null | awk -v now=$(date +%s) '{split($1,a,":"); st=strtonum("0x" a[5]); en=strtonum("0x" $2); if (now-st < 14*86400) printf "%s %s dur=%ds %s %s %s\n", strftime("%FT%T%z",st), strftime("%T",en), en-st, a[6], a[7], $3}' | /usr/bin/grep -v -E ' (vncproxy|vncshell|spiceproxy) ' | tail -150 +h "runs of timer services, last 10 days (start/finish lines)"; for t in $(systemctl list-unit-files --type=timer --no-legend --no-pager | awk '{print $1}' | /usr/bin/grep -i -E 'felhom|pve-daily|apt-daily|pbs'); do s=$(systemctl show $t -p Unit --value); echo "--- $s"; journalctl -u "$s" --since '-10 days' -o short-iso --no-pager -q 2>/dev/null | /usr/bin/grep -E 'Starting|Started|Finished|Failed|Deactivated|Consumed' | tail -9 | cut -c1-220 | M; done +h "agent log: schedule words, last 3 days (tail 260)"; journalctl -u felhom-agent --since '-3 days' -o short-iso --no-pager -q 2>/dev/null | /usr/bin/grep -i -E 'night|vzdump|backup|restore.?test|osupdate|os-update|kernel|slot|window|schedul|prune|pbs|report (sent|fail)|clock|skew|drift|missed|catch.?up|deadline|timeout' | cut -c1-260 | M | tail -260 +h "agent log: line count per hour, last 26 h"; journalctl -u felhom-agent --since '-26 hours' -o short-iso --no-pager -q 2>/dev/null | cut -c1-13 | sort | uniq -c | tr '\n' ' ' +h "agent version"; /usr/local/bin/felhom-agent --version 2>&1 | head -1 diff --git a/documentation/audits/clock-list-2026-11-11/tools/host-night.sh b/documentation/audits/clock-list-2026-11-11/tools/host-night.sh new file mode 100644 index 00000000..f4e69321 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/tools/host-night.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# host-night.sh — read-only: PVE task index (start, end, minutes, type, id, status) for the last 14 days; agent night lines. +printf '=== pve tasks, last 14 days (local time) ===\n' +cat /var/log/pve/tasks/index.1 /var/log/pve/tasks/index 2>/dev/null | perl -ne ' + next unless /^UPID:[^:]+:[0-9A-Fa-f]+:[0-9A-Fa-f]+:([0-9A-Fa-f]+):([^:]+):([^:]*):[^:]+: ([0-9A-Fa-f]+) (.*)$/; + my ($s,$t,$id,$e,$st)=(hex($1),$2,$3,hex($4),$5); next if $s < time-14*86400; + my @a=localtime($s); my @b=localtime($e); + printf "%04d-%02d-%02d %02d:%02d:%02d -> %02d:%02d:%02d %6.1f min %-12s %-8s %s\n",$a[5]+1900,$a[4]+1,$a[3],$a[2],$a[1],$a[0],$b[2],$b[1],$b[0],($e-$s)/60,$t,$id,substr($st,0,60);' | sort | grep -v -E ' (vncproxy|vncshell|spiceproxy) ' | tail -150 +printf '\n=== agent: night words, 20:00 yesterday to 09:00 today (first 3 lines per message) ===\n' +journalctl -u felhom-agent --since "yesterday 20:00" --until "today 09:00" --no-pager -o short-iso 2>/dev/null | grep -E 'level=(INFO|WARN|ERROR)' | grep -i -E 'backup|vzdump|verify|restore-test|prune|update|osupdate|escrow|gc|sweep|retention|offsite|window|sched' | perl -ne 'my ($m)=/msg="([^"]{0,70})/; $m//="?"; $c{$m}++; if($c{$m}<=3){ s/^(\S+) \S+ \S+ time=\S+ /$1 /; print substr($_,0,260),"\n" } END{ print "--- counts\n"; printf "%6d %s\n",$c{$_},$_ for sort keys %c }' diff --git a/documentation/audits/clock-list-2026-11-11/tools/listconf.py b/documentation/audits/clock-list-2026-11-11/tools/listconf.py new file mode 100644 index 00000000..13a937c4 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/tools/listconf.py @@ -0,0 +1,9 @@ +"""listconf.py — the Clock list's own words for make-table.py and make-matrix.py. Data only.""" +HELPERS = ("clock-controller", "clock-agent", "clock-hub") +KINDS = ["Night", "Interval", "Watchdog", "Hub daily", "Hub sweep", "DooPlex and ep0"] +WORD, WORDS = "job", "Jobs" +TITLE = "The Clock list" +ID_RE = r"C[CAH]\d\d" +QUESTIONS = ["what runs, owner, what starts it", "when, how often, order", "how long, limit, 10x data", "if it does not run", + "late, twice, meets another job", "clock wrong or jumps", "who is told, when", "how do we know"] +CLASS_ROW = {"W1": "R-1383", "W2": "R-1384", "W3": "R-1385", "W4": "R-1386", "W5": "R-1387"} diff --git a/documentation/audits/clock-list-2026-11-11/tools/make-matrix.py b/documentation/audits/clock-list-2026-11-11/tools/make-matrix.py new file mode 100644 index 00000000..9528b0f5 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/tools/make-matrix.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python3 +"""Build MATRIX.md from TABLE.md, and check the table's shape. + +The same tool as `audits/endings-list-2026-11-10/tools/make-matrix.py`, made general: the list's own words are in `listconf.py`. + +Read-only on everything but MATRIX.md in this folder. It runs on DooPlex against the git clone only. + +Usage: make-matrix.py write ../MATRIX.md, print the totals + make-matrix.py --check write nothing; rc 1 when the table breaks a rule + +Rules checked (a broken rule is printed and makes rc 1): + - a heading is `### — · · NAMED|MENTIONED|UNNAMED` + - each block has exactly eight answer rows, numbered 1..8, each with one mark A, D, K or N + - an A row carries a backticked pointer; a K row an `R-`; an N row one class from listconf.py and a search S +""" +import os +import re +import sys +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import listconf as C + +HERE = os.path.dirname(os.path.abspath(__file__)) +AUDIT = os.path.dirname(HERE) +HEAD = re.compile(r"^### (" + C.ID_RE + r") — (.+?) · (?:" + "|".join(C.KINDS) + r") · (NAMED|MENTIONED|UNNAMED)\s*$") +ROW = re.compile(r"^\| ([1-8]) \| (.*) \| ([ADKN]) \| (.*) \|\s*$") +QUESTIONS = C.QUESTIONS +CLASS_ROW = C.CLASS_ROW +CLS = re.compile(r"\b(?:" + "|".join(C.CLASS_ROW) + r")\b") + + +def parse(path): + places, errors, cur = [], [], None + with open(path, encoding="utf-8") as fh: + for ln, line in enumerate(fh, 1): + if line.startswith("## TEST-ONLY") or line.startswith("## DOC-ONLY"): + cur = None + continue + if line.startswith("### "): + m = HEAD.match(line) + if not m: + errors.append(f"line {ln}: heading has the wrong shape: {line.strip()}") + cur = None + continue + cur = {"id": m.group(1), "name": m.group(2), "named": m.group(3), "marks": {}, "n": {}, "k": {}} + places.append(cur) + continue + m = ROW.match(line) + if not m or cur is None: + continue + q, mark, ptr = int(m.group(1)), m.group(3), m.group(4) + if q in cur["marks"]: + errors.append(f"{cur['id']}: question {q} is answered twice") + cur["marks"][q] = mark + if mark == "A" and "`" not in ptr: + errors.append(f"{cur['id']}-{q}: A with no pointer") + if mark == "K": + rows = re.findall(r"R-\d+", ptr) + if not rows: + errors.append(f"{cur['id']}-{q}: K with no row") + cur["k"][q] = rows + if mark == "N": + cls = CLS.findall(ptr) + if len(cls) != 1: + errors.append(f"{cur['id']}-{q}: N needs exactly one class, found {cls}") + elif not re.search(r"\bS\d+\b", ptr) and "NOBODY KNOWS" not in m.group(2): + errors.append(f"{cur['id']}-{q}: N with no search id") + cur["n"][q] = cls[0] if cls else "?" + for p in places: + if sorted(p["marks"]) != list(range(1, 9)): + errors.append(f"{p['id']}: has answers {sorted(p['marks'])}, wants 1..8") + return places, errors + + +def build(places): + out = ["# " + C.TITLE + " — the matrix", "", + "Made by `tools/make-matrix.py` from `TABLE.md`. Do not edit it by hand.", "", + "`A` answered · `D` decided, not built · `K` known gap (has a row) · `N` nobody asked. Columns are the eight", + "questions: " + " · ".join(f"{i} {q}" for i, q in enumerate(QUESTIONS, 1)) + ".", + "", "| " + C.WORD.capitalize() + " | Named | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | N |", "|---|---|---|---|---|---|---|---|---|---|---|"] + tot = {m: 0 for m in "ADKN"} + perq = {q: {m: 0 for m in "ADKN"} for q in range(1, 9)} + named = {"NAMED": 0, "MENTIONED": 0, "UNNAMED": 0} + members, krows = {c: [] for c in CLASS_ROW}, {} + for p in places: + named[p["named"]] += 1 + cells = [] + for q in range(1, 9): + mk = p["marks"].get(q, "?") + cells.append(f"**{mk}**" if mk == "N" else mk) + if mk in tot: + tot[mk] += 1 + perq[q][mk] += 1 + for q, c in p["n"].items(): + members.setdefault(c, []).append(f"{p['id']}-{q}") + for q, rows in p["k"].items(): + for r in rows: + krows.setdefault(r, []).append(f"{p['id']}-{q}") + out.append(f"| {p['id']} {p['name']} | {p['named'].lower()} | " + " | ".join(cells) + f" | {len(p['n'])} |") + n = len(places) + out += ["", "## Totals", "", + f"- {C.WORDS}: **{n}** in the table · named **{named['NAMED']}** · mentioned only **{named['MENTIONED']}** · " + f"named by no architecture document **{named['UNNAMED']}**.", + f"- Answers: **{n * 8}** · A **{tot['A']}** · D **{tot['D']}** · K **{tot['K']}** · N **{tot['N']}**.", + f"- {C.WORDS} with no N: **{sum(1 for p in places if not p['n'])}** of {n}.", + "", "| Question | A | D | K | N |", "|---|---|---|---|---|"] + for q in range(1, 9): + out.append(f"| {q} {QUESTIONS[q - 1]} | {perq[q]['A']} | {perq[q]['D']} | {perq[q]['K']} | {perq[q]['N']} |") + out += ["", "## The N answers, by class row", "", "| Class | Row | Count | Members (id-question) |", "|---|---|---|---|"] + for c in sorted(members): + out.append(f"| {c} | {CLASS_ROW.get(c, '?')} | {len(members[c])} | {', '.join(members[c])} |") + out += ["", "## The K answers, by existing row", "", "| Row | Count | Members |", "|---|---|---|"] + for r in sorted(krows, key=lambda x: int(x[2:])): + out.append(f"| {r} | {len(krows[r])} | {', '.join(krows[r])} |") + return "\n".join(out) + "\n", tot, named, members + + +def main() -> int: + places, errors = parse(os.path.join(AUDIT, "TABLE.md")) + for e in errors: + print("BROKEN " + e) + text, tot, named, members = build(places) + if "--check" not in sys.argv[1:]: + with open(os.path.join(AUDIT, "MATRIX.md"), "w", encoding="utf-8") as fh: + fh.write(text) + print(f"places {len(places)} named {named} marks {tot} " + + " ".join(f"{c}={len(m)}" for c, m in sorted(members.items())) + f" broken {len(errors)}") + return 1 if errors else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/documentation/audits/clock-list-2026-11-11/tools/make-table.py b/documentation/audits/clock-list-2026-11-11/tools/make-table.py new file mode 100644 index 00000000..9cf1a41d --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/tools/make-table.py @@ -0,0 +1,126 @@ +#!/usr/bin/env python3 +"""make-table.py — build TABLE.md from the helpers' answers (evidence/helpers/resolved-*.json), marks.py and listconf.py. + +The same tool as `audits/endings-list-2026-11-10/tools/make-table.py`, made general: what differs between two lists +(the heading words, the helper names, the kinds, the questions) is in `listconf.py` beside it. + +A cell with an entry in marks.py carries the MAIN SESSION's words. Every other cell carries the helper's words and the +mark A; its pointers were opened by resolve-helper.py (the file and the line exist; the name the helper gave stands at +the line). A dead pointer is dropped here. Read-only but for TABLE.md. +""" +import json, os, re, subprocess, sys +import marks, listconf as C +HERE = os.path.dirname(os.path.abspath(__file__)); AUDIT = os.path.dirname(HERE) +REPO = os.path.abspath(os.path.join(HERE, "..", "..", "..", "..")); WS = os.path.dirname(REPO) +PREFIX = {"AG": os.path.join(WS, "felhom-agent"), "CT": os.path.join(WS, "felhom-controller", "controller"), + "CAT": os.path.join(WS, "app-catalog-felhom.eu"), "HUB": os.path.join(REPO, "hub", "internal"), "E": REPO, + "AR": os.path.join(REPO, "documentation", "architecture"), "RB": os.path.join(REPO, "documentation", "runbooks"), + "EV": os.path.join(AUDIT, "evidence")} +TOK = re.compile(r"`((?:AG|CT|CAT|HUB|E|AR|RB|EV)/[^`:\s]+)(?::(\d+)(?:-(\d+))?)?`") +R = {h: json.load(open(os.path.join(AUDIT, "evidence", "helpers", f"resolved-{h}.json")))["endings"] for h in C.HELPERS} +dropped = [] + + +def live(tok): + m = TOK.fullmatch(tok) + pre, rest = m.group(1).split("/", 1) + p = os.path.join(PREFIX[pre], rest) + if not os.path.exists(p): + return False + if m.group(2): + n = sum(1 for _ in open(p, encoding="utf-8", errors="replace")) + return int(m.group(3) or m.group(2)) <= n + return True + + +def split(ans, eid, q): + """helper answer -> (plain text, [live pointers])""" + ptrs = [] + for m in TOK.finditer(ans): + t = m.group(0) + if live(t): + if t not in ptrs: + ptrs.append(t) + else: + dropped.append(f"{eid}-{q} {t}") + t = TOK.sub("", ans) + t = re.sub(r"`~?:\d+(?:-\d+)?`", "", t) + t = re.sub(r"\(\s*(?:called|creator|see)?\s*[,;]?\s*\)", "", t) + t = re.sub(r"\s+([,.;:)])", r"\1", t) + t = re.sub(r"\(\s+", "(", t) + t = re.sub(r"([,;:])(?:\s*[,;:])+", r"\1", t) + t = re.sub(r"\s{2,}", " ", t).replace("|", "/").strip(" ;,·") + t = re.sub(r"\(\)", "", t) + if len(t) > 380: + cut = max(t.rfind(". ", 0, 380), t.rfind("; ", 0, 380)) + t = t[:cut + 1 if cut > 150 else 377].rstrip(" ;") + " …" + return t, ptrs + + +def named(m): + kind, why = m["named"] + if kind != "grep": + return kind, f"architecture: {why}" + r = subprocess.run(["/usr/bin/grep", "-rEic", "--include=*.md", why, PREFIX["AR"]], capture_output=True, text=True) + files = [l for l in r.stdout.split("\n") if l and not l.endswith(":0")] + lines = sum(int(l.rsplit(":", 1)[1]) for l in files) + return ("MENTIONED" if lines else "UNNAMED"), f"architecture folder: {lines} lines in {len(files)} files match `/{why}/` (a machine count, not a reading)" + + +def main(): + out = [open(os.path.join(HERE, "table-head.md"), encoding="utf-8").read().rstrip(), ""] + kindmap, bad, body = {}, [], [] + for eid, m in sorted(marks.M.items()): + e = R[m["src"][0]][m["src"][1]] + st, why = named(m) + worst, rows = "A", [] + for q in range(1, 9): + if q in m["o"]: + mark, text, ptr = m["o"][q] + if ptr is None or ptr.startswith("+"): # "+ rest" = the helper's pointers, then rest + _, ps = split(e["answers"].get(str(q), ""), eid, q) + ptr = (" ".join(ps[:4]) + (ptr[1:] if ptr else "")).strip(" ·") + else: + mark = "A" + text, ps = split(e["answers"].get(str(q), ""), eid, q) + ptr = " ".join(ps[:4]) + if q == 8 and eid in marks.MEASURED: + mt, mp = marks.MEASURED[eid] + text = mt + " · " + text + ptr = (mp + " " + ptr).strip() + if mark == "A" and "`" not in ptr: # borrow the code path from question 1, then 2 + for alt in (1, 2, 3): + _, ps = split(e["answers"].get(str(alt), ""), eid, alt) + if ps: + ptr = " ".join(ps[:2]) + f" (the code path of question {alt})" + break + if mark == "A" and "`" not in ptr: + bad.append(f"{eid}-{q}: A with no pointer") + if not text: + bad.append(f"{eid}-{q}: empty answer") + rows.append(f"| {q} | {text} | {mark} | {ptr} |") + worst = max(worst, mark, key="ADKN".index) + kindmap.setdefault(m["seed"], []).append((eid, st, worst)) + body += [f"### {eid} — {m['name']} · {m['seed']} · {st}", "", + f"{why} · helper section: {m['src'][0]} „{e['title'].strip()}”", "", + "| # | Answer | Mark | Pointer |", "|---|---|---|---|"] + rows + ["", f"**Hand step in a runbook?** {m['rb']}", ""] + out += [f"## The kinds, and where each {C.WORD} is", "", f"| Kind | {C.WORDS} (worst mark) | Named · mentioned · named nowhere |", "|---|---|---|"] + for kind in C.KINDS: + v = kindmap.get(kind, []) + c = {s: sum(1 for x in v if x[1] == s) for s in ("NAMED", "MENTIONED", "UNNAMED")} + out.append(f"| {kind} | " + ", ".join(f"{x[0]} ({x[2]})" for x in v) + f" | {c['NAMED']} · {c['MENTIONED']} · {c['UNNAMED']} |") + for k in kindmap: + if k not in C.KINDS: + bad.append(f"kind {k} is not in listconf.KINDS") + out += ["", "## Not in the table", "", + "**Tiny (named once, not tabled):** " + " · ".join(marks.TINY) + ".", "", + "**Left out for another reason:**", ""] + [f"- {a} — {b}." for a, b in marks.LEFT_OUT] + out += ["", "---", ""] + body + open(os.path.join(AUDIT, "TABLE.md"), "w", encoding="utf-8").write("\n".join(out).rstrip() + "\n") + print(f"{C.WORDS} {len(marks.M)} · dead pointers dropped {len(dropped)} · broken {len(bad)}") + for b in bad + dropped: + print(" " + b) + return 1 if bad else 0 + + +sys.exit(main()) diff --git a/documentation/audits/clock-list-2026-11-11/tools/marks.py b/documentation/audits/clock-list-2026-11-11/tools/marks.py new file mode 100644 index 00000000..317038c4 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/tools/marks.py @@ -0,0 +1,290 @@ +"""marks.py — the main session's rulings for the Clock list (2026-11-11). Data only; make-table.py reads it. + +One entry for each job that is in the table: + src (helper, index) the helper section the default answers come from (evidence/helpers/resolved-.json) + name the heading + seed the kind (Night, Interval, Watchdog, Hub daily, Hub sweep, DooPlex and ep0) + named ("grep", REGEX): 0 lines in the architecture folder = UNNAMED, else MENTIONED + o {question: (mark, answer, pointers)} — the MAIN SESSION's words. + pointers "+ rest" = the helper's pointers for that question (opened by the tool), then rest. + A pointer written out here was opened by eye. + A question with no entry here takes the helper's answer and the mark A. + rb the ninth line: is the hand step written in a runbook? +""" +NM = "NOT MEASURED (DooPlex and ep0 are outside this run). " + + +def N(cls, s, text, p="+"): + return ("N", text, f"{p} · class {cls} · S{s:02d}") + + +def K(row, text, p="+"): + return ("K", text, f"{p} · R-{row}") + + +def A(text, p="+"): + return ("A", text, p) + + +NOLIMIT = "No time limit in the source, and no duration was measured. Nobody knows how long it takes on a big box." +NH = "no hand step" +_M = [ + # ---------------------------------------------------------------- controller (in the guest) + ("CC01", "clock-controller", 0, "Night dump leg: databases and volumes · CUSTOMER", "Night", r"db-dump|dump leg", + {3: A("MEASURED, the night of 2026-11-11: 2 min 33 s on demo-hp, 40 s on Tester 1, 18 s on the N100. One database dump has a 5-minute limit. The whole leg has no limit. It grows with the number and size of the apps; all three stores are small.", + "`EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt` `CT/internal/appbackup/dbdump.go:291`")}, NH), + ("CC02", "clock-controller", 1, "Second-drive copy leg · CUSTOMER", "Night", r"tier.?2|second.drive|cross.drive", + {5: N("W3", 14, "It waits for the dump leg by a start time only (60 minutes later). It is skipped while a migration runs. The file holds no lock of its own. With a backup time between 02:00 and 02:59 on the spring hour-change night the two legs start at the same instant; that case is row R-1260."), + 7: A("The household and the hub are told of a failed copy: the copy's result goes to the notifier.", "`CT/cmd/controller/main.go:1427`")}, NH), + ("CC03", "clock-controller", 2, "Off-site push leg, with the clean-up of old snapshots · CUSTOMER", "Night", r"offbox|off.site push|offsite push", + {5: K(1186, "When another backup holds the flag, the push is skipped with an INFO line and the leg is marked as ended. Only the hub's 48-hour check notices. A long second-drive leg can cost the night's off-site copy.", + "`CT/internal/backup/offbox.go:1054-1057`")}, + "NO — no runbook says how to see or repair a skipped off-site night"), + ("CC04", "clock-controller", 3, "Missed-night catch-up and the resume watch · CUSTOMER", "Night", r"catch.?up|nightchain|missed night", + {6: N("W2", 8, "The watch reads a gap of more than 5 minutes in the wall clock as 'the box was asleep'. A clock that is corrected forward by more than 5 minutes looks the same. The design text says this was not measured live.")}, NH), + ("CC05", "clock-controller", 4, "Missed-backup banner (26 hours) · CUSTOMER", "Night", r"26 h|missed.backup banner|banner", + {6: N("W2", 8, "The age is wall-clock arithmetic. A clock that is set back hides the banner for as long as the step. No test sets the clock back.")}, NH), + ("CC06", "clock-controller", 5, "Whole-guest backup driver: stop apps, ask the agent, start apps · CUSTOMER", "Night", r"quiesce|whole.guest|vzdump", + {6: K(1260, "The gate counts wall-clock minutes; the poll is a monotonic ticker. On the hour-change night the two disagree. No test covers that night.")}, NH), + ("CC07", "clock-controller", 6, "Automatic app-update leg · CUSTOMER", "Night", r"update.leg|unattended", + {6: K(1260, "The leg's deadline is the backup time plus 5 hours of real time, and the scheduler's slots are in Budapest time. On the hour-change night a step can start after the backup gate stopped waiting. A daily fire that is more than 60 minutes late is skipped, so no update runs at noon."), + 7: N("W1", 6, "Nobody. When the leg does not run, the only trace is one log line on the box. No event goes to the hub.", "`CT/internal/stacks/unattended.go:374-375`")}, NH), + ("CC08", "clock-controller", 7, "Off-site integrity check · CUSTOMER", "Night", r"integrity", + {3: K(401, "MEASURED 2026-11-11: under 1 second on all three boxes (small stores). The check reads the whole store; the time on a large store rests on one measurement.", "`EV/box/demo-hp-guest-night.txt`"), + 7: A("A failed check goes to the notifier: the household and the hub are told.", "`CT/cmd/controller/main.go:4144-4186`")}, NH), + ("CC09", "clock-controller", 8, "Off-site proof restore · CUSTOMER", "Night", r"proof restore|offsite.proof|proof", + {5: N("W3", 12, "MEASURED 2026-11-11: on Tester 1 and on the N100 the whole-guest backup started at about 05:30 local time, the same minute as this job. This job is not behind the whole-guest gate. It takes the same flag as the off-site push; a meeting there is a silent skip (R-1186).", + "`EV/box/tester-1-host-night.txt` `EV/box/demo-felhom-host-night.txt` `EV/box/tester-1-guest-night.txt`"), + 7: A("An empty or failed proof goes to the notifier: the household and the hub are told.", "`CT/cmd/controller/main.go:4144-4186`")}, NH), + ("CC10", "clock-controller", 9, "Hub report: every 15 minutes, on change, and the hanging wait · CUSTOMER", "Interval", r"15.min|hub report|pusher", + {7: A("Nobody on the box: the box stores the time of the last attempt only. The outside sign is the hub's liveness check (CH01).", "`CT/internal/report/pusher.go:109-111` `HUB/monitor/host_staleness.go:59-60`")}, NH), + ("CC11", "clock-controller", 10, "Controller self-update: check, floor, nightly auto · CUSTOMER", "Interval", r"self.?update", + {3: N("W4", 15, "The registry call has a 15-second limit. The swap of the container has no limit in the file, and no duration was measured in this run."), + 5: N("W3", 12, "The auto-update waits while a backup or an app update runs. Its default time, 04:30, is the minute at which the whole-guest gate opens for a 02:30 backup time. Nothing orders the two.", "`CT/internal/config/config.go:419` `CT/internal/selfupdate/updater.go:533-544`"), + 6: A("The check uses a monotonic ticker. The nightly auto-update is a daily slot (see CC22). The switch is off unless the config turns it on; the default time is 04:30.", "`CT/internal/config/config.go:278-279` `CT/internal/config/config.go:419`")}, NH), + ("CC12", "clock-controller", 11, "Catalog sync: app templates from git · CUSTOMER", "Interval", r"catalog sync|git.sync|sync", + {3: N("W4", 15, "Each git command has its own limit. One whole sync has none, and no duration was measured.", "`CT/internal/sync/sync.go:724` `CT/internal/sync/sync.go:215-216`"), + 4: N("W1", 23, "A missed sync is done at the next tick. A sync that fails for days writes an ERROR line on the box at each tick. Nobody is told.", "`CT/internal/sync/sync.go:236` `CT/internal/sync/sync.go:247`"), + 5: A("Two syncs cannot run at once: `doSync` takes the syncer's lock first. The 30-second guard is for the button only.", "`CT/internal/sync/sync.go:215-216` `CT/internal/sync/sync.go:153-155`")}, NH), + ("CC13", "clock-controller", 12, "System health check and base-stack self-heal · CUSTOMER", "Interval", r"health check|self.heal", + {3: N("W4", 15, NOLIMIT)}, NH), + ("CC14", "clock-controller", 13, "Dead-app scan · CUSTOMER", "Interval", r"stack.scan|dead.app|crash.?loop", + {3: N("W4", 15, NOLIMIT + " It grows with the number of apps."), + 6: N("W2", 8, "The quiet time after a start uses the wall clock. A clock that steps back after the start makes the quiet time longer. No test.")}, NH), + ("CC15", "clock-controller", 14, "Disk-fill watch · CUSTOMER", "Interval", r"fill.?watch|disk.fill", + {3: N("W4", 15, NOLIMIT), + 5: A("The night slot and the interval job are two jobs with two names. The watcher holds one lock, so the two cannot change its state at the same time.", "`CT/internal/fillwatch/fillwatch.go:162` `CT/cmd/controller/main.go:1804-1805`"), + 7: A("The household: the watcher pushes a customer event when a disk fills.", "`CT/internal/fillwatch/fillwatch.go:80-153`")}, NH), + ("CC16", "clock-controller", 15, "Drive health check, hourly · CUSTOMER", "Interval", r"SMART|disk.health|drive health", + {3: N("W4", 15, "One fetch was measured at about 0.8 s (a comment in the source). No time limit in the source."), + 7: A("The household and the hub: the check emits `disk_health_degraded`.", "`CT/internal/web/disk_health.go:175-280`")}, NH), + ("CC17", "clock-controller", 16, "Agent channel health, every 60 seconds · CUSTOMER", "Interval", r"agent channel|channel health", {}, NH), + ("CC18", "clock-controller", 17, "Held-app report to the hub · CUSTOMER", "Interval", r"held.app|held app", + {6: N("W2", 8, "The daily resend is wall-clock arithmetic. A clock that is set back delays the resend by the size of the step.")}, NH), + ("CC19", "clock-controller", 18, "Abandoned off-site store sweep · CUSTOMER", "Night", r"abandon", + {3: A("One sweep has a 1-minute limit.", "`CT/internal/backup/offbox_abandon.go:180`"), + 5: N("W3", 11, "The sweep takes no lock. MEASURED: on the three boxes it ran 55 minutes after the off-site push ended, so they did not meet on these small stores.", "`CT/internal/backup/offbox_abandon.go:180` `EV/box/demo-hp-guest-night.txt`"), + 7: N("W1", 6, "Nobody. A sweep that fails writes a WARN or ERROR line on the box only.", "`CT/internal/backup/offbox_abandon.go:180`")}, NH), + ("CC20", "clock-controller", 19, "Restore-copy sweep and conversion-copy release · CUSTOMER", "Interval", r"restore.copy|restore copy", + {3: N("W4", 15, NOLIMIT), + 4: N("W1", 25, "The first run is 6 hours after the controller starts. A box that is on for under 6 hours a day never runs it; old restore copies stay. Nobody is told."), + 7: A("The household is told when a restore copy is deleted.", "`CT/internal/notify/notifier.go:1601`")}, NH), + ("CC21", "clock-controller", 21, "Upload and CPU watch, torrent limit rule, line-test hold · CUSTOMER", "Interval", r"upwatch|unexplained|upload", + {5: N("W3", 14, "Two loops with their own tickers. The file holds no lock between them."), + 7: A("The household and the hub: the watch emits `app_unexplained_upload` and `app_unexplained_cpu`.", "`CT/internal/upwatch/watch.go:309-382`")}, NH), + ("CC22", "clock-controller", 23, "The scheduler itself: daily slots and interval jobs · CUSTOMER", "Night", r"scheduler|daily slot", + {6: K(1260, "Slots are in Budapest time. On the spring night 02:30 does not exist; on the autumn night it comes twice. No test covers either night. A clock set back while a timer waits does not matter: the timer runs on monotonic time.", "`CT/internal/scheduler/scheduler.go:499-519`"), + 8: A("SOURCE. MEASURED 2026-11-11: the night's order and the start times on three boxes (see the README's night table).", "`CT/internal/scheduler/scheduler.go:344-347` `EV/box/demo-hp-guest-night.txt`")}, NH), + # ---------------------------------------------------------------- agent (on the host) + ("CA01", "clock-agent", 0, "Whole-guest backup: due-check and run · CUSTOMER", "Night", r"vzdump|whole.guest|due.check", + {6: K(1189, "The age is wall-clock time. A host clock that was ahead during one backup gives a negative age afterwards; the tier reads 'not due' until real time passes it."), + 8: K(1189, "TEST and DOC for the due-check. MEASURED 2026-11-11: demo-hp 11 to 13 minutes, Tester 1 about 3 minutes, the N100 under 1 minute; on the N100 the start time moved 5 minutes later each day. A clock jump has no test.", "`EV/box/demo-hp-host-night.txt` `EV/box/tester-1-host-night.txt` `EV/box/demo-felhom-host-night.txt`")}, NH), + ("CA02", "clock-agent", 1, "OS-update night leg: guest, host, Docker, Proxmox · CUSTOMER", "Night", r"os.?update|OS update", + {6: N("W2", 8, "The 20-hour gap compares the clock with a time read from a file. A stamp that was written while the clock was ahead keeps the leg skipped until real time passes that stamp plus 20 hours.", "`AG/internal/osupdate/leg.go:661-663`"), + 8: N("W2", 8, "TEST and DOC for the normal path. A stamp from the future: NOBODY KNOWS.")}, NH), + ("CA03", "clock-agent", 2, "Kernel lane: night reboot, 20-minute boot judge, one self-revert · CUSTOMER", "Night", r"kernel", + {6: K(1301, "The 20-minute judge runs on monotonic time; a clock step does not move it. Nothing in the installer sets up time keeping, and a host can start with a wrong clock: the N100's host clock is 32 s behind today."), + 8: A("TEST and DOC. The design text itself says that a real freeze was not measured.", "`AR/11-os-updates.md:599-601` `AG/internal/osupdate/kernel_test.go:79`")}, + "YES — `RB/crash-guard.md`"), + ("CA04", "clock-agent", 3, "Crash guard: early-boot unit and the hourly re-arm timer · CUSTOMER", "Watchdog", r"crash.guard|crash guard", + {5: N("W3", 14, "The second unclean boot inside 60 minutes trips it; it arms again after 24 hours. The boot unit and the hourly check share one state file and take no lock."), + 6: N("W2", 22, "The script reads the wall clock at early boot, before any time service runs. A clock that is wrong at boot can move unclean boots into or out of the 60-minute window."), + 8: N("W2", 22, "TEST and DOC for the windows. A wrong clock at boot: NOBODY KNOWS.")}, + "YES — `RB/crash-guard.md`"), + ("CA05", "clock-agent", 4, "Scheduled restore-test evaluation · CUSTOMER", "Night", r"restore.test", + {6: N("W2", 8, "The 'settled' cut-off uses the wall clock. A clock that is set back makes no archive look settled, so no test starts. The ticker itself is monotonic."), + 8: N("W2", 8, "TEST and DOC. MEASURED 2026-11-11: restore tests took 1 to 14 minutes on the three boxes. A clock set back: NOBODY KNOWS.", "`EV/box/demo-hp-host-night.txt`")}, NH), + ("CA06", "clock-agent", 5, "Controller supervisor: restarts a dead or silent controller · CUSTOMER", "Watchdog", r"supervisor", + {3: N("W4", 15, "No time limit on one sweep in the source, and no duration was measured. Nothing grows with data."), + 6: N("W2", 8, "The windows (3 restarts in 15 minutes, the 24-hour counter) use the wall clock in UTC. Nobody tested a clock jump.", "`AG/internal/localapi/controllersupervisor.go:550-552` `AG/internal/localapi/controllersupervisor.go:308`")}, NH), + ("CA07", "clock-agent", 6, "Guest-power watchdog: starts a guest that must run · CUSTOMER", "Watchdog", r"guest.power|guest power", + {6: N("W2", 8, "The next try time is kept in memory from the wall clock. A clock set back delays the next try; an agent restart tries again at once.", "`AG/internal/localapi/guestpower.go:234` `AG/internal/localapi/guestpower.go:253`"), + 7: N("W1", 4, "When the watchdog gives up, it writes one ERROR line in the host's journal. No event goes to the hub. The household's box is down and the outside sign is only the hub's liveness check.", "`AG/internal/localapi/guestpower.go:200-206`"), + 8: N("W1", 4, "TEST for the start and the back-off. Who hears the give-up: NOBODY KNOWS.")}, + "NO — no runbook says what to do after the watchdog gave up"), + ("CA08", "clock-agent", 7, "Hub report and heartbeat loop · CUSTOMER", "Interval", r"heartbeat|poll interval|report loop", + {6: A("The ticker is monotonic. The report loop follows the hub's interval override, clamped to a range. The reconcile engine keeps the interval that it read at the start.", "`AG/internal/hub/loop.go:188-196` `AG/cmd/felhom-agent/main.go:969`")}, NH), + ("CA09", "clock-agent", 8, "Signed jobs: the validity window against the host clock · OPERATOR", "Interval", r"signed job|signed op|not.before|valid", + {4: N("W1", 9, "A job that the agent refuses as expired is not run. The agent's log line is the only trace on the box; no path that tells the hub or the operator 'refused: expired' was found."), + 6: N("W2", 9, "The window is checked against the host's wall clock. A host clock that is behind refuses every job as 'not yet valid'; one that is ahead refuses them as 'expired'. So a box with a wrong clock cannot take an agent update.", "`AG/internal/authz/verifier.go:153-162`"), + 7: N("W1", 9, "The runner logs a finished or failed job on the host. For a job refused on its dates, no message to the hub was found.", "`AG/internal/signedjobs/runner.go:176-190`")}, + "NO — no runbook names a wrong host clock as a cause of refused signed jobs"), + ("CA10", "clock-agent", 9, "Storage watchdog and the 20-second mount re-assert · CUSTOMER", "Watchdog", r"storage watchdog|re.?assert|ReassertGuestBinds", + {3: N("W4", 15, "No time limit on one tick in the source, and no duration was measured. The work grows with the number of drives."), + 6: N("W2", 8, "The back-off uses the wall clock in UTC. A clock set back stretches the back-off; a clock set forward ends it early. The tickers are monotonic."), + 8: N("W4", 15, "TEST for the reconcile and the back-off. The 20-second ticker has no test: NOBODY KNOWS.")}, NH), + ("CA11", "clock-agent", 10, "Weekly guest disk trim · CUSTOMER", "Interval", r"fstrim|trim", + {6: N("W2", 8, "The 'done this week' check uses host-local wall time. An attempt that was stamped while the clock was ahead keeps the trim 'not due' until real time passes it.", "`AG/internal/fstrim/fstrim.go:179-182`"), + 7: A("The operator, on the hub's System page (one cell). A failed trim is a WARN line on the host. Nobody is mailed.", "`HUB/web/system.go:45` `HUB/web/system.go:206` `AG/internal/fstrim/fstrim.go:318-319`"), + 8: N("W2", 20, "TEST for the week anchor. The hour change and a clock jump: NOBODY KNOWS.")}, NH), + ("CA12", "clock-agent", 11, "Weekly upload line test · CUSTOMER", "Interval", r"line.?test|linetest", + {6: N("W2", 8, "The check uses host-local wall time. A success that was stamped from the future blocks the tests until that time.", "`AG/internal/linetest/linetest.go:211-213`"), + 7: A("The operator, on the hub's System page; the cell is judged on the last success, not on the attempt. Nobody is mailed.", "`AR/11-os-updates.md:409-411`")}, NH), + ("CA13", "clock-agent", 12, "PBS datastore verify loop · CUSTOMER", "Interval", r"verify loop|PBS verify|verify", + {5: N("W3", 13, "The verify is not behind the heavy-work gate; a comment in the source says so. It can run beside a whole-guest backup."), + 7: N("W1", 5, "A snapshot that fails the verify is an ERROR line on the host and a field in the report. No hub event or mail for it was found."), + 8: N("W4", 15, "TEST for the record of a failed verify. A verify that takes longer than its 30-minute cadence: NOBODY KNOWS.")}, NH), + ("CA14", "clock-agent", 13, "Janitor: leftover scratch guests and stale backup locks · CUSTOMER", "Interval", r"janitor|stale lock", {}, NH), + ("CA15", "clock-agent", 14, "Guest network watchdog: DHCP client heal · CUSTOMER", "Watchdog", r"guest.?net|DHCP", + {3: N("W4", 15, "No time limit on one probe in the source, and no duration was measured."), + 7: A("The operator, on the hub's Hosts page: the report carries the heal count for each guest. Nobody is mailed.", "`HUB/web/hosts.go:298-397` `AG/internal/guestnet/watchdog.go:166-170`")}, NH), + ("CA16", "clock-agent", 15, "Host networking self-heal · CUSTOMER", "Watchdog", r"self.?heal", {}, NH), + ("CA17", "clock-agent", 16, "Management-plane watchdog timer: keeps SSH reachable · OPERATOR", "Watchdog", r"mgmt.watchdog|management.plane", + {5: A("The script changes nothing when the directory is healthy. MEASURED: the timer is active on the three hosts and fires each 60 seconds. `Persistent=true` in the unit has no effect, because the timer has no calendar line.", "`AG/configs/felhom-mgmt-watchdog.sh:13-15` `AG/configs/felhom-mgmt-watchdog.timer:11-14` `EV/box/demo-hp-host-clock.txt`"), + 8: A("SOURCE only. The script has no test file.", "`AG/configs/felhom-mgmt-watchdog.sh:1-68` `AG/configs/felhom-mgmt-watchdog.timer:3-6`")}, NH), + ("CA18", "clock-agent", 17, "Agent service: restart, start limit, rollback · CUSTOMER", "Watchdog", r"Restart=|start limit|rollback", + {4: A("After the start limit the unit stays failed. Nothing on the host starts it again. The outside sign is the hub's 'host stale' alarm at 45 minutes.", "`AG/configs/felhom-agent.service:41-42` `AR/11-os-updates.md:572-573`"), + 6: N("W5", 17, "The design text says `Restart=always` and a systemd watchdog that kills a hung agent. The unit has `Restart=on-failure` and no watchdog.", "`AR/03-host-agent.md:72` `AG/configs/felhom-agent.service:41-42`"), + 8: N("W5", 17, "MEASURED values and a TEST for the rollback. A hung agent (alive, not working): NOBODY KNOWS; no watchdog restarts it.", "`AG/configs/felhom-agent.service:41-42`")}, + "PARTLY — `RB/break-glass.md` names `reset-failed`; no runbook names the start limit of the agent"), + ("CA19", "clock-agent", 18, "Off-site tunnel loop and the clock-ahead re-key · CUSTOMER", "Interval", r"wgtunnel|re.?key|tunnel", + {3: N("W4", 15, "No time limit on one pass in the source, and no duration was measured. Nothing grows with data.")}, NH), + ("CA20", "clock-agent", 19, "Whole-guest backup cap: rolling windows · CUSTOMER", "Night", r"backup cap|rolling window|cap", + {6: N("W2", 8, "The windows use the wall clock. Successes that were stamped while the clock was ahead stay inside the window longer and can refuse real runs. NOBODY KNOWS.")}, NH), + ("CA21", "clock-agent", 21, "Unsent OS-update report sender · OPERATOR", "Interval", r"unsent", + {7: A("The operator sees the result late: at most 5 minutes after the hub is back. The sender has no alarm of its own.")}, NH), + # ---------------------------------------------------------------- hub (source only) + ("CH01", "clock-hub", 0, "Box liveness: 'your server cannot be reached' · CUSTOMER", "Hub sweep", r"host_stale|liveness|staleness", {}, NH), + ("CH02", "clock-hub", 1, "Missed-backup deadline check (05:00) · CUSTOMER", "Hub daily", r"deadline check|backup.deadline|05:00", + {4: N("W1", 1, "When the hub is down at 05:00, that day's check is skipped. The 'last run' value lives in memory and the next slot is the next day. Nobody notices. The next day's check uses longer limits, so a missed backup is seen one day late.", "`E/hub/cmd/hub/main.go:1069-1112`"), + 6: N("W2", 7, "The check compares the BOX's backup time with the hub's clock. A box whose clock runs ahead looks fresh. No guard in the file. (R-1206 is the same cause seen on the box's own pages.)", "`HUB/monitor/deadline.go:243`"), + 8: N("W1", 1, "TEST for a down box and for 'the same slot never runs twice'. A skipped day: NOBODY KNOWS.")}, + "NO — no runbook names a hub that was down at 05:00"), + ("CH03", "clock-hub", 2, "Mail cooldowns and the weekly outage mail · CUSTOMER", "Hub sweep", r"cooldown", + {3: A("Each mail call has a 10-second limit. The cooldown lists are in memory; an entry is removed when its alarm clears.", "`HUB/notify/dispatcher.go:86` `HUB/notify/dispatcher.go:684` `HUB/notify/dispatcher.go:1182`"), + 6: N("W2", 10, "Cooldowns use the hub's wall clock only. A jump back stretches a quiet period; a jump forward ends it early. Nothing handles it."), + 7: N("W1", 2, "A swallowed operator alarm leaves a 'suppressed' row. A household mail that fails is logged and is not sent again; only operator mails are retried.", "`HUB/notify/dispatcher.go:1091-1094`")}, NH), + ("CH04", "clock-hub", 3, "Retry of a failed operator mail · OPERATOR", "Hub sweep", r"retry", + {4: N("W1", 3, "The retry timers are in memory. A hub restart inside the 21 minutes loses them, and nothing sends the mail later. After the third failure the hub writes one ERROR line.", "`HUB/notify/dispatcher.go:90-102`"), + 6: N("W2", 10, "The timers are durations; the hour change has no effect. A jump of the hub's wall clock: NOBODY KNOWS."), + 7: N("W1", 3, "Nobody is mailed when the retries give up. The trace is the ERROR line and the 'failed' rows in the notification log.", "`HUB/notify/dispatcher.go:90-102` `AR/08-alarm-ladder.md:363`"), + 8: N("W1", 3, "TEST for the retry. A restart during the retry window: NOBODY KNOWS.")}, NH), + ("CH05", "clock-hub", 4, "Kernel mail: 'your box restarts tonight' · CUSTOMER", "Hub sweep", r"kernel", + {8: N("W3", 14, "TESTS for 'no mail, no step'. A mail that goes out late in the day against the box's own night slot: NOBODY KNOWS.")}, NH), + ("CH06", "clock-hub", 5, "OS-update approval tick and the hourly OS alarms · OPERATOR", "Hub sweep", r"approval|ring", + {6: N("W2", 7, "A box's claimed reboot time is allowed 1 minute of error. A larger error of the box's clock: NOBODY KNOWS. (The N100's host is 32 s behind today.)"), + 8: N("W4", 16, "TESTS and DOC. The cost of the pass each minute with 100 boxes: NOBODY KNOWS.")}, NH), + ("CH07", "clock-hub", 6, "Off-site copy too old, and escrow never finished · CUSTOMER", "Hub sweep", r"offsite_stale|off.site.{0,20}stale|48 h", + {4: N("W1", 3, "With the hub down nothing is judged. After a start the state lists in memory are empty. If a tier that is still too old is mailed again after a restart: NOBODY KNOWS."), + 6: N("W2", 7, "The last-success time is the BOX's time, compared with the hub's clock. A box clock that is ahead hides an old copy. A time that cannot be read counts as too old.", "`HUB/monitor/offsite.go:203-207`"), + 8: N("W2", 7, "TESTS for the limits and for a newborn box. A box clock that is ahead: NOBODY KNOWS.")}, NH), + ("CH08", "clock-hub", 7, "Restore-test proof too old · OPERATOR", "Hub sweep", r"restore.test", + {6: N("W2", 7, "First contact is the hub's receive time. The proof times come from the box's reports. An error of the box's clock: NOBODY KNOWS."), + 8: N("W4", 16, "TEST that a restart does not mail again. The cost with 100 boxes (14 days of reports per customer, each minute): NOBODY KNOWS.")}, NH), + ("CH09", "clock-hub", 8, "Off-site clean-up window (20 minutes, weekly) · CUSTOMER", "Hub sweep", r"clean.?up window|prune window|deleting key", + {4: N("W1", 1, "If the hub stops while a window is open, the key line that can delete stays on the storage account until the hub is back; the sweep then closes it. Nobody is told during the gap."), + 6: N("W2", 10, "The window runs on the hub's clock; the box's clock does not decide. A hub clock that jumps back can keep a window open longer: NOBODY KNOWS.")}, NH), + ("CH10", "clock-hub", 9, "Delayed deletion of a set-aside off-site copy · CUSTOMER", "Hub sweep", r"set.aside|delayed delet|superseded", + {3: N("W4", 15, "One remote delete for each due request, in the same sweep as the window. How long a large delete takes: NOBODY KNOWS."), + 6: A("The hub's clock only. The length of the wait is one setting, read at the start.", "`E/hub/cmd/hub/main.go:598-605`")}, NH), + ("CH11", "clock-hub", 10, "Off-site key audit (07:10) · OPERATOR", "Hub daily", r"key audit|07:10", + {3: N("W4", 16, "The audit has a 10-minute limit and reads one customer after the other from the storage account. With 100 boxes the 10 minutes may not be enough; nobody measured it."), + 4: N("W1", 1, "When the hub is down at 07:10, that day's audit is skipped. No catch-up. Nobody notices.", "`E/hub/cmd/hub/main.go:1069-1112`"), + 8: N("W4", 16, "TEST and DOC. The 10-minute limit at fleet size: NOBODY KNOWS.")}, NH), + ("CH12", "clock-hub", 11, "Off-site credential stuck, and its self-heal · OPERATOR", "Hub sweep", r"offsiteheal|credential.{0,30}(stuck|heal)", + {8: A("SOURCE and DOC only. No test name for the timing was found.")}, NH), + ("CH13", "clock-hub", 12, "PBS whole-box credential self-heal · OPERATOR", "Hub sweep", r"pbsdrheal|PBS.{0,30}heal", + {8: N("W4", 15, "SOURCE only. A pass that hangs on the remote side: NOBODY KNOWS; the pass has no time limit.")}, NH), + ("CH14", "clock-hub", 13, "WireGuard peer list push to ep0 · OPERATOR", "Hub sweep", r"wgsync|peer list", + {4: N("W1", 5, "With the hub down ep0 keeps the last list. A push that fails is logged and tried again at the next tick. Nobody is mailed when it fails again and again; a new box then cannot reach the off-site store.", "`HUB/wgsync/reconciler.go:100-102`")}, NH), + ("CH15", "clock-hub", 14, "Box clock ran ahead: the tunnel alarm · OPERATOR", "Hub sweep", r"wg_tunnel|stamped in the future|clock.ahead", {}, NH), + ("CH16", "clock-hub", 15, "The daily scheduler and the hour change · OPERATOR", "Hub daily", r"scheduleDaily|nextDailyRun|D24", + {6: N("W2", 10, "Days of 23 and 25 hours are handled (D24, with tests). While the scheduler sleeps, a jump of the hub's clock does not move the timer: NOBODY KNOWS what then runs when.")}, NH), + ("CH17", "clock-hub", 16, "Hub database snapshot (four slots) · OPERATOR", "Hub daily", r"snapshot", + {6: N("W5", 18, "The slots are safe on the hour-change night. The design text still says 'older than 24 h' and '02:30' in its first paragraph; the source has four slots.", "`AR/05-hub-architecture.md:618-620`")}, + "YES — `RB/RUNBOOK-hub-db-offsite-backup.md`"), + ("CH18", "clock-hub", 17, "Daily prune of old hub rows (04:30) · OPERATOR", "Hub daily", r"prune", + {3: N("W4", 15, "No time limit and no duration in the log. The deletes grow with the reports of each box each day."), + 6: N("W2", 10, "One cut-off is made from the pod's local time and compared with a UTC column; the other uses UTC. The error is 1 or 2 hours on 90 days."), + 8: A("SOURCE only. No test for the cut-off's time zone was found.")}, NH), + ("CH19", "clock-hub", 18, "The other checks on the 60-second sweep · OPERATOR", "Hub sweep", r"60.second|sweep", + {3: N("W4", 15, "All checkers run one after the other in one loop, with no limit for each. A slow checker delays all the others, the liveness check too. Each reads the whole fleet at each pass.", "`E/hub/cmd/hub/main.go:935-962`"), + 8: N("W4", 16, "SOURCE only. How long one whole sweep takes: NOBODY KNOWS.")}, NH), + ("CH20", "clock-hub", 19, "Reminder about a held app · CUSTOMER", "Hub sweep", r"held.app|held app", + {6: N("W2", 7, "The reminder counts from a time that the box reports. A box clock that is behind makes the reminder early; one that is ahead makes it late.")}, NH), + ("CH21", "clock-hub", 20, "'A golden is published but not vouched' · OPERATOR", "Hub sweep", r"vouch", {}, NH), + ("CH22", "clock-hub", 21, "Shared storage boxes: fill and reachability · OPERATOR", "Hub sweep", r"storage box|pool.box|oversub", {}, NH), + ("CH23", "clock-hub", 25, "DooPlex: push of the hub database snapshot off-site · OPERATOR", "DooPlex and ep0", r"hub.db|database snapshot", + {6: K(1293, NM + "The timer's zone and the hub's zone must match for the 30-minute offset. On the autumn hour-change day the 02:00 copy never reaches ep0 and the 20:00 copy is pushed twice."), + 7: N("W5", 18, "The runbook says in one place that the alarm fires after 13 hours, and the alarm rule printed in the same runbook says 26 hours.", "`RB/RUNBOOK-hub-db-offsite-backup.md:100` `RB/RUNBOOK-hub-db-offsite-backup.md:152-153`"), + 8: A(NM + "A test file for the script exists and is run by hand. The live alarm rule is in another repository and on DooPlex; this run did not read it.", "`RB/RUNBOOK-hub-db-offsite-backup.md:107-108` `E/scripts/hub-db-backup/test_hub_db_backup.py:309-316`")}, + "YES — `RB/RUNBOOK-hub-db-offsite-backup.md`"), + ("CH24", "clock-hub", 26, "DooPlex: Sunday restore tests · OPERATOR", "DooPlex and ep0", r"restore.test", + {5: N("W3", 14, NM + "Sunday 04:30 is between the 02:00 snapshot push and the 05:00 PBS pull. The script takes no lock against them."), + 6: N("W2", 20, NM + "On the last Sunday of October the hour 02:00 to 03:00 comes twice. What the timer does at 04:30 that day: NOBODY KNOWS."), + 8: A(NM + "DOC only. This run did not read if the alarm rule is deployed.", "`RB/RUNBOOK-hub-db-offsite-backup.md:152-161`")}, + "YES — `RB/RUNBOOK-hub-db-offsite-backup.md`"), + ("CH25", "clock-hub", 27, "DooPlex: nightly off-site push and the release copy · OPERATOR", "DooPlex and ep0", r"release.copy|release copy", + {3: N("W3", 14, NM + "The units have limits of 90 minutes and 2 hours. A 90-minute run from 00:20 ends at 01:50, after the 01:40 start of the next job. No lock between them."), + 8: A(NM + "SOURCE only. The release-copy timer is switched on by hand; this run did not read if it is on.", "`E/scripts/release-copy/install.sh:4`")}, NH), + ("CH26", "clock-hub", 28, "DooPlex and ep0: PBS copy, prune, verify, clean-up · OPERATOR", "DooPlex and ep0", r"ep0|PBS.{0,20}(prune|sync|pull)", + {3: A(NM + "The PBS jobs are in PBS's own config, not in the repository. They grow with the backup size of every box.", "`RB/ep0-datastore-copy.md:19`"), + 5: N("W3", 14, NM + "The order is by the clock only: prune 07:30, clean-up 08:00. A pull that runs past 07:30 meets the prune: NOBODY KNOWS what then happens."), + 6: N("W3", 14, NM + "Two machines, two clocks. 'The ep0 prune at 03:30 is before the 05:00 pull' is true only while both clocks and both zones agree. No document gives their zones."), + 7: N("W1", 1, NM + "PBS has its own notification target. The clean-up unit has no failure hook and no time limit; a clean-up that fails tells nobody.", "`E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.service:9` `RB/ep0-datastore-copy.md:19`")}, + "YES — `RB/ep0-datastore-copy.md`"), + ("CH27", "clock-hub", 29, "Nightly CI: the gates and the security watch · OPERATOR", "DooPlex and ep0", r"nightly|security.watch|cron", + {4: N("W1", 1, NM + "A missed night is not made up. The watch's weekly summary repairs itself: it is sent when the last one is 8 or more days old. A runner that drops the job tells nobody."), + 8: N("W1", 1, NM + "DOC. A night with no run at all: NOBODY KNOWS who notices.", "`AR/08-alarm-ladder.md:708-722`")}, NH), +] + +M = {i: dict(src=(h, n), name=name, seed=seed, named=("grep", rx), o=o, rb=rb) for i, h, n, name, seed, rx, o, rb in _M} + +# ------------------------------------------------------------------ second pass +# Helper answers that still said "not found" or "candidate" after the first build: the main session's words in their place. +M["CC20"]["o"][2] = A("The sweep runs each 6 hours; the release each hour. A restore copy that nobody used goes after 30 days (the constant, D62). A comment in the wiring still says 14 days; the comment is stale.", "`CT/cmd/controller/main.go:1412`") +M["CH01"]["o"][3] = N("W4", 16, "No time limit. The checkers run one after the other in one loop: one read of all hosts at each pass, then a loop over the hosts. The time grows with the number of boxes. Nobody measured one pass.", "`E/hub/cmd/hub/main.go:935-962` `HUB/monitor/host_staleness.go:113`") +M["CH01"]["o"][4] = A("With the hub down nothing is judged. At the start, a box that is already silent gets one alarm at the first pass, unless that alarm was raised before. The controller's checker starts silent for such a box.", "`HUB/monitor/host_staleness.go:59-68` `HUB/monitor/staleness.go:86-91`") +M["CH02"]["o"][3] = N("W4", 16, "No time limit; the check runs inside the scheduler's own loop. It reads up to 7 days of evidence for every customer. The time grows with boxes times reports. Nobody measured it.", "`E/hub/cmd/hub/main.go:1090`") +M["CH08"]["o"][3] = N("W4", 16, "No time limit. Each pass reads 14 days of host reports for each customer, each minute. It is the heaviest read on the sweep. Nobody measured it.", "`HUB/monitor/restoretest.go:79` `HUB/monitor/restoretest.go:252`") +M["CH16"]["o"][2] = A("Budapest wall-clock time. If the zone cannot be loaded, the fallback is fixed CET, which is one hour off in summer. The pod's zone is set to Europe/Budapest.", "`E/hub/cmd/hub/main.go:1072`") +M["CH16"]["o"][4] = N("W1", 1, "A slot that is missed while the hub is down is not made up; the snapshot and the prune are the two exceptions. A time setting that cannot be read becomes 00:00 with no message.", "`E/hub/cmd/hub/main.go:1069-1112`") +M["CH23"]["o"][2] = A(NM + "02:30, 08:30, 14:30 and 20:30 with up to 2 minutes of random delay, in the host's local time (the unit names no zone).") + +NIGHT = "`EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt`" +MEASURED = { + "CC01": ("MEASURED 2026-11-11, one night, small stores: 2 min 33 s · 40 s · 18 s (demo-hp · Tester 1 · N100)", NIGHT), + "CC02": ("MEASURED 2026-11-11: 39 s · 15 s · 11 s", NIGHT), + "CC03": ("MEASURED 2026-11-11: 3 min 09 s · 59 s · 38 s", NIGHT), + "CC08": ("MEASURED 2026-11-11: under 1 s on each box", NIGHT), + "CC09": ("MEASURED 2026-11-11: 5 s · 5 s · 3 s", NIGHT), + "CC19": ("MEASURED 2026-11-11: the slot ran on each box", NIGHT), + "CA04": ("MEASURED 2026-11-11: the hourly timer is active on the three hosts", "`EV/box/demo-hp-host-clock.txt` `EV/box/tester-1-host-clock.txt` `EV/box/demo-felhom-host-clock.txt`"), +} + +TINY = [ + "session clean-up of dashboard sign-ins on the box (a new sign-in repairs it)", + "old image clean-up after a controller start (log only; deletes nothing when it cannot read its keep list)", + "operator restart and reboot limits on the agent (one hour; an operator action)", + "the hub's purges of log bundles, access log and sessions (nobody is told, nothing depends on them)", + "the hub's version check and catalog template fetch (a WARN line and a stored error; a page shows it)", + "the guest swap check, once after an agent start (2-minute limit)", + "the 14-day hold of alarms after a deliberate power-off (a window, not a job)", +] +LEFT_OUT = [ + ("The agent's local-API certificate (10 years)", "it is an end date: the End-dates list, DB03"), + ("Links and tokens that end by themselves", "they are end dates: the End-dates list, DH22 and the by-ruling list there"), + ("The 14-day countdown after a household stops the off-site copy", "the End-dates list, DB04; its daily sweep is CC19 here"), + ("Debian's and Proxmox's own timers on a host (apt-daily, pve-daily-update, logrotate, fstrim.timer, man-db)", "they are the OS's jobs, not ours; MEASURED present on the three hosts, `evidence/box/*-host-clock.txt`"), + ("TEST-ONLY jobs", "none was found: every job in the table runs on a household's box or for the operator"), +] diff --git a/documentation/audits/clock-list-2026-11-11/tools/resolve-helper.py b/documentation/audits/clock-list-2026-11-11/tools/resolve-helper.py new file mode 100644 index 00000000..13152287 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/tools/resolve-helper.py @@ -0,0 +1,155 @@ +#!/usr/bin/env python3 +"""resolve-helper.py HELPER.md [--json OUT] — open every pointer in a helper's answer BEFORE it may enter the table. + +A helper (a read-only subagent) writes pointers in three shapes: a full one (`CT/internal/stacks/delete.go:624`), a +bare file name (`delete.go:756`) and a short one (`:1025`, "same file as the last one"). This tool + 1. resolves each to a full `PREFIX/path:line[-line]`, + 2. says DEAD when the file or the line does not exist, UNRESOLVED when a bare name fits no file or more than one, + 3. checks the NAME the helper put in front of the pointer (a backticked identifier within 90 characters, with no other + pointer between): it must stand within 3 lines of the cited range. IDENT-OK / IDENT-MISS / NO-IDENT. +It does not judge the claim. A claim that carries a mark in TABLE.md is read by the main session (check-pointers.py +prints the lines). Read-only: it opens files in the git clones on DooPlex and prints. +""" +import json, os, re, sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +REPO = os.path.abspath(os.path.join(HERE, "..", "..", "..", "..")) +WS = os.path.dirname(REPO) +PREFIX = {"AG": os.path.join(WS, "felhom-agent"), "CT": os.path.join(WS, "felhom-controller", "controller"), + "CAT": os.path.join(WS, "app-catalog-felhom.eu"), "HUB": os.path.join(REPO, "hub", "internal"), "E": REPO, + "AR": os.path.join(REPO, "documentation", "architecture"), "RB": os.path.join(REPO, "documentation", "runbooks")} +RANGES = r"~?:?(\d+(?:-\d+)?(?:\s?,\s?:?\d+(?:-\d+)?)*)" +PTR = re.compile(r"(?P(?:%s)/[\w./@+-]+\.\w+)(?::(?P\d+(?:-\d+)?(?:,\s?:?\d+(?:-\d+)?)*))?" + r"|(?[\w.-]+\.(?:go|sh|py|md|yml|yaml|html|json|service|timer))(?::(?P
\d+(?:-\d+)?(?:,\s?:?\d+(?:-\d+)?)*))" + r"|(?\d+(?:-\d+)?)(?![\d:])" % "|".join(PREFIX)) +IDENT = re.compile(r"`([^`]{2,80})`") +_cache, _walk = {}, {} + + +def lines_of(path): + if path not in _cache: + try: + _cache[path] = open(path, encoding="utf-8", errors="replace").read().split("\n") + except OSError: + _cache[path] = None + return _cache[path] + + +def find_base(pre, base): + if pre not in _walk: + idx = {} + for d, dn, fn in os.walk(PREFIX[pre]): + dn[:] = [x for x in dn if x not in (".git", "node_modules", "vendor", "evidence")] + for f in fn: + idx.setdefault(f, []).append(os.path.relpath(os.path.join(d, f), PREFIX[pre])) + _walk[pre] = idx + return _walk[pre].get(base, []) + + +def check(full, rng, text, pos): + pre, rest = full.split("/", 1) + ls = lines_of(os.path.join(PREFIX[pre], rest)) + if ls is None: + return "DEAD", "NO-IDENT" + if not rng: + return "OK", "NO-IDENT" + a, _, b = rng.partition("-") + a, b = int(a), int(b or a) + if b > len(ls): + return "DEAD", "NO-IDENT" + back = text[max(0, pos - 90):pos] + cut = max(back.rfind(":"), back.rfind(";"), back.rfind(",")) # a pointer or a clause break in between: do not reach over it + ids = [m for m in IDENT.finditer(back) if m.start() > cut - 1 or cut < 0] + ids = ids or list(IDENT.finditer(back))[-1:] + if not ids: + return "OK", "NO-IDENT" + name = re.sub(r"[()…]|\.\.\.", "", ids[-1].group(1)).strip().split(" ")[0].split("(")[0] + name = name.split(".")[-1] if re.fullmatch(r"[\w.]+", name) else name + win = "\n".join(ls[max(0, a - 4):b + 3]) + return "OK", ("IDENT-OK" if name and name in win else f"IDENT-MISS({name})") + + +def resolve(text, known, default_pre): + """Returns (text with every pointer rewritten as a backticked full token, [records]).""" + out, recs, last, i = [], [], None, 0 + for m in PTR.finditer(text): + out.append(text[i:m.start()]) + i = m.end() + if m.group("full"): + full, rngs = m.group("full"), m.group("fr") + elif m.group("bare"): + base, rngs = m.group("bare"), m.group("br") + full = known.get(base) + if not full: + c = find_base(default_pre, base) + full = f"{default_pre}/{c[0]}" if len(c) == 1 else None + if not full: + recs.append({"ptr": m.group(0), "file": "UNRESOLVED", "ident": "NO-IDENT"}) + out.append(m.group(0)) + continue + else: + full, rngs = last, m.group("sr") + if not full: + out.append(m.group(0)) + continue + last = full + known.setdefault(os.path.basename(full), full) + toks = [] + for r in re.split(r"\s?,\s?:?", rngs) if rngs else [None]: + st, idn = check(full, r, text, m.start()) + tok = f"{full}:{r}" if r else full + recs.append({"ptr": tok, "file": st, "ident": idn}) + toks.append(f"`{tok}`") + out.append(" ".join(toks)) + out.append(text[i:]) + return "".join(out), recs + + +def main(): + src = sys.argv[1] + text = open(src, encoding="utf-8").read() + default_pre = "CT" if "controller" in src else "AG" if "agent" in src else "HUB" if "hub" in src else "AR" + known = {} + for m in re.finditer(r"((?:%s)/[\w./@+-]+\.\w+)" % "|".join(PREFIX), text): # first pass: every full path the helper named + known.setdefault(os.path.basename(m.group(1)), m.group(1)) + endings, cur, allrecs, lastq = [], None, [], None + for line in text.split("\n"): + h = re.match(r"^### (.+)$", line) + if h: + cur = {"title": h.group(1).strip(), "answers": {}, "extra": []} + endings.append(cur) + lastq = None + continue + if cur is None: + continue + a = re.match(r"^([1-8])\. (.*)$", line) + body = a.group(2) if a else line + new, recs = resolve(re.sub(r"`((?:%s)/[^`]+)`" % "|".join(PREFIX), r"\1", body), known, default_pre) + for r in recs: + r["ending"] = cur["title"] + r["q"] = a.group(1) if a else (lastq or "-") + allrecs += recs + if a: + cur["answers"][a.group(1)] = new + lastq = a.group(1) + elif line.strip() and lastq and not line.startswith(("#", "---")): + cur["answers"][lastq] += " · " + new.strip().lstrip("-* ") # a continuation line belongs to the answer above it + elif line.strip(): + cur["extra"].append(new) + n = len(allrecs) + dead = [r for r in allrecs if r["file"] != "OK"] + miss = [r for r in allrecs if r["ident"].startswith("IDENT-MISS")] + ok = sum(1 for r in allrecs if r["ident"] == "IDENT-OK") + print(f"{os.path.basename(src)}: endings {len(endings)} · pointers {n} · dead/unresolved {len(dead)} · " + f"name found at the line {ok} · name NOT at the line {len(miss)} · no name to check {n - ok - len(miss) - len(dead)}") + for r in dead: + print(f" {r['file']:<10} {r['ptr']} [{r['ending'][:50]} q{r['q']}]") + for r in miss: + print(f" {r['ident']:<28} {r['ptr']} [{r['ending'][:50]} q{r['q']}]") + if "--json" in sys.argv: + json.dump({"endings": endings, "pointers": allrecs}, open(sys.argv[sys.argv.index("--json") + 1], "w"), indent=1) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/documentation/audits/clock-list-2026-11-11/tools/run-box.sh b/documentation/audits/clock-list-2026-11-11/tools/run-box.sh new file mode 100644 index 00000000..161d4f21 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/tools/run-box.sh @@ -0,0 +1,26 @@ +#!/bin/bash +# run-box.sh KIND NAME SSHTARGET — run tools/host-KIND.sh on the host and tools/guest-KIND.sh in guest 9201, read-only. +# Output: box/NAME-host-KIND.txt and box/NAME-guest-KIND.txt beside this folder. +set -u +D=$(cd "$(dirname "$0")/.." && pwd) +k=$1; n=$2; t=$3 +SSH="ssh -o BatchMode=yes -o ConnectTimeout=10" +{ + printf '# host-%s %s taken %s from DooPlex\n' "$k" "$n" "$(date -u +%FT%TZ)" + timeout 240 $SSH "$t" 'bash -s' < "$D/tools/host-$k.sh" + printf 'RC=%s\n' "$?" +} > "$D/box/$n-host-$k.txt" 2>&1 +{ + printf '# guest-%s %s taken %s from DooPlex\n' "$k" "$n" "$(date -u +%FT%TZ)" + timeout 240 $SSH "$t" 'pct exec 9201 -- bash -s' < "$D/tools/guest-$k.sh" + printf 'RC=%s\n' "$?" +} > "$D/box/$n-guest-$k.txt" 2>&1 +if [ "$k" = clock ]; then + { + printf '# offset %s: box clock minus DooPlex clock (midpoint of the call), seconds\n' "$n" + for i in 1 2 3 4 5; do + a=$(date +%s.%N); b=$($SSH "$t" 'date +%s.%N'); c=$(date +%s.%N) + awk -v a="$a" -v b="$b" -v c="$c" 'BEGIN{printf "offset=%.3f rtt=%.3f\n", b-(a+c)/2, c-a}' + done + } > "$D/box/$n-offset.txt" 2>&1 +fi diff --git a/documentation/audits/clock-list-2026-11-11/tools/run-night.sh b/documentation/audits/clock-list-2026-11-11/tools/run-night.sh new file mode 100644 index 00000000..69b6187c --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/tools/run-night.sh @@ -0,0 +1,7 @@ +#!/bin/bash +set -u +D=$(cd "$(dirname "$0")/.." && pwd); n=$1; t=$2 +SSH="ssh -o BatchMode=yes -o ConnectTimeout=10" +{ printf '# host-night %s taken %s\n' "$n" "$(date -u +%FT%TZ)"; timeout 200 $SSH "$t" 'bash -s' < "$D/tools/host-night.sh"; printf 'RC=%s\n' "$?"; } > "$D/box/$n-host-night.txt" 2>&1 +{ printf '# guest-night %s taken %s\n' "$n" "$(date -u +%FT%TZ)"; timeout 200 $SSH "$t" 'pct exec 9201 -- bash -s' < "$D/tools/guest-night.sh"; printf 'RC=%s\n' "$?"; } > "$D/box/$n-guest-night.txt" 2>&1 +tail -1 "$D/box/$n-host-night.txt" "$D/box/$n-guest-night.txt" diff --git a/documentation/audits/clock-list-2026-11-11/tools/searches.sh b/documentation/audits/clock-list-2026-11-11/tools/searches.sh new file mode 100644 index 00000000..040c2d90 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/tools/searches.sh @@ -0,0 +1,43 @@ +#!/bin/bash +# searches.sh — the searches an `N` in TABLE.md must survive. Read-only; prints to stdout (saved as evidence/searches.txt). +# The same tool as audits/end-dates-list-2026-11-11/tools/searches.sh with the Clock list's searches. +# Each search: an ASCII fragment (no accented letter), case-insensitive, over (ar) documentation/architecture, +# (op) backlog/OPEN-ITEMS.md, (cl) backlog/CLOSED-ITEMS.md, (rm) backlog/ROADMAP.md. For register files it prints the row ids +# that match, so a reader can open them. Controls: C+ must hit, C- must not. +E="$(cd "$(dirname "$0")/../../../.." && pwd)" +B="$E/documentation/backlog"; AR="$E/documentation/architecture" +G=/usr/bin/grep +ids() { $G -i -E -- "$1" "$2" 2>/dev/null | $G -o -E '^\| \*\*R-[0-9a-z]+\*\*' | tr -d '|* ' | sort -u | tr '\n' ' '; } +s() { # id, what, pattern + local n_ar; n_ar=$($G -r -i -l -E -- "$3" "$AR" 2>/dev/null | wc -l) + printf '%s | %s | /%s/ | ar files=%s | open: %s| closed: %s| roadmap=%s\n' "$1" "$2" "$3" "$n_ar" "$(ids "$3" "$B/OPEN-ITEMS.md")" "$(ids "$3" "$B/CLOSED-ITEMS.md")" "$($G -c -i -E -- "$3" "$B/ROADMAP.md")" +} +date -u +'searches run %Y-%m-%d %H:%M:%S UTC' +s C+ "positive control (a row that exists)" "R-1301" +s C+2 "positive control (a word in all four)" "backup" +s C- "negative control" "zzqxclocklistzz" +s S01 "a daily hub job missed while the hub is down" "(missed|skipped).{0,60}(slot|check|audit).{0,80}(pod|restart|hub (is|was) down)|catch.?up.{0,80}(hub job|scheduleDaily|daily job)" +s S02 "a failed household mail is not sent again" "(customer|household).{0,30}(mail|e-mail).{0,80}(retry|retried|not sent again)" +s S03 "the operator mail retry is lost at a hub restart" "retry.{0,100}(restart|in memory|lost)|GAVE UP" +s S04 "the guest-power watchdog gives up and tells nobody" "guest.power.{0,160}(give|gave|giving).up|GIVING UP" +s S05 "the peer list push to ep0 fails again and again" "wgsync.{0,120}(fail|alarm|event)|peer list.{0,80}(fail|stale|alarm)" +s S06 "the app-update leg did not run: who is told" "update.leg.{0,120}(did not run|missed|skipped|lost).{0,100}(told|mail|alarm|event|nobody)" +s S07 "a verdict trusts the box's clock (fresh backup, off-site age, proof age)" "(box|host).{0,12}clock.{0,120}(ahead|future|wrong).{0,160}(fresh|stale|latest|backup|proof)|timestamp.{0,60}(in the future|ahead)" +s S08 "an 'already ran' stamp written by a clock that was ahead" "(stamp|already ran|last run|LastOKAt).{0,100}(clock|future|ahead)" +s S09 "a wrong clock refuses every signed job" "(signed|job).{0,80}(not.yet.valid|expired).{0,120}clock|clock.{0,120}(signed job|issued_at|ErrNotYetValid)" +s S10 "the hub's own clock jumps" "hub.{0,20}clock.{0,80}(jump|step|wrong|drift)" +s S11 "the off-site push is skipped because another backup runs" "off-?site.{0,100}skipped.{0,100}(another|running)|another backup is running" +s S12 "the 05:30 proof restore can fall inside the whole-guest backup" "proof.{0,120}(05:30|gate|whole-guest|vzdump|meets)" +s S13 "the PBS verify runs beside a backup" "verify.{0,80}(heavy-op|gate|beside|during a backup)" +s S14 "two DooPlex timers overlap, or two machines' clocks order the jobs" "(timer|slot).{0,80}(overlap|meet|same time)|holds only (if|while) both" +s S15 "an interval job with no time limit" "(time limit|timeout|deadline).{0,80}(sweep|interval job|health check|scan)|slow checker|no per-check" +s S16 "how long the hub's sweep takes with many boxes" "(100|hundred|1000) (boxes|customers|households)|sweep.{0,80}(duration|takes|seconds).{0,60}(boxes|fleet)" +s S17 "the agent unit: Restart=always and a watchdog" "Restart=always|WatchdogSec|systemd watchdog" +s S18 "hub database staleness alarm: 13 h or 26 h" "HubDBBackupStale" +s S19 "a list of every timer" "every timer|list of (all )?(timers|scheduled jobs)|all (timers|scheduled jobs)|job (table|inventory)|timer inventory" +s S20 "the hour change" "DST|clock change|daylight|spring forward|2027-03-28" +s S21 "who keeps the time on a box" "chrony|timesyncd|NTP" +s S22 "the crash guard reads the clock at early boot" "crash.guard.{0,160}(clock|RTC|time is wrong)" +s S23 "the catalog sync fails for a long time" "(catalog|template).sync.{0,100}(fail|stale|days|alarm)" +s S24 "the 05:30 proof restore is not made up" "proof.{0,120}(not made up|catch.?up|missed)" +s S25 "the restore-copy sweep on a box that is on for under six hours" "(sweep|every 6 h).{0,120}(first run|on for under|short uptime)" diff --git a/documentation/audits/clock-list-2026-11-11/tools/table-head.md b/documentation/audits/clock-list-2026-11-11/tools/table-head.md new file mode 100644 index 00000000..2bf1cae1 --- /dev/null +++ b/documentation/audits/clock-list-2026-11-11/tools/table-head.md @@ -0,0 +1,41 @@ +# The Clock list — the table + +Built 2026-11-11 from the source of the four repositories and from read-only measurements on the three test boxes +(Tester 1, demo-hp, the N100; host and guest). One block for each job that a clock starts: the night steps on a box, +the interval jobs, the watchdogs, the hub's daily jobs and its 60-second sweep, and the timers on DooPlex and ep0. +`tools/make-table.py` writes this file from the helpers' answers (`evidence/helpers/`) and from `tools/marks.py`; do +not edit it by hand. + +**NOT MEASURED, by the operator's ruling:** DooPlex and ep0, and the hub's database. What this table says about the +hub's jobs and about the timers on those two machines comes from the repository's own source, scripts and documents. + +**The eight questions:** 1 what runs, who owns it, what starts it (with `CUSTOMER`, `TEST-ONLY` or `OPERATOR`) · +2 when, how often, in what order · 3 how long it takes (measured where a box shows it), its time limit, and 10 or 100 +times the data · 4 what happens if it does not run · 5 it runs late, or twice, or meets another job (the lock is +named) · 6 the clock is wrong or jumps · 7 who is told, and when · 8 how do we know (`MEASURED`, a test, `SOURCE`, a +document, or nobody knows). A ninth line, outside the marks: is the hand step written in a runbook? + +**Every measured duration comes from a small test store** (three boxes, a few apps, one night of controller logs). +No number in this table says how long a job takes on a full household disk. + +**Marks:** `A` answered, with a pointer · `D` decided, not built · `K` known gap, with its row · `N` nobody asked — it +survived the searches in `EV/searches.txt` (S01…S25; controls in the same file). Every `N` names its class row: +**W1** = R-1383 a job does not run, gives up, or loses its state at a restart, and nobody is told · **W2** = R-1384 a +verdict or an "already ran" check trusts a clock that can be wrong · **W3** = R-1385 two jobs can meet with no lock and +no order · **W4** = R-1386 no time limit and no measured duration · **W5** = R-1387 a document and the source disagree +on a time. + +**Who read what.** A cell whose mark is `K` or `N`, and every `A` that `tools/marks.py` carries, holds the main +session's words. Where its pointer was opened by eye, the pointer is written out in `tools/marks.py`; where +`tools/marks.py` shows `+`, the pointer is the helper's and `tools/resolve-helper.py` opened it (the file and the line +exist, and the name the helper gave stands at that line). Every other cell holds a helper's words with the mark `A`. +The main session read every helper answer to questions 4 to 7; it did not open each of their pointers. Treat those +cells as a map, not as proof. + +**Pointers:** `AG/` = `felhom-agent/` · `CT/` = `felhom-controller/controller/` · `CAT/` = `app-catalog-felhom.eu/` · +`HUB/` = `felhom.eu/hub/internal/` · `E/` = `felhom.eu/` · `AR/` = `felhom.eu/documentation/architecture/` · +`RB/` = `felhom.eu/documentation/runbooks/` · `EV/` = this folder's `evidence/`. `tools/check-pointers.py` prints the +text behind each one. + +**Named status** (does an architecture document name this job?): the status comes from a search of the architecture +folder: 0 lines = `UNNAMED`, else `MENTIONED`. It is a machine count, not a reading. diff --git a/documentation/audits/end-dates-list-2026-11-11/MATRIX.md b/documentation/audits/end-dates-list-2026-11-11/MATRIX.md new file mode 100644 index 00000000..be2326db --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/MATRIX.md @@ -0,0 +1,82 @@ +# The End-dates list — the matrix + +Made by `tools/make-matrix.py` from `TABLE.md`. Do not edit it by hand. + +`A` answered · `D` decided, not built · `K` known gap (has a row) · `N` nobody asked. Columns are the eight +questions: 1 what, owner, where · 2 when it ends · 3 who renews, how early · 4 what breaks · 5 who is warned before · 6 a box off past the date · 7 if renewal fails · 8 how do we know. + +| Thing | Named | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | N | +|---|---|---|---|---|---|---|---|---|---|---| +| DB01 Web certificate of the household's address | unnamed | A | A | A | A | **N** | **N** | **N** | A | 3 | +| DB02 Proxmox's own certificate, pinned by the agent | unnamed | A | A | **N** | **N** | **N** | **N** | **N** | A | 5 | +| DB03 The agent's local-API certificate, pinned in the guest | named | A | A | A | A | **N** | A | A | A | 1 | +| DB04 Off-site copy after the household stops it: the 14-day countdown | mentioned | A | A | A | A | A | **N** | A | A | 1 | +| DB05 Off-site history: how far back a household can go | mentioned | A | A | A | A | A | A | K | A | 0 | +| DB06 Off-site credentials: SSH key, host key, store password, PBS pin | mentioned | A | A | A | A | A | A | A | A | 0 | +| DB08 Claim code (72 hours) and the one-use gate token | mentioned | A | A | A | A | A | A | A | A | 0 | +| DB09 Share link | mentioned | A | A | A | A | A | A | A | A | 0 | +| DB11 Signed operator jobs: the validity window | named | A | A | A | A | A | A | A | A | 0 | +| DB12 Operator signing keys on each box | named | A | A | D | A | A | A | K | A | 0 | +| DB13 The agent's Proxmox access token | mentioned | A | A | A | A | A | A | A | A | 0 | +| DB16 Package signing keys: Docker in the guest, Proxmox and Debian on the host | mentioned | A | A | A | A | **N** | **N** | A | A | 2 | +| DB17 Debian 13 and Proxmox 9: the series the product is tied to | mentioned | A | **N** | **N** | A | **N** | A | A | A | 3 | +| DB18 Catalog app images and their recorded digests | mentioned | A | A | A | A | **N** | A | A | A | 1 | +| DB19 Images the controller itself starts (Traefik, tunnel client, file browser, Samba) | mentioned | A | **N** | A | A | **N** | **N** | A | A | 3 | +| DB20 Published agent versions: the newest five stay | named | A | A | A | A | A | A | A | A | 0 | +| DB21 The golden image's age (7 days) and the waiver (14 days) | named | A | A | A | A | A | A | K | A | 0 | +| DB24 Copies kept before a restore: deleted after 30 unused days | mentioned | A | A | A | A | **N** | **N** | A | A | 2 | +| DH01 Connect link in the household's e-mail (7 days) | mentioned | A | A | A | A | A | A | A | A | 0 | +| DH05 'Switched off on purpose' hold (14 days) | mentioned | A | A | A | A | A | A | A | A | 0 | +| DH08 Storage Box sub-account password, and the hub's seal key | mentioned | A | A | D | A | A | A | A | A | 0 | +| DH09 PBS off-site token on ep0 | named | A | A | A | A | A | A | A | A | 0 | +| DH10 WireGuard keys of the box's tunnel to the off-site endpoint | named | A | A | A | A | A | A | A | A | 0 | +| DH14 Recovery mode for a host (30 minutes to 4 hours) | mentioned | A | A | A | A | A | A | A | A | 0 | +| DH15 Break-glass password of a host | mentioned | A | A | A | A | A | A | A | A | 0 | +| DH19 The installer ISO | mentioned | A | A | **N** | A | A | A | A | A | 1 | +| DH20 The certificate that the hub's address serves | mentioned | A | A | A | A | **N** | A | A | A | 1 | +| DH22 Operator access tokens, among them CC's hub token | named | A | A | A | A | **N** | A | A | A | 1 | +| DH23 A household's own Cloudflare token | mentioned | A | **N** | A | A | **N** | A | A | A | 2 | +| DH24 The mail service key (all household and operator mail) | mentioned | A | **N** | A | A | **N** | A | A | A | 2 | +| DH25 The hub's global bearer key | named | A | A | A | A | A | A | A | A | 0 | +| DH26 Pinned tool versions in builds and CI | mentioned | A | A | A | A | A | A | A | A | 0 | +| DH27 Domains and registrars | mentioned | A | **N** | A | A | **N** | A | A | A | 2 | +| DH28 Cloudflare account, the site publish token, the ISO store keys | mentioned | A | **N** | A | A | **N** | A | A | A | 2 | +| DH29 Hetzner account: the off-site endpoint and the storage boxes | mentioned | A | **N** | A | A | **N** | A | A | A | 2 | +| DH30 Tailscale node keys (the operator's remote way in) | mentioned | A | A | A | A | **N** | A | A | A | 1 | +| DH31 Gitea and registry credentials | mentioned | A | **N** | A | A | **N** | A | A | A | 2 | +| DH32 CC's secret store: the tokens inside | mentioned | A | **N** | A | A | **N** | A | A | A | 2 | +| DH33 SSH keys between the hub, DooPlex and the endpoint | mentioned | A | A | A | A | A | A | A | A | 0 | + +## Totals + +- Things: **39** in the table · named **9** · mentioned only **28** · named by no architecture document **2**. +- Answers: **312** · A **268** · D **2** · K **3** · N **39**. +- Things with no N: **19** of 39. + +| Question | A | D | K | N | +|---|---|---|---|---| +| 1 what, owner, where | 39 | 0 | 0 | 0 | +| 2 when it ends | 30 | 0 | 0 | 9 | +| 3 who renews, how early | 34 | 2 | 0 | 3 | +| 4 what breaks | 38 | 0 | 0 | 1 | +| 5 who is warned before | 21 | 0 | 0 | 18 | +| 6 a box off past the date | 33 | 0 | 0 | 6 | +| 7 if renewal fails | 34 | 0 | 3 | 2 | +| 8 how do we know | 39 | 0 | 0 | 0 | + +## The N answers, by class row + +| Class | Row | Count | Members (id-question) | +|---|---|---|---| +| T1 | R-1379 | 18 | DB01-5, DB01-7, DB03-5, DB16-5, DB17-5, DB18-5, DB19-5, DB24-5, DH20-5, DH22-5, DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH30-5, DH31-5, DH32-5 | +| T2 | R-1380 | 5 | DB02-3, DB02-4, DB02-5, DB02-6, DB02-7 | +| T3 | R-1381 | 11 | DB17-2, DB17-3, DB19-2, DH19-3, DH23-2, DH24-2, DH27-2, DH28-2, DH29-2, DH31-2, DH32-2 | +| T4 | R-1382 | 5 | DB01-6, DB04-6, DB16-6, DB19-6, DB24-6 | + +## The K answers, by existing row + +| Row | Count | Members | +|---|---|---| +| R-468 | 1 | DB21-7 | +| R-924 | 1 | DB12-7 | +| R-1294 | 1 | DB05-7 | diff --git a/documentation/audits/end-dates-list-2026-11-11/README.md b/documentation/audits/end-dates-list-2026-11-11/README.md new file mode 100644 index 00000000..3fd2ebe8 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/README.md @@ -0,0 +1,97 @@ +# The End-dates list — 2026-11-11 + +The fourth list of the gap review. One block for each thing that stops working, or stops being safe, on a date or after +an age, with eight questions each. Built from the source of four repositories, from read-only measurements on the three +test boxes, and from what any visitor can read with no login. **No document lists every end date** — the claim in the +brief is true (search S18, 0 hits). DooPlex, ep0 and every account are `NOT MEASURED`, by ruling. +Files: `TABLE.md` (the answers), `MATRIX.md` (the marks), `evidence/`, `tools/`. + +## Five lines + +1. **Things found: 58** helper blocks. 39 are in the table. The others are tiny (8), or the same thing seen from the + other side, or not an end date (all are named in the table's head). +2. **Things no architecture document names: 2** of 39 by the search; 28 more are only mentioned. No document gives a + date for any of them except one Tailscale key. +3. **Marks (312 answers):** `A` 268 · `D` 2 · `K` 3 · `N` 39. +4. **The worst `N`:** each box's agent trusts one exact Proxmox certificate. Proxmox makes a new one by itself about 14 + days before the old one ends. The agent does not follow. From that day the agent cannot start, stop, back up or + restore a guest. First box: the N100, about **2028-07-02**; then demo-hp and Tester 1 within three months. + Every box installed in the same season fails in the same fortnight (DB02). +5. **The operator decides** the two questions at the end of this page, and does the two "do" items. + +## Ends in the next 90 days (to 2027-02-09), soonest first + +| Ends | What | Who renews | Renewal proven? | +|---|---|---|---| +| **2026-11-10 (past)** | The golden image's 7-day age limit | You: bake and vouch (Sunday) | No. The gate fails today; code pushes to `felhom.eu` are refused | +| each 7 days | Debian's package lists (`Valid-Until`) | Debian's servers, at each fetch | Yes, measured rolling on three boxes | +| 2026-12-23 | Certificate of the download address (`files.felhom.eu`) | DooPlex, by itself | Not measured | +| 2026-12-27 | Certificate of the ISO address (`iso.felhom.eu`) | Cloudflare, by itself | Not measured | +| 2026-12-31 | Web certificate of the N100's household address | The box (Traefik), by itself | On demo-hp yes; on this box not yet | +| 2027-01-02 | Web certificate of Tester 1's household address | The box, by itself | As above | +| 2027-01-02 | Cloudflare's edge certificate for the N100's name | Cloudflare, by itself | Not measured | +| 2027-01-03 | Certificate of the Gitea address | DooPlex, by itself | Not measured | +| 2027-01-12 | Certificate of the hub's address | DooPlex, by itself | One renewal seen from outside (issued 2026-10-14) | +| 2027-01-17 | Tailscale key of demo-hp (date from the document) | **You**, in the Tailscale console | No. Tailscale is not running on demo-hp today | +| 2027-01-19 | Web certificate of demo-hp's household address | The box, by itself | Yes: issued again 2026-10-21 | +| 2027-01-19, -20 | Cloudflare's edge certificates for the two other household names | Cloudflare, by itself | Not measured | +| 2027-01-28 | Certificate of the website | Cloudflare, by itself | Not measured | +| not known | CC's hub token (90 days from its making) | **You**: make a new one, put it in the store | No. The date is on the hub's token page only | + +Not in this table: two Debian keys that end 2027-02-03 are in the "removed keys" file; nothing uses them. + +## Only you can read these dates + +| What | Where you read the date | +|---|---| +| The domains: `felhom.eu`, `demo-felhom.eu`, `enkisfelhom.hu`, `enkicsifelhom.hu`, `dooplex.hu` | Each registrar's account page (this run could not read the public records) | +| Cloudflare: the account's plan, the site publish token, the ISO store keys | Cloudflare dashboard: the API tokens page, and the R2 tokens page | +| Hetzner: the payment state, the API token, the storage boxes | Hetzner console: the security page (tokens) and the invoices | +| The mail service key and plan | The mail service dashboard: the API keys page | +| Tailscale keys of the nodes | Tailscale console: the Machines page | +| CC's hub token, and every other operator token | The hub: the access tokens page | +| The GitHub advisory token of the nightly security watch | GitHub: Settings, Developer settings, tokens | +| Gitea tokens (the scoped token, CC's token) | Gitea: Settings, Applications | +| A tester household's own Cloudflare token | That household's Cloudflare dashboard | +| The day security fixes stop for Debian 13 and for Proxmox 9 | The public release pages of Debian and Proxmox | + +## The `N` answers, by class (one register row each) + +| Class | Row | What | Members | +|---|---|---|---| +| T1 | R-1379 | An end exists and nobody is warned before it | web certificate (DB01-5, -7) · hub address certificate (DH20-5) · operator tokens (DH22-5) · Tailscale key (DH30-5) · package keys (DB16-5) · restore copy (DB24-5) · OS support (DB17-5) · app images (DB18-5, DB19-5) · mail key, domains, accounts, tokens (DH23/24/27/28/29/31/32-5) · local-API certificate (DB03-5) | +| T2 | R-1380 | Another party renews and our pin does not follow | Proxmox's certificate and the agent's pin (DB02-3 to -7) | +| T3 | R-1381 | No document says when it ends, or where to read the date | OS support (DB17-2, -3) · tunnel client (DB19-2) · installer ISO (DH19-3) · the outside accounts and tokens (DH23/24/27/28/29/31/32-2) | +| T4 | R-1382 | A box that was off past the date: nobody knows if it comes back | web certificate (DB01-6) · package keys (DB16-6) · old tunnel client (DB19-6) · restore copy (DB24-6) · the 14-day countdown (DB04-6) | + +## No end date, or a fixed window, by ruling (each is an `A`) + +The share link (no end) · operator signing keys (no end; a change is designed, DB12) · the agent's Proxmox token (no +end, measured) · the local-API certificate (10 years) · WireGuard keys (no end) · the break-glass password (paper: each +year) · the off-site history window (7 daily, 4 weekly, monthly) · the 14-day countdown after a household stops the +off-site copy · restore copies, 30 days (D62) · the newest five agent versions (D50) · the golden's 7 days and the +waiver's 14 (decision 250) · the 72-hour claim code · the 7-day connect link · signed jobs, 30 minutes to 24 hours. + +## Decisions for the operator (also in `STATUS.md`) + +1. **Shall the agent follow Proxmox's new certificate by itself?** Yes (CC's pick): the agent reads the certificate file + on its own host and takes the new fingerprint; one agent change, long before 2028. No: a hand step in a runbook and a + date in your calendar for each box; the cost is that it depends on memory, on every box, for ever. If nothing + happens: in July 2028 the first box loses its backups and guest control, and no alarm says why. +2. **Shall the product warn before an end date?** Yes (CC's pick): the box reads its web certificate's date and the hub + reads its tokens' dates; an alarm goes to you 14 days before. No: you keep the dates in a calendar by hand; the cost + is that the list above is the only guard. If nothing happens: a renewal that fails is silent until the day. + +**Do, not decide:** (a) bake and vouch a golden, or write a dated waiver; (b) read the dates of the table "Only you can +read these dates" once, and write them in one place. + +## Method verdict (fourth list) + +- **It worked, and the measurement found the worst gap.** Two read-only helpers gave 58 raw things. The pin gap (DB02) + is in no helper answer as a date: it came from reading the certificate's date on the boxes and then Proxmox's own + renewal rule. +- **Read by eye: 85 of 312 cells** (every `D`, `K` and `N`, and the 41 `A` cells that `tools/marks.py` carries). The main session + read all helper answers to questions 4 to 7, and it did not open each of their pointers; a tool opened all 259. +- **The weak point:** for an outside account a list from documents can only say "no document says". Ten lines of this + list wait for a date that only the operator can read. +- **The ninth line:** 11 things have a hand step that no runbook holds; 2 more have it in part. diff --git a/documentation/audits/end-dates-list-2026-11-11/TABLE.md b/documentation/audits/end-dates-list-2026-11-11/TABLE.md new file mode 100644 index 00000000..b47e2cf8 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/TABLE.md @@ -0,0 +1,724 @@ +# The End-dates list — the table + +Built 2026-11-11 from the source of the four repositories, from read-only measurements on the three test boxes +(Tester 1, demo-hp, the N100; host and guest) and from what any visitor can read with no login. One block for each +thing that stops working, or stops being safe, on a date or after an age. `tools/make-table.py` writes this file from +the helpers' answers (`evidence/helpers/`) and from `tools/marks.py`; do not edit it by hand. + +**NOT MEASURED, by the operator's ruling of 2026-11-10:** DooPlex and ep0, and every account (Cloudflare, Hetzner, the +mail service, the registrars). What this table says about them comes from the repository's own scripts and documents. + +**The eight questions:** 1 what it is, who owns it, where it lives · 2 when it ends (a real date for each box where one +was measured, else the rule) · 3 who renews it, and how long before · 4 what breaks, and for whom · 5 who is warned +before the end · 6 a box that was off past the end date: does it come back by itself? · 7 what happens if the renewal +fails · 8 how do we know (`MEASURED`, a test, `SOURCE`, a document, or nobody knows). +A ninth line, outside the marks: is the hand step written in a runbook? + +**Marks:** `A` answered, with a pointer · `D` decided, not built · `K` known gap, with its row · `N` nobody asked — it +survived the searches in `EV/searches.txt` (S01…S23; controls in the same file). Every `N` names its class row: +**T1** = R-1379 an end exists and nobody is warned before it · **T2** = R-1380 another party renews and our pin does +not follow · **T3** = R-1381 no document says when it ends, or where to read the date · **T4** = R-1382 a box that was +off past the date: nobody knows if it comes back. + +**Who read what.** A cell whose mark is `D`, `K` or `N`, and every `A` that `tools/marks.py` carries, holds the main +session's words: the pointer was opened and read. Every other cell holds a helper's words with the mark `A`: +`tools/resolve-helper.py` opened each of its pointers (the file and the line exist, and the name the helper gave stands +at that line). The main session read every helper answer to questions 4 to 7; it did not open each of their pointers. +Treat those cells as a map, not as proof. + +**Pointers:** `AG/` = `felhom-agent/` · `CT/` = `felhom-controller/controller/` · `CAT/` = `app-catalog-felhom.eu/` · +`HUB/` = `felhom.eu/hub/internal/` · `E/` = `felhom.eu/` · `AR/` = `felhom.eu/documentation/architecture/` · +`RB/` = `felhom.eu/documentation/runbooks/` · `EV/` = this folder's `evidence/`. `tools/check-pointers.py` prints the +text behind each one. + +**Named status** (does an architecture document name this end date?): `NAMED` and `MENTIONED` with a pointer were read +in the documents. Where the line says "a machine count", the status comes from a search of the architecture folder: +0 lines = `UNNAMED`, else `MENTIONED`. + +## The kinds, and where each thing is + +| Kind | Things (worst mark) | Named · mentioned · named nowhere | +|---|---|---| +| Certificate | DB01 (N), DB02 (N), DB03 (N), DH20 (N) | 1 · 1 · 2 | +| Key | DB06 (A), DB12 (K), DB13 (A), DB16 (N), DH08 (D), DH09 (A), DH10 (A), DH15 (A), DH22 (N), DH23 (N), DH25 (A), DH30 (N), DH31 (N), DH32 (N), DH33 (A) | 5 · 10 · 0 | +| Window | DB04 (N), DB05 (K), DB08 (A), DB09 (A), DB11 (A), DB24 (N), DH01 (A), DH05 (A), DH14 (A) | 1 · 8 · 0 | +| Version | DB18 (N), DB19 (N), DB20 (A), DB21 (K), DH19 (N), DH26 (A) | 2 · 4 · 0 | +| Account | DH24 (N), DH27 (N), DH28 (N), DH29 (N) | 0 · 4 · 0 | +| OS | DB17 (N) | 0 · 1 · 0 | + +## Not in the table + +**Tiny (named once, not tabled):** login sessions on the box and on the hub (hours to 7 days; a new login repairs it) · the one-use codes of the recovery ceremony (minutes) · the guest's token for the agent and the hub key (no date) · pinned build tools of the controller (a build fails loudly) · the hub's claim code and pairing code (no date; three resets a day) · operator actions in the queue (24 hours; a restart order 30 minutes) · the vouch alarm (2 hours) · the Debian template that the golden is built from (a bake fails loudly; the operator changes the default). + +**Left out for another reason:** + +- The recovery passphrase — it stops on an event (a new one is made), not on a date. +- Kept escrow packages — they have no end date by ruling; the 14-day countdown that removes one is DB04. +- The off-site clean-up window (20 minutes, 6.5 days) — it is a timer, so it is on the Clock list. +- The hub's view of signed jobs, signing keys, the golden's age, the agent versions, and OS support — the same things as DB11, DB12, DB21, DB20 and DB17. +- WireGuard re-key on the box — the same thing as DH10. + +--- + +### DB01 — Web certificate of the household's address · Certificate · UNNAMED + +architecture folder: 0 lines in 0 files match `/acme\.json|Let.s Encrypt|letsencrypt/` (a machine count, not a reading) · helper section: dates-box „1. Web certificate of the household's address (Let's Encrypt through Traefik)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The public TLS certificate for the dashboard and apps. Product-owned Traefik asks Let's Encrypt with the household's e-mail `ACMEEmail`; stored in `acme.json`. | A | `CT/internal/stacks/infra.go:130` `CT/internal/infra/templates/traefik.yml.tmpl:51` | +| 2 | MEASURED 2026-11-11: the N100's certificate ends 2026-12-31, Tester 1's 2027-01-02, demo-hp's 2027-01-19. The life (90 days) is the issuer's rule, not ours. | A | `EV/box/demo-felhom-guest-dates.txt` `EV/box/tester-1-guest-dates.txt` `EV/box/demo-hp-guest-dates.txt` | +| 3 | Traefik renews by itself, by its own default lead time. MEASURED on demo-hp: it got a new certificate on 2026-10-21, 90 days before the end date 2027-01-19. | A | `CT/internal/infra/templates/traefik.yml.tmpl:53` `CT/internal/infra/templates/traefik.yml.tmpl:59` | +| 4 | Browsers show a certificate error on every app address of the household. With no e-mail set Traefik serves a self-signed certificate, see comment at `TraefikData`. | A | `CT/internal/infra/infra.go:111-118` | +| 5 | Nobody. No part of the product reads the certificate's end date. If the renewal fails (the household's Cloudflare token ends, or the name check fails), the first sign is a browser warning on the end date. | N | `CT/internal/infra/infra.go:157-159` · class T1 · S01 | +| 6 | NOBODY KNOWS. A box that was off past the end date starts with a dead certificate. Traefik must get a new one at the start. Nobody has tested this. | N | `CT/internal/infra/infra.go:157-159` · class T4 · S11 | +| 7 | Traefik tries again by itself. The old certificate works until its end date. Nothing in the product reads the result, so a renewal that fails for weeks is silent. | N | `CT/internal/infra/templates/traefik.yml.tmpl:51` · class T1 · S01 | +| 8 | MEASURED: the three end dates, and one renewal by the product on demo-hp (the certificate was issued again on 2026-10-21) · SOURCE for the setup; NOBODY KNOWS for renewal timing, warning and the switched-off case. | A | `EV/box/demo-hp-guest-dates.txt` `CT/internal/infra/templates/traefik.yml.tmpl:48-59` | + +**Hand step in a runbook?** NO — no runbook says what to do when a household's certificate does not renew + +### DB02 — Proxmox's own certificate, pinned by the agent · Certificate · UNNAMED + +architecture folder: 0 lines in 0 files match `/pve-ssl|tls\.fingerprint|TLS pin/` (a machine count, not a reading) · helper section: dates-box „2. Proxmox web certificate pinned by the agent” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The agent talks to Proxmox on the host and trusts ONE certificate by fingerprint. The installer reads the served leaf once, `fp=`; the agent checks it in `TLSConfig.build`. Product-owned. | A | `E/scripts/felhom-host-install.sh:3182-3185` `AG/internal/proxmox/tls.go:39-59` | +| 2 | MEASURED 2026-11-11: the certificate ends 2028-07-16 on the N100, 2028-08-19 on demo-hp, 2028-10-02 on Tester 1. On all three the agent's pin equals this certificate. The pin itself has no date. | A | `EV/box/pin.txt` `AG/internal/proxmox/tls.go:40-58` | +| 3 | Proxmox renews it by itself, about 14 days before the end (its daily update job). No agent code reads the new fingerprint. The installer writes the pin once. | N | `EV/box/pveupdate-excerpt.txt` `AG/internal/proxmox/tls.go:40-58` `E/scripts/felhom-host-install.sh:3182-3185` · class T2 · S02 | +| 4 | After Proxmox makes the new certificate, every call from the agent to Proxmox fails with 'TLS pin mismatch'. That is: guest start and stop, whole-guest backups, restore tests, disk work. First box: the N100, about 2028-07-02. The other two follow within three months. | N | `AG/internal/proxmox/tls.go:55` `EV/box/pin.txt` · class T2 · S02 | +| 5 | Nobody before the day. No code compares the pin with the certificate's end date. | N | `AG/internal/proxmox/tls.go:40-58` · class T2 · S02 | +| 6 | No. Proxmox also renews a certificate that has ended, at its next daily job. The box comes back with a new certificate and the old pin. | N | `EV/box/pveupdate-excerpt.txt` · class T2 · S02 | +| 7 | Here the renewal that works is the failure. The repair is by hand: write the new fingerprint into the agent's config and restart the agent. No code does it. | N | `AG/internal/proxmox/tls.go:40-58` · class T2 · S02 | +| 8 | MEASURED: the three end dates, pin equals certificate on all three, and Proxmox's own renewal rule read on demo-hp · TEST `TestTLSConfig_Build` for the pin check; NOBODY KNOWS for the renewal case. | A | `EV/box/pin.txt` `EV/box/pveupdate-excerpt.txt` `AG/internal/proxmox/tls_test.go:28` | + +**Hand step in a runbook?** NO — a search of the runbooks for 'tls.fingerprint', 're-pin' and 'pin mismatch' finds nothing + +### DB03 — The agent's local-API certificate, pinned in the guest · Certificate · NAMED + +architecture: `AR/03-host-agent.md:420-423` · helper section: dates-box „3. Agent local-API certificate and its pin in the guest” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The self-signed leaf the agent serves to the in-guest controller, made by `generateLeaf`. The controller pins its fingerprint in `New`. Product-owned, on the host. | A | `AG/internal/localapi/cert.go:56` `CT/internal/agentapi/client.go:105` | +| 2 | `certValidity = 10 * 365 * 24 * time.Hour`, set as `NotAfter`. So a box installed in 2026 has a leaf that ends in 2036. | A | `AG/internal/localapi/cert.go:25` `AG/internal/localapi/cert.go:70` | +| 3 | Nobody: `EnsureLeaf` loads the existing pair and never looks at its date. The comment says rotation is an operator action for a later slice. | A | `AG/internal/localapi/cert.go:38-52` `AG/internal/localapi/cert.go:21-24` | +| 4 | At the end date nothing breaks: the controller compares the fingerprint only, not the date. A NEW certificate is what breaks the link. | A | `CT/internal/agentapi/client.go:118-128` `CT/internal/agentapi/client.go:125` | +| 5 | Nobody. The certificate lives 10 years and nothing reads its end date. The first one ends about 2036. | N | `AG/internal/localapi/cert.go:25` · class T1 · S01 | +| 6 | Yes, by itself: the leaf is generated once and kept, and the reinstall path keeps it with `--preserve-state-from`. | A | `AR/03-host-agent.md:420-423` `E/scripts/felhom-host-install.sh:2749-2786` | +| 7 | No renewal exists, so no failure state. A reinstall that makes a new leaf needs `--allow-new-leaf` and then the guest's pin is stale. | A | `E/scripts/felhom-host-install.sh:2749-2786` | +| 8 | TEST `TestEnsureLeaf_StableFingerprintAcrossReload` and `TestClient_WrongPinFailsClosed`; the 10-year end itself: SOURCE only. | A | `AG/internal/localapi/cert_test.go:12` `CT/internal/agentapi/client_test.go:55` | + +**Hand step in a runbook?** no hand step + +### DB04 — Off-site copy after the household stops it: the 14-day countdown · Window · MENTIONED + +architecture folder: 21 lines in 9 files match `/14.day countdown|abandon/` (a machine count, not a reading) · helper section: dates-box „4. Off-site copy after the household stops it: 14-day countdown” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | When the household abandons the off-site backup, the copies are kept for a grace period and then deleted. `startAbandonCountdown`. The household starts it; the product runs it. | A | `CT/internal/backup/offbox_abandon.go:68-70` | +| 2 | `abandonGraceDays = 14`. | A | `CT/internal/backup/offbox_abandon.go:31` | +| 3 | The household can stop or extend it; extending never shortens, test `TestD1_ExtendNeverShortensTheCountdown`. | A | `CT/internal/backup/offbox_abandon_levers_d1_test.go:20` | +| 4 | The off-site copies are gone for good; the household loses its only copy outside the house. The deleting side is the hub (see the hub helper); test name `TestD1_ExtendRefusedWhileTheDeletionIsTheHubs`. | A | `CT/internal/backup/offbox_abandon_levers_d1_test.go:44` | +| 5 | The household, at 5, 3 and 1 days left: `AbandonRemindAtDays`. | A | `CT/internal/backup/offbox_abandon.go:40` | +| 6 | NOBODY KNOWS for a box that is off during the whole countdown: the reminders at 5, 3 and 1 days come from the box. Nobody traced if the old store is removed with no reminder shown. | N | `CT/internal/backup/offbox_abandon.go:31-40` · class T4 · S23 | +| 7 | A failed hub cancel is reported, not hidden: `TestD1_StopReportsFailureWhenTheHubCancelFails`. | A | `CT/internal/backup/offbox_abandon_levers_d1_test.go:69` | +| 8 | TEST `TestR241_ScenarioE_AbandonStartsAReversibleCountdown`. | A | `CT/internal/backup/offbox_abandon_r241_test.go:48` | + +**Hand step in a runbook?** no hand step + +### DB05 — Off-site history: how far back a household can go · Window · MENTIONED + +architecture folder: 13 lines in 5 files match `/keepDaily|keep-daily|7 daily/` (a machine count, not a reading) · helper section: dates-box „5. Off-site backup history: how far back a household can go” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The restic off-site store keeps a thinning history. Policy in `retentionPolicy`. Product rule, household's data. | A | `CT/internal/backup/offbox_window.go:81-82` | +| 2 | `keepDaily = 7`, `keepWeekly = 4`, `keepMonthly = 6`. Older than about six months: gone. | A | `CT/internal/backup/offbox_window.go:47` `CT/internal/backup/offbox_window.go:48` `CT/internal/backup/offbox_window.go:49` | +| 3 | Nobody renews; each run forgets by the same rule, `retentionPolicy`. | A | `CT/internal/backup/offbox_window.go:81-82` | +| 4 | A file deleted or damaged more than six monthly points ago cannot be brought back from off-site. On-host copies are fewer: `KeepLast`, default 3 per comment. | A | `AG/internal/config/config.go:436` `AG/internal/config/config.go:409` | +| 5 | Nobody, and by ruling: the window is the rule. A point leaves it with no warning. | A | `CT/internal/backup/offbox_window.go:81-82` (the code path of question 1) | +| 6 | Yes. A box that was off simply has no points for that time; the next run applies the same rule. PBS copies are not pruned by the box by default, comment. | A | `AG/internal/backup/runner.go:51-58` | +| 7 | A clean-up that the guard refuses leaves more history than the rule says, not less. | K | R-1294 | +| 8 | SOURCE. | A | `CT/internal/backup/offbox_window.go:47-49` | + +**Hand step in a runbook?** no hand step + +### DB06 — Off-site credentials: SSH key, host key, store password, PBS pin · Key · MENTIONED + +architecture folder: 37 lines in 9 files match `/host key|known_hosts|sub-account/` (a machine count, not a reading) · helper section: dates-box „6. Off-site credentials: SSH key, host key, store password, PBS pin” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The box reaches its off-site store with an SSH key and a pinned host key, `StrictHostKeyChecking=yes`, and a password file `RESTIC_PASSWORD_FILE`. PBS is pinned in `pinnedTLS`. | A | `CT/internal/backup/offbox.go:345` `CT/internal/backup/offbox.go:898` `AG/internal/pbs/pin.go:17-30` | +| 2 | NO END DATE in box code: no expiry field on the key, the host key or the password (grep `expir` in found none). The PBS pin ignores the date, `VerifyPeerCertificate`. The end comes from outside: the provider changes its host key or the hub ends the account. | A | `CT/internal/backup/offbox.go` `AG/internal/pbs/pin.go:24` | +| 3 | Nobody on a schedule. A new PBS descriptor is checked before use: `probeFP`. | A | `AG/internal/pbsdr/manager.go:375-379` | +| 4 | If the provider's host key changes, every off-site run fails at the SSH step, `StrictHostKeyChecking=yes`. The household has no off-site copy from that night on. | A | `CT/internal/backup/offbox.go:895` | +| 5 | Nobody, because there is no date. A failed night run is the sign. | A | `CT/internal/backup/offbox.go:345` `CT/internal/backup/offbox.go:898` (the code path of question 1) | +| 6 | Yes if nothing changed outside. A guest that was built again gets a new store password and cannot open the old store; the 14-day countdown (DB04) is the way out. | A | `CT/internal/backup/offbox.go:910` | +| 7 | PBS: state `verify_failed` and a retry at the next tick, `probeFP`. Restic: the run fails and the next night tries again with the same pins. | A | `AG/internal/pbsdr/manager.go:375-379` | +| 8 | SOURCE; provider-side dates: NOBODY KNOWS from the box. | A | `CT/internal/backup/offbox.go:345` | + +**Hand step in a runbook?** yes — `RB/offsite-endpoint.md` + +### DB08 — Claim code (72 hours) and the one-use gate token · Window · MENTIONED + +architecture folder: 12 lines in 8 files match `/claim code|claimCodeTTL|72 h/` (a machine count, not a reading) · helper section: dates-box „8. Claim code and the one-use gate token” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The one-time code a household uses to set its own password, and the short token that carries a pass between addresses. Product-made. `claimCodeTTL`; `gateTokenLife`. | A | `CT/internal/web/claim.go:30` `CT/internal/web/setup_gate.go:47` | +| 2 | `claimCodeTTL = 72 * time.Hour`; `gateTokenLife = 60 * time.Second`, also stated in. | A | `CT/internal/web/claim.go:30` `CT/internal/web/setup_gate.go:47` `AR/01-topology-and-trust.md:192` | +| 3 | The operator sends a new code (hub side, see the hub helper). Nothing on the box mints one by itself. | A | `CT/internal/web/claim.go:30` `CT/internal/web/setup_gate.go:47` (the code path of question 1) | +| 4 | A household that opens the mail after three days cannot set its password until a new code arrives. Checked against the controller's clock `s.claimNow`. | A | `CT/internal/web/claim.go:333` | +| 5 | Nobody before the end. The household sees the dead code at the form, and the operator sends a new one. | A | `CT/internal/web/claim.go:30` `CT/internal/web/setup_gate.go:47` (the code path of question 1) | +| 6 | A box off for more than 72 hours comes back with a dead code; first failing step is the claim form,. | A | `CT/internal/web/claim.go:333` | +| 7 | Wrong tries lock the form for a time: `claimLockoutWindow`. | A | `CT/internal/web/claim.go:33` | +| 8 | TEST `TestClaimSubmit_ExpiredCodeRefused`. | A | `CT/internal/web/claim_test.go:189` | + +**Hand step in a runbook?** no hand step + +### DB09 — Share link · Window · MENTIONED + +architecture folder: 5 lines in 4 files match `//s/|share token|LauncherShareToken/` (a machine count, not a reading) · helper section: dates-box „9. Share link `/s/`” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The link a household gives to guests to see its app tiles. Household-owned, stored as `LauncherShareToken`. | A | `CT/internal/settings/settings.go:45` | +| 2 | NO END DATE: the token has no expiry field; it ends only when the household rotates or disables it, `SetLauncherShareToken`. | A | `CT/internal/settings/settings.go:1111` | +| 3 | The household, by hand. No schedule. | A | `CT/internal/settings/settings.go:45` (the code path of question 1) | +| 4 | Nothing ends. A link that was given away years ago still opens the page, until the household changes it. | A | `CT/internal/web/share.go:31` | +| 5 | Nobody, because there is no date. The link ends only when the household changes it. | A | `CT/internal/settings/settings.go:45` (the code path of question 1) | +| 6 | Yes, unchanged. | A | `CT/internal/settings/settings.go:45` (the code path of question 1) | +| 7 | No renewal to fail. | A | `CT/internal/settings/settings.go:45` (the code path of question 1) | +| 8 | TEST `TestShareGuest_RotateAndDisable`. | A | `CT/internal/web/share_test.go:256` | + +**Hand step in a runbook?** no hand step + +### DB11 — Signed operator jobs: the validity window · Window · NAMED + +architecture: `AR/04-control-plane-authorization.md:139-150` · helper section: dates-box „11. Signed operator jobs: the validity window” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | An order signed by the operator (update the agent, apply OS packages, destructive acts). Operator-owned; it carries a nonce and an expiry,. Checked by `Verify` and by the root wrapper. | A | `AR/03-host-agent.md:268` `AG/internal/authz/verifier.go:104` | +| 2 | At most `MaxWindow = 24 * time.Hour`; the signer's default is 30 minutes, `-ttl`. Wrapper: `MAX_WINDOW_S = 24 * 3600` AG/configs/felhom-os-apply:194. Design:. | A | `AG/internal/authz/verifier.go:26` `AG/cmd/felhom-opsign/main.go:57` `AR/04-control-plane-authorization.md:78-84` | +| 3 | The operator signs a new one. There is no renewal of a signed job. | A | `AR/03-host-agent.md:268` `AG/internal/authz/verifier.go:104` (the code path of question 1) | +| 4 | The job is refused and cleared; the executor is never called,. The box stays on its old agent or old packages. `ErrExpired`. | A | `AG/internal/signedjobs/runner.go:163-170` `AG/internal/authz/verifier.go:161-162` | +| 5 | Nobody before. A job that ends with no result is what the operator sees. A new job is signed. | A | `AR/04-control-plane-authorization.md:184` | +| 6 | A box off for longer than the window never runs that job; it is checked against the BOX's clock, `time.Now.UTC`, and in the wrapper `if now > exp or now < iss - 120` AG/configs/felhom-os-apply:676. A wrong box clock decides. | A | `AG/internal/authz/verifier.go:90` | +| 7 | The refused job is logged as a security event and removed, no retry. Used nonces are kept until their end, comment. | A | `AG/internal/signedjobs/runner.go:163-170` `AG/internal/authz/noncestore.go:47-50` | +| 8 | TEST `test_expired_signed_job_is_refused`, `test_a_window_longer_than_24h_is_refused`, `TestVerify_RejectsPerStage`. | A | `AG/configs/test_felhom_os_apply.py:833` `AG/configs/test_felhom_os_apply.py:836` `AG/internal/authz/verify_test.go:82` | + +**Hand step in a runbook?** yes — the build skill's agent section + +### DB12 — Operator signing keys on each box · Key · NAMED + +architecture: `AR/04-control-plane-authorization.md:139-150` · helper section: dates-box „12. Operator signing keys on each box” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The two public keys every box trusts for signed jobs: `OPERATOR_KEY_OPERATIONAL_ID` and `OPERATOR_KEY_RECOVERY_ID`. Written to `TRUST_SIGNERS` AG/configs/felhom-os-apply:124 by `_write_slow_lane_trust`. Operator-owned. | A | `E/scripts/felhom-host-install.sh:212` `E/scripts/felhom-host-install.sh:214` `E/scripts/felhom-host-install.sh:3424` | +| 2 | NO END DATE: the keys carry no expiry; trust is by key material,. The key id is advisory,. | A | `AG/internal/authz/verifier.go:122-126` `AG/internal/authz/verifier.go:170-175` | +| 3 | The keys have no end date. A change of key is designed as a signed order of its own. Only the name of that order exists in code. | D | `AR/04-control-plane-authorization.md:139-150` | +| 4 | If the operator loses both private keys, no box can be updated or repaired by a signed job again; households keep running on what they have. Design text. | A | `AR/04-control-plane-authorization.md:139-150` | +| 5 | Nobody; there is no date to warn about. | A | `E/scripts/felhom-host-install.sh:212` `E/scripts/felhom-host-install.sh:214` (the code path of question 1) | +| 6 | Yes, the keys are in the installer and on disk,. | A | `E/scripts/felhom-host-install.sh:3452-3453` | +| 7 | If the keys are lost, no box accepts a signed job again without a visit. The keys exist on DooPlex only. | K | R-924 | +| 8 | DOC; for a real rotation: NOBODY KNOWS. | A | `AR/04-control-plane-authorization.md:139-150` | + +**Hand step in a runbook?** no — the design exists, a procedure does not + +### DB13 — The agent's Proxmox access token · Key · MENTIONED + +architecture folder: 33 lines in 12 files match `/pveum|API token|privsep/` (a machine count, not a reading) · helper section: dates-box „13. Proxmox API token of the agent” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The token the agent uses for every Proxmox call. Made by the installer in `step_token`. Product-owned, on the host. | A | `E/scripts/felhom-host-install.sh:2405` | +| 2 | MEASURED 2026-11-11 on the three boxes: the user and its token have no end date (expire 0). | A | `EV/box/pin.txt` `E/scripts/felhom-host-install.sh:2448` | +| 3 | Nobody on a schedule. A re-run of the installer reuses it or makes a new one,. | A | `E/scripts/felhom-host-install.sh:2424-2445` | +| 4 | Nothing ends by date. If someone sets an expiry by hand in Proxmox, every agent call fails that day. | A | `E/scripts/felhom-host-install.sh:2405` (the code path of question 1) | +| 5 | Nobody, because there is no date. | A | `EV/box/pin.txt` | +| 6 | Yes. | A | `E/scripts/felhom-host-install.sh:2405` (the code path of question 1) | +| 7 | No renewal to fail. | A | `E/scripts/felhom-host-install.sh:2405` (the code path of question 1) | +| 8 | MEASURED: expire 0 on three boxes · SOURCE. | A | `EV/box/pin.txt` `E/scripts/felhom-host-install.sh:2448` | + +**Hand step in a runbook?** no hand step + +### DB16 — Package signing keys: Docker in the guest, Proxmox and Debian on the host · Key · MENTIONED + +architecture: `AR/11-os-updates.md:657` · helper section: dates-box „16. Package signing keys: Docker in the guest, Proxmox and Debian on the host” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The keys apt uses to trust packages. Docker's key is fetched once at golden bake into `/etc/apt/keyrings/docker.asc`. Proxmox's keyring path default is in `align_apt_repos`. Owned by Docker, Proxmox, Debian. | A | `AG/configs/build-golden.sh:129` `E/scripts/felhom-host-install.sh:2616` | +| 2 | MEASURED 2026-11-11 on the three boxes: Docker's key never ends. The Debian keys in use end 2029-01-15 at the soonest. The Proxmox key for this series ends 2034-11-10. Two keys that end 2027-02-03 and 2027-04-12 are in the 'removed keys' file; nothing uses them. | A | `EV/box/keyrings.txt` | +| 3 | Debian and Proxmox keys arrive as packages through the normal update. Docker's key has no package: a change is "a slow-lane act for a person",. | A | `AR/11-os-updates.md:657` | +| 4 | The OS update is refused with code R7 when `apt-get update` fails, AG/configs/felhom-os-apply:1125-1127. The box keeps running but stops getting Docker security updates. Household unaware; operator sees the report. | A | `AG/configs/build-golden.sh:129` `E/scripts/felhom-host-install.sh:2616` (the code path of question 1) | +| 5 | Nobody before the end. After it the box reports a failed update. The first date is 2029-01-15. | N | `AR/11-os-updates.md:657` `EV/box/keyrings.txt` · class T1 · S10 | +| 6 | NOBODY KNOWS. A box that was off for years has old keys. The new keys come as a package, and the package list is checked with the old keys. | N | `AR/11-os-updates.md:657` · class T4 · S11 | +| 7 | The update fails cleanly and nothing is half-installed, the next signed job tries again and fails the same way until a person replaces the key. | A | `AR/11-os-updates.md:657` | +| 8 | MEASURED: every key's end date on three hosts and three guests · DOC; TEST for the refusal `test_R7_not_downloadable_and_no_snapshot`; key dates: NOBODY KNOWS. | A | `EV/box/keyrings.txt` `AR/11-os-updates.md:657` `AG/configs/test_felhom_os_apply.py:488` | + +**Hand step in a runbook?** NO — the document says a key change is 'a slow-lane act for a person'; no procedure is written + +### DB17 — Debian 13 and Proxmox 9: the series the product is tied to · OS · MENTIONED + +architecture folder: 9 lines in 2 files match `/trixie|Debian 13|Proxmox (VE )?9/` (a machine count, not a reading) · helper section: dates-box „17. Debian 13 (trixie) and Proxmox 9: the series the product is tied to” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The OS series under everything. Written into source: suite default `trixie`, Docker repo `trixie stable`, base images CT/Dockerfile:52. Owned by Debian and Proxmox. | A | `E/scripts/felhom-host-install.sh:2615` `AG/configs/build-golden.sh:130` | +| 2 | No document gives the day when security fixes stop for Debian 13 or for Proxmox 9. MEASURED: the three boxes run Debian 13.7 and Proxmox 9.2. | N | `EV/box/demo-hp-host-dates.txt` `AR/11-os-updates.md:1` · class T3 · S09 | +| 3 | No plan with a date. The move to the next series is not designed. | N | `AR/11-os-updates.md:1` · class T3 · S09 | +| 4 | After the series ends: no security updates for host and guest; a new install on a newer Proxmox is refused in byo mode and only warned in appliance mode,. | A | `E/scripts/felhom-host-install.sh:2146-2152` | +| 5 | Nobody. The dates are public; nothing in the product or in a document reads them. | N | `AR/11-os-updates.md:1` · class T1 · S09 | +| 6 | Yes while the archives still serve trixie. Pinned OS updates fall back to snapshot.debian.org in `add_snapshot_sources` AG/configs/felhom-os-apply:1451-1462; if that source stops, the update is refused. | A | `E/scripts/felhom-host-install.sh:2615` `AG/configs/build-golden.sh:130` (the code path of question 1) | +| 7 | Refused update, box unchanged, AG/configs/felhom-os-apply:1125-1127. | A | `E/scripts/felhom-host-install.sh:2615` `AG/configs/build-golden.sh:130` (the code path of question 1) | +| 8 | SOURCE; the dates: NOBODY KNOWS. | A | `E/scripts/felhom-host-install.sh:2146-2152` | + +**Hand step in a runbook?** NO + +### DB18 — Catalog app images and their recorded digests · Version · MENTIONED + +architecture: `AR/09-update-architecture.md:391` · helper section: dates-box „18. Catalog app images and their recorded digests” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | 65 app templates under CAT/templates/; 84 different image references; registries docker.io 47, ghcr.io 27, lscr.io 8, gitea.dooplex.hu 1, registry.gitlab.com 1. The update ladders record 149 different digests, e.g.. Owned by each app's publisher. | A | `CAT/templates/actualbudget/.felhom.yml:196` | +| 2 | NO END DATE in source. The end is the day a registry stops serving a tag or a digest. Moving tags are banned: `BANNED_TAGS`. | A | `CAT/scripts/check-image-pins.py:28` | +| 3 | The product, when it tests a new version and records its digest. reports how far behind a pin is and gates nothing. | A | `CAT/scripts/check-currency.py:22` | +| 4 | The box renders `tag@sha256` in `RenderWithLadderDigests`; a digest the registry no longer serves makes the pull fail on every box,. A household cannot install or update that app; a running app keeps running. | A | `CT/internal/stacks/digest.go:94` `AR/09-update-architecture.md:391` | +| 5 | Nobody. No check asks the registries if every recorded image is still served. The document speaks of the day a version is recorded, not of the days after. | N | `AR/09-update-architecture.md:391` `CAT/scripts/check-currency.py:22` · class T1 · S21 | +| 6 | Running apps come back (images are local). The first failing step is the next install, update or restore that must pull,. | A | `CT/internal/stacks/digest.go:10-23` | +| 7 | The pull fails and the app stays on its current image; no automatic repair. Retry happens only when the household presses again. | A | `CAT/templates/actualbudget/.felhom.yml:196` (the code path of question 1) | +| 8 | DOC; SOURCE; which digests are still served today: NOBODY KNOWS. | A | `AR/09-update-architecture.md:391` `CT/internal/stacks/digest.go:94` | + +**Hand step in a runbook?** no hand step + +### DB19 — Images the controller itself starts (Traefik, tunnel client, file browser, Samba) · Version · MENTIONED + +architecture folder: 41 lines in 15 files match `/cloudflared|TraefikImage|traefik:v/` (a machine count, not a reading) · helper section: dates-box „19. Images the controller itself starts (Traefik, tunnel, file browser, Samba)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Four fixed images in the controller binary: `TraefikImage`, `CloudflaredImage`, `FileBrowserImage`, `SambaImage`. Product-owned pins of other people's images. | A | `CT/internal/infra/infra.go:33` `CT/internal/infra/infra.go:34` `CT/internal/infra/infra.go:35` `CT/internal/infra/infra.go:50` | +| 2 | No document says for how long Cloudflare accepts an old tunnel client. MEASURED on the N100: tunnel client 2026.10.3, Traefik v3.7.16. | N | `EV/box/demo-felhom-guest-dates.txt` · class T3 · S12 | +| 3 | The product, by a controller release that names newer tags. | A | `CT/internal/infra/infra.go:33` `CT/internal/infra/infra.go:34` (the code path of question 1) | +| 4 | If Cloudflare stops accepting an old tunnel client, the household's address stops answering from outside while the box is healthy. If a registry drops a tag, a box that is built again cannot start that part. | A | `CT/internal/infra/infra.go:33` `CT/internal/infra/infra.go:34` (the code path of question 1) | +| 5 | Nobody. | N | `EV/box/demo-felhom-guest-dates.txt` · class T1 · S12 | +| 6 | NOBODY KNOWS if the tunnel of a box that was off for a year still connects with its old client. Without the tunnel the box gets no update from outside. | N | `EV/box/demo-felhom-guest-dates.txt` · class T4 · S12 | +| 7 | Not traced. | A | `CT/internal/infra/infra.go:33` `CT/internal/infra/infra.go:34` (the code path of question 1) | +| 8 | SOURCE; NOBODY KNOWS for the tunnel client's age limit. | A | `CT/internal/infra/infra.go:33-50` | + +**Hand step in a runbook?** no hand step + +### DB20 — Published agent versions: the newest five stay · Version · NAMED + +architecture: `AR/09-update-architecture.md` §3, D50 · helper section: dates-box „20. Published agent versions: only the newest five stay downloadable” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The registry of agent binaries. Rule file `generic_versions_kept`. Operator-owned; he prunes by hand,. | A | `AG/scripts/retention-policy.json:23` `AG/scripts/retention-policy.json:7-10` | +| 2 | Newest 5 versions plus two pins: `hub_min_agent` and `day0_agent`. Decision D50,. | A | `AG/scripts/retention-policy.json:30` `AG/scripts/retention-policy.json:31` `AR/09-update-architecture.md:1370-1375` | +| 3 | The session that records a vouch or a new floor updates the pins in the same commit,. | A | `AG/scripts/retention-policy.json:12-15` | +| 4 | A box told to install a version that was pruned cannot download it. A pin that names a missing version is a CI failure,. Deleting a git tag also ends the ability to install that version,. | A | `AG/scripts/retention-policy.json:14-15` `AG/scripts/retention-policy.json:20-21` | +| 5 | The operator, through CI:. | A | `AG/scripts/retention-policy.json:25` | +| 6 | A box off for long is on an old agent; it updates only by a fresh signed job that names a version still kept. First failing step otherwise: the download in the wrapper, `SELFUPDATE_OP` AG/configs/felhom-os-apply:180. | A | `AG/scripts/retention-policy.json:23` `AG/scripts/retention-policy.json:7-10` (the code path of question 1) | +| 7 | Not traced on a failed download. | A | `AG/scripts/retention-policy.json:23` `AG/scripts/retention-policy.json:7-10` (the code path of question 1) | +| 8 | SOURCE; DOC. | A | `AG/scripts/retention-policy.json:23-31` `AR/09-update-architecture.md:1370-1375` | + +**Hand step in a runbook?** yes — the build skill's agent section + +### DB21 — The golden image's age (7 days) and the waiver (14 days) · Version · NAMED + +architecture: `AR/09-update-architecture.md:1010-1018` · helper section: dates-box „21. Golden image age and the waiver file” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The ready-made guest image a new box is installed from, baked weekly,. Operator vouches. Gate:. | A | `AR/09-update-architecture.md:1010-1018` `E/scripts/golden_currency_gate.py:169` | +| 2 | MEASURED 2026-11-11: the newest golden is 8 days old (baked 2026-11-03). The limit is 7. No waiver file exists. The gate fails today. | A | `E/scripts/golden_currency_gate.py:169` `EV/box/golden-gate.txt` | +| 3 | The operator, weekly (Sunday night per). | A | `AR/09-update-architecture.md:1010-1018` | +| 4 | The gate goes red for the product team; no household is touched. A new install gets an older image and must update more on first start. | A | `AR/09-update-architecture.md:1010-1018` `E/scripts/golden_currency_gate.py:169` (the code path of question 1) | +| 5 | The operator, by the gate itself; no earlier warning found. | A | `AR/09-update-architecture.md:1010-1018` `E/scripts/golden_currency_gate.py:169` (the code path of question 1) | +| 6 | Not about a box. | A | `AR/09-update-architecture.md:1010-1018` `E/scripts/golden_currency_gate.py:169` (the code path of question 1) | +| 7 | A push of code to this repository is refused until a golden is baked or a dated waiver is written. A push of documents passes with a notice. | K | R-468 | +| 8 | MEASURED: the gate's own output today · SOURCE; tests exist in (names not matched to this rule). | A | `EV/box/golden-gate.txt` `E/scripts/golden_currency_gate.py:162-169` `E/scripts/test_golden_currency_gate.py:1` | + +**Hand step in a runbook?** yes — `RB/RUNBOOK-manual-build.md` + +### DB24 — Copies kept before a restore: deleted after 30 unused days · Window · MENTIONED + +architecture folder: 18 lines in 4 files match `/restore cop|restore-undo|D62/` (a machine count, not a reading) · helper section: dates-box „24. Kept copies made before a restore: deleted after 30 unused days” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The copy of an app's data the box keeps when the household restores an older state. Household's data, product rule D62. `restoreCopyUnusedAge`. | A | `CT/internal/backup/restore_copy_sweep.go:29` | +| 2 | `restoreCopyUnusedAge = 30 * 24 * time.Hour`. | A | `CT/internal/backup/restore_copy_sweep.go:29` | +| 3 | Nobody; using the copy is what keeps it. | A | `CT/internal/backup/restore_copy_sweep.go:29` (the code path of question 1) | +| 4 | The household loses the way back to the state from before the restore. | A | `CT/internal/backup/restore_copy_sweep.go:29` (the code path of question 1) | +| 5 | Nobody. The household is not told before the copy goes. The household can delete it sooner by hand. | N | `CT/internal/backup/restore_copy_sweep.go:18-29` · class T1 · S05 | +| 6 | A box that was off for more than 30 days deletes the copy at its first sweep after the start: the age is counted from the folder's time stamp by the wall clock. Nobody asked if the off time must count. | N | `CT/internal/backup/restore_copy_sweep.go:21-29` · class T4 · S11 | +| 7 | Not traced. | A | `CT/internal/backup/restore_copy_sweep.go:29` (the code path of question 1) | +| 8 | SOURCE. · Mail relay's in-memory self-signed leaf, 10 years, remade at each start:. · Language cookie, 365 days:. · HSTS max-age one day: `HSTSMaxAgeSeconds`. · A backup counts as fresh enough for an update for 24 hours: `DefaultUpdateBackupMaxAge`. · Night-backup banner turns stale after 26 hours: `StaleAfter`. … | A | `CT/internal/backup/restore_copy_sweep.go:29` `CT/internal/mailrelay/server.go:386` `CT/internal/web/i18n_web.go:478` `CT/internal/infra/infra.go:108` | + +**Hand step in a runbook?** no hand step + +### DH01 — Connect link in the household's e-mail (7 days) · Window · MENTIONED + +architecture folder: 32 lines in 7 files match `/self-bind|selfbind|/bind//` (a machine count, not a reading) · helper section: dates-hub „1. Self-bind (connect) link in the household's e-mail” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | A one-use link that lets the household tie a new box to its account. Household uses it, product mints it. Stored hashed in the hub table `selfbind_tokens`; minted by `selfBindTTL` code. | A | `HUB/store/store.go:796-806` `HUB/web/selfbind_mint.go:20` | +| 2 | 7 days after minting: `selfBindTTL`, passed to the store at; `Expired`. Also locks at the 5th wrong answer, `SelfBindMaxAttempts`. | A | `HUB/web/selfbind_mint.go:20` `HUB/web/selfbind_mint.go:77` `HUB/store/selfbind.go:39` `HUB/store/selfbind.go:21` | +| 3 | The household itself: the expired page has a button, `handleBindResend`, at most one new link an hour, `bindResendEvery`. No renewal before the end. The hub also mints at customer creation and RESET. | A | `HUB/web/selfbind.go:384` `HUB/web/selfbind.go:382` `HUB/web/selfbind_mint.go:97` | +| 4 | The household cannot bind the box from the mail; unknown and expired links show the same "expired" page. The box itself keeps waiting; nothing on it is lost. | A | `HUB/web/selfbind.go:206` | +| 5 | Nobody before, by design: the page of a dead link has a button that sends a new one. | A | `HUB/web/selfbind_mint.go:1-110` `HUB/web/selfbind_mint.go:93` | +| 6 | A box first switched on after day 7: the link fails at the first step (opening it); the household presses the resend button. If an old host is still linked no link is sent and the operator gets `eventBindOldHostLinked`, logged at. | A | `HUB/web/selfbind.go:440` `HUB/web/selfbind.go:413` | +| 7 | If the mail send fails at mint the token is deleted, so no dead link is left. No automatic retry; the household presses again after an hour. | A | `HUB/web/selfbind_mint.go:82` | +| 8 | TEST `TestR719_AnExpiredLinkOffersAndSendsAFreshOne`; SOURCE; DOC. | A | `HUB/web/selfbind_resend_test.go:21` `HUB/web/selfbind_mint.go:20` `AR/05-hub-architecture.md:359` | + +**Hand step in a runbook?** no hand step + +### DH05 — 'Switched off on purpose' hold (14 days) · Window · MENTIONED + +architecture folder: 10 lines in 4 files match `/deliberate.off|switched off on purpose/` (a machine count, not a reading) · helper section: dates-hub „5. "Switched off on purpose" hold (14 days of silence allowed)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | When a household switches the box off through the product, the hub stays quiet about the silence. Household act, hub rule: `DeliberateOffHold`. | A | `HUB/store/deliberateoff.go:18` | +| 2 | 14 days after the switch-off: `DeliberateOffHold`, judged by `DeliberateOffHeld`. | A | `HUB/store/deliberateoff.go:18` `HUB/store/deliberateoff.go:21` | +| 3 | Nobody renews. The hold ends by itself when the box reports again or at day 14. | A | `HUB/store/deliberateoff.go:18` (the code path of question 1) | +| 4 | After day 14 the silence and missed-backup alarms start again for that household; the operator is told. | A | `HUB/store/deliberateoff.go:18` (the code path of question 1) | +| 5 | The operator gets an event at switch-off time naming the moment, message built at. No warning before day 14 was found (grep `DeliberateOffHold` in hub). | A | `HUB/api/deliberate_off.go:42` | +| 6 | Yes, it comes back by itself: a box switched on at any time reports and the hold clears. TEST `TestDeliberateOff_BackOnIsQuiet`. | A | `HUB/monitor/deliberate_off_test.go:138` | +| 7 | Not applicable: nothing is renewed. | A | `HUB/store/deliberateoff.go:18` (the code path of question 1) | +| 8 | TEST `TestDeliberateOff_AlarmsAfterFourteenDays` and `TestDeliberateOff_NoSilenceAlarmDuringTheHold`. | A | `HUB/monitor/deliberate_off_test.go:112` `HUB/monitor/deliberate_off_test.go:90` | + +**Hand step in a runbook?** no hand step + +### DH08 — Storage Box sub-account password, and the hub's seal key · Key · MENTIONED + +architecture folder: 27 lines in 8 files match `/seal key|sub-account/` (a machine count, not a reading) · helper section: dates-hub „8. Storage Box sub-account password and the hub's seal key” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Each household's off-site store account at the provider, minted by the hub, and the hub key that seals stored secrets (`OFFSITE_SECRET_KEY`, named in). Operator and product. | A | `E/manifests/hub.yaml:167-171` | +| 2 | NO END DATE in code: no lifetime on the sub-account password. A clean reinstall's re-issue resets it, DOC; `ResetBoxPassword`. | A | `AR/05-hub-architecture.md:274-279` `HUB/hetznerapi/hetznerapi.go:297` | +| 3 | NOT MEASURED (DooPlex, ep0 and the accounts are outside this run). The password changes when the operator presses Re-issue. A change of the seal key is 'not built'. | D | `RB/secrets.md:159` | +| 4 | If the seal key is lost or changed, the old sealed rows cannot be opened, DOC: operator loses stored off-site secrets and break-glass passwords until re-issued per household. | A | `RB/secrets.md:159` | +| 5 | Nobody, because there is no date. | A | `HUB/hetznerapi/hetznerapi.go:1-320` | +| 6 | A box off for a long time comes back by itself as long as its account was not re-issued; it uses its SSH key, the password is used only by the hub, DOC. | A | `AR/05-hub-architecture.md:274-279` | +| 7 | No renewal on a date exists, so none can fail on a date. | A | `E/manifests/hub.yaml:167-171` (the code path of question 1) | +| 8 | DOC and; SOURCE. No test of a lifetime, because there is none. | A | `RB/secrets.md:159` `AR/05-hub-architecture.md:274-279` `HUB/hetznerapi/hetznerapi.go:297` | + +**Hand step in a runbook?** partly — `RB/secrets.md:159` says the seal key change is not built + +### DH09 — PBS off-site token on ep0 · Key · NAMED + +architecture: `AR/05-hub-architecture.md:379-387` · helper section: dates-hub „9. PBS off-site token on ep0 and its descriptor on the hub” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The per-household token on the rented endpoint for the whole-guest off-site copy, and the hub record that must agree with it. Product mints, operator can re-issue. DOC. | A | `AR/05-hub-architecture.md:373-387` | +| 2 | NO END DATE stated: the lifecycle table has only events (provision, re-key, deprovision). One age rule: a staged secret not consumed in 15 minutes is wrong, `unconsumedGrace`. | A | `AR/05-hub-architecture.md:379-387` `HUB/pbsdrheal/reconciler.go:63` | +| 3 | The hub re-keys on re-issue (delete and recreate). No scheduled renewal. | A | `AR/05-hub-architecture.md:382` | +| 4 | A re-key the box did not pick up leaves the box "serving a revoked credential": the household has no whole-guest off-site copy while both tiers look green, comment at. | A | `HUB/pbsdrheal/reconciler.go:55-63` | +| 5 | Operator only, after the fact: `pbsdr_provision_failed` on a final failure, DOC. | A | `AR/05-hub-architecture.md:400-402` | +| 6 | Host delete keeps the token; a box off and back uses the same token. After a re-issue while it was off, it must consume the new secret on its next report. | A | `AR/05-hub-architecture.md:384` | +| 7 | A failed self-check rolls back and is repeated 3 times, 7 s apart, then raises the operator event, DOC. | A | `AR/05-hub-architecture.md:400-402` | +| 8 | DOC; SOURCE. Lifetime of the token on the endpoint itself: NOBODY KNOWS from these sources (ep0 not read). | A | `AR/05-hub-architecture.md:373-403` `HUB/pbsdrheal/reconciler.go:63` | + +**Hand step in a runbook?** yes — `RB/offsite-endpoint.md` + +### DH10 — WireGuard keys of the box's tunnel to the off-site endpoint · Key · NAMED + +architecture: `AR/06-offsite-connectivity.md:200-205` · helper section: dates-hub „10. WireGuard peers (the box's tunnel to the off-site endpoint)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | One peer per box on the endpoint, registered through the hub; the private key is in the household's sealed bundle. Product. DOC. | A | `AR/06-offsite-connectivity.md:172-182` | +| 2 | NO END DATE: the hub store only removes a peer on request, `RemoveWGPeer`. Scheduled key rotation is an open point, DOC. | A | `HUB/store/wg.go:345` `AR/06-offsite-connectivity.md:368` | +| 3 | Nobody on a schedule. The agent makes a new key itself only after a handshake stamped in the future for 15 minutes, DOC. | A | `AR/06-offsite-connectivity.md:205-207` | +| 4 | Revocation: the box is off the off-site network at once, DOC; the household's off-site copies stop. | A | `AR/06-offsite-connectivity.md:180-182` | +| 5 | Nobody, because there is no date. | A | `HUB/store/wg.go:1-360` | +| 6 | Yes, a box off for months keeps its peer. A revoked box never re-registers on its own, DOC. | A | `AR/06-offsite-connectivity.md:200-205` | +| 7 | A failed fetch is never read as a teardown, DOC. Re-add is an operator call. | A | `AR/06-offsite-connectivity.md:207-208` | +| 8 | DOC; SOURCE. Box-side tests belong to the box helper. | A | `AR/06-offsite-connectivity.md:200-208` `HUB/store/wg.go:345` | + +**Hand step in a runbook?** yes — `RB/offsite-endpoint.md` + +### DH14 — Recovery mode for a host (30 minutes to 4 hours) · Window · MENTIONED + +architecture folder: 6 lines in 4 files match `/recovery mode|defaultRecoveryTTL/` (a machine count, not a reading) · helper section: dates-hub „14. Recovery mode for a host (the door a rebuilt box uses)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | A switch the operator sets on a host so a rebuilt box may re-enrol and fetch its restore data. Operator, global key only, `handleSetRecoveryMode`. | A | `HUB/api/dr.go:34` | +| 2 | 30 minutes by default, `defaultRecoveryTTL`; at most 4 hours, `maxRecoveryTTL`; clamp at. | A | `HUB/api/dr.go:21` `HUB/api/dr.go:22` `HUB/api/dr.go:52` | +| 3 | The operator sets it again. No lead time. | A | `HUB/api/dr.go:34` (the code path of question 1) | +| 4 | A restore that takes longer than the window: the box's re-enrol call is refused and the household's recovery stops until the operator re-opens it. | A | `HUB/api/dr.go:34` (the code path of question 1) | +| 5 | Nobody before. The operator who opened the window is present; he opens it again. | A | `HUB/api/dr.go:34` (the code path of question 1) | +| 6 | No: a box that starts its recovery after the window fails at the re-enrol step. TEST `TestReEnroll_GatedOnRecoveryMode`. | A | `HUB/api/dr_test.go:14` | +| 7 | Nothing is left half-done by the expiry itself; the operator opens a new window. | A | `HUB/api/dr.go:34` (the code path of question 1) | +| 8 | TEST `TestRestoreDirective_GatedAndExpires`; SOURCE. | A | `HUB/api/dr_test.go:70` `HUB/api/dr.go:21-22` | + +**Hand step in a runbook?** yes — the recovery runbooks + +### DH15 — Break-glass password of a host · Key · MENTIONED + +architecture folder: 42 lines in 13 files match `/break-glass|host_recovery/` (a machine count, not a reading) · helper section: dates-hub „15. Break-glass credential (the host's emergency root password)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | A per-host emergency password, sealed in the hub, shown only to a logged-in operator. Operator. `SaveHostRecoveryCredential`; reveal. | A | `HUB/store/host_recovery.go:28` `HUB/web/hosts.go:983` | +| 2 | NO END DATE: last write wins, day-0 sets it and a rotate re-sets it. Paper rule: re-print the sheet "when a key rotates, and yearly". | A | `HUB/store/host_recovery.go:20` `RB/break-glass-sheet.md:75` | +| 3 | The operator, yearly by the paper rule; no code reminds him. | A | `HUB/store/host_recovery.go:28` `HUB/web/hosts.go:983` (the code path of question 1) | +| 4 | If the sealed copy cannot be opened (seal key, block 8) or the paper is stale, the operator has no way into a host whose agent is dead. | A | `HUB/store/host_recovery.go:28` `HUB/web/hosts.go:983` (the code path of question 1) | +| 5 | Nobody before. Each reveal writes the event `recovery_credential_revealed`. | A | `HUB/web/hosts.go:1010` | +| 6 | Yes, the password does not age on the box. | A | `HUB/store/host_recovery.go:28` `HUB/web/hosts.go:983` (the code path of question 1) | +| 7 | No renewal on a date exists, so none can fail on a date. | A | `HUB/store/host_recovery.go:20-43` | +| 8 | SOURCE; session-only route; DOC. A yearly re-print has no record: NOBODY KNOWS. | A | `HUB/store/host_recovery.go:20-28` `HUB/web/routes.go:130` `RB/break-glass-sheet.md:75` | + +**Hand step in a runbook?** yes — `RB/break-glass-sheet.md:75` (print the sheet again each year) + +### DH19 — The installer ISO · Version · MENTIONED + +architecture folder: 3 lines in 3 files match `/ISO_VERSION|installer ISO|iso\.felhom\.eu/` (a machine count, not a reading) · helper section: dates-hub „19. The installer ISO (Proxmox base, installer version, first boot)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The USB image a household or volunteer installs from. Product; built by, served from Cloudflare R2, DOC. | A | `E/scripts/iso/build-felhom-iso.sh:51` `RB/total-loss-of-dooplex.md:125` | +| 2 | NO END DATE in code. `ISO_VERSION` is 1.30.0. The Proxmox ISO and its checksum are build ARGUMENTS, not pinned: the example names 9.2-1. | A | `E/scripts/iso/build-felhom-iso.sh:51` `E/scripts/iso/build-felhom-iso.sh:194` `E/scripts/iso/README.md:40` | +| 3 | No document says how old a public ISO may be, or when to build a new one. The ISO carries a Proxmox base and a first-boot program that age. | N | `E/scripts/iso/build-felhom-iso.sh:51` `RB/iso-release-gate.md:1` · class T3 · S22 | +| 4 | An old ISO installs an old Proxmox base; the install script itself is fetched fresh at first boot (comment block at), so the hub decides the agent and golden. | A | `E/scripts/iso/build-felhom-iso.sh:44-52` | +| 5 | Nobody. The build warns only if golden and floor are not given: unenforced branch. | A | `E/scripts/iso/build-felhom-iso.sh:79` | +| 6 | A too-old ISO on first boot: registers, pairs, fetches the current install script; first failing step would be that fetch or the manifest. No "ISO too old" check found (grep `too old/min.*version` in bootstrap). | A | `E/scripts/felhom-host-install.sh:472` | +| 7 | A failed install is retried every 30 s; a failed script fetch is retried. | A | `E/scripts/iso/felhom-bootstrap.sh:430` `E/scripts/iso/felhom-bootstrap.sh:396` | +| 8 | SOURCE (checksum of the given ISO is verified). How old an ISO still installs cleanly: NOBODY KNOWS. Build image is `FROM debian:trixie`, not pinned. | A | `E/scripts/iso/build-felhom-iso.sh:222` `E/scripts/iso/Dockerfile.assistant:10` | + +**Hand step in a runbook?** yes for a build — `RB/iso-release-gate.md`; NO for 'when' + +### DH20 — The certificate that the hub's address serves · Certificate · MENTIONED + +architecture folder: 1 lines in 1 files match `/cert-manager|letsencrypt-prod/` (a machine count, not a reading) · helper section: dates-hub „20. The certificate the hub's address serves” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The TLS certificate for the hub's public name, used by every box and the operator. Operator's cluster issues it: annotation, stored under the secret name at. | A | `E/manifests/hub.yaml:342` `E/manifests/hub.yaml:360` | +| 2 | Read from outside 2026-11-11, with no login: the certificate of the hub's address ends 2027-01-12. The download address ends 2026-12-23, the ISO address 2026-12-27, the Gitea address 2027-01-03, the website 2027-01-28. | A | `EV/box/public-reads.txt` | +| 3 | NOT MEASURED on DooPlex. cert-manager on DooPlex renews by itself. Read from outside: the certificate served today was issued on 2026-10-14, so a renewal worked. | A | `E/manifests/hub.yaml:342` `EV/box/public-reads.txt` | +| 4 | If it lapses: boxes that pin or verify the hub cannot report or fetch config; households get no alarms and no updates; the operator page shows a browser warning. | A | `E/manifests/hub.yaml:342` `E/manifests/hub.yaml:360` (the code path of question 1) | +| 5 | NOT MEASURED (DooPlex, ep0 and the accounts are outside this run). In the Felhom repositories nothing warns before this date. If it ends, every box fails to report, and the hub then sees all boxes as silent. | N | `E/manifests/hub.yaml:342` `EV/box/public-reads.txt` · class T1 · S01 | +| 6 | Yes. The agent trusts the public issuers, not one certificate, so a renewed certificate is accepted. | A | `AG/internal/hub/client.go:41-51` | +| 7 | The old certificate serves until its end date. Nothing in the Felhom repositories warns (question 5). | A | `E/manifests/hub.yaml:342` | +| 8 | READ FROM OUTSIDE, no login · SOURCE and only. Dates: NOBODY KNOWS from files; needs a measurement. | A | `EV/box/public-reads.txt` `E/manifests/hub.yaml:342` `E/manifests/hub.yaml:360` | + +**Hand step in a runbook?** NO + +### DH22 — Operator access tokens, among them CC's hub token · Key · NAMED + +architecture: `AR/05-hub-architecture.md` §17 · helper section: dates-hub „22. Operator access tokens, among them CC's hub token” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Named, scoped tokens for scripts, shown once, stored hashed. Operator creates them in a browser session; CC holds one, with `overview:read` and `floor:write`, DOC. | A | `AR/09-update-architecture.md:1022` | +| 2 | 90 days by default, `tokenDefaultDays`; at most 365, `tokenMaxDays`; "never" is not offered. End computed at. | A | `HUB/web/tokens.go:29` `HUB/web/tokens.go:30` `HUB/web/tokens.go:254` `HUB/web/tokens.go:180` | +| 3 | The operator makes a new token and puts it in CC's store. There is no renew; no lead time is set anywhere. | A | `AR/09-update-architecture.md:1022` (the code path of question 1) | +| 4 | CC's floor raise and overview reads get 401; a controller release cannot be carried to boxes by CC until the operator acts. Households wait for the update. | A | `HUB/web/tokens.go:95` | +| 5 | Nobody. A token lives 90 days by default and 365 at most. The token page shows the date; no monitor reads it. On the end date the release step that sets a floor answers 401. | N | `HUB/web/tokens.go:29-30` `HUB/web/tokens.go:180` · class T1 · S03 | +| 6 | Not a box matter. | A | `AR/09-update-architecture.md:1022` (the code path of question 1) | +| 7 | Not applicable; an expired token stays listed as expired. Column `expires_at`. | A | `HUB/store/api_tokens.go:52` | +| 8 | TEST `TestToken_NoTokenUnknownExpiredRevoked401`; DOC. The end DATE of CC's token is in no document (grep `HUB_TOKEN` in AR, RB, operations, STATUS): NOBODY KNOWS without the hub page. | A | `HUB/web/tokens_test.go:310` `AR/05-hub-architecture.md:655-669` | + +**Hand step in a runbook?** partly — the build skill says '401 = token expired'; no step says to look at the date before + +### DH23 — A household's own Cloudflare token · Key · MENTIONED + +architecture folder: 9 lines in 4 files match `/Cloudflare token|CF_DNS_API_TOKEN/` (a machine count, not a reading) · helper section: dates-hub „23. A household's own Cloudflare token (hub side)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | A household that brings its own domain gives a Cloudflare API token; the hub checks what it can reach before saving. Household owns it.. | A | `HUB/cloudflare/reach.go:14` | +| 2 | The hub and the box do not know the end date that the household set at Cloudflare. No document tells the household to set none. | N | `HUB/cloudflare/reach.go:14` · class T3 · S15 | +| 3 | The household, at Cloudflare; then the operator saves the new token in the hub. | A | `HUB/cloudflare/reach.go:14` (the code path of question 1) | +| 4 | If it lapses, the household's remote address stops being managed; candidate: first sign is a failed DNS or tunnel call on the box. | A | `HUB/cloudflare/reach.go:14` (the code path of question 1) | +| 5 | Nobody. When the token ends, the web certificate stops renewing (see DB01), with no sign until the certificate's own end. | N | `CT/internal/infra/infra.go:157-159` · class T1 · S15 | +| 6 | The running apps and the tunnel do not use this token. The certificate renewal does. | A | `CT/internal/infra/infra.go:157-159` | +| 7 | The renewal is the household's hand step at Cloudflare. If it is not done, the web certificate stops renewing (DB01). | A | `HUB/cloudflare/reach.go:14` (the code path of question 1) | +| 8 | SOURCE; the rest NOBODY KNOWS from hub files. | A | `HUB/cloudflare/reach.go:14` | + +**Hand step in a runbook?** NO + +### DH24 — The mail service key (all household and operator mail) · Account · MENTIONED + +architecture: `AR/_recovery-inventory-2026-07-28.md` · helper section: dates-hub „24. The mail service key (all household and operator mail)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The key the hub uses to send every mail: alarms, bind links, claim codes. Operator's account at the mail service; named in, value not opened. | A | `E/manifests/hub.yaml:148-152` | +| 2 | NOT MEASURED (DooPlex, ep0 and the accounts are outside this run). No document gives an end date for the key or for the account. | N | `RB/secrets.md:33` · class T3 · S08 | +| 3 | The operator re-mints at the service dashboard, DOC. | A | `RB/total-loss-of-dooplex.md:47` | +| 4 | All mail stops: no alarm reaches the operator or a household, no bind link, no claim code. Everyone. | A | `E/manifests/hub.yaml:148-152` (the code path of question 1) | +| 5 | Nobody. No scheduled check proves that mail still goes out. If the key ends, no alarm mail can say so. | N | `RB/secrets.md:33` · class T1 · S08 | +| 6 | Not a box matter; boxes send through the hub. | A | `E/manifests/hub.yaml:148-152` (the code path of question 1) | +| 7 | If the key ends and no new key is made, no mail goes out, alarm mail included (question 5). | A | `HUB/web/selfbind_mint.go:82` | +| 8 | DOC and. Whether the key or the plan has an end date: NOBODY KNOWS from files. | A | `RB/secrets.md:33` `RB/total-loss-of-dooplex.md:47` | + +**Hand step in a runbook?** yes for a new key — `RB/secrets.md:33` + +### DH25 — The hub's global bearer key · Key · NAMED + +architecture: `AR/13-threat-model.md:70` · helper section: dates-hub „25. The hub's global bearer key” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The operator-tier key for the hub API (recovery mode, peer re-add). Operator. Named in; inventory row. | A | `E/manifests/hub.yaml:157-161` `AR/_recovery-inventory-2026-07-28.md:501` | +| 2 | NO END DATE. Rotation is a supervised runbook. | A | `RB/secrets.md:124` | +| 3 | The operator, with an ordered procedure; no cadence stated. | A | `RB/secrets.md:124` | +| 4 | A rotation done out of order locks out whatever still holds the old key; DOC calls the order load-bearing. | A | `RB/secrets.md:124` | +| 5 | Nobody; no date. | A | `E/manifests/hub.yaml:157-161` `AR/_recovery-inventory-2026-07-28.md:501` (the code path of question 1) | +| 6 | No date drives it. Boxes use their own keys since 2026-10-12. What a box does that holds only the old global key after a change was not traced in this run. | A | `AR/13-threat-model.md:70` | +| 7 | No renewal on a date exists, so none can fail on a date. | A | `E/manifests/hub.yaml:157-161` `AR/_recovery-inventory-2026-07-28.md:501` (the code path of question 1) | +| 8 | DOC only. Two documents disagree on whether it was rotated: "rotated" at, "when last rotated" unknown at. Candidate for a doc fix. | A | `AR/13-threat-model.md:70` `AR/_recovery-inventory-2026-07-28.md:1030` | + +**Hand step in a runbook?** yes — `RB/secrets.md:124` + +### DH26 — Pinned tool versions in builds and CI · Version · MENTIONED + +architecture folder: 8 lines in 2 files match `/security.watch|pinned by digest/` (a machine count, not a reading) · helper section: dates-hub „26. Pinned tool versions in builds and CI” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Versions of the tools that build and check the product. Operator and CC. Hub build image E/hub/Dockerfile:3, run image E/hub/Dockerfile:24, site publisher. | A | `E/scripts/publish-site-cloudflare.sh:28` | +| 2 | NO END DATE in code. Hub images are pinned by digest E/hub/Dockerfile:3; the ISO helper image is the moving tag at; the mailer image is `latest` at. | A | `E/scripts/iso/Dockerfile.assistant:10` `E/manifests/contact-mailer.yaml:54` | +| 3 | Whoever bumps the pin. The nightly security watch runs at 03:40 UTC; nightly gates at 03:20. | A | `E/.gitea/workflows/security-watch.yml:22` `E/.gitea/workflows/gates.yml:27` | +| 4 | A pinned digest removed upstream breaks the hub build for the operator; a moving tag changes the ISO build without a commit. | A | `E/scripts/publish-site-cloudflare.sh:28` (the code path of question 1) | +| 5 | The security watch treats a release line with no fixed version as end of life, it reports to the operator. | A | `E/scripts/security_watch.py:292` | +| 6 | Not a box matter. | A | `E/scripts/publish-site-cloudflare.sh:28` (the code path of question 1) | +| 7 | A CI job that does not run leaves no result. Whether a missed night run is seen is a question of the Clock list. | A | `E/.gitea/workflows/gates.yml:27` | +| 8 | SOURCE E/hub/Dockerfile:3 and. Support end dates of Go, Alpine, wrangler: no document states one. | A | `E/scripts/security_watch.py:292` | + +**Hand step in a runbook?** no hand step + +### DH27 — Domains and registrars · Account · MENTIONED + +architecture folder: 4 lines in 3 files match `/registrar|domain renew/` (a machine count, not a reading) · helper section: dates-hub „27. Domains, registrars and the home dynamic-DNS name” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | NOT MEASURED. The product's two domains and the dynamic name the hub's address points to. Operator. Listed as row S13; the hub name is a CNAME to the dynamic name. | A | `RB/total-loss-of-dooplex.md:46` `RB/total-loss-of-dooplex.md:73` | +| 2 | NOT MEASURED (DooPlex, ep0 and the accounts are outside this run). No document gives a registrar or a renewal date for any of the product's names. This run could not read the public records: no whois tool here, and the public lookups for .eu and .hu gave no answer. | N | `EV/box/public-reads.txt` · class T3 · S06 | +| 3 | The operator, at the registrar. | A | `RB/total-loss-of-dooplex.md:46` `RB/total-loss-of-dooplex.md:73` (the code path of question 1) | +| 4 | If a domain or the dynamic name lapses, every box loses the hub (reports, alarms, updates, bind links) and the site and installer disappear. Every household. | A | `RB/total-loss-of-dooplex.md:46` `RB/total-loss-of-dooplex.md:73` (the code path of question 1) | +| 5 | Nobody that a document names. A name that ends takes down the hub address, the website, the mail and every household address under it. | N | `EV/box/public-reads.txt` · class T1 · S06 | +| 6 | A box comes back by itself once the name resolves again; while it does not, the first failing step is the hub lookup. | A | `RB/total-loss-of-dooplex.md:46` `RB/total-loss-of-dooplex.md:73` (the code path of question 1) | +| 7 | The renewal is the operator's hand step at the outside account. If it is not done, the thing ends (question 4). | A | `RB/total-loss-of-dooplex.md:46` `RB/total-loss-of-dooplex.md:73` (the code path of question 1) | +| 8 | DOC. WHERE THE OPERATOR READS THE DATE: NOT FOUND. NOBODY KNOWS from files. | A | `RB/total-loss-of-dooplex.md:46` | + +**Hand step in a runbook?** NO + +### DH28 — Cloudflare account, the site publish token, the ISO store keys · Account · MENTIONED + +architecture folder: 3 lines in 3 files match `/CF_PAGES_TOKEN|R2 /` (a machine count, not a reading) · helper section: dates-hub „28. Cloudflare account, the site publish token and the ISO store keys” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | NOT MEASURED. The account that serves DNS for both domains, the site, the installer script and the ISO store. Operator. Row S11; site and ISO there since 2026-10-30. | A | `RB/total-loss-of-dooplex.md:44` `RB/total-loss-of-dooplex.md:125` | +| 2 | NOT MEASURED (DooPlex, ep0 and the accounts are outside this run). No document gives an end date for the account, for the publish token or for the store keys. | N | `E/scripts/publish-site-cloudflare.sh:12` · class T3 · S07 | +| 3 | The operator at the Cloudflare dashboard. | A | `RB/total-loss-of-dooplex.md:44` `RB/total-loss-of-dooplex.md:125` (the code path of question 1) | +| 4 | Token lapse: CC cannot publish the site or installer script; the old one stays served. Account trouble: DNS, site, ISO downloads and households' tunnels, per row S11. | A | `RB/total-loss-of-dooplex.md:44` | +| 5 | Nobody. | N | `E/scripts/publish-site-cloudflare.sh:12` · class T1 · S07 | +| 6 | A new box needs the installer script from the site at first boot; it retries until it is served. | A | `E/scripts/iso/felhom-bootstrap.sh:396` | +| 7 | A failed publish leaves the previous site live (candidate; not stated in the script header). | A | `E/scripts/publish-site-cloudflare.sh:12` | +| 8 | DOC. WHERE THE OPERATOR READS THE DATE: NOT FOUND. | A | `RB/total-loss-of-dooplex.md:44` | + +**Hand step in a runbook?** NO + +### DH29 — Hetzner account: the off-site endpoint and the storage boxes · Account · MENTIONED + +architecture: `AR/_recovery-inventory-2026-07-28.md:1019` · helper section: dates-hub „29. Hetzner account, the rented endpoint and the storage boxes” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | NOT MEASURED. The paid account behind ep0 (tunnel endpoint and off-site store) and the Storage Boxes. Operator. Row S8; API token named in. | A | `RB/total-loss-of-dooplex.md:41` `E/manifests/hub.yaml:222-226` | +| 2 | NOT MEASURED (DooPlex, ep0 and the accounts are outside this run). No document gives a payment day, a contract end or the end of the API token. The recovery inventory asks the question and no document answers it. | N | `AR/_recovery-inventory-2026-07-28.md:1019` · class T3 · S07 | +| 3 | The operator; paid monthly by the provider's own cycle, not stated in any document read. | A | `RB/total-loss-of-dooplex.md:41` `E/manifests/hub.yaml:222-226` (the code path of question 1) | +| 4 | If the account is closed or unpaid: every household's off-site copies and the PBS tunnel stop; the hub cannot mint or reset sub-accounts. | A | `RB/total-loss-of-dooplex.md:41` `E/manifests/hub.yaml:222-226` (the code path of question 1) | +| 5 | Nobody before. After the end, the hub's 'off-site copy is old' alarm fires at 8 days. | N | `HUB/monitor/deadline_tiers.go:1` · class T1 · S07 | +| 6 | A box comes back by itself when the endpoint is back; first failing step is the off-site push. | A | `RB/total-loss-of-dooplex.md:41` `E/manifests/hub.yaml:222-226` (the code path of question 1) | +| 7 | The renewal is the operator's hand step at the outside account. If it is not done, the thing ends (question 4). | A | `RB/total-loss-of-dooplex.md:41` `E/manifests/hub.yaml:222-226` (the code path of question 1) | +| 8 | DOC. WHERE THE OPERATOR READS THE DATE: NOT FOUND. | A | `RB/total-loss-of-dooplex.md:41` | + +**Hand step in a runbook?** NO + +### DH30 — Tailscale node keys (the operator's remote way in) · Key · MENTIONED + +architecture folder: 4 lines in 2 files match `/tailscale|tailnet/` (a machine count, not a reading) · helper section: dates-hub „30. Tailscale node keys (operator's remote way in)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | NOT MEASURED. The operator's private network to DooPlex and the two demo boxes. Operator. Table. | A | `E/documentation/operations/tailscale.md:11-13` | +| 2 | DOCUMENT: demo-hp's key ends 2027-01-17; the other two nodes have no end. MEASURED 2026-11-11: the Tailscale service is not running on demo-hp and Tester 1; it runs on the N100. So the date could not be read on the box. | A | `E/documentation/operations/tailscale.md:41-43` `EV/box/pin.txt` | +| 3 | The operator, in the admin console (Machines, Disable key expiry). No lead time set. | A | `E/documentation/operations/tailscale.md:43` | +| 4 | That node drops off the private network; the operator loses remote reach to it. No household is on it. | A | `E/documentation/operations/tailscale.md:91` | +| 5 | Nobody. The date is on the Machines page of the Tailscale console. | N | `E/documentation/operations/tailscale.md:41-43` · class T1 · S04 | +| 6 | The box keeps running; only the operator's remote path is gone until someone re-authenticates on site or over the LAN. | A | `E/documentation/operations/tailscale.md:11-13` (the code path of question 1) | +| 7 | The renewal is the operator's hand step at the outside account. If it is not done, the thing ends (question 4). | A | `E/documentation/operations/tailscale.md:11-13` (the code path of question 1) | +| 8 | DOC. Candidate: the section at says "both" nodes while the table has three. | A | `E/documentation/operations/tailscale.md:41` `E/documentation/operations/tailscale.md:89-92` | + +**Hand step in a runbook?** yes — `E/documentation/operations/tailscale.md:41-43` + +### DH31 — Gitea and registry credentials · Key · MENTIONED + +architecture folder: 2 lines in 1 files match `/gitea.{0,40}(token|credential)/` (a machine count, not a reading) · helper section: dates-hub „31. Gitea and registry credentials” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | NOT MEASURED. The login the cluster and the operator use to pull and push images and code. Operator. Row S15; the pull secret is a scoped token. | A | `RB/total-loss-of-dooplex.md:48` `RB/secrets.md:231` | +| 2 | NOT MEASURED (DooPlex, ep0 and the accounts are outside this run). No document gives an end date for the scoped token or for CC's Gitea token. | N | `RB/secrets.md:1` · class T3 · S13 | +| 3 | The operator, in Gitea. | A | `RB/total-loss-of-dooplex.md:48` `RB/secrets.md:231` (the code path of question 1) | +| 4 | If the pull token lapses: the hub pod cannot be re-pulled on a restart, and boxes pulling the controller image through the same registry would fail (box helper's item). | A | `RB/total-loss-of-dooplex.md:48` `RB/secrets.md:231` (the code path of question 1) | +| 5 | Nobody. | N | `RB/secrets.md:1` · class T1 · S13 | +| 6 | Not a box matter: a box pulls its images with no login. | A | `RB/secrets.md:1` | +| 7 | The renewal is the operator's hand step at the outside account. If it is not done, the thing ends (question 4). | A | `RB/total-loss-of-dooplex.md:48` `RB/secrets.md:231` (the code path of question 1) | +| 8 | DOC and. WHERE THE OPERATOR READS THE DATE: NOT FOUND. | A | `RB/secrets.md:231` `RB/total-loss-of-dooplex.md:48` | + +**Hand step in a runbook?** yes for a new token — `RB/secrets.md` + +### DH32 — CC's secret store: the tokens inside · Key · MENTIONED + +architecture folder: 3 lines in 2 files match `/cc-secrets|sec run/` (a machine count, not a reading) · helper section: dates-hub „32. CC's secret store: its key and the tokens inside (GitHub advisory token, hub token)” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | NOT MEASURED. The store CC uses through `sec`, and the key that opens it. Operator owns; row S19; design in /mnt/5_hdd/felhom.eu/git/cc-secrets/ARCHITECTURE.md (lines 47-48 give the key location). | A | `RB/total-loss-of-dooplex.md:52` | +| 2 | The store holds a value and no end date. A token with an end at its issuer (the GitHub advisory token, the hub token) ends with no record here. | N | `E/scripts/security_watch.py:1` · class T3 · S14 | +| 3 | The operator stores a new value; only he may delete one. | A | `RB/total-loss-of-dooplex.md:52` (the code path of question 1) | +| 4 | An expired GitHub token: the nightly security watch loses its source. An expired hub token: block 22. | A | `E/.gitea/workflows/security-watch.yml:22` | +| 5 | Nobody. The nightly security watch and the release step fail on the day. | N | `E/scripts/security_watch.py:1` · class T1 · S14 | +| 6 | Not a box matter. | A | `RB/total-loss-of-dooplex.md:52` (the code path of question 1) | +| 7 | The renewal is the operator's hand step at the outside account. If it is not done, the thing ends (question 4). | A | `RB/total-loss-of-dooplex.md:52` (the code path of question 1) | +| 8 | DOC. WHERE THE OPERATOR READS THE DATES: NOT FOUND. NOBODY KNOWS from files. | A | `RB/total-loss-of-dooplex.md:52` | + +**Hand step in a runbook?** NO + +### DH33 — SSH keys between the hub, DooPlex and the endpoint · Key · MENTIONED + +architecture folder: 5 lines in 4 files match `/ssh key|id_ed25519/` (a machine count, not a reading) · helper section: dates-hub „33. SSH keys between the hub, DooPlex and the endpoint” + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | NOT MEASURED. Keys the hub uses to reach the endpoint (three key files named at,) and the root key, row S9. Operator. | A | `E/manifests/hub.yaml:191` `E/manifests/hub.yaml:202` `E/manifests/hub.yaml:209` `RB/total-loss-of-dooplex.md:42` | +| 2 | NO END DATE in any document. The only rule: rotate on an endpoint rebuild, DOC. | A | `RB/offsite-endpoint.md:313` | +| 3 | The operator, on rebuild. | A | `E/manifests/hub.yaml:191` `E/manifests/hub.yaml:202` (the code path of question 1) | +| 4 | A lost or changed key: the hub cannot add tunnel peers, mint off-site tokens or poke the endpoint; new households get no off-site tier. | A | `E/manifests/hub.yaml:191` `E/manifests/hub.yaml:202` (the code path of question 1) | +| 5 | Nobody; no date. | A | `E/manifests/hub.yaml:191` `E/manifests/hub.yaml:202` (the code path of question 1) | +| 6 | Existing boxes keep their own tunnel and token; only new provisioning fails. | A | `E/manifests/hub.yaml:191` `E/manifests/hub.yaml:202` (the code path of question 1) | +| 7 | No renewal on a date exists, so none can fail on a date. | A | `E/manifests/hub.yaml:191` `E/manifests/hub.yaml:202` (the code path of question 1) | +| 8 | DOC; row S9 says one authorized key's owner is "not recorded". NOBODY KNOWS who holds it. | A | `RB/offsite-endpoint.md:313` `RB/total-loss-of-dooplex.md:42` | + +**Hand step in a runbook?** yes — `RB/offsite-endpoint.md:313` diff --git a/documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-felhom-guest-dates.txt b/documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-felhom-guest-dates.txt new file mode 100644 index 00000000..f72d513f --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-felhom-guest-dates.txt @@ -0,0 +1,124 @@ +# guest-dates demo-felhom taken 2026-11-11T10:06:04Z from DooPlex + +=== now === +2026-11-11T10:05:32Z + +=== layout === +stacks +actualbudget adventurelog audiobookshelf bentopdf bookstack calcom calibre-web claper cloudflared code-server crafty-controller dawarich docmost emby filebrowser ghost gitea glance gokapi grafana gramps-web grimmory grocy home-assistant homebox homepage immich jellyfin karakeep kimai komga lubelogger mealie metube n8n navidrome nextcloud nzbget onlyoffice opengist outline paperless-ngx papra plant-it plex privatebin prowlarr qbittorrent radarr radicale rallly recipe-importer romm sabnzbd seerr sonarr sparkyfitness tandoor termix traefik transmission uptime-kuma vaultwarden vikunja wanderer wger wishlist zipline +felhom-controller-data + +=== certificate files (system paths and infra stacks only) === + +=== acme stores (certificate fields only) === +--- /opt/docker/stacks/traefik/acme.json mtime=2026-10-02T19:50:44Z +letsencrypt *.demo-felhom.eu | subject=CN=*.demo-felhom.eu issuer=C=US, O=Let's Encrypt, CN=YR1 notBefore=Oct 2 18:52:11 2026 GMT notAfter=Dec 31 18:52:10 2026 GMT sha256 Fingerprint=7F:2C:1A:DB:15:29:2E:EC:B4:0F:84:B3:20:A2:59:88:29:33:4A:39:CD:1D:F5:B8:7F:2E:BC:98:FC:6C:96:95 + +=== served certificates (TLS handshake to own listeners) === +*:22 -> no TLS +0.0.0.0:443 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 04:37:09 2026 GMT notAfter=Nov 11 04:37:09 2027 GMT sha256 Fingerprint=B2:11:42:FE:A7:ED:57:0B:A7:B1:70:7A:09:A1:C3:21:44:D2:8D:7A:A3:B9:D7:90:97:89:E7:B3:64:EE:61:AC +0.0.0.0:80 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 04:37:09 2026 GMT notAfter=Nov 11 04:37:09 2027 GMT sha256 Fingerprint=B2:11:42:FE:A7:ED:57:0B:A7:B1:70:7A:09:A1:C3:21:44:D2:8D:7A:A3:B9:D7:90:97:89:E7:B3:64:EE:61:AC +127.0.0.1:25 -> no TLS +[::1]:25 -> no TLS +[::]:443 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 04:37:09 2026 GMT notAfter=Nov 11 04:37:09 2027 GMT sha256 Fingerprint=B2:11:42:FE:A7:ED:57:0B:A7:B1:70:7A:09:A1:C3:21:44:D2:8D:7A:A3:B9:D7:90:97:89:E7:B3:64:EE:61:AC +[::]:80 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 04:37:09 2026 GMT notAfter=Nov 11 04:37:09 2027 GMT sha256 Fingerprint=B2:11:42:FE:A7:ED:57:0B:A7:B1:70:7A:09:A1:C3:21:44:D2:8D:7A:A3:B9:D7:90:97:89:E7:B3:64:EE:61:AC + +=== os === +Debian GNU/Linux 13 (trixie) / trixie +13.7 + +=== docker === +client=29.9.3 server=29.9.3 +containerd.io 2.4.2-3~debian.13~trixie +docker-ce 5:29.9.3-1~debian.13~trixie +docker-compose-plugin 5.7.0-1~debian.13~trixie +gitea.dooplex.hu/admin/felhom-controller:0.352.0 + +=== restic and tools inside the controller === +restic 0.19.1 compiled with go1.26.4 on linux/amd64 +PRETTY_NAME="Debian GNU/Linux 13 (trixie)" +NAME="Debian GNU/Linux" +VERSION_ID="13" + +=== images: repo tag digest created === +alpine:latest sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b 2026-06-16 00:01:29 +0000 UTC +cloudflare/cloudflared:2026.10.3 sha256:faa8e0a1475f137c910be6025f43c8e4092fdd35461b31e19fd1390657a1ff39 2026-10-21 13:26:38 +0000 UTC +cloudflare/cloudflared:2026.10.4 sha256:c6c7ae1d5dfdcb094ad344ed68ccffc26635c9c6be80965664c33baf064b0515 2026-10-26 11:27:24 +0000 UTC +gitea.dooplex.hu/admin/felhom-controller:0.351.0 sha256:1d90563c443022770b2db90a7730f1ea5aea45cc5b0fa64fec9823d60d42d97f 2026-11-09 12:14:14 +0000 UTC +gitea.dooplex.hu/admin/felhom-controller:0.352.0 sha256:3c785dd396b091f19d4c777db9d7a206e75ab018b6238b5d887e1a4c6cfd3b29 2026-11-10 15:16:20 +0000 UTC +gitea.dooplex.hu/admin/felhom-samba:1.1.0 sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10 2026-07-20 11:36:16 +0000 UTC +gtstef/filebrowser:1.5.6-stable sha256:7c5d7ac8ffda31294d278063cf9d2e04303b39e6dce1f4c691342240ca7703b8 2026-09-04 21:33:58 +0000 UTC +gtstef/filebrowser:1.5.8-stable sha256:68455d4953bad8e984e1bc5d112ed1e9c918a0a04f104c13c9ece50790308894 2026-10-04 21:15:40 +0000 UTC +hello-world:latest sha256:c3cbe1cc1aa588a64951ac6286e0df7b27fe2e6324b1001c619bb358770c0178 2026-03-23 21:33:59 +0000 UTC +traefik:v3.7.16 sha256:1336e85aaed293bd63d005086e0dd013b04078b26aaeb1c18b26a2f928af7e65 2026-10-21 22:55:46 +0000 UTC +traefik:v3.7.17 sha256:68ae5fe5b64ca166704b746a9e8afcd385e36b3f569a5691401bf652d57b3748 2026-10-30 20:17:26 +0000 UTC + +=== apt sources === +--- /etc/apt/sources.list.d/debian.sources +Types: deb +URIs: http://security.debian.org +Suites: trixie-security +Components: contrib main +Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg +Types: deb +URIs: http://deb.debian.org/debian +Suites: trixie trixie-updates +Components: contrib main +Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg +--- /etc/apt/sources.list.d/docker.list +deb [signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian trixie stable + +=== apt keyring files === + + +3817 2026-08-03 docker.asc + + + +11861 2025-04-09 debian-archive-bookworm-automatic.asc +11873 2025-04-09 debian-archive-bookworm-security-automatic.asc +461 2025-04-09 debian-archive-bookworm-stable.asc +11861 2025-04-09 debian-archive-bullseye-automatic.asc +11873 2025-04-09 debian-archive-bullseye-security-automatic.asc +3403 2025-04-09 debian-archive-bullseye-stable.asc +11861 2025-04-09 debian-archive-trixie-automatic.asc +11873 2025-04-09 debian-archive-trixie-security-automatic.asc +1384 2025-04-09 debian-archive-trixie-stable.asc + + + +37 2025-04-09 debian-archive-bookworm-automatic.gpg +8700 2025-04-09 debian-archive-bookworm-automatic.pgp +46 2025-04-09 debian-archive-bookworm-security-automatic.gpg +8709 2025-04-09 debian-archive-bookworm-security-automatic.pgp +34 2025-04-09 debian-archive-bookworm-stable.gpg +280 2025-04-09 debian-archive-bookworm-stable.pgp +37 2025-04-09 debian-archive-bullseye-automatic.gpg +8700 2025-04-09 debian-archive-bullseye-automatic.pgp +46 2025-04-09 debian-archive-bullseye-security-automatic.gpg +8709 2025-04-09 debian-archive-bullseye-security-automatic.pgp +34 2025-04-09 debian-archive-bullseye-stable.gpg +2453 2025-04-09 debian-archive-bullseye-stable.pgp +26 2025-04-09 debian-archive-keyring.gpg +55918 2025-04-09 debian-archive-keyring.pgp +31 2025-04-09 debian-archive-removed-keys.gpg +72636 2025-04-09 debian-archive-removed-keys.pgp +35 2025-04-09 debian-archive-trixie-automatic.gpg +8698 2025-04-09 debian-archive-trixie-automatic.pgp +44 2025-04-09 debian-archive-trixie-security-automatic.gpg +8707 2025-04-09 debian-archive-trixie-security-automatic.pgp +32 2025-04-09 debian-archive-trixie-stable.gpg +962 2025-04-09 debian-archive-trixie-stable.pgp + +=== apt release files: Valid-Until === +deb.debian.org_debian_dists_trixie-updates_InRelease | Date: Wed, 11 Nov 2026 02:14:24 UTC | Valid-Until: Wed, 18 Nov 2026 02:14:24 UTC +deb.debian.org_debian_dists_trixie_InRelease | Date: Sat, 12 Sep 2026 07:55:41 UTC | Valid-Until: +download.docker.com_linux_debian_dists_trixie_InRelease | Date: Fri, 06 Nov 2026 03:19:25 +0000 | Valid-Until: +security.debian.org_dists_trixie-security_InRelease | Date: Wed, 11 Nov 2026 00:23:32 UTC | Valid-Until: Wed, 18 Nov 2026 00:23:32 UTC + +=== controller data folder (top level: names and dates only) === +--- felhom-controller-data +total +-rw------- 2993 2026-09-01 08:32 controller.yaml +drwxr-xr-x 4096 2026-11-11 10:05 data +RC=0 diff --git a/documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-felhom-host-dates.txt b/documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-felhom-host-dates.txt new file mode 100644 index 00000000..95ee6307 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-felhom-host-dates.txt @@ -0,0 +1,235 @@ +# host-dates demo-felhom taken 2026-11-11T10:05:43Z from DooPlex + +=== now === +2026-11-11T10:05:10Z + +=== certificate files === +/etc/pve/local/pve-ssl.pem [certs=1 mtime=2026-07-18] subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-felhom.local issuer=CN=Proxmox Virtual Environment, OU=5bd8f1aa-d679-4c28-ba2a-d2d16cf957c3, O=PVE Cluster Manager CA notBefore=Jul 17 16:17:07 2026 GMT notAfter=Jul 16 16:17:07 2028 GMT sha256 Fingerprint=60:8F:4C:50:C3:8E:4F:F7:49:01:0A:70:6D:40:13:AE:18:C4:43:60:A8:5B:94:F4:6D:28:23:78:C6:E2:F1:78 +/etc/pve/nodes/demo-felhom/pve-ssl.pem [certs=1 mtime=2026-07-18] subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-felhom.local issuer=CN=Proxmox Virtual Environment, OU=5bd8f1aa-d679-4c28-ba2a-d2d16cf957c3, O=PVE Cluster Manager CA notBefore=Jul 17 16:17:07 2026 GMT notAfter=Jul 16 16:17:07 2028 GMT sha256 Fingerprint=60:8F:4C:50:C3:8E:4F:F7:49:01:0A:70:6D:40:13:AE:18:C4:43:60:A8:5B:94:F4:6D:28:23:78:C6:E2:F1:78 +/etc/pve/pve-root-ca.pem [certs=1 mtime=2026-07-18] subject=CN=Proxmox Virtual Environment, OU=5bd8f1aa-d679-4c28-ba2a-d2d16cf957c3, O=PVE Cluster Manager CA issuer=CN=Proxmox Virtual Environment, OU=5bd8f1aa-d679-4c28-ba2a-d2d16cf957c3, O=PVE Cluster Manager CA notBefore=Jul 17 16:17:07 2026 GMT notAfter=Jul 14 16:17:07 2036 GMT sha256 Fingerprint=53:17:92:A0:1E:4B:20:FA:7D:1F:26:AA:FE:C9:DC:9F:BD:37:66:32:12:0F:36:F7:61:06:EA:04:49:3E:E4:C7 +/var/lib/felhom-agent/local-api.crt [certs=1 mtime=2026-08-03] subject=CN=felhom-agent local-api issuer=CN=felhom-agent local-api notBefore=Aug 3 06:15:30 2026 GMT notAfter=Jul 31 06:15:30 2036 GMT sha256 Fingerprint=C4:B9:6B:F0:F9:7E:FC:D4:62:B1:1D:33:BD:39:A2:81:33:B4:B3:64:85:35:EE:5B:49:7C:00:5E:AD:F3:E5:1E + +=== system CA store === +148 +ca-certificates 20250419+deb13u1 + +=== served certificates (TLS handshake to own listeners) === +0.0.0.0:111 -> no TLS +0.0.0.0:22 -> no TLS +0.0.0.0:8822 -> no TLS +100.70.170.35%tailscale0:36209 -> no TLS +[::]:111 -> no TLS +[::1]:25 -> no TLS +127.0.0.1:25 -> no TLS +127.0.0.1:53 -> no TLS +127.0.0.1:85 -> no TLS +169.254.253.1:8443 -> subject=CN=felhom-agent local-api issuer=CN=felhom-agent local-api notBefore=Aug 3 06:15:30 2026 GMT notAfter=Jul 31 06:15:30 2036 GMT sha256 Fingerprint=C4:B9:6B:F0:F9:7E:FC:D4:62:B1:1D:33:BD:39:A2:81:33:B4:B3:64:85:35:EE:5B:49:7C:00:5E:AD:F3:E5:1E +192.168.0.162:53 -> no TLS +[::]:22 -> no TLS +*:3128 -> no TLS +*:8006 -> subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-felhom.local issuer=CN=Proxmox Virtual Environment, OU=5bd8f1aa-d679-4c28-ba2a-d2d16cf957c3, O=PVE Cluster Manager CA notBefore=Jul 17 16:17:07 2026 GMT notAfter=Jul 16 16:17:07 2028 GMT sha256 Fingerprint=60:8F:4C:50:C3:8E:4F:F7:49:01:0A:70:6D:40:13:AE:18:C4:43:60:A8:5B:94:F4:6D:28:23:78:C6:E2:F1:78 +[::]:8822 -> no TLS +[fd7a:115c:a1e0::5236:aa24]%tailscale0:61570 -> no TLS + +=== listeners (port, process) === +0.0.0.0:111 users:(("rpcbind",),("systemd",)) +0.0.0.0:22 users:(("sshd",)) +0.0.0.0:8822 users:(("sshd",)) +100.70.170.35%tailscale0:36209 users:(("tailscaled",)) +[::]:111 users:(("rpcbind",),("systemd",)) +[::1]:25 users:(("smtpd",),("master",)) +127.0.0.1:25 users:(("smtpd",),("master",)) +127.0.0.1:53 users:(("dnsmasq",)) +127.0.0.1:85 users:(("pvedaemon +169.254.253.1:8443 users:(("felhom-agent",)) +192.168.0.162:53 users:(("dnsmasq",)) +[::]:22 users:(("sshd",)) +*:3128 users:(("spiceproxy +*:8006 users:(("pveproxy +[::]:8822 users:(("sshd",)) +[fd7a:115c:a1e0::5236:aa24]%tailscale0:61570 users:(("tailscaled",)) + +=== pve users and tokens (expire: 0 = never) === +user felhom-agent@pve enable 1 expire 0 tokens [] +user root@pam enable 1 expire 0 tokens [] + +=== acme / subscription === +message: There is no subscription key +status: notfound + +=== os === +Debian GNU/Linux 13 (trixie) / trixie +13.7 +pve-manager/9.2.26/41a0e2e09b9e6f9d (running kernel: 7.0.14-28-pve) +7.0.14-28-pve +Manually selected kernels: +None. + +Automatically selected kernels: +7.0.14-26-pve +7.0.14-28-pve +proxmox-kernel-7.0.14-20-pve-signed 7.0.14-20 +proxmox-kernel-7.0.14-22-pve-signed 7.0.14-22 +proxmox-kernel-7.0.14-23-pve-signed 7.0.14-23 +proxmox-kernel-7.0.14-24-pve-signed 7.0.14-24 +proxmox-kernel-7.0.14-25-pve-signed 7.0.14-25 +proxmox-kernel-7.0.14-26-pve-signed 7.0.14-26 +proxmox-kernel-7.0.14-28-pve-signed 7.0.14-28 +proxmox-kernel-7.0.2-6-pve-signed 7.0.2-6 + +=== apt sources === +--- /etc/apt/sources.list.d/ceph.sources +Types: deb +URIs: https://enterprise.proxmox.com/debian/ceph-squid +Suites: trixie +Components: enterprise +Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg +Enabled: no +--- /etc/apt/sources.list.d/debian.sources +Types: deb +URIs: http://deb.debian.org/debian/ +Suites: trixie trixie-updates +Components: main contrib non-free-firmware +Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg +Types: deb +URIs: http://security.debian.org/debian-security/ +Suites: trixie-security +Components: main contrib non-free-firmware +Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg +--- /etc/apt/sources.list.d/pve-enterprise.sources +Types: deb +URIs: https://enterprise.proxmox.com/debian/pve +Suites: trixie +Components: pve-enterprise +Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg +Enabled: no +--- /etc/apt/sources.list.d/pve-no-subscription.sources +Types: deb +URIs: http://download.proxmox.com/debian/pve +Suites: trixie +Components: pve-no-subscription +Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg +--- /etc/apt/sources.list.d/tailscale.list +deb [signed-by=/usr/share/keyrings/tailscale-archive-keyring.gpg] https://pkgs.tailscale.com/stable/debian trixie main +--- auth files (names only): /etc/apt/auth.conf.d: + +=== apt keyring files === + + + + + +11861 2025-04-10 debian-archive-bookworm-automatic.asc +11873 2025-04-10 debian-archive-bookworm-security-automatic.asc +461 2025-04-10 debian-archive-bookworm-stable.asc +11861 2025-04-10 debian-archive-bullseye-automatic.asc +11873 2025-04-10 debian-archive-bullseye-security-automatic.asc +3403 2025-04-10 debian-archive-bullseye-stable.asc +11861 2025-04-10 debian-archive-trixie-automatic.asc +11873 2025-04-10 debian-archive-trixie-security-automatic.asc +1384 2025-04-10 debian-archive-trixie-stable.asc +1187 2022-11-27 proxmox-release-bookworm.gpg +1185 2024-11-10 proxmox-release-trixie.gpg + + + +37 2025-04-10 debian-archive-bookworm-automatic.gpg +8700 2025-04-10 debian-archive-bookworm-automatic.pgp +46 2025-04-10 debian-archive-bookworm-security-automatic.gpg +8709 2025-04-10 debian-archive-bookworm-security-automatic.pgp +34 2025-04-10 debian-archive-bookworm-stable.gpg +280 2025-04-10 debian-archive-bookworm-stable.pgp +37 2025-04-10 debian-archive-bullseye-automatic.gpg +8700 2025-04-10 debian-archive-bullseye-automatic.pgp +46 2025-04-10 debian-archive-bullseye-security-automatic.gpg +8709 2025-04-10 debian-archive-bullseye-security-automatic.pgp +34 2025-04-10 debian-archive-bullseye-stable.gpg +2453 2025-04-10 debian-archive-bullseye-stable.pgp +26 2025-04-10 debian-archive-keyring.gpg +55918 2025-04-10 debian-archive-keyring.pgp +31 2025-04-10 debian-archive-removed-keys.gpg +72636 2025-04-10 debian-archive-removed-keys.pgp +35 2025-04-10 debian-archive-trixie-automatic.gpg +8698 2025-04-10 debian-archive-trixie-automatic.pgp +44 2025-04-10 debian-archive-trixie-security-automatic.gpg +8707 2025-04-10 debian-archive-trixie-security-automatic.pgp +32 2025-04-10 debian-archive-trixie-stable.gpg +962 2025-04-10 debian-archive-trixie-stable.pgp +2372 2025-07-09 proxmox-archive-keyring.gpg +113 2022-11-27 proxmox-offline-signing-key.pub +1187 2022-11-27 proxmox-release-bookworm.gpg +1185 2024-11-10 proxmox-release-trixie.gpg +2288 2023-01-23 tailscale-archive-keyring.gpg + +=== apt release files: Valid-Until === +deb.debian.org_debian_dists_trixie_InRelease | Date: Sat, 12 Sep 2026 07:55:41 UTC | Valid-Until: +deb.debian.org_debian_dists_trixie-updates_InRelease | Date: Wed, 11 Nov 2026 02:14:24 UTC | Valid-Until: Wed, 18 Nov 2026 02:14:24 UTC +download.proxmox.com_debian_pve_dists_trixie_InRelease | Date: Wed, 11 Nov 2026 01:41:17 +0000 | Valid-Until: +pkgs.tailscale.com_stable_debian_dists_trixie_InRelease | Date: Wed, 11 Nov 2026 00:00:32 UTC | Valid-Until: +security.debian.org_debian-security_dists_trixie-security_InRelease | Date: Wed, 11 Nov 2026 00:23:32 UTC | Valid-Until: Wed, 18 Nov 2026 00:23:32 UTC + +=== templates and golden === +/var/lib/vz 129954319 2026-07-14 debian-13-standard_13.6-1_amd64.tar.zst + +=== agent files (names and dates only) === +/etc/felhom: +total +-rw-r--r-- root 7 2026-07-18 18:17 appliance-pairing-code +-rw-r--r-- root 3897 2026-11-07 23:20 config-bundle.json +-rw-r--r-- root 456 2026-10-04 16:12 crash-guard.conf +-rw-r--r-- root 131 2026-10-04 16:12 operator-signers +-rw-r--r-- root 65 2026-10-04 18:34 os-trust.json + +/etc/felhom-agent: +total +-rw------- felhom-agent 2467 2026-09-24 21:36 agent.json +-rw------- root 2467 2026-08-04 09:32 agent.json.bak-r191 +-rw------- felhom-agent 2465 2026-07-27 11:31 agent.json.campaign8-before +-rw------- root 2465 2026-07-28 17:36 agent.json.campaign9-before +-rw------- root 2465 2026-07-27 11:31 agent.json.campaign9-prev +-rw------- felhom-agent 2514 2026-09-24 15:33 agent.json.night-0925-off +-rw------- felhom-agent 2465 2026-07-28 18:10 agent.json.pre-e-target-move +-rw------- felhom-agent 2464 2026-07-27 08:08 agent.json.pre-prunegate.bak +-rw------- root 2467 2026-08-08 20:04 agent.json.pre-r672 + +/var/lib/felhom-agent: +total +-rw------- felhom-agent 631 2026-11-11 05:37 backup-success-state.json +-rw------- felhom-agent 290 2026-11-10 16:16 controller-swap-9201.json +-rw------- felhom-agent 61 2026-08-03 09:15 drive-intents.json +drwx------ felhom-agent 4096 2026-10-03 17:03 escrow-stage +drwx------ felhom-agent 4096 2026-08-03 09:15 felhom-sshd +-rw------- felhom-agent 67 2026-10-14 15:29 guest-binds.json +-rw------- felhom-agent 243 2026-11-11 10:55 guest-disk-trim.json +drwx------ felhom-agent 4096 2026-08-03 09:17 guests +-rw------- felhom-agent 65682 2026-11-11 10:22 journal.log +-rw------- felhom-agent 305 2026-11-10 10:22 line-test.json +-rw-r--r-- felhom-agent 591 2026-08-03 09:15 local-api.crt +-rw------- felhom-agent 227 2026-08-03 09:15 local-api.key +-rw------- felhom-agent 164 2026-08-09 09:51 local-tokens.log +-rw------- felhom-agent 5508 2026-11-10 15:50 nonces.log +drwx------ felhom-agent 4096 2026-11-11 10:50 os +drwx------ felhom-agent 4096 2026-08-03 09:15 pbsdr +-rw------- felhom-agent 77 2026-10-13 20:52 poweroff.json +-rw------- felhom-agent 383 2026-11-11 10:22 restore-test-state.json +drwxr-x--- felhom-agent 4096 2026-11-10 15:51 selfupdate +-rw------- felhom-agent 40 2026-11-03 08:49 store-grant-seen.json +drwx------ felhom-agent 4096 2026-08-03 09:15 wg + +=== agent === +felhom-agent 0.173.0 + +=== wireguard (interface and latest handshake time only) === +wg-felhom peer f3d1ZI… 2026-11-11T10:05:30Z + +=== tailscale === +self key expiry: None online: True + +=== ssh host keys / authorized keys (counts only) === +3 +/root/.ssh/authorized_keys lines=3 + +=== smart: power-on hours and wear (no serials) === +sda: 9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 5155; +sdb: 9 Power_On_Hours 0x0032 085 085 000 Old_age Always - 6093; +RC=0 diff --git a/documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-hp-guest-dates.txt b/documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-hp-guest-dates.txt new file mode 100644 index 00000000..0c045334 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-hp-guest-dates.txt @@ -0,0 +1,137 @@ +# guest-dates demo-hp taken 2026-11-11T10:06:05Z from DooPlex + +=== now === +2026-11-11T10:06:06Z + +=== layout === +stacks +actualbudget adventurelog audiobookshelf bentopdf bookstack calcom calibre-web claper cloudflared code-server crafty-controller dawarich docmost emby filebrowser ghost gitea glance gokapi grafana gramps-web grimmory grocy home-assistant homebox homepage immich jellyfin karakeep kimai komga lubelogger mealie metube n8n navidrome nextcloud nzbget onlyoffice opengist outline paperless-ngx papra plant-it plex privatebin prowlarr qbittorrent radarr radicale rallly recipe-importer romm sabnzbd samba seerr sonarr sparkyfitness tandoor termix traefik transmission uptime-kuma vaultwarden vikunja wanderer wger wishlist zipline +felhom-controller-data + +=== certificate files (system paths and infra stacks only) === + +=== acme stores (certificate fields only) === +--- /opt/docker/stacks/traefik/acme.json mtime=2026-10-21T02:44:33Z +letsencrypt *.enkisfelhom.hu | subject=CN=*.enkisfelhom.hu issuer=C=US, O=Let's Encrypt, CN=YR2 notBefore=Oct 21 01:46:01 2026 GMT notAfter=Jan 19 01:46:00 2027 GMT sha256 Fingerprint=A1:C4:17:B6:34:2C:DC:FD:B2:A0:75:AA:22:EA:C1:E0:3B:CE:AA:8D:A9:CF:79:86:F2:A9:B5:55:F2:91:00:5C + +=== served certificates (TLS handshake to own listeners) === +*:22 -> no TLS +0.0.0.0:443 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 03:35:36 2026 GMT notAfter=Nov 11 03:35:36 2027 GMT sha256 Fingerprint=06:13:E1:5D:F0:97:57:20:45:3A:94:9D:4B:F3:A9:B4:13:91:07:89:72:E9:DD:9F:AA:9F:5F:9B:96:8D:D2:2D +0.0.0.0:80 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 03:35:36 2026 GMT notAfter=Nov 11 03:35:36 2027 GMT sha256 Fingerprint=06:13:E1:5D:F0:97:57:20:45:3A:94:9D:4B:F3:A9:B4:13:91:07:89:72:E9:DD:9F:AA:9F:5F:9B:96:8D:D2:2D +127.0.0.1:25 -> no TLS +[::1]:25 -> no TLS +[::]:443 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 03:35:36 2026 GMT notAfter=Nov 11 03:35:36 2027 GMT sha256 Fingerprint=06:13:E1:5D:F0:97:57:20:45:3A:94:9D:4B:F3:A9:B4:13:91:07:89:72:E9:DD:9F:AA:9F:5F:9B:96:8D:D2:2D +[::]:80 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 03:35:36 2026 GMT notAfter=Nov 11 03:35:36 2027 GMT sha256 Fingerprint=06:13:E1:5D:F0:97:57:20:45:3A:94:9D:4B:F3:A9:B4:13:91:07:89:72:E9:DD:9F:AA:9F:5F:9B:96:8D:D2:2D + +=== os === +Debian GNU/Linux 13 (trixie) / trixie +13.7 + +=== docker === +client=29.9.3 server=29.9.3 +containerd.io 2.4.2-3~debian.13~trixie +docker-ce 5:29.9.3-1~debian.13~trixie +docker-compose-plugin 5.7.0-1~debian.13~trixie +gitea.dooplex.hu/admin/felhom-controller:0.352.0 + +=== restic and tools inside the controller === +restic 0.19.1 compiled with go1.26.4 on linux/amd64 +PRETTY_NAME="Debian GNU/Linux 13 (trixie)" +NAME="Debian GNU/Linux" +VERSION_ID="13" + +=== images: repo tag digest created === +alpine:latest sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b 2026-06-16 00:01:29 +0000 UTC +cloudflare/cloudflared:2026.10.3 sha256:faa8e0a1475f137c910be6025f43c8e4092fdd35461b31e19fd1390657a1ff39 2026-10-21 13:26:38 +0000 UTC +cloudflare/cloudflared:2026.10.4 sha256:c6c7ae1d5dfdcb094ad344ed68ccffc26635c9c6be80965664c33baf064b0515 2026-10-26 11:27:24 +0000 UTC +crocodilestick/calibre-web-automated:v4.0.6 sha256:c31a738b6d5ec6982c050063dd3f063b6943eb1051fc81144789f840d9093a8d 2026-02-04 23:42:57 +0000 UTC +docmost/docmost:0.96.0 sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a 2026-09-08 15:49:25 +0000 UTC +ghcr.io/alam00000/bentopdf:v2.8.6 sha256:eaeea1e447205a79cb61d7efdc6966f37311dc1bc9c36a3a5c897bf79107c2c3 2026-06-28 06:30:01 +0000 UTC +ghcr.io/paperless-ngx/paperless-ngx:2.20.15 sha256:6c86cad803970ea782683a8e80e7403444c5bf3cf70de63b4d3c8e87500db92f 2026-04-27 02:45:05 +0000 UTC +ghcr.io/thomiceli/opengist:1.15 sha256:7edc91273ee7d10a17406da8a08c803158baaff984c39b789c1313473d068f21 2026-08-30 02:07:48 +0000 UTC +gitea.dooplex.hu/admin/felhom-controller:0.351.0 sha256:1d90563c443022770b2db90a7730f1ea5aea45cc5b0fa64fec9823d60d42d97f 2026-11-09 12:14:14 +0000 UTC +gitea.dooplex.hu/admin/felhom-controller:0.352.0 sha256:3c785dd396b091f19d4c777db9d7a206e75ab018b6238b5d887e1a4c6cfd3b29 2026-11-10 15:16:20 +0000 UTC +gitea.dooplex.hu/admin/felhom-samba:1.2.0 sha256:da96f25033827fbb77d0ee398e86fe5b744e2f29be9de898994533340c97acf9 2026-10-16 18:25:34 +0000 UTC +gtstef/filebrowser:1.5.6-stable sha256:7c5d7ac8ffda31294d278063cf9d2e04303b39e6dce1f4c691342240ca7703b8 2026-09-04 21:33:58 +0000 UTC +gtstef/filebrowser:1.5.8-stable sha256:68455d4953bad8e984e1bc5d112ed1e9c918a0a04f104c13c9ece50790308894 2026-10-04 21:15:40 +0000 UTC +mariadb:11.8 sha256:79d59758afc91b89b120b0a8904d637f5a3b3e1c4900f29b740d6d46c72fef68 2026-09-16 03:26:24 +0000 UTC +mariadb:12.3 sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 2026-09-16 03:26:24 +0000 UTC +plexinc/pms-docker:1.41.4.9463-630c9f557 sha256:b3a376a04fa1912af17a61755089ad2324502fe1b2f8bbf042eab31afe36fc25 2025-02-07 10:23:03 +0000 UTC +plexinc/pms-docker:1.43.5.11029-6bf8af3f0 sha256:01bdf0ad6d53198da4331540c1e73ab65db7a72e961fff97ecb0847d46055c5f 2026-10-13 07:57:12 +0000 UTC +postgis/postgis:16-3.5-alpine sha256:47e961a569fd52ff31f0fe205ed91eeab17d9f5fff6722e6d7ea6b588748b293 2026-08-31 11:36:26 +0000 UTC +privatebin/pdo:2.0.6 sha256:4c141b2326f8b353598ce9ce7507a9cfecf2dad5c60a39fea903d430e296d8f5 2026-08-08 06:24:54 +0000 UTC +redis:7-alpine sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 2026-09-17 21:32:46 +0000 UTC +traefik:v3.7.16 sha256:1336e85aaed293bd63d005086e0dd013b04078b26aaeb1c18b26a2f928af7e65 2026-10-21 22:55:46 +0000 UTC +traefik:v3.7.17 sha256:68ae5fe5b64ca166704b746a9e8afcd385e36b3f569a5691401bf652d57b3748 2026-10-30 20:17:26 +0000 UTC + +=== apt sources === +--- /etc/apt/sources.list.d/debian.sources +Types: deb +URIs: http://security.debian.org +Suites: trixie-security +Components: contrib main +Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg +Types: deb +URIs: http://deb.debian.org/debian +Suites: trixie trixie-updates +Components: contrib main +Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg +--- /etc/apt/sources.list.d/docker.list +deb [signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian trixie stable + +=== apt keyring files === + + +3817 2026-08-18 docker.asc + + + +11861 2025-04-09 debian-archive-bookworm-automatic.asc +11873 2025-04-09 debian-archive-bookworm-security-automatic.asc +461 2025-04-09 debian-archive-bookworm-stable.asc +11861 2025-04-09 debian-archive-bullseye-automatic.asc +11873 2025-04-09 debian-archive-bullseye-security-automatic.asc +3403 2025-04-09 debian-archive-bullseye-stable.asc +11861 2025-04-09 debian-archive-trixie-automatic.asc +11873 2025-04-09 debian-archive-trixie-security-automatic.asc +1384 2025-04-09 debian-archive-trixie-stable.asc + + + +37 2025-04-09 debian-archive-bookworm-automatic.gpg +8700 2025-04-09 debian-archive-bookworm-automatic.pgp +46 2025-04-09 debian-archive-bookworm-security-automatic.gpg +8709 2025-04-09 debian-archive-bookworm-security-automatic.pgp +34 2025-04-09 debian-archive-bookworm-stable.gpg +280 2025-04-09 debian-archive-bookworm-stable.pgp +37 2025-04-09 debian-archive-bullseye-automatic.gpg +8700 2025-04-09 debian-archive-bullseye-automatic.pgp +46 2025-04-09 debian-archive-bullseye-security-automatic.gpg +8709 2025-04-09 debian-archive-bullseye-security-automatic.pgp +34 2025-04-09 debian-archive-bullseye-stable.gpg +2453 2025-04-09 debian-archive-bullseye-stable.pgp +26 2025-04-09 debian-archive-keyring.gpg +55918 2025-04-09 debian-archive-keyring.pgp +31 2025-04-09 debian-archive-removed-keys.gpg +72636 2025-04-09 debian-archive-removed-keys.pgp +35 2025-04-09 debian-archive-trixie-automatic.gpg +8698 2025-04-09 debian-archive-trixie-automatic.pgp +44 2025-04-09 debian-archive-trixie-security-automatic.gpg +8707 2025-04-09 debian-archive-trixie-security-automatic.pgp +32 2025-04-09 debian-archive-trixie-stable.gpg +962 2025-04-09 debian-archive-trixie-stable.pgp + +=== apt release files: Valid-Until === +deb.debian.org_debian_dists_trixie-updates_InRelease | Date: Wed, 11 Nov 2026 02:14:24 UTC | Valid-Until: Wed, 18 Nov 2026 02:14:24 UTC +deb.debian.org_debian_dists_trixie_InRelease | Date: Sat, 12 Sep 2026 07:55:41 UTC | Valid-Until: +download.docker.com_linux_debian_dists_trixie_InRelease | Date: Fri, 06 Nov 2026 03:19:25 +0000 | Valid-Until: +security.debian.org_dists_trixie-security_InRelease | Date: Wed, 11 Nov 2026 00:23:32 UTC | Valid-Until: Wed, 18 Nov 2026 00:23:32 UTC + +=== controller data folder (top level: names and dates only) === +--- felhom-controller-data +total +-rw------- 2989 2026-08-31 08:32 controller.yaml +-rw------- 2989 2026-08-31 08:32 controller.yaml.pre-gate-day +-rw------- 2989 2026-08-31 08:32 controller.yaml.pre-undofleet +drwxr-xr-x 4096 2026-11-11 10:06 data +RC=0 diff --git a/documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-hp-host-dates.txt b/documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-hp-host-dates.txt new file mode 100644 index 00000000..75bb8046 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-hp-host-dates.txt @@ -0,0 +1,218 @@ +# host-dates demo-hp taken 2026-11-11T10:05:43Z from DooPlex + +=== now === +2026-11-11T10:05:43Z + +=== certificate files === +/etc/pve/local/pve-ssl.pem [certs=1 mtime=2026-08-21] subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA notBefore=Aug 20 15:44:42 2026 GMT notAfter=Aug 19 15:44:42 2028 GMT sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB +/etc/pve/nodes/demo-hp/pve-ssl.pem [certs=1 mtime=2026-08-21] subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA notBefore=Aug 20 15:44:42 2026 GMT notAfter=Aug 19 15:44:42 2028 GMT sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB +/etc/pve/pve-root-ca.pem [certs=1 mtime=2026-08-21] subject=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA notBefore=Aug 20 15:44:42 2026 GMT notAfter=Aug 17 15:44:42 2036 GMT sha256 Fingerprint=E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7 +/var/lib/felhom-agent/local-api.crt [certs=1 mtime=2026-08-21] subject=CN=felhom-agent local-api issuer=CN=felhom-agent local-api notBefore=Aug 21 14:59:12 2026 GMT notAfter=Aug 18 14:59:12 2036 GMT sha256 Fingerprint=34:12:2A:C2:AC:E9:91:68:5A:D9:AD:AA:F8:82:70:25:81:AE:93:18:37:20:3D:BE:B2:5F:0D:9E:49:3A:83:7D + +=== system CA store === +148 +ca-certificates 20250419+deb13u1 + +=== served certificates (TLS handshake to own listeners) === +0.0.0.0:111 -> no TLS +0.0.0.0:22 -> no TLS +0.0.0.0:8822 -> no TLS +[::]:111 -> no TLS +[::1]:25 -> no TLS +127.0.0.1:25 -> no TLS +127.0.0.1:53 -> no TLS +127.0.0.1:85 -> no TLS +169.254.253.1:8443 -> subject=CN=felhom-agent local-api issuer=CN=felhom-agent local-api notBefore=Aug 21 14:59:12 2026 GMT notAfter=Aug 18 14:59:12 2036 GMT sha256 Fingerprint=34:12:2A:C2:AC:E9:91:68:5A:D9:AD:AA:F8:82:70:25:81:AE:93:18:37:20:3D:BE:B2:5F:0D:9E:49:3A:83:7D +192.168.0.104:53 -> no TLS +[::]:22 -> no TLS +*:3128 -> no TLS +*:8006 -> subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA notBefore=Aug 20 15:44:42 2026 GMT notAfter=Aug 19 15:44:42 2028 GMT sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB +[::]:8822 -> no TLS + +=== listeners (port, process) === +0.0.0.0:111 users:(("rpcbind",),("systemd",)) +0.0.0.0:22 users:(("sshd",)) +0.0.0.0:8822 users:(("sshd",)) +[::]:111 users:(("rpcbind",),("systemd",)) +[::1]:25 users:(("smtpd",),("master",)) +127.0.0.1:25 users:(("smtpd",),("master",)) +127.0.0.1:53 users:(("dnsmasq",)) +127.0.0.1:85 users:(("pvedaemon +169.254.253.1:8443 users:(("felhom-agent",)) +192.168.0.104:53 users:(("dnsmasq",)) +[::]:22 users:(("sshd",)) +*:3128 users:(("spiceproxy +*:8006 users:(("pveproxy +[::]:8822 users:(("sshd",)) + +=== pve users and tokens (expire: 0 = never) === +user felhom-agent@pve enable 1 expire 0 tokens [] +user root@pam enable 1 expire 0 tokens [] + +=== acme / subscription === +message: There is no subscription key +status: notfound + +=== os === +Debian GNU/Linux 13 (trixie) / trixie +13.7 +pve-manager/9.2.26/41a0e2e09b9e6f9d (running kernel: 7.0.14-28-pve) +7.0.14-28-pve +Manually selected kernels: +None. + +Automatically selected kernels: +7.0.14-26-pve +7.0.14-28-pve +proxmox-kernel-7.0.14-26-pve-signed 7.0.14-26 +proxmox-kernel-7.0.14-28-pve-signed 7.0.14-28 + +=== apt sources === +--- /etc/apt/sources.list.d/ceph.sources +Types: deb +URIs: https://enterprise.proxmox.com/debian/ceph-squid +Suites: trixie +Components: enterprise +Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg +Enabled: no +--- /etc/apt/sources.list.d/debian.sources +Types: deb +URIs: http://deb.debian.org/debian/ +Suites: trixie trixie-updates +Components: main contrib non-free-firmware +Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg +Types: deb +URIs: http://security.debian.org/debian-security/ +Suites: trixie-security +Components: main contrib non-free-firmware +Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg +--- /etc/apt/sources.list.d/pve-enterprise.sources +Types: deb +URIs: https://enterprise.proxmox.com/debian/pve +Suites: trixie +Components: pve-enterprise +Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg +Enabled: no +--- /etc/apt/sources.list.d/pve-no-subscription.sources +Types: deb +URIs: http://download.proxmox.com/debian/pve +Suites: trixie +Components: pve-no-subscription +Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg +--- auth files (names only): /etc/apt/auth.conf.d: + +=== apt keyring files === + + + + + +11861 2025-04-10 debian-archive-bookworm-automatic.asc +11873 2025-04-10 debian-archive-bookworm-security-automatic.asc +461 2025-04-10 debian-archive-bookworm-stable.asc +11861 2025-04-10 debian-archive-bullseye-automatic.asc +11873 2025-04-10 debian-archive-bullseye-security-automatic.asc +3403 2025-04-10 debian-archive-bullseye-stable.asc +11861 2025-04-10 debian-archive-trixie-automatic.asc +11873 2025-04-10 debian-archive-trixie-security-automatic.asc +1384 2025-04-10 debian-archive-trixie-stable.asc +1187 2022-11-27 proxmox-release-bookworm.gpg +1185 2024-11-10 proxmox-release-trixie.gpg + + + +37 2025-04-10 debian-archive-bookworm-automatic.gpg +8700 2025-04-10 debian-archive-bookworm-automatic.pgp +46 2025-04-10 debian-archive-bookworm-security-automatic.gpg +8709 2025-04-10 debian-archive-bookworm-security-automatic.pgp +34 2025-04-10 debian-archive-bookworm-stable.gpg +280 2025-04-10 debian-archive-bookworm-stable.pgp +37 2025-04-10 debian-archive-bullseye-automatic.gpg +8700 2025-04-10 debian-archive-bullseye-automatic.pgp +46 2025-04-10 debian-archive-bullseye-security-automatic.gpg +8709 2025-04-10 debian-archive-bullseye-security-automatic.pgp +34 2025-04-10 debian-archive-bullseye-stable.gpg +2453 2025-04-10 debian-archive-bullseye-stable.pgp +26 2025-04-10 debian-archive-keyring.gpg +55918 2025-04-10 debian-archive-keyring.pgp +31 2025-04-10 debian-archive-removed-keys.gpg +72636 2025-04-10 debian-archive-removed-keys.pgp +35 2025-04-10 debian-archive-trixie-automatic.gpg +8698 2025-04-10 debian-archive-trixie-automatic.pgp +44 2025-04-10 debian-archive-trixie-security-automatic.gpg +8707 2025-04-10 debian-archive-trixie-security-automatic.pgp +32 2025-04-10 debian-archive-trixie-stable.gpg +962 2025-04-10 debian-archive-trixie-stable.pgp +2372 2025-07-09 proxmox-archive-keyring.gpg +113 2022-11-27 proxmox-offline-signing-key.pub +1187 2022-11-27 proxmox-release-bookworm.gpg +1185 2024-11-10 proxmox-release-trixie.gpg + +=== apt release files: Valid-Until === +deb.debian.org_debian_dists_trixie_InRelease | Date: Sat, 12 Sep 2026 07:55:41 UTC | Valid-Until: +deb.debian.org_debian_dists_trixie-updates_InRelease | Date: Wed, 11 Nov 2026 02:14:24 UTC | Valid-Until: Wed, 18 Nov 2026 02:14:24 UTC +download.proxmox.com_debian_pve_dists_trixie_InRelease | Date: Wed, 11 Nov 2026 01:41:17 +0000 | Valid-Until: +security.debian.org_debian-security_dists_trixie-security_InRelease | Date: Wed, 11 Nov 2026 00:23:32 UTC | Valid-Until: Wed, 18 Nov 2026 00:23:32 UTC + +=== templates and golden === +/var/lib/vz 129954319 2026-07-14 debian-13-standard_13.6-1_amd64.tar.zst + +=== agent files (names and dates only) === +/etc/felhom: +total +-rw-r--r-- root 7 2026-08-21 17:44 appliance-pairing-code +-rw-r--r-- root 3897 2026-11-07 23:27 config-bundle.json +-rw-r--r-- root 456 2026-10-04 16:08 crash-guard.conf +-rw-r--r-- root 131 2026-10-04 16:08 operator-signers +-rw-r--r-- root 61 2026-10-04 18:34 os-trust.json + +/etc/felhom-agent: +total +-rw------- felhom-agent 2454 2026-09-28 16:25 agent.json +-rw------- felhom-agent 2498 2026-09-24 15:33 agent.json.night-0925-off +-rw------- felhom-agent 2451 2026-09-24 21:56 agent.json.pre-a4-retention +-rw------- felhom-agent 2451 2026-09-24 21:57 agent.json.pre-d44 +-rw------- root 2451 2026-08-21 18:09 agent.json.pre-r672 + +/var/lib/felhom-agent: +total +-rw------- felhom-agent 593 2026-11-11 04:44 backup-success-state.json +-rw------- felhom-agent 290 2026-11-10 16:17 controller-swap-9201.json +-rw------- felhom-agent 126 2026-10-17 16:23 drive-intents.json +drwx------ felhom-agent 4096 2026-10-19 03:48 escrow-stage +drwx------ felhom-agent 4096 2026-08-21 17:59 felhom-sshd +-rw------- felhom-agent 301 2026-10-19 03:49 format-job.json +-rw------- felhom-agent 67 2026-10-17 17:08 guest-binds.json +-rw------- felhom-agent 241 2026-11-11 10:03 guest-disk-trim.json +drwx------ felhom-agent 4096 2026-09-13 22:16 guests +-rw------- felhom-agent 36314 2026-11-11 04:31 journal.log +-rw------- felhom-agent 305 2026-11-10 11:36 line-test.json +-rw-r--r-- felhom-agent 591 2026-08-21 17:59 local-api.crt +-rw------- felhom-agent 227 2026-08-21 17:59 local-api.key +-rw------- felhom-agent 82 2026-08-21 18:00 local-tokens.log +-rw------- felhom-agent 6120 2026-11-10 15:58 nonces.log +drwx------ felhom-agent 4096 2026-11-11 10:58 os +drwx------ felhom-agent 4096 2026-08-21 18:09 pbsdr +-rw------- felhom-agent 367 2026-11-11 04:31 restore-test-state.json +drwxr-x--- felhom-agent 4096 2026-11-10 15:59 selfupdate +-rw------- felhom-agent 32 2026-11-03 08:56 store-grant-seen.json +drwx------ felhom-agent 4096 2026-10-17 16:11 units +drwx------ felhom-agent 4096 2026-08-21 17:59 wg + +=== agent === +felhom-agent 0.173.0 + +=== wireguard (interface and latest handshake time only) === +wg-felhom peer f3d1ZI… 2026-11-11T10:05:07Z + +=== tailscale === +no tailscale + +=== ssh host keys / authorized keys (counts only) === +3 +/root/.ssh/authorized_keys lines=2 + +=== smart: power-on hours and wear (no serials) === +sda: 9 Power_On_Hours 0x0032 100 100 --- Old_age Always - 32293;177 Wear_Leveling_Count 0x0032 100 100 --- Old_age Always - 33; +nvme0n1: Available Spare: 100%;Percentage Used: 8%;Power On Hours: 30,963; +RC=0 diff --git a/documentation/audits/end-dates-list-2026-11-11/evidence/box/golden-gate.txt b/documentation/audits/end-dates-list-2026-11-11/evidence/box/golden-gate.txt new file mode 100644 index 00000000..f1dfe7c0 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/evidence/box/golden-gate.txt @@ -0,0 +1,10 @@ +# python3 scripts/golden_currency_gate.py, run on DooPlex 2026-11-11T10:27:34Z (reads the clone only) + newest released controller : 0.352.0 (## v0.352.0 (2026-11-10) — R-1371: one fill warning per filesystem — RELEASED (image `sha2) + newest golden baked : 0.350.0 (documentation/tests/golden-0.350.0-2026-11-03 [sha 082e2e040de5…]) + golden age : 8 day(s) (baked 2026-11-03; the limit is 7 — `09` §3 decision 250) + +GOLDEN CURRENCY GATE FAILED: controller v0.352.0 is released and the newest golden (0.350.0) is 8 day(s) old — more than 7 (decision 250: the golden is baked weekly; 2 release(s) behind). +A machine installed right now would receive v0.350.0 — the release is written, tested and pushed, and NOT delivered. +Fix: bake the weekly golden at the floor's version per documentation/runbooks/RUNBOOK-manual-build.md §4.1 (runbooks/nightly-rotation.md, Sunday) and put „vouch golden X" on the operator's morning list — the vouch is the operator's (a THREE-field change: golden_version + agent_version + min_agent). +If the cadence ruling covers this release, issue a DATED waiver at documentation/tests/golden-waiver.yml (RUNBOOK-manual-build.md §4.2) — never a bypass. +RC=1 diff --git a/documentation/audits/end-dates-list-2026-11-11/evidence/box/keyrings.txt b/documentation/audits/end-dates-list-2026-11-11/evidence/box/keyrings.txt new file mode 100644 index 00000000..5b3b8c74 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/evidence/box/keyrings.txt @@ -0,0 +1,798 @@ +=== demo-hp host: keyring files 26 +etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | uid Debian Archive Automatic Signing Key (12/bookworm) +etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (12/bookworm) +etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21 +etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | uid Debian Stable Release Key (12/bookworm) +etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | uid Debian Archive Automatic Signing Key (11/bullseye) +etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (11/bullseye) +etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11 +etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | uid Debian Stable Release Key (11/bullseye) +etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | uid Debian Archive Automatic Signing Key (13/trixie) +etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (13/trixie) +etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22 +etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | uid Debian Stable Release Key (13/trixie) +etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg | pub 1140AF8F639E0C39 algo=1 bits=4096 | created 2022-11-27 | expires 2032-11-24 +etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg | uid Proxmox Bookworm Release Key +etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg | pub A7BCD1420BFE778E algo=1 bits=4096 | created 2024-11-10 | expires 2034-11-10 +etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg | uid Proxmox Trixie Release Key +usr/share/keyrings/debian-archive-bookworm-automatic.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-automatic.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-bookworm-automatic.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-stable.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21 +usr/share/keyrings/debian-archive-bookworm-stable.pgp | uid Debian Stable Release Key (12/bookworm) +usr/share/keyrings/debian-archive-bullseye-automatic.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-automatic.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-bullseye-automatic.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-stable.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11 +usr/share/keyrings/debian-archive-bullseye-stable.pgp | uid Debian Stable Release Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (12/bookworm) +usr/share/keyrings/debian-archive-keyring.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-keyring.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-keyring.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-keyring.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-keyring.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (13/trixie) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FFA8EF91DB114E0 algo=1 bits=1024 | created 2004-01-15 | expires 2005-01-27 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2004) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub F1D53D8C4F368D5D algo=17 bits=1024 | created 2005-01-31 | expires 2006-01-31 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2005) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub E415B2B4B5F5BBED algo=17 bits=1024 | created 2005-04-24 | expires never +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian AMD64 Archive Key +usr/share/keyrings/debian-archive-removed-keys.pgp | sub B7A50B4134FC6FE5 algo=16 bits=2048 | created 2005-04-24 | expires never +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 010908312D230C5F algo=17 bits=1024 | created 2006-01-03 | expires 2007-02-07 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2006) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub A70DAF536070D3A1 algo=17 bits=1024 | created 2006-11-20 | expires 2009-07-01 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (4.0/etch) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub B5D0C804ADB11277 algo=17 bits=1024 | created 2006-09-17 | expires never +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Etch Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub EC61E0B0BBE55AB3 algo=17 bits=1024 | created 2007-03-31 | expires 2010-03-30 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (4.0/etch) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 0A3B614236CA98F3 algo=16 bits=2048 | created 2007-03-31 | expires 2010-03-30 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9AA38DCD55BE302B algo=1 bits=4096 | created 2009-01-27 | expires 2012-12-31 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (5.0/lenny) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4D270D06F42584E6 algo=17 bits=1024 | created 2008-04-06 | expires 2012-05-15 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Lenny Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub DFD993306D849617 algo=1 bits=2048 | created 2009-01-24 | expires 2013-01-23 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (5.0/lenny) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 64481591B98321F9 algo=1 bits=4096 | created 2010-08-07 | expires 2017-08-05 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Squeeze Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub AED4B06F473041FA algo=1 bits=4096 | created 2010-08-27 | expires 2018-03-05 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (6.0/squeeze) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 8B48AD6246925553 algo=1 bits=4096 | created 2012-04-27 | expires 2020-04-25 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (7.0/wheezy) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FB2A1C265FFB764 algo=1 bits=4096 | created 2012-05-08 | expires 2019-05-07 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Wheezy Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub CBF8D6FD518E17E1 algo=1 bits=4096 | created 2013-08-17 | expires 2021-08-15 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Jessie Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 7638D0442B90D010 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (8/jessie) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9D6D8F6BC857C906 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (8/jessie) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub EF0F382A1A7B6500 algo=1 bits=4096 | created 2017-05-20 | expires 2025-05-18 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (9/stretch) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub E0B11894F66AEC98 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (9/stretch) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 04EE7237B7D453EC algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub EDA0D2388AE22BA9 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (9/stretch) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub AA8E81B4331F7F50 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub DCC9EFBF77E11517 algo=1 bits=4096 | created 2019-02-05 | expires 2027-02-03 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (10/buster) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub DC30D7C23CBBABEE algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (10/buster) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 648ACFD622F3D138 algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4DFAB270CAA96DFA algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (10/buster) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 112695A0E562B32A algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-trixie-automatic.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-automatic.pgp | uid Debian Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-trixie-automatic.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-stable.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22 +usr/share/keyrings/debian-archive-trixie-stable.pgp | uid Debian Stable Release Key (13/trixie) +usr/share/keyrings/proxmox-archive-keyring.gpg | pub 1140AF8F639E0C39 algo=1 bits=4096 | created 2022-11-27 | expires 2032-11-24 +usr/share/keyrings/proxmox-archive-keyring.gpg | uid Proxmox Bookworm Release Key +usr/share/keyrings/proxmox-archive-keyring.gpg | pub A7BCD1420BFE778E algo=1 bits=4096 | created 2024-11-10 | expires 2034-11-10 +usr/share/keyrings/proxmox-archive-keyring.gpg | uid Proxmox Trixie Release Key +usr/share/keyrings/proxmox-release-bookworm.gpg | pub 1140AF8F639E0C39 algo=1 bits=4096 | created 2022-11-27 | expires 2032-11-24 +usr/share/keyrings/proxmox-release-bookworm.gpg | uid Proxmox Bookworm Release Key +usr/share/keyrings/proxmox-release-trixie.gpg | pub A7BCD1420BFE778E algo=1 bits=4096 | created 2024-11-10 | expires 2034-11-10 +usr/share/keyrings/proxmox-release-trixie.gpg | uid Proxmox Trixie Release Key +=== demo-hp guest: keyring files 21 +etc/apt/keyrings/docker.asc | pub 8D81803C0EBFCD88 algo=1 bits=4096 | created 2017-02-22 | expires never +etc/apt/keyrings/docker.asc | uid Docker Release (CE deb) +etc/apt/keyrings/docker.asc | sub 7EA0A9C3F273FCD8 algo=1 bits=4096 | created 2017-02-22 | expires never +etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | uid Debian Archive Automatic Signing Key (12/bookworm) +etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (12/bookworm) +etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21 +etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | uid Debian Stable Release Key (12/bookworm) +etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | uid Debian Archive Automatic Signing Key (11/bullseye) +etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (11/bullseye) +etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11 +etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | uid Debian Stable Release Key (11/bullseye) +etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | uid Debian Archive Automatic Signing Key (13/trixie) +etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (13/trixie) +etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22 +etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | uid Debian Stable Release Key (13/trixie) +usr/share/keyrings/debian-archive-bookworm-automatic.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-automatic.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-bookworm-automatic.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-stable.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21 +usr/share/keyrings/debian-archive-bookworm-stable.pgp | uid Debian Stable Release Key (12/bookworm) +usr/share/keyrings/debian-archive-bullseye-automatic.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-automatic.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-bullseye-automatic.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-stable.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11 +usr/share/keyrings/debian-archive-bullseye-stable.pgp | uid Debian Stable Release Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (12/bookworm) +usr/share/keyrings/debian-archive-keyring.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-keyring.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-keyring.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-keyring.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-keyring.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (13/trixie) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FFA8EF91DB114E0 algo=1 bits=1024 | created 2004-01-15 | expires 2005-01-27 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2004) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub F1D53D8C4F368D5D algo=17 bits=1024 | created 2005-01-31 | expires 2006-01-31 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2005) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub E415B2B4B5F5BBED algo=17 bits=1024 | created 2005-04-24 | expires never +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian AMD64 Archive Key +usr/share/keyrings/debian-archive-removed-keys.pgp | sub B7A50B4134FC6FE5 algo=16 bits=2048 | created 2005-04-24 | expires never +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 010908312D230C5F algo=17 bits=1024 | created 2006-01-03 | expires 2007-02-07 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2006) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub A70DAF536070D3A1 algo=17 bits=1024 | created 2006-11-20 | expires 2009-07-01 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (4.0/etch) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub B5D0C804ADB11277 algo=17 bits=1024 | created 2006-09-17 | expires never +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Etch Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub EC61E0B0BBE55AB3 algo=17 bits=1024 | created 2007-03-31 | expires 2010-03-30 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (4.0/etch) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 0A3B614236CA98F3 algo=16 bits=2048 | created 2007-03-31 | expires 2010-03-30 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9AA38DCD55BE302B algo=1 bits=4096 | created 2009-01-27 | expires 2012-12-31 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (5.0/lenny) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4D270D06F42584E6 algo=17 bits=1024 | created 2008-04-06 | expires 2012-05-15 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Lenny Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub DFD993306D849617 algo=1 bits=2048 | created 2009-01-24 | expires 2013-01-23 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (5.0/lenny) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 64481591B98321F9 algo=1 bits=4096 | created 2010-08-07 | expires 2017-08-05 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Squeeze Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub AED4B06F473041FA algo=1 bits=4096 | created 2010-08-27 | expires 2018-03-05 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (6.0/squeeze) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 8B48AD6246925553 algo=1 bits=4096 | created 2012-04-27 | expires 2020-04-25 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (7.0/wheezy) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FB2A1C265FFB764 algo=1 bits=4096 | created 2012-05-08 | expires 2019-05-07 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Wheezy Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub CBF8D6FD518E17E1 algo=1 bits=4096 | created 2013-08-17 | expires 2021-08-15 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Jessie Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 7638D0442B90D010 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (8/jessie) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9D6D8F6BC857C906 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (8/jessie) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub EF0F382A1A7B6500 algo=1 bits=4096 | created 2017-05-20 | expires 2025-05-18 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (9/stretch) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub E0B11894F66AEC98 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (9/stretch) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 04EE7237B7D453EC algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub EDA0D2388AE22BA9 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (9/stretch) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub AA8E81B4331F7F50 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub DCC9EFBF77E11517 algo=1 bits=4096 | created 2019-02-05 | expires 2027-02-03 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (10/buster) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub DC30D7C23CBBABEE algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (10/buster) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 648ACFD622F3D138 algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4DFAB270CAA96DFA algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (10/buster) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 112695A0E562B32A algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-trixie-automatic.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-automatic.pgp | uid Debian Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-trixie-automatic.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-stable.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22 +usr/share/keyrings/debian-archive-trixie-stable.pgp | uid Debian Stable Release Key (13/trixie) +=== tester-1 host: keyring files 26 +etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | uid Debian Archive Automatic Signing Key (12/bookworm) +etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (12/bookworm) +etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21 +etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | uid Debian Stable Release Key (12/bookworm) +etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | uid Debian Archive Automatic Signing Key (11/bullseye) +etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (11/bullseye) +etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11 +etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | uid Debian Stable Release Key (11/bullseye) +etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | uid Debian Archive Automatic Signing Key (13/trixie) +etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (13/trixie) +etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22 +etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | uid Debian Stable Release Key (13/trixie) +etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg | pub 1140AF8F639E0C39 algo=1 bits=4096 | created 2022-11-27 | expires 2032-11-24 +etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg | uid Proxmox Bookworm Release Key +etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg | pub A7BCD1420BFE778E algo=1 bits=4096 | created 2024-11-10 | expires 2034-11-10 +etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg | uid Proxmox Trixie Release Key +usr/share/keyrings/debian-archive-bookworm-automatic.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-automatic.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-bookworm-automatic.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-stable.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21 +usr/share/keyrings/debian-archive-bookworm-stable.pgp | uid Debian Stable Release Key (12/bookworm) +usr/share/keyrings/debian-archive-bullseye-automatic.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-automatic.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-bullseye-automatic.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-stable.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11 +usr/share/keyrings/debian-archive-bullseye-stable.pgp | uid Debian Stable Release Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (12/bookworm) +usr/share/keyrings/debian-archive-keyring.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-keyring.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-keyring.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-keyring.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-keyring.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (13/trixie) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FFA8EF91DB114E0 algo=1 bits=1024 | created 2004-01-15 | expires 2005-01-27 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2004) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub F1D53D8C4F368D5D algo=17 bits=1024 | created 2005-01-31 | expires 2006-01-31 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2005) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub E415B2B4B5F5BBED algo=17 bits=1024 | created 2005-04-24 | expires never +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian AMD64 Archive Key +usr/share/keyrings/debian-archive-removed-keys.pgp | sub B7A50B4134FC6FE5 algo=16 bits=2048 | created 2005-04-24 | expires never +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 010908312D230C5F algo=17 bits=1024 | created 2006-01-03 | expires 2007-02-07 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2006) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub A70DAF536070D3A1 algo=17 bits=1024 | created 2006-11-20 | expires 2009-07-01 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (4.0/etch) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub B5D0C804ADB11277 algo=17 bits=1024 | created 2006-09-17 | expires never +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Etch Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub EC61E0B0BBE55AB3 algo=17 bits=1024 | created 2007-03-31 | expires 2010-03-30 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (4.0/etch) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 0A3B614236CA98F3 algo=16 bits=2048 | created 2007-03-31 | expires 2010-03-30 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9AA38DCD55BE302B algo=1 bits=4096 | created 2009-01-27 | expires 2012-12-31 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (5.0/lenny) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4D270D06F42584E6 algo=17 bits=1024 | created 2008-04-06 | expires 2012-05-15 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Lenny Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub DFD993306D849617 algo=1 bits=2048 | created 2009-01-24 | expires 2013-01-23 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (5.0/lenny) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 64481591B98321F9 algo=1 bits=4096 | created 2010-08-07 | expires 2017-08-05 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Squeeze Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub AED4B06F473041FA algo=1 bits=4096 | created 2010-08-27 | expires 2018-03-05 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (6.0/squeeze) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 8B48AD6246925553 algo=1 bits=4096 | created 2012-04-27 | expires 2020-04-25 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (7.0/wheezy) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FB2A1C265FFB764 algo=1 bits=4096 | created 2012-05-08 | expires 2019-05-07 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Wheezy Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub CBF8D6FD518E17E1 algo=1 bits=4096 | created 2013-08-17 | expires 2021-08-15 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Jessie Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 7638D0442B90D010 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (8/jessie) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9D6D8F6BC857C906 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (8/jessie) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub EF0F382A1A7B6500 algo=1 bits=4096 | created 2017-05-20 | expires 2025-05-18 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (9/stretch) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub E0B11894F66AEC98 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (9/stretch) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 04EE7237B7D453EC algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub EDA0D2388AE22BA9 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (9/stretch) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub AA8E81B4331F7F50 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub DCC9EFBF77E11517 algo=1 bits=4096 | created 2019-02-05 | expires 2027-02-03 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (10/buster) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub DC30D7C23CBBABEE algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (10/buster) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 648ACFD622F3D138 algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4DFAB270CAA96DFA algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (10/buster) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 112695A0E562B32A algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-trixie-automatic.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-automatic.pgp | uid Debian Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-trixie-automatic.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-stable.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22 +usr/share/keyrings/debian-archive-trixie-stable.pgp | uid Debian Stable Release Key (13/trixie) +usr/share/keyrings/proxmox-archive-keyring.gpg | pub 1140AF8F639E0C39 algo=1 bits=4096 | created 2022-11-27 | expires 2032-11-24 +usr/share/keyrings/proxmox-archive-keyring.gpg | uid Proxmox Bookworm Release Key +usr/share/keyrings/proxmox-archive-keyring.gpg | pub A7BCD1420BFE778E algo=1 bits=4096 | created 2024-11-10 | expires 2034-11-10 +usr/share/keyrings/proxmox-archive-keyring.gpg | uid Proxmox Trixie Release Key +usr/share/keyrings/proxmox-release-bookworm.gpg | pub 1140AF8F639E0C39 algo=1 bits=4096 | created 2022-11-27 | expires 2032-11-24 +usr/share/keyrings/proxmox-release-bookworm.gpg | uid Proxmox Bookworm Release Key +usr/share/keyrings/proxmox-release-trixie.gpg | pub A7BCD1420BFE778E algo=1 bits=4096 | created 2024-11-10 | expires 2034-11-10 +usr/share/keyrings/proxmox-release-trixie.gpg | uid Proxmox Trixie Release Key +=== tester-1 guest: keyring files 21 +etc/apt/keyrings/docker.asc | pub 8D81803C0EBFCD88 algo=1 bits=4096 | created 2017-02-22 | expires never +etc/apt/keyrings/docker.asc | uid Docker Release (CE deb) +etc/apt/keyrings/docker.asc | sub 7EA0A9C3F273FCD8 algo=1 bits=4096 | created 2017-02-22 | expires never +etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | uid Debian Archive Automatic Signing Key (12/bookworm) +etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (12/bookworm) +etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21 +etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | uid Debian Stable Release Key (12/bookworm) +etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | uid Debian Archive Automatic Signing Key (11/bullseye) +etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (11/bullseye) +etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11 +etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | uid Debian Stable Release Key (11/bullseye) +etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | uid Debian Archive Automatic Signing Key (13/trixie) +etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (13/trixie) +etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22 +etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | uid Debian Stable Release Key (13/trixie) +usr/share/keyrings/debian-archive-bookworm-automatic.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-automatic.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-bookworm-automatic.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-stable.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21 +usr/share/keyrings/debian-archive-bookworm-stable.pgp | uid Debian Stable Release Key (12/bookworm) +usr/share/keyrings/debian-archive-bullseye-automatic.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-automatic.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-bullseye-automatic.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-stable.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11 +usr/share/keyrings/debian-archive-bullseye-stable.pgp | uid Debian Stable Release Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (12/bookworm) +usr/share/keyrings/debian-archive-keyring.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-keyring.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-keyring.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-keyring.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-keyring.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (13/trixie) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FFA8EF91DB114E0 algo=1 bits=1024 | created 2004-01-15 | expires 2005-01-27 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2004) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub F1D53D8C4F368D5D algo=17 bits=1024 | created 2005-01-31 | expires 2006-01-31 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2005) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub E415B2B4B5F5BBED algo=17 bits=1024 | created 2005-04-24 | expires never +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian AMD64 Archive Key +usr/share/keyrings/debian-archive-removed-keys.pgp | sub B7A50B4134FC6FE5 algo=16 bits=2048 | created 2005-04-24 | expires never +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 010908312D230C5F algo=17 bits=1024 | created 2006-01-03 | expires 2007-02-07 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2006) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub A70DAF536070D3A1 algo=17 bits=1024 | created 2006-11-20 | expires 2009-07-01 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (4.0/etch) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub B5D0C804ADB11277 algo=17 bits=1024 | created 2006-09-17 | expires never +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Etch Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub EC61E0B0BBE55AB3 algo=17 bits=1024 | created 2007-03-31 | expires 2010-03-30 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (4.0/etch) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 0A3B614236CA98F3 algo=16 bits=2048 | created 2007-03-31 | expires 2010-03-30 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9AA38DCD55BE302B algo=1 bits=4096 | created 2009-01-27 | expires 2012-12-31 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (5.0/lenny) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4D270D06F42584E6 algo=17 bits=1024 | created 2008-04-06 | expires 2012-05-15 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Lenny Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub DFD993306D849617 algo=1 bits=2048 | created 2009-01-24 | expires 2013-01-23 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (5.0/lenny) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 64481591B98321F9 algo=1 bits=4096 | created 2010-08-07 | expires 2017-08-05 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Squeeze Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub AED4B06F473041FA algo=1 bits=4096 | created 2010-08-27 | expires 2018-03-05 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (6.0/squeeze) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 8B48AD6246925553 algo=1 bits=4096 | created 2012-04-27 | expires 2020-04-25 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (7.0/wheezy) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FB2A1C265FFB764 algo=1 bits=4096 | created 2012-05-08 | expires 2019-05-07 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Wheezy Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub CBF8D6FD518E17E1 algo=1 bits=4096 | created 2013-08-17 | expires 2021-08-15 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Jessie Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 7638D0442B90D010 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (8/jessie) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9D6D8F6BC857C906 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (8/jessie) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub EF0F382A1A7B6500 algo=1 bits=4096 | created 2017-05-20 | expires 2025-05-18 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (9/stretch) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub E0B11894F66AEC98 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (9/stretch) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 04EE7237B7D453EC algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub EDA0D2388AE22BA9 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (9/stretch) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub AA8E81B4331F7F50 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub DCC9EFBF77E11517 algo=1 bits=4096 | created 2019-02-05 | expires 2027-02-03 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (10/buster) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub DC30D7C23CBBABEE algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (10/buster) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 648ACFD622F3D138 algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4DFAB270CAA96DFA algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (10/buster) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 112695A0E562B32A algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-trixie-automatic.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-automatic.pgp | uid Debian Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-trixie-automatic.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-stable.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22 +usr/share/keyrings/debian-archive-trixie-stable.pgp | uid Debian Stable Release Key (13/trixie) +=== demo-felhom host: keyring files 27 +etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | uid Debian Archive Automatic Signing Key (12/bookworm) +etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (12/bookworm) +etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21 +etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | uid Debian Stable Release Key (12/bookworm) +etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | uid Debian Archive Automatic Signing Key (11/bullseye) +etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (11/bullseye) +etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11 +etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | uid Debian Stable Release Key (11/bullseye) +etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | uid Debian Archive Automatic Signing Key (13/trixie) +etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (13/trixie) +etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22 +etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | uid Debian Stable Release Key (13/trixie) +etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg | pub 1140AF8F639E0C39 algo=1 bits=4096 | created 2022-11-27 | expires 2032-11-24 +etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg | uid Proxmox Bookworm Release Key +etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg | pub A7BCD1420BFE778E algo=1 bits=4096 | created 2024-11-10 | expires 2034-11-10 +etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg | uid Proxmox Trixie Release Key +usr/share/keyrings/debian-archive-bookworm-automatic.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-automatic.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-bookworm-automatic.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-stable.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21 +usr/share/keyrings/debian-archive-bookworm-stable.pgp | uid Debian Stable Release Key (12/bookworm) +usr/share/keyrings/debian-archive-bullseye-automatic.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-automatic.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-bullseye-automatic.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-stable.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11 +usr/share/keyrings/debian-archive-bullseye-stable.pgp | uid Debian Stable Release Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (12/bookworm) +usr/share/keyrings/debian-archive-keyring.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-keyring.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-keyring.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-keyring.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-keyring.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (13/trixie) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FFA8EF91DB114E0 algo=1 bits=1024 | created 2004-01-15 | expires 2005-01-27 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2004) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub F1D53D8C4F368D5D algo=17 bits=1024 | created 2005-01-31 | expires 2006-01-31 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2005) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub E415B2B4B5F5BBED algo=17 bits=1024 | created 2005-04-24 | expires never +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian AMD64 Archive Key +usr/share/keyrings/debian-archive-removed-keys.pgp | sub B7A50B4134FC6FE5 algo=16 bits=2048 | created 2005-04-24 | expires never +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 010908312D230C5F algo=17 bits=1024 | created 2006-01-03 | expires 2007-02-07 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2006) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub A70DAF536070D3A1 algo=17 bits=1024 | created 2006-11-20 | expires 2009-07-01 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (4.0/etch) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub B5D0C804ADB11277 algo=17 bits=1024 | created 2006-09-17 | expires never +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Etch Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub EC61E0B0BBE55AB3 algo=17 bits=1024 | created 2007-03-31 | expires 2010-03-30 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (4.0/etch) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 0A3B614236CA98F3 algo=16 bits=2048 | created 2007-03-31 | expires 2010-03-30 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9AA38DCD55BE302B algo=1 bits=4096 | created 2009-01-27 | expires 2012-12-31 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (5.0/lenny) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4D270D06F42584E6 algo=17 bits=1024 | created 2008-04-06 | expires 2012-05-15 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Lenny Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub DFD993306D849617 algo=1 bits=2048 | created 2009-01-24 | expires 2013-01-23 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (5.0/lenny) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 64481591B98321F9 algo=1 bits=4096 | created 2010-08-07 | expires 2017-08-05 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Squeeze Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub AED4B06F473041FA algo=1 bits=4096 | created 2010-08-27 | expires 2018-03-05 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (6.0/squeeze) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 8B48AD6246925553 algo=1 bits=4096 | created 2012-04-27 | expires 2020-04-25 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (7.0/wheezy) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FB2A1C265FFB764 algo=1 bits=4096 | created 2012-05-08 | expires 2019-05-07 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Wheezy Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub CBF8D6FD518E17E1 algo=1 bits=4096 | created 2013-08-17 | expires 2021-08-15 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Jessie Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 7638D0442B90D010 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (8/jessie) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9D6D8F6BC857C906 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (8/jessie) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub EF0F382A1A7B6500 algo=1 bits=4096 | created 2017-05-20 | expires 2025-05-18 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (9/stretch) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub E0B11894F66AEC98 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (9/stretch) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 04EE7237B7D453EC algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub EDA0D2388AE22BA9 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (9/stretch) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub AA8E81B4331F7F50 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub DCC9EFBF77E11517 algo=1 bits=4096 | created 2019-02-05 | expires 2027-02-03 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (10/buster) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub DC30D7C23CBBABEE algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (10/buster) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 648ACFD622F3D138 algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4DFAB270CAA96DFA algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (10/buster) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 112695A0E562B32A algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-trixie-automatic.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-automatic.pgp | uid Debian Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-trixie-automatic.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-stable.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22 +usr/share/keyrings/debian-archive-trixie-stable.pgp | uid Debian Stable Release Key (13/trixie) +usr/share/keyrings/proxmox-archive-keyring.gpg | pub 1140AF8F639E0C39 algo=1 bits=4096 | created 2022-11-27 | expires 2032-11-24 +usr/share/keyrings/proxmox-archive-keyring.gpg | uid Proxmox Bookworm Release Key +usr/share/keyrings/proxmox-archive-keyring.gpg | pub A7BCD1420BFE778E algo=1 bits=4096 | created 2024-11-10 | expires 2034-11-10 +usr/share/keyrings/proxmox-archive-keyring.gpg | uid Proxmox Trixie Release Key +usr/share/keyrings/proxmox-release-bookworm.gpg | pub 1140AF8F639E0C39 algo=1 bits=4096 | created 2022-11-27 | expires 2032-11-24 +usr/share/keyrings/proxmox-release-bookworm.gpg | uid Proxmox Bookworm Release Key +usr/share/keyrings/proxmox-release-trixie.gpg | pub A7BCD1420BFE778E algo=1 bits=4096 | created 2024-11-10 | expires 2034-11-10 +usr/share/keyrings/proxmox-release-trixie.gpg | uid Proxmox Trixie Release Key +usr/share/keyrings/tailscale-archive-keyring.gpg | pub 458CA832957F5868 algo=1 bits=4096 | created 2020-02-25 | expires never +usr/share/keyrings/tailscale-archive-keyring.gpg | uid Tailscale Inc. (Package repository signing key) +usr/share/keyrings/tailscale-archive-keyring.gpg | sub B1547A3DDAAF03C6 algo=1 bits=4096 | created 2020-02-25 | expires never +=== demo-felhom guest: keyring files 21 +etc/apt/keyrings/docker.asc | pub 8D81803C0EBFCD88 algo=1 bits=4096 | created 2017-02-22 | expires never +etc/apt/keyrings/docker.asc | uid Docker Release (CE deb) +etc/apt/keyrings/docker.asc | sub 7EA0A9C3F273FCD8 algo=1 bits=4096 | created 2017-02-22 | expires never +etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | uid Debian Archive Automatic Signing Key (12/bookworm) +etc/apt/trusted.gpg.d/debian-archive-bookworm-automatic.asc | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (12/bookworm) +etc/apt/trusted.gpg.d/debian-archive-bookworm-security-automatic.asc | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21 +etc/apt/trusted.gpg.d/debian-archive-bookworm-stable.asc | uid Debian Stable Release Key (12/bookworm) +etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | uid Debian Archive Automatic Signing Key (11/bullseye) +etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.asc | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (11/bullseye) +etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.asc | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11 +etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.asc | uid Debian Stable Release Key (11/bullseye) +etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | uid Debian Archive Automatic Signing Key (13/trixie) +etc/apt/trusted.gpg.d/debian-archive-trixie-automatic.asc | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | uid Debian Security Archive Automatic Signing Key (13/trixie) +etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22 +etc/apt/trusted.gpg.d/debian-archive-trixie-stable.asc | uid Debian Stable Release Key (13/trixie) +usr/share/keyrings/debian-archive-bookworm-automatic.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-automatic.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-bookworm-automatic.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-bookworm-security-automatic.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-bookworm-stable.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21 +usr/share/keyrings/debian-archive-bookworm-stable.pgp | uid Debian Stable Release Key (12/bookworm) +usr/share/keyrings/debian-archive-bullseye-automatic.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-automatic.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-bullseye-automatic.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-bullseye-security-automatic.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-bullseye-stable.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11 +usr/share/keyrings/debian-archive-bullseye-stable.pgp | uid Debian Stable Release Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | pub 73A4F27B8DD47936 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | sub 0E98404D386FA1D9 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | pub A48449044AAD5C5D algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | sub 54404762BBB6E853 algo=1 bits=4096 | created 2021-01-17 | expires 2029-01-15 +usr/share/keyrings/debian-archive-keyring.pgp | pub 605C66F00D6C9793 algo=1 bits=4096 | created 2021-02-13 | expires 2029-02-11 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (11/bullseye) +usr/share/keyrings/debian-archive-keyring.pgp | pub F8D2585B8783D481 algo=22 bits=255 | created 2023-01-23 | expires 2031-01-21 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (12/bookworm) +usr/share/keyrings/debian-archive-keyring.pgp | pub B7C5D7D6350947F8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-keyring.pgp | sub 6ED0E7B82643E131 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | pub 254CF3B5AEC0A8F0 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (12/bookworm) +usr/share/keyrings/debian-archive-keyring.pgp | sub BDE6D2B9216EC7A8 algo=1 bits=4096 | created 2023-01-21 | expires 2031-01-19 +usr/share/keyrings/debian-archive-keyring.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-keyring.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-keyring.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-keyring.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22 +usr/share/keyrings/debian-archive-keyring.pgp | uid Debian Stable Release Key (13/trixie) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FFA8EF91DB114E0 algo=1 bits=1024 | created 2004-01-15 | expires 2005-01-27 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2004) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub F1D53D8C4F368D5D algo=17 bits=1024 | created 2005-01-31 | expires 2006-01-31 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2005) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub E415B2B4B5F5BBED algo=17 bits=1024 | created 2005-04-24 | expires never +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian AMD64 Archive Key +usr/share/keyrings/debian-archive-removed-keys.pgp | sub B7A50B4134FC6FE5 algo=16 bits=2048 | created 2005-04-24 | expires never +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 010908312D230C5F algo=17 bits=1024 | created 2006-01-03 | expires 2007-02-07 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (2006) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub A70DAF536070D3A1 algo=17 bits=1024 | created 2006-11-20 | expires 2009-07-01 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (4.0/etch) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub B5D0C804ADB11277 algo=17 bits=1024 | created 2006-09-17 | expires never +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Etch Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub EC61E0B0BBE55AB3 algo=17 bits=1024 | created 2007-03-31 | expires 2010-03-30 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (4.0/etch) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 0A3B614236CA98F3 algo=16 bits=2048 | created 2007-03-31 | expires 2010-03-30 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9AA38DCD55BE302B algo=1 bits=4096 | created 2009-01-27 | expires 2012-12-31 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (5.0/lenny) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4D270D06F42584E6 algo=17 bits=1024 | created 2008-04-06 | expires 2012-05-15 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Lenny Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub DFD993306D849617 algo=1 bits=2048 | created 2009-01-24 | expires 2013-01-23 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian-Volatile Archive Automatic Signing Key (5.0/lenny) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 64481591B98321F9 algo=1 bits=4096 | created 2010-08-07 | expires 2017-08-05 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Squeeze Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub AED4B06F473041FA algo=1 bits=4096 | created 2010-08-27 | expires 2018-03-05 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (6.0/squeeze) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 8B48AD6246925553 algo=1 bits=4096 | created 2012-04-27 | expires 2020-04-25 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (7.0/wheezy) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 6FB2A1C265FFB764 algo=1 bits=4096 | created 2012-05-08 | expires 2019-05-07 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Wheezy Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub CBF8D6FD518E17E1 algo=1 bits=4096 | created 2013-08-17 | expires 2021-08-15 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Jessie Stable Release Key +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 7638D0442B90D010 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (8/jessie) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 9D6D8F6BC857C906 algo=1 bits=4096 | created 2014-11-21 | expires 2022-11-19 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (8/jessie) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub EF0F382A1A7B6500 algo=1 bits=4096 | created 2017-05-20 | expires 2025-05-18 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (9/stretch) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub E0B11894F66AEC98 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (9/stretch) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 04EE7237B7D453EC algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub EDA0D2388AE22BA9 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (9/stretch) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub AA8E81B4331F7F50 algo=1 bits=4096 | created 2017-05-22 | expires 2025-05-20 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub DCC9EFBF77E11517 algo=1 bits=4096 | created 2019-02-05 | expires 2027-02-03 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Stable Release Key (10/buster) +usr/share/keyrings/debian-archive-removed-keys.pgp | pub DC30D7C23CBBABEE algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Archive Automatic Signing Key (10/buster) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 648ACFD622F3D138 algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-removed-keys.pgp | pub 4DFAB270CAA96DFA algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-removed-keys.pgp | uid Debian Security Archive Automatic Signing Key (10/buster) +usr/share/keyrings/debian-archive-removed-keys.pgp | sub 112695A0E562B32A algo=1 bits=4096 | created 2019-04-14 | expires 2027-04-12 +usr/share/keyrings/debian-archive-trixie-automatic.pgp | pub 225629DF75B188BD algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-automatic.pgp | uid Debian Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-trixie-automatic.pgp | sub 78DBA3BC47EF2265 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | pub 9904613D4CCE68C6 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | uid Debian Security Archive Automatic Signing Key (13/trixie) +usr/share/keyrings/debian-archive-trixie-security-automatic.pgp | sub 8E9F831205B4BA95 algo=1 bits=4096 | created 2025-03-30 | expires 2035-03-28 +usr/share/keyrings/debian-archive-trixie-stable.pgp | pub 762F67A0B2C39DE4 algo=22 bits=255 | created 2025-03-24 | expires 2033-03-22 +usr/share/keyrings/debian-archive-trixie-stable.pgp | uid Debian Stable Release Key (13/trixie) diff --git a/documentation/audits/end-dates-list-2026-11-11/evidence/box/pin.txt b/documentation/audits/end-dates-list-2026-11-11/evidence/box/pin.txt new file mode 100644 index 00000000..a674ba3e --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/evidence/box/pin.txt @@ -0,0 +1,16 @@ +taken 2026-11-11T10:17:10Z from DooPlex +== demo-hp +pve-ssl notAfter=Aug 19 15:44:42 2028 GMT +proxmox.tls.fingerprint set equals pve-ssl leaf: True +[{"expire":0,"privsep":1,"tokenid":"agent"}] +inactive +== root@192.168.0.154 +pve-ssl notAfter=Oct 2 19:38:05 2028 GMT +proxmox.tls.fingerprint set equals pve-ssl leaf: True +[{"expire":0,"privsep":1,"tokenid":"agent"}] +inactive +== felhom-pve +pve-ssl notAfter=Jul 16 16:17:07 2028 GMT +proxmox.tls.fingerprint set equals pve-ssl leaf: True +[{"expire":0,"privsep":1,"tokenid":"agent"}] +active diff --git a/documentation/audits/end-dates-list-2026-11-11/evidence/box/public-reads.txt b/documentation/audits/end-dates-list-2026-11-11/evidence/box/public-reads.txt new file mode 100644 index 00000000..2eb990c6 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/evidence/box/public-reads.txt @@ -0,0 +1,29 @@ +taken 2026-11-11T10:06:59Z from DooPlex + +=== certificates served by public names === +felhom.eu [resolves here to 188.114.96.3] -> subject=CN=felhom.eu issuer=C=US, O=Google Trust Services, CN=WE1 notBefore=Oct 30 09:04:46 2026 GMT notAfter=Jan 28 10:04:43 2027 GMT sha256 Fingerprint=90:5A:85:D7:9E:45:7B:62:CF:13:C3:88:59:3B:E7:DF:D1:45:72:20:80:33:A6:D2:5F:92:42:12:46:77:8F:FD +www.felhom.eu [resolves here to 188.114.97.3] -> subject=CN=www.felhom.eu issuer=C=US, O=Google Trust Services, CN=WE1 notBefore=Oct 30 09:04:46 2026 GMT notAfter=Jan 28 10:04:45 2027 GMT sha256 Fingerprint=8F:A4:81:B4:E6:4C:33:E4:26:33:BC:8D:73:EC:37:12:D4:43:DC:3D:A0:72:AC:12:9C:C2:14:1F:5F:41:F1:53 +hub.felhom.eu [resolves here to 37.191.56.193] -> subject=CN=hub.felhom.eu issuer=C=US, O=Let's Encrypt, CN=YR2 notBefore=Oct 14 07:36:58 2026 GMT notAfter=Jan 12 07:36:57 2027 GMT sha256 Fingerprint=05:C2:F0:F2:56:D4:5D:33:0B:A1:95:69:1D:5A:7E:E3:2E:76:37:35:2C:E7:AF:5C:72:0B:3A:06:DC:45:32:53 +iso.felhom.eu [resolves here to 188.114.97.3] -> subject=CN=b7d41af6.sni.cloudflaressl.com issuer=C=US, O=Google Trust Services, CN=WE1 notBefore=Sep 28 08:12:03 2026 GMT notAfter=Dec 27 09:11:59 2026 GMT sha256 Fingerprint=8C:D9:5A:7A:55:BC:F3:2A:D8:1B:52:2D:A4:C0:20:0A:B9:6A:D7:9B:62:D1:AE:CF:2A:7E:D4:93:45:1A:49:1A +files.felhom.eu [resolves here to 37.191.56.193] -> subject=CN=files.felhom.eu issuer=C=US, O=Let's Encrypt, CN=YR1 notBefore=Sep 24 04:36:30 2026 GMT notAfter=Dec 23 04:36:29 2026 GMT sha256 Fingerprint=04:5A:A5:37:38:FB:BE:61:BE:9B:5C:3B:32:78:D2:90:B0:28:D0:11:B8:E6:D1:D2:E2:68:21:9D:0F:D6:24:EA +gitea.dooplex.hu [resolves here to 37.191.56.193] -> subject=CN=gitea.dooplex.hu issuer=C=US, O=Let's Encrypt, CN=YR2 notBefore=Oct 5 03:52:51 2026 GMT notAfter=Jan 3 03:52:50 2027 GMT sha256 Fingerprint=00:55:A8:AB:39:49:D6:51:7F:8B:8C:37:95:6F:6B:2B:83:8B:3A:06:44:96:EE:42:1E:9C:31:B9:10:CB:7B:B0 +felhom.enkisfelhom.hu [resolves here to 188.114.96.3] -> subject=CN=enkisfelhom.hu issuer=C=US, O=Let's Encrypt, CN=YE2 notBefore=Oct 22 02:42:30 2026 GMT notAfter=Jan 20 02:42:29 2027 GMT sha256 Fingerprint=69:DC:B3:59:83:C6:96:2A:2A:5E:D0:75:D5:34:F1:11:CF:6B:F1:3E:90:24:32:D7:80:E2:6D:83:18:92:91:96 +felhom.enkicsifelhom.hu [resolves here to 104.21.22.132] -> subject=CN=enkicsifelhom.hu issuer=C=US, O=Let's Encrypt, CN=YE2 notBefore=Oct 21 13:19:02 2026 GMT notAfter=Jan 19 13:19:01 2027 GMT sha256 Fingerprint=92:6F:C1:DD:25:6C:0A:66:ED:1B:F3:23:5A:10:58:D5:66:6C:6D:C4:B3:1B:58:C4:1F:1D:B8:28:97:5D:E6:AB +felhom.demo-felhom.eu [resolves here to 188.114.97.3] -> subject=CN=demo-felhom.eu issuer=C=US, O=Google Trust Services, CN=WE1 notBefore=Oct 4 06:07:02 2026 GMT notAfter=Jan 2 07:05:44 2027 GMT sha256 Fingerprint=12:78:83:48:BC:B7:9E:7C:06:13:19:25:24:2B:6E:40:1D:24:CF:F7:59:0F:2D:BB:22:D2:9E:AA:D7:C4:F3:A5 + +=== public registry records (domains) === +--- felhom.eu +no whois tool on DooPlex +rdap: no JSON answer +--- demo-felhom.eu +no whois tool on DooPlex +rdap: no JSON answer +--- enkisfelhom.hu +no whois tool on DooPlex +rdap: no JSON answer +--- enkicsifelhom.hu +no whois tool on DooPlex +rdap: no JSON answer +--- dooplex.hu +no whois tool on DooPlex +rdap: no JSON answer diff --git a/documentation/audits/end-dates-list-2026-11-11/evidence/box/pveupdate-excerpt.txt b/documentation/audits/end-dates-list-2026-11-11/evidence/box/pveupdate-excerpt.txt new file mode 100644 index 00000000..496487ab --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/evidence/box/pveupdate-excerpt.txt @@ -0,0 +1,73 @@ +# /usr/bin/pveupdate lines 100-170 on demo-hp, taken 2026-11-11T10:24:12Z (read-only) + ); + } + } +}; +syslog('err', "Renewing ACME certificate failed: $@") if $@; + +eval { PVE::Cluster::Setup::renew_pveca_cert() }; +syslog('err', "Renewing cluster CA certificate failed: $@") if $@; + +eval { + my $certpath = PVE::CertHelpers::default_cert_path_prefix($nodename) . ".pem"; + my $capath = "/etc/pve/pve-root-ca.pem"; + + my $check_issued_by_ca = sub { + # get CA info + my $cainfo = PVE::Certificate::get_certificate_info($capath); + + # get cert and check issuer and chain metadata + my $certinfo = PVE::Certificate::get_certificate_info($certpath); + if ($certinfo->{issuer} ne $cainfo->{subject}) { + die "SSL certificate ($certpath) is not issued by root CA ($capath)!\n"; + } + + # check if cert is really signed by the ca + # TODO: replace by low level ssleay interface if version 1.86 is available + my $cmd = [ + '/usr/bin/openssl', 'verify', '-no_check_time', '-CAfile', $capath, '--', $certpath, + ]; + PVE::Tools::run_command($cmd); + }; + + my $renew = sub { + my ($msg, $replaced_ca) = @_; + + # one from a replaced cluster CA fails these checks, but is still ours to replace + $check_issued_by_ca->() if !$replaced_ca; + + print "PVE certificate $msg\n"; + # create new certificate + my $ip = PVE::Cluster::remote_node_ip($nodename); + PVE::Cluster::Setup::gen_pve_ssl_cert(1, $nodename, $ip); + my $expiry = localtime(PVE::Certificate::get_expiration_as_epoch($certpath)); + print "Renewed PVE certificate is valid until $expiry\n"; + + print "Restarting pveproxy after renewing certificate\n"; + PVE::Tools::run_command(['systemctl', 'reload-or-restart', 'pveproxy']); + }; + + my $status = eval { PVE::Cluster::Setup::pve_ssl_cert_status($nodename) }; + syslog('err', "Checking SSL certificate failed: $@") if $@; + + if ($status && $status->{'issued-by'} eq 'replaced-ca') { + $renew->("was issued by a replaced cluster CA, renewing...", 1); + } elsif (PVE::Certificate::check_expiry($certpath)) { + # already expired + $renew->("expired, renewing..."); + } elsif (PVE::Certificate::check_expiry($certpath, time() + 14 * 24 * 60 * 60)) { + # expires in next 2 weeks + $renew->("expires soon, renewing..."); + } elsif (!PVE::Certificate::check_expiry($certpath, time() + 2 * 365 * 24 * 60 * 60)) { + # expires in more than 2 years + $renew->( + "expires in more than 2 years, renewing to reduce certificate life-span for client compatibility..." + ); + } +}; +syslog('err', "Checking/Renewing SSL certificate failed: $@") if $@; + +sub cleanup_tasks { + + my $taskdir = "/var/log/pve/tasks"; +RC=0 diff --git a/documentation/audits/end-dates-list-2026-11-11/evidence/box/tester-1-guest-dates.txt b/documentation/audits/end-dates-list-2026-11-11/evidence/box/tester-1-guest-dates.txt new file mode 100644 index 00000000..c2e9d65c --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/evidence/box/tester-1-guest-dates.txt @@ -0,0 +1,126 @@ +# guest-dates tester-1 taken 2026-11-11T10:06:05Z from DooPlex + +=== now === +2026-11-11T10:06:06Z + +=== layout === +stacks +actualbudget adventurelog audiobookshelf bentopdf bookstack calcom calibre-web claper cloudflared code-server crafty-controller dawarich docmost emby filebrowser ghost gitea glance gokapi grafana gramps-web grimmory grocy home-assistant homebox homepage immich jellyfin karakeep kimai komga lubelogger mealie metube n8n navidrome nextcloud nzbget onlyoffice opengist outline paperless-ngx papra plant-it plex privatebin prowlarr qbittorrent radarr radicale rallly recipe-importer romm sabnzbd seerr sonarr sparkyfitness tandoor termix traefik transmission uptime-kuma vaultwarden vikunja wanderer wger wishlist zipline +felhom-controller-data + +=== certificate files (system paths and infra stacks only) === + +=== acme stores (certificate fields only) === +--- /opt/docker/stacks/traefik/acme.json mtime=2026-10-14T21:15:22Z +letsencrypt *.enkicsifelhom.hu | subject=CN=*.enkicsifelhom.hu issuer=C=US, O=Let's Encrypt, CN=YR1 notBefore=Oct 4 18:44:07 2026 GMT notAfter=Jan 2 18:44:06 2027 GMT sha256 Fingerprint=41:37:66:A9:A3:80:76:6D:3D:E1:64:AD:D2:CC:98:98:47:05:92:22:1D:00:F1:6F:E3:82:9C:1A:0D:E6:30:FD + +=== served certificates (TLS handshake to own listeners) === +*:22 -> no TLS +0.0.0.0:443 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 04:33:46 2026 GMT notAfter=Nov 11 04:33:46 2027 GMT sha256 Fingerprint=68:F4:15:12:FB:45:7A:81:06:62:56:3E:B5:DA:77:AD:0E:9C:B8:B1:9E:21:CF:79:C0:D9:DB:82:2A:9A:16:84 +0.0.0.0:80 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 04:33:46 2026 GMT notAfter=Nov 11 04:33:46 2027 GMT sha256 Fingerprint=68:F4:15:12:FB:45:7A:81:06:62:56:3E:B5:DA:77:AD:0E:9C:B8:B1:9E:21:CF:79:C0:D9:DB:82:2A:9A:16:84 +127.0.0.1:25 -> no TLS +[::1]:25 -> no TLS +[::]:443 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 04:33:46 2026 GMT notAfter=Nov 11 04:33:46 2027 GMT sha256 Fingerprint=68:F4:15:12:FB:45:7A:81:06:62:56:3E:B5:DA:77:AD:0E:9C:B8:B1:9E:21:CF:79:C0:D9:DB:82:2A:9A:16:84 +[::]:80 -> subject=CN=TRAEFIK DEFAULT CERT issuer=CN=TRAEFIK DEFAULT CERT notBefore=Nov 11 04:33:46 2026 GMT notAfter=Nov 11 04:33:46 2027 GMT sha256 Fingerprint=68:F4:15:12:FB:45:7A:81:06:62:56:3E:B5:DA:77:AD:0E:9C:B8:B1:9E:21:CF:79:C0:D9:DB:82:2A:9A:16:84 + +=== os === +Debian GNU/Linux 13 (trixie) / trixie +13.7 + +=== docker === +client=29.8.2 server=29.8.2 +containerd.io 2.3.6-1~debian.13~trixie +docker-ce 5:29.8.2-1~debian.13~trixie +docker-compose-plugin 5.6.0-1~debian.13~trixie +gitea.dooplex.hu/admin/felhom-controller:0.352.0 + +=== restic and tools inside the controller === +restic 0.19.1 compiled with go1.26.4 on linux/amd64 +PRETTY_NAME="Debian GNU/Linux 13 (trixie)" +NAME="Debian GNU/Linux" +VERSION_ID="13" + +=== images: repo tag digest created === +alpine:latest sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6 2026-09-17 20:37:20 +0000 UTC +cloudflare/cloudflared:2026.10.3 sha256:faa8e0a1475f137c910be6025f43c8e4092fdd35461b31e19fd1390657a1ff39 2026-10-21 13:26:38 +0000 UTC +cloudflare/cloudflared:2026.10.4 sha256:c6c7ae1d5dfdcb094ad344ed68ccffc26635c9c6be80965664c33baf064b0515 2026-10-26 11:27:24 +0000 UTC +gitea.dooplex.hu/admin/felhom-controller:0.351.0 sha256:1d90563c443022770b2db90a7730f1ea5aea45cc5b0fa64fec9823d60d42d97f 2026-11-09 12:14:14 +0000 UTC +gitea.dooplex.hu/admin/felhom-controller:0.352.0 sha256:3c785dd396b091f19d4c777db9d7a206e75ab018b6238b5d887e1a4c6cfd3b29 2026-11-10 15:16:20 +0000 UTC +gitea.dooplex.hu/admin/felhom-samba:1.1.0 sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10 2026-07-20 11:36:16 +0000 UTC +gtstef/filebrowser:1.5.6-stable sha256:7c5d7ac8ffda31294d278063cf9d2e04303b39e6dce1f4c691342240ca7703b8 2026-09-04 21:33:58 +0000 UTC +gtstef/filebrowser:1.5.8-stable sha256:68455d4953bad8e984e1bc5d112ed1e9c918a0a04f104c13c9ece50790308894 2026-10-04 21:15:40 +0000 UTC +hello-world:latest sha256:5e23090353324d887c48ad5e5c56d294eab81588df9605b07d1afe895f9cc8f8 2026-03-23 21:33:59 +0000 UTC +traefik/whoami:v1.10.3 sha256:43a68d10b9dfcfc3ffbfe4dd42100dc9aeaf29b3a5636c856337a5940f1b4f1c 2024-08-22 08:07:12 +0000 UTC +traefik:v3.7.16 sha256:1336e85aaed293bd63d005086e0dd013b04078b26aaeb1c18b26a2f928af7e65 2026-10-21 22:55:46 +0000 UTC +traefik:v3.7.17 sha256:68ae5fe5b64ca166704b746a9e8afcd385e36b3f569a5691401bf652d57b3748 2026-10-30 20:17:26 +0000 UTC + +=== apt sources === +--- /etc/apt/sources.list.d/debian.sources +Types: deb +URIs: http://security.debian.org +Suites: trixie-security +Components: contrib main +Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg +Types: deb +URIs: http://deb.debian.org/debian +Suites: trixie trixie-updates +Components: contrib main +Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg +--- /etc/apt/sources.list.d/docker.list +deb [signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian trixie stable + +=== apt keyring files === + + +3817 2026-10-04 docker.asc + + + +11861 2025-04-09 debian-archive-bookworm-automatic.asc +11873 2025-04-09 debian-archive-bookworm-security-automatic.asc +461 2025-04-09 debian-archive-bookworm-stable.asc +11861 2025-04-09 debian-archive-bullseye-automatic.asc +11873 2025-04-09 debian-archive-bullseye-security-automatic.asc +3403 2025-04-09 debian-archive-bullseye-stable.asc +11861 2025-04-09 debian-archive-trixie-automatic.asc +11873 2025-04-09 debian-archive-trixie-security-automatic.asc +1384 2025-04-09 debian-archive-trixie-stable.asc + + + +37 2025-04-09 debian-archive-bookworm-automatic.gpg +8700 2025-04-09 debian-archive-bookworm-automatic.pgp +46 2025-04-09 debian-archive-bookworm-security-automatic.gpg +8709 2025-04-09 debian-archive-bookworm-security-automatic.pgp +34 2025-04-09 debian-archive-bookworm-stable.gpg +280 2025-04-09 debian-archive-bookworm-stable.pgp +37 2025-04-09 debian-archive-bullseye-automatic.gpg +8700 2025-04-09 debian-archive-bullseye-automatic.pgp +46 2025-04-09 debian-archive-bullseye-security-automatic.gpg +8709 2025-04-09 debian-archive-bullseye-security-automatic.pgp +34 2025-04-09 debian-archive-bullseye-stable.gpg +2453 2025-04-09 debian-archive-bullseye-stable.pgp +26 2025-04-09 debian-archive-keyring.gpg +55918 2025-04-09 debian-archive-keyring.pgp +31 2025-04-09 debian-archive-removed-keys.gpg +72636 2025-04-09 debian-archive-removed-keys.pgp +35 2025-04-09 debian-archive-trixie-automatic.gpg +8698 2025-04-09 debian-archive-trixie-automatic.pgp +44 2025-04-09 debian-archive-trixie-security-automatic.gpg +8707 2025-04-09 debian-archive-trixie-security-automatic.pgp +32 2025-04-09 debian-archive-trixie-stable.gpg +962 2025-04-09 debian-archive-trixie-stable.pgp + +=== apt release files: Valid-Until === +deb.debian.org_debian_dists_trixie-updates_InRelease | Date: Wed, 11 Nov 2026 02:14:24 UTC | Valid-Until: Wed, 18 Nov 2026 02:14:24 UTC +deb.debian.org_debian_dists_trixie_InRelease | Date: Sat, 12 Sep 2026 07:55:41 UTC | Valid-Until: +download.docker.com_linux_debian_dists_trixie_InRelease | Date: Fri, 06 Nov 2026 03:19:25 +0000 | Valid-Until: +security.debian.org_dists_trixie-security_InRelease | Date: Wed, 11 Nov 2026 00:23:32 UTC | Valid-Until: Wed, 18 Nov 2026 00:23:32 UTC + +=== controller data folder (top level: names and dates only) === +--- felhom-controller-data +total +-rw------- 3044 2026-10-04 19:41 controller.yaml +-rw------- 3044 2026-10-04 19:41 controller.yaml.pre-media-2026-10-14 +drwxr-xr-x 4096 2026-11-11 10:06 data +RC=0 diff --git a/documentation/audits/end-dates-list-2026-11-11/evidence/box/tester-1-host-dates.txt b/documentation/audits/end-dates-list-2026-11-11/evidence/box/tester-1-host-dates.txt new file mode 100644 index 00000000..b0db9214 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/evidence/box/tester-1-host-dates.txt @@ -0,0 +1,214 @@ +# host-dates tester-1 taken 2026-11-11T10:05:43Z from DooPlex + +=== now === +2026-11-11T10:05:43Z + +=== certificate files === +/etc/pve/local/pve-ssl.pem [certs=1 mtime=2026-10-04] subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=felhom.enkicsifelhom.hu issuer=CN=Proxmox Virtual Environment, OU=f9c69a56-050d-43dd-87c8-d596f6a79d19, O=PVE Cluster Manager CA notBefore=Oct 3 19:38:05 2026 GMT notAfter=Oct 2 19:38:05 2028 GMT sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B +/etc/pve/nodes/felhom/pve-ssl.pem [certs=1 mtime=2026-10-04] subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=felhom.enkicsifelhom.hu issuer=CN=Proxmox Virtual Environment, OU=f9c69a56-050d-43dd-87c8-d596f6a79d19, O=PVE Cluster Manager CA notBefore=Oct 3 19:38:05 2026 GMT notAfter=Oct 2 19:38:05 2028 GMT sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B +/etc/pve/pve-root-ca.pem [certs=1 mtime=2026-10-04] subject=CN=Proxmox Virtual Environment, OU=f9c69a56-050d-43dd-87c8-d596f6a79d19, O=PVE Cluster Manager CA issuer=CN=Proxmox Virtual Environment, OU=f9c69a56-050d-43dd-87c8-d596f6a79d19, O=PVE Cluster Manager CA notBefore=Oct 3 19:38:05 2026 GMT notAfter=Sep 30 19:38:05 2036 GMT sha256 Fingerprint=4E:46:8D:BC:45:A2:2C:CD:AB:EF:82:C0:A4:B1:4D:6C:5C:DA:5A:7F:6F:D7:3B:E9:B7:50:CA:ED:C4:0C:80:CF +/var/lib/felhom-agent/local-api.crt [certs=1 mtime=2026-10-04] subject=CN=felhom-agent local-api issuer=CN=felhom-agent local-api notBefore=Oct 4 18:40:41 2026 GMT notAfter=Oct 1 18:40:41 2036 GMT sha256 Fingerprint=93:5A:5D:8A:FA:36:BE:87:4D:4E:FE:7D:83:8A:27:3E:89:7C:83:CD:32:00:CE:47:93:79:47:46:FF:63:DA:C6 + +=== system CA store === +148 +ca-certificates 20250419+deb13u1 + +=== served certificates (TLS handshake to own listeners) === +0.0.0.0:111 -> no TLS +0.0.0.0:22 -> no TLS +0.0.0.0:8822 -> no TLS +[::]:111 -> no TLS +[::1]:25 -> no TLS +127.0.0.1:25 -> no TLS +127.0.0.1:53 -> no TLS +127.0.0.1:85 -> no TLS +169.254.253.1:8443 -> subject=CN=felhom-agent local-api issuer=CN=felhom-agent local-api notBefore=Oct 4 18:40:41 2026 GMT notAfter=Oct 1 18:40:41 2036 GMT sha256 Fingerprint=93:5A:5D:8A:FA:36:BE:87:4D:4E:FE:7D:83:8A:27:3E:89:7C:83:CD:32:00:CE:47:93:79:47:46:FF:63:DA:C6 +192.168.0.154:53 -> no TLS +[::]:22 -> no TLS +*:3128 -> no TLS +*:8006 -> subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=felhom.enkicsifelhom.hu issuer=CN=Proxmox Virtual Environment, OU=f9c69a56-050d-43dd-87c8-d596f6a79d19, O=PVE Cluster Manager CA notBefore=Oct 3 19:38:05 2026 GMT notAfter=Oct 2 19:38:05 2028 GMT sha256 Fingerprint=BD:EE:9F:C4:FA:73:3A:45:F5:99:FB:F5:3B:3A:23:AE:35:2F:B6:B6:E9:E4:A8:52:20:3A:71:4E:C4:AF:A2:7B +[::]:8822 -> no TLS + +=== listeners (port, process) === +0.0.0.0:111 users:(("rpcbind",),("systemd",)) +0.0.0.0:22 users:(("sshd",)) +0.0.0.0:8822 users:(("sshd",)) +[::]:111 users:(("rpcbind",),("systemd",)) +[::1]:25 users:(("smtpd",),("master",)) +127.0.0.1:25 users:(("smtpd",),("master",)) +127.0.0.1:53 users:(("dnsmasq",)) +127.0.0.1:85 users:(("pvedaemon +169.254.253.1:8443 users:(("felhom-agent",)) +192.168.0.154:53 users:(("dnsmasq",)) +[::]:22 users:(("sshd",)) +*:3128 users:(("spiceproxy +*:8006 users:(("pveproxy +[::]:8822 users:(("sshd",)) + +=== pve users and tokens (expire: 0 = never) === +user felhom-agent@pve enable 1 expire 0 tokens [] +user root@pam enable 1 expire 0 tokens [] + +=== acme / subscription === +message: There is no subscription key +status: notfound + +=== os === +Debian GNU/Linux 13 (trixie) / trixie +13.7 +pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.14-22-pve) +7.0.14-22-pve +Manually selected kernels: +None. + +Automatically selected kernels: +7.0.14-20-pve +7.0.14-22-pve +proxmox-kernel-7.0.14-20-pve-signed 7.0.14-20 +proxmox-kernel-7.0.14-22-pve-signed 7.0.14-22 +proxmox-kernel-7.0.2-6-pve-signed 7.0.2-6 + +=== apt sources === +--- /etc/apt/sources.list.d/ceph.sources +Types: deb +URIs: https://enterprise.proxmox.com/debian/ceph-squid +Suites: trixie +Components: enterprise +Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg +Enabled: no +--- /etc/apt/sources.list.d/debian.sources +Types: deb +URIs: http://deb.debian.org/debian/ +Suites: trixie trixie-updates +Components: main contrib non-free-firmware +Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg +Types: deb +URIs: http://security.debian.org/debian-security/ +Suites: trixie-security +Components: main contrib non-free-firmware +Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg +--- /etc/apt/sources.list.d/pve-enterprise.sources +Types: deb +URIs: https://enterprise.proxmox.com/debian/pve +Suites: trixie +Components: pve-enterprise +Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg +Enabled: no +--- /etc/apt/sources.list.d/pve-no-subscription.sources +Types: deb +URIs: http://download.proxmox.com/debian/pve +Suites: trixie +Components: pve-no-subscription +Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg +--- auth files (names only): /etc/apt/auth.conf.d: + +=== apt keyring files === + + + + + +11861 2025-04-10 debian-archive-bookworm-automatic.asc +11873 2025-04-10 debian-archive-bookworm-security-automatic.asc +461 2025-04-10 debian-archive-bookworm-stable.asc +11861 2025-04-10 debian-archive-bullseye-automatic.asc +11873 2025-04-10 debian-archive-bullseye-security-automatic.asc +3403 2025-04-10 debian-archive-bullseye-stable.asc +11861 2025-04-10 debian-archive-trixie-automatic.asc +11873 2025-04-10 debian-archive-trixie-security-automatic.asc +1384 2025-04-10 debian-archive-trixie-stable.asc +1187 2022-11-27 proxmox-release-bookworm.gpg +1185 2024-11-10 proxmox-release-trixie.gpg + + + +37 2025-04-10 debian-archive-bookworm-automatic.gpg +8700 2025-04-10 debian-archive-bookworm-automatic.pgp +46 2025-04-10 debian-archive-bookworm-security-automatic.gpg +8709 2025-04-10 debian-archive-bookworm-security-automatic.pgp +34 2025-04-10 debian-archive-bookworm-stable.gpg +280 2025-04-10 debian-archive-bookworm-stable.pgp +37 2025-04-10 debian-archive-bullseye-automatic.gpg +8700 2025-04-10 debian-archive-bullseye-automatic.pgp +46 2025-04-10 debian-archive-bullseye-security-automatic.gpg +8709 2025-04-10 debian-archive-bullseye-security-automatic.pgp +34 2025-04-10 debian-archive-bullseye-stable.gpg +2453 2025-04-10 debian-archive-bullseye-stable.pgp +26 2025-04-10 debian-archive-keyring.gpg +55918 2025-04-10 debian-archive-keyring.pgp +31 2025-04-10 debian-archive-removed-keys.gpg +72636 2025-04-10 debian-archive-removed-keys.pgp +35 2025-04-10 debian-archive-trixie-automatic.gpg +8698 2025-04-10 debian-archive-trixie-automatic.pgp +44 2025-04-10 debian-archive-trixie-security-automatic.gpg +8707 2025-04-10 debian-archive-trixie-security-automatic.pgp +32 2025-04-10 debian-archive-trixie-stable.gpg +962 2025-04-10 debian-archive-trixie-stable.pgp +2372 2025-07-09 proxmox-archive-keyring.gpg +113 2022-11-27 proxmox-offline-signing-key.pub +1187 2022-11-27 proxmox-release-bookworm.gpg +1185 2024-11-10 proxmox-release-trixie.gpg + +=== apt release files: Valid-Until === +deb.debian.org_debian_dists_trixie_InRelease | Date: Sat, 12 Sep 2026 07:55:41 UTC | Valid-Until: +deb.debian.org_debian_dists_trixie-updates_InRelease | Date: Wed, 11 Nov 2026 02:14:24 UTC | Valid-Until: Wed, 18 Nov 2026 02:14:24 UTC +download.proxmox.com_debian_pve_dists_trixie_InRelease | Date: Wed, 11 Nov 2026 01:41:17 +0000 | Valid-Until: +security.debian.org_debian-security_dists_trixie-security_InRelease | Date: Wed, 11 Nov 2026 00:23:32 UTC | Valid-Until: Wed, 18 Nov 2026 00:23:32 UTC + +=== templates and golden === + +=== agent files (names and dates only) === +/etc/felhom: +total +-rw-r--r-- root 7 2026-10-04 21:38 appliance-pairing-code +-rw-r--r-- root 3897 2026-11-07 23:28 config-bundle.json +-rw-r--r-- root 456 2026-10-04 21:40 crash-guard.conf +-rw-r--r-- root 131 2026-10-04 21:40 operator-signers +-rw-r--r-- root 63 2026-10-04 21:40 os-trust.json + +/etc/felhom-agent: +total +-rw------- felhom-agent 2451 2026-10-04 21:55 agent.json + +/var/lib/felhom-agent: +total +-rw------- felhom-agent 593 2026-11-11 05:35 backup-success-state.json +-rw------- felhom-agent 290 2026-11-10 16:17 controller-swap-9201.json +-rw------- felhom-agent 321 2026-10-17 10:57 drive-intents.json +drwx------ felhom-agent 4096 2026-10-29 10:34 escrow-stage +drwx------ felhom-agent 4096 2026-10-04 21:40 felhom-sshd +-rw------- felhom-agent 308 2026-10-15 10:04 format-job.json +-rw------- felhom-agent 67 2026-10-17 10:57 guest-binds.json +-rw------- felhom-agent 242 2026-11-11 10:19 guest-disk-trim.json +drwx------ felhom-agent 4096 2026-10-04 21:41 guests +-rw------- felhom-agent 24760 2026-11-11 10:45 journal.log +-rw------- felhom-agent 304 2026-11-10 13:35 line-test.json +-rw-r--r-- felhom-agent 591 2026-10-04 21:40 local-api.crt +-rw------- felhom-agent 227 2026-10-04 21:40 local-api.key +-rw------- felhom-agent 82 2026-10-04 21:41 local-tokens.log +-rw------- felhom-agent 4556 2026-11-10 16:14 nonces.log +drwx------ felhom-agent 4096 2026-11-11 10:59 os +drwx------ felhom-agent 4096 2026-10-04 21:55 pbsdr +-rw------- felhom-agent 367 2026-11-11 10:45 restore-test-state.json +drwxr-x--- felhom-agent 4096 2026-11-10 16:14 selfupdate +-rw------- felhom-agent 32 2026-11-03 08:57 store-grant-seen.json +drwx------ felhom-agent 4096 2026-10-15 10:04 units +drwx------ felhom-agent 4096 2026-10-04 21:40 wg + +=== agent === +felhom-agent 0.173.0 + +=== wireguard (interface and latest handshake time only) === +wg-felhom peer f3d1ZI… 2026-11-11T10:04:06Z + +=== tailscale === +no tailscale + +=== ssh host keys / authorized keys (counts only) === +3 +/root/.ssh/authorized_keys lines=3 + +=== smart: power-on hours and wear (no serials) === +sda: +sdb: +RC=0 diff --git a/documentation/audits/end-dates-list-2026-11-11/evidence/helpers/helper-dates-box.md b/documentation/audits/end-dates-list-2026-11-11/evidence/helpers/helper-dates-box.md new file mode 100644 index 00000000..5851c7f7 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/evidence/helpers/helper-dates-box.md @@ -0,0 +1,265 @@ +# End dates on the box: agent, controller, catalog, installer (helper result, candidates only) + +### 1. Web certificate of the household's address (Let's Encrypt through Traefik) +1. The public TLS certificate for the dashboard and apps. Product-owned Traefik asks Let's Encrypt with the household's e-mail `ACMEEmail` `CT/internal/stacks/infra.go:130`; stored in `acme.json` `CT/internal/infra/templates/traefik.yml.tmpl:51`. +2. The lifetime is the issuer's, not ours: no validity constant in our source. Resolver `letsencrypt` `CT/internal/infra/templates/traefik.yml.tmpl:48-59`. Lifetime value: NOT FOUND (grep `NotAfter|renew|90 ?days` in `CT/internal/infra`). +3. Traefik renews by itself; we configure only the challenge: `dnsChallenge` with a Cloudflare token `CT/internal/infra/templates/traefik.yml.tmpl:53`, else `httpChallenge` `CT/internal/infra/templates/traefik.yml.tmpl:59`. How long before the end: NOT FOUND in our source (Traefik's own default). +4. Browsers show a certificate error on every app address of the household. With no e-mail set Traefik serves a self-signed certificate, see comment at `TraefikData` `CT/internal/infra/infra.go:111-118`. +5. NOT FOUND: no expiry check or warning in the controller (grep `cert.*expir|expir.*cert|NotAfter|acme\.json` in `CT/internal` hit only `CT/internal/stacks/infra.go:160-168` and the mail relay). The issuer mails `ACMEEmail`, which is the household. +6. NOBODY KNOWS from our source: renewal on start is Traefik's behaviour. The file survives because it is created once, mode 0600, `CT/internal/stacks/infra.go:160-168`. The DNS route also needs the Cloudflare token `CFAPIToken` `CT/internal/infra/infra.go:116-118` to be still valid. +7. NOT FOUND in our source: retry is inside Traefik. The old certificate stays in `acme.json` `CT/internal/infra/templates/traefik.yml.tmpl:51` until it ends. Nothing in the controller reads the result (same grep as line 5). +8. SOURCE `CT/internal/infra/templates/traefik.yml.tmpl:48-59` for the setup; NOBODY KNOWS for renewal timing, warning and the switched-off case. + +### 2. Proxmox web certificate pinned by the agent +1. The agent talks to Proxmox on the host and trusts ONE certificate by fingerprint. The installer reads the served leaf once, `fp=` `E/scripts/felhom-host-install.sh:3182-3185`; the agent checks it in `TLSConfig.build` `AG/internal/proxmox/tls.go:39-59`. Product-owned. +2. The pin has NO END DATE: the date of the certificate is never looked at, `VerifyPeerCertificate` `AG/internal/proxmox/tls.go:49`. The end is the day Proxmox serves a different leaf. Proxmox's own certificate lifetime: NOT FOUND (grep `pve-ssl|updatecerts|pveproxy cert` in `AG/`, `E/scripts`, `AR/`, `RB/`). +3. NOT FOUND: no code re-reads the fingerprint (grep `TLS pin mismatch|tls\.fingerprint|\.Fingerprint =` in `AG/internal`). The pin is loaded from config `AG/internal/config/config.go:917` and used at start `AG/cmd/felhom-agent/main.go:340`. +4. Candidate: if Proxmox replaces its certificate, every agent call to Proxmox fails with the mismatch error `AG/internal/proxmox/tls.go:55`: no backup, restore, guest start or resize through the agent. Household and operator both. +5. NOT FOUND: no warning before (same greps as line 2 and 3). After the fact the calls fail; which alarm fires was not traced. +6. NOBODY KNOWS: depends on whether Proxmox regenerates its certificate at boot, which is not in our source. If it does, the first failing step is the first Proxmox API call after start, `AG/internal/proxmox/tls.go:55`. +7. The agent keeps the old pin and keeps failing closed, `AG/internal/proxmox/tls.go:55`. Repair path found: none in code. An empty pin falls back to system trust, warning at `E/scripts/felhom-host-install.sh:3185`. +8. TEST `TestTLSConfig_Build` `AG/internal/proxmox/tls_test.go:28` for the pin check; NOBODY KNOWS for the renewal case. + +### 3. Agent local-API certificate and its pin in the guest +1. The self-signed leaf the agent serves to the in-guest controller, made by `generateLeaf` `AG/internal/localapi/cert.go:56`. The controller pins its fingerprint in `New` `CT/internal/agentapi/client.go:105`. Product-owned, on the host. +2. `certValidity = 10 * 365 * 24 * time.Hour` `AG/internal/localapi/cert.go:25`, set as `NotAfter` `AG/internal/localapi/cert.go:70`. So a box installed in 2026 has a leaf that ends in 2036. +3. Nobody: `EnsureLeaf` loads the existing pair and never looks at its date `AG/internal/localapi/cert.go:38-52`. The comment says rotation is an operator action for a later slice `AG/internal/localapi/cert.go:21-24`. +4. Candidate: at the end date nothing breaks in the controller, because it sets `InsecureSkipVerify` and compares only the fingerprint `CT/internal/agentapi/client.go:118-128`. What breaks things is a NEW leaf: mismatch `CT/internal/agentapi/client.go:125`. +5. Nobody. NOT FOUND: no expiry reading of this leaf (grep `NotAfter` in `AG/internal` hit only `AG/internal/localapi/cert.go:70`). +6. Yes, by itself: the leaf is generated once and kept, `AR/03-host-agent.md:420-423`, and the reinstall path keeps it with `--preserve-state-from` `E/scripts/felhom-host-install.sh:2749-2786`. +7. No renewal exists, so no failure state. A reinstall that makes a new leaf needs `--allow-new-leaf` `E/scripts/felhom-host-install.sh:2749-2786` and then the guest's pin is stale. +8. TEST `TestEnsureLeaf_StableFingerprintAcrossReload` `AG/internal/localapi/cert_test.go:12` and `TestClient_WrongPinFailsClosed` `CT/internal/agentapi/client_test.go:55`; the 10-year end itself: SOURCE only. + +### 4. Off-site copy after the household stops it: 14-day countdown +1. When the household abandons the off-site backup, the copies are kept for a grace period and then deleted. `startAbandonCountdown` `CT/internal/backup/offbox_abandon.go:68-70`. The household starts it; the product runs it. +2. `abandonGraceDays = 14` `CT/internal/backup/offbox_abandon.go:31`. +3. The household can stop or extend it; extending never shortens, test `TestD1_ExtendNeverShortensTheCountdown` `CT/internal/backup/offbox_abandon_levers_d1_test.go:20`. +4. The off-site copies are gone for good; the household loses its only copy outside the house. The deleting side is the hub (see the hub helper); test name `TestD1_ExtendRefusedWhileTheDeletionIsTheHubs` `CT/internal/backup/offbox_abandon_levers_d1_test.go:44`. +5. The household, at 5, 3 and 1 days left: `AbandonRemindAtDays` `CT/internal/backup/offbox_abandon.go:40`. +6. NOT FOUND for a box that is off during the whole countdown: whether the hub deletes without the box's reminders was not traced on the box side (grep `abandonNow|AbandonRemindAtDays` in `CT/internal/backup`). +7. A failed hub cancel is reported, not hidden: `TestD1_StopReportsFailureWhenTheHubCancelFails` `CT/internal/backup/offbox_abandon_levers_d1_test.go:69`. +8. TEST `TestR241_ScenarioE_AbandonStartsAReversibleCountdown` `CT/internal/backup/offbox_abandon_r241_test.go:48`. + +### 5. Off-site backup history: how far back a household can go +1. The restic off-site store keeps a thinning history. Policy in `retentionPolicy` `CT/internal/backup/offbox_window.go:81-82`. Product rule, household's data. +2. `keepDaily = 7` `CT/internal/backup/offbox_window.go:47`, `keepWeekly = 4` `CT/internal/backup/offbox_window.go:48`, `keepMonthly = 6` `CT/internal/backup/offbox_window.go:49`. Older than about six months: gone. +3. Nobody renews; each run forgets by the same rule, `retentionPolicy` `CT/internal/backup/offbox_window.go:81-82`. +4. A file deleted or damaged more than six monthly points ago cannot be brought back from off-site. On-host copies are fewer: `KeepLast` `AG/internal/config/config.go:436`, default 3 per comment `AG/internal/config/config.go:409`. +5. NOT FOUND: no warning that a point is about to leave the window (grep `keepMonthly|about to|will be removed` in `CT/internal/backup`). +6. Yes. A box that was off simply has no points for that time; the next run applies the same rule. PBS copies are not pruned by the box by default, comment `AG/internal/backup/runner.go:51-58`. +7. NOT FOUND for a failed forget (grep `forget` failure handling not traced). The weekly content check has its own age `defaultIntegrityMaxAgeDays = 7` `CT/internal/backup/offbox_integrity.go:81`. +8. SOURCE `CT/internal/backup/offbox_window.go:47-49`. + +### 6. Off-site credentials: SSH key, host key, store password, PBS pin +1. The box reaches its off-site store with an SSH key and a pinned host key, `StrictHostKeyChecking=yes` `CT/internal/backup/offbox.go:345`, and a password file `RESTIC_PASSWORD_FILE` `CT/internal/backup/offbox.go:898`. PBS is pinned in `pinnedTLS` `AG/internal/pbs/pin.go:17-30`. +2. NO END DATE in box code: no expiry field on the key, the host key or the password (grep `expir` in `CT/internal/backup/offbox.go` found none). The PBS pin ignores the date, `VerifyPeerCertificate` `AG/internal/pbs/pin.go:24`. The end comes from outside: the provider changes its host key or the hub ends the account. +3. Nobody on a schedule. A new PBS descriptor is checked before use: `probeFP` `AG/internal/pbsdr/manager.go:375-379`. +4. If the provider's host key changes, every off-site run fails at the SSH step, `StrictHostKeyChecking=yes` `CT/internal/backup/offbox.go:895`. The household has no off-site copy from that night on. +5. NOT FOUND before the end. After it, the failed run is the signal (alarm path not traced here). +6. Yes if nothing changed outside. A rebuilt guest gets a fresh store password and cannot open the old store: NOT FOUND in this pass (grep `RESTIC_PASSWORD_FILE` `CT/internal/backup/offbox.go:910` shows use only). +7. PBS: state `verify_failed` and a retry at the next tick, `probeFP` `AG/internal/pbsdr/manager.go:375-379`. Restic: the run fails and the next night tries again with the same pins. +8. SOURCE `CT/internal/backup/offbox.go:345`; provider-side dates: NOBODY KNOWS from the box. + +### 7. Sign-in sessions and cookies (dashboard, setup gate, family, share) — merged +1. Merged: dashboard session, setup-gate pass, family session, share-guest cookie. All product-made, in the household's browser and the controller's store, `loadSessions` `CT/internal/web/session_store.go:73`. +2. `sessionMaxAge = 7 * 24 * time.Hour` `CT/internal/web/auth.go:30`; `gateCookieLife = 7 * 24 * time.Hour` `CT/internal/web/setup_gate.go:46`; `SessionLife = 30 * 24 * time.Hour` `CT/internal/family/family.go:42`; `shareCookieMaxAge = 30 * 24 * time.Hour` `CT/internal/web/share.go:31`. +3. The family session renews itself at half-life `CT/internal/family/family.go:506-514`. The dashboard session has a fixed end set at sign-in `CT/internal/web/auth.go:308`. +4. The person is asked to sign in again. Nothing is lost. +5. Nobody; no warning is needed. +6. Yes. Expired rows are dropped on load `CT/internal/web/session_store.go:105` and by `cleanupSessions` `CT/internal/web/auth.go:369`; the person signs in again. +7. No renewal to fail. +8. TEST `TestR35_ExpiredRowNotLoaded` `CT/internal/web/session_store_test.go:144` and `TestSetupGate_ARestartKeepsTheHouseholdsPass` `CT/internal/web/setup_gate_test.go:182`. + +### 8. Claim code and the one-use gate token +1. The one-time code a household uses to set its own password, and the short token that carries a pass between addresses. Product-made. `claimCodeTTL` `CT/internal/web/claim.go:30`; `gateTokenLife` `CT/internal/web/setup_gate.go:47`. +2. `claimCodeTTL = 72 * time.Hour` `CT/internal/web/claim.go:30`; `gateTokenLife = 60 * time.Second` `CT/internal/web/setup_gate.go:47`, also stated in `AR/01-topology-and-trust.md:192`. +3. The operator sends a new code (hub side, see the hub helper). Nothing on the box mints one by itself. +4. A household that opens the mail after three days cannot set its password until a new code arrives. Checked against the controller's clock `s.claimNow()` `CT/internal/web/claim.go:333`. +5. NOT FOUND: no reminder before the 72 hours end (grep `claimCodeTTL` in `CT/internal/web`). +6. A box off for more than 72 hours comes back with a dead code; first failing step is the claim form, `CT/internal/web/claim.go:333`. +7. Wrong tries lock the form for a time: `claimLockoutWindow` `CT/internal/web/claim.go:33`. +8. TEST `TestClaimSubmit_ExpiredCodeRefused` `CT/internal/web/claim_test.go:189`. + +### 9. Share link `/s/` +1. The link a household gives to guests to see its app tiles. Household-owned, stored as `LauncherShareToken` `CT/internal/settings/settings.go:45`. +2. NO END DATE: the token has no expiry field; it ends only when the household rotates or disables it, `SetLauncherShareToken` `CT/internal/settings/settings.go:1111`. +3. The household, by hand. No schedule. +4. Nothing ends. Candidate for the list: a link given away years ago still opens the page. The guest's cookie lasts `shareCookieMaxAge` `CT/internal/web/share.go:31`. +5. Nobody; NOT FOUND any reminder of an old link (grep `LauncherShareToken` in `CT/internal`). +6. Yes, unchanged. +7. No renewal to fail. +8. TEST `TestShareGuest_RotateAndDisable` `CT/internal/web/share_test.go:256`. + +### 10. Recovery-code ceremony windows +1. The steps where the household confirms it wrote down its recovery code. Product-made, held by the agent. `escrowConfirmTTL` `AG/internal/localapi/escrow_ceremony.go:158`. +2. `escrowConfirmTTL = 30 * time.Minute` `AG/internal/localapi/escrow_ceremony.go:158`; `escrowClaimTTL = 10 * time.Minute` `AG/internal/localapi/escrow_ceremony.go:166`; the page allows `escrowCeremonyGraceWindow` `CT/internal/web/handlers.go:1169`. +3. The household starts the ceremony again. +4. A household that walks away mid-step must start over. The written-down code itself: NO END DATE found (grep `recovery.*expir|expir.*recovery` in `AG/internal`, `CT/internal` hit only the unclaimed-ceremony refusal `AG/internal/localapi/escrow_ceremony.go:506`). +5. Nobody. +6. Yes; an unfinished ceremony is simply started again. +7. A ceremony that timed out answers "no longer available" (HTTP 410) `AG/internal/localapi/escrow_ceremony.go:506`; the comment says an abandoned one leaves the household's written-down code as it was, `AG/internal/localapi/escrow_ceremony.go:38`. No automatic retry found. +8. SOURCE `AG/internal/localapi/escrow_ceremony.go:158-166`. + +### 11. Signed operator jobs: the validity window +1. An order signed by the operator (update the agent, apply OS packages, destructive acts). Operator-owned; it carries a nonce and an expiry, `AR/03-host-agent.md:268`. Checked by `Verify` `AG/internal/authz/verifier.go:104` and by the root wrapper. +2. At most `MaxWindow = 24 * time.Hour` `AG/internal/authz/verifier.go:26`; the signer's default is 30 minutes, `-ttl` `AG/cmd/felhom-opsign/main.go:57`. Wrapper: `MAX_WINDOW_S = 24 * 3600` `AG/configs/felhom-os-apply:194`. Design: `AR/04-control-plane-authorization.md:78-84`. +3. The operator signs a new one. There is no renewal of a signed job. +4. The job is refused and cleared; the executor is never called, `AG/internal/signedjobs/runner.go:163-170`. The box stays on its old agent or old packages. `ErrExpired` `AG/internal/authz/verifier.go:161-162`. +5. NOT FOUND a warning before a job ends. The design says an order unanswered after 24 hours is expired, `AR/04-control-plane-authorization.md:184`. +6. A box off for longer than the window never runs that job; it is checked against the BOX's clock, `time.Now().UTC()` `AG/internal/authz/verifier.go:90`, and in the wrapper `if now > exp or now < iss - 120` `AG/configs/felhom-os-apply:676`. A wrong box clock decides. +7. The refused job is logged as a security event and removed, `AG/internal/signedjobs/runner.go:163-170`; no retry. Used nonces are kept until their end, comment `AG/internal/authz/noncestore.go:47-50`. +8. TEST `test_expired_signed_job_is_refused` `AG/configs/test_felhom_os_apply.py:833`, `test_a_window_longer_than_24h_is_refused` `AG/configs/test_felhom_os_apply.py:836`, `TestVerify_RejectsPerStage` `AG/internal/authz/verify_test.go:82`. + +### 12. Operator signing keys on each box +1. The two public keys every box trusts for signed jobs: `OPERATOR_KEY_OPERATIONAL_ID` `E/scripts/felhom-host-install.sh:212` and `OPERATOR_KEY_RECOVERY_ID` `E/scripts/felhom-host-install.sh:214`. Written to `TRUST_SIGNERS` `AG/configs/felhom-os-apply:124` by `_write_slow_lane_trust` `E/scripts/felhom-host-install.sh:3424`. Operator-owned. +2. NO END DATE: the keys carry no expiry; trust is by key material, `AG/internal/authz/verifier.go:122-126`. The key id is advisory, `AG/internal/authz/verifier.go:170-175`. +3. Design only: every re-pin is itself a signed order, `AR/04-control-plane-authorization.md:139-150`. In code only the class name exists, `ClassKeyRotation` `AG/internal/reconcile/classify.go:41`. +4. If the operator loses both private keys, no box can be updated or repaired by a signed job again; households keep running on what they have. Design text `AR/04-control-plane-authorization.md:139-150`. +5. Nobody; there is no date to warn about. +6. Yes, the keys are in the installer and on disk, `E/scripts/felhom-host-install.sh:3452-3453`. +7. NOT FOUND: an executor for key rotation (grep `OpRepin|repin_key|key_rotation|OpKeyRotat|rotate_operator` in `AG/` hit only `AG/internal/reconcile/classify.go:41`). Candidate: rotation is designed, not built. +8. DOC `AR/04-control-plane-authorization.md:139-150`; for a real rotation: NOBODY KNOWS. + +### 13. Proxmox API token of the agent +1. The token the agent uses for every Proxmox call. Made by the installer in `step_token` `E/scripts/felhom-host-install.sh:2405`. Product-owned, on the host. +2. NO END DATE: minted with `pveum user token add ... --privsep 1` and no expire option, `E/scripts/felhom-host-install.sh:2448`. +3. Nobody on a schedule. A re-run of the installer reuses it or makes a new one, `E/scripts/felhom-host-install.sh:2424-2445`. +4. Nothing ends by date. If someone sets an expiry by hand in Proxmox, every agent call fails that day. +5. Nobody; NOT FOUND any check of the token's expiry (grep `expire` in `E/scripts/felhom-host-install.sh` near `step_token` found none). +6. Yes. +7. No renewal to fail. +8. SOURCE `E/scripts/felhom-host-install.sh:2448`. + +### 14. Guest token for the agent and the box's hub key — merged +1. Merged: the per-guest token the controller shows to the agent, kept as a hash in `TokenStore` `AG/internal/localapi/tokenstore.go:24-33`, and the agent's hub key used in `NewClient` `AG/internal/hub/client.go:60`. Product-owned. +2. NO END DATE for both: no expiry field in `TokenStore` `AG/internal/localapi/tokenstore.go:24-33`; none seen in `NewClient` `AG/internal/hub/client.go:36`. +3. Nobody on a schedule; a new guest token replaces the old one for that guest (last write wins), `AG/internal/localapi/tokenstore.go:24-33`. +4. Nothing ends by date. The hub connection also depends on public root certificates of the host's system store, `AG/internal/hub/client.go:41`, which age with the OS. +5. Nobody. +6. Yes; first step that can fail after years off is the TLS handshake to the hub if the host's root store is too old, `AG/internal/hub/client.go:41`. Candidate, not tested. +7. No renewal to fail. +8. SOURCE `AG/internal/localapi/tokenstore.go:24-33`; old root store: NOBODY KNOWS. + +### 15. WireGuard key of the operator tunnel +1. The box's key for the tunnel the operator uses to reach it. Product-owned, on the host. `rotateKey` `AG/internal/wgtunnel/rekey.go:110`. +2. NO END DATE and no periodic rotation: a new key is made only when the handshake time sits in the future for `rekeyAfterDefault = 15 * time.Minute` `AG/internal/wgtunnel/rekey.go:36`. Reason in the comment `AG/internal/wgtunnel/rekey.go:16-30`. +3. The agent by itself, at most once per `rekeyMinGap = time.Hour` `AG/internal/wgtunnel/rekey.go:38`; the old key is kept as `prevKeyName` `AG/internal/wgtunnel/rekey.go:39`. +4. Without the re-key, a box whose clock once ran ahead has a tunnel that never connects: the operator cannot reach it. The household notices nothing. +5. Nobody before; it is a repair, not a date. +6. Yes. This is exactly the case the re-key is for (a clock that jumped), `AG/internal/wgtunnel/rekey.go:16-30`. +7. A failed registration is retried by the existing path: `TestD30_AFailedRegistrationIsRetriedByTheExistingPath` `AG/internal/wgtunnel/rekey_test.go:119`. +8. TEST `TestD30_AFutureHandshakeThatStaysReKeys` `AG/internal/wgtunnel/rekey_test.go:35`, `TestD30_TheHourBoundSurvivesARestart` `AG/internal/wgtunnel/rekey_test.go:182`. + +### 16. Package signing keys: Docker in the guest, Proxmox and Debian on the host +1. The keys apt uses to trust packages. Docker's key is fetched once at golden bake into `/etc/apt/keyrings/docker.asc` `AG/configs/build-golden.sh:129`. Proxmox's keyring path default is in `align_apt_repos` `E/scripts/felhom-host-install.sh:2616`. Owned by Docker, Proxmox, Debian. +2. NOT FOUND in our source: the keys' own end dates (grep `expire|valid-until|keyring` in `AG/configs/build-golden.sh` and `E/scripts/felhom-host-install.sh`). The end is set by the publisher. +3. Debian and Proxmox keys arrive as packages through the normal update. Docker's key has no package: a change is "a slow-lane act for a person", `AR/11-os-updates.md:657`. +4. The OS update is refused with code R7 when `apt-get update` fails, `AG/configs/felhom-os-apply:1125-1127`. The box keeps running but stops getting Docker security updates. Household unaware; operator sees the report. +5. Nobody before the end. After it the box reports the failed update, `AR/11-os-updates.md:657`. +6. A box off for years comes back with old keys; the first failing step is `apt-get update` in the wrapper, `AG/configs/felhom-os-apply:1125-1127`. Whether the old key still verifies: NOBODY KNOWS. +7. The update fails cleanly and nothing is half-installed, `AR/11-os-updates.md:657`; the next signed job tries again and fails the same way until a person replaces the key. +8. DOC `AR/11-os-updates.md:657`; TEST for the refusal `test_R7_not_downloadable_and_no_snapshot` `AG/configs/test_felhom_os_apply.py:488`; key dates: NOBODY KNOWS. + +### 17. Debian 13 (trixie) and Proxmox 9: the series the product is tied to +1. The OS series under everything. Written into source: suite default `trixie` `E/scripts/felhom-host-install.sh:2615`, Docker repo `trixie stable` `AG/configs/build-golden.sh:130`, base images `CT/Dockerfile:52`. Owned by Debian and Proxmox. +2. NOT FOUND in source: the end-of-support dates of trixie and Proxmox 9 (grep `end of life|EOL|end-of-life|bookworm|forky` in `AR/11-os-updates.md` and `E/scripts`). The step to Proxmox 10 is named as later work, `AR/11-os-updates.md:725`. +3. The operator and the product; no date-driven plan found in the documents read. +4. After the series ends: no security updates for host and guest; a new install on a newer Proxmox is refused in byo mode and only warned in appliance mode, `E/scripts/felhom-host-install.sh:2146-2152`. +5. NOT FOUND (same grep as line 2). +6. Yes while the archives still serve trixie. Pinned OS updates fall back to snapshot.debian.org in `add_snapshot_sources` `AG/configs/felhom-os-apply:1451-1462`; if that source stops, the update is refused. +7. Refused update, box unchanged, `AG/configs/felhom-os-apply:1125-1127`. +8. SOURCE `E/scripts/felhom-host-install.sh:2146-2152`; the dates: NOBODY KNOWS. + +### 18. Catalog app images and their recorded digests +1. 65 app templates under `CAT/templates/`; 84 different image references; registries docker.io 47, ghcr.io 27, lscr.io 8, gitea.dooplex.hu 1, registry.gitlab.com 1. The update ladders record 149 different digests, e.g. `CAT/templates/actualbudget/.felhom.yml:196`. Owned by each app's publisher. +2. NO END DATE in source. The end is the day a registry stops serving a tag or a digest. Moving tags are banned: `BANNED_TAGS` `CAT/scripts/check-image-pins.py:28`. +3. The product, when it tests a new version and records its digest. `CAT/scripts/check-currency.py:22` reports how far behind a pin is and gates nothing. +4. The box renders `tag@sha256` in `RenderWithLadderDigests` `CT/internal/stacks/digest.go:94`; a digest the registry no longer serves makes the pull fail on every box, `AR/09-update-architecture.md:391`. A household cannot install or update that app; a running app keeps running. +5. NOT FOUND: a check that every recorded digest is still served (grep `manifest inspect|still served|HEAD` in `CAT/scripts`). `CAT/scripts/check-currency.py:22` reads tags, not old digests. +6. Running apps come back (images are local). The first failing step is the next install, update or restore that must pull, `CT/internal/stacks/digest.go:10-23`. +7. The pull fails and the app stays on its current image; no automatic repair. Retry happens only when the household presses again. +8. DOC `AR/09-update-architecture.md:391`; SOURCE `CT/internal/stacks/digest.go:94`; which digests are still served today: NOBODY KNOWS. + +### 19. Images the controller itself starts (Traefik, tunnel, file browser, Samba) +1. Four fixed images in the controller binary: `TraefikImage` `CT/internal/infra/infra.go:33`, `CloudflaredImage` `CT/internal/infra/infra.go:34`, `FileBrowserImage` `CT/internal/infra/infra.go:35`, `SambaImage` `CT/internal/infra/infra.go:50`. Product-owned pins of other people's images. +2. NO END DATE in source; tags only, no digest. Cloudflare's support window for an old cloudflared: NOT FOUND (grep `one year|1 year|12 months|support.*cloudflared` in `AR/01-topology-and-trust.md`). +3. The product, by a controller release that names newer tags. +4. Candidate: if Cloudflare stops accepting an old tunnel client, the household's address stops answering from outside though the box is healthy. If a registry drops a tag, a rebuilt box cannot start that part. +5. NOT FOUND (same grep as line 2). +6. A box off for a long time comes back with the old images it already holds; whether the tunnel still connects with an old client: NOBODY KNOWS. +7. Not traced. +8. SOURCE `CT/internal/infra/infra.go:33-50`; NOBODY KNOWS for the tunnel client's age limit. + +### 20. Published agent versions: only the newest five stay downloadable +1. The registry of agent binaries. Rule file `generic_versions_kept` `AG/scripts/retention-policy.json:23`. Operator-owned; he prunes by hand, `AG/scripts/retention-policy.json:7-10`. +2. Newest 5 versions plus two pins: `hub_min_agent` `AG/scripts/retention-policy.json:30` and `day0_agent` `AG/scripts/retention-policy.json:31`. Decision D50, `AR/09-update-architecture.md:1370-1375`. +3. The session that records a vouch or a new floor updates the pins in the same commit, `AG/scripts/retention-policy.json:12-15`. +4. A box told to install a version that was pruned cannot download it. A pin that names a missing version is a CI failure, `AG/scripts/retention-policy.json:14-15`. Deleting a git tag also ends the ability to install that version, `AG/scripts/retention-policy.json:20-21`. +5. The operator, through CI: `AG/scripts/retention-policy.json:25`. +6. A box off for long is on an old agent; it updates only by a fresh signed job that names a version still kept. First failing step otherwise: the download in the wrapper, `SELFUPDATE_OP` `AG/configs/felhom-os-apply:180`. +7. Not traced on a failed download. +8. SOURCE `AG/scripts/retention-policy.json:23-31`; DOC `AR/09-update-architecture.md:1370-1375`. + +### 21. Golden image age and the waiver file +1. The ready-made guest image a new box is installed from, baked weekly, `AR/09-update-architecture.md:1010-1018`. Operator vouches. Gate: `E/scripts/golden_currency_gate.py:169`. +2. `GOLDEN_MAX_AGE_DAYS = 7` `E/scripts/golden_currency_gate.py:169`. A waiver may span at most `WAIVER_MAX_DAYS = 14` `E/scripts/golden_currency_gate.py:167`, checked in `read_waiver` `E/scripts/golden_currency_gate.py:329-333`; path `WAIVER_PATH` `E/scripts/golden_currency_gate.py:162-163`. +3. The operator, weekly (Sunday night per `AR/09-update-architecture.md:1010-1018`). +4. The gate goes red for the product team; no household is touched. A new install gets an older image and must update more on first start. +5. The operator, by the gate itself; no earlier warning found. +6. Not about a box. +7. A waiver that has run out is no longer honoured, `E/scripts/golden_currency_gate.py:329-333`; no waiver file exists in the clones today (searched by file name `golden-waiver`). +8. SOURCE `E/scripts/golden_currency_gate.py:162-169`; tests exist in `E/scripts/test_golden_currency_gate.py:1` (names not matched to this rule). + +### 22. Debian container template the golden is built from +1. The upstream Debian LXC template: `TEMPLATE` default `debian-13-standard_13.1-2` `AG/configs/build-golden.sh:67`, used in `pct create` `AG/configs/build-golden.sh:101`. Owned by Proxmox's template mirror. +2. NO END DATE in source; the end is the day the mirror drops this exact file name. +3. The operator, by changing the default at a bake. +4. The weekly bake fails; no household is touched until a fresh golden is needed. +5. NOT FOUND (grep `pveam|TEMPLATE` in `AG/configs/build-golden.sh` shows only lines 23, 67, 79, 101: the script does not download or check the template). +6. Not about a box. +7. The bake stops at `pct create` `AG/configs/build-golden.sh:101`. +8. SOURCE `AG/configs/build-golden.sh:67`. + +### 23. Build-time pins of the controller image +1. What the controller image is built from: Go image `CT/Dockerfile:11`, Debian base `CT/Dockerfile:52`, restic `RESTIC_VERSION` `CT/Dockerfile:55` with checksums `CT/Dockerfile:56-57` and a vendored release key `CT/Dockerfile:58`, Docker's apt key fetched at build `CT/Dockerfile:110`. Product-owned. +2. NO END DATE in source. The base images are pinned by digest, so the end is the day the registry stops serving that digest. The vendored restic key's own expiry: NOT FOUND (not opened; key files are not read by this helper). +3. The product, at a release. +4. A new controller release cannot be built; boxes keep the image they have. A version number is pushed once, probe at `CT/build.sh:113-125`. +5. Nobody before; the build fails. +6. Not about a box. +7. The build fails with nothing pushed. +8. SOURCE `CT/Dockerfile:52-58`. + +### 24. Kept copies made before a restore: deleted after 30 unused days +1. The copy of an app's data the box keeps when the household restores an older state. Household's data, product rule D62. `restoreCopyUnusedAge` `CT/internal/backup/restore_copy_sweep.go:29`. +2. `restoreCopyUnusedAge = 30 * 24 * time.Hour` `CT/internal/backup/restore_copy_sweep.go:29`. +3. Nobody; using the copy is what keeps it. +4. The household loses the way back to the state from before the restore. +5. NOT FOUND a warning before the sweep (grep `restoreCopyUnusedAge` in `CT/internal`). +6. Candidate: a box off for more than 30 days deletes the copy at the first sweep after start, since age is counted by the wall clock, `CT/internal/backup/restore_copy_sweep.go:29`. A clock tolerance exists elsewhere: `KeptClockSkew` `CT/internal/backup/kept_load.go:217`. +7. Not traced. +8. SOURCE `CT/internal/backup/restore_copy_sweep.go:29`. + +## Tiny +- Mail relay's in-memory self-signed leaf, 10 years, remade at each start: `CT/internal/mailrelay/server.go:386`. +- Language cookie, 365 days: `CT/internal/web/i18n_web.go:478`. +- HSTS max-age one day: `HSTSMaxAgeSeconds` `CT/internal/infra/infra.go:108`. +- A backup counts as fresh enough for an update for 24 hours: `DefaultUpdateBackupMaxAge` `CT/internal/config/config.go:70`. +- Night-backup banner turns stale after 26 hours: `StaleAfter` `CT/internal/nightchain/banner.go:23`. +- Operator action brake 30 minutes and forget after 24 hours: `opBrakeRefusal` `AG/internal/localapi/opactions.go:42`. +- Backup-cap refusal held 24 hours: `backupCapRefusalHold` `AG/internal/localapi/backup_cap.go:38`. +- Reboot notice shown 24 hours: `opRebootNoticeFor` `AG/internal/localapi/opreboot.go:48`. +- Controller floor is kept in memory only: `floor` `CT/internal/selfupdate/updater.go:71`. +- Catalog sync uses a git user and token from config, no end date in code: `CT/internal/sync/sync.go:399`. +- Operator SSH key file comes from the hub's desired state, no end date seen: `AuthKeysUserPath` `AG/internal/felhomsshd/manager.go:20`. +- Kernel pin file for host updates: `KERNEL_PIN_FILE` `AG/configs/felhom-os-apply:151`. + +## Not found +- Proxmox's own certificate renewal and any re-pin after it: grep `pve-ssl|pveproxy cert|updatecerts|certificate.*(renew|expire)` in `AG/`, `E/scripts`, `AR/`, `RB/`. +- A warning before the web certificate ends: grep `cert.*expir|expir.*cert|NotAfter|acme\.json` in `CT/internal`. +- Cloudflare origin certificates: no use found; the box uses ACME only (grep `origin.?cert|origin_ca` in `CT/internal`, `AG/internal`). +- An executor for operator-key rotation: grep `OpRepin|repin_key|key_rotation|OpKeyRotat|rotate_operator` in `AG/`. +- End dates of the Docker, Debian and Proxmox apt keys, and of trixie and Proxmox 9: grep `expire|valid-until|EOL|end of life` in `AG/configs`, `E/scripts`, `AR/11-os-updates.md`. +- The PBS server certificate's validity: the pin ignores the date, `AG/internal/pbs/pin.go:24`. +- Retained (superseded) escrow keys' lifetime on the box side: grep `retain.*key|superseded` in `AG/internal`, `CT/internal` shows the agent recognising an earlier package, `AG/internal/localapi/escrow_recover.go:122-129`, but no age rule; see the hub helper. +- Expiry of the Cloudflare DNS token used for the web certificate: grep `CFAPIToken` in `CT/internal` shows use only. diff --git a/documentation/audits/end-dates-list-2026-11-11/evidence/helpers/helper-dates-hub.md b/documentation/audits/end-dates-list-2026-11-11/evidence/helpers/helper-dates-hub.md new file mode 100644 index 00000000..a594c683 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/evidence/helpers/helper-dates-hub.md @@ -0,0 +1,379 @@ +# End-dates list: hub, operator tooling and document-only things (helper result, candidates only) + +### 1. Self-bind (connect) link in the household's e-mail +1. A one-use link that lets the household tie a new box to its account. Household uses it, product mints it. Stored hashed in the hub table `selfbind_tokens` `HUB/store/store.go:796-806`; minted by `selfBindTTL` code `HUB/web/selfbind_mint.go:20`. +2. 7 days after minting: `selfBindTTL` `HUB/web/selfbind_mint.go:20`, passed to the store at `HUB/web/selfbind_mint.go:77`; `Expired` `HUB/store/selfbind.go:39`. Also locks at the 5th wrong answer, `SelfBindMaxAttempts` `HUB/store/selfbind.go:21`. +3. The household itself: the expired page has a button, `handleBindResend` `HUB/web/selfbind.go:384`, at most one new link an hour, `bindResendEvery` `HUB/web/selfbind.go:382`. No renewal before the end. The hub also mints at customer creation and RESET `HUB/web/selfbind_mint.go:97`. +4. The household cannot bind the box from the mail; unknown and expired links show the same "expired" page `HUB/web/selfbind.go:206`. The box itself keeps waiting; nothing on it is lost. +5. Nobody is warned before the end. NOT FOUND: grep `expir|remind` in `HUB/web/selfbind_mint.go:1-110` shows no reminder. The mint log line says "valid 7 days" `HUB/web/selfbind_mint.go:93`. +6. A box first switched on after day 7: the link fails at the first step (opening it); the household presses the resend button. If an old host is still linked no link is sent and the operator gets `eventBindOldHostLinked` `HUB/web/selfbind.go:440`, logged at `HUB/web/selfbind.go:413`. +7. If the mail send fails at mint the token is deleted, so no dead link is left `HUB/web/selfbind_mint.go:82`. No automatic retry; the household presses again after an hour. +8. TEST `TestR719_AnExpiredLinkOffersAndSendsAFreshOne` `HUB/web/selfbind_resend_test.go:21`; SOURCE `HUB/web/selfbind_mint.go:20`; DOC `AR/05-hub-architecture.md:359`. + +### 2. Customer claim code (the household's first password step) +1. A code the hub mails so the household can set its own box password. Household uses it; the hub stores a hash and a generation. `RotateClaimCode` `HUB/store/store.go:2084`. +2. NO END DATE: `RotateClaimCode` `HUB/store/store.go:2084` bumps a generation and writes no expiry. It ends when a newer code replaces it. Limit by count, not age: `maxResetPerDay` = 3 `HUB/claim/engine.go:30`. +3. The household asks for a reset; the hub mints a new code. No lead time, because there is no end. +4. After 3 resets in a day the 4th is refused, `maxResetPerDay` `HUB/claim/engine.go:30`, counted by `BumpResetCount` `HUB/store/store.go:2124`. The box side has its own 5-attempt, 15-minute lock, named at `HUB/claim/engine.go:27`. +5. Nobody; there is no end to warn about. +6. Comes back by itself: an old code still works until it is rotated. SOURCE `HUB/store/store.go:2084`. +7. NOT FOUND for a failed code mail: searched grep `expire|ttl|valid` in `HUB/claim/engine.go:1-60`; only the daily count limit is there. +8. SOURCE `HUB/claim/engine.go:30` and `HUB/store/store.go:2084`. No test for "a code never ages" was found (grep `Expire` in the claim package): NOBODY KNOWS whether that is intended. + +### 3. Retrieval passphrase: the moment it stops giving keys +1. The household's retrieval passphrase opens the box's keys at the hub only for a while. Household owns it; rule lives in `passphraseKeysClosed` `HUB/api/passphrase_scope.go:46`. +2. Ends on an EVENT, not a date: once the box has a key of its own and uses it, the passphrase gives no keys. Rule text `HUB/api/passphrase_scope.go:22`; needs controller `boxConfigMinController` 0.334.0 `HUB/api/passphrase_scope.go:40`. +3. Not renewed. To open it again the old host is deleted at the hub first (operator act), then a reinstall mints anew. DOC `AR/05-hub-architecture.md:270-272`. +4. A reinstall that reuses the passphrase without the old host deleted is refused; the household's new box gets no keys. Event `eventPassphraseKeysRefused` `HUB/api/passphrase_scope.go:42`. +5. Nobody before; the operator sees the event after a refusal, `eventPassphraseKeysRefused` `HUB/api/passphrase_scope.go:42`. +6. A running box is not touched: it uses its own key. A wiped box coming back fails at the key fetch step until the operator deletes the old host. +7. The refusal changes nothing; the same request works after the host delete. No retry by the hub. +8. TEST `TestR1159_PassphraseGivesKeysOnlyUntilTheBoxUsesItsOwn` `HUB/api/r1159_passphrase_scope_test.go:110`. + +### 4. Pairing code and appliance registration (box console code) +1. The 6-character code a new box shows on its screen, and the hub row behind it. Product mints, household types it. `PairingCode` `HUB/store/appliance.go:35`; box side `E/scripts/iso/felhom-bootstrap.sh:50`. +2. NO END DATE: `ApplianceByPairingCode` `HUB/store/appliance.go:155` has no age check. After 7 days without a poll the row only gets a "stale" badge, `applianceStaleAfter` `HUB/web/appliances.go:21`. +3. Nobody; the code stays the same across re-registration `HUB/store/appliance.go:35`. +4. Nothing breaks by age. An operator who discards the row makes the box log "appliance token not recognized" and keep trying `E/scripts/iso/felhom-bootstrap.sh:589`. +5. Operator only, by the stale badge on the appliances page `HUB/web/appliances.go:78`. No mail. +6. Comes back by itself: the box polls every `POLL_INTERVAL` 30 s `E/scripts/iso/felhom-bootstrap.sh:54` and retries the network gate every `GATE_RETRY_INTERVAL` 60 s `E/scripts/iso/felhom-bootstrap.sh:180`, with no give-up found. +7. A failed registration is retried by the unit `E/scripts/iso/felhom-bootstrap.sh:495`. +8. SOURCE `HUB/store/appliance.go:155` and `HUB/web/appliances.go:21`. No test of "never expires" found (grep `Stale|expire` in appliance files). + +### 5. "Switched off on purpose" hold (14 days of silence allowed) +1. When a household switches the box off through the product, the hub stays quiet about the silence. Household act, hub rule: `DeliberateOffHold` `HUB/store/deliberateoff.go:18`. +2. 14 days after the switch-off: `DeliberateOffHold` `HUB/store/deliberateoff.go:18`, judged by `DeliberateOffHeld` `HUB/store/deliberateoff.go:21`. +3. Nobody renews. The hold ends by itself when the box reports again or at day 14. +4. After day 14 the silence and missed-backup alarms start again for that household; the operator is told. +5. The operator gets an event at switch-off time naming the moment, message built at `HUB/api/deliberate_off.go:42`. No warning before day 14 was found (grep `DeliberateOffHold` in hub). +6. Yes, it comes back by itself: a box switched on at any time reports and the hold clears. TEST `TestDeliberateOff_BackOnIsQuiet` `HUB/monitor/deliberate_off_test.go:138`. +7. Not applicable: nothing is renewed. +8. TEST `TestDeliberateOff_AlarmsAfterFourteenDays` `HUB/monitor/deliberate_off_test.go:112` and `TestDeliberateOff_NoSilenceAlarmDuringTheHold` `HUB/monitor/deliberate_off_test.go:90`. + +### 6. Retained old escrow packages and the 7-day abandon of set-aside off-site data +1. Old sealed key packages the hub keeps after a re-key, and off-site data set aside from a previous install. Household's data, hub keeps it. Table `host_escrow_superseded` `HUB/store/store.go:440`. +2. Packages: NO END DATE, the schema comment says no pruning `HUB/store/store.go:417`. Set-aside data: deleted 7 days after the household asks to abandon it, `DefaultAbandonDelay` `HUB/offsitekeys/service.go:448`. +3. Nobody renews. The household can cancel inside the 7 days; request path `RequestAbandon` `HUB/offsitekeys/service.go:473`. +4. After the abandon runs, the old off-site data and its retained packages are gone for that household, `PurgeAbandonedEscrowForCustomer` `HUB/store/store.go:4360`. A customer delete also purges them `HUB/store/store.go:1985`. +5. NOT FOUND: no reminder before day 7 (grep `DefaultAbandonDelay|remind` in `HUB/offsitekeys/service.go:440-560`). A failure raises the event at `HUB/offsitekeys/service.go:540`. +6. Yes for the box: the sweep runs on the hub, `SweepAbandons` `HUB/offsitekeys/service.go:529`, and needs no box. A box that was off simply finds the old data gone. +7. A failed delete is marked and tried again on the next sweep, with one event `HUB/offsitekeys/service.go:540`. +8. TEST `TestAbandon_DelayCancelAndScope` `HUB/offsitekeys/service_test.go:134`; SOURCE `HUB/store/store.go:417`. + +### 7. Off-site clean-up window (the short time a box may delete off-site) +1. A time-boxed permission the hub opens so a box can prune its own off-site store. Product. `windowLength` `HUB/offsitekeys/service.go:263`. +2. 20 minutes per window, `windowLength` `HUB/offsitekeys/service.go:263`; a new one is allowed about weekly, `windowCadence` 6 days 12 hours `HUB/offsitekeys/service.go:264`. +3. The hub closes it; the box asks for the next one on its own night run. No human. +4. If the prune does not finish in 20 minutes it waits a week; the off-site store grows meanwhile. Operator and, in time, the household's off-site space. +5. NOT FOUND for a warning before close (grep `windowLength` in hub). The operator gets an event only when a window cannot be closed, logged at `HUB/offsitekeys/service.go:425`. +6. Yes: a box that was off asks again on its next run. First failing step if the cadence has not passed: the grant is refused. +7. A window past its time that cannot be closed is retried at the next sweep, `SweepExpiredWindows` `HUB/offsitekeys/service.go:413`, log `HUB/offsitekeys/service.go:425`. State left: delete right still open. +8. TEST `TestWindow_LeftOpenIsClosedByTheSweep` `HUB/offsitekeys/service_test.go:99` and `TestWindow_GrantOpenCloseAndDropAlarm` `HUB/offsitekeys/service_test.go:37`. + +### 8. Storage Box sub-account password and the hub's seal key +1. Each household's off-site store account at the provider, minted by the hub, and the hub key that seals stored secrets (`OFFSITE_SECRET_KEY`, named in `E/manifests/hub.yaml:167-171`). Operator and product. +2. NO END DATE in code: no lifetime on the sub-account password. A clean reinstall's re-issue resets it, DOC `AR/05-hub-architecture.md:274-279`; `ResetBoxPassword` `HUB/hetznerapi/hetznerapi.go:297`. +3. Operator, by pressing Re-issue. Seal key rotation is "not built", DOC `RB/secrets.md:159`. +4. If the seal key is lost or changed, the old sealed rows cannot be opened, DOC `RB/secrets.md:159`: operator loses stored off-site secrets and break-glass passwords until re-issued per household. +5. Nobody. NOT FOUND: grep `expir|rotat` in `HUB/hetznerapi/hetznerapi.go:1-320` finds no age rule or warning. +6. A box off for a long time comes back by itself as long as its account was not re-issued; it uses its SSH key, the password is used only by the hub, DOC `AR/05-hub-architecture.md:274-279`. +7. NOT FOUND for a half-done reset (searched `ResetBoxPassword` callers only by name). Candidate question for the main session. +8. DOC `RB/secrets.md:159` and `AR/05-hub-architecture.md:274-279`; SOURCE `HUB/hetznerapi/hetznerapi.go:297`. No test of a lifetime, because there is none. + +### 9. PBS off-site token on ep0 and its descriptor on the hub +1. The per-household token on the rented endpoint for the whole-guest off-site copy, and the hub record that must agree with it. Product mints, operator can re-issue. DOC `AR/05-hub-architecture.md:373-387`. +2. NO END DATE stated: the lifecycle table has only events (provision, re-key, deprovision) `AR/05-hub-architecture.md:379-387`. One age rule: a staged secret not consumed in 15 minutes is wrong, `unconsumedGrace` `HUB/pbsdrheal/reconciler.go:63`. +3. The hub re-keys on re-issue (delete and recreate) `AR/05-hub-architecture.md:382`. No scheduled renewal. +4. A re-key the box did not pick up leaves the box "serving a revoked credential": the household has no whole-guest off-site copy while both tiers look green, comment at `HUB/pbsdrheal/reconciler.go:55-63`. +5. Operator only, after the fact: `pbsdr_provision_failed` on a final failure, DOC `AR/05-hub-architecture.md:400-402`. +6. Host delete keeps the token `AR/05-hub-architecture.md:384`; a box off and back uses the same token. After a re-issue while it was off, it must consume the new secret on its next report. +7. A failed self-check rolls back and is repeated 3 times, 7 s apart, then raises the operator event, DOC `AR/05-hub-architecture.md:400-402`. +8. DOC `AR/05-hub-architecture.md:373-403`; SOURCE `HUB/pbsdrheal/reconciler.go:63`. Lifetime of the token on the endpoint itself: NOBODY KNOWS from these sources (ep0 not read). + +### 10. WireGuard peers (the box's tunnel to the off-site endpoint) +1. One peer per box on the endpoint, registered through the hub; the private key is in the household's sealed bundle. Product. DOC `AR/06-offsite-connectivity.md:172-182`. +2. NO END DATE: the hub store only removes a peer on request, `RemoveWGPeer` `HUB/store/wg.go:345`. Scheduled key rotation is an open point, DOC `AR/06-offsite-connectivity.md:368`. +3. Nobody on a schedule. The agent makes a new key itself only after a handshake stamped in the future for 15 minutes, DOC `AR/06-offsite-connectivity.md:205-207`. +4. Revocation: the box is off the off-site network at once, DOC `AR/06-offsite-connectivity.md:180-182`; the household's off-site copies stop. +5. Nobody before. NOT FOUND: grep `expire|rotat|stale|handshake` in `HUB/store/wg.go:1-360` finds no age rule. +6. Yes, a box off for months keeps its peer. A revoked box never re-registers on its own, DOC `AR/06-offsite-connectivity.md:200-205`. +7. A failed fetch is never read as a teardown, DOC `AR/06-offsite-connectivity.md:207-208`. Re-add is an operator call. +8. DOC `AR/06-offsite-connectivity.md:200-208`; SOURCE `HUB/store/wg.go:345`. Box-side tests belong to the box helper. + +### 11. Signed jobs (agent update, config bundle, OS step) +1. An operator-signed order the hub queues for one box. Operator signs with `felhom-opsign`; hub stores an opaque blob, `EnqueueSignedJob` `HUB/store/store.go:3764`; the agent checks it. +2. 30 minutes by default: flag `ttl` `AG/cmd/felhom-opsign/main.go:57`, end set at `AG/cmd/felhom-opsign/main.go:130`. Never more than 24 hours: `MaxWindow` `AG/internal/authz/verifier.go:26`, refused at signing `AG/cmd/felhom-opsign/main.go:126`. +3. The operator (or CC, see block 12) signs a new job. Nothing renews by itself. +4. A box that does not poll inside the window does not get the update or the root-file change; the operator must sign again for that box. +5. Nobody before. The agent logs the rejection and clears the job `AG/internal/signedjobs/runner.go:163`. The hub has no expiry of its own on the queue (grep `expire` near `HUB/store/store.go:3764`). +6. No. A box off past the end polls, gets the job, and fails at the time check `AG/internal/authz/verifier.go:161`. Clock skew allowed: `DefaultClockSkew` 2 minutes `AG/internal/authz/verifier.go:21`. +7. A rejected job is cleared; a job that passes the check but fails to run is also cleared and must be re-signed, default branch at `AG/internal/signedjobs/runner.go:186`. No retry. +8. TEST `TestVerify_RejectsPerStage` `AG/internal/authz/verify_test.go:82`; DOC `AR/04-control-plane-authorization.md:78`. + +### 12. Operational signing key and cold recovery key +1. The two keys every agent trusts for destructive orders. Operator owns them; both live on DooPlex today, DOC `AR/04-control-plane-authorization.md:111` and `AR/04-control-plane-authorization.md:96`. +2. NO END DATE on either key in any source. The only dated rule: "Revisit on the first sale", in §3.1 starting `AR/04-control-plane-authorization.md:111`. +3. The operator. Planned rotation is a key-signs-key order `AR/04-control-plane-authorization.md:143`; rotating a signer is described as not built `AR/04-control-plane-authorization.md:133`. +4. Both keys lost: no update, no config bundle, no OS step for any box until someone visits each box, DOC `AR/04-control-plane-authorization.md:148`. Every household. +5. Nobody; there is no date. The loss case is in the recovery runbook row S6 `RB/total-loss-of-dooplex.md:39`. +6. Yes. A box off for any time still trusts the same pinned keys. If the key was rotated while it was off: NOT FOUND how it learns the new key (searched §4 `AR/04-control-plane-authorization.md:137-149`). +7. NOT FOUND: no source says what state a half-done re-pin leaves (searched §4 `AR/04-control-plane-authorization.md:137-149`). +8. DOC `AR/04-control-plane-authorization.md:137-149`. Rotation never run on a real box as far as these files say: NOBODY KNOWS. + +### 13. Operator actions waiting for a box (24 hours; reboot 30 minutes) +1. Buttons the operator presses in the hub that a box must answer (restart, reboot and so on). Operator. `OperatorActionTTL` `HUB/store/opactions.go:62`. +2. 24 hours unanswered: `OperatorActionTTL` `HUB/store/opactions.go:62`. A host reboot order: 30 minutes, `HostOperatorActionRebootTTL` `HUB/store/opactions.go:47`. +3. The operator presses again. No renewal. +4. The action is marked expired with "did not answer within a day" `HUB/store/opactions.go:222` and `HUB/store/opactions.go:296`; nothing happens on the box. +5. Nobody before; the operator sees the expired state on the hub page. +6. Safe by design: a box back after the end does not run the stale order. The operator presses again. +7. Not applicable; an expired action is a final state. +8. DOC `AR/04-control-plane-authorization.md:184` names `TestOpActions_ClosedList` and `TestOperatorActions_ClosedList`; SOURCE `HUB/store/opactions.go:62`. + +### 14. Recovery mode for a host (the door a rebuilt box uses) +1. A switch the operator sets on a host so a rebuilt box may re-enrol and fetch its restore data. Operator, global key only, `handleSetRecoveryMode` `HUB/api/dr.go:34`. +2. 30 minutes by default, `defaultRecoveryTTL` `HUB/api/dr.go:21`; at most 4 hours, `maxRecoveryTTL` `HUB/api/dr.go:22`; clamp at `HUB/api/dr.go:52`. +3. The operator sets it again. No lead time. +4. A restore that takes longer than the window: the box's re-enrol call is refused and the household's recovery stops until the operator re-opens it. +5. Nobody. NOT FOUND: grep `defaultRecoveryTTL|maxRecoveryTTL` shows no warning before the end. +6. No: a box that starts its recovery after the window fails at the re-enrol step. TEST `TestReEnroll_GatedOnRecoveryMode` `HUB/api/dr_test.go:14`. +7. Nothing is left half-done by the expiry itself; the operator opens a new window. +8. TEST `TestRestoreDirective_GatedAndExpires` `HUB/api/dr_test.go:70`; SOURCE `HUB/api/dr.go:21-22`. + +### 15. Break-glass credential (the host's emergency root password) +1. A per-host emergency password, sealed in the hub, shown only to a logged-in operator. Operator. `SaveHostRecoveryCredential` `HUB/store/host_recovery.go:28`; reveal `HUB/web/hosts.go:983`. +2. NO END DATE: last write wins, day-0 sets it and a rotate re-sets it `HUB/store/host_recovery.go:20`. Paper rule: re-print the sheet "when a key rotates, and yearly" `RB/break-glass-sheet.md:75`. +3. The operator, yearly by the paper rule; no code reminds him. +4. If the sealed copy cannot be opened (seal key, block 8) or the paper is stale, the operator has no way into a host whose agent is dead. +5. Nobody before. Each reveal writes the event `recovery_credential_revealed` `HUB/web/hosts.go:1010`. +6. Yes, the password does not age on the box. +7. NOT FOUND what a half-done rotate leaves (hub updated, box not, or the reverse); searched `HUB/store/host_recovery.go:20-43` only. +8. SOURCE `HUB/store/host_recovery.go:20-28`; session-only route `HUB/web/routes.go:130`; DOC `RB/break-glass-sheet.md:75`. A yearly re-print has no record: NOBODY KNOWS. + +### 16. Vouched artifact manifest and the "golden not vouched" alarm +1. The hub's list of what a fresh install may fetch (agent, golden, checksums), vouched by the operator. Read by `resolve_artifacts` `E/scripts/felhom-host-install.sh:467`. +2. NO END DATE on a vouch. An age rule on the gap: a published golden not vouched after 2 hours raises an alarm, `goldenVouchGrace` `HUB/monitor/golden_vouch.go:42`, checked every 15 minutes `HUB/monitor/golden_vouch.go:43`. +3. The operator vouches; CC never does, DOC `AR/09-update-architecture.md:1022`. +4. An old vouch means a fresh install gets the older controller, message at `HUB/monitor/golden_vouch.go:92`. If the manifest is not served the install dies `E/scripts/felhom-host-install.sh:472`. +5. The operator, 2 hours after the publish, by the event built at `HUB/monitor/golden_vouch.go:92`. If Gitea cannot be read there is no verdict `HUB/monitor/golden_vouch.go:65`. +6. A new box after a long time installs what is vouched and then updates to the floor; a vouched version already pruned from the registry fails at the download (see block 18). +7. An artifact with no expected checksum is refused `E/scripts/felhom-host-install.sh:531`; the bootstrap unit retries the install every 30 s `E/scripts/iso/felhom-bootstrap.sh:430`. +8. TEST `TestGoldenVouchChecker` `HUB/monitor/golden_vouch_test.go:15`; SOURCE `E/scripts/felhom-host-install.sh:467-472`. + +### 17. The golden's age rule and its waiver file +1. A gate that fails a push when the newest release is ahead of the newest baked golden and that golden is old. Operator and CC. `GOLDEN_MAX_AGE_DAYS` `E/scripts/golden_currency_gate.py:169`. +2. Golden: 7 days from the date in its evidence folder name, `GOLDEN_MAX_AGE_DAYS` `E/scripts/golden_currency_gate.py:169`. Waiver: its own `expires` line, at most 14 days after issue, `WAIVER_MAX_DAYS` `E/scripts/golden_currency_gate.py:167`. +3. Operator bakes and vouches weekly, Sunday night, DOC `AR/09-update-architecture.md:997`. A waiver is renewed by editing two dates, DOC `RB/RUNBOOK-manual-build.md:291`. +4. The gate goes red and blocks pushes for the operator and CC; households are not touched. Expired check at `E/scripts/golden_currency_gate.py:354`. +5. The gate prints the age on every run `E/scripts/golden_currency_gate.py:422`. No warning before day 7 or before the waiver's end was found. +6. Not a box matter. +7. A waiver that is malformed or too long is refused out loud (exit 2), `read_waiver` `E/scripts/golden_currency_gate.py:287`. Candidate: the waiver file `WAIVER_PATH` `E/scripts/golden_currency_gate.py:162` is absent in this clone, and the newest evidence folder under documentation/tests is dated 2026-11-03, 8 days before today. +8. SOURCE `E/scripts/golden_currency_gate.py:162-169`; DOC `AR/09-update-architecture.md:1362` and `RB/RUNBOOK-manual-build.md:279`. Test names in the gate's test file NOT FOUND (grep `def test` returned nothing). + +### 18. Which agent versions stay downloadable (D50) +1. The rule for which published agent packages are kept in the registry. Operator prunes by hand; CI checks. File `AG/scripts/retention-policy.json:23-32`. +2. By count, not date: newest 5 (`generic_versions_kept` `AG/scripts/retention-policy.json:23`) plus two pins, `hub_min_agent` and `day0_agent` `AG/scripts/retention-policy.json:30-31`. Recorded 2026-11-10 `AG/scripts/retention-policy.json:27`. +3. The session that records a vouch or a new floor updates the pins in the same commit, rule text `AG/scripts/retention-policy.json:12-15`. +4. A box on a version outside the set cannot fetch its own binary again, DOC `AR/09-update-architecture.md:1370-1373`. A deleted git tag ends the ability to install that version `AG/scripts/retention-policy.json:20-21`. +5. CI only: `AG/scripts/check-published-versions.py:102` reads the file. No warning to anyone when a box's version is about to fall out of the newest 5 was found. +6. A box off while 5 newer versions ship comes back and updates forward by a signed job; what it does if it needs its OLD binary: fails at the download, per `AR/09-update-architecture.md:1372`. +7. A pin naming a version with no tag is a CI failure, never a skip `AG/scripts/retention-policy.json:14-15`. +8. DOC `AR/09-update-architecture.md:1370-1376`; SOURCE `AG/scripts/retention-policy.json:23-32`. The pins are hand copies of hub state `AG/scripts/retention-policy.json:12`: whether they match today, NOBODY KNOWS from files. + +### 19. The installer ISO (Proxmox base, installer version, first boot) +1. The USB image a household or volunteer installs from. Product; built by `E/scripts/iso/build-felhom-iso.sh:51`, served from Cloudflare R2, DOC `RB/total-loss-of-dooplex.md:125`. +2. NO END DATE in code. `ISO_VERSION` is 1.30.0 `E/scripts/iso/build-felhom-iso.sh:51`. The Proxmox ISO and its checksum are build ARGUMENTS, not pinned: `E/scripts/iso/build-felhom-iso.sh:194`; the example names 9.2-1 `E/scripts/iso/README.md:40`. +3. The operator rebuilds. No cadence found (grep `days|stale|too old` in `RB/iso-release-gate.md:1-230` returned nothing on age). +4. An old ISO installs an old Proxmox base; the install script itself is fetched fresh at first boot (comment block at `E/scripts/iso/build-felhom-iso.sh:44-52`), so the hub decides the agent and golden. +5. Nobody. The build warns only if golden and floor are not given: unenforced branch `E/scripts/iso/build-felhom-iso.sh:79`. +6. A too-old ISO on first boot: registers, pairs, fetches the current install script; first failing step would be that fetch or the manifest `E/scripts/felhom-host-install.sh:472`. No "ISO too old" check found (grep `too old|min.*version` in bootstrap). +7. A failed install is retried every 30 s `E/scripts/iso/felhom-bootstrap.sh:430`; a failed script fetch is retried `E/scripts/iso/felhom-bootstrap.sh:396`. +8. SOURCE `E/scripts/iso/build-felhom-iso.sh:222` (checksum of the given ISO is verified). How old an ISO still installs cleanly: NOBODY KNOWS. Build image is `FROM debian:trixie`, not pinned `E/scripts/iso/Dockerfile.assistant:10`. + +### 20. The certificate the hub's address serves +1. The TLS certificate for the hub's public name, used by every box and the operator. Operator's cluster issues it: annotation `E/manifests/hub.yaml:342`, stored under the secret name at `E/manifests/hub.yaml:360`. +2. NOT FOUND in these sources: the issuer `letsencrypt-prod` is only named `E/manifests/hub.yaml:342`; its definition and renewal timing are in the DooPlex cluster repository, outside this task. Searched grep `ClusterIssuer|renewBefore|duration` in E/manifests. +3. cert-manager on DooPlex, by the annotation `E/manifests/hub.yaml:342`. Lead time NOT FOUND (same search). +4. If it lapses: boxes that pin or verify the hub cannot report or fetch config; households get no alarms and no updates; the operator page shows a browser warning. +5. NOT FOUND: grep `cert|expir` in `HUB/monitor/golden_vouch.go:1-100` and across HUB/monitor finds no certificate-age alarm. +6. A box off past a renewal: depends on what it pins (box helper's item). From hub files: NOT FOUND. +7. NOT FOUND. The renewal needs the challenge path reachable; the ingress is limited to two countries in the block starting `E/manifests/hub.yaml:342`. Candidate question: does that block the issuer's check. +8. SOURCE `E/manifests/hub.yaml:342` and `E/manifests/hub.yaml:360` only. Dates: NOBODY KNOWS from files; needs a measurement. + +### 21. Operator login session on the hub +1. The operator's browser session cookie. Operator. `SessionCookieName` `HUB/web/server.go:930`; sessions are held in memory. +2. 7 days from login: session end `HUB/web/server.go:949`, cookie `MaxAge` `HUB/web/server.go:964`. Also ends at any hub restart, because the map is in memory (cleanup `CleanupSessions` `HUB/web/server.go:199`). +3. The operator logs in again. No sliding renewal found: the check only compares the fixed end `HUB/web/server.go:901`. +4. Session-only buttons (delete, reset, reveal, tokens) stop until he logs in; route rule example `HUB/web/routes.go:130`. Scripts with a token are not touched. +5. Nobody; the next click shows the login page. +6. Not a box matter. +7. Not applicable. Basic auth with the password still works as a second door `HUB/web/server.go:909`. +8. TEST `TestR136_LoginSetsHostPrefixedCookie` `HUB/web/r136_host_cookie_test.go:16` (cookie name); the 7 days are SOURCE only `HUB/web/server.go:949`. + +### 22. Operator access tokens, among them CC's hub token +1. Named, scoped tokens for scripts, shown once, stored hashed. Operator creates them in a browser session; CC holds one, with `overview:read` and `floor:write`, DOC `AR/09-update-architecture.md:1022`. +2. 90 days by default, `tokenDefaultDays` `HUB/web/tokens.go:29`; at most 365, `tokenMaxDays` `HUB/web/tokens.go:30`; "never" is not offered `HUB/web/tokens.go:254`. End computed at `HUB/web/tokens.go:180`. +3. The operator makes a new token and puts it in CC's store. There is no renew; no lead time is set anywhere. +4. CC's floor raise and overview reads get 401 `HUB/web/tokens.go:95`; a controller release cannot be carried to boxes by CC until the operator acts. Households wait for the update. +5. Nobody is warned. NOT FOUND: grep `token.*(expiring|days left)|token_expir` in the hub finds only comments. The page shows the date and the state "expired" `HUB/web/tokens.go:245`. +6. Not a box matter. +7. Not applicable; an expired token stays listed as expired. Column `expires_at` `HUB/store/api_tokens.go:52`. +8. TEST `TestToken_NoTokenUnknownExpiredRevoked401` `HUB/web/tokens_test.go:310`; DOC `AR/05-hub-architecture.md:655-669`. The end DATE of CC's token is in no document (grep `HUB_TOKEN` in AR, RB, operations, STATUS): NOBODY KNOWS without the hub page. + +### 23. A household's own Cloudflare token (hub side) +1. A household that brings its own domain gives a Cloudflare API token; the hub checks what it can reach before saving. Household owns it. `HUB/cloudflare/reach.go:14`. +2. NO END DATE known to the hub: the check at `HUB/cloudflare/reach.go:14` is about zones, not age. Whatever end the household set at Cloudflare is not read (grep `expire|rotat` in the cloudflare package: nothing). +3. The household, at Cloudflare; then the operator saves the new token in the hub. +4. If it lapses, the household's remote address stops being managed; candidate: first sign is a failed DNS or tunnel call on the box. +5. Nobody. NOT FOUND: same grep, no expiry field stored. +6. Box helper's item; from hub files NOT FOUND. +7. NOT FOUND. +8. SOURCE `HUB/cloudflare/reach.go:14`; the rest NOBODY KNOWS from hub files. + +### 24. The mail service key (all household and operator mail) +1. The key the hub uses to send every mail: alarms, bind links, claim codes. Operator's account at the mail service; named in `E/manifests/hub.yaml:148-152`, value not opened. +2. NO END DATE stated in any document read: `RB/secrets.md:33` describes create and rotate, with no date; inventory row `AR/_recovery-inventory-2026-07-28.md:502`. +3. The operator re-mints at the service dashboard, DOC `RB/total-loss-of-dooplex.md:47`. +4. All mail stops: no alarm reaches the operator or a household, no bind link, no claim code. Everyone. +5. NOT FOUND: no check that mail still goes out on a schedule was found in the documents read (grep `expir|renew` in RB/secrets.md and STATUS.md). +6. Not a box matter; boxes send through the hub. +7. A failed bind mail deletes its token `HUB/web/selfbind_mint.go:82`; for other mail kinds NOT FOUND in this pass. +8. DOC `RB/secrets.md:33` and `RB/total-loss-of-dooplex.md:47`. Whether the key or the plan has an end date: NOBODY KNOWS from files. + +### 25. The hub's global bearer key +1. The operator-tier key for the hub API (recovery mode, peer re-add). Operator. Named in `E/manifests/hub.yaml:157-161`; inventory row `AR/_recovery-inventory-2026-07-28.md:501`. +2. NO END DATE. Rotation is a supervised runbook `RB/secrets.md:124`. +3. The operator, with an ordered procedure `RB/secrets.md:124`; no cadence stated. +4. A rotation done out of order locks out whatever still holds the old key; DOC `RB/secrets.md:124` calls the order load-bearing. +5. Nobody; no date. +6. Boxes use per-box keys bound to their box since 2026-10-12, DOC `AR/13-threat-model.md:70`; a box still holding only the old global key after a rotation: NOT FOUND what it does. +7. NOT FOUND beyond the runbook order. +8. DOC only. Two documents disagree on whether it was rotated: "rotated" at `AR/13-threat-model.md:70`, "when last rotated" unknown at `AR/_recovery-inventory-2026-07-28.md:1030`. Candidate for a doc fix. + +### 26. Pinned tool versions in builds and CI +1. Versions of the tools that build and check the product. Operator and CC. Hub build image `E/hub/Dockerfile:3`, run image `E/hub/Dockerfile:24`, site publisher `E/scripts/publish-site-cloudflare.sh:28`. +2. NO END DATE in code. Hub images are pinned by digest `E/hub/Dockerfile:3`; the ISO helper image is the moving tag at `E/scripts/iso/Dockerfile.assistant:10`; the mailer image is `latest` at `E/manifests/contact-mailer.yaml:54`. +3. Whoever bumps the pin. The nightly security watch runs at 03:40 UTC `E/.gitea/workflows/security-watch.yml:22`; nightly gates at 03:20 `E/.gitea/workflows/gates.yml:27`. +4. A pinned digest removed upstream breaks the hub build for the operator; a moving tag changes the ISO build without a commit. +5. The security watch treats a release line with no fixed version as end of life, `E/scripts/security_watch.py:292`; it reports to the operator. +6. Not a box matter. +7. A CI job that does not run leaves no result; whether a missed nightly is noticed: NOT FOUND in the workflow file `E/.gitea/workflows/gates.yml:27`. +8. SOURCE `E/hub/Dockerfile:3` and `E/scripts/security_watch.py:292`. Support end dates of Go, Alpine, wrangler: no document states one. + +### 27. Domains, registrars and the home dynamic-DNS name +1. NOT MEASURED. The product's two domains and the dynamic name the hub's address points to. Operator. Listed as row S13 `RB/total-loss-of-dooplex.md:46`; the hub name is a CNAME to the dynamic name `RB/total-loss-of-dooplex.md:73`. +2. NOT FOUND: no document gives a registrar name or a renewal date. Searched grep `expir|renew|valid until|until 20` in documentation/operations, RB, AR and STATUS.md. +3. The operator. NOT FOUND how long before. +4. If a domain or the dynamic name lapses, every box loses the hub (reports, alarms, updates, bind links) and the site and installer disappear. Every household. +5. NOT FOUND. Presumably the registrar's own mails to the operator's mailbox, row S12 `RB/total-loss-of-dooplex.md:45`; no document says so. +6. A box comes back by itself once the name resolves again; while it does not, the first failing step is the hub lookup. +7. NOT FOUND. +8. DOC `RB/total-loss-of-dooplex.md:46`. WHERE THE OPERATOR READS THE DATE: NOT FOUND. NOBODY KNOWS from files. + +### 28. Cloudflare account, the site publish token and the ISO store keys +1. NOT MEASURED. The account that serves DNS for both domains, the site, the installer script and the ISO store. Operator. Row S11 `RB/total-loss-of-dooplex.md:44`; site and ISO there since 2026-10-30 `RB/total-loss-of-dooplex.md:125`. +2. NOT FOUND: no end date for the account, the token `CF_PAGES_TOKEN` (named at `E/scripts/publish-site-cloudflare.sh:12`) or any R2 key. Searched grep `R2_ACCESS|R2 key|expir` in AR, RB, operations. +3. The operator at the Cloudflare dashboard. +4. Token lapse: CC cannot publish the site or installer script; the old one stays served. Account trouble: DNS, site, ISO downloads and households' tunnels, per row S11 `RB/total-loss-of-dooplex.md:44`. +5. NOT FOUND. +6. A new box needs the installer script from the site at first boot `E/scripts/iso/felhom-bootstrap.sh:396`; it retries until it is served. +7. A failed publish leaves the previous site live (candidate; not stated in the script header `E/scripts/publish-site-cloudflare.sh:12`). +8. DOC `RB/total-loss-of-dooplex.md:44`. WHERE THE OPERATOR READS THE DATE: NOT FOUND. + +### 29. Hetzner account, the rented endpoint and the storage boxes +1. NOT MEASURED. The paid account behind ep0 (tunnel endpoint and off-site store) and the Storage Boxes. Operator. Row S8 `RB/total-loss-of-dooplex.md:41`; API token named in `E/manifests/hub.yaml:222-226`. +2. NOT FOUND: no document gives a billing date, a contract end or an API token end. Searched grep `expir|renew|invoice|billing` in RB, AR, operations. +3. The operator; paid monthly by the provider's own cycle, not stated in any document read. +4. If the account is closed or unpaid: every household's off-site copies and the PBS tunnel stop; the hub cannot mint or reset sub-accounts. +5. NOT FOUND. Hub-side sign would be the off-site stale alarm after 8 days, `offsiteBackupStaleAfter` `HUB/monitor/deadline_tiers.go:28`. +6. A box comes back by itself when the endpoint is back; first failing step is the off-site push. +7. NOT FOUND. +8. DOC `RB/total-loss-of-dooplex.md:41`. WHERE THE OPERATOR READS THE DATE: NOT FOUND. + +### 30. Tailscale node keys (operator's remote way in) +1. NOT MEASURED. The operator's private network to DooPlex and the two demo boxes. Operator. Table `E/documentation/operations/tailscale.md:11-13`. +2. Document date: the HP demo box's key expires 2027-01-17, written as OPEN at `E/documentation/operations/tailscale.md:41`. The other two have expiry disabled `E/documentation/operations/tailscale.md:11-12`. +3. The operator, in the admin console (Machines, Disable key expiry) `E/documentation/operations/tailscale.md:43`. No lead time set. +4. That node drops off the private network `E/documentation/operations/tailscale.md:91`; the operator loses remote reach to it. No household is on it. +5. NOT FOUND: no reminder is named. WHERE THE OPERATOR READS THE DATE: the Machines page, "Expiry disabled" badge `E/documentation/operations/tailscale.md:92`. +6. The box keeps running; only the operator's remote path is gone until someone re-authenticates on site or over the LAN. +7. NOT FOUND. +8. DOC `E/documentation/operations/tailscale.md:41`. Candidate: the section at `E/documentation/operations/tailscale.md:89-92` says "both" nodes while the table has three. + +### 31. Gitea and registry credentials +1. NOT MEASURED. The login the cluster and the operator use to pull and push images and code. Operator. Row S15 `RB/total-loss-of-dooplex.md:48`; the pull secret is a scoped token `RB/secrets.md:231`. +2. NOT FOUND: no end date for the scoped token or for CC's own Gitea token. Searched grep `expir` in RB/secrets.md and cc-secrets ARCHITECTURE.md. +3. The operator, in Gitea. +4. If the pull token lapses: the hub pod cannot be re-pulled on a restart, and boxes pulling the controller image through the same registry would fail (box helper's item). +5. NOT FOUND. +6. NOT FOUND from these files. +7. NOT FOUND. +8. DOC `RB/secrets.md:231` and `RB/total-loss-of-dooplex.md:48`. WHERE THE OPERATOR READS THE DATE: NOT FOUND. + +### 32. CC's secret store: its key and the tokens inside (GitHub advisory token, hub token) +1. NOT MEASURED. The store CC uses through `sec`, and the key that opens it. Operator owns; row S19 `RB/total-loss-of-dooplex.md:52`; design in /mnt/5_hdd/felhom.eu/git/cc-secrets/ARCHITECTURE.md (lines 47-48 give the key location). +2. NOT FOUND: the design document states no expiry for any stored secret (grep `expir|rotat|days|lifetime`). The hub token has one by code, block 22. A GitHub advisory token is named only in CC's memory index, in no document (grep `GH_ADVISORY_TOKEN` in AR, RB, operations, STATUS). +3. The operator stores a new value; only he may delete one. +4. An expired GitHub token: the nightly security watch `E/.gitea/workflows/security-watch.yml:22` loses its source. An expired hub token: block 22. +5. NOT FOUND. +6. Not a box matter. +7. NOT FOUND. +8. DOC `RB/total-loss-of-dooplex.md:52`. WHERE THE OPERATOR READS THE DATES: NOT FOUND. NOBODY KNOWS from files. + +### 33. SSH keys between the hub, DooPlex and the endpoint +1. NOT MEASURED. Keys the hub uses to reach the endpoint (three key files named at `E/manifests/hub.yaml:191`, `E/manifests/hub.yaml:202`, `E/manifests/hub.yaml:209`) and the root key, row S9 `RB/total-loss-of-dooplex.md:42`. Operator. +2. NO END DATE in any document. The only rule: rotate on an endpoint rebuild, DOC `RB/offsite-endpoint.md:313`. +3. The operator, on rebuild. +4. A lost or changed key: the hub cannot add tunnel peers, mint off-site tokens or poke the endpoint; new households get no off-site tier. +5. Nobody; no date. +6. Existing boxes keep their own tunnel and token; only new provisioning fails. +7. NOT FOUND. +8. DOC `RB/offsite-endpoint.md:313`; row S9 says one authorized key's owner is "not recorded" `RB/total-loss-of-dooplex.md:42`. NOBODY KNOWS who holds it. + +### 34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS) +1. NOT MEASURED. The base systems under every box and the endpoint. Product. Host is Proxmox VE 9.2 on Debian 13, DOC `AR/11-os-updates.md:103`. +2. NOT FOUND: no document states a support end date. Searched grep `EOL|end of life|support ends|until 20` in AR, RB, operations. The major upgrade 9 to 10 is "later" `AR/11-os-updates.md:91` and `AR/11-os-updates.md:725`. +3. The operator, by a drilled major upgrade first on a test box `AR/11-os-updates.md:725`. No date. +4. After the upstream end, boxes stop getting security fixes; every household, silently. +5. NOT FOUND. The security watch flags a runtime line with no fixed version as end of life `E/scripts/security_watch.py:292`; whether Debian or Proxmox lines are in its list was not checked. +6. A box off across a major release: NOT FOUND what its updater does first. +7. NOT FOUND. +8. DOC `AR/11-os-updates.md:103`. The paid repository is a later money decision `AR/11-os-updates.md:662`. WHERE THE OPERATOR READS THE DATE: NOT FOUND. + +## Tiny +- Diagnostic log bundles kept 72 hours: `logBundleTTL` `HUB/store/logbundle.go:29`. +- Hub access log kept 90 days: `DefaultKeep` `HUB/accesslog/accesslog.go:44`. +- Deleted-customer audit rows kept 365 days: `DeletedCustomerAuditKeep` `HUB/store/deleted_customer_audit.go:23`. +- App telemetry pruned after 90 days and stale issues after 30: `E/hub/cmd/hub/main.go:1134-1139`. +- Hub prune runs daily at 04:30: `prune_schedule` `E/manifests/hub.yaml:83`. +- Kernel-reboot notice valid 24 hours: `KernelNoticeValid` `HUB/osupdates/kernel.go:48`. +- Local backup counted stale after 26 hours: `backupStaleAfter` `HUB/monitor/deadline.go:29`. +- Off-site backup counted stale after 8 days: `offsiteBackupStaleAfter` `HUB/monitor/deadline_tiers.go:28`. +- Restore test counted proven for 7 to 12 days: `HUB/monitor/restoretest.go:68-74`. +- Household quiet-box mail after 7 days: `householdLivenessQuiet` `HUB/notify/dispatcher.go:1018`. +- Line-test result stale after 15 days: `lineStaleAfter` `HUB/web/system.go:174`. +- Hub database snapshot catch-up age 6 hours: `CatchUpAge` `HUB/dbsnap/dbsnap.go:36`. +- Escrow still pending after 7 days raises an operator alarm: `escrowPendingAfter` `HUB/monitor/offsite_escrow_pending.go:37`. +- Escrow ceremony: an unclaimed recovery code is void after 10 minutes: `RB/RUNBOOK-escrow-ceremony.md:46`. +- Appliance row badged stale after 7 days without a poll: `applianceStaleAfter` `HUB/web/appliances.go:21`. +- Waiver example in the runbook carries dates from September 2026: `RB/RUNBOOK-manual-build.md:278-279`. +- Hub image tag in the manifest: `E/manifests/hub.yaml:135`. +- Same issuer annotation on the website ingress: `E/manifests/webpage.yaml:141`. + +## Not found +- The definition of the certificate issuer, certificate lifetime and renew-before: searched `ClusterIssuer|renewBefore|duration` in E/manifests; it lives in the DooPlex cluster repository, outside this task. +- Any warning before an operator access token ends: searched `token.*(expiring|expires soon|about to expire|days left)|token_expir` in the hub. +- The end date of CC's hub token in a document: searched `HUB_TOKEN` in AR, RB, documentation/operations and STATUS.md. +- An expiry on the claim code or on the pairing code: searched `expire|ttl|valid` in the claim engine and the appliance store. +- A hub-side expiry on queued signed jobs: searched `expire` around `EnqueueSignedJob`. +- A pinned Proxmox ISO version or checksum in the build script: they are arguments; searched `sha256|pve-iso|9\.[0-9]-[0-9]` in the ISO script and the release gate runbook. +- An "ISO too old" check at first boot: searched `too old|min.*version|ISO_VERSION` in the bootstrap script. +- The waiver file documentation/tests/golden-waiver.yml in this clone (it exists only in sibling worktrees, which are not sources). +- Test names in the golden gate's test file: `def test` returned nothing. +- Registrar names, domain renewal dates, the dynamic-DNS account's renewal: searched `expir|renew|valid until|until 20|No-IP|noip`. +- Billing or contract dates for Hetzner, Cloudflare, the mail service, Tailscale: searched `expir|renew|invoice|billing`. +- Expiry of Gitea tokens, registry logins, R2 keys, the GitHub advisory token, SSH keys: searched `expir|GH_ADVISORY_TOKEN|R2_ACCESS|R2 key` in AR, RB, operations, STATUS.md and the cc-secrets design document. +- Support end dates for Debian 13, Proxmox VE 9 or PBS: searched `EOL|end of life|support ends`. +- Anything dated in STATUS.md: searched `expir|renew|HUB_TOKEN|waiver|trixie|2027|2028|2029|2030`. +- WireGuard peer expiry or rotation in the hub: searched `expire|rotat|stale|handshake` in the hub's wg files. +- Merged: self-bind mint and resend into block 1; escrow retention and abandon into 6; sub-account password and seal key into 8; vouch and manifest into 16; golden age and waiver into 17; CC store, hub token date and GitHub token into 32. diff --git a/documentation/audits/end-dates-list-2026-11-11/evidence/helpers/resolved-dates-box.json b/documentation/audits/end-dates-list-2026-11-11/evidence/helpers/resolved-dates-box.json new file mode 100644 index 00000000..549089c8 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/evidence/helpers/resolved-dates-box.json @@ -0,0 +1,1864 @@ +{ + "endings": [ + { + "title": "1. Web certificate of the household's address (Let's Encrypt through Traefik)", + "answers": { + "1": "The public TLS certificate for the dashboard and apps. Product-owned Traefik asks Let's Encrypt with the household's e-mail `ACMEEmail` `CT/internal/stacks/infra.go:130`; stored in `acme.json` `CT/internal/infra/templates/traefik.yml.tmpl:51`.", + "2": "The lifetime is the issuer's, not ours: no validity constant in our source. Resolver `letsencrypt` `CT/internal/infra/templates/traefik.yml.tmpl:48-59`. Lifetime value: NOT FOUND (grep `NotAfter|renew|90 ?days` in CT/internal/infra).", + "3": "Traefik renews by itself; we configure only the challenge: `dnsChallenge` with a Cloudflare token `CT/internal/infra/templates/traefik.yml.tmpl:53`, else `httpChallenge` `CT/internal/infra/templates/traefik.yml.tmpl:59`. How long before the end: NOT FOUND in our source (Traefik's own default).", + "4": "Browsers show a certificate error on every app address of the household. With no e-mail set Traefik serves a self-signed certificate, see comment at `TraefikData` `CT/internal/infra/infra.go:111-118`.", + "5": "NOT FOUND: no expiry check or warning in the controller (grep `cert.*expir|expir.*cert|NotAfter|acme\\.json` in CT/internal hit only `CT/internal/stacks/infra.go:160-168` and the mail relay). The issuer mails `ACMEEmail`, which is the household.", + "6": "NOBODY KNOWS from our source: renewal on start is Traefik's behaviour. The file survives because it is created once, mode 0600, `CT/internal/stacks/infra.go:160-168`. The DNS route also needs the Cloudflare token `CFAPIToken` `CT/internal/infra/infra.go:116-118` to be still valid.", + "7": "NOT FOUND in our source: retry is inside Traefik. The old certificate stays in `acme.json` `CT/internal/infra/templates/traefik.yml.tmpl:51` until it ends. Nothing in the controller reads the result (same grep as line 5).", + "8": "SOURCE `CT/internal/infra/templates/traefik.yml.tmpl:48-59` for the setup; NOBODY KNOWS for renewal timing, warning and the switched-off case." + }, + "extra": [] + }, + { + "title": "2. Proxmox web certificate pinned by the agent", + "answers": { + "1": "The agent talks to Proxmox on the host and trusts ONE certificate by fingerprint. The installer reads the served leaf once, `fp=` `E/scripts/felhom-host-install.sh:3182-3185`; the agent checks it in `TLSConfig.build` `AG/internal/proxmox/tls.go:39-59`. Product-owned.", + "2": "The pin has NO END DATE: the date of the certificate is never looked at, `VerifyPeerCertificate` `AG/internal/proxmox/tls.go:49`. The end is the day Proxmox serves a different leaf. Proxmox's own certificate lifetime: NOT FOUND (grep `pve-ssl|updatecerts|pveproxy cert` in `AG/`, E/scripts, `AR/`, `RB/`).", + "3": "NOT FOUND: no code re-reads the fingerprint (grep `TLS pin mismatch|tls\\.fingerprint|\\.Fingerprint =` in AG/internal). The pin is loaded from config `AG/internal/config/config.go:917` and used at start `AG/cmd/felhom-agent/main.go:340`.", + "4": "Candidate: if Proxmox replaces its certificate, every agent call to Proxmox fails with the mismatch error `AG/internal/proxmox/tls.go:55`: no backup, restore, guest start or resize through the agent. Household and operator both.", + "5": "NOT FOUND: no warning before (same greps as line 2 and 3). After the fact the calls fail; which alarm fires was not traced.", + "6": "NOBODY KNOWS: depends on whether Proxmox regenerates its certificate at boot, which is not in our source. If it does, the first failing step is the first Proxmox API call after start, `AG/internal/proxmox/tls.go:55`.", + "7": "The agent keeps the old pin and keeps failing closed, `AG/internal/proxmox/tls.go:55`. Repair path found: none in code. An empty pin falls back to system trust, warning at `E/scripts/felhom-host-install.sh:3185`.", + "8": "TEST `TestTLSConfig_Build` `AG/internal/proxmox/tls_test.go:28` for the pin check; NOBODY KNOWS for the renewal case." + }, + "extra": [] + }, + { + "title": "3. Agent local-API certificate and its pin in the guest", + "answers": { + "1": "The self-signed leaf the agent serves to the in-guest controller, made by `generateLeaf` `AG/internal/localapi/cert.go:56`. The controller pins its fingerprint in `New` `CT/internal/agentapi/client.go:105`. Product-owned, on the host.", + "2": "`certValidity = 10 * 365 * 24 * time.Hour` `AG/internal/localapi/cert.go:25`, set as `NotAfter` `AG/internal/localapi/cert.go:70`. So a box installed in 2026 has a leaf that ends in 2036.", + "3": "Nobody: `EnsureLeaf` loads the existing pair and never looks at its date `AG/internal/localapi/cert.go:38-52`. The comment says rotation is an operator action for a later slice `AG/internal/localapi/cert.go:21-24`.", + "4": "Candidate: at the end date nothing breaks in the controller, because it sets `InsecureSkipVerify` and compares only the fingerprint `CT/internal/agentapi/client.go:118-128`. What breaks things is a NEW leaf: mismatch `CT/internal/agentapi/client.go:125`.", + "5": "Nobody. NOT FOUND: no expiry reading of this leaf (grep `NotAfter` in AG/internal hit only `AG/internal/localapi/cert.go:70`).", + "6": "Yes, by itself: the leaf is generated once and kept, `AR/03-host-agent.md:420-423`, and the reinstall path keeps it with `--preserve-state-from` `E/scripts/felhom-host-install.sh:2749-2786`.", + "7": "No renewal exists, so no failure state. A reinstall that makes a new leaf needs `--allow-new-leaf` `E/scripts/felhom-host-install.sh:2749-2786` and then the guest's pin is stale.", + "8": "TEST `TestEnsureLeaf_StableFingerprintAcrossReload` `AG/internal/localapi/cert_test.go:12` and `TestClient_WrongPinFailsClosed` `CT/internal/agentapi/client_test.go:55`; the 10-year end itself: SOURCE only." + }, + "extra": [] + }, + { + "title": "4. Off-site copy after the household stops it: 14-day countdown", + "answers": { + "1": "When the household abandons the off-site backup, the copies are kept for a grace period and then deleted. `startAbandonCountdown` `CT/internal/backup/offbox_abandon.go:68-70`. The household starts it; the product runs it.", + "2": "`abandonGraceDays = 14` `CT/internal/backup/offbox_abandon.go:31`.", + "3": "The household can stop or extend it; extending never shortens, test `TestD1_ExtendNeverShortensTheCountdown` `CT/internal/backup/offbox_abandon_levers_d1_test.go:20`.", + "4": "The off-site copies are gone for good; the household loses its only copy outside the house. The deleting side is the hub (see the hub helper); test name `TestD1_ExtendRefusedWhileTheDeletionIsTheHubs` `CT/internal/backup/offbox_abandon_levers_d1_test.go:44`.", + "5": "The household, at 5, 3 and 1 days left: `AbandonRemindAtDays` `CT/internal/backup/offbox_abandon.go:40`.", + "6": "NOT FOUND for a box that is off during the whole countdown: whether the hub deletes without the box's reminders was not traced on the box side (grep `abandonNow|AbandonRemindAtDays` in CT/internal/backup).", + "7": "A failed hub cancel is reported, not hidden: `TestD1_StopReportsFailureWhenTheHubCancelFails` `CT/internal/backup/offbox_abandon_levers_d1_test.go:69`.", + "8": "TEST `TestR241_ScenarioE_AbandonStartsAReversibleCountdown` `CT/internal/backup/offbox_abandon_r241_test.go:48`." + }, + "extra": [] + }, + { + "title": "5. Off-site backup history: how far back a household can go", + "answers": { + "1": "The restic off-site store keeps a thinning history. Policy in `retentionPolicy` `CT/internal/backup/offbox_window.go:81-82`. Product rule, household's data.", + "2": "`keepDaily = 7` `CT/internal/backup/offbox_window.go:47`, `keepWeekly = 4` `CT/internal/backup/offbox_window.go:48`, `keepMonthly = 6` `CT/internal/backup/offbox_window.go:49`. Older than about six months: gone.", + "3": "Nobody renews; each run forgets by the same rule, `retentionPolicy` `CT/internal/backup/offbox_window.go:81-82`.", + "4": "A file deleted or damaged more than six monthly points ago cannot be brought back from off-site. On-host copies are fewer: `KeepLast` `AG/internal/config/config.go:436`, default 3 per comment `AG/internal/config/config.go:409`.", + "5": "NOT FOUND: no warning that a point is about to leave the window (grep `keepMonthly|about to|will be removed` in CT/internal/backup).", + "6": "Yes. A box that was off simply has no points for that time; the next run applies the same rule. PBS copies are not pruned by the box by default, comment `AG/internal/backup/runner.go:51-58`.", + "7": "NOT FOUND for a failed forget (grep `forget` failure handling not traced). The weekly content check has its own age `defaultIntegrityMaxAgeDays = 7` `CT/internal/backup/offbox_integrity.go:81`.", + "8": "SOURCE `CT/internal/backup/offbox_window.go:47-49`." + }, + "extra": [] + }, + { + "title": "6. Off-site credentials: SSH key, host key, store password, PBS pin", + "answers": { + "1": "The box reaches its off-site store with an SSH key and a pinned host key, `StrictHostKeyChecking=yes` `CT/internal/backup/offbox.go:345`, and a password file `RESTIC_PASSWORD_FILE` `CT/internal/backup/offbox.go:898`. PBS is pinned in `pinnedTLS` `AG/internal/pbs/pin.go:17-30`.", + "2": "NO END DATE in box code: no expiry field on the key, the host key or the password (grep `expir` in `CT/internal/backup/offbox.go` found none). The PBS pin ignores the date, `VerifyPeerCertificate` `AG/internal/pbs/pin.go:24`. The end comes from outside: the provider changes its host key or the hub ends the account.", + "3": "Nobody on a schedule. A new PBS descriptor is checked before use: `probeFP` `AG/internal/pbsdr/manager.go:375-379`.", + "4": "If the provider's host key changes, every off-site run fails at the SSH step, `StrictHostKeyChecking=yes` `CT/internal/backup/offbox.go:895`. The household has no off-site copy from that night on.", + "5": "NOT FOUND before the end. After it, the failed run is the signal (alarm path not traced here).", + "6": "Yes if nothing changed outside. A rebuilt guest gets a fresh store password and cannot open the old store: NOT FOUND in this pass (grep `RESTIC_PASSWORD_FILE` `CT/internal/backup/offbox.go:910` shows use only).", + "7": "PBS: state `verify_failed` and a retry at the next tick, `probeFP` `AG/internal/pbsdr/manager.go:375-379`. Restic: the run fails and the next night tries again with the same pins.", + "8": "SOURCE `CT/internal/backup/offbox.go:345`; provider-side dates: NOBODY KNOWS from the box." + }, + "extra": [] + }, + { + "title": "7. Sign-in sessions and cookies (dashboard, setup gate, family, share) \u2014 merged", + "answers": { + "1": "Merged: dashboard session, setup-gate pass, family session, share-guest cookie. All product-made, in the household's browser and the controller's store, `loadSessions` `CT/internal/web/session_store.go:73`.", + "2": "`sessionMaxAge = 7 * 24 * time.Hour` `CT/internal/web/auth.go:30`; `gateCookieLife = 7 * 24 * time.Hour` `CT/internal/web/setup_gate.go:46`; `SessionLife = 30 * 24 * time.Hour` `CT/internal/family/family.go:42`; `shareCookieMaxAge = 30 * 24 * time.Hour` `CT/internal/web/share.go:31`.", + "3": "The family session renews itself at half-life `CT/internal/family/family.go:506-514`. The dashboard session has a fixed end set at sign-in `CT/internal/web/auth.go:308`.", + "4": "The person is asked to sign in again. Nothing is lost.", + "5": "Nobody; no warning is needed.", + "6": "Yes. Expired rows are dropped on load `CT/internal/web/session_store.go:105` and by `cleanupSessions` `CT/internal/web/auth.go:369`; the person signs in again.", + "7": "No renewal to fail.", + "8": "TEST `TestR35_ExpiredRowNotLoaded` `CT/internal/web/session_store_test.go:144` and `TestSetupGate_ARestartKeepsTheHouseholdsPass` `CT/internal/web/setup_gate_test.go:182`." + }, + "extra": [] + }, + { + "title": "8. Claim code and the one-use gate token", + "answers": { + "1": "The one-time code a household uses to set its own password, and the short token that carries a pass between addresses. Product-made. `claimCodeTTL` `CT/internal/web/claim.go:30`; `gateTokenLife` `CT/internal/web/setup_gate.go:47`.", + "2": "`claimCodeTTL = 72 * time.Hour` `CT/internal/web/claim.go:30`; `gateTokenLife = 60 * time.Second` `CT/internal/web/setup_gate.go:47`, also stated in `AR/01-topology-and-trust.md:192`.", + "3": "The operator sends a new code (hub side, see the hub helper). Nothing on the box mints one by itself.", + "4": "A household that opens the mail after three days cannot set its password until a new code arrives. Checked against the controller's clock `s.claimNow()` `CT/internal/web/claim.go:333`.", + "5": "NOT FOUND: no reminder before the 72 hours end (grep `claimCodeTTL` in CT/internal/web).", + "6": "A box off for more than 72 hours comes back with a dead code; first failing step is the claim form, `CT/internal/web/claim.go:333`.", + "7": "Wrong tries lock the form for a time: `claimLockoutWindow` `CT/internal/web/claim.go:33`.", + "8": "TEST `TestClaimSubmit_ExpiredCodeRefused` `CT/internal/web/claim_test.go:189`." + }, + "extra": [] + }, + { + "title": "9. Share link `/s/`", + "answers": { + "1": "The link a household gives to guests to see its app tiles. Household-owned, stored as `LauncherShareToken` `CT/internal/settings/settings.go:45`.", + "2": "NO END DATE: the token has no expiry field; it ends only when the household rotates or disables it, `SetLauncherShareToken` `CT/internal/settings/settings.go:1111`.", + "3": "The household, by hand. No schedule.", + "4": "Nothing ends. Candidate for the list: a link given away years ago still opens the page. The guest's cookie lasts `shareCookieMaxAge` `CT/internal/web/share.go:31`.", + "5": "Nobody; NOT FOUND any reminder of an old link (grep `LauncherShareToken` in CT/internal).", + "6": "Yes, unchanged.", + "7": "No renewal to fail.", + "8": "TEST `TestShareGuest_RotateAndDisable` `CT/internal/web/share_test.go:256`." + }, + "extra": [] + }, + { + "title": "10. Recovery-code ceremony windows", + "answers": { + "1": "The steps where the household confirms it wrote down its recovery code. Product-made, held by the agent. `escrowConfirmTTL` `AG/internal/localapi/escrow_ceremony.go:158`.", + "2": "`escrowConfirmTTL = 30 * time.Minute` `AG/internal/localapi/escrow_ceremony.go:158`; `escrowClaimTTL = 10 * time.Minute` `AG/internal/localapi/escrow_ceremony.go:166`; the page allows `escrowCeremonyGraceWindow` `CT/internal/web/handlers.go:1169`.", + "3": "The household starts the ceremony again.", + "4": "A household that walks away mid-step must start over. The written-down code itself: NO END DATE found (grep `recovery.*expir|expir.*recovery` in AG/internal, CT/internal hit only the unclaimed-ceremony refusal `AG/internal/localapi/escrow_ceremony.go:506`).", + "5": "Nobody.", + "6": "Yes; an unfinished ceremony is simply started again.", + "7": "A ceremony that timed out answers \"no longer available\" (HTTP 410) `AG/internal/localapi/escrow_ceremony.go:506`; the comment says an abandoned one leaves the household's written-down code as it was, `AG/internal/localapi/escrow_ceremony.go:38`. No automatic retry found.", + "8": "SOURCE `AG/internal/localapi/escrow_ceremony.go:158-166`." + }, + "extra": [] + }, + { + "title": "11. Signed operator jobs: the validity window", + "answers": { + "1": "An order signed by the operator (update the agent, apply OS packages, destructive acts). Operator-owned; it carries a nonce and an expiry, `AR/03-host-agent.md:268`. Checked by `Verify` `AG/internal/authz/verifier.go:104` and by the root wrapper.", + "2": "At most `MaxWindow = 24 * time.Hour` `AG/internal/authz/verifier.go:26`; the signer's default is 30 minutes, `-ttl` `AG/cmd/felhom-opsign/main.go:57`. Wrapper: `MAX_WINDOW_S = 24 * 3600` AG/configs/felhom-os-apply:194. Design: `AR/04-control-plane-authorization.md:78-84`.", + "3": "The operator signs a new one. There is no renewal of a signed job.", + "4": "The job is refused and cleared; the executor is never called, `AG/internal/signedjobs/runner.go:163-170`. The box stays on its old agent or old packages. `ErrExpired` `AG/internal/authz/verifier.go:161-162`.", + "5": "NOT FOUND a warning before a job ends. The design says an order unanswered after 24 hours is expired, `AR/04-control-plane-authorization.md:184`.", + "6": "A box off for longer than the window never runs that job; it is checked against the BOX's clock, `time.Now().UTC()` `AG/internal/authz/verifier.go:90`, and in the wrapper `if now > exp or now < iss - 120` AG/configs/felhom-os-apply:676. A wrong box clock decides.", + "7": "The refused job is logged as a security event and removed, `AG/internal/signedjobs/runner.go:163-170`; no retry. Used nonces are kept until their end, comment `AG/internal/authz/noncestore.go:47-50`.", + "8": "TEST `test_expired_signed_job_is_refused` `AG/configs/test_felhom_os_apply.py:833`, `test_a_window_longer_than_24h_is_refused` `AG/configs/test_felhom_os_apply.py:836`, `TestVerify_RejectsPerStage` `AG/internal/authz/verify_test.go:82`." + }, + "extra": [] + }, + { + "title": "12. Operator signing keys on each box", + "answers": { + "1": "The two public keys every box trusts for signed jobs: `OPERATOR_KEY_OPERATIONAL_ID` `E/scripts/felhom-host-install.sh:212` and `OPERATOR_KEY_RECOVERY_ID` `E/scripts/felhom-host-install.sh:214`. Written to `TRUST_SIGNERS` AG/configs/felhom-os-apply:124 by `_write_slow_lane_trust` `E/scripts/felhom-host-install.sh:3424`. Operator-owned.", + "2": "NO END DATE: the keys carry no expiry; trust is by key material, `AG/internal/authz/verifier.go:122-126`. The key id is advisory, `AG/internal/authz/verifier.go:170-175`.", + "3": "Design only: every re-pin is itself a signed order, `AR/04-control-plane-authorization.md:139-150`. In code only the class name exists, `ClassKeyRotation` `AG/internal/reconcile/classify.go:41`.", + "4": "If the operator loses both private keys, no box can be updated or repaired by a signed job again; households keep running on what they have. Design text `AR/04-control-plane-authorization.md:139-150`.", + "5": "Nobody; there is no date to warn about.", + "6": "Yes, the keys are in the installer and on disk, `E/scripts/felhom-host-install.sh:3452-3453`.", + "7": "NOT FOUND: an executor for key rotation (grep `OpRepin|repin_key|key_rotation|OpKeyRotat|rotate_operator` in `AG/` hit only `AG/internal/reconcile/classify.go:41`). Candidate: rotation is designed, not built.", + "8": "DOC `AR/04-control-plane-authorization.md:139-150`; for a real rotation: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "13. Proxmox API token of the agent", + "answers": { + "1": "The token the agent uses for every Proxmox call. Made by the installer in `step_token` `E/scripts/felhom-host-install.sh:2405`. Product-owned, on the host.", + "2": "NO END DATE: minted with `pveum user token add ... --privsep 1` and no expire option, `E/scripts/felhom-host-install.sh:2448`.", + "3": "Nobody on a schedule. A re-run of the installer reuses it or makes a new one, `E/scripts/felhom-host-install.sh:2424-2445`.", + "4": "Nothing ends by date. If someone sets an expiry by hand in Proxmox, every agent call fails that day.", + "5": "Nobody; NOT FOUND any check of the token's expiry (grep `expire` in `E/scripts/felhom-host-install.sh` near `step_token` found none).", + "6": "Yes.", + "7": "No renewal to fail.", + "8": "SOURCE `E/scripts/felhom-host-install.sh:2448`." + }, + "extra": [] + }, + { + "title": "14. Guest token for the agent and the box's hub key \u2014 merged", + "answers": { + "1": "Merged: the per-guest token the controller shows to the agent, kept as a hash in `TokenStore` `AG/internal/localapi/tokenstore.go:24-33`, and the agent's hub key used in `NewClient` `AG/internal/hub/client.go:60`. Product-owned.", + "2": "NO END DATE for both: no expiry field in `TokenStore` `AG/internal/localapi/tokenstore.go:24-33`; none seen in `NewClient` `AG/internal/hub/client.go:36`.", + "3": "Nobody on a schedule; a new guest token replaces the old one for that guest (last write wins), `AG/internal/localapi/tokenstore.go:24-33`.", + "4": "Nothing ends by date. The hub connection also depends on public root certificates of the host's system store, `AG/internal/hub/client.go:41`, which age with the OS.", + "5": "Nobody.", + "6": "Yes; first step that can fail after years off is the TLS handshake to the hub if the host's root store is too old, `AG/internal/hub/client.go:41`. Candidate, not tested.", + "7": "No renewal to fail.", + "8": "SOURCE `AG/internal/localapi/tokenstore.go:24-33`; old root store: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "15. WireGuard key of the operator tunnel", + "answers": { + "1": "The box's key for the tunnel the operator uses to reach it. Product-owned, on the host. `rotateKey` `AG/internal/wgtunnel/rekey.go:110`.", + "2": "NO END DATE and no periodic rotation: a new key is made only when the handshake time sits in the future for `rekeyAfterDefault = 15 * time.Minute` `AG/internal/wgtunnel/rekey.go:36`. Reason in the comment `AG/internal/wgtunnel/rekey.go:16-30`.", + "3": "The agent by itself, at most once per `rekeyMinGap = time.Hour` `AG/internal/wgtunnel/rekey.go:38`; the old key is kept as `prevKeyName` `AG/internal/wgtunnel/rekey.go:39`.", + "4": "Without the re-key, a box whose clock once ran ahead has a tunnel that never connects: the operator cannot reach it. The household notices nothing.", + "5": "Nobody before; it is a repair, not a date.", + "6": "Yes. This is exactly the case the re-key is for (a clock that jumped), `AG/internal/wgtunnel/rekey.go:16-30`.", + "7": "A failed registration is retried by the existing path: `TestD30_AFailedRegistrationIsRetriedByTheExistingPath` `AG/internal/wgtunnel/rekey_test.go:119`.", + "8": "TEST `TestD30_AFutureHandshakeThatStaysReKeys` `AG/internal/wgtunnel/rekey_test.go:35`, `TestD30_TheHourBoundSurvivesARestart` `AG/internal/wgtunnel/rekey_test.go:182`." + }, + "extra": [] + }, + { + "title": "16. Package signing keys: Docker in the guest, Proxmox and Debian on the host", + "answers": { + "1": "The keys apt uses to trust packages. Docker's key is fetched once at golden bake into `/etc/apt/keyrings/docker.asc` `AG/configs/build-golden.sh:129`. Proxmox's keyring path default is in `align_apt_repos` `E/scripts/felhom-host-install.sh:2616`. Owned by Docker, Proxmox, Debian.", + "2": "NOT FOUND in our source: the keys' own end dates (grep `expire|valid-until|keyring` in `AG/configs/build-golden.sh` and `E/scripts/felhom-host-install.sh`). The end is set by the publisher.", + "3": "Debian and Proxmox keys arrive as packages through the normal update. Docker's key has no package: a change is \"a slow-lane act for a person\", `AR/11-os-updates.md:657`.", + "4": "The OS update is refused with code R7 when `apt-get update` fails, AG/configs/felhom-os-apply:1125-1127. The box keeps running but stops getting Docker security updates. Household unaware; operator sees the report.", + "5": "Nobody before the end. After it the box reports the failed update, `AR/11-os-updates.md:657`.", + "6": "A box off for years comes back with old keys; the first failing step is `apt-get update` in the wrapper, AG/configs/felhom-os-apply:1125-1127. Whether the old key still verifies: NOBODY KNOWS.", + "7": "The update fails cleanly and nothing is half-installed, `AR/11-os-updates.md:657`; the next signed job tries again and fails the same way until a person replaces the key.", + "8": "DOC `AR/11-os-updates.md:657`; TEST for the refusal `test_R7_not_downloadable_and_no_snapshot` `AG/configs/test_felhom_os_apply.py:488`; key dates: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "17. Debian 13 (trixie) and Proxmox 9: the series the product is tied to", + "answers": { + "1": "The OS series under everything. Written into source: suite default `trixie` `E/scripts/felhom-host-install.sh:2615`, Docker repo `trixie stable` `AG/configs/build-golden.sh:130`, base images CT/Dockerfile:52. Owned by Debian and Proxmox.", + "2": "NOT FOUND in source: the end-of-support dates of trixie and Proxmox 9 (grep `end of life|EOL|end-of-life|bookworm|forky` in `AR/11-os-updates.md` and E/scripts). The step to Proxmox 10 is named as later work, `AR/11-os-updates.md:725`.", + "3": "The operator and the product; no date-driven plan found in the documents read.", + "4": "After the series ends: no security updates for host and guest; a new install on a newer Proxmox is refused in byo mode and only warned in appliance mode, `E/scripts/felhom-host-install.sh:2146-2152`.", + "5": "NOT FOUND (same grep as line 2).", + "6": "Yes while the archives still serve trixie. Pinned OS updates fall back to snapshot.debian.org in `add_snapshot_sources` AG/configs/felhom-os-apply:1451-1462; if that source stops, the update is refused.", + "7": "Refused update, box unchanged, AG/configs/felhom-os-apply:1125-1127.", + "8": "SOURCE `E/scripts/felhom-host-install.sh:2146-2152`; the dates: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "18. Catalog app images and their recorded digests", + "answers": { + "1": "65 app templates under CAT/templates/; 84 different image references; registries docker.io 47, ghcr.io 27, lscr.io 8, gitea.dooplex.hu 1, registry.gitlab.com 1. The update ladders record 149 different digests, e.g. `CAT/templates/actualbudget/.felhom.yml:196`. Owned by each app's publisher.", + "2": "NO END DATE in source. The end is the day a registry stops serving a tag or a digest. Moving tags are banned: `BANNED_TAGS` `CAT/scripts/check-image-pins.py:28`.", + "3": "The product, when it tests a new version and records its digest. `CAT/scripts/check-currency.py:22` reports how far behind a pin is and gates nothing.", + "4": "The box renders `tag@sha256` in `RenderWithLadderDigests` `CT/internal/stacks/digest.go:94`; a digest the registry no longer serves makes the pull fail on every box, `AR/09-update-architecture.md:391`. A household cannot install or update that app; a running app keeps running.", + "5": "NOT FOUND: a check that every recorded digest is still served (grep `manifest inspect|still served|HEAD` in CAT/scripts). `CAT/scripts/check-currency.py:22` reads tags, not old digests.", + "6": "Running apps come back (images are local). The first failing step is the next install, update or restore that must pull, `CT/internal/stacks/digest.go:10-23`.", + "7": "The pull fails and the app stays on its current image; no automatic repair. Retry happens only when the household presses again.", + "8": "DOC `AR/09-update-architecture.md:391`; SOURCE `CT/internal/stacks/digest.go:94`; which digests are still served today: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "19. Images the controller itself starts (Traefik, tunnel, file browser, Samba)", + "answers": { + "1": "Four fixed images in the controller binary: `TraefikImage` `CT/internal/infra/infra.go:33`, `CloudflaredImage` `CT/internal/infra/infra.go:34`, `FileBrowserImage` `CT/internal/infra/infra.go:35`, `SambaImage` `CT/internal/infra/infra.go:50`. Product-owned pins of other people's images.", + "2": "NO END DATE in source; tags only, no digest. Cloudflare's support window for an old cloudflared: NOT FOUND (grep `one year|1 year|12 months|support.*cloudflared` in `AR/01-topology-and-trust.md`).", + "3": "The product, by a controller release that names newer tags.", + "4": "Candidate: if Cloudflare stops accepting an old tunnel client, the household's address stops answering from outside though the box is healthy. If a registry drops a tag, a rebuilt box cannot start that part.", + "5": "NOT FOUND (same grep as line 2).", + "6": "A box off for a long time comes back with the old images it already holds; whether the tunnel still connects with an old client: NOBODY KNOWS.", + "7": "Not traced.", + "8": "SOURCE `CT/internal/infra/infra.go:33-50`; NOBODY KNOWS for the tunnel client's age limit." + }, + "extra": [] + }, + { + "title": "20. Published agent versions: only the newest five stay downloadable", + "answers": { + "1": "The registry of agent binaries. Rule file `generic_versions_kept` `AG/scripts/retention-policy.json:23`. Operator-owned; he prunes by hand, `AG/scripts/retention-policy.json:7-10`.", + "2": "Newest 5 versions plus two pins: `hub_min_agent` `AG/scripts/retention-policy.json:30` and `day0_agent` `AG/scripts/retention-policy.json:31`. Decision D50, `AR/09-update-architecture.md:1370-1375`.", + "3": "The session that records a vouch or a new floor updates the pins in the same commit, `AG/scripts/retention-policy.json:12-15`.", + "4": "A box told to install a version that was pruned cannot download it. A pin that names a missing version is a CI failure, `AG/scripts/retention-policy.json:14-15`. Deleting a git tag also ends the ability to install that version, `AG/scripts/retention-policy.json:20-21`.", + "5": "The operator, through CI: `AG/scripts/retention-policy.json:25`.", + "6": "A box off for long is on an old agent; it updates only by a fresh signed job that names a version still kept. First failing step otherwise: the download in the wrapper, `SELFUPDATE_OP` AG/configs/felhom-os-apply:180.", + "7": "Not traced on a failed download.", + "8": "SOURCE `AG/scripts/retention-policy.json:23-31`; DOC `AR/09-update-architecture.md:1370-1375`." + }, + "extra": [] + }, + { + "title": "21. Golden image age and the waiver file", + "answers": { + "1": "The ready-made guest image a new box is installed from, baked weekly, `AR/09-update-architecture.md:1010-1018`. Operator vouches. Gate: `E/scripts/golden_currency_gate.py:169`.", + "2": "`GOLDEN_MAX_AGE_DAYS = 7` `E/scripts/golden_currency_gate.py:169`. A waiver may span at most `WAIVER_MAX_DAYS = 14` `E/scripts/golden_currency_gate.py:167`, checked in `read_waiver` `E/scripts/golden_currency_gate.py:329-333`; path `WAIVER_PATH` `E/scripts/golden_currency_gate.py:162-163`.", + "3": "The operator, weekly (Sunday night per `AR/09-update-architecture.md:1010-1018`).", + "4": "The gate goes red for the product team; no household is touched. A new install gets an older image and must update more on first start.", + "5": "The operator, by the gate itself; no earlier warning found.", + "6": "Not about a box.", + "7": "A waiver that has run out is no longer honoured, `E/scripts/golden_currency_gate.py:329-333`; no waiver file exists in the clones today (searched by file name `golden-waiver`).", + "8": "SOURCE `E/scripts/golden_currency_gate.py:162-169`; tests exist in `E/scripts/test_golden_currency_gate.py:1` (names not matched to this rule)." + }, + "extra": [] + }, + { + "title": "22. Debian container template the golden is built from", + "answers": { + "1": "The upstream Debian LXC template: `TEMPLATE` default `debian-13-standard_13.1-2` `AG/configs/build-golden.sh:67`, used in `pct create` `AG/configs/build-golden.sh:101`. Owned by Proxmox's template mirror.", + "2": "NO END DATE in source; the end is the day the mirror drops this exact file name.", + "3": "The operator, by changing the default at a bake.", + "4": "The weekly bake fails; no household is touched until a fresh golden is needed.", + "5": "NOT FOUND (grep `pveam|TEMPLATE` in `AG/configs/build-golden.sh` shows only lines 23, 67, 79, 101: the script does not download or check the template).", + "6": "Not about a box.", + "7": "The bake stops at `pct create` `AG/configs/build-golden.sh:101`.", + "8": "SOURCE `AG/configs/build-golden.sh:67`." + }, + "extra": [] + }, + { + "title": "23. Build-time pins of the controller image", + "answers": { + "1": "What the controller image is built from: Go image CT/Dockerfile:11, Debian base CT/Dockerfile:52, restic `RESTIC_VERSION` CT/Dockerfile:55 with checksums CT/Dockerfile:56-57 and a vendored release key CT/Dockerfile:58, Docker's apt key fetched at build CT/Dockerfile:110. Product-owned.", + "2": "NO END DATE in source. The base images are pinned by digest, so the end is the day the registry stops serving that digest. The vendored restic key's own expiry: NOT FOUND (not opened; key files are not read by this helper).", + "3": "The product, at a release.", + "4": "A new controller release cannot be built; boxes keep the image they have. A version number is pushed once, probe at `CT/build.sh:113-125`.", + "5": "Nobody before; the build fails.", + "6": "Not about a box.", + "7": "The build fails with nothing pushed.", + "8": "SOURCE CT/Dockerfile:52-58." + }, + "extra": [] + }, + { + "title": "24. Kept copies made before a restore: deleted after 30 unused days", + "answers": { + "1": "The copy of an app's data the box keeps when the household restores an older state. Household's data, product rule D62. `restoreCopyUnusedAge` `CT/internal/backup/restore_copy_sweep.go:29`.", + "2": "`restoreCopyUnusedAge = 30 * 24 * time.Hour` `CT/internal/backup/restore_copy_sweep.go:29`.", + "3": "Nobody; using the copy is what keeps it.", + "4": "The household loses the way back to the state from before the restore.", + "5": "NOT FOUND a warning before the sweep (grep `restoreCopyUnusedAge` in CT/internal).", + "6": "Candidate: a box off for more than 30 days deletes the copy at the first sweep after start, since age is counted by the wall clock, `CT/internal/backup/restore_copy_sweep.go:29`. A clock tolerance exists elsewhere: `KeptClockSkew` `CT/internal/backup/kept_load.go:217`.", + "7": "Not traced.", + "8": "SOURCE `CT/internal/backup/restore_copy_sweep.go:29`. \u00b7 Mail relay's in-memory self-signed leaf, 10 years, remade at each start: `CT/internal/mailrelay/server.go:386`. \u00b7 Language cookie, 365 days: `CT/internal/web/i18n_web.go:478`. \u00b7 HSTS max-age one day: `HSTSMaxAgeSeconds` `CT/internal/infra/infra.go:108`. \u00b7 A backup counts as fresh enough for an update for 24 hours: `DefaultUpdateBackupMaxAge` `CT/internal/config/config.go:70`. \u00b7 Night-backup banner turns stale after 26 hours: `StaleAfter` `CT/internal/nightchain/banner.go:23`. \u00b7 Operator action brake 30 minutes and forget after 24 hours: `opBrakeRefusal` `AG/internal/localapi/opactions.go:42`. \u00b7 Backup-cap refusal held 24 hours: `backupCapRefusalHold` `AG/internal/localapi/backup_cap.go:38`. \u00b7 Reboot notice shown 24 hours: `opRebootNoticeFor` `AG/internal/localapi/opreboot.go:48`. \u00b7 Controller floor is kept in memory only: `floor` `CT/internal/selfupdate/updater.go:71`. \u00b7 Catalog sync uses a git user and token from config, no end date in code: `CT/internal/sync/sync.go:399`. \u00b7 Operator SSH key file comes from the hub's desired state, no end date seen: `AuthKeysUserPath` `AG/internal/felhomsshd/manager.go:20`. \u00b7 Kernel pin file for host updates: `KERNEL_PIN_FILE` AG/configs/felhom-os-apply:151. \u00b7 Proxmox's own certificate renewal and any re-pin after it: grep `pve-ssl|pveproxy cert|updatecerts|certificate.*(renew|expire)` in `AG/`, E/scripts, `AR/`, `RB/`. \u00b7 A warning before the web certificate ends: grep `cert.*expir|expir.*cert|NotAfter|acme\\.json` in CT/internal. \u00b7 Cloudflare origin certificates: no use found; the box uses ACME only (grep `origin.?cert|origin_ca` in CT/internal, AG/internal). \u00b7 An executor for operator-key rotation: grep `OpRepin|repin_key|key_rotation|OpKeyRotat|rotate_operator` in `AG/`. \u00b7 End dates of the Docker, Debian and Proxmox apt keys, and of trixie and Proxmox 9: grep `expire|valid-until|EOL|end of life` in AG/configs, E/scripts, `AR/11-os-updates.md`. \u00b7 The PBS server certificate's validity: the pin ignores the date, `AG/internal/pbs/pin.go:24`. \u00b7 Retained (superseded) escrow keys' lifetime on the box side: grep `retain.*key|superseded` in AG/internal, CT/internal shows the agent recognising an earlier package, `AG/internal/localapi/escrow_recover.go:122-129`, but no age rule; see the hub helper. \u00b7 Expiry of the Cloudflare DNS token used for the web certificate: grep `CFAPIToken` in CT/internal shows use only." + }, + "extra": [ + "## Tiny", + "## Not found" + ] + } + ], + "pointers": [ + { + "ptr": "CT/internal/stacks/infra.go:130", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Web certificate of the household's address (Let's Encrypt through Traefik)", + "q": "1" + }, + { + "ptr": "CT/internal/infra/templates/traefik.yml.tmpl:51", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Web certificate of the household's address (Let's Encrypt through Traefik)", + "q": "1" + }, + { + "ptr": "CT/internal/infra/templates/traefik.yml.tmpl:48-59", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Web certificate of the household's address (Let's Encrypt through Traefik)", + "q": "2" + }, + { + "ptr": "CT/internal/infra/templates/traefik.yml.tmpl:53", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Web certificate of the household's address (Let's Encrypt through Traefik)", + "q": "3" + }, + { + "ptr": "CT/internal/infra/templates/traefik.yml.tmpl:59", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Web certificate of the household's address (Let's Encrypt through Traefik)", + "q": "3" + }, + { + "ptr": "CT/internal/infra/infra.go:111-118", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Web certificate of the household's address (Let's Encrypt through Traefik)", + "q": "4" + }, + { + "ptr": "CT/internal/stacks/infra.go:160-168", + "file": "OK", + "ident": "IDENT-MISS(cert.*expir|expir.*cert|NotAfter|acme\\.json)", + "ending": "1. Web certificate of the household's address (Let's Encrypt through Traefik)", + "q": "5" + }, + { + "ptr": "CT/internal/stacks/infra.go:160-168", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Web certificate of the household's address (Let's Encrypt through Traefik)", + "q": "6" + }, + { + "ptr": "CT/internal/infra/infra.go:116-118", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Web certificate of the household's address (Let's Encrypt through Traefik)", + "q": "6" + }, + { + "ptr": "CT/internal/infra/templates/traefik.yml.tmpl:51", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Web certificate of the household's address (Let's Encrypt through Traefik)", + "q": "7" + }, + { + "ptr": "CT/internal/infra/templates/traefik.yml.tmpl:48-59", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Web certificate of the household's address (Let's Encrypt through Traefik)", + "q": "8" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:3182-3185", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Proxmox web certificate pinned by the agent", + "q": "1" + }, + { + "ptr": "AG/internal/proxmox/tls.go:39-59", + "file": "OK", + "ident": "IDENT-MISS(E/scripts/felhom-host-install.sh:3182-3185;)", + "ending": "2. Proxmox web certificate pinned by the agent", + "q": "1" + }, + { + "ptr": "AG/internal/proxmox/tls.go:49", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Proxmox web certificate pinned by the agent", + "q": "2" + }, + { + "ptr": "AG/internal/config/config.go:917", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Proxmox web certificate pinned by the agent", + "q": "3" + }, + { + "ptr": "AG/cmd/felhom-agent/main.go:340", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Proxmox web certificate pinned by the agent", + "q": "3" + }, + { + "ptr": "AG/internal/proxmox/tls.go:55", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Proxmox web certificate pinned by the agent", + "q": "4" + }, + { + "ptr": "AG/internal/proxmox/tls.go:55", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Proxmox web certificate pinned by the agent", + "q": "6" + }, + { + "ptr": "AG/internal/proxmox/tls.go:55", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Proxmox web certificate pinned by the agent", + "q": "7" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:3185", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Proxmox web certificate pinned by the agent", + "q": "7" + }, + { + "ptr": "AG/internal/proxmox/tls_test.go:28", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Proxmox web certificate pinned by the agent", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/cert.go:56", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Agent local-API certificate and its pin in the guest", + "q": "1" + }, + { + "ptr": "CT/internal/agentapi/client.go:105", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/localapi/cert.go:56.)", + "ending": "3. Agent local-API certificate and its pin in the guest", + "q": "1" + }, + { + "ptr": "AG/internal/localapi/cert.go:25", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Agent local-API certificate and its pin in the guest", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/cert.go:70", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/localapi/cert.go:25,)", + "ending": "3. Agent local-API certificate and its pin in the guest", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/cert.go:38-52", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Agent local-API certificate and its pin in the guest", + "q": "3" + }, + { + "ptr": "AG/internal/localapi/cert.go:21-24", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Agent local-API certificate and its pin in the guest", + "q": "3" + }, + { + "ptr": "CT/internal/agentapi/client.go:118-128", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Agent local-API certificate and its pin in the guest", + "q": "4" + }, + { + "ptr": "CT/internal/agentapi/client.go:125", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Agent local-API certificate and its pin in the guest", + "q": "4" + }, + { + "ptr": "AG/internal/localapi/cert.go:70", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Agent local-API certificate and its pin in the guest", + "q": "5" + }, + { + "ptr": "AR/03-host-agent.md:420-423", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Agent local-API certificate and its pin in the guest", + "q": "6" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:2749-2786", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Agent local-API certificate and its pin in the guest", + "q": "6" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:2749-2786", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Agent local-API certificate and its pin in the guest", + "q": "7" + }, + { + "ptr": "AG/internal/localapi/cert_test.go:12", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Agent local-API certificate and its pin in the guest", + "q": "8" + }, + { + "ptr": "CT/internal/agentapi/client_test.go:55", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/localapi/cert_test.go:12)", + "ending": "3. Agent local-API certificate and its pin in the guest", + "q": "8" + }, + { + "ptr": "CT/internal/backup/offbox_abandon.go:68-70", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Off-site copy after the household stops it: 14-day countdown", + "q": "1" + }, + { + "ptr": "CT/internal/backup/offbox_abandon.go:31", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Off-site copy after the household stops it: 14-day countdown", + "q": "2" + }, + { + "ptr": "CT/internal/backup/offbox_abandon_levers_d1_test.go:20", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Off-site copy after the household stops it: 14-day countdown", + "q": "3" + }, + { + "ptr": "CT/internal/backup/offbox_abandon_levers_d1_test.go:44", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Off-site copy after the household stops it: 14-day countdown", + "q": "4" + }, + { + "ptr": "CT/internal/backup/offbox_abandon.go:40", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Off-site copy after the household stops it: 14-day countdown", + "q": "5" + }, + { + "ptr": "CT/internal/backup/offbox_abandon_levers_d1_test.go:69", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Off-site copy after the household stops it: 14-day countdown", + "q": "7" + }, + { + "ptr": "CT/internal/backup/offbox_abandon_r241_test.go:48", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Off-site copy after the household stops it: 14-day countdown", + "q": "8" + }, + { + "ptr": "CT/internal/backup/offbox_window.go:81-82", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Off-site backup history: how far back a household can go", + "q": "1" + }, + { + "ptr": "CT/internal/backup/offbox_window.go:47", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Off-site backup history: how far back a household can go", + "q": "2" + }, + { + "ptr": "CT/internal/backup/offbox_window.go:48", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Off-site backup history: how far back a household can go", + "q": "2" + }, + { + "ptr": "CT/internal/backup/offbox_window.go:49", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Off-site backup history: how far back a household can go", + "q": "2" + }, + { + "ptr": "CT/internal/backup/offbox_window.go:81-82", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Off-site backup history: how far back a household can go", + "q": "3" + }, + { + "ptr": "AG/internal/config/config.go:436", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Off-site backup history: how far back a household can go", + "q": "4" + }, + { + "ptr": "AG/internal/config/config.go:409", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Off-site backup history: how far back a household can go", + "q": "4" + }, + { + "ptr": "AG/internal/backup/runner.go:51-58", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Off-site backup history: how far back a household can go", + "q": "6" + }, + { + "ptr": "CT/internal/backup/offbox_integrity.go:81", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. Off-site backup history: how far back a household can go", + "q": "7" + }, + { + "ptr": "CT/internal/backup/offbox_window.go:47-49", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. Off-site backup history: how far back a household can go", + "q": "8" + }, + { + "ptr": "CT/internal/backup/offbox.go:345", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Off-site credentials: SSH key, host key, store password, PBS pin", + "q": "1" + }, + { + "ptr": "CT/internal/backup/offbox.go:898", + "file": "OK", + "ident": "IDENT-MISS(CT/internal/backup/offbox.go:345,)", + "ending": "6. Off-site credentials: SSH key, host key, store password, PBS pin", + "q": "1" + }, + { + "ptr": "AG/internal/pbs/pin.go:17-30", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Off-site credentials: SSH key, host key, store password, PBS pin", + "q": "1" + }, + { + "ptr": "CT/internal/backup/offbox.go", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Off-site credentials: SSH key, host key, store password, PBS pin", + "q": "2" + }, + { + "ptr": "AG/internal/pbs/pin.go:24", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Off-site credentials: SSH key, host key, store password, PBS pin", + "q": "2" + }, + { + "ptr": "AG/internal/pbsdr/manager.go:375-379", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Off-site credentials: SSH key, host key, store password, PBS pin", + "q": "3" + }, + { + "ptr": "CT/internal/backup/offbox.go:895", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Off-site credentials: SSH key, host key, store password, PBS pin", + "q": "4" + }, + { + "ptr": "CT/internal/backup/offbox.go:910", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Off-site credentials: SSH key, host key, store password, PBS pin", + "q": "6" + }, + { + "ptr": "AG/internal/pbsdr/manager.go:375-379", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Off-site credentials: SSH key, host key, store password, PBS pin", + "q": "7" + }, + { + "ptr": "CT/internal/backup/offbox.go:345", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Off-site credentials: SSH key, host key, store password, PBS pin", + "q": "8" + }, + { + "ptr": "CT/internal/web/session_store.go:73", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Sign-in sessions and cookies (dashboard, setup gate, family, share) \u2014 merged", + "q": "1" + }, + { + "ptr": "CT/internal/web/auth.go:30", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Sign-in sessions and cookies (dashboard, setup gate, family, share) \u2014 merged", + "q": "2" + }, + { + "ptr": "CT/internal/web/setup_gate.go:46", + "file": "OK", + "ident": "IDENT-MISS(CT/internal/web/auth.go:30;)", + "ending": "7. Sign-in sessions and cookies (dashboard, setup gate, family, share) \u2014 merged", + "q": "2" + }, + { + "ptr": "CT/internal/family/family.go:42", + "file": "OK", + "ident": "IDENT-MISS(CT/internal/web/setup_gate.go:46;)", + "ending": "7. Sign-in sessions and cookies (dashboard, setup gate, family, share) \u2014 merged", + "q": "2" + }, + { + "ptr": "CT/internal/web/share.go:31", + "file": "OK", + "ident": "IDENT-MISS(CT/internal/family/family.go:42;)", + "ending": "7. Sign-in sessions and cookies (dashboard, setup gate, family, share) \u2014 merged", + "q": "2" + }, + { + "ptr": "CT/internal/family/family.go:506-514", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Sign-in sessions and cookies (dashboard, setup gate, family, share) \u2014 merged", + "q": "3" + }, + { + "ptr": "CT/internal/web/auth.go:308", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Sign-in sessions and cookies (dashboard, setup gate, family, share) \u2014 merged", + "q": "3" + }, + { + "ptr": "CT/internal/web/session_store.go:105", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Sign-in sessions and cookies (dashboard, setup gate, family, share) \u2014 merged", + "q": "6" + }, + { + "ptr": "CT/internal/web/auth.go:369", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Sign-in sessions and cookies (dashboard, setup gate, family, share) \u2014 merged", + "q": "6" + }, + { + "ptr": "CT/internal/web/session_store_test.go:144", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Sign-in sessions and cookies (dashboard, setup gate, family, share) \u2014 merged", + "q": "8" + }, + { + "ptr": "CT/internal/web/setup_gate_test.go:182", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Sign-in sessions and cookies (dashboard, setup gate, family, share) \u2014 merged", + "q": "8" + }, + { + "ptr": "CT/internal/web/claim.go:30", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Claim code and the one-use gate token", + "q": "1" + }, + { + "ptr": "CT/internal/web/setup_gate.go:47", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Claim code and the one-use gate token", + "q": "1" + }, + { + "ptr": "CT/internal/web/claim.go:30", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Claim code and the one-use gate token", + "q": "2" + }, + { + "ptr": "CT/internal/web/setup_gate.go:47", + "file": "OK", + "ident": "IDENT-MISS(CT/internal/web/claim.go:30;)", + "ending": "8. Claim code and the one-use gate token", + "q": "2" + }, + { + "ptr": "AR/01-topology-and-trust.md:192", + "file": "OK", + "ident": "IDENT-MISS(gateTokenLife)", + "ending": "8. Claim code and the one-use gate token", + "q": "2" + }, + { + "ptr": "CT/internal/web/claim.go:333", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Claim code and the one-use gate token", + "q": "4" + }, + { + "ptr": "CT/internal/web/claim.go:333", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Claim code and the one-use gate token", + "q": "6" + }, + { + "ptr": "CT/internal/web/claim.go:33", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Claim code and the one-use gate token", + "q": "7" + }, + { + "ptr": "CT/internal/web/claim_test.go:189", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Claim code and the one-use gate token", + "q": "8" + }, + { + "ptr": "CT/internal/settings/settings.go:45", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Share link `/s/`", + "q": "1" + }, + { + "ptr": "CT/internal/settings/settings.go:1111", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Share link `/s/`", + "q": "2" + }, + { + "ptr": "CT/internal/web/share.go:31", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Share link `/s/`", + "q": "4" + }, + { + "ptr": "CT/internal/web/share_test.go:256", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. Share link `/s/`", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/escrow_ceremony.go:158", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Recovery-code ceremony windows", + "q": "1" + }, + { + "ptr": "AG/internal/localapi/escrow_ceremony.go:158", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Recovery-code ceremony windows", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/escrow_ceremony.go:166", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/localapi/escrow_ceremony.go:158;)", + "ending": "10. Recovery-code ceremony windows", + "q": "2" + }, + { + "ptr": "CT/internal/web/handlers.go:1169", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. Recovery-code ceremony windows", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/escrow_ceremony.go:506", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Recovery-code ceremony windows", + "q": "4" + }, + { + "ptr": "AG/internal/localapi/escrow_ceremony.go:506", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Recovery-code ceremony windows", + "q": "7" + }, + { + "ptr": "AG/internal/localapi/escrow_ceremony.go:38", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Recovery-code ceremony windows", + "q": "7" + }, + { + "ptr": "AG/internal/localapi/escrow_ceremony.go:158-166", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. Recovery-code ceremony windows", + "q": "8" + }, + { + "ptr": "AR/03-host-agent.md:268", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Signed operator jobs: the validity window", + "q": "1" + }, + { + "ptr": "AG/internal/authz/verifier.go:104", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Signed operator jobs: the validity window", + "q": "1" + }, + { + "ptr": "AG/internal/authz/verifier.go:26", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Signed operator jobs: the validity window", + "q": "2" + }, + { + "ptr": "AG/cmd/felhom-opsign/main.go:57", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/authz/verifier.go:26;)", + "ending": "11. Signed operator jobs: the validity window", + "q": "2" + }, + { + "ptr": "AR/04-control-plane-authorization.md:78-84", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Signed operator jobs: the validity window", + "q": "2" + }, + { + "ptr": "AG/internal/signedjobs/runner.go:163-170", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Signed operator jobs: the validity window", + "q": "4" + }, + { + "ptr": "AG/internal/authz/verifier.go:161-162", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Signed operator jobs: the validity window", + "q": "4" + }, + { + "ptr": "AR/04-control-plane-authorization.md:184", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Signed operator jobs: the validity window", + "q": "5" + }, + { + "ptr": "AG/internal/authz/verifier.go:90", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Signed operator jobs: the validity window", + "q": "6" + }, + { + "ptr": "AG/internal/signedjobs/runner.go:163-170", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Signed operator jobs: the validity window", + "q": "7" + }, + { + "ptr": "AG/internal/authz/noncestore.go:47-50", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Signed operator jobs: the validity window", + "q": "7" + }, + { + "ptr": "AG/configs/test_felhom_os_apply.py:833", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Signed operator jobs: the validity window", + "q": "8" + }, + { + "ptr": "AG/configs/test_felhom_os_apply.py:836", + "file": "OK", + "ident": "IDENT-MISS(AG/configs/test_felhom_os_apply.py:833,)", + "ending": "11. Signed operator jobs: the validity window", + "q": "8" + }, + { + "ptr": "AG/internal/authz/verify_test.go:82", + "file": "OK", + "ident": "IDENT-MISS(AG/configs/test_felhom_os_apply.py:836,)", + "ending": "11. Signed operator jobs: the validity window", + "q": "8" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:212", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Operator signing keys on each box", + "q": "1" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:214", + "file": "OK", + "ident": "IDENT-MISS(E/scripts/felhom-host-install.sh:212)", + "ending": "12. Operator signing keys on each box", + "q": "1" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:3424", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Operator signing keys on each box", + "q": "1" + }, + { + "ptr": "AG/internal/authz/verifier.go:122-126", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Operator signing keys on each box", + "q": "2" + }, + { + "ptr": "AG/internal/authz/verifier.go:170-175", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Operator signing keys on each box", + "q": "2" + }, + { + "ptr": "AR/04-control-plane-authorization.md:139-150", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Operator signing keys on each box", + "q": "3" + }, + { + "ptr": "AG/internal/reconcile/classify.go:41", + "file": "OK", + "ident": "IDENT-OK", + "ending": "12. Operator signing keys on each box", + "q": "3" + }, + { + "ptr": "AR/04-control-plane-authorization.md:139-150", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Operator signing keys on each box", + "q": "4" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:3452-3453", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Operator signing keys on each box", + "q": "6" + }, + { + "ptr": "AG/internal/reconcile/classify.go:41", + "file": "OK", + "ident": "IDENT-MISS(AG/)", + "ending": "12. Operator signing keys on each box", + "q": "7" + }, + { + "ptr": "AR/04-control-plane-authorization.md:139-150", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Operator signing keys on each box", + "q": "8" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:2405", + "file": "OK", + "ident": "IDENT-OK", + "ending": "13. Proxmox API token of the agent", + "q": "1" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:2448", + "file": "OK", + "ident": "IDENT-OK", + "ending": "13. Proxmox API token of the agent", + "q": "2" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:2424-2445", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. Proxmox API token of the agent", + "q": "3" + }, + { + "ptr": "E/scripts/felhom-host-install.sh", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. Proxmox API token of the agent", + "q": "5" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:2448", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. Proxmox API token of the agent", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/tokenstore.go:24-33", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. Guest token for the agent and the box's hub key \u2014 merged", + "q": "1" + }, + { + "ptr": "AG/internal/hub/client.go:60", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/localapi/tokenstore.go:24-33,)", + "ending": "14. Guest token for the agent and the box's hub key \u2014 merged", + "q": "1" + }, + { + "ptr": "AG/internal/localapi/tokenstore.go:24-33", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. Guest token for the agent and the box's hub key \u2014 merged", + "q": "2" + }, + { + "ptr": "AG/internal/hub/client.go:36", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. Guest token for the agent and the box's hub key \u2014 merged", + "q": "2" + }, + { + "ptr": "AG/internal/localapi/tokenstore.go:24-33", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. Guest token for the agent and the box's hub key \u2014 merged", + "q": "3" + }, + { + "ptr": "AG/internal/hub/client.go:41", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. Guest token for the agent and the box's hub key \u2014 merged", + "q": "4" + }, + { + "ptr": "AG/internal/hub/client.go:41", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. Guest token for the agent and the box's hub key \u2014 merged", + "q": "6" + }, + { + "ptr": "AG/internal/localapi/tokenstore.go:24-33", + "file": "OK", + "ident": "NO-IDENT", + "ending": "14. Guest token for the agent and the box's hub key \u2014 merged", + "q": "8" + }, + { + "ptr": "AG/internal/wgtunnel/rekey.go:110", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. WireGuard key of the operator tunnel", + "q": "1" + }, + { + "ptr": "AG/internal/wgtunnel/rekey.go:36", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. WireGuard key of the operator tunnel", + "q": "2" + }, + { + "ptr": "AG/internal/wgtunnel/rekey.go:16-30", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. WireGuard key of the operator tunnel", + "q": "2" + }, + { + "ptr": "AG/internal/wgtunnel/rekey.go:38", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. WireGuard key of the operator tunnel", + "q": "3" + }, + { + "ptr": "AG/internal/wgtunnel/rekey.go:39", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/wgtunnel/rekey.go:38;)", + "ending": "15. WireGuard key of the operator tunnel", + "q": "3" + }, + { + "ptr": "AG/internal/wgtunnel/rekey.go:16-30", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. WireGuard key of the operator tunnel", + "q": "6" + }, + { + "ptr": "AG/internal/wgtunnel/rekey_test.go:119", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. WireGuard key of the operator tunnel", + "q": "7" + }, + { + "ptr": "AG/internal/wgtunnel/rekey_test.go:35", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. WireGuard key of the operator tunnel", + "q": "8" + }, + { + "ptr": "AG/internal/wgtunnel/rekey_test.go:182", + "file": "OK", + "ident": "IDENT-MISS(AG/internal/wgtunnel/rekey_test.go:35,)", + "ending": "15. WireGuard key of the operator tunnel", + "q": "8" + }, + { + "ptr": "AG/configs/build-golden.sh:129", + "file": "OK", + "ident": "IDENT-OK", + "ending": "16. Package signing keys: Docker in the guest, Proxmox and Debian on the host", + "q": "1" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:2616", + "file": "OK", + "ident": "IDENT-MISS(AG/configs/build-golden.sh:129.)", + "ending": "16. Package signing keys: Docker in the guest, Proxmox and Debian on the host", + "q": "1" + }, + { + "ptr": "AG/configs/build-golden.sh", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Package signing keys: Docker in the guest, Proxmox and Debian on the host", + "q": "2" + }, + { + "ptr": "E/scripts/felhom-host-install.sh", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Package signing keys: Docker in the guest, Proxmox and Debian on the host", + "q": "2" + }, + { + "ptr": "AR/11-os-updates.md:657", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Package signing keys: Docker in the guest, Proxmox and Debian on the host", + "q": "3" + }, + { + "ptr": "AR/11-os-updates.md:657", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Package signing keys: Docker in the guest, Proxmox and Debian on the host", + "q": "5" + }, + { + "ptr": "AR/11-os-updates.md:657", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Package signing keys: Docker in the guest, Proxmox and Debian on the host", + "q": "7" + }, + { + "ptr": "AR/11-os-updates.md:657", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Package signing keys: Docker in the guest, Proxmox and Debian on the host", + "q": "8" + }, + { + "ptr": "AG/configs/test_felhom_os_apply.py:488", + "file": "OK", + "ident": "IDENT-OK", + "ending": "16. Package signing keys: Docker in the guest, Proxmox and Debian on the host", + "q": "8" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:2615", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. Debian 13 (trixie) and Proxmox 9: the series the product is tied to", + "q": "1" + }, + { + "ptr": "AG/configs/build-golden.sh:130", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. Debian 13 (trixie) and Proxmox 9: the series the product is tied to", + "q": "1" + }, + { + "ptr": "AR/11-os-updates.md", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Debian 13 (trixie) and Proxmox 9: the series the product is tied to", + "q": "2" + }, + { + "ptr": "AR/11-os-updates.md:725", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Debian 13 (trixie) and Proxmox 9: the series the product is tied to", + "q": "2" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:2146-2152", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Debian 13 (trixie) and Proxmox 9: the series the product is tied to", + "q": "4" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:2146-2152", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. Debian 13 (trixie) and Proxmox 9: the series the product is tied to", + "q": "8" + }, + { + "ptr": "CAT/templates/actualbudget/.felhom.yml:196", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Catalog app images and their recorded digests", + "q": "1" + }, + { + "ptr": "CAT/scripts/check-image-pins.py:28", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Catalog app images and their recorded digests", + "q": "2" + }, + { + "ptr": "CAT/scripts/check-currency.py:22", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Catalog app images and their recorded digests", + "q": "3" + }, + { + "ptr": "CT/internal/stacks/digest.go:94", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Catalog app images and their recorded digests", + "q": "4" + }, + { + "ptr": "AR/09-update-architecture.md:391", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Catalog app images and their recorded digests", + "q": "4" + }, + { + "ptr": "CAT/scripts/check-currency.py:22", + "file": "OK", + "ident": "IDENT-MISS(manifest)", + "ending": "18. Catalog app images and their recorded digests", + "q": "5" + }, + { + "ptr": "CT/internal/stacks/digest.go:10-23", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Catalog app images and their recorded digests", + "q": "6" + }, + { + "ptr": "AR/09-update-architecture.md:391", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Catalog app images and their recorded digests", + "q": "8" + }, + { + "ptr": "CT/internal/stacks/digest.go:94", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Catalog app images and their recorded digests", + "q": "8" + }, + { + "ptr": "CT/internal/infra/infra.go:33", + "file": "OK", + "ident": "IDENT-OK", + "ending": "19. Images the controller itself starts (Traefik, tunnel, file browser, Samba)", + "q": "1" + }, + { + "ptr": "CT/internal/infra/infra.go:34", + "file": "OK", + "ident": "IDENT-OK", + "ending": "19. Images the controller itself starts (Traefik, tunnel, file browser, Samba)", + "q": "1" + }, + { + "ptr": "CT/internal/infra/infra.go:35", + "file": "OK", + "ident": "IDENT-OK", + "ending": "19. Images the controller itself starts (Traefik, tunnel, file browser, Samba)", + "q": "1" + }, + { + "ptr": "CT/internal/infra/infra.go:50", + "file": "OK", + "ident": "IDENT-OK", + "ending": "19. Images the controller itself starts (Traefik, tunnel, file browser, Samba)", + "q": "1" + }, + { + "ptr": "AR/01-topology-and-trust.md", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. Images the controller itself starts (Traefik, tunnel, file browser, Samba)", + "q": "2" + }, + { + "ptr": "CT/internal/infra/infra.go:33-50", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. Images the controller itself starts (Traefik, tunnel, file browser, Samba)", + "q": "8" + }, + { + "ptr": "AG/scripts/retention-policy.json:23", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Published agent versions: only the newest five stay downloadable", + "q": "1" + }, + { + "ptr": "AG/scripts/retention-policy.json:7-10", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Published agent versions: only the newest five stay downloadable", + "q": "1" + }, + { + "ptr": "AG/scripts/retention-policy.json:30", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Published agent versions: only the newest five stay downloadable", + "q": "2" + }, + { + "ptr": "AG/scripts/retention-policy.json:31", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. Published agent versions: only the newest five stay downloadable", + "q": "2" + }, + { + "ptr": "AR/09-update-architecture.md:1370-1375", + "file": "OK", + "ident": "IDENT-MISS(day0_agent)", + "ending": "20. Published agent versions: only the newest five stay downloadable", + "q": "2" + }, + { + "ptr": "AG/scripts/retention-policy.json:12-15", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Published agent versions: only the newest five stay downloadable", + "q": "3" + }, + { + "ptr": "AG/scripts/retention-policy.json:14-15", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Published agent versions: only the newest five stay downloadable", + "q": "4" + }, + { + "ptr": "AG/scripts/retention-policy.json:20-21", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Published agent versions: only the newest five stay downloadable", + "q": "4" + }, + { + "ptr": "AG/scripts/retention-policy.json:25", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Published agent versions: only the newest five stay downloadable", + "q": "5" + }, + { + "ptr": "AG/scripts/retention-policy.json:23-31", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Published agent versions: only the newest five stay downloadable", + "q": "8" + }, + { + "ptr": "AR/09-update-architecture.md:1370-1375", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. Published agent versions: only the newest five stay downloadable", + "q": "8" + }, + { + "ptr": "AR/09-update-architecture.md:1010-1018", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Golden image age and the waiver file", + "q": "1" + }, + { + "ptr": "E/scripts/golden_currency_gate.py:169", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Golden image age and the waiver file", + "q": "1" + }, + { + "ptr": "E/scripts/golden_currency_gate.py:169", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. Golden image age and the waiver file", + "q": "2" + }, + { + "ptr": "E/scripts/golden_currency_gate.py:167", + "file": "OK", + "ident": "IDENT-MISS(E/scripts/golden_currency_gate.py:169.)", + "ending": "21. Golden image age and the waiver file", + "q": "2" + }, + { + "ptr": "E/scripts/golden_currency_gate.py:329-333", + "file": "OK", + "ident": "IDENT-MISS(read_waiver)", + "ending": "21. Golden image age and the waiver file", + "q": "2" + }, + { + "ptr": "E/scripts/golden_currency_gate.py:162-163", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. Golden image age and the waiver file", + "q": "2" + }, + { + "ptr": "AR/09-update-architecture.md:1010-1018", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Golden image age and the waiver file", + "q": "3" + }, + { + "ptr": "E/scripts/golden_currency_gate.py:329-333", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Golden image age and the waiver file", + "q": "7" + }, + { + "ptr": "E/scripts/golden_currency_gate.py:162-169", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Golden image age and the waiver file", + "q": "8" + }, + { + "ptr": "E/scripts/test_golden_currency_gate.py:1", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Golden image age and the waiver file", + "q": "8" + }, + { + "ptr": "AG/configs/build-golden.sh:67", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Debian container template the golden is built from", + "q": "1" + }, + { + "ptr": "AG/configs/build-golden.sh:101", + "file": "OK", + "ident": "IDENT-MISS(AG/configs/build-golden.sh:67,)", + "ending": "22. Debian container template the golden is built from", + "q": "1" + }, + { + "ptr": "AG/configs/build-golden.sh", + "file": "OK", + "ident": "NO-IDENT", + "ending": "22. Debian container template the golden is built from", + "q": "5" + }, + { + "ptr": "AG/configs/build-golden.sh:101", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Debian container template the golden is built from", + "q": "7" + }, + { + "ptr": "AG/configs/build-golden.sh:67", + "file": "OK", + "ident": "NO-IDENT", + "ending": "22. Debian container template the golden is built from", + "q": "8" + }, + { + "ptr": "CT/build.sh:113-125", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. Build-time pins of the controller image", + "q": "4" + }, + { + "ptr": "CT/internal/backup/restore_copy_sweep.go:29", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "1" + }, + { + "ptr": "CT/internal/backup/restore_copy_sweep.go:29", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "2" + }, + { + "ptr": "CT/internal/backup/restore_copy_sweep.go:29", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "6" + }, + { + "ptr": "CT/internal/backup/kept_load.go:217", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "6" + }, + { + "ptr": "CT/internal/backup/restore_copy_sweep.go:29", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "8" + }, + { + "ptr": "CT/internal/mailrelay/server.go:386", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "8" + }, + { + "ptr": "CT/internal/web/i18n_web.go:478", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "8" + }, + { + "ptr": "CT/internal/infra/infra.go:108", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "8" + }, + { + "ptr": "CT/internal/config/config.go:70", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "8" + }, + { + "ptr": "CT/internal/nightchain/banner.go:23", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/opactions.go:42", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/backup_cap.go:38", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/opreboot.go:48", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "8" + }, + { + "ptr": "CT/internal/selfupdate/updater.go:71", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "8" + }, + { + "ptr": "CT/internal/sync/sync.go:399", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "8" + }, + { + "ptr": "AG/internal/felhomsshd/manager.go:20", + "file": "OK", + "ident": "IDENT-OK", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "8" + }, + { + "ptr": "AR/11-os-updates.md", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "8" + }, + { + "ptr": "AG/internal/pbs/pin.go:24", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "8" + }, + { + "ptr": "AG/internal/localapi/escrow_recover.go:122-129", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. Kept copies made before a restore: deleted after 30 unused days", + "q": "8" + } + ] +} \ No newline at end of file diff --git a/documentation/audits/end-dates-list-2026-11-11/evidence/helpers/resolved-dates-hub.json b/documentation/audits/end-dates-list-2026-11-11/evidence/helpers/resolved-dates-hub.json new file mode 100644 index 00000000..cf5ddfb3 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/evidence/helpers/resolved-dates-hub.json @@ -0,0 +1,2697 @@ +{ + "endings": [ + { + "title": "1. Self-bind (connect) link in the household's e-mail", + "answers": { + "1": "A one-use link that lets the household tie a new box to its account. Household uses it, product mints it. Stored hashed in the hub table `selfbind_tokens` `HUB/store/store.go:796-806`; minted by `selfBindTTL` code `HUB/web/selfbind_mint.go:20`.", + "2": "7 days after minting: `selfBindTTL` `HUB/web/selfbind_mint.go:20`, passed to the store at `HUB/web/selfbind_mint.go:77`; `Expired` `HUB/store/selfbind.go:39`. Also locks at the 5th wrong answer, `SelfBindMaxAttempts` `HUB/store/selfbind.go:21`.", + "3": "The household itself: the expired page has a button, `handleBindResend` `HUB/web/selfbind.go:384`, at most one new link an hour, `bindResendEvery` `HUB/web/selfbind.go:382`. No renewal before the end. The hub also mints at customer creation and RESET `HUB/web/selfbind_mint.go:97`.", + "4": "The household cannot bind the box from the mail; unknown and expired links show the same \"expired\" page `HUB/web/selfbind.go:206`. The box itself keeps waiting; nothing on it is lost.", + "5": "Nobody is warned before the end. NOT FOUND: grep `expir|remind` in `HUB/web/selfbind_mint.go:1-110` shows no reminder. The mint log line says \"valid 7 days\" `HUB/web/selfbind_mint.go:93`.", + "6": "A box first switched on after day 7: the link fails at the first step (opening it); the household presses the resend button. If an old host is still linked no link is sent and the operator gets `eventBindOldHostLinked` `HUB/web/selfbind.go:440`, logged at `HUB/web/selfbind.go:413`.", + "7": "If the mail send fails at mint the token is deleted, so no dead link is left `HUB/web/selfbind_mint.go:82`. No automatic retry; the household presses again after an hour.", + "8": "TEST `TestR719_AnExpiredLinkOffersAndSendsAFreshOne` `HUB/web/selfbind_resend_test.go:21`; SOURCE `HUB/web/selfbind_mint.go:20`; DOC `AR/05-hub-architecture.md:359`." + }, + "extra": [] + }, + { + "title": "2. Customer claim code (the household's first password step)", + "answers": { + "1": "A code the hub mails so the household can set its own box password. Household uses it; the hub stores a hash and a generation. `RotateClaimCode` `HUB/store/store.go:2084`.", + "2": "NO END DATE: `RotateClaimCode` `HUB/store/store.go:2084` bumps a generation and writes no expiry. It ends when a newer code replaces it. Limit by count, not age: `maxResetPerDay` = 3 `HUB/claim/engine.go:30`.", + "3": "The household asks for a reset; the hub mints a new code. No lead time, because there is no end.", + "4": "After 3 resets in a day the 4th is refused, `maxResetPerDay` `HUB/claim/engine.go:30`, counted by `BumpResetCount` `HUB/store/store.go:2124`. The box side has its own 5-attempt, 15-minute lock, named at `HUB/claim/engine.go:27`.", + "5": "Nobody; there is no end to warn about.", + "6": "Comes back by itself: an old code still works until it is rotated. SOURCE `HUB/store/store.go:2084`.", + "7": "NOT FOUND for a failed code mail: searched grep `expire|ttl|valid` in `HUB/claim/engine.go:1-60`; only the daily count limit is there.", + "8": "SOURCE `HUB/claim/engine.go:30` and `HUB/store/store.go:2084`. No test for \"a code never ages\" was found (grep `Expire` in the claim package): NOBODY KNOWS whether that is intended." + }, + "extra": [] + }, + { + "title": "3. Retrieval passphrase: the moment it stops giving keys", + "answers": { + "1": "The household's retrieval passphrase opens the box's keys at the hub only for a while. Household owns it; rule lives in `passphraseKeysClosed` `HUB/api/passphrase_scope.go:46`.", + "2": "Ends on an EVENT, not a date: once the box has a key of its own and uses it, the passphrase gives no keys. Rule text `HUB/api/passphrase_scope.go:22`; needs controller `boxConfigMinController` 0.334.0 `HUB/api/passphrase_scope.go:40`.", + "3": "Not renewed. To open it again the old host is deleted at the hub first (operator act), then a reinstall mints anew. DOC `AR/05-hub-architecture.md:270-272`.", + "4": "A reinstall that reuses the passphrase without the old host deleted is refused; the household's new box gets no keys. Event `eventPassphraseKeysRefused` `HUB/api/passphrase_scope.go:42`.", + "5": "Nobody before; the operator sees the event after a refusal, `eventPassphraseKeysRefused` `HUB/api/passphrase_scope.go:42`.", + "6": "A running box is not touched: it uses its own key. A wiped box coming back fails at the key fetch step until the operator deletes the old host.", + "7": "The refusal changes nothing; the same request works after the host delete. No retry by the hub.", + "8": "TEST `TestR1159_PassphraseGivesKeysOnlyUntilTheBoxUsesItsOwn` `HUB/api/r1159_passphrase_scope_test.go:110`." + }, + "extra": [] + }, + { + "title": "4. Pairing code and appliance registration (box console code)", + "answers": { + "1": "The 6-character code a new box shows on its screen, and the hub row behind it. Product mints, household types it. `PairingCode` `HUB/store/appliance.go:35`; box side `E/scripts/iso/felhom-bootstrap.sh:50`.", + "2": "NO END DATE: `ApplianceByPairingCode` `HUB/store/appliance.go:155` has no age check. After 7 days without a poll the row only gets a \"stale\" badge, `applianceStaleAfter` `HUB/web/appliances.go:21`.", + "3": "Nobody; the code stays the same across re-registration `HUB/store/appliance.go:35`.", + "4": "Nothing breaks by age. An operator who discards the row makes the box log \"appliance token not recognized\" and keep trying `E/scripts/iso/felhom-bootstrap.sh:589`.", + "5": "Operator only, by the stale badge on the appliances page `HUB/web/appliances.go:78`. No mail.", + "6": "Comes back by itself: the box polls every `POLL_INTERVAL` 30 s `E/scripts/iso/felhom-bootstrap.sh:54` and retries the network gate every `GATE_RETRY_INTERVAL` 60 s `E/scripts/iso/felhom-bootstrap.sh:180`, with no give-up found.", + "7": "A failed registration is retried by the unit `E/scripts/iso/felhom-bootstrap.sh:495`.", + "8": "SOURCE `HUB/store/appliance.go:155` and `HUB/web/appliances.go:21`. No test of \"never expires\" found (grep `Stale|expire` in appliance files)." + }, + "extra": [] + }, + { + "title": "5. \"Switched off on purpose\" hold (14 days of silence allowed)", + "answers": { + "1": "When a household switches the box off through the product, the hub stays quiet about the silence. Household act, hub rule: `DeliberateOffHold` `HUB/store/deliberateoff.go:18`.", + "2": "14 days after the switch-off: `DeliberateOffHold` `HUB/store/deliberateoff.go:18`, judged by `DeliberateOffHeld` `HUB/store/deliberateoff.go:21`.", + "3": "Nobody renews. The hold ends by itself when the box reports again or at day 14.", + "4": "After day 14 the silence and missed-backup alarms start again for that household; the operator is told.", + "5": "The operator gets an event at switch-off time naming the moment, message built at `HUB/api/deliberate_off.go:42`. No warning before day 14 was found (grep `DeliberateOffHold` in hub).", + "6": "Yes, it comes back by itself: a box switched on at any time reports and the hold clears. TEST `TestDeliberateOff_BackOnIsQuiet` `HUB/monitor/deliberate_off_test.go:138`.", + "7": "Not applicable: nothing is renewed.", + "8": "TEST `TestDeliberateOff_AlarmsAfterFourteenDays` `HUB/monitor/deliberate_off_test.go:112` and `TestDeliberateOff_NoSilenceAlarmDuringTheHold` `HUB/monitor/deliberate_off_test.go:90`." + }, + "extra": [] + }, + { + "title": "6. Retained old escrow packages and the 7-day abandon of set-aside off-site data", + "answers": { + "1": "Old sealed key packages the hub keeps after a re-key, and off-site data set aside from a previous install. Household's data, hub keeps it. Table `host_escrow_superseded` `HUB/store/store.go:440`.", + "2": "Packages: NO END DATE, the schema comment says no pruning `HUB/store/store.go:417`. Set-aside data: deleted 7 days after the household asks to abandon it, `DefaultAbandonDelay` `HUB/offsitekeys/service.go:448`.", + "3": "Nobody renews. The household can cancel inside the 7 days; request path `RequestAbandon` `HUB/offsitekeys/service.go:473`.", + "4": "After the abandon runs, the old off-site data and its retained packages are gone for that household, `PurgeAbandonedEscrowForCustomer` `HUB/store/store.go:4360`. A customer delete also purges them `HUB/store/store.go:1985`.", + "5": "NOT FOUND: no reminder before day 7 (grep `DefaultAbandonDelay|remind` in `HUB/offsitekeys/service.go:440-560`). A failure raises the event at `HUB/offsitekeys/service.go:540`.", + "6": "Yes for the box: the sweep runs on the hub, `SweepAbandons` `HUB/offsitekeys/service.go:529`, and needs no box. A box that was off simply finds the old data gone.", + "7": "A failed delete is marked and tried again on the next sweep, with one event `HUB/offsitekeys/service.go:540`.", + "8": "TEST `TestAbandon_DelayCancelAndScope` `HUB/offsitekeys/service_test.go:134`; SOURCE `HUB/store/store.go:417`." + }, + "extra": [] + }, + { + "title": "7. Off-site clean-up window (the short time a box may delete off-site)", + "answers": { + "1": "A time-boxed permission the hub opens so a box can prune its own off-site store. Product. `windowLength` `HUB/offsitekeys/service.go:263`.", + "2": "20 minutes per window, `windowLength` `HUB/offsitekeys/service.go:263`; a new one is allowed about weekly, `windowCadence` 6 days 12 hours `HUB/offsitekeys/service.go:264`.", + "3": "The hub closes it; the box asks for the next one on its own night run. No human.", + "4": "If the prune does not finish in 20 minutes it waits a week; the off-site store grows meanwhile. Operator and, in time, the household's off-site space.", + "5": "NOT FOUND for a warning before close (grep `windowLength` in hub). The operator gets an event only when a window cannot be closed, logged at `HUB/offsitekeys/service.go:425`.", + "6": "Yes: a box that was off asks again on its next run. First failing step if the cadence has not passed: the grant is refused.", + "7": "A window past its time that cannot be closed is retried at the next sweep, `SweepExpiredWindows` `HUB/offsitekeys/service.go:413`, log `HUB/offsitekeys/service.go:425`. State left: delete right still open.", + "8": "TEST `TestWindow_LeftOpenIsClosedByTheSweep` `HUB/offsitekeys/service_test.go:99` and `TestWindow_GrantOpenCloseAndDropAlarm` `HUB/offsitekeys/service_test.go:37`." + }, + "extra": [] + }, + { + "title": "8. Storage Box sub-account password and the hub's seal key", + "answers": { + "1": "Each household's off-site store account at the provider, minted by the hub, and the hub key that seals stored secrets (`OFFSITE_SECRET_KEY`, named in `E/manifests/hub.yaml:167-171`). Operator and product.", + "2": "NO END DATE in code: no lifetime on the sub-account password. A clean reinstall's re-issue resets it, DOC `AR/05-hub-architecture.md:274-279`; `ResetBoxPassword` `HUB/hetznerapi/hetznerapi.go:297`.", + "3": "Operator, by pressing Re-issue. Seal key rotation is \"not built\", DOC `RB/secrets.md:159`.", + "4": "If the seal key is lost or changed, the old sealed rows cannot be opened, DOC `RB/secrets.md:159`: operator loses stored off-site secrets and break-glass passwords until re-issued per household.", + "5": "Nobody. NOT FOUND: grep `expir|rotat` in `HUB/hetznerapi/hetznerapi.go:1-320` finds no age rule or warning.", + "6": "A box off for a long time comes back by itself as long as its account was not re-issued; it uses its SSH key, the password is used only by the hub, DOC `AR/05-hub-architecture.md:274-279`.", + "7": "NOT FOUND for a half-done reset (searched `ResetBoxPassword` callers only by name). Candidate question for the main session.", + "8": "DOC `RB/secrets.md:159` and `AR/05-hub-architecture.md:274-279`; SOURCE `HUB/hetznerapi/hetznerapi.go:297`. No test of a lifetime, because there is none." + }, + "extra": [] + }, + { + "title": "9. PBS off-site token on ep0 and its descriptor on the hub", + "answers": { + "1": "The per-household token on the rented endpoint for the whole-guest off-site copy, and the hub record that must agree with it. Product mints, operator can re-issue. DOC `AR/05-hub-architecture.md:373-387`.", + "2": "NO END DATE stated: the lifecycle table has only events (provision, re-key, deprovision) `AR/05-hub-architecture.md:379-387`. One age rule: a staged secret not consumed in 15 minutes is wrong, `unconsumedGrace` `HUB/pbsdrheal/reconciler.go:63`.", + "3": "The hub re-keys on re-issue (delete and recreate) `AR/05-hub-architecture.md:382`. No scheduled renewal.", + "4": "A re-key the box did not pick up leaves the box \"serving a revoked credential\": the household has no whole-guest off-site copy while both tiers look green, comment at `HUB/pbsdrheal/reconciler.go:55-63`.", + "5": "Operator only, after the fact: `pbsdr_provision_failed` on a final failure, DOC `AR/05-hub-architecture.md:400-402`.", + "6": "Host delete keeps the token `AR/05-hub-architecture.md:384`; a box off and back uses the same token. After a re-issue while it was off, it must consume the new secret on its next report.", + "7": "A failed self-check rolls back and is repeated 3 times, 7 s apart, then raises the operator event, DOC `AR/05-hub-architecture.md:400-402`.", + "8": "DOC `AR/05-hub-architecture.md:373-403`; SOURCE `HUB/pbsdrheal/reconciler.go:63`. Lifetime of the token on the endpoint itself: NOBODY KNOWS from these sources (ep0 not read)." + }, + "extra": [] + }, + { + "title": "10. WireGuard peers (the box's tunnel to the off-site endpoint)", + "answers": { + "1": "One peer per box on the endpoint, registered through the hub; the private key is in the household's sealed bundle. Product. DOC `AR/06-offsite-connectivity.md:172-182`.", + "2": "NO END DATE: the hub store only removes a peer on request, `RemoveWGPeer` `HUB/store/wg.go:345`. Scheduled key rotation is an open point, DOC `AR/06-offsite-connectivity.md:368`.", + "3": "Nobody on a schedule. The agent makes a new key itself only after a handshake stamped in the future for 15 minutes, DOC `AR/06-offsite-connectivity.md:205-207`.", + "4": "Revocation: the box is off the off-site network at once, DOC `AR/06-offsite-connectivity.md:180-182`; the household's off-site copies stop.", + "5": "Nobody before. NOT FOUND: grep `expire|rotat|stale|handshake` in `HUB/store/wg.go:1-360` finds no age rule.", + "6": "Yes, a box off for months keeps its peer. A revoked box never re-registers on its own, DOC `AR/06-offsite-connectivity.md:200-205`.", + "7": "A failed fetch is never read as a teardown, DOC `AR/06-offsite-connectivity.md:207-208`. Re-add is an operator call.", + "8": "DOC `AR/06-offsite-connectivity.md:200-208`; SOURCE `HUB/store/wg.go:345`. Box-side tests belong to the box helper." + }, + "extra": [] + }, + { + "title": "11. Signed jobs (agent update, config bundle, OS step)", + "answers": { + "1": "An operator-signed order the hub queues for one box. Operator signs with `felhom-opsign`; hub stores an opaque blob, `EnqueueSignedJob` `HUB/store/store.go:3764`; the agent checks it.", + "2": "30 minutes by default: flag `ttl` `AG/cmd/felhom-opsign/main.go:57`, end set at `AG/cmd/felhom-opsign/main.go:130`. Never more than 24 hours: `MaxWindow` `AG/internal/authz/verifier.go:26`, refused at signing `AG/cmd/felhom-opsign/main.go:126`.", + "3": "The operator (or CC, see block 12) signs a new job. Nothing renews by itself.", + "4": "A box that does not poll inside the window does not get the update or the root-file change; the operator must sign again for that box.", + "5": "Nobody before. The agent logs the rejection and clears the job `AG/internal/signedjobs/runner.go:163`. The hub has no expiry of its own on the queue (grep `expire` near `HUB/store/store.go:3764`).", + "6": "No. A box off past the end polls, gets the job, and fails at the time check `AG/internal/authz/verifier.go:161`. Clock skew allowed: `DefaultClockSkew` 2 minutes `AG/internal/authz/verifier.go:21`.", + "7": "A rejected job is cleared; a job that passes the check but fails to run is also cleared and must be re-signed, default branch at `AG/internal/signedjobs/runner.go:186`. No retry.", + "8": "TEST `TestVerify_RejectsPerStage` `AG/internal/authz/verify_test.go:82`; DOC `AR/04-control-plane-authorization.md:78`." + }, + "extra": [] + }, + { + "title": "12. Operational signing key and cold recovery key", + "answers": { + "1": "The two keys every agent trusts for destructive orders. Operator owns them; both live on DooPlex today, DOC `AR/04-control-plane-authorization.md:111` and `AR/04-control-plane-authorization.md:96`.", + "2": "NO END DATE on either key in any source. The only dated rule: \"Revisit on the first sale\", in \u00a73.1 starting `AR/04-control-plane-authorization.md:111`.", + "3": "The operator. Planned rotation is a key-signs-key order `AR/04-control-plane-authorization.md:143`; rotating a signer is described as not built `AR/04-control-plane-authorization.md:133`.", + "4": "Both keys lost: no update, no config bundle, no OS step for any box until someone visits each box, DOC `AR/04-control-plane-authorization.md:148`. Every household.", + "5": "Nobody; there is no date. The loss case is in the recovery runbook row S6 `RB/total-loss-of-dooplex.md:39`.", + "6": "Yes. A box off for any time still trusts the same pinned keys. If the key was rotated while it was off: NOT FOUND how it learns the new key (searched \u00a74 `AR/04-control-plane-authorization.md:137-149`).", + "7": "NOT FOUND: no source says what state a half-done re-pin leaves (searched \u00a74 `AR/04-control-plane-authorization.md:137-149`).", + "8": "DOC `AR/04-control-plane-authorization.md:137-149`. Rotation never run on a real box as far as these files say: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "13. Operator actions waiting for a box (24 hours; reboot 30 minutes)", + "answers": { + "1": "Buttons the operator presses in the hub that a box must answer (restart, reboot and so on). Operator. `OperatorActionTTL` `HUB/store/opactions.go:62`.", + "2": "24 hours unanswered: `OperatorActionTTL` `HUB/store/opactions.go:62`. A host reboot order: 30 minutes, `HostOperatorActionRebootTTL` `HUB/store/opactions.go:47`.", + "3": "The operator presses again. No renewal.", + "4": "The action is marked expired with \"did not answer within a day\" `HUB/store/opactions.go:222` and `HUB/store/opactions.go:296`; nothing happens on the box.", + "5": "Nobody before; the operator sees the expired state on the hub page.", + "6": "Safe by design: a box back after the end does not run the stale order. The operator presses again.", + "7": "Not applicable; an expired action is a final state.", + "8": "DOC `AR/04-control-plane-authorization.md:184` names `TestOpActions_ClosedList` and `TestOperatorActions_ClosedList`; SOURCE `HUB/store/opactions.go:62`." + }, + "extra": [] + }, + { + "title": "14. Recovery mode for a host (the door a rebuilt box uses)", + "answers": { + "1": "A switch the operator sets on a host so a rebuilt box may re-enrol and fetch its restore data. Operator, global key only, `handleSetRecoveryMode` `HUB/api/dr.go:34`.", + "2": "30 minutes by default, `defaultRecoveryTTL` `HUB/api/dr.go:21`; at most 4 hours, `maxRecoveryTTL` `HUB/api/dr.go:22`; clamp at `HUB/api/dr.go:52`.", + "3": "The operator sets it again. No lead time.", + "4": "A restore that takes longer than the window: the box's re-enrol call is refused and the household's recovery stops until the operator re-opens it.", + "5": "Nobody. NOT FOUND: grep `defaultRecoveryTTL|maxRecoveryTTL` shows no warning before the end.", + "6": "No: a box that starts its recovery after the window fails at the re-enrol step. TEST `TestReEnroll_GatedOnRecoveryMode` `HUB/api/dr_test.go:14`.", + "7": "Nothing is left half-done by the expiry itself; the operator opens a new window.", + "8": "TEST `TestRestoreDirective_GatedAndExpires` `HUB/api/dr_test.go:70`; SOURCE `HUB/api/dr.go:21-22`." + }, + "extra": [] + }, + { + "title": "15. Break-glass credential (the host's emergency root password)", + "answers": { + "1": "A per-host emergency password, sealed in the hub, shown only to a logged-in operator. Operator. `SaveHostRecoveryCredential` `HUB/store/host_recovery.go:28`; reveal `HUB/web/hosts.go:983`.", + "2": "NO END DATE: last write wins, day-0 sets it and a rotate re-sets it `HUB/store/host_recovery.go:20`. Paper rule: re-print the sheet \"when a key rotates, and yearly\" `RB/break-glass-sheet.md:75`.", + "3": "The operator, yearly by the paper rule; no code reminds him.", + "4": "If the sealed copy cannot be opened (seal key, block 8) or the paper is stale, the operator has no way into a host whose agent is dead.", + "5": "Nobody before. Each reveal writes the event `recovery_credential_revealed` `HUB/web/hosts.go:1010`.", + "6": "Yes, the password does not age on the box.", + "7": "NOT FOUND what a half-done rotate leaves (hub updated, box not, or the reverse); searched `HUB/store/host_recovery.go:20-43` only.", + "8": "SOURCE `HUB/store/host_recovery.go:20-28`; session-only route `HUB/web/routes.go:130`; DOC `RB/break-glass-sheet.md:75`. A yearly re-print has no record: NOBODY KNOWS." + }, + "extra": [] + }, + { + "title": "16. Vouched artifact manifest and the \"golden not vouched\" alarm", + "answers": { + "1": "The hub's list of what a fresh install may fetch (agent, golden, checksums), vouched by the operator. Read by `resolve_artifacts` `E/scripts/felhom-host-install.sh:467`.", + "2": "NO END DATE on a vouch. An age rule on the gap: a published golden not vouched after 2 hours raises an alarm, `goldenVouchGrace` `HUB/monitor/golden_vouch.go:42`, checked every 15 minutes `HUB/monitor/golden_vouch.go:43`.", + "3": "The operator vouches; CC never does, DOC `AR/09-update-architecture.md:1022`.", + "4": "An old vouch means a fresh install gets the older controller, message at `HUB/monitor/golden_vouch.go:92`. If the manifest is not served the install dies `E/scripts/felhom-host-install.sh:472`.", + "5": "The operator, 2 hours after the publish, by the event built at `HUB/monitor/golden_vouch.go:92`. If Gitea cannot be read there is no verdict `HUB/monitor/golden_vouch.go:65`.", + "6": "A new box after a long time installs what is vouched and then updates to the floor; a vouched version already pruned from the registry fails at the download (see block 18).", + "7": "An artifact with no expected checksum is refused `E/scripts/felhom-host-install.sh:531`; the bootstrap unit retries the install every 30 s `E/scripts/iso/felhom-bootstrap.sh:430`.", + "8": "TEST `TestGoldenVouchChecker` `HUB/monitor/golden_vouch_test.go:15`; SOURCE `E/scripts/felhom-host-install.sh:467-472`." + }, + "extra": [] + }, + { + "title": "17. The golden's age rule and its waiver file", + "answers": { + "1": "A gate that fails a push when the newest release is ahead of the newest baked golden and that golden is old. Operator and CC. `GOLDEN_MAX_AGE_DAYS` `E/scripts/golden_currency_gate.py:169`.", + "2": "Golden: 7 days from the date in its evidence folder name, `GOLDEN_MAX_AGE_DAYS` `E/scripts/golden_currency_gate.py:169`. Waiver: its own `expires` line, at most 14 days after issue, `WAIVER_MAX_DAYS` `E/scripts/golden_currency_gate.py:167`.", + "3": "Operator bakes and vouches weekly, Sunday night, DOC `AR/09-update-architecture.md:997`. A waiver is renewed by editing two dates, DOC `RB/RUNBOOK-manual-build.md:291`.", + "4": "The gate goes red and blocks pushes for the operator and CC; households are not touched. Expired check at `E/scripts/golden_currency_gate.py:354`.", + "5": "The gate prints the age on every run `E/scripts/golden_currency_gate.py:422`. No warning before day 7 or before the waiver's end was found.", + "6": "Not a box matter.", + "7": "A waiver that is malformed or too long is refused out loud (exit 2), `read_waiver` `E/scripts/golden_currency_gate.py:287`. Candidate: the waiver file `WAIVER_PATH` `E/scripts/golden_currency_gate.py:162` is absent in this clone, and the newest evidence folder under documentation/tests is dated 2026-11-03, 8 days before today.", + "8": "SOURCE `E/scripts/golden_currency_gate.py:162-169`; DOC `AR/09-update-architecture.md:1362` and `RB/RUNBOOK-manual-build.md:279`. Test names in the gate's test file NOT FOUND (grep `def test` returned nothing)." + }, + "extra": [] + }, + { + "title": "18. Which agent versions stay downloadable (D50)", + "answers": { + "1": "The rule for which published agent packages are kept in the registry. Operator prunes by hand; CI checks. File `AG/scripts/retention-policy.json:23-32`.", + "2": "By count, not date: newest 5 (`generic_versions_kept` `AG/scripts/retention-policy.json:23`) plus two pins, `hub_min_agent` and `day0_agent` `AG/scripts/retention-policy.json:30-31`. Recorded 2026-11-10 `AG/scripts/retention-policy.json:27`.", + "3": "The session that records a vouch or a new floor updates the pins in the same commit, rule text `AG/scripts/retention-policy.json:12-15`.", + "4": "A box on a version outside the set cannot fetch its own binary again, DOC `AR/09-update-architecture.md:1370-1373`. A deleted git tag ends the ability to install that version `AG/scripts/retention-policy.json:20-21`.", + "5": "CI only: `AG/scripts/check-published-versions.py:102` reads the file. No warning to anyone when a box's version is about to fall out of the newest 5 was found.", + "6": "A box off while 5 newer versions ship comes back and updates forward by a signed job; what it does if it needs its OLD binary: fails at the download, per `AR/09-update-architecture.md:1372`.", + "7": "A pin naming a version with no tag is a CI failure, never a skip `AG/scripts/retention-policy.json:14-15`.", + "8": "DOC `AR/09-update-architecture.md:1370-1376`; SOURCE `AG/scripts/retention-policy.json:23-32`. The pins are hand copies of hub state `AG/scripts/retention-policy.json:12`: whether they match today, NOBODY KNOWS from files." + }, + "extra": [] + }, + { + "title": "19. The installer ISO (Proxmox base, installer version, first boot)", + "answers": { + "1": "The USB image a household or volunteer installs from. Product; built by `E/scripts/iso/build-felhom-iso.sh:51`, served from Cloudflare R2, DOC `RB/total-loss-of-dooplex.md:125`.", + "2": "NO END DATE in code. `ISO_VERSION` is 1.30.0 `E/scripts/iso/build-felhom-iso.sh:51`. The Proxmox ISO and its checksum are build ARGUMENTS, not pinned: `E/scripts/iso/build-felhom-iso.sh:194`; the example names 9.2-1 `E/scripts/iso/README.md:40`.", + "3": "The operator rebuilds. No cadence found (grep `days|stale|too old` in `RB/iso-release-gate.md:1-230` returned nothing on age).", + "4": "An old ISO installs an old Proxmox base; the install script itself is fetched fresh at first boot (comment block at `E/scripts/iso/build-felhom-iso.sh:44-52`), so the hub decides the agent and golden.", + "5": "Nobody. The build warns only if golden and floor are not given: unenforced branch `E/scripts/iso/build-felhom-iso.sh:79`.", + "6": "A too-old ISO on first boot: registers, pairs, fetches the current install script; first failing step would be that fetch or the manifest `E/scripts/felhom-host-install.sh:472`. No \"ISO too old\" check found (grep `too old|min.*version` in bootstrap).", + "7": "A failed install is retried every 30 s `E/scripts/iso/felhom-bootstrap.sh:430`; a failed script fetch is retried `E/scripts/iso/felhom-bootstrap.sh:396`.", + "8": "SOURCE `E/scripts/iso/build-felhom-iso.sh:222` (checksum of the given ISO is verified). How old an ISO still installs cleanly: NOBODY KNOWS. Build image is `FROM debian:trixie`, not pinned `E/scripts/iso/Dockerfile.assistant:10`." + }, + "extra": [] + }, + { + "title": "20. The certificate the hub's address serves", + "answers": { + "1": "The TLS certificate for the hub's public name, used by every box and the operator. Operator's cluster issues it: annotation `E/manifests/hub.yaml:342`, stored under the secret name at `E/manifests/hub.yaml:360`.", + "2": "NOT FOUND in these sources: the issuer `letsencrypt-prod` is only named `E/manifests/hub.yaml:342`; its definition and renewal timing are in the DooPlex cluster repository, outside this task. Searched grep `ClusterIssuer|renewBefore|duration` in E/manifests.", + "3": "cert-manager on DooPlex, by the annotation `E/manifests/hub.yaml:342`. Lead time NOT FOUND (same search).", + "4": "If it lapses: boxes that pin or verify the hub cannot report or fetch config; households get no alarms and no updates; the operator page shows a browser warning.", + "5": "NOT FOUND: grep `cert|expir` in `HUB/monitor/golden_vouch.go:1-100` and across HUB/monitor finds no certificate-age alarm.", + "6": "A box off past a renewal: depends on what it pins (box helper's item). From hub files: NOT FOUND.", + "7": "NOT FOUND. The renewal needs the challenge path reachable; the ingress is limited to two countries in the block starting `E/manifests/hub.yaml:342`. Candidate question: does that block the issuer's check.", + "8": "SOURCE `E/manifests/hub.yaml:342` and `E/manifests/hub.yaml:360` only. Dates: NOBODY KNOWS from files; needs a measurement." + }, + "extra": [] + }, + { + "title": "21. Operator login session on the hub", + "answers": { + "1": "The operator's browser session cookie. Operator. `SessionCookieName` `HUB/web/server.go:930`; sessions are held in memory.", + "2": "7 days from login: session end `HUB/web/server.go:949`, cookie `MaxAge` `HUB/web/server.go:964`. Also ends at any hub restart, because the map is in memory (cleanup `CleanupSessions` `HUB/web/server.go:199`).", + "3": "The operator logs in again. No sliding renewal found: the check only compares the fixed end `HUB/web/server.go:901`.", + "4": "Session-only buttons (delete, reset, reveal, tokens) stop until he logs in; route rule example `HUB/web/routes.go:130`. Scripts with a token are not touched.", + "5": "Nobody; the next click shows the login page.", + "6": "Not a box matter.", + "7": "Not applicable. Basic auth with the password still works as a second door `HUB/web/server.go:909`.", + "8": "TEST `TestR136_LoginSetsHostPrefixedCookie` `HUB/web/r136_host_cookie_test.go:16` (cookie name); the 7 days are SOURCE only `HUB/web/server.go:949`." + }, + "extra": [] + }, + { + "title": "22. Operator access tokens, among them CC's hub token", + "answers": { + "1": "Named, scoped tokens for scripts, shown once, stored hashed. Operator creates them in a browser session; CC holds one, with `overview:read` and `floor:write`, DOC `AR/09-update-architecture.md:1022`.", + "2": "90 days by default, `tokenDefaultDays` `HUB/web/tokens.go:29`; at most 365, `tokenMaxDays` `HUB/web/tokens.go:30`; \"never\" is not offered `HUB/web/tokens.go:254`. End computed at `HUB/web/tokens.go:180`.", + "3": "The operator makes a new token and puts it in CC's store. There is no renew; no lead time is set anywhere.", + "4": "CC's floor raise and overview reads get 401 `HUB/web/tokens.go:95`; a controller release cannot be carried to boxes by CC until the operator acts. Households wait for the update.", + "5": "Nobody is warned. NOT FOUND: grep `token.*(expiring|days left)|token_expir` in the hub finds only comments. The page shows the date and the state \"expired\" `HUB/web/tokens.go:245`.", + "6": "Not a box matter.", + "7": "Not applicable; an expired token stays listed as expired. Column `expires_at` `HUB/store/api_tokens.go:52`.", + "8": "TEST `TestToken_NoTokenUnknownExpiredRevoked401` `HUB/web/tokens_test.go:310`; DOC `AR/05-hub-architecture.md:655-669`. The end DATE of CC's token is in no document (grep `HUB_TOKEN` in AR, RB, operations, STATUS): NOBODY KNOWS without the hub page." + }, + "extra": [] + }, + { + "title": "23. A household's own Cloudflare token (hub side)", + "answers": { + "1": "A household that brings its own domain gives a Cloudflare API token; the hub checks what it can reach before saving. Household owns it. `HUB/cloudflare/reach.go:14`.", + "2": "NO END DATE known to the hub: the check at `HUB/cloudflare/reach.go:14` is about zones, not age. Whatever end the household set at Cloudflare is not read (grep `expire|rotat` in the cloudflare package: nothing).", + "3": "The household, at Cloudflare; then the operator saves the new token in the hub.", + "4": "If it lapses, the household's remote address stops being managed; candidate: first sign is a failed DNS or tunnel call on the box.", + "5": "Nobody. NOT FOUND: same grep, no expiry field stored.", + "6": "Box helper's item; from hub files NOT FOUND.", + "7": "NOT FOUND.", + "8": "SOURCE `HUB/cloudflare/reach.go:14`; the rest NOBODY KNOWS from hub files." + }, + "extra": [] + }, + { + "title": "24. The mail service key (all household and operator mail)", + "answers": { + "1": "The key the hub uses to send every mail: alarms, bind links, claim codes. Operator's account at the mail service; named in `E/manifests/hub.yaml:148-152`, value not opened.", + "2": "NO END DATE stated in any document read: `RB/secrets.md:33` describes create and rotate, with no date; inventory row `AR/_recovery-inventory-2026-07-28.md:502`.", + "3": "The operator re-mints at the service dashboard, DOC `RB/total-loss-of-dooplex.md:47`.", + "4": "All mail stops: no alarm reaches the operator or a household, no bind link, no claim code. Everyone.", + "5": "NOT FOUND: no check that mail still goes out on a schedule was found in the documents read (grep `expir|renew` in `RB/secrets.md` and STATUS.md).", + "6": "Not a box matter; boxes send through the hub.", + "7": "A failed bind mail deletes its token `HUB/web/selfbind_mint.go:82`; for other mail kinds NOT FOUND in this pass.", + "8": "DOC `RB/secrets.md:33` and `RB/total-loss-of-dooplex.md:47`. Whether the key or the plan has an end date: NOBODY KNOWS from files." + }, + "extra": [] + }, + { + "title": "25. The hub's global bearer key", + "answers": { + "1": "The operator-tier key for the hub API (recovery mode, peer re-add). Operator. Named in `E/manifests/hub.yaml:157-161`; inventory row `AR/_recovery-inventory-2026-07-28.md:501`.", + "2": "NO END DATE. Rotation is a supervised runbook `RB/secrets.md:124`.", + "3": "The operator, with an ordered procedure `RB/secrets.md:124`; no cadence stated.", + "4": "A rotation done out of order locks out whatever still holds the old key; DOC `RB/secrets.md:124` calls the order load-bearing.", + "5": "Nobody; no date.", + "6": "Boxes use per-box keys bound to their box since 2026-10-12, DOC `AR/13-threat-model.md:70`; a box still holding only the old global key after a rotation: NOT FOUND what it does.", + "7": "NOT FOUND beyond the runbook order.", + "8": "DOC only. Two documents disagree on whether it was rotated: \"rotated\" at `AR/13-threat-model.md:70`, \"when last rotated\" unknown at `AR/_recovery-inventory-2026-07-28.md:1030`. Candidate for a doc fix." + }, + "extra": [] + }, + { + "title": "26. Pinned tool versions in builds and CI", + "answers": { + "1": "Versions of the tools that build and check the product. Operator and CC. Hub build image E/hub/Dockerfile:3, run image E/hub/Dockerfile:24, site publisher `E/scripts/publish-site-cloudflare.sh:28`.", + "2": "NO END DATE in code. Hub images are pinned by digest E/hub/Dockerfile:3; the ISO helper image is the moving tag at `E/scripts/iso/Dockerfile.assistant:10`; the mailer image is `latest` at `E/manifests/contact-mailer.yaml:54`.", + "3": "Whoever bumps the pin. The nightly security watch runs at 03:40 UTC `E/.gitea/workflows/security-watch.yml:22`; nightly gates at 03:20 `E/.gitea/workflows/gates.yml:27`.", + "4": "A pinned digest removed upstream breaks the hub build for the operator; a moving tag changes the ISO build without a commit.", + "5": "The security watch treats a release line with no fixed version as end of life, `E/scripts/security_watch.py:292`; it reports to the operator.", + "6": "Not a box matter.", + "7": "A CI job that does not run leaves no result; whether a missed nightly is noticed: NOT FOUND in the workflow file `E/.gitea/workflows/gates.yml:27`.", + "8": "SOURCE E/hub/Dockerfile:3 and `E/scripts/security_watch.py:292`. Support end dates of Go, Alpine, wrangler: no document states one." + }, + "extra": [] + }, + { + "title": "27. Domains, registrars and the home dynamic-DNS name", + "answers": { + "1": "NOT MEASURED. The product's two domains and the dynamic name the hub's address points to. Operator. Listed as row S13 `RB/total-loss-of-dooplex.md:46`; the hub name is a CNAME to the dynamic name `RB/total-loss-of-dooplex.md:73`.", + "2": "NOT FOUND: no document gives a registrar name or a renewal date. Searched grep `expir|renew|valid until|until 20` in documentation/operations, RB, AR and STATUS.md.", + "3": "The operator. NOT FOUND how long before.", + "4": "If a domain or the dynamic name lapses, every box loses the hub (reports, alarms, updates, bind links) and the site and installer disappear. Every household.", + "5": "NOT FOUND. Presumably the registrar's own mails to the operator's mailbox, row S12 `RB/total-loss-of-dooplex.md:45`; no document says so.", + "6": "A box comes back by itself once the name resolves again; while it does not, the first failing step is the hub lookup.", + "7": "NOT FOUND.", + "8": "DOC `RB/total-loss-of-dooplex.md:46`. WHERE THE OPERATOR READS THE DATE: NOT FOUND. NOBODY KNOWS from files." + }, + "extra": [] + }, + { + "title": "28. Cloudflare account, the site publish token and the ISO store keys", + "answers": { + "1": "NOT MEASURED. The account that serves DNS for both domains, the site, the installer script and the ISO store. Operator. Row S11 `RB/total-loss-of-dooplex.md:44`; site and ISO there since 2026-10-30 `RB/total-loss-of-dooplex.md:125`.", + "2": "NOT FOUND: no end date for the account, the token `CF_PAGES_TOKEN` (named at `E/scripts/publish-site-cloudflare.sh:12`) or any R2 key. Searched grep `R2_ACCESS|R2 key|expir` in AR, RB, operations.", + "3": "The operator at the Cloudflare dashboard.", + "4": "Token lapse: CC cannot publish the site or installer script; the old one stays served. Account trouble: DNS, site, ISO downloads and households' tunnels, per row S11 `RB/total-loss-of-dooplex.md:44`.", + "5": "NOT FOUND.", + "6": "A new box needs the installer script from the site at first boot `E/scripts/iso/felhom-bootstrap.sh:396`; it retries until it is served.", + "7": "A failed publish leaves the previous site live (candidate; not stated in the script header `E/scripts/publish-site-cloudflare.sh:12`).", + "8": "DOC `RB/total-loss-of-dooplex.md:44`. WHERE THE OPERATOR READS THE DATE: NOT FOUND." + }, + "extra": [] + }, + { + "title": "29. Hetzner account, the rented endpoint and the storage boxes", + "answers": { + "1": "NOT MEASURED. The paid account behind ep0 (tunnel endpoint and off-site store) and the Storage Boxes. Operator. Row S8 `RB/total-loss-of-dooplex.md:41`; API token named in `E/manifests/hub.yaml:222-226`.", + "2": "NOT FOUND: no document gives a billing date, a contract end or an API token end. Searched grep `expir|renew|invoice|billing` in RB, AR, operations.", + "3": "The operator; paid monthly by the provider's own cycle, not stated in any document read.", + "4": "If the account is closed or unpaid: every household's off-site copies and the PBS tunnel stop; the hub cannot mint or reset sub-accounts.", + "5": "NOT FOUND. Hub-side sign would be the off-site stale alarm after 8 days, `offsiteBackupStaleAfter` `HUB/monitor/deadline_tiers.go:28`.", + "6": "A box comes back by itself when the endpoint is back; first failing step is the off-site push.", + "7": "NOT FOUND.", + "8": "DOC `RB/total-loss-of-dooplex.md:41`. WHERE THE OPERATOR READS THE DATE: NOT FOUND." + }, + "extra": [] + }, + { + "title": "30. Tailscale node keys (operator's remote way in)", + "answers": { + "1": "NOT MEASURED. The operator's private network to DooPlex and the two demo boxes. Operator. Table `E/documentation/operations/tailscale.md:11-13`.", + "2": "Document date: the HP demo box's key expires 2027-01-17, written as OPEN at `E/documentation/operations/tailscale.md:41`. The other two have expiry disabled `E/documentation/operations/tailscale.md:11-12`.", + "3": "The operator, in the admin console (Machines, Disable key expiry) `E/documentation/operations/tailscale.md:43`. No lead time set.", + "4": "That node drops off the private network `E/documentation/operations/tailscale.md:91`; the operator loses remote reach to it. No household is on it.", + "5": "NOT FOUND: no reminder is named. WHERE THE OPERATOR READS THE DATE: the Machines page, \"Expiry disabled\" badge `E/documentation/operations/tailscale.md:92`.", + "6": "The box keeps running; only the operator's remote path is gone until someone re-authenticates on site or over the LAN.", + "7": "NOT FOUND.", + "8": "DOC `E/documentation/operations/tailscale.md:41`. Candidate: the section at `E/documentation/operations/tailscale.md:89-92` says \"both\" nodes while the table has three." + }, + "extra": [] + }, + { + "title": "31. Gitea and registry credentials", + "answers": { + "1": "NOT MEASURED. The login the cluster and the operator use to pull and push images and code. Operator. Row S15 `RB/total-loss-of-dooplex.md:48`; the pull secret is a scoped token `RB/secrets.md:231`.", + "2": "NOT FOUND: no end date for the scoped token or for CC's own Gitea token. Searched grep `expir` in `RB/secrets.md` and cc-secrets ARCHITECTURE.md.", + "3": "The operator, in Gitea.", + "4": "If the pull token lapses: the hub pod cannot be re-pulled on a restart, and boxes pulling the controller image through the same registry would fail (box helper's item).", + "5": "NOT FOUND.", + "6": "NOT FOUND from these files.", + "7": "NOT FOUND.", + "8": "DOC `RB/secrets.md:231` and `RB/total-loss-of-dooplex.md:48`. WHERE THE OPERATOR READS THE DATE: NOT FOUND." + }, + "extra": [] + }, + { + "title": "32. CC's secret store: its key and the tokens inside (GitHub advisory token, hub token)", + "answers": { + "1": "NOT MEASURED. The store CC uses through `sec`, and the key that opens it. Operator owns; row S19 `RB/total-loss-of-dooplex.md:52`; design in /mnt/5_hdd/felhom.eu/git/cc-secrets/ARCHITECTURE.md (lines 47-48 give the key location).", + "2": "NOT FOUND: the design document states no expiry for any stored secret (grep `expir|rotat|days|lifetime`). The hub token has one by code, block 22. A GitHub advisory token is named only in CC's memory index, in no document (grep `GH_ADVISORY_TOKEN` in AR, RB, operations, STATUS).", + "3": "The operator stores a new value; only he may delete one.", + "4": "An expired GitHub token: the nightly security watch `E/.gitea/workflows/security-watch.yml:22` loses its source. An expired hub token: block 22.", + "5": "NOT FOUND.", + "6": "Not a box matter.", + "7": "NOT FOUND.", + "8": "DOC `RB/total-loss-of-dooplex.md:52`. WHERE THE OPERATOR READS THE DATES: NOT FOUND. NOBODY KNOWS from files." + }, + "extra": [] + }, + { + "title": "33. SSH keys between the hub, DooPlex and the endpoint", + "answers": { + "1": "NOT MEASURED. Keys the hub uses to reach the endpoint (three key files named at `E/manifests/hub.yaml:191`, `E/manifests/hub.yaml:202`, `E/manifests/hub.yaml:209`) and the root key, row S9 `RB/total-loss-of-dooplex.md:42`. Operator.", + "2": "NO END DATE in any document. The only rule: rotate on an endpoint rebuild, DOC `RB/offsite-endpoint.md:313`.", + "3": "The operator, on rebuild.", + "4": "A lost or changed key: the hub cannot add tunnel peers, mint off-site tokens or poke the endpoint; new households get no off-site tier.", + "5": "Nobody; no date.", + "6": "Existing boxes keep their own tunnel and token; only new provisioning fails.", + "7": "NOT FOUND.", + "8": "DOC `RB/offsite-endpoint.md:313`; row S9 says one authorized key's owner is \"not recorded\" `RB/total-loss-of-dooplex.md:42`. NOBODY KNOWS who holds it." + }, + "extra": [] + }, + { + "title": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "answers": { + "1": "NOT MEASURED. The base systems under every box and the endpoint. Product. Host is Proxmox VE 9.2 on Debian 13, DOC `AR/11-os-updates.md:103`.", + "2": "NOT FOUND: no document states a support end date. Searched grep `EOL|end of life|support ends|until 20` in AR, RB, operations. The major upgrade 9 to 10 is \"later\" `AR/11-os-updates.md:91` and `AR/11-os-updates.md:725`.", + "3": "The operator, by a drilled major upgrade first on a test box `AR/11-os-updates.md:725`. No date.", + "4": "After the upstream end, boxes stop getting security fixes; every household, silently.", + "5": "NOT FOUND. The security watch flags a runtime line with no fixed version as end of life `E/scripts/security_watch.py:292`; whether Debian or Proxmox lines are in its list was not checked.", + "6": "A box off across a major release: NOT FOUND what its updater does first.", + "7": "NOT FOUND.", + "8": "DOC `AR/11-os-updates.md:103`. The paid repository is a later money decision `AR/11-os-updates.md:662`. WHERE THE OPERATOR READS THE DATE: NOT FOUND. \u00b7 Diagnostic log bundles kept 72 hours: `logBundleTTL` `HUB/store/logbundle.go:29`. \u00b7 Hub access log kept 90 days: `DefaultKeep` `HUB/accesslog/accesslog.go:44`. \u00b7 Deleted-customer audit rows kept 365 days: `DeletedCustomerAuditKeep` `HUB/store/deleted_customer_audit.go:23`. \u00b7 App telemetry pruned after 90 days and stale issues after 30: `E/hub/cmd/hub/main.go:1134-1139`. \u00b7 Hub prune runs daily at 04:30: `prune_schedule` `E/manifests/hub.yaml:83`. \u00b7 Kernel-reboot notice valid 24 hours: `KernelNoticeValid` `HUB/osupdates/kernel.go:48`. \u00b7 Local backup counted stale after 26 hours: `backupStaleAfter` `HUB/monitor/deadline.go:29`. \u00b7 Off-site backup counted stale after 8 days: `offsiteBackupStaleAfter` `HUB/monitor/deadline_tiers.go:28`. \u00b7 Restore test counted proven for 7 to 12 days: `HUB/monitor/restoretest.go:68-74`. \u00b7 Household quiet-box mail after 7 days: `householdLivenessQuiet` `HUB/notify/dispatcher.go:1018`. \u00b7 Line-test result stale after 15 days: `lineStaleAfter` `HUB/web/system.go:174`. \u00b7 Hub database snapshot catch-up age 6 hours: `CatchUpAge` `HUB/dbsnap/dbsnap.go:36`. \u00b7 Escrow still pending after 7 days raises an operator alarm: `escrowPendingAfter` `HUB/monitor/offsite_escrow_pending.go:37`. \u00b7 Escrow ceremony: an unclaimed recovery code is void after 10 minutes: `RB/RUNBOOK-escrow-ceremony.md:46`. \u00b7 Appliance row badged stale after 7 days without a poll: `applianceStaleAfter` `HUB/web/appliances.go:21`. \u00b7 Waiver example in the runbook carries dates from September 2026: `RB/RUNBOOK-manual-build.md:278-279`. \u00b7 Hub image tag in the manifest: `E/manifests/hub.yaml:135`. \u00b7 Same issuer annotation on the website ingress: `E/manifests/webpage.yaml:141`. \u00b7 The definition of the certificate issuer, certificate lifetime and renew-before: searched `ClusterIssuer|renewBefore|duration` in E/manifests; it lives in the DooPlex cluster repository, outside this task. \u00b7 Any warning before an operator access token ends: searched `token.*(expiring|expires soon|about to expire|days left)|token_expir` in the hub. \u00b7 The end date of CC's hub token in a document: searched `HUB_TOKEN` in AR, RB, documentation/operations and STATUS.md. \u00b7 An expiry on the claim code or on the pairing code: searched `expire|ttl|valid` in the claim engine and the appliance store. \u00b7 A hub-side expiry on queued signed jobs: searched `expire` around `EnqueueSignedJob`. \u00b7 A pinned Proxmox ISO version or checksum in the build script: they are arguments; searched `sha256|pve-iso|9\\.[0-9]-[0-9]` in the ISO script and the release gate runbook. \u00b7 An \"ISO too old\" check at first boot: searched `too old|min.*version|ISO_VERSION` in the bootstrap script. \u00b7 The waiver file documentation/tests/golden-waiver.yml in this clone (it exists only in sibling worktrees, which are not sources). \u00b7 Test names in the golden gate's test file: `def test` returned nothing. \u00b7 Registrar names, domain renewal dates, the dynamic-DNS account's renewal: searched `expir|renew|valid until|until 20|No-IP|noip`. \u00b7 Billing or contract dates for Hetzner, Cloudflare, the mail service, Tailscale: searched `expir|renew|invoice|billing`. \u00b7 Expiry of Gitea tokens, registry logins, R2 keys, the GitHub advisory token, SSH keys: searched `expir|GH_ADVISORY_TOKEN|R2_ACCESS|R2 key` in AR, RB, operations, STATUS.md and the cc-secrets design document. \u00b7 Support end dates for Debian 13, Proxmox VE 9 or PBS: searched `EOL|end of life|support ends`. \u00b7 Anything dated in STATUS.md: searched `expir|renew|HUB_TOKEN|waiver|trixie|2027|2028|2029|2030`. \u00b7 WireGuard peer expiry or rotation in the hub: searched `expire|rotat|stale|handshake` in the hub's wg files. \u00b7 Merged: self-bind mint and resend into block 1; escrow retention and abandon into 6; sub-account password and seal key into 8; vouch and manifest into 16; golden age and waiver into 17; CC store, hub token date and GitHub token into 32." + }, + "extra": [ + "## Tiny", + "## Not found" + ] + } + ], + "pointers": [ + { + "ptr": "HUB/store/store.go:796-806", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Self-bind (connect) link in the household's e-mail", + "q": "1" + }, + { + "ptr": "HUB/web/selfbind_mint.go:20", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Self-bind (connect) link in the household's e-mail", + "q": "1" + }, + { + "ptr": "HUB/web/selfbind_mint.go:20", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Self-bind (connect) link in the household's e-mail", + "q": "2" + }, + { + "ptr": "HUB/web/selfbind_mint.go:77", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Self-bind (connect) link in the household's e-mail", + "q": "2" + }, + { + "ptr": "HUB/store/selfbind.go:39", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Self-bind (connect) link in the household's e-mail", + "q": "2" + }, + { + "ptr": "HUB/store/selfbind.go:21", + "file": "OK", + "ident": "IDENT-MISS(HUB/store/selfbind.go:39.)", + "ending": "1. Self-bind (connect) link in the household's e-mail", + "q": "2" + }, + { + "ptr": "HUB/web/selfbind.go:384", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Self-bind (connect) link in the household's e-mail", + "q": "3" + }, + { + "ptr": "HUB/web/selfbind.go:382", + "file": "OK", + "ident": "IDENT-MISS(HUB/web/selfbind.go:384,)", + "ending": "1. Self-bind (connect) link in the household's e-mail", + "q": "3" + }, + { + "ptr": "HUB/web/selfbind_mint.go:97", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Self-bind (connect) link in the household's e-mail", + "q": "3" + }, + { + "ptr": "HUB/web/selfbind.go:206", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Self-bind (connect) link in the household's e-mail", + "q": "4" + }, + { + "ptr": "HUB/web/selfbind_mint.go:1-110", + "file": "OK", + "ident": "IDENT-MISS(expir|remind)", + "ending": "1. Self-bind (connect) link in the household's e-mail", + "q": "5" + }, + { + "ptr": "HUB/web/selfbind_mint.go:93", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Self-bind (connect) link in the household's e-mail", + "q": "5" + }, + { + "ptr": "HUB/web/selfbind.go:440", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Self-bind (connect) link in the household's e-mail", + "q": "6" + }, + { + "ptr": "HUB/web/selfbind.go:413", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Self-bind (connect) link in the household's e-mail", + "q": "6" + }, + { + "ptr": "HUB/web/selfbind_mint.go:82", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Self-bind (connect) link in the household's e-mail", + "q": "7" + }, + { + "ptr": "HUB/web/selfbind_resend_test.go:21", + "file": "OK", + "ident": "IDENT-OK", + "ending": "1. Self-bind (connect) link in the household's e-mail", + "q": "8" + }, + { + "ptr": "HUB/web/selfbind_mint.go:20", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Self-bind (connect) link in the household's e-mail", + "q": "8" + }, + { + "ptr": "AR/05-hub-architecture.md:359", + "file": "OK", + "ident": "NO-IDENT", + "ending": "1. Self-bind (connect) link in the household's e-mail", + "q": "8" + }, + { + "ptr": "HUB/store/store.go:2084", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Customer claim code (the household's first password step)", + "q": "1" + }, + { + "ptr": "HUB/store/store.go:2084", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Customer claim code (the household's first password step)", + "q": "2" + }, + { + "ptr": "HUB/claim/engine.go:30", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Customer claim code (the household's first password step)", + "q": "2" + }, + { + "ptr": "HUB/claim/engine.go:30", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Customer claim code (the household's first password step)", + "q": "4" + }, + { + "ptr": "HUB/store/store.go:2124", + "file": "OK", + "ident": "IDENT-OK", + "ending": "2. Customer claim code (the household's first password step)", + "q": "4" + }, + { + "ptr": "HUB/claim/engine.go:27", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Customer claim code (the household's first password step)", + "q": "4" + }, + { + "ptr": "HUB/store/store.go:2084", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Customer claim code (the household's first password step)", + "q": "6" + }, + { + "ptr": "HUB/claim/engine.go:1-60", + "file": "OK", + "ident": "IDENT-MISS(expire|ttl|valid)", + "ending": "2. Customer claim code (the household's first password step)", + "q": "7" + }, + { + "ptr": "HUB/claim/engine.go:30", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Customer claim code (the household's first password step)", + "q": "8" + }, + { + "ptr": "HUB/store/store.go:2084", + "file": "OK", + "ident": "NO-IDENT", + "ending": "2. Customer claim code (the household's first password step)", + "q": "8" + }, + { + "ptr": "HUB/api/passphrase_scope.go:46", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Retrieval passphrase: the moment it stops giving keys", + "q": "1" + }, + { + "ptr": "HUB/api/passphrase_scope.go:22", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Retrieval passphrase: the moment it stops giving keys", + "q": "2" + }, + { + "ptr": "HUB/api/passphrase_scope.go:40", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Retrieval passphrase: the moment it stops giving keys", + "q": "2" + }, + { + "ptr": "AR/05-hub-architecture.md:270-272", + "file": "OK", + "ident": "NO-IDENT", + "ending": "3. Retrieval passphrase: the moment it stops giving keys", + "q": "3" + }, + { + "ptr": "HUB/api/passphrase_scope.go:42", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Retrieval passphrase: the moment it stops giving keys", + "q": "4" + }, + { + "ptr": "HUB/api/passphrase_scope.go:42", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Retrieval passphrase: the moment it stops giving keys", + "q": "5" + }, + { + "ptr": "HUB/api/r1159_passphrase_scope_test.go:110", + "file": "OK", + "ident": "IDENT-OK", + "ending": "3. Retrieval passphrase: the moment it stops giving keys", + "q": "8" + }, + { + "ptr": "HUB/store/appliance.go:35", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Pairing code and appliance registration (box console code)", + "q": "1" + }, + { + "ptr": "E/scripts/iso/felhom-bootstrap.sh:50", + "file": "OK", + "ident": "IDENT-MISS(PairingCode)", + "ending": "4. Pairing code and appliance registration (box console code)", + "q": "1" + }, + { + "ptr": "HUB/store/appliance.go:155", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Pairing code and appliance registration (box console code)", + "q": "2" + }, + { + "ptr": "HUB/web/appliances.go:21", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Pairing code and appliance registration (box console code)", + "q": "2" + }, + { + "ptr": "HUB/store/appliance.go:35", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Pairing code and appliance registration (box console code)", + "q": "3" + }, + { + "ptr": "E/scripts/iso/felhom-bootstrap.sh:589", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Pairing code and appliance registration (box console code)", + "q": "4" + }, + { + "ptr": "HUB/web/appliances.go:78", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Pairing code and appliance registration (box console code)", + "q": "5" + }, + { + "ptr": "E/scripts/iso/felhom-bootstrap.sh:54", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Pairing code and appliance registration (box console code)", + "q": "6" + }, + { + "ptr": "E/scripts/iso/felhom-bootstrap.sh:180", + "file": "OK", + "ident": "IDENT-OK", + "ending": "4. Pairing code and appliance registration (box console code)", + "q": "6" + }, + { + "ptr": "E/scripts/iso/felhom-bootstrap.sh:495", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Pairing code and appliance registration (box console code)", + "q": "7" + }, + { + "ptr": "HUB/store/appliance.go:155", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Pairing code and appliance registration (box console code)", + "q": "8" + }, + { + "ptr": "HUB/web/appliances.go:21", + "file": "OK", + "ident": "NO-IDENT", + "ending": "4. Pairing code and appliance registration (box console code)", + "q": "8" + }, + { + "ptr": "HUB/store/deliberateoff.go:18", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. \"Switched off on purpose\" hold (14 days of silence allowed)", + "q": "1" + }, + { + "ptr": "HUB/store/deliberateoff.go:18", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. \"Switched off on purpose\" hold (14 days of silence allowed)", + "q": "2" + }, + { + "ptr": "HUB/store/deliberateoff.go:21", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. \"Switched off on purpose\" hold (14 days of silence allowed)", + "q": "2" + }, + { + "ptr": "HUB/api/deliberate_off.go:42", + "file": "OK", + "ident": "NO-IDENT", + "ending": "5. \"Switched off on purpose\" hold (14 days of silence allowed)", + "q": "5" + }, + { + "ptr": "HUB/monitor/deliberate_off_test.go:138", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. \"Switched off on purpose\" hold (14 days of silence allowed)", + "q": "6" + }, + { + "ptr": "HUB/monitor/deliberate_off_test.go:112", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. \"Switched off on purpose\" hold (14 days of silence allowed)", + "q": "8" + }, + { + "ptr": "HUB/monitor/deliberate_off_test.go:90", + "file": "OK", + "ident": "IDENT-OK", + "ending": "5. \"Switched off on purpose\" hold (14 days of silence allowed)", + "q": "8" + }, + { + "ptr": "HUB/store/store.go:440", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Retained old escrow packages and the 7-day abandon of set-aside off-site data", + "q": "1" + }, + { + "ptr": "HUB/store/store.go:417", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Retained old escrow packages and the 7-day abandon of set-aside off-site data", + "q": "2" + }, + { + "ptr": "HUB/offsitekeys/service.go:448", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Retained old escrow packages and the 7-day abandon of set-aside off-site data", + "q": "2" + }, + { + "ptr": "HUB/offsitekeys/service.go:473", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Retained old escrow packages and the 7-day abandon of set-aside off-site data", + "q": "3" + }, + { + "ptr": "HUB/store/store.go:4360", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Retained old escrow packages and the 7-day abandon of set-aside off-site data", + "q": "4" + }, + { + "ptr": "HUB/store/store.go:1985", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Retained old escrow packages and the 7-day abandon of set-aside off-site data", + "q": "4" + }, + { + "ptr": "HUB/offsitekeys/service.go:440-560", + "file": "OK", + "ident": "IDENT-MISS(DefaultAbandonDelay|remind)", + "ending": "6. Retained old escrow packages and the 7-day abandon of set-aside off-site data", + "q": "5" + }, + { + "ptr": "HUB/offsitekeys/service.go:540", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Retained old escrow packages and the 7-day abandon of set-aside off-site data", + "q": "5" + }, + { + "ptr": "HUB/offsitekeys/service.go:529", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Retained old escrow packages and the 7-day abandon of set-aside off-site data", + "q": "6" + }, + { + "ptr": "HUB/offsitekeys/service.go:540", + "file": "OK", + "ident": "NO-IDENT", + "ending": "6. Retained old escrow packages and the 7-day abandon of set-aside off-site data", + "q": "7" + }, + { + "ptr": "HUB/offsitekeys/service_test.go:134", + "file": "OK", + "ident": "IDENT-OK", + "ending": "6. Retained old escrow packages and the 7-day abandon of set-aside off-site data", + "q": "8" + }, + { + "ptr": "HUB/store/store.go:417", + "file": "OK", + "ident": "IDENT-MISS(TestAbandon_DelayCancelAndScope)", + "ending": "6. Retained old escrow packages and the 7-day abandon of set-aside off-site data", + "q": "8" + }, + { + "ptr": "HUB/offsitekeys/service.go:263", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Off-site clean-up window (the short time a box may delete off-site)", + "q": "1" + }, + { + "ptr": "HUB/offsitekeys/service.go:263", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Off-site clean-up window (the short time a box may delete off-site)", + "q": "2" + }, + { + "ptr": "HUB/offsitekeys/service.go:264", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Off-site clean-up window (the short time a box may delete off-site)", + "q": "2" + }, + { + "ptr": "HUB/offsitekeys/service.go:425", + "file": "OK", + "ident": "NO-IDENT", + "ending": "7. Off-site clean-up window (the short time a box may delete off-site)", + "q": "5" + }, + { + "ptr": "HUB/offsitekeys/service.go:413", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Off-site clean-up window (the short time a box may delete off-site)", + "q": "7" + }, + { + "ptr": "HUB/offsitekeys/service.go:425", + "file": "OK", + "ident": "IDENT-MISS(SweepExpiredWindows)", + "ending": "7. Off-site clean-up window (the short time a box may delete off-site)", + "q": "7" + }, + { + "ptr": "HUB/offsitekeys/service_test.go:99", + "file": "OK", + "ident": "IDENT-OK", + "ending": "7. Off-site clean-up window (the short time a box may delete off-site)", + "q": "8" + }, + { + "ptr": "HUB/offsitekeys/service_test.go:37", + "file": "OK", + "ident": "IDENT-MISS(HUB/offsitekeys/service_test.go:99)", + "ending": "7. Off-site clean-up window (the short time a box may delete off-site)", + "q": "8" + }, + { + "ptr": "E/manifests/hub.yaml:167-171", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Storage Box sub-account password and the hub's seal key", + "q": "1" + }, + { + "ptr": "AR/05-hub-architecture.md:274-279", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Storage Box sub-account password and the hub's seal key", + "q": "2" + }, + { + "ptr": "HUB/hetznerapi/hetznerapi.go:297", + "file": "OK", + "ident": "IDENT-OK", + "ending": "8. Storage Box sub-account password and the hub's seal key", + "q": "2" + }, + { + "ptr": "RB/secrets.md:159", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Storage Box sub-account password and the hub's seal key", + "q": "3" + }, + { + "ptr": "RB/secrets.md:159", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Storage Box sub-account password and the hub's seal key", + "q": "4" + }, + { + "ptr": "HUB/hetznerapi/hetznerapi.go:1-320", + "file": "OK", + "ident": "IDENT-MISS(expir|rotat)", + "ending": "8. Storage Box sub-account password and the hub's seal key", + "q": "5" + }, + { + "ptr": "AR/05-hub-architecture.md:274-279", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Storage Box sub-account password and the hub's seal key", + "q": "6" + }, + { + "ptr": "RB/secrets.md:159", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Storage Box sub-account password and the hub's seal key", + "q": "8" + }, + { + "ptr": "AR/05-hub-architecture.md:274-279", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Storage Box sub-account password and the hub's seal key", + "q": "8" + }, + { + "ptr": "HUB/hetznerapi/hetznerapi.go:297", + "file": "OK", + "ident": "NO-IDENT", + "ending": "8. Storage Box sub-account password and the hub's seal key", + "q": "8" + }, + { + "ptr": "AR/05-hub-architecture.md:373-387", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. PBS off-site token on ep0 and its descriptor on the hub", + "q": "1" + }, + { + "ptr": "AR/05-hub-architecture.md:379-387", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. PBS off-site token on ep0 and its descriptor on the hub", + "q": "2" + }, + { + "ptr": "HUB/pbsdrheal/reconciler.go:63", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. PBS off-site token on ep0 and its descriptor on the hub", + "q": "2" + }, + { + "ptr": "AR/05-hub-architecture.md:382", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. PBS off-site token on ep0 and its descriptor on the hub", + "q": "3" + }, + { + "ptr": "HUB/pbsdrheal/reconciler.go:55-63", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. PBS off-site token on ep0 and its descriptor on the hub", + "q": "4" + }, + { + "ptr": "AR/05-hub-architecture.md:400-402", + "file": "OK", + "ident": "IDENT-OK", + "ending": "9. PBS off-site token on ep0 and its descriptor on the hub", + "q": "5" + }, + { + "ptr": "AR/05-hub-architecture.md:384", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. PBS off-site token on ep0 and its descriptor on the hub", + "q": "6" + }, + { + "ptr": "AR/05-hub-architecture.md:400-402", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. PBS off-site token on ep0 and its descriptor on the hub", + "q": "7" + }, + { + "ptr": "AR/05-hub-architecture.md:373-403", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. PBS off-site token on ep0 and its descriptor on the hub", + "q": "8" + }, + { + "ptr": "HUB/pbsdrheal/reconciler.go:63", + "file": "OK", + "ident": "NO-IDENT", + "ending": "9. PBS off-site token on ep0 and its descriptor on the hub", + "q": "8" + }, + { + "ptr": "AR/06-offsite-connectivity.md:172-182", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. WireGuard peers (the box's tunnel to the off-site endpoint)", + "q": "1" + }, + { + "ptr": "HUB/store/wg.go:345", + "file": "OK", + "ident": "IDENT-OK", + "ending": "10. WireGuard peers (the box's tunnel to the off-site endpoint)", + "q": "2" + }, + { + "ptr": "AR/06-offsite-connectivity.md:368", + "file": "OK", + "ident": "IDENT-MISS(RemoveWGPeer)", + "ending": "10. WireGuard peers (the box's tunnel to the off-site endpoint)", + "q": "2" + }, + { + "ptr": "AR/06-offsite-connectivity.md:205-207", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. WireGuard peers (the box's tunnel to the off-site endpoint)", + "q": "3" + }, + { + "ptr": "AR/06-offsite-connectivity.md:180-182", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. WireGuard peers (the box's tunnel to the off-site endpoint)", + "q": "4" + }, + { + "ptr": "HUB/store/wg.go:1-360", + "file": "OK", + "ident": "IDENT-MISS(expire|rotat|stale|handshake)", + "ending": "10. WireGuard peers (the box's tunnel to the off-site endpoint)", + "q": "5" + }, + { + "ptr": "AR/06-offsite-connectivity.md:200-205", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. WireGuard peers (the box's tunnel to the off-site endpoint)", + "q": "6" + }, + { + "ptr": "AR/06-offsite-connectivity.md:207-208", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. WireGuard peers (the box's tunnel to the off-site endpoint)", + "q": "7" + }, + { + "ptr": "AR/06-offsite-connectivity.md:200-208", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. WireGuard peers (the box's tunnel to the off-site endpoint)", + "q": "8" + }, + { + "ptr": "HUB/store/wg.go:345", + "file": "OK", + "ident": "NO-IDENT", + "ending": "10. WireGuard peers (the box's tunnel to the off-site endpoint)", + "q": "8" + }, + { + "ptr": "HUB/store/store.go:3764", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Signed jobs (agent update, config bundle, OS step)", + "q": "1" + }, + { + "ptr": "AG/cmd/felhom-opsign/main.go:57", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Signed jobs (agent update, config bundle, OS step)", + "q": "2" + }, + { + "ptr": "AG/cmd/felhom-opsign/main.go:130", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Signed jobs (agent update, config bundle, OS step)", + "q": "2" + }, + { + "ptr": "AG/internal/authz/verifier.go:26", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Signed jobs (agent update, config bundle, OS step)", + "q": "2" + }, + { + "ptr": "AG/cmd/felhom-opsign/main.go:126", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Signed jobs (agent update, config bundle, OS step)", + "q": "2" + }, + { + "ptr": "AG/internal/signedjobs/runner.go:163", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Signed jobs (agent update, config bundle, OS step)", + "q": "5" + }, + { + "ptr": "HUB/store/store.go:3764", + "file": "OK", + "ident": "IDENT-MISS(expire)", + "ending": "11. Signed jobs (agent update, config bundle, OS step)", + "q": "5" + }, + { + "ptr": "AG/internal/authz/verifier.go:161", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Signed jobs (agent update, config bundle, OS step)", + "q": "6" + }, + { + "ptr": "AG/internal/authz/verifier.go:21", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Signed jobs (agent update, config bundle, OS step)", + "q": "6" + }, + { + "ptr": "AG/internal/signedjobs/runner.go:186", + "file": "OK", + "ident": "NO-IDENT", + "ending": "11. Signed jobs (agent update, config bundle, OS step)", + "q": "7" + }, + { + "ptr": "AG/internal/authz/verify_test.go:82", + "file": "OK", + "ident": "IDENT-OK", + "ending": "11. Signed jobs (agent update, config bundle, OS step)", + "q": "8" + }, + { + "ptr": "AR/04-control-plane-authorization.md:78", + "file": "OK", + "ident": "IDENT-MISS(TestVerify_RejectsPerStage)", + "ending": "11. Signed jobs (agent update, config bundle, OS step)", + "q": "8" + }, + { + "ptr": "AR/04-control-plane-authorization.md:111", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Operational signing key and cold recovery key", + "q": "1" + }, + { + "ptr": "AR/04-control-plane-authorization.md:96", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Operational signing key and cold recovery key", + "q": "1" + }, + { + "ptr": "AR/04-control-plane-authorization.md:111", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Operational signing key and cold recovery key", + "q": "2" + }, + { + "ptr": "AR/04-control-plane-authorization.md:143", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Operational signing key and cold recovery key", + "q": "3" + }, + { + "ptr": "AR/04-control-plane-authorization.md:133", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Operational signing key and cold recovery key", + "q": "3" + }, + { + "ptr": "AR/04-control-plane-authorization.md:148", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Operational signing key and cold recovery key", + "q": "4" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:39", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Operational signing key and cold recovery key", + "q": "5" + }, + { + "ptr": "AR/04-control-plane-authorization.md:137-149", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Operational signing key and cold recovery key", + "q": "6" + }, + { + "ptr": "AR/04-control-plane-authorization.md:137-149", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Operational signing key and cold recovery key", + "q": "7" + }, + { + "ptr": "AR/04-control-plane-authorization.md:137-149", + "file": "OK", + "ident": "NO-IDENT", + "ending": "12. Operational signing key and cold recovery key", + "q": "8" + }, + { + "ptr": "HUB/store/opactions.go:62", + "file": "OK", + "ident": "IDENT-OK", + "ending": "13. Operator actions waiting for a box (24 hours; reboot 30 minutes)", + "q": "1" + }, + { + "ptr": "HUB/store/opactions.go:62", + "file": "OK", + "ident": "IDENT-OK", + "ending": "13. Operator actions waiting for a box (24 hours; reboot 30 minutes)", + "q": "2" + }, + { + "ptr": "HUB/store/opactions.go:47", + "file": "OK", + "ident": "IDENT-OK", + "ending": "13. Operator actions waiting for a box (24 hours; reboot 30 minutes)", + "q": "2" + }, + { + "ptr": "HUB/store/opactions.go:222", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. Operator actions waiting for a box (24 hours; reboot 30 minutes)", + "q": "4" + }, + { + "ptr": "HUB/store/opactions.go:296", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. Operator actions waiting for a box (24 hours; reboot 30 minutes)", + "q": "4" + }, + { + "ptr": "AR/04-control-plane-authorization.md:184", + "file": "OK", + "ident": "NO-IDENT", + "ending": "13. Operator actions waiting for a box (24 hours; reboot 30 minutes)", + "q": "8" + }, + { + "ptr": "HUB/store/opactions.go:62", + "file": "OK", + "ident": "IDENT-MISS(TestOperatorActions_ClosedList)", + "ending": "13. Operator actions waiting for a box (24 hours; reboot 30 minutes)", + "q": "8" + }, + { + "ptr": "HUB/api/dr.go:34", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. Recovery mode for a host (the door a rebuilt box uses)", + "q": "1" + }, + { + "ptr": "HUB/api/dr.go:21", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. Recovery mode for a host (the door a rebuilt box uses)", + "q": "2" + }, + { + "ptr": "HUB/api/dr.go:22", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. Recovery mode for a host (the door a rebuilt box uses)", + "q": "2" + }, + { + "ptr": "HUB/api/dr.go:52", + "file": "OK", + "ident": "IDENT-MISS(HUB/api/dr.go:21;)", + "ending": "14. Recovery mode for a host (the door a rebuilt box uses)", + "q": "2" + }, + { + "ptr": "HUB/api/dr_test.go:14", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. Recovery mode for a host (the door a rebuilt box uses)", + "q": "6" + }, + { + "ptr": "HUB/api/dr_test.go:70", + "file": "OK", + "ident": "IDENT-OK", + "ending": "14. Recovery mode for a host (the door a rebuilt box uses)", + "q": "8" + }, + { + "ptr": "HUB/api/dr.go:21-22", + "file": "OK", + "ident": "IDENT-MISS(TestRestoreDirective_GatedAndExpires)", + "ending": "14. Recovery mode for a host (the door a rebuilt box uses)", + "q": "8" + }, + { + "ptr": "HUB/store/host_recovery.go:28", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Break-glass credential (the host's emergency root password)", + "q": "1" + }, + { + "ptr": "HUB/web/hosts.go:983", + "file": "OK", + "ident": "IDENT-MISS(SaveHostRecoveryCredential)", + "ending": "15. Break-glass credential (the host's emergency root password)", + "q": "1" + }, + { + "ptr": "HUB/store/host_recovery.go:20", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. Break-glass credential (the host's emergency root password)", + "q": "2" + }, + { + "ptr": "RB/break-glass-sheet.md:75", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. Break-glass credential (the host's emergency root password)", + "q": "2" + }, + { + "ptr": "HUB/web/hosts.go:1010", + "file": "OK", + "ident": "IDENT-OK", + "ending": "15. Break-glass credential (the host's emergency root password)", + "q": "5" + }, + { + "ptr": "HUB/store/host_recovery.go:20-43", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. Break-glass credential (the host's emergency root password)", + "q": "7" + }, + { + "ptr": "HUB/store/host_recovery.go:20-28", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. Break-glass credential (the host's emergency root password)", + "q": "8" + }, + { + "ptr": "HUB/web/routes.go:130", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. Break-glass credential (the host's emergency root password)", + "q": "8" + }, + { + "ptr": "RB/break-glass-sheet.md:75", + "file": "OK", + "ident": "NO-IDENT", + "ending": "15. Break-glass credential (the host's emergency root password)", + "q": "8" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:467", + "file": "OK", + "ident": "IDENT-OK", + "ending": "16. Vouched artifact manifest and the \"golden not vouched\" alarm", + "q": "1" + }, + { + "ptr": "HUB/monitor/golden_vouch.go:42", + "file": "OK", + "ident": "IDENT-OK", + "ending": "16. Vouched artifact manifest and the \"golden not vouched\" alarm", + "q": "2" + }, + { + "ptr": "HUB/monitor/golden_vouch.go:43", + "file": "OK", + "ident": "IDENT-OK", + "ending": "16. Vouched artifact manifest and the \"golden not vouched\" alarm", + "q": "2" + }, + { + "ptr": "AR/09-update-architecture.md:1022", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Vouched artifact manifest and the \"golden not vouched\" alarm", + "q": "3" + }, + { + "ptr": "HUB/monitor/golden_vouch.go:92", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Vouched artifact manifest and the \"golden not vouched\" alarm", + "q": "4" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:472", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Vouched artifact manifest and the \"golden not vouched\" alarm", + "q": "4" + }, + { + "ptr": "HUB/monitor/golden_vouch.go:92", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Vouched artifact manifest and the \"golden not vouched\" alarm", + "q": "5" + }, + { + "ptr": "HUB/monitor/golden_vouch.go:65", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Vouched artifact manifest and the \"golden not vouched\" alarm", + "q": "5" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:531", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Vouched artifact manifest and the \"golden not vouched\" alarm", + "q": "7" + }, + { + "ptr": "E/scripts/iso/felhom-bootstrap.sh:430", + "file": "OK", + "ident": "NO-IDENT", + "ending": "16. Vouched artifact manifest and the \"golden not vouched\" alarm", + "q": "7" + }, + { + "ptr": "HUB/monitor/golden_vouch_test.go:15", + "file": "OK", + "ident": "IDENT-OK", + "ending": "16. Vouched artifact manifest and the \"golden not vouched\" alarm", + "q": "8" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:467-472", + "file": "OK", + "ident": "IDENT-MISS(TestGoldenVouchChecker)", + "ending": "16. Vouched artifact manifest and the \"golden not vouched\" alarm", + "q": "8" + }, + { + "ptr": "E/scripts/golden_currency_gate.py:169", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. The golden's age rule and its waiver file", + "q": "1" + }, + { + "ptr": "E/scripts/golden_currency_gate.py:169", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. The golden's age rule and its waiver file", + "q": "2" + }, + { + "ptr": "E/scripts/golden_currency_gate.py:167", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. The golden's age rule and its waiver file", + "q": "2" + }, + { + "ptr": "AR/09-update-architecture.md:997", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. The golden's age rule and its waiver file", + "q": "3" + }, + { + "ptr": "RB/RUNBOOK-manual-build.md:291", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. The golden's age rule and its waiver file", + "q": "3" + }, + { + "ptr": "E/scripts/golden_currency_gate.py:354", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. The golden's age rule and its waiver file", + "q": "4" + }, + { + "ptr": "E/scripts/golden_currency_gate.py:422", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. The golden's age rule and its waiver file", + "q": "5" + }, + { + "ptr": "E/scripts/golden_currency_gate.py:287", + "file": "OK", + "ident": "IDENT-OK", + "ending": "17. The golden's age rule and its waiver file", + "q": "7" + }, + { + "ptr": "E/scripts/golden_currency_gate.py:162", + "file": "OK", + "ident": "IDENT-MISS(E/scripts/golden_currency_gate.py:287.)", + "ending": "17. The golden's age rule and its waiver file", + "q": "7" + }, + { + "ptr": "E/scripts/golden_currency_gate.py:162-169", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. The golden's age rule and its waiver file", + "q": "8" + }, + { + "ptr": "AR/09-update-architecture.md:1362", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. The golden's age rule and its waiver file", + "q": "8" + }, + { + "ptr": "RB/RUNBOOK-manual-build.md:279", + "file": "OK", + "ident": "NO-IDENT", + "ending": "17. The golden's age rule and its waiver file", + "q": "8" + }, + { + "ptr": "AG/scripts/retention-policy.json:23-32", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Which agent versions stay downloadable (D50)", + "q": "1" + }, + { + "ptr": "AG/scripts/retention-policy.json:23", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Which agent versions stay downloadable (D50)", + "q": "2" + }, + { + "ptr": "AG/scripts/retention-policy.json:30-31", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Which agent versions stay downloadable (D50)", + "q": "2" + }, + { + "ptr": "AG/scripts/retention-policy.json:27", + "file": "OK", + "ident": "IDENT-OK", + "ending": "18. Which agent versions stay downloadable (D50)", + "q": "2" + }, + { + "ptr": "AG/scripts/retention-policy.json:12-15", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Which agent versions stay downloadable (D50)", + "q": "3" + }, + { + "ptr": "AR/09-update-architecture.md:1370-1373", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Which agent versions stay downloadable (D50)", + "q": "4" + }, + { + "ptr": "AG/scripts/retention-policy.json:20-21", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Which agent versions stay downloadable (D50)", + "q": "4" + }, + { + "ptr": "AG/scripts/check-published-versions.py:102", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Which agent versions stay downloadable (D50)", + "q": "5" + }, + { + "ptr": "AR/09-update-architecture.md:1372", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Which agent versions stay downloadable (D50)", + "q": "6" + }, + { + "ptr": "AG/scripts/retention-policy.json:14-15", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Which agent versions stay downloadable (D50)", + "q": "7" + }, + { + "ptr": "AR/09-update-architecture.md:1370-1376", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Which agent versions stay downloadable (D50)", + "q": "8" + }, + { + "ptr": "AG/scripts/retention-policy.json:23-32", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Which agent versions stay downloadable (D50)", + "q": "8" + }, + { + "ptr": "AG/scripts/retention-policy.json:12", + "file": "OK", + "ident": "NO-IDENT", + "ending": "18. Which agent versions stay downloadable (D50)", + "q": "8" + }, + { + "ptr": "E/scripts/iso/build-felhom-iso.sh:51", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. The installer ISO (Proxmox base, installer version, first boot)", + "q": "1" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:125", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. The installer ISO (Proxmox base, installer version, first boot)", + "q": "1" + }, + { + "ptr": "E/scripts/iso/build-felhom-iso.sh:51", + "file": "OK", + "ident": "IDENT-OK", + "ending": "19. The installer ISO (Proxmox base, installer version, first boot)", + "q": "2" + }, + { + "ptr": "E/scripts/iso/build-felhom-iso.sh:194", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. The installer ISO (Proxmox base, installer version, first boot)", + "q": "2" + }, + { + "ptr": "E/scripts/iso/README.md:40", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. The installer ISO (Proxmox base, installer version, first boot)", + "q": "2" + }, + { + "ptr": "RB/iso-release-gate.md:1-230", + "file": "OK", + "ident": "IDENT-MISS(days|stale|too)", + "ending": "19. The installer ISO (Proxmox base, installer version, first boot)", + "q": "3" + }, + { + "ptr": "E/scripts/iso/build-felhom-iso.sh:44-52", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. The installer ISO (Proxmox base, installer version, first boot)", + "q": "4" + }, + { + "ptr": "E/scripts/iso/build-felhom-iso.sh:79", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. The installer ISO (Proxmox base, installer version, first boot)", + "q": "5" + }, + { + "ptr": "E/scripts/felhom-host-install.sh:472", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. The installer ISO (Proxmox base, installer version, first boot)", + "q": "6" + }, + { + "ptr": "E/scripts/iso/felhom-bootstrap.sh:430", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. The installer ISO (Proxmox base, installer version, first boot)", + "q": "7" + }, + { + "ptr": "E/scripts/iso/felhom-bootstrap.sh:396", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. The installer ISO (Proxmox base, installer version, first boot)", + "q": "7" + }, + { + "ptr": "E/scripts/iso/build-felhom-iso.sh:222", + "file": "OK", + "ident": "NO-IDENT", + "ending": "19. The installer ISO (Proxmox base, installer version, first boot)", + "q": "8" + }, + { + "ptr": "E/scripts/iso/Dockerfile.assistant:10", + "file": "OK", + "ident": "IDENT-OK", + "ending": "19. The installer ISO (Proxmox base, installer version, first boot)", + "q": "8" + }, + { + "ptr": "E/manifests/hub.yaml:342", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. The certificate the hub's address serves", + "q": "1" + }, + { + "ptr": "E/manifests/hub.yaml:360", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. The certificate the hub's address serves", + "q": "1" + }, + { + "ptr": "E/manifests/hub.yaml:342", + "file": "OK", + "ident": "IDENT-OK", + "ending": "20. The certificate the hub's address serves", + "q": "2" + }, + { + "ptr": "E/manifests/hub.yaml:342", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. The certificate the hub's address serves", + "q": "3" + }, + { + "ptr": "HUB/monitor/golden_vouch.go:1-100", + "file": "DEAD", + "ident": "NO-IDENT", + "ending": "20. The certificate the hub's address serves", + "q": "5" + }, + { + "ptr": "E/manifests/hub.yaml:342", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. The certificate the hub's address serves", + "q": "7" + }, + { + "ptr": "E/manifests/hub.yaml:342", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. The certificate the hub's address serves", + "q": "8" + }, + { + "ptr": "E/manifests/hub.yaml:360", + "file": "OK", + "ident": "NO-IDENT", + "ending": "20. The certificate the hub's address serves", + "q": "8" + }, + { + "ptr": "HUB/web/server.go:930", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. Operator login session on the hub", + "q": "1" + }, + { + "ptr": "HUB/web/server.go:949", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Operator login session on the hub", + "q": "2" + }, + { + "ptr": "HUB/web/server.go:964", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. Operator login session on the hub", + "q": "2" + }, + { + "ptr": "HUB/web/server.go:199", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. Operator login session on the hub", + "q": "2" + }, + { + "ptr": "HUB/web/server.go:901", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Operator login session on the hub", + "q": "3" + }, + { + "ptr": "HUB/web/routes.go:130", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Operator login session on the hub", + "q": "4" + }, + { + "ptr": "HUB/web/server.go:909", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Operator login session on the hub", + "q": "7" + }, + { + "ptr": "HUB/web/r136_host_cookie_test.go:16", + "file": "OK", + "ident": "IDENT-OK", + "ending": "21. Operator login session on the hub", + "q": "8" + }, + { + "ptr": "HUB/web/server.go:949", + "file": "OK", + "ident": "NO-IDENT", + "ending": "21. Operator login session on the hub", + "q": "8" + }, + { + "ptr": "AR/09-update-architecture.md:1022", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Operator access tokens, among them CC's hub token", + "q": "1" + }, + { + "ptr": "HUB/web/tokens.go:29", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Operator access tokens, among them CC's hub token", + "q": "2" + }, + { + "ptr": "HUB/web/tokens.go:30", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Operator access tokens, among them CC's hub token", + "q": "2" + }, + { + "ptr": "HUB/web/tokens.go:254", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Operator access tokens, among them CC's hub token", + "q": "2" + }, + { + "ptr": "HUB/web/tokens.go:180", + "file": "OK", + "ident": "NO-IDENT", + "ending": "22. Operator access tokens, among them CC's hub token", + "q": "2" + }, + { + "ptr": "HUB/web/tokens.go:95", + "file": "OK", + "ident": "NO-IDENT", + "ending": "22. Operator access tokens, among them CC's hub token", + "q": "4" + }, + { + "ptr": "HUB/web/tokens.go:245", + "file": "OK", + "ident": "NO-IDENT", + "ending": "22. Operator access tokens, among them CC's hub token", + "q": "5" + }, + { + "ptr": "HUB/store/api_tokens.go:52", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Operator access tokens, among them CC's hub token", + "q": "7" + }, + { + "ptr": "HUB/web/tokens_test.go:310", + "file": "OK", + "ident": "IDENT-OK", + "ending": "22. Operator access tokens, among them CC's hub token", + "q": "8" + }, + { + "ptr": "AR/05-hub-architecture.md:655-669", + "file": "OK", + "ident": "IDENT-MISS(TestToken_NoTokenUnknownExpiredRevoked401)", + "ending": "22. Operator access tokens, among them CC's hub token", + "q": "8" + }, + { + "ptr": "HUB/cloudflare/reach.go:14", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. A household's own Cloudflare token (hub side)", + "q": "1" + }, + { + "ptr": "HUB/cloudflare/reach.go:14", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. A household's own Cloudflare token (hub side)", + "q": "2" + }, + { + "ptr": "HUB/cloudflare/reach.go:14", + "file": "OK", + "ident": "NO-IDENT", + "ending": "23. A household's own Cloudflare token (hub side)", + "q": "8" + }, + { + "ptr": "E/manifests/hub.yaml:148-152", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The mail service key (all household and operator mail)", + "q": "1" + }, + { + "ptr": "RB/secrets.md:33", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The mail service key (all household and operator mail)", + "q": "2" + }, + { + "ptr": "AR/_recovery-inventory-2026-07-28.md:502", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The mail service key (all household and operator mail)", + "q": "2" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:47", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The mail service key (all household and operator mail)", + "q": "3" + }, + { + "ptr": "RB/secrets.md", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The mail service key (all household and operator mail)", + "q": "5" + }, + { + "ptr": "HUB/web/selfbind_mint.go:82", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The mail service key (all household and operator mail)", + "q": "7" + }, + { + "ptr": "RB/secrets.md:33", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The mail service key (all household and operator mail)", + "q": "8" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:47", + "file": "OK", + "ident": "NO-IDENT", + "ending": "24. The mail service key (all household and operator mail)", + "q": "8" + }, + { + "ptr": "E/manifests/hub.yaml:157-161", + "file": "OK", + "ident": "NO-IDENT", + "ending": "25. The hub's global bearer key", + "q": "1" + }, + { + "ptr": "AR/_recovery-inventory-2026-07-28.md:501", + "file": "OK", + "ident": "NO-IDENT", + "ending": "25. The hub's global bearer key", + "q": "1" + }, + { + "ptr": "RB/secrets.md:124", + "file": "OK", + "ident": "NO-IDENT", + "ending": "25. The hub's global bearer key", + "q": "2" + }, + { + "ptr": "RB/secrets.md:124", + "file": "OK", + "ident": "NO-IDENT", + "ending": "25. The hub's global bearer key", + "q": "3" + }, + { + "ptr": "RB/secrets.md:124", + "file": "OK", + "ident": "NO-IDENT", + "ending": "25. The hub's global bearer key", + "q": "4" + }, + { + "ptr": "AR/13-threat-model.md:70", + "file": "OK", + "ident": "NO-IDENT", + "ending": "25. The hub's global bearer key", + "q": "6" + }, + { + "ptr": "AR/13-threat-model.md:70", + "file": "OK", + "ident": "NO-IDENT", + "ending": "25. The hub's global bearer key", + "q": "8" + }, + { + "ptr": "AR/_recovery-inventory-2026-07-28.md:1030", + "file": "OK", + "ident": "NO-IDENT", + "ending": "25. The hub's global bearer key", + "q": "8" + }, + { + "ptr": "E/scripts/publish-site-cloudflare.sh:28", + "file": "OK", + "ident": "NO-IDENT", + "ending": "26. Pinned tool versions in builds and CI", + "q": "1" + }, + { + "ptr": "E/scripts/iso/Dockerfile.assistant:10", + "file": "OK", + "ident": "NO-IDENT", + "ending": "26. Pinned tool versions in builds and CI", + "q": "2" + }, + { + "ptr": "E/manifests/contact-mailer.yaml:54", + "file": "OK", + "ident": "IDENT-OK", + "ending": "26. Pinned tool versions in builds and CI", + "q": "2" + }, + { + "ptr": "E/.gitea/workflows/security-watch.yml:22", + "file": "OK", + "ident": "NO-IDENT", + "ending": "26. Pinned tool versions in builds and CI", + "q": "3" + }, + { + "ptr": "E/.gitea/workflows/gates.yml:27", + "file": "OK", + "ident": "NO-IDENT", + "ending": "26. Pinned tool versions in builds and CI", + "q": "3" + }, + { + "ptr": "E/scripts/security_watch.py:292", + "file": "OK", + "ident": "NO-IDENT", + "ending": "26. Pinned tool versions in builds and CI", + "q": "5" + }, + { + "ptr": "E/.gitea/workflows/gates.yml:27", + "file": "OK", + "ident": "NO-IDENT", + "ending": "26. Pinned tool versions in builds and CI", + "q": "7" + }, + { + "ptr": "E/scripts/security_watch.py:292", + "file": "OK", + "ident": "NO-IDENT", + "ending": "26. Pinned tool versions in builds and CI", + "q": "8" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:46", + "file": "OK", + "ident": "NO-IDENT", + "ending": "27. Domains, registrars and the home dynamic-DNS name", + "q": "1" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:73", + "file": "OK", + "ident": "NO-IDENT", + "ending": "27. Domains, registrars and the home dynamic-DNS name", + "q": "1" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:45", + "file": "OK", + "ident": "NO-IDENT", + "ending": "27. Domains, registrars and the home dynamic-DNS name", + "q": "5" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:46", + "file": "OK", + "ident": "NO-IDENT", + "ending": "27. Domains, registrars and the home dynamic-DNS name", + "q": "8" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:44", + "file": "OK", + "ident": "NO-IDENT", + "ending": "28. Cloudflare account, the site publish token and the ISO store keys", + "q": "1" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:125", + "file": "OK", + "ident": "NO-IDENT", + "ending": "28. Cloudflare account, the site publish token and the ISO store keys", + "q": "1" + }, + { + "ptr": "E/scripts/publish-site-cloudflare.sh:12", + "file": "OK", + "ident": "IDENT-OK", + "ending": "28. Cloudflare account, the site publish token and the ISO store keys", + "q": "2" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:44", + "file": "OK", + "ident": "NO-IDENT", + "ending": "28. Cloudflare account, the site publish token and the ISO store keys", + "q": "4" + }, + { + "ptr": "E/scripts/iso/felhom-bootstrap.sh:396", + "file": "OK", + "ident": "NO-IDENT", + "ending": "28. Cloudflare account, the site publish token and the ISO store keys", + "q": "6" + }, + { + "ptr": "E/scripts/publish-site-cloudflare.sh:12", + "file": "OK", + "ident": "NO-IDENT", + "ending": "28. Cloudflare account, the site publish token and the ISO store keys", + "q": "7" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:44", + "file": "OK", + "ident": "NO-IDENT", + "ending": "28. Cloudflare account, the site publish token and the ISO store keys", + "q": "8" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:41", + "file": "OK", + "ident": "NO-IDENT", + "ending": "29. Hetzner account, the rented endpoint and the storage boxes", + "q": "1" + }, + { + "ptr": "E/manifests/hub.yaml:222-226", + "file": "OK", + "ident": "NO-IDENT", + "ending": "29. Hetzner account, the rented endpoint and the storage boxes", + "q": "1" + }, + { + "ptr": "HUB/monitor/deadline_tiers.go:28", + "file": "OK", + "ident": "IDENT-OK", + "ending": "29. Hetzner account, the rented endpoint and the storage boxes", + "q": "5" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:41", + "file": "OK", + "ident": "NO-IDENT", + "ending": "29. Hetzner account, the rented endpoint and the storage boxes", + "q": "8" + }, + { + "ptr": "E/documentation/operations/tailscale.md:11-13", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Tailscale node keys (operator's remote way in)", + "q": "1" + }, + { + "ptr": "E/documentation/operations/tailscale.md:41", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Tailscale node keys (operator's remote way in)", + "q": "2" + }, + { + "ptr": "E/documentation/operations/tailscale.md:11-12", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Tailscale node keys (operator's remote way in)", + "q": "2" + }, + { + "ptr": "E/documentation/operations/tailscale.md:43", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Tailscale node keys (operator's remote way in)", + "q": "3" + }, + { + "ptr": "E/documentation/operations/tailscale.md:91", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Tailscale node keys (operator's remote way in)", + "q": "4" + }, + { + "ptr": "E/documentation/operations/tailscale.md:92", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Tailscale node keys (operator's remote way in)", + "q": "5" + }, + { + "ptr": "E/documentation/operations/tailscale.md:41", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Tailscale node keys (operator's remote way in)", + "q": "8" + }, + { + "ptr": "E/documentation/operations/tailscale.md:89-92", + "file": "OK", + "ident": "NO-IDENT", + "ending": "30. Tailscale node keys (operator's remote way in)", + "q": "8" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:48", + "file": "OK", + "ident": "NO-IDENT", + "ending": "31. Gitea and registry credentials", + "q": "1" + }, + { + "ptr": "RB/secrets.md:231", + "file": "OK", + "ident": "NO-IDENT", + "ending": "31. Gitea and registry credentials", + "q": "1" + }, + { + "ptr": "RB/secrets.md", + "file": "OK", + "ident": "NO-IDENT", + "ending": "31. Gitea and registry credentials", + "q": "2" + }, + { + "ptr": "RB/secrets.md:231", + "file": "OK", + "ident": "NO-IDENT", + "ending": "31. Gitea and registry credentials", + "q": "8" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:48", + "file": "OK", + "ident": "NO-IDENT", + "ending": "31. Gitea and registry credentials", + "q": "8" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:52", + "file": "OK", + "ident": "IDENT-OK", + "ending": "32. CC's secret store: its key and the tokens inside (GitHub advisory token, hub token)", + "q": "1" + }, + { + "ptr": "E/.gitea/workflows/security-watch.yml:22", + "file": "OK", + "ident": "NO-IDENT", + "ending": "32. CC's secret store: its key and the tokens inside (GitHub advisory token, hub token)", + "q": "4" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:52", + "file": "OK", + "ident": "NO-IDENT", + "ending": "32. CC's secret store: its key and the tokens inside (GitHub advisory token, hub token)", + "q": "8" + }, + { + "ptr": "E/manifests/hub.yaml:191", + "file": "OK", + "ident": "NO-IDENT", + "ending": "33. SSH keys between the hub, DooPlex and the endpoint", + "q": "1" + }, + { + "ptr": "E/manifests/hub.yaml:202", + "file": "OK", + "ident": "NO-IDENT", + "ending": "33. SSH keys between the hub, DooPlex and the endpoint", + "q": "1" + }, + { + "ptr": "E/manifests/hub.yaml:209", + "file": "OK", + "ident": "NO-IDENT", + "ending": "33. SSH keys between the hub, DooPlex and the endpoint", + "q": "1" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:42", + "file": "OK", + "ident": "NO-IDENT", + "ending": "33. SSH keys between the hub, DooPlex and the endpoint", + "q": "1" + }, + { + "ptr": "RB/offsite-endpoint.md:313", + "file": "OK", + "ident": "NO-IDENT", + "ending": "33. SSH keys between the hub, DooPlex and the endpoint", + "q": "2" + }, + { + "ptr": "RB/offsite-endpoint.md:313", + "file": "OK", + "ident": "NO-IDENT", + "ending": "33. SSH keys between the hub, DooPlex and the endpoint", + "q": "8" + }, + { + "ptr": "RB/total-loss-of-dooplex.md:42", + "file": "OK", + "ident": "NO-IDENT", + "ending": "33. SSH keys between the hub, DooPlex and the endpoint", + "q": "8" + }, + { + "ptr": "AR/11-os-updates.md:103", + "file": "OK", + "ident": "NO-IDENT", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "1" + }, + { + "ptr": "AR/11-os-updates.md:91", + "file": "OK", + "ident": "NO-IDENT", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "2" + }, + { + "ptr": "AR/11-os-updates.md:725", + "file": "OK", + "ident": "NO-IDENT", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "2" + }, + { + "ptr": "AR/11-os-updates.md:725", + "file": "OK", + "ident": "NO-IDENT", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "3" + }, + { + "ptr": "E/scripts/security_watch.py:292", + "file": "OK", + "ident": "NO-IDENT", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "5" + }, + { + "ptr": "AR/11-os-updates.md:103", + "file": "OK", + "ident": "NO-IDENT", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "AR/11-os-updates.md:662", + "file": "OK", + "ident": "NO-IDENT", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "HUB/store/logbundle.go:29", + "file": "OK", + "ident": "IDENT-OK", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "HUB/accesslog/accesslog.go:44", + "file": "OK", + "ident": "IDENT-OK", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "HUB/store/deleted_customer_audit.go:23", + "file": "OK", + "ident": "IDENT-OK", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "E/hub/cmd/hub/main.go:1134-1139", + "file": "OK", + "ident": "NO-IDENT", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "E/manifests/hub.yaml:83", + "file": "OK", + "ident": "IDENT-OK", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "HUB/osupdates/kernel.go:48", + "file": "OK", + "ident": "IDENT-OK", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "HUB/monitor/deadline.go:29", + "file": "OK", + "ident": "IDENT-OK", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "HUB/monitor/deadline_tiers.go:28", + "file": "OK", + "ident": "IDENT-OK", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "HUB/monitor/restoretest.go:68-74", + "file": "OK", + "ident": "NO-IDENT", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "HUB/notify/dispatcher.go:1018", + "file": "OK", + "ident": "IDENT-OK", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "HUB/web/system.go:174", + "file": "OK", + "ident": "IDENT-OK", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "HUB/dbsnap/dbsnap.go:36", + "file": "OK", + "ident": "IDENT-OK", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "HUB/monitor/offsite_escrow_pending.go:37", + "file": "OK", + "ident": "IDENT-OK", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "RB/RUNBOOK-escrow-ceremony.md:46", + "file": "OK", + "ident": "NO-IDENT", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "HUB/web/appliances.go:21", + "file": "OK", + "ident": "IDENT-OK", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "RB/RUNBOOK-manual-build.md:278-279", + "file": "OK", + "ident": "NO-IDENT", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "E/manifests/hub.yaml:135", + "file": "OK", + "ident": "NO-IDENT", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + }, + { + "ptr": "E/manifests/webpage.yaml:141", + "file": "OK", + "ident": "NO-IDENT", + "ending": "34. Operating-system support ends (Debian 13, Proxmox VE 9, PBS)", + "q": "8" + } + ] +} \ No newline at end of file diff --git a/documentation/audits/end-dates-list-2026-11-11/evidence/searches.txt b/documentation/audits/end-dates-list-2026-11-11/evidence/searches.txt new file mode 100644 index 00000000..613d2d3b --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/evidence/searches.txt @@ -0,0 +1,27 @@ +searches run 2026-11-11 10:25:25 UTC +C+ | positive control (a row that exists) | /R-1301/ | ar files=0 | open: R-1301 | closed: | roadmap=0 +C+2 | positive control (a word in all four) | /certificate/ | ar files=3 | open: R-1202 R-904 | closed: R-1090 R-1300 R-510 | roadmap=0 +C- | negative control | /zzqxenddatezz/ | ar files=0 | open: | closed: | roadmap=0 +S01 | web certificate: warning before it ends | /(certificate|cert).{0,80}(expir|renew|runs out|days left)|acme\.json|NotAfter/ | ar files=0 | open: | closed: | roadmap=0 +S02 | Proxmox renews its certificate, the agent's pin does not follow | /pve-ssl|pveproxy.{0,60}cert|(tls|certificate|cert) pin.{0,120}(renew|expir|rotat|re-pin|repin)|fingerprint.{0,80}(renew|expir|stale)/ | ar files=0 | open: | closed: | roadmap=0 +S03 | operator access token ends, nobody warned | /(access|api|hub|operator).token.{0,120}(expir|90 days|days left|renew)|HUB_TOKEN.{0,80}(expir|ends|renew)|token_expir/ | ar files=0 | open: | closed: | roadmap=0 +S04 | Tailscale key expiry | /tailscale.{0,120}(expir|key)|key expiry/ | ar files=1 | open: | closed: | roadmap=0 +S05 | restore copy swept after 30 days, no warning | /restore cop.{0,100}(30 days|sweep|unused)|restoreCopyUnusedAge/ | ar files=1 | open: | closed: | roadmap=0 +S06 | domain renewal / registrar | /registrar|domain.{0,60}(renew|expir|lapse)|whois/ | ar files=3 | open: | closed: R-1158 R-820 R-826 | roadmap=0 +S07 | paid accounts: billing, contract end | /(hetzner|cloudflare|resend|mail service).{0,100}(billing|invoice|unpaid|contract|account (is )?closed|payment)|credit card/ | ar files=1 | open: | closed: | roadmap=0 +S08 | mail service key end / check that mail still goes | /(resend|mail).{0,40}(key|token).{0,80}(expir|rotat|ends)|canary mail|test mail.{0,40}(daily|weekly)/ | ar files=0 | open: | closed: | roadmap=0 +S09 | OS support end (Debian 13, Proxmox 9) | /EOL|end of life|end-of-life|support (ends|end date)|trixie.{0,80}(until|ends|support)/ | ar files=5 | open: | closed: R-1203 R-921 | roadmap=0 +S10 | apt signing keys end | /(signing|apt|archive|repo) key.{0,100}(expir|ends|rotat)|keyring.{0,80}(expir|old)|Valid-Until/ | ar files=1 | open: | closed: | roadmap=0 +S11 | a box off for a long time comes back | /(off|unplugged|powered off|switched off|in a drawer).{0,60}(months|a year|years)|long.{0,10}(off|absence)|cold box/ | ar files=3 | open: | closed: | roadmap=0 +S12 | old tunnel client / infra image stops being accepted | /cloudflared.{0,100}(old|version|support|deprecat)|tunnel client.{0,60}(old|support)/ | ar files=0 | open: | closed: | roadmap=0 +S13 | Gitea / registry token end | /(gitea|registry).{0,60}(token|credential|password).{0,80}(expir|ends|rotat)/ | ar files=1 | open: R-925 | closed: R-616 | roadmap=0 +S14 | GitHub advisory token / CC store token end | /GH_ADVISORY_TOKEN|advisory token|github.{0,40}token.{0,60}(expir|ends)/ | ar files=0 | open: | closed: | roadmap=0 +S15 | household's own Cloudflare token end | /(cloudflare|cf).{0,20}token.{0,100}(expir|ends|ttl|revok)/ | ar files=0 | open: | closed: | roadmap=0 +S16 | operator signing key rotation | /key.rotation|re-pin|rotat.{0,40}(operational|signing|cold) key/ | ar files=5 | open: R-924 | closed: | roadmap=0 +S17 | seal key rotation | /seal.{0,10}key.{0,100}(rotat|lost|change)/ | ar files=0 | open: | closed: | roadmap=0 +S18 | a list of what ends when | /end dates|expiry (list|calendar|table)|renewal calendar|what ends/ | ar files=0 | open: | closed: | roadmap=0 +S19 | golden age: warning before day 7 | /golden.{0,80}(7 days|too old|age).{0,80}(warn|remind)|waiver.{0,60}(expir|ends)/ | ar files=0 | open: R-468 | closed: | roadmap=0 +S20 | served certificate of the hub address (cert-manager) | /cert-manager|letsencrypt-prod|ClusterIssuer/ | ar files=1 | open: | closed: | roadmap=0 +S21 | a recorded image digest is no longer served | /digest.{0,100}(still served|no longer|gone|deleted|removed from)|registry.{0,60}(drops|deletes|garbage)/ | ar files=1 | open: | closed: | roadmap=0 +S22 | the installer ISO grows old | /ISO.{0,80}(too old|stale|age|rebuild every|rebuilt every)/ | ar files=5 | open: R-1214 R-504 R-548 | closed: R-1176 R-472 R-531 R-556 R-588 R-725 | roadmap=0 +S23 | a box off during the 14-day off-site countdown | /(abandon|countdown).{0,120}(box is off|switched off|offline|unreachable)/ | ar files=0 | open: | closed: | roadmap=0 diff --git a/documentation/audits/end-dates-list-2026-11-11/tools/check-pointers.py b/documentation/audits/end-dates-list-2026-11-11/tools/check-pointers.py new file mode 100644 index 00000000..480f3fb7 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/tools/check-pointers.py @@ -0,0 +1,78 @@ +#!/usr/bin/env python3 +"""Print the text behind every pointer in a table file, so a person can read it. + +Read-only by construction: it opens files for reading and prints. It runs on DooPlex against the +git clones only; it never touches a box. + +Usage: check-pointers.py [FILE ...] (default: ../TABLE.md) + check-pointers.py --quiet [FILE] (only the verdict lines; rc 1 when a pointer is dead) + +A pointer is a backticked token `PREFIX/path[:line[-line]]` with one of the prefixes below. +A pointer is DEAD when the file is missing or the line range lies beyond the end of the file. +The script cannot judge whether the line says what the table claims — that is the reader's job. +""" +import os +import re +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +AUDIT = os.path.dirname(HERE) +REPO = os.path.abspath(os.path.join(AUDIT, "..", "..", "..")) # felhom.eu +WORKSPACE = os.path.dirname(REPO) + +PREFIX = { + "AG": os.path.join(WORKSPACE, "felhom-agent"), + "CT": os.path.join(WORKSPACE, "felhom-controller", "controller"), + "CAT": os.path.join(WORKSPACE, "app-catalog-felhom.eu"), + "HUB": os.path.join(REPO, "hub", "internal"), + "E": REPO, + "AR": os.path.join(REPO, "documentation", "architecture"), + "RB": os.path.join(REPO, "documentation", "runbooks"), + "EV": os.path.join(AUDIT, "evidence"), +} +TOKEN = re.compile(r"`((?:%s)/[^`:\s]+)(?::(\d+)(?:-(\d+))?)?`" % "|".join(PREFIX)) + + +def main() -> int: + args = [a for a in sys.argv[1:] if a != "--quiet"] + quiet = "--quiet" in sys.argv[1:] + files = args or [os.path.join(AUDIT, "TABLE.md")] + seen, dead, total = set(), 0, 0 + for table in files: + with open(table, encoding="utf-8") as fh: + text = fh.read() + for m in TOKEN.finditer(text): + key = m.group(0) + if key in seen: + continue + seen.add(key) + total += 1 + pre, rest = m.group(1).split("/", 1) + path = os.path.join(PREFIX[pre], rest) + first = int(m.group(2)) if m.group(2) else None + last = int(m.group(3)) if m.group(3) else first + if not os.path.exists(path): + dead += 1 + print(f"DEAD {key} (no such file: {path})") + continue + if first is None: + if not quiet: + print(f"OK {key} (file exists)") + continue + with open(path, encoding="utf-8", errors="replace") as fh: + lines = fh.readlines() + if last > len(lines): + dead += 1 + print(f"DEAD {key} (file has {len(lines)} lines)") + continue + if quiet: + continue + print(f"OK {key}") + for n in range(first, min(last, first + 11) + 1): + print(f" {n:>5} {lines[n - 1].rstrip()[:200]}") + print(f"pointers: {total} dead: {dead}") + return 1 if dead else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/documentation/audits/end-dates-list-2026-11-11/tools/guest-dates.sh b/documentation/audits/end-dates-list-2026-11-11/tools/guest-dates.sh new file mode 100644 index 00000000..7d18cca1 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/tools/guest-dates.sh @@ -0,0 +1,30 @@ +#!/bin/bash +# End-dates list 2026-11-11 — guest 9201. READ-ONLY. Prints for a certificate only subject, issuer, dates, fingerprint. +# It never prints a key or a token value. acme.json holds private keys: only its certificate fields are decoded. +set -u +h() { echo; echo "=== $* ==="; } +U() { sed -E 's#(https?://)[^/@ ]+@#\1***@#g'; } +cert() { openssl x509 -noout -subject -issuer -startdate -enddate -fingerprint -sha256 2>/dev/null | tr '\n' ' '; echo; } +h "now"; date -u +%FT%TZ +h "layout"; ls /opt/docker 2>/dev/null | tr '\n' ' '; echo; ls /opt/docker/stacks 2>/dev/null | tr '\n' ' '; echo; docker volume ls --format '{{.Name}}' | /usr/bin/grep -i -E 'felhom|traefik|cloudflare|controller' | tr '\n' ' '; echo +h "certificate files (system paths and infra stacks only)"; for f in $( { /usr/bin/grep -rlIs 'BEGIN CERTIFICATE' /etc/felhom* /etc/docker /usr/local/share/ca-certificates /etc/ssl/private 2>/dev/null; find /opt/docker -maxdepth 4 \( -name '*.pem' -o -name '*.crt' -o -name '*.cert' \) -size -200k 2>/dev/null; for v in $(docker volume ls --format '{{.Name}}' | /usr/bin/grep -i -E 'felhom|traefik|controller'); do find /var/lib/docker/volumes/$v/_data -maxdepth 3 \( -name '*.pem' -o -name '*.crt' -o -name '*.cert' \) -size -200k 2>/dev/null; done; } | sort -u); do /usr/bin/grep -qs 'BEGIN CERTIFICATE' "$f" || continue; printf '%s [mtime=%s] ' "$f" "$(date -u -r "$f" +%F)"; cert < "$f"; done +h "acme stores (certificate fields only)"; for f in $(find /opt/docker /var/lib/docker/volumes -maxdepth 5 -name 'acme*.json' 2>/dev/null); do echo "--- $f mtime=$(date -u -r "$f" +%FT%TZ)"; python3 - "$f" <<'P' +import json,sys,base64,subprocess +try: d=json.load(open(sys.argv[1])) +except Exception as e: print("unreadable:", type(e).__name__); sys.exit(0) +for res,v in (d.items() if isinstance(d,dict) else []): + for c in (v or {}).get("Certificates") or []: + pem=base64.b64decode(c.get("certificate","")) + o=subprocess.run(["openssl","x509","-noout","-subject","-issuer","-startdate","-enddate","-fingerprint","-sha256"],input=pem,capture_output=True) + print(res, (c.get("domain") or {}).get("main"), "|", o.stdout.decode().replace("\n"," ")) +P +done +h "served certificates (TLS handshake to own listeners)"; ss -H -tln | awk '{print $4}' | sort -u | while read a; do p=${a##*:}; ip=${a%:*}; ip=${ip#[}; ip=${ip%]}; case "$ip" in '*'|'0.0.0.0'|'::') ip=127.0.0.1;; esac; out=$(echo | timeout 4 openssl s_client -connect "$ip:$p" 2>/dev/null | cert); [ -n "${out// /}" ] && echo "$a -> $out" || echo "$a -> no TLS"; done +h "os"; . /etc/os-release; echo "$PRETTY_NAME / $VERSION_CODENAME"; cat /etc/debian_version +h "docker"; docker version --format 'client={{.Client.Version}} server={{.Server.Version}}' 2>&1; dpkg -l docker-ce containerd.io docker-compose-plugin 2>/dev/null | awk '/^ii/{print $2,$3}'; cat /etc/felhom-controller-image 2>/dev/null +h "restic and tools inside the controller"; docker exec felhom-controller restic version 2>&1 | head -1; docker exec felhom-controller sh -c 'cat /etc/os-release | head -3; command -v rclone age docker 2>/dev/null' 2>&1 | head -8 +h "images: repo tag digest created"; docker images --digests --format '{{.Repository}}:{{.Tag}} {{.Digest}} {{.CreatedAt}}' | cut -c1-200 | sort +h "apt sources"; for f in /etc/apt/sources.list /etc/apt/sources.list.d/*; do [ -f "$f" ] && { echo "--- $f"; /usr/bin/grep -v -E '^\s*(#|$)' "$f" | U; }; done +h "apt keyring files"; ls -l --time-style=long-iso /etc/apt/trusted.gpg /etc/apt/trusted.gpg.d/ /etc/apt/keyrings/ /usr/share/keyrings/ 2>/dev/null | awk '{print $5,$6,$8}' +h "apt release files: Valid-Until"; for f in /var/lib/apt/lists/*Release; do [ -f "$f" ] && printf '%s | Date: %s | Valid-Until: %s\n' "$(basename "$f")" "$(/usr/bin/grep -m1 '^Date:' "$f" | cut -c7-)" "$(/usr/bin/grep -m1 '^Valid-Until:' "$f" | cut -c14-)"; done +h "controller data folder (top level: names and dates only)"; for v in $(docker volume ls --format '{{.Name}}' | /usr/bin/grep -i -E 'controller'); do echo "--- $v"; ls -l --time-style=long-iso /var/lib/docker/volumes/$v/_data 2>/dev/null | awk '{print $1,$5,$6,$7,$8}' | head -60; done diff --git a/documentation/audits/end-dates-list-2026-11-11/tools/host-dates.sh b/documentation/audits/end-dates-list-2026-11-11/tools/host-dates.sh new file mode 100644 index 00000000..9eed6bc4 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/tools/host-dates.sh @@ -0,0 +1,31 @@ +#!/bin/bash +# End-dates list 2026-11-11 — host. READ-ONLY: openssl x509 -noout (public fields only), a TLS handshake to the box's +# own listening ports, pveum/pvenode/pvesubscription reads, ls, cat of apt source lists, dpkg -l. It prints NO key, +# NO token value and NO certificate body: for a certificate only subject, issuer, the two dates and the fingerprint. +set -u +h() { echo; echo "=== $* ==="; } +U() { sed -E 's#(https?://)[^/@ ]+@#\1***@#g'; } +cert() { openssl x509 -noout -subject -issuer -startdate -enddate -fingerprint -sha256 2>/dev/null | tr '\n' ' '; echo; } +h "now"; date -u +%FT%TZ +h "certificate files"; for f in $(/usr/bin/grep -rlIs 'BEGIN CERTIFICATE' /etc/pve/local /etc/pve/pve-root-ca.pem /etc/pve/nodes /etc/felhom /etc/felhom-agent /var/lib/felhom-agent /var/lib/felhom-install /etc/ssl/private /usr/local/share/ca-certificates /etc/proxmox-backup 2>/dev/null | sort -u); do n=$(/usr/bin/grep -c 'BEGIN CERTIFICATE' "$f"); printf '%s [certs=%s mtime=%s] ' "$f" "$n" "$(date -u -r "$f" +%F)"; cert < "$f"; done +h "system CA store"; ls /etc/ssl/certs/*.pem 2>/dev/null | wc -l; dpkg -l ca-certificates 2>/dev/null | tail -1 | awk '{print $2,$3}' +h "served certificates (TLS handshake to own listeners)"; ss -H -tln | awk '{print $4}' | sort -u | while read a; do p=${a##*:}; ip=${a%:*}; ip=${ip#[}; ip=${ip%]}; case "$ip" in '*'|'0.0.0.0'|'::') ip=127.0.0.1;; esac; out=$(echo | timeout 4 openssl s_client -connect "$ip:$p" 2>/dev/null | cert); [ -n "${out// /}" ] && echo "$a -> $out" || echo "$a -> no TLS"; done +h "listeners (port, process)"; ss -H -tlnp | awk '{print $4, $6}' | sed -E 's/pid=[0-9]+,fd=[0-9]+//g' | sort -u +h "pve users and tokens (expire: 0 = never)"; pveum user list --output-format json 2>/dev/null | python3 -c ' +import json,sys,time +for u in json.load(sys.stdin): + print("user", u.get("userid"), "enable", u.get("enable"), "expire", u.get("expire"), "tokens", [(t.get("tokenid"), "expire", t.get("expire")) for t in (u.get("tokens") or [])])' +h "acme / subscription"; pvenode acme account list 2>&1 | head -5; pvenode config get 2>&1 | head -10; pvesubscription get 2>&1 | /usr/bin/grep -E 'status|message|nextduedate' | head +h "os"; . /etc/os-release; echo "$PRETTY_NAME / $VERSION_CODENAME"; cat /etc/debian_version; pveversion 2>&1; uname -r; proxmox-boot-tool kernel list 2>&1 | head -12; dpkg -l 'proxmox-kernel-*-signed' 'pve-kernel-*' 2>/dev/null | awk '/^ii/{print $2,$3}' | head -12 +h "apt sources"; for f in /etc/apt/sources.list /etc/apt/sources.list.d/*; do [ -f "$f" ] && { echo "--- $f"; /usr/bin/grep -v -E '^\s*(#|$)' "$f" | U; }; done; echo "--- auth files (names only): $(ls /etc/apt/auth.conf /etc/apt/auth.conf.d 2>/dev/null | tr '\n' ' ')" +h "apt keyring files"; ls -l --time-style=long-iso /etc/apt/trusted.gpg /etc/apt/trusted.gpg.d/ /etc/apt/keyrings/ /usr/share/keyrings/ 2>/dev/null | awk '{print $5,$6,$8}' +h "apt release files: Valid-Until"; for f in /var/lib/apt/lists/*Release; do [ -f "$f" ] && printf '%s | Date: %s | Valid-Until: %s\n' "$(basename "$f")" "$(/usr/bin/grep -m1 '^Date:' "$f" | cut -c7-)" "$(/usr/bin/grep -m1 '^Valid-Until:' "$f" | cut -c14-)"; done +h "templates and golden"; for d in $(awk '/^dir:/{f=1;next} /^[a-z]+:/{f=0} f&&$1=="path"{print $2}' /etc/pve/storage.cfg 2>/dev/null) /var/lib/vz; do ls -l --time-style=long-iso "$d/template/cache" 2>/dev/null | awk -v d="$d" 'NR>1{print d, $5,$6,$8}'; done | sort -u +h "agent files (names and dates only)"; ls -l --time-style=long-iso /etc/felhom-agent /etc/felhom /var/lib/felhom-agent 2>/dev/null | awk '{print $1,$3,$5,$6,$7,$8}' +h "agent"; /usr/local/bin/felhom-agent --version 2>&1 | head -1 +h "wireguard (interface and latest handshake time only)"; wg show all latest-handshakes 2>/dev/null | awk '{print $1, "peer", substr($2,1,6) "…", strftime("%FT%TZ",$3,1)}' +h "tailscale"; command -v tailscale >/dev/null && tailscale status --json 2>/dev/null | python3 -c ' +import json,sys +d=json.load(sys.stdin); s=d.get("Self",{}); print("self key expiry:", s.get("KeyExpiry"), "online:", s.get("Online"))' || echo "no tailscale" +h "ssh host keys / authorized keys (counts only)"; ls /etc/ssh/ssh_host_*_key.pub 2>/dev/null | wc -l; for f in /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys; do [ -f "$f" ] && echo "$f lines=$(/usr/bin/grep -c -v -E '^\s*(#|$)' "$f")"; done +h "smart: power-on hours and wear (no serials)"; for d in $(lsblk -dno NAME,TYPE | awk '$2=="disk"{print $1}'); do printf '%s: ' $d; smartctl -A /dev/$d 2>/dev/null | /usr/bin/grep -i -E 'Power_On_Hours|Percentage Used|Power On Hours|Wear_Leveling|Media_Wearout|Available Spare:' | awk '{$1=$1;print}' | tr '\n' ';'; echo; done diff --git a/documentation/audits/end-dates-list-2026-11-11/tools/host-pin.py b/documentation/audits/end-dates-list-2026-11-11/tools/host-pin.py new file mode 100644 index 00000000..3f64853d --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/tools/host-pin.py @@ -0,0 +1,17 @@ +# host-pin.py — read-only: does the agent's pinned Proxmox fingerprint equal the certificate Proxmox serves? +# Prints the certificate's end date and yes/no. It prints no value from agent.json. +import json, subprocess, re +c = json.load(open("/etc/felhom-agent/agent.json")) +def find(o, p=""): + if isinstance(o, dict): + for k, v in o.items(): + if "ingerprint" in k and isinstance(v, str): + yield p + k, v + else: + yield from find(v, p + k + ".") +fp = subprocess.run(["openssl", "x509", "-in", "/etc/pve/local/pve-ssl.pem", "-noout", "-fingerprint", "-sha256", "-enddate"], capture_output=True, text=True).stdout +leaf = re.sub(r"[^0-9a-f]", "", fp.split("\n")[0].split("=")[1].lower()) +print("pve-ssl", fp.strip().split("\n")[1]) +for k, v in find(c): + n = re.sub(r"[^0-9a-f]", "", v.lower()) + print(k, "set" if n else "EMPTY", "equals pve-ssl leaf:", n == leaf) diff --git a/documentation/audits/end-dates-list-2026-11-11/tools/keyrings.sh b/documentation/audits/end-dates-list-2026-11-11/tools/keyrings.sh new file mode 100644 index 00000000..9c2d1ef2 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/tools/keyrings.sh @@ -0,0 +1,17 @@ +#!/bin/bash +# keyrings.sh NAME SSHTARGET — copy the PUBLIC apt keyring files of host and guest to DooPlex and print each key's +# id, creation date and end date. Public keys only. Read-only on the box (tar to stdout). +set -u +D=$(cd "$(dirname "$0")/.." && pwd); n=$1; t=$2; SSH="ssh -o BatchMode=yes -o ConnectTimeout=10" +export GNUPGHOME="$D/gnupg"; mkdir -p "$GNUPGHOME"; chmod 700 "$GNUPGHOME" +for where in host guest; do + w="$D/keyrings/$n/$where"; mkdir -p "$w" + cmd='tar -C / -cf - --ignore-failed-read etc/apt/trusted.gpg etc/apt/trusted.gpg.d etc/apt/keyrings usr/share/keyrings 2>/dev/null' + if [ $where = host ]; then $SSH "$t" "$cmd" | tar -C "$w" -xf - 2>/dev/null; else $SSH "$t" "pct exec 9201 -- sh -c '$cmd'" | tar -C "$w" -xf - 2>/dev/null; fi + printf '=== %s %s: keyring files %s\n' "$n" "$where" "$(find "$w" -type f | wc -l)" + find "$w" -type f | sort | while read f; do + gpg --batch --no-options --show-keys --with-colons "$f" 2>/dev/null | awk -F: -v f="${f#$w/}" ' + $1=="pub"||$1=="sub"{ printf "%s | %s %s algo=%s bits=%s | created %s | expires %s\n", f, $1, $5, $4, $3, strftime("%F",$6,1), ($7==""?"never":strftime("%F",$7,1)) } + $1=="uid"{ printf "%s | uid %s\n", f, $10 }' + done +done diff --git a/documentation/audits/end-dates-list-2026-11-11/tools/listconf.py b/documentation/audits/end-dates-list-2026-11-11/tools/listconf.py new file mode 100644 index 00000000..1e1f7d05 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/tools/listconf.py @@ -0,0 +1,9 @@ +"""listconf.py — the End-dates list's own words for make-table.py and make-matrix.py. Data only.""" +HELPERS = ("dates-box", "dates-hub") +KINDS = ["Certificate", "Key", "Window", "Version", "Account", "OS"] +WORD, WORDS = "thing", "Things" +TITLE = "The End-dates list" +ID_RE = r"D[BH]\d\d" +QUESTIONS = ["what, owner, where", "when it ends", "who renews, how early", "what breaks", "who is warned before", + "a box off past the date", "if renewal fails", "how do we know"] +CLASS_ROW = {"T1": "R-1379", "T2": "R-1380", "T3": "R-1381", "T4": "R-1382"} diff --git a/documentation/audits/end-dates-list-2026-11-11/tools/make-matrix.py b/documentation/audits/end-dates-list-2026-11-11/tools/make-matrix.py new file mode 100644 index 00000000..9528b0f5 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/tools/make-matrix.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python3 +"""Build MATRIX.md from TABLE.md, and check the table's shape. + +The same tool as `audits/endings-list-2026-11-10/tools/make-matrix.py`, made general: the list's own words are in `listconf.py`. + +Read-only on everything but MATRIX.md in this folder. It runs on DooPlex against the git clone only. + +Usage: make-matrix.py write ../MATRIX.md, print the totals + make-matrix.py --check write nothing; rc 1 when the table breaks a rule + +Rules checked (a broken rule is printed and makes rc 1): + - a heading is `### — · · NAMED|MENTIONED|UNNAMED` + - each block has exactly eight answer rows, numbered 1..8, each with one mark A, D, K or N + - an A row carries a backticked pointer; a K row an `R-`; an N row one class from listconf.py and a search S +""" +import os +import re +import sys +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import listconf as C + +HERE = os.path.dirname(os.path.abspath(__file__)) +AUDIT = os.path.dirname(HERE) +HEAD = re.compile(r"^### (" + C.ID_RE + r") — (.+?) · (?:" + "|".join(C.KINDS) + r") · (NAMED|MENTIONED|UNNAMED)\s*$") +ROW = re.compile(r"^\| ([1-8]) \| (.*) \| ([ADKN]) \| (.*) \|\s*$") +QUESTIONS = C.QUESTIONS +CLASS_ROW = C.CLASS_ROW +CLS = re.compile(r"\b(?:" + "|".join(C.CLASS_ROW) + r")\b") + + +def parse(path): + places, errors, cur = [], [], None + with open(path, encoding="utf-8") as fh: + for ln, line in enumerate(fh, 1): + if line.startswith("## TEST-ONLY") or line.startswith("## DOC-ONLY"): + cur = None + continue + if line.startswith("### "): + m = HEAD.match(line) + if not m: + errors.append(f"line {ln}: heading has the wrong shape: {line.strip()}") + cur = None + continue + cur = {"id": m.group(1), "name": m.group(2), "named": m.group(3), "marks": {}, "n": {}, "k": {}} + places.append(cur) + continue + m = ROW.match(line) + if not m or cur is None: + continue + q, mark, ptr = int(m.group(1)), m.group(3), m.group(4) + if q in cur["marks"]: + errors.append(f"{cur['id']}: question {q} is answered twice") + cur["marks"][q] = mark + if mark == "A" and "`" not in ptr: + errors.append(f"{cur['id']}-{q}: A with no pointer") + if mark == "K": + rows = re.findall(r"R-\d+", ptr) + if not rows: + errors.append(f"{cur['id']}-{q}: K with no row") + cur["k"][q] = rows + if mark == "N": + cls = CLS.findall(ptr) + if len(cls) != 1: + errors.append(f"{cur['id']}-{q}: N needs exactly one class, found {cls}") + elif not re.search(r"\bS\d+\b", ptr) and "NOBODY KNOWS" not in m.group(2): + errors.append(f"{cur['id']}-{q}: N with no search id") + cur["n"][q] = cls[0] if cls else "?" + for p in places: + if sorted(p["marks"]) != list(range(1, 9)): + errors.append(f"{p['id']}: has answers {sorted(p['marks'])}, wants 1..8") + return places, errors + + +def build(places): + out = ["# " + C.TITLE + " — the matrix", "", + "Made by `tools/make-matrix.py` from `TABLE.md`. Do not edit it by hand.", "", + "`A` answered · `D` decided, not built · `K` known gap (has a row) · `N` nobody asked. Columns are the eight", + "questions: " + " · ".join(f"{i} {q}" for i, q in enumerate(QUESTIONS, 1)) + ".", + "", "| " + C.WORD.capitalize() + " | Named | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | N |", "|---|---|---|---|---|---|---|---|---|---|---|"] + tot = {m: 0 for m in "ADKN"} + perq = {q: {m: 0 for m in "ADKN"} for q in range(1, 9)} + named = {"NAMED": 0, "MENTIONED": 0, "UNNAMED": 0} + members, krows = {c: [] for c in CLASS_ROW}, {} + for p in places: + named[p["named"]] += 1 + cells = [] + for q in range(1, 9): + mk = p["marks"].get(q, "?") + cells.append(f"**{mk}**" if mk == "N" else mk) + if mk in tot: + tot[mk] += 1 + perq[q][mk] += 1 + for q, c in p["n"].items(): + members.setdefault(c, []).append(f"{p['id']}-{q}") + for q, rows in p["k"].items(): + for r in rows: + krows.setdefault(r, []).append(f"{p['id']}-{q}") + out.append(f"| {p['id']} {p['name']} | {p['named'].lower()} | " + " | ".join(cells) + f" | {len(p['n'])} |") + n = len(places) + out += ["", "## Totals", "", + f"- {C.WORDS}: **{n}** in the table · named **{named['NAMED']}** · mentioned only **{named['MENTIONED']}** · " + f"named by no architecture document **{named['UNNAMED']}**.", + f"- Answers: **{n * 8}** · A **{tot['A']}** · D **{tot['D']}** · K **{tot['K']}** · N **{tot['N']}**.", + f"- {C.WORDS} with no N: **{sum(1 for p in places if not p['n'])}** of {n}.", + "", "| Question | A | D | K | N |", "|---|---|---|---|---|"] + for q in range(1, 9): + out.append(f"| {q} {QUESTIONS[q - 1]} | {perq[q]['A']} | {perq[q]['D']} | {perq[q]['K']} | {perq[q]['N']} |") + out += ["", "## The N answers, by class row", "", "| Class | Row | Count | Members (id-question) |", "|---|---|---|---|"] + for c in sorted(members): + out.append(f"| {c} | {CLASS_ROW.get(c, '?')} | {len(members[c])} | {', '.join(members[c])} |") + out += ["", "## The K answers, by existing row", "", "| Row | Count | Members |", "|---|---|---|"] + for r in sorted(krows, key=lambda x: int(x[2:])): + out.append(f"| {r} | {len(krows[r])} | {', '.join(krows[r])} |") + return "\n".join(out) + "\n", tot, named, members + + +def main() -> int: + places, errors = parse(os.path.join(AUDIT, "TABLE.md")) + for e in errors: + print("BROKEN " + e) + text, tot, named, members = build(places) + if "--check" not in sys.argv[1:]: + with open(os.path.join(AUDIT, "MATRIX.md"), "w", encoding="utf-8") as fh: + fh.write(text) + print(f"places {len(places)} named {named} marks {tot} " + + " ".join(f"{c}={len(m)}" for c, m in sorted(members.items())) + f" broken {len(errors)}") + return 1 if errors else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/documentation/audits/end-dates-list-2026-11-11/tools/make-table.py b/documentation/audits/end-dates-list-2026-11-11/tools/make-table.py new file mode 100644 index 00000000..6220861d --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/tools/make-table.py @@ -0,0 +1,126 @@ +#!/usr/bin/env python3 +"""make-table.py — build TABLE.md from the helpers' answers (evidence/helpers/resolved-*.json), marks.py and listconf.py. + +The same tool as `audits/endings-list-2026-11-10/tools/make-table.py`, made general: what differs between two lists +(the heading words, the helper names, the kinds, the questions) is in `listconf.py` beside it. + +A cell with an entry in marks.py carries the MAIN SESSION's words. Every other cell carries the helper's words and the +mark A; its pointers were opened by resolve-helper.py (the file and the line exist; the name the helper gave stands at +the line). A dead pointer is dropped here. Read-only but for TABLE.md. +""" +import json, os, re, subprocess, sys +import marks, listconf as C +HERE = os.path.dirname(os.path.abspath(__file__)); AUDIT = os.path.dirname(HERE) +REPO = os.path.abspath(os.path.join(HERE, "..", "..", "..", "..")); WS = os.path.dirname(REPO) +PREFIX = {"AG": os.path.join(WS, "felhom-agent"), "CT": os.path.join(WS, "felhom-controller", "controller"), + "CAT": os.path.join(WS, "app-catalog-felhom.eu"), "HUB": os.path.join(REPO, "hub", "internal"), "E": REPO, + "AR": os.path.join(REPO, "documentation", "architecture"), "RB": os.path.join(REPO, "documentation", "runbooks"), + "EV": os.path.join(AUDIT, "evidence")} +TOK = re.compile(r"`((?:AG|CT|CAT|HUB|E|AR|RB|EV)/[^`:\s]+)(?::(\d+)(?:-(\d+))?)?`") +R = {h: json.load(open(os.path.join(AUDIT, "evidence", "helpers", f"resolved-{h}.json")))["endings"] for h in C.HELPERS} +dropped = [] + + +def live(tok): + m = TOK.fullmatch(tok) + pre, rest = m.group(1).split("/", 1) + p = os.path.join(PREFIX[pre], rest) + if not os.path.exists(p): + return False + if m.group(2): + n = sum(1 for _ in open(p, encoding="utf-8", errors="replace")) + return int(m.group(3) or m.group(2)) <= n + return True + + +def split(ans, eid, q): + """helper answer -> (plain text, [live pointers])""" + ptrs = [] + for m in TOK.finditer(ans): + t = m.group(0) + if live(t): + if t not in ptrs: + ptrs.append(t) + else: + dropped.append(f"{eid}-{q} {t}") + t = TOK.sub("", ans) + t = re.sub(r"`~?:\d+(?:-\d+)?`", "", t) + t = re.sub(r"\(\s*(?:called|creator|see)?\s*[,;]?\s*\)", "", t) + t = re.sub(r"\s+([,.;:)])", r"\1", t) + t = re.sub(r"\(\s+", "(", t) + t = re.sub(r"([,;:])(?:\s*[,;:])+", r"\1", t) + t = re.sub(r"\s{2,}", " ", t).replace("|", "/").strip(" ;,·") + t = re.sub(r"\(\)", "", t) + if len(t) > 380: + cut = max(t.rfind(". ", 0, 380), t.rfind("; ", 0, 380)) + t = t[:cut + 1 if cut > 150 else 377].rstrip(" ;") + " …" + return t, ptrs + + +def named(m): + kind, why = m["named"] + if kind != "grep": + return kind, f"architecture: {why}" + r = subprocess.run(["/usr/bin/grep", "-rEic", "--include=*.md", why, PREFIX["AR"]], capture_output=True, text=True) + files = [l for l in r.stdout.split("\n") if l and not l.endswith(":0")] + lines = sum(int(l.rsplit(":", 1)[1]) for l in files) + return ("MENTIONED" if lines else "UNNAMED"), f"architecture folder: {lines} lines in {len(files)} files match `/{why}/` (a machine count, not a reading)" + + +def main(): + out = [open(os.path.join(HERE, "table-head.md"), encoding="utf-8").read().rstrip(), ""] + kindmap, bad, body = {}, [], [] + for eid, m in sorted(marks.M.items()): + e = R[m["src"][0]][m["src"][1]] + st, why = named(m) + worst, rows = "A", [] + for q in range(1, 9): + if q in m["o"]: + mark, text, ptr = m["o"][q] + if ptr is None: + _, ps = split(e["answers"].get(str(q), ""), eid, q) + ptr = " ".join(ps[:4]) + else: + mark = "A" + text, ps = split(e["answers"].get(str(q), ""), eid, q) + ptr = " ".join(ps[:4]) + if q == 8 and eid in marks.MEASURED: + mt, mp = marks.MEASURED[eid] + text = mt + " · " + text + ptr = (mp + " " + ptr).strip() + if mark == "A" and "`" not in ptr: # borrow the code path from question 1, then 2 + for alt in (1, 2, 3): + _, ps = split(e["answers"].get(str(alt), ""), eid, alt) + if ps: + ptr = " ".join(ps[:2]) + f" (the code path of question {alt})" + break + if mark == "A" and "`" not in ptr: + bad.append(f"{eid}-{q}: A with no pointer") + if not text: + bad.append(f"{eid}-{q}: empty answer") + rows.append(f"| {q} | {text} | {mark} | {ptr} |") + worst = max(worst, mark, key="ADKN".index) + kindmap.setdefault(m["seed"], []).append((eid, st, worst)) + body += [f"### {eid} — {m['name']} · {m['seed']} · {st}", "", + f"{why} · helper section: {m['src'][0]} „{e['title'].strip()}”", "", + "| # | Answer | Mark | Pointer |", "|---|---|---|---|"] + rows + ["", f"**Hand step in a runbook?** {m['rb']}", ""] + out += [f"## The kinds, and where each {C.WORD} is", "", f"| Kind | {C.WORDS} (worst mark) | Named · mentioned · named nowhere |", "|---|---|---|"] + for kind in C.KINDS: + v = kindmap.get(kind, []) + c = {s: sum(1 for x in v if x[1] == s) for s in ("NAMED", "MENTIONED", "UNNAMED")} + out.append(f"| {kind} | " + ", ".join(f"{x[0]} ({x[2]})" for x in v) + f" | {c['NAMED']} · {c['MENTIONED']} · {c['UNNAMED']} |") + for k in kindmap: + if k not in C.KINDS: + bad.append(f"kind {k} is not in listconf.KINDS") + out += ["", "## Not in the table", "", + "**Tiny (named once, not tabled):** " + " · ".join(marks.TINY) + ".", "", + "**Left out for another reason:**", ""] + [f"- {a} — {b}." for a, b in marks.LEFT_OUT] + out += ["", "---", ""] + body + open(os.path.join(AUDIT, "TABLE.md"), "w", encoding="utf-8").write("\n".join(out).rstrip() + "\n") + print(f"{C.WORDS} {len(marks.M)} · dead pointers dropped {len(dropped)} · broken {len(bad)}") + for b in bad + dropped: + print(" " + b) + return 1 if bad else 0 + + +sys.exit(main()) diff --git a/documentation/audits/end-dates-list-2026-11-11/tools/marks.py b/documentation/audits/end-dates-list-2026-11-11/tools/marks.py new file mode 100644 index 00000000..9d3f791e --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/tools/marks.py @@ -0,0 +1,266 @@ +"""marks.py — the main session's rulings for the End-dates list (2026-11-11). Data only; make-table.py reads it. + +One entry for each thing that is in the table: + src (helper, index) the helper section the default answers come from (evidence/helpers/resolved-.json) + name the heading + seed the kind (Certificate, Key, Window, Version, Account, OS) + named a fixed status with its reason, or ("grep", REGEX): 0 lines in the architecture folder = UNNAMED, else MENTIONED + o {question: (mark, answer, pointers)} — an answer the MAIN SESSION wrote after it read the pointer by eye. + pointers None = keep the helper's pointers for that question. + A question with no entry here takes the helper's answer and the mark A. + rb the ninth line: is the hand step written in a runbook? +""" +NM = "NOT MEASURED (DooPlex, ep0 and the accounts are outside this run). " + +M = { + # ------------------------------------------------------------------ on a box + "DB01": dict(src=("dates-box", 0), name="Web certificate of the household's address", seed="Certificate", + named=("grep", r"acme\.json|Let.s Encrypt|letsencrypt"), + o={2: ("A", "MEASURED 2026-11-11: the N100's certificate ends 2026-12-31, Tester 1's 2027-01-02, demo-hp's 2027-01-19. The life (90 days) is the issuer's rule, not ours.", + "`EV/box/demo-felhom-guest-dates.txt` `EV/box/tester-1-guest-dates.txt` `EV/box/demo-hp-guest-dates.txt`"), + 5: ("N", "Nobody. No part of the product reads the certificate's end date. If the renewal fails (the household's Cloudflare token ends, or the name check fails), the first sign is a browser warning on the end date.", + "`CT/internal/infra/infra.go:157-159` · class T1 · S01"), + 6: ("N", "NOBODY KNOWS. A box that was off past the end date starts with a dead certificate. Traefik must get a new one at the start. Nobody has tested this.", + "`CT/internal/infra/infra.go:157-159` · class T4 · S11")}, + rb="NO — no runbook says what to do when a household's certificate does not renew"), + "DB02": dict(src=("dates-box", 1), name="Proxmox's own certificate, pinned by the agent", seed="Certificate", + named=("grep", r"pve-ssl|tls\.fingerprint|TLS pin"), + o={2: ("A", "MEASURED 2026-11-11: the certificate ends 2028-07-16 on the N100, 2028-08-19 on demo-hp, 2028-10-02 on Tester 1. On all three the agent's pin equals this certificate. The pin itself has no date.", + "`EV/box/pin.txt` `AG/internal/proxmox/tls.go:40-58`"), + 3: ("N", "Proxmox renews it by itself, about 14 days before the end (its daily update job). No agent code reads the new fingerprint. The installer writes the pin once.", + "`EV/box/pveupdate-excerpt.txt` `AG/internal/proxmox/tls.go:40-58` `E/scripts/felhom-host-install.sh:3182-3185` · class T2 · S02"), + 4: ("N", "After Proxmox makes the new certificate, every call from the agent to Proxmox fails with 'TLS pin mismatch'. That is: guest start and stop, whole-guest backups, restore tests, disk work. First box: the N100, about 2028-07-02. The other two follow within three months.", + "`AG/internal/proxmox/tls.go:55` `EV/box/pin.txt` · class T2 · S02"), + 5: ("N", "Nobody before the day. No code compares the pin with the certificate's end date.", "`AG/internal/proxmox/tls.go:40-58` · class T2 · S02"), + 6: ("N", "No. Proxmox also renews a certificate that has ended, at its next daily job. The box comes back with a new certificate and the old pin.", + "`EV/box/pveupdate-excerpt.txt` · class T2 · S02"), + 7: ("N", "Here the renewal that works is the failure. The repair is by hand: write the new fingerprint into the agent's config and restart the agent. No code does it.", + "`AG/internal/proxmox/tls.go:40-58` · class T2 · S02")}, + rb="NO — a search of the runbooks for 'tls.fingerprint', 're-pin' and 'pin mismatch' finds nothing"), + "DB03": dict(src=("dates-box", 2), name="The agent's local-API certificate, pinned in the guest", seed="Certificate", + named=("NAMED", "`AR/03-host-agent.md:420-423`"), + o={5: ("N", "Nobody. The certificate lives 10 years and nothing reads its end date. The first one ends about 2036.", "`AG/internal/localapi/cert.go:25` · class T1 · S01")}, + rb="no hand step"), + "DB04": dict(src=("dates-box", 3), name="Off-site copy after the household stops it: the 14-day countdown", seed="Window", + named=("grep", r"14.day countdown|abandon"), + o={6: ("N", "NOBODY KNOWS for a box that is off during the whole countdown: the reminders at 5, 3 and 1 days come from the box. Nobody traced if the old store is removed with no reminder shown.", + "`CT/internal/backup/offbox_abandon.go:31-40` · class T4 · S23")}, + rb="no hand step"), + "DB05": dict(src=("dates-box", 4), name="Off-site history: how far back a household can go", seed="Window", + named=("grep", r"keepDaily|keep-daily|7 daily"), + o={5: ("A", "Nobody, and by ruling: the window is the rule. A point leaves it with no warning.", None)}, + rb="no hand step"), + "DB06": dict(src=("dates-box", 5), name="Off-site credentials: SSH key, host key, store password, PBS pin", seed="Key", + named=("grep", r"host key|known_hosts|sub-account"), + o={5: ("A", "Nobody, because there is no date. A failed night run is the sign.", None)}, + rb="yes — `RB/offsite-endpoint.md`"), + "DB08": dict(src=("dates-box", 7), name="Claim code (72 hours) and the one-use gate token", seed="Window", + named=("grep", r"claim code|claimCodeTTL|72 h"), + o={5: ("A", "Nobody before the end. The household sees the dead code at the form, and the operator sends a new one.", None)}, + rb="no hand step"), + "DB09": dict(src=("dates-box", 8), name="Share link", seed="Window", named=("grep", r"/s/|share token|LauncherShareToken"), + o={5: ("A", "Nobody, because there is no date. The link ends only when the household changes it.", None)}, + rb="no hand step"), + "DB11": dict(src=("dates-box", 10), name="Signed operator jobs: the validity window", seed="Window", + named=("NAMED", "`AR/04-control-plane-authorization.md:139-150`"), + o={5: ("A", "Nobody before. A job that ends with no result is what the operator sees. A new job is signed.", None)}, + rb="yes — the build skill's agent section"), + "DB12": dict(src=("dates-box", 11), name="Operator signing keys on each box", seed="Key", + named=("NAMED", "`AR/04-control-plane-authorization.md:139-150`"), + o={3: ("D", "The keys have no end date. A change of key is designed as a signed order of its own. Only the name of that order exists in code.", + "`AR/04-control-plane-authorization.md:139-150`"), + 7: ("K", "If the keys are lost, no box accepts a signed job again without a visit. The keys exist on DooPlex only.", "R-924")}, + rb="no — the design exists, a procedure does not"), + "DB13": dict(src=("dates-box", 12), name="The agent's Proxmox access token", seed="Key", + named=("grep", r"pveum|API token|privsep"), + o={2: ("A", "MEASURED 2026-11-11 on the three boxes: the user and its token have no end date (expire 0).", "`EV/box/pin.txt` `E/scripts/felhom-host-install.sh:2448`"), + 5: ("A", "Nobody, because there is no date.", "`EV/box/pin.txt`")}, + rb="no hand step"), + "DB16": dict(src=("dates-box", 15), name="Package signing keys: Docker in the guest, Proxmox and Debian on the host", seed="Key", + named=("MENTIONED", "`AR/11-os-updates.md:657`"), + o={2: ("A", "MEASURED 2026-11-11 on the three boxes: Docker's key never ends. The Debian keys in use end 2029-01-15 at the soonest. The Proxmox key for this series ends 2034-11-10. Two keys that end 2027-02-03 and 2027-04-12 are in the 'removed keys' file; nothing uses them.", + "`EV/box/keyrings.txt`"), + 5: ("N", "Nobody before the end. After it the box reports a failed update. The first date is 2029-01-15.", "`AR/11-os-updates.md:657` `EV/box/keyrings.txt` · class T1 · S10"), + 6: ("N", "NOBODY KNOWS. A box that was off for years has old keys. The new keys come as a package, and the package list is checked with the old keys.", + "`AR/11-os-updates.md:657` · class T4 · S11")}, + rb="NO — the document says a key change is 'a slow-lane act for a person'; no procedure is written"), + "DB17": dict(src=("dates-box", 16), name="Debian 13 and Proxmox 9: the series the product is tied to", seed="OS", + named=("grep", r"trixie|Debian 13|Proxmox (VE )?9"), + o={2: ("N", "No document gives the day when security fixes stop for Debian 13 or for Proxmox 9. MEASURED: the three boxes run Debian 13.7 and Proxmox 9.2.", + "`EV/box/demo-hp-host-dates.txt` `AR/11-os-updates.md:1` · class T3 · S09"), + 3: ("N", "No plan with a date. The move to the next series is not designed.", "`AR/11-os-updates.md:1` · class T3 · S09"), + 5: ("N", "Nobody. The dates are public; nothing in the product or in a document reads them.", "`AR/11-os-updates.md:1` · class T1 · S09")}, + rb="NO"), + "DB18": dict(src=("dates-box", 17), name="Catalog app images and their recorded digests", seed="Version", + named=("MENTIONED", "`AR/09-update-architecture.md:391`"), + o={5: ("N", "Nobody. No check asks the registries if every recorded image is still served. The document speaks of the day a version is recorded, not of the days after.", + "`AR/09-update-architecture.md:391` `CAT/scripts/check-currency.py:22` · class T1 · S21")}, + rb="no hand step"), + "DB19": dict(src=("dates-box", 18), name="Images the controller itself starts (Traefik, tunnel client, file browser, Samba)", seed="Version", + named=("grep", r"cloudflared|TraefikImage|traefik:v"), + o={2: ("N", "No document says for how long Cloudflare accepts an old tunnel client. MEASURED on the N100: tunnel client 2026.10.3, Traefik v3.7.16.", + "`EV/box/demo-felhom-guest-dates.txt` · class T3 · S12"), + 5: ("N", "Nobody.", "`EV/box/demo-felhom-guest-dates.txt` · class T1 · S12"), + 6: ("N", "NOBODY KNOWS if the tunnel of a box that was off for a year still connects with its old client. Without the tunnel the box gets no update from outside.", + "`EV/box/demo-felhom-guest-dates.txt` · class T4 · S12")}, + rb="no hand step"), + "DB20": dict(src=("dates-box", 19), name="Published agent versions: the newest five stay", seed="Version", + named=("NAMED", "`AR/09-update-architecture.md` §3, D50"), o={}, rb="yes — the build skill's agent section"), + "DB21": dict(src=("dates-box", 20), name="The golden image's age (7 days) and the waiver (14 days)", seed="Version", + named=("NAMED", "`AR/09-update-architecture.md:1010-1018`"), + o={2: ("A", "MEASURED 2026-11-11: the newest golden is 8 days old (baked 2026-11-03). The limit is 7. No waiver file exists. The gate fails today.", + "`E/scripts/golden_currency_gate.py:169` `EV/box/golden-gate.txt`"), + 7: ("K", "A push of code to this repository is refused until a golden is baked or a dated waiver is written. A push of documents passes with a notice.", "R-468")}, + rb="yes — `RB/RUNBOOK-manual-build.md`"), + "DB24": dict(src=("dates-box", 23), name="Copies kept before a restore: deleted after 30 unused days", seed="Window", + named=("grep", r"restore cop|restore-undo|D62"), + o={5: ("N", "Nobody. The household is not told before the copy goes. The household can delete it sooner by hand.", + "`CT/internal/backup/restore_copy_sweep.go:18-29` · class T1 · S05"), + 6: ("N", "A box that was off for more than 30 days deletes the copy at its first sweep after the start: the age is counted from the folder's time stamp by the wall clock. Nobody asked if the off time must count.", + "`CT/internal/backup/restore_copy_sweep.go:21-29` · class T4 · S11")}, + rb="no hand step"), + # ------------------------------------------------------------------ hub, operator, outside + "DH01": dict(src=("dates-hub", 0), name="Connect link in the household's e-mail (7 days)", seed="Window", + named=("grep", r"self-bind|selfbind|/bind/"), + o={5: ("A", "Nobody before, by design: the page of a dead link has a button that sends a new one.", None)}, + rb="no hand step"), + "DH05": dict(src=("dates-hub", 4), name="'Switched off on purpose' hold (14 days)", seed="Window", + named=("grep", r"deliberate.off|switched off on purpose"), o={}, rb="no hand step"), + "DH08": dict(src=("dates-hub", 7), name="Storage Box sub-account password, and the hub's seal key", seed="Key", + named=("grep", r"seal key|sub-account"), + o={3: ("D", NM + "The password changes when the operator presses Re-issue. A change of the seal key is 'not built'.", "`RB/secrets.md:159`"), + 5: ("A", "Nobody, because there is no date.", None)}, + rb="partly — `RB/secrets.md:159` says the seal key change is not built"), + "DH09": dict(src=("dates-hub", 8), name="PBS off-site token on ep0", seed="Key", named=("NAMED", "`AR/05-hub-architecture.md:379-387`"), o={}, + rb="yes — `RB/offsite-endpoint.md`"), + "DH10": dict(src=("dates-hub", 9), name="WireGuard keys of the box's tunnel to the off-site endpoint", seed="Key", + named=("NAMED", "`AR/06-offsite-connectivity.md:200-205`"), + o={5: ("A", "Nobody, because there is no date.", None)}, rb="yes — `RB/offsite-endpoint.md`"), + "DH14": dict(src=("dates-hub", 13), name="Recovery mode for a host (30 minutes to 4 hours)", seed="Window", + named=("grep", r"recovery mode|defaultRecoveryTTL"), + o={5: ("A", "Nobody before. The operator who opened the window is present; he opens it again.", None)}, + rb="yes — the recovery runbooks"), + "DH15": dict(src=("dates-hub", 14), name="Break-glass password of a host", seed="Key", named=("grep", r"break-glass|host_recovery"), o={}, + rb="yes — `RB/break-glass-sheet.md:75` (print the sheet again each year)"), + "DH19": dict(src=("dates-hub", 18), name="The installer ISO", seed="Version", named=("grep", r"ISO_VERSION|installer ISO|iso\.felhom\.eu"), + o={3: ("N", "No document says how old a public ISO may be, or when to build a new one. The ISO carries a Proxmox base and a first-boot program that age.", + "`E/scripts/iso/build-felhom-iso.sh:51` `RB/iso-release-gate.md:1` · class T3 · S22")}, + rb="yes for a build — `RB/iso-release-gate.md`; NO for 'when'"), + "DH20": dict(src=("dates-hub", 19), name="The certificate that the hub's address serves", seed="Certificate", + named=("grep", r"cert-manager|letsencrypt-prod"), + o={2: ("A", "Read from outside 2026-11-11, with no login: the certificate of the hub's address ends 2027-01-12. The download address ends 2026-12-23, the ISO address 2026-12-27, the Gitea address 2027-01-03, the website 2027-01-28.", + "`EV/box/public-reads.txt`"), + 5: ("N", NM + "In the Felhom repositories nothing warns before this date. If it ends, every box fails to report, and the hub then sees all boxes as silent.", + "`E/manifests/hub.yaml:342` `EV/box/public-reads.txt` · class T1 · S01")}, + rb="NO"), + "DH22": dict(src=("dates-hub", 21), name="Operator access tokens, among them CC's hub token", seed="Key", + named=("NAMED", "`AR/05-hub-architecture.md` §17"), + o={5: ("N", "Nobody. A token lives 90 days by default and 365 at most. The token page shows the date; no monitor reads it. On the end date the release step that sets a floor answers 401.", + "`HUB/web/tokens.go:29-30` `HUB/web/tokens.go:180` · class T1 · S03")}, + rb="partly — the build skill says '401 = token expired'; no step says to look at the date before"), + "DH23": dict(src=("dates-hub", 22), name="A household's own Cloudflare token", seed="Key", named=("grep", r"Cloudflare token|CF_DNS_API_TOKEN"), + o={2: ("N", "The hub and the box do not know the end date that the household set at Cloudflare. No document tells the household to set none.", + "`HUB/cloudflare/reach.go:14` · class T3 · S15"), + 5: ("N", "Nobody. When the token ends, the web certificate stops renewing (see DB01), with no sign until the certificate's own end.", + "`CT/internal/infra/infra.go:157-159` · class T1 · S15"), + 6: ("A", "The running apps and the tunnel do not use this token. The certificate renewal does.", "`CT/internal/infra/infra.go:157-159`")}, + rb="NO"), + "DH24": dict(src=("dates-hub", 23), name="The mail service key (all household and operator mail)", seed="Account", + named=("MENTIONED", "`AR/_recovery-inventory-2026-07-28.md`"), + o={2: ("N", NM + "No document gives an end date for the key or for the account.", "`RB/secrets.md:33` · class T3 · S08"), + 5: ("N", "Nobody. No scheduled check proves that mail still goes out. If the key ends, no alarm mail can say so.", "`RB/secrets.md:33` · class T1 · S08")}, + rb="yes for a new key — `RB/secrets.md:33`"), + "DH25": dict(src=("dates-hub", 24), name="The hub's global bearer key", seed="Key", named=("NAMED", "`AR/13-threat-model.md:70`"), o={}, + rb="yes — `RB/secrets.md:124`"), + "DH26": dict(src=("dates-hub", 25), name="Pinned tool versions in builds and CI", seed="Version", named=("grep", r"security.watch|pinned by digest"), o={}, + rb="no hand step"), + "DH27": dict(src=("dates-hub", 26), name="Domains and registrars", seed="Account", named=("grep", r"registrar|domain renew"), + o={2: ("N", NM + "No document gives a registrar or a renewal date for any of the product's names. This run could not read the public records: no whois tool here, and the public lookups for .eu and .hu gave no answer.", + "`EV/box/public-reads.txt` · class T3 · S06"), + 5: ("N", "Nobody that a document names. A name that ends takes down the hub address, the website, the mail and every household address under it.", + "`EV/box/public-reads.txt` · class T1 · S06")}, + rb="NO"), + "DH28": dict(src=("dates-hub", 27), name="Cloudflare account, the site publish token, the ISO store keys", seed="Account", + named=("grep", r"CF_PAGES_TOKEN|R2 "), + o={2: ("N", NM + "No document gives an end date for the account, for the publish token or for the store keys.", "`E/scripts/publish-site-cloudflare.sh:12` · class T3 · S07"), + 5: ("N", "Nobody.", "`E/scripts/publish-site-cloudflare.sh:12` · class T1 · S07")}, + rb="NO"), + "DH29": dict(src=("dates-hub", 28), name="Hetzner account: the off-site endpoint and the storage boxes", seed="Account", + named=("MENTIONED", "`AR/_recovery-inventory-2026-07-28.md:1019`"), + o={2: ("N", NM + "No document gives a payment day, a contract end or the end of the API token. The recovery inventory asks the question and no document answers it.", + "`AR/_recovery-inventory-2026-07-28.md:1019` · class T3 · S07"), + 5: ("N", "Nobody before. After the end, the hub's 'off-site copy is old' alarm fires at 8 days.", "`HUB/monitor/deadline_tiers.go:1` · class T1 · S07")}, + rb="NO"), + "DH30": dict(src=("dates-hub", 29), name="Tailscale node keys (the operator's remote way in)", seed="Key", + named=("grep", r"tailscale|tailnet"), + o={2: ("A", "DOCUMENT: demo-hp's key ends 2027-01-17; the other two nodes have no end. MEASURED 2026-11-11: the Tailscale service is not running on demo-hp and Tester 1; it runs on the N100. So the date could not be read on the box.", + "`E/documentation/operations/tailscale.md:41-43` `EV/box/pin.txt`"), + 5: ("N", "Nobody. The date is on the Machines page of the Tailscale console.", "`E/documentation/operations/tailscale.md:41-43` · class T1 · S04")}, + rb="yes — `E/documentation/operations/tailscale.md:41-43`"), + "DH31": dict(src=("dates-hub", 30), name="Gitea and registry credentials", seed="Key", named=("grep", r"gitea.{0,40}(token|credential)"), + o={2: ("N", NM + "No document gives an end date for the scoped token or for CC's Gitea token.", "`RB/secrets.md:1` · class T3 · S13"), + 5: ("N", "Nobody.", "`RB/secrets.md:1` · class T1 · S13"), + 6: ("A", "Not a box matter: a box pulls its images with no login.", "`RB/secrets.md:1`")}, + rb="yes for a new token — `RB/secrets.md`"), + "DH32": dict(src=("dates-hub", 31), name="CC's secret store: the tokens inside", seed="Key", named=("grep", r"cc-secrets|sec run"), + o={2: ("N", "The store holds a value and no end date. A token with an end at its issuer (the GitHub advisory token, the hub token) ends with no record here.", + "`E/scripts/security_watch.py:1` · class T3 · S14"), + 5: ("N", "Nobody. The nightly security watch and the release step fail on the day.", "`E/scripts/security_watch.py:1` · class T1 · S14")}, + rb="NO"), + "DH33": dict(src=("dates-hub", 32), name="SSH keys between the hub, DooPlex and the endpoint", seed="Key", + named=("grep", r"ssh key|id_ed25519"), o={}, rb="yes — `RB/offsite-endpoint.md:313`"), +} + +MEASURED = { + "DB01": ("MEASURED: the three end dates, and one renewal by the product on demo-hp (the certificate was issued again on 2026-10-21)", "`EV/box/demo-hp-guest-dates.txt`"), + "DB02": ("MEASURED: the three end dates, pin equals certificate on all three, and Proxmox's own renewal rule read on demo-hp", "`EV/box/pin.txt` `EV/box/pveupdate-excerpt.txt`"), + "DB13": ("MEASURED: expire 0 on three boxes", "`EV/box/pin.txt`"), + "DB16": ("MEASURED: every key's end date on three hosts and three guests", "`EV/box/keyrings.txt`"), + "DB21": ("MEASURED: the gate's own output today", "`EV/box/golden-gate.txt`"), + "DH20": ("READ FROM OUTSIDE, no login", "`EV/box/public-reads.txt`"), +} + +TINY = [ + "login sessions on the box and on the hub (hours to 7 days; a new login repairs it)", + "the one-use codes of the recovery ceremony (minutes)", + "the guest's token for the agent and the hub key (no date)", + "pinned build tools of the controller (a build fails loudly)", + "the hub's claim code and pairing code (no date; three resets a day)", + "operator actions in the queue (24 hours; a restart order 30 minutes)", + "the vouch alarm (2 hours)", + "the Debian template that the golden is built from (a bake fails loudly; the operator changes the default)", +] +LEFT_OUT = [ + ("The recovery passphrase", "it stops on an event (a new one is made), not on a date"), + ("Kept escrow packages", "they have no end date by ruling; the 14-day countdown that removes one is DB04"), + ("The off-site clean-up window (20 minutes, 6.5 days)", "it is a timer, so it is on the Clock list"), + ("The hub's view of signed jobs, signing keys, the golden's age, the agent versions, and OS support", "the same things as DB11, DB12, DB21, DB20 and DB17"), + ("WireGuard re-key on the box", "the same thing as DH10"), +] + +# ------------------------------------------------------------------ second pass +# Helper answers to questions 3 to 7 that said "not found" or "candidate": the main session's words in their place. +NODATE = "No renewal on a date exists, so none can fail on a date." +HAND = "The renewal is the operator's hand step at the outside account. If it is not done, the thing ends (question 4)." +for _id, _q, _cell in [ + ("DB01", 3, ("A", "Traefik renews by itself, by its own default lead time. MEASURED on demo-hp: it got a new certificate on 2026-10-21, 90 days before the end date 2027-01-19.", None)), + ("DB01", 7, ("N", "Traefik tries again by itself. The old certificate works until its end date. Nothing in the product reads the result, so a renewal that fails for weeks is silent.", + "`CT/internal/infra/templates/traefik.yml.tmpl:51` · class T1 · S01")), + ("DB03", 4, ("A", "At the end date nothing breaks: the controller compares the fingerprint only, not the date. A NEW certificate is what breaks the link.", None)), + ("DB05", 7, ("K", "A clean-up that the guard refuses leaves more history than the rule says, not less.", "R-1294")), + ("DB06", 6, ("A", "Yes if nothing changed outside. A guest that was built again gets a new store password and cannot open the old store; the 14-day countdown (DB04) is the way out.", None)), + ("DB09", 4, ("A", "Nothing ends. A link that was given away years ago still opens the page, until the household changes it.", None)), + ("DB19", 4, ("A", "If Cloudflare stops accepting an old tunnel client, the household's address stops answering from outside while the box is healthy. If a registry drops a tag, a box that is built again cannot start that part.", None)), + ("DH08", 7, ("A", NODATE, None)), ("DH15", 7, ("A", NODATE, None)), ("DH25", 7, ("A", NODATE, None)), ("DH33", 7, ("A", NODATE, None)), + ("DH23", 7, ("A", "The renewal is the household's hand step at Cloudflare. If it is not done, the web certificate stops renewing (DB01).", None)), + ("DH27", 7, ("A", HAND, None)), ("DH29", 7, ("A", HAND, None)), ("DH30", 7, ("A", HAND, None)), ("DH31", 7, ("A", HAND, None)), ("DH32", 7, ("A", HAND, None)), + ("DH27", 3, ("A", "The operator, at the registrar.", None)), + ("DH24", 7, ("A", "If the key ends and no new key is made, no mail goes out, alarm mail included (question 5).", None)), + ("DH25", 6, ("A", "No date drives it. Boxes use their own keys since 2026-10-12. What a box does that holds only the old global key after a change was not traced in this run.", None)), + ("DH26", 7, ("A", "A CI job that does not run leaves no result. Whether a missed night run is seen is a question of the Clock list.", None)), + ("DH20", 3, ("A", "NOT MEASURED on DooPlex. cert-manager on DooPlex renews by itself. Read from outside: the certificate served today was issued on 2026-10-14, so a renewal worked.", + "`E/manifests/hub.yaml:342` `EV/box/public-reads.txt`")), + ("DH20", 6, ("A", "Yes. The agent trusts the public issuers, not one certificate, so a renewed certificate is accepted.", "`AG/internal/hub/client.go:41-51`")), + ("DH20", 7, ("A", "The old certificate serves until its end date. Nothing in the Felhom repositories warns (question 5).", "`E/manifests/hub.yaml:342`")), +]: + M[_id]["o"][_q] = _cell diff --git a/documentation/audits/end-dates-list-2026-11-11/tools/public-reads.sh b/documentation/audits/end-dates-list-2026-11-11/tools/public-reads.sh new file mode 100644 index 00000000..ef2f7c89 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/tools/public-reads.sh @@ -0,0 +1,29 @@ +#!/bin/bash +# public-reads.sh — what any visitor can read with no login: the TLS certificate a public name serves, and the public +# registry record of a domain. Read-only; no account call. For a certificate: subject, issuer, two dates, fingerprint. +set -u +printf 'taken %s from DooPlex\n' "$(date -u +%FT%TZ)" +printf '\n=== certificates served by public names ===\n' +for n in "$@"; do + case "$n" in */*) continue;; esac + ip=$(getent ahostsv4 "$n" | awk 'NR==1{print $1}') + out=$(printf '' | timeout 8 openssl s_client -connect "$n:443" -servername "$n" 2>/dev/null | openssl x509 -noout -subject -issuer -startdate -enddate -fingerprint -sha256 2>/dev/null | tr '\n' ' ') + printf '%s [resolves here to %s] -> %s\n' "$n" "${ip:-none}" "${out:-NO CERTIFICATE READ}" +done +printf '\n=== public registry records (domains) ===\n' +for d in "$@"; do + case "$d" in */*) d=${d#domain/};; *) continue;; esac + printf -- '--- %s\n' "$d" + if command -v whois >/dev/null; then + timeout 15 whois "$d" 2>&1 | /usr/bin/grep -i -E 'expir|renew|paid-till|registered|created|record created|registrar:|status|^domain' | /usr/bin/grep -v -i -E 'terms|abuse|notice' | head -12 + else + printf 'no whois tool on DooPlex\n' + fi + tld=${d##*.} + case "$tld" in eu) u="https://rdap.eu/domain/$d";; hu) u="https://rdap.nic.hu/domain/$d";; *) u="https://rdap.org/domain/$d";; esac + timeout 15 curl -s -L -m 12 -H 'Accept: application/rdap+json' "$u" | python3 -c ' +import json,sys +try: d=json.load(sys.stdin) +except Exception as e: print("rdap: no JSON answer"); sys.exit(0) +print("rdap status:", d.get("status"), "| events:", [(e.get("eventAction"), e.get("eventDate")) for e in d.get("events",[])] or "none", "| error:", d.get("errorCode"))' +done diff --git a/documentation/audits/end-dates-list-2026-11-11/tools/resolve-helper.py b/documentation/audits/end-dates-list-2026-11-11/tools/resolve-helper.py new file mode 100644 index 00000000..13152287 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/tools/resolve-helper.py @@ -0,0 +1,155 @@ +#!/usr/bin/env python3 +"""resolve-helper.py HELPER.md [--json OUT] — open every pointer in a helper's answer BEFORE it may enter the table. + +A helper (a read-only subagent) writes pointers in three shapes: a full one (`CT/internal/stacks/delete.go:624`), a +bare file name (`delete.go:756`) and a short one (`:1025`, "same file as the last one"). This tool + 1. resolves each to a full `PREFIX/path:line[-line]`, + 2. says DEAD when the file or the line does not exist, UNRESOLVED when a bare name fits no file or more than one, + 3. checks the NAME the helper put in front of the pointer (a backticked identifier within 90 characters, with no other + pointer between): it must stand within 3 lines of the cited range. IDENT-OK / IDENT-MISS / NO-IDENT. +It does not judge the claim. A claim that carries a mark in TABLE.md is read by the main session (check-pointers.py +prints the lines). Read-only: it opens files in the git clones on DooPlex and prints. +""" +import json, os, re, sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +REPO = os.path.abspath(os.path.join(HERE, "..", "..", "..", "..")) +WS = os.path.dirname(REPO) +PREFIX = {"AG": os.path.join(WS, "felhom-agent"), "CT": os.path.join(WS, "felhom-controller", "controller"), + "CAT": os.path.join(WS, "app-catalog-felhom.eu"), "HUB": os.path.join(REPO, "hub", "internal"), "E": REPO, + "AR": os.path.join(REPO, "documentation", "architecture"), "RB": os.path.join(REPO, "documentation", "runbooks")} +RANGES = r"~?:?(\d+(?:-\d+)?(?:\s?,\s?:?\d+(?:-\d+)?)*)" +PTR = re.compile(r"(?P(?:%s)/[\w./@+-]+\.\w+)(?::(?P\d+(?:-\d+)?(?:,\s?:?\d+(?:-\d+)?)*))?" + r"|(?[\w.-]+\.(?:go|sh|py|md|yml|yaml|html|json|service|timer))(?::(?P
\d+(?:-\d+)?(?:,\s?:?\d+(?:-\d+)?)*))" + r"|(?\d+(?:-\d+)?)(?![\d:])" % "|".join(PREFIX)) +IDENT = re.compile(r"`([^`]{2,80})`") +_cache, _walk = {}, {} + + +def lines_of(path): + if path not in _cache: + try: + _cache[path] = open(path, encoding="utf-8", errors="replace").read().split("\n") + except OSError: + _cache[path] = None + return _cache[path] + + +def find_base(pre, base): + if pre not in _walk: + idx = {} + for d, dn, fn in os.walk(PREFIX[pre]): + dn[:] = [x for x in dn if x not in (".git", "node_modules", "vendor", "evidence")] + for f in fn: + idx.setdefault(f, []).append(os.path.relpath(os.path.join(d, f), PREFIX[pre])) + _walk[pre] = idx + return _walk[pre].get(base, []) + + +def check(full, rng, text, pos): + pre, rest = full.split("/", 1) + ls = lines_of(os.path.join(PREFIX[pre], rest)) + if ls is None: + return "DEAD", "NO-IDENT" + if not rng: + return "OK", "NO-IDENT" + a, _, b = rng.partition("-") + a, b = int(a), int(b or a) + if b > len(ls): + return "DEAD", "NO-IDENT" + back = text[max(0, pos - 90):pos] + cut = max(back.rfind(":"), back.rfind(";"), back.rfind(",")) # a pointer or a clause break in between: do not reach over it + ids = [m for m in IDENT.finditer(back) if m.start() > cut - 1 or cut < 0] + ids = ids or list(IDENT.finditer(back))[-1:] + if not ids: + return "OK", "NO-IDENT" + name = re.sub(r"[()…]|\.\.\.", "", ids[-1].group(1)).strip().split(" ")[0].split("(")[0] + name = name.split(".")[-1] if re.fullmatch(r"[\w.]+", name) else name + win = "\n".join(ls[max(0, a - 4):b + 3]) + return "OK", ("IDENT-OK" if name and name in win else f"IDENT-MISS({name})") + + +def resolve(text, known, default_pre): + """Returns (text with every pointer rewritten as a backticked full token, [records]).""" + out, recs, last, i = [], [], None, 0 + for m in PTR.finditer(text): + out.append(text[i:m.start()]) + i = m.end() + if m.group("full"): + full, rngs = m.group("full"), m.group("fr") + elif m.group("bare"): + base, rngs = m.group("bare"), m.group("br") + full = known.get(base) + if not full: + c = find_base(default_pre, base) + full = f"{default_pre}/{c[0]}" if len(c) == 1 else None + if not full: + recs.append({"ptr": m.group(0), "file": "UNRESOLVED", "ident": "NO-IDENT"}) + out.append(m.group(0)) + continue + else: + full, rngs = last, m.group("sr") + if not full: + out.append(m.group(0)) + continue + last = full + known.setdefault(os.path.basename(full), full) + toks = [] + for r in re.split(r"\s?,\s?:?", rngs) if rngs else [None]: + st, idn = check(full, r, text, m.start()) + tok = f"{full}:{r}" if r else full + recs.append({"ptr": tok, "file": st, "ident": idn}) + toks.append(f"`{tok}`") + out.append(" ".join(toks)) + out.append(text[i:]) + return "".join(out), recs + + +def main(): + src = sys.argv[1] + text = open(src, encoding="utf-8").read() + default_pre = "CT" if "controller" in src else "AG" if "agent" in src else "HUB" if "hub" in src else "AR" + known = {} + for m in re.finditer(r"((?:%s)/[\w./@+-]+\.\w+)" % "|".join(PREFIX), text): # first pass: every full path the helper named + known.setdefault(os.path.basename(m.group(1)), m.group(1)) + endings, cur, allrecs, lastq = [], None, [], None + for line in text.split("\n"): + h = re.match(r"^### (.+)$", line) + if h: + cur = {"title": h.group(1).strip(), "answers": {}, "extra": []} + endings.append(cur) + lastq = None + continue + if cur is None: + continue + a = re.match(r"^([1-8])\. (.*)$", line) + body = a.group(2) if a else line + new, recs = resolve(re.sub(r"`((?:%s)/[^`]+)`" % "|".join(PREFIX), r"\1", body), known, default_pre) + for r in recs: + r["ending"] = cur["title"] + r["q"] = a.group(1) if a else (lastq or "-") + allrecs += recs + if a: + cur["answers"][a.group(1)] = new + lastq = a.group(1) + elif line.strip() and lastq and not line.startswith(("#", "---")): + cur["answers"][lastq] += " · " + new.strip().lstrip("-* ") # a continuation line belongs to the answer above it + elif line.strip(): + cur["extra"].append(new) + n = len(allrecs) + dead = [r for r in allrecs if r["file"] != "OK"] + miss = [r for r in allrecs if r["ident"].startswith("IDENT-MISS")] + ok = sum(1 for r in allrecs if r["ident"] == "IDENT-OK") + print(f"{os.path.basename(src)}: endings {len(endings)} · pointers {n} · dead/unresolved {len(dead)} · " + f"name found at the line {ok} · name NOT at the line {len(miss)} · no name to check {n - ok - len(miss) - len(dead)}") + for r in dead: + print(f" {r['file']:<10} {r['ptr']} [{r['ending'][:50]} q{r['q']}]") + for r in miss: + print(f" {r['ident']:<28} {r['ptr']} [{r['ending'][:50]} q{r['q']}]") + if "--json" in sys.argv: + json.dump({"endings": endings, "pointers": allrecs}, open(sys.argv[sys.argv.index("--json") + 1], "w"), indent=1) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/documentation/audits/end-dates-list-2026-11-11/tools/run-box.sh b/documentation/audits/end-dates-list-2026-11-11/tools/run-box.sh new file mode 100644 index 00000000..161d4f21 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/tools/run-box.sh @@ -0,0 +1,26 @@ +#!/bin/bash +# run-box.sh KIND NAME SSHTARGET — run tools/host-KIND.sh on the host and tools/guest-KIND.sh in guest 9201, read-only. +# Output: box/NAME-host-KIND.txt and box/NAME-guest-KIND.txt beside this folder. +set -u +D=$(cd "$(dirname "$0")/.." && pwd) +k=$1; n=$2; t=$3 +SSH="ssh -o BatchMode=yes -o ConnectTimeout=10" +{ + printf '# host-%s %s taken %s from DooPlex\n' "$k" "$n" "$(date -u +%FT%TZ)" + timeout 240 $SSH "$t" 'bash -s' < "$D/tools/host-$k.sh" + printf 'RC=%s\n' "$?" +} > "$D/box/$n-host-$k.txt" 2>&1 +{ + printf '# guest-%s %s taken %s from DooPlex\n' "$k" "$n" "$(date -u +%FT%TZ)" + timeout 240 $SSH "$t" 'pct exec 9201 -- bash -s' < "$D/tools/guest-$k.sh" + printf 'RC=%s\n' "$?" +} > "$D/box/$n-guest-$k.txt" 2>&1 +if [ "$k" = clock ]; then + { + printf '# offset %s: box clock minus DooPlex clock (midpoint of the call), seconds\n' "$n" + for i in 1 2 3 4 5; do + a=$(date +%s.%N); b=$($SSH "$t" 'date +%s.%N'); c=$(date +%s.%N) + awk -v a="$a" -v b="$b" -v c="$c" 'BEGIN{printf "offset=%.3f rtt=%.3f\n", b-(a+c)/2, c-a}' + done + } > "$D/box/$n-offset.txt" 2>&1 +fi diff --git a/documentation/audits/end-dates-list-2026-11-11/tools/run-pin.sh b/documentation/audits/end-dates-list-2026-11-11/tools/run-pin.sh new file mode 100644 index 00000000..a0251167 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/tools/run-pin.sh @@ -0,0 +1,9 @@ +#!/bin/bash +# run-pin.sh — run host-pin.py on the three test hosts, read-only. Output: box/pin.txt +D=$(cd "$(dirname "$0")/.." && pwd) +{ printf 'taken %s from DooPlex\n' "$(date -u +%FT%TZ)" +for t in demo-hp root@192.168.0.154 felhom-pve; do + printf '== %s\n' "$t" + ssh -o BatchMode=yes -o ConnectTimeout=10 "$t" 'python3 -' < "$D/tools/host-pin.py" + ssh -o BatchMode=yes -o ConnectTimeout=10 "$t" 'pveum user token list felhom-agent@pve --output-format json; systemctl is-active tailscaled' +done; } > "$D/box/pin.txt" 2>&1 diff --git a/documentation/audits/end-dates-list-2026-11-11/tools/searches.sh b/documentation/audits/end-dates-list-2026-11-11/tools/searches.sh new file mode 100644 index 00000000..b4505e5d --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/tools/searches.sh @@ -0,0 +1,41 @@ +#!/bin/bash +# searches.sh — the searches an `N` in TABLE.md must survive. Read-only; prints to stdout (saved as evidence/searches.txt). +# The same tool as audits/endings-list-2026-11-10/tools/searches.sh with this list's searches. +# Each search: an ASCII fragment (no accented letter), case-insensitive, over (ar) documentation/architecture, +# (op) backlog/OPEN-ITEMS.md, (cl) backlog/CLOSED-ITEMS.md, (rm) backlog/ROADMAP.md. For register files it prints the row ids +# that match, so a reader can open them. Controls: C+ must hit, C- must not. +E="$(cd "$(dirname "$0")/../../../.." && pwd)" +B="$E/documentation/backlog"; AR="$E/documentation/architecture" +G=/usr/bin/grep +ids() { $G -i -E -- "$1" "$2" 2>/dev/null | $G -o -E '^\| \*\*R-[0-9a-z]+\*\*' | tr -d '|* ' | sort -u | tr '\n' ' '; } +s() { # id, what, pattern + local n_ar; n_ar=$($G -r -i -l -E -- "$3" "$AR" 2>/dev/null | wc -l) + printf '%s | %s | /%s/ | ar files=%s | open: %s| closed: %s| roadmap=%s\n' "$1" "$2" "$3" "$n_ar" "$(ids "$3" "$B/OPEN-ITEMS.md")" "$(ids "$3" "$B/CLOSED-ITEMS.md")" "$($G -c -i -E -- "$3" "$B/ROADMAP.md")" +} +date -u +'searches run %Y-%m-%d %H:%M:%S UTC' +s C+ "positive control (a row that exists)" "R-1301" +s C+2 "positive control (a word in all four)" "certificate" +s C- "negative control" "zzqxenddatezz" +s S01 "web certificate: warning before it ends" "(certificate|cert).{0,80}(expir|renew|runs out|days left)|acme\.json|NotAfter" +s S02 "Proxmox renews its certificate, the agent's pin does not follow" "pve-ssl|pveproxy.{0,60}cert|(tls|certificate|cert) pin.{0,120}(renew|expir|rotat|re-pin|repin)|fingerprint.{0,80}(renew|expir|stale)" +s S03 "operator access token ends, nobody warned" "(access|api|hub|operator).token.{0,120}(expir|90 days|days left|renew)|HUB_TOKEN.{0,80}(expir|ends|renew)|token_expir" +s S04 "Tailscale key expiry" "tailscale.{0,120}(expir|key)|key expiry" +s S05 "restore copy swept after 30 days, no warning" "restore cop.{0,100}(30 days|sweep|unused)|restoreCopyUnusedAge" +s S06 "domain renewal / registrar" "registrar|domain.{0,60}(renew|expir|lapse)|whois" +s S07 "paid accounts: billing, contract end" "(hetzner|cloudflare|resend|mail service).{0,100}(billing|invoice|unpaid|contract|account (is )?closed|payment)|credit card" +s S08 "mail service key end / check that mail still goes" "(resend|mail).{0,40}(key|token).{0,80}(expir|rotat|ends)|canary mail|test mail.{0,40}(daily|weekly)" +s S09 "OS support end (Debian 13, Proxmox 9)" "EOL|end of life|end-of-life|support (ends|end date)|trixie.{0,80}(until|ends|support)" +s S10 "apt signing keys end" "(signing|apt|archive|repo) key.{0,100}(expir|ends|rotat)|keyring.{0,80}(expir|old)|Valid-Until" +s S11 "a box off for a long time comes back" "(off|unplugged|powered off|switched off|in a drawer).{0,60}(months|a year|years)|long.{0,10}(off|absence)|cold box" +s S12 "old tunnel client / infra image stops being accepted" "cloudflared.{0,100}(old|version|support|deprecat)|tunnel client.{0,60}(old|support)" +s S13 "Gitea / registry token end" "(gitea|registry).{0,60}(token|credential|password).{0,80}(expir|ends|rotat)" +s S14 "GitHub advisory token / CC store token end" "GH_ADVISORY_TOKEN|advisory token|github.{0,40}token.{0,60}(expir|ends)" +s S15 "household's own Cloudflare token end" "(cloudflare|cf).{0,20}token.{0,100}(expir|ends|ttl|revok)" +s S16 "operator signing key rotation" "key.rotation|re-pin|rotat.{0,40}(operational|signing|cold) key" +s S17 "seal key rotation" "seal.{0,10}key.{0,100}(rotat|lost|change)" +s S18 "a list of what ends when" "end dates|expiry (list|calendar|table)|renewal calendar|what ends" +s S19 "golden age: warning before day 7" "golden.{0,80}(7 days|too old|age).{0,80}(warn|remind)|waiver.{0,60}(expir|ends)" +s S20 "served certificate of the hub address (cert-manager)" "cert-manager|letsencrypt-prod|ClusterIssuer" +s S21 "a recorded image digest is no longer served" "digest.{0,100}(still served|no longer|gone|deleted|removed from)|registry.{0,60}(drops|deletes|garbage)" +s S22 "the installer ISO grows old" "ISO.{0,80}(too old|stale|age|rebuild every|rebuilt every)" +s S23 "a box off during the 14-day off-site countdown" "(abandon|countdown).{0,120}(box is off|switched off|offline|unreachable)" diff --git a/documentation/audits/end-dates-list-2026-11-11/tools/table-head.md b/documentation/audits/end-dates-list-2026-11-11/tools/table-head.md new file mode 100644 index 00000000..c0b55c92 --- /dev/null +++ b/documentation/audits/end-dates-list-2026-11-11/tools/table-head.md @@ -0,0 +1,36 @@ +# The End-dates list — the table + +Built 2026-11-11 from the source of the four repositories, from read-only measurements on the three test boxes +(Tester 1, demo-hp, the N100; host and guest) and from what any visitor can read with no login. One block for each +thing that stops working, or stops being safe, on a date or after an age. `tools/make-table.py` writes this file from +the helpers' answers (`evidence/helpers/`) and from `tools/marks.py`; do not edit it by hand. + +**NOT MEASURED, by the operator's ruling of 2026-11-10:** DooPlex and ep0, and every account (Cloudflare, Hetzner, the +mail service, the registrars). What this table says about them comes from the repository's own scripts and documents. + +**The eight questions:** 1 what it is, who owns it, where it lives · 2 when it ends (a real date for each box where one +was measured, else the rule) · 3 who renews it, and how long before · 4 what breaks, and for whom · 5 who is warned +before the end · 6 a box that was off past the end date: does it come back by itself? · 7 what happens if the renewal +fails · 8 how do we know (`MEASURED`, a test, `SOURCE`, a document, or nobody knows). +A ninth line, outside the marks: is the hand step written in a runbook? + +**Marks:** `A` answered, with a pointer · `D` decided, not built · `K` known gap, with its row · `N` nobody asked — it +survived the searches in `EV/searches.txt` (S01…S23; controls in the same file). Every `N` names its class row: +**T1** = R-1379 an end exists and nobody is warned before it · **T2** = R-1380 another party renews and our pin does +not follow · **T3** = R-1381 no document says when it ends, or where to read the date · **T4** = R-1382 a box that was +off past the date: nobody knows if it comes back. + +**Who read what.** A cell whose mark is `D`, `K` or `N`, and every `A` that `tools/marks.py` carries, holds the main +session's words: the pointer was opened and read. Every other cell holds a helper's words with the mark `A`: +`tools/resolve-helper.py` opened each of its pointers (the file and the line exist, and the name the helper gave stands +at that line). The main session read every helper answer to questions 4 to 7; it did not open each of their pointers. +Treat those cells as a map, not as proof. + +**Pointers:** `AG/` = `felhom-agent/` · `CT/` = `felhom-controller/controller/` · `CAT/` = `app-catalog-felhom.eu/` · +`HUB/` = `felhom.eu/hub/internal/` · `E/` = `felhom.eu/` · `AR/` = `felhom.eu/documentation/architecture/` · +`RB/` = `felhom.eu/documentation/runbooks/` · `EV/` = this folder's `evidence/`. `tools/check-pointers.py` prints the +text behind each one. + +**Named status** (does an architecture document name this end date?): `NAMED` and `MENTIONED` with a pointer were read +in the documents. Where the line says "a machine count", the status comes from a search of the architecture folder: +0 lines = `UNNAMED`, else `MENTIONED`. diff --git a/documentation/audits/no-token-in-address-2026-11-11/fix-clone.sh b/documentation/audits/no-token-in-address-2026-11-11/fix-clone.sh new file mode 100644 index 00000000..9a4a2c6d --- /dev/null +++ b/documentation/audits/no-token-in-address-2026-11-11/fix-clone.sh @@ -0,0 +1,12 @@ +#!/bin/bash +# fix-clone.sh CLONE LOGIN — take the credential out of the address; point git at the store helper. Prints no address. +set -u +r=$1; login=$2 +u=$(git -C "$r" config --get remote.origin.url | sed -E 's#^([a-z]+://)[^/@]*@#\1#') +case "$u" in *@*) printf 'still holds an @, stop\n'; exit 1;; esac +git -C "$r" config --local remote.origin.url "$u" || exit 1 +git -C "$r" config --local --unset-all credential.helper 2>/dev/null +git -C "$r" config --local credential.helper "" || exit 1 +git -C "$r" config --local --add credential.helper /mnt/5_hdd/felhom.eu/git/felhom.eu/scripts/git-credential-sec.sh || exit 1 +git -C "$r" config --local credential.username "$login" || exit 1 +printf '%s: fixed. address now: %s\n' "$r" "$u" diff --git a/documentation/audits/no-token-in-address-2026-11-11/measure-after.txt b/documentation/audits/no-token-in-address-2026-11-11/measure-after.txt new file mode 100644 index 00000000..b8a5ff00 --- /dev/null +++ b/documentation/audits/no-token-in-address-2026-11-11/measure-after.txt @@ -0,0 +1,30 @@ +taken 2026-11-11T10:46:07Z +=== clones: remote address holds a credential? === +build/felhom-agent url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +clones-plugmeter-2026-10-13/app-catalog-felhom.eu url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +clones-plugmeter-2026-10-13/felhom-agent url=yes extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +clones-plugmeter-2026-10-13/felhom-controller url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +clones-plugmeter-2026-10-13/felhom.eu url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +drill/app-catalog-drill url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +git/app-catalog-felhom.eu url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +git/app-catalog-felhom.eu/output/.git-repo url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +git/cc-secrets url=yes extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +git/felhom-agent url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +git/felhom-controller url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +git/felhom.eu url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +git/felhom.eu-media url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +git/felhom.eu-sizing-push url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +git/felhom.eu-wt-sizing url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +wt/secreview-clone url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu + +=== global and system git settings === +--global: url-with-credential=no extraHeader-lines=0 insteadOf-with-credential=no +--global: credential helper = store +--system: url-with-credential=no extraHeader-lines=0 insteadOf-with-credential=no +--system: credential helper = +clear-text credential file /home/kisfenyo/.git-credentials: EXISTS, 3 line(s), mode 600, hosts: gitea.dooplex.hu github.com +clear-text credential file /home/kisfenyo/.config/git/credentials: none +clear-text credential file /home/kisfenyo/.netrc: none +env: GIT_ASKPASS= SSH_ASKPASS= GIT_CONFIG_COUNT=unset + +=== scripts and skills that write a credential into an address (file:line, the line is NOT printed) === diff --git a/documentation/audits/no-token-in-address-2026-11-11/measure-before.txt b/documentation/audits/no-token-in-address-2026-11-11/measure-before.txt new file mode 100644 index 00000000..2a0a2ad7 --- /dev/null +++ b/documentation/audits/no-token-in-address-2026-11-11/measure-before.txt @@ -0,0 +1,30 @@ +taken 2026-11-11T10:10:20Z +=== clones: remote address holds a credential? === +build/felhom-agent url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +clones-plugmeter-2026-10-13/app-catalog-felhom.eu url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +clones-plugmeter-2026-10-13/felhom-agent url=yes extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +clones-plugmeter-2026-10-13/felhom-controller url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +clones-plugmeter-2026-10-13/felhom.eu url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +drill/app-catalog-drill url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +git/app-catalog-felhom.eu url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +git/app-catalog-felhom.eu/output/.git-repo url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +git/cc-secrets url=yes extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +git/felhom-agent url=yes extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +git/felhom-controller url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +git/felhom.eu url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +git/felhom.eu-media url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +git/felhom.eu-sizing-push url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +git/felhom.eu-wt-sizing url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu +wt/secreview-clone url=no extraHeader=0 insteadOf=no gitmodules=no host=gitea.dooplex.hu + +=== global and system git settings === +--global: url-with-credential=no extraHeader-lines=0 insteadOf-with-credential=no +--global: credential helper = store +--system: url-with-credential=no extraHeader-lines=0 insteadOf-with-credential=no +--system: credential helper = +clear-text credential file /home/kisfenyo/.git-credentials: EXISTS, 3 line(s), mode 600, hosts: gitea.dooplex.hu github.com +clear-text credential file /home/kisfenyo/.config/git/credentials: none +clear-text credential file /home/kisfenyo/.netrc: none +env: GIT_ASKPASS= SSH_ASKPASS= GIT_CONFIG_COUNT=unset + +=== scripts and skills that write a credential into an address (file:line, the line is NOT printed) === diff --git a/documentation/audits/no-token-in-address-2026-11-11/measure.sh b/documentation/audits/no-token-in-address-2026-11-11/measure.sh new file mode 100644 index 00000000..dd748cc9 --- /dev/null +++ b/documentation/audits/no-token-in-address-2026-11-11/measure.sh @@ -0,0 +1,31 @@ +#!/bin/bash +# measure.sh — does any git address or setting hold a credential? Prints names and yes/no only, never an address. +set -u +has() { /usr/bin/grep -q -E '://[^/@[:space:]]+@' ; } +yn() { if has; then printf yes; else printf no; fi; } +printf 'taken %s\n=== clones: remote address holds a credential? ===\n' "$(date -u +%FT%TZ)" +for g in $(find /mnt/5_hdd/felhom.eu /home/kisfenyo/build /home/kisfenyo/maintenance /home/kisfenyo/.cache/felhom* /tmp/felhom* -maxdepth 5 -name .git 2>/dev/null | sort); do + r=${g%/.git} + [ -e "$g" ] || continue + u=$(git -C "$r" config --get-regexp '^remote\..*\.(url|pushurl)$' 2>/dev/null | yn) + x=$(git -C "$r" config --local --get-regexp 'extraheader' 2>/dev/null | /usr/bin/grep -c . ) + i=$(git -C "$r" config --local --get-regexp 'insteadof' 2>/dev/null | yn) + sm=no; [ -f "$r/.gitmodules" ] && sm=$(yn < "$r/.gitmodules") + host=$(git -C "$r" config --get remote.origin.url 2>/dev/null | sed -E 's#^[a-z]+://([^/@]*@)?([^/:]+).*#\2#; s#^[^@]*@([^:]+):.*#\1#' | cut -c1-40) + printf '%-62s url=%-3s extraHeader=%s insteadOf=%s gitmodules=%s host=%s\n' "${r#/mnt/5_hdd/felhom.eu/}" "$u" "$x" "$i" "$sm" "$host" +done +printf '\n=== global and system git settings ===\n' +for sc in --global --system; do + printf '%s: url-with-credential=%s extraHeader-lines=%s insteadOf-with-credential=%s\n' "$sc" \ + "$(git config $sc --get-regexp 'url' 2>/dev/null | yn)" \ + "$(git config $sc --get-regexp -i 'extraheader' 2>/dev/null | /usr/bin/grep -c .)" \ + "$(git config $sc --get-regexp -i 'insteadof' 2>/dev/null | yn)" + printf '%s: credential helper = %s\n' "$sc" "$(git config $sc --get-all credential.helper 2>/dev/null | tr '\n' ' ')" + git config $sc --name-only --get-regexp -i 'credential|askpass' 2>/dev/null | sed "s/^/$sc key: /" +done +for f in /home/kisfenyo/.git-credentials /home/kisfenyo/.config/git/credentials /home/kisfenyo/.netrc; do + if [ -f "$f" ]; then printf 'clear-text credential file %s: EXISTS, %s line(s), mode %s, hosts: %s\n' "$f" "$(wc -l < "$f")" "$(stat -c %a "$f")" "$(sed -E 's#^[a-z]+://[^@]*@##; s#^machine +([^ ]+).*#\1#; s#/.*##' "$f" | sort -u | tr '\n' ' ')"; else printf 'clear-text credential file %s: none\n' "$f"; fi +done +printf 'env: GIT_ASKPASS=%s SSH_ASKPASS=%s GIT_CONFIG_COUNT=%s\n' "${GIT_ASKPASS:+set}" "${SSH_ASKPASS:+set}" "${GIT_CONFIG_COUNT:-unset}" +printf '\n=== scripts and skills that write a credential into an address (file:line, the line is NOT printed) ===\n' +/usr/bin/grep -rn -I -E 'https?://[^/"[:space:]]*(\$\{?[A-Za-z_]+\}?|%s)[^/"[:space:]]*@' /mnt/5_hdd/felhom.eu/git/*/scripts /mnt/5_hdd/felhom.eu/git/felhom.eu/skills /mnt/5_hdd/felhom.eu/git/*/.gitea /mnt/5_hdd/felhom.eu/git/felhom-controller/controller/build.sh 2>/dev/null | /usr/bin/grep -i -E 'git|clone|gitea' | cut -d: -f1,2 | head -40 diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index d57c7afa..4f52ccef 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -168,7 +168,7 @@ stopping line that lies. | **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC | | **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. **2026-10-27 (security triage):** two notices with NO fix released (GHSA-p8cf-g8rv-gvrq: a user overwrites another user's Withings link; GHSA-cmmf-53wf-r5gv: owner-only photos readable by URL) — both need a household account or a leaked URL; pick in `audits/security-app-updates-2026-10-27/PICKS.md` (CC's pick: leave it offered, take the fix with this big step). **2026-10-28 (R-1332): HTTP2Shell has NO fixed release yet** — v0.17.3's server carries Node.js 24.18.0 (`tsx index.ts`, behind the setup gate) and the newest v1.9.0 still Node.js 24.22.0 (fixed: 24.22.1); the web image is nginx, no Node. The big step waits for a v1.x with Node ≥ 24.22.1, or takes the runtime fix later as a lane step. | **OPEN** — **RULED D56 = keep it offered** (operator 2026-10-27 21:33, `09` §3 decision 272) — the fix rides its big step. **Earlier:** **OPEN — rank P3-LOW; owner: CC (after R-784)** **2026-11-02:** v1.10.0 and v1.11.0 read clean of HTTP2Shell (Node 24.22.1); v1.9.0 is still bad. **The big step v0.17.3 → v1.11.0 is proven on the drill:** bench, 9202 (peak 29.4 %), the undo (`audits/security-2026-11-02/bench/sparkyfitness/`, `walks/sparkyfitness-drill/`). Not written: R-784 (the licence) stands before it. | — | — | CC | -## Backup & restore — 48 rows (P2 4, P3 26, P4 18) +## Backup & restore — 49 rows (P2 4, P3 27, P4 18) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -177,7 +177,7 @@ stopping line that lies. | **R-923** | Backup & restore | P2 | **The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies.** Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: `vaultwarden-system/vaultwarden` (1.37.3), PVC `vaultwarden-data` 7.4 MB (`db.sqlite3` + WAL, `rsa_key.pem`, no attachments); its only backups were Longhorn on DooPlex. **A key in a password manager that runs on DooPlex is not off DooPlex** — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its `ADMIN_TOKEN` is plain text (an Argon2 hash is its guidance). | NARROWED — **2026-10-23 (D32):** the sheet is current — names and places only, one A4, HU + EN (`runbooks/break-glass-sheet.md`); every place checked to exist. LEFT: the operator prints it, fills it by hand, keeps it outside the home. **NARROWED 2026-10-09 (operator yes in chat):** Vaultwarden's own `backup` copy + `rsa_key.pem` ride the nightly encrypted DooPlex→ep0 job (`scripts/dooplex-offsite/`), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: `runbooks/break-glass-sheet.md`, `runbooks/total-loss-of-dooplex.md`. **LEFT:** the operator prints the sheet and stores it away from home; the `ADMIN_TOKEN` hash (a DooPlex change, operator's word). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G1/G3/G10):** unchanged; the sheet is still the one act that breaks the circle, and S5's offline form is unknown to CC — on the sheet too (decide list in STATUS). | operator: print the sheet | Operator: print + store the sheet; then close | operator | | **R-1254** | Backup & restore | P2 | **The hub saves a household's new escrow package in TWO writes; a failure between them leaves the new repository hash beside the OLD identity package, and the box then marks itself escrowed — on a total loss the written code opens nothing.** Found 2026-10-24 (HUNT-3 H1), read at `file:line`: `hub/internal/api/handler.go:1415` `SaveHostEscrow` commits blob + hash, `:1445` `SaveHostDRBundle` sets `identity_blob` in a second statement (`internal/store/store.go:3788-3796`; the upsert `:3490-3511` never touches it); the controller's confirm reads the hash (`felhom-controller/controller/internal/report/escrow_confirm.go:195-217`); the agent sends once. Same family (H2): a lost or late answer (30 s) tells the household „the previous code still works” when the hub may already hold the new package (`felhom-agent/internal/localapi/escrow_ceremony.go:79,532-534,639-641`). Static. **The hub half is fixed on main 2026-10-24 night** (felhom.eu `dcf4f84c`; Part 3): key, hash and identity blob in ONE transaction (`SaveHostEscrowWithIdentity`), a bad identity refused before any write; hub image `0.170.0-rc.r1254` built locally (`67db584a45d7`), NOT pushed, NOT deployed. The agent ceremony half (H2: „the previous code still works” after a lost answer) is still open. | **NARROWED** — **released 2026-10-29:** agent 0.168.0 (vouched) + controller 0.347.0 (the floor); a ceremony ran on both (the happy path); the lost-answer branch has tests only (a live proof needs a hub answer lost on purpose). **the hub half CLOSED 2026-10-25:** hub **0.170.0** deployed on the operator's yes (manifest bump, ArgoCD sync of the hub Deployment only, image digest `6478498d…`; 30 minutes and more with no alarm mail, the mailbox as the control); Tester 1 ran the escrow ceremony end to end through the page's endpoints and the hub stored the new package in one transaction („the new package is on the hub", `audits/release-2026-10-26/escrow-ceremony-tester1.txt`). The ceremony half stays open. **2026-10-28 night: the ceremony half FIXED ON MAIN** (agent `5ddd124` CI 2473, controller `e33cffa` CI 2474; tests red first; `03` §8 [DESIGN] 2026-10-28): after a failed upload the agent asks the hub which package it holds (the identity blob compared) — the new one → success, another → 502 and „the previous code still works", no answer → 504 that promises nothing; the controller words 504 and its own failed call as „Nem tudjuk biztosan… Őrizd meg mindkét kódot…" and waits 60 s for the claim/confirm (was 15 s, shorter than the agent's 30 s upload). | the hub release (attended); the agent half | deploy hub v0.170.0 (Monday list); then the ceremony asks the hub what it holds before it says „the previous code still works” | CC | | **R-314** | Backup & restore | P3 | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-314-279-177.md` — the box's own 14-day countdown has no operator door (the hub's 7-day phase does, decision 74); one door for R-314 + R-279: a fixed list of operator actions carried in the report reply. Question D1 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D1 (`09` §3 decision 185):** yes — option A, a fixed list of operator actions carried in the box's report reply. | **VERIFY — 2026-10-09: DELIVERED (hub 0.144.0 + controller 0.304.0); `run_job fill-watch` proven live on the Tester 1 box (hub row „done — ran to its end"; control: the box log „checked 3 filesystem(s)"; an unknown action is refused 400). `abandon_stop` / `abandon_extend` NOT yet proven — no box has a deletion countdown; prove them on the first one.** **VERIFY — built on main 2026-10-08 (D1, decision 185), ships with tomorrow's controller + hub releases; closes after the live proof on 9202** (`abandon_stop` / `abandon_extend` through the hub's buttons). **READY (S) — NEW 2026-08-12, RANK 3** **2026-10-05 (burn-down night): NEEDS A DESIGN.** There is no operator door into a RUNNING controller (its HTTP surface is the household's session; the CLI runs in a separate process and cannot safely trigger a job in the running one). Same design as R-177 and R-279. | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC | -| **R-401** | Backup & restore | P3 | **Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store.** Controller v0.228.0 (R-399) made `--read-data-subset=100%` the default for every box. The whole justification is a single data point: `demo-hp`, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% **39.2 s** — four seconds. Re-proven live 2026-08-31 at 38.7 s. **It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA.** A 50 GB store is ~370x the data and this curve says nothing about it. **Nothing was invented from that one point** — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. **`readDataSubsetRe` already accepts `n/m`**, so a rotating schedule (`1/7` on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. **THE TRIGGER IS AN EVENT, NOT A DATE:** the slow-check WARN from v0.228.0 firing on any box (`integritySlowNoticeThreshold`, 5 min) — that line names the duration, the depth and this row. **WHAT HAPPENS IF NOBODY ACTS:** every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. **Whoever acts must also revisit `integrityCheckTimeout` (30 min)**, which is now the number a large store meets first. | **OPEN — WATCHING** | — | When the WARN fires: measure the curve on that store, then choose between a rotation (`n/m`), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. | CC | +| **R-401** | Backup & restore | P3 | **Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store.** Controller v0.228.0 (R-399) made `--read-data-subset=100%` the default for every box. The whole justification is a single data point: `demo-hp`, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% **39.2 s** — four seconds. Re-proven live 2026-08-31 at 38.7 s. **It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA.** A 50 GB store is ~370x the data and this curve says nothing about it. **Nothing was invented from that one point** — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. **`readDataSubsetRe` already accepts `n/m`**, so a rotating schedule (`1/7` on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. **THE TRIGGER IS AN EVENT, NOT A DATE:** the slow-check WARN from v0.228.0 firing on any box (`integritySlowNoticeThreshold`, 5 min) — that line names the duration, the depth and this row. **WHAT HAPPENS IF NOBODY ACTS:** every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. **Whoever acts must also revisit `integrityCheckTimeout` (30 min)**, which is now the number a large store meets first. **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CC08-3). | **OPEN — WATCHING** | — | When the WARN fires: measure the curve on that store, then choose between a rotation (`n/m`), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. | CC | | **R-409** | Backup & restore | P3 | **Nothing in the product can vouch for the bytes of a restored recovery unit — the only hash record covers 0.002 % of it.** MEASURED on demo-hp 2026-08-31 against kimai's restored unit: `manifest.json`'s `checksums` object carries sha256 for `.felhom.yml` (2 235 B), `app.yaml` (488 B) and `docker-compose.yml` (2 195 B) — **4 918 bytes of a 213 231 242-byte unit**. The database dump (48 217 B) and the two named-volume tars (160 331 776 B + 52 845 056 B) — the recoverable data, 99.998 % of the bytes — have no recorded hash anywhere. **And nothing else supplies one:** restic 0.14.0's `restore --verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving one-byte corruption of a 160 MB tar passed clean, red-proofed), and `restic ls --json` file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps and **no content hash**. **So "the restore produced correct files" is currently unanswerable by any automated means.** **What is NOT claimed here:** `restic check --read-data-subset=100%` already proves the STORE's packs, and the config files that ARE hashed are the ones a wrong-content failure would be hardest to spot in. | **OPEN — MEDIUM** | R-87, R-361 | Cheapest fix, and it is already half-built: extend the capture's `checksums` to cover `db_dumps` and `volume_dumps` — R-361 already computes a canonical dump sha256 to prove itself, so the value exists at capture time. Then a restore-test has a real reference and R-87's narrow version becomes a content check rather than a completeness one. Evidence: `audits/SPIKE-restic-restore-test-2026-08-31.md` §Q2, §Q3. | CC | | **R-433** | Backup & restore | P3 | **A sub-account cannot reach ANY Storage Box snapshot, by any name — so clause (b) of the 2026-09-01 R-95 re-scope ("the rest is recoverable file by file") is NOT SUPPORTED.** MEASURED 2026-09-01 on `demo-hp` over the credential the box already holds, read-only, no delete verb issued. **The sweep:** a batched `stat -c %n` over Hetzner's port-23 restricted shell (500–600 paths per round trip, stdout carrying only paths that exist) tried **777,600** names of the vendor form `YYYY-MM-DDTHH-MM-SS` across nine full days at second granularity, and 126 alternative shapes — **zero resolved.** **The control is what makes the zero mean anything:** the identical 600-name batch with one real path appended returned it in 6 of 6 batches. **The structural cause:** `/home` (the customer data, `u629488-sub3`) is **st_dev 0,82**; `/.zfs/snapshot` is **st_dev 0,276**; `/home/.zfs` does not exist. A snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding `felhom-repo`. **What still stands:** clause (a) — the box can delete its live repository but cannot WRITE into `/.zfs/snapshot` — is unchanged and re-confirmed. **What is now open again:** the only routes to the older copy are a panel rollback of the WHOLE Storage Box (deletes newer snapshots, hits every customer on it) and the provider API (fenced by §11-D, and `hub/internal/hetznerapi/hetznerapi.go` has **no snapshot method at all**, so it needs new code regardless). **NOT ESTABLISHED, and it is the question that decides whether per-file recovery exists for anyone:** whether the MAIN account can see the snapshots. No main-account credential exists in this project. **The ranking is Viktor's; I am not re-ranking R-95 — but the argument that moved it down is the argument this drill removed.** `audits/evidence-drill-r95-recovery-2026-09-01/` **BLOCKED-ON-PROVIDER 2026-09-01.** The one question that can move this is drafted and ready to send: **Question 1** of `documentation/runbooks/provider-questions-2026-09-01.md` — *can the MAIN account retrieve individual files from a snapshot, without a whole-box restore?* **Neither answer leaves this row where it is:** "yes" makes per-file recovery real but permanently operator-only, and closes this; "no, full restore only" means the snapshots do not bound a single customer's exposure at all, because using them costs every other customer on the box their newer snapshots — **and R-95 becomes urgent.** Nothing here can progress without it, and it is not CC's to send (§11-D). Tracked by a dated check in the DUE-CHECKS block. **⚠ A TRAP ON THE WAY TO THIS ANSWER, found by the operator 2026-09-01 and armed against in the questions file: HETZNER HAS TWO SIMILARLY-NAMED PRODUCTS WHOSE DOCS SAY OPPOSITE THINGS.** **Storage SHARE** (a managed Nextcloud, NOT us) documents *"Currently, we only support restores for the full backup ZFS snapshot to a specific point in time"* (`docs.hetzner.com/storage/storage-share/faq/backup-snapshot/`). **Storage BOX** (ours) documents the opposite — *"You can download individual files or entire directories as usual"* (`docs.hetzner.com/storage/storage-box/snapshots/`). **A web search for the obvious phrasing surfaces the SHARE page, and it reads like a definitive NO.** Tell them apart by the giveaways: the Share page talks about *Nextcloud's data cache*, a *database dump* and the *konsoleH* interface, and never mentions Storage Box. **Why this is filed and not left as trivia: a support agent could answer Question 1 from the wrong page, and that answer would push this row to the top of the register for no reason.** Question 1 now names the product, quotes the Storage Box line, and says explicitly that we know what the Share FAQ says. **If an answer cites full-snapshot-only, check WHICH PRODUCT it is about before acting on it.** **Nothing in this repository ever leaned on the Share claim** — verified by grep at the time; the only vendor line we cite is the Storage Box one. **RE-DATED 2026-09-15 → 2026-09-22 (due-checks gate):** the operator mailbox read through the Gmail connector (`(from:hetzner OR subject:hetzner OR "Storage Box") after:2026/09/01`) holds no Hetzner reply — one match, our own `offsite_snapshots_dropped` alarm. Whether the two tickets were ever sent is not visible to CC; if they were not, sending them is the operator's act. **-- ANSWERED. Hetzner replied on ticket #2026090103040671; the operator supplied the thread on 2026-09-22 after this session searched for it and WRONGLY reported no answer (see the instrument note below and R-628).** **Q1 - can the MAIN account read individual files and directories out of a specific snapshot, without restoring the whole box?** Hetzner: *"With the main account it should be possible to download files and directories from the snapshot as it is stated in the documentation"*, citing `docs.hetzner.com/storage/storage-box/snapshots#access-to-snapshots`; and separately *"A restore of a snapshot will revert the Storage Box completely to the state it was in when the snapshot was taken."* **So the Storage BOX documentation governs, not the Storage Share FAQ** - which is exactly the trap `provider-questions-2026-09-01.md` warned the reader about, and the answer came back on the right side of it. **File-level snapshot access is a MAIN-ACCOUNT capability; the sub-account cannot do it**, which matches R-433's own measured finding (777,600 names swept from a sub-account, zero hits). **HEDGE, kept because it is in the reply: "should be possible" is not "is", and nobody has yet read a file out of a snapshot from the main account on `u629488`. That is now the measurement this row needs, and it is cheap.** **Q2 - is `--append-only` enforced by Hetzner, or taken from what the client sends?** Hetzner: *"You can use --append-only and it would look like this: `command="rclone serve restic --stdio --append-only path/to/repo" `"*, with `fluix.one/blog/hetzner-restic-append-only/`. **So it is enforced by US, by pinning a FORCED COMMAND to the SSH key in the Storage Box's `authorized_keys`** - not by Hetzner globally, and not by anything the client asks for. A key without that prefix still gets a deleting server. **This is the answer R-95 has been blocked on** and it says append-only IS expressible on a Storage Box, which R-95 records as impossible over SFTP - see R-95 and R-436. **NOT YET MEASURED, and that distinction is the whole of what this row is worth: this is a support answer, not a proof.** What is owed is a real forced-command key on `u629488`, a restic `forget --prune` through it that is REFUSED, and a backup through it that still succeeds. | **BLOCKED** — **BLOCKED-ON-PROVIDER — Question 1 of `runbooks/provider-questions-2026-09-01.md`** | — | — | CC + operator | | **R-540** | Backup & restore | P3 | **[P3-LOW] The hub knows exactly ONE off-site pool box, so there is no rule for what happens when it fills.** Read from source 2026-09-16 while making off-site the default: `HETZNER_POOL_BOX_ID` is a single value, and every shared customer becomes a sub-account on that box. With off-site now ON for every new customer (hub v0.116.0) the box fills faster, and the fill warning (80%/90% of the box, `monitor/offsite.go`) tells the operator it is filling but nothing says which box a new customer should land on. **Needs a selection rule** (least-full, or explicit per-customer), not a bigger box. No customer is at risk today: the pool box read 0.3% full (2.7 GB of 1 TB), Σ shared quota 150 GB, oversub 0.15x. | **READY — rank P3-LOW; owner: CC (hub)** **2026-10-05 (burn-down night): NEEDS A DESIGN (and money).** A selection rule needs a multi-box configuration and a second box. Next: an operator pick of the rule and its trigger (e.g. add box 2 at 70 %). **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place O1, listed from documents only (the off-site box was not read in this run — second pass). | — | — | CC | @@ -204,22 +204,23 @@ stopping line that lies. | **R-1173** | Backup & restore | P3 | **R-921's consequence (no app stop for an off-site tier the agent will refuse) has not been seen live.** Agent 0.161.0 serves `host_busy` (proven live on demo-hp 2026-10-20 during a press: busy `backup:local`), and controller ≥ 0.335.0 reads it — but the case only arises on the night schedule (a manual press backs up the local tier only), so the no-stop branch is proven by its red-proved test alone. | OPEN — filed 2026-10-20 **2026-10-23 dated check — DONE (the other outcome):** demo-hp's off-site (PBS) tier ran 2026-10-22 20:42:53–20:49:48 UTC with nothing in flight (quiesce → started → snapshotted → early resume → done); no `host is busy` case arose, so the R-921 no-stop branch is still proven by its test alone. Tonight only the local tier was due (refused by space, R-548). Kept open; no new date — it reads itself the night a busy host meets a due off-site tier. | `audits/night2-2026-10-20/P1/1b-host-busy-poll.txt` | Read demo-hp's controller log after the night its off-site (pbs) tier is due (~2026-10-22): a `host is busy … no app is stopped (R-921)` line, or the off-site tier starting with nothing in flight | CC | | **R-1184** | Backup & restore | P3 | **A cut-off shared-folder restore can be „placed" and reported restored while files are missing; the download is then deleted.** Readiness is only „the scratch folder is not empty" (`internal/backup/shares_restore.go:131-140`) — the R-358 class fixed for apps, not for shares; 30-min download window (`web/offbox_handlers.go:716`); fixed 2 GiB free-space gate that fails open (`:107`). CONFIRMED in code. Data-loss hunt 2026-10-20 (C1 reviewer 1, R3). | OPEN | — | Fix: a completion marker after restic succeeds, as for apps | CC | | **R-1185** | Backup & restore | P3 | **A database dump that fails its own content check has already replaced last night's good dump, and the second drive copies it an hour later.** `appbackup/dbdump.go:388-422`: size check, rename over the good file (`:399`), then `ValidateDump` (`:419`). Volume tars are promoted with no content check either (`backup.go:902`). Tier 1 is one point per app by design (07 §6.1); the ORDER is the defect. PROVEN against production code (`audits/dataloss-hunt-2026-10-20/C3-D5-dump-replaced/`, a docker stub prints a header-only dump): `ValidateDump` says invalid („dump file too small"), `DumpOneTo` returns no error, and last night's good dump (with its `CREATE TABLE`) is gone from disk. Data-loss hunt 2026-10-20 (D5). | OPEN | — | Fix: validate the temp file before the rename; keep the last good dump on a failed check | CC | -| **R-1186** | Backup & restore | P3 | **The nightly off-site copy is skipped silently when another backup holds the flag, and the catch-up never re-runs it.** `offbox.go:981-996` returns nil with an INFO line on the nightly path; `withLeg` marks the leg ended anyway (`cmd/controller/main.go:1246`). The 05:30 proof and 06:00 integrity check take the same flag, so a backup time near them collides every night. Only the hub's 48 h `offsite_stale` notices. CONFIRMED in code. Data-loss hunt 2026-10-20 (T5). | OPEN | — | Fix: mark the leg NOT ended on a busy skip so the catch-up retries; say it on the backups page | CC | +| **R-1186** | Backup & restore | P3 | **The nightly off-site copy is skipped silently when another backup holds the flag, and the catch-up never re-runs it.** `offbox.go:981-996` returns nil with an INFO line on the nightly path; `withLeg` marks the leg ended anyway (`cmd/controller/main.go:1246`). The 05:30 proof and 06:00 integrity check take the same flag, so a backup time near them collides every night. Only the hub's 48 h `offsite_stale` notices. CONFIRMED in code. Data-loss hunt 2026-10-20 (T5). **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CC03-5, CC09). | OPEN | — | Fix: mark the leg NOT ended on a busy skip so the catch-up retries; say it on the backups page | CC | | **R-1188** | Backup & restore | P4 | **Restore-path candidates from the data-loss hunt — what is left is unlikely or a sentence:** (b) a failed `volume rm` is ignored before the untar, which then lands in the old volume (`backup/restore.go:218`; `StopStack` is `compose down`, so the rm rarely fails); (d) the full off-site download stops at 30 min per press and the household is not told to press again (`web/offbox_handlers.go:455`; the scratch is kept, so a second press probably continues — not measured); (e) place and full restore re-read the LATEST snapshot, not the downloaded one (`offbox_restore.go:675-679`, `_ = id`). Closed parts: (a) with R-1195 (controller 0.338.0), (c) in controller 0.337.0 (`TestR1188c_ALeftoverTempDoesNotBlockTheRetry`). Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`, `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN — **2026-10-22 night: re-ranked P3→P4**: (a) and (c) are shipped; the rest is low-chance or a sentence. | — | (d): measure a second press on a large download on 9202, then write the sentence; (b), (e): unit tests when the file is next touched | CC | -| **R-1189** | Backup & restore | P3 | **Time and concurrency candidates from the data-loss hunt — what is left:** (a) a host clock ahead during one whole-guest backup silences that tier until real time passes it: the agent turns a future archive time into a negative age (`localapi/server.go:1179-1192`); half-live 2026-10-21 (B6 #8, `s8-wrong-clock/README.md`: a local archive dated 2028-10-21); agent code, so it rides an agent release and the operator's vouch; (c) the whole-guest quiesce waits for the catch-up and the update leg, not for a restore (`quiesce.go:213-275, 565`); (f) the OS leg and kernel reboot follow a daytime valve backup (agent `server.go:982-986`, `kernel.go:101-114`) — a design question; (g) spring forward 2027-03-28 moves 02:30 onto the 03:30 second copy; (h) self-update busy check before the pull only (`updater.go:532-545, 764`; not re-read); (i) a double-clicked restore shows failed while it runs (check and begin not atomic, `web/handlers.go:1860, 1881`). **Moved out 2026-10-22 night:** (b) the skipped nights after an off-cycle run were FIXED by R-899 (controller 0.304.0; demo-hp's journal shows a night run after each of three off-cycle runs since), the creep → R-1221; (d) **FIXED in controller main `d5f88a2`, unreleased (rides 0.340.0):** the catch-up now runs the off-site proof and the integrity check once after its legs, the false comment corrected; red-proved (`nightchain/r1189d_test.go`, `cmd/controller/r1189d_wiring_test.go`); (e) → R-1222 (hub). Evidence `audits/dataloss-hunt-2026-10-20/C1-time-concurrency-candidates.md`, `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN — **2026-10-23: (d) released in controller 0.340.0;** (a), (c), (f), (g) stay as written. OPEN — 2026-10-22 night: (b) half closed, (d) fixed in code, (e) split out. | — | (a) with the next agent release (a unit test with a future archive); (c) a 9202 recipe; (g) a unit test with the 2027 clock; (f) to the operator at the next design review | CC | +| **R-1189** | Backup & restore | P3 | **Time and concurrency candidates from the data-loss hunt — what is left:** (a) a host clock ahead during one whole-guest backup silences that tier until real time passes it: the agent turns a future archive time into a negative age (`localapi/server.go:1179-1192`); half-live 2026-10-21 (B6 #8, `s8-wrong-clock/README.md`: a local archive dated 2028-10-21); agent code, so it rides an agent release and the operator's vouch; (c) the whole-guest quiesce waits for the catch-up and the update leg, not for a restore (`quiesce.go:213-275, 565`); (f) the OS leg and kernel reboot follow a daytime valve backup (agent `server.go:982-986`, `kernel.go:101-114`) — a design question; (g) spring forward 2027-03-28 moves 02:30 onto the 03:30 second copy; (h) self-update busy check before the pull only (`updater.go:532-545, 764`; not re-read); (i) a double-clicked restore shows failed while it runs (check and begin not atomic, `web/handlers.go:1860, 1881`). **Moved out 2026-10-22 night:** (b) the skipped nights after an off-cycle run were FIXED by R-899 (controller 0.304.0; demo-hp's journal shows a night run after each of three off-cycle runs since), the creep → R-1221; (d) **FIXED in controller main `d5f88a2`, unreleased (rides 0.340.0):** the catch-up now runs the off-site proof and the integrity check once after its legs, the false comment corrected; red-proved (`nightchain/r1189d_test.go`, `cmd/controller/r1189d_wiring_test.go`); (e) → R-1222 (hub). Evidence `audits/dataloss-hunt-2026-10-20/C1-time-concurrency-candidates.md`, `audits/night-2026-10-22/P3-candidates/analysis.md`. **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CA01-6, CA01-8). | OPEN — **2026-10-23: (d) released in controller 0.340.0;** (a), (c), (f), (g) stay as written. OPEN — 2026-10-22 night: (b) half closed, (d) fixed in code, (e) split out. | — | (a) with the next agent release (a unit test with a future archive); (c) a 9202 recipe; (g) a unit test with the 2027 clock; (f) to the operator at the next design review | CC | | **R-1190** | Backup & restore | P3 | **Key and retention candidates from the data-loss hunt — what is left:** (c) PLAUSIBLE: after a reuse-path reinstall the new PBS key K is never escrowed and nothing compares fingerprints (`felhom-pbs-apply:99`, `escrow_confirm.go:182-205`); (d) whole-guest off-site copies run before any escrow exists — a design question against `07` §6.1's Tier-3 gate (the operator's); (e) the escrow PUT is two statements; (f) the operator's off-site re-issue resets the provider password before saving it (hub `offsite.go:217-245`); (h) a unit restore can point an app back at an old drive and F5 then prunes the current drive's unit (`backup.go:1517-1574`) — (e), (f), (h) not re-read. **Moved out 2026-10-22 night:** (a) → R-1218, (b) → R-1219, (g) → R-1220. Evidence `audits/dataloss-hunt-2026-10-20/C1-keys-candidates.md`, `C1-delete-retention-candidates.md`, `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN | — | (c) a read-only fingerprint compare on a reinstalled test box; (e), (f), (h) re-read, then unit tests; (d) to the operator at the next design review | CC | | **R-1198** | Backup & restore | P3 | **A backup disk the family adds later is never offered as the place for the whole-box backup; the page keeps offering the disk that holds the apps' own data.** Rehearsal 2026-10-20 B6 #13 (`audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/`): a 50 GB disk plugged in and initialised as „Mentolemez" → the backups page still offers only „Családi adatlemez — Kijelölöm"; `internal/web/backup_target_offer.go:101` `firstOfferableDrive` offers the FIRST eligible drive, never a choice, and the drive's card has no „use for backups". | OPEN | — | Offer every eligible drive (largest empty first) and let the household pick | CC | | **R-1200** | Backup & restore | P3 | **A box installed in the afternoon gets no whole-box backup on its first night: the install's own backup starts the 24 h clock, so the family's whole first day reaches a whole-box copy only on the SECOND night (~38 h after install).** Rehearsal 2026-10-20 B5 (`audits/rehearsal-2026-10-20/B5-night/README.md`): Tester4's install backup at 12:09Z; at the 02:30–06:30Z window the agent answered „within cadence window" (`felhom-agent internal/localapi/server.go:1187`, `age >= tier.Cadence`, default 24 h). R-899 (closed) fixed the same shape for a daytime PRESS only. The other tiers did run that night (database copy, second drive, off-site — all PASS), so app data was covered; the whole-box (DR) tier was not. | **OPEN** — a design question: should the first scheduled night always run (like a press), or is one tier enough on night one? | — | decide; if „always run": treat the install backup like a press in the due rule, with a test on the first night | CC | -| **R-1206** | Backup & restore | P3 | **Backups written while the box's clock ran ahead stay „the latest" after the clock is fixed.** The rehearsal's scenario #10 (clock set to 2026-10-28) left off-site snapshots dated 2026-10-28; on 2026-10-21 the rebuilt box's recovery page listed every app's „Legutóbbi mentés: 2026. 10. 28. 07:39", and the full restore said „mentés: 2026. 10. 28. 07:37" — a date in the future (`audits/fixes-2026-10-21/P1-live/07-unlock-right.txt`, `11-nextcloud-full-restore.txt`). restic orders snapshots by their own time, so the future ones are „newest" to every retention rule for a week. NOT measured: whether the box's keep-daily prune then drops a real snapshot. Related R-1202, R-1204 (clock). | NARROWED — the NAS tier only. **2026-10-22 night (read, `backup/offbox_window.go:22-24, 130-137`):** on the Felhom off-site tier no real snapshot is lost — the guard refuses a prune while any snapshot is future-dated (R-822, by design). The NAS tier prunes without that guard: open. **2026-10-25:** the kept-data choice no longer takes such a future-dated copy as the newest (R-1271, controller `bd9f4f7`); retention is unchanged — this row. | — | A pure unit test with the policy simulator on the NAS tier and one future-dated snapshot; if a real one is dropped, give the NAS tier the same guard | CC | +| **R-1206** | Backup & restore | P3 | **Backups written while the box's clock ran ahead stay „the latest" after the clock is fixed.** The rehearsal's scenario #10 (clock set to 2026-10-28) left off-site snapshots dated 2026-10-28; on 2026-10-21 the rebuilt box's recovery page listed every app's „Legutóbbi mentés: 2026. 10. 28. 07:39", and the full restore said „mentés: 2026. 10. 28. 07:37" — a date in the future (`audits/fixes-2026-10-21/P1-live/07-unlock-right.txt`, `11-nextcloud-full-restore.txt`). restic orders snapshots by their own time, so the future ones are „newest" to every retention rule for a week. NOT measured: whether the box's keep-daily prune then drops a real snapshot. Related R-1202, R-1204 (clock). **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CH02; class R-1384). | NARROWED — the NAS tier only. **2026-10-22 night (read, `backup/offbox_window.go:22-24, 130-137`):** on the Felhom off-site tier no real snapshot is lost — the guard refuses a prune while any snapshot is future-dated (R-822, by design). The NAS tier prunes without that guard: open. **2026-10-25:** the kept-data choice no longer takes such a future-dated copy as the newest (R-1271, controller `bd9f4f7`); retention is unchanged — this row. | — | A pure unit test with the policy simulator on the NAS tier and one future-dated snapshot; if a real one is dropped, give the NAS tier the same guard | CC | | **R-1212** | Backup & restore | P4 | **Two household texts on the backup path do not follow the box's language:** the share-restore summary (`SharesRestoreResult.FlashMessage`, `controller/internal/backup/shares_restore.go`) is built from Hungarian Go literals, and the off-site `LastError` for an interrupted run is the Hungarian literal „megszakadt futás (a vezérlő újraindult futás közben)" (`controller/internal/backup/backup.go`). An English household reads Hungarian there. Found by the night helper 2026-10-21. | OPEN | — | Bundle keys for both (rendered in the box language when written, R-557), with parity fixtures | CC | | **R-1221** | Backup & restore | P4 | **The nightly whole-box backup starts ~1.5–5 min later each night, until one night is skipped (after ~70–80 nights with no press between).** REAL, low harm (night 2026-10-22; split from R-1189 (b)): agent `localapi/server.go:1184` is `age >= Cadence` with the age counted from the archive START; the controller polls every 5 min (`quiesce.go:150`) and the archive starts after the apps stop. Measured on demo-hp: 04:31:07 → 04:33:18 → 04:36:47 → 04:38:14, and 04:33:37 → 04:38:44 → 04:42:40. At W+6h one night is skipped (44 h is below the 48 h valve, `quiesce.go:807-818`); the hub's 26 h line (`deadline.go:29`) mails the operator. Any manual or off-cycle run resets the drift. Evidence `audits/night-2026-10-22/P3-candidates/demohp-whole-box-completions.txt`. | OPEN | — | Fix: inside the night window a tier is due at `age >= cadence - 4h`; the agent answers due-ness, so it is agent code (a unit test with an injected clock) and rides an agent release | CC | | **R-1229** | Backup & restore | P4 | **ArgoCD's per-app auto-sync settings live only on the cluster, not in git.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G9): the Applications differ (`enabled` true/false, prune, selfHeal); a rebuilt cluster from `homelab-manifests` would not have them. | OPEN | — | Write each Application's sync policy into `homelab-manifests` (a homelab change, the operator's yes for the commit that changes live behaviour) | operator | | **R-1231** | Backup & restore | P4 | **The recovery page says the box's host service is „older” when the controller only cannot tell.** Test night 2026-10-23 (`P4-testnight/07a-unlock-refused-fixture-version.txt`): with the agent's capability `unknown` (a version string it could not compare), the page read „…a hozzá tartozó házon belüli szolgáltatás régebbi… A gép magától frissül”. The same page is reached by a merely-down agent, for which „older” and „updates itself” are not true. | OPEN | — | A separate sentence for `unknown` („a gép most nem éri el a házon belüli szolgáltatást — próbáld újra pár perc múlva”); a render test per state; rides a controller release | CC | | **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC | | **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC | -| **R-1294** | Backup & restore | P3 | **demo-hp's weekly off-site clean-up was refused by its own guard on 2026-10-26 (window 16): the plan would remove 209 of 359 snapshots, above the week's cap (179) — nothing was deleted, and each refused week adds to the next plan.** Seen in the operator mailbox during the 2026-10-26 release run (03:17Z, `offsite_prune_guard_refused`, error; details `count_before 359, count_after 359, outcome guard-refused`). The cap refusal is kept on purpose (R-824, R-867 closed the false ones); the open question is whether a store that once exceeds the cap can ever prune again by itself, or refuses every week while it grows. demo-hp is the Tier-0 box with many hand-made backups (sweeps, probes). Earlier cap refusals: demo-hp 2026-10-03 (window 1), demo-felhom 2026-10-05 (window 3). | **OPEN — the grant is SET (D55 = 260, `09` §3 decision 268)**: 2026-10-26 19:10:31Z through the operator's one-purpose token `cc-d55-window`, read back `max:260`; it is used at demo-hp's next window request (after its nightly off-site run, ~03:15Z). Close when that window prunes; then the operator revokes the token. **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 evening: CC could NOT set the D54 grant — `POST /offsite/window-grant/demo-hp` (`max_remove=220`) with CC's hub token answered **403 „missing scope offsite:write"** (the hub log shows the refusal; nothing granted). The plan also grows: restic's dry-run 2026-10-26 18:56Z still 359 copies / plan 209 / default cap 179 (before tonight's run); a simulation of the keep rule adds ~14 per night (14 of 18 groups back up nightly), so ~223 at the night of 10-27 and ~237 at 10-28 — a cap of 220 would be refused again. STATUS D55: the operator grants it, with the cap. The grant opens the box's NEXT window request, which comes after its next nightly off-site run. Ruled D54 = prune once (`09` §3 decision 266). **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 (read only, `audits/r1294-offsite-cap-2026-10-26/`): the plan is HONEST (restic's own dry-run: keep 150, remove 209; every removal 7–30 days old; every app keeps ≥ 9 or its last copies; the bulk is one test day, 2026-10-16, 152 snapshots). It can NOT recover by itself: the cap is half the store and grows ~49/week while the plan grows ~84/week. Only demo-hp has this shape (demo-felhom and Tester 1 pruned normally 10-26). The pick is STATUS D54. **Earlier:** OPEN **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place O1, listed from documents only (the off-site box was not read in this run — second pass). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EC23-7). | — | read the guard's cap rule (controller off-site prune) against demo-hp's snapshot list: is the plan honest (old, superseded) or not; if honest, say how such a store gets back under the cap (an attended one-time prune, or a cap that follows the store) — a design question for the operator if it changes what a box may delete | CC | +| **R-1294** | Backup & restore | P3 | **demo-hp's weekly off-site clean-up was refused by its own guard on 2026-10-26 (window 16): the plan would remove 209 of 359 snapshots, above the week's cap (179) — nothing was deleted, and each refused week adds to the next plan.** Seen in the operator mailbox during the 2026-10-26 release run (03:17Z, `offsite_prune_guard_refused`, error; details `count_before 359, count_after 359, outcome guard-refused`). The cap refusal is kept on purpose (R-824, R-867 closed the false ones); the open question is whether a store that once exceeds the cap can ever prune again by itself, or refuses every week while it grows. demo-hp is the Tier-0 box with many hand-made backups (sweeps, probes). Earlier cap refusals: demo-hp 2026-10-03 (window 1), demo-felhom 2026-10-05 (window 3). **2026-11-11:** the End-dates list leans on this row (`documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB05). | **OPEN — the grant is SET (D55 = 260, `09` §3 decision 268)**: 2026-10-26 19:10:31Z through the operator's one-purpose token `cc-d55-window`, read back `max:260`; it is used at demo-hp's next window request (after its nightly off-site run, ~03:15Z). Close when that window prunes; then the operator revokes the token. **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 evening: CC could NOT set the D54 grant — `POST /offsite/window-grant/demo-hp` (`max_remove=220`) with CC's hub token answered **403 „missing scope offsite:write"** (the hub log shows the refusal; nothing granted). The plan also grows: restic's dry-run 2026-10-26 18:56Z still 359 copies / plan 209 / default cap 179 (before tonight's run); a simulation of the keep rule adds ~14 per night (14 of 18 groups back up nightly), so ~223 at the night of 10-27 and ~237 at 10-28 — a cap of 220 would be refused again. STATUS D55: the operator grants it, with the cap. The grant opens the box's NEXT window request, which comes after its next nightly off-site run. Ruled D54 = prune once (`09` §3 decision 266). **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 (read only, `audits/r1294-offsite-cap-2026-10-26/`): the plan is HONEST (restic's own dry-run: keep 150, remove 209; every removal 7–30 days old; every app keeps ≥ 9 or its last copies; the bulk is one test day, 2026-10-16, 152 snapshots). It can NOT recover by itself: the cap is half the store and grows ~49/week while the plan grows ~84/week. Only demo-hp has this shape (demo-felhom and Tester 1 pruned normally 10-26). The pick is STATUS D54. **Earlier:** OPEN **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place O1, listed from documents only (the off-site box was not read in this run — second pass). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EC23-7). | — | read the guard's cap rule (controller off-site prune) against demo-hp's snapshot list: is the plan honest (old, superseded) or not; if honest, say how such a store gets back under the cap (an attended one-time prune, or a cap that follows the store) — a design question for the operator if it changes what a box may delete | CC | | **R-1362** | Backup & restore | P3 | **With the internet down, the Backups page takes over 2 minutes (135 s and 145 s measured; 5 s with only the hub down).** Walked 2026-11-03 on Tester 1: every other page answers at once on the home network. The page waits on something off-site inside its handler; a household that lost its line and opens exactly this page sees a hang. `audits/security-2026-11-02/part5/outage-wan.log` | OPEN — filed 2026-11-03 | — | name the wait with a timing, then give it a short deadline and show „the off-site state could not be read" | CC | | **R-1374** | Backup & restore | P3 | **Four kinds of copy are kept with no end named, and nobody is reminded of them.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC02, EC23, EC18, EC27; read in source; 9202 has no off-site store, so not measured). (1) **Off-site snapshots of a removed app**: the removal keeps them by ruling (closed R-474), and the weekly clean-up keeps the last snapshots of each app for ever (`internal/backup/offbox_window.go:80-82`), also for an app that is gone. No rule says when they end. (2) **A whole restore from the second drive** moves the old files beside the restored ones (`internal/backup/tier2_whole.go:52`, `:132`, `:345`); nothing removes them later and no page lists them. (3) **Exported app files** stay in the export folder of the drive; the pages can start, list and read an export, none deletes one, and nothing ages it (`internal/web/handler_export.go:30-87`). **Members (answers marked N):** EC02-4, EC23-4, EC18-4, EC27-4. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D72 = yes, by the household's own choice; `09` §3 decision 292).** Not built. | — | Controller (D72): when the household pressed „delete backup data” for an app, its off-site copies end 30 days later, at the next clean-up after that day (the number of a restore copy, D62); when the household kept the backups, the off-site copies stay. The page shows the end date before anything is deleted — write that into `07-backup-architecture.md` before the build. Parts (2) and (3): a delete on the export page; a line on the kept-data page for the files a whole restore set aside. | CC | +| **R-1385** | Backup & restore | P3 | **Two timed jobs can meet with no lock and no order (12 answers).** Found 2026-11-11 (the Clock list, class W3, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED on the night of 2026-11-11: on Tester 1 and the N100 the whole-guest backup started at about 05:30 local, the minute of the off-site proof restore (CC09-5); on demo-hp one whole-guest backup started at 02:30:38, with the dump leg; on the N100 the whole-guest backup starts about 5 minutes later each day. From source: the controller's auto-update default is the minute the whole-guest gate opens (CC11-5); the second-drive copy and the abandon sweep wait only for the clock (CC02-5, CC19-5, CC21-5); the PBS verify can run beside a backup (CA13-5, CA04-5); the timers on DooPlex and ep0 are not ordered against the hub's copies (CH05-8, CH24-5, CH25-3, CH26-5, CH26-6; from documents only, not measured). No harm was seen; the durations are from small test stores. | OPEN — filed 2026-11-11 (clock list) | — | Measure one night with a large store on Tier 0 (as a task), then decide per pair: a shared flag, a fixed offset, or „harmless, written down”. | CC | ## Storage & devices — 15 rows (P3 6, P4 9) @@ -255,13 +256,13 @@ stopping line that lies. | **R-1258** | Security & access | P3 | **ep0's iperf3 line-test server takes any tunnel peer's test: one box (root on its host) can keep it busy for the fleet or ask a long reverse test that fills ep0's uplink.** Found 2026-10-24 (SEC-2 X2): `felhom.eu/scripts/ep0-iperf3/felhom-iperf3.service:17` — no authorized users, no server-side duration or rate cap; one client at a time (`felhom-agent/internal/linetest/iperf.go:263-264` → `busy`; 6 tries a day). Static (no ep0 change in this run). | OPEN | — | per-box iperf3 users or server-side caps, tried on a scratch VM first | CC | | **R-1307** | Security & access | P3 | **Packages inside our own images are not watched** — the Debian packages of the controller image, the Alpine packages of the hub and Samba images (Samba's version is whatever Alpine 3.24 ships). The security watch (R-1303) reads only the base image's point release (BEHIND); Debian's DSAs and Alpine's secdb are not read. `audits/security-watch-2026-10-26/SPIKE.md` „cannot see" | OPEN — filed 2026-10-26 | — | add Alpine secdb (machine-readable JSON per release) and Debian's tracker for the packages the images install, or a scanner run on the built images in CI | CC | | **R-955** | Security & access | P4 | **The cross-box check of the hub API is done in source and tests but not fully proven: if a later check masks a missing id check, a box's key could read or change another box's hub record without any test noticing.** Read 2026-10-12: every box-facing `/api/v1` route takes the box from its own key or refuses a different id; 14 routes already had a refusal test, 5 got one (`crossbox_refusal_test.go`). Left: red-prove the 4 new tests that were not red-proved (as the restore-directive one was), and the live read-only check with one demo box's key against the other demo box's id. | OPEN — filed 2026-10-12 by the security review | — | Red-prove the 4 tests; run the live GET check on the two demo boxes | CC | -| **R-924** | Security & access | P2 | **The operator signing keys exist only on DooPlex, with no passphrase and in no backup — a loss of DooPlex means no box ever accepts a signed agent update, config bundle or OS step again without an on-site re-enrollment.** MEASURED 2026-10-09: `/mnt/5_hdd/felhom.eu/felhom-op-operational`, `felhom-rec-recovery`, `felhom_op_ed25519` (0600 `kisfenyo`); `ssh-keygen -y -P ''` opens all three (no passphrase); `/mnt/5_hdd/felhom.eu` is in no backup set, the off-site job does not carry them. `04` §1 says the operational key is "passphrase-protected" and the recovery key "kept offline" — today neither holds (custody ruling 1 of 2026-09-16 put both on DooPlex for this phase; `04` §3.1). Both keys share one fate, so the cold key cannot do its one job (re-pin after losing the operational key). `runbooks/total-loss-of-dooplex.md` S6. | NARROWED — **2026-10-23 (D32):** S6a/S6b are on the current sheet. LEFT: the operator prints it. **NARROWED 2026-10-09 — operator ruling: option A** (print both AND copy them). The three key files and their public halves ride the nightly encrypted DooPlex→ep0 copy (`scripts/dooplex-offsite/`, a missing key refuses the push); the Sunday restore test proves each private key still derives its own public half (never printed). **LEFT:** the operator prints S6a/S6b (`runbooks/break-glass-sheet.md`); the keys still have no passphrase (custody ruling 1 of 2026-09-16 stands until the first sale). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G2):** the three key files are on `sda1`, the same disk as DooPlex's local backup tree; their only off-box copy is the S1-encrypted nightly one. Passphrase: unchanged by the custody ruling. | operator: print | Operator: print the two keys; then close | operator | +| **R-924** | Security & access | P2 | **The operator signing keys exist only on DooPlex, with no passphrase and in no backup — a loss of DooPlex means no box ever accepts a signed agent update, config bundle or OS step again without an on-site re-enrollment.** MEASURED 2026-10-09: `/mnt/5_hdd/felhom.eu/felhom-op-operational`, `felhom-rec-recovery`, `felhom_op_ed25519` (0600 `kisfenyo`); `ssh-keygen -y -P ''` opens all three (no passphrase); `/mnt/5_hdd/felhom.eu` is in no backup set, the off-site job does not carry them. `04` §1 says the operational key is "passphrase-protected" and the recovery key "kept offline" — today neither holds (custody ruling 1 of 2026-09-16 put both on DooPlex for this phase; `04` §3.1). Both keys share one fate, so the cold key cannot do its one job (re-pin after losing the operational key). `runbooks/total-loss-of-dooplex.md` S6. **2026-11-11:** the End-dates list leans on this row (`documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB12). | NARROWED — **2026-10-23 (D32):** S6a/S6b are on the current sheet. LEFT: the operator prints it. **NARROWED 2026-10-09 — operator ruling: option A** (print both AND copy them). The three key files and their public halves ride the nightly encrypted DooPlex→ep0 copy (`scripts/dooplex-offsite/`, a missing key refuses the push); the Sunday restore test proves each private key still derives its own public half (never printed). **LEFT:** the operator prints S6a/S6b (`runbooks/break-glass-sheet.md`); the keys still have no passphrase (custody ruling 1 of 2026-09-16 stands until the first sale). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G2):** the three key files are on `sda1`, the same disk as DooPlex's local backup tree; their only off-box copy is the S1-encrypted nightly one. Passphrase: unchanged by the custody ruling. | operator: print | Operator: print the two keys; then close | operator | | **R-132** | Security & access | P3 | **`curl -w '%{redirect_url}'` reconstructs the request URL WITH its basic-auth credential** — so a `-u ":$HUB_PW"` call that never put the password in a URL still printed it **Merged 2026-10-05 from R-350 (duplicate):** (1) 2026-08-20 occurrence: POST /configuration/artifacts answers 303; leak lives only in the CC transcript under ~/.claude/projects/ on DooPlex, not in git/evidence (checked then). (2) `-v` and `--libcurl` also re-render the credential, not only %{redirect_url}; confirm redirects with %{http_code} + follow-up GET. (3) Rotation path: hub /configuration form (current_password/new_password/confirm_password); DB override wins over ConfigMap (break-glass); CC can rotate file-to-file without printing (operator-present-one-time-secrets) if asked. | **NARROWED 2026-10-20 — the rotation is DONE:** the operator changed the hub password on 2026-10-15 and the `HUB_PW` line is gone (`audits/hub-tokens-2026-10-15/C153-live.txt:18`; CC uses its own token since hub 0.152.0). **Left, the operator's:** should the break-glass `password_hash` in `manifests/hub.yaml` (a public repository) be emptied or replaced? The DB override wins at login, so the old hash no longer opens the hub. **WAITING-ON-OPERATOR** — **ACTION: rotate `HUB_PW`** **Folded R-580 2026-10-03** (the same `curl -w %{redirect_url}` credential echo, seen again 2026-09-18). **Note 2026-10-12 (security review):** `manifests/hub.yaml` also carries the hub login's bcrypt hash in this public repository (the break-glass field); when `HUB_PW` is rotated, decide whether that field should stay empty. | the operator's word on the `manifests/hub.yaml` hash field | Happened on 2026-07-31 while red-proofing the R-120 gate: the hub operator password was written to the session transcript by the write-out format, not by the request. `-u` is safe; the *reporting* was not. Rule: read the redirect from `-D -` and grep `^Location:`, never `%{redirect_url}`, on any authenticated call. Rotate the hub password (`/configuration` → Login password; ConfigMap `auth.password_hash` is the reset path) and update `~/.config/credentials` | Viktor | | **R-137** | Security & access | P3 | **Cloudflare geo-WAF rules are zone-scoped and non-namespaced — four cross-tenant faults.** `globalRuleDesc = "[felhom-geo] Global"` (`waf.go:18`) is one literal description per ZONE; `appRuleDescPrefix` keys by app name with no customer (`waf.go:21`); `BuildGlobalExpression` has no positive hostname scoping (`waf.go:241`); `applyDiff` deletes every `[felhom-geo]` rule not in THIS box's desired set (`geosync.go:320`) | READY (M) — **blocks shared-zone onboarding** | — | With two customers in one zone: they overwrite each other's Global rule forever; one customer's country policy applies zone-wide; per-app rules collide by name; and disabling the feature for one (or the hub's `RemoveGeoRules`) wipes them all. Interim mitigation, no code: keep geo-restriction OFF for every shared-zone customer. Fix = namespace descriptions by `customer_id` + add `http.host ends_with ""` to both expressions — a TWO-REPO change (controller + hub `RemoveGeoRules`). Same audit §5.1 | CC | | **R-775** | Security & access | P3 | **[P2-MEDIUM] Grimmory: a stranger's 5 wrong sign-ins lock EVERY visitor out of the web login for 15 minutes — so Grimmory was not published.** MEASURED 2026-10-01 on 9202 (drill catalog, v3.4.1, through traefik): after 5 wrong tries for `admin` every further sign-in answered 429 — the household's right password AND a different name — and stayed 429 for 10+ minutes of retries. Read in the jar: `AuthRateLimitService` — Caffeine `expireAfterWrite(ofMinutes(15))`, `MAX_ATTEMPTS 5`, keys `login:ip:` and `login:user:`; Spring `forward-headers-strategy: native` takes the address from X-Forwarded-For, and behind the tunnel every visitor is the tunnel container's address (R-753) — the wger shape (R-752), with no setting to change it. Everything else in the checklist passed (bench + box step v3.4.1 → v3.5.0, gate by its own probe, OPDS through traefik); two smaller findings for the publishing session: on a reinstall over the first install's kept books, a new upload was saved to the drive but not added to the library (`box/grimmory/reinstall-c1.txt`, not investigated); and the remove + restore round trip (2.5) cannot be shown on 9202 for a drive app — its backup lives on the scratch drive, which is not a registered drive (R-756). The template waits in `audits/new-apps-2026-10-01/wip/grimmory/`. **Needs (operator):** (A) publish with a sentence on the page that wrong guesses by others can lock the login for 15 minutes (MEASURED: during the lock an e-reader's OPDS feed still answered 200 with its own login, wrong 401 — `box/grimmory/opds-under-lock.txt`), or (B) wait until the box passes each visitor's real address (R-753). `audits/new-apps-2026-10-01/box/grimmory/throttle.txt` **-- 2026-10-01 (evening):** option B's precondition SHIPPED (controller v0.286.1, R-753): Grimmory's Tomcat RemoteIpValve walks from the right and counts `172.16.0.0/12` as a proxy (READ in source, Spring Boot 4.1.1 — not yet measured with Grimmory's own lock), so `login:ip:` becomes per visitor; `login:user:` still lets a stranger lock the public name `admin` 15 min. A third route was spiked and passed: Grimmory behind the permanent family gate with its e-reader paths excepted (R-780). Recommendation: publish behind the family gate if R-780 is built; otherwise B with a measured 3.6. **UPDATE 2026-10-02 — NARROWED, Grimmory PUBLISHED behind the family gate:** a stranger cannot reach Grimmory's web sign-in at all (6 tries through the simulated tunnel: the gate's 401, then the household signs in 200 — `audits/family-gate-2026-10-02/A/items.txt`), and the e-reader exceptions keep Grimmory's own login. The 2.5 round trip now WORKS on 9202 (`audits/family-gate-2026-10-02/B/box/life.txt`). **What is left:** a family member past the gate can still lock a NAME (the admin's) for 15 minutes with 5 wrong tries — hard-coded in Grimmory; and the reinstall-over-kept-books finding (`new-apps-2026-10-01/box/grimmory/reinstall-c1.txt`) is still not investigated. | **WATCHING — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P2→P3: Grimmory is now behind the family gate; only a family member can still lock a name.** | — | — | CC | | **R-908** | Security & access | P3 | **An old Resend API key is still in the history of the `homelab-manifests` repository; it was replaced on 2026-06-29, but whether it is also revoked at Resend is unknown.** FOUND 2026-10-08 by the contact-mailer source search (`audits/mailer-source-2026-10-08/SEARCH.md`, „A side finding"): commit c9648cd (2026-02-05, „added mailer pod") put a key literal in the old `felhom-system/contact-mailer.yaml` comment; the file was deleted in ee93b50 but the history keeps it. Compared by hash only, never printed: it is NOT today's key (`Secret/resend-api`, rotated 2026-06-29, felhom.eu feea0606). If the old key still works at Resend, anyone with read access to that repository can send mail as felhom.eu. | **OPEN — owner: operator** **Note 2026-10-12 (security review):** gitleaks 8.24.3 with its default rules did NOT flag this key in the full history scan, so a scanner gate alone would not have caught it (R-986). | — | In the Resend console, check that the old key is revoked (revoke it if not); rewriting the repository's history is NOT needed once it is revoked | operator | | **R-525** | Security & access | P4 | **[P3-LOW] FileBrowser has its own login; putting it behind the dashboard session (traefik forwardAuth or Quantum proxy auth) is a new mechanism nobody has measured.** Filed 2026-09-15 by the P1-fixes task (B.5). R-513 closed the default-password hole with a generated password; a household still has two logins. **What it needs:** a spike on a scratch guest — forwardAuth to the controller session, and what FileBrowser Quantum does with a trusted header. | **READY — rank P3-LOW; owner: CC (spike)** **Re-ranked 2026-10-03: P3->P4: comfort feature needing a new unmeasured mechanism; the default password hole is closed.** | — | — | CC | -| **R-925** | Security & access | P1 | **`manifests/felhom.secret.yaml` carries REAL secret values, and the Gitea repo that holds it is readable by anyone on the internet with no login.** FOUND 2026-10-09 by a background security review of the legal-pages commit, which flagged the served `` comments; chasing what those comments pointed at found this instead. **MEASURED, in this order:** (1) `scripts/manifest_bearer_gate.py` itself prints `manifests/felhom.secret.yaml:39 KNOWN-BACKLOG committed secret 65cee3c4…7a86`; (2) the file holds **live-shaped values, not placeholders** — `SECRET_KEY` (69 chars), `SUPERUSER_EMAIL`/`SUPERUSER_PASSWORD` (18), Umami `APP_SECRET` (66) and `POSTGRES_PASSWORD` (34), and a `username`/`password` pair (18); values were never printed, only measured by length; (3) `gitea.dooplex.hu` resolves to **37.191.56.193**, the same public address as the website; (4) an **anonymous** `curl` returns HTTP 200 and 1686 bytes for that exact file, and 200 for `hub/internal/store/store.go` and `manifests/hub.yaml`; (5) **the off-network control**, which is what makes (4) mean anything — fetched from outside the operator's network entirely: a real path returns the file's first line, a nonsense path returns 404, so the 200 is genuine anonymous read from the internet and not a LAN-only ACL. **This contradicts the project's own written model**: `documentation/runbooks/secrets.md:3-4` says *“Secret values are never committed to git. The manifests in `manifests/` carry only placeholders + comments that point here.”* That promise is false today, which is the P1 wording in this register's own scale — see the ranking note below. The same runbook already states the rule that governs the fix: *“The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing.”* **Known neighbours, none of which cover this:** R-887 records that an outside crawler walks the public Gitea pages and leaves *“Gitea's exposure to the crawler”* as the operator's undone item; R-580 still owes a Gitea admin token rotation. Neither says a secret file is world-readable. **Why P2 and not P1, stated so the operator can overrule it:** the exposed credentials guard **analytics and an undeployed healthchecks instance, not household data** — measured: `umami-db` is a **ClusterIP** service on 5432 with no external IP, so the Postgres password is not reachable from the internet; the realistic harm is session forgery against the public `stats.felhom.eu` via `APP_SECRET`, and reuse of those passwords anywhere else. No customer box, hub token or escrow key is in this file. **CC did NOT change anything**: making the repo private could break the public day-0 path (the installer is fetched from a public tag, R-110), and rotation plus repository visibility are operator decisions on production infrastructure. **PARTLY FIXED 2026-10-09, and RE-RANKED P2 -> P1 on what it turned out to be.** The exposed value was not an analytics password: it was the **Gitea `admin` account password** (`is_admin: true`; `/api/v1/admin/users` answered 200), published in a repo `gitea.dooplex.hu` serves anonymously to the internet. That is push access to every repo — including the one whose `website/` is git-synced live and whose `scripts/` is published by tag to **every new box installer** (R-110), i.e. a supply-chain path onto customer hardware. My first ranking said P2 because I had only measured the analytics blast radius; the credential test is what corrected it. **DONE (CC, operator-instructed):** (1) `umami-config` `APP_SECRET` + `POSTGRES_PASSWORD` rotated — the password changed **inside Postgres** too (`ALTER USER`), because the env var is only read at first init; verified by a real beacon returning 200 with a bogus-site-id control still returning 400. (2) `healthchecks-config` `SECRET_KEY` + `SUPERUSER_PASSWORD` rotated (nothing consumes them — there is no healthchecks Deployment). (3) `gitea-creds` no longer holds the admin password at all: it holds a **scoped token** (`read:package` + `read:repository`). **Both scopes are load-bearing and the second was found by breaking it** — a package-only token made the hub log `Template fetch: unexpected status 403`, because the template fetcher reads a raw file out of the `felhom-controller` repo. (4) The Gitea **admin password** was changed and `gitea-system/gitea-admin` updated; **verified: new password 200, old published password 401 — the leak is dead.** (5) The file is removed from git and `.gitignore`'s `*secret*` now applies to it; `manifest_bearer_gate.py`'s `KNOWN_BACKLOG` carve-out is gone, red-proofed with a decoy (exit 1 with, 0 without). **AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part:** the `rollout restart` needed to pick up the new umami secret put umami into **CrashLoopBackOff and took `stats.felhom.eu` down (503)**. Cause was not the rotation — at `memory: 512Mi` the pod runs for months but **cannot restart**, because startup (Prisma + Next.js) peaks over the limit and is OOMKilled (exit 137). Raised to 1Gi, **in the manifest, not just live** (the `.claude/rules/manifests.md` rule: a bare `kubectl set` is reverted by the next sync and the fix silently disappears). Service restored and verified. New values are in a 0600 file on DooPlex (`~/rotated-secrets-2026-10-09.txt`), never echoed; the operator moves them to the password manager and deletes it. **WHAT REMAINS, and it is bigger than what was fixed:** that one password is still the admin password in about **12 other namespaces** — `nextcloud`, `paperless`, `bookstack` (a **database root** password), `tandoor`, `calibre`, `adventurelog`, `gokapi`, `qbittorrent` (x2), `servarr`, `homepage` (x2). Rotating Gitea does not touch them: each is its own login and each is still the string that was published. **Also owed:** a Gitea token for user `kisfenyo` sits in plaintext in the `origin` URL of the local `homelab-manifests` clone (`.git/config`); CC printed it to a session transcript while investigating, so it should be rotated regardless — this is R-580's shape (store the remote without credentials). **Not a finding:** `homelab-manifests` itself is private (404 anonymously) and does not contain the password; ArgoCD's repo credential is a separate token, so the rotation did not touch it. | **NARROWED 2026-10-09 — the felhom.eu half is DONE and verified (Gitea admin password rotated, old one now 401; umami + healthchecks rotated; file de-gitted; gate carve-out removed). What remains is the ~12 reused logins elsewhere in the homelab and the repo-visibility decision; owner: operator** **2026-10-10 (release session), two consequences measured:** (a) DooPlex's own Docker login for `gitea.dooplex.hu` (`~/.docker/config.json`) still holds the OLD admin password — the registry token endpoint answers 401 for it and 200 for the current one — so every image push from DooPlex fails `unauthorized` until it is refreshed; the 0.145.0/0.305.0 pushes used a one-off login in a scratch `DOCKER_CONFIG`, shredded after. (b) The `felhom` ArgoCD app reads OutOfSync for `Secret/gitea-creds`, `Secret/healthchecks-config`, `Secret/umami-config` and `Deployment/umami`: a whole-app sync would push git's (de-gitted) view over the rotated live Secrets — the hub deploy synced ONLY `Deployment/hub` (`audits/release-2026-10-10/hub-0.145.0-deploy.txt`). Refreshing the DooPlex login is a DooPlex change (operator's word). **2026-10-22 night (Part 6, read only, `audits/night-2026-10-22/P6-repos-private/01-readers.md`):** homelab-manifests is already private; felhom-controller can go private after one change — CI clones with a read token (check first that the hub's `gitea-creds` token user can read a private `admin` repo); felhom.eu must stay public now (the website git-sync has no credential), felhom-agent too (every install fetches 16 config files from `raw/tag` with no login), and the catalog too (every box clones it with no login, and a token on a customer box is readable by its owner). Making a repository private does NOT hide its images or packages: they follow the user `admin`, which is public — and boxes need that (NOT-A-FINDING). **Count corrected:** the served `` comments are 61 on four pages (adatkezeles 33, feltetelek 15, teszteles 7, en/testing 6), not „32 on /adatkezeles”. The visibility question is decision D29 in STATUS. **2026-10-22 evening — D29 = no (operator):** the controller repository stays public for now; this row's secret rotation is unchanged. | — | **OPERATOR RULED 2026-10-09: (a) he rotates the remaining services himself — CC's scope stopped at the Felhom boundary; (b) the repo STAYS anonymously readable for now**, because making it private breaks the website git-sync and the installer tag fetch, which both clone with no credentials (R-110). The standing consequence of (b): no secret may ever enter this repo again, which `manifest_bearer_gate.py` now enforces with no exemption; and if (b) is reversed, the 61 served `` comments (four pages, 33 on /adatkezeles) must be stripped in the same change. **The exact checklist of the 12 remaining secrets** (namespace / secret / key, re-measured after the rotation, with the two traps that bite — a DB password is not changed by editing the Secret, and a pod that has run for months may not restart) is in `documentation/runbooks/secrets.md`. Operator: **(1)** work that checklist, `bookstack-db/root-password` first because it is a database root password and every one of these hosts answers on the public internet; (`bookstack` first — it is a database root password); **(2)** rotate the `kisfenyo` Gitea token embedded in the local `homelab-manifests` remote URL, and store the remote without credentials; **(3)** decide whether `gitea.dooplex.hu` should answer anonymously at all — remembering the website git-sync clones it with no credentials and the installer is fetched from a public tag, so making it private breaks both unless they get credentials first, and the 61 served `` comments become a map of it; **(4)** move `~/rotated-secrets-2026-10-09.txt` into the password manager and delete it. If nothing is done: the published password keeps opening a dozen services, even though Gitea itself is now safe | operator | +| **R-925** | Security & access | P1 | **`manifests/felhom.secret.yaml` carries REAL secret values, and the Gitea repo that holds it is readable by anyone on the internet with no login.** FOUND 2026-10-09 by a background security review of the legal-pages commit, which flagged the served `` comments; chasing what those comments pointed at found this instead. **MEASURED, in this order:** (1) `scripts/manifest_bearer_gate.py` itself prints `manifests/felhom.secret.yaml:39 KNOWN-BACKLOG committed secret 65cee3c4…7a86`; (2) the file holds **live-shaped values, not placeholders** — `SECRET_KEY` (69 chars), `SUPERUSER_EMAIL`/`SUPERUSER_PASSWORD` (18), Umami `APP_SECRET` (66) and `POSTGRES_PASSWORD` (34), and a `username`/`password` pair (18); values were never printed, only measured by length; (3) `gitea.dooplex.hu` resolves to **37.191.56.193**, the same public address as the website; (4) an **anonymous** `curl` returns HTTP 200 and 1686 bytes for that exact file, and 200 for `hub/internal/store/store.go` and `manifests/hub.yaml`; (5) **the off-network control**, which is what makes (4) mean anything — fetched from outside the operator's network entirely: a real path returns the file's first line, a nonsense path returns 404, so the 200 is genuine anonymous read from the internet and not a LAN-only ACL. **This contradicts the project's own written model**: `documentation/runbooks/secrets.md:3-4` says *“Secret values are never committed to git. The manifests in `manifests/` carry only placeholders + comments that point here.”* That promise is false today, which is the P1 wording in this register's own scale — see the ranking note below. The same runbook already states the rule that governs the fix: *“The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing.”* **Known neighbours, none of which cover this:** R-887 records that an outside crawler walks the public Gitea pages and leaves *“Gitea's exposure to the crawler”* as the operator's undone item; R-580 still owes a Gitea admin token rotation. Neither says a secret file is world-readable. **Why P2 and not P1, stated so the operator can overrule it:** the exposed credentials guard **analytics and an undeployed healthchecks instance, not household data** — measured: `umami-db` is a **ClusterIP** service on 5432 with no external IP, so the Postgres password is not reachable from the internet; the realistic harm is session forgery against the public `stats.felhom.eu` via `APP_SECRET`, and reuse of those passwords anywhere else. No customer box, hub token or escrow key is in this file. **CC did NOT change anything**: making the repo private could break the public day-0 path (the installer is fetched from a public tag, R-110), and rotation plus repository visibility are operator decisions on production infrastructure. **PARTLY FIXED 2026-10-09, and RE-RANKED P2 -> P1 on what it turned out to be.** The exposed value was not an analytics password: it was the **Gitea `admin` account password** (`is_admin: true`; `/api/v1/admin/users` answered 200), published in a repo `gitea.dooplex.hu` serves anonymously to the internet. That is push access to every repo — including the one whose `website/` is git-synced live and whose `scripts/` is published by tag to **every new box installer** (R-110), i.e. a supply-chain path onto customer hardware. My first ranking said P2 because I had only measured the analytics blast radius; the credential test is what corrected it. **DONE (CC, operator-instructed):** (1) `umami-config` `APP_SECRET` + `POSTGRES_PASSWORD` rotated — the password changed **inside Postgres** too (`ALTER USER`), because the env var is only read at first init; verified by a real beacon returning 200 with a bogus-site-id control still returning 400. (2) `healthchecks-config` `SECRET_KEY` + `SUPERUSER_PASSWORD` rotated (nothing consumes them — there is no healthchecks Deployment). (3) `gitea-creds` no longer holds the admin password at all: it holds a **scoped token** (`read:package` + `read:repository`). **Both scopes are load-bearing and the second was found by breaking it** — a package-only token made the hub log `Template fetch: unexpected status 403`, because the template fetcher reads a raw file out of the `felhom-controller` repo. (4) The Gitea **admin password** was changed and `gitea-system/gitea-admin` updated; **verified: new password 200, old published password 401 — the leak is dead.** (5) The file is removed from git and `.gitignore`'s `*secret*` now applies to it; `manifest_bearer_gate.py`'s `KNOWN_BACKLOG` carve-out is gone, red-proofed with a decoy (exit 1 with, 0 without). **AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part:** the `rollout restart` needed to pick up the new umami secret put umami into **CrashLoopBackOff and took `stats.felhom.eu` down (503)**. Cause was not the rotation — at `memory: 512Mi` the pod runs for months but **cannot restart**, because startup (Prisma + Next.js) peaks over the limit and is OOMKilled (exit 137). Raised to 1Gi, **in the manifest, not just live** (the `.claude/rules/manifests.md` rule: a bare `kubectl set` is reverted by the next sync and the fix silently disappears). Service restored and verified. New values are in a 0600 file on DooPlex (`~/rotated-secrets-2026-10-09.txt`), never echoed; the operator moves them to the password manager and deletes it. **WHAT REMAINS, and it is bigger than what was fixed:** that one password is still the admin password in about **12 other namespaces** — `nextcloud`, `paperless`, `bookstack` (a **database root** password), `tandoor`, `calibre`, `adventurelog`, `gokapi`, `qbittorrent` (x2), `servarr`, `homepage` (x2). Rotating Gitea does not touch them: each is its own login and each is still the string that was published. **Also owed:** a Gitea token for user `kisfenyo` sits in plaintext in the `origin` URL of the local `homelab-manifests` clone (`.git/config`); CC printed it to a session transcript while investigating, so it should be rotated regardless — this is R-580's shape (store the remote without credentials). **Not a finding:** `homelab-manifests` itself is private (404 anonymously) and does not contain the password; ArgoCD's repo credential is a separate token, so the rotation did not touch it. **2026-11-11:** CC's `felhom-agent` clone no longer holds a login and a token in its address: git gets the token from the secret store for one command (`scripts/git-credential-sec.sh`, on disk only until a code push passes the golden gate; proof and the before/after measurement in `documentation/audits/no-token-in-address-2026-11-11/`). Two clones still hold one (`cc-secrets`, `clones-plugmeter-2026-10-13/felhom-agent`): the operator's yes did not come in the 30 minutes the brief gave. The token of login `kisfenyo` was printed into a session on 2026-11-10; the operator rotates it once, into the store only (`GITEA_GIT_TOKEN`). | **NARROWED 2026-10-09 — the felhom.eu half is DONE and verified (Gitea admin password rotated, old one now 401; umami + healthchecks rotated; file de-gitted; gate carve-out removed). What remains is the ~12 reused logins elsewhere in the homelab and the repo-visibility decision; owner: operator** **2026-10-10 (release session), two consequences measured:** (a) DooPlex's own Docker login for `gitea.dooplex.hu` (`~/.docker/config.json`) still holds the OLD admin password — the registry token endpoint answers 401 for it and 200 for the current one — so every image push from DooPlex fails `unauthorized` until it is refreshed; the 0.145.0/0.305.0 pushes used a one-off login in a scratch `DOCKER_CONFIG`, shredded after. (b) The `felhom` ArgoCD app reads OutOfSync for `Secret/gitea-creds`, `Secret/healthchecks-config`, `Secret/umami-config` and `Deployment/umami`: a whole-app sync would push git's (de-gitted) view over the rotated live Secrets — the hub deploy synced ONLY `Deployment/hub` (`audits/release-2026-10-10/hub-0.145.0-deploy.txt`). Refreshing the DooPlex login is a DooPlex change (operator's word). **2026-10-22 night (Part 6, read only, `audits/night-2026-10-22/P6-repos-private/01-readers.md`):** homelab-manifests is already private; felhom-controller can go private after one change — CI clones with a read token (check first that the hub's `gitea-creds` token user can read a private `admin` repo); felhom.eu must stay public now (the website git-sync has no credential), felhom-agent too (every install fetches 16 config files from `raw/tag` with no login), and the catalog too (every box clones it with no login, and a token on a customer box is readable by its owner). Making a repository private does NOT hide its images or packages: they follow the user `admin`, which is public — and boxes need that (NOT-A-FINDING). **Count corrected:** the served `` comments are 61 on four pages (adatkezeles 33, feltetelek 15, teszteles 7, en/testing 6), not „32 on /adatkezeles”. The visibility question is decision D29 in STATUS. **2026-10-22 evening — D29 = no (operator):** the controller repository stays public for now; this row's secret rotation is unchanged. | — | **OPERATOR RULED 2026-10-09: (a) he rotates the remaining services himself — CC's scope stopped at the Felhom boundary; (b) the repo STAYS anonymously readable for now**, because making it private breaks the website git-sync and the installer tag fetch, which both clone with no credentials (R-110). The standing consequence of (b): no secret may ever enter this repo again, which `manifest_bearer_gate.py` now enforces with no exemption; and if (b) is reversed, the 61 served `` comments (four pages, 33 on /adatkezeles) must be stripped in the same change. **The exact checklist of the 12 remaining secrets** (namespace / secret / key, re-measured after the rotation, with the two traps that bite — a DB password is not changed by editing the Secret, and a pod that has run for months may not restart) is in `documentation/runbooks/secrets.md`. Operator: **(1)** work that checklist, `bookstack-db/root-password` first because it is a database root password and every one of these hosts answers on the public internet; (`bookstack` first — it is a database root password); **(2)** rotate the `kisfenyo` Gitea token embedded in the local `homelab-manifests` remote URL, and store the remote without credentials; **(3)** decide whether `gitea.dooplex.hu` should answer anonymously at all — remembering the website git-sync clones it with no credentials and the installer is fetched from a public tag, so making it private breaks both unless they get credentials first, and the 61 served `` comments become a map of it; **(4)** move `~/rotated-secrets-2026-10-09.txt` into the password manager and delete it. If nothing is done: the published password keeps opening a dozen services, even though Gitea itself is now safe | operator | | **R-1243** | Security & access | P2 | **The hub's Longhorn backups sit unencrypted in the NFS folder any DooPlex pod can read (R-1241) — the two there now (2026-10-18, 2026-10-23) hold every household's Cloudflare and tunnel tokens in plaintext, because they predate hub 0.168.0.** Measured 2026-10-23 night (`audits/hub-token-seal-2026-10-23/` §3): the hub volume is `encrypted=false` and in Longhorn's `default` backup group (daily 04:00 + weekly Sunday 05:00, retain 1 each); the hub's own copy to ep0 is encrypted at rest (16/16 snapshots `crypt-mode: encrypt`, key on DooPlex's root disk and on paper) and in transit (SSH tunnel + pinned PBS TLS), every 6 h, restore-tested weekly. DECIDE: (a) an encrypted Longhorn volume for the hub (~5–10 min downtime, a homelab StorageClass + key in the cluster) or (b) take the hub volume out of the Longhorn backup — ep0's copy is the backup; the access log then has no backup until it rides the ep0 push. **CC's pick: (b).** And: delete the two plain copies (the operator's call; with (b) they never rotate out). | **VERIFY** — **2026-10-24: D40 put into effect** (operator ruling 07:22): the hub PVC `hub-data` carries `recurring-job-group.longhorn.io/default: disabled` (felhom.eu `3363ffa8`, ArgoCD sync of the PVC only) and the Longhorn volume `pvc-486c9809…` reads `disabled` too (labelled live to match, unchanged after 2 min); the NFS folder holds no copy of the volume (no `pvc-486c9809*` directory, no backup config naming it; Longhorn lists no backup volume for it; 53 other volumes backed up this morning). Closes after the dated check: tonight's 04:00 job makes no new hub backup while the others run. WAITING-ON-OPERATOR **2026-10-25 dated check DONE (read-only, the restore-truth session):** `kubectl -n longhorn-system get backups.longhorn.io` lists 106 backups made today (the newest 04:08:28Z — the job ran) and none of `pvc-486c9809` — the hub volume is out of the Longhorn backup as D40 meant. | the operator's pick; a homelab change for either option | Operator: pick (a) or (b), and say whether to delete the two plain copies after hub 0.168.0 is live | CC (the dated check) | | **R-1355** | Security & access | P2 | **DooPlex's own Gitea 1.26.2 is inside 19 new notices (2026-11-02, fixed 28.3.0: `GHSA-8p3x` OAuth2 account takeover, `GHSA-37xv` and `GHSA-8g99` public-only tokens reach private data, `GHSA-32h8` Actions secrets) — on the host that holds our code and registry.** Also new on DooPlex: Vaultwarden 1.37.4 (fix 1.37.5), Uptime Kuma 2.5.6 (fix 2.5.9), Paperless 3.3.4 (fix 3.4.0). Felhom changes nothing on DooPlex. Read in `documentation/security-watch/state.json`, group DooPlex. | OPEN — filed 2026-11-03 | the homelab session | add them to `audits/http2shell-2026-10-27/HOMELAB-LIST.md`'s work; which notices reach a self-hosted Gitea with our settings is the homelab session's first read | operator (homelab session) | | **R-779** | Security & access | P4 | **[P3-LOW] Part A's "two outside addresses seen as two" is proven through the simulated tunnel only; on the REAL tunnel the second outside address (ep0, one request allowed) was refused by Cloudflare's edge with 403 and never reached the box.** Measured 2026-10-01 19:51 UTC (`audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt`): no log line on demo-hp; demo-hp's box has no geo restriction in its settings, so a Cloudflare ZONE rule (country or bot, not read) refused a German datacenter address. DooPlex's own address on the real tunnel was seen as itself. **Needs:** one sign-in from a second Hungarian address (the operator's phone off wifi) while DooPlex is locked out — 2 minutes; and say which Cloudflare rule refused ep0. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (a phone), CC reads the logs** **Re-ranked 2026-10-03: P3→P4: a proof gap on the real tunnel; operator-only follow-up.** | — | — | CC + operator | @@ -274,12 +275,12 @@ stopping line that lies. | **R-1248** | Security & access | P3 | **An app's own numbers can lie: an attacker with code inside an app's container can answer the app's API with whatever matches its flood, and the upload watch (R-1244) would then call it explained.** Measured remedy, not built: a per-container upload ceiling with `tc` from a helper container in the app's network namespace — on scratch 9202 (unprivileged) a flood was held at 19.9–20.1 Mbit/s and the app kept answering (`audits/SPIKE-upload-watch-2026-10-23.md` §5). Costs: a helper image with `tc` (a new external image, or `tc` in the controller image), `NET_ADMIN` for that helper, re-applied after each app restart. | OPEN | an operator decision (the new image / capability) | Decide; if yes, CC builds the ceiling for the download and media apps first | operator | | **R-1309** | Security & access | P4 | **What no automated source shows: a fix announced only by mail, forum or a blog we do not read; a security fix nobody calls one (cloudflared 2026.10.2's „bump distroless image"); closed-source apps (Plex, Emby).** The watch (R-1303) shows these only as BEHIND (weekly). The vendor mails the operator receives stay a source. `audits/security-watch-2026-10-26/SPIKE.md` | OPEN — filed 2026-10-26 (a stated limit) | — | none until a vendor offers a feed; add a vendor's feed to `security_watch.FEEDS` when one exists | CC | -## Box system & updates — 18 rows (P3 13, P4 5) +## Box system & updates — 21 rows (P3 16, P4 5) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| -| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 **2026-11-10:** still so. The host clock is 32 s behind DooPlex (it was 14 s on 2026-10-26); `chrony` failed again at the 2026-11-08 05:35 boot with the same line; `NTPSynchronized=no`. Seen in a release run, where the box's log times read half a minute early (`documentation/audits/release-space-fixes-2026-11-10/agent/13-n100-clock.txt`). Not fixed by hand: the N100 takes no hand step in a release run. | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC | -| **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC | +| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` **2026-11-11:** the Clock list measured it again (32.6 s behind, time service not running) and leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CA03-6; class R-1384). | OPEN — filed 2026-10-26 **2026-11-10:** still so. The host clock is 32 s behind DooPlex (it was 14 s on 2026-10-26); `chrony` failed again at the 2026-11-08 05:35 boot with the same line; `NTPSynchronized=no`. Seen in a release run, where the box's log times read half a minute early (`documentation/audits/release-space-fixes-2026-11-10/agent/13-n100-clock.txt`). Not fixed by hand: the N100 takes no hand step in a release run. | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC | +| **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CC02, CC06-6, CC07-6, CC22-6). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC | | **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC | | **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. **2026-11-10 (release run) — part (2) RELEASED, VOUCHED and PROVEN on three test boxes** (agent 0.173.0, vouched by the operator 15:29 UTC; `documentation/audits/release-space-fixes-2026-11-10/agent/`): the folder went 49 → 1, 48 → 1 and 33 → 1 files (about 700 / 690 / 485 MB back) at the commit of THIS update, not at the next one (the earlier sentence here and in the agent's CHANGELOG was wrong, and is corrected there). A rollback after the clean-up worked on Tester 1: a signed update to the older version was crashed once, 9 seconds into the 60-second dwell, and the wrapper put the released version back in the same second; the failed version's file stays until the next good update (by design). The kept older kernel on demo-hp booted to healthy in 313 s and the box went back (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **Still open:** (1) the kernel clean-up as a program (`11-os-updates.md` §5.11.1, not built) and a second box; (3) template archives; (4) the `.bak-*` copies — the release does not reach them; (5) the failed-run logs. A household box gets part (2) at its first good agent update after the vouch. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC | | **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) **2026-11-10 (afternoon) — the runbook is written:** `documentation/runbooks/disk-full.md`, five parts, each hand step marked PROVEN (date and evidence; all on demo-hp) or NOT PROVEN. Still open: one rehearsal of the whole runbook on demo-hp; the NOT PROVEN steps (copies on a former target, template archives, the `.bak-*` binaries, the guest root disk); what fails when the EFI partition or a temp folder is full. | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC | @@ -290,15 +291,18 @@ stopping line that lies. | **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | WATCHING — **2026-10-24 02:4xZ dated check read (weekend-night session, read only):** Tester 1 (8 GB host, guest 4 GB, controller 0.341.0) ran its 03:30 copy at 01:30:00–01:30:13Z (3 apps, 157 MB + 86 MB + 16 KB; no „capped copy container did not start" WARN, no leftover `felhom-tier2-copy-*`); the guest cgroup's memory „full" stall counter on the host (`/sys/fs/cgroup/lxc/9201/memory.pressure`) read **total 444 µs** over the whole 11.1 h since the host started (15:32Z), so „full" stayed far under 20 % (20 % of 10 s = 2 s); 0 OOM kills. **Not shown:** that the copy ran in the capped container (the INFO log does not name it; Docker keeps only ~200 events, from 02:36Z). **Weak load:** 0.24 GB, not the soak's large copy — so the row stays WATCHING; close after a night whose copy moves ≥ 1 GB, read from the same counter. WATCHING — **RELEASED 2026-10-23 in controller 0.340.0** (floor 06:41 UTC; on all three test boxes, start-up „no leftover”, 0 copy containers; Tester 1's manual second-drive run: 3 apps, 0 containers left). **Open only for the consequence:** dated check 2026-10-24 — Tester 1's first night copy ran in the capped container and memory „full” pressure stayed under 20 % (the box's own night log). READY — rides controller 0.340.0. READY — waits for the controller 0.340.0 release. **2026-10-22 night — cause measured, fix built and proven on 9202, waits for the controller release.** Cause: the copy's page cache is charged to the guest and evicts the apps' working set. On 9202 (4 GB RAM, 1.8 GB held by a hog, a 6 GB copy): plain rsync full stall 5 702 / 2 464 ms, peak „full” 16.8 / 6.9 %; the same copy in a 512 MB memory-capped container 374 / 320 ms, 0.9 / 1.0 %. Controller main `03c0a69` + `ae39467` (unreleased, rides 0.340.0): the copy runs capped only for clean `/mnt/` paths with no symlink (security review); a docker error (exit 125) falls back to plain rsync. A real `POST /api/backup/tier2` on rc.4: 6 capped containers, each exit 0; Immich's copy equals its source (dry-run 0 differences, 44 = 44 files). Evidence `audits/night-2026-10-22/P4-R1214/`. **2026-10-23 night (stray-copy fix, night brief Part 1).** The capped copy's container outlived the copy (a timeout or a controller restart killed only the `docker` CLI; `--delete` went on outside the R-1180 guard). Controller main `66a69d4` + `912e592` (unreleased, rides 0.340.0): a fixed name + label per copy, removed by name on any error, every leftover removed at start before the scheduler and the web server, a restore refused while a copy runs or may run; `--mount` (a path the daemon does not see is refused, not created empty) and the image ID with `--pull never` (review C4, C7). Proven on 9202 with rc.5 (`audits/night-2026-10-23/P1-straycopy/`): a restart in the middle of a 16 GB copy → the new controller removed `felhom-tier2-copy-immich-948f2280` at start, the restore probe was refused during the copy, the redone copy equals the source; a 15 s test build → the timed-out copy's container removed, the partial copy stopped growing. A second review of that code (N2–N5) and the commit's own security review (two races) → `9d9e404`, `1babfe9`: a stray clears when a later removal works, the start-up check never removes this process's own copy and holds the destination's lock, one copy per destination at a time, `RunTier2` refuses during a restore for every caller and records itself in one step with that check, every leg asks again. Re-proven on 9202 with rc.8 (`audits/night-2026-10-23/P1-straycopy/1*`). 9202 back on 0.339.0. | — | Release 0.340.0; then re-measure the 03:30 job on the 8 GB soak box and close if „full” stays low | CC | | **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a new install from the public download replaces the bootstrap and the off-site check; `pilot/TESTER2-handover.md`).** | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator | | **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC | -| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement **2026-11-03: the Sunday 2026-11-01 bake was missed (it was in no brief): golden 0.344.0 is 8 days old, the floor 0.349.0; a CODE push to felhom.eu is refused until a bake and the vouch or a dated waiver. The night's close-out was documents only, which the gate exempts; no waiver written. First on the morning list.** | — | — | CC | +| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. **2026-11-11:** the End-dates list leans on this row (`documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB21); the newest golden passed its 7-day limit on 2026-11-10 and no waiver is in force, so the gate refuses code pushes to `felhom.eu`. | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement **2026-11-03: the Sunday 2026-11-01 bake was missed (it was in no brief): golden 0.344.0 is 8 days old, the floor 0.349.0; a CODE push to felhom.eu is refused until a bake and the vouch or a dated waiver. The night's close-out was documents only, which the gate exempts; no waiver written. First on the morning list.** | — | — | CC | | **R-1358** | Box system & updates | P3 | **cloudflared 2026.10.4 misses two dependency fixes (VULN-183456 chi, 2026.10.5; VULN-184686 go-jose, 2026.10.6; the newest is 2026.10.7).** No public text on reach. Not pinned tonight: the tunnel is a box's only door, and 9202 has none to prove it on. | OPEN — filed 2026-11-03 | — | raise `CloudflaredImage` on the next controller release with a tunnel proof on Tester 1 | CC | +| **R-1380** | Box system & updates | P3 | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** Found 2026-11-11 (the End-dates list, class T2, `documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB02-3 to DB02-7). MEASURED: the agent trusts one exact Proxmox certificate (a pin), and the certificate on the three test boxes ends about two years after install — the N100's about 2028-07-16, then demo-hp and Tester 1 within three months. Proxmox makes a new certificate by itself about 14 days before the old one ends (so about 2028-07-02 on the N100). No code path gives the agent the new fingerprint, no alarm names the cause, no runbook holds the hand step, and nobody has tried it. Every box installed in the same season fails in the same fortnight. P3 only because the first date is in 2028; it is certain, not a risk. Decision D76 in `STATUS.md`. | OPEN — filed 2026-11-11 (end-dates list) | — | After D76: the agent reads the certificate file on its own host and takes the new fingerprint (design in `03-host-agent.md` first); prove on a scratch host with a forced renewal. | CC | +| **R-1382** | Box system & updates | P3 | **A box that was off past an end date: nobody knows if it comes back by itself (5 answers).** Found 2026-11-11 (the End-dates list, class T4, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; read in source and documents, never tried). After months without power: the web certificate has ended (DB01-6: does Traefik renew at the first start, and what does the household see until then), Debian's package keys may be too old to verify an update (DB16-6), the tunnel client may be below what Cloudflare accepts (DB19-6), a 30-day restore copy is past its date (DB24-6), and the 14-day countdown after a stopped off-site copy ran out while the box was off (DB04-6). Tester 2 has been off since 2026-10-04 and is the natural test. | OPEN — filed 2026-11-11 (end-dates list) | — | When Tester 2 comes back (or with a scratch host and a moved clock on Tier 0, as a task): record what renews by itself and what needs a hand; put the hand steps in a runbook. | CC | +| **R-1384** | Box system & updates | P3 | **A verdict or an „already ran” check trusts a clock that can be wrong (30 answers).** Found 2026-11-11 (the Clock list, class W2, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED: the N100's host is 32.6 s behind and its time service does not run (R-1301); the other two boxes are in step. Nothing checks a box's clock except one hub alarm for a tunnel handshake stamped in the future. The hub takes the box's own time for backup age, off-site age and proof age (CH02-6, CH07-6, CH07-8, CH08-6); a box with a wrong clock refuses every signed job (CA09-6); a stamp from the future blocks the OS-update leg, the trim and the line test until the clock passes it (CA02-6, CA02-8, CA11-6, CA11-8, CA12-6); the crash guard reads the clock at early boot (CA04-6, CA04-8); a hub clock jump moves its own daily jobs (CH03-6, CH04-6, CH06-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6). The others: CA05-6, CA05-8, CA06-6, CA07-6, CA10-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6. Known parts stay in their rows: R-1189, R-1206, R-1260. Decision D79 in `STATUS.md`. | OPEN — filed 2026-11-11 (clock list) | — | After D79: the hub compares the time in each report with its own, alarms the operator above one minute, and does not take a time from the future as „fresh”. | CC | | **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC | | **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC | | **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC | | **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-27: agent v0.167.0 RELEASED; in the NORMAL signed delivery the pass now runs right after the commit on all three boxes („running the bundle policy pass after the self-update commit (R-1285)” → `BUNDLE POLICY DONE install_mode=appliance removed=0`; `audits/release-2026-10-27/agent-0167-tester1-demofelhom.txt`). Not live: a BYO box coming FROM 0.164.0 (none in the fleet). **Earlier:** **VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:** **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC | -## Monitoring & notifications — 26 rows (P2 3, P3 15, P4 8) +## Monitoring & notifications — 28 rows (P2 3, P3 17, P4 8) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -323,13 +327,15 @@ stopping line that lies. | **R-1296** | Monitoring & notifications | P4 | **When ep0's tenancy script refuses, the hub log shows only its one-line error — PBS's own reason (the 403 `missing Datastore.Audit|Datastore.Backup on …`) stays in the script's stderr, which the hub drops on a refusal.** Found 2026-10-26 diagnosing R-1295: the cause was visible only by reading ep0's proxy journal over root SSH. `hub/internal/tenantsync/client.go` `parseResponse` returns the JSON error and ignores stderr; the script keeps secrets off stderr by design (its header), so the tail is safe to log. | OPEN | — | append the last ~3 stderr lines to the refusal error (bounded, never stdout); a test that a secret on stdout never appears | CC | | **R-1328** | Monitoring & notifications | P2 | **DooPlex's own workloads were not in the security watch at all** — about ten public apps there carry a bad Brotli/Node (HTTP2Shell), on the host that runs the hub, Gitea, the registry and the backups. | **VERIFY** — BUILT 2026-10-27 night (commit only): `scripts/dooplex_inventory.py` lists every `image:` of homelab-manifests (89 images, `documentation/security-watch/dooplex-workloads.txt`; NOT Helm-chart images — said in the file); the watch reads them as the group „dooplex" (operator mails only, own block, „Next: the homelab session"); an image in the private registry is NOT CHECKED with the reason. Tests W12, red-proved. The homelab session's list: `documentation/audits/http2shell-2026-10-27/HOMELAB-LIST.md`. Felhom changes nothing on DooPlex. | — | the first CI night shows the DooPlex block; the homelab session works its list. 2026-10-28: `minio/minio:latest` answers HTTP 401 on Docker Hub (the image is no longer published there) — for the homelab session: the pin cannot be checked anonymously any more | CC | | **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | WATCHING — **released 2026-10-29:** agent 0.168.0 (vouched 10:21:56 CET) + controller 0.347.0 (the global floor). Proven on the release on Tester 1: a hog in a 128 MB Gokapi — host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail (`audits/d63-2026-10-29/run-p4-oom.log`; the repeated WARN is R-1341). Closes at the dated check 2026-11-12 if it holds. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC | -| **R-1361** | Monitoring & notifications | P2 | **An event the box sends while the hub cannot be reached is lost: 3 tries over 9 s, then dropped.** Walked 2026-11-03 on Tester 1 (hub blackholed 13 min): the box ran normally, an app install worked, a crash loop stopped the app and held it, and `app_stopped_unhealthy` and `app_held` failed after 3 attempts (`notifier.go`). Afterwards NOTHING was re-sent: the hub's log shows no event from the outage, its „Held apps" had no row (`a row existed: false`), and the hold reporter had already recorded the app as reported (`holdreport.go`, the push is asynchronous) so only its periodic re-send can repair it. Any DooPlex restart (2026-11-01 was one) can swallow a household's and the operator's mails this way. `audits/security-2026-11-02/part5/outage-hub.log` | OPEN — filed 2026-11-03 **2026-11-03, a second cost seen:** the walk's cut (01:17–01:39 UTC) covered Tester 1's 01:30 dump, its `db_dump_completed` was lost, and at 04:00 UTC the hub mailed `expected_dbdump_missed` „No DB dump completed or failed since midnight", while the box's own report read `last_db_dump` 03:15 UTC. A lost event becomes a false alarm. | — | a small on-disk outbox for events, sent in order when the hub answers (a design: which types, how long kept); the hold reporter marks „reported" only after a 2xx. Test red first | CC | +| **R-1361** | Monitoring & notifications | P2 | **An event the box sends while the hub cannot be reached is lost: 3 tries over 9 s, then dropped.** Walked 2026-11-03 on Tester 1 (hub blackholed 13 min): the box ran normally, an app install worked, a crash loop stopped the app and held it, and `app_stopped_unhealthy` and `app_held` failed after 3 attempts (`notifier.go`). Afterwards NOTHING was re-sent: the hub's log shows no event from the outage, its „Held apps" had no row (`a row existed: false`), and the hold reporter had already recorded the app as reported (`holdreport.go`, the push is asynchronous) so only its periodic re-send can repair it. Any DooPlex restart (2026-11-01 was one) can swallow a household's and the operator's mails this way. `audits/security-2026-11-02/part5/outage-hub.log` **2026-11-11:** the Clock list's class „nobody is told” (R-1383) is beside this row; it does not repeat it. | OPEN — filed 2026-11-03 **2026-11-03, a second cost seen:** the walk's cut (01:17–01:39 UTC) covered Tester 1's 01:30 dump, its `db_dump_completed` was lost, and at 04:00 UTC the hub mailed `expected_dbdump_missed` „No DB dump completed or failed since midnight", while the box's own report read `last_db_dump` 03:15 UTC. A lost event becomes a false alarm. | — | a small on-disk outbox for events, sent in order when the hub answers (a design: which types, how long kept); the hold reporter marks „reported" only after a 2xx. Test red first | CC | | **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC | | **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) | | **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC | +| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. | OPEN — filed 2026-11-11 (end-dates list) | — | After D77: the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC | +| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. Decision D78 in `STATUS.md`. | OPEN — filed 2026-11-11 (clock list) | — | After D78: one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC | | **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC | -## Hub & operator — 16 rows (P2 1, P3 4, P4 11) +## Hub & operator — 17 rows (P2 1, P3 5, P4 11) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -344,11 +350,12 @@ stopping line that lies. | **R-1002** | Hub & operator | P4 | **A box the household switched off reads ONLINE on the hub's hosts page for its first 45 minutes** (seen live 2026-10-13, R-998 proof, `audits/poweroff-2026-10-13/B-hub-watch.log`: the „Power: Kikapcsolva…" line at 18:53, the KIKAPCSOLVA badge only at 19:38, when the host report turned stale). The customer page read „warn" in part of that gap (seen, not captured). No alarm and no mail fired; the operator page is only slow to say it. | **READY** — filed 2026-10-13 | — | CC: let `hostDisplayStatus` read `deliberate_off_at` before staleness, with a render test per state; next hub release | CC | | **R-1056** | Hub & operator | P4 | **Five operator routes fit no token group, so they are session only until the operator places them:** apps reset-telemetry and dismiss-issues, the Configuration asset re-seed, the off-site WireGuard endpoint save, the off-site abandon-cancel (`05` §17.3). Nothing needs them by token today. | WAITING-ON-OPERATOR — filed 2026-10-15 by the token build | operator decision | Place each in a group, or leave session only | operator | | **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator | -| **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC | +| **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CH23-6). | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC | | **R-1345** | Hub & operator | P4 | **An access token's scopes cannot be changed after it is made — widening or narrowing one means a new token (and a new secret).** The operator, 2026-10-29: to give CC `hosts:read` + `hosts:write` for the live proofs he had to make `CC_HOSTS` (now in `sec`) beside `cc-releases-2`. Wanted: edit a token's scopes in the session (an audit line per change), so a scope can also be taken away after a session without retiring the secret. | OPEN | — | design + build the scope edit (session only), red-first test that an unticked scope is refused at once | CC | | **R-1353** | Hub & operator | P4 | **Statements still over 1 s, named by the new slow-statement line (hub 0.178.0) in its first hour:** a background deadline/restore-test check reading days of full host-report JSON (`GetHostReportsSince`, `store.go`; 1.5–9.2 s cold, from `monitor/deadline.go` and `monitor/restoretest.go` — no page waits on it); the start-up prune of `events` and `app_log_issues` (1.1 s and 1.8 s; no index on `created_at` / `last_seen`); the customer page's newest-report read (`GetCustomer`, 1.3 s cold in the pod probe). With the 1 GiB limit most of this should come from the cache once warm. Evidence: the hub log 2026-10-30 09:22–10:05 UTC; `audits/release-2026-10-30/p2-r1346/`. | OPEN **2026-11-02 read (`audits/security-2026-11-02/part4/hub-slow-lines-2026-11-01-02.txt`; the log covers only the 31 h since DooPlex restarted on 2026-11-01 16:11 CET, the lines from 10-30 are gone with the old pod):** 35 lines, 15 statements. What repeats: `VACUUM INTO` 5× (avg 110 s, the 6-hourly snapshot, expected); `GetHostReportsSince` 9× (avg 7.9 s, max 26.6 s); `GetCustomer`'s newest report 3× (1.1 s). Once, at the cold start: the prune of `events` (10.4 s) and `app_log_issues` (21 s, again 04:30), the OS-release and host lists (2–22 s). No fix before the full week (11-06). **2026-11-10 read (the hub's start after a deploy, `audits/release-space-fixes-2026-11-10/hub/10-watch.txt`):** 11 slow lines in the first 2 minutes of the new pod, the longest 19.1 s (`GetHostReportsSince`), then the OS-report reads (3–4 s) and the two prunes (2.7 s, 4.2 s); none in the 28 minutes after. The 13:02 UTC snapshot's `VACUUM INTO` took 127.8 s. The same statements as the 11-02 read; no new one. | — | read a week of slow-statement lines; index or narrow what still repeats (the history read needs only a few fields, not the whole JSON) | CC | | **R-1375** | Hub & operator | P2 | **The outside part of three endings is done by hand or not at all — the worst: an off-site account can stay at the provider with no record in the hub.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH06, EH01, EA21 and the ending "a tester ends"; read in source and documents; no account was called). (1) **Unticking the off-site tier** of a household takes nothing down and leaves no reminder (`hub/internal/web/configs.go:1737-1743`). A later RESET takes the account down only when the tier is ticked (`hub/internal/web/customer_reset.go:203-245`), so the sub-account and its data stay, are paid for, and the hub holds no record that they exist. (2) **Switching the second off-site tier off** takes nothing down: storage entry, token and copies stay; the source calls the teardown 'a deliberate future op' (agent `internal/pbsdr/manager.go:274`). (3) **A tester ends**: no handler and no runbook; done by hand once (2026-09-25); no list says what to collect. **Members (answers marked N):** EH06-4, EH01-4, EA21-4, and 'a tester ends' in the table's head. Related: R-810 (a household leaves), R-32. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D71 = yes, D74 = yes; `09` §3 decisions 291, 294).** Not built. | — | Hub (D71): keep the account record after an untick; RESET and delete take the account down, ticked or not. Docs (D74): the runbook 'a household or a tester leaves', from the five code doors, in a later documents-only session. Parts (2) and (3) of this row have no ruling of their own. | CC | | **R-1376** | Hub & operator | P3 | **Records of a thing that is gone stay in six places.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EH00, EH02, EH21, EA00, EA11, EA27; read in source, `documentation/audits/endings-list-2026-11-10/evidence/hub-delete-statements.txt`; no hub database was read). (1) **Hub:** thirteen tables have no delete statement anywhere. Deleting a box leaves its update settings, its update reports and its kernel notices; deleting a household leaves its off-site key rows, off-site window rows and abandon requests. Most are small; the update reports of each box grow with time. (2) **Host uninstall** leaves the mount units and host mounts of household drives (said when R-179 closed; no open row carried it). (3) **A drive wiped by a signed job** keeps a record and a mount unit that name the old filesystem (agent `internal/signedjobs/wipe.go:61-110`). (4) **The token of a destroyed guest** stays valid until a new guest takes its number: the token store has no remove (agent `internal/localapi/tokenstore.go:127-210`). **Members (answers marked N):** EH00-4, EH02-4, EH21-4, EA00-4, EA11-4, EA27-4. | OPEN — filed 2026-11-10 (endings list) **2026-11-10 15:19 — RULED (D73 = yes, with one exception; `09` §3 decision 293).** Not built. | — | Hub (D73): add the tables with no delete to the two purges and to the daily clean-up; the delete journal (`host_deletions`, `customer_resets`) stays on purpose, and the privacy text says so. Agent: remove a guest's token when the guest is destroyed. Parts (2) and (3) have no ruling of their own. | CC | +| **R-1386** | Hub & operator | P3 | **Timed jobs with no time limit and no measured duration (25 answers).** Found 2026-11-11 (the Clock list, class W4, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source; the hub was not measured, by ruling). The hub's 60-second sweep runs its checks one after the other with no limit for each: one slow check delays all the others, the liveness check too (CH19-3, CH19-8, CH01-3, CH08-3, CH08-8, CH10-3, CH13-8, CH18-3, CH06-8, CH02-3); the 07:10 audit has 10 minutes, measured with three boxes only (CH11-3, CH11-8). On the box: the interval jobs have no limit of their own (CC11-3, CC12-3, CC13-3, CC14-3, CC15-3, CC16-3, CC20-3, CA06-3, CA10-3, CA10-8, CA13-8, CA15-3, CA19-3). Every measured duration is from a small test store. | OPEN — filed 2026-11-11 (clock list) | — | Hub: log each sweep step's duration and alarm when one sweep takes longer than its interval; then read a week of numbers and set limits. Box: the same for the controller's interval jobs. | CC | ## Business & legal — 14 rows (P2 4, P3 2, P4 8) @@ -367,9 +374,9 @@ stopping line that lies. | **R-930** | Business & legal | P3 | **The site's primary call-to-action fails WCAG AA: white on `--blue` #0083D8 measures 4.01:1 at 16px/600, against a 4.5:1 floor.** MEASURED 2026-10-10 in the browser, on the LIVE home page and on the redesign preview: identical, so this is **pre-existing and site-wide**, not a regression of the redesign. It is the „Jelentkezem tesztelőnek" button and every `.cta-button` on every page. `.cta-button-secondary` is fine (7.17:1). Three fixes, all needing the operator because each touches brand: (a) darken `--blue` until the ratio clears 4.5 (changes a token used across hub, controller and website); (b) make the label 700 weight at 18.66px+, which moves it into WCAG's „large text" band where 3:1 applies and 4.01 passes, at the cost of a bigger button everywhere; (c) accept it and record the decision. **Not changed unilaterally**: `felhom-web-design` §6 puts a new accent colour behind the operator's word. | **WAITING-ON-OPERATOR** — filed 2026-10-10 by the home-page redesign session; evidence `audits/redesign-2026-10-10/AUDIT.md` | operator's choice between (a), (b), (c) | pick one of the three | operator | | **R-989** | Business & legal | P4 | **Operator: read `documentation/audits/selfhst-survey-2026-10-12/FELHOM.md` §6 and rule on each recommendation.** READ-ONLY research 2026-10-12: the selfh.st self-hosting survey 2023–2025 (1,869 / 3,655 / 4,081 respondents, hobby users) against our catalog, features, pitch and hardware. Ten ranked recommendations, none acted on; the top three: a supported-hardware page (none exists — finding, §4), off-site backup as a *why* reason in the pitch (only 25.2 % of the crowd backs up off-site), Syncthing as the largest catalog gap (25.9 %). Tables: `TABLE.md`; method and limits: `METHOD.md`. | **WAITING-ON-OPERATOR** — filed 2026-10-12 | the operator's reading | Operator: rule on each of the ten (yes / no / later). If nothing is done: nothing changes; the research ages (the 2026 survey replaces it) | operator | | **R-990** | Business & legal | P4 | **Operator: read `documentation/audits/forum-research-2026-10-12/FELHOM.md` §5 and rule on each recommendation.** READ-ONLY research 2026-10-12: three years (#7943–#11092, 2,908 posts, 195 people, 82 beginner threads) of the public PROHARDVER! topic „Home server / házi szerver építése”, tagged and counted by person (`FINDINGS.md`, method and spot-check in `METHOD.md`; no username or post text in git). Ten ranked recommendations, none acted on; the top three: a „which machine works” page (8 of 33 beginners who name their hardware own only a Synology, a router disk or a Raspberry Pi — the installer cannot run there; the same gap is R-989's first recommendation from a different source, and the shipped-page half is R-991), „you do not choose an operating system — we do” (the biggest beginner question, 20 of 82 threads), and „bring your old office PC” as the cheap entry. Catalog gap (≥ 3 people): a DLNA server (7), an ad/DNS blocker (5), a camera recorder (5). | **WAITING-ON-OPERATOR** — filed 2026-10-12 | the operator's reading | Operator: rule on each of the ten (yes / no / later). If nothing is done: nothing changes; the research ages | operator | -| **R-1035** | Business & legal | P4 | **Did the „felhő tárhely” page earn anything? Two dated checks.** **Baseline (reported by the operator from Search Console, 2026-10-14, last 3 months):** 1 click, 48 impressions, CTR 2.1 %, average position **73.6**; „felhő tárhely” 19 impressions, „felhőtárhely” 5, „felhő alapú tárhely” 3. CC measured separately that the phrase „felhő tárhely” appeared **0 times** on the home, FAQ and backup pages — Google was showing the home page for those queries by accident. `/sajat-felho-tarhely` + `/en/own-cloud-storage` published **2026-10-14**; full baseline, price sources and the power provenance in `audits/seo-2026-10-14/BASELINE.md`. **What to measure, both times:** impressions and average position for „felhő tárhely” and „saját felhő tárhely”, plus the new page's own impressions, clicks and position. **A null result at +4 weeks means nothing on its own** — a new page on a site at position 73.6 takes weeks to months to move, and „felhő tárhely” alone is held by Microsoft, Google and the big list sites; the page aims at the two longer phrases. That is why there are two checks and not one, and why neither should be read as a verdict on the copy. **There is no Search Console connector in this project, so CC cannot read these numbers** — the operator reads them, CC records and compares. **Blocked until the operator asks for indexing** (Search Console → URL-ellenőrzés → the URL → „Indexelés kérése”): until that happens, or Google finds the page itself, both checks read zero for a reason that is not the page. **This is a slow channel and does not replace talking to people.** | **WATCHING** | operator: request indexing, then supply the Search Console read | 2026-11-11: first read (+4 weeks); 2026-12-09: second (+8 weeks), compared with the first | operator + CC | +| **R-1035** | Business & legal | P4 | **Did the „felhő tárhely” page earn anything? Two dated checks.** **Baseline (reported by the operator from Search Console, 2026-10-14, last 3 months):** 1 click, 48 impressions, CTR 2.1 %, average position **73.6**; „felhő tárhely” 19 impressions, „felhőtárhely” 5, „felhő alapú tárhely” 3. CC measured separately that the phrase „felhő tárhely” appeared **0 times** on the home, FAQ and backup pages — Google was showing the home page for those queries by accident. `/sajat-felho-tarhely` + `/en/own-cloud-storage` published **2026-10-14**; full baseline, price sources and the power provenance in `audits/seo-2026-10-14/BASELINE.md`. **What to measure, both times:** impressions and average position for „felhő tárhely” and „saját felhő tárhely”, plus the new page's own impressions, clicks and position. **A null result at +4 weeks means nothing on its own** — a new page on a site at position 73.6 takes weeks to months to move, and „felhő tárhely” alone is held by Microsoft, Google and the big list sites; the page aims at the two longer phrases. That is why there are two checks and not one, and why neither should be read as a verdict on the copy. **There is no Search Console connector in this project, so CC cannot read these numbers** — the operator reads them, CC records and compares. **Blocked until the operator asks for indexing** (Search Console → URL-ellenőrzés → the URL → „Indexelés kérése”): until that happens, or Google finds the page itself, both checks read zero for a reason that is not the page. **This is a slow channel and does not replace talking to people.** **2026-11-11:** the 4-week check came due today in a run with no operator present; only the operator can read Search Console, so the date moved to 2026-11-18 (the gate's rule allows a move with a reason). The 8-week check keeps its date. | **WATCHING** | operator: request indexing, then supply the Search Console read | 2026-11-11: first read (+4 weeks); 2026-12-09: second (+8 weeks), compared with the first | operator + CC | -## Process & tooling — 32 rows (P3 8, P4 24) +## Process & tooling — 34 rows (P3 8, P4 26) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -415,8 +422,10 @@ stopping line that lies. |---|---|---| | R-1293 | 2027-03-29 | the spring clock change: the hub's 02:00 snapshot and the 02:30 push on 2027-03-28 (02:00–03:00 does not exist) — which file each push took; detail in the R-1293 row | | R-1050 | 2026-11-15 | HSTS one year if no https trouble since 2026-10-15; detail in the R-1050 row | -| R-1035 | 2026-11-11 | „felhő tárhely” + „saját felhő tárhely” impressions and average position, and the new page's own impressions/clicks/position, 4 weeks after publishing; operator supplies the read; detail in the R-1035 row | +| R-1035 | 2026-11-18 | „felhő tárhely” + „saját felhő tárhely” impressions and average position, and the new page's own impressions/clicks/position, 4 weeks after publishing; operator supplies the read; detail in the R-1035 row | | R-528 | 2026-11-12 | no failed oom_check / unexplained app_oom since 2026-10-12; detail in the R-528 row | | R-1035 | 2026-12-09 | the same three numbers 8 weeks after publishing, compared with the 4-week read; a null at 4 weeks is not a verdict, both reads are needed; detail in the R-1035 row | | **R-1297** | Process & tooling | P3 | **CI waits dominate a working day: felhom.eu's CI run takes ~13 min and the ONE runner makes every repo's job wait behind it (up to 10–17 min queued).** Measured 2026-10-26 (operator: „these CI waiting times are getting out of hand"): since 10-25 felhom.eu runs median 793 s, the controller/agent/catalog 57–102 s; queue median 5 s, max 548–1023 s. In run 2366, ~10 of 13 minutes are four suites — `test_gate_decoys.py` 232 s, `test_due_checks_gate.py` 133 s, `test_repo_gates.py` 132 s (it runs the whole gate set a SECOND time), `dooplex-offsite/test_dooplex_offsite.py` 69 s — plus `hub-gofmt` ~110 s (a cold Go cache per run). **Ruled 2026-10-26 by the operator: BOTH** — (A) trim felhom.eu CI: the heavy suites only when their own files change, no duplicate full run, a Go cache for hub-gofmt; (B) a second CI runner on DooPlex. | **OPEN — (A) BUILT 2026-10-26 evening, closes after the first nightly run (2026-10-27 03:20 UTC) is read.** Both runners proven at once (jobs 2374 + 2375 on `felhom-gates-runner` / `-2`, 17:11:41Z). felhom.eu CI, one push each: **(a) docs only** 810 s (old runner) → 351 s (2 CPUs) → see the close-out; **(b) a gate script** 351 → 203 s (`ecd82aba`, job 2376); **(c) a hub change** 351 → 118 s (`7e57e824`, job 2378). Heavy suites by path + tag + nightly; `test_repo_gates.py` and due-checks group G test the wiring (130 s each → under 3 s). Decoys through CI's entry point with the filter ON: a docs-only register break → red; an inert gate → GREEN at first (a decoy hole, fixed in `e33ec821`) → red. No Go cache: CI has no Go (the 110 s was security-scan, now 27 s). `audits/r1297-ci-trim-2026-10-26/`. (B) DONE by the operator (`09` §3 decision 267). | — | (A) CC, a run of its own (a CI scope change, ruled); (B) a DooPlex change — a brief or the homelab session; and CC waits on CI only before a release or at the end of a run | CC | +| **R-1381** | Process & tooling | P4 | **No document says when eleven things end, or where to read the date (11 answers).** Found 2026-11-11 (the End-dates list, class T3, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; search S18 found no document that lists end dates). OS support for Debian 13 and Proxmox 9 (DB17-2, DB17-3), the tunnel client's oldest allowed version (DB19-2), the installer ISO (DH19-3), and the outside accounts and tokens (DH23-2, DH24-2, DH27-2, DH28-2, DH29-2, DH31-2, DH32-2). This run could not read these dates (no account call, by its fences). The operator reads them once: the table „Only you can read these dates” in `documentation/audits/end-dates-list-2026-11-11/README.md`. | OPEN — filed 2026-11-11 (end-dates list) | — | When the operator has read the dates: write them in one place (a short table in `documentation/operations/`), and give the due-checks block the ones inside a year. | CC | +| **R-1387** | Process & tooling | P4 | **A document and the source disagree on a time, in four places (4 answers).** Found 2026-11-11 (the Clock list, class W5, `documentation/audits/clock-list-2026-11-11/TABLE.md`). (1) `documentation/architecture/03-host-agent.md` names `Restart=always` and a watchdog for the agent's unit; the unit has `Restart=on-failure` and no watchdog (CA18-6, CA18-8). Which one is the design is not stated, so this session did not edit either. (2) The hub-database copy alarm: 13 hours in one place, 26 in another (CH23-7; a DooPlex procedure, listed for the operator, not edited). (3) Old „24 h / 02:30” text for the hub's daily job (CH17-6). **Seen beside the lists, same kind, not in the count:** (4) the comment at `felhom-controller/controller/cmd/controller/main.go:1412` says 14 days where the code keeps restore copies 30 days (D62); (5) `documentation/architecture/07-backup-architecture.md:455-457` cites `main.go` lines that have moved; (6) `documentation/operations/tailscale.md` says „both nodes” where three test hosts exist, and Tailscale was not running on demo-hp and Tester 1 on 2026-11-11 (not known if that is meant); (7) the threat model and the recovery inventory do not say the same about how often the hub bearer is rotated. | OPEN — filed 2026-11-11 (clock list) | — | Operator or a task: say which side is right for (1) and (2); then one docs commit. (3) is a text fix in the same commit. | CC | diff --git a/documentation/backlog/STATUS-HISTORY.md b/documentation/backlog/STATUS-HISTORY.md index 8cd51988..80f75efe 100644 --- a/documentation/backlog/STATUS-HISTORY.md +++ b/documentation/backlog/STATUS-HISTORY.md @@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S **Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.** +## Moved from STATUS on 2026-11-11 (midday) + +**Updated 2026-11-10 (evening).** Three releases are live: the agent that cleans its own update folder (you vouched it), the controller that sends one warning for one full disk (every box takes it by itself), and the hub that sends the host-disk alarm to you only. You ruled D71–D75 at 15:19 and none is built; the smallest system disk is 200 GB, and what the installer does below 200 GB (refuse, or warn) is open (R-1370). The report of the run is `REPORT-release-space-fixes-2026-11-10.md`. + ## Moved from STATUS on 2026-11-10 (evening) **Updated 2026-11-10 (late afternoon).** You ruled D71–D75 at 15:19 (`documentation/architecture/09-update-architecture.md` §3, decisions 291–296); none is built. The smallest system disk is 200 GB, and what the installer does below 200 GB (refuse, or warn) is open (R-1370). The report of this run is `REPORT-release-space-fixes-2026-11-10.md`.