Files
felhom.eu/documentation/audits/clock-list-2026-11-11/TABLE.md
T
admin 978666dde3
gates / gates (push) Successful in 2m2s
lists: the Clock list (70 jobs) and the End-dates list (39 things), 9 register rows, 4 decisions
Read-only review of every job on a timer and of everything that ends on a date,
with measurements from the three test boxes. Nothing ends in the next 14 days.
Register 236 -> 245 (R-1379..R-1387), pointer lines on twelve rows, D76-D79 in
STATUS.md. Part C: felhom-agent's clone address holds no token now; the other
clones wait for the operator's yes. R-1035's 4-week check moved to 2026-11-18
(operator read needed).

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-11 11:54:42 +01:00

1257 lines
146 KiB
Markdown

# The Clock list — the table
Built 2026-11-11 from the source of the four repositories and from read-only measurements on the three test boxes
(Tester 1, demo-hp, the N100; host and guest). One block for each job that a clock starts: the night steps on a box,
the interval jobs, the watchdogs, the hub's daily jobs and its 60-second sweep, and the timers on DooPlex and ep0.
`tools/make-table.py` writes this file from the helpers' answers (`evidence/helpers/`) and from `tools/marks.py`; do
not edit it by hand.
**NOT MEASURED, by the operator's ruling:** DooPlex and ep0, and the hub's database. What this table says about the
hub's jobs and about the timers on those two machines comes from the repository's own source, scripts and documents.
**The eight questions:** 1 what runs, who owns it, what starts it (with `CUSTOMER`, `TEST-ONLY` or `OPERATOR`) ·
2 when, how often, in what order · 3 how long it takes (measured where a box shows it), its time limit, and 10 or 100
times the data · 4 what happens if it does not run · 5 it runs late, or twice, or meets another job (the lock is
named) · 6 the clock is wrong or jumps · 7 who is told, and when · 8 how do we know (`MEASURED`, a test, `SOURCE`, a
document, or nobody knows). A ninth line, outside the marks: is the hand step written in a runbook?
**Every measured duration comes from a small test store** (three boxes, a few apps, one night of controller logs).
No number in this table says how long a job takes on a full household disk.
**Marks:** `A` answered, with a pointer · `D` decided, not built · `K` known gap, with its row · `N` nobody asked — it
survived the searches in `EV/searches.txt` (S01…S25; controls in the same file). Every `N` names its class row:
**W1** = R-1383 a job does not run, gives up, or loses its state at a restart, and nobody is told · **W2** = R-1384 a
verdict or an "already ran" check trusts a clock that can be wrong · **W3** = R-1385 two jobs can meet with no lock and
no order · **W4** = R-1386 no time limit and no measured duration · **W5** = R-1387 a document and the source disagree
on a time.
**Who read what.** A cell whose mark is `K` or `N`, and every `A` that `tools/marks.py` carries, holds the main
session's words. Where its pointer was opened by eye, the pointer is written out in `tools/marks.py`; where
`tools/marks.py` shows `+`, the pointer is the helper's and `tools/resolve-helper.py` opened it (the file and the line
exist, and the name the helper gave stands at that line). Every other cell holds a helper's words with the mark `A`.
The main session read every helper answer to questions 4 to 7; it did not open each of their pointers. Treat those
cells as a map, not as proof.
**Pointers:** `AG/` = `felhom-agent/` · `CT/` = `felhom-controller/controller/` · `CAT/` = `app-catalog-felhom.eu/` ·
`HUB/` = `felhom.eu/hub/internal/` · `E/` = `felhom.eu/` · `AR/` = `felhom.eu/documentation/architecture/` ·
`RB/` = `felhom.eu/documentation/runbooks/` · `EV/` = this folder's `evidence/`. `tools/check-pointers.py` prints the
text behind each one.
**Named status** (does an architecture document name this job?): the status comes from a search of the architecture
folder: 0 lines = `UNNAMED`, else `MENTIONED`. It is a machine count, not a reading.
## The kinds, and where each job is
| Kind | Jobs (worst mark) | Named · mentioned · named nowhere |
|---|---|---|
| Night | CA01 (K), CA02 (N), CA03 (K), CA05 (N), CA20 (N), CC01 (A), CC02 (N), CC03 (K), CC04 (N), CC05 (N), CC06 (K), CC07 (N), CC08 (K), CC09 (N), CC19 (N), CC22 (K) | 0 · 16 · 0 |
| Interval | CA08 (A), CA09 (N), CA11 (N), CA12 (N), CA13 (N), CA14 (A), CA19 (N), CA21 (A), CC10 (A), CC11 (N), CC12 (N), CC13 (N), CC14 (N), CC15 (N), CC16 (N), CC17 (A), CC18 (N), CC20 (N), CC21 (N) | 0 · 18 · 1 |
| Watchdog | CA04 (N), CA06 (N), CA07 (N), CA10 (N), CA15 (N), CA16 (A), CA17 (A), CA18 (N) | 0 · 8 · 0 |
| Hub daily | CH02 (N), CH11 (N), CH16 (N), CH17 (N), CH18 (N) | 0 · 5 · 0 |
| Hub sweep | CH01 (N), CH03 (N), CH04 (N), CH05 (N), CH06 (N), CH07 (N), CH08 (N), CH09 (N), CH10 (N), CH12 (A), CH13 (N), CH14 (N), CH15 (A), CH19 (N), CH20 (N), CH21 (A), CH22 (A) | 0 · 17 · 0 |
| DooPlex and ep0 | CH23 (N), CH24 (N), CH25 (N), CH26 (N), CH27 (N) | 0 · 5 · 0 |
## Not in the table
**Tiny (named once, not tabled):** session clean-up of dashboard sign-ins on the box (a new sign-in repairs it) · old image clean-up after a controller start (log only; deletes nothing when it cannot read its keep list) · operator restart and reboot limits on the agent (one hour; an operator action) · the hub's purges of log bundles, access log and sessions (nobody is told, nothing depends on them) · the hub's version check and catalog template fetch (a WARN line and a stored error; a page shows it) · the guest swap check, once after an agent start (2-minute limit) · the 14-day hold of alarms after a deliberate power-off (a window, not a job).
**Left out for another reason:**
- The agent's local-API certificate (10 years) — it is an end date: the End-dates list, DB03.
- Links and tokens that end by themselves — they are end dates: the End-dates list, DH22 and the by-ruling list there.
- The 14-day countdown after a household stops the off-site copy — the End-dates list, DB04; its daily sweep is CC19 here.
- Debian's and Proxmox's own timers on a host (apt-daily, pve-daily-update, logrotate, fstrim.timer, man-db) — they are the OS's jobs, not ours; MEASURED present on the three hosts, `evidence/box/*-host-clock.txt`.
- TEST-ONLY jobs — none was found: every job in the table runs on a household's box or for the operator.
---
### CA01 — Whole-guest backup: due-check and run · CUSTOMER · Night · MENTIONED
architecture folder: 210 lines in 20 files match `/vzdump|whole.guest|due.check/` (a machine count, not a reading) · helper section: clock-agent „1. Whole-guest backup: due-check and run”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The in-guest controller asks `handleBackupDue` and starts `handleBackup`; the host has no timer for it; the agent runs the vzdump. CUSTOMER. | A | `AG/internal/localapi/server.go:1134-1217` `AG/internal/localapi/server.go:878-1015` |
| 2 | Due when the newest archive's age reaches the cadence, `age >= tier.Cadence`; default `BackupCadence` 24 h; the night window is the controller's; the agent compares ages only, no time zone. | A | `AG/internal/localapi/server.go:1204-1213` `AG/internal/config/config.go:576-580` `AR/07-backup-architecture.md:551` |
| 3 | Task wait `defaultPrimaryTierWaitTimeout` 30 min and `defaultExtraTierWaitTimeout` 12 h; outer bound is the tier wait plus 15 min; the copy grows with the guest's size. | A | `AG/internal/config/config.go:464-465` `AG/internal/localapi/server.go:971-975` |
| 4 | A tier not backed up stays due at every later ask; the controller has a 48 h safety valve; the hub judges 72 h without a whole-guest backup. | A | `AR/07-backup-architecture.md:551-552` `AR/07-backup-architecture.md:579-581` |
| 5 | Same tier in flight answers 202 with the same job; another tier answers 409; the host-wide gate is `TryAcquire`, in memory. | A | `AG/internal/localapi/server.go:913-917` `AG/internal/localapi/server.go:919-925` `AG/internal/backup/inflight.go:40-59` |
| 6 | The age is wall-clock time. A host clock that was ahead during one backup gives a negative age afterwards; the tier reads 'not due' until real time passes it. | K | `AG/internal/localapi/server.go:509` `AG/internal/localapi/server.go:1176-1182` · R-1189 |
| 7 | The hub raises `expected_backup_missed` to the operator; a flood refused by the cap raises an operator event, rationale. | A | `AR/07-backup-architecture.md:579-582` `AG/internal/localapi/backup_cap.go:12-35` |
| 8 | TEST and DOC for the due-check. MEASURED 2026-11-11: demo-hp 11 to 13 minutes, Tester 1 about 3 minutes, the N100 under 1 minute; on the N100 the start time moved 5 minutes later each day. A clock jump has no test. | K | `EV/box/demo-hp-host-night.txt` `EV/box/tester-1-host-night.txt` `EV/box/demo-felhom-host-night.txt` · R-1189 |
**Hand step in a runbook?** no hand step
### CA02 — OS-update night leg: guest, host, Docker, Proxmox · CUSTOMER · Night · MENTIONED
architecture folder: 51 lines in 10 files match `/os.?update|OS update/` (a machine count, not a reading) · helper section: clock-agent „2. OS-update night leg (guest, host, Docker, Proxmox)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The agent runs it after a successful primary, non-manual whole-guest backup: `SetAfterPrimaryBackup`, trigger at; `Run`. CUSTOMER. | A | `AG/cmd/felhom-agent/main.go:1390-1398` `AG/internal/localapi/server.go:1008-1011` `AG/internal/osupdate/leg.go:538-586` |
| 2 | 90 s after the backup ends; at most once per `MinGap`, default 20 h; order guest, host, Docker, Proxmox, kernel; no clock slot of its own. | A | `AG/cmd/felhom-agent/main.go:1395` `AG/internal/osupdate/leg.go:656-667` `AG/internal/osupdate/leg.go:538-586` `AR/11-os-updates.md:378-387` |
| 3 | Wrapper limits: host command 600 s and guest command 1800 s AG/configs/felhom-os-apply:279-284; health wait default 5 min; time grows with the number of packages; measured 23 to 70 s. | A | `AG/internal/osupdate/leg.go:810-813` `AR/07-backup-architecture.md:449-450` |
| 4 | A failed or missed backup means no OS leg that night; no catch-up found; the hub shows how far behind a box is. | A | `AR/07-backup-architecture.md:447` `AR/11-os-updates.md:406-408` |
| 5 | It runs holding the heavy-op gate taken by the backup; one pass at a time by flock `tryLockPass`; a second night run inside 20 h is skipped. | A | `AR/07-backup-architecture.md:444-446` `AG/internal/osupdate/unsent.go:35-52` `AG/internal/osupdate/leg.go:663` |
| 6 | The 20-hour gap compares the clock with a time read from a file. A stamp that was written while the clock was ahead keeps the leg skipped until real time passes that stamp plus 20 hours. | N | `AG/internal/osupdate/leg.go:661-663` · class W2 · S08 |
| 7 | The result is posted to the hub by `finish`; operator gets a hub event per update and failure, the household one timeline line. | A | `AG/internal/osupdate/leg.go:922-939` `AR/11-os-updates.md:406-423` |
| 8 | TEST and DOC for the normal path. A stamp from the future: NOBODY KNOWS. | N | `AG/internal/osupdate/leg_test.go:367` `AG/internal/localapi/afterbackup_test.go:17` `AR/11-os-updates.md:366-387` · class W2 · S08 |
**Hand step in a runbook?** no hand step
### CA03 — Kernel lane: night reboot, 20-minute boot judge, one self-revert · CUSTOMER · Night · MENTIONED
architecture folder: 165 lines in 14 files match `/kernel/` (a machine count, not a reading) · helper section: clock-agent „3. Kernel lane: night reboot, 20-minute boot judge, one self-revert”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent: `runKernel` ends the night leg; at every agent start `KernelAfterBoot` judges the boot. CUSTOMER (the box restarts at night). | A | `AG/internal/osupdate/kernel.go:119-178` `AG/cmd/felhom-agent/main.go:1018` |
| 2 | Only on trigger night and only when the hub's block says `Tonight`; the household is mailed the day before, 09:00 to 20:00 Budapest time, by the hub. | A | `AG/internal/osupdate/kernel.go:101-115` `AR/11-os-updates.md:580-585` |
| 3 | Judge wait `DefaultKernelJudgeWait` 20 min, polled every 30 s; measured healthy at 68 s and 272 s. | A | `AG/internal/osupdate/kernel.go:37` `AG/internal/osupdate/kernel.go:251` `AR/11-os-updates.md:570-572` |
| 4 | No mail accepted means no step; a step is never retried by itself; a box that never comes back raises `host_stale` at 45 min and `host_down` at 90 min. | A | `AR/11-os-updates.md:584-585` `AR/11-os-updates.md:581-583` `AR/11-os-updates.md:572-573` |
| 5 | At most one per night under the same heavy-op gate; never two steps within 20 h (wrapper refusal R22); one self-revert, never retried. | A | `AR/11-os-updates.md:560-562` `AR/11-os-updates.md:557-558` `AG/internal/osupdate/kernel.go:300-312` |
| 6 | The 20-minute judge runs on monotonic time; a clock step does not move it. Nothing in the installer sets up time keeping, and a host can start with a wrong clock: the N100's host clock is 32 s behind today. | K | `AG/internal/osupdate/leg.go:291-296` `E/scripts/felhom-host-install.sh:1` · R-1301 |
| 7 | The hub hears staged before the reboot; operator events `os_kernel_step` and `os_kernel_notice`; a frozen new kernel needs a person, accepted. | A | `AG/internal/osupdate/kernel.go:165` `AR/11-os-updates.md:585-586` `AR/11-os-updates.md:534-535` |
| 8 | TEST and DOC. The design text itself says that a real freeze was not measured. | A | `AR/11-os-updates.md:599-601` `AG/internal/osupdate/kernel_test.go:79` |
**Hand step in a runbook?** YES — `RB/crash-guard.md`
### CA04 — Crash guard: early-boot unit and the hourly re-arm timer · CUSTOMER · Watchdog · MENTIONED
architecture folder: 23 lines in 7 files match `/crash.guard|crash guard/` (a machine count, not a reading) · helper section: clock-agent „4. Crash guard: early-boot unit plus hourly re-arm timer”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Root systemd units installed by the host installer: boot unit and timer. CUSTOMER (decides if a crashed box restarts). | A | `E/scripts/felhom-host-install.sh:2944-2963` `AG/configs/felhom-crash-guard.service:15-16` `AG/configs/felhom-crash-guard-check.timer:5-6` |
| 2 | Timer as written: `OnBootSec=15min`, `OnUnitActiveSec=1h`; no OnCalendar, no RandomizedDelaySec, no Persistent; boot unit runs before sysinit. | A | `AG/configs/felhom-crash-guard-check.timer:5-6` `AG/configs/felhom-crash-guard.service:8` |
| 3 | Oneshot with no time limit written; history is cut to 7 days AG/configs/felhom-crash-guard:148, so nothing grows. | A | `AG/configs/felhom-crash-guard-check.service:6-7` |
| 4 | A missed hourly check only delays the re-arm; a crash before the boot unit runs leaves the box off, the safe side; the System page shows the guard. | A | `AR/11-os-updates.md:484-485` `AR/11-os-updates.md:487-488` |
| 5 | The second unclean boot inside 60 minutes trips it; it arms again after 24 hours. The boot unit and the hourly check share one state file and take no lock. | N | `AG/configs/crash-guard.conf:3-8` · class W3 · S14 |
| 6 | The script reads the wall clock at early boot, before any time service runs. A clock that is wrong at boot can move unclean boots into or out of the 60-minute window. | N | class W2 · S22 |
| 7 | Each unclean boot is an operator mail `host_crash_restart` and a household line; the trip is alarm `host_crash_guard_tripped`; the trip mail came 15 min after the boot. | A | `AR/11-os-updates.md:487-488` `AR/11-os-updates.md:489-490` |
| 8 | MEASURED 2026-11-11: the hourly timer is active on the three hosts · TEST and DOC for the windows. A wrong clock at boot: NOBODY KNOWS. | N | `EV/box/demo-hp-host-clock.txt` `EV/box/tester-1-host-clock.txt` `EV/box/demo-felhom-host-clock.txt` `AG/configs/test_felhom_crash_guard.py:75` `AG/configs/test_felhom_crash_guard.py:102` `AR/11-os-updates.md:473-490` · class W2 · S22 |
**Hand step in a runbook?** YES — `RB/crash-guard.md`
### CA05 — Scheduled restore-test evaluation · CUSTOMER · Night · MENTIONED
architecture folder: 121 lines in 19 files match `/restore.test/` (a machine count, not a reading) · helper section: clock-agent „5. Scheduled restore-test evaluation”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine `Run`, wired at; restores an archive into a scratch guest on the host. CUSTOMER (proof the backup opens). | A | `AG/internal/backup/schedule.go:148-177` `AG/cmd/felhom-agent/main.go:1968-2018` |
| 2 | First look `DefaultFirstEval` 30 min after agent start, then every `defaultRestoreTestEvalInterval` 6 h; an archive counts after `defaultRestoreTestSettle` 24 h; not windowed, any hour. | A | `AG/internal/backup/schedule.go:76` `AG/internal/config/config.go:636` `AG/internal/config/config.go:641` `AR/11-os-updates.md:385-386` |
| 3 | Restore task wait: `RestoreTestPBSRestoreTimeout` 120 min for the PBS tier, else the 10 min default; boot wait `DefaultBootTimeout` 2 min; grows with guest size. | A | `AG/internal/config/config.go:584-588` `AG/cmd/felhom-agent/main.go:1917-1919` `AG/internal/reconcile/restoretest.go:28` |
| 4 | The ticker restarts with the agent, so a missed look comes 30 min after the next start; the rule is per archive, so a tier stays due until proven; hub proof-age window 7 to 12 days. | A | `AR/07-backup-architecture.md:583-584` `AR/07-backup-architecture.md:179-180` `AR/07-backup-architecture.md:215-217` |
| 5 | Takes the heavy-op gate with `TryAcquire("restore-test")`; busy means deferred to the next 6 h look; proof credit only on a pass. | A | `AG/internal/backup/schedule.go:228` `AG/internal/backup/schedule.go:230` `AG/internal/backup/schedule.go:248-255` |
| 6 | The 'settled' cut-off uses the wall clock. A clock that is set back makes no archive look settled, so no test starts. The ticker itself is monotonic. | N | `AG/internal/backup/schedule.go:119` `AG/internal/backup/restoretest_due.go:96-101` · class W2 · S08 |
| 7 | A failed test is an ERROR log and a record the hub turns into `restore_test_failed` for the operator, as the janitor comment says. | A | `AG/internal/backup/schedule.go:262` `AG/cmd/felhom-agent/janitor.go:42-44` |
| 8 | TEST and DOC. MEASURED 2026-11-11: restore tests took 1 to 14 minutes on the three boxes. A clock set back: NOBODY KNOWS. | N | `EV/box/demo-hp-host-night.txt` · class W2 · S08 |
**Hand step in a runbook?** no hand step
### CA06 — Controller supervisor: restarts a dead or silent controller · CUSTOMER · Watchdog · MENTIONED
architecture folder: 12 lines in 5 files match `/supervisor/` (a machine count, not a reading) · helper section: clock-agent „6. Controller supervisor (restarts a dead or silent controller)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine `WatchControllers`, started at. CUSTOMER (the household's dashboard comes back). | A | `AG/internal/localapi/controllersupervisor.go:143-163` `AG/cmd/felhom-agent/main.go:1714` |
| 2 | Every `controllerSupervisorInterval` 30 s; restart on the 2nd not-running sweep; a running but unhealthy one is alarmed at 6 sweeps and restarted at 10. | A | `AG/internal/localapi/controllersupervisor.go:54` `AG/internal/localapi/controllersupervisor.go:56` `AG/internal/localapi/controllersupervisor.go:92-100` |
| 3 | No time limit on one sweep in the source, and no duration was measured. Nothing grows with data. | N | `AG/internal/localapi/controllersupervisor.go:1-200` `AG/internal/localapi/controllersupervisor.go:64` · class W4 · S15 |
| 4 | If the agent is down nothing restarts the controller: measured 33 min of 502 before this existed; the first sweep is 30 s after agent start (ticker only). | A | `AG/internal/localapi/controllersupervisor.go:20-25` `AG/internal/localapi/controllersupervisor.go:153-161` |
| 5 | Brake: `controllerCrashloopMax` 3 in `controllerCrashloopWindow` 15 min, then `controllerCrashloopPause` 30 min; not during a swap, a parked or locked guest, or a vzdump. | A | `AG/internal/localapi/controllersupervisor.go:59-61` `AG/internal/localapi/controllersupervisor.go:34-42` |
| 6 | The windows (3 restarts in 15 minutes, the 24-hour counter) use the wall clock in UTC. Nobody tested a clock jump. | N | `AG/internal/localapi/controllersupervisor.go:550-552` `AG/internal/localapi/controllersupervisor.go:308` · class W2 · S08 |
| 7 | Hub mints `controller_restarted_by_agent` (info) and `controller_crashloop` (error, operator only) from the host report; a warning at the 5th restart in 24 h. | A | `AG/internal/localapi/controllersupervisor.go:44-49` `AG/internal/localapi/controllersupervisor.go:82-83` |
| 8 | TEST `TestControllerSupervisor_KilledControllerIsRestarted`, `TestSupervisor_UnresponsiveIsAlarmedThenRestarted`; DOC. | A | `AG/internal/localapi/controllersupervisor_test.go:101` `AG/internal/localapi/supervisor_unresponsive_test.go:32` `AR/03-host-agent.md:186-192` |
**Hand step in a runbook?** no hand step
### CA07 — Guest-power watchdog: starts a guest that must run · CUSTOMER · Watchdog · MENTIONED
architecture folder: 8 lines in 4 files match `/guest.power|guest power/` (a machine count, not a reading) · helper section: clock-agent „7. Guest-power watchdog (starts a guest that should be running)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine `WatchGuestPower`, started at. CUSTOMER (the whole box's apps are off while the guest is stopped). | A | `AG/internal/localapi/guestpower.go:89-105` `AG/cmd/felhom-agent/main.go:1710` |
| 2 | Every `guestPowerInterval` 60 s; acts only on a stopped, unlocked guest with onboot set; first sweep one interval after start (ticker only). | A | `AG/internal/localapi/guestpower.go:52` `AG/internal/localapi/guestpower.go:167-180` `AG/internal/localapi/guestpower.go:95-102` |
| 3 | At most `guestPowerMaxAttempts` 3 starts with waits of 1, 2 and 4 min, `guestPowerBackoff`; about 7 min in all. | A | `AG/internal/localapi/guestpower.go:56` `AG/internal/localapi/guestpower.go:75` `AG/internal/localapi/guestpower.go:71-74` |
| 4 | Before it existed a guest stayed down 9 min 47 s until a person started it; with the agent down nobody starts it (the platform's onboot acts only at host boot). | A | `AG/internal/localapi/guestpower.go:14-18` `AG/internal/localapi/guestpower.go:42-43` |
| 5 | Never starts under a running vzdump and fails safe on doubt; leaves a locked guest to the stale-lock path; skips a guest the operator is rebooting. | A | `AG/internal/localapi/guestpower.go:182-194` `AG/internal/localapi/guestpower.go:176-180` `AG/internal/localapi/guestpower.go:157-159` |
| 6 | The next try time is kept in memory from the wall clock. A clock set back delays the next try; an agent restart tries again at once. | N | `AG/internal/localapi/guestpower.go:234` `AG/internal/localapi/guestpower.go:253` · class W2 · S08 |
| 7 | When the watchdog gives up, it writes one ERROR line in the host's journal. No event goes to the hub. The household's box is down and the outside sign is only the hub's liveness check. | N | `AG/internal/localapi/guestpower.go:200-206` · class W1 · S04 |
| 8 | TEST for the start and the back-off. Who hears the give-up: NOBODY KNOWS. | N | `AG/internal/localapi/guestpower_test.go:162` `AG/internal/localapi/guestpower_test.go:119` · class W1 · S04 |
**Hand step in a runbook?** NO — no runbook says what to do after the watchdog gave up
### CA08 — Hub report and heartbeat loop · CUSTOMER · Interval · MENTIONED
architecture folder: 23 lines in 10 files match `/heartbeat|poll interval|report loop/` (a machine count, not a reading) · helper section: clock-agent „8. Hub report and heartbeat loop (with reconcile, fast tick and poke)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine `Run`, built at; it carries every status the hub alarms on. OPERATOR and CUSTOMER (alarms depend on it). Merged here: the reconcile engine, fast tick and poke. | A | `AG/internal/hub/loop.go:107-138` `AG/cmd/felhom-agent/main.go:958` |
| 2 | `PollSeconds` default 900; first report at once, then a ticker; the hub may change it, clamped by `MinPollSeconds` 60 and `MaxPollSeconds` 3600; fast tick `DefaultInterval` 30 s. | A | `AG/internal/config/config.go:797` `AG/internal/hub/loop.go:107-138` `AG/internal/hub/loop.go:11-12` `AG/internal/fasttick/fasttick.go:24` |
| 3 | Per-request limit `TimeoutSeconds` default 30; the system facts in it are read at most every 10 min; the report grows with guests and drives. | A | `AG/internal/config/config.go:798` `AG/cmd/felhom-agent/main.go:4149` |
| 4 | A failed cycle is logged and the interval kept; no backlog, the next report replaces it; the hub raises `host_stale` at 45 min and `host_down` at 90 min. | A | `AG/internal/hub/loop.go:162-164` `AR/11-os-updates.md:572-573` |
| 5 | Extra reports come by the trigger channel; a poke is debounced by `DebounceWindow` 2 s; the reconcile engine runs at once and then on the start-time interval,. | A | `AG/internal/hub/loop.go:125-135` `AG/internal/poke/poke.go:40` `AG/internal/reconcile/engine.go:332-344` `AG/cmd/felhom-agent/main.go:1525` |
| 6 | The ticker is monotonic. The report loop follows the hub's interval override, clamped to a range. The reconcile engine keeps the interval that it read at the start. | A | `AG/internal/hub/loop.go:188-196` `AG/cmd/felhom-agent/main.go:969` |
| 7 | The operator, by the hub's staleness alarms; the household hears cannot-be-reached at most once per 7 days. | A | `AR/11-os-updates.md:572-573` `AR/07-backup-architecture.md:582-583` |
| 8 | TEST `TestLoop_RunImmediateAndResilientAfterError`, `TestClampInterval`; DOC. | A | `AG/internal/hub/loop_test.go:140` `AG/internal/hub/loop_test.go:81` `AR/03-host-agent.md:276-279` |
**Hand step in a runbook?** no hand step
### CA09 — Signed jobs: the validity window against the host clock · OPERATOR · Interval · MENTIONED
architecture folder: 155 lines in 27 files match `/signed job|signed op|not.before|valid/` (a machine count, not a reading) · helper section: clock-agent „9. Signed jobs: validity window against the host clock (agent updates ride it)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent `OnEnvelope` runs pending signed jobs when a heartbeat says there are some; an agent update arrives only this way. OPERATOR (and CUSTOMER: fixes reach the box this way). | A | `AG/internal/signedjobs/runner.go:93-102` |
| 2 | Checked at every hub cycle (item 8); a job is valid from issue minus `DefaultClockSkew` 2 min to its expiry, and a window over `MaxWindow` 24 h is refused. | A | `AG/internal/authz/verifier.go:21` `AG/internal/authz/verifier.go:26` |
| 3 | The self-update download has a 5 min client limit; the new binary must run `DwellSeconds` 60 s before it commits. | A | `AG/internal/selfupdate/executor.go:73` `AG/internal/config/config.go:97-99` |
| 4 | A job that the agent refuses as expired is not run. The agent's log line is the only trace on the box; no path that tells the hub or the operator 'refused: expired' was found. | N | `AG/internal/authz/verifier.go:161-162` `AG/internal/selfupdate/commit.go:122-125` · class W1 · S09 |
| 5 | One run at a time, `RunOnce` single-flight; a replay is refused by the nonce store. | A | `AG/internal/signedjobs/runner.go:107-115` `AG/internal/authz/verifier.go:166` |
| 6 | The window is checked against the host's wall clock. A host clock that is behind refuses every job as 'not yet valid'; one that is ahead refuses them as 'expired'. So a box with a wrong clock cannot take an agent update. | N | `AG/internal/authz/verifier.go:153-162` · class W2 · S09 |
| 7 | The runner logs a finished or failed job on the host. For a job refused on its dates, no message to the hub was found. | N | `AG/internal/signedjobs/runner.go:176-190` · class W1 · S09 |
| 8 | TEST `TestVerify_RejectsPerStage`, `TestRunner_ExpiredRejected`, `TestManager_ShutdownBeforeDwellLeavesPending`. | A | `AG/internal/authz/verify_test.go:82` `AG/internal/signedjobs/runner_test.go:168` `AG/internal/selfupdate/commit_test.go:102` |
**Hand step in a runbook?** NO — no runbook names a wrong host clock as a cause of refused signed jobs
### CA10 — Storage watchdog and the 20-second mount re-assert · CUSTOMER · Watchdog · MENTIONED
architecture folder: 2 lines in 2 files match `/storage watchdog|re.?assert|ReassertGuestBinds/` (a machine count, not a reading) · helper section: clock-agent „10. Storage watchdog and the 20-second mount and bind re-assert”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent: the watchdog `Run` wired at, and a 20 s ticker that re-asserts mounts and guest binds. CUSTOMER (a dropped USB drive). | A | `AG/internal/storage/watchdog.go:196-215` `AG/cmd/felhom-agent/main.go:1142` `AG/cmd/felhom-agent/main.go:1613-1631` |
| 2 | `DefaultWatchdogInterval` 8 s and `DefaultWatchdogDebounce` 30 s; setting `WatchdogIntervalSeconds`, 0 means the default; the re-assert is 20 s, below the controller's 30 s tick. | A | `AG/internal/storage/watchdog.go:15-16` `AG/internal/config/config.go:770-778` `AG/cmd/felhom-agent/main.go:1612-1614` |
| 3 | No time limit on one tick in the source, and no duration was measured. The work grows with the number of drives. | N | class W4 · S15 |
| 4 | A missed tick is made up by the next one; with the agent down a dropped drive is seen only at the next 15 min report, the reason the watchdog exists. | A | `AR/03-host-agent.md:461` |
| 5 | A flapping drive backs off: `backoffWindow`, alert after 4 failed cycles; the retry test is `now.Sub(last)`. | A | `AG/internal/storage/watchdog.go:183-192` `AG/internal/storage/watchdog.go:77-79` `AG/internal/storage/watchdog.go:290` |
| 6 | The back-off uses the wall clock in UTC. A clock set back stretches the back-off; a clock set forward ends it early. The tickers are monotonic. | N | `AG/internal/storage/watchdog.go:160` · class W2 · S08 |
| 7 | A drop requests an immediate hub report through the trigger channel; the hub then decides who is mailed (hub side, not read). | A | `AG/internal/hub/loop.go:125-135` |
| 8 | TEST for the reconcile and the back-off. The 20-second ticker has no test: NOBODY KNOWS. | N | `AG/internal/storage/watchdog_test.go:112` `AG/internal/storage/watchdog_test.go:460` `AG/cmd/felhom-agent/main.go:1613` · class W4 · S15 |
**Hand step in a runbook?** no hand step
### CA11 — Weekly guest disk trim · CUSTOMER · Interval · MENTIONED
architecture folder: 9 lines in 5 files match `/fstrim|trim/` (a machine count, not a reading) · helper section: clock-agent „11. Weekly guest disk trim”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine `Run`, started at; runs `pct fstrim` per owned running guest. CUSTOMER (a full thin pool turns every guest read-only). | A | `AG/internal/fstrim/fstrim.go:186-199` `AG/cmd/felhom-agent/main.go:1741` `AG/internal/fstrim/fstrim.go:3-5` |
| 2 | Due from Wednesday 10:00 host-local, starts only 10:00 to 20:59; looks 5 min after start then hourly, `FirstTickDelay` and `TickInterval`; zone is `time.Local`. | A | `AG/internal/fstrim/fstrim.go:42-46` `AG/internal/fstrim/fstrim.go:48-50` `AG/internal/fstrim/fstrim.go:110` |
| 3 | `PerGuestTimeout` 30 min; measured 24.4 s for 84 GiB; grows with freed blocks. | A | `AG/internal/fstrim/fstrim.go:52` `AG/internal/fstrim/fstrim.go:6` |
| 4 | A box off on Wednesday catches up at its next eligible hour; a failure is retried at most `MaxAttemptsPerWeek` 3 times; the result rides the host report. | A | `AG/internal/fstrim/fstrim.go:10-11` `AG/internal/fstrim/fstrim.go:46` `AG/internal/fstrim/fstrim.go:326-330` |
| 5 | Holds the heavy-op gate for the whole pass; busy means deferred to the next hour; the persisted record stops a second trim after a restart. | A | `AG/internal/fstrim/fstrim.go:247-251` `AG/internal/fstrim/fstrim.go:17` |
| 6 | The 'done this week' check uses host-local wall time. An attempt that was stamped while the clock was ahead keeps the trim 'not due' until real time passes it. | N | `AG/internal/fstrim/fstrim.go:179-182` · class W2 · S08 |
| 7 | The operator, on the hub's System page (one cell). A failed trim is a WARN line on the host. Nobody is mailed. | A | `HUB/web/system.go:45` `HUB/web/system.go:206` `AG/internal/fstrim/fstrim.go:318-319` |
| 8 | TEST for the week anchor. The hour change and a clock jump: NOBODY KNOWS. | N | `AG/internal/fstrim/fstrim_test.go:161` `AG/internal/fstrim/fstrim_test.go:175` `AG/internal/fstrim/fstrim_test.go:104` · class W2 · S20 |
**Hand step in a runbook?** no hand step
### CA12 — Weekly upload line test · CUSTOMER · Interval · MENTIONED
architecture folder: 28 lines in 10 files match `/line.?test|linetest/` (a machine count, not a reading) · helper section: clock-agent „12. Weekly upload line test”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine `Run`, started at; a 10 s iperf3 upload over the tunnel. CUSTOMER (sets the torrent app's upload limit). | A | `AG/internal/linetest/linetest.go:256-268` `AG/cmd/felhom-agent/main.go:1762` `AG/internal/linetest/linetest.go:3-4` |
| 2 | Due from Tuesday 10:00 host-local, starts 10:00 to 15:59; each host from its own minute, `SlotMinute`; ticks every minute after a 5 min delay; zone `time.Local`. | A | `AG/internal/linetest/linetest.go:47-49` `AG/internal/linetest/linetest.go:148-152` `AG/internal/linetest/linetest.go:60-61` `AG/internal/linetest/linetest.go:131` |
| 3 | Three runs of 10 s, each bounded by `RunTimeout` 25 s, 30 s apart; nothing grows with data. | A | `AG/internal/linetest/linetest.go:53-57` |
| 4 | A box that was off catches up on a later day inside the window; the hub page turns amber when the last success is over 15 days old. | A | `AG/internal/linetest/linetest.go:10-11` `AR/11-os-updates.md:409-411` |
| 5 | Needs 10 quiet minutes and the heavy-op gate; server busy is retried at the next tick, at most `MaxTriesPerDay` 6; no tunnel means no test. | A | `AG/internal/linetest/linetest.go:316-325` `AG/internal/linetest/linetest.go:58` `AG/internal/linetest/linetest.go:297-300` |
| 6 | The check uses host-local wall time. A success that was stamped from the future blocks the tests until that time. | N | `AG/internal/linetest/linetest.go:211-213` · class W2 · S08 |
| 7 | The operator, on the hub's System page; the cell is judged on the last success, not on the attempt. Nobody is mailed. | A | `AR/11-os-updates.md:409-411` |
| 8 | TEST `TestDueWeeklyFromTuesday`, `TestNeverTestsInTheNightOrBeforeTuesday`, `TestSlotDelaysTheFirstStartOnTheDueDayOnly`. | A | `AG/internal/linetest/linetest_test.go:206` `AG/internal/linetest/linetest_test.go:363` `AG/internal/linetest/linetest_test.go:189` |
**Hand step in a runbook?** no hand step
### CA13 — PBS datastore verify loop · CUSTOMER · Interval · MENTIONED
architecture folder: 64 lines in 17 files match `/verify loop|PBS verify|verify/` (a machine count, not a reading) · helper section: clock-agent „13. PBS datastore verify loop”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine `Run`, cadence wired at; asks the PBS server to verify the box's snapshots. CUSTOMER (is the off-site copy readable). | A | `AG/internal/pbs/verify.go:64-83` `AG/cmd/felhom-agent/main.go:1188` |
| 2 | `DefaultVerifyCadence` 6 h; setting through `PBSVerifyCadence`; first pass at once at agent start, then the ticker; no time zone. | A | `AG/internal/pbs/verify.go:13` `AG/internal/config/config.go:710-719` `AG/internal/pbs/verify.go:64-83` |
| 3 | One verify is polled every 2 s for at most 30 min; grows with the datastore's size. | A | `AG/internal/pbs/verify.go:107` |
| 4 | No catch-up state: every agent start verifies again at once; a negative cadence switches it off, test `TestVerifyLoop_DisabledByNegativeCadence`. | A | `AG/internal/pbs/verify.go:64-83` `AG/internal/pbs/verify_test.go:87` |
| 5 | The verify is not behind the heavy-work gate; a comment in the source says so. It can run beside a whole-guest backup. | N | `AG/internal/lanresolver/loop.go:17` `AG/internal/pbs/verify.go:1` · class W3 · S13 |
| 6 | Ticker only, monotonic; no wall-clock decision found in. | A | `AG/internal/pbs/verify.go:64-83` |
| 7 | A snapshot that fails the verify is an ERROR line on the host and a field in the report. No hub event or mail for it was found. | N | `AG/internal/pbs/verify.go:127` · class W1 · S05 |
| 8 | TEST for the record of a failed verify. A verify that takes longer than its 30-minute cadence: NOBODY KNOWS. | N | `AG/internal/pbs/verify_test.go:52` · class W4 · S15 |
**Hand step in a runbook?** no hand step
### CA14 — Janitor: leftover scratch guests and stale backup locks · CUSTOMER · Interval · MENTIONED
architecture folder: 2 lines in 2 files match `/janitor|stale lock/` (a machine count, not a reading) · helper section: clock-agent „14. Janitor: leftover scratch guests and stale backup locks”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine `runJanitor`, started at. CUSTOMER (a stale lock blocked whole-box backups for five hours). | A | `AG/cmd/felhom-agent/janitor.go:69-81` `AG/cmd/felhom-agent/main.go:1725` `AG/cmd/felhom-agent/janitor.go:14-17` |
| 2 | Every `janitorInterval` 10 min; ticker only, so the first pass is 10 min after agent start; no time zone. | A | `AG/cmd/felhom-agent/janitor.go:18` `AG/cmd/felhom-agent/janitor.go:71-78` |
| 3 | No time limit on a pass found in; work grows with the number of leftover scratch guests. | A | `AG/cmd/felhom-agent/janitor.go:34-66` |
| 4 | A missed pass is retried 10 min later; a scratch guest that cannot be removed after the retry limit is reported as a failed restore-test record. | A | `AG/cmd/felhom-agent/janitor.go:41-54` |
| 5 | The stale-lock sweep runs only holding the heavy-op gate, else it is skipped this pass; rule stated at. | A | `AG/cmd/felhom-agent/janitor.go:57-64` `AG/cmd/felhom-agent/janitor.go:30-33` |
| 6 | Ticker only, monotonic; a clock step must not kill a live task: `TestVzdumpInFlight_ClockStepCannotKillALiveTask`. | A | `AG/internal/localapi/stalelock_inflight_test.go:97` |
| 7 | The operator, through the hub's `restore_test_failed`, once per distinct archive. | A | `AG/cmd/felhom-agent/janitor.go:42-44` |
| 8 | TEST `TestJanitor_StaleLockSweepWaitsForTheHeavyGate`, `TestJanitor_GaveUpIsReportedAsAFailure`. | A | `AG/cmd/felhom-agent/janitor_test.go:40` `AG/cmd/felhom-agent/janitor_test.go:22` |
**Hand step in a runbook?** no hand step
### CA15 — Guest network watchdog: DHCP client heal · CUSTOMER · Watchdog · MENTIONED
architecture folder: 17 lines in 5 files match `/guest.?net|DHCP/` (a machine count, not a reading) · helper section: clock-agent „15. Guest network watchdog (DHCP client heal)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine `Watch`, built at. CUSTOMER (a guest without network). | A | `AG/internal/guestnet/watchdog.go:105-121` `AG/cmd/felhom-agent/main.go:1686` |
| 2 | `DefaultInterval` 60 s; immediate first sweep; settings interval 60, `MinHealIntervalSeconds` 600, `MaxHealsPerHour` 3. | A | `AG/internal/guestnet/watchdog.go:23` `AG/internal/guestnet/watchdog.go:111` `AG/internal/config/config.go:176-196` |
| 3 | No time limit on one probe in the source, and no duration was measured. | N | `AG/internal/guestnet/watchdog.go:1-200` · class W4 · S15 |
| 4 | A missed sweep is made up a minute later; beyond 3 heals an hour it only reports. | A | `AG/internal/guestnet/watchdog.go:26-28` |
| 5 | Two consecutive bad probes before a heal, `requiredBadProbes`; 10 min between heals, `DefaultMinHealInterval`; boot-race guard `DefaultSettle` 3 min. | A | `AG/internal/guestnet/watchdog.go:32-33` `AG/internal/guestnet/watchdog.go:25` `AG/internal/guestnet/watchdog.go:29-31` |
| 6 | `now` is plain `time.Now`, so in-process gaps are monotonic; heal history is in memory. | A | `AG/internal/guestnet/watchdog.go:80` `AG/internal/guestnet/watchdog.go:56-62` |
| 7 | The operator, on the hub's Hosts page: the report carries the heal count for each guest. Nobody is mailed. | A | `HUB/web/hosts.go:298-397` `AG/internal/guestnet/watchdog.go:166-170` |
| 8 | TEST `TestWatchdog_DampingCeilings`, `TestWatchdog_MinimumIntervalBetweenHeals`, `TestWatchdog_BootRacesObserveOnly`. | A | `AG/internal/guestnet/watchdog_test.go:360` `AG/internal/guestnet/watchdog_test.go:384` `AG/internal/guestnet/watchdog_test.go:416` |
**Hand step in a runbook?** no hand step
### CA16 — Host networking self-heal · CUSTOMER · Watchdog · MENTIONED
architecture folder: 39 lines in 12 files match `/self.?heal/` (a machine count, not a reading) · helper section: clock-agent „16. Host networking self-heal”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine `Watch`, interval set at; starts `networking.service` on an appliance only. CUSTOMER. | A | `AG/internal/selfheal/selfheal.go:67-82` `AG/cmd/felhom-agent/main.go:1648` |
| 2 | One pass at agent start, then every 60 s, no setting found for the interval (hard-coded). | A | `AG/internal/selfheal/selfheal.go:67-82` `AG/cmd/felhom-agent/main.go:1648` |
| 3 | A remedy is 3 start attempts with waits of 10, 30 and 60 s, `networkingBackoffs`. | A | `AG/internal/selfheal/selfheal.go:114-116` |
| 4 | A missed pass is made up a minute later; on a BYO host the remedy is skipped and only a WARN is written. | A | `AG/internal/selfheal/selfheal.go:54-58` |
| 5 | No lock seen; the pass is sequential inside one goroutine. | A | `AG/internal/selfheal/selfheal.go:46-64` |
| 6 | Ticker only, monotonic; no wall-clock decision in. | A | `AG/internal/selfheal/selfheal.go:67-82` |
| 7 | A give-up is an ERROR journal line; with networking down the hub cannot be told, so the hub's stale alarm is the only signal. | A | `AG/internal/selfheal/selfheal.go:61` `AR/11-os-updates.md:572-573` |
| 8 | TEST `TestNetworkingHeal_GivesUpAfterAllAttempts`, `TestManager_ByoNeverRemediates`. | A | `AG/internal/selfheal/selfheal_test.go:109` `AG/internal/selfheal/selfheal_test.go:32` |
**Hand step in a runbook?** no hand step
### CA17 — Management-plane watchdog timer: keeps SSH reachable · OPERATOR · Watchdog · MENTIONED
architecture folder: 5 lines in 4 files match `/mgmt.watchdog|management.plane/` (a machine count, not a reading) · helper section: clock-agent „17. Management-plane watchdog timer (keeps SSH reachable)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Root systemd timer and oneshot installed by the host installer; script; works without the agent. OPERATOR. | A | `E/scripts/felhom-host-install.sh:3067-3076` `AG/configs/felhom-mgmt-watchdog.sh:26-68` `AG/configs/felhom-mgmt-watchdog.service:3-7` |
| 2 | As written: `OnBootSec=30s`, `OnUnitActiveSec=60s`, `AccuracySec=5s`, `Persistent=true`; no OnCalendar, no RandomizedDelaySec. | A | `AG/configs/felhom-mgmt-watchdog.timer:11-14` |
| 3 | No time limit written in the unit; the script is a few stat and systemctl calls. | A | `AG/configs/felhom-mgmt-watchdog.service:16-18` `AG/configs/felhom-mgmt-watchdog.sh:36-61` |
| 4 | A missed tick is retried by the next one; the heal budget is one tick. | A | `AG/configs/felhom-mgmt-watchdog.sh:23-24` `AG/configs/felhom-mgmt-watchdog.timer:3-6` |
| 5 | The script changes nothing when the directory is healthy. MEASURED: the timer is active on the three hosts and fires each 60 seconds. `Persistent=true` in the unit has no effect, because the timer has no calendar line. | A | `AG/configs/felhom-mgmt-watchdog.sh:13-15` `AG/configs/felhom-mgmt-watchdog.timer:11-14` `EV/box/demo-hp-host-clock.txt` |
| 6 | Monotonic timer settings only; the heal marker is a UTC stamp from `date -u`. | A | `AG/configs/felhom-mgmt-watchdog.timer:11-12` `AG/configs/felhom-mgmt-watchdog.sh:65` |
| 7 | The marker lets the agent report the heal and the hub raise a warning,. | A | `AG/configs/felhom-mgmt-watchdog.sh:18-19` `AG/configs/felhom-mgmt-watchdog.sh:63` |
| 8 | SOURCE only. The script has no test file. | A | `AG/configs/felhom-mgmt-watchdog.sh:1-68` `AG/configs/felhom-mgmt-watchdog.timer:3-6` |
**Hand step in a runbook?** no hand step
### CA18 — Agent service: restart, start limit, rollback · CUSTOMER · Watchdog · MENTIONED
architecture folder: 72 lines in 15 files match `/Restart=|start limit|rollback/` (a machine count, not a reading) · helper section: clock-agent „18. Agent service restart, start limit and rollback”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | systemd restarts the agent: `Restart=on-failure`, `RestartSec=5s`; drop-in limits. OPERATOR (every loop in this list dies with the agent). | A | `AG/configs/felhom-agent.service:41-42` `AG/configs/felhom-agent-limits.conf:17-19` |
| 2 | `StartLimitIntervalSec=120`, `StartLimitBurst=4`, `OnFailure=felhom-agent-rollback.service`; a crash-looping binary reaches failed about 20 s after the first crash. | A | `AG/configs/felhom-agent-limits.conf:17-19` `AG/configs/felhom-agent-limits.conf:5-8` |
| 3 | No WatchdogSec in the unit (grep `WatchdogSec` in finds nothing); a new binary commits after a 60 s dwell. | A | `AG/configs/felhom-agent.service:1` `AG/internal/selfupdate/commit.go:40-42` |
| 4 | After the start limit the unit stays failed. Nothing on the host starts it again. The outside sign is the hub's 'host stale' alarm at 45 minutes. | A | `AG/configs/felhom-agent.service:41-42` `AR/11-os-updates.md:572-573` |
| 5 | The rollback unit fires on every crash and is guarded by a pending marker; a shutdown before the dwell leaves the update pending. | A | `AG/configs/felhom-agent-limits.conf:14-15` `AG/internal/selfupdate/commit.go:112-116` |
| 6 | The design text says `Restart=always` and a systemd watchdog that kills a hung agent. The unit has `Restart=on-failure` and no watchdog. | N | `AR/03-host-agent.md:72` `AG/configs/felhom-agent.service:41-42` · class W5 · S17 |
| 7 | The operator, by the hub's stale and down alarms only; a restore-test crash loop that never evaluates is covered by `TestR874_CrashLoopNeverEvaluates`. | A | `AR/11-os-updates.md:572-573` `AG/internal/backup/r874_first_eval_test.go:44` |
| 8 | MEASURED values and a TEST for the rollback. A hung agent (alive, not working): NOBODY KNOWS; no watchdog restarts it. | N | `AG/configs/felhom-agent.service:41-42` · class W5 · S17 |
**Hand step in a runbook?** PARTLY — `RB/break-glass.md` names `reset-failed`; no runbook names the start limit of the agent
### CA19 — Off-site tunnel loop and the clock-ahead re-key · CUSTOMER · Interval · MENTIONED
architecture folder: 236 lines in 22 files match `/wgtunnel|re.?key|tunnel/` (a machine count, not a reading) · helper section: clock-agent „19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutines of one shape: tunnel `Run`, PBS-DR, OOB sshd, LAN resolver. CUSTOMER (off-site backup path) and OPERATOR. Four loops merged. | A | `AG/internal/wgtunnel/loop.go:62-78` `AG/internal/pbsdr/loop.go:62-77` `AG/internal/felhomsshd/loop.go:67-80` `AG/internal/lanresolver/loop.go:35-52` |
| 2 | Tunnel, PBS-DR and OOB every 60 s, resolver every 5 min; each applies at once at start and on a desired-state nudge. | A | `AG/internal/config/config.go:132-133` `AG/cmd/felhom-agent/main.go:1504` `AG/internal/config/config.go:74-75` `AG/internal/lanresolver/loop.go:27-29` |
| 3 | No time limit on one pass in the source, and no duration was measured. Nothing grows with data. | N | class W4 · S15 |
| 4 | A missed pass is made up at the next tick; a handshake older than `StaleAfterSeconds` 180 re-resolves the endpoint,. | A | `AG/internal/config/config.go:124-127` `AG/internal/config/config.go:138-140` |
| 5 | A handshake stamped in the future for `rekeyAfterDefault` 15 min makes a new key, at most once per `rekeyMinGap` 1 h; the hour bound survives a restart, the 15 min count does not. | A | `AG/internal/wgtunnel/rekey.go:36-38` `AR/03-host-agent.md:376-378` |
| 6 | A box whose clock once ran ahead is locked out of the off-site endpoint until real time passes that moment; healed by the re-key; known gap: a reboot after the clock was corrected forgets the future stamp. | A | `AR/03-host-agent.md:142` `AR/03-host-agent.md:144-150` `AR/03-host-agent.md:155-156` |
| 7 | The heartbeat carries `handshake_future_s` and the hub alarms on it. | A | `AR/03-host-agent.md:142` |
| 8 | TEST `TestD30_AFutureHandshakeThatStaysReKeys`, `TestD30_ALastReKeyFromTheFutureDoesNotBlock`, `TestR1204_AFutureHandshakeIsReported`; measured live. | A | `AG/internal/wgtunnel/rekey_test.go:35` `AG/internal/wgtunnel/rekey_test.go:213` `AG/internal/wgtunnel/manager_test.go:704` `AR/03-host-agent.md:151-155` |
**Hand step in a runbook?** no hand step
### CA20 — Whole-guest backup cap: rolling windows · CUSTOMER · Night · MENTIONED
architecture folder: 577 lines in 27 files match `/backup cap|rolling window|cap/` (a machine count, not a reading) · helper section: clock-agent „20. Whole-guest backup cap (rolling windows)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent local API check `capRefuses` before a backup starts; rule `backupCadenceCap`. CUSTOMER (protects older copies from a flood of new ones). | A | `AG/internal/localapi/server.go:929` `AG/internal/localapi/backup_cap.go:41-53` |
| 2 | Tier of 24 h or less: 3 successes per rolling 24 h; longer tier: 1 per the larger of cadence minus 24 h and half the cadence; a refusal holds for `backupCapRefusalHold` 24 h. | A | `AG/internal/localapi/backup_cap.go:12-35` `AG/internal/localapi/backup_cap.go:38` |
| 3 | A lookup, no duration of its own; the success list is bounded by the window. | A | `AG/internal/localapi/backup_cap.go:41-53` |
| 4 | Does not apply to a missed run: failures do not count, `TestBackupCap_FailuresDoNotCount`. | A | `AG/internal/localapi/backup_cap_test.go:266` |
| 5 | The scheduled daily run is never refused, `TestBackupCap_TheScheduledDayIsNeverRefused`; the count survives a restart, `TestBackupCap_SurvivesARestart`. | A | `AG/internal/localapi/backup_cap_test.go:220` `AG/internal/localapi/backup_cap_test.go:248` |
| 6 | The windows use the wall clock. Successes that were stamped while the clock was ahead stay inside the window longer and can refuse real runs. NOBODY KNOWS. | N | `AG/internal/localapi/server.go:509` · class W2 · S08 |
| 7 | The operator, by event `whole_guest_backup_refused` named in; test `TestBackupCap_RefusalRaisesTheOperatorSignal`. | A | `AG/internal/localapi/backup_cap.go:12-35` `AG/internal/localapi/backup_cap_test.go:285` |
| 8 | TEST `TestBackupCap_LocalFloodRunsAtMostThreeADay`, `TestBackupCap_OffsiteFloodKeepsAPreCompromiseCopy`. | A | `AG/internal/localapi/backup_cap_test.go:198` `AG/internal/localapi/backup_cap_test.go:151` |
**Hand step in a runbook?** no hand step
### CA21 — Unsent OS-update report sender · OPERATOR · Interval · UNNAMED
architecture folder: 0 lines in 0 files match `/unsent/` (a machine count, not a reading) · helper section: clock-agent „22. Unsent OS-update report sender”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine `SendUnsentLoop`, started at; posts OS-update results the hub has not received. OPERATOR. | A | `AG/internal/osupdate/unsent.go:71-82` `AG/cmd/felhom-agent/main.go:1009` |
| 2 | Every 5 min, by `time.After`; one send at start, `SendUnsent`. | A | `AG/cmd/felhom-agent/main.go:1009` `AG/internal/osupdate/unsent.go:79` `AG/internal/osupdate/unsent.go:56-64` |
| 3 | A report post has a 1 min limit in `finish`; the kept copies grow only while the hub is away. | A | `AG/internal/osupdate/leg.go:922-939` |
| 4 | Hub away keeps the copy for a later pass, `TestR868_HubAwayKeepsTheCopy`. | A | `AG/internal/osupdate/unsent_test.go:87` |
| 5 | The sender stays off while a pass holds the flock, `tryLockPass`; test `TestR868_RunningPassKeepsTheSenderOff`. | A | `AG/internal/osupdate/unsent.go:35-52` `AG/internal/osupdate/unsent_test.go:110` |
| 6 | `time.After` is monotonic; no wall-clock decision found in. | A | `AG/internal/osupdate/unsent.go:79` `AG/internal/osupdate/unsent.go:56-82` |
| 7 | The operator sees the result late: at most 5 minutes after the hub is back. The sender has no alarm of its own. | A | `AG/internal/osupdate/unsent.go:1` |
| 8 | TEST `TestR868_ACopyWrittenAfterTheStartIsSentByTheLoop`, `TestR868_KilledPassIsReportedAtStart`. | A | `AG/internal/osupdate/unsent_test.go:127` `AG/internal/osupdate/unsent_test.go:20` |
**Hand step in a runbook?** no hand step
### CC01 — Night dump leg: databases and volumes · CUSTOMER · Night · MENTIONED
architecture folder: 17 lines in 5 files match `/db-dump|dump leg/` (a machine count, not a reading) · helper section: clock-controller „1. Night dump leg (databases and volumes) at W”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | `dbDumpLeg` runs `RunDBDumps`; owner: controller; started by the daily job "db-dump". CUSTOMER. | A | `CT/cmd/controller/main.go:1336-1344` `CT/internal/backup/backup.go:564` `CT/cmd/controller/main.go:1345` |
| 2 | Daily at W. W = household setting, else `DBDumpSchedule`, else `DefaultWindow` "02:30", decided in `EffectiveWindow`. First leg. Zone: Europe/Budapest, hard-coded, `getBudapestLocation`. | A | `CT/internal/backupwindow/backupwindow.go:12` `CT/internal/backupwindow/backupwindow.go:86-94` `CT/internal/scheduler/scheduler.go:17-27` |
| 3 | MEASURED, the night of 2026-11-11: 2 min 33 s on demo-hp, 40 s on Tester 1, 18 s on the N100. One database dump has a 5-minute limit. The whole leg has no limit. It grows with the number and size of the apps; all three stores are small. | A | `EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt` `CT/internal/appbackup/dbdump.go:291` |
| 4 | Box off at W: `Missed` sees it from the ledger and the catch-up runs it (block 4). The hub judges a missing dump at its 05:00 deadline. | A | `CT/internal/nightchain/nightchain.go:197-218` `AR/07-backup-architecture.md:579-584` |
| 5 | `withLeg` takes the leg lock; `acquireRunning` refuses a second backup; a job still running is skipped `executeJob`. A running migration skips the leg and returns no error. | A | `CT/cmd/controller/main.go:1321-1335` `CT/internal/backup/backup.go:1105-1113` `CT/internal/scheduler/scheduler.go:354-364` `CT/internal/backup/backup.go:565-568` |
| 6 | Slot from the wall clock, waited on a monotonic timer `runDailyJob`; fired over 60 min late = skipped `DailyLateLimit`. See block 24 for the hour change. | A | `CT/internal/scheduler/scheduler.go:300-352` `CT/internal/scheduler/scheduler.go:67` |
| 7 | Failure: `NotifyDBDumpFailed`. A made-up miss: one timeline line, never mailed. A miss not made up: banner after 26 h `StaleAfter`. | A | `CT/cmd/controller/main.go:1339` `AR/07-backup-architecture.md:562-564` `CT/internal/nightchain/banner.go:23` |
| 8 | MEASURED 2026-11-11, one night, small stores: 2 min 33 s · 40 s · 18 s (demo-hp · Tester 1 · N100) · TEST `TestCatchUp_OffAtWThenOn_OneCatchUpAfter15Min`; TEST `TestDaily_LateFireIsSkipped`. | A | `EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt` `CT/internal/nightchain/nightchain_test.go:92` `CT/internal/scheduler/scheduler_test.go:87` |
**Hand step in a runbook?** no hand step
### CC02 — Second-drive copy leg · CUSTOMER · Night · MENTIONED
architecture folder: 238 lines in 20 files match `/tier.?2|second.drive|cross.drive/` (a machine count, not a reading) · helper section: clock-controller „2. Second-drive copy leg at W+60 min”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | `RunAllTier2` copies each app's data to the second drive; owner: controller; started by the daily job "tier2-backup". CUSTOMER. | A | `CT/internal/backup/tier2.go:534` `CT/cmd/controller/main.go:1442` |
| 2 | Daily at W+60 min, `tier2OffsetMin`; no setting of its own. It does not wait for the dump leg: "NOTHING waits on the leg before it". Zone as block 1. | A | `CT/internal/backupwindow/backupwindow.go:17` `AR/09-update-architecture.md:2751-2757` |
| 3 | One copy: 60 min, `tier2CopyTimeout`, applied at. No limit for the whole leg found (same grep as block 1). Grows with app count and data size. | A | `CT/internal/backup/tier2_copy_lifecycle.go:40` `CT/internal/backup/tier2.go:828` |
| 4 | Catch-up as block 4 (the leg is in the ledger through `withLeg`). The leg function always returns nil, so the scheduler never sees a failure. | A | `CT/cmd/controller/main.go:1438-1441` |
| 5 | It waits for the dump leg by a start time only (60 minutes later). It is skipped while a migration runs. The file holds no lock of its own. With a backup time between 02:00 and 02:59 on the spring hour-change night the two legs start at the same instant; that case is row R-1260. | N | `CT/cmd/controller/main.go:1321-1335` `CT/internal/backup/tier2.go:538-541` `CT/internal/backup/tier2_copy_lifecycle.go:172` · class W3 · S14 |
| 6 | As block 1: wall-clock slot, monotonic wait, 60 min late limit. | A | `CT/internal/scheduler/scheduler.go:67` |
| 7 | The household and the hub are told of a failed copy: the copy's result goes to the notifier. | A | `CT/cmd/controller/main.go:1427` |
| 8 | MEASURED 2026-11-11: 39 s · 15 s · 11 s · SOURCE; catch-up part TEST `TestCatchUp_PowerCutMidChain_FinishesTheRest`. | A | `EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt` `CT/internal/backup/tier2.go:534` `CT/internal/nightchain/nightchain_test.go:145` |
**Hand step in a runbook?** no hand step
### CC03 — Off-site push leg, with the clean-up of old snapshots · CUSTOMER · Night · MENTIONED
architecture folder: 82 lines in 13 files match `/offbox|off.site push|offsite push/` (a machine count, not a reading) · helper section: clock-controller „3. Off-site push leg at W+105 min (with clean-up of old snapshots)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | `RunOffboxBackup` pushes dumps to the remote store, then runs retention; owner: controller; started by the daily job "offbox-backup". CUSTOMER. | A | `CT/internal/backup/offbox.go:1018` `CT/cmd/controller/main.go:1641-1643` |
| 2 | Daily at W+105 min, `offboxOffsetMin`. Retention keeps 7 daily, `keepDaily`. Zone as block 1. | A | `CT/internal/backupwindow/backupwindow.go:18` `CT/internal/backup/offbox_window.go:47` |
| 3 | Per app 2 h, `offboxBackupTimeout`, applied at; SSH connect 10 s. Grows with data changed and line speed. One real night: 3 min 28 s. | A | `CT/internal/backup/offbox.go:39` `CT/internal/backup/offbox.go:1564` `CT/internal/backup/offbox.go:37` `AR/09-update-architecture.md:2751-2757` |
| 4 | Catch-up as block 4. No escrowed key, orphaned store or migration: skipped with an INFO log line and nil. | A | `CT/internal/backup/offbox.go:1040-1053` |
| 5 | When another backup holds the flag, the push is skipped with an INFO line and the leg is marked as ended. Only the hub's 48-hour check notices. A long second-drive leg can cost the night's off-site copy. | K | `CT/internal/backup/offbox.go:1054-1057` · R-1186 |
| 6 | As block 1. The clean-up window has a one-hour clock tolerance `offsiteGuardSkew`. | A | `CT/internal/backup/offbox_window.go:42` |
| 7 | Failure: `NotifyOffboxBackupFailed` with a cleaned message. The hub's daily deadline is the independent signal. | A | `CT/cmd/controller/main.go:1446-1452` `CT/cmd/controller/main.go:1524-1528` |
| 8 | MEASURED 2026-11-11: 3 min 09 s · 59 s · 38 s · SOURCE; DOC. | A | `EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt` `CT/internal/backup/offbox.go:1040-1057` `AR/07-backup-architecture.md:455-457` |
**Hand step in a runbook?** NO — no runbook says how to see or repair a skipped off-site night
### CC04 — Missed-night catch-up and the resume watch · CUSTOMER · Night · MENTIONED
architecture folder: 43 lines in 12 files match `/catch.?up|nightchain|missed night/` (a machine count, not a reading) · helper section: clock-controller „4. Missed-night catch-up and resume watch”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | `CatchUp.Evaluate` runs the backup legs a box missed; owner: controller; started at controller start and by the resume watch. CUSTOMER. | A | `CT/internal/nightchain/nightchain.go:281-299` `CT/cmd/controller/main.go:1676` `CT/cmd/controller/main.go:1677` |
| 2 | 15 min after the trigger, `DefaultDelay`; a leg due within 30 min is left alone, `leaveToNormal`. Values decided by CC. No setting. | A | `CT/internal/nightchain/nightchain.go:62` `CT/internal/nightchain/nightchain.go:64` `AR/07-backup-architecture.md:574-577` |
| 3 | As long as the legs it runs. It waits up to 2 h for a whole-guest backup, `quiesceWaitMax`, then drops with a WARN. | A | `CT/internal/nightchain/nightchain.go:66` `CT/internal/nightchain/nightchain.go:319` |
| 4 | If the catch-up itself does not run: banner after 26 h (block 5) and the hub's longer lines, 48 h without a dump. Two missed nights give one catch-up, not two. | A | `AR/07-backup-architecture.md:579-584` |
| 5 | One at a time: pending/running guard; leg lock. It never runs the update leg. | A | `CT/internal/nightchain/nightchain.go:288-292` `CT/cmd/controller/main.go:1321-1335` `CT/cmd/controller/main.go:1660-1664` |
| 6 | The watch reads a gap of more than 5 minutes in the wall clock as 'the box was asleep'. A clock that is corrected forward by more than 5 minutes looks the same. The design text says this was not measured live. | N | `CT/internal/nightchain/nightchain.go:375` `CT/cmd/controller/main.go:1681` `AR/07-backup-architecture.md:551-559` · class W2 · S08 |
| 7 | Household: one timeline line, never mailed, event sent at. Dropped catch-up: log line only. | A | `AR/07-backup-architecture.md:562-564` `CT/internal/nightchain/nightchain.go:357-359` `CT/internal/nightchain/nightchain.go:319` |
| 8 | TEST `TestCatchUp_TwoMissedNights_One`; TEST `TestResumeWatch_SeesASuspend`; TEST `TestR871_CatchUpWiring`. | A | `CT/internal/nightchain/nightchain_test.go:126` `CT/internal/nightchain/nightchain_test.go:217` `CT/cmd/controller/r871_catchup_wiring_test.go:16` |
**Hand step in a runbook?** no hand step
### CC05 — Missed-backup banner (26 hours) · CUSTOMER · Night · MENTIONED
architecture folder: 48 lines in 14 files match `/26 h|missed.backup banner|banner/` (a machine count, not a reading) · helper section: clock-controller „5. Missed-backup banner (26 hours)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | `ComputeBanner` shows the household a banner when no night backup ran; owner: controller; computed when the page is drawn (no timer of its own). CUSTOMER. | A | `CT/internal/nightchain/banner.go:56-88` |
| 2 | Shown when the last backup is older than 26 h, `StaleAfter`. After 5 missed days it suggests another hour, `suggestMinDays`, `Suggest`. | A | `CT/internal/nightchain/banner.go:23` `CT/internal/nightchain/banner.go:26` `CT/internal/nightchain/banner.go:95-113` |
| 3 | Instant; reads the ledger only. | A | `CT/internal/nightchain/banner.go:56-88` |
| 4 | Nobody opens the page: nobody sees it. The hub's deadline check is the other channel. | A | `AR/07-backup-architecture.md:579-584` |
| 5 | Not a job; no lock needed. Dismissed, it comes back after a new miss, design at. | A | `AR/07-backup-architecture.md:566-572` |
| 6 | The age is wall-clock arithmetic. A clock that is set back hides the banner for as long as the step. No test sets the clock back. | N | `CT/internal/nightchain/banner.go:56-88` · class W2 · S08 |
| 7 | The household only, on the page; no mail. | A | `AR/07-backup-architecture.md:566-572` |
| 8 | TEST `TestBanner_ShownWithReasonAndSuggestion`; TEST `TestBanner_DismissedThenBackAfterANewMiss`. | A | `CT/internal/nightchain/banner_test.go:29` `CT/internal/nightchain/banner_test.go:59` |
**Hand step in a runbook?** no hand step
### CC06 — Whole-guest backup driver: stop apps, ask the agent, start apps · CUSTOMER · Night · MENTIONED
architecture folder: 264 lines in 21 files match `/quiesce|whole.guest|vzdump/` (a machine count, not a reading) · helper section: clock-controller „6. Whole-guest backup driver (stop apps, ask the host agent, start apps)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | `Loop.Run` stops the apps and asks the agent for the whole-guest backup; owner: controller drives, agent does the backup; started by `startQuiesceLoop`. CUSTOMER. | A | `CT/internal/quiesce/quiesce.go:191-206` `CT/cmd/controller/main.go:4008` |
| 2 | Polls every 5 min; runs once per 24 h cadence, only inside W+2h to W+6h, `gateOpenOffsetMin`, `gateSpanMin`. Zone `budapestLocation`. | A | `CT/internal/quiesce/quiesce.go:149-151` `CT/internal/quiesce/quiesce.go:161-163` `CT/internal/quiesce/quiesce.go:760` `CT/internal/quiesce/quiesce.go:761` |
| 3 | Apps stay down at most 30 min, `MaxQuiesce` default, setting `max_quiesce_duration`. Manual run limit = that + 5 min. Grows with guest disk size. | A | `CT/cmd/controller/main.go:3990` `CT/internal/config/config.go:132` `CT/internal/quiesce/quiesce.go:430` |
| 4 | Outside the window it still runs when the last one is older than cadence + 24 h (the 48 h valve). Apps left stopped by a crash are started by `Recover`, called at. | A | `CT/internal/quiesce/quiesce.go:801-818` `CT/internal/quiesce/quiesce.go:177-188` `CT/cmd/controller/main.go:4007` |
| 5 | Single-flight `TryLock`; waits for a catch-up; waits for the update leg, never past W+5h30 `legStepGraceMin`. | A | `CT/internal/quiesce/quiesce.go:213-216` `CT/internal/quiesce/quiesce.go:250-253` `CT/internal/quiesce/quiesce.go:271-274` `CT/internal/quiesce/quiesce.go:935` |
| 6 | The gate counts wall-clock minutes; the poll is a monotonic ticker. On the hour-change night the two disagree. No test covers that night. | K | `CT/internal/quiesce/quiesce.go:801-818` `CT/internal/quiesce/quiesce.go:191-206` · R-1260 |
| 7 | First failure notifies once `noteTierFailure`; retry back-off 15 min to 4 h, busy agent: retry 15 min, alarm after 3 h,. | A | `CT/internal/quiesce/quiesce.go:283-298` `CT/internal/quiesce/breaker.go:47` `CT/internal/quiesce/breaker.go:54` `CT/internal/quiesce/contention.go:53` |
| 8 | TEST `TestScheduledRunAllowed`; TEST `TestRunOnce_ValveRunsOutsideWindow`; TEST `TestBreaker_BackoffSchedule`. | A | `CT/internal/quiesce/quiesce_window_test.go:23` `CT/internal/quiesce/quiesce_window_test.go:94` `CT/internal/quiesce/breaker_test.go:218` |
**Hand step in a runbook?** no hand step
### CC07 — Automatic app-update leg · CUSTOMER · Night · MENTIONED
architecture folder: 136 lines in 17 files match `/update.leg|unattended/` (a machine count, not a reading) · helper section: clock-controller „7. Automatic app-update leg (after the off-site push)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | `RunUpdateLeg` presses one update step per app; owner: controller; started by `chainUpdateLeg` when the off-site half ends, on every path. CUSTOMER. | A | `CT/internal/stacks/unattended.go:230` `CT/cmd/controller/main.go:1641-1643` |
| 2 | Each night after the off-site leg; a box with no off-site tier runs it at W+105 min. One step per app per night, switch default ON. | A | `AR/07-backup-architecture.md:459-465` `AR/09-update-architecture.md:2806-2817` |
| 3 | No new step at or after W+5h: `LegDeadline`, `UpdateLegStopOffsetMin`; check at. Grows with the number of apps that have an update. | A | `CT/internal/stacks/unattended.go:212-216` `CT/internal/backupwindow/backupwindow.go:27` `CT/internal/stacks/unattended.go:374-375` |
| 4 | Missed night: no catch-up by design; the rest "waits for the next night". Controller restarted mid-leg: the rest of the night is lost (R-686). | A | `CT/cmd/controller/main.go:1660-1664` `CT/internal/stacks/unattended.go:374-375` `AR/09-update-architecture.md:2815-2817` |
| 5 | A passing refusal (busy, updating) is retried once after 2 min, `RetryWait`, rule at. The whole-guest gate waits for the leg. | A | `CT/internal/stacks/unattended.go:167-171` `AR/09-update-architecture.md:2882-2883` `CT/internal/quiesce/quiesce.go:271-274` |
| 6 | The leg's deadline is the backup time plus 5 hours of real time, and the scheduler's slots are in Budapest time. On the hour-change night a step can start after the backup gate stopped waiting. A daily fire that is more than 60 minutes late is skipped, so no update runs at noon. | K | `CT/internal/stacks/unattended.go:212-216` `CT/internal/scheduler/scheduler.go:17-27` `AR/07-backup-architecture.md:551-559` · R-1260 |
| 7 | Nobody. When the leg does not run, the only trace is one log line on the box. No event goes to the hub. | N | `CT/internal/stacks/unattended.go:374-375` · class W1 · S06 |
| 8 | TEST `TestChainUpdateLeg_EveryPath`; TEST `TestLegDeadlineAndGateShareW5h`. | A | `CT/cmd/controller/update_leg_wiring_test.go:19` `CT/internal/quiesce/d20_update_leg_test.go:78` |
**Hand step in a runbook?** no hand step
### CC08 — Off-site integrity check · CUSTOMER · Night · MENTIONED
architecture folder: 23 lines in 9 files match `/integrity/` (a machine count, not a reading) · helper section: clock-controller „8. Off-site integrity check at 06:00”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Daily job "offsite-integrity" checks the remote store's data; owner: controller. CUSTOMER. | A | `CT/cmd/controller/main.go:1724` |
| 2 | Fires daily at 06:00 (fixed, no setting for the hour) but works only when due: last good check older than 7 days, `defaultIntegrityMaxAgeDays`, `IntegrityDue`. Zone as block 1. | A | `CT/internal/backup/offbox_integrity.go:81` `CT/internal/backup/offbox_integrity.go:221` |
| 3 | MEASURED 2026-11-11: under 1 second on all three boxes (small stores). The check reads the whole store; the time on a large store rests on one measurement. | K | `EV/box/demo-hp-guest-night.txt` · R-401 |
| 4 | Box off at 06:00: caught up at the next 06:00 by due-ness; also run after a catch-up. | A | `CT/internal/backup/offbox_integrity.go:212-220` `CT/cmd/controller/main.go:1665-1668` |
| 5 | Skipped when "a backup or restore is already running"; then it waits for the next 06:00 (source comment). | A | `CT/internal/backup/offbox_integrity.go:298` `CT/cmd/controller/main.go:1710-1723` |
| 6 | Due-ness compares wall-clock stamps; a never-checked or unreadable stamp counts as due. Slot behaviour as block 24. | A | `CT/internal/backup/offbox_integrity.go:231` `CT/internal/backup/offbox_integrity.go:223-229` |
| 7 | A failed check goes to the notifier: the household and the hub are told. | A | `CT/cmd/controller/main.go:4144-4186` |
| 8 | MEASURED 2026-11-11: under 1 s on each box · SOURCE; TEST `TestR1189d_TheChecksRunAfterTheCatchUpLegs`. | A | `EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt` `CT/internal/backup/offbox_integrity.go:212-232` `CT/internal/nightchain/r1189d_test.go:13` |
**Hand step in a runbook?** no hand step
### CC09 — Off-site proof restore · CUSTOMER · Night · MENTIONED
architecture folder: 136 lines in 21 files match `/proof restore|offsite.proof|proof/` (a machine count, not a reading) · helper section: clock-controller „9. Off-site proof restore at 05:30”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Daily job "offsite-proof" restores one due snapshot to prove it opens, `ProveOffboxUnit`; owner: controller. CUSTOMER. | A | `CT/cmd/controller/main.go:1746` `CT/internal/backup/offbox_proof.go:140` |
| 2 | Daily at 05:30, fixed, no setting. One app per night in rotation (comment). Zone as block 1. | A | `CT/cmd/controller/main.go:1746` `CT/internal/backup/offbox_proof.go:265` |
| 3 | Limit 10 min, `proofRestoreTimeout`. Grows with the size of the one unit. | A | `CT/internal/backup/offbox_proof.go:80` |
| 4 | Box on for the backup legs but off at 05:30: "that day's proof is not made up". Made up only together with a missed backup leg. | A | `CT/cmd/controller/main.go:1660-1664` `CT/cmd/controller/main.go:1665-1668` |
| 5 | MEASURED 2026-11-11: on Tester 1 and on the N100 the whole-guest backup started at about 05:30 local time, the same minute as this job. This job is not behind the whole-guest gate. It takes the same flag as the off-site push; a meeting there is a silent skip (R-1186). | N | `EV/box/tester-1-host-night.txt` `EV/box/demo-felhom-host-night.txt` `EV/box/tester-1-guest-night.txt` · class W3 · S12 |
| 6 | As block 24. | A | `CT/cmd/controller/main.go:1746` `CT/internal/backup/offbox_proof.go:140` (the code path of question 1) |
| 7 | An empty or failed proof goes to the notifier: the household and the hub are told. | A | `CT/cmd/controller/main.go:4144-4186` |
| 8 | MEASURED 2026-11-11: 5 s · 5 s · 3 s · SOURCE; TEST `TestC5_TheWholeGuestBackupDoesNotWaitForTheChecks`. | A | `EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt` `CT/cmd/controller/main.go:1660-1664` `CT/internal/nightchain/r1189d_test.go:51` |
**Hand step in a runbook?** no hand step
### CC10 — Hub report: every 15 minutes, on change, and the hanging wait · CUSTOMER · Interval · MENTIONED
architecture folder: 91 lines in 22 files match `/15.min|hub report|pusher/` (a machine count, not a reading) · helper section: clock-controller „10. Hub report every 15 minutes (plus push on change and hanging wait)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Job "hub-report" sends the box state to the hub; owner: controller. The hub's answer carries the version floor and config. CUSTOMER and OPERATOR. | A | `CT/cmd/controller/main.go:1835` `CT/cmd/controller/main.go:1233-1290` |
| 2 | Every `PushInterval`, default "15m", fallback. Also 5 s after start, and on change with a 15 s floor, `triggerMinInterval`. No time zone. | A | `CT/internal/config/config.go:416` `CT/cmd/controller/main.go:1831-1834` `CT/cmd/controller/main.go:1932-1936` `CT/internal/report/trigger.go:30` |
| 3 | HTTP limit 30 s; 3 tries, 5 s apart. The wait connection holds up to 300 s. | A | `CT/internal/report/pusher.go:85` `CT/internal/report/pusher.go:114-117` `CT/internal/report/waiter.go:35` |
| 4 | A failed push is not queued: "next cycle reconciles". The hub notices silence (hub side, not read here). | A | `CT/internal/report/trigger.go:98` |
| 5 | A report still running is skipped. Bursts are merged after a 2 s quiet window. | A | `CT/internal/scheduler/scheduler.go:354-364` `CT/internal/report/trigger.go:27` |
| 6 | Monotonic ticker; a wrong wall clock does not change the rhythm, only the stamps inside the report. | A | `CT/internal/scheduler/scheduler.go:281-298` |
| 7 | Nobody on the box: the box stores the time of the last attempt only. The outside sign is the hub's liveness check (CH01). | A | `CT/internal/report/pusher.go:109-111` `HUB/monitor/host_staleness.go:59-60` |
| 8 | TEST `TestTrigger_BurstCoalescesTrailingEdge`; TEST `TestWaiter_FiresOnceOnGenChange`. | A | `CT/internal/report/trigger_test.go:114` `CT/internal/report/waiter_test.go:60` |
**Hand step in a runbook?** no hand step
### CC11 — Controller self-update: check, floor, nightly auto · CUSTOMER · Interval · MENTIONED
architecture folder: 65 lines in 13 files match `/self.?update/` (a machine count, not a reading) · helper section: clock-controller „11. Controller self-update: check, floor, nightly auto”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | `CheckForUpdate` looks for a newer controller image; `MaybeAutoUpdate` moves a box below the floor. Owner: controller, the agent swaps the image. CUSTOMER and OPERATOR. | A | `CT/internal/selfupdate/updater.go:153` `CT/internal/selfupdate/updater.go:612` |
| 2 | Check every `CheckInterval`, default "6h", and 25 s after start. Floor: after every hub report. Auto: daily at `AutoUpdateTime`, default "04:30", only if `AutoUpdate`. | A | `CT/internal/config/config.go:417` `CT/cmd/controller/main.go:1981-1989` `CT/cmd/controller/main.go:1249-1252` `CT/internal/config/config.go:419` |
| 3 | The registry call has a 15-second limit. The swap of the container has no limit in the file, and no duration was measured in this run. | N | `CT/internal/selfupdate/updater.go:245` · class W4 · S15 |
| 4 | No catch-up needed for the floor: it is tried again on each report until done, but one attempt per floor value. A failed attempt at a floor is not repeated. | A | `CT/internal/selfupdate/updater.go:656-661` `CT/internal/selfupdate/updater.go:604-606` |
| 5 | The auto-update waits while a backup or an app update runs. Its default time, 04:30, is the minute at which the whole-guest gate opens for a 02:30 backup time. Nothing orders the two. | N | `CT/internal/config/config.go:419` `CT/internal/selfupdate/updater.go:533-544` · class W3 · S12 |
| 6 | The check uses a monotonic ticker. The nightly auto-update is a daily slot (see CC22). The switch is off unless the config turns it on; the default time is 04:30. | A | `CT/internal/config/config.go:278-279` `CT/internal/config/config.go:419` |
| 7 | Auto-update refused: WARN log only. A version the floor cannot compare: WARN once. | A | `CT/cmd/controller/main.go:1879-1881` `CT/internal/selfupdate/updater.go:639` |
| 8 | TEST `TestMaybeAutoUpdate_NoFlap`; TEST `TestMaybeAutoUpdate_BelowFloor_UpdatesToFloor`. | A | `CT/internal/selfupdate/floor_test.go:93` `CT/internal/selfupdate/floor_test.go:11` |
**Hand step in a runbook?** no hand step
### CC12 — Catalog sync: app templates from git · CUSTOMER · Interval · MENTIONED
architecture folder: 147 lines in 18 files match `/catalog sync|git.sync|sync/` (a machine count, not a reading) · helper section: clock-controller „12. Catalog sync (app templates from git)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | `Syncer.Start` pulls the app catalog; owner: controller; started at controller start. CUSTOMER. | A | `CT/internal/sync/sync.go:125` |
| 2 | Once at start, then every `SyncInterval`, default "15m"; a bad value falls back to 15 min. No time zone. | A | `CT/internal/sync/sync.go:140-143` `CT/internal/config/config.go:399` `CT/internal/sync/sync.go:131-135` |
| 3 | Each git command has its own limit. One whole sync has none, and no duration was measured. | N | `CT/internal/sync/sync.go:724` `CT/internal/sync/sync.go:215-216` · class W4 · S15 |
| 4 | A missed sync is done at the next tick. A sync that fails for days writes an ERROR line on the box at each tick. Nobody is told. | N | `CT/internal/sync/sync.go:236` `CT/internal/sync/sync.go:247` · class W1 · S23 |
| 5 | Two syncs cannot run at once: `doSync` takes the syncer's lock first. The 30-second guard is for the button only. | A | `CT/internal/sync/sync.go:215-216` `CT/internal/sync/sync.go:153-155` |
| 6 | Monotonic ticker; own ticker, not the shared scheduler, so it has no "still running" skip from. | A | `CT/internal/sync/sync.go:146` `CT/internal/scheduler/scheduler.go:354-364` |
| 7 | Log line per sync only. | A | `CT/internal/sync/sync.go:155` |
| 8 | SOURCE. | A | `CT/internal/sync/sync.go:124-159` |
**Hand step in a runbook?** no hand step
### CC13 — System health check and base-stack self-heal · CUSTOMER · Interval · MENTIONED
architecture folder: 57 lines in 16 files match `/health check|self.heal/` (a machine count, not a reading) · helper section: clock-controller „13. System health check and base-stack self-heal”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Job "system-health" checks disk, memory and base services, and puts the base stack back; owner: controller. CUSTOMER. | A | `CT/cmd/controller/main.go:981-1006` |
| 2 | Every `SystemHealthInterval`, default "5m", fallback; env override. First run one interval after start. | A | `CT/internal/config/config.go:409` `CT/cmd/controller/main.go:977-980` `CT/internal/config/config.go:453` `CT/internal/scheduler/scheduler.go:281-298` |
| 3 | No time limit in the source, and no duration was measured. Nobody knows how long it takes on a big box. | N | `CT/cmd/controller/main.go:981-1006` · class W4 · S15 |
| 4 | No catch-up; next tick. A controller that is down checks nothing; the hub's missing reports are the outside sign (block 10). | A | `CT/cmd/controller/main.go:981-1006` (the code path of question 1) |
| 5 | Skipped while the previous run is still going. | A | `CT/internal/scheduler/scheduler.go:354-364` |
| 6 | Monotonic ticker; not sensitive to wall clock. | A | `CT/cmd/controller/main.go:981-1006` (the code path of question 1) |
| 7 | `NotifyHealthChange` on a change of state. | A | `CT/cmd/controller/main.go:1004` |
| 8 | SOURCE. | A | `CT/cmd/controller/main.go:981-1006` |
**Hand step in a runbook?** no hand step
### CC14 — Dead-app scan · CUSTOMER · Interval · MENTIONED
architecture folder: 62 lines in 13 files match `/stack.scan|dead.app|crash.?loop/` (a machine count, not a reading) · helper section: clock-controller „14. Dead-app scan every 30 seconds”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Job "deadapp-check" finds apps that should run and do not; owner: controller. CUSTOMER. | A | `CT/cmd/controller/main.go:1086-1092` |
| 2 | Every 30 s; silent for 90 s after start, about 15 min after a crash boot. | A | `CT/cmd/controller/main.go:1087-1088` |
| 3 | No time limit in the source, and no duration was measured. Nobody knows how long it takes on a big box. It grows with the number of apps. | N | `CT/cmd/controller/main.go:1086-1143` · class W4 · S15 |
| 4 | Next tick; no catch-up needed. | A | `CT/cmd/controller/main.go:1086-1092` (the code path of question 1) |
| 5 | Skipped if still running. It reads the whole-guest loop state so a planned stop is not an alarm. | A | `CT/internal/scheduler/scheduler.go:354-364` `CT/cmd/controller/main.go:1090` |
| 6 | The quiet time after a start uses the wall clock. A clock that steps back after the start makes the quiet time longer. No test. | N | `CT/cmd/controller/main.go:1087` · class W2 · S08 |
| 7 | Alert on the page via `SetDeadAppAlerts`. | A | `CT/cmd/controller/main.go:1091` |
| 8 | SOURCE. | A | `CT/cmd/controller/main.go:1086-1092` |
**Hand step in a runbook?** no hand step
### CC15 — Disk-fill watch · CUSTOMER · Interval · MENTIONED
architecture folder: 4 lines in 3 files match `/fill.?watch|disk.fill/` (a machine count, not a reading) · helper section: clock-controller „15. Disk-fill watch”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | `fillWatcher.Check` watches how full the drives are; owner: controller; three starters. CUSTOMER. | A | `CT/cmd/controller/main.go:1804-1805` |
| 2 | Every 10 min, `fillWatchInterval`; daily at 03:30; once 90 s after start, `fillWatchStartupDelay`. Cadence stated at. | A | `CT/cmd/controller/main.go:2690` `CT/cmd/controller/main.go:1804` `CT/cmd/controller/main.go:2685` `CT/cmd/controller/main.go:1772` |
| 3 | No time limit in the source, and no duration was measured. Nobody knows how long it takes on a big box. | N | `CT/cmd/controller/main.go:1804-1805` · class W4 · S15 |
| 4 | Next 10-minute tick. | A | `CT/cmd/controller/main.go:1804-1805` (the code path of question 1) |
| 5 | The night slot and the interval job are two jobs with two names. The watcher holds one lock, so the two cannot change its state at the same time. | A | `CT/internal/fillwatch/fillwatch.go:162` `CT/cmd/controller/main.go:1804-1805` |
| 6 | Interval is monotonic; the 03:30 slot as block 24. | A | `CT/cmd/controller/main.go:1804-1805` (the code path of question 1) |
| 7 | The household: the watcher pushes a customer event when a disk fills. | A | `CT/internal/fillwatch/fillwatch.go:80-153` |
| 8 | TEST `TestFillWatchRunsOnAnInterval`. | A | `CT/cmd/controller/r363_fillwatch_interval_test.go:17` |
**Hand step in a runbook?** no hand step
### CC16 — Drive health check, hourly · CUSTOMER · Interval · MENTIONED
architecture folder: 15 lines in 6 files match `/SMART|disk.health|drive health/` (a machine count, not a reading) · helper section: clock-controller „16. Drive health check, hourly”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Job "disk-health-check" asks the agent for drive health; owner: controller reads, agent measures. CUSTOMER. | A | `CT/cmd/controller/main.go:2226` |
| 2 | Every 1 h; reason for hourly, not 6 h:. Only when the agent channel is configured. | A | `CT/cmd/controller/main.go:2219-2224` `CT/cmd/controller/main.go:2225` |
| 3 | One fetch was measured at about 0.8 s (a comment in the source). No time limit in the source. | N | `CT/cmd/controller/main.go:2219-2224` · class W4 · S15 |
| 4 | Next hour; first run one hour after start. | A | `CT/internal/scheduler/scheduler.go:281-298` |
| 5 | Skipped if still running. | A | `CT/internal/scheduler/scheduler.go:354-364` |
| 6 | Monotonic ticker. | A | `CT/cmd/controller/main.go:2226` (the code path of question 1) |
| 7 | The household and the hub: the check emits `disk_health_degraded`. | A | `CT/internal/web/disk_health.go:175-280` |
| 8 | SOURCE. | A | `CT/cmd/controller/main.go:2219-2226` |
**Hand step in a runbook?** no hand step
### CC17 — Agent channel health, every 60 seconds · CUSTOMER · Interval · MENTIONED
architecture folder: 2 lines in 2 files match `/agent channel|channel health/` (a machine count, not a reading) · helper section: clock-controller „17. Agent channel health, every 60 seconds”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Job "agent-channel-health" probes the controller-to-agent channel; owner: controller. OPERATOR. | A | `CT/cmd/controller/main.go:2289` |
| 2 | Every 60 s. A passing fault needs 2 misses in a row, `debounceThreshold`. | A | `CT/internal/channelhealth/checker.go:29` |
| 3 | Probe limit 15 s, `probeTimeout`. | A | `CT/internal/channelhealth/checker.go:32` |
| 4 | Next minute. | A | `CT/cmd/controller/main.go:2289` (the code path of question 1) |
| 5 | Skipped if still running. An alert marker file survives a restart. | A | `CT/internal/scheduler/scheduler.go:354-364` `CT/cmd/controller/main.go:2288` |
| 6 | Monotonic ticker. | A | `CT/cmd/controller/main.go:2289` (the code path of question 1) |
| 7 | Sink = notifier and page alert. | A | `CT/cmd/controller/main.go:2286` |
| 8 | TEST `TestDebounce_TransientBlipSuppressed`. | A | `CT/internal/channelhealth/checker_test.go:119` |
**Hand step in a runbook?** no hand step
### CC18 — Held-app report to the hub · CUSTOMER · Interval · MENTIONED
architecture folder: 32 lines in 7 files match `/held.app|held app/` (a machine count, not a reading) · helper section: clock-controller „18. Held-app report to the hub”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | `Reporter.Tick` tells the hub which apps are held; owner: controller; job "hold-report". OPERATOR. | A | `CT/internal/holdreport/holdreport.go:89` `CT/cmd/controller/main.go:1081-1084` |
| 2 | Looks every minute; each hold is sent again once a day, `ResendEvery`, and after each start. | A | `CT/internal/holdreport/holdreport.go:36` `CT/cmd/controller/main.go:1071-1073` |
| 3 | Instant; reads the hold store. | A | `CT/internal/holdreport/holdreport.go:89` `CT/cmd/controller/main.go:1081-1084` (the code path of question 1) |
| 4 | Next minute. | A | `CT/internal/holdreport/holdreport.go:89` `CT/cmd/controller/main.go:1081-1084` (the code path of question 1) |
| 5 | Skipped if still running. | A | `CT/internal/scheduler/scheduler.go:354-364` |
| 6 | The daily resend is wall-clock arithmetic. A clock that is set back delays the resend by the size of the step. | N | `CT/cmd/controller/main.go:1082` · class W2 · S08 |
| 7 | The hub, by event. | A | `CT/cmd/controller/main.go:1074` |
| 8 | TEST `TestD64_TheHoldReporterTellsTheHubAfterTheClear`. | A | `CT/cmd/controller/holdclear_test.go:333` |
**Hand step in a runbook?** no hand step
### CC19 — Abandoned off-site store sweep · CUSTOMER · Night · MENTIONED
architecture folder: 21 lines in 9 files match `/abandon/` (a machine count, not a reading) · helper section: clock-controller „19. Abandoned off-site store sweep at 05:10”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Daily job "offsite-abandon-sweep" runs `AbandonSweep`; owner: controller. CUSTOMER. | A | `CT/cmd/controller/main.go:1693` `CT/internal/backup/offbox_abandon.go:214` |
| 2 | Daily at 05:10, fixed. The due date is counted in UTC days, the slot in Europe/Budapest. | A | `CT/internal/backup/offbox_abandon.go:366` `CT/internal/scheduler/scheduler.go:17-27` |
| 3 | One sweep has a 1-minute limit. | A | `CT/internal/backup/offbox_abandon.go:180` |
| 4 | Box off at 05:10: next day; no catch-up (it is not a ledger leg, see). | A | `CT/cmd/controller/main.go:1660-1664` |
| 5 | The sweep takes no lock. MEASURED: on the three boxes it ran 55 minutes after the off-site push ended, so they did not meet on these small stores. | N | `CT/internal/backup/offbox_abandon.go:180` `EV/box/demo-hp-guest-night.txt` · class W3 · S11 |
| 6 | As block 24. | A | `CT/cmd/controller/main.go:1693` `CT/internal/backup/offbox_abandon.go:214` (the code path of question 1) |
| 7 | Nobody. A sweep that fails writes a WARN or ERROR line on the box only. | N | `CT/internal/backup/offbox_abandon.go:180` · class W1 · S06 |
| 8 | MEASURED 2026-11-11: the slot ran on each box · SOURCE. | A | `EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt` `CT/cmd/controller/main.go:1693` |
**Hand step in a runbook?** no hand step
### CC20 — Restore-copy sweep and conversion-copy release · CUSTOMER · Interval · MENTIONED
architecture folder: 14 lines in 3 files match `/restore.copy|restore copy/` (a machine count, not a reading) · helper section: clock-controller „20. Restore-copy sweep and conversion-copy release”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | `SweepUnusedRestoreCopies` deletes old kept copies; job "restore-copy-sweep"; job "conversion-copy-release". CUSTOMER. | A | `CT/internal/backup/restore_copy_sweep.go:36` `CT/cmd/controller/main.go:1414` `CT/cmd/controller/main.go:1407` |
| 2 | The sweep runs each 6 hours; the release each hour. A restore copy that nobody used goes after 30 days (the constant, D62). A comment in the wiring still says 14 days; the comment is stale. | A | `CT/cmd/controller/main.go:1412` |
| 3 | No time limit in the source, and no duration was measured. Nobody knows how long it takes on a big box. | N | class W4 · S15 |
| 4 | The first run is 6 hours after the controller starts. A box that is on for under 6 hours a day never runs it; old restore copies stay. Nobody is told. | N | `CT/internal/scheduler/scheduler.go:281-298` · class W1 · S25 |
| 5 | Skipped if still running. | A | `CT/internal/scheduler/scheduler.go:354-364` |
| 6 | Age is wall-clock arithmetic on `now`. | A | `CT/internal/backup/restore_copy_sweep.go:36` |
| 7 | The household is told when a restore copy is deleted. | A | `CT/internal/notify/notifier.go:1601` |
| 8 | SOURCE. | A | `CT/internal/backup/restore_copy_sweep.go:29` |
**Hand step in a runbook?** no hand step
### CC21 — Upload and CPU watch, torrent limit rule, line-test hold · CUSTOMER · Interval · MENTIONED
architecture folder: 70 lines in 16 files match `/upwatch|unexplained|upload/` (a machine count, not a reading) · helper section: clock-controller „22. Upload and CPU watch, torrent limit rule, line-test hold”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | `startUpwatch` runs the minute watcher and a 15-minute rule that holds the torrent app for the line test; owner: controller. CUSTOMER. | A | `CT/cmd/controller/upwatch_wiring.go:136` |
| 2 | Watcher every 1 min; rule every 15 min. Baseline: 7 days of data, cached 24 h. | A | `CT/cmd/controller/upwatch_wiring.go:139` `CT/cmd/controller/upwatch_wiring.go:141` `CT/cmd/controller/upwatch_wiring.go:124-127` |
| 3 | Hold call limit 30 s, `holdTimeout`; drain wait at most 90 s. | A | `CT/internal/upwatch/linehold.go:79` `CT/internal/upwatch/linehold.go:216` |
| 4 | A limit left at the test value by a stopped controller is put back at start, `RecoverHold`. | A | `CT/cmd/controller/upwatch_wiring.go:137` |
| 5 | Two loops with their own tickers. The file holds no lock between them. | N | class W3 · S14 |
| 6 | Monotonic tickers. | A | `CT/internal/upwatch/watch.go:151` |
| 7 | The household and the hub: the watch emits `app_unexplained_upload` and `app_unexplained_cpu`. | A | `CT/internal/upwatch/watch.go:309-382` |
| 8 | SOURCE. | A | `CT/cmd/controller/upwatch_wiring.go:134-152` |
**Hand step in a runbook?** no hand step
### CC22 — The scheduler itself: daily slots and interval jobs · CUSTOMER · Night · MENTIONED
architecture folder: 24 lines in 10 files match `/scheduler|daily slot/` (a machine count, not a reading) · helper section: clock-controller „24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | `Daily` and `Every` carry every named job; owner: controller. CUSTOMER. | A | `CT/internal/scheduler/scheduler.go:131-157` `CT/internal/scheduler/scheduler.go:105-127` |
| 2 | Daily slots are HH:MM in Europe/Budapest, hard-coded; if the zone data cannot load it silently uses UTC. Interval jobs do not run at start: first run after one interval. | A | `CT/internal/scheduler/scheduler.go:17-27` `CT/internal/scheduler/scheduler.go:281-298` |
| 3 | No time limit of its own; shutdown waits 30 s. A panic in a job is caught. | A | `CT/internal/scheduler/scheduler.go:264` `CT/internal/scheduler/scheduler.go:438-447` |
| 4 | A daily job has no catch-up in the scheduler; a fire over 60 min late is skipped. Only the three backup legs are made up (block 4). | A | `CT/internal/scheduler/scheduler.go:344-347` |
| 5 | One run per job at a time: "still running" skip. No rule between different jobs here. | A | `CT/internal/scheduler/scheduler.go:354-364` |
| 6 | Slots are in Budapest time. On the spring night 02:30 does not exist; on the autumn night it comes twice. No test covers either night. A clock set back while a timer waits does not matter: the timer runs on monotonic time. | K | `CT/internal/scheduler/scheduler.go:499-519` · R-1260 |
| 7 | A skipped late fire is a log line only. | A | `CT/internal/scheduler/scheduler.go:344-347` |
| 8 | SOURCE. MEASURED 2026-11-11: the night's order and the start times on three boxes (see the README's night table). | A | `CT/internal/scheduler/scheduler.go:344-347` `EV/box/demo-hp-guest-night.txt` |
**Hand step in a runbook?** no hand step
### CH01 — Box liveness: 'your server cannot be reached' · CUSTOMER · Hub sweep · MENTIONED
architecture folder: 50 lines in 16 files match `/host_stale|liveness|staleness/` (a machine count, not a reading) · helper section: clock-hub „1. Box liveness: "your server cannot be reached"”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER. The hub's 60 s sweep runs `stalenessChecker.Check` and `hostStaleness.Check`; they raise `host_stale`, `host_down`, `host_recovered`. Owner: hub pod. Merged here: the controller-report checker and the host-report checker. | A | `E/hub/cmd/hub/main.go:935-936` `HUB/monitor/host_staleness.go:207-221` |
| 2 | Every 60 s. Stale after `stale_threshold`, default `"30m"`, deployed 45 m; down after 2x. No time zone: ages are durations. | A | `E/hub/cmd/hub/main.go:927` `E/hub/cmd/hub/main.go:1027` `E/manifests/hub.yaml:88` `HUB/monitor/host_staleness.go:39` |
| 3 | No time limit. The checkers run one after the other in one loop: one read of all hosts at each pass, then a loop over the hosts. The time grows with the number of boxes. Nobody measured one pass. | N | `E/hub/cmd/hub/main.go:935-962` `HUB/monitor/host_staleness.go:113` · class W4 · S16 |
| 4 | With the hub down nothing is judged. At the start, a box that is already silent gets one alarm at the first pass, unless that alarm was raised before. The controller's checker starts silent for such a box. | A | `HUB/monitor/host_staleness.go:59-68` `HUB/monitor/staleness.go:86-91` |
| 5 | One pod, `replicas: 1`, `Recreate`; state under a mutex. Events fire on transitions only. A household power-off holds alarms for `DeliberateOffHold`. | A | `E/manifests/hub.yaml:122` `E/manifests/hub.yaml:123-124` `HUB/monitor/host_staleness.go:119-120` `HUB/monitor/host_staleness.go:170-172` |
| 6 | Judged on the hub's receive time, not the box clock: `last_report_at = datetime('now')`. A wrong box clock cannot hide silence. A hub clock jump moves every age at once; DST has no effect (durations). | A | `HUB/store/store.go:3916` `HUB/monitor/host_staleness.go:131` |
| 7 | Operator: every edge; liveness events bypass the 1 h quiet hour, 5 min dedupe. Household: at most once per 7 days, `householdLivenessQuiet`. If the hub itself is down nobody is told by this job. | A | `HUB/notify/dispatcher.go:609-612` `HUB/notify/dispatcher.go:1018` |
| 8 | TEST `TestHostStaleness_R1001_RestartDoesNotReMail`; TEST `TestR873_HouseholdHearsAnOutageAtMostWeekly`; DOC. | A | `HUB/monitor/r1001_restart_test.go:56` `HUB/notify/r873_liveness_weekly_test.go:17` `AR/05-hub-architecture.md:88-118` |
**Hand step in a runbook?** no hand step
### CH02 — Missed-backup deadline check (05:00) · CUSTOMER · Hub daily · MENTIONED
architecture folder: 28 lines in 11 files match `/deadline check|backup.deadline|05:00/` (a machine count, not a reading) · helper section: clock-hub „2. Missed-backup deadline check (05:00)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER. `scheduleDaily(ctx, "deadline-check", "05:00",...)` runs `CheckBackupDeadlines`; it raises `expected_backup_missed`. Owner: hub pod. | A | `E/hub/cmd/hub/main.go:986-988` `HUB/monitor/deadline.go:409` `HUB/monitor/deadline.go:539-544` |
| 2 | Once a day at 05:00 Budapest wall-clock; the slot is a literal in, no setting. Zone chosen in `scheduleDaily`. Limits: `backupStaleAfter` 26 h, off-site `offsiteBackupStaleAfter` 8 d. | A | `E/hub/cmd/hub/main.go:986` `E/hub/cmd/hub/main.go:1072` `HUB/monitor/deadline.go:29` `HUB/monitor/deadline_tiers.go:28` |
| 3 | No time limit; the check runs inside the scheduler's own loop. It reads up to 7 days of evidence for every customer. The time grows with boxes times reports. Nobody measured it. | N | `E/hub/cmd/hub/main.go:1090` · class W4 · S16 |
| 4 | When the hub is down at 05:00, that day's check is skipped. The 'last run' value lives in memory and the next slot is the next day. Nobody notices. The next day's check uses longer limits, so a missed backup is seen one day late. | N | `E/hub/cmd/hub/main.go:1069-1112` · class W1 · S01 |
| 5 | Cannot run twice for one slot: the `last` guard in `nextDailyRun`. No lock against the 60 s sweep; they are separate goroutines and. A down box is judged on longer lines, 48 h / 72 h. | A | `E/hub/cmd/hub/main.go:1095-1112` `E/hub/cmd/hub/main.go:927` `E/hub/cmd/hub/main.go:986` `HUB/monitor/deadline.go:340-343` |
| 6 | The check compares the BOX's backup time with the hub's clock. A box whose clock runs ahead looks fresh. No guard in the file. (R-1206 is the same cause seen on the box's own pages.) | N | `HUB/monitor/deadline.go:243` · class W2 · S07 |
| 7 | Household and operator through the dispatcher on `expected_backup_missed`; at the 05:00 run only. A box that never bound is "unknown", not alarmed; disabled / deliberately-off are skipped. | A | `HUB/monitor/deadline.go:539-544` `HUB/monitor/deadline.go:486-494` `HUB/monitor/deadline.go:432-442` |
| 8 | TEST for a down box and for 'the same slot never runs twice'. A skipped day: NOBODY KNOWS. | N | `HUB/monitor/r872_down_box_test.go:57` `E/hub/cmd/hub/r1183_dst_schedule_test.go:82` · class W1 · S01 |
**Hand step in a runbook?** NO — no runbook names a hub that was down at 05:00
### CH03 — Mail cooldowns and the weekly outage mail · CUSTOMER · Hub sweep · MENTIONED
architecture folder: 38 lines in 8 files match `/cooldown/` (a machine count, not a reading) · helper section: clock-hub „3. Mail cooldowns and the weekly outage mail”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER. Not a job but a clock rule inside every mail: `processCustomer` and `processOperator` decide whether an alarm is mailed now or swallowed. Owner: hub dispatcher. | A | `HUB/notify/dispatcher.go:1020-1098` `HUB/notify/dispatcher.go:626-696` |
| 2 | Operator: `operatorCooldown` 1 h, liveness 5 min, storage fill `storageFillCooldown` 6 h. Household: default 6 h; outage mails once per 7 days. First hour of a new box is suppressed, `firstHourOfABox`. | A | `HUB/notify/dispatcher.go:609-612` `HUB/notify/dispatcher.go:573` `HUB/notify/dispatcher.go:1065-1069` `HUB/notify/dispatcher.go:1054-1062` |
| 3 | Each mail call has a 10-second limit. The cooldown lists are in memory; an entry is removed when its alarm clears. | A | `HUB/notify/dispatcher.go:86` `HUB/notify/dispatcher.go:684` `HUB/notify/dispatcher.go:1182` |
| 4 | Pod restart: the cooldown maps are in memory, so quiet periods restart from zero; only the 7-day household rule is read from the persisted notification log. Design text:. | A | `HUB/notify/dispatcher.go:1054-1062` `AR/08-alarm-ladder.md:435-439` |
| 5 | The operator cooldown is armed before the send, so two alarms close together collapse to one; a suppression is recorded. A MAIL-HOLD drops the mail. | A | `HUB/notify/dispatcher.go:674` `HUB/notify/dispatcher.go:665-672` `HUB/notify/dispatcher.go:680-687` |
| 6 | Cooldowns use the hub's wall clock only. A jump back stretches a quiet period; a jump forward ends it early. Nothing handles it. | N | `HUB/notify/dispatcher.go:1` · class W2 · S10 |
| 7 | A swallowed operator alarm leaves a 'suppressed' row. A household mail that fails is logged and is not sent again; only operator mails are retried. | N | `HUB/notify/dispatcher.go:1091-1094` · class W1 · S02 |
| 8 | TEST `TestOperatorCooldown_NodeLivenessBypassesQuietHour`; TEST `TestR873_HouseholdHearsAnOutageAtMostWeekly`; DOC. | A | `HUB/notify/node_liveness_cooldown_test.go:17` `HUB/notify/r873_liveness_weekly_test.go:17` `AR/08-alarm-ladder.md:165-168` |
**Hand step in a runbook?** no hand step
### CH04 — Retry of a failed operator mail · OPERATOR · Hub sweep · MENTIONED
architecture folder: 27 lines in 10 files match `/retry/` (a machine count, not a reading) · helper section: clock-hub „4. Retry of a failed operator mail”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR. `retryOperatorEmail` re-sends an operator mail whose send failed; started by the failure branch of `processOperator`. | A | `HUB/notify/dispatcher.go:99-104` `HUB/notify/dispatcher.go:688-692` |
| 2 | After 1, 5 and 15 minutes: `operatorRetryDelays`, armed with `afterFn = time.AfterFunc`. No setting. Durations, no time zone. | A | `HUB/notify/dispatcher.go:76` `HUB/notify/dispatcher.go:92` |
| 3 | Each try is bounded by the 10 s client timeout. One timer per failed mail; grows only with failures. | A | `HUB/notify/dispatcher.go:86` |
| 4 | The retry timers are in memory. A hub restart inside the 21 minutes loses them, and nothing sends the mail later. After the third failure the hub writes one ERROR line. | N | `HUB/notify/dispatcher.go:90-102` · class W1 · S03 |
| 5 | The cooldown for that alarm was armed before the first send, so a repeat alarm in the same hour does not mail either; the retry is the only path. Explained in. | A | `HUB/notify/dispatcher.go:674` `AR/08-alarm-ladder.md:358-363` |
| 6 | The timers are durations; the hour change has no effect. A jump of the hub's wall clock: NOBODY KNOWS. | N | class W2 · S10 |
| 7 | Nobody is mailed when the retries give up. The trace is the ERROR line and the 'failed' rows in the notification log. | N | `HUB/notify/dispatcher.go:90-102` `AR/08-alarm-ladder.md:363` · class W1 · S03 |
| 8 | TEST for the retry. A restart during the retry window: NOBODY KNOWS. | N | `HUB/notify/operator_mail_retry_test.go:17` · class W1 · S03 |
**Hand step in a runbook?** no hand step
### CH05 — Kernel mail: 'your box restarts tonight' · CUSTOMER · Hub sweep · MENTIONED
architecture folder: 165 lines in 14 files match `/kernel/` (a machine count, not a reading) · helper section: clock-hub „5. Kernel "your box restarts tonight" mail”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER. `KernelNotify` mails each due household the day before a kernel step; called from `Evaluate`, which the 60 s OS ticker calls. | A | `HUB/osupdates/kernel.go:191` `HUB/osupdates/service.go:436` `E/hub/cmd/hub/main.go:564` |
| 2 | Every 60 s, but acts only 09:00-20:00 Budapest: `KernelNoticeFromHour`/`KernelNoticeToHour`, checked at. Valid 24 h, gap 20 h, at most 3 mails. Zone from `budapestLoc`. Constants, no setting. | A | `HUB/osupdates/kernel.go:46-47` `HUB/osupdates/kernel.go:193-196` `HUB/osupdates/kernel.go:48-50` `HUB/osupdates/kernel.go:73-78` |
| 3 | No time limit. Loops all hosts every minute with store reads per host; grows linearly with boxes, 1440 passes a day. | A | `HUB/osupdates/kernel.go:203-208` |
| 4 | Pod down all day 09-20: no mail, and without a mail there is no step: `KernelBlockFor` sets `Tonight` only for a notice younger than 24 h. It mails on the next daytime tick; the notices are persisted. | A | `HUB/osupdates/kernel.go:179-185` `HUB/osupdates/kernel.go:236` |
| 5 | Twice is blocked by the stored notice and the 20 h gap; a restart cannot double-mail because the row is in the store. A mail sent but not recorded means no step tonight. | A | `HUB/osupdates/kernel.go:208-211` `HUB/osupdates/kernel.go:236-239` |
| 6 | Hub clock decides the daytime window. A mail sent at 19:59 is valid until 19:59 next day, so it covers one night. DST: the hour test uses Budapest local time; the 20 h / 24 h are durations. A deliberately-off box gets no mail. | A | `HUB/osupdates/kernel.go:182` `HUB/osupdates/kernel.go:193-194` `HUB/osupdates/kernel.go:201` |
| 7 | Mail failure: operator event `os_kernel_notice` warning, once per box, kernel and day. After 3 mails with no step: operator warning, once. | A | `HUB/osupdates/kernel.go:226-233` `HUB/osupdates/kernel.go:212-219` |
| 8 | TESTS for 'no mail, no step'. A mail that goes out late in the day against the box's own night slot: NOBODY KNOWS. | N | `HUB/osupdates/kernel_test.go:192` `HUB/osupdates/kernel_test.go:79` `HUB/osupdates/kernel_test.go:115` · class W3 · S14 |
**Hand step in a runbook?** no hand step
### CH06 — OS-update approval tick and the hourly OS alarms · OPERATOR · Hub sweep · MENTIONED
architecture folder: 468 lines in 28 files match `/approval|ring/` (a machine count, not a reading) · helper section: clock-hub „6. OS update approval tick and hourly OS alarms”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR (the outcome reaches households as updates). 60 s ticker: `osSvc.Evaluate` approves a set when its rule holds; `osSvc.Alarms` raises the fleet alarms. | A | `E/hub/cmd/hub/main.go:554-586` `E/hub/cmd/hub/main.go:564` `E/hub/cmd/hub/main.go:577` |
| 2 | Evaluate every 60 s; Alarms when `time.Since(lastAlarms) >= time.Hour`. Rule default `ApproveAfter: 24 * time.Hour, NightsRequired: 1`, env `OS_APPROVE_AFTER`. Alarm limits 7 d / 14 d, env table. | A | `E/hub/cmd/hub/main.go:574` `E/hub/cmd/hub/main.go:476` `E/hub/cmd/hub/main.go:484-491` `E/hub/cmd/hub/main.go:536-541` |
| 3 | No time limit. The alarm pass logs its own duration. Evaluate walks every layer; grows with boxes times layers, every minute. | A | `E/hub/cmd/hub/main.go:581` `HUB/osupdates/service.go:414-422` |
| 4 | Pod restart: `lastAlarms` is a local variable, zero at start, so the first tick runs the alarms at once. Re-mail is prevented by the stored raise time, 7 days. | A | `E/hub/cmd/hub/main.go:558` `HUB/osupdates/service.go:1102` |
| 5 | One goroutine, so Evaluate and Alarms never overlap. `raise` clears the mark when the condition ends so the next occurrence is announced. | A | `E/hub/cmd/hub/main.go:559-585` `HUB/osupdates/service.go:1094-1108` |
| 6 | A box's claimed reboot time is allowed 1 minute of error. A larger error of the box's clock: NOBODY KNOWS. (The N100's host is 32 s behind today.) | N | `HUB/osupdates/service.go:1102` `HUB/osupdates/service.go:970` · class W2 · S07 |
| 7 | Operator, by event through the dispatcher: `os_update_stale`, `os_ring0_stalled` and others, listed in. Repeats weekly while it holds. | A | `AR/08-alarm-ladder.md:375-383` `HUB/osupdates/service.go:1092-1093` |
| 8 | TESTS and DOC. The cost of the pass each minute with 100 boxes: NOBODY KNOWS. | N | `HUB/osupdates/alarms_test.go:226` `AR/08-alarm-ladder.md:365` · class W4 · S16 |
**Hand step in a runbook?** no hand step
### CH07 — Off-site copy too old, and escrow never finished · CUSTOMER · Hub sweep · MENTIONED
architecture folder: 12 lines in 6 files match `/offsite_stale|off.site.{0,20}stale|48 h/` (a machine count, not a reading) · helper section: clock-hub „7. Off-site copy too old, and escrow never finished”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER. `offsiteChecker.Check` on the 60 s sweep; `isStale` and `checkEscrowPending`. Owner: hub. | A | `E/hub/cmd/hub/main.go:943` `HUB/monitor/offsite.go:166` `HUB/monitor/offsite_escrow_pending.go:66` |
| 2 | Every 60 s. Stale after `defaultOffsiteStaleAfter` 48 h, counted from the last SUCCESS. Escrow pending after `escrowPendingAfter` 7 d, re-raised each 7 d. | A | `HUB/monitor/offsite.go:54` `HUB/monitor/offsite.go:203-207` `HUB/monitor/offsite_escrow_pending.go:37` `HUB/monitor/offsite_escrow_pending.go:110` |
| 3 | No time limit. Reads every customer's latest report each minute; grows linearly with boxes. | A | `HUB/monitor/offsite.go:443-459` |
| 4 | With the hub down nothing is judged. After a start the state lists in memory are empty. If a tier that is still too old is mailed again after a restart: NOBODY KNOWS. | N | `HUB/monitor/offsite.go:463-468` `HUB/monitor/offsite.go:1` · class W1 · S03 |
| 5 | One mutex per pass; a customer listed twice is handled once. A never-ran tier is owned by the delivery checker, "one state, one owner". | A | `HUB/monitor/offsite.go:448-449` `HUB/monitor/offsite.go:453-458` `HUB/monitor/offsite.go:160-162` |
| 6 | The last-success time is the BOX's time, compared with the hub's clock. A box clock that is ahead hides an old copy. A time that cannot be read counts as too old. | N | `HUB/monitor/offsite.go:203-207` · class W2 · S07 |
| 7 | Dispatcher mail on the stale edge; `offsite_escrow_pending` is operator-only, see. | A | `HUB/monitor/offsite.go:636` `AR/08-alarm-ladder.md:383` |
| 8 | TESTS for the limits and for a newborn box. A box clock that is ahead: NOBODY KNOWS. | N | `HUB/monitor/offsite_r100_test.go:55` `HUB/monitor/offsite_r100_test.go:74` · class W2 · S07 |
**Hand step in a runbook?** no hand step
### CH08 — Restore-test proof too old · OPERATOR · Hub sweep · MENTIONED
architecture folder: 121 lines in 19 files match `/restore.test/` (a machine count, not a reading) · helper section: clock-hub „8. Restore-test proof too old”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER (protects the promise that a backup restores; mail goes by event). `restoreTestChecker.Check` on the 60 s sweep; `checkStaleness`. | A | `E/hub/cmd/hub/main.go:944` `HUB/monitor/restoretest.go:242` |
| 2 | Every 60 s. Window per tier: 4 archive generations, never under `restoreProvenWindowFloor` 7 d, never over `restoreProvenWindowCap` 12 d. Constants, no setting. | A | `HUB/monitor/restoretest.go:62` `HUB/monitor/restoretest.go:68` `HUB/monitor/restoretest.go:74` |
| 3 | No time limit. Each pass reads 14 days of host reports for each customer, each minute. It is the heaviest read on the sweep. Nobody measured it. | N | `HUB/monitor/restoretest.go:79` `HUB/monitor/restoretest.go:252` · class W4 · S16 |
| 4 | Pod restart: the first pass does not re-send a stale signal already in the events table, `staleAlreadyReported`. | A | `HUB/monitor/restoretest.go:281-283` `HUB/monitor/restoretest.go:300-312` |
| 5 | Edge-triggered per customer and tier under a mutex. A deliberately-off box is skipped. | A | `HUB/monitor/restoretest.go:270-278` `HUB/monitor/restoretest.go:192-194` |
| 6 | First contact is the hub's receive time. The proof times come from the box's reports. An error of the box's clock: NOBODY KNOWS. | N | `HUB/monitor/restoretest.go:326-342` `HUB/store/store.go:3994-4005` · class W2 · S07 |
| 7 | Event `restore_test_stale` warning through the dispatcher; a failed run is a separate event, `checkFailure`. | A | `HUB/monitor/restoretest.go:284-285` `HUB/monitor/restoretest.go:205` |
| 8 | TEST that a restart does not mail again. The cost with 100 boxes (14 days of reports per customer, each minute): NOBODY KNOWS. | N | `HUB/monitor/r1001_restart_test.go:147` · class W4 · S16 |
**Hand step in a runbook?** no hand step
### CH09 — Off-site clean-up window (20 minutes, weekly) · CUSTOMER · Hub sweep · MENTIONED
architecture folder: 11 lines in 5 files match `/clean.?up window|prune window|deleting key/` (a machine count, not a reading) · helper section: clock-hub „9. Off-site clean-up window (20 minutes, weekly)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER DATA. The box asks; `OpenWindowFor` adds a deleting key line for 20 minutes; the hub's 60 s sweep `SweepExpiredWindows` closes a window left open. Sweep loop:. | A | `HUB/offsitekeys/service.go:334` `HUB/offsitekeys/service.go:413` `E/hub/cmd/hub/main.go:636-650` |
| 2 | Granted when the last window is at least `windowCadence` 6 d 12 h old, test at; length `windowLength` 20 min. Sweep every 60 s. Constants; the weekly switch is a store setting. | A | `HUB/offsitekeys/service.go:264` `HUB/offsitekeys/service.go:337` `HUB/offsitekeys/service.go:263` `HUB/offsitekeys/service.go:338` |
| 3 | Each sweep has a 2-minute context. Work is per expired window, one remote call each; grows with boxes that crash mid-prune. | A | `E/hub/cmd/hub/main.go:644` `HUB/offsitekeys/service.go:419-422` |
| 4 | If the hub stops while a window is open, the key line that can delete stays on the storage account until the hub is back; the sweep then closes it. Nobody is told during the gap. | N | `HUB/offsitekeys/service.go:419-431` `HUB/offsitekeys/service.go:424-426` · class W1 · S01 |
| 5 | One window per cadence via the ledger row. The daily key audit knows an open window. No lock against the abandon sweep; both run in the same goroutine. | A | `HUB/offsitekeys/service.go:364` `HUB/offsitekeys/service.go:212-213` `E/hub/cmd/hub/main.go:636-650` |
| 6 | The window runs on the hub's clock; the box's clock does not decide. A hub clock that jumps back can keep a window open longer: NOBODY KNOWS. | N | `HUB/offsitekeys/service.go:337` · class W2 · S10 |
| 7 | Operator event `offsite_window_failed` warning when the hub had to close it; `offsite_window_drop` error when too much went. | A | `HUB/offsitekeys/service.go:430-431` `HUB/offsitekeys/service.go:259` |
| 8 | TEST `TestWindow_LeftOpenIsClosedByTheSweep`; TEST `TestWindow_GrantOpenCloseAndDropAlarm`; DOC. | A | `HUB/offsitekeys/service_test.go:99` `HUB/offsitekeys/service_test.go:37` `AR/07-backup-architecture.md:256` |
**Hand step in a runbook?** no hand step
### CH10 — Delayed deletion of a set-aside off-site copy · CUSTOMER · Hub sweep · MENTIONED
architecture folder: 51 lines in 16 files match `/set.aside|delayed delet|superseded/` (a machine count, not a reading) · helper section: clock-hub „10. Delayed deletion of a set-aside off-site copy”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER DATA. `SweepAbandons` deletes a household's set-aside copy once its request is due; called from the 60 s sweep. | A | `HUB/offsitekeys/service.go:529` `E/hub/cmd/hub/main.go:636-650` |
| 2 | Due `DefaultAbandonDelay` 7 days after the request; override `OFFSITE_ABANDON_DELAY`. Checked every 60 s. Hub clock. | A | `HUB/offsitekeys/service.go:448` `E/hub/cmd/hub/main.go:598-605` |
| 3 | One remote delete for each due request, in the same sweep as the window. How long a large delete takes: NOBODY KNOWS. | N | `E/hub/cmd/hub/main.go:644` `HUB/offsitekeys/service.go:536-538` · class W4 · S15 |
| 4 | Pod down at the due time: deleted on the first sweep after start; the due list is in the store. A failed delete is marked and retried every sweep. | A | `HUB/offsitekeys/service.go:530` `HUB/offsitekeys/service.go:540-547` |
| 5 | A request is deleted once, then marked. A cancel before the due time wins: design text. | A | `HUB/offsitekeys/service.go:549` `HUB/offsitekeys/service.go:438-441` |
| 6 | The hub's clock only. The length of the wait is one setting, read at the start. | A | `E/hub/cmd/hub/main.go:598-605` |
| 7 | Operator events: requested, cancelled, deleted, failed; failure mails once per request. | A | `HUB/offsitekeys/service.go:444-447` `HUB/offsitekeys/service.go:543-546` |
| 8 | TEST `TestAbandon_DelayCancelAndScope`. | A | `HUB/offsitekeys/service_test.go:134` |
**Hand step in a runbook?** no hand step
### CH11 — Off-site key audit (07:10) · OPERATOR · Hub daily · MENTIONED
architecture folder: 2 lines in 2 files match `/key audit|07:10/` (a machine count, not a reading) · helper section: clock-hub „11. Off-site key audit (07:10)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR (guards household copies against deletion). `scheduleDaily(ctx, "offsite-key-audit", "07:10",...)` runs `AuditAll`. | A | `E/hub/cmd/hub/main.go:651-655` `HUB/offsitekeys/service.go:195` |
| 2 | Daily 07:10 Budapest, literal slot. Zone from `scheduleDaily`. | A | `E/hub/cmd/hub/main.go:651` `E/hub/cmd/hub/main.go:1072` |
| 3 | The audit has a 10-minute limit and reads one customer after the other from the storage account. With 100 boxes the 10 minutes may not be enough; nobody measured it. | N | `E/hub/cmd/hub/main.go:652` `HUB/offsitekeys/service.go:202-215` · class W4 · S16 |
| 4 | When the hub is down at 07:10, that day's audit is skipped. No catch-up. Nobody notices. | N | `E/hub/cmd/hub/main.go:1069-1112` · class W1 · S01 |
| 5 | Runs once per day by the `last` guard. Meets an open clean-up window: the audit is told the window is open. | A | `E/hub/cmd/hub/main.go:1095-1112` `HUB/offsitekeys/service.go:212-213` |
| 6 | Wall-clock slot, DST-safe by `nextDailyRun`. | A | `E/hub/cmd/hub/main.go:1095-1112` |
| 7 | Operator: `offsite_key_unlocked` error, or `offsite_key_audit_failed` when a file cannot be read. | A | `HUB/offsitekeys/service.go:192-194` |
| 8 | TEST and DOC. The 10-minute limit at fleet size: NOBODY KNOWS. | N | `HUB/offsitekeys/offsitekeys_test.go:158` `AR/05-hub-architecture.md:642` · class W4 · S16 |
**Hand step in a runbook?** no hand step
### CH12 — Off-site credential stuck, and its self-heal · OPERATOR · Hub sweep · MENTIONED
architecture folder: 4 lines in 3 files match `/offsiteheal|credential.{0,30}(stuck|heal)/` (a machine count, not a reading) · helper section: clock-hub „12. Off-site credential stuck and its self-heal”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER. Two jobs, merged: `offsiteDeliveryChecker.Check` (60 s sweep) and the `offsiteheal` reconciler, started at. | A | `E/hub/cmd/hub/main.go:945` `HUB/offsiteheal/reconciler.go:165-177` `E/hub/cmd/hub/main.go:846-854` |
| 2 | Stuck after `stuckAfter` 1 h; cooldowns 24 h. Heal sweep every `tickIntervalDefault` 5 min, acts after 2 distinct reports. | A | `HUB/monitor/offsite_delivery.go:38` `HUB/monitor/offsite_delivery.go:40-41` `HUB/offsiteheal/reconciler.go:74` `HUB/offsiteheal/reconciler.go:70` |
| 3 | Heal call limit 3 minutes. Both list every customer config per pass and; linear in boxes. | A | `HUB/monitor/offsite_delivery.go:275` `HUB/monitor/offsite_delivery.go:72` `HUB/offsiteheal/reconciler.go:183` |
| 4 | Pod down: nothing; on return the next tick re-reads state. The cooldowns are "durable via the events table". | A | `HUB/monitor/offsite_delivery.go:39` |
| 5 | Two healers could meet: the checker re-reads the secret row immediately before acting, and a box declaration is owned by offsiteheal. | A | `HUB/monitor/offsite_delivery.go:28-31` `HUB/monitor/offsite_delivery.go:97-99` |
| 6 | Age counted from the hub-held `status.Since`. Debounce is counted in reports, not ticks. | A | `HUB/monitor/offsite_delivery.go:93` `HUB/offsiteheal/reconciler.go:61-69` |
| 7 | Operator events `offsite_delivery_stuck`, `offsite_credential_restaged`; `offsite_selfheal_restaged` / `_reissued`. | A | `HUB/monitor/offsite_delivery.go:34-35` `HUB/offsiteheal/reconciler.go:57-58` |
| 8 | SOURCE and DOC only. No test name for the timing was found. | A | `HUB/offsiteheal/reconciler.go:61-74` `AR/06-offsite-connectivity.md:386` `HUB/monitor/offsite_delivery.go:1` |
**Hand step in a runbook?** no hand step
### CH13 — PBS whole-box credential self-heal · OPERATOR · Hub sweep · MENTIONED
architecture folder: 9 lines in 5 files match `/pbsdrheal|PBS.{0,30}heal/` (a machine count, not a reading) · helper section: clock-hub „13. PBS whole-box credential self-heal”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER. `pbsdrheal` reconciler, started at; repairs a box whose off-site PBS credential did not land. | A | `HUB/pbsdrheal/reconciler.go:153-165` `E/hub/cmd/hub/main.go:829-837` |
| 2 | Every 5 minutes, or on a trigger; acts after 2 reports. A staged secret unconsumed for `unconsumedGrace` 15 min is flagged. | A | `HUB/pbsdrheal/reconciler.go:130` `HUB/pbsdrheal/reconciler.go:160` `HUB/pbsdrheal/reconciler.go:131` `HUB/pbsdrheal/reconciler.go:63` |
| 3 | No time limit on a pass, `reconcileOnce(ctx)`. Reads all heal states; linear in boxes. | A | `HUB/pbsdrheal/reconciler.go:163` `HUB/pbsdrheal/reconciler.go:170` |
| 4 | Pod down: no heal; the debounce maps are in memory, so after a restart a box waits two more reports. | A | `HUB/pbsdrheal/reconciler.go:134-135` |
| 5 | One goroutine; a read error does nothing this tick. | A | `HUB/pbsdrheal/reconciler.go:171-174` |
| 6 | Grace is hub time. Box clock: not used here. | A | `HUB/pbsdrheal/reconciler.go:153-165` `E/hub/cmd/hub/main.go:829-837` (the code path of question 1) |
| 7 | Operator audit events `pbsdr_selfheal_*` and `pbsdr_unconsumed_secret`. | A | `HUB/pbsdrheal/reconciler.go:50-54` |
| 8 | SOURCE only. A pass that hangs on the remote side: NOBODY KNOWS; the pass has no time limit. | N | `HUB/pbsdrheal/reconciler.go:57-63` · class W4 · S15 |
**Hand step in a runbook?** no hand step
### CH14 — WireGuard peer list push to ep0 · OPERATOR · Hub sweep · MENTIONED
architecture folder: 8 lines in 3 files match `/wgsync|peer list/` (a machine count, not a reading) · helper section: clock-hub „14. WireGuard peer list push to ep0”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR (keeps every box's tunnel allowed). `wgReconciler.Run`, loop; pushes the FULL peer list over SSH. | A | `E/hub/cmd/hub/main.go:752` `HUB/wgsync/reconciler.go:90-106` |
| 2 | Every 5 minutes and on every change via `Trigger`. No setting. | A | `HUB/wgsync/reconciler.go:41` `HUB/wgsync/reconciler.go:48-53` |
| 3 | 60 s limit per push. The payload grows with peers. | A | `HUB/wgsync/reconciler.go:100` `HUB/wgsync/reconciler.go:71-73` |
| 4 | With the hub down ep0 keeps the last list. A push that fails is logged and tried again at the next tick. Nobody is mailed when it fails again and again; a new box then cannot reach the off-site store. | N | `HUB/wgsync/reconciler.go:100-102` · class W1 · S05 |
| 5 | One loop; a pending trigger covers later ones. Full list each time erases drift. | A | `HUB/wgsync/reconciler.go:46-47` `HUB/wgsync/reconciler.go:20-23` |
| 6 | No clock in the payload. | A | `E/hub/cmd/hub/main.go:752` `HUB/wgsync/reconciler.go:90-106` (the code path of question 1) |
| 7 | Log line only. | A | `HUB/wgsync/reconciler.go:102` |
| 8 | TEST `TestReconciler_FailureRetriedOnTick`; TEST `TestReconciler_TriggerPushesFullList`. | A | `HUB/wgsync/reconciler_test.go:119` `HUB/wgsync/reconciler_test.go:82` |
**Hand step in a runbook?** no hand step
### CH15 — Box clock ran ahead: the tunnel alarm · OPERATOR · Hub sweep · MENTIONED
architecture folder: 13 lines in 4 files match `/wg_tunnel|stamped in the future|clock.ahead/` (a machine count, not a reading) · helper section: clock-hub „15. Box clock ran ahead: tunnel alarm”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR (the household's whole-box copies stop). `checkWGTunnel`, called on every host report; not a timer, it runs when the box reports. | A | `HUB/api/host_tunnel.go:61` `HUB/api/handler.go:1010` |
| 2 | Fires when `handshake_future_s` > 0 or the last handshake is older than `wgTunnelStaleAfterS` 6 h. Constant. | A | `HUB/api/host_tunnel.go:41` `HUB/api/host_tunnel.go:23` |
| 3 | One store read per report. | A | `HUB/api/host_tunnel.go:66` |
| 4 | If the box stops reporting, this check never runs; block 1 covers that. An older agent that sends neither field is not judged. | A | `HUB/api/host_tunnel.go:62-64` |
| 5 | Edge-triggered from the newest stored tunnel event, so a restart does not re-mail. | A | `HUB/api/host_tunnel.go:71-83` |
| 6 | This IS the clock-ahead handling: the future stamp is measured by the box's own clock; repair is a new key after 15 minutes on the agent side. Known gaps are listed in. | A | `HUB/api/host_tunnel.go:31` `HUB/api/host_tunnel.go:14-17` `AR/05-hub-architecture.md:489-490` |
| 7 | Operator only, `wg_tunnel_stale` warning; `wg_tunnel_ok` is stored, not mailed. | A | `HUB/api/host_tunnel.go:12-14` |
| 8 | TEST `TestD30_TheAlarmClearsWhenTheNewKeyShakesHands`; DOC. | A | `HUB/api/host_tunnel_test.go:83` `AR/05-hub-architecture.md:482-490` |
**Hand step in a runbook?** no hand step
### CH16 — The daily scheduler and the hour change · OPERATOR · Hub daily · MENTIONED
architecture folder: 2 lines in 1 files match `/scheduleDaily|nextDailyRun|D24/` (a machine count, not a reading) · helper section: clock-hub „16. The daily scheduler and the clock change”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR (mechanism under blocks 2, 11, 17, 18). `scheduleDaily` with `nextDailyRun`. | A | `E/hub/cmd/hub/main.go:1069-1093` `E/hub/cmd/hub/main.go:1095-1112` |
| 2 | Budapest wall-clock time. If the zone cannot be loaded, the fallback is fixed CET, which is one hour off in summer. The pod's zone is set to Europe/Budapest. | A | `E/hub/cmd/hub/main.go:1072` |
| 3 | Sleeps with `time.After(delay)`; the job runs inline, so a long job delays only its own next slot. | A | `E/hub/cmd/hub/main.go:1088` `E/hub/cmd/hub/main.go:1090` |
| 4 | A slot that is missed while the hub is down is not made up; the snapshot and the prune are the two exceptions. A time setting that cannot be read becomes 00:00 with no message. | N | `E/hub/cmd/hub/main.go:1069-1112` · class W1 · S01 |
| 5 | Next run is the calendar day after and the slot that ran last never runs again. | A | `AR/05-hub-architecture.md:642-645` |
| 6 | Days of 23 and 25 hours are handled (D24, with tests). While the scheduler sleeps, a jump of the hub's clock does not move the timer: NOBODY KNOWS what then runs when. | N | `AR/05-hub-architecture.md:640-646` · class W2 · S10 |
| 7 | Nobody; start-up logs only. | A | `E/hub/cmd/hub/main.go:1069-1093` `E/hub/cmd/hub/main.go:1095-1112` (the code path of question 1) |
| 8 | TEST `TestR1183_DailyJobRunsOnceOnTheDayTheClocksGoBack`; TEST `TestR1183_DailyJobRunsOnceOnTheDayTheClocksGoForward`. | A | `E/hub/cmd/hub/r1183_dst_schedule_test.go:31` `E/hub/cmd/hub/r1183_dst_schedule_test.go:56` |
**Hand step in a runbook?** no hand step
### CH17 — Hub database snapshot (four slots) · OPERATOR · Hub daily · MENTIONED
architecture folder: 236 lines in 18 files match `/snapshot/` (a machine count, not a reading) · helper section: clock-hub „17. Hub database snapshot (four slots)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR (the hub holds every household's escrow and settings). `runSnapshot` scheduled over `dbsnap.Slots`; `Make`. | A | `E/hub/cmd/hub/main.go:967-983` `HUB/dbsnap/dbsnap.go:72-117` |
| 2 | 02:00, 08:00, 14:00, 20:00 Budapest: `Slots`. Keeps `Keep = 2`. File name carries UTC. | A | `HUB/dbsnap/dbsnap.go:32` `HUB/dbsnap/dbsnap.go:27` `HUB/dbsnap/dbsnap.go:89` |
| 3 | 10-minute limit. Time grows with database size; written to a tmp file then renamed; duration is logged. | A | `E/hub/cmd/hub/main.go:972` `HUB/dbsnap/dbsnap.go:91-104` `HUB/dbsnap/dbsnap.go:113-114` |
| 4 | The one job WITH catch-up: at start, if the newest snapshot is older than `CatchUpAge` 6 h, one is made, `NeedsCatchUp`. | A | `E/hub/cmd/hub/main.go:978-980` `HUB/dbsnap/dbsnap.go:148-159` |
| 5 | A running flag refuses a second run with `ErrBusy`. The DooPlex push timer runs 30 min later and waits for the tmp file (block 26). | A | `HUB/dbsnap/dbsnap.go:73-80` |
| 6 | The slots are safe on the hour-change night. The design text still says 'older than 24 h' and '02:30' in its first paragraph; the source has four slots. | N | `AR/05-hub-architecture.md:618-620` · class W5 · S18 |
| 7 | Failure is a log line; the outside alarm is the push timer's (block 26). | A | `E/hub/cmd/hub/main.go:967-983` `HUB/dbsnap/dbsnap.go:72-117` (the code path of question 1) |
| 8 | TEST `TestD34_SlotsAreEverySixHours`; TEST `TestR173_MainSchedulesTheDBSnapshot`. | A | `HUB/dbsnap/dbsnap_test.go:214` `E/hub/cmd/hub/r173_wiring_test.go:14` |
**Hand step in a runbook?** YES — `RB/RUNBOOK-hub-db-offsite-backup.md`
### CH18 — Daily prune of old hub rows (04:30) · OPERATOR · Hub daily · MENTIONED
architecture folder: 54 lines in 12 files match `/prune/` (a machine count, not a reading) · helper section: clock-hub „18. Daily prune of old hub rows (04:30)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR. `pruneAll`, at start and by `scheduleDaily(ctx, "prune",...)`. | A | `E/hub/cmd/hub/main.go:1123-1150` `E/hub/cmd/hub/main.go:860-865` |
| 2 | Daily at `prune_schedule`, default `"04:30"`, deployed; keeps `max_days` 90. Also app telemetry 90 d, stale issues 30 d, deleted-customer audit 1 year. | A | `E/hub/cmd/hub/main.go:1026` `E/manifests/hub.yaml:83` `E/manifests/hub.yaml:82` `E/hub/cmd/hub/main.go:1134` |
| 3 | No time limit and no duration in the log. The deletes grow with the reports of each box each day. | N | `E/hub/cmd/hub/main.go:1123-1150` · class W4 · S15 |
| 4 | Runs at every start, so a missed slot is caught up by the next restart; with `MaxDays` 0 it never runs. | A | `E/hub/cmd/hub/main.go:860-865` |
| 5 | May meet the 05:00 deadline check or a snapshot only if it runs over 30 min; no lock found. | A | `E/hub/cmd/hub/main.go:1123-1150` `E/hub/cmd/hub/main.go:860-865` (the code path of question 1) |
| 6 | One cut-off is made from the pod's local time and compared with a UTC column; the other uses UTC. The error is 1 or 2 hours on 90 days. | N | `HUB/store/store.go:1607` `HUB/store/store.go:129` `HUB/store/store.go:2873` · class W2 · S10 |
| 7 | Log only. | A | `E/hub/cmd/hub/main.go:1123-1150` `E/hub/cmd/hub/main.go:860-865` (the code path of question 1) |
| 8 | SOURCE only. No test for the cut-off's time zone was found. | A | `E/hub/cmd/hub/main.go:1123-1150` `HUB/store/store.go:1606` |
**Hand step in a runbook?** no hand step
### CH19 — The other checks on the 60-second sweep · OPERATOR · Hub sweep · MENTIONED
architecture folder: 59 lines in 16 files match `/60.second|sweep/` (a machine count, not a reading) · helper section: clock-hub „19. The other checks on the 60-second sweep”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER and OPERATOR mixed. Merged: host capability, host leaf, host disk, storage fill, management plane, OOB, controller supervisor. All run in the one sweep goroutine. | A | `E/hub/cmd/hub/main.go:937-946` `E/hub/cmd/hub/main.go:927-965` |
| 2 | Every 60 s in the fixed order of. A backup-cadence refusal repeats every `refusalRepeatEvery` 24 h. | A | `E/hub/cmd/hub/main.go:935-962` `HUB/monitor/host_capability.go:72` |
| 3 | All checkers run one after the other in one loop, with no limit for each. A slow checker delays all the others, the liveness check too. Each reads the whole fleet at each pass. | N | `E/hub/cmd/hub/main.go:935-962` · class W4 · S15 |
| 4 | Pod down: nothing. After start, band states are rebuilt from the latest reports. | A | `E/hub/cmd/hub/main.go:937-946` `E/hub/cmd/hub/main.go:927-965` (the code path of question 1) |
| 5 | One goroutine, so no overlap; a Go ticker drops ticks when a pass overruns. | A | `E/hub/cmd/hub/main.go:937-946` `E/hub/cmd/hub/main.go:927-965` (the code path of question 1) |
| 6 | They judge the latest report's content, not time, except the 24 h repeat. | A | `HUB/monitor/host_capability.go:172` |
| 7 | Dispatcher mails; storage fill has its own 6 h grain. | A | `AR/08-alarm-ladder.md:305` |
| 8 | SOURCE only. How long one whole sweep takes: NOBODY KNOWS. | N | `E/hub/cmd/hub/main.go:927-965` · class W4 · S16 |
**Hand step in a runbook?** no hand step
### CH20 — Reminder about a held app · CUSTOMER · Hub sweep · MENTIONED
architecture folder: 32 lines in 7 files match `/held.app|held app/` (a machine count, not a reading) · helper section: clock-hub „20. Reminder about a held app”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR (D53: no household mail). `heldApps.Check`; `HeldAppsChecker.Check`. | A | `E/hub/cmd/hub/main.go:947` `HUB/monitor/held_apps.go:102` |
| 2 | One pass per `heldAppCheckEvery` 15 min. First mail after a grace of 6 h / 72 h / 24 h by reason; then every `heldAppRepeat` 7 d. | A | `HUB/monitor/held_apps.go:36` `HUB/monitor/held_apps.go:40-48` `HUB/monitor/held_apps.go:34` |
| 3 | One list of all held apps. | A | `HUB/monitor/held_apps.go:114` |
| 4 | Pod down: the reminder comes on the first pass after start; the reminded-at time is stored BEFORE sending, so a restart cannot mail twice. | A | `HUB/monitor/held_apps.go:141-145` |
| 5 | A row the box has not confirmed for `heldAppStale` 50 h is not reminded. | A | `HUB/monitor/held_apps.go:125-129` |
| 6 | The reminder counts from a time that the box reports. A box clock that is behind makes the reminder early; one that is ahead makes it late. | N | `HUB/monitor/held_apps.go:130-134` `HUB/monitor/held_apps.go:22` · class W2 · S07 |
| 7 | Operator, `app_held_reminder` warning. | A | `HUB/monitor/held_apps.go:157-163` |
| 8 | TEST file named in source; DOC. | A | `HUB/monitor/held_apps.go:31` `AR/08-alarm-ladder.md:655` |
**Hand step in a runbook?** no hand step
### CH21 — 'A golden is published but not vouched' · OPERATOR · Hub sweep · MENTIONED
architecture folder: 57 lines in 17 files match `/vouch/` (a machine count, not a reading) · helper section: clock-hub „21. "A golden is published but not vouched"”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR. `goldenVouch` check on the sweep; `GoldenVouchChecker.Check`. | A | `E/hub/cmd/hub/main.go:948-950` `HUB/monitor/golden_vouch.go:54` |
| 2 | One Gitea listing per `goldenVouchCheckEvery` 15 min; alarm after `goldenVouchGrace` 2 h. | A | `HUB/monitor/golden_vouch.go:42-43` |
| 3 | Listing limit `goldenVouchListDeadline` 20 s; it runs inside the sweep, so a slow Gitea holds the sweep for up to 20 s. | A | `HUB/monitor/golden_vouch.go:44` `HUB/monitor/golden_vouch.go:62-64` |
| 4 | Pod restart: the gap clock and the alerted mark are in memory, so the 2 h starts again and the alarm is mailed again ("one alarm per unvouched version per process"). | A | `HUB/monitor/golden_vouch.go:34-37` `HUB/monitor/golden_vouch.go:23` |
| 5 | Gitea unreachable or empty list: no verdict, never an alarm. | A | `HUB/monitor/golden_vouch.go:65-67` |
| 6 | Hub clock only. | A | `E/hub/cmd/hub/main.go:948-950` `HUB/monitor/golden_vouch.go:54` (the code path of question 1) |
| 7 | Operator event `golden_not_vouched`. | A | `HUB/monitor/golden_vouch.go:95-96` |
| 8 | TEST `TestGoldenVouchChecker` named at. | A | `HUB/monitor/golden_vouch.go:24` |
**Hand step in a runbook?** no hand step
### CH22 — Shared storage boxes: fill and reachability · OPERATOR · Hub sweep · MENTIONED
architecture folder: 37 lines in 9 files match `/storage box|pool.box|oversub/` (a machine count, not a reading) · helper section: clock-hub „22. Shared storage boxes: fill and reachability”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR (every household's off-site space). `offsiteBox` and `pbsdrBox` checks. | A | `E/hub/cmd/hub/main.go:951-956` |
| 2 | On the 60 s sweep, but one provider read per `boxFetchInterval` 15 min, test at and. Warn 80 %, critical 90 %. | A | `HUB/monitor/offsite_box.go:72` `HUB/monitor/offsite_box.go:132` `HUB/monitor/pbsdr_box.go:130` `HUB/monitor/offsite_box.go:67-68` |
| 3 | Provider client limit 30 s. Does not grow with boxes. | A | `HUB/hetznerapi/hetznerapi.go:143-145` |
| 4 | Pod down: no reading. Unreachable is declared after consecutive failed 15-minute windows. | A | `HUB/monitor/pbsdr_box.go:80-82` |
| 5 | Throttled by `lastFetch` in memory; first pass after a restart fetches at once. | A | `HUB/monitor/offsite_box.go:132` |
| 6 | Hub clock only. | A | `E/hub/cmd/hub/main.go:951-956` (the code path of question 1) |
| 7 | Operator events for fill, oversubscription, unreachable and recovered. | A | `HUB/monitor/offsite_box.go:266-330` |
| 8 | SOURCE. | A | `HUB/monitor/offsite_box.go:70-72` |
**Hand step in a runbook?** no hand step
### CH23 — DooPlex: push of the hub database snapshot off-site · OPERATOR · DooPlex and ep0 · MENTIONED
architecture folder: 33 lines in 8 files match `/hub.db|database snapshot/` (a machine count, not a reading) · helper section: clock-hub „26. DooPlex: push of the hub database snapshot off-site”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | NOT MEASURED. OPERATOR. systemd timer `felhom-hub-db-backup.timer` on DooPlex, runs the script E/scripts/hub-db-backup/felhom-hub-db-backup:19; installed by hand per. | A | `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10` `RB/RUNBOOK-hub-db-offsite-backup.md:107-108` |
| 2 | NOT MEASURED (DooPlex and ep0 are outside this run). 02:30, 08:30, 14:30 and 20:30 with up to 2 minutes of random delay, in the host's local time (the unit names no zone). | A | `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10` `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:12` |
| 3 | `TimeoutStartSec=45min`. Waits up to `WAIT_S` 1200 s for the hub's tmp file E/scripts/hub-db-backup/felhom-hub-db-backup:20, loop E/scripts/hub-db-backup/felhom-hub-db-backup:35-40. Grows with database size. | A | `E/scripts/hub-db-backup/felhom-hub-db-backup.service:13` |
| 4 | `Persistent=true`: a slot missed while DooPlex was off runs at boot. It refuses a snapshot older than `MAX_AGE_H` 7 E/scripts/hub-db-backup/felhom-hub-db-backup:19, line E/scripts/hub-db-backup/felhom-hub-db-backup:50. | A | `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:11` |
| 5 | No flock in the script (grep `flock/lock` returned nothing); systemd does not start a oneshot unit twice. A catch-up run right after boot can meet a hub that has not made its catch-up snapshot yet: history of a false failure in E/scripts/hub-db-backup/felhom-hub-db-backup:15-18. | A | `E/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10` `RB/RUNBOOK-hub-db-offsite-backup.md:107-108` (the code path of question 1) |
| 6 | NOT MEASURED (DooPlex and ep0 are outside this run). The timer's zone and the hub's zone must match for the 30-minute offset. On the autumn hour-change day the 02:00 copy never reaches ep0 and the 20:00 copy is pushed twice. | K | `E/scripts/hub-db-backup/test_hub_db_backup.py:309-316` · R-1293 |
| 7 | The runbook says in one place that the alarm fires after 13 hours, and the alarm rule printed in the same runbook says 26 hours. | N | `RB/RUNBOOK-hub-db-offsite-backup.md:100` `RB/RUNBOOK-hub-db-offsite-backup.md:152-153` · class W5 · S18 |
| 8 | NOT MEASURED (DooPlex and ep0 are outside this run). A test file for the script exists and is run by hand. The live alarm rule is in another repository and on DooPlex; this run did not read it. | A | `RB/RUNBOOK-hub-db-offsite-backup.md:107-108` `E/scripts/hub-db-backup/test_hub_db_backup.py:309-316` |
**Hand step in a runbook?** YES — `RB/RUNBOOK-hub-db-offsite-backup.md`
### CH24 — DooPlex: Sunday restore tests · OPERATOR · DooPlex and ep0 · MENTIONED
architecture folder: 121 lines in 19 files match `/restore.test/` (a machine count, not a reading) · helper section: clock-hub „27. DooPlex: Sunday restore tests”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | NOT MEASURED. OPERATOR. Two timers, merged: `felhom-hub-db-restore-test.timer` and `felhom-dooplex-offsite-restore-test.timer`. | A | `E/scripts/hub-db-backup/felhom-hub-db-restore-test.timer:6` `E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test.timer:6` |
| 2 | Sunday 04:30 and Sunday 05:30, host local time; `Persistent=true`. | A | `E/scripts/hub-db-backup/felhom-hub-db-restore-test.timer:7` |
| 3 | Limits 45 min and 90 min. Grow with database and dump size. | A | `E/scripts/hub-db-backup/felhom-hub-db-restore-test.service:13` `E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test.service:13` |
| 4 | Missed Sunday runs at next boot (Persistent). Each refuses a copy older than `MAX_AGE_H` 50 E/scripts/hub-db-backup/felhom-hub-db-restore-test:13 and E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test:14. | A | `E/scripts/hub-db-backup/felhom-hub-db-restore-test.timer:6` `E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test.timer:6` (the code path of question 1) |
| 5 | NOT MEASURED (DooPlex and ep0 are outside this run). Sunday 04:30 is between the 02:00 snapshot push and the 05:00 PBS pull. The script takes no lock against them. | N | class W3 · S14 |
| 6 | NOT MEASURED (DooPlex and ep0 are outside this run). On the last Sunday of October the hour 02:00 to 03:00 comes twice. What the timer does at 04:30 that day: NOBODY KNOWS. | N | class W2 · S20 |
| 7 | Staleness alarm `HubDBRestoreTestStale` after 8 days. | A | `RB/RUNBOOK-hub-db-offsite-backup.md:158` |
| 8 | NOT MEASURED (DooPlex and ep0 are outside this run). DOC only. This run did not read if the alarm rule is deployed. | A | `RB/RUNBOOK-hub-db-offsite-backup.md:152-161` |
**Hand step in a runbook?** YES — `RB/RUNBOOK-hub-db-offsite-backup.md`
### CH25 — DooPlex: nightly off-site push and the release copy · OPERATOR · DooPlex and ep0 · MENTIONED
architecture folder: 7 lines in 2 files match `/release.copy|release copy/` (a machine count, not a reading) · helper section: clock-hub „28. DooPlex: nightly off-site push and release copy”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | NOT MEASURED. OPERATOR. Merged: `felhom-dooplex-offsite.timer` and `felhom-release-copy.timer`. | A | `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:6` `E/scripts/release-copy/felhom-release-copy.timer:7` |
| 2 | 00:20 and 01:40 host local time; the order is stated in a comment. Random delay 2 min and 5 min,. | A | `E/scripts/release-copy/felhom-release-copy.timer:6` `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:8` `E/scripts/release-copy/felhom-release-copy.timer:9` |
| 3 | NOT MEASURED (DooPlex and ep0 are outside this run). The units have limits of 90 minutes and 2 hours. A 90-minute run from 00:20 ends at 01:50, after the 01:40 start of the next job. No lock between them. | N | `E/scripts/dooplex-offsite/felhom-dooplex-offsite.service:13` `E/scripts/release-copy/felhom-release-copy.service:12` · class W3 · S14 |
| 4 | Both `Persistent=true`,. The push refuses a dump older than `DUMP_MAX_AGE_H` 7 E/scripts/dooplex-offsite/felhom-dooplex-offsite:23, message E/scripts/dooplex-offsite/felhom-dooplex-offsite:55. | A | `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:7` `E/scripts/release-copy/felhom-release-copy.timer:8` |
| 5 | The push depends on a dump CronJob that is not in this clone (named at E/scripts/dooplex-offsite/felhom-dooplex-offsite:55). One retry E/scripts/dooplex-offsite/felhom-dooplex-offsite:66. | A | `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:6` `E/scripts/release-copy/felhom-release-copy.timer:7` (the code path of question 1) |
| 6 | 00:20 and 01:40 are outside the 02:00-03:00 change hour. | A | `E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:6` `E/scripts/release-copy/felhom-release-copy.timer:7` (the code path of question 1) |
| 7 | `OnFailure` mail on both, a metric file is written E/scripts/dooplex-offsite/felhom-dooplex-offsite:143-153. | A | `E/scripts/dooplex-offsite/felhom-dooplex-offsite.service:6` `E/scripts/release-copy/felhom-release-copy.service:6` |
| 8 | NOT MEASURED (DooPlex and ep0 are outside this run). SOURCE only. The release-copy timer is switched on by hand; this run did not read if it is on. | A | `E/scripts/release-copy/install.sh:4` |
**Hand step in a runbook?** no hand step
### CH26 — DooPlex and ep0: PBS copy, prune, verify, clean-up · OPERATOR · DooPlex and ep0 · MENTIONED
architecture folder: 124 lines in 17 files match `/ep0|PBS.{0,20}(prune|sync|pull)/` (a machine count, not a reading) · helper section: clock-hub „29. DooPlex and ep0: PBS copy, prune, verify, clean-up”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | NOT MEASURED. OPERATOR (holds the second copy of every household's whole-box backup). PBS jobs described in, plus `felhom-ep0-copy-gc.timer`. | A | `RB/ep0-datastore-copy.md:15-18` `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:5` |
| 2 | ep0 prune 03:30, pull to DooPlex 05:00; DooPlex prune 07:30; copy clean-up 08:00; verify Saturday 06:30; GC Sunday 08:30; hub-DB prune on ep0 03:45. | A | `RB/ep0-datastore-copy.md:15` `RB/ep0-datastore-copy.md:17` `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:5` `RB/ep0-datastore-copy.md:16` |
| 3 | NOT MEASURED (DooPlex and ep0 are outside this run). The PBS jobs are in PBS's own config, not in the repository. They grow with the backup size of every box. | A | `RB/ep0-datastore-copy.md:19` |
| 4 | The gc timer is `Persistent=true`. PBS jobs: catch-up behaviour not in repo. | A | `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:6` |
| 5 | NOT MEASURED (DooPlex and ep0 are outside this run). The order is by the clock only: prune 07:30, clean-up 08:00. A pull that runs past 07:30 meets the prune: NOBODY KNOWS what then happens. | N | `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:2` · class W3 · S14 |
| 6 | NOT MEASURED (DooPlex and ep0 are outside this run). Two machines, two clocks. 'The ep0 prune at 03:30 is before the 05:00 pull' is true only while both clocks and both zones agree. No document gives their zones. | N | `RB/ep0-datastore-copy.md:1` · class W3 · S14 |
| 7 | NOT MEASURED (DooPlex and ep0 are outside this run). PBS has its own notification target. The clean-up unit has no failure hook and no time limit; a clean-up that fails tells nobody. | N | `E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.service:9` `RB/ep0-datastore-copy.md:19` · class W1 · S01 |
| 8 | DOC. | A | `RB/ep0-datastore-copy.md:15-19` |
**Hand step in a runbook?** YES — `RB/ep0-datastore-copy.md`
### CH27 — Nightly CI: the gates and the security watch · OPERATOR · DooPlex and ep0 · MENTIONED
architecture folder: 85 lines in 19 files match `/nightly|security.watch|cron/` (a machine count, not a reading) · helper section: clock-hub „30. Nightly CI: gates and the security watch”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | NOT MEASURED. OPERATOR. Gitea CI crons in felhom.eu: `gates` and `security-watch`. The other three repos have no scheduled job (grep `schedule:/cron:` in their `.gitea/workflows/` returned nothing). | A | `E/.gitea/workflows/gates.yml:26-27` `E/.gitea/workflows/security-watch.yml:21-22` |
| 2 | 03:20 UTC and 03:40 UTC every night; the reason for the order is in. CI cron is UTC per, so it moves one hour against the Budapest night jobs at each clock change. | A | `E/.gitea/workflows/gates.yml:21-23` `AR/08-alarm-ladder.md:710` |
| 3 | Limits: 5 min min. The watch grows with the number of pinned programs. | A | `E/.gitea/workflows/gates.yml:53` `E/.gitea/workflows/gates.yml:20` `E/.gitea/workflows/security-watch.yml:33` |
| 4 | NOT MEASURED (DooPlex and ep0 are outside this run). A missed night is not made up. The watch's weekly summary repairs itself: it is sent when the last one is 8 or more days old. A runner that drops the job tells nobody. | N | `AR/08-alarm-ladder.md:717` · class W1 · S01 |
| 5 | Both run on `felhom-gates` runners; the watch commits its memory to main with 3 attempts. A push race with a human commit is retried, then fails. | A | `E/.gitea/workflows/security-watch.yml:32` `E/.gitea/workflows/security-watch.yml:95-106` |
| 6 | UTC cron, no DST. Reminder ages are computed by the script from its memory file. | A | `AR/08-alarm-ladder.md:716-720` |
| 7 | Failure mail to admin: "[felhom CI] security watch FAILED"; gates has its own alarm step. Whether the alarm step runs for a reaped job is recorded as unknown. | A | `E/.gitea/workflows/gates.yml:245` `E/.gitea/workflows/gates.yml:50-52` |
| 8 | NOT MEASURED (DooPlex and ep0 are outside this run). DOC. A night with no run at all: NOBODY KNOWS who notices. | N | `AR/08-alarm-ladder.md:708-722` · class W1 · S01 |
**Hand step in a runbook?** no hand step