Read-only review of every job on a timer and of everything that ends on a date, with measurements from the three test boxes. Nothing ends in the next 14 days. Register 236 -> 245 (R-1379..R-1387), pointer lines on twelve rows, D76-D79 in STATUS.md. Part C: felhom-agent's clone address holds no token now; the other clones wait for the operator's yes. R-1035's 4-week check moved to 2026-11-18 (operator read needed). Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
146 KiB
The Clock list — the table
Built 2026-11-11 from the source of the four repositories and from read-only measurements on the three test boxes
(Tester 1, demo-hp, the N100; host and guest). One block for each job that a clock starts: the night steps on a box,
the interval jobs, the watchdogs, the hub's daily jobs and its 60-second sweep, and the timers on DooPlex and ep0.
tools/make-table.py writes this file from the helpers' answers (evidence/helpers/) and from tools/marks.py; do
not edit it by hand.
NOT MEASURED, by the operator's ruling: DooPlex and ep0, and the hub's database. What this table says about the hub's jobs and about the timers on those two machines comes from the repository's own source, scripts and documents.
The eight questions: 1 what runs, who owns it, what starts it (with CUSTOMER, TEST-ONLY or OPERATOR) ·
2 when, how often, in what order · 3 how long it takes (measured where a box shows it), its time limit, and 10 or 100
times the data · 4 what happens if it does not run · 5 it runs late, or twice, or meets another job (the lock is
named) · 6 the clock is wrong or jumps · 7 who is told, and when · 8 how do we know (MEASURED, a test, SOURCE, a
document, or nobody knows). A ninth line, outside the marks: is the hand step written in a runbook?
Every measured duration comes from a small test store (three boxes, a few apps, one night of controller logs). No number in this table says how long a job takes on a full household disk.
Marks: A answered, with a pointer · D decided, not built · K known gap, with its row · N nobody asked — it
survived the searches in EV/searches.txt (S01…S25; controls in the same file). Every N names its class row:
W1 = R-1383 a job does not run, gives up, or loses its state at a restart, and nobody is told · W2 = R-1384 a
verdict or an "already ran" check trusts a clock that can be wrong · W3 = R-1385 two jobs can meet with no lock and
no order · W4 = R-1386 no time limit and no measured duration · W5 = R-1387 a document and the source disagree
on a time.
Who read what. A cell whose mark is K or N, and every A that tools/marks.py carries, holds the main
session's words. Where its pointer was opened by eye, the pointer is written out in tools/marks.py; where
tools/marks.py shows +, the pointer is the helper's and tools/resolve-helper.py opened it (the file and the line
exist, and the name the helper gave stands at that line). Every other cell holds a helper's words with the mark A.
The main session read every helper answer to questions 4 to 7; it did not open each of their pointers. Treat those
cells as a map, not as proof.
Pointers: AG/ = felhom-agent/ · CT/ = felhom-controller/controller/ · CAT/ = app-catalog-felhom.eu/ ·
HUB/ = felhom.eu/hub/internal/ · E/ = felhom.eu/ · AR/ = felhom.eu/documentation/architecture/ ·
RB/ = felhom.eu/documentation/runbooks/ · EV/ = this folder's evidence/. tools/check-pointers.py prints the
text behind each one.
Named status (does an architecture document name this job?): the status comes from a search of the architecture
folder: 0 lines = UNNAMED, else MENTIONED. It is a machine count, not a reading.
The kinds, and where each job is
| Kind | Jobs (worst mark) | Named · mentioned · named nowhere |
|---|---|---|
| Night | CA01 (K), CA02 (N), CA03 (K), CA05 (N), CA20 (N), CC01 (A), CC02 (N), CC03 (K), CC04 (N), CC05 (N), CC06 (K), CC07 (N), CC08 (K), CC09 (N), CC19 (N), CC22 (K) | 0 · 16 · 0 |
| Interval | CA08 (A), CA09 (N), CA11 (N), CA12 (N), CA13 (N), CA14 (A), CA19 (N), CA21 (A), CC10 (A), CC11 (N), CC12 (N), CC13 (N), CC14 (N), CC15 (N), CC16 (N), CC17 (A), CC18 (N), CC20 (N), CC21 (N) | 0 · 18 · 1 |
| Watchdog | CA04 (N), CA06 (N), CA07 (N), CA10 (N), CA15 (N), CA16 (A), CA17 (A), CA18 (N) | 0 · 8 · 0 |
| Hub daily | CH02 (N), CH11 (N), CH16 (N), CH17 (N), CH18 (N) | 0 · 5 · 0 |
| Hub sweep | CH01 (N), CH03 (N), CH04 (N), CH05 (N), CH06 (N), CH07 (N), CH08 (N), CH09 (N), CH10 (N), CH12 (A), CH13 (N), CH14 (N), CH15 (A), CH19 (N), CH20 (N), CH21 (A), CH22 (A) | 0 · 17 · 0 |
| DooPlex and ep0 | CH23 (N), CH24 (N), CH25 (N), CH26 (N), CH27 (N) | 0 · 5 · 0 |
Not in the table
Tiny (named once, not tabled): session clean-up of dashboard sign-ins on the box (a new sign-in repairs it) · old image clean-up after a controller start (log only; deletes nothing when it cannot read its keep list) · operator restart and reboot limits on the agent (one hour; an operator action) · the hub's purges of log bundles, access log and sessions (nobody is told, nothing depends on them) · the hub's version check and catalog template fetch (a WARN line and a stored error; a page shows it) · the guest swap check, once after an agent start (2-minute limit) · the 14-day hold of alarms after a deliberate power-off (a window, not a job).
Left out for another reason:
- The agent's local-API certificate (10 years) — it is an end date: the End-dates list, DB03.
- Links and tokens that end by themselves — they are end dates: the End-dates list, DH22 and the by-ruling list there.
- The 14-day countdown after a household stops the off-site copy — the End-dates list, DB04; its daily sweep is CC19 here.
- Debian's and Proxmox's own timers on a host (apt-daily, pve-daily-update, logrotate, fstrim.timer, man-db) — they are the OS's jobs, not ours; MEASURED present on the three hosts,
evidence/box/*-host-clock.txt. - TEST-ONLY jobs — none was found: every job in the table runs on a household's box or for the operator.
CA01 — Whole-guest backup: due-check and run · CUSTOMER · Night · MENTIONED
architecture folder: 210 lines in 20 files match /vzdump|whole.guest|due.check/ (a machine count, not a reading) · helper section: clock-agent „1. Whole-guest backup: due-check and run”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The in-guest controller asks handleBackupDue and starts handleBackup; the host has no timer for it; the agent runs the vzdump. CUSTOMER. |
A | AG/internal/localapi/server.go:1134-1217 AG/internal/localapi/server.go:878-1015 |
| 2 | Due when the newest archive's age reaches the cadence, age >= tier.Cadence; default BackupCadence 24 h; the night window is the controller's; the agent compares ages only, no time zone. |
A | AG/internal/localapi/server.go:1204-1213 AG/internal/config/config.go:576-580 AR/07-backup-architecture.md:551 |
| 3 | Task wait defaultPrimaryTierWaitTimeout 30 min and defaultExtraTierWaitTimeout 12 h; outer bound is the tier wait plus 15 min; the copy grows with the guest's size. |
A | AG/internal/config/config.go:464-465 AG/internal/localapi/server.go:971-975 |
| 4 | A tier not backed up stays due at every later ask; the controller has a 48 h safety valve; the hub judges 72 h without a whole-guest backup. | A | AR/07-backup-architecture.md:551-552 AR/07-backup-architecture.md:579-581 |
| 5 | Same tier in flight answers 202 with the same job; another tier answers 409; the host-wide gate is TryAcquire, in memory. |
A | AG/internal/localapi/server.go:913-917 AG/internal/localapi/server.go:919-925 AG/internal/backup/inflight.go:40-59 |
| 6 | The age is wall-clock time. A host clock that was ahead during one backup gives a negative age afterwards; the tier reads 'not due' until real time passes it. | K | AG/internal/localapi/server.go:509 AG/internal/localapi/server.go:1176-1182 · R-1189 |
| 7 | The hub raises expected_backup_missed to the operator; a flood refused by the cap raises an operator event, rationale. |
A | AR/07-backup-architecture.md:579-582 AG/internal/localapi/backup_cap.go:12-35 |
| 8 | TEST and DOC for the due-check. MEASURED 2026-11-11: demo-hp 11 to 13 minutes, Tester 1 about 3 minutes, the N100 under 1 minute; on the N100 the start time moved 5 minutes later each day. A clock jump has no test. | K | EV/box/demo-hp-host-night.txt EV/box/tester-1-host-night.txt EV/box/demo-felhom-host-night.txt · R-1189 |
Hand step in a runbook? no hand step
CA02 — OS-update night leg: guest, host, Docker, Proxmox · CUSTOMER · Night · MENTIONED
architecture folder: 51 lines in 10 files match /os.?update|OS update/ (a machine count, not a reading) · helper section: clock-agent „2. OS-update night leg (guest, host, Docker, Proxmox)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The agent runs it after a successful primary, non-manual whole-guest backup: SetAfterPrimaryBackup, trigger at; Run. CUSTOMER. |
A | AG/cmd/felhom-agent/main.go:1390-1398 AG/internal/localapi/server.go:1008-1011 AG/internal/osupdate/leg.go:538-586 |
| 2 | 90 s after the backup ends; at most once per MinGap, default 20 h; order guest, host, Docker, Proxmox, kernel; no clock slot of its own. |
A | AG/cmd/felhom-agent/main.go:1395 AG/internal/osupdate/leg.go:656-667 AG/internal/osupdate/leg.go:538-586 AR/11-os-updates.md:378-387 |
| 3 | Wrapper limits: host command 600 s and guest command 1800 s AG/configs/felhom-os-apply:279-284; health wait default 5 min; time grows with the number of packages; measured 23 to 70 s. | A | AG/internal/osupdate/leg.go:810-813 AR/07-backup-architecture.md:449-450 |
| 4 | A failed or missed backup means no OS leg that night; no catch-up found; the hub shows how far behind a box is. | A | AR/07-backup-architecture.md:447 AR/11-os-updates.md:406-408 |
| 5 | It runs holding the heavy-op gate taken by the backup; one pass at a time by flock tryLockPass; a second night run inside 20 h is skipped. |
A | AR/07-backup-architecture.md:444-446 AG/internal/osupdate/unsent.go:35-52 AG/internal/osupdate/leg.go:663 |
| 6 | The 20-hour gap compares the clock with a time read from a file. A stamp that was written while the clock was ahead keeps the leg skipped until real time passes that stamp plus 20 hours. | N | AG/internal/osupdate/leg.go:661-663 · class W2 · S08 |
| 7 | The result is posted to the hub by finish; operator gets a hub event per update and failure, the household one timeline line. |
A | AG/internal/osupdate/leg.go:922-939 AR/11-os-updates.md:406-423 |
| 8 | TEST and DOC for the normal path. A stamp from the future: NOBODY KNOWS. | N | AG/internal/osupdate/leg_test.go:367 AG/internal/localapi/afterbackup_test.go:17 AR/11-os-updates.md:366-387 · class W2 · S08 |
Hand step in a runbook? no hand step
CA03 — Kernel lane: night reboot, 20-minute boot judge, one self-revert · CUSTOMER · Night · MENTIONED
architecture folder: 165 lines in 14 files match /kernel/ (a machine count, not a reading) · helper section: clock-agent „3. Kernel lane: night reboot, 20-minute boot judge, one self-revert”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent: runKernel ends the night leg; at every agent start KernelAfterBoot judges the boot. CUSTOMER (the box restarts at night). |
A | AG/internal/osupdate/kernel.go:119-178 AG/cmd/felhom-agent/main.go:1018 |
| 2 | Only on trigger night and only when the hub's block says Tonight; the household is mailed the day before, 09:00 to 20:00 Budapest time, by the hub. |
A | AG/internal/osupdate/kernel.go:101-115 AR/11-os-updates.md:580-585 |
| 3 | Judge wait DefaultKernelJudgeWait 20 min, polled every 30 s; measured healthy at 68 s and 272 s. |
A | AG/internal/osupdate/kernel.go:37 AG/internal/osupdate/kernel.go:251 AR/11-os-updates.md:570-572 |
| 4 | No mail accepted means no step; a step is never retried by itself; a box that never comes back raises host_stale at 45 min and host_down at 90 min. |
A | AR/11-os-updates.md:584-585 AR/11-os-updates.md:581-583 AR/11-os-updates.md:572-573 |
| 5 | At most one per night under the same heavy-op gate; never two steps within 20 h (wrapper refusal R22); one self-revert, never retried. | A | AR/11-os-updates.md:560-562 AR/11-os-updates.md:557-558 AG/internal/osupdate/kernel.go:300-312 |
| 6 | The 20-minute judge runs on monotonic time; a clock step does not move it. Nothing in the installer sets up time keeping, and a host can start with a wrong clock: the N100's host clock is 32 s behind today. | K | AG/internal/osupdate/leg.go:291-296 E/scripts/felhom-host-install.sh:1 · R-1301 |
| 7 | The hub hears staged before the reboot; operator events os_kernel_step and os_kernel_notice; a frozen new kernel needs a person, accepted. |
A | AG/internal/osupdate/kernel.go:165 AR/11-os-updates.md:585-586 AR/11-os-updates.md:534-535 |
| 8 | TEST and DOC. The design text itself says that a real freeze was not measured. | A | AR/11-os-updates.md:599-601 AG/internal/osupdate/kernel_test.go:79 |
Hand step in a runbook? YES — RB/crash-guard.md
CA04 — Crash guard: early-boot unit and the hourly re-arm timer · CUSTOMER · Watchdog · MENTIONED
architecture folder: 23 lines in 7 files match /crash.guard|crash guard/ (a machine count, not a reading) · helper section: clock-agent „4. Crash guard: early-boot unit plus hourly re-arm timer”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Root systemd units installed by the host installer: boot unit and timer. CUSTOMER (decides if a crashed box restarts). | A | E/scripts/felhom-host-install.sh:2944-2963 AG/configs/felhom-crash-guard.service:15-16 AG/configs/felhom-crash-guard-check.timer:5-6 |
| 2 | Timer as written: OnBootSec=15min, OnUnitActiveSec=1h; no OnCalendar, no RandomizedDelaySec, no Persistent; boot unit runs before sysinit. |
A | AG/configs/felhom-crash-guard-check.timer:5-6 AG/configs/felhom-crash-guard.service:8 |
| 3 | Oneshot with no time limit written; history is cut to 7 days AG/configs/felhom-crash-guard:148, so nothing grows. | A | AG/configs/felhom-crash-guard-check.service:6-7 |
| 4 | A missed hourly check only delays the re-arm; a crash before the boot unit runs leaves the box off, the safe side; the System page shows the guard. | A | AR/11-os-updates.md:484-485 AR/11-os-updates.md:487-488 |
| 5 | The second unclean boot inside 60 minutes trips it; it arms again after 24 hours. The boot unit and the hourly check share one state file and take no lock. | N | AG/configs/crash-guard.conf:3-8 · class W3 · S14 |
| 6 | The script reads the wall clock at early boot, before any time service runs. A clock that is wrong at boot can move unclean boots into or out of the 60-minute window. | N | class W2 · S22 |
| 7 | Each unclean boot is an operator mail host_crash_restart and a household line; the trip is alarm host_crash_guard_tripped; the trip mail came 15 min after the boot. |
A | AR/11-os-updates.md:487-488 AR/11-os-updates.md:489-490 |
| 8 | MEASURED 2026-11-11: the hourly timer is active on the three hosts · TEST and DOC for the windows. A wrong clock at boot: NOBODY KNOWS. | N | EV/box/demo-hp-host-clock.txt EV/box/tester-1-host-clock.txt EV/box/demo-felhom-host-clock.txt AG/configs/test_felhom_crash_guard.py:75 AG/configs/test_felhom_crash_guard.py:102 AR/11-os-updates.md:473-490 · class W2 · S22 |
Hand step in a runbook? YES — RB/crash-guard.md
CA05 — Scheduled restore-test evaluation · CUSTOMER · Night · MENTIONED
architecture folder: 121 lines in 19 files match /restore.test/ (a machine count, not a reading) · helper section: clock-agent „5. Scheduled restore-test evaluation”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine Run, wired at; restores an archive into a scratch guest on the host. CUSTOMER (proof the backup opens). |
A | AG/internal/backup/schedule.go:148-177 AG/cmd/felhom-agent/main.go:1968-2018 |
| 2 | First look DefaultFirstEval 30 min after agent start, then every defaultRestoreTestEvalInterval 6 h; an archive counts after defaultRestoreTestSettle 24 h; not windowed, any hour. |
A | AG/internal/backup/schedule.go:76 AG/internal/config/config.go:636 AG/internal/config/config.go:641 AR/11-os-updates.md:385-386 |
| 3 | Restore task wait: RestoreTestPBSRestoreTimeout 120 min for the PBS tier, else the 10 min default; boot wait DefaultBootTimeout 2 min; grows with guest size. |
A | AG/internal/config/config.go:584-588 AG/cmd/felhom-agent/main.go:1917-1919 AG/internal/reconcile/restoretest.go:28 |
| 4 | The ticker restarts with the agent, so a missed look comes 30 min after the next start; the rule is per archive, so a tier stays due until proven; hub proof-age window 7 to 12 days. | A | AR/07-backup-architecture.md:583-584 AR/07-backup-architecture.md:179-180 AR/07-backup-architecture.md:215-217 |
| 5 | Takes the heavy-op gate with TryAcquire("restore-test"); busy means deferred to the next 6 h look; proof credit only on a pass. |
A | AG/internal/backup/schedule.go:228 AG/internal/backup/schedule.go:230 AG/internal/backup/schedule.go:248-255 |
| 6 | The 'settled' cut-off uses the wall clock. A clock that is set back makes no archive look settled, so no test starts. The ticker itself is monotonic. | N | AG/internal/backup/schedule.go:119 AG/internal/backup/restoretest_due.go:96-101 · class W2 · S08 |
| 7 | A failed test is an ERROR log and a record the hub turns into restore_test_failed for the operator, as the janitor comment says. |
A | AG/internal/backup/schedule.go:262 AG/cmd/felhom-agent/janitor.go:42-44 |
| 8 | TEST and DOC. MEASURED 2026-11-11: restore tests took 1 to 14 minutes on the three boxes. A clock set back: NOBODY KNOWS. | N | EV/box/demo-hp-host-night.txt · class W2 · S08 |
Hand step in a runbook? no hand step
CA06 — Controller supervisor: restarts a dead or silent controller · CUSTOMER · Watchdog · MENTIONED
architecture folder: 12 lines in 5 files match /supervisor/ (a machine count, not a reading) · helper section: clock-agent „6. Controller supervisor (restarts a dead or silent controller)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine WatchControllers, started at. CUSTOMER (the household's dashboard comes back). |
A | AG/internal/localapi/controllersupervisor.go:143-163 AG/cmd/felhom-agent/main.go:1714 |
| 2 | Every controllerSupervisorInterval 30 s; restart on the 2nd not-running sweep; a running but unhealthy one is alarmed at 6 sweeps and restarted at 10. |
A | AG/internal/localapi/controllersupervisor.go:54 AG/internal/localapi/controllersupervisor.go:56 AG/internal/localapi/controllersupervisor.go:92-100 |
| 3 | No time limit on one sweep in the source, and no duration was measured. Nothing grows with data. | N | AG/internal/localapi/controllersupervisor.go:1-200 AG/internal/localapi/controllersupervisor.go:64 · class W4 · S15 |
| 4 | If the agent is down nothing restarts the controller: measured 33 min of 502 before this existed; the first sweep is 30 s after agent start (ticker only). | A | AG/internal/localapi/controllersupervisor.go:20-25 AG/internal/localapi/controllersupervisor.go:153-161 |
| 5 | Brake: controllerCrashloopMax 3 in controllerCrashloopWindow 15 min, then controllerCrashloopPause 30 min; not during a swap, a parked or locked guest, or a vzdump. |
A | AG/internal/localapi/controllersupervisor.go:59-61 AG/internal/localapi/controllersupervisor.go:34-42 |
| 6 | The windows (3 restarts in 15 minutes, the 24-hour counter) use the wall clock in UTC. Nobody tested a clock jump. | N | AG/internal/localapi/controllersupervisor.go:550-552 AG/internal/localapi/controllersupervisor.go:308 · class W2 · S08 |
| 7 | Hub mints controller_restarted_by_agent (info) and controller_crashloop (error, operator only) from the host report; a warning at the 5th restart in 24 h. |
A | AG/internal/localapi/controllersupervisor.go:44-49 AG/internal/localapi/controllersupervisor.go:82-83 |
| 8 | TEST TestControllerSupervisor_KilledControllerIsRestarted, TestSupervisor_UnresponsiveIsAlarmedThenRestarted; DOC. |
A | AG/internal/localapi/controllersupervisor_test.go:101 AG/internal/localapi/supervisor_unresponsive_test.go:32 AR/03-host-agent.md:186-192 |
Hand step in a runbook? no hand step
CA07 — Guest-power watchdog: starts a guest that must run · CUSTOMER · Watchdog · MENTIONED
architecture folder: 8 lines in 4 files match /guest.power|guest power/ (a machine count, not a reading) · helper section: clock-agent „7. Guest-power watchdog (starts a guest that should be running)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine WatchGuestPower, started at. CUSTOMER (the whole box's apps are off while the guest is stopped). |
A | AG/internal/localapi/guestpower.go:89-105 AG/cmd/felhom-agent/main.go:1710 |
| 2 | Every guestPowerInterval 60 s; acts only on a stopped, unlocked guest with onboot set; first sweep one interval after start (ticker only). |
A | AG/internal/localapi/guestpower.go:52 AG/internal/localapi/guestpower.go:167-180 AG/internal/localapi/guestpower.go:95-102 |
| 3 | At most guestPowerMaxAttempts 3 starts with waits of 1, 2 and 4 min, guestPowerBackoff; about 7 min in all. |
A | AG/internal/localapi/guestpower.go:56 AG/internal/localapi/guestpower.go:75 AG/internal/localapi/guestpower.go:71-74 |
| 4 | Before it existed a guest stayed down 9 min 47 s until a person started it; with the agent down nobody starts it (the platform's onboot acts only at host boot). | A | AG/internal/localapi/guestpower.go:14-18 AG/internal/localapi/guestpower.go:42-43 |
| 5 | Never starts under a running vzdump and fails safe on doubt; leaves a locked guest to the stale-lock path; skips a guest the operator is rebooting. | A | AG/internal/localapi/guestpower.go:182-194 AG/internal/localapi/guestpower.go:176-180 AG/internal/localapi/guestpower.go:157-159 |
| 6 | The next try time is kept in memory from the wall clock. A clock set back delays the next try; an agent restart tries again at once. | N | AG/internal/localapi/guestpower.go:234 AG/internal/localapi/guestpower.go:253 · class W2 · S08 |
| 7 | When the watchdog gives up, it writes one ERROR line in the host's journal. No event goes to the hub. The household's box is down and the outside sign is only the hub's liveness check. | N | AG/internal/localapi/guestpower.go:200-206 · class W1 · S04 |
| 8 | TEST for the start and the back-off. Who hears the give-up: NOBODY KNOWS. | N | AG/internal/localapi/guestpower_test.go:162 AG/internal/localapi/guestpower_test.go:119 · class W1 · S04 |
Hand step in a runbook? NO — no runbook says what to do after the watchdog gave up
CA08 — Hub report and heartbeat loop · CUSTOMER · Interval · MENTIONED
architecture folder: 23 lines in 10 files match /heartbeat|poll interval|report loop/ (a machine count, not a reading) · helper section: clock-agent „8. Hub report and heartbeat loop (with reconcile, fast tick and poke)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine Run, built at; it carries every status the hub alarms on. OPERATOR and CUSTOMER (alarms depend on it). Merged here: the reconcile engine, fast tick and poke. |
A | AG/internal/hub/loop.go:107-138 AG/cmd/felhom-agent/main.go:958 |
| 2 | PollSeconds default 900; first report at once, then a ticker; the hub may change it, clamped by MinPollSeconds 60 and MaxPollSeconds 3600; fast tick DefaultInterval 30 s. |
A | AG/internal/config/config.go:797 AG/internal/hub/loop.go:107-138 AG/internal/hub/loop.go:11-12 AG/internal/fasttick/fasttick.go:24 |
| 3 | Per-request limit TimeoutSeconds default 30; the system facts in it are read at most every 10 min; the report grows with guests and drives. |
A | AG/internal/config/config.go:798 AG/cmd/felhom-agent/main.go:4149 |
| 4 | A failed cycle is logged and the interval kept; no backlog, the next report replaces it; the hub raises host_stale at 45 min and host_down at 90 min. |
A | AG/internal/hub/loop.go:162-164 AR/11-os-updates.md:572-573 |
| 5 | Extra reports come by the trigger channel; a poke is debounced by DebounceWindow 2 s; the reconcile engine runs at once and then on the start-time interval,. |
A | AG/internal/hub/loop.go:125-135 AG/internal/poke/poke.go:40 AG/internal/reconcile/engine.go:332-344 AG/cmd/felhom-agent/main.go:1525 |
| 6 | The ticker is monotonic. The report loop follows the hub's interval override, clamped to a range. The reconcile engine keeps the interval that it read at the start. | A | AG/internal/hub/loop.go:188-196 AG/cmd/felhom-agent/main.go:969 |
| 7 | The operator, by the hub's staleness alarms; the household hears cannot-be-reached at most once per 7 days. | A | AR/11-os-updates.md:572-573 AR/07-backup-architecture.md:582-583 |
| 8 | TEST TestLoop_RunImmediateAndResilientAfterError, TestClampInterval; DOC. |
A | AG/internal/hub/loop_test.go:140 AG/internal/hub/loop_test.go:81 AR/03-host-agent.md:276-279 |
Hand step in a runbook? no hand step
CA09 — Signed jobs: the validity window against the host clock · OPERATOR · Interval · MENTIONED
architecture folder: 155 lines in 27 files match /signed job|signed op|not.before|valid/ (a machine count, not a reading) · helper section: clock-agent „9. Signed jobs: validity window against the host clock (agent updates ride it)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent OnEnvelope runs pending signed jobs when a heartbeat says there are some; an agent update arrives only this way. OPERATOR (and CUSTOMER: fixes reach the box this way). |
A | AG/internal/signedjobs/runner.go:93-102 |
| 2 | Checked at every hub cycle (item 8); a job is valid from issue minus DefaultClockSkew 2 min to its expiry, and a window over MaxWindow 24 h is refused. |
A | AG/internal/authz/verifier.go:21 AG/internal/authz/verifier.go:26 |
| 3 | The self-update download has a 5 min client limit; the new binary must run DwellSeconds 60 s before it commits. |
A | AG/internal/selfupdate/executor.go:73 AG/internal/config/config.go:97-99 |
| 4 | A job that the agent refuses as expired is not run. The agent's log line is the only trace on the box; no path that tells the hub or the operator 'refused: expired' was found. | N | AG/internal/authz/verifier.go:161-162 AG/internal/selfupdate/commit.go:122-125 · class W1 · S09 |
| 5 | One run at a time, RunOnce single-flight; a replay is refused by the nonce store. |
A | AG/internal/signedjobs/runner.go:107-115 AG/internal/authz/verifier.go:166 |
| 6 | The window is checked against the host's wall clock. A host clock that is behind refuses every job as 'not yet valid'; one that is ahead refuses them as 'expired'. So a box with a wrong clock cannot take an agent update. | N | AG/internal/authz/verifier.go:153-162 · class W2 · S09 |
| 7 | The runner logs a finished or failed job on the host. For a job refused on its dates, no message to the hub was found. | N | AG/internal/signedjobs/runner.go:176-190 · class W1 · S09 |
| 8 | TEST TestVerify_RejectsPerStage, TestRunner_ExpiredRejected, TestManager_ShutdownBeforeDwellLeavesPending. |
A | AG/internal/authz/verify_test.go:82 AG/internal/signedjobs/runner_test.go:168 AG/internal/selfupdate/commit_test.go:102 |
Hand step in a runbook? NO — no runbook names a wrong host clock as a cause of refused signed jobs
CA10 — Storage watchdog and the 20-second mount re-assert · CUSTOMER · Watchdog · MENTIONED
architecture folder: 2 lines in 2 files match /storage watchdog|re.?assert|ReassertGuestBinds/ (a machine count, not a reading) · helper section: clock-agent „10. Storage watchdog and the 20-second mount and bind re-assert”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent: the watchdog Run wired at, and a 20 s ticker that re-asserts mounts and guest binds. CUSTOMER (a dropped USB drive). |
A | AG/internal/storage/watchdog.go:196-215 AG/cmd/felhom-agent/main.go:1142 AG/cmd/felhom-agent/main.go:1613-1631 |
| 2 | DefaultWatchdogInterval 8 s and DefaultWatchdogDebounce 30 s; setting WatchdogIntervalSeconds, 0 means the default; the re-assert is 20 s, below the controller's 30 s tick. |
A | AG/internal/storage/watchdog.go:15-16 AG/internal/config/config.go:770-778 AG/cmd/felhom-agent/main.go:1612-1614 |
| 3 | No time limit on one tick in the source, and no duration was measured. The work grows with the number of drives. | N | class W4 · S15 |
| 4 | A missed tick is made up by the next one; with the agent down a dropped drive is seen only at the next 15 min report, the reason the watchdog exists. | A | AR/03-host-agent.md:461 |
| 5 | A flapping drive backs off: backoffWindow, alert after 4 failed cycles; the retry test is now.Sub(last). |
A | AG/internal/storage/watchdog.go:183-192 AG/internal/storage/watchdog.go:77-79 AG/internal/storage/watchdog.go:290 |
| 6 | The back-off uses the wall clock in UTC. A clock set back stretches the back-off; a clock set forward ends it early. The tickers are monotonic. | N | AG/internal/storage/watchdog.go:160 · class W2 · S08 |
| 7 | A drop requests an immediate hub report through the trigger channel; the hub then decides who is mailed (hub side, not read). | A | AG/internal/hub/loop.go:125-135 |
| 8 | TEST for the reconcile and the back-off. The 20-second ticker has no test: NOBODY KNOWS. | N | AG/internal/storage/watchdog_test.go:112 AG/internal/storage/watchdog_test.go:460 AG/cmd/felhom-agent/main.go:1613 · class W4 · S15 |
Hand step in a runbook? no hand step
CA11 — Weekly guest disk trim · CUSTOMER · Interval · MENTIONED
architecture folder: 9 lines in 5 files match /fstrim|trim/ (a machine count, not a reading) · helper section: clock-agent „11. Weekly guest disk trim”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine Run, started at; runs pct fstrim per owned running guest. CUSTOMER (a full thin pool turns every guest read-only). |
A | AG/internal/fstrim/fstrim.go:186-199 AG/cmd/felhom-agent/main.go:1741 AG/internal/fstrim/fstrim.go:3-5 |
| 2 | Due from Wednesday 10:00 host-local, starts only 10:00 to 20:59; looks 5 min after start then hourly, FirstTickDelay and TickInterval; zone is time.Local. |
A | AG/internal/fstrim/fstrim.go:42-46 AG/internal/fstrim/fstrim.go:48-50 AG/internal/fstrim/fstrim.go:110 |
| 3 | PerGuestTimeout 30 min; measured 24.4 s for 84 GiB; grows with freed blocks. |
A | AG/internal/fstrim/fstrim.go:52 AG/internal/fstrim/fstrim.go:6 |
| 4 | A box off on Wednesday catches up at its next eligible hour; a failure is retried at most MaxAttemptsPerWeek 3 times; the result rides the host report. |
A | AG/internal/fstrim/fstrim.go:10-11 AG/internal/fstrim/fstrim.go:46 AG/internal/fstrim/fstrim.go:326-330 |
| 5 | Holds the heavy-op gate for the whole pass; busy means deferred to the next hour; the persisted record stops a second trim after a restart. | A | AG/internal/fstrim/fstrim.go:247-251 AG/internal/fstrim/fstrim.go:17 |
| 6 | The 'done this week' check uses host-local wall time. An attempt that was stamped while the clock was ahead keeps the trim 'not due' until real time passes it. | N | AG/internal/fstrim/fstrim.go:179-182 · class W2 · S08 |
| 7 | The operator, on the hub's System page (one cell). A failed trim is a WARN line on the host. Nobody is mailed. | A | HUB/web/system.go:45 HUB/web/system.go:206 AG/internal/fstrim/fstrim.go:318-319 |
| 8 | TEST for the week anchor. The hour change and a clock jump: NOBODY KNOWS. | N | AG/internal/fstrim/fstrim_test.go:161 AG/internal/fstrim/fstrim_test.go:175 AG/internal/fstrim/fstrim_test.go:104 · class W2 · S20 |
Hand step in a runbook? no hand step
CA12 — Weekly upload line test · CUSTOMER · Interval · MENTIONED
architecture folder: 28 lines in 10 files match /line.?test|linetest/ (a machine count, not a reading) · helper section: clock-agent „12. Weekly upload line test”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine Run, started at; a 10 s iperf3 upload over the tunnel. CUSTOMER (sets the torrent app's upload limit). |
A | AG/internal/linetest/linetest.go:256-268 AG/cmd/felhom-agent/main.go:1762 AG/internal/linetest/linetest.go:3-4 |
| 2 | Due from Tuesday 10:00 host-local, starts 10:00 to 15:59; each host from its own minute, SlotMinute; ticks every minute after a 5 min delay; zone time.Local. |
A | AG/internal/linetest/linetest.go:47-49 AG/internal/linetest/linetest.go:148-152 AG/internal/linetest/linetest.go:60-61 AG/internal/linetest/linetest.go:131 |
| 3 | Three runs of 10 s, each bounded by RunTimeout 25 s, 30 s apart; nothing grows with data. |
A | AG/internal/linetest/linetest.go:53-57 |
| 4 | A box that was off catches up on a later day inside the window; the hub page turns amber when the last success is over 15 days old. | A | AG/internal/linetest/linetest.go:10-11 AR/11-os-updates.md:409-411 |
| 5 | Needs 10 quiet minutes and the heavy-op gate; server busy is retried at the next tick, at most MaxTriesPerDay 6; no tunnel means no test. |
A | AG/internal/linetest/linetest.go:316-325 AG/internal/linetest/linetest.go:58 AG/internal/linetest/linetest.go:297-300 |
| 6 | The check uses host-local wall time. A success that was stamped from the future blocks the tests until that time. | N | AG/internal/linetest/linetest.go:211-213 · class W2 · S08 |
| 7 | The operator, on the hub's System page; the cell is judged on the last success, not on the attempt. Nobody is mailed. | A | AR/11-os-updates.md:409-411 |
| 8 | TEST TestDueWeeklyFromTuesday, TestNeverTestsInTheNightOrBeforeTuesday, TestSlotDelaysTheFirstStartOnTheDueDayOnly. |
A | AG/internal/linetest/linetest_test.go:206 AG/internal/linetest/linetest_test.go:363 AG/internal/linetest/linetest_test.go:189 |
Hand step in a runbook? no hand step
CA13 — PBS datastore verify loop · CUSTOMER · Interval · MENTIONED
architecture folder: 64 lines in 17 files match /verify loop|PBS verify|verify/ (a machine count, not a reading) · helper section: clock-agent „13. PBS datastore verify loop”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine Run, cadence wired at; asks the PBS server to verify the box's snapshots. CUSTOMER (is the off-site copy readable). |
A | AG/internal/pbs/verify.go:64-83 AG/cmd/felhom-agent/main.go:1188 |
| 2 | DefaultVerifyCadence 6 h; setting through PBSVerifyCadence; first pass at once at agent start, then the ticker; no time zone. |
A | AG/internal/pbs/verify.go:13 AG/internal/config/config.go:710-719 AG/internal/pbs/verify.go:64-83 |
| 3 | One verify is polled every 2 s for at most 30 min; grows with the datastore's size. | A | AG/internal/pbs/verify.go:107 |
| 4 | No catch-up state: every agent start verifies again at once; a negative cadence switches it off, test TestVerifyLoop_DisabledByNegativeCadence. |
A | AG/internal/pbs/verify.go:64-83 AG/internal/pbs/verify_test.go:87 |
| 5 | The verify is not behind the heavy-work gate; a comment in the source says so. It can run beside a whole-guest backup. | N | AG/internal/lanresolver/loop.go:17 AG/internal/pbs/verify.go:1 · class W3 · S13 |
| 6 | Ticker only, monotonic; no wall-clock decision found in. | A | AG/internal/pbs/verify.go:64-83 |
| 7 | A snapshot that fails the verify is an ERROR line on the host and a field in the report. No hub event or mail for it was found. | N | AG/internal/pbs/verify.go:127 · class W1 · S05 |
| 8 | TEST for the record of a failed verify. A verify that takes longer than its 30-minute cadence: NOBODY KNOWS. | N | AG/internal/pbs/verify_test.go:52 · class W4 · S15 |
Hand step in a runbook? no hand step
CA14 — Janitor: leftover scratch guests and stale backup locks · CUSTOMER · Interval · MENTIONED
architecture folder: 2 lines in 2 files match /janitor|stale lock/ (a machine count, not a reading) · helper section: clock-agent „14. Janitor: leftover scratch guests and stale backup locks”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine runJanitor, started at. CUSTOMER (a stale lock blocked whole-box backups for five hours). |
A | AG/cmd/felhom-agent/janitor.go:69-81 AG/cmd/felhom-agent/main.go:1725 AG/cmd/felhom-agent/janitor.go:14-17 |
| 2 | Every janitorInterval 10 min; ticker only, so the first pass is 10 min after agent start; no time zone. |
A | AG/cmd/felhom-agent/janitor.go:18 AG/cmd/felhom-agent/janitor.go:71-78 |
| 3 | No time limit on a pass found in; work grows with the number of leftover scratch guests. | A | AG/cmd/felhom-agent/janitor.go:34-66 |
| 4 | A missed pass is retried 10 min later; a scratch guest that cannot be removed after the retry limit is reported as a failed restore-test record. | A | AG/cmd/felhom-agent/janitor.go:41-54 |
| 5 | The stale-lock sweep runs only holding the heavy-op gate, else it is skipped this pass; rule stated at. | A | AG/cmd/felhom-agent/janitor.go:57-64 AG/cmd/felhom-agent/janitor.go:30-33 |
| 6 | Ticker only, monotonic; a clock step must not kill a live task: TestVzdumpInFlight_ClockStepCannotKillALiveTask. |
A | AG/internal/localapi/stalelock_inflight_test.go:97 |
| 7 | The operator, through the hub's restore_test_failed, once per distinct archive. |
A | AG/cmd/felhom-agent/janitor.go:42-44 |
| 8 | TEST TestJanitor_StaleLockSweepWaitsForTheHeavyGate, TestJanitor_GaveUpIsReportedAsAFailure. |
A | AG/cmd/felhom-agent/janitor_test.go:40 AG/cmd/felhom-agent/janitor_test.go:22 |
Hand step in a runbook? no hand step
CA15 — Guest network watchdog: DHCP client heal · CUSTOMER · Watchdog · MENTIONED
architecture folder: 17 lines in 5 files match /guest.?net|DHCP/ (a machine count, not a reading) · helper section: clock-agent „15. Guest network watchdog (DHCP client heal)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine Watch, built at. CUSTOMER (a guest without network). |
A | AG/internal/guestnet/watchdog.go:105-121 AG/cmd/felhom-agent/main.go:1686 |
| 2 | DefaultInterval 60 s; immediate first sweep; settings interval 60, MinHealIntervalSeconds 600, MaxHealsPerHour 3. |
A | AG/internal/guestnet/watchdog.go:23 AG/internal/guestnet/watchdog.go:111 AG/internal/config/config.go:176-196 |
| 3 | No time limit on one probe in the source, and no duration was measured. | N | AG/internal/guestnet/watchdog.go:1-200 · class W4 · S15 |
| 4 | A missed sweep is made up a minute later; beyond 3 heals an hour it only reports. | A | AG/internal/guestnet/watchdog.go:26-28 |
| 5 | Two consecutive bad probes before a heal, requiredBadProbes; 10 min between heals, DefaultMinHealInterval; boot-race guard DefaultSettle 3 min. |
A | AG/internal/guestnet/watchdog.go:32-33 AG/internal/guestnet/watchdog.go:25 AG/internal/guestnet/watchdog.go:29-31 |
| 6 | now is plain time.Now, so in-process gaps are monotonic; heal history is in memory. |
A | AG/internal/guestnet/watchdog.go:80 AG/internal/guestnet/watchdog.go:56-62 |
| 7 | The operator, on the hub's Hosts page: the report carries the heal count for each guest. Nobody is mailed. | A | HUB/web/hosts.go:298-397 AG/internal/guestnet/watchdog.go:166-170 |
| 8 | TEST TestWatchdog_DampingCeilings, TestWatchdog_MinimumIntervalBetweenHeals, TestWatchdog_BootRacesObserveOnly. |
A | AG/internal/guestnet/watchdog_test.go:360 AG/internal/guestnet/watchdog_test.go:384 AG/internal/guestnet/watchdog_test.go:416 |
Hand step in a runbook? no hand step
CA16 — Host networking self-heal · CUSTOMER · Watchdog · MENTIONED
architecture folder: 39 lines in 12 files match /self.?heal/ (a machine count, not a reading) · helper section: clock-agent „16. Host networking self-heal”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine Watch, interval set at; starts networking.service on an appliance only. CUSTOMER. |
A | AG/internal/selfheal/selfheal.go:67-82 AG/cmd/felhom-agent/main.go:1648 |
| 2 | One pass at agent start, then every 60 s, no setting found for the interval (hard-coded). | A | AG/internal/selfheal/selfheal.go:67-82 AG/cmd/felhom-agent/main.go:1648 |
| 3 | A remedy is 3 start attempts with waits of 10, 30 and 60 s, networkingBackoffs. |
A | AG/internal/selfheal/selfheal.go:114-116 |
| 4 | A missed pass is made up a minute later; on a BYO host the remedy is skipped and only a WARN is written. | A | AG/internal/selfheal/selfheal.go:54-58 |
| 5 | No lock seen; the pass is sequential inside one goroutine. | A | AG/internal/selfheal/selfheal.go:46-64 |
| 6 | Ticker only, monotonic; no wall-clock decision in. | A | AG/internal/selfheal/selfheal.go:67-82 |
| 7 | A give-up is an ERROR journal line; with networking down the hub cannot be told, so the hub's stale alarm is the only signal. | A | AG/internal/selfheal/selfheal.go:61 AR/11-os-updates.md:572-573 |
| 8 | TEST TestNetworkingHeal_GivesUpAfterAllAttempts, TestManager_ByoNeverRemediates. |
A | AG/internal/selfheal/selfheal_test.go:109 AG/internal/selfheal/selfheal_test.go:32 |
Hand step in a runbook? no hand step
CA17 — Management-plane watchdog timer: keeps SSH reachable · OPERATOR · Watchdog · MENTIONED
architecture folder: 5 lines in 4 files match /mgmt.watchdog|management.plane/ (a machine count, not a reading) · helper section: clock-agent „17. Management-plane watchdog timer (keeps SSH reachable)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Root systemd timer and oneshot installed by the host installer; script; works without the agent. OPERATOR. | A | E/scripts/felhom-host-install.sh:3067-3076 AG/configs/felhom-mgmt-watchdog.sh:26-68 AG/configs/felhom-mgmt-watchdog.service:3-7 |
| 2 | As written: OnBootSec=30s, OnUnitActiveSec=60s, AccuracySec=5s, Persistent=true; no OnCalendar, no RandomizedDelaySec. |
A | AG/configs/felhom-mgmt-watchdog.timer:11-14 |
| 3 | No time limit written in the unit; the script is a few stat and systemctl calls. | A | AG/configs/felhom-mgmt-watchdog.service:16-18 AG/configs/felhom-mgmt-watchdog.sh:36-61 |
| 4 | A missed tick is retried by the next one; the heal budget is one tick. | A | AG/configs/felhom-mgmt-watchdog.sh:23-24 AG/configs/felhom-mgmt-watchdog.timer:3-6 |
| 5 | The script changes nothing when the directory is healthy. MEASURED: the timer is active on the three hosts and fires each 60 seconds. Persistent=true in the unit has no effect, because the timer has no calendar line. |
A | AG/configs/felhom-mgmt-watchdog.sh:13-15 AG/configs/felhom-mgmt-watchdog.timer:11-14 EV/box/demo-hp-host-clock.txt |
| 6 | Monotonic timer settings only; the heal marker is a UTC stamp from date -u. |
A | AG/configs/felhom-mgmt-watchdog.timer:11-12 AG/configs/felhom-mgmt-watchdog.sh:65 |
| 7 | The marker lets the agent report the heal and the hub raise a warning,. | A | AG/configs/felhom-mgmt-watchdog.sh:18-19 AG/configs/felhom-mgmt-watchdog.sh:63 |
| 8 | SOURCE only. The script has no test file. | A | AG/configs/felhom-mgmt-watchdog.sh:1-68 AG/configs/felhom-mgmt-watchdog.timer:3-6 |
Hand step in a runbook? no hand step
CA18 — Agent service: restart, start limit, rollback · CUSTOMER · Watchdog · MENTIONED
architecture folder: 72 lines in 15 files match /Restart=|start limit|rollback/ (a machine count, not a reading) · helper section: clock-agent „18. Agent service restart, start limit and rollback”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | systemd restarts the agent: Restart=on-failure, RestartSec=5s; drop-in limits. OPERATOR (every loop in this list dies with the agent). |
A | AG/configs/felhom-agent.service:41-42 AG/configs/felhom-agent-limits.conf:17-19 |
| 2 | StartLimitIntervalSec=120, StartLimitBurst=4, OnFailure=felhom-agent-rollback.service; a crash-looping binary reaches failed about 20 s after the first crash. |
A | AG/configs/felhom-agent-limits.conf:17-19 AG/configs/felhom-agent-limits.conf:5-8 |
| 3 | No WatchdogSec in the unit (grep WatchdogSec in finds nothing); a new binary commits after a 60 s dwell. |
A | AG/configs/felhom-agent.service:1 AG/internal/selfupdate/commit.go:40-42 |
| 4 | After the start limit the unit stays failed. Nothing on the host starts it again. The outside sign is the hub's 'host stale' alarm at 45 minutes. | A | AG/configs/felhom-agent.service:41-42 AR/11-os-updates.md:572-573 |
| 5 | The rollback unit fires on every crash and is guarded by a pending marker; a shutdown before the dwell leaves the update pending. | A | AG/configs/felhom-agent-limits.conf:14-15 AG/internal/selfupdate/commit.go:112-116 |
| 6 | The design text says Restart=always and a systemd watchdog that kills a hung agent. The unit has Restart=on-failure and no watchdog. |
N | AR/03-host-agent.md:72 AG/configs/felhom-agent.service:41-42 · class W5 · S17 |
| 7 | The operator, by the hub's stale and down alarms only; a restore-test crash loop that never evaluates is covered by TestR874_CrashLoopNeverEvaluates. |
A | AR/11-os-updates.md:572-573 AG/internal/backup/r874_first_eval_test.go:44 |
| 8 | MEASURED values and a TEST for the rollback. A hung agent (alive, not working): NOBODY KNOWS; no watchdog restarts it. | N | AG/configs/felhom-agent.service:41-42 · class W5 · S17 |
Hand step in a runbook? PARTLY — RB/break-glass.md names reset-failed; no runbook names the start limit of the agent
CA19 — Off-site tunnel loop and the clock-ahead re-key · CUSTOMER · Interval · MENTIONED
architecture folder: 236 lines in 22 files match /wgtunnel|re.?key|tunnel/ (a machine count, not a reading) · helper section: clock-agent „19. Off-site tunnel loop and the clock-ahead re-key (with PBS-DR, OOB sshd and LAN resolver loops)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutines of one shape: tunnel Run, PBS-DR, OOB sshd, LAN resolver. CUSTOMER (off-site backup path) and OPERATOR. Four loops merged. |
A | AG/internal/wgtunnel/loop.go:62-78 AG/internal/pbsdr/loop.go:62-77 AG/internal/felhomsshd/loop.go:67-80 AG/internal/lanresolver/loop.go:35-52 |
| 2 | Tunnel, PBS-DR and OOB every 60 s, resolver every 5 min; each applies at once at start and on a desired-state nudge. | A | AG/internal/config/config.go:132-133 AG/cmd/felhom-agent/main.go:1504 AG/internal/config/config.go:74-75 AG/internal/lanresolver/loop.go:27-29 |
| 3 | No time limit on one pass in the source, and no duration was measured. Nothing grows with data. | N | class W4 · S15 |
| 4 | A missed pass is made up at the next tick; a handshake older than StaleAfterSeconds 180 re-resolves the endpoint,. |
A | AG/internal/config/config.go:124-127 AG/internal/config/config.go:138-140 |
| 5 | A handshake stamped in the future for rekeyAfterDefault 15 min makes a new key, at most once per rekeyMinGap 1 h; the hour bound survives a restart, the 15 min count does not. |
A | AG/internal/wgtunnel/rekey.go:36-38 AR/03-host-agent.md:376-378 |
| 6 | A box whose clock once ran ahead is locked out of the off-site endpoint until real time passes that moment; healed by the re-key; known gap: a reboot after the clock was corrected forgets the future stamp. | A | AR/03-host-agent.md:142 AR/03-host-agent.md:144-150 AR/03-host-agent.md:155-156 |
| 7 | The heartbeat carries handshake_future_s and the hub alarms on it. |
A | AR/03-host-agent.md:142 |
| 8 | TEST TestD30_AFutureHandshakeThatStaysReKeys, TestD30_ALastReKeyFromTheFutureDoesNotBlock, TestR1204_AFutureHandshakeIsReported; measured live. |
A | AG/internal/wgtunnel/rekey_test.go:35 AG/internal/wgtunnel/rekey_test.go:213 AG/internal/wgtunnel/manager_test.go:704 AR/03-host-agent.md:151-155 |
Hand step in a runbook? no hand step
CA20 — Whole-guest backup cap: rolling windows · CUSTOMER · Night · MENTIONED
architecture folder: 577 lines in 27 files match /backup cap|rolling window|cap/ (a machine count, not a reading) · helper section: clock-agent „20. Whole-guest backup cap (rolling windows)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent local API check capRefuses before a backup starts; rule backupCadenceCap. CUSTOMER (protects older copies from a flood of new ones). |
A | AG/internal/localapi/server.go:929 AG/internal/localapi/backup_cap.go:41-53 |
| 2 | Tier of 24 h or less: 3 successes per rolling 24 h; longer tier: 1 per the larger of cadence minus 24 h and half the cadence; a refusal holds for backupCapRefusalHold 24 h. |
A | AG/internal/localapi/backup_cap.go:12-35 AG/internal/localapi/backup_cap.go:38 |
| 3 | A lookup, no duration of its own; the success list is bounded by the window. | A | AG/internal/localapi/backup_cap.go:41-53 |
| 4 | Does not apply to a missed run: failures do not count, TestBackupCap_FailuresDoNotCount. |
A | AG/internal/localapi/backup_cap_test.go:266 |
| 5 | The scheduled daily run is never refused, TestBackupCap_TheScheduledDayIsNeverRefused; the count survives a restart, TestBackupCap_SurvivesARestart. |
A | AG/internal/localapi/backup_cap_test.go:220 AG/internal/localapi/backup_cap_test.go:248 |
| 6 | The windows use the wall clock. Successes that were stamped while the clock was ahead stay inside the window longer and can refuse real runs. NOBODY KNOWS. | N | AG/internal/localapi/server.go:509 · class W2 · S08 |
| 7 | The operator, by event whole_guest_backup_refused named in; test TestBackupCap_RefusalRaisesTheOperatorSignal. |
A | AG/internal/localapi/backup_cap.go:12-35 AG/internal/localapi/backup_cap_test.go:285 |
| 8 | TEST TestBackupCap_LocalFloodRunsAtMostThreeADay, TestBackupCap_OffsiteFloodKeepsAPreCompromiseCopy. |
A | AG/internal/localapi/backup_cap_test.go:198 AG/internal/localapi/backup_cap_test.go:151 |
Hand step in a runbook? no hand step
CA21 — Unsent OS-update report sender · OPERATOR · Interval · UNNAMED
architecture folder: 0 lines in 0 files match /unsent/ (a machine count, not a reading) · helper section: clock-agent „22. Unsent OS-update report sender”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Agent goroutine SendUnsentLoop, started at; posts OS-update results the hub has not received. OPERATOR. |
A | AG/internal/osupdate/unsent.go:71-82 AG/cmd/felhom-agent/main.go:1009 |
| 2 | Every 5 min, by time.After; one send at start, SendUnsent. |
A | AG/cmd/felhom-agent/main.go:1009 AG/internal/osupdate/unsent.go:79 AG/internal/osupdate/unsent.go:56-64 |
| 3 | A report post has a 1 min limit in finish; the kept copies grow only while the hub is away. |
A | AG/internal/osupdate/leg.go:922-939 |
| 4 | Hub away keeps the copy for a later pass, TestR868_HubAwayKeepsTheCopy. |
A | AG/internal/osupdate/unsent_test.go:87 |
| 5 | The sender stays off while a pass holds the flock, tryLockPass; test TestR868_RunningPassKeepsTheSenderOff. |
A | AG/internal/osupdate/unsent.go:35-52 AG/internal/osupdate/unsent_test.go:110 |
| 6 | time.After is monotonic; no wall-clock decision found in. |
A | AG/internal/osupdate/unsent.go:79 AG/internal/osupdate/unsent.go:56-82 |
| 7 | The operator sees the result late: at most 5 minutes after the hub is back. The sender has no alarm of its own. | A | AG/internal/osupdate/unsent.go:1 |
| 8 | TEST TestR868_ACopyWrittenAfterTheStartIsSentByTheLoop, TestR868_KilledPassIsReportedAtStart. |
A | AG/internal/osupdate/unsent_test.go:127 AG/internal/osupdate/unsent_test.go:20 |
Hand step in a runbook? no hand step
CC01 — Night dump leg: databases and volumes · CUSTOMER · Night · MENTIONED
architecture folder: 17 lines in 5 files match /db-dump|dump leg/ (a machine count, not a reading) · helper section: clock-controller „1. Night dump leg (databases and volumes) at W”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | dbDumpLeg runs RunDBDumps; owner: controller; started by the daily job "db-dump". CUSTOMER. |
A | CT/cmd/controller/main.go:1336-1344 CT/internal/backup/backup.go:564 CT/cmd/controller/main.go:1345 |
| 2 | Daily at W. W = household setting, else DBDumpSchedule, else DefaultWindow "02:30", decided in EffectiveWindow. First leg. Zone: Europe/Budapest, hard-coded, getBudapestLocation. |
A | CT/internal/backupwindow/backupwindow.go:12 CT/internal/backupwindow/backupwindow.go:86-94 CT/internal/scheduler/scheduler.go:17-27 |
| 3 | MEASURED, the night of 2026-11-11: 2 min 33 s on demo-hp, 40 s on Tester 1, 18 s on the N100. One database dump has a 5-minute limit. The whole leg has no limit. It grows with the number and size of the apps; all three stores are small. | A | EV/box/demo-hp-guest-night.txt EV/box/tester-1-guest-night.txt EV/box/demo-felhom-guest-night.txt CT/internal/appbackup/dbdump.go:291 |
| 4 | Box off at W: Missed sees it from the ledger and the catch-up runs it (block 4). The hub judges a missing dump at its 05:00 deadline. |
A | CT/internal/nightchain/nightchain.go:197-218 AR/07-backup-architecture.md:579-584 |
| 5 | withLeg takes the leg lock; acquireRunning refuses a second backup; a job still running is skipped executeJob. A running migration skips the leg and returns no error. |
A | CT/cmd/controller/main.go:1321-1335 CT/internal/backup/backup.go:1105-1113 CT/internal/scheduler/scheduler.go:354-364 CT/internal/backup/backup.go:565-568 |
| 6 | Slot from the wall clock, waited on a monotonic timer runDailyJob; fired over 60 min late = skipped DailyLateLimit. See block 24 for the hour change. |
A | CT/internal/scheduler/scheduler.go:300-352 CT/internal/scheduler/scheduler.go:67 |
| 7 | Failure: NotifyDBDumpFailed. A made-up miss: one timeline line, never mailed. A miss not made up: banner after 26 h StaleAfter. |
A | CT/cmd/controller/main.go:1339 AR/07-backup-architecture.md:562-564 CT/internal/nightchain/banner.go:23 |
| 8 | MEASURED 2026-11-11, one night, small stores: 2 min 33 s · 40 s · 18 s (demo-hp · Tester 1 · N100) · TEST TestCatchUp_OffAtWThenOn_OneCatchUpAfter15Min; TEST TestDaily_LateFireIsSkipped. |
A | EV/box/demo-hp-guest-night.txt EV/box/tester-1-guest-night.txt EV/box/demo-felhom-guest-night.txt CT/internal/nightchain/nightchain_test.go:92 CT/internal/scheduler/scheduler_test.go:87 |
Hand step in a runbook? no hand step
CC02 — Second-drive copy leg · CUSTOMER · Night · MENTIONED
architecture folder: 238 lines in 20 files match /tier.?2|second.drive|cross.drive/ (a machine count, not a reading) · helper section: clock-controller „2. Second-drive copy leg at W+60 min”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | RunAllTier2 copies each app's data to the second drive; owner: controller; started by the daily job "tier2-backup". CUSTOMER. |
A | CT/internal/backup/tier2.go:534 CT/cmd/controller/main.go:1442 |
| 2 | Daily at W+60 min, tier2OffsetMin; no setting of its own. It does not wait for the dump leg: "NOTHING waits on the leg before it". Zone as block 1. |
A | CT/internal/backupwindow/backupwindow.go:17 AR/09-update-architecture.md:2751-2757 |
| 3 | One copy: 60 min, tier2CopyTimeout, applied at. No limit for the whole leg found (same grep as block 1). Grows with app count and data size. |
A | CT/internal/backup/tier2_copy_lifecycle.go:40 CT/internal/backup/tier2.go:828 |
| 4 | Catch-up as block 4 (the leg is in the ledger through withLeg). The leg function always returns nil, so the scheduler never sees a failure. |
A | CT/cmd/controller/main.go:1438-1441 |
| 5 | It waits for the dump leg by a start time only (60 minutes later). It is skipped while a migration runs. The file holds no lock of its own. With a backup time between 02:00 and 02:59 on the spring hour-change night the two legs start at the same instant; that case is row R-1260. | N | CT/cmd/controller/main.go:1321-1335 CT/internal/backup/tier2.go:538-541 CT/internal/backup/tier2_copy_lifecycle.go:172 · class W3 · S14 |
| 6 | As block 1: wall-clock slot, monotonic wait, 60 min late limit. | A | CT/internal/scheduler/scheduler.go:67 |
| 7 | The household and the hub are told of a failed copy: the copy's result goes to the notifier. | A | CT/cmd/controller/main.go:1427 |
| 8 | MEASURED 2026-11-11: 39 s · 15 s · 11 s · SOURCE; catch-up part TEST TestCatchUp_PowerCutMidChain_FinishesTheRest. |
A | EV/box/demo-hp-guest-night.txt EV/box/tester-1-guest-night.txt EV/box/demo-felhom-guest-night.txt CT/internal/backup/tier2.go:534 CT/internal/nightchain/nightchain_test.go:145 |
Hand step in a runbook? no hand step
CC03 — Off-site push leg, with the clean-up of old snapshots · CUSTOMER · Night · MENTIONED
architecture folder: 82 lines in 13 files match /offbox|off.site push|offsite push/ (a machine count, not a reading) · helper section: clock-controller „3. Off-site push leg at W+105 min (with clean-up of old snapshots)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | RunOffboxBackup pushes dumps to the remote store, then runs retention; owner: controller; started by the daily job "offbox-backup". CUSTOMER. |
A | CT/internal/backup/offbox.go:1018 CT/cmd/controller/main.go:1641-1643 |
| 2 | Daily at W+105 min, offboxOffsetMin. Retention keeps 7 daily, keepDaily. Zone as block 1. |
A | CT/internal/backupwindow/backupwindow.go:18 CT/internal/backup/offbox_window.go:47 |
| 3 | Per app 2 h, offboxBackupTimeout, applied at; SSH connect 10 s. Grows with data changed and line speed. One real night: 3 min 28 s. |
A | CT/internal/backup/offbox.go:39 CT/internal/backup/offbox.go:1564 CT/internal/backup/offbox.go:37 AR/09-update-architecture.md:2751-2757 |
| 4 | Catch-up as block 4. No escrowed key, orphaned store or migration: skipped with an INFO log line and nil. | A | CT/internal/backup/offbox.go:1040-1053 |
| 5 | When another backup holds the flag, the push is skipped with an INFO line and the leg is marked as ended. Only the hub's 48-hour check notices. A long second-drive leg can cost the night's off-site copy. | K | CT/internal/backup/offbox.go:1054-1057 · R-1186 |
| 6 | As block 1. The clean-up window has a one-hour clock tolerance offsiteGuardSkew. |
A | CT/internal/backup/offbox_window.go:42 |
| 7 | Failure: NotifyOffboxBackupFailed with a cleaned message. The hub's daily deadline is the independent signal. |
A | CT/cmd/controller/main.go:1446-1452 CT/cmd/controller/main.go:1524-1528 |
| 8 | MEASURED 2026-11-11: 3 min 09 s · 59 s · 38 s · SOURCE; DOC. | A | EV/box/demo-hp-guest-night.txt EV/box/tester-1-guest-night.txt EV/box/demo-felhom-guest-night.txt CT/internal/backup/offbox.go:1040-1057 AR/07-backup-architecture.md:455-457 |
Hand step in a runbook? NO — no runbook says how to see or repair a skipped off-site night
CC04 — Missed-night catch-up and the resume watch · CUSTOMER · Night · MENTIONED
architecture folder: 43 lines in 12 files match /catch.?up|nightchain|missed night/ (a machine count, not a reading) · helper section: clock-controller „4. Missed-night catch-up and resume watch”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CatchUp.Evaluate runs the backup legs a box missed; owner: controller; started at controller start and by the resume watch. CUSTOMER. |
A | CT/internal/nightchain/nightchain.go:281-299 CT/cmd/controller/main.go:1676 CT/cmd/controller/main.go:1677 |
| 2 | 15 min after the trigger, DefaultDelay; a leg due within 30 min is left alone, leaveToNormal. Values decided by CC. No setting. |
A | CT/internal/nightchain/nightchain.go:62 CT/internal/nightchain/nightchain.go:64 AR/07-backup-architecture.md:574-577 |
| 3 | As long as the legs it runs. It waits up to 2 h for a whole-guest backup, quiesceWaitMax, then drops with a WARN. |
A | CT/internal/nightchain/nightchain.go:66 CT/internal/nightchain/nightchain.go:319 |
| 4 | If the catch-up itself does not run: banner after 26 h (block 5) and the hub's longer lines, 48 h without a dump. Two missed nights give one catch-up, not two. | A | AR/07-backup-architecture.md:579-584 |
| 5 | One at a time: pending/running guard; leg lock. It never runs the update leg. | A | CT/internal/nightchain/nightchain.go:288-292 CT/cmd/controller/main.go:1321-1335 CT/cmd/controller/main.go:1660-1664 |
| 6 | The watch reads a gap of more than 5 minutes in the wall clock as 'the box was asleep'. A clock that is corrected forward by more than 5 minutes looks the same. The design text says this was not measured live. | N | CT/internal/nightchain/nightchain.go:375 CT/cmd/controller/main.go:1681 AR/07-backup-architecture.md:551-559 · class W2 · S08 |
| 7 | Household: one timeline line, never mailed, event sent at. Dropped catch-up: log line only. | A | AR/07-backup-architecture.md:562-564 CT/internal/nightchain/nightchain.go:357-359 CT/internal/nightchain/nightchain.go:319 |
| 8 | TEST TestCatchUp_TwoMissedNights_One; TEST TestResumeWatch_SeesASuspend; TEST TestR871_CatchUpWiring. |
A | CT/internal/nightchain/nightchain_test.go:126 CT/internal/nightchain/nightchain_test.go:217 CT/cmd/controller/r871_catchup_wiring_test.go:16 |
Hand step in a runbook? no hand step
CC05 — Missed-backup banner (26 hours) · CUSTOMER · Night · MENTIONED
architecture folder: 48 lines in 14 files match /26 h|missed.backup banner|banner/ (a machine count, not a reading) · helper section: clock-controller „5. Missed-backup banner (26 hours)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | ComputeBanner shows the household a banner when no night backup ran; owner: controller; computed when the page is drawn (no timer of its own). CUSTOMER. |
A | CT/internal/nightchain/banner.go:56-88 |
| 2 | Shown when the last backup is older than 26 h, StaleAfter. After 5 missed days it suggests another hour, suggestMinDays, Suggest. |
A | CT/internal/nightchain/banner.go:23 CT/internal/nightchain/banner.go:26 CT/internal/nightchain/banner.go:95-113 |
| 3 | Instant; reads the ledger only. | A | CT/internal/nightchain/banner.go:56-88 |
| 4 | Nobody opens the page: nobody sees it. The hub's deadline check is the other channel. | A | AR/07-backup-architecture.md:579-584 |
| 5 | Not a job; no lock needed. Dismissed, it comes back after a new miss, design at. | A | AR/07-backup-architecture.md:566-572 |
| 6 | The age is wall-clock arithmetic. A clock that is set back hides the banner for as long as the step. No test sets the clock back. | N | CT/internal/nightchain/banner.go:56-88 · class W2 · S08 |
| 7 | The household only, on the page; no mail. | A | AR/07-backup-architecture.md:566-572 |
| 8 | TEST TestBanner_ShownWithReasonAndSuggestion; TEST TestBanner_DismissedThenBackAfterANewMiss. |
A | CT/internal/nightchain/banner_test.go:29 CT/internal/nightchain/banner_test.go:59 |
Hand step in a runbook? no hand step
CC06 — Whole-guest backup driver: stop apps, ask the agent, start apps · CUSTOMER · Night · MENTIONED
architecture folder: 264 lines in 21 files match /quiesce|whole.guest|vzdump/ (a machine count, not a reading) · helper section: clock-controller „6. Whole-guest backup driver (stop apps, ask the host agent, start apps)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Loop.Run stops the apps and asks the agent for the whole-guest backup; owner: controller drives, agent does the backup; started by startQuiesceLoop. CUSTOMER. |
A | CT/internal/quiesce/quiesce.go:191-206 CT/cmd/controller/main.go:4008 |
| 2 | Polls every 5 min; runs once per 24 h cadence, only inside W+2h to W+6h, gateOpenOffsetMin, gateSpanMin. Zone budapestLocation. |
A | CT/internal/quiesce/quiesce.go:149-151 CT/internal/quiesce/quiesce.go:161-163 CT/internal/quiesce/quiesce.go:760 CT/internal/quiesce/quiesce.go:761 |
| 3 | Apps stay down at most 30 min, MaxQuiesce default, setting max_quiesce_duration. Manual run limit = that + 5 min. Grows with guest disk size. |
A | CT/cmd/controller/main.go:3990 CT/internal/config/config.go:132 CT/internal/quiesce/quiesce.go:430 |
| 4 | Outside the window it still runs when the last one is older than cadence + 24 h (the 48 h valve). Apps left stopped by a crash are started by Recover, called at. |
A | CT/internal/quiesce/quiesce.go:801-818 CT/internal/quiesce/quiesce.go:177-188 CT/cmd/controller/main.go:4007 |
| 5 | Single-flight TryLock; waits for a catch-up; waits for the update leg, never past W+5h30 legStepGraceMin. |
A | CT/internal/quiesce/quiesce.go:213-216 CT/internal/quiesce/quiesce.go:250-253 CT/internal/quiesce/quiesce.go:271-274 CT/internal/quiesce/quiesce.go:935 |
| 6 | The gate counts wall-clock minutes; the poll is a monotonic ticker. On the hour-change night the two disagree. No test covers that night. | K | CT/internal/quiesce/quiesce.go:801-818 CT/internal/quiesce/quiesce.go:191-206 · R-1260 |
| 7 | First failure notifies once noteTierFailure; retry back-off 15 min to 4 h, busy agent: retry 15 min, alarm after 3 h,. |
A | CT/internal/quiesce/quiesce.go:283-298 CT/internal/quiesce/breaker.go:47 CT/internal/quiesce/breaker.go:54 CT/internal/quiesce/contention.go:53 |
| 8 | TEST TestScheduledRunAllowed; TEST TestRunOnce_ValveRunsOutsideWindow; TEST TestBreaker_BackoffSchedule. |
A | CT/internal/quiesce/quiesce_window_test.go:23 CT/internal/quiesce/quiesce_window_test.go:94 CT/internal/quiesce/breaker_test.go:218 |
Hand step in a runbook? no hand step
CC07 — Automatic app-update leg · CUSTOMER · Night · MENTIONED
architecture folder: 136 lines in 17 files match /update.leg|unattended/ (a machine count, not a reading) · helper section: clock-controller „7. Automatic app-update leg (after the off-site push)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | RunUpdateLeg presses one update step per app; owner: controller; started by chainUpdateLeg when the off-site half ends, on every path. CUSTOMER. |
A | CT/internal/stacks/unattended.go:230 CT/cmd/controller/main.go:1641-1643 |
| 2 | Each night after the off-site leg; a box with no off-site tier runs it at W+105 min. One step per app per night, switch default ON. | A | AR/07-backup-architecture.md:459-465 AR/09-update-architecture.md:2806-2817 |
| 3 | No new step at or after W+5h: LegDeadline, UpdateLegStopOffsetMin; check at. Grows with the number of apps that have an update. |
A | CT/internal/stacks/unattended.go:212-216 CT/internal/backupwindow/backupwindow.go:27 CT/internal/stacks/unattended.go:374-375 |
| 4 | Missed night: no catch-up by design; the rest "waits for the next night". Controller restarted mid-leg: the rest of the night is lost (R-686). | A | CT/cmd/controller/main.go:1660-1664 CT/internal/stacks/unattended.go:374-375 AR/09-update-architecture.md:2815-2817 |
| 5 | A passing refusal (busy, updating) is retried once after 2 min, RetryWait, rule at. The whole-guest gate waits for the leg. |
A | CT/internal/stacks/unattended.go:167-171 AR/09-update-architecture.md:2882-2883 CT/internal/quiesce/quiesce.go:271-274 |
| 6 | The leg's deadline is the backup time plus 5 hours of real time, and the scheduler's slots are in Budapest time. On the hour-change night a step can start after the backup gate stopped waiting. A daily fire that is more than 60 minutes late is skipped, so no update runs at noon. | K | CT/internal/stacks/unattended.go:212-216 CT/internal/scheduler/scheduler.go:17-27 AR/07-backup-architecture.md:551-559 · R-1260 |
| 7 | Nobody. When the leg does not run, the only trace is one log line on the box. No event goes to the hub. | N | CT/internal/stacks/unattended.go:374-375 · class W1 · S06 |
| 8 | TEST TestChainUpdateLeg_EveryPath; TEST TestLegDeadlineAndGateShareW5h. |
A | CT/cmd/controller/update_leg_wiring_test.go:19 CT/internal/quiesce/d20_update_leg_test.go:78 |
Hand step in a runbook? no hand step
CC08 — Off-site integrity check · CUSTOMER · Night · MENTIONED
architecture folder: 23 lines in 9 files match /integrity/ (a machine count, not a reading) · helper section: clock-controller „8. Off-site integrity check at 06:00”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Daily job "offsite-integrity" checks the remote store's data; owner: controller. CUSTOMER. | A | CT/cmd/controller/main.go:1724 |
| 2 | Fires daily at 06:00 (fixed, no setting for the hour) but works only when due: last good check older than 7 days, defaultIntegrityMaxAgeDays, IntegrityDue. Zone as block 1. |
A | CT/internal/backup/offbox_integrity.go:81 CT/internal/backup/offbox_integrity.go:221 |
| 3 | MEASURED 2026-11-11: under 1 second on all three boxes (small stores). The check reads the whole store; the time on a large store rests on one measurement. | K | EV/box/demo-hp-guest-night.txt · R-401 |
| 4 | Box off at 06:00: caught up at the next 06:00 by due-ness; also run after a catch-up. | A | CT/internal/backup/offbox_integrity.go:212-220 CT/cmd/controller/main.go:1665-1668 |
| 5 | Skipped when "a backup or restore is already running"; then it waits for the next 06:00 (source comment). | A | CT/internal/backup/offbox_integrity.go:298 CT/cmd/controller/main.go:1710-1723 |
| 6 | Due-ness compares wall-clock stamps; a never-checked or unreadable stamp counts as due. Slot behaviour as block 24. | A | CT/internal/backup/offbox_integrity.go:231 CT/internal/backup/offbox_integrity.go:223-229 |
| 7 | A failed check goes to the notifier: the household and the hub are told. | A | CT/cmd/controller/main.go:4144-4186 |
| 8 | MEASURED 2026-11-11: under 1 s on each box · SOURCE; TEST TestR1189d_TheChecksRunAfterTheCatchUpLegs. |
A | EV/box/demo-hp-guest-night.txt EV/box/tester-1-guest-night.txt EV/box/demo-felhom-guest-night.txt CT/internal/backup/offbox_integrity.go:212-232 CT/internal/nightchain/r1189d_test.go:13 |
Hand step in a runbook? no hand step
CC09 — Off-site proof restore · CUSTOMER · Night · MENTIONED
architecture folder: 136 lines in 21 files match /proof restore|offsite.proof|proof/ (a machine count, not a reading) · helper section: clock-controller „9. Off-site proof restore at 05:30”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Daily job "offsite-proof" restores one due snapshot to prove it opens, ProveOffboxUnit; owner: controller. CUSTOMER. |
A | CT/cmd/controller/main.go:1746 CT/internal/backup/offbox_proof.go:140 |
| 2 | Daily at 05:30, fixed, no setting. One app per night in rotation (comment). Zone as block 1. | A | CT/cmd/controller/main.go:1746 CT/internal/backup/offbox_proof.go:265 |
| 3 | Limit 10 min, proofRestoreTimeout. Grows with the size of the one unit. |
A | CT/internal/backup/offbox_proof.go:80 |
| 4 | Box on for the backup legs but off at 05:30: "that day's proof is not made up". Made up only together with a missed backup leg. | A | CT/cmd/controller/main.go:1660-1664 CT/cmd/controller/main.go:1665-1668 |
| 5 | MEASURED 2026-11-11: on Tester 1 and on the N100 the whole-guest backup started at about 05:30 local time, the same minute as this job. This job is not behind the whole-guest gate. It takes the same flag as the off-site push; a meeting there is a silent skip (R-1186). | N | EV/box/tester-1-host-night.txt EV/box/demo-felhom-host-night.txt EV/box/tester-1-guest-night.txt · class W3 · S12 |
| 6 | As block 24. | A | CT/cmd/controller/main.go:1746 CT/internal/backup/offbox_proof.go:140 (the code path of question 1) |
| 7 | An empty or failed proof goes to the notifier: the household and the hub are told. | A | CT/cmd/controller/main.go:4144-4186 |
| 8 | MEASURED 2026-11-11: 5 s · 5 s · 3 s · SOURCE; TEST TestC5_TheWholeGuestBackupDoesNotWaitForTheChecks. |
A | EV/box/demo-hp-guest-night.txt EV/box/tester-1-guest-night.txt EV/box/demo-felhom-guest-night.txt CT/cmd/controller/main.go:1660-1664 CT/internal/nightchain/r1189d_test.go:51 |
Hand step in a runbook? no hand step
CC10 — Hub report: every 15 minutes, on change, and the hanging wait · CUSTOMER · Interval · MENTIONED
architecture folder: 91 lines in 22 files match /15.min|hub report|pusher/ (a machine count, not a reading) · helper section: clock-controller „10. Hub report every 15 minutes (plus push on change and hanging wait)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Job "hub-report" sends the box state to the hub; owner: controller. The hub's answer carries the version floor and config. CUSTOMER and OPERATOR. | A | CT/cmd/controller/main.go:1835 CT/cmd/controller/main.go:1233-1290 |
| 2 | Every PushInterval, default "15m", fallback. Also 5 s after start, and on change with a 15 s floor, triggerMinInterval. No time zone. |
A | CT/internal/config/config.go:416 CT/cmd/controller/main.go:1831-1834 CT/cmd/controller/main.go:1932-1936 CT/internal/report/trigger.go:30 |
| 3 | HTTP limit 30 s; 3 tries, 5 s apart. The wait connection holds up to 300 s. | A | CT/internal/report/pusher.go:85 CT/internal/report/pusher.go:114-117 CT/internal/report/waiter.go:35 |
| 4 | A failed push is not queued: "next cycle reconciles". The hub notices silence (hub side, not read here). | A | CT/internal/report/trigger.go:98 |
| 5 | A report still running is skipped. Bursts are merged after a 2 s quiet window. | A | CT/internal/scheduler/scheduler.go:354-364 CT/internal/report/trigger.go:27 |
| 6 | Monotonic ticker; a wrong wall clock does not change the rhythm, only the stamps inside the report. | A | CT/internal/scheduler/scheduler.go:281-298 |
| 7 | Nobody on the box: the box stores the time of the last attempt only. The outside sign is the hub's liveness check (CH01). | A | CT/internal/report/pusher.go:109-111 HUB/monitor/host_staleness.go:59-60 |
| 8 | TEST TestTrigger_BurstCoalescesTrailingEdge; TEST TestWaiter_FiresOnceOnGenChange. |
A | CT/internal/report/trigger_test.go:114 CT/internal/report/waiter_test.go:60 |
Hand step in a runbook? no hand step
CC11 — Controller self-update: check, floor, nightly auto · CUSTOMER · Interval · MENTIONED
architecture folder: 65 lines in 13 files match /self.?update/ (a machine count, not a reading) · helper section: clock-controller „11. Controller self-update: check, floor, nightly auto”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CheckForUpdate looks for a newer controller image; MaybeAutoUpdate moves a box below the floor. Owner: controller, the agent swaps the image. CUSTOMER and OPERATOR. |
A | CT/internal/selfupdate/updater.go:153 CT/internal/selfupdate/updater.go:612 |
| 2 | Check every CheckInterval, default "6h", and 25 s after start. Floor: after every hub report. Auto: daily at AutoUpdateTime, default "04:30", only if AutoUpdate. |
A | CT/internal/config/config.go:417 CT/cmd/controller/main.go:1981-1989 CT/cmd/controller/main.go:1249-1252 CT/internal/config/config.go:419 |
| 3 | The registry call has a 15-second limit. The swap of the container has no limit in the file, and no duration was measured in this run. | N | CT/internal/selfupdate/updater.go:245 · class W4 · S15 |
| 4 | No catch-up needed for the floor: it is tried again on each report until done, but one attempt per floor value. A failed attempt at a floor is not repeated. | A | CT/internal/selfupdate/updater.go:656-661 CT/internal/selfupdate/updater.go:604-606 |
| 5 | The auto-update waits while a backup or an app update runs. Its default time, 04:30, is the minute at which the whole-guest gate opens for a 02:30 backup time. Nothing orders the two. | N | CT/internal/config/config.go:419 CT/internal/selfupdate/updater.go:533-544 · class W3 · S12 |
| 6 | The check uses a monotonic ticker. The nightly auto-update is a daily slot (see CC22). The switch is off unless the config turns it on; the default time is 04:30. | A | CT/internal/config/config.go:278-279 CT/internal/config/config.go:419 |
| 7 | Auto-update refused: WARN log only. A version the floor cannot compare: WARN once. | A | CT/cmd/controller/main.go:1879-1881 CT/internal/selfupdate/updater.go:639 |
| 8 | TEST TestMaybeAutoUpdate_NoFlap; TEST TestMaybeAutoUpdate_BelowFloor_UpdatesToFloor. |
A | CT/internal/selfupdate/floor_test.go:93 CT/internal/selfupdate/floor_test.go:11 |
Hand step in a runbook? no hand step
CC12 — Catalog sync: app templates from git · CUSTOMER · Interval · MENTIONED
architecture folder: 147 lines in 18 files match /catalog sync|git.sync|sync/ (a machine count, not a reading) · helper section: clock-controller „12. Catalog sync (app templates from git)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Syncer.Start pulls the app catalog; owner: controller; started at controller start. CUSTOMER. |
A | CT/internal/sync/sync.go:125 |
| 2 | Once at start, then every SyncInterval, default "15m"; a bad value falls back to 15 min. No time zone. |
A | CT/internal/sync/sync.go:140-143 CT/internal/config/config.go:399 CT/internal/sync/sync.go:131-135 |
| 3 | Each git command has its own limit. One whole sync has none, and no duration was measured. | N | CT/internal/sync/sync.go:724 CT/internal/sync/sync.go:215-216 · class W4 · S15 |
| 4 | A missed sync is done at the next tick. A sync that fails for days writes an ERROR line on the box at each tick. Nobody is told. | N | CT/internal/sync/sync.go:236 CT/internal/sync/sync.go:247 · class W1 · S23 |
| 5 | Two syncs cannot run at once: doSync takes the syncer's lock first. The 30-second guard is for the button only. |
A | CT/internal/sync/sync.go:215-216 CT/internal/sync/sync.go:153-155 |
| 6 | Monotonic ticker; own ticker, not the shared scheduler, so it has no "still running" skip from. | A | CT/internal/sync/sync.go:146 CT/internal/scheduler/scheduler.go:354-364 |
| 7 | Log line per sync only. | A | CT/internal/sync/sync.go:155 |
| 8 | SOURCE. | A | CT/internal/sync/sync.go:124-159 |
Hand step in a runbook? no hand step
CC13 — System health check and base-stack self-heal · CUSTOMER · Interval · MENTIONED
architecture folder: 57 lines in 16 files match /health check|self.heal/ (a machine count, not a reading) · helper section: clock-controller „13. System health check and base-stack self-heal”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Job "system-health" checks disk, memory and base services, and puts the base stack back; owner: controller. CUSTOMER. | A | CT/cmd/controller/main.go:981-1006 |
| 2 | Every SystemHealthInterval, default "5m", fallback; env override. First run one interval after start. |
A | CT/internal/config/config.go:409 CT/cmd/controller/main.go:977-980 CT/internal/config/config.go:453 CT/internal/scheduler/scheduler.go:281-298 |
| 3 | No time limit in the source, and no duration was measured. Nobody knows how long it takes on a big box. | N | CT/cmd/controller/main.go:981-1006 · class W4 · S15 |
| 4 | No catch-up; next tick. A controller that is down checks nothing; the hub's missing reports are the outside sign (block 10). | A | CT/cmd/controller/main.go:981-1006 (the code path of question 1) |
| 5 | Skipped while the previous run is still going. | A | CT/internal/scheduler/scheduler.go:354-364 |
| 6 | Monotonic ticker; not sensitive to wall clock. | A | CT/cmd/controller/main.go:981-1006 (the code path of question 1) |
| 7 | NotifyHealthChange on a change of state. |
A | CT/cmd/controller/main.go:1004 |
| 8 | SOURCE. | A | CT/cmd/controller/main.go:981-1006 |
Hand step in a runbook? no hand step
CC14 — Dead-app scan · CUSTOMER · Interval · MENTIONED
architecture folder: 62 lines in 13 files match /stack.scan|dead.app|crash.?loop/ (a machine count, not a reading) · helper section: clock-controller „14. Dead-app scan every 30 seconds”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Job "deadapp-check" finds apps that should run and do not; owner: controller. CUSTOMER. | A | CT/cmd/controller/main.go:1086-1092 |
| 2 | Every 30 s; silent for 90 s after start, about 15 min after a crash boot. | A | CT/cmd/controller/main.go:1087-1088 |
| 3 | No time limit in the source, and no duration was measured. Nobody knows how long it takes on a big box. It grows with the number of apps. | N | CT/cmd/controller/main.go:1086-1143 · class W4 · S15 |
| 4 | Next tick; no catch-up needed. | A | CT/cmd/controller/main.go:1086-1092 (the code path of question 1) |
| 5 | Skipped if still running. It reads the whole-guest loop state so a planned stop is not an alarm. | A | CT/internal/scheduler/scheduler.go:354-364 CT/cmd/controller/main.go:1090 |
| 6 | The quiet time after a start uses the wall clock. A clock that steps back after the start makes the quiet time longer. No test. | N | CT/cmd/controller/main.go:1087 · class W2 · S08 |
| 7 | Alert on the page via SetDeadAppAlerts. |
A | CT/cmd/controller/main.go:1091 |
| 8 | SOURCE. | A | CT/cmd/controller/main.go:1086-1092 |
Hand step in a runbook? no hand step
CC15 — Disk-fill watch · CUSTOMER · Interval · MENTIONED
architecture folder: 4 lines in 3 files match /fill.?watch|disk.fill/ (a machine count, not a reading) · helper section: clock-controller „15. Disk-fill watch”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | fillWatcher.Check watches how full the drives are; owner: controller; three starters. CUSTOMER. |
A | CT/cmd/controller/main.go:1804-1805 |
| 2 | Every 10 min, fillWatchInterval; daily at 03:30; once 90 s after start, fillWatchStartupDelay. Cadence stated at. |
A | CT/cmd/controller/main.go:2690 CT/cmd/controller/main.go:1804 CT/cmd/controller/main.go:2685 CT/cmd/controller/main.go:1772 |
| 3 | No time limit in the source, and no duration was measured. Nobody knows how long it takes on a big box. | N | CT/cmd/controller/main.go:1804-1805 · class W4 · S15 |
| 4 | Next 10-minute tick. | A | CT/cmd/controller/main.go:1804-1805 (the code path of question 1) |
| 5 | The night slot and the interval job are two jobs with two names. The watcher holds one lock, so the two cannot change its state at the same time. | A | CT/internal/fillwatch/fillwatch.go:162 CT/cmd/controller/main.go:1804-1805 |
| 6 | Interval is monotonic; the 03:30 slot as block 24. | A | CT/cmd/controller/main.go:1804-1805 (the code path of question 1) |
| 7 | The household: the watcher pushes a customer event when a disk fills. | A | CT/internal/fillwatch/fillwatch.go:80-153 |
| 8 | TEST TestFillWatchRunsOnAnInterval. |
A | CT/cmd/controller/r363_fillwatch_interval_test.go:17 |
Hand step in a runbook? no hand step
CC16 — Drive health check, hourly · CUSTOMER · Interval · MENTIONED
architecture folder: 15 lines in 6 files match /SMART|disk.health|drive health/ (a machine count, not a reading) · helper section: clock-controller „16. Drive health check, hourly”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Job "disk-health-check" asks the agent for drive health; owner: controller reads, agent measures. CUSTOMER. | A | CT/cmd/controller/main.go:2226 |
| 2 | Every 1 h; reason for hourly, not 6 h:. Only when the agent channel is configured. | A | CT/cmd/controller/main.go:2219-2224 CT/cmd/controller/main.go:2225 |
| 3 | One fetch was measured at about 0.8 s (a comment in the source). No time limit in the source. | N | CT/cmd/controller/main.go:2219-2224 · class W4 · S15 |
| 4 | Next hour; first run one hour after start. | A | CT/internal/scheduler/scheduler.go:281-298 |
| 5 | Skipped if still running. | A | CT/internal/scheduler/scheduler.go:354-364 |
| 6 | Monotonic ticker. | A | CT/cmd/controller/main.go:2226 (the code path of question 1) |
| 7 | The household and the hub: the check emits disk_health_degraded. |
A | CT/internal/web/disk_health.go:175-280 |
| 8 | SOURCE. | A | CT/cmd/controller/main.go:2219-2226 |
Hand step in a runbook? no hand step
CC17 — Agent channel health, every 60 seconds · CUSTOMER · Interval · MENTIONED
architecture folder: 2 lines in 2 files match /agent channel|channel health/ (a machine count, not a reading) · helper section: clock-controller „17. Agent channel health, every 60 seconds”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Job "agent-channel-health" probes the controller-to-agent channel; owner: controller. OPERATOR. | A | CT/cmd/controller/main.go:2289 |
| 2 | Every 60 s. A passing fault needs 2 misses in a row, debounceThreshold. |
A | CT/internal/channelhealth/checker.go:29 |
| 3 | Probe limit 15 s, probeTimeout. |
A | CT/internal/channelhealth/checker.go:32 |
| 4 | Next minute. | A | CT/cmd/controller/main.go:2289 (the code path of question 1) |
| 5 | Skipped if still running. An alert marker file survives a restart. | A | CT/internal/scheduler/scheduler.go:354-364 CT/cmd/controller/main.go:2288 |
| 6 | Monotonic ticker. | A | CT/cmd/controller/main.go:2289 (the code path of question 1) |
| 7 | Sink = notifier and page alert. | A | CT/cmd/controller/main.go:2286 |
| 8 | TEST TestDebounce_TransientBlipSuppressed. |
A | CT/internal/channelhealth/checker_test.go:119 |
Hand step in a runbook? no hand step
CC18 — Held-app report to the hub · CUSTOMER · Interval · MENTIONED
architecture folder: 32 lines in 7 files match /held.app|held app/ (a machine count, not a reading) · helper section: clock-controller „18. Held-app report to the hub”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Reporter.Tick tells the hub which apps are held; owner: controller; job "hold-report". OPERATOR. |
A | CT/internal/holdreport/holdreport.go:89 CT/cmd/controller/main.go:1081-1084 |
| 2 | Looks every minute; each hold is sent again once a day, ResendEvery, and after each start. |
A | CT/internal/holdreport/holdreport.go:36 CT/cmd/controller/main.go:1071-1073 |
| 3 | Instant; reads the hold store. | A | CT/internal/holdreport/holdreport.go:89 CT/cmd/controller/main.go:1081-1084 (the code path of question 1) |
| 4 | Next minute. | A | CT/internal/holdreport/holdreport.go:89 CT/cmd/controller/main.go:1081-1084 (the code path of question 1) |
| 5 | Skipped if still running. | A | CT/internal/scheduler/scheduler.go:354-364 |
| 6 | The daily resend is wall-clock arithmetic. A clock that is set back delays the resend by the size of the step. | N | CT/cmd/controller/main.go:1082 · class W2 · S08 |
| 7 | The hub, by event. | A | CT/cmd/controller/main.go:1074 |
| 8 | TEST TestD64_TheHoldReporterTellsTheHubAfterTheClear. |
A | CT/cmd/controller/holdclear_test.go:333 |
Hand step in a runbook? no hand step
CC19 — Abandoned off-site store sweep · CUSTOMER · Night · MENTIONED
architecture folder: 21 lines in 9 files match /abandon/ (a machine count, not a reading) · helper section: clock-controller „19. Abandoned off-site store sweep at 05:10”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Daily job "offsite-abandon-sweep" runs AbandonSweep; owner: controller. CUSTOMER. |
A | CT/cmd/controller/main.go:1693 CT/internal/backup/offbox_abandon.go:214 |
| 2 | Daily at 05:10, fixed. The due date is counted in UTC days, the slot in Europe/Budapest. | A | CT/internal/backup/offbox_abandon.go:366 CT/internal/scheduler/scheduler.go:17-27 |
| 3 | One sweep has a 1-minute limit. | A | CT/internal/backup/offbox_abandon.go:180 |
| 4 | Box off at 05:10: next day; no catch-up (it is not a ledger leg, see). | A | CT/cmd/controller/main.go:1660-1664 |
| 5 | The sweep takes no lock. MEASURED: on the three boxes it ran 55 minutes after the off-site push ended, so they did not meet on these small stores. | N | CT/internal/backup/offbox_abandon.go:180 EV/box/demo-hp-guest-night.txt · class W3 · S11 |
| 6 | As block 24. | A | CT/cmd/controller/main.go:1693 CT/internal/backup/offbox_abandon.go:214 (the code path of question 1) |
| 7 | Nobody. A sweep that fails writes a WARN or ERROR line on the box only. | N | CT/internal/backup/offbox_abandon.go:180 · class W1 · S06 |
| 8 | MEASURED 2026-11-11: the slot ran on each box · SOURCE. | A | EV/box/demo-hp-guest-night.txt EV/box/tester-1-guest-night.txt EV/box/demo-felhom-guest-night.txt CT/cmd/controller/main.go:1693 |
Hand step in a runbook? no hand step
CC20 — Restore-copy sweep and conversion-copy release · CUSTOMER · Interval · MENTIONED
architecture folder: 14 lines in 3 files match /restore.copy|restore copy/ (a machine count, not a reading) · helper section: clock-controller „20. Restore-copy sweep and conversion-copy release”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | SweepUnusedRestoreCopies deletes old kept copies; job "restore-copy-sweep"; job "conversion-copy-release". CUSTOMER. |
A | CT/internal/backup/restore_copy_sweep.go:36 CT/cmd/controller/main.go:1414 CT/cmd/controller/main.go:1407 |
| 2 | The sweep runs each 6 hours; the release each hour. A restore copy that nobody used goes after 30 days (the constant, D62). A comment in the wiring still says 14 days; the comment is stale. | A | CT/cmd/controller/main.go:1412 |
| 3 | No time limit in the source, and no duration was measured. Nobody knows how long it takes on a big box. | N | class W4 · S15 |
| 4 | The first run is 6 hours after the controller starts. A box that is on for under 6 hours a day never runs it; old restore copies stay. Nobody is told. | N | CT/internal/scheduler/scheduler.go:281-298 · class W1 · S25 |
| 5 | Skipped if still running. | A | CT/internal/scheduler/scheduler.go:354-364 |
| 6 | Age is wall-clock arithmetic on now. |
A | CT/internal/backup/restore_copy_sweep.go:36 |
| 7 | The household is told when a restore copy is deleted. | A | CT/internal/notify/notifier.go:1601 |
| 8 | SOURCE. | A | CT/internal/backup/restore_copy_sweep.go:29 |
Hand step in a runbook? no hand step
CC21 — Upload and CPU watch, torrent limit rule, line-test hold · CUSTOMER · Interval · MENTIONED
architecture folder: 70 lines in 16 files match /upwatch|unexplained|upload/ (a machine count, not a reading) · helper section: clock-controller „22. Upload and CPU watch, torrent limit rule, line-test hold”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | startUpwatch runs the minute watcher and a 15-minute rule that holds the torrent app for the line test; owner: controller. CUSTOMER. |
A | CT/cmd/controller/upwatch_wiring.go:136 |
| 2 | Watcher every 1 min; rule every 15 min. Baseline: 7 days of data, cached 24 h. | A | CT/cmd/controller/upwatch_wiring.go:139 CT/cmd/controller/upwatch_wiring.go:141 CT/cmd/controller/upwatch_wiring.go:124-127 |
| 3 | Hold call limit 30 s, holdTimeout; drain wait at most 90 s. |
A | CT/internal/upwatch/linehold.go:79 CT/internal/upwatch/linehold.go:216 |
| 4 | A limit left at the test value by a stopped controller is put back at start, RecoverHold. |
A | CT/cmd/controller/upwatch_wiring.go:137 |
| 5 | Two loops with their own tickers. The file holds no lock between them. | N | class W3 · S14 |
| 6 | Monotonic tickers. | A | CT/internal/upwatch/watch.go:151 |
| 7 | The household and the hub: the watch emits app_unexplained_upload and app_unexplained_cpu. |
A | CT/internal/upwatch/watch.go:309-382 |
| 8 | SOURCE. | A | CT/cmd/controller/upwatch_wiring.go:134-152 |
Hand step in a runbook? no hand step
CC22 — The scheduler itself: daily slots and interval jobs · CUSTOMER · Night · MENTIONED
architecture folder: 24 lines in 10 files match /scheduler|daily slot/ (a machine count, not a reading) · helper section: clock-controller „24. The clock itself: daily slots and interval jobs (merged: scheduler rules for all jobs above)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Daily and Every carry every named job; owner: controller. CUSTOMER. |
A | CT/internal/scheduler/scheduler.go:131-157 CT/internal/scheduler/scheduler.go:105-127 |
| 2 | Daily slots are HH:MM in Europe/Budapest, hard-coded; if the zone data cannot load it silently uses UTC. Interval jobs do not run at start: first run after one interval. | A | CT/internal/scheduler/scheduler.go:17-27 CT/internal/scheduler/scheduler.go:281-298 |
| 3 | No time limit of its own; shutdown waits 30 s. A panic in a job is caught. | A | CT/internal/scheduler/scheduler.go:264 CT/internal/scheduler/scheduler.go:438-447 |
| 4 | A daily job has no catch-up in the scheduler; a fire over 60 min late is skipped. Only the three backup legs are made up (block 4). | A | CT/internal/scheduler/scheduler.go:344-347 |
| 5 | One run per job at a time: "still running" skip. No rule between different jobs here. | A | CT/internal/scheduler/scheduler.go:354-364 |
| 6 | Slots are in Budapest time. On the spring night 02:30 does not exist; on the autumn night it comes twice. No test covers either night. A clock set back while a timer waits does not matter: the timer runs on monotonic time. | K | CT/internal/scheduler/scheduler.go:499-519 · R-1260 |
| 7 | A skipped late fire is a log line only. | A | CT/internal/scheduler/scheduler.go:344-347 |
| 8 | SOURCE. MEASURED 2026-11-11: the night's order and the start times on three boxes (see the README's night table). | A | CT/internal/scheduler/scheduler.go:344-347 EV/box/demo-hp-guest-night.txt |
Hand step in a runbook? no hand step
CH01 — Box liveness: 'your server cannot be reached' · CUSTOMER · Hub sweep · MENTIONED
architecture folder: 50 lines in 16 files match /host_stale|liveness|staleness/ (a machine count, not a reading) · helper section: clock-hub „1. Box liveness: "your server cannot be reached"”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER. The hub's 60 s sweep runs stalenessChecker.Check and hostStaleness.Check; they raise host_stale, host_down, host_recovered. Owner: hub pod. Merged here: the controller-report checker and the host-report checker. |
A | E/hub/cmd/hub/main.go:935-936 HUB/monitor/host_staleness.go:207-221 |
| 2 | Every 60 s. Stale after stale_threshold, default "30m", deployed 45 m; down after 2x. No time zone: ages are durations. |
A | E/hub/cmd/hub/main.go:927 E/hub/cmd/hub/main.go:1027 E/manifests/hub.yaml:88 HUB/monitor/host_staleness.go:39 |
| 3 | No time limit. The checkers run one after the other in one loop: one read of all hosts at each pass, then a loop over the hosts. The time grows with the number of boxes. Nobody measured one pass. | N | E/hub/cmd/hub/main.go:935-962 HUB/monitor/host_staleness.go:113 · class W4 · S16 |
| 4 | With the hub down nothing is judged. At the start, a box that is already silent gets one alarm at the first pass, unless that alarm was raised before. The controller's checker starts silent for such a box. | A | HUB/monitor/host_staleness.go:59-68 HUB/monitor/staleness.go:86-91 |
| 5 | One pod, replicas: 1, Recreate; state under a mutex. Events fire on transitions only. A household power-off holds alarms for DeliberateOffHold. |
A | E/manifests/hub.yaml:122 E/manifests/hub.yaml:123-124 HUB/monitor/host_staleness.go:119-120 HUB/monitor/host_staleness.go:170-172 |
| 6 | Judged on the hub's receive time, not the box clock: last_report_at = datetime('now'). A wrong box clock cannot hide silence. A hub clock jump moves every age at once; DST has no effect (durations). |
A | HUB/store/store.go:3916 HUB/monitor/host_staleness.go:131 |
| 7 | Operator: every edge; liveness events bypass the 1 h quiet hour, 5 min dedupe. Household: at most once per 7 days, householdLivenessQuiet. If the hub itself is down nobody is told by this job. |
A | HUB/notify/dispatcher.go:609-612 HUB/notify/dispatcher.go:1018 |
| 8 | TEST TestHostStaleness_R1001_RestartDoesNotReMail; TEST TestR873_HouseholdHearsAnOutageAtMostWeekly; DOC. |
A | HUB/monitor/r1001_restart_test.go:56 HUB/notify/r873_liveness_weekly_test.go:17 AR/05-hub-architecture.md:88-118 |
Hand step in a runbook? no hand step
CH02 — Missed-backup deadline check (05:00) · CUSTOMER · Hub daily · MENTIONED
architecture folder: 28 lines in 11 files match /deadline check|backup.deadline|05:00/ (a machine count, not a reading) · helper section: clock-hub „2. Missed-backup deadline check (05:00)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER. scheduleDaily(ctx, "deadline-check", "05:00",...) runs CheckBackupDeadlines; it raises expected_backup_missed. Owner: hub pod. |
A | E/hub/cmd/hub/main.go:986-988 HUB/monitor/deadline.go:409 HUB/monitor/deadline.go:539-544 |
| 2 | Once a day at 05:00 Budapest wall-clock; the slot is a literal in, no setting. Zone chosen in scheduleDaily. Limits: backupStaleAfter 26 h, off-site offsiteBackupStaleAfter 8 d. |
A | E/hub/cmd/hub/main.go:986 E/hub/cmd/hub/main.go:1072 HUB/monitor/deadline.go:29 HUB/monitor/deadline_tiers.go:28 |
| 3 | No time limit; the check runs inside the scheduler's own loop. It reads up to 7 days of evidence for every customer. The time grows with boxes times reports. Nobody measured it. | N | E/hub/cmd/hub/main.go:1090 · class W4 · S16 |
| 4 | When the hub is down at 05:00, that day's check is skipped. The 'last run' value lives in memory and the next slot is the next day. Nobody notices. The next day's check uses longer limits, so a missed backup is seen one day late. | N | E/hub/cmd/hub/main.go:1069-1112 · class W1 · S01 |
| 5 | Cannot run twice for one slot: the last guard in nextDailyRun. No lock against the 60 s sweep; they are separate goroutines and. A down box is judged on longer lines, 48 h / 72 h. |
A | E/hub/cmd/hub/main.go:1095-1112 E/hub/cmd/hub/main.go:927 E/hub/cmd/hub/main.go:986 HUB/monitor/deadline.go:340-343 |
| 6 | The check compares the BOX's backup time with the hub's clock. A box whose clock runs ahead looks fresh. No guard in the file. (R-1206 is the same cause seen on the box's own pages.) | N | HUB/monitor/deadline.go:243 · class W2 · S07 |
| 7 | Household and operator through the dispatcher on expected_backup_missed; at the 05:00 run only. A box that never bound is "unknown", not alarmed; disabled / deliberately-off are skipped. |
A | HUB/monitor/deadline.go:539-544 HUB/monitor/deadline.go:486-494 HUB/monitor/deadline.go:432-442 |
| 8 | TEST for a down box and for 'the same slot never runs twice'. A skipped day: NOBODY KNOWS. | N | HUB/monitor/r872_down_box_test.go:57 E/hub/cmd/hub/r1183_dst_schedule_test.go:82 · class W1 · S01 |
Hand step in a runbook? NO — no runbook names a hub that was down at 05:00
CH03 — Mail cooldowns and the weekly outage mail · CUSTOMER · Hub sweep · MENTIONED
architecture folder: 38 lines in 8 files match /cooldown/ (a machine count, not a reading) · helper section: clock-hub „3. Mail cooldowns and the weekly outage mail”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER. Not a job but a clock rule inside every mail: processCustomer and processOperator decide whether an alarm is mailed now or swallowed. Owner: hub dispatcher. |
A | HUB/notify/dispatcher.go:1020-1098 HUB/notify/dispatcher.go:626-696 |
| 2 | Operator: operatorCooldown 1 h, liveness 5 min, storage fill storageFillCooldown 6 h. Household: default 6 h; outage mails once per 7 days. First hour of a new box is suppressed, firstHourOfABox. |
A | HUB/notify/dispatcher.go:609-612 HUB/notify/dispatcher.go:573 HUB/notify/dispatcher.go:1065-1069 HUB/notify/dispatcher.go:1054-1062 |
| 3 | Each mail call has a 10-second limit. The cooldown lists are in memory; an entry is removed when its alarm clears. | A | HUB/notify/dispatcher.go:86 HUB/notify/dispatcher.go:684 HUB/notify/dispatcher.go:1182 |
| 4 | Pod restart: the cooldown maps are in memory, so quiet periods restart from zero; only the 7-day household rule is read from the persisted notification log. Design text:. | A | HUB/notify/dispatcher.go:1054-1062 AR/08-alarm-ladder.md:435-439 |
| 5 | The operator cooldown is armed before the send, so two alarms close together collapse to one; a suppression is recorded. A MAIL-HOLD drops the mail. | A | HUB/notify/dispatcher.go:674 HUB/notify/dispatcher.go:665-672 HUB/notify/dispatcher.go:680-687 |
| 6 | Cooldowns use the hub's wall clock only. A jump back stretches a quiet period; a jump forward ends it early. Nothing handles it. | N | HUB/notify/dispatcher.go:1 · class W2 · S10 |
| 7 | A swallowed operator alarm leaves a 'suppressed' row. A household mail that fails is logged and is not sent again; only operator mails are retried. | N | HUB/notify/dispatcher.go:1091-1094 · class W1 · S02 |
| 8 | TEST TestOperatorCooldown_NodeLivenessBypassesQuietHour; TEST TestR873_HouseholdHearsAnOutageAtMostWeekly; DOC. |
A | HUB/notify/node_liveness_cooldown_test.go:17 HUB/notify/r873_liveness_weekly_test.go:17 AR/08-alarm-ladder.md:165-168 |
Hand step in a runbook? no hand step
CH04 — Retry of a failed operator mail · OPERATOR · Hub sweep · MENTIONED
architecture folder: 27 lines in 10 files match /retry/ (a machine count, not a reading) · helper section: clock-hub „4. Retry of a failed operator mail”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR. retryOperatorEmail re-sends an operator mail whose send failed; started by the failure branch of processOperator. |
A | HUB/notify/dispatcher.go:99-104 HUB/notify/dispatcher.go:688-692 |
| 2 | After 1, 5 and 15 minutes: operatorRetryDelays, armed with afterFn = time.AfterFunc. No setting. Durations, no time zone. |
A | HUB/notify/dispatcher.go:76 HUB/notify/dispatcher.go:92 |
| 3 | Each try is bounded by the 10 s client timeout. One timer per failed mail; grows only with failures. | A | HUB/notify/dispatcher.go:86 |
| 4 | The retry timers are in memory. A hub restart inside the 21 minutes loses them, and nothing sends the mail later. After the third failure the hub writes one ERROR line. | N | HUB/notify/dispatcher.go:90-102 · class W1 · S03 |
| 5 | The cooldown for that alarm was armed before the first send, so a repeat alarm in the same hour does not mail either; the retry is the only path. Explained in. | A | HUB/notify/dispatcher.go:674 AR/08-alarm-ladder.md:358-363 |
| 6 | The timers are durations; the hour change has no effect. A jump of the hub's wall clock: NOBODY KNOWS. | N | class W2 · S10 |
| 7 | Nobody is mailed when the retries give up. The trace is the ERROR line and the 'failed' rows in the notification log. | N | HUB/notify/dispatcher.go:90-102 AR/08-alarm-ladder.md:363 · class W1 · S03 |
| 8 | TEST for the retry. A restart during the retry window: NOBODY KNOWS. | N | HUB/notify/operator_mail_retry_test.go:17 · class W1 · S03 |
Hand step in a runbook? no hand step
CH05 — Kernel mail: 'your box restarts tonight' · CUSTOMER · Hub sweep · MENTIONED
architecture folder: 165 lines in 14 files match /kernel/ (a machine count, not a reading) · helper section: clock-hub „5. Kernel "your box restarts tonight" mail”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER. KernelNotify mails each due household the day before a kernel step; called from Evaluate, which the 60 s OS ticker calls. |
A | HUB/osupdates/kernel.go:191 HUB/osupdates/service.go:436 E/hub/cmd/hub/main.go:564 |
| 2 | Every 60 s, but acts only 09:00-20:00 Budapest: KernelNoticeFromHour/KernelNoticeToHour, checked at. Valid 24 h, gap 20 h, at most 3 mails. Zone from budapestLoc. Constants, no setting. |
A | HUB/osupdates/kernel.go:46-47 HUB/osupdates/kernel.go:193-196 HUB/osupdates/kernel.go:48-50 HUB/osupdates/kernel.go:73-78 |
| 3 | No time limit. Loops all hosts every minute with store reads per host; grows linearly with boxes, 1440 passes a day. | A | HUB/osupdates/kernel.go:203-208 |
| 4 | Pod down all day 09-20: no mail, and without a mail there is no step: KernelBlockFor sets Tonight only for a notice younger than 24 h. It mails on the next daytime tick; the notices are persisted. |
A | HUB/osupdates/kernel.go:179-185 HUB/osupdates/kernel.go:236 |
| 5 | Twice is blocked by the stored notice and the 20 h gap; a restart cannot double-mail because the row is in the store. A mail sent but not recorded means no step tonight. | A | HUB/osupdates/kernel.go:208-211 HUB/osupdates/kernel.go:236-239 |
| 6 | Hub clock decides the daytime window. A mail sent at 19:59 is valid until 19:59 next day, so it covers one night. DST: the hour test uses Budapest local time; the 20 h / 24 h are durations. A deliberately-off box gets no mail. | A | HUB/osupdates/kernel.go:182 HUB/osupdates/kernel.go:193-194 HUB/osupdates/kernel.go:201 |
| 7 | Mail failure: operator event os_kernel_notice warning, once per box, kernel and day. After 3 mails with no step: operator warning, once. |
A | HUB/osupdates/kernel.go:226-233 HUB/osupdates/kernel.go:212-219 |
| 8 | TESTS for 'no mail, no step'. A mail that goes out late in the day against the box's own night slot: NOBODY KNOWS. | N | HUB/osupdates/kernel_test.go:192 HUB/osupdates/kernel_test.go:79 HUB/osupdates/kernel_test.go:115 · class W3 · S14 |
Hand step in a runbook? no hand step
CH06 — OS-update approval tick and the hourly OS alarms · OPERATOR · Hub sweep · MENTIONED
architecture folder: 468 lines in 28 files match /approval|ring/ (a machine count, not a reading) · helper section: clock-hub „6. OS update approval tick and hourly OS alarms”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR (the outcome reaches households as updates). 60 s ticker: osSvc.Evaluate approves a set when its rule holds; osSvc.Alarms raises the fleet alarms. |
A | E/hub/cmd/hub/main.go:554-586 E/hub/cmd/hub/main.go:564 E/hub/cmd/hub/main.go:577 |
| 2 | Evaluate every 60 s; Alarms when time.Since(lastAlarms) >= time.Hour. Rule default ApproveAfter: 24 * time.Hour, NightsRequired: 1, env OS_APPROVE_AFTER. Alarm limits 7 d / 14 d, env table. |
A | E/hub/cmd/hub/main.go:574 E/hub/cmd/hub/main.go:476 E/hub/cmd/hub/main.go:484-491 E/hub/cmd/hub/main.go:536-541 |
| 3 | No time limit. The alarm pass logs its own duration. Evaluate walks every layer; grows with boxes times layers, every minute. | A | E/hub/cmd/hub/main.go:581 HUB/osupdates/service.go:414-422 |
| 4 | Pod restart: lastAlarms is a local variable, zero at start, so the first tick runs the alarms at once. Re-mail is prevented by the stored raise time, 7 days. |
A | E/hub/cmd/hub/main.go:558 HUB/osupdates/service.go:1102 |
| 5 | One goroutine, so Evaluate and Alarms never overlap. raise clears the mark when the condition ends so the next occurrence is announced. |
A | E/hub/cmd/hub/main.go:559-585 HUB/osupdates/service.go:1094-1108 |
| 6 | A box's claimed reboot time is allowed 1 minute of error. A larger error of the box's clock: NOBODY KNOWS. (The N100's host is 32 s behind today.) | N | HUB/osupdates/service.go:1102 HUB/osupdates/service.go:970 · class W2 · S07 |
| 7 | Operator, by event through the dispatcher: os_update_stale, os_ring0_stalled and others, listed in. Repeats weekly while it holds. |
A | AR/08-alarm-ladder.md:375-383 HUB/osupdates/service.go:1092-1093 |
| 8 | TESTS and DOC. The cost of the pass each minute with 100 boxes: NOBODY KNOWS. | N | HUB/osupdates/alarms_test.go:226 AR/08-alarm-ladder.md:365 · class W4 · S16 |
Hand step in a runbook? no hand step
CH07 — Off-site copy too old, and escrow never finished · CUSTOMER · Hub sweep · MENTIONED
architecture folder: 12 lines in 6 files match /offsite_stale|off.site.{0,20}stale|48 h/ (a machine count, not a reading) · helper section: clock-hub „7. Off-site copy too old, and escrow never finished”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER. offsiteChecker.Check on the 60 s sweep; isStale and checkEscrowPending. Owner: hub. |
A | E/hub/cmd/hub/main.go:943 HUB/monitor/offsite.go:166 HUB/monitor/offsite_escrow_pending.go:66 |
| 2 | Every 60 s. Stale after defaultOffsiteStaleAfter 48 h, counted from the last SUCCESS. Escrow pending after escrowPendingAfter 7 d, re-raised each 7 d. |
A | HUB/monitor/offsite.go:54 HUB/monitor/offsite.go:203-207 HUB/monitor/offsite_escrow_pending.go:37 HUB/monitor/offsite_escrow_pending.go:110 |
| 3 | No time limit. Reads every customer's latest report each minute; grows linearly with boxes. | A | HUB/monitor/offsite.go:443-459 |
| 4 | With the hub down nothing is judged. After a start the state lists in memory are empty. If a tier that is still too old is mailed again after a restart: NOBODY KNOWS. | N | HUB/monitor/offsite.go:463-468 HUB/monitor/offsite.go:1 · class W1 · S03 |
| 5 | One mutex per pass; a customer listed twice is handled once. A never-ran tier is owned by the delivery checker, "one state, one owner". | A | HUB/monitor/offsite.go:448-449 HUB/monitor/offsite.go:453-458 HUB/monitor/offsite.go:160-162 |
| 6 | The last-success time is the BOX's time, compared with the hub's clock. A box clock that is ahead hides an old copy. A time that cannot be read counts as too old. | N | HUB/monitor/offsite.go:203-207 · class W2 · S07 |
| 7 | Dispatcher mail on the stale edge; offsite_escrow_pending is operator-only, see. |
A | HUB/monitor/offsite.go:636 AR/08-alarm-ladder.md:383 |
| 8 | TESTS for the limits and for a newborn box. A box clock that is ahead: NOBODY KNOWS. | N | HUB/monitor/offsite_r100_test.go:55 HUB/monitor/offsite_r100_test.go:74 · class W2 · S07 |
Hand step in a runbook? no hand step
CH08 — Restore-test proof too old · OPERATOR · Hub sweep · MENTIONED
architecture folder: 121 lines in 19 files match /restore.test/ (a machine count, not a reading) · helper section: clock-hub „8. Restore-test proof too old”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER (protects the promise that a backup restores; mail goes by event). restoreTestChecker.Check on the 60 s sweep; checkStaleness. |
A | E/hub/cmd/hub/main.go:944 HUB/monitor/restoretest.go:242 |
| 2 | Every 60 s. Window per tier: 4 archive generations, never under restoreProvenWindowFloor 7 d, never over restoreProvenWindowCap 12 d. Constants, no setting. |
A | HUB/monitor/restoretest.go:62 HUB/monitor/restoretest.go:68 HUB/monitor/restoretest.go:74 |
| 3 | No time limit. Each pass reads 14 days of host reports for each customer, each minute. It is the heaviest read on the sweep. Nobody measured it. | N | HUB/monitor/restoretest.go:79 HUB/monitor/restoretest.go:252 · class W4 · S16 |
| 4 | Pod restart: the first pass does not re-send a stale signal already in the events table, staleAlreadyReported. |
A | HUB/monitor/restoretest.go:281-283 HUB/monitor/restoretest.go:300-312 |
| 5 | Edge-triggered per customer and tier under a mutex. A deliberately-off box is skipped. | A | HUB/monitor/restoretest.go:270-278 HUB/monitor/restoretest.go:192-194 |
| 6 | First contact is the hub's receive time. The proof times come from the box's reports. An error of the box's clock: NOBODY KNOWS. | N | HUB/monitor/restoretest.go:326-342 HUB/store/store.go:3994-4005 · class W2 · S07 |
| 7 | Event restore_test_stale warning through the dispatcher; a failed run is a separate event, checkFailure. |
A | HUB/monitor/restoretest.go:284-285 HUB/monitor/restoretest.go:205 |
| 8 | TEST that a restart does not mail again. The cost with 100 boxes (14 days of reports per customer, each minute): NOBODY KNOWS. | N | HUB/monitor/r1001_restart_test.go:147 · class W4 · S16 |
Hand step in a runbook? no hand step
CH09 — Off-site clean-up window (20 minutes, weekly) · CUSTOMER · Hub sweep · MENTIONED
architecture folder: 11 lines in 5 files match /clean.?up window|prune window|deleting key/ (a machine count, not a reading) · helper section: clock-hub „9. Off-site clean-up window (20 minutes, weekly)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER DATA. The box asks; OpenWindowFor adds a deleting key line for 20 minutes; the hub's 60 s sweep SweepExpiredWindows closes a window left open. Sweep loop:. |
A | HUB/offsitekeys/service.go:334 HUB/offsitekeys/service.go:413 E/hub/cmd/hub/main.go:636-650 |
| 2 | Granted when the last window is at least windowCadence 6 d 12 h old, test at; length windowLength 20 min. Sweep every 60 s. Constants; the weekly switch is a store setting. |
A | HUB/offsitekeys/service.go:264 HUB/offsitekeys/service.go:337 HUB/offsitekeys/service.go:263 HUB/offsitekeys/service.go:338 |
| 3 | Each sweep has a 2-minute context. Work is per expired window, one remote call each; grows with boxes that crash mid-prune. | A | E/hub/cmd/hub/main.go:644 HUB/offsitekeys/service.go:419-422 |
| 4 | If the hub stops while a window is open, the key line that can delete stays on the storage account until the hub is back; the sweep then closes it. Nobody is told during the gap. | N | HUB/offsitekeys/service.go:419-431 HUB/offsitekeys/service.go:424-426 · class W1 · S01 |
| 5 | One window per cadence via the ledger row. The daily key audit knows an open window. No lock against the abandon sweep; both run in the same goroutine. | A | HUB/offsitekeys/service.go:364 HUB/offsitekeys/service.go:212-213 E/hub/cmd/hub/main.go:636-650 |
| 6 | The window runs on the hub's clock; the box's clock does not decide. A hub clock that jumps back can keep a window open longer: NOBODY KNOWS. | N | HUB/offsitekeys/service.go:337 · class W2 · S10 |
| 7 | Operator event offsite_window_failed warning when the hub had to close it; offsite_window_drop error when too much went. |
A | HUB/offsitekeys/service.go:430-431 HUB/offsitekeys/service.go:259 |
| 8 | TEST TestWindow_LeftOpenIsClosedByTheSweep; TEST TestWindow_GrantOpenCloseAndDropAlarm; DOC. |
A | HUB/offsitekeys/service_test.go:99 HUB/offsitekeys/service_test.go:37 AR/07-backup-architecture.md:256 |
Hand step in a runbook? no hand step
CH10 — Delayed deletion of a set-aside off-site copy · CUSTOMER · Hub sweep · MENTIONED
architecture folder: 51 lines in 16 files match /set.aside|delayed delet|superseded/ (a machine count, not a reading) · helper section: clock-hub „10. Delayed deletion of a set-aside off-site copy”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER DATA. SweepAbandons deletes a household's set-aside copy once its request is due; called from the 60 s sweep. |
A | HUB/offsitekeys/service.go:529 E/hub/cmd/hub/main.go:636-650 |
| 2 | Due DefaultAbandonDelay 7 days after the request; override OFFSITE_ABANDON_DELAY. Checked every 60 s. Hub clock. |
A | HUB/offsitekeys/service.go:448 E/hub/cmd/hub/main.go:598-605 |
| 3 | One remote delete for each due request, in the same sweep as the window. How long a large delete takes: NOBODY KNOWS. | N | E/hub/cmd/hub/main.go:644 HUB/offsitekeys/service.go:536-538 · class W4 · S15 |
| 4 | Pod down at the due time: deleted on the first sweep after start; the due list is in the store. A failed delete is marked and retried every sweep. | A | HUB/offsitekeys/service.go:530 HUB/offsitekeys/service.go:540-547 |
| 5 | A request is deleted once, then marked. A cancel before the due time wins: design text. | A | HUB/offsitekeys/service.go:549 HUB/offsitekeys/service.go:438-441 |
| 6 | The hub's clock only. The length of the wait is one setting, read at the start. | A | E/hub/cmd/hub/main.go:598-605 |
| 7 | Operator events: requested, cancelled, deleted, failed; failure mails once per request. | A | HUB/offsitekeys/service.go:444-447 HUB/offsitekeys/service.go:543-546 |
| 8 | TEST TestAbandon_DelayCancelAndScope. |
A | HUB/offsitekeys/service_test.go:134 |
Hand step in a runbook? no hand step
CH11 — Off-site key audit (07:10) · OPERATOR · Hub daily · MENTIONED
architecture folder: 2 lines in 2 files match /key audit|07:10/ (a machine count, not a reading) · helper section: clock-hub „11. Off-site key audit (07:10)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR (guards household copies against deletion). scheduleDaily(ctx, "offsite-key-audit", "07:10",...) runs AuditAll. |
A | E/hub/cmd/hub/main.go:651-655 HUB/offsitekeys/service.go:195 |
| 2 | Daily 07:10 Budapest, literal slot. Zone from scheduleDaily. |
A | E/hub/cmd/hub/main.go:651 E/hub/cmd/hub/main.go:1072 |
| 3 | The audit has a 10-minute limit and reads one customer after the other from the storage account. With 100 boxes the 10 minutes may not be enough; nobody measured it. | N | E/hub/cmd/hub/main.go:652 HUB/offsitekeys/service.go:202-215 · class W4 · S16 |
| 4 | When the hub is down at 07:10, that day's audit is skipped. No catch-up. Nobody notices. | N | E/hub/cmd/hub/main.go:1069-1112 · class W1 · S01 |
| 5 | Runs once per day by the last guard. Meets an open clean-up window: the audit is told the window is open. |
A | E/hub/cmd/hub/main.go:1095-1112 HUB/offsitekeys/service.go:212-213 |
| 6 | Wall-clock slot, DST-safe by nextDailyRun. |
A | E/hub/cmd/hub/main.go:1095-1112 |
| 7 | Operator: offsite_key_unlocked error, or offsite_key_audit_failed when a file cannot be read. |
A | HUB/offsitekeys/service.go:192-194 |
| 8 | TEST and DOC. The 10-minute limit at fleet size: NOBODY KNOWS. | N | HUB/offsitekeys/offsitekeys_test.go:158 AR/05-hub-architecture.md:642 · class W4 · S16 |
Hand step in a runbook? no hand step
CH12 — Off-site credential stuck, and its self-heal · OPERATOR · Hub sweep · MENTIONED
architecture folder: 4 lines in 3 files match /offsiteheal|credential.{0,30}(stuck|heal)/ (a machine count, not a reading) · helper section: clock-hub „12. Off-site credential stuck and its self-heal”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER. Two jobs, merged: offsiteDeliveryChecker.Check (60 s sweep) and the offsiteheal reconciler, started at. |
A | E/hub/cmd/hub/main.go:945 HUB/offsiteheal/reconciler.go:165-177 E/hub/cmd/hub/main.go:846-854 |
| 2 | Stuck after stuckAfter 1 h; cooldowns 24 h. Heal sweep every tickIntervalDefault 5 min, acts after 2 distinct reports. |
A | HUB/monitor/offsite_delivery.go:38 HUB/monitor/offsite_delivery.go:40-41 HUB/offsiteheal/reconciler.go:74 HUB/offsiteheal/reconciler.go:70 |
| 3 | Heal call limit 3 minutes. Both list every customer config per pass and; linear in boxes. | A | HUB/monitor/offsite_delivery.go:275 HUB/monitor/offsite_delivery.go:72 HUB/offsiteheal/reconciler.go:183 |
| 4 | Pod down: nothing; on return the next tick re-reads state. The cooldowns are "durable via the events table". | A | HUB/monitor/offsite_delivery.go:39 |
| 5 | Two healers could meet: the checker re-reads the secret row immediately before acting, and a box declaration is owned by offsiteheal. | A | HUB/monitor/offsite_delivery.go:28-31 HUB/monitor/offsite_delivery.go:97-99 |
| 6 | Age counted from the hub-held status.Since. Debounce is counted in reports, not ticks. |
A | HUB/monitor/offsite_delivery.go:93 HUB/offsiteheal/reconciler.go:61-69 |
| 7 | Operator events offsite_delivery_stuck, offsite_credential_restaged; offsite_selfheal_restaged / _reissued. |
A | HUB/monitor/offsite_delivery.go:34-35 HUB/offsiteheal/reconciler.go:57-58 |
| 8 | SOURCE and DOC only. No test name for the timing was found. | A | HUB/offsiteheal/reconciler.go:61-74 AR/06-offsite-connectivity.md:386 HUB/monitor/offsite_delivery.go:1 |
Hand step in a runbook? no hand step
CH13 — PBS whole-box credential self-heal · OPERATOR · Hub sweep · MENTIONED
architecture folder: 9 lines in 5 files match /pbsdrheal|PBS.{0,30}heal/ (a machine count, not a reading) · helper section: clock-hub „13. PBS whole-box credential self-heal”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER. pbsdrheal reconciler, started at; repairs a box whose off-site PBS credential did not land. |
A | HUB/pbsdrheal/reconciler.go:153-165 E/hub/cmd/hub/main.go:829-837 |
| 2 | Every 5 minutes, or on a trigger; acts after 2 reports. A staged secret unconsumed for unconsumedGrace 15 min is flagged. |
A | HUB/pbsdrheal/reconciler.go:130 HUB/pbsdrheal/reconciler.go:160 HUB/pbsdrheal/reconciler.go:131 HUB/pbsdrheal/reconciler.go:63 |
| 3 | No time limit on a pass, reconcileOnce(ctx). Reads all heal states; linear in boxes. |
A | HUB/pbsdrheal/reconciler.go:163 HUB/pbsdrheal/reconciler.go:170 |
| 4 | Pod down: no heal; the debounce maps are in memory, so after a restart a box waits two more reports. | A | HUB/pbsdrheal/reconciler.go:134-135 |
| 5 | One goroutine; a read error does nothing this tick. | A | HUB/pbsdrheal/reconciler.go:171-174 |
| 6 | Grace is hub time. Box clock: not used here. | A | HUB/pbsdrheal/reconciler.go:153-165 E/hub/cmd/hub/main.go:829-837 (the code path of question 1) |
| 7 | Operator audit events pbsdr_selfheal_* and pbsdr_unconsumed_secret. |
A | HUB/pbsdrheal/reconciler.go:50-54 |
| 8 | SOURCE only. A pass that hangs on the remote side: NOBODY KNOWS; the pass has no time limit. | N | HUB/pbsdrheal/reconciler.go:57-63 · class W4 · S15 |
Hand step in a runbook? no hand step
CH14 — WireGuard peer list push to ep0 · OPERATOR · Hub sweep · MENTIONED
architecture folder: 8 lines in 3 files match /wgsync|peer list/ (a machine count, not a reading) · helper section: clock-hub „14. WireGuard peer list push to ep0”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR (keeps every box's tunnel allowed). wgReconciler.Run, loop; pushes the FULL peer list over SSH. |
A | E/hub/cmd/hub/main.go:752 HUB/wgsync/reconciler.go:90-106 |
| 2 | Every 5 minutes and on every change via Trigger. No setting. |
A | HUB/wgsync/reconciler.go:41 HUB/wgsync/reconciler.go:48-53 |
| 3 | 60 s limit per push. The payload grows with peers. | A | HUB/wgsync/reconciler.go:100 HUB/wgsync/reconciler.go:71-73 |
| 4 | With the hub down ep0 keeps the last list. A push that fails is logged and tried again at the next tick. Nobody is mailed when it fails again and again; a new box then cannot reach the off-site store. | N | HUB/wgsync/reconciler.go:100-102 · class W1 · S05 |
| 5 | One loop; a pending trigger covers later ones. Full list each time erases drift. | A | HUB/wgsync/reconciler.go:46-47 HUB/wgsync/reconciler.go:20-23 |
| 6 | No clock in the payload. | A | E/hub/cmd/hub/main.go:752 HUB/wgsync/reconciler.go:90-106 (the code path of question 1) |
| 7 | Log line only. | A | HUB/wgsync/reconciler.go:102 |
| 8 | TEST TestReconciler_FailureRetriedOnTick; TEST TestReconciler_TriggerPushesFullList. |
A | HUB/wgsync/reconciler_test.go:119 HUB/wgsync/reconciler_test.go:82 |
Hand step in a runbook? no hand step
CH15 — Box clock ran ahead: the tunnel alarm · OPERATOR · Hub sweep · MENTIONED
architecture folder: 13 lines in 4 files match /wg_tunnel|stamped in the future|clock.ahead/ (a machine count, not a reading) · helper section: clock-hub „15. Box clock ran ahead: tunnel alarm”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR (the household's whole-box copies stop). checkWGTunnel, called on every host report; not a timer, it runs when the box reports. |
A | HUB/api/host_tunnel.go:61 HUB/api/handler.go:1010 |
| 2 | Fires when handshake_future_s > 0 or the last handshake is older than wgTunnelStaleAfterS 6 h. Constant. |
A | HUB/api/host_tunnel.go:41 HUB/api/host_tunnel.go:23 |
| 3 | One store read per report. | A | HUB/api/host_tunnel.go:66 |
| 4 | If the box stops reporting, this check never runs; block 1 covers that. An older agent that sends neither field is not judged. | A | HUB/api/host_tunnel.go:62-64 |
| 5 | Edge-triggered from the newest stored tunnel event, so a restart does not re-mail. | A | HUB/api/host_tunnel.go:71-83 |
| 6 | This IS the clock-ahead handling: the future stamp is measured by the box's own clock; repair is a new key after 15 minutes on the agent side. Known gaps are listed in. | A | HUB/api/host_tunnel.go:31 HUB/api/host_tunnel.go:14-17 AR/05-hub-architecture.md:489-490 |
| 7 | Operator only, wg_tunnel_stale warning; wg_tunnel_ok is stored, not mailed. |
A | HUB/api/host_tunnel.go:12-14 |
| 8 | TEST TestD30_TheAlarmClearsWhenTheNewKeyShakesHands; DOC. |
A | HUB/api/host_tunnel_test.go:83 AR/05-hub-architecture.md:482-490 |
Hand step in a runbook? no hand step
CH16 — The daily scheduler and the hour change · OPERATOR · Hub daily · MENTIONED
architecture folder: 2 lines in 1 files match /scheduleDaily|nextDailyRun|D24/ (a machine count, not a reading) · helper section: clock-hub „16. The daily scheduler and the clock change”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR (mechanism under blocks 2, 11, 17, 18). scheduleDaily with nextDailyRun. |
A | E/hub/cmd/hub/main.go:1069-1093 E/hub/cmd/hub/main.go:1095-1112 |
| 2 | Budapest wall-clock time. If the zone cannot be loaded, the fallback is fixed CET, which is one hour off in summer. The pod's zone is set to Europe/Budapest. | A | E/hub/cmd/hub/main.go:1072 |
| 3 | Sleeps with time.After(delay); the job runs inline, so a long job delays only its own next slot. |
A | E/hub/cmd/hub/main.go:1088 E/hub/cmd/hub/main.go:1090 |
| 4 | A slot that is missed while the hub is down is not made up; the snapshot and the prune are the two exceptions. A time setting that cannot be read becomes 00:00 with no message. | N | E/hub/cmd/hub/main.go:1069-1112 · class W1 · S01 |
| 5 | Next run is the calendar day after and the slot that ran last never runs again. | A | AR/05-hub-architecture.md:642-645 |
| 6 | Days of 23 and 25 hours are handled (D24, with tests). While the scheduler sleeps, a jump of the hub's clock does not move the timer: NOBODY KNOWS what then runs when. | N | AR/05-hub-architecture.md:640-646 · class W2 · S10 |
| 7 | Nobody; start-up logs only. | A | E/hub/cmd/hub/main.go:1069-1093 E/hub/cmd/hub/main.go:1095-1112 (the code path of question 1) |
| 8 | TEST TestR1183_DailyJobRunsOnceOnTheDayTheClocksGoBack; TEST TestR1183_DailyJobRunsOnceOnTheDayTheClocksGoForward. |
A | E/hub/cmd/hub/r1183_dst_schedule_test.go:31 E/hub/cmd/hub/r1183_dst_schedule_test.go:56 |
Hand step in a runbook? no hand step
CH17 — Hub database snapshot (four slots) · OPERATOR · Hub daily · MENTIONED
architecture folder: 236 lines in 18 files match /snapshot/ (a machine count, not a reading) · helper section: clock-hub „17. Hub database snapshot (four slots)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR (the hub holds every household's escrow and settings). runSnapshot scheduled over dbsnap.Slots; Make. |
A | E/hub/cmd/hub/main.go:967-983 HUB/dbsnap/dbsnap.go:72-117 |
| 2 | 02:00, 08:00, 14:00, 20:00 Budapest: Slots. Keeps Keep = 2. File name carries UTC. |
A | HUB/dbsnap/dbsnap.go:32 HUB/dbsnap/dbsnap.go:27 HUB/dbsnap/dbsnap.go:89 |
| 3 | 10-minute limit. Time grows with database size; written to a tmp file then renamed; duration is logged. | A | E/hub/cmd/hub/main.go:972 HUB/dbsnap/dbsnap.go:91-104 HUB/dbsnap/dbsnap.go:113-114 |
| 4 | The one job WITH catch-up: at start, if the newest snapshot is older than CatchUpAge 6 h, one is made, NeedsCatchUp. |
A | E/hub/cmd/hub/main.go:978-980 HUB/dbsnap/dbsnap.go:148-159 |
| 5 | A running flag refuses a second run with ErrBusy. The DooPlex push timer runs 30 min later and waits for the tmp file (block 26). |
A | HUB/dbsnap/dbsnap.go:73-80 |
| 6 | The slots are safe on the hour-change night. The design text still says 'older than 24 h' and '02:30' in its first paragraph; the source has four slots. | N | AR/05-hub-architecture.md:618-620 · class W5 · S18 |
| 7 | Failure is a log line; the outside alarm is the push timer's (block 26). | A | E/hub/cmd/hub/main.go:967-983 HUB/dbsnap/dbsnap.go:72-117 (the code path of question 1) |
| 8 | TEST TestD34_SlotsAreEverySixHours; TEST TestR173_MainSchedulesTheDBSnapshot. |
A | HUB/dbsnap/dbsnap_test.go:214 E/hub/cmd/hub/r173_wiring_test.go:14 |
Hand step in a runbook? YES — RB/RUNBOOK-hub-db-offsite-backup.md
CH18 — Daily prune of old hub rows (04:30) · OPERATOR · Hub daily · MENTIONED
architecture folder: 54 lines in 12 files match /prune/ (a machine count, not a reading) · helper section: clock-hub „18. Daily prune of old hub rows (04:30)”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR. pruneAll, at start and by scheduleDaily(ctx, "prune",...). |
A | E/hub/cmd/hub/main.go:1123-1150 E/hub/cmd/hub/main.go:860-865 |
| 2 | Daily at prune_schedule, default "04:30", deployed; keeps max_days 90. Also app telemetry 90 d, stale issues 30 d, deleted-customer audit 1 year. |
A | E/hub/cmd/hub/main.go:1026 E/manifests/hub.yaml:83 E/manifests/hub.yaml:82 E/hub/cmd/hub/main.go:1134 |
| 3 | No time limit and no duration in the log. The deletes grow with the reports of each box each day. | N | E/hub/cmd/hub/main.go:1123-1150 · class W4 · S15 |
| 4 | Runs at every start, so a missed slot is caught up by the next restart; with MaxDays 0 it never runs. |
A | E/hub/cmd/hub/main.go:860-865 |
| 5 | May meet the 05:00 deadline check or a snapshot only if it runs over 30 min; no lock found. | A | E/hub/cmd/hub/main.go:1123-1150 E/hub/cmd/hub/main.go:860-865 (the code path of question 1) |
| 6 | One cut-off is made from the pod's local time and compared with a UTC column; the other uses UTC. The error is 1 or 2 hours on 90 days. | N | HUB/store/store.go:1607 HUB/store/store.go:129 HUB/store/store.go:2873 · class W2 · S10 |
| 7 | Log only. | A | E/hub/cmd/hub/main.go:1123-1150 E/hub/cmd/hub/main.go:860-865 (the code path of question 1) |
| 8 | SOURCE only. No test for the cut-off's time zone was found. | A | E/hub/cmd/hub/main.go:1123-1150 HUB/store/store.go:1606 |
Hand step in a runbook? no hand step
CH19 — The other checks on the 60-second sweep · OPERATOR · Hub sweep · MENTIONED
architecture folder: 59 lines in 16 files match /60.second|sweep/ (a machine count, not a reading) · helper section: clock-hub „19. The other checks on the 60-second sweep”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | CUSTOMER and OPERATOR mixed. Merged: host capability, host leaf, host disk, storage fill, management plane, OOB, controller supervisor. All run in the one sweep goroutine. | A | E/hub/cmd/hub/main.go:937-946 E/hub/cmd/hub/main.go:927-965 |
| 2 | Every 60 s in the fixed order of. A backup-cadence refusal repeats every refusalRepeatEvery 24 h. |
A | E/hub/cmd/hub/main.go:935-962 HUB/monitor/host_capability.go:72 |
| 3 | All checkers run one after the other in one loop, with no limit for each. A slow checker delays all the others, the liveness check too. Each reads the whole fleet at each pass. | N | E/hub/cmd/hub/main.go:935-962 · class W4 · S15 |
| 4 | Pod down: nothing. After start, band states are rebuilt from the latest reports. | A | E/hub/cmd/hub/main.go:937-946 E/hub/cmd/hub/main.go:927-965 (the code path of question 1) |
| 5 | One goroutine, so no overlap; a Go ticker drops ticks when a pass overruns. | A | E/hub/cmd/hub/main.go:937-946 E/hub/cmd/hub/main.go:927-965 (the code path of question 1) |
| 6 | They judge the latest report's content, not time, except the 24 h repeat. | A | HUB/monitor/host_capability.go:172 |
| 7 | Dispatcher mails; storage fill has its own 6 h grain. | A | AR/08-alarm-ladder.md:305 |
| 8 | SOURCE only. How long one whole sweep takes: NOBODY KNOWS. | N | E/hub/cmd/hub/main.go:927-965 · class W4 · S16 |
Hand step in a runbook? no hand step
CH20 — Reminder about a held app · CUSTOMER · Hub sweep · MENTIONED
architecture folder: 32 lines in 7 files match /held.app|held app/ (a machine count, not a reading) · helper section: clock-hub „20. Reminder about a held app”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR (D53: no household mail). heldApps.Check; HeldAppsChecker.Check. |
A | E/hub/cmd/hub/main.go:947 HUB/monitor/held_apps.go:102 |
| 2 | One pass per heldAppCheckEvery 15 min. First mail after a grace of 6 h / 72 h / 24 h by reason; then every heldAppRepeat 7 d. |
A | HUB/monitor/held_apps.go:36 HUB/monitor/held_apps.go:40-48 HUB/monitor/held_apps.go:34 |
| 3 | One list of all held apps. | A | HUB/monitor/held_apps.go:114 |
| 4 | Pod down: the reminder comes on the first pass after start; the reminded-at time is stored BEFORE sending, so a restart cannot mail twice. | A | HUB/monitor/held_apps.go:141-145 |
| 5 | A row the box has not confirmed for heldAppStale 50 h is not reminded. |
A | HUB/monitor/held_apps.go:125-129 |
| 6 | The reminder counts from a time that the box reports. A box clock that is behind makes the reminder early; one that is ahead makes it late. | N | HUB/monitor/held_apps.go:130-134 HUB/monitor/held_apps.go:22 · class W2 · S07 |
| 7 | Operator, app_held_reminder warning. |
A | HUB/monitor/held_apps.go:157-163 |
| 8 | TEST file named in source; DOC. | A | HUB/monitor/held_apps.go:31 AR/08-alarm-ladder.md:655 |
Hand step in a runbook? no hand step
CH21 — 'A golden is published but not vouched' · OPERATOR · Hub sweep · MENTIONED
architecture folder: 57 lines in 17 files match /vouch/ (a machine count, not a reading) · helper section: clock-hub „21. "A golden is published but not vouched"”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR. goldenVouch check on the sweep; GoldenVouchChecker.Check. |
A | E/hub/cmd/hub/main.go:948-950 HUB/monitor/golden_vouch.go:54 |
| 2 | One Gitea listing per goldenVouchCheckEvery 15 min; alarm after goldenVouchGrace 2 h. |
A | HUB/monitor/golden_vouch.go:42-43 |
| 3 | Listing limit goldenVouchListDeadline 20 s; it runs inside the sweep, so a slow Gitea holds the sweep for up to 20 s. |
A | HUB/monitor/golden_vouch.go:44 HUB/monitor/golden_vouch.go:62-64 |
| 4 | Pod restart: the gap clock and the alerted mark are in memory, so the 2 h starts again and the alarm is mailed again ("one alarm per unvouched version per process"). | A | HUB/monitor/golden_vouch.go:34-37 HUB/monitor/golden_vouch.go:23 |
| 5 | Gitea unreachable or empty list: no verdict, never an alarm. | A | HUB/monitor/golden_vouch.go:65-67 |
| 6 | Hub clock only. | A | E/hub/cmd/hub/main.go:948-950 HUB/monitor/golden_vouch.go:54 (the code path of question 1) |
| 7 | Operator event golden_not_vouched. |
A | HUB/monitor/golden_vouch.go:95-96 |
| 8 | TEST TestGoldenVouchChecker named at. |
A | HUB/monitor/golden_vouch.go:24 |
Hand step in a runbook? no hand step
CH22 — Shared storage boxes: fill and reachability · OPERATOR · Hub sweep · MENTIONED
architecture folder: 37 lines in 9 files match /storage box|pool.box|oversub/ (a machine count, not a reading) · helper section: clock-hub „22. Shared storage boxes: fill and reachability”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | OPERATOR (every household's off-site space). offsiteBox and pbsdrBox checks. |
A | E/hub/cmd/hub/main.go:951-956 |
| 2 | On the 60 s sweep, but one provider read per boxFetchInterval 15 min, test at and. Warn 80 %, critical 90 %. |
A | HUB/monitor/offsite_box.go:72 HUB/monitor/offsite_box.go:132 HUB/monitor/pbsdr_box.go:130 HUB/monitor/offsite_box.go:67-68 |
| 3 | Provider client limit 30 s. Does not grow with boxes. | A | HUB/hetznerapi/hetznerapi.go:143-145 |
| 4 | Pod down: no reading. Unreachable is declared after consecutive failed 15-minute windows. | A | HUB/monitor/pbsdr_box.go:80-82 |
| 5 | Throttled by lastFetch in memory; first pass after a restart fetches at once. |
A | HUB/monitor/offsite_box.go:132 |
| 6 | Hub clock only. | A | E/hub/cmd/hub/main.go:951-956 (the code path of question 1) |
| 7 | Operator events for fill, oversubscription, unreachable and recovered. | A | HUB/monitor/offsite_box.go:266-330 |
| 8 | SOURCE. | A | HUB/monitor/offsite_box.go:70-72 |
Hand step in a runbook? no hand step
CH23 — DooPlex: push of the hub database snapshot off-site · OPERATOR · DooPlex and ep0 · MENTIONED
architecture folder: 33 lines in 8 files match /hub.db|database snapshot/ (a machine count, not a reading) · helper section: clock-hub „26. DooPlex: push of the hub database snapshot off-site”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | NOT MEASURED. OPERATOR. systemd timer felhom-hub-db-backup.timer on DooPlex, runs the script E/scripts/hub-db-backup/felhom-hub-db-backup:19; installed by hand per. |
A | E/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10 RB/RUNBOOK-hub-db-offsite-backup.md:107-108 |
| 2 | NOT MEASURED (DooPlex and ep0 are outside this run). 02:30, 08:30, 14:30 and 20:30 with up to 2 minutes of random delay, in the host's local time (the unit names no zone). | A | E/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10 E/scripts/hub-db-backup/felhom-hub-db-backup.timer:12 |
| 3 | TimeoutStartSec=45min. Waits up to WAIT_S 1200 s for the hub's tmp file E/scripts/hub-db-backup/felhom-hub-db-backup:20, loop E/scripts/hub-db-backup/felhom-hub-db-backup:35-40. Grows with database size. |
A | E/scripts/hub-db-backup/felhom-hub-db-backup.service:13 |
| 4 | Persistent=true: a slot missed while DooPlex was off runs at boot. It refuses a snapshot older than MAX_AGE_H 7 E/scripts/hub-db-backup/felhom-hub-db-backup:19, line E/scripts/hub-db-backup/felhom-hub-db-backup:50. |
A | E/scripts/hub-db-backup/felhom-hub-db-backup.timer:11 |
| 5 | No flock in the script (grep flock/lock returned nothing); systemd does not start a oneshot unit twice. A catch-up run right after boot can meet a hub that has not made its catch-up snapshot yet: history of a false failure in E/scripts/hub-db-backup/felhom-hub-db-backup:15-18. |
A | E/scripts/hub-db-backup/felhom-hub-db-backup.timer:7-10 RB/RUNBOOK-hub-db-offsite-backup.md:107-108 (the code path of question 1) |
| 6 | NOT MEASURED (DooPlex and ep0 are outside this run). The timer's zone and the hub's zone must match for the 30-minute offset. On the autumn hour-change day the 02:00 copy never reaches ep0 and the 20:00 copy is pushed twice. | K | E/scripts/hub-db-backup/test_hub_db_backup.py:309-316 · R-1293 |
| 7 | The runbook says in one place that the alarm fires after 13 hours, and the alarm rule printed in the same runbook says 26 hours. | N | RB/RUNBOOK-hub-db-offsite-backup.md:100 RB/RUNBOOK-hub-db-offsite-backup.md:152-153 · class W5 · S18 |
| 8 | NOT MEASURED (DooPlex and ep0 are outside this run). A test file for the script exists and is run by hand. The live alarm rule is in another repository and on DooPlex; this run did not read it. | A | RB/RUNBOOK-hub-db-offsite-backup.md:107-108 E/scripts/hub-db-backup/test_hub_db_backup.py:309-316 |
Hand step in a runbook? YES — RB/RUNBOOK-hub-db-offsite-backup.md
CH24 — DooPlex: Sunday restore tests · OPERATOR · DooPlex and ep0 · MENTIONED
architecture folder: 121 lines in 19 files match /restore.test/ (a machine count, not a reading) · helper section: clock-hub „27. DooPlex: Sunday restore tests”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | NOT MEASURED. OPERATOR. Two timers, merged: felhom-hub-db-restore-test.timer and felhom-dooplex-offsite-restore-test.timer. |
A | E/scripts/hub-db-backup/felhom-hub-db-restore-test.timer:6 E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test.timer:6 |
| 2 | Sunday 04:30 and Sunday 05:30, host local time; Persistent=true. |
A | E/scripts/hub-db-backup/felhom-hub-db-restore-test.timer:7 |
| 3 | Limits 45 min and 90 min. Grow with database and dump size. | A | E/scripts/hub-db-backup/felhom-hub-db-restore-test.service:13 E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test.service:13 |
| 4 | Missed Sunday runs at next boot (Persistent). Each refuses a copy older than MAX_AGE_H 50 E/scripts/hub-db-backup/felhom-hub-db-restore-test:13 and E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test:14. |
A | E/scripts/hub-db-backup/felhom-hub-db-restore-test.timer:6 E/scripts/dooplex-offsite/felhom-dooplex-offsite-restore-test.timer:6 (the code path of question 1) |
| 5 | NOT MEASURED (DooPlex and ep0 are outside this run). Sunday 04:30 is between the 02:00 snapshot push and the 05:00 PBS pull. The script takes no lock against them. | N | class W3 · S14 |
| 6 | NOT MEASURED (DooPlex and ep0 are outside this run). On the last Sunday of October the hour 02:00 to 03:00 comes twice. What the timer does at 04:30 that day: NOBODY KNOWS. | N | class W2 · S20 |
| 7 | Staleness alarm HubDBRestoreTestStale after 8 days. |
A | RB/RUNBOOK-hub-db-offsite-backup.md:158 |
| 8 | NOT MEASURED (DooPlex and ep0 are outside this run). DOC only. This run did not read if the alarm rule is deployed. | A | RB/RUNBOOK-hub-db-offsite-backup.md:152-161 |
Hand step in a runbook? YES — RB/RUNBOOK-hub-db-offsite-backup.md
CH25 — DooPlex: nightly off-site push and the release copy · OPERATOR · DooPlex and ep0 · MENTIONED
architecture folder: 7 lines in 2 files match /release.copy|release copy/ (a machine count, not a reading) · helper section: clock-hub „28. DooPlex: nightly off-site push and release copy”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | NOT MEASURED. OPERATOR. Merged: felhom-dooplex-offsite.timer and felhom-release-copy.timer. |
A | E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:6 E/scripts/release-copy/felhom-release-copy.timer:7 |
| 2 | 00:20 and 01:40 host local time; the order is stated in a comment. Random delay 2 min and 5 min,. | A | E/scripts/release-copy/felhom-release-copy.timer:6 E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:8 E/scripts/release-copy/felhom-release-copy.timer:9 |
| 3 | NOT MEASURED (DooPlex and ep0 are outside this run). The units have limits of 90 minutes and 2 hours. A 90-minute run from 00:20 ends at 01:50, after the 01:40 start of the next job. No lock between them. | N | E/scripts/dooplex-offsite/felhom-dooplex-offsite.service:13 E/scripts/release-copy/felhom-release-copy.service:12 · class W3 · S14 |
| 4 | Both Persistent=true,. The push refuses a dump older than DUMP_MAX_AGE_H 7 E/scripts/dooplex-offsite/felhom-dooplex-offsite:23, message E/scripts/dooplex-offsite/felhom-dooplex-offsite:55. |
A | E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:7 E/scripts/release-copy/felhom-release-copy.timer:8 |
| 5 | The push depends on a dump CronJob that is not in this clone (named at E/scripts/dooplex-offsite/felhom-dooplex-offsite:55). One retry E/scripts/dooplex-offsite/felhom-dooplex-offsite:66. | A | E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:6 E/scripts/release-copy/felhom-release-copy.timer:7 (the code path of question 1) |
| 6 | 00:20 and 01:40 are outside the 02:00-03:00 change hour. | A | E/scripts/dooplex-offsite/felhom-dooplex-offsite.timer:6 E/scripts/release-copy/felhom-release-copy.timer:7 (the code path of question 1) |
| 7 | OnFailure mail on both, a metric file is written E/scripts/dooplex-offsite/felhom-dooplex-offsite:143-153. |
A | E/scripts/dooplex-offsite/felhom-dooplex-offsite.service:6 E/scripts/release-copy/felhom-release-copy.service:6 |
| 8 | NOT MEASURED (DooPlex and ep0 are outside this run). SOURCE only. The release-copy timer is switched on by hand; this run did not read if it is on. | A | E/scripts/release-copy/install.sh:4 |
Hand step in a runbook? no hand step
CH26 — DooPlex and ep0: PBS copy, prune, verify, clean-up · OPERATOR · DooPlex and ep0 · MENTIONED
architecture folder: 124 lines in 17 files match /ep0|PBS.{0,20}(prune|sync|pull)/ (a machine count, not a reading) · helper section: clock-hub „29. DooPlex and ep0: PBS copy, prune, verify, clean-up”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | NOT MEASURED. OPERATOR (holds the second copy of every household's whole-box backup). PBS jobs described in, plus felhom-ep0-copy-gc.timer. |
A | RB/ep0-datastore-copy.md:15-18 E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:5 |
| 2 | ep0 prune 03:30, pull to DooPlex 05:00; DooPlex prune 07:30; copy clean-up 08:00; verify Saturday 06:30; GC Sunday 08:30; hub-DB prune on ep0 03:45. | A | RB/ep0-datastore-copy.md:15 RB/ep0-datastore-copy.md:17 E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:5 RB/ep0-datastore-copy.md:16 |
| 3 | NOT MEASURED (DooPlex and ep0 are outside this run). The PBS jobs are in PBS's own config, not in the repository. They grow with the backup size of every box. | A | RB/ep0-datastore-copy.md:19 |
| 4 | The gc timer is Persistent=true. PBS jobs: catch-up behaviour not in repo. |
A | E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:6 |
| 5 | NOT MEASURED (DooPlex and ep0 are outside this run). The order is by the clock only: prune 07:30, clean-up 08:00. A pull that runs past 07:30 meets the prune: NOBODY KNOWS what then happens. | N | E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.timer:2 · class W3 · S14 |
| 6 | NOT MEASURED (DooPlex and ep0 are outside this run). Two machines, two clocks. 'The ep0 prune at 03:30 is before the 05:00 pull' is true only while both clocks and both zones agree. No document gives their zones. | N | RB/ep0-datastore-copy.md:1 · class W3 · S14 |
| 7 | NOT MEASURED (DooPlex and ep0 are outside this run). PBS has its own notification target. The clean-up unit has no failure hook and no time limit; a clean-up that fails tells nobody. | N | E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.service:9 RB/ep0-datastore-copy.md:19 · class W1 · S01 |
| 8 | DOC. | A | RB/ep0-datastore-copy.md:15-19 |
Hand step in a runbook? YES — RB/ep0-datastore-copy.md
CH27 — Nightly CI: the gates and the security watch · OPERATOR · DooPlex and ep0 · MENTIONED
architecture folder: 85 lines in 19 files match /nightly|security.watch|cron/ (a machine count, not a reading) · helper section: clock-hub „30. Nightly CI: gates and the security watch”
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | NOT MEASURED. OPERATOR. Gitea CI crons in felhom.eu: gates and security-watch. The other three repos have no scheduled job (grep schedule:/cron: in their .gitea/workflows/ returned nothing). |
A | E/.gitea/workflows/gates.yml:26-27 E/.gitea/workflows/security-watch.yml:21-22 |
| 2 | 03:20 UTC and 03:40 UTC every night; the reason for the order is in. CI cron is UTC per, so it moves one hour against the Budapest night jobs at each clock change. | A | E/.gitea/workflows/gates.yml:21-23 AR/08-alarm-ladder.md:710 |
| 3 | Limits: 5 min min. The watch grows with the number of pinned programs. | A | E/.gitea/workflows/gates.yml:53 E/.gitea/workflows/gates.yml:20 E/.gitea/workflows/security-watch.yml:33 |
| 4 | NOT MEASURED (DooPlex and ep0 are outside this run). A missed night is not made up. The watch's weekly summary repairs itself: it is sent when the last one is 8 or more days old. A runner that drops the job tells nobody. | N | AR/08-alarm-ladder.md:717 · class W1 · S01 |
| 5 | Both run on felhom-gates runners; the watch commits its memory to main with 3 attempts. A push race with a human commit is retried, then fails. |
A | E/.gitea/workflows/security-watch.yml:32 E/.gitea/workflows/security-watch.yml:95-106 |
| 6 | UTC cron, no DST. Reminder ages are computed by the script from its memory file. | A | AR/08-alarm-ladder.md:716-720 |
| 7 | Failure mail to admin: "[felhom CI] security watch FAILED"; gates has its own alarm step. Whether the alarm step runs for a reaped job is recorded as unknown. | A | E/.gitea/workflows/gates.yml:245 E/.gitea/workflows/gates.yml:50-52 |
| 8 | NOT MEASURED (DooPlex and ep0 are outside this run). DOC. A night with no run at all: NOBODY KNOWS who notices. | N | AR/08-alarm-ladder.md:708-722 · class W1 · S01 |
Hand step in a runbook? no hand step