gates / gates (push) Successful in 2m2s
Read-only review of every job on a timer and of everything that ends on a date, with measurements from the three test boxes. Nothing ends in the next 14 days. Register 236 -> 245 (R-1379..R-1387), pointer lines on twelve rows, D76-D79 in STATUS.md. Part C: felhom-agent's clone address holds no token now; the other clones wait for the operator's yes. R-1035's 4-week check moved to 2026-11-18 (operator read needed). Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
291 lines
38 KiB
Python
291 lines
38 KiB
Python
"""marks.py — the main session's rulings for the Clock list (2026-11-11). Data only; make-table.py reads it.
|
|
|
|
One entry for each job that is in the table:
|
|
src (helper, index) the helper section the default answers come from (evidence/helpers/resolved-<helper>.json)
|
|
name the heading
|
|
seed the kind (Night, Interval, Watchdog, Hub daily, Hub sweep, DooPlex and ep0)
|
|
named ("grep", REGEX): 0 lines in the architecture folder = UNNAMED, else MENTIONED
|
|
o {question: (mark, answer, pointers)} — the MAIN SESSION's words.
|
|
pointers "+ rest" = the helper's pointers for that question (opened by the tool), then rest.
|
|
A pointer written out here was opened by eye.
|
|
A question with no entry here takes the helper's answer and the mark A.
|
|
rb the ninth line: is the hand step written in a runbook?
|
|
"""
|
|
NM = "NOT MEASURED (DooPlex and ep0 are outside this run). "
|
|
|
|
|
|
def N(cls, s, text, p="+"):
|
|
return ("N", text, f"{p} · class {cls} · S{s:02d}")
|
|
|
|
|
|
def K(row, text, p="+"):
|
|
return ("K", text, f"{p} · R-{row}")
|
|
|
|
|
|
def A(text, p="+"):
|
|
return ("A", text, p)
|
|
|
|
|
|
NOLIMIT = "No time limit in the source, and no duration was measured. Nobody knows how long it takes on a big box."
|
|
NH = "no hand step"
|
|
_M = [
|
|
# ---------------------------------------------------------------- controller (in the guest)
|
|
("CC01", "clock-controller", 0, "Night dump leg: databases and volumes · CUSTOMER", "Night", r"db-dump|dump leg",
|
|
{3: A("MEASURED, the night of 2026-11-11: 2 min 33 s on demo-hp, 40 s on Tester 1, 18 s on the N100. One database dump has a 5-minute limit. The whole leg has no limit. It grows with the number and size of the apps; all three stores are small.",
|
|
"`EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt` `CT/internal/appbackup/dbdump.go:291`")}, NH),
|
|
("CC02", "clock-controller", 1, "Second-drive copy leg · CUSTOMER", "Night", r"tier.?2|second.drive|cross.drive",
|
|
{5: N("W3", 14, "It waits for the dump leg by a start time only (60 minutes later). It is skipped while a migration runs. The file holds no lock of its own. With a backup time between 02:00 and 02:59 on the spring hour-change night the two legs start at the same instant; that case is row R-1260."),
|
|
7: A("The household and the hub are told of a failed copy: the copy's result goes to the notifier.", "`CT/cmd/controller/main.go:1427`")}, NH),
|
|
("CC03", "clock-controller", 2, "Off-site push leg, with the clean-up of old snapshots · CUSTOMER", "Night", r"offbox|off.site push|offsite push",
|
|
{5: K(1186, "When another backup holds the flag, the push is skipped with an INFO line and the leg is marked as ended. Only the hub's 48-hour check notices. A long second-drive leg can cost the night's off-site copy.",
|
|
"`CT/internal/backup/offbox.go:1054-1057`")},
|
|
"NO — no runbook says how to see or repair a skipped off-site night"),
|
|
("CC04", "clock-controller", 3, "Missed-night catch-up and the resume watch · CUSTOMER", "Night", r"catch.?up|nightchain|missed night",
|
|
{6: N("W2", 8, "The watch reads a gap of more than 5 minutes in the wall clock as 'the box was asleep'. A clock that is corrected forward by more than 5 minutes looks the same. The design text says this was not measured live.")}, NH),
|
|
("CC05", "clock-controller", 4, "Missed-backup banner (26 hours) · CUSTOMER", "Night", r"26 h|missed.backup banner|banner",
|
|
{6: N("W2", 8, "The age is wall-clock arithmetic. A clock that is set back hides the banner for as long as the step. No test sets the clock back.")}, NH),
|
|
("CC06", "clock-controller", 5, "Whole-guest backup driver: stop apps, ask the agent, start apps · CUSTOMER", "Night", r"quiesce|whole.guest|vzdump",
|
|
{6: K(1260, "The gate counts wall-clock minutes; the poll is a monotonic ticker. On the hour-change night the two disagree. No test covers that night.")}, NH),
|
|
("CC07", "clock-controller", 6, "Automatic app-update leg · CUSTOMER", "Night", r"update.leg|unattended",
|
|
{6: K(1260, "The leg's deadline is the backup time plus 5 hours of real time, and the scheduler's slots are in Budapest time. On the hour-change night a step can start after the backup gate stopped waiting. A daily fire that is more than 60 minutes late is skipped, so no update runs at noon."),
|
|
7: N("W1", 6, "Nobody. When the leg does not run, the only trace is one log line on the box. No event goes to the hub.", "`CT/internal/stacks/unattended.go:374-375`")}, NH),
|
|
("CC08", "clock-controller", 7, "Off-site integrity check · CUSTOMER", "Night", r"integrity",
|
|
{3: K(401, "MEASURED 2026-11-11: under 1 second on all three boxes (small stores). The check reads the whole store; the time on a large store rests on one measurement.", "`EV/box/demo-hp-guest-night.txt`"),
|
|
7: A("A failed check goes to the notifier: the household and the hub are told.", "`CT/cmd/controller/main.go:4144-4186`")}, NH),
|
|
("CC09", "clock-controller", 8, "Off-site proof restore · CUSTOMER", "Night", r"proof restore|offsite.proof|proof",
|
|
{5: N("W3", 12, "MEASURED 2026-11-11: on Tester 1 and on the N100 the whole-guest backup started at about 05:30 local time, the same minute as this job. This job is not behind the whole-guest gate. It takes the same flag as the off-site push; a meeting there is a silent skip (R-1186).",
|
|
"`EV/box/tester-1-host-night.txt` `EV/box/demo-felhom-host-night.txt` `EV/box/tester-1-guest-night.txt`"),
|
|
7: A("An empty or failed proof goes to the notifier: the household and the hub are told.", "`CT/cmd/controller/main.go:4144-4186`")}, NH),
|
|
("CC10", "clock-controller", 9, "Hub report: every 15 minutes, on change, and the hanging wait · CUSTOMER", "Interval", r"15.min|hub report|pusher",
|
|
{7: A("Nobody on the box: the box stores the time of the last attempt only. The outside sign is the hub's liveness check (CH01).", "`CT/internal/report/pusher.go:109-111` `HUB/monitor/host_staleness.go:59-60`")}, NH),
|
|
("CC11", "clock-controller", 10, "Controller self-update: check, floor, nightly auto · CUSTOMER", "Interval", r"self.?update",
|
|
{3: N("W4", 15, "The registry call has a 15-second limit. The swap of the container has no limit in the file, and no duration was measured in this run."),
|
|
5: N("W3", 12, "The auto-update waits while a backup or an app update runs. Its default time, 04:30, is the minute at which the whole-guest gate opens for a 02:30 backup time. Nothing orders the two.", "`CT/internal/config/config.go:419` `CT/internal/selfupdate/updater.go:533-544`"),
|
|
6: A("The check uses a monotonic ticker. The nightly auto-update is a daily slot (see CC22). The switch is off unless the config turns it on; the default time is 04:30.", "`CT/internal/config/config.go:278-279` `CT/internal/config/config.go:419`")}, NH),
|
|
("CC12", "clock-controller", 11, "Catalog sync: app templates from git · CUSTOMER", "Interval", r"catalog sync|git.sync|sync",
|
|
{3: N("W4", 15, "Each git command has its own limit. One whole sync has none, and no duration was measured.", "`CT/internal/sync/sync.go:724` `CT/internal/sync/sync.go:215-216`"),
|
|
4: N("W1", 23, "A missed sync is done at the next tick. A sync that fails for days writes an ERROR line on the box at each tick. Nobody is told.", "`CT/internal/sync/sync.go:236` `CT/internal/sync/sync.go:247`"),
|
|
5: A("Two syncs cannot run at once: `doSync` takes the syncer's lock first. The 30-second guard is for the button only.", "`CT/internal/sync/sync.go:215-216` `CT/internal/sync/sync.go:153-155`")}, NH),
|
|
("CC13", "clock-controller", 12, "System health check and base-stack self-heal · CUSTOMER", "Interval", r"health check|self.heal",
|
|
{3: N("W4", 15, NOLIMIT)}, NH),
|
|
("CC14", "clock-controller", 13, "Dead-app scan · CUSTOMER", "Interval", r"stack.scan|dead.app|crash.?loop",
|
|
{3: N("W4", 15, NOLIMIT + " It grows with the number of apps."),
|
|
6: N("W2", 8, "The quiet time after a start uses the wall clock. A clock that steps back after the start makes the quiet time longer. No test.")}, NH),
|
|
("CC15", "clock-controller", 14, "Disk-fill watch · CUSTOMER", "Interval", r"fill.?watch|disk.fill",
|
|
{3: N("W4", 15, NOLIMIT),
|
|
5: A("The night slot and the interval job are two jobs with two names. The watcher holds one lock, so the two cannot change its state at the same time.", "`CT/internal/fillwatch/fillwatch.go:162` `CT/cmd/controller/main.go:1804-1805`"),
|
|
7: A("The household: the watcher pushes a customer event when a disk fills.", "`CT/internal/fillwatch/fillwatch.go:80-153`")}, NH),
|
|
("CC16", "clock-controller", 15, "Drive health check, hourly · CUSTOMER", "Interval", r"SMART|disk.health|drive health",
|
|
{3: N("W4", 15, "One fetch was measured at about 0.8 s (a comment in the source). No time limit in the source."),
|
|
7: A("The household and the hub: the check emits `disk_health_degraded`.", "`CT/internal/web/disk_health.go:175-280`")}, NH),
|
|
("CC17", "clock-controller", 16, "Agent channel health, every 60 seconds · CUSTOMER", "Interval", r"agent channel|channel health", {}, NH),
|
|
("CC18", "clock-controller", 17, "Held-app report to the hub · CUSTOMER", "Interval", r"held.app|held app",
|
|
{6: N("W2", 8, "The daily resend is wall-clock arithmetic. A clock that is set back delays the resend by the size of the step.")}, NH),
|
|
("CC19", "clock-controller", 18, "Abandoned off-site store sweep · CUSTOMER", "Night", r"abandon",
|
|
{3: A("One sweep has a 1-minute limit.", "`CT/internal/backup/offbox_abandon.go:180`"),
|
|
5: N("W3", 11, "The sweep takes no lock. MEASURED: on the three boxes it ran 55 minutes after the off-site push ended, so they did not meet on these small stores.", "`CT/internal/backup/offbox_abandon.go:180` `EV/box/demo-hp-guest-night.txt`"),
|
|
7: N("W1", 6, "Nobody. A sweep that fails writes a WARN or ERROR line on the box only.", "`CT/internal/backup/offbox_abandon.go:180`")}, NH),
|
|
("CC20", "clock-controller", 19, "Restore-copy sweep and conversion-copy release · CUSTOMER", "Interval", r"restore.copy|restore copy",
|
|
{3: N("W4", 15, NOLIMIT),
|
|
4: N("W1", 25, "The first run is 6 hours after the controller starts. A box that is on for under 6 hours a day never runs it; old restore copies stay. Nobody is told."),
|
|
7: A("The household is told when a restore copy is deleted.", "`CT/internal/notify/notifier.go:1601`")}, NH),
|
|
("CC21", "clock-controller", 21, "Upload and CPU watch, torrent limit rule, line-test hold · CUSTOMER", "Interval", r"upwatch|unexplained|upload",
|
|
{5: N("W3", 14, "Two loops with their own tickers. The file holds no lock between them."),
|
|
7: A("The household and the hub: the watch emits `app_unexplained_upload` and `app_unexplained_cpu`.", "`CT/internal/upwatch/watch.go:309-382`")}, NH),
|
|
("CC22", "clock-controller", 23, "The scheduler itself: daily slots and interval jobs · CUSTOMER", "Night", r"scheduler|daily slot",
|
|
{6: K(1260, "Slots are in Budapest time. On the spring night 02:30 does not exist; on the autumn night it comes twice. No test covers either night. A clock set back while a timer waits does not matter: the timer runs on monotonic time.", "`CT/internal/scheduler/scheduler.go:499-519`"),
|
|
8: A("SOURCE. MEASURED 2026-11-11: the night's order and the start times on three boxes (see the README's night table).", "`CT/internal/scheduler/scheduler.go:344-347` `EV/box/demo-hp-guest-night.txt`")}, NH),
|
|
# ---------------------------------------------------------------- agent (on the host)
|
|
("CA01", "clock-agent", 0, "Whole-guest backup: due-check and run · CUSTOMER", "Night", r"vzdump|whole.guest|due.check",
|
|
{6: K(1189, "The age is wall-clock time. A host clock that was ahead during one backup gives a negative age afterwards; the tier reads 'not due' until real time passes it."),
|
|
8: K(1189, "TEST and DOC for the due-check. MEASURED 2026-11-11: demo-hp 11 to 13 minutes, Tester 1 about 3 minutes, the N100 under 1 minute; on the N100 the start time moved 5 minutes later each day. A clock jump has no test.", "`EV/box/demo-hp-host-night.txt` `EV/box/tester-1-host-night.txt` `EV/box/demo-felhom-host-night.txt`")}, NH),
|
|
("CA02", "clock-agent", 1, "OS-update night leg: guest, host, Docker, Proxmox · CUSTOMER", "Night", r"os.?update|OS update",
|
|
{6: N("W2", 8, "The 20-hour gap compares the clock with a time read from a file. A stamp that was written while the clock was ahead keeps the leg skipped until real time passes that stamp plus 20 hours.", "`AG/internal/osupdate/leg.go:661-663`"),
|
|
8: N("W2", 8, "TEST and DOC for the normal path. A stamp from the future: NOBODY KNOWS.")}, NH),
|
|
("CA03", "clock-agent", 2, "Kernel lane: night reboot, 20-minute boot judge, one self-revert · CUSTOMER", "Night", r"kernel",
|
|
{6: K(1301, "The 20-minute judge runs on monotonic time; a clock step does not move it. Nothing in the installer sets up time keeping, and a host can start with a wrong clock: the N100's host clock is 32 s behind today."),
|
|
8: A("TEST and DOC. The design text itself says that a real freeze was not measured.", "`AR/11-os-updates.md:599-601` `AG/internal/osupdate/kernel_test.go:79`")},
|
|
"YES — `RB/crash-guard.md`"),
|
|
("CA04", "clock-agent", 3, "Crash guard: early-boot unit and the hourly re-arm timer · CUSTOMER", "Watchdog", r"crash.guard|crash guard",
|
|
{5: N("W3", 14, "The second unclean boot inside 60 minutes trips it; it arms again after 24 hours. The boot unit and the hourly check share one state file and take no lock."),
|
|
6: N("W2", 22, "The script reads the wall clock at early boot, before any time service runs. A clock that is wrong at boot can move unclean boots into or out of the 60-minute window."),
|
|
8: N("W2", 22, "TEST and DOC for the windows. A wrong clock at boot: NOBODY KNOWS.")},
|
|
"YES — `RB/crash-guard.md`"),
|
|
("CA05", "clock-agent", 4, "Scheduled restore-test evaluation · CUSTOMER", "Night", r"restore.test",
|
|
{6: N("W2", 8, "The 'settled' cut-off uses the wall clock. A clock that is set back makes no archive look settled, so no test starts. The ticker itself is monotonic."),
|
|
8: N("W2", 8, "TEST and DOC. MEASURED 2026-11-11: restore tests took 1 to 14 minutes on the three boxes. A clock set back: NOBODY KNOWS.", "`EV/box/demo-hp-host-night.txt`")}, NH),
|
|
("CA06", "clock-agent", 5, "Controller supervisor: restarts a dead or silent controller · CUSTOMER", "Watchdog", r"supervisor",
|
|
{3: N("W4", 15, "No time limit on one sweep in the source, and no duration was measured. Nothing grows with data."),
|
|
6: N("W2", 8, "The windows (3 restarts in 15 minutes, the 24-hour counter) use the wall clock in UTC. Nobody tested a clock jump.", "`AG/internal/localapi/controllersupervisor.go:550-552` `AG/internal/localapi/controllersupervisor.go:308`")}, NH),
|
|
("CA07", "clock-agent", 6, "Guest-power watchdog: starts a guest that must run · CUSTOMER", "Watchdog", r"guest.power|guest power",
|
|
{6: N("W2", 8, "The next try time is kept in memory from the wall clock. A clock set back delays the next try; an agent restart tries again at once.", "`AG/internal/localapi/guestpower.go:234` `AG/internal/localapi/guestpower.go:253`"),
|
|
7: N("W1", 4, "When the watchdog gives up, it writes one ERROR line in the host's journal. No event goes to the hub. The household's box is down and the outside sign is only the hub's liveness check.", "`AG/internal/localapi/guestpower.go:200-206`"),
|
|
8: N("W1", 4, "TEST for the start and the back-off. Who hears the give-up: NOBODY KNOWS.")},
|
|
"NO — no runbook says what to do after the watchdog gave up"),
|
|
("CA08", "clock-agent", 7, "Hub report and heartbeat loop · CUSTOMER", "Interval", r"heartbeat|poll interval|report loop",
|
|
{6: A("The ticker is monotonic. The report loop follows the hub's interval override, clamped to a range. The reconcile engine keeps the interval that it read at the start.", "`AG/internal/hub/loop.go:188-196` `AG/cmd/felhom-agent/main.go:969`")}, NH),
|
|
("CA09", "clock-agent", 8, "Signed jobs: the validity window against the host clock · OPERATOR", "Interval", r"signed job|signed op|not.before|valid",
|
|
{4: N("W1", 9, "A job that the agent refuses as expired is not run. The agent's log line is the only trace on the box; no path that tells the hub or the operator 'refused: expired' was found."),
|
|
6: N("W2", 9, "The window is checked against the host's wall clock. A host clock that is behind refuses every job as 'not yet valid'; one that is ahead refuses them as 'expired'. So a box with a wrong clock cannot take an agent update.", "`AG/internal/authz/verifier.go:153-162`"),
|
|
7: N("W1", 9, "The runner logs a finished or failed job on the host. For a job refused on its dates, no message to the hub was found.", "`AG/internal/signedjobs/runner.go:176-190`")},
|
|
"NO — no runbook names a wrong host clock as a cause of refused signed jobs"),
|
|
("CA10", "clock-agent", 9, "Storage watchdog and the 20-second mount re-assert · CUSTOMER", "Watchdog", r"storage watchdog|re.?assert|ReassertGuestBinds",
|
|
{3: N("W4", 15, "No time limit on one tick in the source, and no duration was measured. The work grows with the number of drives."),
|
|
6: N("W2", 8, "The back-off uses the wall clock in UTC. A clock set back stretches the back-off; a clock set forward ends it early. The tickers are monotonic."),
|
|
8: N("W4", 15, "TEST for the reconcile and the back-off. The 20-second ticker has no test: NOBODY KNOWS.")}, NH),
|
|
("CA11", "clock-agent", 10, "Weekly guest disk trim · CUSTOMER", "Interval", r"fstrim|trim",
|
|
{6: N("W2", 8, "The 'done this week' check uses host-local wall time. An attempt that was stamped while the clock was ahead keeps the trim 'not due' until real time passes it.", "`AG/internal/fstrim/fstrim.go:179-182`"),
|
|
7: A("The operator, on the hub's System page (one cell). A failed trim is a WARN line on the host. Nobody is mailed.", "`HUB/web/system.go:45` `HUB/web/system.go:206` `AG/internal/fstrim/fstrim.go:318-319`"),
|
|
8: N("W2", 20, "TEST for the week anchor. The hour change and a clock jump: NOBODY KNOWS.")}, NH),
|
|
("CA12", "clock-agent", 11, "Weekly upload line test · CUSTOMER", "Interval", r"line.?test|linetest",
|
|
{6: N("W2", 8, "The check uses host-local wall time. A success that was stamped from the future blocks the tests until that time.", "`AG/internal/linetest/linetest.go:211-213`"),
|
|
7: A("The operator, on the hub's System page; the cell is judged on the last success, not on the attempt. Nobody is mailed.", "`AR/11-os-updates.md:409-411`")}, NH),
|
|
("CA13", "clock-agent", 12, "PBS datastore verify loop · CUSTOMER", "Interval", r"verify loop|PBS verify|verify",
|
|
{5: N("W3", 13, "The verify is not behind the heavy-work gate; a comment in the source says so. It can run beside a whole-guest backup."),
|
|
7: N("W1", 5, "A snapshot that fails the verify is an ERROR line on the host and a field in the report. No hub event or mail for it was found."),
|
|
8: N("W4", 15, "TEST for the record of a failed verify. A verify that takes longer than its 30-minute cadence: NOBODY KNOWS.")}, NH),
|
|
("CA14", "clock-agent", 13, "Janitor: leftover scratch guests and stale backup locks · CUSTOMER", "Interval", r"janitor|stale lock", {}, NH),
|
|
("CA15", "clock-agent", 14, "Guest network watchdog: DHCP client heal · CUSTOMER", "Watchdog", r"guest.?net|DHCP",
|
|
{3: N("W4", 15, "No time limit on one probe in the source, and no duration was measured."),
|
|
7: A("The operator, on the hub's Hosts page: the report carries the heal count for each guest. Nobody is mailed.", "`HUB/web/hosts.go:298-397` `AG/internal/guestnet/watchdog.go:166-170`")}, NH),
|
|
("CA16", "clock-agent", 15, "Host networking self-heal · CUSTOMER", "Watchdog", r"self.?heal", {}, NH),
|
|
("CA17", "clock-agent", 16, "Management-plane watchdog timer: keeps SSH reachable · OPERATOR", "Watchdog", r"mgmt.watchdog|management.plane",
|
|
{5: A("The script changes nothing when the directory is healthy. MEASURED: the timer is active on the three hosts and fires each 60 seconds. `Persistent=true` in the unit has no effect, because the timer has no calendar line.", "`AG/configs/felhom-mgmt-watchdog.sh:13-15` `AG/configs/felhom-mgmt-watchdog.timer:11-14` `EV/box/demo-hp-host-clock.txt`"),
|
|
8: A("SOURCE only. The script has no test file.", "`AG/configs/felhom-mgmt-watchdog.sh:1-68` `AG/configs/felhom-mgmt-watchdog.timer:3-6`")}, NH),
|
|
("CA18", "clock-agent", 17, "Agent service: restart, start limit, rollback · CUSTOMER", "Watchdog", r"Restart=|start limit|rollback",
|
|
{4: A("After the start limit the unit stays failed. Nothing on the host starts it again. The outside sign is the hub's 'host stale' alarm at 45 minutes.", "`AG/configs/felhom-agent.service:41-42` `AR/11-os-updates.md:572-573`"),
|
|
6: N("W5", 17, "The design text says `Restart=always` and a systemd watchdog that kills a hung agent. The unit has `Restart=on-failure` and no watchdog.", "`AR/03-host-agent.md:72` `AG/configs/felhom-agent.service:41-42`"),
|
|
8: N("W5", 17, "MEASURED values and a TEST for the rollback. A hung agent (alive, not working): NOBODY KNOWS; no watchdog restarts it.", "`AG/configs/felhom-agent.service:41-42`")},
|
|
"PARTLY — `RB/break-glass.md` names `reset-failed`; no runbook names the start limit of the agent"),
|
|
("CA19", "clock-agent", 18, "Off-site tunnel loop and the clock-ahead re-key · CUSTOMER", "Interval", r"wgtunnel|re.?key|tunnel",
|
|
{3: N("W4", 15, "No time limit on one pass in the source, and no duration was measured. Nothing grows with data.")}, NH),
|
|
("CA20", "clock-agent", 19, "Whole-guest backup cap: rolling windows · CUSTOMER", "Night", r"backup cap|rolling window|cap",
|
|
{6: N("W2", 8, "The windows use the wall clock. Successes that were stamped while the clock was ahead stay inside the window longer and can refuse real runs. NOBODY KNOWS.")}, NH),
|
|
("CA21", "clock-agent", 21, "Unsent OS-update report sender · OPERATOR", "Interval", r"unsent",
|
|
{7: A("The operator sees the result late: at most 5 minutes after the hub is back. The sender has no alarm of its own.")}, NH),
|
|
# ---------------------------------------------------------------- hub (source only)
|
|
("CH01", "clock-hub", 0, "Box liveness: 'your server cannot be reached' · CUSTOMER", "Hub sweep", r"host_stale|liveness|staleness", {}, NH),
|
|
("CH02", "clock-hub", 1, "Missed-backup deadline check (05:00) · CUSTOMER", "Hub daily", r"deadline check|backup.deadline|05:00",
|
|
{4: N("W1", 1, "When the hub is down at 05:00, that day's check is skipped. The 'last run' value lives in memory and the next slot is the next day. Nobody notices. The next day's check uses longer limits, so a missed backup is seen one day late.", "`E/hub/cmd/hub/main.go:1069-1112`"),
|
|
6: N("W2", 7, "The check compares the BOX's backup time with the hub's clock. A box whose clock runs ahead looks fresh. No guard in the file. (R-1206 is the same cause seen on the box's own pages.)", "`HUB/monitor/deadline.go:243`"),
|
|
8: N("W1", 1, "TEST for a down box and for 'the same slot never runs twice'. A skipped day: NOBODY KNOWS.")},
|
|
"NO — no runbook names a hub that was down at 05:00"),
|
|
("CH03", "clock-hub", 2, "Mail cooldowns and the weekly outage mail · CUSTOMER", "Hub sweep", r"cooldown",
|
|
{3: A("Each mail call has a 10-second limit. The cooldown lists are in memory; an entry is removed when its alarm clears.", "`HUB/notify/dispatcher.go:86` `HUB/notify/dispatcher.go:684` `HUB/notify/dispatcher.go:1182`"),
|
|
6: N("W2", 10, "Cooldowns use the hub's wall clock only. A jump back stretches a quiet period; a jump forward ends it early. Nothing handles it."),
|
|
7: N("W1", 2, "A swallowed operator alarm leaves a 'suppressed' row. A household mail that fails is logged and is not sent again; only operator mails are retried.", "`HUB/notify/dispatcher.go:1091-1094`")}, NH),
|
|
("CH04", "clock-hub", 3, "Retry of a failed operator mail · OPERATOR", "Hub sweep", r"retry",
|
|
{4: N("W1", 3, "The retry timers are in memory. A hub restart inside the 21 minutes loses them, and nothing sends the mail later. After the third failure the hub writes one ERROR line.", "`HUB/notify/dispatcher.go:90-102`"),
|
|
6: N("W2", 10, "The timers are durations; the hour change has no effect. A jump of the hub's wall clock: NOBODY KNOWS."),
|
|
7: N("W1", 3, "Nobody is mailed when the retries give up. The trace is the ERROR line and the 'failed' rows in the notification log.", "`HUB/notify/dispatcher.go:90-102` `AR/08-alarm-ladder.md:363`"),
|
|
8: N("W1", 3, "TEST for the retry. A restart during the retry window: NOBODY KNOWS.")}, NH),
|
|
("CH05", "clock-hub", 4, "Kernel mail: 'your box restarts tonight' · CUSTOMER", "Hub sweep", r"kernel",
|
|
{8: N("W3", 14, "TESTS for 'no mail, no step'. A mail that goes out late in the day against the box's own night slot: NOBODY KNOWS.")}, NH),
|
|
("CH06", "clock-hub", 5, "OS-update approval tick and the hourly OS alarms · OPERATOR", "Hub sweep", r"approval|ring",
|
|
{6: N("W2", 7, "A box's claimed reboot time is allowed 1 minute of error. A larger error of the box's clock: NOBODY KNOWS. (The N100's host is 32 s behind today.)"),
|
|
8: N("W4", 16, "TESTS and DOC. The cost of the pass each minute with 100 boxes: NOBODY KNOWS.")}, NH),
|
|
("CH07", "clock-hub", 6, "Off-site copy too old, and escrow never finished · CUSTOMER", "Hub sweep", r"offsite_stale|off.site.{0,20}stale|48 h",
|
|
{4: N("W1", 3, "With the hub down nothing is judged. After a start the state lists in memory are empty. If a tier that is still too old is mailed again after a restart: NOBODY KNOWS."),
|
|
6: N("W2", 7, "The last-success time is the BOX's time, compared with the hub's clock. A box clock that is ahead hides an old copy. A time that cannot be read counts as too old.", "`HUB/monitor/offsite.go:203-207`"),
|
|
8: N("W2", 7, "TESTS for the limits and for a newborn box. A box clock that is ahead: NOBODY KNOWS.")}, NH),
|
|
("CH08", "clock-hub", 7, "Restore-test proof too old · OPERATOR", "Hub sweep", r"restore.test",
|
|
{6: N("W2", 7, "First contact is the hub's receive time. The proof times come from the box's reports. An error of the box's clock: NOBODY KNOWS."),
|
|
8: N("W4", 16, "TEST that a restart does not mail again. The cost with 100 boxes (14 days of reports per customer, each minute): NOBODY KNOWS.")}, NH),
|
|
("CH09", "clock-hub", 8, "Off-site clean-up window (20 minutes, weekly) · CUSTOMER", "Hub sweep", r"clean.?up window|prune window|deleting key",
|
|
{4: N("W1", 1, "If the hub stops while a window is open, the key line that can delete stays on the storage account until the hub is back; the sweep then closes it. Nobody is told during the gap."),
|
|
6: N("W2", 10, "The window runs on the hub's clock; the box's clock does not decide. A hub clock that jumps back can keep a window open longer: NOBODY KNOWS.")}, NH),
|
|
("CH10", "clock-hub", 9, "Delayed deletion of a set-aside off-site copy · CUSTOMER", "Hub sweep", r"set.aside|delayed delet|superseded",
|
|
{3: N("W4", 15, "One remote delete for each due request, in the same sweep as the window. How long a large delete takes: NOBODY KNOWS."),
|
|
6: A("The hub's clock only. The length of the wait is one setting, read at the start.", "`E/hub/cmd/hub/main.go:598-605`")}, NH),
|
|
("CH11", "clock-hub", 10, "Off-site key audit (07:10) · OPERATOR", "Hub daily", r"key audit|07:10",
|
|
{3: N("W4", 16, "The audit has a 10-minute limit and reads one customer after the other from the storage account. With 100 boxes the 10 minutes may not be enough; nobody measured it."),
|
|
4: N("W1", 1, "When the hub is down at 07:10, that day's audit is skipped. No catch-up. Nobody notices.", "`E/hub/cmd/hub/main.go:1069-1112`"),
|
|
8: N("W4", 16, "TEST and DOC. The 10-minute limit at fleet size: NOBODY KNOWS.")}, NH),
|
|
("CH12", "clock-hub", 11, "Off-site credential stuck, and its self-heal · OPERATOR", "Hub sweep", r"offsiteheal|credential.{0,30}(stuck|heal)",
|
|
{8: A("SOURCE and DOC only. No test name for the timing was found.")}, NH),
|
|
("CH13", "clock-hub", 12, "PBS whole-box credential self-heal · OPERATOR", "Hub sweep", r"pbsdrheal|PBS.{0,30}heal",
|
|
{8: N("W4", 15, "SOURCE only. A pass that hangs on the remote side: NOBODY KNOWS; the pass has no time limit.")}, NH),
|
|
("CH14", "clock-hub", 13, "WireGuard peer list push to ep0 · OPERATOR", "Hub sweep", r"wgsync|peer list",
|
|
{4: N("W1", 5, "With the hub down ep0 keeps the last list. A push that fails is logged and tried again at the next tick. Nobody is mailed when it fails again and again; a new box then cannot reach the off-site store.", "`HUB/wgsync/reconciler.go:100-102`")}, NH),
|
|
("CH15", "clock-hub", 14, "Box clock ran ahead: the tunnel alarm · OPERATOR", "Hub sweep", r"wg_tunnel|stamped in the future|clock.ahead", {}, NH),
|
|
("CH16", "clock-hub", 15, "The daily scheduler and the hour change · OPERATOR", "Hub daily", r"scheduleDaily|nextDailyRun|D24",
|
|
{6: N("W2", 10, "Days of 23 and 25 hours are handled (D24, with tests). While the scheduler sleeps, a jump of the hub's clock does not move the timer: NOBODY KNOWS what then runs when.")}, NH),
|
|
("CH17", "clock-hub", 16, "Hub database snapshot (four slots) · OPERATOR", "Hub daily", r"snapshot",
|
|
{6: N("W5", 18, "The slots are safe on the hour-change night. The design text still says 'older than 24 h' and '02:30' in its first paragraph; the source has four slots.", "`AR/05-hub-architecture.md:618-620`")},
|
|
"YES — `RB/RUNBOOK-hub-db-offsite-backup.md`"),
|
|
("CH18", "clock-hub", 17, "Daily prune of old hub rows (04:30) · OPERATOR", "Hub daily", r"prune",
|
|
{3: N("W4", 15, "No time limit and no duration in the log. The deletes grow with the reports of each box each day."),
|
|
6: N("W2", 10, "One cut-off is made from the pod's local time and compared with a UTC column; the other uses UTC. The error is 1 or 2 hours on 90 days."),
|
|
8: A("SOURCE only. No test for the cut-off's time zone was found.")}, NH),
|
|
("CH19", "clock-hub", 18, "The other checks on the 60-second sweep · OPERATOR", "Hub sweep", r"60.second|sweep",
|
|
{3: N("W4", 15, "All checkers run one after the other in one loop, with no limit for each. A slow checker delays all the others, the liveness check too. Each reads the whole fleet at each pass.", "`E/hub/cmd/hub/main.go:935-962`"),
|
|
8: N("W4", 16, "SOURCE only. How long one whole sweep takes: NOBODY KNOWS.")}, NH),
|
|
("CH20", "clock-hub", 19, "Reminder about a held app · CUSTOMER", "Hub sweep", r"held.app|held app",
|
|
{6: N("W2", 7, "The reminder counts from a time that the box reports. A box clock that is behind makes the reminder early; one that is ahead makes it late.")}, NH),
|
|
("CH21", "clock-hub", 20, "'A golden is published but not vouched' · OPERATOR", "Hub sweep", r"vouch", {}, NH),
|
|
("CH22", "clock-hub", 21, "Shared storage boxes: fill and reachability · OPERATOR", "Hub sweep", r"storage box|pool.box|oversub", {}, NH),
|
|
("CH23", "clock-hub", 25, "DooPlex: push of the hub database snapshot off-site · OPERATOR", "DooPlex and ep0", r"hub.db|database snapshot",
|
|
{6: K(1293, NM + "The timer's zone and the hub's zone must match for the 30-minute offset. On the autumn hour-change day the 02:00 copy never reaches ep0 and the 20:00 copy is pushed twice."),
|
|
7: N("W5", 18, "The runbook says in one place that the alarm fires after 13 hours, and the alarm rule printed in the same runbook says 26 hours.", "`RB/RUNBOOK-hub-db-offsite-backup.md:100` `RB/RUNBOOK-hub-db-offsite-backup.md:152-153`"),
|
|
8: A(NM + "A test file for the script exists and is run by hand. The live alarm rule is in another repository and on DooPlex; this run did not read it.", "`RB/RUNBOOK-hub-db-offsite-backup.md:107-108` `E/scripts/hub-db-backup/test_hub_db_backup.py:309-316`")},
|
|
"YES — `RB/RUNBOOK-hub-db-offsite-backup.md`"),
|
|
("CH24", "clock-hub", 26, "DooPlex: Sunday restore tests · OPERATOR", "DooPlex and ep0", r"restore.test",
|
|
{5: N("W3", 14, NM + "Sunday 04:30 is between the 02:00 snapshot push and the 05:00 PBS pull. The script takes no lock against them."),
|
|
6: N("W2", 20, NM + "On the last Sunday of October the hour 02:00 to 03:00 comes twice. What the timer does at 04:30 that day: NOBODY KNOWS."),
|
|
8: A(NM + "DOC only. This run did not read if the alarm rule is deployed.", "`RB/RUNBOOK-hub-db-offsite-backup.md:152-161`")},
|
|
"YES — `RB/RUNBOOK-hub-db-offsite-backup.md`"),
|
|
("CH25", "clock-hub", 27, "DooPlex: nightly off-site push and the release copy · OPERATOR", "DooPlex and ep0", r"release.copy|release copy",
|
|
{3: N("W3", 14, NM + "The units have limits of 90 minutes and 2 hours. A 90-minute run from 00:20 ends at 01:50, after the 01:40 start of the next job. No lock between them."),
|
|
8: A(NM + "SOURCE only. The release-copy timer is switched on by hand; this run did not read if it is on.", "`E/scripts/release-copy/install.sh:4`")}, NH),
|
|
("CH26", "clock-hub", 28, "DooPlex and ep0: PBS copy, prune, verify, clean-up · OPERATOR", "DooPlex and ep0", r"ep0|PBS.{0,20}(prune|sync|pull)",
|
|
{3: A(NM + "The PBS jobs are in PBS's own config, not in the repository. They grow with the backup size of every box.", "`RB/ep0-datastore-copy.md:19`"),
|
|
5: N("W3", 14, NM + "The order is by the clock only: prune 07:30, clean-up 08:00. A pull that runs past 07:30 meets the prune: NOBODY KNOWS what then happens."),
|
|
6: N("W3", 14, NM + "Two machines, two clocks. 'The ep0 prune at 03:30 is before the 05:00 pull' is true only while both clocks and both zones agree. No document gives their zones."),
|
|
7: N("W1", 1, NM + "PBS has its own notification target. The clean-up unit has no failure hook and no time limit; a clean-up that fails tells nobody.", "`E/scripts/ep0-copy-gc/felhom-ep0-copy-gc.service:9` `RB/ep0-datastore-copy.md:19`")},
|
|
"YES — `RB/ep0-datastore-copy.md`"),
|
|
("CH27", "clock-hub", 29, "Nightly CI: the gates and the security watch · OPERATOR", "DooPlex and ep0", r"nightly|security.watch|cron",
|
|
{4: N("W1", 1, NM + "A missed night is not made up. The watch's weekly summary repairs itself: it is sent when the last one is 8 or more days old. A runner that drops the job tells nobody."),
|
|
8: N("W1", 1, NM + "DOC. A night with no run at all: NOBODY KNOWS who notices.", "`AR/08-alarm-ladder.md:708-722`")}, NH),
|
|
]
|
|
|
|
M = {i: dict(src=(h, n), name=name, seed=seed, named=("grep", rx), o=o, rb=rb) for i, h, n, name, seed, rx, o, rb in _M}
|
|
|
|
# ------------------------------------------------------------------ second pass
|
|
# Helper answers that still said "not found" or "candidate" after the first build: the main session's words in their place.
|
|
M["CC20"]["o"][2] = A("The sweep runs each 6 hours; the release each hour. A restore copy that nobody used goes after 30 days (the constant, D62). A comment in the wiring still says 14 days; the comment is stale.", "`CT/cmd/controller/main.go:1412`")
|
|
M["CH01"]["o"][3] = N("W4", 16, "No time limit. The checkers run one after the other in one loop: one read of all hosts at each pass, then a loop over the hosts. The time grows with the number of boxes. Nobody measured one pass.", "`E/hub/cmd/hub/main.go:935-962` `HUB/monitor/host_staleness.go:113`")
|
|
M["CH01"]["o"][4] = A("With the hub down nothing is judged. At the start, a box that is already silent gets one alarm at the first pass, unless that alarm was raised before. The controller's checker starts silent for such a box.", "`HUB/monitor/host_staleness.go:59-68` `HUB/monitor/staleness.go:86-91`")
|
|
M["CH02"]["o"][3] = N("W4", 16, "No time limit; the check runs inside the scheduler's own loop. It reads up to 7 days of evidence for every customer. The time grows with boxes times reports. Nobody measured it.", "`E/hub/cmd/hub/main.go:1090`")
|
|
M["CH08"]["o"][3] = N("W4", 16, "No time limit. Each pass reads 14 days of host reports for each customer, each minute. It is the heaviest read on the sweep. Nobody measured it.", "`HUB/monitor/restoretest.go:79` `HUB/monitor/restoretest.go:252`")
|
|
M["CH16"]["o"][2] = A("Budapest wall-clock time. If the zone cannot be loaded, the fallback is fixed CET, which is one hour off in summer. The pod's zone is set to Europe/Budapest.", "`E/hub/cmd/hub/main.go:1072`")
|
|
M["CH16"]["o"][4] = N("W1", 1, "A slot that is missed while the hub is down is not made up; the snapshot and the prune are the two exceptions. A time setting that cannot be read becomes 00:00 with no message.", "`E/hub/cmd/hub/main.go:1069-1112`")
|
|
M["CH23"]["o"][2] = A(NM + "02:30, 08:30, 14:30 and 20:30 with up to 2 minutes of random delay, in the host's local time (the unit names no zone).")
|
|
|
|
NIGHT = "`EV/box/demo-hp-guest-night.txt` `EV/box/tester-1-guest-night.txt` `EV/box/demo-felhom-guest-night.txt`"
|
|
MEASURED = {
|
|
"CC01": ("MEASURED 2026-11-11, one night, small stores: 2 min 33 s · 40 s · 18 s (demo-hp · Tester 1 · N100)", NIGHT),
|
|
"CC02": ("MEASURED 2026-11-11: 39 s · 15 s · 11 s", NIGHT),
|
|
"CC03": ("MEASURED 2026-11-11: 3 min 09 s · 59 s · 38 s", NIGHT),
|
|
"CC08": ("MEASURED 2026-11-11: under 1 s on each box", NIGHT),
|
|
"CC09": ("MEASURED 2026-11-11: 5 s · 5 s · 3 s", NIGHT),
|
|
"CC19": ("MEASURED 2026-11-11: the slot ran on each box", NIGHT),
|
|
"CA04": ("MEASURED 2026-11-11: the hourly timer is active on the three hosts", "`EV/box/demo-hp-host-clock.txt` `EV/box/tester-1-host-clock.txt` `EV/box/demo-felhom-host-clock.txt`"),
|
|
}
|
|
|
|
TINY = [
|
|
"session clean-up of dashboard sign-ins on the box (a new sign-in repairs it)",
|
|
"old image clean-up after a controller start (log only; deletes nothing when it cannot read its keep list)",
|
|
"operator restart and reboot limits on the agent (one hour; an operator action)",
|
|
"the hub's purges of log bundles, access log and sessions (nobody is told, nothing depends on them)",
|
|
"the hub's version check and catalog template fetch (a WARN line and a stored error; a page shows it)",
|
|
"the guest swap check, once after an agent start (2-minute limit)",
|
|
"the 14-day hold of alarms after a deliberate power-off (a window, not a job)",
|
|
]
|
|
LEFT_OUT = [
|
|
("The agent's local-API certificate (10 years)", "it is an end date: the End-dates list, DB03"),
|
|
("Links and tokens that end by themselves", "they are end dates: the End-dates list, DH22 and the by-ruling list there"),
|
|
("The 14-day countdown after a household stops the off-site copy", "the End-dates list, DB04; its daily sweep is CC19 here"),
|
|
("Debian's and Proxmox's own timers on a host (apt-daily, pve-daily-update, logrotate, fstrim.timer, man-db)", "they are the OS's jobs, not ours; MEASURED present on the three hosts, `evidence/box/*-host-clock.txt`"),
|
|
("TEST-ONLY jobs", "none was found: every job in the table runs on a household's box or for the operator"),
|
|
]
|