From 1550b1da5aec9b86d77659e93932a9c55f4fa59e Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 26 Oct 2026 08:23:31 +0100 Subject: [PATCH] =?UTF-8?q?R-1183=20CLOSED=20(measured=202026-10-26,=20two?= =?UTF-8?q?=20channels:=20the=20hub=20log=20and=20DooPlex's=20push=20journ?= =?UTF-8?q?al=20=E2=80=94=20each=20daily=20job=20once,=20on=20its=20CET=20?= =?UTF-8?q?hour;=20the=20morning=20of=2010-25's=20hub=20log=20is=20gone,?= =?UTF-8?q?=20named).=20R-1293=20filed=20(P4):=20on=20the=20fall-back=20ni?= =?UTF-8?q?ght=20the=2002:30=20push=20runs=20before=20the=20hub's=20second?= =?UTF-8?q?=2002:00,=20so=20that=20copy=20never=20reaches=20ep0;=20dated?= =?UTF-8?q?=20check=202027-03-29.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- STATUS.md | 4 +- .../r1183-measurement-2026-10-26.txt | 55 +++++++++++++++++++ documentation/backlog/CLOSED-ITEMS.md | 6 ++ documentation/backlog/OPEN-ITEMS.md | 7 +-- 4 files changed, 66 insertions(+), 6 deletions(-) create mode 100644 documentation/audits/night-2026-10-25/r1183-measurement-2026-10-26.txt diff --git a/STATUS.md b/STATUS.md index 7d53d34d..2e7d16b6 100644 --- a/STATUS.md +++ b/STATUS.md @@ -23,7 +23,7 @@ - DooPlex: something sets the shared checkout to „bare" (R-935). ## What we work on -- Dated checks, run by CC. 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the qBittorrent line test on a seeding box, with DooPlex's qBittorrent as the peer (R-1250, your yes); the weekly clean-up window (R-435; the logs of the last one were lost to restarts). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes). +- Dated checks, run by CC. 2026-10-27: the qBittorrent line test on a seeding box, with DooPlex's qBittorrent as the peer (R-1250, your yes); the weekly clean-up window (R-435; the logs of the last one were lost to restarts). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes). ## Waiting on you — decide @@ -47,7 +47,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te 6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14. 7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12. -**Monday 2026-10-26 (attended; each a STOP).** 1. Hub v0.171.0 (R-1287, R-1289). 2. Agent v0.166.0 + your vouch (R-1280, R-1285). 3. Controller v0.344.0 — before any household gets agent 0.165.0 (R-1261, R-1281, R-1288, R-1282, R-1286, R-1278). 4. Catalog: MeTube (R-1283). 5. Dated: R-1183 today, R-1250 Tuesday. 6. Hub Hosts page: delete the dead `Tester4-84bbe7`. If not: the fixes stay unreleased; the host alarms. +**Monday 2026-10-26 (attended; each a STOP).** 1. Hub v0.171.0 (R-1287, R-1289). 2. Agent v0.166.0 + your vouch (R-1280, R-1285). 3. Controller v0.344.0 — before any household gets agent 0.165.0 (R-1261, R-1281, R-1288, R-1282, R-1286, R-1278). 4. Catalog: MeTube (R-1283). 5. Dated: R-1250 Tuesday. 6. Hub Hosts page: delete the dead `Tester4-84bbe7`. If not: the fixes stay unreleased; the host alarms. **2026-10-23 do list.** 1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens. 2. On the hub's Hosts page: Discard appliance 49. If not: a dead entry alarms. diff --git a/documentation/audits/night-2026-10-25/r1183-measurement-2026-10-26.txt b/documentation/audits/night-2026-10-25/r1183-measurement-2026-10-26.txt new file mode 100644 index 00000000..cfbf4fb6 --- /dev/null +++ b/documentation/audits/night-2026-10-25/r1183-measurement-2026-10-26.txt @@ -0,0 +1,55 @@ +# R-1183 measurement, 2026-10-26 ~07:20Z (CC, operator attended). Read only. + +## 1. hub pod log (channel A): pod start + every daily-job line (host reports dropped) +pod=hub-76cb98d46c-xmm64 started=2026-10-25T15:50:49Z restarts=0 image=gitea.dooplex.hu/admin/felhom-hub:0.170.0 +2026/10/25 16:50:51 [INFO] felhom-hub 0.170.0 starting +2026/10/25 16:50:53 [INFO] offsite-key-audit: next run at 2026-10-26 07:10 CET (in 14h19m7s) +2026/10/25 16:51:26 [INFO] Pruned 204 old report rows +2026/10/25 16:51:27 [INFO] Pruned 7 old event rows +2026/10/25 16:51:28 [INFO] Pruned 355 old app telemetry rows +2026/10/25 16:51:32 [INFO] Pruned 2 stale app issues +2026/10/25 16:51:32 [INFO] prune: next run at 2026-10-26 04:30 CET (in 11h38m28s) +2026/10/25 16:51:34 [INFO] db-snapshot 08:00: next run at 2026-10-26 08:00 CET (in 15h8m26s) +2026/10/25 16:51:34 [INFO] db-snapshot 02:00: next run at 2026-10-26 02:00 CET (in 9h8m26s) +2026/10/25 16:51:34 [INFO] db-snapshot 14:00: next run at 2026-10-26 14:00 CET (in 21h8m26s) +2026/10/25 16:51:34 [INFO] deadline-check: next run at 2026-10-26 05:00 CET (in 12h8m26s) +2026/10/25 16:51:34 [INFO] db-snapshot 20:00: next run at 2026-10-25 20:00 CET (in 3h8m26s) +2026/10/25 20:03:19 [INFO] db snapshot written: hub-20261025T190000Z.db (543191040 bytes, 3m19.044s); pruned 1, keeping 2 +2026/10/25 20:03:19 [INFO] db-snapshot 20:00: next run at 2026-10-26 20:00 CET (in 23h56m41s) +2026/10/26 02:02:38 [INFO] db snapshot written: hub-20261026T010000Z.db (546148352 bytes, 2m38.714s); pruned 1, keeping 2 +2026/10/26 02:02:38 [INFO] db-snapshot 02:00: next run at 2026-10-27 02:00 CET (in 23h57m21s) +2026/10/26 04:16:00 [INFO] offsitekeys: clean-up window 14 CLOSED for demo-felhom: outcome=pruned, 63 -> 48 (drop 15, allowed 31) +2026/10/26 04:16:06 [INFO] offsitekeys: clean-up window 15 CLOSED for tester-1: outcome=pruned, 78 -> 57 (drop 21, allowed 39) +2026/10/26 04:30:37 [INFO] Pruned 187 old report rows +2026/10/26 04:30:38 [INFO] Pruned 43 old event rows +2026/10/26 04:30:39 [INFO] Pruned 325 old app telemetry rows +2026/10/26 04:30:43 [INFO] Pruned 19 stale app issues +2026/10/26 04:30:43 [INFO] prune: next run at 2026-10-27 04:30 CET (in 23h59m16s) +2026/10/26 05:00:00 [INFO] Deadline check: Tester-2 is DOWN — not judged yet: first host report unknown ago (under 48h0m0s, nothing expected yet) +2026/10/26 05:00:00 [INFO] Deadline check: Tester4 is DOWN — not judged yet: first host report 8h35m0s ago (under 48h0m0s, nothing expected yet) +2026/10/26 05:00:00 [INFO] Deadline check: 5 customers, 0 backup missed, 0 backup unknown (deferred), 0 dbdump missed, 2 skipped (down), 0 unknown (no host ever bound) +2026/10/26 05:00:00 [INFO] deadline-check: next run at 2026-10-27 05:00 CET (in 23h59m59s) +2026/10/26 07:10:09 [INFO] off-site key check: 4 sub-account(s) read in 9.289s +2026/10/26 07:10:09 [INFO] offsite-key-audit: next run at 2026-10-27 07:10 CET (in 23h59m51s) +2026/10/26 08:02:30 [INFO] db snapshot written: hub-20261026T070000Z.db (546680832 bytes, 2m30.132s); pruned 1, keeping 2 +2026/10/26 08:02:30 [INFO] db-snapshot 08:00: next run at 2026-10-27 08:00 CET (in 23h57m30s) + +## 2. /data/snapshots now (keep 2) +-rw------- 1 root root 546148352 Oct 26 02:02 hub-20261026T010000Z.db +-rw------- 1 root root 546680832 Oct 26 08:02 hub-20261026T070000Z.db + +## 3. DooPlex push service journal (channel B, a different process and clock source): which snapshot each push found +Oct 25 02:31:27 dooplex felhom-hub-db-backup[718752]: felhom-hub-db-backup: snapshot hub-20261024T180000Z.db, 391 min old +Oct 25 02:31:44 dooplex felhom-hub-db-backup[718752]: felhom-hub-db-backup: pushed hub-20261024T180000Z.db to ep0 (ns operator) in 2 s +Oct 25 08:30:27 dooplex felhom-hub-db-backup[417378]: felhom-hub-db-backup: snapshot hub-20261025T070000Z.db, 30 min old +Oct 25 08:30:43 dooplex felhom-hub-db-backup[417378]: felhom-hub-db-backup: pushed hub-20261025T070000Z.db to ep0 (ns operator) in 3 s +Oct 25 14:31:29 dooplex felhom-hub-db-backup[4047100]: felhom-hub-db-backup: snapshot hub-20261025T130000Z.db, 31 min old +Oct 25 14:31:45 dooplex felhom-hub-db-backup[4047100]: felhom-hub-db-backup: pushed hub-20261025T130000Z.db to ep0 (ns operator) in 3 s +Oct 25 20:30:32 dooplex felhom-hub-db-backup[3654710]: felhom-hub-db-backup: snapshot hub-20261025T190000Z.db, 30 min old +Oct 25 20:30:52 dooplex felhom-hub-db-backup[3654710]: felhom-hub-db-backup: pushed hub-20261025T190000Z.db to ep0 (ns operator) in 3 s +Oct 26 02:31:49 dooplex felhom-hub-db-backup[3240669]: felhom-hub-db-backup: snapshot hub-20261026T010000Z.db, 31 min old +Oct 26 02:32:08 dooplex felhom-hub-db-backup[3240669]: felhom-hub-db-backup: pushed hub-20261026T010000Z.db to ep0 (ns operator) in 4 s + +## 4. Go's choice for the ambiguous 02:00 (the hub's nextDailyRun uses time.Date) +time.Date(2026-10-25 02:00 Budapest) = 2026-10-25T02:00:00+01:00 = 01:00Z +time.Date(2026-10-25 08:00 Budapest) = 2026-10-25T08:00:00+01:00 = 07:00Z diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index aeb117d4..97162019 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -26,6 +26,12 @@ --- +## 2026-10-26 — R-1183 measured (full text of the row: `git show b00d5c61:documentation/backlog/OPEN-ITEMS.md`) + +| Row | What | Closed | Evidence | +|---|---|---|---| +| **R-1183** | **On the day the clocks go back, every hub daily job after 02:00 ran twice** (P3) | CLOSED 2026-10-26 — hub v0.163.0's calendar-day scheduler held across 2026-10-25. Two channels: the DooPlex push journal (a different process) found the 08:00, 14:00 and 20:00 snapshots at 07:00Z, 13:00Z and 19:00Z — each on its CET hour, the 08:00 one scheduled before the change and run after it; no 02:00 CEST (00:00Z) copy existed at 00:31Z, so the 02:00 slot did not run twice. The hub log (pod since 2026-10-25 15:50Z) shows each of the four jobs once on 2026-10-26 at its CET hour and every next run one calendar day on. **Not seen:** prune, deadline-check and the key audit on the morning of 2026-10-25 — that log ended with the 0.170.0 restart; they share the same scheduler function (`nextDailyRun`) and its `TestR1183_*`. Found on the way: R-1293 (the 02:30 push runs before the second 02:00). | `audits/night-2026-10-25/r1183-measurement-2026-10-26.txt` | + ## 2026-10-25 (evening) — installer 1.36.1 published; D50 (full text of the rows: `git show b8411926:documentation/backlog/OPEN-ITEMS.md`) | Row | What | Closed | Evidence | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 831e8c91..7fa1051e 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -316,7 +316,7 @@ stopping line that lies. | **R-1291** | Monitoring & notifications | P3 | **A held app gives the operator no lasting signal: one event at the stop, then nothing — demo-hp's Kimai sat held 11 days unseen.** Found 2026-10-25 night (R-1286). No held-app state reaches the hub (the report carries none; the hub shows none); the crash-loop stop sends `app_stopped_unhealthy` once. On a household box an app held after a crash loop, a failed update or a failed restore stays stopped until someone opens its page. | OPEN | — | a two-repo design: the controller reports its holds (stack, reason, since), the hub shows them on the host and mails a reminder after N days (the D-level of the reason); one row's worth of design first | CC | -## Hub & operator — 13 rows (P3 5, P4 8) +## Hub & operator — 13 rows (P3 4, P4 9) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -324,7 +324,6 @@ stopping line that lies. | **R-31** | Hub & operator | P3 | **[P2-HIGH] Offsite provisioning is synchronous with no status affordance.** Save runs the Hetzner sync in-request, so the request can hit the nginx 504 **while succeeding server-side**: the operator cannot tell failed from slow, and a retry races the first attempt. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **Re-ranked 2026-10-03: P2→P3: operator-only; a known workaround (click once, wait, verify) exists.** **2026-10-06 night: the race half fixed on felhom.eu main (hub, unreleased):** a second Save while the first still provisions is refused with 409 („already running — wait about a minute, then reload; do not save again"); nothing saved, nothing created; per customer, in memory. `TestProvision_R31_*`, red-proof `audits/night-burndown-2026-10-06/hub/R-31-red.txt`. LEFT: the async save with a status card (the escrow-card idiom). | — | Direction: make it async + a status card, reusing the proven **awaiting-card/poll idiom** (v0.138.0 escrow card). **Interim mitigation belongs in R-3 as an operator note: click once, wait, verify — do not re-click.** | CC | | **R-882** | Hub & operator | P3 | **Longhorn on DooPlex could not grow a volume online: its `instance-manager` (116 days up) called a host process that no longer existed** — `nsenter: cannot open /host/proc/196610/ns/mnt` on every expansion retry, and an offline growth was blocked by the expansion's own attachment ticket (found 2026-10-05 growing `hub-data` to 2 Gi). A restart of the instance-manager (operator-approved) fixed it: 77/77 volumes back `attached/healthy` in 110 s. **Why the cached PID went stale was not established** (likely a containerd/k3s or iscsid restart after the instance-manager started), so it will recur after the next such restart and stay invisible until a volume needs to grow. `audits/hub-db-offsite-2026-10-05/partA/step1-*.txt` | **OPEN** | — | Find which host process the PID was and whether Longhorn 1.10.x re-resolves it; until then, before growing any volume, check the instance-manager's age against the last k3s/containerd restart | operator | | **R-883** | Hub & operator | P3 | **8 DooPlex workloads run an image by a moving tag (`:latest` or none), so any pod restart is a silent upgrade.** Measured 2026-10-05: the Longhorn restart restarted zipline on `ghcr.io/diced/zipline:latest` (pull Always), which pulled 4.8.0; 4.8.0 refused its database (`cannot safely migrate from prisma to drizzle: expected migration 20260508022000 … was not applied`) and crash-looped. Fixed for zipline by pinning `4.7.0` (homelab-manifests `90f60e4`, `4c8ec7a`; 4.7.0 applied the four missing migrations; a dump from before is kept out-of-band). The other 7 were counted, not named or changed. **Checked from source 2026-10-05 (burn-down round 2):** homelab-manifests@87dfc29 still has moving tags (grep image lines without a numeric tag): admin-system/toolbox.yaml:12 nicolaka/netshoot:latest (a bare Pod); calibre-system/cwa.yaml:826 calibre-web-automated:dev; outline-system/outline.yaml:270 minio/minio:latest; tandoor-system/recipe-importer.yaml:26 gitea.dooplex.hu/admin/recipe-importer:latest (pull Always :27); adventurelog-system/adventurelog.yaml:100 and :256 adventurelog-backend/frontend:latest (pull Always :101,:257); jarrs-system/jarr- | **OPEN** | — | List the 7 (`kubectl get deploy,sts -A` images without a fixed tag), pin each to the running version, and let Renovate move them | operator | -| **R-1183** | Hub & operator | P3 | **On Sunday 2026-10-25 (the clocks go back) every hub daily job after 02:00 runs TWICE: the alarm check at 04:00 and 05:00, the hub database snapshot twice (keep 2 → the previous day's copy pruned a day early), the prune and the key audit twice.** `hub/cmd/hub/main.go:1057-1062` adds 24 h to an absolute time; the controller fixed this (`scheduler.go:511-516`), the hub did not. PROVEN by replaying the hub's exact arithmetic (`audits/dataloss-hunt-2026-10-20/C3-T9-hub-dst/replay.txt`). On 2027-03-28 each runs once, an hour late. No architecture document mentions DST. Data-loss hunt 2026-10-20 (T9). | VERIFY **FIXED in hub v0.163.0, deployed 2026-10-21 11:46 UTC with the operator (D24 = yes):** `scheduleDaily` takes the calendar day after (`nextDailyRun`, `time.Date` day+1) and never repeats the slot that ran last; `TestR1183_*` for 2026-10-25 and 2027-03-28, red-proved against the old arithmetic (`04:00 job ran 2 time(s) on 2026-10-25`; the 02:00 snapshot also ran at 02:00 CEST and 02:00 CET). No other hub scheduler computes a wall-clock time. `05` §16.5 **2026-10-25 (partial read, the attended release run):** `/data/snapshots` holds `hub-20261025T070000Z.db` and `hub-20261025T130000Z.db` — one file per slot, each on its hour (keep 2, so the 02:00 slot is pruned); the hub restarted at 16:51 (the 0.170.0 deploy), so the day's earlier log is gone. The full read stays on 2026-10-26 (the 20:00 slot and the log from 16:51). | Sunday 2026-10-25 has to happen | On 2026-10-26 read the hub log: every daily job (db-snapshot, prune, deadline-check, offsite-key-audit) logged ONE run on 2026-10-25, and `/data/snapshots` holds one file for that day; then close | CC | | **R-264** | Hub & operator | P4 | **Twenty-one facts the boxes report that the hub can now decode nowhere, each allowlisted with a reason rather than silently skipped — and for these the reason is "no consumer today, and one is arguably owed".** Split out of R-260 on 2026-08-08 so that closing the CLASS (gated) and fixing its sharpest instance (`operator_key_configured`) could not be mistaken for having decided what the hub should do with the rest. **The list, grouped by what a consumer would be for.** **(a) Guest-network health — `guest_net` and its seven children** (`checked_at`, `has_route`, `dhclient_alive`, `heal_succeeded`, `heals_last_hour`, `last_heal_at`, `damped`). The R-54 watchdog reports per-guest network state and self-heal counts every cycle and the hub — the component that emails the operator — models none of it. There is a live incident in this project's own record where a killed `dhclient` took a tunnel down for 1 h 15 m (`audits/INCIDENT-guest-dhclient-killed-2026-07-20.md`); a recurring-heal signal is exactly what would have surfaced it. **This is the strongest candidate of the twenty-one.** **(b) `selfupdate_pending` + `selfupdate_pending_version`** — an agent that has flipped its binary and never committed reports pending on every heartbeat so that "the operator sees WHY the version isn't advancing", and no operator can see it. **(c) `mgmt_plane.healed_recently`** — bounded: the hub DOES alarm on the `privsep_healed_at` timestamp beside it, so the recurring-clobber signal is not lost, only this flag. **(d) `restore_tests.mount_parity` + `mount_inventory`** — R-262's subject; the verdict is not lost (a mismatch fails before `Pass` is set) but the hub cannot tell a full-fidelity pass from a boot-only one. **(e) `pbs_dr.applied_at`.** **(f) Controller-side: `config_hash`, `reporting_disabled`, `stacks`, `storage.migrated_to`, `backup.last_db_dump`, `backup.last_integrity_check`** — the last two are backup-integrity timestamps, which is the "presence is not success" neighbourhood. **For each the question is the same and is NOT answered here: is it wanted? If the hub should act on it, model it and name what consults it. If it should not, the honest end is that the emitter stops sending it** — a fact emitted forever and consumed nowhere is a future false green waiting for someone to write a check against it. **Deliberately not decided in the G-1 session**, whose scope was the gate plus the operator-access instance; unilaterally removing emitters would also break the byte-identical cross-repo host-report golden and is a coordinated two-repo change. **⚠ DISPOSITIONS RECORDED 2026-08-13 — and the first thing to say is that they were NOT in this register before today.** The rulings were made on 2026-08-12; this row still read **READY — owner Viktor** and the gate's twenty entries still all said *"arguably owed"*, so a session told to *"re-read the dispositions from the register"* would have found none. They are written down now, which is the point of writing them down. **THE COUNT WAS ALSO WRONG:** this row says *twenty-one*; the gate's allowlist held **twenty**, measured. Twenty is the number the dispositions below account for, exactly. **(1) BUILD A READER — four groups, fourteen facts.** (a) guest-network health, (b) the staged-update-pending pair, (d) the restore-test depth pair, (f-part) the two backup-integrity timestamps. **(2) NO READER WANTED — five facts, now recorded as `not consumed, DELIBERATELY` with the ruling and its date** in `scripts/wire_contract_gate.py`, each with its own reason rather than a bare refusal: `mgmt_plane.healed_recently` (the hub already alarms on the timestamp beside it), `pbs_dr.applied_at` (`pbs_dr.state` is the verdict; the timestamp alone is the attempt-read-as-result trap), `config_hash` (the hub authors the config and knows its own generation), `stacks` (the app view is built from the purpose-built `app_telemetry` wire), `storage.migrated_to` (box-local bookkeeping with no hub-side intent to reconcile against). **The emitters are deliberately left alone** — removing one is a coordinated two-repo change and breaks the host-report golden; the honest end here is a recorded decision, not a deletion. The gate grew a THIRD entry kind to carry them, because an undecided fact and a decided one must not read alike. **(3) `reporting_disabled`, decided on its own merits: RECLASSIFIED `redundant`** — `health.status = "disabled"` travels in the same minimal report, is decoded into `reports.health_status`, and IS rendered. **The decision surfaced a real defect the flag would not have fixed: the staleness checker is age-only, so a deliberately-silent box still alarms → R-321.** **PROGRESS, 2026-08-13: the first reader is BUILT — guest-network health, R-319.** Its eight allowlist entries are **removed** (an allowlisted tag is skipped, so leaving them would have meant the new reader's fields were never checked); the gate's checked-tag count rose **182 → 190** and skipped fell **88 → 80**, which is the positive control that the wiring is real. **WHERE THE TWENTY NOW STAND: 8 read · 5 deliberately unread · 1 redundant · 6 still owed a reader** (`selfupdate_pending`, `selfupdate_pending_version`, `restore_tests.mount_parity`, `restore_tests.mount_inventory`, `backup.last_db_dump`, `backup.last_integrity_check`) — counts measured from the allowlist, not estimated. **Only ONE reader was built on purpose:** four at once is a design session pretending to be an implementation, and this one now tells us what the other three cost | **READY** — operator ruling 2026-10-12 (S7, `09` §3 decision 202): CC owns the six owed readers and builds them when the queue is idle, one group per session, attended hub deploys. **OPEN — 6 of 20 still owed a reader; dispositions recorded 2026-08-13, first reader shipped (R-319)** — owner Viktor | — | Last in the 2026-10-12 arc | CC | | **R-451** | Hub & operator | P4 | **[P3-LOW] UPDATE ARC SLICE 7 — a fleet sweep: the operator can SEE, and MOVE, how far behind every box is.** Slices 1 and 2 make one box's state visible on that box's own pages. The operator has no fleet view, and **it is not derivable from what is already reported: the hub's report payload carries container name, state, CPU and memory, and NO image field at all** (spike §5, which is why Peti's box could only be recorded UNKNOWN). So this is a hub-side change as well as a controller one. Rank LOW today because the fleet is two enrolled boxes; it rises with the fleet. `architecture/09-update-architecture.md` §6, §8.4 **BOTH SIDES VERIFIED 2026-09-21, and it is cheaper than this row implies.** The controller's payload carries no image (`internal/report/types.go` L98–103) and the hub's `Store.SaveReport` (`hub/internal/store/store.go:965`) denormalises only container **counts** — but **the hub stores the raw report JSON whole**, so a new controller field lands there the day it is sent. What is missing is the denormalisation and the page, not the transport. Shape in `09` §6.2–6.3; the payload question is `09` §3b **Q7**. **-- RULED 2026-09-23 (`09` §3 decision 18):** the report carries, per compose service (database included), the installed reference, the catalog reference and the badge state. **Built later, when the fleet grows** — Q7's recommendation, confirmed. | **DEFERRED** (2026-10-03 triage: the row's verdict was finished, but it names open work no other row carries — the fleet sweep (report field, hub denormalisation, fleet page) is ruled and not built — deferred until the fleet grows (`09` §6.3)) — **RULED — build deferred until the fleet grows; owner: CC** | — | — | CC | | **R-814** | Hub & operator | P4 | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | **VERIFY** (2026-10-03 triage: a July watch row with no id; given R-814. WAITING-ON-OPERATOR — no record found that the box was deleted.) — WAITING-ON-OPERATOR | operator console | Delete the box | operator | @@ -333,7 +332,7 @@ stopping line that lies. | **R-1056** | Hub & operator | P4 | **Five operator routes fit no token group, so they are session only until the operator places them:** apps reset-telemetry and dismiss-issues, the Configuration asset re-seed, the off-site WireGuard endpoint save, the off-site abandon-cancel (`05` §17.3). Nothing needs them by token today. | WAITING-ON-OPERATOR — filed 2026-10-15 by the token build | operator decision | Place each in a group, or leave session only | operator | | **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator | | **R-1289** | Hub & operator | P4 | **The hub's „newest controller the fleet reported" can be a dev build that no box runs: `semver.Compare` returns 0 (equal) for any version with a suffix, so „0.343.0-dev.r1251a" — once read first — beats nothing and is never beaten.** Seen 2026-10-25 in the vouch line (attended release run): „golden 0.331.0 unchanged (behind the fleet's 0.343.0-dev.r1251a …)" while the four hosts run 0.343.0 / 0.342.0. `store.NewestReportedControllerVersion` scans every report ever stored (`hub/internal/store/store.go:2210`), and `hub/internal/semver/semver.go:14` gives up on `0-dev…` / `0-rc.N`. Used by the R-120 golden gate (`hub/internal/web/configs.go:1548,1554`): with a suffixed „newest" the gate compares against whatever row came first — today it only words a log line and fails open. | **VERIFY** — fixed on main 2026-10-25 night, waits for the hub release (v0.171.0): suffixed versions are skipped (`TestR1289_ATestBuildIsNeverTheFleetsNewest`, red-proved) | — | skip suffixed versions in `NewestReportedControllerVersion` (or compare with the pre-release rule the controller uses, `util.ParseVersionPre`) + a test with a dev and an rc row ahead of a release row (red first); one hub release | CC | - +| **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC | ## Business & legal — 14 rows (P2 4, P3 2, P4 8) @@ -400,8 +399,8 @@ stopping line that lies. | item | due (UTC) | what to measure | |---|---|---| | R-435 | 2026-10-27 | Tester 1's weekly off-site clean-up ran inside a hub window with no drop alarm (box log + mailbox); detail in the R-435 row | -| R-1183 | 2026-10-26 | every hub daily job ran ONCE on 2026-10-25 (hub log + one snapshot file per slot — four slots since hub 0.167.0, D34, the 02:00 one not twice); detail in the R-1183 row | | R-1250 | 2026-10-27 | the released line hold, live: a seeding qBittorrent on demo-hp (peer DooPlex's, operator yes 2026-10-25) through the host's due window; the line value and the limit back; detail in the R-1250 row | +| R-1293 | 2027-03-29 | the spring clock change: the hub's 02:00 snapshot and the 02:30 push on 2027-03-28 (02:00–03:00 does not exist) — which file each push took; detail in the R-1293 row | | R-1228 | 2026-10-30 | D35 (yes, after the release week): start moving the website + installer to Cloudflare; detail in the R-1228 row | | R-1174 | 2026-10-27 | the status gate refuses from today: one conviction seen exiting 1 (a hand run on a copy of the page is enough); detail in the R-1174 row | | R-1050 | 2026-11-15 | HSTS one year if no https trouble since 2026-10-15; detail in the R-1050 row |