hub (unreleased): the operator is told when a box's clock is more than one minute from the hub's (R-1384, D79)
gates / gates (push) Successful in 2m11s

The host report's new sent_at (the box clock at the moment of sending) is compared with the hub's clock at arrival.
Both directions; told once; clears within 30 s. A report that waited on the box raises nothing (reported_at is not
read). A second box off in the same direction points at the hub's own clock. Operator-only. Not deployed.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-14 17:05:40 +01:00
1 parent 26f4f7660c
commit 88e636445a
9 files changed
+318 -2

No files matched your search

+1 -1
View File
@@ -294,7 +294,7 @@ stopping line that lies.
| **R-1358** | Box system & updates | P3 | **cloudflared 2026.10.4 misses two dependency fixes (VULN-183456 chi, 2026.10.5; VULN-184686 go-jose, 2026.10.6; the newest is 2026.10.7).** No public text on reach. Not pinned tonight: the tunnel is a box's only door, and 9202 has none to prove it on. | OPEN — filed 2026-11-03 | — | raise `CloudflaredImage` on the next controller release with a tunnel proof on Tester 1 | CC |
| **R-1380** | Box system & updates | P3 | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** Found 2026-11-11 (the End-dates list, class T2, `documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB02-3 to DB02-7). MEASURED: the agent trusts one exact Proxmox certificate (a pin), and the certificate on the three test boxes ends about two years after install — the N100's about 2028-07-16, then demo-hp and Tester 1 within three months. Proxmox makes a new certificate by itself about 14 days before the old one ends (so about 2028-07-02 on the N100). No code path gives the agent the new fingerprint, no alarm names the cause, no runbook holds the hand step, and nobody has tried it. Every box installed in the same season fails in the same fortnight. P3 only because the first date is in 2028; it is certain, not a risk. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 297). Not built.** **2026-11-11 (afternoon): tried on demo-hp (`documentation/audits/certpin-spike-2026-11-11/README.md`).** A forced new certificate (`pvecm updatecerts --force`, then a reload of the web service; Proxmox's own renewal does that reload by itself) blinds the agent at once: `TLS pin mismatch`, no reconcile, no host report to the hub, 502 to the controller; guests and apps stay up. One operator mail names a symptom (`agent_channel_unknown`), none names the cause; the hub's Hosts page still read ONLINE eleven minutes after the last report (when it turns offline was not measured). The hand repair works and is written down (`documentation/runbooks/RUNBOOK-agent-proxmox-pin-repair.md`, 2.1 s from restart to a working call). **A CA file in place of the fingerprint works with no agent code:** with `proxmox.tls.ca_file` on a readable copy of the box's root CA and the fingerprint empty, the agent went through a second new certificate with no hand step; a wrong CA file was refused. Two facts for the fix: the agent's user cannot read Proxmox's certificate files, and a set fingerprint wins over the CA file. The design with both ways is `03-host-agent.md` §12a; CC's pick differs from the ruled shape, so it is decision D80. | READY — **2026-11-14 night: BUILT, not released** (felhom-agent `386a0c4` + `a9ab47e`; the installer on felhom.eu `main`, not tagged). The root step keeps the copy; the agent moves a pinned box in the order copy → proof → config, clears the fingerprint last, and a pinned box whose certificate changed heals at the first failed call; the installer gives a new box the copy and no fingerprint; the cause of a trust failure is named. `03` §12a is `[FACT — built, not released]`. **LEFT: the proof on a box** (demo-hp, a forced new certificate, twice; then the release with both signed jobs and the installer tag). What the hub does not tell yet: R-1396. Earlier: READY — **2026-11-14 09:55 — RULED (D80, `09` §3 decision 303): the root CA copy.** Built in release 2 (the host release). OPEN — filed 2026-11-11 (end-dates list) | — | the operator rules D80 (CA file, or follow the fingerprint); then the design's open parts in `03-host-agent.md` §12a (the root step that refreshes the copy; an alarm that names the cause), then build, then prove on a scratch host with a forced renewal | CC |
| **R-1382** | Box system & updates | P3 | **A box that was off past an end date: nobody knows if it comes back by itself (5 answers).** Found 2026-11-11 (the End-dates list, class T4, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; read in source and documents, never tried). After months without power: the web certificate has ended (DB01-6: does Traefik renew at the first start, and what does the household see until then), Debian's package keys may be too old to verify an update (DB16-6), the tunnel client may be below what Cloudflare accepts (DB19-6), a 30-day restore copy is past its date (DB24-6), and the 14-day countdown after a stopped off-site copy ran out while the box was off (DB04-6). Tester 2 has been off since 2026-10-04 and is the natural test. | OPEN — filed 2026-11-11 (end-dates list) | — | When Tester 2 comes back (or with a scratch host and a moved clock on Tier 0, as a task): record what renews by itself and what needs a hand; put the hand steps in a runbook. | CC |
| **R-1384** | Box system & updates | P3 | **A verdict or an „already ran” check trusts a clock that can be wrong (30 answers).** Found 2026-11-11 (the Clock list, class W2, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED: the N100's host is 32.6 s behind and its time service does not run (R-1301); the other two boxes are in step. Nothing checks a box's clock except one hub alarm for a tunnel handshake stamped in the future. The hub takes the box's own time for backup age, off-site age and proof age (CH02-6, CH07-6, CH07-8, CH08-6); a box with a wrong clock refuses every signed job (CA09-6); a stamp from the future blocks the OS-update leg, the trim and the line test until the clock passes it (CA02-6, CA02-8, CA11-6, CA11-8, CA12-6); the crash guard reads the clock at early boot (CA04-6, CA04-8); a hub clock jump moves its own daily jobs (CH03-6, CH04-6, CH06-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6). The others: CA05-6, CA05-8, CA06-6, CA07-6, CA10-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6. Known parts stay in their rows: R-1189, R-1206, R-1260. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 300). Not built.** **2026-11-11 (afternoon): why the N100's clock is wrong is known (R-1301, `documentation/audits/n100-clock-2026-11-11/README.md`).** Numbers from source: a host clock more than 120 s behind refuses every signed job (`felhom-agent/internal/authz/verifier.go:21`, `:153-162`); at about 1.25 s a day the N100 gets there in about 70 days (mid-January 2027, an estimate from one measurement). | OPEN — **2026-11-14: the one box whose clock was wrong (the N100, 32.6 s) is right again (R-1301 closed); this row's check is still not built.** Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): the hub compares the time in each report with its own, alarms the operator above one minute, and does not take a time from the future as „fresh”. | CC |
| **R-1384** | Box system & updates | P3 | **A verdict or an „already ran” check trusts a clock that can be wrong (30 answers).** Found 2026-11-11 (the Clock list, class W2, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED: the N100's host is 32.6 s behind and its time service does not run (R-1301); the other two boxes are in step. Nothing checks a box's clock except one hub alarm for a tunnel handshake stamped in the future. The hub takes the box's own time for backup age, off-site age and proof age (CH02-6, CH07-6, CH07-8, CH08-6); a box with a wrong clock refuses every signed job (CA09-6); a stamp from the future blocks the OS-update leg, the trim and the line test until the clock passes it (CA02-6, CA02-8, CA11-6, CA11-8, CA12-6); the crash guard reads the clock at early boot (CA04-6, CA04-8); a hub clock jump moves its own daily jobs (CH03-6, CH04-6, CH06-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6). The others: CA05-6, CA05-8, CA06-6, CA07-6, CA10-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6. Known parts stay in their rows: R-1189, R-1206, R-1260. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 300). Not built.** **2026-11-11 (afternoon): why the N100's clock is wrong is known (R-1301, `documentation/audits/n100-clock-2026-11-11/README.md`).** Numbers from source: a host clock more than 120 s behind refuses every signed job (`felhom-agent/internal/authz/verifier.go:21`, `:153-162`); at about 1.25 s a day the N100 gets there in about 70 days (mid-January 2027, an estimate from one measurement). | NARROWED — the hub's check is built, not released. **2026-11-14 (night build):** the agent stamps its clock at the moment of sending (`sent_at`) and the hub alarms the operator when it is more than one minute from the hub's clock, in either direction, and says so when it is back (`08-alarm-ladder.md`, `box_clock_wrong`). A report that waited on the box raises nothing (tested). Not released, not seen on a box. LEFT after the release: the 30 answers themselves still trust the clock; the alarm only makes a wrong clock known. OPEN — **2026-11-14: the one box whose clock was wrong (the N100, 32.6 s) is right again (R-1301 closed); this row's check is still not built.** Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): the hub compares the time in each report with its own, alarms the operator above one minute, and does not take a time from the future as „fresh”. | CC |
| **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC |
| **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC |
| **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC |