diff --git a/documentation/architecture/03-host-agent.md b/documentation/architecture/03-host-agent.md index ebc31cce..a862598c 100644 --- a/documentation/architecture/03-host-agent.md +++ b/documentation/architecture/03-host-agent.md @@ -994,8 +994,9 @@ before, and the agent moves by itself later. `ca_file_unreadable`, `name_mismatch`, and from the move: `ca_copy_unavailable`, `ca_copy_does_not_verify`, `ca_copy_ends_soon` (30 days), `config_not_written`. Each is an ERROR line in the agent's journal and rides the host report as the `proxmox_trust` stanza (`mode`, `problem`, `detail`, `ca_not_after`, `ca_days_left`, `moved_at`). -**Two gaps, both in R-1396:** the hub does not read the stanza yet (no operator alarm from it), and a box that is -fully blind sends no host report at all, so that case still reaches the operator only as „host stale". +**The hub reads the stanza** (hub `main` after v0.181.0, built, not deployed): one operator-only +`proxmox_trust_problem` per (box, cause), told once, and one `proxmox_trust_ok` at the recovery. **One gap, R-1396:** +a box that is fully blind sends no host report at all, so that case still reaches the operator only as „host stale". **5. Both keys set.** The order in `proxmox.TLSConfig.build` is unchanged: skip-verify, then the fingerprint, then `ca_file`, then the system's roots. **A set fingerprint wins over a CA file.** So: the move clears the fingerprint diff --git a/documentation/architecture/08-alarm-ladder.md b/documentation/architecture/08-alarm-ladder.md index f2055161..4edbd06c 100644 --- a/documentation/architecture/08-alarm-ladder.md +++ b/documentation/architecture/08-alarm-ladder.md @@ -372,6 +372,7 @@ one info line beside `host_crash_restart`; `operatorOnlyEvents`, pinned by | Event | Severity | Raised when | Cleared | Pinned by | |---|---|---|---|---| | `tunnel_down` | error | the box's two newest host reports say the tunnel is `not_running` (more than one report cycle, 15 min) and the one before did not | the first `running` after it → `tunnel_recovered` (info) | `api/tunnel_test.go` | +| `proxmox_trust_problem` | error (warning for „the root CA copy ends within 30 days”) | **[built, not deployed — hub `main` after v0.181.0]** the host report's `proxmox_trust` stanza names a cause (`03` §12a); once per (box, cause), not at every report | the first report with no problem → `proxmox_trust_ok` (info), one for all causes | `api/proxmox_trust_test.go` | | `os_update_stale` | warning | no successful OS leg for **7 days** while the switch is ON (agents that can run the leg only); the mail names the likely reason (box not reporting / the last leg's failure / no good night backup) | a successful leg | `TestAlarm_StaleLeg`, `TestAlarm_StaleNamesTheReason` | | `os_reboot_needed` | warning | the host has needed a reboot for **14 days** (from the FIRST scanned report that said so) | a scanned pass that finds nothing (agent ≥ 0.141.1 scans the host every pass) | `TestAlarm_RebootNeeded`, `TestRebootNeeded_ClearedByAScannedPass` | | `os_ring0_stalled` | error | ring 0 approved nothing for **7 days** in a layer while it has pending FAST-lane updates (a pending kernel does not count) | a new release | `TestAlarm_Ring0Stalled` | diff --git a/documentation/audits/night-build-2026-11-14/B/redproof-hub-proxmox-trust.txt b/documentation/audits/night-build-2026-11-14/B/redproof-hub-proxmox-trust.txt new file mode 100644 index 00000000..c266bc57 --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/B/redproof-hub-proxmox-trust.txt @@ -0,0 +1,26 @@ +== MUTATION: a cause is told at EVERY report (the 'said once' check removed) + RED on the mutant, GREEN on the real file + --- FAIL: TestProxmoxTrust_CauseOnceThenRecovery (0.05s) + proxmox_trust_test.go:58: problems=3 ok=0, want ONE problem for three reports of the same cause + --- FAIL: TestProxmoxTrust_UnknownCause (0.04s) + proxmox_trust_test.go:106: events = [{ID:1 CustomerID:c1 EventType:proxmox_trust_problem Severity:error Message:Trust in Proxmox on h1: a cause this hub does not know (mode: ca). Design: `03` §12a. DetailsJSON:{"ca_not_after":""," +== MUTATION: no recovery event + RED on the mutant, GREEN on the real file + --- FAIL: TestProxmoxTrust_CauseOnceThenRecovery (0.05s) + proxmox_trust_test.go:84: problems=2 ok=0, want ONE recovery for both causes +== MUTATION: a cause the hub does not know is passed on as text + RED on the mutant, GREEN on the real file + --- FAIL: TestProxmoxTrust_UnknownCause (0.04s) + proxmox_trust_test.go:106: events = [{ID:1 CustomerID:c1 EventType:proxmox_trust_problem Severity:error Message:Trust in Proxmox on h1: Kedves Ügyfél, kattints ide (mode: ca). Design: `03` §12a. DetailsJSON:{"ca_not_after":"","cau +== MUTATION: 'ends soon' is an error like the rest + RED on the mutant, GREEN on the real file + --- FAIL: TestProxmoxTrust_CauseOnceThenRecovery (0.05s) + proxmox_trust_test.go:78: ends-soon event = {ID:2 CustomerID:c1 EventType:proxmox_trust_problem Severity:error Message:Trust in Proxmox on h1: the root CA copy ends within 30 days (mode: ca). The root CA copy ends in 12 day(s) (20 +== MUTATION: the check is not called from the report handler + RED on the mutant, GREEN on the real file + --- FAIL: TestProxmoxTrust_CauseOnceThenRecovery (0.05s) + proxmox_trust_test.go:58: problems=0 ok=0, want ONE problem for three reports of the same cause + --- FAIL: TestProxmoxTrust_UnknownCause (0.04s) + proxmox_trust_test.go:106: events = [] + +5 mutations, 0 not a red-proof diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 5cf491b5..f1c8c7cc 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -333,7 +333,7 @@ stopping line that lies. | **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC | | **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC | | **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | OPEN — **2026-11-14 (release 2a), two more things the hub does not tell:** (1) a kernel clean-up that removed kernels shows only as „cleaned