hub (unreleased): the operator is told why a box's agent does not trust Proxmox (proxmox_trust stanza → proxmox_trust_problem / _ok, operator-only; R-1396)
gates / gates (push) Successful in 3m35s
gates / gates (push) Successful in 3m35s
This commit is contained in:
1 parent
d809a1d444
commit
26f4f7660c
8 files changed
+255
-4
No files matched your search
@@ -994,8 +994,9 @@ before, and the agent moves by itself later.
|
||||
`ca_file_unreadable`, `name_mismatch`, and from the move: `ca_copy_unavailable`, `ca_copy_does_not_verify`,
|
||||
`ca_copy_ends_soon` (30 days), `config_not_written`. Each is an ERROR line in the agent's journal and rides the host
|
||||
report as the `proxmox_trust` stanza (`mode`, `problem`, `detail`, `ca_not_after`, `ca_days_left`, `moved_at`).
|
||||
**Two gaps, both in R-1396:** the hub does not read the stanza yet (no operator alarm from it), and a box that is
|
||||
fully blind sends no host report at all, so that case still reaches the operator only as „host stale".
|
||||
**The hub reads the stanza** (hub `main` after v0.181.0, built, not deployed): one operator-only
|
||||
`proxmox_trust_problem` per (box, cause), told once, and one `proxmox_trust_ok` at the recovery. **One gap, R-1396:**
|
||||
a box that is fully blind sends no host report at all, so that case still reaches the operator only as „host stale".
|
||||
|
||||
**5. Both keys set.** The order in `proxmox.TLSConfig.build` is unchanged: skip-verify, then the fingerprint, then
|
||||
`ca_file`, then the system's roots. **A set fingerprint wins over a CA file.** So: the move clears the fingerprint
|
||||
|
||||
@@ -372,6 +372,7 @@ one info line beside `host_crash_restart`; `operatorOnlyEvents`, pinned by
|
||||
| Event | Severity | Raised when | Cleared | Pinned by |
|
||||
|---|---|---|---|---|
|
||||
| `tunnel_down` | error | the box's two newest host reports say the tunnel is `not_running` (more than one report cycle, 15 min) and the one before did not | the first `running` after it → `tunnel_recovered` (info) | `api/tunnel_test.go` |
|
||||
| `proxmox_trust_problem` | error (warning for „the root CA copy ends within 30 days”) | **[built, not deployed — hub `main` after v0.181.0]** the host report's `proxmox_trust` stanza names a cause (`03` §12a); once per (box, cause), not at every report | the first report with no problem → `proxmox_trust_ok` (info), one for all causes | `api/proxmox_trust_test.go` |
|
||||
| `os_update_stale` | warning | no successful OS leg for **7 days** while the switch is ON (agents that can run the leg only); the mail names the likely reason (box not reporting / the last leg's failure / no good night backup) | a successful leg | `TestAlarm_StaleLeg`, `TestAlarm_StaleNamesTheReason` |
|
||||
| `os_reboot_needed` | warning | the host has needed a reboot for **14 days** (from the FIRST scanned report that said so) | a scanned pass that finds nothing (agent ≥ 0.141.1 scans the host every pass) | `TestAlarm_RebootNeeded`, `TestRebootNeeded_ClearedByAScannedPass` |
|
||||
| `os_ring0_stalled` | error | ring 0 approved nothing for **7 days** in a layer while it has pending FAST-lane updates (a pending kernel does not count) | a new release | `TestAlarm_Ring0Stalled` |
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
== MUTATION: a cause is told at EVERY report (the 'said once' check removed)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestProxmoxTrust_CauseOnceThenRecovery (0.05s)
|
||||
proxmox_trust_test.go:58: problems=3 ok=0, want ONE problem for three reports of the same cause
|
||||
--- FAIL: TestProxmoxTrust_UnknownCause (0.04s)
|
||||
proxmox_trust_test.go:106: events = [{ID:1 CustomerID:c1 EventType:proxmox_trust_problem Severity:error Message:Trust in Proxmox on h1: a cause this hub does not know (mode: ca). Design: `03` §12a. DetailsJSON:{"ca_not_after":"","
|
||||
== MUTATION: no recovery event
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestProxmoxTrust_CauseOnceThenRecovery (0.05s)
|
||||
proxmox_trust_test.go:84: problems=2 ok=0, want ONE recovery for both causes
|
||||
== MUTATION: a cause the hub does not know is passed on as text
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestProxmoxTrust_UnknownCause (0.04s)
|
||||
proxmox_trust_test.go:106: events = [{ID:1 CustomerID:c1 EventType:proxmox_trust_problem Severity:error Message:Trust in Proxmox on h1: Kedves Ügyfél, kattints ide (mode: ca). Design: `03` §12a. DetailsJSON:{"ca_not_after":"","cau
|
||||
== MUTATION: 'ends soon' is an error like the rest
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestProxmoxTrust_CauseOnceThenRecovery (0.05s)
|
||||
proxmox_trust_test.go:78: ends-soon event = {ID:2 CustomerID:c1 EventType:proxmox_trust_problem Severity:error Message:Trust in Proxmox on h1: the root CA copy ends within 30 days (mode: ca). The root CA copy ends in 12 day(s) (20
|
||||
== MUTATION: the check is not called from the report handler
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestProxmoxTrust_CauseOnceThenRecovery (0.05s)
|
||||
proxmox_trust_test.go:58: problems=0 ok=0, want ONE problem for three reports of the same cause
|
||||
--- FAIL: TestProxmoxTrust_UnknownCause (0.04s)
|
||||
proxmox_trust_test.go:106: events = []
|
||||
|
||||
5 mutations, 0 not a red-proof
|
||||
@@ -333,7 +333,7 @@ stopping line that lies.
|
||||
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
|
||||
| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC |
|
||||
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | OPEN — **2026-11-14 (release 2a), two more things the hub does not tell:** (1) a kernel clean-up that removed kernels shows only as „cleaned <time ago>” in the System page's „Kernel step” cell — what was removed, kept and gained is stored and shown nowhere; (2) the security-profile refresh sends nothing to the hub at all, also when it restarted a failed service (it did on the N100). Both for release 3. Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
|
||||
| **R-1396** | Monitoring & notifications | P3 | **The agent now names WHY its trust in Proxmox is wrong, and the hub does not tell the operator yet.** Found 2026-11-14 night (release 2b, built). The agent's host report carries a `proxmox_trust` stanza (`mode`, `problem`, `detail`, the CA copy's end date; `03-host-agent.md` §12a). Two gaps: (1) the hub does not read the stanza — a box whose CA copy does not verify, ends within 30 days, or whose move failed is said only in the box's journal; (2) a box that is fully blind (neither the pinned certificate nor the CA copy works) sends NO host report, because the report starts with a call to Proxmox — that case reaches the operator only as „host stale” after 45 minutes, with no cause. With release 2b a pinned box heals itself at the first failed call, so (2) needs both ways to fail at once. | OPEN — filed 2026-11-14 | — | hub: an operator-only event from `proxmox_trust.problem` (raise once per cause, clear when it is empty), and the mode on the System page; agent: send a report without the Proxmox part when the failure is one of trust (a design choice: what the hub does with a report that has no guests) | CC |
|
||||
| **R-1396** | Monitoring & notifications | P3 | **The agent now names WHY its trust in Proxmox is wrong, and the hub does not tell the operator yet.** Found 2026-11-14 night (release 2b, built). The agent's host report carries a `proxmox_trust` stanza (`mode`, `problem`, `detail`, the CA copy's end date; `03-host-agent.md` §12a). Two gaps: (1) the hub does not read the stanza — a box whose CA copy does not verify, ends within 30 days, or whose move failed is said only in the box's journal; (2) a box that is fully blind (neither the pinned certificate nor the CA copy works) sends NO host report, because the report starts with a call to Proxmox — that case reaches the operator only as „host stale” after 45 minutes, with no cause. With release 2b a pinned box heals itself at the first failed call, so (2) needs both ways to fail at once. | NARROWED — **2026-11-14 night: gap (1) is BUILT, not deployed** — the hub reads `proxmox_trust` and raises one operator-only `proxmox_trust_problem` per (box, cause), and `proxmox_trust_ok` at the recovery (hub `main` after v0.181.0; the wire-contract gate asked for the receiver in the same push as the sender). **LEFT: gap (2), the fully blind box.** Earlier: OPEN — filed 2026-11-14 | — | hub: an operator-only event from `proxmox_trust.problem` (raise once per cause, clear when it is empty), and the mode on the System page; agent: send a report without the Proxmox part when the failure is one of trust (a design choice: what the hub does with a report that has no guests) | CC |
|
||||
| **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC |
|
||||
|
||||
## Hub & operator — 18 rows (P2 1, P3 5, P4 12)
|
||||
|
||||
@@ -2,6 +2,15 @@
|
||||
|
||||
- `bundlefiles`: the line „bundle X holds the same files as the vouched Y” is logged once per pair (INFO), not at
|
||||
every view of the System page (it was 6 DEBUG lines per view, seen at the 0.181.0 deploy). No behaviour change.
|
||||
- **Release 3 („the hub tells you”), built 2026-11-14 night, NOT deployed. Every new alarm is operator-only.**
|
||||
- **Why a box's agent does not trust Proxmox (R-1396, R-1380; `03` §12a).** The host report's new `proxmox_trust`
|
||||
stanza (agents of release 2b) is read: one `proxmox_trust_problem` per (box, cause) — told once, not at every
|
||||
report; `error`, and `warning` for „the root CA copy ends within 30 days” — and one `proxmox_trust_ok` when the box
|
||||
reports no problem again. The causes are a closed list; a cause the hub does not know is told as „other” and the
|
||||
box's own text never reaches a mail. An agent without the stanza neither raises nor clears. Both events are in
|
||||
`notify.operatorOnlyEvents`. Tests `TestProxmoxTrust_*`; 5 mutations seen red
|
||||
(`documentation/audits/night-build-2026-11-14/B/redproof-hub-proxmox-trust.txt`). **Not built:** a fully blind box
|
||||
sends no host report at all (R-1396, what is left).
|
||||
|
||||
## v0.181.0 (2026-11-14) — R-1392: root files are „behind” only when the FILES differ; Go 1.27.4 (R-1391) (image `sha256:be2e9e5676eec7622622402a830d8f92913a93fe746e421db60f949138d1f429` built from `83316fc9`, CI run id 2636; the operator's yes 2026-11-14 ~09:25 UTC; the manifest bump is the commit that carries this line)
|
||||
|
||||
|
||||
+103
-1
@@ -775,7 +775,10 @@ type hostReportPayload struct {
|
||||
} `json:"host"`
|
||||
// Wireguard (R-1204): the box's own view of its tunnel to ep0. nil = an agent with no tunnel stanza.
|
||||
Wireguard *hostWireguard `json:"wireguard"`
|
||||
Guests []struct {
|
||||
// ProxmoxTrust (R-1380, R-1396; `03` §12a): how the agent trusts Proxmox's certificate and, when something about it
|
||||
// is wrong, the CAUSE. nil = an agent without the stanza (older than release 2b).
|
||||
ProxmoxTrust *hostProxmoxTrust `json:"proxmox_trust"`
|
||||
Guests []struct {
|
||||
VMID int `json:"vmid"`
|
||||
Name string `json:"name"`
|
||||
Status string `json:"status"`
|
||||
@@ -1008,6 +1011,7 @@ func (h *Handler) handleHostReport(w http.ResponseWriter, r *http.Request) {
|
||||
h.checkTunnel(hostID, custID, rep.Cloudflared.Status, rep.Cloudflared.Detail)
|
||||
h.checkHostSwap(hostID, custID, rep.Host.HostSwapMB, rep.Host.GuestSwapMB)
|
||||
h.checkWGTunnel(hostID, custID, rep.Wireguard)
|
||||
h.checkProxmoxTrust(hostID, custID, rep.ProxmoxTrust)
|
||||
h.checkCrash(hostID, custID, body)
|
||||
h.checkDeliberateOff(hostID, custID, &rep)
|
||||
|
||||
@@ -3236,6 +3240,104 @@ func (h *Handler) checkTunnel(hostID, custID, status, detail string) {
|
||||
}
|
||||
}
|
||||
|
||||
// Proxmox trust alarm (R-1396, R-1380; `03-host-agent.md` §12a). The agent (release 2b and later) reports how it trusts
|
||||
// Proxmox's certificate and names the cause when that is wrong: the proof with the CA copy failed, the copy ends
|
||||
// within 30 days, the root step gave no copy, the config could not be written. One operator-only event per (box,
|
||||
// cause) — raised once, not at every report — and one `proxmox_trust_ok` when the box reports no problem again.
|
||||
// A box with NO stanza (an older agent) never alarms and never clears. Pinned by TestProxmoxTrust_*.
|
||||
const (
|
||||
eventProxmoxTrustProblem = "proxmox_trust_problem"
|
||||
eventProxmoxTrustOK = "proxmox_trust_ok"
|
||||
)
|
||||
|
||||
type hostProxmoxTrust struct {
|
||||
Mode string `json:"mode"`
|
||||
Problem string `json:"problem"`
|
||||
Detail string `json:"detail"`
|
||||
CANotAfter string `json:"ca_not_after"`
|
||||
CADaysLeft *int `json:"ca_days_left"`
|
||||
MovedAt string `json:"moved_at"`
|
||||
}
|
||||
|
||||
// proxmoxTrustProblems is the CLOSED list of causes the hub alarms on (the agent's internal/pvetrust and
|
||||
// proxmox.Trust* names). A cause outside it is logged and stored under "other" — never dropped, never trusted as text.
|
||||
var proxmoxTrustProblems = map[string]string{
|
||||
"pinned_cert_changed": "the pinned certificate is not the one Proxmox serves any more",
|
||||
"ca_mismatch": "the root CA copy did not sign the certificate Proxmox serves",
|
||||
"cert_expired": "the certificate Proxmox serves, or its root CA, has ended",
|
||||
"ca_file_unreadable": "the root CA copy is missing or holds no certificate",
|
||||
"name_mismatch": "the certificate Proxmox serves does not name the address the agent calls",
|
||||
"ca_copy_unavailable": "the box's root step gave no copy of Proxmox's root CA",
|
||||
"ca_copy_does_not_verify": "a call to Proxmox with the root CA copy alone failed; the box keeps its pinned certificate",
|
||||
"ca_copy_ends_soon": "the root CA copy ends within 30 days",
|
||||
"config_not_written": "the agent could not save its new trust settings; the box keeps its pinned certificate",
|
||||
}
|
||||
|
||||
func (h *Handler) checkProxmoxTrust(hostID, custID string, t *hostProxmoxTrust) {
|
||||
if t == nil {
|
||||
return
|
||||
}
|
||||
now := time.Now()
|
||||
emit := func(typ, sev, msg string, details map[string]any) {
|
||||
dj, _ := json.Marshal(details)
|
||||
if _, err := h.store.SaveEvent(custID, typ, sev, msg, string(dj), "hub"); err != nil {
|
||||
h.logger.Printf("[WARN] %s event save FAILED for %s: %v", typ, hostID, err)
|
||||
} else if h.dispatcher != nil {
|
||||
go h.dispatcher.ProcessEvent(custID, typ, sev, msg, string(dj), "hub")
|
||||
}
|
||||
}
|
||||
if t.Problem == "" {
|
||||
// the box reports no problem: clear every cause that was raised for it, with ONE recovery event
|
||||
var cleared []string
|
||||
for cause := range proxmoxTrustProblems {
|
||||
key := "pve_trust:" + hostID + ":" + cause
|
||||
if !h.store.OSAlarmRaised(key).IsZero() {
|
||||
_ = h.store.SetOSAlarmRaised(key, time.Time{})
|
||||
cleared = append(cleared, cause)
|
||||
}
|
||||
}
|
||||
if key := "pve_trust:" + hostID + ":other"; !h.store.OSAlarmRaised(key).IsZero() {
|
||||
_ = h.store.SetOSAlarmRaised(key, time.Time{})
|
||||
cleared = append(cleared, "other")
|
||||
}
|
||||
if len(cleared) > 0 {
|
||||
sort.Strings(cleared)
|
||||
h.logger.Printf("[INFO] host %s proxmox trust: no problem any more (was: %s; mode %s)", hostID, strings.Join(cleared, ", "), t.Mode)
|
||||
emit(eventProxmoxTrustOK, "info", fmt.Sprintf("The agent of %s trusts Proxmox again (mode: %s). Was: %s.", hostID, t.Mode, strings.Join(cleared, ", ")),
|
||||
map[string]any{"host_id": hostID, "mode": t.Mode, "was": cleared})
|
||||
}
|
||||
return
|
||||
}
|
||||
cause, text := t.Problem, proxmoxTrustProblems[t.Problem]
|
||||
if text == "" {
|
||||
cause, text = "other", "a cause this hub does not know"
|
||||
}
|
||||
key := "pve_trust:" + hostID + ":" + cause
|
||||
if !h.store.OSAlarmRaised(key).IsZero() {
|
||||
return // said once; the box repeats it at every report
|
||||
}
|
||||
_ = h.store.SetOSAlarmRaised(key, now)
|
||||
sev := "error"
|
||||
if cause == "ca_copy_ends_soon" {
|
||||
sev = "warning"
|
||||
}
|
||||
days := ""
|
||||
if t.CADaysLeft != nil {
|
||||
days = fmt.Sprintf(" The root CA copy ends in %d day(s) (%s).", *t.CADaysLeft, t.CANotAfter)
|
||||
}
|
||||
h.logger.Printf("[WARN] host %s proxmox trust: %s (mode %s)", hostID, cause, t.Mode)
|
||||
emit(eventProxmoxTrustProblem, sev, fmt.Sprintf("Trust in Proxmox on %s: %s (mode: %s).%s Design: `03` §12a.", hostID, text, t.Mode, days),
|
||||
map[string]any{"host_id": hostID, "cause": cause, "mode": t.Mode, "reported_cause": truncateRunes(t.Problem, 60), "ca_not_after": t.CANotAfter})
|
||||
}
|
||||
|
||||
func truncateRunes(s string, n int) string {
|
||||
r := []rune(s)
|
||||
if len(r) <= n {
|
||||
return s
|
||||
}
|
||||
return string(r[:n])
|
||||
}
|
||||
|
||||
// recordHeldApp applies one app_held / app_hold_cleared to the held_apps mirror (D53, `08` §10). The hub never
|
||||
// clears a hold on a box; it only forgets one the box says is gone.
|
||||
func (h *Handler) recordHeldApp(customerID, eventType string, raw json.RawMessage) {
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
func trustReport(stanza string) string {
|
||||
if stanza == "-" {
|
||||
return validReportBody("h1")
|
||||
}
|
||||
b := validReportBody("h1")
|
||||
return b[:strings.LastIndex(b, "}")] + `,"proxmox_trust":` + stanza + "}"
|
||||
}
|
||||
|
||||
func trustSetup(t *testing.T) (*Handler, *store.Store) {
|
||||
t.Helper()
|
||||
h, st, _ := newTestHandler(t)
|
||||
st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ckey", RetrievalPassword: "p"})
|
||||
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||
return h, st
|
||||
}
|
||||
|
||||
func trustCounts(t *testing.T, st *store.Store) (problems, oks []store.Event) {
|
||||
t.Helper()
|
||||
since := time.Now().Add(-time.Hour)
|
||||
problems, _ = st.GetEventsByType("c1", eventProxmoxTrustProblem, since)
|
||||
oks, _ = st.GetEventsByType("c1", eventProxmoxTrustOK, since)
|
||||
return
|
||||
}
|
||||
|
||||
// R-1396: a cause is told ONCE (the box repeats it at every report), a second cause is told too, and one recovery
|
||||
// event follows when the box reports no problem. A healthy stanza and an agent without the stanza raise nothing.
|
||||
// COMPANION RED-PROOFS (observed, audits/night-build-2026-11-14/B/redproof-hub.txt).
|
||||
func TestProxmoxTrust_CauseOnceThenRecovery(t *testing.T) {
|
||||
h, st := trustSetup(t)
|
||||
post := func(stanza string) {
|
||||
t.Helper()
|
||||
if rr := do(h, http.MethodPost, "/host-report", "HKEY", trustReport(stanza)); rr.Code != 200 {
|
||||
t.Fatalf("report: %d %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
}
|
||||
post(`{"mode":"ca"}`)
|
||||
post("-")
|
||||
if p, o := trustCounts(t, st); len(p) != 0 || len(o) != 0 {
|
||||
t.Fatalf("a healthy box raised %d problem(s), %d ok", len(p), len(o))
|
||||
}
|
||||
bad := `{"mode":"fingerprint","problem":"ca_copy_does_not_verify","detail":"<script>x</script>","ca_days_left":3560,"ca_not_after":"2036-08-17T15:44:42Z"}`
|
||||
post(bad)
|
||||
post(bad)
|
||||
post(bad)
|
||||
p, o := trustCounts(t, st)
|
||||
if len(p) != 1 || len(o) != 0 {
|
||||
t.Fatalf("problems=%d ok=%d, want ONE problem for three reports of the same cause", len(p), len(o))
|
||||
}
|
||||
if p[0].Severity != "error" || !strings.Contains(p[0].Message, "keeps its pinned certificate") || !strings.Contains(p[0].Message, "h1") {
|
||||
t.Fatalf("event = %+v", p[0])
|
||||
}
|
||||
if strings.Contains(p[0].Message, "<script>") || strings.Contains(p[0].DetailsJSON, "<script>") {
|
||||
t.Fatalf("the box's free text reached the operator's mail: %+v", p[0])
|
||||
}
|
||||
post(`{"mode":"ca","problem":"ca_copy_ends_soon","ca_days_left":12,"ca_not_after":"2026-11-26T00:00:00Z"}`)
|
||||
p, _ = trustCounts(t, st)
|
||||
if len(p) != 2 {
|
||||
t.Fatalf("a second cause was not told: %d", len(p))
|
||||
}
|
||||
var soon store.Event
|
||||
for _, e := range p {
|
||||
if strings.Contains(e.Message, "ends within 30 days") {
|
||||
soon = e
|
||||
}
|
||||
}
|
||||
if soon.Severity != "warning" || !strings.Contains(soon.Message, "12 day") {
|
||||
t.Fatalf("ends-soon event = %+v", soon)
|
||||
}
|
||||
post(`{"mode":"ca"}`)
|
||||
post(`{"mode":"ca"}`)
|
||||
p, o = trustCounts(t, st)
|
||||
if len(p) != 2 || len(o) != 1 {
|
||||
t.Fatalf("problems=%d ok=%d, want ONE recovery for both causes", len(p), len(o))
|
||||
}
|
||||
// after the recovery the same cause is a NEW problem
|
||||
post(bad)
|
||||
if p, _ = trustCounts(t, st); len(p) != 3 {
|
||||
t.Fatalf("the cause came back and was not told again: %d", len(p))
|
||||
}
|
||||
// an agent WITHOUT the stanza neither clears nor raises
|
||||
post("-")
|
||||
if _, o = trustCounts(t, st); len(o) != 1 {
|
||||
t.Fatalf("a report with no stanza cleared the problem: ok=%d", len(o))
|
||||
}
|
||||
}
|
||||
|
||||
// A cause the hub does not know is told as "other" — once — and its text is not passed on.
|
||||
func TestProxmoxTrust_UnknownCause(t *testing.T) {
|
||||
h, st := trustSetup(t)
|
||||
for i := 0; i < 2; i++ {
|
||||
do(h, http.MethodPost, "/host-report", "HKEY", trustReport(`{"mode":"ca","problem":"Kedves Ügyfél, kattints ide","detail":"x"}`))
|
||||
}
|
||||
p, _ := trustCounts(t, st)
|
||||
if len(p) != 1 || !strings.Contains(p[0].Message, "a cause this hub does not know") || strings.Contains(p[0].Message, "Kedves") {
|
||||
t.Fatalf("events = %+v", p)
|
||||
}
|
||||
}
|
||||
@@ -841,6 +841,10 @@ var operatorOnlyEvents = map[string]bool{
|
||||
// R-841 (hub v0.131.0): the tunnel alarm — a box fact the household can do nothing about from inside.
|
||||
"tunnel_down": true,
|
||||
"tunnel_recovered": true,
|
||||
// R-1396 / R-1380 (hub v0.182.0): why a box's agent does not trust Proxmox's certificate, and its recovery — a box
|
||||
// fact only the operator can act on. Hub-origin only (minted from the host report's `proxmox_trust` stanza).
|
||||
"proxmox_trust_problem": true,
|
||||
"proxmox_trust_ok": true,
|
||||
// R-836 (hub v0.143.0): the kernel lane's step results and the day-before notice's own record — operator facts. The
|
||||
// household's word is the day-before mail itself (SendKernelNotice), never these events.
|
||||
"os_kernel_step": true,
|
||||
|
||||
Reference in new issue
Block a user