hub v0.75.0: R-81 — "no signal" is not "bad signal" (anchor the backup deadline check)
Third instance of one class (hub v0.12.0, v0.73.0, this), fixed as a class. On 2026-07-26 03:00 UTC expected_backup_missed fired on demo-felhom, demo-hp and drill-r50 at once; the demo-felhom one reached the CUSTOMER channel claiming "newest backup is 176h0m0s old". Nothing was wrong — three vzdump archives were on disk. Cause: the agent backup store is in-memory, so the R-50 fleet restart emptied `backups` until the next run, and the hub read empty as "no backup exists". - assessBackupFreshness returns OK/UNKNOWN/MISSED instead of `missed bool`; absence is UNKNOWN until it outlives an anchored window. Still pure. - store.GetHostReportsSince + monitor.newestBackupEvidence read the hubs own retained history (bounded 7-day lookback, early-exit on fresh evidence) — "when did I last SEE evidence of a backup?" The anchor was free: the hub already retains 90 days. No agent change, no new persisted state. - store.GetFirstHostReportAt anchors absence at first contact, reusing the existing 26h threshold as the grace (no new knob, the v0.73.0 shape). - Deferrals logged + counted; reason strings kept distinct. - backupStaleAfter untouched; landmine recorded (a weekly PBS snapshot would alarm six days in seven) and owned by R-82. Tests 493->508. Red-proofs A/B/C observed and restored; A reproduces the live message verbatim. Replayed the real 03:00 reports (600/417/77 rows): all three now silent. Source: documentation/audits/DIAG-backup-missed-2026-07-26.md
This commit is contained in:
1 parent
add5b9bbbb
commit
f5a5e2b911
9 files changed
+843
-28
No files matched your search
@@ -111,7 +111,8 @@
|
||||
| Paired recovery notifications + prefs seeding at claim + priority headers (power-outage audit F11/F12/F14-light) | hub v0.71.0 | **IMPLEMENTED** (recovery leg **PARTIAL** until a live staleness cycle fires it) | `hub/CHANGELOG.md` v0.71.0; 17 tests + 4 red-proofs (`REPORT.md` 2026-07-22); Resend `headers` mechanism probed live (HTTP 200) pre-implementation; operator+customer `test` rows live-fired via the controller's own test endpoint | Recovery = explicit eventType branch, severity semantics frozen; customer gate = PAIRING (`notification_log` evidence), not `enabled_events`. Live legs pending: a natural `*_recovered` mail (next real staleness cycle or the reboot-drill arc — never fabricated by blocking reports) and seed-at-claim on a real claim (Peti Friday reinstall). F14-full (operator push channel, ntfy/Telegram) stays open → R-69 |
|
||||
| System + container metrics (SQLite, Chart.js, 30-day downsampling) | controller | **IMPLEMENTED** | metrics collection + `/monitoring` render present (page 200) | The cited `CAMPAIGN-2` T-RES-CPU/T-SOAK-LOOP are H1/H2 harness artifacts (auth-302), not metrics tests; SQLite/Chart.js/30-day downsampling validated in no campaign. Demoted |
|
||||
| Always-on debug rings + on-demand log-bundle pulls with TTL/custody | controller v0.116, agent v0.83, hub v0.46 | **PROVEN-LIVE** | debug rings live-exercised `CAMPAIGN-3` fix-6 (1000-cap ring, ~55min horizon under load) | The **log-bundle-pull TTL/custody** half is changelog-only (no dedicated observability audit doc); ring persistence across restart is a known gap |
|
||||
| Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | operator infra, stated in production since 02-04; no corpus validation doc | Per the status enum, no citation → not PROVEN-LIVE. Demoted pending an operator-cited live alert (candidate re-upgrade — see REPORT) |
|
||||
| Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | operator infra, stated in production since 02-04; no corpus validation doc |
|
||||
| Backup-deadline alerting (`expected_backup_missed`) is ANCHORED — absence of signal is UNKNOWN, not failure | hub v0.75.0 | **IMPLEMENTED** | `audits/DIAG-backup-missed-2026-07-26.md` + red-proofs A/B/C + replay of the real 2026-07-26 03:00 reports (all three silent) | **No row status flips** — this signal had a FALSE-POSITIVE class (three instances: hub v0.12.0, v0.73.0, R-81), now anchored at first contact and read across retained host-report history. Still unit-proven only, not live-fired at a real deadline. The *underlying* PBS/offsite-DR tier gap it exposed is → R-82. | Per the status enum, no citation → not PROVEN-LIVE. Demoted pending an operator-cited live alert (candidate re-upgrade — see REPORT) |
|
||||
|
||||
## G. Fleet & operator (hub)
|
||||
|
||||
|
||||
@@ -108,7 +108,11 @@
|
||||
| R-77 | **Endpoint-drift detection, samba protected-set gate, channel log honesty** | S | **SHIPPED (controller v0.173.0 + hub v0.74.0, 2026-07-26)** | Fixes the two defects in `audits/DIAG-agent-channel-2026-07-26.md`. `bootstrap.DetectEndpointDrift` names a `controller.yaml` vs `bootstrap.json` `local_api.endpoint` divergence with its own event type `local_api_endpoint_drift` and its own banner — and **writes nothing** (see R-78). `EffectiveProtected` now gates samba on `Enabled && UserSet`, mirroring both of `reconcileSambaAt`'s early returns, plus the doc-comment correction. Channel log distinguishes born-down (`unseeded->down`) from a real transition, logging-only. **Part 0 repaired both production boxes** (endpoint → `169.254.253.1:8443`; fingerprint+token agreed). Tests 951→959; red-proofs A/E/F. |
|
||||
| R-78 | **`local_api` authority ruling — auto-reconcile vs detect-only** | M | idea (deferred OUT of R-77 on purpose) | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. |
|
||||
| R-79 | **`report.Issues` / `report.Warnings` are English on customer-facing surfaces** | M | idea | **Whole-surface, not a one-off** (DIAG §6): every producer is English — `"SSD/HDD disk usage critical"`, `"Docker: %v"`, `"Protected container not running: %s"`, and all six `Warnings` strings. They render on the customer's Hungarian dashboard, and the `health_critical` path has reached the **customer** email channel three times historically. Deliberately NOT bundled into R-77: a copy sweep across every producer would have buried two safety fixes in string churn, and the seam is not obvious — translate at the producer, or at the render/notification boundary where operator-English and customer-Hungarian already diverge? Pick the seam in a spike; the strings are mechanical after. |
|
||||
| R-80 | **`expected_backup_missed` firing nightly on all three customers — one reaching a CUSTOMER channel** | M | idea — **likely outranks R-77** | 2026-07-26 03:00 fired for demo-felhom, demo-hp and drill-r50; the demo-felhom one went to the **customer** channel reporting `newest backup is 176h0m0s old` — **7.3 days**. **The causal link the DIAG hedged on is unlikely to be the whole story: 7.3 days materially exceeds the ~1.5-day channel outage (2026-07-25 12:44 → 2026-07-26 07:00), so backups were already stale for ~6 days BEFORE the channel broke.** The channel outage may compound it but cannot explain it. This is a customer-visible claim about their data protection and needs its own diagnostic: is it a real backup gap, a stale-report artefact, or a threshold/derivation defect? Do not assume it resolves with R-77's repair — verify against PBS snapshot reality. |
|
||||
| R-80 | **`expected_backup_missed` false alarm — diagnosed + class-fixed** | M | **SHIPPED (hub v0.75.0, 2026-07-26)** | Diagnosed in `audits/DIAG-backup-missed-2026-07-26.md`, fixed as R-81. **The premise was wrong on both counts:** it fired ONCE (not nightly), and no real external customer was notified — the one customer-channel mail went to the operator's own mailbox; `peti-felhom` (the only real external customer) did not fire. **The 7.3 days did NOT predate the channel outage** — it is the age of the *PBS* snapshot, reached only as a fallback once the vzdump array went empty. Local vzdump was never stale (archives on disk 07-24/25/26). Cause: the agent's backup store is in-memory, so the R-50 fleet restart at 12:44 UTC emptied `backups` until the next backup at 07:03, and the hub read empty as "no backup exists". Fixed by anchoring (hub v0.75.0). **Left standing as a REAL finding → R-82:** the PBS/offsite-DR tier has no schedule at all. |
|
||||
| R-81 | **"No signal" is not "bad signal" — anchor the backup deadline check** | S | **SHIPPED (hub v0.75.0, 2026-07-26)** | Third instance of one class (hub v0.12.0, v0.73.0, this). `assessBackupFreshness` now returns OK/UNKNOWN/MISSED; absence is UNKNOWN until it outlives an anchored window. The hub reads its own retained host-report history (`GetHostReportsSince` + `newestBackupEvidence`, 7-day bounded lookback) to answer "when did I last SEE evidence of a backup?", and anchors absence at first contact (`GetFirstHostReportAt`), reusing the existing 26 h threshold as the grace — no new knob, the v0.73.0 shape. Deferrals are logged; reason strings kept distinct. Tests 493→508; red-proofs A/B/C. **Not silence** — a genuinely dead box still alarms, which is the half the naive fix breaks. |
|
||||
| R-82 | **The backup target split — local daily + PBS weekly is NOT EXPRESSIBLE today** | M | idea — **the real finding behind R-80** | `backup.local_backup_target` is a SINGLE target and `BackupCadence()` a SINGLE 24 h window, so a box cannot run local-daily and PBS-weekly at all. `felhom-host-install.sh:2291` pins every new box to `local`, leaving the DR tier provisioned, authenticating and **empty**: demo-felhom holds ONE PBS snapshot (2026-07-18, not from a recurring job), demo-hp has `pbs_dr` applied since 07-21 and **zero snapshots ever**, and there is no `/etc/pve/jobs.cfg`. The offsite-DR promise is currently unbacked on both demo boxes. Must also: (a) **move `backupStaleAfter`** — it is applied to whichever tier is newest with no tier-awareness, so a healthy WEEKLY snapshot would be >26 h old six days in seven and alarm (landmine recorded in the constant's comment, hub v0.75.0); (b) enumerate the state that exists ONLY in the whole-guest snapshot — guest rootfs, `controller.yaml`, `/etc/felhom-bootstrap`, systemd units, offbox SSH key, `settings.json` — and confirm none of it is catastrophic at 7 days old; (c) update `07-backup-architecture.md`. |
|
||||
| R-83 | **Ratify or retire `07-backup-architecture.md`** | S | idea | DRAFT since 2026-07-14, verified against controller **v0.132.0** — now 41 versions stale (live is v0.173.0) — and cited as authoritative meanwhile. Either ratify it against current reality or mark it superseded; per the new CONTEXT ruling it must in any case carry an honest status header naming its verified-against version. |
|
||||
| R-84 | **Persist the agent's backup `Store`** | S | idea — the CAUSE-fix behind R-81 | `felhom-agent/internal/backup/store.go` is in-memory by design ("lost on restart; the cadence re-populates"), so every agent restart blanks `backups` in the host-report until the next run. R-81 makes the hub interpret that correctly; this makes the report **truthful** rather than merely defensively interpreted. Either persist latest-per-target to the agent state dir (crash-safe journal/marker patterns already exist) or have the collector fall back to querying PVE for the newest archive on the target — the query path is arguably better, being ground truth rather than remembered state, and would also have surfaced the R-82 PBS gap. Small in code, but it is an agent deploy plus a MinAgent floor, so it is its own task. |
|
||||
| R-76 | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** | S | idea (surfaced by the R-75 spike, 2026-07-26) | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say the wrapper fails to achieve the intended mode either. **(b)** `import/calibre` is live on demo-felhom at `755 1000:1000` where every sibling is `2775 root:1000` — with `media/books` on the same box, same app, same deploy, at `2775` as the control, and no parser asymmetry (checked: `ParseComposeUserdataMounts` picks up both calibre-web binds). Consistent with the consuming app rewriting the mode of its own ingest dir after the deploy belt sets it; **not confirmed causally**. Together they mean **a drop-zone directory's mode is not stable against either the customer or the consuming app** — verify before building anything that assumes `import/*` stays 2775. Flips no capability-map row today (latent); would become customer-visible the moment a non-1000 content app enters the catalog |
|
||||
| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18)** | **PHYSICAL CLOSURE 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the generic pairing ISO v1.20.0 (`--loader mkimage`, SB off) **booted the very AMI board that F1 blocked**, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware and the rehearsal dependency on this item is discharged.** Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → **SB must be OFF**. The installer's GRUB menu still offers interactive installers → new item **R-38**. — Prior: **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needLine truncated
|
||||
| R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v <volumeName>:/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. **2026-07-18 rehearsal note:** the run's finding list independently re-raised "assign the pre-existing `docker_run_volume_path_gate` failure its ID so red stops normalizing" — **that is this item; no second ID was minted.** |
|
||||
|
||||
Reference in new issue
Block a user