admin
ca013c8d27
Records the valve-licence table, why a missing field means legacy rather than unknown, and the near-miss: TieredBackend is runtime-asserted, so the signature change built and vetted clean while the adapter silently stopped satisfying it.
85 lines
4.9 KiB
Markdown
85 lines
4.9 KiB
Markdown
# REPORT — R-88 Part 2 (controller) + R-97c follow-through (v0.178.0) (2026-07-27)
|
|
|
|
**Overwritten** per the standing rule. Controller **v0.177.0 → v0.178.0**.
|
|
**MinAgent: 0.105.0** — the `age_state` semantics. Deployed AFTER agent v0.105.0 on both boxes.
|
|
|
|
## What changed
|
|
|
|
`scheduledRunAllowed` fired the safety valve on ANY nil age. It now requires a licence:
|
|
|
|
| `age_state` | licenses the valve? | why |
|
|
|---|---|---|
|
|
| `absent` | **yes** | the agent looked; nothing is there |
|
|
| `unknown` | **no** | unreadable storage — the whole point of R-88 |
|
|
| `known` | n/a | a real age; the comparison decides |
|
|
| *(empty)* | **yes** | pre-v0.105.0 agent — deliberate, see below |
|
|
|
|
**A missing field means LEGACY, not unknown.** Reading an old agent's silence as "unknown" looks
|
|
safer and regresses Scenario D: the valve would stop firing on every un-upgraded box, so a genuinely
|
|
new box would never take its first backup outside its window and nobody would notice for weeks. The
|
|
MinAgent floor drives the upgrade; the valve is not the place to force it. Logged **once** per
|
|
process (the `logTierDegradeOnce` shape). An unrecognised FUTURE value also maps to legacy — a newer
|
|
agent inventing a fourth state must not inherit "unknown" semantics from a controller that has never
|
|
heard of it.
|
|
|
|
**Untouched, as required:** the R-88 Part 1 breaker and its timings, `dropBackedOffTiers`, the window
|
|
bounds, and `TriggerNow` (still ungated by everything).
|
|
|
|
## The near-miss worth reporting
|
|
|
|
`TieredBackend` is satisfied by a **runtime** type assertion in `resolveDueTiers`
|
|
(`l.backend.(TieredBackend)`), not at compile time. When `DueFor`'s signature changed, the whole repo
|
|
**built and vetted clean** while `quiesceBackend` silently stopped satisfying the interface. That
|
|
would have shipped, degraded every box to the untargeted single-tier path, and lost R-82's multi-tier
|
|
backups entirely — **with no error anywhere**. `TestQuiesceBackendSatisfiesTieredBackend` is now the
|
|
compile-time witness; it caught this during the change. Sixth instance of the inert-seam class.
|
|
|
|
## R-97c follow-through
|
|
|
|
The comment in `internal/notify/notifier.go` claiming these event types are operator-only "because
|
|
they have no customerMessages entry" was **wrong** and is corrected. The hub falls back to the raw
|
|
message when the entry is missing, and the only customer gate is `prefs.EnabledEvents` — pure
|
|
configuration. Enforcement is hub-side `operatorOnlyEvents` (hub >= v0.79.0).
|
|
|
|
## Tests — 27 packages ok, 0 failed; `build`/`vet`/`test` each rc=0, run separately, vet unpiped
|
|
|
|
+7 (6 age-state + 1 interface witness). Red-proofs observed and restored:
|
|
|
|
| Scenario | Mutation | Observed |
|
|
|---|---|---|
|
|
| **A** | `unknown` licenses the valve (pre-fix) | `R-88 Part 2: an UNKNOWN age bypassed the backup window and stopped 1 stack(s) — an unreadable storage must not masquerade as a first-ever backup` |
|
|
| **B** | drop `absent` from the licence | `a genuine first-ever backup (absent) must RUN outside the window; 0 stack(s) stopped — the safety valve was lost and a new box would starve` |
|
|
| **C** | treat legacy as unknown | `C: a pre-v0.105.0 agent must behave EXACTLY as before — nil age fires the valve. 0 stack(s) stopped; an un-upgraded box just silently stopped backing up outside its window` |
|
|
|
|
**C asserts behaviour, not the log line** — a controller that logs the degrade and then defers would
|
|
pass a log-only assertion while silently changing behaviour on every un-upgraded box.
|
|
|
|
## Deployed + verified
|
|
|
|
Agent v0.105.0 first, then controller v0.178.0 — demo-hp, then demo-felhom. Both `Up (healthy)`.
|
|
|
|
**The MinAgent floor is satisfied, proven positively:** zero `pre-v0.105.0` degrade warnings on
|
|
either box (they would fire on the first due tier against an old agent), and the live wire carries
|
|
the field:
|
|
|
|
```
|
|
tier=local due=True age_state=known age_seconds=91097 reason=older than cadence
|
|
tier=felhom-pbs due=False age_state=known age_seconds=100172 reason=within cadence window
|
|
```
|
|
|
|
**And the gate is demonstrably working on a known age:** `local` is genuinely due (25.3 h against a
|
|
24 h cadence) at 18:11 Budapest — outside `[04:30, 08:30)` and well under the 48 h valve threshold —
|
|
so it correctly **defers**. No vzdump has run since the incident's last failure at 07:13:06 UTC.
|
|
Zero spurious backoff/suppression lines, zero startup errors on both boxes.
|
|
|
|
## NOT yet live-validated
|
|
|
|
- **No `unknown` state has ever been observed on real hardware** — both boxes report `known`. The
|
|
fix's central path is unit-proven only; reproducing it live means making PBS unreadable, which must
|
|
not be done to ep0.
|
|
- **No R-88 backoff has ever armed on real hardware** (carried forward, unchanged).
|
|
- **R-97b's app-alarm suppression has never fired live** — it needs a real quiesce cycle.
|
|
- **The legacy-agent path is now unobservable on this fleet** — both agents are v0.105.0, so
|
|
Scenario C rests on tests alone from here.
|
|
- demo-felhom's first in-window scheduled backup lands 2026-07-28 at the earliest.
|