docs: REPORT for controller v0.178.0 (R-88 Part 2 + R-97c)
Records the valve-licence table, why a missing field means legacy rather than unknown, and the near-miss: TieredBackend is runtime-asserted, so the signature change built and vetted clean while the adapter silently stopped satisfying it.
This commit is contained in:
@@ -1,118 +1,84 @@
|
||||
# REPORT — R-97: a failing backup is HEARD, and stops blaming the apps (v0.177.0) (2026-07-27)
|
||||
# REPORT — R-88 Part 2 (controller) + R-97c follow-through (v0.178.0) (2026-07-27)
|
||||
|
||||
**Overwritten** per the standing rule. Controller **v0.176.0 → v0.177.0**; **hub v0.77.0 → v0.78.0**
|
||||
(required — the controller emits event types the hub must allowlist). **MinAgent unchanged.**
|
||||
**Overwritten** per the standing rule. Controller **v0.177.0 → v0.178.0**.
|
||||
**MinAgent: 0.105.0** — the `age_state` semantics. Deployed AFTER agent v0.105.0 on both boxes.
|
||||
|
||||
## Part 0 — ops
|
||||
## What changed
|
||||
|
||||
| Item | Result |
|
||||
|---|---|
|
||||
| **0.1 swap on ep0** | **4 GiB live and persistent.** `swapon --show` → `/swapfile file 4G`; `free -h` → `Swap: 4.0Gi`. Persistence verified **without rebooting**: `systemd-fstab-generator` produced `swapfile.swap` (`RequiredBy=swap.target`, `is-enabled: generated`, `ActiveState=active`), and `findmnt --verify --fstab` → **0 parse errors, 0 errors**. `vm.swappiness` **unchanged at the default 60** — no sysctl override added. Done ahead of Sunday's first-ever GC. |
|
||||
| **0.2 R-95 mitigation** | **Snapshot plan IS set** on `storage-box-pool-1` (611714): `{"max_snapshots":7,"minute":0,"hour":0,"day_of_week":null,"day_of_month":null}` → **daily at 00:00, keep 7**. But **0 snapshots exist and `size_snapshots=0`** — it is ARMED, not yet PROVEN; the first fires tonight. The **root exposure remains open**: subaccounts are still `readonly=False` and `restic forget --prune` still runs from the box. |
|
||||
| **0.3 R-91 gate** | **NOT satisfied — do not delete.** demo-felhom's newest offsite snapshot is still `2026-07-26T12:21:48Z`, i.e. **before** the volume migration (attached 2026-07-27 06:29:40 UTC). demo-hp has two post-migration (07-27 08:25:47Z, 09:37:29Z). `/srv/pbs-felhom` remains 13 G. |
|
||||
| **0.4 residue** | Removed `/root/.ssh/known_hosts.pre-storagebox.bak` and stripped the 3 `u629193` lines from `/root/.bash_history` (0 remaining). `CONTEXT.md` corrected: the datastore is `/mnt/pbs-datastore`, not `/srv/pbs-felhom`. |
|
||||
| **0.4 R-88 watcher** | **The first in-window scheduled backup has NOT landed, and could not have.** The window is 02:30–06:30 UTC and the fix deployed at 14:23 UTC, so the next opportunity is 2026-07-28. Last vzdump of any kind: the incident's `07:13:06Z FAIL`. This is **timing, not health** — reporting it as "quiet therefore fine" would be the absent-log-line fallacy the new rule 3 forbids. |
|
||||
`scheduledRunAllowed` fired the safety valve on ANY nil age. It now requires a licence:
|
||||
|
||||
## The cooldown keying finding
|
||||
|
||||
**Per-event-type only, and it would have masked one tier with the other.** `dispatcher.go:233` keyed
|
||||
the operator cooldown `customerID + ":" + eventType` with a hardcoded 1 h window. Correct for an event
|
||||
describing ONE thing; wrong for one describing ONE TIER when a box has two — `felhom-pbs` failing at
|
||||
09:00 would swallow `local` failing at 09:20 for the rest of the hour.
|
||||
|
||||
Two clarifications that changed the fix's shape:
|
||||
- it gates the **email only** — `SaveEvent` runs before dispatch, so both events are always stored;
|
||||
- widening the key for **everything** would be a flood, not a fix (one hourly `app_start_failed`
|
||||
becomes one per app).
|
||||
|
||||
So the fix is **narrow**: `cooldownTierSuffix` appends `:<tier>` **only when the event details carry a
|
||||
`tier`**. No existing event type's behaviour changes. Pinned by `TestCooldownTierSuffix_EmptyForEverythingElse`.
|
||||
|
||||
## The grace window (Part 2)
|
||||
|
||||
**`quiesceAlarmGrace = 180 s`**, derived rather than picked round:
|
||||
- the **deploy flow already allows 120 s** for a stack to come up healthy — the project's own existing
|
||||
answer to "how long is too long";
|
||||
- the slowest catalog healthcheck `start_period` is **Mealie's 60 s**, after which a couple of check
|
||||
intervals must still elapse before a verdict means anything.
|
||||
|
||||
180 s clears both with margin, and is deliberately not longer: the app-state scan runs on its own
|
||||
cadence, so a genuinely dead app alarms on the first scan after the window closes. The cost is a
|
||||
**bounded delay** in reporting a real failure, never its loss.
|
||||
|
||||
**Why a cycle-keyed window and not a state test:** v0.164.0's filter is
|
||||
`IsDownState(st.State) && st.State != StateStopped` and suppresses *deliberately stopped* apps.
|
||||
BookStack alarmed because the third cycle caught it **mid-restart** — `starting`/`unhealthy`, not
|
||||
`StateStopped`. No state classification can distinguish "restarting because a backup stopped me" from
|
||||
"restarting because I keep crashing"; the distinguishing fact is that *we* stopped it, and we know it.
|
||||
Consumed at the **same single derivation point** (`classifyRunStates`) that computes both the banner
|
||||
dead-list and the notifier Down-set — still one place.
|
||||
|
||||
## Tests — controller 27 pkg ok / hub 17 pkg ok, 0 failed; `build`/`vet`/`test` each rc=0, run separately, vet unpiped
|
||||
|
||||
Controller +9 (8 quiesce + 1 wiring reachability); hub +8. Red-proofs observed and restored:
|
||||
|
||||
| Scenario | Mutation | Observed failure |
|
||||
| `age_state` | licenses the valve? | why |
|
||||
|---|---|---|
|
||||
| **C** | key the breaker on a constant (one global failure record) | `both tiers must be reported; got local=1 felhom-pbs=0 — one broken tier must not mask another` |
|
||||
| **E** | `SuppressedStacks` returns nil (pre-fix shape) | `R-97b: the apps we stopped for a backup must not be reported down; bookstack still alarmed` |
|
||||
| **F** | suppression never expires (permanent) | `CONTRACT VIOLATED: suppression must EXPIRE — bookstack is still suppressed 3m1s after the cycle; permanent suppression is a silent real alarm` |
|
||||
| `absent` | **yes** | the agent looked; nothing is there |
|
||||
| `unknown` | **no** | unreadable storage — the whole point of R-88 |
|
||||
| `known` | n/a | a real age; the comparison decides |
|
||||
| *(empty)* | **yes** | pre-v0.105.0 agent — deliberate, see below |
|
||||
|
||||
## The end-to-end event evidence
|
||||
**A missing field means LEGACY, not unknown.** Reading an old agent's silence as "unknown" looks
|
||||
safer and regresses Scenario D: the valve would stop firing on every un-upgraded box, so a genuinely
|
||||
new box would never take its first backup outside its window and nobody would notice for weeks. The
|
||||
MinAgent floor drives the upgrade; the valve is not the place to force it. Logged **once** per
|
||||
process (the `logTierDegradeOnce` shape). An unrecognised FUTURE value also maps to legacy — a newer
|
||||
agent inventing a fourth state must not inherit "unknown" semantics from a controller that has never
|
||||
heard of it.
|
||||
|
||||
**Induced deliberately, and it travelled.** A real `whole_guest_backup_failed` was POSTed from **inside
|
||||
guest 9201**, with the controller's own hub URL and API key, over the live `/api/v1/event` route — the
|
||||
exact path `PushEvent` uses. Message prefixed `[R-97a VERIFICATION - induced by Claude Code, NOT a real
|
||||
fault]`, details `{"tier":"felhom-pbs",...}`.
|
||||
**Untouched, as required:** the R-88 Part 1 breaker and its timings, `dropBackedOffTiers`, the window
|
||||
bounds, and `TriggerNow` (still ungated by everything).
|
||||
|
||||
## The near-miss worth reporting
|
||||
|
||||
`TieredBackend` is satisfied by a **runtime** type assertion in `resolveDueTiers`
|
||||
(`l.backend.(TieredBackend)`), not at compile time. When `DueFor`'s signature changed, the whole repo
|
||||
**built and vetted clean** while `quiesceBackend` silently stopped satisfying the interface. That
|
||||
would have shipped, degraded every box to the untargeted single-tier path, and lost R-82's multi-tier
|
||||
backups entirely — **with no error anywhere**. `TestQuiesceBackendSatisfiesTieredBackend` is now the
|
||||
compile-time witness; it caught this during the change. Sixth instance of the inert-seam class.
|
||||
|
||||
## R-97c follow-through
|
||||
|
||||
The comment in `internal/notify/notifier.go` claiming these event types are operator-only "because
|
||||
they have no customerMessages entry" was **wrong** and is corrected. The hub falls back to the raw
|
||||
message when the entry is missing, and the only customer gate is `prefs.EnabledEvents` — pure
|
||||
configuration. Enforcement is hub-side `operatorOnlyEvents` (hub >= v0.79.0).
|
||||
|
||||
## Tests — 27 packages ok, 0 failed; `build`/`vet`/`test` each rc=0, run separately, vet unpiped
|
||||
|
||||
+7 (6 age-state + 1 interface witness). Red-proofs observed and restored:
|
||||
|
||||
| Scenario | Mutation | Observed |
|
||||
|---|---|---|
|
||||
| **A** | `unknown` licenses the valve (pre-fix) | `R-88 Part 2: an UNKNOWN age bypassed the backup window and stopped 1 stack(s) — an unreadable storage must not masquerade as a first-ever backup` |
|
||||
| **B** | drop `absent` from the licence | `a genuine first-ever backup (absent) must RUN outside the window; 0 stack(s) stopped — the safety valve was lost and a new box would starve` |
|
||||
| **C** | treat legacy as unknown | `C: a pre-v0.105.0 agent must behave EXACTLY as before — nil age fires the valve. 0 stack(s) stopped; an un-upgraded box just silently stopped backing up outside its window` |
|
||||
|
||||
**C asserts behaviour, not the log line** — a controller that logs the degrade and then defers would
|
||||
pass a log-only assertion while silently changing behaviour on every un-upgraded box.
|
||||
|
||||
## Deployed + verified
|
||||
|
||||
Agent v0.105.0 first, then controller v0.178.0 — demo-hp, then demo-felhom. Both `Up (healthy)`.
|
||||
|
||||
**The MinAgent floor is satisfied, proven positively:** zero `pre-v0.105.0` degrade warnings on
|
||||
either box (they would fire on the first due tier against an old agent), and the live wire carries
|
||||
the field:
|
||||
|
||||
```
|
||||
1. the new type → HTTP 200 {"ok":true}
|
||||
2. CONTROL: bogus type → HTTP 400 "Invalid event_type: whole_guest_backup_bogus"
|
||||
tier=local due=True age_state=known age_seconds=91097 reason=older than cadence
|
||||
tier=felhom-pbs due=False age_state=known age_seconds=100172 reason=within cadence window
|
||||
```
|
||||
|
||||
The control matters: without it, a 200 is equally consistent with "my type is allowlisted" and "the
|
||||
gate is off". At the far end, from a **freshly copied** hub DB (mtime checked — a stale copy already
|
||||
produced one confident wrong answer today):
|
||||
|
||||
```
|
||||
events: 2026-07-27 15:04:57 error whole_guest_backup_failed [R-97a VERIFICATION …]
|
||||
notification_log: 2026-07-27 15:04:58 whole_guest_backup_failed channel=operator status=sent
|
||||
OPERATOR rows: 1 CUSTOMER rows: 0
|
||||
```
|
||||
|
||||
**Operator notified, customer not** — the operator-only property confirmed positively, not assumed.
|
||||
No storage target was altered and nothing was pointed at a dead endpoint, so nothing needed restoring.
|
||||
|
||||
## Deployed
|
||||
|
||||
- **hub v0.78.0** — GitOps: manifest bumped, ArgoCD hard-refresh + deliberate sync → `Synced`,
|
||||
rollout complete, pod image `felhom-hub:0.78.0`, startup clean. **Deployed FIRST**, so the
|
||||
controller never emits a type the hub would 400.
|
||||
- **controller v0.177.0** — demo-hp first, then demo-felhom; both `Up (healthy)`.
|
||||
|
||||
Post-deploy on both boxes: **0 spurious suppression/backoff lines, 0 startup errors**, quiesce loop
|
||||
started, all app stacks up (10 on demo-felhom, 3 on demo-hp). The positive observable still lands —
|
||||
both tiers polled every 5 min with `200`s in `pveproxy/access.log`.
|
||||
**And the gate is demonstrably working on a known age:** `local` is genuinely due (25.3 h against a
|
||||
24 h cadence) at 18:11 Budapest — outside `[04:30, 08:30)` and well under the 48 h valve threshold —
|
||||
so it correctly **defers**. No vzdump has run since the incident's last failure at 07:13:06 UTC.
|
||||
Zero spurious backoff/suppression lines, zero startup errors on both boxes.
|
||||
|
||||
## NOT yet live-validated
|
||||
|
||||
- **The quiesce → notify link is UNIT-PROVEN ONLY.** The hub half is proven live end-to-end with a
|
||||
control (above), and the adapter's reachability is pinned by `TestQuiesceTierNotifierIsWired` — but
|
||||
no real quiesce failure has fired the seam on hardware, because inducing one means an app outage on
|
||||
a demo box. The join is proven by test + inspection, not observation.
|
||||
- **No R-88 backoff has ever armed on real hardware.** Both boxes are healthy, so the breaker's
|
||||
behaviour under a failing tier still rests on unit tests alone. Unchanged by this task.
|
||||
- **R-97b's suppression has never fired live** — it needs a real quiesce cycle, i.e. tomorrow's
|
||||
in-window backup at the earliest.
|
||||
- **The 180 s grace has not been measured against a real restart** on these boxes; it is derived from
|
||||
the deploy flow's 120 s and Mealie's 60 s, not observed.
|
||||
- **The snapshot plan (0.2) is armed but has taken 0 snapshots** — first one due tonight at 00:00.
|
||||
- **demo-felhom's first in-window scheduled backup** under v0.176.0/v0.177.0 lands 2026-07-28 at the
|
||||
earliest.
|
||||
|
||||
## OPEN-ITEMS rows touched
|
||||
|
||||
**Closed:** R-97 (both halves shipped) · R-96 (three standing rules committed to `CLAUDE.md`) ·
|
||||
R-90 (swap done; the CX33 rescale stays open as the durable fix).
|
||||
**Re-ranked:** R-95 split — mitigation ARMED (awaiting first snapshot), root exposure still READY.
|
||||
**Unchanged:** R-91 (gate not satisfied) · R-88b (agent wire change) · R-86 · R-87.
|
||||
- **No `unknown` state has ever been observed on real hardware** — both boxes report `known`. The
|
||||
fix's central path is unit-proven only; reproducing it live means making PBS unreadable, which must
|
||||
not be done to ep0.
|
||||
- **No R-88 backoff has ever armed on real hardware** (carried forward, unchanged).
|
||||
- **R-97b's app-alarm suppression has never fired live** — it needs a real quiesce cycle.
|
||||
- **The legacy-agent path is now unobservable on this fleet** — both agents are v0.105.0, so
|
||||
Scenario C rests on tests alone from here.
|
||||
- demo-felhom's first in-window scheduled backup lands 2026-07-28 at the earliest.
|
||||
|
||||
Reference in New Issue
Block a user