admin/felhom-controller
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: REPORT for controller v0.178.0 (R-88 Part 2 + R-97c)

Browse Source 
Records the valve-licence table, why a missing field means legacy rather than
unknown, and the near-miss: TieredBackend is runtime-asserted, so the signature
change built and vetted clean while the adapter silently stopped satisfying it.
This commit is contained in:
authentik Default Admin
2026-07-27 18:12:35 +02:00
parent 86ea482fc1
commit ca013c8d27
1 changed files with 70 additions and 104 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+70 -104
View File
@@ -1,118 +1,84 @@
# REPORT — R-97: a failing backup is HEARD, and stops blaming the apps (v0.177.0) (2026-07-27) # REPORT — R-88 Part 2 (controller) + R-97c follow-through (v0.178.0) (2026-07-27)
**Overwritten** per the standing rule. Controller **v0.176.0 → v0.177.0**; **hub v0.77.0 → v0.78.0** **Overwritten** per the standing rule. Controller **v0.177.0 → v0.178.0**.
(required — the controller emits event types the hub must allowlist). **MinAgent unchanged.** **MinAgent: 0.105.0** — the `age_state` semantics. Deployed AFTER agent v0.105.0 on both boxes.
## Part 0 — ops ## What changed
| Item | Result | `scheduledRunAllowed` fired the safety valve on ANY nil age. It now requires a licence:
|---|---|
| **0.1 swap on ep0** | **4 GiB live and persistent.** `swapon --show` → `/swapfile file 4G`; `free -h` → `Swap: 4.0Gi`. Persistence verified **without rebooting**: `systemd-fstab-generator` produced `swapfile.swap` (`RequiredBy=swap.target`, `is-enabled: generated`, `ActiveState=active`), and `findmnt --verify --fstab` → **0 parse errors, 0 errors**. `vm.swappiness` **unchanged at the default 60** — no sysctl override added. Done ahead of Sunday's first-ever GC. |
| **0.2 R-95 mitigation** | **Snapshot plan IS set** on `storage-box-pool-1` (611714): `{"max_snapshots":7,"minute":0,"hour":0,"day_of_week":null,"day_of_month":null}` → **daily at 00:00, keep 7**. But **0 snapshots exist and `size_snapshots=0`** — it is ARMED, not yet PROVEN; the first fires tonight. The **root exposure remains open**: subaccounts are still `readonly=False` and `restic forget --prune` still runs from the box. |
| **0.3 R-91 gate** | **NOT satisfied — do not delete.** demo-felhom's newest offsite snapshot is still `2026-07-26T12:21:48Z`, i.e. **before** the volume migration (attached 2026-07-27 06:29:40 UTC). demo-hp has two post-migration (07-27 08:25:47Z, 09:37:29Z). `/srv/pbs-felhom` remains 13 G. |
| **0.4 residue** | Removed `/root/.ssh/known_hosts.pre-storagebox.bak` and stripped the 3 `u629193` lines from `/root/.bash_history` (0 remaining). `CONTEXT.md` corrected: the datastore is `/mnt/pbs-datastore`, not `/srv/pbs-felhom`. |
| **0.4 R-88 watcher** | **The first in-window scheduled backup has NOT landed, and could not have.** The window is 02:30–06:30 UTC and the fix deployed at 14:23 UTC, so the next opportunity is 2026-07-28. Last vzdump of any kind: the incident's `07:13:06Z FAIL`. This is **timing, not health** — reporting it as "quiet therefore fine" would be the absent-log-line fallacy the new rule 3 forbids. |
## The cooldown keying finding | `age_state` | licenses the valve? | why |
**Per-event-type only, and it would have masked one tier with the other.** `dispatcher.go:233` keyed
the operator cooldown `customerID + ":" + eventType` with a hardcoded 1 h window. Correct for an event
describing ONE thing; wrong for one describing ONE TIER when a box has two — `felhom-pbs` failing at
09:00 would swallow `local` failing at 09:20 for the rest of the hour.
Two clarifications that changed the fix's shape:
- it gates the **email only** — `SaveEvent` runs before dispatch, so both events are always stored;
- widening the key for **everything** would be a flood, not a fix (one hourly `app_start_failed`
becomes one per app).
So the fix is **narrow**: `cooldownTierSuffix` appends `:<tier>` **only when the event details carry a
`tier`**. No existing event type's behaviour changes. Pinned by `TestCooldownTierSuffix_EmptyForEverythingElse`.
## The grace window (Part 2)
**`quiesceAlarmGrace = 180 s`**, derived rather than picked round:
- the **deploy flow already allows 120 s** for a stack to come up healthy — the project's own existing
answer to "how long is too long";
- the slowest catalog healthcheck `start_period` is **Mealie's 60 s**, after which a couple of check
intervals must still elapse before a verdict means anything.
180 s clears both with margin, and is deliberately not longer: the app-state scan runs on its own
cadence, so a genuinely dead app alarms on the first scan after the window closes. The cost is a
**bounded delay** in reporting a real failure, never its loss.
**Why a cycle-keyed window and not a state test:** v0.164.0's filter is
`IsDownState(st.State) && st.State != StateStopped` and suppresses *deliberately stopped* apps.
BookStack alarmed because the third cycle caught it **mid-restart** — `starting`/`unhealthy`, not
`StateStopped`. No state classification can distinguish "restarting because a backup stopped me" from
"restarting because I keep crashing"; the distinguishing fact is that *we* stopped it, and we know it.
Consumed at the **same single derivation point** (`classifyRunStates`) that computes both the banner
dead-list and the notifier Down-set — still one place.
## Tests — controller 27 pkg ok / hub 17 pkg ok, 0 failed; `build`/`vet`/`test` each rc=0, run separately, vet unpiped
Controller +9 (8 quiesce + 1 wiring reachability); hub +8. Red-proofs observed and restored:
| Scenario | Mutation | Observed failure |
|---|---|---| |---|---|---|
| **C** | key the breaker on a constant (one global failure record) | `both tiers must be reported; got local=1 felhom-pbs=0 — one broken tier must not mask another` | | `absent` | **yes** | the agent looked; nothing is there |
| **E** | `SuppressedStacks` returns nil (pre-fix shape) | `R-97b: the apps we stopped for a backup must not be reported down; bookstack still alarmed` | | `unknown` | **no** | unreadable storage — the whole point of R-88 |
| **F** | suppression never expires (permanent) | `CONTRACT VIOLATED: suppression must EXPIRE — bookstack is still suppressed 3m1s after the cycle; permanent suppression is a silent real alarm` | | `known` | n/a | a real age; the comparison decides |
| *(empty)* | **yes** | pre-v0.105.0 agent — deliberate, see below |
## The end-to-end event evidence **A missing field means LEGACY, not unknown.** Reading an old agent's silence as "unknown" looks
safer and regresses Scenario D: the valve would stop firing on every un-upgraded box, so a genuinely
new box would never take its first backup outside its window and nobody would notice for weeks. The
MinAgent floor drives the upgrade; the valve is not the place to force it. Logged **once** per
process (the `logTierDegradeOnce` shape). An unrecognised FUTURE value also maps to legacy — a newer
agent inventing a fourth state must not inherit "unknown" semantics from a controller that has never
heard of it.
**Induced deliberately, and it travelled.** A real `whole_guest_backup_failed` was POSTed from **inside **Untouched, as required:** the R-88 Part 1 breaker and its timings, `dropBackedOffTiers`, the window
guest 9201**, with the controller's own hub URL and API key, over the live `/api/v1/event` route — the bounds, and `TriggerNow` (still ungated by everything).
exact path `PushEvent` uses. Message prefixed `[R-97a VERIFICATION - induced by Claude Code, NOT a real
fault]`, details `{"tier":"felhom-pbs",...}`. ## The near-miss worth reporting
`TieredBackend` is satisfied by a **runtime** type assertion in `resolveDueTiers`
(`l.backend.(TieredBackend)`), not at compile time. When `DueFor`'s signature changed, the whole repo
**built and vetted clean** while `quiesceBackend` silently stopped satisfying the interface. That
would have shipped, degraded every box to the untargeted single-tier path, and lost R-82's multi-tier
backups entirely — **with no error anywhere**. `TestQuiesceBackendSatisfiesTieredBackend` is now the
compile-time witness; it caught this during the change. Sixth instance of the inert-seam class.
## R-97c follow-through
The comment in `internal/notify/notifier.go` claiming these event types are operator-only "because
they have no customerMessages entry" was **wrong** and is corrected. The hub falls back to the raw
message when the entry is missing, and the only customer gate is `prefs.EnabledEvents` — pure
configuration. Enforcement is hub-side `operatorOnlyEvents` (hub >= v0.79.0).
## Tests — 27 packages ok, 0 failed; `build`/`vet`/`test` each rc=0, run separately, vet unpiped
+7 (6 age-state + 1 interface witness). Red-proofs observed and restored:
| Scenario | Mutation | Observed |
|---|---|---|
| **A** | `unknown` licenses the valve (pre-fix) | `R-88 Part 2: an UNKNOWN age bypassed the backup window and stopped 1 stack(s) — an unreadable storage must not masquerade as a first-ever backup` |
| **B** | drop `absent` from the licence | `a genuine first-ever backup (absent) must RUN outside the window; 0 stack(s) stopped — the safety valve was lost and a new box would starve` |
| **C** | treat legacy as unknown | `C: a pre-v0.105.0 agent must behave EXACTLY as before — nil age fires the valve. 0 stack(s) stopped; an un-upgraded box just silently stopped backing up outside its window` |
**C asserts behaviour, not the log line** — a controller that logs the degrade and then defers would
pass a log-only assertion while silently changing behaviour on every un-upgraded box.
## Deployed + verified
Agent v0.105.0 first, then controller v0.178.0 — demo-hp, then demo-felhom. Both `Up (healthy)`.
**The MinAgent floor is satisfied, proven positively:** zero `pre-v0.105.0` degrade warnings on
either box (they would fire on the first due tier against an old agent), and the live wire carries
the field:
``` ```
1. the new type → HTTP 200 {"ok":true} tier=local due=True age_state=known age_seconds=91097 reason=older than cadence
2. CONTROL: bogus type → HTTP 400 "Invalid event_type: whole_guest_backup_bogus" tier=felhom-pbs due=False age_state=known age_seconds=100172 reason=within cadence window
``` ```
The control matters: without it, a 200 is equally consistent with "my type is allowlisted" and "the **And the gate is demonstrably working on a known age:** `local` is genuinely due (25.3 h against a
gate is off". At the far end, from a **freshly copied** hub DB (mtime checked — a stale copy already 24 h cadence) at 18:11 Budapest — outside `[04:30, 08:30)` and well under the 48 h valve threshold —
produced one confident wrong answer today): so it correctly **defers**. No vzdump has run since the incident's last failure at 07:13:06 UTC.
Zero spurious backoff/suppression lines, zero startup errors on both boxes.
```
events: 2026-07-27 15:04:57 error whole_guest_backup_failed [R-97a VERIFICATION …]
notification_log: 2026-07-27 15:04:58 whole_guest_backup_failed channel=operator status=sent
OPERATOR rows: 1 CUSTOMER rows: 0
```
**Operator notified, customer not** — the operator-only property confirmed positively, not assumed.
No storage target was altered and nothing was pointed at a dead endpoint, so nothing needed restoring.
## Deployed
- **hub v0.78.0** — GitOps: manifest bumped, ArgoCD hard-refresh + deliberate sync → `Synced`,
rollout complete, pod image `felhom-hub:0.78.0`, startup clean. **Deployed FIRST**, so the
controller never emits a type the hub would 400.
- **controller v0.177.0** — demo-hp first, then demo-felhom; both `Up (healthy)`.
Post-deploy on both boxes: **0 spurious suppression/backoff lines, 0 startup errors**, quiesce loop
started, all app stacks up (10 on demo-felhom, 3 on demo-hp). The positive observable still lands —
both tiers polled every 5 min with `200`s in `pveproxy/access.log`.
## NOT yet live-validated ## NOT yet live-validated
- **The quiesce → notify link is UNIT-PROVEN ONLY.** The hub half is proven live end-to-end with a - **No `unknown` state has ever been observed on real hardware** — both boxes report `known`. The
control (above), and the adapter's reachability is pinned by `TestQuiesceTierNotifierIsWired` — but fix's central path is unit-proven only; reproducing it live means making PBS unreadable, which must
no real quiesce failure has fired the seam on hardware, because inducing one means an app outage on not be done to ep0.
a demo box. The join is proven by test + inspection, not observation. - **No R-88 backoff has ever armed on real hardware** (carried forward, unchanged).
- **No R-88 backoff has ever armed on real hardware.** Both boxes are healthy, so the breaker's - **R-97b's app-alarm suppression has never fired live** — it needs a real quiesce cycle.
behaviour under a failing tier still rests on unit tests alone. Unchanged by this task. - **The legacy-agent path is now unobservable on this fleet** — both agents are v0.105.0, so
- **R-97b's suppression has never fired live** — it needs a real quiesce cycle, i.e. tomorrow's Scenario C rests on tests alone from here.
in-window backup at the earliest. - demo-felhom's first in-window scheduled backup lands 2026-07-28 at the earliest.
- **The 180 s grace has not been measured against a real restart** on these boxes; it is derived from
the deploy flow's 120 s and Mealie's 60 s, not observed.
- **The snapshot plan (0.2) is armed but has taken 0 snapshots** — first one due tonight at 00:00.
- **demo-felhom's first in-window scheduled backup** under v0.176.0/v0.177.0 lands 2026-07-28 at the
earliest.
## OPEN-ITEMS rows touched
**Closed:** R-97 (both halves shipped) · R-96 (three standing rules committed to `CLAUDE.md`) ·
R-90 (swap done; the CX33 rescale stays open as the durable fix).
**Re-ranked:** R-95 split — mitigation ARMED (awaiting first snapshot), root exposure still READY.
**Unchanged:** R-91 (gate not satisfied) · R-88b (agent wire change) · R-86 · R-87.