docs: R-88 Part 1 shipped; Phase 0 root cause; R-97 minted

R-88 split: Part 1 (the failure breaker) SHIPPED in controller v0.176.0 and live
on both boxes; Part 2 (unknown != never) stays OPEN and is agent-side.

Phase 0 established the root cause at source: newestArchiveOn's (time.Time, bool)
signature cannot represent 'unknown', so a storage read ERROR collapses into a
positive 'no successful backup recorded yet'. The errored and genuine-never paths
are byte-identical on the wire, which is why Part 2 cannot be done controller-side.

R-97: the whole-guest backup tier has no failure signal to the hub at all —
internal/quiesce never imports internal/notify, so three failed backups and three
app-stack outages produced zero backup_failed events. Its only trace was a
customer-tier Hungarian app_start_failed for an app the backup itself had stopped.
This commit is contained in:
2026-07-27 16:27:31 +02:00
parent 7826bd7179
commit 65409aecd1
2 changed files with 8 additions and 6 deletions
+5 -4
View File
@@ -8,7 +8,8 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
| ID | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|
| **R-88** | Unreachable backup target reads as *no backup exists* → quiesce loop, no backoff, maintenance window bypassed | **READY #1** | — | Make an unreachable read UNKNOWN; fix the `AgeSecs==nil` window bypass in the same slice | CC |
| **R-88a** | ~~Failing backup re-quiesces every 5 min, no backoff~~ | **SHIPPED** (controller v0.176.0, 2026-07-27) | — | Live on both boxes; breaker 15m→4h, per-tier, never permanent | |
| **R-88b** | `/backup/due` cannot say *unknown* — "read errored" and "never backed up" are byte-identical, so nil still bypasses the window gate | **READY #1** | — | Agent wire change: give unknown its own representation; compat rule both ways + MinAgent floor | CC |
| **R-95** | restic offsite credential **can delete** (`readonly=False`, `forget --prune` runs from the box); SFTP cannot express append-only | **READY #2** | — | Snapshot plan as the stopgap (row below), then split prune off-box or move to REST `--append-only` | CC |
| **R-94** | Hub hands out host-install `1.19.0`; `1.20.0` is what carries R-82's backup default | **READY #3** | — | Bump `configs.go:28`, and stop hand-syncing a version constant across repos | CC |
| **R-86** | Restore-tests are interval-scheduled, not backup-aligned | **READY #4** | R-90 (ep0 headroom) informs cadence | Trigger a tier ~24 h after **its own** newest archive | CC |
@@ -27,9 +28,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
## Why the READY rows rank this way
1. **R-88** — the only item causing *active harm*: it stops every customer app stack every 5 minutes,
with no backoff, and the `AgeSecs==nil` path also disables the maintenance-window gate, so it runs
in business hours. Its trigger is a PBS read failure, which R-90 shows is live.
1. **R-88b** — the loop itself is fixed (R-88a, v0.176.0), so the active harm is gone; what remains is
that an unknown still fires the safety valve, so a read failure can take **one** out-of-window
quiesce. Bounded now, but it is the fourth appearance of this class and the only one still open.
2. **R-95** — the largest *data* exposure: the tier holding the customer's documents and photos is the
one whose credential can delete, and the mitigation is a console click nobody has made.
3. **R-94** — a one-line constant, but until it moves every hub-driven install gets the pre-R-82