admin
d0cec9d3f3
F-CRIT-2 marked SHIPPED + PROVEN-LIVE: NewestArchiveTime now counts only plausibly-complete entries (measured 1 MiB floor), verified by replaying campaign fault 2 on demo-hp in both directions. Files two follow-ups: - R-99: prune never REMOVES phantoms. It does NOT count them toward keep-last (dry-run kept 2 real + the phantom), so there is no retention/data-loss bug — but they accumulate one per aborted upload. - F-CRIT-1: filed READY-HIGHEST. Campaign 8's other HIGH finding, untouched by this fix.
42 lines
6.0 KiB
Markdown
42 lines
6.0 KiB
Markdown
# OPEN-ITEMS — the single source of truth for open work
|
|
|
|
**Rebuilt 2026-07-27 by read-only triage.** `ROADMAP.md` keeps the full history and reasoning; this
|
|
page keeps only what is **open**, and it is the file to read first. `REPORT.md` is per-session and
|
|
**overwritten** — nothing durable may live only there.
|
|
|
|
State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row has an owner.
|
|
|
|
| ID | What | State | Blocked on | Next action | Owner |
|
|
|---|---|---|---|---|---|
|
|
| **R-88a** | ~~Failing backup re-quiesces every 5 min, no backoff~~ | **SHIPPED** (controller v0.176.0, 2026-07-27) | — | Live on both boxes; breaker 15m→4h, per-tier, never permanent | — |
|
|
| **R-88b** | ~~`/backup/due` cannot say *unknown*~~ | **SHIPPED + PROVEN-LIVE** (agent v0.105.0 + controller v0.178.0, 2026-07-27) | — | `age_state=unknown` captured on real hardware during a deliberate ep0 outage; controller deferred, **zero app stacks stopped** | — |
|
|
| **R-95** | restic offsite credential **can delete** (`readonly=False`, `forget --prune` runs from the box); SFTP cannot express append-only | **READY #1** | — | Root exposure still open. Mitigation now ARMED — split prune off-box or move to REST `--append-only` | CC |
|
|
| **R-94** | Hub hands out host-install `1.19.0`; `1.20.0` is what carries R-82's backup default | **READY #2** | — | Bump `configs.go:28`, and stop hand-syncing a version constant across repos | CC |
|
|
| **R-86** | Restore-tests are interval-scheduled, not backup-aligned | **READY #3** | R-90 (ep0 headroom) informs cadence | Trigger a tier ~24 h after **its own** newest archive | CC |
|
|
| **R-87** | The restic tier is never restore-tested | **READY #4** | — | Design a controller-side test (no scratch-guest analogue transfers) | CC |
|
|
| — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC |
|
|
| — | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | WAITING-ON-OPERATOR | operator console | Delete the box | operator |
|
|
| **R-90** | ep0 RAM headroom — 4 GiB swap **survived its first reboot** 2026-07-27; 3.8 GB RAM unchanged | **BLOCKED** (interim proven) | Hetzner CX33 availability — **confirmed unavailable even powered OFF**, so it is the Cost-Optimized "Limited availability", not the power state | Re-check CX33; escape hatch if urgent = CPX/CCX lines (no availability warning, higher cost) | operator |
|
|
| **R-91** | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC |
|
|
| — | First-ever **GC** on `felhom-offsite` (armed today 13:11 UTC, never run) | WATCHING | schedule | **Sun 2026-08-02 04:30 UTC** — confirm it completes | CC |
|
|
| — | demo-felhom's next weekly PBS backup (newest is 2026-07-26) | WATCHING | schedule | ~2026-08-02; also releases R-91 | CC |
|
|
| — | demo-felhom's next restore-test (84 h cadence, last 2026-07-27 06:38 UTC) | WATCHING | schedule | ~2026-07-30 18:38 UTC | CC |
|
|
| **R-97** | ~~Whole-guest backup tier had no hub signal; quiesce blamed the apps~~ | **SHIPPED** (controller v0.177.0 + hub v0.78.0/v0.79.0, 2026-07-27) | — | v0.79.0 (R-97c) replaced a FALSE operator-only comment with a real `operatorOnlyEvents` register | — |
|
|
| **F-CRIT-2** | ~~A failed offsite backup left a phantom snapshot (1 B, manifest-less, NEWEST) that RESET the tier's freshness clock — 7 days silent on the real 168h cadence, invisible to both the R-88 breaker and the hub deadline monitor~~ | **SHIPPED + PROVEN-LIVE** (agent v0.106.0, 2026-07-28) | — | `NewestArchiveTime` now counts only plausibly-complete entries (measured 1 MiB floor; undecidable ⇒ not counted). Campaign fault 2 replayed on demo-hp: phantom rejected + logged once, tier correctly DUE and backed up, and **no thrash** on the inverse | — |
|
|
| **R-99** | Server-side prune **never removes** a phantom snapshot. Confirmed it does NOT count them toward `keep-last` (dry-run kept 2 real + the phantom) so there is **no retention/data-loss bug** — but one accumulates per aborted upload, forever | READY (S) | — | Decide a cleanup path. Deletion on a **customer** datastore is a separate ruling — detection shipped, removal deliberately not automated | CC |
|
|
| **F-CRIT-1** | An app that **fails to restart** after a quiesce never alarms on any channel. Two independent causes: `restartAll` discards the error (returns nothing), and `StateStopped` is whitelisted at `main.go:1237` on invariant I1, which is false for the failed-restart path | **READY — HIGHEST** | — | Campaign 8's other HIGH finding, untouched by the F-CRIT-2 fix. Evidence: `audits/CAMPAIGN-8-backup-restore-2026-07-27.md` | CC |
|
|
| **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC |
|
|
| **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC |
|
|
| **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC |
|
|
|
|
## Why the READY rows rank this way
|
|
|
|
1. **R-95** — the largest *data* exposure: the tier holding the customer's documents and photos is the
|
|
one whose credential can delete. The snapshot mitigation is now **armed** (daily 00:00, keep 7),
|
|
but it has taken zero snapshots so far and it does not touch the root cause — the box can still
|
|
`forget --prune` its own repo.
|
|
2. **R-94** — a one-line constant, but until it moves every hub-driven install gets the pre-R-82
|
|
backup default. Cheapest high-consequence fix on the list.
|
|
3. **R-86** — an operator ruling already exists; it only waits on knowing what load ep0 can take.
|
|
4. **R-87** — real and unbuilt, but needs its own design, so it should not jump work that is specified.
|