admin
955083c0fc
Phase A is the headline and it passed on live hardware, through the real endpoints the UI
posts to: a customer who deletes files — or their entire app data directory — gets everything
back byte-identical, and the app works afterwards (paperless served the restored bytes over
its own API at the exact pre-deletion sha256). A1's two non-destruction promises both hold.
Three defects, recorded not fixed:
C9-F1 (HIGH) the Tier-2 restore button is offered for apps it can never restore, takes a
real outage, and reports "nothing was missing" — indistinguishable from a
genuine result, while 156 MB of that app's data sits unread in the same copy.
C9-F2 (HIGH) an app in a crash loop never alarms on any channel; StateRestarting is in no
down-set, so F-OBS's own heartbeat printed "0 currently down" for 9 minutes.
C9-F3 (MEDIUM) an interrupted offsite run leaves a lock the self-heal cannot reach; the tier
is dead until a human unlocks and the operator is told "unknown reason".
This answers Phase C item 8.
Two candidates were deliberately NOT filed: a recovery-unit poisoning the catalog sync healed
in ~3 min, and a snapshot_id that is documented as logging-only. Reporting either would have
been reporting an artifact.
Stopped at the end of Phase B (plus D10), then full recovery — both boxes healthy, real
cadences, offsite tier proven working again, no leaked scratch guests, peti untouched.
D11's approved staleAfter compression turned out not to be a knob; reported, not worked around.
14 KiB
14 KiB
OPEN-ITEMS — the single source of truth for open work
Rebuilt 2026-07-27 by read-only triage. ROADMAP.md keeps the full history and reasoning; this
page keeps only what is open, and it is the file to read first. REPORT.md is per-session and
overwritten — nothing durable may live only there.
State: BLOCKED · READY · WAITING-ON-OPERATOR · WATCHING. Every row has an owner.
| ID | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|
| R-88a | SHIPPED (controller v0.176.0, 2026-07-27) | — | Live on both boxes; breaker 15m→4h, per-tier, never permanent | — | |
| R-88b | /backup/due cannot say unknown |
SHIPPED + PROVEN-LIVE (agent v0.105.0 + controller v0.178.0, 2026-07-27) | — | age_state=unknown captured on real hardware during a deliberate ep0 outage; controller deferred, zero app stacks stopped |
— |
| R-95 | restic offsite credential can delete (readonly=False, forget --prune runs from the box); SFTP cannot express append-only |
READY #1 | — | Root exposure still open. Mitigation now ARMED — split prune off-box or move to REST --append-only |
CC |
| R-94 | Hub hands out host-install 1.19.0; 1.20.0 is what carries R-82's backup default |
READY #2 | — | Bump configs.go:28, and stop hand-syncing a version constant across repos |
CC |
| R-86 | Restore-tests are interval-scheduled, not backup-aligned | READY #3 | R-90 (ep0 headroom) informs cadence | Trigger a tier ~24 h after its own newest archive | CC |
| R-87 | The restic tier is never restore-tested | READY #4 | — | Design a controller-side test (no scratch-guest analogue transfers) | CC |
| — | Storage Box snapshots on storage-box-pool-1 — plan SET (daily 00:00, keep 7) but 0 taken yet |
WATCHING | first run tonight 00:00 | Confirm size_snapshots > 0 tomorrow; until then the mitigation is armed, not proven |
CC |
| — | PBS-storage-1 (u629193, box 611421) still status=active, 19.9 MB |
WAITING-ON-OPERATOR | operator console | Delete the box | operator |
| R-90 | ep0 RAM headroom — 4 GiB swap survived its first reboot 2026-07-27; 3.8 GB RAM unchanged | BLOCKED (interim proven) | Hetzner CX33 availability — confirmed unavailable even powered OFF, so it is the Cost-Optimized "Limited availability", not the power state | Re-check CX33; escape hatch if urgent = CPX/CCX lines (no availability warning, higher cost) | operator |
| R-91 | Old 13 GB datastore copy at /srv/pbs-felhom on ep0's root disk |
WATCHING | demo-felhom's first post-migration PBS backup | Delete once it lands; fix CONTEXT.md:1018 same commit |
CC |
| — | First-ever GC on felhom-offsite (armed today 13:11 UTC, never run) |
WATCHING | schedule | Sun 2026-08-02 04:30 UTC — confirm it completes | CC |
| — | demo-felhom's next weekly PBS backup (newest is 2026-07-26) | WATCHING | schedule | ~2026-08-02; also releases R-91 | CC |
| — | demo-felhom's next restore-test (84 h cadence, last 2026-07-27 06:38 UTC) | WATCHING | schedule | ~2026-07-30 18:38 UTC | CC |
| R-97 | SHIPPED (controller v0.177.0 + hub v0.78.0/v0.79.0, 2026-07-27) | — | v0.79.0 (R-97c) replaced a FALSE operator-only comment with a real operatorOnlyEvents register |
— | |
| F-CRIT-2 | SHIPPED + PROVEN-LIVE (agent v0.106.0, 2026-07-28) | — | NewestArchiveTime now counts only plausibly-complete entries (measured 1 MiB floor; undecidable ⇒ not counted). Campaign fault 2 replayed on demo-hp: phantom rejected + logged once, tier correctly DUE and backed up, and no thrash on the inverse |
— | |
| R-99 | Server-side prune never removes a phantom snapshot. Confirmed it does NOT count them toward keep-last (dry-run kept 2 real + the phantom) so there is no retention/data-loss bug — but one accumulates per aborted upload, forever |
READY (S) | — | Decide a cleanup path. Deletion on a customer datastore is a separate ruling — detection shipped, removal deliberately not automated | CC |
| F-CRIT-1 | restartAll discarded the error AND StateStopped was whitelisted on invariant I1, which the quiesce path had made false |
SHIPPED + PROVEN-LIVE (controller v0.179.0, 2026-07-28) | — | Both causes fixed. Live on demo-hp: alarmed 9s after grace expiry, banner shows (stopped); a deliberate user stop stayed silent through 9 dead-app scans |
— |
| F-A1 | SHIPPED + PROVEN-LIVE (controller v0.179.0, 2026-07-28) | — | 409 → contention: tier stays DUE, dropped before anything stops (15m), and BLOCKED alarm if contention outlives the agent's 120m ceiling (3h). Hub DB: 409 → 0 operator emails, real failure → 1 | — | |
| R-100 | isStale counted from LastRun, which the controller writes unconditionally on failure |
SHIPPED + PROVEN-LIVE (controller v0.181.0 + hub v0.80.0, 2026-07-28) | — | Anchored on a new last_success. Severity corrected during Phase 0: this was NOT a silence — backup_failed does fire nightly and reaches the operator (live DB: 5 sends). The real defect is defeated defence in depth: the hub-side pull net was anchored on a field the failing controller keeps refreshing, so it could not compensate for a lost push (cf. F-HUB). Live on demo-hp: induced failure → last_run advanced 11:25:48Z, last_success held 11:24:20Z; demo-felhom healthy → anchor advanced. Legacy degrade logged once per customer, live |
— |
| R-101 | LastRun is written on failure and rendered to the customer as „Legutóbbi másolat" — including in the restore confirm dialog |
SHIPPED + PROVEN-LIVE (controller v0.182.0, 2026-07-28) | — | CrossDriveBackup.LastSuccess + SuccessTracked; the dialog names the last successful copy and discloses a failed newest attempt. Legacy rows migrate truthfully on first touch (an ok row adopts its time; an error row seeds nothing) — without the marker all 7 fleet rows would have flipped to „Még nincs sikeres másolat" on deploy. Part 2: the three record* sites rebuilt the whole struct; replaced by tier2Update (copy-and-overlay, safe by construction) — the naive fix would have had recordTier2Failure CLEAR the anchor. Live on demo-hp, rendered dialog read in both states |
— |
| C9-F1 | Tier-2 „Fájlok visszaállítása" is offered for apps whose Tier-2 copy has no restorable file leg (BookStack, Docmost = 2 of the 4 Tier-2 apps on demo-felhom). Clicking it stops the app, restores 0 files, and reports „Nincs hiányzó fájl — minden fájl megvan a helyén." — indistinguishable from a genuine "nothing was missing", on every click, forever, while 156 MB of that app's data sits unread in recovery-unit/ in the same copy |
READY (HIGH) | — | Either don't render the button when the copy has no hdd/+userdata/ leg, or make the message distinguish "nothing missing" from "nothing restorable here". tier2_restore.go:101-104 reads only those two subtrees; tier2_restore_test.go:219 pins 0-files-as-success and explicitly accepts the outage; tier2_honesty_test.go Scenario D asserts the button MUST render for exactly BookStack's shape |
CC |
| C9-F2 | An app in a Docker crash loop never alarms on any channel — no dashboard banner, no app_start_failed, no customer email, no hub event. aggregateState returns StateRestarting (manager.go:753-756, checked before the degraded branch) and IsDownState (manager.go:54-56) excludes it, so classifyRunStates computes down=false. With restart: unless-stopped this is permanent silence. F-CRIT-1's fix covers StateStopped only |
READY (HIGH) | — | Add StateRestarting to the down set with a debounce, or watch RestartCount. Proven live on demo-felhom: 9+ min crash loop, restartcount 18, while the F-OBS heartbeat printed "180 scans since boot, 4 deployed app(s) evaluated, 0 currently down" |
CC |
| C9-F3 | An interrupted offsite run leaves an exclusive restic lock the self-heal cannot reach: resticStep (offbox.go:634-648) has unlock --remove-all, but ensureOffboxRepo's probe fails first, classifyResticProbe (offbox.go:77-93) has no lock case → "other" → fail-fast. Tier dead until a human unlocks; ClassifyOffsiteFailure likewise has no lock case so the operator is told „A távoli mentés ismeretlen okból nem sikerült" for a precisely-known, self-healable condition |
READY (MEDIUM) | — | Add a lock case to both classifiers and let the probe path escalate to unlock --remove-all. Answers Phase C item 8: the repo is NOT usable after a killed run. Cleared manually this run; tier proven working again (ok, 1m35s). Reachable by any interruption — container restart, OOM, host reboot mid-backup |
CC |
| F-DIAG | SHIPPED (controller v0.182.0, 2026-07-28) | — | ClassifyOffsiteFailure → quota / orphaned / no_repo / no_units / transport / unknown, each with its own Hungarian message. Unclassifiable says so rather than being folded into a neighbour. Secrets: the old message was a raw err.Error() passthrough carrying sftp:<user>@<host>:<path>; redaction is now by the target's actual host/user/path (a first regex-only attempt leaked on a bare hostname and its own test caught it). Unit-proven; not yet exercised by a live offsite failure of each class |
— | |
| F-OPS | pct restore inherits the source guest's bind mounts — during a real DR, on a different host, under pressure |
DOCUMENTED (2026-07-28) | — | documentation/runbooks/RUNBOOK-manual-guest-restore.md: which mpN are volumes vs host binds, the mp9 source-VMID trap (it can bind another guest's bootstrap credentials), strip-and-re-add before first boot, and a positive pre-start verification. Docs only by design — the agent already neutralises binds on its own restore paths, and a second implementation would drift |
— |
| F-REBOOT | SHIPPED + PROVEN-LIVE (agent v0.107.0, 2026-07-28) | — | 60 s guest-power watchdog; onboot is the deliberate-stop discriminator (already the stale-lock path's, and what pve-guests consults), retry bounded 3x/1m-2m-4m then escalates once. Live on demo-hp: 120 s unattended vs the incident's 587 s with a human; Scenario B proven (an onboot:0 guest left stopped) |
— | |
| F-LEAK | VM.Allocate); the 10-slot VMID band shrinks silently |
SHIPPED + PROVEN-LIVE (agent v0.110.0 + host-install v1.21.0, 2026-07-28) | — | Three attempts, two refuted live. (1) Pool adoption: PUT /pools/{pool} also needs VM.Allocate on the VM — membership cannot bootstrap its own authority. (2) Per-path /vms/990000..990009 ACLs: work, but PVE's destroy calls remove_vm_access (LXC.pm:906) which deletes every ACL at /vms/<vmid> — consumed by the op it authorises, one use per slot. (3) SHIPPED: 4th root-fenced exception, band enforced in sudoers literally (pct destroy 99000[0-9] --purge) + in code + at the caller; API destroy still tried first. Live: band PERMITTED, 9201/9100/9999/990010/1 REFUSED, and pct start 990000 REFUSED too |
— |
| F-OBS | deadapp-check leaves NO positive observable on a default (info-level) box — "no alarms" was indistinguishable from "never ran" |
SHIPPED + PROVEN-LIVE (controller v0.180.0 + agent v0.109.0, 2026-07-28) | — | INFO summary every 20th scan carrying scans/evaluated/down. Agent v0.109.0 fixes the same shape in the guest-power watchdog shipped hours earlier in v0.107.0 — it logged only at startup and when it acted, so its health could be read only from absence | — |
| R-89 | Retention as a per-customer commercial policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC |
| R-92 | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC |
| R-93 | drill-r50 is both a blocked customer and the only drift fixture |
READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC |
Why the READY rows rank this way
- R-95 — the largest data exposure: the tier holding the customer's documents and photos is the
one whose credential can delete. The snapshot mitigation is now armed (daily 00:00, keep 7),
but it has taken zero snapshots so far and it does not touch the root cause — the box can still
forget --pruneits own repo. - R-94 — a one-line constant, but until it moves every hub-driven install gets the pre-R-82 backup default. Cheapest high-consequence fix on the list.
- R-86 — an operator ruling already exists; it only waits on knowing what load ep0 can take.
- R-87 — real and unbuilt, but needs its own design, so it should not jump work that is specified.