admin
955083c0fc
Phase A is the headline and it passed on live hardware, through the real endpoints the UI
posts to: a customer who deletes files — or their entire app data directory — gets everything
back byte-identical, and the app works afterwards (paperless served the restored bytes over
its own API at the exact pre-deletion sha256). A1's two non-destruction promises both hold.
Three defects, recorded not fixed:
C9-F1 (HIGH) the Tier-2 restore button is offered for apps it can never restore, takes a
real outage, and reports "nothing was missing" — indistinguishable from a
genuine result, while 156 MB of that app's data sits unread in the same copy.
C9-F2 (HIGH) an app in a crash loop never alarms on any channel; StateRestarting is in no
down-set, so F-OBS's own heartbeat printed "0 currently down" for 9 minutes.
C9-F3 (MEDIUM) an interrupted offsite run leaves a lock the self-heal cannot reach; the tier
is dead until a human unlocks and the operator is told "unknown reason".
This answers Phase C item 8.
Two candidates were deliberately NOT filed: a recovery-unit poisoning the catalog sync healed
in ~3 min, and a snapshot_id that is documented as logging-only. Reporting either would have
been reporting an artifact.
Stopped at the end of Phase B (plus D10), then full recovery — both boxes healthy, real
cadences, offsite tier proven working again, no leaked scratch guests, peti untouched.
D11's approved staleAfter compression turned out not to be a knob; reported, not worked around.
86 lines
6.1 KiB
Markdown
86 lines
6.1 KiB
Markdown
# REPORT — CAMPAIGN 9: the restore paths, proven (2026-07-28)
|
|
|
|
**Overwritten** per the standing rule. **No production code shipped** — this was a proof campaign,
|
|
and findings are recorded, never fixed inline. Full write-up:
|
|
`documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md`.
|
|
Evidence: `DooPlex:~/campaign9/evidence/` (69 files, 221 MB, 7 collectors, written continuously).
|
|
|
|
Fleet unchanged and healthy at close: hub v0.80.0, agent v0.110.0, controller v0.182.0 on both boxes.
|
|
**`peti-felhom` was never touched.** The ep0 rollback copy `/srv/pbs-felhom` (13 G) is intact.
|
|
|
|
## The headline — two never-proven restore paths are now proven
|
|
|
|
Driven through the **real endpoints the UI posts to**, over https through traefik with a real session
|
|
and CSRF token, on live hardware.
|
|
|
|
| proof | result |
|
|
|---|---|
|
|
| **A1** — Tier-2 restore of ordinary app data (`paperless-ngx`, demo-hp) | 6 deleted files back **byte-identical** (`sha256sum -c` all OK) |
|
|
| A1 — „A meglévő fájlok NEM módosulnak és NEM törlődnek" | 2 created files survived; 1 locally-edited file **not overwritten** (edit marker intact) |
|
|
| A1 — app stopped/restarted and healthy | stop→copy→start in 39 s, `paperless-webserver` healthy |
|
|
| A1 — data **usable by the app**, not just on disk | paperless resolved all 3 docs, checksums matched its own DB, and **served the restored bytes over its own HTTP API** at the exact pre-deletion sha256 |
|
|
| **A2** — Tier-1 recovery-unit restore is a **distinct** path | `POST /backup/restore` → `RestoreFromRecoveryUnit`; ran end-to-end in 18 s, 1 volume restored, app healthy |
|
|
| **A3** — restore after **total loss** (whole appdata dir `rm -rf`) | loss proven by doc download going **200 → 404**; restore returned **43/43 files byte-identical**, `documents_ok 16 of 16`, downloads back to 200 |
|
|
|
|
The honest boundary A1+A3 together establish: **existing files are untouched; destroyed files return
|
|
at their last-backup state.**
|
|
|
|
## Findings — 3 defects, ranked (none fixed)
|
|
|
|
| # | finding | severity |
|
|
|---|---|---|
|
|
| **C9-F1** | The Tier-2 restore button is offered for apps it can **never** restore (BookStack, Docmost). It takes a real app outage, restores 0 files, and reports „Nincs hiányzó fájl — minden fájl megvan a helyén." — while 156 MB of that app's data sits unread in the same copy | **HIGH** |
|
|
| **C9-F2** | An app in a **crash loop never alarms on any channel**. `StateRestarting` is in no down-set, so the dead-app heartbeat printed *"180 scans … 0 currently down"* while the app had been looping for 9 minutes | **HIGH** |
|
|
| **C9-F3** | An **interrupted offsite run** leaves an exclusive restic lock the existing self-heal cannot reach; the tier is dead until a human unlocks, and the operator is told *"unknown reason"* | **MEDIUM** |
|
|
|
|
Two things were deliberately **not** filed as defects: a recovery-unit poisoning that the catalog
|
|
sync self-healed within ~3 minutes (proven live — reporting it would have been reporting an
|
|
artifact), and a `snapshot_id` that looked ignored but is documented as logging-only and confirmed
|
|
so live.
|
|
|
|
## Mechanisms confirmed working, live
|
|
|
|
R-82's one-quiesce rule under mixed outcomes (2 tiers due, apps stopped **once**, per-target
|
|
breaker); R-88's breaker (edge-triggered, one WARN, one event, three silent DEBUG skips, **no app
|
|
thrash**); F-A1's contention deferral (409 → no breaker, no event, prompt restart — both sides of
|
|
the seam captured in the same second); **F-CRIT-2's size filter against a real 1-byte phantom** on
|
|
demo-hp, confirmed independently on ep0's filesystem; R-100's success anchor twice; **F-DIAG's
|
|
sanitiser on the exact bare-hostname case that defeated its first version** (nothing raw reaches the
|
|
hub event or the report); F-OBS's positive observable — which is precisely what made C9-F2 provable;
|
|
F-LEAK's fenced destroy (no leaked `990000` guests across ~10 restore-tests).
|
|
|
|
## Where it stopped, and what remains
|
|
|
|
Stopped at the **end of Phase B**, plus Phase D item 10, then full recovery. Phase C item 6 (host
|
|
reboot mid-backup) was deliberately not started — a large new fault class against boxes that are
|
|
remote until ~08-02, and starting it would have meant rushing it or leaving the fleet unknown.
|
|
|
|
**Approved but impossible:** Phase 0 cleared compressing the hub's `staleAfter` for R-100's
|
|
threshold test. It is **not a knob** — `cmd/hub/main.go:552` passes `0`, selecting the compile-time
|
|
`defaultOffsiteStaleAfter = 48h`. Compressing it needed a hub code change, which the campaign
|
|
forbids. Reported rather than worked around. The no-code-change alternative (age the controller's
|
|
reported `last_success` past 48 h and let the hub judge at its real threshold) is the recommended
|
|
method next time.
|
|
|
|
**The honest residue — still not proven:** Tier-1 **content** recovery after real loss (A2 ran on an
|
|
intact app; A3 used Tier-2) — now the most valuable open item; host reboot mid-backup; three-way
|
|
concurrency with GC; Scenario C live; `offsite_stale` actually firing; F-HUB `SQLITE_BUSY`.
|
|
|
|
## Recovery
|
|
|
|
Every config reverted from `evidence/config-before/REVERT.md`, each verified with a **positive
|
|
observable**: agent cadences back to `0 / 302400 / 604800` on both hosts (`is-active` = active),
|
|
windows back to `02:30`, `pvesm` shows `felhom-pbs active` on both, 0 campaign iptables rules on
|
|
either host or guest, 0 scratch guests in the `990000` band, all stacks healthy on both boxes, and
|
|
the offsite tier not merely unblocked but **proven working again** (`ok`, 1m35s, 8 snapshots).
|
|
|
|
One benign residue: the in-memory R-88 breaker still holds a `felhom-pbs` failure count on each box.
|
|
Its `until` is long past so it blocks nothing; it clears on the next successful backup or any
|
|
controller restart (by design, not persisted). Clearing it would have cost another app outage for no
|
|
benefit.
|
|
|
|
**One operational lesson worth a runbook line:** a hand-run `docker compose up -d` in
|
|
`/opt/docker/stacks/<app>` starts a Felhom app **without its secrets** — they are injected by the
|
|
controller's `stackEnv` at start time, not stored in a `.env`. It turned a healthy docmost into a
|
|
crash loop during recovery. Manual recovery must go through `POST /api/stacks/<name>/restart`.
|