Phase A is the headline and it passed on live hardware, through the real endpoints the UI
posts to: a customer who deletes files — or their entire app data directory — gets everything
back byte-identical, and the app works afterwards (paperless served the restored bytes over
its own API at the exact pre-deletion sha256). A1's two non-destruction promises both hold.
Three defects, recorded not fixed:
C9-F1 (HIGH) the Tier-2 restore button is offered for apps it can never restore, takes a
real outage, and reports "nothing was missing" — indistinguishable from a
genuine result, while 156 MB of that app's data sits unread in the same copy.
C9-F2 (HIGH) an app in a crash loop never alarms on any channel; StateRestarting is in no
down-set, so F-OBS's own heartbeat printed "0 currently down" for 9 minutes.
C9-F3 (MEDIUM) an interrupted offsite run leaves a lock the self-heal cannot reach; the tier
is dead until a human unlocks and the operator is told "unknown reason".
This answers Phase C item 8.
Two candidates were deliberately NOT filed: a recovery-unit poisoning the catalog sync healed
in ~3 min, and a snapshot_id that is documented as logging-only. Reporting either would have
been reporting an artifact.
Stopped at the end of Phase B (plus D10), then full recovery — both boxes healthy, real
cadences, offsite tier proven working again, no leaked scratch guests, peti untouched.
D11's approved staleAfter compression turned out not to be a knob; reported, not worked around.
Evidence: `DooPlex:~/campaign9/evidence/` (69 files, 221 MB, 7 collectors, written continuously).
Fleet unchanged and healthy at close: hub v0.80.0, agent v0.110.0, controller v0.182.0 on both boxes.
**`peti-felhom` was never touched.** The ep0 rollback copy `/srv/pbs-felhom` (13 G) is intact.
## The headline — two never-proven restore paths are now proven
Driven through the **real endpoints the UI posts to**, over https through traefik with a real session
and CSRF token, on live hardware.
| proof | result |
|---|---|
| **A1** — Tier-2 restore of ordinary app data (`paperless-ngx`, demo-hp) | 6 deleted files back **byte-identical** (`sha256sum -c` all OK) |
| A1 — „A meglévő fájlok NEM módosulnak és NEM törlődnek" | 2 created files survived; 1 locally-edited file **not overwritten** (edit marker intact) |
| A1 — app stopped/restarted and healthy | stop→copy→start in 39 s, `paperless-webserver` healthy |
| A1 — data **usable by the app**, not just on disk | paperless resolved all 3 docs, checksums matched its own DB, and **served the restored bytes over its own HTTP API** at the exact pre-deletion sha256 |
| **A2** — Tier-1 recovery-unit restore is a **distinct** path | `POST /backup/restore` → `RestoreFromRecoveryUnit`; ran end-to-end in 18 s, 1 volume restored, app healthy |
| **A3** — restore after **total loss** (whole appdata dir `rm -rf`) | loss proven by doc download going **200 → 404**; restore returned **43/43 files byte-identical**, `documents_ok 16 of 16`, downloads back to 200 |
The honest boundary A1+A3 together establish: **existing files are untouched; destroyed files return
at their last-backup state.**
## Findings — 3 defects, ranked (none fixed)
| # | finding | severity |
|---|---|---|
| **C9-F1** | The Tier-2 restore button is offered for apps it can **never** restore (BookStack, Docmost). It takes a real app outage, restores 0 files, and reports „Nincs hiányzó fájl — minden fájl megvan a helyén." — while 156 MB of that app's data sits unread in the same copy | **HIGH** |
| **C9-F2** | An app in a **crash loop never alarms on any channel**. `StateRestarting` is in no down-set, so the dead-app heartbeat printed *"180 scans … 0 currently down"* while the app had been looping for 9 minutes | **HIGH** |
| **C9-F3** | An **interrupted offsite run** leaves an exclusive restic lock the existing self-heal cannot reach; the tier is dead until a human unlocks, and the operator is told *"unknown reason"* | **MEDIUM** |
Two things were deliberately **not** filed as defects: a recovery-unit poisoning that the catalog
sync self-healed within ~3 minutes (proven live — reporting it would have been reporting an
artifact), and a `snapshot_id` that looked ignored but is documented as logging-only and confirmed
| Schedules | `gc-schedule sun 04:30`, `verify-new 1`, 2 prune jobs ✅ |
| Filesystem | `EXT4-fs (sda): recovery complete` — journal replayed after the unclean power-off, then mounted r/w; **0** I/O or EXT4 errors ✅ |
| **A1** — Tier-2 restore of ordinary app data (`paperless-ngx`, demo-hp) | 6 deleted files back **byte-identical** (`sha256sum -c` all OK) |
| A1 — „A meglévő fájlok NEM módosulnak és NEM törlődnek" | 2 created files survived; 1 locally-edited file **not overwritten** (edit marker intact) |
| A1 — app stopped/restarted and healthy | stop→copy→start in 39 s, `paperless-webserver` healthy |
| A1 — data **usable by the app**, not just on disk | paperless resolved all 3 docs, checksums matched its own DB, and **served the restored bytes over its own HTTP API** at the exact pre-deletion sha256 |
| **A2** — Tier-1 recovery-unit restore is a **distinct** path | `POST /backup/restore` → `RestoreFromRecoveryUnit`; ran end-to-end in 18 s, 1 volume restored, app healthy |
| **A3** — restore after **total loss** (whole appdata dir `rm -rf`) | loss proven by doc download going **200 → 404**; restore returned **43/43 files byte-identical**, `documents_ok 16 of 16`, downloads back to 200 |
### The unplanned finding: the volume changed device name
The honest boundary A1+A3 together establish: **existing files are untouched; destroyed files return
at their last-backup state.**
```
before: volume=/dev/sdb root=/dev/sda1
after: volume=/dev/sda root=/dev/sdb1
```
## Findings — 3 defects, ranked (none fixed)
The kernel enumerated the disks in the opposite order across the reboot. The mount survived only
because fstab uses `/dev/disk/by-id/scsi-0HC_Volume_106469259`. **A `/dev/sdb` entry would have failed
to mount — or mounted the root disk's device node.** That choice was load-bearing and had never been
tested; this reboot tested it.
| # | finding | severity |
|---|---|---|
| **C9-F1** | The Tier-2 restore button is offered for apps it can **never** restore (BookStack, Docmost). It takes a real app outage, restores 0 files, and reports „Nincs hiányzó fájl — minden fájl megvan a helyén." — while 156 MB of that app's data sits unread in the same copy | **HIGH** |
| **C9-F2** | An app in a **crash loop never alarms on any channel**. `StateRestarting` is in no down-set, so the dead-app heartbeat printed *"180 scans … 0 currently down"* while the app had been looping for 9 minutes | **HIGH** |
| **C9-F3** | An **interrupted offsite run** leaves an exclusive restic lock the existing self-heal cannot reach; the tier is dead until a human unlocks, and the operator is told *"unknown reason"* | **MEDIUM** |
### Recovery
Two things were deliberately **not** filed as defects: a recovery-unit poisoning that the catalog
sync self-healed within ~3 minutes (proven live — reporting it would have been reporting an
artifact), and a `snapshot_id` that looked ignored but is documented as logging-only and confirmed
so live.
Both boxes returned to `age_state=known` on both tiers within ~60 s of boot; `felhom-pbs` on
demo-felhom back to `due=false, within cadence window, age_seconds=102102`. No breaker was armed, so
none needed clearing; no failure event fired, so no recovery event was expected — and none appeared.
15:53:17 [INFO] [backup] Recovery unit captured for docmost → …/backups/primary/docmost (images=3, …)
```
Verified afterwards on all three surfaces:
```
live compose image: image: docmost/docmost:0.95.0
Tier-1 recovery unit pins: ['docmost/docmost:0.95.0', 'postgres:16-alpine', 'redis:7-alpine']
Tier-2 copy's unit pins: ['docmost/docmost:0.95.0', 'postgres:16-alpine', 'redis:7-alpine']
```
**Conclusion.** Compose drift is self-healing and the restore point follows it, bounded by the sync
interval — the poisoned window here was **~3 minutes**. The system does not leave you in the state I
manufactured, so **reporting this as a defect would have been reporting an artifact.** The absence
of validation in `CaptureRecoveryUnit` is real, but it is compensated, and the compensating control
was proven live rather than assumed.
**The one residue worth an operator's attention** — narrow, and NOT demonstrated here: the sync makes
the *catalog* the source of truth, so a bad image tag **in the catalog** would propagate to every
box's only restore point unhealed, because sync would be restoring it rather than reverting it. That
is a catalog-quality concern owned by the catalog gates, not a controller defect.
### F-3 — an interrupted offsite run leaves a lock the self-heal cannot reach, reported as "unknown" — **DEFECT, MEDIUM**
**This answers Phase C item 8** ("is the repo usable after a killed run, or does it need an
unlock?"). It was found by accident — the campaign severed its own in-flight run — which makes it a
genuine interruption rather than a constructed one.
**Sequence.** An offsite run started at 15:58:12 and took the repo's exclusive lock at 15:59:00. At
15:59:33 its transport was severed. The run died holding the lock. Every later run then failed:
```
16:15:48 [offbox] pre-push dump leg completed in 20.056s — snapshot pair is coherent
16:15:50 [ERROR] [offbox] backup failed after 23s: offbox repo unreachable: exit status 1:
unable to create lock in backend: repository is already locked exclusively by PID 14328 on demo-hp by root
lock was created at 2026-07-28 15:59:00 (16m49s ago)
the `unlock` command can be used to remove stale locks
```
**The remedy exists in the code and was never reached.**`resticStep`
(`internal/backup/offbox.go:634-648`) detects `repository is already locked`, escalates to
`unlock --remove-all`, and retries once — its comment states plainly that this is required because
"the recreated container has a new hostname, so restic can't verify the dead PID and won't treat the
lock as stale for ~30 min (the overnight-campaign C2 finding)".
But the failure happens **before** any `resticStep` call. `ensureOffboxRepo`
(`offbox.go:676-696`) probes the repo first; that probe hits the lock, `classifyResticProbe`
(`offbox.go:77-93`) has no case for a lock error so it returns `"other"`, and the default branch
returns `offbox repo unreachable: …` — a fail-fast with **no unlock escalation**. The self-heal sits
behind a gate the failure never opens.
**And the operator is told the wrong thing.**`ClassifyOffsiteFailure` likewise has no lock case, so
it falls through to `OffsiteFailUnknown`. Hub event 1919:
```
"A távoli mentés ismeretlen okból nem sikerült (23s): offbox repo unreachable: … repository is
already locked exclusively by PID 14328 … the `unlock` command can be used to remove stale locks"
```
"The remote backup failed **for an unknown reason**" — for a condition that is precisely known,
named in the attached detail, and self-healable by code already in the binary.
**Impact.** The offsite tier stays **dead until a human intervenes**. Verified: it took a manual
`restic unlock --remove-all` to restore it (`evidence/phaseC/C8-unlock.txt`).
**Reachability is ordinary, not exotic.** Any interruption of an in-flight run leaves this lock — a
controller container restart, an OOM, a network drop, or **a host reboot mid-backup, which is
Phase C item 6's scenario**. The code's own comment describes exactly this ("the C2 crash lock").
**Severity MEDIUM, argued rather than asserted.** Against HIGH: an error event *is* raised
immediately on the operator channel, its detail literally contains the remedy, and R-100's 48 h
`offsite_stale` is a second backstop, so this is loud-but-mislabelled rather than silent — unlike
F-1 and F-2. Against LOW: a backup tier is down until manual intervention, and the headline actively
misdirects triage.
**DEFECT.** No cadence compression was involved in the mechanism; only the interruption was
deliberate.
---
### O-2 — a hand-run `docker compose up -d` starts a Felhom app WITHOUT its secrets — **OPERATIONAL, worth a runbook line**
Encountered during recovery, by me. Restoring docmost with a plain
`cd /opt/docker/stacks/docmost && docker compose up -d` produced a container that crash-looped on:
```
{"minLength":"APP_SECRET must be longer than or equal to 32 characters","isNotEmpty":"APP_SECRET should not be empty"}
```
Secrets are injected by the controller's `stackEnv` at start time (`StartStack`,
`internal/stacks/manager.go:912-948`); they are **not** in a `.env` file beside the compose. A manual
`up -d` therefore recreates the container with empty secrets — turning a healthy app into a broken
one, and looking exactly like the fault you were trying to fix. Recovery must go through the
controller (`POST /api/stacks/<name>/restart`), which is what finally fixed it.
---
## 7. Phase C / D / E — reached, and not reached
| # | item | status |
|---|---|---|
| C6 | host reboot mid-backup | **NOT REACHED.** Approved by the operator; not attempted. Its most likely consequence was nonetheless observed by another route — see F-3, which is exactly the state a reboot mid-backup would leave. |
| C7 | three-way concurrency (backup + restore-test + GC) | **PARTIALLY.** Backup-vs-restore-test contention was proven (B1); GC was not added as a third party. |
| C8 | restic transport interrupted mid-run — is the repo usable after? | **ANSWERED: no, it needs an unlock.** See **F-3**. |
| D9 | Scenario C — never-succeeded Tier-2 on a throwaway app | **NOT REACHED.** No throwaway app was created. |
| D10 | F-DIAG's failure classes + no credential/repo-URL/raw-error passthrough | **PROVEN for the surfaces that leave the box** — see below. |
| D11 | R-100's threshold — see `offsite_stale` actually fire | **NOT REACHED**, and the approved method turned out not to exist — see below. |
iptables : 0 campaign9 rules on either host and in either guest
scratch VMs : none in the 990000 band on either host
ep0 : /srv/pbs-felhom rollback copy intact (13G); datastore 14% used
peti-felhom : never touched
```
**One benign residue, stated for completeness.** The in-memory R-88 breaker on each box still holds
a `felhom-pbs` consecutive-failure count from the induced failures. Its `until` timestamps are long
past, so it blocks nothing; the only effect is that a *next* failure would back off 30 m instead of
15 m. It clears on the next successful whole-guest backup, and by design it also clears on any
controller restart (`breaker.go` — deliberately not persisted). Nothing was done about it because
clearing it would have meant taking another app outage for no benefit.
---
## 9. Where this stopped, and why
**Stopped at the end of Phase B, with Phase D item 10 additionally completed**, then Phase F.
Phase A — the campaign's stated headline and its whole justification — is complete and proven. Phase
B produced two of its five combinations cleanly, plus a HIGH finding. Phase C item 8 was answered by
an accident that turned out to be a genuine defect. Phase D item 10 is proven; D11's approved method
did not exist.
Phase C item 6 (host reboot mid-backup) was deliberately **not started**. It is a large new fault
class against two boxes that are remote until ~08-02, and beginning it would have meant either
rushing it or leaving the fleet in an unknown state. The campaign's own rule — *stop cleanly at a
phase boundary; a broken test node is fine, an unreported one is not* — made the choice: finish the
recovery properly rather than start a fault I could not supervise to completion. F-3 partly covers
its most likely consequence anyway, since an interrupted run is exactly what a reboot mid-backup
produces.
---
## 10. Findings, ranked
| # | finding | severity | class |
|---|---|---|---|
| **F-1** | the Tier-2 restore button reports success for apps it can never restore (BookStack, Docmost — half the fleet's Tier-2 apps); takes a real outage; message is indistinguishable from a genuine "nothing was missing" | **HIGH** | DEFECT |
| **F-2** | an app in a crash loop never alarms on any channel — `StateRestarting` is in no down-set, and the dead-app heartbeat positively reports "0 currently down" while it loops | **HIGH** | DEFECT |
| **F-3** | an interrupted offsite run leaves an exclusive lock the existing self-heal cannot reach; tier dead until manual `unlock`; reported to the operator as "unknown reason" | **MEDIUM** | DEFECT |
| O-1 | recovery-unit capture validates nothing, but the catalog sync reverts drift within ~3 min and the unit follows | — | OBSERVATION (nearly mis-filed as a defect) |
| O-2 | a hand-run `docker compose up -d` starts a Felhom app without its secrets | — | OPERATIONAL |
**Mechanisms confirmed working, on live hardware:** R-82's one-quiesce rule under mixed outcomes;
R-88's breaker (per-target, edge-triggered, one event, no app thrash); F-A1's contention deferral
(no breaker, no event, prompt restart); F-CRIT-2's size filter (against a real 1-byte phantom, not a
manufactured one); R-100's success anchor (twice); F-DIAG's sanitiser (on the exact bare-hostname
case that defeated its first version); F-OBS's positive observable (which is what made F-2
provable); F-LEAK's fenced destroy (no leaked scratch guests across ~10 restore-tests).
---
## 11. What came off the never-proven list — and the honest residue
**Off the list:**
- Tier-2 cross-drive restore of **ordinary app data** — the customer's actual button (A1)
- Tier-2's "existing files are NOT modified or deleted" promise, both halves (A1)
- Tier-2 restore after **total loss** of an app's data directory (A3)
- Restored data **usable by the app**, proven through the app's own API (A1/A3)
- Tier-1 recovery-unit restore is a **distinct** path, and it executes end-to-end (A2)
- The six recent fixes **do** compose: R-82 + R-88 + F-A1 + F-CRIT-2 + R-100 produced one coherent
operator story in every combination reached
- Whether an interrupted restic run leaves a usable repo (C8 — it does not)
**Still not proven — the residue:**
- **Tier-1 content recovery after real loss.** A2 ran against an intact app; A3 used Tier-2. Nobody
has yet destroyed an app's data and recovered it *from the recovery unit*. This is now the single
most valuable unproven item.
- **Host reboot mid-backup** (C6) — untested, and F-REBOOT proved the guest-reboot sibling was real.
- **Three-way concurrency** with GC as the third party (C7).
- **Scenario C** — a never-succeeded Tier-2, live (D9).
- **`offsite_stale` actually firing** (D11) — never observed. Needs the anchor-ageing method above,
not the `staleAfter` compression that was approved but does not exist.
- **F-HUB** `SQLITE_BUSY` characterisation (E12).
- **Whether F-1 and F-2 have siblings**: both are "a surface reports health it did not establish".
Nothing in this campaign searched systematically for others.
@@ -28,6 +28,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
| **F-A1** | ~~A restore-test in flight made a healthy backup report as FAILED (HTTP 409 read as a tier failure): breaker armed + operator emailed, on both boxes~~ | **SHIPPED + PROVEN-LIVE** (controller v0.179.0, 2026-07-28) | — | 409 → contention: tier stays DUE, dropped before anything stops (15m), and BLOCKED alarm if contention outlives the agent's 120m ceiling (3h). Hub DB: 409 → **0** operator emails, real failure → **1** | — |
| **R-100** | ~~A restic offsite tier that fails every night never goes stale on the hub — `isStale` counted from `LastRun`, which the controller writes unconditionally on failure~~ | **SHIPPED + PROVEN-LIVE** (controller v0.181.0 + hub v0.80.0, 2026-07-28) | — | Anchored on a new `last_success`. **Severity corrected during Phase 0: this was NOT a silence** — `backup_failed` does fire nightly and reaches the operator (live DB: 5 sends). The real defect is **defeated defence in depth**: the hub-side *pull* net was anchored on a field the failing controller keeps refreshing, so it could not compensate for a lost *push* (cf. F-HUB). Live on demo-hp: induced failure → `last_run` advanced 11:25:48Z, `last_success`**held** 11:24:20Z; demo-felhom healthy → anchor advanced. Legacy degrade logged once per customer, live | — |
| **R-101** | ~~Tier-2 `LastRun` is written on failure and rendered to the customer as „Legutóbbi másolat" — including in the restore confirm dialog~~ | **SHIPPED + PROVEN-LIVE** (controller v0.182.0, 2026-07-28) | — | `CrossDriveBackup.LastSuccess` + `SuccessTracked`; the dialog names the last **successful** copy and discloses a failed newest attempt. Legacy rows migrate truthfully on first touch (an `ok` row adopts its time; an `error` row seeds nothing) — without the marker all 7 fleet rows would have flipped to „Még nincs sikeres másolat" on deploy. **Part 2:** the three `record*` sites rebuilt the whole struct; replaced by `tier2Update` (copy-and-overlay, safe by construction) — the naive fix would have had `recordTier2Failure` CLEAR the anchor. Live on demo-hp, rendered dialog read in both states | — |
| **C9-F1** | Tier-2 „Fájlok visszaállítása" is offered for apps whose Tier-2 copy has **no restorable file leg** (BookStack, Docmost = 2 of the 4 Tier-2 apps on demo-felhom). Clicking it stops the app, restores 0 files, and reports **„Nincs hiányzó fájl — minden fájl megvan a helyén."** — indistinguishable from a genuine "nothing was missing", on every click, forever, while 156 MB of that app's data sits unread in `recovery-unit/` in the same copy | **READY (HIGH)** | — | Either don't render the button when the copy has no `hdd/`+`userdata/` leg, or make the message distinguish "nothing missing" from "nothing restorable here". `tier2_restore.go:101-104` reads only those two subtrees; `tier2_restore_test.go:219` pins 0-files-as-success and explicitly accepts the outage; `tier2_honesty_test.go` Scenario D asserts the button MUST render for exactly BookStack's shape | CC |
| **C9-F2** | An app in a **Docker crash loop never alarms on any channel** — no dashboard banner, no `app_start_failed`, no customer email, no hub event. `aggregateState` returns `StateRestarting` (`manager.go:753-756`, checked before the degraded branch) and `IsDownState` (`manager.go:54-56`) excludes it, so `classifyRunStates` computes `down=false`. With `restart: unless-stopped` this is **permanent** silence. F-CRIT-1's fix covers `StateStopped` only | **READY (HIGH)** | — | Add `StateRestarting` to the down set with a debounce, or watch `RestartCount`. Proven live on demo-felhom: 9+ min crash loop, restartcount 18, while the F-OBS heartbeat printed **"180 scans since boot, 4 deployed app(s) evaluated, 0 currently down"** | CC |
| **C9-F3** | An **interrupted offsite run leaves an exclusive restic lock the self-heal cannot reach**: `resticStep` (`offbox.go:634-648`) has `unlock --remove-all`, but `ensureOffboxRepo`'s probe fails first, `classifyResticProbe` (`offbox.go:77-93`) has no lock case → `"other"` → fail-fast. Tier dead until a human unlocks; `ClassifyOffsiteFailure` likewise has no lock case so the operator is told **„A távoli mentés ismeretlen okból nem sikerült"** for a precisely-known, self-healable condition | **READY (MEDIUM)** | — | Add a lock case to both classifiers and let the probe path escalate to `unlock --remove-all`. Answers Phase C item 8: the repo is NOT usable after a killed run. Cleared manually this run; tier proven working again (`ok`, 1m35s). Reachable by any interruption — container restart, OOM, **host reboot mid-backup** | CC |
| **F-DIAG** | ~~Four distinct offsite failure causes collapse into two operator-visible strings~~ | **SHIPPED** (controller v0.182.0, 2026-07-28) | — | `ClassifyOffsiteFailure` → quota / orphaned / no_repo / no_units / transport / **unknown**, each with its own Hungarian message. Unclassifiable says so rather than being folded into a neighbour. **Secrets:** the old message was a raw `err.Error()` passthrough carrying `sftp:<user>@<host>:<path>`; redaction is now by the target's **actual** host/user/path (a first regex-only attempt leaked on a bare hostname and its own test caught it). Unit-proven; **not** yet exercised by a live offsite failure of each class | — |
| **F-OPS** | ~~A manual `pct restore` inherits the source guest's bind mounts — during a real DR, on a different host, under pressure~~ | **DOCUMENTED** (2026-07-28) | — | `documentation/runbooks/RUNBOOK-manual-guest-restore.md`: which `mpN` are volumes vs host binds, the `mp9` source-VMID trap (it can bind **another guest's bootstrap credentials**), strip-and-re-add before first boot, and a positive pre-start verification. Docs only by design — the agent already neutralises binds on its own restore paths, and a second implementation would drift | — |
| **F-REBOOT** | ~~A guest rebooted during its backup does not come back — shutdown completes, start never happens, no self-heal; 9m47s total appliance outage with every alarm silent~~ | **SHIPPED + PROVEN-LIVE** (agent v0.107.0, 2026-07-28) | — | 60 s guest-power watchdog; `onboot` is the deliberate-stop discriminator (already the stale-lock path's, and what `pve-guests` consults), retry bounded 3x/1m-2m-4m then escalates once. Live on demo-hp: **120 s unattended** vs the incident's 587 s with a human; Scenario B proven (an `onboot:0` guest left stopped) | — |
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
