C9-F1 (honesty half) + C9-F2 shipped in controller v0.183.0; C9-F1b and C9-F4 filed
Phase 0 sized C9-F1 properly before anything was designed: 43 of the 53 catalog apps have NO subtree the Tier-2 restore can read (not 2), 9 are covered only for their file legs and never their database or volumes, 1 is stateless. The asymmetry is Tier-2's alone — Tier-1 and offsite both restore the unit and replay volume dumps, so BookStack always had a working restore and only this button lied. Shipped: the restore refuses BEFORE stopping the app and names the action that does work; a run that proceeds claims only what it EXAMINED and discloses that the database and volumes are not covered. C9-F2 alarms after a 5-minute sustained-restarting threshold, set above the 120s deploy timeout, Mealie's 60s start_period and R-97b's 180s grace; StateRestarting is deliberately NOT added to IsDownState. Live: silent through ten 30s samples then app_start_failed at 5m25s, heartbeat now reads "1 currently down" where Campaign 9 recorded 0; a real deploy stayed silent; bookstack refused with its uptime unbroken; paperless re-restored 43/43 byte-identical, 16/16 docs clean. Filed, not fixed: C9-F1b (route to the Tier-1 restore — its own task because it puts a destructive operation behind a non-destructive button) and C9-F4 (nothing reads the Tier-2 copy's recovery-unit/ mirror, so the second local copy that exists for drive loss is unreachable by any customer action — potentially larger than C9-F1).
This commit is contained in:
@@ -1,85 +1,103 @@
|
||||
# REPORT — CAMPAIGN 9: the restore paths, proven (2026-07-28)
|
||||
# REPORT — C9-F1 + C9-F2: a restore that restored nothing, and a crash loop nobody saw (2026-07-28)
|
||||
|
||||
**Overwritten** per the standing rule. **No production code shipped** — this was a proof campaign,
|
||||
and findings are recorded, never fixed inline. Full write-up:
|
||||
`documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md`.
|
||||
Evidence: `DooPlex:~/campaign9/evidence/` (69 files, 221 MB, 7 collectors, written continuously).
|
||||
**Overwritten** per the standing rule. **Shipped: controller v0.183.0**, live on **both** demo boxes.
|
||||
Fleet: hub v0.80.0, agent v0.110.0, controller **0.183.0**. `peti-felhom` untouched.
|
||||
|
||||
Fleet unchanged and healthy at close: hub v0.80.0, agent v0.110.0, controller v0.182.0 on both boxes.
|
||||
**`peti-felhom` was never touched.** The ep0 rollback copy `/srv/pbs-felhom` (13 G) is intact.
|
||||
Both defects are the same shape — the system reporting healthy while the customer is not — and both
|
||||
live in the same status-derivation code.
|
||||
|
||||
## The headline — two never-proven restore paths are now proven
|
||||
## Phase 0 — the asymmetry, sized before designing anything
|
||||
|
||||
Driven through the **real endpoints the UI posts to**, over https through traefik with a real session
|
||||
and CSRF token, on live hardware.
|
||||
Tier-2 writes **two** things on every run: the capture legs (`hdd/`, `userdata/`) and, always, a full
|
||||
`recovery-unit/` (DB dumps + named-volume tarballs). `RestoreTier2Files` reads **only the two legs**
|
||||
(`tier2_restore.go:101-104`) and has never opened `recovery-unit/`.
|
||||
|
||||
| proof | result |
|
||||
|---|---|
|
||||
| **A1** — Tier-2 restore of ordinary app data (`paperless-ngx`, demo-hp) | 6 deleted files back **byte-identical** (`sha256sum -c` all OK) |
|
||||
| A1 — „A meglévő fájlok NEM módosulnak és NEM törlődnek" | 2 created files survived; 1 locally-edited file **not overwritten** (edit marker intact) |
|
||||
| A1 — app stopped/restarted and healthy | stop→copy→start in 39 s, `paperless-webserver` healthy |
|
||||
| A1 — data **usable by the app**, not just on disk | paperless resolved all 3 docs, checksums matched its own DB, and **served the restored bytes over its own HTTP API** at the exact pre-deletion sha256 |
|
||||
| **A2** — Tier-1 recovery-unit restore is a **distinct** path | `POST /backup/restore` → `RestoreFromRecoveryUnit`; ran end-to-end in 18 s, 1 volume restored, app healthy |
|
||||
| **A3** — restore after **total loss** (whole appdata dir `rm -rf`) | loss proven by doc download going **200 → 404**; restore returned **43/43 files byte-identical**, `documents_ok 16 of 16`, downloads back to 200 |
|
||||
All 53 catalog templates enumerated, cross-checked against both boxes' actual copies:
|
||||
|
||||
The honest boundary A1+A3 together establish: **existing files are untouched; destroyed files return
|
||||
at their last-backup state.**
|
||||
|
||||
## Findings — 3 defects, ranked (none fixed)
|
||||
|
||||
| # | finding | severity |
|
||||
| class | count | what the restore can return |
|
||||
|---|---|---|
|
||||
| **C9-F1** | The Tier-2 restore button is offered for apps it can **never** restore (BookStack, Docmost). It takes a real app outage, restores 0 files, and reports „Nincs hiányzó fájl — minden fájl megvan a helyén." — while 156 MB of that app's data sits unread in the same copy | **HIGH** |
|
||||
| **C9-F2** | An app in a **crash loop never alarms on any channel**. `StateRestarting` is in no down-set, so the dead-app heartbeat printed *"180 scans … 0 currently down"* while the app had been looping for 9 minutes | **HIGH** |
|
||||
| **C9-F3** | An **interrupted offsite run** leaves an exclusive restic lock the existing self-heal cannot reach; the tier is dead until a human unlocks, and the operator is told *"unknown reason"* | **MEDIUM** |
|
||||
| **A** | **9** | the file legs only — **never** their database or named volumes |
|
||||
| **B** | **43** | **nothing at all** — a guaranteed no-op, forever |
|
||||
| C | 1 | `bentopdf`, stateless |
|
||||
|
||||
Two things were deliberately **not** filed as defects: a recovery-unit poisoning that the catalog
|
||||
sync self-healed within ~3 minutes (proven live — reporting it would have been reporting an
|
||||
artifact), and a `snapshot_id` that looked ignored but is documented as logging-only and confirmed
|
||||
so live.
|
||||
Four apps (`plex`, `jellyfin`, `emby`, `navidrome`) are in B only because their single bind is a
|
||||
`:ro` media mount, which `ClassifyBinds` correctly excludes. **81% of the catalog.**
|
||||
|
||||
## Mechanisms confirmed working, live
|
||||
**The asymmetry is Tier-2's alone.** Tier-1 (`RestoreFromRecoveryUnit`) and offsite both restore the
|
||||
unit and replay volume dumps — so BookStack already had a working restore; only this button lied.
|
||||
|
||||
R-82's one-quiesce rule under mixed outcomes (2 tiers due, apps stopped **once**, per-target
|
||||
breaker); R-88's breaker (edge-triggered, one WARN, one event, three silent DEBUG skips, **no app
|
||||
thrash**); F-A1's contention deferral (409 → no breaker, no event, prompt restart — both sides of
|
||||
the seam captured in the same second); **F-CRIT-2's size filter against a real 1-byte phantom** on
|
||||
demo-hp, confirmed independently on ep0's filesystem; R-100's success anchor twice; **F-DIAG's
|
||||
sanitiser on the exact bare-hostname case that defeated its first version** (nothing raw reaches the
|
||||
hub event or the report); F-OBS's positive observable — which is precisely what made C9-F2 provable;
|
||||
F-LEAK's fenced destroy (no leaked `990000` guests across ~10 restore-tests).
|
||||
## Shipped — Part 1a (honesty)
|
||||
|
||||
## Where it stopped, and what remains
|
||||
- **Refuses UP FRONT.** `Tier2RestoreCoverage` is consulted before any op begins; a class-B app is
|
||||
refused **without being stopped**. Live: BookStack uptime stayed `Up About an hour` (Campaign 9
|
||||
left it at `Up 25 seconds`).
|
||||
- **Names the working action** rather than dead-ending 81% of the catalog:
|
||||
> „Ennek az alkalmazásnak az adatai nem ebből a másolatból állíthatók vissza — az alkalmazás nem
|
||||
> állt le. Használd a Visszaállítás indítása gombot a Biztonsági mentés → Visszaállítás oldalon."
|
||||
- **Claims only what was examined** (the QUIET half — immich's 1.3 GB Postgres unit is not covered,
|
||||
so the old blanket sentence was a clean bill of health over data never opened):
|
||||
> „Minden vizsgált fájl megvan a helyén." + „Az alkalmazás adatbázisa és belső kötetei nem
|
||||
> tartoznak ebbe a visszaállításba."
|
||||
|
||||
Stopped at the **end of Phase B**, plus Phase D item 10, then full recovery. Phase C item 6 (host
|
||||
reboot mid-backup) was deliberately not started — a large new fault class against boxes that are
|
||||
remote until ~08-02, and starting it would have meant rushing it or leaving the fleet unknown.
|
||||
## Shipped — Part 2 (C9-F2)
|
||||
|
||||
**Approved but impossible:** Phase 0 cleared compressing the hub's `staleAfter` for R-100's
|
||||
threshold test. It is **not a knob** — `cmd/hub/main.go:552` passes `0`, selecting the compile-time
|
||||
`defaultOffsiteStaleAfter = 48h`. Compressing it needed a hub code change, which the campaign
|
||||
forbids. Reported rather than worked around. The no-code-change alternative (age the controller's
|
||||
reported `last_success` past 48 h and let the hub judge at its real threshold) is the recommended
|
||||
method next time.
|
||||
`StateRestarting` is deliberately **NOT** added to `IsDownState` — that alarms on every deploy and
|
||||
update fleet-wide. A **sustained** run becomes down after `crashLoopAfter = 5m`, set above the three
|
||||
real numbers already in the codebase: the deploy flow's **120 s** health timeout, Mealie's **60 s**
|
||||
`start_period`, and R-97b's **180 s** quiesce grace (so the windows compose into one bounded delay
|
||||
instead of leaving a gap). Docker's backoff caps at 60 s, so a real loop registers ≥4 attempts inside
|
||||
it. Carried by `Stack.RestartingSince` (not persisted) + `Stack.CrashLooping(now)`, used by **both**
|
||||
the alarm and the dashboard counter — which previously counted `restarting` as running and so
|
||||
contradicted the alarm on the same screen.
|
||||
|
||||
**The honest residue — still not proven:** Tier-1 **content** recovery after real loss (A2 ran on an
|
||||
intact app; A3 used Tier-2) — now the most valuable open item; host reboot mid-backup; three-way
|
||||
concurrency with GC; Scenario C live; `offsite_stale` actually firing; F-HUB `SQLITE_BUSY`.
|
||||
## Live replay (demo-hp + demo-felhom)
|
||||
|
||||
## Recovery
|
||||
| # | scenario | result |
|
||||
|---|---|---|
|
||||
| 1 | **crash loop alarms** | Ten consecutive 30 s samples **silent** through the threshold window, then `17:05:40 Event pushed: app_start_failed (warn) — Telepített alkalmazás nem fut: Uptime Kuma` — **5m25s** after the loop began (5 min + one scan). |
|
||||
| 1b | **heartbeat COUNTS it** | `17:06:10 [deadapp] check alive: 20 scans since boot, 2 deployed app(s) evaluated, **1 currently down**` — Campaign 9's evidence was `0 currently down` while an app looped. |
|
||||
| 2 | **normal deploy is silent** | A real `uptime-kuma` deploy produced only `app_deployed (info)`; no alarm, with deadapp-check running every 30 s throughout. |
|
||||
| 3 | **restore refuses without an outage** | The honest message rendered; `[WARN] Tier-2 file restore refused up front: stack=bookstack has no restorable subtree in its copy (unit_present=true) — app NOT stopped`; BookStack uptime unbroken. |
|
||||
| 4 | **paperless still restores** (regression guard on Campaign 9's headline) | 3 files deleted → restored → **43/43 byte-identical to the pre-deletion sha256 set**, `documents_ok 16 of 16 problems []`. |
|
||||
|
||||
Every config reverted from `evidence/config-before/REVERT.md`, each verified with a **positive
|
||||
observable**: agent cadences back to `0 / 302400 / 604800` on both hosts (`is-active` = active),
|
||||
windows back to `02:30`, `pvesm` shows `felhom-pbs active` on both, 0 campaign iptables rules on
|
||||
either host or guest, 0 scratch guests in the `990000` band, all stacks healthy on both boxes, and
|
||||
the offsite tier not merely unblocked but **proven working again** (`ok`, 1m35s, 8 snapshots).
|
||||
## Filed, not fixed
|
||||
|
||||
One benign residue: the in-memory R-88 breaker still holds a `felhom-pbs` failure count on each box.
|
||||
Its `until` is long past so it blocks nothing; it clears on the next successful backup or any
|
||||
controller restart (by design, not persisted). Clearing it would have cost another app outage for no
|
||||
benefit.
|
||||
- **C9-F1b** — route class-B apps to the Tier-1 unit restore from the card the customer already
|
||||
opened. Its own task **deliberately**: it puts a DESTRUCTIVE operation (overwrites live data with
|
||||
the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry
|
||||
that difference.
|
||||
- **C9-F4** — **nothing reads the Tier-2 copy's `recovery-unit/` mirror.** Written by every Tier-2 run
|
||||
(`tier2.go:369`), read by no path: `RecoveryUnitPath` resolves to `backups/**primary**/`
|
||||
(`appbackup/paths.go:46-48`), and the only reader of the secondary tree is `tier2_restore.go:79`.
|
||||
Tier-2 exists for the case where the PRIMARY drive is lost — and in exactly that case the primary
|
||||
unit is gone while this mirror survives on the second drive, unreachable by any customer action,
|
||||
leaving offsite as the only route. **Potentially larger than C9-F1.**
|
||||
|
||||
**One operational lesson worth a runbook line:** a hand-run `docker compose up -d` in
|
||||
`/opt/docker/stacks/<app>` starts a Felhom app **without its secrets** — they are injected by the
|
||||
controller's `stackEnv` at start time, not stored in a `.env`. It turned a healthy docmost into a
|
||||
crash loop during recovery. Manual recovery must go through `POST /api/stacks/<name>/restart`.
|
||||
## Tests
|
||||
|
||||
`go test ./...` **rc=0, 27 packages** — run and `rc` read *separately* from the commit. Six red-proofs
|
||||
all observed, including the one that matters most: adding `StateRestarting` to `IsDownState` fails the
|
||||
brief-restart test with *"every deploy and update would page the operator"*.
|
||||
|
||||
## Observations
|
||||
|
||||
- **A seventh shipped-invariant-comment.** `controller/README.md` stated that faults "still surface as
|
||||
`exited`/`degraded`/**`restarting`**/`unhealthy`" — but `restarting` was in no down set at all. The
|
||||
sentence was a wish with no test pinning it. Corrected in place, with the threshold rule documented
|
||||
beside it.
|
||||
- **Pre-existing gate failure, not mine:** `scripts/docker_run_volume_path_gate.py` fails on
|
||||
`internal/appexport/estimate.go:179` (an unreviewed `docker run -v`). Verified it fails identically
|
||||
on clean HEAD; left alone as out of scope rather than silently "fixed".
|
||||
- The other six gates pass (`template_id`, `emoji`, `mojibake`, `native_confirm`, `app_row_dedup`,
|
||||
`offbox_rename`).
|
||||
|
||||
## NOT yet live-validated — carried forward
|
||||
|
||||
- **Tier-1 content recovery after real loss** — still the most valuable unproven item (Campaign 9's A2
|
||||
ran against an intact app; A3 used Tier-2). Unchanged by this work.
|
||||
- The C9-F2 threshold under a **quiesce** cycle (Scenario C) is unit-proven but was not replayed live;
|
||||
it needs a backup window plus an app that fails to come back.
|
||||
- C9-F1's refusal for the other 42 class-B apps is proven by enumeration and by BookStack live, not
|
||||
app-by-app.
|
||||
- Host reboot mid-backup, three-way concurrency with GC, Scenario C live, `offsite_stale` firing,
|
||||
F-HUB — all still open from Campaign 9.
|
||||
|
||||
@@ -214,6 +214,17 @@ defect — each is the correct consequence of not being in the last successful c
|
||||
|
||||
## 3. Findings
|
||||
|
||||
> **SHIPPED 2026-07-28 — controller v0.183.0.** F-1's honesty half and F-2 in full. Phase 0 of the fix
|
||||
> sized F-1 properly: **43 of the 53 catalog apps** have no restorable subtree (not 2), 9 are covered
|
||||
> only for their file legs, 1 is stateless. The restore now refuses **before** stopping the app and
|
||||
> names the action that works; a run that proceeds claims only what it **examined** and discloses that
|
||||
> the database and volumes are not covered. F-2 alarms after a 5-minute sustained-restarting threshold
|
||||
> — proven live: silent through ten 30 s samples, then `app_start_failed` at 5m25s, and the F-OBS
|
||||
> heartbeat now reads `1 currently down` where Campaign 9 recorded `0`. Completeness is filed as
|
||||
> **C9-F1b**, and **C9-F4** was filed during the fix: nothing reads the Tier-2 copy's `recovery-unit/`
|
||||
> mirror at all. See `REPORT.md` and CHANGELOG v0.183.0.
|
||||
|
||||
|
||||
### F-1 — the restore button reports success for apps it can never restore — **DEFECT, HIGH**
|
||||
|
||||
**Applies to:** every app whose data lives entirely in Docker named volumes and which therefore has
|
||||
|
||||
@@ -28,9 +28,11 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **F-A1** | ~~A restore-test in flight made a healthy backup report as FAILED (HTTP 409 read as a tier failure): breaker armed + operator emailed, on both boxes~~ | **SHIPPED + PROVEN-LIVE** (controller v0.179.0, 2026-07-28) | — | 409 → contention: tier stays DUE, dropped before anything stops (15m), and BLOCKED alarm if contention outlives the agent's 120m ceiling (3h). Hub DB: 409 → **0** operator emails, real failure → **1** | — |
|
||||
| **R-100** | ~~A restic offsite tier that fails every night never goes stale on the hub — `isStale` counted from `LastRun`, which the controller writes unconditionally on failure~~ | **SHIPPED + PROVEN-LIVE** (controller v0.181.0 + hub v0.80.0, 2026-07-28) | — | Anchored on a new `last_success`. **Severity corrected during Phase 0: this was NOT a silence** — `backup_failed` does fire nightly and reaches the operator (live DB: 5 sends). The real defect is **defeated defence in depth**: the hub-side *pull* net was anchored on a field the failing controller keeps refreshing, so it could not compensate for a lost *push* (cf. F-HUB). Live on demo-hp: induced failure → `last_run` advanced 11:25:48Z, `last_success` **held** 11:24:20Z; demo-felhom healthy → anchor advanced. Legacy degrade logged once per customer, live | — |
|
||||
| **R-101** | ~~Tier-2 `LastRun` is written on failure and rendered to the customer as „Legutóbbi másolat" — including in the restore confirm dialog~~ | **SHIPPED + PROVEN-LIVE** (controller v0.182.0, 2026-07-28) | — | `CrossDriveBackup.LastSuccess` + `SuccessTracked`; the dialog names the last **successful** copy and discloses a failed newest attempt. Legacy rows migrate truthfully on first touch (an `ok` row adopts its time; an `error` row seeds nothing) — without the marker all 7 fleet rows would have flipped to „Még nincs sikeres másolat" on deploy. **Part 2:** the three `record*` sites rebuilt the whole struct; replaced by `tier2Update` (copy-and-overlay, safe by construction) — the naive fix would have had `recordTier2Failure` CLEAR the anchor. Live on demo-hp, rendered dialog read in both states | — |
|
||||
| **C9-F1** | Tier-2 „Fájlok visszaállítása" is offered for apps whose Tier-2 copy has **no restorable file leg** (BookStack, Docmost = 2 of the 4 Tier-2 apps on demo-felhom). Clicking it stops the app, restores 0 files, and reports **„Nincs hiányzó fájl — minden fájl megvan a helyén."** — indistinguishable from a genuine "nothing was missing", on every click, forever, while 156 MB of that app's data sits unread in `recovery-unit/` in the same copy | **READY (HIGH)** | — | Either don't render the button when the copy has no `hdd/`+`userdata/` leg, or make the message distinguish "nothing missing" from "nothing restorable here". `tier2_restore.go:101-104` reads only those two subtrees; `tier2_restore_test.go:219` pins 0-files-as-success and explicitly accepts the outage; `tier2_honesty_test.go` Scenario D asserts the button MUST render for exactly BookStack's shape | CC |
|
||||
| **C9-F2** | An app in a **Docker crash loop never alarms on any channel** — no dashboard banner, no `app_start_failed`, no customer email, no hub event. `aggregateState` returns `StateRestarting` (`manager.go:753-756`, checked before the degraded branch) and `IsDownState` (`manager.go:54-56`) excludes it, so `classifyRunStates` computes `down=false`. With `restart: unless-stopped` this is **permanent** silence. F-CRIT-1's fix covers `StateStopped` only | **READY (HIGH)** | — | Add `StateRestarting` to the down set with a debounce, or watch `RestartCount`. Proven live on demo-felhom: 9+ min crash loop, restartcount 18, while the F-OBS heartbeat printed **"180 scans since boot, 4 deployed app(s) evaluated, 0 currently down"** | CC |
|
||||
| **C9-F1** | ~~Tier-2 „Fájlok visszaállítása" is offered for apps whose copy has no restorable file leg; stops the app, restores 0 files, reports „Nincs hiányzó fájl — minden fájl megvan a helyén."~~ | **SHIPPED + PROVEN-LIVE** (controller v0.183.0, 2026-07-28) | — | **Phase 0 sized it: 43 of 53 catalog apps read NOTHING, 9 read file legs but never their DB/volumes, 1 stateless.** Honesty half shipped: `Tier2RestoreCoverage` refuses UP FRONT without stopping the app and NAMES the working action; a run that proceeds claims only what it **examined** and discloses that the database and volumes are not covered. Live on demo-felhom: bookstack refused, uptime stayed „Up About an hour" (was „Up 25 seconds"); paperless A1 re-run still byte-identical, 16/16 docs clean | — |
|
||||
| **C9-F2** | ~~An app in a Docker crash loop never alarms on any channel; `StateRestarting` is in no down-set~~ | **SHIPPED + PROVEN-LIVE** (controller v0.183.0, 2026-07-28) | — | `StateRestarting` deliberately NOT added to `IsDownState` (that alarms on every deploy fleet-wide); a SUSTAINED run becomes down after `crashLoopAfter`=5m, set above the 120s deploy timeout, Mealie's 60s start_period and R-97b's 180s grace. Dashboard counter uses the same predicate so it no longer contradicts the alarm. Red-proof that matters: the naive `IsDownState` change fails the brief-restart test | — |
|
||||
| **C9-F3** | An **interrupted offsite run leaves an exclusive restic lock the self-heal cannot reach**: `resticStep` (`offbox.go:634-648`) has `unlock --remove-all`, but `ensureOffboxRepo`'s probe fails first, `classifyResticProbe` (`offbox.go:77-93`) has no lock case → `"other"` → fail-fast. Tier dead until a human unlocks; `ClassifyOffsiteFailure` likewise has no lock case so the operator is told **„A távoli mentés ismeretlen okból nem sikerült"** for a precisely-known, self-healable condition | **READY (MEDIUM)** | — | Add a lock case to both classifiers and let the probe path escalate to `unlock --remove-all`. Answers Phase C item 8: the repo is NOT usable after a killed run. Cleared manually this run; tier proven working again (`ok`, 1m35s). Reachable by any interruption — container restart, OOM, **host reboot mid-backup** | CC |
|
||||
| **C9-F1b** | Tier-2's restore cannot cover 43 of 53 apps; the action that CAN is the keep-side unit restore (`POST /backup/restore` → `RestoreFromRecoveryUnit`, replays volume tars + DB dumps). v0.183.0 NAMES it in the refusal text but does not route to it | **READY** | — | Put the working action in the card the customer already opened. **Deliberately its own task:** it places a DESTRUCTIVE operation (overwrites live data with the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry that difference — the reason it was not folded into v0.183.0 | CC |
|
||||
| **C9-F4** | **Nothing reads the Tier-2 copy's `recovery-unit/` mirror.** It is written by EVERY Tier-2 run (`tier2.go:369`, „Unit leg (always)") and read by no code path: `RecoveryUnitPath` resolves to `backups/**primary**/` (`appbackup/paths.go:46-48`), and the only reader of the secondary tree is `tier2_restore.go:79`, which reads `hdd/`+`userdata/` only | **READY (potentially > C9-F1)** | — | Tier-2 exists for the case where the PRIMARY drive is lost — and in exactly that case the primary unit is gone while this mirror survives on the second drive, unreachable by any customer action, leaving offsite as the only route. Verified by enumeration: 6 references to `"secondary"` in the tree, one writer, one reader, one wipe-warning lister | CC |
|
||||
| **F-DIAG** | ~~Four distinct offsite failure causes collapse into two operator-visible strings~~ | **SHIPPED** (controller v0.182.0, 2026-07-28) | — | `ClassifyOffsiteFailure` → quota / orphaned / no_repo / no_units / transport / **unknown**, each with its own Hungarian message. Unclassifiable says so rather than being folded into a neighbour. **Secrets:** the old message was a raw `err.Error()` passthrough carrying `sftp:<user>@<host>:<path>`; redaction is now by the target's **actual** host/user/path (a first regex-only attempt leaked on a bare hostname and its own test caught it). Unit-proven; **not** yet exercised by a live offsite failure of each class | — |
|
||||
| **F-OPS** | ~~A manual `pct restore` inherits the source guest's bind mounts — during a real DR, on a different host, under pressure~~ | **DOCUMENTED** (2026-07-28) | — | `documentation/runbooks/RUNBOOK-manual-guest-restore.md`: which `mpN` are volumes vs host binds, the `mp9` source-VMID trap (it can bind **another guest's bootstrap credentials**), strip-and-re-add before first boot, and a positive pre-start verification. Docs only by design — the agent already neutralises binds on its own restore paths, and a second implementation would drift | — |
|
||||
| **F-REBOOT** | ~~A guest rebooted during its backup does not come back — shutdown completes, start never happens, no self-heal; 9m47s total appliance outage with every alarm silent~~ | **SHIPPED + PROVEN-LIVE** (agent v0.107.0, 2026-07-28) | — | 60 s guest-power watchdog; `onboot` is the deliberate-stop discriminator (already the stale-lock path's, and what `pve-guests` consults), retry bounded 3x/1m-2m-4m then escalates once. Live on demo-hp: **120 s unattended** vs the incident's 587 s with a human; Scenario B proven (an `onboot:0` guest left stopped) | — |
|
||||
|
||||
Reference in New Issue
Block a user