Move the local whole-guest backup off the guest's own device (demo-hp + demo-felhom)
Supervised operational run. No code, no version bump, nothing deleted.
Primary backup tier on both demo boxes moved from `local` (a dir storage on
/var/lib/vz -- the SAME physical device as the guest) to `felhom-backup`, a dir
storage on each box's secondary drive:
demo-hp /mnt/nvme-1tb uuid:91d2dc2d-... archive 2,256,044,492 B
demo-felhom /mnt/hdd_1 uuid:47a3361a-... archive 5,957,878,962 B
Both proven end to end via the real UI path: archive lands on the secondary
drive (df delta matches the archive byte-for-byte), restore-test auto-selects it
and passes with mount_parity: ok, and freshness survives an agent restart with
an empty in-memory store -- so the age can only have come from the new storage.
Phase 0: the target is CONFIGURATION, not converged (the sole writer of
agent.json touches only escrow.pbs_storage_id and preserves unknown keys), so
the runbook's STOP did not fire. No consumer hardcodes "local" on the backup path.
Findings:
- F-1 the storage path must BE the mountpoint; a subdirectory fails exactMount
and the target reports disconnected permanently (observe.go:321)
- F-2 --is_mountpoint 1 is load-bearing; proven live, an unguarded storage on a
non-mounted path reports active with the ROOT filesystem's free space and
had already created dump/ on pve-root -- a silent retarget onto the very
device this change escapes
- F-3 FelhomAgentStore is granted per storage path; without it every backup
403s. felhom-host-install.sh must issue it for new installs
- R-109 (new) the DR recipe records no backup target, and each box now carries
two content=backup dir storages, one live and one frozen
- R-105 narrowed and TRACED: dr_recipe drives was [] fleet-wide because the
enrolled drives were never PVE storages, so isUserDataDrive never saw
them. Both boxes now populate drives; SMART on the backup drives too
Absent-drive behaviour today is fail-loudly with no silent retarget (PVE half
live-proven; agent half source-traced). That is NOT the intended fall-back-and-
alarm design -- filed as E-2 with the honest single-drive label.
Reported in full in the record: the agent was restarted with a felhom-pbs backup
in flight, producing a spurious tier failure. The backup had in fact succeeded
(PVE task OK, 6,264,034,053 B snapshot) and the spurious failure reached no
channel -- R-84 ground truth superseded it.
Outstanding: full drive-loss recovery (needs physical access) and the agent half
of the absent-drive behaviour.
This commit is contained in:
@@ -1,103 +1,96 @@
|
||||
# REPORT — C9-F1 + C9-F2: a restore that restored nothing, and a crash loop nobody saw (2026-07-28)
|
||||
# REPORT — the local whole-guest backup moved off the guest's own device (2026-07-28)
|
||||
|
||||
**Overwritten** per the standing rule. **Shipped: controller v0.183.0**, live on **both** demo boxes.
|
||||
Fleet: hub v0.80.0, agent v0.110.0, controller **0.183.0**. `peti-felhom` untouched.
|
||||
**Overwritten** per the standing rule (the prior C9-F1/C9-F2 text stays in git history).
|
||||
**Supervised operational run — no code, no version bump.** Fleet unchanged: hub v0.80.0,
|
||||
agent v0.110.0, controller v0.183.0. `peti-felhom` untouched. **Nothing was deleted.**
|
||||
|
||||
Both defects are the same shape — the system reporting healthy while the customer is not — and both
|
||||
live in the same status-derivation code.
|
||||
Record: `documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md`.
|
||||
|
||||
## Phase 0 — the asymmetry, sized before designing anything
|
||||
---
|
||||
|
||||
Tier-2 writes **two** things on every run: the capture legs (`hdd/`, `userdata/`) and, always, a full
|
||||
`recovery-unit/` (DB dumps + named-volume tarballs). `RestoreTier2Files` reads **only the two legs**
|
||||
(`tier2_restore.go:101-104`) and has never opened `recovery-unit/`.
|
||||
## What changed
|
||||
|
||||
All 53 catalog templates enumerated, cross-checked against both boxes' actual copies:
|
||||
On both demo boxes the primary backup tier moved from `local` — a `dir` storage on `/var/lib/vz`,
|
||||
i.e. **the same physical device as the guest itself** — to **`felhom-backup`**, a `dir` storage on
|
||||
the secondary drive.
|
||||
|
||||
| class | count | what the restore can return |
|
||||
| | demo-felhom | demo-hp |
|
||||
|---|---|---|
|
||||
| **A** | **9** | the file legs only — **never** their database or named volumes |
|
||||
| **B** | **43** | **nothing at all** — a guaranteed no-op, forever |
|
||||
| C | 1 | `bentopdf`, stateless |
|
||||
| Target drive | `/dev/sdb` USB HDD → `/mnt/hdd_1` | `/dev/nvme0n1` → `/mnt/nvme-1tb` |
|
||||
| Durable id | `uuid:47a3361a-91e0-4831-a69d-27f540ed3f48` | `uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae` |
|
||||
| Archive proven | 5,957,878,962 B | 2,256,044,492 B |
|
||||
| Restore-test | `pass`, **`mount_parity: ok`**, 1 m 24 s | `pass`, **`mount_parity: ok`**, 1 m 54 s |
|
||||
|
||||
Four apps (`plex`, `jellyfin`, `emby`, `navidrome`) are in B only because their single bind is a
|
||||
`:ro` media mount, which `ClassifyBinds` correctly excludes. **81% of the catalog.**
|
||||
Three steps per box: `pvesm add dir … --is_mountpoint 1`, two `pveum acl modify`, and a one-line
|
||||
`local_backup_target` edit in `agent.json`.
|
||||
|
||||
**The asymmetry is Tier-2's alone.** Tier-1 (`RestoreFromRecoveryUnit`) and offsite both restore the
|
||||
unit and replay volume dumps — so BookStack already had a working restore; only this button lied.
|
||||
**Drive loss is now locally recoverable in principle on both boxes** — the whole-guest archive lives
|
||||
on different hardware from the guest, and a restore from it boots and passes mount parity. The proof
|
||||
that closes matrix row 4 (pull the system drive and restore for real) needs physical access and is
|
||||
outstanding.
|
||||
|
||||
## Shipped — Part 1a (honesty)
|
||||
## Phase 0 verdict
|
||||
|
||||
- **Refuses UP FRONT.** `Tier2RestoreCoverage` is consulted before any op begins; a class-B app is
|
||||
refused **without being stopped**. Live: BookStack uptime stayed `Up About an hour` (Campaign 9
|
||||
left it at `Up 25 seconds`).
|
||||
- **Names the working action** rather than dead-ending 81% of the catalog:
|
||||
> „Ennek az alkalmazásnak az adatai nem ebből a másolatból állíthatók vissza — az alkalmazás nem
|
||||
> állt le. Használd a Visszaállítás indítása gombot a Biztonsági mentés → Visszaállítás oldalon."
|
||||
- **Claims only what was examined** (the QUIET half — immich's 1.3 GB Postgres unit is not covered,
|
||||
so the old blanket sentence was a clean bill of health over data never opened):
|
||||
> „Minden vizsgált fájl megvan a helyén." + „Az alkalmazás adatbázisa és belső kötetei nem
|
||||
> tartoznak ebbe a visszaállításba."
|
||||
**The target is configuration, not converged — the runbook held and the §3 STOP did not fire.**
|
||||
Exactly one writer of `agent.json` exists (`pbsdr.seedEscrowStorageID`); it touches only
|
||||
`escrow.pbs_storage_id` and preserves unknown keys verbatim via a `map[string]json.RawMessage`
|
||||
read-modify-write. Nothing on the hub pushes agent config. No consumer hardcodes `"local"` on the
|
||||
backup path — runners are one-per-tier, `NewestArchiveTime` reads its own target, and
|
||||
`restoreTierForArchive` classifies from the archive, not from config.
|
||||
|
||||
## Shipped — Part 2 (C9-F2)
|
||||
## What the run found
|
||||
|
||||
`StateRestarting` is deliberately **NOT** added to `IsDownState` — that alarms on every deploy and
|
||||
update fleet-wide. A **sustained** run becomes down after `crashLoopAfter = 5m`, set above the three
|
||||
real numbers already in the codebase: the deploy flow's **120 s** health timeout, Mealie's **60 s**
|
||||
`start_period`, and R-97b's **180 s** quiesce grace (so the windows compose into one bounded delay
|
||||
instead of leaving a gap). Docker's backoff caps at 60 s, so a real loop registers ≥4 attempts inside
|
||||
it. Carried by `Stack.RestartingSince` (not persisted) + `Stack.CrashLooping(now)`, used by **both**
|
||||
the alarm and the dashboard counter — which previously counted `restarting` as running and so
|
||||
contradicted the alarm on the same screen.
|
||||
- **F-1 (before any command).** The storage `path` must **be** the drive's mountpoint. A
|
||||
subdirectory fails `exactMount` (`internal/storage/observe.go:321`), so the target would report
|
||||
`disconnected` **permanently** and its durable id would degrade off the filesystem UUID.
|
||||
- **F-2 (before any command).** `--is_mountpoint 1` is load-bearing. **Proven live:** an unguarded
|
||||
`dir` storage on a non-mounted path reports `active`, advertises the **root filesystem's** free
|
||||
space, and had already created `dump/` on `/dev/mapper/pve-root` — a silent retarget onto the
|
||||
exact device this change exists to escape. The guarded one refuses outright.
|
||||
- **F-3 (found by the first real backup).** `FelhomAgentStore` is granted **per storage path**; a
|
||||
new target without its own grant 403s every backup. **`felhom-host-install.sh` must issue it for
|
||||
new installs**, or a new box ships with a tier that fails on its first run.
|
||||
- **R-109 (new).** The DR recipe records **no backup target** — and each box now carries two
|
||||
`content=backup` dir storages, one live and one holding frozen 2026-07-28 archives.
|
||||
- **R-105 narrowed and traced.** `dr_recipe.host_half.drives` was `[]` fleet-wide with the cause
|
||||
untraced. Cause: the enrolled drives were never PVE storages, so `isUserDataDrive` never saw them.
|
||||
Both demo boxes now populate `drives`, and the backup drives gained SMART reporting. R-105's other
|
||||
two fields are untouched.
|
||||
|
||||
## Live replay (demo-hp + demo-felhom)
|
||||
## Absent-drive behaviour (Part 3)
|
||||
|
||||
| # | scenario | result |
|
||||
|---|---|---|
|
||||
| 1 | **crash loop alarms** | Ten consecutive 30 s samples **silent** through the threshold window, then `17:05:40 Event pushed: app_start_failed (warn) — Telepített alkalmazás nem fut: Uptime Kuma` — **5m25s** after the loop began (5 min + one scan). |
|
||||
| 1b | **heartbeat COUNTS it** | `17:06:10 [deadapp] check alive: 20 scans since boot, 2 deployed app(s) evaluated, **1 currently down**` — Campaign 9's evidence was `0 currently down` while an app looped. |
|
||||
| 2 | **normal deploy is silent** | A real `uptime-kuma` deploy produced only `app_deployed (info)`; no alarm, with deadapp-check running every 30 s throughout. |
|
||||
| 3 | **restore refuses without an outage** | The honest message rendered; `[WARN] Tier-2 file restore refused up front: stack=bookstack has no restorable subtree in its copy (unit_present=true) — app NOT stopped`; BookStack uptime unbroken. |
|
||||
| 4 | **paperless still restores** (regression guard on Campaign 9's headline) | 3 files deleted → restored → **43/43 byte-identical to the pre-deletion sha256 set**, `documents_ok 16 of 16 problems []`. |
|
||||
Today: **fail loudly, no silent retarget.** The PVE half is live-proven with throwaway storages (no
|
||||
live drive was unmounted). The agent half is source-traced: `targetStoragePresent` checks name
|
||||
presence only, never `Reachable`, so the tier stays DUE, the controller quiesces, vzdump is refused,
|
||||
and the run fails and alarms.
|
||||
|
||||
## Filed, not fixed
|
||||
**This is not §6's intended design** (fall back to the system drive and alarm). There is no fallback
|
||||
at all, so an absent drive means no local backup until a human intervenes. Filed as **E-2**, together
|
||||
with the honest single-drive label — a one-drive box protects against corruption only, and two
|
||||
drives is effectively a hardware requirement for drive-loss protection.
|
||||
|
||||
- **C9-F1b** — route class-B apps to the Tier-1 unit restore from the card the customer already
|
||||
opened. Its own task **deliberately**: it puts a DESTRUCTIVE operation (overwrites live data with
|
||||
the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry
|
||||
that difference.
|
||||
- **C9-F4** — **nothing reads the Tier-2 copy's `recovery-unit/` mirror.** Written by every Tier-2 run
|
||||
(`tier2.go:369`), read by no path: `RecoveryUnitPath` resolves to `backups/**primary**/`
|
||||
(`appbackup/paths.go:46-48`), and the only reader of the secondary tree is `tier2_restore.go:79`.
|
||||
Tier-2 exists for the case where the PRIMARY drive is lost — and in exactly that case the primary
|
||||
unit is gone while this mirror survives on the second drive, unreachable by any customer action,
|
||||
leaving offsite as the only route. **Potentially larger than C9-F1.**
|
||||
## One operational error, reported in full
|
||||
|
||||
## Tests
|
||||
**The agent was restarted on demo-hp with a `felhom-pbs` backup in flight**, against §4.3. The
|
||||
in-flight check was done before the first restart and not repeated before the second. It produced a
|
||||
**spurious tier failure** (`context canceled` while waiting) — the F-A1 class the project already
|
||||
fixed once.
|
||||
|
||||
`go test ./...` **rc=0, 27 packages** — run and `rc` read *separately* from the commit. Six red-proofs
|
||||
all observed, including the one that matters most: adding `StateRestarting` to `IsDownState` fails the
|
||||
brief-restart test with *"every deploy and update would page the operator"*.
|
||||
**The backup had not failed:** the PVE task returned OK and the PBS snapshot `2026-07-28T19:19:45Z`
|
||||
is 6,264,034,053 B. The spurious failure **reached no channel** — the agent died with its in-memory
|
||||
record and R-84 ground truth superseded it; the controller's event trail for the window shows only
|
||||
the real 403 failure and its recovery. The design absorbed it; the mistake was still a mistake, and
|
||||
on a slower tier the same slip could have aborted a multi-hour WAN upload.
|
||||
|
||||
## Observations
|
||||
Also recorded: `pgrep -f vzdump` self-matches a polling script's own command line and is not a safe
|
||||
in-flight check — use the PVE task list.
|
||||
|
||||
- **A seventh shipped-invariant-comment.** `controller/README.md` stated that faults "still surface as
|
||||
`exited`/`degraded`/**`restarting`**/`unhealthy`" — but `restarting` was in no down set at all. The
|
||||
sentence was a wish with no test pinning it. Corrected in place, with the threshold rule documented
|
||||
beside it.
|
||||
- **Pre-existing gate failure, not mine:** `scripts/docker_run_volume_path_gate.py` fails on
|
||||
`internal/appexport/estimate.go:179` (an unreviewed `docker run -v`). Verified it fails identically
|
||||
on clean HEAD; left alone as out of scope rather than silently "fixed".
|
||||
- The other six gates pass (`template_id`, `emoji`, `mojibake`, `native_confirm`, `app_row_dedup`,
|
||||
`offbox_rename`).
|
||||
## State at close
|
||||
|
||||
## NOT yet live-validated — carried forward
|
||||
Both boxes healthy. Primary and offsite tiers `due=false` with `age_state=known` on both; breakers
|
||||
clear; agents `active`; no thrash and no spurious staleness. Target drives at 1–2 % used with SMART
|
||||
`PASSED`. The old archives (18 GB demo-felhom, 5.2 GB demo-hp) are **left in place** as the rollback
|
||||
and as the only evidence of what the previous configuration produced.
|
||||
|
||||
- **Tier-1 content recovery after real loss** — still the most valuable unproven item (Campaign 9's A2
|
||||
ran against an intact app; A3 used Tier-2). Unchanged by this work.
|
||||
- The C9-F2 threshold under a **quiesce** cycle (Scenario C) is unit-proven but was not replayed live;
|
||||
it needs a backup window plus an app that fails to come back.
|
||||
- C9-F1's refusal for the other 42 class-B apps is proven by enumeration and by BookStack live, not
|
||||
app-by-app.
|
||||
- Host reboot mid-backup, three-way concurrency with GC, Scenario C live, `offsite_stale` firing,
|
||||
F-HUB — all still open from Campaign 9.
|
||||
**Outstanding:** full drive-loss recovery (physical access), and the agent half of the absent-drive
|
||||
behaviour (needs a drive unmount that would break the guest bind on a remote box).
|
||||
|
||||
Reference in New Issue
Block a user