# REPORT — R-82: the backup target split (2026-07-26) Local **daily** + offsite **weekly**, made expressible at all. Spans four artifacts: agent **v0.97.0 → v0.103.0**, controller **v0.174.0 → v0.175.0**, hub **v0.76.0**, host-install **1.20.0**. Phase-0 gates: `documentation/audits/SPIKE-r82-phase0-2026-07-26.md`. --- ## 1. What was wrong `BackupTarget()` returned ONE string and `BackupCadence()` ONE 24 h window, so "local daily **and** PBS weekly" could not be said. The consequence was not theoretical: the DR tier reported `applied` since 2026-07-21 while demo-felhom held **one** snapshot (2026-07-18, a healing artifact) and demo-hp held **zero, ever**. R-39 was "applied and dead"; this was **applied and empty** — the same shape, quieter, and it would have surfaced first at a real restore. ## 2. Phase 0 — three gates | Gate | Verdict | |---|---| | P0.1 what is exposed for 7 days | **weekly CONFIRMED.** The only 7-day-exposed state is the non-SMB half of `settings.json`. `encryption.key` and the offbox credentials are **stable files unchanged since first boot**, so a week-old copy is byte-identical — that was the risk that could have overturned it | | P0.2 the `pvesm status` 0/0/0 anomaly | **RESOLVED, benign.** PBS returns HTTP 200 with zeroed usage to a namespace-scoped token (`DatastoreBackup`, not `Datastore.Audit`). Ground truth via the hub's ep0 `df`: the datastore is real and writable | | P0.3 capacity | **STOP raised; operator ruled to proceed and grow later.** 37.2 GB total. Per-tenant encryption means **no cross-customer dedup** | **Capacity, now measured rather than bracketed:** the second weekly snapshot cost **+2.7 GB on disk** against 14.46 GB logical (~81 % dedup). Weekly top-ups are cheap; **first** snapshots are not — one customer at two retained snapshots costs ~13.5 GB, so the 80 % warn arrives at roughly the **first** additional customer, not the second as I first estimated. Recorded in `07-backup-architecture.md` §9.1. ## 3. What shipped - **Agent** — `backup_targets[]`: each tier carries its **own** cadence, retention and wait bound (`keep-last=3` is three DAYS on a daily tier and three WEEKS on a weekly one; one shared knob guarantees one of them is wrong). `/backup/due?target=` judges a tier against **its own** newest successful backup. `GET /backup/tiers` is the controller's capability probe. One runner per tier. - **Controller** — every due tier collected up front and run in **ONE quiesce window**. Two cycles on the weekly night would mean two app outages for one night's work. The app stays quiesced until the **last** tier snapshots, so every tier is app-consistent. - **Hub** — per-tier thresholds (host 26 h, offsite 8 d), preserving R-81's three-valued verdicts, anchored absence and distinct reason strings. Classification is by **target type** (`target_id` → `storage_targets[].name` → `.type`), never by array membership. - **Installer** — a fresh box defaults to local-daily + offsite-weekly; an unprovisioned tier **defers** rather than firing at a storage that does not exist. **The untargeted local-API contract is frozen.** No `?target=` ⇒ the primary tier, same response **bytes** (`Target` is `omitempty` and stays empty). An old controller cannot tell the new agent from the old one; a new controller against an old agent degrades on a 404 probe, logs once, and **still takes the backup**. ## 4. Operator rulings (2026-07-26), all implemented | Ruling | Implementation | |---|---| | Two weeks of offsite backups | `keep_last=2`; the blanket PBS-prune refusal scoped to *additional* tiers with an explicit setting — the primary keeps the absolute refusal, because its target **and** retention both default and could prune the DR by accident | | Grow the datastore before any real tester | recorded in `07` §9.1; no action taken | | First backup runs as long as needed; nothing else starts until done | wait bound → 12 h (measured ~5 h for a first full snapshot); **one backup at a time per guest** — a second tier gets a 409 naming the busy tier, with no job id it could mistake for its own; a tier overrunning the quiesce bound defers the rest | | Drill box is temporary | dropped from the rollout | | Restore test, then next slice | done — see §6 | ## 5. Four defects found by RUNNING it, not reviewing it 1. **30-minute wait bound vs a 41-minute backup** (v0.98.0). The agent recorded `success:false` **while the vzdump was still running**, and it later completed `TASK OK`. Not "the backup didn't happen" but worse: the tier stays permanently due and the retry collides with the guest lock. 2. **The restore tier read from the configured target, not the archive** (v0.100.0). A `felhom-pbs:` archive was classified `local` and got the 10-minute bound against a 14.46 GB WAN restore, failing at 600 s. **This was a silent regression of the S4.1 fix** — the mechanism was never removed, its *input* changed when `local_backup_target` was retargeted to `local`. The lesson is not "add a timeout" (one was already there) but that a fix keyed on *"the configured target"* stops holding the moment more than one target exists. Recorded in `06-offsite-connectivity.md`. 3. **A leaked scratch guest kept `onboot: 1`** (v0.101.0) — a host reboot would have started a clone of the live guest. Now `onboot=0` is set **at restore time**, because "after" is the path that leaks. 4. **A tier fires at a not-yet-provisioned storage** (v0.102.0) — would have quiesced the apps and failed every cadence on a fresh box until DR provisioning. ## 6. Live validation **demo-felhom** — the first real PBS-targeted backup: **`TASK OK`, 41 minutes, 14.46 GB snapshot**, and it **restored cleanly** (`vzrestore: stopped OK`, all volumes back). Both tiers armed and verified over the real local API; the untargeted response confirmed byte-identical. **demo-hp** — reached via the documented break-glass path; binary and config backed up first. Its **first ever** PBS backup **landed**: 4.25 GB into a namespace that was verifiably empty. **THE RESTORE ROUND-TRIP PASSED** — the bar for calling a tier real: ``` source_archive : felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z source_tier : pbs <- the v0.100.0 fix; the earlier attempt said "local" and died at 600s pass : true verified : boot+running mount_parity : ok <- mp0=/var/lib/docker 50G, mp1=/mnt/sys_drive 20G, mp8/mp9 stand-ins duration : 4m5s (restore + boot + verify + teardown) ``` `mount_parity` is the non-hollow half: a boot-only verify cannot see a missing data volume. The scratch guest tore down **cleanly** — no 403, no leak — which confirms `06-offsite-connectivity.md`'s reading that the teardown 403 was a **phantom** (a consequence of the short timeout, not an ACL gap), and corrects my earlier framing of it as a standing privilege gap. Afterwards: scratch band empty, thin pool back to its exact pre-restore figure, live guest running, snapshot intact. **THE MULTI-TIER QUIESCE RAN LIVE**, through the real UI endpoint (`POST /api/guest-backup/trigger`, session auth + CSRF — the exact call the "Mentés most" button makes): ``` 17:01:39 manual backup requested — quiescing now 17:01:39 backup due on 2 tier(s) — quiescing 1 stack(s): [paperless-ngx] <- ONE stop 17:01:46 tier local: backup job ... started 17:02:56 tier local: ... done — next tier may start (app still quiesced) <- app stays DOWN 17:02:56 tier felhom-pbs: backup job backup-9201-felhom-pbs-... started 17:03:06 tier felhom-pbs: ... snapshotted — resuming app early (8B.2) 17:03:06 unquiescing (snapshotted (early resume, last tier)): restarting 1 stack(s) <- ONE start ``` **Exactly one stop/start pair with both backups inside it** — the assertion that matters, since "both backups ran" would also pass against an implementation that quiesces twice. Tier order was local-first/PBS-last as designed, the app stayed quiesced through the non-last tier (app-consistency preserved on the DR tier), and it resumed at the **last** tier's snapshot rather than waiting for the upload. **Total app downtime 1m27s for both tiers**; paperless came back healthy. **Hub Slice C replayed against the live DB before deploying:** ``` demo-felhom host=07-26T14:38Z offsite=07-26T12:21Z -> OK demo-hp host=07-26T07:06Z offsite=none -> UNKNOWN (119h of a 192h grace) drill-r50 host=none offsite=not expected -> MISSED (no evidence in 29h) ``` **No customer email results from the deploy.** demo-hp defers correctly and will alarm in ~3 days if its offsite tier stays empty — the true finding arriving on schedule, not a false alarm. ## 7. A correction I had to make mid-arc I reported that the restore-test would boot a scratch guest carrying the live guest's MAC, static island IP and hostname, and so would break the controller→agent link. **That was wrong.** `RunRestoreTest` step 2 link-downs **every** interface before the guest is started, and it is unit-tested. I read a restored config artifact, inferred the boot behaviour from it, and escalated before reading the code path that consumes it. I also disabled the scheduled restore-test on that basis, which was an unnecessary reduction in safety coverage; it is re-enabled. The residual hazard was real but far narrower — it needed the restore to fail *before* the link-down step, which is what defect 1 caused — and that is what v0.101.0 fixes. Separately, `06-offsite-connectivity.md` records that the teardown `403` I flagged as a standing privilege gap was **already diagnosed in S4.1 as a phantom**: it is a consequence of the short timeout, not an ACL problem. With the timeout fixed the guest is pool-associated by teardown time. ## 8. Tests | Repo | Result | |---|---| | felhom-agent | `build/vet/test` rc=0, **29 packages** | | felhom-controller | `build/vet/test` rc=0, **27 packages** | | felhom.eu (hub) | `build/vet/test` rc=0, **17 packages** | Red-proofs observed and restored for every mandatory scenario: old-controller compat, new-controller degrade (the hollow version asserts "no error" while silently skipping the backup), the both-due night (**the COUNT is the assertion** — asserting only "both ran" passes against a double-quiesce), the merged threshold, the per-tier wait bound, the `onboot` override, and the overrun defer. **A process failure worth recording:** I ran the agent suite and committed in the same command, read `packages ok: 28`, and pushed **without reading `rc=1`**. Five of my own Slice A tests were failing — a harness artifact, not a product bug, but the commit went out red. Fixed in `13ca2d9`. This is the exact exit-code trap recorded twice earlier in this arc. ## 9. NOT done — explicitly 1. **The offsite tier is never AUTOMATICALLY restore-tested.** The scheduled restore-test picks candidates from a runner built on the primary target, so it can never select a PBS archive. The **manual/selftest** path is now proven end-to-end; the **unattended** one is not. Needs a per-tick spec. 2. **The hub infers "PBS ⇒ weekly" from storage TYPE.** `defaultBackupTarget` is `felhom-pbs`, so a box that never sets `local_backup_target` would run PBS as its **daily** tier and be judged against 8 days — seven days of blindness. No box is in that shape today. The real fix is the agent reporting each tier's actual cadence. 3. **The installer-default fleet flip** (Slice D step 4) waits on a full weekly cycle holding — a genuine gate, not an oversight. 5. **`07-backup-architecture.md` is NOT ratified** — brought current with an honest staleness header; ratification is Viktor's review of the §10 list. ## 10. Observations - The `felhom-pbs` PVE storage will permanently show **0 %** in the PVE UI (namespace-scoped token). Operators must read fill from the hub's PBS-DR gauge. Worth a runbook line. - demo-felhom's guest grew **9.74 → 14.46 GB logical in eight days**. Probably one-off from app testing, but if it is a rate the capacity sizing changes quickly. - demo-felhom was enabled **before** demo-hp, out of the specified rollout order, because Slice A could not be validated otherwise. - The customer-facing Hungarian copy still overstates scope ("A mai biztonsági mentés nem készült el a határidőig!" covers only the host/PBS tier). Unchanged; flagged since the R-80 diagnostic.