Claude Code
e4f22f4c4f
Four defects found by running it rather than reviewing it, the frozen untargeted contract verified live, the fail-safe directions stated once, and what is NOT done — including that the scheduled restore-test never selects the offsite tier and that R-84 is now closer to a prerequisite than a tidy-up.
93 lines
5.2 KiB
Markdown
93 lines
5.2 KiB
Markdown
# REPORT — R-82: per-target backup tiers, agent v0.97.0 → v0.102.0 (2026-07-26)
|
|
|
|
**Overwritten** per the standing rule. Live on **demo-felhom** and **demo-hp**.
|
|
Full cross-repo arc + the Phase-0 gates: `felhom.eu/REPORT.md`.
|
|
|
|
## Versions in this arc
|
|
|
|
| | |
|
|
|---|---|
|
|
| **v0.97.0** | Slice A — per-target cadence/retention, `/backup/tiers`, per-tier due + runner + job keys |
|
|
| **v0.98.0** | the 30-minute false failure — per-tier vzdump wait bound |
|
|
| **v0.99.0** | operator rulings — 2-week offsite retention (scoped PBS prune opt-in) + one backup at a time per guest |
|
|
| **v0.100.0** | the restore tier comes from the ARCHIVE, not the configured target |
|
|
| **v0.101.0** | a leaked restore-test scratch can no longer auto-start (`onboot=0` at restore time) |
|
|
| **v0.102.0** | an unprovisioned tier DEFERS instead of firing at a missing storage |
|
|
|
|
## Four defects found by RUNNING it, not reviewing it
|
|
|
|
1. **v0.98.0** — a 41-minute PBS backup was recorded `success:false` at 30 minutes **while it was
|
|
still running**, and it later completed `TASK OK`. Worse than "the backup didn't happen": the
|
|
tier stays permanently due and the retry collides with the guest lock.
|
|
2. **v0.100.0** — a `felhom-pbs:` archive was classified `local` and got the 10-minute bound against
|
|
a 14.46 GB WAN restore, failing at 600 s. **A silent regression of the S4.1 fix** — the mechanism
|
|
was never removed; its *input* changed when `local_backup_target` was retargeted to `local`. A fix
|
|
keyed on *"the configured target"* stops holding the moment more than one target exists.
|
|
3. **v0.101.0** — a leaked scratch guest kept `onboot: 1`, so a host reboot would have started a
|
|
clone of the live guest. `onboot=0` is now set **at restore time**, because "after" is the path
|
|
that leaks.
|
|
4. **v0.102.0** — a tier fires at a storage that does not exist yet (a fresh box before DR
|
|
provisioning). Now deferred, going live with no restart once the storage appears.
|
|
|
|
## The frozen contract
|
|
|
|
Untargeted `/backup/due`, `POST /backup` and `/backup/status` keep the **primary tier and the
|
|
pre-R-82 response bytes** (`Target` is `omitempty` and stays empty); the primary's job-id format is
|
|
unchanged. Verified live on demo-felhom:
|
|
|
|
```
|
|
/backup/due (untargeted) {"vmid":9201,"due":true,"reason":"no successful backup recorded yet"}
|
|
/backup/due?target=felhom-pbs {...,"target":"felhom-pbs"}
|
|
/backup/due?target=bogus 400 "unknown backup target: bogus"
|
|
```
|
|
|
|
An unknown `?target=` is a **400, never a silent fallback** — a caller asking about a tier this agent
|
|
does not serve must find out, not act on another tier's freshness.
|
|
|
|
## Fail-safe directions, stated once
|
|
|
|
- Unparseable timestamp → **due** (a spurious backup is cheap; a skipped one is not).
|
|
- Storage-view error → **present** (`"I could not check"` is not `"not there"` — reading it that way
|
|
would silently suppress backups, the rule this project has relearned three times).
|
|
- Missing tier cadence → **rejected, logged at ERROR** (never defaulted: a weekly DR tier silently
|
|
running daily would fill the datastore).
|
|
- Unset `keep_last` → **never prune** (a DR tier must not start pruning itself because someone forgot
|
|
a field). The primary tier keeps the **absolute** PBS-prune refusal, because its target *and*
|
|
retention both default and could prune the DR by accident.
|
|
|
|
## Live validation (demo-felhom)
|
|
|
|
Both tiers armed: `local 24h keep_last=3 wait 30m` + `felhom-pbs 168h keep_last=2 wait 12h
|
|
prune_pbs_allowed=true`. First real PBS-targeted backup: **`TASK OK`, 41 minutes, 14.46 GB
|
|
snapshot**, and it **restored cleanly** (`vzrestore: stopped OK`). Measured incremental cost:
|
|
**+2.7 GB on disk** for a 14.46 GB logical snapshot (~81 % dedup).
|
|
|
|
demo-hp: v0.102.0 deployed via the documented break-glass path (binary + config backed up first),
|
|
tier armed, first-ever PBS backup triggered against a verifiably empty namespace.
|
|
|
|
## Tests
|
|
|
|
`go build ./... && go vet ./... && go test ./...` — **rc=0, 29 packages**, vet run unpiped.
|
|
|
|
Red-proofs observed and restored: old-controller compat (`"target":"local"` leaking into the
|
|
untargeted body), the per-tier wait bound, and the `onboot` override. A per-tier single-flight defect
|
|
was caught **by its own test** before it ever shipped — job ids were unique only by clock luck, so
|
|
the PBS request was handed the local job's id.
|
|
|
|
**Process failure recorded:** I ran the suite and committed in the same command, read
|
|
`packages ok: 28` and pushed without checking `rc=1`; five of my own tests were failing (a harness
|
|
artifact, fixed in the follow-up). That is the exact exit-code trap recorded twice earlier in this
|
|
arc.
|
|
|
|
## NOT done
|
|
|
|
1. The **scheduled** restore-test still only ever selects the **primary** tier, so the offsite tier
|
|
is never automatically restore-tested — arguably the more important half of "is the DR tier real?".
|
|
2. The agent does not report per-tier **cadences**, so the hub must infer "PBS ⇒ weekly" from storage
|
|
type.
|
|
3. The backup `Store` is still in-memory (**R-84**): every agent restart re-triggers a backup on every
|
|
tier. On the local tier that is wasted minutes; on the offsite tier it is a wasted multi-hour
|
|
upload after every deploy — which makes R-84 closer to a prerequisite than a tidy-up.
|
|
4. The boot+verify half of the restore round-trip has not run under the fixed code.
|
|
|