diff --git a/REPORT.md b/REPORT.md index 367fa5e..ec5028f 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,126 +1,70 @@ -# REPORT — R-82: per-target backup tiers, agent v0.97.0 → v0.102.0 (2026-07-26) +# REPORT — R-88 Part 2: the agent can say "unknown" (v0.105.0) (2026-07-27) -**Overwritten** per the standing rule. Live on **demo-felhom** and **demo-hp**. -Full cross-repo arc + the Phase-0 gates: `felhom.eu/REPORT.md`. +**Overwritten** per the standing rule. Agent **v0.104.0 → v0.105.0**. Producer half of a wire +contract; the controller consumer is v0.178.0. -## Versions in this arc +## What was broken -| | | +`newestArchiveOn` promised in its own doc comment that *"errors and unsupported services degrade to +'unknown', never to 'no backup'"*. Its `(time.Time, bool)` signature made that **impossible**: an +error and a genuine not-found both returned `(zero, false)`, so `/backup/due` emitted a POSITIVE +`"no successful backup recorded yet"` with a nil age. The controller read that as *never backed up*, +fired its window-gate safety valve, and quiesced customer app stacks **outside** the backup window — +what happened during the 2026-07-27 PBS outage. The comment described an intent the type forbade. + +## The wire encoding, and how the zero value is handled + +**A string enum**, matching the repo's existing convention (`phase` on `/backup/status`): + +``` +age_state: "known" | "absent" | "unknown" (omitted entirely by a pre-v0.105.0 agent) +``` + +The zero value is `""`, and it means **"legacy agent, no information"** — never "unknown". A `bool` +would have made the legacy case indistinguishable from a real `false`, which is the trap the task +named. The controller maps `""` (and any unrecognised future value) to `AgeStateLegacy` explicitly. + +Internally `archiveLookup` (`found`/`absent`/`unknown`) replaces the old bool, so the doc comment's +promise is now something the type can actually express. + +## Two decisions worth stating + +**The fail-safe direction is unchanged: unknown is still DUE.** An agent that cannot read the storage +must never suppress a backup. Only the *window-gate bypass* narrows. + +**A service with NO lister deliberately stays `absent`, not `unknown`.** This broke a pre-existing +test (`TestBackupDue_ServiceWithoutLister_UnchangedBehaviour`) and the test was right: calling it +"unknown" would stop the controller firing its first-backup valve on a pre-R-84 build, so a genuinely +new box would never back up outside its window and nobody would notice for weeks. On that path the +in-memory record is the only registry that exists, so its absence means "no backup recorded" in the +only terms available. `unknown` is reserved for a lister that was asked and could not answer. + +An **unparseable** in-memory timestamp is now `unknown` too — a backup did happen, we simply cannot +date it. It previously fell through to the same positive "never" claim. + +## Tests — 29 packages ok, 0 failed; `build`/`vet`/`test` each rc=0, run separately, vet unpiped + ++5. Red-proofs observed and restored, in both directions: + +| Mutation | Observed | |---|---| -| **v0.97.0** | Slice A — per-target cadence/retention, `/backup/tiers`, per-tier due + runner + job keys | -| **v0.98.0** | the 30-minute false failure — per-tier vzdump wait bound | -| **v0.99.0** | operator rulings — 2-week offsite retention (scoped PBS prune opt-in) + one backup at a time per guest | -| **v0.100.0** | the restore tier comes from the ARCHIVE, not the configured target | -| **v0.101.0** | a leaked restore-test scratch can no longer auto-start (`onboot=0` at restore time) | -| **v0.102.0** | an unprovisioned tier DEFERS instead of firing at a missing storage | -| **v0.103.0** | **R-84** — an agent restart no longer triggers a redundant backup | +| error path → `archiveAbsent` (the pre-fix collapse) | `an unreadable storage must report age_state="unknown", got "absent" — that is a POSITIVE claim of 'never backed up' built out of two absences` | +| `!found` → `archiveUnknown` (the over-correction) | `a genuine never-backed-up tier must report age_state="absent", got "unknown" — the controller only licenses a first backup outside the window on ABSENT` | -## Four defects found by RUNNING it, not reviewing it +## Deployed — producer BEFORE consumer -1. **v0.98.0** — a 41-minute PBS backup was recorded `success:false` at 30 minutes **while it was - still running**, and it later completed `TASK OK`. Worse than "the backup didn't happen": the - tier stays permanently due and the retry collides with the guest lock. -2. **v0.100.0** — a `felhom-pbs:` archive was classified `local` and got the 10-minute bound against - a 14.46 GB WAN restore, failing at 600 s. **A silent regression of the S4.1 fix** — the mechanism - was never removed; its *input* changed when `local_backup_target` was retargeted to `local`. A fix - keyed on *"the configured target"* stops holding the moment more than one target exists. -3. **v0.101.0** — a leaked scratch guest kept `onboot: 1`, so a host reboot would have started a - clone of the live guest. `onboot=0` is now set **at restore time**, because "after" is the path - that leaks. -4. **v0.102.0** — a tier fires at a storage that does not exist yet (a fresh box before DR - provisioning). Now deferred, going live with no restart once the storage appears. -5. **v0.103.0 (R-84)** — three redundant local backups ran on demo-felhom in one afternoon of - deploys, because the in-memory `Store` is empty after a restart and the due-check read that as - "no backup ever". On the offsite tier that is a wasted multi-hour WAN upload after every agent - deploy. The due-check now consults the STORAGE (ground truth: a pruned archive correctly stops - counting, where a persisted record would keep claiming a backup that no longer exists). - -## The frozen contract - -Untargeted `/backup/due`, `POST /backup` and `/backup/status` keep the **primary tier and the -pre-R-82 response bytes** (`Target` is `omitempty` and stays empty); the primary's job-id format is -unchanged. Verified live on demo-felhom: +demo-hp then demo-felhom, both `felhom-agent 0.105.0`, service active, previous binary kept as +`.bak-0.104.0`. **Verified on the live wire** from inside guest 9201, over the exact route the +controller calls: ``` -/backup/due (untargeted) {"vmid":9201,"due":true,"reason":"no successful backup recorded yet"} -/backup/due?target=felhom-pbs {...,"target":"felhom-pbs"} -/backup/due?target=bogus 400 "unknown backup target: bogus" +tier=local due=True age_state=known age_seconds=91097 reason=older than cadence +tier=felhom-pbs due=False age_state=known age_seconds=100172 reason=within cadence window ``` -An unknown `?target=` is a **400, never a silent fallback** — a caller asking about a tier this agent -does not serve must find out, not act on another tier's freshness. - -## Fail-safe directions, stated once - -- Unparseable timestamp → **due** (a spurious backup is cheap; a skipped one is not). -- Storage-view error → **present** (`"I could not check"` is not `"not there"` — reading it that way - would silently suppress backups, the rule this project has relearned three times). -- Missing tier cadence → **rejected, logged at ERROR** (never defaulted: a weekly DR tier silently - running daily would fill the datastore). -- Unset `keep_last` → **never prune** (a DR tier must not start pruning itself because someone forgot - a field). The primary tier keeps the **absolute** PBS-prune refusal, because its target *and* - retention both default and could prune the DR by accident. - -## Live validation (demo-felhom) - -Both tiers armed: `local 24h keep_last=3 wait 30m` + `felhom-pbs 168h keep_last=2 wait 12h -prune_pbs_allowed=true`. First real PBS-targeted backup: **`TASK OK`, 41 minutes, 14.46 GB -snapshot**, and it **restored cleanly** (`vzrestore: stopped OK`). Measured incremental cost: -**+2.7 GB on disk** for a 14.46 GB logical snapshot (~81 % dedup). - -**demo-hp — the box that had ZERO offsite snapshots, ever:** deployed via the documented break-glass -path (binary + config backed up first), tier armed, and its **first ever** PBS backup **landed** — -`felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z`, **4.25 GB**, into a namespace that was verifiably -empty beforehand. That is the R-82 finding closed on the box where it was worst. - -**R-84 proven live on both boxes** — immediately after a restart, with the in-memory store cold, the -due-check reports real ages read from the storage instead of "no successful backup recorded yet": - -``` -demo-felhom local age=5298s felhom-pbs age=14373s both due:false -demo-hp local age=498s felhom-pbs age=2346s both due:false -``` - -Before v0.103.0 every one of those would have said "no successful backup recorded yet" and triggered -a redundant backup — on the offsite tier, a multi-hour upload. - -## The restore round-trip — PASSED (demo-hp, 2026-07-26) - -``` -source_archive : felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z -source_tier : pbs <- the v0.100.0 fix; the earlier attempt said "local" and died at 600s -pass : true -verified : boot+running -mount_parity : ok <- mp0=/var/lib/docker 50G, mp1=/mnt/sys_drive 20G, mp8/mp9 stand-ins -duration : 4m5s (restore + boot + verify + teardown) -``` - -`mount_parity` is the non-hollow half — a boot-only verify cannot see a missing data volume. The -scratch tore down **cleanly** (no 403, no leak), confirming that the teardown 403 was a **phantom** -(a short-timeout consequence, as `06-offsite-connectivity.md` already recorded) and correcting my -earlier framing of it as a standing privilege gap. - -## Tests - -`go build ./... && go vet ./... && go test ./...` — **rc=0, 29 packages**, vet run unpiped. - -Red-proofs observed and restored: old-controller compat (`"target":"local"` leaking into the -untargeted body), the per-tier wait bound, and the `onboot` override. A per-tier single-flight defect -was caught **by its own test** before it ever shipped — job ids were unique only by clock luck, so -the PBS request was handed the local job's id. - -**Process failure recorded:** I ran the suite and committed in the same command, read -`packages ok: 28` and pushed without checking `rc=1`; five of my own tests were failing (a harness -artifact, fixed in the follow-up). That is the exact exit-code trap recorded twice earlier in this -arc. - -## NOT done - -1. The **scheduled** restore-test still only ever selects the **primary** tier, so the offsite tier - is never automatically restore-tested — arguably the more important half of "is the DR tier real?". -2. The agent does not report per-tier **cadences**, so the hub must infer "PBS ⇒ weekly" from storage - type. -3. The **unattended** offsite restore-test — the scheduled path still only selects the primary tier. - (The manual/selftest round-trip is now PROVEN: `pass:true`, `verified:"boot+running"`, - `mount_parity:"ok"`, `source_tier:"pbs"`, 4m5s, clean teardown on demo-hp.) +## NOT yet live-validated +- **No `unknown` state has ever been observed on real hardware.** Both boxes report `known`; the + unknown path is unit-proven only, and reproducing it live means making PBS unreadable, which must + not be done to ep0. +- The `absent` path is likewise unobserved live — it needs a genuinely fresh tier.