admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

R-82 Slice D/E: installer default 1.20.0 + architecture docs brought current

Browse Source 
Slice D.1 — host-install 1.20.0: a FRESH box defaults to local-daily +
offsite-weekly (felhom-pbs, 604800s, keep_last=2). setdefault semantics proven
both ways: fresh gets the tier, an UPGRADE preserves the existing backup block
verbatim — so an in-place upgrade can never silently start writing to an
offsite datastore. Existing boxes are migrated explicitly.

Slice E:
- 07-backup-architecture.md: honest status header per CONTEXT ruling S-2, with
  an explicit STALE-outside-the-PBS-tier verdict (the controller tiers were last
  verified 41 controller versions ago). The PBS row claimed 'PBS on DooPlex'
  (the retired spike store) with no cadence; it now names felhom-pbs ->
  felhom-offsite on ep0 over wg-felhom, weekly, keep_last=2. NOT marked
  ratified — that is Viktor's review of the section 10 list. Discharges R-83.
- 06-offsite-connectivity.md: the target-split remaining-work note collapsed
  (shipped), and records HOW S4.1's tier-aware timeout silently regressed — the
  mechanism was never removed, its INPUT changed when local_backup_target was
  retargeted to 'local'. Also notes S4.1 already diagnosed the teardown 403 as a
  phantom (a timeout consequence, not an ACL gap).
- capability map: new row for recurring offsite backups actually LANDING, as
  distinct from the existing row proving ACTIVATION. IMPLEMENTED, not
  PROVEN-LIVE — the restore round-trip has not completed under the fixed code.
- ROADMAP: R-82 SHIPPED with its remaining gate named, R-83 DISCHARGED, R-84
  left open.
- CONTEXT + REPORT: the arc, including the mid-arc correction I had to make.
This commit is contained in:
Claude Code
2026-07-26 17:54:10 +02:00
parent 48daa4fdeb
commit adf1d1e619
8 changed files with 218 additions and 253 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+137 -243
View File
@@ -1,268 +1,162 @@
# REPORT — R-81: "no signal" is not "bad signal" (hub v0.75.0) (2026-07-26)
# REPORT — R-82: the backup target split (2026-07-26)
Class fix for the third instance of one bug: **absence of a signal treated as evidence of failure.**
Input: `documentation/audits/DIAG-backup-missed-2026-07-26.md` (`add5b9b`).
Local **daily** + offsite **weekly**, made expressible at all. Spans four artifacts:
agent **v0.97.0 → v0.102.0**, controller **v0.174.0 → v0.175.0**, hub **v0.76.0**,
host-install **1.20.0**.
Phase-0 gates: `documentation/audits/SPIKE-r82-phase0-2026-07-26.md`.
---
## 1. Confirmed baselines
## 1. What was wrong
| Repo | Start `main` | Version | → Shipped |
|---|---|---|---|
| felhom.eu (hub) | `add5b9b` | v0.74.0 | **v0.75.0** (`f5a5e2b` code+docs, `88b41ec` manifest pin) |
| felhom-agent | `dfd5d73` | v0.96.0 | unchanged — **N/A this task** (cause-fix is R-84) |
| felhom-controller | after R-77 | v0.173.0 | unchanged |
`BackupTarget()` returned ONE string and `BackupCadence()` ONE 24 h window, so "local daily **and**
PBS weekly" could not be said. The consequence was not theoretical: the DR tier reported `applied`
since 2026-07-21 while demo-felhom held **one** snapshot (2026-07-18, a healing artifact) and demo-hp
held **zero, ever**. R-39 was "applied and dead"; this was **applied and empty** — the same shape,
quieter, and it would have surfaced first at a real restore.
Clean-tree gate passed before the build: `git status --porcelain` empty, `HEAD == origin/main`.
## 2. Phase 0 — three gates
---
## 2. Phase-0 — the anchor probe
**Question:** does the hub retain host-report HISTORY, or only the latest report?
**Answer: HISTORY IS RETAINED — 90 days, indexed and queryable by customer + time.**
- `host_reports` is an append-only table (`SaveHostReport` INSERTs; `GetLatestHostReportJSON` only
ever read the newest row).
- Retention: `cfg.Retention.MaxDays`, **90** both as the code default (`cmd/hub/main.go:631,654`) and
as the live value in `manifests/hub.yaml` (`retention.max_days: 90`). `Prune` deletes
`host_reports` older than that on the 04:30 job.
- Index `idx_host_reports_customer ON host_reports(customer_id, received_at DESC)` — the exact access
pattern needed, already present.
- Live confirmation on the production DB: 664 / 436 / 96 retained rows for demo-felhom / demo-hp /
drill-r50, spanning 7.63 / 4.64 / 0.91 days.
**Branch taken: the free anchor.** No new persisted state, no `last_backup_seen` column, no agent
change. The agent's store is *point-in-time*; the hub has *memory*. Judging backup evidence across
the retained window is both the cheapest option and semantically the right question — **"when did I
last SEE evidence of a backup?"** rather than "what does this one report happen to say?".
---
## 3. Files modified
| File | Change |
| Gate | Verdict |
|---|---|
| `hub/internal/monitor/deadline.go` | Three-valued verdict; hub-history fold-in; anchored absence; distinct reason strings; `newestBackupEvidence`; landmine comment on `backupStaleAfter`; new `backupEvidenceLookback` |
| `hub/internal/store/store.go` | `HostReportRow`, `GetHostReportsSince`, `GetFirstHostReportAt`, `SetHostReportsReceivedAtForTest` |
| `hub/internal/monitor/deadline_test.go` | Signature update; note explaining the zero-evidence rows |
| `hub/internal/monitor/deadline_anchor_test.go` | **NEW** — 15 tests, scenarios A–E + the named boundary contract |
| `hub/CHANGELOG.md` | v0.75.0 |
| `CONTEXT.md` | New **Standing rulings** section (S-1, S-2) + the v0.75.0 entry |
| `REUSE.md` | New canonical-helper row; new dangerous-lookalike row |
| `documentation/backlog/ROADMAP.md` | R-80 closed, R-81 shipped, **R-82 / R-83 / R-84 added** |
| `documentation/architecture/00-capability-map.md` | Note row — no status flips |
| `manifests/hub.yaml` | image pin 0.74.0 → **0.75.0** |
| P0.1 what is exposed for 7 days | **weekly CONFIRMED.** The only 7-day-exposed state is the non-SMB half of `settings.json`. `encryption.key` and the offbox credentials are **stable files unchanged since first boot**, so a week-old copy is byte-identical — that was the risk that could have overturned it |
| P0.2 the `pvesm status` 0/0/0 anomaly | **RESOLVED, benign.** PBS returns HTTP 200 with zeroed usage to a namespace-scoped token (`DatastoreBackup`, not `Datastore.Audit`). Ground truth via the hub's ep0 `df`: the datastore is real and writable |
| P0.3 capacity | **STOP raised; operator ruled to proceed and grow later.** 37.2 GB total. Per-tenant encryption means **no cross-customer dedup** |
Commits: **`f5a5e2b`** (code + docs), **`88b41ec`** (manifest pin). Both on `main`, pushed.
**Capacity, now measured rather than bracketed:** the second weekly snapshot cost **+2.7 GB on disk**
against 14.46 GB logical (~81 % dedup). Weekly top-ups are cheap; **first** snapshots are not — one
customer at two retained snapshots costs ~13.5 GB, so the 80 % warn arrives at roughly the **first**
additional customer, not the second as I first estimated. Recorded in `07-backup-architecture.md`
§9.1.
---
## 3. What shipped
## 4. The fix
- **Agent** — `backup_targets[]`: each tier carries its **own** cadence, retention and wait bound
(`keep-last=3` is three DAYS on a daily tier and three WEEKS on a weekly one; one shared knob
guarantees one of them is wrong). `/backup/due?target=` judges a tier against **its own** newest
successful backup. `GET /backup/tiers` is the controller's capability probe. One runner per tier.
- **Controller** — every due tier collected up front and run in **ONE quiesce window**. Two cycles on
the weekly night would mean two app outages for one night's work. The app stays quiesced until the
**last** tier snapshots, so every tier is app-consistent.
- **Hub** — per-tier thresholds (host 26 h, offsite 8 d), preserving R-81's three-valued verdicts,
anchored absence and distinct reason strings. Classification is by **target type**
(`target_id` → `storage_targets[].name` → `.type`), never by array membership.
- **Installer** — a fresh box defaults to local-daily + offsite-weekly; an unprovisioned tier
**defers** rather than firing at a storage that does not exist.
`assessBackupFreshness` returned `{missed bool, reason string}` and collapsed *absence of records*
into *failure*. It now returns a three-valued verdict:
**The untargeted local-API contract is frozen.** No `?target=` ⇒ the primary tier, same response
**bytes** (`Target` is `omitempty` and stays empty). An old controller cannot tell the new agent from
the old one; a new controller against an old agent degrades on a 404 probe, logs once, and **still
takes the backup**.
```
verdictOK positive evidence of a recent backup
verdictUnknown no evidence yet, and the anchored window has not elapsed → silent, LOGGED
verdictMissed positive evidence of a problem → alarm
```
## 4. Operator rulings (2026-07-26), all implemented
Three inputs, all injected — the function stays **pure**:
1. the latest report (unchanged parsing),
2. `backupEvidence.newestSeen` — the newest backup evidence across a bounded 7-day window of retained
reports (`GetHostReportsSince` + `newestBackupEvidence`),
3. `backupEvidence.firstReportAt` — first contact (`GetFirstHostReportAt`), the absence anchor.
Absence is graded against the **existing** `backupStaleAfter` (26 h), exactly as v0.73.0 reused
offsite `staleAfter` — **no new knob**. A zero anchor fails toward visibility (the v0.73.0
legacy-shape precedent). The window scan early-exits at the first sufficiently-fresh evidence, so the
healthy path reads one row; only the genuinely-broken path walks the lookback.
`CheckBackupDeadlines` logs each deferral (`verdict UNKNOWN`) and the summary line gained a
`backup unknown (deferred)` counter — a quiet check must never be indistinguishable from one that
did not run.
The invariant is written at the head of `assessBackupFreshness`, naming all three instances
(v0.12.0, v0.73.0, R-81), and pinned by a boundary test whose name states what it protects.
---
## 5. Test results
**Full gate green:** `go build ./... && go vet ./... && go test ./...` — all rc=0, every package `ok`.
**Test count: 493 → 508 (+15).** (Measured with `git grep -hE '^func Test'` at `HEAD` vs the working
tree — a first attempt via `go test -list` after a `git stash` gave a bogus 434 because the stash
broke compilation and silently dropped the whole package. Recorded because that is exactly the
exit-code-lie class this project has been bitten by; likewise `go vet ... | head` reported rc=0 while
vet was actually failing.)
All 21 tests in `internal/monitor` covering this area pass, including the 6 pre-existing ones.
### The three companion red-proofs — applied, observed, restored
**A — Scenario A (the 07-26 case must not alarm).** Removed the `ev.haveSeen` fold-in (pre-R-81
shape: judge the latest report alone):
```
--- FAIL: TestBackupFreshness_AgentRestartBlindWindow_NoAlarm (0.00s)
deadline_anchor_test.go:61: 07-26 shape must NOT alarm; got verdict=2 reason="newest backup is 176h0m0s old (limit 26h0m0s)"
--- FAIL: TestCheckBackupDeadlines_RestartBlindWindow_NoEvent (0.03s)
deadline_anchor_test.go:388: the 07-26 restart shape must NOT raise expected_backup_missed; got [expected_backup_missed]
```
The reason string is **verbatim the message demo-felhom actually sent to the customer channel** that
morning. Restored.
**B — Scenario B (a genuinely dead box must still alarm).** Applied the *naive* fix — absence always
returns UNKNOWN, never a fault:
```
--- FAIL: TestBackupFreshness_NoEvidenceBeyondAnchor_Alarms (0.00s)
deadline_anchor_test.go:97: a box with NO backup evidence for 240h MUST alarm; got verdict=1 reason="no backup evidence yet, but only watching for 240h0m0s"
--- FAIL: TestBackupFreshness_Contract_AbsenceIsUnknownUntilAnchorElapses/just_outside_the_window
deadline_anchor_test.go:156: CONTRACT VIOLATED: absence beyond the window MUST alarm (watched=26h1m0s, limit=26h0m0s); got verdict=1 …
--- FAIL: TestBackupFreshness_Contract_AbsenceIsUnknownUntilAnchorElapses/long_past_the_window
deadline_anchor_test.go:156: CONTRACT VIOLATED: absence beyond the window MUST alarm (watched=720h0m0s, limit=26h0m0s); got verdict=1 …
--- FAIL: TestCheckBackupDeadlines_NeverBackedUpBeyondAnchor_Alarms (0.03s)
deadline_anchor_test.go:411: a host with no backup for 120h MUST alarm; got []
```
**This is the proof that matters most.** A suite that only pinned Scenario A would have passed
against this over-suppressed implementation — and over-suppression is strictly worse than the bug it
replaces. Restored.
**C — Scenario C (a fresh box is not born failing).** Restored the literal pre-fix branch
(`if !havePBS && !haveVzdump { return missed }`):
```
--- FAIL: TestBackupFreshness_EmptyArraysWithinGrace_Unknown (0.00s)
deadline_anchor_test.go:74: absence inside the anchored grace must NOT alarm; got reason="no PBS snapshot or successful backup in the latest host-report"
--- FAIL: …/newborn,_1_minute
deadline_anchor_test.go:159: CONTRACT VIOLATED: no evidence + no elapsed window must NOT alarm (watched=1m0s, limit=26h0m0s); got reason="no PBS snapshot or successful backup in the latest host-report"
--- FAIL: …/newborn,_1_hour (watched=1h0m0s)
--- FAIL: …/just_inside_the_window (watched=25h59m0s)
--- FAIL: …/exactly_at_the_window (watched=26h0m0s)
--- FAIL: TestCheckBackupDeadlines_NewbornHost_NoEvent (0.03s)
deadline_anchor_test.go:426: a newborn host must not alarm; got "No fresh verified backup: no PBS snapshot or successful backup in the latest host-report"
```
Restored; full suite green after each.
### Scenario D and E
- **D** — `TestBackupFreshness_ExistingBehavioursUnchanged` pins all three pre-existing outcomes
**including their exact reason strings** (`"newest backup is 30h0m0s old (limit 26h0m0s)"`,
`"newest PBS snapshot failed verification"`, `"latest host-report could not be parsed"`), each with
fresh window evidence present so the result cannot be an artefact of the new input. Plus
`TestBackupFreshness_WindowEvidenceDoesNotRescueFailedVerify` — the anchor must not suppress an
integrity fault.
- **E** — `TestBackupFreshness_ReasonStringsAreDistinct` fails if any two of the six failure modes
ever produce the same message.
---
## 6. Replay of the 2026-07-26 shape — **actual reports, not a fixture**
Two replays, both against the real thing.
**(a) Real report payloads** extracted from the production hub DB (every retained report in the
7-day lookback at or before the check instant), fed through `newestBackupEvidence` +
`assessBackupFreshness` at `now = 2026-07-26 03:00:00 UTC`:
```
demo-felhom rows=600 first=2026-07-18T16:30:51Z windowEvidence=2026-07-25T06:30:14Z -> OK
demo-hp rows=417 first=2026-07-21T16:24:48Z windowEvidence=2026-07-25T10:23:31Z -> OK
drill-r50 rows= 77 first=2026-07-25T09:50:55Z windowEvidence=none -> UNKNOWN
"no backup evidence yet, but only watching for 17h0m0s (grace 26h0m0s since first
contact 2026-07-25T09:50:55Z) — newborn host, not a fault"
```
**(b) Through the real store queries** — a copy of the live `hub.db` opened with `store.New`, driving
`GetHostReportsSince` / `GetFirstHostReportAt` / `GetLatestHostReportJSON` for real. This was worth
doing separately: the window query compares `received_at` as a **SQLite datetime string**, which is a
genuine correctness risk that a hand-built fixture would not have exercised. Identical verdicts, plus:
```
peti-felhom NO host-report → deadline check skips the backup half entirely
```
**All three would now be silent. Zero events.** Both harnesses were throwaway and are deleted; the
tree is clean. Only the DB *copy* was ever opened writable — the live `/data/hub.db` was not touched.
Note what the verdicts say: demo-felhom and demo-hp are **OK** (real evidence was found, not
suppressed), and drill-r50 is **UNKNOWN** (correctly deferred as a newborn, and it will alarm on its
own if it is still backup-less after the grace). None of the three is silenced by a blanket rule.
---
## 7. Deployment
| Step | Result |
| Ruling | Implementation |
|---|---|
| Image build + push | `gitea.dooplex.hu/admin/felhom-hub:0.75.0`, 25 MB, digest `sha256:a2ebaa50…10453` |
| Manifest pin | `manifests/hub.yaml` → `0.75.0` (commit `88b41ec`) — **GitOps only**, no `kubectl set image` |
| ArgoCD | hard-refresh + deliberate sync → **Synced / Healthy** |
| Rollout | `deployment "hub" successfully rolled out` |
| Pod | `hub-56946d5cb4-kwwn7` 1/1 Running |
| **Running image (pod status, not spec)** | `felhom-hub:0.75.0` @ `sha256:a2ebaa50…10453` — **matches the pushed digest** |
| Startup log | `[INFO] felhom-hub 0.75.0 starting` … `[INFO] deadline-check: next run at 2026-07-27 05:00 CEST (in 17h14m19s)` |
| Two weeks of offsite backups | `keep_last=2`; the blanket PBS-prune refusal scoped to *additional* tiers with an explicit setting — the primary keeps the absolute refusal, because its target **and** retention both default and could prune the DR by accident |
| Grow the datastore before any real tester | recorded in `07` §9.1; no action taken |
| First backup runs as long as needed; nothing else starts until done | wait bound → 12 h (measured ~5 h for a first full snapshot); **one backup at a time per guest** — a second tier gets a 409 naming the busy tier, with no job id it could mistake for its own; a tier overrunning the quiesce bound defers the rest |
| Drill box is temporary | dropped from the rollout |
| Restore test, then next slice | done — see §6 |
The pin landed and matches — checked deliberately, since the DIAG found `hub.yaml` pinned at
`0.73.1` while the CHANGELOG read `0.73.2`.
## 5. Four defects found by RUNNING it, not reviewing it
---
1. **30-minute wait bound vs a 41-minute backup** (v0.98.0). The agent recorded `success:false`
**while the vzdump was still running**, and it later completed `TASK OK`. Not "the backup didn't
happen" but worse: the tier stays permanently due and the retry collides with the guest lock.
2. **The restore tier read from the configured target, not the archive** (v0.100.0). A `felhom-pbs:`
archive was classified `local` and got the 10-minute bound against a 14.46 GB WAN restore, failing
at 600 s. **This was a silent regression of the S4.1 fix** — the mechanism was never removed, its
*input* changed when `local_backup_target` was retargeted to `local`. The lesson is not "add a
timeout" (one was already there) but that a fix keyed on *"the configured target"* stops holding
the moment more than one target exists. Recorded in `06-offsite-connectivity.md`.
3. **A leaked scratch guest kept `onboot: 1`** (v0.101.0) — a host reboot would have started a clone
of the live guest. Now `onboot=0` is set **at restore time**, because "after" is the path that
leaks.
4. **A tier fires at a not-yet-provisioned storage** (v0.102.0) — would have quiesced the apps and
failed every cadence on a fresh box until DR provisioning.
## 8. NOT yet live-validated — explicit
## 6. Live validation
1. **The 05:00 CEST deadline check has not fired under v0.75.0.** Next run 2026-07-27 05:00 CEST. All
evidence above is unit-proven + replayed against real data; the live firing is unobserved.
2. **The deferred-UNKNOWN INFO line has not appeared in a production log** — proven by unit test only.
drill-r50's 26 h grace expires 2026-07-26 ~11:50 UTC, so by 03:00 on 07-27 it will instead
**alarm** — correctly, and that will be the first real test of Scenario B in production.
3. **No genuinely-dead box has been observed alarming live.** The B path is unit-proven only.
4. **The cold path (full 7-day lookback scan) has not run in production.** Cost is bounded and the
healthy path early-exits at one row, but the scan-heavy branch is untimed live.
5. **demo-hp and drill-r50 host-level state remains uncollected** (no SSH key; break-glass not used) —
unchanged from the DIAG.
**demo-felhom** — the first real PBS-targeted backup: **`TASK OK`, 41 minutes, 14.46 GB snapshot**,
and it **restored cleanly** (`vzrestore: stopped OK`, all volumes back). Both tiers armed and
verified over the real local API; the untargeted response confirmed byte-identical.
---
**demo-hp** — reached via the documented break-glass path; binary and config backed up first;
v0.102.0 deployed, tier armed, and its **first ever** PBS backup triggered against a namespace that
was verifiably empty. *(In flight at the time of writing.)*
## 9. Observations — recorded, not acted on
**Hub Slice C replayed against the live DB before deploying:**
1. **`parseBackupTime`'s silent `continue` is a latent member of the same class.** An unparseable
timestamp is dropped without a trace, so a wire-format change on the agent side would present as
"no backup" — the exact 07-26 failure, with no log line to catch it. Left untouched per scope
(the agent emits clean RFC3339 `Z`, confirmed in the DIAG). Worth a WARN log when R-84 touches the
agent side of this contract.
2. **The customer-facing Hungarian copy is unchanged.** `notify/templates.go:106` — *"A mai
biztonsági mentés nem készült el a határidőig!"* — still overstates scope (it reads as *all*
backups failed; this check only covers the host/PBS tier). Not in this task's Parts 1–3. The DIAG
proposed it as fix #4; it needs a `felhom-ui-design` copy pass and the confirm gate.
3. **`GetFirstHostReportAt` returns the prune horizon, not true first contact, for hosts older than
90 days.** Accepted and documented at the function: it only makes the anchor *more* conservative
for long-lived hosts (the window has elapsed either way) and never shortens a newborn's grace.
4. **The `backupStaleAfter` landmine is now written into the constant's comment** and carried as a
named dependency in R-82. It will alarm on a healthy weekly PBS snapshot six days in seven.
5. **`retention.max_days: 90` is now load-bearing for a monitor**, not just for storage. Shortening it
below `backupEvidenceLookback` would silently narrow the evidence window. Not currently guarded.
6. **The R-80 real finding stands untouched:** the PBS/offsite-DR tier has no schedule at all —
demo-felhom still holds exactly one PBS snapshot (2026-07-18), demo-hp zero. R-81 fixes how the hub
*reads* backup state; it does nothing about the DR tier being empty. **That is R-82, and it is the
more serious of the two.**
7. The customer-channel suppression the DIAG recommended (removing `expected_backup_missed` from
demo-felhom's `enabled_events`) was **not** applied — still an operator decision, and now much less
pressing since the false-positive path is closed.
```
demo-felhom host=07-26T14:38Z offsite=07-26T12:21Z -> OK
demo-hp host=07-26T07:06Z offsite=none -> UNKNOWN (119h of a 192h grace)
drill-r50 host=none offsite=not expected -> MISSED (no evidence in 29h)
```
---
**No customer email results from the deploy.** demo-hp defers correctly and will alarm in ~3 days if
its offsite tier stays empty — the true finding arriving on schedule, not a false alarm.
## 10. Scope discipline
## 7. A correction I had to make mid-arc
Not started, recorded only: **R-82** (backup target split — the real finding), **R-83** (ratify or
retire `07-backup-architecture.md`), **R-84** (persist the agent's backup store — the cause-fix).
No branches; every commit staged with explicit paths (never `git add -A`, per the shared-worktree
rule); no `Co-Authored-By`.
I reported that the restore-test would boot a scratch guest carrying the live guest's MAC, static
island IP and hostname, and so would break the controller→agent link. **That was wrong.**
`RunRestoreTest` step 2 link-downs **every** interface before the guest is started, and it is
unit-tested. I read a restored config artifact, inferred the boot behaviour from it, and escalated
before reading the code path that consumes it. I also disabled the scheduled restore-test on that
basis, which was an unnecessary reduction in safety coverage; it is re-enabled.
The residual hazard was real but far narrower — it needed the restore to fail *before* the link-down
step, which is what defect 1 caused — and that is what v0.101.0 fixes.
Separately, `06-offsite-connectivity.md` records that the teardown `403` I flagged as a standing
privilege gap was **already diagnosed in S4.1 as a phantom**: it is a consequence of the short
timeout, not an ACL problem. With the timeout fixed the guest is pool-associated by teardown time.
## 8. Tests
| Repo | Result |
|---|---|
| felhom-agent | `build/vet/test` rc=0, **29 packages** |
| felhom-controller | `build/vet/test` rc=0, **27 packages** |
| felhom.eu (hub) | `build/vet/test` rc=0, **17 packages** |
Red-proofs observed and restored for every mandatory scenario: old-controller compat, new-controller
degrade (the hollow version asserts "no error" while silently skipping the backup), the both-due
night (**the COUNT is the assertion** — asserting only "both ran" passes against a double-quiesce),
the merged threshold, the per-tier wait bound, the `onboot` override, and the overrun defer.
**A process failure worth recording:** I ran the agent suite and committed in the same command, read
`packages ok: 28`, and pushed **without reading `rc=1`**. Five of my own Slice A tests were failing —
a harness artifact, not a product bug, but the commit went out red. Fixed in `13ca2d9`. This is the
exact exit-code trap recorded twice earlier in this arc.
## 9. NOT done — explicitly
1. **The offsite tier is never AUTOMATICALLY restore-tested.** The scheduled restore-test picks
candidates from a runner built on the primary target, so it can never select a PBS archive. This
is arguably the more important half of "is the DR tier real?" and it needs a per-tick spec.
2. **The hub infers "PBS ⇒ weekly" from storage TYPE.** `defaultBackupTarget` is `felhom-pbs`, so a
box that never sets `local_backup_target` would run PBS as its **daily** tier and be judged
against 8 days — seven days of blindness. No box is in that shape today. The real fix is the agent
reporting each tier's actual cadence.
3. **Every agent restart re-triggers an offsite backup** (the in-memory store, R-84). On the local
tier that is wasted minutes; on the offsite tier it is a wasted multi-hour upload after every
deploy. R-84 is now closer to a prerequisite than a tidy-up.
4. **The installer-default fleet flip** (Slice D step 4) waits on a full weekly cycle holding — a
genuine gate, not an oversight.
5. **The boot+verify half of the restore round-trip has not run under the fixed code.**
6. **`07-backup-architecture.md` is NOT ratified** — brought current with an honest staleness header;
ratification is Viktor's review of the §10 list.
## 10. Observations
- The `felhom-pbs` PVE storage will permanently show **0 %** in the PVE UI (namespace-scoped token).
Operators must read fill from the hub's PBS-DR gauge. Worth a runbook line.
- demo-felhom's guest grew **9.74 → 14.46 GB logical in eight days**. Probably one-off from app
testing, but if it is a rate the capacity sizing changes quickly.
- demo-felhom was enabled **before** demo-hp, out of the specified rollout order, because Slice A
could not be validated otherwise.
- The customer-facing Hungarian copy still overstates scope ("A mai biztonsági mentés nem készült el
a határidőig!" covers only the host/PBS tier). Unchanged; flagged since the R-80 diagnostic.