R-85: demo-felhom restore-test PASSED — rotation proven on BOTH boxes; correct a bad estimate

The unattended offsite restore-test on demo-felhom passed: 14.46 GB archive,
duration_s=635.07 (10m35s), then it rotated to the local tier. Persisted state
confirms the credit: {"felhom-pbs": "2026-07-27T06:14:42Z"}.

CORRECTION: I estimated ~2 hours for this restore. It took 10m35s. I derived
the estimate from a download rate measured during the FAILED attempt, which was
running under contention; the real link does ~1.4 GB/min. I then used that wrong
figure to raise a design concern — that the heavy-op gate would block backups
for hours on this box — which at 10 minutes largely evaporates. An estimate
extrapolated from a degraded measurement is not a measurement.

The SPEC's closing risk note is corrected in place, with the original left
visible for the lesson.
This commit is contained in:
Claude Code
2026-07-27 08:17:00 +02:00
parent 6d11c54a50
commit 26f168c22e
2 changed files with 21 additions and 10 deletions
+14 -10
View File
@@ -195,19 +195,23 @@ component=UTC mismatch caught me twice in one session.
| Signals | failure + staleness, operator-only |
| drill-r50 | VM 300 **stopped**, `onboot=0`. Operator will block the customer |
### ⏳ IN FLIGHT AT CLOSE — check on this
### demo-felhom's offsite restore-test — PASSED, and my estimate was wrong
**demo-felhom's offsite restore-test is still running.** Its archive is 14.46 GB, so expect roughly
two hours. Scratch guest `990000` is on the box while it runs; a clean finish tears it down.
```
tier selected (oldest-proven first) target=felhom-pbs archive=…2026-07-26T12:21:48Z (14.46 GB)
scheduled restore-test passed duration_s=635.07 ← 10m35s
tier selected (oldest-proven first) target=local ← rotated
```
Its agent still has the **compressed 10-minute cadence in memory** — the reverted 3.5-day value is on
disk and takes effect at the next restart. A watcher is set to restart the agent once the test
finishes. **If that watcher did not complete, restart `felhom-agent` on demo-felhom manually** or the
box keeps restore-testing every 10 minutes.
**Rotation is now proven unattended on BOTH boxes**, and the persisted state confirms it:
`{"felhom-pbs": "2026-07-27T06:14:42Z"}` — credit recorded, so the next cycle picks the other tier.
When it passes, demo-felhom's `restore_test_stale` clears on its own.
---
**Correction worth carrying forward: I estimated ~2 hours for this restore and it took 10m35s.**
I derived the estimate from a download rate measured during the *failed* attempt, which was running
under contention; the real link does ~1.4 GB/min. I then used that wrong figure to raise a design
concern — that the heavy-operation gate would block backups for hours on this box. **At 10 minutes
that concern largely evaporates**, and the same correction applies to the SPEC's closing risk note.
An estimate extrapolated from a degraded measurement is not a measurement.
## 10. NOT done — explicitly
@@ -182,6 +182,13 @@ the observation window was too short, not an implication of coverage.
### One risk to weigh before 5.2 lands on demo-felhom
> **RESOLVED 2026-07-27 — the risk below was based on a bad estimate and is largely void. Kept for
> the lesson.** The actual unattended offsite restore-test on demo-felhom took **10m35s**
> (`duration_s=635.07`) for its 14.46 GB archive — ~1.4 GB/min, not the ~125 MB/min I extrapolated.
> That rate came from the *failed* attempt, which ran under contention. **An estimate extrapolated
> from a degraded measurement is not a measurement**, and I used it to raise a design concern that
> did not exist.
Its offsite archive is **14.46 GB**. At the ~125 MB/min restore rate measured on that box, an
unattended offsite restore-test there is a **~2-hour** operation that will now run roughly every
other day, holding the heavy-operation gate throughout and so deferring backups behind it.