R-85: demo-felhom restore-test PASSED — rotation proven on BOTH boxes; correct a bad estimate
The unattended offsite restore-test on demo-felhom passed: 14.46 GB archive,
duration_s=635.07 (10m35s), then it rotated to the local tier. Persisted state
confirms the credit: {"felhom-pbs": "2026-07-27T06:14:42Z"}.
CORRECTION: I estimated ~2 hours for this restore. It took 10m35s. I derived
the estimate from a download rate measured during the FAILED attempt, which was
running under contention; the real link does ~1.4 GB/min. I then used that wrong
figure to raise a design concern — that the heavy-op gate would block backups
for hours on this box — which at 10 minutes largely evaporates. An estimate
extrapolated from a degraded measurement is not a measurement.
The SPEC's closing risk note is corrected in place, with the original left
visible for the lesson.
This commit is contained in:
@@ -195,19 +195,23 @@ component=UTC mismatch caught me twice in one session.
|
||||
| Signals | failure + staleness, operator-only |
|
||||
| drill-r50 | VM 300 **stopped**, `onboot=0`. Operator will block the customer |
|
||||
|
||||
### ⏳ IN FLIGHT AT CLOSE — check on this
|
||||
### demo-felhom's offsite restore-test — PASSED, and my estimate was wrong
|
||||
|
||||
**demo-felhom's offsite restore-test is still running.** Its archive is 14.46 GB, so expect roughly
|
||||
two hours. Scratch guest `990000` is on the box while it runs; a clean finish tears it down.
|
||||
```
|
||||
tier selected (oldest-proven first) target=felhom-pbs archive=…2026-07-26T12:21:48Z (14.46 GB)
|
||||
scheduled restore-test passed duration_s=635.07 ← 10m35s
|
||||
tier selected (oldest-proven first) target=local ← rotated
|
||||
```
|
||||
|
||||
Its agent still has the **compressed 10-minute cadence in memory** — the reverted 3.5-day value is on
|
||||
disk and takes effect at the next restart. A watcher is set to restart the agent once the test
|
||||
finishes. **If that watcher did not complete, restart `felhom-agent` on demo-felhom manually** or the
|
||||
box keeps restore-testing every 10 minutes.
|
||||
**Rotation is now proven unattended on BOTH boxes**, and the persisted state confirms it:
|
||||
`{"felhom-pbs": "2026-07-27T06:14:42Z"}` — credit recorded, so the next cycle picks the other tier.
|
||||
|
||||
When it passes, demo-felhom's `restore_test_stale` clears on its own.
|
||||
|
||||
---
|
||||
**Correction worth carrying forward: I estimated ~2 hours for this restore and it took 10m35s.**
|
||||
I derived the estimate from a download rate measured during the *failed* attempt, which was running
|
||||
under contention; the real link does ~1.4 GB/min. I then used that wrong figure to raise a design
|
||||
concern — that the heavy-operation gate would block backups for hours on this box. **At 10 minutes
|
||||
that concern largely evaporates**, and the same correction applies to the SPEC's closing risk note.
|
||||
An estimate extrapolated from a degraded measurement is not a measurement.
|
||||
|
||||
## 10. NOT done — explicitly
|
||||
|
||||
|
||||
@@ -182,6 +182,13 @@ the observation window was too short, not an implication of coverage.
|
||||
|
||||
### One risk to weigh before 5.2 lands on demo-felhom
|
||||
|
||||
> **RESOLVED 2026-07-27 — the risk below was based on a bad estimate and is largely void. Kept for
|
||||
> the lesson.** The actual unattended offsite restore-test on demo-felhom took **10m35s**
|
||||
> (`duration_s=635.07`) for its 14.46 GB archive — ~1.4 GB/min, not the ~125 MB/min I extrapolated.
|
||||
> That rate came from the *failed* attempt, which ran under contention. **An estimate extrapolated
|
||||
> from a degraded measurement is not a measurement**, and I used it to raise a design concern that
|
||||
> did not exist.
|
||||
|
||||
Its offsite archive is **14.46 GB**. At the ~125 MB/min restore rate measured on that box, an
|
||||
unattended offsite restore-test there is a **~2-hour** operation that will now run roughly every
|
||||
other day, holding the heavy-operation gate throughout and so deferring backups behind it.
|
||||
|
||||
Reference in New Issue
Block a user