Claude Code
26f168c22e
The unattended offsite restore-test on demo-felhom passed: 14.46 GB archive,
duration_s=635.07 (10m35s), then it rotated to the local tier. Persisted state
confirms the credit: {"felhom-pbs": "2026-07-27T06:14:42Z"}.
CORRECTION: I estimated ~2 hours for this restore. It took 10m35s. I derived
the estimate from a download rate measured during the FAILED attempt, which was
running under contention; the real link does ~1.4 GB/min. I then used that wrong
figure to raise a design concern — that the heavy-op gate would block backups
for hours on this box — which at 10 minutes largely evaporates. An estimate
extrapolated from a degraded measurement is not a measurement.
The SPEC's closing risk note is corrected in place, with the original left
visible for the lesson.
199 lines
11 KiB
Markdown
199 lines
11 KiB
Markdown
# SPEC — R-85 Phases 4 & 5: docs, then deploy
|
||
|
||
**Written** 2026-07-26 by Claude Code, at the operator's request, after Phases 0–3 shipped.
|
||
**Status:** ready to implement. **Phase 6 (live observation) is deliberately NOT in scope here** —
|
||
it is wall-clock-bound and gets its own acceptance note.
|
||
|
||
---
|
||
|
||
## 0. State this spec was written against (verified, not assumed)
|
||
|
||
| Artifact | Committed | Built | Deployed | Note |
|
||
|---|---|---|---|---|
|
||
| felhom-agent | `043c762` — CHANGELOG says **`v0.104.0-dev`** | — | **v0.103.0** on both boxes | version not finalised |
|
||
| felhom.eu hub | `ce4e03d` — **v0.77.0** | ❌ **no image** | **v0.76.0** (manifest pins 0.76.0) | **code shipped, signal not live** |
|
||
| felhom-controller | `3f0420f` | ✅ 0.175.0 | ✅ 0.175.0 both boxes | unchanged by R-85 |
|
||
|
||
### Three gaps found while writing this, which the original §10 order does not cover
|
||
|
||
1. **The hub is committed but not deployed.** `manifests/hub.yaml` pins `0.76.0` and the live pod
|
||
runs `0.76.0`. The R-85 signal exists only in git. The original Phase 5 says "deploy agent" and
|
||
never mentions the hub.
|
||
2. **R-85 has no ROADMAP row.** R-82/83/84 were collapsed; R-85 was never added.
|
||
3. **The agent version is `v0.104.0-dev`.** It must be finalised before any build — an ldflags
|
||
version that disagrees with the CHANGELOG is the reconciliation problem this project already
|
||
hit with hub 0.73.2.
|
||
|
||
### One ordering correction
|
||
|
||
The original §10 reads `deploy agent → drill → demo-hp → demo-felhom`. Two changes:
|
||
|
||
- **The drill box is retired** (operator ruling 2026-07-26). Rollout is **demo-hp → demo-felhom**.
|
||
- **THE HUB GOES FIRST.** Agent v0.104.0 makes the offsite tier testable; hub v0.77.0 is what makes a
|
||
failure audible. Deploying the agent first means rotation begins with **nothing listening** — two
|
||
tiers able to fail silently instead of one, which is the exact fault R-85 exists to end. Reversing
|
||
the order costs nothing: the hub monitor is inert on boxes whose agents never rotate.
|
||
|
||
---
|
||
|
||
## Phase 4 — documentation
|
||
|
||
### 4.1 ROADMAP — add R-85
|
||
|
||
Take the next free `R-n` (85 is free; the ROADMAP is the source of truth). The row must record:
|
||
|
||
- **What was broken, all three parts:** the scheduler only ever saw
|
||
`cfg.Backup.BackupTarget()`; the spec was frozen at daemon start (an immediately-invoked function,
|
||
so a storage-type or config change was stale until restart); and a failed restore-test was a
|
||
`[WARN]` line with **no event, no notification, no gauge** — true for the local tier that was
|
||
already being tested.
|
||
- **The selection ruling:** oldest-first (Option 1), operator, 2026-07-26. Record that the
|
||
per-tier last-proven timestamp is now reported **so the achieved interval can be measured before
|
||
anyone decides whether a cadence knob is needed** — measure, then configure.
|
||
- **Status:** SHIPPED for the code; **rotation NOT yet observed live across consecutive cadences.**
|
||
Do not write SHIPPED as though Phase 6 had happened.
|
||
|
||
### 4.2 Capability map
|
||
|
||
The R-82 row currently reads **PROVEN-LIVE** for recurring offsite backups landing and restoring —
|
||
earned by a **manual** `--selftest=restore-test`. Do not touch that claim; it is accurate.
|
||
|
||
Add or extend a row for **unattended** restore-proof, and keep the distinction explicit:
|
||
|
||
> the manual path is proven; the **scheduled** path is not, until Phase 6 observes both tiers across
|
||
> consecutive cadences.
|
||
|
||
Status: **IMPLEMENTED**, with `→ R-85`. **Not PROVEN-LIVE.** The whole point of the R-82 row split
|
||
(activation vs arrival) was that a capability claim must name which half it earned; the same applies
|
||
here to manual vs unattended.
|
||
|
||
### 4.3 Architecture docs
|
||
|
||
- **`03-host-agent.md`** — the restore-test section must say the scheduler rotates across every
|
||
configured tier, oldest-proven first, and that a restore-test joins the one-heavy-operation gate.
|
||
If the section describes a single-target restore-test, it is now wrong.
|
||
- **`07-backup-architecture.md`** — the §2 tier table gained real cadences in R-82; add
|
||
**restore-proof** as a property of each tier. Keep the honest status header. **Do NOT mark the doc
|
||
ratified** — that remains Viktor's review of the §10 list.
|
||
- **`06-offsite-connectivity.md`** — S4.1's follow-up note claims the offsite restore-test "now runs
|
||
unattended on the agent-token path". That was true when written and **stopped being true** when
|
||
`local_backup_target` was retargeted to `local`, because the scheduler then only ever saw the local
|
||
tier. Correct it, and say what made it silently stop — the same input-changed-underneath-a-correct-
|
||
mechanism failure already recorded there for the timeout.
|
||
|
||
### 4.4 CONTEXT + REPORTs
|
||
|
||
- `CONTEXT.md`: one entry for R-85 — the three defects, the ruling, and **what is not yet proven**.
|
||
- `felhom-agent/REPORT.md` and `felhom.eu/REPORT.md`: overwrite per the standing rule.
|
||
- **`REPORT.md` must be updated in the same session as any deploy it describes.** A REPORT saying
|
||
"not deployed" 26 minutes before a deploy is how the record disagreed with reality earlier today.
|
||
|
||
### 4.5 REUSE.md
|
||
|
||
New shared helpers must be registered in the same commit: `backup.InFlight`,
|
||
`backup.RestoreTestState`, `backup.SpecBuilder` / `TierPicker`,
|
||
`BackupRunner.PickRestoreCandidateOn`, `monitor.RestoreTestChecker`.
|
||
|
||
The traps worth naming there, because each is a decision someone could unknowingly undo:
|
||
- rotation credit is given **only on success**, or a permanently failing tier looks freshly proven;
|
||
- `InFlight` callers **defer, never cancel**;
|
||
- a tier with no archive **no-ops, it does not fail**;
|
||
- `SourceTier` comes from **the archive**, never the configured target.
|
||
|
||
---
|
||
|
||
## Phase 5 — deploy
|
||
|
||
### 5.0 Preflight — do these BEFORE any build. Any failure is a STOP.
|
||
|
||
1. **Finalise the agent version.** `v0.104.0-dev` → `v0.104.0` in the CHANGELOG; build with matching
|
||
ldflags.
|
||
2. **Verify the state-file path is writable by the agent's non-root user.** The agent runs as
|
||
`felhom-agent`; the new rotation state lands at `/var/lib/felhom-agent/restore-test-state.json`.
|
||
**If that directory is not writable by `felhom-agent`, rotation silently loses persistence** —
|
||
`RecordSuccess` logs a warning and continues, so the failure mode is a quiet return to Scenario E
|
||
(one tier starved), not a crash. Check ownership on **both** boxes before deploying. This is the
|
||
single most likely silent failure in this phase.
|
||
3. **Confirm `restore_test_cadence_seconds` on each box.** It was disabled on demo-felhom mid-arc and
|
||
re-enabled; demo-hp has not been checked. A disabled cadence means Phase 6 observes nothing.
|
||
4. **Clean-tree gate** in every repo being built.
|
||
|
||
### 5.1 Hub v0.77.0 — FIRST
|
||
|
||
Standard GitOps: build+push image → bump `manifests/hub.yaml` → commit → deliberate ArgoCD sync →
|
||
verify Synced/Healthy, rollout, **pod-status image**, and the startup log.
|
||
|
||
Acceptance: the pod reports `felhom-hub 0.77.0 starting` and the deadline-check schedules.
|
||
**No `restore_test_*` event should fire on the healthy fleet** — if one does immediately, the anchor
|
||
is wrong and that is a STOP, not a curiosity.
|
||
|
||
### 5.2 Agent v0.104.0 — demo-hp first, then demo-felhom. **STOP between them.**
|
||
|
||
demo-hp first because it is the smaller guest (4.25 GB offsite archive vs demo-felhom's 14.46 GB), so
|
||
its first unattended offsite restore-test costs minutes rather than hours — a cheaper place to
|
||
discover a mistake.
|
||
|
||
Per box: back up the binary **and** `agent.json` first; install; restart; then verify:
|
||
|
||
- `felhom-agent 0.104.0`, service active, capabilities clean
|
||
- **the rotation state file exists and is owned by `felhom-agent`** after the first successful run
|
||
- `backup tier armed` for both tiers, unchanged from v0.103.0
|
||
- **no restore-test starts while a backup is running**, and vice versa — the gate is new and this is
|
||
its first live exercise
|
||
|
||
### 5.3 Force a failure end-to-end (§11 deliverable 6)
|
||
|
||
A restore-test that fails must produce an operator notification. The cheap way: point a one-shot
|
||
`--selftest=restore-test` at a **non-existent archive** on a box, and assert the hub emits
|
||
`restore_test_failed` naming the tier.
|
||
|
||
**Use the one-shot selftest, not a doctored config** — it fails at candidate resolution without
|
||
touching the scheduled path or leaving a scratch guest behind.
|
||
|
||
---
|
||
|
||
## What NOT to do
|
||
|
||
- Do NOT deploy the agent before the hub. Rotation with nothing listening is the fault this fixes.
|
||
- Do NOT write SHIPPED/PROVEN-LIVE for anything Phase 6 has not observed.
|
||
- Do NOT mark `07-backup-architecture.md` ratified.
|
||
- Do NOT stand the drill box back up for this; it is retired.
|
||
- Do NOT skip the state-file ownership check — it is the silent failure of this phase.
|
||
- Do NOT combine a test run and a commit in one command, and **read `rc` in the directory you
|
||
intended** — both traps fired today, the second returning a bogus `rc=1` from the wrong `cd`.
|
||
- Do NOT pipe `go vet` into `head`.
|
||
- No branches, no `git add -A`, no "Co-Authored-By".
|
||
|
||
---
|
||
|
||
## Acceptance
|
||
|
||
**Phase 4** is done when a reader who was not here can tell, from the docs alone, that the
|
||
**unattended** path is unproven while the **manual** one is proven.
|
||
|
||
**Phase 5** is done when hub v0.77.0 and agent v0.104.0 are live on both boxes, the rotation state
|
||
file is confirmed writable, and a deliberately failed restore-test has produced an operator
|
||
notification end-to-end.
|
||
|
||
**Phase 6 is NOT in scope and will almost certainly be open at the end of this session.** It requires
|
||
observing both tiers selected across consecutive unattended cadences — at a 24 h cadence that is a
|
||
**multi-day** window. The honest close-out is an explicit entry in "NOT yet live-validated" saying
|
||
the observation window was too short, not an implication of coverage.
|
||
|
||
### One risk to weigh before 5.2 lands on demo-felhom
|
||
|
||
> **RESOLVED 2026-07-27 — the risk below was based on a bad estimate and is largely void. Kept for
|
||
> the lesson.** The actual unattended offsite restore-test on demo-felhom took **10m35s**
|
||
> (`duration_s=635.07`) for its 14.46 GB archive — ~1.4 GB/min, not the ~125 MB/min I extrapolated.
|
||
> That rate came from the *failed* attempt, which ran under contention. **An estimate extrapolated
|
||
> from a degraded measurement is not a measurement**, and I used it to raise a design concern that
|
||
> did not exist.
|
||
|
||
Its offsite archive is **14.46 GB**. At the ~125 MB/min restore rate measured on that box, an
|
||
unattended offsite restore-test there is a **~2-hour** operation that will now run roughly every
|
||
other day, holding the heavy-operation gate throughout and so deferring backups behind it.
|
||
|
||
That is the design working as ruled — but it was ruled when the only measured restore was demo-hp's
|
||
4-minute one. If a ~2-hour recurring operation on that box is not wanted, the lever is the
|
||
restore-test cadence, and it is an operator decision, not a code change.
|