# SPEC — R-85 Phases 4 & 5: docs, then deploy **Written** 2026-07-26 by Claude Code, at the operator's request, after Phases 0–3 shipped. **Status:** ready to implement. **Phase 6 (live observation) is deliberately NOT in scope here** — it is wall-clock-bound and gets its own acceptance note. --- ## 0. State this spec was written against (verified, not assumed) | Artifact | Committed | Built | Deployed | Note | |---|---|---|---|---| | felhom-agent | `043c762` — CHANGELOG says **`v0.104.0-dev`** | — | **v0.103.0** on both boxes | version not finalised | | felhom.eu hub | `ce4e03d` — **v0.77.0** | ❌ **no image** | **v0.76.0** (manifest pins 0.76.0) | **code shipped, signal not live** | | felhom-controller | `3f0420f` | ✅ 0.175.0 | ✅ 0.175.0 both boxes | unchanged by R-85 | ### Three gaps found while writing this, which the original §10 order does not cover 1. **The hub is committed but not deployed.** `manifests/hub.yaml` pins `0.76.0` and the live pod runs `0.76.0`. The R-85 signal exists only in git. The original Phase 5 says "deploy agent" and never mentions the hub. 2. **R-85 has no ROADMAP row.** R-82/83/84 were collapsed; R-85 was never added. 3. **The agent version is `v0.104.0-dev`.** It must be finalised before any build — an ldflags version that disagrees with the CHANGELOG is the reconciliation problem this project already hit with hub 0.73.2. ### One ordering correction The original §10 reads `deploy agent → drill → demo-hp → demo-felhom`. Two changes: - **The drill box is retired** (operator ruling 2026-07-26). Rollout is **demo-hp → demo-felhom**. - **THE HUB GOES FIRST.** Agent v0.104.0 makes the offsite tier testable; hub v0.77.0 is what makes a failure audible. Deploying the agent first means rotation begins with **nothing listening** — two tiers able to fail silently instead of one, which is the exact fault R-85 exists to end. Reversing the order costs nothing: the hub monitor is inert on boxes whose agents never rotate. --- ## Phase 4 — documentation ### 4.1 ROADMAP — add R-85 Take the next free `R-n` (85 is free; the ROADMAP is the source of truth). The row must record: - **What was broken, all three parts:** the scheduler only ever saw `cfg.Backup.BackupTarget()`; the spec was frozen at daemon start (an immediately-invoked function, so a storage-type or config change was stale until restart); and a failed restore-test was a `[WARN]` line with **no event, no notification, no gauge** — true for the local tier that was already being tested. - **The selection ruling:** oldest-first (Option 1), operator, 2026-07-26. Record that the per-tier last-proven timestamp is now reported **so the achieved interval can be measured before anyone decides whether a cadence knob is needed** — measure, then configure. - **Status:** SHIPPED for the code; **rotation NOT yet observed live across consecutive cadences.** Do not write SHIPPED as though Phase 6 had happened. ### 4.2 Capability map The R-82 row currently reads **PROVEN-LIVE** for recurring offsite backups landing and restoring — earned by a **manual** `--selftest=restore-test`. Do not touch that claim; it is accurate. Add or extend a row for **unattended** restore-proof, and keep the distinction explicit: > the manual path is proven; the **scheduled** path is not, until Phase 6 observes both tiers across > consecutive cadences. Status: **IMPLEMENTED**, with `→ R-85`. **Not PROVEN-LIVE.** The whole point of the R-82 row split (activation vs arrival) was that a capability claim must name which half it earned; the same applies here to manual vs unattended. ### 4.3 Architecture docs - **`03-host-agent.md`** — the restore-test section must say the scheduler rotates across every configured tier, oldest-proven first, and that a restore-test joins the one-heavy-operation gate. If the section describes a single-target restore-test, it is now wrong. - **`07-backup-architecture.md`** — the §2 tier table gained real cadences in R-82; add **restore-proof** as a property of each tier. Keep the honest status header. **Do NOT mark the doc ratified** — that remains Viktor's review of the §10 list. - **`06-offsite-connectivity.md`** — S4.1's follow-up note claims the offsite restore-test "now runs unattended on the agent-token path". That was true when written and **stopped being true** when `local_backup_target` was retargeted to `local`, because the scheduler then only ever saw the local tier. Correct it, and say what made it silently stop — the same input-changed-underneath-a-correct- mechanism failure already recorded there for the timeout. ### 4.4 CONTEXT + REPORTs - `CONTEXT.md`: one entry for R-85 — the three defects, the ruling, and **what is not yet proven**. - `felhom-agent/REPORT.md` and `felhom.eu/REPORT.md`: overwrite per the standing rule. - **`REPORT.md` must be updated in the same session as any deploy it describes.** A REPORT saying "not deployed" 26 minutes before a deploy is how the record disagreed with reality earlier today. ### 4.5 REUSE.md New shared helpers must be registered in the same commit: `backup.InFlight`, `backup.RestoreTestState`, `backup.SpecBuilder` / `TierPicker`, `BackupRunner.PickRestoreCandidateOn`, `monitor.RestoreTestChecker`. The traps worth naming there, because each is a decision someone could unknowingly undo: - rotation credit is given **only on success**, or a permanently failing tier looks freshly proven; - `InFlight` callers **defer, never cancel**; - a tier with no archive **no-ops, it does not fail**; - `SourceTier` comes from **the archive**, never the configured target. --- ## Phase 5 — deploy ### 5.0 Preflight — do these BEFORE any build. Any failure is a STOP. 1. **Finalise the agent version.** `v0.104.0-dev` → `v0.104.0` in the CHANGELOG; build with matching ldflags. 2. **Verify the state-file path is writable by the agent's non-root user.** The agent runs as `felhom-agent`; the new rotation state lands at `/var/lib/felhom-agent/restore-test-state.json`. **If that directory is not writable by `felhom-agent`, rotation silently loses persistence** — `RecordSuccess` logs a warning and continues, so the failure mode is a quiet return to Scenario E (one tier starved), not a crash. Check ownership on **both** boxes before deploying. This is the single most likely silent failure in this phase. 3. **Confirm `restore_test_cadence_seconds` on each box.** It was disabled on demo-felhom mid-arc and re-enabled; demo-hp has not been checked. A disabled cadence means Phase 6 observes nothing. 4. **Clean-tree gate** in every repo being built. ### 5.1 Hub v0.77.0 — FIRST Standard GitOps: build+push image → bump `manifests/hub.yaml` → commit → deliberate ArgoCD sync → verify Synced/Healthy, rollout, **pod-status image**, and the startup log. Acceptance: the pod reports `felhom-hub 0.77.0 starting` and the deadline-check schedules. **No `restore_test_*` event should fire on the healthy fleet** — if one does immediately, the anchor is wrong and that is a STOP, not a curiosity. ### 5.2 Agent v0.104.0 — demo-hp first, then demo-felhom. **STOP between them.** demo-hp first because it is the smaller guest (4.25 GB offsite archive vs demo-felhom's 14.46 GB), so its first unattended offsite restore-test costs minutes rather than hours — a cheaper place to discover a mistake. Per box: back up the binary **and** `agent.json` first; install; restart; then verify: - `felhom-agent 0.104.0`, service active, capabilities clean - **the rotation state file exists and is owned by `felhom-agent`** after the first successful run - `backup tier armed` for both tiers, unchanged from v0.103.0 - **no restore-test starts while a backup is running**, and vice versa — the gate is new and this is its first live exercise ### 5.3 Force a failure end-to-end (§11 deliverable 6) A restore-test that fails must produce an operator notification. The cheap way: point a one-shot `--selftest=restore-test` at a **non-existent archive** on a box, and assert the hub emits `restore_test_failed` naming the tier. **Use the one-shot selftest, not a doctored config** — it fails at candidate resolution without touching the scheduled path or leaving a scratch guest behind. --- ## What NOT to do - Do NOT deploy the agent before the hub. Rotation with nothing listening is the fault this fixes. - Do NOT write SHIPPED/PROVEN-LIVE for anything Phase 6 has not observed. - Do NOT mark `07-backup-architecture.md` ratified. - Do NOT stand the drill box back up for this; it is retired. - Do NOT skip the state-file ownership check — it is the silent failure of this phase. - Do NOT combine a test run and a commit in one command, and **read `rc` in the directory you intended** — both traps fired today, the second returning a bogus `rc=1` from the wrong `cd`. - Do NOT pipe `go vet` into `head`. - No branches, no `git add -A`, no "Co-Authored-By". --- ## Acceptance **Phase 4** is done when a reader who was not here can tell, from the docs alone, that the **unattended** path is unproven while the **manual** one is proven. **Phase 5** is done when hub v0.77.0 and agent v0.104.0 are live on both boxes, the rotation state file is confirmed writable, and a deliberately failed restore-test has produced an operator notification end-to-end. **Phase 6 is NOT in scope and will almost certainly be open at the end of this session.** It requires observing both tiers selected across consecutive unattended cadences — at a 24 h cadence that is a **multi-day** window. The honest close-out is an explicit entry in "NOT yet live-validated" saying the observation window was too short, not an implication of coverage. ### One risk to weigh before 5.2 lands on demo-felhom > **RESOLVED 2026-07-27 — the risk below was based on a bad estimate and is largely void. Kept for > the lesson.** The actual unattended offsite restore-test on demo-felhom took **10m35s** > (`duration_s=635.07`) for its 14.46 GB archive — ~1.4 GB/min, not the ~125 MB/min I extrapolated. > That rate came from the *failed* attempt, which ran under contention. **An estimate extrapolated > from a degraded measurement is not a measurement**, and I used it to raise a design concern that > did not exist. Its offsite archive is **14.46 GB**. At the ~125 MB/min restore rate measured on that box, an unattended offsite restore-test there is a **~2-hour** operation that will now run roughly every other day, holding the heavy-operation gate throughout and so deferring backups behind it. That is the design working as ruled — but it was ruled when the only measured restore was demo-hp's 4-minute one. If a ~2-hour recurring operation on that box is not wanted, the lever is the restore-test cadence, and it is an operator decision, not a code change.