# SPEC — R-85 Phases 4 & 5: docs, then deploy

**Written** 2026-07-26 by Claude Code, at the operator's request, after Phases 0–3 shipped.
**Status:** ready to implement. **Phase 6 (live observation) is deliberately NOT in scope here** —
it is wall-clock-bound and gets its own acceptance note.

---

## 0. State this spec was written against (verified, not assumed)

| Artifact | Committed | Built | Deployed | Note |
|---|---|---|---|---|
| felhom-agent | `043c762` — CHANGELOG says **`v0.104.0-dev`** | — | **v0.103.0** on both boxes | version not finalised |
| felhom.eu hub | `ce4e03d` — **v0.77.0** | ❌ **no image** | **v0.76.0** (manifest pins 0.76.0) | **code shipped, signal not live** |
| felhom-controller | `3f0420f` | ✅ 0.175.0 | ✅ 0.175.0 both boxes | unchanged by R-85 |

### Three gaps found while writing this, which the original §10 order does not cover

1. **The hub is committed but not deployed.** `manifests/hub.yaml` pins `0.76.0` and the live pod
   runs `0.76.0`. The R-85 signal exists only in git. The original Phase 5 says "deploy agent" and
   never mentions the hub.
2. **R-85 has no ROADMAP row.** R-82/83/84 were collapsed; R-85 was never added.
3. **The agent version is `v0.104.0-dev`.** It must be finalised before any build — an ldflags
   version that disagrees with the CHANGELOG is the reconciliation problem this project already
   hit with hub 0.73.2.

### One ordering correction

The original §10 reads `deploy agent → drill → demo-hp → demo-felhom`. Two changes:

- **The drill box is retired** (operator ruling 2026-07-26). Rollout is **demo-hp → demo-felhom**.
- **THE HUB GOES FIRST.** Agent v0.104.0 makes the offsite tier testable; hub v0.77.0 is what makes a
  failure audible. Deploying the agent first means rotation begins with **nothing listening** — two
  tiers able to fail silently instead of one, which is the exact fault R-85 exists to end. Reversing
  the order costs nothing: the hub monitor is inert on boxes whose agents never rotate.

---

## Phase 4 — documentation

### 4.1 ROADMAP — add R-85

Take the next free `R-n` (85 is free; the ROADMAP is the source of truth). The row must record:

- **What was broken, all three parts:** the scheduler only ever saw
  `cfg.Backup.BackupTarget()`; the spec was frozen at daemon start (an immediately-invoked function,
  so a storage-type or config change was stale until restart); and a failed restore-test was a
  `[WARN]` line with **no event, no notification, no gauge** — true for the local tier that was
  already being tested.
- **The selection ruling:** oldest-first (Option 1), operator, 2026-07-26. Record that the
  per-tier last-proven timestamp is now reported **so the achieved interval can be measured before
  anyone decides whether a cadence knob is needed** — measure, then configure.
- **Status:** SHIPPED for the code; **rotation NOT yet observed live across consecutive cadences.**
  Do not write SHIPPED as though Phase 6 had happened.

### 4.2 Capability map

The R-82 row currently reads **PROVEN-LIVE** for recurring offsite backups landing and restoring —
earned by a **manual** `--selftest=restore-test`. Do not touch that claim; it is accurate.

Add or extend a row for **unattended** restore-proof, and keep the distinction explicit:

> the manual path is proven; the **scheduled** path is not, until Phase 6 observes both tiers across
> consecutive cadences.

Status: **IMPLEMENTED**, with `→ R-85`. **Not PROVEN-LIVE.** The whole point of the R-82 row split
(activation vs arrival) was that a capability claim must name which half it earned; the same applies
here to manual vs unattended.

### 4.3 Architecture docs

- **`03-host-agent.md`** — the restore-test section must say the scheduler rotates across every
  configured tier, oldest-proven first, and that a restore-test joins the one-heavy-operation gate.
  If the section describes a single-target restore-test, it is now wrong.
- **`07-backup-architecture.md`** — the §2 tier table gained real cadences in R-82; add
  **restore-proof** as a property of each tier. Keep the honest status header. **Do NOT mark the doc
  ratified** — that remains Viktor's review of the §10 list.
- **`06-offsite-connectivity.md`** — S4.1's follow-up note claims the offsite restore-test "now runs
  unattended on the agent-token path". That was true when written and **stopped being true** when
  `local_backup_target` was retargeted to `local`, because the scheduler then only ever saw the local
  tier. Correct it, and say what made it silently stop — the same input-changed-underneath-a-correct-
  mechanism failure already recorded there for the timeout.

### 4.4 CONTEXT + REPORTs

- `CONTEXT.md`: one entry for R-85 — the three defects, the ruling, and **what is not yet proven**.
- `felhom-agent/REPORT.md` and `felhom.eu/REPORT.md`: overwrite per the standing rule.
- **`REPORT.md` must be updated in the same session as any deploy it describes.** A REPORT saying
  "not deployed" 26 minutes before a deploy is how the record disagreed with reality earlier today.

### 4.5 REUSE.md

New shared helpers must be registered in the same commit: `backup.InFlight`,
`backup.RestoreTestState`, `backup.SpecBuilder` / `TierPicker`,
`BackupRunner.PickRestoreCandidateOn`, `monitor.RestoreTestChecker`.

The traps worth naming there, because each is a decision someone could unknowingly undo:
- rotation credit is given **only on success**, or a permanently failing tier looks freshly proven;
- `InFlight` callers **defer, never cancel**;
- a tier with no archive **no-ops, it does not fail**;
- `SourceTier` comes from **the archive**, never the configured target.

---

## Phase 5 — deploy

### 5.0 Preflight — do these BEFORE any build. Any failure is a STOP.

1. **Finalise the agent version.** `v0.104.0-dev` → `v0.104.0` in the CHANGELOG; build with matching
   ldflags.
2. **Verify the state-file path is writable by the agent's non-root user.** The agent runs as
   `felhom-agent`; the new rotation state lands at `/var/lib/felhom-agent/restore-test-state.json`.
   **If that directory is not writable by `felhom-agent`, rotation silently loses persistence** —
   `RecordSuccess` logs a warning and continues, so the failure mode is a quiet return to Scenario E
   (one tier starved), not a crash. Check ownership on **both** boxes before deploying. This is the
   single most likely silent failure in this phase.
3. **Confirm `restore_test_cadence_seconds` on each box.** It was disabled on demo-felhom mid-arc and
   re-enabled; demo-hp has not been checked. A disabled cadence means Phase 6 observes nothing.
4. **Clean-tree gate** in every repo being built.

### 5.1 Hub v0.77.0 — FIRST

Standard GitOps: build+push image → bump `manifests/hub.yaml` → commit → deliberate ArgoCD sync →
verify Synced/Healthy, rollout, **pod-status image**, and the startup log.

Acceptance: the pod reports `felhom-hub 0.77.0 starting` and the deadline-check schedules.
**No `restore_test_*` event should fire on the healthy fleet** — if one does immediately, the anchor
is wrong and that is a STOP, not a curiosity.

### 5.2 Agent v0.104.0 — demo-hp first, then demo-felhom. **STOP between them.**

demo-hp first because it is the smaller guest (4.25 GB offsite archive vs demo-felhom's 14.46 GB), so
its first unattended offsite restore-test costs minutes rather than hours — a cheaper place to
discover a mistake.

Per box: back up the binary **and** `agent.json` first; install; restart; then verify:

- `felhom-agent 0.104.0`, service active, capabilities clean
- **the rotation state file exists and is owned by `felhom-agent`** after the first successful run
- `backup tier armed` for both tiers, unchanged from v0.103.0
- **no restore-test starts while a backup is running**, and vice versa — the gate is new and this is
  its first live exercise

### 5.3 Force a failure end-to-end (§11 deliverable 6)

A restore-test that fails must produce an operator notification. The cheap way: point a one-shot
`--selftest=restore-test` at a **non-existent archive** on a box, and assert the hub emits
`restore_test_failed` naming the tier.

**Use the one-shot selftest, not a doctored config** — it fails at candidate resolution without
touching the scheduled path or leaving a scratch guest behind.

---

## What NOT to do

- Do NOT deploy the agent before the hub. Rotation with nothing listening is the fault this fixes.
- Do NOT write SHIPPED/PROVEN-LIVE for anything Phase 6 has not observed.
- Do NOT mark `07-backup-architecture.md` ratified.
- Do NOT stand the drill box back up for this; it is retired.
- Do NOT skip the state-file ownership check — it is the silent failure of this phase.
- Do NOT combine a test run and a commit in one command, and **read `rc` in the directory you
  intended** — both traps fired today, the second returning a bogus `rc=1` from the wrong `cd`.
- Do NOT pipe `go vet` into `head`.
- No branches, no `git add -A`, no "Co-Authored-By".

---

## Acceptance

**Phase 4** is done when a reader who was not here can tell, from the docs alone, that the
**unattended** path is unproven while the **manual** one is proven.

**Phase 5** is done when hub v0.77.0 and agent v0.104.0 are live on both boxes, the rotation state
file is confirmed writable, and a deliberately failed restore-test has produced an operator
notification end-to-end.

**Phase 6 is NOT in scope and will almost certainly be open at the end of this session.** It requires
observing both tiers selected across consecutive unattended cadences — at a 24 h cadence that is a
**multi-day** window. The honest close-out is an explicit entry in "NOT yet live-validated" saying
the observation window was too short, not an implication of coverage.

### One risk to weigh before 5.2 lands on demo-felhom

> **RESOLVED 2026-07-27 — the risk below was based on a bad estimate and is largely void. Kept for
> the lesson.** The actual unattended offsite restore-test on demo-felhom took **10m35s**
> (`duration_s=635.07`) for its 14.46 GB archive — ~1.4 GB/min, not the ~125 MB/min I extrapolated.
> That rate came from the *failed* attempt, which ran under contention. **An estimate extrapolated
> from a degraded measurement is not a measurement**, and I used it to raise a design concern that
> did not exist.

Its offsite archive is **14.46 GB**. At the ~125 MB/min restore rate measured on that box, an
unattended offsite restore-test there is a **~2-hour** operation that will now run roughly every
other day, holding the heavy-operation gate throughout and so deferring backups behind it.

That is the design working as ruled — but it was ruled when the only measured restore was demo-hp's
4-minute one. If a ~2-hour recurring operation on that box is not wanted, the lever is the
restore-test cadence, and it is an operator decision, not a code change.
