diff --git a/documentation/backlog/SPEC-r85-phase4-5-2026-07-26.md b/documentation/backlog/SPEC-r85-phase4-5-2026-07-26.md new file mode 100644 index 0000000..0456c6d --- /dev/null +++ b/documentation/backlog/SPEC-r85-phase4-5-2026-07-26.md @@ -0,0 +1,191 @@ +# SPEC — R-85 Phases 4 & 5: docs, then deploy + +**Written** 2026-07-26 by Claude Code, at the operator's request, after Phases 0–3 shipped. +**Status:** ready to implement. **Phase 6 (live observation) is deliberately NOT in scope here** — +it is wall-clock-bound and gets its own acceptance note. + +--- + +## 0. State this spec was written against (verified, not assumed) + +| Artifact | Committed | Built | Deployed | Note | +|---|---|---|---|---| +| felhom-agent | `043c762` — CHANGELOG says **`v0.104.0-dev`** | — | **v0.103.0** on both boxes | version not finalised | +| felhom.eu hub | `ce4e03d` — **v0.77.0** | ❌ **no image** | **v0.76.0** (manifest pins 0.76.0) | **code shipped, signal not live** | +| felhom-controller | `3f0420f` | ✅ 0.175.0 | ✅ 0.175.0 both boxes | unchanged by R-85 | + +### Three gaps found while writing this, which the original §10 order does not cover + +1. **The hub is committed but not deployed.** `manifests/hub.yaml` pins `0.76.0` and the live pod + runs `0.76.0`. The R-85 signal exists only in git. The original Phase 5 says "deploy agent" and + never mentions the hub. +2. **R-85 has no ROADMAP row.** R-82/83/84 were collapsed; R-85 was never added. +3. **The agent version is `v0.104.0-dev`.** It must be finalised before any build — an ldflags + version that disagrees with the CHANGELOG is the reconciliation problem this project already + hit with hub 0.73.2. + +### One ordering correction + +The original §10 reads `deploy agent → drill → demo-hp → demo-felhom`. Two changes: + +- **The drill box is retired** (operator ruling 2026-07-26). Rollout is **demo-hp → demo-felhom**. +- **THE HUB GOES FIRST.** Agent v0.104.0 makes the offsite tier testable; hub v0.77.0 is what makes a + failure audible. Deploying the agent first means rotation begins with **nothing listening** — two + tiers able to fail silently instead of one, which is the exact fault R-85 exists to end. Reversing + the order costs nothing: the hub monitor is inert on boxes whose agents never rotate. + +--- + +## Phase 4 — documentation + +### 4.1 ROADMAP — add R-85 + +Take the next free `R-n` (85 is free; the ROADMAP is the source of truth). The row must record: + +- **What was broken, all three parts:** the scheduler only ever saw + `cfg.Backup.BackupTarget()`; the spec was frozen at daemon start (an immediately-invoked function, + so a storage-type or config change was stale until restart); and a failed restore-test was a + `[WARN]` line with **no event, no notification, no gauge** — true for the local tier that was + already being tested. +- **The selection ruling:** oldest-first (Option 1), operator, 2026-07-26. Record that the + per-tier last-proven timestamp is now reported **so the achieved interval can be measured before + anyone decides whether a cadence knob is needed** — measure, then configure. +- **Status:** SHIPPED for the code; **rotation NOT yet observed live across consecutive cadences.** + Do not write SHIPPED as though Phase 6 had happened. + +### 4.2 Capability map + +The R-82 row currently reads **PROVEN-LIVE** for recurring offsite backups landing and restoring — +earned by a **manual** `--selftest=restore-test`. Do not touch that claim; it is accurate. + +Add or extend a row for **unattended** restore-proof, and keep the distinction explicit: + +> the manual path is proven; the **scheduled** path is not, until Phase 6 observes both tiers across +> consecutive cadences. + +Status: **IMPLEMENTED**, with `→ R-85`. **Not PROVEN-LIVE.** The whole point of the R-82 row split +(activation vs arrival) was that a capability claim must name which half it earned; the same applies +here to manual vs unattended. + +### 4.3 Architecture docs + +- **`03-host-agent.md`** — the restore-test section must say the scheduler rotates across every + configured tier, oldest-proven first, and that a restore-test joins the one-heavy-operation gate. + If the section describes a single-target restore-test, it is now wrong. +- **`07-backup-architecture.md`** — the §2 tier table gained real cadences in R-82; add + **restore-proof** as a property of each tier. Keep the honest status header. **Do NOT mark the doc + ratified** — that remains Viktor's review of the §10 list. +- **`06-offsite-connectivity.md`** — S4.1's follow-up note claims the offsite restore-test "now runs + unattended on the agent-token path". That was true when written and **stopped being true** when + `local_backup_target` was retargeted to `local`, because the scheduler then only ever saw the local + tier. Correct it, and say what made it silently stop — the same input-changed-underneath-a-correct- + mechanism failure already recorded there for the timeout. + +### 4.4 CONTEXT + REPORTs + +- `CONTEXT.md`: one entry for R-85 — the three defects, the ruling, and **what is not yet proven**. +- `felhom-agent/REPORT.md` and `felhom.eu/REPORT.md`: overwrite per the standing rule. +- **`REPORT.md` must be updated in the same session as any deploy it describes.** A REPORT saying + "not deployed" 26 minutes before a deploy is how the record disagreed with reality earlier today. + +### 4.5 REUSE.md + +New shared helpers must be registered in the same commit: `backup.InFlight`, +`backup.RestoreTestState`, `backup.SpecBuilder` / `TierPicker`, +`BackupRunner.PickRestoreCandidateOn`, `monitor.RestoreTestChecker`. + +The traps worth naming there, because each is a decision someone could unknowingly undo: +- rotation credit is given **only on success**, or a permanently failing tier looks freshly proven; +- `InFlight` callers **defer, never cancel**; +- a tier with no archive **no-ops, it does not fail**; +- `SourceTier` comes from **the archive**, never the configured target. + +--- + +## Phase 5 — deploy + +### 5.0 Preflight — do these BEFORE any build. Any failure is a STOP. + +1. **Finalise the agent version.** `v0.104.0-dev` → `v0.104.0` in the CHANGELOG; build with matching + ldflags. +2. **Verify the state-file path is writable by the agent's non-root user.** The agent runs as + `felhom-agent`; the new rotation state lands at `/var/lib/felhom-agent/restore-test-state.json`. + **If that directory is not writable by `felhom-agent`, rotation silently loses persistence** — + `RecordSuccess` logs a warning and continues, so the failure mode is a quiet return to Scenario E + (one tier starved), not a crash. Check ownership on **both** boxes before deploying. This is the + single most likely silent failure in this phase. +3. **Confirm `restore_test_cadence_seconds` on each box.** It was disabled on demo-felhom mid-arc and + re-enabled; demo-hp has not been checked. A disabled cadence means Phase 6 observes nothing. +4. **Clean-tree gate** in every repo being built. + +### 5.1 Hub v0.77.0 — FIRST + +Standard GitOps: build+push image → bump `manifests/hub.yaml` → commit → deliberate ArgoCD sync → +verify Synced/Healthy, rollout, **pod-status image**, and the startup log. + +Acceptance: the pod reports `felhom-hub 0.77.0 starting` and the deadline-check schedules. +**No `restore_test_*` event should fire on the healthy fleet** — if one does immediately, the anchor +is wrong and that is a STOP, not a curiosity. + +### 5.2 Agent v0.104.0 — demo-hp first, then demo-felhom. **STOP between them.** + +demo-hp first because it is the smaller guest (4.25 GB offsite archive vs demo-felhom's 14.46 GB), so +its first unattended offsite restore-test costs minutes rather than hours — a cheaper place to +discover a mistake. + +Per box: back up the binary **and** `agent.json` first; install; restart; then verify: + +- `felhom-agent 0.104.0`, service active, capabilities clean +- **the rotation state file exists and is owned by `felhom-agent`** after the first successful run +- `backup tier armed` for both tiers, unchanged from v0.103.0 +- **no restore-test starts while a backup is running**, and vice versa — the gate is new and this is + its first live exercise + +### 5.3 Force a failure end-to-end (§11 deliverable 6) + +A restore-test that fails must produce an operator notification. The cheap way: point a one-shot +`--selftest=restore-test` at a **non-existent archive** on a box, and assert the hub emits +`restore_test_failed` naming the tier. + +**Use the one-shot selftest, not a doctored config** — it fails at candidate resolution without +touching the scheduled path or leaving a scratch guest behind. + +--- + +## What NOT to do + +- Do NOT deploy the agent before the hub. Rotation with nothing listening is the fault this fixes. +- Do NOT write SHIPPED/PROVEN-LIVE for anything Phase 6 has not observed. +- Do NOT mark `07-backup-architecture.md` ratified. +- Do NOT stand the drill box back up for this; it is retired. +- Do NOT skip the state-file ownership check — it is the silent failure of this phase. +- Do NOT combine a test run and a commit in one command, and **read `rc` in the directory you + intended** — both traps fired today, the second returning a bogus `rc=1` from the wrong `cd`. +- Do NOT pipe `go vet` into `head`. +- No branches, no `git add -A`, no "Co-Authored-By". + +--- + +## Acceptance + +**Phase 4** is done when a reader who was not here can tell, from the docs alone, that the +**unattended** path is unproven while the **manual** one is proven. + +**Phase 5** is done when hub v0.77.0 and agent v0.104.0 are live on both boxes, the rotation state +file is confirmed writable, and a deliberately failed restore-test has produced an operator +notification end-to-end. + +**Phase 6 is NOT in scope and will almost certainly be open at the end of this session.** It requires +observing both tiers selected across consecutive unattended cadences — at a 24 h cadence that is a +**multi-day** window. The honest close-out is an explicit entry in "NOT yet live-validated" saying +the observation window was too short, not an implication of coverage. + +### One risk to weigh before 5.2 lands on demo-felhom + +Its offsite archive is **14.46 GB**. At the ~125 MB/min restore rate measured on that box, an +unattended offsite restore-test there is a **~2-hour** operation that will now run roughly every +other day, holding the heavy-operation gate throughout and so deferring backups behind it. + +That is the design working as ruled — but it was ruled when the only measured restore was demo-hp's +4-minute one. If a ~2-hour recurring operation on that box is not wanted, the lever is the +restore-test cadence, and it is an operator decision, not a code change.