felhom.eu/documentation/backlog/SPEC-r85-phase4-5-2026-07-26.md

SPEC — R-85 Phases 4 & 5: docs, then deploy

Written 2026-07-26 by Claude Code, at the operator's request, after Phases 0–3 shipped. Status: ready to implement. Phase 6 (live observation) is deliberately NOT in scope here — it is wall-clock-bound and gets its own acceptance note.

---

0. State this spec was written against (verified, not assumed)

Artifact	Committed	Built	Deployed	Note
felhom-agent	043c762 — CHANGELOG says v0.104.0-dev	—	v0.103.0 on both boxes	version not finalised
felhom.eu hub	ce4e03d — v0.77.0	❌ no image	v0.76.0 (manifest pins 0.76.0)	code shipped, signal not live
felhom-controller	3f0420f	✅ 0.175.0	✅ 0.175.0 both boxes	unchanged by R-85

Three gaps found while writing this, which the original §10 order does not cover

1. The hub is committed but not deployed. manifests/hub.yaml pins 0.76.0 and the live pod runs 0.76.0. The R-85 signal exists only in git. The original Phase 5 says "deploy agent" and never mentions the hub.
2. R-85 has no ROADMAP row. R-82/83/84 were collapsed; R-85 was never added.
3. The agent version is v0.104.0-dev. It must be finalised before any build — an ldflags version that disagrees with the CHANGELOG is the reconciliation problem this project already hit with hub 0.73.2.

One ordering correction

The original §10 reads deploy agent → drill → demo-hp → demo-felhom. Two changes:

• The drill box is retired (operator ruling 2026-07-26). Rollout is demo-hp → demo-felhom.
• THE HUB GOES FIRST. Agent v0.104.0 makes the offsite tier testable; hub v0.77.0 is what makes a failure audible. Deploying the agent first means rotation begins with nothing listening — two tiers able to fail silently instead of one, which is the exact fault R-85 exists to end. Reversing the order costs nothing: the hub monitor is inert on boxes whose agents never rotate.

---

Phase 4 — documentation

4.1 ROADMAP — add R-85

Take the next free R-n (85 is free; the ROADMAP is the source of truth). The row must record:

• What was broken, all three parts: the scheduler only ever saw cfg.Backup.BackupTarget(); the spec was frozen at daemon start (an immediately-invoked function, so a storage-type or config change was stale until restart); and a failed restore-test was a [WARN] line with no event, no notification, no gauge — true for the local tier that was already being tested.
• The selection ruling: oldest-first (Option 1), operator, 2026-07-26. Record that the per-tier last-proven timestamp is now reported so the achieved interval can be measured before anyone decides whether a cadence knob is needed — measure, then configure.
• Status: SHIPPED for the code; rotation NOT yet observed live across consecutive cadences. Do not write SHIPPED as though Phase 6 had happened.

4.2 Capability map

The R-82 row currently reads PROVEN-LIVE for recurring offsite backups landing and restoring — earned by a manual --selftest=restore-test. Do not touch that claim; it is accurate.

Add or extend a row for unattended restore-proof, and keep the distinction explicit:

the manual path is proven; the scheduled path is not, until Phase 6 observes both tiers across consecutive cadences.

Status: IMPLEMENTED, with → R-85. Not PROVEN-LIVE. The whole point of the R-82 row split (activation vs arrival) was that a capability claim must name which half it earned; the same applies here to manual vs unattended.

4.3 Architecture docs

• 03-host-agent.md — the restore-test section must say the scheduler rotates across every configured tier, oldest-proven first, and that a restore-test joins the one-heavy-operation gate. If the section describes a single-target restore-test, it is now wrong.
• 07-backup-architecture.md — the §2 tier table gained real cadences in R-82; add restore-proof as a property of each tier. Keep the honest status header. Do NOT mark the doc ratified — that remains Viktor's review of the §10 list.
• 06-offsite-connectivity.md — S4.1's follow-up note claims the offsite restore-test "now runs unattended on the agent-token path". That was true when written and stopped being true when local_backup_target was retargeted to local, because the scheduler then only ever saw the local tier. Correct it, and say what made it silently stop — the same input-changed-underneath-a-correct- mechanism failure already recorded there for the timeout.

4.4 CONTEXT + REPORTs

• CONTEXT.md: one entry for R-85 — the three defects, the ruling, and what is not yet proven.
• felhom-agent/REPORT.md and felhom.eu/REPORT.md: overwrite per the standing rule.
• REPORT.md must be updated in the same session as any deploy it describes. A REPORT saying "not deployed" 26 minutes before a deploy is how the record disagreed with reality earlier today.

4.5 REUSE.md

New shared helpers must be registered in the same commit: backup.InFlight, backup.RestoreTestState, backup.SpecBuilder / TierPicker, BackupRunner.PickRestoreCandidateOn, monitor.RestoreTestChecker.

The traps worth naming there, because each is a decision someone could unknowingly undo:

• rotation credit is given only on success, or a permanently failing tier looks freshly proven;
• InFlight callers defer, never cancel;
• a tier with no archive no-ops, it does not fail;
• SourceTier comes from the archive, never the configured target.

---

Phase 5 — deploy

5.0 Preflight — do these BEFORE any build. Any failure is a STOP.

1. Finalise the agent version. v0.104.0-dev → v0.104.0 in the CHANGELOG; build with matching ldflags.
2. Verify the state-file path is writable by the agent's non-root user. The agent runs as felhom-agent; the new rotation state lands at /var/lib/felhom-agent/restore-test-state.json. If that directory is not writable by felhom-agent, rotation silently loses persistence — RecordSuccess logs a warning and continues, so the failure mode is a quiet return to Scenario E (one tier starved), not a crash. Check ownership on both boxes before deploying. This is the single most likely silent failure in this phase.
3. Confirm restore_test_cadence_seconds on each box. It was disabled on demo-felhom mid-arc and re-enabled; demo-hp has not been checked. A disabled cadence means Phase 6 observes nothing.
4. Clean-tree gate in every repo being built.

5.1 Hub v0.77.0 — FIRST

Standard GitOps: build+push image → bump manifests/hub.yaml → commit → deliberate ArgoCD sync → verify Synced/Healthy, rollout, pod-status image, and the startup log.

Acceptance: the pod reports felhom-hub 0.77.0 starting and the deadline-check schedules. No restore_test_* event should fire on the healthy fleet — if one does immediately, the anchor is wrong and that is a STOP, not a curiosity.

5.2 Agent v0.104.0 — demo-hp first, then demo-felhom. STOP between them.

demo-hp first because it is the smaller guest (4.25 GB offsite archive vs demo-felhom's 14.46 GB), so its first unattended offsite restore-test costs minutes rather than hours — a cheaper place to discover a mistake.

Per box: back up the binary and agent.json first; install; restart; then verify:

• felhom-agent 0.104.0, service active, capabilities clean
• the rotation state file exists and is owned by felhom-agent after the first successful run
• backup tier armed for both tiers, unchanged from v0.103.0
• no restore-test starts while a backup is running, and vice versa — the gate is new and this is its first live exercise

5.3 Force a failure end-to-end (§11 deliverable 6)

A restore-test that fails must produce an operator notification. The cheap way: point a one-shot --selftest=restore-test at a non-existent archive on a box, and assert the hub emits restore_test_failed naming the tier.

Use the one-shot selftest, not a doctored config — it fails at candidate resolution without touching the scheduled path or leaving a scratch guest behind.

---

What NOT to do

• Do NOT deploy the agent before the hub. Rotation with nothing listening is the fault this fixes.
• Do NOT write SHIPPED/PROVEN-LIVE for anything Phase 6 has not observed.
• Do NOT mark 07-backup-architecture.md ratified.
• Do NOT stand the drill box back up for this; it is retired.
• Do NOT skip the state-file ownership check — it is the silent failure of this phase.
• Do NOT combine a test run and a commit in one command, and read rc in the directory you intended — both traps fired today, the second returning a bogus rc=1 from the wrong cd.
• Do NOT pipe go vet into head.
• No branches, no git add -A, no "Co-Authored-By".

---

Acceptance

Phase 4 is done when a reader who was not here can tell, from the docs alone, that the unattended path is unproven while the manual one is proven.

Phase 5 is done when hub v0.77.0 and agent v0.104.0 are live on both boxes, the rotation state file is confirmed writable, and a deliberately failed restore-test has produced an operator notification end-to-end.

Phase 6 is NOT in scope and will almost certainly be open at the end of this session. It requires observing both tiers selected across consecutive unattended cadences — at a 24 h cadence that is a multi-day window. The honest close-out is an explicit entry in "NOT yet live-validated" saying the observation window was too short, not an implication of coverage.

One risk to weigh before 5.2 lands on demo-felhom

RESOLVED 2026-07-27 — the risk below was based on a bad estimate and is largely void. Kept for the lesson. The actual unattended offsite restore-test on demo-felhom took 10m35s (duration_s=635.07) for its 14.46 GB archive — ~1.4 GB/min, not the ~125 MB/min I extrapolated. That rate came from the failed attempt, which ran under contention. An estimate extrapolated from a degraded measurement is not a measurement, and I used it to raise a design concern that did not exist.

Its offsite archive is 14.46 GB. At the ~125 MB/min restore rate measured on that box, an unattended offsite restore-test there is a ~2-hour operation that will now run roughly every other day, holding the heavy-operation gate throughout and so deferring backups behind it.

That is the design working as ruled — but it was ruled when the only measured restore was demo-hp's 4-minute one. If a ~2-hour recurring operation on that box is not wanted, the lever is the restore-test cadence, and it is an operator decision, not a code change.