docs(backlog): SPEC for R-85 Phases 4 & 5 (docs, then deploy)
Written against verified state, not assumption. Records three gaps the original Phase 5 ordering does not cover: 1. hub v0.77.0 is COMMITTED BUT NOT DEPLOYED — manifest pins 0.76.0 and the pod runs 0.76.0, so the R-85 signal exists only in git. The original Phase 5 only mentions the agent. 2. R-85 has no ROADMAP row. 3. The agent CHANGELOG says v0.104.0-dev; an ldflags version disagreeing with the CHANGELOG is the reconciliation problem hub 0.73.2 already caused. One ordering correction: THE HUB GOES FIRST. Agent v0.104.0 makes the offsite tier testable; hub v0.77.0 makes a failure audible. Agent-first means rotation begins with nothing listening — two tiers able to fail silently instead of one, which is the fault R-85 exists to end. Also drops the retired drill box, so the rollout is demo-hp -> demo-felhom. Names the phase's most likely SILENT failure: the new rotation state lands at /var/lib/felhom-agent/restore-test-state.json and the agent is non-root. If that is not writable, RecordSuccess warns and continues — a quiet return to one tier being starved, not a crash. Flags for operator judgement: demo-felhom's 14.46 GB offsite archive makes its unattended restore-test a ~2h operation every other day, holding the heavy-op gate throughout. Ruled when the only measured restore was demo-hp's 4 minutes.
This commit is contained in:
Claude Code
@@ -0,0 +1,191 @@
|
||||
# SPEC — R-85 Phases 4 & 5: docs, then deploy
|
||||
|
||||
**Written** 2026-07-26 by Claude Code, at the operator's request, after Phases 0–3 shipped.
|
||||
**Status:** ready to implement. **Phase 6 (live observation) is deliberately NOT in scope here** —
|
||||
it is wall-clock-bound and gets its own acceptance note.
|
||||
|
||||
---
|
||||
|
||||
## 0. State this spec was written against (verified, not assumed)
|
||||
|
||||
| Artifact | Committed | Built | Deployed | Note |
|
||||
|---|---|---|---|---|
|
||||
| felhom-agent | `043c762` — CHANGELOG says **`v0.104.0-dev`** | — | **v0.103.0** on both boxes | version not finalised |
|
||||
| felhom.eu hub | `ce4e03d` — **v0.77.0** | ❌ **no image** | **v0.76.0** (manifest pins 0.76.0) | **code shipped, signal not live** |
|
||||
| felhom-controller | `3f0420f` | ✅ 0.175.0 | ✅ 0.175.0 both boxes | unchanged by R-85 |
|
||||
|
||||
### Three gaps found while writing this, which the original §10 order does not cover
|
||||
|
||||
1. **The hub is committed but not deployed.** `manifests/hub.yaml` pins `0.76.0` and the live pod
|
||||
runs `0.76.0`. The R-85 signal exists only in git. The original Phase 5 says "deploy agent" and
|
||||
never mentions the hub.
|
||||
2. **R-85 has no ROADMAP row.** R-82/83/84 were collapsed; R-85 was never added.
|
||||
3. **The agent version is `v0.104.0-dev`.** It must be finalised before any build — an ldflags
|
||||
version that disagrees with the CHANGELOG is the reconciliation problem this project already
|
||||
hit with hub 0.73.2.
|
||||
|
||||
### One ordering correction
|
||||
|
||||
The original §10 reads `deploy agent → drill → demo-hp → demo-felhom`. Two changes:
|
||||
|
||||
- **The drill box is retired** (operator ruling 2026-07-26). Rollout is **demo-hp → demo-felhom**.
|
||||
- **THE HUB GOES FIRST.** Agent v0.104.0 makes the offsite tier testable; hub v0.77.0 is what makes a
|
||||
failure audible. Deploying the agent first means rotation begins with **nothing listening** — two
|
||||
tiers able to fail silently instead of one, which is the exact fault R-85 exists to end. Reversing
|
||||
the order costs nothing: the hub monitor is inert on boxes whose agents never rotate.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — documentation
|
||||
|
||||
### 4.1 ROADMAP — add R-85
|
||||
|
||||
Take the next free `R-n` (85 is free; the ROADMAP is the source of truth). The row must record:
|
||||
|
||||
- **What was broken, all three parts:** the scheduler only ever saw
|
||||
`cfg.Backup.BackupTarget()`; the spec was frozen at daemon start (an immediately-invoked function,
|
||||
so a storage-type or config change was stale until restart); and a failed restore-test was a
|
||||
`[WARN]` line with **no event, no notification, no gauge** — true for the local tier that was
|
||||
already being tested.
|
||||
- **The selection ruling:** oldest-first (Option 1), operator, 2026-07-26. Record that the
|
||||
per-tier last-proven timestamp is now reported **so the achieved interval can be measured before
|
||||
anyone decides whether a cadence knob is needed** — measure, then configure.
|
||||
- **Status:** SHIPPED for the code; **rotation NOT yet observed live across consecutive cadences.**
|
||||
Do not write SHIPPED as though Phase 6 had happened.
|
||||
|
||||
### 4.2 Capability map
|
||||
|
||||
The R-82 row currently reads **PROVEN-LIVE** for recurring offsite backups landing and restoring —
|
||||
earned by a **manual** `--selftest=restore-test`. Do not touch that claim; it is accurate.
|
||||
|
||||
Add or extend a row for **unattended** restore-proof, and keep the distinction explicit:
|
||||
|
||||
> the manual path is proven; the **scheduled** path is not, until Phase 6 observes both tiers across
|
||||
> consecutive cadences.
|
||||
|
||||
Status: **IMPLEMENTED**, with `→ R-85`. **Not PROVEN-LIVE.** The whole point of the R-82 row split
|
||||
(activation vs arrival) was that a capability claim must name which half it earned; the same applies
|
||||
here to manual vs unattended.
|
||||
|
||||
### 4.3 Architecture docs
|
||||
|
||||
- **`03-host-agent.md`** — the restore-test section must say the scheduler rotates across every
|
||||
configured tier, oldest-proven first, and that a restore-test joins the one-heavy-operation gate.
|
||||
If the section describes a single-target restore-test, it is now wrong.
|
||||
- **`07-backup-architecture.md`** — the §2 tier table gained real cadences in R-82; add
|
||||
**restore-proof** as a property of each tier. Keep the honest status header. **Do NOT mark the doc
|
||||
ratified** — that remains Viktor's review of the §10 list.
|
||||
- **`06-offsite-connectivity.md`** — S4.1's follow-up note claims the offsite restore-test "now runs
|
||||
unattended on the agent-token path". That was true when written and **stopped being true** when
|
||||
`local_backup_target` was retargeted to `local`, because the scheduler then only ever saw the local
|
||||
tier. Correct it, and say what made it silently stop — the same input-changed-underneath-a-correct-
|
||||
mechanism failure already recorded there for the timeout.
|
||||
|
||||
### 4.4 CONTEXT + REPORTs
|
||||
|
||||
- `CONTEXT.md`: one entry for R-85 — the three defects, the ruling, and **what is not yet proven**.
|
||||
- `felhom-agent/REPORT.md` and `felhom.eu/REPORT.md`: overwrite per the standing rule.
|
||||
- **`REPORT.md` must be updated in the same session as any deploy it describes.** A REPORT saying
|
||||
"not deployed" 26 minutes before a deploy is how the record disagreed with reality earlier today.
|
||||
|
||||
### 4.5 REUSE.md
|
||||
|
||||
New shared helpers must be registered in the same commit: `backup.InFlight`,
|
||||
`backup.RestoreTestState`, `backup.SpecBuilder` / `TierPicker`,
|
||||
`BackupRunner.PickRestoreCandidateOn`, `monitor.RestoreTestChecker`.
|
||||
|
||||
The traps worth naming there, because each is a decision someone could unknowingly undo:
|
||||
- rotation credit is given **only on success**, or a permanently failing tier looks freshly proven;
|
||||
- `InFlight` callers **defer, never cancel**;
|
||||
- a tier with no archive **no-ops, it does not fail**;
|
||||
- `SourceTier` comes from **the archive**, never the configured target.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — deploy
|
||||
|
||||
### 5.0 Preflight — do these BEFORE any build. Any failure is a STOP.
|
||||
|
||||
1. **Finalise the agent version.** `v0.104.0-dev` → `v0.104.0` in the CHANGELOG; build with matching
|
||||
ldflags.
|
||||
2. **Verify the state-file path is writable by the agent's non-root user.** The agent runs as
|
||||
`felhom-agent`; the new rotation state lands at `/var/lib/felhom-agent/restore-test-state.json`.
|
||||
**If that directory is not writable by `felhom-agent`, rotation silently loses persistence** —
|
||||
`RecordSuccess` logs a warning and continues, so the failure mode is a quiet return to Scenario E
|
||||
(one tier starved), not a crash. Check ownership on **both** boxes before deploying. This is the
|
||||
single most likely silent failure in this phase.
|
||||
3. **Confirm `restore_test_cadence_seconds` on each box.** It was disabled on demo-felhom mid-arc and
|
||||
re-enabled; demo-hp has not been checked. A disabled cadence means Phase 6 observes nothing.
|
||||
4. **Clean-tree gate** in every repo being built.
|
||||
|
||||
### 5.1 Hub v0.77.0 — FIRST
|
||||
|
||||
Standard GitOps: build+push image → bump `manifests/hub.yaml` → commit → deliberate ArgoCD sync →
|
||||
verify Synced/Healthy, rollout, **pod-status image**, and the startup log.
|
||||
|
||||
Acceptance: the pod reports `felhom-hub 0.77.0 starting` and the deadline-check schedules.
|
||||
**No `restore_test_*` event should fire on the healthy fleet** — if one does immediately, the anchor
|
||||
is wrong and that is a STOP, not a curiosity.
|
||||
|
||||
### 5.2 Agent v0.104.0 — demo-hp first, then demo-felhom. **STOP between them.**
|
||||
|
||||
demo-hp first because it is the smaller guest (4.25 GB offsite archive vs demo-felhom's 14.46 GB), so
|
||||
its first unattended offsite restore-test costs minutes rather than hours — a cheaper place to
|
||||
discover a mistake.
|
||||
|
||||
Per box: back up the binary **and** `agent.json` first; install; restart; then verify:
|
||||
|
||||
- `felhom-agent 0.104.0`, service active, capabilities clean
|
||||
- **the rotation state file exists and is owned by `felhom-agent`** after the first successful run
|
||||
- `backup tier armed` for both tiers, unchanged from v0.103.0
|
||||
- **no restore-test starts while a backup is running**, and vice versa — the gate is new and this is
|
||||
its first live exercise
|
||||
|
||||
### 5.3 Force a failure end-to-end (§11 deliverable 6)
|
||||
|
||||
A restore-test that fails must produce an operator notification. The cheap way: point a one-shot
|
||||
`--selftest=restore-test` at a **non-existent archive** on a box, and assert the hub emits
|
||||
`restore_test_failed` naming the tier.
|
||||
|
||||
**Use the one-shot selftest, not a doctored config** — it fails at candidate resolution without
|
||||
touching the scheduled path or leaving a scratch guest behind.
|
||||
|
||||
---
|
||||
|
||||
## What NOT to do
|
||||
|
||||
- Do NOT deploy the agent before the hub. Rotation with nothing listening is the fault this fixes.
|
||||
- Do NOT write SHIPPED/PROVEN-LIVE for anything Phase 6 has not observed.
|
||||
- Do NOT mark `07-backup-architecture.md` ratified.
|
||||
- Do NOT stand the drill box back up for this; it is retired.
|
||||
- Do NOT skip the state-file ownership check — it is the silent failure of this phase.
|
||||
- Do NOT combine a test run and a commit in one command, and **read `rc` in the directory you
|
||||
intended** — both traps fired today, the second returning a bogus `rc=1` from the wrong `cd`.
|
||||
- Do NOT pipe `go vet` into `head`.
|
||||
- No branches, no `git add -A`, no "Co-Authored-By".
|
||||
|
||||
---
|
||||
|
||||
## Acceptance
|
||||
|
||||
**Phase 4** is done when a reader who was not here can tell, from the docs alone, that the
|
||||
**unattended** path is unproven while the **manual** one is proven.
|
||||
|
||||
**Phase 5** is done when hub v0.77.0 and agent v0.104.0 are live on both boxes, the rotation state
|
||||
file is confirmed writable, and a deliberately failed restore-test has produced an operator
|
||||
notification end-to-end.
|
||||
|
||||
**Phase 6 is NOT in scope and will almost certainly be open at the end of this session.** It requires
|
||||
observing both tiers selected across consecutive unattended cadences — at a 24 h cadence that is a
|
||||
**multi-day** window. The honest close-out is an explicit entry in "NOT yet live-validated" saying
|
||||
the observation window was too short, not an implication of coverage.
|
||||
|
||||
### One risk to weigh before 5.2 lands on demo-felhom
|
||||
|
||||
Its offsite archive is **14.46 GB**. At the ~125 MB/min restore rate measured on that box, an
|
||||
unattended offsite restore-test there is a **~2-hour** operation that will now run roughly every
|
||||
other day, holding the heavy-operation gate throughout and so deferring backups behind it.
|
||||
|
||||
That is the design working as ruled — but it was ruled when the only measured restore was demo-hp's
|
||||
4-minute one. If a ~2-hour recurring operation on that box is not wanted, the lever is the
|
||||
restore-test cadence, and it is an operator decision, not a code change.
|
||||
Reference in New Issue
Block a user