hub v0.13.0: DR recipe — assemble + store + view the secret-free reconstruction recipe

DR recipe slice (hub half), grounded in SPIKE-dr-recipe-2026-06-16. The hub
receives two additive dr_recipe halves on the existing report paths (agent
storage/guest/PBS on host-report; controller customer/apps on the controller
report), stores them PLAINTEXT in a DEDICATED dr_recipe table keyed by customer
(each half preserves the other), and AssembleDRRecipe stitches them into one
operator-readable recipe (ignore-unknown + version-skew tolerant).

View: a DR-recipe panel on the customer page + GET /customers/{id}/dr-recipe.json
download (operator-auth, no secrets to redact). Plaintext-at-rest is correct —
the recipe is the clean inverse of the retired infra-backup.

Tests: store round-trip (each half preserves the other), assemble-matches-golden,
ignore-unknown + version skew, partial halves, no-secrets sweep. Manifest tag
bumped to v0.13.0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-16 18:49:45 +02:00
parent 228dac4c06
commit 5f5e3c54a1
13 changed files with 597 additions and 69 deletions
+45 -64
View File
@@ -4,80 +4,61 @@
---
# Hub v0.12.0 — retire Infra Backup, purge plaintext secrets, fix backup-deadline email
## REPORT — hub v0.13.0: DR recipe (assemble + store + view)
**Date:** 2026-06-16
**Scope:** Phase-1 of `documentation/audits/SPIKE-infra-backup-2026-06-15.md` (hub side).
**Deployed:** `felhom-hub:v0.12.0` live on k3s (ArgoCD `felhom` app Synced); commit `0635640`.
**Companion repo:** `felhom-controller` v0.69.0 (own REPORT there).
**TASK — DR recipe slice (hub half).** Receive the two secret-free recipe halves, assemble them into one
customer recipe, store PLAINTEXT in a dedicated table, and expose an operator view/download. Grounded in
`documentation/audits/SPIKE-dr-recipe-2026-06-16.md`. Pairs with felhom-agent v0.38.0 (storage/guest/PBS
half) + felhom-controller v0.73.0 (customer/apps half, the boundary-enforcing emitter).
## What shipped
### Store (`hub/internal/store/dr_recipe.go`)
1. **Backup-deadline check repointed to PBS freshness** (`internal/monitor/deadline.go`).
The backup half no longer queries for a `backup_completed` event (nothing emits it post-slice-8C,
so it fired daily for every healthy customer). It now reads the customer's **latest agent
host-report** (`store.GetLatestHostReportJSON`) and raises `expected_backup_missed` only on
positive evidence: no PBS snapshot / successful vzdump at all, newest backup older than **26h**, or
the newest PBS snapshot's `verify_state == "failed"`. A fresh-but-unverified snapshot is **not** a
failure (PBS verifies on its own cadence — alarming on it would recreate the false alarm). The
**db-dump half is unchanged**. A customer with **no host-report** gets no backup alarm here
(liveness is the host-staleness checker's job).
- New `dr_recipe` table (migrate()) — DEDICATED, separate from the opaque `host_escrow` and from the
dropped `infra_backup*` tables. Keyed by `customer_id`; columns `recipe_version, host_id,
host_half_json, app_half_json, updated_at`.
- `SaveDRRecipeHostHalf` / `SaveDRRecipeAppHalf` — each upserts its half and PRESERVES the other
(last-write-wins per half; a re-report of one half never clobbers the other).
- `AssembleDRRecipe(rec)` — stitches the two halves into `AssembledRecipe{recipe_version, customer,
guests, pbs, drives, pve_storage, apps}`. Sub-sections pass through as `json.RawMessage` (verbatim);
**ignore-unknown** at the top level + version-skew tolerant (`recipe_version` = max) for forward-compat
across the three repos. Pure → unit-tested.
2. **Infra Backup feature removed** (`api/handler.go`, `store/store.go`, `web/configs.go`,
`templates/customer_unified.html`, `templates/customer.html`): the `POST/GET /api/v1/infra-backup[…]`
endpoints + handlers, the store methods/types (`SaveInfraBackup`/`GetInfraBackup`/`GetInfraBackupByID`/
`GetInfraBackupMeta`/`ListInfraBackupVersions`/`pruneInfraBackups`, `InfraBackupMeta`/`InfraBackupVersion`),
and the operator "Infra Backup" panel. `GET /api/v1/recovery/{id}` now returns **config_yaml only**.
The customer-page config-drift badge (diffed against the stored controller.yaml) is hidden; the live
"Show Diff" path is unaffected.
### Ingest (`hub/internal/api/handler.go`)
3. **Plaintext secret purge** (`store/store.go migrate()`): `DROP infra_backup_versions; DROP
infra_backups; VACUUM; wal_checkpoint(TRUNCATE)` — gated on table existence so normal restarts skip
it. VACUUM physically reclaims the freed pages so the plaintext keys/tokens are not merely delinked.
- `handleHostReport` persists the `dr_recipe` host-half (keyed by the host's customer).
- `handleReport` persists the `dr_recipe` app-half (keyed by `customer_id`), mirroring the
`app_telemetry` pattern. Both backward-compatible (old agents/controllers omit the field) and never
fatal to the heartbeat.
## Gate (STEP 1a) — PASSED before any change
### View (`hub/internal/web/dr_recipe.go` + customer page)
The demo's latest host-report (id 699, agent v0.36.7) carried **5 PBS snapshots, all `verify=ok`,
newest `2026-06-15T18:41:29Z` (13.9 h old, < 26 h)** plus a matching successful vzdump. The repoint's
data source is present and fresh, so the repoint alone clears the email.
- A DR-recipe panel on the customer detail page (which half landed + last-updated) with a
**Download recipe (JSON)** link.
- `GET /customers/{id}/dr-recipe.json` serves the assembled recipe (operator dashboard-auth, pretty
JSON, `Content-Disposition` attachment). No decrypt, nothing to redact.
## Verification (live, non-hollow)
### Boundary
- **Build + tests:** `go build ./... && go test ./...` green. `deadline_test.go` covers fresh+verified→
quiet (the **companion**), stale→alarm, failed-verify→alarm, no-host-report→quiet, db-dump half
preserved, and a pure `assessBackupFreshness` table. Red-proof: with the old event-based half
temporarily restored, the companion + no-report + db-dump-preserved tests **fail**, while the
pure-helper test still passes — the behavioral tests are sensitive to the logic.
- **Deploy:** hub v0.12.0 rolled out; startup log: `Retired infra-backup: dropped 2 table(s) and
VACUUMed to reclaim plaintext pages`.
- **Tables gone:** `.tables` on the live `/data/hub.db` shows no `infra_backup*`. DB shrank
**65.8 MB → 52.2 MB**; no `-wal` sidecar lingers.
- **Secret purge proof (grep on the live post-VACUUM DB file):** the infra-backup-exclusive markers
are gone — `encryption_key_b64` 17→**0**, `controller_config_b64` 11→**0**, and the actual
AES-key value from spike record 109 present→**0** (physically reclaimed, not just delinked). The
residual `cf_api_token`/`cf_tunnel_token` hits are in `customer_configs.config_json` (the hub's
legitimate config store), not the infra-backup blobs (which held CF tokens base64-encoded inside the
now-zero `controller_config_b64`).
- **Endpoints retired:** `POST /api/v1/infra-backup` and `GET …/versions` return **404**; control
`POST /api/v1/report` still returns **401** (routed, unauth) — confirming the 404s are route removal,
not a blanket failure.
- **Demo email fix:** the next 03:00 deadline run cannot be observed within this session, but the live
demo host-report (fresh verified PBS, 13.9 h) exercises the not-missed path, and demo emits
`db_dump_completed` daily (never missed) — so `expected_backup_missed` will no longer fire.
The recipe is PLAINTEXT-at-rest because it carries NO secrets — only identifiers/intents/sizes/
coordinates. The PBS key stays in escrow, the access token in identity-escrow, the restic password in
escrow. This is the clean inverse of the retired infra-backup (which shipped `restic_password`/
`cf_api_token` and was a zero-knowledge violation). The leak-preventing allowlist is enforced at the
controller emitter; the hub adds a defense-in-depth no-secrets sweep.
## Flagged for the operator (out of scope here)
### Tests
- **Rotate the exposed credentials** that were in the dropped blobs (Cloudflare API + tunnel tokens
for `demo-felhom.eu` first; hub/session secrets if shared). They remain valid until rotated; the
purge removes the at-rest copy but not their validity.
- **Separate historical leak:** the legacy `reports` table holds thousands of rows with a plaintext
`restic_password` value from old controller versions. The **live** controller no longer sends it
(removed in controller v0.69.0), but the historical rows persist — a distinct purge/rotation
decision, deliberately not done here.
- **`peti-felhom` is a defunct customer marked `active`** (no host-report or controller report since
Feb 2026). Its daily `expected_backup_missed` stops with this change (no host-report → no backup
alarm), but it should be marked inactive to silence all residual noise.
`TestDRRecipe_StoreRoundTrip`, `TestAssembleDRRecipe_MatchesGolden` (assembled shape pinned in
`testdata/dr-recipe.golden.json`), `TestAssembleDRRecipe_IgnoreUnknownAndVersionSkew`,
`TestAssembleDRRecipe_PartialHalves`, `TestAssembleDRRecipe_NoSecrets`. `go build`/`go vet`/`go test ./...`
green.
## Out of scope (untouched)
### Cross-repo golden discipline
Credential rotation; the Komga healthcheck; the secret-free DR "recipe" (the later DR slice).
The recipe wire spans three repos. The agent's `host-report.golden.json` `dr_recipe` section and the
hub's host-half test literal must stay key-consistent; the controller's emitter and the hub's app-half
test literal likewise. On any wire change, manually checksum-diff the golden across the three repos
(there is no shared types module yet).
### Deploy
Built + pushed `felhom-hub:0.13.0`; manifest tag bumped + ArgoCD `felhom` app synced.