hub v0.13.0: DR recipe — assemble + store + view the secret-free reconstruction recipe
DR recipe slice (hub half), grounded in SPIKE-dr-recipe-2026-06-16. The hub
receives two additive dr_recipe halves on the existing report paths (agent
storage/guest/PBS on host-report; controller customer/apps on the controller
report), stores them PLAINTEXT in a DEDICATED dr_recipe table keyed by customer
(each half preserves the other), and AssembleDRRecipe stitches them into one
operator-readable recipe (ignore-unknown + version-skew tolerant).
View: a DR-recipe panel on the customer page + GET /customers/{id}/dr-recipe.json
download (operator-auth, no secrets to redact). Plaintext-at-rest is correct —
the recipe is the clean inverse of the retired infra-backup.
Tests: store round-trip (each half preserves the other), assemble-matches-golden,
ignore-unknown + version skew, partial halves, no-secrets sweep. Manifest tag
bumped to v0.13.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -4,80 +4,61 @@
|
||||
|
||||
---
|
||||
|
||||
# Hub v0.12.0 — retire Infra Backup, purge plaintext secrets, fix backup-deadline email
|
||||
## REPORT — hub v0.13.0: DR recipe (assemble + store + view)
|
||||
|
||||
**Date:** 2026-06-16
|
||||
**Scope:** Phase-1 of `documentation/audits/SPIKE-infra-backup-2026-06-15.md` (hub side).
|
||||
**Deployed:** `felhom-hub:v0.12.0` live on k3s (ArgoCD `felhom` app Synced); commit `0635640`.
|
||||
**Companion repo:** `felhom-controller` v0.69.0 (own REPORT there).
|
||||
**TASK — DR recipe slice (hub half).** Receive the two secret-free recipe halves, assemble them into one
|
||||
customer recipe, store PLAINTEXT in a dedicated table, and expose an operator view/download. Grounded in
|
||||
`documentation/audits/SPIKE-dr-recipe-2026-06-16.md`. Pairs with felhom-agent v0.38.0 (storage/guest/PBS
|
||||
half) + felhom-controller v0.73.0 (customer/apps half, the boundary-enforcing emitter).
|
||||
|
||||
## What shipped
|
||||
### Store (`hub/internal/store/dr_recipe.go`)
|
||||
|
||||
1. **Backup-deadline check repointed to PBS freshness** (`internal/monitor/deadline.go`).
|
||||
The backup half no longer queries for a `backup_completed` event (nothing emits it post-slice-8C,
|
||||
so it fired daily for every healthy customer). It now reads the customer's **latest agent
|
||||
host-report** (`store.GetLatestHostReportJSON`) and raises `expected_backup_missed` only on
|
||||
positive evidence: no PBS snapshot / successful vzdump at all, newest backup older than **26h**, or
|
||||
the newest PBS snapshot's `verify_state == "failed"`. A fresh-but-unverified snapshot is **not** a
|
||||
failure (PBS verifies on its own cadence — alarming on it would recreate the false alarm). The
|
||||
**db-dump half is unchanged**. A customer with **no host-report** gets no backup alarm here
|
||||
(liveness is the host-staleness checker's job).
|
||||
- New `dr_recipe` table (migrate()) — DEDICATED, separate from the opaque `host_escrow` and from the
|
||||
dropped `infra_backup*` tables. Keyed by `customer_id`; columns `recipe_version, host_id,
|
||||
host_half_json, app_half_json, updated_at`.
|
||||
- `SaveDRRecipeHostHalf` / `SaveDRRecipeAppHalf` — each upserts its half and PRESERVES the other
|
||||
(last-write-wins per half; a re-report of one half never clobbers the other).
|
||||
- `AssembleDRRecipe(rec)` — stitches the two halves into `AssembledRecipe{recipe_version, customer,
|
||||
guests, pbs, drives, pve_storage, apps}`. Sub-sections pass through as `json.RawMessage` (verbatim);
|
||||
**ignore-unknown** at the top level + version-skew tolerant (`recipe_version` = max) for forward-compat
|
||||
across the three repos. Pure → unit-tested.
|
||||
|
||||
2. **Infra Backup feature removed** (`api/handler.go`, `store/store.go`, `web/configs.go`,
|
||||
`templates/customer_unified.html`, `templates/customer.html`): the `POST/GET /api/v1/infra-backup[…]`
|
||||
endpoints + handlers, the store methods/types (`SaveInfraBackup`/`GetInfraBackup`/`GetInfraBackupByID`/
|
||||
`GetInfraBackupMeta`/`ListInfraBackupVersions`/`pruneInfraBackups`, `InfraBackupMeta`/`InfraBackupVersion`),
|
||||
and the operator "Infra Backup" panel. `GET /api/v1/recovery/{id}` now returns **config_yaml only**.
|
||||
The customer-page config-drift badge (diffed against the stored controller.yaml) is hidden; the live
|
||||
"Show Diff" path is unaffected.
|
||||
### Ingest (`hub/internal/api/handler.go`)
|
||||
|
||||
3. **Plaintext secret purge** (`store/store.go migrate()`): `DROP infra_backup_versions; DROP
|
||||
infra_backups; VACUUM; wal_checkpoint(TRUNCATE)` — gated on table existence so normal restarts skip
|
||||
it. VACUUM physically reclaims the freed pages so the plaintext keys/tokens are not merely delinked.
|
||||
- `handleHostReport` persists the `dr_recipe` host-half (keyed by the host's customer).
|
||||
- `handleReport` persists the `dr_recipe` app-half (keyed by `customer_id`), mirroring the
|
||||
`app_telemetry` pattern. Both backward-compatible (old agents/controllers omit the field) and never
|
||||
fatal to the heartbeat.
|
||||
|
||||
## Gate (STEP 1a) — PASSED before any change
|
||||
### View (`hub/internal/web/dr_recipe.go` + customer page)
|
||||
|
||||
The demo's latest host-report (id 699, agent v0.36.7) carried **5 PBS snapshots, all `verify=ok`,
|
||||
newest `2026-06-15T18:41:29Z` (13.9 h old, < 26 h)** plus a matching successful vzdump. The repoint's
|
||||
data source is present and fresh, so the repoint alone clears the email.
|
||||
- A DR-recipe panel on the customer detail page (which half landed + last-updated) with a
|
||||
**Download recipe (JSON)** link.
|
||||
- `GET /customers/{id}/dr-recipe.json` serves the assembled recipe (operator dashboard-auth, pretty
|
||||
JSON, `Content-Disposition` attachment). No decrypt, nothing to redact.
|
||||
|
||||
## Verification (live, non-hollow)
|
||||
### Boundary
|
||||
|
||||
- **Build + tests:** `go build ./... && go test ./...` green. `deadline_test.go` covers fresh+verified→
|
||||
quiet (the **companion**), stale→alarm, failed-verify→alarm, no-host-report→quiet, db-dump half
|
||||
preserved, and a pure `assessBackupFreshness` table. Red-proof: with the old event-based half
|
||||
temporarily restored, the companion + no-report + db-dump-preserved tests **fail**, while the
|
||||
pure-helper test still passes — the behavioral tests are sensitive to the logic.
|
||||
- **Deploy:** hub v0.12.0 rolled out; startup log: `Retired infra-backup: dropped 2 table(s) and
|
||||
VACUUMed to reclaim plaintext pages`.
|
||||
- **Tables gone:** `.tables` on the live `/data/hub.db` shows no `infra_backup*`. DB shrank
|
||||
**65.8 MB → 52.2 MB**; no `-wal` sidecar lingers.
|
||||
- **Secret purge proof (grep on the live post-VACUUM DB file):** the infra-backup-exclusive markers
|
||||
are gone — `encryption_key_b64` 17→**0**, `controller_config_b64` 11→**0**, and the actual
|
||||
AES-key value from spike record 109 present→**0** (physically reclaimed, not just delinked). The
|
||||
residual `cf_api_token`/`cf_tunnel_token` hits are in `customer_configs.config_json` (the hub's
|
||||
legitimate config store), not the infra-backup blobs (which held CF tokens base64-encoded inside the
|
||||
now-zero `controller_config_b64`).
|
||||
- **Endpoints retired:** `POST /api/v1/infra-backup` and `GET …/versions` return **404**; control
|
||||
`POST /api/v1/report` still returns **401** (routed, unauth) — confirming the 404s are route removal,
|
||||
not a blanket failure.
|
||||
- **Demo email fix:** the next 03:00 deadline run cannot be observed within this session, but the live
|
||||
demo host-report (fresh verified PBS, 13.9 h) exercises the not-missed path, and demo emits
|
||||
`db_dump_completed` daily (never missed) — so `expected_backup_missed` will no longer fire.
|
||||
The recipe is PLAINTEXT-at-rest because it carries NO secrets — only identifiers/intents/sizes/
|
||||
coordinates. The PBS key stays in escrow, the access token in identity-escrow, the restic password in
|
||||
escrow. This is the clean inverse of the retired infra-backup (which shipped `restic_password`/
|
||||
`cf_api_token` and was a zero-knowledge violation). The leak-preventing allowlist is enforced at the
|
||||
controller emitter; the hub adds a defense-in-depth no-secrets sweep.
|
||||
|
||||
## Flagged for the operator (out of scope here)
|
||||
### Tests
|
||||
|
||||
- **Rotate the exposed credentials** that were in the dropped blobs (Cloudflare API + tunnel tokens
|
||||
for `demo-felhom.eu` first; hub/session secrets if shared). They remain valid until rotated; the
|
||||
purge removes the at-rest copy but not their validity.
|
||||
- **Separate historical leak:** the legacy `reports` table holds thousands of rows with a plaintext
|
||||
`restic_password` value from old controller versions. The **live** controller no longer sends it
|
||||
(removed in controller v0.69.0), but the historical rows persist — a distinct purge/rotation
|
||||
decision, deliberately not done here.
|
||||
- **`peti-felhom` is a defunct customer marked `active`** (no host-report or controller report since
|
||||
Feb 2026). Its daily `expected_backup_missed` stops with this change (no host-report → no backup
|
||||
alarm), but it should be marked inactive to silence all residual noise.
|
||||
`TestDRRecipe_StoreRoundTrip`, `TestAssembleDRRecipe_MatchesGolden` (assembled shape pinned in
|
||||
`testdata/dr-recipe.golden.json`), `TestAssembleDRRecipe_IgnoreUnknownAndVersionSkew`,
|
||||
`TestAssembleDRRecipe_PartialHalves`, `TestAssembleDRRecipe_NoSecrets`. `go build`/`go vet`/`go test ./...`
|
||||
green.
|
||||
|
||||
## Out of scope (untouched)
|
||||
### Cross-repo golden discipline
|
||||
|
||||
Credential rotation; the Komga healthcheck; the secret-free DR "recipe" (the later DR slice).
|
||||
The recipe wire spans three repos. The agent's `host-report.golden.json` `dr_recipe` section and the
|
||||
hub's host-half test literal must stay key-consistent; the controller's emitter and the hub's app-half
|
||||
test literal likewise. On any wire change, manually checksum-diff the golden across the three repos
|
||||
(there is no shared types module yet).
|
||||
|
||||
### Deploy
|
||||
|
||||
Built + pushed `felhom-hub:0.13.0`; manifest tag bumped + ArgoCD `felhom` app synced.
|
||||
|
||||
Reference in New Issue
Block a user