hub v0.13.0: DR recipe — assemble + store + view the secret-free reconstruction recipe

DR recipe slice (hub half), grounded in SPIKE-dr-recipe-2026-06-16. The hub
receives two additive dr_recipe halves on the existing report paths (agent
storage/guest/PBS on host-report; controller customer/apps on the controller
report), stores them PLAINTEXT in a DEDICATED dr_recipe table keyed by customer
(each half preserves the other), and AssembleDRRecipe stitches them into one
operator-readable recipe (ignore-unknown + version-skew tolerant).

View: a DR-recipe panel on the customer page + GET /customers/{id}/dr-recipe.json
download (operator-auth, no secrets to redact). Plaintext-at-rest is correct —
the recipe is the clean inverse of the retired infra-backup.

Tests: store round-trip (each half preserves the other), assemble-matches-golden,
ignore-unknown + version skew, partial halves, no-secrets sweep. Manifest tag
bumped to v0.13.0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-16 18:49:45 +02:00
parent 228dac4c06
commit 5f5e3c54a1
13 changed files with 597 additions and 69 deletions
+45 -64
View File
@@ -4,80 +4,61 @@
---
# Hub v0.12.0 — retire Infra Backup, purge plaintext secrets, fix backup-deadline email
## REPORT — hub v0.13.0: DR recipe (assemble + store + view)
**Date:** 2026-06-16
**Scope:** Phase-1 of `documentation/audits/SPIKE-infra-backup-2026-06-15.md` (hub side).
**Deployed:** `felhom-hub:v0.12.0` live on k3s (ArgoCD `felhom` app Synced); commit `0635640`.
**Companion repo:** `felhom-controller` v0.69.0 (own REPORT there).
**TASK — DR recipe slice (hub half).** Receive the two secret-free recipe halves, assemble them into one
customer recipe, store PLAINTEXT in a dedicated table, and expose an operator view/download. Grounded in
`documentation/audits/SPIKE-dr-recipe-2026-06-16.md`. Pairs with felhom-agent v0.38.0 (storage/guest/PBS
half) + felhom-controller v0.73.0 (customer/apps half, the boundary-enforcing emitter).
## What shipped
### Store (`hub/internal/store/dr_recipe.go`)
1. **Backup-deadline check repointed to PBS freshness** (`internal/monitor/deadline.go`).
The backup half no longer queries for a `backup_completed` event (nothing emits it post-slice-8C,
so it fired daily for every healthy customer). It now reads the customer's **latest agent
host-report** (`store.GetLatestHostReportJSON`) and raises `expected_backup_missed` only on
positive evidence: no PBS snapshot / successful vzdump at all, newest backup older than **26h**, or
the newest PBS snapshot's `verify_state == "failed"`. A fresh-but-unverified snapshot is **not** a
failure (PBS verifies on its own cadence — alarming on it would recreate the false alarm). The
**db-dump half is unchanged**. A customer with **no host-report** gets no backup alarm here
(liveness is the host-staleness checker's job).
- New `dr_recipe` table (migrate()) — DEDICATED, separate from the opaque `host_escrow` and from the
dropped `infra_backup*` tables. Keyed by `customer_id`; columns `recipe_version, host_id,
host_half_json, app_half_json, updated_at`.
- `SaveDRRecipeHostHalf` / `SaveDRRecipeAppHalf` — each upserts its half and PRESERVES the other
(last-write-wins per half; a re-report of one half never clobbers the other).
- `AssembleDRRecipe(rec)` — stitches the two halves into `AssembledRecipe{recipe_version, customer,
guests, pbs, drives, pve_storage, apps}`. Sub-sections pass through as `json.RawMessage` (verbatim);
**ignore-unknown** at the top level + version-skew tolerant (`recipe_version` = max) for forward-compat
across the three repos. Pure → unit-tested.
2. **Infra Backup feature removed** (`api/handler.go`, `store/store.go`, `web/configs.go`,
`templates/customer_unified.html`, `templates/customer.html`): the `POST/GET /api/v1/infra-backup[…]`
endpoints + handlers, the store methods/types (`SaveInfraBackup`/`GetInfraBackup`/`GetInfraBackupByID`/
`GetInfraBackupMeta`/`ListInfraBackupVersions`/`pruneInfraBackups`, `InfraBackupMeta`/`InfraBackupVersion`),
and the operator "Infra Backup" panel. `GET /api/v1/recovery/{id}` now returns **config_yaml only**.
The customer-page config-drift badge (diffed against the stored controller.yaml) is hidden; the live
"Show Diff" path is unaffected.
### Ingest (`hub/internal/api/handler.go`)
3. **Plaintext secret purge** (`store/store.go migrate()`): `DROP infra_backup_versions; DROP
infra_backups; VACUUM; wal_checkpoint(TRUNCATE)` — gated on table existence so normal restarts skip
it. VACUUM physically reclaims the freed pages so the plaintext keys/tokens are not merely delinked.
- `handleHostReport` persists the `dr_recipe` host-half (keyed by the host's customer).
- `handleReport` persists the `dr_recipe` app-half (keyed by `customer_id`), mirroring the
`app_telemetry` pattern. Both backward-compatible (old agents/controllers omit the field) and never
fatal to the heartbeat.
## Gate (STEP 1a) — PASSED before any change
### View (`hub/internal/web/dr_recipe.go` + customer page)
The demo's latest host-report (id 699, agent v0.36.7) carried **5 PBS snapshots, all `verify=ok`,
newest `2026-06-15T18:41:29Z` (13.9 h old, < 26 h)** plus a matching successful vzdump. The repoint's
data source is present and fresh, so the repoint alone clears the email.
- A DR-recipe panel on the customer detail page (which half landed + last-updated) with a
**Download recipe (JSON)** link.
- `GET /customers/{id}/dr-recipe.json` serves the assembled recipe (operator dashboard-auth, pretty
JSON, `Content-Disposition` attachment). No decrypt, nothing to redact.
## Verification (live, non-hollow)
### Boundary
- **Build + tests:** `go build ./... && go test ./...` green. `deadline_test.go` covers fresh+verified→
quiet (the **companion**), stale→alarm, failed-verify→alarm, no-host-report→quiet, db-dump half
preserved, and a pure `assessBackupFreshness` table. Red-proof: with the old event-based half
temporarily restored, the companion + no-report + db-dump-preserved tests **fail**, while the
pure-helper test still passes — the behavioral tests are sensitive to the logic.
- **Deploy:** hub v0.12.0 rolled out; startup log: `Retired infra-backup: dropped 2 table(s) and
VACUUMed to reclaim plaintext pages`.
- **Tables gone:** `.tables` on the live `/data/hub.db` shows no `infra_backup*`. DB shrank
**65.8 MB → 52.2 MB**; no `-wal` sidecar lingers.
- **Secret purge proof (grep on the live post-VACUUM DB file):** the infra-backup-exclusive markers
are gone — `encryption_key_b64` 17→**0**, `controller_config_b64` 11→**0**, and the actual
AES-key value from spike record 109 present→**0** (physically reclaimed, not just delinked). The
residual `cf_api_token`/`cf_tunnel_token` hits are in `customer_configs.config_json` (the hub's
legitimate config store), not the infra-backup blobs (which held CF tokens base64-encoded inside the
now-zero `controller_config_b64`).
- **Endpoints retired:** `POST /api/v1/infra-backup` and `GET …/versions` return **404**; control
`POST /api/v1/report` still returns **401** (routed, unauth) — confirming the 404s are route removal,
not a blanket failure.
- **Demo email fix:** the next 03:00 deadline run cannot be observed within this session, but the live
demo host-report (fresh verified PBS, 13.9 h) exercises the not-missed path, and demo emits
`db_dump_completed` daily (never missed) — so `expected_backup_missed` will no longer fire.
The recipe is PLAINTEXT-at-rest because it carries NO secrets — only identifiers/intents/sizes/
coordinates. The PBS key stays in escrow, the access token in identity-escrow, the restic password in
escrow. This is the clean inverse of the retired infra-backup (which shipped `restic_password`/
`cf_api_token` and was a zero-knowledge violation). The leak-preventing allowlist is enforced at the
controller emitter; the hub adds a defense-in-depth no-secrets sweep.
## Flagged for the operator (out of scope here)
### Tests
- **Rotate the exposed credentials** that were in the dropped blobs (Cloudflare API + tunnel tokens
for `demo-felhom.eu` first; hub/session secrets if shared). They remain valid until rotated; the
purge removes the at-rest copy but not their validity.
- **Separate historical leak:** the legacy `reports` table holds thousands of rows with a plaintext
`restic_password` value from old controller versions. The **live** controller no longer sends it
(removed in controller v0.69.0), but the historical rows persist — a distinct purge/rotation
decision, deliberately not done here.
- **`peti-felhom` is a defunct customer marked `active`** (no host-report or controller report since
Feb 2026). Its daily `expected_backup_missed` stops with this change (no host-report → no backup
alarm), but it should be marked inactive to silence all residual noise.
`TestDRRecipe_StoreRoundTrip`, `TestAssembleDRRecipe_MatchesGolden` (assembled shape pinned in
`testdata/dr-recipe.golden.json`), `TestAssembleDRRecipe_IgnoreUnknownAndVersionSkew`,
`TestAssembleDRRecipe_PartialHalves`, `TestAssembleDRRecipe_NoSecrets`. `go build`/`go vet`/`go test ./...`
green.
## Out of scope (untouched)
### Cross-repo golden discipline
Credential rotation; the Komga healthcheck; the secret-free DR "recipe" (the later DR slice).
The recipe wire spans three repos. The agent's `host-report.golden.json` `dr_recipe` section and the
hub's host-half test literal must stay key-consistent; the controller's emitter and the hub's app-half
test literal likewise. On any wire change, manually checksum-diff the golden across the three repos
(there is no shared types module yet).
### Deploy
Built + pushed `felhom-hub:0.13.0`; manifest tag bumped + ArgoCD `felhom` app synced.
+30
View File
@@ -1,5 +1,35 @@
# Felhom Hub — Changelog
## v0.13.0 — DR recipe: assemble + store + view the secret-free reconstruction recipe (2026-06-16)
**DR recipe slice (hub half)** — the assemble-store-view side of the secret-free reconstruction recipe
(`documentation/audits/SPIKE-dr-recipe-2026-06-16.md`). The hub receives two additive halves via the
existing report paths — the agent's storage/guest/PBS half (on the host-report) and the controller's
customer/apps half (on the controller report) — and assembles them into one operator-readable recipe per
customer. This is the clean inverse of the retired infra-backup: same "re-provision plan" goal, but
PLAINTEXT-at-rest is *correct* because the recipe has zero secrets.
- **Store** (`internal/store/dr_recipe.go`): a DEDICATED `dr_recipe` table (NOT `host_escrow`, NOT the
dropped `infra_backup*` tables) keyed by `customer_id`, holding `host_half_json` + `app_half_json` +
`recipe_version` + `host_id`. `SaveDRRecipeHostHalf` / `SaveDRRecipeAppHalf` each upsert their half and
PRESERVE the other (last-write-wins per half). `AssembleDRRecipe` stitches the two into an
`AssembledRecipe{recipe_version, customer, guests, pbs, drives, pve_storage, apps}` — sub-sections pass
through as `json.RawMessage` (verbatim), **ignore-unknown** at the top level and version-skew tolerant
(`recipe_version` = max of the two halves) for forward-compat across the three repos.
- **Ingest** (`internal/api/handler.go`): `handleHostReport` persists the `dr_recipe` host-half (keyed by
the host's customer); `handleReport` persists the `dr_recipe` app-half (keyed by `customer_id`) —
mirroring the `app_telemetry` pattern, backward-compatible (old agents/controllers omit the field), and
never fatal to the heartbeat.
- **View** (`internal/web/dr_recipe.go` + customer page): a DR-recipe panel on the customer detail page
(which half has landed + last-updated) with a **Download recipe (JSON)** link →
`GET /customers/{id}/dr-recipe.json` serves the assembled recipe (operator dashboard-auth, pretty JSON,
`Content-Disposition` attachment). No decrypt, nothing to redact.
- Tests: `TestDRRecipe_StoreRoundTrip` (each half preserves the other), `TestAssembleDRRecipe_MatchesGolden`
(the assembled wire shape pinned in `testdata/dr-recipe.golden.json`),
`TestAssembleDRRecipe_IgnoreUnknownAndVersionSkew` (a forward-compat half still assembles; version = max),
`TestAssembleDRRecipe_PartialHalves` (one half present), `TestAssembleDRRecipe_NoSecrets` (defense-in-depth
credential-key sweep). Pairs with felhom-agent v0.38.0 + felhom-controller v0.73.0.
## v0.12.0 — retire Infra Backup + purge its plaintext secrets + fix the daily backup-deadline email (2026-06-16)
Phase-1 of the Infra Backup retirement (per `documentation/audits/SPIKE-infra-backup-2026-06-15.md`).
+4
View File
@@ -62,6 +62,10 @@ All API endpoints require `Authorization: Bearer <api_key>` (except `/healthz` a
The `POST /api/v1/report` handler (v0.4.0+) automatically parses the optional `app_telemetry` JSON array from the request body and stores it in `app_telemetry` / `app_log_issues` tables. Old controllers (no `app_telemetry` key) continue to work unchanged.
### DR Recipe — secret-free reconstruction recipe (hub v0.13.0)
The hub assembles + stores + serves the **secret-free DR recipe** (`documentation/audits/SPIKE-dr-recipe-2026-06-16.md`) — the non-secret re-provision plan that complements escrow (keys) + PBS/restic (bytes). It arrives as two additive `dr_recipe` halves on the existing report paths: the **agent's** storage/guest/PBS half on `POST /api/v1/host-report` and the **controller's** customer/apps half on `POST /api/v1/report` (both backward-compatible, ignore-unknown). The hub stores them PLAINTEXT in a dedicated `dr_recipe` table keyed by `customer_id` (`SaveDRRecipeHostHalf` / `SaveDRRecipeAppHalf`, each preserving the other half), and `AssembleDRRecipe` stitches them into one operator-readable recipe (`{recipe_version, customer, guests, pbs, drives, pve_storage, apps}`; sub-sections pass through verbatim, version = max of the two halves). An operator views the panel on the customer page and downloads the assembled JSON at `GET /customers/{id}/dr-recipe.json`. **Plaintext-at-rest is correct here** — the recipe carries only identifiers/intents/sizes/coordinates, never a key/password/token (the boundary is enforced at the controller emitter). This is the clean inverse of the retired infra-backup.
### Infrastructure Backup — RETIRED (Phase-1, 2026-06-16, hub v0.12.0)
The Infra Backup mechanism (`POST/GET /api/v1/infra-backup`, the operator panel, the
+35
View File
@@ -241,6 +241,21 @@ func (h *Handler) handleReport(w http.ResponseWriter, r *http.Request) {
}
}
// DR recipe — persist the controller's secret-free customer/apps half (preserving any host half).
// Backward-compatible (old controllers won't have this field); a failure must not drop the report.
var drPayload struct {
DRRecipe json.RawMessage `json:"dr_recipe"`
}
if err := json.Unmarshal(body, &drPayload); err == nil && len(drPayload.DRRecipe) > 0 {
var ver drRecipeVersionOnly
_ = json.Unmarshal(drPayload.DRRecipe, &ver)
if err := h.store.SaveDRRecipeAppHalf(payload.CustomerID, ver.RecipeVersion, drPayload.DRRecipe); err != nil {
h.logger.Printf("[WARN] Failed to save DR-recipe app-half for %s: %v", payload.CustomerID, err)
} else {
h.logger.Printf("[INFO] DR-recipe app-half stored for customer %s (v%d)", payload.CustomerID, ver.RecipeVersion)
}
}
h.logger.Printf("[INFO] Received report from %s (%d bytes)", payload.CustomerID, len(body))
// Build response with optional customer_blocked flag
@@ -297,6 +312,14 @@ type hostReportPayload struct {
Cloudflared struct {
Status string `json:"status"`
} `json:"cloudflared"`
// DR recipe — the agent's storage/guest/PBS half (secret-free). RawMessage = stored verbatim,
// ignore-unknown (forward-compat). Persisted to dr_recipe, assembled with the controller half.
DRRecipe json.RawMessage `json:"dr_recipe"`
}
// drRecipeVersionOnly extracts just recipe_version from a half's JSON (ignore-unknown). 0 if absent.
type drRecipeVersionOnly struct {
RecipeVersion int `json:"recipe_version"`
}
// hostPBSSnapshot mirrors the agent's hub.PBSSnapshot wire contract (slice 6 Phase B). The
@@ -525,6 +548,18 @@ func (h *Handler) handleHostReport(w http.ResponseWriter, r *http.Request) {
h.logger.Printf("[INFO] host-report from %s (%d guests, %d storage targets, %d backups, %d restore-tests, %d pbs-snapshots, %d bytes)",
hostID, len(rep.Guests), len(rep.StorageTargets), len(rep.Backups), len(rep.RestoreTests), len(rep.PBSSnapshots), len(body))
// DR recipe — persist the agent's secret-free storage/guest/PBS half (preserving any app half).
// A failure here must NOT drop the heartbeat (the report already saved); just warn.
if len(rep.DRRecipe) > 0 && custID != "" {
var ver drRecipeVersionOnly
_ = json.Unmarshal(rep.DRRecipe, &ver)
if err := h.store.SaveDRRecipeHostHalf(custID, hostID, ver.RecipeVersion, rep.DRRecipe); err != nil {
h.logger.Printf("[WARN] Failed to save DR-recipe host-half for customer %s (host %s): %v", custID, hostID, err)
} else {
h.logger.Printf("[INFO] DR-recipe host-half stored for customer %s (host %s, v%d)", custID, hostID, ver.RecipeVersion)
}
}
blocked := false
if cc, err := h.store.GetCustomerConfig(custID); err == nil && cc != nil && cc.Status == "blocked" {
blocked = true
+130
View File
@@ -0,0 +1,130 @@
package store
import (
"database/sql"
"encoding/json"
)
// DR recipe (SPIKE-dr-recipe-2026-06-16) — the secret-free reconstruction recipe, stored PLAINTEXT
// because it has NO secrets (it is the clean inverse of the retired infra-backup). The hub receives two
// halves via the existing report paths — the agent's storage/guest/PBS half (on the host-report) and
// the controller's customer/apps half (on the controller report) — and assembles them into one record
// keyed by customer. Both halves carry ONLY identifiers/intents/sizes/coordinates; the table is a
// DEDICATED `dr_recipe`, NOT host_escrow (opaque) and NOT the dropped infra_backup* tables.
// DRRecipe is the stored two-half record for one customer.
type DRRecipe struct {
CustomerID string
RecipeVersion int
HostID string
HostHalfJSON string // the agent half (guests/pbs/drives/pve_storage); "" until a host-report lands
AppHalfJSON string // the controller half (customer/apps); "" until a controller report lands
UpdatedAt string
}
// SaveDRRecipeHostHalf upserts the agent (storage/guest/PBS) half for a customer, preserving any
// app half already stored. Last-write-wins on the host half + host_id + recipe_version.
func (s *Store) SaveDRRecipeHostHalf(customerID, hostID string, recipeVersion int, hostHalf []byte) error {
_, err := s.db.Exec(`
INSERT INTO dr_recipe (customer_id, recipe_version, host_id, host_half_json, app_half_json, updated_at)
VALUES (?, ?, ?, ?, '', datetime('now'))
ON CONFLICT(customer_id) DO UPDATE SET
recipe_version = excluded.recipe_version,
host_id = excluded.host_id,
host_half_json = excluded.host_half_json,
updated_at = datetime('now')`,
customerID, recipeVersion, hostID, string(hostHalf),
)
return err
}
// SaveDRRecipeAppHalf upserts the controller (customer/apps) half for a customer, preserving any
// host half already stored. Last-write-wins on the app half + recipe_version.
func (s *Store) SaveDRRecipeAppHalf(customerID string, recipeVersion int, appHalf []byte) error {
_, err := s.db.Exec(`
INSERT INTO dr_recipe (customer_id, recipe_version, host_id, host_half_json, app_half_json, updated_at)
VALUES (?, ?, '', '', ?, datetime('now'))
ON CONFLICT(customer_id) DO UPDATE SET
recipe_version = excluded.recipe_version,
app_half_json = excluded.app_half_json,
updated_at = datetime('now')`,
customerID, recipeVersion, string(appHalf),
)
return err
}
// GetDRRecipe returns the stored two-half record for a customer (nil if none).
func (s *Store) GetDRRecipe(customerID string) (*DRRecipe, error) {
var r DRRecipe
err := s.db.QueryRow(`
SELECT customer_id, recipe_version, host_id, host_half_json, app_half_json, updated_at
FROM dr_recipe WHERE customer_id = ?`, customerID).
Scan(&r.CustomerID, &r.RecipeVersion, &r.HostID, &r.HostHalfJSON, &r.AppHalfJSON, &r.UpdatedAt)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, err
}
return &r, nil
}
// AssembledRecipe is the operator-facing single recipe: the two halves stitched together. The
// sub-sections are passed through as json.RawMessage so the assembly is robust to forward-compat
// additions inside either half (ignore-unknown at the top level, verbatim passthrough below).
type AssembledRecipe struct {
RecipeVersion int `json:"recipe_version"`
Customer json.RawMessage `json:"customer,omitempty"`
Guests json.RawMessage `json:"guests,omitempty"`
PBS json.RawMessage `json:"pbs,omitempty"`
Drives json.RawMessage `json:"drives,omitempty"`
PVEStorage json.RawMessage `json:"pve_storage,omitempty"`
Apps json.RawMessage `json:"apps,omitempty"`
}
// hostHalfShape / appHalfShape capture only the top-level keys the assembly stitches; encoding/json
// drops any unknown top-level key (forward-compat — a newer half with extra sections still parses).
type hostHalfShape struct {
RecipeVersion int `json:"recipe_version"`
Guests json.RawMessage `json:"guests"`
PBS json.RawMessage `json:"pbs"`
Drives json.RawMessage `json:"drives"`
PVEStorage json.RawMessage `json:"pve_storage"`
}
type appHalfShape struct {
RecipeVersion int `json:"recipe_version"`
Customer json.RawMessage `json:"customer"`
Apps json.RawMessage `json:"apps"`
}
// AssembleDRRecipe stitches the two stored halves into one operator-facing recipe. Either half may be
// empty (not yet reported); the assembly fills what it has. recipe_version = the max of the two halves'
// versions (1 if both absent). Ignore-unknown: extra top-level keys in either half are dropped, nested
// shapes pass through verbatim — so the three repos can evolve the recipe without silent drift here.
func AssembleDRRecipe(rec *DRRecipe) (AssembledRecipe, error) {
out := AssembledRecipe{RecipeVersion: 1}
if rec == nil {
return out, nil
}
if rec.HostHalfJSON != "" {
var h hostHalfShape
if err := json.Unmarshal([]byte(rec.HostHalfJSON), &h); err != nil {
return out, err
}
out.Guests, out.PBS, out.Drives, out.PVEStorage = h.Guests, h.PBS, h.Drives, h.PVEStorage
if h.RecipeVersion > out.RecipeVersion {
out.RecipeVersion = h.RecipeVersion
}
}
if rec.AppHalfJSON != "" {
var a appHalfShape
if err := json.Unmarshal([]byte(rec.AppHalfJSON), &a); err != nil {
return out, err
}
out.Customer, out.Apps = a.Customer, a.Apps
if a.RecipeVersion > out.RecipeVersion {
out.RecipeVersion = a.RecipeVersion
}
}
return out, nil
}
+182
View File
@@ -0,0 +1,182 @@
package store
import (
"encoding/json"
"os"
"reflect"
"regexp"
"sort"
"strings"
"testing"
)
// The two halves as the agent (host) and controller (app) emit them — keys must match the cross-repo
// golden (the agent's host-report.golden.json dr_recipe section + the controller's emitter).
const drHostHalf = `{
"recipe_version": 1,
"guests": [ { "vmid": 9201, "cores": 4, "memory_bytes": 12884901888, "disk_bytes": 34359738368 } ],
"pbs": { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" },
"drives": [ { "durable_id": "uuid:da9e7089-cf8e-4617-adcb-a377743fae00", "role": "bulk-data", "mount_path": "/mnt/felhom-usb", "intent": "enrolled", "total_bytes": 1000000000000 } ],
"pve_storage": [ { "name": "local-lvm", "type": "lvmthin", "content": "rootdir,images" }, { "name": "felhom-usb", "type": "usb", "content": "backup" } ]
}`
const drAppHalf = `{
"recipe_version": 1,
"customer": { "id": "cust-demo", "display": "Demo Customer", "domain": "demo-felhom.eu" },
"apps": [ { "catalog_ref": "romm", "enabled": true, "storage_bindings": [ { "container_path": "/roms", "drive": "felhom-flash", "subpath": "userdata/roms" } ] } ]
}`
// TestDRRecipe_StoreRoundTrip: each half upserts independently and preserves the other; GetDRRecipe
// returns both.
func TestDRRecipe_StoreRoundTrip(t *testing.T) {
s := newTestStore(t)
// App half lands first.
if err := s.SaveDRRecipeAppHalf("cust-demo", 1, []byte(drAppHalf)); err != nil {
t.Fatal(err)
}
rec, _ := s.GetDRRecipe("cust-demo")
if rec == nil || rec.AppHalfJSON == "" || rec.HostHalfJSON != "" {
t.Fatalf("after app-half: want app set, host empty, got %+v", rec)
}
// Host half lands later — must NOT clobber the app half.
if err := s.SaveDRRecipeHostHalf("cust-demo", "host-01", 1, []byte(drHostHalf)); err != nil {
t.Fatal(err)
}
rec, _ = s.GetDRRecipe("cust-demo")
if rec == nil || rec.AppHalfJSON == "" || rec.HostHalfJSON == "" || rec.HostID != "host-01" {
t.Fatalf("after host-half: both halves must be present + host_id set, got %+v", rec)
}
// A re-report of the host half preserves the app half (and vice-versa).
if err := s.SaveDRRecipeHostHalf("cust-demo", "host-01", 1, []byte(drHostHalf)); err != nil {
t.Fatal(err)
}
rec, _ = s.GetDRRecipe("cust-demo")
if rec.AppHalfJSON == "" {
t.Fatal("re-reporting the host half clobbered the app half")
}
// Absent customer → nil, no error.
if got, err := s.GetDRRecipe("nobody"); err != nil || got != nil {
t.Fatalf("absent customer should be (nil,nil), got (%v,%v)", got, err)
}
}
// TestAssembleDRRecipe_MatchesGolden: assembling both halves yields the golden's key shape (the
// cross-repo wire pin) and the correct stitched values.
func TestAssembleDRRecipe_MatchesGolden(t *testing.T) {
rec := &DRRecipe{CustomerID: "cust-demo", RecipeVersion: 1, HostHalfJSON: drHostHalf, AppHalfJSON: drAppHalf}
asm, err := AssembleDRRecipe(rec)
if err != nil {
t.Fatal(err)
}
b, _ := json.Marshal(asm)
var got map[string]any
json.Unmarshal(b, &got)
raw, err := os.ReadFile("testdata/dr-recipe.golden.json")
if err != nil {
t.Fatal(err)
}
var golden map[string]any
if err := json.Unmarshal(raw, &golden); err != nil {
t.Fatalf("golden invalid: %v", err)
}
// Top-level key set must match the golden (the assembled wire shape).
if ga, gb := keysOf(golden), keysOf(got); !reflect.DeepEqual(ga, gb) {
t.Errorf("assembled key drift:\n golden=%v\n got =%v", ga, gb)
}
// And the stitched values: customer from the app half, drives/pbs from the host half.
if asm.RecipeVersion != 1 {
t.Errorf("recipe_version=%d want 1", asm.RecipeVersion)
}
if !jsonContains(t, asm.Customer, "cust-demo") || !jsonContains(t, asm.Customer, "demo-felhom.eu") {
t.Errorf("customer not stitched from app half: %s", asm.Customer)
}
if !jsonContains(t, asm.Drives, "uuid:da9e7089-cf8e-4617-adcb-a377743fae00") {
t.Errorf("drives not stitched from host half: %s", asm.Drives)
}
if !jsonContains(t, asm.Apps, "romm") {
t.Errorf("apps not stitched from app half: %s", asm.Apps)
}
}
// TestAssembleDRRecipe_IgnoreUnknownAndVersionSkew: a half carrying an UNKNOWN top-level field and a
// HIGHER recipe_version still assembles (forward-compat), and recipe_version reflects the max.
func TestAssembleDRRecipe_IgnoreUnknownAndVersionSkew(t *testing.T) {
futureHost := `{ "recipe_version": 2, "drives": [], "pve_storage": [], "guests": [],
"future_section": { "whatever": 1 }, "network_topology": ["a","b"] }`
rec := &DRRecipe{CustomerID: "c", RecipeVersion: 2, HostHalfJSON: futureHost, AppHalfJSON: drAppHalf}
asm, err := AssembleDRRecipe(rec)
if err != nil {
t.Fatalf("ignore-unknown failed to parse a forward-compat half: %v", err)
}
if asm.RecipeVersion != 2 {
t.Errorf("recipe_version=%d, want max(2,1)=2", asm.RecipeVersion)
}
// The unknown sections are dropped (not in AssembledRecipe), but the assembly did not error.
b, _ := json.Marshal(asm)
if string(b) == "" {
t.Fatal("empty assembly")
}
}
// TestAssembleDRRecipe_PartialHalves: only one half present → assemble what we have, no error.
func TestAssembleDRRecipe_PartialHalves(t *testing.T) {
onlyApp, err := AssembleDRRecipe(&DRRecipe{AppHalfJSON: drAppHalf})
if err != nil || onlyApp.Apps == nil || onlyApp.Drives != nil {
t.Errorf("only-app assembly wrong: %+v err=%v", onlyApp, err)
}
onlyHost, err := AssembleDRRecipe(&DRRecipe{HostHalfJSON: drHostHalf})
if err != nil || onlyHost.Drives == nil || onlyHost.Customer != nil {
t.Errorf("only-host assembly wrong: %+v err=%v", onlyHost, err)
}
empty, err := AssembleDRRecipe(nil)
if err != nil || empty.RecipeVersion != 1 {
t.Errorf("nil assembly should be a v1 empty recipe, got %+v err=%v", empty, err)
}
}
// TestAssembleDRRecipe_NoSecrets: defense-in-depth — the assembled output carries no credential-shaped
// key. (The load-bearing boundary is enforced at the controller emitter; this guards the hub side.)
func TestAssembleDRRecipe_NoSecrets(t *testing.T) {
asm, _ := AssembleDRRecipe(&DRRecipe{HostHalfJSON: drHostHalf, AppHalfJSON: drAppHalf})
b, _ := json.Marshal(asm)
re := regexp.MustCompile(`(?i)(password|secret|token|hash|passphrase|api[_-]?key|\bkey\b|enc:)`)
var v any
json.Unmarshal(b, &v)
var walk func(any)
walk = func(n any) {
switch x := n.(type) {
case map[string]any:
for k, c := range x {
if re.MatchString(k) {
t.Errorf("secret-shaped key %q in assembled recipe", k)
}
walk(c)
}
case []any:
for _, c := range x {
walk(c)
}
}
}
walk(v)
}
func jsonContains(t *testing.T, raw json.RawMessage, substr string) bool {
t.Helper()
return len(raw) > 0 && string(raw) != "null" && strings.Contains(string(raw), substr)
}
func keysOf(m map[string]any) []string {
ks := make([]string, 0, len(m))
for k := range m {
ks = append(ks, k)
}
sort.Strings(ks)
return ks
}
+20
View File
@@ -309,6 +309,26 @@ func (s *Store) migrate() error {
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN identity_blob BLOB`)
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN directive_json TEXT NOT NULL DEFAULT '{}'`)
// dr_recipe (SPIKE-dr-recipe-2026-06-16): the secret-free DR reconstruction recipe, stored
// PLAINTEXT (it has NO secrets — the clean inverse of the retired infra_backup). Two halves keyed
// by customer: the agent's storage/guest/PBS half (host_half_json, from the host-report) and the
// controller's customer/apps half (app_half_json, from the controller report); the hub assembles
// them on read. DEDICATED table, separate from the opaque host_escrow. One row per customer;
// each half is last-write-wins and preserves the other.
_, err = s.db.Exec(`
CREATE TABLE IF NOT EXISTS dr_recipe (
customer_id TEXT PRIMARY KEY,
recipe_version INTEGER NOT NULL DEFAULT 1,
host_id TEXT NOT NULL DEFAULT '',
host_half_json TEXT NOT NULL DEFAULT '',
app_half_json TEXT NOT NULL DEFAULT '',
updated_at DATETIME NOT NULL DEFAULT (datetime('now'))
);
`)
if err != nil {
return err
}
return nil
}
+30
View File
@@ -0,0 +1,30 @@
{
"recipe_version": 1,
"customer": { "id": "cust-demo", "display": "Demo Customer", "domain": "demo-felhom.eu" },
"guests": [
{ "vmid": 9201, "cores": 4, "memory_bytes": 12884901888, "disk_bytes": 34359738368 }
],
"pbs": { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" },
"drives": [
{
"durable_id": "uuid:da9e7089-cf8e-4617-adcb-a377743fae00",
"role": "bulk-data",
"mount_path": "/mnt/felhom-usb",
"intent": "enrolled",
"total_bytes": 1000000000000
}
],
"pve_storage": [
{ "name": "local-lvm", "type": "lvmthin", "content": "rootdir,images" },
{ "name": "felhom-usb", "type": "usb", "content": "backup" }
],
"apps": [
{
"catalog_ref": "romm",
"enabled": true,
"storage_bindings": [
{ "container_path": "/roms", "drive": "felhom-flash", "subpath": "userdata/roms" }
]
}
]
}
+25 -4
View File
@@ -254,12 +254,28 @@ func (s *Server) handleCustomerUnified(w http.ResponseWriter, r *http.Request, c
AppTelemetry []store.CustomerAppSummary
HasAppTelemetry bool
HasDRRecipe bool
DRRecipeUpdatedAt string
DRRecipeHasHost bool
DRRecipeHasApps bool
Flash string
ActiveNav string
CSRFField template.HTML
CSRFToken string
}
// DR recipe presence — show the secret-free reconstruction recipe panel + download link when
// either half has landed (host-report and/or controller report).
var hasDR, drHost, drApps bool
var drUpdated string
if rec, err := s.store.GetDRRecipe(customerID); err == nil && rec != nil {
hasDR = rec.HostHalfJSON != "" || rec.AppHalfJSON != ""
drHost = rec.HostHalfJSON != ""
drApps = rec.AppHalfJSON != ""
drUpdated = rec.UpdatedAt
}
data := pageData{
CustomerID: customerID,
CustomerName: name,
@@ -293,6 +309,11 @@ func (s *Server) handleCustomerUnified(w http.ResponseWriter, r *http.Request, c
AppTelemetry: appTelemetry,
HasAppTelemetry: len(appTelemetry) > 0,
HasDRRecipe: hasDR,
DRRecipeUpdatedAt: drUpdated,
DRRecipeHasHost: drHost,
DRRecipeHasApps: drApps,
Flash: r.URL.Query().Get("flash"),
ActiveNav: "configs",
CSRFField: s.csrfField(r),
@@ -741,10 +762,10 @@ func flattenYAML(m map[string]interface{}, prefix string) map[string]string {
// configDiff represents a single key-value difference between two configs.
type configDiff struct {
Key string `json:"key"`
HubValue string `json:"hub"`
CtrlValue string `json:"controller"`
Status string `json:"status"` // "changed", "hub_only", "controller_only"
Key string `json:"key"`
HubValue string `json:"hub"`
CtrlValue string `json:"controller"`
Status string `json:"status"` // "changed", "hub_only", "controller_only"
}
// compareYAMLValues parses two YAML strings and returns their value differences.
+61
View File
@@ -0,0 +1,61 @@
package web
import (
"encoding/json"
"net/http"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// handleDRRecipeDownload serves the assembled secret-free DR recipe for a customer as a JSON download
// (SPIKE-dr-recipe-2026-06-16). The recipe is PLAINTEXT because it carries NO secrets — only the
// reconstruction scaffolding (guest sizing, drive inventory, PVE storage, PBS coordinates, app
// inventory + storage bindings). Operator (dashboard-auth) only; no decrypt, nothing to redact.
func (s *Server) handleDRRecipeDownload(w http.ResponseWriter, r *http.Request, customerID string) {
if r.Method != http.MethodGet {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
rec, err := s.store.GetDRRecipe(customerID)
if err != nil {
s.logger.Printf("[ERROR] DR-recipe lookup failed for %s: %v", customerID, err)
http.Error(w, "Internal error", http.StatusInternalServerError)
return
}
if rec == nil {
http.Error(w, "No DR recipe for this customer yet (awaiting a host-report + controller report)", http.StatusNotFound)
return
}
assembled, err := store.AssembleDRRecipe(rec)
if err != nil {
s.logger.Printf("[ERROR] DR-recipe assemble failed for %s: %v", customerID, err)
http.Error(w, "Internal error", http.StatusInternalServerError)
return
}
out, err := json.MarshalIndent(assembled, "", " ")
if err != nil {
http.Error(w, "Internal error", http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("Content-Disposition", "attachment; filename=\"dr-recipe-"+safeFilename(customerID)+".json\"")
w.Write(out)
s.logger.Printf("[INFO] DR-recipe downloaded for customer %s (v%d)", customerID, assembled.RecipeVersion)
}
// safeFilename keeps a customer id safe for a Content-Disposition filename (alnum/-/_ only).
func safeFilename(s string) string {
out := make([]rune, 0, len(s))
for _, r := range s {
switch {
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '-', r == '_':
out = append(out, r)
default:
out = append(out, '_')
}
}
if len(out) == 0 {
return "customer"
}
return string(out)
}
+4
View File
@@ -238,6 +238,10 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
} else {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
}
case strings.HasPrefix(path, "/customers/") && strings.HasSuffix(path, "/dr-recipe.json"):
customerID := strings.TrimPrefix(path, "/customers/")
customerID = strings.TrimSuffix(customerID, "/dr-recipe.json")
s.handleDRRecipeDownload(w, r, customerID)
case strings.HasPrefix(path, "/customers/"):
customerID := strings.TrimPrefix(path, "/customers/")
s.handleCustomerUnified(w, r, customerID)
@@ -559,6 +559,36 @@
</section>
{{end}}
{{if .HasDRRecipe}}
<!-- DR recipe (secret-free reconstruction recipe) -->
<section class="card">
<h2>DR Recipe <span class="text-muted" style="font-size: 0.85em; font-weight: normal;">(secret-free reconstruction plan)</span></h2>
<p class="text-muted" style="margin-top: 0;">
The non-secret re-provision plan — guest sizing, drive inventory (durable-id → role → mount → intent),
PVE storage defs, PBS coordinates, and app inventory + storage bindings. It complements escrow (keys)
and PBS/restic (bytes): <strong>it contains no key, password, or token</strong>. Use it to rebuild the
host/guest/storage scaffolding before the PBS bytes land.
</p>
<div class="info-grid">
<div class="info-item">
<span class="label">Storage / guest / PBS half (agent)</span>
<span class="value">{{if .DRRecipeHasHost}}<span class="badge badge-ok">present</span>{{else}}<span class="badge badge-warn">awaiting host-report</span>{{end}}</span>
</div>
<div class="info-item">
<span class="label">Customer / apps half (controller)</span>
<span class="value">{{if .DRRecipeHasApps}}<span class="badge badge-ok">present</span>{{else}}<span class="badge badge-warn">awaiting controller report</span>{{end}}</span>
</div>
<div class="info-item">
<span class="label">Last updated</span>
<span class="value">{{if .DRRecipeUpdatedAt}}{{.DRRecipeUpdatedAt}}{{else}}—{{end}}</span>
</div>
</div>
<div style="margin-top: 1rem;">
<a href="/customers/{{.CustomerID}}/dr-recipe.json" class="btn" download>Download recipe (JSON)</a>
</div>
</section>
{{end}}
<!-- Notifications -->
<section class="card">
<h2>Notifications</h2>
+1 -1
View File
@@ -117,7 +117,7 @@ spec:
spec:
containers:
- name: hub
image: gitea.dooplex.hu/admin/felhom-hub:v0.12.0
image: gitea.dooplex.hu/admin/felhom-hub:v0.13.0
ports:
- containerPort: 8080
name: http