felhom.eu/documentation/audits/SPIKE-infra-backup-2026-06-15.md

# SPIKE — Characterize the Infra Backup mechanism (revive vs retire)

**Date:** 2026-06-15 (investigation completed 2026-06-16)
**Type:** Report-only audit. Validated against pushed source in `felhom-controller`, `felhom-agent`, `felhom.eu/hub`; the hub's actually-stored infra-backup records inspected **read-only** (DB copied off the live pod, queried on the build host, copy deleted afterwards).
**Status:** findings complete. **No code changes, no version bump.** Recommendation + fork list at the end.
**Scope note:** the Komga healthcheck fix is independent and explicitly out of scope.

> The Komga healthcheck issue can ship separately whenever; it is not part of this spike.

---

## One-screen summary

**The Infra Backup mechanism is DEAD CODE that left a plaintext-secret liability standing on the hub.**

- The controller's payload **builder** (`report/infra_backup.go`, `backup/local_infra.go`, `setup/scanner`, `report/infra_pull`) was **deleted in slice 8C** (`abe4e8e`, controller **v0.37.0**, 2026-06-10) when the disk tier moved controller→agent. The **push method** (`PushInfraBackup`) and the **`backup_completed` emitter** (`NotifyBackupCompleted`) survive as **orphaned, caller-less stubs**.
- The agent **never re-implemented** infra-backup. Slice 8C on the agent is disk format/wipe/assign/eject + crash-consistent vzdump only. **Zero** `infra.backup` matches in the agent tree.
- The hub still **accepts** pushes (`POST /api/v1/infra-backup` is live) and still **displays** the panel, but **nothing pushes anymore**. The panel is frozen at the last pre-8C controller run.
- **The "stopped ~2026-06-05" hypothesis is CONFIRMED, and it is the slice-8C move.** The demo customer's last stored record is from controller **v0.34.0** (pre-8C), timestamp **2026-06-05T05:35:23Z**. After the demo guest self-updated past 8C, no controller code path produces an infra-backup or a `backup_completed` event — which is **exactly why the hub's daily deadline check fires** (it expects `backup_completed` and never sees one).
- **SECURITY (confirmed live, empirically):** the hub stores each infra-backup as a **plaintext JSON blob at rest** (SQLite TEXT column). The latest demo record contains, in the clear: the **AES app-secret master key** (`encryption_key_b64`), the **restic password**, and a controller.yaml carrying **Cloudflare API + tunnel tokens, the hub API key, the dashboard password hash, and the session secret** — together with the apps' `ENC:`-encrypted secrets that that very key unlocks. **This is a zero-knowledge violation** and a standing credential exposure, independent of revive/retire.

**Recommendation (phased):**
1. **Now — retire the liability:** purge the 15 stale plaintext records, rotate any still-valid credentials they contain, delete the orphaned stubs + dead endpoint/panel, and **repoint the deadline check** to the real current backup signal (the agent's host-report PBS/Backups freshness). This stops the daily false-alarm email and removes the security exposure.
2. **Later (DR re-enrollment slice) — revive the *concept*, not the code:** a slim, **secret-free** "DR recipe" (customer → apps → drive/durable-id intents → guest sizing) that complements PBS's "ingredients". Secrets are never in it; they are recovered from the PBS whole-CT snapshot, never regenerated.

Do **not** revive the current mechanism as-is: its format is bare-metal-era obsolete and it is a security liability.

---

## Evidence map (where each claim is grounded)

| Repo | What | Location |
|---|---|---|
| controller | payload builder `BuildInfraBackup` / `InfraBackup` struct | `report/infra_backup.go` — **DELETED** in `abe4e8e` (v0.37.0) |
| controller | local DR mirror + drive scanner | `backup/local_infra.go`, `setup/scanner.go`, `report/infra_pull*` — **DELETED** in `abe4e8e` |
| controller | orphaned push method (no callers) | `controller/internal/report/pusher.go:147` `PushInfraBackup` → `POST {hub}/api/v1/infra-backup` |
| controller | orphaned emitter (no callers) | `controller/internal/notify/notifier.go:280` `NotifyBackupCompleted` → event `backup_completed` |
| controller | 8C boundary comment | `controller/internal/backup/backup.go:18-21` ("Disk-tier backup (restic, cross-drive, drive-recovery, infra-backup) has been moved out of the controller into the host agent (slice 8C)") |
| controller | DR-pull client retired comment | `controller/internal/report/config_pull.go:15` |
| agent | infra-backup builder/pusher | **does not exist** — 0 grep matches for `infra.backup`/`InfraBackup` |
| agent | host-report `Backups[]` / `PBSSnapshots[]` | `felhom-agent/internal/hub/report.go:176-194`, `:223-234`; built `hub/collect.go:99`; pushed `hub/loop.go` (60–3600 s) |
| hub | receive endpoint (still live) | `hub/internal/api/handler.go:172` → `handleInfraBackupPush` `:1247` |
| hub | get / versions / recovery | `handler.go:174` (`/versions`), `:178` (`/{cid}`), `:1333` (`/recovery/{cid}`, needs `X-Retrieval-Password`) |
| hub | storage tables | `hub/internal/store/store.go:199` `infra_backup_versions`; `:97` legacy `infra_backups` |
| hub | persist + GFS prune | `store.go:637` `SaveInfraBackup`; `:759` `pruneInfraBackups` |
| hub | operator panel (view-only) | `hub/internal/web/templates/customer_unified.html:305-353`; data wired `web/configs.go:244-248` |
| hub | deadline check (the email) | `hub/internal/monitor/deadline.go:31-86` (daily 05:00 Budapest); allowed event `handler.go:911` |
| hub | zero-knowledge posture (escrow only) | `handler.go:627-634`, `:678` ("OPAQUE bytes. No decrypt path exists.") |

---

## Q1 — CONTENT: what one infra-backup version actually contains

### (a) The live stored records (read-only inspection of `/data/hub.db`)

Two customers have records; **both are dead** (no recent writes):

| customer | versions | oldest | newest | last controller_version |
|---|---|---|---|---|
| `demo-felhom` | 7 | 2026-03-29 | **2026-06-05 05:35:22** | **v0.34.0** (pre-8C) |
| `peti-felhom` | 8 | 2026-02-27 | 2026-02-28 09:42:27 | (Feb-era) |

The latest demo record (`infra_backup_versions.id=109`, 44 001 bytes) decodes to these top-level fields:

```
customer_id            demo-felhom
domain                 demo-felhom.eu
controller_version     v0.34.0
timestamp              2026-06-05T05:35:23Z
controller_config_b64  <base64 of controller.yaml, 2601 chars decoded>
settings_json_b64      <base64 of settings.json>
disk_layout            { mounts: [...] }            # bare-metal fstab/blkid model
deployed_stacks        [5 apps]                     # each: compose + app.yaml + .felhom.yml
restic_password        <PLAINTEXT, 60 chars>
encryption_key_b64     <PLAINTEXT, 44 chars = base64 of a 32-byte AES key>
```

`deployed_stacks` (the **stale** set — confirms the panel's complaint; current live apps are RomM/ActualBudget/etc.):

| stack | needs_hdd | compose | app.yaml | `ENC:` secrets in app.yaml |
|---|---|---|---|---|
| bentopdf | false | ✓ | ✓ | 0 |
| immich | true | ✓ | ✓ | 1 |
| mealie | false | ✓ | ✓ | 0 |
| nextcloud | true | ✓ | ✓ | 3 |
| tandoor | false | ✓ | ✓ | 2 |

So **one version = the full re-provision bundle for the whole customer**: identity (customer/domain/version/timestamp), the controller's own config + settings, the disk registry, and every deployed app's compose + env + metadata.

### (b) The builder that produced it (pre-8C, now deleted)

The payload was assembled by `report.BuildInfraBackup` into:

```go
type InfraBackup struct {
    CustomerID, Domain, ControllerVersion, Timestamp string
    ControllerConfigB64 string   // controller.yaml
    SettingsJSONB64     string   // settings.json
    DiskLayout          backup.DiskLayout
    DeployedStacks      []InfraStack   // name/display/hdd_path/needs_hdd + compose/app.yaml/.felhom.yml (all b64)
    ResticPassword      string
    CrossDrivePassword  string
    EncryptionKeyB64    string
}
```

(`cross_drive_password` is **absent** in the live demo record — that path wasn't populated on this node — but the field was part of the wire shape.)

### SECURITY FLAG — **CONFIRMED zero-knowledge violation, plaintext at rest**

The hub holds, **unencrypted in SQLite**, for `demo-felhom` (and `peti-felhom`):

- `encryption_key_b64` — **the AES master key that decrypts every app's `ENC:` secrets**, shipped in the **same blob** as those encrypted secrets. The encryption is therefore meaningless against a hub-side reader.
- `restic_password` — plaintext.
- `controller_config_b64` → controller.yaml, whose decoded text contains non-empty values for: `cf_api_token`, `cf_tunnel_token` (Cloudflare), `api_key` (hub), `token`, `password_hash` (dashboard), `session_secret`, `restic_password_file`.

(Values were **not** extracted into this report — only key names and presence/length were inspected.)

This directly contradicts the hub's stated posture, which is zero-knowledge **only for the PBS recovery-code escrow** (`handler.go:627` "the hub stores the bytes and NEVER decrypts them"; `:678` "No decrypt path exists"). Infra-backup bypasses that posture entirely. **Any revive must be secret-free or encrypt-before-push; and the existing records should be purged + the credentials rotated regardless of the revive/retire decision.**

---

## Q2 — STORAGE: where, encryption, retention

- **Where:** SQLite at `/data/hub.db` on the hub pod (`felhom-system/hub-…`). Two tables: `infra_backup_versions(id, customer_id, backup_json TEXT, created_at)` (`store.go:199`, the versioned history behind the panel) and legacy `infra_backups(customer_id PK, backup_json TEXT, updated_at)` (`store.go:97`, last-write-wins mirror; demo's legacy row matches version 109).
- **Encryption at rest:** **none.** `backup_json` is the raw JSON string (`SaveInfraBackup`, `store.go:637`). The DB file itself is not encrypted. Plaintext secrets (Q1) therefore sit in the clear on disk and in any DB backup of the hub.
- **Retention / the "7 versions":** GFS prune on every write (`pruneInfraBackups`, `store.go:759`): keep all < 24 h, then newest-per-day for 7 days, newest-per-week for 4 weeks, newest-per-month for 3 months; delete > 3 months. Cap ≈ 14. Demo shows **7** because writes stopped 2026-06-05 — the older dailies/weeklies have aged out of the window and nothing new arrives, so the count is decaying toward 0 over time, not held at 7.
- **Also inside restic?** The pre-8C `_infra/` mirror (`<drive>/felhom-data/backups/secondary/_infra/` = controller.yaml + stacks/) was swept into the secondary cross-drive **restic** repo, and a local copy lived at `<drive>/.felhom-infra-backup/` (backup.json + metadata.json + history, last 5). **All three locations are retired** with slice 8C (restic, cross-drive, and `local_infra.go` are gone). Today the only surviving copy of the old bundle is the hub DB rows.

---

## Q3 — PUSH PATH + WHY IT STOPPED

**Pre-8C push path:** the controller's nightly backup cycle (daily ~01:02/02:00) ran the disk tiers (DB dumps, restic, cross-drive), then `pushInfraBackup(...)` → `Pusher.PushInfraBackup` → `POST {hub}/api/v1/infra-backup` (Bearer-authed, 3 retries / 5 s). It was **also** wired to fire on startup, after config-apply, and after a cross-drive run. The **same nightly cycle** emitted `NotifyBackupCompleted` → event `backup_completed` to the hub.

**Why it stopped — confirmed:** **slice 8C** (`abe4e8e`, controller **v0.37.0**, 2026-06-10) deleted the builder and unwired the scheduler job (the move of the disk tier controller→agent, retiring ~12.3k LOC). The agent **did not** pick up infra-backup — only restic/cross-drive/drive-recovery's *replacements* (vzdump + PBS) and disk management. What remains in the controller are two **dead stubs with no callers**: `PushInfraBackup` (`pusher.go:147`) and `NotifyBackupCompleted` (`notifier.go:280`).

The live records corroborate the timeline exactly: the demo customer's last record is **controller v0.34.0** (pre-8C) at **2026-06-05T05:35:23Z**. Once the demo guest's bootstrap-managed controller self-updated past v0.37.0, no code path emits either the bundle or the event. (GFS pruning means the retained weekly cadence — …05-17, 05-18, 06-05 — can't be read as the true write cadence; the 05:35 timestamp vs the usual 01:02 suggests the final push was a config-apply/manual trigger rather than the scheduled job, which had likely already gone quiet as the node transitioned. The mechanism is dead in current code regardless of the exact last beat.)

**Link to the deadline email — confirmed:** `monitor.CheckBackupDeadlines` (`deadline.go:31`, daily 05:00 Budapest) looks for a `backup_completed`/`backup_failed` event since midnight; finding none it raises `expected_backup_missed` (severity error) → dispatcher → customer + operator email. Because the only emitter of `backup_completed` was the now-removed nightly cycle (`NotifyBackupCompleted`, now caller-less), the check **can never pass** → it fires **every day**. The agent does **not** emit `backup_completed` (it records backup state in-memory and lets the hub poll it via host-report), so nothing else satisfies the check. The infra-backup death and the daily false alarm are the **same root cause**.

---

## Q4 — PROXMOX FIT: what's stale, what it should become

The stored format is **bare-metal-controller-era** and no longer matches the platform:

- **`disk_layout` is fstab/blkid bare-metal.** Live demo record: `mounts: [ {/dev/sda1 → /mnt/sys_drive, ext4, fstab_options "defaults,noatime", role system_data}, {/dev/sdb1 → /mnt/hdd_1, role hdd_storage} ]`. The current model is an **unprivileged LXC guest** whose drives are agent-owned and surfaced via the **intermediary mount** at `/mnt/felhom-drives/<name>`, identified by **durable-ids** (`uuid:…`, wipe-binding `byid:<wwn>`/`byuuid:`), with the agent — not the controller — as the storage source of truth. `/mnt/sys_drive` + `/dev/sdaN` + `fstab_options` are meaningless under LXC/Proxmox.
- **`restic_password` / `cross_drive_password` describe retired tiers.** There is no restic and no cross-drive backup anymore; whole-CT durability is **PBS**.
- **The "self-contained DR bundle living on the controller" premise is gone.** Secrets (app.yaml + the AES key) live on the **guest rootfs**, which is already inside the **PBS whole-CT snapshot**. Re-shipping them to the hub is both redundant and the security violation above.
- **The app set drifts and isn't reconciled.** The panel shows a 5-app set that no longer exists; nothing prunes/refreshes it because nothing writes.

**What it should become — a "recipe", not a "bundle".** PBS provides the *ingredients* (the bytes: rootfs, app.yaml-encrypted, the key, DBs, volumes). The missing piece for DR is the *recipe*: the **non-secret** statement of intent needed to rebuild a customer from scratch —
- customer id + domain + tier/sizing (guest RAM/disk, e.g. 9201 = 12 GiB),
- deployed apps as **catalog references + non-secret deploy fields** (not full compose blobs — those come from the catalog/PBS),
- storage intent: which drives (by **durable-id**), their roles (primary / bulk-data / vzdump-target / pbs-offsite), and userdata layout,
- PBS coordinates for the whole-CT snapshot to restore.

This is a small, secret-free document that turns "I have PBS bytes" into "I can re-provision *this* customer". Anything secret stays out and is recovered from PBS, never regenerated (consistent with the controller's existing recovery-unit **fail-closed `data_key` gate**: refuse rather than regenerate a data-encrypting key).

---

## Q5 — HUB MANAGEABILITY: now vs should-be

**Now:** the operator panel (`customer_unified.html:305-353`) is **view-only** — last-updated age, stack count, disk count, version count, and a collapsible history table (date / apps / disk count). There are **no** download/restore/re-provision buttons. The data-serving endpoints exist (`GET /infra-backup/{cid}`, `…/versions`, and `GET /recovery/{cid}` gated by `X-Retrieval-Password`) but are wired for the now-defunct controller restore flow, not operator actions.

**Should be** (for a revived recipe): from the hub the operator can (1) **download** the DR definition, (2) **prune / pin** versions (protect a known-good one from GFS), and (3) **drive a DR re-provision** — feed the recipe into the agent's re-enrollment so a replacement guest is rebuilt and the PBS snapshot restored into it. The panel should also surface **staleness honestly** (it currently shows an 11-day-old, wrong-app-set record with no warning that the feed is dead).

---

## Q6 — DR RELATIONSHIP + RECOMMENDATION

### How it relates to the rest of DR

- **PBS whole-CT snapshot** = the **ingredients**. Per-guest, captures the entire rootfs (app.yaml encrypted **+** the encryption key **+** DBs + configs) and bind-mounted data per coverage. This is the authoritative byte-level recovery source and already holds everything the infra-backup tried to ship as secrets.
- **Host-report `Backups[]` / `PBSSnapshots[]`** (`report.go:176`, `:223`) = the **freshness/health signal**. The agent enumerates PBS snapshots (verify state, encrypted, size, time) and last vzdump per target, pushed every cycle. This is *current* and *agent-sourced* — the right input for "did backups happen", which is what the deadline check actually wants.
- **Future DR re-enrollment** = needs the **recipe** (Q4) to map customer→apps→drives→sizing. PBS gives bytes but not this mapping; reverse-engineering the app set from a restored rootfs is fragile.

So the three are complementary: PBS = bytes, host-report = freshness, recipe = re-provision plan. The **current** infra-backup tried to be all three at once and, by also carrying live secrets, became a liability.

### Recommendation — **phased: RETIRE the implementation now, REVIVE the concept later (secret-free)**

**Why not keep/revive as-is:** it is dead code, its format is bare-metal-obsolete, and it is an active credential exposure. **Why not pure-retire forever:** PBS + host-report give bytes + freshness but **not** the re-provision recipe, which has real DR value once re-enrollment lands.

**Phase 1 — now (security + hygiene, do regardless):**
1. **Purge** the 15 stale plaintext records (`demo-felhom` ×7 + `peti-felhom` ×8) from `infra_backup_versions` and `infra_backups`. They are wrong-app-set, bare-metal-era, and a standing secret leak.
2. **Rotate** any credentials in them that may still be valid — prioritise the **Cloudflare API + tunnel tokens** for `demo-felhom.eu` (and the hub `api_key` / `session_secret` if shared). The demo guest's AES key + restic password are likely obsolete (fresh LXC re-provision), but confirm before assuming.
3. **Delete the dead stubs:** controller `PushInfraBackup` (`pusher.go`) + `NotifyBackupCompleted` (`notifier.go`); decide whether to also drop the hub `POST /infra-backup` endpoint + panel now or park them for the recipe (fork below).
4. **Fix the deadline email** (independent win): repoint `CheckBackupDeadlines` to the **agent-sourced** backup-freshness in the latest host-report (`Backups[]` / `PBSSnapshots[]` recency per customer) instead of the dead `backup_completed` event. Optionally *also* have the controller's quiesce loop emit a real `backup_completed` on each successful app-data backup so the event becomes truthful again. Either stops the daily false alarm; the host-report route is preferred because the agent is now the backup source of truth.

**Phase 2 — DR re-enrollment slice (revive the concept):** introduce a **secret-free DR recipe** (rename away from "Infra Backup" to e.g. *DR Manifest* / *Infra Definition* to kill the secret-bundle mental model). Built from current truth — the **controller** emits the secret-free app recipe (catalog refs + non-secret deploy fields + storage-role intents), the **agent** contributes the storage/guest facts (durable-ids, roles, sizing, PBS coordinates). Pushed to the hub, stored (encrypt-before-push only if any residual sensitive field survives — ideally none), **hub-manageable** (download / prune / pin / re-provision), and positioned as the *recipe* alongside PBS's *ingredients*. Secrets are recovered from the PBS whole-CT snapshot at restore time, never shipped, never regenerated.

In that world the deadline-check email is already correct (fixed in Phase 1 against host-report freshness), and the recipe is a clean DR input rather than a credential pile.

---

## Fork list (decisions for project-Claude / operator)

1. **Cadence of the recommendation:** accept the phased plan (retire-now + revive-later), or pure-retire (drop the concept entirely and rely on PBS + host-report + future reverse-engineered app set)? — *recommend phased.*
2. **Purge + rotate (Phase 1.1/1.2):** approve deleting the 15 stale records now? Confirm which embedded creds are still live and must rotate (Cloudflare tokens first). — *recommend yes, purge + rotate CF.*
3. **Deadline-check repoint (Phase 1.4):** (a) consume host-report `Backups`/`PBSSnapshots` recency, (b) controller quiesce emits real `backup_completed`, or (c) both? — *recommend (a), optionally (c).*
4. **Dead endpoint/panel:** delete the hub `POST /api/v1/infra-backup` + `GET …/versions` + the panel now, or keep the endpoint dormant for the Phase-2 recipe? — *recommend delete the panel + push endpoint now; keep a clean slate for the recipe.*
5. **If reviving (Phase 2):** confirm the recipe holds **zero** secrets; confirm the controller-builds-app-recipe / agent-builds-storage-recipe split and the storage table (new vs reuse `infra_backup_versions`); confirm the hub-manageable actions (download / prune / pin / re-provision).
6. **Naming:** rename "Infra Backup" → "DR Manifest"/"Infra Definition" to prevent the secret-bundle reappearing by association.

---

## Appendix — investigation method (read-only, reproducible)

- Source validated against the pushed trees in `e:\git\felhom-controller`, `e:\git\felhom-agent`, `e:\git\felhom.eu` (citations above).
- Live records inspected by `kubectl cp` of `felhom-system/hub-…:/data/hub.db` → build host `192.168.0.180` (`sudo sqlite3` + `python3`), querying `infra_backup_versions` / `infra_backups`. **Only key names, presence, and lengths of secret fields were read; no secret values were extracted into this report.** The DB copy and all temp files were deleted from the build host after inspection.
- No write was performed against the hub, its DB, or any live host.