felhom.eu/documentation/audits/SPIKE-infra-backup-2026-06-15.md

SPIKE — Characterize the Infra Backup mechanism (revive vs retire)

Date: 2026-06-15 (investigation completed 2026-06-16) Type: Report-only audit. Validated against pushed source in felhom-controller, felhom-agent, felhom.eu/hub; the hub's actually-stored infra-backup records inspected read-only (DB copied off the live pod, queried on the build host, copy deleted afterwards). Status: findings complete. No code changes, no version bump. Recommendation + fork list at the end. Scope note: the Komga healthcheck fix is independent and explicitly out of scope.

The Komga healthcheck issue can ship separately whenever; it is not part of this spike.

---

One-screen summary

The Infra Backup mechanism is DEAD CODE that left a plaintext-secret liability standing on the hub.

• The controller's payload builder (report/infra_backup.go, backup/local_infra.go, setup/scanner, report/infra_pull) was deleted in slice 8C (abe4e8e, controller v0.37.0, 2026-06-10) when the disk tier moved controller→agent. The push method (PushInfraBackup) and the backup_completed emitter (NotifyBackupCompleted) survive as orphaned, caller-less stubs.
• The agent never re-implemented infra-backup. Slice 8C on the agent is disk format/wipe/assign/eject + crash-consistent vzdump only. Zero infra.backup matches in the agent tree.
• The hub still accepts pushes (POST /api/v1/infra-backup is live) and still displays the panel, but nothing pushes anymore. The panel is frozen at the last pre-8C controller run.
• The "stopped ~2026-06-05" hypothesis is CONFIRMED, and it is the slice-8C move. The demo customer's last stored record is from controller v0.34.0 (pre-8C), timestamp 2026-06-05T05:35:23Z. After the demo guest self-updated past 8C, no controller code path produces an infra-backup or a backup_completed event — which is exactly why the hub's daily deadline check fires (it expects backup_completed and never sees one).
• SECURITY (confirmed live, empirically): the hub stores each infra-backup as a plaintext JSON blob at rest (SQLite TEXT column). The latest demo record contains, in the clear: the AES app-secret master key (encryption_key_b64), the restic password, and a controller.yaml carrying Cloudflare API + tunnel tokens, the hub API key, the dashboard password hash, and the session secret — together with the apps' ENC:-encrypted secrets that that very key unlocks. This is a zero-knowledge violation and a standing credential exposure, independent of revive/retire.

Recommendation (phased):

1. Now — retire the liability: purge the 15 stale plaintext records, rotate any still-valid credentials they contain, delete the orphaned stubs + dead endpoint/panel, and repoint the deadline check to the real current backup signal (the agent's host-report PBS/Backups freshness). This stops the daily false-alarm email and removes the security exposure.
2. Later (DR re-enrollment slice) — revive the concept, not the code: a slim, secret-free "DR recipe" (customer → apps → drive/durable-id intents → guest sizing) that complements PBS's "ingredients". Secrets are never in it; they are recovered from the PBS whole-CT snapshot, never regenerated.

Do not revive the current mechanism as-is: its format is bare-metal-era obsolete and it is a security liability.

---

Evidence map (where each claim is grounded)

Repo	What	Location
controller	payload builder BuildInfraBackup / InfraBackup struct	report/infra_backup.go — DELETED in abe4e8e (v0.37.0)
controller	local DR mirror + drive scanner	backup/local_infra.go, setup/scanner.go, report/infra_pull* — DELETED in abe4e8e
controller	orphaned push method (no callers)	controller/internal/report/pusher.go:147 PushInfraBackup → POST {hub}/api/v1/infra-backup
controller	orphaned emitter (no callers)	controller/internal/notify/notifier.go:280 NotifyBackupCompleted → event backup_completed
controller	8C boundary comment	controller/internal/backup/backup.go:18-21 ("Disk-tier backup (restic, cross-drive, drive-recovery, infra-backup) has been moved out of the controller into the host agent (slice 8C)")
controller	DR-pull client retired comment	controller/internal/report/config_pull.go:15
agent	infra-backup builder/pusher	does not exist — 0 grep matches for infra.backup/InfraBackup
agent	host-report Backups[] / PBSSnapshots[]	felhom-agent/internal/hub/report.go:176-194, :223-234; built hub/collect.go:99; pushed hub/loop.go (60–3600 s)
hub	receive endpoint (still live)	hub/internal/api/handler.go:172 → handleInfraBackupPush :1247
hub	get / versions / recovery	handler.go:174 (/versions), :178 (/{cid}), :1333 (/recovery/{cid}, needs X-Retrieval-Password)
hub	storage tables	hub/internal/store/store.go:199 infra_backup_versions; :97 legacy infra_backups
hub	persist + GFS prune	store.go:637 SaveInfraBackup; :759 pruneInfraBackups
hub	operator panel (view-only)	hub/internal/web/templates/customer_unified.html:305-353; data wired web/configs.go:244-248
hub	deadline check (the email)	hub/internal/monitor/deadline.go:31-86 (daily 05:00 Budapest); allowed event handler.go:911
hub	zero-knowledge posture (escrow only)	handler.go:627-634, :678 ("OPAQUE bytes. No decrypt path exists.")

---

Q1 — CONTENT: what one infra-backup version actually contains

(a) The live stored records (read-only inspection of /data/hub.db)

Two customers have records; both are dead (no recent writes):

customer	versions	oldest	newest	last controller_version
demo-felhom	7	2026-03-29	2026-06-05 05:35:22	v0.34.0 (pre-8C)
peti-felhom	8	2026-02-27	2026-02-28 09:42:27	(Feb-era)

The latest demo record (infra_backup_versions.id=109, 44 001 bytes) decodes to these top-level fields:

customer_id            demo-felhom
domain                 demo-felhom.eu
controller_version     v0.34.0
timestamp              2026-06-05T05:35:23Z
controller_config_b64  <base64 of controller.yaml, 2601 chars decoded>
settings_json_b64      <base64 of settings.json>
disk_layout            { mounts: [...] }            # bare-metal fstab/blkid model
deployed_stacks        [5 apps]                     # each: compose + app.yaml + .felhom.yml
restic_password        <PLAINTEXT, 60 chars>
encryption_key_b64     <PLAINTEXT, 44 chars = base64 of a 32-byte AES key>

deployed_stacks (the stale set — confirms the panel's complaint; current live apps are RomM/ActualBudget/etc.):

stack	needs_hdd	compose	app.yaml	ENC: secrets in app.yaml
bentopdf	false	✓	✓	0
immich	true	✓	✓	1
mealie	false	✓	✓	0
nextcloud	true	✓	✓	3
tandoor	false	✓	✓	2

So one version = the full re-provision bundle for the whole customer: identity (customer/domain/version/timestamp), the controller's own config + settings, the disk registry, and every deployed app's compose + env + metadata.

(b) The builder that produced it (pre-8C, now deleted)

The payload was assembled by report.BuildInfraBackup into:

type InfraBackup struct {
    CustomerID, Domain, ControllerVersion, Timestamp string
    ControllerConfigB64 string   // controller.yaml
    SettingsJSONB64     string   // settings.json
    DiskLayout          backup.DiskLayout
    DeployedStacks      []InfraStack   // name/display/hdd_path/needs_hdd + compose/app.yaml/.felhom.yml (all b64)
    ResticPassword      string
    CrossDrivePassword  string
    EncryptionKeyB64    string
}

(cross_drive_password is absent in the live demo record — that path wasn't populated on this node — but the field was part of the wire shape.)

SECURITY FLAG — CONFIRMED zero-knowledge violation, plaintext at rest

The hub holds, unencrypted in SQLite, for demo-felhom (and peti-felhom):

• encryption_key_b64 — the AES master key that decrypts every app's ENC: secrets, shipped in the same blob as those encrypted secrets. The encryption is therefore meaningless against a hub-side reader.
• restic_password — plaintext.
• controller_config_b64 → controller.yaml, whose decoded text contains non-empty values for: cf_api_token, cf_tunnel_token (Cloudflare), api_key (hub), token, password_hash (dashboard), session_secret, restic_password_file.

(Values were not extracted into this report — only key names and presence/length were inspected.)

This directly contradicts the hub's stated posture, which is zero-knowledge only for the PBS recovery-code escrow (handler.go:627 "the hub stores the bytes and NEVER decrypts them"; :678 "No decrypt path exists"). Infra-backup bypasses that posture entirely. Any revive must be secret-free or encrypt-before-push; and the existing records should be purged + the credentials rotated regardless of the revive/retire decision.

---

Q2 — STORAGE: where, encryption, retention

• Where: SQLite at /data/hub.db on the hub pod (felhom-system/hub-…). Two tables: infra_backup_versions(id, customer_id, backup_json TEXT, created_at) (store.go:199, the versioned history behind the panel) and legacy infra_backups(customer_id PK, backup_json TEXT, updated_at) (store.go:97, last-write-wins mirror; demo's legacy row matches version 109).
• Encryption at rest: none. backup_json is the raw JSON string (SaveInfraBackup, store.go:637). The DB file itself is not encrypted. Plaintext secrets (Q1) therefore sit in the clear on disk and in any DB backup of the hub.
• Retention / the "7 versions": GFS prune on every write (pruneInfraBackups, store.go:759): keep all < 24 h, then newest-per-day for 7 days, newest-per-week for 4 weeks, newest-per-month for 3 months; delete > 3 months. Cap ≈ 14. Demo shows 7 because writes stopped 2026-06-05 — the older dailies/weeklies have aged out of the window and nothing new arrives, so the count is decaying toward 0 over time, not held at 7.
• Also inside restic? The pre-8C _infra/ mirror (<drive>/felhom-data/backups/secondary/_infra/ = controller.yaml + stacks/) was swept into the secondary cross-drive restic repo, and a local copy lived at <drive>/.felhom-infra-backup/ (backup.json + metadata.json + history, last 5). All three locations are retired with slice 8C (restic, cross-drive, and local_infra.go are gone). Today the only surviving copy of the old bundle is the hub DB rows.

---

Q3 — PUSH PATH + WHY IT STOPPED

Pre-8C push path: the controller's nightly backup cycle (daily ~01:02/02:00) ran the disk tiers (DB dumps, restic, cross-drive), then pushInfraBackup(...) → Pusher.PushInfraBackup → POST {hub}/api/v1/infra-backup (Bearer-authed, 3 retries / 5 s). It was also wired to fire on startup, after config-apply, and after a cross-drive run. The same nightly cycle emitted NotifyBackupCompleted → event backup_completed to the hub.

Why it stopped — confirmed: slice 8C (abe4e8e, controller v0.37.0, 2026-06-10) deleted the builder and unwired the scheduler job (the move of the disk tier controller→agent, retiring ~12.3k LOC). The agent did not pick up infra-backup — only restic/cross-drive/drive-recovery's replacements (vzdump + PBS) and disk management. What remains in the controller are two dead stubs with no callers: PushInfraBackup (pusher.go:147) and NotifyBackupCompleted (notifier.go:280).

The live records corroborate the timeline exactly: the demo customer's last record is controller v0.34.0 (pre-8C) at 2026-06-05T05:35:23Z. Once the demo guest's bootstrap-managed controller self-updated past v0.37.0, no code path emits either the bundle or the event. (GFS pruning means the retained weekly cadence — …05-17, 05-18, 06-05 — can't be read as the true write cadence; the 05:35 timestamp vs the usual 01:02 suggests the final push was a config-apply/manual trigger rather than the scheduled job, which had likely already gone quiet as the node transitioned. The mechanism is dead in current code regardless of the exact last beat.)

Link to the deadline email — confirmed: monitor.CheckBackupDeadlines (deadline.go:31, daily 05:00 Budapest) looks for a backup_completed/backup_failed event since midnight; finding none it raises expected_backup_missed (severity error) → dispatcher → customer + operator email. Because the only emitter of backup_completed was the now-removed nightly cycle (NotifyBackupCompleted, now caller-less), the check can never pass → it fires every day. The agent does not emit backup_completed (it records backup state in-memory and lets the hub poll it via host-report), so nothing else satisfies the check. The infra-backup death and the daily false alarm are the same root cause.

---

Q4 — PROXMOX FIT: what's stale, what it should become

The stored format is bare-metal-controller-era and no longer matches the platform:

• disk_layout is fstab/blkid bare-metal. Live demo record: mounts: [ {/dev/sda1 → /mnt/sys_drive, ext4, fstab_options "defaults,noatime", role system_data}, {/dev/sdb1 → /mnt/hdd_1, role hdd_storage} ]. The current model is an unprivileged LXC guest whose drives are agent-owned and surfaced via the intermediary mount at /mnt/felhom-drives/<name>, identified by durable-ids (uuid:…, wipe-binding byid:<wwn>/byuuid:), with the agent — not the controller — as the storage source of truth. /mnt/sys_drive + /dev/sdaN + fstab_options are meaningless under LXC/Proxmox.
• restic_password / cross_drive_password describe retired tiers. There is no restic and no cross-drive backup anymore; whole-CT durability is PBS.
• The "self-contained DR bundle living on the controller" premise is gone. Secrets (app.yaml + the AES key) live on the guest rootfs, which is already inside the PBS whole-CT snapshot. Re-shipping them to the hub is both redundant and the security violation above.
• The app set drifts and isn't reconciled. The panel shows a 5-app set that no longer exists; nothing prunes/refreshes it because nothing writes.

What it should become — a "recipe", not a "bundle". PBS provides the ingredients (the bytes: rootfs, app.yaml-encrypted, the key, DBs, volumes). The missing piece for DR is the recipe: the non-secret statement of intent needed to rebuild a customer from scratch —

• customer id + domain + tier/sizing (guest RAM/disk, e.g. 9201 = 12 GiB),
• deployed apps as catalog references + non-secret deploy fields (not full compose blobs — those come from the catalog/PBS),
• storage intent: which drives (by durable-id), their roles (primary / bulk-data / vzdump-target / pbs-offsite), and userdata layout,
• PBS coordinates for the whole-CT snapshot to restore.

This is a small, secret-free document that turns "I have PBS bytes" into "I can re-provision this customer". Anything secret stays out and is recovered from PBS, never regenerated (consistent with the controller's existing recovery-unit fail-closed data_key gate: refuse rather than regenerate a data-encrypting key).

---

Q5 — HUB MANAGEABILITY: now vs should-be

Now: the operator panel (customer_unified.html:305-353) is view-only — last-updated age, stack count, disk count, version count, and a collapsible history table (date / apps / disk count). There are no download/restore/re-provision buttons. The data-serving endpoints exist (GET /infra-backup/{cid}, …/versions, and GET /recovery/{cid} gated by X-Retrieval-Password) but are wired for the now-defunct controller restore flow, not operator actions.

Should be (for a revived recipe): from the hub the operator can (1) download the DR definition, (2) prune / pin versions (protect a known-good one from GFS), and (3) drive a DR re-provision — feed the recipe into the agent's re-enrollment so a replacement guest is rebuilt and the PBS snapshot restored into it. The panel should also surface staleness honestly (it currently shows an 11-day-old, wrong-app-set record with no warning that the feed is dead).

---

Q6 — DR RELATIONSHIP + RECOMMENDATION

How it relates to the rest of DR

• PBS whole-CT snapshot = the ingredients. Per-guest, captures the entire rootfs (app.yaml encrypted + the encryption key + DBs + configs) and bind-mounted data per coverage. This is the authoritative byte-level recovery source and already holds everything the infra-backup tried to ship as secrets.
• Host-report Backups[] / PBSSnapshots[] (report.go:176, :223) = the freshness/health signal. The agent enumerates PBS snapshots (verify state, encrypted, size, time) and last vzdump per target, pushed every cycle. This is current and agent-sourced — the right input for "did backups happen", which is what the deadline check actually wants.
• Future DR re-enrollment = needs the recipe (Q4) to map customer→apps→drives→sizing. PBS gives bytes but not this mapping; reverse-engineering the app set from a restored rootfs is fragile.

So the three are complementary: PBS = bytes, host-report = freshness, recipe = re-provision plan. The current infra-backup tried to be all three at once and, by also carrying live secrets, became a liability.

Recommendation — phased: RETIRE the implementation now, REVIVE the concept later (secret-free)

Why not keep/revive as-is: it is dead code, its format is bare-metal-obsolete, and it is an active credential exposure. Why not pure-retire forever: PBS + host-report give bytes + freshness but not the re-provision recipe, which has real DR value once re-enrollment lands.

Phase 1 — now (security + hygiene, do regardless):

1. Purge the 15 stale plaintext records (demo-felhom ×7 + peti-felhom ×8) from infra_backup_versions and infra_backups. They are wrong-app-set, bare-metal-era, and a standing secret leak.
2. Rotate any credentials in them that may still be valid — prioritise the Cloudflare API + tunnel tokens for demo-felhom.eu (and the hub api_key / session_secret if shared). The demo guest's AES key + restic password are likely obsolete (fresh LXC re-provision), but confirm before assuming.
3. Delete the dead stubs: controller PushInfraBackup (pusher.go) + NotifyBackupCompleted (notifier.go); decide whether to also drop the hub POST /infra-backup endpoint + panel now or park them for the recipe (fork below).
4. Fix the deadline email (independent win): repoint CheckBackupDeadlines to the agent-sourced backup-freshness in the latest host-report (Backups[] / PBSSnapshots[] recency per customer) instead of the dead backup_completed event. Optionally also have the controller's quiesce loop emit a real backup_completed on each successful app-data backup so the event becomes truthful again. Either stops the daily false alarm; the host-report route is preferred because the agent is now the backup source of truth.

Phase 2 — DR re-enrollment slice (revive the concept): introduce a secret-free DR recipe (rename away from "Infra Backup" to e.g. DR Manifest / Infra Definition to kill the secret-bundle mental model). Built from current truth — the controller emits the secret-free app recipe (catalog refs + non-secret deploy fields + storage-role intents), the agent contributes the storage/guest facts (durable-ids, roles, sizing, PBS coordinates). Pushed to the hub, stored (encrypt-before-push only if any residual sensitive field survives — ideally none), hub-manageable (download / prune / pin / re-provision), and positioned as the recipe alongside PBS's ingredients. Secrets are recovered from the PBS whole-CT snapshot at restore time, never shipped, never regenerated.

In that world the deadline-check email is already correct (fixed in Phase 1 against host-report freshness), and the recipe is a clean DR input rather than a credential pile.

---

Fork list (decisions for project-Claude / operator)

1. Cadence of the recommendation: accept the phased plan (retire-now + revive-later), or pure-retire (drop the concept entirely and rely on PBS + host-report + future reverse-engineered app set)? — recommend phased.
2. Purge + rotate (Phase 1.1/1.2): approve deleting the 15 stale records now? Confirm which embedded creds are still live and must rotate (Cloudflare tokens first). — recommend yes, purge + rotate CF.
3. Deadline-check repoint (Phase 1.4): (a) consume host-report Backups/PBSSnapshots recency, (b) controller quiesce emits real backup_completed, or (c) both? — recommend (a), optionally (c).
4. Dead endpoint/panel: delete the hub POST /api/v1/infra-backup + GET …/versions + the panel now, or keep the endpoint dormant for the Phase-2 recipe? — recommend delete the panel + push endpoint now; keep a clean slate for the recipe.
5. If reviving (Phase 2): confirm the recipe holds zero secrets; confirm the controller-builds-app-recipe / agent-builds-storage-recipe split and the storage table (new vs reuse infra_backup_versions); confirm the hub-manageable actions (download / prune / pin / re-provision).
6. Naming: rename "Infra Backup" → "DR Manifest"/"Infra Definition" to prevent the secret-bundle reappearing by association.

---

Appendix — investigation method (read-only, reproducible)

• Source validated against the pushed trees in e:\git\felhom-controller, e:\git\felhom-agent, e:\git\felhom.eu (citations above).
• Live records inspected by kubectl cp of felhom-system/hub-…:/data/hub.db → build host 192.168.0.180 (sudo sqlite3 + python3), querying infra_backup_versions / infra_backups. Only key names, presence, and lengths of secret fields were read; no secret values were extracted into this report. The DB copy and all temp files were deleted from the build host after inspection.
• No write was performed against the hub, its DB, or any live host.