felhom-controller/REPORT.md

# REPORT — felhom-controller v0.54.0 (Phase 2: recovery unit — capture + restore)

Each app's on-drive backup is a self-contained, recreatable **recovery unit** (secret-free), and restore
now **recreates an app from its unit + the guest's own secrets** with a **fail-closed data-key gate**.
Built, unit/integration-tested, shipped to `main`, deployed to guest 9201. (Phase 1, the deploy-side
double-nest GATE, shipped as v0.52.0; Phase 2 capture side as v0.53.x — see git history.)

## The design decision that shaped Phase 2 (secret handling)
The recovery unit carries **no secrets, no data-keys, and not the Docker image**. This was decided after
reading the *actual* hub code (the controller README that implied the hub stores app.yaml is stale
pre-strip):
- The hub is deliberately **zero-knowledge** — it holds a per-host recovery-code-wrapped PBS key it
  cannot decrypt + non-secret config; **no per-app secrets**. Escrowing app secrets there would regress
  that posture, so it was rejected.
- `app.yaml` (encrypted) + the encryption key live on the **guest rootfs** (`local-lvm:vm-9201-disk-0`,
  confirmed via `pct config`) → already inside the **PBS whole-guest snapshot**; the external drive
  (`mp0` bind) is not. So the secret↔data split maps onto the tiers: **secrets ride PBS; bulk userdata
  rides the drive + (Phase 3) Tier 2.**
- Therefore: secret-free unit; restore recovers the original secrets from the guest's own app.yaml
  (live, else PBS); **regenerate nothing**. `data_key` is a fail-closed annotation, not a
  preserve/regenerate decision.

## What shipped (v0.53.0 + v0.53.1)
- **Unit layout** (rooted at the existing `backups/primary/<app>/` — a deliberate low-churn choice, no
  risky dump-dir migration): `compose/` (docker-compose.yml + .felhom.yml + a **secret-stripped**
  app.yaml) + the existing `db-dumps/` + `volume-dumps/` + `manifest.json`. New helpers
  `RecoveryUnit{Path,ComposePath,ManifestPath}` (`internal/appbackup/paths.go`).
- **Secret-free manifest** (`internal/backup/recovery_unit.go`): app id, display name, controller
  version, timestamp, drive, namespace root, **image pins** (image NOT stored — re-pulled on restore),
  the **NAMES** of secret env vars (values never stored), `data_key` env-var names, an explicit
  `secret_source` note, captured config-file list, enumerated dumps, sha256 checksums.
- **Capture needs no secret access:** non-secret env is plaintext in app.yaml, so the capture excludes
  secret-named keys (plus a defensive `crypto.IsEncrypted` guard) and reads no secret value. New
  `StackDataProvider.GetStackRecoveryInfo` + `RecoveryInfo`, implemented by the main.go `stackAdapter`;
  `ParseComposeImages` extracts pins.
- **`data_key`**: `DeployField.DataKey` + `Metadata.DataKeyEnvVars()`; catalog `adventurelog/.felhom.yml`
  `SECRET_KEY` ("Titkosítási kulcs") marked `data_key: true`.
- **Refresh cadence (v0.53.1):** capture runs from the daily DB dump AND the periodic `RefreshCache`
  (startup + every 5m), **idempotent** — content is built in memory and writes are skipped when the unit
  is already current (checksum + dump-set + version), so a spinning USB drive is not thrashed.
- **Tests:** capture is secret-free (a secret in the source app.yaml never appears in the unit) +
  manifest structure + idempotency (unchanged → skip; config change → rewrite). `go build ./...` clean.

## Deploy mechanism (resolved this session)
The controller in guest 9201 is **golden/bootstrap-managed**: `felhom-controller-bootstrap.service` runs
`/usr/local/sbin/felhom-controller-bootstrap.sh`, which `docker run`s the tag from
`/etc/felhom-controller-image` (gitea anon-pull, no login). Deploy = build+push tag → anon-pull → update
that tag file → `systemctl restart felhom-controller-bootstrap.service`. Data volume + encryption key
persist. (This is what "self-update handles version drift" refers to.)

## Live validation (guest 9201, demo-felhom)
- Deployed v0.53.1; on startup `RefreshCache` captured units: **romm** (`images=3, secrets-referenced=3,
  data_keys=0`) and **actualbudget** (`images=1`, system-fallback path `…/sys_drive/felhom-data/…`).
- RomM unit on disk: `compose/{app.yaml,docker-compose.yml,.felhom.yml}` + `db-dumps/romm-mariadb.sql` +
  `manifest.json`. Manifest is secret-free (image pins + secret NAMES + `secret_source`); captured
  app.yaml holds only DOMAIN/HDD_PATH/SUBDOMAIN with the three secret names listed as stripped.
- **Secret-leak grep against the three actual RomM secret values → `NO_LEAK`.** Idempotency confirmed
  (single capture log line; the 5m refresh skips).

## Phase 2b — restore-from-unit + fail-closed gate (v0.54.0)
- **`reconcileRestoreSecrets`** (pure, exhaustively unit-tested): merges the unit's non-secret env with
  the secrets recovered from the guest's live app.yaml. A missing/empty **data-encrypting key** aborts
  the restore (a PBS whole-guest restore is required) — regenerating it would corrupt data. A missing
  resettable secret is non-fatal (warn + proceed). **Regenerates nothing.**
- **`RestoreFromRecoveryUnit`**: manifest → recover secrets from the guest → gate → restore named-volume
  tars → recover the app definition from the unit → redeploy with the reconstructed env (re-pull pinned
  image). Falls back to volume-only `RestoreApp` when no unit exists. Wired into `/backup/restore`.
- Seams: `RecoverStackSecrets` / `RecreateStackFromUnit` (adapter, with `encKey` to decrypt the live
  app.yaml); `stacks.RedeployFromEnv`. `isDebug` made nil-safe.
- **Tests:** the gate (recovered / data-key-missing→refuse / empty-data-key→refuse / resettable-missing
  →proceed, values used verbatim), the full orchestration (success→recreate-with-merged-env;
  data-key-missing→refused, recreate never called), and `data_key` parsing from `.felhom.yml`.

## Validation status
- **Unit/integration-tested (authoritative):** the fail-closed gate, the restore orchestration, secret
  reconciliation (regenerate-nothing), and the catalog→metadata `data_key` flow.
- **Live-validated (guest 9201):** the capture side (v0.53.1, RomM — secret-free, NO_LEAK). For Phase 2b
  on **AdventureLog** (a real data_key app): its unit manifest carries `data_key_env_vars: [SECRET_KEY]`
  (catalog→manifest flow live); and with `SECRET_KEY` made unrecoverable, `POST /backup/restore`
  **refused** with the exact fail-closed message **before any compose-up** (no side effects). The demo
  has no dashboard password → the API is open (auth + CSRF skipped), driven via the public URL.
- **One e2e not run — environment limit, not a code gap:** the full "deploy with data → restore →
  confirm decrypts" — AdventureLog's images do not fit the **8 GB guest rootfs** (deploy hit "no space
  left on device"). That is precisely the Phase 3 rootfs-headroom concern, now observed live.
  Key-preservation is covered by the gate's verbatim-recovery unit test. Demo left clean (AdventureLog
  reverted to not-deployed, no leftovers).

## Still ahead
Phase 3 (auto off-drive Tier 2 with rootfs-headroom guard) and Phase 4 (FileBrowser scoping + deploy-UI
DB-on-SSD note + monitoring sort). The README backup-paths section still shows the stale restic/secondary
layout — rewritten when Tier 2 lands.