From 88362dac0af56f6050e0bfec35bf29769834a908 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 5 Jul 2026 11:56:48 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20v0.99.0=20=E2=80=94=20CHANGELOG/CONTEXT?= =?UTF-8?q?/REUSE/README=20for=20the=20restore-path=20fixes=20(F1/F3/O4)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- CHANGELOG.md | 35 +++++++++++++++++++++++++++++++++++ CONTEXT.md | 10 +++++++++- REUSE.md | 8 ++++++-- controller/README.md | 43 ++++++++++++++++++++++++++++++++----------- 4 files changed, 82 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a1d5869..bbe26f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,40 @@ ## Changelog +### v0.99.0 — restore-path fixes: dead restore UI + volume dumps + blank-secret redeploy (2026-07-05) + +TASK C1 — fixes F1/F3/O4 from the 2026-07-04 restore drill +(`felhom.eu/documentation/audits/DRILL-appdata-restore-2026-07-04.md`). F2 (one-click in-place +class-C restore) deliberately NOT included — product-design work (C2). + +- **F1 (HIGH — the restore panel was dead):** `GET /api/backup/snapshots?stack=` now exists + (`internal/api/router.go` + `backup.Manager.ListRestorePoints`, `internal/backup/restore_points.go`). + The backups.html restore panel fetched this restic-era route, got the catch-all 404, so the + snapshot dropdown never populated and "Visszaállítás indítása" could never enable. Returns the + ONE honest keep-side restore point (the current recovery unit): `time` = newest artifact mtime + (manifest / db-dumps / volume-dumps), `short_id:"helyi"`, `tier:1` always (Tier-2 copies are NOT + restorable via POST /backup/restore — never listed), `drive_label` from the storage registry. + Guards: traversal/empty → 400 (`validStackParam`), unknown stack → 404, no unit yet → `ok:true, data:[]`. + No template change needed — the JS payload contract was honoured server-side. +- **F3 — named-volume data was never backed up:** `DumpAppVolumesSafe` had no production caller. + New `runVolumeDumps` loop in `runDBDumpsInternal` (`internal/backup/backup.go`), running BEFORE + `captureAllRecoveryUnits` so manifests enumerate the fresh tars. Gate order is load-bearing: + protected-stack + has-volumes checks precede the Safe call (which stops the stack before its own + check — unconditional calls would bounce every volume-less app nightly); disconnected/decommissioned + drives skip like the DB loop. Failures land in the run summary and fail the run (no silent + partial). Zero-DB early return removed (volume-only apps still get dumps + unit refresh). + Test seam: `dumpVolumesSafe` func field (F17-style). +- **O4 — missing resettable secret redeployed blank:** the restore proceed-path now generates a + replacement credential via the catalog field's `generate` spec (`stacks.Manager.GenerateSecretForField` + → `backup.SetSecretGenerator` seam, wired in main.go), persisted encrypted through the existing + `RecreateStackFromUnit` → `SaveAppConfig` path. Data-keys are NEVER generated (gate untouched + + generator refuses `data_key` fields); values never logged. No-generator fields keep proceeding + with an upgraded "may fail to start" WARN. Residual case documented: a restored volume tar + carrying the OLD internal credential hash may still need a manual in-DB reset. + +Tests: +12 (api snapshots ×3, backup restore-points ×4, volume-dump gating ×3, secret-gen ×2 files); +all three fixes companion-red-proofed (hollow `[]` endpoint / removed volume gate / no-generation +each fail their test). Full `go build && go vet && go test ./...` green. + ### docs — CLAUDE.md refresh: slim-down to stable orientation (2026-07-03) No code change, no version bump. CLAUDE.md 338 → ~160 lines: full 30-package layout map (was 7); diff --git a/CONTEXT.md b/CONTEXT.md index 0bc45fc..50e6175 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -9,8 +9,16 @@ Last updated: 2026-07-03 (docs: CLAUDE.md refreshed — stable orientation; runbooks live in the felhom skills) +> **2026-07-05 — v0.99.0 restore-path fixes (TASK C1): drill findings F1/F3/O4 RESOLVED.** +> F1: `GET /api/backup/snapshots` implemented (`backup.ListRestorePoints`) — the restore panel +> populates and the restore button enables. F3: `runVolumeDumps` wired into the nightly/manual +> backup run (volume gate BEFORE DumpAppVolumesSafe; before unit capture). O4: unrecoverable +> resettable secrets get a generated replacement (`GenerateSecretForField` + `SetSecretGenerator` +> seam); data-key gate untouched. **F2 (one-click in-place class-C restore) remains OPEN → TASK C2.** +> O4 residual: restored volume tar with an OLD credential hash may still need a manual in-DB reset. + > **2026-07-04 — app-data restore drill** → see `felhom.eu/documentation/audits/DRILL-appdata-restore-2026-07-04.md`. -> Keep-side restore proven live on 9201 (class-A DB replay + fail-closed data-key gate + non-destruction). **F1 (HIGH): UI restore is dead — `/api/backup/snapshots` has no handler, so the restore button never enables.** F2: no one-click in-place class-C (HDD bind-mount) restore. F3: named-volume data never backed up (`DumpAppVolumes*` has no caller). +> Keep-side restore proven live on 9201 (class-A DB replay + fail-closed data-key gate + non-destruction). **F1 (HIGH): UI restore is dead — `/api/backup/snapshots` has no handler, so the restore button never enables.** F2: no one-click in-place class-C (HDD bind-mount) restore. F3: named-volume data never backed up (`DumpAppVolumes*` has no caller). *(F1/F3/O4 resolved in v0.99.0, see above.)* > **2026-07-03 — CLAUDE.md slimmed to stable orientation** (full package map, verified 9201 deploy > summary, no version-pinned state). Deep runbooks/design/testing doctrine now in the personal diff --git a/REUSE.md b/REUSE.md index 8338855..b072229 100644 --- a/REUSE.md +++ b/REUSE.md @@ -67,7 +67,8 @@ | `resolveContainerState` / `aggregateState` | controller/internal/stacks/manager.go | `(dockerState, dockerStatus)` / `([]ContainerInfo)` | State classification | `.State` says "running" even when unhealthy — `.Status` parse is the fix | | `Manager.logPostStartStatus` | controller/internal/stacks/manager.go | `(name, stackDir, env)` | Async post-start verification | compose up exits 0 on crash-loops; this is the detection. Goroutine + 3s, never blocks | | `Manager.EnsureBaseStack` | controller/internal/stacks/infra.go | `() error` | Traefik/cloudflared/FileBrowser infra convergence | Renders from `internal/infra` templates | -| `backup.Manager.DumpAppVolumesSafe` | controller/internal/backup/backup.go | `(stackName) error` | Volume tar of a live app | Stops → dumps → restarts; surfaces BOTH errors (app may be left stopped) | +| `backup.Manager.DumpAppVolumesSafe` | controller/internal/backup/backup.go | `(stackName) error` | Volume tar of a live app | Stops → dumps → restarts; surfaces BOTH errors (app may be left stopped). Check `GetDockerVolumes()!=0` + `IsProtectedStack` BEFORE calling — it stops the stack before its own volume check (see `runVolumeDumps`) | +| `backup.Manager.ListRestorePoints` | controller/internal/backup/restore_points.go | `(stackName) ([]RestorePoint, bool)` | Restorable keep-side backups (the /api/backup/snapshots payload) | ONE point per app (the current unit); tier always 1 — never list Tier-2 (not restorable via /backup/restore) | | `Manager.acquireRunning`/`releaseRunning`, `acquireMigrating` | controller/internal/backup/backup.go, controller/internal/stacks/migrate.go | `() error` | Single-flight for long ops | Copy this mutex-flag pattern for any new long-running manager op | ### Secrets hygiene @@ -78,6 +79,7 @@ | `crypto.LoadOrCreateKey` | controller/internal/crypto/crypto.go | `(path) ([]byte, error)` | The 32-byte key file (0600) | — | | `SaveAppConfig` / `LoadAppConfigDecrypted` | controller/internal/stacks/deploy.go | `(stackDir, cfg, encKey, sensitiveVars)` | app.yaml persistence | Encrypts only `SensitiveEnvVars(meta)`; never write app.yaml directly | | `generateValue` / `randomAlphanumeric` | controller/internal/stacks/deploy.go | `(spec "password:N\|hex:N\|base64key:N\|static:v")` | Auto-generated secrets | crypto/rand-backed; reuse the spec grammar | +| `Manager.GenerateSecretForField` | controller/internal/stacks/deploy.go | `(stackName, envVar) (string, bool)` | Replacement value for a RESETTABLE secret from its catalog `generate` spec (O4 restore path via `backup.SetSecretGenerator`) | REFUSES `data_key` fields, spec-less and non-secret fields; never log the value | | `reconcileRestoreSecrets` | controller/internal/backup/restore_unit.go | `(nonSecretEnv, recoveredSecrets, secretNames, dataKeyNames)` | Recovery-unit restore env merge | Units are secret-FREE by design; secrets come from live app.yaml | | `EncryptFile` / `DecryptFile` / `IsEncryptedFAB` | controller/internal/appexport/crypto.go | password-based file crypto | .fab export bundles | scrypt-derived AES+HMAC keys | | `maskRepoURL` | controller/internal/sync/sync.go | `(url) string` | Logging git URLs | Strips embedded credentials | @@ -174,6 +176,8 @@ | `integrations.Handler` + `StackProvider` | controller/internal/integrations/integrations.go + manager.go | OnlyOffice handlers | table-driven tests in package | | `bootstrap.PullFunc` | controller/internal/bootstrap/bootstrap.go | `report.PullConfig` | injected in bootstrap tests | | `offboxRunner` (func) | controller/internal/backup/offbox.go | `defaultOffboxRunner` (restic exec) | `SetOffboxRunner` injection point | +| `dumpVolumesSafe` (func seam) | controller/internal/backup/backup.go | nil → real `DumpAppVolumesSafe` | injected in controller/internal/backup/volume_dumps_test.go (gating tests without Docker) | +| `generateSecret` (func seam) | controller/internal/backup/backup.go | `stacks.Manager.GenerateSecretForField` via `SetSecretGenerator` (main.go) | injected in controller/internal/backup/restore_secrets_gen_test.go | Cross-repo edges: - `controller/internal/agentapi/client.go` ↔ **felhom-agent** local API (`/storage`, `/disks*`, `/backup*`, `/netstorage*`, `/guest/*`): pinned leaf SHA-256 + per-guest bearer token from bootstrap.json. @@ -205,6 +209,6 @@ Cross-repo edges: | CSRF ×2 | controller/internal/web/csrf.go (session HMAC) vs controller/internal/setup/csrf.go (cookie double-submit) — intentional (pre-auth wizard) but unlabeled | | Budapest timezone loader ×2 | controller/internal/scheduler/scheduler.go `getBudapestLocation` vs controller/internal/web/funcmap.go `getTimezone` | | JSON writers ×5, 3 envelope shapes | api `writeJSON`; web `writeDiskJSON`, `jsonResponse`/`jsonError`, `writeDebugJSON` | -| Safe-name validators ×3 | controller/internal/web/validate.go `validStackName`; controller/internal/backup/offbox.go `isSafeStackName`; controller/internal/appexport/validate.go `ValidateSegment` (strictest) | +| Safe-name validators ×4 | controller/internal/web/validate.go `validStackName`; controller/internal/api/router.go `validStackParam` (same body — api↔web import cycle); controller/internal/backup/offbox.go `isSafeStackName`; controller/internal/appexport/validate.go `ValidateSegment` (strictest) | | DB wait/import ×2 | controller/internal/appbackup/dbdump.go `waitDBReady`/`ImportDump` vs controller/internal/appexport/restore.go `waitForDB`/`importDBDump` | | compose exec ×2 | controller/internal/stacks/manager.go `composeExecCustomEnv` vs controller/internal/appexport/restore.go `composeExecEnv` (the latter has ctx+timeout; the former has the userdata belt) | diff --git a/controller/README.md b/controller/README.md index b5dac46..6b8d5c2 100644 --- a/controller/README.md +++ b/controller/README.md @@ -441,8 +441,14 @@ backups/primary// Docker image** — only the pinned image tag(s) (re-pulled on restore) and the *names* of the secret / `data_key` env vars. Rationale: app.yaml + the encryption key live on the guest rootfs → already in the PBS whole-guest snapshot, and the hub is deliberately zero-knowledge. Restore recovers the - original secrets from the guest's own app.yaml (live, or via PBS) and **regenerates nothing**; for a - `data_key` app it **fails closed** (refuse + warn) if the key can't be recovered. + original secrets from the guest's own app.yaml (live, or via PBS); for a `data_key` app it + **fails closed** (refuse + warn) if the key can't be recovered — data-keys are NEVER generated. + **Resettable secrets (O4, v0.99.0):** an unrecoverable resettable secret (DB password etc.) gets a + **generated replacement** from its catalog `generate` spec (`stacks.GenerateSecretForField` via the + `backup.SetSecretGenerator` seam) instead of redeploying blank (which failed compose-up); the new + value persists encrypted through the normal `RecreateStackFromUnit` → `SaveAppConfig` path. Fields + with no `generate` spec still proceed with a loud "may fail to start" WARN. Residual case: a restored + volume tar carrying the OLD internal credential hash may still need a manual in-DB reset. - Helpers: `RecoveryUnitPath` / `RecoveryUnitComposePath` / `RecoveryUnitManifestPath` (`internal/appbackup/paths.go`). Capture: `Manager.CaptureRecoveryUnit` (`internal/backup/recovery_unit.go`), run from the daily DB dump and the periodic `RefreshCache` (idempotent checksum-skip). The non-secret @@ -492,15 +498,23 @@ re-validates the pin off-disk at run time. `Manager.Tier2Info(stackName)` is the - **Validation** after each dump: checks file size, header presence, counts `CREATE TABLE` - Results cached in `settings.json` surviving container restarts -**Phase 1b — Docker Volume Dumps** (`internal/backup/backup.go`, runs after DB dumps) +**Phase 1b — Docker Volume Dumps** (`internal/backup/backup.go` `runVolumeDumps`, part of the same run) -- Iterates all deployed stacks that have Docker named volumes (`GetDockerVolumes()`) -- **v0.34.0:** Each stack is stopped before dump, restarted after (`DumpAppVolumesSafe()`) — prevents inconsistent tars of live databases. Protected stacks (traefik, etc.) that reject StopStack are skipped with a warning. +- **F3 (v0.99.0):** re-wired into the nightly/manual app-data backup run (`runDBDumpsInternal`) — + after the restic removal `DumpAppVolumesSafe` had no caller, so `volume-dumps/` was never produced. + Runs AFTER the DB dumps and BEFORE `captureAllRecoveryUnits` so the manifests enumerate fresh tars. +- Gate order (load-bearing): protected-stack (`cfg.IsProtectedStack`) and has-volumes + (`GetDockerVolumes()`) checks come BEFORE `DumpAppVolumesSafe` — the Safe variant stops the stack + before its own volume check, so unconditional calls would bounce every volume-less app nightly. + Disconnected/decommissioned drives skip with the same summary style as the DB loop. +- Each volume-bearing stack is stopped before dump, restarted after (`DumpAppVolumesSafe()`) — + prevents inconsistent tars of live databases. - For each volume: `docker run --rm -v :/vol:ro -v :/out alpine tar cf /out/.tar -C /vol .` -- 10-minute timeout per volume; warnings on failure (non-fatal) +- 10-minute timeout per volume; a per-stack failure lands in the run summary as `FAIL volumes:`, + flips the run's Success flag and fails the run (no silent partial) — other stacks still proceed - Stale tars cleaned up (volumes that no longer exist) - Volume names resolved with project prefix via `ResolveDockerVolumeNames()` (e.g., `mealie_mealie_data`) -- Dumps written to `AppVolumeDumpPath(appDrive, stackName)` +- Dumps written to `AppVolumeDumpPath(nsRoot, stackName)` **Phase 2 — Restic Snapshot** (`internal/backup/restic.go`, scheduled 03:00) @@ -589,10 +603,17 @@ appear in the restore dropdown with per-app snapshot filtering. | DB only, no HDD/volumes | Yes | Yes | n/a | n/a | | Config only | Yes | — | n/a | n/a | -**Snapshot API** (`/api/backup/snapshots?stack=`): -- Returns snapshots **only from the app's home drive** primary repo (prevents showing irrelevant snapshots from other drives) -- Appends a synthetic Tier 2 entry (ID `tier2-rsync`) from cross-drive config when last backup was successful -- Dropdown groups by tier: "1. szint — Helyi mentes" and "2. szint — Masodlagos masolat" +**Snapshot API** (`GET /api/backup/snapshots?stack=` — F1, v0.99.0): +- Backed by `backup.Manager.ListRestorePoints` (`internal/backup/restore_points.go`). The keep-side + restore has exactly **one** restore point per app — the current recovery unit — so the endpoint + returns at most one entry: `time` = newest artifact mtime (manifest / db-dumps / volume-dumps), + `short_id:"helyi"`, `tier:1`, `drive_label` from the storage registry (empty on the SSD fallback) +- **Never emits tier-2 entries**: Tier-2 copies are not restorable via `POST /backup/restore` (it + only reads the primary unit) — listing them would silently restore tier-1 data while claiming tier-2 +- Guards: empty/traversal stack name → 400 (`validStackParam`), unknown stack → 404, known stack + with no unit yet → `ok:true, data:[]` (the UI shows "Nincs elérhető mentés") +- History: the route was a restic-era leftover fetched by the template but unregistered — the + dropdown could never populate and the restore button never enabled (drill finding F1) **Restore type info** shown per-app when selected in dropdown (Hungarian banners): - Has HDD or Docker volumes: "Teljes visszaallitas: adatbazis + konfiguracio + felhasznaloi adatok"