Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 88362dac0a | |||
| a52851e79e | |||
| 73378a812c | |||
| f413f9539d |
@@ -1,5 +1,40 @@
|
|||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
### v0.99.0 — restore-path fixes: dead restore UI + volume dumps + blank-secret redeploy (2026-07-05)
|
||||||
|
|
||||||
|
TASK C1 — fixes F1/F3/O4 from the 2026-07-04 restore drill
|
||||||
|
(`felhom.eu/documentation/audits/DRILL-appdata-restore-2026-07-04.md`). F2 (one-click in-place
|
||||||
|
class-C restore) deliberately NOT included — product-design work (C2).
|
||||||
|
|
||||||
|
- **F1 (HIGH — the restore panel was dead):** `GET /api/backup/snapshots?stack=` now exists
|
||||||
|
(`internal/api/router.go` + `backup.Manager.ListRestorePoints`, `internal/backup/restore_points.go`).
|
||||||
|
The backups.html restore panel fetched this restic-era route, got the catch-all 404, so the
|
||||||
|
snapshot dropdown never populated and "Visszaállítás indítása" could never enable. Returns the
|
||||||
|
ONE honest keep-side restore point (the current recovery unit): `time` = newest artifact mtime
|
||||||
|
(manifest / db-dumps / volume-dumps), `short_id:"helyi"`, `tier:1` always (Tier-2 copies are NOT
|
||||||
|
restorable via POST /backup/restore — never listed), `drive_label` from the storage registry.
|
||||||
|
Guards: traversal/empty → 400 (`validStackParam`), unknown stack → 404, no unit yet → `ok:true, data:[]`.
|
||||||
|
No template change needed — the JS payload contract was honoured server-side.
|
||||||
|
- **F3 — named-volume data was never backed up:** `DumpAppVolumesSafe` had no production caller.
|
||||||
|
New `runVolumeDumps` loop in `runDBDumpsInternal` (`internal/backup/backup.go`), running BEFORE
|
||||||
|
`captureAllRecoveryUnits` so manifests enumerate the fresh tars. Gate order is load-bearing:
|
||||||
|
protected-stack + has-volumes checks precede the Safe call (which stops the stack before its own
|
||||||
|
check — unconditional calls would bounce every volume-less app nightly); disconnected/decommissioned
|
||||||
|
drives skip like the DB loop. Failures land in the run summary and fail the run (no silent
|
||||||
|
partial). Zero-DB early return removed (volume-only apps still get dumps + unit refresh).
|
||||||
|
Test seam: `dumpVolumesSafe` func field (F17-style).
|
||||||
|
- **O4 — missing resettable secret redeployed blank:** the restore proceed-path now generates a
|
||||||
|
replacement credential via the catalog field's `generate` spec (`stacks.Manager.GenerateSecretForField`
|
||||||
|
→ `backup.SetSecretGenerator` seam, wired in main.go), persisted encrypted through the existing
|
||||||
|
`RecreateStackFromUnit` → `SaveAppConfig` path. Data-keys are NEVER generated (gate untouched +
|
||||||
|
generator refuses `data_key` fields); values never logged. No-generator fields keep proceeding
|
||||||
|
with an upgraded "may fail to start" WARN. Residual case documented: a restored volume tar
|
||||||
|
carrying the OLD internal credential hash may still need a manual in-DB reset.
|
||||||
|
|
||||||
|
Tests: +12 (api snapshots ×3, backup restore-points ×4, volume-dump gating ×3, secret-gen ×2 files);
|
||||||
|
all three fixes companion-red-proofed (hollow `[]` endpoint / removed volume gate / no-generation
|
||||||
|
each fail their test). Full `go build && go vet && go test ./...` green.
|
||||||
|
|
||||||
### docs — CLAUDE.md refresh: slim-down to stable orientation (2026-07-03)
|
### docs — CLAUDE.md refresh: slim-down to stable orientation (2026-07-03)
|
||||||
|
|
||||||
No code change, no version bump. CLAUDE.md 338 → ~160 lines: full 30-package layout map (was 7);
|
No code change, no version bump. CLAUDE.md 338 → ~160 lines: full 30-package layout map (was 7);
|
||||||
|
|||||||
+9
-1
@@ -9,8 +9,16 @@
|
|||||||
|
|
||||||
Last updated: 2026-07-03 (docs: CLAUDE.md refreshed — stable orientation; runbooks live in the felhom skills)
|
Last updated: 2026-07-03 (docs: CLAUDE.md refreshed — stable orientation; runbooks live in the felhom skills)
|
||||||
|
|
||||||
|
> **2026-07-05 — v0.99.0 restore-path fixes (TASK C1): drill findings F1/F3/O4 RESOLVED.**
|
||||||
|
> F1: `GET /api/backup/snapshots` implemented (`backup.ListRestorePoints`) — the restore panel
|
||||||
|
> populates and the restore button enables. F3: `runVolumeDumps` wired into the nightly/manual
|
||||||
|
> backup run (volume gate BEFORE DumpAppVolumesSafe; before unit capture). O4: unrecoverable
|
||||||
|
> resettable secrets get a generated replacement (`GenerateSecretForField` + `SetSecretGenerator`
|
||||||
|
> seam); data-key gate untouched. **F2 (one-click in-place class-C restore) remains OPEN → TASK C2.**
|
||||||
|
> O4 residual: restored volume tar with an OLD credential hash may still need a manual in-DB reset.
|
||||||
|
|
||||||
> **2026-07-04 — app-data restore drill** → see `felhom.eu/documentation/audits/DRILL-appdata-restore-2026-07-04.md`.
|
> **2026-07-04 — app-data restore drill** → see `felhom.eu/documentation/audits/DRILL-appdata-restore-2026-07-04.md`.
|
||||||
> Keep-side restore proven live on 9201 (class-A DB replay + fail-closed data-key gate + non-destruction). **F1 (HIGH): UI restore is dead — `/api/backup/snapshots` has no handler, so the restore button never enables.** F2: no one-click in-place class-C (HDD bind-mount) restore. F3: named-volume data never backed up (`DumpAppVolumes*` has no caller).
|
> Keep-side restore proven live on 9201 (class-A DB replay + fail-closed data-key gate + non-destruction). **F1 (HIGH): UI restore is dead — `/api/backup/snapshots` has no handler, so the restore button never enables.** F2: no one-click in-place class-C (HDD bind-mount) restore. F3: named-volume data never backed up (`DumpAppVolumes*` has no caller). *(F1/F3/O4 resolved in v0.99.0, see above.)*
|
||||||
|
|
||||||
> **2026-07-03 — CLAUDE.md slimmed to stable orientation** (full package map, verified 9201 deploy
|
> **2026-07-03 — CLAUDE.md slimmed to stable orientation** (full package map, verified 9201 deploy
|
||||||
> summary, no version-pinned state). Deep runbooks/design/testing doctrine now in the personal
|
> summary, no version-pinned state). Deep runbooks/design/testing doctrine now in the personal
|
||||||
|
|||||||
@@ -67,7 +67,8 @@
|
|||||||
| `resolveContainerState` / `aggregateState` | controller/internal/stacks/manager.go | `(dockerState, dockerStatus)` / `([]ContainerInfo)` | State classification | `.State` says "running" even when unhealthy — `.Status` parse is the fix |
|
| `resolveContainerState` / `aggregateState` | controller/internal/stacks/manager.go | `(dockerState, dockerStatus)` / `([]ContainerInfo)` | State classification | `.State` says "running" even when unhealthy — `.Status` parse is the fix |
|
||||||
| `Manager.logPostStartStatus` | controller/internal/stacks/manager.go | `(name, stackDir, env)` | Async post-start verification | compose up exits 0 on crash-loops; this is the detection. Goroutine + 3s, never blocks |
|
| `Manager.logPostStartStatus` | controller/internal/stacks/manager.go | `(name, stackDir, env)` | Async post-start verification | compose up exits 0 on crash-loops; this is the detection. Goroutine + 3s, never blocks |
|
||||||
| `Manager.EnsureBaseStack` | controller/internal/stacks/infra.go | `() error` | Traefik/cloudflared/FileBrowser infra convergence | Renders from `internal/infra` templates |
|
| `Manager.EnsureBaseStack` | controller/internal/stacks/infra.go | `() error` | Traefik/cloudflared/FileBrowser infra convergence | Renders from `internal/infra` templates |
|
||||||
| `backup.Manager.DumpAppVolumesSafe` | controller/internal/backup/backup.go | `(stackName) error` | Volume tar of a live app | Stops → dumps → restarts; surfaces BOTH errors (app may be left stopped) |
|
| `backup.Manager.DumpAppVolumesSafe` | controller/internal/backup/backup.go | `(stackName) error` | Volume tar of a live app | Stops → dumps → restarts; surfaces BOTH errors (app may be left stopped). Check `GetDockerVolumes()!=0` + `IsProtectedStack` BEFORE calling — it stops the stack before its own volume check (see `runVolumeDumps`) |
|
||||||
|
| `backup.Manager.ListRestorePoints` | controller/internal/backup/restore_points.go | `(stackName) ([]RestorePoint, bool)` | Restorable keep-side backups (the /api/backup/snapshots payload) | ONE point per app (the current unit); tier always 1 — never list Tier-2 (not restorable via /backup/restore) |
|
||||||
| `Manager.acquireRunning`/`releaseRunning`, `acquireMigrating` | controller/internal/backup/backup.go, controller/internal/stacks/migrate.go | `() error` | Single-flight for long ops | Copy this mutex-flag pattern for any new long-running manager op |
|
| `Manager.acquireRunning`/`releaseRunning`, `acquireMigrating` | controller/internal/backup/backup.go, controller/internal/stacks/migrate.go | `() error` | Single-flight for long ops | Copy this mutex-flag pattern for any new long-running manager op |
|
||||||
|
|
||||||
### Secrets hygiene
|
### Secrets hygiene
|
||||||
@@ -78,6 +79,7 @@
|
|||||||
| `crypto.LoadOrCreateKey` | controller/internal/crypto/crypto.go | `(path) ([]byte, error)` | The 32-byte key file (0600) | — |
|
| `crypto.LoadOrCreateKey` | controller/internal/crypto/crypto.go | `(path) ([]byte, error)` | The 32-byte key file (0600) | — |
|
||||||
| `SaveAppConfig` / `LoadAppConfigDecrypted` | controller/internal/stacks/deploy.go | `(stackDir, cfg, encKey, sensitiveVars)` | app.yaml persistence | Encrypts only `SensitiveEnvVars(meta)`; never write app.yaml directly |
|
| `SaveAppConfig` / `LoadAppConfigDecrypted` | controller/internal/stacks/deploy.go | `(stackDir, cfg, encKey, sensitiveVars)` | app.yaml persistence | Encrypts only `SensitiveEnvVars(meta)`; never write app.yaml directly |
|
||||||
| `generateValue` / `randomAlphanumeric` | controller/internal/stacks/deploy.go | `(spec "password:N\|hex:N\|base64key:N\|static:v")` | Auto-generated secrets | crypto/rand-backed; reuse the spec grammar |
|
| `generateValue` / `randomAlphanumeric` | controller/internal/stacks/deploy.go | `(spec "password:N\|hex:N\|base64key:N\|static:v")` | Auto-generated secrets | crypto/rand-backed; reuse the spec grammar |
|
||||||
|
| `Manager.GenerateSecretForField` | controller/internal/stacks/deploy.go | `(stackName, envVar) (string, bool)` | Replacement value for a RESETTABLE secret from its catalog `generate` spec (O4 restore path via `backup.SetSecretGenerator`) | REFUSES `data_key` fields, spec-less and non-secret fields; never log the value |
|
||||||
| `reconcileRestoreSecrets` | controller/internal/backup/restore_unit.go | `(nonSecretEnv, recoveredSecrets, secretNames, dataKeyNames)` | Recovery-unit restore env merge | Units are secret-FREE by design; secrets come from live app.yaml |
|
| `reconcileRestoreSecrets` | controller/internal/backup/restore_unit.go | `(nonSecretEnv, recoveredSecrets, secretNames, dataKeyNames)` | Recovery-unit restore env merge | Units are secret-FREE by design; secrets come from live app.yaml |
|
||||||
| `EncryptFile` / `DecryptFile` / `IsEncryptedFAB` | controller/internal/appexport/crypto.go | password-based file crypto | .fab export bundles | scrypt-derived AES+HMAC keys |
|
| `EncryptFile` / `DecryptFile` / `IsEncryptedFAB` | controller/internal/appexport/crypto.go | password-based file crypto | .fab export bundles | scrypt-derived AES+HMAC keys |
|
||||||
| `maskRepoURL` | controller/internal/sync/sync.go | `(url) string` | Logging git URLs | Strips embedded credentials |
|
| `maskRepoURL` | controller/internal/sync/sync.go | `(url) string` | Logging git URLs | Strips embedded credentials |
|
||||||
@@ -174,6 +176,8 @@
|
|||||||
| `integrations.Handler` + `StackProvider` | controller/internal/integrations/integrations.go + manager.go | OnlyOffice handlers | table-driven tests in package |
|
| `integrations.Handler` + `StackProvider` | controller/internal/integrations/integrations.go + manager.go | OnlyOffice handlers | table-driven tests in package |
|
||||||
| `bootstrap.PullFunc` | controller/internal/bootstrap/bootstrap.go | `report.PullConfig` | injected in bootstrap tests |
|
| `bootstrap.PullFunc` | controller/internal/bootstrap/bootstrap.go | `report.PullConfig` | injected in bootstrap tests |
|
||||||
| `offboxRunner` (func) | controller/internal/backup/offbox.go | `defaultOffboxRunner` (restic exec) | `SetOffboxRunner` injection point |
|
| `offboxRunner` (func) | controller/internal/backup/offbox.go | `defaultOffboxRunner` (restic exec) | `SetOffboxRunner` injection point |
|
||||||
|
| `dumpVolumesSafe` (func seam) | controller/internal/backup/backup.go | nil → real `DumpAppVolumesSafe` | injected in controller/internal/backup/volume_dumps_test.go (gating tests without Docker) |
|
||||||
|
| `generateSecret` (func seam) | controller/internal/backup/backup.go | `stacks.Manager.GenerateSecretForField` via `SetSecretGenerator` (main.go) | injected in controller/internal/backup/restore_secrets_gen_test.go |
|
||||||
|
|
||||||
Cross-repo edges:
|
Cross-repo edges:
|
||||||
- `controller/internal/agentapi/client.go` ↔ **felhom-agent** local API (`/storage`, `/disks*`, `/backup*`, `/netstorage*`, `/guest/*`): pinned leaf SHA-256 + per-guest bearer token from bootstrap.json.
|
- `controller/internal/agentapi/client.go` ↔ **felhom-agent** local API (`/storage`, `/disks*`, `/backup*`, `/netstorage*`, `/guest/*`): pinned leaf SHA-256 + per-guest bearer token from bootstrap.json.
|
||||||
@@ -205,6 +209,6 @@ Cross-repo edges:
|
|||||||
| CSRF ×2 | controller/internal/web/csrf.go (session HMAC) vs controller/internal/setup/csrf.go (cookie double-submit) — intentional (pre-auth wizard) but unlabeled |
|
| CSRF ×2 | controller/internal/web/csrf.go (session HMAC) vs controller/internal/setup/csrf.go (cookie double-submit) — intentional (pre-auth wizard) but unlabeled |
|
||||||
| Budapest timezone loader ×2 | controller/internal/scheduler/scheduler.go `getBudapestLocation` vs controller/internal/web/funcmap.go `getTimezone` |
|
| Budapest timezone loader ×2 | controller/internal/scheduler/scheduler.go `getBudapestLocation` vs controller/internal/web/funcmap.go `getTimezone` |
|
||||||
| JSON writers ×5, 3 envelope shapes | api `writeJSON`; web `writeDiskJSON`, `jsonResponse`/`jsonError`, `writeDebugJSON` |
|
| JSON writers ×5, 3 envelope shapes | api `writeJSON`; web `writeDiskJSON`, `jsonResponse`/`jsonError`, `writeDebugJSON` |
|
||||||
| Safe-name validators ×3 | controller/internal/web/validate.go `validStackName`; controller/internal/backup/offbox.go `isSafeStackName`; controller/internal/appexport/validate.go `ValidateSegment` (strictest) |
|
| Safe-name validators ×4 | controller/internal/web/validate.go `validStackName`; controller/internal/api/router.go `validStackParam` (same body — api↔web import cycle); controller/internal/backup/offbox.go `isSafeStackName`; controller/internal/appexport/validate.go `ValidateSegment` (strictest) |
|
||||||
| DB wait/import ×2 | controller/internal/appbackup/dbdump.go `waitDBReady`/`ImportDump` vs controller/internal/appexport/restore.go `waitForDB`/`importDBDump` |
|
| DB wait/import ×2 | controller/internal/appbackup/dbdump.go `waitDBReady`/`ImportDump` vs controller/internal/appexport/restore.go `waitForDB`/`importDBDump` |
|
||||||
| compose exec ×2 | controller/internal/stacks/manager.go `composeExecCustomEnv` vs controller/internal/appexport/restore.go `composeExecEnv` (the latter has ctx+timeout; the former has the userdata belt) |
|
| compose exec ×2 | controller/internal/stacks/manager.go `composeExecCustomEnv` vs controller/internal/appexport/restore.go `composeExecEnv` (the latter has ctx+timeout; the former has the userdata belt) |
|
||||||
|
|||||||
+32
-11
@@ -441,8 +441,14 @@ backups/primary/<app>/
|
|||||||
Docker image** — only the pinned image tag(s) (re-pulled on restore) and the *names* of the secret /
|
Docker image** — only the pinned image tag(s) (re-pulled on restore) and the *names* of the secret /
|
||||||
`data_key` env vars. Rationale: app.yaml + the encryption key live on the guest rootfs → already in
|
`data_key` env vars. Rationale: app.yaml + the encryption key live on the guest rootfs → already in
|
||||||
the PBS whole-guest snapshot, and the hub is deliberately zero-knowledge. Restore recovers the
|
the PBS whole-guest snapshot, and the hub is deliberately zero-knowledge. Restore recovers the
|
||||||
original secrets from the guest's own app.yaml (live, or via PBS) and **regenerates nothing**; for a
|
original secrets from the guest's own app.yaml (live, or via PBS); for a `data_key` app it
|
||||||
`data_key` app it **fails closed** (refuse + warn) if the key can't be recovered.
|
**fails closed** (refuse + warn) if the key can't be recovered — data-keys are NEVER generated.
|
||||||
|
**Resettable secrets (O4, v0.99.0):** an unrecoverable resettable secret (DB password etc.) gets a
|
||||||
|
**generated replacement** from its catalog `generate` spec (`stacks.GenerateSecretForField` via the
|
||||||
|
`backup.SetSecretGenerator` seam) instead of redeploying blank (which failed compose-up); the new
|
||||||
|
value persists encrypted through the normal `RecreateStackFromUnit` → `SaveAppConfig` path. Fields
|
||||||
|
with no `generate` spec still proceed with a loud "may fail to start" WARN. Residual case: a restored
|
||||||
|
volume tar carrying the OLD internal credential hash may still need a manual in-DB reset.
|
||||||
- Helpers: `RecoveryUnitPath` / `RecoveryUnitComposePath` / `RecoveryUnitManifestPath`
|
- Helpers: `RecoveryUnitPath` / `RecoveryUnitComposePath` / `RecoveryUnitManifestPath`
|
||||||
(`internal/appbackup/paths.go`). Capture: `Manager.CaptureRecoveryUnit` (`internal/backup/recovery_unit.go`),
|
(`internal/appbackup/paths.go`). Capture: `Manager.CaptureRecoveryUnit` (`internal/backup/recovery_unit.go`),
|
||||||
run from the daily DB dump and the periodic `RefreshCache` (idempotent checksum-skip). The non-secret
|
run from the daily DB dump and the periodic `RefreshCache` (idempotent checksum-skip). The non-secret
|
||||||
@@ -492,15 +498,23 @@ re-validates the pin off-disk at run time. `Manager.Tier2Info(stackName)` is the
|
|||||||
- **Validation** after each dump: checks file size, header presence, counts `CREATE TABLE`
|
- **Validation** after each dump: checks file size, header presence, counts `CREATE TABLE`
|
||||||
- Results cached in `settings.json` surviving container restarts
|
- Results cached in `settings.json` surviving container restarts
|
||||||
|
|
||||||
**Phase 1b — Docker Volume Dumps** (`internal/backup/backup.go`, runs after DB dumps)
|
**Phase 1b — Docker Volume Dumps** (`internal/backup/backup.go` `runVolumeDumps`, part of the same run)
|
||||||
|
|
||||||
- Iterates all deployed stacks that have Docker named volumes (`GetDockerVolumes()`)
|
- **F3 (v0.99.0):** re-wired into the nightly/manual app-data backup run (`runDBDumpsInternal`) —
|
||||||
- **v0.34.0:** Each stack is stopped before dump, restarted after (`DumpAppVolumesSafe()`) — prevents inconsistent tars of live databases. Protected stacks (traefik, etc.) that reject StopStack are skipped with a warning.
|
after the restic removal `DumpAppVolumesSafe` had no caller, so `volume-dumps/` was never produced.
|
||||||
|
Runs AFTER the DB dumps and BEFORE `captureAllRecoveryUnits` so the manifests enumerate fresh tars.
|
||||||
|
- Gate order (load-bearing): protected-stack (`cfg.IsProtectedStack`) and has-volumes
|
||||||
|
(`GetDockerVolumes()`) checks come BEFORE `DumpAppVolumesSafe` — the Safe variant stops the stack
|
||||||
|
before its own volume check, so unconditional calls would bounce every volume-less app nightly.
|
||||||
|
Disconnected/decommissioned drives skip with the same summary style as the DB loop.
|
||||||
|
- Each volume-bearing stack is stopped before dump, restarted after (`DumpAppVolumesSafe()`) —
|
||||||
|
prevents inconsistent tars of live databases.
|
||||||
- For each volume: `docker run --rm -v <vol>:/vol:ro -v <dumpDir>:/out alpine tar cf /out/<vol>.tar -C /vol .`
|
- For each volume: `docker run --rm -v <vol>:/vol:ro -v <dumpDir>:/out alpine tar cf /out/<vol>.tar -C /vol .`
|
||||||
- 10-minute timeout per volume; warnings on failure (non-fatal)
|
- 10-minute timeout per volume; a per-stack failure lands in the run summary as `FAIL <app> volumes:`,
|
||||||
|
flips the run's Success flag and fails the run (no silent partial) — other stacks still proceed
|
||||||
- Stale tars cleaned up (volumes that no longer exist)
|
- Stale tars cleaned up (volumes that no longer exist)
|
||||||
- Volume names resolved with project prefix via `ResolveDockerVolumeNames()` (e.g., `mealie_mealie_data`)
|
- Volume names resolved with project prefix via `ResolveDockerVolumeNames()` (e.g., `mealie_mealie_data`)
|
||||||
- Dumps written to `AppVolumeDumpPath(appDrive, stackName)`
|
- Dumps written to `AppVolumeDumpPath(nsRoot, stackName)`
|
||||||
|
|
||||||
**Phase 2 — Restic Snapshot** (`internal/backup/restic.go`, scheduled 03:00)
|
**Phase 2 — Restic Snapshot** (`internal/backup/restic.go`, scheduled 03:00)
|
||||||
|
|
||||||
@@ -589,10 +603,17 @@ appear in the restore dropdown with per-app snapshot filtering.
|
|||||||
| DB only, no HDD/volumes | Yes | Yes | n/a | n/a |
|
| DB only, no HDD/volumes | Yes | Yes | n/a | n/a |
|
||||||
| Config only | Yes | — | n/a | n/a |
|
| Config only | Yes | — | n/a | n/a |
|
||||||
|
|
||||||
**Snapshot API** (`/api/backup/snapshots?stack=<name>`):
|
**Snapshot API** (`GET /api/backup/snapshots?stack=<name>` — F1, v0.99.0):
|
||||||
- Returns snapshots **only from the app's home drive** primary repo (prevents showing irrelevant snapshots from other drives)
|
- Backed by `backup.Manager.ListRestorePoints` (`internal/backup/restore_points.go`). The keep-side
|
||||||
- Appends a synthetic Tier 2 entry (ID `tier2-rsync`) from cross-drive config when last backup was successful
|
restore has exactly **one** restore point per app — the current recovery unit — so the endpoint
|
||||||
- Dropdown groups by tier: "1. szint — Helyi mentes" and "2. szint — Masodlagos masolat"
|
returns at most one entry: `time` = newest artifact mtime (manifest / db-dumps / volume-dumps),
|
||||||
|
`short_id:"helyi"`, `tier:1`, `drive_label` from the storage registry (empty on the SSD fallback)
|
||||||
|
- **Never emits tier-2 entries**: Tier-2 copies are not restorable via `POST /backup/restore` (it
|
||||||
|
only reads the primary unit) — listing them would silently restore tier-1 data while claiming tier-2
|
||||||
|
- Guards: empty/traversal stack name → 400 (`validStackParam`), unknown stack → 404, known stack
|
||||||
|
with no unit yet → `ok:true, data:[]` (the UI shows "Nincs elérhető mentés")
|
||||||
|
- History: the route was a restic-era leftover fetched by the template but unregistered — the
|
||||||
|
dropdown could never populate and the restore button never enabled (drill finding F1)
|
||||||
|
|
||||||
**Restore type info** shown per-app when selected in dropdown (Hungarian banners):
|
**Restore type info** shown per-app when selected in dropdown (Hungarian banners):
|
||||||
- Has HDD or Docker volumes: "Teljes visszaallitas: adatbazis + konfiguracio + felhasznaloi adatok"
|
- Has HDD or Docker volumes: "Teljes visszaallitas: adatbazis + konfiguracio + felhasznaloi adatok"
|
||||||
|
|||||||
@@ -226,6 +226,9 @@ func main() {
|
|||||||
backupMgr = backup.NewManager(cfg, sett, logger)
|
backupMgr = backup.NewManager(cfg, sett, logger)
|
||||||
backupMgr.SetStackProvider(stackProv)
|
backupMgr.SetStackProvider(stackProv)
|
||||||
backupMgr.SetVersion(Version)
|
backupMgr.SetVersion(Version)
|
||||||
|
// O4: restore-from-unit generates a replacement for an unrecoverable RESETTABLE secret
|
||||||
|
// (data-keys stay fail-closed) so the app redeploys with a fresh credential, not a blank one.
|
||||||
|
backupMgr.SetSecretGenerator(stackMgr.GenerateSecretForField)
|
||||||
}
|
}
|
||||||
|
|
||||||
// --- Wire the data-migration engine (B1) + backup↔migration mutual exclusion (Change 3) ---
|
// --- Wire the data-migration engine (B1) + backup↔migration mutual exclusion (Change 3) ---
|
||||||
|
|||||||
@@ -0,0 +1,144 @@
|
|||||||
|
package api
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// snapshotsStubProvider implements backup.StackDataProvider for the F1 endpoint tests: one known
|
||||||
|
// stack ("app") living on `hdd`. Everything else is inert.
|
||||||
|
type snapshotsStubProvider struct{ hdd string }
|
||||||
|
|
||||||
|
func (p *snapshotsStubProvider) GetStackComposePath(name string) (string, bool) {
|
||||||
|
if name == "app" {
|
||||||
|
return filepath.Join(p.hdd, "compose", "docker-compose.yml"), true
|
||||||
|
}
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
func (p *snapshotsStubProvider) ListDeployedStacks() []backup.StackSummary { return nil }
|
||||||
|
func (p *snapshotsStubProvider) GetStackHDDMounts(string) []string { return nil }
|
||||||
|
func (p *snapshotsStubProvider) GetStackHDDPath(string) string { return p.hdd }
|
||||||
|
func (p *snapshotsStubProvider) GetDockerVolumes(string) []string { return nil }
|
||||||
|
func (p *snapshotsStubProvider) StopStack(string) error { return nil }
|
||||||
|
func (p *snapshotsStubProvider) StartStack(string) error { return nil }
|
||||||
|
func (p *snapshotsStubProvider) RefreshAndIsRunning(string) bool { return true }
|
||||||
|
func (p *snapshotsStubProvider) GetStackRecoveryInfo(string) (backup.RecoveryInfo, bool) {
|
||||||
|
return backup.RecoveryInfo{}, false
|
||||||
|
}
|
||||||
|
func (p *snapshotsStubProvider) RecoverStackSecrets(string, []string) map[string]string { return nil }
|
||||||
|
func (p *snapshotsStubProvider) RecreateStackFromUnit(string, string, map[string]string) error {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// newSnapshotsRouter wires a Router with a real backup.Manager over a tempdir drive.
|
||||||
|
func newSnapshotsRouter(t *testing.T) (*Router, string) {
|
||||||
|
t.Helper()
|
||||||
|
drive := filepath.Join(t.TempDir(), "drive")
|
||||||
|
cfg := &config.Config{}
|
||||||
|
cfg.Paths.SystemDataPath = filepath.Join(t.TempDir(), "sys")
|
||||||
|
mgr := backup.NewManager(cfg, nil, log.New(io.Discard, "", 0))
|
||||||
|
mgr.SetStackProvider(&snapshotsStubProvider{hdd: drive})
|
||||||
|
return &Router{cfg: cfg, backupMgr: mgr, logger: log.New(io.Discard, "", 0)}, drive
|
||||||
|
}
|
||||||
|
|
||||||
|
// getSnapshots dispatches through Router.ServeHTTP so the tests also prove the ROUTE is
|
||||||
|
// registered — the F1 bug was precisely a fetch to a route that did not exist.
|
||||||
|
func getSnapshots(t *testing.T, r *Router, rawQuery string) *httptest.ResponseRecorder {
|
||||||
|
t.Helper()
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/api/backup/snapshots?"+rawQuery, nil)
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
r.ServeHTTP(rec, req)
|
||||||
|
return rec
|
||||||
|
}
|
||||||
|
|
||||||
|
type snapshotsResp struct {
|
||||||
|
OK bool `json:"ok"`
|
||||||
|
Error string `json:"error"`
|
||||||
|
Data []struct {
|
||||||
|
Time string `json:"time"`
|
||||||
|
ShortID string `json:"short_id"`
|
||||||
|
Tier int `json:"tier"`
|
||||||
|
DriveLabel string `json:"drive_label"`
|
||||||
|
} `json:"data"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeSnapshots(t *testing.T, rec *httptest.ResponseRecorder) snapshotsResp {
|
||||||
|
t.Helper()
|
||||||
|
var v snapshotsResp
|
||||||
|
if err := json.Unmarshal(rec.Body.Bytes(), &v); err != nil {
|
||||||
|
t.Fatalf("decode %q: %v", rec.Body.String(), err)
|
||||||
|
}
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario A (data half): a recovery unit on disk → 200 with EXACTLY ONE tier-1 "helyi" entry the
|
||||||
|
// restore panel JS can render and POST back.
|
||||||
|
func TestBackupSnapshots_UnitOnDisk(t *testing.T) {
|
||||||
|
r, drive := newSnapshotsRouter(t)
|
||||||
|
manifest := backup.RecoveryUnitManifestPath(drive, "app")
|
||||||
|
if err := os.MkdirAll(filepath.Dir(manifest), 0755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(manifest, []byte("{}"), 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
rec := getSnapshots(t, r, "stack=app")
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
v := decodeSnapshots(t, rec)
|
||||||
|
if !v.OK || len(v.Data) != 1 {
|
||||||
|
t.Fatalf("want ok with exactly 1 entry, got %+v", v)
|
||||||
|
}
|
||||||
|
if v.Data[0].Tier != 1 || v.Data[0].ShortID != "helyi" || v.Data[0].Time == "" {
|
||||||
|
t.Errorf("entry = %+v (want tier 1, short_id helyi, non-empty time)", v.Data[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario B: known stack, no recovery unit yet → ok:true with an EMPTY data list (the JS shows
|
||||||
|
// "Nincs elérhető mentés" and keeps the button disabled — reached honestly, not via a 404).
|
||||||
|
func TestBackupSnapshots_NoBackupYet(t *testing.T) {
|
||||||
|
r, _ := newSnapshotsRouter(t)
|
||||||
|
rec := getSnapshots(t, r, "stack=app")
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
v := decodeSnapshots(t, rec)
|
||||||
|
if !v.OK || len(v.Data) != 0 {
|
||||||
|
t.Errorf("want ok with empty list, got %+v", v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario C: guards — traversal and empty names are 400 BEFORE any filesystem work; an unknown
|
||||||
|
// stack is 404; a Router without a backup manager is 400.
|
||||||
|
func TestBackupSnapshots_Guards(t *testing.T) {
|
||||||
|
r, _ := newSnapshotsRouter(t)
|
||||||
|
|
||||||
|
for _, q := range []string{"stack=../../etc", "stack=", "stack=a/b", "stack=.."} {
|
||||||
|
rec := getSnapshots(t, r, q)
|
||||||
|
if rec.Code != http.StatusBadRequest {
|
||||||
|
t.Errorf("%q: status = %d, want 400; body=%s", q, rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
rec := getSnapshots(t, r, "stack=ghost")
|
||||||
|
if rec.Code != http.StatusNotFound {
|
||||||
|
t.Errorf("unknown stack: status = %d, want 404; body=%s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
noMgr := &Router{cfg: &config.Config{}, logger: log.New(io.Discard, "", 0)}
|
||||||
|
rec = getSnapshots(t, noMgr, "stack=app")
|
||||||
|
if rec.Code != http.StatusBadRequest {
|
||||||
|
t.Errorf("nil backupMgr: status = %d, want 400", rec.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -252,6 +252,10 @@ func (r *Router) ServeHTTP(w http.ResponseWriter, req *http.Request) {
|
|||||||
case path == "/backup/status" && req.Method == http.MethodGet:
|
case path == "/backup/status" && req.Method == http.MethodGet:
|
||||||
r.backupStatus(w, req)
|
r.backupStatus(w, req)
|
||||||
|
|
||||||
|
// GET /api/backup/snapshots?stack=<name> — restorable keep-side backups for the restore panel
|
||||||
|
case path == "/backup/snapshots" && req.Method == http.MethodGet:
|
||||||
|
r.backupSnapshots(w, req)
|
||||||
|
|
||||||
// POST /api/backup/run
|
// POST /api/backup/run
|
||||||
case path == "/backup/run" && req.Method == http.MethodPost:
|
case path == "/backup/run" && req.Method == http.MethodPost:
|
||||||
r.triggerBackup(w, req)
|
r.triggerBackup(w, req)
|
||||||
@@ -831,6 +835,44 @@ func (r *Router) backupStatus(w http.ResponseWriter, _ *http.Request) {
|
|||||||
writeJSON(w, http.StatusOK, apiResponse{OK: true, Data: data})
|
writeJSON(w, http.StatusOK, apiResponse{OK: true, Data: data})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// validStackParam reports whether a stack name from a request is a safe single path segment
|
||||||
|
// (same semantics as web.validStackName — see internal/web/validate.go; duplicated here because
|
||||||
|
// api ↔ web would be a circular import). Rejects traversal/escape so the name can never become
|
||||||
|
// a filesystem path outside the app's namespace root.
|
||||||
|
func validStackParam(name string) bool {
|
||||||
|
if name == "" || name == "." || name == ".." {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if strings.ContainsAny(name, "/\\\x00") {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return name == filepath.Clean(name)
|
||||||
|
}
|
||||||
|
|
||||||
|
// backupSnapshots lists the restorable keep-side backups for one app — the data source of the
|
||||||
|
// /backups restore panel's snapshot dropdown (F1: this route was fetched by the template but never
|
||||||
|
// existed, so the dropdown could never populate and the restore button never enabled).
|
||||||
|
func (r *Router) backupSnapshots(w http.ResponseWriter, req *http.Request) {
|
||||||
|
stack := req.URL.Query().Get("stack")
|
||||||
|
if !validStackParam(stack) {
|
||||||
|
r.dbg("backupSnapshots: invalid stack param %q", stack)
|
||||||
|
writeJSON(w, http.StatusBadRequest, apiResponse{OK: false, Error: "invalid stack name"})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if r.backupMgr == nil {
|
||||||
|
writeJSON(w, http.StatusBadRequest, apiResponse{OK: false, Error: "Backup not configured"})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
points, found := r.backupMgr.ListRestorePoints(stack)
|
||||||
|
if !found {
|
||||||
|
writeJSON(w, http.StatusNotFound, apiResponse{OK: false, Error: "stack not found: " + stack})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
r.dbg("backupSnapshots: stack=%s points=%d", stack, len(points))
|
||||||
|
writeJSON(w, http.StatusOK, apiResponse{OK: true, Data: points})
|
||||||
|
}
|
||||||
|
|
||||||
// triggerBackup runs the app-data database dumps. Disk-tier (restic) backup has
|
// triggerBackup runs the app-data database dumps. Disk-tier (restic) backup has
|
||||||
// moved to the host agent (slice 8C).
|
// moved to the host agent (slice 8C).
|
||||||
func (r *Router) triggerBackup(w http.ResponseWriter, _ *http.Request) {
|
func (r *Router) triggerBackup(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
|||||||
@@ -40,6 +40,16 @@ type Manager struct {
|
|||||||
discoverDBs func(ctx context.Context) ([]DiscoveredDB, error)
|
discoverDBs func(ctx context.Context) ([]DiscoveredDB, error)
|
||||||
importDBDump func(ctx context.Context, db DiscoveredDB, dumpPath string) error
|
importDBDump func(ctx context.Context, db DiscoveredDB, dumpPath string) error
|
||||||
|
|
||||||
|
// F3 volume-dump seam — overridable in tests so runVolumeDumps' gating (protected / volume-less /
|
||||||
|
// disconnected) can be unit-tested without Docker. Nil → the real DumpAppVolumesSafe.
|
||||||
|
dumpVolumesSafe func(stackName string) error
|
||||||
|
|
||||||
|
// generateSecret (O4), if set, produces a replacement value for a RESETTABLE secret that could
|
||||||
|
// not be recovered during restore-from-unit (wired to stacks.Manager.GenerateSecretForField in
|
||||||
|
// main.go). Nil / ok=false → the secret stays absent and the restore proceeds with a loud WARN.
|
||||||
|
// NEVER consulted for data-keys — the fail-closed gate refuses those before generation runs.
|
||||||
|
generateSecret func(stackName, envVar string) (string, bool)
|
||||||
|
|
||||||
// migrationRunning, if set, reports whether a data migration is in progress. The scheduled
|
// migrationRunning, if set, reports whether a data migration is in progress. The scheduled
|
||||||
// backup paths skip when it returns true (Change 3 — backup ↔ migration mutual exclusion), so a
|
// backup paths skip when it returns true (Change 3 — backup ↔ migration mutual exclusion), so a
|
||||||
// nightly dump/Tier-2 can't race a migration copy/cleanup on the same drive.
|
// nightly dump/Tier-2 can't race a migration copy/cleanup on the same drive.
|
||||||
@@ -202,20 +212,14 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// F3: no early return on zero DBs — volume-bearing apps without a database still need their
|
||||||
|
// class-B volume dump + recovery-unit refresh below (the DB loop simply has no iterations).
|
||||||
if len(dbs) == 0 {
|
if len(dbs) == 0 {
|
||||||
m.logger.Printf("[INFO] [backup] No database containers found")
|
m.logger.Printf("[INFO] [backup] No database containers found")
|
||||||
m.mu.Lock()
|
} else {
|
||||||
m.lastDBDump = &DBDumpStatus{
|
m.logger.Printf("[INFO] [backup] Discovered %d database(s): %s", len(dbs), dbNames(dbs))
|
||||||
LastRun: time.Now(),
|
|
||||||
Success: true,
|
|
||||||
Duration: time.Since(start),
|
|
||||||
}
|
|
||||||
m.mu.Unlock()
|
|
||||||
return nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
m.logger.Printf("[INFO] [backup] Discovered %d database(s): %s", len(dbs), dbNames(dbs))
|
|
||||||
|
|
||||||
// Dump each DB to its app's drive path
|
// Dump each DB to its app's drive path
|
||||||
var results []DumpResult
|
var results []DumpResult
|
||||||
allOK := true
|
allOK := true
|
||||||
@@ -270,6 +274,13 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// F3: class-B leg — dump each app's named-volume data (stop → tar → restart). MUST run before
|
||||||
|
// captureAllRecoveryUnits so the manifests enumerate the fresh tars into VolumeDumps.
|
||||||
|
dbOK := allOK
|
||||||
|
volSummary, volDumped, volOK := m.runVolumeDumps()
|
||||||
|
summary = append(summary, volSummary...)
|
||||||
|
allOK = dbOK && volOK
|
||||||
|
|
||||||
duration := time.Since(start)
|
duration := time.Since(start)
|
||||||
m.mu.Lock()
|
m.mu.Lock()
|
||||||
m.lastDBDump = &DBDumpStatus{
|
m.lastDBDump = &DBDumpStatus{
|
||||||
@@ -281,22 +292,91 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
|
|||||||
m.mu.Unlock()
|
m.mu.Unlock()
|
||||||
|
|
||||||
if allOK {
|
if allOK {
|
||||||
m.logger.Printf("[INFO] [backup] DB dump completed: %d databases, %s total (%s)",
|
m.logger.Printf("[INFO] [backup] App-data backup completed: %d databases (%s total), %d volume dump(s) (%s)",
|
||||||
len(results), humanizeBytes(totalSize), duration.Round(time.Millisecond))
|
len(results), humanizeBytes(totalSize), volDumped, duration.Round(time.Millisecond))
|
||||||
} else {
|
} else {
|
||||||
// Still refresh recovery units below — a partial DB failure shouldn't leave units stale.
|
// Still refresh recovery units below — a partial failure shouldn't leave units stale.
|
||||||
m.logger.Printf("[WARN] [backup] some database dumps failed; refreshing recovery units anyway")
|
m.logger.Printf("[WARN] [backup] some backup steps failed (%s); refreshing recovery units anyway",
|
||||||
|
strings.Join(failedSummaryLines(summary), "; "))
|
||||||
}
|
}
|
||||||
|
|
||||||
// Phase 2: refresh each deployed app's self-contained recovery unit (compose + manifest).
|
// Phase 2: refresh each deployed app's self-contained recovery unit (compose + manifest).
|
||||||
m.captureAllRecoveryUnits()
|
m.captureAllRecoveryUnits()
|
||||||
|
|
||||||
|
// No silent partials: a DB-dump or volume-dump failure fails the whole run.
|
||||||
if !allOK {
|
if !allOK {
|
||||||
return fmt.Errorf("some database dumps failed")
|
return fmt.Errorf("some backup steps failed: %s", strings.Join(failedSummaryLines(summary), "; "))
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// failedSummaryLines filters a run summary down to its FAIL entries (for logs/errors).
|
||||||
|
func failedSummaryLines(summary []string) []string {
|
||||||
|
var failed []string
|
||||||
|
for _, s := range summary {
|
||||||
|
if strings.HasPrefix(s, "FAIL ") {
|
||||||
|
failed = append(failed, s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return failed
|
||||||
|
}
|
||||||
|
|
||||||
|
// runVolumeDumps exports the Docker named-volume data of every deployed, unprotected stack whose
|
||||||
|
// drive is writable — the class-B leg of the nightly app-data backup. (F3: DumpAppVolumesSafe
|
||||||
|
// previously had NO production caller, so volume-dumps/ was never produced and the granular
|
||||||
|
// restore had nothing to restore for named-volume apps.) Caller must hold the running flag.
|
||||||
|
//
|
||||||
|
// Gate ORDER is load-bearing: the volume check precedes DumpAppVolumesSafe, because the Safe
|
||||||
|
// variant stops the stack before its own volume check — calling it unconditionally would bounce
|
||||||
|
// every volume-less app on every nightly run. Per-stack isolation mirrors the DB loop: one app's
|
||||||
|
// failure is recorded and does not abort the others.
|
||||||
|
func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) {
|
||||||
|
allOK = true
|
||||||
|
if m.stackProvider == nil {
|
||||||
|
return nil, 0, true
|
||||||
|
}
|
||||||
|
dump := m.dumpVolumesSafe
|
||||||
|
if dump == nil {
|
||||||
|
dump = m.DumpAppVolumesSafe
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, stack := range m.stackProvider.ListDeployedStacks() {
|
||||||
|
// Never stop/dump infra stacks (felhom-controller, traefik, cloudflared).
|
||||||
|
if m.cfg != nil && m.cfg.IsProtectedStack(stack.Name) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
// Volume check FIRST — a volume-less stack must not be stopped at all (see gate-order note).
|
||||||
|
if len(m.stackProvider.GetDockerVolumes(stack.Name)) == 0 {
|
||||||
|
if m.isDebug() {
|
||||||
|
m.logger.Printf("[DEBUG] [backup] %s has no named volumes — volume dump skipped", stack.Name)
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
// Same drive-state skip guards as the DB-dump loop.
|
||||||
|
drivePath := m.GetAppDrivePath(stack.Name)
|
||||||
|
if m.settings != nil && m.settings.IsDisconnected(drivePath) {
|
||||||
|
m.logger.Printf("[WARN] [backup] Skipping volume dump for %s — drive disconnected: %s", stack.Name, drivePath)
|
||||||
|
summary = append(summary, fmt.Sprintf("SKIP %s volumes (drive disconnected)", stack.Name))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if m.settings != nil && m.settings.IsDecommissioned(drivePath) {
|
||||||
|
m.logger.Printf("[WARN] [backup] Skipping volume dump for %s — drive decommissioned: %s", stack.Name, drivePath)
|
||||||
|
summary = append(summary, fmt.Sprintf("SKIP %s volumes (drive decommissioned)", stack.Name))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := dump(stack.Name); err != nil {
|
||||||
|
allOK = false
|
||||||
|
summary = append(summary, fmt.Sprintf("FAIL %s volumes: %v", stack.Name, err))
|
||||||
|
m.logger.Printf("[ERROR] [backup] Volume dump failed for %s: %v", stack.Name, err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
dumped++
|
||||||
|
summary = append(summary, fmt.Sprintf("OK %s volumes", stack.Name))
|
||||||
|
}
|
||||||
|
return summary, dumped, allOK
|
||||||
|
}
|
||||||
|
|
||||||
// DumpAppVolumes exports Docker named volumes to tar files for the given stack.
|
// DumpAppVolumes exports Docker named volumes to tar files for the given stack.
|
||||||
// Tars are written to AppVolumeDumpPath(drivePath, stackName)/.
|
// Tars are written to AppVolumeDumpPath(drivePath, stackName)/.
|
||||||
// Uses "docker run alpine tar" (same pattern as appexport).
|
// Uses "docker run alpine tar" (same pattern as appexport).
|
||||||
@@ -432,6 +512,12 @@ func (m *Manager) releaseRunning() {
|
|||||||
m.mu.Unlock()
|
m.mu.Unlock()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SetSecretGenerator wires the O4 resettable-secret generator used by RestoreFromRecoveryUnit
|
||||||
|
// (init-only, same contract as SetStackProvider: call once during single-threaded startup).
|
||||||
|
func (m *Manager) SetSecretGenerator(fn func(stackName, envVar string) (string, bool)) {
|
||||||
|
m.generateSecret = fn
|
||||||
|
}
|
||||||
|
|
||||||
// SetStackProvider sets the stack data provider for app data discovery.
|
// SetStackProvider sets the stack data provider for app data discovery.
|
||||||
//
|
//
|
||||||
// M2: this MUST be called exactly once during single-threaded startup (main.go),
|
// M2: this MUST be called exactly once during single-threaded startup (main.go),
|
||||||
|
|||||||
@@ -0,0 +1,90 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// RestorePoint describes one restorable keep-side backup for the /backups restore panel
|
||||||
|
// (GET /api/backup/snapshots). The field names/shape are the payload contract of the
|
||||||
|
// backups.html restore JS (formatSnapshot): time / short_id / tier / drive_label.
|
||||||
|
//
|
||||||
|
// The keep-side restore has exactly ONE restore point per app — the current recovery unit
|
||||||
|
// (RestoreFromRecoveryUnit reads "the unit", not a history; snapshot_id is logging-only).
|
||||||
|
// Tier is always 1: Tier-2 copies are NOT restorable through POST /backup/restore (it only
|
||||||
|
// reads the app's primary unit), so listing them would silently restore tier-1 data while
|
||||||
|
// claiming tier-2 — never emit them here.
|
||||||
|
type RestorePoint struct {
|
||||||
|
Time string `json:"time"` // RFC3339 — newest artifact in the unit
|
||||||
|
ShortID string `json:"short_id"` // opaque label; POST /backup/restore uses it for logging only
|
||||||
|
Tier int `json:"tier"` // always 1 (see above)
|
||||||
|
DriveLabel string `json:"drive_label"` // registered storage label; empty for the SSD fallback
|
||||||
|
}
|
||||||
|
|
||||||
|
// restorePointShortID is the single keep-side restore point's identifier. Hungarian ("local"),
|
||||||
|
// because the JS renders it verbatim inside the snapshot dropdown label.
|
||||||
|
const restorePointShortID = "helyi"
|
||||||
|
|
||||||
|
// ListRestorePoints returns the app's restorable keep-side backups, and whether the stack is
|
||||||
|
// known at all (found=false → the caller should 404). A known stack with no recovery unit on
|
||||||
|
// disk returns an EMPTY list (a valid answer — "no backup yet"), not an error.
|
||||||
|
//
|
||||||
|
// The single point's Time is the newest mtime among the unit's artifacts (manifest.json,
|
||||||
|
// db-dumps/*.sql, volume-dumps/*.tar): the manifest is only rewritten when the app's config
|
||||||
|
// changes (checksum-skip), so the nightly-refreshed dumps are usually the freshest artifact.
|
||||||
|
func (m *Manager) ListRestorePoints(stackName string) (points []RestorePoint, found bool) {
|
||||||
|
if m.stackProvider == nil {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
if _, ok := m.stackProvider.GetStackComposePath(stackName); !ok {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
|
||||||
|
nsRoot := m.AppNamespaceRoot(stackName)
|
||||||
|
if nsRoot == "" || !filepath.IsAbs(nsRoot) {
|
||||||
|
// Stack is known but its backup location is unresolvable (e.g. systemDataPath unset in a
|
||||||
|
// misconfigured environment) — honest empty list rather than a path walk from "".
|
||||||
|
m.logger.Printf("[WARN] [backup] ListRestorePoints(%s): cannot resolve namespace root", stackName)
|
||||||
|
return []RestorePoint{}, true
|
||||||
|
}
|
||||||
|
|
||||||
|
fi, err := os.Stat(RecoveryUnitManifestPath(nsRoot, stackName))
|
||||||
|
if err != nil {
|
||||||
|
return []RestorePoint{}, true // no recovery unit yet — "no backup" is a valid answer
|
||||||
|
}
|
||||||
|
newest := fi.ModTime()
|
||||||
|
newest = newestArtifact(AppDBDumpPath(nsRoot, stackName), ".sql", newest)
|
||||||
|
newest = newestArtifact(AppVolumeDumpPath(nsRoot, stackName), ".tar", newest)
|
||||||
|
|
||||||
|
driveLabel := ""
|
||||||
|
if drive := m.GetAppDrivePath(stackName); drive != "" && drive != m.systemDataPath && m.settings != nil {
|
||||||
|
driveLabel = m.settings.GetStorageLabel(drive)
|
||||||
|
}
|
||||||
|
|
||||||
|
return []RestorePoint{{
|
||||||
|
Time: newest.UTC().Format(time.RFC3339),
|
||||||
|
ShortID: restorePointShortID,
|
||||||
|
Tier: 1,
|
||||||
|
DriveLabel: driveLabel,
|
||||||
|
}}, true
|
||||||
|
}
|
||||||
|
|
||||||
|
// newestArtifact returns the newest mtime among cur and the files with the given extension in
|
||||||
|
// dir (non-recursive; a missing dir contributes nothing).
|
||||||
|
func newestArtifact(dir, ext string, cur time.Time) time.Time {
|
||||||
|
entries, err := os.ReadDir(dir)
|
||||||
|
if err != nil {
|
||||||
|
return cur
|
||||||
|
}
|
||||||
|
for _, e := range entries {
|
||||||
|
if e.IsDir() || !strings.HasSuffix(e.Name(), ext) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if info, err := e.Info(); err == nil && info.ModTime().After(cur) {
|
||||||
|
cur = info.ModTime()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return cur
|
||||||
|
}
|
||||||
@@ -0,0 +1,141 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
import (
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// newRestorePointsManager builds a Manager over a tempdir unit layout for the ListRestorePoints
|
||||||
|
// tests. drive is the in-guest namespace root (≠ systemDataPath ⇒ treated as a user-data drive).
|
||||||
|
func newRestorePointsManager(t *testing.T, drive string, sett *settings.Settings) *Manager {
|
||||||
|
t.Helper()
|
||||||
|
return &Manager{
|
||||||
|
logger: log.New(io.Discard, "", 0),
|
||||||
|
settings: sett,
|
||||||
|
systemDataPath: filepath.Join(t.TempDir(), "sys"),
|
||||||
|
stackProvider: &fakeRecoveryProvider{hdd: drive},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestListRestorePoints_UnitOnDisk proves the F1 endpoint's data source: a recovery unit on disk
|
||||||
|
// yields EXACTLY ONE tier-1 point whose Time is the NEWEST artifact mtime (here: a db-dump made
|
||||||
|
// fresher than the manifest — the nightly-dump-vs-checksum-skipped-manifest case). This is the
|
||||||
|
// half a hollow "returns []" handler could never pass; the companion below is the same layout
|
||||||
|
// with the manifest deleted.
|
||||||
|
func TestListRestorePoints_UnitOnDisk(t *testing.T) {
|
||||||
|
drive := filepath.Join(t.TempDir(), "drive")
|
||||||
|
mustWrite(t, RecoveryUnitManifestPath(drive, "app"), "{}")
|
||||||
|
dumpPath := filepath.Join(AppDBDumpPath(drive, "app"), "app-postgres.sql")
|
||||||
|
mustWrite(t, dumpPath, "dump")
|
||||||
|
// Make the dump decisively newer than the manifest (mtime granularity safety).
|
||||||
|
dumpTime := time.Now().Add(1 * time.Hour).Truncate(time.Second)
|
||||||
|
if err := os.Chtimes(dumpPath, dumpTime, dumpTime); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
m := newRestorePointsManager(t, drive, nil)
|
||||||
|
points, found := m.ListRestorePoints("app")
|
||||||
|
if !found {
|
||||||
|
t.Fatal("stack should be found")
|
||||||
|
}
|
||||||
|
if len(points) != 1 {
|
||||||
|
t.Fatalf("want exactly 1 restore point, got %d (%v)", len(points), points)
|
||||||
|
}
|
||||||
|
p := points[0]
|
||||||
|
if p.Tier != 1 {
|
||||||
|
t.Errorf("tier = %d, want 1 (tier-2 copies are not restorable via /backup/restore)", p.Tier)
|
||||||
|
}
|
||||||
|
if p.ShortID != "helyi" {
|
||||||
|
t.Errorf("short_id = %q, want %q", p.ShortID, "helyi")
|
||||||
|
}
|
||||||
|
if want := dumpTime.UTC().Format(time.RFC3339); p.Time != want {
|
||||||
|
t.Errorf("time = %q, want newest artifact mtime %q (the db-dump, not the older manifest)", p.Time, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// COMPANION red-proof for the above: the SAME layout minus the manifest must yield an EMPTY list
|
||||||
|
// (found=true — "no backup yet" is a valid answer, not an error). Together the pair kills the
|
||||||
|
// hollow implementations: always-[] fails the first test, always-1-entry fails this one.
|
||||||
|
func TestListRestorePoints_NoUnitYet(t *testing.T) {
|
||||||
|
drive := filepath.Join(t.TempDir(), "drive")
|
||||||
|
// db-dump exists but there is NO manifest — the unit is what makes an app restorable.
|
||||||
|
mustWrite(t, filepath.Join(AppDBDumpPath(drive, "app"), "app-postgres.sql"), "dump")
|
||||||
|
|
||||||
|
m := newRestorePointsManager(t, drive, nil)
|
||||||
|
points, found := m.ListRestorePoints("app")
|
||||||
|
if !found {
|
||||||
|
t.Fatal("a known stack without a unit is still found")
|
||||||
|
}
|
||||||
|
if len(points) != 0 {
|
||||||
|
t.Errorf("want empty list without a recovery unit, got %v", points)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestListRestorePoints_DriveLabel proves drive_label resolution: a registered storage path's
|
||||||
|
// label for drive-resident apps, and EMPTY for the SSD system-data fallback (not a bogus basename).
|
||||||
|
func TestListRestorePoints_DriveLabel(t *testing.T) {
|
||||||
|
sett, err := settings.Load(filepath.Join(t.TempDir(), "settings.json"), log.New(io.Discard, "", 0))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Run("registered drive → label", func(t *testing.T) {
|
||||||
|
drive := filepath.Join(t.TempDir(), "usb")
|
||||||
|
if err := sett.AddStoragePath(settings.StoragePath{Path: drive, Label: "Tárhely (usb)"}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
mustWrite(t, RecoveryUnitManifestPath(drive, "app"), "{}")
|
||||||
|
m := newRestorePointsManager(t, drive, sett)
|
||||||
|
points, _ := m.ListRestorePoints("app")
|
||||||
|
if len(points) != 1 || points[0].DriveLabel != "Tárhely (usb)" {
|
||||||
|
t.Errorf("drive label: %v", points)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("SSD fallback → empty label", func(t *testing.T) {
|
||||||
|
sys := filepath.Join(t.TempDir(), "sysdata")
|
||||||
|
nsRoot := NamespaceRoot(sys, false) // system path appends felhom-data
|
||||||
|
mustWrite(t, RecoveryUnitManifestPath(nsRoot, "app"), "{}")
|
||||||
|
m := &Manager{
|
||||||
|
logger: log.New(io.Discard, "", 0),
|
||||||
|
settings: sett,
|
||||||
|
systemDataPath: sys,
|
||||||
|
stackProvider: &fakeRecoveryProvider{hdd: ""}, // no HDD_PATH → falls back to systemDataPath
|
||||||
|
}
|
||||||
|
points, found := m.ListRestorePoints("app")
|
||||||
|
if !found || len(points) != 1 {
|
||||||
|
t.Fatalf("points: %v found=%v", points, found)
|
||||||
|
}
|
||||||
|
if points[0].DriveLabel != "" {
|
||||||
|
t.Errorf("SSD fallback drive_label = %q, want empty", points[0].DriveLabel)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestListRestorePoints_UnknownStack proves the found=false path (the handler's 404).
|
||||||
|
func TestListRestorePoints_UnknownStack(t *testing.T) {
|
||||||
|
m := &Manager{
|
||||||
|
logger: log.New(io.Discard, "", 0),
|
||||||
|
systemDataPath: t.TempDir(),
|
||||||
|
stackProvider: &unknownStackProvider{},
|
||||||
|
}
|
||||||
|
if _, found := m.ListRestorePoints("ghost"); found {
|
||||||
|
t.Error("unknown stack must report found=false")
|
||||||
|
}
|
||||||
|
// No provider wired at all → also not found (nothing is restorable).
|
||||||
|
m.stackProvider = nil
|
||||||
|
if _, found := m.ListRestorePoints("ghost"); found {
|
||||||
|
t.Error("nil provider must report found=false")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// unknownStackProvider is a fakeRecoveryProvider whose stacks never resolve.
|
||||||
|
type unknownStackProvider struct{ fakeRecoveryProvider }
|
||||||
|
|
||||||
|
func (u *unknownStackProvider) GetStackComposePath(string) (string, bool) { return "", false }
|
||||||
@@ -0,0 +1,134 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"log"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// newSecretGenUnit lays out a recovery unit whose manifest names one resettable secret
|
||||||
|
// (DB_PASSWORD) and one data-key (SECRET_KEY) — the O4 test fixture.
|
||||||
|
func newSecretGenUnit(t *testing.T) (drive string) {
|
||||||
|
t.Helper()
|
||||||
|
drive = filepath.Join(t.TempDir(), "drive")
|
||||||
|
mustWrite(t, filepath.Join(RecoveryUnitComposePath(drive, "app"), "app.yaml"),
|
||||||
|
"deployed: true\nenv:\n SUBDOMAIN: trips\n")
|
||||||
|
man := &RecoveryManifest{SchemaVersion: 1, AppName: "app", ControllerVer: "v",
|
||||||
|
SecretEnvVars: []string{"DB_PASSWORD", "SECRET_KEY"}, DataKeyEnvVars: []string{"SECRET_KEY"}}
|
||||||
|
if err := writeManifest(RecoveryUnitManifestPath(drive, "app"), man); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return drive
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRestoreGeneratesMissingResettableSecret proves Scenario F: with the data-key recovered but
|
||||||
|
// DB_PASSWORD unrecoverable, the restore PROCEEDS and RecreateStackFromUnit receives a NON-EMPTY
|
||||||
|
// generated DB_PASSWORD (pre-O4 it was simply absent → compose deployed blank → exit 1). Also
|
||||||
|
// proves the generator is consulted for the resettable secret ONLY — never a data-key — and that
|
||||||
|
// the generated VALUE never reaches the logs.
|
||||||
|
// COMPANION red-proof: reverting to the pre-fix behaviour (no generation) fails the non-empty
|
||||||
|
// DB_PASSWORD assertion.
|
||||||
|
func TestRestoreGeneratesMissingResettableSecret(t *testing.T) {
|
||||||
|
const genValue = "generated-secret-value-do-not-log"
|
||||||
|
drive := newSecretGenUnit(t)
|
||||||
|
fake := &fakeRecoveryProvider{
|
||||||
|
hdd: drive,
|
||||||
|
running: true,
|
||||||
|
secrets: map[string]string{"SECRET_KEY": "deadbeef"}, // DB_PASSWORD unrecoverable
|
||||||
|
}
|
||||||
|
var logBuf bytes.Buffer
|
||||||
|
m := &Manager{logger: log.New(&logBuf, "", 0), systemDataPath: filepath.Join(drive, "..", "sys"),
|
||||||
|
stackProvider: fake}
|
||||||
|
var genCalls []string
|
||||||
|
m.generateSecret = func(stackName, envVar string) (string, bool) {
|
||||||
|
genCalls = append(genCalls, envVar)
|
||||||
|
return genValue, true
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := m.RestoreFromRecoveryUnit("app"); err != nil {
|
||||||
|
t.Fatalf("restore must proceed for a missing RESETTABLE secret: %v", err)
|
||||||
|
}
|
||||||
|
if fake.gotEnv == nil {
|
||||||
|
t.Fatal("recreate was not called")
|
||||||
|
}
|
||||||
|
if fake.gotEnv["DB_PASSWORD"] != genValue {
|
||||||
|
t.Errorf("DB_PASSWORD = %q, want the generated replacement (pre-O4: absent → blank deploy)", fake.gotEnv["DB_PASSWORD"])
|
||||||
|
}
|
||||||
|
if fake.gotEnv["SECRET_KEY"] != "deadbeef" {
|
||||||
|
t.Errorf("recovered data-key must pass through verbatim, got %q", fake.gotEnv["SECRET_KEY"])
|
||||||
|
}
|
||||||
|
if len(genCalls) != 1 || genCalls[0] != "DB_PASSWORD" {
|
||||||
|
t.Errorf("generator consulted for %v, want exactly [DB_PASSWORD] (never data-keys)", genCalls)
|
||||||
|
}
|
||||||
|
logs := logBuf.String()
|
||||||
|
if !strings.Contains(logs, "generated replacement") || !strings.Contains(logs, "DB_PASSWORD") {
|
||||||
|
t.Errorf("WARN must name the reset credential; logs:\n%s", logs)
|
||||||
|
}
|
||||||
|
// Secrets safety: the generated VALUE must never be logged — names only.
|
||||||
|
if strings.Contains(logs, genValue) {
|
||||||
|
t.Errorf("SECRET LEAK: generated value found in logs:\n%s", logs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRestoreProceedsWhenNoGenerator proves Scenario G: a missing resettable secret with NO
|
||||||
|
// generator (seam returns ok=false, or no seam wired) still proceeds — env var absent, loud WARN
|
||||||
|
// that the app may fail to start — and never invents a default.
|
||||||
|
func TestRestoreProceedsWhenNoGenerator(t *testing.T) {
|
||||||
|
run := func(t *testing.T, wire bool) {
|
||||||
|
drive := newSecretGenUnit(t)
|
||||||
|
fake := &fakeRecoveryProvider{
|
||||||
|
hdd: drive,
|
||||||
|
running: true,
|
||||||
|
secrets: map[string]string{"SECRET_KEY": "deadbeef"},
|
||||||
|
}
|
||||||
|
var logBuf bytes.Buffer
|
||||||
|
m := &Manager{logger: log.New(&logBuf, "", 0), systemDataPath: filepath.Join(drive, "..", "sys"),
|
||||||
|
stackProvider: fake}
|
||||||
|
if wire {
|
||||||
|
m.generateSecret = func(string, string) (string, bool) { return "", false } // no spec (Scenario G)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := m.RestoreFromRecoveryUnit("app"); err != nil {
|
||||||
|
t.Fatalf("restore must still proceed: %v", err)
|
||||||
|
}
|
||||||
|
if _, present := fake.gotEnv["DB_PASSWORD"]; present {
|
||||||
|
t.Errorf("no generator → the secret must stay absent, not be invented: %v", fake.gotEnv)
|
||||||
|
}
|
||||||
|
logs := logBuf.String()
|
||||||
|
if !strings.Contains(logs, "no generator") || !strings.Contains(logs, "may fail to start") || !strings.Contains(logs, "DB_PASSWORD") {
|
||||||
|
t.Errorf("upgraded WARN must name the var and the may-fail consequence; logs:\n%s", logs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
t.Run("generator wired, field has no spec", func(t *testing.T) { run(t, true) })
|
||||||
|
t.Run("no generator wired at all", func(t *testing.T) { run(t, false) })
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRestoreGenerationNeverReachesDataKeys proves the frozen gate is untouched by O4: a missing
|
||||||
|
// DATA-KEY still refuses fail-closed BEFORE any generation — the generator is never consulted and
|
||||||
|
// the app is never recreated, even with a generator eagerly offering values.
|
||||||
|
func TestRestoreGenerationNeverReachesDataKeys(t *testing.T) {
|
||||||
|
drive := newSecretGenUnit(t)
|
||||||
|
fake := &fakeRecoveryProvider{
|
||||||
|
hdd: drive,
|
||||||
|
secrets: map[string]string{"DB_PASSWORD": "pw"}, // SECRET_KEY (data_key) missing
|
||||||
|
}
|
||||||
|
m := &Manager{logger: log.New(bytes.NewBuffer(nil), "", 0), systemDataPath: filepath.Join(drive, "..", "sys"),
|
||||||
|
stackProvider: fake}
|
||||||
|
var genCalls []string
|
||||||
|
m.generateSecret = func(_, envVar string) (string, bool) {
|
||||||
|
genCalls = append(genCalls, envVar)
|
||||||
|
return "eager-value", true
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := m.RestoreFromRecoveryUnit("app"); err == nil {
|
||||||
|
t.Fatal("missing data-key must still refuse fail-closed")
|
||||||
|
}
|
||||||
|
if len(genCalls) != 0 {
|
||||||
|
t.Errorf("generator consulted for %v — must be unreachable when the gate refuses", genCalls)
|
||||||
|
}
|
||||||
|
if fake.gotEnv != nil {
|
||||||
|
t.Errorf("recreate must not run on refusal: %v", fake.gotEnv)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -114,9 +114,36 @@ func (m *Manager) RestoreFromRecoveryUnit(stackName string) error {
|
|||||||
m.logger.Printf("[ERROR] [backup] Restore REFUSED for %s: %v", stackName, err)
|
m.logger.Printf("[ERROR] [backup] Restore REFUSED for %s: %v", stackName, err)
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
// O4: a missing RESETTABLE secret used to redeploy blank (compose "Defaulting to a blank
|
||||||
|
// string" → exit 1). Generate a replacement via the deploy flow's generator instead —
|
||||||
|
// RecreateStackFromUnit persists fullEnv through SaveAppConfig, so the new value lands
|
||||||
|
// encrypted in the guest app.yaml and round-trips on the next backup/restore. Data-keys are
|
||||||
|
// never generated: the fail-closed gate above already refused if one was missing, and the
|
||||||
|
// generator itself refuses data-key fields (defense-in-depth). Values are never logged.
|
||||||
if len(missing) > 0 {
|
if len(missing) > 0 {
|
||||||
m.logger.Printf("[WARN] [backup] Restore %s: %d resettable secret(s) unrecoverable %v — proceeding (may need a credential reset; no data-key affected)",
|
dataKeySet := make(map[string]bool, len(manifest.DataKeyEnvVars))
|
||||||
stackName, len(missing), missing)
|
for _, dk := range manifest.DataKeyEnvVars {
|
||||||
|
dataKeySet[dk] = true
|
||||||
|
}
|
||||||
|
var generated, unresolved []string
|
||||||
|
for _, name := range missing {
|
||||||
|
if !dataKeySet[name] && m.generateSecret != nil {
|
||||||
|
if v, ok := m.generateSecret(stackName, name); ok && v != "" {
|
||||||
|
fullEnv[name] = v
|
||||||
|
generated = append(generated, name)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
}
|
||||||
|
unresolved = append(unresolved, name)
|
||||||
|
}
|
||||||
|
if len(generated) > 0 {
|
||||||
|
m.logger.Printf("[WARN] [backup] Restore %s: generated replacement for %v — the credential was reset (old value unrecoverable); stored data is unaffected (no data-key involved)",
|
||||||
|
stackName, generated)
|
||||||
|
}
|
||||||
|
if len(unresolved) > 0 {
|
||||||
|
m.logger.Printf("[WARN] [backup] Restore %s: %d resettable secret(s) unrecoverable and have no generator %v — proceeding, but the app may fail to start until the credential is set manually",
|
||||||
|
stackName, len(unresolved), unresolved)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
m.logger.Printf("[INFO] [backup] Restoring %s from recovery unit: images=%d, secrets recovered=%d/%d, data_keys=%d",
|
m.logger.Printf("[INFO] [backup] Restoring %s from recovery unit: images=%d, secrets recovered=%d/%d, data_keys=%d",
|
||||||
stackName, len(manifest.ImagePins), len(manifest.SecretEnvVars)-len(missing), len(manifest.SecretEnvVars), len(manifest.DataKeyEnvVars))
|
stackName, len(manifest.ImagePins), len(manifest.SecretEnvVars)-len(missing), len(manifest.SecretEnvVars), len(manifest.DataKeyEnvVars))
|
||||||
|
|||||||
@@ -0,0 +1,193 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
import (
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// volDumpFakeProvider is a StackDataProvider for the runVolumeDumps gating tests: a configurable
|
||||||
|
// stack list with per-stack volumes + drive, and a StopStack recorder (the destructive act the
|
||||||
|
// gates must prevent for volume-less/protected stacks).
|
||||||
|
type volDumpFakeProvider struct {
|
||||||
|
stacks []StackSummary
|
||||||
|
volumes map[string][]string
|
||||||
|
hdd map[string]string
|
||||||
|
stopped []string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *volDumpFakeProvider) GetStackComposePath(string) (string, bool) { return "", false }
|
||||||
|
func (f *volDumpFakeProvider) ListDeployedStacks() []StackSummary { return f.stacks }
|
||||||
|
func (f *volDumpFakeProvider) GetStackHDDMounts(string) []string { return nil }
|
||||||
|
func (f *volDumpFakeProvider) GetStackHDDPath(name string) string { return f.hdd[name] }
|
||||||
|
func (f *volDumpFakeProvider) GetDockerVolumes(name string) []string { return f.volumes[name] }
|
||||||
|
func (f *volDumpFakeProvider) StopStack(name string) error {
|
||||||
|
f.stopped = append(f.stopped, name)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
func (f *volDumpFakeProvider) StartStack(string) error { return nil }
|
||||||
|
func (f *volDumpFakeProvider) RefreshAndIsRunning(string) bool { return true }
|
||||||
|
func (f *volDumpFakeProvider) GetStackRecoveryInfo(string) (RecoveryInfo, bool) {
|
||||||
|
return RecoveryInfo{}, false
|
||||||
|
}
|
||||||
|
func (f *volDumpFakeProvider) RecoverStackSecrets(string, []string) map[string]string { return nil }
|
||||||
|
func (f *volDumpFakeProvider) RecreateStackFromUnit(string, string, map[string]string) error {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRunVolumeDumps_GatesPrecedeDump proves Scenario D/E's gating: the dump is invoked ONLY for
|
||||||
|
// a volume-bearing, unprotected stack on a writable drive. The negatives are the point —
|
||||||
|
// volume-less (rallly-like), protected (traefik-like), and disconnected-drive stacks are never
|
||||||
|
// dumped (and therefore never stopped, since stopping happens inside DumpAppVolumesSafe).
|
||||||
|
// COMPANION red-proof: removing the volume gate makes the seam fire for "rallly" → this fails.
|
||||||
|
func TestRunVolumeDumps_GatesPrecedeDump(t *testing.T) {
|
||||||
|
tmp := t.TempDir()
|
||||||
|
usbDrive := filepath.Join(tmp, "usb")
|
||||||
|
badDrive := filepath.Join(tmp, "gone")
|
||||||
|
|
||||||
|
sett, err := settings.Load(filepath.Join(tmp, "settings.json"), log.New(io.Discard, "", 0))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := sett.AddStoragePath(settings.StoragePath{Path: badDrive, Label: "gone"}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := sett.SetDisconnected(badDrive, true, nil); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg := &config.Config{}
|
||||||
|
cfg.Paths.SystemDataPath = filepath.Join(tmp, "sys")
|
||||||
|
cfg.Stacks.Protected = []string{"traefik"}
|
||||||
|
|
||||||
|
fake := &volDumpFakeProvider{
|
||||||
|
stacks: []StackSummary{
|
||||||
|
{Name: "nextcloud"}, {Name: "rallly"}, {Name: "traefik"}, {Name: "diskapp"},
|
||||||
|
},
|
||||||
|
volumes: map[string][]string{
|
||||||
|
"nextcloud": {"nextcloud_nextcloud_html"},
|
||||||
|
"rallly": nil, // volume-less — must NOT be stopped/dumped
|
||||||
|
"traefik": {"traefik_data"}, // protected — never considered
|
||||||
|
"diskapp": {"diskapp_data"}, // volume-bearing but drive disconnected
|
||||||
|
},
|
||||||
|
hdd: map[string]string{"nextcloud": usbDrive, "diskapp": badDrive},
|
||||||
|
}
|
||||||
|
|
||||||
|
m := &Manager{cfg: cfg, settings: sett, logger: log.New(io.Discard, "", 0),
|
||||||
|
systemDataPath: cfg.Paths.SystemDataPath, stackProvider: fake}
|
||||||
|
var dumpCalls []string
|
||||||
|
m.dumpVolumesSafe = func(name string) error {
|
||||||
|
dumpCalls = append(dumpCalls, name)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
summary, dumped, ok := m.runVolumeDumps()
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("run should be ok, summary=%v", summary)
|
||||||
|
}
|
||||||
|
if len(dumpCalls) != 1 || dumpCalls[0] != "nextcloud" {
|
||||||
|
t.Errorf("dump invoked for %v, want exactly [nextcloud] (gates must exclude volume-less/protected/disconnected)", dumpCalls)
|
||||||
|
}
|
||||||
|
if dumped != 1 {
|
||||||
|
t.Errorf("dumped = %d, want 1", dumped)
|
||||||
|
}
|
||||||
|
if len(fake.stopped) != 0 {
|
||||||
|
t.Errorf("StopStack called for %v — the seam bypasses the real dump, so ANY stop means a gate leaked", fake.stopped)
|
||||||
|
}
|
||||||
|
// The disconnected drive appears as a SKIP in the summary (same style as the DB loop).
|
||||||
|
if !containsSummary(summary, "SKIP diskapp volumes (drive disconnected)") {
|
||||||
|
t.Errorf("summary missing disconnected SKIP entry: %v", summary)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRunVolumeDumps_VolumelessNeverStopped drives the REAL DumpAppVolumesSafe path (no seam) with
|
||||||
|
// only volume-less/protected stacks: the volume gate must keep them from ever being stopped. This
|
||||||
|
// is the direct Scenario D negative — without the gate, DumpAppVolumesSafe stops the stack BEFORE
|
||||||
|
// its own volume check, so this test fails with stopped=[rallly].
|
||||||
|
func TestRunVolumeDumps_VolumelessNeverStopped(t *testing.T) {
|
||||||
|
cfg := &config.Config{}
|
||||||
|
cfg.Paths.SystemDataPath = filepath.Join(t.TempDir(), "sys")
|
||||||
|
cfg.Stacks.Protected = []string{"traefik"}
|
||||||
|
|
||||||
|
fake := &volDumpFakeProvider{
|
||||||
|
stacks: []StackSummary{{Name: "rallly"}, {Name: "traefik"}},
|
||||||
|
volumes: map[string][]string{"traefik": {"traefik_data"}},
|
||||||
|
}
|
||||||
|
m := &Manager{cfg: cfg, logger: log.New(io.Discard, "", 0),
|
||||||
|
systemDataPath: cfg.Paths.SystemDataPath, stackProvider: fake}
|
||||||
|
// deliberately NO seam: the real DumpAppVolumesSafe would record StopStack on the fake.
|
||||||
|
|
||||||
|
if _, dumped, ok := m.runVolumeDumps(); !ok || dumped != 0 {
|
||||||
|
t.Fatalf("expected clean zero-dump run, dumped=%d ok=%v", dumped, ok)
|
||||||
|
}
|
||||||
|
if len(fake.stopped) != 0 {
|
||||||
|
t.Errorf("volume-less/protected stacks were stopped: %v", fake.stopped)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRunVolumeDumps_FailureSurfaces proves no-silent-partial: a per-stack dump failure lands in
|
||||||
|
// the summary as a FAIL entry, flips allOK, and does NOT abort the remaining stacks.
|
||||||
|
func TestRunVolumeDumps_FailureSurfaces(t *testing.T) {
|
||||||
|
cfg := &config.Config{}
|
||||||
|
cfg.Paths.SystemDataPath = filepath.Join(t.TempDir(), "sys")
|
||||||
|
|
||||||
|
fake := &volDumpFakeProvider{
|
||||||
|
stacks: []StackSummary{{Name: "broken"}, {Name: "healthy"}},
|
||||||
|
volumes: map[string][]string{
|
||||||
|
"broken": {"broken_data"},
|
||||||
|
"healthy": {"healthy_data"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
m := &Manager{cfg: cfg, logger: log.New(io.Discard, "", 0),
|
||||||
|
systemDataPath: cfg.Paths.SystemDataPath, stackProvider: fake}
|
||||||
|
m.dumpVolumesSafe = func(name string) error {
|
||||||
|
if name == "broken" {
|
||||||
|
return errTest
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
summary, dumped, ok := m.runVolumeDumps()
|
||||||
|
if ok {
|
||||||
|
t.Error("allOK must be false after a dump failure")
|
||||||
|
}
|
||||||
|
if dumped != 1 {
|
||||||
|
t.Errorf("the healthy stack must still be dumped after another's failure (dumped=%d)", dumped)
|
||||||
|
}
|
||||||
|
if !containsSummaryPrefix(summary, "FAIL broken volumes:") {
|
||||||
|
t.Errorf("summary missing FAIL entry: %v", summary)
|
||||||
|
}
|
||||||
|
// failedSummaryLines feeds the run's returned error — the FAIL entry must survive the filter.
|
||||||
|
if failed := failedSummaryLines(summary); len(failed) != 1 {
|
||||||
|
t.Errorf("failedSummaryLines = %v, want exactly the broken entry", failed)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var errTest = &testErr{}
|
||||||
|
|
||||||
|
type testErr struct{}
|
||||||
|
|
||||||
|
func (*testErr) Error() string { return "tar exploded" }
|
||||||
|
|
||||||
|
func containsSummary(summary []string, want string) bool {
|
||||||
|
for _, s := range summary {
|
||||||
|
if s == want {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func containsSummaryPrefix(summary []string, prefix string) bool {
|
||||||
|
for _, s := range summary {
|
||||||
|
if strings.HasPrefix(s, prefix) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
@@ -827,6 +827,42 @@ func generateValue(spec string) (string, error) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// GenerateSecretForField generates a replacement value for a stack's RESETTABLE secret deploy-field
|
||||||
|
// (O4: the restore-from-unit path uses this — via backup.SetSecretGenerator — when a resettable
|
||||||
|
// secret cannot be recovered from the guest's app.yaml, so the app redeploys with a fresh credential
|
||||||
|
// instead of a blank one that fails compose-up).
|
||||||
|
//
|
||||||
|
// Returns ok=false when the field is unknown, has no generator spec, or — deliberately — is a
|
||||||
|
// DATA-ENCRYPTING key: data-keys are NEVER generated (regenerating one would render stored data
|
||||||
|
// unreadable; the restore's fail-closed gate refuses before this point, this is defense-in-depth).
|
||||||
|
// The generated VALUE is never logged — names only.
|
||||||
|
func (m *Manager) GenerateSecretForField(stackName, envVar string) (string, bool) {
|
||||||
|
s, ok := m.GetStack(stackName)
|
||||||
|
if !ok {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
meta := LoadMetadata(filepath.Dir(s.ComposePath))
|
||||||
|
for _, f := range meta.DeployFields {
|
||||||
|
if f.EnvVar != envVar {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if f.DataKey {
|
||||||
|
m.logger.Printf("[WARN] [stacks] GenerateSecretForField(%s/%s): refusing — field is a data-encrypting key", stackName, envVar)
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
if (f.Type != "secret" && f.Type != "password") || f.Generate == "" {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
value, err := generateValue(f.Generate)
|
||||||
|
if err != nil || value == "" {
|
||||||
|
m.logger.Printf("[ERROR] [stacks] GenerateSecretForField(%s/%s): generator %q failed: %v", stackName, envVar, f.Generate, err)
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
return value, true
|
||||||
|
}
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
|
||||||
// InjectMissingFields checks deployed stacks for new deploy_fields that are not
|
// InjectMissingFields checks deployed stacks for new deploy_fields that are not
|
||||||
// yet in app.yaml and auto-generates values for secret/domain fields.
|
// yet in app.yaml and auto-generates values for secret/domain fields.
|
||||||
// Called after sync (for updated stacks) and on startup (for all deployed stacks).
|
// Called after sync (for updated stacks) and on startup (for all deployed stacks).
|
||||||
|
|||||||
@@ -0,0 +1,89 @@
|
|||||||
|
package stacks
|
||||||
|
|
||||||
|
import (
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// newSecretGenManager builds a Manager with one stack whose .felhom.yml declares the O4 test
|
||||||
|
// fields: a generatable resettable secret, a data-key, and a spec-less secret.
|
||||||
|
func newSecretGenManager(t *testing.T) *Manager {
|
||||||
|
t.Helper()
|
||||||
|
stackDir := filepath.Join(t.TempDir(), "app")
|
||||||
|
if err := os.MkdirAll(stackDir, 0755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
meta := `display_name: App
|
||||||
|
deploy_fields:
|
||||||
|
- env_var: DB_PASSWORD
|
||||||
|
type: secret
|
||||||
|
generate: "password:24"
|
||||||
|
- env_var: SECRET_KEY
|
||||||
|
type: secret
|
||||||
|
generate: "hex:32"
|
||||||
|
data_key: true
|
||||||
|
- env_var: ADMIN_TOKEN
|
||||||
|
type: secret
|
||||||
|
- env_var: SUBDOMAIN
|
||||||
|
type: subdomain
|
||||||
|
default: app
|
||||||
|
`
|
||||||
|
if err := os.WriteFile(filepath.Join(stackDir, ".felhom.yml"), []byte(meta), 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return &Manager{
|
||||||
|
logger: log.New(io.Discard, "", 0),
|
||||||
|
stacks: map[string]*Stack{
|
||||||
|
"app": {Name: "app", ComposePath: filepath.Join(stackDir, "docker-compose.yml")},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestGenerateSecretForField covers the O4 generator seam's contract: spec-conformant values for
|
||||||
|
// resettable secrets, and REFUSAL for data-keys (frozen fail-closed territory), spec-less fields,
|
||||||
|
// non-secret fields, and unknown stacks/vars.
|
||||||
|
func TestGenerateSecretForField(t *testing.T) {
|
||||||
|
m := newSecretGenManager(t)
|
||||||
|
|
||||||
|
t.Run("resettable secret with spec → spec-conformant value", func(t *testing.T) {
|
||||||
|
v, ok := m.GenerateSecretForField("app", "DB_PASSWORD")
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("expected generation for DB_PASSWORD (generate: password:24)")
|
||||||
|
}
|
||||||
|
if len(v) != 24 || !regexp.MustCompile(`^[A-Za-z0-9]+$`).MatchString(v) {
|
||||||
|
t.Errorf("value does not conform to password:24 (len=%d)", len(v))
|
||||||
|
}
|
||||||
|
// Distinct per call (crypto/rand-backed, not a constant).
|
||||||
|
if v2, _ := m.GenerateSecretForField("app", "DB_PASSWORD"); v2 == v {
|
||||||
|
t.Error("two generations returned the same value")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("data-key → REFUSED even with a generate spec", func(t *testing.T) {
|
||||||
|
if v, ok := m.GenerateSecretForField("app", "SECRET_KEY"); ok || v != "" {
|
||||||
|
t.Error("a data-encrypting key must NEVER be generated")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("no generate spec → refused", func(t *testing.T) {
|
||||||
|
if _, ok := m.GenerateSecretForField("app", "ADMIN_TOKEN"); ok {
|
||||||
|
t.Error("spec-less secret must not be generated (Scenario G: proceed-with-warn instead)")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("non-secret field / unknown var / unknown stack → refused", func(t *testing.T) {
|
||||||
|
if _, ok := m.GenerateSecretForField("app", "SUBDOMAIN"); ok {
|
||||||
|
t.Error("non-secret field must not be generated")
|
||||||
|
}
|
||||||
|
if _, ok := m.GenerateSecretForField("app", "NOPE"); ok {
|
||||||
|
t.Error("unknown env var must not be generated")
|
||||||
|
}
|
||||||
|
if _, ok := m.GenerateSecretForField("ghost", "DB_PASSWORD"); ok {
|
||||||
|
t.Error("unknown stack must not be generated")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user