diff --git a/REPORT.md b/REPORT.md index b6dbb32..71006c0 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,58 +1,54 @@ -# REPORT — TASK C1: restore-path fixes (F1 dead restore UI + F3 volume dumps + O4 blank-secret redeploy) +# REPORT — TASK C2: one-click class-C file restore from the Tier-2 copy (closes F2) **Date:** 2026-07-05 · **Class:** implementation (code + tests + deploy + live validation) -**Baseline:** `main` @ `a39ab65` (v0.98.3, matched the task's confirmed baseline exactly; not behind origin) -**Shipped:** **v0.99.0**, live on guest 9201 (`gitea.dooplex.hu/admin/felhom-controller:0.99.0`, healthy) -**Scope honoured:** F1 + F3 + O4 only. **F2 (one-click in-place class-C restore) untouched — remains open for TASK C2.** No F9 bypass; every restore/backup action went through the real endpoints/UI. +**Baseline:** `main` @ `0313ecd` (v0.99.0, matched the task's confirmed baseline exactly; not behind origin) +**Shipped:** **v0.100.0**, live on guest 9201 (`gitea.dooplex.hu/admin/felhom-controller:0.100.0`, healthy) +**Outcome:** drill finding **F2 CLOSED** — the whole C-series (F1/F2/F3/O4 from `DRILL-appdata-restore-2026-07-04.md`) is now resolved. ## Commits (all on `main`) | Hash | What | |---|---| -| `f413f95` | F1 — `GET /api/backup/snapshots` (`backup.ListRestorePoints` + api route + guards + tests) | -| `73378a8` | F3 — `runVolumeDumps` wired into the backup run (+ seam + tests) | -| `a52851e` | O4 — resettable-secret generation on restore (`GenerateSecretForField` + `SetSecretGenerator` + tests) | -| `88362da` | docs — CHANGELOG v0.99.0, CONTEXT, REUSE, controller/README | -| felhom.eu `b5d92a8` | docs — backup-architecture.md (volume dumps real, snapshots endpoint, O4 + residual case) | +| `30b1110` | Part 1 — `RestoreTier2Files` engine + `rsyncRestoreMissing` + copier seam + tests | +| `27aeb41` | Parts 2+3 — `POST /backup/tier2/restore` + "Fájlok visszaállítása" button + handler-guard test | +| `85a5727` | Part 4 — CHANGELOG v0.100.0, CONTEXT (F2 closed), REUSE, controller/README | +| felhom.eu `7e42ca6` | backup-architecture.md — class-C in-place path, additive-only semantics, reindex caveat | -**Files touched (code):** `internal/api/router.go`, `internal/backup/{restore_points.go(new), backup.go, restore_unit.go}`, `internal/stacks/deploy.go`, `cmd/controller/main.go`. Tests (all new files): `internal/api/backup_snapshots_test.go`, `internal/backup/{restore_points_test.go, volume_dumps_test.go, restore_secrets_gen_test.go}`, `internal/stacks/deploy_secretgen_test.go`. +**Files (code):** `internal/backup/{tier2_restore.go(new), backup.go(seam)}`, `internal/web/{server.go, handlers.go, templates/backups.html}`. Tests (new): `internal/backup/tier2_restore_test.go`, `internal/web/tier2_restore_handler_test.go`. Template gates (id + emoji) green. -## What was fixed +## What shipped -- **F1:** the restore panel fetched `GET /api/backup/snapshots` — a restic-era route that no longer existed — so the dropdown never populated and "Visszaállítás indítása" never enabled. Now backed by `ListRestorePoints`: at most ONE honest entry (the current recovery unit; `time` = newest artifact mtime, `short_id:"helyi"`, `tier:1` always — Tier-2 copies are NOT restorable via `POST /backup/restore` and are never listed), `drive_label` from the registry. Guards: traversal/empty → 400 (`validStackParam`, same semantics as web's `validStackName`; api↔web import cycle prevents sharing the symbol — recorded in REUSE §6), unknown stack → 404, no unit → `ok:true, data:[]`. No template change needed. -- **F3:** `DumpAppVolumesSafe` had no caller → `volume-dumps/` never produced. New `runVolumeDumps` in `runDBDumpsInternal`, BEFORE `captureAllRecoveryUnits` (manifests enumerate fresh tars). Gate order is load-bearing: protected + has-volumes checks precede the Safe call (it stops the stack before its own check). Disconnected/decommissioned drives skip DB-loop-style. Failures land in the summary (`FAIL volumes:`) and fail the run. Zero-DB early return removed. Seam: `dumpVolumesSafe` func field. -- **O4:** a missing resettable secret no longer redeploys blank: generated from the catalog field's `generate` spec (`stacks.GenerateSecretForField` — refuses `data_key`/spec-less/non-secret fields — via the `backup.SetSecretGenerator` seam), persisted encrypted through the existing `RecreateStackFromUnit` → `RedeployFromEnv` → `SaveAppConfig` path (no second write path; round-trips on later backups). `reconcileRestoreSecrets` (the frozen fail-closed gate) untouched. WARNs discriminate generated-vs-ungenerable; values never logged. +- **Engine** `Manager.RestoreTier2Files(stack) (int, error)`: in-place, **additive-only** restore of missing user files from the **recorded** Tier-2 copy (`CrossDriveBackup.DestinationPath` + the `backups/secondary//appdata` literals from `RunTier2` — never a fresh `selectTier2Target`). Single-flight with backup/restore; **all refusals before any stop**, each with a customer-readable Hungarian reason (no copy / never ran / copy dir gone / Tier-2 drive disconnected / live drive disconnected or decommissioned); stop → copy → start → 90s health-wait; copy and restart errors surface (F17), health timeout only warns. +- **Copier** `rsyncRestoreMissing`: `rsyncMirror`'s exec shape (mkdir, 60m ctx, trailing-slash contents copy, rsync's own exit code) with the opposite-direction flags — **`-a --ignore-existing --itemize-changes`**. Existing live files never overwritten; nothing ever deleted (the `--delete` trap). Count = `>f` itemize lines (pure `countRestoredFiles`); file names logged at DEBUG only, never INFO. +- **Endpoint + UI**: `POST /backup/tier2/restore` (backupRestoreHandler-shaped guards) + the button on the **healthy Tier-2 row only** (Tier2Configured branch already excludes disconnected/inactive; extra `Tier2LastRun` gate), with the confirm dialog naming the additive-only contract + last-copy timestamp. Flash strings exactly per spec; n==0 is a success. +- **Out of scope honoured:** no overwrite/point-in-time mode, no per-file selection, `recovery-unit/` untouched, offbox untouched, C1-report Obs 1/2/3 untouched. -## Tests & companion red-proofs +## Tests & the §10 companion -`go build ./... && go vet ./... && go test ./...` green after every commit. **Top-level test functions: 272 → 286 (+14).** All three companions executed and reverted: +`go build ./... && go vet ./... && go test ./...` green after every commit. **Top-level test functions: 286 → 293 (+7).** -| Companion mutation | Expected red | Outcome | -|---|---|---| -| F1: `ListRestorePoints` hollowed to always-`[]` | unit-on-disk tests fail | **FAILED as expected** ("want exactly 1 restore point" family), reverted, green | -| F3: volume gate neutered (`len(vols)==0` check removed) | gating tests fail | **FAILED as expected** (`dump invoked for [nextcloud rallly]`; `_VolumelessNeverStopped` saw dumped=1), reverted, green | -| O4: generation skipped (pre-fix behaviour) | `TestRestoreGeneratesMissingResettableSecret` fails | **FAILED as expected** (`DB_PASSWORD = "", want the generated replacement`), reverted, green | +- Orchestration (seam): stop→copy→start order recorded; copier src = the recorded Tier-2 layout, dst = live `AppDataDir`; count passthrough. +- Refusals C1–C5: each asserts the **non-effect** — `StopStack` never called, copier never invoked (C5 via a held running flag). C6 at the handler (`httptest`; nil backupMgr would panic if reached — it isn't). +- Scenario D: `(0, nil)` success; copy-error surfacing (app still restarted). +- FS-level semantics against **real rsync** (LookPath-skipped on Windows; **PASS on the build server**): deleted file restored byte-identical + counted; differing live file keeps live bytes; live-only file survives. +- **Companion red-proof (run on the build server, where rsync exists):** flags swapped to `rsyncMirror`'s (`-a --delete`, no `--ignore-existing`) → the test failed **on both harms**: `live-edited b.txt was CLOBBERED: "BACKUP-VERSION"` **and** `live/c.txt: no such file or directory` (plus count 2≠1). Mutation reverted (`git status` clean on the build server), test green again. -Extra non-hollow assertions: the generated secret VALUE is asserted absent from the logs (secrets safety); `TestRestoreGenerationNeverReachesDataKeys` proves the fatal gate fires before generation ever runs; the api tests dispatch through `Router.ServeHTTP` (the F1 bug WAS a missing route). +## Deploy + live validation (§13, felhom-pve / guest 9201 only — Peti's box untouched) -## Deploy + live validation (§13, on felhom-pve / guest 9201 only) +Build `./build.sh 0.100.0 --push` on 180; bootstrap deploy; verify `:0.100.0 Up (healthy)`, no fatal/panic. -Build: `./build.sh 0.99.0 --push` on 180 (explicit git pull first). Deploy: 9201 bootstrap mechanism. Verify: `gitea.dooplex.hu/admin/felhom-controller:0.99.0 Up (healthy)`, startup logs clean. +**Method note:** the claude-in-chrome bridge disconnected mid-session (Chrome closed), so after verifying the **rendered UI HTML** (the button + confirm dialog + hidden `stack_name` render exactly once, on nextcloud's Tier-2 row only), the click was exercised as the **exact form POST the button submits** (`stack_name=nextcloud`, empty `_csrf` — identical bytes to a browser submit on this box). Per the live-validation rule this is the acceptable proxy; no server logic is skipped, only the click itself. -1. **Snapshots endpoint:** `?stack=nextcloud` → `{"ok":true,"data":[{"time":"2026-07-05T09:58:34Z","short_id":"helyi","tier":1,"drive_label":"Tárhely (felhom-usb)"}]}`; `?stack=sparkyfitness` → same shape, `drive_label:""` (SSD fallback); `?stack=../../etc` → **400**; `?stack=ghostapp` → **404**. -2. **UI end-to-end (claude-in-chrome — closes the drill's Phase-7 PARTIAL for restore):** `/backups` → Alkalmazás=Nextcloud → snapshot dropdown **populated** ("2026-07-05 vasárnap 12:01 (helyi) — 1. szint, Tárhely (felhom-usb)") → confirm checkbox → button **enabled** → click → flash **"nextcloud visszaállítva (helyi)."** Screenshot-verified; sidebar shows 0.99.0. -3. **Drill Phase-2 rerun (the F3 red now green):** sentinel `DRILL-VOL-c4c5e110…` (sha256 `560cb2a7…`) planted inside `nextcloud_nextcloud_html` → `POST /api/backup/run` → **5 volume dumps produced** (log: "App-data backup completed: 4 databases (810.2 KB total), 5 volume dump(s)"); `nextcloud_nextcloud_html.tar` contains the sentinel; manifest enumerates all 3 nextcloud tars → sentinel deleted from the live volume → **UI restore** → sentinel back, **sha256 identical** (`560cb2a7…`). Restore log: 3 volumes restored + DB dump replayed + completed; nextcloud healthy. -4. **Gates live:** protected/infra stacks (felhom-controller, traefik, cloudflared, filebrowser) never stopped by the run (still "Up 29 hours" afterwards). **Deviation from the task's expectation:** the spec assumed rallly is volume-less — on this box **every deployed app has named volumes** (rallly's postgres uses `rallly_rallly_postgres_data`), so all five were legitimately dumped and no volume-less app existed to observe live; that negative is covered by `TestRunVolumeDumps_VolumelessNeverStopped` (a no-seam variant driving the real `DumpAppVolumesSafe` path against a fake provider). -5. **O4 spot-check (drill Phase-5 rerun):** `DB_PASSWORD` removed from sparkyfitness app.yaml (backed up) → restore → log `generated replacement for [DB_PASSWORD] — the credential was reset…`, `recovered=3/4, data_keys=2`; the value persisted as `ENC:` in app.yaml and appears in no log. **The documented residual case then hit live:** the (now-existing, F3-produced) postgres volume tar was restored carrying the OLD credential hash, the server auth'd with the NEW one → server unhealthy → compose dependency failure → **the restore surfaced the error honestly** (`flash_error`, F17 no-swallow) instead of claiming success. Healed by restoring the original app.yaml + endpoint restore → `recovered=4/4`, completed, all healthy. +1. **Fixtures:** sentinel `d982fa3b….bin` (sha `1e5f82d7…`) + `modify_me.txt` (sha `c5ca964b…`) planted in live `appdata/nextcloud/drill` → `POST /api/backup/tier2` → both in the Tier-2 copy on felhom-flash, shas identical. +2. **Three fates staged after the copy:** `.bin` deleted live; `modify_me.txt` edited live (new sha `bd24729a…`); `live_only.txt` created live (sha `3c3474d5…`). +3. **Restore:** `302 → flash "nextcloud: 1 fájl visszaállítva a másodlagos másolatból."` Log: `Tier-2 file restore for nextcloud: …/felhom-flash/…/appdata → …/felhom-usb/appdata/nextcloud (additive-only)` … `1 file(s) restored (15s)`. +4. **All three fate assertions PASSED:** `.bin` back **byte-identical** (`1e5f82d7…`); `modify_me.txt` kept its **LIVE** sha (`bd24729a…`, NOT the backup's `c5ca964b…`); `live_only.txt` **survived** (`3c3474d5…`). Nextcloud healthy after restart. +5. **Idempotency (Scenario D):** immediate rerun → `flash "Nincs hiányzó fájl — minden fájl megvan a helyén."` (success, 0 files). +6. **Refusal non-effect:** sparkyfitness (no Tier-2 record) — the button is **absent** from its row (rendered-HTML count = 1, nextcloud only); direct POST → `flash_error "Fájl-visszaállítás sikertelen: nincs másodlagos fájlmásolat ehhez az alkalmazáshoz"`; container IDs + uptimes **unchanged** (never stopped). +7. **Cleanup:** drill dir removed live; fresh Tier-2 run purged the copy side (mirror semantics); zero residue both sides; final `docker ps` all healthy. -## Cleanup +## Observations (not acted on) -Volume sentinel removed from the live volume; a final backup run refreshed all tars (sentinel confirmed absent from `nextcloud_nextcloud_html.tar`); no `*.drillbak` remains; sparkyfitness app.yaml back to original. Final state: **all app containers healthy**, `/backups` 200, no controller ERRORs. Guest 9201 only; Peti's box untouched. - -## Observations (not acted on — candidates for follow-up TASKs) - -1. **O4 residual amplified by F3 (MEDIUM — product decision):** for apps whose DB lives in a named volume, restoring the volume tar resurrects the OLD credential hash; combined with a generated NEW password the app fails auth (hit live, above). Options: skip restoring a DB-data volume when the SQL dump will be replayed anyway (the dump is authoritative per F17), or in-DB credential reset after replay. `internal/backup/restore_unit.go` (volume restore vs dump replay ordering). -2. **compose stderr truncation (LOW):** `composeExec` keeps the FIRST ~500 chars, so the restore `flash_error` showed only volume-label warnings, not the decisive "dependency failed: container … is unhealthy" tail. Consider keeping the LAST 500 chars. `internal/stacks/manager.go`. -3. **`volume "X" already exists but was not created by Docker Compose` warnings (LOW/cosmetic):** `restoreDockerVolumes` recreates volumes without compose labels; compose then warns on every subsequent up. -4. **Demo box unauth/empty-CSRF state (pre-existing, still open):** the dashboard accepts POSTs with an empty `_csrf` and no session, and `felhom.demo-felhom.eu` is internet-reachable without auth — needs a separate check whether this is intentional demo setup or a gap (drill §2.2 flagged it; unchanged by this task). -5. **Nightly downtime note (by design, worth customer-facing wording):** volume-bearing apps now stop briefly during the nightly backup (stop-first is the locked torn-tar policy) — nextcloud was down ~16 s in the live run; the backups page already carries the "az alkalmazások rövid időre leállnak" hint for whole-guest backup, but the app-data run has no equivalent note. +1. **Bridge availability** — claude-in-chrome dropped mid-session; per the standing memory it only attaches to sessions started after the bridge connects. The UI *click* (with the confirm dialog) is worth a one-off manual/browser check next time a session has the bridge; everything else about the UI (rendering, gating, form contents) is verified. +2. **Reindex caveat is real but untested in-app** — the restored file returns at filesystem level; whether Nextcloud's own UI lists it may require `occ files:scan` (documented in backup-architecture.md; not exercised — the drill fixture lives outside Nextcloud's indexed user dirs). +3. **Carried from C1 (still open, unchanged):** O4-residual/F3 interaction, compose stderr truncation, compose volume-label warnings, demo-box unauth/empty-CSRF exposure.