From cd0ebd20424066c84079a5df9462303fdbe15f5a Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 6 Jul 2026 20:43:11 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20v0.102.0=20REPORT=20+=20CONTEXT=20?= =?UTF-8?q?=E2=80=94=20async=20restore=20family,=20F4=20UX=20fixed=20+=20l?= =?UTF-8?q?ive-validated?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- CONTEXT.md | 12 +++++- REPORT.md | 109 ++++++++++++++++++----------------------------------- 2 files changed, 47 insertions(+), 74 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index 3f20f1d..a5a044b 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -7,7 +7,17 @@ > > Ask Claude Code: "Please update CONTEXT.md with what we did today" -Last updated: 2026-07-06 (v0.101.0 — no-mercy campaign findings F3 + F2-surfacing) +Last updated: 2026-07-06 (v0.102.0 — async restore family; F4 re-adjudicated + fixed) + +> **2026-07-06 — v0.102.0: async restore family (F4 UX fix, LIVE on 9201).** All three restore surfaces +> (`/backup/restore`, `/backup/tier2/restore`, `/backup/offbox/restore`) blocked the HTTP request until +> completion → through cloudflared's 100s cap a customer got an error page while the restore succeeded +> (offbox worse: bounded on `r.Context()`, canceling the SFTP restore mid-flight). Now async (offboxRun +> shape): fast-path IsRunning refuse → background goroutine (offbox ctx off r.Context()→Background+30m) → +> instant redirect. New `GET /api/backup/restore-status` + mutex op-status (`opstatus.go`) + 3s-polling +> `backups.html` banner. Live-proven: restore POST 0.018s internal / **0.235s external (F4 tunnel)**, canary +> bit-identical, status transitions. Restore single-flight unchanged. OPEN: op-status is in-memory (no +> persistence, by design). > **2026-07-06 — v0.101.0: no-mercy campaign findings.** F3: git subprocess deadline in > `internal/sync/sync.go` (`gitCmdTimeout=120s`, `exec.CommandContext`) — a hung remote no longer diff --git a/REPORT.md b/REPORT.md index 2780dd8..ee2905e 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,80 +1,43 @@ -# REPORT — v0.101.0: campaign findings F3 (sync deadline) + F2 evidence gap (agent refusal surfacing) +# REPORT — v0.102.0: async restore family (no proxy-timeout error page on a succeeding restore) -**Date:** 2026-07-06 · **Class:** implementation (code + tests + deploy + live validation) -**Baseline:** `main` @ `af7ea0b` (v0.100.0, matched the task's confirmed baseline exactly). -**Provenance:** `felhom.eu/documentation/audits/CAMPAIGN-nomercy-2026-07-06.md` findings F3, F2. -**Companion repos:** catalog `d86e256` (F1 vaultwarden fix); felhom.eu audit addendum (F1/F2/F3 verdicts). +**Date:** 2026-07-06 · **Class:** implementation (code + tests + deploy + live acceptance). +**Baseline:** `main` @ `9d5a588` (v0.101.0) → `c529a45`. Part B of the pilot-blockers bundle (A = agent +v0.73.0 F2, C = catalog healthcheck sweep). Provenance: RERUN F4 re-adjudication. -## What changed +## The bug (F4 re-adjudicated) +All three restore surfaces blocked the HTTP request until completion: `/backup/restore` (→ +`RestoreFromRecoveryUnit`), `/backup/tier2/restore` (→ `RestoreTier2Files`), `/backup/offbox/restore` (→ +`RestoreOffbox`). Through cloudflared's hard 100s cap + traefik a real customer got an **error page while +the restore silently succeeded**; the off-box handler was worse — it bounded on `r.Context()`, so a proxy +read-timeout **canceled the SFTP restore mid-flight**. -### F3 — git subprocess deadline (`internal/sync/sync.go`) -`runGitInDir` took no context; a hung remote parked the sync goroutine in `cmd.Run()`, the `doSync` -defer never ran, `syncing` stayed true, and every subsequent manual + periodic sync was refused -("Szinkronizálás már folyamatban") until a controller restart. Now: `const gitCmdTimeout = 120s`; -a `gitCmd` wrapper derives a fresh per-command `context.WithTimeout`; `runGitInDir` takes a -`context.Context` and uses `exec.CommandContext`; on `ctx.Err()` the error names the timeout + the -masked git args. All three call sites (clone via `runGit`, fetch, reset) route through it. Debounce -and failed-sync-arms-debounce semantics unchanged. +## The fix +- **Async family** (mirrors the live `offboxRunHandler`): each handler fast-path refuses a concurrent op + (`IsRunning()` → "Egy mentési/visszaállítási művelet már fut."), runs the restore in a **background + goroutine**, and redirects immediately with "Visszaállítás elindult…". The offbox restore's context moved + from `r.Context()` to `context.Background()+30m` (fixes the mid-flight cancel). The restore functions' + internal single-flight acquire is unchanged. +- **Op-status** (`internal/backup/opstatus.go`): mutex-guarded in-memory current-op + terminal + `last{op,stack,ok,message,finished_at}` (deep-copy getter); new `GET /api/backup/restore-status` (distinct + from `/backup/status`, which proxies the agent PBS status). In-memory (lost on restart). +- **UI** (`backups.html`): a banner polls the status every 3s — neutral running (shown even on a fresh page + load mid-op), success on completion, red only on failure. -### F2 evidence gap — agent refusal surfacing (`internal/agentapi/client.go`) -`EjectDisk` + `Decommission` used `c.post`, which drops any non-2xx body, flattening the agent's -informative refusal to a bare `HTTP 403` (the exact campaign evidence: -`agentapi: POST /disks/decommission: HTTP 403`). Both switched to the purpose-built -`postWithStatus` + a new shared `refusalError(path, status, env)` that returns -`agentapi: POST : HTTP : ` (and handles the ok:false-on-2xx -business-refusal case). `truncateErr` mirrors `stacks.truncateStr`. The generic `post` and every -other caller are untouched. `internal/web/storage_handlers.go` already wraps the client error into -its Hungarian message (`"a meghajtó leszerelése sikertelen: %w"` / eject `err.Error()`), so the role -reason now reaches the UI/API response verbatim — verified by reading the handler paths (no -re-truncation layer). +## Tests + red-proof +`opstatus_test.go` (begin→running→terminal, deep-copy, failure); `async_restore_test.go` (handler returns +<500ms while the restore parks in a blocking provider + op-status transitions; double-click refused with no +second launch). Red-proof: reverting the handler to synchronous blocks the request indefinitely (killed at +30s) vs <500ms async. Full gate `go build/vet/test ./...` = PASS. -## Tests + red-proofs (non-hollow) - -| Test | Asserts | Red-proof (pre-fix shape) | -|---|---|---| -| `TestRunGitInDir_CancelledContextKillsSubprocess` (T-C1) | cancelled ctx → error names the ctx cause + "deadline/killed", returns < 5s | reverted to `exec.Command` → **FAIL**: *"got nil (git --version ran to completion — the pre-fix behavior)"* | -| `TestTriggerSync_FailureReleasesSyncingAndAllowsRetry` (T-C2) | failed sync carries "Git hiba", `syncing` false after, immediate retry hits debounce, post-debounce retry EXECUTES | (behavioral; guards the lockup) | -| `TestDecommission_RefusalReasonSurfaced` (T-D1) | error contains "(role: system)" + "HTTP 403" | reverted to `c.post` → **FAIL**: *"agent refusal reason discarded — operator sees only: agentapi: POST /disks/decommission: HTTP 403"* | -| `TestEjectDisk_RefusalReasonSurfaced` (T-D2) | error contains "(role: backup)" | — | -| `TestDecommission_SuccessUnchanged` (T-D3) | 200 envelope decodes to result | — | -| `TestDecommission_OkFalseBusinessRefusal` | ok:false-on-2xx reason surfaced | — | - -Both red-proofs demonstrated live (revert → FAIL with the wrong value shown → restore → pass; -`git diff` clean). Green gate `go build ./... && go vet ./... && go test ./...` = **PASS** (exit 0); -30 passing tests across `internal/sync` + `internal/agentapi` (+2 new test files, +6 new tests). - -## Deploy + live validation (executed, real UI pipeline) - -- Built + pushed `gitea.dooplex.hu/admin/felhom-controller:0.101.0` on 180; deployed to guest 9201 - via golden/bootstrap → `docker ps` `:0.101.0 Up (healthy)`. -- Catalog synced (startup sync pulled the F1 template; `grep _ENABLE_SMTP` present in the guest - `docker-compose.yml` + `.felhom.yml`). -- **Scenario A (F1 acceptance):** stopped + removed the crash-looping vaultwarden via the UI - `/stop` + `/remove` endpoints (remove correctly 409'd until stopped — "still running"); fresh - `POST /api/stacks/vaultwarden/deploy` (app-email default off) → 202 → `running` in ~70s, - `docker inspect` **restarts=0**; env: `_ENABLE_SMTP=false`, `SMTP_HOST=`, `SMTP_FROM=` (empty). - No crash-loop — the pre-fix instance that was `Restarting` is gone. -- **Scenario B (regression):** `POST /stacks/vaultwarden/app-email app_email_enabled=on` → recreate - → healthy, restarts=0, `_ENABLE_SMTP=true`, `SMTP_HOST=felhom-controller`, - `SMTP_FROM=vaultwarden@felhom.eu`; then `app_email_enabled=false` → healthy, `_ENABLE_SMTP=false`. - Left OFF. The send-tested ON path is unchanged. -- **Sync smoke (F3):** `POST /api/sync` → HTTP 200 ("Sablonok naprakészek") — the deadline change - is transparent on the healthy path. - -## F2 verdict (real finding — agent fix deferred) - -Campaign-time agent journal (persistent, survived the P6 reboots): -`"local-api: protected — decommission refused by role" where=/mnt/teszt_enroll role=system`. -`roleForMountPath` (agent `disks.go:1031`) fail-safe-returns `system` when no PVE **storage target** -matches the mount — but `/disks` already classifies teszt_enroll as `user-data`. Two role paths -disagree; a bind-mounted user-data drive that isn't a standalone PVE storage can't be decommissioned. -Over-refusal (fail-safe), not a hole → **MEDIUM**. Full write-up: felhom.eu audit addendum. Agent -untouched (read-only per scope); fix is a separate follow-up task. - -## NOT yet live-validated -Clean backup/restore (F3-volume-tar / C2 Tier-2) coverage on the now-healthy vaultwarden — awaits -the separate targeted P1–P3 campaign re-run (a runbook, not this task). +## Deploy + live acceptance +Built + pushed `:0.102.0` on 180, deployed to guest 9201 (golden/bootstrap), healthy; `GET +/api/backup/restore-status` live. **Restore drill (vaultwarden):** planted canary → backup → vandalized → +`POST /backup/restore` returned in **0.018s** (302, "elindult" flash); status transitioned +`running:true → false` with terminal `{op:restore, ok:true}`; **canary restored bit-identical**. **External +path (F4):** `POST https://felhom.demo-felhom.eu/backup/restore` (through cloudflared) returned in **0.235s** +with 302 — no proxy error. **Tier-2:** `POST /backup/tier2/restore` async 0.007s; its failure landed in the +terminal status (B3 failure-visibility). Double-click refusal covered by the unit test. ## Not changed -felhom-agent (read-only; F2 is diagnosis only). The generic `agentapi.post` error format, the sync -debounce, and the other five smtp-mapped catalog templates. +The restore functions' internal single-flight; `/api/backup/status` (agent-PBS) semantics; the restore-button +CSRF/form mechanism (only the handler tail went async); no op-status persistence (documented).