admin/felhom-controller
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: v0.102.0 REPORT + CONTEXT — async restore family, F4 UX fixed + live-validated

Browse Source 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
authentik Default Admin
2026-07-06 20:43:11 +02:00
parent c529a455af
commit cd0ebd2042
2 changed files with 47 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTEXT.md
+11 -1
View File
@@ -7,7 +7,17 @@
>
> Ask Claude Code: "Please update CONTEXT.md with what we did today"
Last updated: 2026-07-06 (v0.101.0 — no-mercy campaign findings F3 + F2-surfacing)
Last updated: 2026-07-06 (v0.102.0 — async restore family; F4 re-adjudicated + fixed)
> **2026-07-06 — v0.102.0: async restore family (F4 UX fix, LIVE on 9201).** All three restore surfaces
> (`/backup/restore`, `/backup/tier2/restore`, `/backup/offbox/restore`) blocked the HTTP request until
> completion → through cloudflared's 100s cap a customer got an error page while the restore succeeded
> (offbox worse: bounded on `r.Context()`, canceling the SFTP restore mid-flight). Now async (offboxRun
> shape): fast-path IsRunning refuse → background goroutine (offbox ctx off r.Context()→Background+30m) →
> instant redirect. New `GET /api/backup/restore-status` + mutex op-status (`opstatus.go`) + 3s-polling
> `backups.html` banner. Live-proven: restore POST 0.018s internal / **0.235s external (F4 tunnel)**, canary
> bit-identical, status transitions. Restore single-flight unchanged. OPEN: op-status is in-memory (no
> persistence, by design).
> **2026-07-06 — v0.101.0: no-mercy campaign findings.** F3: git subprocess deadline in
> `internal/sync/sync.go` (`gitCmdTimeout=120s`, `exec.CommandContext`) — a hung remote no longer
REPORT.md
+36 -73
View File
@@ -1,80 +1,43 @@
# REPORT — v0.101.0: campaign findings F3 (sync deadline) + F2 evidence gap (agent refusal surfacing)
# REPORT — v0.102.0: async restore family (no proxy-timeout error page on a succeeding restore)
**Date:** 2026-07-06 · **Class:** implementation (code + tests + deploy + live validation)
**Baseline:** `main` @ `af7ea0b` (v0.100.0, matched the task's confirmed baseline exactly).
**Provenance:** `felhom.eu/documentation/audits/CAMPAIGN-nomercy-2026-07-06.md` findings F3, F2.
**Companion repos:** catalog `d86e256` (F1 vaultwarden fix); felhom.eu audit addendum (F1/F2/F3 verdicts).
**Date:** 2026-07-06 · **Class:** implementation (code + tests + deploy + live acceptance).
**Baseline:** `main` @ `9d5a588` (v0.101.0) → `c529a45`. Part B of the pilot-blockers bundle (A = agent
v0.73.0 F2, C = catalog healthcheck sweep). Provenance: RERUN F4 re-adjudication.
## What changed
## The bug (F4 re-adjudicated)
All three restore surfaces blocked the HTTP request until completion: `/backup/restore` (→
`RestoreFromRecoveryUnit`), `/backup/tier2/restore` (→ `RestoreTier2Files`), `/backup/offbox/restore` (→
`RestoreOffbox`). Through cloudflared's hard 100s cap + traefik a real customer got an **error page while
the restore silently succeeded**; the off-box handler was worse — it bounded on `r.Context()`, so a proxy
read-timeout **canceled the SFTP restore mid-flight**.
### F3 — git subprocess deadline (`internal/sync/sync.go`)
`runGitInDir` took no context; a hung remote parked the sync goroutine in `cmd.Run()`, the `doSync`
defer never ran, `syncing` stayed true, and every subsequent manual + periodic sync was refused
("Szinkronizálás már folyamatban") until a controller restart. Now: `const gitCmdTimeout = 120s`;
a `gitCmd` wrapper derives a fresh per-command `context.WithTimeout`; `runGitInDir` takes a
`context.Context` and uses `exec.CommandContext`; on `ctx.Err()` the error names the timeout + the
masked git args. All three call sites (clone via `runGit`, fetch, reset) route through it. Debounce
and failed-sync-arms-debounce semantics unchanged.
## The fix
- **Async family** (mirrors the live `offboxRunHandler`): each handler fast-path refuses a concurrent op
(`IsRunning()` → "Egy mentési/visszaállítási művelet már fut."), runs the restore in a **background
goroutine**, and redirects immediately with "Visszaállítás elindult…". The offbox restore's context moved
from `r.Context()` to `context.Background()+30m` (fixes the mid-flight cancel). The restore functions'
internal single-flight acquire is unchanged.
- **Op-status** (`internal/backup/opstatus.go`): mutex-guarded in-memory current-op + terminal
`last{op,stack,ok,message,finished_at}` (deep-copy getter); new `GET /api/backup/restore-status` (distinct
from `/backup/status`, which proxies the agent PBS status). In-memory (lost on restart).
- **UI** (`backups.html`): a banner polls the status every 3s — neutral running (shown even on a fresh page
load mid-op), success on completion, red only on failure.
### F2 evidence gap — agent refusal surfacing (`internal/agentapi/client.go`)
`EjectDisk` + `Decommission` used `c.post`, which drops any non-2xx body, flattening the agent's
informative refusal to a bare `HTTP 403` (the exact campaign evidence:
`agentapi: POST /disks/decommission: HTTP 403`). Both switched to the purpose-built
`postWithStatus` + a new shared `refusalError(path, status, env)` that returns
`agentapi: POST <path>: HTTP <code>: <agent reason, truncated ~300>` (and handles the ok:false-on-2xx
business-refusal case). `truncateErr` mirrors `stacks.truncateStr`. The generic `post` and every
other caller are untouched. `internal/web/storage_handlers.go` already wraps the client error into
its Hungarian message (`"a meghajtó leszerelése sikertelen: %w"` / eject `err.Error()`), so the role
reason now reaches the UI/API response verbatim — verified by reading the handler paths (no
re-truncation layer).
## Tests + red-proof
`opstatus_test.go` (begin→running→terminal, deep-copy, failure); `async_restore_test.go` (handler returns
<500ms while the restore parks in a blocking provider + op-status transitions; double-click refused with no
second launch). Red-proof: reverting the handler to synchronous blocks the request indefinitely (killed at
30s) vs <500ms async. Full gate `go build/vet/test ./...` = PASS.
## Tests + red-proofs (non-hollow)
| Test | Asserts | Red-proof (pre-fix shape) |
|---|---|---|
| `TestRunGitInDir_CancelledContextKillsSubprocess` (T-C1) | cancelled ctx → error names the ctx cause + "deadline/killed", returns < 5s | reverted to `exec.Command` → **FAIL**: *"got nil (git --version ran to completion — the pre-fix behavior)"* |
| `TestTriggerSync_FailureReleasesSyncingAndAllowsRetry` (T-C2) | failed sync carries "Git hiba", `syncing` false after, immediate retry hits debounce, post-debounce retry EXECUTES | (behavioral; guards the lockup) |
| `TestDecommission_RefusalReasonSurfaced` (T-D1) | error contains "(role: system)" + "HTTP 403" | reverted to `c.post` → **FAIL**: *"agent refusal reason discarded — operator sees only: agentapi: POST /disks/decommission: HTTP 403"* |
| `TestEjectDisk_RefusalReasonSurfaced` (T-D2) | error contains "(role: backup)" | — |
| `TestDecommission_SuccessUnchanged` (T-D3) | 200 envelope decodes to result | — |
| `TestDecommission_OkFalseBusinessRefusal` | ok:false-on-2xx reason surfaced | — |
Both red-proofs demonstrated live (revert → FAIL with the wrong value shown → restore → pass;
`git diff` clean). Green gate `go build ./... && go vet ./... && go test ./...` = **PASS** (exit 0);
30 passing tests across `internal/sync` + `internal/agentapi` (+2 new test files, +6 new tests).
## Deploy + live validation (executed, real UI pipeline)
- Built + pushed `gitea.dooplex.hu/admin/felhom-controller:0.101.0` on 180; deployed to guest 9201
via golden/bootstrap → `docker ps` `:0.101.0 Up (healthy)`.
- Catalog synced (startup sync pulled the F1 template; `grep _ENABLE_SMTP` present in the guest
`docker-compose.yml` + `.felhom.yml`).
- **Scenario A (F1 acceptance):** stopped + removed the crash-looping vaultwarden via the UI
`/stop` + `/remove` endpoints (remove correctly 409'd until stopped — "still running"); fresh
`POST /api/stacks/vaultwarden/deploy` (app-email default off) → 202 → `running` in ~70s,
`docker inspect` **restarts=0**; env: `_ENABLE_SMTP=false`, `SMTP_HOST=`, `SMTP_FROM=` (empty).
No crash-loop — the pre-fix instance that was `Restarting` is gone.
- **Scenario B (regression):** `POST /stacks/vaultwarden/app-email app_email_enabled=on` → recreate
→ healthy, restarts=0, `_ENABLE_SMTP=true`, `SMTP_HOST=felhom-controller`,
`SMTP_FROM=vaultwarden@felhom.eu`; then `app_email_enabled=false` → healthy, `_ENABLE_SMTP=false`.
Left OFF. The send-tested ON path is unchanged.
- **Sync smoke (F3):** `POST /api/sync` → HTTP 200 ("Sablonok naprakészek") — the deadline change
is transparent on the healthy path.
## F2 verdict (real finding — agent fix deferred)
Campaign-time agent journal (persistent, survived the P6 reboots):
`"local-api: protected — decommission refused by role" where=/mnt/teszt_enroll role=system`.
`roleForMountPath` (agent `disks.go:1031`) fail-safe-returns `system` when no PVE **storage target**
matches the mount — but `/disks` already classifies teszt_enroll as `user-data`. Two role paths
disagree; a bind-mounted user-data drive that isn't a standalone PVE storage can't be decommissioned.
Over-refusal (fail-safe), not a hole → **MEDIUM**. Full write-up: felhom.eu audit addendum. Agent
untouched (read-only per scope); fix is a separate follow-up task.
## NOT yet live-validated
Clean backup/restore (F3-volume-tar / C2 Tier-2) coverage on the now-healthy vaultwarden — awaits
the separate targeted P1–P3 campaign re-run (a runbook, not this task).
## Deploy + live acceptance
Built + pushed `:0.102.0` on 180, deployed to guest 9201 (golden/bootstrap), healthy; `GET
/api/backup/restore-status` live. **Restore drill (vaultwarden):** planted canary → backup → vandalized →
`POST /backup/restore` returned in **0.018s** (302, "elindult" flash); status transitioned
`running:true → false` with terminal `{op:restore, ok:true}`; **canary restored bit-identical**. **External
path (F4):** `POST https://felhom.demo-felhom.eu/backup/restore` (through cloudflared) returned in **0.235s**
with 302 — no proxy error. **Tier-2:** `POST /backup/tier2/restore` async 0.007s; its failure landed in the
terminal status (B3 failure-visibility). Double-click refusal covered by the unit test.
## Not changed
felhom-agent (read-only; F2 is diagnosis only). The generic `agentapi.post` error format, the sync
debounce, and the other five smtp-mapped catalog templates.
The restore functions' internal single-flight; `/api/backup/status` (agent-PBS) semantics; the restore-button
CSRF/form mechanism (only the handler tail went async); no op-status persistence (documented).