Compare commits

...

8 Commits

Author SHA1 Message Date
admin d6c428b5dd session log: batch progress 2026-06-14 10:03:26 +02:00
admin 0b9450e356 F17: per-app restore now replays the captured .sql DB dump (.sql wins)
The restore paths (RestoreFromRecoveryUnit + the RestoreApp fallback) repopulated
Docker volume tars but NEVER replayed the captured <stack>-<dbtype>.sql dump, so
DB-resident data (e.g. rows in a DB whose data dir is a bind mount) did not come
back — the romm marker round-trip in the audit lost the row.

New appbackup.ImportDump (read-side counterpart to DumpOne) replays a .sql/.sql.gz
into the running DB using the live container's OWN discovered credentials (no env
threading; reuses DiscoveredDB + getMariaDBPassword). backup.reimportDBDumps
orchestrates it AFTER volume restore + stack bring-up, so the logical dump WINS
over any volume-tar copy of the DB (operator-chosen precedence). pg_dump
--clean --if-exists and mariadb-dump (default --add-drop-table) make replay
idempotent; psql ON_ERROR_STOP=1 surfaces real import errors.

Also: volume-restore per-volume failures and DB-import failures now SURFACE (the
restore returns an error) instead of a swallowed WARN, so a failed data restore
cannot read as success.

Tests (restore_db_test.go, injectable discover/import seams): imports when dump+DB
present, failure surfaces, no-dump skips discovery, dump-but-no-matching-DB is a
non-fatal skip. Live DB round-trip to be validated post-deploy.
2026-06-14 10:02:58 +02:00
admin 803ce50578 F5 (dashboard): surface 'route unpublished' for unhealthy/restarting deployed apps
Traefik only publishes a route to a healthy container, so an unhealthy deployed app
returns 404 at its URL though the container runs — previously shown only as 'Nem
egészséges' with no hint the URL is dead. New routeUnpublished() funcmap helper +
a distinct indicator on the dashboard and stacks cards (gated on .Deployed). Tests:
routeUnpublished across all states, real templateFS parses with the funcmap, and the
card guard renders the indicator only for deployed+unhealthy.
2026-06-14 09:56:30 +02:00
admin 68684892d8 F8: persist controller.yaml as 0600 (holds infra secrets)
The Hub config-apply handler wrote controller.yaml 0644; it holds cf_api_token,
cf_tunnel_token and hub api_key in plaintext. New writeConfig0600 helper writes
0600 atomically (tmp+rename, bind-mount fallback) and chmods to enforce 0600 even
when the file pre-existed 0644 (os.WriteFile doesn't chmod existing files).
Test asserts mode 0600 (Linux; skipped on Windows). Setup path already used 0600.
2026-06-14 09:50:50 +02:00
admin 4938cc8985 F7: tighten status-refresh cadence 30s->10s (dashboard state lag)
The dashboard stacks list served the in-memory map refreshed by a 30s ticker, so
container state lagged Docker health by up to ~30s after a deploy. RefreshStatus
is a cheap docker-ps refresh; 10s (matching health-probes) cuts the lag without
loading Docker.
2026-06-14 09:49:05 +02:00
admin 56fe5749a5 F4+F6: 405 for non-POST /stacks/rescan; deploy POST reports 'started' not 'deployed'
F4: GET /api/stacks/rescan fell through to GET /stacks/{name} → misleading
'stack not found: rescan'. Now returns 405 + Allow: POST.
F6: the deploy POST returns before compose/health complete (async; UI polls).
Message changed 'Stack X deployed' → 'Telepítés elindítva…' and status 200→202
Accepted, so API/script consumers aren't told a deploy finished when it hasn't.
UI checks data.ok (not HTTP status), so 202 is safe.
2026-06-14 09:48:08 +02:00
admin 2cf3fadaec F20-BUG1: surface swallowed format error in agentapi.FormatDisk
A failed agent format (e.g. 502 'device is mounted', ok:false, data:null) fell
through FormatDisk's trailing 'return out, nil', so the web layer reported a
zero-value FormatResult as ok:true — a failed DESTRUCTIVE format read as success.
postWithStatus now returns the full envelope; FormatDisk returns a non-nil error
on any non-2xx/ok:false that is not a recognized refusal (403/needs-confirmation).
Test TestFormat_MountedFailureSurfacesError (502 → non-nil err) fails on old code.
2026-06-14 09:46:28 +02:00
admin 0550b3117e F1: read cgroup memory limit, not host RAM (restores deploy OOM guard)
/api/system/info reported the Proxmox host's 16GB (the controller container
reads host /proc/meminfo with no lxcfs), defeating the deploy memory-headroom
hard-block (deploy.go uses GetMemoryMB). readMemInfo now prefers the cgroup
memory limit (v2 memory.max / v1 memory.limit_in_bytes; sentinels = unlimited)
when finite and below the host total; used = memory.current/usage_in_bytes.
Test info_cgroup_test.go (cgroup v2 cap wins, v2 max sentinel, v1 unlimited,
v1 finite) — fails on pre-fix code.
2026-06-14 09:44:28 +02:00
20 changed files with 817 additions and 43 deletions
+12
View File
@@ -0,0 +1,12 @@
# SESSION 2026-06-14 — Fix Batch 1 (+F17) → controller v0.61.0
Branch: `audit/2026-06-14-live-drive`. Implementing controller-side live-drive fixes per `LIVE-DRIVE-FIXSPEC-2026-06-14.md`.
Exclusions: F9, F20-BUG2, F20-BUG3 (agent/golden — supervised next session).
Operator decisions applied: F17 `.sql` wins; F5 gate route but surface "route unpublished"; F8 0600; F11 warn-and-allow (note: F11 not in this batch's fix list — only F1, F20-BUG1, F4, F6, F7, F8, F5, F17).
## Progress log
- **t0** — Read CLAUDE.md (build workflow), FIXSPEC, target source files. Setting up.
- **Test approach (documented assumption):** controller code is largely `//go:build linux`; dev host is Windows. Per-commit gate = `CGO_ENABLED=0 GOOS=linux go build ./... && go vet ./...` locally (compiles + type-checks code AND `_test.go`). Authoritative `go test ./...` runs on the Linux build server (192.168.0.180) against the pushed branch BEFORE the image build/deploy. Windows cannot execute linux test binaries, so this is the faithful interpretation of the per-commit green gate.
- **t2 — quick wins DONE** (separate commits): F20-BUG1 (agentapi.FormatDisk surfaces non-2xx/ok:false instead of zero-value success; test passes, fails on old code), F4 (405 for non-POST /stacks/rescan), F6 (deploy POST → 202 "Telepítés elindítva"; UI checks data.ok so 202 safe), F7 (status-refresh 30s→10s), F8 (writeConfig0600 helper enforces 0600 even on a pre-existing file; test skips on Windows, asserts on Linux). F5 catalog: uptime-kuma healthcheck fixed (→ `extra/healthcheck` binary + 180s start_period) committed+pushed to app-catalog main. F5 dashboard: `routeUnpublished` funcmap helper + indicator on stacks/dashboard cards + CSS; tests (func + real-template parse + fragment render) pass.
- **t3 — F17 DONE.** Reuse decision: put `ImportDump`/`waitDBReady` in `appbackup` (the DB-domain home with DiscoverDatabases/DumpOne/getMariaDBPassword) rather than reusing appexport's unexported copies — appexport→appbackup already exists, so the reverse would CYCLE; appbackup is the clean shared home and `DiscoveredDB` already carries the live container's creds (no env threading). `backup.reimportDBDumps` (injectable discover/import seams) runs after volume restore in both RestoreFromRecoveryUnit and RestoreApp → `.sql` WINS. Volume-restore + DB-import failures now SURFACE (restore returns error). Unit tests pass; full backup/appbackup/agentapi/web suites green locally. **F17 escape hatch:** shipping it in v0.61.0 contingent on the LIVE DB round-trip passing post-deploy; if it fails, revert F17 to branch fix/f17-restore-db-reimport and ship the rest.
- **t1 — F1 (cgroup memory)** DONE. `internal/system/info_linux.go`: `readMemInfo` now prefers the cgroup memory LIMIT (v2 `memory.max`, v1 `memory.limit_in_bytes`, "max"/near-uint64-max = unlimited→fallback) when finite and below the host `/proc/meminfo` total; used = `memory.current`/`usage_in_bytes`. Restores the deploy OOM guard (deploy.go:162-185 reads GetMemoryMB). Test `info_cgroup_test.go` (4 cases) — fails on pre-fix code (ignored cgroup). GOOS=linux build+vet OK.
+5 -1
View File
@@ -253,7 +253,11 @@ func main() {
sched.SetDebug(cfg.Logging.Level == "debug") sched.SetDebug(cfg.Logging.Level == "debug")
// Existing periodic tasks (migrated from ad-hoc goroutines) // Existing periodic tasks (migrated from ad-hoc goroutines)
sched.Every("status-refresh", 30*time.Second, func(ctx context.Context) error { // F7: 30s left the dashboard list lagging Docker health by up to ~30s after a deploy/state change.
// 10s (matching the health-probes cadence) tightens it; RefreshStatus is a cheap `docker ps`-based
// refresh of the in-memory map, so 10s does not meaningfully load Docker. (The deploy page itself
// already polls per-stack every 3s; this is for the dashboard/stacks list.)
sched.Every("status-refresh", 10*time.Second, func(ctx context.Context) error {
return stackMgr.RefreshStatus() return stackMgr.RefreshStatus()
}) })
sched.Every("stack-scan", 2*time.Minute, func(ctx context.Context) error { sched.Every("stack-scan", 2*time.Minute, func(ctx context.Context) error {
+22 -11
View File
@@ -373,15 +373,15 @@ func (c *Client) FormatDisk(ctx context.Context, device, fstype string, confirme
var out FormatResult var out FormatResult
// Status-aware POST: the agent returns the FULL FormatResponse (incl. pending_op / durable_id) // Status-aware POST: the agent returns the FULL FormatResponse (incl. pending_op / durable_id)
// even on the 403 refusal, so we must read the body on non-2xx rather than discarding it. // even on the 403 refusal, so we must read the body on non-2xx rather than discarding it.
data, status, err := c.postWithStatus(ctx, "/disks/format", map[string]any{ env, status, err := c.postWithStatus(ctx, "/disks/format", map[string]any{
"device": device, "fstype": fstype, "confirmed": confirmed, "durable_id": durableID, "device": device, "fstype": fstype, "confirmed": confirmed, "durable_id": durableID,
}) })
if err != nil { if err != nil {
return out, err return out, err
} }
// data is the envelope's {data:…} payload (present on both success and the 403 refusal). // env.Data is the envelope's {data:…} payload (present on both success and the 403 refusal).
if len(data) > 0 { if len(env.Data) > 0 {
_ = json.Unmarshal(data, &out) // best-effort; fields default on a missing/partial body _ = json.Unmarshal(env.Data, &out) // best-effort; fields default on a missing/partial body
} }
if out.Formatted { if out.Formatted {
return out, nil return out, nil
@@ -394,34 +394,45 @@ func (c *Client) FormatDisk(ctx context.Context, device, fstype string, confirme
out.DataBearing = true out.DataBearing = true
return out, ErrFormatRefused // system/backup: surface the opsign command return out, ErrFormatRefused // system/backup: surface the opsign command
} }
// F20-BUG1: a non-2xx response (or ok:false) that is NOT one of the recognized refusals above is a
// real failure (e.g. the agent's 502 on a mkfs error: "device is mounted"). Returning the zero-value
// result with a nil error here made a failed destructive format read as a silent SUCCESS in the web
// layer. Surface it as an error so the caller (and the dashboard) report the failure.
if status < 200 || status >= 300 || !env.OK {
msg := strings.TrimSpace(env.Error)
if msg == "" {
msg = "format failed"
}
return out, fmt.Errorf("agentapi: format: HTTP %d: %s", status, msg)
}
return out, nil return out, nil
} }
// postWithStatus issues an authenticated JSON POST and returns the envelope's data payload + the HTTP // postWithStatus issues an authenticated JSON POST and returns the envelope's data payload + the HTTP
// status, even on a non-2xx (so callers like FormatDisk can read a 403 refusal body). A transport or // status, even on a non-2xx (so callers like FormatDisk can read a 403 refusal body). A transport or
// envelope-parse failure is still an error; an `ok:false` business refusal is NOT (the data carries it). // envelope-parse failure is still an error; an `ok:false` business refusal is NOT (the data carries it).
func (c *Client) postWithStatus(ctx context.Context, path string, body any) (json.RawMessage, int, error) { func (c *Client) postWithStatus(ctx context.Context, path string, body any) (apiResponse, int, error) {
var env apiResponse
buf, err := json.Marshal(body) buf, err := json.Marshal(body)
if err != nil { if err != nil {
return nil, 0, err return env, 0, err
} }
req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+path, bytes.NewReader(buf)) req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+path, bytes.NewReader(buf))
if err != nil { if err != nil {
return nil, 0, err return env, 0, err
} }
req.Header.Set("Authorization", "Bearer "+c.token) req.Header.Set("Authorization", "Bearer "+c.token)
req.Header.Set("Content-Type", "application/json") req.Header.Set("Content-Type", "application/json")
resp, err := c.hc.Do(req) resp, err := c.hc.Do(req)
if err != nil { if err != nil {
return nil, 0, fmt.Errorf("agentapi: POST %s: %w", path, err) return env, 0, fmt.Errorf("agentapi: POST %s: %w", path, err)
} }
defer resp.Body.Close() defer resp.Body.Close()
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
var env apiResponse
if err := json.Unmarshal(raw, &env); err != nil { if err := json.Unmarshal(raw, &env); err != nil {
return nil, resp.StatusCode, fmt.Errorf("agentapi: POST %s: HTTP %d, bad envelope: %w", path, resp.StatusCode, err) return env, resp.StatusCode, fmt.Errorf("agentapi: POST %s: HTTP %d, bad envelope: %w", path, resp.StatusCode, err)
} }
return env.Data, resp.StatusCode, nil return env, resp.StatusCode, nil
} }
// ---- slice 9: host metrics (the customer host-health view) ------------------------------- // ---- slice 9: host metrics (the customer host-health view) -------------------------------
@@ -34,6 +34,9 @@ func diskStub(t *testing.T) (*httptest.Server, string) {
case strings.Contains(body.Device, "data") && !body.Confirmed: // user-data, not yet confirmed case strings.Contains(body.Device, "data") && !body.Confirmed: // user-data, not yet confirmed
w.WriteHeader(http.StatusForbidden) w.WriteHeader(http.StatusForbidden)
_, _ = w.Write([]byte(`{"ok":false,"data":{"device":"` + body.Device + `","data_bearing":true,"role":"user-data","needs_confirmation":true,"durable_id":"byid:wwn-1"}}`)) _, _ = w.Write([]byte(`{"ok":false,"data":{"device":"` + body.Device + `","data_bearing":true,"role":"user-data","needs_confirmation":true,"durable_id":"byid:wwn-1"}}`))
case strings.Contains(body.Device, "mounted"): // mkfs failed (e.g. device mounted) → agent 502, data:null
w.WriteHeader(http.StatusBadGateway)
_, _ = w.Write([]byte(`{"ok":false,"error":"format failed: /dev/sdb1 is mounted; will not make a filesystem here!","data":null}`))
default: // blank, or user-data confirmed default: // blank, or user-data confirmed
_, _ = w.Write([]byte(`{"ok":true,"data":{"device":"` + body.Device + `","formatted":true,"role":"user-data"}}`)) _, _ = w.Write([]byte(`{"ok":true,"data":{"device":"` + body.Device + `","formatted":true,"role":"user-data"}}`))
} }
@@ -117,6 +120,28 @@ func TestFormat_UserDataConfirmed(t *testing.T) {
} }
} }
// F20-BUG1: a real mkfs failure (agent 502, ok:false, data:null) must surface as a non-nil error —
// NOT a zero-value FormatResult with nil err (which read as a silent SUCCESS in the web layer).
func TestFormat_MountedFailureSurfacesError(t *testing.T) {
s, ep := diskStub(t)
defer s.Close()
c := clientFor(t, s, ep)
res, err := c.FormatDisk(context.Background(), "/dev/sdb1-mounted", "ext4", true, "byid:wwn-1")
if err == nil {
t.Fatalf("expected a non-nil error for a failed format, got (res=%+v, err=nil) — silent success regression", res)
}
if res.Formatted {
t.Fatalf("Formatted must be false on a failed format: %+v", res)
}
if !strings.Contains(err.Error(), "502") || !strings.Contains(err.Error(), "mounted") {
t.Fatalf("error should carry the HTTP status + agent message, got: %v", err)
}
// Must not be misclassified as one of the gated refusals.
if errors.Is(err, ErrNeedsConfirmation) || errors.Is(err, ErrFormatRefused) {
t.Fatalf("a 502 mkfs failure must not be reported as a refusal: %v", err)
}
}
func TestEject_Dependents(t *testing.T) { func TestEject_Dependents(t *testing.T) {
s, ep := diskStub(t) s, ep := diskStub(t)
defer s.Close() defer s.Close()
@@ -0,0 +1,39 @@
package api
import (
"os"
"path/filepath"
"runtime"
"testing"
)
// TestWriteConfig0600 asserts F8: controller.yaml is persisted 0600 (it holds infra secrets), even when
// the target file already existed with looser (0644) permissions. POSIX modes only — skipped on Windows.
func TestWriteConfig0600(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("POSIX file modes not represented on Windows")
}
dir := t.TempDir()
path := filepath.Join(dir, "controller.yaml")
// Pre-create with world-readable 0644 to prove the helper tightens an existing file.
if err := os.WriteFile(path, []byte("old: true\n"), 0o644); err != nil {
t.Fatal(err)
}
if err := writeConfig0600(path, []byte("hub:\n api_key: redacted\n")); err != nil {
t.Fatalf("writeConfig0600: %v", err)
}
fi, err := os.Stat(path)
if err != nil {
t.Fatal(err)
}
if mode := fi.Mode().Perm(); mode != 0o600 {
t.Fatalf("config mode = %o, want 0600", mode)
}
// No leftover temp file.
if _, err := os.Stat(path + ".tmp"); !os.IsNotExist(err) {
t.Fatalf("temp file not cleaned up")
}
}
+37 -17
View File
@@ -112,6 +112,12 @@ func (r *Router) ServeHTTP(w http.ResponseWriter, req *http.Request) {
case path == "/stacks/rescan" && req.Method == http.MethodPost: case path == "/stacks/rescan" && req.Method == http.MethodPost:
r.rescanStacks(w, req) r.rescanStacks(w, req)
// F4: /api/stacks/rescan with a non-POST method must be a clear 405, not fall through to the
// GET /stacks/{name} lookup below (which returned the misleading "stack not found: rescan").
case path == "/stacks/rescan":
w.Header().Set("Allow", http.MethodPost)
writeJSON(w, http.StatusMethodNotAllowed, apiResponse{OK: false, Error: "method not allowed: use POST /api/stacks/rescan"})
// GET /api/stacks/{name} // GET /api/stacks/{name}
case strings.HasPrefix(path, "/stacks/") && req.Method == http.MethodGet && !hasSubpath(path, "/stacks/"): case strings.HasPrefix(path, "/stacks/") && req.Method == http.MethodGet && !hasSubpath(path, "/stacks/"):
r.getStack(w, req, trimSegment(path, "/stacks/")) r.getStack(w, req, trimSegment(path, "/stacks/"))
@@ -378,11 +384,15 @@ func (r *Router) deployStack(w http.ResponseWriter, req *http.Request, name stri
return return
} }
resp := apiResponse{OK: true, Message: "Stack " + name + " deployed"} // F6: the deploy runs asynchronously (compose pull/up + health happen after this returns; the UI
// polls GET /api/stacks/{name}). The old "Stack X deployed" message asserted completion before it
// was true — misleading for API/script consumers. Report that the deploy STARTED, not that it
// finished. 202 Accepted reflects "accepted, processing"; ok:true is preserved for the UI.
resp := apiResponse{OK: true, Message: "Telepítés elindítva az állapot a kártyán követhető"}
if warning != "" { if warning != "" {
resp.Data = map[string]string{"warning": warning} resp.Data = map[string]string{"warning": warning}
} }
writeJSON(w, http.StatusOK, resp) writeJSON(w, http.StatusAccepted, resp)
// Push app deployed event to Hub // Push app deployed event to Hub
if r.notifier != nil { if r.notifier != nil {
@@ -1001,25 +1011,14 @@ func (r *Router) configApply(w http.ResponseWriter, req *http.Request) {
return return
} }
// Write config: try atomic rename first, fall back to direct write // Write config 0600: it holds infra credentials (cf_api_token, cf_tunnel_token, hub api_key) in
// (os.Rename fails on Docker bind mounts with "device or resource busy") // plaintext (F8), so it must not be world-readable. writeConfig0600 enforces the mode even when the
tmpPath := r.configPath + ".tmp" // target file already existed with looser perms (os.WriteFile does not chmod an existing file).
if err := os.WriteFile(tmpPath, body, 0644); err != nil { if err := writeConfig0600(r.configPath, body); err != nil {
r.logger.Printf("[ERROR] [api] Config apply: failed to write temp file: %v", err)
writeJSON(w, http.StatusInternalServerError, apiResponse{OK: false, Error: "failed to write config"})
return
}
if err := os.Rename(tmpPath, r.configPath); err != nil {
os.Remove(tmpPath)
// Rename failed (likely Docker bind mount) — write directly
if err := os.WriteFile(r.configPath, body, 0644); err != nil {
r.logger.Printf("[ERROR] [api] Config apply: failed to write config: %v", err) r.logger.Printf("[ERROR] [api] Config apply: failed to write config: %v", err)
writeJSON(w, http.StatusInternalServerError, apiResponse{OK: false, Error: "failed to apply config"}) writeJSON(w, http.StatusInternalServerError, apiResponse{OK: false, Error: "failed to apply config"})
return return
} }
r.logger.Printf("[INFO] [api] Config apply: rename failed, wrote directly (bind mount)")
}
r.logger.Printf("[INFO] [api] Config applied from Hub (%d bytes), restart needed to take effect", len(body)) r.logger.Printf("[INFO] [api] Config applied from Hub (%d bytes), restart needed to take effect", len(body))
writeJSON(w, http.StatusOK, apiResponse{OK: true, Message: "Config applied. Restart controller to apply changes."}) writeJSON(w, http.StatusOK, apiResponse{OK: true, Message: "Config applied. Restart controller to apply changes."})
@@ -1030,6 +1029,27 @@ func (r *Router) configApply(w http.ResponseWriter, req *http.Request) {
} }
} }
// writeConfig0600 writes config bytes to path with mode 0600, atomically when possible (tmp+rename),
// falling back to a direct write for Docker bind mounts (where os.Rename returns EBUSY). It ALWAYS
// enforces 0600 on the final file — even if it already existed with looser perms — because controller.yaml
// holds infra credentials in plaintext (F8); os.WriteFile only applies the mode when creating a new file.
func writeConfig0600(path string, body []byte) error {
tmpPath := path + ".tmp"
if err := os.WriteFile(tmpPath, body, 0600); err != nil {
return fmt.Errorf("writing temp config: %w", err)
}
if err := os.Rename(tmpPath, path); err != nil {
os.Remove(tmpPath)
if err := os.WriteFile(path, body, 0600); err != nil { // bind-mount fallback
return fmt.Errorf("writing config: %w", err)
}
}
if err := os.Chmod(path, 0600); err != nil {
return fmt.Errorf("chmod config 0600: %w", err)
}
return nil
}
func (r *Router) configHash(w http.ResponseWriter, _ *http.Request) { func (r *Router) configHash(w http.ResponseWriter, _ *http.Request) {
hash, err := config.FileHash(r.configPath) hash, err := config.FileHash(r.configPath)
if err != nil { if err != nil {
+108
View File
@@ -2,6 +2,7 @@ package appbackup
import ( import (
"bufio" "bufio"
"compress/gzip"
"context" "context"
"fmt" "fmt"
"io" "io"
@@ -513,6 +514,113 @@ func populateDBEnv(ctx context.Context, db *DiscoveredDB) error {
return nil return nil
} }
// ImportDump replays a (possibly gzipped) SQL dump into a RUNNING database container — the read-side
// counterpart to DumpOne (F17). It reuses the per-engine clients (psql / mariadb) and the DiscoveredDB's
// OWN credentials (discovered from the live container env), so the caller needs no external env map. The
// container must already be running (the restore flow brings the stack up first); ImportDump briefly
// waits for the engine to accept connections, then pipes the dump in. The backup dumps are produced with
// DROP/CREATE (pg_dump --clean --if-exists; mariadb-dump's default --add-drop-table), so a replay fully
// reconstructs the captured logical state.
func ImportDump(ctx context.Context, db DiscoveredDB, dumpPath string, logger *log.Logger, debug bool) error {
if err := waitDBReady(ctx, db, 30*time.Second); err != nil {
return fmt.Errorf("waiting for %s (%s) readiness: %w", db.ContainerName, db.DBType, err)
}
f, err := os.Open(dumpPath)
if err != nil {
return fmt.Errorf("opening dump %s: %w", dumpPath, err)
}
defer f.Close()
var reader io.Reader = f
if strings.HasSuffix(dumpPath, ".gz") {
gr, err := gzip.NewReader(f)
if err != nil {
return fmt.Errorf("opening gzip %s: %w", dumpPath, err)
}
defer gr.Close()
reader = gr
}
impCtx, cancel := context.WithTimeout(ctx, 30*time.Minute)
defer cancel()
var cmd *exec.Cmd
switch db.DBType {
case DBTypePostgres:
user := db.DBUser
if user == "" {
user = "postgres"
}
dbName := db.DBName
if dbName == "" {
dbName = user
}
// ON_ERROR_STOP=1: a real import error must FAIL (and surface), not silently half-apply.
cmd = exec.CommandContext(impCtx, "docker", "exec", "-i", db.ContainerID,
"psql", "-v", "ON_ERROR_STOP=1", "-U", user, "-d", dbName)
case DBTypeMariaDB:
password := getMariaDBPassword(impCtx, db.ContainerID)
if password == "" {
return fmt.Errorf("could not determine MariaDB root password for %s", db.ContainerName)
}
cmd = exec.CommandContext(impCtx, "docker", "exec", "-i", db.ContainerID,
"mariadb", "-u", "root", "-p"+password, db.DBName)
default:
return fmt.Errorf("unsupported DB type: %s", db.DBType)
}
cmd.Stdin = reader
var stderr strings.Builder
cmd.Stderr = &stderr
if debug && logger != nil {
logger.Printf("[DEBUG] [backup] ImportDump: importing %s into %s (%s)", dumpPath, db.ContainerName, db.DBType)
}
if err := cmd.Run(); err != nil {
msg := strings.TrimSpace(stderr.String())
if len(msg) > 300 {
msg = msg[:300]
}
return fmt.Errorf("%s import into %s failed: %s — %w", db.DBType, db.ContainerName, msg, err)
}
if logger != nil {
logger.Printf("[INFO] [backup] Imported DB dump %s into %s (%s)", filepath.Base(dumpPath), db.ContainerName, db.DBType)
}
return nil
}
// waitDBReady polls until the database accepts connections (pg_isready / mariadb-admin ping).
func waitDBReady(ctx context.Context, db DiscoveredDB, timeout time.Duration) error {
deadline := time.Now().Add(timeout)
for {
c, cancel := context.WithTimeout(ctx, 5*time.Second)
var cmd *exec.Cmd
switch db.DBType {
case DBTypePostgres:
user := db.DBUser
if user == "" {
user = "postgres"
}
cmd = exec.CommandContext(c, "docker", "exec", db.ContainerID, "pg_isready", "-U", user)
case DBTypeMariaDB:
pw := getMariaDBPassword(c, db.ContainerID)
cmd = exec.CommandContext(c, "docker", "exec", db.ContainerID, "mariadb-admin", "ping", "-u", "root", "-p"+pw)
default:
cancel()
return fmt.Errorf("unsupported DB type: %s", db.DBType)
}
err := cmd.Run()
cancel()
if err == nil {
return nil
}
if time.Now().After(deadline) {
return fmt.Errorf("timeout after %s", timeout)
}
time.Sleep(2 * time.Second)
}
}
func getMariaDBPassword(ctx context.Context, containerID string) string { func getMariaDBPassword(ctx context.Context, containerID string) string {
cmd := exec.CommandContext(ctx, "docker", "inspect", containerID, cmd := exec.CommandContext(ctx, "docker", "inspect", containerID,
"--format", "{{range .Config.Env}}{{println .}}{{end}}") "--format", "{{range .Config.Env}}{{println .}}{{end}}")
@@ -59,6 +59,11 @@ func DumpOne(ctx context.Context, db DiscoveredDB, dumpDir string, logger *log.L
return appbackup.DumpOne(ctx, db, dumpDir, logger, debug) return appbackup.DumpOne(ctx, db, dumpDir, logger, debug)
} }
// ImportDump replays a captured .sql dump back into a running DB container (F17 restore path).
func ImportDump(ctx context.Context, db DiscoveredDB, dumpPath string, logger *log.Logger, debug bool) error {
return appbackup.ImportDump(ctx, db, dumpPath, logger, debug)
}
func ValidateDump(filePath string, dbType DBType) DumpValidation { func ValidateDump(filePath string, dbType DBType) DumpValidation {
return appbackup.ValidateDump(filePath, dbType) return appbackup.ValidateDump(filePath, dbType)
} }
+5
View File
@@ -31,6 +31,11 @@ type Manager struct {
// tier2Notify, if set, is called after each Tier 2 copy (success: err==nil) for notifications. // tier2Notify, if set, is called after each Tier 2 copy (success: err==nil) for notifications.
tier2Notify func(stackName, destLabel string, dur time.Duration, err error) tier2Notify func(stackName, destLabel string, dur time.Duration, err error)
// F17 restore seams — overridable in tests so the .sql re-import orchestration can be unit-tested
// without Docker. Default to the real DiscoverDatabases / ImportDump (lazy-init in reimportDBDumps).
discoverDBs func(ctx context.Context) ([]DiscoveredDB, error)
importDBDump func(ctx context.Context, db DiscoveredDB, dumpPath string) error
mu sync.Mutex mu sync.Mutex
lastDBDump *DBDumpStatus lastDBDump *DBDumpStatus
running bool running bool
+28 -3
View File
@@ -56,12 +56,17 @@ func (m *Manager) RestoreApp(stackName, snapshotID string) error {
m.logger.Printf("[WARN] RESTORE could not stop %s: %v (proceeding anyway)", stackName, err) m.logger.Printf("[WARN] RESTORE could not stop %s: %v (proceeding anyway)", stackName, err)
} }
// F17: surface a data-restore failure instead of swallowing it. We still bring the app back up so it
// isn't left dead, but the error is returned at the end so a failed restore can't read as success.
var dataErr error
// Populate Docker volumes from restored tars // Populate Docker volumes from restored tars
if m.isDebug() { if m.isDebug() {
m.logger.Printf("[DEBUG] RestoreApp: step 2/3 — restoring Docker volumes for %s", stackName) m.logger.Printf("[DEBUG] RestoreApp: step 2/3 — restoring Docker volumes for %s", stackName)
} }
if err := m.restoreDockerVolumes(stackName, drivePath); err != nil { if err := m.restoreDockerVolumes(stackName, drivePath); err != nil {
m.logger.Printf("[WARN] RESTORE volume restore failed for %s: %v (continuing)", stackName, err) m.logger.Printf("[ERROR] RESTORE volume restore failed for %s: %v", stackName, err)
dataErr = err
} }
// Restart the app // Restart the app
@@ -72,11 +77,23 @@ func (m *Manager) RestoreApp(stackName, snapshotID string) error {
m.logger.Printf("[WARN] RESTORE could not restart %s after restore: %v", stackName, err) m.logger.Printf("[WARN] RESTORE could not restart %s after restore: %v", stackName, err)
} }
// F17: replay the captured .sql dump into the now-running DB (the legacy path never did this, so
// DB-resident data did not come back). Runs after volume restore so the dump WINS over any tar copy.
if _, err := m.reimportDBDumpsCtx(stackName, m.namespaceRoot(drivePath)); err != nil {
m.logger.Printf("[ERROR] RESTORE DB re-import failed for %s: %v", stackName, err)
if dataErr == nil {
dataErr = err
}
}
// Verify app started successfully // Verify app started successfully
if err := m.waitForHealthy(stackName, 90*time.Second); err != nil { if err := m.waitForHealthy(stackName, 90*time.Second); err != nil {
m.logger.Printf("[WARN] [backup] Restore completed but app health check failed: %v", err) m.logger.Printf("[WARN] [backup] Restore completed but app health check failed: %v", err)
} }
if dataErr != nil {
return fmt.Errorf("restore of %s completed with data errors: %w", stackName, dataErr)
}
m.logger.Printf("[INFO] RESTORE completed: stack=%s", stackName) m.logger.Printf("[INFO] RESTORE completed: stack=%s", stackName)
return nil return nil
} }
@@ -93,6 +110,7 @@ func (m *Manager) restoreDockerVolumes(stackName, drivePath string) error {
} }
var restored int var restored int
var failed []string
for _, entry := range entries { for _, entry := range entries {
if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".tar") { if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".tar") {
continue continue
@@ -106,7 +124,8 @@ func (m *Manager) restoreDockerVolumes(stackName, drivePath string) error {
// Create fresh volume // Create fresh volume
if out, err := exec.Command("docker", "volume", "create", volName).CombinedOutput(); err != nil { if out, err := exec.Command("docker", "volume", "create", volName).CombinedOutput(); err != nil {
m.logger.Printf("[WARN] [backup] Failed to create volume %s: %s — %v", volName, strings.TrimSpace(string(out)), err) m.logger.Printf("[ERROR] [backup] Failed to create volume %s: %s — %v", volName, strings.TrimSpace(string(out)), err)
failed = append(failed, volName)
continue continue
} }
@@ -120,7 +139,8 @@ func (m *Manager) restoreDockerVolumes(stackName, drivePath string) error {
cancel() cancel()
if err != nil { if err != nil {
m.logger.Printf("[WARN] [backup] Failed to populate volume %s: %s — %v", volName, strings.TrimSpace(string(out)), err) m.logger.Printf("[ERROR] [backup] Failed to populate volume %s: %s — %v", volName, strings.TrimSpace(string(out)), err)
failed = append(failed, volName)
continue continue
} }
@@ -133,6 +153,11 @@ func (m *Manager) restoreDockerVolumes(stackName, drivePath string) error {
if restored > 0 { if restored > 0 {
m.logger.Printf("[INFO] [backup] Restored %d Docker volume(s) for %s", restored, stackName) m.logger.Printf("[INFO] [backup] Restored %d Docker volume(s) for %s", restored, stackName)
} }
// F17: a per-volume failure used to be a swallowed WARN; surface it so the restore is reported as
// failed rather than silently partial.
if len(failed) > 0 {
return fmt.Errorf("failed to restore %d volume(s): %v", len(failed), failed)
}
return nil return nil
} }
+90
View File
@@ -0,0 +1,90 @@
package backup
import (
"context"
"fmt"
"os"
"path/filepath"
"time"
)
// reimportDBDumps replays the captured per-app .sql dumps back into the app's now-running database
// container(s) — the F17 fix. The per-app backup captures a logical SQL dump (DumpOne →
// <stack>-<dbtype>.sql) but the legacy restore only repopulated Docker volume tars and NEVER replayed
// the dump, so DB-resident data (e.g. rows in a DB whose data dir is a bind mount, not a named volume)
// did not come back. This runs AFTER volume restore + stack bring-up, so the dump WINS over any
// volume-tar copy of the DB (the operator-chosen precedence: the consistent logical dump is authoritative).
//
// It uses the live container's OWN discovered credentials (DiscoveredDB), so no env threading is needed.
// A dump whose DB container is not found is logged and skipped; an actual import FAILURE is returned
// (surfaced, not swallowed) so a failed data restore cannot read as success.
func (m *Manager) reimportDBDumps(ctx context.Context, stackName, nsRoot string) (int, error) {
dumpDir := AppDBDumpPath(nsRoot, stackName)
entries, err := os.ReadDir(dumpDir)
if err != nil {
if os.IsNotExist(err) {
return 0, nil // no DB dumps for this app
}
return 0, fmt.Errorf("reading db-dump dir: %w", err)
}
hasDump := false
for _, e := range entries {
if !e.IsDir() && filepath.Ext(e.Name()) == ".sql" {
hasDump = true
break
}
}
if !hasDump {
return 0, nil
}
discover := m.discoverDBs
if discover == nil {
discover = func(ctx context.Context) ([]DiscoveredDB, error) {
return DiscoverDatabases(ctx, m.logger, m.isDebug())
}
}
imp := m.importDBDump
if imp == nil {
imp = func(ctx context.Context, db DiscoveredDB, dumpPath string) error {
return ImportDump(ctx, db, dumpPath, m.logger, m.isDebug())
}
}
dbs, err := discover(ctx)
if err != nil {
return 0, fmt.Errorf("discovering DB containers for %s: %w", stackName, err)
}
var imported int
for _, db := range dbs {
if db.StackName != stackName {
continue
}
// The dump for this DB is named "<stack>-<dbtype>.sql" (see DumpOne).
dumpPath := filepath.Join(dumpDir, fmt.Sprintf("%s-%s.sql", stackName, db.DBType))
if _, statErr := os.Stat(dumpPath); statErr != nil {
continue // no dump for this particular DB engine
}
m.logger.Printf("[INFO] [backup] Restore %s: replaying DB dump into %s (%s)", stackName, db.ContainerName, db.DBType)
if err := imp(ctx, db, dumpPath); err != nil {
return imported, fmt.Errorf("importing %s dump for %s: %w", db.DBType, stackName, err)
}
imported++
}
if imported == 0 {
m.logger.Printf("[WARN] [backup] Restore %s: a .sql dump exists but no matching running DB container was found — DB content NOT restored", stackName)
} else {
m.logger.Printf("[INFO] [backup] Restore %s: replayed %d DB dump(s)", stackName, imported)
}
return imported, nil
}
// reimportDBDumpsCtx is a small helper that runs reimportDBDumps with a bounded context so a stuck DB
// import cannot hang the restore indefinitely.
func (m *Manager) reimportDBDumpsCtx(stackName, nsRoot string) (int, error) {
ctx, cancel := context.WithTimeout(context.Background(), 35*time.Minute)
defer cancel()
return m.reimportDBDumps(ctx, stackName, nsRoot)
}
@@ -0,0 +1,125 @@
package backup
import (
"context"
"fmt"
"io"
"log"
"os"
"path/filepath"
"strings"
"testing"
)
func newReimportTestManager() *Manager {
return &Manager{logger: log.New(io.Discard, "", 0)}
}
func writeDump(t *testing.T, nsRoot, stack string, dbType DBType) string {
t.Helper()
dir := AppDBDumpPath(nsRoot, stack)
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
p := filepath.Join(dir, fmt.Sprintf("%s-%s.sql", stack, dbType))
if err := os.WriteFile(p, []byte("-- dump\nDROP TABLE IF EXISTS t;\n"), 0o644); err != nil {
t.Fatal(err)
}
return p
}
// TestReimportDBDumps_ImportsWhenDumpAndDBPresent asserts F17: when a captured .sql dump exists and a
// matching running DB is discovered, reimportDBDumps replays it. The PRE-FIX restore never called any
// import — this orchestration is the fix.
func TestReimportDBDumps_ImportsWhenDumpAndDBPresent(t *testing.T) {
nsRoot := t.TempDir()
wantPath := writeDump(t, nsRoot, "app", DBTypeMariaDB)
m := newReimportTestManager()
m.discoverDBs = func(ctx context.Context) ([]DiscoveredDB, error) {
return []DiscoveredDB{
{StackName: "other", DBType: DBTypePostgres, ContainerName: "other-db"},
{StackName: "app", DBType: DBTypeMariaDB, ContainerName: "app-db", ContainerID: "cid"},
}, nil
}
var gotPath string
var gotDB DiscoveredDB
m.importDBDump = func(ctx context.Context, db DiscoveredDB, dumpPath string) error {
gotPath, gotDB = dumpPath, db
return nil
}
n, err := m.reimportDBDumps(context.Background(), "app", nsRoot)
if err != nil {
t.Fatalf("reimportDBDumps: %v", err)
}
if n != 1 {
t.Fatalf("imported = %d, want 1", n)
}
if gotPath != wantPath {
t.Fatalf("imported path = %q, want %q", gotPath, wantPath)
}
if gotDB.ContainerName != "app-db" {
t.Fatalf("imported into %q, want app-db (must match the stack's own DB)", gotDB.ContainerName)
}
}
// TestReimportDBDumps_FailureSurfaces asserts an import failure is RETURNED, not swallowed (a failed
// data restore must not read as success).
func TestReimportDBDumps_FailureSurfaces(t *testing.T) {
nsRoot := t.TempDir()
writeDump(t, nsRoot, "app", DBTypeMariaDB)
m := newReimportTestManager()
m.discoverDBs = func(ctx context.Context) ([]DiscoveredDB, error) {
return []DiscoveredDB{{StackName: "app", DBType: DBTypeMariaDB, ContainerName: "app-db"}}, nil
}
m.importDBDump = func(ctx context.Context, db DiscoveredDB, dumpPath string) error {
return fmt.Errorf("boom")
}
if _, err := m.reimportDBDumps(context.Background(), "app", nsRoot); err == nil || !strings.Contains(err.Error(), "boom") {
t.Fatalf("expected the import failure to surface, got %v", err)
}
}
// TestReimportDBDumps_NoDumpNoImport asserts apps with no .sql dump never trigger discovery/import.
func TestReimportDBDumps_NoDumpNoImport(t *testing.T) {
nsRoot := t.TempDir()
m := newReimportTestManager()
discoverCalled := false
m.discoverDBs = func(ctx context.Context) ([]DiscoveredDB, error) {
discoverCalled = true
return nil, nil
}
m.importDBDump = func(ctx context.Context, db DiscoveredDB, dumpPath string) error {
t.Fatal("importDBDump must not be called when there is no dump")
return nil
}
n, err := m.reimportDBDumps(context.Background(), "app", nsRoot)
if err != nil || n != 0 {
t.Fatalf("reimportDBDumps with no dump = (%d, %v), want (0, nil)", n, err)
}
if discoverCalled {
t.Fatalf("discovery should be skipped when there is no .sql dump")
}
}
// TestReimportDBDumps_DumpButNoMatchingDB asserts that a dump with no matching running DB container is a
// non-fatal skip (logged), returning 0 imported and no error — the app is up, just no DB matched.
func TestReimportDBDumps_DumpButNoMatchingDB(t *testing.T) {
nsRoot := t.TempDir()
writeDump(t, nsRoot, "app", DBTypeMariaDB)
m := newReimportTestManager()
m.discoverDBs = func(ctx context.Context) ([]DiscoveredDB, error) {
return []DiscoveredDB{{StackName: "different", DBType: DBTypeMariaDB}}, nil
}
m.importDBDump = func(ctx context.Context, db DiscoveredDB, dumpPath string) error {
t.Fatal("must not import when no DB matches the stack")
return nil
}
n, err := m.reimportDBDumps(context.Background(), "app", nsRoot)
if err != nil || n != 0 {
t.Fatalf("= (%d, %v), want (0, nil)", n, err)
}
}
+15 -1
View File
@@ -122,19 +122,33 @@ func (m *Manager) RestoreFromRecoveryUnit(stackName string) error {
stackName, len(manifest.ImagePins), len(manifest.SecretEnvVars)-len(missing), len(manifest.SecretEnvVars), len(manifest.DataKeyEnvVars)) stackName, len(manifest.ImagePins), len(manifest.SecretEnvVars)-len(missing), len(manifest.SecretEnvVars), len(manifest.DataKeyEnvVars))
// Stop, restore named-volume data, then recreate the definition + redeploy with the recovered env. // Stop, restore named-volume data, then recreate the definition + redeploy with the recovered env.
// F17: surface a data-restore failure instead of swallowing it (we still bring the app back up).
var dataErr error
if err := m.stackProvider.StopStack(stackName); err != nil { if err := m.stackProvider.StopStack(stackName); err != nil {
m.logger.Printf("[WARN] [backup] could not stop %s before restore: %v (continuing)", stackName, err) m.logger.Printf("[WARN] [backup] could not stop %s before restore: %v (continuing)", stackName, err)
} }
if err := m.restoreDockerVolumes(stackName, drivePath); err != nil { if err := m.restoreDockerVolumes(stackName, drivePath); err != nil {
m.logger.Printf("[WARN] [backup] volume restore for %s: %v (continuing)", stackName, err) m.logger.Printf("[ERROR] [backup] volume restore for %s: %v", stackName, err)
dataErr = err
} }
if err := m.stackProvider.RecreateStackFromUnit(stackName, composeDir, fullEnv); err != nil { if err := m.stackProvider.RecreateStackFromUnit(stackName, composeDir, fullEnv); err != nil {
return fmt.Errorf("recreating %s from unit: %w", stackName, err) return fmt.Errorf("recreating %s from unit: %w", stackName, err)
} }
// F17: the captured .sql dump is the authoritative logical DB state — replay it into the now-running
// DB container AFTER the volume restore, so the dump WINS over any volume-tar copy of the database.
if _, err := m.reimportDBDumpsCtx(stackName, nsRoot); err != nil {
m.logger.Printf("[ERROR] [backup] DB re-import for %s: %v", stackName, err)
if dataErr == nil {
dataErr = err
}
}
if err := m.waitForHealthy(stackName, 90*time.Second); err != nil { if err := m.waitForHealthy(stackName, 90*time.Second); err != nil {
m.logger.Printf("[WARN] [backup] %s restored but health check failed: %v", stackName, err) m.logger.Printf("[WARN] [backup] %s restored but health check failed: %v", stackName, err)
} }
if dataErr != nil {
return fmt.Errorf("restore of %s from unit completed with data errors: %w", stackName, dataErr)
}
m.logger.Printf("[INFO] [backup] Restore-from-unit completed: %s", stackName) m.logger.Printf("[INFO] [backup] Restore-from-unit completed: %s", stackName)
return nil return nil
} }
@@ -0,0 +1,112 @@
//go:build linux
package system
import (
"os"
"path/filepath"
"testing"
)
// TestReadMemInfoUsesCgroupV2Limit asserts F1: when a cgroup v2 memory.max caps the container well
// below the host /proc/meminfo total, readMemInfo reports the cgroup cap (the guest's real ceiling),
// not the host RAM. On the pre-fix code this test fails because readMemInfo ignored cgroup entirely.
func TestReadMemInfoUsesCgroupV2Limit(t *testing.T) {
dir := t.TempDir()
// 2 GiB limit, 512 MiB current usage.
const twoGiB = uint64(2 * 1024 * 1024 * 1024)
const halfGiB = uint64(512 * 1024 * 1024)
mustWriteFile(t, filepath.Join(dir, "memory.max"), []byte(itoa(twoGiB)))
mustWriteFile(t, filepath.Join(dir, "memory.current"), []byte(itoa(halfGiB)))
old := cgroupRoot
cgroupRoot = dir
defer func() { cgroupRoot = old }()
var info SystemInfo
readMemInfo(&info)
// Host /proc/meminfo total is whatever the test machine has; the cgroup cap must win when smaller.
if info.TotalMemMB != 2048 {
t.Fatalf("TotalMemMB = %d, want 2048 (cgroup cap), not host RAM", info.TotalMemMB)
}
if info.UsedMemMB != 512 {
t.Fatalf("UsedMemMB = %d, want 512 (memory.current)", info.UsedMemMB)
}
if info.AvailMemMB != 1536 {
t.Fatalf("AvailMemMB = %d, want 1536", info.AvailMemMB)
}
if info.MemPercent < 24 || info.MemPercent > 26 {
t.Fatalf("MemPercent = %.1f, want ~25", info.MemPercent)
}
}
// TestReadMemInfoCgroupMaxIsUnlimited asserts that a v2 "max" sentinel does NOT override /proc/meminfo
// (an uncapped container keeps the host view rather than a bogus 0).
func TestReadMemInfoCgroupMaxIsUnlimited(t *testing.T) {
dir := t.TempDir()
mustWriteFile(t, filepath.Join(dir, "memory.max"), []byte("max"))
old := cgroupRoot
cgroupRoot = dir
defer func() { cgroupRoot = old }()
var info SystemInfo
readMemInfo(&info)
if info.TotalMemMB == 0 {
t.Fatalf("TotalMemMB = 0 with an unlimited cgroup; expected the /proc/meminfo host total")
}
}
// TestReadCgroupMemLimitV1Unlimited asserts the v1 near-uint64-max sentinel is treated as unlimited.
func TestReadCgroupMemLimitV1Unlimited(t *testing.T) {
dir := t.TempDir()
memDir := filepath.Join(dir, "memory")
if err := os.MkdirAll(memDir, 0o755); err != nil {
t.Fatal(err)
}
// Typical v1 "unlimited" value.
mustWriteFile(t, filepath.Join(memDir, "memory.limit_in_bytes"), []byte("9223372036854771712"))
if _, ok := readCgroupMemLimitMB(dir); ok {
t.Fatalf("readCgroupMemLimitMB treated the v1 unlimited sentinel as a real limit")
}
}
// TestReadCgroupMemLimitV1Real asserts a finite v1 limit is read.
func TestReadCgroupMemLimitV1Real(t *testing.T) {
dir := t.TempDir()
memDir := filepath.Join(dir, "memory")
if err := os.MkdirAll(memDir, 0o755); err != nil {
t.Fatal(err)
}
const oneGiB = uint64(1024 * 1024 * 1024)
mustWriteFile(t, filepath.Join(memDir, "memory.limit_in_bytes"), []byte(itoa(oneGiB)))
mb, ok := readCgroupMemLimitMB(dir)
if !ok || mb != 1024 {
t.Fatalf("readCgroupMemLimitMB = (%d, %v), want (1024, true)", mb, ok)
}
}
func mustWriteFile(t *testing.T, path string, data []byte) {
t.Helper()
if err := os.WriteFile(path, data, 0o644); err != nil {
t.Fatal(err)
}
}
func itoa(v uint64) string {
if v == 0 {
return "0"
}
var buf [20]byte
i := len(buf)
for v > 0 {
i--
buf[i] = byte('0' + v%10)
v /= 10
}
return string(buf[i:])
}
+71 -3
View File
@@ -8,6 +8,7 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"sort" "sort"
"strconv"
"strings" "strings"
"syscall" "syscall"
"time" "time"
@@ -77,6 +78,9 @@ func GetMemoryMB() (totalMB, usedMB int, err error) {
return int(info.TotalMemMB), int(info.UsedMemMB), nil return int(info.TotalMemMB), int(info.UsedMemMB), nil
} }
// cgroupRoot is the cgroup mount point. Overridable in tests.
var cgroupRoot = "/sys/fs/cgroup"
func readMemInfo(info *SystemInfo) { func readMemInfo(info *SystemInfo) {
f, err := os.Open("/proc/meminfo") f, err := os.Open("/proc/meminfo")
if err != nil { if err != nil {
@@ -100,16 +104,80 @@ func readMemInfo(info *SystemInfo) {
} }
} }
if totalKB > 0 { if totalKB == 0 {
debugf("[DEBUG] [system] readMemInfo: could not parse MemTotal from /proc/meminfo")
return
}
info.TotalMemMB = totalKB / 1024 info.TotalMemMB = totalKB / 1024
info.AvailMemMB = availKB / 1024 info.AvailMemMB = availKB / 1024
info.UsedMemMB = info.TotalMemMB - info.AvailMemMB info.UsedMemMB = info.TotalMemMB - info.AvailMemMB
// F1: the controller runs as a Docker container inside an LXC. /proc/meminfo reports the HOST's
// RAM (no lxcfs in the container), which massively overstates the guest's real ceiling and defeats
// the deploy memory-headroom guard. Prefer the cgroup memory LIMIT when it is finite and below the
// host total — that is the amount this guest can actually use. Fall back to /proc/meminfo otherwise.
if limitMB, ok := readCgroupMemLimitMB(cgroupRoot); ok && limitMB > 0 && limitMB < info.TotalMemMB {
info.TotalMemMB = limitMB
if curMB, okC := readCgroupMemCurrentMB(cgroupRoot); okC && curMB <= limitMB {
info.UsedMemMB = curMB
} else if info.UsedMemMB > limitMB {
info.UsedMemMB = limitMB
}
info.AvailMemMB = info.TotalMemMB - info.UsedMemMB
debugf("[DEBUG] [system] readMemInfo: using cgroup limit=%dMB (host total was %dKB) → used=%dMB avail=%dMB",
limitMB, totalKB, info.UsedMemMB, info.AvailMemMB)
}
if info.TotalMemMB > 0 {
info.MemPercent = float64(info.UsedMemMB) / float64(info.TotalMemMB) * 100 info.MemPercent = float64(info.UsedMemMB) / float64(info.TotalMemMB) * 100
}
debugf("[DEBUG] [system] readMemInfo: totalKB=%d availKB=%d → total=%dMB avail=%dMB used=%dMB (%.1f%%)", debugf("[DEBUG] [system] readMemInfo: totalKB=%d availKB=%d → total=%dMB avail=%dMB used=%dMB (%.1f%%)",
totalKB, availKB, info.TotalMemMB, info.AvailMemMB, info.UsedMemMB, info.MemPercent) totalKB, availKB, info.TotalMemMB, info.AvailMemMB, info.UsedMemMB, info.MemPercent)
} else { }
debugf("[DEBUG] [system] readMemInfo: could not parse MemTotal from /proc/meminfo")
// readCgroupMemLimitMB returns the cgroup memory limit in MB. It tries cgroup v2 (memory.max) first,
// then v1 (memory/memory.limit_in_bytes). A sentinel ("max" on v2, or a near-uint64-max value on v1)
// means "unlimited" → ok=false so the caller keeps the /proc/meminfo value.
func readCgroupMemLimitMB(root string) (mb uint64, ok bool) {
// cgroup v2
if b, err := os.ReadFile(filepath.Join(root, "memory.max")); err == nil {
s := strings.TrimSpace(string(b))
if s == "max" {
return 0, false
} }
if v, err := strconv.ParseUint(s, 10, 64); err == nil && v > 0 {
return v / (1024 * 1024), true
}
}
// cgroup v1
if b, err := os.ReadFile(filepath.Join(root, "memory", "memory.limit_in_bytes")); err == nil {
s := strings.TrimSpace(string(b))
if v, err := strconv.ParseUint(s, 10, 64); err == nil && v > 0 {
// v1 "unlimited" is a huge page-aligned value near uint64 max; treat >= 1 PiB as unlimited.
if v >= (1 << 50) {
return 0, false
}
return v / (1024 * 1024), true
}
}
return 0, false
}
// readCgroupMemCurrentMB returns the cgroup current memory usage in MB (v2 memory.current, v1
// memory/memory.usage_in_bytes). ok=false if unreadable.
func readCgroupMemCurrentMB(root string) (mb uint64, ok bool) {
for _, p := range []string{
filepath.Join(root, "memory.current"),
filepath.Join(root, "memory", "memory.usage_in_bytes"),
} {
if b, err := os.ReadFile(p); err == nil {
if v, err := strconv.ParseUint(strings.TrimSpace(string(b)), 10, 64); err == nil {
return v / (1024 * 1024), true
}
}
}
return 0, false
} }
// parseMemLine extracts the kB value from a /proc/meminfo line like "MemTotal: 16384000 kB" // parseMemLine extracts the kB value from a /proc/meminfo line like "MemTotal: 16384000 kB"
+15
View File
@@ -30,6 +30,20 @@ func getTimezone() *time.Location {
return webTimezone return webTimezone
} }
// routeUnpublished reports whether the reverse proxy (Traefik) is withholding a deployed stack's public
// route because the container is not healthy. Traefik's docker provider only publishes a route to a
// container that is healthy (or has no healthcheck); an unhealthy or restarting container yields a 404
// at its URL even though the card "looks deployed". Templates use this to surface that distinctly (F5),
// so an unhealthy app with a dead URL isn't mistaken for a merely-degraded-but-reachable one.
func routeUnpublished(state stacks.ContainerState) bool {
switch state {
case stacks.StateUnhealthy, stacks.StateRestarting:
return true
default:
return false
}
}
// templateFuncMap returns the FuncMap used by all HTML templates. // templateFuncMap returns the FuncMap used by all HTML templates.
func (s *Server) templateFuncMap() template.FuncMap { func (s *Server) templateFuncMap() template.FuncMap {
loc := getTimezone() loc := getTimezone()
@@ -102,6 +116,7 @@ func (s *Server) templateFuncMap() template.FuncMap {
return false return false
} }
}, },
"routeUnpublished": routeUnpublished,
"logoURL": func(slug string) string { "logoURL": func(slug string) string {
return s.cfg.AppLogoURL(slug) return s.cfg.AppLogoURL(slug)
}, },
@@ -0,0 +1,78 @@
package web
import (
"bytes"
"html/template"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
)
// TestRouteUnpublished asserts F5: routeUnpublished is true exactly for the states where Traefik
// withholds the public route (unhealthy / restarting) and false otherwise.
func TestRouteUnpublished(t *testing.T) {
cases := []struct {
state stacks.ContainerState
want bool
}{
{stacks.StateUnhealthy, true},
{stacks.StateRestarting, true},
{stacks.StateRunning, false},
{stacks.StateStarting, false},
{stacks.StateDeploying, false},
{stacks.StateStopped, false},
{stacks.StateExited, false},
{stacks.StateNotDeployed, false},
{stacks.StatePaused, false},
}
for _, c := range cases {
if got := routeUnpublished(c.state); got != c.want {
t.Errorf("routeUnpublished(%q) = %v, want %v", c.state, got, c.want)
}
}
}
// TestTemplatesParseWithFuncmap asserts the real embedded templates (including the stacks.html /
// dashboard.html edits that reference routeUnpublished) parse with the production funcmap. Catches an
// unregistered func or a template syntax error introduced by the F5 edits.
func TestTemplatesParseWithFuncmap(t *testing.T) {
s := &Server{cfg: &config.Config{}}
if _, err := template.New("").Funcs(s.templateFuncMap()).ParseFS(templateFS, "templates/*.html"); err != nil {
t.Fatalf("templates failed to parse with funcmap: %v", err)
}
}
// TestRouteUnpublishedIndicatorRenders asserts the dashboard/stacks card guard renders the distinct
// indicator for a DEPLOYED + unhealthy stack, and NOT for a healthy one — the exact condition both
// edited templates use ({{if and .Deployed (routeUnpublished .State)}}).
func TestRouteUnpublishedIndicatorRenders(t *testing.T) {
s := &Server{cfg: &config.Config{}}
const frag = `{{if and .Deployed (routeUnpublished .State)}}URL-NOT-PUBLISHED{{end}}`
tmpl, err := template.New("frag").Funcs(s.templateFuncMap()).Parse(frag)
if err != nil {
t.Fatal(err)
}
type row struct {
Deployed bool
State stacks.ContainerState
}
render := func(r row) string {
var b bytes.Buffer
if err := tmpl.Execute(&b, r); err != nil {
t.Fatal(err)
}
return b.String()
}
if got := render(row{Deployed: true, State: stacks.StateUnhealthy}); !strings.Contains(got, "URL-NOT-PUBLISHED") {
t.Errorf("deployed+unhealthy should show the indicator, got %q", got)
}
if got := render(row{Deployed: true, State: stacks.StateRunning}); strings.Contains(got, "URL-NOT-PUBLISHED") {
t.Errorf("deployed+running must NOT show the indicator, got %q", got)
}
if got := render(row{Deployed: false, State: stacks.StateUnhealthy}); strings.Contains(got, "URL-NOT-PUBLISHED") {
t.Errorf("not-deployed must NOT show the indicator, got %q", got)
}
}
@@ -149,6 +149,7 @@
<div class="stack-actions"> <div class="stack-actions">
<span class="stack-state-label">{{stateLabel .State}}</span> <span class="stack-state-label">{{stateLabel .State}}</span>
{{if .Orphaned}}<span class="badge badge-orphaned">Elavult</span>{{end}} {{if .Orphaned}}<span class="badge badge-orphaned">Elavult</span>{{end}}
{{if and .Deployed (routeUnpublished .State)}}<span class="badge badge-route-unpublished" title="A proxy (Traefik) csak egészséges konténerhez publikál nyilvános útvonalat. Amíg az alkalmazás nem egészséges, az URL 404-et ad, pedig a konténer fut.">⚠ URL nem elérhető</span>{{end}}
{{if .Protected}} {{if .Protected}}
<span class="badge badge-protected">Védett</span> <span class="badge badge-protected">Védett</span>
@@ -30,6 +30,9 @@
<a class="subdomain-link" href="https://{{$subdomain}}.{{$.Domain}}" target="_blank"> <a class="subdomain-link" href="https://{{$subdomain}}.{{$.Domain}}" target="_blank">
{{$subdomain}}.{{$.Domain}} ↗ {{$subdomain}}.{{$.Domain}} ↗
</a> </a>
{{if and .Deployed (routeUnpublished .State)}}
<span class="route-unpublished" title="A proxy (Traefik) csak egészséges konténerhez publikál nyilvános útvonalat. Amíg az alkalmazás nem egészséges, az URL nem érhető el (404), pedig a konténer fut.">⚠ URL nem elérhető útvonal nincs publikálva</span>
{{end}}
{{end}} {{end}}
</div> </div>
</div> </div>
@@ -1249,6 +1249,20 @@ a.stat-card:hover {
color: var(--orange); color: var(--orange);
} }
/* F5: an unhealthy/restarting deployed app has its public route withheld by Traefik (404 at the URL). */
.badge-route-unpublished {
background: var(--orange-bg);
color: var(--orange);
white-space: nowrap;
}
.route-unpublished {
display: inline-block;
margin-top: 2px;
font-size: 0.8em;
color: var(--orange);
white-space: nowrap;
}
/* Delete modal */ /* Delete modal */
.modal-overlay { .modal-overlay {
position: fixed; position: fixed;