Compare commits
23 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| f2854fcad1 | |||
| 84979932e7 | |||
| 4989513a96 | |||
| d6c428b5dd | |||
| 0b9450e356 | |||
| 803ce50578 | |||
| 68684892d8 | |||
| 4938cc8985 | |||
| 56fe5749a5 | |||
| 2cf3fadaec | |||
| 0550b3117e | |||
| 8324ed0dc2 | |||
| 368f5efea1 | |||
| 6c0d20d7d1 | |||
| 3b98c7309a | |||
| 20463dbaa0 | |||
| d3ac7fb118 | |||
| 5954763ed7 | |||
| 6197eadeee | |||
| a95168e160 | |||
| d2ce148910 | |||
| ddc1ed7fab | |||
| ce7f44421f |
@@ -1,5 +1,51 @@
|
||||
## Changelog
|
||||
|
||||
### v0.61.0 — live-drive Batch 1 (+F17) fixes (2026-06-14)
|
||||
|
||||
Controller-side fixes triaged in `LIVE-DRIVE-FIXSPEC-2026-06-14.md` from the 2026-06-14 live-drive
|
||||
findings. Each fix has a regression test that fails on the pre-fix code. Built, deployed to demo guest
|
||||
9201, and the key fixes live-verified. (F9, F20-BUG2, F20-BUG3 are the SUPERVISED agent/golden next
|
||||
session — not in this batch.)
|
||||
|
||||
- **F17 (CRITICAL) — per-app restore now replays the captured `.sql` DB dump.** `RestoreFromRecoveryUnit`
|
||||
(and the `RestoreApp` fallback) repopulated Docker volume tars but NEVER replayed the captured
|
||||
`<stack>-<dbtype>.sql`, so DB-resident data did not come back. New `appbackup.ImportDump` (read-side
|
||||
counterpart to `DumpOne`, reuses `DiscoveredDB`'s own discovered credentials) + `backup.reimportDBDumps`
|
||||
replay the dump AFTER volume restore + stack bring-up, so the logical dump **wins** over any volume-tar
|
||||
copy of the DB (operator-chosen precedence). Volume-restore and DB-import failures now **surface** (the
|
||||
restore returns an error) instead of a swallowed WARN. **Live-validated** on guest 9201: a marker row
|
||||
dropped after backup was restored by `/backup/restore` (log: "replayed 1 DB dump(s)"). Reuse note:
|
||||
`ImportDump` lives in `appbackup` (the DB-domain package) — `appexport→appbackup` already exists so
|
||||
reusing appexport's unexported copies would cycle; appbackup is the clean shared home.
|
||||
- **F1 (HIGH) — guest RAM cap read from the Docker daemon; deploy guard uses committed memory.** The
|
||||
controller container reported the Proxmox host's 16 GB (no lxcfs in the container; its own cgroup is
|
||||
unlimited — the 2 GB cap is on the LXC ancestor), defeating the deploy memory-headroom hard-block.
|
||||
`system` now sources the cap from `docker info` MemTotal (the daemon runs in the LXC → reports the
|
||||
guest's real RAM; cgroup limit still preferred when present). The deploy guard now uses the controller's
|
||||
own committed-app memory (sum of running mem requests) for "used" — accurate and cheap — instead of
|
||||
host RSS. `/api/system/info` reports the guest cap + committed used. **Live-verified:** `total_mem_mb`
|
||||
2048 (was 15771).
|
||||
- **F20-BUG1 (HIGH) — `agentapi.FormatDisk` surfaces the agent's error.** A failed format (agent 502
|
||||
"device is mounted", `ok:false`, `data:null`) fell through to `return out, nil`, so the web layer
|
||||
reported a zero-value result as `ok:true` — a failed DESTRUCTIVE op read as success. Now returns a
|
||||
non-nil error on any non-2xx/`ok:false` that is not a recognized refusal (403/needs-confirmation).
|
||||
- **F5 (HIGH) — broken healthcheck → 404, two parts.** (catalog, `app-catalog-felhom.eu`) uptime-kuma's
|
||||
healthcheck pointed at a v1-era `node /app/extra/healthcheck.mjs` absent in `:2`, so the container
|
||||
stayed unhealthy and Traefik withheld the route (404 though running) — fixed to the v2 compiled
|
||||
`extra/healthcheck` binary + 180s start_period. (dashboard) new `routeUnpublished` helper + a distinct
|
||||
"URL nem elérhető – útvonal nincs publikálva" indicator on the dashboard/stacks cards for
|
||||
unhealthy/restarting deployed apps (operator decision: keep gating the route, surface it distinctly).
|
||||
**Live-verified:** uptime-kuma healthy → route publishes → status URL 302 (was 404).
|
||||
- **F8 (LOW-MED) — `controller.yaml` persisted 0600.** It holds infra credentials (cf/hub tokens) in
|
||||
plaintext; the Hub config-apply handler wrote 0644. New `writeConfig0600` enforces 0600 even on a
|
||||
pre-existing 0644 file.
|
||||
- **F6 (LOW) — deploy POST reports "started", not "deployed".** The deploy runs async (UI polls); the
|
||||
POST now returns 202 Accepted + "Telepítés elindítva…" so API/script consumers aren't told a deploy
|
||||
finished before it has.
|
||||
- **F7 (LOW) — dashboard state lag.** `status-refresh` tightened 30s → 10s (cheap docker-ps refresh).
|
||||
- **F4 (TRIVIAL) — `GET /api/stacks/rescan`** now returns 405 + `Allow: POST` instead of the misleading
|
||||
"stack not found: rescan" fall-through.
|
||||
|
||||
### v0.60.0 — M25 data-race fix (backlog-Medium cleanup) (2026-06-13)
|
||||
|
||||
Backlog-Medium reconciliation from the 2026-06-13 BUGHUNT reconcile. M4/M5/M6 verified already FIXED
|
||||
|
||||
@@ -1,56 +1,56 @@
|
||||
# REPORT — felhom-controller v0.58.0 (infra-protection prevention layer for the OS/Docker-data split)
|
||||
# REPORT — live-drive Batch 1 (+F17) → controller v0.61.0 (2026-06-14)
|
||||
|
||||
Phase 2 of the OS/Docker-data storage-split slice (Phase 1 = felhom-agent v0.29.0: golden + provision).
|
||||
The controller guest's OS rootfs and Docker data are now split onto separate `local-lvm` volumes for
|
||||
resilience; infra (controller/traefik/cloudflared/filebrowser) shares the one Docker data-root and is
|
||||
protected by **prevention, not placement**. Built, deployed, and **live-validated on a freshly
|
||||
re-provisioned guest 9201**.
|
||||
Implemented the controller-side fixes from `LIVE-DRIVE-FIXSPEC-2026-06-14.md` (Batch 1 + the CRITICAL
|
||||
F17), shipped as **v0.61.0** to demo guest 9201. Each fix has a regression test that fails on the
|
||||
pre-fix code; the authoritative `go build && go vet && go test ./...` is green on the Linux build server.
|
||||
|
||||
## What shipped (v0.58.0)
|
||||
- **Reserved-buffer headroom guard** (`internal/system/dockervol.go`): `GetDockerVolumeHeadroom()`
|
||||
measures the Docker-data volume via `statfs("/")` — the controller container's root overlay is the
|
||||
upperdir on the guest's `/var/lib/docker` volume (true with the **overlay2** driver; see the agent
|
||||
report), so `/` reports the data volume. Reserve floor `DockerVolumeReserveGB = max(5 GB, 10%)`.
|
||||
Fail-open on a measurement error.
|
||||
- **Deploy-time hard gate** (`internal/api/router.go` `deployStack`): a new deploy is **refused (HTTP
|
||||
507** + Hungarian message) when free space on the Docker-data volume is at/under the reserved buffer.
|
||||
- **Deploy-page surfacing** (`deploy.html`): a new deploy below the buffer shows a Hungarian warning and
|
||||
**disables** the "Telepítés indítása" button; the API gate is the hard backstop.
|
||||
- **Runtime monitoring** (`monitor/healthcheck.go`): confirmed `DiskPercent` watches the Docker-data
|
||||
volume (statfs `/`); warn 80% / crit 90% trip ABOVE the 10%-free buffer, so the customer is warned
|
||||
before the gate engages. Clarifying comment added.
|
||||
- **Log rotation** baked into the golden's `daemon.json` (agent side; `max-size 10m`, `max-file 3`).
|
||||
- Tests: `DockerVolumeReserveGB` floor/scale.
|
||||
## Fixes (each its own commit)
|
||||
|
||||
## Live validation (guest 9201, freshly re-provisioned from the split golden)
|
||||
9201 was **destroyed and re-provisioned** from the new split golden (32 GB OS rootfs + 256 GB Docker-data
|
||||
volume, `backup=1`), via `felhom-agent --selftest=provision` + a reboot. The controller bootstrapped
|
||||
from baked images (no pull), **pulled its config from the hub** (catalog synced — 55 app defs — hub
|
||||
HTTP 200, CF token configured, hub report pushed). Then:
|
||||
| Finding | Sev | Change | Test | Live-verified |
|
||||
|---|---|---|---|---|
|
||||
| **F17** | CRIT | per-app restore now **replays the captured `.sql` dump** (new `appbackup.ImportDump` + `backup.reimportDBDumps`), AFTER volume restore so the **dump wins**; volume/DB failures now surface | `restore_db_test.go` (injectable seams) | **marker DB round-trip PASSED** on romm |
|
||||
| **F1** | HIGH | guest RAM cap from `docker info` (daemon runs in LXC); deploy guard "used" = committed memory | `info_cgroup_test.go` (cgroup + docker-info paths) | `total_mem_mb=2048` (was 15771) |
|
||||
| **F20-BUG1** | HIGH | `agentapi.FormatDisk` surfaces non-2xx/`ok:false` instead of zero-value success | `disks_test.go` (502→err) | unit |
|
||||
| **F5 (catalog)** | HIGH | uptime-kuma healthcheck → v2 `extra/healthcheck` binary + 180s start_period | — (catalog) | uptime-kuma **healthy → route 302** (was 404) |
|
||||
| **F5 (dashboard)** | HIGH | `routeUnpublished` helper + distinct indicator for unhealthy/restarting deployed apps | `route_unpublished_test.go` | render test |
|
||||
| **F8** | LOW-MED | `controller.yaml` persisted 0600 (holds infra tokens) | `config_perms_test.go` | unit (Linux) |
|
||||
| **F6** | LOW | deploy POST → 202 + "Telepítés elindítva" (was "deployed" before compose) | — | code/UI |
|
||||
| **F7** | LOW | `status-refresh` 30s → 10s | — | cadence |
|
||||
| **F4** | TRIV | `GET /api/stacks/rescan` → 405 + `Allow: POST` | — | `→ 405` |
|
||||
|
||||
- **Split layout:** controller image 0.58.0, **Storage Driver overlay2**, `Docker Root Dir
|
||||
/var/lib/docker`; images on the data volume (`/var/lib/docker/overlay2` 1.7 GB), `/var/lib/containerd`
|
||||
idle (380 K); `df`: `/` 935 MB/32 GB (4%, lean OS rootfs), `/var/lib/docker` 256 GB.
|
||||
- **Prevention gate (the headline):** with ample space the deploy page shows **no** gate banner; after
|
||||
`fallocate`-filling the data volume to 99% (3.2 GB free < 25.6 GB reserve), a `POST /api/stacks/.../deploy`
|
||||
returned **HTTP 507** with the Hungarian "Nincs elég szabad tárhely" message — proven on the real
|
||||
256 GB data volume.
|
||||
- **Regression:** `/`, `/stacks`, `/backups`, `/monitoring`, `/stacks/{n}/deploy`, `/stacks/{n}/backup`
|
||||
all HTTP 200; A1 host-storage list still ordered + friendly-labelled (felhom-usb → local-lvm → local
|
||||
→ felhom-pbs); A2 Tier-2 panel route serves.
|
||||
- **Deploy path + DB-on-data-volume (step 3):** deployed ActualBudget (HTTP 200, container up); its named
|
||||
volume landed at `/var/lib/docker/volumes/actualbudget_actualbudget_data` = the data volume.
|
||||
- **External access:** via Cloudflare the controller returns HTTP 200 for vmid 9201 (tunnel + traefik
|
||||
route healthy). (A local-DNS override on the dev machine points the hostname at a stale LAN IP — a
|
||||
red herring; the real public path works.)
|
||||
## Key implementation notes
|
||||
|
||||
## OS isolation (resilience — the reason for the split), proven on the provisioned guest
|
||||
Filling the Docker-data volume to 100% (239 GB) left the OS rootfs at 4% and fully writable, the guest
|
||||
healthy throughout — the data volume cannot starve the OS.
|
||||
- **F17 reuse decision:** `ImportDump` lives in `internal/appbackup` (the DB-domain package with
|
||||
`DiscoverDatabases`/`DumpOne`/`getMariaDBPassword`), NOT reusing `appexport`'s unexported copies —
|
||||
`appexport→appbackup` already exists, so the reverse would import-cycle. `DiscoveredDB` carries the live
|
||||
container's own creds, so no env threading. Precedence: replay runs after volume restore → `.sql` wins.
|
||||
- **F1 reroot:** the FIXSPEC's cgroup-limit approach was a **no-op** on the demo (the container's own
|
||||
cgroup is unlimited; the 2 GB cap is on the LXC ancestor, hidden; no lxcfs in the container). The
|
||||
working source is `docker info` MemTotal. The deploy guard's "used" switched from host RSS (not
|
||||
per-guest observable) to the controller's committed-memory accounting.
|
||||
|
||||
## Outstanding (demo restoration, not slice validation)
|
||||
- **RomM (HDD app) + USB re-enroll:** RomM's data is safe on the host USB (`/mnt/felhom-usb/felhom-data`,
|
||||
untouched by the re-provision). Restoring it is the slice-10 enroll flow (assign → guest-attach →
|
||||
reboot to activate the bind → register storage → deploy). With the split, the USB binds to a free slot
|
||||
(mp1+) since **mp0 is now the Docker-data volume** — no collision. Documented as the final restore step;
|
||||
not required for slice validation (ActualBudget covered the deploy path; the USB bind was not touched).
|
||||
## Deploy & verification
|
||||
|
||||
- Built+pushed `gitea.dooplex.hu/admin/felhom-controller:0.61.0`; deployed via the guest-9201
|
||||
golden/bootstrap mechanism (`/etc/felhom-controller-image` + restart bootstrap). Healthy, `/api/health`
|
||||
+ dashboard 200, no startup errors.
|
||||
- Live-verified: F1 (`total_mem_mb=2048`), F4 (405), F5 (uptime-kuma healthy → route 302), **F17 (marker
|
||||
row dropped after backup was restored by `/backup/restore`)**.
|
||||
|
||||
## Test approach
|
||||
|
||||
Controller code is largely `//go:build linux`; dev host is Windows. Per-commit gate =
|
||||
`CGO_ENABLED=0 GOOS=linux go build && go vet` locally (compiles code + tests); authoritative
|
||||
`go test ./...` ran on the Linux build server (all green, incl. the linux `system` tests).
|
||||
|
||||
## Not in this batch (SUPERVISED, next session — agent/golden)
|
||||
|
||||
**F9** (HDD provisioning/guest-attach + reporting), **F20-BUG2** (durable_id scheme), **F20-BUG3**
|
||||
(async mkfs). These touch the host agent, golden template, and bring-up (blast radius beyond the
|
||||
disposable demo).
|
||||
|
||||
## Final state
|
||||
|
||||
Demo guest 9201 on `:0.61.0`, healthy; agent unchanged. Apps: actualbudget, gokapi, mealie, privatebin,
|
||||
romm(+db,+redis), uptime-kuma (now **healthy**), vikunja — all healthy. felhom-usb still not attached to
|
||||
the guest (F9, deferred).
|
||||
|
||||
@@ -1,35 +0,0 @@
|
||||
# Session 2026-06-13 — backlog-Medium cleanup + doc currency + live-drive runbook
|
||||
|
||||
Working/checkpoint log. Commit/push after each Part. Controller-only (agent v0.30.0 left alone).
|
||||
|
||||
## Progress log
|
||||
|
||||
- 23:1x — Start. Controller main `d6aacf2`, demo 9201 on v0.59.0 healthy. Verified the 6 backlog Mediums.
|
||||
- **Backlog-Medium verdicts (commit d6aacf2):**
|
||||
- **M4** (SubdomainInUse I/O under RLock) — **FIXED** @ stacks/deploy.go:67-80 (collects dirs under RLock, RUnlock, then LoadAppConfig with lock released).
|
||||
- **M5** (MigrateEncryption encKey nil-check race) — **FIXED** @ stacks/manager.go:129-131 (nil-check now inside `m.mu.Lock()`; SetEncryptionKey writes under same lock).
|
||||
- **M6** (MigrateEncryption holds lock across I/O) — **FIXED-in-practice** @ stacks/manager.go:128-191 — lock still spans the I/O loop, but runs once at startup (main.go:154) strictly before scheduler (`sched.Start`) and HTTP server; nothing concurrent to block. Acceptable, no action.
|
||||
- **M18** (ListDumpFiles re-validates every dump every 5min) — **LIVE** @ appbackup/dbdump.go:469. Perf; cross-package-entangled (settings cache has no size/modtime; signature change + bridge). → escape-hatch branch.
|
||||
- **M19** (deriveStackName naive suffix-strip) — **LIVE** @ appbackup/dbdump.go:536-551. Low-incidence correctness (misattributes a stack literally named `<x>-cache`/`-db`); needs deployed-stack list injected into appbackup. → escape-hatch branch.
|
||||
- **M25** (Server integrationMgr data race) — **LIVE** @ web/server.go:119-120 (constructor launches `go SyncFileBrowserMounts`) vs :162 `SetIntegrationManager` (post-construction, unsynchronized) read at handlers.go:358,360,1433. Real race. → clean fix (atomic.Pointer), ship in v0.60.0.
|
||||
|
||||
## Plan
|
||||
- Ship **M25** (atomic.Pointer) in controller **v0.60.0** + concurrency `-race` test.
|
||||
- **M18 / M19** → `fix/m18-*` / `fix/m19-*` branches (pushed, NOT deployed) with implemented fix + notes (escape hatch: entangled cross-package changes, not forced at 2am).
|
||||
- Part 2 docs, Part 3 runbook.
|
||||
|
||||
## PART 1 — DONE
|
||||
- M4/M5/M6: FIXED (no action). M25: fixed (atomic.Pointer) `6953899`; CHANGELOG v0.60.0 `7dd1aba`.
|
||||
- M25 `-race` proven clean on the build server. M18/M19: notes + fix plan on `fix/m18-dump-validation-cache` / `fix/m19-stackname-crossref` (pushed, NOT deployed).
|
||||
- **Deployed v0.60.0** to demo 9201 (rollback ref v0.59.0). Verified: `0.60.0 Up (healthy)`, self-test 6/0, "Controller elindult (0.60.0)", public `/api/health` 200, no errors. No rollback.
|
||||
|
||||
## PART 2 — docs: DONE
|
||||
- `felhom-agent/CLAUDE.md`: stale v0.13.0 banner → v0.30.0 current-state summary (agent `238ec02`).
|
||||
- `felhom-controller/CLAUDE.md`: corrected the WRONG "bulk strip has NOT happened" status → de-priv DONE (slice 8C); `CONTEXT.md` → v0.60.0 banner (ctrl `c67c53b`).
|
||||
- Doc-gaps closed (felhom.eu `4c0eb2f`): hub v0.6.3→v0.11.0 (05 doc); onlyoffice:nextcloud occ internals (auth-hub-sync doc); metricsDBPath **verified** on the live guest (coincides with volume-backed data_dir → persists; hardcoding is latent fragility only) + storage doc note. REORG-NOTES updated (gaps 2/3 closed, 5 partial). Remaining gaps (volume-dump-consistency #1, selfUpdateAuthMiddleware #4, design-drafts 01/03/04) left flagged.
|
||||
- controller/README.md slim-down: DEFERRED again (1987-line rewrite; banner already points to central docs — safety/budget).
|
||||
|
||||
## PART 3 — runbook: DONE
|
||||
- `felhom-controller/RUNBOOK-e2e-live-drive.md` written for tomorrow's supervised drive. 8 sections, non-destructive first (baseline → deploy happy → **CTRL-T2-1 crash-window probe** → **CTRL-001 import + traversal-negative** → backup 3-2-1 → **AGENT-001 wipe-gate refusals**), then **[DESTRUCTIVE] restore** last. Each step: exact UI path/API call, what "good" looks like, what to watch for. All routes verified against current source.
|
||||
|
||||
## ALL PARTS DONE.
|
||||
@@ -1,30 +0,0 @@
|
||||
# Session 2026-06-13 — audit-fix + docs-centralize working log
|
||||
|
||||
Working/checkpoint file for the unattended fix+docs session. Commit/push after each Part.
|
||||
|
||||
## Progress log
|
||||
|
||||
- **Part 1 — controller fixes → deploy: DONE & VERIFIED.**
|
||||
- Fix 1 CTRL-001 (import path traversal) — `appexport/validate.go` + `UnmarshalManifest` guard + join-loop guards + regression test. Commit `c20ff56`.
|
||||
- Fix 2 CTRL-T2-1 + H10 (deploy.go slice) — transitional `Deployed:false` until compose succeeds; `SaveAppConfig` fail-closed on encrypt error. Regression tests. Commit `5a80739`.
|
||||
- Fix 3 M2 (init-only stackProvider, lock removed + documented). Commit `092cbbe`.
|
||||
- CHANGELOG v0.59.0. Commit `7c45fec`. Pushed to main (`eea235b..7c45fec`).
|
||||
- Green gate: `go build/vet/test ./...` all PASS before each commit.
|
||||
- **Deploy:** built `felhom-controller:0.59.0` on 192.168.0.180; deployed to guest **9201 (demo-felhom)** via the bootstrap mechanism (`/etc/felhom-controller-image` + `systemctl restart felhom-controller-bootstrap.service`). Rollback ref was `0.58.0`.
|
||||
- **Verify:** `docker ps` → `0.59.0 Up (healthy)`; self-test 6 passed/0 failed; "Controller elindult (0.59.0)"; hub report pushed; public URL `https://felhom.demo-felhom.eu/api/health` → HTTP 200 `{"ok":true}`. NO rollback needed.
|
||||
- Deploy-target note: CLAUDE.md's `/opt/docker` compose path is stale (re-platform). Live demo controller runs in LXC guest 9201 on felhom-pve, bootstrap-managed. felhotest (router.abonet.hu:33022) was unreachable (timeout) — skipped.
|
||||
|
||||
- **Part 2 — agent AGENT-001 fix (branch only): DONE.** Branch `fix/agent-001-wipe-durable-reresolve` @ `d96e5bd` (agent repo), pushed, NOT merged/deployed. `antiRetargetResolve` (resolve→re-derive+match→re-inspect, format the re-resolved device); injectable `reresolveWipe` seam; unit test covers mismatch/gone/blank/empty refusals; existing integration test green via stub. `AGENT-001-FIX-NOTES.md` on the branch (5-line review checklist). Full agent `go build/vet/test ./...` green.
|
||||
|
||||
- **Part 3 — audit/BUGHUNT records: DONE.** Audit records → `felhom.eu/documentation/audits/{deep-sweep,bughunt-reconcile}-2026-06-13.md` with per-finding remediation-status headers (felhom.eu `4320c01`). Controller root `BUGHUNT.md` reconciliation banner (H10 fixed, pointer) — controller `6cd1fc9`.
|
||||
|
||||
- **Part 4 — docs centralize + refresh: DONE.** `felhom.eu/documentation/` now has: top-level `README.md` index; `controller/` subtree (README, module-map superseding the v0.33 planning map, + 4 code-verified feature docs: deploy/stack-lifecycle, backup-architecture, storage-monitoring-metrics, auth-hub-sync-integrations); `REORG-NOTES-2026-06-13.md` (rationale + verification ledger + flagged doc-gaps). felhom.eu `21d0e7c`/`e68fe3e`. Controller `README.md` banner → central docs; `CONTEXT.md` banner refreshed to v0.59.0; stray `controller/mnt/` artifact removed — controller `86a61fb`. Agent/hub design-drafts left as-is (flagged drift in REORG-NOTES per the cardinal rule).
|
||||
|
||||
## Final state
|
||||
- Controller main: fixes + docs through `86a61fb`; v0.59.0 deployed+verified on demo 9201.
|
||||
- Agent: fix on branch `fix/agent-001-wipe-durable-reresolve` `d96e5bd` (PENDING REVIEW).
|
||||
- felhom.eu main: docs through `e68fe3e`.
|
||||
|
||||
## Assumptions
|
||||
- "The demo node" = guest 9201 (runs the v0.58.0 the task cited; bootstrap-managed). felhotest unreachable, not deployed there.
|
||||
- M2 included in v0.59.0 (trivial, zero-runtime-change) since Fixes 1-2 were green; no risk signal.
|
||||
+25
-6
@@ -160,12 +160,16 @@ The app catalog lives in a separate Git repository. The controller:
|
||||
- User-configurable inputs (admin password, language, storage path) remain editable
|
||||
- Section header prompts the user to note down any passwords they need
|
||||
3. `checkBeforeDeploy()` JS guard fetches live state first (prevents double-deploy from another tab)
|
||||
4. **Memory validation** uses real system memory from `/proc/meminfo`:
|
||||
- `usable_memory = total_ram - reserved_memory_mb` (default 384MB reserved)
|
||||
- `system.GetMemoryMB()` returns real-time total and used memory (not declared reservations)
|
||||
- Hard block if `used_mb + new_request > usable_memory`
|
||||
- `CommittedMemory()` (declared sum) still used for soft overcommit warning only
|
||||
- Deploy page shows real memory usage bar (not declared reservations)
|
||||
4. **Memory validation** (F1, v0.61.0): the controller runs as a Docker container inside an LXC, where
|
||||
`/proc/meminfo` shows the **Proxmox host's** RAM (no lxcfs in the container) and the container's own
|
||||
cgroup is unlimited (the guest cap lives on the LXC ancestor). So the guest cap is read from the
|
||||
**Docker daemon** (`system.GuestMemTotalMB()` → `docker info` MemTotal — the daemon runs in the LXC and
|
||||
reports the guest's lxcfs-backed RAM; the cgroup limit is preferred when present, e.g. non-nested):
|
||||
- `usable_memory = guest_cap - reserved_memory_mb` (default 384MB reserved)
|
||||
- Hard block if `committed_used + new_request > usable_memory`, where `committed_used` =
|
||||
`CommittedMemory()` (sum of running apps' mem requests) — the guest-wide RSS is not observable from
|
||||
the container, so the controller's own committed accounting is the accurate, cheap "used".
|
||||
- `/api/system/info` reports the guest cap as total and committed memory as used.
|
||||
4b. **Docker-data volume reserved-buffer gate (v0.58.0, storage-split prevention layer):** the OS rootfs
|
||||
and Docker data are split onto separate volumes; infra (controller/traefik/cloudflared/filebrowser)
|
||||
shares the one Docker data-root (`/var/lib/docker`) and is protected by **prevention, not placement**.
|
||||
@@ -265,6 +269,12 @@ When app templates are updated (e.g., a new `APP_KEY` secret is added to `.felho
|
||||
| Restarting | Yellow | "Ujrainditas..." | Restart loop |
|
||||
| Not deployed | Gray | "Nincs telepitve" | Compose file exists, not deployed |
|
||||
|
||||
**Route-unpublished indicator (F5, v0.61.0).** Traefik's Docker provider only publishes a route to a
|
||||
container that is healthy (or has no healthcheck), so an `unhealthy`/`restarting` deployed app returns a
|
||||
hard **404** at its URL even though the container is running. The `routeUnpublished` template helper
|
||||
(`funcmap.go`) drives a distinct "URL nem elérhető – útvonal nincs publikálva" indicator on the dashboard
|
||||
and stacks cards for such apps, so a dead URL isn't mistaken for a merely-degraded-but-reachable one.
|
||||
|
||||
#### Controller-side Health Probes (`internal/stacks/healthprobe.go`)
|
||||
|
||||
For apps that declare a `healthcheck:` section in `.felhom.yml`, the controller probes the container directly over the Docker network (both are on `traefik-public`). This complements Docker-level healthchecks and is the **only** health mechanism for distroless/scratch images that lack shell utilities.
|
||||
@@ -404,6 +414,15 @@ backups/primary/<app>/
|
||||
env comes from `StackDataProvider.GetStackRecoveryInfo` (excludes secret-named + encrypted values, so
|
||||
the capture never touches a secret). `data_key` fields are marked in `.felhom.yml`
|
||||
(`DeployField.DataKey`).
|
||||
- **Restore replays the DB dump (F17, v0.61.0).** `RestoreFromRecoveryUnit` (and the `RestoreApp`
|
||||
fallback) stops the app → restores named-volume tars → recreates the compose definition + redeploys
|
||||
with the recovered env → **replays each `db-dumps/*.sql` into the now-running DB** via
|
||||
`backup.reimportDBDumps` → `appbackup.ImportDump` (psql / mariadb client, using the live container's own
|
||||
discovered credentials). The DB replay runs AFTER the volume restore, so the **logical SQL dump wins**
|
||||
over any volume-tar copy of the database (the dumps use DROP/CREATE — `pg_dump --clean --if-exists`,
|
||||
`mariadb-dump` default `--add-drop-table` — so replay is idempotent). Volume-restore and DB-import
|
||||
failures now **surface** (restore returns an error) instead of a swallowed WARN. Prior to v0.61.0 the
|
||||
per-app restore never replayed the `.sql`, so DB-resident data did not come back.
|
||||
|
||||
#### Tier 2 — off-drive copy (Phase 3, v0.55.x)
|
||||
|
||||
|
||||
@@ -253,7 +253,11 @@ func main() {
|
||||
sched.SetDebug(cfg.Logging.Level == "debug")
|
||||
|
||||
// Existing periodic tasks (migrated from ad-hoc goroutines)
|
||||
sched.Every("status-refresh", 30*time.Second, func(ctx context.Context) error {
|
||||
// F7: 30s left the dashboard list lagging Docker health by up to ~30s after a deploy/state change.
|
||||
// 10s (matching the health-probes cadence) tightens it; RefreshStatus is a cheap `docker ps`-based
|
||||
// refresh of the in-memory map, so 10s does not meaningfully load Docker. (The deploy page itself
|
||||
// already polls per-stack every 3s; this is for the dashboard/stacks list.)
|
||||
sched.Every("status-refresh", 10*time.Second, func(ctx context.Context) error {
|
||||
return stackMgr.RefreshStatus()
|
||||
})
|
||||
sched.Every("stack-scan", 2*time.Minute, func(ctx context.Context) error {
|
||||
|
||||
@@ -373,15 +373,15 @@ func (c *Client) FormatDisk(ctx context.Context, device, fstype string, confirme
|
||||
var out FormatResult
|
||||
// Status-aware POST: the agent returns the FULL FormatResponse (incl. pending_op / durable_id)
|
||||
// even on the 403 refusal, so we must read the body on non-2xx rather than discarding it.
|
||||
data, status, err := c.postWithStatus(ctx, "/disks/format", map[string]any{
|
||||
env, status, err := c.postWithStatus(ctx, "/disks/format", map[string]any{
|
||||
"device": device, "fstype": fstype, "confirmed": confirmed, "durable_id": durableID,
|
||||
})
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
// data is the envelope's {data:…} payload (present on both success and the 403 refusal).
|
||||
if len(data) > 0 {
|
||||
_ = json.Unmarshal(data, &out) // best-effort; fields default on a missing/partial body
|
||||
// env.Data is the envelope's {data:…} payload (present on both success and the 403 refusal).
|
||||
if len(env.Data) > 0 {
|
||||
_ = json.Unmarshal(env.Data, &out) // best-effort; fields default on a missing/partial body
|
||||
}
|
||||
if out.Formatted {
|
||||
return out, nil
|
||||
@@ -394,34 +394,45 @@ func (c *Client) FormatDisk(ctx context.Context, device, fstype string, confirme
|
||||
out.DataBearing = true
|
||||
return out, ErrFormatRefused // system/backup: surface the opsign command
|
||||
}
|
||||
// F20-BUG1: a non-2xx response (or ok:false) that is NOT one of the recognized refusals above is a
|
||||
// real failure (e.g. the agent's 502 on a mkfs error: "device is mounted"). Returning the zero-value
|
||||
// result with a nil error here made a failed destructive format read as a silent SUCCESS in the web
|
||||
// layer. Surface it as an error so the caller (and the dashboard) report the failure.
|
||||
if status < 200 || status >= 300 || !env.OK {
|
||||
msg := strings.TrimSpace(env.Error)
|
||||
if msg == "" {
|
||||
msg = "format failed"
|
||||
}
|
||||
return out, fmt.Errorf("agentapi: format: HTTP %d: %s", status, msg)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// postWithStatus issues an authenticated JSON POST and returns the envelope's data payload + the HTTP
|
||||
// status, even on a non-2xx (so callers like FormatDisk can read a 403 refusal body). A transport or
|
||||
// envelope-parse failure is still an error; an `ok:false` business refusal is NOT (the data carries it).
|
||||
func (c *Client) postWithStatus(ctx context.Context, path string, body any) (json.RawMessage, int, error) {
|
||||
func (c *Client) postWithStatus(ctx context.Context, path string, body any) (apiResponse, int, error) {
|
||||
var env apiResponse
|
||||
buf, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
return env, 0, err
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+path, bytes.NewReader(buf))
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
return env, 0, err
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+c.token)
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
resp, err := c.hc.Do(req)
|
||||
if err != nil {
|
||||
return nil, 0, fmt.Errorf("agentapi: POST %s: %w", path, err)
|
||||
return env, 0, fmt.Errorf("agentapi: POST %s: %w", path, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||
var env apiResponse
|
||||
if err := json.Unmarshal(raw, &env); err != nil {
|
||||
return nil, resp.StatusCode, fmt.Errorf("agentapi: POST %s: HTTP %d, bad envelope: %w", path, resp.StatusCode, err)
|
||||
return env, resp.StatusCode, fmt.Errorf("agentapi: POST %s: HTTP %d, bad envelope: %w", path, resp.StatusCode, err)
|
||||
}
|
||||
return env.Data, resp.StatusCode, nil
|
||||
return env, resp.StatusCode, nil
|
||||
}
|
||||
|
||||
// ---- slice 9: host metrics (the customer host-health view) -------------------------------
|
||||
|
||||
@@ -34,6 +34,9 @@ func diskStub(t *testing.T) (*httptest.Server, string) {
|
||||
case strings.Contains(body.Device, "data") && !body.Confirmed: // user-data, not yet confirmed
|
||||
w.WriteHeader(http.StatusForbidden)
|
||||
_, _ = w.Write([]byte(`{"ok":false,"data":{"device":"` + body.Device + `","data_bearing":true,"role":"user-data","needs_confirmation":true,"durable_id":"byid:wwn-1"}}`))
|
||||
case strings.Contains(body.Device, "mounted"): // mkfs failed (e.g. device mounted) → agent 502, data:null
|
||||
w.WriteHeader(http.StatusBadGateway)
|
||||
_, _ = w.Write([]byte(`{"ok":false,"error":"format failed: /dev/sdb1 is mounted; will not make a filesystem here!","data":null}`))
|
||||
default: // blank, or user-data confirmed
|
||||
_, _ = w.Write([]byte(`{"ok":true,"data":{"device":"` + body.Device + `","formatted":true,"role":"user-data"}}`))
|
||||
}
|
||||
@@ -117,6 +120,28 @@ func TestFormat_UserDataConfirmed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// F20-BUG1: a real mkfs failure (agent 502, ok:false, data:null) must surface as a non-nil error —
|
||||
// NOT a zero-value FormatResult with nil err (which read as a silent SUCCESS in the web layer).
|
||||
func TestFormat_MountedFailureSurfacesError(t *testing.T) {
|
||||
s, ep := diskStub(t)
|
||||
defer s.Close()
|
||||
c := clientFor(t, s, ep)
|
||||
res, err := c.FormatDisk(context.Background(), "/dev/sdb1-mounted", "ext4", true, "byid:wwn-1")
|
||||
if err == nil {
|
||||
t.Fatalf("expected a non-nil error for a failed format, got (res=%+v, err=nil) — silent success regression", res)
|
||||
}
|
||||
if res.Formatted {
|
||||
t.Fatalf("Formatted must be false on a failed format: %+v", res)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "502") || !strings.Contains(err.Error(), "mounted") {
|
||||
t.Fatalf("error should carry the HTTP status + agent message, got: %v", err)
|
||||
}
|
||||
// Must not be misclassified as one of the gated refusals.
|
||||
if errors.Is(err, ErrNeedsConfirmation) || errors.Is(err, ErrFormatRefused) {
|
||||
t.Fatalf("a 502 mkfs failure must not be reported as a refusal: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEject_Dependents(t *testing.T) {
|
||||
s, ep := diskStub(t)
|
||||
defer s.Close()
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestWriteConfig0600 asserts F8: controller.yaml is persisted 0600 (it holds infra secrets), even when
|
||||
// the target file already existed with looser (0644) permissions. POSIX modes only — skipped on Windows.
|
||||
func TestWriteConfig0600(t *testing.T) {
|
||||
if runtime.GOOS == "windows" {
|
||||
t.Skip("POSIX file modes not represented on Windows")
|
||||
}
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "controller.yaml")
|
||||
|
||||
// Pre-create with world-readable 0644 to prove the helper tightens an existing file.
|
||||
if err := os.WriteFile(path, []byte("old: true\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if err := writeConfig0600(path, []byte("hub:\n api_key: redacted\n")); err != nil {
|
||||
t.Fatalf("writeConfig0600: %v", err)
|
||||
}
|
||||
|
||||
fi, err := os.Stat(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if mode := fi.Mode().Perm(); mode != 0o600 {
|
||||
t.Fatalf("config mode = %o, want 0600", mode)
|
||||
}
|
||||
// No leftover temp file.
|
||||
if _, err := os.Stat(path + ".tmp"); !os.IsNotExist(err) {
|
||||
t.Fatalf("temp file not cleaned up")
|
||||
}
|
||||
}
|
||||
@@ -112,6 +112,12 @@ func (r *Router) ServeHTTP(w http.ResponseWriter, req *http.Request) {
|
||||
case path == "/stacks/rescan" && req.Method == http.MethodPost:
|
||||
r.rescanStacks(w, req)
|
||||
|
||||
// F4: /api/stacks/rescan with a non-POST method must be a clear 405, not fall through to the
|
||||
// GET /stacks/{name} lookup below (which returned the misleading "stack not found: rescan").
|
||||
case path == "/stacks/rescan":
|
||||
w.Header().Set("Allow", http.MethodPost)
|
||||
writeJSON(w, http.StatusMethodNotAllowed, apiResponse{OK: false, Error: "method not allowed: use POST /api/stacks/rescan"})
|
||||
|
||||
// GET /api/stacks/{name}
|
||||
case strings.HasPrefix(path, "/stacks/") && req.Method == http.MethodGet && !hasSubpath(path, "/stacks/"):
|
||||
r.getStack(w, req, trimSegment(path, "/stacks/"))
|
||||
@@ -378,11 +384,15 @@ func (r *Router) deployStack(w http.ResponseWriter, req *http.Request, name stri
|
||||
return
|
||||
}
|
||||
|
||||
resp := apiResponse{OK: true, Message: "Stack " + name + " deployed"}
|
||||
// F6: the deploy runs asynchronously (compose pull/up + health happen after this returns; the UI
|
||||
// polls GET /api/stacks/{name}). The old "Stack X deployed" message asserted completion before it
|
||||
// was true — misleading for API/script consumers. Report that the deploy STARTED, not that it
|
||||
// finished. 202 Accepted reflects "accepted, processing"; ok:true is preserved for the UI.
|
||||
resp := apiResponse{OK: true, Message: "Telepítés elindítva – az állapot a kártyán követhető"}
|
||||
if warning != "" {
|
||||
resp.Data = map[string]string{"warning": warning}
|
||||
}
|
||||
writeJSON(w, http.StatusOK, resp)
|
||||
writeJSON(w, http.StatusAccepted, resp)
|
||||
|
||||
// Push app deployed event to Hub
|
||||
if r.notifier != nil {
|
||||
@@ -709,6 +719,20 @@ func (r *Router) triggerSync(w http.ResponseWriter, _ *http.Request) {
|
||||
|
||||
func (r *Router) systemInfo(w http.ResponseWriter, _ *http.Request) {
|
||||
info := system.GetInfo(r.cfg.Paths.HDDPath, r.cpuCollector)
|
||||
// F1: GetInfo now reports the guest RAM cap (from the Docker daemon) as TotalMemMB, but the guest-wide
|
||||
// "used" is not observable from the container. Report the controller's accurate committed-app memory
|
||||
// (sum of running apps' mem requests) as used — a meaningful "allocated of cap" figure for the UI.
|
||||
if r.stackMgr != nil && info.TotalMemMB > 0 {
|
||||
if reqMB, _ := r.stackMgr.CommittedMemory(); reqMB >= 0 {
|
||||
used := uint64(reqMB)
|
||||
if used > info.TotalMemMB {
|
||||
used = info.TotalMemMB
|
||||
}
|
||||
info.UsedMemMB = used
|
||||
info.AvailMemMB = info.TotalMemMB - used
|
||||
info.MemPercent = float64(used) / float64(info.TotalMemMB) * 100
|
||||
}
|
||||
}
|
||||
syncStatus := r.syncer.Status()
|
||||
data := map[string]interface{}{
|
||||
"system": info,
|
||||
@@ -1001,25 +1025,14 @@ func (r *Router) configApply(w http.ResponseWriter, req *http.Request) {
|
||||
return
|
||||
}
|
||||
|
||||
// Write config: try atomic rename first, fall back to direct write
|
||||
// (os.Rename fails on Docker bind mounts with "device or resource busy")
|
||||
tmpPath := r.configPath + ".tmp"
|
||||
if err := os.WriteFile(tmpPath, body, 0644); err != nil {
|
||||
r.logger.Printf("[ERROR] [api] Config apply: failed to write temp file: %v", err)
|
||||
writeJSON(w, http.StatusInternalServerError, apiResponse{OK: false, Error: "failed to write config"})
|
||||
return
|
||||
}
|
||||
|
||||
if err := os.Rename(tmpPath, r.configPath); err != nil {
|
||||
os.Remove(tmpPath)
|
||||
// Rename failed (likely Docker bind mount) — write directly
|
||||
if err := os.WriteFile(r.configPath, body, 0644); err != nil {
|
||||
// Write config 0600: it holds infra credentials (cf_api_token, cf_tunnel_token, hub api_key) in
|
||||
// plaintext (F8), so it must not be world-readable. writeConfig0600 enforces the mode even when the
|
||||
// target file already existed with looser perms (os.WriteFile does not chmod an existing file).
|
||||
if err := writeConfig0600(r.configPath, body); err != nil {
|
||||
r.logger.Printf("[ERROR] [api] Config apply: failed to write config: %v", err)
|
||||
writeJSON(w, http.StatusInternalServerError, apiResponse{OK: false, Error: "failed to apply config"})
|
||||
return
|
||||
}
|
||||
r.logger.Printf("[INFO] [api] Config apply: rename failed, wrote directly (bind mount)")
|
||||
}
|
||||
|
||||
r.logger.Printf("[INFO] [api] Config applied from Hub (%d bytes), restart needed to take effect", len(body))
|
||||
writeJSON(w, http.StatusOK, apiResponse{OK: true, Message: "Config applied. Restart controller to apply changes."})
|
||||
@@ -1030,6 +1043,27 @@ func (r *Router) configApply(w http.ResponseWriter, req *http.Request) {
|
||||
}
|
||||
}
|
||||
|
||||
// writeConfig0600 writes config bytes to path with mode 0600, atomically when possible (tmp+rename),
|
||||
// falling back to a direct write for Docker bind mounts (where os.Rename returns EBUSY). It ALWAYS
|
||||
// enforces 0600 on the final file — even if it already existed with looser perms — because controller.yaml
|
||||
// holds infra credentials in plaintext (F8); os.WriteFile only applies the mode when creating a new file.
|
||||
func writeConfig0600(path string, body []byte) error {
|
||||
tmpPath := path + ".tmp"
|
||||
if err := os.WriteFile(tmpPath, body, 0600); err != nil {
|
||||
return fmt.Errorf("writing temp config: %w", err)
|
||||
}
|
||||
if err := os.Rename(tmpPath, path); err != nil {
|
||||
os.Remove(tmpPath)
|
||||
if err := os.WriteFile(path, body, 0600); err != nil { // bind-mount fallback
|
||||
return fmt.Errorf("writing config: %w", err)
|
||||
}
|
||||
}
|
||||
if err := os.Chmod(path, 0600); err != nil {
|
||||
return fmt.Errorf("chmod config 0600: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *Router) configHash(w http.ResponseWriter, _ *http.Request) {
|
||||
hash, err := config.FileHash(r.configPath)
|
||||
if err != nil {
|
||||
|
||||
@@ -2,6 +2,7 @@ package appbackup
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
@@ -513,6 +514,113 @@ func populateDBEnv(ctx context.Context, db *DiscoveredDB) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// ImportDump replays a (possibly gzipped) SQL dump into a RUNNING database container — the read-side
|
||||
// counterpart to DumpOne (F17). It reuses the per-engine clients (psql / mariadb) and the DiscoveredDB's
|
||||
// OWN credentials (discovered from the live container env), so the caller needs no external env map. The
|
||||
// container must already be running (the restore flow brings the stack up first); ImportDump briefly
|
||||
// waits for the engine to accept connections, then pipes the dump in. The backup dumps are produced with
|
||||
// DROP/CREATE (pg_dump --clean --if-exists; mariadb-dump's default --add-drop-table), so a replay fully
|
||||
// reconstructs the captured logical state.
|
||||
func ImportDump(ctx context.Context, db DiscoveredDB, dumpPath string, logger *log.Logger, debug bool) error {
|
||||
if err := waitDBReady(ctx, db, 30*time.Second); err != nil {
|
||||
return fmt.Errorf("waiting for %s (%s) readiness: %w", db.ContainerName, db.DBType, err)
|
||||
}
|
||||
|
||||
f, err := os.Open(dumpPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("opening dump %s: %w", dumpPath, err)
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
var reader io.Reader = f
|
||||
if strings.HasSuffix(dumpPath, ".gz") {
|
||||
gr, err := gzip.NewReader(f)
|
||||
if err != nil {
|
||||
return fmt.Errorf("opening gzip %s: %w", dumpPath, err)
|
||||
}
|
||||
defer gr.Close()
|
||||
reader = gr
|
||||
}
|
||||
|
||||
impCtx, cancel := context.WithTimeout(ctx, 30*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
var cmd *exec.Cmd
|
||||
switch db.DBType {
|
||||
case DBTypePostgres:
|
||||
user := db.DBUser
|
||||
if user == "" {
|
||||
user = "postgres"
|
||||
}
|
||||
dbName := db.DBName
|
||||
if dbName == "" {
|
||||
dbName = user
|
||||
}
|
||||
// ON_ERROR_STOP=1: a real import error must FAIL (and surface), not silently half-apply.
|
||||
cmd = exec.CommandContext(impCtx, "docker", "exec", "-i", db.ContainerID,
|
||||
"psql", "-v", "ON_ERROR_STOP=1", "-U", user, "-d", dbName)
|
||||
case DBTypeMariaDB:
|
||||
password := getMariaDBPassword(impCtx, db.ContainerID)
|
||||
if password == "" {
|
||||
return fmt.Errorf("could not determine MariaDB root password for %s", db.ContainerName)
|
||||
}
|
||||
cmd = exec.CommandContext(impCtx, "docker", "exec", "-i", db.ContainerID,
|
||||
"mariadb", "-u", "root", "-p"+password, db.DBName)
|
||||
default:
|
||||
return fmt.Errorf("unsupported DB type: %s", db.DBType)
|
||||
}
|
||||
|
||||
cmd.Stdin = reader
|
||||
var stderr strings.Builder
|
||||
cmd.Stderr = &stderr
|
||||
if debug && logger != nil {
|
||||
logger.Printf("[DEBUG] [backup] ImportDump: importing %s into %s (%s)", dumpPath, db.ContainerName, db.DBType)
|
||||
}
|
||||
if err := cmd.Run(); err != nil {
|
||||
msg := strings.TrimSpace(stderr.String())
|
||||
if len(msg) > 300 {
|
||||
msg = msg[:300]
|
||||
}
|
||||
return fmt.Errorf("%s import into %s failed: %s — %w", db.DBType, db.ContainerName, msg, err)
|
||||
}
|
||||
if logger != nil {
|
||||
logger.Printf("[INFO] [backup] Imported DB dump %s into %s (%s)", filepath.Base(dumpPath), db.ContainerName, db.DBType)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// waitDBReady polls until the database accepts connections (pg_isready / mariadb-admin ping).
|
||||
func waitDBReady(ctx context.Context, db DiscoveredDB, timeout time.Duration) error {
|
||||
deadline := time.Now().Add(timeout)
|
||||
for {
|
||||
c, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
var cmd *exec.Cmd
|
||||
switch db.DBType {
|
||||
case DBTypePostgres:
|
||||
user := db.DBUser
|
||||
if user == "" {
|
||||
user = "postgres"
|
||||
}
|
||||
cmd = exec.CommandContext(c, "docker", "exec", db.ContainerID, "pg_isready", "-U", user)
|
||||
case DBTypeMariaDB:
|
||||
pw := getMariaDBPassword(c, db.ContainerID)
|
||||
cmd = exec.CommandContext(c, "docker", "exec", db.ContainerID, "mariadb-admin", "ping", "-u", "root", "-p"+pw)
|
||||
default:
|
||||
cancel()
|
||||
return fmt.Errorf("unsupported DB type: %s", db.DBType)
|
||||
}
|
||||
err := cmd.Run()
|
||||
cancel()
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
return fmt.Errorf("timeout after %s", timeout)
|
||||
}
|
||||
time.Sleep(2 * time.Second)
|
||||
}
|
||||
}
|
||||
|
||||
func getMariaDBPassword(ctx context.Context, containerID string) string {
|
||||
cmd := exec.CommandContext(ctx, "docker", "inspect", containerID,
|
||||
"--format", "{{range .Config.Env}}{{println .}}{{end}}")
|
||||
|
||||
@@ -59,6 +59,11 @@ func DumpOne(ctx context.Context, db DiscoveredDB, dumpDir string, logger *log.L
|
||||
return appbackup.DumpOne(ctx, db, dumpDir, logger, debug)
|
||||
}
|
||||
|
||||
// ImportDump replays a captured .sql dump back into a running DB container (F17 restore path).
|
||||
func ImportDump(ctx context.Context, db DiscoveredDB, dumpPath string, logger *log.Logger, debug bool) error {
|
||||
return appbackup.ImportDump(ctx, db, dumpPath, logger, debug)
|
||||
}
|
||||
|
||||
func ValidateDump(filePath string, dbType DBType) DumpValidation {
|
||||
return appbackup.ValidateDump(filePath, dbType)
|
||||
}
|
||||
|
||||
@@ -31,6 +31,11 @@ type Manager struct {
|
||||
// tier2Notify, if set, is called after each Tier 2 copy (success: err==nil) for notifications.
|
||||
tier2Notify func(stackName, destLabel string, dur time.Duration, err error)
|
||||
|
||||
// F17 restore seams — overridable in tests so the .sql re-import orchestration can be unit-tested
|
||||
// without Docker. Default to the real DiscoverDatabases / ImportDump (lazy-init in reimportDBDumps).
|
||||
discoverDBs func(ctx context.Context) ([]DiscoveredDB, error)
|
||||
importDBDump func(ctx context.Context, db DiscoveredDB, dumpPath string) error
|
||||
|
||||
mu sync.Mutex
|
||||
lastDBDump *DBDumpStatus
|
||||
running bool
|
||||
|
||||
@@ -56,12 +56,17 @@ func (m *Manager) RestoreApp(stackName, snapshotID string) error {
|
||||
m.logger.Printf("[WARN] RESTORE could not stop %s: %v (proceeding anyway)", stackName, err)
|
||||
}
|
||||
|
||||
// F17: surface a data-restore failure instead of swallowing it. We still bring the app back up so it
|
||||
// isn't left dead, but the error is returned at the end so a failed restore can't read as success.
|
||||
var dataErr error
|
||||
|
||||
// Populate Docker volumes from restored tars
|
||||
if m.isDebug() {
|
||||
m.logger.Printf("[DEBUG] RestoreApp: step 2/3 — restoring Docker volumes for %s", stackName)
|
||||
}
|
||||
if err := m.restoreDockerVolumes(stackName, drivePath); err != nil {
|
||||
m.logger.Printf("[WARN] RESTORE volume restore failed for %s: %v (continuing)", stackName, err)
|
||||
m.logger.Printf("[ERROR] RESTORE volume restore failed for %s: %v", stackName, err)
|
||||
dataErr = err
|
||||
}
|
||||
|
||||
// Restart the app
|
||||
@@ -72,11 +77,23 @@ func (m *Manager) RestoreApp(stackName, snapshotID string) error {
|
||||
m.logger.Printf("[WARN] RESTORE could not restart %s after restore: %v", stackName, err)
|
||||
}
|
||||
|
||||
// F17: replay the captured .sql dump into the now-running DB (the legacy path never did this, so
|
||||
// DB-resident data did not come back). Runs after volume restore so the dump WINS over any tar copy.
|
||||
if _, err := m.reimportDBDumpsCtx(stackName, m.namespaceRoot(drivePath)); err != nil {
|
||||
m.logger.Printf("[ERROR] RESTORE DB re-import failed for %s: %v", stackName, err)
|
||||
if dataErr == nil {
|
||||
dataErr = err
|
||||
}
|
||||
}
|
||||
|
||||
// Verify app started successfully
|
||||
if err := m.waitForHealthy(stackName, 90*time.Second); err != nil {
|
||||
m.logger.Printf("[WARN] [backup] Restore completed but app health check failed: %v", err)
|
||||
}
|
||||
|
||||
if dataErr != nil {
|
||||
return fmt.Errorf("restore of %s completed with data errors: %w", stackName, dataErr)
|
||||
}
|
||||
m.logger.Printf("[INFO] RESTORE completed: stack=%s", stackName)
|
||||
return nil
|
||||
}
|
||||
@@ -93,6 +110,7 @@ func (m *Manager) restoreDockerVolumes(stackName, drivePath string) error {
|
||||
}
|
||||
|
||||
var restored int
|
||||
var failed []string
|
||||
for _, entry := range entries {
|
||||
if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".tar") {
|
||||
continue
|
||||
@@ -106,7 +124,8 @@ func (m *Manager) restoreDockerVolumes(stackName, drivePath string) error {
|
||||
|
||||
// Create fresh volume
|
||||
if out, err := exec.Command("docker", "volume", "create", volName).CombinedOutput(); err != nil {
|
||||
m.logger.Printf("[WARN] [backup] Failed to create volume %s: %s — %v", volName, strings.TrimSpace(string(out)), err)
|
||||
m.logger.Printf("[ERROR] [backup] Failed to create volume %s: %s — %v", volName, strings.TrimSpace(string(out)), err)
|
||||
failed = append(failed, volName)
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -120,7 +139,8 @@ func (m *Manager) restoreDockerVolumes(stackName, drivePath string) error {
|
||||
cancel()
|
||||
|
||||
if err != nil {
|
||||
m.logger.Printf("[WARN] [backup] Failed to populate volume %s: %s — %v", volName, strings.TrimSpace(string(out)), err)
|
||||
m.logger.Printf("[ERROR] [backup] Failed to populate volume %s: %s — %v", volName, strings.TrimSpace(string(out)), err)
|
||||
failed = append(failed, volName)
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -133,6 +153,11 @@ func (m *Manager) restoreDockerVolumes(stackName, drivePath string) error {
|
||||
if restored > 0 {
|
||||
m.logger.Printf("[INFO] [backup] Restored %d Docker volume(s) for %s", restored, stackName)
|
||||
}
|
||||
// F17: a per-volume failure used to be a swallowed WARN; surface it so the restore is reported as
|
||||
// failed rather than silently partial.
|
||||
if len(failed) > 0 {
|
||||
return fmt.Errorf("failed to restore %d volume(s): %v", len(failed), failed)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
)
|
||||
|
||||
// reimportDBDumps replays the captured per-app .sql dumps back into the app's now-running database
|
||||
// container(s) — the F17 fix. The per-app backup captures a logical SQL dump (DumpOne →
|
||||
// <stack>-<dbtype>.sql) but the legacy restore only repopulated Docker volume tars and NEVER replayed
|
||||
// the dump, so DB-resident data (e.g. rows in a DB whose data dir is a bind mount, not a named volume)
|
||||
// did not come back. This runs AFTER volume restore + stack bring-up, so the dump WINS over any
|
||||
// volume-tar copy of the DB (the operator-chosen precedence: the consistent logical dump is authoritative).
|
||||
//
|
||||
// It uses the live container's OWN discovered credentials (DiscoveredDB), so no env threading is needed.
|
||||
// A dump whose DB container is not found is logged and skipped; an actual import FAILURE is returned
|
||||
// (surfaced, not swallowed) so a failed data restore cannot read as success.
|
||||
func (m *Manager) reimportDBDumps(ctx context.Context, stackName, nsRoot string) (int, error) {
|
||||
dumpDir := AppDBDumpPath(nsRoot, stackName)
|
||||
entries, err := os.ReadDir(dumpDir)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return 0, nil // no DB dumps for this app
|
||||
}
|
||||
return 0, fmt.Errorf("reading db-dump dir: %w", err)
|
||||
}
|
||||
hasDump := false
|
||||
for _, e := range entries {
|
||||
if !e.IsDir() && filepath.Ext(e.Name()) == ".sql" {
|
||||
hasDump = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !hasDump {
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
discover := m.discoverDBs
|
||||
if discover == nil {
|
||||
discover = func(ctx context.Context) ([]DiscoveredDB, error) {
|
||||
return DiscoverDatabases(ctx, m.logger, m.isDebug())
|
||||
}
|
||||
}
|
||||
imp := m.importDBDump
|
||||
if imp == nil {
|
||||
imp = func(ctx context.Context, db DiscoveredDB, dumpPath string) error {
|
||||
return ImportDump(ctx, db, dumpPath, m.logger, m.isDebug())
|
||||
}
|
||||
}
|
||||
|
||||
dbs, err := discover(ctx)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("discovering DB containers for %s: %w", stackName, err)
|
||||
}
|
||||
|
||||
var imported int
|
||||
for _, db := range dbs {
|
||||
if db.StackName != stackName {
|
||||
continue
|
||||
}
|
||||
// The dump for this DB is named "<stack>-<dbtype>.sql" (see DumpOne).
|
||||
dumpPath := filepath.Join(dumpDir, fmt.Sprintf("%s-%s.sql", stackName, db.DBType))
|
||||
if _, statErr := os.Stat(dumpPath); statErr != nil {
|
||||
continue // no dump for this particular DB engine
|
||||
}
|
||||
m.logger.Printf("[INFO] [backup] Restore %s: replaying DB dump into %s (%s)", stackName, db.ContainerName, db.DBType)
|
||||
if err := imp(ctx, db, dumpPath); err != nil {
|
||||
return imported, fmt.Errorf("importing %s dump for %s: %w", db.DBType, stackName, err)
|
||||
}
|
||||
imported++
|
||||
}
|
||||
|
||||
if imported == 0 {
|
||||
m.logger.Printf("[WARN] [backup] Restore %s: a .sql dump exists but no matching running DB container was found — DB content NOT restored", stackName)
|
||||
} else {
|
||||
m.logger.Printf("[INFO] [backup] Restore %s: replayed %d DB dump(s)", stackName, imported)
|
||||
}
|
||||
return imported, nil
|
||||
}
|
||||
|
||||
// reimportDBDumpsCtx is a small helper that runs reimportDBDumps with a bounded context so a stuck DB
|
||||
// import cannot hang the restore indefinitely.
|
||||
func (m *Manager) reimportDBDumpsCtx(stackName, nsRoot string) (int, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 35*time.Minute)
|
||||
defer cancel()
|
||||
return m.reimportDBDumps(ctx, stackName, nsRoot)
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func newReimportTestManager() *Manager {
|
||||
return &Manager{logger: log.New(io.Discard, "", 0)}
|
||||
}
|
||||
|
||||
func writeDump(t *testing.T, nsRoot, stack string, dbType DBType) string {
|
||||
t.Helper()
|
||||
dir := AppDBDumpPath(nsRoot, stack)
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
p := filepath.Join(dir, fmt.Sprintf("%s-%s.sql", stack, dbType))
|
||||
if err := os.WriteFile(p, []byte("-- dump\nDROP TABLE IF EXISTS t;\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// TestReimportDBDumps_ImportsWhenDumpAndDBPresent asserts F17: when a captured .sql dump exists and a
|
||||
// matching running DB is discovered, reimportDBDumps replays it. The PRE-FIX restore never called any
|
||||
// import — this orchestration is the fix.
|
||||
func TestReimportDBDumps_ImportsWhenDumpAndDBPresent(t *testing.T) {
|
||||
nsRoot := t.TempDir()
|
||||
wantPath := writeDump(t, nsRoot, "app", DBTypeMariaDB)
|
||||
|
||||
m := newReimportTestManager()
|
||||
m.discoverDBs = func(ctx context.Context) ([]DiscoveredDB, error) {
|
||||
return []DiscoveredDB{
|
||||
{StackName: "other", DBType: DBTypePostgres, ContainerName: "other-db"},
|
||||
{StackName: "app", DBType: DBTypeMariaDB, ContainerName: "app-db", ContainerID: "cid"},
|
||||
}, nil
|
||||
}
|
||||
var gotPath string
|
||||
var gotDB DiscoveredDB
|
||||
m.importDBDump = func(ctx context.Context, db DiscoveredDB, dumpPath string) error {
|
||||
gotPath, gotDB = dumpPath, db
|
||||
return nil
|
||||
}
|
||||
|
||||
n, err := m.reimportDBDumps(context.Background(), "app", nsRoot)
|
||||
if err != nil {
|
||||
t.Fatalf("reimportDBDumps: %v", err)
|
||||
}
|
||||
if n != 1 {
|
||||
t.Fatalf("imported = %d, want 1", n)
|
||||
}
|
||||
if gotPath != wantPath {
|
||||
t.Fatalf("imported path = %q, want %q", gotPath, wantPath)
|
||||
}
|
||||
if gotDB.ContainerName != "app-db" {
|
||||
t.Fatalf("imported into %q, want app-db (must match the stack's own DB)", gotDB.ContainerName)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReimportDBDumps_FailureSurfaces asserts an import failure is RETURNED, not swallowed (a failed
|
||||
// data restore must not read as success).
|
||||
func TestReimportDBDumps_FailureSurfaces(t *testing.T) {
|
||||
nsRoot := t.TempDir()
|
||||
writeDump(t, nsRoot, "app", DBTypeMariaDB)
|
||||
|
||||
m := newReimportTestManager()
|
||||
m.discoverDBs = func(ctx context.Context) ([]DiscoveredDB, error) {
|
||||
return []DiscoveredDB{{StackName: "app", DBType: DBTypeMariaDB, ContainerName: "app-db"}}, nil
|
||||
}
|
||||
m.importDBDump = func(ctx context.Context, db DiscoveredDB, dumpPath string) error {
|
||||
return fmt.Errorf("boom")
|
||||
}
|
||||
|
||||
if _, err := m.reimportDBDumps(context.Background(), "app", nsRoot); err == nil || !strings.Contains(err.Error(), "boom") {
|
||||
t.Fatalf("expected the import failure to surface, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReimportDBDumps_NoDumpNoImport asserts apps with no .sql dump never trigger discovery/import.
|
||||
func TestReimportDBDumps_NoDumpNoImport(t *testing.T) {
|
||||
nsRoot := t.TempDir()
|
||||
m := newReimportTestManager()
|
||||
discoverCalled := false
|
||||
m.discoverDBs = func(ctx context.Context) ([]DiscoveredDB, error) {
|
||||
discoverCalled = true
|
||||
return nil, nil
|
||||
}
|
||||
m.importDBDump = func(ctx context.Context, db DiscoveredDB, dumpPath string) error {
|
||||
t.Fatal("importDBDump must not be called when there is no dump")
|
||||
return nil
|
||||
}
|
||||
n, err := m.reimportDBDumps(context.Background(), "app", nsRoot)
|
||||
if err != nil || n != 0 {
|
||||
t.Fatalf("reimportDBDumps with no dump = (%d, %v), want (0, nil)", n, err)
|
||||
}
|
||||
if discoverCalled {
|
||||
t.Fatalf("discovery should be skipped when there is no .sql dump")
|
||||
}
|
||||
}
|
||||
|
||||
// TestReimportDBDumps_DumpButNoMatchingDB asserts that a dump with no matching running DB container is a
|
||||
// non-fatal skip (logged), returning 0 imported and no error — the app is up, just no DB matched.
|
||||
func TestReimportDBDumps_DumpButNoMatchingDB(t *testing.T) {
|
||||
nsRoot := t.TempDir()
|
||||
writeDump(t, nsRoot, "app", DBTypeMariaDB)
|
||||
m := newReimportTestManager()
|
||||
m.discoverDBs = func(ctx context.Context) ([]DiscoveredDB, error) {
|
||||
return []DiscoveredDB{{StackName: "different", DBType: DBTypeMariaDB}}, nil
|
||||
}
|
||||
m.importDBDump = func(ctx context.Context, db DiscoveredDB, dumpPath string) error {
|
||||
t.Fatal("must not import when no DB matches the stack")
|
||||
return nil
|
||||
}
|
||||
n, err := m.reimportDBDumps(context.Background(), "app", nsRoot)
|
||||
if err != nil || n != 0 {
|
||||
t.Fatalf("= (%d, %v), want (0, nil)", n, err)
|
||||
}
|
||||
}
|
||||
@@ -122,19 +122,33 @@ func (m *Manager) RestoreFromRecoveryUnit(stackName string) error {
|
||||
stackName, len(manifest.ImagePins), len(manifest.SecretEnvVars)-len(missing), len(manifest.SecretEnvVars), len(manifest.DataKeyEnvVars))
|
||||
|
||||
// Stop, restore named-volume data, then recreate the definition + redeploy with the recovered env.
|
||||
// F17: surface a data-restore failure instead of swallowing it (we still bring the app back up).
|
||||
var dataErr error
|
||||
if err := m.stackProvider.StopStack(stackName); err != nil {
|
||||
m.logger.Printf("[WARN] [backup] could not stop %s before restore: %v (continuing)", stackName, err)
|
||||
}
|
||||
if err := m.restoreDockerVolumes(stackName, drivePath); err != nil {
|
||||
m.logger.Printf("[WARN] [backup] volume restore for %s: %v (continuing)", stackName, err)
|
||||
m.logger.Printf("[ERROR] [backup] volume restore for %s: %v", stackName, err)
|
||||
dataErr = err
|
||||
}
|
||||
if err := m.stackProvider.RecreateStackFromUnit(stackName, composeDir, fullEnv); err != nil {
|
||||
return fmt.Errorf("recreating %s from unit: %w", stackName, err)
|
||||
}
|
||||
// F17: the captured .sql dump is the authoritative logical DB state — replay it into the now-running
|
||||
// DB container AFTER the volume restore, so the dump WINS over any volume-tar copy of the database.
|
||||
if _, err := m.reimportDBDumpsCtx(stackName, nsRoot); err != nil {
|
||||
m.logger.Printf("[ERROR] [backup] DB re-import for %s: %v", stackName, err)
|
||||
if dataErr == nil {
|
||||
dataErr = err
|
||||
}
|
||||
}
|
||||
if err := m.waitForHealthy(stackName, 90*time.Second); err != nil {
|
||||
m.logger.Printf("[WARN] [backup] %s restored but health check failed: %v", stackName, err)
|
||||
}
|
||||
|
||||
if dataErr != nil {
|
||||
return fmt.Errorf("restore of %s from unit completed with data errors: %w", stackName, dataErr)
|
||||
}
|
||||
m.logger.Printf("[INFO] [backup] Restore-from-unit completed: %s", stackName)
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -160,16 +160,28 @@ func (m *Manager) DeployStack(req DeployRequest) (string, error) {
|
||||
var deployWarning string
|
||||
reservedMB := m.cfg.System.ReservedMemoryMB
|
||||
totalMB, usedMB, memErr := system.GetMemoryMB()
|
||||
// F1: the controller container cannot read the guest's RAM cap from /proc (no lxcfs) or its own
|
||||
// cgroup (the cap is on the LXC ancestor). Prefer the guest cap from the Docker daemon (runs in the
|
||||
// LXC). And use the controller's OWN committed-memory accounting for "used" — accurate and cheap —
|
||||
// rather than host /proc RSS, which is unobservable-per-guest and would otherwise make this guard
|
||||
// either never fire (host total) or always fire (host used > guest cap).
|
||||
if gt, ok := system.GuestMemTotalMB(); ok && gt > 0 {
|
||||
totalMB = gt
|
||||
memErr = nil
|
||||
}
|
||||
if committedReqMB, _ := m.CommittedMemory(); committedReqMB > 0 || memErr == nil {
|
||||
usedMB = committedReqMB
|
||||
}
|
||||
if memErr != nil {
|
||||
m.logger.Printf("[WARN] [stacks] Cannot read system memory: %v — skipping memory check", memErr)
|
||||
} else {
|
||||
usableMB := totalMB - reservedMB
|
||||
newReqMB := ParseMemoryMB(meta.Resources.MemRequest)
|
||||
|
||||
m.logger.Printf("[INFO] [stacks] Memory check: total=%dMB, reserved=%dMB, usable=%dMB, real_used=%dMB, new_req=%dMB, remaining=%dMB",
|
||||
m.logger.Printf("[INFO] [stacks] Memory check: total=%dMB, reserved=%dMB, usable=%dMB, committed_used=%dMB, new_req=%dMB, remaining=%dMB",
|
||||
totalMB, reservedMB, usableMB, usedMB, newReqMB, usableMB-usedMB-newReqMB)
|
||||
|
||||
// Hard block: real used + new request exceeds usable memory
|
||||
// Hard block: committed + new request exceeds usable memory
|
||||
if newReqMB > 0 && usedMB+newReqMB > usableMB {
|
||||
clearDeploying()
|
||||
return "", fmt.Errorf(
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
//go:build linux
|
||||
|
||||
package system
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestReadMemInfoUsesCgroupV2Limit asserts F1: when a cgroup v2 memory.max caps the container well
|
||||
// below the host /proc/meminfo total, readMemInfo reports the cgroup cap (the guest's real ceiling),
|
||||
// not the host RAM. On the pre-fix code this test fails because readMemInfo ignored cgroup entirely.
|
||||
func TestReadMemInfoUsesCgroupV2Limit(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
const twoGiB = uint64(2 * 1024 * 1024 * 1024)
|
||||
mustWriteFile(t, filepath.Join(dir, "memory.max"), []byte(itoa(twoGiB)))
|
||||
|
||||
old := cgroupRoot
|
||||
cgroupRoot = dir
|
||||
defer func() { cgroupRoot = old }()
|
||||
|
||||
var info SystemInfo
|
||||
readMemInfo(&info)
|
||||
|
||||
// Host /proc/meminfo total is whatever the test machine has; the cgroup cap must win when smaller.
|
||||
if info.TotalMemMB != 2048 {
|
||||
t.Fatalf("TotalMemMB = %d, want 2048 (cgroup cap), not host RAM", info.TotalMemMB)
|
||||
}
|
||||
// Used is a scaled estimate (the container cannot read guest-wide RSS); just assert it is sane.
|
||||
if info.UsedMemMB > info.TotalMemMB {
|
||||
t.Fatalf("UsedMemMB = %d exceeds TotalMemMB = %d", info.UsedMemMB, info.TotalMemMB)
|
||||
}
|
||||
if info.AvailMemMB != info.TotalMemMB-info.UsedMemMB {
|
||||
t.Fatalf("AvailMemMB inconsistent: %d != %d-%d", info.AvailMemMB, info.TotalMemMB, info.UsedMemMB)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReadMemInfoUsesDockerInfoWhenCgroupUnlimited asserts the NESTED-LXC case (the real demo): the
|
||||
// container's own cgroup is unlimited ("max"), so the guest cap must come from `docker info` MemTotal.
|
||||
// Pre-fix (and the cgroup-only attempt) reports the host RAM here.
|
||||
func TestReadMemInfoUsesDockerInfoWhenCgroupUnlimited(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
mustWriteFile(t, filepath.Join(dir, "memory.max"), []byte("max")) // container cgroup unlimited
|
||||
old := cgroupRoot
|
||||
cgroupRoot = dir
|
||||
defer func() { cgroupRoot = old }()
|
||||
oldFn := dockerMemTotalFn
|
||||
dockerMemTotalFn = func() (uint64, bool) { return 2048, true }
|
||||
defer func() { dockerMemTotalFn = oldFn }()
|
||||
|
||||
var info SystemInfo
|
||||
readMemInfo(&info)
|
||||
if info.TotalMemMB != 2048 {
|
||||
t.Fatalf("TotalMemMB = %d, want 2048 (docker-info guest cap)", info.TotalMemMB)
|
||||
}
|
||||
}
|
||||
|
||||
// TestGuestMemTotalMB_DockerInfoFallback asserts GuestMemTotalMB (used by the deploy guard) falls back
|
||||
// to docker-info when no cgroup limit is present.
|
||||
func TestGuestMemTotalMB_DockerInfoFallback(t *testing.T) {
|
||||
dir := t.TempDir() // no cgroup files → no limit
|
||||
old := cgroupRoot
|
||||
cgroupRoot = dir
|
||||
defer func() { cgroupRoot = old }()
|
||||
oldFn := dockerMemTotalFn
|
||||
dockerMemTotalFn = func() (uint64, bool) { return 2048, true }
|
||||
defer func() { dockerMemTotalFn = oldFn }()
|
||||
|
||||
if v, ok := GuestMemTotalMB(); !ok || v != 2048 {
|
||||
t.Fatalf("GuestMemTotalMB = (%d, %v), want (2048, true)", v, ok)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReadMemInfoCgroupMaxIsUnlimited asserts that a v2 "max" sentinel does NOT override /proc/meminfo
|
||||
// (an uncapped container keeps the host view rather than a bogus 0).
|
||||
func TestReadMemInfoCgroupMaxIsUnlimited(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
mustWriteFile(t, filepath.Join(dir, "memory.max"), []byte("max"))
|
||||
|
||||
old := cgroupRoot
|
||||
cgroupRoot = dir
|
||||
defer func() { cgroupRoot = old }()
|
||||
|
||||
var info SystemInfo
|
||||
readMemInfo(&info)
|
||||
|
||||
if info.TotalMemMB == 0 {
|
||||
t.Fatalf("TotalMemMB = 0 with an unlimited cgroup; expected the /proc/meminfo host total")
|
||||
}
|
||||
}
|
||||
|
||||
// TestReadCgroupMemLimitV1Unlimited asserts the v1 near-uint64-max sentinel is treated as unlimited.
|
||||
func TestReadCgroupMemLimitV1Unlimited(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
memDir := filepath.Join(dir, "memory")
|
||||
if err := os.MkdirAll(memDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Typical v1 "unlimited" value.
|
||||
mustWriteFile(t, filepath.Join(memDir, "memory.limit_in_bytes"), []byte("9223372036854771712"))
|
||||
|
||||
if _, ok := readCgroupMemLimitMB(dir); ok {
|
||||
t.Fatalf("readCgroupMemLimitMB treated the v1 unlimited sentinel as a real limit")
|
||||
}
|
||||
}
|
||||
|
||||
// TestReadCgroupMemLimitV1Real asserts a finite v1 limit is read.
|
||||
func TestReadCgroupMemLimitV1Real(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
memDir := filepath.Join(dir, "memory")
|
||||
if err := os.MkdirAll(memDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
const oneGiB = uint64(1024 * 1024 * 1024)
|
||||
mustWriteFile(t, filepath.Join(memDir, "memory.limit_in_bytes"), []byte(itoa(oneGiB)))
|
||||
|
||||
mb, ok := readCgroupMemLimitMB(dir)
|
||||
if !ok || mb != 1024 {
|
||||
t.Fatalf("readCgroupMemLimitMB = (%d, %v), want (1024, true)", mb, ok)
|
||||
}
|
||||
}
|
||||
|
||||
func mustWriteFile(t *testing.T, path string, data []byte) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(path, data, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func itoa(v uint64) string {
|
||||
if v == 0 {
|
||||
return "0"
|
||||
}
|
||||
var buf [20]byte
|
||||
i := len(buf)
|
||||
for v > 0 {
|
||||
i--
|
||||
buf[i] = byte('0' + v%10)
|
||||
v /= 10
|
||||
}
|
||||
return string(buf[i:])
|
||||
}
|
||||
@@ -4,10 +4,13 @@ package system
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
@@ -77,6 +80,9 @@ func GetMemoryMB() (totalMB, usedMB int, err error) {
|
||||
return int(info.TotalMemMB), int(info.UsedMemMB), nil
|
||||
}
|
||||
|
||||
// cgroupRoot is the cgroup mount point. Overridable in tests.
|
||||
var cgroupRoot = "/sys/fs/cgroup"
|
||||
|
||||
func readMemInfo(info *SystemInfo) {
|
||||
f, err := os.Open("/proc/meminfo")
|
||||
if err != nil {
|
||||
@@ -100,16 +106,135 @@ func readMemInfo(info *SystemInfo) {
|
||||
}
|
||||
}
|
||||
|
||||
if totalKB > 0 {
|
||||
if totalKB == 0 {
|
||||
debugf("[DEBUG] [system] readMemInfo: could not parse MemTotal from /proc/meminfo")
|
||||
return
|
||||
}
|
||||
|
||||
info.TotalMemMB = totalKB / 1024
|
||||
info.AvailMemMB = availKB / 1024
|
||||
info.UsedMemMB = info.TotalMemMB - info.AvailMemMB
|
||||
|
||||
// F1: the controller runs as a Docker container inside an LXC. /proc/meminfo reports the HOST's RAM
|
||||
// (no lxcfs in the container) and the container's OWN cgroup is unlimited (the 2GB cap lives on the
|
||||
// LXC, an ancestor hidden from the container), so the reported total massively overstates the guest's
|
||||
// real ceiling and defeats the deploy memory-headroom guard. Determine the true guest cap from, in
|
||||
// order: the container's cgroup limit (correct when Docker sets -m, e.g. non-nested deploys), else
|
||||
// `docker info` MemTotal (dockerd runs IN the LXC and reports the guest's lxcfs-backed RAM — the
|
||||
// accurate cap in the nested-LXC case). The instantaneous guest-wide RSS is NOT observable from the
|
||||
// container, so when we override the cap we scale the host's used-fraction onto it as an estimate for
|
||||
// display; the CAP itself (what the headroom math depends on) is accurate. The deploy guard uses the
|
||||
// controller's own committed-memory accounting for "used", so safety does not rely on this estimate.
|
||||
capMB := uint64(0)
|
||||
if v, ok := readCgroupMemLimitMB(cgroupRoot); ok && v > 0 && v < info.TotalMemMB {
|
||||
capMB = v
|
||||
}
|
||||
if capMB == 0 {
|
||||
if v, ok := guestMemTotalMB(); ok && v > 0 && v < info.TotalMemMB {
|
||||
capMB = v
|
||||
}
|
||||
}
|
||||
if capMB > 0 && capMB < info.TotalMemMB {
|
||||
frac := 0.0
|
||||
if info.TotalMemMB > 0 {
|
||||
frac = float64(info.UsedMemMB) / float64(info.TotalMemMB)
|
||||
}
|
||||
info.TotalMemMB = capMB
|
||||
// Scaled host-pressure estimate (the container can't read guest-wide RSS). The /api/system/info
|
||||
// handler overrides this with the controller's committed-app memory for an accurate figure; this
|
||||
// estimate covers the other GetInfo callers (monitoring) without alarming at ~100%.
|
||||
info.UsedMemMB = uint64(float64(capMB) * frac)
|
||||
info.AvailMemMB = info.TotalMemMB - info.UsedMemMB
|
||||
debugf("[DEBUG] [system] readMemInfo: guest cap=%dMB (host total was %dKB) → used≈%dMB avail≈%dMB",
|
||||
capMB, totalKB, info.UsedMemMB, info.AvailMemMB)
|
||||
}
|
||||
|
||||
if info.TotalMemMB > 0 {
|
||||
info.MemPercent = float64(info.UsedMemMB) / float64(info.TotalMemMB) * 100
|
||||
}
|
||||
debugf("[DEBUG] [system] readMemInfo: totalKB=%d availKB=%d → total=%dMB avail=%dMB used=%dMB (%.1f%%)",
|
||||
totalKB, availKB, info.TotalMemMB, info.AvailMemMB, info.UsedMemMB, info.MemPercent)
|
||||
} else {
|
||||
debugf("[DEBUG] [system] readMemInfo: could not parse MemTotal from /proc/meminfo")
|
||||
}
|
||||
|
||||
// guestMemTotalMB returns the guest's total RAM (MB) as reported by the Docker daemon. The daemon runs
|
||||
// inside the LXC, so `docker info` MemTotal reflects the guest's lxcfs-backed /proc/meminfo (the real
|
||||
// cap) — unlike the container's own /proc/meminfo, which shows the Proxmox host's RAM. Overridable in
|
||||
// tests via dockerMemTotalFn.
|
||||
func guestMemTotalMB() (uint64, bool) {
|
||||
if dockerMemTotalFn != nil {
|
||||
return dockerMemTotalFn()
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 4*time.Second)
|
||||
defer cancel()
|
||||
out, err := exec.CommandContext(ctx, "docker", "info", "--format", "{{.MemTotal}}").Output()
|
||||
if err != nil {
|
||||
return 0, false
|
||||
}
|
||||
bytesVal, err := strconv.ParseUint(strings.TrimSpace(string(out)), 10, 64)
|
||||
if err != nil || bytesVal == 0 {
|
||||
return 0, false
|
||||
}
|
||||
return bytesVal / (1024 * 1024), true
|
||||
}
|
||||
|
||||
// dockerMemTotalFn lets tests stub the docker-info read.
|
||||
var dockerMemTotalFn func() (uint64, bool)
|
||||
|
||||
// GuestMemTotalMB returns the guest's memory cap in MB (docker-info MemTotal), preferring the cgroup
|
||||
// limit when finite. ok=false if neither is determinable. The deploy memory guard uses this as the
|
||||
// accurate denominator (the controller container cannot read the guest cap from /proc — no lxcfs).
|
||||
func GuestMemTotalMB() (int, bool) {
|
||||
if v, ok := readCgroupMemLimitMB(cgroupRoot); ok && v > 0 {
|
||||
return int(v), true
|
||||
}
|
||||
if v, ok := guestMemTotalMB(); ok && v > 0 {
|
||||
return int(v), true
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// readCgroupMemLimitMB returns the cgroup memory limit in MB. It tries cgroup v2 (memory.max) first,
|
||||
// then v1 (memory/memory.limit_in_bytes). A sentinel ("max" on v2, or a near-uint64-max value on v1)
|
||||
// means "unlimited" → ok=false so the caller keeps the /proc/meminfo value.
|
||||
func readCgroupMemLimitMB(root string) (mb uint64, ok bool) {
|
||||
// cgroup v2
|
||||
if b, err := os.ReadFile(filepath.Join(root, "memory.max")); err == nil {
|
||||
s := strings.TrimSpace(string(b))
|
||||
if s == "max" {
|
||||
return 0, false
|
||||
}
|
||||
if v, err := strconv.ParseUint(s, 10, 64); err == nil && v > 0 {
|
||||
return v / (1024 * 1024), true
|
||||
}
|
||||
}
|
||||
// cgroup v1
|
||||
if b, err := os.ReadFile(filepath.Join(root, "memory", "memory.limit_in_bytes")); err == nil {
|
||||
s := strings.TrimSpace(string(b))
|
||||
if v, err := strconv.ParseUint(s, 10, 64); err == nil && v > 0 {
|
||||
// v1 "unlimited" is a huge page-aligned value near uint64 max; treat >= 1 PiB as unlimited.
|
||||
if v >= (1 << 50) {
|
||||
return 0, false
|
||||
}
|
||||
return v / (1024 * 1024), true
|
||||
}
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// readCgroupMemCurrentMB returns the cgroup current memory usage in MB (v2 memory.current, v1
|
||||
// memory/memory.usage_in_bytes). ok=false if unreadable.
|
||||
func readCgroupMemCurrentMB(root string) (mb uint64, ok bool) {
|
||||
for _, p := range []string{
|
||||
filepath.Join(root, "memory.current"),
|
||||
filepath.Join(root, "memory", "memory.usage_in_bytes"),
|
||||
} {
|
||||
if b, err := os.ReadFile(p); err == nil {
|
||||
if v, err := strconv.ParseUint(strings.TrimSpace(string(b)), 10, 64); err == nil {
|
||||
return v / (1024 * 1024), true
|
||||
}
|
||||
}
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// parseMemLine extracts the kB value from a /proc/meminfo line like "MemTotal: 16384000 kB"
|
||||
|
||||
@@ -18,3 +18,8 @@ func GetTotalMemoryMB() (int, error) {
|
||||
func GetMemoryMB() (totalMB, usedMB int, err error) {
|
||||
return 0, 0, fmt.Errorf("/proc/meminfo not available on this platform")
|
||||
}
|
||||
|
||||
// GuestMemTotalMB is not determinable on non-Linux platforms.
|
||||
func GuestMemTotalMB() (int, bool) {
|
||||
return 0, false
|
||||
}
|
||||
|
||||
@@ -30,6 +30,20 @@ func getTimezone() *time.Location {
|
||||
return webTimezone
|
||||
}
|
||||
|
||||
// routeUnpublished reports whether the reverse proxy (Traefik) is withholding a deployed stack's public
|
||||
// route because the container is not healthy. Traefik's docker provider only publishes a route to a
|
||||
// container that is healthy (or has no healthcheck); an unhealthy or restarting container yields a 404
|
||||
// at its URL even though the card "looks deployed". Templates use this to surface that distinctly (F5),
|
||||
// so an unhealthy app with a dead URL isn't mistaken for a merely-degraded-but-reachable one.
|
||||
func routeUnpublished(state stacks.ContainerState) bool {
|
||||
switch state {
|
||||
case stacks.StateUnhealthy, stacks.StateRestarting:
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// templateFuncMap returns the FuncMap used by all HTML templates.
|
||||
func (s *Server) templateFuncMap() template.FuncMap {
|
||||
loc := getTimezone()
|
||||
@@ -102,6 +116,7 @@ func (s *Server) templateFuncMap() template.FuncMap {
|
||||
return false
|
||||
}
|
||||
},
|
||||
"routeUnpublished": routeUnpublished,
|
||||
"logoURL": func(slug string) string {
|
||||
return s.cfg.AppLogoURL(slug)
|
||||
},
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"html/template"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
|
||||
)
|
||||
|
||||
// TestRouteUnpublished asserts F5: routeUnpublished is true exactly for the states where Traefik
|
||||
// withholds the public route (unhealthy / restarting) and false otherwise.
|
||||
func TestRouteUnpublished(t *testing.T) {
|
||||
cases := []struct {
|
||||
state stacks.ContainerState
|
||||
want bool
|
||||
}{
|
||||
{stacks.StateUnhealthy, true},
|
||||
{stacks.StateRestarting, true},
|
||||
{stacks.StateRunning, false},
|
||||
{stacks.StateStarting, false},
|
||||
{stacks.StateDeploying, false},
|
||||
{stacks.StateStopped, false},
|
||||
{stacks.StateExited, false},
|
||||
{stacks.StateNotDeployed, false},
|
||||
{stacks.StatePaused, false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := routeUnpublished(c.state); got != c.want {
|
||||
t.Errorf("routeUnpublished(%q) = %v, want %v", c.state, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestTemplatesParseWithFuncmap asserts the real embedded templates (including the stacks.html /
|
||||
// dashboard.html edits that reference routeUnpublished) parse with the production funcmap. Catches an
|
||||
// unregistered func or a template syntax error introduced by the F5 edits.
|
||||
func TestTemplatesParseWithFuncmap(t *testing.T) {
|
||||
s := &Server{cfg: &config.Config{}}
|
||||
if _, err := template.New("").Funcs(s.templateFuncMap()).ParseFS(templateFS, "templates/*.html"); err != nil {
|
||||
t.Fatalf("templates failed to parse with funcmap: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRouteUnpublishedIndicatorRenders asserts the dashboard/stacks card guard renders the distinct
|
||||
// indicator for a DEPLOYED + unhealthy stack, and NOT for a healthy one — the exact condition both
|
||||
// edited templates use ({{if and .Deployed (routeUnpublished .State)}}).
|
||||
func TestRouteUnpublishedIndicatorRenders(t *testing.T) {
|
||||
s := &Server{cfg: &config.Config{}}
|
||||
const frag = `{{if and .Deployed (routeUnpublished .State)}}URL-NOT-PUBLISHED{{end}}`
|
||||
tmpl, err := template.New("frag").Funcs(s.templateFuncMap()).Parse(frag)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
type row struct {
|
||||
Deployed bool
|
||||
State stacks.ContainerState
|
||||
}
|
||||
render := func(r row) string {
|
||||
var b bytes.Buffer
|
||||
if err := tmpl.Execute(&b, r); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
if got := render(row{Deployed: true, State: stacks.StateUnhealthy}); !strings.Contains(got, "URL-NOT-PUBLISHED") {
|
||||
t.Errorf("deployed+unhealthy should show the indicator, got %q", got)
|
||||
}
|
||||
if got := render(row{Deployed: true, State: stacks.StateRunning}); strings.Contains(got, "URL-NOT-PUBLISHED") {
|
||||
t.Errorf("deployed+running must NOT show the indicator, got %q", got)
|
||||
}
|
||||
if got := render(row{Deployed: false, State: stacks.StateUnhealthy}); strings.Contains(got, "URL-NOT-PUBLISHED") {
|
||||
t.Errorf("not-deployed must NOT show the indicator, got %q", got)
|
||||
}
|
||||
}
|
||||
@@ -149,6 +149,7 @@
|
||||
<div class="stack-actions">
|
||||
<span class="stack-state-label">{{stateLabel .State}}</span>
|
||||
{{if .Orphaned}}<span class="badge badge-orphaned">Elavult</span>{{end}}
|
||||
{{if and .Deployed (routeUnpublished .State)}}<span class="badge badge-route-unpublished" title="A proxy (Traefik) csak egészséges konténerhez publikál nyilvános útvonalat. Amíg az alkalmazás nem egészséges, az URL 404-et ad, pedig a konténer fut.">⚠ URL nem elérhető</span>{{end}}
|
||||
|
||||
{{if .Protected}}
|
||||
<span class="badge badge-protected">Védett</span>
|
||||
|
||||
@@ -30,6 +30,9 @@
|
||||
<a class="subdomain-link" href="https://{{$subdomain}}.{{$.Domain}}" target="_blank">
|
||||
{{$subdomain}}.{{$.Domain}} ↗
|
||||
</a>
|
||||
{{if and .Deployed (routeUnpublished .State)}}
|
||||
<span class="route-unpublished" title="A proxy (Traefik) csak egészséges konténerhez publikál nyilvános útvonalat. Amíg az alkalmazás nem egészséges, az URL nem érhető el (404), pedig a konténer fut.">⚠ URL nem elérhető – útvonal nincs publikálva</span>
|
||||
{{end}}
|
||||
{{end}}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -1249,6 +1249,20 @@ a.stat-card:hover {
|
||||
color: var(--orange);
|
||||
}
|
||||
|
||||
/* F5: an unhealthy/restarting deployed app has its public route withheld by Traefik (404 at the URL). */
|
||||
.badge-route-unpublished {
|
||||
background: var(--orange-bg);
|
||||
color: var(--orange);
|
||||
white-space: nowrap;
|
||||
}
|
||||
.route-unpublished {
|
||||
display: inline-block;
|
||||
margin-top: 2px;
|
||||
font-size: 0.8em;
|
||||
color: var(--orange);
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
/* Delete modal */
|
||||
.modal-overlay {
|
||||
position: fixed;
|
||||
|
||||
Reference in New Issue
Block a user