v0.58.0: infra-protection prevention layer for the OS/Docker-data split (Phase 2)
Reserved-buffer headroom guard on the Docker-data volume (system/dockervol.go, max(5GB,10%)); deploy-time hard gate refuses (HTTP 507) when below the buffer (api/router.go); deploy page warns + disables the button (deploy.html); runtime disk monitor confirmed to watch the Docker volume above the buffer. Log rotation baked into the golden (agent side). Phase 1 = felhom-agent v0.29.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,5 +1,30 @@
|
||||
## Changelog
|
||||
|
||||
### v0.58.0 — infra-protection prevention layer for the OS/Docker-data split (2026-06-13)
|
||||
|
||||
Phase 2 of the storage-split slice (Phase 1 = felhom-agent golden + provision). The OS rootfs and
|
||||
Docker data are split onto separate volumes for resilience; infra (controller/traefik/cloudflared/
|
||||
filebrowser) shares the one Docker data-root and is protected by **prevention, not placement**.
|
||||
|
||||
- **Reserved-buffer headroom guard (`internal/system/dockervol.go`):** `GetDockerVolumeHeadroom()`
|
||||
measures the Docker-data volume via `statfs("/")` (the controller container's root overlay is backed
|
||||
by the guest's `/var/lib/docker` volume) and computes a reserved floor `DockerVolumeReserveGB` =
|
||||
`max(5 GB, 10% of total)`. Fail-open on a measurement error (the buffer is a safety net, not a
|
||||
security control).
|
||||
- **Deploy-time hard gate (`internal/api/router.go` `deployStack`):** a new deploy is **refused** (HTTP
|
||||
507 + Hungarian message) when free space on the Docker-data volume is at/under the reserved buffer,
|
||||
so customer apps can't fill the volume the infra containers depend on.
|
||||
- **Deploy-page surfacing (`deploy.html`):** for a new deploy, when below the buffer the page shows a
|
||||
clear Hungarian warning and **disables** the "Telepítés indítása" button (mirrors the memory-blocked
|
||||
pattern) — the customer sees it before clicking; the API gate is the hard backstop.
|
||||
- **Runtime monitoring (2C):** confirmed `monitor/healthcheck.go` already watches `sysInfo.DiskPercent`
|
||||
= the Docker-data volume post-split (statfs `/`); warn 80% / crit 90% used trip ABOVE the 10%-free
|
||||
reserved buffer, so the customer is warned before the deploy gate engages. Comment added to make the
|
||||
"SSD disk" alert's target explicit.
|
||||
- **Log rotation (2D):** baked into the golden's `daemon.json` (`max-size 10m`, `max-file 3`) in the
|
||||
felhom-agent golden build — every guest inherits it. Per-app xfs-project-quota caps deferred.
|
||||
- Tests: `DockerVolumeReserveGB` floor/scale.
|
||||
|
||||
### v0.57.0 — UI fixes: stable host-storage list + per-app Tier-2 config panel (2026-06-13)
|
||||
|
||||
Part A of the UI-fixes/storage-spike spec (Part B is a build-nothing findings report).
|
||||
|
||||
@@ -346,6 +346,19 @@ func (r *Router) deployStack(w http.ResponseWriter, req *http.Request, name stri
|
||||
return
|
||||
}
|
||||
|
||||
// Prevention layer (storage-split): refuse a deploy when the Docker-data volume is at/under its
|
||||
// reserved buffer, so customer apps can't fill the volume the infra containers (controller,
|
||||
// traefik, cloudflared, filebrowser) depend on. Fail-OPEN on a measurement error — the buffer is
|
||||
// a safety net, not a security control, so a transient statfs failure must not block all deploys.
|
||||
if hr := system.GetDockerVolumeHeadroom(); hr.OK && hr.BelowReserve {
|
||||
r.logger.Printf("[WARN] [api] Deploy refused for %s: Docker volume below reserved buffer (%.1fG free, reserve %.1fG of %.0fG)",
|
||||
name, hr.AvailGB, hr.ReserveGB, hr.TotalGB)
|
||||
writeJSON(w, http.StatusInsufficientStorage, apiResponse{OK: false, Error: fmt.Sprintf(
|
||||
"Nincs elég szabad tárhely a telepítéshez: csak %.0f GB szabad, és a rendszer %.0f GB tartalékot tart fenn az alapszolgáltatások (vezérlő, proxy) védelmében. Szabadítson fel helyet, vagy bővítse a tárhelyet.",
|
||||
hr.AvailGB, hr.ReserveGB)})
|
||||
return
|
||||
}
|
||||
|
||||
deployReq := stacks.DeployRequest{
|
||||
StackName: name,
|
||||
Values: body.Values,
|
||||
|
||||
@@ -44,7 +44,12 @@ func RunHealthCheck(cfg *config.Config, cpuCollector *system.CPUCollector, stora
|
||||
sysInfo.CPUPercent, sysInfo.TemperatureCelsius, sysInfo.TemperatureSource)
|
||||
}
|
||||
|
||||
// 1. Disk usage (SSD)
|
||||
// 1. Disk usage (SSD). NOTE (storage-split): sysInfo.DiskPercent statfs's the controller
|
||||
// container's "/", whose overlay upperdir lives on the guest's /var/lib/docker volume — so this
|
||||
// IS the Docker-data volume guard (post-split it's the dedicated data volume; pre-split it's the
|
||||
// rootfs — either way it's wherever Docker's data-root lives). Warn at 80% / crit at 90% used
|
||||
// trips ABOVE the prevention layer's 10%-free reserved buffer, so the customer is warned before
|
||||
// the deploy gate even engages.
|
||||
if sysInfo.DiskPercent > 0 {
|
||||
if sysInfo.DiskPercent >= float64(cfg.Monitoring.Thresholds.DiskCritPercent) {
|
||||
report.Issues = append(report.Issues, fmt.Sprintf("SSD disk usage critical: %.0f%%", sysInfo.DiskPercent))
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
package system
|
||||
|
||||
// Docker-data volume headroom — the infra-protection prevention layer (storage-split slice).
|
||||
//
|
||||
// After the OS/Docker-data split, /var/lib/docker is a dedicated volume holding ALL images +
|
||||
// overlay + named volumes (controller/traefik/cloudflared/filebrowser AND customer apps). Infra is
|
||||
// protected by PREVENTION, not placement: a reserved buffer the controller refuses to deploy into,
|
||||
// so the volume can't be filled to the point the infra containers can't write. This file measures
|
||||
// that volume and computes the reserved-buffer verdict; the deploy gate + UI consume it.
|
||||
|
||||
// DockerVolumePath is the path whose filesystem backs Docker's data-root as seen from INSIDE the
|
||||
// controller container. The controller's own root ("/") is an overlay whose upperdir lives on the
|
||||
// guest's /var/lib/docker volume, so statfs("/") reports THAT volume's capacity/free — i.e. the
|
||||
// Docker-data volume the split isolates (and, pre-split, the rootfs — correct either way: it is
|
||||
// always wherever Docker's data-root lives).
|
||||
const DockerVolumePath = "/"
|
||||
|
||||
// DockerVolumeReserveGB returns the reserved-buffer floor (GiB) for the Docker-data volume:
|
||||
// max(5 GB, 10% of total). Deploys are refused once free space reaches this floor so the infra
|
||||
// containers keep running even when apps would otherwise fill the volume. (10% rather than the
|
||||
// Tier-2 guard's 20%: on a large data volume 20% would reserve an absurd amount; infra needs only
|
||||
// modest headroom for logs/overlay writes, and the runtime disk-warning at 80% used trips first.)
|
||||
func DockerVolumeReserveGB(totalGB float64) float64 {
|
||||
reserve := totalGB * 0.10
|
||||
if reserve < 5.0 {
|
||||
reserve = 5.0
|
||||
}
|
||||
return reserve
|
||||
}
|
||||
|
||||
// DockerVolumeHeadroom is the Docker-data volume's capacity view for the prevention layer.
|
||||
type DockerVolumeHeadroom struct {
|
||||
TotalGB float64
|
||||
AvailGB float64
|
||||
ReserveGB float64
|
||||
BelowReserve bool // free space is at/under the reserved buffer → refuse new deploys
|
||||
OK bool // stats were readable (false → callers must FAIL-OPEN, not block)
|
||||
}
|
||||
|
||||
// GetDockerVolumeHeadroom measures the Docker-data volume and computes the reserved-buffer verdict.
|
||||
// OK=false when the stats can't be read; callers MUST fail-open (do not block deploys on a transient
|
||||
// measurement error — the buffer is a safety net, not a security control).
|
||||
func GetDockerVolumeHeadroom() DockerVolumeHeadroom {
|
||||
di := GetDiskUsage(DockerVolumePath)
|
||||
if di == nil || di.TotalGB <= 0 {
|
||||
return DockerVolumeHeadroom{}
|
||||
}
|
||||
reserve := DockerVolumeReserveGB(di.TotalGB)
|
||||
return DockerVolumeHeadroom{
|
||||
TotalGB: di.TotalGB,
|
||||
AvailGB: di.AvailGB,
|
||||
ReserveGB: reserve,
|
||||
BelowReserve: di.AvailGB <= reserve,
|
||||
OK: true,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
package system
|
||||
|
||||
import "testing"
|
||||
|
||||
// DockerVolumeReserveGB = max(5 GB, 10% of total): a flat 5 GB floor for small volumes, scaling to
|
||||
// 10% on larger ones (so a 256 GB data volume reserves ~25.6 GB, not the Tier-2 guard's 20%).
|
||||
func TestDockerVolumeReserveGB(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
totalGB float64
|
||||
want float64
|
||||
}{
|
||||
{"tiny volume uses the 5G floor", 16, 5},
|
||||
{"50G volume: 10% = 5G ties the floor", 50, 5},
|
||||
{"100G volume: 10% dominates", 100, 10},
|
||||
{"256G data volume", 256, 25.6},
|
||||
{"zero total still floors at 5G", 0, 5},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := DockerVolumeReserveGB(c.totalGB); got != c.want {
|
||||
t.Errorf("%s: DockerVolumeReserveGB(%.0f) = %.2f, want %.2f", c.name, c.totalGB, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -332,6 +332,17 @@ func (s *Server) deployHandler(w http.ResponseWriter, r *http.Request, name stri
|
||||
}
|
||||
data["StoragePaths"] = deployPaths
|
||||
|
||||
// Prevention layer (storage-split): surface the Docker-data volume's reserved-buffer state so the
|
||||
// customer sees BEFORE deploying when free space is too low (the API gate also hard-refuses). Only
|
||||
// meaningful for a NEW deploy (an existing app's config save doesn't consume fresh image space).
|
||||
if !alreadyDeployed {
|
||||
if hr := system.GetDockerVolumeHeadroom(); hr.OK {
|
||||
data["DockerBelowReserve"] = hr.BelowReserve
|
||||
data["DockerFreeHuman"] = formatFreeSpace(hr.AvailGB)
|
||||
data["DockerReserveHuman"] = formatFreeSpace(hr.ReserveGB)
|
||||
}
|
||||
}
|
||||
|
||||
// Effective subdomain for "Megnyitás" button
|
||||
if alreadyDeployed && appCfg != nil {
|
||||
if sd, ok := appCfg.Env["SUBDOMAIN"]; ok && sd != "" {
|
||||
|
||||
@@ -427,6 +427,13 @@
|
||||
{{end}}
|
||||
|
||||
<form id="deploy-form" class="deploy-form">
|
||||
{{if .DockerBelowReserve}}
|
||||
<div class="alert alert-warning" style="margin-bottom:1rem">
|
||||
⚠ Nincs elég szabad tárhely a telepítéshez. Jelenleg {{.DockerFreeHuman}} szabad, és a rendszer
|
||||
{{.DockerReserveHuman}} tartalékot tart fenn az alapszolgáltatások (vezérlő, proxy) védelmében.
|
||||
A telepítés ezért átmenetileg le van tiltva — szabadítson fel helyet, vagy bővítse a tárhelyet.
|
||||
</div>
|
||||
{{end}}
|
||||
{{if .AutoFields}}
|
||||
<div class="form-section">
|
||||
<h4>Automatikusan generált értékek</h4>
|
||||
@@ -568,7 +575,7 @@
|
||||
|
||||
{{if not .AlreadyDeployed}}
|
||||
<div class="deploy-actions">
|
||||
<button type="submit" class="btn btn-primary btn-lg"{{if and .MemoryInfo (index .MemoryInfo "Blocked")}} disabled title="Nincs elég memória"{{end}}>Telepítés indítása</button>
|
||||
<button type="submit" class="btn btn-primary btn-lg"{{if .DockerBelowReserve}} disabled title="Nincs elég szabad tárhely"{{else if and .MemoryInfo (index .MemoryInfo "Blocked")}} disabled title="Nincs elég memória"{{end}}>Telepítés indítása</button>
|
||||
<a href="/stacks" class="btn btn-outline">Mégsem</a>
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
Reference in New Issue
Block a user