v0.58.0: infra-protection prevention layer for the OS/Docker-data split (Phase 2)
Reserved-buffer headroom guard on the Docker-data volume (system/dockervol.go, max(5GB,10%)); deploy-time hard gate refuses (HTTP 507) when below the buffer (api/router.go); deploy page warns + disables the button (deploy.html); runtime disk monitor confirmed to watch the Docker volume above the buffer. Log rotation baked into the golden (agent side). Phase 1 = felhom-agent v0.29.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,5 +1,30 @@
|
|||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
### v0.58.0 — infra-protection prevention layer for the OS/Docker-data split (2026-06-13)
|
||||||
|
|
||||||
|
Phase 2 of the storage-split slice (Phase 1 = felhom-agent golden + provision). The OS rootfs and
|
||||||
|
Docker data are split onto separate volumes for resilience; infra (controller/traefik/cloudflared/
|
||||||
|
filebrowser) shares the one Docker data-root and is protected by **prevention, not placement**.
|
||||||
|
|
||||||
|
- **Reserved-buffer headroom guard (`internal/system/dockervol.go`):** `GetDockerVolumeHeadroom()`
|
||||||
|
measures the Docker-data volume via `statfs("/")` (the controller container's root overlay is backed
|
||||||
|
by the guest's `/var/lib/docker` volume) and computes a reserved floor `DockerVolumeReserveGB` =
|
||||||
|
`max(5 GB, 10% of total)`. Fail-open on a measurement error (the buffer is a safety net, not a
|
||||||
|
security control).
|
||||||
|
- **Deploy-time hard gate (`internal/api/router.go` `deployStack`):** a new deploy is **refused** (HTTP
|
||||||
|
507 + Hungarian message) when free space on the Docker-data volume is at/under the reserved buffer,
|
||||||
|
so customer apps can't fill the volume the infra containers depend on.
|
||||||
|
- **Deploy-page surfacing (`deploy.html`):** for a new deploy, when below the buffer the page shows a
|
||||||
|
clear Hungarian warning and **disables** the "Telepítés indítása" button (mirrors the memory-blocked
|
||||||
|
pattern) — the customer sees it before clicking; the API gate is the hard backstop.
|
||||||
|
- **Runtime monitoring (2C):** confirmed `monitor/healthcheck.go` already watches `sysInfo.DiskPercent`
|
||||||
|
= the Docker-data volume post-split (statfs `/`); warn 80% / crit 90% used trip ABOVE the 10%-free
|
||||||
|
reserved buffer, so the customer is warned before the deploy gate engages. Comment added to make the
|
||||||
|
"SSD disk" alert's target explicit.
|
||||||
|
- **Log rotation (2D):** baked into the golden's `daemon.json` (`max-size 10m`, `max-file 3`) in the
|
||||||
|
felhom-agent golden build — every guest inherits it. Per-app xfs-project-quota caps deferred.
|
||||||
|
- Tests: `DockerVolumeReserveGB` floor/scale.
|
||||||
|
|
||||||
### v0.57.0 — UI fixes: stable host-storage list + per-app Tier-2 config panel (2026-06-13)
|
### v0.57.0 — UI fixes: stable host-storage list + per-app Tier-2 config panel (2026-06-13)
|
||||||
|
|
||||||
Part A of the UI-fixes/storage-spike spec (Part B is a build-nothing findings report).
|
Part A of the UI-fixes/storage-spike spec (Part B is a build-nothing findings report).
|
||||||
|
|||||||
@@ -346,6 +346,19 @@ func (r *Router) deployStack(w http.ResponseWriter, req *http.Request, name stri
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Prevention layer (storage-split): refuse a deploy when the Docker-data volume is at/under its
|
||||||
|
// reserved buffer, so customer apps can't fill the volume the infra containers (controller,
|
||||||
|
// traefik, cloudflared, filebrowser) depend on. Fail-OPEN on a measurement error — the buffer is
|
||||||
|
// a safety net, not a security control, so a transient statfs failure must not block all deploys.
|
||||||
|
if hr := system.GetDockerVolumeHeadroom(); hr.OK && hr.BelowReserve {
|
||||||
|
r.logger.Printf("[WARN] [api] Deploy refused for %s: Docker volume below reserved buffer (%.1fG free, reserve %.1fG of %.0fG)",
|
||||||
|
name, hr.AvailGB, hr.ReserveGB, hr.TotalGB)
|
||||||
|
writeJSON(w, http.StatusInsufficientStorage, apiResponse{OK: false, Error: fmt.Sprintf(
|
||||||
|
"Nincs elég szabad tárhely a telepítéshez: csak %.0f GB szabad, és a rendszer %.0f GB tartalékot tart fenn az alapszolgáltatások (vezérlő, proxy) védelmében. Szabadítson fel helyet, vagy bővítse a tárhelyet.",
|
||||||
|
hr.AvailGB, hr.ReserveGB)})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
deployReq := stacks.DeployRequest{
|
deployReq := stacks.DeployRequest{
|
||||||
StackName: name,
|
StackName: name,
|
||||||
Values: body.Values,
|
Values: body.Values,
|
||||||
|
|||||||
@@ -44,7 +44,12 @@ func RunHealthCheck(cfg *config.Config, cpuCollector *system.CPUCollector, stora
|
|||||||
sysInfo.CPUPercent, sysInfo.TemperatureCelsius, sysInfo.TemperatureSource)
|
sysInfo.CPUPercent, sysInfo.TemperatureCelsius, sysInfo.TemperatureSource)
|
||||||
}
|
}
|
||||||
|
|
||||||
// 1. Disk usage (SSD)
|
// 1. Disk usage (SSD). NOTE (storage-split): sysInfo.DiskPercent statfs's the controller
|
||||||
|
// container's "/", whose overlay upperdir lives on the guest's /var/lib/docker volume — so this
|
||||||
|
// IS the Docker-data volume guard (post-split it's the dedicated data volume; pre-split it's the
|
||||||
|
// rootfs — either way it's wherever Docker's data-root lives). Warn at 80% / crit at 90% used
|
||||||
|
// trips ABOVE the prevention layer's 10%-free reserved buffer, so the customer is warned before
|
||||||
|
// the deploy gate even engages.
|
||||||
if sysInfo.DiskPercent > 0 {
|
if sysInfo.DiskPercent > 0 {
|
||||||
if sysInfo.DiskPercent >= float64(cfg.Monitoring.Thresholds.DiskCritPercent) {
|
if sysInfo.DiskPercent >= float64(cfg.Monitoring.Thresholds.DiskCritPercent) {
|
||||||
report.Issues = append(report.Issues, fmt.Sprintf("SSD disk usage critical: %.0f%%", sysInfo.DiskPercent))
|
report.Issues = append(report.Issues, fmt.Sprintf("SSD disk usage critical: %.0f%%", sysInfo.DiskPercent))
|
||||||
|
|||||||
@@ -0,0 +1,56 @@
|
|||||||
|
package system
|
||||||
|
|
||||||
|
// Docker-data volume headroom — the infra-protection prevention layer (storage-split slice).
|
||||||
|
//
|
||||||
|
// After the OS/Docker-data split, /var/lib/docker is a dedicated volume holding ALL images +
|
||||||
|
// overlay + named volumes (controller/traefik/cloudflared/filebrowser AND customer apps). Infra is
|
||||||
|
// protected by PREVENTION, not placement: a reserved buffer the controller refuses to deploy into,
|
||||||
|
// so the volume can't be filled to the point the infra containers can't write. This file measures
|
||||||
|
// that volume and computes the reserved-buffer verdict; the deploy gate + UI consume it.
|
||||||
|
|
||||||
|
// DockerVolumePath is the path whose filesystem backs Docker's data-root as seen from INSIDE the
|
||||||
|
// controller container. The controller's own root ("/") is an overlay whose upperdir lives on the
|
||||||
|
// guest's /var/lib/docker volume, so statfs("/") reports THAT volume's capacity/free — i.e. the
|
||||||
|
// Docker-data volume the split isolates (and, pre-split, the rootfs — correct either way: it is
|
||||||
|
// always wherever Docker's data-root lives).
|
||||||
|
const DockerVolumePath = "/"
|
||||||
|
|
||||||
|
// DockerVolumeReserveGB returns the reserved-buffer floor (GiB) for the Docker-data volume:
|
||||||
|
// max(5 GB, 10% of total). Deploys are refused once free space reaches this floor so the infra
|
||||||
|
// containers keep running even when apps would otherwise fill the volume. (10% rather than the
|
||||||
|
// Tier-2 guard's 20%: on a large data volume 20% would reserve an absurd amount; infra needs only
|
||||||
|
// modest headroom for logs/overlay writes, and the runtime disk-warning at 80% used trips first.)
|
||||||
|
func DockerVolumeReserveGB(totalGB float64) float64 {
|
||||||
|
reserve := totalGB * 0.10
|
||||||
|
if reserve < 5.0 {
|
||||||
|
reserve = 5.0
|
||||||
|
}
|
||||||
|
return reserve
|
||||||
|
}
|
||||||
|
|
||||||
|
// DockerVolumeHeadroom is the Docker-data volume's capacity view for the prevention layer.
|
||||||
|
type DockerVolumeHeadroom struct {
|
||||||
|
TotalGB float64
|
||||||
|
AvailGB float64
|
||||||
|
ReserveGB float64
|
||||||
|
BelowReserve bool // free space is at/under the reserved buffer → refuse new deploys
|
||||||
|
OK bool // stats were readable (false → callers must FAIL-OPEN, not block)
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetDockerVolumeHeadroom measures the Docker-data volume and computes the reserved-buffer verdict.
|
||||||
|
// OK=false when the stats can't be read; callers MUST fail-open (do not block deploys on a transient
|
||||||
|
// measurement error — the buffer is a safety net, not a security control).
|
||||||
|
func GetDockerVolumeHeadroom() DockerVolumeHeadroom {
|
||||||
|
di := GetDiskUsage(DockerVolumePath)
|
||||||
|
if di == nil || di.TotalGB <= 0 {
|
||||||
|
return DockerVolumeHeadroom{}
|
||||||
|
}
|
||||||
|
reserve := DockerVolumeReserveGB(di.TotalGB)
|
||||||
|
return DockerVolumeHeadroom{
|
||||||
|
TotalGB: di.TotalGB,
|
||||||
|
AvailGB: di.AvailGB,
|
||||||
|
ReserveGB: reserve,
|
||||||
|
BelowReserve: di.AvailGB <= reserve,
|
||||||
|
OK: true,
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
package system
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
// DockerVolumeReserveGB = max(5 GB, 10% of total): a flat 5 GB floor for small volumes, scaling to
|
||||||
|
// 10% on larger ones (so a 256 GB data volume reserves ~25.6 GB, not the Tier-2 guard's 20%).
|
||||||
|
func TestDockerVolumeReserveGB(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
totalGB float64
|
||||||
|
want float64
|
||||||
|
}{
|
||||||
|
{"tiny volume uses the 5G floor", 16, 5},
|
||||||
|
{"50G volume: 10% = 5G ties the floor", 50, 5},
|
||||||
|
{"100G volume: 10% dominates", 100, 10},
|
||||||
|
{"256G data volume", 256, 25.6},
|
||||||
|
{"zero total still floors at 5G", 0, 5},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
if got := DockerVolumeReserveGB(c.totalGB); got != c.want {
|
||||||
|
t.Errorf("%s: DockerVolumeReserveGB(%.0f) = %.2f, want %.2f", c.name, c.totalGB, got, c.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -332,6 +332,17 @@ func (s *Server) deployHandler(w http.ResponseWriter, r *http.Request, name stri
|
|||||||
}
|
}
|
||||||
data["StoragePaths"] = deployPaths
|
data["StoragePaths"] = deployPaths
|
||||||
|
|
||||||
|
// Prevention layer (storage-split): surface the Docker-data volume's reserved-buffer state so the
|
||||||
|
// customer sees BEFORE deploying when free space is too low (the API gate also hard-refuses). Only
|
||||||
|
// meaningful for a NEW deploy (an existing app's config save doesn't consume fresh image space).
|
||||||
|
if !alreadyDeployed {
|
||||||
|
if hr := system.GetDockerVolumeHeadroom(); hr.OK {
|
||||||
|
data["DockerBelowReserve"] = hr.BelowReserve
|
||||||
|
data["DockerFreeHuman"] = formatFreeSpace(hr.AvailGB)
|
||||||
|
data["DockerReserveHuman"] = formatFreeSpace(hr.ReserveGB)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Effective subdomain for "Megnyitás" button
|
// Effective subdomain for "Megnyitás" button
|
||||||
if alreadyDeployed && appCfg != nil {
|
if alreadyDeployed && appCfg != nil {
|
||||||
if sd, ok := appCfg.Env["SUBDOMAIN"]; ok && sd != "" {
|
if sd, ok := appCfg.Env["SUBDOMAIN"]; ok && sd != "" {
|
||||||
|
|||||||
@@ -427,6 +427,13 @@
|
|||||||
{{end}}
|
{{end}}
|
||||||
|
|
||||||
<form id="deploy-form" class="deploy-form">
|
<form id="deploy-form" class="deploy-form">
|
||||||
|
{{if .DockerBelowReserve}}
|
||||||
|
<div class="alert alert-warning" style="margin-bottom:1rem">
|
||||||
|
⚠ Nincs elég szabad tárhely a telepítéshez. Jelenleg {{.DockerFreeHuman}} szabad, és a rendszer
|
||||||
|
{{.DockerReserveHuman}} tartalékot tart fenn az alapszolgáltatások (vezérlő, proxy) védelmében.
|
||||||
|
A telepítés ezért átmenetileg le van tiltva — szabadítson fel helyet, vagy bővítse a tárhelyet.
|
||||||
|
</div>
|
||||||
|
{{end}}
|
||||||
{{if .AutoFields}}
|
{{if .AutoFields}}
|
||||||
<div class="form-section">
|
<div class="form-section">
|
||||||
<h4>Automatikusan generált értékek</h4>
|
<h4>Automatikusan generált értékek</h4>
|
||||||
@@ -568,7 +575,7 @@
|
|||||||
|
|
||||||
{{if not .AlreadyDeployed}}
|
{{if not .AlreadyDeployed}}
|
||||||
<div class="deploy-actions">
|
<div class="deploy-actions">
|
||||||
<button type="submit" class="btn btn-primary btn-lg"{{if and .MemoryInfo (index .MemoryInfo "Blocked")}} disabled title="Nincs elég memória"{{end}}>Telepítés indítása</button>
|
<button type="submit" class="btn btn-primary btn-lg"{{if .DockerBelowReserve}} disabled title="Nincs elég szabad tárhely"{{else if and .MemoryInfo (index .MemoryInfo "Blocked")}} disabled title="Nincs elég memória"{{end}}>Telepítés indítása</button>
|
||||||
<a href="/stacks" class="btn btn-outline">Mégsem</a>
|
<a href="/stacks" class="btn btn-outline">Mégsem</a>
|
||||||
</div>
|
</div>
|
||||||
{{end}}
|
{{end}}
|
||||||
|
|||||||
Reference in New Issue
Block a user