v0.237.0: the Update button takes a backup first, and tells the truth (update arc slice 4 — R-448, R-443, R-439)
gates / gates (push) Successful in 13s
gates / gates (push) Successful in 13s
POST /api/stacks/{name}/update is now a guarded job answering 202:
cheap refusals (hold — R-439, busy, migration, deploying, memory via the
deploy's own memoryVerdict, a fixed 2 GB disk floor, and no restorable
Tier-2 copy) → backup-first when the proven copy is older than
update.backup_max_age (24h) → safety dump BEFORE the pin moves → pin →
pull (failure puts the pin back) → up → health (.felhom.yml check or 60 s
settle, update.health_timeout 5m). Not healthy → the app is stopped and
HELD (RestoreHold reason update_failed, same store and gate as R-379) and
the page names the backup to restore from; the pin stays. Success is only
ever update_phase=done after health (R-443). UpdateStack is deleted.
The restorable-unit predicate is EXTRACTED to backup.Tier2UnitRestorePoint
and shared with the backups page (row pinned unchanged). The copy is aged
by the last successful Tier-2 copy, not the manifest created_at — measured
on demo-hp that created_at moves only on definition changes.
Crash safety: update-journal.json before each phase; RecoverUpdates before
the boot sweep, ResumeInterruptedUpdates after the guards are wired.
Three unattended start paths ignored a hold and now honour it: the
drive-return gate (restart + boot recreate) and the nightly volume dump.
The nightly capture and Tier-2 run skip held apps so the restore point
survives. No automatic rollback — measured per-app; route back = restore.
Tests A–H across stacks/backup/api/web/cmd; six red-proofs seen to fail.
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
+48
-4
@@ -472,7 +472,7 @@ through them: `EffectiveLifecycle()`, `CanInstall()`, `IsAbandoned()`.
|
||||
Two additions, and **neither changes how an update behaves**. Slice 1 is a record; slice 2 is a label.
|
||||
|
||||
**Slice 1 — `app.yaml` gains `installed_images`.** After every successful `compose up` from
|
||||
`StartStack`, `RestartStack`, `UpdateStack` and the deploy path, `Manager.recordInstalledImages`
|
||||
`StartStack`, `RestartStack`, the guarded update (v0.237.0; `UpdateStack` before it) and the deploy path, `Manager.recordInstalledImages`
|
||||
(`internal/stacks/installed.go`) reads what each container is ACTUALLY running and writes it down,
|
||||
**keyed by compose SERVICE name**:
|
||||
|
||||
@@ -527,8 +527,8 @@ the current template pins and returns a `*MetaBadge` rendered by the existing `m
|
||||
page. **Known limitation:** for the 23 floating pins (`postgres:16-alpine`, `mariadb:11.6`, …) the
|
||||
reference can be identical while the image behind it has moved, so those apps can read „Naprakész"
|
||||
when they may not be. Digest-level comparison needs a registry query and is deferred.
|
||||
- **Information only.** The badge is wired to no action; `Frissítés`/`Újraindítás`/`Leállítás` are
|
||||
byte-identical to before (`TestScenarioE_TheUpdateButtonIsUntouched`).
|
||||
- **Information only.** The badge is wired to no action. (What the `Frissítés` button itself does
|
||||
changed in v0.237.0 — see "The guarded update" below.)
|
||||
|
||||
Reasoning and the seven-slice plan: `felhom.eu/documentation/architecture/09-update-architecture.md`.
|
||||
|
||||
@@ -539,7 +539,7 @@ deliberate Update moves it. Everything else in a template — health checks, mem
|
||||
fields — still arrives on the 15-minute cycle, and a broken definition still repairs itself.
|
||||
|
||||
- **`app.yaml` gains `pinned_images`** (service → ref): what the app is SUPPOSED to run. **Not**
|
||||
`installed_images`, which is an observation. Written only by the deploy path, `UpdateStack`, a
|
||||
`installed_images`, which is an observation. Written only by the deploy path, the guarded update, a
|
||||
restore, and the one-time `AdoptPins`. **Absent = unpinned = pre-v0.235.0 behaviour.**
|
||||
- **`applied-compose.yml`** in the stack dir stores the exact definition the pin came from. The
|
||||
syncer copies only `docker-compose.yml` and `.felhom.yml`, so that name is safe.
|
||||
@@ -555,6 +555,50 @@ fields — still arrives on the 15-minute cycle, and a broken definition still r
|
||||
|
||||
Reasoning: `felhom.eu/documentation/architecture/09-update-architecture.md` §3, §5.
|
||||
|
||||
#### The guarded update (v0.237.0 — update arc slice 4)
|
||||
|
||||
**`POST /api/stacks/{name}/update` no longer updates on the spot.** It refuses what it must, starts a
|
||||
job, and answers **202**. The page polls `GET /api/stacks/{name}`.
|
||||
|
||||
| field | meaning |
|
||||
|---|---|
|
||||
| `updating` | a guarded update is in progress |
|
||||
| `update_phase` / `update_phase_label` | `checking`, `backing-up`, `safety-dump`, `pinning`, `pulling`, `starting`, `verifying`, `done`, `failed` — and the Hungarian label for each |
|
||||
| `update_error` | the customer sentence when the update did not complete |
|
||||
| `hold_reason` | the hold's sentence while the app is held (failed update OR failed restore) |
|
||||
|
||||
**Refused with 409 before anything moves:** held; a backup, restore, app-data op or quiesce holding
|
||||
the app; a migration; already updating; deploying; not enough memory for the NEW template's request
|
||||
(the deploy's own `memoryVerdict`, releasing the app's current request); less than **2 GB** free on
|
||||
the Docker data root (a fixed floor — image sizes are not known without a registry query); and **no
|
||||
restorable backup** (no openable Tier-2 recovery unit with a proven copy).
|
||||
|
||||
**The sequence.** A proven copy older than `update.backup_max_age` is refreshed first
|
||||
(`RunAppBackupNow`: this app's DB dump, volume dump, unit capture, Tier-2 copy). Then a database
|
||||
safety dump, then the pin moves, then pull, `up`, and the health wait (`.felhom.yml` check, or 60 s of
|
||||
every container running for an app with none; bounded by `update.health_timeout`). **A failed pull
|
||||
puts the pin back. An app that does not become healthy is stopped and HELD** — the pin stays on the
|
||||
new version, and the hold sentence names the backup it can be restored from. A successful unit
|
||||
restore lifts an update hold.
|
||||
|
||||
**Config (`controller.yaml`):**
|
||||
|
||||
```yaml
|
||||
update:
|
||||
backup_max_age: 24h # a proven Tier-2 copy older than this is refreshed before the update
|
||||
health_timeout: 5m # how long the new version has to become healthy before the app is held
|
||||
```
|
||||
|
||||
**Crash safety:** `<data>/update-journal.json` is written before every phase; `RecoverUpdates` (before
|
||||
the boot sweep) puts a pin back or marks an interrupted update for `ResumeInterruptedUpdates`.
|
||||
|
||||
**Every unattended start path honours a hold:** the boot sweep and the app-stop guard (as before), and
|
||||
since v0.237.0 the drive-return gate and the nightly volume dump. The nightly capture and Tier-2 run
|
||||
skip a held app so its restore point is not overwritten.
|
||||
|
||||
**Not done, deliberately:** the old version is never put back automatically — whether that works is
|
||||
per-app and was measured unpredictable. Reasoning: `felhom.eu/documentation/architecture/09-update-architecture.md` §6.
|
||||
|
||||
#### App Info Pages
|
||||
|
||||
Each app can define rich metadata in `.felhom.yml`:
|
||||
|
||||
@@ -476,9 +476,20 @@ func main() {
|
||||
// R-171: hand the boot sweep the settings it needs to answer "is this app's drive live?" BEFORE
|
||||
// the goroutine starts — an unwired gate is silently the pre-v0.190.0 behaviour that started apps
|
||||
// onto absent drives. TestMainWiresBootDriveGate walks this file's AST for the assignment.
|
||||
// --- Slice 4: guarded-update crash recovery (Scenario G) ---
|
||||
// BEFORE the boot reconciler, and the order is load-bearing: an update interrupted after `up` is
|
||||
// marked Updating here, and bootDriveGate refuses an Updating app — started after the sweep, the
|
||||
// sweep could bring up a half-updated app with no record of why. Pin-backs for updates interrupted
|
||||
// before anything ran happen here too. The resumed health wait itself is launched further down,
|
||||
// once the backup side is wired, because a resumed update that fails must be able to HOLD.
|
||||
if resumed := stackMgr.RecoverUpdates(); len(resumed) > 0 {
|
||||
logger.Printf("[WARN] [update] %d interrupted update(s) will resume after the backup side is wired: %v", len(resumed), resumed)
|
||||
}
|
||||
|
||||
bootDriveSettings = sett
|
||||
bootQuiesceLoop = quiesceLoop
|
||||
bootAppStopGuard = appStopGuard
|
||||
bootStackMgr = stackMgr
|
||||
go runBootReconcile(ctx, stackMgr, logger)
|
||||
|
||||
// --- Start CPU collector ---
|
||||
@@ -537,6 +548,16 @@ func main() {
|
||||
backupMgr.SetSharesReconciler(stackMgr.ReconcileSamba)
|
||||
}
|
||||
|
||||
// --- Slice 4: the guarded update's backup side ---
|
||||
// Wired unconditionally: with backup disabled the adapter answers "no restore point" and every
|
||||
// update is refused with the no-backup sentence — the precondition cannot be met, which is true.
|
||||
// An UNWIRED manager also refuses (fail closed); TestSlice4_UpdateGuardsAreWiredAtStartup walks
|
||||
// this file for the call, because a seam built and never wired has shipped here seven times.
|
||||
stackMgr.SetUpdateGuards(&updateGuardsAdapter{b: backupMgr, q: quiesceLoop})
|
||||
if n := stackMgr.ResumeInterruptedUpdates(ctx); n > 0 {
|
||||
logger.Printf("[WARN] [update] resumed %d interrupted update(s)", n)
|
||||
}
|
||||
|
||||
// SLICE 2: the offsite apply-bridge is launched further down, AFTER the self-updater is constructed
|
||||
// (R-71a: the bridge's settle-gate reads the updater's floor/update-running state to defer the
|
||||
// consume past a managed day-0 floor-update). See "offsite apply-bridge" below.
|
||||
@@ -1906,6 +1927,7 @@ var (
|
||||
bootDriveSettings *settings.Settings
|
||||
bootQuiesceLoop *quiesce.Loop
|
||||
bootAppStopGuard *backup.AppStopGuard
|
||||
bootStackMgr *stacks.Manager
|
||||
)
|
||||
|
||||
// bootDriveGate answers bootrecon.StartGate for the real controller. It enforces §8.2: an app that
|
||||
@@ -1943,6 +1965,11 @@ func (g bootDriveGate) MayStart(stackName string) (bool, string) {
|
||||
if bootQuiesceLoop.SuppressedStacks()[stackName] {
|
||||
return false, "a whole-guest backup (quiesce) is holding it — the quiesce loop restarts its own stacks"
|
||||
}
|
||||
// 1b. slice 4: a guarded update is moving this app (including one RecoverUpdates marked for
|
||||
// resumption). The update job owns the bring-up and ends in healthy or held.
|
||||
if bootStackMgr != nil && bootStackMgr.IsUpdating(stackName) {
|
||||
return false, "a guarded update is in progress — the update brings it up and verifies it"
|
||||
}
|
||||
// 2. an app-data operation in flight
|
||||
for _, held := range bootAppStopGuard.HeldStacks() {
|
||||
if held == stackName {
|
||||
@@ -1985,6 +2012,9 @@ func (g driveStartGate) MayStart(stackName string) (bool, string) {
|
||||
// app-stop guard's Recover reaches it directly. A hold only one path honours is not a hold.
|
||||
if g.sett != nil {
|
||||
if h, ok := g.sett.GetRestoreHold(stackName); ok {
|
||||
if h.Reason == settings.HoldReasonUpdateFailed {
|
||||
return false, "held after a failed update (" + h.At + ") — restore it from its backup to start it"
|
||||
}
|
||||
return false, "held after a failed restore whose rollback also failed (" + h.At + ") — clear the hold to start it"
|
||||
}
|
||||
}
|
||||
@@ -2236,7 +2266,11 @@ func scanDeployedAppRunStates(mgr *stacks.Manager, q *quiesce.Loop, g *backup.Ap
|
||||
// `g` covers the per-app operations — the nightly volume dump, an off-site reconstitution and a
|
||||
// .fab export. Both are nil-safe, and the union is taken here rather than inside classifyRunStates
|
||||
// so that pure function keeps its single `quiesced` parameter and its existing tests.
|
||||
return classifyRunStates(mgr.GetStacks(), unionSuppressed(q.SuppressedStacks(), g.SuppressedStacks()), q.FailedRestarts(), time.Now())
|
||||
// Slice 4: a THIRD mechanism moves an app on purpose — the guarded update recreates it and waits
|
||||
// for health, and it ends in healthy or HELD. Counting it as dead mid-update would be R-330's false
|
||||
// alarm one mechanism over.
|
||||
suppressed := unionSuppressed(unionSuppressed(q.SuppressedStacks(), g.SuppressedStacks()), mgr.UpdatingStacks())
|
||||
return classifyRunStates(mgr.GetStacks(), suppressed, q.FailedRestarts(), time.Now())
|
||||
}
|
||||
|
||||
// unionSuppressed merges the suppression sets of the two mechanisms that stop apps on purpose.
|
||||
@@ -3313,3 +3347,61 @@ func integrityOKMsg(res backup.IntegrityResult) string {
|
||||
}
|
||||
return msg + ")"
|
||||
}
|
||||
|
||||
// updateGuardsAdapter implements stacks.UpdateGuards over the backup manager (slice 4). The stacks
|
||||
// package cannot import backup, so this is the one place the two meet. Nil-safe on b: a box with
|
||||
// backup disabled has no restore point, and the update is refused for that true reason.
|
||||
type updateGuardsAdapter struct {
|
||||
b *backup.Manager
|
||||
q *quiesce.Loop
|
||||
}
|
||||
|
||||
func (a *updateGuardsAdapter) HoldFor(name string) (bool, string) {
|
||||
if a.b == nil {
|
||||
return false, ""
|
||||
}
|
||||
return a.b.RestoreHoldFor(name)
|
||||
}
|
||||
|
||||
func (a *updateGuardsAdapter) Busy(name string) (bool, string) {
|
||||
if a.q.SuppressedStacks()[name] {
|
||||
return true, "a whole-guest backup (quiesce) is holding it"
|
||||
}
|
||||
if a.b == nil {
|
||||
return false, ""
|
||||
}
|
||||
return a.b.UpdateBusy(name)
|
||||
}
|
||||
|
||||
func (a *updateGuardsAdapter) RestorePoint(name string) (stacks.UpdateRestorePoint, error) {
|
||||
if a.b == nil {
|
||||
return stacks.UpdateRestorePoint{}, fmt.Errorf("backup is not enabled on this box")
|
||||
}
|
||||
rp, err := a.b.Tier2UnitRestorePoint(name)
|
||||
if err != nil {
|
||||
return stacks.UpdateRestorePoint{}, err
|
||||
}
|
||||
at, proven := rp.ProvenCopyTime()
|
||||
return stacks.UpdateRestorePoint{Restorable: rp.Restorable, Proven: proven, ProvenAt: at}, nil
|
||||
}
|
||||
|
||||
func (a *updateGuardsAdapter) BackupNow(ctx context.Context, name string) error {
|
||||
if a.b == nil {
|
||||
return fmt.Errorf("backup is not enabled on this box")
|
||||
}
|
||||
return a.b.RunAppBackupNow(ctx, name)
|
||||
}
|
||||
|
||||
func (a *updateGuardsAdapter) SafetyDump(ctx context.Context, name string) ([]string, error) {
|
||||
if a.b == nil {
|
||||
return nil, fmt.Errorf("backup is not enabled on this box")
|
||||
}
|
||||
return a.b.WriteUpdateSafetyDump(ctx, name)
|
||||
}
|
||||
|
||||
func (a *updateGuardsAdapter) HoldAfterFailedUpdate(name string, at, provenCopyAt time.Time) error {
|
||||
if a.b == nil {
|
||||
return fmt.Errorf("backup is not enabled on this box — the hold cannot be recorded")
|
||||
}
|
||||
return a.b.HoldAfterFailedUpdate(name, at, provenCopyAt)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// Update arc slice 4 — the seams are WIRED, in the order that makes them true. A seam built and never
|
||||
// wired has shipped in this project seven times; each assertion here fails if a line moves or goes.
|
||||
|
||||
func slice4CallLines(t *testing.T) (map[string][]int, *ast.File, *token.FileSet) {
|
||||
t.Helper()
|
||||
fset := token.NewFileSet()
|
||||
f, err := parser.ParseFile(fset, "main.go", nil, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lines := map[string][]int{}
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
if call, ok := n.(*ast.CallExpr); ok {
|
||||
if sel, ok := call.Fun.(*ast.SelectorExpr); ok {
|
||||
lines[sel.Sel.Name] = append(lines[sel.Sel.Name], fset.Position(call.Pos()).Line)
|
||||
} else if id, ok := call.Fun.(*ast.Ident); ok {
|
||||
lines[id.Name] = append(lines[id.Name], fset.Position(call.Pos()).Line)
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
return lines, f, fset
|
||||
}
|
||||
|
||||
func TestSlice4_UpdateGuardsAreWiredAtStartup(t *testing.T) {
|
||||
lines, _, _ := slice4CallLines(t)
|
||||
if len(lines["SetUpdateGuards"]) == 0 {
|
||||
t.Fatal("SetUpdateGuards is never called — every update would be refused as unwired")
|
||||
}
|
||||
if len(lines["RecoverUpdates"]) == 0 || len(lines["runBootReconcile"]) == 0 {
|
||||
t.Fatal("RecoverUpdates or the boot reconciler is missing from main.go")
|
||||
}
|
||||
if lines["RecoverUpdates"][0] > lines["runBootReconcile"][0] {
|
||||
t.Errorf("RecoverUpdates (line %d) must run BEFORE the boot reconciler (line %d), or the sweep can start a half-updated app",
|
||||
lines["RecoverUpdates"][0], lines["runBootReconcile"][0])
|
||||
}
|
||||
if len(lines["ResumeInterruptedUpdates"]) == 0 || lines["ResumeInterruptedUpdates"][0] < lines["SetUpdateGuards"][0] {
|
||||
t.Error("ResumeInterruptedUpdates must run AFTER SetUpdateGuards — a resumed update that fails must be able to HOLD")
|
||||
}
|
||||
}
|
||||
|
||||
func funcBodySource(t *testing.T, f *ast.File, fset *token.FileSet, recv, name string) string {
|
||||
t.Helper()
|
||||
for _, d := range f.Decls {
|
||||
fn, ok := d.(*ast.FuncDecl)
|
||||
if !ok || fn.Name.Name != name || fn.Body == nil {
|
||||
continue
|
||||
}
|
||||
if recv != "" {
|
||||
if fn.Recv == nil {
|
||||
continue
|
||||
}
|
||||
id, ok := fn.Recv.List[0].Type.(*ast.Ident)
|
||||
if !ok || id.Name != recv {
|
||||
continue
|
||||
}
|
||||
}
|
||||
var names []string
|
||||
ast.Inspect(fn.Body, func(n ast.Node) bool {
|
||||
if sel, ok := n.(*ast.SelectorExpr); ok {
|
||||
names = append(names, sel.Sel.Name)
|
||||
}
|
||||
return true
|
||||
})
|
||||
return strings.Join(names, " ")
|
||||
}
|
||||
t.Fatalf("func %s.%s not found", recv, name)
|
||||
return ""
|
||||
}
|
||||
|
||||
func TestSlice4_BootSweepAndDeadAppAlarmKnowAboutUpdates(t *testing.T) {
|
||||
_, f, fset := slice4CallLines(t)
|
||||
if !strings.Contains(funcBodySource(t, f, fset, "bootDriveGate", "MayStart"), "IsUpdating") {
|
||||
t.Error("the boot sweep must refuse an app a guarded update is moving")
|
||||
}
|
||||
if !strings.Contains(funcBodySource(t, f, fset, "", "scanDeployedAppRunStates"), "UpdatingStacks") {
|
||||
t.Error("the dead-app alarm must not count an app the update itself is recreating")
|
||||
}
|
||||
}
|
||||
|
||||
// The shared start gate names WHICH hold it is honouring — an operator reading the boot log must not
|
||||
// be told a restore failed when an update did.
|
||||
func TestSlice4_DriveStartGate_NamesAnUpdateHold(t *testing.T) {
|
||||
sett := holdTestSettings(t)
|
||||
if err := sett.SetRestoreHold(settings.RestoreHold{Stack: "bookstack", At: "2026-09-13T08:00:00Z", Reason: settings.HoldReasonUpdateFailed}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ok, why := driveStartGate{sett: sett}.MayStart("bookstack")
|
||||
if ok || !strings.Contains(why, "held after a failed update") {
|
||||
t.Errorf("ok=%v why=%q", ok, why)
|
||||
}
|
||||
}
|
||||
@@ -579,7 +579,11 @@ func (r *Router) actionStack(w http.ResponseWriter, action, name string) {
|
||||
// R-379/R-380: an app held after a failed restore + failed rollback must not start from the
|
||||
// customer's button either. Checked BEFORE the drive gate because it applies to driveless apps,
|
||||
// which is the class the hold exists for.
|
||||
if action == "start" || action == "restart" {
|
||||
//
|
||||
// R-439 (slice 4): `update` is in this list. It was not until v0.237.0, so a held app could be
|
||||
// updated — the one action most likely to make a held app's data worse. Pinned by
|
||||
// TestR439_UpdateOfAHeldAppIsRefused.
|
||||
if action == "start" || action == "restart" || action == "update" {
|
||||
if held, why := r.restoreHoldFor(name); held {
|
||||
writeJSON(w, http.StatusConflict, apiResponse{OK: false, Error: why})
|
||||
return
|
||||
@@ -618,6 +622,20 @@ func (r *Router) actionStack(w http.ResponseWriter, action, name string) {
|
||||
}
|
||||
}
|
||||
|
||||
// Slice 4: every cheap refusal of an update — busy, already updating, deploying, memory, disk, and
|
||||
// "no backup to return to" — BEFORE the intent below is recorded, so an update that was never going
|
||||
// to happen records nothing (§8.2). Each is a 409 with the Hungarian sentence.
|
||||
if action == "update" {
|
||||
if ref := r.stackMgr.UpdatePreflight(name); ref != nil {
|
||||
status := http.StatusConflict
|
||||
if ref.Reason == "not_found" {
|
||||
status = http.StatusNotFound
|
||||
}
|
||||
writeJSON(w, status, apiResponse{OK: false, Error: ref.Message})
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// R-166: THE CUSTOMER-INTENT POINT. This switch is where a human's decision about whether their
|
||||
// app should be running enters the system, and until v0.189.0 that decision was recorded nowhere
|
||||
// — so the box had to infer it from container counts, and inferred wrong for a power cut and for
|
||||
@@ -647,12 +665,18 @@ func (r *Router) actionStack(w http.ResponseWriter, action, name string) {
|
||||
case "restart":
|
||||
err = r.stackMgr.RestartStack(name)
|
||||
case "update":
|
||||
err = r.stackMgr.UpdateStack(name)
|
||||
// Slice 4: the GUARDED update. It returns as soon as the job has started; the result is only
|
||||
// ever known from GET /api/stacks/{name} (updating / update_phase / update_error).
|
||||
err = r.stackMgr.StartGuardedUpdate(name)
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
r.logger.Printf("[ERROR] [api] %s failed for %s: %v", action, name, err)
|
||||
status := http.StatusInternalServerError
|
||||
var ref *stacks.UpdateRefusal
|
||||
if errors.As(err, &ref) {
|
||||
status = http.StatusConflict
|
||||
}
|
||||
if strings.Contains(err.Error(), "protected") {
|
||||
status = http.StatusForbidden
|
||||
}
|
||||
@@ -663,6 +687,15 @@ func (r *Router) actionStack(w http.ResponseWriter, action, name string) {
|
||||
return
|
||||
}
|
||||
|
||||
// R-443 (slice 4): an update is NEVER reported completed here. The spike measured this line saying
|
||||
// "update completed" over an app that was already crash-looping. It answers 202 — accepted, not
|
||||
// finished — and "completed" exists only as update_phase=done on GET /api/stacks/{name}, which is
|
||||
// written after the app's health is known. Pinned by TestR443_UpdateIsNeverReportedCompleteSynchronously.
|
||||
if action == "update" {
|
||||
writeJSON(w, http.StatusAccepted, apiResponse{OK: true, Message: "Frissítés elindult – az állapot a kártyán követhető",
|
||||
Data: map[string]interface{}{"accepted": true, "completed": false}})
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, apiResponse{OK: true, Message: "Stack " + name + " " + action + " completed"})
|
||||
|
||||
// Trigger integration lifecycle hooks after successful action
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
|
||||
)
|
||||
|
||||
// Update arc slice 4 through the PRODUCTION handler: actionStack → the real stacks.Manager
|
||||
// (NewManager + ScanStacks) and the real backup.Manager over real settings. No docker is reached:
|
||||
// every path here refuses, or fails at the pin (the app's catalog template is absent on purpose).
|
||||
|
||||
type apiFakeGuards struct {
|
||||
b *backup.Manager
|
||||
rp stacks.UpdateRestorePoint
|
||||
// blindToHolds makes the manager-side preflight NOT see holds, so a test can prove the ROUTER's
|
||||
// own hold check refuses — the two layers are each pinned separately (the preflight's by
|
||||
// TestSlice4_D_CheapRefusals/held). Without it, removing either layer passes inertly, because the
|
||||
// other refuses with the same sentence (observed on the first run of red-proof 4, 2026-09-13).
|
||||
blindToHolds bool
|
||||
}
|
||||
|
||||
func (g *apiFakeGuards) HoldFor(n string) (bool, string) {
|
||||
if g.blindToHolds {
|
||||
return false, ""
|
||||
}
|
||||
return g.b.RestoreHoldFor(n)
|
||||
}
|
||||
func (g *apiFakeGuards) Busy(string) (bool, string) { return false, "" }
|
||||
func (g *apiFakeGuards) RestorePoint(string) (stacks.UpdateRestorePoint, error) {
|
||||
return g.rp, nil
|
||||
}
|
||||
func (g *apiFakeGuards) BackupNow(context.Context, string) error { return nil }
|
||||
func (g *apiFakeGuards) SafetyDump(context.Context, string) ([]string, error) {
|
||||
return nil, nil
|
||||
}
|
||||
func (g *apiFakeGuards) HoldAfterFailedUpdate(string, time.Time, time.Time) error { return nil }
|
||||
|
||||
const slice4AppYAML = "deployed: true\nenv: {}\npinned_images:\n app: nginx:1.27\n"
|
||||
|
||||
func newSlice4Router(t *testing.T) (*Router, *settings.Settings, *apiFakeGuards, string) {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
dir := filepath.Join(root, "stacks", "app")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "docker-compose.yml"), []byte("services:\n app:\n image: nginx:1.27\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "app.yaml"), []byte(slice4AppYAML), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cfg := &config.Config{}
|
||||
cfg.Paths.StacksDir = filepath.Join(root, "stacks")
|
||||
cfg.Paths.DataDir = filepath.Join(root, "data")
|
||||
cfg.Paths.SystemDataPath = filepath.Join(root, "sys")
|
||||
cfg.Stacks.ComposeCommand = "docker compose"
|
||||
lg := log.New(io.Discard, "", 0)
|
||||
m, err := stacks.NewManager(cfg, lg)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := m.ScanStacks(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sett, err := settings.Load(filepath.Join(root, "settings.json"), lg)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b := backup.NewManager(cfg, sett, lg)
|
||||
g := &apiFakeGuards{b: b, rp: stacks.UpdateRestorePoint{Restorable: true, Proven: true, ProvenAt: time.Now().Add(-time.Hour)}}
|
||||
m.SetUpdateGuards(g)
|
||||
return &Router{cfg: cfg, stackMgr: m, backupMgr: b, logger: lg}, sett, g, dir
|
||||
}
|
||||
|
||||
func postUpdate(t *testing.T, r *Router) (int, apiResponse) {
|
||||
t.Helper()
|
||||
w := httptest.NewRecorder()
|
||||
r.actionStack(w, "update", "app")
|
||||
var resp apiResponse
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("non-JSON body %q: %v", w.Body.String(), err)
|
||||
}
|
||||
return w.Code, resp
|
||||
}
|
||||
|
||||
// TestR439_UpdateOfAHeldAppIsRefused — R-439 closed.
|
||||
//
|
||||
// COMPANION RED-PROOF 4 (REPORT.md): remove `|| action == "update"` from actionStack's hold check. The
|
||||
// preflight then refuses on its own grounds with a DIFFERENT sentence, and this test fails on the
|
||||
// message — which is what proves the router line is the one doing it.
|
||||
func TestR439_UpdateOfAHeldAppIsRefused(t *testing.T) {
|
||||
r, sett, g, dir := newSlice4Router(t)
|
||||
g.rp = stacks.UpdateRestorePoint{} // the preflight's own refusal would say "no backup" — not the hold
|
||||
g.blindToHolds = true // only the router's line can produce the hold's sentence
|
||||
if err := sett.SetRestoreHold(settings.RestoreHold{Stack: "app", At: "2026-09-13T08:00:00Z", Reason: settings.HoldReasonUpdateFailed, CopyDate: "2026-09-13T01:30:00Z"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
before, _ := os.ReadFile(filepath.Join(dir, "app.yaml"))
|
||||
code, resp := postUpdate(t, r)
|
||||
_, holdText := r.backupMgr.RestoreHoldFor("app")
|
||||
if code != http.StatusConflict || resp.OK || resp.Error != holdText {
|
||||
t.Fatalf("a HELD app's update must be refused with the hold's own sentence: code=%d ok=%v error=%q", code, resp.OK, resp.Error)
|
||||
}
|
||||
after, _ := os.ReadFile(filepath.Join(dir, "app.yaml"))
|
||||
if string(before) != string(after) {
|
||||
t.Error("a refused update must record no intent — app.yaml changed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlice4_Router_NoBackupIs409AndRecordsNothing(t *testing.T) {
|
||||
r, _, g, dir := newSlice4Router(t)
|
||||
g.rp = stacks.UpdateRestorePoint{Restorable: false}
|
||||
before, _ := os.ReadFile(filepath.Join(dir, "app.yaml"))
|
||||
code, resp := postUpdate(t, r)
|
||||
if code != http.StatusConflict || resp.Error != fmt.Sprintf(stacks.MsgUpdateNoBackupFmt, "app") {
|
||||
t.Fatalf("code=%d error=%q", code, resp.Error)
|
||||
}
|
||||
if after, _ := os.ReadFile(filepath.Join(dir, "app.yaml")); string(before) != string(after) {
|
||||
t.Error("the preflight refusal must come BEFORE the intent write")
|
||||
}
|
||||
}
|
||||
|
||||
// TestR443_UpdateIsNeverReportedCompleteSynchronously — R-443 closed. The handler answers 202 with
|
||||
// completed:false; the job then runs (and here fails at the pin, the catalog being absent), and the
|
||||
// outcome exists ONLY on GET /api/stacks/{name}.
|
||||
func TestR443_UpdateIsNeverReportedCompleteSynchronously(t *testing.T) {
|
||||
r, _, _, _ := newSlice4Router(t)
|
||||
code, resp := postUpdate(t, r)
|
||||
if code != http.StatusAccepted {
|
||||
t.Fatalf("an accepted update must answer 202, got %d (%+v)", code, resp)
|
||||
}
|
||||
data, _ := resp.Data.(map[string]interface{})
|
||||
if data["completed"] != false || data["accepted"] != true {
|
||||
t.Errorf("the body must say accepted and NOT completed, got %v", resp.Data)
|
||||
}
|
||||
if resp.Message == "Stack app update completed" {
|
||||
t.Error("the synchronous response claimed completion — R-443")
|
||||
}
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
if st, ok := r.stackMgr.GetStack("app"); ok && !st.Updating {
|
||||
if st.UpdatePhase != stacks.UpdatePhaseFailed || st.UpdateError != stacks.MsgUpdatePinFailed {
|
||||
t.Errorf("the job's truth must be on the stack: phase=%q err=%q", st.UpdatePhase, st.UpdateError)
|
||||
}
|
||||
return
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
t.Fatal("the job never finished")
|
||||
}
|
||||
@@ -680,6 +680,15 @@ func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) {
|
||||
if m.cfg != nil && m.cfg.IsProtectedStack(stack.Name) {
|
||||
continue
|
||||
}
|
||||
// Slice 4 / R-379: a HELD app is deliberately stopped, and DumpAppVolumesSafe ends in
|
||||
// StartStack — so without this line the nightly backup restarted every held app, every night.
|
||||
// "A hold that only one path honours is not a hold." Checked BEFORE the volume check so a held
|
||||
// app is never stopped or started by this leg at all.
|
||||
if m.isHeld(stack.Name) {
|
||||
m.logger.Printf("[WARN] [backup] Skipping volume dump for %s — the app is HELD stopped (its restore point is preserved)", stack.Name)
|
||||
summary = append(summary, fmt.Sprintf("SKIP %s volumes (held)", stack.Name))
|
||||
continue
|
||||
}
|
||||
// Volume check FIRST — a volume-less stack must not be stopped at all (see gate-order note).
|
||||
if len(m.stackProvider.GetDockerVolumes(stack.Name)) == 0 {
|
||||
if m.isDebug() {
|
||||
|
||||
@@ -337,6 +337,15 @@ func (m *Manager) RestoreHoldFor(stack string) (bool, string) {
|
||||
if !ok {
|
||||
return false, ""
|
||||
}
|
||||
// Slice 4: one storage, two reasons. An update hold names the copy it can be restored from; a
|
||||
// restore hold names nothing, because the restore it refers to already consumed the copy.
|
||||
if h.Reason == settings.HoldReasonUpdateFailed {
|
||||
copyDate := "legutóbbi"
|
||||
if h.CopyDate != "" {
|
||||
copyDate = fmtHoldTime(h.CopyDate)
|
||||
}
|
||||
return true, fmt.Sprintf(UpdateHoldFmt, stack, fmtHoldTime(h.At), copyDate)
|
||||
}
|
||||
when := h.At
|
||||
if t, err := time.Parse(time.RFC3339, h.At); err == nil {
|
||||
when = t.Format("2006-01-02 15:04")
|
||||
|
||||
@@ -385,6 +385,14 @@ func (m *Manager) captureAllRecoveryUnits() {
|
||||
if m.settings != nil && (m.settings.IsDisconnected(drivePath) || m.settings.IsDecommissioned(drivePath)) {
|
||||
continue // drive not writable — skip, the existing unit stays as-is
|
||||
}
|
||||
// Slice 4: a HELD app's unit is its RESTORE POINT, and the hold text names that copy's date.
|
||||
// Re-capturing it would write the definition the app is held ON (after a failed update: the new
|
||||
// version that would not start) into the unit, and the next Tier-2 run would mirror it over the
|
||||
// copy the customer was told to restore from. The unit stays exactly as it was until the hold is
|
||||
// lifted.
|
||||
if m.isHeld(stack.Name) {
|
||||
continue
|
||||
}
|
||||
m.noteAttempted(stack.Name)
|
||||
// The reserve, checked BEFORE anything is written. Per app, and the loop continues.
|
||||
if !m.admitApp(stack.Name) {
|
||||
|
||||
@@ -390,5 +390,8 @@ func (m *Manager) RestoreFromRecoveryUnitAt(stackName, unitDir string) (UnitRest
|
||||
}
|
||||
m.logger.Printf("[INFO] [backup] Restore-from-unit completed: %s — %d volume(s) of %d listed, %d database(s) of %d listed",
|
||||
stackName, res.VolumesReplayed, res.ManifestVolumes, res.DBsReplayed, res.ManifestDBs)
|
||||
// Slice 4: the app is back on its unit's definition and data and was started — the route back an
|
||||
// update hold names. Lift that hold (and only that kind; see clearUpdateHoldAfterRestore).
|
||||
m.clearUpdateHoldAfterRestore(stackName)
|
||||
return res, nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,168 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// Update arc slice 4 — the backup side of the guarded update.
|
||||
|
||||
// The measured case (demo-hp 2026-09-13, bookstack): the mirror's manifest said 2026-09-12T02:15:29Z
|
||||
// while its database dump was written 2026-09-13T00:30Z and the Tier-2 run succeeded at 01:30Z. The
|
||||
// update's age must be the proven COPY time; the manifest date would call a fresh copy stale forever.
|
||||
func TestSlice4_ProvenCopyTime_IsTheLastSuccessNotTheManifestDate(t *testing.T) {
|
||||
rp := restorePointFromCoverage(Tier2Coverage{
|
||||
UnitRestorable: true, UnitPackageDate: "2026-09-12T02:15:29Z",
|
||||
CopyLastRun: "2026-09-13T01:30:00Z", CopyLastSuccess: "2026-09-13T01:30:00Z",
|
||||
})
|
||||
at, ok := rp.ProvenCopyTime()
|
||||
if !ok || !at.Equal(time.Date(2026, 9, 13, 1, 30, 0, 0, time.UTC)) {
|
||||
t.Fatalf("proven copy time = %v ok=%v, want the last successful copy", at, ok)
|
||||
}
|
||||
// The page still names the PACKAGE date (R-403) — the extraction changes nothing it shows.
|
||||
if rp.CopyDate != "2026-09-12T02:15:29Z" || !rp.CopyDateProven || rp.PackagePreserved {
|
||||
t.Errorf("restore point = %+v", rp)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlice4_ProvenCopyTime_PreservedPackageUsesThePackageDate(t *testing.T) {
|
||||
rp := restorePointFromCoverage(Tier2Coverage{
|
||||
UnitRestorable: true, UnitPackageDate: "2026-09-01T02:00:00Z", UnitLegPreserved: true,
|
||||
CopyLastSuccess: "2026-09-13T01:30:00Z",
|
||||
})
|
||||
if at, ok := rp.ProvenCopyTime(); !ok || !at.Equal(time.Date(2026, 9, 1, 2, 0, 0, 0, time.UTC)) {
|
||||
t.Errorf("a PRESERVED package is as old as the package, got %v ok=%v", at, ok)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlice4_ProvenCopyTime_NoProvenOrNoUnitIsNoRestorePoint(t *testing.T) {
|
||||
for _, cov := range []Tier2Coverage{
|
||||
{UnitRestorable: true, CopyLastRun: "2026-09-13T01:30:00Z"}, // attempt, never a success (R-101)
|
||||
{UnitRestorable: false, CopyLastSuccess: "2026-09-13T01:30:00Z"}, // a copy with no openable unit
|
||||
{UnitRestorable: true, CopyLastSuccess: "not-a-date"}, // unparseable is unknown, never "now"
|
||||
} {
|
||||
if _, ok := restorePointFromCoverage(cov).ProvenCopyTime(); ok {
|
||||
t.Errorf("%+v must not yield a proven copy time", cov)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func slice4Settings(t *testing.T) *settings.Settings {
|
||||
t.Helper()
|
||||
s, err := settings.Load(filepath.Join(t.TempDir(), "settings.json"), log.New(io.Discard, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func TestSlice4_UpdateHoldTextNamesTheTimeAndTheCopy(t *testing.T) {
|
||||
sett := slice4Settings(t)
|
||||
m := &Manager{logger: log.New(io.Discard, "", 0), settings: sett}
|
||||
at := time.Date(2026, 9, 13, 8, 0, 0, 0, time.UTC)
|
||||
copyAt := time.Date(2026, 9, 13, 1, 30, 0, 0, time.UTC)
|
||||
if err := m.HoldAfterFailedUpdate("bookstack", at, copyAt); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h, ok := sett.GetRestoreHold("bookstack")
|
||||
if !ok || h.Reason != settings.HoldReasonUpdateFailed || h.CopyDate != "2026-09-13T01:30:00Z" {
|
||||
t.Fatalf("hold = %+v ok=%v", h, ok)
|
||||
}
|
||||
held, why := m.RestoreHoldFor("bookstack")
|
||||
// Budapest is UTC+2 in September: 08:00Z → 10:00, 01:30Z → 03:30.
|
||||
want := fmt.Sprintf(UpdateHoldFmt, "bookstack", "2026-09-13 10:00", "2026-09-13 03:30")
|
||||
if !held || why != want {
|
||||
t.Errorf("hold text =\n%q\nwant\n%q", why, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlice4_RestoreHoldTextIsUnchanged(t *testing.T) {
|
||||
sett := slice4Settings(t)
|
||||
m := &Manager{logger: log.New(io.Discard, "", 0), settings: sett}
|
||||
if err := sett.SetRestoreHold(settings.RestoreHold{Stack: "docmost", At: "2026-08-22T14:00:00Z"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, why := m.RestoreHoldFor("docmost")
|
||||
if !strings.Contains(why, "visszaállítása") || !strings.Contains(why, "Vedd fel velünk a kapcsolatot") || strings.Contains(why, "frissítése") {
|
||||
t.Errorf("an R-379 restore hold must keep its own sentence, got %q", why)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlice4_ASuccessfulRestoreClearsOnlyAnUpdateHold(t *testing.T) {
|
||||
sett := slice4Settings(t)
|
||||
m := &Manager{logger: log.New(io.Discard, "", 0), settings: sett}
|
||||
_ = m.HoldAfterFailedUpdate("upd", time.Now(), time.Now())
|
||||
_ = sett.SetRestoreHold(settings.RestoreHold{Stack: "rst", At: "2026-08-22T14:00:00Z"})
|
||||
m.clearUpdateHoldAfterRestore("upd")
|
||||
m.clearUpdateHoldAfterRestore("rst")
|
||||
if _, ok := sett.GetRestoreHold("upd"); ok {
|
||||
t.Error("a restore is the route back from a failed update — its hold must be lifted")
|
||||
}
|
||||
if _, ok := sett.GetRestoreHold("rst"); !ok {
|
||||
t.Error("an R-379 restore hold stays operator-cleared")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlice4_UpdateBusy(t *testing.T) {
|
||||
m := &Manager{logger: log.New(io.Discard, "", 0)}
|
||||
if busy, _ := m.UpdateBusy("app"); busy {
|
||||
t.Fatal("an idle manager is not busy")
|
||||
}
|
||||
if err := m.acquireRunning(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if busy, _ := m.UpdateBusy("app"); !busy {
|
||||
t.Error("a running backup/restore must make an update wait")
|
||||
}
|
||||
m.releaseRunning()
|
||||
m.BeginRestoreOp("tier2-unit-restore", "other")
|
||||
if busy, _ := m.UpdateBusy("app"); !busy {
|
||||
t.Error("a restore op in flight must make an update wait")
|
||||
}
|
||||
}
|
||||
|
||||
// "A hold that only one path honours is not a hold." The nightly legs are unattended start paths
|
||||
// (DumpAppVolumesSafe ends in StartStack) and writers of the restore point the hold text names.
|
||||
//
|
||||
// COMPANION RED-PROOF (REPORT.md): delete the isHeld skip from runVolumeDumps — the held app is then
|
||||
// stopped (and restarted) by the nightly backup, and this test fails.
|
||||
func TestSlice4_NightlyLegsLeaveAHeldAppAlone(t *testing.T) {
|
||||
h := newAdmissionHarness(t, "held", "free")
|
||||
h.m.settings = slice4Settings(t)
|
||||
if err := h.m.HoldAfterFailedUpdate("held", time.Now(), time.Now()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.m.runVolumeDumps()
|
||||
for _, n := range append(append([]string{}, h.volDumped...), h.prov.stopped...) {
|
||||
if n == "held" {
|
||||
t.Fatalf("the nightly volume dump touched a HELD app (dumped=%v stopped=%v)", h.volDumped, h.prov.stopped)
|
||||
}
|
||||
}
|
||||
if len(h.volDumped) != 1 || h.volDumped[0] != "free" {
|
||||
t.Errorf("positive control: the unheld app must still be dumped, got %v", h.volDumped)
|
||||
}
|
||||
h.m.captureAllRecoveryUnits()
|
||||
for _, n := range h.prov.infoHits {
|
||||
if n == "held" {
|
||||
t.Error("the capture must not rewrite a HELD app's restore point")
|
||||
}
|
||||
}
|
||||
var mirrored []string
|
||||
h.m.perAppTier2 = func(name string) error { mirrored = append(mirrored, name); return nil }
|
||||
h.m.RunAllTier2()
|
||||
for _, n := range mirrored {
|
||||
if n == "held" {
|
||||
t.Error("Tier 2 must not mirror over a HELD app's copy")
|
||||
}
|
||||
}
|
||||
if len(mirrored) != 1 {
|
||||
t.Errorf("positive control: the unheld app must still be mirrored, got %v", mirrored)
|
||||
}
|
||||
}
|
||||
@@ -457,6 +457,12 @@ func (m *Manager) RunAllTier2() {
|
||||
m.settings.IsDecommissioned(m.GetAppDrivePath(stack.Name))) {
|
||||
continue
|
||||
}
|
||||
// Slice 4: never mirror over a HELD app's copy — it is the restore point the hold text names.
|
||||
// See the matching skip in captureAllRecoveryUnits.
|
||||
if m.isHeld(stack.Name) {
|
||||
m.logger.Printf("[WARN] [backup] Tier 2 skipped for %s — the app is HELD; its copy is the restore point and is preserved", stack.Name)
|
||||
continue
|
||||
}
|
||||
runOne := m.perAppTier2
|
||||
if runOne == nil {
|
||||
runOne = m.RunTier2
|
||||
|
||||
@@ -0,0 +1,325 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// ── The backup side of the guarded update (update arc slice 4, controller v0.237.0) ─────────────
|
||||
//
|
||||
// 09-update-architecture.md §3 decision 1 (operator ruling 2026-09-02): the safety copy for an update
|
||||
// is a VERIFIED RECENT BACKUP as a PRECONDITION — not a new copy mechanism invented for the update
|
||||
// path. So everything in this file composes machinery that already exists and is proven live:
|
||||
// the Tier-2 unit restore's own predicate (R-102/R-103), the nightly legs (DB dump, volume dump,
|
||||
// unit capture, Tier-2 mirror), the pre-restore safety dump (R-361) and the R-379 hold.
|
||||
//
|
||||
// The stacks package cannot import this one, so the update job reaches all of it through the
|
||||
// stacks.UpdateGuards interface, implemented by an adapter in cmd/controller/main.go.
|
||||
|
||||
// Tier2RestorePoint is the answer to "could this app be restored from its Tier-2 copy, and from
|
||||
// when?" — the predicate the destructive „Teljes visszaállítás" action is gated on.
|
||||
//
|
||||
// EXTRACTED, NOT DUPLICATED (slice 4). Until v0.237.0 this computation lived inline in the backups
|
||||
// page handler (buildAppBackupRows). The update path needs exactly the same question answered, and
|
||||
// a second copy of a predicate is how this project's two copies of `namespaceRoot` came to differ
|
||||
// (R-203). So there is one function, and the page and the update both call it.
|
||||
type Tier2RestorePoint struct {
|
||||
// Restorable — the copy holds an OPENABLE recovery unit (Tier2Coverage.CanRestoreUnit).
|
||||
Restorable bool
|
||||
// CopyDate — the date the unit restore NAMES: the package's own manifest date, falling back to the
|
||||
// copy date (Tier2Coverage.UnitRestoreDate, R-403). RFC3339 as recorded, "" when unknown.
|
||||
CopyDate string
|
||||
// CopyDateProven — a copy actually SUCCEEDED (LastSuccess is set), never merely an attempt (R-101).
|
||||
CopyDateProven bool
|
||||
// PackagePreserved — the newest run PRESERVED an older package instead of refreshing it (R-403).
|
||||
PackagePreserved bool
|
||||
// CopyLastSuccess — the RFC3339 time of the last Tier-2 copy that succeeded.
|
||||
CopyLastSuccess string
|
||||
}
|
||||
|
||||
// restorePointFromCoverage is the pure half of the predicate.
|
||||
func restorePointFromCoverage(cov Tier2Coverage) Tier2RestorePoint {
|
||||
pkgDate, preserved := cov.UnitRestoreDate()
|
||||
return Tier2RestorePoint{
|
||||
Restorable: cov.CanRestoreUnit(),
|
||||
CopyDate: pkgDate,
|
||||
CopyDateProven: cov.CopyLastSuccess != "",
|
||||
PackagePreserved: preserved,
|
||||
CopyLastSuccess: cov.CopyLastSuccess,
|
||||
}
|
||||
}
|
||||
|
||||
// Tier2UnitRestorePoint resolves the app's recorded Tier-2 copy and returns the restore point. The
|
||||
// error is the same refusal Tier2RestoreCoverage raises (no copy, drive gone, pre-v2 layout).
|
||||
func (m *Manager) Tier2UnitRestorePoint(stackName string) (Tier2RestorePoint, error) {
|
||||
cov, err := m.Tier2RestoreCoverage(stackName)
|
||||
if err != nil {
|
||||
return Tier2RestorePoint{}, err
|
||||
}
|
||||
return restorePointFromCoverage(cov), nil
|
||||
}
|
||||
|
||||
// ProvenCopyTime returns WHEN the data this copy would restore was last proven copied, and false when
|
||||
// there is no proven, restorable copy at all.
|
||||
//
|
||||
// WHY NOT CopyDate, measured rather than assumed. CopyDate is the unit MANIFEST's created_at, and a
|
||||
// capture rewrites the manifest only when the app's DEFINITION changes (compose, app.yaml, controller
|
||||
// version) — a nightly DB dump keeps the same file name, so it does not move it. Measured on demo-hp
|
||||
// 2026-09-13: bookstack's Tier-2 mirror held `bookstack-mariadb.sql` written 2026-09-13T00:30Z while
|
||||
// its manifest still read 2026-09-12T02:15:29Z. Judging "recent" by that date would call a fresh copy
|
||||
// stale — and, worse, a "back up first" run would not move it either on a quiet app, so the update
|
||||
// would be refused forever.
|
||||
//
|
||||
// So the age is the last SUCCESSFUL copy (LastSuccess), which the Tier-2 run records only when it
|
||||
// actually mirrored the unit — EXCEPT when the run preserved an older package (R-403), in which case
|
||||
// the package date is the honest one, because that is what the copy really holds.
|
||||
func (p Tier2RestorePoint) ProvenCopyTime() (time.Time, bool) {
|
||||
if !p.Restorable || !p.CopyDateProven {
|
||||
return time.Time{}, false
|
||||
}
|
||||
src := p.CopyLastSuccess
|
||||
if p.PackagePreserved {
|
||||
src = p.CopyDate
|
||||
}
|
||||
t, err := time.Parse(time.RFC3339, src)
|
||||
if err != nil {
|
||||
return time.Time{}, false
|
||||
}
|
||||
return t, true
|
||||
}
|
||||
|
||||
// UpdateBusy reports whether something else is ALREADY touching this app's data, which refuses an
|
||||
// update before anything moves (slice 4 Scenario D). The reason is operator-English; the customer
|
||||
// sentence is chosen by the caller.
|
||||
//
|
||||
// IsRunning is box-wide, deliberately: the backup/restore single-flight is box-wide, and an update's
|
||||
// "back up first" leg needs that same flag — an update started beside a running backup would either
|
||||
// wait on it invisibly or fail half-way.
|
||||
func (m *Manager) UpdateBusy(stackName string) (bool, string) {
|
||||
if m == nil {
|
||||
return false, ""
|
||||
}
|
||||
if m.IsRunning() {
|
||||
return true, "a backup or restore is running (single-flight held)"
|
||||
}
|
||||
if st := m.RestoreStatus(); st.Running {
|
||||
return true, fmt.Sprintf("restore op %q is running for %q", st.Op, st.Stack)
|
||||
}
|
||||
for _, held := range m.appStop.HeldStacks() {
|
||||
if held == stackName {
|
||||
return true, "an app-data operation (volume dump / export / reconstitute) is holding it"
|
||||
}
|
||||
}
|
||||
return false, ""
|
||||
}
|
||||
|
||||
// ErrUpdateBackupNoUnit is returned when a "back up first" run completed but the app still has no
|
||||
// openable Tier-2 unit — typically Tier 2 is switched off for the app or has no second target.
|
||||
var ErrUpdateBackupNoUnit = errors.New("a frissítés előtti mentés lefutott, de nem jött létre visszaállítható másolat")
|
||||
|
||||
// RunAppBackupNow runs THIS app's backup legs now, in the nightly order, and then its Tier-2 copy:
|
||||
// database dump(s) → volume dump (if the app has named volumes) → recovery-unit capture → Tier-2
|
||||
// mirror. It is the "back up first" of slice 4 Scenario B.
|
||||
//
|
||||
// Composed, not reinvented: every leg is the one runDBDumpsInternal and RunAllTier2 already run,
|
||||
// including the R-181 reserve (admitApp) before the first write and the R-166 app-stop marker inside
|
||||
// DumpAppVolumesSafe. What differs is only the scope — one app instead of all of them — because an
|
||||
// update must not bounce every other app on the box to back up one.
|
||||
func (m *Manager) RunAppBackupNow(ctx context.Context, stackName string) error {
|
||||
if m.stackProvider == nil {
|
||||
return fmt.Errorf("stack provider not configured")
|
||||
}
|
||||
if m.migrationActive() {
|
||||
return fmt.Errorf("adatáthelyezés folyamatban — a mentés most nem indítható")
|
||||
}
|
||||
if err := m.acquireRunning(); err != nil {
|
||||
return err
|
||||
}
|
||||
m.logger.Printf("[INFO] [backup] update pre-backup for %s: starting (DB dump → volume dump → unit capture → Tier 2)", stackName)
|
||||
start := time.Now()
|
||||
legErr := func() error {
|
||||
defer m.releaseRunning()
|
||||
defer m.beginAdmissionRun()()
|
||||
|
||||
drivePath := m.GetAppDrivePath(stackName)
|
||||
if drivePath == "" || !filepath.IsAbs(drivePath) {
|
||||
return fmt.Errorf("az alkalmazás meghajtója nem határozható meg")
|
||||
}
|
||||
if m.settings != nil && (m.settings.IsDisconnected(drivePath) || m.settings.IsDecommissioned(drivePath)) {
|
||||
return fmt.Errorf("az alkalmazás meghajtója nem elérhető (%s)", drivePath)
|
||||
}
|
||||
if !m.admitApp(stackName) {
|
||||
return fmt.Errorf("nincs elég szabad hely a mentéshez a(z) %s meghajtón", drivePath)
|
||||
}
|
||||
nsRoot := m.namespaceRoot(drivePath)
|
||||
|
||||
discover := m.discoverDBs
|
||||
if discover == nil {
|
||||
discover = func(ctx context.Context) ([]DiscoveredDB, error) {
|
||||
return DiscoverDatabases(ctx, m.logger, m.isDebug(), m.knownStackNames())
|
||||
}
|
||||
}
|
||||
dbs, err := discover(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("adatbázis-felderítés sikertelen: %w", err)
|
||||
}
|
||||
dumped := 0
|
||||
for _, db := range dbs {
|
||||
if db.StackName != stackName {
|
||||
continue
|
||||
}
|
||||
res := DumpOne(ctx, db, AppDBDumpPath(nsRoot, stackName), m.logger, m.isDebug())
|
||||
if res.Error != nil {
|
||||
return fmt.Errorf("adatbázis-mentés sikertelen (%s): %w", db.ContainerName, res.Error)
|
||||
}
|
||||
dumped++
|
||||
m.logger.Printf("[INFO] [backup] update pre-backup for %s: database dump OK (%s, %s)", stackName, db.ContainerName, humanizeBytes(res.Size))
|
||||
}
|
||||
|
||||
if len(m.stackProvider.GetDockerVolumes(stackName)) > 0 {
|
||||
dump := m.dumpVolumesSafe
|
||||
if dump == nil {
|
||||
dump = m.DumpAppVolumesSafe
|
||||
}
|
||||
if err := dump(stackName); err != nil {
|
||||
return fmt.Errorf("kötetmentés sikertelen: %w", err)
|
||||
}
|
||||
m.logger.Printf("[INFO] [backup] update pre-backup for %s: volume dump OK", stackName)
|
||||
}
|
||||
|
||||
if err := m.CaptureRecoveryUnit(stackName); err != nil {
|
||||
return fmt.Errorf("a mentési egység rögzítése sikertelen: %w", err)
|
||||
}
|
||||
m.logger.Printf("[INFO] [backup] update pre-backup for %s: recovery unit captured (%d database dump(s))", stackName, dumped)
|
||||
return nil
|
||||
}()
|
||||
if legErr != nil {
|
||||
m.logger.Printf("[ERROR] [backup] update pre-backup for %s FAILED after %s: %v", stackName, time.Since(start).Round(time.Millisecond), legErr)
|
||||
return legErr
|
||||
}
|
||||
|
||||
runOne := m.perAppTier2
|
||||
if runOne == nil {
|
||||
runOne = m.RunTier2
|
||||
}
|
||||
if err := runOne(stackName); err != nil {
|
||||
m.logger.Printf("[ERROR] [backup] update pre-backup for %s: Tier 2 copy FAILED: %v", stackName, err)
|
||||
return fmt.Errorf("a másodlagos másolat elkészítése sikertelen: %w", err)
|
||||
}
|
||||
m.logger.Printf("[INFO] [backup] update pre-backup for %s: complete in %s", stackName, time.Since(start).Round(time.Millisecond))
|
||||
return nil
|
||||
}
|
||||
|
||||
// WriteUpdateSafetyDump takes the last-minute database copy an update makes just before it moves the
|
||||
// pin: "the state the customer was in a minute ago". It is writeSafetyDump (R-361) unchanged — the
|
||||
// same `pre-restore-` undo naming, the same pruning to three, the same never-the-canonical-name rule
|
||||
// — so it is also picked up by the same exclusions (it never enters a manifest's db_dumps).
|
||||
//
|
||||
// Returns the paths written; an app with no database returns (nil, nil), which is a no-op and never a
|
||||
// failure (measured in writeSafetyDump: `len(mine) == 0` returns an empty set).
|
||||
func (m *Manager) WriteUpdateSafetyDump(ctx context.Context, stackName string) ([]string, error) {
|
||||
nsRoot := m.AppNamespaceRoot(stackName)
|
||||
if nsRoot == "" {
|
||||
return nil, fmt.Errorf("az alkalmazás mentési helye nem határozható meg")
|
||||
}
|
||||
set, err := m.writeSafetyDump(ctx, stackName, nsRoot)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var paths []string
|
||||
for _, f := range set.Files {
|
||||
paths = append(paths, f.Path)
|
||||
}
|
||||
if len(paths) == 0 {
|
||||
m.logger.Printf("[INFO] [backup] update safety dump for %s: the app has no database — nothing to copy (no-op)", stackName)
|
||||
} else {
|
||||
m.logger.Printf("[INFO] [backup] update safety dump for %s: %d file(s) %v", stackName, len(paths), paths)
|
||||
}
|
||||
return paths, nil
|
||||
}
|
||||
|
||||
// UpdateHoldFmt is the customer sentence for an app held after a failed update. Arguments: the app,
|
||||
// the time of the failure, and the PROVEN date of the copy it can be restored from. One named string so
|
||||
// a test asserts it verbatim instead of retyping Hungarian (R-364).
|
||||
const UpdateHoldFmt = "A(z) %s frissítése %s-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. " +
|
||||
"Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. " +
|
||||
"Visszaállítható a(z) %s-i biztonsági mentésből a Mentések oldalon."
|
||||
|
||||
// holdTimeZone is where the customer-facing hold sentence renders its times. The same zone the web
|
||||
// layer renders the Mentések page's copy dates in (web.getTimezone), so the date in the hold text and
|
||||
// the date on the page it points at are the same string.
|
||||
func holdTimeZone() *time.Location {
|
||||
if loc, err := time.LoadLocation("Europe/Budapest"); err == nil {
|
||||
return loc
|
||||
}
|
||||
return time.UTC
|
||||
}
|
||||
|
||||
func fmtHoldTime(rfc3339 string) string {
|
||||
t, err := time.Parse(time.RFC3339, rfc3339)
|
||||
if err != nil {
|
||||
return rfc3339
|
||||
}
|
||||
return t.In(holdTimeZone()).Format("2006-01-02 15:04")
|
||||
}
|
||||
|
||||
// HoldAfterFailedUpdate records that an app is held stopped because its new version did not come up
|
||||
// healthy. Same storage and same gate as the R-379 hold — every start path that already refuses a
|
||||
// restore hold refuses this one without being touched.
|
||||
//
|
||||
// It returns the error rather than only logging it, unlike holdAppAfterFailedRollback: the update job
|
||||
// has just stopped the app on the strength of this record, and an unrecorded hold is a stopped app
|
||||
// that the next restart button will quietly start again. The caller logs it at ERROR and keeps the
|
||||
// failure on the page.
|
||||
func (m *Manager) HoldAfterFailedUpdate(stackName string, at time.Time, copyDate time.Time) error {
|
||||
if m == nil || m.settings == nil {
|
||||
return fmt.Errorf("no settings wired — the update hold for %s cannot be persisted", stackName)
|
||||
}
|
||||
h := settings.RestoreHold{
|
||||
Stack: stackName,
|
||||
At: at.UTC().Format(time.RFC3339),
|
||||
Reason: settings.HoldReasonUpdateFailed,
|
||||
}
|
||||
if !copyDate.IsZero() {
|
||||
h.CopyDate = copyDate.UTC().Format(time.RFC3339)
|
||||
}
|
||||
if err := m.settings.SetRestoreHold(h); err != nil {
|
||||
return fmt.Errorf("persisting the update hold for %s: %w", stackName, err)
|
||||
}
|
||||
m.logger.Printf("[WARN] [backup] %s is HELD STOPPED after a failed update (restore point: %s)", stackName, h.CopyDate)
|
||||
return nil
|
||||
}
|
||||
|
||||
// isHeld reports whether an app carries ANY hold. Used by the nightly legs to leave a held app alone.
|
||||
func (m *Manager) isHeld(stackName string) bool {
|
||||
held, _ := m.RestoreHoldFor(stackName)
|
||||
return held
|
||||
}
|
||||
|
||||
// clearUpdateHoldAfterRestore lifts an UPDATE hold once a person has restored the app successfully.
|
||||
//
|
||||
// "A person clears it by restoring" — slice 4 Part 3. The restore just put the app back on the
|
||||
// definition and data of its recovery unit and started it, which is the exact route back the hold
|
||||
// text names; leaving the hold in place would refuse the next restart of an app that is now fine.
|
||||
//
|
||||
// A RESTORE hold (R-379) is deliberately NOT cleared here: that hold means a previous restore already
|
||||
// left the database in an unknown state, and it stays operator-cleared (`-clear-restore-hold`).
|
||||
func (m *Manager) clearUpdateHoldAfterRestore(stackName string) {
|
||||
if m.settings == nil {
|
||||
return
|
||||
}
|
||||
h, ok := m.settings.GetRestoreHold(stackName)
|
||||
if !ok || h.Reason != settings.HoldReasonUpdateFailed {
|
||||
return
|
||||
}
|
||||
if _, err := m.settings.ClearRestoreHold(stackName); err != nil {
|
||||
m.logger.Printf("[ERROR] [backup] %s was restored, but its update hold could not be cleared: %v", stackName, err)
|
||||
return
|
||||
}
|
||||
m.logger.Printf("[INFO] [backup] %s: restore completed — the update hold (set %s) is CLEARED", stackName, h.At)
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
@@ -46,6 +47,49 @@ type Config struct {
|
||||
Quiesce QuiesceConfig `yaml:"quiesce"`
|
||||
MailRelay MailRelayConfig `yaml:"mail_relay"`
|
||||
Offsite OffsiteConfig `yaml:"offsite"`
|
||||
Update UpdateConfig `yaml:"update"`
|
||||
}
|
||||
|
||||
// UpdateConfig tunes the guarded app update (update arc slice 4, v0.237.0).
|
||||
//
|
||||
// Both are OPERATOR knobs, not constants: "recent" and "healthy in time" are judgements the operator
|
||||
// ruled defaults for (2026-09-13) and may want to move per box without a release.
|
||||
type UpdateConfig struct {
|
||||
// BackupMaxAge is how old the app's proven Tier-2 copy may be before an update makes a fresh one
|
||||
// first. Default "24h".
|
||||
BackupMaxAge string `yaml:"backup_max_age"`
|
||||
// HealthTimeout bounds the wait for the new version to become healthy before the app is held.
|
||||
// Default "5m".
|
||||
HealthTimeout string `yaml:"health_timeout"`
|
||||
}
|
||||
|
||||
// DefaultUpdateBackupMaxAge and DefaultUpdateHealthTimeout are the values used when the config is
|
||||
// empty or unparseable. An unparseable value falls back rather than failing the whole config load:
|
||||
// a typo in a tuning knob must not stop a controller from starting.
|
||||
const (
|
||||
DefaultUpdateBackupMaxAge = 24 * time.Hour
|
||||
DefaultUpdateHealthTimeout = 5 * time.Minute
|
||||
)
|
||||
|
||||
// BackupMaxAgeDuration parses BackupMaxAge, falling back to the default on empty/invalid/non-positive.
|
||||
func (u UpdateConfig) BackupMaxAgeDuration() time.Duration {
|
||||
return parsePositiveDuration(u.BackupMaxAge, DefaultUpdateBackupMaxAge)
|
||||
}
|
||||
|
||||
// HealthTimeoutDuration parses HealthTimeout, falling back to the default on empty/invalid/non-positive.
|
||||
func (u UpdateConfig) HealthTimeoutDuration() time.Duration {
|
||||
return parsePositiveDuration(u.HealthTimeout, DefaultUpdateHealthTimeout)
|
||||
}
|
||||
|
||||
func parsePositiveDuration(s string, def time.Duration) time.Duration {
|
||||
if s == "" {
|
||||
return def
|
||||
}
|
||||
d, err := time.ParseDuration(s)
|
||||
if err != nil || d <= 0 {
|
||||
return def
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// MailRelayConfig tunes the in-controller SMTP shim (app email → shim → hub → Resend).
|
||||
@@ -384,6 +428,8 @@ func applyDefaults(cfg *Config) {
|
||||
d(&cfg.MailRelay.TLSListen, ":2465")
|
||||
d(&cfg.MailRelay.PlainNoTLSListen, ":2526")
|
||||
d(&cfg.MailRelay.ShimHost, "felhom-controller")
|
||||
d(&cfg.Update.BackupMaxAge, "24h")
|
||||
d(&cfg.Update.HealthTimeout, "5m")
|
||||
if len(cfg.MailRelay.FromDomains) == 0 {
|
||||
cfg.MailRelay.FromDomains = []string{"felhom.eu"}
|
||||
}
|
||||
|
||||
@@ -1585,8 +1585,27 @@ type RestoreHold struct {
|
||||
ReplayError string `json:"replay_error,omitempty"` // what the restore hit
|
||||
RollbackErr string `json:"rollback_error,omitempty"` // what the rollback then hit
|
||||
SafetyDump string `json:"safety_dump,omitempty"` // basename of the undo copy that could not be applied
|
||||
|
||||
// Reason (update arc slice 4, v0.237.0) says WHICH operation put the hold in place. Empty means
|
||||
// HoldReasonRestoreFailed — every hold written before this field existed was a restore hold, so
|
||||
// the zero value keeps their meaning and no migration is needed.
|
||||
//
|
||||
// ONE STORAGE, ONE GATE, TWO REASONS — deliberately not a second map. Every start path already
|
||||
// consults GetRestoreHold (the customer's button, the boot sweep, the app-stop guard's Recover),
|
||||
// and "a hold that only one path honours is not a hold". A second map would need every one of
|
||||
// those paths found and changed again, and the one that got missed would be the next R-439.
|
||||
Reason string `json:"reason,omitempty"`
|
||||
// CopyDate is the RFC3339 time of the proven backup the customer is told they can restore from.
|
||||
// Only set for HoldReasonUpdateFailed.
|
||||
CopyDate string `json:"copy_date,omitempty"`
|
||||
}
|
||||
|
||||
// Hold reasons. See RestoreHold.Reason.
|
||||
const (
|
||||
HoldReasonRestoreFailed = "" // R-379/R-380: a restore AND its rollback failed
|
||||
HoldReasonUpdateFailed = "update_failed" // slice 4: the new version did not come up healthy
|
||||
)
|
||||
|
||||
// SetRestoreHold records a hold. Modelled on SetDisconnected: a condition, plus what it is holding.
|
||||
func (s *Settings) SetRestoreHold(h RestoreHold) error {
|
||||
s.mu.Lock()
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"crypto/rand"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"math/big"
|
||||
@@ -236,52 +237,12 @@ func (m *Manager) DeployStack(req DeployRequest) (string, error) {
|
||||
}
|
||||
|
||||
// --- Memory validation ---
|
||||
var deployWarning string
|
||||
reservedMB := m.cfg.System.ReservedMemoryMB
|
||||
totalMB, usedMB, memErr := system.GetMemoryMB()
|
||||
// F1: the controller container cannot read the guest's RAM cap from /proc (no lxcfs) or its own
|
||||
// cgroup (the cap is on the LXC ancestor). Prefer the guest cap from the Docker daemon (runs in the
|
||||
// LXC). And use the controller's OWN committed-memory accounting for "used" — accurate and cheap —
|
||||
// rather than host /proc RSS, which is unobservable-per-guest and would otherwise make this guard
|
||||
// either never fire (host total) or always fire (host used > guest cap).
|
||||
if gt, ok := system.GuestMemTotalMB(); ok && gt > 0 {
|
||||
totalMB = gt
|
||||
memErr = nil
|
||||
}
|
||||
if committedReqMB, _ := m.CommittedMemory(); committedReqMB > 0 || memErr == nil {
|
||||
usedMB = committedReqMB
|
||||
}
|
||||
if memErr != nil {
|
||||
m.logger.Printf("[WARN] [stacks] Cannot read system memory: %v — skipping memory check", memErr)
|
||||
} else {
|
||||
usableMB := totalMB - reservedMB
|
||||
newReqMB := ParseMemoryMB(meta.Resources.MemRequest)
|
||||
|
||||
m.logger.Printf("[INFO] [stacks] Memory check: total=%dMB, reserved=%dMB, usable=%dMB, committed_used=%dMB, new_req=%dMB, remaining=%dMB",
|
||||
totalMB, reservedMB, usableMB, usedMB, newReqMB, usableMB-usedMB-newReqMB)
|
||||
|
||||
// Hard block: committed + new request exceeds usable memory
|
||||
if newReqMB > 0 && usedMB+newReqMB > usableMB {
|
||||
clearDeploying()
|
||||
return "", fmt.Errorf(
|
||||
"Nincs elég memória az alkalmazás telepítéséhez. "+
|
||||
"Szükséges: %d MB, Elérhető: %d MB "+
|
||||
"(összesen: %d MB, ebből %d MB használt, %d MB rendszer számára fenntartva)",
|
||||
newReqMB,
|
||||
usableMB-usedMB,
|
||||
totalMB,
|
||||
usedMB,
|
||||
reservedMB,
|
||||
)
|
||||
}
|
||||
|
||||
// Soft warning: limits exceed total (overcommit)
|
||||
_, currentLimitMB := m.CommittedMemory()
|
||||
newLimitMB := ParseMemoryMB(meta.Resources.MemLimit)
|
||||
if newLimitMB > 0 && currentLimitMB+newLimitMB > totalMB {
|
||||
deployWarning = "Az alkalmazások csúcsterhelése meghaladhatja a rendelkezésre álló memóriát. " +
|
||||
"Normál használat mellett ez nem okoz problémát."
|
||||
}
|
||||
// Slice 4: the block moved into memoryVerdict so the guarded update applies the SAME check with
|
||||
// the SAME wording. Behaviour here is unchanged — same inputs, same log line, same refusal text.
|
||||
refusal, deployWarning := m.memoryVerdict(ParseMemoryMB(meta.Resources.MemRequest), ParseMemoryMB(meta.Resources.MemLimit), 0, 0)
|
||||
if refusal != "" {
|
||||
clearDeploying()
|
||||
return "", errors.New(refusal)
|
||||
}
|
||||
|
||||
// Debug: log received values (redact passwords/secrets)
|
||||
@@ -1197,3 +1158,62 @@ func randomAlphanumeric(length int) (string, error) {
|
||||
}
|
||||
return string(result), nil
|
||||
}
|
||||
|
||||
// memoryVerdict is the deploy's memory check, extracted so the guarded update uses it unchanged
|
||||
// (slice 4). releasedReqMB/releasedLimitMB are what the act FREES before it takes the new amount — an
|
||||
// update replaces the app's own current request, so counting both would refuse an update that fits.
|
||||
// A deploy releases nothing and passes 0, 0.
|
||||
//
|
||||
// Returns the refusal (the deploy's own Hungarian wording, "" = admitted) and the soft overcommit
|
||||
// warning. An unreadable memory reading admits with a WARN, exactly as the deploy always has.
|
||||
func (m *Manager) memoryVerdict(newReqMB, newLimitMB, releasedReqMB, releasedLimitMB int) (refusal, warning string) {
|
||||
reservedMB := m.cfg.System.ReservedMemoryMB
|
||||
totalMB, usedMB, memErr := system.GetMemoryMB()
|
||||
// F1: the controller container cannot read the guest's RAM cap from /proc (no lxcfs) or its own
|
||||
// cgroup (the cap is on the LXC ancestor). Prefer the guest cap from the Docker daemon (runs in the
|
||||
// LXC). And use the controller's OWN committed-memory accounting for "used" — accurate and cheap —
|
||||
// rather than host /proc RSS, which is unobservable-per-guest and would otherwise make this guard
|
||||
// either never fire (host total) or always fire (host used > guest cap).
|
||||
if gt, ok := system.GuestMemTotalMB(); ok && gt > 0 {
|
||||
totalMB = gt
|
||||
memErr = nil
|
||||
}
|
||||
if committedReqMB, _ := m.CommittedMemory(); committedReqMB > 0 || memErr == nil {
|
||||
usedMB = committedReqMB
|
||||
}
|
||||
if memErr != nil {
|
||||
m.logger.Printf("[WARN] [stacks] Cannot read system memory: %v — skipping memory check", memErr)
|
||||
return "", ""
|
||||
}
|
||||
usedMB -= releasedReqMB
|
||||
if usedMB < 0 {
|
||||
usedMB = 0
|
||||
}
|
||||
usableMB := totalMB - reservedMB
|
||||
|
||||
m.logger.Printf("[INFO] [stacks] Memory check: total=%dMB, reserved=%dMB, usable=%dMB, committed_used=%dMB, new_req=%dMB, remaining=%dMB",
|
||||
totalMB, reservedMB, usableMB, usedMB, newReqMB, usableMB-usedMB-newReqMB)
|
||||
|
||||
// Hard block: committed + new request exceeds usable memory
|
||||
if newReqMB > 0 && usedMB+newReqMB > usableMB {
|
||||
return fmt.Sprintf(
|
||||
"Nincs elég memória az alkalmazás telepítéséhez. "+
|
||||
"Szükséges: %d MB, Elérhető: %d MB "+
|
||||
"(összesen: %d MB, ebből %d MB használt, %d MB rendszer számára fenntartva)",
|
||||
newReqMB,
|
||||
usableMB-usedMB,
|
||||
totalMB,
|
||||
usedMB,
|
||||
reservedMB,
|
||||
), ""
|
||||
}
|
||||
|
||||
// Soft warning: limits exceed total (overcommit)
|
||||
_, currentLimitMB := m.CommittedMemory()
|
||||
currentLimitMB -= releasedLimitMB
|
||||
if newLimitMB > 0 && currentLimitMB+newLimitMB > totalMB {
|
||||
warning = "Az alkalmazások csúcsterhelése meghaladhatja a rendelkezésre álló memóriát. " +
|
||||
"Normál használat mellett ez nem okoz problémát."
|
||||
}
|
||||
return "", warning
|
||||
}
|
||||
|
||||
@@ -407,7 +407,9 @@ func TestGroupE_RestartStackReachesTheRecorder(t *testing.T) {
|
||||
func TestGroupE_EveryBringUpPathCallsTheRecorder(t *testing.T) {
|
||||
callers := map[string]bool{} // enclosing func name -> calls recordInstalledImages
|
||||
fset := token.NewFileSet()
|
||||
for _, src := range []string{"manager.go", "deploy.go"} {
|
||||
// update.go since v0.237.0: the guarded update replaced UpdateStack, and it records in
|
||||
// verifyAndConclude — only after the app's health is known (slice 4).
|
||||
for _, src := range []string{"manager.go", "deploy.go", "update.go"} {
|
||||
f, err := parser.ParseFile(fset, src, nil, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -433,7 +435,7 @@ func TestGroupE_EveryBringUpPathCallsTheRecorder(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, want := range []string{"StartStack", "RestartStack", "UpdateStack", "runComposeDeploy"} {
|
||||
for _, want := range []string{"StartStack", "RestartStack", "verifyAndConclude", "runComposeDeploy"} {
|
||||
if !callers[want] {
|
||||
t.Errorf("%s does not call recordInstalledImages — a bring-up path that records nothing leaves a stale record standing", want)
|
||||
}
|
||||
|
||||
@@ -2,6 +2,7 @@ package stacks
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
@@ -130,17 +131,28 @@ type HealthCheckDetail struct {
|
||||
|
||||
// Stack represents a docker compose stack on disk.
|
||||
type Stack struct {
|
||||
Name string `json:"name"`
|
||||
Meta Metadata `json:"meta"`
|
||||
ComposePath string `json:"compose_path"`
|
||||
State ContainerState `json:"state"`
|
||||
Deployed bool `json:"deployed"` // Has app.yaml with deployed=true
|
||||
Protected bool `json:"protected"`
|
||||
Orphaned bool `json:"orphaned"` // Deployed but no catalog template
|
||||
Containers []ContainerInfo `json:"containers"`
|
||||
AppConfig *AppConfig `json:"app_config,omitempty"`
|
||||
Deploying bool `json:"deploying"` // compose up in progress
|
||||
DeployError string `json:"deploy_error,omitempty"` // last async deploy error
|
||||
Name string `json:"name"`
|
||||
Meta Metadata `json:"meta"`
|
||||
ComposePath string `json:"compose_path"`
|
||||
State ContainerState `json:"state"`
|
||||
Deployed bool `json:"deployed"` // Has app.yaml with deployed=true
|
||||
Protected bool `json:"protected"`
|
||||
Orphaned bool `json:"orphaned"` // Deployed but no catalog template
|
||||
Containers []ContainerInfo `json:"containers"`
|
||||
AppConfig *AppConfig `json:"app_config,omitempty"`
|
||||
Deploying bool `json:"deploying"` // compose up in progress
|
||||
DeployError string `json:"deploy_error,omitempty"` // last async deploy error
|
||||
// Updating / UpdatePhase / UpdatePhaseLabel / UpdateError (update arc slice 4, v0.237.0) are the
|
||||
// guarded update's in-memory progress, the same shape as Deploying/DeployError: the API answers
|
||||
// 202 at once and the page polls GET /api/stacks/{name}. See update.go.
|
||||
Updating bool `json:"updating"`
|
||||
UpdatePhase string `json:"update_phase,omitempty"`
|
||||
UpdatePhaseLabel string `json:"update_phase_label,omitempty"`
|
||||
UpdateError string `json:"update_error,omitempty"`
|
||||
// HoldReason is the customer sentence of a hold in force on this app (a failed update or a failed
|
||||
// restore), "" when none. Filled on every read from the ONE hold store, never cached, so the page
|
||||
// and the API cannot show a hold the gate has already lifted — or miss one it enforces.
|
||||
HoldReason string `json:"hold_reason,omitempty"`
|
||||
HealthProbe *HealthProbeResult `json:"health_probe,omitempty"` // controller-side probe result
|
||||
LastUpdated time.Time `json:"last_updated"`
|
||||
// RestartingSince (C9-F2) is when this stack was FIRST observed in StateRestarting during the
|
||||
@@ -198,6 +210,16 @@ type Manager struct {
|
||||
restartPolicyCache map[string]string
|
||||
// execFn replaces execCommand's process boundary in tests; nil in production.
|
||||
execFn func(name string, args ...string) (string, error)
|
||||
|
||||
// --- guarded update (slice 4, update.go) ---
|
||||
updateGuards UpdateGuards // init-only, SetUpdateGuards; nil ⇒ every update is REFUSED
|
||||
updateComposeFn func(dir string, env []string, args ...string) (string, error)
|
||||
updateHealthFn func(ctx context.Context, name string, timeout time.Duration) (bool, string)
|
||||
updateMemoryFn func(newReqMB, newLimitMB, releasedReqMB, releasedLimitMB int) (refusal, warning string)
|
||||
updateDiskFreeFn func() (freeGiB float64, ok bool)
|
||||
updateNowFn func() time.Time
|
||||
updateJournalMu sync.Mutex
|
||||
updateResume []string // apps whose update was interrupted after `up`; resumed once guards exist
|
||||
// inspectRestartPolicyFn is the docker-inspect seam for the above; nil in production
|
||||
// (dockerRestartPolicy). Tests inject a scripted lookup and never touch docker.
|
||||
inspectRestartPolicyFn func(containerName string) (string, error)
|
||||
@@ -960,12 +982,15 @@ func aggregateState(containers []ContainerInfo, policyOf restartPolicyLookup) Co
|
||||
|
||||
func (m *Manager) GetStacks() []Stack {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
result := make([]Stack, 0, len(m.stacks))
|
||||
for _, s := range m.stacks {
|
||||
result = append(result, deepCopyStack(s))
|
||||
}
|
||||
g := m.updateGuards
|
||||
m.mu.RUnlock()
|
||||
for i := range result {
|
||||
fillHoldReason(g, &result[i])
|
||||
}
|
||||
|
||||
// Sort alphabetically by display name for consistent UI ordering
|
||||
sort.Slice(result, func(i, j int) bool {
|
||||
@@ -984,6 +1009,7 @@ func (m *Manager) GetStack(name string) (*Stack, bool) {
|
||||
return nil, false
|
||||
}
|
||||
cp := deepCopyStack(s)
|
||||
fillHoldReason(m.updateGuards, &cp)
|
||||
return &cp, true
|
||||
}
|
||||
|
||||
@@ -1224,54 +1250,11 @@ func (m *Manager) RestartStack(name string) error {
|
||||
return m.RefreshStatus()
|
||||
}
|
||||
|
||||
func (m *Manager) UpdateStack(name string) error {
|
||||
stack, ok := m.GetStack(name)
|
||||
if !ok {
|
||||
return fmt.Errorf("stack %q not found", name)
|
||||
}
|
||||
|
||||
m.logger.Printf("[INFO] [stacks] Updating stack: %s", name)
|
||||
start := time.Now()
|
||||
dir := filepath.Dir(stack.ComposePath)
|
||||
|
||||
// v0.235.0 — ADVANCE THE PIN FIRST, AND RE-RENDER BEFORE THE PULL.
|
||||
//
|
||||
// This is the ONE act entitled to move a version; the freeze exists so that nothing else can.
|
||||
// The ordering is load-bearing, not stylistic: `compose pull` and `up -d` act on the file on
|
||||
// disk, so the catalog's current definition has to BE that file before either runs. Setting the
|
||||
// pin afterwards would pull the frozen version and change nothing, while reporting success — and
|
||||
// a button that lies is worse than a button that refuses.
|
||||
//
|
||||
// A FAILED PIN WRITE REFUSES THE UPDATE, deliberately the opposite of recordInstalledImages.
|
||||
// That field is an observation and a failed write is a bookkeeping gap; this one is INTENT, and
|
||||
// an update whose intent could not be recorded leaves the box running a version it has no record
|
||||
// of choosing — the exact ambiguity R-166 closed for desired_state, one field over.
|
||||
if err := m.advancePinToCatalog(name, dir); err != nil {
|
||||
m.logger.Printf("[ERROR] [stacks] Stack %s update refused: %v", name, err)
|
||||
return fmt.Errorf("updating stack %s: %w", name, err)
|
||||
}
|
||||
|
||||
env := m.stackEnv(dir)
|
||||
|
||||
if m.isDebug() {
|
||||
m.checkLocalImages(name, dir)
|
||||
}
|
||||
|
||||
if _, err := m.composeExecCustomEnv(dir, env, "pull"); err != nil {
|
||||
m.logger.Printf("[ERROR] [stacks] Stack %s update (pull) failed after %.1fs: %v", name, time.Since(start).Seconds(), err)
|
||||
return fmt.Errorf("pulling images for %s: %w", name, err)
|
||||
}
|
||||
|
||||
if _, err := m.composeExecCustomEnv(dir, env, "up", "-d", "--remove-orphans"); err != nil {
|
||||
m.logger.Printf("[ERROR] [stacks] Stack %s update (up) failed after %.1fs: %v", name, time.Since(start).Seconds(), err)
|
||||
return fmt.Errorf("recreating %s: %w", name, err)
|
||||
}
|
||||
|
||||
m.logger.Printf("[INFO] [stacks] Stack %s updated successfully (took %.1fs)", name, time.Since(start).Seconds())
|
||||
m.recordInstalledImages(name, dir, env)
|
||||
m.logPostStartStatus(name, dir, env)
|
||||
return m.RefreshStatus()
|
||||
}
|
||||
// UpdateStack was REMOVED in v0.237.0 (update arc slice 4). It advanced the pin, pulled, ran `up -d`
|
||||
// and reported success on the compose exit code — no copy first, no refusals, and HTTP 200 over a
|
||||
// crash loop (R-443). Its only caller was the API, which now runs StartGuardedUpdate (update.go).
|
||||
// Deleting it rather than leaving it is deliberate: an unguarded update path that still compiles is
|
||||
// one caller away from being the next R-439.
|
||||
|
||||
func (m *Manager) GetLogs(name string, lines int) (string, error) {
|
||||
stack, ok := m.GetStack(name)
|
||||
|
||||
@@ -0,0 +1,802 @@
|
||||
package stacks
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/system"
|
||||
)
|
||||
|
||||
// ── The guarded update (update arc slice 4, controller v0.237.0) ─────────────────────────────────
|
||||
//
|
||||
// WHAT IT REPLACED. `UpdateStack` advanced the pin, pulled, ran `up -d` and returned — no copy first,
|
||||
// no check of memory, disk, a running backup or a held app, and a success the moment `up` returned.
|
||||
// SPIKE-app-update-2026-09-01 §4 measured that as HTTP 200 over an app that was already crash-looping
|
||||
// (R-443), and R-439 is that a held app could be updated at all.
|
||||
//
|
||||
// THE SEQUENCE, and the order is the point:
|
||||
//
|
||||
// checking → backing-up (only if the proven copy is too old) → safety-dump → pinning → pulling
|
||||
// → starting → verifying → done | failed
|
||||
//
|
||||
// 1. The PRECONDITION is the app's existing verified backup (operator ruling 2026-09-02,
|
||||
// 09-update-architecture §3 decision 1): an openable Tier-2 unit with a PROVEN copy date. The
|
||||
// same predicate that permits the destructive „Teljes visszaállítás" permits the update — the
|
||||
// route back IS that restore, so an update without it has no route back.
|
||||
// 2. The safety dump is taken BEFORE the pin moves: it is "the state the customer was in a minute ago",
|
||||
// and a minute later the migration may have run.
|
||||
// 3. The pin moves BEFORE the pull (v0.235.0's reason: pull and up act on the file on disk).
|
||||
// 4. A PULL failure puts the pin BACK — nothing ran, so reverting is safe and honest (Scenario E).
|
||||
// 5. A HEALTH failure leaves the pin where it is — the new version's migration may have run, and a
|
||||
// pin claiming the old version would be a record of something untrue (Scenario F). The app is
|
||||
// HELD STOPPED and the customer is told which backup it can be restored from.
|
||||
//
|
||||
// WHAT IT DELIBERATELY DOES NOT DO: put the old version back by itself. SPIKE-upgrade-test-2026-09-06
|
||||
// measured that whether the old image starts on migrated data depends on the app (PrivateBin yes,
|
||||
// Docmost and Nextcloud no) and cannot be predicted. The route back is the restore.
|
||||
//
|
||||
// CRASH SAFETY IS A JOURNAL, NOT A DEFER. A SIGKILL runs no deferred function (Campaign 8 fault 10),
|
||||
// so every phase is written to `update-journal.json` BEFORE it starts, and RecoverUpdates reads it at
|
||||
// the next startup (Scenario G) — the AppStopGuard pattern.
|
||||
|
||||
// Update phases, as recorded in the journal and served as Stack.UpdatePhase.
|
||||
const (
|
||||
UpdatePhaseChecking = "checking"
|
||||
UpdatePhaseBackingUp = "backing-up"
|
||||
UpdatePhaseSafetyDump = "safety-dump"
|
||||
UpdatePhasePinning = "pinning"
|
||||
UpdatePhasePulling = "pulling"
|
||||
UpdatePhaseStarting = "starting"
|
||||
UpdatePhaseVerifying = "verifying"
|
||||
UpdatePhaseDone = "done"
|
||||
UpdatePhaseFailed = "failed"
|
||||
)
|
||||
|
||||
// updatePhaseLabels are the customer labels (slice 4 Part 4, exact). `pinning` has no row in the
|
||||
// specification — it is instantaneous and is the first step of fetching the new version, so it shares
|
||||
// the pull's label rather than inventing a sentence nobody would read.
|
||||
var updatePhaseLabels = map[string]string{
|
||||
UpdatePhaseChecking: "Ellenőrzés…",
|
||||
UpdatePhaseBackingUp: "Biztonsági mentés készül a frissítés előtt…",
|
||||
UpdatePhaseSafetyDump: "Adatbázis pillanatkép…",
|
||||
UpdatePhasePinning: "Új verzió letöltése…",
|
||||
UpdatePhasePulling: "Új verzió letöltése…",
|
||||
UpdatePhaseStarting: "Indítás az új verzióval…",
|
||||
UpdatePhaseVerifying: "Működés ellenőrzése…",
|
||||
UpdatePhaseDone: "Frissítve",
|
||||
UpdatePhaseFailed: "A frissítés nem sikerült",
|
||||
}
|
||||
|
||||
// UpdatePhaseLabel returns the customer label for a phase, "" for an unknown one.
|
||||
func UpdatePhaseLabel(phase string) string { return updatePhaseLabels[phase] }
|
||||
|
||||
// Customer sentences. Named so tests compare against the constant, never a retyped literal (R-364).
|
||||
const (
|
||||
MsgUpdateNoGuards = "A frissítés nem indítható: a frissítés előtti biztonsági ellenőrzés nem érhető el ezen a szerveren."
|
||||
MsgUpdateNotDeployed = "Az alkalmazás nincs telepítve, ezért nem frissíthető."
|
||||
MsgUpdateDeployingFmt = "A(z) %s telepítése még folyamatban van — a frissítés utána indítható."
|
||||
MsgUpdateAlreadyFmt = "A(z) %s frissítése már folyamatban van."
|
||||
MsgUpdateBusy = "A frissítés most nem indítható: mentés/visszaállítás folyamatban. Próbáld újra, ha befejeződött."
|
||||
MsgUpdateMigrating = "A frissítés most nem indítható: adatáthelyezés folyamatban."
|
||||
MsgUpdateNoBackupFmt = "A(z) %s nem frissíthető, mert nincs olyan biztonsági mentése, amelyből vissza lehetne állítani. Kapcsold be a 2. mentést az alkalmazás mentési beállításainál a Mentések oldalon, és várd meg az első sikeres másolatot — utána a frissítés elindítható."
|
||||
MsgUpdateDiskFmt = "Nincs elég szabad hely a frissítéshez: %.1f GB szabad, az új verzió letöltéséhez legalább %.0f GB szükséges."
|
||||
MsgUpdateBackupFailFmt = "A frissítés nem indult el, mert a frissítés előtti biztonsági mentés nem sikerült: %v. Az alkalmazás változatlanul fut tovább."
|
||||
MsgUpdateBackupNoUnit = "A frissítés nem indult el: a frissítés előtti mentés lefutott, de nem jött létre friss, visszaállítható másolat. Az alkalmazás változatlanul fut tovább."
|
||||
MsgUpdateDumpFailFmt = "A frissítés nem indult el, mert az adatbázis pillanatkép nem készült el: %v. Az alkalmazás változatlanul fut tovább."
|
||||
MsgUpdatePinFailed = "A frissítés nem indult el: az új verzió leírása nem olvasható be. Az alkalmazás változatlanul fut tovább."
|
||||
MsgUpdateJournalFailed = "A frissítés nem indult el: a frissítés naplója nem menthető. Az alkalmazás változatlanul fut tovább."
|
||||
MsgUpdatePullFailed = "Az új verzió letöltése nem sikerült, ezért a frissítés elmaradt. Az alkalmazás a korábbi verzióval fut tovább."
|
||||
MsgUpdateInterrupted = "A frissítés megszakadt, mert a vezérlő újraindult, mielőtt az új verzió elindult volna. Az alkalmazás a korábbi verzióval fut tovább."
|
||||
MsgUpdateHoldUnsaved = "A frissítés nem sikerült, az alkalmazás le lett állítva, de a leállítás rögzítése nem sikerült. Ne indítsd újra — vedd fel velünk a kapcsolatot."
|
||||
)
|
||||
|
||||
// updateDiskFloorGiB is the free space the Docker data root must have before a pull. A FIXED FLOOR,
|
||||
// stated as such: the new image set's size is not known without a registry query (the catalog
|
||||
// records tags, not sizes, and §8.1 of 09 already declines registry calls on the customer box), so
|
||||
// the rule is "not less than 2 GB", not "enough for these images".
|
||||
const updateDiskFloorGiB = 2.0
|
||||
|
||||
// updateSettleWindow is the rule for an app with no .felhom.yml health check: every container
|
||||
// running, none restarting, for this long.
|
||||
const updateSettleWindow = 60 * time.Second
|
||||
|
||||
// updatePollEvery is how often the health wait re-reads the stack.
|
||||
const updatePollEvery = 5 * time.Second
|
||||
|
||||
// UpdateRestorePoint is the precondition answer, reduced to what the update needs.
|
||||
type UpdateRestorePoint struct {
|
||||
Restorable bool // an openable recovery unit exists in the Tier-2 copy
|
||||
Proven bool // a copy actually succeeded (never an attempt clock)
|
||||
ProvenAt time.Time // when the data in that copy was last proven copied
|
||||
}
|
||||
|
||||
// UpdateGuards is everything the update needs from the backup side. The stacks package cannot import
|
||||
// backup, so cmd/controller/main.go wires an adapter (TestSlice4_UpdateGuardsAreWiredAtStartup).
|
||||
type UpdateGuards interface {
|
||||
HoldFor(name string) (bool, string)
|
||||
Busy(name string) (bool, string)
|
||||
RestorePoint(name string) (UpdateRestorePoint, error)
|
||||
BackupNow(ctx context.Context, name string) error
|
||||
SafetyDump(ctx context.Context, name string) ([]string, error)
|
||||
HoldAfterFailedUpdate(name string, at, provenCopyAt time.Time) error
|
||||
}
|
||||
|
||||
// SetUpdateGuards wires the backup side. INIT-ONLY. Unwired, every update is refused (fail closed):
|
||||
// an update that cannot see the backup cannot promise a route back.
|
||||
func (m *Manager) SetUpdateGuards(g UpdateGuards) {
|
||||
m.mu.Lock()
|
||||
m.updateGuards = g
|
||||
m.mu.Unlock()
|
||||
}
|
||||
|
||||
func (m *Manager) guards() UpdateGuards {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
return m.updateGuards
|
||||
}
|
||||
|
||||
func fillHoldReason(g UpdateGuards, st *Stack) {
|
||||
if g == nil || st == nil || !st.Deployed {
|
||||
return
|
||||
}
|
||||
if held, why := g.HoldFor(st.Name); held {
|
||||
st.HoldReason = why
|
||||
}
|
||||
}
|
||||
|
||||
// UpdateRefusal is a refusal taken before anything moved. Reason is a stable key for logs and tests;
|
||||
// Message is the customer sentence.
|
||||
type UpdateRefusal struct {
|
||||
Reason string
|
||||
Message string
|
||||
}
|
||||
|
||||
func (r *UpdateRefusal) Error() string { return r.Message }
|
||||
|
||||
func (m *Manager) now() time.Time {
|
||||
if m.updateNowFn != nil {
|
||||
return m.updateNowFn()
|
||||
}
|
||||
return time.Now()
|
||||
}
|
||||
|
||||
func (m *Manager) refuseUpdate(name, reason, msg, detail string) *UpdateRefusal {
|
||||
m.logger.Printf("[ERROR] [stacks] update %s REFUSED (%s): %s", name, reason, detail)
|
||||
return &UpdateRefusal{Reason: reason, Message: msg}
|
||||
}
|
||||
|
||||
// UpdatePreflight runs every CHEAP refusal (slice 4 Part 1 + the precondition's existence), in order,
|
||||
// and returns the first. Nothing is moved and nothing is recorded by it. The router calls it before
|
||||
// recording the customer's intent, so an update that was never going to happen records nothing.
|
||||
func (m *Manager) UpdatePreflight(name string) *UpdateRefusal {
|
||||
st, ok := m.GetStack(name)
|
||||
if !ok {
|
||||
return m.refuseUpdate(name, "not_found", fmt.Sprintf("stack %q not found", name), "no such stack")
|
||||
}
|
||||
if !st.Deployed {
|
||||
return m.refuseUpdate(name, "not_deployed", MsgUpdateNotDeployed, "not deployed")
|
||||
}
|
||||
g := m.guards()
|
||||
if g == nil {
|
||||
return m.refuseUpdate(name, "guards_unwired", MsgUpdateNoGuards, "no UpdateGuards wired — fail closed")
|
||||
}
|
||||
if st.Deploying {
|
||||
return m.refuseUpdate(name, "deploying", fmt.Sprintf(MsgUpdateDeployingFmt, name), "a deploy is in progress")
|
||||
}
|
||||
if st.Updating {
|
||||
return m.refuseUpdate(name, "updating", fmt.Sprintf(MsgUpdateAlreadyFmt, name), "an update is already in progress")
|
||||
}
|
||||
if held, why := g.HoldFor(name); held {
|
||||
return m.refuseUpdate(name, "held", why, "the app is held")
|
||||
}
|
||||
if busy, why := g.Busy(name); busy {
|
||||
return m.refuseUpdate(name, "busy", MsgUpdateBusy, why)
|
||||
}
|
||||
if m.IsMigrating() {
|
||||
return m.refuseUpdate(name, "migrating", MsgUpdateMigrating, "a data migration is running")
|
||||
}
|
||||
rp, err := g.RestorePoint(name)
|
||||
if err != nil || !rp.Restorable || !rp.Proven {
|
||||
return m.refuseUpdate(name, "no_backup", fmt.Sprintf(MsgUpdateNoBackupFmt, name),
|
||||
fmt.Sprintf("no restorable proven Tier-2 unit (restorable=%v proven=%v err=%v)", rp.Restorable, rp.Proven, err))
|
||||
}
|
||||
if ref := m.updateMemoryRefusal(name, st); ref != nil {
|
||||
return ref
|
||||
}
|
||||
free, known := m.updateDiskFree()
|
||||
switch {
|
||||
case !known:
|
||||
m.logger.Printf("[WARN] [stacks] update %s: free space on the Docker data root is unreadable — proceeding without the %.0f GB floor", name, updateDiskFloorGiB)
|
||||
case free < updateDiskFloorGiB:
|
||||
return m.refuseUpdate(name, "disk", fmt.Sprintf(MsgUpdateDiskFmt, free, updateDiskFloorGiB),
|
||||
fmt.Sprintf("%.2f GiB free on the Docker data root, floor %.0f GiB (fixed floor — image size unknown)", free, updateDiskFloorGiB))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// updateMemoryRefusal applies the deploy's memory check to the NEW template's request, releasing the
|
||||
// app's CURRENT request first (an update replaces it). An unknown new request proceeds with a WARN.
|
||||
func (m *Manager) updateMemoryRefusal(name string, st *Stack) *UpdateRefusal {
|
||||
catPath := m.CatalogTemplatePath(name, ".felhom.yml")
|
||||
if _, err := os.Stat(catPath); err != nil {
|
||||
m.logger.Printf("[WARN] [stacks] update %s: the new template's memory request is unknown (%v) — proceeding without the memory check", name, err)
|
||||
return nil
|
||||
}
|
||||
newMeta := LoadMetadata(filepath.Dir(catPath))
|
||||
newReq, newLim := ParseMemoryMB(newMeta.Resources.MemRequest), ParseMemoryMB(newMeta.Resources.MemLimit)
|
||||
if newReq == 0 {
|
||||
m.logger.Printf("[WARN] [stacks] update %s: the new template declares no memory request — proceeding without the memory check", name)
|
||||
return nil
|
||||
}
|
||||
oldReq, oldLim := ParseMemoryMB(st.Meta.Resources.MemRequest), ParseMemoryMB(st.Meta.Resources.MemLimit)
|
||||
verdict := m.updateMemoryFn
|
||||
if verdict == nil {
|
||||
verdict = m.memoryVerdict
|
||||
}
|
||||
if refusal, _ := verdict(newReq, newLim, oldReq, oldLim); refusal != "" {
|
||||
return m.refuseUpdate(name, "memory", refusal, fmt.Sprintf("new_req=%dMB replacing %dMB does not fit", newReq, oldReq))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Manager) updateDiskFree() (float64, bool) {
|
||||
if m.updateDiskFreeFn != nil {
|
||||
return m.updateDiskFreeFn()
|
||||
}
|
||||
du := system.GetDiskUsage(system.DockerVolumePath)
|
||||
if du == nil {
|
||||
return 0, false
|
||||
}
|
||||
return du.AvailGB, true
|
||||
}
|
||||
|
||||
// StartGuardedUpdate re-checks the cheap refusals, claims the Updating flag atomically and launches the
|
||||
// job. It returns as soon as the job has STARTED — the result arrives on GET /api/stacks/{name}.
|
||||
func (m *Manager) StartGuardedUpdate(name string) error {
|
||||
if ref := m.UpdatePreflight(name); ref != nil {
|
||||
return ref
|
||||
}
|
||||
m.mu.Lock()
|
||||
s, ok := m.stacks[name]
|
||||
if !ok {
|
||||
m.mu.Unlock()
|
||||
return &UpdateRefusal{Reason: "not_found", Message: fmt.Sprintf("stack %q not found", name)}
|
||||
}
|
||||
// A second press between the preflight and here is the race this lock closes.
|
||||
if s.Updating || s.Deploying {
|
||||
m.mu.Unlock()
|
||||
return m.refuseUpdate(name, "updating", fmt.Sprintf(MsgUpdateAlreadyFmt, name), "lost the race for the Updating flag")
|
||||
}
|
||||
s.Updating, s.UpdateError = true, ""
|
||||
s.UpdatePhase, s.UpdatePhaseLabel = UpdatePhaseChecking, UpdatePhaseLabel(UpdatePhaseChecking)
|
||||
m.mu.Unlock()
|
||||
|
||||
m.logger.Printf("[INFO] [stacks] update %s: accepted — guarded update started", name)
|
||||
go m.runGuardedUpdate(context.Background(), name)
|
||||
return nil
|
||||
}
|
||||
|
||||
// IsUpdating reports whether a guarded update is in progress for the app.
|
||||
func (m *Manager) IsUpdating(name string) bool {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
s, ok := m.stacks[name]
|
||||
return ok && s.Updating
|
||||
}
|
||||
|
||||
// UpdatingStacks is the set of apps an update is currently moving — for the dead-app alarm, which
|
||||
// must not count an app the update itself is recreating (R-330's class, a third mechanism).
|
||||
func (m *Manager) UpdatingStacks() map[string]bool {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
var out map[string]bool
|
||||
for name, s := range m.stacks {
|
||||
if s.Updating {
|
||||
if out == nil {
|
||||
out = map[string]bool{}
|
||||
}
|
||||
out[name] = true
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (m *Manager) setUpdatePhase(name, phase string) {
|
||||
m.mu.Lock()
|
||||
if s, ok := m.stacks[name]; ok {
|
||||
s.UpdatePhase, s.UpdatePhaseLabel = phase, UpdatePhaseLabel(phase)
|
||||
}
|
||||
m.mu.Unlock()
|
||||
}
|
||||
|
||||
// finishUpdate is the ONE place Updating goes false. msg is the customer sentence on failure.
|
||||
func (m *Manager) finishUpdate(name, phase, msg string) {
|
||||
m.mu.Lock()
|
||||
if s, ok := m.stacks[name]; ok {
|
||||
s.Updating = false
|
||||
s.UpdatePhase, s.UpdatePhaseLabel = phase, UpdatePhaseLabel(phase)
|
||||
s.UpdateError = msg
|
||||
}
|
||||
m.mu.Unlock()
|
||||
}
|
||||
|
||||
func (m *Manager) updateCompose(dir string, env []string, args ...string) (string, error) {
|
||||
if m.updateComposeFn != nil {
|
||||
return m.updateComposeFn(dir, env, args...)
|
||||
}
|
||||
return m.composeExecCustomEnv(dir, env, args...)
|
||||
}
|
||||
|
||||
func (m *Manager) updateHealth(ctx context.Context, name string, timeout time.Duration) (bool, string) {
|
||||
if m.updateHealthFn != nil {
|
||||
return m.updateHealthFn(ctx, name, timeout)
|
||||
}
|
||||
return m.waitUpdateHealthy(ctx, name, timeout)
|
||||
}
|
||||
|
||||
func (m *Manager) healthTimeout() time.Duration {
|
||||
if m.cfg == nil {
|
||||
return 5 * time.Minute
|
||||
}
|
||||
return m.cfg.Update.HealthTimeoutDuration()
|
||||
}
|
||||
|
||||
func (m *Manager) backupMaxAge() time.Duration {
|
||||
if m.cfg == nil {
|
||||
return 24 * time.Hour
|
||||
}
|
||||
return m.cfg.Update.BackupMaxAgeDuration()
|
||||
}
|
||||
|
||||
// pre-update copies, kept in the stack dir so they travel with it. Neither name is one the syncer
|
||||
// copies (it copies exactly docker-compose.yml and .felhom.yml).
|
||||
const (
|
||||
preUpdateComposeFile = "pre-update-compose.yml"
|
||||
preUpdateAppliedFile = "pre-update-applied.yml"
|
||||
)
|
||||
|
||||
func (m *Manager) runGuardedUpdate(ctx context.Context, name string) {
|
||||
start := m.now()
|
||||
st, ok := m.GetStack(name)
|
||||
if !ok {
|
||||
m.finishUpdate(name, UpdatePhaseFailed, fmt.Sprintf("stack %q not found", name))
|
||||
return
|
||||
}
|
||||
dir := filepath.Dir(st.ComposePath)
|
||||
g := m.guards()
|
||||
entry := updateJournalEntry{StartedAt: start}
|
||||
fail := func(msg, detail string) {
|
||||
m.logger.Printf("[ERROR] [stacks] update %s FAILED in phase %s after %s — nothing was moved: %s", name, entry.Phase, m.now().Sub(start).Round(time.Millisecond), detail)
|
||||
m.clearJournal(name)
|
||||
m.finishUpdate(name, UpdatePhaseFailed, msg)
|
||||
}
|
||||
|
||||
if !m.enterUpdatePhase(name, &entry, UpdatePhaseChecking) {
|
||||
m.finishUpdate(name, UpdatePhaseFailed, MsgUpdateJournalFailed)
|
||||
return
|
||||
}
|
||||
if g == nil {
|
||||
fail(MsgUpdateNoGuards, "no UpdateGuards wired")
|
||||
return
|
||||
}
|
||||
rp, err := g.RestorePoint(name)
|
||||
if err != nil || !rp.Restorable || !rp.Proven {
|
||||
fail(fmt.Sprintf(MsgUpdateNoBackupFmt, name), fmt.Sprintf("precondition vanished: restorable=%v proven=%v err=%v", rp.Restorable, rp.Proven, err))
|
||||
return
|
||||
}
|
||||
|
||||
maxAge := m.backupMaxAge()
|
||||
if age := start.Sub(rp.ProvenAt); age > maxAge {
|
||||
m.logger.Printf("[INFO] [stacks] update %s: the proven copy is %s old (limit %s) — backing up first", name, age.Round(time.Minute), maxAge)
|
||||
if !m.enterUpdatePhase(name, &entry, UpdatePhaseBackingUp) {
|
||||
fail(MsgUpdateJournalFailed, "journal write failed")
|
||||
return
|
||||
}
|
||||
if err := g.BackupNow(ctx, name); err != nil {
|
||||
fail(fmt.Sprintf(MsgUpdateBackupFailFmt, err), "pre-update backup: "+err.Error())
|
||||
return
|
||||
}
|
||||
rp, err = g.RestorePoint(name)
|
||||
if err != nil || !rp.Restorable || !rp.Proven || m.now().Sub(rp.ProvenAt) > maxAge {
|
||||
fail(MsgUpdateBackupNoUnit, fmt.Sprintf("after the backup: restorable=%v proven=%v at=%s err=%v", rp.Restorable, rp.Proven, rp.ProvenAt.Format(time.RFC3339), err))
|
||||
return
|
||||
}
|
||||
} else {
|
||||
m.logger.Printf("[INFO] [stacks] update %s: precondition met — proven copy from %s (%s old, limit %s)", name, rp.ProvenAt.UTC().Format(time.RFC3339), age.Round(time.Minute), maxAge)
|
||||
}
|
||||
entry.ProvenCopyAt = rp.ProvenAt.UTC().Format(time.RFC3339)
|
||||
|
||||
// SAFETY DUMP BEFORE THE PIN MOVES — "a minute ago", before any migration can have run.
|
||||
if !m.enterUpdatePhase(name, &entry, UpdatePhaseSafetyDump) {
|
||||
fail(MsgUpdateJournalFailed, "journal write failed")
|
||||
return
|
||||
}
|
||||
paths, err := g.SafetyDump(ctx, name)
|
||||
if err != nil {
|
||||
fail(fmt.Sprintf(MsgUpdateDumpFailFmt, err), "safety dump: "+err.Error())
|
||||
return
|
||||
}
|
||||
m.logger.Printf("[INFO] [stacks] update %s: safety dump done (%d file(s)) %v", name, len(paths), paths)
|
||||
|
||||
// PINNING — the previous definition is copied aside and journaled BEFORE the pin moves, so a crash
|
||||
// at any later instant can put it back (Scenario G).
|
||||
prevLive, err := os.ReadFile(st.ComposePath)
|
||||
if err != nil {
|
||||
fail(MsgUpdatePinFailed, "reading the live compose file: "+err.Error())
|
||||
return
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, preUpdateComposeFile), prevLive, 0o644); err != nil {
|
||||
fail(MsgUpdateJournalFailed, "saving the pre-update compose copy: "+err.Error())
|
||||
return
|
||||
}
|
||||
entry.PrevCompose = filepath.Join(dir, preUpdateComposeFile)
|
||||
if applied, aerr := LoadAppliedDefinition(dir); aerr == nil {
|
||||
if err := os.WriteFile(filepath.Join(dir, preUpdateAppliedFile), applied, 0o644); err == nil {
|
||||
entry.PrevApplied = filepath.Join(dir, preUpdateAppliedFile)
|
||||
}
|
||||
}
|
||||
if cfg := LoadAppConfig(dir); cfg != nil && len(cfg.PinnedImages) > 0 {
|
||||
entry.PrevPin = map[string]string{}
|
||||
for k, v := range cfg.PinnedImages {
|
||||
entry.PrevPin[k] = v
|
||||
}
|
||||
}
|
||||
if !m.enterUpdatePhase(name, &entry, UpdatePhasePinning) {
|
||||
m.removePreUpdateCopies(dir)
|
||||
fail(MsgUpdateJournalFailed, "journal write failed")
|
||||
return
|
||||
}
|
||||
if err := m.advancePinToCatalog(name, dir); err != nil {
|
||||
m.pinBack(name, dir, entry)
|
||||
fail(MsgUpdatePinFailed, "advancing the pin: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
env := m.stackEnv(dir)
|
||||
if !m.enterUpdatePhase(name, &entry, UpdatePhasePulling) {
|
||||
m.pinBack(name, dir, entry)
|
||||
fail(MsgUpdateJournalFailed, "journal write failed")
|
||||
return
|
||||
}
|
||||
if _, err := m.updateCompose(dir, env, "pull"); err != nil {
|
||||
// Scenario E: NOTHING RAN. The containers are the old ones and still running, so the honest
|
||||
// state is the old pin and the old file — put both back.
|
||||
m.pinBack(name, dir, entry)
|
||||
m.logger.Printf("[ERROR] [stacks] update %s: pull failed — pin and definition PUT BACK; the app was not touched. Docker said: %v", name, err)
|
||||
fail(MsgUpdatePullFailed, "pull failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
if !m.enterUpdatePhase(name, &entry, UpdatePhaseStarting) {
|
||||
m.failAndHold(ctx, name, dir, env, rp.ProvenAt, "journal write failed before up")
|
||||
return
|
||||
}
|
||||
if _, err := m.updateCompose(dir, env, "up", "-d", "--remove-orphans"); err != nil {
|
||||
// Containers may already have been recreated on the new image — something may have run.
|
||||
m.failAndHold(ctx, name, dir, env, rp.ProvenAt, "compose up failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
m.verifyAndConclude(ctx, name, dir, env, rp.ProvenAt, start, &entry)
|
||||
}
|
||||
|
||||
// verifyAndConclude is the TRUTH half (R-443): success is declared only after the app's health is
|
||||
// known, and a failure holds the app.
|
||||
func (m *Manager) verifyAndConclude(ctx context.Context, name, dir string, env []string, provenAt, start time.Time, entry *updateJournalEntry) {
|
||||
if !m.enterUpdatePhase(name, entry, UpdatePhaseVerifying) {
|
||||
m.logger.Printf("[ERROR] [stacks] update %s: could not journal the verifying phase — verifying anyway", name)
|
||||
}
|
||||
timeout := m.healthTimeout()
|
||||
waitStart := m.now()
|
||||
healthy, detail := m.updateHealth(ctx, name, timeout)
|
||||
if !healthy {
|
||||
m.failAndHold(ctx, name, dir, env, provenAt, "not healthy: "+detail)
|
||||
return
|
||||
}
|
||||
m.logger.Printf("[INFO] [stacks] update %s: healthy after %s (%s)", name, m.now().Sub(waitStart).Round(time.Second), detail)
|
||||
m.recordInstalledImages(name, dir, env)
|
||||
_ = m.RefreshStatus()
|
||||
m.clearJournal(name)
|
||||
m.removePreUpdateCopies(dir)
|
||||
m.finishUpdate(name, UpdatePhaseDone, "")
|
||||
m.logger.Printf("[INFO] [stacks] update %s: DONE in %s", name, m.now().Sub(start).Round(time.Second))
|
||||
}
|
||||
|
||||
// failAndHold is Scenario F: stop the app, record the hold, tell the customer the route back.
|
||||
func (m *Manager) failAndHold(ctx context.Context, name, dir string, env []string, provenAt time.Time, why string) {
|
||||
m.logger.Printf("[ERROR] [stacks] update %s FAILED after the new version was started: %s — stopping and HOLDING the app; the pin stays on the new version (its migration may have run)", name, why)
|
||||
if _, err := m.updateCompose(dir, env, "down"); err != nil {
|
||||
m.logger.Printf("[ERROR] [stacks] update %s: stopping the failed app also failed: %v", name, err)
|
||||
}
|
||||
msg := MsgUpdateHoldUnsaved
|
||||
if g := m.guards(); g == nil {
|
||||
m.logger.Printf("[ERROR] [stacks] update %s: no UpdateGuards — the hold CANNOT be recorded", name)
|
||||
} else if err := g.HoldAfterFailedUpdate(name, m.now(), provenAt); err != nil {
|
||||
m.logger.Printf("[ERROR] [stacks] update %s: %v", name, err)
|
||||
} else if _, why := g.HoldFor(name); why != "" {
|
||||
msg = why
|
||||
}
|
||||
_ = m.RefreshStatus()
|
||||
m.clearJournal(name)
|
||||
m.removePreUpdateCopies(dir)
|
||||
m.finishUpdate(name, UpdatePhaseFailed, msg)
|
||||
}
|
||||
|
||||
// pinBack restores the pin, the stored definition and the live file from the journaled copies.
|
||||
func (m *Manager) pinBack(name, dir string, entry updateJournalEntry) {
|
||||
prevLive, lerr := os.ReadFile(entry.PrevCompose)
|
||||
if lerr != nil {
|
||||
m.logger.Printf("[ERROR] [stacks] update %s: cannot read the pre-update compose copy (%v) — the definition could NOT be put back", name, lerr)
|
||||
}
|
||||
if len(entry.PrevPin) > 0 {
|
||||
applied := prevLive
|
||||
if entry.PrevApplied != "" {
|
||||
if b, err := os.ReadFile(entry.PrevApplied); err == nil {
|
||||
applied = b
|
||||
}
|
||||
}
|
||||
if err := m.SetPin(name, dir, entry.PrevPin, applied); err != nil {
|
||||
m.logger.Printf("[ERROR] [stacks] update %s: putting the pin back failed: %v", name, err)
|
||||
}
|
||||
}
|
||||
if lerr == nil {
|
||||
if err := os.WriteFile(ComposePathIn(dir), prevLive, 0o644); err != nil {
|
||||
m.logger.Printf("[ERROR] [stacks] update %s: re-rendering the previous definition failed: %v", name, err)
|
||||
}
|
||||
}
|
||||
m.removePreUpdateCopies(dir)
|
||||
m.logger.Printf("[INFO] [stacks] update %s: pin and definition PUT BACK to the pre-update version (%s)", name, summarisePin(entry.PrevPin))
|
||||
}
|
||||
|
||||
func (m *Manager) removePreUpdateCopies(dir string) {
|
||||
_ = os.Remove(filepath.Join(dir, preUpdateComposeFile))
|
||||
_ = os.Remove(filepath.Join(dir, preUpdateAppliedFile))
|
||||
}
|
||||
|
||||
// waitUpdateHealthy is the production health wait: the app's own .felhom.yml health check through the
|
||||
// existing probe, or — for an app with none — every container running and none restarting for
|
||||
// updateSettleWindow. NEVER the compose exit code, and never logPostStartStatus's delayed log line.
|
||||
func (m *Manager) waitUpdateHealthy(ctx context.Context, name string, timeout time.Duration) (bool, string) {
|
||||
deadline := m.now().Add(timeout)
|
||||
var runningSince time.Time
|
||||
last := "no observation yet"
|
||||
for {
|
||||
_ = m.RefreshStatus()
|
||||
st, ok := m.GetStack(name)
|
||||
switch {
|
||||
case !ok:
|
||||
last = "stack vanished"
|
||||
runningSince = time.Time{}
|
||||
case st.State == StateRunning:
|
||||
if hc := st.Meta.HealthCheck; hc != nil && len(hc.Checks) > 0 {
|
||||
if c := findProbeContainer(name, st.Containers); c != "" {
|
||||
res := m.runChecks(probeTarget{stackName: name, containerName: c, checks: hc.Checks})
|
||||
m.mu.Lock()
|
||||
if s, ok := m.stacks[name]; ok {
|
||||
s.HealthProbe = res
|
||||
}
|
||||
m.mu.Unlock()
|
||||
if res.Healthy {
|
||||
return true, "the app's health check passed"
|
||||
}
|
||||
last = "health check failing"
|
||||
} else {
|
||||
last = "no probe container"
|
||||
}
|
||||
} else {
|
||||
if runningSince.IsZero() {
|
||||
runningSince = m.now()
|
||||
}
|
||||
if m.now().Sub(runningSince) >= updateSettleWindow {
|
||||
return true, fmt.Sprintf("all containers running, none restarting, for %s (no health check declared)", updateSettleWindow)
|
||||
}
|
||||
last = "running, settling"
|
||||
}
|
||||
default:
|
||||
runningSince = time.Time{}
|
||||
last = "state " + string(st.State)
|
||||
}
|
||||
if !m.now().Before(deadline) {
|
||||
return false, fmt.Sprintf("not healthy within %s (last: %s)", timeout, last)
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return false, "cancelled: " + ctx.Err().Error()
|
||||
case <-time.After(updatePollEvery):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── the journal ──────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
type updateJournalEntry struct {
|
||||
Phase string `json:"phase"`
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
PrevPin map[string]string `json:"prev_pin,omitempty"`
|
||||
PrevCompose string `json:"prev_compose,omitempty"`
|
||||
PrevApplied string `json:"prev_applied,omitempty"`
|
||||
ProvenCopyAt string `json:"proven_copy_at,omitempty"`
|
||||
}
|
||||
|
||||
type updateJournal struct {
|
||||
Updates map[string]updateJournalEntry `json:"updates"`
|
||||
}
|
||||
|
||||
func (m *Manager) updateJournalPath() string {
|
||||
return filepath.Join(m.cfg.Paths.DataDir, "update-journal.json")
|
||||
}
|
||||
|
||||
func (m *Manager) readUpdateJournal() updateJournal {
|
||||
j := updateJournal{Updates: map[string]updateJournalEntry{}}
|
||||
data, err := os.ReadFile(m.updateJournalPath())
|
||||
if err != nil {
|
||||
return j
|
||||
}
|
||||
if err := json.Unmarshal(data, &j); err != nil {
|
||||
m.logger.Printf("[WARN] [stacks] update journal at %s is corrupt (%v) — quarantining", m.updateJournalPath(), err)
|
||||
_ = os.Rename(m.updateJournalPath(), fmt.Sprintf("%s.corrupt-%d", m.updateJournalPath(), time.Now().Unix()))
|
||||
return updateJournal{Updates: map[string]updateJournalEntry{}}
|
||||
}
|
||||
if j.Updates == nil {
|
||||
j.Updates = map[string]updateJournalEntry{}
|
||||
}
|
||||
return j
|
||||
}
|
||||
|
||||
// writeUpdateJournal is atomic and fsynced (the AppStopGuard shape): the point is surviving a power cut.
|
||||
func (m *Manager) writeUpdateJournal(j updateJournal) error {
|
||||
p := m.updateJournalPath()
|
||||
if len(j.Updates) == 0 {
|
||||
if err := os.Remove(p); err != nil && !os.IsNotExist(err) {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
data, err := json.MarshalIndent(j, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := p + ".tmp"
|
||||
f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := f.Write(data); err != nil {
|
||||
f.Close()
|
||||
os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
if err := f.Sync(); err != nil {
|
||||
f.Close()
|
||||
os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
if err := f.Close(); err != nil {
|
||||
os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, p)
|
||||
}
|
||||
|
||||
// enterUpdatePhase journals the phase BEFORE it starts and mirrors it for the UI. False means the
|
||||
// journal could not be written — the caller must not perform a mutation it could not record.
|
||||
func (m *Manager) enterUpdatePhase(name string, entry *updateJournalEntry, phase string) bool {
|
||||
entry.Phase = phase
|
||||
m.updateJournalMu.Lock()
|
||||
j := m.readUpdateJournal()
|
||||
j.Updates[name] = *entry
|
||||
err := m.writeUpdateJournal(j)
|
||||
m.updateJournalMu.Unlock()
|
||||
m.setUpdatePhase(name, phase)
|
||||
if err != nil {
|
||||
m.logger.Printf("[ERROR] [stacks] update %s: journal write for phase %s failed: %v", name, phase, err)
|
||||
return false
|
||||
}
|
||||
m.logger.Printf("[INFO] [stacks] update %s: phase %s", name, phase)
|
||||
return true
|
||||
}
|
||||
|
||||
func (m *Manager) clearJournal(name string) {
|
||||
m.updateJournalMu.Lock()
|
||||
defer m.updateJournalMu.Unlock()
|
||||
j := m.readUpdateJournal()
|
||||
delete(j.Updates, name)
|
||||
if err := m.writeUpdateJournal(j); err != nil {
|
||||
m.logger.Printf("[ERROR] [stacks] update %s: clearing the journal entry failed: %v", name, err)
|
||||
}
|
||||
}
|
||||
|
||||
// RecoverUpdates reads the journal at startup (Scenario G). Call it BEFORE the boot reconciler.
|
||||
//
|
||||
// - interrupted BEFORE the pin moved (checking, backing-up, safety-dump): nothing moved — the entry is
|
||||
// dropped and the app carries the "interrupted" sentence;
|
||||
// - interrupted while pinning or pulling: nothing RAN — the pin and definition are put back (as E);
|
||||
// - interrupted while starting or verifying: something may have run — the app is marked Updating
|
||||
// (so the boot sweep and the dead-app alarm leave it alone) and queued for ResumeInterruptedUpdates,
|
||||
// which re-runs `up -d` and the health wait, ending in A or F.
|
||||
//
|
||||
// It needs no backup wiring, because nothing here holds an app — that is left to the resumed job.
|
||||
func (m *Manager) RecoverUpdates() []string {
|
||||
m.updateJournalMu.Lock()
|
||||
j := m.readUpdateJournal()
|
||||
m.updateJournalMu.Unlock()
|
||||
if len(j.Updates) == 0 {
|
||||
return nil
|
||||
}
|
||||
var resumed []string
|
||||
for name, e := range j.Updates {
|
||||
st, ok := m.GetStack(name)
|
||||
if !ok {
|
||||
m.logger.Printf("[WARN] [stacks] update recovery: %s is in the journal (phase %s) but no longer exists — dropping the entry", name, e.Phase)
|
||||
m.clearJournal(name)
|
||||
continue
|
||||
}
|
||||
dir := filepath.Dir(st.ComposePath)
|
||||
switch e.Phase {
|
||||
case UpdatePhaseChecking, UpdatePhaseBackingUp, UpdatePhaseSafetyDump:
|
||||
m.logger.Printf("[WARN] [stacks] update recovery: %s was interrupted in %s (started %s) — nothing had moved; dropping it", name, e.Phase, e.StartedAt.Format(time.RFC3339))
|
||||
m.clearJournal(name)
|
||||
m.finishUpdate(name, UpdatePhaseFailed, MsgUpdateInterrupted)
|
||||
case UpdatePhasePinning, UpdatePhasePulling:
|
||||
m.logger.Printf("[WARN] [stacks] update recovery: %s was interrupted in %s (started %s) — nothing had run; putting the pin back", name, e.Phase, e.StartedAt.Format(time.RFC3339))
|
||||
m.pinBack(name, dir, e)
|
||||
m.clearJournal(name)
|
||||
m.finishUpdate(name, UpdatePhaseFailed, MsgUpdateInterrupted)
|
||||
case UpdatePhaseStarting, UpdatePhaseVerifying:
|
||||
m.logger.Printf("[WARN] [stacks] update recovery: %s was interrupted in %s (started %s) — the new version may have run; marking it Updating and RESUMING the health wait", name, e.Phase, e.StartedAt.Format(time.RFC3339))
|
||||
m.mu.Lock()
|
||||
if s, ok := m.stacks[name]; ok {
|
||||
s.Updating, s.UpdateError = true, ""
|
||||
s.UpdatePhase, s.UpdatePhaseLabel = UpdatePhaseVerifying, UpdatePhaseLabel(UpdatePhaseVerifying)
|
||||
}
|
||||
m.updateResume = append(m.updateResume, name)
|
||||
m.mu.Unlock()
|
||||
resumed = append(resumed, name)
|
||||
default:
|
||||
m.logger.Printf("[WARN] [stacks] update recovery: %s has unknown phase %q — dropping the entry", name, e.Phase)
|
||||
m.clearJournal(name)
|
||||
}
|
||||
}
|
||||
return resumed
|
||||
}
|
||||
|
||||
// ResumeInterruptedUpdates continues the updates RecoverUpdates queued, once the backup side is wired
|
||||
// (a resumed update that fails must be able to HOLD). Returns how many were resumed.
|
||||
func (m *Manager) ResumeInterruptedUpdates(ctx context.Context) int {
|
||||
m.mu.Lock()
|
||||
names := m.updateResume
|
||||
m.updateResume = nil
|
||||
m.mu.Unlock()
|
||||
for _, name := range names {
|
||||
st, ok := m.GetStack(name)
|
||||
if !ok {
|
||||
m.finishUpdate(name, UpdatePhaseFailed, MsgUpdateInterrupted)
|
||||
continue
|
||||
}
|
||||
m.updateJournalMu.Lock()
|
||||
e, ok := m.readUpdateJournal().Updates[name]
|
||||
m.updateJournalMu.Unlock()
|
||||
if !ok {
|
||||
m.finishUpdate(name, UpdatePhaseFailed, MsgUpdateInterrupted)
|
||||
continue
|
||||
}
|
||||
provenAt, _ := time.Parse(time.RFC3339, e.ProvenCopyAt)
|
||||
dir := filepath.Dir(st.ComposePath)
|
||||
go func(name, dir string, e updateJournalEntry, provenAt time.Time) {
|
||||
env := m.stackEnv(dir)
|
||||
m.logger.Printf("[INFO] [stacks] update %s: resuming after a controller restart — `up -d` then the health wait", name)
|
||||
if _, err := m.updateCompose(dir, env, "up", "-d", "--remove-orphans"); err != nil {
|
||||
m.failAndHold(ctx, name, dir, env, provenAt, "resumed compose up failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
m.verifyAndConclude(ctx, name, dir, env, provenAt, e.StartedAt, &e)
|
||||
}(name, dir, e, provenAt)
|
||||
}
|
||||
return len(names)
|
||||
}
|
||||
@@ -0,0 +1,569 @@
|
||||
package stacks
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Update arc slice 4 — the guarded update. Scenarios A–G of the task, through the real job
|
||||
// (runGuardedUpdate) with the four process boundaries injected: compose, the health wait, the backup
|
||||
// side (UpdateGuards) and the clock. Every assertion reads the EFFECT back — the pin in app.yaml, the
|
||||
// bytes of the live compose file, the journal on disk, Updating/UpdatePhase/UpdateError — never "no error".
|
||||
|
||||
var slice4T0 = time.Date(2026, 9, 13, 10, 0, 0, 0, time.UTC)
|
||||
|
||||
type fakeGuards struct {
|
||||
mu sync.Mutex
|
||||
calls []string
|
||||
held bool
|
||||
holdWhy string
|
||||
busy bool
|
||||
rp UpdateRestorePoint
|
||||
rpErr error
|
||||
rpAfterBackup *UpdateRestorePoint
|
||||
backupErr error
|
||||
dumpErr error
|
||||
holdErr error
|
||||
holdProvenAt time.Time
|
||||
pinAtDump string
|
||||
stackDir string
|
||||
}
|
||||
|
||||
func (f *fakeGuards) note(c string) { f.mu.Lock(); f.calls = append(f.calls, c); f.mu.Unlock() }
|
||||
func (f *fakeGuards) callList() []string {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([]string(nil), f.calls...)
|
||||
}
|
||||
func (f *fakeGuards) HoldFor(string) (bool, string) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return f.held, f.holdWhy
|
||||
}
|
||||
func (f *fakeGuards) Busy(string) (bool, string) { return f.busy, "fake busy" }
|
||||
func (f *fakeGuards) RestorePoint(string) (UpdateRestorePoint, error) {
|
||||
f.note("RestorePoint")
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return f.rp, f.rpErr
|
||||
}
|
||||
func (f *fakeGuards) BackupNow(context.Context, string) error {
|
||||
f.note("BackupNow")
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if f.backupErr == nil && f.rpAfterBackup != nil {
|
||||
f.rp = *f.rpAfterBackup
|
||||
}
|
||||
return f.backupErr
|
||||
}
|
||||
func (f *fakeGuards) SafetyDump(context.Context, string) ([]string, error) {
|
||||
f.note("SafetyDump")
|
||||
if cfg := LoadAppConfig(f.stackDir); cfg != nil {
|
||||
f.mu.Lock()
|
||||
f.pinAtDump = cfg.PinnedImages["web"]
|
||||
f.mu.Unlock()
|
||||
}
|
||||
return []string{"/fake/pre-restore-x.sql"}, f.dumpErr
|
||||
}
|
||||
func (f *fakeGuards) HoldAfterFailedUpdate(_ string, _ time.Time, provenAt time.Time) error {
|
||||
f.note("HoldAfterFailedUpdate")
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if f.holdErr != nil {
|
||||
return f.holdErr
|
||||
}
|
||||
f.held, f.holdWhy, f.holdProvenAt = true, "HELD-SENTENCE", provenAt
|
||||
return nil
|
||||
}
|
||||
|
||||
type composeRec struct {
|
||||
mu sync.Mutex
|
||||
calls []string
|
||||
fail map[string]error // first arg → error
|
||||
}
|
||||
|
||||
func (c *composeRec) fn(_ string, _ []string, args ...string) (string, error) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
c.calls = append(c.calls, strings.Join(args, " "))
|
||||
if err := c.fail[args[0]]; err != nil {
|
||||
return "", err
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
func (c *composeRec) list() []string {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return append([]string(nil), c.calls...)
|
||||
}
|
||||
|
||||
// newSlice4Manager: a pinned nextcloud on the OLD version, the catalog offering the NEW one, fresh
|
||||
// proven copy, and every boundary faked. Returns the manager, its stack dir, the guards and compose.
|
||||
func newSlice4Manager(t *testing.T) (*Manager, string, *fakeGuards, *composeRec) {
|
||||
t.Helper()
|
||||
m, dir := newPinManager(t, pinTplOld, pinTplNew,
|
||||
"deployed: true\nenv: {}\npinned_images:\n web: nextcloud:31.0.14-apache\n")
|
||||
mustWrite(t, AppliedComposePath(dir), pinTplOld)
|
||||
g := &fakeGuards{rp: UpdateRestorePoint{Restorable: true, Proven: true, ProvenAt: slice4T0.Add(-1 * time.Hour)}, stackDir: dir}
|
||||
c := &composeRec{fail: map[string]error{}}
|
||||
m.updateGuards = g
|
||||
m.updateComposeFn = c.fn
|
||||
m.updateHealthFn = func(context.Context, string, time.Duration) (bool, string) { return true, "fake healthy" }
|
||||
m.updateMemoryFn = func(int, int, int, int) (string, string) { return "", "" }
|
||||
m.updateDiskFreeFn = func() (float64, bool) { return 50, true }
|
||||
m.updateNowFn = func() time.Time { return slice4T0 } // R-457: the SAME clock the age check reads
|
||||
m.execFn = func(string, ...string) (string, error) { return "", nil }
|
||||
return m, dir, g, c
|
||||
}
|
||||
|
||||
func waitUpdateDone(t *testing.T, m *Manager, name string) *Stack {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
if st, ok := m.GetStack(name); ok && !st.Updating {
|
||||
return st
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
t.Fatal("the update never finished")
|
||||
return nil
|
||||
}
|
||||
|
||||
func pinOf(t *testing.T, dir string) string { return readPin(t, dir).PinnedImages["web"] }
|
||||
|
||||
func fileBody(t *testing.T, p string) string {
|
||||
t.Helper()
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func journalExists(m *Manager) bool {
|
||||
_, err := os.Stat(m.updateJournalPath())
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// ── A: the happy path, and the truth ─────────────────────────────────────────────────────────────
|
||||
|
||||
// TestSlice4_A_SuccessIsDeclaredOnlyAfterHealth. The health wait BLOCKS until the test releases it;
|
||||
// while it blocks, the app must read Updating=true / phase=verifying / no error — and the safety dump
|
||||
// must have run while the pin still named the OLD version.
|
||||
//
|
||||
// COMPANION RED-PROOF 1 (REPORT.md): delete the updateHealth call from verifyAndConclude so success is
|
||||
// declared on the compose exit code. This test then fails at "Updating went false before health was
|
||||
// known" — which is R-443 exactly.
|
||||
func TestSlice4_A_SuccessIsDeclaredOnlyAfterHealth(t *testing.T) {
|
||||
m, dir, g, c := newSlice4Manager(t)
|
||||
release := make(chan struct{})
|
||||
healthCalled := make(chan struct{}, 1)
|
||||
m.updateHealthFn = func(ctx context.Context, name string, timeout time.Duration) (bool, string) {
|
||||
healthCalled <- struct{}{}
|
||||
<-release
|
||||
return true, "fake healthy"
|
||||
}
|
||||
if err := m.StartGuardedUpdate("nextcloud"); err != nil {
|
||||
t.Fatalf("a fully-qualified update must start: %v", err)
|
||||
}
|
||||
select {
|
||||
case <-healthCalled:
|
||||
case <-time.After(5 * time.Second):
|
||||
st, _ := m.GetStack("nextcloud")
|
||||
t.Fatalf("the health wait was never reached; state: updating=%v phase=%s err=%q", st.Updating, st.UpdatePhase, st.UpdateError)
|
||||
}
|
||||
st, _ := m.GetStack("nextcloud")
|
||||
if !st.Updating {
|
||||
t.Fatal("Updating went false before health was known — success reported on the compose exit code (R-443)")
|
||||
}
|
||||
if st.UpdatePhase != UpdatePhaseVerifying || st.UpdatePhaseLabel != "Működés ellenőrzése…" {
|
||||
t.Errorf("while waiting for health the phase must be verifying, got %q / %q", st.UpdatePhase, st.UpdatePhaseLabel)
|
||||
}
|
||||
if st.UpdateError != "" {
|
||||
t.Errorf("no error may be shown while verifying, got %q", st.UpdateError)
|
||||
}
|
||||
if !journalExists(m) {
|
||||
t.Error("the journal must exist while the update is in flight (Scenario G depends on it)")
|
||||
}
|
||||
if got := pinOf(t, dir); got != "nextcloud:34.0.1-apache" {
|
||||
t.Errorf("by verifying, the pin must have advanced; got %q", got)
|
||||
}
|
||||
close(release)
|
||||
st = waitUpdateDone(t, m, "nextcloud")
|
||||
if st.UpdatePhase != UpdatePhaseDone || st.UpdateError != "" || st.UpdatePhaseLabel != "Frissítve" {
|
||||
t.Fatalf("after health the update is done: phase=%q label=%q err=%q", st.UpdatePhase, st.UpdatePhaseLabel, st.UpdateError)
|
||||
}
|
||||
if journalExists(m) {
|
||||
t.Error("a completed update must clear its journal entry")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, preUpdateComposeFile)); err == nil {
|
||||
t.Error("the pre-update copy must be removed after success")
|
||||
}
|
||||
if g.pinAtDump != "nextcloud:31.0.14-apache" {
|
||||
t.Errorf("the safety dump must run BEFORE the pin moves (\"a minute ago\"); the pin at dump time was %q", g.pinAtDump)
|
||||
}
|
||||
if got, want := strings.Join(c.list(), " | "), "pull | up -d --remove-orphans"; got != want {
|
||||
t.Errorf("compose calls = %q, want %q", got, want)
|
||||
}
|
||||
// RestorePoint twice by design: once in the preflight (the refusal), once inside the job (the
|
||||
// precondition must still hold when the job actually starts).
|
||||
if got := strings.Join(g.callList(), ","); got != "RestorePoint,RestorePoint,SafetyDump" {
|
||||
t.Errorf("a fresh copy needs no backup-first; guard calls = %s", got)
|
||||
}
|
||||
}
|
||||
|
||||
// ── B: the proven copy is too old ─────────────────────────────────────────────────────────────────
|
||||
|
||||
func TestSlice4_B_StaleCopyIsRefreshedFirst(t *testing.T) {
|
||||
m, dir, g, _ := newSlice4Manager(t)
|
||||
g.rp.ProvenAt = slice4T0.Add(-30 * time.Hour) // > 24 h default
|
||||
g.rpAfterBackup = &UpdateRestorePoint{Restorable: true, Proven: true, ProvenAt: slice4T0.Add(-1 * time.Minute)}
|
||||
if err := m.StartGuardedUpdate("nextcloud"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st := waitUpdateDone(t, m, "nextcloud")
|
||||
if st.UpdatePhase != UpdatePhaseDone {
|
||||
t.Fatalf("with a successful backup-first the update completes, got phase=%q err=%q", st.UpdatePhase, st.UpdateError)
|
||||
}
|
||||
calls := strings.Join(g.callList(), ",")
|
||||
if !strings.HasPrefix(calls, "RestorePoint,RestorePoint,BackupNow,RestorePoint,SafetyDump") {
|
||||
t.Errorf("a stale copy must be backed up FIRST and the precondition re-read; calls = %s", calls)
|
||||
}
|
||||
if got := pinOf(t, dir); got != "nextcloud:34.0.1-apache" {
|
||||
t.Errorf("pin = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlice4_B_BackupFailureMovesNothing(t *testing.T) {
|
||||
m, dir, g, c := newSlice4Manager(t)
|
||||
g.rp.ProvenAt = slice4T0.Add(-30 * time.Hour)
|
||||
g.backupErr = errors.New("disk full")
|
||||
if err := m.StartGuardedUpdate("nextcloud"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st := waitUpdateDone(t, m, "nextcloud")
|
||||
if want := fmt.Sprintf(MsgUpdateBackupFailFmt, g.backupErr); st.UpdateError != want {
|
||||
t.Errorf("UpdateError = %q, want the backup's own error in the sentence %q", st.UpdateError, want)
|
||||
}
|
||||
if got := pinOf(t, dir); got != "nextcloud:31.0.14-apache" {
|
||||
t.Errorf("a failed backup must move nothing; pin = %q", got)
|
||||
}
|
||||
if len(c.list()) != 0 {
|
||||
t.Errorf("a failed backup must reach no compose call; got %v", c.list())
|
||||
}
|
||||
if journalExists(m) {
|
||||
t.Error("the journal must be cleared on a refusal")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlice4_B_BackupThatYieldsNoFreshUnitRefuses(t *testing.T) {
|
||||
m, dir, g, c := newSlice4Manager(t)
|
||||
g.rp.ProvenAt = slice4T0.Add(-30 * time.Hour) // stays stale: rpAfterBackup nil
|
||||
if err := m.StartGuardedUpdate("nextcloud"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st := waitUpdateDone(t, m, "nextcloud")
|
||||
if st.UpdateError != MsgUpdateBackupNoUnit {
|
||||
t.Errorf("UpdateError = %q", st.UpdateError)
|
||||
}
|
||||
if pinOf(t, dir) != "nextcloud:31.0.14-apache" || len(c.list()) != 0 {
|
||||
t.Error("nothing may move when the backup did not produce a fresh restorable copy")
|
||||
}
|
||||
}
|
||||
|
||||
// ── C: no backup exists that could restore this app ───────────────────────────────────────────────
|
||||
|
||||
// COMPANION RED-PROOF 2 (REPORT.md): make the precondition in UpdatePreflight proceed when
|
||||
// !rp.Restorable. This test then fails with the update started.
|
||||
func TestSlice4_C_NoRestorableCopyRefusesBeforeAnythingMoves(t *testing.T) {
|
||||
m, dir, g, c := newSlice4Manager(t)
|
||||
// A PROVEN, FRESH copy whose unit cannot be opened — the realistic half-copied mirror. Proven and
|
||||
// fresh on purpose: a fixture that is also unproven would be refused by the proven check alone,
|
||||
// and a red-proof that drops the restorable check would then pass inertly (observed on the first
|
||||
// run of red-proof 2, 2026-09-13).
|
||||
g.rp = UpdateRestorePoint{Restorable: false, Proven: true, ProvenAt: slice4T0.Add(-time.Hour)}
|
||||
err := m.StartGuardedUpdate("nextcloud")
|
||||
var ref *UpdateRefusal
|
||||
if !errors.As(err, &ref) || ref.Reason != "no_backup" {
|
||||
t.Fatalf("an app with no restorable copy must be REFUSED (no_backup), got %v", err)
|
||||
}
|
||||
if want := fmt.Sprintf(MsgUpdateNoBackupFmt, "nextcloud"); ref.Message != want {
|
||||
t.Errorf("message = %q", ref.Message)
|
||||
}
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
if st, _ := m.GetStack("nextcloud"); st.Updating {
|
||||
t.Error("a refused update must not set Updating")
|
||||
}
|
||||
if pinOf(t, dir) != "nextcloud:31.0.14-apache" || len(c.list()) != 0 {
|
||||
t.Error("a refused update must move nothing")
|
||||
}
|
||||
// A copy that exists but was never PROVEN is not a copy (R-101).
|
||||
g.rp = UpdateRestorePoint{Restorable: true, Proven: false}
|
||||
if ref := m.UpdatePreflight("nextcloud"); ref == nil || ref.Reason != "no_backup" {
|
||||
t.Errorf("an unproven copy must refuse too, got %v", ref)
|
||||
}
|
||||
}
|
||||
|
||||
// ── D: the cheap refusals, each one ─────────────────────────────────────────────────────────────────
|
||||
|
||||
func TestSlice4_D_CheapRefusals(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
setup func(m *Manager, g *fakeGuards, dir string)
|
||||
reason string
|
||||
msg string
|
||||
}{
|
||||
{"held", func(m *Manager, g *fakeGuards, _ string) { g.held, g.holdWhy = true, "THE HOLD TEXT" }, "held", "THE HOLD TEXT"},
|
||||
{"busy", func(m *Manager, g *fakeGuards, _ string) { g.busy = true }, "busy", MsgUpdateBusy},
|
||||
{"already updating", func(m *Manager, _ *fakeGuards, _ string) { m.stacks["nextcloud"].Updating = true }, "updating", fmt.Sprintf(MsgUpdateAlreadyFmt, "nextcloud")},
|
||||
{"deploying", func(m *Manager, _ *fakeGuards, _ string) { m.stacks["nextcloud"].Deploying = true }, "deploying", fmt.Sprintf(MsgUpdateDeployingFmt, "nextcloud")},
|
||||
{"memory", func(m *Manager, _ *fakeGuards, _ string) {
|
||||
catDir := filepath.Join(m.cfg.Paths.DataDir, "catalog-cache", "templates", "nextcloud")
|
||||
if err := os.WriteFile(filepath.Join(catDir, ".felhom.yml"), []byte("resources:\n mem_request: 900M\n"), 0o644); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
m.updateMemoryFn = func(newReq, _, _, _ int) (string, string) {
|
||||
return fmt.Sprintf("Nincs elég memória (%d MB)", newReq), ""
|
||||
}
|
||||
}, "memory", "Nincs elég memória (900 MB)"},
|
||||
{"disk", func(m *Manager, _ *fakeGuards, _ string) {
|
||||
m.updateDiskFreeFn = func() (float64, bool) { return 1.0, true }
|
||||
}, "disk", fmt.Sprintf(MsgUpdateDiskFmt, 1.0, updateDiskFloorGiB)},
|
||||
{"guards unwired", func(m *Manager, _ *fakeGuards, _ string) { m.updateGuards = nil }, "guards_unwired", MsgUpdateNoGuards},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
m, dir, g, c := newSlice4Manager(t)
|
||||
tc.setup(m, g, dir)
|
||||
ref := m.UpdatePreflight("nextcloud")
|
||||
if ref == nil || ref.Reason != tc.reason {
|
||||
t.Fatalf("want refusal %q, got %+v", tc.reason, ref)
|
||||
}
|
||||
if ref.Message != tc.msg {
|
||||
t.Errorf("message = %q, want %q", ref.Message, tc.msg)
|
||||
}
|
||||
if err := m.StartGuardedUpdate("nextcloud"); err == nil {
|
||||
t.Fatal("StartGuardedUpdate must refuse the same")
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
if len(c.list()) != 0 || pinOf(t, dir) != "nextcloud:31.0.14-apache" {
|
||||
t.Errorf("a cheap refusal reached the act: compose=%v pin=%s", c.list(), pinOf(t, dir))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// ── E: the pull fails ────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
func TestSlice4_E_PullFailurePutsThePinBack(t *testing.T) {
|
||||
m, dir, g, c := newSlice4Manager(t)
|
||||
c.fail["pull"] = errors.New("exit code 1\nstderr: manifest unknown")
|
||||
if err := m.StartGuardedUpdate("nextcloud"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st := waitUpdateDone(t, m, "nextcloud")
|
||||
if st.UpdateError != MsgUpdatePullFailed {
|
||||
t.Errorf("UpdateError = %q, want the Hungarian sentence and never raw stderr", st.UpdateError)
|
||||
}
|
||||
if strings.Contains(st.UpdateError, "manifest unknown") {
|
||||
t.Error("raw Docker stderr leaked into the customer sentence")
|
||||
}
|
||||
if got := pinOf(t, dir); got != "nextcloud:31.0.14-apache" {
|
||||
t.Errorf("the pin must be PUT BACK after a failed pull, got %q", got)
|
||||
}
|
||||
if got := fileBody(t, filepath.Join(dir, "docker-compose.yml")); got != pinTplOld {
|
||||
t.Errorf("the live file must be re-rendered to the old version:\n%s", got)
|
||||
}
|
||||
if got := fileBody(t, AppliedComposePath(dir)); got != pinTplOld {
|
||||
t.Errorf("the stored definition must be the old one again:\n%s", got)
|
||||
}
|
||||
if got := strings.Join(c.list(), " | "); got != "pull" {
|
||||
t.Errorf("after a failed pull nothing else runs; compose calls = %q", got)
|
||||
}
|
||||
for _, call := range g.callList() {
|
||||
if call == "HoldAfterFailedUpdate" {
|
||||
t.Error("a failed PULL ran nothing and must not hold the app")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── F: the new version does not come up ─────────────────────────────────────────────────────────
|
||||
|
||||
// COMPANION RED-PROOF 3 (REPORT.md): remove the HoldAfterFailedUpdate call from failAndHold. This test
|
||||
// then fails: no hold, and the customer is not told the route back.
|
||||
func TestSlice4_F_HealthFailureHoldsTheAppAndKeepsTheNewPin(t *testing.T) {
|
||||
m, dir, g, c := newSlice4Manager(t)
|
||||
m.updateHealthFn = func(context.Context, string, time.Duration) (bool, string) { return false, "crash loop" }
|
||||
if err := m.StartGuardedUpdate("nextcloud"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st := waitUpdateDone(t, m, "nextcloud")
|
||||
if st.UpdatePhase != UpdatePhaseFailed {
|
||||
t.Errorf("phase = %q", st.UpdatePhase)
|
||||
}
|
||||
held, _ := g.HoldFor("nextcloud")
|
||||
if !held {
|
||||
t.Fatal("an app that did not come up must be HELD")
|
||||
}
|
||||
if !g.holdProvenAt.Equal(g.rp.ProvenAt) {
|
||||
t.Errorf("the hold must name the PROVEN copy date %s, got %s", g.rp.ProvenAt, g.holdProvenAt)
|
||||
}
|
||||
if st.UpdateError != "HELD-SENTENCE" {
|
||||
t.Errorf("the page must carry the hold's own sentence, got %q", st.UpdateError)
|
||||
}
|
||||
if st.HoldReason != "HELD-SENTENCE" {
|
||||
t.Errorf("GetStack must carry the hold text, got %q", st.HoldReason)
|
||||
}
|
||||
if got := pinOf(t, dir); got != "nextcloud:34.0.1-apache" {
|
||||
t.Errorf("the pin must STAY on the new version (its migration may have run), got %q", got)
|
||||
}
|
||||
if got := strings.Join(c.list(), " | "); got != "pull | up -d --remove-orphans | down" {
|
||||
t.Errorf("the failed app must be stopped; compose calls = %q", got)
|
||||
}
|
||||
if journalExists(m) {
|
||||
t.Error("the journal is cleared once the hold (the durable record) is written")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlice4_F_UnsavedHoldSaysSo(t *testing.T) {
|
||||
m, _, g, _ := newSlice4Manager(t)
|
||||
m.updateHealthFn = func(context.Context, string, time.Duration) (bool, string) { return false, "crash loop" }
|
||||
g.holdErr = errors.New("settings.json read-only")
|
||||
if err := m.StartGuardedUpdate("nextcloud"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if st := waitUpdateDone(t, m, "nextcloud"); st.UpdateError != MsgUpdateHoldUnsaved {
|
||||
t.Errorf("an unrecorded hold must be disclosed, got %q", st.UpdateError)
|
||||
}
|
||||
}
|
||||
|
||||
// ── G: the controller restarts mid-update ────────────────────────────────────────────────────────
|
||||
|
||||
func writeTestJournal(t *testing.T, m *Manager, name string, e updateJournalEntry) {
|
||||
t.Helper()
|
||||
if err := m.writeUpdateJournal(updateJournal{Updates: map[string]updateJournalEntry{name: e}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// simulateAdvanced puts the stack in the state a crash AFTER the pin moved would leave: pin, live file
|
||||
// and stored definition all new, the pre-update copy on disk.
|
||||
func simulateAdvanced(t *testing.T, m *Manager, dir string) updateJournalEntry {
|
||||
t.Helper()
|
||||
mustWrite(t, filepath.Join(dir, preUpdateComposeFile), pinTplOld)
|
||||
mustWrite(t, filepath.Join(dir, preUpdateAppliedFile), pinTplOld)
|
||||
if err := m.advancePinToCatalog("nextcloud", dir); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return updateJournalEntry{
|
||||
StartedAt: slice4T0, PrevPin: map[string]string{"web": "nextcloud:31.0.14-apache"},
|
||||
PrevCompose: filepath.Join(dir, preUpdateComposeFile), PrevApplied: filepath.Join(dir, preUpdateAppliedFile),
|
||||
ProvenCopyAt: slice4T0.Add(-time.Hour).Format(time.RFC3339),
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlice4_G_InterruptedBeforeUpIsPutBack(t *testing.T) {
|
||||
m, dir, _, c := newSlice4Manager(t)
|
||||
e := simulateAdvanced(t, m, dir)
|
||||
e.Phase = UpdatePhasePulling
|
||||
writeTestJournal(t, m, "nextcloud", e)
|
||||
|
||||
if resumed := m.RecoverUpdates(); len(resumed) != 0 {
|
||||
t.Fatalf("an update interrupted before `up` is not resumed, got %v", resumed)
|
||||
}
|
||||
if got := pinOf(t, dir); got != "nextcloud:31.0.14-apache" {
|
||||
t.Errorf("the pin must be put back, got %q", got)
|
||||
}
|
||||
if got := fileBody(t, filepath.Join(dir, "docker-compose.yml")); got != pinTplOld {
|
||||
t.Errorf("the live file must be the old definition again:\n%s", got)
|
||||
}
|
||||
st, _ := m.GetStack("nextcloud")
|
||||
if st.Updating || st.UpdateError != MsgUpdateInterrupted {
|
||||
t.Errorf("updating=%v err=%q", st.Updating, st.UpdateError)
|
||||
}
|
||||
if journalExists(m) || len(c.list()) != 0 {
|
||||
t.Error("recovery of a pre-up interruption runs nothing and clears the journal")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlice4_G_InterruptedBeforeThePinIsDroppedUntouched(t *testing.T) {
|
||||
m, dir, _, _ := newSlice4Manager(t)
|
||||
writeTestJournal(t, m, "nextcloud", updateJournalEntry{Phase: UpdatePhaseSafetyDump, StartedAt: slice4T0})
|
||||
m.RecoverUpdates()
|
||||
if pinOf(t, dir) != "nextcloud:31.0.14-apache" || journalExists(m) {
|
||||
t.Error("an update interrupted before the pin moved must leave the pin and clear the journal")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlice4_G_InterruptedAfterUpResumesTheHealthWait(t *testing.T) {
|
||||
m, dir, g, c := newSlice4Manager(t)
|
||||
e := simulateAdvanced(t, m, dir)
|
||||
e.Phase = UpdatePhaseVerifying
|
||||
writeTestJournal(t, m, "nextcloud", e)
|
||||
guards := m.updateGuards
|
||||
m.updateGuards = nil // at RecoverUpdates time the backup side is NOT wired yet (main.go order)
|
||||
|
||||
resumed := m.RecoverUpdates()
|
||||
if len(resumed) != 1 || !m.IsUpdating("nextcloud") || !m.UpdatingStacks()["nextcloud"] {
|
||||
t.Fatalf("an update interrupted after `up` must be marked Updating for the boot sweep; resumed=%v", resumed)
|
||||
}
|
||||
if got := pinOf(t, dir); got != "nextcloud:34.0.1-apache" {
|
||||
t.Errorf("after `up` the pin is NOT put back — something may have run; got %q", got)
|
||||
}
|
||||
|
||||
m.updateGuards = guards
|
||||
m.updateHealthFn = func(context.Context, string, time.Duration) (bool, string) { return false, "still broken" }
|
||||
if n := m.ResumeInterruptedUpdates(context.Background()); n != 1 {
|
||||
t.Fatalf("resumed %d, want 1", n)
|
||||
}
|
||||
st := waitUpdateDone(t, m, "nextcloud")
|
||||
if held, _ := g.HoldFor("nextcloud"); !held || st.UpdatePhase != UpdatePhaseFailed {
|
||||
t.Errorf("a resumed update that is still unhealthy must end HELD; held=%v phase=%q", held, st.UpdatePhase)
|
||||
}
|
||||
if got := strings.Join(c.list(), " | "); got != "up -d --remove-orphans | down" {
|
||||
t.Errorf("resumption re-runs `up` then stops the failed app; compose calls = %q", got)
|
||||
}
|
||||
if !g.holdProvenAt.Equal(slice4T0.Add(-time.Hour)) {
|
||||
t.Errorf("the resumed hold must name the journaled proven copy date, got %s", g.holdProvenAt)
|
||||
}
|
||||
}
|
||||
|
||||
// ── the page reads the hold from the ONE store ──────────────────────────────────────────────────
|
||||
|
||||
func TestSlice4_GetStacksCarriesTheHoldText(t *testing.T) {
|
||||
m, _, g, _ := newSlice4Manager(t)
|
||||
g.held, g.holdWhy = true, "HOLD"
|
||||
for _, st := range m.GetStacks() {
|
||||
if st.Name == "nextcloud" && st.HoldReason != "HOLD" {
|
||||
t.Errorf("GetStacks HoldReason = %q", st.HoldReason)
|
||||
}
|
||||
}
|
||||
g.held = false
|
||||
if st, _ := m.GetStack("nextcloud"); st.HoldReason != "" {
|
||||
t.Errorf("a lifted hold must disappear on the next read, got %q", st.HoldReason)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlice4_PhaseLabelsAreTheSpecifiedCopy(t *testing.T) {
|
||||
want := map[string]string{
|
||||
UpdatePhaseChecking: "Ellenőrzés…",
|
||||
UpdatePhaseBackingUp: "Biztonsági mentés készül a frissítés előtt…",
|
||||
UpdatePhaseSafetyDump: "Adatbázis pillanatkép…",
|
||||
UpdatePhasePulling: "Új verzió letöltése…",
|
||||
UpdatePhaseStarting: "Indítás az új verzióval…",
|
||||
UpdatePhaseVerifying: "Működés ellenőrzése…",
|
||||
UpdatePhaseDone: "Frissítve",
|
||||
}
|
||||
for p, l := range want {
|
||||
if got := UpdatePhaseLabel(p); got != l {
|
||||
t.Errorf("label(%s) = %q, want %q", p, got, l)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1429,12 +1429,15 @@ func (s *Server) buildAppBackupRows(status *backup.FullBackupStatus) []AppBackup
|
||||
// a disconnected destination, a pre-v2 layout — so an offer is never rendered for a
|
||||
// copy the action would refuse. On any refusal the action is simply not offered; the
|
||||
// row keeps rendering everything else it already showed.
|
||||
if cov, covErr := s.backupMgr.Tier2RestoreCoverage(app.StackName); covErr == nil {
|
||||
row.Tier2UnitRestorable = cov.CanRestoreUnit()
|
||||
//
|
||||
// Slice 4: the computation lives in backup.Tier2UnitRestorePoint, because the guarded
|
||||
// update asks the same question and a second copy of a predicate drifts (R-203).
|
||||
if rp, rpErr := s.backupMgr.Tier2UnitRestorePoint(app.StackName); rpErr == nil {
|
||||
row.Tier2UnitRestorable = rp.Restorable
|
||||
// R-403: the UNIT action names the PACKAGE's date, not the run's. After a
|
||||
// preserved leg those are different dates and the run's is the flattering one.
|
||||
pkgDate, stale := cov.UnitRestoreDate()
|
||||
row.Tier2CopyDate, row.Tier2CopyDateProven = pkgDate, cov.CopyLastSuccess != ""
|
||||
pkgDate, stale := rp.CopyDate, rp.PackagePreserved
|
||||
row.Tier2CopyDate, row.Tier2CopyDateProven = pkgDate, rp.CopyDateProven
|
||||
row.Tier2UnitConfirm = tier2UnitConfirmWithStaleness(pkgDate, row.Tier2CopyDateProven, stale)
|
||||
if stale && pkgDate != "" {
|
||||
row.Tier2UnitStaleNotice = fmt.Sprintf(tier2UnitStaleNoticeFmt, fmtRFC3339Local(pkgDate))
|
||||
|
||||
@@ -219,6 +219,12 @@ func (s *Server) stopAppsOnPath(storagePath string) []string {
|
||||
// restartStacks starts each named stack (the gate-stopped set on drive return). Best-effort per app.
|
||||
func (s *Server) restartStacks(names []string) {
|
||||
for _, name := range names {
|
||||
// Slice 4 / R-379: the drive-return gate starts apps UNATTENDED, so it must honour a hold
|
||||
// exactly as the customer's button does. Until v0.237.0 it did not — a held app whose drive
|
||||
// blinked would have been started again. "A hold that only one path honours is not a hold."
|
||||
if s.appHeld(name) {
|
||||
continue
|
||||
}
|
||||
if err := s.stackMgr.StartStack(name); err != nil {
|
||||
s.logger.Printf("[WARN] [gate] restart %s: %v", name, err)
|
||||
}
|
||||
@@ -454,6 +460,9 @@ func (s *Server) processGuestBootChange() {
|
||||
}
|
||||
recreate := func(bs bootStack) {
|
||||
s.logger.Printf("[INFO] [gate] boot %s: live bind confirmed — recreating drive-backed app %s (state=%s) onto %s", resp.GuestBootID, bs.name, bs.state, bs.hdd)
|
||||
if s.appHeld(bs.name) {
|
||||
return
|
||||
}
|
||||
_ = s.stackMgr.StopStack(bs.name)
|
||||
if serr := s.stackMgr.StartStack(bs.name); serr != nil {
|
||||
s.logger.Printf("[WARN] [gate] boot recreate %s: %v", bs.name, serr)
|
||||
@@ -694,3 +703,16 @@ func (s *Server) notifyDriveReturned(path string, isTarget map[string]bool) {
|
||||
}
|
||||
s.notifier.NotifyStorageReconnected(label)
|
||||
}
|
||||
|
||||
// appHeld reports whether an app carries a hold (failed update or failed restore) and logs the skip.
|
||||
// Nil-safe: no backup manager means no hold store, so nothing is held.
|
||||
func (s *Server) appHeld(name string) bool {
|
||||
if s.backupMgr == nil {
|
||||
return false
|
||||
}
|
||||
held, _ := s.backupMgr.RestoreHoldFor(name)
|
||||
if held {
|
||||
s.logger.Printf("[WARN] [gate] NOT starting %s — the app is HELD (a failed update or restore); a person releases it", name)
|
||||
}
|
||||
return held
|
||||
}
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// Slice 4 — the predicate moved from this package into backup.Tier2UnitRestorePoint. The page must
|
||||
// render IDENTICALLY: the row's four unit-restore fields are compared against the coverage computed
|
||||
// the old way (the inline expression that used to live in buildAppBackupRows).
|
||||
func TestSlice4_BackupRowUnitFieldsAreUnchangedByTheExtraction(t *testing.T) {
|
||||
s, sett, m := newOffboxWebServer(t)
|
||||
dest := t.TempDir()
|
||||
if err := sett.AddStoragePath(settings.StoragePath{Path: dest, Label: "flash"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := sett.SetCrossDriveConfig("app", &settings.CrossDriveBackup{
|
||||
Enabled: true, Method: "rsync", DestinationPath: dest,
|
||||
LastRun: "2026-09-13T01:30:00Z", LastStatus: "ok", LastSuccess: "2026-09-13T01:30:00Z", SuccessTracked: true,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
unit := filepath.Join(dest, "backups", "secondary", "app", "recovery-unit")
|
||||
if err := os.MkdirAll(unit, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(unit, "manifest.json"), []byte(`{"schema_version":2,"app_name":"app","created_at":"2026-09-12T02:15:29Z"}`), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dest, "backups", "secondary", "app", ".felhom-tier2-layout"), []byte("2"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
cov, err := m.Tier2RestoreCoverage("app")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
wantDate, wantStale := cov.UnitRestoreDate()
|
||||
rows := s.buildAppBackupRows(&backup.FullBackupStatus{AppDataInfo: []backup.AppBackupInfo{{StackName: "app", DisplayName: "App"}}})
|
||||
row := findRow(rows, "app")
|
||||
if row == nil {
|
||||
t.Fatal("no row")
|
||||
}
|
||||
if !cov.CanRestoreUnit() {
|
||||
t.Fatal("fixture: the copy must hold an openable unit")
|
||||
}
|
||||
if row.Tier2UnitRestorable != cov.CanRestoreUnit() || row.Tier2CopyDate != wantDate ||
|
||||
row.Tier2CopyDateProven != (cov.CopyLastSuccess != "") ||
|
||||
row.Tier2UnitConfirm != tier2UnitConfirmWithStaleness(wantDate, cov.CopyLastSuccess != "", wantStale) {
|
||||
t.Errorf("the row changed: restorable=%v date=%q proven=%v confirm=%q", row.Tier2UnitRestorable, row.Tier2CopyDate, row.Tier2CopyDateProven, row.Tier2UnitConfirm)
|
||||
}
|
||||
}
|
||||
|
||||
// The drive-return gate starts apps UNATTENDED. A held app must be skipped.
|
||||
//
|
||||
// COMPANION RED-PROOF (REPORT.md): delete the appHeld check from restartStacks. The server has NO stack
|
||||
// manager here on purpose, so the unguarded StartStack call panics and this test fails.
|
||||
func TestSlice4_DriveReturnGateSkipsAHeldApp(t *testing.T) {
|
||||
s, _, m := newOffboxWebServer(t)
|
||||
if err := m.HoldAfterFailedUpdate("held", time.Now(), time.Now()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
t.Fatalf("the drive-return gate tried to START a held app: %v", r)
|
||||
}
|
||||
}()
|
||||
s.restartStacks([]string{"held"})
|
||||
}
|
||||
Reference in New Issue
Block a user