clear_hold (D64 = B) + the box command through the running controller (R-1343), R-1340, R-1341 — tests red first; commit only
gates / gates (push) Successful in 31s

- clear_hold: a fifth verb in the closed operator-action list; lifts a hold of any reason in the RUNNING
  controller (memory + settings.json together), forgets the update's held flag, drops a waiting put-back
  from the app-stop marker (copy kept). Refused on a bad name or no hold. No new event type.
- --clear-restore-hold asks the running controller over <data dir>/ctl/hold.sock (unix, 0600, peer uid
  checked, no TCP); edits settings.json only when no controller answers, and says so (in-guest restart).
- R-1340: the operator's HELD STOPPED mail is built from the hold record (restore kind, mixed cause,
  failed put-back step, database replay or not, kept copy) — never the option-C database story by default.
- R-1341: the OOM WARN line once per new kill (container, start time, kill count); event unchanged.
This commit is contained in:
admin committed 2026-10-29 12:49:46 +01:00
1 parent c7bb776e00
commit 5b3b541602
25 files changed
+1809 -83

No files matched your search

+46
View File
@@ -1,3 +1,49 @@
## Unreleased (2026-10-29, afternoon) — clear_hold (D64 = B), the box command through the running controller (R-1343), R-1340, R-1341; commit only, rides the next release
**MinAgent: 0.158.1** (unchanged). The next release also carries R-1342 (`652a690`, already on main, the entry below).
**No new event type** — the hub records the operator action's result as its own `operator_action` event.
- **`clear_hold` — a fifth verb in the CLOSED operator-action list (operator ruling D64 = B, 2026-10-29 12:10)**
(`internal/report/opactions.go`, wired in `cmd/controller/opactions_wiring.go`; arg = the app's stack name, letters,
digits, `-`, `_`, ≤64 — checked before any handler). It lifts the app's hold of ANY reason in the RUNNING controller
(`backup.AppStopGuard.ClearHoldByOperator` → `settings.ClearRestoreHold`: memory and `settings.json` together), so no
later save can write it back. It starts nothing. Refused, nothing changed: a bad name, an app with no hold. One log
line names who: „hold CLEARED by the operator via the hub, action #<id>". Every reader of the hold (page, start gate,
API, `RestoreHoldFor`, hold reporter) reads the same settings record; the ONE held flag cached elsewhere —
`stacks.Manager`'s „the last update ended held" (it made the crash-loop observer and the nightly update leg skip the
app) — is forgotten too (`ForgetUpdateHeld`). A `putback_pending` hold is also dropped from the app-stop marker, or the
next start with the drive bound would pour the old copy over what the app wrote after the clear; the copy stays on
disk. The hold reporter needs no change: its next tick (≤1 min) sends `app_hold_cleared`.
Tests: `TestOpActions_ClosedList` (updated deliberately), `TestOpActions_ClearHoldBadArgumentIsRefusedAndCallsNothing`,
`TestOpActions_ClearHoldDoneRefusedFailed`, `TestD64_ClearHoldViaTheHub`, `TestD64_ClearHoldRefusals`,
`TestD64_ClearHoldStaysGoneAfterALaterSave`, `TestD64_ClearPutBackPendingDropsTheMarkerEntry`,
`TestD64_TheHoldReporterTellsTheHubAfterTheClear`, `TestD64_ForgetUpdateHeld`, `TestD64_MainWiresTheNewPieces`.
- **R-1343 — the box command `--clear-restore-hold <app>` goes through the running controller** (`cmd/controller/holdclear.go`):
a unix socket `<data dir>/ctl/hold.sock`, 0600 in a 0700 dir, peer uid checked (SO_PEERCRED), nothing on TCP — not
reachable through traefik, the LAN or an app container. Output: „Cleared by the RUNNING controller … no restart is
needed". Only when no controller answers (no socket, or a stale one) does it edit `settings.json`, and it says so,
naming the restart INSIDE the guest (`pct exec <vmid> -- systemctl restart felhom-controller-bootstrap.service`). The
flag's help names the exact guest command. `--restore-holds` (the list) is unchanged.
Tests: `TestR1343_TheBoxCommandClearsThroughTheRunningController` (red with the socket skipped: „the hold came back
after a later save of the running controller" — R-1343 reproduced), `TestR1343_TheBoxCommandFallsBackWhenNoControllerRuns`,
`TestHoldCtl_NotReachableFromOutside`, `TestHoldCtl_UnknownRequestIsRefused`, `TestHoldCtl_ANonSocketFileIsNotReplaced`.
- **R-1340 — the operator's „HELD STOPPED" mail names the real cause** (`cmd/controller/holdmail.go`
`heldStoppedOperatorText`, built from the hold record). Hold writers now record `RestoreKind` (off-site / local),
`MixedCause` (`definition_write` / `put_back` / `interrupted_put_back`), `FailedStep` (the put-back step that failed,
with its error), `DBReplay`, and the kept copy's folder (`settings.RestoreHold`, all `omitempty`). The mail never says
„database" unless the failure was a database replay, never „off-site" for a local restore; a pre-R-1340 mixed hold
names only what its record says. Legs unchanged (`restore-hold`, `restore-hold-mixed`); a local restore's RunKind is
`restore`. Tests (red with the old sentence/record): `TestR1340_UnhealthyStop`, `_UpdateFailed`,
`_RestoreFailedWithADatabase`, `_PutBackWithoutADatabase` (the measured Radicale case), `_PutBackAfterADatabaseReplay`,
`_DefinitionWrite`, `_LocalRestore`, `_PutBackPending`, `_InterruptedPutBack`, `_LegacyMixedHoldAssumesNothing`,
`_NotifyReadsTheHoldRecord`; writer side `TestR1340_AVolumeFailureWhosePutBackStopFailsRecordsTheStepAndNoDatabase`,
`_ADatabaseReplayFailureRecordsTheDatabase`, `_ADefinitionWriteFailureRecordsItsCause`, `_ALocalRestoreRecordsItsKind`,
`_AnInterruptedPutBackRecordsItsCause`. `TestEventMessageWireTextIsFrozen` is untouched (this text is passed through).
- **R-1341 — the OOM line once per NEW kill** (`cmd/controller/oomlog.go` `oomLogGate`, key = container + start time +
kill count): a second scan of the same kill logs nothing; a new kill or a restarted container logs again. The
`app_oom` event call is unchanged. Tests: `TestR1341_ASecondScanOfTheSameKillLogsNothing`, `_ANewKillLogsAgain`,
`_ARestartedContainerLogsAgain`, `_ContainersAreJudgedApart`.
## Unreleased (2026-10-29, after v0.347.0) — R-1342: the restore copy is safe during the stable-start watch; a removed app does not alarm; commit only, rides the next release
**MinAgent: 0.158.1** (unchanged). Found live on Tester 1 at 10:39 CET: a false `backup_run_failures` mail („Gokapi did
+3 -2
View File
@@ -82,12 +82,13 @@
| `restoreOpBlocked` | controller/internal/web/restore_wizard.go | `() (msg string, blocked bool)` | THE refusal gate before starting ANY restore | **Use this, never a bare `IsRunning()`.** It reads BOTH flags: `RestoreStatus().Running` (set synchronously by `BeginRestoreOp`, true for the whole off-box restore) and `IsRunning()` (the concurrency flag, the only one the nightly backup holds). **R-351: all seven handlers read only `IsRunning()`, which the goroutine acquires AFTER the handler returns — a second press started a second run and was told „…elindult".** Returns the Hungarian refusal, which names the running app and a route | **R-360 (v0.226.0): `offboxVerifyCopyDeleteHandler` was the LAST holdout and its doc comment claimed it already did this — the sentence is why nobody looked. `IsRunning()` is FALSE for the whole of a verification restore, so the copy a restore was writing into could be deleted from the UI (observed live 2026-08-21 22:35). No app-name comparison: refusing during ANY restore is stronger and uniform.**
| `Server.beginRestoreOp` → `backup.TryBeginRestoreOp` / `RestoreBusy` (R-1256 H6) | controller/internal/web/restore_wizard.go; controller/internal/backup/opstatus.go | `(op, stack) (msg string, ok bool)` / `(op, stack) (busy, mirrorApp string)` | THE way a restore op begins: the check for every other operation (a restore op, the backup run, a second-drive copy) and the record in ONE step under `m.mu` — the mirror asks under the same lock (`beginTier2Run`) | **Never call `BeginRestoreOp` from a handler** (`TestR1256_H6_NoHandlerBeginsARestoreUnchecked`): `restoreOpBlocked` asked early can be stale by the time the form is read, and a mirror that began in between ran beside the restore. `restoreOpBlocked` stays as the EARLY refusal; the kept loads begin inside `LoadKeptApp` / `LoadKeptOffsite` and return `ErrRestoreBusy` |
| `Manager.RestoreFromRecoveryUnitAt` / `RestoreFromRecoveryUnit` + `UnitRestoreResult` (R-353 v0.226.0; R-102 v0.229.0) | controller/internal/backup/restore_unit.go | `(stack, unitDir string) (UnitRestoreResult, error)` / `(stack string) (…)` | THE local recovery-unit restore, and the facts its surface must state | **Returns a RESULT, not just an error** — volumes replayed, DBs replayed, and what the manifest LISTED. The Manifest* counts are load-bearing: zero-replayed has two causes (the backup held no data / the backup listed data that did not come back) and they are opposite news. Pair it with `unitRestoreOutcomeMsg`; do NOT write a new sentence. **A claim about the APP is forbidden on this path** — it has no `SafetyDump` discriminator, unlike the off-site twin (CONTEXT.md ruling, 07-backup-architecture §6.3). `restoreDockerVolumes` now returns `(int, error)`; `restoreDockerVolumesFrom` is unchanged and still the shared implementation. **R-102: `…At` holds the whole body and takes the unit DIRECTORY; the one-argument form is the thin caller naming the primary unit.** THE SOURCE MOVES; THE DESTINATION DOES NOT — `GetAppDrivePath` still resolves where the data lands. The R-47 mutation order, the unit-over-guest secret precedence, the fail-closed data-key gate and the no-unit `RestoreApp` fallback are all pinned and unchanged. The volume leg goes through the R-354 `volumeReplayFrom` seam so the source directory is assertable without Docker |
| `Manager.putBackAfterFailedRestore` + `liveVolumeCopy` (R-893 option A, v0.336.0) | controller/internal/backup/offbox_restore_undo.go | `(stack, cause, live, copy, volumesMoved, filesPlaced, restart, holdKey) error` | THE restore undo — off-site (v0.336.0) and, since D22 (unreleased), the LOCAL unit restore (`RestoreFromRecoveryUnitAtWith`, so also the second drive's and a kept load): the live volumes it will replace, measured before the stop (`planLiveVolumeCopy`), copied right after it (`copyLiveVolumes`), poured back on any failure | One copy per restore under `<ns>/backups/restore-undo/<app>/` (no capture set reads `backups/`); the put-back uses the restore's own `volumeReplayFrom`/`restoreDockerVolumesFrom`, so there is one volume-replay implementation. A put-back that fails sets `keep` and HOLDS (option C) — never start on volumes that are not known to be back. Seams for tests without Docker: `undoVolumeProbeFn`, `undoVolumeTarFn`, `undoVolumeRemoveFn`; operator notice `SetRestorePutBackNotify` (leg `restore-put-back`). Pinned by `r893_putback_test.go` and `r1177_local_putback_test.go` |
| `Manager.putBackAfterFailedRestore` + `liveVolumeCopy` (R-893 option A, v0.336.0) | controller/internal/backup/offbox_restore_undo.go | `(stack, cause, live, copy, volumesMoved, filesPlaced, restart, holdKey, restoreKind) error` (R-1340: `restoreKind` = `settings.RestoreKindOffsite`/`Local`; a failed put-back records its step in the hold's `MixedCause`/`FailedStep`/`DBReplay`) | THE restore undo — off-site (v0.336.0) and, since D22 (unreleased), the LOCAL unit restore (`RestoreFromRecoveryUnitAtWith`, so also the second drive's and a kept load): the live volumes it will replace, measured before the stop (`planLiveVolumeCopy`), copied right after it (`copyLiveVolumes`), poured back on any failure | One copy per restore under `<ns>/backups/restore-undo/<app>/` (no capture set reads `backups/`); the put-back uses the restore's own `volumeReplayFrom`/`restoreDockerVolumesFrom`, so there is one volume-replay implementation. A put-back that fails sets `keep` and HOLDS (option C) — never start on volumes that are not known to be back. Seams for tests without Docker: `undoVolumeProbeFn`, `undoVolumeTarFn`, `undoVolumeRemoveFn`; operator notice `SetRestorePutBackNotify` (leg `restore-put-back`). Pinned by `r893_putback_test.go` and `r1177_local_putback_test.go` |
| `Manager.SweepUnusedRestoreCopies` (R-1256 H8) | controller/internal/backup/restore_copy_sweep.go | `(now) []string` | THE only AUTOMATIC remover of a restore copy nothing names any more (`restore-copy-sweep`, every 6 h); the household's „Törlés most" is `DeleteRestoreCopy` (D62, restore_copies.go — same „needed" rule via `restoreCopyNeededFor`, check + removal under `m.mu`) | Removes a copy only when ALL hold: older than 30 days (D62), the app has no restore hold of ANY kind, the app-stop marker is readable and does not name the app, no restore op runs. It answers „no" when it cannot answer — a held or waiting copy is support's way back |
| `CrashPutBack` + `Manager.persistUndoForCrash` + `AppStopGuard.SetUndo` (D22 / R-1195, v0.338.0) | controller/internal/backup/restore_crash_putback.go | `NewCrashPutBack(stop, writeDef, logger)`; `PutBackInterruptedRestore(stack, undoDir) error` | A restore cut by a power loss is put back at start: the copy (volumes + `definition/` + `undo.json`) is recorded in the app-stop marker once whole; `Recover` pours it back BEFORE the restart | Call `persistUndoForCrash` right after `copyLiveVolumes` in any new restore path. `Recover` runs before the backup manager exists, so the put-back is wired in main.go with `stackMgr.StopStack` and the stack adapter's definition writer. A failed put-back = a FAILED recovery (app not started, marker kept). The household reads `note.restore.interrupted_put_back` (`LoadRestoreRecordAfterRecovery`). Pinned by `TestR1195_*` |
| `tier2CaptureSetFull(...) (legs, missing, warns, classified)` + `keptCopyRels` + `dropKeptCopiesUnder` (R-1252, unreleased) | controller/internal/backup/tier2_capture.go, tier2_kept.go | `classified` = the legs come from a VALID backup block | **The second-drive reconcile deletes only under a valid classification**; a rejected or absent block keeps every dest dir; kept copies (D25) are always in the keep set; a mirror deleted by Remove drops its kept records | A new reconcile caller must take `classified` and the kept rels too — never reconcile against the legacy legs. A rejected block is told by `Manager.BackupBlockRejected` (stacks; via the adapter's optional `BackupBlockRejected` method) and signalled once a day (`SetTier2SettingsUnreadableNotify`). Pinned by `r1252_rejected_block_keeps_copies_test.go` |
| `holdreport.Reporter` + `settings.RestoreHold.Why` / `WireReason` (D53, unreleased) | controller/internal/holdreport/holdreport.go · controller/internal/settings/settings.go | `New(store, notifier.PushEvent, logger)`; `Tick(now)` from the scheduler | Tells the hub every hold in place (`app_held`), every hold gone (`app_hold_cleared`), each again daily | **One observer of the hold store — never a send inside a hold writer.** A new hold writer needs nothing here. The hold sentence is `RestoreHold.Why()` (one per reason) — never a second copy. Pinned by `internal/holdreport/holdreport_test.go` |
| `AppStopGuard.ResumePutBack` / `ResumeWaitingPutBacks` + `HoldReasonPutBackPending` (R-1251, unreleased) | controller/internal/backup/appstop_putback_wait.go | `ResumePutBack(stack) bool` (true = nothing owed, the caller may start); `ResumeWaitingPutBacks() []string` (puts back, then starts through the gated starter) | A cut restore's copy on a drive that is not bound is WAITING: Recover writes the restore hold `putback_pending` (copy in `SafetyDump`) and starts nothing; the put-back runs when the drive binds, before any start | **Any new path that starts an app when its drive arrives calls `ResumePutBack` first** (the boot recreate and `restartStacks` do, in web via `Server.resumePutBack`). Every other start path is covered by the hold store it already honours — do not add a second check. A failed put-back (at start or on resume) writes the D20 hold (`restore_mixed`) and drops the app from the marker. Needs `SetHolds` + `SetUndoReachable` before `Recover` (main.go). Pinned by `r1251_putback_waits_test.go`, `TestR1251_DriveStartPathsPutBackBeforeTheyStart`, `TestR1251_MainWiresThePutBackWait` |
| `AppStopGuard.ClearHoldByOperator` + `clearHoldNow` / `listenHoldCtl` / `clearRestoreHoldCLI` (D64 = B, R-1343, unreleased) | controller/internal/backup/hold_clear.go · controller/cmd/controller/holdclear.go | `ClearHoldByOperator(stack) (RestoreHold, error)`; `clearHoldNow(deps, stack, who) (msg, error)` | THE operator's hold clear, any reason — the hub's `clear_hold` and the box command `--clear-restore-hold` both go through `clearHoldNow` in the RUNNING controller | **Never clear a hold from a second process** — the running controller's next save writes it back (R-1343, measured). The box command reaches the running controller over `<data dir>/ctl/hold.sock` (unix, 0600, peer uid checked; never TCP/HTTP). A `putback_pending` clear also drops the app from the app-stop marker (the copy stays). Refusals (`ErrHoldBadStack`, `ErrNotHeld`) change nothing. The update's in-memory held flag goes via `stacks.Manager.ForgetUpdateHeld`. Pinned by `cmd/controller/holdclear_test.go` |
| `Manager.RestoreTier2Unit` + `Tier2Coverage.CanRestoreUnit` / `Tier2CopyDate` (R-102/R-103, v0.229.0) | controller/internal/backup/tier2_restore.go | `(stack string) (UnitRestoreResult, error)` / `() bool` / `() (string, bool)` | THE full restore from the SECOND DRIVE's mirror, and the predicate that gates it | **Do NOT widen `CanRestore()`** — it answers only "can the additive file restore run?" and one predicate answering two questions is R-356. Fail-closed: `tier2UnitIsOpenable` requires a parseable manifest, because a directory is not a package. The single-writer flag is taken INSIDE `RestoreFromRecoveryUnitAt` — a second `acquireRunning()` here would refuse the restore it guards. `Tier2CopyDate` prefers the SUCCESS anchor over the attempt clock (R-101) and reports which it returned |
| `unitRestoreOutcomeMsg` + its three message constants (R-353, v0.226.0) | controller/internal/web/handlers.go | `(app string, res backup.UnitRestoreResult) string` | THE customer sentence for a completed LOCAL restore | Twin of `reconstituteOutcomeMsg`; copy its SHAPE (clauses earned by having done the thing, no filesystem path, base names only), never its text. The constants are named because `r353_unit_outcome_test.go` asserts them verbatim — a silent edit is how an honest message drifts back into a comforting one, which is the documented history of the sentence it replaces |
| `offsiteNoSpaceMsgFmt` + `offsiteSizeUnknownMsg` (R-357, v0.226.0) | controller/internal/backup/offbox_restore.go | two consts | EVERY headroom refusal on the off-site restore surface | **All four gates share these** (prepare, scratch, place, and the destructive reconstitute). A customer meeting one wording on one path and a different one on another has to work out whether it is the same problem. **The reconstitute gate uses NO ×1.1 margin** — it copies a measured tree; `OffboxRestorePrepareFull`'s ×1.1 predicts a download. **Fail closed when either probe reads ≤ 0**: `free < need` with `need == 0` is FALSE, so an unmeasurable input sails through — a gate present and inert |
@@ -456,7 +457,7 @@ Cross-repo edges:
| timeAgo switch body ×2 | controller/internal/web/funcmap.go `timeAgo` vs `timeAgoStr` (identical formatting logic) |
| Budapest timezone loader ×4 | controller/internal/scheduler/scheduler.go `getBudapestLocation` vs controller/internal/web/funcmap.go `getTimezone` vs controller/internal/quiesce/quiesce.go `budapestLocation` vs controller/internal/api/kept_install.go `budapest` (v0.277.0, the kept choice's dates) (v0.168.0 window gate — Budapest wall-clock, kept local to avoid a scheduler↔quiesce import edge) |
| JSON writers ×5, 3 envelope shapes | api `writeJSON`; web `writeDiskJSON`, `jsonResponse`/`jsonError`, `writeDebugJSON` |
| Safe-name validators ×4 | controller/internal/web/validate.go `validStackName`; controller/internal/api/router.go `validStackParam` (same body — api↔web import cycle); controller/internal/backup/offbox.go `isSafeStackName`; controller/internal/appexport/validate.go `ValidateSegment` (strictest) |
| Safe-name validators ×5 | controller/internal/web/validate.go `validStackName`; controller/internal/api/router.go `validStackParam` (same body — api↔web import cycle); controller/internal/backup/offbox.go `isSafeStackName` (exported as `IsValidHoldStack` for the hold commands); controller/internal/report/opactions.go `validHoldStack` (same alphabet as `isSafeStackName` — report cannot import backup; clear_hold's argument check); controller/internal/appexport/validate.go `ValidateSegment` (strictest) |
| DB wait/import ×2 | controller/internal/appbackup/dbdump.go `waitDBReady`/`ImportDump` vs controller/internal/appexport/restore.go `waitForDB`/`importDBDump` |
| qBittorrent Web API login ×2 | controller/internal/mediawire/mediawire.go `qbLogin` (writes the drive wiring) vs controller/internal/upwatch/explain.go `QBittorrent.login` (reads every minute; keeps its session cookie, which mediawire's per-run login does not need) |
| compose exec ×2 | controller/internal/stacks/manager.go `composeExecCustomEnv` vs controller/internal/appexport/restore.go `composeExecEnv` (the latter has ctx+timeout; the former has the userdata belt) |
+15 -1
View File
@@ -1017,7 +1017,8 @@ Each app can define rich metadata in `.felhom.yml`:
the hold is persisted in `Settings.RestoreHolds`, every start path refuses it (customer button,
app-stop `Recover()`, boot sweep, via `driveStartGate` **above** its driveless early return), the
app-stop marker is ended so nothing auto-restarts it, and the row goes red. Cleared with
`--clear-restore-hold <app>`, **which requires a controller restart**. `--single-transaction` on
`--clear-restore-hold <app>` (since 2026-10-29, R-1343: through the running controller, no restart — see
„Operator actions") or the hub's „Clear hold" (`clear_hold`). `--single-transaction` on
the Postgres import is a belt only; MariaDB DDL is not transactional, which is why the rollback
is the fix.
- **Off-site restore of one app: a failed restore puts the app back EXACTLY as it was and starts it (v0.336.0,
@@ -3655,6 +3656,19 @@ Anything else is answered `refused` and calls nothing. Each id runs once per pro
(`operator_action_results: [{id, outcome, message}]`, outcome `done` / `refused` / `failed`). No action deletes data,
starts a countdown or shortens one (`TestOpActions_ClosedList`). The box's log records every action it receives.
- **`clear_hold` (2026-10-29, operator ruling D64 = B; arg = the app's stack name, letters/digits/`-`/`_`, ≤64):** lifts
the app's hold of ANY reason, in the RUNNING controller (`backup.AppStopGuard.ClearHoldByOperator`, the door
`clearHoldNow` in `cmd/controller/holdclear.go`): memory and `settings.json` together, so no later save writes it back
(R-1343). It starts nothing — the household's Start (or a restore) does. A waiting put-back (`putback_pending`) is
dropped from the app-stop marker too; its copy stays on disk. The update's in-memory „held" flag
(`stacks.Manager.ForgetUpdateHeld`) goes with it; the hold reporter's next tick (≤1 min) sends `app_hold_cleared`.
Refused, nothing changed: a bad name, or an app with no hold. No new event type: the hub records the result.
- **The box command `--clear-restore-hold <app>`** (`pct exec <vmid> -- docker exec felhom-controller felhom-controller
--clear-restore-hold <app>`) asks the running controller over a unix socket, `<data dir>/ctl/hold.sock` (0600 in a
0700 dir, peer uid checked, nothing on TCP — not reachable through traefik, the LAN or an app), and says no restart is
needed. Only when no controller answers does it edit `settings.json` itself, and it says so, naming the in-guest
restart. `--restore-holds` lists the holds (read-only).
#### Report Push (`internal/report/`)
Periodic JSON push (default every 15 min) to the central felhom-hub service:
+260
View File
@@ -0,0 +1,260 @@
package main
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net"
"os"
"path/filepath"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
"gitea.dooplex.hu/admin/felhom-controller/internal/report"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// ── D64 = B + R-1343: the operator clears a hold in the RUNNING controller ─────────────────────────────────────────
//
// Two routes reach the same door (clearHoldNow):
// - the hub's „Clear hold" → the operator action clear_hold (internal/report/opactions.go), run in this process;
// - the box command `felhom-controller --clear-restore-hold <app>` (run with `docker exec` inside the guest) → the
// local socket below → the running controller.
// The running controller clears the hold in memory AND saves settings.json at once (backup.ClearHoldByOperator), so
// no later save — the hold reporter's, a hub report's — can write it back (R-1343). The household's „held" sentence
// goes with it: every reader (the page, the start gate, the API, the hold reporter) reads the same settings record.
// The ONE held flag kept elsewhere — stacks.Manager's „the last update ended held" — is forgotten here too.
// holdClearDeps are clearHoldNow's seams (production: holdClearDepsFor).
type holdClearDeps struct {
clear func(stack string) (settings.RestoreHold, error) // backup.AppStopGuard.ClearHoldByOperator
forgetUpdateHeld func(stack string) bool // stacks.Manager.ForgetUpdateHeld (nil = none)
logger *log.Logger
}
// clearHoldNow lifts stack's hold and returns the result sentence (English, operator-facing). who names the person
// and the route for the box's own log, e.g. „the operator via the hub, action #12". A refusal (bad name, no hold) is
// wrapped in report.ErrActionRefused, and nothing was changed.
func clearHoldNow(d holdClearDeps, stack, who string) (string, error) {
if d.clear == nil {
return "", fmt.Errorf("%w: clearing a hold is not available on this box", report.ErrActionRefused)
}
h, err := d.clear(stack)
switch {
case errors.Is(err, backup.ErrHoldBadStack):
return "", fmt.Errorf("%w: %q is not a valid app name — nothing was changed", report.ErrActionRefused, stack)
case errors.Is(err, backup.ErrNotHeld):
return "", fmt.Errorf("%w: %s has no hold — nothing was changed", report.ErrActionRefused, stack)
case err != nil:
return "", fmt.Errorf("clearing the hold on %s failed: %v", stack, err)
}
if d.forgetUpdateHeld != nil && d.forgetUpdateHeld(stack) {
if d.logger != nil {
d.logger.Printf("[INFO] [hold] %s: the update's held state is forgotten with the hold", stack)
}
}
if d.logger != nil {
d.logger.Printf("[WARN] [hold] %s: hold CLEARED by %s — it was %s (set %s); the app is not started by this", stack, who, h.WireReason(), h.At)
}
msg := fmt.Sprintf("the hold on %s is cleared (it was: %s). The app is NOT started by this — the household's Start, or a restore, starts it", stack, h.Why())
if h.Reason == settings.HoldReasonPutBackPending {
dir := h.SafetyDump
if dir == "" {
dir = "(not named in the hold)"
}
msg += ". Its waiting put-back is dropped: the pre-restore copy stays on disk at " + dir + " and is never poured back by itself"
}
return msg, nil
}
// ── the local channel (R-1343) ────────────────────────────────────────────────────────────────────────────────────
//
// WHY A UNIX SOCKET IN THE CONTROLLER'S DATA DIR, mode 0600 in a 0700 directory, and nothing on TCP:
// - traefik, the LAN and every app reach the controller only over TCP (:8080, :8081). A unix socket has no port,
// so none of them can reach it, whatever a proxy header or a forwarded request says;
// - the data dir is the controller's own volume: no app container mounts it, and only root in the guest
// (`pct exec <vmid> -- docker exec felhom-controller …`, the operator gate this project already uses) can enter
// the controller's container to use it;
// - the directory is created 0700 BEFORE the socket is bound, so the moment between bind and chmod is not a window;
// - the peer's uid must equal this process's (SO_PEERCRED, holdctl_linux.go) — a second lock on the same door.
// The customer's HTTP surface is deliberately NOT used: it authenticates as the household, and a household clearing
// its own hold is what the hold exists to prevent. Pinned by TestHoldCtl_NotReachableFromOutside.
const (
holdCtlDir = "ctl"
holdCtlSock = "hold.sock"
// holdCtlOpClear is the one request this channel answers.
holdCtlOpClear = "clear_hold"
// holdCtlMaxRequest bounds what one request may send.
holdCtlMaxRequest = 4096
)
// holdCtlPath is the socket's path under the controller's data dir.
func holdCtlPath(dataDir string) string { return filepath.Join(dataDir, holdCtlDir, holdCtlSock) }
type holdCtlRequest struct {
Op string `json:"op"`
Stack string `json:"stack"`
}
type holdCtlResponse struct {
Outcome string `json:"outcome"` // done | refused | failed (report.Outcome*)
Message string `json:"message"`
}
// listenHoldCtl binds the socket and serves it until the listener is closed (closing it removes the socket file).
// clear is clearHoldNow bound to the running controller's deps.
func listenHoldCtl(path string, clear func(stack, who string) (string, error), logger *log.Logger) (net.Listener, error) {
dir := filepath.Dir(path)
if err := os.MkdirAll(dir, 0o700); err != nil {
return nil, err
}
if err := os.Chmod(dir, 0o700); err != nil {
return nil, err
}
// A socket left by a controller that died is replaced; anything else at that path is refused, never removed.
if fi, err := os.Lstat(path); err == nil {
if fi.Mode()&os.ModeSocket == 0 {
return nil, fmt.Errorf("%s exists and is not a socket — refusing to replace it", path)
}
if err := os.Remove(path); err != nil {
return nil, err
}
}
ln, err := net.Listen("unix", path)
if err != nil {
return nil, err
}
if err := os.Chmod(path, 0o600); err != nil {
ln.Close()
return nil, err
}
go func() {
for {
conn, err := ln.Accept()
if err != nil {
if !errors.Is(err, net.ErrClosed) && logger != nil {
logger.Printf("[WARN] [hold] local channel stopped: %v", err)
}
return
}
go serveHoldCtl(conn, clear, logger)
}
}()
return ln, nil
}
func serveHoldCtl(conn net.Conn, clear func(stack, who string) (string, error), logger *log.Logger) {
defer conn.Close()
_ = conn.SetDeadline(time.Now().Add(30 * time.Second))
answer := func(r holdCtlResponse) { _ = json.NewEncoder(conn).Encode(r) }
if ok, why := holdCtlPeerAllowed(conn); !ok {
if logger != nil {
logger.Printf("[WARN] [hold] local channel: a request was refused — %s", why)
}
answer(holdCtlResponse{Outcome: report.OutcomeRefused, Message: "refused: " + why})
return
}
line, err := bufio.NewReader(io.LimitReader(conn, holdCtlMaxRequest)).ReadBytes('\n')
if err != nil && len(line) == 0 {
answer(holdCtlResponse{Outcome: report.OutcomeRefused, Message: "no request was read"})
return
}
var req holdCtlRequest
if err := json.Unmarshal(line, &req); err != nil || req.Op != holdCtlOpClear {
answer(holdCtlResponse{Outcome: report.OutcomeRefused, Message: "unknown request — this channel only clears a hold"})
return
}
msg, err := clear(req.Stack, "the operator on the box (--clear-restore-hold)")
switch {
case err == nil:
answer(holdCtlResponse{Outcome: report.OutcomeDone, Message: msg})
case errors.Is(err, report.ErrActionRefused):
answer(holdCtlResponse{Outcome: report.OutcomeRefused, Message: strings.TrimPrefix(err.Error(), report.ErrActionRefused.Error()+": ")})
default:
answer(holdCtlResponse{Outcome: report.OutcomeFailed, Message: err.Error()})
}
}
// askRunningControllerToClear sends one clear request. reached=false: no controller answered at path (no socket, or
// a socket nobody listens on) — nothing was asked, so the caller may edit settings.json itself. reached=true with an
// error: the request may have been applied; the caller must not edit the file.
func askRunningControllerToClear(path, stack string) (resp holdCtlResponse, reached bool, err error) {
conn, err := net.DialTimeout("unix", path, 3*time.Second)
if err != nil {
return holdCtlResponse{}, false, err
}
defer conn.Close()
_ = conn.SetDeadline(time.Now().Add(60 * time.Second))
if err := json.NewEncoder(conn).Encode(holdCtlRequest{Op: holdCtlOpClear, Stack: stack}); err != nil {
return holdCtlResponse{}, true, err
}
if err := json.NewDecoder(io.LimitReader(conn, 64*1024)).Decode(&resp); err != nil {
return holdCtlResponse{}, true, err
}
return resp, true, nil
}
// clearRestoreHoldCLI is the box command `--clear-restore-hold <app>` (R-379, R-1343). It asks the running
// controller first; only when none answers does it edit settings.json itself, and it says which one happened.
// Returns the process exit code: 0 cleared, 2 refused (nothing changed), 1 failed or unknown.
func clearRestoreHoldCLI(dataDir, stack string, stdout, stderr io.Writer) int {
path := holdCtlPath(dataDir)
resp, reached, err := askRunningControllerToClear(path, stack)
if reached {
if err != nil {
fmt.Fprintf(stderr, "the running controller was reached, but its answer was lost: %v\n", err)
fmt.Fprintf(stderr, " The hold may or may not be cleared. Check with: felhom-controller --restore-holds\n")
fmt.Fprintf(stderr, " settings.json was NOT edited by this command.\n")
return 1
}
switch resp.Outcome {
case report.OutcomeDone:
fmt.Fprintf(stdout, "%s.\n", resp.Message)
fmt.Fprintf(stdout, " Cleared by the RUNNING controller (memory and settings.json together) — no restart is needed.\n")
return 0
case report.OutcomeRefused:
fmt.Fprintf(stderr, "%s\n", resp.Message)
return 2
default:
fmt.Fprintf(stderr, "the running controller could not clear the hold: %s\n", resp.Message)
return 1
}
}
// No controller answered: it is not running (or is older than this channel). The file is the only copy.
fmt.Fprintf(stdout, "No running controller answered on %s (%v).\n", path, err)
fmt.Fprintf(stdout, " Editing settings.json directly instead.\n")
if !backup.IsValidHoldStack(stack) {
fmt.Fprintf(stderr, "%q is not a valid app name — nothing was changed\n", stack)
return 2
}
lg := log.New(stderr, "", 0)
sett, lerr := settings.Load(filepath.Join(dataDir, "settings.json"), lg)
if lerr != nil {
fmt.Fprintf(stderr, "restore-hold: loading settings: %v\n", lerr)
return 1
}
cleared, cerr := sett.ClearRestoreHold(stack)
if cerr != nil {
fmt.Fprintf(stderr, "restore-hold: clearing %s: %v\n", stack, cerr)
return 1
}
if !cleared {
// "there was nothing to clear" is not success — a silent 0 here would let an operator believe they had
// unblocked an app whose name they mistyped.
fmt.Fprintf(stderr, "no restore hold in force for %q — nothing was changed\n", stack)
return 2
}
fmt.Fprintf(stdout, "restore hold cleared for %s in settings.json (no controller was running to ask).\n", stack)
fmt.Fprintf(stdout, " The controller reads it when it starts. If a controller IS running in this guest after all, restart it NOW,\n")
fmt.Fprintf(stdout, " INSIDE THE GUEST — from the Proxmox host: pct exec <vmid> -- systemctl restart felhom-controller-bootstrap.service\n")
fmt.Fprintf(stdout, " — or its next settings save writes the hold back (R-1343).\n")
return 0
}
// holdClearCommandHint is the exact guest command, for the help text and the operator's mail.
const holdClearCommandHint = "pct exec <vmid> -- docker exec felhom-controller felhom-controller --clear-restore-hold <app>"
+516
View File
@@ -0,0 +1,516 @@
package main
import (
"bytes"
"context"
"encoding/json"
"go/ast"
"go/parser"
"go/token"
"io"
"log"
"net"
"os"
"path/filepath"
"strconv"
"strings"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
"gitea.dooplex.hu/admin/felhom-controller/internal/holdreport"
"gitea.dooplex.hu/admin/felhom-controller/internal/report"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// D64 = B (operator ruling 2026-10-29 12:10) + R-1343: the operator clears a hold — from the hub (clear_hold) or with
// the box command — IN THE RUNNING CONTROLLER, so the hold is gone from memory and from settings.json together and no
// later save brings it back.
//
// RED-PROOFS (2026-10-29):
// - TestR1343_TheBoxCommandClearsThroughTheRunningController: with clearRestoreHoldCLI's socket call skipped (the
// old second-process edit of settings.json), it fails on „the hold came back after a later save of the running
// controller" — the R-1343 measurement, in a test;
// - TestD64_ClearPutBackPendingDropsTheMarkerEntry: with the dropFromMarker call removed from
// ClearHoldByOperator, it fails on „a later controller start poured the old copy back";
// - TestD64_ClearHoldViaTheHub: with the forgetUpdateHeld call removed from clearHoldNow, it fails on „the
// update's held flag was not forgotten".
// - TestD64_ClearHoldViaTheHub + TestD64_ClearHoldStaysGoneAfterALaterSave: with ClearHoldByOperator clearing a
// SECOND Settings loaded from disk (the old CLI's shape) instead of the running one, they fail on „the running
// controller still holds the app" / „the hold came back after a later save of the running controller".
// d64Rig is a running controller's hold side: its Settings, the app-stop guard wired to them, and a log.
type d64Rig struct {
dir string
sett *settings.Settings
guard *backup.AppStopGuard
logBuf *syncBuffer
logger *log.Logger
forgot []string
holdDeps holdClearDeps
}
type syncBuffer struct {
mu sync.Mutex
b bytes.Buffer
}
func (s *syncBuffer) Write(p []byte) (int, error) {
s.mu.Lock()
defer s.mu.Unlock()
return s.b.Write(p)
}
func (s *syncBuffer) String() string { s.mu.Lock(); defer s.mu.Unlock(); return s.b.String() }
func newD64Rig(t *testing.T) *d64Rig {
t.Helper()
r := &d64Rig{dir: t.TempDir(), logBuf: &syncBuffer{}}
r.logger = log.New(r.logBuf, "", 0)
sett, err := settings.Load(filepath.Join(r.dir, "settings.json"), r.logger)
if err != nil {
t.Fatal(err)
}
r.sett = sett
r.guard = backup.NewAppStopGuard(filepath.Join(r.dir, "appstop-state.json"), r.logger)
r.guard.SetHolds(sett)
r.holdDeps = holdClearDeps{
clear: r.guard.ClearHoldByOperator,
forgetUpdateHeld: func(s string) bool { r.forgot = append(r.forgot, s); return true },
logger: r.logger,
}
return r
}
func (r *d64Rig) hold(t *testing.T, h settings.RestoreHold) {
t.Helper()
if h.At == "" {
h.At = "2026-10-29T08:18:00Z"
}
if err := r.sett.SetRestoreHold(h); err != nil {
t.Fatal(err)
}
}
func (r *d64Rig) onDisk(t *testing.T, stack string) bool {
t.Helper()
s, err := settings.Load(filepath.Join(r.dir, "settings.json"), log.New(io.Discard, "", 0))
if err != nil {
t.Fatal(err)
}
_, held := s.GetRestoreHold(stack)
return held
}
func (r *d64Rig) actions() *report.OperatorActions {
return report.NewOperatorActions(context.Background(), operatorActionHandlers(nil, nil, r.holdDeps, nil, r.logger), r.logger)
}
func onlyResult(t *testing.T, o *report.OperatorActions, id int64) report.OperatorActionResult {
t.Helper()
for _, x := range o.Results() {
if x.ID == id {
return x
}
}
t.Fatalf("no result for #%d: %+v", id, o.Results())
return report.OperatorActionResult{}
}
// The hub's clear_hold: the hold is gone (memory and disk), the household's held sentence is gone, the update's
// held flag is forgotten, the box logs who did it, and the result names what was cleared.
func TestD64_ClearHoldViaTheHub(t *testing.T) {
r := newD64Rig(t)
r.hold(t, settings.RestoreHold{Stack: "paperless", ReplayError: "ERROR: relation exists", RollbackErr: "psql: refused"})
cfg := &config.Config{}
cfg.Paths.DataDir = r.dir
mgr := backup.NewManager(cfg, r.sett, r.logger)
if held, why := mgr.RestoreHoldForLang("paperless", "hu"); !held || why == "" {
t.Fatal("fixture: the household must read a held sentence before the clear")
}
o := r.actions()
o.Reconcile([]report.OperatorAction{{ID: 12, Action: report.OpClearHold, Arg: "paperless"}})
res := onlyResult(t, o, 12)
if res.Outcome != report.OutcomeDone {
t.Fatalf("result %+v", res)
}
for _, want := range []string{"the hold on paperless is cleared", "held after a failed restore whose rollback also failed", "NOT started by this"} {
if !strings.Contains(res.Message, want) {
t.Errorf("result message %q lacks %q", res.Message, want)
}
}
if _, held := r.sett.GetRestoreHold("paperless"); held {
t.Fatal("the running controller still holds the app")
}
if r.onDisk(t, "paperless") {
t.Fatal("settings.json still holds the app")
}
if held, why := mgr.RestoreHoldForLang("paperless", "hu"); held || why != "" {
t.Fatalf("the household still reads a held sentence: %q", why)
}
if len(r.forgot) != 1 || r.forgot[0] != "paperless" {
t.Fatalf("the update's held flag was not forgotten: %v", r.forgot)
}
if lg := r.logBuf.String(); !strings.Contains(lg, "paperless: hold CLEARED by the operator via the hub, action #12") {
t.Fatalf("the box's log must say who cleared it:\n%s", lg)
}
}
// Refusals change nothing: an app nobody ever held, a second press on an app already cleared, a bad name.
func TestD64_ClearHoldRefusals(t *testing.T) {
r := newD64Rig(t)
r.hold(t, settings.RestoreHold{Stack: "kimai", Reason: settings.HoldReasonUpdateFailed})
o := r.actions()
o.Reconcile([]report.OperatorAction{{ID: 1, Action: report.OpClearHold, Arg: "nosuchapp"}})
if res := onlyResult(t, o, 1); res.Outcome != report.OutcomeRefused || !strings.Contains(res.Message, "nosuchapp has no hold") {
t.Fatalf("unknown app: %+v", res)
}
if _, held := r.sett.GetRestoreHold("kimai"); !held || !r.onDisk(t, "kimai") {
t.Fatal("a refusal changed another app's hold")
}
if len(r.forgot) != 0 {
t.Fatalf("a refusal touched the update state: %v", r.forgot)
}
o.Reconcile([]report.OperatorAction{{ID: 2, Action: report.OpClearHold, Arg: "kimai"}})
o.Reconcile([]report.OperatorAction{{ID: 3, Action: report.OpClearHold, Arg: "kimai"}})
if res := onlyResult(t, o, 3); res.Outcome != report.OutcomeRefused || !strings.Contains(res.Message, "kimai has no hold") {
t.Fatalf("not held (second press): %+v", res)
}
if _, err := clearHoldNow(r.holdDeps, "../kimai", "test"); err == nil || !strings.Contains(err.Error(), "not a valid app name") {
t.Fatalf("a bad name must be refused by the door itself too: %v", err)
}
}
// R-1343's order: clear, then ANOTHER component saves settings (the hold reporter, a hub report), then the file is
// read back — the hold must stay gone, in memory and on disk.
func TestD64_ClearHoldStaysGoneAfterALaterSave(t *testing.T) {
r := newD64Rig(t)
r.hold(t, settings.RestoreHold{Stack: "radicale", Reason: settings.HoldReasonRestoreMixed})
o := r.actions()
o.Reconcile([]report.OperatorAction{{ID: 5, Action: report.OpClearHold, Arg: "radicale"}})
if res := onlyResult(t, o, 5); res.Outcome != report.OutcomeDone {
t.Fatalf("result %+v", res)
}
if err := r.sett.SetReportedHolds(map[string]string{"other": "x|y"}); err != nil { // another component's save
t.Fatal(err)
}
if r.onDisk(t, "radicale") {
t.Fatal("the hold came back after a later save of the running controller")
}
if _, held := r.sett.GetRestoreHold("radicale"); held {
t.Fatal("the running controller holds the app again")
}
}
// The same order through the BOX COMMAND: it reaches the running controller over the socket; no restart.
func TestR1343_TheBoxCommandClearsThroughTheRunningController(t *testing.T) {
r := newD64Rig(t)
r.hold(t, settings.RestoreHold{Stack: "radicale", Reason: settings.HoldReasonRestoreMixed})
ln, err := listenHoldCtl(holdCtlPath(r.dir), func(stack, who string) (string, error) { return clearHoldNow(r.holdDeps, stack, who) }, r.logger)
if err != nil {
t.Fatal(err)
}
defer ln.Close()
var out, errb bytes.Buffer
if code := clearRestoreHoldCLI(r.dir, "radicale", &out, &errb); code != 0 {
t.Fatalf("exit %d, stdout %q, stderr %q", code, out.String(), errb.String())
}
if err := r.sett.SetReportedHolds(map[string]string{"other": "x|y"}); err != nil { // the running controller's next save
t.Fatal(err)
}
if r.onDisk(t, "radicale") {
t.Fatal("the hold came back after a later save of the running controller")
}
if !strings.Contains(out.String(), "Cleared by the RUNNING controller") || !strings.Contains(out.String(), "no restart is needed") {
t.Fatalf("the command must say it went through the running controller and needs no restart:\n%s", out.String())
}
if !strings.Contains(r.logBuf.String(), "hold CLEARED by the operator on the box (--clear-restore-hold)") {
t.Fatalf("the box's log must say who cleared it:\n%s", r.logBuf.String())
}
// A second run: refused through the socket, exit 2, nothing edited.
out.Reset()
errb.Reset()
if code := clearRestoreHoldCLI(r.dir, "radicale", &out, &errb); code != 2 || !strings.Contains(errb.String(), "radicale has no hold") {
t.Fatalf("not held: exit %d, stderr %q", code, errb.String())
}
}
// No controller runs (no socket, or a socket nobody listens on): the command edits settings.json and says so plainly,
// naming the guest restart for the case a controller runs after all.
func TestR1343_TheBoxCommandFallsBackWhenNoControllerRuns(t *testing.T) {
for _, stale := range []bool{false, true} {
r := newD64Rig(t)
r.hold(t, settings.RestoreHold{Stack: "kimai", Reason: settings.HoldReasonUpdateFailed})
if stale { // a socket left by a controller that died
ln, err := listenHoldCtl(holdCtlPath(r.dir), func(string, string) (string, error) { return "", nil }, r.logger)
if err != nil {
t.Fatal(err)
}
ln.(*net.UnixListener).SetUnlinkOnClose(false)
ln.Close()
}
var out, errb bytes.Buffer
if code := clearRestoreHoldCLI(r.dir, "kimai", &out, &errb); code != 0 {
t.Fatalf("stale=%v: exit %d, stderr %q", stale, code, errb.String())
}
for _, want := range []string{"No running controller answered", "Editing settings.json directly", "INSIDE THE GUEST", "pct exec <vmid> -- systemctl restart felhom-controller-bootstrap.service"} {
if !strings.Contains(out.String(), want) {
t.Errorf("stale=%v: output lacks %q:\n%s", stale, want, out.String())
}
}
if r.onDisk(t, "kimai") {
t.Fatalf("stale=%v: settings.json still holds the app", stale)
}
out.Reset()
errb.Reset()
if code := clearRestoreHoldCLI(r.dir, "kimai", &out, &errb); code != 2 {
t.Fatalf("stale=%v: not held must exit 2, got %d", stale, code)
}
if code := clearRestoreHoldCLI(r.dir, "../x", &out, &errb); code != 2 {
t.Fatalf("stale=%v: a bad name must exit 2, got %d", stale, code)
}
}
}
// A put-back WAITING for its drive is also in the app-stop marker: clearing the hold must drop it there too, or the
// next controller start with the drive bound would pour the old copy over what the app wrote after the clear. The
// copy itself stays on disk (nothing is deleted).
func TestD64_ClearPutBackPendingDropsTheMarkerEntry(t *testing.T) {
r := newD64Rig(t)
undo := filepath.Join(r.dir, "drive", "felhom-data", "backups", "restore-undo", "immich", "20261029T080000Z")
if err := os.MkdirAll(undo, 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(undo, "undo.json"), []byte(`{"stack":"immich"}`), 0o600); err != nil {
t.Fatal(err)
}
if err := r.guard.Begin("unit-restore:immich", backup.ReasonUnitRestore, []string{"immich"}); err != nil {
t.Fatal(err)
}
if err := r.guard.SetUndo("immich", undo); err != nil {
t.Fatal(err)
}
r.hold(t, settings.RestoreHold{Stack: "immich", Reason: settings.HoldReasonPutBackPending, SafetyDump: undo})
o := r.actions()
o.Reconcile([]report.OperatorAction{{ID: 3, Action: report.OpClearHold, Arg: "immich"}})
res := onlyResult(t, o, 3)
if res.Outcome != report.OutcomeDone || !strings.Contains(res.Message, "waiting put-back is dropped") || !strings.Contains(res.Message, undo) {
t.Fatalf("result %+v", res)
}
if _, err := os.Stat(filepath.Join(undo, "undo.json")); err != nil {
t.Fatal("the copy must stay on disk")
}
// The consequence: a later controller start with the drive bound pours nothing back.
poured := &recordingPutBacker{}
g2 := backup.NewAppStopGuard(filepath.Join(r.dir, "appstop-state.json"), r.logger)
g2.SetHolds(r.sett)
g2.SetPutBacker(poured)
g2.SetStarter(nopStarter{})
g2.SetUndoReachable(func(string, string) bool { return true })
g2.Recover()
if len(poured.stacks) != 0 {
t.Fatalf("a later controller start poured the old copy back over the app: %v", poured.stacks)
}
}
type recordingPutBacker struct{ stacks []string }
func (p *recordingPutBacker) PutBackInterruptedRestore(stack, _ string) error {
p.stacks = append(p.stacks, stack)
return nil
}
type nopStarter struct{}
func (nopStarter) StartStack(string) error { return nil }
// The hold reporter's next tick tells the hub the hold is gone (it reads the same settings record; nothing cached).
func TestD64_TheHoldReporterTellsTheHubAfterTheClear(t *testing.T) {
r := newD64Rig(t)
r.hold(t, settings.RestoreHold{Stack: "kimai", Reason: settings.HoldReasonUpdateFailed})
var events []string
rep := holdreport.New(r.sett, func(et, _, _ string, _ interface{}) { events = append(events, et) }, r.logger)
rep.Tick(timeAt(0))
o := r.actions()
o.Reconcile([]report.OperatorAction{{ID: 1, Action: report.OpClearHold, Arg: "kimai"}})
rep.Tick(timeAt(60))
if strings.Join(events, ",") != holdreport.EventHeld+","+holdreport.EventCleared {
t.Fatalf("events %v — want app_held then app_hold_cleared", events)
}
}
// „Not reachable from outside": the channel is a unix socket (no TCP anywhere in the file), 0600 in a 0700 directory
// — even when the directory existed with a looser mode — and main opens it once, under the data dir.
func TestHoldCtl_NotReachableFromOutside(t *testing.T) {
dir := t.TempDir()
if err := os.MkdirAll(filepath.Join(dir, holdCtlDir), 0o755); err != nil {
t.Fatal(err)
}
ln, err := listenHoldCtl(holdCtlPath(dir), func(string, string) (string, error) { return "", nil }, nil)
if err != nil {
t.Fatal(err)
}
defer ln.Close()
if n := ln.Addr().Network(); n != "unix" {
t.Fatalf("the channel listens on %q, want unix", n)
}
fi, err := os.Lstat(holdCtlPath(dir))
if err != nil || fi.Mode()&os.ModeSocket == 0 || fi.Mode().Perm() != 0o600 {
t.Fatalf("socket mode %v (err %v), want a socket with 0600", fi.Mode(), err)
}
di, _ := os.Stat(filepath.Join(dir, holdCtlDir))
if di.Mode().Perm() != 0o700 {
t.Fatalf("socket dir mode %v, want 0700", di.Mode().Perm())
}
// No TCP and no HTTP in the channel's file: every net.Listen/Dial names "unix".
f, err := parser.ParseFile(token.NewFileSet(), "holdclear.go", nil, parser.ImportsOnly)
if err != nil {
t.Fatal(err)
}
for _, im := range f.Imports {
if p, _ := strconv.Unquote(im.Path.Value); strings.HasPrefix(p, "net/http") {
t.Fatalf("holdclear.go imports %s — the channel must not be HTTP", p)
}
}
full, err := parser.ParseFile(token.NewFileSet(), "holdclear.go", nil, 0)
if err != nil {
t.Fatal(err)
}
nets := 0
ast.Inspect(full, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
sel, ok := call.Fun.(*ast.SelectorExpr)
if !ok {
return true
}
if x, ok := sel.X.(*ast.Ident); ok && x.Name == "net" && len(call.Args) > 0 {
nets++
if lit, ok := call.Args[0].(*ast.BasicLit); !ok || lit.Value != `"unix"` {
t.Errorf("net.%s is called with a network other than \"unix\"", sel.Sel.Name)
}
}
return true
})
if nets < 2 {
t.Fatalf("expected the listen and the dial in holdclear.go, found %d net calls", nets)
}
// main opens it exactly once, at holdCtlPath(cfg.Paths.DataDir).
mf, err := parser.ParseFile(token.NewFileSet(), "main.go", nil, 0)
if err != nil {
t.Fatal(err)
}
opened := 0
ast.Inspect(mf, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
if id, ok := call.Fun.(*ast.Ident); ok && id.Name == "listenHoldCtl" {
opened++
arg, ok := call.Args[0].(*ast.CallExpr)
if fn, ok2 := arg.Fun.(*ast.Ident); !ok || !ok2 || fn.Name != "holdCtlPath" {
t.Error("main must open the channel at holdCtlPath(<data dir>)")
}
}
return true
})
if opened != 1 {
t.Fatalf("main.go opens the channel %d time(s), want 1", opened)
}
}
// The socket answers only its one request; anything else is refused and calls nothing.
func TestHoldCtl_UnknownRequestIsRefused(t *testing.T) {
dir := t.TempDir()
called := 0
ln, err := listenHoldCtl(holdCtlPath(dir), func(string, string) (string, error) { called++; return "x", nil }, nil)
if err != nil {
t.Fatal(err)
}
defer ln.Close()
conn, err := net.Dial("unix", holdCtlPath(dir))
if err != nil {
t.Fatal(err)
}
defer conn.Close()
if _, err := conn.Write([]byte(`{"op":"set_hold","stack":"x"}` + "\n")); err != nil {
t.Fatal(err)
}
var resp holdCtlResponse
if err := json.NewDecoder(conn).Decode(&resp); err != nil {
t.Fatal(err)
}
if resp.Outcome != report.OutcomeRefused || called != 0 {
t.Fatalf("response %+v, calls %d — want refused and nothing called", resp, called)
}
}
// Something at the socket's path that is NOT a socket is refused, never removed.
func TestHoldCtl_ANonSocketFileIsNotReplaced(t *testing.T) {
dir := t.TempDir()
p := holdCtlPath(dir)
if err := os.MkdirAll(filepath.Dir(p), 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte("keep me"), 0o600); err != nil {
t.Fatal(err)
}
if _, err := listenHoldCtl(p, func(string, string) (string, error) { return "", nil }, nil); err == nil {
t.Fatal("a regular file at the socket path must be refused")
}
if b, _ := os.ReadFile(p); string(b) != "keep me" {
t.Fatal("the file at the socket path was changed")
}
}
// main wires every piece once (the seam-built-but-never-wired trap): the clear door into the operator actions, the
// R-1340 text into the hold notification, the R-1341 gate into the OOM loop, and the box command.
func TestD64_MainWiresTheNewPieces(t *testing.T) {
f, err := parser.ParseFile(token.NewFileSet(), "main.go", nil, 0)
if err != nil {
t.Fatal(err)
}
found := map[string]int{}
var opArgs []ast.Expr
ast.Inspect(f, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
switch fn := call.Fun.(type) {
case *ast.SelectorExpr:
if x, ok := fn.X.(*ast.Ident); ok {
found[x.Name+"."+fn.Sel.Name]++
}
case *ast.Ident:
found[fn.Name]++
if fn.Name == "operatorActionHandlers" {
opArgs = call.Args
}
}
return true
})
for _, want := range []string{"heldStoppedOperatorText", "heldHoldFromNotify", "oomLog.fresh", "clearRestoreHoldCLI", "listenHoldCtl", "clearHoldNow"} {
if found[want] != 1 {
t.Errorf("main.go calls %s %d time(s), want exactly 1", want, found[want])
}
}
if len(opArgs) < 3 {
t.Fatal("operatorActionHandlers call not found")
}
if id, ok := opArgs[2].(*ast.Ident); !ok || id.Name != "holdDeps" {
t.Error("main must hand holdDeps (the running controller's clear door) to operatorActionHandlers")
}
}
func timeAt(sec int) time.Time { return time.Date(2026, 10, 29, 12, 0, sec, 0, time.UTC) }
@@ -0,0 +1,34 @@
//go:build linux
package main
import (
"fmt"
"net"
"os"
"syscall"
)
// holdCtlPeerAllowed (R-1343) admits a peer on the hold socket only when its uid equals this process's (SO_PEERCRED).
// The socket's 0600 mode in a 0700 directory already says so; this is the second lock on the same door.
func holdCtlPeerAllowed(conn net.Conn) (bool, string) {
uc, ok := conn.(*net.UnixConn)
if !ok {
return false, "not a unix-socket peer"
}
raw, err := uc.SyscallConn()
if err != nil {
return false, "the peer's identity could not be read"
}
var cred *syscall.Ucred
var cerr error
if err := raw.Control(func(fd uintptr) {
cred, cerr = syscall.GetsockoptUcred(int(fd), syscall.SOL_SOCKET, syscall.SO_PEERCRED)
}); err != nil || cerr != nil || cred == nil {
return false, "the peer's identity could not be read"
}
if int(cred.Uid) != os.Geteuid() {
return false, fmt.Sprintf("the peer's uid %d is not this controller's", cred.Uid)
}
return true, ""
}
@@ -0,0 +1,14 @@
//go:build !linux
package main
import "net"
// holdCtlPeerAllowed: the peer check needs SO_PEERCRED (Linux, where the controller runs). Elsewhere the socket's
// 0600 mode in a 0700 directory is the only lock.
func holdCtlPeerAllowed(conn net.Conn) (bool, string) {
if _, ok := conn.(*net.UnixConn); !ok {
return false, "not a unix-socket peer"
}
return true, ""
}
+155
View File
@@ -0,0 +1,155 @@
package main
import (
"fmt"
"strings"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// ── R-1340: the operator's „HELD STOPPED" mail names the hold's REAL cause ────────────────────────────────────────
//
// Until 2026-10-29 the restore-hold mail told one story for every restore_mixed hold — R-893 option C's „its off-site
// database restore failed. The database was rolled back …, but its files and named volumes are the snapshot's" —
// written when option C was the only mixed hold. Measured on Tester 1 (D63's H5 proof): Radicale, which has no
// database, whose put-back stopped at „the app could not be stopped" with nothing moved, got exactly that text.
// Support read a wrong account of the box.
//
// The text is now built from the HOLD RECORD the writer persisted just before the mail (its reason and the R-1340
// fields: which restore, which branch, which step failed, whether the failure was a database replay, where the copy
// is kept). It never says „database" unless the restore's failure WAS a database replay, and never „off-site" for a
// restore from the app's own backup. One sentence per reason; pinned by TestR1340_* (holdmail_test.go).
// heldStoppedOperatorText is the operator mail for a held app: the message, its leg (the hub reads the leg, so the
// two existing ones are kept: restore-hold for a restore hold, restore-hold-mixed for a mixed one) and a short reason.
func heldStoppedOperatorText(h settings.RestoreHold) (msg, leg, reason string) {
stack := h.Stack
const tail = " The app will not start from any path until the hold is cleared (the hub's Clear hold button, or on the box: " + holdClearCommandHint + ")."
kind := restoreKindPhrase(h.RestoreKind)
switch h.Reason {
case settings.HoldReasonUnhealthyStop:
what := "a crash loop"
if h.UnhealthyKind == "oom_storm" {
what = "an out-of-memory storm"
}
trip := ""
if h.Trip >= 2 {
trip = fmt.Sprintf(" This is stop %d within 24 hours.", h.Trip)
}
msg = fmt.Sprintf("App %q is HELD STOPPED: the box stopped it for %s at %s.%s No restore or update was involved and no data was touched. "+
"The household's Start gives it one more try; clearing the hold lifts it too.", stack, what, h.At, trip)
return msg, "unhealthy-stop", "stopped for " + what
case settings.HoldReasonUpdateFailed:
undo := ""
switch h.UndoState {
case "untouched":
undo = " The automatic undo failed; the data was not touched."
case "half":
undo = " The automatic undo failed half-way; the data may be partly the new version's."
case "not_started":
undo = " The automatic undo did not start."
}
copyLine := " Restoring it from its backup starts it."
if h.NoWholeCopy {
copyLine = " No copy on this box brings it back whole — it needs support."
} else if h.CopyDate != "" {
copyLine = fmt.Sprintf(" Restoring it from its backup (tier %d, %s) starts it.", h.CopyTier, h.CopyDate)
}
msg = fmt.Sprintf("App %q is HELD STOPPED: its update at %s did not come up healthy.%s%s", stack, h.At, undo, copyLine) + tail
return msg, "update-hold", "update failed"
case settings.HoldReasonPutBackPending:
msg = fmt.Sprintf("App %q is HELD STOPPED: a restore was cut after its copy of the live data was taken, and that copy (%s) is on a drive that is not bound. "+
"The copy is poured back and the app started when the drive is bound. Clearing the hold DROPS that put-back; the copy stays on disk.", stack, orNone(h.SafetyDump))
return msg, "putback-pending", "put-back waits for its drive"
case settings.HoldReasonRestoreMixed:
return mixedHoldText(h, kind, tail)
default: // HoldReasonRestoreFailed: a database replay failed AND its rollback failed (holdAppAfterFailedRollback)
what := "its " + kind + " failed"
if h.DBReplay || h.RestoreKind == "" { // every restore hold before R-1340 was written by the database branch
what = "its " + kind + " failed at the database replay AND the rollback to the pre-restore database copy also failed — the database may be half-written"
}
msg = fmt.Sprintf("App %q is HELD STOPPED: %s.", stack, what) + tail +
fmt.Sprintf(" Replay error: %s. Rollback error: %s", orNone(h.ReplayError), orNone(h.RollbackErr))
reason = h.RollbackErr
if reason == "" {
reason = "rollback failed"
}
return msg, "restore-hold", reason
}
}
func mixedHoldText(h settings.RestoreHold, kind, tail string) (msg, leg, reason string) {
stack := h.Stack
leg = "restore-hold-mixed"
failedAt := ""
if h.DBReplay {
failedAt = " at the database replay"
}
copyLine := " There is no copy of the live volumes."
if h.SafetyDump != "" {
copyLine = " The copy of the live data is kept at " + h.SafetyDump + "."
}
switch h.MixedCause {
case settings.MixedCauseDefinitionWrite:
msg = fmt.Sprintf("App %q is HELD STOPPED for support: its %s failed while writing the backup's app definition, before any data was touched, "+
"and %s. The app's data is the live data; the definition on disk may be the backup's.", stack, kind, orNone(h.FailedStep)) + tail +
" Error: " + orNone(h.ReplayError)
return msg, leg, "definition write failed; writing the live definition back failed too"
case settings.MixedCausePutBack:
msg = fmt.Sprintf("App %q is HELD STOPPED for support: its %s failed%s, and putting the live data back FAILED — %s. "+
"The app may be on a mix of the backup's and the live data.%s", stack, kind, failedAt, orNone(h.FailedStep), copyLine) + tail +
" Restore error: " + orNone(h.ReplayError)
return msg, leg, "put-back failed: " + firstClause(h.FailedStep)
case settings.MixedCauseInterruptedPutBack:
msg = fmt.Sprintf("App %q is HELD STOPPED for support: a restore cut by a stop could not be put back at the next start — %s. "+
"The app may be on a mix of the backup's and the live data.%s", stack, orNone(h.FailedStep), copyLine) + tail
return msg, leg, "interrupted restore: put-back failed"
}
// A mixed hold written before R-1340 carries no cause: name only what the record says, and assume nothing.
msg = fmt.Sprintf("App %q is HELD STOPPED for support: a restore stopped half-way (set %s); the app may be on a mix of the backup's and the live data. "+
"The hold record says: %s.", stack, h.At, orNone(h.ReplayError)) + tail
return msg, leg, "restore stopped half-way"
}
func restoreKindPhrase(k string) string {
switch k {
case settings.RestoreKindLocal:
return "restore from its own backup"
case settings.RestoreKindOffsite:
return "off-site restore"
}
return "restore"
}
func orNone(s string) string {
if strings.TrimSpace(s) == "" {
return "(none recorded)"
}
return s
}
// firstClause is a step's text up to its error (the short reason field).
func firstClause(s string) string {
if i := strings.Index(s, ":"); i > 0 {
return s[:i]
}
return s
}
// heldHoldFromNotify is the hold the notification describes: the record the writer persisted just before it, or —
// when that write failed — one rebuilt from the errors handed over (the pre-R-1340 rule: no rollback error = mixed).
func heldHoldFromNotify(get func(string) (settings.RestoreHold, bool), stack string, replayErr, rollbackErr error) settings.RestoreHold {
if get != nil {
if h, ok := get(stack); ok && (h.Reason == settings.HoldReasonRestoreFailed || h.Reason == settings.HoldReasonRestoreMixed) {
return h
}
}
h := settings.RestoreHold{Stack: stack, Reason: settings.HoldReasonRestoreMixed}
if rollbackErr != nil {
h.Reason, h.RollbackErr = settings.HoldReasonRestoreFailed, rollbackErr.Error()
}
if replayErr != nil {
h.ReplayError = replayErr.Error()
}
return h
}
+140
View File
@@ -0,0 +1,140 @@
package main
import (
"errors"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-1340: the operator's „HELD STOPPED" mail names the hold's real cause — one test per reason. Each asserts the words
// that MUST be there and the words that must NOT (the option-C database story was the bug).
//
// RED-PROOF (2026-10-29): with mixedHoldText's body replaced by the pre-R-1340 sentence („its off-site database restore
// failed. The database was rolled back to the pre-restore copy and the live definition written back, but its files
// and named volumes are the snapshot's"), TestR1340_PutBackWithoutADatabase, _DefinitionWrite, _LocalRestore and
// _InterruptedPutBack fail on „database" / „off-site" being present and their own step missing.
const optionCStory = "files and named volumes are the snapshot's"
func mustHave(t *testing.T, msg string, want ...string) {
t.Helper()
for _, w := range want {
if !strings.Contains(msg, w) {
t.Errorf("the mail must say %q:\n%s", w, msg)
}
}
}
func mustNotHave(t *testing.T, msg string, bad ...string) {
t.Helper()
for _, b := range bad {
if strings.Contains(strings.ToLower(msg), strings.ToLower(b)) {
t.Errorf("the mail must NOT say %q:\n%s", b, msg)
}
}
}
func TestR1340_UnhealthyStop(t *testing.T) {
msg, leg, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "romm", At: "2026-10-29T08:00:00Z", Reason: settings.HoldReasonUnhealthyStop, UnhealthyKind: "crash_loop"})
mustHave(t, msg, `"romm"`, "crash loop", "no data was touched", "Start gives it one more try")
mustNotHave(t, msg, "database", "restore failed", optionCStory)
if leg != "unhealthy-stop" {
t.Errorf("leg %q", leg)
}
oom, _, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "romm", Reason: settings.HoldReasonUnhealthyStop, UnhealthyKind: "oom_storm", Trip: 2})
mustHave(t, oom, "out-of-memory storm", "stop 2 within 24 hours")
}
func TestR1340_UpdateFailed(t *testing.T) {
msg, leg, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "kimai", At: "2026-10-29T03:10:00Z", Reason: settings.HoldReasonUpdateFailed,
UndoState: "half", CopyTier: 1, CopyDate: "2026-10-29T02:00:00Z"})
mustHave(t, msg, "update at 2026-10-29T03:10:00Z did not come up healthy", "undo failed half-way", "tier 1, 2026-10-29T02:00:00Z", "--clear-restore-hold")
mustNotHave(t, msg, "database", optionCStory)
if leg != "update-hold" {
t.Errorf("leg %q", leg)
}
}
func TestR1340_RestoreFailedWithADatabase(t *testing.T) {
msg, leg, reason := heldStoppedOperatorText(settings.RestoreHold{Stack: "paperless", Reason: settings.HoldReasonRestoreFailed, RestoreKind: settings.RestoreKindOffsite,
DBReplay: true, ReplayError: "ERROR: relation exists", RollbackErr: "psql: connection refused"})
mustHave(t, msg, "off-site restore failed at the database replay", "rollback to the pre-restore database copy also failed", "ERROR: relation exists", "psql: connection refused")
mustNotHave(t, msg, optionCStory)
if leg != "restore-hold" || reason != "psql: connection refused" {
t.Errorf("leg %q reason %q", leg, reason)
}
}
// The measured case: Radicale has no database; the volume could not be removed and the put-back's stop failed.
func TestR1340_PutBackWithoutADatabase(t *testing.T) {
msg, leg, reason := heldStoppedOperatorText(settings.RestoreHold{Stack: "radicale", Reason: settings.HoldReasonRestoreMixed, RestoreKind: settings.RestoreKindOffsite,
MixedCause: settings.MixedCausePutBack, FailedStep: "stopping the app before the put-back failed, so nothing was put back: compose down: timeout",
ReplayError: "volume rm -f radicale_data: volume is in use", SafetyDump: "/mnt/hdd/felhom-data/backups/restore-undo/radicale/20261029T081800Z"})
mustHave(t, msg, `"radicale"`, "off-site restore failed, and putting the live data back FAILED", "stopping the app before the put-back failed",
"compose down: timeout", "kept at /mnt/hdd/felhom-data/backups/restore-undo/radicale/20261029T081800Z", "volume is in use")
mustNotHave(t, msg, "database", "rolled back", optionCStory)
if leg != "restore-hold-mixed" || reason != "put-back failed: stopping the app before the put-back failed, so nothing was put back" {
t.Errorf("leg %q reason %q", leg, reason)
}
}
func TestR1340_PutBackAfterADatabaseReplay(t *testing.T) {
msg, _, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "immich", Reason: settings.HoldReasonRestoreMixed, RestoreKind: settings.RestoreKindOffsite,
MixedCause: settings.MixedCausePutBack, DBReplay: true, FailedStep: "putting the live volumes back failed: tar xf failed"})
mustHave(t, msg, "failed at the database replay", "putting the live volumes back failed: tar xf failed", "There is no copy of the live volumes")
}
func TestR1340_DefinitionWrite(t *testing.T) {
msg, _, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "gokapi", Reason: settings.HoldReasonRestoreMixed, RestoreKind: settings.RestoreKindOffsite,
MixedCause: settings.MixedCauseDefinitionWrite, FailedStep: "writing the live app definition back failed: app.yaml write failed", ReplayError: "app.yaml write failed"})
mustHave(t, msg, "while writing the backup's app definition, before any data was touched", "writing the live app definition back failed", "data is the live data")
mustNotHave(t, msg, "database", optionCStory)
}
func TestR1340_LocalRestore(t *testing.T) {
msg, _, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "app", Reason: settings.HoldReasonRestoreMixed, RestoreKind: settings.RestoreKindLocal,
MixedCause: settings.MixedCausePutBack, FailedStep: "putting the live volumes back failed: tar xf failed", SafetyDump: "/srv/undo/app/1"})
mustHave(t, msg, "restore from its own backup failed", "putting the live volumes back failed")
mustNotHave(t, msg, "off-site", "database", optionCStory)
}
func TestR1340_PutBackPending(t *testing.T) {
msg, leg, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "immich", Reason: settings.HoldReasonPutBackPending, SafetyDump: "/mnt/hdd/undo/immich/1"})
mustHave(t, msg, "copy (/mnt/hdd/undo/immich/1) is on a drive that is not bound", "poured back and the app started when the drive is bound", "DROPS that put-back")
mustNotHave(t, msg, "database", optionCStory)
if leg != "putback-pending" {
t.Errorf("leg %q", leg)
}
}
func TestR1340_InterruptedPutBack(t *testing.T) {
msg, _, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "immich", Reason: settings.HoldReasonRestoreMixed, MixedCause: settings.MixedCauseInterruptedPutBack,
FailedStep: "putting the interrupted restore's copy back failed: compose down: timeout", SafetyDump: "/mnt/hdd/undo/immich/1"})
mustHave(t, msg, "cut by a stop could not be put back", "compose down: timeout", "kept at /mnt/hdd/undo/immich/1")
mustNotHave(t, msg, "database", "off-site", optionCStory)
}
// A mixed hold written before R-1340 names only what its record says.
func TestR1340_LegacyMixedHoldAssumesNothing(t *testing.T) {
msg, _, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "x", At: "2026-10-01T00:00:00Z", Reason: settings.HoldReasonRestoreMixed, ReplayError: "boom"})
mustHave(t, msg, "stopped half-way", "The hold record says: boom")
mustNotHave(t, msg, "database", optionCStory)
}
// The notification reads the hold the writer persisted; when that write failed, it rebuilds one from the errors.
func TestR1340_NotifyReadsTheHoldRecord(t *testing.T) {
rec := settings.RestoreHold{Stack: "radicale", Reason: settings.HoldReasonRestoreMixed, MixedCause: settings.MixedCausePutBack, FailedStep: "x"}
get := func(s string) (settings.RestoreHold, bool) { return rec, s == "radicale" }
if h := heldHoldFromNotify(get, "radicale", errors.New("e"), nil); h.MixedCause != settings.MixedCausePutBack {
t.Fatalf("the persisted record was not used: %+v", h)
}
h := heldHoldFromNotify(get, "other", errors.New("replay"), errors.New("rollback"))
if h.Reason != settings.HoldReasonRestoreFailed || h.RollbackErr != "rollback" || h.ReplayError != "replay" {
t.Fatalf("fallback hold %+v", h)
}
if h := heldHoldFromNotify(nil, "other", errors.New("replay"), nil); h.Reason != settings.HoldReasonRestoreMixed {
t.Fatalf("no rollback error = mixed, got %+v", h)
}
}
+44 -53
View File
@@ -92,8 +92,9 @@ func main() {
abandonStatus := flag.Bool("abandon-status", false, "R-241: print the state of this box's off-site abandonment countdown (set-aside path, due date, days left) and exit. Read-only.")
abandonExtend := flag.Int("abandon-extend", 0, "R-241 (operator): extend a running abandonment countdown by N days from now, then exit. Refuses when no countdown is running.")
abandonStop := flag.Bool("abandon-stop", false, "R-241 (operator): stop a running abandonment countdown, then exit. The set-aside history is kept and nothing is deleted. Refuses when no countdown is running.")
listRestoreHolds := flag.Bool("restore-holds", false, "R-379 (operator): list apps this controller is deliberately holding stopped after a failed database restore whose rollback also failed, then exit. Read-only.")
clearRestoreHold := flag.String("clear-restore-hold", "", "R-379 (operator): clear the restore hold on APP so it can start again, then exit. Clearing does NOT repair the database — check the app's data first; the undo copies are in its unit's db-dumps dir.")
listRestoreHolds := flag.Bool("restore-holds", false, "R-379 (operator): list the apps this controller is holding stopped (every reason, with why), then exit. Read-only.")
clearRestoreHold := flag.String("clear-restore-hold", "", "R-379/D64 (operator): clear the hold on APP (any reason), then exit. Run it inside the guest: "+holdClearCommandHint+
". It asks the RUNNING controller, so no restart is needed; only when no controller runs does it edit settings.json. Clearing repairs nothing and starts nothing — check the app's data first.")
flag.Parse()
if *showVersion {
@@ -172,21 +173,23 @@ func main() {
// starts, exactly like the other CLI subcommands: each loads config + settings, acts, and exits.
// R-379 (operator): the way OUT of a hold. A CLI subcommand rather than an API route, matching
// --abandon-stop above: the controller's HTTP surface is authenticated as the CUSTOMER, and
// clearing a hold is a decision that should follow an operator looking at the database. Reaching
// clearing a hold is a decision that should follow an operator looking at the data. Reaching
// this needs `docker exec` into the guest, which is the operator gate this project already uses.
// R-1343: the clear goes to the RUNNING controller over its local socket (holdclear.go); the list reads
// settings.json, which the running controller saves at every hold change.
if *listRestoreHolds || *clearRestoreHold != "" {
cfg, err := config.LoadPermissive(*configPath)
if err != nil {
fmt.Fprintf(os.Stderr, "restore-hold: loading config: %v\n", err)
os.Exit(1)
}
lg := log.New(os.Stderr, "", 0)
sett, err := settings.Load(cfg.Paths.DataDir+"/settings.json", lg)
if err != nil {
fmt.Fprintf(os.Stderr, "restore-hold: loading settings: %v\n", err)
os.Exit(1)
}
if *listRestoreHolds {
lg := log.New(os.Stderr, "", 0)
sett, err := settings.Load(cfg.Paths.DataDir+"/settings.json", lg)
if err != nil {
fmt.Fprintf(os.Stderr, "restore-hold: loading settings: %v\n", err)
os.Exit(1)
}
holds := sett.ListRestoreHolds()
if len(holds) == 0 {
fmt.Println("no restore holds in force")
@@ -197,34 +200,9 @@ func main() {
}
os.Exit(0)
}
cleared, err := sett.ClearRestoreHold(*clearRestoreHold)
if err != nil {
fmt.Fprintf(os.Stderr, "restore-hold: clearing %s: %v\n", *clearRestoreHold, err)
os.Exit(1)
}
if !cleared {
// "there was nothing to clear" is not success — a silent 0 here would let an operator
// believe they had unblocked an app whose name they mistyped.
fmt.Fprintf(os.Stderr, "no restore hold in force for %q — nothing was changed\n", *clearRestoreHold)
os.Exit(2)
}
// THE RESTART IS NOT OPTIONAL, and saying so is the difference between a route that works
// and one an operator believes worked. This runs as a SECOND process: it clears the hold in
// settings.json, but the RUNNING controller holds its own in-memory Settings and keeps
// refusing until it reloads. Measured on demo-hp 2026-08-22 — the clear succeeded, the file
// was correct, and the customer's start button still refused until the controller restarted.
//
// There is also a lost-update window while both processes hold the file: if the running
// controller saves settings.json from its own copy before the restart, the clear is undone.
// Restarting promptly closes it. A route that clears through the running controller would
// remove both problems and is the right shape later; it needs an operator tier the
// controller's HTTP surface does not have today (it authenticates as the CUSTOMER, and a
// customer clearing their own hold is precisely what the hold exists to prevent).
fmt.Printf("restore hold cleared for %s in settings.json.\n", *clearRestoreHold)
fmt.Printf(" NOW RESTART THE CONTROLLER, or it will keep refusing to start the app:\n")
fmt.Printf(" systemctl restart felhom-controller-bootstrap.service\n")
fmt.Printf(" Check the app's data first — the undo copies are in its unit's db-dumps dir.\n")
os.Exit(0)
// R-1343: through the RUNNING controller (holdclear.go) — the old second-process edit of settings.json was
// written back by the running controller's next save. Only when no controller answers is the file edited here.
os.Exit(clearRestoreHoldCLI(cfg.Paths.DataDir, *clearRestoreHold, os.Stdout, os.Stderr))
}
if *abandonStatus || *abandonExtend > 0 || *abandonStop {
@@ -485,6 +463,18 @@ func main() {
return stackMgr.DriveLive(hdd)
})
appStopRecovery := appStopGuard.Recover()
// D64 = B / R-1343: the operator's hold clear — ONE door for the hub's clear_hold and the box command, run in THIS
// process so memory and settings.json agree (holdclear.go). The box command reaches it over a unix socket in the
// data dir (0600, nothing on TCP). A socket that cannot be bound is logged; the command then says plainly that no
// running controller answered.
holdDeps := holdClearDeps{clear: appStopGuard.ClearHoldByOperator, forgetUpdateHeld: stackMgr.ForgetUpdateHeld, logger: logger}
if holdLn, herr := listenHoldCtl(holdCtlPath(cfg.Paths.DataDir), func(stack, who string) (string, error) {
return clearHoldNow(holdDeps, stack, who)
}, logger); herr != nil {
logger.Printf("[ERROR] [hold] the local hold-clear channel could not be opened: %v — `--clear-restore-hold` will edit settings.json and need a restart", herr)
} else {
defer holdLn.Close()
}
// R-1251: a put-back that waited for its drive (a hold from an earlier start) runs now if that drive is bound.
if started := appStopGuard.ResumeWaitingPutBacks(); len(started) > 0 {
logger.Printf("[WARN] [appstop] %d app(s) whose interrupted restore waited for its drive are put back and started: %v", len(started), started)
@@ -1081,6 +1071,7 @@ func main() {
holdReporter.Tick(time.Now())
return nil
})
oomLog := &oomLogGate{} // R-1341
sched.Every("deadapp-check", 30*time.Second, func(ctx context.Context) error {
if bootGrace.Within(time.Now()) {
return nil // still inside the startup settle window (90 s, or ~15 min after a crash boot)
@@ -1096,8 +1087,12 @@ func main() {
if oerr != nil {
logger.Printf("[WARN] [deadapp] OOM scan failed: %v", oerr)
} else {
for _, o := range ooms {
// R-1341: the line once per NEW kill (oomlog.go); the event call stays per scan — the notifier's cooldown
// keeps it to one, as before.
for _, o := range oomLog.fresh(ooms) {
logger.Printf("[WARN] [deadapp] %s: container %s was OOM-killed (started %s; kills %d, limit %s, peak %s; seen by %s)", o.Stack, o.Container, o.StartedAt, o.Kills, o.MemLimit, o.Peak, o.Source)
}
for _, o := range ooms {
notifier.NotifyAppOOMFrom(o.Stack, o.Container, o.StartedAt, o.Kills, o.MemLimit, o.Peak, o.Source)
}
}
@@ -1217,7 +1212,7 @@ func main() {
// R-314/R-279/R-177 (`09` §3 decision 185): the operator's closed list of actions, carried in the
// report reply and run in THIS process (opactions_wiring.go). The result rides the next report;
// a finished result fires an out-of-cycle report through the trigger once it exists.
opActions := report.NewOperatorActions(ctx, operatorActionHandlers(backupMgr, sched, func() {
opActions := report.NewOperatorActions(ctx, operatorActionHandlers(backupMgr, sched, holdDeps, func() {
if f, ok := opActionReportFire.Load().(func()); ok && f != nil {
f()
}
@@ -1548,22 +1543,18 @@ func main() {
// DELIBERATELY NOT `backup_failed`: customer-enabled by default, Hungarian "A biztonsági
// mentés sikertelen!", and this is a deliberate hold rather than a backup failure. R-171 one
// path over.
// R-1340: the text is built from the HOLD RECORD the writer persisted just before this call (holdmail.go) — its
// reason, which restore, which branch, which put-back step failed, whether it was a database replay — never the
// option-C database story for every cause.
backupMgr.SetRestoreHoldNotify(func(stack string, replayErr, rollbackErr error) {
d := notify.BackupRunFailuresDetails{RunKind: "offsite-reconstitute", Failed: 1, Attempted: 1}
if rollbackErr == nil { // R-893: the rollback WORKED, but files/volumes/version had already moved
d.Apps = append(d.Apps, notify.RunFailureDetail{App: stack, Leg: "restore-hold-mixed", Reason: "replay failed after files/volumes/version moved"})
msg := fmt.Sprintf("App %q is HELD STOPPED for support: its off-site database restore failed. The database was rolled back to the pre-restore copy and the live definition written back, "+
"but its files and named volumes are the snapshot's — a mixed state (R-893). The app will not start from any path until the hold is cleared. Replay error: %v", stack, replayErr)
notifier.NotifyBackupRunFailures(msg, d)
return
h := heldHoldFromNotify(sett.GetRestoreHold, stack, replayErr, rollbackErr)
msg, leg, reason := heldStoppedOperatorText(h)
runKind := "offsite-reconstitute"
if h.RestoreKind == settings.RestoreKindLocal {
runKind = "restore"
}
reason := "rollback failed"
if rollbackErr != nil {
reason = rollbackErr.Error()
}
d.Apps = append(d.Apps, notify.RunFailureDetail{App: stack, Leg: "restore-hold", Reason: reason})
msg := fmt.Sprintf("App %q is HELD STOPPED: its off-site database restore failed AND the rollback to the customer's own pre-restore copy also failed. "+
"The app will not start from any path until the hold is cleared. Replay error: %v. Rollback error: %v", stack, replayErr, rollbackErr)
d := notify.BackupRunFailuresDetails{RunKind: runKind, Failed: 1, Attempted: 1}
d.Apps = append(d.Apps, notify.RunFailureDetail{App: stack, Leg: leg, Reason: reason})
notifier.NotifyBackupRunFailures(msg, d)
})
// R-893 option A (D20 = A): a failed off-site restore that was PUT BACK — the live volumes and definition poured
+43
View File
@@ -0,0 +1,43 @@
package main
import (
"strconv"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
)
// ── R-1341: the OOM line is logged once per NEW kill ──────────────────────────────────────────────────────────────
//
// ScanOOMKilled reports a container for as long as it lives after a kill (Docker's OOMKilled flag is sticky, and the
// kernel's kill counter never goes down), so the 30-s dead-app scan logged „[WARN] [deadapp] … was OOM-killed" about
// 2 880 times a day per container — measured 2026-10-29 on Tester 1 — while the notifier's cooldown kept the event to
// one. The line now goes out when the kill is NEW: the key is (container, its start time, the kill count), and a line
// is logged only when a container's key changes — a second scan of the same kill logs nothing, a new kill (the count
// went up) or a restarted container (a new start time) logs again. The event call is unchanged (every scan, the
// notifier decides). Pinned by TestR1341_*.
// oomLogGate remembers the last logged key per container. Not safe for concurrent use: the dead-app job runs serially.
type oomLogGate struct {
last map[string]string
}
func oomLogKey(o stacks.OOMContainer) string {
return o.StartedAt + "|" + strconv.FormatInt(o.Kills, 10)
}
// fresh returns the containers whose kill is new since the last scan, and remembers this scan's keys. A container
// absent from this scan is forgotten (it was removed or restarted clean), which also bounds the memory.
func (g *oomLogGate) fresh(ooms []stacks.OOMContainer) []stacks.OOMContainer {
next := make(map[string]string, len(ooms))
var out []stacks.OOMContainer
for _, o := range ooms {
k := oomLogKey(o)
next[o.Container] = k
if prev, ok := g.last[o.Container]; ok && prev == k {
continue
}
out = append(out, o)
}
g.last = next
return out
}
+64
View File
@@ -0,0 +1,64 @@
package main
import (
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
)
// R-1341: the „[WARN] [deadapp] … was OOM-killed" line goes out once per NEW kill, not every 30-s scan.
//
// RED-PROOF (2026-10-29): with oomLogGate.fresh returning its input unchanged (the pre-R-1341 loop logged every
// container every scan), TestR1341_ASecondScanOfTheSameKillLogsNothing fails on „logged 1 line(s), want 0".
func oom(container, started string, kills int64) stacks.OOMContainer {
return stacks.OOMContainer{Stack: "gokapi", Container: container, StartedAt: started, Kills: kills}
}
func TestR1341_ASecondScanOfTheSameKillLogsNothing(t *testing.T) {
g := &oomLogGate{}
scan := []stacks.OOMContainer{oom("gokapi", "2026-10-29T09:02:00Z", 1)}
if got := g.fresh(scan); len(got) != 1 {
t.Fatalf("the first sighting must log, logged %d line(s)", len(got))
}
for i := 0; i < 5; i++ {
if got := g.fresh(scan); len(got) != 0 {
t.Fatalf("scan %d of the same kill logged %d line(s), want 0", i+2, len(got))
}
}
}
func TestR1341_ANewKillLogsAgain(t *testing.T) {
g := &oomLogGate{}
g.fresh([]stacks.OOMContainer{oom("gokapi", "2026-10-29T09:02:00Z", 1)})
got := g.fresh([]stacks.OOMContainer{oom("gokapi", "2026-10-29T09:02:00Z", 2)})
if len(got) != 1 || got[0].Kills != 2 {
t.Fatalf("a new kill (count 1→2) must log once, got %+v", got)
}
if got := g.fresh([]stacks.OOMContainer{oom("gokapi", "2026-10-29T09:02:00Z", 2)}); len(got) != 0 {
t.Fatalf("the same new kill must not log twice, got %+v", got)
}
}
func TestR1341_ARestartedContainerLogsAgain(t *testing.T) {
g := &oomLogGate{}
g.fresh([]stacks.OOMContainer{oom("gokapi", "2026-10-29T09:02:00Z", 1)})
if got := g.fresh([]stacks.OOMContainer{oom("gokapi", "2026-10-29T09:30:00Z", 1)}); len(got) != 1 {
t.Fatalf("a restarted container's kill must log, got %+v", got)
}
}
// Two containers are judged apart; one that leaves the scan is forgotten, so its return logs again.
func TestR1341_ContainersAreJudgedApart(t *testing.T) {
g := &oomLogGate{}
a, b := oom("a", "t1", 1), oom("b", "t1", 1)
if got := g.fresh([]stacks.OOMContainer{a, b}); len(got) != 2 {
t.Fatalf("first scan: %+v", got)
}
if got := g.fresh([]stacks.OOMContainer{a}); len(got) != 0 {
t.Fatalf("a seen again must not log: %+v", got)
}
if got := g.fresh([]stacks.OOMContainer{a, b}); len(got) != 1 || got[0].Container != "b" {
t.Fatalf("b came back after leaving the scan: %+v", got)
}
}
+10 -2
View File
@@ -17,18 +17,26 @@ import (
// executor is built earlier than the trigger, so it reads it through this holder.
var opActionReportFire atomic.Value
// operatorActionHandlers wires the four operator actions (R-314/R-279/R-177, `09` §3 decision 185)
// operatorActionHandlers wires the five operator actions (R-314/R-279/R-177, `09` §3 decision 185)
// to the running process. ONE wiring site, called from main and from the wiring test, so the test
// exercises the doors production uses (the seam-built-but-never-wired trap).
//
// offsite_backup_now makes the SAME four checks as the household's „back up now"
// (offboxRunHandler) and then runs the same call, so an operator press can never do what the
// household's own press cannot.
func operatorActionHandlers(backupMgr *backup.Manager, sched *scheduler.Scheduler, resultReady func(), logger *log.Logger) report.OperatorActionHandlers {
//
// clear_hold (D64 = B) goes through clearHoldNow with hold — the same door the box command's socket uses
// (holdclear.go), so the hub's button and the box command can never differ. hold.clear nil = not available.
func operatorActionHandlers(backupMgr *backup.Manager, sched *scheduler.Scheduler, hold holdClearDeps, resultReady func(), logger *log.Logger) report.OperatorActionHandlers {
h := report.OperatorActionHandlers{ResultReady: resultReady}
if sched != nil {
h.RunJob = sched.RunNow
}
if hold.clear != nil {
h.ClearHold = func(stack string, actionID int64) (string, error) {
return clearHoldNow(hold, stack, fmt.Sprintf("the operator via the hub, action #%d", actionID))
}
}
if backupMgr == nil {
return h
}
@@ -57,7 +57,7 @@ func TestD1_AbandonStopActsOnTheRunningManagerAndDisk(t *testing.T) {
t.Fatal("fixture: a countdown must be running")
}
o := report.NewOperatorActions(context.Background(), operatorActionHandlers(mgr, nil, nil, logger), logger)
o := report.NewOperatorActions(context.Background(), operatorActionHandlers(mgr, nil, holdClearDeps{}, nil, logger), logger)
o.Reconcile([]report.OperatorAction{{ID: 1, Action: report.OpAbandonStop}})
if mgr.AbandonStatus().Active {
@@ -97,7 +97,7 @@ func TestD1_OffsiteBackupNowRefusesWithoutATarget(t *testing.T) {
t.Fatal("no restic may run")
return nil, nil
})
o := report.NewOperatorActions(context.Background(), operatorActionHandlers(mgr, nil, nil, logger), logger)
o := report.NewOperatorActions(context.Background(), operatorActionHandlers(mgr, nil, holdClearDeps{}, nil, logger), logger)
o.Reconcile([]report.OperatorAction{{ID: 9, Action: report.OpOffsiteBackupNow}})
if r := o.Results(); len(r) != 1 || r[0].Outcome != report.OutcomeRefused {
t.Fatalf("result = %+v", r)
@@ -115,7 +115,7 @@ func TestD1_RunJobUsesTheScheduler(t *testing.T) {
defer cancel()
sched.Start(ctx)
defer sched.Stop()
o := report.NewOperatorActions(ctx, operatorActionHandlers(nil, sched, nil, logger), logger)
o := report.NewOperatorActions(ctx, operatorActionHandlers(nil, sched, holdClearDeps{}, nil, logger), logger)
o.Reconcile([]report.OperatorAction{{ID: 1, Action: report.OpRunJob, Arg: "fill-watch"}, {ID: 2, Action: report.OpRunJob, Arg: "offsite-proof"}})
select {
case od := <-ran:
@@ -119,7 +119,8 @@ func (g *AppStopGuard) holdPutBackFailed(stack, dir string, cause error) bool {
return false
}
h := settings.RestoreHold{Stack: stack, At: g.now().UTC().Format(time.RFC3339), Reason: settings.HoldReasonRestoreMixed,
ReplayError: "interrupted restore, put-back failed: " + cause.Error(), SafetyDump: dir}
ReplayError: "interrupted restore, put-back failed: " + cause.Error(), SafetyDump: dir,
MixedCause: settings.MixedCauseInterruptedPutBack, FailedStep: "putting the interrupted restore's copy back failed: " + cause.Error()}
if err := g.holds.SetRestoreHold(h); err != nil {
g.logger.Printf("[ERROR] [appstop] %s: the failed put-back could not be recorded as a hold: %v — the marker is kept", stack, err)
return false
+66
View File
@@ -0,0 +1,66 @@
package backup
import (
"errors"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// ── D64 = B (operator ruling 2026-10-29 12:10) + R-1343: the operator clears a hold IN THE RUNNING CONTROLLER ──────
//
// One door for both of the operator's routes — the hub's „Clear hold" (operator action clear_hold) and the box command
// `--clear-restore-hold`, which now asks the running controller over its local socket (cmd/controller/holdctl.go).
//
// WHY IN THIS PROCESS (R-1343, measured 2026-10-29 on Tester 1): the CLI used to run as a SECOND process that edited
// settings.json while the running controller kept the hold in its own in-memory Settings — and that controller's next
// save (after a hub report) wrote the hold back. Clearing through the running Settings removes the hold from memory
// AND saves the file at once, so no later save of any component can bring it back.
// Pinned by TestD64_ClearHoldStaysGoneAfterALaterSave (cmd/controller).
//
// EVERY REASON (D64 = B): the hold's reason does not matter — R-379 now reads „a person — on the box, or the operator
// on the hub". One reason needs a second act: a put-back WAITING for its drive (HoldReasonPutBackPending) is also
// recorded in the app-stop marker, and a controller start with the drive bound would pour that old copy back over
// whatever the app wrote after the clear. So the clear also drops the app from the marker; the copy stays on disk
// (nothing is deleted). It runs under resumeMu, so it can never interleave with a put-back of the same app.
// Pinned by TestD64_ClearPutBackPendingDropsTheMarkerEntry.
// ErrHoldBadStack: the argument is not a valid stack name. ErrNotHeld: the app has no hold. Both are refusals —
// nothing was changed.
var (
ErrHoldBadStack = errors.New("not a valid app name")
ErrNotHeld = errors.New("this app has no hold")
)
// IsValidHoldStack reports whether s may name an app for the hold commands (the backup package's strict
// isSafeStackName: letters, digits, '-' and '_', at most 64).
func IsValidHoldStack(s string) bool { return isSafeStackName(s) }
// ClearHoldByOperator lifts stack's hold, whatever its reason, in the hold store the running controller uses, and
// returns the hold it lifted. A waiting put-back is dropped from the app-stop marker too (see above). Refusals
// (ErrHoldBadStack, ErrNotHeld) change nothing.
func (g *AppStopGuard) ClearHoldByOperator(stack string) (settings.RestoreHold, error) {
if !isSafeStackName(stack) {
return settings.RestoreHold{}, ErrHoldBadStack
}
if g == nil || g.holds == nil {
return settings.RestoreHold{}, errors.New("no hold store is wired on this box")
}
resumeMu.Lock()
defer resumeMu.Unlock()
h, ok := g.holds.GetRestoreHold(stack)
if !ok {
return settings.RestoreHold{}, ErrNotHeld
}
cleared, err := g.holds.ClearRestoreHold(stack)
if err != nil {
return h, err
}
if !cleared {
return settings.RestoreHold{}, ErrNotHeld
}
if h.Reason == settings.HoldReasonPutBackPending {
g.dropFromMarker(stack)
g.logger.Printf("[WARN] [appstop] %s: its waiting put-back is DROPPED with the hold — the copy stays on disk at %s and is never poured back by itself", stack, h.SafetyDump)
}
return h, nil
}
@@ -425,9 +425,12 @@ func (m *Manager) holdAppAfterFailedRollback(stack string, replayErr, rollbackEr
m.logger.Printf("[ERROR] [offbox] %s: cannot persist the restore hold — no settings wired; the app is stopped but NOTHING will refuse a restart", stack)
return
}
// R-1340: this branch is only ever an off-site restore's database replay (offbox_reconstitute's rollback).
h := settings.RestoreHold{
Stack: stack,
At: time.Now().UTC().Format(time.RFC3339),
Stack: stack,
At: time.Now().UTC().Format(time.RFC3339),
RestoreKind: settings.RestoreKindOffsite,
DBReplay: true,
}
if replayErr != nil {
h.ReplayError = replayErr.Error()
@@ -447,6 +450,15 @@ func (m *Manager) holdAppAfterFailedRollback(stack string, replayErr, rollbackEr
}
}
// mixedHoldWhy (R-1340) is what a restore_mixed hold records about its cause (settings.RestoreHold's R-1340 fields).
type mixedHoldWhy struct {
kind string // settings.RestoreKind*
cause string // settings.MixedCause*
step string // the step that failed, with its error
dbReplay bool // the restore's own failure was its database replay
copyDir string // where the copy of the live volumes is kept; "" = none kept
}
// liveDefinition is the app's definition as it ran before an off-site restore wrote the snapshot's
// (R-893): a copy of its compose files in a temporary folder, and its full env held in memory only.
type liveDefinition struct {
@@ -511,7 +523,11 @@ func (m *Manager) captureLiveDefinition(stack string) (*liveDefinition, error) {
// written back (when the snapshot's was written), the database service is stopped again, and the app is
// held with HoldReasonRestoreMixed. The operator is notified with no rollback error — that is how the
// notification tells this case from R-379's double failure.
func (m *Manager) holdAppAfterMixedRestore(stack string, replayErr error, live *liveDefinition) {
//
// why (R-1340) is what the hold records about its cause, so the operator's mail tells the real story — this hold is
// written by three different branches now (a definition write, a put-back that failed at one of its steps), not
// only by option C's database replay.
func (m *Manager) holdAppAfterMixedRestore(stack string, replayErr error, live *liveDefinition, why mixedHoldWhy) {
// The hold is PERSISTED FIRST (security review 2026-10-08): a controller that dies between the stop and
// the hold would otherwise restart the app from the R-166 marker with nothing refusing it. When the hold
// cannot be persisted, the app-stop marker is KEPT, so nothing reads the window as finished.
@@ -519,7 +535,8 @@ func (m *Manager) holdAppAfterMixedRestore(stack string, replayErr error, live *
if m.settings == nil {
m.logger.Printf("[ERROR] [offbox] %s: cannot persist the restore hold — no settings wired; the app is stopped but NOTHING will refuse a restart", stack)
} else {
h := settings.RestoreHold{Stack: stack, At: time.Now().UTC().Format(time.RFC3339), Reason: settings.HoldReasonRestoreMixed}
h := settings.RestoreHold{Stack: stack, At: time.Now().UTC().Format(time.RFC3339), Reason: settings.HoldReasonRestoreMixed,
RestoreKind: why.kind, MixedCause: why.cause, FailedStep: why.step, DBReplay: why.dbReplay, SafetyDump: why.copyDir}
if replayErr != nil {
h.ReplayError = replayErr.Error()
}
@@ -557,7 +574,8 @@ func (m *Manager) restartAtLiveDefinition(stack string, live *liveDefinition, ca
if live != nil {
if err := m.stackProvider.RecreateStackDefinitionFromUnit(stack, live.dir, live.env); err != nil {
m.logger.Printf("[ERROR] [offbox] %s: the definition write failed (%v) and writing the live definition back failed too: %v — holding the app stopped", stack, cause, err)
m.holdAppAfterMixedRestore(stack, cause, nil)
m.holdAppAfterMixedRestore(stack, cause, nil, mixedHoldWhy{kind: settings.RestoreKindOffsite, cause: settings.MixedCauseDefinitionWrite,
step: "writing the live app definition back failed: " + err.Error()})
return
}
m.logger.Printf("[INFO] [offbox] %s: the definition write failed (%v); the live definition is written back before the start", stack, cause)
@@ -1060,7 +1078,7 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack
// R-893 option A: no volume has moved yet, so only the live definition is written back and the app starts.
// The failing copy may have placed part of its files, so the sentence names placed files (+1).
if definitionMoved {
return res, m.putBackAfterFailedRestore(stack, cErr, liveDef, undo, false, res.FilesPlaced+1, restartStack, "err.backup.restore_failed_held_mixed")
return res, m.putBackAfterFailedRestore(stack, cErr, liveDef, undo, false, res.FilesPlaced+1, restartStack, "err.backup.restore_failed_held_mixed", settings.RestoreKindOffsite)
}
// Nothing moved yet: best-effort bring-up — a failed restore must not also be an outage.
if sErr := restartStack(); sErr != nil {
@@ -1095,7 +1113,7 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack
// R-893 option A: the replay removes a volume BEFORE it recreates it, so even nVols == 0 may have lost one —
// with a copy, it is always poured back; with no copy the old rule decides (and holds when one moved).
if definitionMoved || nVols > 0 || undo != nil {
return res, m.putBackAfterFailedRestore(stack, vErr, liveDef, undo, nVols > 0 || undo != nil, res.FilesPlaced, restartStack, "err.backup.restore_failed_held_mixed")
return res, m.putBackAfterFailedRestore(stack, vErr, liveDef, undo, nVols > 0 || undo != nil, res.FilesPlaced, restartStack, "err.backup.restore_failed_held_mixed", settings.RestoreKindOffsite)
}
// Nothing replaced yet: bring the app back up rather than leaving an outage, then surface it.
if sErr := restartStack(); sErr != nil {
@@ -1114,7 +1132,7 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack
if err := m.stackProvider.StartStackServices(stack, dbServices); err != nil {
// R-893 option C (security review 2026-10-08): volumes or the definition already moved → HOLD.
if definitionMoved || res.VolumesReplayed > 0 {
return res, m.putBackAfterFailedRestore(stack, err, liveDef, undo, res.VolumesReplayed > 0, res.FilesPlaced, restartStack, "err.backup.restore_failed_held_mixed")
return res, m.putBackAfterFailedRestore(stack, err, liveDef, undo, res.VolumesReplayed > 0, res.FilesPlaced, restartStack, "err.backup.restore_failed_held_mixed", settings.RestoreKindOffsite)
}
// Best-effort bring-up: a failed restore must not also be an outage.
if sErr := restartStack(); sErr != nil {
@@ -1166,7 +1184,7 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack
// under them — the copy of the live volumes is poured back over the rolled-back database, the live
// definition written back, and the app starts exactly as it was. A put-back that fails holds, as before.
if definitionMoved || res.VolumesReplayed > 0 {
return res, m.putBackAfterFailedRestore(stack, iErr, liveDef, undo, res.VolumesReplayed > 0, res.FilesPlaced, restartStack, "err.backup.db_restore_failed_held_mixed")
return res, m.putBackAfterFailedRestore(stack, iErr, liveDef, undo, res.VolumesReplayed > 0, res.FilesPlaced, restartStack, "err.backup.db_restore_failed_held_mixed", settings.RestoreKindOffsite)
}
if sErr := restartStack(); sErr != nil {
m.logger.Printf("[WARN] [offbox] %s: start after a successful rollback failed: %v", stack, sErr)
@@ -11,6 +11,7 @@ import (
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/dockerexec"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
"gitea.dooplex.hu/admin/felhom-controller/internal/util"
)
@@ -219,11 +220,23 @@ func (m *Manager) putBack(stack string, c *liveVolumeCopy) error {
// failure's own held sentence) and the copy stays on disk. volumesMoved with no copy is such a case: nothing can
// put them back. filesPlaced chooses the household's sentence: placed files are not taken back (as ruled for
// option C, a file mix does not hold), so the sentence then says so instead of „nothing changed".
//
// restoreKind (R-1340) is settings.RestoreKindOffsite or RestoreKindLocal: the hold records it, with the step that
// failed and whether the restore's failure was a database replay (holdKey says so — it is the callers' own choice of
// the household's sentence), so the operator's mail names the real cause.
func (m *Manager) putBackAfterFailedRestore(stack string, cause error, live *liveDefinition, c *liveVolumeCopy, volumesMoved bool,
filesPlaced int, restart func() error, holdKey string) error {
filesPlaced int, restart func() error, holdKey, restoreKind string) error {
why := func(step string) mixedHoldWhy {
w := mixedHoldWhy{kind: restoreKind, cause: settings.MixedCausePutBack, step: step,
dbReplay: holdKey == "err.backup.db_restore_failed_held_mixed"}
if c != nil && c.keep {
w.copyDir = c.dir
}
return w
}
if volumesMoved && c == nil {
m.logger.Printf("[ERROR] [offbox] %s: volumes were replaced and there is no copy to put back — holding the app", stack)
m.holdAppAfterMixedRestore(stack, cause, live)
m.holdAppAfterMixedRestore(stack, cause, live, why("the restore had already changed the app's data and there was no copy of the live volumes to put back"))
return util.MsgError(holdKey, stack)
}
// R-1256 H5: a put-back under an app that could not be stopped could mix the copy with what the app writes.
@@ -231,14 +244,14 @@ func (m *Manager) putBackAfterFailedRestore(stack string, cause error, live *liv
if err := m.stackProvider.StopStack(stack); err != nil {
c.keepIfAny()
m.logger.Printf("[ERROR] [offbox] %s: stopping the app before the put-back FAILED: %v — nothing is put back; the copy stays at %s; holding the app", stack, err, c.dirOrNone())
m.holdAppAfterMixedRestore(stack, cause, live)
m.holdAppAfterMixedRestore(stack, cause, live, why("stopping the app before the put-back failed, so nothing was put back: "+err.Error()))
return util.MsgError(holdKey, stack)
}
if volumesMoved {
if err := m.putBack(stack, c); err != nil {
c.keep = true
m.logger.Printf("[ERROR] [offbox] %s: putting the live volumes back FAILED: %v — the copy stays at %s; holding the app", stack, err, c.dir)
m.holdAppAfterMixedRestore(stack, cause, live)
m.holdAppAfterMixedRestore(stack, cause, live, why("putting the live volumes back failed: "+err.Error()))
return util.MsgError(holdKey, stack)
}
}
@@ -246,7 +259,7 @@ func (m *Manager) putBackAfterFailedRestore(stack string, cause error, live *liv
if err := m.stackProvider.RecreateStackDefinitionFromUnit(stack, live.dir, live.env); err != nil {
c.keepIfAny() // R-1256 H5 (D20): a put-back that fails keeps its copy for support
m.logger.Printf("[ERROR] [offbox] %s: writing the live definition back FAILED: %v — the copy stays at %s; holding the app", stack, err, c.dirOrNone())
m.holdAppAfterMixedRestore(stack, cause, live)
m.holdAppAfterMixedRestore(stack, cause, live, why("writing the live app definition back failed: "+err.Error()))
return util.MsgError(holdKey, stack)
}
}
@@ -0,0 +1,133 @@
package backup
import (
"context"
"errors"
"io"
"log"
"os"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-1340: a restore_mixed hold records WHAT put it in place — which restore, which branch, which put-back step failed,
// whether the restore's failure was a database replay, and where the copy is kept — so the operator's mail
// (cmd/controller heldStoppedOperatorText, pinned by TestR1340_* there) tells the real story. Measured 2026-10-29 on
// Tester 1: Radicale (no database), its volume could not be removed, the put-back's stop failed, and the mail said its
// DATABASE restore failed and was rolled back.
//
// RED-PROOF (2026-10-29): with the R-1340 fields dropped from holdAppAfterMixedRestore's hold literal and from
// holdPutBackFailed's (the pre-R-1340 record, which carried no cause), all five tests below fail on „mixed cause """ /
// „kind """.
func holdOf(t *testing.T, m *Manager, stack string) settings.RestoreHold {
t.Helper()
if m.settings == nil {
t.Fatal("fixture: no settings wired")
}
h, ok := m.settings.GetRestoreHold(stack)
if !ok {
t.Fatalf("%s is not held", stack)
}
return h
}
// The Radicale shape: a VOLUME restore fails (no database involved), then the put-back's stop fails.
func TestR1340_AVolumeFailureWhosePutBackStopFailsRecordsTheStepAndNoDatabase(t *testing.T) {
m, vp, rig, _, notified, _ := r893PutBackFixture(t)
rig.snapReplayErr = errors.New("volume rm -f immich_pgdata: volume is in use")
p := &r1256StopFailProvider{r893Provider: vp, failAt: 2} // 1 = the restore's stop, 2 = the put-back's
m.SetStackProvider(p)
if _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false); err == nil {
t.Fatal("a failed restore must be surfaced")
}
h := holdOf(t, m, "immich")
if h.Reason != settings.HoldReasonRestoreMixed || h.MixedCause != settings.MixedCausePutBack {
t.Fatalf("reason %q, mixed cause %q — want restore_mixed / put_back", h.Reason, h.MixedCause)
}
if !strings.Contains(h.FailedStep, "stopping the app before the put-back failed") || !strings.Contains(h.FailedStep, "compose down: timeout") {
t.Errorf("failed step %q — want the stop and its error", h.FailedStep)
}
if h.DBReplay {
t.Error("a volume failure must not be recorded as a database replay")
}
if h.RestoreKind != settings.RestoreKindOffsite {
t.Errorf("restore kind %q, want offsite", h.RestoreKind)
}
if h.SafetyDump == "" {
t.Error("the kept copy's folder must be recorded")
} else if _, err := os.Stat(h.SafetyDump); err != nil {
t.Errorf("the recorded copy %s is not on disk: %v", h.SafetyDump, err)
}
if *notified != 1 {
t.Errorf("hold notices %d, want 1", *notified)
}
}
// The database replay fails (option C's shape), then the put-back's stop fails: DBReplay is recorded.
func TestR1340_ADatabaseReplayFailureRecordsTheDatabase(t *testing.T) {
m, vp, _, _, _, _ := r893PutBackFixture(t)
m.SetStackProvider(&r1256StopFailProvider{r893Provider: vp, failAt: 2})
if _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false); err == nil {
t.Fatal("a failed restore must be surfaced")
}
h := holdOf(t, m, "immich")
if h.MixedCause != settings.MixedCausePutBack || !h.DBReplay {
t.Fatalf("mixed cause %q, db replay %v — want put_back and true", h.MixedCause, h.DBReplay)
}
}
// R-1256 H4: the backup's definition cannot be written and the live one cannot be written back — no data moved.
func TestR1340_ADefinitionWriteFailureRecordsItsCause(t *testing.T) {
m, _, _ := r1256DefFailFixture(t, 2)
if _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false); err == nil {
t.Fatal("a failed definition write must be surfaced")
}
h := holdOf(t, m, "immich")
if h.MixedCause != settings.MixedCauseDefinitionWrite || h.DBReplay {
t.Fatalf("mixed cause %q, db replay %v — want definition_write and false", h.MixedCause, h.DBReplay)
}
if !strings.Contains(h.FailedStep, "writing the live app definition back failed") {
t.Errorf("failed step %q", h.FailedStep)
}
}
// A LOCAL restore (from the app's own backup) records itself as local, never off-site.
func TestR1340_ALocalRestoreRecordsItsKind(t *testing.T) {
m, _, rig, _ := r1177Fixture(t)
if m.settings == nil {
sett, err := settings.Load(t.TempDir()+"/settings.json", log.New(io.Discard, "", 0))
if err != nil {
t.Fatal(err)
}
m.settings = sett
}
rig.unitErr = errors.New("tar: short read")
rig.putBackErr = errors.New("tar xf failed")
if _, err := m.RestoreFromRecoveryUnit("app"); err == nil {
t.Fatal("a failed put-back must be surfaced")
}
h := holdOf(t, m, "app")
if h.RestoreKind != settings.RestoreKindLocal || h.MixedCause != settings.MixedCausePutBack || h.DBReplay {
t.Fatalf("kind %q, cause %q, db %v — want local / put_back / false", h.RestoreKind, h.MixedCause, h.DBReplay)
}
if !strings.Contains(h.FailedStep, "putting the live volumes back failed") || !strings.Contains(h.FailedStep, "tar xf failed") {
t.Errorf("failed step %q", h.FailedStep)
}
}
// A restore cut by a stop whose put-back fails at the next start (R-1251 H2) records its own cause.
func TestR1340_AnInterruptedPutBackRecordsItsCause(t *testing.T) {
r := newR1251Rig(t)
r.putErr = errors.New("compose down: timeout")
r.guard().Recover()
h, ok := r.sett.GetRestoreHold("immich")
if !ok || h.Reason != settings.HoldReasonRestoreMixed || h.MixedCause != settings.MixedCauseInterruptedPutBack {
t.Fatalf("hold %+v (held %v) — want restore_mixed / interrupted_put_back", h, ok)
}
if !strings.Contains(h.FailedStep, "compose down: timeout") {
t.Errorf("failed step %q", h.FailedStep)
}
}
+2 -1
View File
@@ -3,6 +3,7 @@ package backup
import (
"errors"
"fmt"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
"gitea.dooplex.hu/admin/felhom-controller/internal/util"
"os"
"path/filepath"
@@ -530,7 +531,7 @@ func (m *Manager) RestoreFromRecoveryUnitAtWith(stackName, unitDir string, opt U
// without one (the unit holds no volume) only a database replay that began can have changed data — and with no
// copy to pour back that is the R-379 half state: held, never started (operator ruling 2026-08-22).
putBack := func(cause error, holdKey string, moved bool) (UnitRestoreResult, error) {
return res, m.putBackAfterFailedRestore(stackName, cause, liveDef, undo, moved || undo != nil, opt.FilesPlaced, restartStack, holdKey)
return res, m.putBackAfterFailedRestore(stackName, cause, liveDef, undo, moved || undo != nil, opt.FilesPlaced, restartStack, holdKey, settings.RestoreKindLocal)
}
// R-353: the count is captured even when the replay errors — a partial replay is a fact the
// customer's sentence has to be built from, and discarding it on the error path is how Scenario C
+42 -4
View File
@@ -21,9 +21,11 @@ import (
// problem, main.go). The result rides the next report as `operator_action_results: [{id, outcome,
// message}]`; the hub lists an action until its result has arrived, then stops.
//
// THE LIST IS CLOSED. Four actions, and run_job names one of four jobs. Nothing here deletes data,
// THE LIST IS CLOSED. Five actions, and run_job names one of four jobs. Nothing here deletes data,
// starts a countdown or shortens one — `03` §4 asks for a signing key only to destroy or overwrite the
// only copy, and none of these does. An unknown action, an unknown job or an argument out of range is
// only copy, and none of these does. clear_hold (D64 = B, operator ruling 2026-10-29 12:10) lifts an app's
// hold of ANY reason with no operator signature: it deletes nothing and starts nothing — the app stays
// stopped until the household (or the operator) starts it, and a waiting put-back's copy stays on disk. An unknown action, an unknown job or an argument out of range is
// answered `refused` and NOTHING is called. Pinned by TestOpActions_ClosedList (which fails when an
// entry is added, so a new action is a decision, not an edit) and TestOpActions_UnknownIsRefused.
//
@@ -38,6 +40,7 @@ const (
OpAbandonStop = "abandon_stop"
OpAbandonExtend = "abandon_extend"
OpRunJob = "run_job"
OpClearHold = "clear_hold"
OutcomeDone = "done"
OutcomeRefused = "refused"
@@ -57,7 +60,7 @@ const (
)
// operatorActionNames is THE closed list. Adding an entry fails TestOpActions_ClosedList by design.
var operatorActionNames = []string{OpOffsiteBackupNow, OpAbandonStop, OpAbandonExtend, OpRunJob}
var operatorActionNames = []string{OpOffsiteBackupNow, OpAbandonStop, OpAbandonExtend, OpRunJob, OpClearHold}
// operatorJobNames is the fixed set run_job may name — the scheduler's own job names (main.go).
var operatorJobNames = []string{"fill-watch", "offsite-integrity", "offsite-proof", "disk-health-check"}
@@ -83,6 +86,21 @@ func contains(list []string, v string) bool {
return false
}
// validHoldStack is clear_hold's argument check: the same alphabet as backup.isSafeStackName (report cannot import
// backup; REUSE.md „Safe-name validators").
func validHoldStack(s string) bool {
if s == "" || len(s) > 64 {
return false
}
for _, c := range s {
if (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '-' || c == '_' {
continue
}
return false
}
return true
}
// ErrActionRefused, wrapped by a handler, turns its error into a `refused` outcome (the action could
// not START — e.g. an off-site run already in flight) instead of `failed` (it started and went wrong).
var ErrActionRefused = errors.New("refused")
@@ -101,7 +119,7 @@ type OperatorActionResult struct {
Message string `json:"message"`
}
// OperatorActionHandlers are the four doors, wired by main.go. A nil handler answers `refused`
// OperatorActionHandlers are the five doors, wired by main.go. A nil handler answers `refused`
// („not available on this box").
type OperatorActionHandlers struct {
// OffsiteBackupNow returns a refusal sentence when the run may not start (the household's
@@ -111,6 +129,9 @@ type OperatorActionHandlers struct {
AbandonExtend func(days int) (time.Time, error)
// RunJob is Scheduler.RunNow: an error = not started (unknown/running); the channel = the job's end.
RunJob func(name string) (<-chan error, error)
// ClearHold lifts the app's hold in the RUNNING controller (D64 = B, R-1343) and returns the result
// sentence. actionID names the action in the box's log. An error wrapping ErrActionRefused = nothing changed.
ClearHold func(stack string, actionID int64) (string, error)
// ResultReady sends an out-of-cycle report so a result reaches the hub in seconds (optional).
ResultReady func()
}
@@ -317,6 +338,23 @@ func (o *OperatorActions) dispatch(a OperatorAction) {
}
o.finish(a, outcomeOf(jerr), msg)
}()
case OpClearHold:
// The argument is checked HERE, before any handler is touched (the closed list's rule): one app name,
// the backup package's strict alphabet (letters, digits, '-' and '_', at most 64).
if !validHoldStack(a.Arg) {
o.finish(a, OutcomeRefused, "the argument must be one app name (letters, digits, '-' and '_')")
return
}
if o.h.ClearHold == nil {
o.finish(a, OutcomeRefused, "not available on this box")
return
}
msg, err := o.h.ClearHold(a.Arg, a.ID)
if err != nil {
o.finish(a, outcomeOf(err), err.Error())
return
}
o.finish(a, OutcomeDone, msg)
default:
o.finish(a, OutcomeRefused, "unknown action")
}
+79 -2
View File
@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"errors"
"fmt"
"reflect"
"strings"
"sync"
@@ -15,9 +16,11 @@ import (
// The list is CLOSED. This test fails when an action or a job is added, by design: a new entry is an
// operator decision (no action may delete data, start a countdown or shorten one), not an edit. The
// hub pins the same four in its own test (hub internal/store opactions_test.go).
// hub pins the same list in its own test (hub internal/store opactions_test.go).
// 2026-10-29: clear_hold added deliberately — operator ruling D64 = B (12:10): the hub's „Clear hold" works for
// every hold reason with no operator signature. It deletes nothing and starts nothing.
func TestOpActions_ClosedList(t *testing.T) {
if got, want := OperatorActionNames(), []string{"abandon_extend", "abandon_stop", "offsite_backup_now", "run_job"}; !reflect.DeepEqual(got, want) {
if got, want := OperatorActionNames(), []string{"abandon_extend", "abandon_stop", "clear_hold", "offsite_backup_now", "run_job"}; !reflect.DeepEqual(got, want) {
t.Fatalf("operator actions = %v, want exactly %v — a new action needs the operator's word (decision 185)", got, want)
}
if got, want := OperatorJobNames(), []string{"disk-health-check", "fill-watch", "offsite-integrity", "offsite-proof"}; !reflect.DeepEqual(got, want) {
@@ -56,6 +59,10 @@ func recordingHandlers(c *calls) OperatorActionHandlers {
close(ch)
return ch, nil
},
ClearHold: func(stack string, id int64) (string, error) {
c.add("clear:" + stack)
return "the hold on " + stack + " is cleared", nil
},
}
}
@@ -197,3 +204,73 @@ func TestOpActions_WireShapes(t *testing.T) {
t.Fatalf("report carried %s", b)
}
}
// clear_hold (D64 = B): the argument is one app name in the strict alphabet — never a path, never empty, never two
// names. A bad one is refused and the handler is NOT called (the closed list's rule: validate before any door).
// RED-PROOF (2026-10-29): with validHoldStack returning true, this fails on „a refused action called [clear:…]".
func TestOpActions_ClearHoldBadArgumentIsRefusedAndCallsNothing(t *testing.T) {
c := &calls{}
o := NewOperatorActions(context.Background(), recordingHandlers(c), nil)
list := []OperatorAction{
{ID: 1, Action: OpClearHold, Arg: ""},
{ID: 2, Action: OpClearHold, Arg: "../settings"},
{ID: 3, Action: OpClearHold, Arg: "immich paperless"},
{ID: 4, Action: OpClearHold, Arg: "immich/x"},
{ID: 5, Action: OpClearHold, Arg: strings.Repeat("a", 65)},
{ID: 6, Action: OpClearHold, Arg: "immich\n"},
{ID: 7, Action: OpClearHold, Arg: "."},
}
o.Reconcile(list)
if got := c.get(); len(got) != 0 {
t.Fatalf("a refused action called %v", got)
}
for _, a := range list {
if r := resultFor(t, o, a.ID); r.Outcome != OutcomeRefused || !strings.Contains(r.Message, "one app name") {
t.Errorf("#%d clear_hold(%q): %+v, want refused naming the rule", a.ID, a.Arg, r)
}
}
// control: a good name does reach the handler
o.Reconcile([]OperatorAction{{ID: 8, Action: OpClearHold, Arg: "paperless-ngx_2"}})
if got := c.get(); len(got) != 1 || got[0] != "clear:paperless-ngx_2" {
t.Fatalf("a valid name must reach the handler, calls %v", got)
}
}
// clear_hold (D64 = B): a valid name reaches the handler with the action's id, and its sentence is the result; a
// handler refusal (the app has no hold) is answered refused, an error that is not a refusal failed.
// RED-PROOF (2026-10-29): with the OpClearHold case removed from dispatch, this fails on „unknown action".
func TestOpActions_ClearHoldDoneRefusedFailed(t *testing.T) {
var gotStack string
var gotID int64
h := OperatorActionHandlers{ClearHold: func(stack string, id int64) (string, error) {
gotStack, gotID = stack, id
switch stack {
case "notheld":
return "", fmt.Errorf("%w: notheld has no hold — nothing was changed", ErrActionRefused)
case "broken":
return "", errors.New("saving settings: disk full")
}
return "the hold on " + stack + " is cleared (it was: held after a failed update)", nil
}}
o := NewOperatorActions(context.Background(), h, nil)
o.Reconcile([]OperatorAction{{ID: 7, Action: OpClearHold, Arg: "kimai"}})
if gotStack != "kimai" || gotID != 7 {
t.Fatalf("handler called with (%q, %d), want (kimai, 7)", gotStack, gotID)
}
if r := resultFor(t, o, 7); r.Outcome != OutcomeDone || r.Message != "the hold on kimai is cleared (it was: held after a failed update)" {
t.Fatalf("result %+v", r)
}
o.Reconcile([]OperatorAction{{ID: 8, Action: OpClearHold, Arg: "notheld"}, {ID: 9, Action: OpClearHold, Arg: "broken"}})
if r := resultFor(t, o, 8); r.Outcome != OutcomeRefused || !strings.Contains(r.Message, "has no hold") {
t.Errorf("not held: %+v", r)
}
if r := resultFor(t, o, 9); r.Outcome != OutcomeFailed {
t.Errorf("a save error must be failed, not refused: %+v", r)
}
// nil handler → refused, not a crash
o2 := NewOperatorActions(context.Background(), OperatorActionHandlers{}, nil)
o2.Reconcile([]OperatorAction{{ID: 1, Action: OpClearHold, Arg: "kimai"}})
if r := resultFor(t, o2, 1); r.Outcome != OutcomeRefused {
t.Errorf("nil handler: %+v", r)
}
}
+28
View File
@@ -1982,8 +1982,36 @@ type RestoreHold struct {
// first stop or a repeat within 24 h (Trip >= 2 → the sentence says support is informed).
UnhealthyKind string `json:"unhealthy_kind,omitempty"`
Trip int `json:"trip,omitempty"`
// R-1340 (2026-10-29): what put a restore hold in place, so the operator's „HELD STOPPED" mail tells the real
// story instead of the option-C database story for every cause. All empty on a hold written before them: the mail
// then names only what the record carries (cmd/controller heldStoppedOperatorText).
//
// RestoreKind is which restore: RestoreKindOffsite or RestoreKindLocal.
RestoreKind string `json:"restore_kind,omitempty"`
// MixedCause (restore_mixed only) is the branch that held the app: MixedCause*.
MixedCause string `json:"mixed_cause,omitempty"`
// FailedStep is the put-back step that failed, with its error (MixedCausePutBack).
FailedStep string `json:"failed_step,omitempty"`
// DBReplay: the restore's own failure was its database replay. False = the app's database was never touched
// by the failure, so the mail never says „database".
DBReplay bool `json:"db_replay,omitempty"`
}
// Restore kinds and mixed-hold causes (R-1340). See RestoreHold.RestoreKind / MixedCause.
const (
RestoreKindOffsite = "offsite"
RestoreKindLocal = "local"
// MixedCauseDefinitionWrite: the backup's definition could not be written (no data moved yet) and writing the
// live definition back failed too (R-1256 H4).
MixedCauseDefinitionWrite = "definition_write"
// MixedCausePutBack: the restore failed and putting the live data back failed at FailedStep (D20/R-1256 H5).
MixedCausePutBack = "put_back"
// MixedCauseInterruptedPutBack: a restore cut by a stop could not be put back at the next start (R-1251 H2).
MixedCauseInterruptedPutBack = "interrupted_put_back"
)
// Hold reasons. See RestoreHold.Reason.
const (
HoldReasonRestoreFailed = "" // R-379/R-380: a restore AND its rollback failed
@@ -0,0 +1,46 @@
package stacks
import (
"io"
"log"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
)
// D64 = B / R-1343: when the operator clears a hold, the ONE held flag kept outside the hold store — „the last update
// ended held" — goes too. Left set, the crash-loop observer and the nightly update leg kept skipping an app whose hold
// was gone, until the controller restarted.
// RED-PROOF (2026-10-29): with ForgetUpdateHeld's body reduced to `return false`, this fails on „a held update must be forgotten".
func TestD64_ForgetUpdateHeld(t *testing.T) {
m := &Manager{cfg: &config.Config{}, logger: log.New(io.Discard, "", 0), stacks: map[string]*Stack{
"kimai": {Name: "kimai", Deployed: true},
"running": {Name: "running", Deployed: true, Updating: true},
}}
m.finishUpdateKey("kimai", UpdatePhaseFailed, UpdateErrorKeyHeld, "a kimai leállítva marad")
m.markUpdateHeld("kimai")
m.markUpdateHeld("running")
if !m.ForgetUpdateHeld("kimai") {
t.Fatal("a held update must be forgotten")
}
st, _ := m.GetStack("kimai")
if st.updateHeld || st.UpdateErrorKey != "" || st.UpdateError != "" || st.UpdatePhase != "" {
t.Fatalf("held state left: held=%v key=%q err=%q phase=%q", st.updateHeld, st.UpdateErrorKey, st.UpdateError, st.UpdatePhase)
}
// The two readers of the flag (ObserveUnhealthy's skip, unhealthy.go; legCandidate's LegSkipHeld, unattended.go)
// read the stored stack — not called here, because ObserveUnhealthy reaches docker (R-650).
m.mu.RLock()
skipped := m.stacks["kimai"].updateHeld || m.stacks["kimai"].UpdateErrorKey == UpdateErrorKeyHeld
m.mu.RUnlock()
if skipped {
t.Fatal("the observer and the update leg still skip kimai")
}
// A running update is never touched; an app with nothing held reports false.
if m.ForgetUpdateHeld("running") {
t.Fatal("a running update must not be touched")
}
if m.ForgetUpdateHeld("kimai") || m.ForgetUpdateHeld("nosuch") {
t.Fatal("nothing held → false")
}
}
+19
View File
@@ -209,6 +209,25 @@ func (m *Manager) ClearUpdateState(name string) {
s.HealthProbe = nil
}
// ForgetUpdateHeld (D64 = B, R-1343) forgets that the app's last update ended HELD, once the hold itself is cleared
// by the operator. updateHeld is the ONE held flag kept outside the hold store: it makes the crash-loop observer
// (ObserveUnhealthy) and the nightly update leg (legCandidate) skip the app, and keeps the hold's sentence as the
// update error. Left set, an app whose hold was cleared would stay unobserved and never be updated by night until
// the controller restarts. A running update is not touched. Returns whether anything was forgotten.
// Pinned by TestD64_ForgetUpdateHeld (this package) and TestD64_ClearHoldForgetsTheUpdateHeldFlag (cmd/controller).
func (m *Manager) ForgetUpdateHeld(name string) bool {
m.mu.Lock()
defer m.mu.Unlock()
s, ok := m.stacks[name]
if !ok || s.Updating || (!s.updateHeld && s.UpdateErrorKey != UpdateErrorKeyHeld) {
return false
}
s.updateHeld = false
s.UpdatePhase, s.UpdatePhaseLabel = "", ""
s.UpdateError, s.UpdateErrorKey, s.UpdateErrorArgs = "", "", nil
return true
}
func (m *Manager) markUpdateHeld(name string) {
m.mu.Lock()
if s, ok := m.stacks[name]; ok {