Compare commits

25 Commits
Author SHA1 Message Date
adminandClaude Opus 5.5 27360d6abe v0.351.0 (unreleased): old -rc controller images and old infra images are deleted (decision 56 amended, decision 285)
gates / gates (push) Successful in 31s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-09 10:23:06 +01:00
admin 64b99b74e3 REPORT + CONTEXT: v0.350.0 the global floor; Traefik v3.7.17 proven by behaviour on 9202
gates / gates (push) Successful in 34s
2026-11-03 10:29:37 +01:00
admin 5c336d828c v0.350.0 CHANGELOG heading (MinAgent 0.158.1, unchanged) — the release build's commit; the Traefik load test's figures
gates / gates (push) Successful in 31s
2026-11-03 08:02:35 +01:00
admin 13e34d4ed4 REPORT: 2026-11-02 night, commit only (Traefik pin, R-1347, R-1348, R-1339, R-1351, R-1323)
gates / gates (push) Successful in 30s
2026-11-03 04:13:24 +01:00
admin 37644458c7 R-1348: the mail of an interrupted put-back names a restore, not an off-site run (seen on the live proof); changelog: R-1347, R-1348 proven live on 9202. Commit only.
gates / gates (push) Successful in 32s
2026-11-03 01:12:19 +01:00
admin c4b6844d34 Traefik v3.7.17 (GHSA-xv5x-242w-7hf7); R-1347 a restore forgets the update-held flag; R-1348 a failed put-back at start-up mails; R-1339, R-1351, R-1323 truthful start-up lines and one lock over FileBrowser's compose. Commit only.
gates / gates (push) Successful in 29s
2026-11-02 23:14:16 +01:00
admin a90b62323f REPORT + CONTEXT: v0.349.0 the global floor; busy file, household line, R-1350 proven live
gates / gates (push) Successful in 31s
2026-10-29 20:24:36 +01:00
admin 94049a7200 v0.349.0 CHANGELOG heading (MinAgent 0.158.1, unchanged) — the release build's commit
gates / gates (push) Successful in 31s
2026-10-29 19:18:56 +01:00
admin b69e90bdad D65 = A: the busy file the agent reads before an operator reboot, and the household's 'Felhom support restarted your box' line; R-1350: a hold clear records the wanted stop so no app_start_failed follows it
gates / gates (push) Successful in 32s
2026-10-29 16:24:40 +01:00
admin a28db2d17f REPORT + CONTEXT: v0.348.0 the global floor; clear_hold proven on the release build
gates / gates (push) Successful in 31s
2026-10-29 15:11:17 +01:00
admin 2a0a17a760 v0.348.0: CHANGELOG heading — the release build's commit
gates / gates (push) Successful in 31s
2026-10-29 14:35:30 +01:00
admin 99fac06aa7 R-1340 second producer: a restore_mixed hold's Why() names the cause from its record (found live on rc.1: the hub's Held apps line said off-site/database for a local restore with no database) — test red first
gates / gates (push) Successful in 30s
2026-10-29 14:35:07 +01:00
admin 6077ac0083 R-1343 fallback: the box command's own settings.json edit (no controller running) goes through the same clear door, so a waiting put-back is dropped from the marker there too — test red first
gates / gates (push) Successful in 30s
2026-10-29 12:59:41 +01:00
admin 5b3b541602 clear_hold (D64 = B) + the box command through the running controller (R-1343), R-1340, R-1341 — tests red first; commit only
gates / gates (push) Successful in 31s
- clear_hold: a fifth verb in the closed operator-action list; lifts a hold of any reason in the RUNNING
  controller (memory + settings.json together), forgets the update's held flag, drops a waiting put-back
  from the app-stop marker (copy kept). Refused on a bad name or no hold. No new event type.
- --clear-restore-hold asks the running controller over <data dir>/ctl/hold.sock (unix, 0600, peer uid
  checked, no TCP); edits settings.json only when no controller answers, and says so (in-guest restart).
- R-1340: the operator's HELD STOPPED mail is built from the hold record (restore kind, mixed cause,
  failed put-back step, database replay or not, kept copy) — never the option-C database story by default.
- R-1341: the OOM WARN line once per new kill (container, start time, kill count); event unchanged.
2026-10-29 12:49:46 +01:00
admin c7bb776e00 REPORT: R-1342 after the release (fixed on main 652a690, not released)
gates / gates (push) Successful in 28s
2026-10-29 11:16:36 +01:00
admin 652a690e5d R-1342: a restore copy under the stable-start watch is not deletable (D62 gap); an app removed during the watch raises no alarm — tests red first; commit only
gates / gates (push) Successful in 31s
2026-10-29 11:16:13 +01:00
admin 6c72e3ff34 v0.347.0 RELEASED (the global floor, MinAgent 0.158.1): CHANGELOG heading, CONTEXT, REPORT
gates / gates (push) Successful in 33s
2026-10-29 10:48:22 +01:00
admin fd75c7c717 v0.347.0: CHANGELOG heading (D62, R-1256 all proven live, R-1254 controller half, R-528, 404) — the release build's commit
gates / gates (push) Successful in 30s
2026-10-29 09:55:42 +01:00
admin e3fce09d96 D62: restore copies kept 30 days, listed with their date, deletable by hand (typed name, never a needed copy, operator event); D63: a test-build-only fault switch for the R-1256 H4/H5 live proofs — tests red first
gates / gates (push) Successful in 33s
2026-10-29 08:50:59 +01:00
admin a815b37fbd Night 2026-10-28 docs: REPORT (H5 + H8 live on 9202, teardown), README (restore edges, host OOM count — unreleased), CONTEXT, CHANGELOG live line
gates / gates (push) Successful in 29s
2026-10-29 05:14:05 +01:00
admin 20c5877056 An action on a name that is no app answers 404, not 'settings cannot be saved' (found walking capability map B2 on 9202)
gates / gates (push) Successful in 27s
Test TestAnActionOnANameThatIsNoAppIsA404 red first. Commit only.
2026-10-28 23:56:36 +01:00
admin eed23e7594 R-528: the 30-s OOM scan also reads the host's per-container kernel counter (agent GET /guest/oom-kills)
gates / gates (push) Successful in 32s
A count above 0 in a running container's current run fires the existing app_oom line (source cgroup/docker in the
details), whatever Docker's OOMKilled says. Old agent: 404 back-off; no agent: Docker's flag only.
Tests TestR528_* red first / red-proved. Commit only.
2026-10-28 23:22:46 +01:00
admin e33cffaee1 R-1254 (controller half): escrow claim/confirm outwait the agent (60 s); an unknown confirm outcome promises nothing
gates / gates (push) Successful in 28s
504 from the agent (it cannot learn which package the hub holds) or a failed call -> api.escrow.confirm_store_unknown
(hu+en); only 502 keeps 'your previous code still works'. Tests TestR1254_* red before the fix. Commit only.
2026-10-28 22:59:52 +01:00
admin 342a394bff R-1256: four restore edges (review round 2 HUNT-1 H4/H5/H6/H8) — each test seen failing first
gates / gates (push) Successful in 31s
H4: a failed definition write in the off-site restore writes the live definition back before the start;
    a failed write-back holds the app (restore_mixed).
H5: a put-back whose stop fails pours nothing, keeps the copy and holds the app (off-site, local, power-cut);
    a failed definition write-back keeps the copy (D20); a failed 'volume rm -f' fails that volume (no merge).
H6: TryBeginRestoreOp/RestoreBusy: check every operation and record the restore in one step under m.mu;
    every web handler and the kept loads begin through it; install-time kept load asks about the mirror too.
H8: SweepUnusedRestoreCopies every 6 h: removes a copy only with no hold, no marker naming it, no restore
    running, and older than 14 days.

Commit only; rides the next controller release.
2026-10-28 22:24:16 +01:00
admin ed3927f03a REPORT: controller v0.346.0 released; R-1250 fixed the same day
gates / gates (push) Successful in 1m18s
2026-10-27 15:49:59 +01:00
104 changed files with 7258 additions and 182 deletions

No files matched your search

+249
View File
@@ -1,3 +1,252 @@
## v0.351.0 (unreleased, 2026-11-09) — old test builds and old infrastructure images are deleted (decision 56 amended, decision 285)
### 2026-11-09
**MinAgent: 0.158.1** (unchanged).
**Why (measured 2026-11-09, demo-hp guest 9201, `docker system df -v`):** the whole-guest backup copies `/var/lib/docker` (mp0,
`backup=1`), and demo-hp's local tier was refused for space again (felhom.eu R-548). Of 27.5 GB of images, 13.2 GB ran in
containers, 9.8 GB were the apps' undo images (decision 53, kept on purpose), 2.2 GB were 28 old controller images and
1.35 GB were old infrastructure and helper images. Nothing deleted the last two kinds.
**Controller images: a test build is a version (decision 56 amended).** `RetainControllerImages` left every `-rc` tag alone,
so 25 test builds stayed. Tags, the running version and the previous now parse with `util.ParseVersionPre` (an rc orders
below its release): an old rc goes like an old release; an rc above the running one or between the previous and the
running one stays; `latest` stays. A box that RUNS a test build is judged (it was skipped). The fallback previous prefers
a release over a newer test build. Tests `TestControllerRetention_OldTestBuildsGo`, `_BoxOnATestBuildIsJudged`; the
existing test's rc expectation changed with the rule.
**Infrastructure images (decision 285, new).** `RetainInfraImages(infra.Images())`, 3 minutes after every start: per pinned
repository the pin, newer versions, the previous pin and every in-use image stay; older and untagged ones go by exact
name/ID; never forced, never pruned; not while an app pulls (R-863, retried every 2 minutes, 30 tries) or while the
controller swaps itself. Tests `TestInfraRetention_*` (4). Customer boxes get a new traefik or cloudflared pin every few
weeks (~65–195 MB each); this is the part a customer box needs.
Red-proofs: six code mutants, each test red (the previous-pin keep, the in-use check, the R-863 lock, the tag parser, the
running-version parser, the release preference). Green gate: `go build`, `go vet`, `go test ./...` rc 0; `controller_gates.py --fast` OK.
## v0.350.0 (2026-11-03) — Traefik v3.7.17 (GHSA-xv5x-242w-7hf7); a restore forgets the update-held flag (R-1347); a failed put-back at start-up mails (R-1348); three false start-up lines (R-1339, R-1351, R-1323)
### 2026-11-03
**MinAgent: 0.158.1** (unchanged).
**Load test, scratch 9202 only, from the LAN, 2026-11-03** (felhom.eu `audits/release-2026-11-03/part1/`): the same many short TLS connections with an
`Authorization: NTLM` header to the dashboard route. v3.7.16: Traefik's anonymous memory 18.7 → 79.0 MiB after 920 connections, not released (stopped at
twice its start, as the test's bound says). v3.7.17: 16.2 → 25-26 MiB, flat for 25,116 connections and the whole 300 s.
**Traefik v3.7.17 (GHSA-xv5x-242w-7hf7, high, published 2026-11-02).** v3.7.14–v3.7.16: an unauthenticated client that
sends a `Negotiate` or `NTLM` `Authorization` header gets a dedicated backend transport per connection, and it is not
released when the connection closes: many short connections exhaust Traefik's memory, whether or not an NTLM backend
exists. Traefik fronts every box. `TraefikImage` v3.7.16 → v3.7.17; a running box takes it by the reconcile at start
(`TestEnsureTraefik_ANewImageRecreatesTraefik`). The upgrade notes name Go 1.27 (GODEBUG TLS settings: we set none; 500
request-header values at most, else 431) and the ingress-nginx provider (not used). Found by reading the release itself:
the watch had flagged it on a word-only line (felhom.eu R-1310).
**R-1347 — a restore that lifts an update hold forgets the „update held" flag.** `clearUpdateHoldAfterRestore` cleared the
hold in settings, but `stacks.Manager` kept `updateHeld`: the crash-loop watcher did not watch the app and the night leg
skipped it until the controller restarted. New seam `backup.SetUpdateHeldForgetter` (`stackMgr.ForgetUpdateHeld`). Tests
`TestR1347_*` (the seam, the wiring, and the consequence: the same restarts trip the watcher once the flag is gone).
**R-1348 — a failed put-back at start-up mails the operator.** `holdPutBackFailed` writes the D20 hold at `Recover`, before any
notifier exists, so only the hold reporter's info and the hub's later reminder said anything. `AppStopGuard.SetHeldNotify`
hands over what `Recover` held, once, and mails a later failure (a drive that returned) at once: the R-1340 sentence for
`MixedCauseInterruptedPutBack`, from the hold record. The mail's run name says „restore" for it, not „offsite-reconstitute"
(`heldRunKind`, seen on the live proof). Tests `TestR1348_*`. **Live proof 2026-11-03, scratch 9202, test image of c4b6844** (felhom.eu
`audits/security-2026-11-02/part3/`): a restore cut at „the restore copy is recorded", the volume held by a created-only
container, the controller started: hold `restore_mixed` / `interrupted_put_back`, the event `backup_run_failures` with the sentence in the sink.
**R-1347's live proof, same image:** an update with its undo copy cut off → hold `update_failed`; the household's restore → „the update hold is CLEARED", „its
update-held flag is forgotten"; six `kill -9` → the crash-loop watcher stops the app (`unhealthy_stop`), so it watches again.
**R-1339 — no false „drive … not mounted".** A per-app folder under a mounted drive is never a mountpoint itself; nine false lines
at every start on 9202. The folder still gets no bind and no skeleton of its own (the row's „not skipped" would write a
`userdata/` inside the app's folder and recreate the file manager, so the behaviour stays and only the line changes). A drive that
is really absent keeps its line, and a folder of it names the drive. `TestR1339_*`.
**R-1351 — FileBrowser's two start paths at boot take one lock.** `EnsureBaseStack`'s `compose up` and
`web.syncFileBrowserMounts`' `compose up --force-recreate` in the same second gave „Conflict. The container name … is already in
use", two ERROR lines and a first-boot WARN for an app that then ran healthy. `stacks.Manager.LockFileBrowserStack`.
`TestR1351_*`.
**R-1323 — „No such container" is not „every app can still reach it".** The checked container was replaced between the check and
the leave; the line now reads the CURRENT container's networks and says the truth. A current container on the app network, an
unreadable one, or any other failure keeps the alarm. `TestR1323_*`.
## v0.349.0 (2026-10-29) — the household line after an operator reboot and the busy file for it (D65 = A); a hold clear records the wanted stop (R-1350)
### 2026-10-29 (evening)
**MinAgent: 0.158.1** (unchanged). The new `GET /guest/operator-reboot` is asked once at start; an older agent answers
404, and such an agent cannot reboot from the hub either — nothing to show, nothing broken. The busy file is read BY the
agent (≥ the reboot_guest commit); an older agent never reads it.
- **D65 = A — the busy file (`internal/busymark`).** `<data dir>/ctl/busy`, rewritten every 15 s while a backup, a
restore, an off-site run, the night catch-up, an app update or an app export/import runs; removed when idle and at
start. The agent refuses the operator's „Reboot guest" while it is fresh. Tests `TestBusyFileFollowsTheWork`,
`TestBusyMarkPathIsTheOneTheAgentReads` (the path is the agent's guest path), `TestBusyMarkIsWiredAtStartup`.
- **D65 = A — „Felhom support restarted your box at <time>."** The dashboard's one neutral line for a day after an
operator reboot, HU/EN (`dashboard.operator_reboot`); the idle dashboard is byte-identical (the parity fixtures pass
unchanged). Tests `TestOperatorRebootNotice_*`, `TestWatchOperatorReboot_RetriesAndKeepsTheTime`.
- **R-1350 — no „installed app is not running" after the operator's hold clear.** Measured 2026-10-29 14:57 on Tester 1:
gokapi's crash-loop hold cleared from the hub, the app stayed stopped, and 30 s later `app_start_failed` went out —
the hold had kept it out of the dead-app scan while the household's recorded wish was still „running". The clear now
records the stop as wanted (desired state „stopped") for an app that is not up, once the hold is found and before it
is lifted; a running app's wish is left alone; a refused clear records nothing; the household's Start ends it.
Tests `TestR1350_*` (the consequence through `classifyRunStates`, the order against the lift, the refusal, the up
check, the wiring).
- Every new test seen failing by a mutation (17 mutants; two first-run mutants that did not compile were re-made so they
compile, and then failed).
## v0.348.0 (2026-10-29) — clear_hold from the hub (D64 = B); the box command clears through the running controller (R-1343); the held-app mail and line name the real cause (R-1340); the OOM line once per kill (R-1341); the restore copy safe during the stable-start watch (R-1342) — built from the commit carrying this heading
### 2026-10-29 (afternoon)
**MinAgent: 0.158.1** (unchanged). The next release also carries R-1342 (`652a690`, already on main, the entry below).
**No new event type** — the hub records the operator action's result as its own `operator_action` event.
- **`clear_hold` — a fifth verb in the CLOSED operator-action list (operator ruling D64 = B, 2026-10-29 12:10)**
(`internal/report/opactions.go`, wired in `cmd/controller/opactions_wiring.go`; arg = the app's stack name, letters,
digits, `-`, `_`, ≤64 — checked before any handler). It lifts the app's hold of ANY reason in the RUNNING controller
(`backup.AppStopGuard.ClearHoldByOperator` → `settings.ClearRestoreHold`: memory and `settings.json` together), so no
later save can write it back. It starts nothing. Refused, nothing changed: a bad name, an app with no hold. One log
line names who: „hold CLEARED by the operator via the hub, action #<id>". Every reader of the hold (page, start gate,
API, `RestoreHoldFor`, hold reporter) reads the same settings record; the ONE held flag cached elsewhere —
`stacks.Manager`'s „the last update ended held" (it made the crash-loop observer and the nightly update leg skip the
app) — is forgotten too (`ForgetUpdateHeld`). A `putback_pending` hold is also dropped from the app-stop marker, or the
next start with the drive bound would pour the old copy over what the app wrote after the clear; the copy stays on
disk. The hold reporter needs no change: its next tick (≤1 min) sends `app_hold_cleared`.
Tests: `TestOpActions_ClosedList` (updated deliberately), `TestOpActions_ClearHoldBadArgumentIsRefusedAndCallsNothing`,
`TestOpActions_ClearHoldDoneRefusedFailed`, `TestD64_ClearHoldViaTheHub`, `TestD64_ClearHoldRefusals`,
`TestD64_ClearHoldStaysGoneAfterALaterSave`, `TestD64_ClearPutBackPendingDropsTheMarkerEntry`,
`TestD64_TheHoldReporterTellsTheHubAfterTheClear`, `TestD64_ForgetUpdateHeld`, `TestD64_MainWiresTheNewPieces`.
- **R-1343 — the box command `--clear-restore-hold <app>` goes through the running controller** (`cmd/controller/holdclear.go`):
a unix socket `<data dir>/ctl/hold.sock`, 0600 in a 0700 dir, peer uid checked (SO_PEERCRED), nothing on TCP — not
reachable through traefik, the LAN or an app container. Output: „Cleared by the RUNNING controller … no restart is
needed". Only when no controller answers (no socket, or a stale one) does it edit `settings.json`, and it says so,
naming the restart INSIDE the guest (`pct exec <vmid> -- systemctl restart felhom-controller-bootstrap.service`); that
edit goes through the same door, so a waiting put-back is dropped from the marker there too. The
flag's help names the exact guest command. `--restore-holds` (the list) is unchanged.
Tests: `TestR1343_TheBoxCommandClearsThroughTheRunningController` (red with the socket skipped: „the hold came back
after a later save of the running controller" — R-1343 reproduced), `TestR1343_TheBoxCommandFallsBackWhenNoControllerRuns`,
`TestR1343_TheFallbackDropsAWaitingPutBackToo`, `TestHoldCtl_NotReachableFromOutside`,
`TestHoldCtl_UnknownRequestIsRefused`, `TestHoldCtl_ANonSocketFileIsNotReplaced`.
- **R-1340, second producer (found live 2026-10-29 on rc.1, Tester 1):** the hold's `Why()` — the hub's „Held apps" line,
the box's `app_held` event and `--restore-holds` — still said „held after an off-site restore failed its database
replay" for a LOCAL restore of an app with no database. It is now built from the record (`RestoreKind`, `MixedCause`,
`FailedStep`, `DBReplay`); a record from before these fields gets the neutral „held after a restore stopped half-way".
Test `TestR1340_WhyNamesTheMixedCause` (13 assertions failed on the old sentence).
- **R-1340 — the operator's „HELD STOPPED" mail names the real cause** (`cmd/controller/holdmail.go`
`heldStoppedOperatorText`, built from the hold record). Hold writers now record `RestoreKind` (off-site / local),
`MixedCause` (`definition_write` / `put_back` / `interrupted_put_back`), `FailedStep` (the put-back step that failed,
with its error), `DBReplay`, and the kept copy's folder (`settings.RestoreHold`, all `omitempty`). The mail never says
„database" unless the failure was a database replay, never „off-site" for a local restore; a pre-R-1340 mixed hold
names only what its record says. Legs unchanged (`restore-hold`, `restore-hold-mixed`); a local restore's RunKind is
`restore`. Tests (red with the old sentence/record): `TestR1340_UnhealthyStop`, `_UpdateFailed`,
`_RestoreFailedWithADatabase`, `_PutBackWithoutADatabase` (the measured Radicale case), `_PutBackAfterADatabaseReplay`,
`_DefinitionWrite`, `_LocalRestore`, `_PutBackPending`, `_InterruptedPutBack`, `_LegacyMixedHoldAssumesNothing`,
`_NotifyReadsTheHoldRecord`; writer side `TestR1340_AVolumeFailureWhosePutBackStopFailsRecordsTheStepAndNoDatabase`,
`_ADatabaseReplayFailureRecordsTheDatabase`, `_ADefinitionWriteFailureRecordsItsCause`, `_ALocalRestoreRecordsItsKind`,
`_AnInterruptedPutBackRecordsItsCause`. `TestEventMessageWireTextIsFrozen` is untouched (this text is passed through).
- **R-1341 — the OOM line once per NEW kill** (`cmd/controller/oomlog.go` `oomLogGate`, key = container + start time +
kill count): a second scan of the same kill logs nothing; a new kill or a restarted container logs again. The
`app_oom` event call is unchanged. Tests: `TestR1341_ASecondScanOfTheSameKillLogsNothing`, `_ANewKillLogsAgain`,
`_ARestartedContainerLogsAgain`, `_ContainersAreJudgedApart`.
### 2026-10-29 (after v0.347.0) — R-1342: the restore copy is safe during the stable-start watch; a removed app does not alarm
**MinAgent: 0.158.1** (unchanged). Found live on Tester 1 at 10:39 CET: a false `backup_run_failures` mail („Gokapi did
not start … the pre-restore copy is kept") after the copy had been deleted by hand mid-watch and the app removed.
- **(1) D62 gap — `internal/backup/restore_copies.go`:** while a stable-start watch (`releaseAfterStableStart`, and the
late-start watcher) holds a copy, the copy reads „checking" in the list (one line, no delete date) and
`DeleteRestoreCopy` refuses it (`ErrRestoreCopyNeeded`) — the operator's rule is „never while an operation runs on the
app". The mark lives in `Manager.watching` under `m.mu`, the lock the delete checks under.
- **(2) R-1270 watch — `internal/backup/restore_verdict.go`:** the watch ends quietly (`stableRemoved`, one INFO line, no
alarm) when the provider says the app was removed (`StackRemovalReporter`; the production `stackAdapter.StackRemoved`
with a compile-time assertion in `cmd/controller/main.go`). An installed app that never runs stably still alarms once.
- Tests `TestR1342_ACopyUnderTheStableWatchIsNotDeletable`, `TestR1342_AnAppRemovedDuringTheWatchRaisesNoAlarm` (each
red by a mutation), the control `TestR1342_AnInstalledUnstableAppStillAlarms`, and `TestR1342_EveryNeededReasonHasItsSentence`.
## v0.347.0 (2026-10-29) — D62: restore copies kept 30 days, listed, deletable by hand; R-1256: four restore edges, all proven live (D63); R-1254 controller half; R-528: the host's OOM count; an unknown app is a 404 — RELEASED (image `sha256:c79b06e6…`, from `fd75c7c`, CI 2491; rc.1 = `e3fce09` on scratch 9202); per-box floors 09:57 CET, global floor 2026-10-29 10:31 CET on the operator's yes (MinAgent 0.158.1)
### 2026-10-29 (attended) — D62 and D63
**MinAgent: 0.158.1** (unchanged). Needs hub ≥ the release carrying `restore_copy_deleted` (felhom.eu `19afefab`): an
older hub answers that one event 400 and the record is lost (the delete itself is unaffected).
- **D62 (operator ruling 2026-10-29 08:17) — the age: 30 days** (`restoreCopyUnusedAge`, was 14, the night's safest
reading). `TestD62_AnUnusedCopyIsKept30Days` (red at 14: the 20-day copy was swept).
- **D62 — the list (`backup.RestoreCopies`, `web.restoreCopyRows`, `backups_restore.html` „Megőrzött másolatok"):** each
restore copy shows the app, the date of the restore it protects, its size and „automatikusan törlődik: <date>". A copy
a hold or a pending put-back needs (a hold; the putback_pending hold; the app-stop marker naming it; an unreadable
marker) shows why in one line and its button is off — it never gets a date, because the sweep never takes it.
- **D62 — „Törlés most" (`POST /backup/restore-copy/delete`, `backup.DeleteRestoreCopy`):** the warning names the app
and the restore's date (the operator's text, hu + en); the household types the app's name. JSON answer: 200; 400 a
wrong name; 404 no such copy; 409 the copy is needed, or a backup/restore runs, or the app updates. The check and the
removal are one step under the backup manager's lock (no restore can begin in between). Path guard: both names one
path element, the copy a real directory (not a link) directly under `backups/restore-undo/<app>/`. One `[WARN]` log
line and one operator event `restore_copy_deleted` (who, app, restore date, size) per delete. CSRF as every form.
Tests `TestD62_*` (backup, web, notify) — each seen failing by a mutation of the check it pins.
- **D63 — the fault switch, TEST BUILDS ONLY (`cmd/controller/faultpoint_{on,off}.go`):** `go build -tags faultinject`
arms two points from a file (`<data>/faults/<point>` = „<skip> <fail>"): `defwrite-after-compose` (the definition
writer fails after the compose was written — H4) and `stop-<app>` (the app's stop fails — H5). Without the tag it
is a no-op that reads nothing; `TestD63_TheReleaseBuildHasNoFaultSwitch` pins that, and that build.sh and the
Dockerfile pass no build tag (red-proofed both ways).
### 2026-10-28 night — four restore edges (felhom.eu R-1256, review round 2 HUNT-1 H4/H5/H6/H8)
**MinAgent: 0.158.1** (unchanged). Each edge: a test seen failing first, then the fix; live proof on 9202 in the felhom.eu
night report (`audits/night-2026-10-28/`). **Live 2026-10-29 on 9202:** H5's busy volume (not written into, the live data
put back, no hold) and H8 (the 03:29 sweep removed the two old copies, kept the young one). **Live 2026-10-29 on Tester 1
(D63, a `-tags faultinject` test build):** H4 (the live definition written back before the start) and H5's failed stop
(nothing put back, the copy kept, the app held). H6: tests only (the old code refused at the early check too).
- **H4 — `internal/backup/offbox_reconstitute.go`:** an off-site restore whose definition write (or database-password
write) fails no longer starts the app at once. The writer writes the compose BEFORE the app's settings, so the stack
dir could hold the snapshot's (older) compose: the live definition is written back first, then the app starts
(`restartAtLiveDefinition`); when the write-back fails too, the app is held (`restore_mixed`), never started on the
snapshot's definition. Tests `TestR1256_H4_*`.
- **H5 — a put-back whose stop fails does not go on.** `putBackAfterFailedRestore` (off-site and local unit restore):
a failed stop pours nothing, keeps the copy and holds the app; a failed definition write-back keeps the copy too (D20:
a put-back that fails keeps its copy). `CrashPutBack.PutBackInterruptedRestore` (a restore cut by a power loss): a
failed stop is an error, so the start holds the app (R-1251 H2) instead of pouring under a running app.
`restoreDockerVolumesFrom`: a `docker volume rm -f` that fails (the volume is in use) counts as a failed volume — the
following `volume create` was a no-op and the tar was unpacked ON TOP of the old content (a merge). Tests
`TestR1256_H5_*`. One log line corrected: a failed start after a put-back no longer says the copy stays (it is removed;
the volumes are back).
- **H6 — a restore begins in the same step as the check for every other operation.** New
`backup.TryBeginRestoreOp` / `RestoreBusy` (under `m.mu`, the lock the second-drive mirror's `beginTier2Run` asks
under): a restore op in flight, the backup run, a second-drive copy. Every web restore handler begins through
`Server.beginRestoreOp` (a source test refuses a direct `BeginRestoreOp`); the kept loads (`LoadKeptApp`,
`LoadKeptOffsite`) return `ErrRestoreBusy` with nothing started, and `LoadKeptApp` takes the dated folder's file move
as `prepare`, inside the operation. The install-time „use my kept data" asks about the second-drive copy too
(`RestoreBusy`, was RestoreStatus/IsRunning only). Tests `TestR1256_H6_*` (backup, web, api).
- **R-1254 H2, the controller half (felhom.eu R-1254; agent `5ddd124`):** the escrow claim and confirm reach the agent
with their own 60 s timeout (`agentapi.escrowCallTimeout`; was the client's 15 s) — the agent may spend its 30 s
upload plus 15 s asking the hub before it answers, and a claim abandoned at 15 s could still store a package whose
code reached nobody. The confirm's household sentence: 502 (the hub holds another package) keeps „…a régi kódod
továbbra is érvényes…"; 504 (the agent cannot learn the outcome) or a failed call (0) now reads
`api.escrow.confirm_store_unknown` („Nem tudjuk biztosan… Őrizd meg mindkét kódot…", hu + en) and never promises the
previous code. `TestR1151_ConfirmAnswersInTheBoxLanguage`'s status-0 case changes with it, on purpose. Tests
`TestR1254_*` (agentapi, web), red before the fix.
- **R-528 — the OOM line sees a kill Docker does not report (felhom.eu R-528; agent `GET /guest/oom-kills`).** The
30-s dead-app scan (`stacks.ScanOOMKilled`) now also asks the host's agent once per scan for the kernel's per-container
`oom_kill` counter (`agentapi.GuestOOMKills`, wired in `cmd/controller/hostoom_wiring.go`, 5 s cap). A running
container whose current run has a count above 0 is an OOM kill whatever Docker's `OOMKilled` says: the existing
`app_oom` line (once per run, R-514/R-636) fires, its details carry `source` (`docker`, `cgroup` or `docker,cgroup`),
and the host's count is the run's kill count. An agent without the route (404) is asked again only after 10 minutes
and said once; no agent (9202) = Docker's flag only, as before. The inspect now reads `{{.Id}}` too. Tests
`TestR528_*` (stacks, cmd/controller), red before the fix (the host-counted kill was not seen) or red-proved by
mutation (the back-off, the wiring).
- **Found walking the capability map's B2 on 9202 (small, fixed in the session):** a Stop/Start of a name that is no
app on the box (the bootstrap-managed `felhom-controller`) answered 500 „…az alkalmazás beállításai nem menthetők." —
the desired-state write's „not found" read as a save failure. `stacks.SetDesiredState` returns the
`ErrStackNotFound` kind with the household's sentence (`err.stacks.alkalmazas_nem_talalhato`); `actionStack` answers
404 before recording any intent. Nothing was ever stopped. Test `TestAnActionOnANameThatIsNoAppIsA404` (red first).
- **H8 — `internal/backup/restore_copy_sweep.go`:** an unused restore copy goes after 14 days. Every 6 hours
(`restore-copy-sweep`) a copy is removed only when the app has no restore hold of any kind, the app-stop marker is
readable and does not name the app, no restore runs, and the copy is older than 14 days. Tests `TestR1256_H8_*`
(and the wiring test).
## v0.346.1 (2026-10-27) — SECURITY: traefik v3.7.16, the file manager 1.5.8-stable, cloudflared 2026.10.4 — built from v0.346.0 (`b1b9be3`) + one commit (felhom.eu R-1304, R-1305)
**MinAgent: 0.158.1** (unchanged). Box programs only; no other change from v0.346.0.
+38
View File
@@ -1,5 +1,43 @@
# CONTEXT.md — Project Memory
> **2026-11-09 (attended) — v0.351.0 in code, not released.** Operator's word 2026-11-09 („go with A"): decision 56 amended
> (an old `-rc` controller image is deleted like an old release) and decision 285 (pinned infra images keep the pin + the
> previous pin). Measured cause: demo-hp's whole-guest local copy refused for space (felhom.eu R-548); 3.5 GB of its 27.5 GB
> of images were these kinds. The bigger lever — the whole-guest copy carries every Docker image — is a design question
> (felhom.eu register, filed the same day).
> **2026-11-03 (day, attended) — v0.350.0 RELEASED, the global floor since 08:18 CET (MinAgent 0.158.1).** Image `sha256:9ee53908…` from `5c336d8`. Traefik v3.7.17 (GHSA-xv5x), R-1347, R-1348, R-1339, R-1351, R-1323. Golden 0.350.0 vouched 10:18:10 CET.
>
> **2026-10-29 (evening, attended) — v0.349.0 RELEASED, the global floor since 19:24 CET (MinAgent 0.158.1).** Image
> `sha256:c5341883…` from `94049a7`. D65 = A: `internal/busymark` keeps `<data>/ctl/busy` fresh while the controller's
> own work runs (the agent refuses „Reboot guest" meanwhile); the dashboard shows „A Felhom ügyfélszolgálata HH:MM-kor
> újraindította a dobozt." for a day after an operator reboot (`GET /guest/operator-reboot`, asked once at start).
> R-1350: the operator's hold clear records the stop as wanted (desired state „stopped") before the hold is lifted —
> proven on Tester 1, no `app_start_failed` in 6.5 minutes. 30-minute watch clean; per-box floors cleared.
> **2026-10-29 (afternoon, attended) — v0.348.0 RELEASED, the global floor since 14:48 CET (MinAgent 0.158.1).** Image
> `sha256:672fc1c1…` from `2a0a17a` (CI 2515). D64 = B (`09` §3 decision 283): `clear_hold` is the fifth verb of the closed
> operator-action list; any hold reason; the running controller clears memory and `settings.json` together through
> `<data>/ctl/hold.sock` (0600, peer uid checked), and `--clear-restore-hold` goes through it when a controller runs
> (R-1343). R-1340 (both producers name the real cause), R-1341 (OOM line once per kill), R-1342 (no delete during the
> stable-start watch). Proven live on Tester 1 on the release build: a crash-loop hold cleared from the hub stayed
> cleared after the next settings save. Open: R-1350 (an `app_start_failed` 30 s after a clear), R-1347, R-1348.
> **2026-10-29 (attended release run) — v0.347.0 RELEASED, the global floor (MinAgent 0.158.1).** Image `sha256:c79b06e6…`
> from `fd75c7c` (CI 2491); rc.1 = `e3fce09` on scratch 9202. Carries the night's R-1256 (all four edges, H4 + H5's
> failed stop proven live on Tester 1 with a `-tags faultinject` test build — D63), R-1254 controller half, R-528, the
> unknown-app 404, and D62 (restore copies 30 days; „Törlés most" in the kept-copies list; `restore_copy_deleted` needs hub
> ≥ 0.175.0). The fault switch (`cmd/controller/faultpoint_{on,off}.go`) is a no-op in every build without the tag
> (`TestD63_TheReleaseBuildHasNoFaultSwitch`). Next here: R-1340 (the mixed-hold operator mail), R-1341 (the OOM WARN every
> 30 s), R-1339 (the false „not mounted" line).
> **2026-10-28 night (unattended) — R-1256, R-1254 (controller half), R-528 and the unknown-app 404 on `main`, NOT
> released (`342a394`, `e33cffa`, `eed23e7`, `20c5877`; MinAgent unchanged).** Restore ops begin through
> `Manager.TryBeginRestoreOp` / `web.beginRestoreOp` only (`TestR1256_H6_NoHandlerBeginsARestoreUnchecked` refuses a direct
> `BeginRestoreOp` in a handler). **Decided by CC unattended — operator may reverse:** an unused restore copy goes after
> 14 days (`restoreCopyUnusedAge`; felhom.eu `07` §6.3). H4 and H5's failed stop: tests only, live proof owed. H5 (busy
> volume) proven live on 9202, R-528 on Tester 1, with the local test tag `0.347.0-rc.n1028` (never pushed).
> **2026-10-26 (evening) — D53 ruled (felhom.eu `09` §3 decision 265): held apps reach the OPERATOR first.** Next here:
> the controller reports its holds (stack, reason, since) to the hub as `app_held` / `app_hold_cleared` events plus a
> daily re-send (felhom.eu `08` §10). Commit only; the release rides a later attended run.
+16 -20
View File
@@ -1,24 +1,20 @@
# REPORT — D53: the hub learns which apps are held — committed, NOT released (2026-10-26 evening)
# REPORT — v0.351.0 in code: old test builds and old infrastructure images are deleted (2026-11-09, attended)
**Plain words first.** A held app (stopped on purpose until a person acts) now reaches the hub: the box sends one
message when a hold appears or changes, one when it goes, and repeats every hold once a day. The hub shows the list
and mails the operator after a waiting time (felhom.eu `7e57e824`). Not released (this run allows no release).
**Plain words first.** demo-hp's whole-box copy was too big for its system disk. Part of it was old images that
nothing deletes: 25 old controller test builds and old copies of the system images (traefik, cloudflared, FileBrowser,
Samba). The controller now deletes both kinds by itself, 3 minutes after each start. **Not released yet.**
## The change (`5bd03ad`, CI 2377 ✓)
| | |
|---|---|
| Version | 0.351.0 (CHANGELOG heading „unreleased"), MinAgent 0.158.1 unchanged |
| Rulings | `09` §3 decision 56 amended, decision 285 new — operator 2026-11-09 („go with A") |
| Tests | `TestControllerRetention_OldTestBuildsGo`, `_BoxOnATestBuildIsJudged`, `TestInfraRetention_*` (4); one existing rc expectation changed with the rule |
| Red-proof | six code mutants, each test red: previous-pin keep, in-use check, R-863 lock, tag parser, running-version parser, release preference |
| Green gate | `go build ./...`, `go vet ./...`, `go test -count=1 ./...` rc 0; `controller_gates.py --fast` OK |
- `internal/holdreport`: one observer of the hold store, ticked every minute by the scheduler (`hold-report`). It sends
`app_held` (stack, reason, since, the hold's sentence) for a new or changed hold, `app_hold_cleared` when a hold is
gone — also one cleared while the controller was down (`settings.reported_holds`) — and every hold again daily and at
start. Severity info.
- `settings.RestoreHold.Why()` / `WireReason()`: the hold sentence lives in one place (`holdWhy` calls it); the legacy
empty reason is sent as `restore_failed`.
- Tests `internal/holdreport/holdreport_test.go` (8). Red-proved: an inert `Tick` failed all six behaviour tests; the
scheduler line commented out failed the wiring test.
**Measured cause** (felhom.eu `REPORT.md`): on demo-hp guest 9201, 2.2 GB of old controller images and 1.35 GB of old
infra and helper images, of 27.5 GB. The same images were removed there by hand the same day, so a live proof needs
a box that still has them (Tester 1: 37 images, 3.8 GB unused).
## Live proof (felhom.eu `documentation/audits/d53-held-apps-2026-10-26/`)
9202 ran a local-only image of this commit (`0.346.0-d53test`, never pushed to the registry) against a test hub process.
A throwaway Radicale, crashed seven times, was stopped and held by the box (17:37:52Z); `app_held` reached the hub at the
next tick; one operator reminder after the 3-minute test grace; the household's Start lifted the hold and
`app_hold_cleared` removed the row. The first tick also reported four holds 9202 has carried since 2026-09-24 for apps
that are not installed (felhom.eu R-1298). 9202 is back on 0.343.0 with its own config.
**Not done:** no image built, no box moved, no floor. The release path is the skill's: an `-rc` on the three test boxes,
the box walk, then the tag and the floor.
+8 -3
View File
@@ -35,7 +35,8 @@
| `dockerexec.BeginImageWork(args)` / `TryImageCleanup()` / `ImagePulling(args)` (v0.294.0, R-863) | controller/internal/dockerexec/imagework.go | `defer dockerexec.BeginImageWork(args)()` | **Every NEW place that runs `compose up/pull/create/run` must hold it** (the stacks compose helpers, appexport's restore and the FileBrowser sync already do) | An image pulled by compose is named by no container until compose creates one; a clean-up pass in that window deleted it (measured, R-863). The pass takes the lock exclusively without waiting; pulling commands wait for a running pass (seconds) |
| `nightchain.Ledger` / `CatchUp` / `ResumeWatch` / `ComputeBanner` (v0.295.0, R-871) | controller/internal/nightchain/ | `Open(path, now)`, `MarkEnded(leg, t)`, `Missed(now, W, loc)`, `Evaluate(ctx, why)` | "was a night missed?" and making it up; the missed-backup banner's rule | The ledger is an ATTEMPT record (ran to its end) — never evidence a backup exists; a NEW nightly backup leg must be wrapped by `withLeg` in main.go and listed in `nightchain.Order`, or the catch-up never makes it up; never add the update leg to `catchUpLegs` |
| `scheduler.DailyLateLimit` (v0.295.0) | controller/internal/scheduler/scheduler.go | — | a daily timer firing > 60 min late (host suspend) is skipped | do not "fix" a late fire by running it: the app-update leg would run at noon |
| `stacks.RetainControllerImages(ControllerImageRecord)` · `selfupdate.UpdateState.RecordedPrevious` (v0.285.0, decision 56) | controller/internal/stacks/controller_image_retention.go | `({Repo, Running, Previous})` | Deletes controller images older than the running + previous one | The previous comes from the SWAP RECORD (success onto the running version), version order only as the fallback; versions above the running one and non-version tags are kept; skipped while the controller swaps itself; same `imageDocker` seam |
| `stacks.RetainControllerImages(ControllerImageRecord)` · `selfupdate.UpdateState.RecordedPrevious` (v0.285.0, decision 56) | controller/internal/stacks/controller_image_retention.go | `({Repo, Running, Previous})` | Deletes controller images older than the running + previous one | The previous comes from the SWAP RECORD (success onto the running version), version order only as the fallback (a release preferred over a test build, v0.351.0); versions above the running one and non-version tags (`latest`) are kept — an `-rc` tag IS a version (`util.ParseVersionPre`) since v0.351.0; skipped while the controller swaps itself; same `imageDocker` seam |
| `stacks.RetainInfraImages(pins)` / `RunInfraImageRetention(ctx, pins, every, tries)` (v0.351.0, decision 285) | controller/internal/stacks/infra_image_retention.go | `(infra.Images())` | Deletes pinned infra images older than the pin + the previous pin | Per repository: the pin, newer versions, the previous pin and in-use images stay; untagged unused go; non-version tags stay; skipped (errImageRetentionBusy) while an app pulls (R-863) or the controller swaps; same `imageDocker` seam |
| `stacks.CloseSignupNow` / `CloseSignupOffered` / `applyNativeLock` (v0.282.0, decisions 47/49) | controller/internal/stacks/after_setup.go | `(name)` | The app's own sign-up switch after the setup; "close sign-up now" for an app installed before the rule | **Check the installed compose reads the variable** (an old install carries the old compose until its next update) — never record a lock that is not there. One run per app at a time (`nativeLockBusy`) |
| `backup.judgeCopy` / `HollowCopies` / `SetHollowCopyNotify` (Part D, v0.279.0) | controller/internal/backup/hollow_watch.go | `(app, tier, unitDir)` | A RUNNING app whose newest copy holds no data → operator digest once/day + page sentence | Uses `unitCarriesData` (the manifest, never size); a stopped held app is never flagged |
| `web.nightChain` (R-705, v0.279.0) | controller/internal/web/night_chain.go | `POST /api/debug/backup/night-chain` | The night's four legs now, in order | Refuses while any op/update/chain runs; the leg uses `RunUpdateLegNow` |
@@ -80,12 +81,15 @@
| `restoreOpInFlight` + `hasRecentRestoreResult` | controller/internal/web/restore_wizard.go | `(backup.RestoreOpStatus) bool` / `(st, app, now) bool` | THE "is a restore running / did one just finish" display reads | **TRAP (v0.154.0 shipped this bug): `Manager` has TWO running flags.** `IsRunning()` reads the CONCURRENCY flag, acquired inside the goroutine — and `RestoreOffboxScratch` never acquires it, so it is false for the whole verification restore. Display must read `RestoreStatus().Running` (set synchronously by `BeginRestoreOp`). Read the status ONCE per render or the strip and the suppression can disagree. `hasRecentRestoreResult` is app-bound and window-bounded — a process-wide result must not light another app's „Eredmény" |
| `restoreWizardPath` / `deriveWizardStep` / `resolveWizardApp` | controller/internal/web/restore_wizard.go | `(app) string` / `(restoreWizardInput) restoreWizardView` / `([]OffboxAppRow, name) *OffboxAppRow` | R-48 offsite restore wizard: URL builder + the PURE step/unlock derivation + the app-resolution refusals | The step is **never** taken from the request. Precedence is load-bearing: op-running outranks a stale `?full_prep=`, else a commit button reappears mid-restore. Truth table + red-proof: `restore_wizard_test.go`. Adding a form here that posts anywhere new breaks `TestRestoreWizard_NoNewMutationEndpoints` **by design** — R-48 adds no mutation surface |
| `restoreOpBlocked` | controller/internal/web/restore_wizard.go | `() (msg string, blocked bool)` | THE refusal gate before starting ANY restore | **Use this, never a bare `IsRunning()`.** It reads BOTH flags: `RestoreStatus().Running` (set synchronously by `BeginRestoreOp`, true for the whole off-box restore) and `IsRunning()` (the concurrency flag, the only one the nightly backup holds). **R-351: all seven handlers read only `IsRunning()`, which the goroutine acquires AFTER the handler returns — a second press started a second run and was told „…elindult".** Returns the Hungarian refusal, which names the running app and a route | **R-360 (v0.226.0): `offboxVerifyCopyDeleteHandler` was the LAST holdout and its doc comment claimed it already did this — the sentence is why nobody looked. `IsRunning()` is FALSE for the whole of a verification restore, so the copy a restore was writing into could be deleted from the UI (observed live 2026-08-21 22:35). No app-name comparison: refusing during ANY restore is stronger and uniform.**
| `Server.beginRestoreOp` → `backup.TryBeginRestoreOp` / `RestoreBusy` (R-1256 H6) | controller/internal/web/restore_wizard.go; controller/internal/backup/opstatus.go | `(op, stack) (msg string, ok bool)` / `(op, stack) (busy, mirrorApp string)` | THE way a restore op begins: the check for every other operation (a restore op, the backup run, a second-drive copy) and the record in ONE step under `m.mu` — the mirror asks under the same lock (`beginTier2Run`) | **Never call `BeginRestoreOp` from a handler** (`TestR1256_H6_NoHandlerBeginsARestoreUnchecked`): `restoreOpBlocked` asked early can be stale by the time the form is read, and a mirror that began in between ran beside the restore. `restoreOpBlocked` stays as the EARLY refusal; the kept loads begin inside `LoadKeptApp` / `LoadKeptOffsite` and return `ErrRestoreBusy` |
| `Manager.RestoreFromRecoveryUnitAt` / `RestoreFromRecoveryUnit` + `UnitRestoreResult` (R-353 v0.226.0; R-102 v0.229.0) | controller/internal/backup/restore_unit.go | `(stack, unitDir string) (UnitRestoreResult, error)` / `(stack string) (…)` | THE local recovery-unit restore, and the facts its surface must state | **Returns a RESULT, not just an error** — volumes replayed, DBs replayed, and what the manifest LISTED. The Manifest* counts are load-bearing: zero-replayed has two causes (the backup held no data / the backup listed data that did not come back) and they are opposite news. Pair it with `unitRestoreOutcomeMsg`; do NOT write a new sentence. **A claim about the APP is forbidden on this path** — it has no `SafetyDump` discriminator, unlike the off-site twin (CONTEXT.md ruling, 07-backup-architecture §6.3). `restoreDockerVolumes` now returns `(int, error)`; `restoreDockerVolumesFrom` is unchanged and still the shared implementation. **R-102: `…At` holds the whole body and takes the unit DIRECTORY; the one-argument form is the thin caller naming the primary unit.** THE SOURCE MOVES; THE DESTINATION DOES NOT — `GetAppDrivePath` still resolves where the data lands. The R-47 mutation order, the unit-over-guest secret precedence, the fail-closed data-key gate and the no-unit `RestoreApp` fallback are all pinned and unchanged. The volume leg goes through the R-354 `volumeReplayFrom` seam so the source directory is assertable without Docker |
| `Manager.putBackAfterFailedRestore` + `liveVolumeCopy` (R-893 option A, v0.336.0) | controller/internal/backup/offbox_restore_undo.go | `(stack, cause, live, copy, volumesMoved, filesPlaced, restart, holdKey) error` | THE restore undo — off-site (v0.336.0) and, since D22 (unreleased), the LOCAL unit restore (`RestoreFromRecoveryUnitAtWith`, so also the second drive's and a kept load): the live volumes it will replace, measured before the stop (`planLiveVolumeCopy`), copied right after it (`copyLiveVolumes`), poured back on any failure | One copy per restore under `<ns>/backups/restore-undo/<app>/` (no capture set reads `backups/`); the put-back uses the restore's own `volumeReplayFrom`/`restoreDockerVolumesFrom`, so there is one volume-replay implementation. A put-back that fails sets `keep` and HOLDS (option C) — never start on volumes that are not known to be back. Seams for tests without Docker: `undoVolumeProbeFn`, `undoVolumeTarFn`, `undoVolumeRemoveFn`; operator notice `SetRestorePutBackNotify` (leg `restore-put-back`). Pinned by `r893_putback_test.go` and `r1177_local_putback_test.go` |
| `Manager.putBackAfterFailedRestore` + `liveVolumeCopy` (R-893 option A, v0.336.0) | controller/internal/backup/offbox_restore_undo.go | `(stack, cause, live, copy, volumesMoved, filesPlaced, restart, holdKey, restoreKind) error` (R-1340: `restoreKind` = `settings.RestoreKindOffsite`/`Local`; a failed put-back records its step in the hold's `MixedCause`/`FailedStep`/`DBReplay`) | THE restore undo — off-site (v0.336.0) and, since D22 (unreleased), the LOCAL unit restore (`RestoreFromRecoveryUnitAtWith`, so also the second drive's and a kept load): the live volumes it will replace, measured before the stop (`planLiveVolumeCopy`), copied right after it (`copyLiveVolumes`), poured back on any failure | One copy per restore under `<ns>/backups/restore-undo/<app>/` (no capture set reads `backups/`); the put-back uses the restore's own `volumeReplayFrom`/`restoreDockerVolumesFrom`, so there is one volume-replay implementation. A put-back that fails sets `keep` and HOLDS (option C) — never start on volumes that are not known to be back. Seams for tests without Docker: `undoVolumeProbeFn`, `undoVolumeTarFn`, `undoVolumeRemoveFn`; operator notice `SetRestorePutBackNotify` (leg `restore-put-back`). Pinned by `r893_putback_test.go` and `r1177_local_putback_test.go` |
| `Manager.SweepUnusedRestoreCopies` (R-1256 H8) | controller/internal/backup/restore_copy_sweep.go | `(now) []string` | THE only AUTOMATIC remover of a restore copy nothing names any more (`restore-copy-sweep`, every 6 h); the household's „Törlés most" is `DeleteRestoreCopy` (D62, restore_copies.go — same „needed" rule via `restoreCopyNeededFor`, check + removal under `m.mu`) | Removes a copy only when ALL hold: older than 30 days (D62), the app has no restore hold of ANY kind, the app-stop marker is readable and does not name the app, no restore op runs. It answers „no" when it cannot answer — a held or waiting copy is support's way back |
| `CrashPutBack` + `Manager.persistUndoForCrash` + `AppStopGuard.SetUndo` (D22 / R-1195, v0.338.0) | controller/internal/backup/restore_crash_putback.go | `NewCrashPutBack(stop, writeDef, logger)`; `PutBackInterruptedRestore(stack, undoDir) error` | A restore cut by a power loss is put back at start: the copy (volumes + `definition/` + `undo.json`) is recorded in the app-stop marker once whole; `Recover` pours it back BEFORE the restart | Call `persistUndoForCrash` right after `copyLiveVolumes` in any new restore path. `Recover` runs before the backup manager exists, so the put-back is wired in main.go with `stackMgr.StopStack` and the stack adapter's definition writer. A failed put-back = a FAILED recovery (app not started, marker kept). The household reads `note.restore.interrupted_put_back` (`LoadRestoreRecordAfterRecovery`). Pinned by `TestR1195_*` |
| `tier2CaptureSetFull(...) (legs, missing, warns, classified)` + `keptCopyRels` + `dropKeptCopiesUnder` (R-1252, unreleased) | controller/internal/backup/tier2_capture.go, tier2_kept.go | `classified` = the legs come from a VALID backup block | **The second-drive reconcile deletes only under a valid classification**; a rejected or absent block keeps every dest dir; kept copies (D25) are always in the keep set; a mirror deleted by Remove drops its kept records | A new reconcile caller must take `classified` and the kept rels too — never reconcile against the legacy legs. A rejected block is told by `Manager.BackupBlockRejected` (stacks; via the adapter's optional `BackupBlockRejected` method) and signalled once a day (`SetTier2SettingsUnreadableNotify`). Pinned by `r1252_rejected_block_keeps_copies_test.go` |
| `holdreport.Reporter` + `settings.RestoreHold.Why` / `WireReason` (D53, unreleased) | controller/internal/holdreport/holdreport.go · controller/internal/settings/settings.go | `New(store, notifier.PushEvent, logger)`; `Tick(now)` from the scheduler | Tells the hub every hold in place (`app_held`), every hold gone (`app_hold_cleared`), each again daily | **One observer of the hold store — never a send inside a hold writer.** A new hold writer needs nothing here. The hold sentence is `RestoreHold.Why()` (one per reason) — never a second copy. Pinned by `internal/holdreport/holdreport_test.go` |
| `AppStopGuard.ResumePutBack` / `ResumeWaitingPutBacks` + `HoldReasonPutBackPending` (R-1251, unreleased) | controller/internal/backup/appstop_putback_wait.go | `ResumePutBack(stack) bool` (true = nothing owed, the caller may start); `ResumeWaitingPutBacks() []string` (puts back, then starts through the gated starter) | A cut restore's copy on a drive that is not bound is WAITING: Recover writes the restore hold `putback_pending` (copy in `SafetyDump`) and starts nothing; the put-back runs when the drive binds, before any start | **Any new path that starts an app when its drive arrives calls `ResumePutBack` first** (the boot recreate and `restartStacks` do, in web via `Server.resumePutBack`). Every other start path is covered by the hold store it already honours — do not add a second check. A failed put-back (at start or on resume) writes the D20 hold (`restore_mixed`) and drops the app from the marker. Needs `SetHolds` + `SetUndoReachable` before `Recover` (main.go). Pinned by `r1251_putback_waits_test.go`, `TestR1251_DriveStartPathsPutBackBeforeTheyStart`, `TestR1251_MainWiresThePutBackWait` |
| `AppStopGuard.ClearHoldByOperator` + `clearHoldNow` / `listenHoldCtl` / `clearRestoreHoldCLI` (D64 = B, R-1343, unreleased) | controller/internal/backup/hold_clear.go · controller/cmd/controller/holdclear.go | `ClearHoldByOperator(stack) (RestoreHold, error)`; `clearHoldNow(deps, stack, who) (msg, error)` | THE operator's hold clear, any reason — the hub's `clear_hold` and the box command `--clear-restore-hold` both go through `clearHoldNow` in the RUNNING controller | **Never clear a hold from a second process** — the running controller's next save writes it back (R-1343, measured). The box command reaches the running controller over `<data dir>/ctl/hold.sock` (unix, 0600, peer uid checked; never TCP/HTTP). A `putback_pending` clear also drops the app from the app-stop marker (the copy stays). Refusals (`ErrHoldBadStack`, `ErrNotHeld`) change nothing. The update's in-memory held flag goes via `stacks.Manager.ForgetUpdateHeld`. Pinned by `cmd/controller/holdclear_test.go` |
| `Manager.RestoreTier2Unit` + `Tier2Coverage.CanRestoreUnit` / `Tier2CopyDate` (R-102/R-103, v0.229.0) | controller/internal/backup/tier2_restore.go | `(stack string) (UnitRestoreResult, error)` / `() bool` / `() (string, bool)` | THE full restore from the SECOND DRIVE's mirror, and the predicate that gates it | **Do NOT widen `CanRestore()`** — it answers only "can the additive file restore run?" and one predicate answering two questions is R-356. Fail-closed: `tier2UnitIsOpenable` requires a parseable manifest, because a directory is not a package. The single-writer flag is taken INSIDE `RestoreFromRecoveryUnitAt` — a second `acquireRunning()` here would refuse the restore it guards. `Tier2CopyDate` prefers the SUCCESS anchor over the attempt clock (R-101) and reports which it returned |
| `unitRestoreOutcomeMsg` + its three message constants (R-353, v0.226.0) | controller/internal/web/handlers.go | `(app string, res backup.UnitRestoreResult) string` | THE customer sentence for a completed LOCAL restore | Twin of `reconstituteOutcomeMsg`; copy its SHAPE (clauses earned by having done the thing, no filesystem path, base names only), never its text. The constants are named because `r353_unit_outcome_test.go` asserts them verbatim — a silent edit is how an honest message drifts back into a comforting one, which is the documented history of the sentence it replaces |
| `offsiteNoSpaceMsgFmt` + `offsiteSizeUnknownMsg` (R-357, v0.226.0) | controller/internal/backup/offbox_restore.go | two consts | EVERY headroom refusal on the off-site restore surface | **All four gates share these** (prepare, scratch, place, and the destructive reconstitute). A customer meeting one wording on one path and a different one on another has to work out whether it is the same problem. **The reconstitute gate uses NO ×1.1 margin** — it copies a measured tree; `OffboxRestorePrepareFull`'s ×1.1 predicts a download. **Fail closed when either probe reads ≤ 0**: `free < need` with `need == 0` is FALSE, so an unmeasurable input sails through — a gate present and inert |
@@ -357,6 +361,7 @@
|---|---|---|---|
| `diskAgent` | controller/internal/web/storage_handlers.go | `*agentapi.Client` | `mockAgent` in controller/internal/web/storage_handlers_test.go |
| `netAgent` + `Server.netAgentFn/netProbeFn/netListFn` | controller/internal/web/netstorage_job.go (+ server.go fields) | `*agentapi.Client` / `runNetProbe` (linux re-exec) / `agent.ListNetStorage` | `fakeNetAgent` + fn injections in controller/internal/web/netstorage_job_test.go — the NAS add orchestration never shells/TLS-dials in tests |
| `faultPoint(point) error` (D63, TEST BUILDS ONLY) | controller/cmd/controller/faultpoint_{off,on}.go | a no-op without `-tags faultinject` (the release image); with it, a file `<data>/faults/<point>` = „<skip> <fail>" fails the named point | `TestD63_TheReleaseBuildHasNoFaultSwitch` (release build: no-op; build.sh/Dockerfile pass no tag), `TestD63_TheTestBuildFailsTheArmedPointOnDemand` (`go test -tags faultinject`). Points: `defwrite-after-compose`, `stop-<app>`. A live proof of a branch Docker cannot be made to take on demand goes here — never an env switch in the release binary |
| `healthSettle` / `healthInterval` / `healthTimeout` (package vars, R-488) | controller/internal/backup/restore.go | production 3 s / 5 s / 90 s — `waitForHealthy`'s clock, used by every restore path | shortened ONCE in controller/internal/backup/main_test.go `TestMain` (0 / 2 ms / 50 ms; the package went 444 s → ~2 s); `TestR488_HealthWaitDefaultsAreProduction` pins the production values. A new real-clock wait in this package goes behind the same kind of var, not a per-test sleep. Also: `newOffboxManager` installs a failing `SetOffboxSSH` fake, so no backup test spawns a real `ssh` |
| `Server.agentLogsFn` (func seam) | controller/internal/web/server.go | nil → `agentClient().DebugLogs` (agent GET /debug/logs) | injected in controller/internal/web/observability_test.go (incl. the pre-0.83 typed-404 notice path) |
| `escrowAgent` + `Server.escrowAgentFn/escrowStageFn/escrowStaleFn` | controller/internal/web/escrow_handlers.go (+ server.go fields) | `*agentapi.Client` / `PushOffboxPasswordForEscrow` / `report.EscrowAutoConfirmer.StaleBlob` (SetEscrowStale) | `fakeEscrowAgent` + fn injections in escrow_wizard_test.go — call-ORDER assertions (stage BEFORE trigger) + agent-never-called gates. The claim leg is the ONLY surface R crosses: no-store, never logged, never templated |
@@ -453,7 +458,7 @@ Cross-repo edges:
| timeAgo switch body ×2 | controller/internal/web/funcmap.go `timeAgo` vs `timeAgoStr` (identical formatting logic) |
| Budapest timezone loader ×4 | controller/internal/scheduler/scheduler.go `getBudapestLocation` vs controller/internal/web/funcmap.go `getTimezone` vs controller/internal/quiesce/quiesce.go `budapestLocation` vs controller/internal/api/kept_install.go `budapest` (v0.277.0, the kept choice's dates) (v0.168.0 window gate — Budapest wall-clock, kept local to avoid a scheduler↔quiesce import edge) |
| JSON writers ×5, 3 envelope shapes | api `writeJSON`; web `writeDiskJSON`, `jsonResponse`/`jsonError`, `writeDebugJSON` |
| Safe-name validators ×4 | controller/internal/web/validate.go `validStackName`; controller/internal/api/router.go `validStackParam` (same body — api↔web import cycle); controller/internal/backup/offbox.go `isSafeStackName`; controller/internal/appexport/validate.go `ValidateSegment` (strictest) |
| Safe-name validators ×5 | controller/internal/web/validate.go `validStackName`; controller/internal/api/router.go `validStackParam` (same body — api↔web import cycle); controller/internal/backup/offbox.go `isSafeStackName` (exported as `IsValidHoldStack` for the hold commands); controller/internal/report/opactions.go `validHoldStack` (same alphabet as `isSafeStackName` — report cannot import backup; clear_hold's argument check); controller/internal/appexport/validate.go `ValidateSegment` (strictest) |
| DB wait/import ×2 | controller/internal/appbackup/dbdump.go `waitDBReady`/`ImportDump` vs controller/internal/appexport/restore.go `waitForDB`/`importDBDump` |
| qBittorrent Web API login ×2 | controller/internal/mediawire/mediawire.go `qbLogin` (writes the drive wiring) vs controller/internal/upwatch/explain.go `QBittorrent.login` (reads every minute; keeps its session cookie, which mediawire's per-run login does not need) |
| compose exec ×2 | controller/internal/stacks/manager.go `composeExecCustomEnv` vs controller/internal/appexport/restore.go `composeExecEnv` (the latter has ctx+timeout; the former has the userdata belt) |
+51 -2
View File
@@ -1017,7 +1017,8 @@ Each app can define rich metadata in `.felhom.yml`:
the hold is persisted in `Settings.RestoreHolds`, every start path refuses it (customer button,
app-stop `Recover()`, boot sweep, via `driveStartGate` **above** its driveless early return), the
app-stop marker is ended so nothing auto-restarts it, and the row goes red. Cleared with
`--clear-restore-hold <app>`, **which requires a controller restart**. `--single-transaction` on
`--clear-restore-hold <app>` (since 2026-10-29, R-1343: through the running controller, no restart — see
„Operator actions") or the hub's „Clear hold" (`clear_hold`). `--single-transaction` on
the Postgres import is a belt only; MariaDB DDL is not transactional, which is why the rollback
is the fix.
- **Off-site restore of one app: a failed restore puts the app back EXACTLY as it was and starts it (v0.336.0,
@@ -1034,6 +1035,18 @@ Each app can define rich metadata in `.felhom.yml`:
`healthy`; the copy then goes after a further 3 min of the same (10 s samples, watched up to 30 min; a new restore
of the app ends the watch). A restore whose app does not start reads „…még nem indult el…", keeps the copy and
alarms the operator once (D49 = A: nothing is poured back by itself). `restore_verdict.go`. `offbox_restore_undo.go`.
- **Restore edges (unreleased, R-1256, night 2026-10-28).** A put-back whose stop fails pours nothing, keeps the copy
and holds the app; a failed definition write-back keeps the copy; a volume whose `docker volume rm -f` fails is a
failed volume (never unpacked on top of the old content). An off-site restore whose definition write fails writes
the LIVE definition back before the start, or holds the app. Every restore begins in one step with the check for a
restore, the backup run and a second-drive copy (`TryBeginRestoreOp`, also both kept loads). A copy that nothing
needs (no hold of any kind, not named by the app-stop marker, no restore running) goes after 30 days, checked every
6 h (`restore_copy_sweep.go`; operator ruling D62, 2026-10-29).
- **Restore copies in the kept-copies list (unreleased, D62).** The restore page's „Megőrzött másolatok" lists each
restore copy: the app, the restore's date, the size, „automatikusan törlődik: <date>". „Törlés most" deletes one
earlier, after a warning naming the app and the date and the app's name typed (`POST /backup/restore-copy/delete`,
JSON; 409 while a backup/restore runs or the app updates). A copy a hold or a pending put-back needs: the button is
off, the reason on one line. Each delete: one log line and the operator event `restore_copy_deleted`.
- **A restore cut by a power loss, on a drive that comes back late (unreleased, R-1251).** The copy of the live
data lives on the app's drive. When it cannot be reached at start because the drive is not bound yet, the app is
HELD (`putback_pending`; „…A visszaállítás előtti adatai egy meghajtón vannak, amely most nem érhető el…") and no
@@ -1488,6 +1501,10 @@ The nightly backup has two phases that run sequentially. All paths are **per-dri
> **OOM visibility (v0.243.0, R-514).** The 30 s dead-app check also reads `State.OOMKilled` for running app containers (`Manager.ScanOOMKilled`); the dashboard shows „Memória elfogyott" and the hub gets `app_oom` once per container run. The controller does not restart the app.
> **OOM storm (v0.265.0, R-636).** For a flagged container the scan also reads the kernel's own kill counter (`memory.events` `oom_kill`, plus `memory.max` / `memory.peak`) with one `docker exec … cat` — `OOMKilled` is sticky, so it cannot count. When the SAME container run's counter rises by **20 or more within 30 minutes**, the notifier sends **`app_oom_storm`** (severity `error`, operator-only, details `{app, container, kills, window_min, mem_limit, peak}`) — ONCE per container run. `app_oom` itself is unchanged. An unreadable counter (an image without `cat`) never escalates.
> **Host OOM count (unreleased, R-528, night 2026-10-28).** Docker inside an LXC guest misses real OOM kills. The scan
> also asks the agent for the host's per-container `oom_kill` (`GET /guest/oom-kills`, 5 s cap; a 404 from an older
> agent → 10 minutes before the next ask); a count above zero raises `app_oom` whatever `OOMKilled` says, with `source`
> (`docker`, `cgroup` or `docker,cgroup`) in the details. `cmd/controller/hostoom_wiring.go`, `internal/stacks/oom.go`.
> **Multi-tier whole-guest backup (v0.174.0, R-82 Slice B).** The agent can serve SEVERAL whole-guest
> backup tiers with independent cadences — "local daily + PBS weekly" (agent >= v0.97.0,
@@ -2227,10 +2244,16 @@ that folder is never a dead end, and an install never runs into it silently (R-6
- **Controller image retention (v0.285.0, decision 56)** — 3 minutes after every start the box keeps the controller
image it runs and the one before it (the swap record `update-state.json` `previous_image` of a swap that ended on the
running version; without one, the highest version below the running one), every version above the running one (a
pulled target) and every non-version tag (`latest`, `-rc`); older and untagged controller images are deleted by exact
pulled target) and every non-version tag (`latest`); since v0.351.0 a test build (`X.Y.Z-rc.N`) is a version below its
release, so an old one goes too, and the fallback previous prefers a release; older and untagged controller images are deleted by exact
name/ID after the in-use check. Nothing while the controller swaps itself; registry tags are never touched. The
agent's own roll-back needs only the running image (it restores what `/etc/felhom-controller-image` named when the
swap began). See `internal/stacks/controller_image_retention.go`.
- **Infrastructure image retention (v0.351.0, decision 285)** — in the same start-up pass, for each pinned infra image
(`infra.Images()`: traefik, cloudflared, FileBrowser, Samba) the box keeps the pin, every newer version, the highest
version below the pin (the previous pin) and every image a container uses; older and untagged images of that
repository are deleted by exact name/ID. Non-version tags stay. Skipped while an app pulls (R-863, retried every 2
minutes) or while the controller swaps itself. See `internal/stacks/infra_image_retention.go`.
- **The setup gate (v0.280.0, decision 46)** — `.felhom.yml` `setup_gate: true` + optional `setup_done_probe: {url, field,
done}`. A FRESH install is closed to everyone but the household: the traefik file
`<stacks>/traefik/dynamic/setup-gate-<app>.yml` is written BEFORE the first start (a failed write refuses the install) and
@@ -3639,6 +3662,32 @@ Anything else is answered `refused` and calls nothing. Each id runs once per pro
(`operator_action_results: [{id, outcome, message}]`, outcome `done` / `refused` / `failed`). No action deletes data,
starts a countdown or shortens one (`TestOpActions_ClosedList`). The box's log records every action it receives.
- **`clear_hold` (2026-10-29, operator ruling D64 = B; arg = the app's stack name, letters/digits/`-`/`_`, ≤64):** lifts
the app's hold of ANY reason, in the RUNNING controller (`backup.AppStopGuard.ClearHoldByOperator`, the door
`clearHoldNow` in `cmd/controller/holdclear.go`): memory and `settings.json` together, so no later save writes it back
(R-1343). It starts nothing — the household's Start (or a restore) does. A waiting put-back (`putback_pending`) is
dropped from the app-stop marker too; its copy stays on disk. The update's in-memory „held" flag
(`stacks.Manager.ForgetUpdateHeld`) goes with it; the hold reporter's next tick (≤1 min) sends `app_hold_cleared`.
Refused, nothing changed: a bad name, or an app with no hold. No new event type: the hub records the result.
**R-1350 (2026-10-29):** for an app that is not up, the clear records the stop as WANTED (desired state „stopped",
`stacks.Manager.RecordStopWantedIfDown`) once the hold is found and BEFORE it is lifted
(`AppStopGuard.ClearHoldByOperatorThen`), so the dead-app scan sends no „Telepített alkalmazás nem fut" for it and the
boot reconciler does not start it; the household's Start records „running" and ends it.
- **The busy file for the host agent (D65 = A, 2026-10-29; `internal/busymark`).** While a backup, a restore, an
off-site run, the night catch-up, an app update or an app export/import runs, the controller rewrites
`<data dir>/ctl/busy` every 15 s (line 1 the UTC time, line 2 what runs, 0600) and removes it when idle. The agent reads
it before the operator's „Reboot guest" and refuses while it is fresh (≤ 90 s); a controller that stopped writing it
does not block the reboot.
- **„Felhom support restarted your box" (D65 = A).** At start the controller asks the agent `GET /guest/operator-reboot`
(retrying for up to 10 minutes; a 404 from an older agent ends it) and the dashboard shows one neutral line,
„A Felhom ügyfélszolgálata HH:MM-kor újraindította a dobozt." / „Felhom support restarted your box at HH:MM.", until a
day has passed since the reboot.
- **The box command `--clear-restore-hold <app>`** (`pct exec <vmid> -- docker exec felhom-controller felhom-controller
--clear-restore-hold <app>`) asks the running controller over a unix socket, `<data dir>/ctl/hold.sock` (0600 in a
0700 dir, peer uid checked, nothing on TCP — not reachable through traefik, the LAN or an app), and says no restart is
needed. Only when no controller answers does it edit `settings.json` itself, and it says so, naming the in-guest
restart. `--restore-holds` lists the holds (read-only).
#### Report Push (`internal/report/`)
Periodic JSON push (default every 15 min) to the central felhom-hub service:
@@ -0,0 +1,49 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/busymark"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
)
// D65 = A: the agent reads ONE fixed guest path — the controller-data volume + data/ctl/busy
// (felhom-agent ControllerBusyMarkerGuestPath, and its sudoers line). The volume is mounted at
// /opt/docker/felhom-controller, so the controller must write <default data dir>/ctl/busy.
// RED-PROOF: rename holdCtlDir or busymark.FileName → the path moves → FAILS.
func TestBusyMarkPathIsTheOneTheAgentReads(t *testing.T) {
cfg, err := config.LoadFromBytes([]byte("customer:\n id: t\n domain: t.example\n"))
if err != nil {
t.Fatal(err)
}
got := filepath.Join(cfg.Paths.DataDir, holdCtlDir, busymark.FileName)
want := "/opt/docker/felhom-controller/data/ctl/busy" // = the agent's /var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy
if got != want {
t.Fatalf("the busy file is %s, the agent reads %s — an operator reboot would never be refused for a backup", got, want)
}
}
// Every kind of work has its probe and the writer is started — a seam built and never wired is this project's commonest
// defect. RED-PROOF: delete any AddProbe line or the Run → FAILS.
func TestBusyMarkIsWiredAtStartup(t *testing.T) {
raw, err := os.ReadFile("main.go")
if err != nil {
t.Fatal(err)
}
src := string(raw)
for _, want := range []string{
`busyMark.AddProbe("an app update is running", stackMgr.AnyUpdating)`,
`return backupMgr.IsRunning() || backupMgr.RestoreStatus().Running`,
`busyMark.AddProbe("the night backup catch-up is running", catchUp.Running)`,
`busyMark.AddProbe("an app export or import is running", appExporter.IsRunning)`,
`go busyMark.Run(ctx)`,
`go webServer.WatchOperatorReboot(ctx)`, // the household line after an operator reboot
} {
if !strings.Contains(src, want) {
t.Fatalf("main.go does not contain %q", want)
}
}
}
@@ -0,0 +1,34 @@
//go:build faultinject
package main
import (
"os"
"path/filepath"
"testing"
)
// D63 (test build only): "<skip> <fail>" lets <skip> passes through, fails the next <fail>, then removes the file.
func TestD63_TheTestBuildFailsTheArmedPointOnDemand(t *testing.T) {
dir := t.TempDir()
t.Setenv("FELHOM_FAULT_DIR", dir)
p := filepath.Join(dir, "stop-radicale")
if err := os.WriteFile(p, []byte("1 1\n"), 0o600); err != nil {
t.Fatal(err)
}
if err := faultPoint("stop-radicale"); err != nil {
t.Fatalf("the first pass must go through: %v", err)
}
if err := faultPoint("stop-other"); err != nil {
t.Fatalf("another point is not armed: %v", err)
}
if err := faultPoint("stop-radicale"); err == nil {
t.Fatal("the second pass must fail")
}
if _, err := os.Stat(p); !os.IsNotExist(err) {
t.Fatalf("the spent file must be gone: %v", err)
}
if err := faultPoint("stop-radicale"); err != nil {
t.Fatalf("a spent point fails nothing: %v", err)
}
}
@@ -0,0 +1,40 @@
//go:build !faultinject
package main
import (
"os"
"path/filepath"
"regexp"
"strings"
"testing"
)
// goBuildTag is a `go build -tags …` flag (build.sh's own `--print-tags` option is not one).
var goBuildTag = regexp.MustCompile(`\s-tags[ =]`)
// D63: the fault switch exists only in a `-tags faultinject` test build. In this (the release) build an armed fault
// file injects nothing, and neither build.sh nor the Dockerfile passes a build tag.
// RED-PROOF: build the release build with the switch (drop the !faultinject line from faultpoint_off.go and the
// faultinject line from faultpoint_on.go) → the armed point fails → FAILS.
func TestD63_TheReleaseBuildHasNoFaultSwitch(t *testing.T) {
dir := t.TempDir()
t.Setenv("FELHOM_FAULT_DIR", dir)
for _, p := range []string{"defwrite-after-compose", "stop-radicale"} {
if err := os.WriteFile(filepath.Join(dir, p), []byte("0 5\n"), 0o600); err != nil {
t.Fatal(err)
}
if err := faultPoint(p); err != nil {
t.Fatalf("the release build injected a fault at %s: %v", p, err)
}
}
for _, f := range []string{"../../build.sh", "../../Dockerfile"} {
b, err := os.ReadFile(f)
if err != nil {
t.Fatal(err)
}
if goBuildTag.MatchString(string(b)) || strings.Contains(string(b), "faultinject") {
t.Fatalf("%s passes a build tag — the release image could carry the fault switch", f)
}
}
}
@@ -0,0 +1,9 @@
//go:build !faultinject
package main
// faultPoint is the D63 live-proof fault switch (operator ruling 2026-10-29: prove R-1256 H4 and H5's failed stop live,
// with a fault switch in a TEST build only). In every build without the `faultinject` tag — the release image among
// them: build.sh and the Dockerfile pass no build tags — it does nothing and reads nothing. Pinned by
// TestD63_TheReleaseBuildHasNoFaultSwitch; the switch itself is faultpoint_on.go.
func faultPoint(string) error { return nil }
@@ -0,0 +1,64 @@
//go:build faultinject
package main
import (
"fmt"
"log"
"os"
"path/filepath"
"strconv"
"strings"
"sync"
)
// faultPoint (TEST BUILDS ONLY — `go build -tags faultinject`; never the release image, faultpoint_off.go) fails a named
// point on demand, for the D63 live proofs of R-1256 H4 (a definition write that fails after the compose was written)
// and H5 (a put-back whose stop fails). Armed by a file <dir>/<point> holding "<skip> <fail>": the next <skip> passes go
// through, the <fail> after them fail, and the file is removed when both are spent. <dir> is $FELHOM_FAULT_DIR, else
// /opt/docker/felhom-controller/faults (the controller's data volume). A file that does not parse fails nothing.
var faultMu sync.Mutex
func faultPoint(point string) error {
faultMu.Lock()
defer faultMu.Unlock()
dir := os.Getenv("FELHOM_FAULT_DIR")
if dir == "" {
dir = "/opt/docker/felhom-controller/faults"
}
p := filepath.Join(dir, point)
data, err := os.ReadFile(p)
if err != nil {
return nil
}
f := strings.Fields(string(data))
if len(f) != 2 {
log.Printf("[WARN] [faultinject] %s: %q does not parse as \"<skip> <fail>\" — nothing injected", p, strings.TrimSpace(string(data)))
return nil
}
skip, e1 := strconv.Atoi(f[0])
fail, e2 := strconv.Atoi(f[1])
if e1 != nil || e2 != nil || skip < 0 || fail < 0 {
return nil
}
write := func() {
if skip == 0 && fail == 0 {
_ = os.Remove(p)
return
}
_ = os.WriteFile(p, []byte(fmt.Sprintf("%d %d\n", skip, fail)), 0o600)
}
if skip > 0 {
skip--
write()
log.Printf("[WARN] [faultinject] %s: passed (skip left %d, fail left %d)", point, skip, fail)
return nil
}
if fail > 0 {
fail--
write()
log.Printf("[ERROR] [faultinject] %s: FAULT INJECTED (fail left %d)", point, fail)
return fmt.Errorf("FAULT INJECTED at %s (D63 test build)", point)
}
return nil
}
+282
View File
@@ -0,0 +1,282 @@
package main
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net"
"os"
"path/filepath"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
"gitea.dooplex.hu/admin/felhom-controller/internal/report"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// ── D64 = B + R-1343: the operator clears a hold in the RUNNING controller ─────────────────────────────────────────
//
// Two routes reach the same door (clearHoldNow):
// - the hub's „Clear hold" → the operator action clear_hold (internal/report/opactions.go), run in this process;
// - the box command `felhom-controller --clear-restore-hold <app>` (run with `docker exec` inside the guest) → the
// local socket below → the running controller.
// The running controller clears the hold in memory AND saves settings.json at once (backup.ClearHoldByOperator), so
// no later save — the hold reporter's, a hub report's — can write it back (R-1343). The household's „held" sentence
// goes with it: every reader (the page, the start gate, the API, the hold reporter) reads the same settings record.
// The ONE held flag kept elsewhere — stacks.Manager's „the last update ended held" — is forgotten here too.
// holdClearDeps are clearHoldNow's seams (production: holdClearDepsFor).
type holdClearDeps struct {
clear func(stack string, beforeLift func()) (settings.RestoreHold, error) // backup.AppStopGuard.ClearHoldByOperatorThen
forgetUpdateHeld func(stack string) bool // stacks.Manager.ForgetUpdateHeld (nil = none)
// keepStopped (R-1350) records that the stop is wanted for an app that is not up — run once the hold is found and
// BEFORE it is lifted: stacks.Manager.RecordStopWantedIfDown (nil = none).
keepStopped func(stack string) (bool, error)
logger *log.Logger
}
// clearHoldNow lifts stack's hold and returns the result sentence (English, operator-facing). who names the person
// and the route for the box's own log, e.g. „the operator via the hub, action #12". A refusal (bad name, no hold) is
// wrapped in report.ErrActionRefused, and nothing was changed.
func clearHoldNow(d holdClearDeps, stack, who string) (string, error) {
if d.clear == nil {
return "", fmt.Errorf("%w: clearing a hold is not available on this box", report.ErrActionRefused)
}
var before func()
if d.keepStopped != nil {
before = func() {
rec, kerr := d.keepStopped(stack)
switch {
case kerr != nil && d.logger != nil:
d.logger.Printf("[WARN] [hold] %s: the stop could not be recorded as wanted (%v) — the dead-app alarm may follow the clear (R-1350)", stack, kerr)
case rec && d.logger != nil:
d.logger.Printf("[INFO] [hold] %s: the stop is recorded as wanted (desired state: stopped) — no dead-app alarm; the household's Start ends it (R-1350)", stack)
}
}
}
h, err := d.clear(stack, before)
switch {
case errors.Is(err, backup.ErrHoldBadStack):
return "", fmt.Errorf("%w: %q is not a valid app name — nothing was changed", report.ErrActionRefused, stack)
case errors.Is(err, backup.ErrNotHeld):
return "", fmt.Errorf("%w: %s has no hold — nothing was changed", report.ErrActionRefused, stack)
case err != nil:
return "", fmt.Errorf("clearing the hold on %s failed: %v", stack, err)
}
if d.forgetUpdateHeld != nil && d.forgetUpdateHeld(stack) {
if d.logger != nil {
d.logger.Printf("[INFO] [hold] %s: the update's held state is forgotten with the hold", stack)
}
}
if d.logger != nil {
d.logger.Printf("[WARN] [hold] %s: hold CLEARED by %s — it was %s (set %s); the app is not started by this", stack, who, h.WireReason(), h.At)
}
msg := fmt.Sprintf("the hold on %s is cleared (it was: %s). The app is NOT started by this — the household's Start, or a restore, starts it", stack, h.Why())
if h.Reason == settings.HoldReasonPutBackPending {
dir := h.SafetyDump
if dir == "" {
dir = "(not named in the hold)"
}
msg += ". Its waiting put-back is dropped: the pre-restore copy stays on disk at " + dir + " and is never poured back by itself"
}
return msg, nil
}
// ── the local channel (R-1343) ────────────────────────────────────────────────────────────────────────────────────
//
// WHY A UNIX SOCKET IN THE CONTROLLER'S DATA DIR, mode 0600 in a 0700 directory, and nothing on TCP:
// - traefik, the LAN and every app reach the controller only over TCP (:8080, :8081). A unix socket has no port,
// so none of them can reach it, whatever a proxy header or a forwarded request says;
// - the data dir is the controller's own volume: no app container mounts it, and only root in the guest
// (`pct exec <vmid> -- docker exec felhom-controller …`, the operator gate this project already uses) can enter
// the controller's container to use it;
// - the directory is created 0700 BEFORE the socket is bound, so the moment between bind and chmod is not a window;
// - the peer's uid must equal this process's (SO_PEERCRED, holdctl_linux.go) — a second lock on the same door.
// The customer's HTTP surface is deliberately NOT used: it authenticates as the household, and a household clearing
// its own hold is what the hold exists to prevent. Pinned by TestHoldCtl_NotReachableFromOutside.
const (
holdCtlDir = "ctl"
holdCtlSock = "hold.sock"
// holdCtlOpClear is the one request this channel answers.
holdCtlOpClear = "clear_hold"
// holdCtlMaxRequest bounds what one request may send.
holdCtlMaxRequest = 4096
)
// holdCtlPath is the socket's path under the controller's data dir.
func holdCtlPath(dataDir string) string { return filepath.Join(dataDir, holdCtlDir, holdCtlSock) }
type holdCtlRequest struct {
Op string `json:"op"`
Stack string `json:"stack"`
}
type holdCtlResponse struct {
Outcome string `json:"outcome"` // done | refused | failed (report.Outcome*)
Message string `json:"message"`
}
// listenHoldCtl binds the socket and serves it until the listener is closed (closing it removes the socket file).
// clear is clearHoldNow bound to the running controller's deps.
func listenHoldCtl(path string, clear func(stack, who string) (string, error), logger *log.Logger) (net.Listener, error) {
dir := filepath.Dir(path)
if err := os.MkdirAll(dir, 0o700); err != nil {
return nil, err
}
if err := os.Chmod(dir, 0o700); err != nil {
return nil, err
}
// A socket left by a controller that died is replaced; anything else at that path is refused, never removed.
if fi, err := os.Lstat(path); err == nil {
if fi.Mode()&os.ModeSocket == 0 {
return nil, fmt.Errorf("%s exists and is not a socket — refusing to replace it", path)
}
if err := os.Remove(path); err != nil {
return nil, err
}
}
ln, err := net.Listen("unix", path)
if err != nil {
return nil, err
}
if err := os.Chmod(path, 0o600); err != nil {
ln.Close()
return nil, err
}
go func() {
for {
conn, err := ln.Accept()
if err != nil {
if !errors.Is(err, net.ErrClosed) && logger != nil {
logger.Printf("[WARN] [hold] local channel stopped: %v", err)
}
return
}
go serveHoldCtl(conn, clear, logger)
}
}()
return ln, nil
}
func serveHoldCtl(conn net.Conn, clear func(stack, who string) (string, error), logger *log.Logger) {
defer conn.Close()
_ = conn.SetDeadline(time.Now().Add(30 * time.Second))
answer := func(r holdCtlResponse) { _ = json.NewEncoder(conn).Encode(r) }
if ok, why := holdCtlPeerAllowed(conn); !ok {
if logger != nil {
logger.Printf("[WARN] [hold] local channel: a request was refused — %s", why)
}
answer(holdCtlResponse{Outcome: report.OutcomeRefused, Message: "refused: " + why})
return
}
line, err := bufio.NewReader(io.LimitReader(conn, holdCtlMaxRequest)).ReadBytes('\n')
if err != nil && len(line) == 0 {
answer(holdCtlResponse{Outcome: report.OutcomeRefused, Message: "no request was read"})
return
}
var req holdCtlRequest
if err := json.Unmarshal(line, &req); err != nil || req.Op != holdCtlOpClear {
answer(holdCtlResponse{Outcome: report.OutcomeRefused, Message: "unknown request — this channel only clears a hold"})
return
}
msg, err := clear(req.Stack, "the operator on the box (--clear-restore-hold)")
switch {
case err == nil:
answer(holdCtlResponse{Outcome: report.OutcomeDone, Message: msg})
case errors.Is(err, report.ErrActionRefused):
answer(holdCtlResponse{Outcome: report.OutcomeRefused, Message: strings.TrimPrefix(err.Error(), report.ErrActionRefused.Error()+": ")})
default:
answer(holdCtlResponse{Outcome: report.OutcomeFailed, Message: err.Error()})
}
}
// askRunningControllerToClear sends one clear request. reached=false: no controller answered at path (no socket, or
// a socket nobody listens on) — nothing was asked, so the caller may edit settings.json itself. reached=true with an
// error: the request may have been applied; the caller must not edit the file.
func askRunningControllerToClear(path, stack string) (resp holdCtlResponse, reached bool, err error) {
conn, err := net.DialTimeout("unix", path, 3*time.Second)
if err != nil {
return holdCtlResponse{}, false, err
}
defer conn.Close()
_ = conn.SetDeadline(time.Now().Add(60 * time.Second))
if err := json.NewEncoder(conn).Encode(holdCtlRequest{Op: holdCtlOpClear, Stack: stack}); err != nil {
return holdCtlResponse{}, true, err
}
if err := json.NewDecoder(io.LimitReader(conn, 64*1024)).Decode(&resp); err != nil {
return holdCtlResponse{}, true, err
}
return resp, true, nil
}
// clearRestoreHoldCLI is the box command `--clear-restore-hold <app>` (R-379, R-1343). It asks the running
// controller first; only when none answers does it edit settings.json itself, and it says which one happened.
// Returns the process exit code: 0 cleared, 2 refused (nothing changed), 1 failed or unknown.
func clearRestoreHoldCLI(dataDir, stack string, stdout, stderr io.Writer) int {
path := holdCtlPath(dataDir)
resp, reached, err := askRunningControllerToClear(path, stack)
if reached {
if err != nil {
fmt.Fprintf(stderr, "the running controller was reached, but its answer was lost: %v\n", err)
fmt.Fprintf(stderr, " The hold may or may not be cleared. Check with: felhom-controller --restore-holds\n")
fmt.Fprintf(stderr, " settings.json was NOT edited by this command.\n")
return 1
}
switch resp.Outcome {
case report.OutcomeDone:
fmt.Fprintf(stdout, "%s.\n", resp.Message)
fmt.Fprintf(stdout, " Cleared by the RUNNING controller (memory and settings.json together) — no restart is needed.\n")
return 0
case report.OutcomeRefused:
fmt.Fprintf(stderr, "%s\n", resp.Message)
return 2
default:
fmt.Fprintf(stderr, "the running controller could not clear the hold: %s\n", resp.Message)
return 1
}
}
// No controller answered: it is not running (or is older than this channel). The file is the only copy.
fmt.Fprintf(stdout, "No running controller answered on %s (%v).\n", path, err)
fmt.Fprintf(stdout, " Editing settings.json directly instead.\n")
if !backup.IsValidHoldStack(stack) {
fmt.Fprintf(stderr, "%q is not a valid app name — nothing was changed\n", stack)
return 2
}
lg := log.New(stderr, "", 0)
sett, lerr := settings.Load(filepath.Join(dataDir, "settings.json"), lg)
if lerr != nil {
fmt.Fprintf(stderr, "restore-hold: loading settings: %v\n", lerr)
return 1
}
// The same door as the running controller's (a waiting put-back is dropped from the app-stop marker too), over
// this process's copy — the only copy while no controller runs.
g := backup.NewAppStopGuard(filepath.Join(dataDir, "appstop-state.json"), lg)
g.SetHolds(sett)
h, cerr := g.ClearHoldByOperator(stack)
switch {
case errors.Is(cerr, backup.ErrNotHeld):
// "there was nothing to clear" is not success — a silent 0 here would let an operator believe they had
// unblocked an app whose name they mistyped.
fmt.Fprintf(stderr, "no restore hold in force for %q — nothing was changed\n", stack)
return 2
case cerr != nil:
fmt.Fprintf(stderr, "restore-hold: clearing %s: %v\n", stack, cerr)
return 1
}
fmt.Fprintf(stdout, "restore hold cleared for %s in settings.json (no controller was running to ask; it was: %s).\n", stack, h.Why())
if h.Reason == settings.HoldReasonPutBackPending {
fmt.Fprintf(stdout, " Its waiting put-back is dropped; the pre-restore copy stays on disk at %s.\n", orNone(h.SafetyDump))
}
fmt.Fprintf(stdout, " The controller reads it when it starts. If a controller IS running in this guest after all, restart it NOW,\n")
fmt.Fprintf(stdout, " INSIDE THE GUEST — from the Proxmox host: pct exec <vmid> -- systemctl restart felhom-controller-bootstrap.service\n")
fmt.Fprintf(stdout, " — or its next settings save writes the hold back (R-1343).\n")
return 0
}
// holdClearCommandHint is the exact guest command, for the help text and the operator's mail.
const holdClearCommandHint = "pct exec <vmid> -- docker exec felhom-controller felhom-controller --clear-restore-hold <app>"
+559
View File
@@ -0,0 +1,559 @@
package main
import (
"bytes"
"context"
"encoding/json"
"go/ast"
"go/parser"
"go/token"
"io"
"log"
"net"
"os"
"path/filepath"
"strconv"
"strings"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
"gitea.dooplex.hu/admin/felhom-controller/internal/holdreport"
"gitea.dooplex.hu/admin/felhom-controller/internal/report"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// D64 = B (operator ruling 2026-10-29 12:10) + R-1343: the operator clears a hold — from the hub (clear_hold) or with
// the box command — IN THE RUNNING CONTROLLER, so the hold is gone from memory and from settings.json together and no
// later save brings it back.
//
// RED-PROOFS (2026-10-29):
// - TestR1343_TheBoxCommandClearsThroughTheRunningController: with clearRestoreHoldCLI's socket call skipped (the
// old second-process edit of settings.json), it fails on „the hold came back after a later save of the running
// controller" — the R-1343 measurement, in a test;
// - TestD64_ClearPutBackPendingDropsTheMarkerEntry: with the dropFromMarker call removed from
// ClearHoldByOperator, it fails on „a later controller start poured the old copy back";
// - TestD64_ClearHoldViaTheHub: with the forgetUpdateHeld call removed from clearHoldNow, it fails on „the
// update's held flag was not forgotten".
// - TestD64_ClearHoldViaTheHub + TestD64_ClearHoldStaysGoneAfterALaterSave: with ClearHoldByOperator clearing a
// SECOND Settings loaded from disk (the old CLI's shape) instead of the running one, they fail on „the running
// controller still holds the app" / „the hold came back after a later save of the running controller".
// d64Rig is a running controller's hold side: its Settings, the app-stop guard wired to them, and a log.
type d64Rig struct {
dir string
sett *settings.Settings
guard *backup.AppStopGuard
logBuf *syncBuffer
logger *log.Logger
forgot []string
holdDeps holdClearDeps
}
type syncBuffer struct {
mu sync.Mutex
b bytes.Buffer
}
func (s *syncBuffer) Write(p []byte) (int, error) {
s.mu.Lock()
defer s.mu.Unlock()
return s.b.Write(p)
}
func (s *syncBuffer) String() string { s.mu.Lock(); defer s.mu.Unlock(); return s.b.String() }
func newD64Rig(t *testing.T) *d64Rig {
t.Helper()
r := &d64Rig{dir: t.TempDir(), logBuf: &syncBuffer{}}
r.logger = log.New(r.logBuf, "", 0)
sett, err := settings.Load(filepath.Join(r.dir, "settings.json"), r.logger)
if err != nil {
t.Fatal(err)
}
r.sett = sett
r.guard = backup.NewAppStopGuard(filepath.Join(r.dir, "appstop-state.json"), r.logger)
r.guard.SetHolds(sett)
r.holdDeps = holdClearDeps{
clear: r.guard.ClearHoldByOperatorThen,
forgetUpdateHeld: func(s string) bool { r.forgot = append(r.forgot, s); return true },
logger: r.logger,
}
return r
}
func (r *d64Rig) hold(t *testing.T, h settings.RestoreHold) {
t.Helper()
if h.At == "" {
h.At = "2026-10-29T08:18:00Z"
}
if err := r.sett.SetRestoreHold(h); err != nil {
t.Fatal(err)
}
}
func (r *d64Rig) onDisk(t *testing.T, stack string) bool {
t.Helper()
s, err := settings.Load(filepath.Join(r.dir, "settings.json"), log.New(io.Discard, "", 0))
if err != nil {
t.Fatal(err)
}
_, held := s.GetRestoreHold(stack)
return held
}
func (r *d64Rig) actions() *report.OperatorActions {
return report.NewOperatorActions(context.Background(), operatorActionHandlers(nil, nil, r.holdDeps, nil, r.logger), r.logger)
}
func onlyResult(t *testing.T, o *report.OperatorActions, id int64) report.OperatorActionResult {
t.Helper()
for _, x := range o.Results() {
if x.ID == id {
return x
}
}
t.Fatalf("no result for #%d: %+v", id, o.Results())
return report.OperatorActionResult{}
}
// The hub's clear_hold: the hold is gone (memory and disk), the household's held sentence is gone, the update's
// held flag is forgotten, the box logs who did it, and the result names what was cleared.
func TestD64_ClearHoldViaTheHub(t *testing.T) {
r := newD64Rig(t)
r.hold(t, settings.RestoreHold{Stack: "paperless", ReplayError: "ERROR: relation exists", RollbackErr: "psql: refused"})
cfg := &config.Config{}
cfg.Paths.DataDir = r.dir
mgr := backup.NewManager(cfg, r.sett, r.logger)
if held, why := mgr.RestoreHoldForLang("paperless", "hu"); !held || why == "" {
t.Fatal("fixture: the household must read a held sentence before the clear")
}
o := r.actions()
o.Reconcile([]report.OperatorAction{{ID: 12, Action: report.OpClearHold, Arg: "paperless"}})
res := onlyResult(t, o, 12)
if res.Outcome != report.OutcomeDone {
t.Fatalf("result %+v", res)
}
for _, want := range []string{"the hold on paperless is cleared", "held after a failed restore whose rollback also failed", "NOT started by this"} {
if !strings.Contains(res.Message, want) {
t.Errorf("result message %q lacks %q", res.Message, want)
}
}
if _, held := r.sett.GetRestoreHold("paperless"); held {
t.Fatal("the running controller still holds the app")
}
if r.onDisk(t, "paperless") {
t.Fatal("settings.json still holds the app")
}
if held, why := mgr.RestoreHoldForLang("paperless", "hu"); held || why != "" {
t.Fatalf("the household still reads a held sentence: %q", why)
}
if len(r.forgot) != 1 || r.forgot[0] != "paperless" {
t.Fatalf("the update's held flag was not forgotten: %v", r.forgot)
}
if lg := r.logBuf.String(); !strings.Contains(lg, "paperless: hold CLEARED by the operator via the hub, action #12") {
t.Fatalf("the box's log must say who cleared it:\n%s", lg)
}
}
// Refusals change nothing: an app nobody ever held, a second press on an app already cleared, a bad name.
func TestD64_ClearHoldRefusals(t *testing.T) {
r := newD64Rig(t)
r.hold(t, settings.RestoreHold{Stack: "kimai", Reason: settings.HoldReasonUpdateFailed})
o := r.actions()
o.Reconcile([]report.OperatorAction{{ID: 1, Action: report.OpClearHold, Arg: "nosuchapp"}})
if res := onlyResult(t, o, 1); res.Outcome != report.OutcomeRefused || !strings.Contains(res.Message, "nosuchapp has no hold") {
t.Fatalf("unknown app: %+v", res)
}
if _, held := r.sett.GetRestoreHold("kimai"); !held || !r.onDisk(t, "kimai") {
t.Fatal("a refusal changed another app's hold")
}
if len(r.forgot) != 0 {
t.Fatalf("a refusal touched the update state: %v", r.forgot)
}
o.Reconcile([]report.OperatorAction{{ID: 2, Action: report.OpClearHold, Arg: "kimai"}})
o.Reconcile([]report.OperatorAction{{ID: 3, Action: report.OpClearHold, Arg: "kimai"}})
if res := onlyResult(t, o, 3); res.Outcome != report.OutcomeRefused || !strings.Contains(res.Message, "kimai has no hold") {
t.Fatalf("not held (second press): %+v", res)
}
if _, err := clearHoldNow(r.holdDeps, "../kimai", "test"); err == nil || !strings.Contains(err.Error(), "not a valid app name") {
t.Fatalf("a bad name must be refused by the door itself too: %v", err)
}
}
// R-1343's order: clear, then ANOTHER component saves settings (the hold reporter, a hub report), then the file is
// read back — the hold must stay gone, in memory and on disk.
func TestD64_ClearHoldStaysGoneAfterALaterSave(t *testing.T) {
r := newD64Rig(t)
r.hold(t, settings.RestoreHold{Stack: "radicale", Reason: settings.HoldReasonRestoreMixed})
o := r.actions()
o.Reconcile([]report.OperatorAction{{ID: 5, Action: report.OpClearHold, Arg: "radicale"}})
if res := onlyResult(t, o, 5); res.Outcome != report.OutcomeDone {
t.Fatalf("result %+v", res)
}
if err := r.sett.SetReportedHolds(map[string]string{"other": "x|y"}); err != nil { // another component's save
t.Fatal(err)
}
if r.onDisk(t, "radicale") {
t.Fatal("the hold came back after a later save of the running controller")
}
if _, held := r.sett.GetRestoreHold("radicale"); held {
t.Fatal("the running controller holds the app again")
}
}
// The same order through the BOX COMMAND: it reaches the running controller over the socket; no restart.
func TestR1343_TheBoxCommandClearsThroughTheRunningController(t *testing.T) {
r := newD64Rig(t)
r.hold(t, settings.RestoreHold{Stack: "radicale", Reason: settings.HoldReasonRestoreMixed})
ln, err := listenHoldCtl(holdCtlPath(r.dir), func(stack, who string) (string, error) { return clearHoldNow(r.holdDeps, stack, who) }, r.logger)
if err != nil {
t.Fatal(err)
}
defer ln.Close()
var out, errb bytes.Buffer
if code := clearRestoreHoldCLI(r.dir, "radicale", &out, &errb); code != 0 {
t.Fatalf("exit %d, stdout %q, stderr %q", code, out.String(), errb.String())
}
if err := r.sett.SetReportedHolds(map[string]string{"other": "x|y"}); err != nil { // the running controller's next save
t.Fatal(err)
}
if r.onDisk(t, "radicale") {
t.Fatal("the hold came back after a later save of the running controller")
}
if !strings.Contains(out.String(), "Cleared by the RUNNING controller") || !strings.Contains(out.String(), "no restart is needed") {
t.Fatalf("the command must say it went through the running controller and needs no restart:\n%s", out.String())
}
if !strings.Contains(r.logBuf.String(), "hold CLEARED by the operator on the box (--clear-restore-hold)") {
t.Fatalf("the box's log must say who cleared it:\n%s", r.logBuf.String())
}
// A second run: refused through the socket, exit 2, nothing edited.
out.Reset()
errb.Reset()
if code := clearRestoreHoldCLI(r.dir, "radicale", &out, &errb); code != 2 || !strings.Contains(errb.String(), "radicale has no hold") {
t.Fatalf("not held: exit %d, stderr %q", code, errb.String())
}
}
// No controller runs (no socket, or a socket nobody listens on): the command edits settings.json and says so plainly,
// naming the guest restart for the case a controller runs after all.
func TestR1343_TheBoxCommandFallsBackWhenNoControllerRuns(t *testing.T) {
for _, stale := range []bool{false, true} {
r := newD64Rig(t)
r.hold(t, settings.RestoreHold{Stack: "kimai", Reason: settings.HoldReasonUpdateFailed})
if stale { // a socket left by a controller that died
ln, err := listenHoldCtl(holdCtlPath(r.dir), func(string, string) (string, error) { return "", nil }, r.logger)
if err != nil {
t.Fatal(err)
}
ln.(*net.UnixListener).SetUnlinkOnClose(false)
ln.Close()
}
var out, errb bytes.Buffer
if code := clearRestoreHoldCLI(r.dir, "kimai", &out, &errb); code != 0 {
t.Fatalf("stale=%v: exit %d, stderr %q", stale, code, errb.String())
}
for _, want := range []string{"No running controller answered", "Editing settings.json directly", "INSIDE THE GUEST", "pct exec <vmid> -- systemctl restart felhom-controller-bootstrap.service"} {
if !strings.Contains(out.String(), want) {
t.Errorf("stale=%v: output lacks %q:\n%s", stale, want, out.String())
}
}
if r.onDisk(t, "kimai") {
t.Fatalf("stale=%v: settings.json still holds the app", stale)
}
out.Reset()
errb.Reset()
if code := clearRestoreHoldCLI(r.dir, "kimai", &out, &errb); code != 2 {
t.Fatalf("stale=%v: not held must exit 2, got %d", stale, code)
}
if code := clearRestoreHoldCLI(r.dir, "../x", &out, &errb); code != 2 {
t.Fatalf("stale=%v: a bad name must exit 2, got %d", stale, code)
}
}
}
// A put-back WAITING for its drive is also in the app-stop marker: clearing the hold must drop it there too, or the
// next controller start with the drive bound would pour the old copy over what the app wrote after the clear. The
// copy itself stays on disk (nothing is deleted).
func TestD64_ClearPutBackPendingDropsTheMarkerEntry(t *testing.T) {
r := newD64Rig(t)
undo := filepath.Join(r.dir, "drive", "felhom-data", "backups", "restore-undo", "immich", "20261029T080000Z")
if err := os.MkdirAll(undo, 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(undo, "undo.json"), []byte(`{"stack":"immich"}`), 0o600); err != nil {
t.Fatal(err)
}
if err := r.guard.Begin("unit-restore:immich", backup.ReasonUnitRestore, []string{"immich"}); err != nil {
t.Fatal(err)
}
if err := r.guard.SetUndo("immich", undo); err != nil {
t.Fatal(err)
}
r.hold(t, settings.RestoreHold{Stack: "immich", Reason: settings.HoldReasonPutBackPending, SafetyDump: undo})
o := r.actions()
o.Reconcile([]report.OperatorAction{{ID: 3, Action: report.OpClearHold, Arg: "immich"}})
res := onlyResult(t, o, 3)
if res.Outcome != report.OutcomeDone || !strings.Contains(res.Message, "waiting put-back is dropped") || !strings.Contains(res.Message, undo) {
t.Fatalf("result %+v", res)
}
if _, err := os.Stat(filepath.Join(undo, "undo.json")); err != nil {
t.Fatal("the copy must stay on disk")
}
// The consequence: a later controller start with the drive bound pours nothing back.
poured := &recordingPutBacker{}
g2 := backup.NewAppStopGuard(filepath.Join(r.dir, "appstop-state.json"), r.logger)
g2.SetHolds(r.sett)
g2.SetPutBacker(poured)
g2.SetStarter(nopStarter{})
g2.SetUndoReachable(func(string, string) bool { return true })
g2.Recover()
if len(poured.stacks) != 0 {
t.Fatalf("a later controller start poured the old copy back over the app: %v", poured.stacks)
}
}
type recordingPutBacker struct{ stacks []string }
func (p *recordingPutBacker) PutBackInterruptedRestore(stack, _ string) error {
p.stacks = append(p.stacks, stack)
return nil
}
type nopStarter struct{}
func (nopStarter) StartStack(string) error { return nil }
// The hold reporter's next tick tells the hub the hold is gone (it reads the same settings record; nothing cached).
func TestD64_TheHoldReporterTellsTheHubAfterTheClear(t *testing.T) {
r := newD64Rig(t)
r.hold(t, settings.RestoreHold{Stack: "kimai", Reason: settings.HoldReasonUpdateFailed})
var events []string
rep := holdreport.New(r.sett, func(et, _, _ string, _ interface{}) { events = append(events, et) }, r.logger)
rep.Tick(timeAt(0))
o := r.actions()
o.Reconcile([]report.OperatorAction{{ID: 1, Action: report.OpClearHold, Arg: "kimai"}})
rep.Tick(timeAt(60))
if strings.Join(events, ",") != holdreport.EventHeld+","+holdreport.EventCleared {
t.Fatalf("events %v — want app_held then app_hold_cleared", events)
}
}
// „Not reachable from outside": the channel is a unix socket (no TCP anywhere in the file), 0600 in a 0700 directory
// — even when the directory existed with a looser mode — and main opens it once, under the data dir.
func TestHoldCtl_NotReachableFromOutside(t *testing.T) {
dir := t.TempDir()
if err := os.MkdirAll(filepath.Join(dir, holdCtlDir), 0o755); err != nil {
t.Fatal(err)
}
ln, err := listenHoldCtl(holdCtlPath(dir), func(string, string) (string, error) { return "", nil }, nil)
if err != nil {
t.Fatal(err)
}
defer ln.Close()
if n := ln.Addr().Network(); n != "unix" {
t.Fatalf("the channel listens on %q, want unix", n)
}
fi, err := os.Lstat(holdCtlPath(dir))
if err != nil || fi.Mode()&os.ModeSocket == 0 || fi.Mode().Perm() != 0o600 {
t.Fatalf("socket mode %v (err %v), want a socket with 0600", fi.Mode(), err)
}
di, _ := os.Stat(filepath.Join(dir, holdCtlDir))
if di.Mode().Perm() != 0o700 {
t.Fatalf("socket dir mode %v, want 0700", di.Mode().Perm())
}
// No TCP and no HTTP in the channel's file: every net.Listen/Dial names "unix".
f, err := parser.ParseFile(token.NewFileSet(), "holdclear.go", nil, parser.ImportsOnly)
if err != nil {
t.Fatal(err)
}
for _, im := range f.Imports {
if p, _ := strconv.Unquote(im.Path.Value); strings.HasPrefix(p, "net/http") {
t.Fatalf("holdclear.go imports %s — the channel must not be HTTP", p)
}
}
full, err := parser.ParseFile(token.NewFileSet(), "holdclear.go", nil, 0)
if err != nil {
t.Fatal(err)
}
nets := 0
ast.Inspect(full, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
sel, ok := call.Fun.(*ast.SelectorExpr)
if !ok {
return true
}
if x, ok := sel.X.(*ast.Ident); ok && x.Name == "net" && len(call.Args) > 0 {
nets++
if lit, ok := call.Args[0].(*ast.BasicLit); !ok || lit.Value != `"unix"` {
t.Errorf("net.%s is called with a network other than \"unix\"", sel.Sel.Name)
}
}
return true
})
if nets < 2 {
t.Fatalf("expected the listen and the dial in holdclear.go, found %d net calls", nets)
}
// main opens it exactly once, at holdCtlPath(cfg.Paths.DataDir).
mf, err := parser.ParseFile(token.NewFileSet(), "main.go", nil, 0)
if err != nil {
t.Fatal(err)
}
opened := 0
ast.Inspect(mf, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
if id, ok := call.Fun.(*ast.Ident); ok && id.Name == "listenHoldCtl" {
opened++
arg, ok := call.Args[0].(*ast.CallExpr)
if fn, ok2 := arg.Fun.(*ast.Ident); !ok || !ok2 || fn.Name != "holdCtlPath" {
t.Error("main must open the channel at holdCtlPath(<data dir>)")
}
}
return true
})
if opened != 1 {
t.Fatalf("main.go opens the channel %d time(s), want 1", opened)
}
}
// The socket answers only its one request; anything else is refused and calls nothing.
func TestHoldCtl_UnknownRequestIsRefused(t *testing.T) {
dir := t.TempDir()
called := 0
ln, err := listenHoldCtl(holdCtlPath(dir), func(string, string) (string, error) { called++; return "x", nil }, nil)
if err != nil {
t.Fatal(err)
}
defer ln.Close()
conn, err := net.Dial("unix", holdCtlPath(dir))
if err != nil {
t.Fatal(err)
}
defer conn.Close()
if _, err := conn.Write([]byte(`{"op":"set_hold","stack":"x"}` + "\n")); err != nil {
t.Fatal(err)
}
var resp holdCtlResponse
if err := json.NewDecoder(conn).Decode(&resp); err != nil {
t.Fatal(err)
}
if resp.Outcome != report.OutcomeRefused || called != 0 {
t.Fatalf("response %+v, calls %d — want refused and nothing called", resp, called)
}
}
// Something at the socket's path that is NOT a socket is refused, never removed.
func TestHoldCtl_ANonSocketFileIsNotReplaced(t *testing.T) {
dir := t.TempDir()
p := holdCtlPath(dir)
if err := os.MkdirAll(filepath.Dir(p), 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte("keep me"), 0o600); err != nil {
t.Fatal(err)
}
if _, err := listenHoldCtl(p, func(string, string) (string, error) { return "", nil }, nil); err == nil {
t.Fatal("a regular file at the socket path must be refused")
}
if b, _ := os.ReadFile(p); string(b) != "keep me" {
t.Fatal("the file at the socket path was changed")
}
}
// main wires every piece once (the seam-built-but-never-wired trap): the clear door into the operator actions, the
// R-1340 text into the hold notification, the R-1341 gate into the OOM loop, and the box command.
func TestD64_MainWiresTheNewPieces(t *testing.T) {
f, err := parser.ParseFile(token.NewFileSet(), "main.go", nil, 0)
if err != nil {
t.Fatal(err)
}
found := map[string]int{}
var opArgs []ast.Expr
ast.Inspect(f, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
switch fn := call.Fun.(type) {
case *ast.SelectorExpr:
if x, ok := fn.X.(*ast.Ident); ok {
found[x.Name+"."+fn.Sel.Name]++
}
case *ast.Ident:
found[fn.Name]++
if fn.Name == "operatorActionHandlers" {
opArgs = call.Args
}
}
return true
})
for _, want := range []string{"heldStoppedOperatorText", "heldHoldFromNotify", "oomLog.fresh", "clearRestoreHoldCLI", "listenHoldCtl", "clearHoldNow"} {
if found[want] != 1 {
t.Errorf("main.go calls %s %d time(s), want exactly 1", want, found[want])
}
}
if len(opArgs) < 3 {
t.Fatal("operatorActionHandlers call not found")
}
if id, ok := opArgs[2].(*ast.Ident); !ok || id.Name != "holdDeps" {
t.Error("main must hand holdDeps (the running controller's clear door) to operatorActionHandlers")
}
}
func timeAt(sec int) time.Time { return time.Date(2026, 10, 29, 12, 0, sec, 0, time.UTC) }
// No controller runs: the box command's own clear drops a waiting put-back from the app-stop marker too (the same
// door), so the next start with the drive bound pours nothing back.
// RED-PROOF (2026-10-29): with the fallback clearing through settings.ClearRestoreHold alone (its first shape in
// 5b3b541), this fails on „a later controller start poured the old copy back".
func TestR1343_TheFallbackDropsAWaitingPutBackToo(t *testing.T) {
r := newD64Rig(t)
undo := filepath.Join(r.dir, "drive", "felhom-data", "backups", "restore-undo", "immich", "20261029T080000Z")
if err := os.MkdirAll(undo, 0o700); err != nil {
t.Fatal(err)
}
if err := r.guard.Begin("unit-restore:immich", backup.ReasonUnitRestore, []string{"immich"}); err != nil {
t.Fatal(err)
}
if err := r.guard.SetUndo("immich", undo); err != nil {
t.Fatal(err)
}
r.hold(t, settings.RestoreHold{Stack: "immich", Reason: settings.HoldReasonPutBackPending, SafetyDump: undo})
var out, errb bytes.Buffer
if code := clearRestoreHoldCLI(r.dir, "immich", &out, &errb); code != 0 {
t.Fatalf("exit %d, stderr %q", code, errb.String())
}
if !strings.Contains(out.String(), "waiting put-back is dropped") {
t.Errorf("output must say the put-back is dropped:\n%s", out.String())
}
sett2, err := settings.Load(filepath.Join(r.dir, "settings.json"), log.New(io.Discard, "", 0))
if err != nil {
t.Fatal(err)
}
poured := &recordingPutBacker{}
g2 := backup.NewAppStopGuard(filepath.Join(r.dir, "appstop-state.json"), r.logger)
g2.SetHolds(sett2)
g2.SetPutBacker(poured)
g2.SetStarter(nopStarter{})
g2.SetUndoReachable(func(string, string) bool { return true })
g2.Recover()
if len(poured.stacks) != 0 {
t.Fatalf("a later controller start poured the old copy back over the app: %v", poured.stacks)
}
if _, err := os.Stat(undo); err != nil {
t.Fatal("the copy must stay on disk")
}
}
@@ -0,0 +1,34 @@
//go:build linux
package main
import (
"fmt"
"net"
"os"
"syscall"
)
// holdCtlPeerAllowed (R-1343) admits a peer on the hold socket only when its uid equals this process's (SO_PEERCRED).
// The socket's 0600 mode in a 0700 directory already says so; this is the second lock on the same door.
func holdCtlPeerAllowed(conn net.Conn) (bool, string) {
uc, ok := conn.(*net.UnixConn)
if !ok {
return false, "not a unix-socket peer"
}
raw, err := uc.SyscallConn()
if err != nil {
return false, "the peer's identity could not be read"
}
var cred *syscall.Ucred
var cerr error
if err := raw.Control(func(fd uintptr) {
cred, cerr = syscall.GetsockoptUcred(int(fd), syscall.SOL_SOCKET, syscall.SO_PEERCRED)
}); err != nil || cerr != nil || cred == nil {
return false, "the peer's identity could not be read"
}
if int(cred.Uid) != os.Geteuid() {
return false, fmt.Sprintf("the peer's uid %d is not this controller's", cred.Uid)
}
return true, ""
}
@@ -0,0 +1,14 @@
//go:build !linux
package main
import "net"
// holdCtlPeerAllowed: the peer check needs SO_PEERCRED (Linux, where the controller runs). Elsewhere the socket's
// 0600 mode in a 0700 directory is the only lock.
func holdCtlPeerAllowed(conn net.Conn) (bool, string) {
if _, ok := conn.(*net.UnixConn); !ok {
return false, "not a unix-socket peer"
}
return true, ""
}
+165
View File
@@ -0,0 +1,165 @@
package main
import (
"fmt"
"strings"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// ── R-1340: the operator's „HELD STOPPED" mail names the hold's REAL cause ────────────────────────────────────────
//
// Until 2026-10-29 the restore-hold mail told one story for every restore_mixed hold — R-893 option C's „its off-site
// database restore failed. The database was rolled back …, but its files and named volumes are the snapshot's" —
// written when option C was the only mixed hold. Measured on Tester 1 (D63's H5 proof): Radicale, which has no
// database, whose put-back stopped at „the app could not be stopped" with nothing moved, got exactly that text.
// Support read a wrong account of the box.
//
// The text is now built from the HOLD RECORD the writer persisted just before the mail (its reason and the R-1340
// fields: which restore, which branch, which step failed, whether the failure was a database replay, where the copy
// is kept). It never says „database" unless the restore's failure WAS a database replay, and never „off-site" for a
// restore from the app's own backup. One sentence per reason; pinned by TestR1340_* (holdmail_test.go).
// heldStoppedOperatorText is the operator mail for a held app: the message, its leg (the hub reads the leg, so the
// two existing ones are kept: restore-hold for a restore hold, restore-hold-mixed for a mixed one) and a short reason.
func heldStoppedOperatorText(h settings.RestoreHold) (msg, leg, reason string) {
stack := h.Stack
const tail = " The app will not start from any path until the hold is cleared (the hub's Clear hold button, or on the box: " + holdClearCommandHint + ")."
kind := restoreKindPhrase(h.RestoreKind)
switch h.Reason {
case settings.HoldReasonUnhealthyStop:
what := "a crash loop"
if h.UnhealthyKind == "oom_storm" {
what = "an out-of-memory storm"
}
trip := ""
if h.Trip >= 2 {
trip = fmt.Sprintf(" This is stop %d within 24 hours.", h.Trip)
}
msg = fmt.Sprintf("App %q is HELD STOPPED: the box stopped it for %s at %s.%s No restore or update was involved and no data was touched. "+
"The household's Start gives it one more try; clearing the hold lifts it too.", stack, what, h.At, trip)
return msg, "unhealthy-stop", "stopped for " + what
case settings.HoldReasonUpdateFailed:
undo := ""
switch h.UndoState {
case "untouched":
undo = " The automatic undo failed; the data was not touched."
case "half":
undo = " The automatic undo failed half-way; the data may be partly the new version's."
case "not_started":
undo = " The automatic undo did not start."
}
copyLine := " Restoring it from its backup starts it."
if h.NoWholeCopy {
copyLine = " No copy on this box brings it back whole — it needs support."
} else if h.CopyDate != "" {
copyLine = fmt.Sprintf(" Restoring it from its backup (tier %d, %s) starts it.", h.CopyTier, h.CopyDate)
}
msg = fmt.Sprintf("App %q is HELD STOPPED: its update at %s did not come up healthy.%s%s", stack, h.At, undo, copyLine) + tail
return msg, "update-hold", "update failed"
case settings.HoldReasonPutBackPending:
msg = fmt.Sprintf("App %q is HELD STOPPED: a restore was cut after its copy of the live data was taken, and that copy (%s) is on a drive that is not bound. "+
"The copy is poured back and the app started when the drive is bound. Clearing the hold DROPS that put-back; the copy stays on disk.", stack, orNone(h.SafetyDump))
return msg, "putback-pending", "put-back waits for its drive"
case settings.HoldReasonRestoreMixed:
return mixedHoldText(h, kind, tail)
default: // HoldReasonRestoreFailed: a database replay failed AND its rollback failed (holdAppAfterFailedRollback)
what := "its " + kind + " failed"
if h.DBReplay || h.RestoreKind == "" { // every restore hold before R-1340 was written by the database branch
what = "its " + kind + " failed at the database replay AND the rollback to the pre-restore database copy also failed — the database may be half-written"
}
msg = fmt.Sprintf("App %q is HELD STOPPED: %s.", stack, what) + tail +
fmt.Sprintf(" Replay error: %s. Rollback error: %s", orNone(h.ReplayError), orNone(h.RollbackErr))
reason = h.RollbackErr
if reason == "" {
reason = "rollback failed"
}
return msg, "restore-hold", reason
}
}
// heldRunKind names the run in the mail's subject. A hold that does not say which restore wrote it is an off-site one, the
// pre-R-1340 rule, EXCEPT an interrupted put-back: its record names no kind, and its restore may be either, so „restore" (R-1348,
// seen live 2026-11-03: a local restore's mail read „offsite-reconstitute"). Pinned by TestR1348_HeldRunKind.
func heldRunKind(h settings.RestoreHold) string {
if h.RestoreKind == settings.RestoreKindLocal || h.MixedCause == settings.MixedCauseInterruptedPutBack {
return "restore"
}
return "offsite-reconstitute"
}
func mixedHoldText(h settings.RestoreHold, kind, tail string) (msg, leg, reason string) {
stack := h.Stack
leg = "restore-hold-mixed"
failedAt := ""
if h.DBReplay {
failedAt = " at the database replay"
}
copyLine := " There is no copy of the live volumes."
if h.SafetyDump != "" {
copyLine = " The copy of the live data is kept at " + h.SafetyDump + "."
}
switch h.MixedCause {
case settings.MixedCauseDefinitionWrite:
msg = fmt.Sprintf("App %q is HELD STOPPED for support: its %s failed while writing the backup's app definition, before any data was touched, "+
"and %s. The app's data is the live data; the definition on disk may be the backup's.", stack, kind, orNone(h.FailedStep)) + tail +
" Error: " + orNone(h.ReplayError)
return msg, leg, "definition write failed; writing the live definition back failed too"
case settings.MixedCausePutBack:
msg = fmt.Sprintf("App %q is HELD STOPPED for support: its %s failed%s, and putting the live data back FAILED — %s. "+
"The app may be on a mix of the backup's and the live data.%s", stack, kind, failedAt, orNone(h.FailedStep), copyLine) + tail +
" Restore error: " + orNone(h.ReplayError)
return msg, leg, "put-back failed: " + firstClause(h.FailedStep)
case settings.MixedCauseInterruptedPutBack:
msg = fmt.Sprintf("App %q is HELD STOPPED for support: a restore cut by a stop could not be put back at the next start — %s. "+
"The app may be on a mix of the backup's and the live data.%s", stack, orNone(h.FailedStep), copyLine) + tail
return msg, leg, "interrupted restore: put-back failed"
}
// A mixed hold written before R-1340 carries no cause: name only what the record says, and assume nothing.
msg = fmt.Sprintf("App %q is HELD STOPPED for support: a restore stopped half-way (set %s); the app may be on a mix of the backup's and the live data. "+
"The hold record says: %s.", stack, h.At, orNone(h.ReplayError)) + tail
return msg, leg, "restore stopped half-way"
}
func restoreKindPhrase(k string) string {
switch k {
case settings.RestoreKindLocal:
return "restore from its own backup"
case settings.RestoreKindOffsite:
return "off-site restore"
}
return "restore"
}
func orNone(s string) string {
if strings.TrimSpace(s) == "" {
return "(none recorded)"
}
return s
}
// firstClause is a step's text up to its error (the short reason field).
func firstClause(s string) string {
if i := strings.Index(s, ":"); i > 0 {
return s[:i]
}
return s
}
// heldHoldFromNotify is the hold the notification describes: the record the writer persisted just before it, or —
// when that write failed — one rebuilt from the errors handed over (the pre-R-1340 rule: no rollback error = mixed).
func heldHoldFromNotify(get func(string) (settings.RestoreHold, bool), stack string, replayErr, rollbackErr error) settings.RestoreHold {
if get != nil {
if h, ok := get(stack); ok && (h.Reason == settings.HoldReasonRestoreFailed || h.Reason == settings.HoldReasonRestoreMixed) {
return h
}
}
h := settings.RestoreHold{Stack: stack, Reason: settings.HoldReasonRestoreMixed}
if rollbackErr != nil {
h.Reason, h.RollbackErr = settings.HoldReasonRestoreFailed, rollbackErr.Error()
}
if replayErr != nil {
h.ReplayError = replayErr.Error()
}
return h
}
+140
View File
@@ -0,0 +1,140 @@
package main
import (
"errors"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-1340: the operator's „HELD STOPPED" mail names the hold's real cause — one test per reason. Each asserts the words
// that MUST be there and the words that must NOT (the option-C database story was the bug).
//
// RED-PROOF (2026-10-29): with mixedHoldText's body replaced by the pre-R-1340 sentence („its off-site database restore
// failed. The database was rolled back to the pre-restore copy and the live definition written back, but its files
// and named volumes are the snapshot's"), TestR1340_PutBackWithoutADatabase, _DefinitionWrite, _LocalRestore and
// _InterruptedPutBack fail on „database" / „off-site" being present and their own step missing.
const optionCStory = "files and named volumes are the snapshot's"
func mustHave(t *testing.T, msg string, want ...string) {
t.Helper()
for _, w := range want {
if !strings.Contains(msg, w) {
t.Errorf("the mail must say %q:\n%s", w, msg)
}
}
}
func mustNotHave(t *testing.T, msg string, bad ...string) {
t.Helper()
for _, b := range bad {
if strings.Contains(strings.ToLower(msg), strings.ToLower(b)) {
t.Errorf("the mail must NOT say %q:\n%s", b, msg)
}
}
}
func TestR1340_UnhealthyStop(t *testing.T) {
msg, leg, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "romm", At: "2026-10-29T08:00:00Z", Reason: settings.HoldReasonUnhealthyStop, UnhealthyKind: "crash_loop"})
mustHave(t, msg, `"romm"`, "crash loop", "no data was touched", "Start gives it one more try")
mustNotHave(t, msg, "database", "restore failed", optionCStory)
if leg != "unhealthy-stop" {
t.Errorf("leg %q", leg)
}
oom, _, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "romm", Reason: settings.HoldReasonUnhealthyStop, UnhealthyKind: "oom_storm", Trip: 2})
mustHave(t, oom, "out-of-memory storm", "stop 2 within 24 hours")
}
func TestR1340_UpdateFailed(t *testing.T) {
msg, leg, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "kimai", At: "2026-10-29T03:10:00Z", Reason: settings.HoldReasonUpdateFailed,
UndoState: "half", CopyTier: 1, CopyDate: "2026-10-29T02:00:00Z"})
mustHave(t, msg, "update at 2026-10-29T03:10:00Z did not come up healthy", "undo failed half-way", "tier 1, 2026-10-29T02:00:00Z", "--clear-restore-hold")
mustNotHave(t, msg, "database", optionCStory)
if leg != "update-hold" {
t.Errorf("leg %q", leg)
}
}
func TestR1340_RestoreFailedWithADatabase(t *testing.T) {
msg, leg, reason := heldStoppedOperatorText(settings.RestoreHold{Stack: "paperless", Reason: settings.HoldReasonRestoreFailed, RestoreKind: settings.RestoreKindOffsite,
DBReplay: true, ReplayError: "ERROR: relation exists", RollbackErr: "psql: connection refused"})
mustHave(t, msg, "off-site restore failed at the database replay", "rollback to the pre-restore database copy also failed", "ERROR: relation exists", "psql: connection refused")
mustNotHave(t, msg, optionCStory)
if leg != "restore-hold" || reason != "psql: connection refused" {
t.Errorf("leg %q reason %q", leg, reason)
}
}
// The measured case: Radicale has no database; the volume could not be removed and the put-back's stop failed.
func TestR1340_PutBackWithoutADatabase(t *testing.T) {
msg, leg, reason := heldStoppedOperatorText(settings.RestoreHold{Stack: "radicale", Reason: settings.HoldReasonRestoreMixed, RestoreKind: settings.RestoreKindOffsite,
MixedCause: settings.MixedCausePutBack, FailedStep: "stopping the app before the put-back failed, so nothing was put back: compose down: timeout",
ReplayError: "volume rm -f radicale_data: volume is in use", SafetyDump: "/mnt/hdd/felhom-data/backups/restore-undo/radicale/20261029T081800Z"})
mustHave(t, msg, `"radicale"`, "off-site restore failed, and putting the live data back FAILED", "stopping the app before the put-back failed",
"compose down: timeout", "kept at /mnt/hdd/felhom-data/backups/restore-undo/radicale/20261029T081800Z", "volume is in use")
mustNotHave(t, msg, "database", "rolled back", optionCStory)
if leg != "restore-hold-mixed" || reason != "put-back failed: stopping the app before the put-back failed, so nothing was put back" {
t.Errorf("leg %q reason %q", leg, reason)
}
}
func TestR1340_PutBackAfterADatabaseReplay(t *testing.T) {
msg, _, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "immich", Reason: settings.HoldReasonRestoreMixed, RestoreKind: settings.RestoreKindOffsite,
MixedCause: settings.MixedCausePutBack, DBReplay: true, FailedStep: "putting the live volumes back failed: tar xf failed"})
mustHave(t, msg, "failed at the database replay", "putting the live volumes back failed: tar xf failed", "There is no copy of the live volumes")
}
func TestR1340_DefinitionWrite(t *testing.T) {
msg, _, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "gokapi", Reason: settings.HoldReasonRestoreMixed, RestoreKind: settings.RestoreKindOffsite,
MixedCause: settings.MixedCauseDefinitionWrite, FailedStep: "writing the live app definition back failed: app.yaml write failed", ReplayError: "app.yaml write failed"})
mustHave(t, msg, "while writing the backup's app definition, before any data was touched", "writing the live app definition back failed", "data is the live data")
mustNotHave(t, msg, "database", optionCStory)
}
func TestR1340_LocalRestore(t *testing.T) {
msg, _, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "app", Reason: settings.HoldReasonRestoreMixed, RestoreKind: settings.RestoreKindLocal,
MixedCause: settings.MixedCausePutBack, FailedStep: "putting the live volumes back failed: tar xf failed", SafetyDump: "/srv/undo/app/1"})
mustHave(t, msg, "restore from its own backup failed", "putting the live volumes back failed")
mustNotHave(t, msg, "off-site", "database", optionCStory)
}
func TestR1340_PutBackPending(t *testing.T) {
msg, leg, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "immich", Reason: settings.HoldReasonPutBackPending, SafetyDump: "/mnt/hdd/undo/immich/1"})
mustHave(t, msg, "copy (/mnt/hdd/undo/immich/1) is on a drive that is not bound", "poured back and the app started when the drive is bound", "DROPS that put-back")
mustNotHave(t, msg, "database", optionCStory)
if leg != "putback-pending" {
t.Errorf("leg %q", leg)
}
}
func TestR1340_InterruptedPutBack(t *testing.T) {
msg, _, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "immich", Reason: settings.HoldReasonRestoreMixed, MixedCause: settings.MixedCauseInterruptedPutBack,
FailedStep: "putting the interrupted restore's copy back failed: compose down: timeout", SafetyDump: "/mnt/hdd/undo/immich/1"})
mustHave(t, msg, "cut by a stop could not be put back", "compose down: timeout", "kept at /mnt/hdd/undo/immich/1")
mustNotHave(t, msg, "database", "off-site", optionCStory)
}
// A mixed hold written before R-1340 names only what its record says.
func TestR1340_LegacyMixedHoldAssumesNothing(t *testing.T) {
msg, _, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "x", At: "2026-10-01T00:00:00Z", Reason: settings.HoldReasonRestoreMixed, ReplayError: "boom"})
mustHave(t, msg, "stopped half-way", "The hold record says: boom")
mustNotHave(t, msg, "database", optionCStory)
}
// The notification reads the hold the writer persisted; when that write failed, it rebuilds one from the errors.
func TestR1340_NotifyReadsTheHoldRecord(t *testing.T) {
rec := settings.RestoreHold{Stack: "radicale", Reason: settings.HoldReasonRestoreMixed, MixedCause: settings.MixedCausePutBack, FailedStep: "x"}
get := func(s string) (settings.RestoreHold, bool) { return rec, s == "radicale" }
if h := heldHoldFromNotify(get, "radicale", errors.New("e"), nil); h.MixedCause != settings.MixedCausePutBack {
t.Fatalf("the persisted record was not used: %+v", h)
}
h := heldHoldFromNotify(get, "other", errors.New("replay"), errors.New("rollback"))
if h.Reason != settings.HoldReasonRestoreFailed || h.RollbackErr != "rollback" || h.ReplayError != "replay" {
t.Fatalf("fallback hold %+v", h)
}
if h := heldHoldFromNotify(nil, "other", errors.New("replay"), nil); h.Reason != settings.HoldReasonRestoreMixed {
t.Fatalf("no rollback error = mixed, got %+v", h)
}
}
@@ -0,0 +1,50 @@
package main
import (
"context"
"errors"
"log"
"net/http"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
)
// hostOOMSource (R-528) is the 30-s dead-app scan's host-side OOM counter: one agent call per scan (5 s cap). An agent
// without the route (404) is not asked again for hostOOMRetryAfter, and said once; any other failure is a nil answer
// for this scan only (Docker's flag still works). Pinned by TestR528_HostOOMSource*.
const hostOOMRetryAfter = 10 * time.Minute
func hostOOMSource(ask func(context.Context) (agentapi.GuestOOMKillsState, error), logger *log.Logger, now func() time.Time) func() map[string]int64 {
var mu sync.Mutex
var until time.Time
said := false
return func() map[string]int64 {
mu.Lock()
defer mu.Unlock()
if now().Before(until) {
return nil
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
st, err := ask(ctx)
if err != nil {
var se *agentapi.StatusError
if errors.As(err, &se) && se.Code == http.StatusNotFound {
until = now().Add(hostOOMRetryAfter)
if !said {
logger.Printf("[INFO] [deadapp] the agent has no OOM counter route (R-528) — Docker's OOM flag only; asked again in %s", hostOOMRetryAfter)
said = true
}
return nil
}
logger.Printf("[WARN] [deadapp] the host OOM counter could not be read: %v — Docker's OOM flag only for this scan", err)
return nil
}
if !st.Available {
return nil
}
return st.Containers
}
}
+88 -51
View File
@@ -34,6 +34,7 @@ import (
"gitea.dooplex.hu/admin/felhom-controller/internal/backupwindow"
"gitea.dooplex.hu/admin/felhom-controller/internal/bootrecon"
"gitea.dooplex.hu/admin/felhom-controller/internal/bootstrap"
"gitea.dooplex.hu/admin/felhom-controller/internal/busymark"
"gitea.dooplex.hu/admin/felhom-controller/internal/channelhealth"
cf "gitea.dooplex.hu/admin/felhom-controller/internal/cloudflare"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
@@ -92,8 +93,9 @@ func main() {
abandonStatus := flag.Bool("abandon-status", false, "R-241: print the state of this box's off-site abandonment countdown (set-aside path, due date, days left) and exit. Read-only.")
abandonExtend := flag.Int("abandon-extend", 0, "R-241 (operator): extend a running abandonment countdown by N days from now, then exit. Refuses when no countdown is running.")
abandonStop := flag.Bool("abandon-stop", false, "R-241 (operator): stop a running abandonment countdown, then exit. The set-aside history is kept and nothing is deleted. Refuses when no countdown is running.")
listRestoreHolds := flag.Bool("restore-holds", false, "R-379 (operator): list apps this controller is deliberately holding stopped after a failed database restore whose rollback also failed, then exit. Read-only.")
clearRestoreHold := flag.String("clear-restore-hold", "", "R-379 (operator): clear the restore hold on APP so it can start again, then exit. Clearing does NOT repair the database — check the app's data first; the undo copies are in its unit's db-dumps dir.")
listRestoreHolds := flag.Bool("restore-holds", false, "R-379 (operator): list the apps this controller is holding stopped (every reason, with why), then exit. Read-only.")
clearRestoreHold := flag.String("clear-restore-hold", "", "R-379/D64 (operator): clear the hold on APP (any reason), then exit. Run it inside the guest: "+holdClearCommandHint+
". It asks the RUNNING controller, so no restart is needed; only when no controller runs does it edit settings.json. Clearing repairs nothing and starts nothing — check the app's data first.")
flag.Parse()
if *showVersion {
@@ -172,21 +174,23 @@ func main() {
// starts, exactly like the other CLI subcommands: each loads config + settings, acts, and exits.
// R-379 (operator): the way OUT of a hold. A CLI subcommand rather than an API route, matching
// --abandon-stop above: the controller's HTTP surface is authenticated as the CUSTOMER, and
// clearing a hold is a decision that should follow an operator looking at the database. Reaching
// clearing a hold is a decision that should follow an operator looking at the data. Reaching
// this needs `docker exec` into the guest, which is the operator gate this project already uses.
// R-1343: the clear goes to the RUNNING controller over its local socket (holdclear.go); the list reads
// settings.json, which the running controller saves at every hold change.
if *listRestoreHolds || *clearRestoreHold != "" {
cfg, err := config.LoadPermissive(*configPath)
if err != nil {
fmt.Fprintf(os.Stderr, "restore-hold: loading config: %v\n", err)
os.Exit(1)
}
if *listRestoreHolds {
lg := log.New(os.Stderr, "", 0)
sett, err := settings.Load(cfg.Paths.DataDir+"/settings.json", lg)
if err != nil {
fmt.Fprintf(os.Stderr, "restore-hold: loading settings: %v\n", err)
os.Exit(1)
}
if *listRestoreHolds {
holds := sett.ListRestoreHolds()
if len(holds) == 0 {
fmt.Println("no restore holds in force")
@@ -197,34 +201,9 @@ func main() {
}
os.Exit(0)
}
cleared, err := sett.ClearRestoreHold(*clearRestoreHold)
if err != nil {
fmt.Fprintf(os.Stderr, "restore-hold: clearing %s: %v\n", *clearRestoreHold, err)
os.Exit(1)
}
if !cleared {
// "there was nothing to clear" is not success — a silent 0 here would let an operator
// believe they had unblocked an app whose name they mistyped.
fmt.Fprintf(os.Stderr, "no restore hold in force for %q — nothing was changed\n", *clearRestoreHold)
os.Exit(2)
}
// THE RESTART IS NOT OPTIONAL, and saying so is the difference between a route that works
// and one an operator believes worked. This runs as a SECOND process: it clears the hold in
// settings.json, but the RUNNING controller holds its own in-memory Settings and keeps
// refusing until it reloads. Measured on demo-hp 2026-08-22 — the clear succeeded, the file
// was correct, and the customer's start button still refused until the controller restarted.
//
// There is also a lost-update window while both processes hold the file: if the running
// controller saves settings.json from its own copy before the restart, the clear is undone.
// Restarting promptly closes it. A route that clears through the running controller would
// remove both problems and is the right shape later; it needs an operator tier the
// controller's HTTP surface does not have today (it authenticates as the CUSTOMER, and a
// customer clearing their own hold is precisely what the hold exists to prevent).
fmt.Printf("restore hold cleared for %s in settings.json.\n", *clearRestoreHold)
fmt.Printf(" NOW RESTART THE CONTROLLER, or it will keep refusing to start the app:\n")
fmt.Printf(" systemctl restart felhom-controller-bootstrap.service\n")
fmt.Printf(" Check the app's data first — the undo copies are in its unit's db-dumps dir.\n")
os.Exit(0)
// R-1343: through the RUNNING controller (holdclear.go) — the old second-process edit of settings.json was
// written back by the running controller's next save. Only when no controller answers is the file edited here.
os.Exit(clearRestoreHoldCLI(cfg.Paths.DataDir, *clearRestoreHold, os.Stdout, os.Stderr))
}
if *abandonStatus || *abandonExtend > 0 || *abandonStop {
@@ -485,6 +464,19 @@ func main() {
return stackMgr.DriveLive(hdd)
})
appStopRecovery := appStopGuard.Recover()
// D64 = B / R-1343: the operator's hold clear — ONE door for the hub's clear_hold and the box command, run in THIS
// process so memory and settings.json agree (holdclear.go). The box command reaches it over a unix socket in the
// data dir (0600, nothing on TCP). A socket that cannot be bound is logged; the command then says plainly that no
// running controller answered.
holdDeps := holdClearDeps{clear: appStopGuard.ClearHoldByOperatorThen, forgetUpdateHeld: stackMgr.ForgetUpdateHeld,
keepStopped: stackMgr.RecordStopWantedIfDown, logger: logger}
if holdLn, herr := listenHoldCtl(holdCtlPath(cfg.Paths.DataDir), func(stack, who string) (string, error) {
return clearHoldNow(holdDeps, stack, who)
}, logger); herr != nil {
logger.Printf("[ERROR] [hold] the local hold-clear channel could not be opened: %v — `--clear-restore-hold` will edit settings.json and need a restart", herr)
} else {
defer holdLn.Close()
}
// R-1251: a put-back that waited for its drive (a hold from an earlier start) runs now if that drive is bound.
if started := appStopGuard.ResumeWaitingPutBacks(); len(started) > 0 {
logger.Printf("[WARN] [appstop] %d app(s) whose interrupted restore waited for its drive are put back and started: %v", len(started), started)
@@ -587,6 +579,11 @@ func main() {
logger.Println("[INFO] Healthchecks ping UUIDs configured but no longer used — monitoring is now handled by the Hub")
}
// D65 = A (2026-10-29): the busy file the host agent reads before an operator reboot of the guest (internal/busymark).
// Probes are added where each component is built; the writer starts once the last one exists (after the exporter).
busyMark := &busymark.Writer{Path: filepath.Join(cfg.Paths.DataDir, holdCtlDir, busymark.FileName), Logger: logger}
busyMark.AddProbe("an app update is running", stackMgr.AnyUpdating)
// --- Initialize backup manager (app-data only: DB dumps + Docker-volume tars) ---
var backupMgr *backup.Manager
// R-550: the restore the last stop interrupted, found when the record loads; raised once the notifier exists.
@@ -598,6 +595,9 @@ func main() {
}
if cfg.Backup.Enabled {
backupMgr = backup.NewManager(cfg, sett, logger)
busyMark.AddProbe("a backup, a restore or an off-site run is running", func() bool {
return backupMgr.IsRunning() || backupMgr.RestoreStatus().Running
})
// The stray-copy fix: a second-drive copy container a previous controller process lost (restart, self-update)
// still runs `rsync --delete`. Remove every one BEFORE the scheduler or the web server can start a backup or a
// restore; until that succeeds, restores are refused. Pinned by TestStrayCopy_ClearIsWiredBeforeTheScheduler.
@@ -682,6 +682,7 @@ func main() {
backupMgr.SetVersionCheck(stackMgr.PinNotRunning)
// R-671 (v0.272.0): a restore that lifts an update hold removes the undo copies that hold kept.
backupMgr.SetUndoCopyRemover(stackMgr.RemoveUndoCopies)
backupMgr.SetUpdateHeldForgetter(stackMgr.ForgetUpdateHeld) // R-1347
}
if n := stackMgr.ResumeInterruptedUpdates(ctx); n > 0 {
logger.Printf("[WARN] [update] resumed %d interrupted update(s)", n)
@@ -1035,6 +1036,9 @@ func main() {
logger.Printf("[WARN] [deadapp] agent client init failed (%v) — the boot grace cannot learn of a crash boot; 90 s", err)
} else {
crashProbe = crashboot.AgentProbe(ac.CrashGuard)
// R-528: the 30-s scan also asks the host for the kernel's per-container OOM counter (Docker's own flag can
// stay false inside an LXC guest). TestR528_HostOOMSourceIsWiredAtStartup.
stackMgr.SetHostOOMKills(hostOOMSource(ac.GuestOOMKills, logger, time.Now))
}
}
bootGrace := crashboot.New(startTime, crashProbe, logger)
@@ -1078,6 +1082,7 @@ func main() {
holdReporter.Tick(time.Now())
return nil
})
oomLog := &oomLogGate{} // R-1341
sched.Every("deadapp-check", 30*time.Second, func(ctx context.Context) error {
if bootGrace.Within(time.Now()) {
return nil // still inside the startup settle window (90 s, or ~15 min after a crash boot)
@@ -1093,9 +1098,13 @@ func main() {
if oerr != nil {
logger.Printf("[WARN] [deadapp] OOM scan failed: %v", oerr)
} else {
// R-1341: the line once per NEW kill (oomlog.go); the event call stays per scan — the notifier's cooldown
// keeps it to one, as before.
for _, o := range oomLog.fresh(ooms) {
logger.Printf("[WARN] [deadapp] %s: container %s was OOM-killed (started %s; kills %d, limit %s, peak %s; seen by %s)", o.Stack, o.Container, o.StartedAt, o.Kills, o.MemLimit, o.Peak, o.Source)
}
for _, o := range ooms {
logger.Printf("[WARN] [deadapp] %s: container %s was OOM-killed (started %s; kills %d, limit %s, peak %s)", o.Stack, o.Container, o.StartedAt, o.Kills, o.MemLimit, o.Peak)
notifier.NotifyAppOOM(o.Stack, o.Container, o.StartedAt, o.Kills, o.MemLimit, o.Peak)
notifier.NotifyAppOOMFrom(o.Stack, o.Container, o.StartedAt, o.Kills, o.MemLimit, o.Peak, o.Source)
}
}
// v0.269.0 (`09` §3 decision 28, R-667): a crash loop or an OOM storm is STOPPED by the box.
@@ -1214,7 +1223,7 @@ func main() {
// R-314/R-279/R-177 (`09` §3 decision 185): the operator's closed list of actions, carried in the
// report reply and run in THIS process (opactions_wiring.go). The result rides the next report;
// a finished result fires an out-of-cycle report through the trigger once it exists.
opActions := report.NewOperatorActions(ctx, operatorActionHandlers(backupMgr, sched, func() {
opActions := report.NewOperatorActions(ctx, operatorActionHandlers(backupMgr, sched, holdDeps, func() {
if f, ok := opActionReportFire.Load().(func()); ok && f != nil {
f()
}
@@ -1400,6 +1409,13 @@ func main() {
return nil
})
// R-1256 H8: a restore copy no hold, no pending put-back and no running restore needs goes after 14 days. Every 6
// hours — a copy outliving its age by hours costs disk, never data. TestR1256_H8_TheSweepIsWiredAtStartup.
sched.Every("restore-copy-sweep", 6*time.Hour, func(ctx context.Context) error {
backupMgr.SweepUnusedRestoreCopies(time.Now())
return nil
})
// Tier 2: off-drive copy of each HDD app's recovery unit + userdata (auto-enabled, auto-target).
// Runs after the DB dump so it copies a fresh unit.
backupMgr.SetTier2Notifier(func(stackName, destLabel string, dur time.Duration, err error) {
@@ -1538,24 +1554,20 @@ func main() {
// DELIBERATELY NOT `backup_failed`: customer-enabled by default, Hungarian "A biztonsági
// mentés sikertelen!", and this is a deliberate hold rather than a backup failure. R-171 one
// path over.
backupMgr.SetRestoreHoldNotify(func(stack string, replayErr, rollbackErr error) {
d := notify.BackupRunFailuresDetails{RunKind: "offsite-reconstitute", Failed: 1, Attempted: 1}
if rollbackErr == nil { // R-893: the rollback WORKED, but files/volumes/version had already moved
d.Apps = append(d.Apps, notify.RunFailureDetail{App: stack, Leg: "restore-hold-mixed", Reason: "replay failed after files/volumes/version moved"})
msg := fmt.Sprintf("App %q is HELD STOPPED for support: its off-site database restore failed. The database was rolled back to the pre-restore copy and the live definition written back, "+
"but its files and named volumes are the snapshot's — a mixed state (R-893). The app will not start from any path until the hold is cleared. Replay error: %v", stack, replayErr)
// R-1340: the text is built from the HOLD RECORD the writer persisted just before this call (holdmail.go) — its
// reason, which restore, which branch, which put-back step failed, whether it was a database replay — never the
// option-C database story for every cause.
holdNotify := func(stack string, replayErr, rollbackErr error) {
h := heldHoldFromNotify(sett.GetRestoreHold, stack, replayErr, rollbackErr)
msg, leg, reason := heldStoppedOperatorText(h)
d := notify.BackupRunFailuresDetails{RunKind: heldRunKind(h), Failed: 1, Attempted: 1}
d.Apps = append(d.Apps, notify.RunFailureDetail{App: stack, Leg: leg, Reason: reason})
notifier.NotifyBackupRunFailures(msg, d)
return
}
reason := "rollback failed"
if rollbackErr != nil {
reason = rollbackErr.Error()
}
d.Apps = append(d.Apps, notify.RunFailureDetail{App: stack, Leg: "restore-hold", Reason: reason})
msg := fmt.Sprintf("App %q is HELD STOPPED: its off-site database restore failed AND the rollback to the customer's own pre-restore copy also failed. "+
"The app will not start from any path until the hold is cleared. Replay error: %v. Rollback error: %v", stack, replayErr, rollbackErr)
notifier.NotifyBackupRunFailures(msg, d)
})
backupMgr.SetRestoreHoldNotify(holdNotify)
// R-1348: the hold an interrupted put-back wrote at START-UP (Recover, before any notifier existed) or on a resume.
// The same sentence (R-1340, MixedCauseInterruptedPutBack), the same mail.
appStopGuard.SetHeldNotify(func(stack string) { holdNotify(stack, nil, nil) })
// R-893 option A (D20 = A): a failed off-site restore that was PUT BACK — the live volumes and definition poured
// back and the app started. The same operator-tier event as the hold above (no new event type), its own leg.
// D47 follow-up: a unit of an earlier install was set aside whole — the operator sees it (its own leg).
@@ -1660,6 +1672,7 @@ func main() {
if quiesceLoop != nil {
quiesceLoop.SetCatchUpFn(catchUp.Running)
}
busyMark.AddProbe("the night backup catch-up is running", catchUp.Running)
catchUp.Evaluate(ctx, "controller start")
resumeTick := time.NewTicker(time.Minute)
go nightchain.ResumeWatch(ctx, resumeTick.C,
@@ -2049,6 +2062,10 @@ func main() {
prevCtl = st.RecordedPrevious(Version)
}
_, _ = stackMgr.RetainControllerImages(stacks.ControllerImageRecord{Repo: cfg.SelfUpdate.Image, Running: Version, Previous: prevCtl})
// decision 285 (2026-11-09): the pinned infrastructure images (traefik, cloudflared, FileBrowser, Samba) — keep
// the pin and the one before it. A pin moves only with a controller release, so once per start is enough; the
// infra stacks were reconciled long before these 3 minutes. Retried every 2 minutes while an app pulls (R-863).
stackMgr.RunInfraImageRetention(ctx, infra.Images(), 2*time.Minute, 30)
}()
// --- Initialize API router ---
@@ -2118,6 +2135,8 @@ func main() {
// this the export path would be the uncovered sibling of two covered ones, which is how a reader
// concludes the whole class is handled (§2.2).
appExporter.SetStopGuard(exportStopGuard{g: appStopGuard})
busyMark.AddProbe("an app export or import is running", appExporter.IsRunning)
go busyMark.Run(ctx)
// R-1279 (D48 for the import): the import downloads first, with the restores' helper and sentences.
if backupMgr != nil {
appExporter.SetImagePreparer(backupMgr.PrepareRestoreImages)
@@ -2126,6 +2145,7 @@ func main() {
// --- Initialize web server ---
webServer := web.NewServer(cfg, stackMgr, cpuCollector, backupMgr, sched, sett, alertMgr, notifier, updater, logger, Version)
go webServer.WatchOperatorReboot(ctx) // D65: „Felhom support restarted your box at …" after an operator reboot
// v0.316.0: which BUILD this is, not just which version it calls itself. Every shell's stylesheet
// URL is derived from it, so two builds of one version can never share a cache entry. Init-only,
// before the HTTP server starts.
@@ -3181,9 +3201,23 @@ func (a *stackAdapter) ListDeployedStacks() []backup.StackSummary {
}
func (a *stackAdapter) StopStack(name string) error {
if err := faultPoint("stop-" + name); err != nil { // D63: a no-op outside a -tags faultinject test build
return err
}
return a.mgr.StopStack(name)
}
// StackRemoved answers backup's optional StackRemovalReporter (R-1342): the app is known and no longer installed — a
// stable-start watch then ends quietly instead of alarming „did not start". An app this box does not know at all is
// not „removed" (no answer → the watch goes on as before).
func (a *stackAdapter) StackRemoved(name string) bool {
s, ok := a.mgr.GetStack(name)
return ok && !s.Deployed
}
// The watch reaches StackRemoved through a type assertion; this line makes the adapter's answer a compile-time fact.
var _ backup.StackRemovalReporter = (*stackAdapter)(nil)
// IsDeploying answers backup's optional deployingReporter (R-634): the volume leg asks it again
// immediately before it stops an app, because its list is taken once at the start of the run.
func (a *stackAdapter) IsDeploying(name string) bool {
@@ -3425,6 +3459,9 @@ func (a *stackAdapter) RecreateStackDefinitionFromUnit(name, composeSrcDir strin
return fmt.Errorf("restoring %s from unit: %w", fname, err)
}
}
if err := faultPoint("defwrite-after-compose"); err != nil { // D63: a no-op outside a -tags faultinject test build
return err
}
// R-1057 (v0.314.0): the gate as the backup recorded it — the unit's app.yaml, read BEFORE the write.
gate := stacks.ReadGateAtBackup(filepath.Join(composeSrcDir, "app.yaml"))
if err := a.mgr.PersistUnitRedeployConfigGate(name, fullEnv, gate); err != nil {
+43
View File
@@ -0,0 +1,43 @@
package main
import (
"strconv"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
)
// ── R-1341: the OOM line is logged once per NEW kill ──────────────────────────────────────────────────────────────
//
// ScanOOMKilled reports a container for as long as it lives after a kill (Docker's OOMKilled flag is sticky, and the
// kernel's kill counter never goes down), so the 30-s dead-app scan logged „[WARN] [deadapp] … was OOM-killed" about
// 2 880 times a day per container — measured 2026-10-29 on Tester 1 — while the notifier's cooldown kept the event to
// one. The line now goes out when the kill is NEW: the key is (container, its start time, the kill count), and a line
// is logged only when a container's key changes — a second scan of the same kill logs nothing, a new kill (the count
// went up) or a restarted container (a new start time) logs again. The event call is unchanged (every scan, the
// notifier decides). Pinned by TestR1341_*.
// oomLogGate remembers the last logged key per container. Not safe for concurrent use: the dead-app job runs serially.
type oomLogGate struct {
last map[string]string
}
func oomLogKey(o stacks.OOMContainer) string {
return o.StartedAt + "|" + strconv.FormatInt(o.Kills, 10)
}
// fresh returns the containers whose kill is new since the last scan, and remembers this scan's keys. A container
// absent from this scan is forgotten (it was removed or restarted clean), which also bounds the memory.
func (g *oomLogGate) fresh(ooms []stacks.OOMContainer) []stacks.OOMContainer {
next := make(map[string]string, len(ooms))
var out []stacks.OOMContainer
for _, o := range ooms {
k := oomLogKey(o)
next[o.Container] = k
if prev, ok := g.last[o.Container]; ok && prev == k {
continue
}
out = append(out, o)
}
g.last = next
return out
}
+64
View File
@@ -0,0 +1,64 @@
package main
import (
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
)
// R-1341: the „[WARN] [deadapp] … was OOM-killed" line goes out once per NEW kill, not every 30-s scan.
//
// RED-PROOF (2026-10-29): with oomLogGate.fresh returning its input unchanged (the pre-R-1341 loop logged every
// container every scan), TestR1341_ASecondScanOfTheSameKillLogsNothing fails on „logged 1 line(s), want 0".
func oom(container, started string, kills int64) stacks.OOMContainer {
return stacks.OOMContainer{Stack: "gokapi", Container: container, StartedAt: started, Kills: kills}
}
func TestR1341_ASecondScanOfTheSameKillLogsNothing(t *testing.T) {
g := &oomLogGate{}
scan := []stacks.OOMContainer{oom("gokapi", "2026-10-29T09:02:00Z", 1)}
if got := g.fresh(scan); len(got) != 1 {
t.Fatalf("the first sighting must log, logged %d line(s)", len(got))
}
for i := 0; i < 5; i++ {
if got := g.fresh(scan); len(got) != 0 {
t.Fatalf("scan %d of the same kill logged %d line(s), want 0", i+2, len(got))
}
}
}
func TestR1341_ANewKillLogsAgain(t *testing.T) {
g := &oomLogGate{}
g.fresh([]stacks.OOMContainer{oom("gokapi", "2026-10-29T09:02:00Z", 1)})
got := g.fresh([]stacks.OOMContainer{oom("gokapi", "2026-10-29T09:02:00Z", 2)})
if len(got) != 1 || got[0].Kills != 2 {
t.Fatalf("a new kill (count 1→2) must log once, got %+v", got)
}
if got := g.fresh([]stacks.OOMContainer{oom("gokapi", "2026-10-29T09:02:00Z", 2)}); len(got) != 0 {
t.Fatalf("the same new kill must not log twice, got %+v", got)
}
}
func TestR1341_ARestartedContainerLogsAgain(t *testing.T) {
g := &oomLogGate{}
g.fresh([]stacks.OOMContainer{oom("gokapi", "2026-10-29T09:02:00Z", 1)})
if got := g.fresh([]stacks.OOMContainer{oom("gokapi", "2026-10-29T09:30:00Z", 1)}); len(got) != 1 {
t.Fatalf("a restarted container's kill must log, got %+v", got)
}
}
// Two containers are judged apart; one that leaves the scan is forgotten, so its return logs again.
func TestR1341_ContainersAreJudgedApart(t *testing.T) {
g := &oomLogGate{}
a, b := oom("a", "t1", 1), oom("b", "t1", 1)
if got := g.fresh([]stacks.OOMContainer{a, b}); len(got) != 2 {
t.Fatalf("first scan: %+v", got)
}
if got := g.fresh([]stacks.OOMContainer{a}); len(got) != 0 {
t.Fatalf("a seen again must not log: %+v", got)
}
if got := g.fresh([]stacks.OOMContainer{a, b}); len(got) != 1 || got[0].Container != "b" {
t.Fatalf("b came back after leaving the scan: %+v", got)
}
}
+10 -2
View File
@@ -17,18 +17,26 @@ import (
// executor is built earlier than the trigger, so it reads it through this holder.
var opActionReportFire atomic.Value
// operatorActionHandlers wires the four operator actions (R-314/R-279/R-177, `09` §3 decision 185)
// operatorActionHandlers wires the five operator actions (R-314/R-279/R-177, `09` §3 decision 185)
// to the running process. ONE wiring site, called from main and from the wiring test, so the test
// exercises the doors production uses (the seam-built-but-never-wired trap).
//
// offsite_backup_now makes the SAME four checks as the household's „back up now"
// (offboxRunHandler) and then runs the same call, so an operator press can never do what the
// household's own press cannot.
func operatorActionHandlers(backupMgr *backup.Manager, sched *scheduler.Scheduler, resultReady func(), logger *log.Logger) report.OperatorActionHandlers {
//
// clear_hold (D64 = B) goes through clearHoldNow with hold — the same door the box command's socket uses
// (holdclear.go), so the hub's button and the box command can never differ. hold.clear nil = not available.
func operatorActionHandlers(backupMgr *backup.Manager, sched *scheduler.Scheduler, hold holdClearDeps, resultReady func(), logger *log.Logger) report.OperatorActionHandlers {
h := report.OperatorActionHandlers{ResultReady: resultReady}
if sched != nil {
h.RunJob = sched.RunNow
}
if hold.clear != nil {
h.ClearHold = func(stack string, actionID int64) (string, error) {
return clearHoldNow(hold, stack, fmt.Sprintf("the operator via the hub, action #%d", actionID))
}
}
if backupMgr == nil {
return h
}
@@ -57,7 +57,7 @@ func TestD1_AbandonStopActsOnTheRunningManagerAndDisk(t *testing.T) {
t.Fatal("fixture: a countdown must be running")
}
o := report.NewOperatorActions(context.Background(), operatorActionHandlers(mgr, nil, nil, logger), logger)
o := report.NewOperatorActions(context.Background(), operatorActionHandlers(mgr, nil, holdClearDeps{}, nil, logger), logger)
o.Reconcile([]report.OperatorAction{{ID: 1, Action: report.OpAbandonStop}})
if mgr.AbandonStatus().Active {
@@ -97,7 +97,7 @@ func TestD1_OffsiteBackupNowRefusesWithoutATarget(t *testing.T) {
t.Fatal("no restic may run")
return nil, nil
})
o := report.NewOperatorActions(context.Background(), operatorActionHandlers(mgr, nil, nil, logger), logger)
o := report.NewOperatorActions(context.Background(), operatorActionHandlers(mgr, nil, holdClearDeps{}, nil, logger), logger)
o.Reconcile([]report.OperatorAction{{ID: 9, Action: report.OpOffsiteBackupNow}})
if r := o.Results(); len(r) != 1 || r[0].Outcome != report.OutcomeRefused {
t.Fatalf("result = %+v", r)
@@ -115,7 +115,7 @@ func TestD1_RunJobUsesTheScheduler(t *testing.T) {
defer cancel()
sched.Start(ctx)
defer sched.Stop()
o := report.NewOperatorActions(ctx, operatorActionHandlers(nil, sched, nil, logger), logger)
o := report.NewOperatorActions(ctx, operatorActionHandlers(nil, sched, holdClearDeps{}, nil, logger), logger)
o.Reconcile([]report.OperatorAction{{ID: 1, Action: report.OpRunJob, Arg: "fill-watch"}, {ID: 2, Action: report.OpRunJob, Arg: "offsite-proof"}})
select {
case od := <-ran:
@@ -0,0 +1,13 @@
package main
import "testing"
// TestR1256_H8_TheSweepIsWiredAtStartup — R-1256 H8: an unused restore copy goes only through
// SweepUnusedRestoreCopies, so main.go must CALL it (the seam-built-but-never-wired class).
// RED-PROOF: delete the `restore-copy-sweep` job — this fails.
func TestR1256_H8_TheSweepIsWiredAtStartup(t *testing.T) {
lines, _, _ := slice4CallLines(t)
if len(lines["SweepUnusedRestoreCopies"]) == 0 {
t.Fatal("SweepUnusedRestoreCopies is never called — an unused restore copy would stay on the drive for ever")
}
}
@@ -20,7 +20,9 @@ func TestR1286_EveryHoldReasonSaysWhatItIs(t *testing.T) {
}{
{settings.RestoreHold{Stack: "kimai", At: "2026-10-14T09:06:31Z", Reason: settings.HoldReasonUnhealthyStop, UnhealthyKind: "crash_loop"}, "stopped by the box for a crash loop"},
{settings.RestoreHold{Stack: "a", At: "t", Reason: settings.HoldReasonUpdateFailed}, "failed update"},
{settings.RestoreHold{Stack: "b", At: "t", Reason: settings.HoldReasonRestoreMixed}, "off-site restore failed its database replay"},
// R-1340 (2026-10-29): a record with no cause fields no longer claims „off-site … database replay" — that was the
// false story (a local restore of an app with no database read the same); it says what is known.
{settings.RestoreHold{Stack: "b", At: "t", Reason: settings.HoldReasonRestoreMixed}, "restore stopped half-way"},
{settings.RestoreHold{Stack: "c", At: "t", Reason: settings.HoldReasonPutBackPending}, "pre-restore copy is poured back"},
{settings.RestoreHold{Stack: "d", At: "t"}, "failed restore whose rollback also failed"},
} {
@@ -0,0 +1,41 @@
package main
import (
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-1347 and R-1348: each seam is CALLED from main.go (an AST walk) — a seam built and never wired is the project's
// oldest shape.
func TestR1347_UpdateHeldForgetterIsWiredAtStartup(t *testing.T) {
lines, _, _ := slice4CallLines(t)
if len(lines["SetUpdateHeldForgetter"]) == 0 {
t.Fatal("main.go never calls SetUpdateHeldForgetter — a restored app would stay unwatched until the controller restarts")
}
}
func TestR1348_HeldNotifyIsWiredAtStartup(t *testing.T) {
lines, _, _ := slice4CallLines(t)
if len(lines["SetHeldNotify"]) == 0 {
t.Fatal("main.go never calls SetHeldNotify — a failed put-back at start-up would mail nobody")
}
}
// The mail's run name for an interrupted put-back. RED (2026-11-03, live on 9202): the sink showed run_kind offsite-reconstitute for a
// local restore.
func TestR1348_HeldRunKind(t *testing.T) {
for _, c := range []struct {
h settings.RestoreHold
want string
}{
{settings.RestoreHold{Reason: settings.HoldReasonRestoreMixed, MixedCause: settings.MixedCauseInterruptedPutBack}, "restore"},
{settings.RestoreHold{RestoreKind: settings.RestoreKindLocal}, "restore"},
{settings.RestoreHold{RestoreKind: settings.RestoreKindOffsite}, "offsite-reconstitute"},
{settings.RestoreHold{Reason: settings.HoldReasonRestoreMixed}, "offsite-reconstitute"},
} {
if got := heldRunKind(c.h); got != c.want {
t.Errorf("%+v: run kind %q, want %q", c.h, got, c.want)
}
}
}
+80
View File
@@ -0,0 +1,80 @@
package main
import (
"os"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/report"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
)
// R-1350 (measured 2026-10-29 14:57 on Tester 1): the operator cleared gokapi's crash-loop hold from the hub, the app
// stayed stopped (the clear starts nothing), and 30 s later the dead-app scan sent „Telepített alkalmazás nem fut:
// Gokapi" — the hold had kept the app out of the scan, and the household's recorded wish was still „running".
// The fix: the clear records that the stop is WANTED (desired state „stopped") for an app that is not up, BEFORE the
// hold is lifted; a later Start records „running" and ends it.
// The consequence, through the scan's own derivation: a stopped app whose wish is „running" is DOWN (the alarm); the
// same app with the wish „stopped" is not.
func TestR1350_AStopWantedIsNotDown(t *testing.T) {
st := func(desired string) []stacks.Stack {
return []stacks.Stack{{Name: "gokapi", Deployed: true, State: stacks.StateStopped, AppConfig: &stacks.AppConfig{DesiredState: desired}}}
}
if _, states := classifyRunStates(st(stacks.DesiredStateRunning), nil, nil, time.Now()); len(states) != 1 || !states[0].Down {
t.Fatalf("fixture: a stopped app wished running must be down: %+v", states)
}
if _, states := classifyRunStates(st(stacks.DesiredStateStopped), nil, nil, time.Now()); len(states) != 1 || states[0].Down {
t.Fatalf("a stopped app wished stopped is down — the alarm fires: %+v", states)
}
}
// The clear records the wish BEFORE the hold is lifted (a dead-app scan between the two would read „no hold, wished
// running" and mail), and only for an app that IS held. RED-PROOF: drop the keepStopped call from the clear → FAILS
// („not recorded"); call it after the lift → FAILS („after the hold was lifted").
func TestR1350_TheClearRecordsTheStopIsWanted(t *testing.T) {
r := newD64Rig(t)
r.hold(t, settings.RestoreHold{Stack: "gokapi", Reason: settings.HoldReasonUnhealthyStop, UnhealthyKind: "crash_loop", Trip: 1})
var recorded []string
var heldWhenRecorded bool
r.holdDeps.keepStopped = func(stack string) (bool, error) {
recorded = append(recorded, stack)
_, heldWhenRecorded = r.sett.GetRestoreHold(stack)
return true, nil
}
o := r.actions()
o.Reconcile([]report.OperatorAction{{ID: 21, Action: report.OpClearHold, Arg: "gokapi"}})
if res := onlyResult(t, o, 21); res.Outcome != report.OutcomeDone {
t.Fatalf("result %+v", res)
}
if len(recorded) != 1 || recorded[0] != "gokapi" {
t.Fatalf("the stop was not recorded as wanted: %v", recorded)
}
if !heldWhenRecorded {
t.Fatal("the wish was recorded after the hold was lifted — a scan in between mails app_start_failed")
}
if lg := r.logBuf.String(); !strings.Contains(lg, "gokapi: the stop is recorded as wanted") {
t.Fatalf("the box's log must say it:\n%s", lg)
}
// a refusal (no hold) records nothing — a down app with no hold is a real fault the alarm must keep
o.Reconcile([]report.OperatorAction{{ID: 22, Action: report.OpClearHold, Arg: "kimai"}})
if res := onlyResult(t, o, 22); res.Outcome != report.OutcomeRefused {
t.Fatalf("result %+v", res)
}
if len(recorded) != 1 {
t.Fatalf("a refused clear recorded a wish: %v", recorded)
}
}
// The running controller's clear is wired with the record. RED-PROOF: drop keepStopped from main.go's holdDeps → FAILS.
func TestR1350_MainWiresTheRecord(t *testing.T) {
raw, err := os.ReadFile("main.go")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(raw), "keepStopped: stackMgr.RecordStopWantedIfDown") || !strings.Contains(string(raw), "clear: appStopGuard.ClearHoldByOperatorThen") {
t.Fatal("main.go's hold clear does not record the wanted stop (R-1350)")
}
}
@@ -0,0 +1,60 @@
package main
import (
"context"
"errors"
"io"
"log"
"net/http"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
)
// R-528: the scan's host counter source. An old agent (404) is asked again only after hostOOMRetryAfter; a transient
// failure is retried at the next scan; the counters pass through.
// RED-PROOF: drop the 404 back-off → an old agent is asked at every 30-s scan (calls 3, not 1) → FAILS.
func TestR528_HostOOMSourceBacksOffAnOldAgent(t *testing.T) {
clock := time.Date(2026, 10, 28, 23, 0, 0, 0, time.UTC)
calls := 0
var answer error = &agentapi.StatusError{Path: "/guest/oom-kills", Code: http.StatusNotFound}
src := hostOOMSource(func(context.Context) (agentapi.GuestOOMKillsState, error) {
calls++
if answer != nil {
return agentapi.GuestOOMKillsState{}, answer
}
return agentapi.GuestOOMKillsState{Available: true, Containers: map[string]int64{"id": 2}}, nil
}, log.New(io.Discard, "", 0), func() time.Time { return clock })
for i := 0; i < 3; i++ {
if got := src(); got != nil {
t.Fatalf("an old agent: %v", got)
}
clock = clock.Add(30 * time.Second)
}
if calls != 1 {
t.Fatalf("an old agent was asked %d times in 90 s, want 1", calls)
}
clock = clock.Add(hostOOMRetryAfter)
answer = nil
if got := src(); got["id"] != 2 || calls != 2 {
t.Fatalf("after the back-off, an upgraded agent: %v (calls %d)", got, calls)
}
answer = errors.New("timeout")
if got := src(); got != nil || calls != 3 {
t.Fatalf("a transient failure: %v", got)
}
answer = nil
if got := src(); got["id"] != 2 || calls != 4 {
t.Fatalf("the next scan after a transient failure must ask again: %v (calls %d)", got, calls)
}
}
// The source is wired into the stack manager at startup (the seam-built-but-never-wired class).
// RED-PROOF: delete the SetHostOOMKills line in main.go → FAILS.
func TestR528_HostOOMSourceIsWiredAtStartup(t *testing.T) {
lines, _, _ := slice4CallLines(t)
if len(lines["SetHostOOMKills"]) == 0 || len(lines["hostOOMSource"]) == 0 {
t.Fatal("the host OOM counter is never wired — the 30-s scan would read Docker's flag only (R-528)")
}
}
+13 -1
View File
@@ -1071,6 +1071,18 @@ func (c *Client) RemoveNetStorage(ctx context.Context, name string) error {
// status, even on a non-2xx (so callers like FormatDisk can read a 403 refusal body). A transport or
// envelope-parse failure is still an error; an `ok:false` business refusal is NOT (the data carries it).
func (c *Client) postWithStatus(ctx context.Context, path string, body any) (apiResponse, int, error) {
return c.postWithStatusOn(c.hc, ctx, path, body)
}
// postWithStatusTimeout is postWithStatus with its own overall timeout (the escrow claim and confirm, R-1254). It shares
// the pinned transport.
func (c *Client) postWithStatusTimeout(ctx context.Context, path string, body any, timeout time.Duration) (apiResponse, int, error) {
hc := *c.hc
hc.Timeout = timeout
return c.postWithStatusOn(&hc, ctx, path, body)
}
func (c *Client) postWithStatusOn(hc *http.Client, ctx context.Context, path string, body any) (apiResponse, int, error) {
var env apiResponse
buf, err := json.Marshal(body)
if err != nil {
@@ -1083,7 +1095,7 @@ func (c *Client) postWithStatus(ctx context.Context, path string, body any) (api
req.Header.Set("Authorization", "Bearer "+c.token)
req.Header.Set("Content-Type", "application/json")
start := time.Now()
resp, err := c.hc.Do(req)
resp, err := hc.Do(req)
if err != nil {
logx.Debugf(c.logger, "[agentapi] POST %s failed after %dms: %v", path, time.Since(start).Milliseconds(), err)
return env, 0, fmt.Errorf("agentapi: POST %s: %w", path, err)
+10 -2
View File
@@ -6,6 +6,7 @@ import (
"errors"
"fmt"
"net/http"
"time"
)
// Controller-driven escrow ceremony client methods (v0.127.0, agent ≥ v0.88.0). The claim call
@@ -108,8 +109,15 @@ type EscrowClaim struct {
//
// R-1151: the claim asks the agent to keep the new package until the household typed the code back
// (`defer_upload`). An agent older than 0.160.0 ignores the body and uploads as before.
// escrowCallTimeout bounds the claim and the confirm (R-1254 H2). The agent may take its whole upload timeout (30 s)
// plus its question to the hub (15 s) before it answers; the client's own 15 s used to abandon the call while the
// agent went on — a claim abandoned that way could still store the package, and its code reached nobody. 60 s
// outlasts both (TestR1254_ClaimAndConfirmOutwaitTheAgentsUpload; the agent side: escrowUploadTimeout,
// escrowHoldTimeout in felhom-agent internal/localapi/escrow_ceremony.go — if either grows, this must too).
var escrowCallTimeout = 60 * time.Second
func (c *Client) EscrowCeremonyClaim(ctx context.Context) (EscrowClaim, int, error) {
env, status, err := c.postWithStatus(ctx, "/escrow/ceremony/claim", map[string]bool{"defer_upload": true})
env, status, err := c.postWithStatusTimeout(ctx, "/escrow/ceremony/claim", map[string]bool{"defer_upload": true}, escrowCallTimeout)
if err != nil {
return EscrowClaim{}, status, err
}
@@ -141,7 +149,7 @@ func (c *Client) EscrowCeremonyClaim(ctx context.Context) (EscrowClaim, int, err
// match (still waiting), 410 the window ran out, 404 nothing waits (superseded, restarted), 502 the hub failed (still
// waiting). The words are part of the recovery code — never logged.
func (c *Client) EscrowCeremonyConfirm(ctx context.Context, words []string) (int, error) {
env, status, err := c.postWithStatus(ctx, "/escrow/ceremony/confirm", map[string][]string{"words": words})
env, status, err := c.postWithStatusTimeout(ctx, "/escrow/ceremony/confirm", map[string][]string{"words": words}, escrowCallTimeout)
if err != nil {
return status, err
}
+31
View File
@@ -0,0 +1,31 @@
package agentapi
import (
"context"
"encoding/json"
"fmt"
)
// GuestOOMKillsState mirrors the agent's GET /guest/oom-kills (R-528, agent ≥ the R-528 commit): the kernel's oom_kill
// counter of each of THIS guest's containers, read on the host — Docker's own OOMKilled flag can stay false inside an
// LXC guest. Containers maps the full 64-hex container id to the count for the container's current run.
// Available=false: the guest has no cgroup folder the host can read. An agent that predates the route answers 404 (a
// *StatusError) — the controller then keeps Docker's flag only.
type GuestOOMKillsState struct {
VMID int `json:"vmid"`
Available bool `json:"available"`
Containers map[string]int64 `json:"containers"`
}
// GuestOOMKills calls GET /guest/oom-kills.
func (c *Client) GuestOOMKills(ctx context.Context) (GuestOOMKillsState, error) {
var out GuestOOMKillsState
body, err := c.get(ctx, "/guest/oom-kills")
if err != nil {
return out, err
}
if err := json.Unmarshal(body, &out); err != nil {
return out, fmt.Errorf("agentapi: decode /guest/oom-kills: %w", err)
}
return out, nil
}
@@ -0,0 +1,33 @@
package agentapi
import (
"context"
"encoding/json"
"fmt"
"time"
)
// OperatorReboot calls GET /guest/operator-reboot (D65 = A, agent ≥ the reboot_guest commit): the time the operator
// rebooted THIS guest from the hub, when it is less than a day old; the zero time when there is none. An agent that
// predates the route answers 404 (a *StatusError) — such an agent cannot reboot from the hub either, so there is
// nothing to show and no MinAgent is needed.
func (c *Client) OperatorReboot(ctx context.Context) (time.Time, error) {
body, err := c.get(ctx, "/guest/operator-reboot")
if err != nil {
return time.Time{}, err
}
var out struct {
At string `json:"at"`
}
if err := json.Unmarshal(body, &out); err != nil {
return time.Time{}, fmt.Errorf("agentapi: decode /guest/operator-reboot: %w", err)
}
if out.At == "" {
return time.Time{}, nil
}
at, err := time.Parse(time.RFC3339, out.At)
if err != nil {
return time.Time{}, fmt.Errorf("agentapi: /guest/operator-reboot: bad time %q", out.At)
}
return at, nil
}
@@ -0,0 +1,42 @@
package agentapi
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
)
// R-1254 H2: the agent's claim/confirm may take its whole upload timeout (30 s) plus the hub question (15 s) before it
// answers. The controller's agent client gives up at 15 s — and a claim the controller abandons can still complete on
// the agent: the hub then holds a package whose code nobody saw. The two calls carry their own, longer timeout.
// RED-PROOF: send claim/confirm through the plain 15 s client again → the slow agent below times them out → FAILS.
func TestR1254_ClaimAndConfirmOutwaitTheAgentsUpload(t *testing.T) {
if escrowCallTimeout < 45*time.Second+10*time.Second {
t.Fatalf("escrowCallTimeout %s must outlast the agent's upload (30 s) + hub question (15 s) with room", escrowCallTimeout)
}
mux := http.NewServeMux()
mux.HandleFunc("POST /escrow/ceremony/claim", func(w http.ResponseWriter, r *http.Request) {
time.Sleep(300 * time.Millisecond)
_, _ = w.Write([]byte(`{"ok":true,"data":{"recovery_code":"a-b-c"}}`))
})
mux.HandleFunc("POST /escrow/ceremony/confirm", func(w http.ResponseWriter, r *http.Request) {
time.Sleep(300 * time.Millisecond)
_, _ = w.Write([]byte(`{"ok":true,"data":{"uploaded":true}}`))
})
s := httptest.NewTLSServer(mux)
defer s.Close()
c := clientFor(t, s, strings.TrimPrefix(s.URL, "https://"))
c.hc.Timeout = 100 * time.Millisecond // the client's own short timeout, scaled down
old := escrowCallTimeout
escrowCallTimeout = 5 * time.Second
defer func() { escrowCallTimeout = old }()
if claim, st, err := c.EscrowCeremonyClaim(context.Background()); err != nil || st != http.StatusOK || claim.Code != "a-b-c" {
t.Fatalf("a slow claim was abandoned: %+v %d %v", claim, st, err)
}
if st, err := c.EscrowCeremonyConfirm(context.Background(), []string{"a", "c"}); err != nil || st != http.StatusOK {
t.Fatalf("a slow confirm was abandoned: %d %v", st, err)
}
}
+11 -6
View File
@@ -94,7 +94,9 @@ func (r *Router) keptDataAtInstall(w http.ResponseWriter, req *http.Request, nam
writeJSON(w, http.StatusConflict, apiResponse{OK: false, Error: data.UseOff, Data: data})
return true
}
if st := r.backupMgr.RestoreStatus(); st.Running || r.backupMgr.IsRunning() {
// R-1256 H6: every operation — a restore, a backup run, a second-drive copy — not only the first two; and the
// load itself begins in the same step as this question (startKeptLoad → ErrRestoreBusy).
if busy, _ := r.backupMgr.RestoreBusy(); busy != "" {
writeJSON(w, http.StatusConflict, apiResponse{OK: false, Error: r.msgLang(lang, "api.kept.busy")})
return true
}
@@ -104,7 +106,11 @@ func (r *Router) keptDataAtInstall(w http.ResponseWriter, req *http.Request, nam
}
r.logger.Printf("[INFO] [api] Deploy %s: USE MY KEPT DATA — loading from %s (tier %d, %s) under the kept files %v", name, src, cp.c.Tier, cp.c.Time.UTC().Format(time.RFC3339), old)
r.dropLeftoverHold(name, "use-kept-data") // R-704: this is a new install of an app that is not installed
r.startKeptLoad(name, hdd, cp.c, lang, nil)
if err := r.startKeptLoad(name, hdd, cp.c, lang, nil); err != nil {
r.logger.Printf("[WARN] [api] Deploy %s: use my kept data REFUSED at its start: %v", name, err)
writeJSON(w, http.StatusConflict, apiResponse{OK: false, Error: r.msgLang(lang, "api.kept.busy")})
return true
}
writeJSON(w, http.StatusAccepted, apiResponse{OK: true, Message: r.msgLang(lang, "kept.load.started", display)})
return true
default:
@@ -120,7 +126,7 @@ func (r *Router) keptCopyName(lang string, c backup.KeptCopy) string {
// startKeptLoad runs the load and, when it succeeded, the app's after_load. then(ok) runs last. The
// off-site copy goes through LoadKeptOffsite (download the unit alone, judge it, then the same restore).
func (r *Router) startKeptLoad(name, drive string, c backup.KeptCopy, lang string, then func(ok bool)) {
func (r *Router) startKeptLoad(name, drive string, c backup.KeptCopy, lang string, then func(ok bool)) error {
display := name
if st, ok := r.stackMgr.GetStack(name); ok && st.Meta.DisplayName != "" {
display = st.Meta.DisplayName
@@ -128,10 +134,9 @@ func (r *Router) startKeptLoad(name, drive string, c backup.KeptCopy, lang strin
okMsg := func(error) string { return r.msgLang(lang, "kept.load.done", display) }
failMsg := func(err error) string { return r.msgLang(lang, "kept.load.failed", display, err) }
if c.Tier == backup.KeptTierOffsite {
r.backupMgr.LoadKeptOffsite(name, drive, c, nil, okMsg, failMsg, r.keptAfterLoad(name, then))
return
return r.backupMgr.LoadKeptOffsite(name, drive, c, nil, okMsg, failMsg, r.keptAfterLoad(name, then))
}
r.backupMgr.LoadKeptApp(name, c.UnitDir, okMsg, failMsg, r.keptAfterLoad(name, then))
return r.backupMgr.LoadKeptApp(name, c.UnitDir, nil, okMsg, failMsg, r.keptAfterLoad(name, then))
}
// keptAfterLoad runs the app's after_load when the load succeeded, then then(ok).
+44 -1
View File
@@ -162,7 +162,10 @@ func TestR691_InstallChoiceNamesTheOffsiteCopy(t *testing.T) {
// copy, restic and the unit restore behind their seams — no Docker).
// COMPANION RED-PROOF: remove the dropLeftoverHold call from keptDataAtInstall → the hold is still there
// after the 202 and the first assertion fails; remove it from deployStack → the AST check fails.
func TestR704_AFreshInstallDropsALeftoverHold(t *testing.T) {
// keptInstallRig is an installable app with old data on its drive and an off-site copy of its unit; done receives
// once per unit restore the load reaches.
func keptInstallRig(t *testing.T) (*Router, *settings.Settings, string, chan struct{}) {
t.Helper()
dir := t.TempDir()
drive := filepath.Join(dir, "drive")
cfg := &config.Config{}
@@ -234,6 +237,11 @@ func TestR704_AFreshInstallDropsALeftoverHold(t *testing.T) {
return backup.UnitRestoreResult{}, nil
})
r := &Router{stackMgr: m, backupMgr: bm, sett: sett, logger: log.New(io.Discard, "", 0)}
return r, sett, drive, done
}
func TestR704_AFreshInstallDropsALeftoverHold(t *testing.T) {
r, sett, drive, done := keptInstallRig(t)
// A leftover update hold from an earlier install (the demo-hp shape): the new install drops it.
_ = sett.SetRestoreHold(settings.RestoreHold{Stack: "cloudapp", At: "2026-09-13T19:56:32Z", Reason: settings.HoldReasonUpdateFailed})
@@ -287,3 +295,38 @@ func TestR704_AFreshInstallDropsALeftoverHold(t *testing.T) {
t.Fatalf("deployStack must call dropLeftoverHold before DeployStack (drop=%v deploy=%v)", dropAt, deployAt)
}
}
// R-1256 H6: the install-time „use my kept data" asks about EVERY operation — a second-drive copy too, not only a
// restore record and the backup run — and the load begins in the same step as the question.
// RED-PROOF: restore the RestoreStatus/IsRunning-only check and LoadKept*'s plain BeginRestoreOp → 202 and the load
// runs beside the copy → FAILS.
func TestR1256_H6_AKeptLoadAtInstallIsRefusedWhileASecondDriveCopyRuns(t *testing.T) {
r, _, drive, done := keptInstallRig(t)
end := backup.MarkTier2CopyForTest("someapp")
w := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodPost, "/api/stacks/cloudapp/deploy?lang=en", nil)
r.keptDataAtInstall(w, req, "cloudapp", drive, "use")
end()
if w.Code != http.StatusConflict {
t.Fatalf("a load during a second-drive copy: code %d, want 409 — body %s", w.Code, w.Body.String())
}
select {
case <-done:
t.Fatal("the load ran beside the second-drive copy")
case <-time.After(300 * time.Millisecond):
}
if st := r.backupMgr.RestoreStatus(); st.Running {
t.Fatal("a refused load left a restore op recorded")
}
// The copy ended: the same press loads.
w = httptest.NewRecorder()
r.keptDataAtInstall(w, httptest.NewRequest(http.MethodPost, "/api/stacks/cloudapp/deploy?lang=en", nil), "cloudapp", drive, "use")
if w.Code != http.StatusAccepted {
t.Fatalf("after the copy: code %d — body %s", w.Code, w.Body.String())
}
select {
case <-done:
case <-time.After(10 * time.Second):
t.Fatal("the load never reached the unit restore")
}
}
+6
View File
@@ -771,6 +771,12 @@ func (r *Router) actionStack(w http.ResponseWriter, req *http.Request, action, n
// above, so nothing is recorded for an action that was never going to happen.
if desired, ok := desiredStateForAction(action); ok {
if derr := r.stackMgr.SetDesiredState(name, desired); derr != nil {
if errors.Is(derr, stacks.ErrStackNotFound) {
// no app by that name on this box (e.g. the bootstrap-managed controller): a 404, not a save failure
r.logger.Printf("[WARN] [api] %s for %s refused: no such app", action, name)
writeJSON(w, stackOpStatusFor(derr), apiResponse{OK: false, Error: r.errText(req, derr)})
return
}
r.logger.Printf("[ERROR] [api] %s for %s refused: could not record desired state: %v", action, name, derr)
writeJSON(w, http.StatusInternalServerError, apiResponse{
OK: false,
@@ -0,0 +1,29 @@
package api
import (
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
// Found walking the capability map's B2 („App lifecycle") on 9202, 2026-10-28 night: a Stop of a name that is no app
// on this box (`felhom-controller` — bootstrap-managed, not a stack) answered 500 „…az alkalmazás beállításai nem
// menthetők." — the desired-state write's „not found" read as a save failure. It is a 404 now, before any intent is
// recorded. RED-PROOF: drop the ErrStackNotFound branch in actionStack → 500 → FAILS.
func TestAnActionOnANameThatIsNoAppIsA404(t *testing.T) {
r, _, _, _ := newSlice4Router(t)
for _, action := range []string{"stop", "start"} {
w := httptest.NewRecorder()
r.actionStack(w, httptest.NewRequest("POST", "/api/stacks/nosuchapp/"+action, nil), action, "nosuchapp")
var resp apiResponse
_ = json.Unmarshal(w.Body.Bytes(), &resp)
if w.Code != http.StatusNotFound {
t.Fatalf("%s of a name that is no app: %d %q — want 404", action, w.Code, resp.Error)
}
if strings.Contains(strings.ToLower(resp.Error), "mentheto") || strings.Contains(resp.Error, "menthetők") {
t.Fatalf("%s: the answer still blames a save: %q", action, resp.Error)
}
}
}
@@ -137,6 +137,10 @@ type AppStopGuard struct {
// whether the drive a copy lives on is bound. nil = the pre-R-1251 behaviour.
holds AppStopHolds
undoReachable func(stack, dir string) bool
// heldNotify and heldPending (R-1348, appstop_putback_wait.go): the operator mail for a failed put-back. Recover runs
// before the notifier exists, so a hold written then waits in heldPending until SetHeldNotify.
heldNotify func(stack string)
heldPending []string
// liveOp (R-1253) is the operation of THIS process that holds the marker now (set by Begin, cleared by End). The
// marker has one slot; BeginExclusive refuses to replace a live one. A marker left by a crash has no live op.
@@ -119,15 +119,49 @@ func (g *AppStopGuard) holdPutBackFailed(stack, dir string, cause error) bool {
return false
}
h := settings.RestoreHold{Stack: stack, At: g.now().UTC().Format(time.RFC3339), Reason: settings.HoldReasonRestoreMixed,
ReplayError: "interrupted restore, put-back failed: " + cause.Error(), SafetyDump: dir}
ReplayError: "interrupted restore, put-back failed: " + cause.Error(), SafetyDump: dir,
MixedCause: settings.MixedCauseInterruptedPutBack, FailedStep: "putting the interrupted restore's copy back failed: " + cause.Error()}
if err := g.holds.SetRestoreHold(h); err != nil {
g.logger.Printf("[ERROR] [appstop] %s: the failed put-back could not be recorded as a hold: %v — the marker is kept", stack, err)
return false
}
g.logger.Printf("[ERROR] [appstop] %s: the interrupted restore could NOT be put back (%v) — the app is HELD for support, the copy stays at %s (R-1251 H2)", stack, cause, dir)
g.notifyHeld(stack)
return true
}
// SetHeldNotify wires the operator mail for a failed put-back (R-1348; main.go: the R-1340 sentence from the hold record)
// and hands over what Recover held before the notifier existed. Each held app is mailed once. Pinned by TestR1348_* and
// TestR1348_HeldNotifyIsWiredAtStartup.
func (g *AppStopGuard) SetHeldNotify(fn func(stack string)) {
if g == nil {
return
}
g.liveMu.Lock()
g.heldNotify = fn
waiting := g.heldPending
g.heldPending = nil
g.liveMu.Unlock()
if fn == nil {
return
}
for _, stack := range waiting {
fn(stack)
}
}
func (g *AppStopGuard) notifyHeld(stack string) {
g.liveMu.Lock()
fn := g.heldNotify
if fn == nil {
g.heldPending = append(g.heldPending, stack)
}
g.liveMu.Unlock()
if fn != nil {
fn(stack)
}
}
// clearPutBackPending lifts this app's waiting-put-back hold, and only that kind.
func (g *AppStopGuard) clearPutBackPending(stack string) {
if g.holds == nil {
+6
View File
@@ -219,6 +219,9 @@ type Manager struct {
// undoCopyRemover (R-671, v0.272.0) deletes an app's leftover undo copies — stacks.Manager.RemoveUndoCopies,
// wired in main.go (SetUndoCopyRemover). nil-safe: without it the copies stay, as before.
undoCopyRemover func(stackName string) int
// updateHeldForgetter (R-1347) forgets the stack manager's „last update ended held" flag when a restore lifts the update
// hold — stacks.Manager.ForgetUpdateHeld, wired in main.go (SetUpdateHeldForgetter). nil-safe.
updateHeldForgetter func(stackName string) bool
// R-475 update-precondition seams, one per tier. Nil → the real Tier2UnitRestorePoint /
// ListRestorePoints / OffsiteSnapshotTimes. They let a test reach Tier 1 and Tier 3 without a drive
@@ -340,6 +343,9 @@ type Manager struct {
// R-550: persistence of the above (restore_record.go) and the per-app interrupted notices.
opRecordPath string
opInterrupted map[string]RestoreOpResult
// R-1342: the restore copies a stable-start watch still holds (copy dir → true), guarded by mu. While one is here the
// copy is the way back if the restored app turns out unstable: „Törlés most" refuses it (restore_copies.go).
watching map[string]bool
// R-519: the app-data run record (run_record.go) and the start of the last run a stop cut off (zero = none).
runRecordPath string
runInterrupted time.Time
@@ -0,0 +1,185 @@
package backup
import (
"errors"
"os"
"path/filepath"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// D62 (operator ruling 2026-10-29): an unused restore copy is kept 30 days; the household sees each copy (app, the
// restore's date, size, the day it goes) and may delete one earlier. A copy a hold or a pending put-back needs is never
// deleted, by the sweep or by hand.
// RED-PROOF: with the age at 14 days the 20-day copy is swept → FAILS; at 30 days only the 31-day copy goes.
func TestD62_AnUnusedCopyIsKept30Days(t *testing.T) {
m, _ := newOffboxManager(t)
now := time.Now()
day29 := r1256Copy(t, m, "a", 29*24*time.Hour, now)
day20 := r1256Copy(t, m, "b", 20*24*time.Hour, now)
day31 := r1256Copy(t, m, "c", 31*24*time.Hour, now)
got := m.SweepUnusedRestoreCopies(now)
if len(got) != 1 || got[0] != day31 {
t.Fatalf("swept %v — want only the 31-day copy %s", got, day31)
}
if !r1256Exists(day29) || !r1256Exists(day20) {
t.Fatal("a copy younger than 30 days was swept")
}
if restoreCopyUnusedAge != 30*24*time.Hour {
t.Fatalf("restoreCopyUnusedAge = %s, D62 says 30 days", restoreCopyUnusedAge)
}
}
// The list shows each copy with its restore date, its size and the day it goes; a needed copy has no such day and
// says why. RED-PROOF: drop the hold check from restoreCopyNeededFor → the held copy is offered → FAILS.
func TestD62_TheListShowsDateSizeAndWhyACopyIsNeeded(t *testing.T) {
m, sett := newOffboxManager(t)
now := time.Now().UTC().Truncate(time.Second)
free := r1256Copy(t, m, "free", 2*24*time.Hour, now)
r1256Copy(t, m, "held", 2*24*time.Hour, now)
r1256Copy(t, m, "waiting", 2*24*time.Hour, now)
r1256Copy(t, m, "named", 2*24*time.Hour, now)
if err := sett.SetRestoreHold(settings.RestoreHold{Stack: "held", At: now.Format(time.RFC3339), Reason: settings.HoldReasonRestoreMixed}); err != nil {
t.Fatal(err)
}
if err := sett.SetRestoreHold(settings.RestoreHold{Stack: "waiting", At: now.Format(time.RFC3339), Reason: settings.HoldReasonPutBackPending}); err != nil {
t.Fatal(err)
}
if err := m.appStop.Begin("unit-restore:named", ReasonUnitRestore, []string{"named"}); err != nil {
t.Fatal(err)
}
list := m.RestoreCopies()
if len(list) != 4 {
t.Fatalf("listed %d copies, want 4: %+v", len(list), list)
}
want := map[string]string{"free": "", "held": RestoreCopyNeededHold, "waiting": RestoreCopyNeededPutBack, "named": RestoreCopyNeededPutBack}
for _, c := range list {
if c.Needed != want[c.App] {
t.Errorf("%s: needed %q, want %q", c.App, c.Needed, want[c.App])
}
if !c.At.Equal(now.Add(-2 * 24 * time.Hour)) {
t.Errorf("%s: restore date %s, want the stamp's %s", c.App, c.At, now.Add(-2*24*time.Hour))
}
if c.SizeBytes != int64(len("LIVE")) {
t.Errorf("%s: size %d", c.App, c.SizeBytes)
}
if c.Needed == "" && !c.DeleteOn.Equal(c.At.Add(30*24*time.Hour)) {
t.Errorf("%s: deleted on %s, want 30 days after the restore", c.App, c.DeleteOn)
}
if c.Needed != "" && !c.DeleteOn.IsZero() {
t.Errorf("%s: a needed copy shows a delete day %s — the sweep never takes it", c.App, c.DeleteOn)
}
if c.App == "free" && c.Path != free {
t.Errorf("path %s, want %s", c.Path, free)
}
}
}
// A held or waiting copy cannot be deleted by hand. RED-PROOF: drop the Needed check in DeleteRestoreCopy → FAILS.
func TestD62_AHeldCopyCannotBeDeleted(t *testing.T) {
m, sett := newOffboxManager(t)
now := time.Now()
for _, tc := range []struct{ app, reason string }{
{"held", settings.HoldReasonRestoreMixed},
{"waiting", settings.HoldReasonPutBackPending},
{"failed", settings.HoldReasonRestoreFailed},
} {
dir := r1256Copy(t, m, tc.app, time.Hour, now)
if err := sett.SetRestoreHold(settings.RestoreHold{Stack: tc.app, At: now.Format(time.RFC3339), Reason: tc.reason}); err != nil {
t.Fatal(err)
}
if _, err := m.DeleteRestoreCopy(tc.app, filepath.Base(dir)); !errors.Is(err, ErrRestoreCopyNeeded) {
t.Errorf("%s: err %v, want ErrRestoreCopyNeeded", tc.app, err)
}
if !r1256Exists(dir) {
t.Errorf("%s: a held copy was deleted", tc.app)
}
}
// a copy a cut restore names (the marker) is needed too
named := r1256Copy(t, m, "named", time.Hour, now)
if err := m.appStop.Begin("unit-restore:named", ReasonUnitRestore, []string{"named"}); err != nil {
t.Fatal(err)
}
if _, err := m.DeleteRestoreCopy("named", filepath.Base(named)); !errors.Is(err, ErrRestoreCopyNeeded) || !r1256Exists(named) {
t.Errorf("a copy the marker names: err %v, exists %v", err, r1256Exists(named))
}
// an unreadable marker offers nothing
mustWrite(t, m.appStop.path, "{not json")
free := r1256Copy(t, m, "free", time.Hour, now)
if _, err := m.DeleteRestoreCopy("free", filepath.Base(free)); !errors.Is(err, ErrRestoreCopyNeeded) || !r1256Exists(free) {
t.Errorf("an unreadable marker: err %v, exists %v", err, r1256Exists(free))
}
}
// A delete removes ONLY that copy: the app's other copy, another app's copy and anything outside the copy tree stay;
// a name that is not one path element, or a link in the copy's place, is refused.
// RED-PROOF: remove the validCopyName check → "../.." reaches outside → FAILS; remove the Lstat link check → the
// link is followed/removed → FAILS.
func TestD62_ADeleteRemovesOnlyThatCopy(t *testing.T) {
m, _ := newOffboxManager(t)
now := time.Now()
one := r1256Copy(t, m, "app", 3*time.Hour, now)
two := r1256Copy(t, m, "app", 2*time.Hour, now)
other := r1256Copy(t, m, "other", 3*time.Hour, now)
got, err := m.DeleteRestoreCopy("app", filepath.Base(one))
if err != nil {
t.Fatal(err)
}
if got.Path != one || got.SizeBytes != int64(len("LIVE")) || got.App != "app" {
t.Fatalf("returned %+v", got)
}
if r1256Exists(one) || !r1256Exists(two) || !r1256Exists(other) {
t.Fatalf("one gone=%v, two kept=%v, other kept=%v", !r1256Exists(one), r1256Exists(two), r1256Exists(other))
}
// path guard
outside := filepath.Join(m.namespaceRoot(m.systemDataPath), "backups", "primary", "keep.txt")
mustWrite(t, outside, "UNIT")
for _, bad := range [][2]string{{"app", ".."}, {"..", "primary"}, {"app", "../../primary"}, {"", "x"}, {"app", ""}, {"app/x", "y"}} {
if _, err := m.DeleteRestoreCopy(bad[0], bad[1]); !errors.Is(err, ErrRestoreCopyNotFound) {
t.Errorf("DeleteRestoreCopy(%q,%q) = %v, want not found", bad[0], bad[1], err)
}
}
if !r1256Exists(outside) || !r1256Exists(two) {
t.Fatal("a bad name reached outside the copy")
}
// a link in a copy's place is not a copy
target := t.TempDir()
mustWrite(t, filepath.Join(target, "precious"), "X")
link := filepath.Join(filepath.Dir(two), "20260101T000000Z")
if err := os.Symlink(target, link); err != nil {
t.Fatal(err)
}
if _, err := m.DeleteRestoreCopy("app", "20260101T000000Z"); !errors.Is(err, ErrRestoreCopyNotFound) {
t.Errorf("a link: %v, want not found", err)
}
if !r1256Exists(filepath.Join(target, "precious")) {
t.Fatal("the link's target was removed")
}
if _, err := m.DeleteRestoreCopy("app", "20990101T000000Z"); !errors.Is(err, ErrRestoreCopyNotFound) {
t.Errorf("a missing copy: %v", err)
}
}
// While a backup or restore runs, a delete is refused and nothing moves. RED-PROOF: drop the busy check → FAILS.
func TestD62_NoDeleteWhileAnOperationRuns(t *testing.T) {
m, _ := newOffboxManager(t)
c := r1256Copy(t, m, "app", time.Hour, time.Now())
m.BeginRestoreOp("restore", "other")
if _, err := m.DeleteRestoreCopy("app", filepath.Base(c)); !errors.Is(err, ErrRestoreCopyBusy) || !r1256Exists(c) {
t.Fatalf("during a restore: err %v, exists %v", err, r1256Exists(c))
}
m.EndRestoreOp(true, "ok")
if err := m.acquireRunning(); err != nil {
t.Fatalf("could not take the backup single-flight: %v", err)
}
if _, err := m.DeleteRestoreCopy("app", filepath.Base(c)); !errors.Is(err, ErrRestoreCopyBusy) || !r1256Exists(c) {
t.Fatalf("during a backup run: err %v, exists %v", err, r1256Exists(c))
}
m.ReleaseRunningForTest()
if _, err := m.DeleteRestoreCopy("app", filepath.Base(c)); err != nil || r1256Exists(c) {
t.Fatalf("control: with nothing running the delete goes through: %v", err)
}
}
+76
View File
@@ -0,0 +1,76 @@
package backup
import (
"errors"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// ── D64 = B (operator ruling 2026-10-29 12:10) + R-1343: the operator clears a hold IN THE RUNNING CONTROLLER ──────
//
// One door for both of the operator's routes — the hub's „Clear hold" (operator action clear_hold) and the box command
// `--clear-restore-hold`, which now asks the running controller over its local socket (cmd/controller/holdctl.go).
//
// WHY IN THIS PROCESS (R-1343, measured 2026-10-29 on Tester 1): the CLI used to run as a SECOND process that edited
// settings.json while the running controller kept the hold in its own in-memory Settings — and that controller's next
// save (after a hub report) wrote the hold back. Clearing through the running Settings removes the hold from memory
// AND saves the file at once, so no later save of any component can bring it back.
// Pinned by TestD64_ClearHoldStaysGoneAfterALaterSave (cmd/controller).
//
// EVERY REASON (D64 = B): the hold's reason does not matter — R-379 now reads „a person — on the box, or the operator
// on the hub". One reason needs a second act: a put-back WAITING for its drive (HoldReasonPutBackPending) is also
// recorded in the app-stop marker, and a controller start with the drive bound would pour that old copy back over
// whatever the app wrote after the clear. So the clear also drops the app from the marker; the copy stays on disk
// (nothing is deleted). It runs under resumeMu, so it can never interleave with a put-back of the same app.
// Pinned by TestD64_ClearPutBackPendingDropsTheMarkerEntry.
// ErrHoldBadStack: the argument is not a valid stack name. ErrNotHeld: the app has no hold. Both are refusals —
// nothing was changed.
var (
ErrHoldBadStack = errors.New("not a valid app name")
ErrNotHeld = errors.New("this app has no hold")
)
// IsValidHoldStack reports whether s may name an app for the hold commands (the backup package's strict
// isSafeStackName: letters, digits, '-' and '_', at most 64).
func IsValidHoldStack(s string) bool { return isSafeStackName(s) }
// ClearHoldByOperator lifts stack's hold, whatever its reason, in the hold store the running controller uses, and
// returns the hold it lifted. A waiting put-back is dropped from the app-stop marker too (see above). Refusals
// (ErrHoldBadStack, ErrNotHeld) change nothing.
func (g *AppStopGuard) ClearHoldByOperator(stack string) (settings.RestoreHold, error) {
return g.ClearHoldByOperatorThen(stack, nil)
}
// ClearHoldByOperatorThen is ClearHoldByOperator with beforeLift run once the hold is FOUND and before it is lifted,
// under the same lock (R-1350: the household's „the stop is wanted" is recorded before the hold stops keeping the app out
// of the dead-app scan). beforeLift is not run on a refusal.
func (g *AppStopGuard) ClearHoldByOperatorThen(stack string, beforeLift func()) (settings.RestoreHold, error) {
if !isSafeStackName(stack) {
return settings.RestoreHold{}, ErrHoldBadStack
}
if g == nil || g.holds == nil {
return settings.RestoreHold{}, errors.New("no hold store is wired on this box")
}
resumeMu.Lock()
defer resumeMu.Unlock()
h, ok := g.holds.GetRestoreHold(stack)
if !ok {
return settings.RestoreHold{}, ErrNotHeld
}
if beforeLift != nil {
beforeLift()
}
cleared, err := g.holds.ClearRestoreHold(stack)
if err != nil {
return h, err
}
if !cleared {
return settings.RestoreHold{}, ErrNotHeld
}
if h.Reason == settings.HoldReasonPutBackPending {
g.dropFromMarker(stack)
g.logger.Printf("[WARN] [appstop] %s: its waiting put-back is DROPPED with the hold — the copy stays on disk at %s and is never poured back by itself", stack, h.SafetyDump)
}
return h, nil
}
+26 -5
View File
@@ -301,8 +301,12 @@ func (m *Manager) downloadKeptOffsiteUnit(ctx context.Context, app, drive string
// LoadKeptOffsite is LoadKeptApp from the off-site copy c: download the unit alone, judge it, then
// prepare (nil = nothing; a dated folder's files move back here), then the one unit restore, then
// after(ok). A download failure or a refusal never reaches prepare, so the kept files stay as they were.
func (m *Manager) LoadKeptOffsite(app, drive string, c KeptCopy, prepare func() error, okMsg, failMsg func(err error) string, after func(ok bool)) {
m.BeginRestoreOp("restore", app)
// ErrRestoreBusy (nothing started, after not called) when another operation runs (R-1256 H6).
func (m *Manager) LoadKeptOffsite(app, drive string, c KeptCopy, prepare func() error, okMsg, failMsg func(err error) string, after func(ok bool)) error {
if busy, _ := m.TryBeginRestoreOp("restore", app); busy != "" {
m.logger.Printf("[WARN] [backup] kept load %s REFUSED: a %s operation runs (R-1256)", app, busy)
return ErrRestoreBusy
}
go func() {
start := time.Now()
fail := func(err error) {
@@ -345,6 +349,7 @@ func (m *Manager) LoadKeptOffsite(app, drive string, c KeptCopy, prepare func()
after(true)
}
}()
return nil
}
// RemovedUnitOnDrive is the start-fresh hook (stacks.SetKeptUnitFinder): the removed app's OWN unit when
@@ -370,11 +375,26 @@ func (m *Manager) PrimaryUnitHome(app, drive string) string {
// LoadKeptApp runs the load as a restore operation the backup pages already follow (the poll banner):
// Begin → RestoreFromRecoveryUnitAt(app, unitDir) → End, then after(ok) in the same goroutine (the
// after_load command, the kept folder's tidy-up). The caller has checked RestoreStatus/IsRunning.
func (m *Manager) LoadKeptApp(app, unitDir string, okMsg, failMsg func(err error) string, after func(ok bool)) {
m.BeginRestoreOp("restore", app)
// after_load command, the kept folder's tidy-up). prepare (nil = nothing) runs first, inside the operation — a dated
// folder's files move back there. R-1256 H6: the op is begun in the same step as the check for every other operation;
// a refusal returns ErrRestoreBusy with nothing started and nothing moved.
func (m *Manager) LoadKeptApp(app, unitDir string, prepare func() error, okMsg, failMsg func(err error) string, after func(ok bool)) error {
if busy, _ := m.TryBeginRestoreOp("restore", app); busy != "" {
m.logger.Printf("[WARN] [backup] kept load %s REFUSED: a %s operation runs (R-1256)", app, busy)
return ErrRestoreBusy
}
go func() {
start := time.Now()
if prepare != nil {
if perr := prepare(); perr != nil {
m.logger.Printf("[ERROR] [backup] kept load %s: preparing the kept files FAILED: %v — nothing was loaded", app, perr)
m.EndRestoreOp(false, failMsg(perr))
if after != nil {
after(false)
}
return
}
}
res, err := m.keptUnitRestore()(app, unitDir)
if errors.Is(err, ErrRestoreNotStarted) {
m.keptLoadNotStarted(app, err, after)
@@ -395,6 +415,7 @@ func (m *Manager) LoadKeptApp(app, unitDir string, okMsg, failMsg func(err error
after(true)
}
}()
return nil
}
// keptLoadNotStarted (R-1270): the kept data is loaded but the app did not start in time. Never „done": the record says
@@ -425,9 +425,12 @@ func (m *Manager) holdAppAfterFailedRollback(stack string, replayErr, rollbackEr
m.logger.Printf("[ERROR] [offbox] %s: cannot persist the restore hold — no settings wired; the app is stopped but NOTHING will refuse a restart", stack)
return
}
// R-1340: this branch is only ever an off-site restore's database replay (offbox_reconstitute's rollback).
h := settings.RestoreHold{
Stack: stack,
At: time.Now().UTC().Format(time.RFC3339),
RestoreKind: settings.RestoreKindOffsite,
DBReplay: true,
}
if replayErr != nil {
h.ReplayError = replayErr.Error()
@@ -447,6 +450,15 @@ func (m *Manager) holdAppAfterFailedRollback(stack string, replayErr, rollbackEr
}
}
// mixedHoldWhy (R-1340) is what a restore_mixed hold records about its cause (settings.RestoreHold's R-1340 fields).
type mixedHoldWhy struct {
kind string // settings.RestoreKind*
cause string // settings.MixedCause*
step string // the step that failed, with its error
dbReplay bool // the restore's own failure was its database replay
copyDir string // where the copy of the live volumes is kept; "" = none kept
}
// liveDefinition is the app's definition as it ran before an off-site restore wrote the snapshot's
// (R-893): a copy of its compose files in a temporary folder, and its full env held in memory only.
type liveDefinition struct {
@@ -511,7 +523,11 @@ func (m *Manager) captureLiveDefinition(stack string) (*liveDefinition, error) {
// written back (when the snapshot's was written), the database service is stopped again, and the app is
// held with HoldReasonRestoreMixed. The operator is notified with no rollback error — that is how the
// notification tells this case from R-379's double failure.
func (m *Manager) holdAppAfterMixedRestore(stack string, replayErr error, live *liveDefinition) {
//
// why (R-1340) is what the hold records about its cause, so the operator's mail tells the real story — this hold is
// written by three different branches now (a definition write, a put-back that failed at one of its steps), not
// only by option C's database replay.
func (m *Manager) holdAppAfterMixedRestore(stack string, replayErr error, live *liveDefinition, why mixedHoldWhy) {
// The hold is PERSISTED FIRST (security review 2026-10-08): a controller that dies between the stop and
// the hold would otherwise restart the app from the R-166 marker with nothing refusing it. When the hold
// cannot be persisted, the app-stop marker is KEPT, so nothing reads the window as finished.
@@ -519,7 +535,8 @@ func (m *Manager) holdAppAfterMixedRestore(stack string, replayErr error, live *
if m.settings == nil {
m.logger.Printf("[ERROR] [offbox] %s: cannot persist the restore hold — no settings wired; the app is stopped but NOTHING will refuse a restart", stack)
} else {
h := settings.RestoreHold{Stack: stack, At: time.Now().UTC().Format(time.RFC3339), Reason: settings.HoldReasonRestoreMixed}
h := settings.RestoreHold{Stack: stack, At: time.Now().UTC().Format(time.RFC3339), Reason: settings.HoldReasonRestoreMixed,
RestoreKind: why.kind, MixedCause: why.cause, FailedStep: why.step, DBReplay: why.dbReplay, SafetyDump: why.copyDir}
if replayErr != nil {
h.ReplayError = replayErr.Error()
}
@@ -548,6 +565,26 @@ func (m *Manager) holdAppAfterMixedRestore(stack string, replayErr error, live *
}
}
// restartAtLiveDefinition (R-1256 H4) is the failure branch of the definition write, taken while no data has moved.
// The writer writes the compose before the app's settings, so a failure can leave the snapshot's compose in the
// stack dir: the LIVE definition is written back first, and only then is the app started. When the write-back fails
// too, the app is held stopped (restore_mixed) — a start would run the snapshot's definition over the live data.
// Pinned by TestR1256_H4_*.
func (m *Manager) restartAtLiveDefinition(stack string, live *liveDefinition, cause error, restartStack func() error) {
if live != nil {
if err := m.stackProvider.RecreateStackDefinitionFromUnit(stack, live.dir, live.env); err != nil {
m.logger.Printf("[ERROR] [offbox] %s: the definition write failed (%v) and writing the live definition back failed too: %v — holding the app stopped", stack, cause, err)
m.holdAppAfterMixedRestore(stack, cause, nil, mixedHoldWhy{kind: settings.RestoreKindOffsite, cause: settings.MixedCauseDefinitionWrite,
step: "writing the live app definition back failed: " + err.Error()})
return
}
m.logger.Printf("[INFO] [offbox] %s: the definition write failed (%v); the live definition is written back before the start", stack, cause)
}
if sErr := restartStack(); sErr != nil {
m.logger.Printf("[WARN] [offbox] %s: restart after a failed definition write also failed: %v", stack, sErr)
}
}
// SetRestoreHoldNotify wires the operator notification for a held app (cmd/controller/main.go).
func (m *Manager) SetRestoreHoldNotify(fn func(stack string, replayErr, rollbackErr error)) {
m.restoreHoldNotify = fn
@@ -1005,16 +1042,12 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack
// yet — a failure here restarts the app as it was.
if defineFromSnapshot {
if err := m.stackProvider.RecreateStackDefinitionFromUnit(stack, scratchCompose, snapEnv); err != nil {
if sErr := restartStack(); sErr != nil {
m.logger.Printf("[WARN] [offbox] %s: restart after a failed definition write also failed: %v", stack, sErr)
}
m.restartAtLiveDefinition(stack, liveDef, err, restartStack)
return res, fmt.Errorf("restoring %s: writing the snapshot's definition failed: %w", stack, err)
}
} else if len(credOverlay) > 0 {
if err := m.stackProvider.(secretValueSetter).SetStackSecretValues(stack, credOverlay); err != nil {
if sErr := restartStack(); sErr != nil {
m.logger.Printf("[WARN] [offbox] %s: restart after a failed password write also failed: %v", stack, sErr)
}
m.restartAtLiveDefinition(stack, liveDef, err, restartStack)
return res, fmt.Errorf("restoring %s: writing the backup's database password failed: %w", stack, err)
}
m.logger.Printf("[INFO] [offbox] %s: the backup's database password(s) %v written into the app's settings before the data (R-1106)", stack, credNames(credOverlay))
@@ -1045,7 +1078,7 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack
// R-893 option A: no volume has moved yet, so only the live definition is written back and the app starts.
// The failing copy may have placed part of its files, so the sentence names placed files (+1).
if definitionMoved {
return res, m.putBackAfterFailedRestore(stack, cErr, liveDef, undo, false, res.FilesPlaced+1, restartStack, "err.backup.restore_failed_held_mixed")
return res, m.putBackAfterFailedRestore(stack, cErr, liveDef, undo, false, res.FilesPlaced+1, restartStack, "err.backup.restore_failed_held_mixed", settings.RestoreKindOffsite)
}
// Nothing moved yet: best-effort bring-up — a failed restore must not also be an outage.
if sErr := restartStack(); sErr != nil {
@@ -1080,7 +1113,7 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack
// R-893 option A: the replay removes a volume BEFORE it recreates it, so even nVols == 0 may have lost one —
// with a copy, it is always poured back; with no copy the old rule decides (and holds when one moved).
if definitionMoved || nVols > 0 || undo != nil {
return res, m.putBackAfterFailedRestore(stack, vErr, liveDef, undo, nVols > 0 || undo != nil, res.FilesPlaced, restartStack, "err.backup.restore_failed_held_mixed")
return res, m.putBackAfterFailedRestore(stack, vErr, liveDef, undo, nVols > 0 || undo != nil, res.FilesPlaced, restartStack, "err.backup.restore_failed_held_mixed", settings.RestoreKindOffsite)
}
// Nothing replaced yet: bring the app back up rather than leaving an outage, then surface it.
if sErr := restartStack(); sErr != nil {
@@ -1099,7 +1132,7 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack
if err := m.stackProvider.StartStackServices(stack, dbServices); err != nil {
// R-893 option C (security review 2026-10-08): volumes or the definition already moved → HOLD.
if definitionMoved || res.VolumesReplayed > 0 {
return res, m.putBackAfterFailedRestore(stack, err, liveDef, undo, res.VolumesReplayed > 0, res.FilesPlaced, restartStack, "err.backup.restore_failed_held_mixed")
return res, m.putBackAfterFailedRestore(stack, err, liveDef, undo, res.VolumesReplayed > 0, res.FilesPlaced, restartStack, "err.backup.restore_failed_held_mixed", settings.RestoreKindOffsite)
}
// Best-effort bring-up: a failed restore must not also be an outage.
if sErr := restartStack(); sErr != nil {
@@ -1151,7 +1184,7 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack
// under them — the copy of the live volumes is poured back over the rolled-back database, the live
// definition written back, and the app starts exactly as it was. A put-back that fails holds, as before.
if definitionMoved || res.VolumesReplayed > 0 {
return res, m.putBackAfterFailedRestore(stack, iErr, liveDef, undo, res.VolumesReplayed > 0, res.FilesPlaced, restartStack, "err.backup.db_restore_failed_held_mixed")
return res, m.putBackAfterFailedRestore(stack, iErr, liveDef, undo, res.VolumesReplayed > 0, res.FilesPlaced, restartStack, "err.backup.db_restore_failed_held_mixed", settings.RestoreKindOffsite)
}
if sErr := restartStack(); sErr != nil {
m.logger.Printf("[WARN] [offbox] %s: start after a successful rollback failed: %v", stack, sErr)
@@ -11,6 +11,7 @@ import (
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/dockerexec"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
"gitea.dooplex.hu/admin/felhom-controller/internal/util"
)
@@ -219,28 +220,46 @@ func (m *Manager) putBack(stack string, c *liveVolumeCopy) error {
// failure's own held sentence) and the copy stays on disk. volumesMoved with no copy is such a case: nothing can
// put them back. filesPlaced chooses the household's sentence: placed files are not taken back (as ruled for
// option C, a file mix does not hold), so the sentence then says so instead of „nothing changed".
//
// restoreKind (R-1340) is settings.RestoreKindOffsite or RestoreKindLocal: the hold records it, with the step that
// failed and whether the restore's failure was a database replay (holdKey says so — it is the callers' own choice of
// the household's sentence), so the operator's mail names the real cause.
func (m *Manager) putBackAfterFailedRestore(stack string, cause error, live *liveDefinition, c *liveVolumeCopy, volumesMoved bool,
filesPlaced int, restart func() error, holdKey string) error {
filesPlaced int, restart func() error, holdKey, restoreKind string) error {
why := func(step string) mixedHoldWhy {
w := mixedHoldWhy{kind: restoreKind, cause: settings.MixedCausePutBack, step: step,
dbReplay: holdKey == "err.backup.db_restore_failed_held_mixed"}
if c != nil && c.keep {
w.copyDir = c.dir
}
return w
}
if volumesMoved && c == nil {
m.logger.Printf("[ERROR] [offbox] %s: volumes were replaced and there is no copy to put back — holding the app", stack)
m.holdAppAfterMixedRestore(stack, cause, live)
m.holdAppAfterMixedRestore(stack, cause, live, why("the restore had already changed the app's data and there was no copy of the live volumes to put back"))
return util.MsgError(holdKey, stack)
}
// R-1256 H5: a put-back under an app that could not be stopped could mix the copy with what the app writes.
// Nothing is poured; the copy is kept and the app held (D20: a put-back that fails holds and keeps its copy).
if err := m.stackProvider.StopStack(stack); err != nil {
m.logger.Printf("[WARN] [offbox] %s: stopping the app before the put-back failed: %v", stack, err)
c.keepIfAny()
m.logger.Printf("[ERROR] [offbox] %s: stopping the app before the put-back FAILED: %v — nothing is put back; the copy stays at %s; holding the app", stack, err, c.dirOrNone())
m.holdAppAfterMixedRestore(stack, cause, live, why("stopping the app before the put-back failed, so nothing was put back: "+err.Error()))
return util.MsgError(holdKey, stack)
}
if volumesMoved {
if err := m.putBack(stack, c); err != nil {
c.keep = true
m.logger.Printf("[ERROR] [offbox] %s: putting the live volumes back FAILED: %v — the copy stays at %s; holding the app", stack, err, c.dir)
m.holdAppAfterMixedRestore(stack, cause, live)
m.holdAppAfterMixedRestore(stack, cause, live, why("putting the live volumes back failed: "+err.Error()))
return util.MsgError(holdKey, stack)
}
}
if live != nil {
if err := m.stackProvider.RecreateStackDefinitionFromUnit(stack, live.dir, live.env); err != nil {
m.logger.Printf("[ERROR] [offbox] %s: writing the live definition back FAILED: %v — holding the app", stack, err)
m.holdAppAfterMixedRestore(stack, cause, live)
c.keepIfAny() // R-1256 H5 (D20): a put-back that fails keeps its copy for support
m.logger.Printf("[ERROR] [offbox] %s: writing the live definition back FAILED: %v — the copy stays at %s; holding the app", stack, err, c.dirOrNone())
m.holdAppAfterMixedRestore(stack, cause, live, why("writing the live app definition back failed: "+err.Error()))
return util.MsgError(holdKey, stack)
}
}
@@ -249,12 +268,19 @@ func (m *Manager) putBackAfterFailedRestore(stack string, cause error, live *liv
m.restorePutBackNotify(stack, cause)
}
if err := restart(); err != nil {
m.logger.Printf("[ERROR] [offbox] %s: start after the put-back failed: %v (the copy stays at %s)", stack, err, c.dirOrNone())
m.logger.Printf("[ERROR] [offbox] %s: start after the put-back failed: %v (the live volumes and definition are back; the copy %s is removed)", stack, err, c.dirOrNone())
return util.MsgError("err.backup.restore_failed_put_back_not_started", stack)
}
return putBackMessage(stack, filesPlaced)
}
// keepIfAny marks the copy as kept for support (no copy: nothing to keep).
func (c *liveVolumeCopy) keepIfAny() {
if c != nil {
c.keep = true
}
}
func (c *liveVolumeCopy) dirOrNone() string {
if c == nil {
return "(no volume copy)"
+47
View File
@@ -62,6 +62,53 @@ type RestoreOpStatus struct {
func (m *Manager) BeginRestoreOp(op, stack string) {
m.mu.Lock()
defer m.mu.Unlock()
m.beginRestoreOpLocked(op, stack)
}
// Why a new restore may not begin now (R-1256 H6). "" = it may.
const (
RestoreBusyRestore = "restore" // a restore op is in flight (the record the pages follow)
RestoreBusyBackup = "backup" // the single-flight backup/restore run is held (the night run never records an op)
RestoreBusyMirror = "mirror" // a second-drive copy runs, or one a previous process lost may still run
)
// TryBeginRestoreOp is BeginRestoreOp that refuses — in the SAME step, under m.mu — while another restore op, a
// backup run or a second-drive copy runs. R-1256 H6: the pages asked (RestoreStatus, IsRunning,
// Tier2CopyBlocksRestore) and recorded the op later; a mirror that began in between ran beside the restore. The mirror
// asks under the same m.mu (beginTier2Run), so one of the two always sees the other. Lock order m.mu → tier2Copies,
// as in beginTier2Run. Returns the busy reason ("" = begun) and, for a mirror, its app ("" when unknown).
// Pinned by TestR1256_H6_*.
func (m *Manager) TryBeginRestoreOp(op, stack string) (busy, mirrorApp string) {
m.mu.Lock()
defer m.mu.Unlock()
if busy, app := m.restoreBusyLocked(); busy != "" {
return busy, app
}
m.beginRestoreOpLocked(op, stack)
return "", ""
}
// RestoreBusy is TryBeginRestoreOp's question alone — for a refusal asked early, before a request is read further.
func (m *Manager) RestoreBusy() (busy, mirrorApp string) {
m.mu.Lock()
defer m.mu.Unlock()
return m.restoreBusyLocked()
}
func (m *Manager) restoreBusyLocked() (string, string) {
if m.opRunning {
return RestoreBusyRestore, m.opStack
}
if m.running {
return RestoreBusyBackup, ""
}
if app, blocked := m.Tier2CopyBlocksRestore(); blocked {
return RestoreBusyMirror, app
}
return "", ""
}
func (m *Manager) beginRestoreOpLocked(op, stack string) {
m.opRunning = true
m.opName = op
m.opStack = stack
@@ -0,0 +1,358 @@
package backup
import (
"context"
"errors"
"io"
"log"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-1256 (review round 2, HUNT-1): four restore edges — H4, H5, H6 and H8, each pinned below.
//
// H4: an off-site restore writes the snapshot's definition before any data moves. The writer
// (stackAdapter.RecreateStackDefinitionFromUnit) writes the compose FIRST and the app's settings after, so a
// failure can come back with the snapshot's compose already in the stack dir. The failure branch used to start
// the app at once — on the snapshot's (older) compose, over the live data. Now the live definition is written
// back first; only then is the app started. When the write-back fails too, the app is held, never started.
// r1256DefFailProvider fails the FIRST definition write (the snapshot's), after recording it — as the real
// writer does when its compose write succeeded and its settings write did not.
type r1256DefFailProvider struct {
*r893Provider
failWrites int // how many writes fail, from the first
failFrom int // when set, every write from this one (1-based) on fails
}
func (p *r1256DefFailProvider) RecreateStackDefinitionFromUnit(name, composeDir string, env map[string]string) error {
if err := p.r893Provider.RecreateStackDefinitionFromUnit(name, composeDir, env); err != nil {
return err
}
if len(p.defs) <= p.failWrites || (p.failFrom > 0 && len(p.defs) >= p.failFrom) {
return errors.New("app.yaml write failed")
}
return nil
}
func r1256DefFailFixture(t *testing.T, failWrites int) (*Manager, *r1256DefFailProvider, *int) {
t.Helper()
m, vp, _, notified, _ := r893FixtureA(t, true, 0)
p := &r1256DefFailProvider{r893Provider: vp, failWrites: failWrites}
m.SetStackProvider(p)
return m, p, notified
}
// RED-PROOF: remove the live write-back from the definition-write failure branch (restartStack alone) → one
// definition write (the snapshot's), then the start → FAILS.
func TestR1256_H4_AFailedDefinitionWriteStartsTheAppAtItsLiveDefinition(t *testing.T) {
m, p, notified := r1256DefFailFixture(t, 1)
_, err := m.ReconstituteFromOffsite(context.Background(), "immich", false)
if err == nil {
t.Fatal("a failed definition write must be surfaced")
}
if len(p.defs) != 2 || !strings.Contains(strings.Join(p.defs[1], " "), "postgres:18-alpine") {
t.Fatalf("definition writes %v — want the snapshot's, then the LIVE one written back", p.defs)
}
if !p.fullStarted {
t.Fatalf("the app must run again at its live definition — calls %v", p.calls)
}
// The write-back comes BEFORE the start.
lastRecreate, lastStart := -1, -1
for i, c := range p.calls {
switch c {
case "recreate":
lastRecreate = i
case "start":
lastStart = i
}
}
if lastRecreate > lastStart {
t.Fatalf("the app was started before the live definition was written back — calls %v", p.calls)
}
if held, _ := m.RestoreHoldFor("immich"); held {
t.Fatal("the live definition is back: the app must not be held")
}
if *notified != 0 {
t.Errorf("hold notices %d, want 0", *notified)
}
}
// The write-back fails too: the stack dir may still hold the snapshot's compose, so the app is NOT started; it is
// held (restore_mixed: the definition moved and could not be put back) and the operator is told once.
// RED-PROOF: start the app when the write-back fails → fullStarted → FAILS.
func TestR1256_H4_AFailedWriteBackHoldsTheApp(t *testing.T) {
m, p, notified := r1256DefFailFixture(t, 2)
_, err := m.ReconstituteFromOffsite(context.Background(), "immich", false)
if err == nil {
t.Fatal("a failed definition write must be surfaced")
}
if p.fullStarted {
t.Fatalf("the app was STARTED although its live definition could not be written back — calls %v", p.calls)
}
held, _ := m.RestoreHoldFor("immich")
if !held {
t.Fatal("no hold: every start path would start the app on the snapshot's definition")
}
if k := m.HoldKind("immich"); k != settings.HoldReasonRestoreMixed {
t.Errorf("hold kind %q, want %q", k, settings.HoldReasonRestoreMixed)
}
if *notified != 1 {
t.Errorf("the operator must be told once, got %d", *notified)
}
}
// ---- H5: a put-back whose stop fails must not go on; `volume rm -f` errors count; a failed put-back keeps its copy.
// r1256StopFailProvider fails the Nth StopStack (1-based).
type r1256StopFailProvider struct {
*r893Provider
failAt, n int
}
func (p *r1256StopFailProvider) StopStack(name string) error {
p.n++
err := p.r893Provider.StopStack(name)
if p.n == p.failAt {
return errors.New("compose down: timeout")
}
return err
}
// The off-site put-back: the replay fails after the snapshot's volume replaced the live one; the stop before the
// put-back fails. Pouring the copy back under a running app could mix it with what the app writes, so nothing is
// poured, the copy is KEPT, and the app is held.
// RED-PROOF: restore the „(continuing)" WARN in putBackAfterFailedRestore → the copy is poured and the app started
// → FAILS.
func TestR1256_H5_APutBackWhoseStopFailsDoesNotPourAndKeepsTheCopy(t *testing.T) {
m, vp, rig, _, notified, putBack := r893PutBackFixture(t)
p := &r1256StopFailProvider{r893Provider: vp, failAt: 2} // 1 = the restore's stop, 2 = the put-back's
m.SetStackProvider(p)
_, err := m.ReconstituteFromOffsite(context.Background(), "immich", false)
if err == nil {
t.Fatal("a failed restore must be surfaced")
}
if p.n < 2 {
t.Fatalf("the put-back's stop never ran (stops %d) — the fixture did not reach the put-back", p.n)
}
if len(rig.poured) != 0 || len(rig.removed) != 0 {
t.Fatalf("the copy was poured (%v) / volumes removed (%v) after a FAILED stop", rig.poured, rig.removed)
}
if vp.fullStarted {
t.Fatalf("the app was started after a failed put-back — calls %v", vp.calls)
}
if held, _ := m.RestoreHoldFor("immich"); !held {
t.Fatal("a put-back that could not run must hold the app (D20)")
}
if left := undoCopies(t, rig.undoRoot); len(left) == 0 {
t.Fatal("the copy must be KEPT after a failed put-back (D20)")
}
if *notified != 1 || *putBack != 0 {
t.Errorf("hold notices %d (want 1), put-back notices %d (want 0)", *notified, *putBack)
}
}
// The live definition cannot be written back after the volumes were poured: the app is held and the copy is KEPT
// (D20: a put-back that fails keeps its copy for support).
// RED-PROOF: drop `c.keep = true` from the definition write-back failure branch → the copy is removed → FAILS.
func TestR1256_H5_AFailedDefinitionWriteBackKeepsTheCopy(t *testing.T) {
m, vp, rig, _, _, _ := r893PutBackFixture(t)
// A version change, so a live definition exists to be written back; the second write (the write-back) fails.
if err := os.WriteFile(vp.composePath, []byte(vtCompose("18")), 0o644); err != nil {
t.Fatal(err)
}
vp.livePins = ParseComposeImages(vp.composePath)
p := &r1256DefFailProvider{r893Provider: vp, failWrites: 0}
p.failFrom = 2
m.SetStackProvider(p)
_, err := m.ReconstituteFromOffsite(context.Background(), "immich", false)
if err == nil {
t.Fatal("a failed restore must be surfaced")
}
if len(p.defs) < 2 {
t.Fatalf("the write-back never ran: writes %v", p.defs)
}
if held, _ := m.RestoreHoldFor("immich"); !held {
t.Fatal("a failed write-back must hold the app")
}
if left := undoCopies(t, rig.undoRoot); len(left) == 0 {
t.Fatal("the copy must be KEPT after a failed put-back (D20)")
}
}
// The crash put-back (a restore cut by a power loss): the stop before pouring fails → an error (the guard then
// holds the app and keeps the copy, R-1251 H2), nothing poured.
// RED-PROOF: restore the „(continuing)" WARN in PutBackInterruptedRestore → nil error, the copy poured → FAILS.
func TestR1256_H5_ACrashPutBackWhoseStopFailsRefuses(t *testing.T) {
dir := t.TempDir()
undo := filepath.Join(dir, "backups", "restore-undo", "app", "20261028T000000Z")
mustWrite(t, filepath.Join(undo, "undo.json"), `{"stack":"app","copied":["app_data"],"definition":false}`)
mustWrite(t, filepath.Join(undo, "app_data.tar"), "LIVE")
poured := 0
cp := NewCrashPutBack(func(string) error { return errors.New("compose down: timeout") }, nil, log.New(io.Discard, "", 0))
cp.volReplay = func(string, string) (int, error) { poured++; return 1, nil }
if err := cp.PutBackInterruptedRestore("app", undo); err == nil {
t.Fatal("a put-back whose stop failed must return an error")
}
if poured != 0 {
t.Fatal("the copy was poured back under an app that could not be stopped")
}
if _, err := os.Stat(filepath.Join(undo, "app_data.tar")); err != nil {
t.Fatal("the copy must stay on disk")
}
}
// restoreDockerVolumesFrom: a `docker volume rm -f` that fails (the volume is still in use) used to be ignored; the
// `volume create` that follows is a no-op on an existing volume, so the tar was unpacked ON TOP of the old content —
// a merge. Now that volume counts as failed and is not touched.
// RED-PROOF: ignore the rm error again → the create and the tar run for the volume → FAILS.
func TestR1256_H5_AVolumeThatCannotBeRemovedIsNotMergedInto(t *testing.T) {
bin := t.TempDir()
calls := filepath.Join(bin, "calls.log")
stub := "#!/bin/sh\necho \"$@\" >> " + calls + "\n" +
"case \"$*\" in 'compose version --short') echo v2.29.7;; 'volume rm -f app_busy') echo 'volume is in use' >&2; exit 1;; esac\nexit 0\n"
if err := os.WriteFile(filepath.Join(bin, "docker"), []byte(stub), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH"))
dump := t.TempDir()
for _, n := range []string{"app_busy.tar", "app_free.tar"} {
mustWrite(t, filepath.Join(dump, n), "tar")
}
m := &Manager{logger: log.New(io.Discard, "", 0)}
n, err := m.restoreDockerVolumesFrom("app", dump)
if err == nil || n != 1 {
t.Fatalf("restored %d, err %v — want 1 and an error naming app_busy", n, err)
}
if !strings.Contains(err.Error(), "app_busy") {
t.Errorf("the error must name the volume: %v", err)
}
b, _ := os.ReadFile(calls)
for _, l := range strings.Split(string(b), "\n") {
if (strings.HasPrefix(l, "volume create") || strings.HasPrefix(l, "run ")) && strings.Contains(l, "app_busy") {
t.Fatalf("the volume that could not be removed was written into: %q", l)
}
}
}
// ---- H6: a second-drive mirror must not start between the page's refusal and the restore's own record.
// The page asked (nothing ran), then a mirror began, then the restore began: the begin itself refuses, records
// nothing, and the mirror — asked under the same lock — refuses once a restore is recorded.
// RED-PROOF: make TryBeginRestoreOp record the op without asking (the old BeginRestoreOp) → the restore begins beside
// the mirror → FAILS.
func TestR1256_H6_ARestoreCannotBeginBesideAMirrorThatStartedAfterTheCheck(t *testing.T) {
tier2ResetForTest()
defer tier2ResetForTest()
m, _ := newOffboxManager(t)
if busy, _ := m.RestoreBusy(); busy != "" {
t.Fatalf("fixture: nothing runs, got %q", busy)
}
// the window: the page's check passed; now a mirror begins
if !m.beginTier2Run("immich") {
t.Fatal("fixture: the mirror must begin while no restore is recorded")
}
if busy, app := m.TryBeginRestoreOp("restore", "immich"); busy != RestoreBusyMirror || app != "immich" {
t.Fatalf("the restore began beside a running mirror: busy=%q app=%q", busy, app)
}
if m.RestoreStatus().Running {
t.Fatal("a refused begin left a restore op recorded")
}
tier2CopyEnd("immich")
if busy, _ := m.TryBeginRestoreOp("restore", "immich"); busy != "" {
t.Fatalf("with the mirror done the restore must begin, got %q", busy)
}
if m.beginTier2Run("immich") {
t.Fatal("a mirror began while the restore of the same app is recorded")
}
// a second restore is refused while the first runs; the backup run refuses too
if busy, _ := m.TryBeginRestoreOp("restore", "other"); busy != RestoreBusyRestore {
t.Fatalf("a second restore began beside the first: %q", busy)
}
m.EndRestoreOp(true, "ok")
if err := m.AcquireRunningForTest(); err != nil {
t.Fatal(err)
}
if busy, _ := m.TryBeginRestoreOp("restore", "immich"); busy != RestoreBusyBackup {
t.Fatalf("a restore began beside a backup run: %q", busy)
}
m.ReleaseRunningForTest()
}
// ---- H8: an unused restore copy is cleaned up; a copy a hold or a pending put-back needs is never touched.
func r1256Copy(t *testing.T, m *Manager, app string, age time.Duration, now time.Time) string {
t.Helper()
dir := filepath.Join(restoreUndoDir(m.namespaceRoot(m.systemDataPath), app), now.Add(-age).UTC().Format(restoreCopyStampLayout))
mustWrite(t, filepath.Join(dir, app+"_data.tar"), "LIVE")
return dir
}
func r1256Exists(p string) bool { _, err := os.Stat(p); return err == nil }
// RED-PROOF: drop the hold check (or the marker check) from SweepUnusedRestoreCopies → the held / named copy is removed
// → FAILS; drop the age check → the young copy is removed → FAILS.
func TestR1256_H8_OnlyAnUnusedOldCopyIsRemoved(t *testing.T) {
m, sett := newOffboxManager(t)
now := time.Now()
old := 40 * 24 * time.Hour
unused := r1256Copy(t, m, "unused", old, now)
young := r1256Copy(t, m, "young", 24*time.Hour, now)
held := r1256Copy(t, m, "held", old, now)
waiting := r1256Copy(t, m, "waiting", old, now)
named := r1256Copy(t, m, "named", old, now)
if err := sett.SetRestoreHold(settings.RestoreHold{Stack: "held", At: now.Format(time.RFC3339), Reason: settings.HoldReasonRestoreMixed}); err != nil {
t.Fatal(err)
}
if err := sett.SetRestoreHold(settings.RestoreHold{Stack: "waiting", At: now.Format(time.RFC3339), Reason: settings.HoldReasonPutBackPending}); err != nil {
t.Fatal(err)
}
// a restore cut by a power loss: the marker (no longer active after a failed recovery's End) still names the copy
if err := m.appStop.Begin("unit-restore:named", ReasonUnitRestore, []string{"named"}); err != nil {
t.Fatal(err)
}
if err := m.appStop.SetUndo("named", named); err != nil {
t.Fatal(err)
}
got := m.SweepUnusedRestoreCopies(now)
if len(got) != 1 || got[0] != unused || r1256Exists(unused) {
t.Fatalf("removed %v — want exactly the unused old copy %s", got, unused)
}
for _, keep := range []string{young, held, waiting, named} {
if !r1256Exists(keep) {
t.Errorf("a needed copy was removed: %s", keep)
}
}
if r1256Exists(filepath.Dir(unused)) {
t.Error("the emptied per-app folder must go too")
}
}
// While a restore runs nothing is removed; a marker that cannot be read stops the sweep.
// RED-PROOF: drop the RestoreBusy check → the old copy goes during a restore → FAILS.
func TestR1256_H8_NothingIsRemovedDuringARestoreOrWithAnUnreadableMarker(t *testing.T) {
m, _ := newOffboxManager(t)
now := time.Now()
c := r1256Copy(t, m, "app", 40*24*time.Hour, now)
m.BeginRestoreOp("restore", "other")
if got := m.SweepUnusedRestoreCopies(now); len(got) != 0 || !r1256Exists(c) {
t.Fatalf("removed %v during a restore", got)
}
m.EndRestoreOp(true, "ok")
mustWrite(t, m.appStop.path, "{not json")
if got := m.SweepUnusedRestoreCopies(now); len(got) != 0 || !r1256Exists(c) {
t.Fatalf("removed %v with an unreadable marker", got)
}
if err := os.Remove(m.appStop.path); err != nil {
t.Fatal(err)
}
if got := m.SweepUnusedRestoreCopies(now); len(got) != 1 {
t.Fatalf("control: with nothing in the way the old copy goes, removed %v", got)
}
}
@@ -0,0 +1,133 @@
package backup
import (
"context"
"errors"
"io"
"log"
"os"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-1340: a restore_mixed hold records WHAT put it in place — which restore, which branch, which put-back step failed,
// whether the restore's failure was a database replay, and where the copy is kept — so the operator's mail
// (cmd/controller heldStoppedOperatorText, pinned by TestR1340_* there) tells the real story. Measured 2026-10-29 on
// Tester 1: Radicale (no database), its volume could not be removed, the put-back's stop failed, and the mail said its
// DATABASE restore failed and was rolled back.
//
// RED-PROOF (2026-10-29): with the R-1340 fields dropped from holdAppAfterMixedRestore's hold literal and from
// holdPutBackFailed's (the pre-R-1340 record, which carried no cause), all five tests below fail on „mixed cause """ /
// „kind """.
func holdOf(t *testing.T, m *Manager, stack string) settings.RestoreHold {
t.Helper()
if m.settings == nil {
t.Fatal("fixture: no settings wired")
}
h, ok := m.settings.GetRestoreHold(stack)
if !ok {
t.Fatalf("%s is not held", stack)
}
return h
}
// The Radicale shape: a VOLUME restore fails (no database involved), then the put-back's stop fails.
func TestR1340_AVolumeFailureWhosePutBackStopFailsRecordsTheStepAndNoDatabase(t *testing.T) {
m, vp, rig, _, notified, _ := r893PutBackFixture(t)
rig.snapReplayErr = errors.New("volume rm -f immich_pgdata: volume is in use")
p := &r1256StopFailProvider{r893Provider: vp, failAt: 2} // 1 = the restore's stop, 2 = the put-back's
m.SetStackProvider(p)
if _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false); err == nil {
t.Fatal("a failed restore must be surfaced")
}
h := holdOf(t, m, "immich")
if h.Reason != settings.HoldReasonRestoreMixed || h.MixedCause != settings.MixedCausePutBack {
t.Fatalf("reason %q, mixed cause %q — want restore_mixed / put_back", h.Reason, h.MixedCause)
}
if !strings.Contains(h.FailedStep, "stopping the app before the put-back failed") || !strings.Contains(h.FailedStep, "compose down: timeout") {
t.Errorf("failed step %q — want the stop and its error", h.FailedStep)
}
if h.DBReplay {
t.Error("a volume failure must not be recorded as a database replay")
}
if h.RestoreKind != settings.RestoreKindOffsite {
t.Errorf("restore kind %q, want offsite", h.RestoreKind)
}
if h.SafetyDump == "" {
t.Error("the kept copy's folder must be recorded")
} else if _, err := os.Stat(h.SafetyDump); err != nil {
t.Errorf("the recorded copy %s is not on disk: %v", h.SafetyDump, err)
}
if *notified != 1 {
t.Errorf("hold notices %d, want 1", *notified)
}
}
// The database replay fails (option C's shape), then the put-back's stop fails: DBReplay is recorded.
func TestR1340_ADatabaseReplayFailureRecordsTheDatabase(t *testing.T) {
m, vp, _, _, _, _ := r893PutBackFixture(t)
m.SetStackProvider(&r1256StopFailProvider{r893Provider: vp, failAt: 2})
if _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false); err == nil {
t.Fatal("a failed restore must be surfaced")
}
h := holdOf(t, m, "immich")
if h.MixedCause != settings.MixedCausePutBack || !h.DBReplay {
t.Fatalf("mixed cause %q, db replay %v — want put_back and true", h.MixedCause, h.DBReplay)
}
}
// R-1256 H4: the backup's definition cannot be written and the live one cannot be written back — no data moved.
func TestR1340_ADefinitionWriteFailureRecordsItsCause(t *testing.T) {
m, _, _ := r1256DefFailFixture(t, 2)
if _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false); err == nil {
t.Fatal("a failed definition write must be surfaced")
}
h := holdOf(t, m, "immich")
if h.MixedCause != settings.MixedCauseDefinitionWrite || h.DBReplay {
t.Fatalf("mixed cause %q, db replay %v — want definition_write and false", h.MixedCause, h.DBReplay)
}
if !strings.Contains(h.FailedStep, "writing the live app definition back failed") {
t.Errorf("failed step %q", h.FailedStep)
}
}
// A LOCAL restore (from the app's own backup) records itself as local, never off-site.
func TestR1340_ALocalRestoreRecordsItsKind(t *testing.T) {
m, _, rig, _ := r1177Fixture(t)
if m.settings == nil {
sett, err := settings.Load(t.TempDir()+"/settings.json", log.New(io.Discard, "", 0))
if err != nil {
t.Fatal(err)
}
m.settings = sett
}
rig.unitErr = errors.New("tar: short read")
rig.putBackErr = errors.New("tar xf failed")
if _, err := m.RestoreFromRecoveryUnit("app"); err == nil {
t.Fatal("a failed put-back must be surfaced")
}
h := holdOf(t, m, "app")
if h.RestoreKind != settings.RestoreKindLocal || h.MixedCause != settings.MixedCausePutBack || h.DBReplay {
t.Fatalf("kind %q, cause %q, db %v — want local / put_back / false", h.RestoreKind, h.MixedCause, h.DBReplay)
}
if !strings.Contains(h.FailedStep, "putting the live volumes back failed") || !strings.Contains(h.FailedStep, "tar xf failed") {
t.Errorf("failed step %q", h.FailedStep)
}
}
// A restore cut by a stop whose put-back fails at the next start (R-1251 H2) records its own cause.
func TestR1340_AnInterruptedPutBackRecordsItsCause(t *testing.T) {
r := newR1251Rig(t)
r.putErr = errors.New("compose down: timeout")
r.guard().Recover()
h, ok := r.sett.GetRestoreHold("immich")
if !ok || h.Reason != settings.HoldReasonRestoreMixed || h.MixedCause != settings.MixedCauseInterruptedPutBack {
t.Fatalf("hold %+v (held %v) — want restore_mixed / interrupted_put_back", h, ok)
}
if !strings.Contains(h.FailedStep, "compose down: timeout") {
t.Errorf("failed step %q", h.FailedStep)
}
}
@@ -0,0 +1,122 @@
package backup
import (
"errors"
"io"
"log"
"path/filepath"
"sync/atomic"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// r1342Drive registers the rig's app drive, as every drive is registered on a box, so the list and the delete see it.
func r1342Drive(t *testing.T, m *Manager) {
t.Helper()
if m.settings == nil {
sett, err := settings.Load(filepath.Join(t.TempDir(), "settings.json"), log.New(io.Discard, "", 0))
if err != nil {
t.Fatal(err)
}
m.settings = sett
}
if err := m.settings.AddStoragePath(settings.StoragePath{Path: m.GetAppDrivePath("app"), Label: "rig"}); err != nil {
t.Fatal(err)
}
}
// R-1342 (found live on Tester 1, 2026-10-29 10:39 CET): (1) while the box still watches whether a restored app runs
// stably, its restore copy is the way back — „Törlés most" must not take it (D62: never while an operation runs on the
// app); (2) an app REMOVED during that watch is not „did not start" — no operator alarm, no „the copy is kept".
// removableProvider is the stable-watch rig's provider that can answer StackRemoved.
type removableProvider struct {
*sampledProvider
removed atomic.Bool
}
func (p *removableProvider) StackRemoved(string) bool { return p.removed.Load() }
// RED-PROOF: drop the watching check from restoreCopyNeededFor / DeleteRestoreCopy → the copy is offered and deleted
// mid-watch → FAILS.
func TestR1342_ACopyUnderTheStableWatchIsNotDeletable(t *testing.T) {
withStable(t, 400*time.Millisecond, 3*time.Second)
m, rig, alarms := fixARig(t, func(int) bool { return true },
func(int) []ContainerSample { return []ContainerSample{healthy("c1", 0)} })
r1342Drive(t, m)
m.BeginRestoreOp("restore", "app")
if _, err := m.RestoreFromRecoveryUnit("app"); err != nil {
t.Fatal(err)
}
m.EndRestoreOp(true, "ok")
copies := m.RestoreCopies()
if len(copies) != 1 || copies[0].Needed != RestoreCopyNeededChecking || !copies[0].DeleteOn.IsZero() {
t.Fatalf("mid-watch the copy must read „checking\", got %+v", copies)
}
if _, err := m.DeleteRestoreCopy("app", copies[0].Stamp); !errors.Is(err, ErrRestoreCopyNeeded) {
t.Fatalf("a delete mid-watch: %v, want ErrRestoreCopyNeeded", err)
}
if left := undoCopies(t, rig.undoRoot); len(left) != 1 {
t.Fatalf("the copy must stay through the watch, found %v", left)
}
if left := waitNoCopies(t, rig.undoRoot); len(left) != 0 {
t.Fatalf("after the stable window the watch releases it, found %v", left)
}
if len(*alarms) != 0 {
t.Fatalf("no alarm: %v", *alarms)
}
if m.watchingCopy(filepath.Join(rig.undoRoot, copies[0].Stamp)) {
t.Fatal("the watch mark must go when the watch ends")
}
}
// RED-PROOF: drop the stableRemoved case → the removed app „did not run stably" → one alarm → FAILS.
func TestR1342_AnAppRemovedDuringTheWatchRaisesNoAlarm(t *testing.T) {
withStable(t, 2*time.Second, 400*time.Millisecond)
var up atomic.Bool
up.Store(true)
m, rig, alarms := fixARig(t, func(int) bool { return up.Load() },
func(int) []ContainerSample { return []ContainerSample{healthy("c1", 0)} })
rp := &removableProvider{sampledProvider: m.stackProvider.(*sampledProvider)}
m.stackProvider = rp
r1342Drive(t, m)
m.BeginRestoreOp("restore", "app")
if _, err := m.RestoreFromRecoveryUnit("app"); err != nil {
t.Fatal(err)
}
m.EndRestoreOp(true, "ok")
// the household removes the app inside the watch
up.Store(false)
rp.removed.Store(true)
time.Sleep(700 * time.Millisecond)
if len(*alarms) != 0 {
t.Fatalf("an app removed during the watch must not alarm „did not start\": %v", *alarms)
}
copies := m.RestoreCopies()
if len(copies) != 1 || copies[0].Needed != "" {
t.Fatalf("after the watch ended the copy is an unused one (sweep / Törlés most): %+v", copies)
}
_ = rig
}
// Control: an app that stays installed and never runs stably still alarms once (the R-1270 rule is unchanged).
func TestR1342_AnInstalledUnstableAppStillAlarms(t *testing.T) {
withStable(t, 2*time.Second, 400*time.Millisecond)
var up atomic.Bool
up.Store(true)
m, _, alarms := fixARig(t, func(int) bool { return up.Load() },
func(int) []ContainerSample { return []ContainerSample{healthy("c1", 0)} })
m.stackProvider = &removableProvider{sampledProvider: m.stackProvider.(*sampledProvider)}
m.BeginRestoreOp("restore", "app")
if _, err := m.RestoreFromRecoveryUnit("app"); err != nil {
t.Fatal(err)
}
m.EndRestoreOp(true, "ok")
up.Store(false)
time.Sleep(700 * time.Millisecond)
if len(*alarms) != 1 {
t.Fatalf("an installed app that never runs stably alarms once, got %v", *alarms)
}
}
@@ -0,0 +1,53 @@
package backup
import (
"io"
"log"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-1347: a restore that lifts an UPDATE hold must also forget the stack manager's in-memory „last update ended held"
// flag. Left set, the crash-loop watcher does not watch the app and the night leg skips it until the controller restarts
// (D64's sibling: the operator's clear_hold already forgets it, R-1343; the restore did not).
//
// The consequence on the real flag — ObserveUnhealthy and legCandidate — is TestR1347_RestoreLiftsTheWatchersBlindSpot
// (cmd/controller), and the seam's wiring TestR1347_UpdateHeldForgetterIsWiredAtStartup.
//
// RED-PROOF (2026-11-02): seen failing before the seam existed — the build failed on SetUpdateHeldForgetter; with the
// call dropped from clearUpdateHoldAfterRestore it fails on „the flag was not forgotten".
func TestR1347_ARestoreThatLiftsAnUpdateHoldForgetsTheHeldFlag(t *testing.T) {
sett := slice4Settings(t)
m := &Manager{logger: log.New(io.Discard, "", 0), settings: sett}
forgot := map[string]int{}
m.SetUpdateHeldForgetter(func(s string) bool { forgot[s]++; return true })
_ = m.HoldAfterFailedUpdate("nextcloud", time.Now(), time.Now(), UpdateTierLocal)
m.clearUpdateHoldAfterRestore("nextcloud")
if _, held := sett.GetRestoreHold("nextcloud"); held {
t.Fatal("the update hold must be lifted")
}
if forgot["nextcloud"] != 1 {
t.Fatalf("the restore lifted the update hold, but the flag was not forgotten (calls: %v)", forgot)
}
// A RESTORE hold (R-379) stays, so nothing about it is forgotten.
_ = sett.SetRestoreHold(settings.RestoreHold{Stack: "rst", At: "2026-08-22T14:00:00Z"})
m.clearUpdateHoldAfterRestore("rst")
if _, held := sett.GetRestoreHold("rst"); !held || forgot["rst"] != 0 {
t.Fatalf("a restore hold is not lifted by a restore (held %v, forget calls %d)", held, forgot["rst"])
}
// No hold at all: nothing to forget (a plain restore of a healthy app).
m.clearUpdateHoldAfterRestore("plain")
if forgot["plain"] != 0 {
t.Fatalf("no hold was lifted, yet the flag was touched %d time(s)", forgot["plain"])
}
// nil seam: no crash (the pre-R-1347 behaviour).
m2 := &Manager{logger: log.New(io.Discard, "", 0), settings: sett}
_ = m2.HoldAfterFailedUpdate("other", time.Now(), time.Now(), UpdateTierLocal)
m2.clearUpdateHoldAfterRestore("other")
}
@@ -0,0 +1,75 @@
package backup
import (
"errors"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-1348: the restore_mixed hold written at START-UP after an interrupted put-back fails must reach the operator as the
// R-1340 mail. Recover runs long before the notifier exists (main.go), so the guard keeps the name and hands it over when
// SetHeldNotify is wired; a failure after that (a drive that returned at run time) goes out at once.
//
// RED-PROOF (2026-11-02): seen failing before the seam existed (build: SetHeldNotify); with the call dropped from
// holdPutBackFailed these fail on „no mail for a held app".
func TestR1348_AFailedPutBackAtStartupMailsOnceTheNotifierIsWired(t *testing.T) {
r := newR1251Rig(t)
r.putErr = errors.New("compose down: timeout")
g := r.guard()
if res := g.Recover(); res == nil || len(res.Held) != 1 {
t.Fatalf("recovery must hold the app: %+v", res)
}
var mailed []string
g.SetHeldNotify(func(stack string) { mailed = append(mailed, stack) })
if len(mailed) != 1 || mailed[0] != "immich" {
t.Fatalf("no mail for a held app: the hold from Recover must be handed over when the notifier is wired, got %v", mailed)
}
h, ok := r.sett.GetRestoreHold("immich")
if !ok || h.Reason != settings.HoldReasonRestoreMixed || h.MixedCause != settings.MixedCauseInterruptedPutBack ||
!strings.Contains(h.FailedStep, "compose down: timeout") {
t.Fatalf("the mail reads this record, and it is wrong: %+v", h)
}
g.SetHeldNotify(func(stack string) { mailed = append(mailed, "again:"+stack) })
if len(mailed) != 1 {
t.Fatalf("a held app is mailed once, not at every wiring: %v", mailed)
}
}
// A clean put-back, or one still waiting for its drive, is no failure: no mail.
func TestR1348_NoMailWithoutAFailedPutBack(t *testing.T) {
r := newR1251Rig(t)
g := r.guard()
g.Recover()
var mailed []string
g.SetHeldNotify(func(stack string) { mailed = append(mailed, stack) })
if len(mailed) != 0 {
t.Fatalf("a put-back that worked mailed: %v", mailed)
}
w := newR1251Rig(t)
w.unbind(t)
gw := w.guard()
gw.Recover()
gw.SetHeldNotify(func(stack string) { mailed = append(mailed, stack) })
if len(mailed) != 0 {
t.Fatalf("a put-back that waits for its drive has its own hold and reminder, not this mail: %v", mailed)
}
}
// After the wiring a failed put-back (the drive returned at run time: ResumePutBack) mails at once.
func TestR1348_AFailedResumeMailsAtOnce(t *testing.T) {
r := newR1251Rig(t)
r.unbind(t)
g := r.guard()
g.Recover() // waiting for its drive
var mailed []string
g.SetHeldNotify(func(stack string) { mailed = append(mailed, stack) })
r.bind(t)
r.putErr = errors.New("tar xf failed")
g.ResumePutBack("immich")
if len(mailed) != 1 || mailed[0] != "immich" {
t.Fatalf("a put-back that failed on resume must mail: %v", mailed)
}
}
+8 -2
View File
@@ -224,8 +224,14 @@ func (m *Manager) restoreDockerVolumesFrom(stackName, dumpDir string) (int, erro
m.logger.Printf("[INFO] [backup] Restoring Docker volume %s for %s", volName, stackName)
// Remove existing volume (ignore errors — may not exist)
dockerexec.Command("docker", "volume", "rm", "-f", volName).Run()
// Remove the existing volume. `rm -f` of a missing volume succeeds, so an error means the volume EXISTS and
// could not be removed (still in use). R-1256 H5: it used to be ignored — the `volume create` below is a no-op
// on an existing volume, and the tar was unpacked ON TOP of the old content (a merge). That volume fails now.
if out, err := dockerexec.Command("docker", "volume", "rm", "-f", volName).CombinedOutput(); err != nil {
m.logger.Printf("[ERROR] [backup] Failed to remove volume %s before the restore: %s — %v (not written into)", volName, strings.TrimSpace(string(out)), err)
failed = append(failed, volName)
continue
}
// Create fresh volume — WITH the labels compose gives its own volumes (R-658, v0.268.0). A bare
// `volume create` made a volume the update's undo (until v0.267.0) could not find, and one
@@ -0,0 +1,216 @@
package backup
import (
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"sort"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// D62 (operator ruling 2026-10-29; `07` §6.3): the household sees its restore copies in the kept-copies list
// („Megőrzött másolatok") on the restore page — the app, the date of the restore the copy protects, its size, and the
// day the box deletes it by itself (restoreCopyUnusedAge after the restore) — and may delete one earlier with
// „Törlés most", after a warning and the app's name typed.
//
// A copy a hold or a pending put-back needs is NEVER deletable (the same three states the sweep leaves alone, R-1256
// H8): the row says why in one line and the button is off; the handler refuses it too. While any backup or restore
// operation runs on the box, a delete is refused (409). The check and the removal are ONE step under m.mu, so a
// restore cannot begin in between (the R-1256 H6 shape). Pinned by TestD62_*.
// Why a restore copy is needed — the row's one line, and the refusal's reason. "" = not needed: deletable.
const (
RestoreCopyNeededHold = "hold" // the app is held (a failed put-back keeps its copy, D20) — support decides
RestoreCopyNeededPutBack = "putback" // a put-back waits for its drive (R-1251) or a cut restore names it (D22)
RestoreCopyNeededUnknown = "unknown" // the app-stop marker cannot be read: nothing is offered (fail closed)
// RestoreCopyNeededChecking (R-1342): the box still watches whether the restored app runs stably (up to 30 minutes
// after the restore, R-1270) — the copy is the way back if it does not.
RestoreCopyNeededChecking = "checking"
)
var (
ErrRestoreCopyNotFound = errors.New("restore copy not found")
ErrRestoreCopyNeeded = errors.New("restore copy is needed by a hold or a pending put-back")
ErrRestoreCopyBusy = errors.New("a backup or restore operation runs")
)
// RestoreCopy is one restore copy (<namespace>/backups/restore-undo/<app>/<stamp>).
type RestoreCopy struct {
App string
Stamp string // the folder's name; with App, the copy's identity on the page
At time.Time // when the restore began (the stamp; the folder's time when the stamp does not parse)
SizeBytes int64
Path string
Needed string // "" or one of RestoreCopyNeeded*
DeleteOn time.Time // when the sweep removes it; zero when Needed != "" (a needed copy is never swept)
}
// restoreCopyAt is a copy's restore time: its stamp, or the folder's time when the stamp does not parse.
func restoreCopyAt(dir, name string) (time.Time, bool) {
if at, err := time.Parse(restoreCopyStampLayout, name); err == nil {
return at, true
}
fi, err := os.Stat(dir)
if err != nil {
return time.Time{}, false
}
return fi.ModTime(), true
}
// restoreCopyNeededFor is why app's copies are needed ("" = deletable). named is appStopNamedApps' answer, ok its ok.
func (m *Manager) restoreCopyNeededFor(app string, named map[string]bool, ok bool) string {
if !ok || m.settings == nil {
return RestoreCopyNeededUnknown
}
if named[app] {
return RestoreCopyNeededPutBack
}
if h, held := m.settings.GetRestoreHold(app); held {
if h.Reason == settings.HoldReasonPutBackPending {
return RestoreCopyNeededPutBack
}
return RestoreCopyNeededHold
}
return ""
}
func dirSize(dir string) int64 {
var n int64
_ = filepath.WalkDir(dir, func(_ string, d fs.DirEntry, err error) error {
if err == nil && d.Type().IsRegular() {
if fi, e := d.Info(); e == nil {
n += fi.Size()
}
}
return nil
})
return n
}
// RestoreCopies lists every restore copy on the box's drives, oldest first.
func (m *Manager) RestoreCopies() []RestoreCopy {
named, ok := m.appStopNamedApps()
var out []RestoreCopy
for _, nsRoot := range m.primaryUnitRoots() {
base := filepath.Join(nsRoot, "backups", "restore-undo")
apps, err := os.ReadDir(base)
if err != nil {
continue
}
for _, a := range apps {
if !a.IsDir() {
continue
}
app := a.Name()
needed := m.restoreCopyNeededFor(app, named, ok)
copies, err := os.ReadDir(filepath.Join(base, app))
if err != nil {
continue
}
for _, c := range copies {
if !c.IsDir() {
continue
}
dir := filepath.Join(base, app, c.Name())
at, ok := restoreCopyAt(dir, c.Name())
if !ok {
continue
}
rc := RestoreCopy{App: app, Stamp: c.Name(), At: at, SizeBytes: dirSize(dir), Path: dir, Needed: needed}
if rc.Needed == "" && m.watchingCopy(dir) {
rc.Needed = RestoreCopyNeededChecking
}
if rc.Needed == "" {
rc.DeleteOn = at.Add(restoreCopyUnusedAge)
}
out = append(out, rc)
}
}
}
sort.SliceStable(out, func(i, j int) bool { return out[i].At.Before(out[j].At) })
return out
}
// validCopyName is a single path element: no separator, not empty, not "." or "..".
func validCopyName(s string) bool {
return s != "" && s != "." && s != ".." && !strings.ContainsAny(s, `/\`) && filepath.Base(s) == s
}
// DeleteRestoreCopy removes ONE restore copy, named by its app and its folder's name, and nothing else (the path
// guard: both names are single path elements, the folder is a real directory — not a link — directly under
// <namespace>/backups/restore-undo/<app>/). Refused while any backup or restore operation runs (ErrRestoreCopyBusy)
// and for a copy a hold or a pending put-back needs (ErrRestoreCopyNeeded). The check and the removal hold m.mu, so
// no restore can begin in between. Returns the copy as it was (for the log line and the operator event).
func (m *Manager) DeleteRestoreCopy(app, stamp string) (RestoreCopy, error) {
if !validCopyName(app) || !validCopyName(stamp) {
return RestoreCopy{}, ErrRestoreCopyNotFound
}
m.mu.Lock()
defer m.mu.Unlock()
if busy, _ := m.restoreBusyLocked(); busy != "" {
return RestoreCopy{}, ErrRestoreCopyBusy
}
var found *RestoreCopy
for _, nsRoot := range m.primaryUnitRoots() {
appDir := filepath.Join(nsRoot, "backups", "restore-undo", app)
dir := filepath.Join(appDir, stamp)
fi, err := os.Lstat(dir)
if err != nil || !fi.IsDir() || fi.Mode()&os.ModeSymlink != 0 {
continue
}
if filepath.Dir(dir) != appDir || filepath.Dir(appDir) != filepath.Join(nsRoot, "backups", "restore-undo") {
continue
}
at, ok := restoreCopyAt(dir, stamp)
if !ok {
continue
}
found = &RestoreCopy{App: app, Stamp: stamp, At: at, SizeBytes: dirSize(dir), Path: dir}
break
}
if found == nil {
return RestoreCopy{}, ErrRestoreCopyNotFound
}
named, ok := m.appStopNamedApps()
if found.Needed = m.restoreCopyNeededFor(app, named, ok); found.Needed != "" {
return *found, ErrRestoreCopyNeeded
}
if m.watching[found.Path] { // R-1342: under m.mu, the same lock the watch marks its copy under
found.Needed = RestoreCopyNeededChecking
return *found, ErrRestoreCopyNeeded
}
if err := os.RemoveAll(found.Path); err != nil {
return *found, fmt.Errorf("removing the restore copy %s: %w", found.Path, err)
}
_ = os.Remove(filepath.Dir(found.Path)) // the per-app folder, when it is now empty
return *found, nil
}
// setWatching marks (on) or clears a copy a stable-start watch holds (R-1342).
func (m *Manager) setWatching(dir string, on bool) {
if dir == "" {
return
}
m.mu.Lock()
defer m.mu.Unlock()
if on {
if m.watching == nil {
m.watching = map[string]bool{}
}
m.watching[dir] = true
return
}
delete(m.watching, dir)
}
// watchingCopy reports whether a stable-start watch still holds the copy.
func (m *Manager) watchingCopy(dir string) bool {
m.mu.Lock()
defer m.mu.Unlock()
return m.watching[dir]
}
@@ -0,0 +1,130 @@
package backup
import (
"encoding/json"
"os"
"path/filepath"
"sort"
"time"
)
// R-1256 H8 (review round 2, HUNT-1): a restore copy (<namespace>/backups/restore-undo/<app>/<stamp>) that nothing
// will ever use again used to stay on the drive for good. A copy is KEPT on purpose in three states — a put-back that
// failed (the app is held, D20), a put-back waiting for its drive (the putback_pending hold, R-1251) and a restore cut
// by a power loss (the app-stop marker names it, D22) — and for 30 minutes after a restore while the box watches for a
// stable start (R-1270). Every one of those is still needed. A copy outside all of them — the app runs again, or was
// removed, and nothing names the copy — is disk the household cannot see or use.
//
// The rule (the age: operator ruling D62, 2026-10-29 — 30 days, and the household may delete a copy earlier with
// „Törlés most", restore_copies.go): a copy is removed only
// when ALL of these hold, and the sweep removes NOTHING when it cannot answer one of them:
// - it is older than restoreCopyUnusedAge (its folder's stamp; the folder's time when the stamp does not parse);
// - the app has no restore hold of any kind (held apps keep every copy — support decides);
// - the app-stop marker does not name the app (a pending put-back), and the marker is readable;
// - no restore op is in flight on the box (a new restore's copy and the stable-start watcher are left alone).
//
// Run every 6 hours from main.go (TestR1256_H8_TheSweepIsWiredAtStartup). Pinned by TestR1256_H8_*.
// restoreCopyUnusedAge is how long an unused copy stays after its restore (D62: 30 days; TestD62_AnUnusedCopyIsKept30Days).
const restoreCopyUnusedAge = 30 * 24 * time.Hour
// restoreCopyStampLayout is copyLiveVolumes' folder name.
const restoreCopyStampLayout = "20060102T150405Z"
// SweepUnusedRestoreCopies removes the restore copies no hold, no pending put-back and no running restore needs, and
// older than restoreCopyUnusedAge. Returns the folders removed.
func (m *Manager) SweepUnusedRestoreCopies(now time.Time) []string {
if busy, _ := m.RestoreBusy(); busy == RestoreBusyRestore {
m.logger.Printf("[INFO] [backup] restore-copy sweep skipped: a restore runs")
return nil
}
named, ok := m.appStopNamedApps()
if !ok {
m.logger.Printf("[WARN] [backup] restore-copy sweep skipped: the app-stop marker cannot be read — a copy it names must not go")
return nil
}
var removed []string
for _, nsRoot := range m.primaryUnitRoots() {
base := filepath.Join(nsRoot, "backups", "restore-undo")
apps, err := os.ReadDir(base)
if err != nil {
continue
}
for _, a := range apps {
if !a.IsDir() {
continue
}
app := a.Name()
if named[app] {
continue
}
if m.settings == nil {
return removed // no hold store: a held app cannot be told apart
}
if _, held := m.settings.GetRestoreHold(app); held {
continue
}
appDir := filepath.Join(base, app)
copies, err := os.ReadDir(appDir)
if err != nil {
continue
}
sort.Slice(copies, func(i, j int) bool { return copies[i].Name() < copies[j].Name() })
for _, c := range copies {
if !c.IsDir() {
continue
}
dir := filepath.Join(appDir, c.Name())
at, err := time.Parse(restoreCopyStampLayout, c.Name())
if err != nil {
fi, sErr := os.Stat(dir)
if sErr != nil {
continue
}
at = fi.ModTime()
}
if now.Sub(at) < restoreCopyUnusedAge {
continue
}
if err := os.RemoveAll(dir); err != nil {
m.logger.Printf("[WARN] [backup] %s: the unused restore copy %s could not be removed: %v", app, dir, err)
continue
}
m.logger.Printf("[INFO] [backup] %s: the unused restore copy %s (from %s) is removed — no hold, no pending put-back, older than %s (R-1256)",
app, dir, at.UTC().Format(time.RFC3339), restoreCopyUnusedAge)
removed = append(removed, dir)
}
_ = os.Remove(appDir) // the per-app folder, when it is now empty
}
}
return removed
}
// appStopNamedApps reads the app-stop marker without side effects: the apps of an ACTIVE marker and every app its Undo
// names. ok=false when the marker exists and cannot be read — the caller then touches nothing.
func (m *Manager) appStopNamedApps() (map[string]bool, bool) {
named := map[string]bool{}
if m.appStop == nil || m.appStop.path == "" {
return named, true
}
data, err := os.ReadFile(m.appStop.path)
if os.IsNotExist(err) {
return named, true
}
if err != nil {
return nil, false
}
var mk AppStopMarker
if err := json.Unmarshal(data, &mk); err != nil {
return nil, false
}
for app := range mk.Undo {
named[app] = true
}
if mk.Active {
for _, app := range mk.Stacks {
named[app] = true
}
}
return named, true
}
@@ -195,9 +195,11 @@ func (p *CrashPutBack) PutBackInterruptedRestore(stack, undoDir string) error {
if j.Stack != stack {
return fmt.Errorf("the restore copy at %s belongs to %q, not %q", undoDir, j.Stack, stack)
}
// R-1256 H5: never pour the copy back under an app that could not be stopped — a refusal here is a failed
// put-back: the guard holds the app and keeps the copy (R-1251 H2).
if p.stop != nil {
if err := p.stop(stack); err != nil {
p.logger.Printf("[WARN] [appstop] %s: stopping before the put-back failed: %v (continuing)", stack, err)
return fmt.Errorf("stopping %s before the put-back: %w", stack, err)
}
}
mgr := &Manager{logger: p.logger, volumeReplayFrom: p.volReplay, undoVolumeRemoveFn: p.volRemove}
+2 -1
View File
@@ -3,6 +3,7 @@ package backup
import (
"errors"
"fmt"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
"gitea.dooplex.hu/admin/felhom-controller/internal/util"
"os"
"path/filepath"
@@ -530,7 +531,7 @@ func (m *Manager) RestoreFromRecoveryUnitAtWith(stackName, unitDir string, opt U
// without one (the unit holds no volume) only a database replay that began can have changed data — and with no
// copy to pour back that is the R-379 half state: held, never started (operator ruling 2026-08-22).
putBack := func(cause error, holdKey string, moved bool) (UnitRestoreResult, error) {
return res, m.putBackAfterFailedRestore(stackName, cause, liveDef, undo, moved || undo != nil, opt.FilesPlaced, restartStack, holdKey)
return res, m.putBackAfterFailedRestore(stackName, cause, liveDef, undo, moved || undo != nil, opt.FilesPlaced, restartStack, holdKey, settings.RestoreKindLocal)
}
// R-353: the count is captured even when the replay errors — a partial replay is a fact the
// customer's sentence has to be built from, and discarding it on the error path is how Scenario C
@@ -71,9 +71,14 @@ func orNone(s string) string {
// so and the kept copy is removed (the normal rule after a successful restore). A new restore of the app ends the watch
// untouched.
func (m *Manager) watchLateStart(stack, copyDir string, since time.Time, watch time.Duration) {
m.setWatching(copyDir, true) // R-1342: the copy is the way back while the app may still start
defer m.setWatching(copyDir, false)
switch m.waitStableStart(stack, since, since.Add(watch)) {
case stableAborted:
return // a new restore of this app owns it now
case stableRemoved:
m.logger.Printf("[INFO] [backup] %s: the app was removed while the box waited for it to start — the watch ends (R-1342)", stack)
return
case stableNot:
m.logger.Printf("[WARN] [backup] %s: still not running stably %s after its restore — the restore copy stays for support (%s) (R-1270)", stack, lateStartWatch, orNone(copyDir))
return
@@ -133,12 +138,26 @@ type containerSampler interface {
ContainerSamples(name string) ([]ContainerSample, error)
}
// StackRemovalReporter (R-1342) is the provider's optional answer to „was this app removed?" — true only when the app is
// known and no longer installed. A watch that sees it ends quietly: an app the household removed did not „fail to
// start". The production stackAdapter implements it (a compile-time assertion in cmd/controller/main.go).
type StackRemovalReporter interface {
StackRemoved(name string) bool
}
// stackRemoved asks the provider; a provider without the answer never reads „removed".
func (m *Manager) stackRemoved(stack string) bool {
r, ok := m.stackProvider.(StackRemovalReporter)
return ok && r.StackRemoved(stack)
}
type stableResult int
const (
stableYes stableResult = iota
stableNot
stableAborted
stableRemoved // R-1342: the app was removed during the watch
)
// stableSample is one look: ok, and the fingerprint (container id → restart count) a stable window must keep.
@@ -201,6 +220,9 @@ func (m *Manager) waitStableFor(stack string, watchFrom, deadline time.Time, win
if m.restoreBegunAfter(stack, watchFrom) {
return stableAborted
}
if m.stackRemoved(stack) {
return stableRemoved
}
ok, fp := m.stableSample(stack)
switch {
case !ok:
@@ -226,13 +248,17 @@ func (m *Manager) releaseAfterStableStart(stack, kind string, undo *liveVolumeCo
dir := undo.dir
from := time.Now() // taken HERE, before the goroutine: a restore begun after this moment is a new one
until := from.Add(stableReleaseWatch)
m.setWatching(dir, true) // R-1342: „Törlés most" refuses the copy while the watch holds it
go func() {
defer m.setWatching(dir, false)
switch m.waitStableStart(stack, from, until) {
case stableYes:
removeCopyDir(m, stack, dir)
m.logger.Printf("[INFO] [backup] %s: ran stably for %s after its %s — the restore copy is released (R-1270)", stack, stableStartWindow, kind)
case stableAborted:
m.logger.Printf("[INFO] [backup] %s: a new restore began before the %s's start was stable — its restore copy stays for support (%s)", stack, kind, dir)
case stableRemoved:
m.logger.Printf("[INFO] [backup] %s: the app was removed during the stable-start check after its %s — no alarm; the restore copy, if the remove left it, is an unused one (%s) (R-1342)", stack, kind, dir)
default:
m.logger.Printf("[ERROR] [backup] %s: did not run stably within %s after its %s — the restore copy stays for support (%s) (R-1270)", stack, stableReleaseWatch, kind, dir)
if m.restoreNotStartedNotify != nil {
@@ -262,6 +288,8 @@ func (m *Manager) restoredAppStarted(stack string) error {
return nil
case stableAborted:
return fmt.Errorf("a new restore of %s began", stack)
case stableRemoved:
return fmt.Errorf("%s was removed during the restore's start check", stack)
}
return fmt.Errorf("stack %s did not run stably for %s (it restarted or turned unhealthy) after restore", stack, verdictStableWindow)
}
@@ -674,6 +674,12 @@ func (m *Manager) clearUpdateHoldAfterRestore(stackName string) {
return
}
m.logger.Printf("[INFO] [backup] %s: restore completed — the update hold (set %s) is CLEARED", stackName, h.At)
// R-1347: the hold store and the stack manager's in-memory flag go together, as in the operator's clear (D64 = B, R-1343).
// Left set, the crash-loop watcher would not watch the restored app and the night leg would skip it until the next
// controller restart. A running update is not touched (ForgetUpdateHeld refuses it). Pinned by TestR1347_*.
if m.updateHeldForgetter != nil && m.updateHeldForgetter(stackName) {
m.logger.Printf("[INFO] [backup] %s: the restored app is watched and updated again — its update-held flag is forgotten (R-1347)", stackName)
}
// R-671 (v0.272.0): the undo copies the hold kept describe the state this restore just replaced. They
// were kept so the hold's data stayed recoverable; once the app is restored whole they are dead weight
// (measured 2026-09-24 on 9202: three nextcloud copies, ~0.9 GiB, outlived the hold and nothing named
@@ -691,6 +697,12 @@ func (m *Manager) SetUndoCopyRemover(fn func(stackName string) int) {
m.undoCopyRemover = fn
}
// SetUpdateHeldForgetter wires R-1347's seam: stacks.Manager.ForgetUpdateHeld. INIT-ONLY, main.go — pinned by
// TestR1347_UpdateHeldForgetterIsWiredAtStartup (an AST walk). The backup package cannot import stacks.
func (m *Manager) SetUpdateHeldForgetter(fn func(stackName string) bool) {
m.updateHeldForgetter = fn
}
// UpdateHeldStacks is the set of apps held stopped after a failed update (R-660, v0.268.0) — the
// FOURTH way the product stops an app on purpose, and until v0.268.0 the one `classifyRunStates` did
// not know: each hold's `app_update_held` was followed ~11 s later by an `app_start_failed` for the
+132
View File
@@ -0,0 +1,132 @@
// Package busymark writes the controller's „busy" file for the host agent (D65 = A, operator ruling 2026-10-29 15:50;
// felhom.eu `03`, `04` §6.1).
//
// WHY: the operator can reboot the guest from the hub. The agent refuses while a backup, a restore, an off-site run or
// an app update runs — but the controller's own work is invisible to the agent (it runs inside this container). So
// while any of it runs, this package keeps `<data>/ctl/busy` fresh: line 1 the time (RFC 3339, UTC), line 2 what runs.
// The agent reads it over one exact read-only grant (`pct exec <vmid> -- cat <the guest path>`), without Docker.
//
// FRESHNESS IS THE SIGNAL, NOT PRESENCE: the file is rewritten every Every (15 s) while busy and removed when idle.
// A controller that is killed or stuck mid-backup stops rewriting it, and the agent (90 s) reads the old file as „not
// busy" — a stuck controller is exactly the box a reboot is for. A file left by a previous run is removed at start.
// Pinned by busymark_test.go.
package busymark
import (
"context"
"errors"
"log"
"os"
"path/filepath"
"sync"
"time"
)
// FileName is the busy file inside the ctl directory (the directory of the hold-clear socket, 0700).
const FileName = "busy"
// DefaultEvery is the rewrite cadence while busy.
const DefaultEvery = 15 * time.Second
// Writer keeps the file in step with its probes.
type Writer struct {
Path string // <data>/ctl/busy
Every time.Duration
Now func() time.Time
Logger *log.Logger
mu sync.Mutex
probes []probe
last string // what the last tick saw ("" = idle)
}
type probe struct {
what string
busy func() bool
}
// AddProbe adds one kind of work: what is the sentence the agent shows the operator, busy says whether it runs now.
// A nil busy is ignored (a box without that component).
func (w *Writer) AddProbe(what string, busy func() bool) {
if busy == nil {
return
}
w.mu.Lock()
defer w.mu.Unlock()
w.probes = append(w.probes, probe{what: what, busy: busy})
}
// Busy returns the first running work's sentence, or "" when nothing runs.
func (w *Writer) Busy() string {
w.mu.Lock()
ps := append([]probe(nil), w.probes...)
w.mu.Unlock()
for _, p := range ps {
if p.busy() {
return p.what
}
}
return ""
}
func (w *Writer) now() time.Time {
if w.Now != nil {
return w.Now()
}
return time.Now()
}
func (w *Writer) logf(format string, args ...any) {
if w.Logger != nil {
w.Logger.Printf(format, args...)
}
}
// Tick runs one evaluation: write the file when busy, remove it when idle. Exported for the test and the start.
func (w *Writer) Tick() {
what := w.Busy()
if what == "" {
if err := os.Remove(w.Path); err != nil && !errors.Is(err, os.ErrNotExist) {
w.logf("[WARN] [busymark] the busy file could not be removed: %v — the agent reads it as busy for up to 90 s more", err)
}
if w.last != "" {
w.logf("[INFO] [busymark] idle again (was: %s) — an operator reboot of the box is no longer refused for it", w.last)
}
w.last = ""
return
}
body := w.now().UTC().Format(time.RFC3339) + "\n" + what + "\n"
if err := os.MkdirAll(filepath.Dir(w.Path), 0o700); err == nil {
tmp := w.Path + ".tmp"
if err = os.WriteFile(tmp, []byte(body), 0o600); err == nil {
err = os.Rename(tmp, w.Path)
}
if err != nil {
w.logf("[WARN] [busymark] the busy file could not be written: %v — an operator reboot is not refused for %s", err, what)
}
}
if w.last != what {
w.logf("[INFO] [busymark] busy: %s — an operator reboot of the box is refused until it ends", what)
}
w.last = what
}
// Run removes a file left by an earlier run, then ticks every Every until ctx ends (and removes the file then).
func (w *Writer) Run(ctx context.Context) {
every := w.Every
if every <= 0 {
every = DefaultEvery
}
w.Tick()
t := time.NewTicker(every)
defer t.Stop()
for {
select {
case <-ctx.Done():
_ = os.Remove(w.Path)
return
case <-t.C:
w.Tick()
}
}
}
@@ -0,0 +1,65 @@
package busymark
import (
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// D65 = A: the agent refuses an operator reboot while the controller's busy file is fresh.
func newW(t *testing.T) (*Writer, *bool, *time.Time) {
t.Helper()
busy := false
now := time.Date(2026, 10, 29, 16, 0, 0, 0, time.UTC)
w := &Writer{Path: filepath.Join(t.TempDir(), "ctl", FileName), Now: func() time.Time { return now }}
w.AddProbe("a backup or a restore is running", func() bool { return busy })
return w, &busy, &now
}
// Busy → the file holds the time and the sentence; idle → gone. RED-PROOF: skip the write → no file → FAILS;
// skip the remove → the file stays → FAILS.
func TestBusyFileFollowsTheWork(t *testing.T) {
w, busy, now := newW(t)
w.Tick()
if _, err := os.Stat(w.Path); !os.IsNotExist(err) {
t.Fatalf("idle: a busy file exists (%v)", err)
}
*busy = true
w.Tick()
raw, err := os.ReadFile(w.Path)
if err != nil {
t.Fatalf("busy: no file: %v", err)
}
lines := strings.Split(strings.TrimSpace(string(raw)), "\n")
if len(lines) != 2 || lines[0] != now.Format(time.RFC3339) || lines[1] != "a backup or a restore is running" {
t.Fatalf("file %q", raw)
}
if fi, _ := os.Stat(w.Path); fi.Mode().Perm() != 0o600 {
t.Fatalf("mode %v, want 0600", fi.Mode().Perm())
}
// it is REWRITTEN while busy: freshness is the agent's signal
*now = now.Add(15 * time.Second)
w.Tick()
raw, _ = os.ReadFile(w.Path)
if !strings.HasPrefix(string(raw), now.Format(time.RFC3339)) {
t.Fatalf("not rewritten: %q", raw)
}
*busy = false
w.Tick()
if _, err := os.Stat(w.Path); !os.IsNotExist(err) {
t.Fatalf("idle again: the busy file is still there (%v) — the agent would refuse a reboot for 90 s more", err)
}
}
// The first probe that runs names the work; a nil probe is ignored.
func TestBusyNamesTheRunningWork(t *testing.T) {
w, _, _ := newW(t)
w.AddProbe("an app update is running", func() bool { return true })
w.AddProbe("ignored", nil)
if got := w.Busy(); got != "an app update is running" {
t.Fatalf("Busy() = %q", got)
}
}
+16
View File
@@ -953,6 +953,7 @@
"dashboard.tunnel_offline_title": "The secure internet connection is not connected now, so your apps cannot be reached from outside. Check the internet connection.",
"dashboard.utolso_mentes": "Last backup:",
"dashboard.vedett": "Protected",
"dashboard.operator_reboot": "Felhom support restarted your box at %s.",
"dashboard.vezerlopult": "Dashboard",
"datapath.consequence.excluded": "This folder is temporary and is not backed up.",
"datapath.consequence.import_excluded": "Copy the files to be processed in here. The app reads them and then deletes them from here — this folder is temporary and is not backed up.",
@@ -2634,6 +2635,20 @@
"kept.delete.type": "To delete it, type: %s",
"kept.delete.mismatch": "The name you typed does not match (%s), so nothing was deleted.",
"kept.deleted": "The kept data of %s is deleted.",
"restore_copy.title": "Copies from before a restore",
"restore_copy.hint": "When you restore an app, the box first saves the app’s data as it was. It deletes that copy by itself 30 days after the restore.",
"restore_copy.restored_on": "restore on %s",
"restore_copy.auto_delete": "automatically deleted on %s",
"restore_copy.needed.hold": "Cannot be deleted: the app is stopped, and this copy is its way back. Support decides about it.",
"restore_copy.needed.putback": "Cannot be deleted: an interrupted restore puts the data back from this copy.",
"restore_copy.needed.unknown": "Cannot be deleted now: the box cannot tell whether this copy is still needed.",
"restore_copy.needed.checking": "Cannot be deleted now: the box is still checking that the app started well after the restore (up to 30 minutes).",
"restore_copy.delete": "Delete now",
"restore_copy.warning": "This is the copy of %s’s data from before the restore on %s. If you delete it, the restore can no longer be undone, and this copy cannot be brought back.",
"restore_copy.app_busy": "Cannot be deleted now: an operation runs on the app. Try again when it has ended.",
"restore_copy.not_found": "This copy is no longer there, so nothing was deleted.",
"restore_copy.failed": "The copy could not be deleted: %s",
"flash.restore_copy.deleted": "The copy of %s from before its restore is deleted.",
"kept.page.title": "Kept data",
"kept.page.back": "Back",
"kept.page.intro": "Data that removed apps left on the drives. You can look at it, load it into a new install, or delete it. The server never deletes any of it by itself.",
@@ -2672,6 +2687,7 @@
"api.escrow.confirm_expired": "The time to type the new code back has run out — your previous code still works. Start the process again.",
"api.escrow.confirm_lost": "The new code no longer waits for confirmation — your previous code still works. Start the process again.",
"api.escrow.confirm_store_failed": "The new code could not be saved — your previous code still works. Try again.",
"api.escrow.confirm_store_unknown": "We cannot tell yet whether the new code was saved — the server did not answer in time. Keep both codes (the old one and the new one) and try again.",
"api.escrow.confirm_words_mismatch": "The words do not match — check the code you wrote down.",
"api.share.form_invalid": "Invalid form — reload the page.",
"api.share.too_many_attempts": "Too many failed attempts, try again in 1 minute",
+16
View File
@@ -948,6 +948,7 @@
"dashboard.tunnel_offline_title": "A biztonságos internetkapcsolat most nem kapcsolódik, ezért az alkalmazásaid kívülről nem érhetők el. Ellenőrizd az internetkapcsolatot.",
"dashboard.utolso_mentes": "Utolsó mentés:",
"dashboard.vedett": "Védett",
"dashboard.operator_reboot": "A Felhom ügyfélszolgálata %s-kor újraindította a dobozt.",
"dashboard.vezerlopult": "Vezérlőpult",
"datapath.consequence.excluded": "Ez a mappa átmeneti, és nem készül róla biztonsági mentés.",
"datapath.consequence.import_excluded": "Ide másold a feldolgozandó fájlokat. Az alkalmazás beolvassa, majd törli innen — ez a mappa átmeneti, és nem készül róla biztonsági mentés.",
@@ -2622,6 +2623,20 @@
"kept.delete.type": "A törléshez írd be: %s",
"kept.delete.mismatch": "A beírt név nem egyezik (%s), ezért nem töröltünk semmit.",
"kept.deleted": "A(z) %s megőrzött adatai törölve.",
"restore_copy.title": "Visszaállítás előtti másolatok",
"restore_copy.hint": "Visszaállításkor a doboz előbb elmenti az alkalmazás akkori adatait. Ezt a másolatot a visszaállítás után 30 nappal magától törli.",
"restore_copy.restored_on": "visszaállítás: %s",
"restore_copy.auto_delete": "automatikusan törlődik: %s",
"restore_copy.needed.hold": "Nem törölhető: az alkalmazás le van állítva, és ez a másolat az út vissza. Az ügyfélszolgálat dönt róla.",
"restore_copy.needed.putback": "Nem törölhető: egy félbeszakadt visszaállítás ebből a másolatból teszi vissza az adatokat.",
"restore_copy.needed.unknown": "Most nem törölhető: a doboz nem tudja eldönteni, kell-e még ez a másolat.",
"restore_copy.needed.checking": "Most nem törölhető: a doboz még figyeli, hogy az alkalmazás rendben elindult-e a visszaállítás után (legfeljebb 30 percig).",
"restore_copy.delete": "Törlés most",
"restore_copy.warning": "Ez a(z) %s adatainak másolata a %s-i visszaállítás előttről. Ha törlöd, a visszaállítás többé nem vonható vissza, és ez a másolat nem hozható vissza.",
"restore_copy.app_busy": "Most nem törölhető: az alkalmazáson éppen egy művelet fut. Próbáld újra, ha véget ért.",
"restore_copy.not_found": "Ez a másolat már nincs meg, ezért nem töröltünk semmit.",
"restore_copy.failed": "A másolatot nem sikerült törölni: %s",
"flash.restore_copy.deleted": "A(z) %s visszaállítás előtti másolata törölve.",
"kept.page.title": "Megőrzött adatok",
"kept.page.back": "Vissza",
"kept.page.intro": "Az eltávolított alkalmazások lemezen maradt adatai. Megnézheted őket, betöltheted egy új telepítésbe, vagy törölheted. Magától a szerver soha nem töröl közülük semmit.",
@@ -2660,6 +2675,7 @@
"api.escrow.confirm_expired": "Lejárt az idő az új kód visszaírására — a régi kódod továbbra is érvényes. Indítsd újra a folyamatot.",
"api.escrow.confirm_lost": "Az új kód már nem vár megerősítésre — a régi kódod továbbra is érvényes. Indítsd újra a folyamatot.",
"api.escrow.confirm_store_failed": "Az új kódot nem sikerült elmenteni — a régi kódod továbbra is érvényes. Próbáld újra.",
"api.escrow.confirm_store_unknown": "Nem tudjuk biztosan, hogy az új kód mentése sikerült-e — a központ nem válaszolt időben. Őrizd meg mindkét kódot (a régit és az újat is), és próbáld újra.",
"api.escrow.confirm_words_mismatch": "A megadott szavak nem egyeznek — ellenőrizd a felírt kódot.",
"api.share.form_invalid": "Érvénytelen űrlap — töltsd újra az oldalt.",
"api.share.too_many_attempts": "Túl sok sikertelen próbálkozás, próbáld újra 1 perc múlva",
+3 -1
View File
@@ -27,8 +27,10 @@ import (
// notices (felhom.eu R-1304, R-1305; 1.5.7-stable is skipped — it broke existing TOTP sign-ins). Each new image reaches
// a running box by the reconcile at start: TestEnsureTraefik_ANewImageRecreatesTraefik,
// TestEnsureCloudflared_ANewImageRecreatesTheTunnel, TestReconcileFileBrowserImage_MovesOnlyTheImage.
// traefik v3.7.17 since 2026-11-02: GHSA-xv5x-242w-7hf7 (high, v3.7.14–v3.7.16): an unauthenticated client with a
// Negotiate/NTLM header holds a backend transport per connection until memory runs out, NTLM backend or not.
const (
TraefikImage = "traefik:v3.7.16"
TraefikImage = "traefik:v3.7.17"
CloudflaredImage = "cloudflare/cloudflared:2026.10.4"
FileBrowserImage = "gtstef/filebrowser:1.5.8-stable"
// FileBrowserImportMount is the in-container mount point NAME for the canonical drop-zone
@@ -0,0 +1,34 @@
package notify
import (
"encoding/json"
"strings"
"testing"
)
// D62 — restore_copy_deleted: info (the hub's vocabulary), the operator message only (no household sentence — the hub
// bars the type from customers too), and the details carry who, app, the restore's date and the size.
func TestD62_RestoreCopyDeletedEvent(t *testing.T) {
n := &Notifier{}
var et, sev, msg, cust string
var det []byte
n.pushFn = func(eventType, severity, message, customer string, details interface{}) {
et, sev, msg, cust = eventType, severity, message, customer
det, _ = json.Marshal(details)
}
n.NotifyRestoreCopyDeleted(RestoreCopyDeletedDetails{App: "vikunja", RestoreAt: "2026-10-29T08:00:00Z", SizeBytes: 4096,
Who: "the household (dashboard sign-in) from 192.168.0.5", Path: "/x/backups/restore-undo/vikunja/20261029T080000Z"})
if et != "restore_copy_deleted" || sev != "info" || cust != "" {
t.Fatalf("type=%q severity=%q customer=%q", et, sev, cust)
}
for _, want := range []string{"vikunja", "2026-10-29T08:00:00Z", "4096", "the household"} {
if !strings.Contains(msg, want) {
t.Fatalf("message %q lacks %q", msg, want)
}
}
for _, want := range []string{`"app":"vikunja"`, `"restore_at":"2026-10-29T08:00:00Z"`, `"size_bytes":4096`, `"who":"the household`} {
if !strings.Contains(string(det), want) {
t.Fatalf("details %s lack %s", det, want)
}
}
}
+28 -1
View File
@@ -163,6 +163,8 @@ type UpdateDetails struct {
type AppDetails struct {
StackName string `json:"stack_name,omitempty"`
DisplayName string `json:"display_name,omitempty"`
// Source (app_oom only, R-528): what saw the kill — "docker" (State.OOMKilled) or "cgroup" (the host's kernel counter).
Source string `json:"source,omitempty"`
}
// CrossDriveDetails holds structured data for cross-drive backup events.
@@ -804,6 +806,12 @@ func (n *Notifier) NotifyAppStartFailures(apps []AppRunState) {
// that stays OOM-marked does not repeat. "warning" — the hub vocabulary (R-329). Operator-only
// hub-side (hub >= v0.114.0 registers it): the household sees the dashboard tag.
func (n *Notifier) NotifyAppOOM(stack, container, startedAt string, kills int64, memLimit, peak string) {
n.NotifyAppOOMFrom(stack, container, startedAt, kills, memLimit, peak, "")
}
// NotifyAppOOMFrom is NotifyAppOOM with the source that saw the kill (R-528: "docker" or "cgroup"), carried in the
// event's details. One event per container run, whichever source saw it.
func (n *Notifier) NotifyAppOOMFrom(stack, container, startedAt string, kills int64, memLimit, peak, source string) {
key := container + "|" + startedAt
n.mu.Lock()
if n.oomSeen == nil {
@@ -816,7 +824,7 @@ func (n *Notifier) NotifyAppOOM(stack, container, startedAt string, kills int64,
if first {
n.emit("app_oom", "warning",
fmt.Sprintf("Alkalmazás memóriája elfogyott: %s (%s) — egy folyamatát a memóriakorlát leállította", stack, container),
AppDetails{StackName: stack, DisplayName: container})
AppDetails{StackName: stack, DisplayName: container, Source: source})
}
if storm {
n.logger.Printf("[ERROR] [notify] %s: container %s OOM STORM — %d kills in %d min (limit %s, peak %s)",
@@ -1577,6 +1585,25 @@ func (n *Notifier) NotifyAppDefinitionRefused(app, detail string) {
n.pushEventBoth("app_definition_refused", "warning", msg, "", AppDefinitionRefusedDetails{App: app, Refused: detail})
}
// RestoreCopyDeletedDetails is restore_copy_deleted's record: who deleted which app's copy, the restore it protected,
// and its size.
type RestoreCopyDeletedDetails struct {
App string `json:"app"`
RestoreAt string `json:"restore_at"` // RFC 3339, UTC
SizeBytes int64 `json:"size_bytes"`
Who string `json:"who"`
Path string `json:"path"`
}
// NotifyRestoreCopyDeleted (D62, operator 2026-10-29): the household deleted a restore copy with „Törlés most" before
// its 30 days. Info, operator-only on the hub (registered in both of its lists, hub D62): an audit line for support,
// never a household mail. Pinned by TestD62_RestoreCopyDeletedEvent.
func (n *Notifier) NotifyRestoreCopyDeleted(d RestoreCopyDeletedDetails) {
msg := fmt.Sprintf("%s: the restore copy from before the restore of %s (%d bytes) was deleted by hand — %s. That restore can no longer be undone from this box.",
d.App, d.RestoreAt, d.SizeBytes, d.Who)
n.pushEventBoth("restore_copy_deleted", "info", msg, "", d)
}
// healthSeverity is the event severity a health status would be sent at (R-647): the disabled path
// names what it drops, and "warn" is a health STATUS, not a severity the hub knows.
func healthSeverity(status string) string {
+42 -4
View File
@@ -21,9 +21,11 @@ import (
// problem, main.go). The result rides the next report as `operator_action_results: [{id, outcome,
// message}]`; the hub lists an action until its result has arrived, then stops.
//
// THE LIST IS CLOSED. Four actions, and run_job names one of four jobs. Nothing here deletes data,
// THE LIST IS CLOSED. Five actions, and run_job names one of four jobs. Nothing here deletes data,
// starts a countdown or shortens one — `03` §4 asks for a signing key only to destroy or overwrite the
// only copy, and none of these does. An unknown action, an unknown job or an argument out of range is
// only copy, and none of these does. clear_hold (D64 = B, operator ruling 2026-10-29 12:10) lifts an app's
// hold of ANY reason with no operator signature: it deletes nothing and starts nothing — the app stays
// stopped until the household (or the operator) starts it, and a waiting put-back's copy stays on disk. An unknown action, an unknown job or an argument out of range is
// answered `refused` and NOTHING is called. Pinned by TestOpActions_ClosedList (which fails when an
// entry is added, so a new action is a decision, not an edit) and TestOpActions_UnknownIsRefused.
//
@@ -38,6 +40,7 @@ const (
OpAbandonStop = "abandon_stop"
OpAbandonExtend = "abandon_extend"
OpRunJob = "run_job"
OpClearHold = "clear_hold"
OutcomeDone = "done"
OutcomeRefused = "refused"
@@ -57,7 +60,7 @@ const (
)
// operatorActionNames is THE closed list. Adding an entry fails TestOpActions_ClosedList by design.
var operatorActionNames = []string{OpOffsiteBackupNow, OpAbandonStop, OpAbandonExtend, OpRunJob}
var operatorActionNames = []string{OpOffsiteBackupNow, OpAbandonStop, OpAbandonExtend, OpRunJob, OpClearHold}
// operatorJobNames is the fixed set run_job may name — the scheduler's own job names (main.go).
var operatorJobNames = []string{"fill-watch", "offsite-integrity", "offsite-proof", "disk-health-check"}
@@ -83,6 +86,21 @@ func contains(list []string, v string) bool {
return false
}
// validHoldStack is clear_hold's argument check: the same alphabet as backup.isSafeStackName (report cannot import
// backup; REUSE.md „Safe-name validators").
func validHoldStack(s string) bool {
if s == "" || len(s) > 64 {
return false
}
for _, c := range s {
if (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '-' || c == '_' {
continue
}
return false
}
return true
}
// ErrActionRefused, wrapped by a handler, turns its error into a `refused` outcome (the action could
// not START — e.g. an off-site run already in flight) instead of `failed` (it started and went wrong).
var ErrActionRefused = errors.New("refused")
@@ -101,7 +119,7 @@ type OperatorActionResult struct {
Message string `json:"message"`
}
// OperatorActionHandlers are the four doors, wired by main.go. A nil handler answers `refused`
// OperatorActionHandlers are the five doors, wired by main.go. A nil handler answers `refused`
// („not available on this box").
type OperatorActionHandlers struct {
// OffsiteBackupNow returns a refusal sentence when the run may not start (the household's
@@ -111,6 +129,9 @@ type OperatorActionHandlers struct {
AbandonExtend func(days int) (time.Time, error)
// RunJob is Scheduler.RunNow: an error = not started (unknown/running); the channel = the job's end.
RunJob func(name string) (<-chan error, error)
// ClearHold lifts the app's hold in the RUNNING controller (D64 = B, R-1343) and returns the result
// sentence. actionID names the action in the box's log. An error wrapping ErrActionRefused = nothing changed.
ClearHold func(stack string, actionID int64) (string, error)
// ResultReady sends an out-of-cycle report so a result reaches the hub in seconds (optional).
ResultReady func()
}
@@ -317,6 +338,23 @@ func (o *OperatorActions) dispatch(a OperatorAction) {
}
o.finish(a, outcomeOf(jerr), msg)
}()
case OpClearHold:
// The argument is checked HERE, before any handler is touched (the closed list's rule): one app name,
// the backup package's strict alphabet (letters, digits, '-' and '_', at most 64).
if !validHoldStack(a.Arg) {
o.finish(a, OutcomeRefused, "the argument must be one app name (letters, digits, '-' and '_')")
return
}
if o.h.ClearHold == nil {
o.finish(a, OutcomeRefused, "not available on this box")
return
}
msg, err := o.h.ClearHold(a.Arg, a.ID)
if err != nil {
o.finish(a, outcomeOf(err), err.Error())
return
}
o.finish(a, OutcomeDone, msg)
default:
o.finish(a, OutcomeRefused, "unknown action")
}
+79 -2
View File
@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"errors"
"fmt"
"reflect"
"strings"
"sync"
@@ -15,9 +16,11 @@ import (
// The list is CLOSED. This test fails when an action or a job is added, by design: a new entry is an
// operator decision (no action may delete data, start a countdown or shorten one), not an edit. The
// hub pins the same four in its own test (hub internal/store opactions_test.go).
// hub pins the same list in its own test (hub internal/store opactions_test.go).
// 2026-10-29: clear_hold added deliberately — operator ruling D64 = B (12:10): the hub's „Clear hold" works for
// every hold reason with no operator signature. It deletes nothing and starts nothing.
func TestOpActions_ClosedList(t *testing.T) {
if got, want := OperatorActionNames(), []string{"abandon_extend", "abandon_stop", "offsite_backup_now", "run_job"}; !reflect.DeepEqual(got, want) {
if got, want := OperatorActionNames(), []string{"abandon_extend", "abandon_stop", "clear_hold", "offsite_backup_now", "run_job"}; !reflect.DeepEqual(got, want) {
t.Fatalf("operator actions = %v, want exactly %v — a new action needs the operator's word (decision 185)", got, want)
}
if got, want := OperatorJobNames(), []string{"disk-health-check", "fill-watch", "offsite-integrity", "offsite-proof"}; !reflect.DeepEqual(got, want) {
@@ -56,6 +59,10 @@ func recordingHandlers(c *calls) OperatorActionHandlers {
close(ch)
return ch, nil
},
ClearHold: func(stack string, id int64) (string, error) {
c.add("clear:" + stack)
return "the hold on " + stack + " is cleared", nil
},
}
}
@@ -197,3 +204,73 @@ func TestOpActions_WireShapes(t *testing.T) {
t.Fatalf("report carried %s", b)
}
}
// clear_hold (D64 = B): the argument is one app name in the strict alphabet — never a path, never empty, never two
// names. A bad one is refused and the handler is NOT called (the closed list's rule: validate before any door).
// RED-PROOF (2026-10-29): with validHoldStack returning true, this fails on „a refused action called [clear:…]".
func TestOpActions_ClearHoldBadArgumentIsRefusedAndCallsNothing(t *testing.T) {
c := &calls{}
o := NewOperatorActions(context.Background(), recordingHandlers(c), nil)
list := []OperatorAction{
{ID: 1, Action: OpClearHold, Arg: ""},
{ID: 2, Action: OpClearHold, Arg: "../settings"},
{ID: 3, Action: OpClearHold, Arg: "immich paperless"},
{ID: 4, Action: OpClearHold, Arg: "immich/x"},
{ID: 5, Action: OpClearHold, Arg: strings.Repeat("a", 65)},
{ID: 6, Action: OpClearHold, Arg: "immich\n"},
{ID: 7, Action: OpClearHold, Arg: "."},
}
o.Reconcile(list)
if got := c.get(); len(got) != 0 {
t.Fatalf("a refused action called %v", got)
}
for _, a := range list {
if r := resultFor(t, o, a.ID); r.Outcome != OutcomeRefused || !strings.Contains(r.Message, "one app name") {
t.Errorf("#%d clear_hold(%q): %+v, want refused naming the rule", a.ID, a.Arg, r)
}
}
// control: a good name does reach the handler
o.Reconcile([]OperatorAction{{ID: 8, Action: OpClearHold, Arg: "paperless-ngx_2"}})
if got := c.get(); len(got) != 1 || got[0] != "clear:paperless-ngx_2" {
t.Fatalf("a valid name must reach the handler, calls %v", got)
}
}
// clear_hold (D64 = B): a valid name reaches the handler with the action's id, and its sentence is the result; a
// handler refusal (the app has no hold) is answered refused, an error that is not a refusal failed.
// RED-PROOF (2026-10-29): with the OpClearHold case removed from dispatch, this fails on „unknown action".
func TestOpActions_ClearHoldDoneRefusedFailed(t *testing.T) {
var gotStack string
var gotID int64
h := OperatorActionHandlers{ClearHold: func(stack string, id int64) (string, error) {
gotStack, gotID = stack, id
switch stack {
case "notheld":
return "", fmt.Errorf("%w: notheld has no hold — nothing was changed", ErrActionRefused)
case "broken":
return "", errors.New("saving settings: disk full")
}
return "the hold on " + stack + " is cleared (it was: held after a failed update)", nil
}}
o := NewOperatorActions(context.Background(), h, nil)
o.Reconcile([]OperatorAction{{ID: 7, Action: OpClearHold, Arg: "kimai"}})
if gotStack != "kimai" || gotID != 7 {
t.Fatalf("handler called with (%q, %d), want (kimai, 7)", gotStack, gotID)
}
if r := resultFor(t, o, 7); r.Outcome != OutcomeDone || r.Message != "the hold on kimai is cleared (it was: held after a failed update)" {
t.Fatalf("result %+v", r)
}
o.Reconcile([]OperatorAction{{ID: 8, Action: OpClearHold, Arg: "notheld"}, {ID: 9, Action: OpClearHold, Arg: "broken"}})
if r := resultFor(t, o, 8); r.Outcome != OutcomeRefused || !strings.Contains(r.Message, "has no hold") {
t.Errorf("not held: %+v", r)
}
if r := resultFor(t, o, 9); r.Outcome != OutcomeFailed {
t.Errorf("a save error must be failed, not refused: %+v", r)
}
// nil handler → refused, not a crash
o2 := NewOperatorActions(context.Background(), OperatorActionHandlers{}, nil)
o2.Reconcile([]OperatorAction{{ID: 1, Action: OpClearHold, Arg: "kimai"}})
if r := resultFor(t, o2, 1); r.Outcome != OutcomeRefused {
t.Errorf("nil handler: %+v", r)
}
}
@@ -0,0 +1,41 @@
package settings
import (
"strings"
"testing"
)
// R-1340, second producer (found live 2026-10-29 on Tester 1): the hold's Why() — the hub's „Held apps" line and the
// box's app_held event — still said „held after an off-site restore failed its database replay" for a LOCAL restore of
// an app with no database whose put-back stop failed. It must name what the hold record carries.
// RED-PROOF: seen failing on the old fixed sentence.
func TestR1340_WhyNamesTheMixedCause(t *testing.T) {
for _, tc := range []struct {
name string
h RestoreHold
want, not []string
}{
{"local put-back, no database", RestoreHold{Reason: HoldReasonRestoreMixed, At: "T", RestoreKind: RestoreKindLocal, MixedCause: MixedCausePutBack, FailedStep: "stopping the app before the put-back"},
[]string{"local restore", "putting the live data back failed", "stopping the app before the put-back"}, []string{"database", "off-site"}},
{"off-site database replay", RestoreHold{Reason: HoldReasonRestoreMixed, At: "T", RestoreKind: RestoreKindOffsite, MixedCause: MixedCausePutBack, DBReplay: true},
[]string{"off-site restore", "database replay", "putting the live data back failed"}, nil},
{"definition write", RestoreHold{Reason: HoldReasonRestoreMixed, At: "T", RestoreKind: RestoreKindOffsite, MixedCause: MixedCauseDefinitionWrite},
[]string{"definition"}, []string{"database"}},
{"interrupted put-back", RestoreHold{Reason: HoldReasonRestoreMixed, At: "T", MixedCause: MixedCauseInterruptedPutBack},
[]string{"interrupted"}, []string{"database"}},
{"legacy record", RestoreHold{Reason: HoldReasonRestoreMixed, At: "T"},
[]string{"stopped half-way", "clear the hold"}, []string{"database", "off-site"}},
} {
got := tc.h.Why()
for _, w := range tc.want {
if !strings.Contains(got, w) {
t.Errorf("%s: %q lacks %q", tc.name, got, w)
}
}
for _, n := range tc.not {
if strings.Contains(got, n) {
t.Errorf("%s: %q says %q", tc.name, got, n)
}
}
}
}
+59 -1
View File
@@ -1982,8 +1982,36 @@ type RestoreHold struct {
// first stop or a repeat within 24 h (Trip >= 2 → the sentence says support is informed).
UnhealthyKind string `json:"unhealthy_kind,omitempty"`
Trip int `json:"trip,omitempty"`
// R-1340 (2026-10-29): what put a restore hold in place, so the operator's „HELD STOPPED" mail tells the real
// story instead of the option-C database story for every cause. All empty on a hold written before them: the mail
// then names only what the record carries (cmd/controller heldStoppedOperatorText).
//
// RestoreKind is which restore: RestoreKindOffsite or RestoreKindLocal.
RestoreKind string `json:"restore_kind,omitempty"`
// MixedCause (restore_mixed only) is the branch that held the app: MixedCause*.
MixedCause string `json:"mixed_cause,omitempty"`
// FailedStep is the put-back step that failed, with its error (MixedCausePutBack).
FailedStep string `json:"failed_step,omitempty"`
// DBReplay: the restore's own failure was its database replay. False = the app's database was never touched
// by the failure, so the mail never says „database".
DBReplay bool `json:"db_replay,omitempty"`
}
// Restore kinds and mixed-hold causes (R-1340). See RestoreHold.RestoreKind / MixedCause.
const (
RestoreKindOffsite = "offsite"
RestoreKindLocal = "local"
// MixedCauseDefinitionWrite: the backup's definition could not be written (no data moved yet) and writing the
// live definition back failed too (R-1256 H4).
MixedCauseDefinitionWrite = "definition_write"
// MixedCausePutBack: the restore failed and putting the live data back failed at FailedStep (D20/R-1256 H5).
MixedCausePutBack = "put_back"
// MixedCauseInterruptedPutBack: a restore cut by a stop could not be put back at the next start (R-1251 H2).
MixedCauseInterruptedPutBack = "interrupted_put_back"
)
// Hold reasons. See RestoreHold.Reason.
const (
HoldReasonRestoreFailed = "" // R-379/R-380: a restore AND its rollback failed
@@ -2023,7 +2051,7 @@ func (h RestoreHold) Why() string {
}
return "stopped by the box for a " + strings.ReplaceAll(kind, "_", " ") + " (" + h.At + ") — the household's Start gives it one more try"
case HoldReasonRestoreMixed:
return "held after an off-site restore failed its database replay (" + h.At + ") — clear the hold to start it"
return h.mixedWhy() + " (" + h.At + ") — clear the hold to start it"
case HoldReasonPutBackPending:
return "held until the pre-restore copy is poured back (" + h.At + ") — the put-back starts it"
default:
@@ -2031,6 +2059,36 @@ func (h RestoreHold) Why() string {
}
}
// mixedWhy (R-1340) names a restore_mixed hold from what its record carries — never „database" or „off-site" unless the
// record says so. A record written before these fields („legacy") gets the neutral sentence. Pinned by
// TestR1340_WhyNamesTheMixedCause.
func (h RestoreHold) mixedWhy() string {
kind := "a restore"
switch h.RestoreKind {
case RestoreKindOffsite:
kind = "an off-site restore"
case RestoreKindLocal:
kind = "a local restore"
}
switch h.MixedCause {
case MixedCauseDefinitionWrite:
return "held after " + kind + " could not write its definition and writing the live definition back failed too"
case MixedCauseInterruptedPutBack:
return "held after a restore was interrupted and its live data could not be put back at the next start"
case MixedCausePutBack:
what := "failed"
if h.DBReplay {
what = "failed its database replay"
}
s := "held after " + kind + " " + what + " and putting the live data back failed"
if h.FailedStep != "" {
s += " (" + h.FailedStep + ")"
}
return s
}
return "held after a restore stopped half-way"
}
// SetRestoreHold records a hold. Modelled on SetDisconnected: a condition, plus what it is holding.
func (s *Settings) SetRestoreHold(h RestoreHold) error {
s.mu.Lock()
@@ -28,7 +28,11 @@ import (
// of the running version; the PREVIOUS — the self-update's own record (update-state.json previous_image of a swap
// that ended on the running version), or, when there is no such record or its image is gone, the highest version
// below the running one (logged as "by version order"); every version ABOVE the running one (a pulled target the
// swap has not taken yet); every tag that is not a plain X.Y.Z (latest, -rc) — left alone, logged. A candidate is an
// swap has not taken yet); every tag that is not a version at all (latest) — left alone, logged. A test build's tag
// (X.Y.Z-rc.N) IS a version since v0.351.0 (decision 56, amended 2026-11-09): util.ParseVersionPre orders it below its release, so an old rc is a
// candidate like an old release — 25 of them (2.2 GB) sat on demo-hp on 2026-11-09 because the rule left every -rc
// alone. The previous by version order prefers a release; a test build is the previous only when no release is below
// the running one. A box that RUNS a test build is judged too (it was skipped). A candidate is an
// untagged (<none>) controller image or one whose every tag is a version below the previous. Deleted by exact
// name/ID, never forced, never by prune; an ID that also carries another repository's name is left alone. NOTHING is
// deleted while the controller is swapping itself (selfUpdatingNow). Registry tags are never touched.
@@ -45,7 +49,7 @@ type ControllerImageRecord struct {
func (m *Manager) RetainControllerImages(rec ControllerImageRecord) ([]string, error) {
imageRetentionMu.Lock()
defer imageRetentionMu.Unlock()
running, err := util.ParseVersion(rec.Running)
running, err := util.ParseVersionPre(rec.Running)
if err != nil || rec.Repo == "" {
m.logger.Printf("[INFO] [stacks] controller image retention: skipped — running version %q is not a release (or no repository)", rec.Running)
return nil, nil
@@ -82,15 +86,27 @@ func (m *Manager) RetainControllerImages(rec ControllerImageRecord) ([]string, e
}
}
if prevTag == "" {
var best *util.Version
// A release first: a box on 0.350.0 holding 0.349.0 and 0.350.0-rc.3 keeps 0.349.0 for a hand roll-back.
var best, bestPre *util.Version
for _, im := range imgs {
if im.Repo != repo {
continue
}
if v, err := util.ParseVersion(im.Tag); err == nil && v.Compare(running) < 0 && (best == nil || v.Compare(*best) > 0) {
v, err := util.ParseVersionPre(im.Tag)
if err != nil || v.Compare(running) >= 0 {
continue
}
vv := v
if v.Pre == "" {
if best == nil || v.Compare(*best) > 0 {
best = &vv
}
} else if bestPre == nil || v.Compare(*bestPre) > 0 {
bestPre = &vv
}
}
if best == nil {
best = bestPre
}
if best != nil {
prevTag, prevHow = best.Raw, "by version order (no swap record names one present)"
@@ -98,7 +114,7 @@ func (m *Manager) RetainControllerImages(rec ControllerImageRecord) ([]string, e
}
var prev *util.Version
if prevTag != "" {
if v, err := util.ParseVersion(prevTag); err == nil {
if v, err := util.ParseVersionPre(prevTag); err == nil {
prev = &v
}
}
@@ -140,10 +156,10 @@ func (m *Manager) RetainControllerImages(rec ControllerImageRecord) ([]string, e
continue
}
names = append(names, im.Repo+":"+im.Tag)
v, err := util.ParseVersion(im.Tag)
v, err := util.ParseVersionPre(im.Tag)
switch {
case err != nil:
deletable, odd = false, im.Tag // latest, -rc: not ours to judge
deletable, odd = false, im.Tag // latest: not a version, not ours to judge
case v.Compare(running) >= 0:
deletable = false // the running one, or a pulled target not swapped to yet
case prev == nil || v.Compare(*prev) >= 0:
@@ -48,16 +48,17 @@ func TestControllerRetention_KeepsRunningAndPreviousDeletesOlder(t *testing.T) {
if strings.Contains(s, ":0.284.2,") {
t.Fatalf("the previous controller (0.284.2) was deleted: %v", f.rmi)
}
for _, want := range []string{ctlRepo + ":0.284.1", ctlRepo + ":0.283.1", "sha256:CNONE"} {
// 0.273.0-rc1: a test build below the previous goes like an old release (decision 56 amended 2026-11-09).
for _, want := range []string{ctlRepo + ":0.284.1", ctlRepo + ":0.283.1", "sha256:CNONE", ctlRepo + ":0.273.0-rc1"} {
if !strings.Contains(s, ","+want+",") {
t.Fatalf("%s (older than the previous / untagged) was not deleted: rmi=%v", want, f.rmi)
}
}
if strings.Contains(s, "0.273.0-rc1") || strings.Contains(s, "acme/web") || strings.Contains(s, "W3") {
t.Fatalf("a non-version tag or another repository's image was touched: %v", f.rmi)
if strings.Contains(s, "acme/web") || strings.Contains(s, "W3") {
t.Fatalf("another repository's image was touched: %v", f.rmi)
}
if len(got) != 3 {
t.Fatalf("deleted %d, want 3: %v", len(got), got)
if len(got) != 4 {
t.Fatalf("deleted %d, want 4: %v", len(got), got)
}
}
@@ -183,3 +184,67 @@ func TestRetainImagesAfterUpdate_AppGoneDoesNotPanic(t *testing.T) {
t.Fatalf("deleted images for an app that is gone: %v", f.rmi)
}
}
// Decision 56 amended 2026-11-09: demo-hp's guest held 25 old test builds (X.Y.Z-rc.N, 2.2 GB) because every -rc tag was
// "left alone". A test build is now a version (util.ParseVersionPre: below its release): an old one goes; a test build
// ABOVE the running one (a pulled target) and one BETWEEN the previous and the running one stay; `latest` stays.
// COMPANION RED-PROOF: parse tags with util.ParseVersion again → "an old test build (0.330.0-rc.2) was not deleted".
func TestControllerRetention_OldTestBuildsGo(t *testing.T) {
m := retentionManager(t)
f := &fakeImages{
imgs: []localImage{
{ID: "sha256:R350", Repo: ctlRepo, Tag: "0.350.0", Size: "348MB"},
{ID: "sha256:R349", Repo: ctlRepo, Tag: "0.349.0", Size: "348MB"},
{ID: "sha256:RC350", Repo: ctlRepo, Tag: "0.350.0-rc.3", Size: "348MB"},
{ID: "sha256:RC330", Repo: ctlRepo, Tag: "0.330.0-rc.2", Size: "347MB"},
{ID: "sha256:RC351", Repo: ctlRepo, Tag: "0.351.0-rc.1", Size: "348MB"},
{ID: "sha256:LATEST", Repo: ctlRepo, Tag: "latest", Size: "348MB"},
},
containers: map[string]string{"c-ctl": "sha256:R350"},
}
withFakeImages(t, f)
// No swap record: the previous by version order is the RELEASE 0.349.0, not the newer test build 0.350.0-rc.3.
if _, err := m.RetainControllerImages(ControllerImageRecord{Repo: ctlRepo, Running: "0.350.0"}); err != nil {
t.Fatal(err)
}
s := rmiSet(f)
if !strings.Contains(s, ","+ctlRepo+":0.330.0-rc.2,") {
t.Fatalf("an old test build (0.330.0-rc.2) was not deleted: %v", f.rmi)
}
if strings.Contains(s, ":0.349.0,") {
t.Fatalf("the previous RELEASE (0.349.0) was deleted — version order must prefer a release over a test build: %v", f.rmi)
}
if strings.Contains(s, ":0.351.0-rc.1,") || strings.Contains(s, ":latest,") || strings.Contains(s, ":0.350.0,") {
t.Fatalf("a newer test build, latest, or the running one was deleted: %v", f.rmi)
}
// 0.350.0-rc.3 sits between the previous (0.349.0) and the running one: kept, like a release there would be.
if strings.Contains(s, ":0.350.0-rc.3,") {
t.Fatalf("a test build between the previous and the running one was deleted: %v", f.rmi)
}
}
// A box that RUNS a test build (Tester 1 during an rc round) is judged too — it was skipped before.
// COMPANION RED-PROOF: parse rec.Running with util.ParseVersion again → "a box running a test build deleted nothing".
func TestControllerRetention_BoxOnATestBuildIsJudged(t *testing.T) {
m := retentionManager(t)
f := &fakeImages{
imgs: []localImage{
{ID: "sha256:RC2", Repo: ctlRepo, Tag: "0.351.0-rc.2", Size: "348MB"},
{ID: "sha256:RC1", Repo: ctlRepo, Tag: "0.351.0-rc.1", Size: "348MB"},
{ID: "sha256:R350", Repo: ctlRepo, Tag: "0.350.0", Size: "348MB"},
{ID: "sha256:R349", Repo: ctlRepo, Tag: "0.349.0", Size: "348MB"},
},
containers: map[string]string{"c-ctl": "sha256:RC2"},
}
withFakeImages(t, f)
if _, err := m.RetainControllerImages(ControllerImageRecord{Repo: ctlRepo, Running: "0.351.0-rc.2", Previous: ctlRepo + ":0.350.0"}); err != nil {
t.Fatal(err)
}
s := rmiSet(f)
if !strings.Contains(s, ","+ctlRepo+":0.349.0,") {
t.Fatalf("a box running a test build deleted nothing: %v", f.rmi)
}
if strings.Contains(s, ":0.350.0,") || strings.Contains(s, ":0.351.0-rc.2,") || strings.Contains(s, ":0.351.0-rc.1,") {
t.Fatalf("the recorded previous, the running test build or one between them was deleted: %v", f.rmi)
}
}
@@ -0,0 +1,46 @@
package stacks
import (
"io"
"log"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
)
// D64 = B / R-1343: when the operator clears a hold, the ONE held flag kept outside the hold store — „the last update
// ended held" — goes too. Left set, the crash-loop observer and the nightly update leg kept skipping an app whose hold
// was gone, until the controller restarted.
// RED-PROOF (2026-10-29): with ForgetUpdateHeld's body reduced to `return false`, this fails on „a held update must be forgotten".
func TestD64_ForgetUpdateHeld(t *testing.T) {
m := &Manager{cfg: &config.Config{}, logger: log.New(io.Discard, "", 0), stacks: map[string]*Stack{
"kimai": {Name: "kimai", Deployed: true},
"running": {Name: "running", Deployed: true, Updating: true},
}}
m.finishUpdateKey("kimai", UpdatePhaseFailed, UpdateErrorKeyHeld, "a kimai leállítva marad")
m.markUpdateHeld("kimai")
m.markUpdateHeld("running")
if !m.ForgetUpdateHeld("kimai") {
t.Fatal("a held update must be forgotten")
}
st, _ := m.GetStack("kimai")
if st.updateHeld || st.UpdateErrorKey != "" || st.UpdateError != "" || st.UpdatePhase != "" {
t.Fatalf("held state left: held=%v key=%q err=%q phase=%q", st.updateHeld, st.UpdateErrorKey, st.UpdateError, st.UpdatePhase)
}
// The two readers of the flag (ObserveUnhealthy's skip, unhealthy.go; legCandidate's LegSkipHeld, unattended.go)
// read the stored stack — not called here, because ObserveUnhealthy reaches docker (R-650).
m.mu.RLock()
skipped := m.stacks["kimai"].updateHeld || m.stacks["kimai"].UpdateErrorKey == UpdateErrorKeyHeld
m.mu.RUnlock()
if skipped {
t.Fatal("the observer and the update leg still skip kimai")
}
// A running update is never touched; an app with nothing held reports false.
if m.ForgetUpdateHeld("running") {
t.Fatal("a running update must not be touched")
}
if m.ForgetUpdateHeld("kimai") || m.ForgetUpdateHeld("nosuch") {
t.Fatal("nothing held → false")
}
}
+24 -1
View File
@@ -3,6 +3,8 @@ package stacks
import (
"fmt"
"path/filepath"
"gitea.dooplex.hu/admin/felhom-controller/internal/util"
)
// Desired-state values for AppConfig.DesiredState (R-166, decision D-b).
@@ -57,7 +59,8 @@ func (m *Manager) SetDesiredState(name, desired string) error {
stack, ok := m.GetStack(name)
if !ok {
return fmt.Errorf("stack %q not found", name)
// a KIND (404 at the API) and the household's sentence — not a save failure
return util.MsgErrorf(ErrStackNotFound, "err.stacks.alkalmazas_nem_talalhato", name)
}
stackDir := filepath.Dir(stack.ComposePath)
@@ -157,6 +160,26 @@ func isObservedUp(s Stack) bool {
}
}
// RecordStopWantedIfDown (R-1350) is the operator's hold clear (D64 = B) speaking for the household: the clear starts
// nothing, so an app that is NOT up stays stopped — and that stop is now WANTED, recorded as „stopped" so the dead-app
// scan does not mail „Telepített alkalmazás nem fut" 30 s after the clear (measured 2026-10-29 on Tester 1), and the
// boot reconciler does not start it at the next boot. A later Start records „running" and ends it. A running app's wish
// is left alone (recorded=false). The clear is a PERSON's act (R-379: „a person — on the box, or the operator on the
// hub"), so this is the ownership rule's own exception, not a machine writing intent.
func (m *Manager) RecordStopWantedIfDown(name string) (bool, error) {
st, ok := m.GetStack(name)
if !ok {
return false, nil
}
if isObservedUp(*st) {
return false, nil
}
if err := m.SetDesiredState(name, DesiredStateStopped); err != nil {
return false, err
}
return true, nil
}
// WantsStopped (R-721, v0.283.0) reports whether the household's own recorded intent for the stack is
// „stopped". The machines that stop an app for their own purposes and start it again afterwards — the
// whole-guest backup's quiesce, the nightly volume dump — ask this before their restart, so a Stop the
@@ -336,3 +336,27 @@ func TestDesiredStateOf_NilAppConfigIsUnknown(t *testing.T) {
t.Fatalf("a stack with no AppConfig reported desired state %q — absent must read as unknown", got)
}
}
// R-1350: the operator's hold clear records „stopped" for an app that is NOT up, and leaves a running app's wish alone
// (a clear never stops anything). RED-PROOF: write „stopped" without the up check → the running case FAILS.
func TestR1350_RecordStopWantedIfDown(t *testing.T) {
m, root := newDSManager(t, "gokapi", "immich")
for _, n := range []string{"gokapi", "immich"} {
writeAppYAML(t, stackDirOf(root, n), "deployed: true\ndesired_state: running\n")
m.stacks[n].AppConfig = LoadAppConfig(stackDirOf(root, n))
}
m.stacks["gokapi"].State = StateStopped
m.stacks["immich"].State = StateRunning
if rec, err := m.RecordStopWantedIfDown("gokapi"); err != nil || !rec {
t.Fatalf("gokapi: recorded=%v err=%v", rec, err)
}
if got := LoadAppConfig(stackDirOf(root, "gokapi")).DesiredState; got != DesiredStateStopped {
t.Fatalf("gokapi desired_state on disk = %q, want stopped", got)
}
if rec, err := m.RecordStopWantedIfDown("immich"); err != nil || rec {
t.Fatalf("immich (running): recorded=%v err=%v", rec, err)
}
if got := LoadAppConfig(stackDirOf(root, "immich")).DesiredState; got != DesiredStateRunning {
t.Fatalf("a running app's wish was changed to %q", got)
}
}
+22 -1
View File
@@ -109,7 +109,7 @@ func (m *Manager) ensureFileBrowserPublic() {
}
if m.fbNetOn(target, traefikNetwork) {
if err := m.fbNetDisconnect(target, traefikNetwork); err != nil {
m.logger.Printf("[WARN] [infra] filebrowser: could not leave %s — every app can still reach it (R-1281): %v", traefikNetwork, err)
m.reportFileBrowserNotLeft(target, err)
} else {
m.logger.Printf("[INFO] [infra] filebrowser: left %s — no app container reaches it any more (R-1281)", traefikNetwork)
}
@@ -117,6 +117,27 @@ func (m *Manager) ensureFileBrowserPublic() {
m.ensureOfficeOnFilesNetwork()
}
// reportFileBrowserNotLeft says what a failed leave means (R-1323). „No such container" means the CHECKED container was
// replaced between the check and the leave (the image moved at start, a mounts sync): the line then reads the CURRENT
// container's networks and says the truth. A false „every app can still reach it" teaches the reader to ignore the real
// one, which stays: any other failure, a current container on the app network, or one that cannot be read. The new
// container is checked and published by the next round. Pinned by TestR1323_*.
func (m *Manager) reportFileBrowserNotLeft(target string, err error) {
if !strings.Contains(err.Error(), "No such container") {
m.logger.Printf("[WARN] [infra] filebrowser: could not leave %s — every app can still reach it (R-1281): %v", traefikNetwork, err)
return
}
cur := strings.Fields(m.fbStartedAt())
switch {
case len(cur) == 0:
m.logger.Printf("[WARN] [infra] filebrowser: the checked container %s is gone and the current one cannot be read — whether every app can reach it is not known (R-1281, R-1323)", target)
case m.fbNetOn(cur[0], traefikNetwork):
m.logger.Printf("[WARN] [infra] filebrowser: could not leave %s — every app can still reach it (R-1281): the checked container %s was replaced, and the current one %s IS on it", traefikNetwork, target, cur[0])
default:
m.logger.Printf("[INFO] [infra] filebrowser: the checked container %s was replaced before it left %s; the current one %s is not on %s (R-1323)", target, traefikNetwork, cur[0], traefikNetwork)
}
}
// traefikContainer is traefik's container (infra/templates/traefik-compose.yml.tmpl container_name).
const traefikContainer = "traefik"
+24 -1
View File
@@ -254,8 +254,28 @@ func changedInfraFiles(dir string, files map[string]infra.FileSpec) []string {
return changed
}
// LockFileBrowserStack takes the lock over every `docker compose` on FileBrowser's stack at a boot (R-1351): this file's
// ensureFileBrowser and web.syncFileBrowserMounts. Returns the unlock. Lock order: infraMu (the base stack) before this;
// the web sync takes only this one, and starts the gate (infraMu) in a goroutine. Pinned by TestR1351_* and
// TestR1351_TheWebSyncTakesTheLockBeforeCompose (internal/web).
func (m *Manager) LockFileBrowserStack() (unlock func()) {
m.fbStackMu.Lock()
return m.fbStackMu.Unlock
}
// WithFileBrowserStack runs fn under LockFileBrowserStack.
func (m *Manager) WithFileBrowserStack(fn func()) {
defer m.LockFileBrowserStack()()
fn()
}
func (m *Manager) ensureFileBrowser(dir string) error {
if containerRunning("filebrowser") {
defer m.LockFileBrowserStack()()
running := containerRunning
if m.fbRunningFn != nil {
running = func(string) bool { return m.fbRunningFn() }
}
if running("filebrowser") {
return m.reconcileFileBrowserImage(dir)
}
composePath := filepath.Join(dir, "docker-compose.yml")
@@ -263,6 +283,9 @@ func (m *Manager) ensureFileBrowser(dir string) error {
// Already provisioned but not running — bring it up WITHOUT regenerating, so the storage
// mounts that web.SyncFileBrowserMounts manages are preserved. Just `compose up -d`.
m.logger.Printf("[INFO] [infra] filebrowser compose exists — starting without regenerating")
if m.fbComposeUpFn != nil {
return m.fbComposeUpFn(dir)
}
return m.composeUp(dir)
}
m.logger.Printf("[INFO] [infra] deploying filebrowser → %s", dir)
@@ -0,0 +1,186 @@
package stacks
import (
"context"
"fmt"
"sort"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/dockerexec"
"gitea.dooplex.hu/admin/felhom-controller/internal/util"
)
// ── Infrastructure image retention (`09` §3 decision 285, operator 2026-11-09) ──────────────────────────
//
// Decision 53 cleans app images and decision 56 the controller's own; the pinned infrastructure images (traefik,
// cloudflared, FileBrowser, Samba — infra.Images()) were cleaned by nothing, so every pin a box ever ran stayed:
// on demo-hp's guest on 2026-11-09, 2 old traefik, 3 old cloudflared, 2 old FileBrowser and 1 old Samba image
// (about 1 GB). Every one of them rides the whole-guest backup (mp0 holds /var/lib/docker), on a system disk the
// local tier already struggles to fit (R-548).
//
// THE RULE, decision 56's shape applied per pinned repository: a box keeps the PINNED tag (what this controller
// runs), every version ABOVE it (a box rolled back to an older controller keeps the newer pin's image for the way
// forward), the highest version BELOW it (the previous pin — a hand roll-back), and every image a container uses
// (running or stopped). An untagged image of the repository that no container uses is a candidate; a tag that is
// not a version (latest) is left alone. Deleted by exact name/ID, never forced, never by prune; an ID that also
// carries another repository's name is left alone. No pass while an image-pulling compose command runs (R-863),
// and none while the controller swaps itself. Pinned by internal/stacks/infra_image_retention_test.go.
// RetainInfraImages applies the rule once to every pinned infrastructure image. Returns what it deleted, and
// errImageRetentionBusy when image work was in flight (nothing was judged; try again later).
func (m *Manager) RetainInfraImages(pins []string) ([]string, error) {
imageRetentionMu.Lock()
defer imageRetentionMu.Unlock()
if m.selfUpdatingNow() {
m.logger.Printf("[INFO] [stacks] infra image retention: skipped — the controller is swapping itself")
return nil, errImageRetentionBusy
}
endCleanup, ok := dockerexec.TryImageCleanup()
if !ok {
m.logger.Printf("[INFO] [stacks] infra image retention: skipped — an app install, update, restore or undo is pulling images now (R-863); tried again later")
return nil, errImageRetentionBusy
}
defer endCleanup()
imgs, err := listLocalImages()
if err != nil {
return nil, err
}
used, err := imagesUsedByContainers()
if err != nil {
m.logger.Printf("[WARN] [stacks] infra image retention: the containers could not be read (%v) — NOTHING is deleted", err)
return nil, err
}
var deleted []string
for _, pin := range pins {
deleted = append(deleted, m.retainOneInfraRepo(pin, imgs, used)...)
}
return deleted, nil
}
func (m *Manager) retainOneInfraRepo(pin string, imgs []localImage, used map[string]bool) []string {
r, tag, _ := splitRepoTag(pin)
repo := normRepo(r)
pinned, err := util.ParseVersionPre(tag)
if err != nil {
m.logger.Printf("[INFO] [stacks] infra image retention: %s — the pin %q is not a version; the repository is left alone", repo, tag)
return nil
}
byID := map[string][]localImage{}
var prev *util.Version
for _, im := range imgs {
byID[im.ID] = append(byID[im.ID], im)
if im.Repo != repo {
continue
}
if v, err := util.ParseVersionPre(im.Tag); err == nil && v.Compare(pinned) < 0 && (prev == nil || v.Compare(*prev) > 0) {
vv := v
prev = &vv
}
}
prevTag := ""
if prev != nil {
prevTag = prev.Raw
}
ids := make([]string, 0, len(byID))
for id := range byID {
ids = append(ids, id)
}
sort.Strings(ids)
var deleted []string
considered, candidates := 0, 0
defer func() {
// ONE line per repository per pass, whatever it did — the positive observable that the pass ran (R-96 rule 3).
m.logger.Printf("[INFO] [stacks] infra image retention: %s — pass over %d image(s), pinned %s, previous %q, %d candidate(s), %d deleted, the rest kept",
repo, considered, tag, prevTag, candidates, len(deleted))
}()
for _, id := range ids {
group := byID[id]
var mine []localImage
other := false
for _, im := range group {
if im.Repo == repo {
mine = append(mine, im)
} else {
other = true
}
}
if len(mine) == 0 {
continue
}
considered++
if used[id] {
continue
}
deletable, odd := true, ""
var names []string
for _, im := range mine {
if im.Tag == "<none>" || im.Tag == "" {
continue
}
names = append(names, im.Repo+":"+im.Tag)
v, err := util.ParseVersionPre(im.Tag)
switch {
case err != nil:
deletable, odd = false, im.Tag // latest: not a version, not ours to judge
case v.Compare(pinned) >= 0:
deletable = false // the pin, or a newer pin's image
case prev == nil || v.Compare(*prev) >= 0:
deletable = false // the previous pin
}
}
if !deletable {
if odd != "" {
m.logger.Printf("[INFO] [stacks] infra image retention: %s carries a tag that is not a version (%s) — left alone", shortID(id), odd)
}
continue
}
candidates++
if other {
m.logger.Printf("[INFO] [stacks] infra image retention: %s also carries another repository's name — left alone", shortID(id))
continue
}
targets := names
if len(targets) == 0 {
targets = []string{id}
}
ok := true
for _, t := range targets {
if out, err := imageDocker("rmi", t); err != nil {
m.logger.Printf("[WARN] [stacks] infra image retention: docker refused to delete %s (%s): %s", t, shortID(id), truncateStr(strings.TrimSpace(out), 160))
ok = false
break
}
}
if !ok {
continue
}
label := strings.Join(names, ",")
if label == "" {
label = repo + ":<none>"
}
m.logger.Printf("[INFO] [stacks] infra image retention: deleted %s (%s, %s) — older than the previous pin and no container uses it (decision 285)", label, shortID(id), mine[0].Size)
deleted = append(deleted, fmt.Sprintf("%s@%s", label, shortID(id)))
}
return deleted
}
// RunInfraImageRetention runs RetainInfraImages, and again every `every` while image work is in flight, at most
// `tries` times (R-863's retry shape: a household's first installs land in the minutes after a start).
func (m *Manager) RunInfraImageRetention(ctx context.Context, pins []string, every time.Duration, tries int) {
for i := 0; i < tries; i++ {
_, err := m.RetainInfraImages(pins)
if err != errImageRetentionBusy {
if err != nil {
m.logger.Printf("[WARN] [stacks] infra image retention: %v — tried again at the next start", err)
}
return
}
select {
case <-ctx.Done():
return
case <-time.After(every):
}
}
m.logger.Printf("[WARN] [stacks] infra image retention: not run after %d tries — tried again at the next start", tries)
}
@@ -0,0 +1,128 @@
package stacks
import (
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/dockerexec"
)
// Decision 285 (2026-11-09): the pinned infrastructure images keep the pin, every newer version, the previous pin and
// every image a container uses; older ones go. The fixture is demo-hp's guest as measured on 2026-11-09 (before the
// hand clean-up): traefik v3.6.7/v3.7.13/v3.7.16 beside the pinned v3.7.17, FileBrowser 1.3.3/1.5.6 beside 1.5.8.
// Docker is the imageDocker seam (fakeImages) — nothing here reaches a daemon.
var infraPins = []string{"traefik:v3.7.17", "gtstef/filebrowser:1.5.8-stable", "gitea.dooplex.hu/admin/felhom-samba:1.2.0"}
func infraImages() *fakeImages {
return &fakeImages{
imgs: []localImage{
{ID: "sha256:T3717", Repo: "traefik", Tag: "v3.7.17", Size: "195MB"},
{ID: "sha256:T3716", Repo: "traefik", Tag: "v3.7.16", Size: "194MB"},
{ID: "sha256:T3713", Repo: "traefik", Tag: "v3.7.13", Size: "195MB"},
{ID: "sha256:T367", Repo: "traefik", Tag: "v3.6.7", Size: "186MB"},
{ID: "sha256:TNONE", Repo: "traefik", Tag: "<none>", Size: "190MB"},
{ID: "sha256:F158", Repo: "gtstef/filebrowser", Tag: "1.5.8-stable", Size: "196MB"},
{ID: "sha256:F156", Repo: "gtstef/filebrowser", Tag: "1.5.6-stable", Size: "195MB"},
{ID: "sha256:F133", Repo: "gtstef/filebrowser", Tag: "1.3.3-stable", Size: "215MB"},
// Samba is pinned but not enabled on this box (no container): the pin itself must stay.
{ID: "sha256:S120", Repo: "gitea.dooplex.hu/admin/felhom-samba", Tag: "1.2.0", Size: "107MB"},
{ID: "sha256:S110", Repo: "gitea.dooplex.hu/admin/felhom-samba", Tag: "1.1.0", Size: "103MB"},
{ID: "sha256:S100", Repo: "gitea.dooplex.hu/admin/felhom-samba", Tag: "1.0.0", Size: "101MB"},
// not an infra repository: never touched by this pass
{ID: "sha256:A1", Repo: "acme/web", Tag: "1", Size: "100MB"},
{ID: "sha256:ALP", Repo: "alpine", Tag: "latest", Size: "8MB"},
},
containers: map[string]string{"c-traefik": "sha256:T3717", "c-fb": "sha256:F158"},
}
}
// COMPANION RED-PROOF: drop the `prev` case from the keep switch → "the previous pin traefik:v3.7.16 was deleted";
// drop the `v >= pinned` case → "the pinned Samba image (no container) was deleted".
func TestInfraRetention_KeepsPinAndPreviousDeletesOlder(t *testing.T) {
m := retentionManager(t)
f := infraImages()
withFakeImages(t, f)
got, err := m.RetainInfraImages(infraPins)
if err != nil {
t.Fatal(err)
}
s := rmiSet(f)
for _, keep := range []string{"traefik:v3.7.17", "traefik:v3.7.16", "gtstef/filebrowser:1.5.8-stable", "gtstef/filebrowser:1.5.6-stable"} {
if strings.Contains(s, ","+keep+",") {
t.Fatalf("the previous pin %s was deleted: %v", keep, f.rmi)
}
}
if strings.Contains(s, "felhom-samba:1.2.0") || strings.Contains(s, "felhom-samba:1.1.0") {
t.Fatalf("the pinned Samba image (no container) or its previous pin was deleted: %v", f.rmi)
}
for _, want := range []string{"traefik:v3.7.13", "traefik:v3.6.7", "sha256:TNONE", "gtstef/filebrowser:1.3.3-stable", "gitea.dooplex.hu/admin/felhom-samba:1.0.0"} {
if !strings.Contains(s, ","+want+",") {
t.Fatalf("%s (older than the previous pin / untagged) was not deleted: rmi=%v", want, f.rmi)
}
}
if strings.Contains(s, "acme/web") || strings.Contains(s, "sha256:A1") || strings.Contains(s, "alpine") {
t.Fatalf("an image outside the pinned repositories was touched: %v", f.rmi)
}
if len(got) != 5 {
t.Fatalf("deleted %d, want 5: %v", len(got), got)
}
}
// A box rolled back to an older controller pins an OLDER traefik: the newer one stays (the way forward), and a
// container that still runs an old image keeps it.
// COMPANION RED-PROOF: drop the `used[id]` check → "an image a container uses was deleted".
func TestInfraRetention_NewerPinAndInUseKept(t *testing.T) {
m := retentionManager(t)
f := infraImages()
f.containers["c-old-fb"] = "sha256:F133"
withFakeImages(t, f)
if _, err := m.RetainInfraImages([]string{"traefik:v3.7.16", "gtstef/filebrowser:1.5.8-stable"}); err != nil {
t.Fatal(err)
}
s := rmiSet(f)
if strings.Contains(s, "traefik:v3.7.17") {
t.Fatalf("a version above the pin was deleted: %v", f.rmi)
}
if strings.Contains(s, "traefik:v3.7.13") || !strings.Contains(s, "traefik:v3.6.7") {
t.Fatalf("pinned v3.7.16: want v3.7.13 kept (previous) and v3.6.7 deleted: %v", f.rmi)
}
if strings.Contains(s, "1.3.3-stable") {
t.Fatalf("an image a container uses was deleted: %v", f.rmi)
}
}
// While an image-pulling compose command runs (R-863), or the controller swaps itself, NOTHING is deleted.
// COMPANION RED-PROOF: drop the TryImageCleanup check → "deleted while image work was in flight".
func TestInfraRetention_NothingWhileImageWorkOrSwap(t *testing.T) {
m := retentionManager(t)
f := infraImages()
withFakeImages(t, f)
end := dockerexec.HoldImageWork()
_, err := m.RetainInfraImages(infraPins)
end()
if err != errImageRetentionBusy || len(f.rmi) != 0 {
t.Fatalf("deleted while image work was in flight: err=%v rmi=%v", err, f.rmi)
}
m.SetSelfUpdatingCheck(func() bool { return true })
_, err = m.RetainInfraImages(infraPins)
m.SetSelfUpdatingCheck(func() bool { return false })
if err != errImageRetentionBusy || len(f.rmi) != 0 {
t.Fatalf("deleted while the controller swaps itself: err=%v rmi=%v", err, f.rmi)
}
}
// A pin that is not a version, and a tag that is not a version, are left alone.
func TestInfraRetention_NonVersionsLeftAlone(t *testing.T) {
m := retentionManager(t)
f := infraImages()
f.imgs = append(f.imgs, localImage{ID: "sha256:TLATEST", Repo: "traefik", Tag: "latest", Size: "190MB"})
withFakeImages(t, f)
if _, err := m.RetainInfraImages([]string{"traefik:v3.7.17", "gtstef/filebrowser:latest"}); err != nil {
t.Fatal(err)
}
s := rmiSet(f)
if strings.Contains(s, "traefik:latest") || strings.Contains(s, "filebrowser") {
t.Fatalf("a non-version tag or a repository with a non-version pin was touched: %v", f.rmi)
}
}
+7
View File
@@ -284,6 +284,8 @@ type Manager struct {
// R-514: last OOM scan (oom.go), for the dashboard.
oomMu sync.Mutex
oomCache map[string][]string
// hostOOMKills (R-528) is the host agent's per-container OOM counter; nil = Docker's flag only (SetHostOOMKills).
hostOOMKills func() map[string]int64
// R-513: FileBrowser admin-password seams (filebrowser_password.go); nil/"" in production.
fbHTTP fbHTTPDo
fbBaseURL string
@@ -293,6 +295,11 @@ type Manager struct {
fbProbedRun string
// fbNetOnFn / fbNetConnectFn replace the docker network reads and connects of the R-1126 gate in tests.
fbNetOnFn func(container, network string) bool
// fbStackMu (R-1351) is the lock over `docker compose` on FileBrowser's stack: LockFileBrowserStack. fbRunningFn and
// fbComposeUpFn replace the docker read and the bring-up of ensureFileBrowser in tests.
fbStackMu sync.Mutex
fbRunningFn func() bool
fbComposeUpFn func(dir string) error
fbNetConnectFn func(container, network string) error
// R-1281: the disconnect, the network create, and „is OnlyOffice linked to FileBrowser and running" — test seams.
fbNetDisconnectFn func(container, network string) error
+47 -4
View File
@@ -27,6 +27,21 @@ type OOMContainer struct {
Kills int64
// MemLimit / Peak are the cgroup's memory.max and memory.peak, as "<n>M" (or "max"); "" unread.
MemLimit, Peak string
// Source names what flagged it: OOMSourceDocker (State.OOMKilled), OOMSourceCgroup (R-528 — only the host's
// per-container kernel counter saw it; Docker's flag stayed false), or "docker,cgroup" (both).
Source string
}
const (
OOMSourceDocker = "docker"
OOMSourceCgroup = "cgroup"
)
// SetHostOOMKills wires the host-side OOM counter (R-528): container id → the kernel's oom_kill for the container's
// CURRENT run (its cgroup scope is new at each start), read by the host's agent. nil, or a nil answer (no agent, an
// agent without the route, a failed call) = Docker's flag only, as before. INIT-ONLY (main.go).
func (m *Manager) SetHostOOMKills(fn func() map[string]int64) {
m.hostOOMKills = fn
}
// ScanOOMKilled inspects the containers of every deployed, running-ish stack in ONE docker call and
@@ -52,20 +67,45 @@ func (m *Manager) ScanOOMKilled() ([]OOMContainer, error) {
return nil, nil
}
sort.Strings(names)
args := append([]string{"inspect", "-f", "{{.Name}}|{{.State.OOMKilled}}|{{.State.StartedAt}}"}, names...)
args := append([]string{"inspect", "-f", "{{.Name}}|{{.State.OOMKilled}}|{{.State.StartedAt}}|{{.Id}}"}, names...)
out, err := m.execCommand("docker", args...)
if err != nil && strings.TrimSpace(out) == "" {
return nil, err
}
// R-528: the host's counter, asked once per scan (an HTTPS call to the agent); nil = Docker's flag only.
var host map[string]int64
if m.hostOOMKills != nil {
host = m.hostOOMKills()
}
var found []OOMContainer
hostKills := map[int]int64{}
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
f := strings.SplitN(strings.TrimSpace(line), "|", 3)
if len(f) != 3 || f[1] != "true" {
f := strings.SplitN(strings.TrimSpace(line), "|", 4)
if len(f) < 3 {
continue
}
id := ""
if len(f) == 4 {
id = strings.TrimSpace(f[3])
}
hk, counted := host[id]
source := ""
switch {
case f[1] == "true" && id != "" && counted && hk > 0:
source = OOMSourceDocker + "," + OOMSourceCgroup // both saw it
case f[1] == "true":
source = OOMSourceDocker
case id != "" && counted && hk > 0:
source = OOMSourceCgroup
default:
continue
}
cname := strings.TrimPrefix(f[0], "/")
if st, ok := owner[cname]; ok {
found = append(found, OOMContainer{Stack: st, Container: cname, StartedAt: f[2], Kills: -1})
if counted && id != "" {
hostKills[len(found)] = hk
}
found = append(found, OOMContainer{Stack: st, Container: cname, StartedAt: f[2], Kills: -1, Source: source})
}
}
// R-636: only for the containers already flagged — one `docker exec` each, and the flagged set is
@@ -75,6 +115,9 @@ func (m *Manager) ScanOOMKilled() ([]OOMContainer, error) {
out, _ := m.execCommand("docker", "exec", found[i].Container, "cat",
"/sys/fs/cgroup/memory.events", "/sys/fs/cgroup/memory.max", "/sys/fs/cgroup/memory.peak")
found[i].Kills, found[i].MemLimit, found[i].Peak = parseCgroupMemory(out)
if hk, ok := hostKills[i]; ok && hk > found[i].Kills {
found[i].Kills = hk // the host's count is the kernel's own; the in-container read can be -1 (no cat)
}
}
m.setOOMCache(found)
return found, nil
@@ -0,0 +1,78 @@
package stacks
import (
"bytes"
"errors"
"log"
"net/http/httptest"
"strings"
"testing"
)
// R-1323: the checked FileBrowser container can be replaced between the check and the leave (the image moved at start).
// „No such container" then means the OLD one is gone, not that every app still reaches FileBrowser. The line reads the
// CURRENT container's networks and says what is true; a current container that IS on the app network keeps the alarm.
//
// RED-PROOF (2026-11-02): before the fix the first case logged „could not leave traefik-public — every app can still reach
// it … No such container" (seen on 9202, 0.346.1-rc.1, 2026-10-27), and this fails on that line.
func r1323Rig(t *testing.T, currentOnApps bool) (*Manager, *bytes.Buffer) {
t.Helper()
fb := &fakeFB{password: "admin"}
srv := httptest.NewServer(fb.handler())
t.Cleanup(srv.Close)
m, _, _ := fbManager(t, srv.URL)
var buf bytes.Buffer
m.logger = log.New(&buf, "", 0)
runs := []string{"old111 2026-10-27T14:58:15Z"}
m.fbStarted = func() string {
r := runs[0]
if len(runs) > 1 {
runs = runs[1:]
}
return r
}
nets := map[string]bool{"old111@traefik-public": true, "new222@traefik-public": currentOnApps}
m.fbNetOnFn = func(c, n string) bool { return nets[c+"@"+n] }
m.fbNetConnectFn = func(c, n string) error { nets[c+"@"+n] = true; return nil }
m.fbNetDisconnectFn = func(c, n string) error {
if c == "old111" {
runs = []string{"new222 2026-10-27T14:58:17Z"} // the recreate won the race
return errors.New("Error response from daemon: No such container: old111")
}
return nil
}
m.fbNetEnsureFn = func(string) error { return nil }
m.fbOfficeLinkedFn = func() (bool, bool) { return false, false }
return m, &buf
}
func TestR1323_AReplacedContainerIsNotReportedAsExposed(t *testing.T) {
m, buf := r1323Rig(t, false)
m.fileBrowserGateLocked()
got := buf.String()
if strings.Contains(got, "every app can still reach it") {
t.Fatalf("a false alarm for a container that is gone: %q", got)
}
if !strings.Contains(got, "new222") || !strings.Contains(got, "not on traefik-public") {
t.Fatalf("the line must name the current container and the truth: %q", got)
}
}
func TestR1323_ACurrentContainerOnTheAppNetworkKeepsTheAlarm(t *testing.T) {
m, buf := r1323Rig(t, true)
m.fileBrowserGateLocked()
got := buf.String()
if !strings.Contains(got, "every app can still reach it") || !strings.Contains(got, "new222") {
t.Fatalf("the replacement IS on the app network: the alarm stays, naming it: %q", got)
}
}
// Any other failure to leave keeps the alarm as before (control).
func TestR1323_AnotherDisconnectFailureKeepsTheAlarm(t *testing.T) {
m, buf := r1323Rig(t, false)
m.fbNetDisconnectFn = func(c, n string) error { return errors.New("daemon busy") }
m.fileBrowserGateLocked()
if got := buf.String(); !strings.Contains(got, "every app can still reach it") || !strings.Contains(got, "daemon busy") {
t.Fatalf("a real failure to leave must keep its alarm: %q", got)
}
}
@@ -0,0 +1,40 @@
package stacks
import (
"fmt"
"io"
"log"
"testing"
"time"
)
// R-1347, the consequence: an app whose last update ended held is not watched for a crash loop; once its hold is lifted by
// a restore and the flag forgotten (backup.clearUpdateHoldAfterRestore → ForgetUpdateHeld) the same restarts trip the
// watcher again.
func TestR1347_TheWatcherWatchesAgainOnceTheFlagIsForgotten(t *testing.T) {
m := &Manager{logger: log.New(io.Discard, "", 0), stacks: map[string]*Stack{
"nextcloud": {Name: "nextcloud", Deployed: true, Containers: []ContainerInfo{{Name: "nextcloud-web-1"}}},
}}
m.markUpdateHeld("nextcloud")
count := int64(0)
m.execFn = func(string, ...string) (string, error) { return fmt.Sprintf("/nextcloud-web-1|%d\n", count), nil }
t0 := time.Date(2026, 11, 2, 4, 20, 0, 0, time.UTC)
sample := func(from int) []UnhealthyVerdict {
var got []UnhealthyVerdict
for i := from; i < from+12 && len(got) == 0; i++ {
count += 3
got = m.ObserveUnhealthy(t0.Add(time.Duration(i)*time.Minute), nil)
}
return got
}
if got := sample(0); len(got) != 0 {
t.Fatalf("control: an app whose update ended held must not be judged: %+v", got)
}
if !m.ForgetUpdateHeld("nextcloud") {
t.Fatal("the flag was set, so it must be forgotten")
}
got := sample(20)
if len(got) != 1 || got[0].Stack != "nextcloud" || got[0].Kind != UnhealthyCrashLoop {
t.Fatalf("after the restore lifted the hold the watcher must watch the app again; verdicts %+v", got)
}
}
@@ -0,0 +1,58 @@
package stacks
import (
"io"
"log"
"os"
"path/filepath"
"sync"
"testing"
"time"
)
// R-1351: at a guest boot EnsureBaseStack's `compose up` and web.SyncFileBrowserMounts' `compose up --force-recreate` ran
// in the same second; one lost with „Conflict. The container name … is already in use" and a false first-boot WARN. The
// two paths now share one lock: the base stack's bring-up holds it around its FileBrowser step, the web sync around its
// compose call (TestFileBrowserSyncRunsInsideTheStackLock pins that side, internal/web).
//
// RED-PROOF (2026-11-02): seen failing before the lock existed (build: WithFileBrowserStack); with ensureFileBrowser not
// taking the lock it fails on „overlap".
func TestR1351_TheTwoFileBrowserStartPathsNeverOverlap(t *testing.T) {
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "docker-compose.yml"), []byte("services: {}\n"), 0o644); err != nil {
t.Fatal(err)
}
m := &Manager{logger: log.New(io.Discard, "", 0)}
var mu sync.Mutex
inside, overlaps := 0, 0
enter := func() {
mu.Lock()
inside++
if inside > 1 {
overlaps++
}
mu.Unlock()
time.Sleep(30 * time.Millisecond)
mu.Lock()
inside--
mu.Unlock()
}
m.fbComposeUpFn = func(string) error { enter(); return nil }
m.fbRunningFn = func() bool { return false }
var wg sync.WaitGroup
for i := 0; i < 4; i++ {
wg.Add(2)
go func() {
defer wg.Done()
if err := m.ensureFileBrowser(dir); err != nil {
t.Error(err)
}
}()
go func() { defer wg.Done(); m.WithFileBrowserStack(enter) }()
}
wg.Wait()
if overlaps != 0 {
t.Fatalf("overlap: the two start paths ran compose at the same time %d time(s) — the „Conflict” of R-1351", overlaps)
}
}
@@ -0,0 +1,84 @@
package stacks
import (
"io"
"log"
"strings"
"testing"
)
// R-528 — inside a Felhom LXC guest Docker can leave State.OOMKilled false after a memory-limit kill (measured
// 2026-09-15/16). The host's agent reads the kernel's own per-container counter (GET /guest/oom-kills); a counter above
// 0 for the RUNNING container's id is an OOM kill in this run, whatever Docker says. The scan flags it with
// Source "cgroup" and the host's count, and the existing app_oom line (once per run) fires for it.
const r528ID = "4f5e6d7c8b9a0f1e2d3c4b5a69788796a5b4c3d2e1f00112233445566778899a"
// RED-PROOF: ignore the host counter in ScanOOMKilled (only `OOMKilled == true` flags) → the killed container is not
// seen → FAILS.
func TestR528_AHostCounterAboveZeroIsAnOOMKillWhateverDockerSays(t *testing.T) {
var execd []string
m := &Manager{logger: log.New(io.Discard, "", 0), stacks: map[string]*Stack{
"hog": {Name: "hog", Deployed: true, Containers: []ContainerInfo{{Name: "hog", State: StateRunning}}},
"bookstack": {Name: "bookstack", Deployed: true, Containers: []ContainerInfo{{Name: "bookstack", State: StateRunning}}},
}}
m.execFn = func(name string, args ...string) (string, error) {
if args[0] == "exec" {
execd = append(execd, args[1])
return "oom_kill 1\n67108864\n67100000\n", nil
}
if !strings.Contains(strings.Join(args, " "), "{{.Id}}") {
t.Errorf("the inspect must ask for the container id: %v", args)
}
return "/bookstack|false|t0|" + strings.Repeat("0", 64) + "\n/hog|false|2026-10-28T23:00:00Z|" + r528ID + "\n", nil
}
m.SetHostOOMKills(func() map[string]int64 {
return map[string]int64{r528ID: 2, strings.Repeat("0", 64): 0}
})
ooms, err := m.ScanOOMKilled()
if err != nil {
t.Fatal(err)
}
if len(ooms) != 1 || ooms[0].Stack != "hog" || ooms[0].Container != "hog" {
t.Fatalf("the host-counted kill was not seen: %+v", ooms)
}
if ooms[0].Source != OOMSourceCgroup || ooms[0].Kills != 2 || ooms[0].StartedAt != "2026-10-28T23:00:00Z" {
t.Fatalf("want source cgroup, the host's count 2 and the run's start: %+v", ooms[0])
}
if ooms[0].MemLimit != "64M" {
t.Errorf("limit/peak are still read for the flagged container: %+v", ooms[0])
}
if len(execd) != 1 || execd[0] != "hog" {
t.Errorf("exec must reach the flagged container only, got %v", execd)
}
}
// Docker's flag alone still flags (Source docker); no host source (an old agent, 9202 without an agent) = today's
// behaviour; a host counter of 0 never flags.
func TestR528_WithoutAHostCounterTheScanIsAsBefore(t *testing.T) {
m := &Manager{logger: log.New(io.Discard, "", 0), stacks: map[string]*Stack{
"a": {Name: "a", Deployed: true, Containers: []ContainerInfo{{Name: "a", State: StateRunning}}},
"b": {Name: "b", Deployed: true, Containers: []ContainerInfo{{Name: "b", State: StateRunning}}},
}}
m.execFn = func(name string, args ...string) (string, error) {
if args[0] == "exec" {
return "oom_kill 3\n", nil
}
return "/a|true|s1|" + strings.Repeat("a", 64) + "\n/b|false|s2|" + strings.Repeat("b", 64) + "\n", nil
}
ooms, _ := m.ScanOOMKilled()
if len(ooms) != 1 || ooms[0].Stack != "a" || ooms[0].Source != OOMSourceDocker || ooms[0].Kills != 3 {
t.Fatalf("no host source: %+v", ooms)
}
m.SetHostOOMKills(func() map[string]int64 { return map[string]int64{strings.Repeat("b", 64): 0} })
ooms, _ = m.ScanOOMKilled()
if len(ooms) != 1 || ooms[0].Stack != "a" {
t.Fatalf("a host counter of 0 flagged a container: %+v", ooms)
}
// Both saw it: the source names both, the host's count wins over a lower in-container read.
m.SetHostOOMKills(func() map[string]int64 { return map[string]int64{strings.Repeat("a", 64): 7} })
ooms, _ = m.ScanOOMKilled()
if len(ooms) != 1 || ooms[0].Source != OOMSourceDocker+","+OOMSourceCgroup || ooms[0].Kills != 7 {
t.Fatalf("both sources: %+v", ooms)
}
}
+19
View File
@@ -209,6 +209,25 @@ func (m *Manager) ClearUpdateState(name string) {
s.HealthProbe = nil
}
// ForgetUpdateHeld (D64 = B, R-1343) forgets that the app's last update ended HELD, once the hold itself is cleared
// by the operator. updateHeld is the ONE held flag kept outside the hold store: it makes the crash-loop observer
// (ObserveUnhealthy) and the nightly update leg (legCandidate) skip the app, and keeps the hold's sentence as the
// update error. Left set, an app whose hold was cleared would stay unobserved and never be updated by night until
// the controller restarts. A running update is not touched. Returns whether anything was forgotten.
// Pinned by TestD64_ForgetUpdateHeld (this package) and TestD64_ClearHoldViaTheHub (cmd/controller).
func (m *Manager) ForgetUpdateHeld(name string) bool {
m.mu.Lock()
defer m.mu.Unlock()
s, ok := m.stacks[name]
if !ok || s.Updating || (!s.updateHeld && s.UpdateErrorKey != UpdateErrorKeyHeld) {
return false
}
s.updateHeld = false
s.UpdatePhase, s.UpdatePhaseLabel = "", ""
s.UpdateError, s.UpdateErrorKey, s.UpdateErrorArgs = "", "", nil
return true
}
func (m *Manager) markUpdateHeld(name string) {
m.mu.Lock()
if s, ok := m.stacks[name]; ok {
@@ -0,0 +1,248 @@
package web
import (
"encoding/json"
"html/template"
"io"
"log"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// D62 (operator ruling 2026-10-29): restore copies in the kept-copies list with „Törlés most". Real settings, manager,
// disk and a hub stand-in that records the event — the handler is the endpoint the page's form calls.
type d62Fixture struct {
s *Server
sett *settings.Settings
nsRoot string
events chan map[string]interface{}
}
func newD62Fixture(t *testing.T) *d62Fixture {
t.Helper()
tmp := t.TempDir()
lg := log.New(io.Discard, "", 0)
sett, err := settings.Load(filepath.Join(tmp, "settings.json"), lg)
if err != nil {
t.Fatal(err)
}
cfg := &config.Config{}
cfg.Paths.DataDir = tmp
cfg.Paths.SystemDataPath = filepath.Join(tmp, "sys")
events := make(chan map[string]interface{}, 8)
hub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/v1/event" {
var ev map[string]interface{}
_ = json.NewDecoder(r.Body).Decode(&ev)
events <- ev
}
w.WriteHeader(http.StatusOK)
}))
t.Cleanup(hub.Close)
n := notify.New(hub.URL, "test-key", "c1", sett, lg, false)
s := &Server{cfg: cfg, settings: sett, backupMgr: backup.NewManager(cfg, sett, lg), notifier: n, logger: lg}
return &d62Fixture{s: s, sett: sett, nsRoot: backup.NamespaceRootFor(cfg.Paths.SystemDataPath, cfg.Paths.SystemDataPath), events: events}
}
// plant writes one restore copy of app taken at `at`; returns its folder.
func (f *d62Fixture) plant(t *testing.T, app string, at time.Time) string {
t.Helper()
dir := filepath.Join(backup.RestoreUndoDir(f.nsRoot, app), at.UTC().Format("20060102T150405Z"))
if err := os.MkdirAll(dir, 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, app+"_data.tar"), []byte("LIVE-DATA"), 0o600); err != nil {
t.Fatal(err)
}
return dir
}
func (f *d62Fixture) post(form string) (int, map[string]interface{}) {
rr := httptest.NewRecorder()
req := httptest.NewRequest("POST", "/backup/restore-copy/delete", strings.NewReader(form))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.RemoteAddr = "192.168.0.5:5555"
f.s.restoreCopyDeleteHandler(rr, req)
var body map[string]interface{}
_ = json.Unmarshal(rr.Body.Bytes(), &body)
return rr.Code, body
}
func exists(p string) bool { _, err := os.Stat(p); return err == nil }
// The list shows the restore's date, the size and „automatikusan törlődik: <30 days later>"; a held copy shows why and
// offers no warning. RED-PROOF: drop the auto-delete part from restoreCopyRows → FAILS.
func TestD62_TheListShowsTheDateAndTheAutoDeleteDay(t *testing.T) {
f := newD62Fixture(t)
at := time.Date(2026, 10, 29, 8, 0, 0, 0, time.UTC)
f.plant(t, "vikunja", at)
f.plant(t, "immich", at)
if err := f.sett.SetRestoreHold(settings.RestoreHold{Stack: "immich", At: at.Format(time.RFC3339), Reason: settings.HoldReasonRestoreMixed}); err != nil {
t.Fatal(err)
}
rows := f.s.restoreCopyRows("hu")
if len(rows) != 2 {
t.Fatalf("rows %+v", rows)
}
var free, held restoreCopyRow
for _, r := range rows {
if r.App == "vikunja" {
free = r
} else {
held = r
}
}
for _, want := range []string{"visszaállítás: 2026. 10. 29. 09:00", "9 B", "automatikusan törlődik: 2026. 11. 28."} {
if !strings.Contains(free.Line, want) {
t.Errorf("free row line %q lacks %q", free.Line, want)
}
}
if !strings.Contains(free.Warning, "Ez a(z) vikunja adatainak másolata a 2026. 10. 29-i visszaállítás előttről") ||
!strings.Contains(free.Warning, "a visszaállítás többé nem vonható vissza, és ez a másolat nem hozható vissza") {
t.Errorf("warning %q", free.Warning)
}
if free.NeededText != "" || free.TypePrompt == "" {
t.Errorf("free row: needed %q prompt %q", free.NeededText, free.TypePrompt)
}
if held.NeededText == "" || held.Warning != "" || strings.Contains(held.Line, "automatikusan") {
t.Errorf("held row offers a delete or an auto-delete day: %+v", held)
}
en := f.s.restoreCopyRows("en")
for _, r := range en {
if r.App == "vikunja" && (!strings.Contains(r.Line, "automatically deleted on 2026-11-28") ||
!strings.Contains(r.Warning, "This is the copy of vikunja’s data from before the restore on 2026-10-29. If you delete it, the restore can no longer be undone, and this copy cannot be brought back.")) {
t.Errorf("English row %+v", r)
}
}
}
// The page: a deletable copy has the form (app, stamp, the typed name, CSRF); a held one has the button OFF and the
// reason, and no form. RED-PROOF: render the form for a needed copy too → FAILS.
func TestD62_ThePageOffersTheFormOnlyForADeletableCopy(t *testing.T) {
d := i18nLayoutData("backups-restore", "Visszaállítás a mentésből")
d["Backup"] = i18nBackupStatus(false, false)
d["RestoreCopies"] = []restoreCopyRow{
{App: "vikunja", Stamp: "20261029T080000Z", Name: "Vikunja", Line: "L1", Warning: "W1", TypePrompt: "P1"},
{App: "immich", Stamp: "20261020T080000Z", Name: "Immich", Line: "L2", NeededText: "HELD-REASON"},
}
d["CSRFField"] = template.HTML(`<input type="hidden" name="_csrf" value="TOKEN-D62">`)
html := renderBackupPage(t, "backups_restore", d)
for _, want := range []string{"Visszaállítás előtti másolatok", `action="/backup/restore-copy/delete"`, `name="app" value="vikunja"`,
`name="stamp" value="20261029T080000Z"`, `name="confirm"`, "W1", "P1", "Törlés most", `data-restore-copy-off="true"`, "HELD-REASON"} {
if !strings.Contains(html, want) {
t.Errorf("page lacks %q", want)
}
}
if strings.Contains(html, `name="app" value="immich"`) || strings.Contains(html, `data-restore-copy="immich/`) {
t.Error("a held copy got a delete form")
}
form := html[strings.Index(html, `action="/backup/restore-copy/delete"`):]
if !strings.Contains(form[:strings.Index(form, "</form>")], `value="TOKEN-D62"`) {
t.Error("the delete form carries no CSRF field")
}
none := i18nLayoutData("backups-restore", "Visszaállítás a mentésből")
none["Backup"] = i18nBackupStatus(false, false)
if html := renderBackupPage(t, "backups_restore", none); strings.Contains(html, "Megőrzött másolatok") || strings.Contains(html, "restore-copy") {
t.Error("a page with no copy shows the section")
}
}
// A wrong typed name is refused (400) and nothing is deleted. RED-PROOF: drop the name check → FAILS.
func TestD62_AWrongNameIsRefused(t *testing.T) {
f := newD62Fixture(t)
c := f.plant(t, "vikunja", time.Now().Add(-time.Hour))
for _, typed := range []string{"", "vikunj", "immich", "VIKUNJA "} {
code, body := f.post("app=vikunja&stamp=" + filepath.Base(c) + "&confirm=" + typed)
if code != http.StatusBadRequest || body["ok"] != false || !exists(c) {
t.Fatalf("typed %q: code %d body %v exists %v", typed, code, body, exists(c))
}
}
}
// A held copy is refused by the handler too (409, the reason), and stays. RED-PROOF: drop the Needed check in the
// handler AND in DeleteRestoreCopy → FAILS (either alone holds it).
func TestD62_AHeldCopyIsRefusedByTheHandler(t *testing.T) {
f := newD62Fixture(t)
c := f.plant(t, "immich", time.Now().Add(-time.Hour))
if err := f.sett.SetRestoreHold(settings.RestoreHold{Stack: "immich", At: time.Now().Format(time.RFC3339), Reason: settings.HoldReasonPutBackPending}); err != nil {
t.Fatal(err)
}
code, body := f.post("app=immich&stamp=" + filepath.Base(c) + "&confirm=immich")
if code != http.StatusConflict || !exists(c) || !strings.Contains(body["error"].(string), "félbeszakadt visszaállítás") {
t.Fatalf("code %d body %v exists %v", code, body, exists(c))
}
}
// While a restore runs, a delete is 409 with a sentence, and nothing moves. RED-PROOF: drop the busy check → FAILS.
func TestD62_NoDeleteWhileAnOperationRuns(t *testing.T) {
f := newD62Fixture(t)
c := f.plant(t, "vikunja", time.Now().Add(-time.Hour))
f.s.backupMgr.BeginRestoreOp("restore", "other")
code, body := f.post("app=vikunja&stamp=" + filepath.Base(c) + "&confirm=vikunja")
if code != http.StatusConflict || !exists(c) || body["error"] == "" {
t.Fatalf("code %d body %v exists %v", code, body, exists(c))
}
f.s.backupMgr.EndRestoreOp(true, "ok")
}
// The right name: the copy is gone, ONLY it (the app's other copy and another app's copy stay), the answer leads back
// to the page with the flash, and the operator event carries who, app, the restore's date and size.
// RED-PROOF: drop the NotifyRestoreCopyDeleted call → no event → FAILS.
func TestD62_ADeleteRemovesOnlyThatCopyAndStoresTheEvent(t *testing.T) {
f := newD62Fixture(t)
at := time.Date(2026, 10, 29, 8, 0, 0, 0, time.UTC)
c := f.plant(t, "vikunja", at)
sibling := f.plant(t, "vikunja", at.Add(time.Hour))
other := f.plant(t, "immich", at)
code, body := f.post("app=vikunja&stamp=" + filepath.Base(c) + "&confirm=vikunja")
if code != http.StatusOK || body["ok"] != true || !strings.Contains(body["redirect"].(string), "flash.restore_copy.deleted") {
t.Fatalf("code %d body %v", code, body)
}
if exists(c) || !exists(sibling) || !exists(other) {
t.Fatalf("gone=%v sibling=%v other=%v", !exists(c), exists(sibling), exists(other))
}
select {
case ev := <-f.events:
if ev["event_type"] != "restore_copy_deleted" || ev["severity"] != "info" {
t.Fatalf("event %v", ev)
}
det, _ := json.Marshal(ev["details"])
for _, want := range []string{`"app":"vikunja"`, `"restore_at":"2026-10-29T08:00:00Z"`, `"size_bytes":9`, `"who":"the household (dashboard sign-in) from 192.168.0.5"`} {
if !strings.Contains(string(det), want) {
t.Errorf("details %s lack %s", det, want)
}
}
case <-time.After(5 * time.Second):
t.Fatal("no restore_copy_deleted event reached the hub")
}
// a second press: the copy is gone → 404, no second event
if code, _ := f.post("app=vikunja&stamp=" + filepath.Base(c) + "&confirm=vikunja"); code != http.StatusNotFound {
t.Fatalf("second press code %d", code)
}
}
// R-1342: every reason a copy can be needed has its one-line sentence in both languages (the row's key is built from the
// reason, so a missing key would show the raw key). RED-PROOF: remove restore_copy.needed.checking from BOTH bundles →
// FAILS (seen 2026-10-29; a key missing only in en.json falls back to the Hungarian line — i18n_missing_gate counts that).
func TestR1342_EveryNeededReasonHasItsSentence(t *testing.T) {
f := newD62Fixture(t)
for _, why := range []string{backup.RestoreCopyNeededHold, backup.RestoreCopyNeededPutBack, backup.RestoreCopyNeededUnknown, backup.RestoreCopyNeededChecking} {
for _, lang := range []string{"hu", "en"} {
key := "restore_copy.needed." + why
if got := f.s.msgLang(lang, key); got == "" || got == key || strings.Contains(got, "restore_copy.") {
t.Errorf("%s/%s renders %q", lang, key, got)
}
}
}
}
@@ -352,6 +352,12 @@ func (s *Server) escrowConfirmAPIHandler(w http.ResponseWriter, r *http.Request)
escrowJSON(w, http.StatusGone, map[string]any{"restart": true}, s.msg(r, "api.escrow.confirm_expired"))
case http.StatusNotFound:
escrowJSON(w, http.StatusGone, map[string]any{"restart": true}, s.msg(r, "api.escrow.confirm_lost"))
case http.StatusGatewayTimeout, 0:
// R-1254 H2: the agent could not learn which package the hub holds (504), or this call itself failed (0 — it may
// have timed out while the agent still stored the package). Either way the new package MAY be on the hub: no
// promise about the previous code. Only 502 (the hub holds another package) keeps the store-failed sentence.
s.logger.Printf("[WARN] [web] escrow confirm outcome unknown (status %d)", status)
escrowJSON(w, http.StatusGatewayTimeout, nil, s.msg(r, "api.escrow.confirm_store_unknown"))
default:
s.logger.Printf("[WARN] [web] escrow confirm failed (status %d)", status)
escrowJSON(w, http.StatusBadGateway, nil, s.msg(r, "api.escrow.confirm_store_failed"))
@@ -1,6 +1,14 @@
package web
import "testing"
import (
"bytes"
"log"
"os"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// T4: skipFileBrowserPath skips ONLY an external drive path (under StableParentDir) that is not a live
// mountpoint. A mounted external path and any system/local path are never skipped.
@@ -49,3 +57,66 @@ func TestFbNeedsRecreate(t *testing.T) {
}
}
}
// R-1339: a per-app folder under a MOUNTED drive (/mnt/felhom-drives/x/userdata/<app>) is never a mountpoint itself. It is
// shown through the drive's own bind, so it still gets no bind and no skeleton of its own (a skeleton there would write
// userdata/ inside the app's folder), and the log no longer says its drive is „not mounted". A drive that really is
// absent keeps its line, and a folder of such a drive names the absent drive.
//
// RED-PROOF (2026-11-02): before the fix the sub-folder logged „drive …/userdata/nextcloud not mounted" (nine lines at every
// start on 9202, scratch_hdd a live mountpoint) and this fails on that line.
func TestR1339_AFolderOfAMountedDriveIsNotReportedAsUnmounted(t *testing.T) {
sub := StableParentDir + "/scratch_hdd/userdata/nextcloud"
mounted := StableParentDir + "/scratch_hdd"
absent := StableParentDir + "/usb"
absentSub := absent + "/userdata/romm"
isMount := func(p string) bool { return p == mounted }
var buf bytes.Buffer
var skeletons []string
mounts, cfg := buildFileBrowserPaths([]settings.StoragePath{{Path: sub}, {Path: mounted}, {Path: absent}, {Path: absentSub}}, fbPathDeps{
isMount: isMount,
ensureSkeleton: func(p string) error { skeletons = append(skeletons, p); return nil },
logger: log.New(&buf, "", 0),
})
logged := buf.String()
if strings.Contains(logged, sub) {
t.Errorf("a folder of a mounted drive is reported: %q", logged)
}
if !strings.Contains(logged, "drive "+absent+" not mounted") {
t.Errorf("a drive that is really absent keeps its line: %q", logged)
}
if !strings.Contains(logged, "drive "+absent+" not mounted") || strings.Count(logged, "not mounted") != 2 || !strings.Contains(logged, absentSub) {
t.Errorf("a folder of an absent drive is reported, naming that drive: %q", logged)
}
// The behaviour stays: only the mounted drive itself gets a skeleton and a bind.
if len(skeletons) != 1 || skeletons[0] != mounted {
t.Errorf("skeletons %v — only the mounted drive gets one", skeletons)
}
if len(mounts) != 1 || !strings.Contains(mounts[0], ":/srv/scratch_hdd") {
t.Errorf("mounts %v — only the mounted drive's userdata is bound", mounts)
}
if len(cfg) != 4 {
t.Errorf("the source list keeps every registered path: %v", cfg)
}
}
// R-1351: the web sync's compose call runs inside the stack lock it shares with the base stack's bring-up. The lock's
// behaviour is TestR1351_TheTwoFileBrowserStartPathsNeverOverlap (internal/stacks); this pins the web side's use of it.
func TestR1351_TheWebSyncTakesTheLockBeforeCompose(t *testing.T) {
b, err := os.ReadFile("handlers.go")
if err != nil {
t.Fatal(err)
}
src := string(b)
start := strings.Index(src, "func (s *Server) syncFileBrowserMounts(")
if start < 0 {
t.Fatal("syncFileBrowserMounts not found")
}
body := src[start:]
lock := strings.Index(body, "s.stackMgr.LockFileBrowserStack()")
down := strings.Index(body, `"docker", "compose", "down", "-v"`)
up := strings.Index(body, `[]string{"compose", "up", "-d", "--remove-orphans"}`)
if lock < 0 || down < 0 || up < 0 || lock > down || lock > up {
t.Fatalf("the lock must come before both compose calls (lock %d, down %d, up %d)", lock, down, up)
}
}
+42 -4
View File
@@ -186,6 +186,7 @@ func (s *Server) dashboardHandler(w http.ResponseWriter, r *http.Request) {
data["TitleKey"] = "page.title.dashboard" // i18n: the Hungarian title above is what hu renders
s.addRecoveryBanner(data, r) // R-241: the reminder bar, per visit
data["SettingsWarning"] = s.settings.LoadWarning // non-empty if settings.json was recovered from corruption
data["OperatorRebootNotice"] = s.operatorRebootNotice(s.langFor(r), time.Now()) // D65: „Felhom support restarted your box"
// R-560: the app rows carry catalog copy (each row's `.Meta.Description`), so the list the
// TEMPLATE sees is the localised view. The lists the warning helpers below see are the
// originals — they key on stack NAMES, and handing them a translated copy would only widen
@@ -1346,6 +1347,7 @@ func (s *Server) backupsRestoreHandler(w http.ResponseWriter, r *http.Request) {
// D25: second-drive copies whose source is gone — kept until the household removes one.
rows := s.keptCopyRows(s.langFor(r))
data["KeptCopies"] = rows
data["RestoreCopies"] = s.restoreCopyRows(s.langFor(r)) // D62: restore copies in the same list
for _, k := range rows {
if k.Restorable {
data["KeptUnitRestorable"] = true // D47: the confirm script only when a row offers „Visszaállítás"
@@ -1894,7 +1896,10 @@ func (s *Server) backupRestoreHandler(w http.ResponseWriter, r *http.Request) {
}
s.logger.Printf("[WARN] [web] Restore requested (async): stack=%s, snapshot=%s from %s", stackName, snapshotID, r.RemoteAddr)
s.backupMgr.BeginRestoreOp("restore", stackName)
if msg, ok := s.beginRestoreOp("restore", stackName); !ok {
http.Redirect(w, r, "/backups/restore?flash_error="+url.QueryEscape(msg), http.StatusFound)
return
}
go func() {
start := time.Now()
// Phase 2b: restore from the app's recovery unit (recovers secrets from the guest, fail-closed
@@ -2236,7 +2241,10 @@ func (s *Server) backupTier2RestoreHandler(w http.ResponseWriter, r *http.Reques
}
s.logger.Printf("[WARN] [web] Tier-2 file restore requested (async): stack=%s from %s", stackName, r.RemoteAddr)
s.backupMgr.BeginRestoreOp("tier2-restore", stackName)
if msg, ok := s.beginRestoreOp("tier2-restore", stackName); !ok {
http.Redirect(w, r, "/backups/apps?flash_error="+url.QueryEscape(msg), http.StatusFound)
return
}
go func() {
n, err := s.backupMgr.RestoreTier2Files(stackName)
if err != nil {
@@ -2311,7 +2319,10 @@ func (s *Server) backupTier2UnitRestoreHandler(w http.ResponseWriter, r *http.Re
// (R-538), which is what left such an app with no way back from the second drive.
if s.backupMgr.HasDriveFileLegs(stackName) {
s.logger.Printf("[WARN] [web] Tier-2 WHOLE restore requested (async, files + database): stack=%s from %s", stackName, r.RemoteAddr)
s.backupMgr.BeginRestoreOp("tier2-whole-restore", stackName)
if msg, ok := s.beginRestoreOp("tier2-whole-restore", stackName); !ok {
http.Redirect(w, r, "/backups/apps?flash_error="+url.QueryEscape(msg), http.StatusFound)
return
}
go func() {
start := time.Now()
res, err := s.backupMgr.RestoreTier2Whole(stackName)
@@ -2346,7 +2357,10 @@ func (s *Server) backupTier2UnitRestoreHandler(w http.ResponseWriter, r *http.Re
}
s.logger.Printf("[WARN] [web] Tier-2 UNIT restore requested (async, OVERWRITES live data): stack=%s from %s", stackName, r.RemoteAddr)
s.backupMgr.BeginRestoreOp("tier2-unit-restore", stackName)
if msg, ok := s.beginRestoreOp("tier2-unit-restore", stackName); !ok {
http.Redirect(w, r, "/backups/apps?flash_error="+url.QueryEscape(msg), http.StatusFound)
return
}
go func() {
start := time.Now()
res, err := s.backupMgr.RestoreTier2Unit(stackName)
@@ -3755,6 +3769,15 @@ func skipFileBrowserPath(path string, isMount func(string) bool) bool {
return strings.HasPrefix(path, StableParentDir+"/") && !isMount(path)
}
// fbDriveRoot is the drive a path under StableParentDir lives on (…/scratch_hdd/userdata/x → …/scratch_hdd).
func fbDriveRoot(path string) string {
rest := strings.TrimPrefix(path, StableParentDir+"/")
if i := strings.Index(rest, "/"); i >= 0 {
rest = rest[:i]
}
return StableParentDir + "/" + rest
}
func (s *Server) syncFileBrowserMounts(resetDBOnChange bool) {
// R-695 (v0.275.0): SINGLE-FLIGHT. A request is covered by any sync that STARTS reading the state
// after the request was made: a caller that waited behind a running sync returns as soon as one
@@ -3903,6 +3926,13 @@ func (s *Server) syncFileBrowserMounts(resetDBOnChange bool) {
}
s.fbForceMu.Unlock()
// R-1351: the base stack's bring-up may run `compose up` on this same stack in the same second (a guest boot); the loser
// logged „Conflict. The container name … is already in use". One lock over both compose paths.
if s.stackMgr != nil {
unlock := s.stackMgr.LockFileBrowserStack()
defer unlock()
}
// If sources changed and caller requested a DB reset (restore flow),
// nuke the data volume so FileBrowser re-reads config.yaml from scratch.
// Normal operations skip this to preserve user accounts, permissions, and share links.
@@ -4001,8 +4031,16 @@ func buildFileBrowserPaths(paths []settings.StoragePath, d fbPathDeps) (storageM
// FileBrowser this pass. It returns on the next sync after reconnect. Matches planDriveGates'
// external-only rule (system paths, not under StableParentDir, are never skipped).
if skipFileBrowserPath(sp.Path, d.isMount) {
// R-1339: a per-app folder (<drive>/userdata/<app>) is never a mountpoint itself. Under a MOUNTED drive it is shown
// through the drive's own bind, so it gets no bind of its own and there is nothing to report: the old „not mounted"
// line was false nine times at every start on a box whose drive worked. Pinned by TestR1339_*.
if d.logger != nil {
switch root := fbDriveRoot(sp.Path); {
case root == sp.Path:
d.logger.Printf("[INFO] [web] FileBrowser: drive %s not mounted — skipping userdata skeleton", sp.Path)
case !d.isMount(root):
d.logger.Printf("[INFO] [web] FileBrowser: drive %s not mounted — skipping its folder %s", root, sp.Path)
}
}
continue
}
@@ -255,6 +255,21 @@ func i18nCasesB() []i18nCase {
d["KeptUnitRestorable"] = true
return d
}},
// D62 (2026-10-29): restore copies in the kept-copies list — one deletable (the warning and the typed name), one a
// hold needs (the button off, the reason). A NEW case on a box without off-site (the second call site), so no
// existing fixture changes. Line, Warning, TypePrompt and NeededText are composed in Go in the request's language;
// the fixture carries the same data in both renders.
{"backups_restore_restore_copies", "backups_restore", func() map[string]interface{} {
d := i18nLayoutData("backups-restore", "Visszaállítás a mentésből")
d["Backup"] = i18nBackupStatus(false, false)
d["OffboxConfigured"] = false
d["RestoreCopies"] = []restoreCopyRow{
{App: "vikunja", Stamp: "20261029T080000Z", Name: "Vikunja", Line: "2026-10-29 09:00 · 12 MB · 2026-11-28",
Warning: "Vikunja 2026-10-29", TypePrompt: "Vikunja"},
{App: "immich", Stamp: "20261020T080000Z", Name: "Immich Photos", Line: "2026-10-20 10:00 · 2 GB", NeededText: "hold: support decides"},
}
return d
}},
{"backups_restore_known_empty", "backups_restore", func() map[string]interface{} {
d := i18nLayoutData("backups-restore", "Visszaállítás a mentésből")
d["Backup"] = i18nBackupStatus(false, false)
+139 -1
View File
@@ -1,12 +1,16 @@
package web
import (
"encoding/json"
"errors"
"net/http"
"net/url"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
"gitea.dooplex.hu/admin/felhom-controller/internal/i18n"
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
)
// D25 (operator ruling 2026-10-21; R-1175 + R-1181): the restore page lists the second-drive copies whose source is
@@ -106,7 +110,10 @@ func (s *Server) keptUnitRestoreHandler(w http.ResponseWriter, r *http.Request)
return
}
s.logger.Printf("[WARN] [web] Restore of %s from a set-aside unit requested (async): %s from %s", k.Source, k.Path, r.RemoteAddr)
s.backupMgr.BeginRestoreOp("restore", k.Source)
if msg, ok := s.beginRestoreOp("restore", k.Source); !ok {
http.Redirect(w, r, "/backups/restore?flash_error="+url.QueryEscape(msg), http.StatusFound)
return
}
go func() {
app, res, err := s.backupMgr.RestoreSetAsideUnit(id)
if err != nil {
@@ -118,3 +125,134 @@ func (s *Server) keptUnitRestoreHandler(w http.ResponseWriter, r *http.Request)
}()
http.Redirect(w, r, "/backups/restore?"+flashQuery("flash", "flash.restore.started"), http.StatusFound)
}
// ── D62 (operator ruling 2026-10-29): restore copies in the same list ───────────────────────────────────────────────
//
// Each restore copy shows the app, the date of the restore it protects, its size and the day the box deletes it by
// itself (30 days after the restore). „Törlés most" deletes it earlier: a warning that names the app and the date,
// and the app's name typed. A copy a hold or a pending put-back needs shows why in one line and has no button — and
// the handler refuses it too. Pinned by TestD62_* (web).
// restoreCopyRow is one restore copy on the page.
type restoreCopyRow struct {
App string
Stamp string
Name string // the app's display name — what the household types
Line string // „visszaállítás: <date> · <size> · automatikusan törlődik: <date>"
NeededText string // why it cannot be deleted; "" = the button is offered
Warning string // the confirm step's sentence (app and date)
TypePrompt string
}
// restoreCopyAppName is the app's display name, or its slug when the app is gone.
func (s *Server) restoreCopyAppName(app string) string {
if s.stackMgr != nil {
if st, ok := s.stackMgr.GetStack(app); ok && st.Meta.DisplayName != "" {
return st.Meta.DisplayName
}
}
return app
}
func (s *Server) restoreCopyRows(lang string) []restoreCopyRow {
if s.backupMgr == nil {
return nil
}
var out []restoreCopyRow
for _, c := range s.backupMgr.RestoreCopies() {
name := s.restoreCopyAppName(c.App)
at := c.At.In(getTimezone())
line := s.msgLang(lang, "restore_copy.restored_on", fmtLocalTime(lang, c.At)) + " · " +
i18n.LocalizeDecimals(lang, backup.HumanizeBytes(c.SizeBytes))
row := restoreCopyRow{App: c.App, Stamp: c.Stamp, Name: name}
if c.Needed == "" {
line += " · " + s.msgLang(lang, "restore_copy.auto_delete", i18n.FormatDate(lang, c.DeleteOn.In(getTimezone())))
row.Warning = s.msgLang(lang, "restore_copy.warning", name, strings.TrimSuffix(i18n.FormatDate(lang, at), "."))
row.TypePrompt = s.msgLang(lang, "kept.delete.type", name)
} else {
row.NeededText = s.msgLang(lang, "restore_copy.needed."+c.Needed)
}
row.Line = line
out = append(out, row)
}
return out
}
// restoreCopyJSON is the delete's answer: ok with the page to go to, or the refusal's sentence.
func restoreCopyJSON(w http.ResponseWriter, code int, ok bool, msg, redirect string) {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.WriteHeader(code)
_ = json.NewEncoder(w).Encode(map[string]interface{}{"ok": ok, "error": msg, "redirect": redirect})
}
// restoreCopyDeleteHandler is „Törlés most": POST /backup/restore-copy/delete (app, stamp, confirm). JSON answer:
// 200 deleted; 400 the typed name does not match; 404 no such copy; 409 the copy is needed, or an operation runs on
// the box or the app; 500 the removal failed. CSRF as every state-changing form (CsrfProtect).
func (s *Server) restoreCopyDeleteHandler(w http.ResponseWriter, r *http.Request) {
lang := s.langFor(r)
if s.backupMgr == nil {
restoreCopyJSON(w, http.StatusServiceUnavailable, false, s.msgLang(lang, "flash.offbox.mgr_unreachable"), "")
return
}
_ = r.ParseForm()
app, stamp := strings.TrimSpace(r.FormValue("app")), strings.TrimSpace(r.FormValue("stamp"))
var found *backup.RestoreCopy
for _, c := range s.backupMgr.RestoreCopies() {
if c.App == app && c.Stamp == stamp {
c := c
found = &c
break
}
}
if found == nil {
s.logger.Printf("[WARN] [web] restore-copy delete REFUSED: %q/%q is not a listed copy (from %s)", app, stamp, clientIP(r))
restoreCopyJSON(w, http.StatusNotFound, false, s.msgLang(lang, "restore_copy.not_found"), "")
return
}
if found.Needed != "" {
s.logger.Printf("[WARN] [web] restore-copy delete REFUSED for %s: the copy is needed (%s)", found.Path, found.Needed)
restoreCopyJSON(w, http.StatusConflict, false, s.msgLang(lang, "restore_copy.needed."+found.Needed), "")
return
}
name := s.restoreCopyAppName(app)
if strings.TrimSpace(r.FormValue("confirm")) != name {
s.logger.Printf("[WARN] [web] restore-copy delete REFUSED for %s: the typed confirmation did not match", found.Path)
restoreCopyJSON(w, http.StatusBadRequest, false, s.msgLang(lang, "kept.delete.mismatch", name), "")
return
}
if s.stackMgr != nil && s.stackMgr.IsUpdating(app) {
s.logger.Printf("[WARN] [web] restore-copy delete REFUSED for %s: an update of %s runs", found.Path, app)
restoreCopyJSON(w, http.StatusConflict, false, s.msgLang(lang, "restore_copy.app_busy"), "")
return
}
who := "the household (dashboard sign-in) from " + clientIP(r)
c, err := s.backupMgr.DeleteRestoreCopy(app, stamp)
switch {
case errors.Is(err, backup.ErrRestoreCopyBusy):
busy, mirror := s.backupMgr.RestoreBusy()
msg, blocked := s.restoreBusyMessage(busy, mirror)
if !blocked {
msg = s.msgLang(lang, "restore_copy.app_busy")
}
s.logger.Printf("[WARN] [web] restore-copy delete REFUSED for %s: a backup or restore operation runs", found.Path)
restoreCopyJSON(w, http.StatusConflict, false, msg, "")
return
case errors.Is(err, backup.ErrRestoreCopyNeeded):
restoreCopyJSON(w, http.StatusConflict, false, s.msgLang(lang, "restore_copy.needed."+c.Needed), "")
return
case errors.Is(err, backup.ErrRestoreCopyNotFound):
restoreCopyJSON(w, http.StatusNotFound, false, s.msgLang(lang, "restore_copy.not_found"), "")
return
case err != nil:
s.logger.Printf("[ERROR] [web] restore-copy delete of %s failed: %v", found.Path, err)
restoreCopyJSON(w, http.StatusInternalServerError, false, s.msgLang(lang, "restore_copy.failed", err.Error()), "")
return
}
restoreAt := c.At.UTC().Format(time.RFC3339)
s.logger.Printf("[WARN] [web] restore copy DELETED by hand (D62): app=%s restore_at=%s size=%d path=%s who=%s",
app, restoreAt, c.SizeBytes, c.Path, who)
if s.notifier != nil {
s.notifier.NotifyRestoreCopyDeleted(notify.RestoreCopyDeletedDetails{App: app, RestoreAt: restoreAt, SizeBytes: c.SizeBytes, Who: who, Path: c.Path})
}
restoreCopyJSON(w, http.StatusOK, true, "", "/backups/restore?"+flashQuery("flash", "flash.restore_copy.deleted", name))
}
+13 -6
View File
@@ -217,16 +217,23 @@ func (s *Server) keptLoadHandler(w http.ResponseWriter, r *http.Request) {
_, err := s.stackMgr.RestoreKeptFiles(it)
return err
}
// R-1256 H6: the load begins in the same step as the check for every other operation; a refusal moves nothing
// (the dated folder's files move back inside the operation, as prepare).
var lErr error
if c.Tier == backup.KeptTierOffsite {
s.logger.Printf("[INFO] [web] kept LOAD %s: %s from the off-site snapshot %s (from %s)", it.App, it.Path, c.SnapshotID, r.RemoteAddr)
s.backupMgr.LoadKeptOffsite(app, it.Drive, c, moveBack, okMsg, failMsg, after)
lErr = s.backupMgr.LoadKeptOffsite(app, it.Drive, c, moveBack, okMsg, failMsg, after)
} else {
if err := moveBack(); err != nil {
s.keptRedirect(w, r, "flash_error", s.errText(r, err))
return
}
s.logger.Printf("[INFO] [web] kept LOAD %s: %s from %s (from %s)", it.App, it.Path, c.UnitDir, r.RemoteAddr)
s.backupMgr.LoadKeptApp(app, c.UnitDir, okMsg, failMsg, after)
lErr = s.backupMgr.LoadKeptApp(app, c.UnitDir, moveBack, okMsg, failMsg, after)
}
if lErr != nil {
msg, _ := s.restoreOpBlocked()
if msg == "" { // the other operation ended in the meantime
msg, _ = s.restoreBusyMessage(backup.RestoreBusyRestore, "")
}
s.keptRedirect(w, r, "flash_error", msg)
return
}
http.Redirect(w, r, "/backups/restore?"+flashQuery("flash", "flash.restore.started"), http.StatusFound)
}
+20 -5
View File
@@ -450,7 +450,10 @@ func (s *Server) offboxRestoreHandler(w http.ResponseWriter, r *http.Request) {
return
}
full := mode == "full"
s.backupMgr.BeginRestoreOp("offbox-restore", app)
if msg, ok := s.beginRestoreOp("offbox-restore", app); !ok {
offboxRedirectTo(w, r, restoreWizardPath(app), msg, true)
return
}
go func() {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
defer cancel()
@@ -538,7 +541,10 @@ func (s *Server) offboxReconstituteHandler(w http.ResponseWriter, r *http.Reques
// data"; this one answers "yes, into a different place than the backup recorded". One checkbox
// carrying both would be the two-decisions-one-button shape R-48 removed from this surface.
ackPlacement := r.FormValue("ack_placement") == "1"
s.backupMgr.BeginRestoreOp("offbox-reconstitute", app)
if msg, ok := s.beginRestoreOp("offbox-reconstitute", app); !ok {
offboxRedirectTo(w, r, restoreWizardPath(app), msg, true)
return
}
go func() {
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Minute)
defer cancel()
@@ -678,7 +684,10 @@ func (s *Server) offboxPlaceHandler(w http.ResponseWriter, r *http.Request) {
offboxRedirectTo(w, r, restoreWizardPath(app), offsiteScratchIncompleteKey, true)
return
}
s.backupMgr.BeginRestoreOp("offbox-place", app)
if msg, ok := s.beginRestoreOp("offbox-place", app); !ok {
offboxRedirectTo(w, r, restoreWizardPath(app), msg, true)
return
}
go func() {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
defer cancel()
@@ -711,7 +720,10 @@ func (s *Server) sharesRestoreHandler(w http.ResponseWriter, r *http.Request) {
offboxRedirectTo(w, r, "/backups/restore", msg, true)
return
}
s.backupMgr.BeginRestoreOp("shares-restore", backup.SharesDisplayName)
if msg, ok := s.beginRestoreOp("shares-restore", backup.SharesDisplayName); !ok {
offboxRedirectTo(w, r, "/backups/restore", msg, true)
return
}
go func() {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
defer cancel()
@@ -737,7 +749,10 @@ func (s *Server) sharesPlaceHandler(w http.ResponseWriter, r *http.Request) {
offboxRedirectTo(w, r, "/backups/restore", msg, true)
return
}
s.backupMgr.BeginRestoreOp("shares-place", backup.SharesDisplayName)
if msg, ok := s.beginRestoreOp("shares-place", backup.SharesDisplayName); !ok {
offboxRedirectTo(w, r, "/backups/restore", msg, true)
return
}
go func() {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
defer cancel()
@@ -0,0 +1,81 @@
package web
import (
"context"
"errors"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
)
// „Felhom support restarted your box at <time>" (D65 = A, operator ruling 2026-10-29 15:50; felhom.eu `04` §6.1).
//
// The operator can reboot the guest from the hub; the household must see afterwards that Felhom did it. The agent
// writes its record BEFORE the reboot, so this controller — which starts with the guest — asks for it once at start
// (GET /guest/operator-reboot) and the dashboard shows one neutral line until a day has passed since the reboot.
// Pinned by operator_reboot_notice_test.go.
// operatorRebootNoticeFor is how long the line shows (the agent also stops answering after a day).
const operatorRebootNoticeFor = 24 * time.Hour
// SetOperatorReboot records the last operator reboot (the zero time = none).
func (s *Server) SetOperatorReboot(at time.Time) {
s.opRebootMu.Lock()
defer s.opRebootMu.Unlock()
s.opRebootAt = at
}
// operatorRebootNotice is the dashboard's line for lang at now ("" = none).
func (s *Server) operatorRebootNotice(lang string, now time.Time) string {
s.opRebootMu.Lock()
at := s.opRebootAt
s.opRebootMu.Unlock()
if at.IsZero() || now.Sub(at) >= operatorRebootNoticeFor || now.Before(at) {
return ""
}
return s.i18n.Msgf(lang, "dashboard.operator_reboot", at.In(budapestLocation()).Format("15:04"))
}
// WatchOperatorReboot asks the agent once at start, retrying while the agent does not answer (it may start after us),
// at most for opRebootAskFor. A 404 (an older agent) ends it at once.
func (s *Server) WatchOperatorReboot(ctx context.Context) {
fetch := s.opRebootFetch
if fetch == nil {
fetch = func(ctx context.Context) (time.Time, error) {
c, err := s.agentClient()
if err != nil {
return time.Time{}, err
}
return c.OperatorReboot(ctx)
}
}
every := s.opRebootRetry
if every <= 0 {
every = 30 * time.Second
}
for deadline := time.Now().Add(10 * time.Minute); ; {
cctx, cancel := context.WithTimeout(ctx, 10*time.Second)
at, err := fetch(cctx)
cancel()
var se *agentapi.StatusError
switch {
case err == nil:
if !at.IsZero() {
s.SetOperatorReboot(at)
s.logger.Printf("[INFO] [web] the operator rebooted this box from the hub at %s — the dashboard says so for a day", at.UTC().Format(time.RFC3339))
}
return
case errors.As(err, &se) && se.Code == 404:
return // an agent older than the route cannot reboot from the hub
}
if !time.Now().Before(deadline) {
s.logger.Printf("[WARN] [web] could not ask the agent about an operator reboot: %v — no notice on the dashboard", err)
return
}
select {
case <-ctx.Done():
return
case <-time.After(every):
}
}
}
@@ -0,0 +1,79 @@
package web
import (
"context"
"errors"
"io"
"log"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
)
// D65 = A: after the operator rebooted the box from the hub, the household's dashboard says so — one neutral line,
// for one day, then gone.
// The real dashboard, through ServeHTTP: the line is there after an operator reboot, in Hungarian, with the Budapest
// time; it is gone a day later. RED-PROOF: drop the data["OperatorRebootNotice"] line from dashboardHandler → FAILS;
// drop the 24-hour check → the day-later page still shows it → FAILS.
func TestOperatorRebootNotice_OnTheDashboardForADay(t *testing.T) {
s := newDashboardServer(t, time.Time{})
if body := getPage(t, s, "/dashboard").Body.String(); strings.Contains(body, "data-operator-reboot") {
t.Fatal("no reboot yet, but the dashboard shows the line")
}
at := time.Now().Add(-2 * time.Hour)
s.SetOperatorReboot(at)
body := getPage(t, s, "/dashboard").Body.String()
want := "A Felhom ügyfélszolgálata " + at.In(budapestLocation()).Format("15:04") + "-kor újraindította a dobozt."
if !strings.Contains(body, want) {
t.Fatalf("the dashboard lacks %q", want)
}
s.SetOperatorReboot(time.Now().Add(-25 * time.Hour))
if body := getPage(t, s, "/dashboard").Body.String(); strings.Contains(body, "data-operator-reboot") {
t.Fatal("a day later the line must be gone")
}
}
// English: the reader's language.
func TestOperatorRebootNotice_English(t *testing.T) {
s := newDashboardServer(t, time.Time{})
at := time.Date(2026, 10, 30, 9, 41, 0, 0, time.UTC)
s.SetOperatorReboot(at)
got := s.operatorRebootNotice("en", at.Add(time.Hour))
if got != "Felhom support restarted your box at "+at.In(budapestLocation()).Format("15:04")+"." {
t.Fatalf("en: %q", got)
}
}
// The start-up ask: it retries while the agent does not answer, keeps the time it gets, and stops at once on a 404
// (an agent older than the route cannot reboot from the hub). RED-PROOF: drop SetOperatorReboot from the success
// branch → FAILS; return on the first error → the retried answer is lost → FAILS.
func TestWatchOperatorReboot_RetriesAndKeepsTheTime(t *testing.T) {
s := &Server{logger: log.New(io.Discard, "", 0), opRebootRetry: time.Millisecond}
at := time.Date(2026, 10, 30, 9, 41, 0, 0, time.UTC)
calls := 0
s.opRebootFetch = func(context.Context) (time.Time, error) {
calls++
if calls < 3 {
return time.Time{}, errors.New("agentapi: GET /guest/operator-reboot: connection refused")
}
return at, nil
}
s.WatchOperatorReboot(context.Background())
if calls != 3 || !s.opRebootAt.Equal(at) {
t.Fatalf("calls=%d at=%v, want 3 and %v", calls, s.opRebootAt, at)
}
s2 := &Server{logger: log.New(io.Discard, "", 0), opRebootRetry: time.Millisecond}
calls = 0
s2.opRebootFetch = func(context.Context) (time.Time, error) {
calls++
return time.Time{}, &agentapi.StatusError{Path: "/guest/operator-reboot", Code: 404}
}
s2.WatchOperatorReboot(context.Background())
if calls != 1 || !s2.opRebootAt.IsZero() {
t.Fatalf("a 404 must end the ask at once: calls=%d at=%v", calls, s2.opRebootAt)
}
}
@@ -93,7 +93,9 @@ func TestR1151_ConfirmAnswersInTheBoxLanguage(t *testing.T) {
{http.StatusGone, http.StatusGone, true, "Lejárt az idő", "has run out"},
{http.StatusNotFound, http.StatusGone, true, "már nem vár megerősítésre", "no longer waits"},
{http.StatusBadGateway, http.StatusBadGateway, false, "nem sikerült elmenteni", "could not be saved"},
{0, http.StatusBadGateway, false, "nem sikerült elmenteni", "could not be saved"},
// R-1254 H2 (2026-10-28): a failed call (0) or an agent that cannot learn the outcome (504) promises nothing.
{0, http.StatusGatewayTimeout, false, "Nem tudjuk biztosan", "cannot tell yet"},
{http.StatusGatewayTimeout, http.StatusGatewayTimeout, false, "Nem tudjuk biztosan", "cannot tell yet"},
}
for _, c := range cases {
for _, lang := range []string{"hu", "en"} {
@@ -0,0 +1,48 @@
package web
import (
"errors"
"net/http"
"strings"
"testing"
)
// R-1254 H2 (the household's sentence): a confirm whose upload answer is late or lost must not promise „your previous
// code still works" — the hub may hold the new package. Since agent 0.168.0 the agent asks the hub and answers 502 only
// when the hub holds ANOTHER package; 504 = unknown. A controller-side transport failure (status 0 — the call itself
// timed out while the agent may still store) is unknown too.
// RED-PROOF: drop the 504/0 case from escrowConfirmAPIHandler → the default sentence („a régi kódod továbbra is
// érvényes") is shown → FAILS.
func TestR1254_AnUnknownConfirmOutcomePromisesNothing(t *testing.T) {
for _, tc := range []struct {
name string
status int
}{{"agent 504", http.StatusGatewayTimeout}, {"transport failure", 0}} {
t.Run(tc.name, func(t *testing.T) {
h := newEscrowWizardHarness(t)
h.configureOffbox(t, "pending")
h.agent.confirmStatus, h.agent.confirmErr = tc.status, errors.New("late")
for _, lang := range []string{"hu", "en"} {
w := postConfirm(t, h.s, lang, `{"words":["ketto","negy"]}`)
low := strings.ToLower(w.Body.String())
// ASCII fragments (the accented forms carry these stems): „régi kódod továbbra is érvényes" / „still works".
if strings.Contains(low, "tovabbra is") || strings.Contains(low, "továbbra is") || strings.Contains(low, "still works") {
t.Fatalf("%s: an unknown outcome promised the previous code: %s", lang, w.Body.String())
}
if !strings.Contains(low, "mindket") && !strings.Contains(low, "mindkét") && !strings.Contains(low, "both") {
t.Fatalf("%s: the sentence must tell the household to keep both codes: %s", lang, w.Body.String())
}
}
if ceremonyStamped(t, h) {
t.Fatal("an unknown outcome stamped the ceremony completed")
}
})
}
// Control: the agent says the hub holds another package (502) → the old sentence, now true.
h := newEscrowWizardHarness(t)
h.configureOffbox(t, "pending")
h.agent.confirmStatus, h.agent.confirmErr = http.StatusBadGateway, errors.New("not stored")
if w := postConfirm(t, h.s, "en", `{"words":["ketto","negy"]}`); !strings.Contains(w.Body.String(), "still works") {
t.Fatalf("502 must keep the store-failed sentence: %s", w.Body.String())
}
}
@@ -0,0 +1,60 @@
package web
import (
"go/ast"
"go/parser"
"go/token"
"os"
"path/filepath"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
)
// R-1256 H6: every restore handler records its op through beginRestoreOp, which asks about every other operation in
// the same step (backup.TryBeginRestoreOp). A handler that calls BeginRestoreOp directly re-opens the window between
// the early refusal and the record. RED-PROOF: put one `s.backupMgr.BeginRestoreOp(...)` back → FAILS, naming the file.
func TestR1256_H6_NoHandlerBeginsARestoreUnchecked(t *testing.T) {
files, _ := filepath.Glob("*.go")
for _, f := range files {
if strings.HasSuffix(f, "_test.go") {
continue
}
src, err := os.ReadFile(f)
if err != nil {
t.Fatal(err)
}
fs := token.NewFileSet()
af, err := parser.ParseFile(fs, f, src, 0)
if err != nil {
t.Fatal(err)
}
ast.Inspect(af, func(n ast.Node) bool {
if sel, ok := n.(*ast.SelectorExpr); ok && sel.Sel.Name == "BeginRestoreOp" {
t.Errorf("%s: a direct BeginRestoreOp — begin through s.beginRestoreOp (R-1256 H6)", fs.Position(sel.Pos()))
}
return true
})
}
}
// The handler's begin refuses with the household's sentence when a second-drive copy began after the early check.
func TestR1256_H6_TheBeginRefusesWithTheCopySentence(t *testing.T) {
s, _, _ := newOffboxWebServer(t)
if _, blocked := s.restoreOpBlocked(); blocked {
t.Fatal("fixture: nothing runs")
}
done := backup.MarkTier2CopyForTest("immich")
msg, ok := s.beginRestoreOp("offbox-reconstitute", "immich")
done()
if ok || !strings.Contains(msg, "második meghajtóra") {
t.Fatalf("the begin must refuse with the copy named; ok=%v %q", ok, msg)
}
if s.backupMgr.RestoreStatus().Running {
t.Fatal("a refused begin recorded a restore")
}
if _, ok := s.beginRestoreOp("offbox-reconstitute", "immich"); !ok {
t.Fatal("after the copy the begin must work")
}
}
+32 -10
View File
@@ -209,28 +209,50 @@ func (s *Server) restoreOpBlocked() (string, bool) {
if s.backupMgr == nil {
return "", false
}
if st := s.backupMgr.RestoreStatus(); restoreOpInFlight(st) {
busy, app := s.backupMgr.RestoreBusy()
return s.restoreBusyMessage(busy, app)
}
// beginRestoreOp records a restore op in the SAME step as the check for every other operation (R-1256 H6): the early
// restoreOpBlocked answer can be stale by the time a handler has read its form, and a second-drive copy that began in
// between ran beside the restore. A refusal returns the same sentence as restoreOpBlocked; the caller redirects with it
// and starts nothing. Every restore handler begins through here — TestR1256_H6_NoHandlerBeginsARestoreUnchecked.
func (s *Server) beginRestoreOp(op, stack string) (string, bool) {
busy, app := s.backupMgr.TryBeginRestoreOp(op, stack)
if busy == "" {
return "", true
}
if s.logger != nil {
s.logger.Printf("[WARN] [restore] %s of %s refused at its start — a %s operation runs (R-1256)", op, stack, busy)
}
msg, _ := s.restoreBusyMessage(busy, app)
return msg, false
}
// restoreBusyMessage is the household's sentence for a RestoreBusy reason ("" = not busy).
func (s *Server) restoreBusyMessage(busy, app string) (string, bool) {
switch busy {
case "":
return "", false
case backup.RestoreBusyRestore:
subject := "Egy visszaállítási művelet"
if st.Stack != "" {
subject = "Egy visszaállítási művelet (" + st.Stack + ")"
if app != "" {
subject = "Egy visszaállítási művelet (" + app + ")"
}
return subject + " már fut, ezért most nem indítható újabb. Az állapotát ezen az oldalon " +
"követheted; amint befejeződik, újra indíthatsz visszaállítást.", true
}
if s.backupMgr.IsRunning() {
case backup.RestoreBusyBackup:
return "Egy mentési művelet már fut, ezért most nem indítható visszaállítás. Az állapotát " +
"ezen az oldalon követheted; amint befejeződik, újra indíthatsz visszaállítást.", true
}
// The stray-copy fix: a second-drive copy (rsync --delete) runs now, or one a previous controller process lost may
// still run — a restore beside it could lose the files it brings back (R-1180's other direction).
if app, blocked := s.backupMgr.Tier2CopyBlocksRestore(); blocked {
default:
// The stray-copy fix: a second-drive copy (rsync --delete) runs now, or one a previous controller process lost
// may still run — a restore beside it could lose the files it brings back (R-1180's other direction).
if s.logger != nil {
s.logger.Printf("[WARN] [restore] restore refused — a second-drive copy is running or may still run (app=%q)", app)
}
return "Éppen fut a második meghajtóra készülő mentési másolat, ezért most nem indítható visszaállítás. " +
"Néhány perc múlva próbáld újra.", true
}
return "", false
}
// backupsRestoreWizardHandler renders GET /backups/restore/app?name=<app> — the single entry the
+8
View File
@@ -136,6 +136,11 @@ type Server struct {
agentCli *agentapi.Client
agentCliErr error
agentCliOnce sync.Once
// D65 = A: the last operator reboot of this guest (operator_reboot_notice.go); the fetch and retry are test seams.
opRebootMu sync.Mutex
opRebootAt time.Time
opRebootFetch func(ctx context.Context) (time.Time, error)
opRebootRetry time.Duration
// initialCredsFn is the R-254 read seam for an app's generated first-login credential. nil → the
// real live container read (stackMgr.ReadInitialCredentials). ONE definition, used by BOTH the
@@ -844,6 +849,9 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// D25 (2026-10-21): „Eltávolítás" of ONE kept second-drive copy whose source is gone.
case path == "/backup/kept/remove" && r.Method == http.MethodPost:
s.keptCopyRemoveHandler(w, r)
// D62 (2026-10-29): „Törlés most" of ONE restore copy (typed name; never a copy a hold or a put-back needs).
case path == "/backup/restore-copy/delete" && r.Method == http.MethodPost:
s.restoreCopyDeleteHandler(w, r)
// D47 (2026-10-25): „Visszaállítás" from a set-aside recovery unit of an earlier install.
case path == "/backup/kept/restore" && r.Method == http.MethodPost:
s.keptUnitRestoreHandler(w, r)
@@ -429,9 +429,9 @@ function submitRestore() {
{{/* D25 + D47: the kept copies (second-drive copies whose source is gone; set-aside units of an earlier install). Shown
inside the off-site card when the box has off-site, and after it when it has none — before 2026-10-25 it sat only inside
the off-site block, so a box without off-site never listed its kept copies. One body, two call sites. */}}
{{define "kept_copies_section"}}{{if .KeptCopies}}
{{define "kept_copies_section"}}{{if or .KeptCopies .RestoreCopies}}
<div class="backup-tier-divider" style="margin-top:1.5rem"></div>
<h3 style="margin-bottom:.5rem">{{T "backups_restore.megorzott_masolatok"}}</h3>
<h3 style="margin-bottom:.5rem">{{T "backups_restore.megorzott_masolatok"}}</h3>{{if .KeptCopies}}
<p class="form-hint" style="margin-bottom:.75rem">{{T "backups_restore.megorzott_masolatok_hint"}}</p>
<div class="app-row-list">
{{range .KeptCopies}}
@@ -467,4 +467,43 @@ the off-site block, so a box without off-site never listed its kept copies. One
});
}
</script>
{{end}}{{end}}
{{end}}{{if .RestoreCopies}}{{/* D62 (operator ruling 2026-10-29): restore copies — the app, the restore's date, the size, the day the box deletes
it; „Törlés most" after the warning and the app's name typed. A needed copy: the button is off, the reason in one line. */}}
<h4 style="margin:1rem 0 .5rem">{{T "restore_copy.title"}}</h4>
<p class="form-hint" style="margin-bottom:.75rem">{{T "restore_copy.hint"}}</p>
<div class="app-row-list" data-restore-copies="true">
{{range .RestoreCopies}}
{{template "app_list_row" dict "Slug" .App "Name" .Name "Secondary" .Line}}{{if .NeededText}}
<button type="button" class="btn btn-xs btn-danger-outline" disabled data-restore-copy-off="true">{{T "restore_copy.delete"}}</button>{{end}}
{{template "app_list_row_end"}}{{if .NeededText}}
<p class="form-hint" data-restore-copy-needed="true">{{.NeededText}}</p>{{else}}
<details class="kept-delete" data-restore-copy="{{.App}}/{{.Stamp}}">
<summary class="btn btn-xs btn-danger-outline">{{T "restore_copy.delete"}}</summary>
<form method="POST" action="/backup/restore-copy/delete" class="kept-delete-form" onsubmit="return deleteRestoreCopy(event, this)">
{{$.CSRFField}}
<input type="hidden" name="app" value="{{.App}}">
<input type="hidden" name="stamp" value="{{.Stamp}}">
<p class="kept-delete-text">{{.Warning}}</p>
<label class="form-hint">{{.TypePrompt}}</label>
<input type="text" name="confirm" autocomplete="off" required>
<button type="submit" class="btn btn-sm btn-danger">{{T "restore_copy.delete"}}</button>
<p class="kept-delete-text" data-restore-copy-error="true" hidden></p>
</form>
</details>{{end}}
{{end}}
</div>
<script>
function deleteRestoreCopy(ev, f){
ev.preventDefault();
var err = f.querySelector('[data-restore-copy-error]');
fetch(f.action, {method: 'POST', headers: Object.assign({'Accept': 'application/json'}, csrfHeaders()), body: new URLSearchParams(new FormData(f))})
.then(function(r){ return r.json(); })
.then(function(d){
if (d.ok) { location.href = d.redirect; return; }
err.textContent = d.error || ''; err.hidden = false;
})
.catch(function(e){ err.textContent = '{{T "layout.halozati_hiba"}} ' + e; err.hidden = false; });
return false;
}
</script>
{{end}}{{end}}{{end}}
Loaded 100 of 104 files, more files were not shown because too many files have changed in this diff. Show more