Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
27360d6abe | ||
|
|
64b99b74e3 | ||
|
|
5c336d828c | ||
|
|
13e34d4ed4 | ||
|
|
37644458c7 | ||
|
|
c4b6844d34 | ||
|
|
a90b62323f | ||
|
|
94049a7200 | ||
|
|
b69e90bdad | ||
|
|
a28db2d17f | ||
|
|
2a0a17a760 | ||
|
|
99fac06aa7 | ||
|
|
6077ac0083 | ||
|
|
5b3b541602 | ||
|
|
c7bb776e00 | ||
|
|
652a690e5d | ||
|
|
6c72e3ff34 | ||
|
|
fd75c7c717 | ||
|
|
e3fce09d96 | ||
|
|
a815b37fbd | ||
|
|
20c5877056 | ||
|
|
eed23e7594 | ||
|
|
e33cffaee1 | ||
|
|
342a394bff | ||
|
|
ed3927f03a |
No files matched your search
+249
@@ -1,3 +1,252 @@
|
||||
## v0.351.0 (unreleased, 2026-11-09) — old test builds and old infrastructure images are deleted (decision 56 amended, decision 285)
|
||||
|
||||
### 2026-11-09
|
||||
|
||||
**MinAgent: 0.158.1** (unchanged).
|
||||
|
||||
**Why (measured 2026-11-09, demo-hp guest 9201, `docker system df -v`):** the whole-guest backup copies `/var/lib/docker` (mp0,
|
||||
`backup=1`), and demo-hp's local tier was refused for space again (felhom.eu R-548). Of 27.5 GB of images, 13.2 GB ran in
|
||||
containers, 9.8 GB were the apps' undo images (decision 53, kept on purpose), 2.2 GB were 28 old controller images and
|
||||
1.35 GB were old infrastructure and helper images. Nothing deleted the last two kinds.
|
||||
|
||||
**Controller images: a test build is a version (decision 56 amended).** `RetainControllerImages` left every `-rc` tag alone,
|
||||
so 25 test builds stayed. Tags, the running version and the previous now parse with `util.ParseVersionPre` (an rc orders
|
||||
below its release): an old rc goes like an old release; an rc above the running one or between the previous and the
|
||||
running one stays; `latest` stays. A box that RUNS a test build is judged (it was skipped). The fallback previous prefers
|
||||
a release over a newer test build. Tests `TestControllerRetention_OldTestBuildsGo`, `_BoxOnATestBuildIsJudged`; the
|
||||
existing test's rc expectation changed with the rule.
|
||||
|
||||
**Infrastructure images (decision 285, new).** `RetainInfraImages(infra.Images())`, 3 minutes after every start: per pinned
|
||||
repository the pin, newer versions, the previous pin and every in-use image stay; older and untagged ones go by exact
|
||||
name/ID; never forced, never pruned; not while an app pulls (R-863, retried every 2 minutes, 30 tries) or while the
|
||||
controller swaps itself. Tests `TestInfraRetention_*` (4). Customer boxes get a new traefik or cloudflared pin every few
|
||||
weeks (~65–195 MB each); this is the part a customer box needs.
|
||||
|
||||
Red-proofs: six code mutants, each test red (the previous-pin keep, the in-use check, the R-863 lock, the tag parser, the
|
||||
running-version parser, the release preference). Green gate: `go build`, `go vet`, `go test ./...` rc 0; `controller_gates.py --fast` OK.
|
||||
|
||||
## v0.350.0 (2026-11-03) — Traefik v3.7.17 (GHSA-xv5x-242w-7hf7); a restore forgets the update-held flag (R-1347); a failed put-back at start-up mails (R-1348); three false start-up lines (R-1339, R-1351, R-1323)
|
||||
|
||||
### 2026-11-03
|
||||
|
||||
**MinAgent: 0.158.1** (unchanged).
|
||||
|
||||
**Load test, scratch 9202 only, from the LAN, 2026-11-03** (felhom.eu `audits/release-2026-11-03/part1/`): the same many short TLS connections with an
|
||||
`Authorization: NTLM` header to the dashboard route. v3.7.16: Traefik's anonymous memory 18.7 → 79.0 MiB after 920 connections, not released (stopped at
|
||||
twice its start, as the test's bound says). v3.7.17: 16.2 → 25-26 MiB, flat for 25,116 connections and the whole 300 s.
|
||||
|
||||
**Traefik v3.7.17 (GHSA-xv5x-242w-7hf7, high, published 2026-11-02).** v3.7.14–v3.7.16: an unauthenticated client that
|
||||
sends a `Negotiate` or `NTLM` `Authorization` header gets a dedicated backend transport per connection, and it is not
|
||||
released when the connection closes: many short connections exhaust Traefik's memory, whether or not an NTLM backend
|
||||
exists. Traefik fronts every box. `TraefikImage` v3.7.16 → v3.7.17; a running box takes it by the reconcile at start
|
||||
(`TestEnsureTraefik_ANewImageRecreatesTraefik`). The upgrade notes name Go 1.27 (GODEBUG TLS settings: we set none; 500
|
||||
request-header values at most, else 431) and the ingress-nginx provider (not used). Found by reading the release itself:
|
||||
the watch had flagged it on a word-only line (felhom.eu R-1310).
|
||||
|
||||
**R-1347 — a restore that lifts an update hold forgets the „update held" flag.** `clearUpdateHoldAfterRestore` cleared the
|
||||
hold in settings, but `stacks.Manager` kept `updateHeld`: the crash-loop watcher did not watch the app and the night leg
|
||||
skipped it until the controller restarted. New seam `backup.SetUpdateHeldForgetter` (`stackMgr.ForgetUpdateHeld`). Tests
|
||||
`TestR1347_*` (the seam, the wiring, and the consequence: the same restarts trip the watcher once the flag is gone).
|
||||
|
||||
**R-1348 — a failed put-back at start-up mails the operator.** `holdPutBackFailed` writes the D20 hold at `Recover`, before any
|
||||
notifier exists, so only the hold reporter's info and the hub's later reminder said anything. `AppStopGuard.SetHeldNotify`
|
||||
hands over what `Recover` held, once, and mails a later failure (a drive that returned) at once: the R-1340 sentence for
|
||||
`MixedCauseInterruptedPutBack`, from the hold record. The mail's run name says „restore" for it, not „offsite-reconstitute"
|
||||
(`heldRunKind`, seen on the live proof). Tests `TestR1348_*`. **Live proof 2026-11-03, scratch 9202, test image of c4b6844** (felhom.eu
|
||||
`audits/security-2026-11-02/part3/`): a restore cut at „the restore copy is recorded", the volume held by a created-only
|
||||
container, the controller started: hold `restore_mixed` / `interrupted_put_back`, the event `backup_run_failures` with the sentence in the sink.
|
||||
**R-1347's live proof, same image:** an update with its undo copy cut off → hold `update_failed`; the household's restore → „the update hold is CLEARED", „its
|
||||
update-held flag is forgotten"; six `kill -9` → the crash-loop watcher stops the app (`unhealthy_stop`), so it watches again.
|
||||
|
||||
**R-1339 — no false „drive … not mounted".** A per-app folder under a mounted drive is never a mountpoint itself; nine false lines
|
||||
at every start on 9202. The folder still gets no bind and no skeleton of its own (the row's „not skipped" would write a
|
||||
`userdata/` inside the app's folder and recreate the file manager, so the behaviour stays and only the line changes). A drive that
|
||||
is really absent keeps its line, and a folder of it names the drive. `TestR1339_*`.
|
||||
|
||||
**R-1351 — FileBrowser's two start paths at boot take one lock.** `EnsureBaseStack`'s `compose up` and
|
||||
`web.syncFileBrowserMounts`' `compose up --force-recreate` in the same second gave „Conflict. The container name … is already in
|
||||
use", two ERROR lines and a first-boot WARN for an app that then ran healthy. `stacks.Manager.LockFileBrowserStack`.
|
||||
`TestR1351_*`.
|
||||
|
||||
**R-1323 — „No such container" is not „every app can still reach it".** The checked container was replaced between the check and
|
||||
the leave; the line now reads the CURRENT container's networks and says the truth. A current container on the app network, an
|
||||
unreadable one, or any other failure keeps the alarm. `TestR1323_*`.
|
||||
|
||||
## v0.349.0 (2026-10-29) — the household line after an operator reboot and the busy file for it (D65 = A); a hold clear records the wanted stop (R-1350)
|
||||
|
||||
### 2026-10-29 (evening)
|
||||
|
||||
**MinAgent: 0.158.1** (unchanged). The new `GET /guest/operator-reboot` is asked once at start; an older agent answers
|
||||
404, and such an agent cannot reboot from the hub either — nothing to show, nothing broken. The busy file is read BY the
|
||||
agent (≥ the reboot_guest commit); an older agent never reads it.
|
||||
|
||||
- **D65 = A — the busy file (`internal/busymark`).** `<data dir>/ctl/busy`, rewritten every 15 s while a backup, a
|
||||
restore, an off-site run, the night catch-up, an app update or an app export/import runs; removed when idle and at
|
||||
start. The agent refuses the operator's „Reboot guest" while it is fresh. Tests `TestBusyFileFollowsTheWork`,
|
||||
`TestBusyMarkPathIsTheOneTheAgentReads` (the path is the agent's guest path), `TestBusyMarkIsWiredAtStartup`.
|
||||
- **D65 = A — „Felhom support restarted your box at <time>."** The dashboard's one neutral line for a day after an
|
||||
operator reboot, HU/EN (`dashboard.operator_reboot`); the idle dashboard is byte-identical (the parity fixtures pass
|
||||
unchanged). Tests `TestOperatorRebootNotice_*`, `TestWatchOperatorReboot_RetriesAndKeepsTheTime`.
|
||||
- **R-1350 — no „installed app is not running" after the operator's hold clear.** Measured 2026-10-29 14:57 on Tester 1:
|
||||
gokapi's crash-loop hold cleared from the hub, the app stayed stopped, and 30 s later `app_start_failed` went out —
|
||||
the hold had kept it out of the dead-app scan while the household's recorded wish was still „running". The clear now
|
||||
records the stop as wanted (desired state „stopped") for an app that is not up, once the hold is found and before it
|
||||
is lifted; a running app's wish is left alone; a refused clear records nothing; the household's Start ends it.
|
||||
Tests `TestR1350_*` (the consequence through `classifyRunStates`, the order against the lift, the refusal, the up
|
||||
check, the wiring).
|
||||
- Every new test seen failing by a mutation (17 mutants; two first-run mutants that did not compile were re-made so they
|
||||
compile, and then failed).
|
||||
|
||||
## v0.348.0 (2026-10-29) — clear_hold from the hub (D64 = B); the box command clears through the running controller (R-1343); the held-app mail and line name the real cause (R-1340); the OOM line once per kill (R-1341); the restore copy safe during the stable-start watch (R-1342) — built from the commit carrying this heading
|
||||
|
||||
### 2026-10-29 (afternoon)
|
||||
|
||||
**MinAgent: 0.158.1** (unchanged). The next release also carries R-1342 (`652a690`, already on main, the entry below).
|
||||
**No new event type** — the hub records the operator action's result as its own `operator_action` event.
|
||||
|
||||
- **`clear_hold` — a fifth verb in the CLOSED operator-action list (operator ruling D64 = B, 2026-10-29 12:10)**
|
||||
(`internal/report/opactions.go`, wired in `cmd/controller/opactions_wiring.go`; arg = the app's stack name, letters,
|
||||
digits, `-`, `_`, ≤64 — checked before any handler). It lifts the app's hold of ANY reason in the RUNNING controller
|
||||
(`backup.AppStopGuard.ClearHoldByOperator` → `settings.ClearRestoreHold`: memory and `settings.json` together), so no
|
||||
later save can write it back. It starts nothing. Refused, nothing changed: a bad name, an app with no hold. One log
|
||||
line names who: „hold CLEARED by the operator via the hub, action #<id>". Every reader of the hold (page, start gate,
|
||||
API, `RestoreHoldFor`, hold reporter) reads the same settings record; the ONE held flag cached elsewhere —
|
||||
`stacks.Manager`'s „the last update ended held" (it made the crash-loop observer and the nightly update leg skip the
|
||||
app) — is forgotten too (`ForgetUpdateHeld`). A `putback_pending` hold is also dropped from the app-stop marker, or the
|
||||
next start with the drive bound would pour the old copy over what the app wrote after the clear; the copy stays on
|
||||
disk. The hold reporter needs no change: its next tick (≤1 min) sends `app_hold_cleared`.
|
||||
Tests: `TestOpActions_ClosedList` (updated deliberately), `TestOpActions_ClearHoldBadArgumentIsRefusedAndCallsNothing`,
|
||||
`TestOpActions_ClearHoldDoneRefusedFailed`, `TestD64_ClearHoldViaTheHub`, `TestD64_ClearHoldRefusals`,
|
||||
`TestD64_ClearHoldStaysGoneAfterALaterSave`, `TestD64_ClearPutBackPendingDropsTheMarkerEntry`,
|
||||
`TestD64_TheHoldReporterTellsTheHubAfterTheClear`, `TestD64_ForgetUpdateHeld`, `TestD64_MainWiresTheNewPieces`.
|
||||
- **R-1343 — the box command `--clear-restore-hold <app>` goes through the running controller** (`cmd/controller/holdclear.go`):
|
||||
a unix socket `<data dir>/ctl/hold.sock`, 0600 in a 0700 dir, peer uid checked (SO_PEERCRED), nothing on TCP — not
|
||||
reachable through traefik, the LAN or an app container. Output: „Cleared by the RUNNING controller … no restart is
|
||||
needed". Only when no controller answers (no socket, or a stale one) does it edit `settings.json`, and it says so,
|
||||
naming the restart INSIDE the guest (`pct exec <vmid> -- systemctl restart felhom-controller-bootstrap.service`); that
|
||||
edit goes through the same door, so a waiting put-back is dropped from the marker there too. The
|
||||
flag's help names the exact guest command. `--restore-holds` (the list) is unchanged.
|
||||
Tests: `TestR1343_TheBoxCommandClearsThroughTheRunningController` (red with the socket skipped: „the hold came back
|
||||
after a later save of the running controller" — R-1343 reproduced), `TestR1343_TheBoxCommandFallsBackWhenNoControllerRuns`,
|
||||
`TestR1343_TheFallbackDropsAWaitingPutBackToo`, `TestHoldCtl_NotReachableFromOutside`,
|
||||
`TestHoldCtl_UnknownRequestIsRefused`, `TestHoldCtl_ANonSocketFileIsNotReplaced`.
|
||||
- **R-1340, second producer (found live 2026-10-29 on rc.1, Tester 1):** the hold's `Why()` — the hub's „Held apps" line,
|
||||
the box's `app_held` event and `--restore-holds` — still said „held after an off-site restore failed its database
|
||||
replay" for a LOCAL restore of an app with no database. It is now built from the record (`RestoreKind`, `MixedCause`,
|
||||
`FailedStep`, `DBReplay`); a record from before these fields gets the neutral „held after a restore stopped half-way".
|
||||
Test `TestR1340_WhyNamesTheMixedCause` (13 assertions failed on the old sentence).
|
||||
- **R-1340 — the operator's „HELD STOPPED" mail names the real cause** (`cmd/controller/holdmail.go`
|
||||
`heldStoppedOperatorText`, built from the hold record). Hold writers now record `RestoreKind` (off-site / local),
|
||||
`MixedCause` (`definition_write` / `put_back` / `interrupted_put_back`), `FailedStep` (the put-back step that failed,
|
||||
with its error), `DBReplay`, and the kept copy's folder (`settings.RestoreHold`, all `omitempty`). The mail never says
|
||||
„database" unless the failure was a database replay, never „off-site" for a local restore; a pre-R-1340 mixed hold
|
||||
names only what its record says. Legs unchanged (`restore-hold`, `restore-hold-mixed`); a local restore's RunKind is
|
||||
`restore`. Tests (red with the old sentence/record): `TestR1340_UnhealthyStop`, `_UpdateFailed`,
|
||||
`_RestoreFailedWithADatabase`, `_PutBackWithoutADatabase` (the measured Radicale case), `_PutBackAfterADatabaseReplay`,
|
||||
`_DefinitionWrite`, `_LocalRestore`, `_PutBackPending`, `_InterruptedPutBack`, `_LegacyMixedHoldAssumesNothing`,
|
||||
`_NotifyReadsTheHoldRecord`; writer side `TestR1340_AVolumeFailureWhosePutBackStopFailsRecordsTheStepAndNoDatabase`,
|
||||
`_ADatabaseReplayFailureRecordsTheDatabase`, `_ADefinitionWriteFailureRecordsItsCause`, `_ALocalRestoreRecordsItsKind`,
|
||||
`_AnInterruptedPutBackRecordsItsCause`. `TestEventMessageWireTextIsFrozen` is untouched (this text is passed through).
|
||||
- **R-1341 — the OOM line once per NEW kill** (`cmd/controller/oomlog.go` `oomLogGate`, key = container + start time +
|
||||
kill count): a second scan of the same kill logs nothing; a new kill or a restarted container logs again. The
|
||||
`app_oom` event call is unchanged. Tests: `TestR1341_ASecondScanOfTheSameKillLogsNothing`, `_ANewKillLogsAgain`,
|
||||
`_ARestartedContainerLogsAgain`, `_ContainersAreJudgedApart`.
|
||||
|
||||
### 2026-10-29 (after v0.347.0) — R-1342: the restore copy is safe during the stable-start watch; a removed app does not alarm
|
||||
|
||||
**MinAgent: 0.158.1** (unchanged). Found live on Tester 1 at 10:39 CET: a false `backup_run_failures` mail („Gokapi did
|
||||
not start … the pre-restore copy is kept") after the copy had been deleted by hand mid-watch and the app removed.
|
||||
- **(1) D62 gap — `internal/backup/restore_copies.go`:** while a stable-start watch (`releaseAfterStableStart`, and the
|
||||
late-start watcher) holds a copy, the copy reads „checking" in the list (one line, no delete date) and
|
||||
`DeleteRestoreCopy` refuses it (`ErrRestoreCopyNeeded`) — the operator's rule is „never while an operation runs on the
|
||||
app". The mark lives in `Manager.watching` under `m.mu`, the lock the delete checks under.
|
||||
- **(2) R-1270 watch — `internal/backup/restore_verdict.go`:** the watch ends quietly (`stableRemoved`, one INFO line, no
|
||||
alarm) when the provider says the app was removed (`StackRemovalReporter`; the production `stackAdapter.StackRemoved`
|
||||
with a compile-time assertion in `cmd/controller/main.go`). An installed app that never runs stably still alarms once.
|
||||
- Tests `TestR1342_ACopyUnderTheStableWatchIsNotDeletable`, `TestR1342_AnAppRemovedDuringTheWatchRaisesNoAlarm` (each
|
||||
red by a mutation), the control `TestR1342_AnInstalledUnstableAppStillAlarms`, and `TestR1342_EveryNeededReasonHasItsSentence`.
|
||||
|
||||
## v0.347.0 (2026-10-29) — D62: restore copies kept 30 days, listed, deletable by hand; R-1256: four restore edges, all proven live (D63); R-1254 controller half; R-528: the host's OOM count; an unknown app is a 404 — RELEASED (image `sha256:c79b06e6…`, from `fd75c7c`, CI 2491; rc.1 = `e3fce09` on scratch 9202); per-box floors 09:57 CET, global floor 2026-10-29 10:31 CET on the operator's yes (MinAgent 0.158.1)
|
||||
|
||||
### 2026-10-29 (attended) — D62 and D63
|
||||
|
||||
**MinAgent: 0.158.1** (unchanged). Needs hub ≥ the release carrying `restore_copy_deleted` (felhom.eu `19afefab`): an
|
||||
older hub answers that one event 400 and the record is lost (the delete itself is unaffected).
|
||||
|
||||
- **D62 (operator ruling 2026-10-29 08:17) — the age: 30 days** (`restoreCopyUnusedAge`, was 14, the night's safest
|
||||
reading). `TestD62_AnUnusedCopyIsKept30Days` (red at 14: the 20-day copy was swept).
|
||||
- **D62 — the list (`backup.RestoreCopies`, `web.restoreCopyRows`, `backups_restore.html` „Megőrzött másolatok"):** each
|
||||
restore copy shows the app, the date of the restore it protects, its size and „automatikusan törlődik: <date>". A copy
|
||||
a hold or a pending put-back needs (a hold; the putback_pending hold; the app-stop marker naming it; an unreadable
|
||||
marker) shows why in one line and its button is off — it never gets a date, because the sweep never takes it.
|
||||
- **D62 — „Törlés most" (`POST /backup/restore-copy/delete`, `backup.DeleteRestoreCopy`):** the warning names the app
|
||||
and the restore's date (the operator's text, hu + en); the household types the app's name. JSON answer: 200; 400 a
|
||||
wrong name; 404 no such copy; 409 the copy is needed, or a backup/restore runs, or the app updates. The check and the
|
||||
removal are one step under the backup manager's lock (no restore can begin in between). Path guard: both names one
|
||||
path element, the copy a real directory (not a link) directly under `backups/restore-undo/<app>/`. One `[WARN]` log
|
||||
line and one operator event `restore_copy_deleted` (who, app, restore date, size) per delete. CSRF as every form.
|
||||
Tests `TestD62_*` (backup, web, notify) — each seen failing by a mutation of the check it pins.
|
||||
- **D63 — the fault switch, TEST BUILDS ONLY (`cmd/controller/faultpoint_{on,off}.go`):** `go build -tags faultinject`
|
||||
arms two points from a file (`<data>/faults/<point>` = „<skip> <fail>"): `defwrite-after-compose` (the definition
|
||||
writer fails after the compose was written — H4) and `stop-<app>` (the app's stop fails — H5). Without the tag it
|
||||
is a no-op that reads nothing; `TestD63_TheReleaseBuildHasNoFaultSwitch` pins that, and that build.sh and the
|
||||
Dockerfile pass no build tag (red-proofed both ways).
|
||||
|
||||
### 2026-10-28 night — four restore edges (felhom.eu R-1256, review round 2 HUNT-1 H4/H5/H6/H8)
|
||||
|
||||
**MinAgent: 0.158.1** (unchanged). Each edge: a test seen failing first, then the fix; live proof on 9202 in the felhom.eu
|
||||
night report (`audits/night-2026-10-28/`). **Live 2026-10-29 on 9202:** H5's busy volume (not written into, the live data
|
||||
put back, no hold) and H8 (the 03:29 sweep removed the two old copies, kept the young one). **Live 2026-10-29 on Tester 1
|
||||
(D63, a `-tags faultinject` test build):** H4 (the live definition written back before the start) and H5's failed stop
|
||||
(nothing put back, the copy kept, the app held). H6: tests only (the old code refused at the early check too).
|
||||
|
||||
- **H4 — `internal/backup/offbox_reconstitute.go`:** an off-site restore whose definition write (or database-password
|
||||
write) fails no longer starts the app at once. The writer writes the compose BEFORE the app's settings, so the stack
|
||||
dir could hold the snapshot's (older) compose: the live definition is written back first, then the app starts
|
||||
(`restartAtLiveDefinition`); when the write-back fails too, the app is held (`restore_mixed`), never started on the
|
||||
snapshot's definition. Tests `TestR1256_H4_*`.
|
||||
- **H5 — a put-back whose stop fails does not go on.** `putBackAfterFailedRestore` (off-site and local unit restore):
|
||||
a failed stop pours nothing, keeps the copy and holds the app; a failed definition write-back keeps the copy too (D20:
|
||||
a put-back that fails keeps its copy). `CrashPutBack.PutBackInterruptedRestore` (a restore cut by a power loss): a
|
||||
failed stop is an error, so the start holds the app (R-1251 H2) instead of pouring under a running app.
|
||||
`restoreDockerVolumesFrom`: a `docker volume rm -f` that fails (the volume is in use) counts as a failed volume — the
|
||||
following `volume create` was a no-op and the tar was unpacked ON TOP of the old content (a merge). Tests
|
||||
`TestR1256_H5_*`. One log line corrected: a failed start after a put-back no longer says the copy stays (it is removed;
|
||||
the volumes are back).
|
||||
- **H6 — a restore begins in the same step as the check for every other operation.** New
|
||||
`backup.TryBeginRestoreOp` / `RestoreBusy` (under `m.mu`, the lock the second-drive mirror's `beginTier2Run` asks
|
||||
under): a restore op in flight, the backup run, a second-drive copy. Every web restore handler begins through
|
||||
`Server.beginRestoreOp` (a source test refuses a direct `BeginRestoreOp`); the kept loads (`LoadKeptApp`,
|
||||
`LoadKeptOffsite`) return `ErrRestoreBusy` with nothing started, and `LoadKeptApp` takes the dated folder's file move
|
||||
as `prepare`, inside the operation. The install-time „use my kept data" asks about the second-drive copy too
|
||||
(`RestoreBusy`, was RestoreStatus/IsRunning only). Tests `TestR1256_H6_*` (backup, web, api).
|
||||
- **R-1254 H2, the controller half (felhom.eu R-1254; agent `5ddd124`):** the escrow claim and confirm reach the agent
|
||||
with their own 60 s timeout (`agentapi.escrowCallTimeout`; was the client's 15 s) — the agent may spend its 30 s
|
||||
upload plus 15 s asking the hub before it answers, and a claim abandoned at 15 s could still store a package whose
|
||||
code reached nobody. The confirm's household sentence: 502 (the hub holds another package) keeps „…a régi kódod
|
||||
továbbra is érvényes…"; 504 (the agent cannot learn the outcome) or a failed call (0) now reads
|
||||
`api.escrow.confirm_store_unknown` („Nem tudjuk biztosan… Őrizd meg mindkét kódot…", hu + en) and never promises the
|
||||
previous code. `TestR1151_ConfirmAnswersInTheBoxLanguage`'s status-0 case changes with it, on purpose. Tests
|
||||
`TestR1254_*` (agentapi, web), red before the fix.
|
||||
- **R-528 — the OOM line sees a kill Docker does not report (felhom.eu R-528; agent `GET /guest/oom-kills`).** The
|
||||
30-s dead-app scan (`stacks.ScanOOMKilled`) now also asks the host's agent once per scan for the kernel's per-container
|
||||
`oom_kill` counter (`agentapi.GuestOOMKills`, wired in `cmd/controller/hostoom_wiring.go`, 5 s cap). A running
|
||||
container whose current run has a count above 0 is an OOM kill whatever Docker's `OOMKilled` says: the existing
|
||||
`app_oom` line (once per run, R-514/R-636) fires, its details carry `source` (`docker`, `cgroup` or `docker,cgroup`),
|
||||
and the host's count is the run's kill count. An agent without the route (404) is asked again only after 10 minutes
|
||||
and said once; no agent (9202) = Docker's flag only, as before. The inspect now reads `{{.Id}}` too. Tests
|
||||
`TestR528_*` (stacks, cmd/controller), red before the fix (the host-counted kill was not seen) or red-proved by
|
||||
mutation (the back-off, the wiring).
|
||||
- **Found walking the capability map's B2 on 9202 (small, fixed in the session):** a Stop/Start of a name that is no
|
||||
app on the box (the bootstrap-managed `felhom-controller`) answered 500 „…az alkalmazás beállításai nem menthetők." —
|
||||
the desired-state write's „not found" read as a save failure. `stacks.SetDesiredState` returns the
|
||||
`ErrStackNotFound` kind with the household's sentence (`err.stacks.alkalmazas_nem_talalhato`); `actionStack` answers
|
||||
404 before recording any intent. Nothing was ever stopped. Test `TestAnActionOnANameThatIsNoAppIsA404` (red first).
|
||||
- **H8 — `internal/backup/restore_copy_sweep.go`:** an unused restore copy goes after 14 days. Every 6 hours
|
||||
(`restore-copy-sweep`) a copy is removed only when the app has no restore hold of any kind, the app-stop marker is
|
||||
readable and does not name the app, no restore runs, and the copy is older than 14 days. Tests `TestR1256_H8_*`
|
||||
(and the wiring test).
|
||||
|
||||
## v0.346.1 (2026-10-27) — SECURITY: traefik v3.7.16, the file manager 1.5.8-stable, cloudflared 2026.10.4 — built from v0.346.0 (`b1b9be3`) + one commit (felhom.eu R-1304, R-1305)
|
||||
|
||||
**MinAgent: 0.158.1** (unchanged). Box programs only; no other change from v0.346.0.
|
||||
|
||||
+38
@@ -1,5 +1,43 @@
|
||||
# CONTEXT.md — Project Memory
|
||||
|
||||
> **2026-11-09 (attended) — v0.351.0 in code, not released.** Operator's word 2026-11-09 („go with A"): decision 56 amended
|
||||
> (an old `-rc` controller image is deleted like an old release) and decision 285 (pinned infra images keep the pin + the
|
||||
> previous pin). Measured cause: demo-hp's whole-guest local copy refused for space (felhom.eu R-548); 3.5 GB of its 27.5 GB
|
||||
> of images were these kinds. The bigger lever — the whole-guest copy carries every Docker image — is a design question
|
||||
> (felhom.eu register, filed the same day).
|
||||
|
||||
> **2026-11-03 (day, attended) — v0.350.0 RELEASED, the global floor since 08:18 CET (MinAgent 0.158.1).** Image `sha256:9ee53908…` from `5c336d8`. Traefik v3.7.17 (GHSA-xv5x), R-1347, R-1348, R-1339, R-1351, R-1323. Golden 0.350.0 vouched 10:18:10 CET.
|
||||
>
|
||||
> **2026-10-29 (evening, attended) — v0.349.0 RELEASED, the global floor since 19:24 CET (MinAgent 0.158.1).** Image
|
||||
> `sha256:c5341883…` from `94049a7`. D65 = A: `internal/busymark` keeps `<data>/ctl/busy` fresh while the controller's
|
||||
> own work runs (the agent refuses „Reboot guest" meanwhile); the dashboard shows „A Felhom ügyfélszolgálata HH:MM-kor
|
||||
> újraindította a dobozt." for a day after an operator reboot (`GET /guest/operator-reboot`, asked once at start).
|
||||
> R-1350: the operator's hold clear records the stop as wanted (desired state „stopped") before the hold is lifted —
|
||||
> proven on Tester 1, no `app_start_failed` in 6.5 minutes. 30-minute watch clean; per-box floors cleared.
|
||||
|
||||
> **2026-10-29 (afternoon, attended) — v0.348.0 RELEASED, the global floor since 14:48 CET (MinAgent 0.158.1).** Image
|
||||
> `sha256:672fc1c1…` from `2a0a17a` (CI 2515). D64 = B (`09` §3 decision 283): `clear_hold` is the fifth verb of the closed
|
||||
> operator-action list; any hold reason; the running controller clears memory and `settings.json` together through
|
||||
> `<data>/ctl/hold.sock` (0600, peer uid checked), and `--clear-restore-hold` goes through it when a controller runs
|
||||
> (R-1343). R-1340 (both producers name the real cause), R-1341 (OOM line once per kill), R-1342 (no delete during the
|
||||
> stable-start watch). Proven live on Tester 1 on the release build: a crash-loop hold cleared from the hub stayed
|
||||
> cleared after the next settings save. Open: R-1350 (an `app_start_failed` 30 s after a clear), R-1347, R-1348.
|
||||
|
||||
> **2026-10-29 (attended release run) — v0.347.0 RELEASED, the global floor (MinAgent 0.158.1).** Image `sha256:c79b06e6…`
|
||||
> from `fd75c7c` (CI 2491); rc.1 = `e3fce09` on scratch 9202. Carries the night's R-1256 (all four edges, H4 + H5's
|
||||
> failed stop proven live on Tester 1 with a `-tags faultinject` test build — D63), R-1254 controller half, R-528, the
|
||||
> unknown-app 404, and D62 (restore copies 30 days; „Törlés most" in the kept-copies list; `restore_copy_deleted` needs hub
|
||||
> ≥ 0.175.0). The fault switch (`cmd/controller/faultpoint_{on,off}.go`) is a no-op in every build without the tag
|
||||
> (`TestD63_TheReleaseBuildHasNoFaultSwitch`). Next here: R-1340 (the mixed-hold operator mail), R-1341 (the OOM WARN every
|
||||
> 30 s), R-1339 (the false „not mounted" line).
|
||||
|
||||
> **2026-10-28 night (unattended) — R-1256, R-1254 (controller half), R-528 and the unknown-app 404 on `main`, NOT
|
||||
> released (`342a394`, `e33cffa`, `eed23e7`, `20c5877`; MinAgent unchanged).** Restore ops begin through
|
||||
> `Manager.TryBeginRestoreOp` / `web.beginRestoreOp` only (`TestR1256_H6_NoHandlerBeginsARestoreUnchecked` refuses a direct
|
||||
> `BeginRestoreOp` in a handler). **Decided by CC unattended — operator may reverse:** an unused restore copy goes after
|
||||
> 14 days (`restoreCopyUnusedAge`; felhom.eu `07` §6.3). H4 and H5's failed stop: tests only, live proof owed. H5 (busy
|
||||
> volume) proven live on 9202, R-528 on Tester 1, with the local test tag `0.347.0-rc.n1028` (never pushed).
|
||||
|
||||
> **2026-10-26 (evening) — D53 ruled (felhom.eu `09` §3 decision 265): held apps reach the OPERATOR first.** Next here:
|
||||
> the controller reports its holds (stack, reason, since) to the hub as `app_held` / `app_hold_cleared` events plus a
|
||||
> daily re-send (felhom.eu `08` §10). Commit only; the release rides a later attended run.
|
||||
|
||||
@@ -1,24 +1,20 @@
|
||||
# REPORT — D53: the hub learns which apps are held — committed, NOT released (2026-10-26 evening)
|
||||
# REPORT — v0.351.0 in code: old test builds and old infrastructure images are deleted (2026-11-09, attended)
|
||||
|
||||
**Plain words first.** A held app (stopped on purpose until a person acts) now reaches the hub: the box sends one
|
||||
message when a hold appears or changes, one when it goes, and repeats every hold once a day. The hub shows the list
|
||||
and mails the operator after a waiting time (felhom.eu `7e57e824`). Not released (this run allows no release).
|
||||
**Plain words first.** demo-hp's whole-box copy was too big for its system disk. Part of it was old images that
|
||||
nothing deletes: 25 old controller test builds and old copies of the system images (traefik, cloudflared, FileBrowser,
|
||||
Samba). The controller now deletes both kinds by itself, 3 minutes after each start. **Not released yet.**
|
||||
|
||||
## The change (`5bd03ad`, CI 2377 ✓)
|
||||
| | |
|
||||
|---|---|
|
||||
| Version | 0.351.0 (CHANGELOG heading „unreleased"), MinAgent 0.158.1 unchanged |
|
||||
| Rulings | `09` §3 decision 56 amended, decision 285 new — operator 2026-11-09 („go with A") |
|
||||
| Tests | `TestControllerRetention_OldTestBuildsGo`, `_BoxOnATestBuildIsJudged`, `TestInfraRetention_*` (4); one existing rc expectation changed with the rule |
|
||||
| Red-proof | six code mutants, each test red: previous-pin keep, in-use check, R-863 lock, tag parser, running-version parser, release preference |
|
||||
| Green gate | `go build ./...`, `go vet ./...`, `go test -count=1 ./...` rc 0; `controller_gates.py --fast` OK |
|
||||
|
||||
- `internal/holdreport`: one observer of the hold store, ticked every minute by the scheduler (`hold-report`). It sends
|
||||
`app_held` (stack, reason, since, the hold's sentence) for a new or changed hold, `app_hold_cleared` when a hold is
|
||||
gone — also one cleared while the controller was down (`settings.reported_holds`) — and every hold again daily and at
|
||||
start. Severity info.
|
||||
- `settings.RestoreHold.Why()` / `WireReason()`: the hold sentence lives in one place (`holdWhy` calls it); the legacy
|
||||
empty reason is sent as `restore_failed`.
|
||||
- Tests `internal/holdreport/holdreport_test.go` (8). Red-proved: an inert `Tick` failed all six behaviour tests; the
|
||||
scheduler line commented out failed the wiring test.
|
||||
**Measured cause** (felhom.eu `REPORT.md`): on demo-hp guest 9201, 2.2 GB of old controller images and 1.35 GB of old
|
||||
infra and helper images, of 27.5 GB. The same images were removed there by hand the same day, so a live proof needs
|
||||
a box that still has them (Tester 1: 37 images, 3.8 GB unused).
|
||||
|
||||
## Live proof (felhom.eu `documentation/audits/d53-held-apps-2026-10-26/`)
|
||||
|
||||
9202 ran a local-only image of this commit (`0.346.0-d53test`, never pushed to the registry) against a test hub process.
|
||||
A throwaway Radicale, crashed seven times, was stopped and held by the box (17:37:52Z); `app_held` reached the hub at the
|
||||
next tick; one operator reminder after the 3-minute test grace; the household's Start lifted the hold and
|
||||
`app_hold_cleared` removed the row. The first tick also reported four holds 9202 has carried since 2026-09-24 for apps
|
||||
that are not installed (felhom.eu R-1298). 9202 is back on 0.343.0 with its own config.
|
||||
**Not done:** no image built, no box moved, no floor. The release path is the skill's: an `-rc` on the three test boxes,
|
||||
the box walk, then the tag and the floor.
|
||||
@@ -35,7 +35,8 @@
|
||||
| `dockerexec.BeginImageWork(args)` / `TryImageCleanup()` / `ImagePulling(args)` (v0.294.0, R-863) | controller/internal/dockerexec/imagework.go | `defer dockerexec.BeginImageWork(args)()` | **Every NEW place that runs `compose up/pull/create/run` must hold it** (the stacks compose helpers, appexport's restore and the FileBrowser sync already do) | An image pulled by compose is named by no container until compose creates one; a clean-up pass in that window deleted it (measured, R-863). The pass takes the lock exclusively without waiting; pulling commands wait for a running pass (seconds) |
|
||||
| `nightchain.Ledger` / `CatchUp` / `ResumeWatch` / `ComputeBanner` (v0.295.0, R-871) | controller/internal/nightchain/ | `Open(path, now)`, `MarkEnded(leg, t)`, `Missed(now, W, loc)`, `Evaluate(ctx, why)` | "was a night missed?" and making it up; the missed-backup banner's rule | The ledger is an ATTEMPT record (ran to its end) — never evidence a backup exists; a NEW nightly backup leg must be wrapped by `withLeg` in main.go and listed in `nightchain.Order`, or the catch-up never makes it up; never add the update leg to `catchUpLegs` |
|
||||
| `scheduler.DailyLateLimit` (v0.295.0) | controller/internal/scheduler/scheduler.go | — | a daily timer firing > 60 min late (host suspend) is skipped | do not "fix" a late fire by running it: the app-update leg would run at noon |
|
||||
| `stacks.RetainControllerImages(ControllerImageRecord)` · `selfupdate.UpdateState.RecordedPrevious` (v0.285.0, decision 56) | controller/internal/stacks/controller_image_retention.go | `({Repo, Running, Previous})` | Deletes controller images older than the running + previous one | The previous comes from the SWAP RECORD (success onto the running version), version order only as the fallback; versions above the running one and non-version tags are kept; skipped while the controller swaps itself; same `imageDocker` seam |
|
||||
| `stacks.RetainControllerImages(ControllerImageRecord)` · `selfupdate.UpdateState.RecordedPrevious` (v0.285.0, decision 56) | controller/internal/stacks/controller_image_retention.go | `({Repo, Running, Previous})` | Deletes controller images older than the running + previous one | The previous comes from the SWAP RECORD (success onto the running version), version order only as the fallback (a release preferred over a test build, v0.351.0); versions above the running one and non-version tags (`latest`) are kept — an `-rc` tag IS a version (`util.ParseVersionPre`) since v0.351.0; skipped while the controller swaps itself; same `imageDocker` seam |
|
||||
| `stacks.RetainInfraImages(pins)` / `RunInfraImageRetention(ctx, pins, every, tries)` (v0.351.0, decision 285) | controller/internal/stacks/infra_image_retention.go | `(infra.Images())` | Deletes pinned infra images older than the pin + the previous pin | Per repository: the pin, newer versions, the previous pin and in-use images stay; untagged unused go; non-version tags stay; skipped (errImageRetentionBusy) while an app pulls (R-863) or the controller swaps; same `imageDocker` seam |
|
||||
| `stacks.CloseSignupNow` / `CloseSignupOffered` / `applyNativeLock` (v0.282.0, decisions 47/49) | controller/internal/stacks/after_setup.go | `(name)` | The app's own sign-up switch after the setup; "close sign-up now" for an app installed before the rule | **Check the installed compose reads the variable** (an old install carries the old compose until its next update) — never record a lock that is not there. One run per app at a time (`nativeLockBusy`) |
|
||||
| `backup.judgeCopy` / `HollowCopies` / `SetHollowCopyNotify` (Part D, v0.279.0) | controller/internal/backup/hollow_watch.go | `(app, tier, unitDir)` | A RUNNING app whose newest copy holds no data → operator digest once/day + page sentence | Uses `unitCarriesData` (the manifest, never size); a stopped held app is never flagged |
|
||||
| `web.nightChain` (R-705, v0.279.0) | controller/internal/web/night_chain.go | `POST /api/debug/backup/night-chain` | The night's four legs now, in order | Refuses while any op/update/chain runs; the leg uses `RunUpdateLegNow` |
|
||||
@@ -80,12 +81,15 @@
|
||||
| `restoreOpInFlight` + `hasRecentRestoreResult` | controller/internal/web/restore_wizard.go | `(backup.RestoreOpStatus) bool` / `(st, app, now) bool` | THE "is a restore running / did one just finish" display reads | **TRAP (v0.154.0 shipped this bug): `Manager` has TWO running flags.** `IsRunning()` reads the CONCURRENCY flag, acquired inside the goroutine — and `RestoreOffboxScratch` never acquires it, so it is false for the whole verification restore. Display must read `RestoreStatus().Running` (set synchronously by `BeginRestoreOp`). Read the status ONCE per render or the strip and the suppression can disagree. `hasRecentRestoreResult` is app-bound and window-bounded — a process-wide result must not light another app's „Eredmény" |
|
||||
| `restoreWizardPath` / `deriveWizardStep` / `resolveWizardApp` | controller/internal/web/restore_wizard.go | `(app) string` / `(restoreWizardInput) restoreWizardView` / `([]OffboxAppRow, name) *OffboxAppRow` | R-48 offsite restore wizard: URL builder + the PURE step/unlock derivation + the app-resolution refusals | The step is **never** taken from the request. Precedence is load-bearing: op-running outranks a stale `?full_prep=`, else a commit button reappears mid-restore. Truth table + red-proof: `restore_wizard_test.go`. Adding a form here that posts anywhere new breaks `TestRestoreWizard_NoNewMutationEndpoints` **by design** — R-48 adds no mutation surface |
|
||||
| `restoreOpBlocked` | controller/internal/web/restore_wizard.go | `() (msg string, blocked bool)` | THE refusal gate before starting ANY restore | **Use this, never a bare `IsRunning()`.** It reads BOTH flags: `RestoreStatus().Running` (set synchronously by `BeginRestoreOp`, true for the whole off-box restore) and `IsRunning()` (the concurrency flag, the only one the nightly backup holds). **R-351: all seven handlers read only `IsRunning()`, which the goroutine acquires AFTER the handler returns — a second press started a second run and was told „…elindult".** Returns the Hungarian refusal, which names the running app and a route | **R-360 (v0.226.0): `offboxVerifyCopyDeleteHandler` was the LAST holdout and its doc comment claimed it already did this — the sentence is why nobody looked. `IsRunning()` is FALSE for the whole of a verification restore, so the copy a restore was writing into could be deleted from the UI (observed live 2026-08-21 22:35). No app-name comparison: refusing during ANY restore is stronger and uniform.**
|
||||
| `Server.beginRestoreOp` → `backup.TryBeginRestoreOp` / `RestoreBusy` (R-1256 H6) | controller/internal/web/restore_wizard.go; controller/internal/backup/opstatus.go | `(op, stack) (msg string, ok bool)` / `(op, stack) (busy, mirrorApp string)` | THE way a restore op begins: the check for every other operation (a restore op, the backup run, a second-drive copy) and the record in ONE step under `m.mu` — the mirror asks under the same lock (`beginTier2Run`) | **Never call `BeginRestoreOp` from a handler** (`TestR1256_H6_NoHandlerBeginsARestoreUnchecked`): `restoreOpBlocked` asked early can be stale by the time the form is read, and a mirror that began in between ran beside the restore. `restoreOpBlocked` stays as the EARLY refusal; the kept loads begin inside `LoadKeptApp` / `LoadKeptOffsite` and return `ErrRestoreBusy` |
|
||||
| `Manager.RestoreFromRecoveryUnitAt` / `RestoreFromRecoveryUnit` + `UnitRestoreResult` (R-353 v0.226.0; R-102 v0.229.0) | controller/internal/backup/restore_unit.go | `(stack, unitDir string) (UnitRestoreResult, error)` / `(stack string) (…)` | THE local recovery-unit restore, and the facts its surface must state | **Returns a RESULT, not just an error** — volumes replayed, DBs replayed, and what the manifest LISTED. The Manifest* counts are load-bearing: zero-replayed has two causes (the backup held no data / the backup listed data that did not come back) and they are opposite news. Pair it with `unitRestoreOutcomeMsg`; do NOT write a new sentence. **A claim about the APP is forbidden on this path** — it has no `SafetyDump` discriminator, unlike the off-site twin (CONTEXT.md ruling, 07-backup-architecture §6.3). `restoreDockerVolumes` now returns `(int, error)`; `restoreDockerVolumesFrom` is unchanged and still the shared implementation. **R-102: `…At` holds the whole body and takes the unit DIRECTORY; the one-argument form is the thin caller naming the primary unit.** THE SOURCE MOVES; THE DESTINATION DOES NOT — `GetAppDrivePath` still resolves where the data lands. The R-47 mutation order, the unit-over-guest secret precedence, the fail-closed data-key gate and the no-unit `RestoreApp` fallback are all pinned and unchanged. The volume leg goes through the R-354 `volumeReplayFrom` seam so the source directory is assertable without Docker |
|
||||
| `Manager.putBackAfterFailedRestore` + `liveVolumeCopy` (R-893 option A, v0.336.0) | controller/internal/backup/offbox_restore_undo.go | `(stack, cause, live, copy, volumesMoved, filesPlaced, restart, holdKey) error` | THE restore undo — off-site (v0.336.0) and, since D22 (unreleased), the LOCAL unit restore (`RestoreFromRecoveryUnitAtWith`, so also the second drive's and a kept load): the live volumes it will replace, measured before the stop (`planLiveVolumeCopy`), copied right after it (`copyLiveVolumes`), poured back on any failure | One copy per restore under `<ns>/backups/restore-undo/<app>/` (no capture set reads `backups/`); the put-back uses the restore's own `volumeReplayFrom`/`restoreDockerVolumesFrom`, so there is one volume-replay implementation. A put-back that fails sets `keep` and HOLDS (option C) — never start on volumes that are not known to be back. Seams for tests without Docker: `undoVolumeProbeFn`, `undoVolumeTarFn`, `undoVolumeRemoveFn`; operator notice `SetRestorePutBackNotify` (leg `restore-put-back`). Pinned by `r893_putback_test.go` and `r1177_local_putback_test.go` |
|
||||
| `Manager.putBackAfterFailedRestore` + `liveVolumeCopy` (R-893 option A, v0.336.0) | controller/internal/backup/offbox_restore_undo.go | `(stack, cause, live, copy, volumesMoved, filesPlaced, restart, holdKey, restoreKind) error` (R-1340: `restoreKind` = `settings.RestoreKindOffsite`/`Local`; a failed put-back records its step in the hold's `MixedCause`/`FailedStep`/`DBReplay`) | THE restore undo — off-site (v0.336.0) and, since D22 (unreleased), the LOCAL unit restore (`RestoreFromRecoveryUnitAtWith`, so also the second drive's and a kept load): the live volumes it will replace, measured before the stop (`planLiveVolumeCopy`), copied right after it (`copyLiveVolumes`), poured back on any failure | One copy per restore under `<ns>/backups/restore-undo/<app>/` (no capture set reads `backups/`); the put-back uses the restore's own `volumeReplayFrom`/`restoreDockerVolumesFrom`, so there is one volume-replay implementation. A put-back that fails sets `keep` and HOLDS (option C) — never start on volumes that are not known to be back. Seams for tests without Docker: `undoVolumeProbeFn`, `undoVolumeTarFn`, `undoVolumeRemoveFn`; operator notice `SetRestorePutBackNotify` (leg `restore-put-back`). Pinned by `r893_putback_test.go` and `r1177_local_putback_test.go` |
|
||||
| `Manager.SweepUnusedRestoreCopies` (R-1256 H8) | controller/internal/backup/restore_copy_sweep.go | `(now) []string` | THE only AUTOMATIC remover of a restore copy nothing names any more (`restore-copy-sweep`, every 6 h); the household's „Törlés most" is `DeleteRestoreCopy` (D62, restore_copies.go — same „needed" rule via `restoreCopyNeededFor`, check + removal under `m.mu`) | Removes a copy only when ALL hold: older than 30 days (D62), the app has no restore hold of ANY kind, the app-stop marker is readable and does not name the app, no restore op runs. It answers „no" when it cannot answer — a held or waiting copy is support's way back |
|
||||
| `CrashPutBack` + `Manager.persistUndoForCrash` + `AppStopGuard.SetUndo` (D22 / R-1195, v0.338.0) | controller/internal/backup/restore_crash_putback.go | `NewCrashPutBack(stop, writeDef, logger)`; `PutBackInterruptedRestore(stack, undoDir) error` | A restore cut by a power loss is put back at start: the copy (volumes + `definition/` + `undo.json`) is recorded in the app-stop marker once whole; `Recover` pours it back BEFORE the restart | Call `persistUndoForCrash` right after `copyLiveVolumes` in any new restore path. `Recover` runs before the backup manager exists, so the put-back is wired in main.go with `stackMgr.StopStack` and the stack adapter's definition writer. A failed put-back = a FAILED recovery (app not started, marker kept). The household reads `note.restore.interrupted_put_back` (`LoadRestoreRecordAfterRecovery`). Pinned by `TestR1195_*` |
|
||||
| `tier2CaptureSetFull(...) (legs, missing, warns, classified)` + `keptCopyRels` + `dropKeptCopiesUnder` (R-1252, unreleased) | controller/internal/backup/tier2_capture.go, tier2_kept.go | `classified` = the legs come from a VALID backup block | **The second-drive reconcile deletes only under a valid classification**; a rejected or absent block keeps every dest dir; kept copies (D25) are always in the keep set; a mirror deleted by Remove drops its kept records | A new reconcile caller must take `classified` and the kept rels too — never reconcile against the legacy legs. A rejected block is told by `Manager.BackupBlockRejected` (stacks; via the adapter's optional `BackupBlockRejected` method) and signalled once a day (`SetTier2SettingsUnreadableNotify`). Pinned by `r1252_rejected_block_keeps_copies_test.go` |
|
||||
| `holdreport.Reporter` + `settings.RestoreHold.Why` / `WireReason` (D53, unreleased) | controller/internal/holdreport/holdreport.go · controller/internal/settings/settings.go | `New(store, notifier.PushEvent, logger)`; `Tick(now)` from the scheduler | Tells the hub every hold in place (`app_held`), every hold gone (`app_hold_cleared`), each again daily | **One observer of the hold store — never a send inside a hold writer.** A new hold writer needs nothing here. The hold sentence is `RestoreHold.Why()` (one per reason) — never a second copy. Pinned by `internal/holdreport/holdreport_test.go` |
|
||||
| `AppStopGuard.ResumePutBack` / `ResumeWaitingPutBacks` + `HoldReasonPutBackPending` (R-1251, unreleased) | controller/internal/backup/appstop_putback_wait.go | `ResumePutBack(stack) bool` (true = nothing owed, the caller may start); `ResumeWaitingPutBacks() []string` (puts back, then starts through the gated starter) | A cut restore's copy on a drive that is not bound is WAITING: Recover writes the restore hold `putback_pending` (copy in `SafetyDump`) and starts nothing; the put-back runs when the drive binds, before any start | **Any new path that starts an app when its drive arrives calls `ResumePutBack` first** (the boot recreate and `restartStacks` do, in web via `Server.resumePutBack`). Every other start path is covered by the hold store it already honours — do not add a second check. A failed put-back (at start or on resume) writes the D20 hold (`restore_mixed`) and drops the app from the marker. Needs `SetHolds` + `SetUndoReachable` before `Recover` (main.go). Pinned by `r1251_putback_waits_test.go`, `TestR1251_DriveStartPathsPutBackBeforeTheyStart`, `TestR1251_MainWiresThePutBackWait` |
|
||||
| `AppStopGuard.ClearHoldByOperator` + `clearHoldNow` / `listenHoldCtl` / `clearRestoreHoldCLI` (D64 = B, R-1343, unreleased) | controller/internal/backup/hold_clear.go · controller/cmd/controller/holdclear.go | `ClearHoldByOperator(stack) (RestoreHold, error)`; `clearHoldNow(deps, stack, who) (msg, error)` | THE operator's hold clear, any reason — the hub's `clear_hold` and the box command `--clear-restore-hold` both go through `clearHoldNow` in the RUNNING controller | **Never clear a hold from a second process** — the running controller's next save writes it back (R-1343, measured). The box command reaches the running controller over `<data dir>/ctl/hold.sock` (unix, 0600, peer uid checked; never TCP/HTTP). A `putback_pending` clear also drops the app from the app-stop marker (the copy stays). Refusals (`ErrHoldBadStack`, `ErrNotHeld`) change nothing. The update's in-memory held flag goes via `stacks.Manager.ForgetUpdateHeld`. Pinned by `cmd/controller/holdclear_test.go` |
|
||||
| `Manager.RestoreTier2Unit` + `Tier2Coverage.CanRestoreUnit` / `Tier2CopyDate` (R-102/R-103, v0.229.0) | controller/internal/backup/tier2_restore.go | `(stack string) (UnitRestoreResult, error)` / `() bool` / `() (string, bool)` | THE full restore from the SECOND DRIVE's mirror, and the predicate that gates it | **Do NOT widen `CanRestore()`** — it answers only "can the additive file restore run?" and one predicate answering two questions is R-356. Fail-closed: `tier2UnitIsOpenable` requires a parseable manifest, because a directory is not a package. The single-writer flag is taken INSIDE `RestoreFromRecoveryUnitAt` — a second `acquireRunning()` here would refuse the restore it guards. `Tier2CopyDate` prefers the SUCCESS anchor over the attempt clock (R-101) and reports which it returned |
|
||||
| `unitRestoreOutcomeMsg` + its three message constants (R-353, v0.226.0) | controller/internal/web/handlers.go | `(app string, res backup.UnitRestoreResult) string` | THE customer sentence for a completed LOCAL restore | Twin of `reconstituteOutcomeMsg`; copy its SHAPE (clauses earned by having done the thing, no filesystem path, base names only), never its text. The constants are named because `r353_unit_outcome_test.go` asserts them verbatim — a silent edit is how an honest message drifts back into a comforting one, which is the documented history of the sentence it replaces |
|
||||
| `offsiteNoSpaceMsgFmt` + `offsiteSizeUnknownMsg` (R-357, v0.226.0) | controller/internal/backup/offbox_restore.go | two consts | EVERY headroom refusal on the off-site restore surface | **All four gates share these** (prepare, scratch, place, and the destructive reconstitute). A customer meeting one wording on one path and a different one on another has to work out whether it is the same problem. **The reconstitute gate uses NO ×1.1 margin** — it copies a measured tree; `OffboxRestorePrepareFull`'s ×1.1 predicts a download. **Fail closed when either probe reads ≤ 0**: `free < need` with `need == 0` is FALSE, so an unmeasurable input sails through — a gate present and inert |
|
||||
@@ -357,6 +361,7 @@
|
||||
|---|---|---|---|
|
||||
| `diskAgent` | controller/internal/web/storage_handlers.go | `*agentapi.Client` | `mockAgent` in controller/internal/web/storage_handlers_test.go |
|
||||
| `netAgent` + `Server.netAgentFn/netProbeFn/netListFn` | controller/internal/web/netstorage_job.go (+ server.go fields) | `*agentapi.Client` / `runNetProbe` (linux re-exec) / `agent.ListNetStorage` | `fakeNetAgent` + fn injections in controller/internal/web/netstorage_job_test.go — the NAS add orchestration never shells/TLS-dials in tests |
|
||||
| `faultPoint(point) error` (D63, TEST BUILDS ONLY) | controller/cmd/controller/faultpoint_{off,on}.go | a no-op without `-tags faultinject` (the release image); with it, a file `<data>/faults/<point>` = „<skip> <fail>" fails the named point | `TestD63_TheReleaseBuildHasNoFaultSwitch` (release build: no-op; build.sh/Dockerfile pass no tag), `TestD63_TheTestBuildFailsTheArmedPointOnDemand` (`go test -tags faultinject`). Points: `defwrite-after-compose`, `stop-<app>`. A live proof of a branch Docker cannot be made to take on demand goes here — never an env switch in the release binary |
|
||||
| `healthSettle` / `healthInterval` / `healthTimeout` (package vars, R-488) | controller/internal/backup/restore.go | production 3 s / 5 s / 90 s — `waitForHealthy`'s clock, used by every restore path | shortened ONCE in controller/internal/backup/main_test.go `TestMain` (0 / 2 ms / 50 ms; the package went 444 s → ~2 s); `TestR488_HealthWaitDefaultsAreProduction` pins the production values. A new real-clock wait in this package goes behind the same kind of var, not a per-test sleep. Also: `newOffboxManager` installs a failing `SetOffboxSSH` fake, so no backup test spawns a real `ssh` |
|
||||
| `Server.agentLogsFn` (func seam) | controller/internal/web/server.go | nil → `agentClient().DebugLogs` (agent GET /debug/logs) | injected in controller/internal/web/observability_test.go (incl. the pre-0.83 typed-404 notice path) |
|
||||
| `escrowAgent` + `Server.escrowAgentFn/escrowStageFn/escrowStaleFn` | controller/internal/web/escrow_handlers.go (+ server.go fields) | `*agentapi.Client` / `PushOffboxPasswordForEscrow` / `report.EscrowAutoConfirmer.StaleBlob` (SetEscrowStale) | `fakeEscrowAgent` + fn injections in escrow_wizard_test.go — call-ORDER assertions (stage BEFORE trigger) + agent-never-called gates. The claim leg is the ONLY surface R crosses: no-store, never logged, never templated |
|
||||
@@ -453,7 +458,7 @@ Cross-repo edges:
|
||||
| timeAgo switch body ×2 | controller/internal/web/funcmap.go `timeAgo` vs `timeAgoStr` (identical formatting logic) |
|
||||
| Budapest timezone loader ×4 | controller/internal/scheduler/scheduler.go `getBudapestLocation` vs controller/internal/web/funcmap.go `getTimezone` vs controller/internal/quiesce/quiesce.go `budapestLocation` vs controller/internal/api/kept_install.go `budapest` (v0.277.0, the kept choice's dates) (v0.168.0 window gate — Budapest wall-clock, kept local to avoid a scheduler↔quiesce import edge) |
|
||||
| JSON writers ×5, 3 envelope shapes | api `writeJSON`; web `writeDiskJSON`, `jsonResponse`/`jsonError`, `writeDebugJSON` |
|
||||
| Safe-name validators ×4 | controller/internal/web/validate.go `validStackName`; controller/internal/api/router.go `validStackParam` (same body — api↔web import cycle); controller/internal/backup/offbox.go `isSafeStackName`; controller/internal/appexport/validate.go `ValidateSegment` (strictest) |
|
||||
| Safe-name validators ×5 | controller/internal/web/validate.go `validStackName`; controller/internal/api/router.go `validStackParam` (same body — api↔web import cycle); controller/internal/backup/offbox.go `isSafeStackName` (exported as `IsValidHoldStack` for the hold commands); controller/internal/report/opactions.go `validHoldStack` (same alphabet as `isSafeStackName` — report cannot import backup; clear_hold's argument check); controller/internal/appexport/validate.go `ValidateSegment` (strictest) |
|
||||
| DB wait/import ×2 | controller/internal/appbackup/dbdump.go `waitDBReady`/`ImportDump` vs controller/internal/appexport/restore.go `waitForDB`/`importDBDump` |
|
||||
| qBittorrent Web API login ×2 | controller/internal/mediawire/mediawire.go `qbLogin` (writes the drive wiring) vs controller/internal/upwatch/explain.go `QBittorrent.login` (reads every minute; keeps its session cookie, which mediawire's per-run login does not need) |
|
||||
| compose exec ×2 | controller/internal/stacks/manager.go `composeExecCustomEnv` vs controller/internal/appexport/restore.go `composeExecEnv` (the latter has ctx+timeout; the former has the userdata belt) |
|
||||
+51
-2
@@ -1017,7 +1017,8 @@ Each app can define rich metadata in `.felhom.yml`:
|
||||
the hold is persisted in `Settings.RestoreHolds`, every start path refuses it (customer button,
|
||||
app-stop `Recover()`, boot sweep, via `driveStartGate` **above** its driveless early return), the
|
||||
app-stop marker is ended so nothing auto-restarts it, and the row goes red. Cleared with
|
||||
`--clear-restore-hold <app>`, **which requires a controller restart**. `--single-transaction` on
|
||||
`--clear-restore-hold <app>` (since 2026-10-29, R-1343: through the running controller, no restart — see
|
||||
„Operator actions") or the hub's „Clear hold" (`clear_hold`). `--single-transaction` on
|
||||
the Postgres import is a belt only; MariaDB DDL is not transactional, which is why the rollback
|
||||
is the fix.
|
||||
- **Off-site restore of one app: a failed restore puts the app back EXACTLY as it was and starts it (v0.336.0,
|
||||
@@ -1034,6 +1035,18 @@ Each app can define rich metadata in `.felhom.yml`:
|
||||
`healthy`; the copy then goes after a further 3 min of the same (10 s samples, watched up to 30 min; a new restore
|
||||
of the app ends the watch). A restore whose app does not start reads „…még nem indult el…", keeps the copy and
|
||||
alarms the operator once (D49 = A: nothing is poured back by itself). `restore_verdict.go`. `offbox_restore_undo.go`.
|
||||
- **Restore edges (unreleased, R-1256, night 2026-10-28).** A put-back whose stop fails pours nothing, keeps the copy
|
||||
and holds the app; a failed definition write-back keeps the copy; a volume whose `docker volume rm -f` fails is a
|
||||
failed volume (never unpacked on top of the old content). An off-site restore whose definition write fails writes
|
||||
the LIVE definition back before the start, or holds the app. Every restore begins in one step with the check for a
|
||||
restore, the backup run and a second-drive copy (`TryBeginRestoreOp`, also both kept loads). A copy that nothing
|
||||
needs (no hold of any kind, not named by the app-stop marker, no restore running) goes after 30 days, checked every
|
||||
6 h (`restore_copy_sweep.go`; operator ruling D62, 2026-10-29).
|
||||
- **Restore copies in the kept-copies list (unreleased, D62).** The restore page's „Megőrzött másolatok" lists each
|
||||
restore copy: the app, the restore's date, the size, „automatikusan törlődik: <date>". „Törlés most" deletes one
|
||||
earlier, after a warning naming the app and the date and the app's name typed (`POST /backup/restore-copy/delete`,
|
||||
JSON; 409 while a backup/restore runs or the app updates). A copy a hold or a pending put-back needs: the button is
|
||||
off, the reason on one line. Each delete: one log line and the operator event `restore_copy_deleted`.
|
||||
- **A restore cut by a power loss, on a drive that comes back late (unreleased, R-1251).** The copy of the live
|
||||
data lives on the app's drive. When it cannot be reached at start because the drive is not bound yet, the app is
|
||||
HELD (`putback_pending`; „…A visszaállítás előtti adatai egy meghajtón vannak, amely most nem érhető el…") and no
|
||||
@@ -1488,6 +1501,10 @@ The nightly backup has two phases that run sequentially. All paths are **per-dri
|
||||
|
||||
> **OOM visibility (v0.243.0, R-514).** The 30 s dead-app check also reads `State.OOMKilled` for running app containers (`Manager.ScanOOMKilled`); the dashboard shows „Memória elfogyott" and the hub gets `app_oom` once per container run. The controller does not restart the app.
|
||||
> **OOM storm (v0.265.0, R-636).** For a flagged container the scan also reads the kernel's own kill counter (`memory.events` `oom_kill`, plus `memory.max` / `memory.peak`) with one `docker exec … cat` — `OOMKilled` is sticky, so it cannot count. When the SAME container run's counter rises by **20 or more within 30 minutes**, the notifier sends **`app_oom_storm`** (severity `error`, operator-only, details `{app, container, kills, window_min, mem_limit, peak}`) — ONCE per container run. `app_oom` itself is unchanged. An unreadable counter (an image without `cat`) never escalates.
|
||||
> **Host OOM count (unreleased, R-528, night 2026-10-28).** Docker inside an LXC guest misses real OOM kills. The scan
|
||||
> also asks the agent for the host's per-container `oom_kill` (`GET /guest/oom-kills`, 5 s cap; a 404 from an older
|
||||
> agent → 10 minutes before the next ask); a count above zero raises `app_oom` whatever `OOMKilled` says, with `source`
|
||||
> (`docker`, `cgroup` or `docker,cgroup`) in the details. `cmd/controller/hostoom_wiring.go`, `internal/stacks/oom.go`.
|
||||
|
||||
> **Multi-tier whole-guest backup (v0.174.0, R-82 Slice B).** The agent can serve SEVERAL whole-guest
|
||||
> backup tiers with independent cadences — "local daily + PBS weekly" (agent >= v0.97.0,
|
||||
@@ -2227,10 +2244,16 @@ that folder is never a dead end, and an install never runs into it silently (R-6
|
||||
- **Controller image retention (v0.285.0, decision 56)** — 3 minutes after every start the box keeps the controller
|
||||
image it runs and the one before it (the swap record `update-state.json` `previous_image` of a swap that ended on the
|
||||
running version; without one, the highest version below the running one), every version above the running one (a
|
||||
pulled target) and every non-version tag (`latest`, `-rc`); older and untagged controller images are deleted by exact
|
||||
pulled target) and every non-version tag (`latest`); since v0.351.0 a test build (`X.Y.Z-rc.N`) is a version below its
|
||||
release, so an old one goes too, and the fallback previous prefers a release; older and untagged controller images are deleted by exact
|
||||
name/ID after the in-use check. Nothing while the controller swaps itself; registry tags are never touched. The
|
||||
agent's own roll-back needs only the running image (it restores what `/etc/felhom-controller-image` named when the
|
||||
swap began). See `internal/stacks/controller_image_retention.go`.
|
||||
- **Infrastructure image retention (v0.351.0, decision 285)** — in the same start-up pass, for each pinned infra image
|
||||
(`infra.Images()`: traefik, cloudflared, FileBrowser, Samba) the box keeps the pin, every newer version, the highest
|
||||
version below the pin (the previous pin) and every image a container uses; older and untagged images of that
|
||||
repository are deleted by exact name/ID. Non-version tags stay. Skipped while an app pulls (R-863, retried every 2
|
||||
minutes) or while the controller swaps itself. See `internal/stacks/infra_image_retention.go`.
|
||||
- **The setup gate (v0.280.0, decision 46)** — `.felhom.yml` `setup_gate: true` + optional `setup_done_probe: {url, field,
|
||||
done}`. A FRESH install is closed to everyone but the household: the traefik file
|
||||
`<stacks>/traefik/dynamic/setup-gate-<app>.yml` is written BEFORE the first start (a failed write refuses the install) and
|
||||
@@ -3639,6 +3662,32 @@ Anything else is answered `refused` and calls nothing. Each id runs once per pro
|
||||
(`operator_action_results: [{id, outcome, message}]`, outcome `done` / `refused` / `failed`). No action deletes data,
|
||||
starts a countdown or shortens one (`TestOpActions_ClosedList`). The box's log records every action it receives.
|
||||
|
||||
- **`clear_hold` (2026-10-29, operator ruling D64 = B; arg = the app's stack name, letters/digits/`-`/`_`, ≤64):** lifts
|
||||
the app's hold of ANY reason, in the RUNNING controller (`backup.AppStopGuard.ClearHoldByOperator`, the door
|
||||
`clearHoldNow` in `cmd/controller/holdclear.go`): memory and `settings.json` together, so no later save writes it back
|
||||
(R-1343). It starts nothing — the household's Start (or a restore) does. A waiting put-back (`putback_pending`) is
|
||||
dropped from the app-stop marker too; its copy stays on disk. The update's in-memory „held" flag
|
||||
(`stacks.Manager.ForgetUpdateHeld`) goes with it; the hold reporter's next tick (≤1 min) sends `app_hold_cleared`.
|
||||
Refused, nothing changed: a bad name, or an app with no hold. No new event type: the hub records the result.
|
||||
**R-1350 (2026-10-29):** for an app that is not up, the clear records the stop as WANTED (desired state „stopped",
|
||||
`stacks.Manager.RecordStopWantedIfDown`) once the hold is found and BEFORE it is lifted
|
||||
(`AppStopGuard.ClearHoldByOperatorThen`), so the dead-app scan sends no „Telepített alkalmazás nem fut" for it and the
|
||||
boot reconciler does not start it; the household's Start records „running" and ends it.
|
||||
- **The busy file for the host agent (D65 = A, 2026-10-29; `internal/busymark`).** While a backup, a restore, an
|
||||
off-site run, the night catch-up, an app update or an app export/import runs, the controller rewrites
|
||||
`<data dir>/ctl/busy` every 15 s (line 1 the UTC time, line 2 what runs, 0600) and removes it when idle. The agent reads
|
||||
it before the operator's „Reboot guest" and refuses while it is fresh (≤ 90 s); a controller that stopped writing it
|
||||
does not block the reboot.
|
||||
- **„Felhom support restarted your box" (D65 = A).** At start the controller asks the agent `GET /guest/operator-reboot`
|
||||
(retrying for up to 10 minutes; a 404 from an older agent ends it) and the dashboard shows one neutral line,
|
||||
„A Felhom ügyfélszolgálata HH:MM-kor újraindította a dobozt." / „Felhom support restarted your box at HH:MM.", until a
|
||||
day has passed since the reboot.
|
||||
- **The box command `--clear-restore-hold <app>`** (`pct exec <vmid> -- docker exec felhom-controller felhom-controller
|
||||
--clear-restore-hold <app>`) asks the running controller over a unix socket, `<data dir>/ctl/hold.sock` (0600 in a
|
||||
0700 dir, peer uid checked, nothing on TCP — not reachable through traefik, the LAN or an app), and says no restart is
|
||||
needed. Only when no controller answers does it edit `settings.json` itself, and it says so, naming the in-guest
|
||||
restart. `--restore-holds` lists the holds (read-only).
|
||||
|
||||
#### Report Push (`internal/report/`)
|
||||
|
||||
Periodic JSON push (default every 15 min) to the central felhom-hub service:
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/busymark"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
|
||||
)
|
||||
|
||||
// D65 = A: the agent reads ONE fixed guest path — the controller-data volume + data/ctl/busy
|
||||
// (felhom-agent ControllerBusyMarkerGuestPath, and its sudoers line). The volume is mounted at
|
||||
// /opt/docker/felhom-controller, so the controller must write <default data dir>/ctl/busy.
|
||||
// RED-PROOF: rename holdCtlDir or busymark.FileName → the path moves → FAILS.
|
||||
func TestBusyMarkPathIsTheOneTheAgentReads(t *testing.T) {
|
||||
cfg, err := config.LoadFromBytes([]byte("customer:\n id: t\n domain: t.example\n"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := filepath.Join(cfg.Paths.DataDir, holdCtlDir, busymark.FileName)
|
||||
want := "/opt/docker/felhom-controller/data/ctl/busy" // = the agent's /var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy
|
||||
if got != want {
|
||||
t.Fatalf("the busy file is %s, the agent reads %s — an operator reboot would never be refused for a backup", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// Every kind of work has its probe and the writer is started — a seam built and never wired is this project's commonest
|
||||
// defect. RED-PROOF: delete any AddProbe line or the Run → FAILS.
|
||||
func TestBusyMarkIsWiredAtStartup(t *testing.T) {
|
||||
raw, err := os.ReadFile("main.go")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
src := string(raw)
|
||||
for _, want := range []string{
|
||||
`busyMark.AddProbe("an app update is running", stackMgr.AnyUpdating)`,
|
||||
`return backupMgr.IsRunning() || backupMgr.RestoreStatus().Running`,
|
||||
`busyMark.AddProbe("the night backup catch-up is running", catchUp.Running)`,
|
||||
`busyMark.AddProbe("an app export or import is running", appExporter.IsRunning)`,
|
||||
`go busyMark.Run(ctx)`,
|
||||
`go webServer.WatchOperatorReboot(ctx)`, // the household line after an operator reboot
|
||||
} {
|
||||
if !strings.Contains(src, want) {
|
||||
t.Fatalf("main.go does not contain %q", want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
//go:build faultinject
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// D63 (test build only): "<skip> <fail>" lets <skip> passes through, fails the next <fail>, then removes the file.
|
||||
func TestD63_TheTestBuildFailsTheArmedPointOnDemand(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
t.Setenv("FELHOM_FAULT_DIR", dir)
|
||||
p := filepath.Join(dir, "stop-radicale")
|
||||
if err := os.WriteFile(p, []byte("1 1\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := faultPoint("stop-radicale"); err != nil {
|
||||
t.Fatalf("the first pass must go through: %v", err)
|
||||
}
|
||||
if err := faultPoint("stop-other"); err != nil {
|
||||
t.Fatalf("another point is not armed: %v", err)
|
||||
}
|
||||
if err := faultPoint("stop-radicale"); err == nil {
|
||||
t.Fatal("the second pass must fail")
|
||||
}
|
||||
if _, err := os.Stat(p); !os.IsNotExist(err) {
|
||||
t.Fatalf("the spent file must be gone: %v", err)
|
||||
}
|
||||
if err := faultPoint("stop-radicale"); err != nil {
|
||||
t.Fatalf("a spent point fails nothing: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
//go:build !faultinject
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// goBuildTag is a `go build -tags …` flag (build.sh's own `--print-tags` option is not one).
|
||||
var goBuildTag = regexp.MustCompile(`\s-tags[ =]`)
|
||||
|
||||
// D63: the fault switch exists only in a `-tags faultinject` test build. In this (the release) build an armed fault
|
||||
// file injects nothing, and neither build.sh nor the Dockerfile passes a build tag.
|
||||
// RED-PROOF: build the release build with the switch (drop the !faultinject line from faultpoint_off.go and the
|
||||
// faultinject line from faultpoint_on.go) → the armed point fails → FAILS.
|
||||
func TestD63_TheReleaseBuildHasNoFaultSwitch(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
t.Setenv("FELHOM_FAULT_DIR", dir)
|
||||
for _, p := range []string{"defwrite-after-compose", "stop-radicale"} {
|
||||
if err := os.WriteFile(filepath.Join(dir, p), []byte("0 5\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := faultPoint(p); err != nil {
|
||||
t.Fatalf("the release build injected a fault at %s: %v", p, err)
|
||||
}
|
||||
}
|
||||
for _, f := range []string{"../../build.sh", "../../Dockerfile"} {
|
||||
b, err := os.ReadFile(f)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if goBuildTag.MatchString(string(b)) || strings.Contains(string(b), "faultinject") {
|
||||
t.Fatalf("%s passes a build tag — the release image could carry the fault switch", f)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
//go:build !faultinject
|
||||
|
||||
package main
|
||||
|
||||
// faultPoint is the D63 live-proof fault switch (operator ruling 2026-10-29: prove R-1256 H4 and H5's failed stop live,
|
||||
// with a fault switch in a TEST build only). In every build without the `faultinject` tag — the release image among
|
||||
// them: build.sh and the Dockerfile pass no build tags — it does nothing and reads nothing. Pinned by
|
||||
// TestD63_TheReleaseBuildHasNoFaultSwitch; the switch itself is faultpoint_on.go.
|
||||
func faultPoint(string) error { return nil }
|
||||
@@ -0,0 +1,64 @@
|
||||
//go:build faultinject
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// faultPoint (TEST BUILDS ONLY — `go build -tags faultinject`; never the release image, faultpoint_off.go) fails a named
|
||||
// point on demand, for the D63 live proofs of R-1256 H4 (a definition write that fails after the compose was written)
|
||||
// and H5 (a put-back whose stop fails). Armed by a file <dir>/<point> holding "<skip> <fail>": the next <skip> passes go
|
||||
// through, the <fail> after them fail, and the file is removed when both are spent. <dir> is $FELHOM_FAULT_DIR, else
|
||||
// /opt/docker/felhom-controller/faults (the controller's data volume). A file that does not parse fails nothing.
|
||||
var faultMu sync.Mutex
|
||||
|
||||
func faultPoint(point string) error {
|
||||
faultMu.Lock()
|
||||
defer faultMu.Unlock()
|
||||
dir := os.Getenv("FELHOM_FAULT_DIR")
|
||||
if dir == "" {
|
||||
dir = "/opt/docker/felhom-controller/faults"
|
||||
}
|
||||
p := filepath.Join(dir, point)
|
||||
data, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
f := strings.Fields(string(data))
|
||||
if len(f) != 2 {
|
||||
log.Printf("[WARN] [faultinject] %s: %q does not parse as \"<skip> <fail>\" — nothing injected", p, strings.TrimSpace(string(data)))
|
||||
return nil
|
||||
}
|
||||
skip, e1 := strconv.Atoi(f[0])
|
||||
fail, e2 := strconv.Atoi(f[1])
|
||||
if e1 != nil || e2 != nil || skip < 0 || fail < 0 {
|
||||
return nil
|
||||
}
|
||||
write := func() {
|
||||
if skip == 0 && fail == 0 {
|
||||
_ = os.Remove(p)
|
||||
return
|
||||
}
|
||||
_ = os.WriteFile(p, []byte(fmt.Sprintf("%d %d\n", skip, fail)), 0o600)
|
||||
}
|
||||
if skip > 0 {
|
||||
skip--
|
||||
write()
|
||||
log.Printf("[WARN] [faultinject] %s: passed (skip left %d, fail left %d)", point, skip, fail)
|
||||
return nil
|
||||
}
|
||||
if fail > 0 {
|
||||
fail--
|
||||
write()
|
||||
log.Printf("[ERROR] [faultinject] %s: FAULT INJECTED (fail left %d)", point, fail)
|
||||
return fmt.Errorf("FAULT INJECTED at %s (D63 test build)", point)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,282 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/report"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// ── D64 = B + R-1343: the operator clears a hold in the RUNNING controller ─────────────────────────────────────────
|
||||
//
|
||||
// Two routes reach the same door (clearHoldNow):
|
||||
// - the hub's „Clear hold" → the operator action clear_hold (internal/report/opactions.go), run in this process;
|
||||
// - the box command `felhom-controller --clear-restore-hold <app>` (run with `docker exec` inside the guest) → the
|
||||
// local socket below → the running controller.
|
||||
// The running controller clears the hold in memory AND saves settings.json at once (backup.ClearHoldByOperator), so
|
||||
// no later save — the hold reporter's, a hub report's — can write it back (R-1343). The household's „held" sentence
|
||||
// goes with it: every reader (the page, the start gate, the API, the hold reporter) reads the same settings record.
|
||||
// The ONE held flag kept elsewhere — stacks.Manager's „the last update ended held" — is forgotten here too.
|
||||
|
||||
// holdClearDeps are clearHoldNow's seams (production: holdClearDepsFor).
|
||||
type holdClearDeps struct {
|
||||
clear func(stack string, beforeLift func()) (settings.RestoreHold, error) // backup.AppStopGuard.ClearHoldByOperatorThen
|
||||
forgetUpdateHeld func(stack string) bool // stacks.Manager.ForgetUpdateHeld (nil = none)
|
||||
// keepStopped (R-1350) records that the stop is wanted for an app that is not up — run once the hold is found and
|
||||
// BEFORE it is lifted: stacks.Manager.RecordStopWantedIfDown (nil = none).
|
||||
keepStopped func(stack string) (bool, error)
|
||||
logger *log.Logger
|
||||
}
|
||||
|
||||
// clearHoldNow lifts stack's hold and returns the result sentence (English, operator-facing). who names the person
|
||||
// and the route for the box's own log, e.g. „the operator via the hub, action #12". A refusal (bad name, no hold) is
|
||||
// wrapped in report.ErrActionRefused, and nothing was changed.
|
||||
func clearHoldNow(d holdClearDeps, stack, who string) (string, error) {
|
||||
if d.clear == nil {
|
||||
return "", fmt.Errorf("%w: clearing a hold is not available on this box", report.ErrActionRefused)
|
||||
}
|
||||
var before func()
|
||||
if d.keepStopped != nil {
|
||||
before = func() {
|
||||
rec, kerr := d.keepStopped(stack)
|
||||
switch {
|
||||
case kerr != nil && d.logger != nil:
|
||||
d.logger.Printf("[WARN] [hold] %s: the stop could not be recorded as wanted (%v) — the dead-app alarm may follow the clear (R-1350)", stack, kerr)
|
||||
case rec && d.logger != nil:
|
||||
d.logger.Printf("[INFO] [hold] %s: the stop is recorded as wanted (desired state: stopped) — no dead-app alarm; the household's Start ends it (R-1350)", stack)
|
||||
}
|
||||
}
|
||||
}
|
||||
h, err := d.clear(stack, before)
|
||||
switch {
|
||||
case errors.Is(err, backup.ErrHoldBadStack):
|
||||
return "", fmt.Errorf("%w: %q is not a valid app name — nothing was changed", report.ErrActionRefused, stack)
|
||||
case errors.Is(err, backup.ErrNotHeld):
|
||||
return "", fmt.Errorf("%w: %s has no hold — nothing was changed", report.ErrActionRefused, stack)
|
||||
case err != nil:
|
||||
return "", fmt.Errorf("clearing the hold on %s failed: %v", stack, err)
|
||||
}
|
||||
if d.forgetUpdateHeld != nil && d.forgetUpdateHeld(stack) {
|
||||
if d.logger != nil {
|
||||
d.logger.Printf("[INFO] [hold] %s: the update's held state is forgotten with the hold", stack)
|
||||
}
|
||||
}
|
||||
if d.logger != nil {
|
||||
d.logger.Printf("[WARN] [hold] %s: hold CLEARED by %s — it was %s (set %s); the app is not started by this", stack, who, h.WireReason(), h.At)
|
||||
}
|
||||
msg := fmt.Sprintf("the hold on %s is cleared (it was: %s). The app is NOT started by this — the household's Start, or a restore, starts it", stack, h.Why())
|
||||
if h.Reason == settings.HoldReasonPutBackPending {
|
||||
dir := h.SafetyDump
|
||||
if dir == "" {
|
||||
dir = "(not named in the hold)"
|
||||
}
|
||||
msg += ". Its waiting put-back is dropped: the pre-restore copy stays on disk at " + dir + " and is never poured back by itself"
|
||||
}
|
||||
return msg, nil
|
||||
}
|
||||
|
||||
// ── the local channel (R-1343) ────────────────────────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// WHY A UNIX SOCKET IN THE CONTROLLER'S DATA DIR, mode 0600 in a 0700 directory, and nothing on TCP:
|
||||
// - traefik, the LAN and every app reach the controller only over TCP (:8080, :8081). A unix socket has no port,
|
||||
// so none of them can reach it, whatever a proxy header or a forwarded request says;
|
||||
// - the data dir is the controller's own volume: no app container mounts it, and only root in the guest
|
||||
// (`pct exec <vmid> -- docker exec felhom-controller …`, the operator gate this project already uses) can enter
|
||||
// the controller's container to use it;
|
||||
// - the directory is created 0700 BEFORE the socket is bound, so the moment between bind and chmod is not a window;
|
||||
// - the peer's uid must equal this process's (SO_PEERCRED, holdctl_linux.go) — a second lock on the same door.
|
||||
// The customer's HTTP surface is deliberately NOT used: it authenticates as the household, and a household clearing
|
||||
// its own hold is what the hold exists to prevent. Pinned by TestHoldCtl_NotReachableFromOutside.
|
||||
|
||||
const (
|
||||
holdCtlDir = "ctl"
|
||||
holdCtlSock = "hold.sock"
|
||||
// holdCtlOpClear is the one request this channel answers.
|
||||
holdCtlOpClear = "clear_hold"
|
||||
// holdCtlMaxRequest bounds what one request may send.
|
||||
holdCtlMaxRequest = 4096
|
||||
)
|
||||
|
||||
// holdCtlPath is the socket's path under the controller's data dir.
|
||||
func holdCtlPath(dataDir string) string { return filepath.Join(dataDir, holdCtlDir, holdCtlSock) }
|
||||
|
||||
type holdCtlRequest struct {
|
||||
Op string `json:"op"`
|
||||
Stack string `json:"stack"`
|
||||
}
|
||||
|
||||
type holdCtlResponse struct {
|
||||
Outcome string `json:"outcome"` // done | refused | failed (report.Outcome*)
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
// listenHoldCtl binds the socket and serves it until the listener is closed (closing it removes the socket file).
|
||||
// clear is clearHoldNow bound to the running controller's deps.
|
||||
func listenHoldCtl(path string, clear func(stack, who string) (string, error), logger *log.Logger) (net.Listener, error) {
|
||||
dir := filepath.Dir(path)
|
||||
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := os.Chmod(dir, 0o700); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// A socket left by a controller that died is replaced; anything else at that path is refused, never removed.
|
||||
if fi, err := os.Lstat(path); err == nil {
|
||||
if fi.Mode()&os.ModeSocket == 0 {
|
||||
return nil, fmt.Errorf("%s exists and is not a socket — refusing to replace it", path)
|
||||
}
|
||||
if err := os.Remove(path); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
ln, err := net.Listen("unix", path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := os.Chmod(path, 0o600); err != nil {
|
||||
ln.Close()
|
||||
return nil, err
|
||||
}
|
||||
go func() {
|
||||
for {
|
||||
conn, err := ln.Accept()
|
||||
if err != nil {
|
||||
if !errors.Is(err, net.ErrClosed) && logger != nil {
|
||||
logger.Printf("[WARN] [hold] local channel stopped: %v", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
go serveHoldCtl(conn, clear, logger)
|
||||
}
|
||||
}()
|
||||
return ln, nil
|
||||
}
|
||||
|
||||
func serveHoldCtl(conn net.Conn, clear func(stack, who string) (string, error), logger *log.Logger) {
|
||||
defer conn.Close()
|
||||
_ = conn.SetDeadline(time.Now().Add(30 * time.Second))
|
||||
answer := func(r holdCtlResponse) { _ = json.NewEncoder(conn).Encode(r) }
|
||||
if ok, why := holdCtlPeerAllowed(conn); !ok {
|
||||
if logger != nil {
|
||||
logger.Printf("[WARN] [hold] local channel: a request was refused — %s", why)
|
||||
}
|
||||
answer(holdCtlResponse{Outcome: report.OutcomeRefused, Message: "refused: " + why})
|
||||
return
|
||||
}
|
||||
line, err := bufio.NewReader(io.LimitReader(conn, holdCtlMaxRequest)).ReadBytes('\n')
|
||||
if err != nil && len(line) == 0 {
|
||||
answer(holdCtlResponse{Outcome: report.OutcomeRefused, Message: "no request was read"})
|
||||
return
|
||||
}
|
||||
var req holdCtlRequest
|
||||
if err := json.Unmarshal(line, &req); err != nil || req.Op != holdCtlOpClear {
|
||||
answer(holdCtlResponse{Outcome: report.OutcomeRefused, Message: "unknown request — this channel only clears a hold"})
|
||||
return
|
||||
}
|
||||
msg, err := clear(req.Stack, "the operator on the box (--clear-restore-hold)")
|
||||
switch {
|
||||
case err == nil:
|
||||
answer(holdCtlResponse{Outcome: report.OutcomeDone, Message: msg})
|
||||
case errors.Is(err, report.ErrActionRefused):
|
||||
answer(holdCtlResponse{Outcome: report.OutcomeRefused, Message: strings.TrimPrefix(err.Error(), report.ErrActionRefused.Error()+": ")})
|
||||
default:
|
||||
answer(holdCtlResponse{Outcome: report.OutcomeFailed, Message: err.Error()})
|
||||
}
|
||||
}
|
||||
|
||||
// askRunningControllerToClear sends one clear request. reached=false: no controller answered at path (no socket, or
|
||||
// a socket nobody listens on) — nothing was asked, so the caller may edit settings.json itself. reached=true with an
|
||||
// error: the request may have been applied; the caller must not edit the file.
|
||||
func askRunningControllerToClear(path, stack string) (resp holdCtlResponse, reached bool, err error) {
|
||||
conn, err := net.DialTimeout("unix", path, 3*time.Second)
|
||||
if err != nil {
|
||||
return holdCtlResponse{}, false, err
|
||||
}
|
||||
defer conn.Close()
|
||||
_ = conn.SetDeadline(time.Now().Add(60 * time.Second))
|
||||
if err := json.NewEncoder(conn).Encode(holdCtlRequest{Op: holdCtlOpClear, Stack: stack}); err != nil {
|
||||
return holdCtlResponse{}, true, err
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(conn, 64*1024)).Decode(&resp); err != nil {
|
||||
return holdCtlResponse{}, true, err
|
||||
}
|
||||
return resp, true, nil
|
||||
}
|
||||
|
||||
// clearRestoreHoldCLI is the box command `--clear-restore-hold <app>` (R-379, R-1343). It asks the running
|
||||
// controller first; only when none answers does it edit settings.json itself, and it says which one happened.
|
||||
// Returns the process exit code: 0 cleared, 2 refused (nothing changed), 1 failed or unknown.
|
||||
func clearRestoreHoldCLI(dataDir, stack string, stdout, stderr io.Writer) int {
|
||||
path := holdCtlPath(dataDir)
|
||||
resp, reached, err := askRunningControllerToClear(path, stack)
|
||||
if reached {
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "the running controller was reached, but its answer was lost: %v\n", err)
|
||||
fmt.Fprintf(stderr, " The hold may or may not be cleared. Check with: felhom-controller --restore-holds\n")
|
||||
fmt.Fprintf(stderr, " settings.json was NOT edited by this command.\n")
|
||||
return 1
|
||||
}
|
||||
switch resp.Outcome {
|
||||
case report.OutcomeDone:
|
||||
fmt.Fprintf(stdout, "%s.\n", resp.Message)
|
||||
fmt.Fprintf(stdout, " Cleared by the RUNNING controller (memory and settings.json together) — no restart is needed.\n")
|
||||
return 0
|
||||
case report.OutcomeRefused:
|
||||
fmt.Fprintf(stderr, "%s\n", resp.Message)
|
||||
return 2
|
||||
default:
|
||||
fmt.Fprintf(stderr, "the running controller could not clear the hold: %s\n", resp.Message)
|
||||
return 1
|
||||
}
|
||||
}
|
||||
// No controller answered: it is not running (or is older than this channel). The file is the only copy.
|
||||
fmt.Fprintf(stdout, "No running controller answered on %s (%v).\n", path, err)
|
||||
fmt.Fprintf(stdout, " Editing settings.json directly instead.\n")
|
||||
if !backup.IsValidHoldStack(stack) {
|
||||
fmt.Fprintf(stderr, "%q is not a valid app name — nothing was changed\n", stack)
|
||||
return 2
|
||||
}
|
||||
lg := log.New(stderr, "", 0)
|
||||
sett, lerr := settings.Load(filepath.Join(dataDir, "settings.json"), lg)
|
||||
if lerr != nil {
|
||||
fmt.Fprintf(stderr, "restore-hold: loading settings: %v\n", lerr)
|
||||
return 1
|
||||
}
|
||||
// The same door as the running controller's (a waiting put-back is dropped from the app-stop marker too), over
|
||||
// this process's copy — the only copy while no controller runs.
|
||||
g := backup.NewAppStopGuard(filepath.Join(dataDir, "appstop-state.json"), lg)
|
||||
g.SetHolds(sett)
|
||||
h, cerr := g.ClearHoldByOperator(stack)
|
||||
switch {
|
||||
case errors.Is(cerr, backup.ErrNotHeld):
|
||||
// "there was nothing to clear" is not success — a silent 0 here would let an operator believe they had
|
||||
// unblocked an app whose name they mistyped.
|
||||
fmt.Fprintf(stderr, "no restore hold in force for %q — nothing was changed\n", stack)
|
||||
return 2
|
||||
case cerr != nil:
|
||||
fmt.Fprintf(stderr, "restore-hold: clearing %s: %v\n", stack, cerr)
|
||||
return 1
|
||||
}
|
||||
fmt.Fprintf(stdout, "restore hold cleared for %s in settings.json (no controller was running to ask; it was: %s).\n", stack, h.Why())
|
||||
if h.Reason == settings.HoldReasonPutBackPending {
|
||||
fmt.Fprintf(stdout, " Its waiting put-back is dropped; the pre-restore copy stays on disk at %s.\n", orNone(h.SafetyDump))
|
||||
}
|
||||
fmt.Fprintf(stdout, " The controller reads it when it starts. If a controller IS running in this guest after all, restart it NOW,\n")
|
||||
fmt.Fprintf(stdout, " INSIDE THE GUEST — from the Proxmox host: pct exec <vmid> -- systemctl restart felhom-controller-bootstrap.service\n")
|
||||
fmt.Fprintf(stdout, " — or its next settings save writes the hold back (R-1343).\n")
|
||||
return 0
|
||||
}
|
||||
|
||||
// holdClearCommandHint is the exact guest command, for the help text and the operator's mail.
|
||||
const holdClearCommandHint = "pct exec <vmid> -- docker exec felhom-controller felhom-controller --clear-restore-hold <app>"
|
||||
@@ -0,0 +1,559 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"io"
|
||||
"log"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/holdreport"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/report"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// D64 = B (operator ruling 2026-10-29 12:10) + R-1343: the operator clears a hold — from the hub (clear_hold) or with
|
||||
// the box command — IN THE RUNNING CONTROLLER, so the hold is gone from memory and from settings.json together and no
|
||||
// later save brings it back.
|
||||
//
|
||||
// RED-PROOFS (2026-10-29):
|
||||
// - TestR1343_TheBoxCommandClearsThroughTheRunningController: with clearRestoreHoldCLI's socket call skipped (the
|
||||
// old second-process edit of settings.json), it fails on „the hold came back after a later save of the running
|
||||
// controller" — the R-1343 measurement, in a test;
|
||||
// - TestD64_ClearPutBackPendingDropsTheMarkerEntry: with the dropFromMarker call removed from
|
||||
// ClearHoldByOperator, it fails on „a later controller start poured the old copy back";
|
||||
// - TestD64_ClearHoldViaTheHub: with the forgetUpdateHeld call removed from clearHoldNow, it fails on „the
|
||||
// update's held flag was not forgotten".
|
||||
// - TestD64_ClearHoldViaTheHub + TestD64_ClearHoldStaysGoneAfterALaterSave: with ClearHoldByOperator clearing a
|
||||
// SECOND Settings loaded from disk (the old CLI's shape) instead of the running one, they fail on „the running
|
||||
// controller still holds the app" / „the hold came back after a later save of the running controller".
|
||||
|
||||
// d64Rig is a running controller's hold side: its Settings, the app-stop guard wired to them, and a log.
|
||||
type d64Rig struct {
|
||||
dir string
|
||||
sett *settings.Settings
|
||||
guard *backup.AppStopGuard
|
||||
logBuf *syncBuffer
|
||||
logger *log.Logger
|
||||
forgot []string
|
||||
holdDeps holdClearDeps
|
||||
}
|
||||
|
||||
type syncBuffer struct {
|
||||
mu sync.Mutex
|
||||
b bytes.Buffer
|
||||
}
|
||||
|
||||
func (s *syncBuffer) Write(p []byte) (int, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.b.Write(p)
|
||||
}
|
||||
func (s *syncBuffer) String() string { s.mu.Lock(); defer s.mu.Unlock(); return s.b.String() }
|
||||
|
||||
func newD64Rig(t *testing.T) *d64Rig {
|
||||
t.Helper()
|
||||
r := &d64Rig{dir: t.TempDir(), logBuf: &syncBuffer{}}
|
||||
r.logger = log.New(r.logBuf, "", 0)
|
||||
sett, err := settings.Load(filepath.Join(r.dir, "settings.json"), r.logger)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.sett = sett
|
||||
r.guard = backup.NewAppStopGuard(filepath.Join(r.dir, "appstop-state.json"), r.logger)
|
||||
r.guard.SetHolds(sett)
|
||||
r.holdDeps = holdClearDeps{
|
||||
clear: r.guard.ClearHoldByOperatorThen,
|
||||
forgetUpdateHeld: func(s string) bool { r.forgot = append(r.forgot, s); return true },
|
||||
logger: r.logger,
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
func (r *d64Rig) hold(t *testing.T, h settings.RestoreHold) {
|
||||
t.Helper()
|
||||
if h.At == "" {
|
||||
h.At = "2026-10-29T08:18:00Z"
|
||||
}
|
||||
if err := r.sett.SetRestoreHold(h); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func (r *d64Rig) onDisk(t *testing.T, stack string) bool {
|
||||
t.Helper()
|
||||
s, err := settings.Load(filepath.Join(r.dir, "settings.json"), log.New(io.Discard, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, held := s.GetRestoreHold(stack)
|
||||
return held
|
||||
}
|
||||
|
||||
func (r *d64Rig) actions() *report.OperatorActions {
|
||||
return report.NewOperatorActions(context.Background(), operatorActionHandlers(nil, nil, r.holdDeps, nil, r.logger), r.logger)
|
||||
}
|
||||
|
||||
func onlyResult(t *testing.T, o *report.OperatorActions, id int64) report.OperatorActionResult {
|
||||
t.Helper()
|
||||
for _, x := range o.Results() {
|
||||
if x.ID == id {
|
||||
return x
|
||||
}
|
||||
}
|
||||
t.Fatalf("no result for #%d: %+v", id, o.Results())
|
||||
return report.OperatorActionResult{}
|
||||
}
|
||||
|
||||
// The hub's clear_hold: the hold is gone (memory and disk), the household's held sentence is gone, the update's
|
||||
// held flag is forgotten, the box logs who did it, and the result names what was cleared.
|
||||
func TestD64_ClearHoldViaTheHub(t *testing.T) {
|
||||
r := newD64Rig(t)
|
||||
r.hold(t, settings.RestoreHold{Stack: "paperless", ReplayError: "ERROR: relation exists", RollbackErr: "psql: refused"})
|
||||
cfg := &config.Config{}
|
||||
cfg.Paths.DataDir = r.dir
|
||||
mgr := backup.NewManager(cfg, r.sett, r.logger)
|
||||
if held, why := mgr.RestoreHoldForLang("paperless", "hu"); !held || why == "" {
|
||||
t.Fatal("fixture: the household must read a held sentence before the clear")
|
||||
}
|
||||
|
||||
o := r.actions()
|
||||
o.Reconcile([]report.OperatorAction{{ID: 12, Action: report.OpClearHold, Arg: "paperless"}})
|
||||
res := onlyResult(t, o, 12)
|
||||
if res.Outcome != report.OutcomeDone {
|
||||
t.Fatalf("result %+v", res)
|
||||
}
|
||||
for _, want := range []string{"the hold on paperless is cleared", "held after a failed restore whose rollback also failed", "NOT started by this"} {
|
||||
if !strings.Contains(res.Message, want) {
|
||||
t.Errorf("result message %q lacks %q", res.Message, want)
|
||||
}
|
||||
}
|
||||
if _, held := r.sett.GetRestoreHold("paperless"); held {
|
||||
t.Fatal("the running controller still holds the app")
|
||||
}
|
||||
if r.onDisk(t, "paperless") {
|
||||
t.Fatal("settings.json still holds the app")
|
||||
}
|
||||
if held, why := mgr.RestoreHoldForLang("paperless", "hu"); held || why != "" {
|
||||
t.Fatalf("the household still reads a held sentence: %q", why)
|
||||
}
|
||||
if len(r.forgot) != 1 || r.forgot[0] != "paperless" {
|
||||
t.Fatalf("the update's held flag was not forgotten: %v", r.forgot)
|
||||
}
|
||||
if lg := r.logBuf.String(); !strings.Contains(lg, "paperless: hold CLEARED by the operator via the hub, action #12") {
|
||||
t.Fatalf("the box's log must say who cleared it:\n%s", lg)
|
||||
}
|
||||
}
|
||||
|
||||
// Refusals change nothing: an app nobody ever held, a second press on an app already cleared, a bad name.
|
||||
func TestD64_ClearHoldRefusals(t *testing.T) {
|
||||
r := newD64Rig(t)
|
||||
r.hold(t, settings.RestoreHold{Stack: "kimai", Reason: settings.HoldReasonUpdateFailed})
|
||||
o := r.actions()
|
||||
o.Reconcile([]report.OperatorAction{{ID: 1, Action: report.OpClearHold, Arg: "nosuchapp"}})
|
||||
if res := onlyResult(t, o, 1); res.Outcome != report.OutcomeRefused || !strings.Contains(res.Message, "nosuchapp has no hold") {
|
||||
t.Fatalf("unknown app: %+v", res)
|
||||
}
|
||||
if _, held := r.sett.GetRestoreHold("kimai"); !held || !r.onDisk(t, "kimai") {
|
||||
t.Fatal("a refusal changed another app's hold")
|
||||
}
|
||||
if len(r.forgot) != 0 {
|
||||
t.Fatalf("a refusal touched the update state: %v", r.forgot)
|
||||
}
|
||||
o.Reconcile([]report.OperatorAction{{ID: 2, Action: report.OpClearHold, Arg: "kimai"}})
|
||||
o.Reconcile([]report.OperatorAction{{ID: 3, Action: report.OpClearHold, Arg: "kimai"}})
|
||||
if res := onlyResult(t, o, 3); res.Outcome != report.OutcomeRefused || !strings.Contains(res.Message, "kimai has no hold") {
|
||||
t.Fatalf("not held (second press): %+v", res)
|
||||
}
|
||||
if _, err := clearHoldNow(r.holdDeps, "../kimai", "test"); err == nil || !strings.Contains(err.Error(), "not a valid app name") {
|
||||
t.Fatalf("a bad name must be refused by the door itself too: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// R-1343's order: clear, then ANOTHER component saves settings (the hold reporter, a hub report), then the file is
|
||||
// read back — the hold must stay gone, in memory and on disk.
|
||||
func TestD64_ClearHoldStaysGoneAfterALaterSave(t *testing.T) {
|
||||
r := newD64Rig(t)
|
||||
r.hold(t, settings.RestoreHold{Stack: "radicale", Reason: settings.HoldReasonRestoreMixed})
|
||||
o := r.actions()
|
||||
o.Reconcile([]report.OperatorAction{{ID: 5, Action: report.OpClearHold, Arg: "radicale"}})
|
||||
if res := onlyResult(t, o, 5); res.Outcome != report.OutcomeDone {
|
||||
t.Fatalf("result %+v", res)
|
||||
}
|
||||
if err := r.sett.SetReportedHolds(map[string]string{"other": "x|y"}); err != nil { // another component's save
|
||||
t.Fatal(err)
|
||||
}
|
||||
if r.onDisk(t, "radicale") {
|
||||
t.Fatal("the hold came back after a later save of the running controller")
|
||||
}
|
||||
if _, held := r.sett.GetRestoreHold("radicale"); held {
|
||||
t.Fatal("the running controller holds the app again")
|
||||
}
|
||||
}
|
||||
|
||||
// The same order through the BOX COMMAND: it reaches the running controller over the socket; no restart.
|
||||
func TestR1343_TheBoxCommandClearsThroughTheRunningController(t *testing.T) {
|
||||
r := newD64Rig(t)
|
||||
r.hold(t, settings.RestoreHold{Stack: "radicale", Reason: settings.HoldReasonRestoreMixed})
|
||||
ln, err := listenHoldCtl(holdCtlPath(r.dir), func(stack, who string) (string, error) { return clearHoldNow(r.holdDeps, stack, who) }, r.logger)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer ln.Close()
|
||||
|
||||
var out, errb bytes.Buffer
|
||||
if code := clearRestoreHoldCLI(r.dir, "radicale", &out, &errb); code != 0 {
|
||||
t.Fatalf("exit %d, stdout %q, stderr %q", code, out.String(), errb.String())
|
||||
}
|
||||
if err := r.sett.SetReportedHolds(map[string]string{"other": "x|y"}); err != nil { // the running controller's next save
|
||||
t.Fatal(err)
|
||||
}
|
||||
if r.onDisk(t, "radicale") {
|
||||
t.Fatal("the hold came back after a later save of the running controller")
|
||||
}
|
||||
if !strings.Contains(out.String(), "Cleared by the RUNNING controller") || !strings.Contains(out.String(), "no restart is needed") {
|
||||
t.Fatalf("the command must say it went through the running controller and needs no restart:\n%s", out.String())
|
||||
}
|
||||
if !strings.Contains(r.logBuf.String(), "hold CLEARED by the operator on the box (--clear-restore-hold)") {
|
||||
t.Fatalf("the box's log must say who cleared it:\n%s", r.logBuf.String())
|
||||
}
|
||||
|
||||
// A second run: refused through the socket, exit 2, nothing edited.
|
||||
out.Reset()
|
||||
errb.Reset()
|
||||
if code := clearRestoreHoldCLI(r.dir, "radicale", &out, &errb); code != 2 || !strings.Contains(errb.String(), "radicale has no hold") {
|
||||
t.Fatalf("not held: exit %d, stderr %q", code, errb.String())
|
||||
}
|
||||
}
|
||||
|
||||
// No controller runs (no socket, or a socket nobody listens on): the command edits settings.json and says so plainly,
|
||||
// naming the guest restart for the case a controller runs after all.
|
||||
func TestR1343_TheBoxCommandFallsBackWhenNoControllerRuns(t *testing.T) {
|
||||
for _, stale := range []bool{false, true} {
|
||||
r := newD64Rig(t)
|
||||
r.hold(t, settings.RestoreHold{Stack: "kimai", Reason: settings.HoldReasonUpdateFailed})
|
||||
if stale { // a socket left by a controller that died
|
||||
ln, err := listenHoldCtl(holdCtlPath(r.dir), func(string, string) (string, error) { return "", nil }, r.logger)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ln.(*net.UnixListener).SetUnlinkOnClose(false)
|
||||
ln.Close()
|
||||
}
|
||||
var out, errb bytes.Buffer
|
||||
if code := clearRestoreHoldCLI(r.dir, "kimai", &out, &errb); code != 0 {
|
||||
t.Fatalf("stale=%v: exit %d, stderr %q", stale, code, errb.String())
|
||||
}
|
||||
for _, want := range []string{"No running controller answered", "Editing settings.json directly", "INSIDE THE GUEST", "pct exec <vmid> -- systemctl restart felhom-controller-bootstrap.service"} {
|
||||
if !strings.Contains(out.String(), want) {
|
||||
t.Errorf("stale=%v: output lacks %q:\n%s", stale, want, out.String())
|
||||
}
|
||||
}
|
||||
if r.onDisk(t, "kimai") {
|
||||
t.Fatalf("stale=%v: settings.json still holds the app", stale)
|
||||
}
|
||||
out.Reset()
|
||||
errb.Reset()
|
||||
if code := clearRestoreHoldCLI(r.dir, "kimai", &out, &errb); code != 2 {
|
||||
t.Fatalf("stale=%v: not held must exit 2, got %d", stale, code)
|
||||
}
|
||||
if code := clearRestoreHoldCLI(r.dir, "../x", &out, &errb); code != 2 {
|
||||
t.Fatalf("stale=%v: a bad name must exit 2, got %d", stale, code)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A put-back WAITING for its drive is also in the app-stop marker: clearing the hold must drop it there too, or the
|
||||
// next controller start with the drive bound would pour the old copy over what the app wrote after the clear. The
|
||||
// copy itself stays on disk (nothing is deleted).
|
||||
func TestD64_ClearPutBackPendingDropsTheMarkerEntry(t *testing.T) {
|
||||
r := newD64Rig(t)
|
||||
undo := filepath.Join(r.dir, "drive", "felhom-data", "backups", "restore-undo", "immich", "20261029T080000Z")
|
||||
if err := os.MkdirAll(undo, 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(undo, "undo.json"), []byte(`{"stack":"immich"}`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := r.guard.Begin("unit-restore:immich", backup.ReasonUnitRestore, []string{"immich"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := r.guard.SetUndo("immich", undo); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.hold(t, settings.RestoreHold{Stack: "immich", Reason: settings.HoldReasonPutBackPending, SafetyDump: undo})
|
||||
|
||||
o := r.actions()
|
||||
o.Reconcile([]report.OperatorAction{{ID: 3, Action: report.OpClearHold, Arg: "immich"}})
|
||||
res := onlyResult(t, o, 3)
|
||||
if res.Outcome != report.OutcomeDone || !strings.Contains(res.Message, "waiting put-back is dropped") || !strings.Contains(res.Message, undo) {
|
||||
t.Fatalf("result %+v", res)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(undo, "undo.json")); err != nil {
|
||||
t.Fatal("the copy must stay on disk")
|
||||
}
|
||||
// The consequence: a later controller start with the drive bound pours nothing back.
|
||||
poured := &recordingPutBacker{}
|
||||
g2 := backup.NewAppStopGuard(filepath.Join(r.dir, "appstop-state.json"), r.logger)
|
||||
g2.SetHolds(r.sett)
|
||||
g2.SetPutBacker(poured)
|
||||
g2.SetStarter(nopStarter{})
|
||||
g2.SetUndoReachable(func(string, string) bool { return true })
|
||||
g2.Recover()
|
||||
if len(poured.stacks) != 0 {
|
||||
t.Fatalf("a later controller start poured the old copy back over the app: %v", poured.stacks)
|
||||
}
|
||||
}
|
||||
|
||||
type recordingPutBacker struct{ stacks []string }
|
||||
|
||||
func (p *recordingPutBacker) PutBackInterruptedRestore(stack, _ string) error {
|
||||
p.stacks = append(p.stacks, stack)
|
||||
return nil
|
||||
}
|
||||
|
||||
type nopStarter struct{}
|
||||
|
||||
func (nopStarter) StartStack(string) error { return nil }
|
||||
|
||||
// The hold reporter's next tick tells the hub the hold is gone (it reads the same settings record; nothing cached).
|
||||
func TestD64_TheHoldReporterTellsTheHubAfterTheClear(t *testing.T) {
|
||||
r := newD64Rig(t)
|
||||
r.hold(t, settings.RestoreHold{Stack: "kimai", Reason: settings.HoldReasonUpdateFailed})
|
||||
var events []string
|
||||
rep := holdreport.New(r.sett, func(et, _, _ string, _ interface{}) { events = append(events, et) }, r.logger)
|
||||
rep.Tick(timeAt(0))
|
||||
o := r.actions()
|
||||
o.Reconcile([]report.OperatorAction{{ID: 1, Action: report.OpClearHold, Arg: "kimai"}})
|
||||
rep.Tick(timeAt(60))
|
||||
if strings.Join(events, ",") != holdreport.EventHeld+","+holdreport.EventCleared {
|
||||
t.Fatalf("events %v — want app_held then app_hold_cleared", events)
|
||||
}
|
||||
}
|
||||
|
||||
// „Not reachable from outside": the channel is a unix socket (no TCP anywhere in the file), 0600 in a 0700 directory
|
||||
// — even when the directory existed with a looser mode — and main opens it once, under the data dir.
|
||||
func TestHoldCtl_NotReachableFromOutside(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.MkdirAll(filepath.Join(dir, holdCtlDir), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ln, err := listenHoldCtl(holdCtlPath(dir), func(string, string) (string, error) { return "", nil }, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer ln.Close()
|
||||
if n := ln.Addr().Network(); n != "unix" {
|
||||
t.Fatalf("the channel listens on %q, want unix", n)
|
||||
}
|
||||
fi, err := os.Lstat(holdCtlPath(dir))
|
||||
if err != nil || fi.Mode()&os.ModeSocket == 0 || fi.Mode().Perm() != 0o600 {
|
||||
t.Fatalf("socket mode %v (err %v), want a socket with 0600", fi.Mode(), err)
|
||||
}
|
||||
di, _ := os.Stat(filepath.Join(dir, holdCtlDir))
|
||||
if di.Mode().Perm() != 0o700 {
|
||||
t.Fatalf("socket dir mode %v, want 0700", di.Mode().Perm())
|
||||
}
|
||||
|
||||
// No TCP and no HTTP in the channel's file: every net.Listen/Dial names "unix".
|
||||
f, err := parser.ParseFile(token.NewFileSet(), "holdclear.go", nil, parser.ImportsOnly)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, im := range f.Imports {
|
||||
if p, _ := strconv.Unquote(im.Path.Value); strings.HasPrefix(p, "net/http") {
|
||||
t.Fatalf("holdclear.go imports %s — the channel must not be HTTP", p)
|
||||
}
|
||||
}
|
||||
full, err := parser.ParseFile(token.NewFileSet(), "holdclear.go", nil, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
nets := 0
|
||||
ast.Inspect(full, func(n ast.Node) bool {
|
||||
call, ok := n.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
sel, ok := call.Fun.(*ast.SelectorExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
if x, ok := sel.X.(*ast.Ident); ok && x.Name == "net" && len(call.Args) > 0 {
|
||||
nets++
|
||||
if lit, ok := call.Args[0].(*ast.BasicLit); !ok || lit.Value != `"unix"` {
|
||||
t.Errorf("net.%s is called with a network other than \"unix\"", sel.Sel.Name)
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
if nets < 2 {
|
||||
t.Fatalf("expected the listen and the dial in holdclear.go, found %d net calls", nets)
|
||||
}
|
||||
|
||||
// main opens it exactly once, at holdCtlPath(cfg.Paths.DataDir).
|
||||
mf, err := parser.ParseFile(token.NewFileSet(), "main.go", nil, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
opened := 0
|
||||
ast.Inspect(mf, func(n ast.Node) bool {
|
||||
call, ok := n.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
if id, ok := call.Fun.(*ast.Ident); ok && id.Name == "listenHoldCtl" {
|
||||
opened++
|
||||
arg, ok := call.Args[0].(*ast.CallExpr)
|
||||
if fn, ok2 := arg.Fun.(*ast.Ident); !ok || !ok2 || fn.Name != "holdCtlPath" {
|
||||
t.Error("main must open the channel at holdCtlPath(<data dir>)")
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
if opened != 1 {
|
||||
t.Fatalf("main.go opens the channel %d time(s), want 1", opened)
|
||||
}
|
||||
}
|
||||
|
||||
// The socket answers only its one request; anything else is refused and calls nothing.
|
||||
func TestHoldCtl_UnknownRequestIsRefused(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
called := 0
|
||||
ln, err := listenHoldCtl(holdCtlPath(dir), func(string, string) (string, error) { called++; return "x", nil }, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer ln.Close()
|
||||
conn, err := net.Dial("unix", holdCtlPath(dir))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer conn.Close()
|
||||
if _, err := conn.Write([]byte(`{"op":"set_hold","stack":"x"}` + "\n")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var resp holdCtlResponse
|
||||
if err := json.NewDecoder(conn).Decode(&resp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if resp.Outcome != report.OutcomeRefused || called != 0 {
|
||||
t.Fatalf("response %+v, calls %d — want refused and nothing called", resp, called)
|
||||
}
|
||||
}
|
||||
|
||||
// Something at the socket's path that is NOT a socket is refused, never removed.
|
||||
func TestHoldCtl_ANonSocketFileIsNotReplaced(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
p := holdCtlPath(dir)
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(p, []byte("keep me"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := listenHoldCtl(p, func(string, string) (string, error) { return "", nil }, nil); err == nil {
|
||||
t.Fatal("a regular file at the socket path must be refused")
|
||||
}
|
||||
if b, _ := os.ReadFile(p); string(b) != "keep me" {
|
||||
t.Fatal("the file at the socket path was changed")
|
||||
}
|
||||
}
|
||||
|
||||
// main wires every piece once (the seam-built-but-never-wired trap): the clear door into the operator actions, the
|
||||
// R-1340 text into the hold notification, the R-1341 gate into the OOM loop, and the box command.
|
||||
func TestD64_MainWiresTheNewPieces(t *testing.T) {
|
||||
f, err := parser.ParseFile(token.NewFileSet(), "main.go", nil, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
found := map[string]int{}
|
||||
var opArgs []ast.Expr
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
call, ok := n.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
switch fn := call.Fun.(type) {
|
||||
case *ast.SelectorExpr:
|
||||
if x, ok := fn.X.(*ast.Ident); ok {
|
||||
found[x.Name+"."+fn.Sel.Name]++
|
||||
}
|
||||
case *ast.Ident:
|
||||
found[fn.Name]++
|
||||
if fn.Name == "operatorActionHandlers" {
|
||||
opArgs = call.Args
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
for _, want := range []string{"heldStoppedOperatorText", "heldHoldFromNotify", "oomLog.fresh", "clearRestoreHoldCLI", "listenHoldCtl", "clearHoldNow"} {
|
||||
if found[want] != 1 {
|
||||
t.Errorf("main.go calls %s %d time(s), want exactly 1", want, found[want])
|
||||
}
|
||||
}
|
||||
if len(opArgs) < 3 {
|
||||
t.Fatal("operatorActionHandlers call not found")
|
||||
}
|
||||
if id, ok := opArgs[2].(*ast.Ident); !ok || id.Name != "holdDeps" {
|
||||
t.Error("main must hand holdDeps (the running controller's clear door) to operatorActionHandlers")
|
||||
}
|
||||
}
|
||||
|
||||
func timeAt(sec int) time.Time { return time.Date(2026, 10, 29, 12, 0, sec, 0, time.UTC) }
|
||||
|
||||
// No controller runs: the box command's own clear drops a waiting put-back from the app-stop marker too (the same
|
||||
// door), so the next start with the drive bound pours nothing back.
|
||||
// RED-PROOF (2026-10-29): with the fallback clearing through settings.ClearRestoreHold alone (its first shape in
|
||||
// 5b3b541), this fails on „a later controller start poured the old copy back".
|
||||
func TestR1343_TheFallbackDropsAWaitingPutBackToo(t *testing.T) {
|
||||
r := newD64Rig(t)
|
||||
undo := filepath.Join(r.dir, "drive", "felhom-data", "backups", "restore-undo", "immich", "20261029T080000Z")
|
||||
if err := os.MkdirAll(undo, 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := r.guard.Begin("unit-restore:immich", backup.ReasonUnitRestore, []string{"immich"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := r.guard.SetUndo("immich", undo); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.hold(t, settings.RestoreHold{Stack: "immich", Reason: settings.HoldReasonPutBackPending, SafetyDump: undo})
|
||||
var out, errb bytes.Buffer
|
||||
if code := clearRestoreHoldCLI(r.dir, "immich", &out, &errb); code != 0 {
|
||||
t.Fatalf("exit %d, stderr %q", code, errb.String())
|
||||
}
|
||||
if !strings.Contains(out.String(), "waiting put-back is dropped") {
|
||||
t.Errorf("output must say the put-back is dropped:\n%s", out.String())
|
||||
}
|
||||
sett2, err := settings.Load(filepath.Join(r.dir, "settings.json"), log.New(io.Discard, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
poured := &recordingPutBacker{}
|
||||
g2 := backup.NewAppStopGuard(filepath.Join(r.dir, "appstop-state.json"), r.logger)
|
||||
g2.SetHolds(sett2)
|
||||
g2.SetPutBacker(poured)
|
||||
g2.SetStarter(nopStarter{})
|
||||
g2.SetUndoReachable(func(string, string) bool { return true })
|
||||
g2.Recover()
|
||||
if len(poured.stacks) != 0 {
|
||||
t.Fatalf("a later controller start poured the old copy back over the app: %v", poured.stacks)
|
||||
}
|
||||
if _, err := os.Stat(undo); err != nil {
|
||||
t.Fatal("the copy must stay on disk")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
//go:build linux
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
// holdCtlPeerAllowed (R-1343) admits a peer on the hold socket only when its uid equals this process's (SO_PEERCRED).
|
||||
// The socket's 0600 mode in a 0700 directory already says so; this is the second lock on the same door.
|
||||
func holdCtlPeerAllowed(conn net.Conn) (bool, string) {
|
||||
uc, ok := conn.(*net.UnixConn)
|
||||
if !ok {
|
||||
return false, "not a unix-socket peer"
|
||||
}
|
||||
raw, err := uc.SyscallConn()
|
||||
if err != nil {
|
||||
return false, "the peer's identity could not be read"
|
||||
}
|
||||
var cred *syscall.Ucred
|
||||
var cerr error
|
||||
if err := raw.Control(func(fd uintptr) {
|
||||
cred, cerr = syscall.GetsockoptUcred(int(fd), syscall.SOL_SOCKET, syscall.SO_PEERCRED)
|
||||
}); err != nil || cerr != nil || cred == nil {
|
||||
return false, "the peer's identity could not be read"
|
||||
}
|
||||
if int(cred.Uid) != os.Geteuid() {
|
||||
return false, fmt.Sprintf("the peer's uid %d is not this controller's", cred.Uid)
|
||||
}
|
||||
return true, ""
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
//go:build !linux
|
||||
|
||||
package main
|
||||
|
||||
import "net"
|
||||
|
||||
// holdCtlPeerAllowed: the peer check needs SO_PEERCRED (Linux, where the controller runs). Elsewhere the socket's
|
||||
// 0600 mode in a 0700 directory is the only lock.
|
||||
func holdCtlPeerAllowed(conn net.Conn) (bool, string) {
|
||||
if _, ok := conn.(*net.UnixConn); !ok {
|
||||
return false, "not a unix-socket peer"
|
||||
}
|
||||
return true, ""
|
||||
}
|
||||
@@ -0,0 +1,165 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// ── R-1340: the operator's „HELD STOPPED" mail names the hold's REAL cause ────────────────────────────────────────
|
||||
//
|
||||
// Until 2026-10-29 the restore-hold mail told one story for every restore_mixed hold — R-893 option C's „its off-site
|
||||
// database restore failed. The database was rolled back …, but its files and named volumes are the snapshot's" —
|
||||
// written when option C was the only mixed hold. Measured on Tester 1 (D63's H5 proof): Radicale, which has no
|
||||
// database, whose put-back stopped at „the app could not be stopped" with nothing moved, got exactly that text.
|
||||
// Support read a wrong account of the box.
|
||||
//
|
||||
// The text is now built from the HOLD RECORD the writer persisted just before the mail (its reason and the R-1340
|
||||
// fields: which restore, which branch, which step failed, whether the failure was a database replay, where the copy
|
||||
// is kept). It never says „database" unless the restore's failure WAS a database replay, and never „off-site" for a
|
||||
// restore from the app's own backup. One sentence per reason; pinned by TestR1340_* (holdmail_test.go).
|
||||
|
||||
// heldStoppedOperatorText is the operator mail for a held app: the message, its leg (the hub reads the leg, so the
|
||||
// two existing ones are kept: restore-hold for a restore hold, restore-hold-mixed for a mixed one) and a short reason.
|
||||
func heldStoppedOperatorText(h settings.RestoreHold) (msg, leg, reason string) {
|
||||
stack := h.Stack
|
||||
const tail = " The app will not start from any path until the hold is cleared (the hub's Clear hold button, or on the box: " + holdClearCommandHint + ")."
|
||||
kind := restoreKindPhrase(h.RestoreKind)
|
||||
switch h.Reason {
|
||||
case settings.HoldReasonUnhealthyStop:
|
||||
what := "a crash loop"
|
||||
if h.UnhealthyKind == "oom_storm" {
|
||||
what = "an out-of-memory storm"
|
||||
}
|
||||
trip := ""
|
||||
if h.Trip >= 2 {
|
||||
trip = fmt.Sprintf(" This is stop %d within 24 hours.", h.Trip)
|
||||
}
|
||||
msg = fmt.Sprintf("App %q is HELD STOPPED: the box stopped it for %s at %s.%s No restore or update was involved and no data was touched. "+
|
||||
"The household's Start gives it one more try; clearing the hold lifts it too.", stack, what, h.At, trip)
|
||||
return msg, "unhealthy-stop", "stopped for " + what
|
||||
case settings.HoldReasonUpdateFailed:
|
||||
undo := ""
|
||||
switch h.UndoState {
|
||||
case "untouched":
|
||||
undo = " The automatic undo failed; the data was not touched."
|
||||
case "half":
|
||||
undo = " The automatic undo failed half-way; the data may be partly the new version's."
|
||||
case "not_started":
|
||||
undo = " The automatic undo did not start."
|
||||
}
|
||||
copyLine := " Restoring it from its backup starts it."
|
||||
if h.NoWholeCopy {
|
||||
copyLine = " No copy on this box brings it back whole — it needs support."
|
||||
} else if h.CopyDate != "" {
|
||||
copyLine = fmt.Sprintf(" Restoring it from its backup (tier %d, %s) starts it.", h.CopyTier, h.CopyDate)
|
||||
}
|
||||
msg = fmt.Sprintf("App %q is HELD STOPPED: its update at %s did not come up healthy.%s%s", stack, h.At, undo, copyLine) + tail
|
||||
return msg, "update-hold", "update failed"
|
||||
case settings.HoldReasonPutBackPending:
|
||||
msg = fmt.Sprintf("App %q is HELD STOPPED: a restore was cut after its copy of the live data was taken, and that copy (%s) is on a drive that is not bound. "+
|
||||
"The copy is poured back and the app started when the drive is bound. Clearing the hold DROPS that put-back; the copy stays on disk.", stack, orNone(h.SafetyDump))
|
||||
return msg, "putback-pending", "put-back waits for its drive"
|
||||
case settings.HoldReasonRestoreMixed:
|
||||
return mixedHoldText(h, kind, tail)
|
||||
default: // HoldReasonRestoreFailed: a database replay failed AND its rollback failed (holdAppAfterFailedRollback)
|
||||
what := "its " + kind + " failed"
|
||||
if h.DBReplay || h.RestoreKind == "" { // every restore hold before R-1340 was written by the database branch
|
||||
what = "its " + kind + " failed at the database replay AND the rollback to the pre-restore database copy also failed — the database may be half-written"
|
||||
}
|
||||
msg = fmt.Sprintf("App %q is HELD STOPPED: %s.", stack, what) + tail +
|
||||
fmt.Sprintf(" Replay error: %s. Rollback error: %s", orNone(h.ReplayError), orNone(h.RollbackErr))
|
||||
reason = h.RollbackErr
|
||||
if reason == "" {
|
||||
reason = "rollback failed"
|
||||
}
|
||||
return msg, "restore-hold", reason
|
||||
}
|
||||
}
|
||||
|
||||
// heldRunKind names the run in the mail's subject. A hold that does not say which restore wrote it is an off-site one, the
|
||||
// pre-R-1340 rule, EXCEPT an interrupted put-back: its record names no kind, and its restore may be either, so „restore" (R-1348,
|
||||
// seen live 2026-11-03: a local restore's mail read „offsite-reconstitute"). Pinned by TestR1348_HeldRunKind.
|
||||
func heldRunKind(h settings.RestoreHold) string {
|
||||
if h.RestoreKind == settings.RestoreKindLocal || h.MixedCause == settings.MixedCauseInterruptedPutBack {
|
||||
return "restore"
|
||||
}
|
||||
return "offsite-reconstitute"
|
||||
}
|
||||
|
||||
func mixedHoldText(h settings.RestoreHold, kind, tail string) (msg, leg, reason string) {
|
||||
stack := h.Stack
|
||||
leg = "restore-hold-mixed"
|
||||
failedAt := ""
|
||||
if h.DBReplay {
|
||||
failedAt = " at the database replay"
|
||||
}
|
||||
copyLine := " There is no copy of the live volumes."
|
||||
if h.SafetyDump != "" {
|
||||
copyLine = " The copy of the live data is kept at " + h.SafetyDump + "."
|
||||
}
|
||||
switch h.MixedCause {
|
||||
case settings.MixedCauseDefinitionWrite:
|
||||
msg = fmt.Sprintf("App %q is HELD STOPPED for support: its %s failed while writing the backup's app definition, before any data was touched, "+
|
||||
"and %s. The app's data is the live data; the definition on disk may be the backup's.", stack, kind, orNone(h.FailedStep)) + tail +
|
||||
" Error: " + orNone(h.ReplayError)
|
||||
return msg, leg, "definition write failed; writing the live definition back failed too"
|
||||
case settings.MixedCausePutBack:
|
||||
msg = fmt.Sprintf("App %q is HELD STOPPED for support: its %s failed%s, and putting the live data back FAILED — %s. "+
|
||||
"The app may be on a mix of the backup's and the live data.%s", stack, kind, failedAt, orNone(h.FailedStep), copyLine) + tail +
|
||||
" Restore error: " + orNone(h.ReplayError)
|
||||
return msg, leg, "put-back failed: " + firstClause(h.FailedStep)
|
||||
case settings.MixedCauseInterruptedPutBack:
|
||||
msg = fmt.Sprintf("App %q is HELD STOPPED for support: a restore cut by a stop could not be put back at the next start — %s. "+
|
||||
"The app may be on a mix of the backup's and the live data.%s", stack, orNone(h.FailedStep), copyLine) + tail
|
||||
return msg, leg, "interrupted restore: put-back failed"
|
||||
}
|
||||
// A mixed hold written before R-1340 carries no cause: name only what the record says, and assume nothing.
|
||||
msg = fmt.Sprintf("App %q is HELD STOPPED for support: a restore stopped half-way (set %s); the app may be on a mix of the backup's and the live data. "+
|
||||
"The hold record says: %s.", stack, h.At, orNone(h.ReplayError)) + tail
|
||||
return msg, leg, "restore stopped half-way"
|
||||
}
|
||||
|
||||
func restoreKindPhrase(k string) string {
|
||||
switch k {
|
||||
case settings.RestoreKindLocal:
|
||||
return "restore from its own backup"
|
||||
case settings.RestoreKindOffsite:
|
||||
return "off-site restore"
|
||||
}
|
||||
return "restore"
|
||||
}
|
||||
|
||||
func orNone(s string) string {
|
||||
if strings.TrimSpace(s) == "" {
|
||||
return "(none recorded)"
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// firstClause is a step's text up to its error (the short reason field).
|
||||
func firstClause(s string) string {
|
||||
if i := strings.Index(s, ":"); i > 0 {
|
||||
return s[:i]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// heldHoldFromNotify is the hold the notification describes: the record the writer persisted just before it, or —
|
||||
// when that write failed — one rebuilt from the errors handed over (the pre-R-1340 rule: no rollback error = mixed).
|
||||
func heldHoldFromNotify(get func(string) (settings.RestoreHold, bool), stack string, replayErr, rollbackErr error) settings.RestoreHold {
|
||||
if get != nil {
|
||||
if h, ok := get(stack); ok && (h.Reason == settings.HoldReasonRestoreFailed || h.Reason == settings.HoldReasonRestoreMixed) {
|
||||
return h
|
||||
}
|
||||
}
|
||||
h := settings.RestoreHold{Stack: stack, Reason: settings.HoldReasonRestoreMixed}
|
||||
if rollbackErr != nil {
|
||||
h.Reason, h.RollbackErr = settings.HoldReasonRestoreFailed, rollbackErr.Error()
|
||||
}
|
||||
if replayErr != nil {
|
||||
h.ReplayError = replayErr.Error()
|
||||
}
|
||||
return h
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// R-1340: the operator's „HELD STOPPED" mail names the hold's real cause — one test per reason. Each asserts the words
|
||||
// that MUST be there and the words that must NOT (the option-C database story was the bug).
|
||||
//
|
||||
// RED-PROOF (2026-10-29): with mixedHoldText's body replaced by the pre-R-1340 sentence („its off-site database restore
|
||||
// failed. The database was rolled back to the pre-restore copy and the live definition written back, but its files
|
||||
// and named volumes are the snapshot's"), TestR1340_PutBackWithoutADatabase, _DefinitionWrite, _LocalRestore and
|
||||
// _InterruptedPutBack fail on „database" / „off-site" being present and their own step missing.
|
||||
|
||||
const optionCStory = "files and named volumes are the snapshot's"
|
||||
|
||||
func mustHave(t *testing.T, msg string, want ...string) {
|
||||
t.Helper()
|
||||
for _, w := range want {
|
||||
if !strings.Contains(msg, w) {
|
||||
t.Errorf("the mail must say %q:\n%s", w, msg)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func mustNotHave(t *testing.T, msg string, bad ...string) {
|
||||
t.Helper()
|
||||
for _, b := range bad {
|
||||
if strings.Contains(strings.ToLower(msg), strings.ToLower(b)) {
|
||||
t.Errorf("the mail must NOT say %q:\n%s", b, msg)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestR1340_UnhealthyStop(t *testing.T) {
|
||||
msg, leg, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "romm", At: "2026-10-29T08:00:00Z", Reason: settings.HoldReasonUnhealthyStop, UnhealthyKind: "crash_loop"})
|
||||
mustHave(t, msg, `"romm"`, "crash loop", "no data was touched", "Start gives it one more try")
|
||||
mustNotHave(t, msg, "database", "restore failed", optionCStory)
|
||||
if leg != "unhealthy-stop" {
|
||||
t.Errorf("leg %q", leg)
|
||||
}
|
||||
oom, _, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "romm", Reason: settings.HoldReasonUnhealthyStop, UnhealthyKind: "oom_storm", Trip: 2})
|
||||
mustHave(t, oom, "out-of-memory storm", "stop 2 within 24 hours")
|
||||
}
|
||||
|
||||
func TestR1340_UpdateFailed(t *testing.T) {
|
||||
msg, leg, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "kimai", At: "2026-10-29T03:10:00Z", Reason: settings.HoldReasonUpdateFailed,
|
||||
UndoState: "half", CopyTier: 1, CopyDate: "2026-10-29T02:00:00Z"})
|
||||
mustHave(t, msg, "update at 2026-10-29T03:10:00Z did not come up healthy", "undo failed half-way", "tier 1, 2026-10-29T02:00:00Z", "--clear-restore-hold")
|
||||
mustNotHave(t, msg, "database", optionCStory)
|
||||
if leg != "update-hold" {
|
||||
t.Errorf("leg %q", leg)
|
||||
}
|
||||
}
|
||||
|
||||
func TestR1340_RestoreFailedWithADatabase(t *testing.T) {
|
||||
msg, leg, reason := heldStoppedOperatorText(settings.RestoreHold{Stack: "paperless", Reason: settings.HoldReasonRestoreFailed, RestoreKind: settings.RestoreKindOffsite,
|
||||
DBReplay: true, ReplayError: "ERROR: relation exists", RollbackErr: "psql: connection refused"})
|
||||
mustHave(t, msg, "off-site restore failed at the database replay", "rollback to the pre-restore database copy also failed", "ERROR: relation exists", "psql: connection refused")
|
||||
mustNotHave(t, msg, optionCStory)
|
||||
if leg != "restore-hold" || reason != "psql: connection refused" {
|
||||
t.Errorf("leg %q reason %q", leg, reason)
|
||||
}
|
||||
}
|
||||
|
||||
// The measured case: Radicale has no database; the volume could not be removed and the put-back's stop failed.
|
||||
func TestR1340_PutBackWithoutADatabase(t *testing.T) {
|
||||
msg, leg, reason := heldStoppedOperatorText(settings.RestoreHold{Stack: "radicale", Reason: settings.HoldReasonRestoreMixed, RestoreKind: settings.RestoreKindOffsite,
|
||||
MixedCause: settings.MixedCausePutBack, FailedStep: "stopping the app before the put-back failed, so nothing was put back: compose down: timeout",
|
||||
ReplayError: "volume rm -f radicale_data: volume is in use", SafetyDump: "/mnt/hdd/felhom-data/backups/restore-undo/radicale/20261029T081800Z"})
|
||||
mustHave(t, msg, `"radicale"`, "off-site restore failed, and putting the live data back FAILED", "stopping the app before the put-back failed",
|
||||
"compose down: timeout", "kept at /mnt/hdd/felhom-data/backups/restore-undo/radicale/20261029T081800Z", "volume is in use")
|
||||
mustNotHave(t, msg, "database", "rolled back", optionCStory)
|
||||
if leg != "restore-hold-mixed" || reason != "put-back failed: stopping the app before the put-back failed, so nothing was put back" {
|
||||
t.Errorf("leg %q reason %q", leg, reason)
|
||||
}
|
||||
}
|
||||
|
||||
func TestR1340_PutBackAfterADatabaseReplay(t *testing.T) {
|
||||
msg, _, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "immich", Reason: settings.HoldReasonRestoreMixed, RestoreKind: settings.RestoreKindOffsite,
|
||||
MixedCause: settings.MixedCausePutBack, DBReplay: true, FailedStep: "putting the live volumes back failed: tar xf failed"})
|
||||
mustHave(t, msg, "failed at the database replay", "putting the live volumes back failed: tar xf failed", "There is no copy of the live volumes")
|
||||
}
|
||||
|
||||
func TestR1340_DefinitionWrite(t *testing.T) {
|
||||
msg, _, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "gokapi", Reason: settings.HoldReasonRestoreMixed, RestoreKind: settings.RestoreKindOffsite,
|
||||
MixedCause: settings.MixedCauseDefinitionWrite, FailedStep: "writing the live app definition back failed: app.yaml write failed", ReplayError: "app.yaml write failed"})
|
||||
mustHave(t, msg, "while writing the backup's app definition, before any data was touched", "writing the live app definition back failed", "data is the live data")
|
||||
mustNotHave(t, msg, "database", optionCStory)
|
||||
}
|
||||
|
||||
func TestR1340_LocalRestore(t *testing.T) {
|
||||
msg, _, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "app", Reason: settings.HoldReasonRestoreMixed, RestoreKind: settings.RestoreKindLocal,
|
||||
MixedCause: settings.MixedCausePutBack, FailedStep: "putting the live volumes back failed: tar xf failed", SafetyDump: "/srv/undo/app/1"})
|
||||
mustHave(t, msg, "restore from its own backup failed", "putting the live volumes back failed")
|
||||
mustNotHave(t, msg, "off-site", "database", optionCStory)
|
||||
}
|
||||
|
||||
func TestR1340_PutBackPending(t *testing.T) {
|
||||
msg, leg, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "immich", Reason: settings.HoldReasonPutBackPending, SafetyDump: "/mnt/hdd/undo/immich/1"})
|
||||
mustHave(t, msg, "copy (/mnt/hdd/undo/immich/1) is on a drive that is not bound", "poured back and the app started when the drive is bound", "DROPS that put-back")
|
||||
mustNotHave(t, msg, "database", optionCStory)
|
||||
if leg != "putback-pending" {
|
||||
t.Errorf("leg %q", leg)
|
||||
}
|
||||
}
|
||||
|
||||
func TestR1340_InterruptedPutBack(t *testing.T) {
|
||||
msg, _, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "immich", Reason: settings.HoldReasonRestoreMixed, MixedCause: settings.MixedCauseInterruptedPutBack,
|
||||
FailedStep: "putting the interrupted restore's copy back failed: compose down: timeout", SafetyDump: "/mnt/hdd/undo/immich/1"})
|
||||
mustHave(t, msg, "cut by a stop could not be put back", "compose down: timeout", "kept at /mnt/hdd/undo/immich/1")
|
||||
mustNotHave(t, msg, "database", "off-site", optionCStory)
|
||||
}
|
||||
|
||||
// A mixed hold written before R-1340 names only what its record says.
|
||||
func TestR1340_LegacyMixedHoldAssumesNothing(t *testing.T) {
|
||||
msg, _, _ := heldStoppedOperatorText(settings.RestoreHold{Stack: "x", At: "2026-10-01T00:00:00Z", Reason: settings.HoldReasonRestoreMixed, ReplayError: "boom"})
|
||||
mustHave(t, msg, "stopped half-way", "The hold record says: boom")
|
||||
mustNotHave(t, msg, "database", optionCStory)
|
||||
}
|
||||
|
||||
// The notification reads the hold the writer persisted; when that write failed, it rebuilds one from the errors.
|
||||
func TestR1340_NotifyReadsTheHoldRecord(t *testing.T) {
|
||||
rec := settings.RestoreHold{Stack: "radicale", Reason: settings.HoldReasonRestoreMixed, MixedCause: settings.MixedCausePutBack, FailedStep: "x"}
|
||||
get := func(s string) (settings.RestoreHold, bool) { return rec, s == "radicale" }
|
||||
if h := heldHoldFromNotify(get, "radicale", errors.New("e"), nil); h.MixedCause != settings.MixedCausePutBack {
|
||||
t.Fatalf("the persisted record was not used: %+v", h)
|
||||
}
|
||||
h := heldHoldFromNotify(get, "other", errors.New("replay"), errors.New("rollback"))
|
||||
if h.Reason != settings.HoldReasonRestoreFailed || h.RollbackErr != "rollback" || h.ReplayError != "replay" {
|
||||
t.Fatalf("fallback hold %+v", h)
|
||||
}
|
||||
if h := heldHoldFromNotify(nil, "other", errors.New("replay"), nil); h.Reason != settings.HoldReasonRestoreMixed {
|
||||
t.Fatalf("no rollback error = mixed, got %+v", h)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log"
|
||||
"net/http"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
|
||||
)
|
||||
|
||||
// hostOOMSource (R-528) is the 30-s dead-app scan's host-side OOM counter: one agent call per scan (5 s cap). An agent
|
||||
// without the route (404) is not asked again for hostOOMRetryAfter, and said once; any other failure is a nil answer
|
||||
// for this scan only (Docker's flag still works). Pinned by TestR528_HostOOMSource*.
|
||||
const hostOOMRetryAfter = 10 * time.Minute
|
||||
|
||||
func hostOOMSource(ask func(context.Context) (agentapi.GuestOOMKillsState, error), logger *log.Logger, now func() time.Time) func() map[string]int64 {
|
||||
var mu sync.Mutex
|
||||
var until time.Time
|
||||
said := false
|
||||
return func() map[string]int64 {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if now().Before(until) {
|
||||
return nil
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
st, err := ask(ctx)
|
||||
if err != nil {
|
||||
var se *agentapi.StatusError
|
||||
if errors.As(err, &se) && se.Code == http.StatusNotFound {
|
||||
until = now().Add(hostOOMRetryAfter)
|
||||
if !said {
|
||||
logger.Printf("[INFO] [deadapp] the agent has no OOM counter route (R-528) — Docker's OOM flag only; asked again in %s", hostOOMRetryAfter)
|
||||
said = true
|
||||
}
|
||||
return nil
|
||||
}
|
||||
logger.Printf("[WARN] [deadapp] the host OOM counter could not be read: %v — Docker's OOM flag only for this scan", err)
|
||||
return nil
|
||||
}
|
||||
if !st.Available {
|
||||
return nil
|
||||
}
|
||||
return st.Containers
|
||||
}
|
||||
}
|
||||
@@ -34,6 +34,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/backupwindow"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/bootrecon"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/bootstrap"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/busymark"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/channelhealth"
|
||||
cf "gitea.dooplex.hu/admin/felhom-controller/internal/cloudflare"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
|
||||
@@ -92,8 +93,9 @@ func main() {
|
||||
abandonStatus := flag.Bool("abandon-status", false, "R-241: print the state of this box's off-site abandonment countdown (set-aside path, due date, days left) and exit. Read-only.")
|
||||
abandonExtend := flag.Int("abandon-extend", 0, "R-241 (operator): extend a running abandonment countdown by N days from now, then exit. Refuses when no countdown is running.")
|
||||
abandonStop := flag.Bool("abandon-stop", false, "R-241 (operator): stop a running abandonment countdown, then exit. The set-aside history is kept and nothing is deleted. Refuses when no countdown is running.")
|
||||
listRestoreHolds := flag.Bool("restore-holds", false, "R-379 (operator): list apps this controller is deliberately holding stopped after a failed database restore whose rollback also failed, then exit. Read-only.")
|
||||
clearRestoreHold := flag.String("clear-restore-hold", "", "R-379 (operator): clear the restore hold on APP so it can start again, then exit. Clearing does NOT repair the database — check the app's data first; the undo copies are in its unit's db-dumps dir.")
|
||||
listRestoreHolds := flag.Bool("restore-holds", false, "R-379 (operator): list the apps this controller is holding stopped (every reason, with why), then exit. Read-only.")
|
||||
clearRestoreHold := flag.String("clear-restore-hold", "", "R-379/D64 (operator): clear the hold on APP (any reason), then exit. Run it inside the guest: "+holdClearCommandHint+
|
||||
". It asks the RUNNING controller, so no restart is needed; only when no controller runs does it edit settings.json. Clearing repairs nothing and starts nothing — check the app's data first.")
|
||||
flag.Parse()
|
||||
|
||||
if *showVersion {
|
||||
@@ -172,21 +174,23 @@ func main() {
|
||||
// starts, exactly like the other CLI subcommands: each loads config + settings, acts, and exits.
|
||||
// R-379 (operator): the way OUT of a hold. A CLI subcommand rather than an API route, matching
|
||||
// --abandon-stop above: the controller's HTTP surface is authenticated as the CUSTOMER, and
|
||||
// clearing a hold is a decision that should follow an operator looking at the database. Reaching
|
||||
// clearing a hold is a decision that should follow an operator looking at the data. Reaching
|
||||
// this needs `docker exec` into the guest, which is the operator gate this project already uses.
|
||||
// R-1343: the clear goes to the RUNNING controller over its local socket (holdclear.go); the list reads
|
||||
// settings.json, which the running controller saves at every hold change.
|
||||
if *listRestoreHolds || *clearRestoreHold != "" {
|
||||
cfg, err := config.LoadPermissive(*configPath)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "restore-hold: loading config: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if *listRestoreHolds {
|
||||
lg := log.New(os.Stderr, "", 0)
|
||||
sett, err := settings.Load(cfg.Paths.DataDir+"/settings.json", lg)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "restore-hold: loading settings: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if *listRestoreHolds {
|
||||
holds := sett.ListRestoreHolds()
|
||||
if len(holds) == 0 {
|
||||
fmt.Println("no restore holds in force")
|
||||
@@ -197,34 +201,9 @@ func main() {
|
||||
}
|
||||
os.Exit(0)
|
||||
}
|
||||
cleared, err := sett.ClearRestoreHold(*clearRestoreHold)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "restore-hold: clearing %s: %v\n", *clearRestoreHold, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if !cleared {
|
||||
// "there was nothing to clear" is not success — a silent 0 here would let an operator
|
||||
// believe they had unblocked an app whose name they mistyped.
|
||||
fmt.Fprintf(os.Stderr, "no restore hold in force for %q — nothing was changed\n", *clearRestoreHold)
|
||||
os.Exit(2)
|
||||
}
|
||||
// THE RESTART IS NOT OPTIONAL, and saying so is the difference between a route that works
|
||||
// and one an operator believes worked. This runs as a SECOND process: it clears the hold in
|
||||
// settings.json, but the RUNNING controller holds its own in-memory Settings and keeps
|
||||
// refusing until it reloads. Measured on demo-hp 2026-08-22 — the clear succeeded, the file
|
||||
// was correct, and the customer's start button still refused until the controller restarted.
|
||||
//
|
||||
// There is also a lost-update window while both processes hold the file: if the running
|
||||
// controller saves settings.json from its own copy before the restart, the clear is undone.
|
||||
// Restarting promptly closes it. A route that clears through the running controller would
|
||||
// remove both problems and is the right shape later; it needs an operator tier the
|
||||
// controller's HTTP surface does not have today (it authenticates as the CUSTOMER, and a
|
||||
// customer clearing their own hold is precisely what the hold exists to prevent).
|
||||
fmt.Printf("restore hold cleared for %s in settings.json.\n", *clearRestoreHold)
|
||||
fmt.Printf(" NOW RESTART THE CONTROLLER, or it will keep refusing to start the app:\n")
|
||||
fmt.Printf(" systemctl restart felhom-controller-bootstrap.service\n")
|
||||
fmt.Printf(" Check the app's data first — the undo copies are in its unit's db-dumps dir.\n")
|
||||
os.Exit(0)
|
||||
// R-1343: through the RUNNING controller (holdclear.go) — the old second-process edit of settings.json was
|
||||
// written back by the running controller's next save. Only when no controller answers is the file edited here.
|
||||
os.Exit(clearRestoreHoldCLI(cfg.Paths.DataDir, *clearRestoreHold, os.Stdout, os.Stderr))
|
||||
}
|
||||
|
||||
if *abandonStatus || *abandonExtend > 0 || *abandonStop {
|
||||
@@ -485,6 +464,19 @@ func main() {
|
||||
return stackMgr.DriveLive(hdd)
|
||||
})
|
||||
appStopRecovery := appStopGuard.Recover()
|
||||
// D64 = B / R-1343: the operator's hold clear — ONE door for the hub's clear_hold and the box command, run in THIS
|
||||
// process so memory and settings.json agree (holdclear.go). The box command reaches it over a unix socket in the
|
||||
// data dir (0600, nothing on TCP). A socket that cannot be bound is logged; the command then says plainly that no
|
||||
// running controller answered.
|
||||
holdDeps := holdClearDeps{clear: appStopGuard.ClearHoldByOperatorThen, forgetUpdateHeld: stackMgr.ForgetUpdateHeld,
|
||||
keepStopped: stackMgr.RecordStopWantedIfDown, logger: logger}
|
||||
if holdLn, herr := listenHoldCtl(holdCtlPath(cfg.Paths.DataDir), func(stack, who string) (string, error) {
|
||||
return clearHoldNow(holdDeps, stack, who)
|
||||
}, logger); herr != nil {
|
||||
logger.Printf("[ERROR] [hold] the local hold-clear channel could not be opened: %v — `--clear-restore-hold` will edit settings.json and need a restart", herr)
|
||||
} else {
|
||||
defer holdLn.Close()
|
||||
}
|
||||
// R-1251: a put-back that waited for its drive (a hold from an earlier start) runs now if that drive is bound.
|
||||
if started := appStopGuard.ResumeWaitingPutBacks(); len(started) > 0 {
|
||||
logger.Printf("[WARN] [appstop] %d app(s) whose interrupted restore waited for its drive are put back and started: %v", len(started), started)
|
||||
@@ -587,6 +579,11 @@ func main() {
|
||||
logger.Println("[INFO] Healthchecks ping UUIDs configured but no longer used — monitoring is now handled by the Hub")
|
||||
}
|
||||
|
||||
// D65 = A (2026-10-29): the busy file the host agent reads before an operator reboot of the guest (internal/busymark).
|
||||
// Probes are added where each component is built; the writer starts once the last one exists (after the exporter).
|
||||
busyMark := &busymark.Writer{Path: filepath.Join(cfg.Paths.DataDir, holdCtlDir, busymark.FileName), Logger: logger}
|
||||
busyMark.AddProbe("an app update is running", stackMgr.AnyUpdating)
|
||||
|
||||
// --- Initialize backup manager (app-data only: DB dumps + Docker-volume tars) ---
|
||||
var backupMgr *backup.Manager
|
||||
// R-550: the restore the last stop interrupted, found when the record loads; raised once the notifier exists.
|
||||
@@ -598,6 +595,9 @@ func main() {
|
||||
}
|
||||
if cfg.Backup.Enabled {
|
||||
backupMgr = backup.NewManager(cfg, sett, logger)
|
||||
busyMark.AddProbe("a backup, a restore or an off-site run is running", func() bool {
|
||||
return backupMgr.IsRunning() || backupMgr.RestoreStatus().Running
|
||||
})
|
||||
// The stray-copy fix: a second-drive copy container a previous controller process lost (restart, self-update)
|
||||
// still runs `rsync --delete`. Remove every one BEFORE the scheduler or the web server can start a backup or a
|
||||
// restore; until that succeeds, restores are refused. Pinned by TestStrayCopy_ClearIsWiredBeforeTheScheduler.
|
||||
@@ -682,6 +682,7 @@ func main() {
|
||||
backupMgr.SetVersionCheck(stackMgr.PinNotRunning)
|
||||
// R-671 (v0.272.0): a restore that lifts an update hold removes the undo copies that hold kept.
|
||||
backupMgr.SetUndoCopyRemover(stackMgr.RemoveUndoCopies)
|
||||
backupMgr.SetUpdateHeldForgetter(stackMgr.ForgetUpdateHeld) // R-1347
|
||||
}
|
||||
if n := stackMgr.ResumeInterruptedUpdates(ctx); n > 0 {
|
||||
logger.Printf("[WARN] [update] resumed %d interrupted update(s)", n)
|
||||
@@ -1035,6 +1036,9 @@ func main() {
|
||||
logger.Printf("[WARN] [deadapp] agent client init failed (%v) — the boot grace cannot learn of a crash boot; 90 s", err)
|
||||
} else {
|
||||
crashProbe = crashboot.AgentProbe(ac.CrashGuard)
|
||||
// R-528: the 30-s scan also asks the host for the kernel's per-container OOM counter (Docker's own flag can
|
||||
// stay false inside an LXC guest). TestR528_HostOOMSourceIsWiredAtStartup.
|
||||
stackMgr.SetHostOOMKills(hostOOMSource(ac.GuestOOMKills, logger, time.Now))
|
||||
}
|
||||
}
|
||||
bootGrace := crashboot.New(startTime, crashProbe, logger)
|
||||
@@ -1078,6 +1082,7 @@ func main() {
|
||||
holdReporter.Tick(time.Now())
|
||||
return nil
|
||||
})
|
||||
oomLog := &oomLogGate{} // R-1341
|
||||
sched.Every("deadapp-check", 30*time.Second, func(ctx context.Context) error {
|
||||
if bootGrace.Within(time.Now()) {
|
||||
return nil // still inside the startup settle window (90 s, or ~15 min after a crash boot)
|
||||
@@ -1093,9 +1098,13 @@ func main() {
|
||||
if oerr != nil {
|
||||
logger.Printf("[WARN] [deadapp] OOM scan failed: %v", oerr)
|
||||
} else {
|
||||
// R-1341: the line once per NEW kill (oomlog.go); the event call stays per scan — the notifier's cooldown
|
||||
// keeps it to one, as before.
|
||||
for _, o := range oomLog.fresh(ooms) {
|
||||
logger.Printf("[WARN] [deadapp] %s: container %s was OOM-killed (started %s; kills %d, limit %s, peak %s; seen by %s)", o.Stack, o.Container, o.StartedAt, o.Kills, o.MemLimit, o.Peak, o.Source)
|
||||
}
|
||||
for _, o := range ooms {
|
||||
logger.Printf("[WARN] [deadapp] %s: container %s was OOM-killed (started %s; kills %d, limit %s, peak %s)", o.Stack, o.Container, o.StartedAt, o.Kills, o.MemLimit, o.Peak)
|
||||
notifier.NotifyAppOOM(o.Stack, o.Container, o.StartedAt, o.Kills, o.MemLimit, o.Peak)
|
||||
notifier.NotifyAppOOMFrom(o.Stack, o.Container, o.StartedAt, o.Kills, o.MemLimit, o.Peak, o.Source)
|
||||
}
|
||||
}
|
||||
// v0.269.0 (`09` §3 decision 28, R-667): a crash loop or an OOM storm is STOPPED by the box.
|
||||
@@ -1214,7 +1223,7 @@ func main() {
|
||||
// R-314/R-279/R-177 (`09` §3 decision 185): the operator's closed list of actions, carried in the
|
||||
// report reply and run in THIS process (opactions_wiring.go). The result rides the next report;
|
||||
// a finished result fires an out-of-cycle report through the trigger once it exists.
|
||||
opActions := report.NewOperatorActions(ctx, operatorActionHandlers(backupMgr, sched, func() {
|
||||
opActions := report.NewOperatorActions(ctx, operatorActionHandlers(backupMgr, sched, holdDeps, func() {
|
||||
if f, ok := opActionReportFire.Load().(func()); ok && f != nil {
|
||||
f()
|
||||
}
|
||||
@@ -1400,6 +1409,13 @@ func main() {
|
||||
return nil
|
||||
})
|
||||
|
||||
// R-1256 H8: a restore copy no hold, no pending put-back and no running restore needs goes after 14 days. Every 6
|
||||
// hours — a copy outliving its age by hours costs disk, never data. TestR1256_H8_TheSweepIsWiredAtStartup.
|
||||
sched.Every("restore-copy-sweep", 6*time.Hour, func(ctx context.Context) error {
|
||||
backupMgr.SweepUnusedRestoreCopies(time.Now())
|
||||
return nil
|
||||
})
|
||||
|
||||
// Tier 2: off-drive copy of each HDD app's recovery unit + userdata (auto-enabled, auto-target).
|
||||
// Runs after the DB dump so it copies a fresh unit.
|
||||
backupMgr.SetTier2Notifier(func(stackName, destLabel string, dur time.Duration, err error) {
|
||||
@@ -1538,24 +1554,20 @@ func main() {
|
||||
// DELIBERATELY NOT `backup_failed`: customer-enabled by default, Hungarian "A biztonsági
|
||||
// mentés sikertelen!", and this is a deliberate hold rather than a backup failure. R-171 one
|
||||
// path over.
|
||||
backupMgr.SetRestoreHoldNotify(func(stack string, replayErr, rollbackErr error) {
|
||||
d := notify.BackupRunFailuresDetails{RunKind: "offsite-reconstitute", Failed: 1, Attempted: 1}
|
||||
if rollbackErr == nil { // R-893: the rollback WORKED, but files/volumes/version had already moved
|
||||
d.Apps = append(d.Apps, notify.RunFailureDetail{App: stack, Leg: "restore-hold-mixed", Reason: "replay failed after files/volumes/version moved"})
|
||||
msg := fmt.Sprintf("App %q is HELD STOPPED for support: its off-site database restore failed. The database was rolled back to the pre-restore copy and the live definition written back, "+
|
||||
"but its files and named volumes are the snapshot's — a mixed state (R-893). The app will not start from any path until the hold is cleared. Replay error: %v", stack, replayErr)
|
||||
// R-1340: the text is built from the HOLD RECORD the writer persisted just before this call (holdmail.go) — its
|
||||
// reason, which restore, which branch, which put-back step failed, whether it was a database replay — never the
|
||||
// option-C database story for every cause.
|
||||
holdNotify := func(stack string, replayErr, rollbackErr error) {
|
||||
h := heldHoldFromNotify(sett.GetRestoreHold, stack, replayErr, rollbackErr)
|
||||
msg, leg, reason := heldStoppedOperatorText(h)
|
||||
d := notify.BackupRunFailuresDetails{RunKind: heldRunKind(h), Failed: 1, Attempted: 1}
|
||||
d.Apps = append(d.Apps, notify.RunFailureDetail{App: stack, Leg: leg, Reason: reason})
|
||||
notifier.NotifyBackupRunFailures(msg, d)
|
||||
return
|
||||
}
|
||||
reason := "rollback failed"
|
||||
if rollbackErr != nil {
|
||||
reason = rollbackErr.Error()
|
||||
}
|
||||
d.Apps = append(d.Apps, notify.RunFailureDetail{App: stack, Leg: "restore-hold", Reason: reason})
|
||||
msg := fmt.Sprintf("App %q is HELD STOPPED: its off-site database restore failed AND the rollback to the customer's own pre-restore copy also failed. "+
|
||||
"The app will not start from any path until the hold is cleared. Replay error: %v. Rollback error: %v", stack, replayErr, rollbackErr)
|
||||
notifier.NotifyBackupRunFailures(msg, d)
|
||||
})
|
||||
backupMgr.SetRestoreHoldNotify(holdNotify)
|
||||
// R-1348: the hold an interrupted put-back wrote at START-UP (Recover, before any notifier existed) or on a resume.
|
||||
// The same sentence (R-1340, MixedCauseInterruptedPutBack), the same mail.
|
||||
appStopGuard.SetHeldNotify(func(stack string) { holdNotify(stack, nil, nil) })
|
||||
// R-893 option A (D20 = A): a failed off-site restore that was PUT BACK — the live volumes and definition poured
|
||||
// back and the app started. The same operator-tier event as the hold above (no new event type), its own leg.
|
||||
// D47 follow-up: a unit of an earlier install was set aside whole — the operator sees it (its own leg).
|
||||
@@ -1660,6 +1672,7 @@ func main() {
|
||||
if quiesceLoop != nil {
|
||||
quiesceLoop.SetCatchUpFn(catchUp.Running)
|
||||
}
|
||||
busyMark.AddProbe("the night backup catch-up is running", catchUp.Running)
|
||||
catchUp.Evaluate(ctx, "controller start")
|
||||
resumeTick := time.NewTicker(time.Minute)
|
||||
go nightchain.ResumeWatch(ctx, resumeTick.C,
|
||||
@@ -2049,6 +2062,10 @@ func main() {
|
||||
prevCtl = st.RecordedPrevious(Version)
|
||||
}
|
||||
_, _ = stackMgr.RetainControllerImages(stacks.ControllerImageRecord{Repo: cfg.SelfUpdate.Image, Running: Version, Previous: prevCtl})
|
||||
// decision 285 (2026-11-09): the pinned infrastructure images (traefik, cloudflared, FileBrowser, Samba) — keep
|
||||
// the pin and the one before it. A pin moves only with a controller release, so once per start is enough; the
|
||||
// infra stacks were reconciled long before these 3 minutes. Retried every 2 minutes while an app pulls (R-863).
|
||||
stackMgr.RunInfraImageRetention(ctx, infra.Images(), 2*time.Minute, 30)
|
||||
}()
|
||||
|
||||
// --- Initialize API router ---
|
||||
@@ -2118,6 +2135,8 @@ func main() {
|
||||
// this the export path would be the uncovered sibling of two covered ones, which is how a reader
|
||||
// concludes the whole class is handled (§2.2).
|
||||
appExporter.SetStopGuard(exportStopGuard{g: appStopGuard})
|
||||
busyMark.AddProbe("an app export or import is running", appExporter.IsRunning)
|
||||
go busyMark.Run(ctx)
|
||||
// R-1279 (D48 for the import): the import downloads first, with the restores' helper and sentences.
|
||||
if backupMgr != nil {
|
||||
appExporter.SetImagePreparer(backupMgr.PrepareRestoreImages)
|
||||
@@ -2126,6 +2145,7 @@ func main() {
|
||||
|
||||
// --- Initialize web server ---
|
||||
webServer := web.NewServer(cfg, stackMgr, cpuCollector, backupMgr, sched, sett, alertMgr, notifier, updater, logger, Version)
|
||||
go webServer.WatchOperatorReboot(ctx) // D65: „Felhom support restarted your box at …" after an operator reboot
|
||||
// v0.316.0: which BUILD this is, not just which version it calls itself. Every shell's stylesheet
|
||||
// URL is derived from it, so two builds of one version can never share a cache entry. Init-only,
|
||||
// before the HTTP server starts.
|
||||
@@ -3181,9 +3201,23 @@ func (a *stackAdapter) ListDeployedStacks() []backup.StackSummary {
|
||||
}
|
||||
|
||||
func (a *stackAdapter) StopStack(name string) error {
|
||||
if err := faultPoint("stop-" + name); err != nil { // D63: a no-op outside a -tags faultinject test build
|
||||
return err
|
||||
}
|
||||
return a.mgr.StopStack(name)
|
||||
}
|
||||
|
||||
// StackRemoved answers backup's optional StackRemovalReporter (R-1342): the app is known and no longer installed — a
|
||||
// stable-start watch then ends quietly instead of alarming „did not start". An app this box does not know at all is
|
||||
// not „removed" (no answer → the watch goes on as before).
|
||||
func (a *stackAdapter) StackRemoved(name string) bool {
|
||||
s, ok := a.mgr.GetStack(name)
|
||||
return ok && !s.Deployed
|
||||
}
|
||||
|
||||
// The watch reaches StackRemoved through a type assertion; this line makes the adapter's answer a compile-time fact.
|
||||
var _ backup.StackRemovalReporter = (*stackAdapter)(nil)
|
||||
|
||||
// IsDeploying answers backup's optional deployingReporter (R-634): the volume leg asks it again
|
||||
// immediately before it stops an app, because its list is taken once at the start of the run.
|
||||
func (a *stackAdapter) IsDeploying(name string) bool {
|
||||
@@ -3425,6 +3459,9 @@ func (a *stackAdapter) RecreateStackDefinitionFromUnit(name, composeSrcDir strin
|
||||
return fmt.Errorf("restoring %s from unit: %w", fname, err)
|
||||
}
|
||||
}
|
||||
if err := faultPoint("defwrite-after-compose"); err != nil { // D63: a no-op outside a -tags faultinject test build
|
||||
return err
|
||||
}
|
||||
// R-1057 (v0.314.0): the gate as the backup recorded it — the unit's app.yaml, read BEFORE the write.
|
||||
gate := stacks.ReadGateAtBackup(filepath.Join(composeSrcDir, "app.yaml"))
|
||||
if err := a.mgr.PersistUnitRedeployConfigGate(name, fullEnv, gate); err != nil {
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"strconv"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
|
||||
)
|
||||
|
||||
// ── R-1341: the OOM line is logged once per NEW kill ──────────────────────────────────────────────────────────────
|
||||
//
|
||||
// ScanOOMKilled reports a container for as long as it lives after a kill (Docker's OOMKilled flag is sticky, and the
|
||||
// kernel's kill counter never goes down), so the 30-s dead-app scan logged „[WARN] [deadapp] … was OOM-killed" about
|
||||
// 2 880 times a day per container — measured 2026-10-29 on Tester 1 — while the notifier's cooldown kept the event to
|
||||
// one. The line now goes out when the kill is NEW: the key is (container, its start time, the kill count), and a line
|
||||
// is logged only when a container's key changes — a second scan of the same kill logs nothing, a new kill (the count
|
||||
// went up) or a restarted container (a new start time) logs again. The event call is unchanged (every scan, the
|
||||
// notifier decides). Pinned by TestR1341_*.
|
||||
|
||||
// oomLogGate remembers the last logged key per container. Not safe for concurrent use: the dead-app job runs serially.
|
||||
type oomLogGate struct {
|
||||
last map[string]string
|
||||
}
|
||||
|
||||
func oomLogKey(o stacks.OOMContainer) string {
|
||||
return o.StartedAt + "|" + strconv.FormatInt(o.Kills, 10)
|
||||
}
|
||||
|
||||
// fresh returns the containers whose kill is new since the last scan, and remembers this scan's keys. A container
|
||||
// absent from this scan is forgotten (it was removed or restarted clean), which also bounds the memory.
|
||||
func (g *oomLogGate) fresh(ooms []stacks.OOMContainer) []stacks.OOMContainer {
|
||||
next := make(map[string]string, len(ooms))
|
||||
var out []stacks.OOMContainer
|
||||
for _, o := range ooms {
|
||||
k := oomLogKey(o)
|
||||
next[o.Container] = k
|
||||
if prev, ok := g.last[o.Container]; ok && prev == k {
|
||||
continue
|
||||
}
|
||||
out = append(out, o)
|
||||
}
|
||||
g.last = next
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
|
||||
)
|
||||
|
||||
// R-1341: the „[WARN] [deadapp] … was OOM-killed" line goes out once per NEW kill, not every 30-s scan.
|
||||
//
|
||||
// RED-PROOF (2026-10-29): with oomLogGate.fresh returning its input unchanged (the pre-R-1341 loop logged every
|
||||
// container every scan), TestR1341_ASecondScanOfTheSameKillLogsNothing fails on „logged 1 line(s), want 0".
|
||||
|
||||
func oom(container, started string, kills int64) stacks.OOMContainer {
|
||||
return stacks.OOMContainer{Stack: "gokapi", Container: container, StartedAt: started, Kills: kills}
|
||||
}
|
||||
|
||||
func TestR1341_ASecondScanOfTheSameKillLogsNothing(t *testing.T) {
|
||||
g := &oomLogGate{}
|
||||
scan := []stacks.OOMContainer{oom("gokapi", "2026-10-29T09:02:00Z", 1)}
|
||||
if got := g.fresh(scan); len(got) != 1 {
|
||||
t.Fatalf("the first sighting must log, logged %d line(s)", len(got))
|
||||
}
|
||||
for i := 0; i < 5; i++ {
|
||||
if got := g.fresh(scan); len(got) != 0 {
|
||||
t.Fatalf("scan %d of the same kill logged %d line(s), want 0", i+2, len(got))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestR1341_ANewKillLogsAgain(t *testing.T) {
|
||||
g := &oomLogGate{}
|
||||
g.fresh([]stacks.OOMContainer{oom("gokapi", "2026-10-29T09:02:00Z", 1)})
|
||||
got := g.fresh([]stacks.OOMContainer{oom("gokapi", "2026-10-29T09:02:00Z", 2)})
|
||||
if len(got) != 1 || got[0].Kills != 2 {
|
||||
t.Fatalf("a new kill (count 1→2) must log once, got %+v", got)
|
||||
}
|
||||
if got := g.fresh([]stacks.OOMContainer{oom("gokapi", "2026-10-29T09:02:00Z", 2)}); len(got) != 0 {
|
||||
t.Fatalf("the same new kill must not log twice, got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestR1341_ARestartedContainerLogsAgain(t *testing.T) {
|
||||
g := &oomLogGate{}
|
||||
g.fresh([]stacks.OOMContainer{oom("gokapi", "2026-10-29T09:02:00Z", 1)})
|
||||
if got := g.fresh([]stacks.OOMContainer{oom("gokapi", "2026-10-29T09:30:00Z", 1)}); len(got) != 1 {
|
||||
t.Fatalf("a restarted container's kill must log, got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Two containers are judged apart; one that leaves the scan is forgotten, so its return logs again.
|
||||
func TestR1341_ContainersAreJudgedApart(t *testing.T) {
|
||||
g := &oomLogGate{}
|
||||
a, b := oom("a", "t1", 1), oom("b", "t1", 1)
|
||||
if got := g.fresh([]stacks.OOMContainer{a, b}); len(got) != 2 {
|
||||
t.Fatalf("first scan: %+v", got)
|
||||
}
|
||||
if got := g.fresh([]stacks.OOMContainer{a}); len(got) != 0 {
|
||||
t.Fatalf("a seen again must not log: %+v", got)
|
||||
}
|
||||
if got := g.fresh([]stacks.OOMContainer{a, b}); len(got) != 1 || got[0].Container != "b" {
|
||||
t.Fatalf("b came back after leaving the scan: %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -17,18 +17,26 @@ import (
|
||||
// executor is built earlier than the trigger, so it reads it through this holder.
|
||||
var opActionReportFire atomic.Value
|
||||
|
||||
// operatorActionHandlers wires the four operator actions (R-314/R-279/R-177, `09` §3 decision 185)
|
||||
// operatorActionHandlers wires the five operator actions (R-314/R-279/R-177, `09` §3 decision 185)
|
||||
// to the running process. ONE wiring site, called from main and from the wiring test, so the test
|
||||
// exercises the doors production uses (the seam-built-but-never-wired trap).
|
||||
//
|
||||
// offsite_backup_now makes the SAME four checks as the household's „back up now"
|
||||
// (offboxRunHandler) and then runs the same call, so an operator press can never do what the
|
||||
// household's own press cannot.
|
||||
func operatorActionHandlers(backupMgr *backup.Manager, sched *scheduler.Scheduler, resultReady func(), logger *log.Logger) report.OperatorActionHandlers {
|
||||
//
|
||||
// clear_hold (D64 = B) goes through clearHoldNow with hold — the same door the box command's socket uses
|
||||
// (holdclear.go), so the hub's button and the box command can never differ. hold.clear nil = not available.
|
||||
func operatorActionHandlers(backupMgr *backup.Manager, sched *scheduler.Scheduler, hold holdClearDeps, resultReady func(), logger *log.Logger) report.OperatorActionHandlers {
|
||||
h := report.OperatorActionHandlers{ResultReady: resultReady}
|
||||
if sched != nil {
|
||||
h.RunJob = sched.RunNow
|
||||
}
|
||||
if hold.clear != nil {
|
||||
h.ClearHold = func(stack string, actionID int64) (string, error) {
|
||||
return clearHoldNow(hold, stack, fmt.Sprintf("the operator via the hub, action #%d", actionID))
|
||||
}
|
||||
}
|
||||
if backupMgr == nil {
|
||||
return h
|
||||
}
|
||||
|
||||
@@ -57,7 +57,7 @@ func TestD1_AbandonStopActsOnTheRunningManagerAndDisk(t *testing.T) {
|
||||
t.Fatal("fixture: a countdown must be running")
|
||||
}
|
||||
|
||||
o := report.NewOperatorActions(context.Background(), operatorActionHandlers(mgr, nil, nil, logger), logger)
|
||||
o := report.NewOperatorActions(context.Background(), operatorActionHandlers(mgr, nil, holdClearDeps{}, nil, logger), logger)
|
||||
o.Reconcile([]report.OperatorAction{{ID: 1, Action: report.OpAbandonStop}})
|
||||
|
||||
if mgr.AbandonStatus().Active {
|
||||
@@ -97,7 +97,7 @@ func TestD1_OffsiteBackupNowRefusesWithoutATarget(t *testing.T) {
|
||||
t.Fatal("no restic may run")
|
||||
return nil, nil
|
||||
})
|
||||
o := report.NewOperatorActions(context.Background(), operatorActionHandlers(mgr, nil, nil, logger), logger)
|
||||
o := report.NewOperatorActions(context.Background(), operatorActionHandlers(mgr, nil, holdClearDeps{}, nil, logger), logger)
|
||||
o.Reconcile([]report.OperatorAction{{ID: 9, Action: report.OpOffsiteBackupNow}})
|
||||
if r := o.Results(); len(r) != 1 || r[0].Outcome != report.OutcomeRefused {
|
||||
t.Fatalf("result = %+v", r)
|
||||
@@ -115,7 +115,7 @@ func TestD1_RunJobUsesTheScheduler(t *testing.T) {
|
||||
defer cancel()
|
||||
sched.Start(ctx)
|
||||
defer sched.Stop()
|
||||
o := report.NewOperatorActions(ctx, operatorActionHandlers(nil, sched, nil, logger), logger)
|
||||
o := report.NewOperatorActions(ctx, operatorActionHandlers(nil, sched, holdClearDeps{}, nil, logger), logger)
|
||||
o.Reconcile([]report.OperatorAction{{ID: 1, Action: report.OpRunJob, Arg: "fill-watch"}, {ID: 2, Action: report.OpRunJob, Arg: "offsite-proof"}})
|
||||
select {
|
||||
case od := <-ran:
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
package main
|
||||
|
||||
import "testing"
|
||||
|
||||
// TestR1256_H8_TheSweepIsWiredAtStartup — R-1256 H8: an unused restore copy goes only through
|
||||
// SweepUnusedRestoreCopies, so main.go must CALL it (the seam-built-but-never-wired class).
|
||||
// RED-PROOF: delete the `restore-copy-sweep` job — this fails.
|
||||
func TestR1256_H8_TheSweepIsWiredAtStartup(t *testing.T) {
|
||||
lines, _, _ := slice4CallLines(t)
|
||||
if len(lines["SweepUnusedRestoreCopies"]) == 0 {
|
||||
t.Fatal("SweepUnusedRestoreCopies is never called — an unused restore copy would stay on the drive for ever")
|
||||
}
|
||||
}
|
||||
@@ -20,7 +20,9 @@ func TestR1286_EveryHoldReasonSaysWhatItIs(t *testing.T) {
|
||||
}{
|
||||
{settings.RestoreHold{Stack: "kimai", At: "2026-10-14T09:06:31Z", Reason: settings.HoldReasonUnhealthyStop, UnhealthyKind: "crash_loop"}, "stopped by the box for a crash loop"},
|
||||
{settings.RestoreHold{Stack: "a", At: "t", Reason: settings.HoldReasonUpdateFailed}, "failed update"},
|
||||
{settings.RestoreHold{Stack: "b", At: "t", Reason: settings.HoldReasonRestoreMixed}, "off-site restore failed its database replay"},
|
||||
// R-1340 (2026-10-29): a record with no cause fields no longer claims „off-site … database replay" — that was the
|
||||
// false story (a local restore of an app with no database read the same); it says what is known.
|
||||
{settings.RestoreHold{Stack: "b", At: "t", Reason: settings.HoldReasonRestoreMixed}, "restore stopped half-way"},
|
||||
{settings.RestoreHold{Stack: "c", At: "t", Reason: settings.HoldReasonPutBackPending}, "pre-restore copy is poured back"},
|
||||
{settings.RestoreHold{Stack: "d", At: "t"}, "failed restore whose rollback also failed"},
|
||||
} {
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// R-1347 and R-1348: each seam is CALLED from main.go (an AST walk) — a seam built and never wired is the project's
|
||||
// oldest shape.
|
||||
func TestR1347_UpdateHeldForgetterIsWiredAtStartup(t *testing.T) {
|
||||
lines, _, _ := slice4CallLines(t)
|
||||
if len(lines["SetUpdateHeldForgetter"]) == 0 {
|
||||
t.Fatal("main.go never calls SetUpdateHeldForgetter — a restored app would stay unwatched until the controller restarts")
|
||||
}
|
||||
}
|
||||
|
||||
func TestR1348_HeldNotifyIsWiredAtStartup(t *testing.T) {
|
||||
lines, _, _ := slice4CallLines(t)
|
||||
if len(lines["SetHeldNotify"]) == 0 {
|
||||
t.Fatal("main.go never calls SetHeldNotify — a failed put-back at start-up would mail nobody")
|
||||
}
|
||||
}
|
||||
|
||||
// The mail's run name for an interrupted put-back. RED (2026-11-03, live on 9202): the sink showed run_kind offsite-reconstitute for a
|
||||
// local restore.
|
||||
func TestR1348_HeldRunKind(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
h settings.RestoreHold
|
||||
want string
|
||||
}{
|
||||
{settings.RestoreHold{Reason: settings.HoldReasonRestoreMixed, MixedCause: settings.MixedCauseInterruptedPutBack}, "restore"},
|
||||
{settings.RestoreHold{RestoreKind: settings.RestoreKindLocal}, "restore"},
|
||||
{settings.RestoreHold{RestoreKind: settings.RestoreKindOffsite}, "offsite-reconstitute"},
|
||||
{settings.RestoreHold{Reason: settings.HoldReasonRestoreMixed}, "offsite-reconstitute"},
|
||||
} {
|
||||
if got := heldRunKind(c.h); got != c.want {
|
||||
t.Errorf("%+v: run kind %q, want %q", c.h, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/report"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
|
||||
)
|
||||
|
||||
// R-1350 (measured 2026-10-29 14:57 on Tester 1): the operator cleared gokapi's crash-loop hold from the hub, the app
|
||||
// stayed stopped (the clear starts nothing), and 30 s later the dead-app scan sent „Telepített alkalmazás nem fut:
|
||||
// Gokapi" — the hold had kept the app out of the scan, and the household's recorded wish was still „running".
|
||||
// The fix: the clear records that the stop is WANTED (desired state „stopped") for an app that is not up, BEFORE the
|
||||
// hold is lifted; a later Start records „running" and ends it.
|
||||
|
||||
// The consequence, through the scan's own derivation: a stopped app whose wish is „running" is DOWN (the alarm); the
|
||||
// same app with the wish „stopped" is not.
|
||||
func TestR1350_AStopWantedIsNotDown(t *testing.T) {
|
||||
st := func(desired string) []stacks.Stack {
|
||||
return []stacks.Stack{{Name: "gokapi", Deployed: true, State: stacks.StateStopped, AppConfig: &stacks.AppConfig{DesiredState: desired}}}
|
||||
}
|
||||
if _, states := classifyRunStates(st(stacks.DesiredStateRunning), nil, nil, time.Now()); len(states) != 1 || !states[0].Down {
|
||||
t.Fatalf("fixture: a stopped app wished running must be down: %+v", states)
|
||||
}
|
||||
if _, states := classifyRunStates(st(stacks.DesiredStateStopped), nil, nil, time.Now()); len(states) != 1 || states[0].Down {
|
||||
t.Fatalf("a stopped app wished stopped is down — the alarm fires: %+v", states)
|
||||
}
|
||||
}
|
||||
|
||||
// The clear records the wish BEFORE the hold is lifted (a dead-app scan between the two would read „no hold, wished
|
||||
// running" and mail), and only for an app that IS held. RED-PROOF: drop the keepStopped call from the clear → FAILS
|
||||
// („not recorded"); call it after the lift → FAILS („after the hold was lifted").
|
||||
func TestR1350_TheClearRecordsTheStopIsWanted(t *testing.T) {
|
||||
r := newD64Rig(t)
|
||||
r.hold(t, settings.RestoreHold{Stack: "gokapi", Reason: settings.HoldReasonUnhealthyStop, UnhealthyKind: "crash_loop", Trip: 1})
|
||||
var recorded []string
|
||||
var heldWhenRecorded bool
|
||||
r.holdDeps.keepStopped = func(stack string) (bool, error) {
|
||||
recorded = append(recorded, stack)
|
||||
_, heldWhenRecorded = r.sett.GetRestoreHold(stack)
|
||||
return true, nil
|
||||
}
|
||||
o := r.actions()
|
||||
o.Reconcile([]report.OperatorAction{{ID: 21, Action: report.OpClearHold, Arg: "gokapi"}})
|
||||
if res := onlyResult(t, o, 21); res.Outcome != report.OutcomeDone {
|
||||
t.Fatalf("result %+v", res)
|
||||
}
|
||||
if len(recorded) != 1 || recorded[0] != "gokapi" {
|
||||
t.Fatalf("the stop was not recorded as wanted: %v", recorded)
|
||||
}
|
||||
if !heldWhenRecorded {
|
||||
t.Fatal("the wish was recorded after the hold was lifted — a scan in between mails app_start_failed")
|
||||
}
|
||||
if lg := r.logBuf.String(); !strings.Contains(lg, "gokapi: the stop is recorded as wanted") {
|
||||
t.Fatalf("the box's log must say it:\n%s", lg)
|
||||
}
|
||||
// a refusal (no hold) records nothing — a down app with no hold is a real fault the alarm must keep
|
||||
o.Reconcile([]report.OperatorAction{{ID: 22, Action: report.OpClearHold, Arg: "kimai"}})
|
||||
if res := onlyResult(t, o, 22); res.Outcome != report.OutcomeRefused {
|
||||
t.Fatalf("result %+v", res)
|
||||
}
|
||||
if len(recorded) != 1 {
|
||||
t.Fatalf("a refused clear recorded a wish: %v", recorded)
|
||||
}
|
||||
}
|
||||
|
||||
// The running controller's clear is wired with the record. RED-PROOF: drop keepStopped from main.go's holdDeps → FAILS.
|
||||
func TestR1350_MainWiresTheRecord(t *testing.T) {
|
||||
raw, err := os.ReadFile("main.go")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(string(raw), "keepStopped: stackMgr.RecordStopWantedIfDown") || !strings.Contains(string(raw), "clear: appStopGuard.ClearHoldByOperatorThen") {
|
||||
t.Fatal("main.go's hold clear does not record the wanted stop (R-1350)")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
|
||||
)
|
||||
|
||||
// R-528: the scan's host counter source. An old agent (404) is asked again only after hostOOMRetryAfter; a transient
|
||||
// failure is retried at the next scan; the counters pass through.
|
||||
// RED-PROOF: drop the 404 back-off → an old agent is asked at every 30-s scan (calls 3, not 1) → FAILS.
|
||||
func TestR528_HostOOMSourceBacksOffAnOldAgent(t *testing.T) {
|
||||
clock := time.Date(2026, 10, 28, 23, 0, 0, 0, time.UTC)
|
||||
calls := 0
|
||||
var answer error = &agentapi.StatusError{Path: "/guest/oom-kills", Code: http.StatusNotFound}
|
||||
src := hostOOMSource(func(context.Context) (agentapi.GuestOOMKillsState, error) {
|
||||
calls++
|
||||
if answer != nil {
|
||||
return agentapi.GuestOOMKillsState{}, answer
|
||||
}
|
||||
return agentapi.GuestOOMKillsState{Available: true, Containers: map[string]int64{"id": 2}}, nil
|
||||
}, log.New(io.Discard, "", 0), func() time.Time { return clock })
|
||||
for i := 0; i < 3; i++ {
|
||||
if got := src(); got != nil {
|
||||
t.Fatalf("an old agent: %v", got)
|
||||
}
|
||||
clock = clock.Add(30 * time.Second)
|
||||
}
|
||||
if calls != 1 {
|
||||
t.Fatalf("an old agent was asked %d times in 90 s, want 1", calls)
|
||||
}
|
||||
clock = clock.Add(hostOOMRetryAfter)
|
||||
answer = nil
|
||||
if got := src(); got["id"] != 2 || calls != 2 {
|
||||
t.Fatalf("after the back-off, an upgraded agent: %v (calls %d)", got, calls)
|
||||
}
|
||||
answer = errors.New("timeout")
|
||||
if got := src(); got != nil || calls != 3 {
|
||||
t.Fatalf("a transient failure: %v", got)
|
||||
}
|
||||
answer = nil
|
||||
if got := src(); got["id"] != 2 || calls != 4 {
|
||||
t.Fatalf("the next scan after a transient failure must ask again: %v (calls %d)", got, calls)
|
||||
}
|
||||
}
|
||||
|
||||
// The source is wired into the stack manager at startup (the seam-built-but-never-wired class).
|
||||
// RED-PROOF: delete the SetHostOOMKills line in main.go → FAILS.
|
||||
func TestR528_HostOOMSourceIsWiredAtStartup(t *testing.T) {
|
||||
lines, _, _ := slice4CallLines(t)
|
||||
if len(lines["SetHostOOMKills"]) == 0 || len(lines["hostOOMSource"]) == 0 {
|
||||
t.Fatal("the host OOM counter is never wired — the 30-s scan would read Docker's flag only (R-528)")
|
||||
}
|
||||
}
|
||||
@@ -1071,6 +1071,18 @@ func (c *Client) RemoveNetStorage(ctx context.Context, name string) error {
|
||||
// status, even on a non-2xx (so callers like FormatDisk can read a 403 refusal body). A transport or
|
||||
// envelope-parse failure is still an error; an `ok:false` business refusal is NOT (the data carries it).
|
||||
func (c *Client) postWithStatus(ctx context.Context, path string, body any) (apiResponse, int, error) {
|
||||
return c.postWithStatusOn(c.hc, ctx, path, body)
|
||||
}
|
||||
|
||||
// postWithStatusTimeout is postWithStatus with its own overall timeout (the escrow claim and confirm, R-1254). It shares
|
||||
// the pinned transport.
|
||||
func (c *Client) postWithStatusTimeout(ctx context.Context, path string, body any, timeout time.Duration) (apiResponse, int, error) {
|
||||
hc := *c.hc
|
||||
hc.Timeout = timeout
|
||||
return c.postWithStatusOn(&hc, ctx, path, body)
|
||||
}
|
||||
|
||||
func (c *Client) postWithStatusOn(hc *http.Client, ctx context.Context, path string, body any) (apiResponse, int, error) {
|
||||
var env apiResponse
|
||||
buf, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
@@ -1083,7 +1095,7 @@ func (c *Client) postWithStatus(ctx context.Context, path string, body any) (api
|
||||
req.Header.Set("Authorization", "Bearer "+c.token)
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
start := time.Now()
|
||||
resp, err := c.hc.Do(req)
|
||||
resp, err := hc.Do(req)
|
||||
if err != nil {
|
||||
logx.Debugf(c.logger, "[agentapi] POST %s failed after %dms: %v", path, time.Since(start).Milliseconds(), err)
|
||||
return env, 0, fmt.Errorf("agentapi: POST %s: %w", path, err)
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Controller-driven escrow ceremony client methods (v0.127.0, agent ≥ v0.88.0). The claim call
|
||||
@@ -108,8 +109,15 @@ type EscrowClaim struct {
|
||||
//
|
||||
// R-1151: the claim asks the agent to keep the new package until the household typed the code back
|
||||
// (`defer_upload`). An agent older than 0.160.0 ignores the body and uploads as before.
|
||||
// escrowCallTimeout bounds the claim and the confirm (R-1254 H2). The agent may take its whole upload timeout (30 s)
|
||||
// plus its question to the hub (15 s) before it answers; the client's own 15 s used to abandon the call while the
|
||||
// agent went on — a claim abandoned that way could still store the package, and its code reached nobody. 60 s
|
||||
// outlasts both (TestR1254_ClaimAndConfirmOutwaitTheAgentsUpload; the agent side: escrowUploadTimeout,
|
||||
// escrowHoldTimeout in felhom-agent internal/localapi/escrow_ceremony.go — if either grows, this must too).
|
||||
var escrowCallTimeout = 60 * time.Second
|
||||
|
||||
func (c *Client) EscrowCeremonyClaim(ctx context.Context) (EscrowClaim, int, error) {
|
||||
env, status, err := c.postWithStatus(ctx, "/escrow/ceremony/claim", map[string]bool{"defer_upload": true})
|
||||
env, status, err := c.postWithStatusTimeout(ctx, "/escrow/ceremony/claim", map[string]bool{"defer_upload": true}, escrowCallTimeout)
|
||||
if err != nil {
|
||||
return EscrowClaim{}, status, err
|
||||
}
|
||||
@@ -141,7 +149,7 @@ func (c *Client) EscrowCeremonyClaim(ctx context.Context) (EscrowClaim, int, err
|
||||
// match (still waiting), 410 the window ran out, 404 nothing waits (superseded, restarted), 502 the hub failed (still
|
||||
// waiting). The words are part of the recovery code — never logged.
|
||||
func (c *Client) EscrowCeremonyConfirm(ctx context.Context, words []string) (int, error) {
|
||||
env, status, err := c.postWithStatus(ctx, "/escrow/ceremony/confirm", map[string][]string{"words": words})
|
||||
env, status, err := c.postWithStatusTimeout(ctx, "/escrow/ceremony/confirm", map[string][]string{"words": words}, escrowCallTimeout)
|
||||
if err != nil {
|
||||
return status, err
|
||||
}
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
package agentapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// GuestOOMKillsState mirrors the agent's GET /guest/oom-kills (R-528, agent ≥ the R-528 commit): the kernel's oom_kill
|
||||
// counter of each of THIS guest's containers, read on the host — Docker's own OOMKilled flag can stay false inside an
|
||||
// LXC guest. Containers maps the full 64-hex container id to the count for the container's current run.
|
||||
// Available=false: the guest has no cgroup folder the host can read. An agent that predates the route answers 404 (a
|
||||
// *StatusError) — the controller then keeps Docker's flag only.
|
||||
type GuestOOMKillsState struct {
|
||||
VMID int `json:"vmid"`
|
||||
Available bool `json:"available"`
|
||||
Containers map[string]int64 `json:"containers"`
|
||||
}
|
||||
|
||||
// GuestOOMKills calls GET /guest/oom-kills.
|
||||
func (c *Client) GuestOOMKills(ctx context.Context) (GuestOOMKillsState, error) {
|
||||
var out GuestOOMKillsState
|
||||
body, err := c.get(ctx, "/guest/oom-kills")
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
if err := json.Unmarshal(body, &out); err != nil {
|
||||
return out, fmt.Errorf("agentapi: decode /guest/oom-kills: %w", err)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
package agentapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// OperatorReboot calls GET /guest/operator-reboot (D65 = A, agent ≥ the reboot_guest commit): the time the operator
|
||||
// rebooted THIS guest from the hub, when it is less than a day old; the zero time when there is none. An agent that
|
||||
// predates the route answers 404 (a *StatusError) — such an agent cannot reboot from the hub either, so there is
|
||||
// nothing to show and no MinAgent is needed.
|
||||
func (c *Client) OperatorReboot(ctx context.Context) (time.Time, error) {
|
||||
body, err := c.get(ctx, "/guest/operator-reboot")
|
||||
if err != nil {
|
||||
return time.Time{}, err
|
||||
}
|
||||
var out struct {
|
||||
At string `json:"at"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &out); err != nil {
|
||||
return time.Time{}, fmt.Errorf("agentapi: decode /guest/operator-reboot: %w", err)
|
||||
}
|
||||
if out.At == "" {
|
||||
return time.Time{}, nil
|
||||
}
|
||||
at, err := time.Parse(time.RFC3339, out.At)
|
||||
if err != nil {
|
||||
return time.Time{}, fmt.Errorf("agentapi: /guest/operator-reboot: bad time %q", out.At)
|
||||
}
|
||||
return at, nil
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
package agentapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-1254 H2: the agent's claim/confirm may take its whole upload timeout (30 s) plus the hub question (15 s) before it
|
||||
// answers. The controller's agent client gives up at 15 s — and a claim the controller abandons can still complete on
|
||||
// the agent: the hub then holds a package whose code nobody saw. The two calls carry their own, longer timeout.
|
||||
// RED-PROOF: send claim/confirm through the plain 15 s client again → the slow agent below times them out → FAILS.
|
||||
func TestR1254_ClaimAndConfirmOutwaitTheAgentsUpload(t *testing.T) {
|
||||
if escrowCallTimeout < 45*time.Second+10*time.Second {
|
||||
t.Fatalf("escrowCallTimeout %s must outlast the agent's upload (30 s) + hub question (15 s) with room", escrowCallTimeout)
|
||||
}
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("POST /escrow/ceremony/claim", func(w http.ResponseWriter, r *http.Request) {
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
_, _ = w.Write([]byte(`{"ok":true,"data":{"recovery_code":"a-b-c"}}`))
|
||||
})
|
||||
mux.HandleFunc("POST /escrow/ceremony/confirm", func(w http.ResponseWriter, r *http.Request) {
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
_, _ = w.Write([]byte(`{"ok":true,"data":{"uploaded":true}}`))
|
||||
})
|
||||
s := httptest.NewTLSServer(mux)
|
||||
defer s.Close()
|
||||
c := clientFor(t, s, strings.TrimPrefix(s.URL, "https://"))
|
||||
c.hc.Timeout = 100 * time.Millisecond // the client's own short timeout, scaled down
|
||||
old := escrowCallTimeout
|
||||
escrowCallTimeout = 5 * time.Second
|
||||
defer func() { escrowCallTimeout = old }()
|
||||
if claim, st, err := c.EscrowCeremonyClaim(context.Background()); err != nil || st != http.StatusOK || claim.Code != "a-b-c" {
|
||||
t.Fatalf("a slow claim was abandoned: %+v %d %v", claim, st, err)
|
||||
}
|
||||
if st, err := c.EscrowCeremonyConfirm(context.Background(), []string{"a", "c"}); err != nil || st != http.StatusOK {
|
||||
t.Fatalf("a slow confirm was abandoned: %d %v", st, err)
|
||||
}
|
||||
}
|
||||
@@ -94,7 +94,9 @@ func (r *Router) keptDataAtInstall(w http.ResponseWriter, req *http.Request, nam
|
||||
writeJSON(w, http.StatusConflict, apiResponse{OK: false, Error: data.UseOff, Data: data})
|
||||
return true
|
||||
}
|
||||
if st := r.backupMgr.RestoreStatus(); st.Running || r.backupMgr.IsRunning() {
|
||||
// R-1256 H6: every operation — a restore, a backup run, a second-drive copy — not only the first two; and the
|
||||
// load itself begins in the same step as this question (startKeptLoad → ErrRestoreBusy).
|
||||
if busy, _ := r.backupMgr.RestoreBusy(); busy != "" {
|
||||
writeJSON(w, http.StatusConflict, apiResponse{OK: false, Error: r.msgLang(lang, "api.kept.busy")})
|
||||
return true
|
||||
}
|
||||
@@ -104,7 +106,11 @@ func (r *Router) keptDataAtInstall(w http.ResponseWriter, req *http.Request, nam
|
||||
}
|
||||
r.logger.Printf("[INFO] [api] Deploy %s: USE MY KEPT DATA — loading from %s (tier %d, %s) under the kept files %v", name, src, cp.c.Tier, cp.c.Time.UTC().Format(time.RFC3339), old)
|
||||
r.dropLeftoverHold(name, "use-kept-data") // R-704: this is a new install of an app that is not installed
|
||||
r.startKeptLoad(name, hdd, cp.c, lang, nil)
|
||||
if err := r.startKeptLoad(name, hdd, cp.c, lang, nil); err != nil {
|
||||
r.logger.Printf("[WARN] [api] Deploy %s: use my kept data REFUSED at its start: %v", name, err)
|
||||
writeJSON(w, http.StatusConflict, apiResponse{OK: false, Error: r.msgLang(lang, "api.kept.busy")})
|
||||
return true
|
||||
}
|
||||
writeJSON(w, http.StatusAccepted, apiResponse{OK: true, Message: r.msgLang(lang, "kept.load.started", display)})
|
||||
return true
|
||||
default:
|
||||
@@ -120,7 +126,7 @@ func (r *Router) keptCopyName(lang string, c backup.KeptCopy) string {
|
||||
|
||||
// startKeptLoad runs the load and, when it succeeded, the app's after_load. then(ok) runs last. The
|
||||
// off-site copy goes through LoadKeptOffsite (download the unit alone, judge it, then the same restore).
|
||||
func (r *Router) startKeptLoad(name, drive string, c backup.KeptCopy, lang string, then func(ok bool)) {
|
||||
func (r *Router) startKeptLoad(name, drive string, c backup.KeptCopy, lang string, then func(ok bool)) error {
|
||||
display := name
|
||||
if st, ok := r.stackMgr.GetStack(name); ok && st.Meta.DisplayName != "" {
|
||||
display = st.Meta.DisplayName
|
||||
@@ -128,10 +134,9 @@ func (r *Router) startKeptLoad(name, drive string, c backup.KeptCopy, lang strin
|
||||
okMsg := func(error) string { return r.msgLang(lang, "kept.load.done", display) }
|
||||
failMsg := func(err error) string { return r.msgLang(lang, "kept.load.failed", display, err) }
|
||||
if c.Tier == backup.KeptTierOffsite {
|
||||
r.backupMgr.LoadKeptOffsite(name, drive, c, nil, okMsg, failMsg, r.keptAfterLoad(name, then))
|
||||
return
|
||||
return r.backupMgr.LoadKeptOffsite(name, drive, c, nil, okMsg, failMsg, r.keptAfterLoad(name, then))
|
||||
}
|
||||
r.backupMgr.LoadKeptApp(name, c.UnitDir, okMsg, failMsg, r.keptAfterLoad(name, then))
|
||||
return r.backupMgr.LoadKeptApp(name, c.UnitDir, nil, okMsg, failMsg, r.keptAfterLoad(name, then))
|
||||
}
|
||||
|
||||
// keptAfterLoad runs the app's after_load when the load succeeded, then then(ok).
|
||||
|
||||
@@ -162,7 +162,10 @@ func TestR691_InstallChoiceNamesTheOffsiteCopy(t *testing.T) {
|
||||
// copy, restic and the unit restore behind their seams — no Docker).
|
||||
// COMPANION RED-PROOF: remove the dropLeftoverHold call from keptDataAtInstall → the hold is still there
|
||||
// after the 202 and the first assertion fails; remove it from deployStack → the AST check fails.
|
||||
func TestR704_AFreshInstallDropsALeftoverHold(t *testing.T) {
|
||||
// keptInstallRig is an installable app with old data on its drive and an off-site copy of its unit; done receives
|
||||
// once per unit restore the load reaches.
|
||||
func keptInstallRig(t *testing.T) (*Router, *settings.Settings, string, chan struct{}) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
drive := filepath.Join(dir, "drive")
|
||||
cfg := &config.Config{}
|
||||
@@ -234,6 +237,11 @@ func TestR704_AFreshInstallDropsALeftoverHold(t *testing.T) {
|
||||
return backup.UnitRestoreResult{}, nil
|
||||
})
|
||||
r := &Router{stackMgr: m, backupMgr: bm, sett: sett, logger: log.New(io.Discard, "", 0)}
|
||||
return r, sett, drive, done
|
||||
}
|
||||
|
||||
func TestR704_AFreshInstallDropsALeftoverHold(t *testing.T) {
|
||||
r, sett, drive, done := keptInstallRig(t)
|
||||
|
||||
// A leftover update hold from an earlier install (the demo-hp shape): the new install drops it.
|
||||
_ = sett.SetRestoreHold(settings.RestoreHold{Stack: "cloudapp", At: "2026-09-13T19:56:32Z", Reason: settings.HoldReasonUpdateFailed})
|
||||
@@ -287,3 +295,38 @@ func TestR704_AFreshInstallDropsALeftoverHold(t *testing.T) {
|
||||
t.Fatalf("deployStack must call dropLeftoverHold before DeployStack (drop=%v deploy=%v)", dropAt, deployAt)
|
||||
}
|
||||
}
|
||||
|
||||
// R-1256 H6: the install-time „use my kept data" asks about EVERY operation — a second-drive copy too, not only a
|
||||
// restore record and the backup run — and the load begins in the same step as the question.
|
||||
// RED-PROOF: restore the RestoreStatus/IsRunning-only check and LoadKept*'s plain BeginRestoreOp → 202 and the load
|
||||
// runs beside the copy → FAILS.
|
||||
func TestR1256_H6_AKeptLoadAtInstallIsRefusedWhileASecondDriveCopyRuns(t *testing.T) {
|
||||
r, _, drive, done := keptInstallRig(t)
|
||||
end := backup.MarkTier2CopyForTest("someapp")
|
||||
w := httptest.NewRecorder()
|
||||
req := httptest.NewRequest(http.MethodPost, "/api/stacks/cloudapp/deploy?lang=en", nil)
|
||||
r.keptDataAtInstall(w, req, "cloudapp", drive, "use")
|
||||
end()
|
||||
if w.Code != http.StatusConflict {
|
||||
t.Fatalf("a load during a second-drive copy: code %d, want 409 — body %s", w.Code, w.Body.String())
|
||||
}
|
||||
select {
|
||||
case <-done:
|
||||
t.Fatal("the load ran beside the second-drive copy")
|
||||
case <-time.After(300 * time.Millisecond):
|
||||
}
|
||||
if st := r.backupMgr.RestoreStatus(); st.Running {
|
||||
t.Fatal("a refused load left a restore op recorded")
|
||||
}
|
||||
// The copy ended: the same press loads.
|
||||
w = httptest.NewRecorder()
|
||||
r.keptDataAtInstall(w, httptest.NewRequest(http.MethodPost, "/api/stacks/cloudapp/deploy?lang=en", nil), "cloudapp", drive, "use")
|
||||
if w.Code != http.StatusAccepted {
|
||||
t.Fatalf("after the copy: code %d — body %s", w.Code, w.Body.String())
|
||||
}
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Fatal("the load never reached the unit restore")
|
||||
}
|
||||
}
|
||||
@@ -771,6 +771,12 @@ func (r *Router) actionStack(w http.ResponseWriter, req *http.Request, action, n
|
||||
// above, so nothing is recorded for an action that was never going to happen.
|
||||
if desired, ok := desiredStateForAction(action); ok {
|
||||
if derr := r.stackMgr.SetDesiredState(name, desired); derr != nil {
|
||||
if errors.Is(derr, stacks.ErrStackNotFound) {
|
||||
// no app by that name on this box (e.g. the bootstrap-managed controller): a 404, not a save failure
|
||||
r.logger.Printf("[WARN] [api] %s for %s refused: no such app", action, name)
|
||||
writeJSON(w, stackOpStatusFor(derr), apiResponse{OK: false, Error: r.errText(req, derr)})
|
||||
return
|
||||
}
|
||||
r.logger.Printf("[ERROR] [api] %s for %s refused: could not record desired state: %v", action, name, derr)
|
||||
writeJSON(w, http.StatusInternalServerError, apiResponse{
|
||||
OK: false,
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Found walking the capability map's B2 („App lifecycle") on 9202, 2026-10-28 night: a Stop of a name that is no app
|
||||
// on this box (`felhom-controller` — bootstrap-managed, not a stack) answered 500 „…az alkalmazás beállításai nem
|
||||
// menthetők." — the desired-state write's „not found" read as a save failure. It is a 404 now, before any intent is
|
||||
// recorded. RED-PROOF: drop the ErrStackNotFound branch in actionStack → 500 → FAILS.
|
||||
func TestAnActionOnANameThatIsNoAppIsA404(t *testing.T) {
|
||||
r, _, _, _ := newSlice4Router(t)
|
||||
for _, action := range []string{"stop", "start"} {
|
||||
w := httptest.NewRecorder()
|
||||
r.actionStack(w, httptest.NewRequest("POST", "/api/stacks/nosuchapp/"+action, nil), action, "nosuchapp")
|
||||
var resp apiResponse
|
||||
_ = json.Unmarshal(w.Body.Bytes(), &resp)
|
||||
if w.Code != http.StatusNotFound {
|
||||
t.Fatalf("%s of a name that is no app: %d %q — want 404", action, w.Code, resp.Error)
|
||||
}
|
||||
if strings.Contains(strings.ToLower(resp.Error), "mentheto") || strings.Contains(resp.Error, "menthetők") {
|
||||
t.Fatalf("%s: the answer still blames a save: %q", action, resp.Error)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -137,6 +137,10 @@ type AppStopGuard struct {
|
||||
// whether the drive a copy lives on is bound. nil = the pre-R-1251 behaviour.
|
||||
holds AppStopHolds
|
||||
undoReachable func(stack, dir string) bool
|
||||
// heldNotify and heldPending (R-1348, appstop_putback_wait.go): the operator mail for a failed put-back. Recover runs
|
||||
// before the notifier exists, so a hold written then waits in heldPending until SetHeldNotify.
|
||||
heldNotify func(stack string)
|
||||
heldPending []string
|
||||
|
||||
// liveOp (R-1253) is the operation of THIS process that holds the marker now (set by Begin, cleared by End). The
|
||||
// marker has one slot; BeginExclusive refuses to replace a live one. A marker left by a crash has no live op.
|
||||
|
||||
@@ -119,15 +119,49 @@ func (g *AppStopGuard) holdPutBackFailed(stack, dir string, cause error) bool {
|
||||
return false
|
||||
}
|
||||
h := settings.RestoreHold{Stack: stack, At: g.now().UTC().Format(time.RFC3339), Reason: settings.HoldReasonRestoreMixed,
|
||||
ReplayError: "interrupted restore, put-back failed: " + cause.Error(), SafetyDump: dir}
|
||||
ReplayError: "interrupted restore, put-back failed: " + cause.Error(), SafetyDump: dir,
|
||||
MixedCause: settings.MixedCauseInterruptedPutBack, FailedStep: "putting the interrupted restore's copy back failed: " + cause.Error()}
|
||||
if err := g.holds.SetRestoreHold(h); err != nil {
|
||||
g.logger.Printf("[ERROR] [appstop] %s: the failed put-back could not be recorded as a hold: %v — the marker is kept", stack, err)
|
||||
return false
|
||||
}
|
||||
g.logger.Printf("[ERROR] [appstop] %s: the interrupted restore could NOT be put back (%v) — the app is HELD for support, the copy stays at %s (R-1251 H2)", stack, cause, dir)
|
||||
g.notifyHeld(stack)
|
||||
return true
|
||||
}
|
||||
|
||||
// SetHeldNotify wires the operator mail for a failed put-back (R-1348; main.go: the R-1340 sentence from the hold record)
|
||||
// and hands over what Recover held before the notifier existed. Each held app is mailed once. Pinned by TestR1348_* and
|
||||
// TestR1348_HeldNotifyIsWiredAtStartup.
|
||||
func (g *AppStopGuard) SetHeldNotify(fn func(stack string)) {
|
||||
if g == nil {
|
||||
return
|
||||
}
|
||||
g.liveMu.Lock()
|
||||
g.heldNotify = fn
|
||||
waiting := g.heldPending
|
||||
g.heldPending = nil
|
||||
g.liveMu.Unlock()
|
||||
if fn == nil {
|
||||
return
|
||||
}
|
||||
for _, stack := range waiting {
|
||||
fn(stack)
|
||||
}
|
||||
}
|
||||
|
||||
func (g *AppStopGuard) notifyHeld(stack string) {
|
||||
g.liveMu.Lock()
|
||||
fn := g.heldNotify
|
||||
if fn == nil {
|
||||
g.heldPending = append(g.heldPending, stack)
|
||||
}
|
||||
g.liveMu.Unlock()
|
||||
if fn != nil {
|
||||
fn(stack)
|
||||
}
|
||||
}
|
||||
|
||||
// clearPutBackPending lifts this app's waiting-put-back hold, and only that kind.
|
||||
func (g *AppStopGuard) clearPutBackPending(stack string) {
|
||||
if g.holds == nil {
|
||||
|
||||
@@ -219,6 +219,9 @@ type Manager struct {
|
||||
// undoCopyRemover (R-671, v0.272.0) deletes an app's leftover undo copies — stacks.Manager.RemoveUndoCopies,
|
||||
// wired in main.go (SetUndoCopyRemover). nil-safe: without it the copies stay, as before.
|
||||
undoCopyRemover func(stackName string) int
|
||||
// updateHeldForgetter (R-1347) forgets the stack manager's „last update ended held" flag when a restore lifts the update
|
||||
// hold — stacks.Manager.ForgetUpdateHeld, wired in main.go (SetUpdateHeldForgetter). nil-safe.
|
||||
updateHeldForgetter func(stackName string) bool
|
||||
|
||||
// R-475 update-precondition seams, one per tier. Nil → the real Tier2UnitRestorePoint /
|
||||
// ListRestorePoints / OffsiteSnapshotTimes. They let a test reach Tier 1 and Tier 3 without a drive
|
||||
@@ -340,6 +343,9 @@ type Manager struct {
|
||||
// R-550: persistence of the above (restore_record.go) and the per-app interrupted notices.
|
||||
opRecordPath string
|
||||
opInterrupted map[string]RestoreOpResult
|
||||
// R-1342: the restore copies a stable-start watch still holds (copy dir → true), guarded by mu. While one is here the
|
||||
// copy is the way back if the restored app turns out unstable: „Törlés most" refuses it (restore_copies.go).
|
||||
watching map[string]bool
|
||||
// R-519: the app-data run record (run_record.go) and the start of the last run a stop cut off (zero = none).
|
||||
runRecordPath string
|
||||
runInterrupted time.Time
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// D62 (operator ruling 2026-10-29): an unused restore copy is kept 30 days; the household sees each copy (app, the
|
||||
// restore's date, size, the day it goes) and may delete one earlier. A copy a hold or a pending put-back needs is never
|
||||
// deleted, by the sweep or by hand.
|
||||
|
||||
// RED-PROOF: with the age at 14 days the 20-day copy is swept → FAILS; at 30 days only the 31-day copy goes.
|
||||
func TestD62_AnUnusedCopyIsKept30Days(t *testing.T) {
|
||||
m, _ := newOffboxManager(t)
|
||||
now := time.Now()
|
||||
day29 := r1256Copy(t, m, "a", 29*24*time.Hour, now)
|
||||
day20 := r1256Copy(t, m, "b", 20*24*time.Hour, now)
|
||||
day31 := r1256Copy(t, m, "c", 31*24*time.Hour, now)
|
||||
got := m.SweepUnusedRestoreCopies(now)
|
||||
if len(got) != 1 || got[0] != day31 {
|
||||
t.Fatalf("swept %v — want only the 31-day copy %s", got, day31)
|
||||
}
|
||||
if !r1256Exists(day29) || !r1256Exists(day20) {
|
||||
t.Fatal("a copy younger than 30 days was swept")
|
||||
}
|
||||
if restoreCopyUnusedAge != 30*24*time.Hour {
|
||||
t.Fatalf("restoreCopyUnusedAge = %s, D62 says 30 days", restoreCopyUnusedAge)
|
||||
}
|
||||
}
|
||||
|
||||
// The list shows each copy with its restore date, its size and the day it goes; a needed copy has no such day and
|
||||
// says why. RED-PROOF: drop the hold check from restoreCopyNeededFor → the held copy is offered → FAILS.
|
||||
func TestD62_TheListShowsDateSizeAndWhyACopyIsNeeded(t *testing.T) {
|
||||
m, sett := newOffboxManager(t)
|
||||
now := time.Now().UTC().Truncate(time.Second)
|
||||
free := r1256Copy(t, m, "free", 2*24*time.Hour, now)
|
||||
r1256Copy(t, m, "held", 2*24*time.Hour, now)
|
||||
r1256Copy(t, m, "waiting", 2*24*time.Hour, now)
|
||||
r1256Copy(t, m, "named", 2*24*time.Hour, now)
|
||||
if err := sett.SetRestoreHold(settings.RestoreHold{Stack: "held", At: now.Format(time.RFC3339), Reason: settings.HoldReasonRestoreMixed}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := sett.SetRestoreHold(settings.RestoreHold{Stack: "waiting", At: now.Format(time.RFC3339), Reason: settings.HoldReasonPutBackPending}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := m.appStop.Begin("unit-restore:named", ReasonUnitRestore, []string{"named"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
list := m.RestoreCopies()
|
||||
if len(list) != 4 {
|
||||
t.Fatalf("listed %d copies, want 4: %+v", len(list), list)
|
||||
}
|
||||
want := map[string]string{"free": "", "held": RestoreCopyNeededHold, "waiting": RestoreCopyNeededPutBack, "named": RestoreCopyNeededPutBack}
|
||||
for _, c := range list {
|
||||
if c.Needed != want[c.App] {
|
||||
t.Errorf("%s: needed %q, want %q", c.App, c.Needed, want[c.App])
|
||||
}
|
||||
if !c.At.Equal(now.Add(-2 * 24 * time.Hour)) {
|
||||
t.Errorf("%s: restore date %s, want the stamp's %s", c.App, c.At, now.Add(-2*24*time.Hour))
|
||||
}
|
||||
if c.SizeBytes != int64(len("LIVE")) {
|
||||
t.Errorf("%s: size %d", c.App, c.SizeBytes)
|
||||
}
|
||||
if c.Needed == "" && !c.DeleteOn.Equal(c.At.Add(30*24*time.Hour)) {
|
||||
t.Errorf("%s: deleted on %s, want 30 days after the restore", c.App, c.DeleteOn)
|
||||
}
|
||||
if c.Needed != "" && !c.DeleteOn.IsZero() {
|
||||
t.Errorf("%s: a needed copy shows a delete day %s — the sweep never takes it", c.App, c.DeleteOn)
|
||||
}
|
||||
if c.App == "free" && c.Path != free {
|
||||
t.Errorf("path %s, want %s", c.Path, free)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A held or waiting copy cannot be deleted by hand. RED-PROOF: drop the Needed check in DeleteRestoreCopy → FAILS.
|
||||
func TestD62_AHeldCopyCannotBeDeleted(t *testing.T) {
|
||||
m, sett := newOffboxManager(t)
|
||||
now := time.Now()
|
||||
for _, tc := range []struct{ app, reason string }{
|
||||
{"held", settings.HoldReasonRestoreMixed},
|
||||
{"waiting", settings.HoldReasonPutBackPending},
|
||||
{"failed", settings.HoldReasonRestoreFailed},
|
||||
} {
|
||||
dir := r1256Copy(t, m, tc.app, time.Hour, now)
|
||||
if err := sett.SetRestoreHold(settings.RestoreHold{Stack: tc.app, At: now.Format(time.RFC3339), Reason: tc.reason}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := m.DeleteRestoreCopy(tc.app, filepath.Base(dir)); !errors.Is(err, ErrRestoreCopyNeeded) {
|
||||
t.Errorf("%s: err %v, want ErrRestoreCopyNeeded", tc.app, err)
|
||||
}
|
||||
if !r1256Exists(dir) {
|
||||
t.Errorf("%s: a held copy was deleted", tc.app)
|
||||
}
|
||||
}
|
||||
// a copy a cut restore names (the marker) is needed too
|
||||
named := r1256Copy(t, m, "named", time.Hour, now)
|
||||
if err := m.appStop.Begin("unit-restore:named", ReasonUnitRestore, []string{"named"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := m.DeleteRestoreCopy("named", filepath.Base(named)); !errors.Is(err, ErrRestoreCopyNeeded) || !r1256Exists(named) {
|
||||
t.Errorf("a copy the marker names: err %v, exists %v", err, r1256Exists(named))
|
||||
}
|
||||
// an unreadable marker offers nothing
|
||||
mustWrite(t, m.appStop.path, "{not json")
|
||||
free := r1256Copy(t, m, "free", time.Hour, now)
|
||||
if _, err := m.DeleteRestoreCopy("free", filepath.Base(free)); !errors.Is(err, ErrRestoreCopyNeeded) || !r1256Exists(free) {
|
||||
t.Errorf("an unreadable marker: err %v, exists %v", err, r1256Exists(free))
|
||||
}
|
||||
}
|
||||
|
||||
// A delete removes ONLY that copy: the app's other copy, another app's copy and anything outside the copy tree stay;
|
||||
// a name that is not one path element, or a link in the copy's place, is refused.
|
||||
// RED-PROOF: remove the validCopyName check → "../.." reaches outside → FAILS; remove the Lstat link check → the
|
||||
// link is followed/removed → FAILS.
|
||||
func TestD62_ADeleteRemovesOnlyThatCopy(t *testing.T) {
|
||||
m, _ := newOffboxManager(t)
|
||||
now := time.Now()
|
||||
one := r1256Copy(t, m, "app", 3*time.Hour, now)
|
||||
two := r1256Copy(t, m, "app", 2*time.Hour, now)
|
||||
other := r1256Copy(t, m, "other", 3*time.Hour, now)
|
||||
got, err := m.DeleteRestoreCopy("app", filepath.Base(one))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Path != one || got.SizeBytes != int64(len("LIVE")) || got.App != "app" {
|
||||
t.Fatalf("returned %+v", got)
|
||||
}
|
||||
if r1256Exists(one) || !r1256Exists(two) || !r1256Exists(other) {
|
||||
t.Fatalf("one gone=%v, two kept=%v, other kept=%v", !r1256Exists(one), r1256Exists(two), r1256Exists(other))
|
||||
}
|
||||
// path guard
|
||||
outside := filepath.Join(m.namespaceRoot(m.systemDataPath), "backups", "primary", "keep.txt")
|
||||
mustWrite(t, outside, "UNIT")
|
||||
for _, bad := range [][2]string{{"app", ".."}, {"..", "primary"}, {"app", "../../primary"}, {"", "x"}, {"app", ""}, {"app/x", "y"}} {
|
||||
if _, err := m.DeleteRestoreCopy(bad[0], bad[1]); !errors.Is(err, ErrRestoreCopyNotFound) {
|
||||
t.Errorf("DeleteRestoreCopy(%q,%q) = %v, want not found", bad[0], bad[1], err)
|
||||
}
|
||||
}
|
||||
if !r1256Exists(outside) || !r1256Exists(two) {
|
||||
t.Fatal("a bad name reached outside the copy")
|
||||
}
|
||||
// a link in a copy's place is not a copy
|
||||
target := t.TempDir()
|
||||
mustWrite(t, filepath.Join(target, "precious"), "X")
|
||||
link := filepath.Join(filepath.Dir(two), "20260101T000000Z")
|
||||
if err := os.Symlink(target, link); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := m.DeleteRestoreCopy("app", "20260101T000000Z"); !errors.Is(err, ErrRestoreCopyNotFound) {
|
||||
t.Errorf("a link: %v, want not found", err)
|
||||
}
|
||||
if !r1256Exists(filepath.Join(target, "precious")) {
|
||||
t.Fatal("the link's target was removed")
|
||||
}
|
||||
if _, err := m.DeleteRestoreCopy("app", "20990101T000000Z"); !errors.Is(err, ErrRestoreCopyNotFound) {
|
||||
t.Errorf("a missing copy: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// While a backup or restore runs, a delete is refused and nothing moves. RED-PROOF: drop the busy check → FAILS.
|
||||
func TestD62_NoDeleteWhileAnOperationRuns(t *testing.T) {
|
||||
m, _ := newOffboxManager(t)
|
||||
c := r1256Copy(t, m, "app", time.Hour, time.Now())
|
||||
m.BeginRestoreOp("restore", "other")
|
||||
if _, err := m.DeleteRestoreCopy("app", filepath.Base(c)); !errors.Is(err, ErrRestoreCopyBusy) || !r1256Exists(c) {
|
||||
t.Fatalf("during a restore: err %v, exists %v", err, r1256Exists(c))
|
||||
}
|
||||
m.EndRestoreOp(true, "ok")
|
||||
if err := m.acquireRunning(); err != nil {
|
||||
t.Fatalf("could not take the backup single-flight: %v", err)
|
||||
}
|
||||
if _, err := m.DeleteRestoreCopy("app", filepath.Base(c)); !errors.Is(err, ErrRestoreCopyBusy) || !r1256Exists(c) {
|
||||
t.Fatalf("during a backup run: err %v, exists %v", err, r1256Exists(c))
|
||||
}
|
||||
m.ReleaseRunningForTest()
|
||||
if _, err := m.DeleteRestoreCopy("app", filepath.Base(c)); err != nil || r1256Exists(c) {
|
||||
t.Fatalf("control: with nothing running the delete goes through: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"errors"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// ── D64 = B (operator ruling 2026-10-29 12:10) + R-1343: the operator clears a hold IN THE RUNNING CONTROLLER ──────
|
||||
//
|
||||
// One door for both of the operator's routes — the hub's „Clear hold" (operator action clear_hold) and the box command
|
||||
// `--clear-restore-hold`, which now asks the running controller over its local socket (cmd/controller/holdctl.go).
|
||||
//
|
||||
// WHY IN THIS PROCESS (R-1343, measured 2026-10-29 on Tester 1): the CLI used to run as a SECOND process that edited
|
||||
// settings.json while the running controller kept the hold in its own in-memory Settings — and that controller's next
|
||||
// save (after a hub report) wrote the hold back. Clearing through the running Settings removes the hold from memory
|
||||
// AND saves the file at once, so no later save of any component can bring it back.
|
||||
// Pinned by TestD64_ClearHoldStaysGoneAfterALaterSave (cmd/controller).
|
||||
//
|
||||
// EVERY REASON (D64 = B): the hold's reason does not matter — R-379 now reads „a person — on the box, or the operator
|
||||
// on the hub". One reason needs a second act: a put-back WAITING for its drive (HoldReasonPutBackPending) is also
|
||||
// recorded in the app-stop marker, and a controller start with the drive bound would pour that old copy back over
|
||||
// whatever the app wrote after the clear. So the clear also drops the app from the marker; the copy stays on disk
|
||||
// (nothing is deleted). It runs under resumeMu, so it can never interleave with a put-back of the same app.
|
||||
// Pinned by TestD64_ClearPutBackPendingDropsTheMarkerEntry.
|
||||
|
||||
// ErrHoldBadStack: the argument is not a valid stack name. ErrNotHeld: the app has no hold. Both are refusals —
|
||||
// nothing was changed.
|
||||
var (
|
||||
ErrHoldBadStack = errors.New("not a valid app name")
|
||||
ErrNotHeld = errors.New("this app has no hold")
|
||||
)
|
||||
|
||||
// IsValidHoldStack reports whether s may name an app for the hold commands (the backup package's strict
|
||||
// isSafeStackName: letters, digits, '-' and '_', at most 64).
|
||||
func IsValidHoldStack(s string) bool { return isSafeStackName(s) }
|
||||
|
||||
// ClearHoldByOperator lifts stack's hold, whatever its reason, in the hold store the running controller uses, and
|
||||
// returns the hold it lifted. A waiting put-back is dropped from the app-stop marker too (see above). Refusals
|
||||
// (ErrHoldBadStack, ErrNotHeld) change nothing.
|
||||
func (g *AppStopGuard) ClearHoldByOperator(stack string) (settings.RestoreHold, error) {
|
||||
return g.ClearHoldByOperatorThen(stack, nil)
|
||||
}
|
||||
|
||||
// ClearHoldByOperatorThen is ClearHoldByOperator with beforeLift run once the hold is FOUND and before it is lifted,
|
||||
// under the same lock (R-1350: the household's „the stop is wanted" is recorded before the hold stops keeping the app out
|
||||
// of the dead-app scan). beforeLift is not run on a refusal.
|
||||
func (g *AppStopGuard) ClearHoldByOperatorThen(stack string, beforeLift func()) (settings.RestoreHold, error) {
|
||||
if !isSafeStackName(stack) {
|
||||
return settings.RestoreHold{}, ErrHoldBadStack
|
||||
}
|
||||
if g == nil || g.holds == nil {
|
||||
return settings.RestoreHold{}, errors.New("no hold store is wired on this box")
|
||||
}
|
||||
resumeMu.Lock()
|
||||
defer resumeMu.Unlock()
|
||||
h, ok := g.holds.GetRestoreHold(stack)
|
||||
if !ok {
|
||||
return settings.RestoreHold{}, ErrNotHeld
|
||||
}
|
||||
if beforeLift != nil {
|
||||
beforeLift()
|
||||
}
|
||||
cleared, err := g.holds.ClearRestoreHold(stack)
|
||||
if err != nil {
|
||||
return h, err
|
||||
}
|
||||
if !cleared {
|
||||
return settings.RestoreHold{}, ErrNotHeld
|
||||
}
|
||||
if h.Reason == settings.HoldReasonPutBackPending {
|
||||
g.dropFromMarker(stack)
|
||||
g.logger.Printf("[WARN] [appstop] %s: its waiting put-back is DROPPED with the hold — the copy stays on disk at %s and is never poured back by itself", stack, h.SafetyDump)
|
||||
}
|
||||
return h, nil
|
||||
}
|
||||
@@ -301,8 +301,12 @@ func (m *Manager) downloadKeptOffsiteUnit(ctx context.Context, app, drive string
|
||||
// LoadKeptOffsite is LoadKeptApp from the off-site copy c: download the unit alone, judge it, then
|
||||
// prepare (nil = nothing; a dated folder's files move back here), then the one unit restore, then
|
||||
// after(ok). A download failure or a refusal never reaches prepare, so the kept files stay as they were.
|
||||
func (m *Manager) LoadKeptOffsite(app, drive string, c KeptCopy, prepare func() error, okMsg, failMsg func(err error) string, after func(ok bool)) {
|
||||
m.BeginRestoreOp("restore", app)
|
||||
// ErrRestoreBusy (nothing started, after not called) when another operation runs (R-1256 H6).
|
||||
func (m *Manager) LoadKeptOffsite(app, drive string, c KeptCopy, prepare func() error, okMsg, failMsg func(err error) string, after func(ok bool)) error {
|
||||
if busy, _ := m.TryBeginRestoreOp("restore", app); busy != "" {
|
||||
m.logger.Printf("[WARN] [backup] kept load %s REFUSED: a %s operation runs (R-1256)", app, busy)
|
||||
return ErrRestoreBusy
|
||||
}
|
||||
go func() {
|
||||
start := time.Now()
|
||||
fail := func(err error) {
|
||||
@@ -345,6 +349,7 @@ func (m *Manager) LoadKeptOffsite(app, drive string, c KeptCopy, prepare func()
|
||||
after(true)
|
||||
}
|
||||
}()
|
||||
return nil
|
||||
}
|
||||
|
||||
// RemovedUnitOnDrive is the start-fresh hook (stacks.SetKeptUnitFinder): the removed app's OWN unit when
|
||||
@@ -370,11 +375,26 @@ func (m *Manager) PrimaryUnitHome(app, drive string) string {
|
||||
|
||||
// LoadKeptApp runs the load as a restore operation the backup pages already follow (the poll banner):
|
||||
// Begin → RestoreFromRecoveryUnitAt(app, unitDir) → End, then after(ok) in the same goroutine (the
|
||||
// after_load command, the kept folder's tidy-up). The caller has checked RestoreStatus/IsRunning.
|
||||
func (m *Manager) LoadKeptApp(app, unitDir string, okMsg, failMsg func(err error) string, after func(ok bool)) {
|
||||
m.BeginRestoreOp("restore", app)
|
||||
// after_load command, the kept folder's tidy-up). prepare (nil = nothing) runs first, inside the operation — a dated
|
||||
// folder's files move back there. R-1256 H6: the op is begun in the same step as the check for every other operation;
|
||||
// a refusal returns ErrRestoreBusy with nothing started and nothing moved.
|
||||
func (m *Manager) LoadKeptApp(app, unitDir string, prepare func() error, okMsg, failMsg func(err error) string, after func(ok bool)) error {
|
||||
if busy, _ := m.TryBeginRestoreOp("restore", app); busy != "" {
|
||||
m.logger.Printf("[WARN] [backup] kept load %s REFUSED: a %s operation runs (R-1256)", app, busy)
|
||||
return ErrRestoreBusy
|
||||
}
|
||||
go func() {
|
||||
start := time.Now()
|
||||
if prepare != nil {
|
||||
if perr := prepare(); perr != nil {
|
||||
m.logger.Printf("[ERROR] [backup] kept load %s: preparing the kept files FAILED: %v — nothing was loaded", app, perr)
|
||||
m.EndRestoreOp(false, failMsg(perr))
|
||||
if after != nil {
|
||||
after(false)
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
res, err := m.keptUnitRestore()(app, unitDir)
|
||||
if errors.Is(err, ErrRestoreNotStarted) {
|
||||
m.keptLoadNotStarted(app, err, after)
|
||||
@@ -395,6 +415,7 @@ func (m *Manager) LoadKeptApp(app, unitDir string, okMsg, failMsg func(err error
|
||||
after(true)
|
||||
}
|
||||
}()
|
||||
return nil
|
||||
}
|
||||
|
||||
// keptLoadNotStarted (R-1270): the kept data is loaded but the app did not start in time. Never „done": the record says
|
||||
|
||||
@@ -425,9 +425,12 @@ func (m *Manager) holdAppAfterFailedRollback(stack string, replayErr, rollbackEr
|
||||
m.logger.Printf("[ERROR] [offbox] %s: cannot persist the restore hold — no settings wired; the app is stopped but NOTHING will refuse a restart", stack)
|
||||
return
|
||||
}
|
||||
// R-1340: this branch is only ever an off-site restore's database replay (offbox_reconstitute's rollback).
|
||||
h := settings.RestoreHold{
|
||||
Stack: stack,
|
||||
At: time.Now().UTC().Format(time.RFC3339),
|
||||
RestoreKind: settings.RestoreKindOffsite,
|
||||
DBReplay: true,
|
||||
}
|
||||
if replayErr != nil {
|
||||
h.ReplayError = replayErr.Error()
|
||||
@@ -447,6 +450,15 @@ func (m *Manager) holdAppAfterFailedRollback(stack string, replayErr, rollbackEr
|
||||
}
|
||||
}
|
||||
|
||||
// mixedHoldWhy (R-1340) is what a restore_mixed hold records about its cause (settings.RestoreHold's R-1340 fields).
|
||||
type mixedHoldWhy struct {
|
||||
kind string // settings.RestoreKind*
|
||||
cause string // settings.MixedCause*
|
||||
step string // the step that failed, with its error
|
||||
dbReplay bool // the restore's own failure was its database replay
|
||||
copyDir string // where the copy of the live volumes is kept; "" = none kept
|
||||
}
|
||||
|
||||
// liveDefinition is the app's definition as it ran before an off-site restore wrote the snapshot's
|
||||
// (R-893): a copy of its compose files in a temporary folder, and its full env held in memory only.
|
||||
type liveDefinition struct {
|
||||
@@ -511,7 +523,11 @@ func (m *Manager) captureLiveDefinition(stack string) (*liveDefinition, error) {
|
||||
// written back (when the snapshot's was written), the database service is stopped again, and the app is
|
||||
// held with HoldReasonRestoreMixed. The operator is notified with no rollback error — that is how the
|
||||
// notification tells this case from R-379's double failure.
|
||||
func (m *Manager) holdAppAfterMixedRestore(stack string, replayErr error, live *liveDefinition) {
|
||||
//
|
||||
// why (R-1340) is what the hold records about its cause, so the operator's mail tells the real story — this hold is
|
||||
// written by three different branches now (a definition write, a put-back that failed at one of its steps), not
|
||||
// only by option C's database replay.
|
||||
func (m *Manager) holdAppAfterMixedRestore(stack string, replayErr error, live *liveDefinition, why mixedHoldWhy) {
|
||||
// The hold is PERSISTED FIRST (security review 2026-10-08): a controller that dies between the stop and
|
||||
// the hold would otherwise restart the app from the R-166 marker with nothing refusing it. When the hold
|
||||
// cannot be persisted, the app-stop marker is KEPT, so nothing reads the window as finished.
|
||||
@@ -519,7 +535,8 @@ func (m *Manager) holdAppAfterMixedRestore(stack string, replayErr error, live *
|
||||
if m.settings == nil {
|
||||
m.logger.Printf("[ERROR] [offbox] %s: cannot persist the restore hold — no settings wired; the app is stopped but NOTHING will refuse a restart", stack)
|
||||
} else {
|
||||
h := settings.RestoreHold{Stack: stack, At: time.Now().UTC().Format(time.RFC3339), Reason: settings.HoldReasonRestoreMixed}
|
||||
h := settings.RestoreHold{Stack: stack, At: time.Now().UTC().Format(time.RFC3339), Reason: settings.HoldReasonRestoreMixed,
|
||||
RestoreKind: why.kind, MixedCause: why.cause, FailedStep: why.step, DBReplay: why.dbReplay, SafetyDump: why.copyDir}
|
||||
if replayErr != nil {
|
||||
h.ReplayError = replayErr.Error()
|
||||
}
|
||||
@@ -548,6 +565,26 @@ func (m *Manager) holdAppAfterMixedRestore(stack string, replayErr error, live *
|
||||
}
|
||||
}
|
||||
|
||||
// restartAtLiveDefinition (R-1256 H4) is the failure branch of the definition write, taken while no data has moved.
|
||||
// The writer writes the compose before the app's settings, so a failure can leave the snapshot's compose in the
|
||||
// stack dir: the LIVE definition is written back first, and only then is the app started. When the write-back fails
|
||||
// too, the app is held stopped (restore_mixed) — a start would run the snapshot's definition over the live data.
|
||||
// Pinned by TestR1256_H4_*.
|
||||
func (m *Manager) restartAtLiveDefinition(stack string, live *liveDefinition, cause error, restartStack func() error) {
|
||||
if live != nil {
|
||||
if err := m.stackProvider.RecreateStackDefinitionFromUnit(stack, live.dir, live.env); err != nil {
|
||||
m.logger.Printf("[ERROR] [offbox] %s: the definition write failed (%v) and writing the live definition back failed too: %v — holding the app stopped", stack, cause, err)
|
||||
m.holdAppAfterMixedRestore(stack, cause, nil, mixedHoldWhy{kind: settings.RestoreKindOffsite, cause: settings.MixedCauseDefinitionWrite,
|
||||
step: "writing the live app definition back failed: " + err.Error()})
|
||||
return
|
||||
}
|
||||
m.logger.Printf("[INFO] [offbox] %s: the definition write failed (%v); the live definition is written back before the start", stack, cause)
|
||||
}
|
||||
if sErr := restartStack(); sErr != nil {
|
||||
m.logger.Printf("[WARN] [offbox] %s: restart after a failed definition write also failed: %v", stack, sErr)
|
||||
}
|
||||
}
|
||||
|
||||
// SetRestoreHoldNotify wires the operator notification for a held app (cmd/controller/main.go).
|
||||
func (m *Manager) SetRestoreHoldNotify(fn func(stack string, replayErr, rollbackErr error)) {
|
||||
m.restoreHoldNotify = fn
|
||||
@@ -1005,16 +1042,12 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack
|
||||
// yet — a failure here restarts the app as it was.
|
||||
if defineFromSnapshot {
|
||||
if err := m.stackProvider.RecreateStackDefinitionFromUnit(stack, scratchCompose, snapEnv); err != nil {
|
||||
if sErr := restartStack(); sErr != nil {
|
||||
m.logger.Printf("[WARN] [offbox] %s: restart after a failed definition write also failed: %v", stack, sErr)
|
||||
}
|
||||
m.restartAtLiveDefinition(stack, liveDef, err, restartStack)
|
||||
return res, fmt.Errorf("restoring %s: writing the snapshot's definition failed: %w", stack, err)
|
||||
}
|
||||
} else if len(credOverlay) > 0 {
|
||||
if err := m.stackProvider.(secretValueSetter).SetStackSecretValues(stack, credOverlay); err != nil {
|
||||
if sErr := restartStack(); sErr != nil {
|
||||
m.logger.Printf("[WARN] [offbox] %s: restart after a failed password write also failed: %v", stack, sErr)
|
||||
}
|
||||
m.restartAtLiveDefinition(stack, liveDef, err, restartStack)
|
||||
return res, fmt.Errorf("restoring %s: writing the backup's database password failed: %w", stack, err)
|
||||
}
|
||||
m.logger.Printf("[INFO] [offbox] %s: the backup's database password(s) %v written into the app's settings before the data (R-1106)", stack, credNames(credOverlay))
|
||||
@@ -1045,7 +1078,7 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack
|
||||
// R-893 option A: no volume has moved yet, so only the live definition is written back and the app starts.
|
||||
// The failing copy may have placed part of its files, so the sentence names placed files (+1).
|
||||
if definitionMoved {
|
||||
return res, m.putBackAfterFailedRestore(stack, cErr, liveDef, undo, false, res.FilesPlaced+1, restartStack, "err.backup.restore_failed_held_mixed")
|
||||
return res, m.putBackAfterFailedRestore(stack, cErr, liveDef, undo, false, res.FilesPlaced+1, restartStack, "err.backup.restore_failed_held_mixed", settings.RestoreKindOffsite)
|
||||
}
|
||||
// Nothing moved yet: best-effort bring-up — a failed restore must not also be an outage.
|
||||
if sErr := restartStack(); sErr != nil {
|
||||
@@ -1080,7 +1113,7 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack
|
||||
// R-893 option A: the replay removes a volume BEFORE it recreates it, so even nVols == 0 may have lost one —
|
||||
// with a copy, it is always poured back; with no copy the old rule decides (and holds when one moved).
|
||||
if definitionMoved || nVols > 0 || undo != nil {
|
||||
return res, m.putBackAfterFailedRestore(stack, vErr, liveDef, undo, nVols > 0 || undo != nil, res.FilesPlaced, restartStack, "err.backup.restore_failed_held_mixed")
|
||||
return res, m.putBackAfterFailedRestore(stack, vErr, liveDef, undo, nVols > 0 || undo != nil, res.FilesPlaced, restartStack, "err.backup.restore_failed_held_mixed", settings.RestoreKindOffsite)
|
||||
}
|
||||
// Nothing replaced yet: bring the app back up rather than leaving an outage, then surface it.
|
||||
if sErr := restartStack(); sErr != nil {
|
||||
@@ -1099,7 +1132,7 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack
|
||||
if err := m.stackProvider.StartStackServices(stack, dbServices); err != nil {
|
||||
// R-893 option C (security review 2026-10-08): volumes or the definition already moved → HOLD.
|
||||
if definitionMoved || res.VolumesReplayed > 0 {
|
||||
return res, m.putBackAfterFailedRestore(stack, err, liveDef, undo, res.VolumesReplayed > 0, res.FilesPlaced, restartStack, "err.backup.restore_failed_held_mixed")
|
||||
return res, m.putBackAfterFailedRestore(stack, err, liveDef, undo, res.VolumesReplayed > 0, res.FilesPlaced, restartStack, "err.backup.restore_failed_held_mixed", settings.RestoreKindOffsite)
|
||||
}
|
||||
// Best-effort bring-up: a failed restore must not also be an outage.
|
||||
if sErr := restartStack(); sErr != nil {
|
||||
@@ -1151,7 +1184,7 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack
|
||||
// under them — the copy of the live volumes is poured back over the rolled-back database, the live
|
||||
// definition written back, and the app starts exactly as it was. A put-back that fails holds, as before.
|
||||
if definitionMoved || res.VolumesReplayed > 0 {
|
||||
return res, m.putBackAfterFailedRestore(stack, iErr, liveDef, undo, res.VolumesReplayed > 0, res.FilesPlaced, restartStack, "err.backup.db_restore_failed_held_mixed")
|
||||
return res, m.putBackAfterFailedRestore(stack, iErr, liveDef, undo, res.VolumesReplayed > 0, res.FilesPlaced, restartStack, "err.backup.db_restore_failed_held_mixed", settings.RestoreKindOffsite)
|
||||
}
|
||||
if sErr := restartStack(); sErr != nil {
|
||||
m.logger.Printf("[WARN] [offbox] %s: start after a successful rollback failed: %v", stack, sErr)
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/dockerexec"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/util"
|
||||
)
|
||||
|
||||
@@ -219,28 +220,46 @@ func (m *Manager) putBack(stack string, c *liveVolumeCopy) error {
|
||||
// failure's own held sentence) and the copy stays on disk. volumesMoved with no copy is such a case: nothing can
|
||||
// put them back. filesPlaced chooses the household's sentence: placed files are not taken back (as ruled for
|
||||
// option C, a file mix does not hold), so the sentence then says so instead of „nothing changed".
|
||||
//
|
||||
// restoreKind (R-1340) is settings.RestoreKindOffsite or RestoreKindLocal: the hold records it, with the step that
|
||||
// failed and whether the restore's failure was a database replay (holdKey says so — it is the callers' own choice of
|
||||
// the household's sentence), so the operator's mail names the real cause.
|
||||
func (m *Manager) putBackAfterFailedRestore(stack string, cause error, live *liveDefinition, c *liveVolumeCopy, volumesMoved bool,
|
||||
filesPlaced int, restart func() error, holdKey string) error {
|
||||
filesPlaced int, restart func() error, holdKey, restoreKind string) error {
|
||||
why := func(step string) mixedHoldWhy {
|
||||
w := mixedHoldWhy{kind: restoreKind, cause: settings.MixedCausePutBack, step: step,
|
||||
dbReplay: holdKey == "err.backup.db_restore_failed_held_mixed"}
|
||||
if c != nil && c.keep {
|
||||
w.copyDir = c.dir
|
||||
}
|
||||
return w
|
||||
}
|
||||
if volumesMoved && c == nil {
|
||||
m.logger.Printf("[ERROR] [offbox] %s: volumes were replaced and there is no copy to put back — holding the app", stack)
|
||||
m.holdAppAfterMixedRestore(stack, cause, live)
|
||||
m.holdAppAfterMixedRestore(stack, cause, live, why("the restore had already changed the app's data and there was no copy of the live volumes to put back"))
|
||||
return util.MsgError(holdKey, stack)
|
||||
}
|
||||
// R-1256 H5: a put-back under an app that could not be stopped could mix the copy with what the app writes.
|
||||
// Nothing is poured; the copy is kept and the app held (D20: a put-back that fails holds and keeps its copy).
|
||||
if err := m.stackProvider.StopStack(stack); err != nil {
|
||||
m.logger.Printf("[WARN] [offbox] %s: stopping the app before the put-back failed: %v", stack, err)
|
||||
c.keepIfAny()
|
||||
m.logger.Printf("[ERROR] [offbox] %s: stopping the app before the put-back FAILED: %v — nothing is put back; the copy stays at %s; holding the app", stack, err, c.dirOrNone())
|
||||
m.holdAppAfterMixedRestore(stack, cause, live, why("stopping the app before the put-back failed, so nothing was put back: "+err.Error()))
|
||||
return util.MsgError(holdKey, stack)
|
||||
}
|
||||
if volumesMoved {
|
||||
if err := m.putBack(stack, c); err != nil {
|
||||
c.keep = true
|
||||
m.logger.Printf("[ERROR] [offbox] %s: putting the live volumes back FAILED: %v — the copy stays at %s; holding the app", stack, err, c.dir)
|
||||
m.holdAppAfterMixedRestore(stack, cause, live)
|
||||
m.holdAppAfterMixedRestore(stack, cause, live, why("putting the live volumes back failed: "+err.Error()))
|
||||
return util.MsgError(holdKey, stack)
|
||||
}
|
||||
}
|
||||
if live != nil {
|
||||
if err := m.stackProvider.RecreateStackDefinitionFromUnit(stack, live.dir, live.env); err != nil {
|
||||
m.logger.Printf("[ERROR] [offbox] %s: writing the live definition back FAILED: %v — holding the app", stack, err)
|
||||
m.holdAppAfterMixedRestore(stack, cause, live)
|
||||
c.keepIfAny() // R-1256 H5 (D20): a put-back that fails keeps its copy for support
|
||||
m.logger.Printf("[ERROR] [offbox] %s: writing the live definition back FAILED: %v — the copy stays at %s; holding the app", stack, err, c.dirOrNone())
|
||||
m.holdAppAfterMixedRestore(stack, cause, live, why("writing the live app definition back failed: "+err.Error()))
|
||||
return util.MsgError(holdKey, stack)
|
||||
}
|
||||
}
|
||||
@@ -249,12 +268,19 @@ func (m *Manager) putBackAfterFailedRestore(stack string, cause error, live *liv
|
||||
m.restorePutBackNotify(stack, cause)
|
||||
}
|
||||
if err := restart(); err != nil {
|
||||
m.logger.Printf("[ERROR] [offbox] %s: start after the put-back failed: %v (the copy stays at %s)", stack, err, c.dirOrNone())
|
||||
m.logger.Printf("[ERROR] [offbox] %s: start after the put-back failed: %v (the live volumes and definition are back; the copy %s is removed)", stack, err, c.dirOrNone())
|
||||
return util.MsgError("err.backup.restore_failed_put_back_not_started", stack)
|
||||
}
|
||||
return putBackMessage(stack, filesPlaced)
|
||||
}
|
||||
|
||||
// keepIfAny marks the copy as kept for support (no copy: nothing to keep).
|
||||
func (c *liveVolumeCopy) keepIfAny() {
|
||||
if c != nil {
|
||||
c.keep = true
|
||||
}
|
||||
}
|
||||
|
||||
func (c *liveVolumeCopy) dirOrNone() string {
|
||||
if c == nil {
|
||||
return "(no volume copy)"
|
||||
|
||||
@@ -62,6 +62,53 @@ type RestoreOpStatus struct {
|
||||
func (m *Manager) BeginRestoreOp(op, stack string) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.beginRestoreOpLocked(op, stack)
|
||||
}
|
||||
|
||||
// Why a new restore may not begin now (R-1256 H6). "" = it may.
|
||||
const (
|
||||
RestoreBusyRestore = "restore" // a restore op is in flight (the record the pages follow)
|
||||
RestoreBusyBackup = "backup" // the single-flight backup/restore run is held (the night run never records an op)
|
||||
RestoreBusyMirror = "mirror" // a second-drive copy runs, or one a previous process lost may still run
|
||||
)
|
||||
|
||||
// TryBeginRestoreOp is BeginRestoreOp that refuses — in the SAME step, under m.mu — while another restore op, a
|
||||
// backup run or a second-drive copy runs. R-1256 H6: the pages asked (RestoreStatus, IsRunning,
|
||||
// Tier2CopyBlocksRestore) and recorded the op later; a mirror that began in between ran beside the restore. The mirror
|
||||
// asks under the same m.mu (beginTier2Run), so one of the two always sees the other. Lock order m.mu → tier2Copies,
|
||||
// as in beginTier2Run. Returns the busy reason ("" = begun) and, for a mirror, its app ("" when unknown).
|
||||
// Pinned by TestR1256_H6_*.
|
||||
func (m *Manager) TryBeginRestoreOp(op, stack string) (busy, mirrorApp string) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if busy, app := m.restoreBusyLocked(); busy != "" {
|
||||
return busy, app
|
||||
}
|
||||
m.beginRestoreOpLocked(op, stack)
|
||||
return "", ""
|
||||
}
|
||||
|
||||
// RestoreBusy is TryBeginRestoreOp's question alone — for a refusal asked early, before a request is read further.
|
||||
func (m *Manager) RestoreBusy() (busy, mirrorApp string) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return m.restoreBusyLocked()
|
||||
}
|
||||
|
||||
func (m *Manager) restoreBusyLocked() (string, string) {
|
||||
if m.opRunning {
|
||||
return RestoreBusyRestore, m.opStack
|
||||
}
|
||||
if m.running {
|
||||
return RestoreBusyBackup, ""
|
||||
}
|
||||
if app, blocked := m.Tier2CopyBlocksRestore(); blocked {
|
||||
return RestoreBusyMirror, app
|
||||
}
|
||||
return "", ""
|
||||
}
|
||||
|
||||
func (m *Manager) beginRestoreOpLocked(op, stack string) {
|
||||
m.opRunning = true
|
||||
m.opName = op
|
||||
m.opStack = stack
|
||||
|
||||
@@ -0,0 +1,358 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// R-1256 (review round 2, HUNT-1): four restore edges — H4, H5, H6 and H8, each pinned below.
|
||||
//
|
||||
// H4: an off-site restore writes the snapshot's definition before any data moves. The writer
|
||||
// (stackAdapter.RecreateStackDefinitionFromUnit) writes the compose FIRST and the app's settings after, so a
|
||||
// failure can come back with the snapshot's compose already in the stack dir. The failure branch used to start
|
||||
// the app at once — on the snapshot's (older) compose, over the live data. Now the live definition is written
|
||||
// back first; only then is the app started. When the write-back fails too, the app is held, never started.
|
||||
|
||||
// r1256DefFailProvider fails the FIRST definition write (the snapshot's), after recording it — as the real
|
||||
// writer does when its compose write succeeded and its settings write did not.
|
||||
type r1256DefFailProvider struct {
|
||||
*r893Provider
|
||||
failWrites int // how many writes fail, from the first
|
||||
failFrom int // when set, every write from this one (1-based) on fails
|
||||
}
|
||||
|
||||
func (p *r1256DefFailProvider) RecreateStackDefinitionFromUnit(name, composeDir string, env map[string]string) error {
|
||||
if err := p.r893Provider.RecreateStackDefinitionFromUnit(name, composeDir, env); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(p.defs) <= p.failWrites || (p.failFrom > 0 && len(p.defs) >= p.failFrom) {
|
||||
return errors.New("app.yaml write failed")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func r1256DefFailFixture(t *testing.T, failWrites int) (*Manager, *r1256DefFailProvider, *int) {
|
||||
t.Helper()
|
||||
m, vp, _, notified, _ := r893FixtureA(t, true, 0)
|
||||
p := &r1256DefFailProvider{r893Provider: vp, failWrites: failWrites}
|
||||
m.SetStackProvider(p)
|
||||
return m, p, notified
|
||||
}
|
||||
|
||||
// RED-PROOF: remove the live write-back from the definition-write failure branch (restartStack alone) → one
|
||||
// definition write (the snapshot's), then the start → FAILS.
|
||||
func TestR1256_H4_AFailedDefinitionWriteStartsTheAppAtItsLiveDefinition(t *testing.T) {
|
||||
m, p, notified := r1256DefFailFixture(t, 1)
|
||||
_, err := m.ReconstituteFromOffsite(context.Background(), "immich", false)
|
||||
if err == nil {
|
||||
t.Fatal("a failed definition write must be surfaced")
|
||||
}
|
||||
if len(p.defs) != 2 || !strings.Contains(strings.Join(p.defs[1], " "), "postgres:18-alpine") {
|
||||
t.Fatalf("definition writes %v — want the snapshot's, then the LIVE one written back", p.defs)
|
||||
}
|
||||
if !p.fullStarted {
|
||||
t.Fatalf("the app must run again at its live definition — calls %v", p.calls)
|
||||
}
|
||||
// The write-back comes BEFORE the start.
|
||||
lastRecreate, lastStart := -1, -1
|
||||
for i, c := range p.calls {
|
||||
switch c {
|
||||
case "recreate":
|
||||
lastRecreate = i
|
||||
case "start":
|
||||
lastStart = i
|
||||
}
|
||||
}
|
||||
if lastRecreate > lastStart {
|
||||
t.Fatalf("the app was started before the live definition was written back — calls %v", p.calls)
|
||||
}
|
||||
if held, _ := m.RestoreHoldFor("immich"); held {
|
||||
t.Fatal("the live definition is back: the app must not be held")
|
||||
}
|
||||
if *notified != 0 {
|
||||
t.Errorf("hold notices %d, want 0", *notified)
|
||||
}
|
||||
}
|
||||
|
||||
// The write-back fails too: the stack dir may still hold the snapshot's compose, so the app is NOT started; it is
|
||||
// held (restore_mixed: the definition moved and could not be put back) and the operator is told once.
|
||||
// RED-PROOF: start the app when the write-back fails → fullStarted → FAILS.
|
||||
func TestR1256_H4_AFailedWriteBackHoldsTheApp(t *testing.T) {
|
||||
m, p, notified := r1256DefFailFixture(t, 2)
|
||||
_, err := m.ReconstituteFromOffsite(context.Background(), "immich", false)
|
||||
if err == nil {
|
||||
t.Fatal("a failed definition write must be surfaced")
|
||||
}
|
||||
if p.fullStarted {
|
||||
t.Fatalf("the app was STARTED although its live definition could not be written back — calls %v", p.calls)
|
||||
}
|
||||
held, _ := m.RestoreHoldFor("immich")
|
||||
if !held {
|
||||
t.Fatal("no hold: every start path would start the app on the snapshot's definition")
|
||||
}
|
||||
if k := m.HoldKind("immich"); k != settings.HoldReasonRestoreMixed {
|
||||
t.Errorf("hold kind %q, want %q", k, settings.HoldReasonRestoreMixed)
|
||||
}
|
||||
if *notified != 1 {
|
||||
t.Errorf("the operator must be told once, got %d", *notified)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- H5: a put-back whose stop fails must not go on; `volume rm -f` errors count; a failed put-back keeps its copy.
|
||||
|
||||
// r1256StopFailProvider fails the Nth StopStack (1-based).
|
||||
type r1256StopFailProvider struct {
|
||||
*r893Provider
|
||||
failAt, n int
|
||||
}
|
||||
|
||||
func (p *r1256StopFailProvider) StopStack(name string) error {
|
||||
p.n++
|
||||
err := p.r893Provider.StopStack(name)
|
||||
if p.n == p.failAt {
|
||||
return errors.New("compose down: timeout")
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// The off-site put-back: the replay fails after the snapshot's volume replaced the live one; the stop before the
|
||||
// put-back fails. Pouring the copy back under a running app could mix it with what the app writes, so nothing is
|
||||
// poured, the copy is KEPT, and the app is held.
|
||||
// RED-PROOF: restore the „(continuing)" WARN in putBackAfterFailedRestore → the copy is poured and the app started
|
||||
// → FAILS.
|
||||
func TestR1256_H5_APutBackWhoseStopFailsDoesNotPourAndKeepsTheCopy(t *testing.T) {
|
||||
m, vp, rig, _, notified, putBack := r893PutBackFixture(t)
|
||||
p := &r1256StopFailProvider{r893Provider: vp, failAt: 2} // 1 = the restore's stop, 2 = the put-back's
|
||||
m.SetStackProvider(p)
|
||||
_, err := m.ReconstituteFromOffsite(context.Background(), "immich", false)
|
||||
if err == nil {
|
||||
t.Fatal("a failed restore must be surfaced")
|
||||
}
|
||||
if p.n < 2 {
|
||||
t.Fatalf("the put-back's stop never ran (stops %d) — the fixture did not reach the put-back", p.n)
|
||||
}
|
||||
if len(rig.poured) != 0 || len(rig.removed) != 0 {
|
||||
t.Fatalf("the copy was poured (%v) / volumes removed (%v) after a FAILED stop", rig.poured, rig.removed)
|
||||
}
|
||||
if vp.fullStarted {
|
||||
t.Fatalf("the app was started after a failed put-back — calls %v", vp.calls)
|
||||
}
|
||||
if held, _ := m.RestoreHoldFor("immich"); !held {
|
||||
t.Fatal("a put-back that could not run must hold the app (D20)")
|
||||
}
|
||||
if left := undoCopies(t, rig.undoRoot); len(left) == 0 {
|
||||
t.Fatal("the copy must be KEPT after a failed put-back (D20)")
|
||||
}
|
||||
if *notified != 1 || *putBack != 0 {
|
||||
t.Errorf("hold notices %d (want 1), put-back notices %d (want 0)", *notified, *putBack)
|
||||
}
|
||||
}
|
||||
|
||||
// The live definition cannot be written back after the volumes were poured: the app is held and the copy is KEPT
|
||||
// (D20: a put-back that fails keeps its copy for support).
|
||||
// RED-PROOF: drop `c.keep = true` from the definition write-back failure branch → the copy is removed → FAILS.
|
||||
func TestR1256_H5_AFailedDefinitionWriteBackKeepsTheCopy(t *testing.T) {
|
||||
m, vp, rig, _, _, _ := r893PutBackFixture(t)
|
||||
// A version change, so a live definition exists to be written back; the second write (the write-back) fails.
|
||||
if err := os.WriteFile(vp.composePath, []byte(vtCompose("18")), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
vp.livePins = ParseComposeImages(vp.composePath)
|
||||
p := &r1256DefFailProvider{r893Provider: vp, failWrites: 0}
|
||||
p.failFrom = 2
|
||||
m.SetStackProvider(p)
|
||||
_, err := m.ReconstituteFromOffsite(context.Background(), "immich", false)
|
||||
if err == nil {
|
||||
t.Fatal("a failed restore must be surfaced")
|
||||
}
|
||||
if len(p.defs) < 2 {
|
||||
t.Fatalf("the write-back never ran: writes %v", p.defs)
|
||||
}
|
||||
if held, _ := m.RestoreHoldFor("immich"); !held {
|
||||
t.Fatal("a failed write-back must hold the app")
|
||||
}
|
||||
if left := undoCopies(t, rig.undoRoot); len(left) == 0 {
|
||||
t.Fatal("the copy must be KEPT after a failed put-back (D20)")
|
||||
}
|
||||
}
|
||||
|
||||
// The crash put-back (a restore cut by a power loss): the stop before pouring fails → an error (the guard then
|
||||
// holds the app and keeps the copy, R-1251 H2), nothing poured.
|
||||
// RED-PROOF: restore the „(continuing)" WARN in PutBackInterruptedRestore → nil error, the copy poured → FAILS.
|
||||
func TestR1256_H5_ACrashPutBackWhoseStopFailsRefuses(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
undo := filepath.Join(dir, "backups", "restore-undo", "app", "20261028T000000Z")
|
||||
mustWrite(t, filepath.Join(undo, "undo.json"), `{"stack":"app","copied":["app_data"],"definition":false}`)
|
||||
mustWrite(t, filepath.Join(undo, "app_data.tar"), "LIVE")
|
||||
poured := 0
|
||||
cp := NewCrashPutBack(func(string) error { return errors.New("compose down: timeout") }, nil, log.New(io.Discard, "", 0))
|
||||
cp.volReplay = func(string, string) (int, error) { poured++; return 1, nil }
|
||||
if err := cp.PutBackInterruptedRestore("app", undo); err == nil {
|
||||
t.Fatal("a put-back whose stop failed must return an error")
|
||||
}
|
||||
if poured != 0 {
|
||||
t.Fatal("the copy was poured back under an app that could not be stopped")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(undo, "app_data.tar")); err != nil {
|
||||
t.Fatal("the copy must stay on disk")
|
||||
}
|
||||
}
|
||||
|
||||
// restoreDockerVolumesFrom: a `docker volume rm -f` that fails (the volume is still in use) used to be ignored; the
|
||||
// `volume create` that follows is a no-op on an existing volume, so the tar was unpacked ON TOP of the old content —
|
||||
// a merge. Now that volume counts as failed and is not touched.
|
||||
// RED-PROOF: ignore the rm error again → the create and the tar run for the volume → FAILS.
|
||||
func TestR1256_H5_AVolumeThatCannotBeRemovedIsNotMergedInto(t *testing.T) {
|
||||
bin := t.TempDir()
|
||||
calls := filepath.Join(bin, "calls.log")
|
||||
stub := "#!/bin/sh\necho \"$@\" >> " + calls + "\n" +
|
||||
"case \"$*\" in 'compose version --short') echo v2.29.7;; 'volume rm -f app_busy') echo 'volume is in use' >&2; exit 1;; esac\nexit 0\n"
|
||||
if err := os.WriteFile(filepath.Join(bin, "docker"), []byte(stub), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH"))
|
||||
dump := t.TempDir()
|
||||
for _, n := range []string{"app_busy.tar", "app_free.tar"} {
|
||||
mustWrite(t, filepath.Join(dump, n), "tar")
|
||||
}
|
||||
m := &Manager{logger: log.New(io.Discard, "", 0)}
|
||||
n, err := m.restoreDockerVolumesFrom("app", dump)
|
||||
if err == nil || n != 1 {
|
||||
t.Fatalf("restored %d, err %v — want 1 and an error naming app_busy", n, err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "app_busy") {
|
||||
t.Errorf("the error must name the volume: %v", err)
|
||||
}
|
||||
b, _ := os.ReadFile(calls)
|
||||
for _, l := range strings.Split(string(b), "\n") {
|
||||
if (strings.HasPrefix(l, "volume create") || strings.HasPrefix(l, "run ")) && strings.Contains(l, "app_busy") {
|
||||
t.Fatalf("the volume that could not be removed was written into: %q", l)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---- H6: a second-drive mirror must not start between the page's refusal and the restore's own record.
|
||||
|
||||
// The page asked (nothing ran), then a mirror began, then the restore began: the begin itself refuses, records
|
||||
// nothing, and the mirror — asked under the same lock — refuses once a restore is recorded.
|
||||
// RED-PROOF: make TryBeginRestoreOp record the op without asking (the old BeginRestoreOp) → the restore begins beside
|
||||
// the mirror → FAILS.
|
||||
func TestR1256_H6_ARestoreCannotBeginBesideAMirrorThatStartedAfterTheCheck(t *testing.T) {
|
||||
tier2ResetForTest()
|
||||
defer tier2ResetForTest()
|
||||
m, _ := newOffboxManager(t)
|
||||
if busy, _ := m.RestoreBusy(); busy != "" {
|
||||
t.Fatalf("fixture: nothing runs, got %q", busy)
|
||||
}
|
||||
// the window: the page's check passed; now a mirror begins
|
||||
if !m.beginTier2Run("immich") {
|
||||
t.Fatal("fixture: the mirror must begin while no restore is recorded")
|
||||
}
|
||||
if busy, app := m.TryBeginRestoreOp("restore", "immich"); busy != RestoreBusyMirror || app != "immich" {
|
||||
t.Fatalf("the restore began beside a running mirror: busy=%q app=%q", busy, app)
|
||||
}
|
||||
if m.RestoreStatus().Running {
|
||||
t.Fatal("a refused begin left a restore op recorded")
|
||||
}
|
||||
tier2CopyEnd("immich")
|
||||
if busy, _ := m.TryBeginRestoreOp("restore", "immich"); busy != "" {
|
||||
t.Fatalf("with the mirror done the restore must begin, got %q", busy)
|
||||
}
|
||||
if m.beginTier2Run("immich") {
|
||||
t.Fatal("a mirror began while the restore of the same app is recorded")
|
||||
}
|
||||
// a second restore is refused while the first runs; the backup run refuses too
|
||||
if busy, _ := m.TryBeginRestoreOp("restore", "other"); busy != RestoreBusyRestore {
|
||||
t.Fatalf("a second restore began beside the first: %q", busy)
|
||||
}
|
||||
m.EndRestoreOp(true, "ok")
|
||||
if err := m.AcquireRunningForTest(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if busy, _ := m.TryBeginRestoreOp("restore", "immich"); busy != RestoreBusyBackup {
|
||||
t.Fatalf("a restore began beside a backup run: %q", busy)
|
||||
}
|
||||
m.ReleaseRunningForTest()
|
||||
}
|
||||
|
||||
// ---- H8: an unused restore copy is cleaned up; a copy a hold or a pending put-back needs is never touched.
|
||||
|
||||
func r1256Copy(t *testing.T, m *Manager, app string, age time.Duration, now time.Time) string {
|
||||
t.Helper()
|
||||
dir := filepath.Join(restoreUndoDir(m.namespaceRoot(m.systemDataPath), app), now.Add(-age).UTC().Format(restoreCopyStampLayout))
|
||||
mustWrite(t, filepath.Join(dir, app+"_data.tar"), "LIVE")
|
||||
return dir
|
||||
}
|
||||
|
||||
func r1256Exists(p string) bool { _, err := os.Stat(p); return err == nil }
|
||||
|
||||
// RED-PROOF: drop the hold check (or the marker check) from SweepUnusedRestoreCopies → the held / named copy is removed
|
||||
// → FAILS; drop the age check → the young copy is removed → FAILS.
|
||||
func TestR1256_H8_OnlyAnUnusedOldCopyIsRemoved(t *testing.T) {
|
||||
m, sett := newOffboxManager(t)
|
||||
now := time.Now()
|
||||
old := 40 * 24 * time.Hour
|
||||
unused := r1256Copy(t, m, "unused", old, now)
|
||||
young := r1256Copy(t, m, "young", 24*time.Hour, now)
|
||||
held := r1256Copy(t, m, "held", old, now)
|
||||
waiting := r1256Copy(t, m, "waiting", old, now)
|
||||
named := r1256Copy(t, m, "named", old, now)
|
||||
if err := sett.SetRestoreHold(settings.RestoreHold{Stack: "held", At: now.Format(time.RFC3339), Reason: settings.HoldReasonRestoreMixed}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := sett.SetRestoreHold(settings.RestoreHold{Stack: "waiting", At: now.Format(time.RFC3339), Reason: settings.HoldReasonPutBackPending}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// a restore cut by a power loss: the marker (no longer active after a failed recovery's End) still names the copy
|
||||
if err := m.appStop.Begin("unit-restore:named", ReasonUnitRestore, []string{"named"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := m.appStop.SetUndo("named", named); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
got := m.SweepUnusedRestoreCopies(now)
|
||||
if len(got) != 1 || got[0] != unused || r1256Exists(unused) {
|
||||
t.Fatalf("removed %v — want exactly the unused old copy %s", got, unused)
|
||||
}
|
||||
for _, keep := range []string{young, held, waiting, named} {
|
||||
if !r1256Exists(keep) {
|
||||
t.Errorf("a needed copy was removed: %s", keep)
|
||||
}
|
||||
}
|
||||
if r1256Exists(filepath.Dir(unused)) {
|
||||
t.Error("the emptied per-app folder must go too")
|
||||
}
|
||||
}
|
||||
|
||||
// While a restore runs nothing is removed; a marker that cannot be read stops the sweep.
|
||||
// RED-PROOF: drop the RestoreBusy check → the old copy goes during a restore → FAILS.
|
||||
func TestR1256_H8_NothingIsRemovedDuringARestoreOrWithAnUnreadableMarker(t *testing.T) {
|
||||
m, _ := newOffboxManager(t)
|
||||
now := time.Now()
|
||||
c := r1256Copy(t, m, "app", 40*24*time.Hour, now)
|
||||
m.BeginRestoreOp("restore", "other")
|
||||
if got := m.SweepUnusedRestoreCopies(now); len(got) != 0 || !r1256Exists(c) {
|
||||
t.Fatalf("removed %v during a restore", got)
|
||||
}
|
||||
m.EndRestoreOp(true, "ok")
|
||||
mustWrite(t, m.appStop.path, "{not json")
|
||||
if got := m.SweepUnusedRestoreCopies(now); len(got) != 0 || !r1256Exists(c) {
|
||||
t.Fatalf("removed %v with an unreadable marker", got)
|
||||
}
|
||||
if err := os.Remove(m.appStop.path); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := m.SweepUnusedRestoreCopies(now); len(got) != 1 {
|
||||
t.Fatalf("control: with nothing in the way the old copy goes, removed %v", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// R-1340: a restore_mixed hold records WHAT put it in place — which restore, which branch, which put-back step failed,
|
||||
// whether the restore's failure was a database replay, and where the copy is kept — so the operator's mail
|
||||
// (cmd/controller heldStoppedOperatorText, pinned by TestR1340_* there) tells the real story. Measured 2026-10-29 on
|
||||
// Tester 1: Radicale (no database), its volume could not be removed, the put-back's stop failed, and the mail said its
|
||||
// DATABASE restore failed and was rolled back.
|
||||
//
|
||||
// RED-PROOF (2026-10-29): with the R-1340 fields dropped from holdAppAfterMixedRestore's hold literal and from
|
||||
// holdPutBackFailed's (the pre-R-1340 record, which carried no cause), all five tests below fail on „mixed cause """ /
|
||||
// „kind """.
|
||||
|
||||
func holdOf(t *testing.T, m *Manager, stack string) settings.RestoreHold {
|
||||
t.Helper()
|
||||
if m.settings == nil {
|
||||
t.Fatal("fixture: no settings wired")
|
||||
}
|
||||
h, ok := m.settings.GetRestoreHold(stack)
|
||||
if !ok {
|
||||
t.Fatalf("%s is not held", stack)
|
||||
}
|
||||
return h
|
||||
}
|
||||
|
||||
// The Radicale shape: a VOLUME restore fails (no database involved), then the put-back's stop fails.
|
||||
func TestR1340_AVolumeFailureWhosePutBackStopFailsRecordsTheStepAndNoDatabase(t *testing.T) {
|
||||
m, vp, rig, _, notified, _ := r893PutBackFixture(t)
|
||||
rig.snapReplayErr = errors.New("volume rm -f immich_pgdata: volume is in use")
|
||||
p := &r1256StopFailProvider{r893Provider: vp, failAt: 2} // 1 = the restore's stop, 2 = the put-back's
|
||||
m.SetStackProvider(p)
|
||||
if _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false); err == nil {
|
||||
t.Fatal("a failed restore must be surfaced")
|
||||
}
|
||||
h := holdOf(t, m, "immich")
|
||||
if h.Reason != settings.HoldReasonRestoreMixed || h.MixedCause != settings.MixedCausePutBack {
|
||||
t.Fatalf("reason %q, mixed cause %q — want restore_mixed / put_back", h.Reason, h.MixedCause)
|
||||
}
|
||||
if !strings.Contains(h.FailedStep, "stopping the app before the put-back failed") || !strings.Contains(h.FailedStep, "compose down: timeout") {
|
||||
t.Errorf("failed step %q — want the stop and its error", h.FailedStep)
|
||||
}
|
||||
if h.DBReplay {
|
||||
t.Error("a volume failure must not be recorded as a database replay")
|
||||
}
|
||||
if h.RestoreKind != settings.RestoreKindOffsite {
|
||||
t.Errorf("restore kind %q, want offsite", h.RestoreKind)
|
||||
}
|
||||
if h.SafetyDump == "" {
|
||||
t.Error("the kept copy's folder must be recorded")
|
||||
} else if _, err := os.Stat(h.SafetyDump); err != nil {
|
||||
t.Errorf("the recorded copy %s is not on disk: %v", h.SafetyDump, err)
|
||||
}
|
||||
if *notified != 1 {
|
||||
t.Errorf("hold notices %d, want 1", *notified)
|
||||
}
|
||||
}
|
||||
|
||||
// The database replay fails (option C's shape), then the put-back's stop fails: DBReplay is recorded.
|
||||
func TestR1340_ADatabaseReplayFailureRecordsTheDatabase(t *testing.T) {
|
||||
m, vp, _, _, _, _ := r893PutBackFixture(t)
|
||||
m.SetStackProvider(&r1256StopFailProvider{r893Provider: vp, failAt: 2})
|
||||
if _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false); err == nil {
|
||||
t.Fatal("a failed restore must be surfaced")
|
||||
}
|
||||
h := holdOf(t, m, "immich")
|
||||
if h.MixedCause != settings.MixedCausePutBack || !h.DBReplay {
|
||||
t.Fatalf("mixed cause %q, db replay %v — want put_back and true", h.MixedCause, h.DBReplay)
|
||||
}
|
||||
}
|
||||
|
||||
// R-1256 H4: the backup's definition cannot be written and the live one cannot be written back — no data moved.
|
||||
func TestR1340_ADefinitionWriteFailureRecordsItsCause(t *testing.T) {
|
||||
m, _, _ := r1256DefFailFixture(t, 2)
|
||||
if _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false); err == nil {
|
||||
t.Fatal("a failed definition write must be surfaced")
|
||||
}
|
||||
h := holdOf(t, m, "immich")
|
||||
if h.MixedCause != settings.MixedCauseDefinitionWrite || h.DBReplay {
|
||||
t.Fatalf("mixed cause %q, db replay %v — want definition_write and false", h.MixedCause, h.DBReplay)
|
||||
}
|
||||
if !strings.Contains(h.FailedStep, "writing the live app definition back failed") {
|
||||
t.Errorf("failed step %q", h.FailedStep)
|
||||
}
|
||||
}
|
||||
|
||||
// A LOCAL restore (from the app's own backup) records itself as local, never off-site.
|
||||
func TestR1340_ALocalRestoreRecordsItsKind(t *testing.T) {
|
||||
m, _, rig, _ := r1177Fixture(t)
|
||||
if m.settings == nil {
|
||||
sett, err := settings.Load(t.TempDir()+"/settings.json", log.New(io.Discard, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m.settings = sett
|
||||
}
|
||||
rig.unitErr = errors.New("tar: short read")
|
||||
rig.putBackErr = errors.New("tar xf failed")
|
||||
if _, err := m.RestoreFromRecoveryUnit("app"); err == nil {
|
||||
t.Fatal("a failed put-back must be surfaced")
|
||||
}
|
||||
h := holdOf(t, m, "app")
|
||||
if h.RestoreKind != settings.RestoreKindLocal || h.MixedCause != settings.MixedCausePutBack || h.DBReplay {
|
||||
t.Fatalf("kind %q, cause %q, db %v — want local / put_back / false", h.RestoreKind, h.MixedCause, h.DBReplay)
|
||||
}
|
||||
if !strings.Contains(h.FailedStep, "putting the live volumes back failed") || !strings.Contains(h.FailedStep, "tar xf failed") {
|
||||
t.Errorf("failed step %q", h.FailedStep)
|
||||
}
|
||||
}
|
||||
|
||||
// A restore cut by a stop whose put-back fails at the next start (R-1251 H2) records its own cause.
|
||||
func TestR1340_AnInterruptedPutBackRecordsItsCause(t *testing.T) {
|
||||
r := newR1251Rig(t)
|
||||
r.putErr = errors.New("compose down: timeout")
|
||||
r.guard().Recover()
|
||||
h, ok := r.sett.GetRestoreHold("immich")
|
||||
if !ok || h.Reason != settings.HoldReasonRestoreMixed || h.MixedCause != settings.MixedCauseInterruptedPutBack {
|
||||
t.Fatalf("hold %+v (held %v) — want restore_mixed / interrupted_put_back", h, ok)
|
||||
}
|
||||
if !strings.Contains(h.FailedStep, "compose down: timeout") {
|
||||
t.Errorf("failed step %q", h.FailedStep)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"path/filepath"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// r1342Drive registers the rig's app drive, as every drive is registered on a box, so the list and the delete see it.
|
||||
func r1342Drive(t *testing.T, m *Manager) {
|
||||
t.Helper()
|
||||
if m.settings == nil {
|
||||
sett, err := settings.Load(filepath.Join(t.TempDir(), "settings.json"), log.New(io.Discard, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m.settings = sett
|
||||
}
|
||||
if err := m.settings.AddStoragePath(settings.StoragePath{Path: m.GetAppDrivePath("app"), Label: "rig"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// R-1342 (found live on Tester 1, 2026-10-29 10:39 CET): (1) while the box still watches whether a restored app runs
|
||||
// stably, its restore copy is the way back — „Törlés most" must not take it (D62: never while an operation runs on the
|
||||
// app); (2) an app REMOVED during that watch is not „did not start" — no operator alarm, no „the copy is kept".
|
||||
|
||||
// removableProvider is the stable-watch rig's provider that can answer StackRemoved.
|
||||
type removableProvider struct {
|
||||
*sampledProvider
|
||||
removed atomic.Bool
|
||||
}
|
||||
|
||||
func (p *removableProvider) StackRemoved(string) bool { return p.removed.Load() }
|
||||
|
||||
// RED-PROOF: drop the watching check from restoreCopyNeededFor / DeleteRestoreCopy → the copy is offered and deleted
|
||||
// mid-watch → FAILS.
|
||||
func TestR1342_ACopyUnderTheStableWatchIsNotDeletable(t *testing.T) {
|
||||
withStable(t, 400*time.Millisecond, 3*time.Second)
|
||||
m, rig, alarms := fixARig(t, func(int) bool { return true },
|
||||
func(int) []ContainerSample { return []ContainerSample{healthy("c1", 0)} })
|
||||
r1342Drive(t, m)
|
||||
m.BeginRestoreOp("restore", "app")
|
||||
if _, err := m.RestoreFromRecoveryUnit("app"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m.EndRestoreOp(true, "ok")
|
||||
copies := m.RestoreCopies()
|
||||
if len(copies) != 1 || copies[0].Needed != RestoreCopyNeededChecking || !copies[0].DeleteOn.IsZero() {
|
||||
t.Fatalf("mid-watch the copy must read „checking\", got %+v", copies)
|
||||
}
|
||||
if _, err := m.DeleteRestoreCopy("app", copies[0].Stamp); !errors.Is(err, ErrRestoreCopyNeeded) {
|
||||
t.Fatalf("a delete mid-watch: %v, want ErrRestoreCopyNeeded", err)
|
||||
}
|
||||
if left := undoCopies(t, rig.undoRoot); len(left) != 1 {
|
||||
t.Fatalf("the copy must stay through the watch, found %v", left)
|
||||
}
|
||||
if left := waitNoCopies(t, rig.undoRoot); len(left) != 0 {
|
||||
t.Fatalf("after the stable window the watch releases it, found %v", left)
|
||||
}
|
||||
if len(*alarms) != 0 {
|
||||
t.Fatalf("no alarm: %v", *alarms)
|
||||
}
|
||||
if m.watchingCopy(filepath.Join(rig.undoRoot, copies[0].Stamp)) {
|
||||
t.Fatal("the watch mark must go when the watch ends")
|
||||
}
|
||||
}
|
||||
|
||||
// RED-PROOF: drop the stableRemoved case → the removed app „did not run stably" → one alarm → FAILS.
|
||||
func TestR1342_AnAppRemovedDuringTheWatchRaisesNoAlarm(t *testing.T) {
|
||||
withStable(t, 2*time.Second, 400*time.Millisecond)
|
||||
var up atomic.Bool
|
||||
up.Store(true)
|
||||
m, rig, alarms := fixARig(t, func(int) bool { return up.Load() },
|
||||
func(int) []ContainerSample { return []ContainerSample{healthy("c1", 0)} })
|
||||
rp := &removableProvider{sampledProvider: m.stackProvider.(*sampledProvider)}
|
||||
m.stackProvider = rp
|
||||
r1342Drive(t, m)
|
||||
m.BeginRestoreOp("restore", "app")
|
||||
if _, err := m.RestoreFromRecoveryUnit("app"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m.EndRestoreOp(true, "ok")
|
||||
// the household removes the app inside the watch
|
||||
up.Store(false)
|
||||
rp.removed.Store(true)
|
||||
time.Sleep(700 * time.Millisecond)
|
||||
if len(*alarms) != 0 {
|
||||
t.Fatalf("an app removed during the watch must not alarm „did not start\": %v", *alarms)
|
||||
}
|
||||
copies := m.RestoreCopies()
|
||||
if len(copies) != 1 || copies[0].Needed != "" {
|
||||
t.Fatalf("after the watch ended the copy is an unused one (sweep / Törlés most): %+v", copies)
|
||||
}
|
||||
_ = rig
|
||||
}
|
||||
|
||||
// Control: an app that stays installed and never runs stably still alarms once (the R-1270 rule is unchanged).
|
||||
func TestR1342_AnInstalledUnstableAppStillAlarms(t *testing.T) {
|
||||
withStable(t, 2*time.Second, 400*time.Millisecond)
|
||||
var up atomic.Bool
|
||||
up.Store(true)
|
||||
m, _, alarms := fixARig(t, func(int) bool { return up.Load() },
|
||||
func(int) []ContainerSample { return []ContainerSample{healthy("c1", 0)} })
|
||||
m.stackProvider = &removableProvider{sampledProvider: m.stackProvider.(*sampledProvider)}
|
||||
m.BeginRestoreOp("restore", "app")
|
||||
if _, err := m.RestoreFromRecoveryUnit("app"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m.EndRestoreOp(true, "ok")
|
||||
up.Store(false)
|
||||
time.Sleep(700 * time.Millisecond)
|
||||
if len(*alarms) != 1 {
|
||||
t.Fatalf("an installed app that never runs stably alarms once, got %v", *alarms)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// R-1347: a restore that lifts an UPDATE hold must also forget the stack manager's in-memory „last update ended held"
|
||||
// flag. Left set, the crash-loop watcher does not watch the app and the night leg skips it until the controller restarts
|
||||
// (D64's sibling: the operator's clear_hold already forgets it, R-1343; the restore did not).
|
||||
//
|
||||
// The consequence on the real flag — ObserveUnhealthy and legCandidate — is TestR1347_RestoreLiftsTheWatchersBlindSpot
|
||||
// (cmd/controller), and the seam's wiring TestR1347_UpdateHeldForgetterIsWiredAtStartup.
|
||||
//
|
||||
// RED-PROOF (2026-11-02): seen failing before the seam existed — the build failed on SetUpdateHeldForgetter; with the
|
||||
// call dropped from clearUpdateHoldAfterRestore it fails on „the flag was not forgotten".
|
||||
func TestR1347_ARestoreThatLiftsAnUpdateHoldForgetsTheHeldFlag(t *testing.T) {
|
||||
sett := slice4Settings(t)
|
||||
m := &Manager{logger: log.New(io.Discard, "", 0), settings: sett}
|
||||
forgot := map[string]int{}
|
||||
m.SetUpdateHeldForgetter(func(s string) bool { forgot[s]++; return true })
|
||||
|
||||
_ = m.HoldAfterFailedUpdate("nextcloud", time.Now(), time.Now(), UpdateTierLocal)
|
||||
m.clearUpdateHoldAfterRestore("nextcloud")
|
||||
if _, held := sett.GetRestoreHold("nextcloud"); held {
|
||||
t.Fatal("the update hold must be lifted")
|
||||
}
|
||||
if forgot["nextcloud"] != 1 {
|
||||
t.Fatalf("the restore lifted the update hold, but the flag was not forgotten (calls: %v)", forgot)
|
||||
}
|
||||
|
||||
// A RESTORE hold (R-379) stays, so nothing about it is forgotten.
|
||||
_ = sett.SetRestoreHold(settings.RestoreHold{Stack: "rst", At: "2026-08-22T14:00:00Z"})
|
||||
m.clearUpdateHoldAfterRestore("rst")
|
||||
if _, held := sett.GetRestoreHold("rst"); !held || forgot["rst"] != 0 {
|
||||
t.Fatalf("a restore hold is not lifted by a restore (held %v, forget calls %d)", held, forgot["rst"])
|
||||
}
|
||||
|
||||
// No hold at all: nothing to forget (a plain restore of a healthy app).
|
||||
m.clearUpdateHoldAfterRestore("plain")
|
||||
if forgot["plain"] != 0 {
|
||||
t.Fatalf("no hold was lifted, yet the flag was touched %d time(s)", forgot["plain"])
|
||||
}
|
||||
|
||||
// nil seam: no crash (the pre-R-1347 behaviour).
|
||||
m2 := &Manager{logger: log.New(io.Discard, "", 0), settings: sett}
|
||||
_ = m2.HoldAfterFailedUpdate("other", time.Now(), time.Now(), UpdateTierLocal)
|
||||
m2.clearUpdateHoldAfterRestore("other")
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// R-1348: the restore_mixed hold written at START-UP after an interrupted put-back fails must reach the operator as the
|
||||
// R-1340 mail. Recover runs long before the notifier exists (main.go), so the guard keeps the name and hands it over when
|
||||
// SetHeldNotify is wired; a failure after that (a drive that returned at run time) goes out at once.
|
||||
//
|
||||
// RED-PROOF (2026-11-02): seen failing before the seam existed (build: SetHeldNotify); with the call dropped from
|
||||
// holdPutBackFailed these fail on „no mail for a held app".
|
||||
func TestR1348_AFailedPutBackAtStartupMailsOnceTheNotifierIsWired(t *testing.T) {
|
||||
r := newR1251Rig(t)
|
||||
r.putErr = errors.New("compose down: timeout")
|
||||
g := r.guard()
|
||||
if res := g.Recover(); res == nil || len(res.Held) != 1 {
|
||||
t.Fatalf("recovery must hold the app: %+v", res)
|
||||
}
|
||||
var mailed []string
|
||||
g.SetHeldNotify(func(stack string) { mailed = append(mailed, stack) })
|
||||
if len(mailed) != 1 || mailed[0] != "immich" {
|
||||
t.Fatalf("no mail for a held app: the hold from Recover must be handed over when the notifier is wired, got %v", mailed)
|
||||
}
|
||||
h, ok := r.sett.GetRestoreHold("immich")
|
||||
if !ok || h.Reason != settings.HoldReasonRestoreMixed || h.MixedCause != settings.MixedCauseInterruptedPutBack ||
|
||||
!strings.Contains(h.FailedStep, "compose down: timeout") {
|
||||
t.Fatalf("the mail reads this record, and it is wrong: %+v", h)
|
||||
}
|
||||
g.SetHeldNotify(func(stack string) { mailed = append(mailed, "again:"+stack) })
|
||||
if len(mailed) != 1 {
|
||||
t.Fatalf("a held app is mailed once, not at every wiring: %v", mailed)
|
||||
}
|
||||
}
|
||||
|
||||
// A clean put-back, or one still waiting for its drive, is no failure: no mail.
|
||||
func TestR1348_NoMailWithoutAFailedPutBack(t *testing.T) {
|
||||
r := newR1251Rig(t)
|
||||
g := r.guard()
|
||||
g.Recover()
|
||||
var mailed []string
|
||||
g.SetHeldNotify(func(stack string) { mailed = append(mailed, stack) })
|
||||
if len(mailed) != 0 {
|
||||
t.Fatalf("a put-back that worked mailed: %v", mailed)
|
||||
}
|
||||
|
||||
w := newR1251Rig(t)
|
||||
w.unbind(t)
|
||||
gw := w.guard()
|
||||
gw.Recover()
|
||||
gw.SetHeldNotify(func(stack string) { mailed = append(mailed, stack) })
|
||||
if len(mailed) != 0 {
|
||||
t.Fatalf("a put-back that waits for its drive has its own hold and reminder, not this mail: %v", mailed)
|
||||
}
|
||||
}
|
||||
|
||||
// After the wiring a failed put-back (the drive returned at run time: ResumePutBack) mails at once.
|
||||
func TestR1348_AFailedResumeMailsAtOnce(t *testing.T) {
|
||||
r := newR1251Rig(t)
|
||||
r.unbind(t)
|
||||
g := r.guard()
|
||||
g.Recover() // waiting for its drive
|
||||
var mailed []string
|
||||
g.SetHeldNotify(func(stack string) { mailed = append(mailed, stack) })
|
||||
r.bind(t)
|
||||
r.putErr = errors.New("tar xf failed")
|
||||
g.ResumePutBack("immich")
|
||||
if len(mailed) != 1 || mailed[0] != "immich" {
|
||||
t.Fatalf("a put-back that failed on resume must mail: %v", mailed)
|
||||
}
|
||||
}
|
||||
@@ -224,8 +224,14 @@ func (m *Manager) restoreDockerVolumesFrom(stackName, dumpDir string) (int, erro
|
||||
|
||||
m.logger.Printf("[INFO] [backup] Restoring Docker volume %s for %s", volName, stackName)
|
||||
|
||||
// Remove existing volume (ignore errors — may not exist)
|
||||
dockerexec.Command("docker", "volume", "rm", "-f", volName).Run()
|
||||
// Remove the existing volume. `rm -f` of a missing volume succeeds, so an error means the volume EXISTS and
|
||||
// could not be removed (still in use). R-1256 H5: it used to be ignored — the `volume create` below is a no-op
|
||||
// on an existing volume, and the tar was unpacked ON TOP of the old content (a merge). That volume fails now.
|
||||
if out, err := dockerexec.Command("docker", "volume", "rm", "-f", volName).CombinedOutput(); err != nil {
|
||||
m.logger.Printf("[ERROR] [backup] Failed to remove volume %s before the restore: %s — %v (not written into)", volName, strings.TrimSpace(string(out)), err)
|
||||
failed = append(failed, volName)
|
||||
continue
|
||||
}
|
||||
|
||||
// Create fresh volume — WITH the labels compose gives its own volumes (R-658, v0.268.0). A bare
|
||||
// `volume create` made a volume the update's undo (until v0.267.0) could not find, and one
|
||||
|
||||
@@ -0,0 +1,216 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// D62 (operator ruling 2026-10-29; `07` §6.3): the household sees its restore copies in the kept-copies list
|
||||
// („Megőrzött másolatok") on the restore page — the app, the date of the restore the copy protects, its size, and the
|
||||
// day the box deletes it by itself (restoreCopyUnusedAge after the restore) — and may delete one earlier with
|
||||
// „Törlés most", after a warning and the app's name typed.
|
||||
//
|
||||
// A copy a hold or a pending put-back needs is NEVER deletable (the same three states the sweep leaves alone, R-1256
|
||||
// H8): the row says why in one line and the button is off; the handler refuses it too. While any backup or restore
|
||||
// operation runs on the box, a delete is refused (409). The check and the removal are ONE step under m.mu, so a
|
||||
// restore cannot begin in between (the R-1256 H6 shape). Pinned by TestD62_*.
|
||||
|
||||
// Why a restore copy is needed — the row's one line, and the refusal's reason. "" = not needed: deletable.
|
||||
const (
|
||||
RestoreCopyNeededHold = "hold" // the app is held (a failed put-back keeps its copy, D20) — support decides
|
||||
RestoreCopyNeededPutBack = "putback" // a put-back waits for its drive (R-1251) or a cut restore names it (D22)
|
||||
RestoreCopyNeededUnknown = "unknown" // the app-stop marker cannot be read: nothing is offered (fail closed)
|
||||
// RestoreCopyNeededChecking (R-1342): the box still watches whether the restored app runs stably (up to 30 minutes
|
||||
// after the restore, R-1270) — the copy is the way back if it does not.
|
||||
RestoreCopyNeededChecking = "checking"
|
||||
)
|
||||
|
||||
var (
|
||||
ErrRestoreCopyNotFound = errors.New("restore copy not found")
|
||||
ErrRestoreCopyNeeded = errors.New("restore copy is needed by a hold or a pending put-back")
|
||||
ErrRestoreCopyBusy = errors.New("a backup or restore operation runs")
|
||||
)
|
||||
|
||||
// RestoreCopy is one restore copy (<namespace>/backups/restore-undo/<app>/<stamp>).
|
||||
type RestoreCopy struct {
|
||||
App string
|
||||
Stamp string // the folder's name; with App, the copy's identity on the page
|
||||
At time.Time // when the restore began (the stamp; the folder's time when the stamp does not parse)
|
||||
SizeBytes int64
|
||||
Path string
|
||||
Needed string // "" or one of RestoreCopyNeeded*
|
||||
DeleteOn time.Time // when the sweep removes it; zero when Needed != "" (a needed copy is never swept)
|
||||
}
|
||||
|
||||
// restoreCopyAt is a copy's restore time: its stamp, or the folder's time when the stamp does not parse.
|
||||
func restoreCopyAt(dir, name string) (time.Time, bool) {
|
||||
if at, err := time.Parse(restoreCopyStampLayout, name); err == nil {
|
||||
return at, true
|
||||
}
|
||||
fi, err := os.Stat(dir)
|
||||
if err != nil {
|
||||
return time.Time{}, false
|
||||
}
|
||||
return fi.ModTime(), true
|
||||
}
|
||||
|
||||
// restoreCopyNeededFor is why app's copies are needed ("" = deletable). named is appStopNamedApps' answer, ok its ok.
|
||||
func (m *Manager) restoreCopyNeededFor(app string, named map[string]bool, ok bool) string {
|
||||
if !ok || m.settings == nil {
|
||||
return RestoreCopyNeededUnknown
|
||||
}
|
||||
if named[app] {
|
||||
return RestoreCopyNeededPutBack
|
||||
}
|
||||
if h, held := m.settings.GetRestoreHold(app); held {
|
||||
if h.Reason == settings.HoldReasonPutBackPending {
|
||||
return RestoreCopyNeededPutBack
|
||||
}
|
||||
return RestoreCopyNeededHold
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func dirSize(dir string) int64 {
|
||||
var n int64
|
||||
_ = filepath.WalkDir(dir, func(_ string, d fs.DirEntry, err error) error {
|
||||
if err == nil && d.Type().IsRegular() {
|
||||
if fi, e := d.Info(); e == nil {
|
||||
n += fi.Size()
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return n
|
||||
}
|
||||
|
||||
// RestoreCopies lists every restore copy on the box's drives, oldest first.
|
||||
func (m *Manager) RestoreCopies() []RestoreCopy {
|
||||
named, ok := m.appStopNamedApps()
|
||||
var out []RestoreCopy
|
||||
for _, nsRoot := range m.primaryUnitRoots() {
|
||||
base := filepath.Join(nsRoot, "backups", "restore-undo")
|
||||
apps, err := os.ReadDir(base)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for _, a := range apps {
|
||||
if !a.IsDir() {
|
||||
continue
|
||||
}
|
||||
app := a.Name()
|
||||
needed := m.restoreCopyNeededFor(app, named, ok)
|
||||
copies, err := os.ReadDir(filepath.Join(base, app))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for _, c := range copies {
|
||||
if !c.IsDir() {
|
||||
continue
|
||||
}
|
||||
dir := filepath.Join(base, app, c.Name())
|
||||
at, ok := restoreCopyAt(dir, c.Name())
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
rc := RestoreCopy{App: app, Stamp: c.Name(), At: at, SizeBytes: dirSize(dir), Path: dir, Needed: needed}
|
||||
if rc.Needed == "" && m.watchingCopy(dir) {
|
||||
rc.Needed = RestoreCopyNeededChecking
|
||||
}
|
||||
if rc.Needed == "" {
|
||||
rc.DeleteOn = at.Add(restoreCopyUnusedAge)
|
||||
}
|
||||
out = append(out, rc)
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return out[i].At.Before(out[j].At) })
|
||||
return out
|
||||
}
|
||||
|
||||
// validCopyName is a single path element: no separator, not empty, not "." or "..".
|
||||
func validCopyName(s string) bool {
|
||||
return s != "" && s != "." && s != ".." && !strings.ContainsAny(s, `/\`) && filepath.Base(s) == s
|
||||
}
|
||||
|
||||
// DeleteRestoreCopy removes ONE restore copy, named by its app and its folder's name, and nothing else (the path
|
||||
// guard: both names are single path elements, the folder is a real directory — not a link — directly under
|
||||
// <namespace>/backups/restore-undo/<app>/). Refused while any backup or restore operation runs (ErrRestoreCopyBusy)
|
||||
// and for a copy a hold or a pending put-back needs (ErrRestoreCopyNeeded). The check and the removal hold m.mu, so
|
||||
// no restore can begin in between. Returns the copy as it was (for the log line and the operator event).
|
||||
func (m *Manager) DeleteRestoreCopy(app, stamp string) (RestoreCopy, error) {
|
||||
if !validCopyName(app) || !validCopyName(stamp) {
|
||||
return RestoreCopy{}, ErrRestoreCopyNotFound
|
||||
}
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if busy, _ := m.restoreBusyLocked(); busy != "" {
|
||||
return RestoreCopy{}, ErrRestoreCopyBusy
|
||||
}
|
||||
var found *RestoreCopy
|
||||
for _, nsRoot := range m.primaryUnitRoots() {
|
||||
appDir := filepath.Join(nsRoot, "backups", "restore-undo", app)
|
||||
dir := filepath.Join(appDir, stamp)
|
||||
fi, err := os.Lstat(dir)
|
||||
if err != nil || !fi.IsDir() || fi.Mode()&os.ModeSymlink != 0 {
|
||||
continue
|
||||
}
|
||||
if filepath.Dir(dir) != appDir || filepath.Dir(appDir) != filepath.Join(nsRoot, "backups", "restore-undo") {
|
||||
continue
|
||||
}
|
||||
at, ok := restoreCopyAt(dir, stamp)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
found = &RestoreCopy{App: app, Stamp: stamp, At: at, SizeBytes: dirSize(dir), Path: dir}
|
||||
break
|
||||
}
|
||||
if found == nil {
|
||||
return RestoreCopy{}, ErrRestoreCopyNotFound
|
||||
}
|
||||
named, ok := m.appStopNamedApps()
|
||||
if found.Needed = m.restoreCopyNeededFor(app, named, ok); found.Needed != "" {
|
||||
return *found, ErrRestoreCopyNeeded
|
||||
}
|
||||
if m.watching[found.Path] { // R-1342: under m.mu, the same lock the watch marks its copy under
|
||||
found.Needed = RestoreCopyNeededChecking
|
||||
return *found, ErrRestoreCopyNeeded
|
||||
}
|
||||
if err := os.RemoveAll(found.Path); err != nil {
|
||||
return *found, fmt.Errorf("removing the restore copy %s: %w", found.Path, err)
|
||||
}
|
||||
_ = os.Remove(filepath.Dir(found.Path)) // the per-app folder, when it is now empty
|
||||
return *found, nil
|
||||
}
|
||||
|
||||
// setWatching marks (on) or clears a copy a stable-start watch holds (R-1342).
|
||||
func (m *Manager) setWatching(dir string, on bool) {
|
||||
if dir == "" {
|
||||
return
|
||||
}
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if on {
|
||||
if m.watching == nil {
|
||||
m.watching = map[string]bool{}
|
||||
}
|
||||
m.watching[dir] = true
|
||||
return
|
||||
}
|
||||
delete(m.watching, dir)
|
||||
}
|
||||
|
||||
// watchingCopy reports whether a stable-start watch still holds the copy.
|
||||
func (m *Manager) watchingCopy(dir string) bool {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return m.watching[dir]
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-1256 H8 (review round 2, HUNT-1): a restore copy (<namespace>/backups/restore-undo/<app>/<stamp>) that nothing
|
||||
// will ever use again used to stay on the drive for good. A copy is KEPT on purpose in three states — a put-back that
|
||||
// failed (the app is held, D20), a put-back waiting for its drive (the putback_pending hold, R-1251) and a restore cut
|
||||
// by a power loss (the app-stop marker names it, D22) — and for 30 minutes after a restore while the box watches for a
|
||||
// stable start (R-1270). Every one of those is still needed. A copy outside all of them — the app runs again, or was
|
||||
// removed, and nothing names the copy — is disk the household cannot see or use.
|
||||
//
|
||||
// The rule (the age: operator ruling D62, 2026-10-29 — 30 days, and the household may delete a copy earlier with
|
||||
// „Törlés most", restore_copies.go): a copy is removed only
|
||||
// when ALL of these hold, and the sweep removes NOTHING when it cannot answer one of them:
|
||||
// - it is older than restoreCopyUnusedAge (its folder's stamp; the folder's time when the stamp does not parse);
|
||||
// - the app has no restore hold of any kind (held apps keep every copy — support decides);
|
||||
// - the app-stop marker does not name the app (a pending put-back), and the marker is readable;
|
||||
// - no restore op is in flight on the box (a new restore's copy and the stable-start watcher are left alone).
|
||||
//
|
||||
// Run every 6 hours from main.go (TestR1256_H8_TheSweepIsWiredAtStartup). Pinned by TestR1256_H8_*.
|
||||
|
||||
// restoreCopyUnusedAge is how long an unused copy stays after its restore (D62: 30 days; TestD62_AnUnusedCopyIsKept30Days).
|
||||
const restoreCopyUnusedAge = 30 * 24 * time.Hour
|
||||
|
||||
// restoreCopyStampLayout is copyLiveVolumes' folder name.
|
||||
const restoreCopyStampLayout = "20060102T150405Z"
|
||||
|
||||
// SweepUnusedRestoreCopies removes the restore copies no hold, no pending put-back and no running restore needs, and
|
||||
// older than restoreCopyUnusedAge. Returns the folders removed.
|
||||
func (m *Manager) SweepUnusedRestoreCopies(now time.Time) []string {
|
||||
if busy, _ := m.RestoreBusy(); busy == RestoreBusyRestore {
|
||||
m.logger.Printf("[INFO] [backup] restore-copy sweep skipped: a restore runs")
|
||||
return nil
|
||||
}
|
||||
named, ok := m.appStopNamedApps()
|
||||
if !ok {
|
||||
m.logger.Printf("[WARN] [backup] restore-copy sweep skipped: the app-stop marker cannot be read — a copy it names must not go")
|
||||
return nil
|
||||
}
|
||||
var removed []string
|
||||
for _, nsRoot := range m.primaryUnitRoots() {
|
||||
base := filepath.Join(nsRoot, "backups", "restore-undo")
|
||||
apps, err := os.ReadDir(base)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for _, a := range apps {
|
||||
if !a.IsDir() {
|
||||
continue
|
||||
}
|
||||
app := a.Name()
|
||||
if named[app] {
|
||||
continue
|
||||
}
|
||||
if m.settings == nil {
|
||||
return removed // no hold store: a held app cannot be told apart
|
||||
}
|
||||
if _, held := m.settings.GetRestoreHold(app); held {
|
||||
continue
|
||||
}
|
||||
appDir := filepath.Join(base, app)
|
||||
copies, err := os.ReadDir(appDir)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
sort.Slice(copies, func(i, j int) bool { return copies[i].Name() < copies[j].Name() })
|
||||
for _, c := range copies {
|
||||
if !c.IsDir() {
|
||||
continue
|
||||
}
|
||||
dir := filepath.Join(appDir, c.Name())
|
||||
at, err := time.Parse(restoreCopyStampLayout, c.Name())
|
||||
if err != nil {
|
||||
fi, sErr := os.Stat(dir)
|
||||
if sErr != nil {
|
||||
continue
|
||||
}
|
||||
at = fi.ModTime()
|
||||
}
|
||||
if now.Sub(at) < restoreCopyUnusedAge {
|
||||
continue
|
||||
}
|
||||
if err := os.RemoveAll(dir); err != nil {
|
||||
m.logger.Printf("[WARN] [backup] %s: the unused restore copy %s could not be removed: %v", app, dir, err)
|
||||
continue
|
||||
}
|
||||
m.logger.Printf("[INFO] [backup] %s: the unused restore copy %s (from %s) is removed — no hold, no pending put-back, older than %s (R-1256)",
|
||||
app, dir, at.UTC().Format(time.RFC3339), restoreCopyUnusedAge)
|
||||
removed = append(removed, dir)
|
||||
}
|
||||
_ = os.Remove(appDir) // the per-app folder, when it is now empty
|
||||
}
|
||||
}
|
||||
return removed
|
||||
}
|
||||
|
||||
// appStopNamedApps reads the app-stop marker without side effects: the apps of an ACTIVE marker and every app its Undo
|
||||
// names. ok=false when the marker exists and cannot be read — the caller then touches nothing.
|
||||
func (m *Manager) appStopNamedApps() (map[string]bool, bool) {
|
||||
named := map[string]bool{}
|
||||
if m.appStop == nil || m.appStop.path == "" {
|
||||
return named, true
|
||||
}
|
||||
data, err := os.ReadFile(m.appStop.path)
|
||||
if os.IsNotExist(err) {
|
||||
return named, true
|
||||
}
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
var mk AppStopMarker
|
||||
if err := json.Unmarshal(data, &mk); err != nil {
|
||||
return nil, false
|
||||
}
|
||||
for app := range mk.Undo {
|
||||
named[app] = true
|
||||
}
|
||||
if mk.Active {
|
||||
for _, app := range mk.Stacks {
|
||||
named[app] = true
|
||||
}
|
||||
}
|
||||
return named, true
|
||||
}
|
||||
@@ -195,9 +195,11 @@ func (p *CrashPutBack) PutBackInterruptedRestore(stack, undoDir string) error {
|
||||
if j.Stack != stack {
|
||||
return fmt.Errorf("the restore copy at %s belongs to %q, not %q", undoDir, j.Stack, stack)
|
||||
}
|
||||
// R-1256 H5: never pour the copy back under an app that could not be stopped — a refusal here is a failed
|
||||
// put-back: the guard holds the app and keeps the copy (R-1251 H2).
|
||||
if p.stop != nil {
|
||||
if err := p.stop(stack); err != nil {
|
||||
p.logger.Printf("[WARN] [appstop] %s: stopping before the put-back failed: %v (continuing)", stack, err)
|
||||
return fmt.Errorf("stopping %s before the put-back: %w", stack, err)
|
||||
}
|
||||
}
|
||||
mgr := &Manager{logger: p.logger, volumeReplayFrom: p.volReplay, undoVolumeRemoveFn: p.volRemove}
|
||||
|
||||
@@ -3,6 +3,7 @@ package backup
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/util"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -530,7 +531,7 @@ func (m *Manager) RestoreFromRecoveryUnitAtWith(stackName, unitDir string, opt U
|
||||
// without one (the unit holds no volume) only a database replay that began can have changed data — and with no
|
||||
// copy to pour back that is the R-379 half state: held, never started (operator ruling 2026-08-22).
|
||||
putBack := func(cause error, holdKey string, moved bool) (UnitRestoreResult, error) {
|
||||
return res, m.putBackAfterFailedRestore(stackName, cause, liveDef, undo, moved || undo != nil, opt.FilesPlaced, restartStack, holdKey)
|
||||
return res, m.putBackAfterFailedRestore(stackName, cause, liveDef, undo, moved || undo != nil, opt.FilesPlaced, restartStack, holdKey, settings.RestoreKindLocal)
|
||||
}
|
||||
// R-353: the count is captured even when the replay errors — a partial replay is a fact the
|
||||
// customer's sentence has to be built from, and discarding it on the error path is how Scenario C
|
||||
|
||||
@@ -71,9 +71,14 @@ func orNone(s string) string {
|
||||
// so and the kept copy is removed (the normal rule after a successful restore). A new restore of the app ends the watch
|
||||
// untouched.
|
||||
func (m *Manager) watchLateStart(stack, copyDir string, since time.Time, watch time.Duration) {
|
||||
m.setWatching(copyDir, true) // R-1342: the copy is the way back while the app may still start
|
||||
defer m.setWatching(copyDir, false)
|
||||
switch m.waitStableStart(stack, since, since.Add(watch)) {
|
||||
case stableAborted:
|
||||
return // a new restore of this app owns it now
|
||||
case stableRemoved:
|
||||
m.logger.Printf("[INFO] [backup] %s: the app was removed while the box waited for it to start — the watch ends (R-1342)", stack)
|
||||
return
|
||||
case stableNot:
|
||||
m.logger.Printf("[WARN] [backup] %s: still not running stably %s after its restore — the restore copy stays for support (%s) (R-1270)", stack, lateStartWatch, orNone(copyDir))
|
||||
return
|
||||
@@ -133,12 +138,26 @@ type containerSampler interface {
|
||||
ContainerSamples(name string) ([]ContainerSample, error)
|
||||
}
|
||||
|
||||
// StackRemovalReporter (R-1342) is the provider's optional answer to „was this app removed?" — true only when the app is
|
||||
// known and no longer installed. A watch that sees it ends quietly: an app the household removed did not „fail to
|
||||
// start". The production stackAdapter implements it (a compile-time assertion in cmd/controller/main.go).
|
||||
type StackRemovalReporter interface {
|
||||
StackRemoved(name string) bool
|
||||
}
|
||||
|
||||
// stackRemoved asks the provider; a provider without the answer never reads „removed".
|
||||
func (m *Manager) stackRemoved(stack string) bool {
|
||||
r, ok := m.stackProvider.(StackRemovalReporter)
|
||||
return ok && r.StackRemoved(stack)
|
||||
}
|
||||
|
||||
type stableResult int
|
||||
|
||||
const (
|
||||
stableYes stableResult = iota
|
||||
stableNot
|
||||
stableAborted
|
||||
stableRemoved // R-1342: the app was removed during the watch
|
||||
)
|
||||
|
||||
// stableSample is one look: ok, and the fingerprint (container id → restart count) a stable window must keep.
|
||||
@@ -201,6 +220,9 @@ func (m *Manager) waitStableFor(stack string, watchFrom, deadline time.Time, win
|
||||
if m.restoreBegunAfter(stack, watchFrom) {
|
||||
return stableAborted
|
||||
}
|
||||
if m.stackRemoved(stack) {
|
||||
return stableRemoved
|
||||
}
|
||||
ok, fp := m.stableSample(stack)
|
||||
switch {
|
||||
case !ok:
|
||||
@@ -226,13 +248,17 @@ func (m *Manager) releaseAfterStableStart(stack, kind string, undo *liveVolumeCo
|
||||
dir := undo.dir
|
||||
from := time.Now() // taken HERE, before the goroutine: a restore begun after this moment is a new one
|
||||
until := from.Add(stableReleaseWatch)
|
||||
m.setWatching(dir, true) // R-1342: „Törlés most" refuses the copy while the watch holds it
|
||||
go func() {
|
||||
defer m.setWatching(dir, false)
|
||||
switch m.waitStableStart(stack, from, until) {
|
||||
case stableYes:
|
||||
removeCopyDir(m, stack, dir)
|
||||
m.logger.Printf("[INFO] [backup] %s: ran stably for %s after its %s — the restore copy is released (R-1270)", stack, stableStartWindow, kind)
|
||||
case stableAborted:
|
||||
m.logger.Printf("[INFO] [backup] %s: a new restore began before the %s's start was stable — its restore copy stays for support (%s)", stack, kind, dir)
|
||||
case stableRemoved:
|
||||
m.logger.Printf("[INFO] [backup] %s: the app was removed during the stable-start check after its %s — no alarm; the restore copy, if the remove left it, is an unused one (%s) (R-1342)", stack, kind, dir)
|
||||
default:
|
||||
m.logger.Printf("[ERROR] [backup] %s: did not run stably within %s after its %s — the restore copy stays for support (%s) (R-1270)", stack, stableReleaseWatch, kind, dir)
|
||||
if m.restoreNotStartedNotify != nil {
|
||||
@@ -262,6 +288,8 @@ func (m *Manager) restoredAppStarted(stack string) error {
|
||||
return nil
|
||||
case stableAborted:
|
||||
return fmt.Errorf("a new restore of %s began", stack)
|
||||
case stableRemoved:
|
||||
return fmt.Errorf("%s was removed during the restore's start check", stack)
|
||||
}
|
||||
return fmt.Errorf("stack %s did not run stably for %s (it restarted or turned unhealthy) after restore", stack, verdictStableWindow)
|
||||
}
|
||||
|
||||
@@ -674,6 +674,12 @@ func (m *Manager) clearUpdateHoldAfterRestore(stackName string) {
|
||||
return
|
||||
}
|
||||
m.logger.Printf("[INFO] [backup] %s: restore completed — the update hold (set %s) is CLEARED", stackName, h.At)
|
||||
// R-1347: the hold store and the stack manager's in-memory flag go together, as in the operator's clear (D64 = B, R-1343).
|
||||
// Left set, the crash-loop watcher would not watch the restored app and the night leg would skip it until the next
|
||||
// controller restart. A running update is not touched (ForgetUpdateHeld refuses it). Pinned by TestR1347_*.
|
||||
if m.updateHeldForgetter != nil && m.updateHeldForgetter(stackName) {
|
||||
m.logger.Printf("[INFO] [backup] %s: the restored app is watched and updated again — its update-held flag is forgotten (R-1347)", stackName)
|
||||
}
|
||||
// R-671 (v0.272.0): the undo copies the hold kept describe the state this restore just replaced. They
|
||||
// were kept so the hold's data stayed recoverable; once the app is restored whole they are dead weight
|
||||
// (measured 2026-09-24 on 9202: three nextcloud copies, ~0.9 GiB, outlived the hold and nothing named
|
||||
@@ -691,6 +697,12 @@ func (m *Manager) SetUndoCopyRemover(fn func(stackName string) int) {
|
||||
m.undoCopyRemover = fn
|
||||
}
|
||||
|
||||
// SetUpdateHeldForgetter wires R-1347's seam: stacks.Manager.ForgetUpdateHeld. INIT-ONLY, main.go — pinned by
|
||||
// TestR1347_UpdateHeldForgetterIsWiredAtStartup (an AST walk). The backup package cannot import stacks.
|
||||
func (m *Manager) SetUpdateHeldForgetter(fn func(stackName string) bool) {
|
||||
m.updateHeldForgetter = fn
|
||||
}
|
||||
|
||||
// UpdateHeldStacks is the set of apps held stopped after a failed update (R-660, v0.268.0) — the
|
||||
// FOURTH way the product stops an app on purpose, and until v0.268.0 the one `classifyRunStates` did
|
||||
// not know: each hold's `app_update_held` was followed ~11 s later by an `app_start_failed` for the
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
// Package busymark writes the controller's „busy" file for the host agent (D65 = A, operator ruling 2026-10-29 15:50;
|
||||
// felhom.eu `03`, `04` §6.1).
|
||||
//
|
||||
// WHY: the operator can reboot the guest from the hub. The agent refuses while a backup, a restore, an off-site run or
|
||||
// an app update runs — but the controller's own work is invisible to the agent (it runs inside this container). So
|
||||
// while any of it runs, this package keeps `<data>/ctl/busy` fresh: line 1 the time (RFC 3339, UTC), line 2 what runs.
|
||||
// The agent reads it over one exact read-only grant (`pct exec <vmid> -- cat <the guest path>`), without Docker.
|
||||
//
|
||||
// FRESHNESS IS THE SIGNAL, NOT PRESENCE: the file is rewritten every Every (15 s) while busy and removed when idle.
|
||||
// A controller that is killed or stuck mid-backup stops rewriting it, and the agent (90 s) reads the old file as „not
|
||||
// busy" — a stuck controller is exactly the box a reboot is for. A file left by a previous run is removed at start.
|
||||
// Pinned by busymark_test.go.
|
||||
package busymark
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// FileName is the busy file inside the ctl directory (the directory of the hold-clear socket, 0700).
|
||||
const FileName = "busy"
|
||||
|
||||
// DefaultEvery is the rewrite cadence while busy.
|
||||
const DefaultEvery = 15 * time.Second
|
||||
|
||||
// Writer keeps the file in step with its probes.
|
||||
type Writer struct {
|
||||
Path string // <data>/ctl/busy
|
||||
Every time.Duration
|
||||
Now func() time.Time
|
||||
Logger *log.Logger
|
||||
|
||||
mu sync.Mutex
|
||||
probes []probe
|
||||
last string // what the last tick saw ("" = idle)
|
||||
}
|
||||
|
||||
type probe struct {
|
||||
what string
|
||||
busy func() bool
|
||||
}
|
||||
|
||||
// AddProbe adds one kind of work: what is the sentence the agent shows the operator, busy says whether it runs now.
|
||||
// A nil busy is ignored (a box without that component).
|
||||
func (w *Writer) AddProbe(what string, busy func() bool) {
|
||||
if busy == nil {
|
||||
return
|
||||
}
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
w.probes = append(w.probes, probe{what: what, busy: busy})
|
||||
}
|
||||
|
||||
// Busy returns the first running work's sentence, or "" when nothing runs.
|
||||
func (w *Writer) Busy() string {
|
||||
w.mu.Lock()
|
||||
ps := append([]probe(nil), w.probes...)
|
||||
w.mu.Unlock()
|
||||
for _, p := range ps {
|
||||
if p.busy() {
|
||||
return p.what
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (w *Writer) now() time.Time {
|
||||
if w.Now != nil {
|
||||
return w.Now()
|
||||
}
|
||||
return time.Now()
|
||||
}
|
||||
|
||||
func (w *Writer) logf(format string, args ...any) {
|
||||
if w.Logger != nil {
|
||||
w.Logger.Printf(format, args...)
|
||||
}
|
||||
}
|
||||
|
||||
// Tick runs one evaluation: write the file when busy, remove it when idle. Exported for the test and the start.
|
||||
func (w *Writer) Tick() {
|
||||
what := w.Busy()
|
||||
if what == "" {
|
||||
if err := os.Remove(w.Path); err != nil && !errors.Is(err, os.ErrNotExist) {
|
||||
w.logf("[WARN] [busymark] the busy file could not be removed: %v — the agent reads it as busy for up to 90 s more", err)
|
||||
}
|
||||
if w.last != "" {
|
||||
w.logf("[INFO] [busymark] idle again (was: %s) — an operator reboot of the box is no longer refused for it", w.last)
|
||||
}
|
||||
w.last = ""
|
||||
return
|
||||
}
|
||||
body := w.now().UTC().Format(time.RFC3339) + "\n" + what + "\n"
|
||||
if err := os.MkdirAll(filepath.Dir(w.Path), 0o700); err == nil {
|
||||
tmp := w.Path + ".tmp"
|
||||
if err = os.WriteFile(tmp, []byte(body), 0o600); err == nil {
|
||||
err = os.Rename(tmp, w.Path)
|
||||
}
|
||||
if err != nil {
|
||||
w.logf("[WARN] [busymark] the busy file could not be written: %v — an operator reboot is not refused for %s", err, what)
|
||||
}
|
||||
}
|
||||
if w.last != what {
|
||||
w.logf("[INFO] [busymark] busy: %s — an operator reboot of the box is refused until it ends", what)
|
||||
}
|
||||
w.last = what
|
||||
}
|
||||
|
||||
// Run removes a file left by an earlier run, then ticks every Every until ctx ends (and removes the file then).
|
||||
func (w *Writer) Run(ctx context.Context) {
|
||||
every := w.Every
|
||||
if every <= 0 {
|
||||
every = DefaultEvery
|
||||
}
|
||||
w.Tick()
|
||||
t := time.NewTicker(every)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
_ = os.Remove(w.Path)
|
||||
return
|
||||
case <-t.C:
|
||||
w.Tick()
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
package busymark
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// D65 = A: the agent refuses an operator reboot while the controller's busy file is fresh.
|
||||
|
||||
func newW(t *testing.T) (*Writer, *bool, *time.Time) {
|
||||
t.Helper()
|
||||
busy := false
|
||||
now := time.Date(2026, 10, 29, 16, 0, 0, 0, time.UTC)
|
||||
w := &Writer{Path: filepath.Join(t.TempDir(), "ctl", FileName), Now: func() time.Time { return now }}
|
||||
w.AddProbe("a backup or a restore is running", func() bool { return busy })
|
||||
return w, &busy, &now
|
||||
}
|
||||
|
||||
// Busy → the file holds the time and the sentence; idle → gone. RED-PROOF: skip the write → no file → FAILS;
|
||||
// skip the remove → the file stays → FAILS.
|
||||
func TestBusyFileFollowsTheWork(t *testing.T) {
|
||||
w, busy, now := newW(t)
|
||||
w.Tick()
|
||||
if _, err := os.Stat(w.Path); !os.IsNotExist(err) {
|
||||
t.Fatalf("idle: a busy file exists (%v)", err)
|
||||
}
|
||||
*busy = true
|
||||
w.Tick()
|
||||
raw, err := os.ReadFile(w.Path)
|
||||
if err != nil {
|
||||
t.Fatalf("busy: no file: %v", err)
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(string(raw)), "\n")
|
||||
if len(lines) != 2 || lines[0] != now.Format(time.RFC3339) || lines[1] != "a backup or a restore is running" {
|
||||
t.Fatalf("file %q", raw)
|
||||
}
|
||||
if fi, _ := os.Stat(w.Path); fi.Mode().Perm() != 0o600 {
|
||||
t.Fatalf("mode %v, want 0600", fi.Mode().Perm())
|
||||
}
|
||||
// it is REWRITTEN while busy: freshness is the agent's signal
|
||||
*now = now.Add(15 * time.Second)
|
||||
w.Tick()
|
||||
raw, _ = os.ReadFile(w.Path)
|
||||
if !strings.HasPrefix(string(raw), now.Format(time.RFC3339)) {
|
||||
t.Fatalf("not rewritten: %q", raw)
|
||||
}
|
||||
*busy = false
|
||||
w.Tick()
|
||||
if _, err := os.Stat(w.Path); !os.IsNotExist(err) {
|
||||
t.Fatalf("idle again: the busy file is still there (%v) — the agent would refuse a reboot for 90 s more", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The first probe that runs names the work; a nil probe is ignored.
|
||||
func TestBusyNamesTheRunningWork(t *testing.T) {
|
||||
w, _, _ := newW(t)
|
||||
w.AddProbe("an app update is running", func() bool { return true })
|
||||
w.AddProbe("ignored", nil)
|
||||
if got := w.Busy(); got != "an app update is running" {
|
||||
t.Fatalf("Busy() = %q", got)
|
||||
}
|
||||
}
|
||||
@@ -953,6 +953,7 @@
|
||||
"dashboard.tunnel_offline_title": "The secure internet connection is not connected now, so your apps cannot be reached from outside. Check the internet connection.",
|
||||
"dashboard.utolso_mentes": "Last backup:",
|
||||
"dashboard.vedett": "Protected",
|
||||
"dashboard.operator_reboot": "Felhom support restarted your box at %s.",
|
||||
"dashboard.vezerlopult": "Dashboard",
|
||||
"datapath.consequence.excluded": "This folder is temporary and is not backed up.",
|
||||
"datapath.consequence.import_excluded": "Copy the files to be processed in here. The app reads them and then deletes them from here — this folder is temporary and is not backed up.",
|
||||
@@ -2634,6 +2635,20 @@
|
||||
"kept.delete.type": "To delete it, type: %s",
|
||||
"kept.delete.mismatch": "The name you typed does not match (%s), so nothing was deleted.",
|
||||
"kept.deleted": "The kept data of %s is deleted.",
|
||||
"restore_copy.title": "Copies from before a restore",
|
||||
"restore_copy.hint": "When you restore an app, the box first saves the app’s data as it was. It deletes that copy by itself 30 days after the restore.",
|
||||
"restore_copy.restored_on": "restore on %s",
|
||||
"restore_copy.auto_delete": "automatically deleted on %s",
|
||||
"restore_copy.needed.hold": "Cannot be deleted: the app is stopped, and this copy is its way back. Support decides about it.",
|
||||
"restore_copy.needed.putback": "Cannot be deleted: an interrupted restore puts the data back from this copy.",
|
||||
"restore_copy.needed.unknown": "Cannot be deleted now: the box cannot tell whether this copy is still needed.",
|
||||
"restore_copy.needed.checking": "Cannot be deleted now: the box is still checking that the app started well after the restore (up to 30 minutes).",
|
||||
"restore_copy.delete": "Delete now",
|
||||
"restore_copy.warning": "This is the copy of %s’s data from before the restore on %s. If you delete it, the restore can no longer be undone, and this copy cannot be brought back.",
|
||||
"restore_copy.app_busy": "Cannot be deleted now: an operation runs on the app. Try again when it has ended.",
|
||||
"restore_copy.not_found": "This copy is no longer there, so nothing was deleted.",
|
||||
"restore_copy.failed": "The copy could not be deleted: %s",
|
||||
"flash.restore_copy.deleted": "The copy of %s from before its restore is deleted.",
|
||||
"kept.page.title": "Kept data",
|
||||
"kept.page.back": "Back",
|
||||
"kept.page.intro": "Data that removed apps left on the drives. You can look at it, load it into a new install, or delete it. The server never deletes any of it by itself.",
|
||||
@@ -2672,6 +2687,7 @@
|
||||
"api.escrow.confirm_expired": "The time to type the new code back has run out — your previous code still works. Start the process again.",
|
||||
"api.escrow.confirm_lost": "The new code no longer waits for confirmation — your previous code still works. Start the process again.",
|
||||
"api.escrow.confirm_store_failed": "The new code could not be saved — your previous code still works. Try again.",
|
||||
"api.escrow.confirm_store_unknown": "We cannot tell yet whether the new code was saved — the server did not answer in time. Keep both codes (the old one and the new one) and try again.",
|
||||
"api.escrow.confirm_words_mismatch": "The words do not match — check the code you wrote down.",
|
||||
"api.share.form_invalid": "Invalid form — reload the page.",
|
||||
"api.share.too_many_attempts": "Too many failed attempts, try again in 1 minute",
|
||||
|
||||
@@ -948,6 +948,7 @@
|
||||
"dashboard.tunnel_offline_title": "A biztonságos internetkapcsolat most nem kapcsolódik, ezért az alkalmazásaid kívülről nem érhetők el. Ellenőrizd az internetkapcsolatot.",
|
||||
"dashboard.utolso_mentes": "Utolsó mentés:",
|
||||
"dashboard.vedett": "Védett",
|
||||
"dashboard.operator_reboot": "A Felhom ügyfélszolgálata %s-kor újraindította a dobozt.",
|
||||
"dashboard.vezerlopult": "Vezérlőpult",
|
||||
"datapath.consequence.excluded": "Ez a mappa átmeneti, és nem készül róla biztonsági mentés.",
|
||||
"datapath.consequence.import_excluded": "Ide másold a feldolgozandó fájlokat. Az alkalmazás beolvassa, majd törli innen — ez a mappa átmeneti, és nem készül róla biztonsági mentés.",
|
||||
@@ -2622,6 +2623,20 @@
|
||||
"kept.delete.type": "A törléshez írd be: %s",
|
||||
"kept.delete.mismatch": "A beírt név nem egyezik (%s), ezért nem töröltünk semmit.",
|
||||
"kept.deleted": "A(z) %s megőrzött adatai törölve.",
|
||||
"restore_copy.title": "Visszaállítás előtti másolatok",
|
||||
"restore_copy.hint": "Visszaállításkor a doboz előbb elmenti az alkalmazás akkori adatait. Ezt a másolatot a visszaállítás után 30 nappal magától törli.",
|
||||
"restore_copy.restored_on": "visszaállítás: %s",
|
||||
"restore_copy.auto_delete": "automatikusan törlődik: %s",
|
||||
"restore_copy.needed.hold": "Nem törölhető: az alkalmazás le van állítva, és ez a másolat az út vissza. Az ügyfélszolgálat dönt róla.",
|
||||
"restore_copy.needed.putback": "Nem törölhető: egy félbeszakadt visszaállítás ebből a másolatból teszi vissza az adatokat.",
|
||||
"restore_copy.needed.unknown": "Most nem törölhető: a doboz nem tudja eldönteni, kell-e még ez a másolat.",
|
||||
"restore_copy.needed.checking": "Most nem törölhető: a doboz még figyeli, hogy az alkalmazás rendben elindult-e a visszaállítás után (legfeljebb 30 percig).",
|
||||
"restore_copy.delete": "Törlés most",
|
||||
"restore_copy.warning": "Ez a(z) %s adatainak másolata a %s-i visszaállítás előttről. Ha törlöd, a visszaállítás többé nem vonható vissza, és ez a másolat nem hozható vissza.",
|
||||
"restore_copy.app_busy": "Most nem törölhető: az alkalmazáson éppen egy művelet fut. Próbáld újra, ha véget ért.",
|
||||
"restore_copy.not_found": "Ez a másolat már nincs meg, ezért nem töröltünk semmit.",
|
||||
"restore_copy.failed": "A másolatot nem sikerült törölni: %s",
|
||||
"flash.restore_copy.deleted": "A(z) %s visszaállítás előtti másolata törölve.",
|
||||
"kept.page.title": "Megőrzött adatok",
|
||||
"kept.page.back": "Vissza",
|
||||
"kept.page.intro": "Az eltávolított alkalmazások lemezen maradt adatai. Megnézheted őket, betöltheted egy új telepítésbe, vagy törölheted. Magától a szerver soha nem töröl közülük semmit.",
|
||||
@@ -2660,6 +2675,7 @@
|
||||
"api.escrow.confirm_expired": "Lejárt az idő az új kód visszaírására — a régi kódod továbbra is érvényes. Indítsd újra a folyamatot.",
|
||||
"api.escrow.confirm_lost": "Az új kód már nem vár megerősítésre — a régi kódod továbbra is érvényes. Indítsd újra a folyamatot.",
|
||||
"api.escrow.confirm_store_failed": "Az új kódot nem sikerült elmenteni — a régi kódod továbbra is érvényes. Próbáld újra.",
|
||||
"api.escrow.confirm_store_unknown": "Nem tudjuk biztosan, hogy az új kód mentése sikerült-e — a központ nem válaszolt időben. Őrizd meg mindkét kódot (a régit és az újat is), és próbáld újra.",
|
||||
"api.escrow.confirm_words_mismatch": "A megadott szavak nem egyeznek — ellenőrizd a felírt kódot.",
|
||||
"api.share.form_invalid": "Érvénytelen űrlap — töltsd újra az oldalt.",
|
||||
"api.share.too_many_attempts": "Túl sok sikertelen próbálkozás, próbáld újra 1 perc múlva",
|
||||
|
||||
@@ -27,8 +27,10 @@ import (
|
||||
// notices (felhom.eu R-1304, R-1305; 1.5.7-stable is skipped — it broke existing TOTP sign-ins). Each new image reaches
|
||||
// a running box by the reconcile at start: TestEnsureTraefik_ANewImageRecreatesTraefik,
|
||||
// TestEnsureCloudflared_ANewImageRecreatesTheTunnel, TestReconcileFileBrowserImage_MovesOnlyTheImage.
|
||||
// traefik v3.7.17 since 2026-11-02: GHSA-xv5x-242w-7hf7 (high, v3.7.14–v3.7.16): an unauthenticated client with a
|
||||
// Negotiate/NTLM header holds a backend transport per connection until memory runs out, NTLM backend or not.
|
||||
const (
|
||||
TraefikImage = "traefik:v3.7.16"
|
||||
TraefikImage = "traefik:v3.7.17"
|
||||
CloudflaredImage = "cloudflare/cloudflared:2026.10.4"
|
||||
FileBrowserImage = "gtstef/filebrowser:1.5.8-stable"
|
||||
// FileBrowserImportMount is the in-container mount point NAME for the canonical drop-zone
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// D62 — restore_copy_deleted: info (the hub's vocabulary), the operator message only (no household sentence — the hub
|
||||
// bars the type from customers too), and the details carry who, app, the restore's date and the size.
|
||||
func TestD62_RestoreCopyDeletedEvent(t *testing.T) {
|
||||
n := &Notifier{}
|
||||
var et, sev, msg, cust string
|
||||
var det []byte
|
||||
n.pushFn = func(eventType, severity, message, customer string, details interface{}) {
|
||||
et, sev, msg, cust = eventType, severity, message, customer
|
||||
det, _ = json.Marshal(details)
|
||||
}
|
||||
n.NotifyRestoreCopyDeleted(RestoreCopyDeletedDetails{App: "vikunja", RestoreAt: "2026-10-29T08:00:00Z", SizeBytes: 4096,
|
||||
Who: "the household (dashboard sign-in) from 192.168.0.5", Path: "/x/backups/restore-undo/vikunja/20261029T080000Z"})
|
||||
if et != "restore_copy_deleted" || sev != "info" || cust != "" {
|
||||
t.Fatalf("type=%q severity=%q customer=%q", et, sev, cust)
|
||||
}
|
||||
for _, want := range []string{"vikunja", "2026-10-29T08:00:00Z", "4096", "the household"} {
|
||||
if !strings.Contains(msg, want) {
|
||||
t.Fatalf("message %q lacks %q", msg, want)
|
||||
}
|
||||
}
|
||||
for _, want := range []string{`"app":"vikunja"`, `"restore_at":"2026-10-29T08:00:00Z"`, `"size_bytes":4096`, `"who":"the household`} {
|
||||
if !strings.Contains(string(det), want) {
|
||||
t.Fatalf("details %s lack %s", det, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -163,6 +163,8 @@ type UpdateDetails struct {
|
||||
type AppDetails struct {
|
||||
StackName string `json:"stack_name,omitempty"`
|
||||
DisplayName string `json:"display_name,omitempty"`
|
||||
// Source (app_oom only, R-528): what saw the kill — "docker" (State.OOMKilled) or "cgroup" (the host's kernel counter).
|
||||
Source string `json:"source,omitempty"`
|
||||
}
|
||||
|
||||
// CrossDriveDetails holds structured data for cross-drive backup events.
|
||||
@@ -804,6 +806,12 @@ func (n *Notifier) NotifyAppStartFailures(apps []AppRunState) {
|
||||
// that stays OOM-marked does not repeat. "warning" — the hub vocabulary (R-329). Operator-only
|
||||
// hub-side (hub >= v0.114.0 registers it): the household sees the dashboard tag.
|
||||
func (n *Notifier) NotifyAppOOM(stack, container, startedAt string, kills int64, memLimit, peak string) {
|
||||
n.NotifyAppOOMFrom(stack, container, startedAt, kills, memLimit, peak, "")
|
||||
}
|
||||
|
||||
// NotifyAppOOMFrom is NotifyAppOOM with the source that saw the kill (R-528: "docker" or "cgroup"), carried in the
|
||||
// event's details. One event per container run, whichever source saw it.
|
||||
func (n *Notifier) NotifyAppOOMFrom(stack, container, startedAt string, kills int64, memLimit, peak, source string) {
|
||||
key := container + "|" + startedAt
|
||||
n.mu.Lock()
|
||||
if n.oomSeen == nil {
|
||||
@@ -816,7 +824,7 @@ func (n *Notifier) NotifyAppOOM(stack, container, startedAt string, kills int64,
|
||||
if first {
|
||||
n.emit("app_oom", "warning",
|
||||
fmt.Sprintf("Alkalmazás memóriája elfogyott: %s (%s) — egy folyamatát a memóriakorlát leállította", stack, container),
|
||||
AppDetails{StackName: stack, DisplayName: container})
|
||||
AppDetails{StackName: stack, DisplayName: container, Source: source})
|
||||
}
|
||||
if storm {
|
||||
n.logger.Printf("[ERROR] [notify] %s: container %s OOM STORM — %d kills in %d min (limit %s, peak %s)",
|
||||
@@ -1577,6 +1585,25 @@ func (n *Notifier) NotifyAppDefinitionRefused(app, detail string) {
|
||||
n.pushEventBoth("app_definition_refused", "warning", msg, "", AppDefinitionRefusedDetails{App: app, Refused: detail})
|
||||
}
|
||||
|
||||
// RestoreCopyDeletedDetails is restore_copy_deleted's record: who deleted which app's copy, the restore it protected,
|
||||
// and its size.
|
||||
type RestoreCopyDeletedDetails struct {
|
||||
App string `json:"app"`
|
||||
RestoreAt string `json:"restore_at"` // RFC 3339, UTC
|
||||
SizeBytes int64 `json:"size_bytes"`
|
||||
Who string `json:"who"`
|
||||
Path string `json:"path"`
|
||||
}
|
||||
|
||||
// NotifyRestoreCopyDeleted (D62, operator 2026-10-29): the household deleted a restore copy with „Törlés most" before
|
||||
// its 30 days. Info, operator-only on the hub (registered in both of its lists, hub D62): an audit line for support,
|
||||
// never a household mail. Pinned by TestD62_RestoreCopyDeletedEvent.
|
||||
func (n *Notifier) NotifyRestoreCopyDeleted(d RestoreCopyDeletedDetails) {
|
||||
msg := fmt.Sprintf("%s: the restore copy from before the restore of %s (%d bytes) was deleted by hand — %s. That restore can no longer be undone from this box.",
|
||||
d.App, d.RestoreAt, d.SizeBytes, d.Who)
|
||||
n.pushEventBoth("restore_copy_deleted", "info", msg, "", d)
|
||||
}
|
||||
|
||||
// healthSeverity is the event severity a health status would be sent at (R-647): the disabled path
|
||||
// names what it drops, and "warn" is a health STATUS, not a severity the hub knows.
|
||||
func healthSeverity(status string) string {
|
||||
|
||||
@@ -21,9 +21,11 @@ import (
|
||||
// problem, main.go). The result rides the next report as `operator_action_results: [{id, outcome,
|
||||
// message}]`; the hub lists an action until its result has arrived, then stops.
|
||||
//
|
||||
// THE LIST IS CLOSED. Four actions, and run_job names one of four jobs. Nothing here deletes data,
|
||||
// THE LIST IS CLOSED. Five actions, and run_job names one of four jobs. Nothing here deletes data,
|
||||
// starts a countdown or shortens one — `03` §4 asks for a signing key only to destroy or overwrite the
|
||||
// only copy, and none of these does. An unknown action, an unknown job or an argument out of range is
|
||||
// only copy, and none of these does. clear_hold (D64 = B, operator ruling 2026-10-29 12:10) lifts an app's
|
||||
// hold of ANY reason with no operator signature: it deletes nothing and starts nothing — the app stays
|
||||
// stopped until the household (or the operator) starts it, and a waiting put-back's copy stays on disk. An unknown action, an unknown job or an argument out of range is
|
||||
// answered `refused` and NOTHING is called. Pinned by TestOpActions_ClosedList (which fails when an
|
||||
// entry is added, so a new action is a decision, not an edit) and TestOpActions_UnknownIsRefused.
|
||||
//
|
||||
@@ -38,6 +40,7 @@ const (
|
||||
OpAbandonStop = "abandon_stop"
|
||||
OpAbandonExtend = "abandon_extend"
|
||||
OpRunJob = "run_job"
|
||||
OpClearHold = "clear_hold"
|
||||
|
||||
OutcomeDone = "done"
|
||||
OutcomeRefused = "refused"
|
||||
@@ -57,7 +60,7 @@ const (
|
||||
)
|
||||
|
||||
// operatorActionNames is THE closed list. Adding an entry fails TestOpActions_ClosedList by design.
|
||||
var operatorActionNames = []string{OpOffsiteBackupNow, OpAbandonStop, OpAbandonExtend, OpRunJob}
|
||||
var operatorActionNames = []string{OpOffsiteBackupNow, OpAbandonStop, OpAbandonExtend, OpRunJob, OpClearHold}
|
||||
|
||||
// operatorJobNames is the fixed set run_job may name — the scheduler's own job names (main.go).
|
||||
var operatorJobNames = []string{"fill-watch", "offsite-integrity", "offsite-proof", "disk-health-check"}
|
||||
@@ -83,6 +86,21 @@ func contains(list []string, v string) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
// validHoldStack is clear_hold's argument check: the same alphabet as backup.isSafeStackName (report cannot import
|
||||
// backup; REUSE.md „Safe-name validators").
|
||||
func validHoldStack(s string) bool {
|
||||
if s == "" || len(s) > 64 {
|
||||
return false
|
||||
}
|
||||
for _, c := range s {
|
||||
if (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '-' || c == '_' {
|
||||
continue
|
||||
}
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// ErrActionRefused, wrapped by a handler, turns its error into a `refused` outcome (the action could
|
||||
// not START — e.g. an off-site run already in flight) instead of `failed` (it started and went wrong).
|
||||
var ErrActionRefused = errors.New("refused")
|
||||
@@ -101,7 +119,7 @@ type OperatorActionResult struct {
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
// OperatorActionHandlers are the four doors, wired by main.go. A nil handler answers `refused`
|
||||
// OperatorActionHandlers are the five doors, wired by main.go. A nil handler answers `refused`
|
||||
// („not available on this box").
|
||||
type OperatorActionHandlers struct {
|
||||
// OffsiteBackupNow returns a refusal sentence when the run may not start (the household's
|
||||
@@ -111,6 +129,9 @@ type OperatorActionHandlers struct {
|
||||
AbandonExtend func(days int) (time.Time, error)
|
||||
// RunJob is Scheduler.RunNow: an error = not started (unknown/running); the channel = the job's end.
|
||||
RunJob func(name string) (<-chan error, error)
|
||||
// ClearHold lifts the app's hold in the RUNNING controller (D64 = B, R-1343) and returns the result
|
||||
// sentence. actionID names the action in the box's log. An error wrapping ErrActionRefused = nothing changed.
|
||||
ClearHold func(stack string, actionID int64) (string, error)
|
||||
// ResultReady sends an out-of-cycle report so a result reaches the hub in seconds (optional).
|
||||
ResultReady func()
|
||||
}
|
||||
@@ -317,6 +338,23 @@ func (o *OperatorActions) dispatch(a OperatorAction) {
|
||||
}
|
||||
o.finish(a, outcomeOf(jerr), msg)
|
||||
}()
|
||||
case OpClearHold:
|
||||
// The argument is checked HERE, before any handler is touched (the closed list's rule): one app name,
|
||||
// the backup package's strict alphabet (letters, digits, '-' and '_', at most 64).
|
||||
if !validHoldStack(a.Arg) {
|
||||
o.finish(a, OutcomeRefused, "the argument must be one app name (letters, digits, '-' and '_')")
|
||||
return
|
||||
}
|
||||
if o.h.ClearHold == nil {
|
||||
o.finish(a, OutcomeRefused, "not available on this box")
|
||||
return
|
||||
}
|
||||
msg, err := o.h.ClearHold(a.Arg, a.ID)
|
||||
if err != nil {
|
||||
o.finish(a, outcomeOf(err), err.Error())
|
||||
return
|
||||
}
|
||||
o.finish(a, OutcomeDone, msg)
|
||||
default:
|
||||
o.finish(a, OutcomeRefused, "unknown action")
|
||||
}
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
@@ -15,9 +16,11 @@ import (
|
||||
|
||||
// The list is CLOSED. This test fails when an action or a job is added, by design: a new entry is an
|
||||
// operator decision (no action may delete data, start a countdown or shorten one), not an edit. The
|
||||
// hub pins the same four in its own test (hub internal/store opactions_test.go).
|
||||
// hub pins the same list in its own test (hub internal/store opactions_test.go).
|
||||
// 2026-10-29: clear_hold added deliberately — operator ruling D64 = B (12:10): the hub's „Clear hold" works for
|
||||
// every hold reason with no operator signature. It deletes nothing and starts nothing.
|
||||
func TestOpActions_ClosedList(t *testing.T) {
|
||||
if got, want := OperatorActionNames(), []string{"abandon_extend", "abandon_stop", "offsite_backup_now", "run_job"}; !reflect.DeepEqual(got, want) {
|
||||
if got, want := OperatorActionNames(), []string{"abandon_extend", "abandon_stop", "clear_hold", "offsite_backup_now", "run_job"}; !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("operator actions = %v, want exactly %v — a new action needs the operator's word (decision 185)", got, want)
|
||||
}
|
||||
if got, want := OperatorJobNames(), []string{"disk-health-check", "fill-watch", "offsite-integrity", "offsite-proof"}; !reflect.DeepEqual(got, want) {
|
||||
@@ -56,6 +59,10 @@ func recordingHandlers(c *calls) OperatorActionHandlers {
|
||||
close(ch)
|
||||
return ch, nil
|
||||
},
|
||||
ClearHold: func(stack string, id int64) (string, error) {
|
||||
c.add("clear:" + stack)
|
||||
return "the hold on " + stack + " is cleared", nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -197,3 +204,73 @@ func TestOpActions_WireShapes(t *testing.T) {
|
||||
t.Fatalf("report carried %s", b)
|
||||
}
|
||||
}
|
||||
|
||||
// clear_hold (D64 = B): the argument is one app name in the strict alphabet — never a path, never empty, never two
|
||||
// names. A bad one is refused and the handler is NOT called (the closed list's rule: validate before any door).
|
||||
// RED-PROOF (2026-10-29): with validHoldStack returning true, this fails on „a refused action called [clear:…]".
|
||||
func TestOpActions_ClearHoldBadArgumentIsRefusedAndCallsNothing(t *testing.T) {
|
||||
c := &calls{}
|
||||
o := NewOperatorActions(context.Background(), recordingHandlers(c), nil)
|
||||
list := []OperatorAction{
|
||||
{ID: 1, Action: OpClearHold, Arg: ""},
|
||||
{ID: 2, Action: OpClearHold, Arg: "../settings"},
|
||||
{ID: 3, Action: OpClearHold, Arg: "immich paperless"},
|
||||
{ID: 4, Action: OpClearHold, Arg: "immich/x"},
|
||||
{ID: 5, Action: OpClearHold, Arg: strings.Repeat("a", 65)},
|
||||
{ID: 6, Action: OpClearHold, Arg: "immich\n"},
|
||||
{ID: 7, Action: OpClearHold, Arg: "."},
|
||||
}
|
||||
o.Reconcile(list)
|
||||
if got := c.get(); len(got) != 0 {
|
||||
t.Fatalf("a refused action called %v", got)
|
||||
}
|
||||
for _, a := range list {
|
||||
if r := resultFor(t, o, a.ID); r.Outcome != OutcomeRefused || !strings.Contains(r.Message, "one app name") {
|
||||
t.Errorf("#%d clear_hold(%q): %+v, want refused naming the rule", a.ID, a.Arg, r)
|
||||
}
|
||||
}
|
||||
// control: a good name does reach the handler
|
||||
o.Reconcile([]OperatorAction{{ID: 8, Action: OpClearHold, Arg: "paperless-ngx_2"}})
|
||||
if got := c.get(); len(got) != 1 || got[0] != "clear:paperless-ngx_2" {
|
||||
t.Fatalf("a valid name must reach the handler, calls %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// clear_hold (D64 = B): a valid name reaches the handler with the action's id, and its sentence is the result; a
|
||||
// handler refusal (the app has no hold) is answered refused, an error that is not a refusal failed.
|
||||
// RED-PROOF (2026-10-29): with the OpClearHold case removed from dispatch, this fails on „unknown action".
|
||||
func TestOpActions_ClearHoldDoneRefusedFailed(t *testing.T) {
|
||||
var gotStack string
|
||||
var gotID int64
|
||||
h := OperatorActionHandlers{ClearHold: func(stack string, id int64) (string, error) {
|
||||
gotStack, gotID = stack, id
|
||||
switch stack {
|
||||
case "notheld":
|
||||
return "", fmt.Errorf("%w: notheld has no hold — nothing was changed", ErrActionRefused)
|
||||
case "broken":
|
||||
return "", errors.New("saving settings: disk full")
|
||||
}
|
||||
return "the hold on " + stack + " is cleared (it was: held after a failed update)", nil
|
||||
}}
|
||||
o := NewOperatorActions(context.Background(), h, nil)
|
||||
o.Reconcile([]OperatorAction{{ID: 7, Action: OpClearHold, Arg: "kimai"}})
|
||||
if gotStack != "kimai" || gotID != 7 {
|
||||
t.Fatalf("handler called with (%q, %d), want (kimai, 7)", gotStack, gotID)
|
||||
}
|
||||
if r := resultFor(t, o, 7); r.Outcome != OutcomeDone || r.Message != "the hold on kimai is cleared (it was: held after a failed update)" {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
o.Reconcile([]OperatorAction{{ID: 8, Action: OpClearHold, Arg: "notheld"}, {ID: 9, Action: OpClearHold, Arg: "broken"}})
|
||||
if r := resultFor(t, o, 8); r.Outcome != OutcomeRefused || !strings.Contains(r.Message, "has no hold") {
|
||||
t.Errorf("not held: %+v", r)
|
||||
}
|
||||
if r := resultFor(t, o, 9); r.Outcome != OutcomeFailed {
|
||||
t.Errorf("a save error must be failed, not refused: %+v", r)
|
||||
}
|
||||
// nil handler → refused, not a crash
|
||||
o2 := NewOperatorActions(context.Background(), OperatorActionHandlers{}, nil)
|
||||
o2.Reconcile([]OperatorAction{{ID: 1, Action: OpClearHold, Arg: "kimai"}})
|
||||
if r := resultFor(t, o2, 1); r.Outcome != OutcomeRefused {
|
||||
t.Errorf("nil handler: %+v", r)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
package settings
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1340, second producer (found live 2026-10-29 on Tester 1): the hold's Why() — the hub's „Held apps" line and the
|
||||
// box's app_held event — still said „held after an off-site restore failed its database replay" for a LOCAL restore of
|
||||
// an app with no database whose put-back stop failed. It must name what the hold record carries.
|
||||
// RED-PROOF: seen failing on the old fixed sentence.
|
||||
func TestR1340_WhyNamesTheMixedCause(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
h RestoreHold
|
||||
want, not []string
|
||||
}{
|
||||
{"local put-back, no database", RestoreHold{Reason: HoldReasonRestoreMixed, At: "T", RestoreKind: RestoreKindLocal, MixedCause: MixedCausePutBack, FailedStep: "stopping the app before the put-back"},
|
||||
[]string{"local restore", "putting the live data back failed", "stopping the app before the put-back"}, []string{"database", "off-site"}},
|
||||
{"off-site database replay", RestoreHold{Reason: HoldReasonRestoreMixed, At: "T", RestoreKind: RestoreKindOffsite, MixedCause: MixedCausePutBack, DBReplay: true},
|
||||
[]string{"off-site restore", "database replay", "putting the live data back failed"}, nil},
|
||||
{"definition write", RestoreHold{Reason: HoldReasonRestoreMixed, At: "T", RestoreKind: RestoreKindOffsite, MixedCause: MixedCauseDefinitionWrite},
|
||||
[]string{"definition"}, []string{"database"}},
|
||||
{"interrupted put-back", RestoreHold{Reason: HoldReasonRestoreMixed, At: "T", MixedCause: MixedCauseInterruptedPutBack},
|
||||
[]string{"interrupted"}, []string{"database"}},
|
||||
{"legacy record", RestoreHold{Reason: HoldReasonRestoreMixed, At: "T"},
|
||||
[]string{"stopped half-way", "clear the hold"}, []string{"database", "off-site"}},
|
||||
} {
|
||||
got := tc.h.Why()
|
||||
for _, w := range tc.want {
|
||||
if !strings.Contains(got, w) {
|
||||
t.Errorf("%s: %q lacks %q", tc.name, got, w)
|
||||
}
|
||||
}
|
||||
for _, n := range tc.not {
|
||||
if strings.Contains(got, n) {
|
||||
t.Errorf("%s: %q says %q", tc.name, got, n)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1982,8 +1982,36 @@ type RestoreHold struct {
|
||||
// first stop or a repeat within 24 h (Trip >= 2 → the sentence says support is informed).
|
||||
UnhealthyKind string `json:"unhealthy_kind,omitempty"`
|
||||
Trip int `json:"trip,omitempty"`
|
||||
|
||||
// R-1340 (2026-10-29): what put a restore hold in place, so the operator's „HELD STOPPED" mail tells the real
|
||||
// story instead of the option-C database story for every cause. All empty on a hold written before them: the mail
|
||||
// then names only what the record carries (cmd/controller heldStoppedOperatorText).
|
||||
//
|
||||
// RestoreKind is which restore: RestoreKindOffsite or RestoreKindLocal.
|
||||
RestoreKind string `json:"restore_kind,omitempty"`
|
||||
// MixedCause (restore_mixed only) is the branch that held the app: MixedCause*.
|
||||
MixedCause string `json:"mixed_cause,omitempty"`
|
||||
// FailedStep is the put-back step that failed, with its error (MixedCausePutBack).
|
||||
FailedStep string `json:"failed_step,omitempty"`
|
||||
// DBReplay: the restore's own failure was its database replay. False = the app's database was never touched
|
||||
// by the failure, so the mail never says „database".
|
||||
DBReplay bool `json:"db_replay,omitempty"`
|
||||
}
|
||||
|
||||
// Restore kinds and mixed-hold causes (R-1340). See RestoreHold.RestoreKind / MixedCause.
|
||||
const (
|
||||
RestoreKindOffsite = "offsite"
|
||||
RestoreKindLocal = "local"
|
||||
|
||||
// MixedCauseDefinitionWrite: the backup's definition could not be written (no data moved yet) and writing the
|
||||
// live definition back failed too (R-1256 H4).
|
||||
MixedCauseDefinitionWrite = "definition_write"
|
||||
// MixedCausePutBack: the restore failed and putting the live data back failed at FailedStep (D20/R-1256 H5).
|
||||
MixedCausePutBack = "put_back"
|
||||
// MixedCauseInterruptedPutBack: a restore cut by a stop could not be put back at the next start (R-1251 H2).
|
||||
MixedCauseInterruptedPutBack = "interrupted_put_back"
|
||||
)
|
||||
|
||||
// Hold reasons. See RestoreHold.Reason.
|
||||
const (
|
||||
HoldReasonRestoreFailed = "" // R-379/R-380: a restore AND its rollback failed
|
||||
@@ -2023,7 +2051,7 @@ func (h RestoreHold) Why() string {
|
||||
}
|
||||
return "stopped by the box for a " + strings.ReplaceAll(kind, "_", " ") + " (" + h.At + ") — the household's Start gives it one more try"
|
||||
case HoldReasonRestoreMixed:
|
||||
return "held after an off-site restore failed its database replay (" + h.At + ") — clear the hold to start it"
|
||||
return h.mixedWhy() + " (" + h.At + ") — clear the hold to start it"
|
||||
case HoldReasonPutBackPending:
|
||||
return "held until the pre-restore copy is poured back (" + h.At + ") — the put-back starts it"
|
||||
default:
|
||||
@@ -2031,6 +2059,36 @@ func (h RestoreHold) Why() string {
|
||||
}
|
||||
}
|
||||
|
||||
// mixedWhy (R-1340) names a restore_mixed hold from what its record carries — never „database" or „off-site" unless the
|
||||
// record says so. A record written before these fields („legacy") gets the neutral sentence. Pinned by
|
||||
// TestR1340_WhyNamesTheMixedCause.
|
||||
func (h RestoreHold) mixedWhy() string {
|
||||
kind := "a restore"
|
||||
switch h.RestoreKind {
|
||||
case RestoreKindOffsite:
|
||||
kind = "an off-site restore"
|
||||
case RestoreKindLocal:
|
||||
kind = "a local restore"
|
||||
}
|
||||
switch h.MixedCause {
|
||||
case MixedCauseDefinitionWrite:
|
||||
return "held after " + kind + " could not write its definition and writing the live definition back failed too"
|
||||
case MixedCauseInterruptedPutBack:
|
||||
return "held after a restore was interrupted and its live data could not be put back at the next start"
|
||||
case MixedCausePutBack:
|
||||
what := "failed"
|
||||
if h.DBReplay {
|
||||
what = "failed its database replay"
|
||||
}
|
||||
s := "held after " + kind + " " + what + " and putting the live data back failed"
|
||||
if h.FailedStep != "" {
|
||||
s += " (" + h.FailedStep + ")"
|
||||
}
|
||||
return s
|
||||
}
|
||||
return "held after a restore stopped half-way"
|
||||
}
|
||||
|
||||
// SetRestoreHold records a hold. Modelled on SetDisconnected: a condition, plus what it is holding.
|
||||
func (s *Settings) SetRestoreHold(h RestoreHold) error {
|
||||
s.mu.Lock()
|
||||
|
||||
@@ -28,7 +28,11 @@ import (
|
||||
// of the running version; the PREVIOUS — the self-update's own record (update-state.json previous_image of a swap
|
||||
// that ended on the running version), or, when there is no such record or its image is gone, the highest version
|
||||
// below the running one (logged as "by version order"); every version ABOVE the running one (a pulled target the
|
||||
// swap has not taken yet); every tag that is not a plain X.Y.Z (latest, -rc) — left alone, logged. A candidate is an
|
||||
// swap has not taken yet); every tag that is not a version at all (latest) — left alone, logged. A test build's tag
|
||||
// (X.Y.Z-rc.N) IS a version since v0.351.0 (decision 56, amended 2026-11-09): util.ParseVersionPre orders it below its release, so an old rc is a
|
||||
// candidate like an old release — 25 of them (2.2 GB) sat on demo-hp on 2026-11-09 because the rule left every -rc
|
||||
// alone. The previous by version order prefers a release; a test build is the previous only when no release is below
|
||||
// the running one. A box that RUNS a test build is judged too (it was skipped). A candidate is an
|
||||
// untagged (<none>) controller image or one whose every tag is a version below the previous. Deleted by exact
|
||||
// name/ID, never forced, never by prune; an ID that also carries another repository's name is left alone. NOTHING is
|
||||
// deleted while the controller is swapping itself (selfUpdatingNow). Registry tags are never touched.
|
||||
@@ -45,7 +49,7 @@ type ControllerImageRecord struct {
|
||||
func (m *Manager) RetainControllerImages(rec ControllerImageRecord) ([]string, error) {
|
||||
imageRetentionMu.Lock()
|
||||
defer imageRetentionMu.Unlock()
|
||||
running, err := util.ParseVersion(rec.Running)
|
||||
running, err := util.ParseVersionPre(rec.Running)
|
||||
if err != nil || rec.Repo == "" {
|
||||
m.logger.Printf("[INFO] [stacks] controller image retention: skipped — running version %q is not a release (or no repository)", rec.Running)
|
||||
return nil, nil
|
||||
@@ -82,15 +86,27 @@ func (m *Manager) RetainControllerImages(rec ControllerImageRecord) ([]string, e
|
||||
}
|
||||
}
|
||||
if prevTag == "" {
|
||||
var best *util.Version
|
||||
// A release first: a box on 0.350.0 holding 0.349.0 and 0.350.0-rc.3 keeps 0.349.0 for a hand roll-back.
|
||||
var best, bestPre *util.Version
|
||||
for _, im := range imgs {
|
||||
if im.Repo != repo {
|
||||
continue
|
||||
}
|
||||
if v, err := util.ParseVersion(im.Tag); err == nil && v.Compare(running) < 0 && (best == nil || v.Compare(*best) > 0) {
|
||||
v, err := util.ParseVersionPre(im.Tag)
|
||||
if err != nil || v.Compare(running) >= 0 {
|
||||
continue
|
||||
}
|
||||
vv := v
|
||||
if v.Pre == "" {
|
||||
if best == nil || v.Compare(*best) > 0 {
|
||||
best = &vv
|
||||
}
|
||||
} else if bestPre == nil || v.Compare(*bestPre) > 0 {
|
||||
bestPre = &vv
|
||||
}
|
||||
}
|
||||
if best == nil {
|
||||
best = bestPre
|
||||
}
|
||||
if best != nil {
|
||||
prevTag, prevHow = best.Raw, "by version order (no swap record names one present)"
|
||||
@@ -98,7 +114,7 @@ func (m *Manager) RetainControllerImages(rec ControllerImageRecord) ([]string, e
|
||||
}
|
||||
var prev *util.Version
|
||||
if prevTag != "" {
|
||||
if v, err := util.ParseVersion(prevTag); err == nil {
|
||||
if v, err := util.ParseVersionPre(prevTag); err == nil {
|
||||
prev = &v
|
||||
}
|
||||
}
|
||||
@@ -140,10 +156,10 @@ func (m *Manager) RetainControllerImages(rec ControllerImageRecord) ([]string, e
|
||||
continue
|
||||
}
|
||||
names = append(names, im.Repo+":"+im.Tag)
|
||||
v, err := util.ParseVersion(im.Tag)
|
||||
v, err := util.ParseVersionPre(im.Tag)
|
||||
switch {
|
||||
case err != nil:
|
||||
deletable, odd = false, im.Tag // latest, -rc: not ours to judge
|
||||
deletable, odd = false, im.Tag // latest: not a version, not ours to judge
|
||||
case v.Compare(running) >= 0:
|
||||
deletable = false // the running one, or a pulled target not swapped to yet
|
||||
case prev == nil || v.Compare(*prev) >= 0:
|
||||
|
||||
@@ -48,16 +48,17 @@ func TestControllerRetention_KeepsRunningAndPreviousDeletesOlder(t *testing.T) {
|
||||
if strings.Contains(s, ":0.284.2,") {
|
||||
t.Fatalf("the previous controller (0.284.2) was deleted: %v", f.rmi)
|
||||
}
|
||||
for _, want := range []string{ctlRepo + ":0.284.1", ctlRepo + ":0.283.1", "sha256:CNONE"} {
|
||||
// 0.273.0-rc1: a test build below the previous goes like an old release (decision 56 amended 2026-11-09).
|
||||
for _, want := range []string{ctlRepo + ":0.284.1", ctlRepo + ":0.283.1", "sha256:CNONE", ctlRepo + ":0.273.0-rc1"} {
|
||||
if !strings.Contains(s, ","+want+",") {
|
||||
t.Fatalf("%s (older than the previous / untagged) was not deleted: rmi=%v", want, f.rmi)
|
||||
}
|
||||
}
|
||||
if strings.Contains(s, "0.273.0-rc1") || strings.Contains(s, "acme/web") || strings.Contains(s, "W3") {
|
||||
t.Fatalf("a non-version tag or another repository's image was touched: %v", f.rmi)
|
||||
if strings.Contains(s, "acme/web") || strings.Contains(s, "W3") {
|
||||
t.Fatalf("another repository's image was touched: %v", f.rmi)
|
||||
}
|
||||
if len(got) != 3 {
|
||||
t.Fatalf("deleted %d, want 3: %v", len(got), got)
|
||||
if len(got) != 4 {
|
||||
t.Fatalf("deleted %d, want 4: %v", len(got), got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -183,3 +184,67 @@ func TestRetainImagesAfterUpdate_AppGoneDoesNotPanic(t *testing.T) {
|
||||
t.Fatalf("deleted images for an app that is gone: %v", f.rmi)
|
||||
}
|
||||
}
|
||||
|
||||
// Decision 56 amended 2026-11-09: demo-hp's guest held 25 old test builds (X.Y.Z-rc.N, 2.2 GB) because every -rc tag was
|
||||
// "left alone". A test build is now a version (util.ParseVersionPre: below its release): an old one goes; a test build
|
||||
// ABOVE the running one (a pulled target) and one BETWEEN the previous and the running one stay; `latest` stays.
|
||||
// COMPANION RED-PROOF: parse tags with util.ParseVersion again → "an old test build (0.330.0-rc.2) was not deleted".
|
||||
func TestControllerRetention_OldTestBuildsGo(t *testing.T) {
|
||||
m := retentionManager(t)
|
||||
f := &fakeImages{
|
||||
imgs: []localImage{
|
||||
{ID: "sha256:R350", Repo: ctlRepo, Tag: "0.350.0", Size: "348MB"},
|
||||
{ID: "sha256:R349", Repo: ctlRepo, Tag: "0.349.0", Size: "348MB"},
|
||||
{ID: "sha256:RC350", Repo: ctlRepo, Tag: "0.350.0-rc.3", Size: "348MB"},
|
||||
{ID: "sha256:RC330", Repo: ctlRepo, Tag: "0.330.0-rc.2", Size: "347MB"},
|
||||
{ID: "sha256:RC351", Repo: ctlRepo, Tag: "0.351.0-rc.1", Size: "348MB"},
|
||||
{ID: "sha256:LATEST", Repo: ctlRepo, Tag: "latest", Size: "348MB"},
|
||||
},
|
||||
containers: map[string]string{"c-ctl": "sha256:R350"},
|
||||
}
|
||||
withFakeImages(t, f)
|
||||
// No swap record: the previous by version order is the RELEASE 0.349.0, not the newer test build 0.350.0-rc.3.
|
||||
if _, err := m.RetainControllerImages(ControllerImageRecord{Repo: ctlRepo, Running: "0.350.0"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := rmiSet(f)
|
||||
if !strings.Contains(s, ","+ctlRepo+":0.330.0-rc.2,") {
|
||||
t.Fatalf("an old test build (0.330.0-rc.2) was not deleted: %v", f.rmi)
|
||||
}
|
||||
if strings.Contains(s, ":0.349.0,") {
|
||||
t.Fatalf("the previous RELEASE (0.349.0) was deleted — version order must prefer a release over a test build: %v", f.rmi)
|
||||
}
|
||||
if strings.Contains(s, ":0.351.0-rc.1,") || strings.Contains(s, ":latest,") || strings.Contains(s, ":0.350.0,") {
|
||||
t.Fatalf("a newer test build, latest, or the running one was deleted: %v", f.rmi)
|
||||
}
|
||||
// 0.350.0-rc.3 sits between the previous (0.349.0) and the running one: kept, like a release there would be.
|
||||
if strings.Contains(s, ":0.350.0-rc.3,") {
|
||||
t.Fatalf("a test build between the previous and the running one was deleted: %v", f.rmi)
|
||||
}
|
||||
}
|
||||
|
||||
// A box that RUNS a test build (Tester 1 during an rc round) is judged too — it was skipped before.
|
||||
// COMPANION RED-PROOF: parse rec.Running with util.ParseVersion again → "a box running a test build deleted nothing".
|
||||
func TestControllerRetention_BoxOnATestBuildIsJudged(t *testing.T) {
|
||||
m := retentionManager(t)
|
||||
f := &fakeImages{
|
||||
imgs: []localImage{
|
||||
{ID: "sha256:RC2", Repo: ctlRepo, Tag: "0.351.0-rc.2", Size: "348MB"},
|
||||
{ID: "sha256:RC1", Repo: ctlRepo, Tag: "0.351.0-rc.1", Size: "348MB"},
|
||||
{ID: "sha256:R350", Repo: ctlRepo, Tag: "0.350.0", Size: "348MB"},
|
||||
{ID: "sha256:R349", Repo: ctlRepo, Tag: "0.349.0", Size: "348MB"},
|
||||
},
|
||||
containers: map[string]string{"c-ctl": "sha256:RC2"},
|
||||
}
|
||||
withFakeImages(t, f)
|
||||
if _, err := m.RetainControllerImages(ControllerImageRecord{Repo: ctlRepo, Running: "0.351.0-rc.2", Previous: ctlRepo + ":0.350.0"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := rmiSet(f)
|
||||
if !strings.Contains(s, ","+ctlRepo+":0.349.0,") {
|
||||
t.Fatalf("a box running a test build deleted nothing: %v", f.rmi)
|
||||
}
|
||||
if strings.Contains(s, ":0.350.0,") || strings.Contains(s, ":0.351.0-rc.2,") || strings.Contains(s, ":0.351.0-rc.1,") {
|
||||
t.Fatalf("the recorded previous, the running test build or one between them was deleted: %v", f.rmi)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
package stacks
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
|
||||
)
|
||||
|
||||
// D64 = B / R-1343: when the operator clears a hold, the ONE held flag kept outside the hold store — „the last update
|
||||
// ended held" — goes too. Left set, the crash-loop observer and the nightly update leg kept skipping an app whose hold
|
||||
// was gone, until the controller restarted.
|
||||
// RED-PROOF (2026-10-29): with ForgetUpdateHeld's body reduced to `return false`, this fails on „a held update must be forgotten".
|
||||
func TestD64_ForgetUpdateHeld(t *testing.T) {
|
||||
m := &Manager{cfg: &config.Config{}, logger: log.New(io.Discard, "", 0), stacks: map[string]*Stack{
|
||||
"kimai": {Name: "kimai", Deployed: true},
|
||||
"running": {Name: "running", Deployed: true, Updating: true},
|
||||
}}
|
||||
m.finishUpdateKey("kimai", UpdatePhaseFailed, UpdateErrorKeyHeld, "a kimai leállítva marad")
|
||||
m.markUpdateHeld("kimai")
|
||||
m.markUpdateHeld("running")
|
||||
|
||||
if !m.ForgetUpdateHeld("kimai") {
|
||||
t.Fatal("a held update must be forgotten")
|
||||
}
|
||||
st, _ := m.GetStack("kimai")
|
||||
if st.updateHeld || st.UpdateErrorKey != "" || st.UpdateError != "" || st.UpdatePhase != "" {
|
||||
t.Fatalf("held state left: held=%v key=%q err=%q phase=%q", st.updateHeld, st.UpdateErrorKey, st.UpdateError, st.UpdatePhase)
|
||||
}
|
||||
// The two readers of the flag (ObserveUnhealthy's skip, unhealthy.go; legCandidate's LegSkipHeld, unattended.go)
|
||||
// read the stored stack — not called here, because ObserveUnhealthy reaches docker (R-650).
|
||||
m.mu.RLock()
|
||||
skipped := m.stacks["kimai"].updateHeld || m.stacks["kimai"].UpdateErrorKey == UpdateErrorKeyHeld
|
||||
m.mu.RUnlock()
|
||||
if skipped {
|
||||
t.Fatal("the observer and the update leg still skip kimai")
|
||||
}
|
||||
// A running update is never touched; an app with nothing held reports false.
|
||||
if m.ForgetUpdateHeld("running") {
|
||||
t.Fatal("a running update must not be touched")
|
||||
}
|
||||
if m.ForgetUpdateHeld("kimai") || m.ForgetUpdateHeld("nosuch") {
|
||||
t.Fatal("nothing held → false")
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,8 @@ package stacks
|
||||
import (
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/util"
|
||||
)
|
||||
|
||||
// Desired-state values for AppConfig.DesiredState (R-166, decision D-b).
|
||||
@@ -57,7 +59,8 @@ func (m *Manager) SetDesiredState(name, desired string) error {
|
||||
|
||||
stack, ok := m.GetStack(name)
|
||||
if !ok {
|
||||
return fmt.Errorf("stack %q not found", name)
|
||||
// a KIND (404 at the API) and the household's sentence — not a save failure
|
||||
return util.MsgErrorf(ErrStackNotFound, "err.stacks.alkalmazas_nem_talalhato", name)
|
||||
}
|
||||
stackDir := filepath.Dir(stack.ComposePath)
|
||||
|
||||
@@ -157,6 +160,26 @@ func isObservedUp(s Stack) bool {
|
||||
}
|
||||
}
|
||||
|
||||
// RecordStopWantedIfDown (R-1350) is the operator's hold clear (D64 = B) speaking for the household: the clear starts
|
||||
// nothing, so an app that is NOT up stays stopped — and that stop is now WANTED, recorded as „stopped" so the dead-app
|
||||
// scan does not mail „Telepített alkalmazás nem fut" 30 s after the clear (measured 2026-10-29 on Tester 1), and the
|
||||
// boot reconciler does not start it at the next boot. A later Start records „running" and ends it. A running app's wish
|
||||
// is left alone (recorded=false). The clear is a PERSON's act (R-379: „a person — on the box, or the operator on the
|
||||
// hub"), so this is the ownership rule's own exception, not a machine writing intent.
|
||||
func (m *Manager) RecordStopWantedIfDown(name string) (bool, error) {
|
||||
st, ok := m.GetStack(name)
|
||||
if !ok {
|
||||
return false, nil
|
||||
}
|
||||
if isObservedUp(*st) {
|
||||
return false, nil
|
||||
}
|
||||
if err := m.SetDesiredState(name, DesiredStateStopped); err != nil {
|
||||
return false, err
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// WantsStopped (R-721, v0.283.0) reports whether the household's own recorded intent for the stack is
|
||||
// „stopped". The machines that stop an app for their own purposes and start it again afterwards — the
|
||||
// whole-guest backup's quiesce, the nightly volume dump — ask this before their restart, so a Stop the
|
||||
|
||||
@@ -336,3 +336,27 @@ func TestDesiredStateOf_NilAppConfigIsUnknown(t *testing.T) {
|
||||
t.Fatalf("a stack with no AppConfig reported desired state %q — absent must read as unknown", got)
|
||||
}
|
||||
}
|
||||
|
||||
// R-1350: the operator's hold clear records „stopped" for an app that is NOT up, and leaves a running app's wish alone
|
||||
// (a clear never stops anything). RED-PROOF: write „stopped" without the up check → the running case FAILS.
|
||||
func TestR1350_RecordStopWantedIfDown(t *testing.T) {
|
||||
m, root := newDSManager(t, "gokapi", "immich")
|
||||
for _, n := range []string{"gokapi", "immich"} {
|
||||
writeAppYAML(t, stackDirOf(root, n), "deployed: true\ndesired_state: running\n")
|
||||
m.stacks[n].AppConfig = LoadAppConfig(stackDirOf(root, n))
|
||||
}
|
||||
m.stacks["gokapi"].State = StateStopped
|
||||
m.stacks["immich"].State = StateRunning
|
||||
if rec, err := m.RecordStopWantedIfDown("gokapi"); err != nil || !rec {
|
||||
t.Fatalf("gokapi: recorded=%v err=%v", rec, err)
|
||||
}
|
||||
if got := LoadAppConfig(stackDirOf(root, "gokapi")).DesiredState; got != DesiredStateStopped {
|
||||
t.Fatalf("gokapi desired_state on disk = %q, want stopped", got)
|
||||
}
|
||||
if rec, err := m.RecordStopWantedIfDown("immich"); err != nil || rec {
|
||||
t.Fatalf("immich (running): recorded=%v err=%v", rec, err)
|
||||
}
|
||||
if got := LoadAppConfig(stackDirOf(root, "immich")).DesiredState; got != DesiredStateRunning {
|
||||
t.Fatalf("a running app's wish was changed to %q", got)
|
||||
}
|
||||
}
|
||||
@@ -109,7 +109,7 @@ func (m *Manager) ensureFileBrowserPublic() {
|
||||
}
|
||||
if m.fbNetOn(target, traefikNetwork) {
|
||||
if err := m.fbNetDisconnect(target, traefikNetwork); err != nil {
|
||||
m.logger.Printf("[WARN] [infra] filebrowser: could not leave %s — every app can still reach it (R-1281): %v", traefikNetwork, err)
|
||||
m.reportFileBrowserNotLeft(target, err)
|
||||
} else {
|
||||
m.logger.Printf("[INFO] [infra] filebrowser: left %s — no app container reaches it any more (R-1281)", traefikNetwork)
|
||||
}
|
||||
@@ -117,6 +117,27 @@ func (m *Manager) ensureFileBrowserPublic() {
|
||||
m.ensureOfficeOnFilesNetwork()
|
||||
}
|
||||
|
||||
// reportFileBrowserNotLeft says what a failed leave means (R-1323). „No such container" means the CHECKED container was
|
||||
// replaced between the check and the leave (the image moved at start, a mounts sync): the line then reads the CURRENT
|
||||
// container's networks and says the truth. A false „every app can still reach it" teaches the reader to ignore the real
|
||||
// one, which stays: any other failure, a current container on the app network, or one that cannot be read. The new
|
||||
// container is checked and published by the next round. Pinned by TestR1323_*.
|
||||
func (m *Manager) reportFileBrowserNotLeft(target string, err error) {
|
||||
if !strings.Contains(err.Error(), "No such container") {
|
||||
m.logger.Printf("[WARN] [infra] filebrowser: could not leave %s — every app can still reach it (R-1281): %v", traefikNetwork, err)
|
||||
return
|
||||
}
|
||||
cur := strings.Fields(m.fbStartedAt())
|
||||
switch {
|
||||
case len(cur) == 0:
|
||||
m.logger.Printf("[WARN] [infra] filebrowser: the checked container %s is gone and the current one cannot be read — whether every app can reach it is not known (R-1281, R-1323)", target)
|
||||
case m.fbNetOn(cur[0], traefikNetwork):
|
||||
m.logger.Printf("[WARN] [infra] filebrowser: could not leave %s — every app can still reach it (R-1281): the checked container %s was replaced, and the current one %s IS on it", traefikNetwork, target, cur[0])
|
||||
default:
|
||||
m.logger.Printf("[INFO] [infra] filebrowser: the checked container %s was replaced before it left %s; the current one %s is not on %s (R-1323)", target, traefikNetwork, cur[0], traefikNetwork)
|
||||
}
|
||||
}
|
||||
|
||||
// traefikContainer is traefik's container (infra/templates/traefik-compose.yml.tmpl container_name).
|
||||
const traefikContainer = "traefik"
|
||||
|
||||
|
||||
@@ -254,8 +254,28 @@ func changedInfraFiles(dir string, files map[string]infra.FileSpec) []string {
|
||||
return changed
|
||||
}
|
||||
|
||||
// LockFileBrowserStack takes the lock over every `docker compose` on FileBrowser's stack at a boot (R-1351): this file's
|
||||
// ensureFileBrowser and web.syncFileBrowserMounts. Returns the unlock. Lock order: infraMu (the base stack) before this;
|
||||
// the web sync takes only this one, and starts the gate (infraMu) in a goroutine. Pinned by TestR1351_* and
|
||||
// TestR1351_TheWebSyncTakesTheLockBeforeCompose (internal/web).
|
||||
func (m *Manager) LockFileBrowserStack() (unlock func()) {
|
||||
m.fbStackMu.Lock()
|
||||
return m.fbStackMu.Unlock
|
||||
}
|
||||
|
||||
// WithFileBrowserStack runs fn under LockFileBrowserStack.
|
||||
func (m *Manager) WithFileBrowserStack(fn func()) {
|
||||
defer m.LockFileBrowserStack()()
|
||||
fn()
|
||||
}
|
||||
|
||||
func (m *Manager) ensureFileBrowser(dir string) error {
|
||||
if containerRunning("filebrowser") {
|
||||
defer m.LockFileBrowserStack()()
|
||||
running := containerRunning
|
||||
if m.fbRunningFn != nil {
|
||||
running = func(string) bool { return m.fbRunningFn() }
|
||||
}
|
||||
if running("filebrowser") {
|
||||
return m.reconcileFileBrowserImage(dir)
|
||||
}
|
||||
composePath := filepath.Join(dir, "docker-compose.yml")
|
||||
@@ -263,6 +283,9 @@ func (m *Manager) ensureFileBrowser(dir string) error {
|
||||
// Already provisioned but not running — bring it up WITHOUT regenerating, so the storage
|
||||
// mounts that web.SyncFileBrowserMounts manages are preserved. Just `compose up -d`.
|
||||
m.logger.Printf("[INFO] [infra] filebrowser compose exists — starting without regenerating")
|
||||
if m.fbComposeUpFn != nil {
|
||||
return m.fbComposeUpFn(dir)
|
||||
}
|
||||
return m.composeUp(dir)
|
||||
}
|
||||
m.logger.Printf("[INFO] [infra] deploying filebrowser → %s", dir)
|
||||
|
||||
@@ -0,0 +1,186 @@
|
||||
package stacks
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/dockerexec"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/util"
|
||||
)
|
||||
|
||||
// ── Infrastructure image retention (`09` §3 decision 285, operator 2026-11-09) ──────────────────────────
|
||||
//
|
||||
// Decision 53 cleans app images and decision 56 the controller's own; the pinned infrastructure images (traefik,
|
||||
// cloudflared, FileBrowser, Samba — infra.Images()) were cleaned by nothing, so every pin a box ever ran stayed:
|
||||
// on demo-hp's guest on 2026-11-09, 2 old traefik, 3 old cloudflared, 2 old FileBrowser and 1 old Samba image
|
||||
// (about 1 GB). Every one of them rides the whole-guest backup (mp0 holds /var/lib/docker), on a system disk the
|
||||
// local tier already struggles to fit (R-548).
|
||||
//
|
||||
// THE RULE, decision 56's shape applied per pinned repository: a box keeps the PINNED tag (what this controller
|
||||
// runs), every version ABOVE it (a box rolled back to an older controller keeps the newer pin's image for the way
|
||||
// forward), the highest version BELOW it (the previous pin — a hand roll-back), and every image a container uses
|
||||
// (running or stopped). An untagged image of the repository that no container uses is a candidate; a tag that is
|
||||
// not a version (latest) is left alone. Deleted by exact name/ID, never forced, never by prune; an ID that also
|
||||
// carries another repository's name is left alone. No pass while an image-pulling compose command runs (R-863),
|
||||
// and none while the controller swaps itself. Pinned by internal/stacks/infra_image_retention_test.go.
|
||||
|
||||
// RetainInfraImages applies the rule once to every pinned infrastructure image. Returns what it deleted, and
|
||||
// errImageRetentionBusy when image work was in flight (nothing was judged; try again later).
|
||||
func (m *Manager) RetainInfraImages(pins []string) ([]string, error) {
|
||||
imageRetentionMu.Lock()
|
||||
defer imageRetentionMu.Unlock()
|
||||
if m.selfUpdatingNow() {
|
||||
m.logger.Printf("[INFO] [stacks] infra image retention: skipped — the controller is swapping itself")
|
||||
return nil, errImageRetentionBusy
|
||||
}
|
||||
endCleanup, ok := dockerexec.TryImageCleanup()
|
||||
if !ok {
|
||||
m.logger.Printf("[INFO] [stacks] infra image retention: skipped — an app install, update, restore or undo is pulling images now (R-863); tried again later")
|
||||
return nil, errImageRetentionBusy
|
||||
}
|
||||
defer endCleanup()
|
||||
imgs, err := listLocalImages()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
used, err := imagesUsedByContainers()
|
||||
if err != nil {
|
||||
m.logger.Printf("[WARN] [stacks] infra image retention: the containers could not be read (%v) — NOTHING is deleted", err)
|
||||
return nil, err
|
||||
}
|
||||
var deleted []string
|
||||
for _, pin := range pins {
|
||||
deleted = append(deleted, m.retainOneInfraRepo(pin, imgs, used)...)
|
||||
}
|
||||
return deleted, nil
|
||||
}
|
||||
|
||||
func (m *Manager) retainOneInfraRepo(pin string, imgs []localImage, used map[string]bool) []string {
|
||||
r, tag, _ := splitRepoTag(pin)
|
||||
repo := normRepo(r)
|
||||
pinned, err := util.ParseVersionPre(tag)
|
||||
if err != nil {
|
||||
m.logger.Printf("[INFO] [stacks] infra image retention: %s — the pin %q is not a version; the repository is left alone", repo, tag)
|
||||
return nil
|
||||
}
|
||||
byID := map[string][]localImage{}
|
||||
var prev *util.Version
|
||||
for _, im := range imgs {
|
||||
byID[im.ID] = append(byID[im.ID], im)
|
||||
if im.Repo != repo {
|
||||
continue
|
||||
}
|
||||
if v, err := util.ParseVersionPre(im.Tag); err == nil && v.Compare(pinned) < 0 && (prev == nil || v.Compare(*prev) > 0) {
|
||||
vv := v
|
||||
prev = &vv
|
||||
}
|
||||
}
|
||||
prevTag := ""
|
||||
if prev != nil {
|
||||
prevTag = prev.Raw
|
||||
}
|
||||
ids := make([]string, 0, len(byID))
|
||||
for id := range byID {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
sort.Strings(ids)
|
||||
var deleted []string
|
||||
considered, candidates := 0, 0
|
||||
defer func() {
|
||||
// ONE line per repository per pass, whatever it did — the positive observable that the pass ran (R-96 rule 3).
|
||||
m.logger.Printf("[INFO] [stacks] infra image retention: %s — pass over %d image(s), pinned %s, previous %q, %d candidate(s), %d deleted, the rest kept",
|
||||
repo, considered, tag, prevTag, candidates, len(deleted))
|
||||
}()
|
||||
for _, id := range ids {
|
||||
group := byID[id]
|
||||
var mine []localImage
|
||||
other := false
|
||||
for _, im := range group {
|
||||
if im.Repo == repo {
|
||||
mine = append(mine, im)
|
||||
} else {
|
||||
other = true
|
||||
}
|
||||
}
|
||||
if len(mine) == 0 {
|
||||
continue
|
||||
}
|
||||
considered++
|
||||
if used[id] {
|
||||
continue
|
||||
}
|
||||
deletable, odd := true, ""
|
||||
var names []string
|
||||
for _, im := range mine {
|
||||
if im.Tag == "<none>" || im.Tag == "" {
|
||||
continue
|
||||
}
|
||||
names = append(names, im.Repo+":"+im.Tag)
|
||||
v, err := util.ParseVersionPre(im.Tag)
|
||||
switch {
|
||||
case err != nil:
|
||||
deletable, odd = false, im.Tag // latest: not a version, not ours to judge
|
||||
case v.Compare(pinned) >= 0:
|
||||
deletable = false // the pin, or a newer pin's image
|
||||
case prev == nil || v.Compare(*prev) >= 0:
|
||||
deletable = false // the previous pin
|
||||
}
|
||||
}
|
||||
if !deletable {
|
||||
if odd != "" {
|
||||
m.logger.Printf("[INFO] [stacks] infra image retention: %s carries a tag that is not a version (%s) — left alone", shortID(id), odd)
|
||||
}
|
||||
continue
|
||||
}
|
||||
candidates++
|
||||
if other {
|
||||
m.logger.Printf("[INFO] [stacks] infra image retention: %s also carries another repository's name — left alone", shortID(id))
|
||||
continue
|
||||
}
|
||||
targets := names
|
||||
if len(targets) == 0 {
|
||||
targets = []string{id}
|
||||
}
|
||||
ok := true
|
||||
for _, t := range targets {
|
||||
if out, err := imageDocker("rmi", t); err != nil {
|
||||
m.logger.Printf("[WARN] [stacks] infra image retention: docker refused to delete %s (%s): %s", t, shortID(id), truncateStr(strings.TrimSpace(out), 160))
|
||||
ok = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
label := strings.Join(names, ",")
|
||||
if label == "" {
|
||||
label = repo + ":<none>"
|
||||
}
|
||||
m.logger.Printf("[INFO] [stacks] infra image retention: deleted %s (%s, %s) — older than the previous pin and no container uses it (decision 285)", label, shortID(id), mine[0].Size)
|
||||
deleted = append(deleted, fmt.Sprintf("%s@%s", label, shortID(id)))
|
||||
}
|
||||
return deleted
|
||||
}
|
||||
|
||||
// RunInfraImageRetention runs RetainInfraImages, and again every `every` while image work is in flight, at most
|
||||
// `tries` times (R-863's retry shape: a household's first installs land in the minutes after a start).
|
||||
func (m *Manager) RunInfraImageRetention(ctx context.Context, pins []string, every time.Duration, tries int) {
|
||||
for i := 0; i < tries; i++ {
|
||||
_, err := m.RetainInfraImages(pins)
|
||||
if err != errImageRetentionBusy {
|
||||
if err != nil {
|
||||
m.logger.Printf("[WARN] [stacks] infra image retention: %v — tried again at the next start", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(every):
|
||||
}
|
||||
}
|
||||
m.logger.Printf("[WARN] [stacks] infra image retention: not run after %d tries — tried again at the next start", tries)
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
package stacks
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/dockerexec"
|
||||
)
|
||||
|
||||
// Decision 285 (2026-11-09): the pinned infrastructure images keep the pin, every newer version, the previous pin and
|
||||
// every image a container uses; older ones go. The fixture is demo-hp's guest as measured on 2026-11-09 (before the
|
||||
// hand clean-up): traefik v3.6.7/v3.7.13/v3.7.16 beside the pinned v3.7.17, FileBrowser 1.3.3/1.5.6 beside 1.5.8.
|
||||
// Docker is the imageDocker seam (fakeImages) — nothing here reaches a daemon.
|
||||
|
||||
var infraPins = []string{"traefik:v3.7.17", "gtstef/filebrowser:1.5.8-stable", "gitea.dooplex.hu/admin/felhom-samba:1.2.0"}
|
||||
|
||||
func infraImages() *fakeImages {
|
||||
return &fakeImages{
|
||||
imgs: []localImage{
|
||||
{ID: "sha256:T3717", Repo: "traefik", Tag: "v3.7.17", Size: "195MB"},
|
||||
{ID: "sha256:T3716", Repo: "traefik", Tag: "v3.7.16", Size: "194MB"},
|
||||
{ID: "sha256:T3713", Repo: "traefik", Tag: "v3.7.13", Size: "195MB"},
|
||||
{ID: "sha256:T367", Repo: "traefik", Tag: "v3.6.7", Size: "186MB"},
|
||||
{ID: "sha256:TNONE", Repo: "traefik", Tag: "<none>", Size: "190MB"},
|
||||
{ID: "sha256:F158", Repo: "gtstef/filebrowser", Tag: "1.5.8-stable", Size: "196MB"},
|
||||
{ID: "sha256:F156", Repo: "gtstef/filebrowser", Tag: "1.5.6-stable", Size: "195MB"},
|
||||
{ID: "sha256:F133", Repo: "gtstef/filebrowser", Tag: "1.3.3-stable", Size: "215MB"},
|
||||
// Samba is pinned but not enabled on this box (no container): the pin itself must stay.
|
||||
{ID: "sha256:S120", Repo: "gitea.dooplex.hu/admin/felhom-samba", Tag: "1.2.0", Size: "107MB"},
|
||||
{ID: "sha256:S110", Repo: "gitea.dooplex.hu/admin/felhom-samba", Tag: "1.1.0", Size: "103MB"},
|
||||
{ID: "sha256:S100", Repo: "gitea.dooplex.hu/admin/felhom-samba", Tag: "1.0.0", Size: "101MB"},
|
||||
// not an infra repository: never touched by this pass
|
||||
{ID: "sha256:A1", Repo: "acme/web", Tag: "1", Size: "100MB"},
|
||||
{ID: "sha256:ALP", Repo: "alpine", Tag: "latest", Size: "8MB"},
|
||||
},
|
||||
containers: map[string]string{"c-traefik": "sha256:T3717", "c-fb": "sha256:F158"},
|
||||
}
|
||||
}
|
||||
|
||||
// COMPANION RED-PROOF: drop the `prev` case from the keep switch → "the previous pin traefik:v3.7.16 was deleted";
|
||||
// drop the `v >= pinned` case → "the pinned Samba image (no container) was deleted".
|
||||
func TestInfraRetention_KeepsPinAndPreviousDeletesOlder(t *testing.T) {
|
||||
m := retentionManager(t)
|
||||
f := infraImages()
|
||||
withFakeImages(t, f)
|
||||
got, err := m.RetainInfraImages(infraPins)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := rmiSet(f)
|
||||
for _, keep := range []string{"traefik:v3.7.17", "traefik:v3.7.16", "gtstef/filebrowser:1.5.8-stable", "gtstef/filebrowser:1.5.6-stable"} {
|
||||
if strings.Contains(s, ","+keep+",") {
|
||||
t.Fatalf("the previous pin %s was deleted: %v", keep, f.rmi)
|
||||
}
|
||||
}
|
||||
if strings.Contains(s, "felhom-samba:1.2.0") || strings.Contains(s, "felhom-samba:1.1.0") {
|
||||
t.Fatalf("the pinned Samba image (no container) or its previous pin was deleted: %v", f.rmi)
|
||||
}
|
||||
for _, want := range []string{"traefik:v3.7.13", "traefik:v3.6.7", "sha256:TNONE", "gtstef/filebrowser:1.3.3-stable", "gitea.dooplex.hu/admin/felhom-samba:1.0.0"} {
|
||||
if !strings.Contains(s, ","+want+",") {
|
||||
t.Fatalf("%s (older than the previous pin / untagged) was not deleted: rmi=%v", want, f.rmi)
|
||||
}
|
||||
}
|
||||
if strings.Contains(s, "acme/web") || strings.Contains(s, "sha256:A1") || strings.Contains(s, "alpine") {
|
||||
t.Fatalf("an image outside the pinned repositories was touched: %v", f.rmi)
|
||||
}
|
||||
if len(got) != 5 {
|
||||
t.Fatalf("deleted %d, want 5: %v", len(got), got)
|
||||
}
|
||||
}
|
||||
|
||||
// A box rolled back to an older controller pins an OLDER traefik: the newer one stays (the way forward), and a
|
||||
// container that still runs an old image keeps it.
|
||||
// COMPANION RED-PROOF: drop the `used[id]` check → "an image a container uses was deleted".
|
||||
func TestInfraRetention_NewerPinAndInUseKept(t *testing.T) {
|
||||
m := retentionManager(t)
|
||||
f := infraImages()
|
||||
f.containers["c-old-fb"] = "sha256:F133"
|
||||
withFakeImages(t, f)
|
||||
if _, err := m.RetainInfraImages([]string{"traefik:v3.7.16", "gtstef/filebrowser:1.5.8-stable"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := rmiSet(f)
|
||||
if strings.Contains(s, "traefik:v3.7.17") {
|
||||
t.Fatalf("a version above the pin was deleted: %v", f.rmi)
|
||||
}
|
||||
if strings.Contains(s, "traefik:v3.7.13") || !strings.Contains(s, "traefik:v3.6.7") {
|
||||
t.Fatalf("pinned v3.7.16: want v3.7.13 kept (previous) and v3.6.7 deleted: %v", f.rmi)
|
||||
}
|
||||
if strings.Contains(s, "1.3.3-stable") {
|
||||
t.Fatalf("an image a container uses was deleted: %v", f.rmi)
|
||||
}
|
||||
}
|
||||
|
||||
// While an image-pulling compose command runs (R-863), or the controller swaps itself, NOTHING is deleted.
|
||||
// COMPANION RED-PROOF: drop the TryImageCleanup check → "deleted while image work was in flight".
|
||||
func TestInfraRetention_NothingWhileImageWorkOrSwap(t *testing.T) {
|
||||
m := retentionManager(t)
|
||||
f := infraImages()
|
||||
withFakeImages(t, f)
|
||||
end := dockerexec.HoldImageWork()
|
||||
_, err := m.RetainInfraImages(infraPins)
|
||||
end()
|
||||
if err != errImageRetentionBusy || len(f.rmi) != 0 {
|
||||
t.Fatalf("deleted while image work was in flight: err=%v rmi=%v", err, f.rmi)
|
||||
}
|
||||
m.SetSelfUpdatingCheck(func() bool { return true })
|
||||
_, err = m.RetainInfraImages(infraPins)
|
||||
m.SetSelfUpdatingCheck(func() bool { return false })
|
||||
if err != errImageRetentionBusy || len(f.rmi) != 0 {
|
||||
t.Fatalf("deleted while the controller swaps itself: err=%v rmi=%v", err, f.rmi)
|
||||
}
|
||||
}
|
||||
|
||||
// A pin that is not a version, and a tag that is not a version, are left alone.
|
||||
func TestInfraRetention_NonVersionsLeftAlone(t *testing.T) {
|
||||
m := retentionManager(t)
|
||||
f := infraImages()
|
||||
f.imgs = append(f.imgs, localImage{ID: "sha256:TLATEST", Repo: "traefik", Tag: "latest", Size: "190MB"})
|
||||
withFakeImages(t, f)
|
||||
if _, err := m.RetainInfraImages([]string{"traefik:v3.7.17", "gtstef/filebrowser:latest"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := rmiSet(f)
|
||||
if strings.Contains(s, "traefik:latest") || strings.Contains(s, "filebrowser") {
|
||||
t.Fatalf("a non-version tag or a repository with a non-version pin was touched: %v", f.rmi)
|
||||
}
|
||||
}
|
||||
@@ -284,6 +284,8 @@ type Manager struct {
|
||||
// R-514: last OOM scan (oom.go), for the dashboard.
|
||||
oomMu sync.Mutex
|
||||
oomCache map[string][]string
|
||||
// hostOOMKills (R-528) is the host agent's per-container OOM counter; nil = Docker's flag only (SetHostOOMKills).
|
||||
hostOOMKills func() map[string]int64
|
||||
// R-513: FileBrowser admin-password seams (filebrowser_password.go); nil/"" in production.
|
||||
fbHTTP fbHTTPDo
|
||||
fbBaseURL string
|
||||
@@ -293,6 +295,11 @@ type Manager struct {
|
||||
fbProbedRun string
|
||||
// fbNetOnFn / fbNetConnectFn replace the docker network reads and connects of the R-1126 gate in tests.
|
||||
fbNetOnFn func(container, network string) bool
|
||||
// fbStackMu (R-1351) is the lock over `docker compose` on FileBrowser's stack: LockFileBrowserStack. fbRunningFn and
|
||||
// fbComposeUpFn replace the docker read and the bring-up of ensureFileBrowser in tests.
|
||||
fbStackMu sync.Mutex
|
||||
fbRunningFn func() bool
|
||||
fbComposeUpFn func(dir string) error
|
||||
fbNetConnectFn func(container, network string) error
|
||||
// R-1281: the disconnect, the network create, and „is OnlyOffice linked to FileBrowser and running" — test seams.
|
||||
fbNetDisconnectFn func(container, network string) error
|
||||
|
||||
@@ -27,6 +27,21 @@ type OOMContainer struct {
|
||||
Kills int64
|
||||
// MemLimit / Peak are the cgroup's memory.max and memory.peak, as "<n>M" (or "max"); "" unread.
|
||||
MemLimit, Peak string
|
||||
// Source names what flagged it: OOMSourceDocker (State.OOMKilled), OOMSourceCgroup (R-528 — only the host's
|
||||
// per-container kernel counter saw it; Docker's flag stayed false), or "docker,cgroup" (both).
|
||||
Source string
|
||||
}
|
||||
|
||||
const (
|
||||
OOMSourceDocker = "docker"
|
||||
OOMSourceCgroup = "cgroup"
|
||||
)
|
||||
|
||||
// SetHostOOMKills wires the host-side OOM counter (R-528): container id → the kernel's oom_kill for the container's
|
||||
// CURRENT run (its cgroup scope is new at each start), read by the host's agent. nil, or a nil answer (no agent, an
|
||||
// agent without the route, a failed call) = Docker's flag only, as before. INIT-ONLY (main.go).
|
||||
func (m *Manager) SetHostOOMKills(fn func() map[string]int64) {
|
||||
m.hostOOMKills = fn
|
||||
}
|
||||
|
||||
// ScanOOMKilled inspects the containers of every deployed, running-ish stack in ONE docker call and
|
||||
@@ -52,20 +67,45 @@ func (m *Manager) ScanOOMKilled() ([]OOMContainer, error) {
|
||||
return nil, nil
|
||||
}
|
||||
sort.Strings(names)
|
||||
args := append([]string{"inspect", "-f", "{{.Name}}|{{.State.OOMKilled}}|{{.State.StartedAt}}"}, names...)
|
||||
args := append([]string{"inspect", "-f", "{{.Name}}|{{.State.OOMKilled}}|{{.State.StartedAt}}|{{.Id}}"}, names...)
|
||||
out, err := m.execCommand("docker", args...)
|
||||
if err != nil && strings.TrimSpace(out) == "" {
|
||||
return nil, err
|
||||
}
|
||||
// R-528: the host's counter, asked once per scan (an HTTPS call to the agent); nil = Docker's flag only.
|
||||
var host map[string]int64
|
||||
if m.hostOOMKills != nil {
|
||||
host = m.hostOOMKills()
|
||||
}
|
||||
var found []OOMContainer
|
||||
hostKills := map[int]int64{}
|
||||
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
|
||||
f := strings.SplitN(strings.TrimSpace(line), "|", 3)
|
||||
if len(f) != 3 || f[1] != "true" {
|
||||
f := strings.SplitN(strings.TrimSpace(line), "|", 4)
|
||||
if len(f) < 3 {
|
||||
continue
|
||||
}
|
||||
id := ""
|
||||
if len(f) == 4 {
|
||||
id = strings.TrimSpace(f[3])
|
||||
}
|
||||
hk, counted := host[id]
|
||||
source := ""
|
||||
switch {
|
||||
case f[1] == "true" && id != "" && counted && hk > 0:
|
||||
source = OOMSourceDocker + "," + OOMSourceCgroup // both saw it
|
||||
case f[1] == "true":
|
||||
source = OOMSourceDocker
|
||||
case id != "" && counted && hk > 0:
|
||||
source = OOMSourceCgroup
|
||||
default:
|
||||
continue
|
||||
}
|
||||
cname := strings.TrimPrefix(f[0], "/")
|
||||
if st, ok := owner[cname]; ok {
|
||||
found = append(found, OOMContainer{Stack: st, Container: cname, StartedAt: f[2], Kills: -1})
|
||||
if counted && id != "" {
|
||||
hostKills[len(found)] = hk
|
||||
}
|
||||
found = append(found, OOMContainer{Stack: st, Container: cname, StartedAt: f[2], Kills: -1, Source: source})
|
||||
}
|
||||
}
|
||||
// R-636: only for the containers already flagged — one `docker exec` each, and the flagged set is
|
||||
@@ -75,6 +115,9 @@ func (m *Manager) ScanOOMKilled() ([]OOMContainer, error) {
|
||||
out, _ := m.execCommand("docker", "exec", found[i].Container, "cat",
|
||||
"/sys/fs/cgroup/memory.events", "/sys/fs/cgroup/memory.max", "/sys/fs/cgroup/memory.peak")
|
||||
found[i].Kills, found[i].MemLimit, found[i].Peak = parseCgroupMemory(out)
|
||||
if hk, ok := hostKills[i]; ok && hk > found[i].Kills {
|
||||
found[i].Kills = hk // the host's count is the kernel's own; the in-container read can be -1 (no cat)
|
||||
}
|
||||
}
|
||||
m.setOOMCache(found)
|
||||
return found, nil
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
package stacks
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"log"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1323: the checked FileBrowser container can be replaced between the check and the leave (the image moved at start).
|
||||
// „No such container" then means the OLD one is gone, not that every app still reaches FileBrowser. The line reads the
|
||||
// CURRENT container's networks and says what is true; a current container that IS on the app network keeps the alarm.
|
||||
//
|
||||
// RED-PROOF (2026-11-02): before the fix the first case logged „could not leave traefik-public — every app can still reach
|
||||
// it … No such container" (seen on 9202, 0.346.1-rc.1, 2026-10-27), and this fails on that line.
|
||||
func r1323Rig(t *testing.T, currentOnApps bool) (*Manager, *bytes.Buffer) {
|
||||
t.Helper()
|
||||
fb := &fakeFB{password: "admin"}
|
||||
srv := httptest.NewServer(fb.handler())
|
||||
t.Cleanup(srv.Close)
|
||||
m, _, _ := fbManager(t, srv.URL)
|
||||
var buf bytes.Buffer
|
||||
m.logger = log.New(&buf, "", 0)
|
||||
runs := []string{"old111 2026-10-27T14:58:15Z"}
|
||||
m.fbStarted = func() string {
|
||||
r := runs[0]
|
||||
if len(runs) > 1 {
|
||||
runs = runs[1:]
|
||||
}
|
||||
return r
|
||||
}
|
||||
nets := map[string]bool{"old111@traefik-public": true, "new222@traefik-public": currentOnApps}
|
||||
m.fbNetOnFn = func(c, n string) bool { return nets[c+"@"+n] }
|
||||
m.fbNetConnectFn = func(c, n string) error { nets[c+"@"+n] = true; return nil }
|
||||
m.fbNetDisconnectFn = func(c, n string) error {
|
||||
if c == "old111" {
|
||||
runs = []string{"new222 2026-10-27T14:58:17Z"} // the recreate won the race
|
||||
return errors.New("Error response from daemon: No such container: old111")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
m.fbNetEnsureFn = func(string) error { return nil }
|
||||
m.fbOfficeLinkedFn = func() (bool, bool) { return false, false }
|
||||
return m, &buf
|
||||
}
|
||||
|
||||
func TestR1323_AReplacedContainerIsNotReportedAsExposed(t *testing.T) {
|
||||
m, buf := r1323Rig(t, false)
|
||||
m.fileBrowserGateLocked()
|
||||
got := buf.String()
|
||||
if strings.Contains(got, "every app can still reach it") {
|
||||
t.Fatalf("a false alarm for a container that is gone: %q", got)
|
||||
}
|
||||
if !strings.Contains(got, "new222") || !strings.Contains(got, "not on traefik-public") {
|
||||
t.Fatalf("the line must name the current container and the truth: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestR1323_ACurrentContainerOnTheAppNetworkKeepsTheAlarm(t *testing.T) {
|
||||
m, buf := r1323Rig(t, true)
|
||||
m.fileBrowserGateLocked()
|
||||
got := buf.String()
|
||||
if !strings.Contains(got, "every app can still reach it") || !strings.Contains(got, "new222") {
|
||||
t.Fatalf("the replacement IS on the app network: the alarm stays, naming it: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Any other failure to leave keeps the alarm as before (control).
|
||||
func TestR1323_AnotherDisconnectFailureKeepsTheAlarm(t *testing.T) {
|
||||
m, buf := r1323Rig(t, false)
|
||||
m.fbNetDisconnectFn = func(c, n string) error { return errors.New("daemon busy") }
|
||||
m.fileBrowserGateLocked()
|
||||
if got := buf.String(); !strings.Contains(got, "every app can still reach it") || !strings.Contains(got, "daemon busy") {
|
||||
t.Fatalf("a real failure to leave must keep its alarm: %q", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
package stacks
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-1347, the consequence: an app whose last update ended held is not watched for a crash loop; once its hold is lifted by
|
||||
// a restore and the flag forgotten (backup.clearUpdateHoldAfterRestore → ForgetUpdateHeld) the same restarts trip the
|
||||
// watcher again.
|
||||
func TestR1347_TheWatcherWatchesAgainOnceTheFlagIsForgotten(t *testing.T) {
|
||||
m := &Manager{logger: log.New(io.Discard, "", 0), stacks: map[string]*Stack{
|
||||
"nextcloud": {Name: "nextcloud", Deployed: true, Containers: []ContainerInfo{{Name: "nextcloud-web-1"}}},
|
||||
}}
|
||||
m.markUpdateHeld("nextcloud")
|
||||
count := int64(0)
|
||||
m.execFn = func(string, ...string) (string, error) { return fmt.Sprintf("/nextcloud-web-1|%d\n", count), nil }
|
||||
t0 := time.Date(2026, 11, 2, 4, 20, 0, 0, time.UTC)
|
||||
sample := func(from int) []UnhealthyVerdict {
|
||||
var got []UnhealthyVerdict
|
||||
for i := from; i < from+12 && len(got) == 0; i++ {
|
||||
count += 3
|
||||
got = m.ObserveUnhealthy(t0.Add(time.Duration(i)*time.Minute), nil)
|
||||
}
|
||||
return got
|
||||
}
|
||||
if got := sample(0); len(got) != 0 {
|
||||
t.Fatalf("control: an app whose update ended held must not be judged: %+v", got)
|
||||
}
|
||||
if !m.ForgetUpdateHeld("nextcloud") {
|
||||
t.Fatal("the flag was set, so it must be forgotten")
|
||||
}
|
||||
got := sample(20)
|
||||
if len(got) != 1 || got[0].Stack != "nextcloud" || got[0].Kind != UnhealthyCrashLoop {
|
||||
t.Fatalf("after the restore lifted the hold the watcher must watch the app again; verdicts %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
package stacks
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-1351: at a guest boot EnsureBaseStack's `compose up` and web.SyncFileBrowserMounts' `compose up --force-recreate` ran
|
||||
// in the same second; one lost with „Conflict. The container name … is already in use" and a false first-boot WARN. The
|
||||
// two paths now share one lock: the base stack's bring-up holds it around its FileBrowser step, the web sync around its
|
||||
// compose call (TestFileBrowserSyncRunsInsideTheStackLock pins that side, internal/web).
|
||||
//
|
||||
// RED-PROOF (2026-11-02): seen failing before the lock existed (build: WithFileBrowserStack); with ensureFileBrowser not
|
||||
// taking the lock it fails on „overlap".
|
||||
func TestR1351_TheTwoFileBrowserStartPathsNeverOverlap(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(dir, "docker-compose.yml"), []byte("services: {}\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m := &Manager{logger: log.New(io.Discard, "", 0)}
|
||||
var mu sync.Mutex
|
||||
inside, overlaps := 0, 0
|
||||
enter := func() {
|
||||
mu.Lock()
|
||||
inside++
|
||||
if inside > 1 {
|
||||
overlaps++
|
||||
}
|
||||
mu.Unlock()
|
||||
time.Sleep(30 * time.Millisecond)
|
||||
mu.Lock()
|
||||
inside--
|
||||
mu.Unlock()
|
||||
}
|
||||
m.fbComposeUpFn = func(string) error { enter(); return nil }
|
||||
m.fbRunningFn = func() bool { return false }
|
||||
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < 4; i++ {
|
||||
wg.Add(2)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
if err := m.ensureFileBrowser(dir); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}()
|
||||
go func() { defer wg.Done(); m.WithFileBrowserStack(enter) }()
|
||||
}
|
||||
wg.Wait()
|
||||
if overlaps != 0 {
|
||||
t.Fatalf("overlap: the two start paths ran compose at the same time %d time(s) — the „Conflict” of R-1351", overlaps)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
package stacks
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-528 — inside a Felhom LXC guest Docker can leave State.OOMKilled false after a memory-limit kill (measured
|
||||
// 2026-09-15/16). The host's agent reads the kernel's own per-container counter (GET /guest/oom-kills); a counter above
|
||||
// 0 for the RUNNING container's id is an OOM kill in this run, whatever Docker says. The scan flags it with
|
||||
// Source "cgroup" and the host's count, and the existing app_oom line (once per run) fires for it.
|
||||
|
||||
const r528ID = "4f5e6d7c8b9a0f1e2d3c4b5a69788796a5b4c3d2e1f00112233445566778899a"
|
||||
|
||||
// RED-PROOF: ignore the host counter in ScanOOMKilled (only `OOMKilled == true` flags) → the killed container is not
|
||||
// seen → FAILS.
|
||||
func TestR528_AHostCounterAboveZeroIsAnOOMKillWhateverDockerSays(t *testing.T) {
|
||||
var execd []string
|
||||
m := &Manager{logger: log.New(io.Discard, "", 0), stacks: map[string]*Stack{
|
||||
"hog": {Name: "hog", Deployed: true, Containers: []ContainerInfo{{Name: "hog", State: StateRunning}}},
|
||||
"bookstack": {Name: "bookstack", Deployed: true, Containers: []ContainerInfo{{Name: "bookstack", State: StateRunning}}},
|
||||
}}
|
||||
m.execFn = func(name string, args ...string) (string, error) {
|
||||
if args[0] == "exec" {
|
||||
execd = append(execd, args[1])
|
||||
return "oom_kill 1\n67108864\n67100000\n", nil
|
||||
}
|
||||
if !strings.Contains(strings.Join(args, " "), "{{.Id}}") {
|
||||
t.Errorf("the inspect must ask for the container id: %v", args)
|
||||
}
|
||||
return "/bookstack|false|t0|" + strings.Repeat("0", 64) + "\n/hog|false|2026-10-28T23:00:00Z|" + r528ID + "\n", nil
|
||||
}
|
||||
m.SetHostOOMKills(func() map[string]int64 {
|
||||
return map[string]int64{r528ID: 2, strings.Repeat("0", 64): 0}
|
||||
})
|
||||
ooms, err := m.ScanOOMKilled()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(ooms) != 1 || ooms[0].Stack != "hog" || ooms[0].Container != "hog" {
|
||||
t.Fatalf("the host-counted kill was not seen: %+v", ooms)
|
||||
}
|
||||
if ooms[0].Source != OOMSourceCgroup || ooms[0].Kills != 2 || ooms[0].StartedAt != "2026-10-28T23:00:00Z" {
|
||||
t.Fatalf("want source cgroup, the host's count 2 and the run's start: %+v", ooms[0])
|
||||
}
|
||||
if ooms[0].MemLimit != "64M" {
|
||||
t.Errorf("limit/peak are still read for the flagged container: %+v", ooms[0])
|
||||
}
|
||||
if len(execd) != 1 || execd[0] != "hog" {
|
||||
t.Errorf("exec must reach the flagged container only, got %v", execd)
|
||||
}
|
||||
}
|
||||
|
||||
// Docker's flag alone still flags (Source docker); no host source (an old agent, 9202 without an agent) = today's
|
||||
// behaviour; a host counter of 0 never flags.
|
||||
func TestR528_WithoutAHostCounterTheScanIsAsBefore(t *testing.T) {
|
||||
m := &Manager{logger: log.New(io.Discard, "", 0), stacks: map[string]*Stack{
|
||||
"a": {Name: "a", Deployed: true, Containers: []ContainerInfo{{Name: "a", State: StateRunning}}},
|
||||
"b": {Name: "b", Deployed: true, Containers: []ContainerInfo{{Name: "b", State: StateRunning}}},
|
||||
}}
|
||||
m.execFn = func(name string, args ...string) (string, error) {
|
||||
if args[0] == "exec" {
|
||||
return "oom_kill 3\n", nil
|
||||
}
|
||||
return "/a|true|s1|" + strings.Repeat("a", 64) + "\n/b|false|s2|" + strings.Repeat("b", 64) + "\n", nil
|
||||
}
|
||||
ooms, _ := m.ScanOOMKilled()
|
||||
if len(ooms) != 1 || ooms[0].Stack != "a" || ooms[0].Source != OOMSourceDocker || ooms[0].Kills != 3 {
|
||||
t.Fatalf("no host source: %+v", ooms)
|
||||
}
|
||||
m.SetHostOOMKills(func() map[string]int64 { return map[string]int64{strings.Repeat("b", 64): 0} })
|
||||
ooms, _ = m.ScanOOMKilled()
|
||||
if len(ooms) != 1 || ooms[0].Stack != "a" {
|
||||
t.Fatalf("a host counter of 0 flagged a container: %+v", ooms)
|
||||
}
|
||||
// Both saw it: the source names both, the host's count wins over a lower in-container read.
|
||||
m.SetHostOOMKills(func() map[string]int64 { return map[string]int64{strings.Repeat("a", 64): 7} })
|
||||
ooms, _ = m.ScanOOMKilled()
|
||||
if len(ooms) != 1 || ooms[0].Source != OOMSourceDocker+","+OOMSourceCgroup || ooms[0].Kills != 7 {
|
||||
t.Fatalf("both sources: %+v", ooms)
|
||||
}
|
||||
}
|
||||
@@ -209,6 +209,25 @@ func (m *Manager) ClearUpdateState(name string) {
|
||||
s.HealthProbe = nil
|
||||
}
|
||||
|
||||
// ForgetUpdateHeld (D64 = B, R-1343) forgets that the app's last update ended HELD, once the hold itself is cleared
|
||||
// by the operator. updateHeld is the ONE held flag kept outside the hold store: it makes the crash-loop observer
|
||||
// (ObserveUnhealthy) and the nightly update leg (legCandidate) skip the app, and keeps the hold's sentence as the
|
||||
// update error. Left set, an app whose hold was cleared would stay unobserved and never be updated by night until
|
||||
// the controller restarts. A running update is not touched. Returns whether anything was forgotten.
|
||||
// Pinned by TestD64_ForgetUpdateHeld (this package) and TestD64_ClearHoldViaTheHub (cmd/controller).
|
||||
func (m *Manager) ForgetUpdateHeld(name string) bool {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
s, ok := m.stacks[name]
|
||||
if !ok || s.Updating || (!s.updateHeld && s.UpdateErrorKey != UpdateErrorKeyHeld) {
|
||||
return false
|
||||
}
|
||||
s.updateHeld = false
|
||||
s.UpdatePhase, s.UpdatePhaseLabel = "", ""
|
||||
s.UpdateError, s.UpdateErrorKey, s.UpdateErrorArgs = "", "", nil
|
||||
return true
|
||||
}
|
||||
|
||||
func (m *Manager) markUpdateHeld(name string) {
|
||||
m.mu.Lock()
|
||||
if s, ok := m.stacks[name]; ok {
|
||||
|
||||
@@ -0,0 +1,248 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"html/template"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// D62 (operator ruling 2026-10-29): restore copies in the kept-copies list with „Törlés most". Real settings, manager,
|
||||
// disk and a hub stand-in that records the event — the handler is the endpoint the page's form calls.
|
||||
|
||||
type d62Fixture struct {
|
||||
s *Server
|
||||
sett *settings.Settings
|
||||
nsRoot string
|
||||
events chan map[string]interface{}
|
||||
}
|
||||
|
||||
func newD62Fixture(t *testing.T) *d62Fixture {
|
||||
t.Helper()
|
||||
tmp := t.TempDir()
|
||||
lg := log.New(io.Discard, "", 0)
|
||||
sett, err := settings.Load(filepath.Join(tmp, "settings.json"), lg)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cfg := &config.Config{}
|
||||
cfg.Paths.DataDir = tmp
|
||||
cfg.Paths.SystemDataPath = filepath.Join(tmp, "sys")
|
||||
events := make(chan map[string]interface{}, 8)
|
||||
hub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/api/v1/event" {
|
||||
var ev map[string]interface{}
|
||||
_ = json.NewDecoder(r.Body).Decode(&ev)
|
||||
events <- ev
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
t.Cleanup(hub.Close)
|
||||
n := notify.New(hub.URL, "test-key", "c1", sett, lg, false)
|
||||
s := &Server{cfg: cfg, settings: sett, backupMgr: backup.NewManager(cfg, sett, lg), notifier: n, logger: lg}
|
||||
return &d62Fixture{s: s, sett: sett, nsRoot: backup.NamespaceRootFor(cfg.Paths.SystemDataPath, cfg.Paths.SystemDataPath), events: events}
|
||||
}
|
||||
|
||||
// plant writes one restore copy of app taken at `at`; returns its folder.
|
||||
func (f *d62Fixture) plant(t *testing.T, app string, at time.Time) string {
|
||||
t.Helper()
|
||||
dir := filepath.Join(backup.RestoreUndoDir(f.nsRoot, app), at.UTC().Format("20060102T150405Z"))
|
||||
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, app+"_data.tar"), []byte("LIVE-DATA"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return dir
|
||||
}
|
||||
|
||||
func (f *d62Fixture) post(form string) (int, map[string]interface{}) {
|
||||
rr := httptest.NewRecorder()
|
||||
req := httptest.NewRequest("POST", "/backup/restore-copy/delete", strings.NewReader(form))
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
req.RemoteAddr = "192.168.0.5:5555"
|
||||
f.s.restoreCopyDeleteHandler(rr, req)
|
||||
var body map[string]interface{}
|
||||
_ = json.Unmarshal(rr.Body.Bytes(), &body)
|
||||
return rr.Code, body
|
||||
}
|
||||
|
||||
func exists(p string) bool { _, err := os.Stat(p); return err == nil }
|
||||
|
||||
// The list shows the restore's date, the size and „automatikusan törlődik: <30 days later>"; a held copy shows why and
|
||||
// offers no warning. RED-PROOF: drop the auto-delete part from restoreCopyRows → FAILS.
|
||||
func TestD62_TheListShowsTheDateAndTheAutoDeleteDay(t *testing.T) {
|
||||
f := newD62Fixture(t)
|
||||
at := time.Date(2026, 10, 29, 8, 0, 0, 0, time.UTC)
|
||||
f.plant(t, "vikunja", at)
|
||||
f.plant(t, "immich", at)
|
||||
if err := f.sett.SetRestoreHold(settings.RestoreHold{Stack: "immich", At: at.Format(time.RFC3339), Reason: settings.HoldReasonRestoreMixed}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rows := f.s.restoreCopyRows("hu")
|
||||
if len(rows) != 2 {
|
||||
t.Fatalf("rows %+v", rows)
|
||||
}
|
||||
var free, held restoreCopyRow
|
||||
for _, r := range rows {
|
||||
if r.App == "vikunja" {
|
||||
free = r
|
||||
} else {
|
||||
held = r
|
||||
}
|
||||
}
|
||||
for _, want := range []string{"visszaállítás: 2026. 10. 29. 09:00", "9 B", "automatikusan törlődik: 2026. 11. 28."} {
|
||||
if !strings.Contains(free.Line, want) {
|
||||
t.Errorf("free row line %q lacks %q", free.Line, want)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(free.Warning, "Ez a(z) vikunja adatainak másolata a 2026. 10. 29-i visszaállítás előttről") ||
|
||||
!strings.Contains(free.Warning, "a visszaállítás többé nem vonható vissza, és ez a másolat nem hozható vissza") {
|
||||
t.Errorf("warning %q", free.Warning)
|
||||
}
|
||||
if free.NeededText != "" || free.TypePrompt == "" {
|
||||
t.Errorf("free row: needed %q prompt %q", free.NeededText, free.TypePrompt)
|
||||
}
|
||||
if held.NeededText == "" || held.Warning != "" || strings.Contains(held.Line, "automatikusan") {
|
||||
t.Errorf("held row offers a delete or an auto-delete day: %+v", held)
|
||||
}
|
||||
en := f.s.restoreCopyRows("en")
|
||||
for _, r := range en {
|
||||
if r.App == "vikunja" && (!strings.Contains(r.Line, "automatically deleted on 2026-11-28") ||
|
||||
!strings.Contains(r.Warning, "This is the copy of vikunja’s data from before the restore on 2026-10-29. If you delete it, the restore can no longer be undone, and this copy cannot be brought back.")) {
|
||||
t.Errorf("English row %+v", r)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The page: a deletable copy has the form (app, stamp, the typed name, CSRF); a held one has the button OFF and the
|
||||
// reason, and no form. RED-PROOF: render the form for a needed copy too → FAILS.
|
||||
func TestD62_ThePageOffersTheFormOnlyForADeletableCopy(t *testing.T) {
|
||||
d := i18nLayoutData("backups-restore", "Visszaállítás a mentésből")
|
||||
d["Backup"] = i18nBackupStatus(false, false)
|
||||
d["RestoreCopies"] = []restoreCopyRow{
|
||||
{App: "vikunja", Stamp: "20261029T080000Z", Name: "Vikunja", Line: "L1", Warning: "W1", TypePrompt: "P1"},
|
||||
{App: "immich", Stamp: "20261020T080000Z", Name: "Immich", Line: "L2", NeededText: "HELD-REASON"},
|
||||
}
|
||||
d["CSRFField"] = template.HTML(`<input type="hidden" name="_csrf" value="TOKEN-D62">`)
|
||||
html := renderBackupPage(t, "backups_restore", d)
|
||||
for _, want := range []string{"Visszaállítás előtti másolatok", `action="/backup/restore-copy/delete"`, `name="app" value="vikunja"`,
|
||||
`name="stamp" value="20261029T080000Z"`, `name="confirm"`, "W1", "P1", "Törlés most", `data-restore-copy-off="true"`, "HELD-REASON"} {
|
||||
if !strings.Contains(html, want) {
|
||||
t.Errorf("page lacks %q", want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(html, `name="app" value="immich"`) || strings.Contains(html, `data-restore-copy="immich/`) {
|
||||
t.Error("a held copy got a delete form")
|
||||
}
|
||||
form := html[strings.Index(html, `action="/backup/restore-copy/delete"`):]
|
||||
if !strings.Contains(form[:strings.Index(form, "</form>")], `value="TOKEN-D62"`) {
|
||||
t.Error("the delete form carries no CSRF field")
|
||||
}
|
||||
none := i18nLayoutData("backups-restore", "Visszaállítás a mentésből")
|
||||
none["Backup"] = i18nBackupStatus(false, false)
|
||||
if html := renderBackupPage(t, "backups_restore", none); strings.Contains(html, "Megőrzött másolatok") || strings.Contains(html, "restore-copy") {
|
||||
t.Error("a page with no copy shows the section")
|
||||
}
|
||||
}
|
||||
|
||||
// A wrong typed name is refused (400) and nothing is deleted. RED-PROOF: drop the name check → FAILS.
|
||||
func TestD62_AWrongNameIsRefused(t *testing.T) {
|
||||
f := newD62Fixture(t)
|
||||
c := f.plant(t, "vikunja", time.Now().Add(-time.Hour))
|
||||
for _, typed := range []string{"", "vikunj", "immich", "VIKUNJA "} {
|
||||
code, body := f.post("app=vikunja&stamp=" + filepath.Base(c) + "&confirm=" + typed)
|
||||
if code != http.StatusBadRequest || body["ok"] != false || !exists(c) {
|
||||
t.Fatalf("typed %q: code %d body %v exists %v", typed, code, body, exists(c))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A held copy is refused by the handler too (409, the reason), and stays. RED-PROOF: drop the Needed check in the
|
||||
// handler AND in DeleteRestoreCopy → FAILS (either alone holds it).
|
||||
func TestD62_AHeldCopyIsRefusedByTheHandler(t *testing.T) {
|
||||
f := newD62Fixture(t)
|
||||
c := f.plant(t, "immich", time.Now().Add(-time.Hour))
|
||||
if err := f.sett.SetRestoreHold(settings.RestoreHold{Stack: "immich", At: time.Now().Format(time.RFC3339), Reason: settings.HoldReasonPutBackPending}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
code, body := f.post("app=immich&stamp=" + filepath.Base(c) + "&confirm=immich")
|
||||
if code != http.StatusConflict || !exists(c) || !strings.Contains(body["error"].(string), "félbeszakadt visszaállítás") {
|
||||
t.Fatalf("code %d body %v exists %v", code, body, exists(c))
|
||||
}
|
||||
}
|
||||
|
||||
// While a restore runs, a delete is 409 with a sentence, and nothing moves. RED-PROOF: drop the busy check → FAILS.
|
||||
func TestD62_NoDeleteWhileAnOperationRuns(t *testing.T) {
|
||||
f := newD62Fixture(t)
|
||||
c := f.plant(t, "vikunja", time.Now().Add(-time.Hour))
|
||||
f.s.backupMgr.BeginRestoreOp("restore", "other")
|
||||
code, body := f.post("app=vikunja&stamp=" + filepath.Base(c) + "&confirm=vikunja")
|
||||
if code != http.StatusConflict || !exists(c) || body["error"] == "" {
|
||||
t.Fatalf("code %d body %v exists %v", code, body, exists(c))
|
||||
}
|
||||
f.s.backupMgr.EndRestoreOp(true, "ok")
|
||||
}
|
||||
|
||||
// The right name: the copy is gone, ONLY it (the app's other copy and another app's copy stay), the answer leads back
|
||||
// to the page with the flash, and the operator event carries who, app, the restore's date and size.
|
||||
// RED-PROOF: drop the NotifyRestoreCopyDeleted call → no event → FAILS.
|
||||
func TestD62_ADeleteRemovesOnlyThatCopyAndStoresTheEvent(t *testing.T) {
|
||||
f := newD62Fixture(t)
|
||||
at := time.Date(2026, 10, 29, 8, 0, 0, 0, time.UTC)
|
||||
c := f.plant(t, "vikunja", at)
|
||||
sibling := f.plant(t, "vikunja", at.Add(time.Hour))
|
||||
other := f.plant(t, "immich", at)
|
||||
code, body := f.post("app=vikunja&stamp=" + filepath.Base(c) + "&confirm=vikunja")
|
||||
if code != http.StatusOK || body["ok"] != true || !strings.Contains(body["redirect"].(string), "flash.restore_copy.deleted") {
|
||||
t.Fatalf("code %d body %v", code, body)
|
||||
}
|
||||
if exists(c) || !exists(sibling) || !exists(other) {
|
||||
t.Fatalf("gone=%v sibling=%v other=%v", !exists(c), exists(sibling), exists(other))
|
||||
}
|
||||
select {
|
||||
case ev := <-f.events:
|
||||
if ev["event_type"] != "restore_copy_deleted" || ev["severity"] != "info" {
|
||||
t.Fatalf("event %v", ev)
|
||||
}
|
||||
det, _ := json.Marshal(ev["details"])
|
||||
for _, want := range []string{`"app":"vikunja"`, `"restore_at":"2026-10-29T08:00:00Z"`, `"size_bytes":9`, `"who":"the household (dashboard sign-in) from 192.168.0.5"`} {
|
||||
if !strings.Contains(string(det), want) {
|
||||
t.Errorf("details %s lack %s", det, want)
|
||||
}
|
||||
}
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("no restore_copy_deleted event reached the hub")
|
||||
}
|
||||
// a second press: the copy is gone → 404, no second event
|
||||
if code, _ := f.post("app=vikunja&stamp=" + filepath.Base(c) + "&confirm=vikunja"); code != http.StatusNotFound {
|
||||
t.Fatalf("second press code %d", code)
|
||||
}
|
||||
}
|
||||
|
||||
// R-1342: every reason a copy can be needed has its one-line sentence in both languages (the row's key is built from the
|
||||
// reason, so a missing key would show the raw key). RED-PROOF: remove restore_copy.needed.checking from BOTH bundles →
|
||||
// FAILS (seen 2026-10-29; a key missing only in en.json falls back to the Hungarian line — i18n_missing_gate counts that).
|
||||
func TestR1342_EveryNeededReasonHasItsSentence(t *testing.T) {
|
||||
f := newD62Fixture(t)
|
||||
for _, why := range []string{backup.RestoreCopyNeededHold, backup.RestoreCopyNeededPutBack, backup.RestoreCopyNeededUnknown, backup.RestoreCopyNeededChecking} {
|
||||
for _, lang := range []string{"hu", "en"} {
|
||||
key := "restore_copy.needed." + why
|
||||
if got := f.s.msgLang(lang, key); got == "" || got == key || strings.Contains(got, "restore_copy.") {
|
||||
t.Errorf("%s/%s renders %q", lang, key, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -352,6 +352,12 @@ func (s *Server) escrowConfirmAPIHandler(w http.ResponseWriter, r *http.Request)
|
||||
escrowJSON(w, http.StatusGone, map[string]any{"restart": true}, s.msg(r, "api.escrow.confirm_expired"))
|
||||
case http.StatusNotFound:
|
||||
escrowJSON(w, http.StatusGone, map[string]any{"restart": true}, s.msg(r, "api.escrow.confirm_lost"))
|
||||
case http.StatusGatewayTimeout, 0:
|
||||
// R-1254 H2: the agent could not learn which package the hub holds (504), or this call itself failed (0 — it may
|
||||
// have timed out while the agent still stored the package). Either way the new package MAY be on the hub: no
|
||||
// promise about the previous code. Only 502 (the hub holds another package) keeps the store-failed sentence.
|
||||
s.logger.Printf("[WARN] [web] escrow confirm outcome unknown (status %d)", status)
|
||||
escrowJSON(w, http.StatusGatewayTimeout, nil, s.msg(r, "api.escrow.confirm_store_unknown"))
|
||||
default:
|
||||
s.logger.Printf("[WARN] [web] escrow confirm failed (status %d)", status)
|
||||
escrowJSON(w, http.StatusBadGateway, nil, s.msg(r, "api.escrow.confirm_store_failed"))
|
||||
|
||||
@@ -1,6 +1,14 @@
|
||||
package web
|
||||
|
||||
import "testing"
|
||||
import (
|
||||
"bytes"
|
||||
"log"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// T4: skipFileBrowserPath skips ONLY an external drive path (under StableParentDir) that is not a live
|
||||
// mountpoint. A mounted external path and any system/local path are never skipped.
|
||||
@@ -49,3 +57,66 @@ func TestFbNeedsRecreate(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// R-1339: a per-app folder under a MOUNTED drive (/mnt/felhom-drives/x/userdata/<app>) is never a mountpoint itself. It is
|
||||
// shown through the drive's own bind, so it still gets no bind and no skeleton of its own (a skeleton there would write
|
||||
// userdata/ inside the app's folder), and the log no longer says its drive is „not mounted". A drive that really is
|
||||
// absent keeps its line, and a folder of such a drive names the absent drive.
|
||||
//
|
||||
// RED-PROOF (2026-11-02): before the fix the sub-folder logged „drive …/userdata/nextcloud not mounted" (nine lines at every
|
||||
// start on 9202, scratch_hdd a live mountpoint) and this fails on that line.
|
||||
func TestR1339_AFolderOfAMountedDriveIsNotReportedAsUnmounted(t *testing.T) {
|
||||
sub := StableParentDir + "/scratch_hdd/userdata/nextcloud"
|
||||
mounted := StableParentDir + "/scratch_hdd"
|
||||
absent := StableParentDir + "/usb"
|
||||
absentSub := absent + "/userdata/romm"
|
||||
isMount := func(p string) bool { return p == mounted }
|
||||
var buf bytes.Buffer
|
||||
var skeletons []string
|
||||
mounts, cfg := buildFileBrowserPaths([]settings.StoragePath{{Path: sub}, {Path: mounted}, {Path: absent}, {Path: absentSub}}, fbPathDeps{
|
||||
isMount: isMount,
|
||||
ensureSkeleton: func(p string) error { skeletons = append(skeletons, p); return nil },
|
||||
logger: log.New(&buf, "", 0),
|
||||
})
|
||||
logged := buf.String()
|
||||
if strings.Contains(logged, sub) {
|
||||
t.Errorf("a folder of a mounted drive is reported: %q", logged)
|
||||
}
|
||||
if !strings.Contains(logged, "drive "+absent+" not mounted") {
|
||||
t.Errorf("a drive that is really absent keeps its line: %q", logged)
|
||||
}
|
||||
if !strings.Contains(logged, "drive "+absent+" not mounted") || strings.Count(logged, "not mounted") != 2 || !strings.Contains(logged, absentSub) {
|
||||
t.Errorf("a folder of an absent drive is reported, naming that drive: %q", logged)
|
||||
}
|
||||
// The behaviour stays: only the mounted drive itself gets a skeleton and a bind.
|
||||
if len(skeletons) != 1 || skeletons[0] != mounted {
|
||||
t.Errorf("skeletons %v — only the mounted drive gets one", skeletons)
|
||||
}
|
||||
if len(mounts) != 1 || !strings.Contains(mounts[0], ":/srv/scratch_hdd") {
|
||||
t.Errorf("mounts %v — only the mounted drive's userdata is bound", mounts)
|
||||
}
|
||||
if len(cfg) != 4 {
|
||||
t.Errorf("the source list keeps every registered path: %v", cfg)
|
||||
}
|
||||
}
|
||||
|
||||
// R-1351: the web sync's compose call runs inside the stack lock it shares with the base stack's bring-up. The lock's
|
||||
// behaviour is TestR1351_TheTwoFileBrowserStartPathsNeverOverlap (internal/stacks); this pins the web side's use of it.
|
||||
func TestR1351_TheWebSyncTakesTheLockBeforeCompose(t *testing.T) {
|
||||
b, err := os.ReadFile("handlers.go")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
src := string(b)
|
||||
start := strings.Index(src, "func (s *Server) syncFileBrowserMounts(")
|
||||
if start < 0 {
|
||||
t.Fatal("syncFileBrowserMounts not found")
|
||||
}
|
||||
body := src[start:]
|
||||
lock := strings.Index(body, "s.stackMgr.LockFileBrowserStack()")
|
||||
down := strings.Index(body, `"docker", "compose", "down", "-v"`)
|
||||
up := strings.Index(body, `[]string{"compose", "up", "-d", "--remove-orphans"}`)
|
||||
if lock < 0 || down < 0 || up < 0 || lock > down || lock > up {
|
||||
t.Fatalf("the lock must come before both compose calls (lock %d, down %d, up %d)", lock, down, up)
|
||||
}
|
||||
}
|
||||
@@ -186,6 +186,7 @@ func (s *Server) dashboardHandler(w http.ResponseWriter, r *http.Request) {
|
||||
data["TitleKey"] = "page.title.dashboard" // i18n: the Hungarian title above is what hu renders
|
||||
s.addRecoveryBanner(data, r) // R-241: the reminder bar, per visit
|
||||
data["SettingsWarning"] = s.settings.LoadWarning // non-empty if settings.json was recovered from corruption
|
||||
data["OperatorRebootNotice"] = s.operatorRebootNotice(s.langFor(r), time.Now()) // D65: „Felhom support restarted your box"
|
||||
// R-560: the app rows carry catalog copy (each row's `.Meta.Description`), so the list the
|
||||
// TEMPLATE sees is the localised view. The lists the warning helpers below see are the
|
||||
// originals — they key on stack NAMES, and handing them a translated copy would only widen
|
||||
@@ -1346,6 +1347,7 @@ func (s *Server) backupsRestoreHandler(w http.ResponseWriter, r *http.Request) {
|
||||
// D25: second-drive copies whose source is gone — kept until the household removes one.
|
||||
rows := s.keptCopyRows(s.langFor(r))
|
||||
data["KeptCopies"] = rows
|
||||
data["RestoreCopies"] = s.restoreCopyRows(s.langFor(r)) // D62: restore copies in the same list
|
||||
for _, k := range rows {
|
||||
if k.Restorable {
|
||||
data["KeptUnitRestorable"] = true // D47: the confirm script only when a row offers „Visszaállítás"
|
||||
@@ -1894,7 +1896,10 @@ func (s *Server) backupRestoreHandler(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
|
||||
s.logger.Printf("[WARN] [web] Restore requested (async): stack=%s, snapshot=%s from %s", stackName, snapshotID, r.RemoteAddr)
|
||||
s.backupMgr.BeginRestoreOp("restore", stackName)
|
||||
if msg, ok := s.beginRestoreOp("restore", stackName); !ok {
|
||||
http.Redirect(w, r, "/backups/restore?flash_error="+url.QueryEscape(msg), http.StatusFound)
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
start := time.Now()
|
||||
// Phase 2b: restore from the app's recovery unit (recovers secrets from the guest, fail-closed
|
||||
@@ -2236,7 +2241,10 @@ func (s *Server) backupTier2RestoreHandler(w http.ResponseWriter, r *http.Reques
|
||||
}
|
||||
|
||||
s.logger.Printf("[WARN] [web] Tier-2 file restore requested (async): stack=%s from %s", stackName, r.RemoteAddr)
|
||||
s.backupMgr.BeginRestoreOp("tier2-restore", stackName)
|
||||
if msg, ok := s.beginRestoreOp("tier2-restore", stackName); !ok {
|
||||
http.Redirect(w, r, "/backups/apps?flash_error="+url.QueryEscape(msg), http.StatusFound)
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
n, err := s.backupMgr.RestoreTier2Files(stackName)
|
||||
if err != nil {
|
||||
@@ -2311,7 +2319,10 @@ func (s *Server) backupTier2UnitRestoreHandler(w http.ResponseWriter, r *http.Re
|
||||
// (R-538), which is what left such an app with no way back from the second drive.
|
||||
if s.backupMgr.HasDriveFileLegs(stackName) {
|
||||
s.logger.Printf("[WARN] [web] Tier-2 WHOLE restore requested (async, files + database): stack=%s from %s", stackName, r.RemoteAddr)
|
||||
s.backupMgr.BeginRestoreOp("tier2-whole-restore", stackName)
|
||||
if msg, ok := s.beginRestoreOp("tier2-whole-restore", stackName); !ok {
|
||||
http.Redirect(w, r, "/backups/apps?flash_error="+url.QueryEscape(msg), http.StatusFound)
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
start := time.Now()
|
||||
res, err := s.backupMgr.RestoreTier2Whole(stackName)
|
||||
@@ -2346,7 +2357,10 @@ func (s *Server) backupTier2UnitRestoreHandler(w http.ResponseWriter, r *http.Re
|
||||
}
|
||||
|
||||
s.logger.Printf("[WARN] [web] Tier-2 UNIT restore requested (async, OVERWRITES live data): stack=%s from %s", stackName, r.RemoteAddr)
|
||||
s.backupMgr.BeginRestoreOp("tier2-unit-restore", stackName)
|
||||
if msg, ok := s.beginRestoreOp("tier2-unit-restore", stackName); !ok {
|
||||
http.Redirect(w, r, "/backups/apps?flash_error="+url.QueryEscape(msg), http.StatusFound)
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
start := time.Now()
|
||||
res, err := s.backupMgr.RestoreTier2Unit(stackName)
|
||||
@@ -3755,6 +3769,15 @@ func skipFileBrowserPath(path string, isMount func(string) bool) bool {
|
||||
return strings.HasPrefix(path, StableParentDir+"/") && !isMount(path)
|
||||
}
|
||||
|
||||
// fbDriveRoot is the drive a path under StableParentDir lives on (…/scratch_hdd/userdata/x → …/scratch_hdd).
|
||||
func fbDriveRoot(path string) string {
|
||||
rest := strings.TrimPrefix(path, StableParentDir+"/")
|
||||
if i := strings.Index(rest, "/"); i >= 0 {
|
||||
rest = rest[:i]
|
||||
}
|
||||
return StableParentDir + "/" + rest
|
||||
}
|
||||
|
||||
func (s *Server) syncFileBrowserMounts(resetDBOnChange bool) {
|
||||
// R-695 (v0.275.0): SINGLE-FLIGHT. A request is covered by any sync that STARTS reading the state
|
||||
// after the request was made: a caller that waited behind a running sync returns as soon as one
|
||||
@@ -3903,6 +3926,13 @@ func (s *Server) syncFileBrowserMounts(resetDBOnChange bool) {
|
||||
}
|
||||
s.fbForceMu.Unlock()
|
||||
|
||||
// R-1351: the base stack's bring-up may run `compose up` on this same stack in the same second (a guest boot); the loser
|
||||
// logged „Conflict. The container name … is already in use". One lock over both compose paths.
|
||||
if s.stackMgr != nil {
|
||||
unlock := s.stackMgr.LockFileBrowserStack()
|
||||
defer unlock()
|
||||
}
|
||||
|
||||
// If sources changed and caller requested a DB reset (restore flow),
|
||||
// nuke the data volume so FileBrowser re-reads config.yaml from scratch.
|
||||
// Normal operations skip this to preserve user accounts, permissions, and share links.
|
||||
@@ -4001,8 +4031,16 @@ func buildFileBrowserPaths(paths []settings.StoragePath, d fbPathDeps) (storageM
|
||||
// FileBrowser this pass. It returns on the next sync after reconnect. Matches planDriveGates'
|
||||
// external-only rule (system paths, not under StableParentDir, are never skipped).
|
||||
if skipFileBrowserPath(sp.Path, d.isMount) {
|
||||
// R-1339: a per-app folder (<drive>/userdata/<app>) is never a mountpoint itself. Under a MOUNTED drive it is shown
|
||||
// through the drive's own bind, so it gets no bind of its own and there is nothing to report: the old „not mounted"
|
||||
// line was false nine times at every start on a box whose drive worked. Pinned by TestR1339_*.
|
||||
if d.logger != nil {
|
||||
switch root := fbDriveRoot(sp.Path); {
|
||||
case root == sp.Path:
|
||||
d.logger.Printf("[INFO] [web] FileBrowser: drive %s not mounted — skipping userdata skeleton", sp.Path)
|
||||
case !d.isMount(root):
|
||||
d.logger.Printf("[INFO] [web] FileBrowser: drive %s not mounted — skipping its folder %s", root, sp.Path)
|
||||
}
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -255,6 +255,21 @@ func i18nCasesB() []i18nCase {
|
||||
d["KeptUnitRestorable"] = true
|
||||
return d
|
||||
}},
|
||||
// D62 (2026-10-29): restore copies in the kept-copies list — one deletable (the warning and the typed name), one a
|
||||
// hold needs (the button off, the reason). A NEW case on a box without off-site (the second call site), so no
|
||||
// existing fixture changes. Line, Warning, TypePrompt and NeededText are composed in Go in the request's language;
|
||||
// the fixture carries the same data in both renders.
|
||||
{"backups_restore_restore_copies", "backups_restore", func() map[string]interface{} {
|
||||
d := i18nLayoutData("backups-restore", "Visszaállítás a mentésből")
|
||||
d["Backup"] = i18nBackupStatus(false, false)
|
||||
d["OffboxConfigured"] = false
|
||||
d["RestoreCopies"] = []restoreCopyRow{
|
||||
{App: "vikunja", Stamp: "20261029T080000Z", Name: "Vikunja", Line: "2026-10-29 09:00 · 12 MB · 2026-11-28",
|
||||
Warning: "Vikunja 2026-10-29", TypePrompt: "Vikunja"},
|
||||
{App: "immich", Stamp: "20261020T080000Z", Name: "Immich Photos", Line: "2026-10-20 10:00 · 2 GB", NeededText: "hold: support decides"},
|
||||
}
|
||||
return d
|
||||
}},
|
||||
{"backups_restore_known_empty", "backups_restore", func() map[string]interface{} {
|
||||
d := i18nLayoutData("backups-restore", "Visszaállítás a mentésből")
|
||||
d["Backup"] = i18nBackupStatus(false, false)
|
||||
|
||||
@@ -1,12 +1,16 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/i18n"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
|
||||
)
|
||||
|
||||
// D25 (operator ruling 2026-10-21; R-1175 + R-1181): the restore page lists the second-drive copies whose source is
|
||||
@@ -106,7 +110,10 @@ func (s *Server) keptUnitRestoreHandler(w http.ResponseWriter, r *http.Request)
|
||||
return
|
||||
}
|
||||
s.logger.Printf("[WARN] [web] Restore of %s from a set-aside unit requested (async): %s from %s", k.Source, k.Path, r.RemoteAddr)
|
||||
s.backupMgr.BeginRestoreOp("restore", k.Source)
|
||||
if msg, ok := s.beginRestoreOp("restore", k.Source); !ok {
|
||||
http.Redirect(w, r, "/backups/restore?flash_error="+url.QueryEscape(msg), http.StatusFound)
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
app, res, err := s.backupMgr.RestoreSetAsideUnit(id)
|
||||
if err != nil {
|
||||
@@ -118,3 +125,134 @@ func (s *Server) keptUnitRestoreHandler(w http.ResponseWriter, r *http.Request)
|
||||
}()
|
||||
http.Redirect(w, r, "/backups/restore?"+flashQuery("flash", "flash.restore.started"), http.StatusFound)
|
||||
}
|
||||
|
||||
// ── D62 (operator ruling 2026-10-29): restore copies in the same list ───────────────────────────────────────────────
|
||||
//
|
||||
// Each restore copy shows the app, the date of the restore it protects, its size and the day the box deletes it by
|
||||
// itself (30 days after the restore). „Törlés most" deletes it earlier: a warning that names the app and the date,
|
||||
// and the app's name typed. A copy a hold or a pending put-back needs shows why in one line and has no button — and
|
||||
// the handler refuses it too. Pinned by TestD62_* (web).
|
||||
|
||||
// restoreCopyRow is one restore copy on the page.
|
||||
type restoreCopyRow struct {
|
||||
App string
|
||||
Stamp string
|
||||
Name string // the app's display name — what the household types
|
||||
Line string // „visszaállítás: <date> · <size> · automatikusan törlődik: <date>"
|
||||
NeededText string // why it cannot be deleted; "" = the button is offered
|
||||
Warning string // the confirm step's sentence (app and date)
|
||||
TypePrompt string
|
||||
}
|
||||
|
||||
// restoreCopyAppName is the app's display name, or its slug when the app is gone.
|
||||
func (s *Server) restoreCopyAppName(app string) string {
|
||||
if s.stackMgr != nil {
|
||||
if st, ok := s.stackMgr.GetStack(app); ok && st.Meta.DisplayName != "" {
|
||||
return st.Meta.DisplayName
|
||||
}
|
||||
}
|
||||
return app
|
||||
}
|
||||
|
||||
func (s *Server) restoreCopyRows(lang string) []restoreCopyRow {
|
||||
if s.backupMgr == nil {
|
||||
return nil
|
||||
}
|
||||
var out []restoreCopyRow
|
||||
for _, c := range s.backupMgr.RestoreCopies() {
|
||||
name := s.restoreCopyAppName(c.App)
|
||||
at := c.At.In(getTimezone())
|
||||
line := s.msgLang(lang, "restore_copy.restored_on", fmtLocalTime(lang, c.At)) + " · " +
|
||||
i18n.LocalizeDecimals(lang, backup.HumanizeBytes(c.SizeBytes))
|
||||
row := restoreCopyRow{App: c.App, Stamp: c.Stamp, Name: name}
|
||||
if c.Needed == "" {
|
||||
line += " · " + s.msgLang(lang, "restore_copy.auto_delete", i18n.FormatDate(lang, c.DeleteOn.In(getTimezone())))
|
||||
row.Warning = s.msgLang(lang, "restore_copy.warning", name, strings.TrimSuffix(i18n.FormatDate(lang, at), "."))
|
||||
row.TypePrompt = s.msgLang(lang, "kept.delete.type", name)
|
||||
} else {
|
||||
row.NeededText = s.msgLang(lang, "restore_copy.needed."+c.Needed)
|
||||
}
|
||||
row.Line = line
|
||||
out = append(out, row)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// restoreCopyJSON is the delete's answer: ok with the page to go to, or the refusal's sentence.
|
||||
func restoreCopyJSON(w http.ResponseWriter, code int, ok bool, msg, redirect string) {
|
||||
w.Header().Set("Content-Type", "application/json; charset=utf-8")
|
||||
w.WriteHeader(code)
|
||||
_ = json.NewEncoder(w).Encode(map[string]interface{}{"ok": ok, "error": msg, "redirect": redirect})
|
||||
}
|
||||
|
||||
// restoreCopyDeleteHandler is „Törlés most": POST /backup/restore-copy/delete (app, stamp, confirm). JSON answer:
|
||||
// 200 deleted; 400 the typed name does not match; 404 no such copy; 409 the copy is needed, or an operation runs on
|
||||
// the box or the app; 500 the removal failed. CSRF as every state-changing form (CsrfProtect).
|
||||
func (s *Server) restoreCopyDeleteHandler(w http.ResponseWriter, r *http.Request) {
|
||||
lang := s.langFor(r)
|
||||
if s.backupMgr == nil {
|
||||
restoreCopyJSON(w, http.StatusServiceUnavailable, false, s.msgLang(lang, "flash.offbox.mgr_unreachable"), "")
|
||||
return
|
||||
}
|
||||
_ = r.ParseForm()
|
||||
app, stamp := strings.TrimSpace(r.FormValue("app")), strings.TrimSpace(r.FormValue("stamp"))
|
||||
var found *backup.RestoreCopy
|
||||
for _, c := range s.backupMgr.RestoreCopies() {
|
||||
if c.App == app && c.Stamp == stamp {
|
||||
c := c
|
||||
found = &c
|
||||
break
|
||||
}
|
||||
}
|
||||
if found == nil {
|
||||
s.logger.Printf("[WARN] [web] restore-copy delete REFUSED: %q/%q is not a listed copy (from %s)", app, stamp, clientIP(r))
|
||||
restoreCopyJSON(w, http.StatusNotFound, false, s.msgLang(lang, "restore_copy.not_found"), "")
|
||||
return
|
||||
}
|
||||
if found.Needed != "" {
|
||||
s.logger.Printf("[WARN] [web] restore-copy delete REFUSED for %s: the copy is needed (%s)", found.Path, found.Needed)
|
||||
restoreCopyJSON(w, http.StatusConflict, false, s.msgLang(lang, "restore_copy.needed."+found.Needed), "")
|
||||
return
|
||||
}
|
||||
name := s.restoreCopyAppName(app)
|
||||
if strings.TrimSpace(r.FormValue("confirm")) != name {
|
||||
s.logger.Printf("[WARN] [web] restore-copy delete REFUSED for %s: the typed confirmation did not match", found.Path)
|
||||
restoreCopyJSON(w, http.StatusBadRequest, false, s.msgLang(lang, "kept.delete.mismatch", name), "")
|
||||
return
|
||||
}
|
||||
if s.stackMgr != nil && s.stackMgr.IsUpdating(app) {
|
||||
s.logger.Printf("[WARN] [web] restore-copy delete REFUSED for %s: an update of %s runs", found.Path, app)
|
||||
restoreCopyJSON(w, http.StatusConflict, false, s.msgLang(lang, "restore_copy.app_busy"), "")
|
||||
return
|
||||
}
|
||||
who := "the household (dashboard sign-in) from " + clientIP(r)
|
||||
c, err := s.backupMgr.DeleteRestoreCopy(app, stamp)
|
||||
switch {
|
||||
case errors.Is(err, backup.ErrRestoreCopyBusy):
|
||||
busy, mirror := s.backupMgr.RestoreBusy()
|
||||
msg, blocked := s.restoreBusyMessage(busy, mirror)
|
||||
if !blocked {
|
||||
msg = s.msgLang(lang, "restore_copy.app_busy")
|
||||
}
|
||||
s.logger.Printf("[WARN] [web] restore-copy delete REFUSED for %s: a backup or restore operation runs", found.Path)
|
||||
restoreCopyJSON(w, http.StatusConflict, false, msg, "")
|
||||
return
|
||||
case errors.Is(err, backup.ErrRestoreCopyNeeded):
|
||||
restoreCopyJSON(w, http.StatusConflict, false, s.msgLang(lang, "restore_copy.needed."+c.Needed), "")
|
||||
return
|
||||
case errors.Is(err, backup.ErrRestoreCopyNotFound):
|
||||
restoreCopyJSON(w, http.StatusNotFound, false, s.msgLang(lang, "restore_copy.not_found"), "")
|
||||
return
|
||||
case err != nil:
|
||||
s.logger.Printf("[ERROR] [web] restore-copy delete of %s failed: %v", found.Path, err)
|
||||
restoreCopyJSON(w, http.StatusInternalServerError, false, s.msgLang(lang, "restore_copy.failed", err.Error()), "")
|
||||
return
|
||||
}
|
||||
restoreAt := c.At.UTC().Format(time.RFC3339)
|
||||
s.logger.Printf("[WARN] [web] restore copy DELETED by hand (D62): app=%s restore_at=%s size=%d path=%s who=%s",
|
||||
app, restoreAt, c.SizeBytes, c.Path, who)
|
||||
if s.notifier != nil {
|
||||
s.notifier.NotifyRestoreCopyDeleted(notify.RestoreCopyDeletedDetails{App: app, RestoreAt: restoreAt, SizeBytes: c.SizeBytes, Who: who, Path: c.Path})
|
||||
}
|
||||
restoreCopyJSON(w, http.StatusOK, true, "", "/backups/restore?"+flashQuery("flash", "flash.restore_copy.deleted", name))
|
||||
}
|
||||
@@ -217,16 +217,23 @@ func (s *Server) keptLoadHandler(w http.ResponseWriter, r *http.Request) {
|
||||
_, err := s.stackMgr.RestoreKeptFiles(it)
|
||||
return err
|
||||
}
|
||||
// R-1256 H6: the load begins in the same step as the check for every other operation; a refusal moves nothing
|
||||
// (the dated folder's files move back inside the operation, as prepare).
|
||||
var lErr error
|
||||
if c.Tier == backup.KeptTierOffsite {
|
||||
s.logger.Printf("[INFO] [web] kept LOAD %s: %s from the off-site snapshot %s (from %s)", it.App, it.Path, c.SnapshotID, r.RemoteAddr)
|
||||
s.backupMgr.LoadKeptOffsite(app, it.Drive, c, moveBack, okMsg, failMsg, after)
|
||||
lErr = s.backupMgr.LoadKeptOffsite(app, it.Drive, c, moveBack, okMsg, failMsg, after)
|
||||
} else {
|
||||
if err := moveBack(); err != nil {
|
||||
s.keptRedirect(w, r, "flash_error", s.errText(r, err))
|
||||
return
|
||||
}
|
||||
s.logger.Printf("[INFO] [web] kept LOAD %s: %s from %s (from %s)", it.App, it.Path, c.UnitDir, r.RemoteAddr)
|
||||
s.backupMgr.LoadKeptApp(app, c.UnitDir, okMsg, failMsg, after)
|
||||
lErr = s.backupMgr.LoadKeptApp(app, c.UnitDir, moveBack, okMsg, failMsg, after)
|
||||
}
|
||||
if lErr != nil {
|
||||
msg, _ := s.restoreOpBlocked()
|
||||
if msg == "" { // the other operation ended in the meantime
|
||||
msg, _ = s.restoreBusyMessage(backup.RestoreBusyRestore, "")
|
||||
}
|
||||
s.keptRedirect(w, r, "flash_error", msg)
|
||||
return
|
||||
}
|
||||
http.Redirect(w, r, "/backups/restore?"+flashQuery("flash", "flash.restore.started"), http.StatusFound)
|
||||
}
|
||||
|
||||
@@ -450,7 +450,10 @@ func (s *Server) offboxRestoreHandler(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
full := mode == "full"
|
||||
s.backupMgr.BeginRestoreOp("offbox-restore", app)
|
||||
if msg, ok := s.beginRestoreOp("offbox-restore", app); !ok {
|
||||
offboxRedirectTo(w, r, restoreWizardPath(app), msg, true)
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
|
||||
defer cancel()
|
||||
@@ -538,7 +541,10 @@ func (s *Server) offboxReconstituteHandler(w http.ResponseWriter, r *http.Reques
|
||||
// data"; this one answers "yes, into a different place than the backup recorded". One checkbox
|
||||
// carrying both would be the two-decisions-one-button shape R-48 removed from this surface.
|
||||
ackPlacement := r.FormValue("ack_placement") == "1"
|
||||
s.backupMgr.BeginRestoreOp("offbox-reconstitute", app)
|
||||
if msg, ok := s.beginRestoreOp("offbox-reconstitute", app); !ok {
|
||||
offboxRedirectTo(w, r, restoreWizardPath(app), msg, true)
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Minute)
|
||||
defer cancel()
|
||||
@@ -678,7 +684,10 @@ func (s *Server) offboxPlaceHandler(w http.ResponseWriter, r *http.Request) {
|
||||
offboxRedirectTo(w, r, restoreWizardPath(app), offsiteScratchIncompleteKey, true)
|
||||
return
|
||||
}
|
||||
s.backupMgr.BeginRestoreOp("offbox-place", app)
|
||||
if msg, ok := s.beginRestoreOp("offbox-place", app); !ok {
|
||||
offboxRedirectTo(w, r, restoreWizardPath(app), msg, true)
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
|
||||
defer cancel()
|
||||
@@ -711,7 +720,10 @@ func (s *Server) sharesRestoreHandler(w http.ResponseWriter, r *http.Request) {
|
||||
offboxRedirectTo(w, r, "/backups/restore", msg, true)
|
||||
return
|
||||
}
|
||||
s.backupMgr.BeginRestoreOp("shares-restore", backup.SharesDisplayName)
|
||||
if msg, ok := s.beginRestoreOp("shares-restore", backup.SharesDisplayName); !ok {
|
||||
offboxRedirectTo(w, r, "/backups/restore", msg, true)
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
|
||||
defer cancel()
|
||||
@@ -737,7 +749,10 @@ func (s *Server) sharesPlaceHandler(w http.ResponseWriter, r *http.Request) {
|
||||
offboxRedirectTo(w, r, "/backups/restore", msg, true)
|
||||
return
|
||||
}
|
||||
s.backupMgr.BeginRestoreOp("shares-place", backup.SharesDisplayName)
|
||||
if msg, ok := s.beginRestoreOp("shares-place", backup.SharesDisplayName); !ok {
|
||||
offboxRedirectTo(w, r, "/backups/restore", msg, true)
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
|
||||
)
|
||||
|
||||
// „Felhom support restarted your box at <time>" (D65 = A, operator ruling 2026-10-29 15:50; felhom.eu `04` §6.1).
|
||||
//
|
||||
// The operator can reboot the guest from the hub; the household must see afterwards that Felhom did it. The agent
|
||||
// writes its record BEFORE the reboot, so this controller — which starts with the guest — asks for it once at start
|
||||
// (GET /guest/operator-reboot) and the dashboard shows one neutral line until a day has passed since the reboot.
|
||||
// Pinned by operator_reboot_notice_test.go.
|
||||
|
||||
// operatorRebootNoticeFor is how long the line shows (the agent also stops answering after a day).
|
||||
const operatorRebootNoticeFor = 24 * time.Hour
|
||||
|
||||
// SetOperatorReboot records the last operator reboot (the zero time = none).
|
||||
func (s *Server) SetOperatorReboot(at time.Time) {
|
||||
s.opRebootMu.Lock()
|
||||
defer s.opRebootMu.Unlock()
|
||||
s.opRebootAt = at
|
||||
}
|
||||
|
||||
// operatorRebootNotice is the dashboard's line for lang at now ("" = none).
|
||||
func (s *Server) operatorRebootNotice(lang string, now time.Time) string {
|
||||
s.opRebootMu.Lock()
|
||||
at := s.opRebootAt
|
||||
s.opRebootMu.Unlock()
|
||||
if at.IsZero() || now.Sub(at) >= operatorRebootNoticeFor || now.Before(at) {
|
||||
return ""
|
||||
}
|
||||
return s.i18n.Msgf(lang, "dashboard.operator_reboot", at.In(budapestLocation()).Format("15:04"))
|
||||
}
|
||||
|
||||
// WatchOperatorReboot asks the agent once at start, retrying while the agent does not answer (it may start after us),
|
||||
// at most for opRebootAskFor. A 404 (an older agent) ends it at once.
|
||||
func (s *Server) WatchOperatorReboot(ctx context.Context) {
|
||||
fetch := s.opRebootFetch
|
||||
if fetch == nil {
|
||||
fetch = func(ctx context.Context) (time.Time, error) {
|
||||
c, err := s.agentClient()
|
||||
if err != nil {
|
||||
return time.Time{}, err
|
||||
}
|
||||
return c.OperatorReboot(ctx)
|
||||
}
|
||||
}
|
||||
every := s.opRebootRetry
|
||||
if every <= 0 {
|
||||
every = 30 * time.Second
|
||||
}
|
||||
for deadline := time.Now().Add(10 * time.Minute); ; {
|
||||
cctx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
at, err := fetch(cctx)
|
||||
cancel()
|
||||
var se *agentapi.StatusError
|
||||
switch {
|
||||
case err == nil:
|
||||
if !at.IsZero() {
|
||||
s.SetOperatorReboot(at)
|
||||
s.logger.Printf("[INFO] [web] the operator rebooted this box from the hub at %s — the dashboard says so for a day", at.UTC().Format(time.RFC3339))
|
||||
}
|
||||
return
|
||||
case errors.As(err, &se) && se.Code == 404:
|
||||
return // an agent older than the route cannot reboot from the hub
|
||||
}
|
||||
if !time.Now().Before(deadline) {
|
||||
s.logger.Printf("[WARN] [web] could not ask the agent about an operator reboot: %v — no notice on the dashboard", err)
|
||||
return
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(every):
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
|
||||
)
|
||||
|
||||
// D65 = A: after the operator rebooted the box from the hub, the household's dashboard says so — one neutral line,
|
||||
// for one day, then gone.
|
||||
|
||||
// The real dashboard, through ServeHTTP: the line is there after an operator reboot, in Hungarian, with the Budapest
|
||||
// time; it is gone a day later. RED-PROOF: drop the data["OperatorRebootNotice"] line from dashboardHandler → FAILS;
|
||||
// drop the 24-hour check → the day-later page still shows it → FAILS.
|
||||
func TestOperatorRebootNotice_OnTheDashboardForADay(t *testing.T) {
|
||||
s := newDashboardServer(t, time.Time{})
|
||||
if body := getPage(t, s, "/dashboard").Body.String(); strings.Contains(body, "data-operator-reboot") {
|
||||
t.Fatal("no reboot yet, but the dashboard shows the line")
|
||||
}
|
||||
at := time.Now().Add(-2 * time.Hour)
|
||||
s.SetOperatorReboot(at)
|
||||
body := getPage(t, s, "/dashboard").Body.String()
|
||||
want := "A Felhom ügyfélszolgálata " + at.In(budapestLocation()).Format("15:04") + "-kor újraindította a dobozt."
|
||||
if !strings.Contains(body, want) {
|
||||
t.Fatalf("the dashboard lacks %q", want)
|
||||
}
|
||||
s.SetOperatorReboot(time.Now().Add(-25 * time.Hour))
|
||||
if body := getPage(t, s, "/dashboard").Body.String(); strings.Contains(body, "data-operator-reboot") {
|
||||
t.Fatal("a day later the line must be gone")
|
||||
}
|
||||
}
|
||||
|
||||
// English: the reader's language.
|
||||
func TestOperatorRebootNotice_English(t *testing.T) {
|
||||
s := newDashboardServer(t, time.Time{})
|
||||
at := time.Date(2026, 10, 30, 9, 41, 0, 0, time.UTC)
|
||||
s.SetOperatorReboot(at)
|
||||
got := s.operatorRebootNotice("en", at.Add(time.Hour))
|
||||
if got != "Felhom support restarted your box at "+at.In(budapestLocation()).Format("15:04")+"." {
|
||||
t.Fatalf("en: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The start-up ask: it retries while the agent does not answer, keeps the time it gets, and stops at once on a 404
|
||||
// (an agent older than the route cannot reboot from the hub). RED-PROOF: drop SetOperatorReboot from the success
|
||||
// branch → FAILS; return on the first error → the retried answer is lost → FAILS.
|
||||
func TestWatchOperatorReboot_RetriesAndKeepsTheTime(t *testing.T) {
|
||||
s := &Server{logger: log.New(io.Discard, "", 0), opRebootRetry: time.Millisecond}
|
||||
at := time.Date(2026, 10, 30, 9, 41, 0, 0, time.UTC)
|
||||
calls := 0
|
||||
s.opRebootFetch = func(context.Context) (time.Time, error) {
|
||||
calls++
|
||||
if calls < 3 {
|
||||
return time.Time{}, errors.New("agentapi: GET /guest/operator-reboot: connection refused")
|
||||
}
|
||||
return at, nil
|
||||
}
|
||||
s.WatchOperatorReboot(context.Background())
|
||||
if calls != 3 || !s.opRebootAt.Equal(at) {
|
||||
t.Fatalf("calls=%d at=%v, want 3 and %v", calls, s.opRebootAt, at)
|
||||
}
|
||||
|
||||
s2 := &Server{logger: log.New(io.Discard, "", 0), opRebootRetry: time.Millisecond}
|
||||
calls = 0
|
||||
s2.opRebootFetch = func(context.Context) (time.Time, error) {
|
||||
calls++
|
||||
return time.Time{}, &agentapi.StatusError{Path: "/guest/operator-reboot", Code: 404}
|
||||
}
|
||||
s2.WatchOperatorReboot(context.Background())
|
||||
if calls != 1 || !s2.opRebootAt.IsZero() {
|
||||
t.Fatalf("a 404 must end the ask at once: calls=%d at=%v", calls, s2.opRebootAt)
|
||||
}
|
||||
}
|
||||
@@ -93,7 +93,9 @@ func TestR1151_ConfirmAnswersInTheBoxLanguage(t *testing.T) {
|
||||
{http.StatusGone, http.StatusGone, true, "Lejárt az idő", "has run out"},
|
||||
{http.StatusNotFound, http.StatusGone, true, "már nem vár megerősítésre", "no longer waits"},
|
||||
{http.StatusBadGateway, http.StatusBadGateway, false, "nem sikerült elmenteni", "could not be saved"},
|
||||
{0, http.StatusBadGateway, false, "nem sikerült elmenteni", "could not be saved"},
|
||||
// R-1254 H2 (2026-10-28): a failed call (0) or an agent that cannot learn the outcome (504) promises nothing.
|
||||
{0, http.StatusGatewayTimeout, false, "Nem tudjuk biztosan", "cannot tell yet"},
|
||||
{http.StatusGatewayTimeout, http.StatusGatewayTimeout, false, "Nem tudjuk biztosan", "cannot tell yet"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
for _, lang := range []string{"hu", "en"} {
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1254 H2 (the household's sentence): a confirm whose upload answer is late or lost must not promise „your previous
|
||||
// code still works" — the hub may hold the new package. Since agent 0.168.0 the agent asks the hub and answers 502 only
|
||||
// when the hub holds ANOTHER package; 504 = unknown. A controller-side transport failure (status 0 — the call itself
|
||||
// timed out while the agent may still store) is unknown too.
|
||||
// RED-PROOF: drop the 504/0 case from escrowConfirmAPIHandler → the default sentence („a régi kódod továbbra is
|
||||
// érvényes") is shown → FAILS.
|
||||
func TestR1254_AnUnknownConfirmOutcomePromisesNothing(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
status int
|
||||
}{{"agent 504", http.StatusGatewayTimeout}, {"transport failure", 0}} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
h := newEscrowWizardHarness(t)
|
||||
h.configureOffbox(t, "pending")
|
||||
h.agent.confirmStatus, h.agent.confirmErr = tc.status, errors.New("late")
|
||||
for _, lang := range []string{"hu", "en"} {
|
||||
w := postConfirm(t, h.s, lang, `{"words":["ketto","negy"]}`)
|
||||
low := strings.ToLower(w.Body.String())
|
||||
// ASCII fragments (the accented forms carry these stems): „régi kódod továbbra is érvényes" / „still works".
|
||||
if strings.Contains(low, "tovabbra is") || strings.Contains(low, "továbbra is") || strings.Contains(low, "still works") {
|
||||
t.Fatalf("%s: an unknown outcome promised the previous code: %s", lang, w.Body.String())
|
||||
}
|
||||
if !strings.Contains(low, "mindket") && !strings.Contains(low, "mindkét") && !strings.Contains(low, "both") {
|
||||
t.Fatalf("%s: the sentence must tell the household to keep both codes: %s", lang, w.Body.String())
|
||||
}
|
||||
}
|
||||
if ceremonyStamped(t, h) {
|
||||
t.Fatal("an unknown outcome stamped the ceremony completed")
|
||||
}
|
||||
})
|
||||
}
|
||||
// Control: the agent says the hub holds another package (502) → the old sentence, now true.
|
||||
h := newEscrowWizardHarness(t)
|
||||
h.configureOffbox(t, "pending")
|
||||
h.agent.confirmStatus, h.agent.confirmErr = http.StatusBadGateway, errors.New("not stored")
|
||||
if w := postConfirm(t, h.s, "en", `{"words":["ketto","negy"]}`); !strings.Contains(w.Body.String(), "still works") {
|
||||
t.Fatalf("502 must keep the store-failed sentence: %s", w.Body.String())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
|
||||
)
|
||||
|
||||
// R-1256 H6: every restore handler records its op through beginRestoreOp, which asks about every other operation in
|
||||
// the same step (backup.TryBeginRestoreOp). A handler that calls BeginRestoreOp directly re-opens the window between
|
||||
// the early refusal and the record. RED-PROOF: put one `s.backupMgr.BeginRestoreOp(...)` back → FAILS, naming the file.
|
||||
func TestR1256_H6_NoHandlerBeginsARestoreUnchecked(t *testing.T) {
|
||||
files, _ := filepath.Glob("*.go")
|
||||
for _, f := range files {
|
||||
if strings.HasSuffix(f, "_test.go") {
|
||||
continue
|
||||
}
|
||||
src, err := os.ReadFile(f)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
fs := token.NewFileSet()
|
||||
af, err := parser.ParseFile(fs, f, src, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ast.Inspect(af, func(n ast.Node) bool {
|
||||
if sel, ok := n.(*ast.SelectorExpr); ok && sel.Sel.Name == "BeginRestoreOp" {
|
||||
t.Errorf("%s: a direct BeginRestoreOp — begin through s.beginRestoreOp (R-1256 H6)", fs.Position(sel.Pos()))
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The handler's begin refuses with the household's sentence when a second-drive copy began after the early check.
|
||||
func TestR1256_H6_TheBeginRefusesWithTheCopySentence(t *testing.T) {
|
||||
s, _, _ := newOffboxWebServer(t)
|
||||
if _, blocked := s.restoreOpBlocked(); blocked {
|
||||
t.Fatal("fixture: nothing runs")
|
||||
}
|
||||
done := backup.MarkTier2CopyForTest("immich")
|
||||
msg, ok := s.beginRestoreOp("offbox-reconstitute", "immich")
|
||||
done()
|
||||
if ok || !strings.Contains(msg, "második meghajtóra") {
|
||||
t.Fatalf("the begin must refuse with the copy named; ok=%v %q", ok, msg)
|
||||
}
|
||||
if s.backupMgr.RestoreStatus().Running {
|
||||
t.Fatal("a refused begin recorded a restore")
|
||||
}
|
||||
if _, ok := s.beginRestoreOp("offbox-reconstitute", "immich"); !ok {
|
||||
t.Fatal("after the copy the begin must work")
|
||||
}
|
||||
}
|
||||
@@ -209,28 +209,50 @@ func (s *Server) restoreOpBlocked() (string, bool) {
|
||||
if s.backupMgr == nil {
|
||||
return "", false
|
||||
}
|
||||
if st := s.backupMgr.RestoreStatus(); restoreOpInFlight(st) {
|
||||
busy, app := s.backupMgr.RestoreBusy()
|
||||
return s.restoreBusyMessage(busy, app)
|
||||
}
|
||||
|
||||
// beginRestoreOp records a restore op in the SAME step as the check for every other operation (R-1256 H6): the early
|
||||
// restoreOpBlocked answer can be stale by the time a handler has read its form, and a second-drive copy that began in
|
||||
// between ran beside the restore. A refusal returns the same sentence as restoreOpBlocked; the caller redirects with it
|
||||
// and starts nothing. Every restore handler begins through here — TestR1256_H6_NoHandlerBeginsARestoreUnchecked.
|
||||
func (s *Server) beginRestoreOp(op, stack string) (string, bool) {
|
||||
busy, app := s.backupMgr.TryBeginRestoreOp(op, stack)
|
||||
if busy == "" {
|
||||
return "", true
|
||||
}
|
||||
if s.logger != nil {
|
||||
s.logger.Printf("[WARN] [restore] %s of %s refused at its start — a %s operation runs (R-1256)", op, stack, busy)
|
||||
}
|
||||
msg, _ := s.restoreBusyMessage(busy, app)
|
||||
return msg, false
|
||||
}
|
||||
|
||||
// restoreBusyMessage is the household's sentence for a RestoreBusy reason ("" = not busy).
|
||||
func (s *Server) restoreBusyMessage(busy, app string) (string, bool) {
|
||||
switch busy {
|
||||
case "":
|
||||
return "", false
|
||||
case backup.RestoreBusyRestore:
|
||||
subject := "Egy visszaállítási művelet"
|
||||
if st.Stack != "" {
|
||||
subject = "Egy visszaállítási művelet (" + st.Stack + ")"
|
||||
if app != "" {
|
||||
subject = "Egy visszaállítási művelet (" + app + ")"
|
||||
}
|
||||
return subject + " már fut, ezért most nem indítható újabb. Az állapotát ezen az oldalon " +
|
||||
"követheted; amint befejeződik, újra indíthatsz visszaállítást.", true
|
||||
}
|
||||
if s.backupMgr.IsRunning() {
|
||||
case backup.RestoreBusyBackup:
|
||||
return "Egy mentési művelet már fut, ezért most nem indítható visszaállítás. Az állapotát " +
|
||||
"ezen az oldalon követheted; amint befejeződik, újra indíthatsz visszaállítást.", true
|
||||
}
|
||||
// The stray-copy fix: a second-drive copy (rsync --delete) runs now, or one a previous controller process lost may
|
||||
// still run — a restore beside it could lose the files it brings back (R-1180's other direction).
|
||||
if app, blocked := s.backupMgr.Tier2CopyBlocksRestore(); blocked {
|
||||
default:
|
||||
// The stray-copy fix: a second-drive copy (rsync --delete) runs now, or one a previous controller process lost
|
||||
// may still run — a restore beside it could lose the files it brings back (R-1180's other direction).
|
||||
if s.logger != nil {
|
||||
s.logger.Printf("[WARN] [restore] restore refused — a second-drive copy is running or may still run (app=%q)", app)
|
||||
}
|
||||
return "Éppen fut a második meghajtóra készülő mentési másolat, ezért most nem indítható visszaállítás. " +
|
||||
"Néhány perc múlva próbáld újra.", true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// backupsRestoreWizardHandler renders GET /backups/restore/app?name=<app> — the single entry the
|
||||
|
||||
@@ -136,6 +136,11 @@ type Server struct {
|
||||
agentCli *agentapi.Client
|
||||
agentCliErr error
|
||||
agentCliOnce sync.Once
|
||||
// D65 = A: the last operator reboot of this guest (operator_reboot_notice.go); the fetch and retry are test seams.
|
||||
opRebootMu sync.Mutex
|
||||
opRebootAt time.Time
|
||||
opRebootFetch func(ctx context.Context) (time.Time, error)
|
||||
opRebootRetry time.Duration
|
||||
|
||||
// initialCredsFn is the R-254 read seam for an app's generated first-login credential. nil → the
|
||||
// real live container read (stackMgr.ReadInitialCredentials). ONE definition, used by BOTH the
|
||||
@@ -844,6 +849,9 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
// D25 (2026-10-21): „Eltávolítás" of ONE kept second-drive copy whose source is gone.
|
||||
case path == "/backup/kept/remove" && r.Method == http.MethodPost:
|
||||
s.keptCopyRemoveHandler(w, r)
|
||||
// D62 (2026-10-29): „Törlés most" of ONE restore copy (typed name; never a copy a hold or a put-back needs).
|
||||
case path == "/backup/restore-copy/delete" && r.Method == http.MethodPost:
|
||||
s.restoreCopyDeleteHandler(w, r)
|
||||
// D47 (2026-10-25): „Visszaállítás" from a set-aside recovery unit of an earlier install.
|
||||
case path == "/backup/kept/restore" && r.Method == http.MethodPost:
|
||||
s.keptUnitRestoreHandler(w, r)
|
||||
|
||||
@@ -429,9 +429,9 @@ function submitRestore() {
|
||||
{{/* D25 + D47: the kept copies (second-drive copies whose source is gone; set-aside units of an earlier install). Shown
|
||||
inside the off-site card when the box has off-site, and after it when it has none — before 2026-10-25 it sat only inside
|
||||
the off-site block, so a box without off-site never listed its kept copies. One body, two call sites. */}}
|
||||
{{define "kept_copies_section"}}{{if .KeptCopies}}
|
||||
{{define "kept_copies_section"}}{{if or .KeptCopies .RestoreCopies}}
|
||||
<div class="backup-tier-divider" style="margin-top:1.5rem"></div>
|
||||
<h3 style="margin-bottom:.5rem">{{T "backups_restore.megorzott_masolatok"}}</h3>
|
||||
<h3 style="margin-bottom:.5rem">{{T "backups_restore.megorzott_masolatok"}}</h3>{{if .KeptCopies}}
|
||||
<p class="form-hint" style="margin-bottom:.75rem">{{T "backups_restore.megorzott_masolatok_hint"}}</p>
|
||||
<div class="app-row-list">
|
||||
{{range .KeptCopies}}
|
||||
@@ -467,4 +467,43 @@ the off-site block, so a box without off-site never listed its kept copies. One
|
||||
});
|
||||
}
|
||||
</script>
|
||||
{{end}}{{end}}
|
||||
{{end}}{{if .RestoreCopies}}{{/* D62 (operator ruling 2026-10-29): restore copies — the app, the restore's date, the size, the day the box deletes
|
||||
it; „Törlés most" after the warning and the app's name typed. A needed copy: the button is off, the reason in one line. */}}
|
||||
<h4 style="margin:1rem 0 .5rem">{{T "restore_copy.title"}}</h4>
|
||||
<p class="form-hint" style="margin-bottom:.75rem">{{T "restore_copy.hint"}}</p>
|
||||
<div class="app-row-list" data-restore-copies="true">
|
||||
{{range .RestoreCopies}}
|
||||
{{template "app_list_row" dict "Slug" .App "Name" .Name "Secondary" .Line}}{{if .NeededText}}
|
||||
<button type="button" class="btn btn-xs btn-danger-outline" disabled data-restore-copy-off="true">{{T "restore_copy.delete"}}</button>{{end}}
|
||||
{{template "app_list_row_end"}}{{if .NeededText}}
|
||||
<p class="form-hint" data-restore-copy-needed="true">{{.NeededText}}</p>{{else}}
|
||||
<details class="kept-delete" data-restore-copy="{{.App}}/{{.Stamp}}">
|
||||
<summary class="btn btn-xs btn-danger-outline">{{T "restore_copy.delete"}}</summary>
|
||||
<form method="POST" action="/backup/restore-copy/delete" class="kept-delete-form" onsubmit="return deleteRestoreCopy(event, this)">
|
||||
{{$.CSRFField}}
|
||||
<input type="hidden" name="app" value="{{.App}}">
|
||||
<input type="hidden" name="stamp" value="{{.Stamp}}">
|
||||
<p class="kept-delete-text">{{.Warning}}</p>
|
||||
<label class="form-hint">{{.TypePrompt}}</label>
|
||||
<input type="text" name="confirm" autocomplete="off" required>
|
||||
<button type="submit" class="btn btn-sm btn-danger">{{T "restore_copy.delete"}}</button>
|
||||
<p class="kept-delete-text" data-restore-copy-error="true" hidden></p>
|
||||
</form>
|
||||
</details>{{end}}
|
||||
{{end}}
|
||||
</div>
|
||||
<script>
|
||||
function deleteRestoreCopy(ev, f){
|
||||
ev.preventDefault();
|
||||
var err = f.querySelector('[data-restore-copy-error]');
|
||||
fetch(f.action, {method: 'POST', headers: Object.assign({'Accept': 'application/json'}, csrfHeaders()), body: new URLSearchParams(new FormData(f))})
|
||||
.then(function(r){ return r.json(); })
|
||||
.then(function(d){
|
||||
if (d.ok) { location.href = d.redirect; return; }
|
||||
err.textContent = d.error || ''; err.hidden = false;
|
||||
})
|
||||
.catch(function(e){ err.textContent = '{{T "layout.halozati_hiba"}} ' + e; err.hidden = false; });
|
||||
return false;
|
||||
}
|
||||
</script>
|
||||
{{end}}{{end}}{{end}}
|
||||
Loaded 100 of 104 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user