Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 admin
|
f3204024e2 |
No files matched your search
+12
-171
@@ -1,180 +1,21 @@
|
||||
## Unreleased (2026-10-28 night) — four restore edges (felhom.eu R-1256, review round 2 HUNT-1 H4/H5/H6/H8); commit only, rides the next release
|
||||
## v0.345.1 (2026-10-26) — SECURITY: cloudflared 2026.10.3 (Cloudflare's eMudhra notice) — built from v0.345.0 + this one commit (R-1300)
|
||||
|
||||
**MinAgent: 0.158.1** (unchanged). Each edge: a test seen failing first, then the fix; live proof on 9202 in the felhom.eu
|
||||
night report (`audits/night-2026-10-28/`). **Live 2026-10-29 on 9202:** H5's busy volume (not written into, the live data
|
||||
put back, no hold) and H8 (the 03:29 sweep removed the two old copies, kept the young one). H4, H5's failed stop and H6:
|
||||
tests only — the attended live proof (or a revert of those two branches) is owed.
|
||||
|
||||
- **H4 — `internal/backup/offbox_reconstitute.go`:** an off-site restore whose definition write (or database-password
|
||||
write) fails no longer starts the app at once. The writer writes the compose BEFORE the app's settings, so the stack
|
||||
dir could hold the snapshot's (older) compose: the live definition is written back first, then the app starts
|
||||
(`restartAtLiveDefinition`); when the write-back fails too, the app is held (`restore_mixed`), never started on the
|
||||
snapshot's definition. Tests `TestR1256_H4_*`.
|
||||
- **H5 — a put-back whose stop fails does not go on.** `putBackAfterFailedRestore` (off-site and local unit restore):
|
||||
a failed stop pours nothing, keeps the copy and holds the app; a failed definition write-back keeps the copy too (D20:
|
||||
a put-back that fails keeps its copy). `CrashPutBack.PutBackInterruptedRestore` (a restore cut by a power loss): a
|
||||
failed stop is an error, so the start holds the app (R-1251 H2) instead of pouring under a running app.
|
||||
`restoreDockerVolumesFrom`: a `docker volume rm -f` that fails (the volume is in use) counts as a failed volume — the
|
||||
following `volume create` was a no-op and the tar was unpacked ON TOP of the old content (a merge). Tests
|
||||
`TestR1256_H5_*`. One log line corrected: a failed start after a put-back no longer says the copy stays (it is removed;
|
||||
the volumes are back).
|
||||
- **H6 — a restore begins in the same step as the check for every other operation.** New
|
||||
`backup.TryBeginRestoreOp` / `RestoreBusy` (under `m.mu`, the lock the second-drive mirror's `beginTier2Run` asks
|
||||
under): a restore op in flight, the backup run, a second-drive copy. Every web restore handler begins through
|
||||
`Server.beginRestoreOp` (a source test refuses a direct `BeginRestoreOp`); the kept loads (`LoadKeptApp`,
|
||||
`LoadKeptOffsite`) return `ErrRestoreBusy` with nothing started, and `LoadKeptApp` takes the dated folder's file move
|
||||
as `prepare`, inside the operation. The install-time „use my kept data" asks about the second-drive copy too
|
||||
(`RestoreBusy`, was RestoreStatus/IsRunning only). Tests `TestR1256_H6_*` (backup, web, api).
|
||||
- **R-1254 H2, the controller half (felhom.eu R-1254; agent `5ddd124`):** the escrow claim and confirm reach the agent
|
||||
with their own 60 s timeout (`agentapi.escrowCallTimeout`; was the client's 15 s) — the agent may spend its 30 s
|
||||
upload plus 15 s asking the hub before it answers, and a claim abandoned at 15 s could still store a package whose
|
||||
code reached nobody. The confirm's household sentence: 502 (the hub holds another package) keeps „…a régi kódod
|
||||
továbbra is érvényes…"; 504 (the agent cannot learn the outcome) or a failed call (0) now reads
|
||||
`api.escrow.confirm_store_unknown` („Nem tudjuk biztosan… Őrizd meg mindkét kódot…", hu + en) and never promises the
|
||||
previous code. `TestR1151_ConfirmAnswersInTheBoxLanguage`'s status-0 case changes with it, on purpose. Tests
|
||||
`TestR1254_*` (agentapi, web), red before the fix.
|
||||
- **R-528 — the OOM line sees a kill Docker does not report (felhom.eu R-528; agent `GET /guest/oom-kills`).** The
|
||||
30-s dead-app scan (`stacks.ScanOOMKilled`) now also asks the host's agent once per scan for the kernel's per-container
|
||||
`oom_kill` counter (`agentapi.GuestOOMKills`, wired in `cmd/controller/hostoom_wiring.go`, 5 s cap). A running
|
||||
container whose current run has a count above 0 is an OOM kill whatever Docker's `OOMKilled` says: the existing
|
||||
`app_oom` line (once per run, R-514/R-636) fires, its details carry `source` (`docker`, `cgroup` or `docker,cgroup`),
|
||||
and the host's count is the run's kill count. An agent without the route (404) is asked again only after 10 minutes
|
||||
and said once; no agent (9202) = Docker's flag only, as before. The inspect now reads `{{.Id}}` too. Tests
|
||||
`TestR528_*` (stacks, cmd/controller), red before the fix (the host-counted kill was not seen) or red-proved by
|
||||
mutation (the back-off, the wiring).
|
||||
- **Found walking the capability map's B2 on 9202 (small, fixed in the session):** a Stop/Start of a name that is no
|
||||
app on the box (the bootstrap-managed `felhom-controller`) answered 500 „…az alkalmazás beállításai nem menthetők." —
|
||||
the desired-state write's „not found" read as a save failure. `stacks.SetDesiredState` returns the
|
||||
`ErrStackNotFound` kind with the household's sentence (`err.stacks.alkalmazas_nem_talalhato`); `actionStack` answers
|
||||
404 before recording any intent. Nothing was ever stopped. Test `TestAnActionOnANameThatIsNoAppIsA404` (red first).
|
||||
- **H8 — `internal/backup/restore_copy_sweep.go`:** an unused restore copy goes after 14 days. Every 6 hours
|
||||
(`restore-copy-sweep`) a copy is removed only when the app has no restore hold of any kind, the app-stop marker is
|
||||
readable and does not name the app, no restore runs, and the copy is older than 14 days. Tests `TestR1256_H8_*`
|
||||
(and the wiring test).
|
||||
|
||||
## v0.346.1 (2026-10-27) — SECURITY: traefik v3.7.16, the file manager 1.5.8-stable, cloudflared 2026.10.4 — built from v0.346.0 (`b1b9be3`) + one commit (felhom.eu R-1304, R-1305)
|
||||
|
||||
**MinAgent: 0.158.1** (unchanged). Box programs only; no other change from v0.346.0.
|
||||
|
||||
- `internal/infra`: `TraefikImage` `traefik:v3.7.13` → `traefik:v3.7.16` (12 notices between; none reaches our config —
|
||||
docker + file providers, no FastProxy, no NTLM backend, no Kubernetes; the v3.7.14–16 migration notes touch only
|
||||
Kubernetes provider names, OTel histograms and a CONNECT request now answered 501). `FileBrowserImage`
|
||||
`gtstef/filebrowser:1.5.6-stable` → `1.5.8-stable` (GHSA-qx86-4v5r-26g5 TOTP re-enrolment, GHSA-3846-gh75-gp3m upload
|
||||
share, and the 1.5.8 fixes; 1.5.7-stable is skipped — it broke existing TOTP sign-ins). `CloudflaredImage`
|
||||
`2026.10.3` → `2026.10.4` (VULN-176088: the metrics server's debug endpoints; ours is not reachable from outside — low).
|
||||
- How a box takes them: the reconcile at start, as for 0.345.1 — traefik's compose changes → `up -d --force-recreate`
|
||||
(new test `TestEnsureTraefik_ANewImageRecreatesTraefik`, red-proved: it fails when a compose-only change is skipped);
|
||||
cloudflared `TestEnsureCloudflared_ANewImageRecreatesTheTunnel`; the file manager
|
||||
`TestReconcileFileBrowserImage_MovesOnlyTheImage` (mounts unchanged; it rejoins the app network only after the
|
||||
password check, R-1281).
|
||||
|
||||
## v0.346.0 (2026-10-27) — D53: the hub learns which apps are held; an unreadable app.yaml keeps its folder (R-1220); the ceremony keeps the off-site key (R-1218, R-1219); safe .fab export/import (R-1187); holds of uninstalled apps are not reported (R-1298); Go 1.27.3 (R-972); the line-test hold waits for the seeder to slow down (R-1250) — RELEASED on the operator's yes (attended release block)
|
||||
|
||||
**MinAgent: 0.158.1** (unchanged). Operator ruling D53 (2026-10-26, felhom.eu `09` §3 decision 265): operator-only
|
||||
reminders first. Needs hub with `app_held` / `app_hold_cleared` in `allowedEventTypes` (felhom.eu, same day) — an older
|
||||
hub answers 400 to both and nothing else changes.
|
||||
|
||||
- `internal/holdreport`: ONE observer of the hold store (`settings.RestoreHolds`), ticked by the scheduler every minute
|
||||
(`hold-report`). `app_held` for a new or changed hold (details `stack_name`, `reason` — the empty legacy reason is sent
|
||||
as `restore_failed` — `since`, `why`), `app_hold_cleared` when a hold is gone; every hold is sent again once a day and
|
||||
at each start (`repeat: true`). The last-reported set is kept in `settings.reported_holds`, so a hold cleared while the
|
||||
controller was down is still reported cleared. Both severity info (the hub mails nothing on receipt).
|
||||
- **R-972: Go 1.27.3** (`go.mod` + the builder image `golang:1.27.3-*` pinned by digest; released 2026-10-19 with a
|
||||
security fix to `crypto/x509`). `govulncheck` v1.8.0: 1 called vulnerability before (GO-2026-6809), 0 after; full
|
||||
`go build/vet/test` green. `golang.org/x/crypto` v0.58.0 is already the newest.
|
||||
- **R-1220 (P2, delete path):** an app whose `app.yaml` EXISTS but cannot be read counts as INSTALLED (fail closed):
|
||||
`stacks.liveDriveBinds(drives)` adds every folder its compose binds on every given drive, so „Megőrzött adatok" never
|
||||
lists — and never offers to delete — that app's live folder; one WARN names the app. Before: a parse error gave a nil
|
||||
`AppConfig`, the app read as not installed, its folder was a „leftover". Test `TestR1220_AnUnreadableAppYamlKeepsTheFolderLive`
|
||||
(with a real leftover as the control), red-proved against the old code.
|
||||
- **R-1218 (P2, key code):** the recovery ceremony stages the off-site repository password whenever the password FILE is
|
||||
on the box (`backup.Manager.OffboxRepoPasswordOnBox`), not only when off-site is configured — a cleared or switched-off
|
||||
target keeps the file, and a ceremony that sealed no password let the hub demote the household's good package (the
|
||||
hub half refuses that demotion: felhom.eu hub, same night). Test `TestR1218_TheCeremonyStagesWhenThePasswordFileExists`
|
||||
(off, cleared, and a box with no password as the control), red-proved („preflight,start" — no stage).
|
||||
- **R-1219 (P2, key code):** during a recovery offer the top bar no longer urges the ceremony, and `POST
|
||||
/api/escrow/start` answers 409 with `api.escrow.recovery_offer_open` (hu + en) before anything is staged or started.
|
||||
Tests `TestR1219_*` (the bar with a real session, and the same box without an offer as the control), red-proved (bar
|
||||
shown, start 200).
|
||||
- **R-1187 (P2, `.fab` export/import) — (a), (b), (d), (f), (g):** (a) the export carries a symlink AS A LINK (never its
|
||||
target's bytes) and stops with `err.appexport.kulonleges_fajl` naming the path on a FIFO, socket or device (a FIFO used
|
||||
to hang it); a drive folder that cannot be packed now FAILS the export (it was left out with a WARN while the export
|
||||
ended „done"). (b) the import keeps owners (`Lchown`) and modes and restores a link that stays inside the app's folder;
|
||||
a link pointing outside is refused and listed in a WARN. (d) `StartImport` refuses (`err.appexport.import_foglalt`)
|
||||
while another operation holds the app-stop marker (`BusyWith`). (f) a bundle whose `app.yaml` carries an empty,
|
||||
relative or unclean `HDD_PATH` is refused before anything is touched (`err.appexport.hdd_path_hibas`). (g) the import
|
||||
classifies databases with the backup side's predicate (`appbackup.DBTypeForImage`: postgis, pgvector, timescaledb) and
|
||||
runs `psql -v ON_ERROR_STOP=1`. Tests `internal/appexport/r1187_abdfg_test.go` (6), red-proved with the old behaviour
|
||||
behind the new function shapes (link packing error, a 10-s hang on the FIFO, link not restored, import not refused,
|
||||
bad paths accepted, postgis missed).
|
||||
**Follow-up (the automated security review, same night):** the import writes only through `os.Root`, so no entry is
|
||||
written through a link already in the folder that points outside; such an entry is refused and listed. Test
|
||||
`TestR1187b_NoWriteThroughALinkThatLeavesTheFolder`, red-proved (the file landed outside with plain `os.Create`).
|
||||
A second review finding: a link entry whose parent path passes through a link („a" → „.", then „a/b/l" → „../..")
|
||||
passed the path-text check while really pointing above the folder — now refused and listed
|
||||
(`TestR1187b_NoLinkThroughALinkedParent`, red-proved). psql takes `--set=ON_ERROR_STOP=1` (the `-v` short form tripped
|
||||
the docker `-v` gate); the four new message keys are listed in `scripts/i18n_go_keys.json` as born keys.
|
||||
A third review finding: a link TARGET through a link already in the folder („x" → „evil/y", evil pointing out). The rule
|
||||
is now strict and simple: a restored link points only DOWNWARD — relative, no „..", no link on its path or in its
|
||||
target; everything else is refused and listed (`safeDownwardLink`; `TestR1187b_NoLinkTargetThroughALink`, red-proved).
|
||||
**Found live on 9202 the same night (grimmory), fixed:** the import ran `compose` without the controller's own path
|
||||
variables (`USERDATA_PATH`, `IMPORT_PATH` — `stacks.policyEnv` adds them at every install and start); compose printed
|
||||
„The USERDATA_PATH variable is not set" and the DB step read that WARNING as the container id, waited 60 s and left the
|
||||
app removed and not started. Now `withImportPathVars` gives the import's compose runs the same derivation (never a
|
||||
bundle's value) and `containerIDFrom` takes only a line that IS an id. Tests `TestR1187_TheContainerIDIsNeverAComposeWarning`,
|
||||
`TestR1187_TheImportEnvCarriesTheControllersPathVars`, red-proved.
|
||||
**And the second cause, found by the same live run:** the import waited with `mysqladmin` and loaded with `mysql` —
|
||||
names the MariaDB 11+ images do not ship (only `mariadb-admin` / `mariadb`; checked in `mariadb:11.4`) — and read the
|
||||
password from two fixed env names, so EVERY `.fab` import of a MariaDB 11+ app timed out and left the app down. The
|
||||
import now loads through the backup side: `appbackup.DiscoverDatabases` (the live container's own credentials) and
|
||||
`appbackup.ImportDump` (mariadb / mariadb-admin; psql `ON_ERROR_STOP` + `--single-transaction`); the import's own
|
||||
`waitForDB` / `importDBDump` are removed. `appbackup.DBTypeForImage` stays the classifier (g). Test
|
||||
`TestR1187_TheImportLoadsTheDumpThroughTheBackupSide` (seams, no Docker), red-proved with the old body.
|
||||
- **R-1298:** the hold report skips an install's hold (`unhealthy_stop`, `update_failed`, `putback_pending`) of an app that
|
||||
is not installed (`holdreport.Reporter.SetInstalled`: the app's `app.yaml` exists — an unreadable one still counts,
|
||||
R-1220) and reports it cleared if it was reported; a restore hold is always reported (R-379). 9202's four orphan holds
|
||||
from a hand removal would otherwise have mailed weekly. Test `TestR1298_OnlyHoldsOfInstalledAppsAreReported`,
|
||||
red-proved; the wiring test pins the `SetInstalled` line.
|
||||
- **R-1250 (the line test for a box that seeds all day):** after the hold the controller now waits until every held app
|
||||
REPORTS its own upload near zero (qBittorrent `transfer/info` `up_info_speed`, Transmission `session-stats`
|
||||
`uploadSpeed`; ≤ 16 KiB/s on two readings 2 s apart, at most 90 s; an unreadable speed waits 20 s) and only then asks
|
||||
the agent (`upwatch.waitDrained`). Measured on demo-hp 2026-10-27: qBittorrent 5.2.4 kept sending at full speed for
|
||||
~14 s after its limit fell to 1 KiB/s, while the agent reads the uplink 5–15 s after the call — four tries in an hour,
|
||||
each refused „not quiet" (4.0–4.2 Mbit/s). New `Holdable.UploadRate`. Tests `TestR1250_*` (five), red-proved by four
|
||||
mutations (no wait; only the first app read; no fallback wait; no maximum → hangs).
|
||||
- `settings.RestoreHold.Why()` / `WireReason()`: the hold sentence moved out of `cmd/controller` (`holdWhy` calls it), so
|
||||
`--restore-holds` and the hub's list say the same thing.
|
||||
- Tests `internal/holdreport/holdreport_test.go` (8): the report field, the daily repeat, cleared once, cleared while
|
||||
down, a changed reason, the wire reason, the wiring in main, the record across a reload. Red-proved with an inert
|
||||
`Tick` (all six behaviour tests failed) and with the scheduler line commented out (the wiring test failed).
|
||||
|
||||
- **R-1303 (tooling, no runtime change): every pin names where its security notices are published.** `internal/infra/
|
||||
upstream.txt` beside `infra.go` (one line per `*Image` const), and a `# upstream:` comment directly above every
|
||||
`FROM` (`Dockerfile`, `infra-images/samba/Dockerfile`) and `ARG RESTIC_VERSION`. Gate `upstream` (`--fast`, SHARED:
|
||||
felhom.eu `scripts/security_watch_inventory.py --gate=controller`) refuses a pin with no line; decoys in
|
||||
`scripts/test_gate_decoys.py` (a new const with no line, the line only in a comment, the comment one line too high,
|
||||
a new FROM with none), red-proved. The nightly security watch (felhom.eu `scripts/security_watch.py`) reads them.
|
||||
|
||||
## v0.345.1 (2026-10-26) — SECURITY: cloudflared 2026.10.3 (Cloudflare's eMudhra notice) — built from v0.345.0 + one commit (R-1300)
|
||||
|
||||
**MinAgent: 0.158.1** (unchanged). **This entry on `main` records the same change as the tag `v0.345.1`** (the v0.345.0
|
||||
commit `6bfb784` + one commit); the image is built from the TAG, so none of the „Unreleased" work above is in it — the
|
||||
next release from `main` carries the bump by itself.
|
||||
**MinAgent: 0.158.1** (unchanged). Nothing else from `main` is in this build: it is the v0.345.0 commit (`6bfb784`) plus
|
||||
this commit only, tagged `v0.345.1`; the same change is on `main`.
|
||||
|
||||
- **Security:** `infra.CloudflaredImage` `cloudflare/cloudflared:2026.9.3` → `2026.10.3` (Docker Hub index digest
|
||||
`sha256:faa8e0a1475f137c910be6025f43c8e4092fdd35461b31e19fd1390657a1ff39`). Cloudflare's notice of 2026-10-20:
|
||||
cloudflared before 2026.10.2 trusts eight eMudhra (emSign) intermediate CAs whose private keys were weak and are
|
||||
recoverable — anyone holding them could pose as any website to cloudflared. The fix is the image's trust store.
|
||||
- After the controller update `ensureCloudflared` sees the compose differ (the `image:` line) and runs `compose up -d`,
|
||||
which recreates cloudflared on the new image. Test `TestEnsureCloudflared_ANewImageRecreatesTheTunnel`, red-proved.
|
||||
- Origin check (read only): every tunnel routes `*.<domain>` → `https://traefik` with `noTLSVerify: true` (R-510) and
|
||||
traefik answers with its self-signed default certificate — no eMudhra chain. felhom.eu
|
||||
recoverable — anyone holding them could pose as any website to cloudflared. The fix is the image's trust store
|
||||
(2026.10.2 rebuilt on a new distroless base; 2026.10.3 a newer base and Go).
|
||||
- How a box takes it: after the controller update, `EnsureBaseStack` → `ensureCloudflared` sees the rendered compose
|
||||
differ (the `image:` line) and runs `compose up -d`, which recreates cloudflared on the new image; the tunnel
|
||||
reconnects in seconds. New test `TestEnsureCloudflared_ANewImageRecreatesTheTunnel` (an image-only change recreates;
|
||||
the same render again does not), red-proved with the change check removed.
|
||||
- Origin check (read only, before the update): every tunnel's remote route is `*.<domain>` → `https://traefik` with
|
||||
`noTLSVerify: true` (the day-0 recipe, R-510); traefik answers with its self-signed `TRAEFIK DEFAULT CERT` — no
|
||||
eMudhra chain, so the new trust store refuses nothing on the origin leg. felhom.eu
|
||||
`documentation/audits/cloudflared-2026.10.3-2026-10-26/`.
|
||||
- **`build.sh` (main only — the operator's yes, 2026-10-26):** a `--push` may also build from the tag `v<VERSION>` AS
|
||||
THE SERVER HAS IT when it points at HEAD — a patch release off a released commit while `main` holds unreleased work.
|
||||
A tag only on the clone, a tag on another commit, a tag of another version, or a server that cannot be asked is a
|
||||
refusal; clean-tree stays required. Four new cases in `scripts/test_build_push_once.sh` (20/20), red-proved both
|
||||
ways (the tag branch accepting anything: 4 fail; removed: 1 fails).
|
||||
|
||||
## v0.345.0 (2026-10-26) — the 50 % rule owns qBittorrent's NORMAL limit only (R-1292) — attended release run (afternoon)
|
||||
|
||||
|
||||
-11
@@ -1,16 +1,5 @@
|
||||
# CONTEXT.md — Project Memory
|
||||
|
||||
> **2026-10-28 night (unattended) — R-1256, R-1254 (controller half), R-528 and the unknown-app 404 on `main`, NOT
|
||||
> released (`342a394`, `e33cffa`, `eed23e7`, `20c5877`; MinAgent unchanged).** Restore ops begin through
|
||||
> `Manager.TryBeginRestoreOp` / `web.beginRestoreOp` only (`TestR1256_H6_NoHandlerBeginsARestoreUnchecked` refuses a direct
|
||||
> `BeginRestoreOp` in a handler). **Decided by CC unattended — operator may reverse:** an unused restore copy goes after
|
||||
> 14 days (`restoreCopyUnusedAge`; felhom.eu `07` §6.3). H4 and H5's failed stop: tests only, live proof owed. H5 (busy
|
||||
> volume) proven live on 9202, R-528 on Tester 1, with the local test tag `0.347.0-rc.n1028` (never pushed).
|
||||
|
||||
> **2026-10-26 (evening) — D53 ruled (felhom.eu `09` §3 decision 265): held apps reach the OPERATOR first.** Next here:
|
||||
> the controller reports its holds (stack, reason, since) to the hub as `app_held` / `app_hold_cleared` events plus a
|
||||
> daily re-send (felhom.eu `08` §10). Commit only; the release rides a later attended run.
|
||||
|
||||
> **2026-10-25 (Sunday, unattended) — D47, D48, R-1270, R-1271 on `main`, NOT released (next v0.343.0, MinAgent 0.158.1
|
||||
> unchanged).** D47 (`02daebe`): `stacks.AppConfig.InstallID` (minted at install/import, taken from the unit by a
|
||||
> restore, `EnsureInstallID` for older apps); `backup/unit_owner.go` `guardUnitWrite`/`guardUnitDir` in every unit writer
|
||||
|
||||
@@ -1,19 +0,0 @@
|
||||
# REPORT — controller v0.346.1: traefik v3.7.16, the file manager 1.5.8-stable, cloudflared 2026.10.4 (security), 2026-10-27 (attended)
|
||||
|
||||
> A sibling of `REPORT.md`, not an overwrite: the release session of the same day (v0.346.0) owns `REPORT.md`.
|
||||
|
||||
## In plain words
|
||||
|
||||
- **What:** three box programs had public security notices: traefik (12 notices, none reaching our config), the file
|
||||
manager (a TOTP re-enrolment and an upload-share flaw; 1.5.7 is skipped, it broke TOTP sign-ins) and cloudflared
|
||||
(debug endpoints on its metrics server, not reachable from outside). Controller **0.346.1** = the released 0.346.0
|
||||
(`b1b9be3`) + only the three new pins.
|
||||
- **How a box takes them:** the reconcile at start recreates each container on its new image (traefik pinned by the new
|
||||
test `TestEnsureTraefik_ANewImageRecreatesTraefik`, red-proved).
|
||||
|
||||
## Gates
|
||||
|
||||
`controller_gates.py` (full) green; `go test ./...` green (41 packages); `go build`, `go vet` green.
|
||||
|
||||
Evidence and the box proofs: felhom.eu `documentation/audits/security-app-updates-2026-10-27/part5/`. Register:
|
||||
felhom.eu R-1304 (traefik), R-1305 (file manager).
|
||||
@@ -1,40 +0,0 @@
|
||||
# REPORT — controller v0.345.1: cloudflared 2026.10.3 (security), 2026-10-26 evening (attended)
|
||||
|
||||
> A sibling of `REPORT.md`, not an overwrite: the night session (D53, R-1218/1219/1220) owns `REPORT.md` tonight and
|
||||
> writes into this repo at the same time.
|
||||
|
||||
## In plain words
|
||||
|
||||
- **What:** Cloudflare said on 2026-10-20 that cloudflared before 2026.10.2 trusts eight bad certificate issuers
|
||||
(eMudhra, weak keys). Every Felhom box ran cloudflared 2026.9.3 in its guest. Controller **0.345.1** = the released
|
||||
0.345.0 + only the new cloudflared pin `2026.10.3`. Nothing from tonight's `main` is in it.
|
||||
- **Did it work:** yes. Every box with a tunnel (Tester 1, demo-hp, demo-felhom) runs controller 0.345.1 and cloudflared
|
||||
2026.10.3, healthy; the hub says „running (connected)" for all three; all 29 public names still redirect http → https.
|
||||
Tester4 moved to 0.345.1 by the global floor (it has no tunnel). Tester 2 has no box.
|
||||
- **Gaps:** the tunnel was down about 2 s (Tester 1), 4 s (demo-felhom) and 11 s (demo-hp) while cloudflared restarted.
|
||||
|
||||
## What changed
|
||||
|
||||
| Where | What |
|
||||
|---|---|
|
||||
| tag `v0.345.1` = `f3204024` (on `6bfb784`) | `infra.CloudflaredImage` → `cloudflare/cloudflared:2026.10.3`; test `TestEnsureCloudflared_ANewImageRecreatesTheTunnel`; CHANGELOG entry |
|
||||
| `main` `a1c65b3d` | the same change, plus `build.sh`: a `--push` may build from the pushed tag `v<VERSION>` when it is HEAD (operator's yes, this session); 4 new cases in `scripts/test_build_push_once.sh` |
|
||||
|
||||
- Red-proofs: the new reconcile test failed with the „compose changed" check removed; the build cases failed both ways
|
||||
(tag branch accepting anything: 4 of 20 fail; tag branch removed: 1 of 20 fails).
|
||||
- Gates `--fast` green on both commits; `go test ./...` green on the tag commit (`internal/stacks` 450 s under the night
|
||||
session's load). CI by commit: `a1c65b3d` job 2392 success; `f3204024` job 2393 success (run_attempt 1 each).
|
||||
- Image `gitea.dooplex.hu/admin/felhom-controller:0.345.1` (and `:latest`) digest
|
||||
`sha256:7cfdf7e1f835628de9c7d536e0a60c06bad5bbe19e8133765cc0a6bb732dbff6`, built from `f3204024`.
|
||||
- cloudflared image index digest `sha256:faa8e0a1475f137c910be6025f43c8e4092fdd35461b31e19fd1390657a1ff39`.
|
||||
- How a box takes it: `controller/internal/stacks/infra.go` `ensureCloudflared` — the rendered compose differs (the
|
||||
`image:` line) → `compose up -d` recreates the container. Seen on all three boxes: `[infra] cloudflared compose
|
||||
changed … recreating cloudflared` 1–2 s after `[selfupdate] Post-update startup: update successful (0.345.0 → 0.345.1)`.
|
||||
|
||||
## For the night session
|
||||
|
||||
`main` carries the bump (`a1c65b3d`). Your next release from `main` keeps cloudflared 2026.10.3 — no action needed.
|
||||
Your CHANGELOG „Unreleased" block is unchanged; the v0.345.1 section sits between it and v0.345.0.
|
||||
|
||||
Evidence: felhom.eu `documentation/audits/cloudflared-2026.10.3-2026-10-26/`. Register: R-1300 (closed), R-1301
|
||||
(opened). Full session report: felhom.eu `REPORT-cloudflared-2026-10-26.md`.
|
||||
@@ -1,12 +0,0 @@
|
||||
# REPORT — the security watch's pin list (felhom.eu R-1303), 2026-10-26 night
|
||||
|
||||
> A sibling of `REPORT.md` (the night session owns it tonight). The full report: felhom.eu
|
||||
> `REPORT-security-watch-2026-10-26.md`.
|
||||
|
||||
- **What changed here:** every pin now names where its security notices are published, and a push gate (`upstream`,
|
||||
`--fast`) refuses a pin without one. The rule and its reader are shared: felhom.eu `scripts/security_watch_inventory.py`.
|
||||
Decoys for the gate were red-proved with a reader that reads comment lines.
|
||||
- **Why:** the nightly security watch (felhom.eu `scripts/security_watch.py`, CI 03:40 UTC) reads these lines. No
|
||||
runtime change: nothing on a box changes, nothing is released.
|
||||
- **First run:** see felhom.eu `documentation/audits/security-watch-2026-10-26/ADVISORIES-2026-10-26.md` and rows
|
||||
R-1304 (traefik), R-1305 (the file manager), R-1306 (27 apps).
|
||||
@@ -1,39 +1,30 @@
|
||||
# REPORT — restore edges, the escrow timeout, the host OOM count: on main, NOT released (night 2026-10-28 → 29, unattended)
|
||||
# REPORT — controller v0.344.0 released (2026-10-26, attended release run)
|
||||
|
||||
**Plain words first.** Four commits are on main, each with CI green and each new test seen failing first. Nothing is
|
||||
released: the night brief allowed no release. The controller release goes on the morning list. It needs no new MinAgent:
|
||||
with an older agent the OOM count answers 404 and the controller waits 10 minutes before it asks again.
|
||||
**Plain words first.** Every box runs 0.344.0. The global floor was set at 09:07:47 UTC (MinAgent 0.158.1) on the
|
||||
operator's yes, after the three test boxes moved through their per-box floors and the release-build checks passed. New
|
||||
installs start on it too: golden 0.344.0 was baked and vouched the same day, and a fresh install came up on 0.344.0
|
||||
with no update step. The file manager restarted once at the update, as the release notes say.
|
||||
|
||||
**Decided by CC unattended — operator may reverse:** an unused restore copy goes after 14 days (options: 14 days, for
|
||||
ever, 3 days; `07` §6.3 [DESIGN] 2026-10-28). H4 and H5's failed-stop branch keep only test proofs; their live proof is
|
||||
owed to an attended session (or revert those two branches of `342a394`).
|
||||
## Release
|
||||
|
||||
## Commits
|
||||
- Image `felhom-controller:0.344.0`, sha256 `9435165dab85b1200d94157ec49d8be065889139e1831a5476dc68f4b9474b8a`, built
|
||||
from `0a59044` (CI 2355 ✓). The rc `0.344.0-rc.4` (same commit) ran on 9202 first; 9202 is back on 0.343.0.
|
||||
- Per-box floors 0.344.0 for Tester 1, demo-hp, demo-felhom (08:15 UTC; each moved by the normal update path, healthy),
|
||||
then the global floor (read back from the hub DB with its WAL and from the hub log), then the per-box floors cleared.
|
||||
- Fleet after the floor: demo-felhom, demo-hp, tester-1 report 0.344.0; Tester-2 has no box since 2026-10-04 (R-862).
|
||||
|
||||
| Commit | CI (run id / #) | What |
|
||||
## Checks on the release build (felhom.eu `documentation/audits/release-monday-2026-10-26/`)
|
||||
|
||||
| Row | Check | Result |
|
||||
|---|---|---|
|
||||
| `342a394` | 2469 / #532 green | **R-1256** four restore edges. H4: an off-site restore whose definition write fails writes the LIVE definition back before the start; a failed write-back holds the app. H5: a put-back whose stop fails pours nothing, keeps the copy, holds the app; a failed definition write-back keeps the copy (D20); a `volume rm -f` that fails fails that volume (no merge on top of the old content). H6: a restore begins in ONE step with the check for a restore, the backup run and a second-drive copy (`TryBeginRestoreOp`); every restore button and both kept loads begin that way. H8: `SweepUnusedRestoreCopies`, every 6 h. |
|
||||
| `e33cffa` | 2472 / #533 green | **R-1254** controller half: escrow claim/confirm wait 60 s (the agent's own upload wait is longer than the old 15 s); an unknown outcome is 504 `api.escrow.confirm_store_unknown` („keep both codes"), hu + en. |
|
||||
| `eed23e7` | 2474 / #534 green | **R-528**: the 30-s OOM scan also reads the host's per-container `oom_kill` (agent `GET /guest/oom-kills`); a counted kill raises `app_oom` whatever Docker's flag says; `source` in the details. |
|
||||
| `20c5877` | 2475 / #535 green | An action on a name that is no app answers 404, not „settings cannot be saved" (found walking the capability map). |
|
||||
| R-1281 | FileBrowser's networks; a forged header from apps; the household's files; OnlyOffice linked | on all three boxes FileBrowser only on `felhom-filebrowser` + `felhom-files`; qBittorrent/Sonarr/Radarr/Prowlarr → 000 (control: controller 404, traefik 301); Tester 1: dashboard → pass → cookie → listing 200, no cookie 401; OnlyOffice joined `felhom-files`, editor config 200, the OnlyOffice container fetched the document 200 — closed |
|
||||
| R-1282 | „Később", then a controller restart | the stored 03:30 kept (start-up line + settings.json) — closed |
|
||||
| R-1278 | install page with a kept backup | hint + link shown; none before or after — closed |
|
||||
| R-1286 | Kimai's install id; the hold sentences | id written 07:01:55Z; `holdWhy` released — closed |
|
||||
| R-1288 | released | proven live on the night's rig — closed |
|
||||
| R-1261 | released | the live line test is Tuesday (R-1250's dated check) — stays VERIFY |
|
||||
| D47 | upgrade count | 0 set aside on all three boxes; every app has an install id |
|
||||
|
||||
Red-proofs: felhom.eu `documentation/audits/night-2026-10-28/redproofs/README.md`.
|
||||
|
||||
## Live proof (test build `0.347.0-rc.n1028`, local tag only, never pushed)
|
||||
|
||||
- **H5, scratch 9202:** vikunja's first volume held by a created-only container, the household's restore pressed through
|
||||
the page: „Failed to remove volume vikunja_vikunja_data before the restore … (not written into)", the restore failed,
|
||||
the put-back poured the live copy of both volumes, the app ran, no hold; a project from before the restore point AND
|
||||
one from after it read back.
|
||||
- **H8, scratch 9202:** three copies planted by hand (2026-10-01, 2026-10-05, 2026-10-28); the first sweep at 03:29 UTC
|
||||
removed the two older than 14 days and kept the young one (8 ms). The young one was then removed by hand.
|
||||
- **R-528, Tester 1:** one `app_oom` „seen by docker,cgroup", one mail with `"source":"docker,cgroup"`.
|
||||
- **Not live:** H4 (off-site only; 9202 has no off-site target), H5's failed stop and crash put-back, H6 (the old
|
||||
code also refused at the early check, so a live press cannot tell the two apart). Tests only.
|
||||
|
||||
Both boxes put back after. **Scratch 9202** (2026-10-29 04:14 UTC): controller 0.346.1 healthy, the test image removed, the
|
||||
live catalog (config byte-identical to the saved copy), no throwaway app, no planted restore copy, no volume made
|
||||
tonight. **Tester 1** (2026-10-28 23:30 UTC): controller 0.346.1, agent 0.167.0 (sha256 `28bdd282…`), the test app
|
||||
removed. The bench is stopped. Hub: nothing provisioned.
|
||||
|
||||
Evidence: felhom.eu `documentation/audits/night-2026-10-28/` (`r1256-restore/`, `r528-oom/`).
|
||||
**Observation (NOT-A-FINDING):** on 9202's rc start the first FileBrowser round logged a WARN „could not join felhom-files
|
||||
(files.<domain> stays unreachable)" because FileBrowser was being re-created in the same second; it joined 2 s later.
|
||||
Log only, no event; the three release boxes logged no WARN.
|
||||
@@ -80,13 +80,10 @@
|
||||
| `restoreOpInFlight` + `hasRecentRestoreResult` | controller/internal/web/restore_wizard.go | `(backup.RestoreOpStatus) bool` / `(st, app, now) bool` | THE "is a restore running / did one just finish" display reads | **TRAP (v0.154.0 shipped this bug): `Manager` has TWO running flags.** `IsRunning()` reads the CONCURRENCY flag, acquired inside the goroutine — and `RestoreOffboxScratch` never acquires it, so it is false for the whole verification restore. Display must read `RestoreStatus().Running` (set synchronously by `BeginRestoreOp`). Read the status ONCE per render or the strip and the suppression can disagree. `hasRecentRestoreResult` is app-bound and window-bounded — a process-wide result must not light another app's „Eredmény" |
|
||||
| `restoreWizardPath` / `deriveWizardStep` / `resolveWizardApp` | controller/internal/web/restore_wizard.go | `(app) string` / `(restoreWizardInput) restoreWizardView` / `([]OffboxAppRow, name) *OffboxAppRow` | R-48 offsite restore wizard: URL builder + the PURE step/unlock derivation + the app-resolution refusals | The step is **never** taken from the request. Precedence is load-bearing: op-running outranks a stale `?full_prep=`, else a commit button reappears mid-restore. Truth table + red-proof: `restore_wizard_test.go`. Adding a form here that posts anywhere new breaks `TestRestoreWizard_NoNewMutationEndpoints` **by design** — R-48 adds no mutation surface |
|
||||
| `restoreOpBlocked` | controller/internal/web/restore_wizard.go | `() (msg string, blocked bool)` | THE refusal gate before starting ANY restore | **Use this, never a bare `IsRunning()`.** It reads BOTH flags: `RestoreStatus().Running` (set synchronously by `BeginRestoreOp`, true for the whole off-box restore) and `IsRunning()` (the concurrency flag, the only one the nightly backup holds). **R-351: all seven handlers read only `IsRunning()`, which the goroutine acquires AFTER the handler returns — a second press started a second run and was told „…elindult".** Returns the Hungarian refusal, which names the running app and a route | **R-360 (v0.226.0): `offboxVerifyCopyDeleteHandler` was the LAST holdout and its doc comment claimed it already did this — the sentence is why nobody looked. `IsRunning()` is FALSE for the whole of a verification restore, so the copy a restore was writing into could be deleted from the UI (observed live 2026-08-21 22:35). No app-name comparison: refusing during ANY restore is stronger and uniform.**
|
||||
| `Server.beginRestoreOp` → `backup.TryBeginRestoreOp` / `RestoreBusy` (R-1256 H6) | controller/internal/web/restore_wizard.go; controller/internal/backup/opstatus.go | `(op, stack) (msg string, ok bool)` / `(op, stack) (busy, mirrorApp string)` | THE way a restore op begins: the check for every other operation (a restore op, the backup run, a second-drive copy) and the record in ONE step under `m.mu` — the mirror asks under the same lock (`beginTier2Run`) | **Never call `BeginRestoreOp` from a handler** (`TestR1256_H6_NoHandlerBeginsARestoreUnchecked`): `restoreOpBlocked` asked early can be stale by the time the form is read, and a mirror that began in between ran beside the restore. `restoreOpBlocked` stays as the EARLY refusal; the kept loads begin inside `LoadKeptApp` / `LoadKeptOffsite` and return `ErrRestoreBusy` |
|
||||
| `Manager.RestoreFromRecoveryUnitAt` / `RestoreFromRecoveryUnit` + `UnitRestoreResult` (R-353 v0.226.0; R-102 v0.229.0) | controller/internal/backup/restore_unit.go | `(stack, unitDir string) (UnitRestoreResult, error)` / `(stack string) (…)` | THE local recovery-unit restore, and the facts its surface must state | **Returns a RESULT, not just an error** — volumes replayed, DBs replayed, and what the manifest LISTED. The Manifest* counts are load-bearing: zero-replayed has two causes (the backup held no data / the backup listed data that did not come back) and they are opposite news. Pair it with `unitRestoreOutcomeMsg`; do NOT write a new sentence. **A claim about the APP is forbidden on this path** — it has no `SafetyDump` discriminator, unlike the off-site twin (CONTEXT.md ruling, 07-backup-architecture §6.3). `restoreDockerVolumes` now returns `(int, error)`; `restoreDockerVolumesFrom` is unchanged and still the shared implementation. **R-102: `…At` holds the whole body and takes the unit DIRECTORY; the one-argument form is the thin caller naming the primary unit.** THE SOURCE MOVES; THE DESTINATION DOES NOT — `GetAppDrivePath` still resolves where the data lands. The R-47 mutation order, the unit-over-guest secret precedence, the fail-closed data-key gate and the no-unit `RestoreApp` fallback are all pinned and unchanged. The volume leg goes through the R-354 `volumeReplayFrom` seam so the source directory is assertable without Docker |
|
||||
| `Manager.putBackAfterFailedRestore` + `liveVolumeCopy` (R-893 option A, v0.336.0) | controller/internal/backup/offbox_restore_undo.go | `(stack, cause, live, copy, volumesMoved, filesPlaced, restart, holdKey) error` | THE restore undo — off-site (v0.336.0) and, since D22 (unreleased), the LOCAL unit restore (`RestoreFromRecoveryUnitAtWith`, so also the second drive's and a kept load): the live volumes it will replace, measured before the stop (`planLiveVolumeCopy`), copied right after it (`copyLiveVolumes`), poured back on any failure | One copy per restore under `<ns>/backups/restore-undo/<app>/` (no capture set reads `backups/`); the put-back uses the restore's own `volumeReplayFrom`/`restoreDockerVolumesFrom`, so there is one volume-replay implementation. A put-back that fails sets `keep` and HOLDS (option C) — never start on volumes that are not known to be back. Seams for tests without Docker: `undoVolumeProbeFn`, `undoVolumeTarFn`, `undoVolumeRemoveFn`; operator notice `SetRestorePutBackNotify` (leg `restore-put-back`). Pinned by `r893_putback_test.go` and `r1177_local_putback_test.go` |
|
||||
| `Manager.SweepUnusedRestoreCopies` (R-1256 H8) | controller/internal/backup/restore_copy_sweep.go | `(now) []string` | THE only remover of a restore copy nothing names any more (`restore-copy-sweep`, every 6 h) | Removes a copy only when ALL hold: older than 14 days, the app has no restore hold of ANY kind, the app-stop marker is readable and does not name the app, no restore op runs. It answers „no" when it cannot answer — a held or waiting copy is support's way back |
|
||||
| `CrashPutBack` + `Manager.persistUndoForCrash` + `AppStopGuard.SetUndo` (D22 / R-1195, v0.338.0) | controller/internal/backup/restore_crash_putback.go | `NewCrashPutBack(stop, writeDef, logger)`; `PutBackInterruptedRestore(stack, undoDir) error` | A restore cut by a power loss is put back at start: the copy (volumes + `definition/` + `undo.json`) is recorded in the app-stop marker once whole; `Recover` pours it back BEFORE the restart | Call `persistUndoForCrash` right after `copyLiveVolumes` in any new restore path. `Recover` runs before the backup manager exists, so the put-back is wired in main.go with `stackMgr.StopStack` and the stack adapter's definition writer. A failed put-back = a FAILED recovery (app not started, marker kept). The household reads `note.restore.interrupted_put_back` (`LoadRestoreRecordAfterRecovery`). Pinned by `TestR1195_*` |
|
||||
| `tier2CaptureSetFull(...) (legs, missing, warns, classified)` + `keptCopyRels` + `dropKeptCopiesUnder` (R-1252, unreleased) | controller/internal/backup/tier2_capture.go, tier2_kept.go | `classified` = the legs come from a VALID backup block | **The second-drive reconcile deletes only under a valid classification**; a rejected or absent block keeps every dest dir; kept copies (D25) are always in the keep set; a mirror deleted by Remove drops its kept records | A new reconcile caller must take `classified` and the kept rels too — never reconcile against the legacy legs. A rejected block is told by `Manager.BackupBlockRejected` (stacks; via the adapter's optional `BackupBlockRejected` method) and signalled once a day (`SetTier2SettingsUnreadableNotify`). Pinned by `r1252_rejected_block_keeps_copies_test.go` |
|
||||
| `holdreport.Reporter` + `settings.RestoreHold.Why` / `WireReason` (D53, unreleased) | controller/internal/holdreport/holdreport.go · controller/internal/settings/settings.go | `New(store, notifier.PushEvent, logger)`; `Tick(now)` from the scheduler | Tells the hub every hold in place (`app_held`), every hold gone (`app_hold_cleared`), each again daily | **One observer of the hold store — never a send inside a hold writer.** A new hold writer needs nothing here. The hold sentence is `RestoreHold.Why()` (one per reason) — never a second copy. Pinned by `internal/holdreport/holdreport_test.go` |
|
||||
| `AppStopGuard.ResumePutBack` / `ResumeWaitingPutBacks` + `HoldReasonPutBackPending` (R-1251, unreleased) | controller/internal/backup/appstop_putback_wait.go | `ResumePutBack(stack) bool` (true = nothing owed, the caller may start); `ResumeWaitingPutBacks() []string` (puts back, then starts through the gated starter) | A cut restore's copy on a drive that is not bound is WAITING: Recover writes the restore hold `putback_pending` (copy in `SafetyDump`) and starts nothing; the put-back runs when the drive binds, before any start | **Any new path that starts an app when its drive arrives calls `ResumePutBack` first** (the boot recreate and `restartStacks` do, in web via `Server.resumePutBack`). Every other start path is covered by the hold store it already honours — do not add a second check. A failed put-back (at start or on resume) writes the D20 hold (`restore_mixed`) and drops the app from the marker. Needs `SetHolds` + `SetUndoReachable` before `Recover` (main.go). Pinned by `r1251_putback_waits_test.go`, `TestR1251_DriveStartPathsPutBackBeforeTheyStart`, `TestR1251_MainWiresThePutBackWait` |
|
||||
| `Manager.RestoreTier2Unit` + `Tier2Coverage.CanRestoreUnit` / `Tier2CopyDate` (R-102/R-103, v0.229.0) | controller/internal/backup/tier2_restore.go | `(stack string) (UnitRestoreResult, error)` / `() bool` / `() (string, bool)` | THE full restore from the SECOND DRIVE's mirror, and the predicate that gates it | **Do NOT widen `CanRestore()`** — it answers only "can the additive file restore run?" and one predicate answering two questions is R-356. Fail-closed: `tier2UnitIsOpenable` requires a parseable manifest, because a directory is not a package. The single-writer flag is taken INSIDE `RestoreFromRecoveryUnitAt` — a second `acquireRunning()` here would refuse the restore it guards. `Tier2CopyDate` prefers the SUCCESS anchor over the attempt clock (R-101) and reports which it returned |
|
||||
| `unitRestoreOutcomeMsg` + its three message constants (R-353, v0.226.0) | controller/internal/web/handlers.go | `(app string, res backup.UnitRestoreResult) string` | THE customer sentence for a completed LOCAL restore | Twin of `reconstituteOutcomeMsg`; copy its SHAPE (clauses earned by having done the thing, no filesystem path, base names only), never its text. The constants are named because `r353_unit_outcome_test.go` asserts them verbatim — a silent edit is how an honest message drifts back into a comforting one, which is the documented history of the sentence it replaces |
|
||||
|
||||
@@ -7,8 +7,7 @@
|
||||
# --- Build stage ---
|
||||
# R-972 (v0.322.0): a supported, patched Go, pinned by version AND digest (no moving tag). Go 1.24 had no security fixes
|
||||
# left (go.dev/doc/devel/release#policy). The builder's Debian does not reach the runtime (a static CGO-free binary).
|
||||
# upstream: osv:Go/stdlib
|
||||
FROM golang:1.27.3-trixie@sha256:fbb7ae40b62ff8afe31662df570d39b4f5052ac32b90dae8e60d7657a20a74f4 AS builder
|
||||
FROM golang:1.27.2-trixie@sha256:47667934874913a449b96faa7ef4777fa285d9ec29126ee80ed61c45b7cccf0e AS builder
|
||||
|
||||
ARG TARGETOS=linux
|
||||
ARG TARGETARCH
|
||||
@@ -48,10 +47,8 @@ RUN CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} go build \
|
||||
# in third_party/restic/, fingerprint CF8F 18F2 8445 7597 3F79 D4E1 91A6 868B D3F7 A907 — no keyserver at build
|
||||
# time); the download matches the published sum; and it matches the sum pinned HERE, so even a re-signed release
|
||||
# cannot swap the binary. Pinned by TestR1105_ResticIsPinnedOutsideTheOS.
|
||||
# upstream: gap:Debian announces package fixes as DSAs on security-tracker.debian.org, which the watch does not read; BEHIND on the point release is the signal
|
||||
FROM debian:13.7-slim@sha256:a29215f6a35e51e22adffa17f89e9d2ef06214e64a2bad10d765c46aea49f11f AS restic
|
||||
ARG TARGETARCH
|
||||
# upstream: github:restic/restic
|
||||
ARG RESTIC_VERSION=0.19.1
|
||||
ARG RESTIC_SHA256_AMD64=f415415624dcc452f2a02b8c33641791a8c6d6d3b65bbb3543fcf9a25151585c
|
||||
ARG RESTIC_SHA256_ARM64=a5f64aaab53d51e311fa3829124c5b703f2d14cf187d8640b6be3b2b49376465
|
||||
@@ -77,7 +74,6 @@ RUN set -eu; \
|
||||
# --- Runtime stage ---
|
||||
# Debian 13 (R-1105, v0.323.0), pinned by point release AND digest. It waited on Debian 12 until the off-site code's
|
||||
# restic assumptions were re-proven on 0.18 (07-backup-architecture.md §6.8); restic itself no longer comes from here.
|
||||
# upstream: gap:Debian announces package fixes as DSAs on security-tracker.debian.org, which the watch does not read; BEHIND on the point release is the signal
|
||||
FROM debian:13.7-slim@sha256:a29215f6a35e51e22adffa17f89e9d2ef06214e64a2bad10d765c46aea49f11f
|
||||
|
||||
# Install runtime dependencies — each one is a program the controller binary runs (R-1105 re-checked every exec):
|
||||
|
||||
@@ -1034,13 +1034,6 @@ Each app can define rich metadata in `.felhom.yml`:
|
||||
`healthy`; the copy then goes after a further 3 min of the same (10 s samples, watched up to 30 min; a new restore
|
||||
of the app ends the watch). A restore whose app does not start reads „…még nem indult el…", keeps the copy and
|
||||
alarms the operator once (D49 = A: nothing is poured back by itself). `restore_verdict.go`. `offbox_restore_undo.go`.
|
||||
- **Restore edges (unreleased, R-1256, night 2026-10-28).** A put-back whose stop fails pours nothing, keeps the copy
|
||||
and holds the app; a failed definition write-back keeps the copy; a volume whose `docker volume rm -f` fails is a
|
||||
failed volume (never unpacked on top of the old content). An off-site restore whose definition write fails writes
|
||||
the LIVE definition back before the start, or holds the app. Every restore begins in one step with the check for a
|
||||
restore, the backup run and a second-drive copy (`TryBeginRestoreOp`, also both kept loads). A copy that nothing
|
||||
needs (no hold of any kind, not named by the app-stop marker, no restore running) goes after 14 days, checked every
|
||||
6 h (`restore_copy_sweep.go`; decided by CC unattended — operator may reverse).
|
||||
- **A restore cut by a power loss, on a drive that comes back late (unreleased, R-1251).** The copy of the live
|
||||
data lives on the app's drive. When it cannot be reached at start because the drive is not bound yet, the app is
|
||||
HELD (`putback_pending`; „…A visszaállítás előtti adatai egy meghajtón vannak, amely most nem érhető el…") and no
|
||||
@@ -1495,10 +1488,6 @@ The nightly backup has two phases that run sequentially. All paths are **per-dri
|
||||
|
||||
> **OOM visibility (v0.243.0, R-514).** The 30 s dead-app check also reads `State.OOMKilled` for running app containers (`Manager.ScanOOMKilled`); the dashboard shows „Memória elfogyott" and the hub gets `app_oom` once per container run. The controller does not restart the app.
|
||||
> **OOM storm (v0.265.0, R-636).** For a flagged container the scan also reads the kernel's own kill counter (`memory.events` `oom_kill`, plus `memory.max` / `memory.peak`) with one `docker exec … cat` — `OOMKilled` is sticky, so it cannot count. When the SAME container run's counter rises by **20 or more within 30 minutes**, the notifier sends **`app_oom_storm`** (severity `error`, operator-only, details `{app, container, kills, window_min, mem_limit, peak}`) — ONCE per container run. `app_oom` itself is unchanged. An unreadable counter (an image without `cat`) never escalates.
|
||||
> **Host OOM count (unreleased, R-528, night 2026-10-28).** Docker inside an LXC guest misses real OOM kills. The scan
|
||||
> also asks the agent for the host's per-container `oom_kill` (`GET /guest/oom-kills`, 5 s cap; a 404 from an older
|
||||
> agent → 10 minutes before the next ask); a count above zero raises `app_oom` whatever `OOMKilled` says, with `source`
|
||||
> (`docker`, `cgroup` or `docker,cgroup`) in the details. `cmd/controller/hostoom_wiring.go`, `internal/stacks/oom.go`.
|
||||
|
||||
> **Multi-tier whole-guest backup (v0.174.0, R-82 Slice B).** The agent can serve SEVERAL whole-guest
|
||||
> backup tiers with independent cadences — "local daily + PBS weekly" (agent >= v0.97.0,
|
||||
@@ -2294,10 +2283,6 @@ that folder is never a dead end, and an install never runs into it silently (R-6
|
||||
- **R-1286 (v0.344.0).** A held app says WHY: each hold reason (a failed restore, an unhealthy stop by the crash-loop
|
||||
guard, …) has its own sentence in the start gate, the boot sweep and `--restore-holds` (an unhealthy stop used to
|
||||
read as a failed restore).
|
||||
- **D53 (next release, felhom.eu `08` §10).** The hub learns which apps are held: `internal/holdreport` reads the hold
|
||||
store every minute and sends `app_held` (stack, reason, since, the hold's sentence) for a new or changed hold and
|
||||
`app_hold_cleared` when one is gone — also one cleared while the controller was down (`settings.reported_holds`) — and
|
||||
repeats every hold daily and at start. Both are severity info; the hub's operator reminder after a grace is the mail.
|
||||
- **R-690 (fixed here).** The removed-app restore (R-487) never found a unit on a DATA drive — it asked
|
||||
`GetStackComposePath`, true for every catalog app — and restored with no env. It now asks `isStackDeployed`.
|
||||
- `.felhom.yml` **`after_load: {service, user, command: [...]}`** — one command run with `docker compose exec -T`
|
||||
@@ -3610,9 +3595,6 @@ it), holds the app at 1 KiB/s (Transmission 1 kB/s), calls `POST /host/line-test
|
||||
back — the household's own included; a change made during the test stays. A killed controller puts it back at start
|
||||
(`RecoverHold`, before any rule); the 50 % rule skips while a record exists. Proven on demo-hp with a real seed (felhom.eu
|
||||
`audits/dooplex-alarms-2026-10-24/`).
|
||||
**v0.346.0 (R-1250):** between the hold and the call the controller waits until every held app reports its own upload
|
||||
near zero (≤ 16 KiB/s twice, 2 s apart, at most 90 s; an unreadable speed waits 20 s) — qBittorrent sends at full
|
||||
speed for ~14 s after its limit falls, which made the agent refuse every test „not quiet" (measured 2026-10-27).
|
||||
**v0.344.0 (R-1261):** EVERY running seeder is held, one record per app (`upwatch.line_test_holds`); a record whose app
|
||||
was removed while held is dropped (it used to stop the 50 % rule for good); qBittorrent's hold names its slot
|
||||
(`up_limit` / `alt_up_limit` through `app/preferences` — `transfer/uploadLimit` follows the CURRENT speed mode), so a
|
||||
|
||||
+6
-19
@@ -153,27 +153,14 @@ if [[ "${ACTION}" == "--push" || "${ACTION}" == "--multiarch" ]]; then
|
||||
fi
|
||||
head_sha=$(cd "${REPO_DIR}" && git rev-parse HEAD 2>/dev/null || echo unknown)
|
||||
main_sha=$(cd "${REPO_DIR}" && git rev-parse origin/main 2>/dev/null || echo unknown)
|
||||
# The one other source that can be fetched again: the tag v<VERSION> AS THE SERVER HAS IT, pointing at HEAD. That
|
||||
# is a patch release built off a released version while main holds unreleased work (v0.345.1, R-1300, cloudflared's
|
||||
# security bump, 2026-10-26 — operator's yes in that session). Only the tag that names THIS version counts; a tag
|
||||
# only on this clone, a tag on another commit, or a server that cannot be asked is a refusal. Pinned by
|
||||
# scripts/test_build_push_once.sh („tag" cases).
|
||||
if [[ "${head_sha}" != "${main_sha}" ]]; then
|
||||
tag_sha=$(cd "${REPO_DIR}" && GIT_TERMINAL_PROMPT=0 git ls-remote origin "refs/tags/v${VERSION}" "refs/tags/v${VERSION}^{}" 2>/dev/null \
|
||||
| awk '$2 ~ /\^\{\}$/ {peeled=$1} $2 !~ /\^\{\}$/ {plain=$1} END {print (peeled != "" ? peeled : plain)}') || tag_sha="" # no origin / unreachable: the error below, not set -e
|
||||
if [[ -n "${tag_sha}" && "${tag_sha}" == "${head_sha}" ]]; then
|
||||
info "Source tree is clean and at the pushed tag v${VERSION} (not origin/main)."
|
||||
else
|
||||
error "${REPO_DIR} is not at origin/main, nor at a pushed tag v${VERSION} — refusing to push."
|
||||
error " HEAD ${head_sha}"
|
||||
error " origin/main ${main_sha}"
|
||||
error " v${VERSION} on origin ${tag_sha:-<none>}"
|
||||
error "An unpushed change does not exist; an image built from one cannot be fetched again."
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
info "Source tree is clean and at origin/main."
|
||||
error "${REPO_DIR} is not at origin/main — refusing to push."
|
||||
error " HEAD ${head_sha}"
|
||||
error " origin/main ${main_sha}"
|
||||
error "An unpushed change does not exist; an image built from one cannot be fetched again."
|
||||
exit 1
|
||||
fi
|
||||
info "Source tree is clean and at origin/main."
|
||||
else
|
||||
error "${REPO_DIR} is not a git checkout — refusing to push a build whose source cannot be identified."
|
||||
exit 1
|
||||
|
||||
@@ -1,50 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log"
|
||||
"net/http"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
|
||||
)
|
||||
|
||||
// hostOOMSource (R-528) is the 30-s dead-app scan's host-side OOM counter: one agent call per scan (5 s cap). An agent
|
||||
// without the route (404) is not asked again for hostOOMRetryAfter, and said once; any other failure is a nil answer
|
||||
// for this scan only (Docker's flag still works). Pinned by TestR528_HostOOMSource*.
|
||||
const hostOOMRetryAfter = 10 * time.Minute
|
||||
|
||||
func hostOOMSource(ask func(context.Context) (agentapi.GuestOOMKillsState, error), logger *log.Logger, now func() time.Time) func() map[string]int64 {
|
||||
var mu sync.Mutex
|
||||
var until time.Time
|
||||
said := false
|
||||
return func() map[string]int64 {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if now().Before(until) {
|
||||
return nil
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
st, err := ask(ctx)
|
||||
if err != nil {
|
||||
var se *agentapi.StatusError
|
||||
if errors.As(err, &se) && se.Code == http.StatusNotFound {
|
||||
until = now().Add(hostOOMRetryAfter)
|
||||
if !said {
|
||||
logger.Printf("[INFO] [deadapp] the agent has no OOM counter route (R-528) — Docker's OOM flag only; asked again in %s", hostOOMRetryAfter)
|
||||
said = true
|
||||
}
|
||||
return nil
|
||||
}
|
||||
logger.Printf("[WARN] [deadapp] the host OOM counter could not be read: %v — Docker's OOM flag only for this scan", err)
|
||||
return nil
|
||||
}
|
||||
if !st.Available {
|
||||
return nil
|
||||
}
|
||||
return st.Containers
|
||||
}
|
||||
}
|
||||
@@ -40,7 +40,6 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/crashboot"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/crypto"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/fillwatch"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/holdreport"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/infra"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/integrations"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/mailrelay"
|
||||
@@ -1035,9 +1034,6 @@ func main() {
|
||||
logger.Printf("[WARN] [deadapp] agent client init failed (%v) — the boot grace cannot learn of a crash boot; 90 s", err)
|
||||
} else {
|
||||
crashProbe = crashboot.AgentProbe(ac.CrashGuard)
|
||||
// R-528: the 30-s scan also asks the host for the kernel's per-container OOM counter (Docker's own flag can
|
||||
// stay false inside an LXC guest). TestR528_HostOOMSourceIsWiredAtStartup.
|
||||
stackMgr.SetHostOOMKills(hostOOMSource(ac.GuestOOMKills, logger, time.Now))
|
||||
}
|
||||
}
|
||||
bootGrace := crashboot.New(startTime, crashProbe, logger)
|
||||
@@ -1067,20 +1063,6 @@ func main() {
|
||||
go func() { defer aubBusy.Unlock(); aubRunner.Tick(time.Now()) }() // a run can take minutes; never block the scheduler
|
||||
return nil
|
||||
})
|
||||
// D53 (felhom.eu `08` §10): tell the hub which apps are held — app_held for a new or changed hold, app_hold_cleared
|
||||
// when one is gone, every hold again once a day and after each start. One observer of the hold store, so every
|
||||
// writer is covered. Pinned by internal/holdreport.
|
||||
holdReporter := holdreport.New(sett, notifier.PushEvent, logger)
|
||||
// R-1298: „installed" = the app's app.yaml file exists (a product removal deletes it; an unreadable one still counts,
|
||||
// the R-1220 rule). An install's hold of an app without one is not reported.
|
||||
holdReporter.SetInstalled(func(stack string) bool {
|
||||
_, err := os.Stat(filepath.Join(cfg.Paths.StacksDir, stack, "app.yaml"))
|
||||
return err == nil
|
||||
})
|
||||
sched.Every("hold-report", time.Minute, func(ctx context.Context) error {
|
||||
holdReporter.Tick(time.Now())
|
||||
return nil
|
||||
})
|
||||
sched.Every("deadapp-check", 30*time.Second, func(ctx context.Context) error {
|
||||
if bootGrace.Within(time.Now()) {
|
||||
return nil // still inside the startup settle window (90 s, or ~15 min after a crash boot)
|
||||
@@ -1097,8 +1079,8 @@ func main() {
|
||||
logger.Printf("[WARN] [deadapp] OOM scan failed: %v", oerr)
|
||||
} else {
|
||||
for _, o := range ooms {
|
||||
logger.Printf("[WARN] [deadapp] %s: container %s was OOM-killed (started %s; kills %d, limit %s, peak %s; seen by %s)", o.Stack, o.Container, o.StartedAt, o.Kills, o.MemLimit, o.Peak, o.Source)
|
||||
notifier.NotifyAppOOMFrom(o.Stack, o.Container, o.StartedAt, o.Kills, o.MemLimit, o.Peak, o.Source)
|
||||
logger.Printf("[WARN] [deadapp] %s: container %s was OOM-killed (started %s; kills %d, limit %s, peak %s)", o.Stack, o.Container, o.StartedAt, o.Kills, o.MemLimit, o.Peak)
|
||||
notifier.NotifyAppOOM(o.Stack, o.Container, o.StartedAt, o.Kills, o.MemLimit, o.Peak)
|
||||
}
|
||||
}
|
||||
// v0.269.0 (`09` §3 decision 28, R-667): a crash loop or an OOM storm is STOPPED by the box.
|
||||
@@ -1403,13 +1385,6 @@ func main() {
|
||||
return nil
|
||||
})
|
||||
|
||||
// R-1256 H8: a restore copy no hold, no pending put-back and no running restore needs goes after 14 days. Every 6
|
||||
// hours — a copy outliving its age by hours costs disk, never data. TestR1256_H8_TheSweepIsWiredAtStartup.
|
||||
sched.Every("restore-copy-sweep", 6*time.Hour, func(ctx context.Context) error {
|
||||
backupMgr.SweepUnusedRestoreCopies(time.Now())
|
||||
return nil
|
||||
})
|
||||
|
||||
// Tier 2: off-drive copy of each HDD app's recovery unit + userdata (auto-enabled, auto-target).
|
||||
// Runs after the DB dump so it copies a fresh unit.
|
||||
backupMgr.SetTier2Notifier(func(stackName, destLabel string, dur time.Duration, err error) {
|
||||
@@ -2587,7 +2562,22 @@ type driveStartGate struct {
|
||||
// stop read as „held after a failed restore whose rollback also failed" for 11 days on demo-hp, so the operator could
|
||||
// not tell a crash loop from a broken restore). Pinned by TestR1286_EveryHoldReasonSaysWhatItIs.
|
||||
func holdWhy(h settings.RestoreHold) string {
|
||||
return h.Why() // one sentence per reason, shared with the hub's „Held apps" list (D53)
|
||||
switch h.Reason {
|
||||
case settings.HoldReasonUpdateFailed:
|
||||
return "held after a failed update (" + h.At + ") — restore it from its backup to start it"
|
||||
case settings.HoldReasonUnhealthyStop:
|
||||
kind := h.UnhealthyKind
|
||||
if kind == "" {
|
||||
kind = "unhealthy"
|
||||
}
|
||||
return "stopped by the box for a " + strings.ReplaceAll(kind, "_", " ") + " (" + h.At + ") — the household's Start gives it one more try"
|
||||
case settings.HoldReasonRestoreMixed:
|
||||
return "held after an off-site restore failed its database replay (" + h.At + ") — clear the hold to start it"
|
||||
case settings.HoldReasonPutBackPending:
|
||||
return "held until the pre-restore copy is poured back (" + h.At + ") — the put-back starts it"
|
||||
default:
|
||||
return "held after a failed restore whose rollback also failed (" + h.At + ") — clear the hold to start it"
|
||||
}
|
||||
}
|
||||
|
||||
func (g driveStartGate) MayStart(stackName string) (bool, string) {
|
||||
|
||||
@@ -1,13 +0,0 @@
|
||||
package main
|
||||
|
||||
import "testing"
|
||||
|
||||
// TestR1256_H8_TheSweepIsWiredAtStartup — R-1256 H8: an unused restore copy goes only through
|
||||
// SweepUnusedRestoreCopies, so main.go must CALL it (the seam-built-but-never-wired class).
|
||||
// RED-PROOF: delete the `restore-copy-sweep` job — this fails.
|
||||
func TestR1256_H8_TheSweepIsWiredAtStartup(t *testing.T) {
|
||||
lines, _, _ := slice4CallLines(t)
|
||||
if len(lines["SweepUnusedRestoreCopies"]) == 0 {
|
||||
t.Fatal("SweepUnusedRestoreCopies is never called — an unused restore copy would stay on the drive for ever")
|
||||
}
|
||||
}
|
||||
@@ -1,60 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
|
||||
)
|
||||
|
||||
// R-528: the scan's host counter source. An old agent (404) is asked again only after hostOOMRetryAfter; a transient
|
||||
// failure is retried at the next scan; the counters pass through.
|
||||
// RED-PROOF: drop the 404 back-off → an old agent is asked at every 30-s scan (calls 3, not 1) → FAILS.
|
||||
func TestR528_HostOOMSourceBacksOffAnOldAgent(t *testing.T) {
|
||||
clock := time.Date(2026, 10, 28, 23, 0, 0, 0, time.UTC)
|
||||
calls := 0
|
||||
var answer error = &agentapi.StatusError{Path: "/guest/oom-kills", Code: http.StatusNotFound}
|
||||
src := hostOOMSource(func(context.Context) (agentapi.GuestOOMKillsState, error) {
|
||||
calls++
|
||||
if answer != nil {
|
||||
return agentapi.GuestOOMKillsState{}, answer
|
||||
}
|
||||
return agentapi.GuestOOMKillsState{Available: true, Containers: map[string]int64{"id": 2}}, nil
|
||||
}, log.New(io.Discard, "", 0), func() time.Time { return clock })
|
||||
for i := 0; i < 3; i++ {
|
||||
if got := src(); got != nil {
|
||||
t.Fatalf("an old agent: %v", got)
|
||||
}
|
||||
clock = clock.Add(30 * time.Second)
|
||||
}
|
||||
if calls != 1 {
|
||||
t.Fatalf("an old agent was asked %d times in 90 s, want 1", calls)
|
||||
}
|
||||
clock = clock.Add(hostOOMRetryAfter)
|
||||
answer = nil
|
||||
if got := src(); got["id"] != 2 || calls != 2 {
|
||||
t.Fatalf("after the back-off, an upgraded agent: %v (calls %d)", got, calls)
|
||||
}
|
||||
answer = errors.New("timeout")
|
||||
if got := src(); got != nil || calls != 3 {
|
||||
t.Fatalf("a transient failure: %v", got)
|
||||
}
|
||||
answer = nil
|
||||
if got := src(); got["id"] != 2 || calls != 4 {
|
||||
t.Fatalf("the next scan after a transient failure must ask again: %v (calls %d)", got, calls)
|
||||
}
|
||||
}
|
||||
|
||||
// The source is wired into the stack manager at startup (the seam-built-but-never-wired class).
|
||||
// RED-PROOF: delete the SetHostOOMKills line in main.go → FAILS.
|
||||
func TestR528_HostOOMSourceIsWiredAtStartup(t *testing.T) {
|
||||
lines, _, _ := slice4CallLines(t)
|
||||
if len(lines["SetHostOOMKills"]) == 0 || len(lines["hostOOMSource"]) == 0 {
|
||||
t.Fatal("the host OOM counter is never wired — the 30-s scan would read Docker's flag only (R-528)")
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
module gitea.dooplex.hu/admin/felhom-controller
|
||||
|
||||
go 1.27.3
|
||||
go 1.27.2
|
||||
|
||||
require (
|
||||
github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6
|
||||
|
||||
@@ -24,7 +24,6 @@
|
||||
# Evidence: felhom.eu/documentation/audits/DIAG-sharing-2026-07-20.md.
|
||||
# 1.2.0 (R-972): Alpine 3.24.2, pinned by digest — the same base as the hub. 3.21 carried 48 fixable high findings
|
||||
# (audits/security-review-2026-10-12/scanners/). Same packages; scanner before → after in the controller's v0.323.0 CHANGELOG.
|
||||
# upstream: gap:Alpine's package fixes (Samba among them) are in its secdb, which the watch does not read; BEHIND on the point release is the signal
|
||||
FROM alpine:3.24.2@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
|
||||
|
||||
# samba = smbd + nmbd + smbpasswd/testparm (meta-package proven installable in the spike);
|
||||
|
||||
@@ -1071,18 +1071,6 @@ func (c *Client) RemoveNetStorage(ctx context.Context, name string) error {
|
||||
// status, even on a non-2xx (so callers like FormatDisk can read a 403 refusal body). A transport or
|
||||
// envelope-parse failure is still an error; an `ok:false` business refusal is NOT (the data carries it).
|
||||
func (c *Client) postWithStatus(ctx context.Context, path string, body any) (apiResponse, int, error) {
|
||||
return c.postWithStatusOn(c.hc, ctx, path, body)
|
||||
}
|
||||
|
||||
// postWithStatusTimeout is postWithStatus with its own overall timeout (the escrow claim and confirm, R-1254). It shares
|
||||
// the pinned transport.
|
||||
func (c *Client) postWithStatusTimeout(ctx context.Context, path string, body any, timeout time.Duration) (apiResponse, int, error) {
|
||||
hc := *c.hc
|
||||
hc.Timeout = timeout
|
||||
return c.postWithStatusOn(&hc, ctx, path, body)
|
||||
}
|
||||
|
||||
func (c *Client) postWithStatusOn(hc *http.Client, ctx context.Context, path string, body any) (apiResponse, int, error) {
|
||||
var env apiResponse
|
||||
buf, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
@@ -1095,7 +1083,7 @@ func (c *Client) postWithStatusOn(hc *http.Client, ctx context.Context, path str
|
||||
req.Header.Set("Authorization", "Bearer "+c.token)
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
start := time.Now()
|
||||
resp, err := hc.Do(req)
|
||||
resp, err := c.hc.Do(req)
|
||||
if err != nil {
|
||||
logx.Debugf(c.logger, "[agentapi] POST %s failed after %dms: %v", path, time.Since(start).Milliseconds(), err)
|
||||
return env, 0, fmt.Errorf("agentapi: POST %s: %w", path, err)
|
||||
|
||||
@@ -6,7 +6,6 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Controller-driven escrow ceremony client methods (v0.127.0, agent ≥ v0.88.0). The claim call
|
||||
@@ -109,15 +108,8 @@ type EscrowClaim struct {
|
||||
//
|
||||
// R-1151: the claim asks the agent to keep the new package until the household typed the code back
|
||||
// (`defer_upload`). An agent older than 0.160.0 ignores the body and uploads as before.
|
||||
// escrowCallTimeout bounds the claim and the confirm (R-1254 H2). The agent may take its whole upload timeout (30 s)
|
||||
// plus its question to the hub (15 s) before it answers; the client's own 15 s used to abandon the call while the
|
||||
// agent went on — a claim abandoned that way could still store the package, and its code reached nobody. 60 s
|
||||
// outlasts both (TestR1254_ClaimAndConfirmOutwaitTheAgentsUpload; the agent side: escrowUploadTimeout,
|
||||
// escrowHoldTimeout in felhom-agent internal/localapi/escrow_ceremony.go — if either grows, this must too).
|
||||
var escrowCallTimeout = 60 * time.Second
|
||||
|
||||
func (c *Client) EscrowCeremonyClaim(ctx context.Context) (EscrowClaim, int, error) {
|
||||
env, status, err := c.postWithStatusTimeout(ctx, "/escrow/ceremony/claim", map[string]bool{"defer_upload": true}, escrowCallTimeout)
|
||||
env, status, err := c.postWithStatus(ctx, "/escrow/ceremony/claim", map[string]bool{"defer_upload": true})
|
||||
if err != nil {
|
||||
return EscrowClaim{}, status, err
|
||||
}
|
||||
@@ -149,7 +141,7 @@ func (c *Client) EscrowCeremonyClaim(ctx context.Context) (EscrowClaim, int, err
|
||||
// match (still waiting), 410 the window ran out, 404 nothing waits (superseded, restarted), 502 the hub failed (still
|
||||
// waiting). The words are part of the recovery code — never logged.
|
||||
func (c *Client) EscrowCeremonyConfirm(ctx context.Context, words []string) (int, error) {
|
||||
env, status, err := c.postWithStatusTimeout(ctx, "/escrow/ceremony/confirm", map[string][]string{"words": words}, escrowCallTimeout)
|
||||
env, status, err := c.postWithStatus(ctx, "/escrow/ceremony/confirm", map[string][]string{"words": words})
|
||||
if err != nil {
|
||||
return status, err
|
||||
}
|
||||
|
||||
@@ -1,31 +0,0 @@
|
||||
package agentapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// GuestOOMKillsState mirrors the agent's GET /guest/oom-kills (R-528, agent ≥ the R-528 commit): the kernel's oom_kill
|
||||
// counter of each of THIS guest's containers, read on the host — Docker's own OOMKilled flag can stay false inside an
|
||||
// LXC guest. Containers maps the full 64-hex container id to the count for the container's current run.
|
||||
// Available=false: the guest has no cgroup folder the host can read. An agent that predates the route answers 404 (a
|
||||
// *StatusError) — the controller then keeps Docker's flag only.
|
||||
type GuestOOMKillsState struct {
|
||||
VMID int `json:"vmid"`
|
||||
Available bool `json:"available"`
|
||||
Containers map[string]int64 `json:"containers"`
|
||||
}
|
||||
|
||||
// GuestOOMKills calls GET /guest/oom-kills.
|
||||
func (c *Client) GuestOOMKills(ctx context.Context) (GuestOOMKillsState, error) {
|
||||
var out GuestOOMKillsState
|
||||
body, err := c.get(ctx, "/guest/oom-kills")
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
if err := json.Unmarshal(body, &out); err != nil {
|
||||
return out, fmt.Errorf("agentapi: decode /guest/oom-kills: %w", err)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -1,42 +0,0 @@
|
||||
package agentapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-1254 H2: the agent's claim/confirm may take its whole upload timeout (30 s) plus the hub question (15 s) before it
|
||||
// answers. The controller's agent client gives up at 15 s — and a claim the controller abandons can still complete on
|
||||
// the agent: the hub then holds a package whose code nobody saw. The two calls carry their own, longer timeout.
|
||||
// RED-PROOF: send claim/confirm through the plain 15 s client again → the slow agent below times them out → FAILS.
|
||||
func TestR1254_ClaimAndConfirmOutwaitTheAgentsUpload(t *testing.T) {
|
||||
if escrowCallTimeout < 45*time.Second+10*time.Second {
|
||||
t.Fatalf("escrowCallTimeout %s must outlast the agent's upload (30 s) + hub question (15 s) with room", escrowCallTimeout)
|
||||
}
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("POST /escrow/ceremony/claim", func(w http.ResponseWriter, r *http.Request) {
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
_, _ = w.Write([]byte(`{"ok":true,"data":{"recovery_code":"a-b-c"}}`))
|
||||
})
|
||||
mux.HandleFunc("POST /escrow/ceremony/confirm", func(w http.ResponseWriter, r *http.Request) {
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
_, _ = w.Write([]byte(`{"ok":true,"data":{"uploaded":true}}`))
|
||||
})
|
||||
s := httptest.NewTLSServer(mux)
|
||||
defer s.Close()
|
||||
c := clientFor(t, s, strings.TrimPrefix(s.URL, "https://"))
|
||||
c.hc.Timeout = 100 * time.Millisecond // the client's own short timeout, scaled down
|
||||
old := escrowCallTimeout
|
||||
escrowCallTimeout = 5 * time.Second
|
||||
defer func() { escrowCallTimeout = old }()
|
||||
if claim, st, err := c.EscrowCeremonyClaim(context.Background()); err != nil || st != http.StatusOK || claim.Code != "a-b-c" {
|
||||
t.Fatalf("a slow claim was abandoned: %+v %d %v", claim, st, err)
|
||||
}
|
||||
if st, err := c.EscrowCeremonyConfirm(context.Background(), []string{"a", "c"}); err != nil || st != http.StatusOK {
|
||||
t.Fatalf("a slow confirm was abandoned: %d %v", st, err)
|
||||
}
|
||||
}
|
||||
@@ -94,9 +94,7 @@ func (r *Router) keptDataAtInstall(w http.ResponseWriter, req *http.Request, nam
|
||||
writeJSON(w, http.StatusConflict, apiResponse{OK: false, Error: data.UseOff, Data: data})
|
||||
return true
|
||||
}
|
||||
// R-1256 H6: every operation — a restore, a backup run, a second-drive copy — not only the first two; and the
|
||||
// load itself begins in the same step as this question (startKeptLoad → ErrRestoreBusy).
|
||||
if busy, _ := r.backupMgr.RestoreBusy(); busy != "" {
|
||||
if st := r.backupMgr.RestoreStatus(); st.Running || r.backupMgr.IsRunning() {
|
||||
writeJSON(w, http.StatusConflict, apiResponse{OK: false, Error: r.msgLang(lang, "api.kept.busy")})
|
||||
return true
|
||||
}
|
||||
@@ -106,11 +104,7 @@ func (r *Router) keptDataAtInstall(w http.ResponseWriter, req *http.Request, nam
|
||||
}
|
||||
r.logger.Printf("[INFO] [api] Deploy %s: USE MY KEPT DATA — loading from %s (tier %d, %s) under the kept files %v", name, src, cp.c.Tier, cp.c.Time.UTC().Format(time.RFC3339), old)
|
||||
r.dropLeftoverHold(name, "use-kept-data") // R-704: this is a new install of an app that is not installed
|
||||
if err := r.startKeptLoad(name, hdd, cp.c, lang, nil); err != nil {
|
||||
r.logger.Printf("[WARN] [api] Deploy %s: use my kept data REFUSED at its start: %v", name, err)
|
||||
writeJSON(w, http.StatusConflict, apiResponse{OK: false, Error: r.msgLang(lang, "api.kept.busy")})
|
||||
return true
|
||||
}
|
||||
r.startKeptLoad(name, hdd, cp.c, lang, nil)
|
||||
writeJSON(w, http.StatusAccepted, apiResponse{OK: true, Message: r.msgLang(lang, "kept.load.started", display)})
|
||||
return true
|
||||
default:
|
||||
@@ -126,7 +120,7 @@ func (r *Router) keptCopyName(lang string, c backup.KeptCopy) string {
|
||||
|
||||
// startKeptLoad runs the load and, when it succeeded, the app's after_load. then(ok) runs last. The
|
||||
// off-site copy goes through LoadKeptOffsite (download the unit alone, judge it, then the same restore).
|
||||
func (r *Router) startKeptLoad(name, drive string, c backup.KeptCopy, lang string, then func(ok bool)) error {
|
||||
func (r *Router) startKeptLoad(name, drive string, c backup.KeptCopy, lang string, then func(ok bool)) {
|
||||
display := name
|
||||
if st, ok := r.stackMgr.GetStack(name); ok && st.Meta.DisplayName != "" {
|
||||
display = st.Meta.DisplayName
|
||||
@@ -134,9 +128,10 @@ func (r *Router) startKeptLoad(name, drive string, c backup.KeptCopy, lang strin
|
||||
okMsg := func(error) string { return r.msgLang(lang, "kept.load.done", display) }
|
||||
failMsg := func(err error) string { return r.msgLang(lang, "kept.load.failed", display, err) }
|
||||
if c.Tier == backup.KeptTierOffsite {
|
||||
return r.backupMgr.LoadKeptOffsite(name, drive, c, nil, okMsg, failMsg, r.keptAfterLoad(name, then))
|
||||
r.backupMgr.LoadKeptOffsite(name, drive, c, nil, okMsg, failMsg, r.keptAfterLoad(name, then))
|
||||
return
|
||||
}
|
||||
return r.backupMgr.LoadKeptApp(name, c.UnitDir, nil, okMsg, failMsg, r.keptAfterLoad(name, then))
|
||||
r.backupMgr.LoadKeptApp(name, c.UnitDir, okMsg, failMsg, r.keptAfterLoad(name, then))
|
||||
}
|
||||
|
||||
// keptAfterLoad runs the app's after_load when the load succeeded, then then(ok).
|
||||
|
||||
@@ -162,10 +162,7 @@ func TestR691_InstallChoiceNamesTheOffsiteCopy(t *testing.T) {
|
||||
// copy, restic and the unit restore behind their seams — no Docker).
|
||||
// COMPANION RED-PROOF: remove the dropLeftoverHold call from keptDataAtInstall → the hold is still there
|
||||
// after the 202 and the first assertion fails; remove it from deployStack → the AST check fails.
|
||||
// keptInstallRig is an installable app with old data on its drive and an off-site copy of its unit; done receives
|
||||
// once per unit restore the load reaches.
|
||||
func keptInstallRig(t *testing.T) (*Router, *settings.Settings, string, chan struct{}) {
|
||||
t.Helper()
|
||||
func TestR704_AFreshInstallDropsALeftoverHold(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
drive := filepath.Join(dir, "drive")
|
||||
cfg := &config.Config{}
|
||||
@@ -237,11 +234,6 @@ func keptInstallRig(t *testing.T) (*Router, *settings.Settings, string, chan str
|
||||
return backup.UnitRestoreResult{}, nil
|
||||
})
|
||||
r := &Router{stackMgr: m, backupMgr: bm, sett: sett, logger: log.New(io.Discard, "", 0)}
|
||||
return r, sett, drive, done
|
||||
}
|
||||
|
||||
func TestR704_AFreshInstallDropsALeftoverHold(t *testing.T) {
|
||||
r, sett, drive, done := keptInstallRig(t)
|
||||
|
||||
// A leftover update hold from an earlier install (the demo-hp shape): the new install drops it.
|
||||
_ = sett.SetRestoreHold(settings.RestoreHold{Stack: "cloudapp", At: "2026-09-13T19:56:32Z", Reason: settings.HoldReasonUpdateFailed})
|
||||
@@ -295,38 +287,3 @@ func TestR704_AFreshInstallDropsALeftoverHold(t *testing.T) {
|
||||
t.Fatalf("deployStack must call dropLeftoverHold before DeployStack (drop=%v deploy=%v)", dropAt, deployAt)
|
||||
}
|
||||
}
|
||||
|
||||
// R-1256 H6: the install-time „use my kept data" asks about EVERY operation — a second-drive copy too, not only a
|
||||
// restore record and the backup run — and the load begins in the same step as the question.
|
||||
// RED-PROOF: restore the RestoreStatus/IsRunning-only check and LoadKept*'s plain BeginRestoreOp → 202 and the load
|
||||
// runs beside the copy → FAILS.
|
||||
func TestR1256_H6_AKeptLoadAtInstallIsRefusedWhileASecondDriveCopyRuns(t *testing.T) {
|
||||
r, _, drive, done := keptInstallRig(t)
|
||||
end := backup.MarkTier2CopyForTest("someapp")
|
||||
w := httptest.NewRecorder()
|
||||
req := httptest.NewRequest(http.MethodPost, "/api/stacks/cloudapp/deploy?lang=en", nil)
|
||||
r.keptDataAtInstall(w, req, "cloudapp", drive, "use")
|
||||
end()
|
||||
if w.Code != http.StatusConflict {
|
||||
t.Fatalf("a load during a second-drive copy: code %d, want 409 — body %s", w.Code, w.Body.String())
|
||||
}
|
||||
select {
|
||||
case <-done:
|
||||
t.Fatal("the load ran beside the second-drive copy")
|
||||
case <-time.After(300 * time.Millisecond):
|
||||
}
|
||||
if st := r.backupMgr.RestoreStatus(); st.Running {
|
||||
t.Fatal("a refused load left a restore op recorded")
|
||||
}
|
||||
// The copy ended: the same press loads.
|
||||
w = httptest.NewRecorder()
|
||||
r.keptDataAtInstall(w, httptest.NewRequest(http.MethodPost, "/api/stacks/cloudapp/deploy?lang=en", nil), "cloudapp", drive, "use")
|
||||
if w.Code != http.StatusAccepted {
|
||||
t.Fatalf("after the copy: code %d — body %s", w.Code, w.Body.String())
|
||||
}
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Fatal("the load never reached the unit restore")
|
||||
}
|
||||
}
|
||||
@@ -771,12 +771,6 @@ func (r *Router) actionStack(w http.ResponseWriter, req *http.Request, action, n
|
||||
// above, so nothing is recorded for an action that was never going to happen.
|
||||
if desired, ok := desiredStateForAction(action); ok {
|
||||
if derr := r.stackMgr.SetDesiredState(name, desired); derr != nil {
|
||||
if errors.Is(derr, stacks.ErrStackNotFound) {
|
||||
// no app by that name on this box (e.g. the bootstrap-managed controller): a 404, not a save failure
|
||||
r.logger.Printf("[WARN] [api] %s for %s refused: no such app", action, name)
|
||||
writeJSON(w, stackOpStatusFor(derr), apiResponse{OK: false, Error: r.errText(req, derr)})
|
||||
return
|
||||
}
|
||||
r.logger.Printf("[ERROR] [api] %s for %s refused: could not record desired state: %v", action, name, derr)
|
||||
writeJSON(w, http.StatusInternalServerError, apiResponse{
|
||||
OK: false,
|
||||
|
||||
@@ -1,29 +0,0 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Found walking the capability map's B2 („App lifecycle") on 9202, 2026-10-28 night: a Stop of a name that is no app
|
||||
// on this box (`felhom-controller` — bootstrap-managed, not a stack) answered 500 „…az alkalmazás beállításai nem
|
||||
// menthetők." — the desired-state write's „not found" read as a save failure. It is a 404 now, before any intent is
|
||||
// recorded. RED-PROOF: drop the ErrStackNotFound branch in actionStack → 500 → FAILS.
|
||||
func TestAnActionOnANameThatIsNoAppIsA404(t *testing.T) {
|
||||
r, _, _, _ := newSlice4Router(t)
|
||||
for _, action := range []string{"stop", "start"} {
|
||||
w := httptest.NewRecorder()
|
||||
r.actionStack(w, httptest.NewRequest("POST", "/api/stacks/nosuchapp/"+action, nil), action, "nosuchapp")
|
||||
var resp apiResponse
|
||||
_ = json.Unmarshal(w.Body.Bytes(), &resp)
|
||||
if w.Code != http.StatusNotFound {
|
||||
t.Fatalf("%s of a name that is no app: %d %q — want 404", action, w.Code, resp.Error)
|
||||
}
|
||||
if strings.Contains(strings.ToLower(resp.Error), "mentheto") || strings.Contains(resp.Error, "menthetők") {
|
||||
t.Fatalf("%s: the answer still blames a save: %q", action, resp.Error)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -42,9 +42,6 @@ func dbTypeForImage(image string) (DBType, bool) {
|
||||
return DBType(""), false
|
||||
}
|
||||
|
||||
// DBTypeForImage is dbTypeForImage for other packages (R-1187 g: the .fab import classifies with the SAME predicate).
|
||||
func DBTypeForImage(image string) (DBType, bool) { return dbTypeForImage(image) }
|
||||
|
||||
// composeServicesDoc is the minimal view of a compose file needed here: the `services:` MAP and each
|
||||
// service's `image:`. Deliberately a real YAML parse and not a line scan — a top-level `volumes:`
|
||||
// block (immich's `immich_ml_cache:`) has exactly the shape a naive scan misreads as a service, and
|
||||
|
||||
@@ -728,11 +728,10 @@ func (e *Exporter) exportHDDData(req ExportRequest, dataDir string, manifest *Ma
|
||||
e.debugf("tarring HDD mount: %s → %s (%d exclude(s))", mount, tarPath, len(excludes))
|
||||
tarStart := time.Now()
|
||||
if err := tarDirectoryExcluding(mount, tarPath, excludes); err != nil {
|
||||
// R-1187 (a): a folder that cannot be packed FAILS the export. It used to be left out with a WARN while the
|
||||
// export ended „done" — a bundle silently missing a whole drive folder. Pinned by TestR1187a_*.
|
||||
e.logger.Printf("[WARN] Export: failed to pack %s — the export FAILS rather than leave it out (R-1187 a): %v", mount, err)
|
||||
// v0.125.0: claim the subdir ONLY on success — a claimed-but-absent tar would trip
|
||||
// the packaging assertion; an honestly-skipped mount stays out of the manifest.
|
||||
e.logger.Printf("[WARN] Export: failed to tar %s (excluded from the bundle): %v", mount, err)
|
||||
os.Remove(tarPath)
|
||||
return err
|
||||
} else {
|
||||
manifest.HDDSubdirs = append(manifest.HDDSubdirs, subdir)
|
||||
if info, _ := os.Stat(tarPath); info != nil {
|
||||
|
||||
@@ -2,7 +2,6 @@ package appexport
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/util"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -290,20 +289,7 @@ func tarDirectoryExcluding(sourceDir, outputPath string, excludeRels []string) e
|
||||
return nil
|
||||
}
|
||||
}
|
||||
// R-1187 (a): a symlink is carried AS A LINK (its target's bytes never — the walk used to copy them into the link
|
||||
// header, the tar failed, and the whole folder was left out); a FIFO, socket or device stops the export with a
|
||||
// sentence naming the path (os.Open on a FIFO used to HANG the export). Pinned by TestR1187a_*.
|
||||
mode := info.Mode()
|
||||
if mode&(os.ModeNamedPipe|os.ModeSocket|os.ModeDevice|os.ModeCharDevice|os.ModeIrregular) != 0 {
|
||||
return util.MsgError("err.appexport.kulonleges_fajl", path)
|
||||
}
|
||||
link := ""
|
||||
if mode&os.ModeSymlink != 0 {
|
||||
if link, err = os.Readlink(path); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
header, err := tar.FileInfoHeader(info, link)
|
||||
header, err := tar.FileInfoHeader(info, "")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -311,7 +297,7 @@ func tarDirectoryExcluding(sourceDir, outputPath string, excludeRels []string) e
|
||||
if err := tw.WriteHeader(header); err != nil {
|
||||
return err
|
||||
}
|
||||
if info.IsDir() || link != "" {
|
||||
if info.IsDir() {
|
||||
return nil
|
||||
}
|
||||
f, err := os.Open(path)
|
||||
|
||||
@@ -1,274 +0,0 @@
|
||||
package appexport
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-1187 (a), (b), (d), (f), (g) — the open parts of the `.fab` data-loss row. Each test asserts the CONSEQUENCE for
|
||||
// the household's files. Red-proved 2026-10-26 night against the code before the fix (function shapes added first with
|
||||
// the old behaviour, so every failure below was a behaviour failure, not a compile error).
|
||||
|
||||
func r1187TarEntries(t *testing.T, path string) map[string]*tar.Header {
|
||||
t.Helper()
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer f.Close()
|
||||
out := map[string]*tar.Header{}
|
||||
tr := tar.NewReader(f)
|
||||
for {
|
||||
h, err := tr.Next()
|
||||
if err == io.EOF {
|
||||
return out
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("reading %s: %v", path, err)
|
||||
}
|
||||
out[h.Name] = h
|
||||
}
|
||||
}
|
||||
|
||||
// (a) A symlink in a drive folder is carried AS A LINK — before, the walk wrote a link header and then copied the
|
||||
// target's bytes into it, the tar failed, and the whole folder was left out with only a WARN (the export ended „done").
|
||||
func TestR1187a_ASymlinkIsCarriedAsALink(t *testing.T) {
|
||||
src := t.TempDir()
|
||||
os.WriteFile(filepath.Join(src, "photo.jpg"), []byte("jpeg"), 0o644)
|
||||
os.Symlink("photo.jpg", filepath.Join(src, "latest.jpg"))
|
||||
os.Symlink("/etc/hostname", filepath.Join(src, "outside"))
|
||||
out := filepath.Join(t.TempDir(), "x.tar")
|
||||
if err := tarDirectoryExcluding(src, out, nil); err != nil {
|
||||
t.Fatalf("a folder with symlinks could not be packed: %v", err)
|
||||
}
|
||||
e := r1187TarEntries(t, out)
|
||||
if h := e["latest.jpg"]; h == nil || h.Typeflag != tar.TypeSymlink || h.Linkname != "photo.jpg" {
|
||||
t.Fatalf("the inside link is not carried as a link: %+v", h)
|
||||
}
|
||||
if h := e["outside"]; h == nil || h.Typeflag != tar.TypeSymlink || h.Size != 0 {
|
||||
t.Fatalf("the outside link must be carried as a bare link (never its target's bytes): %+v", h)
|
||||
}
|
||||
if h := e["photo.jpg"]; h == nil || h.Typeflag != tar.TypeReg {
|
||||
t.Fatalf("the regular file is missing: %+v", h)
|
||||
}
|
||||
}
|
||||
|
||||
// (a) A FIFO stops the export with a sentence naming the path — never a hang on os.Open, never a silent drop. The
|
||||
// export of the folder FAILS as a whole (exportHDDData returns the error; the job then fails).
|
||||
func TestR1187a_AFifoFailsTheExportNamingThePath(t *testing.T) {
|
||||
mount := filepath.Join(t.TempDir(), "appdata-x")
|
||||
os.MkdirAll(mount, 0o755)
|
||||
os.WriteFile(filepath.Join(mount, "keep.txt"), []byte("k"), 0o644)
|
||||
fifo := filepath.Join(mount, "pipe")
|
||||
if err := syscall.Mkfifo(fifo, 0o644); err != nil {
|
||||
t.Skipf("no FIFO here: %v", err)
|
||||
}
|
||||
prov := &r1187HDDProvider{rtProvider: rtProvider{stackDir: t.TempDir()}, mounts: []string{mount}}
|
||||
e := NewExporter(prov, log.New(io.Discard, "", 0), "test")
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- e.exportHDDData(ExportRequest{StackName: "x"}, t.TempDir(), &Manifest{}) }()
|
||||
select {
|
||||
case err := <-done:
|
||||
if err == nil || !strings.Contains(err.Error(), fifo) {
|
||||
t.Fatalf("a FIFO must fail the export naming %s, got %v", fifo, err)
|
||||
}
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Fatal("the export HUNG on a FIFO")
|
||||
}
|
||||
}
|
||||
|
||||
type r1187HDDProvider struct {
|
||||
rtProvider
|
||||
mounts []string
|
||||
}
|
||||
|
||||
func (p *r1187HDDProvider) GetStackHDDMounts(string) []string { return p.mounts }
|
||||
|
||||
// (b) The import keeps owners and modes, restores a link that stays inside the app's folder, and refuses (and lists) a
|
||||
// link that points outside it. Before: only folders and regular files were written — owners lost, every link dropped.
|
||||
func TestR1187b_TheImportKeepsOwnersModesAndInsideLinks(t *testing.T) {
|
||||
tarPath := filepath.Join(t.TempDir(), "d.tar")
|
||||
f, _ := os.Create(tarPath)
|
||||
tw := tar.NewWriter(f)
|
||||
tw.WriteHeader(&tar.Header{Name: "sub/", Typeflag: tar.TypeDir, Mode: 0o750, Uid: 1000, Gid: 1000})
|
||||
tw.WriteHeader(&tar.Header{Name: "sub/a.txt", Typeflag: tar.TypeReg, Mode: 0o640, Size: 2, Uid: 1000, Gid: 1001})
|
||||
tw.Write([]byte("hi"))
|
||||
tw.WriteHeader(&tar.Header{Name: "link-in", Typeflag: tar.TypeSymlink, Linkname: "sub/a.txt", Uid: 1000, Gid: 1000})
|
||||
tw.WriteHeader(&tar.Header{Name: "link-out", Typeflag: tar.TypeSymlink, Linkname: "../../etc/passwd"})
|
||||
tw.WriteHeader(&tar.Header{Name: "link-abs", Typeflag: tar.TypeSymlink, Linkname: "/etc/passwd"})
|
||||
tw.Close()
|
||||
f.Close()
|
||||
|
||||
var chowned []string
|
||||
old := lchownFn
|
||||
lchownFn = func(_ *os.Root, p string, uid, gid int) error {
|
||||
chowned = append(chowned, filepath.Base(p)+":"+strconv.Itoa(uid)+":"+strconv.Itoa(gid))
|
||||
return nil
|
||||
}
|
||||
defer func() { lchownFn = old }()
|
||||
|
||||
dest := t.TempDir()
|
||||
refused, err := extractTar(tarPath, dest)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if fi, err := os.Stat(filepath.Join(dest, "sub/a.txt")); err != nil || fi.Mode().Perm() != 0o640 {
|
||||
t.Fatalf("mode not kept: %v %v", fi, err)
|
||||
}
|
||||
if l, err := os.Readlink(filepath.Join(dest, "link-in")); err != nil || l != "sub/a.txt" {
|
||||
t.Fatalf("the inside link was not restored: %q %v", l, err)
|
||||
}
|
||||
for _, bad := range []string{"link-out", "link-abs"} {
|
||||
if _, err := os.Lstat(filepath.Join(dest, bad)); err == nil {
|
||||
t.Fatalf("a link pointing OUTSIDE the app's folder was restored: %s", bad)
|
||||
}
|
||||
}
|
||||
if strings.Join(refused, ",") != "link-out,link-abs" {
|
||||
t.Fatalf("the refused links must be listed, got %v", refused)
|
||||
}
|
||||
got := strings.Join(chowned, ",")
|
||||
for _, want := range []string{"sub:1000:1000", "a.txt:1000:1001", "link-in:1000:1000"} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Fatalf("owner %s not restored (got %s)", want, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// (d) An import is refused while another operation holds the app-stop marker (a restore, a put-back, a stop-first
|
||||
// export) — before, StartImport checked only its own job slot.
|
||||
func TestR1187d_AnImportWaitsForARestore(t *testing.T) {
|
||||
e := NewExporter(&rtProvider{stackDir: t.TempDir()}, log.New(io.Discard, "", 0), "test")
|
||||
e.SetStopGuard(&busyGuard{})
|
||||
fab := filepath.Join(t.TempDir(), "x.fab")
|
||||
os.WriteFile(fab, []byte("x"), 0o644)
|
||||
err := e.StartImport(ImportRequest{FABPath: fab})
|
||||
if err == nil || !strings.Contains(err.Error(), "unit-restore:bookstack") {
|
||||
t.Fatalf("an import started while a restore holds the box: %v", err)
|
||||
}
|
||||
if e.activeJob != nil {
|
||||
t.Fatal("a refused import created a job")
|
||||
}
|
||||
}
|
||||
|
||||
// (f) A bundle whose app.yaml carries an empty or relative HDD_PATH is refused BEFORE anything is touched — before, an
|
||||
// empty one made the drive tars extract into a RELATIVE path (the controller's working directory).
|
||||
func TestR1187f_AnEmptyOrRelativeHDDPathIsRefused(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
hdd string
|
||||
ok bool
|
||||
}{{"", false}, {"data", false}, {"/mnt/drive/../../etc", false}, {"/mnt/sys_drive", true}} {
|
||||
tmp := t.TempDir()
|
||||
os.MkdirAll(filepath.Join(tmp, "config"), 0o755)
|
||||
os.WriteFile(filepath.Join(tmp, "config", "app.yaml"), []byte("env:\n HDD_PATH: \""+tc.hdd+"\"\n"), 0o644)
|
||||
os.MkdirAll(filepath.Join(tmp, "data", "hdd"), 0o755)
|
||||
os.WriteFile(filepath.Join(tmp, "data", "hdd", "appdata.tar"), []byte("not empty"), 0o644)
|
||||
err := validateBundleData(tmp, &Manifest{HDDSubdirs: []string{"appdata"}, HasHDDData: true})
|
||||
if tc.ok != (err == nil) {
|
||||
t.Errorf("HDD_PATH %q: refused=%v, want refused=%v (%v)", tc.hdd, err != nil, !tc.ok, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// (g) PostGIS / pgvector / timescaledb images are PostgreSQL for the import too (the backup side's predicate, R-484),
|
||||
// and psql stops on the first error (the dump is --clean --if-exists, so ON_ERROR_STOP is safe).
|
||||
func TestR1187g_PostgisIsPostgresAndPsqlStopsOnError(t *testing.T) {
|
||||
for _, img := range []string{"postgis/postgis:16-3.5-alpine", "pgvector/pgvector:pg16", "timescale/timescaledb:2.14.2-pg16", "postgres:16"} {
|
||||
if info := classifyDBImage("db", img, "c"); info == nil || info.DBType != "postgres" {
|
||||
t.Errorf("%s is not classified as PostgreSQL: %+v", img, info)
|
||||
}
|
||||
}
|
||||
// The load itself goes through appbackup.ImportDump (psql -v ON_ERROR_STOP=1 --single-transaction) since the
|
||||
// import's own client calls were removed — TestR1187_TheImportLoadsTheDumpThroughTheBackupSide.
|
||||
}
|
||||
|
||||
// (b), the security review's case: a link already in the destination folder (left by the old app, or made by an earlier
|
||||
// entry) that points OUTSIDE must never be written through — a tar entry „evil/passwd" would otherwise create a file
|
||||
// wherever „evil" points. Every write is confined to the app's folder (os.Root). RED-PROOF: with plain os.Create the
|
||||
// outside file is created.
|
||||
func TestR1187b_NoWriteThroughALinkThatLeavesTheFolder(t *testing.T) {
|
||||
outside := t.TempDir()
|
||||
dest := t.TempDir()
|
||||
os.Symlink(outside, filepath.Join(dest, "evil")) // pre-existing, pointing out
|
||||
tarPath := filepath.Join(t.TempDir(), "d.tar")
|
||||
f, _ := os.Create(tarPath)
|
||||
tw := tar.NewWriter(f)
|
||||
tw.WriteHeader(&tar.Header{Name: "evil/passwd", Typeflag: tar.TypeReg, Mode: 0o644, Size: 3})
|
||||
tw.Write([]byte("pwn"))
|
||||
tw.WriteHeader(&tar.Header{Name: "ok.txt", Typeflag: tar.TypeReg, Mode: 0o644, Size: 2})
|
||||
tw.Write([]byte("ok"))
|
||||
tw.Close()
|
||||
f.Close()
|
||||
old := lchownFn
|
||||
lchownFn = func(*os.Root, string, int, int) error { return nil }
|
||||
defer func() { lchownFn = old }()
|
||||
extractTar(tarPath, dest)
|
||||
if _, err := os.Stat(filepath.Join(outside, "passwd")); err == nil {
|
||||
t.Fatal("a file was written OUTSIDE the app's folder through a pre-existing link")
|
||||
}
|
||||
}
|
||||
|
||||
// (b), the security review's second case — a parser differential: link „a" → „." makes „a/b" really „b", so a later
|
||||
// link „a/b/l" → „../.." passes a path-TEXT check (root/a/b/../.. = root) while it really points one level ABOVE the
|
||||
// app's folder. A link entry whose parent path passes through a link is refused and listed. RED-PROOF: with the text
|
||||
// check only, „a/b/l" is created and points outside.
|
||||
func TestR1187b_NoLinkThroughALinkedParent(t *testing.T) {
|
||||
dest := t.TempDir()
|
||||
tarPath := filepath.Join(t.TempDir(), "d.tar")
|
||||
f, _ := os.Create(tarPath)
|
||||
tw := tar.NewWriter(f)
|
||||
tw.WriteHeader(&tar.Header{Name: "a", Typeflag: tar.TypeSymlink, Linkname: "."})
|
||||
tw.WriteHeader(&tar.Header{Name: "a/b/", Typeflag: tar.TypeDir, Mode: 0o755})
|
||||
tw.WriteHeader(&tar.Header{Name: "a/b/l", Typeflag: tar.TypeSymlink, Linkname: "../.."})
|
||||
tw.Close()
|
||||
f.Close()
|
||||
old := lchownFn
|
||||
lchownFn = func(*os.Root, string, int, int) error { return nil }
|
||||
defer func() { lchownFn = old }()
|
||||
refused, err := extractTar(tarPath, dest)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, p := range []string{filepath.Join(dest, "b", "l"), filepath.Join(dest, "a", "b", "l")} {
|
||||
if _, err := os.Lstat(p); err == nil {
|
||||
real, _ := filepath.EvalSymlinks(p)
|
||||
t.Fatalf("a link through a linked parent was created (%s → %s)", p, real)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(strings.Join(refused, ","), "a/b/l") {
|
||||
t.Fatalf("the refused link must be listed, got %v", refused)
|
||||
}
|
||||
}
|
||||
|
||||
// (b), the security review's third case: a link TARGET that passes through a link already in the folder („evil" → an
|
||||
// outside folder, then „x" → „evil/y") reads as inside by its text and really points outside. Restored links may only
|
||||
// point DOWNWARD to real folders: no absolute target, no „..", no link on the way. RED-PROOF: the text check created „x".
|
||||
func TestR1187b_NoLinkTargetThroughALink(t *testing.T) {
|
||||
outside := t.TempDir()
|
||||
dest := t.TempDir()
|
||||
os.Symlink(outside, filepath.Join(dest, "evil"))
|
||||
tarPath := filepath.Join(t.TempDir(), "d.tar")
|
||||
f, _ := os.Create(tarPath)
|
||||
tw := tar.NewWriter(f)
|
||||
tw.WriteHeader(&tar.Header{Name: "x", Typeflag: tar.TypeSymlink, Linkname: "evil/y"})
|
||||
tw.WriteHeader(&tar.Header{Name: "up", Typeflag: tar.TypeSymlink, Linkname: "sub/../sub"})
|
||||
tw.Close()
|
||||
f.Close()
|
||||
old := lchownFn
|
||||
lchownFn = func(*os.Root, string, int, int) error { return nil }
|
||||
defer func() { lchownFn = old }()
|
||||
refused, _ := extractTar(tarPath, dest)
|
||||
if _, err := os.Lstat(filepath.Join(dest, "x")); err == nil {
|
||||
t.Fatal("a link whose target passes through an outside-pointing link was created")
|
||||
}
|
||||
if strings.Join(refused, ",") != "x,up" {
|
||||
t.Fatalf("both links must be refused and listed, got %v", refused)
|
||||
}
|
||||
}
|
||||
@@ -1,47 +0,0 @@
|
||||
package appexport
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"log"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/appbackup"
|
||||
)
|
||||
|
||||
// R-1187, found live on 9202 2026-10-26 night: the `.fab` import waited with `mysqladmin` and loaded with `mysql` — names
|
||||
// the MariaDB 11+ images do not ship (only mariadb-admin / mariadb; measured in mariadb:11.4) — and read the password
|
||||
// from the app's env under two fixed names (grimmory's is DB_ROOT_PASSWORD). Every import of a MariaDB 11+ app timed
|
||||
// out and left the app down. The import now uses the BACKUP side's discovery (the live container's own credentials)
|
||||
// and appbackup.ImportDump (mariadb / mariadb-admin, psql ON_ERROR_STOP + --single-transaction). The seams keep this
|
||||
// test off the real Docker (R-650). RED-PROOF: the old body (its own waitForDB + importDBDump) never reaches the seams.
|
||||
func TestR1187_TheImportLoadsTheDumpThroughTheBackupSide(t *testing.T) {
|
||||
oldD, oldI := discoverDBsFn, importDumpFn
|
||||
defer func() { discoverDBsFn, importDumpFn = oldD, oldI }()
|
||||
discoverDBsFn = func(ctx context.Context, l *log.Logger, debug bool, known []string) ([]appbackup.DiscoveredDB, error) {
|
||||
return []appbackup.DiscoveredDB{
|
||||
{ContainerName: "paperless-postgres", ContainerID: "1b47ee5f58d2", DBType: appbackup.DBTypePostgres, StackName: "paperless-ngx"},
|
||||
{ContainerName: "grimmory-db", ContainerID: "2310c379bd74", DBType: appbackup.DBTypeMariaDB, DBName: "grimmory", StackName: "grimmory"},
|
||||
}, nil
|
||||
}
|
||||
var got appbackup.DiscoveredDB
|
||||
var gotPath string
|
||||
importDumpFn = func(ctx context.Context, db appbackup.DiscoveredDB, dumpPath string, l *log.Logger, debug bool) error {
|
||||
got, gotPath = db, dumpPath
|
||||
return nil
|
||||
}
|
||||
e := NewExporter(&rtProvider{}, log.New(io.Discard, "", 0), "test")
|
||||
err := e.importIntoContainer("grimmory", "2310c379bd742a9bbb3a78ceb60d1872edb2dfe0ddc15b134edb9142b048e558", "mariadb", "/tmp/x/grimmory-mariadb.sql.gz", nil)
|
||||
if err != nil {
|
||||
t.Fatalf("import: %v", err)
|
||||
}
|
||||
if got.ContainerName != "grimmory-db" || gotPath != "/tmp/x/grimmory-mariadb.sql.gz" {
|
||||
t.Fatalf("the dump did not go through the backup side's ImportDump for the started container: %+v %q", got, gotPath)
|
||||
}
|
||||
// A container the backup side cannot see is a clear refusal, never a guess.
|
||||
err = e.importIntoContainer("grimmory", "ffffffffffff", "mariadb", "/tmp/x/d.sql.gz", nil)
|
||||
if err == nil || !strings.Contains(err.Error(), "ffffffffffff") {
|
||||
t.Fatalf("an undiscovered container must be refused naming it, got %v", err)
|
||||
}
|
||||
}
|
||||
@@ -1,40 +0,0 @@
|
||||
package appexport
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/appbackup"
|
||||
)
|
||||
|
||||
// R-1187, found live on 9202 2026-10-26 night (grimmory): the import ran `compose` WITHOUT the two path variables the
|
||||
// controller adds at every install and start (USERDATA_PATH, IMPORT_PATH — stacks.policyEnv). Compose then printed
|
||||
// „The USERDATA_PATH variable is not set" on its combined output, the DB step read that WARNING as the container id,
|
||||
// waited 60 s for a container that does not exist, and left the app removed and not started. Both halves pinned here.
|
||||
// RED-PROOF: with the old helpers (the first output line; the env unchanged) both tests fail.
|
||||
func TestR1187_TheContainerIDIsNeverAComposeWarning(t *testing.T) {
|
||||
out := []byte("time=\"2026-10-26T20:18:06Z\" level=warning msg=\"The \\\"USERDATA_PATH\\\" variable is not set. Defaulting to a blank string.\"\n" +
|
||||
"1047c698c778ab12cd34ef56ab78cd90ef12ab34cd56ef78ab90cd12ef34ab56\n")
|
||||
if got := containerIDFrom(out); got != "1047c698c778ab12cd34ef56ab78cd90ef12ab34cd56ef78ab90cd12ef34ab56" {
|
||||
t.Fatalf("container id = %q", got)
|
||||
}
|
||||
if got := containerIDFrom([]byte("time=x level=warning msg=y\n")); got != "" {
|
||||
t.Fatalf("a warning alone must give no id, got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestR1187_TheImportEnvCarriesTheControllersPathVars(t *testing.T) {
|
||||
env := map[string]string{"HDD_PATH": "/mnt/felhom-drives/d", "USERDATA_PATH": "/mnt/felhom-drives/d/appdata/other", "X": "1"}
|
||||
got := withImportPathVars(env, "/mnt/felhom-drives/d", "/mnt/sys_drive/felhom-data/userdata/import")
|
||||
if got["USERDATA_PATH"] != appbackup.UserdataDir("/mnt/felhom-drives/d") {
|
||||
t.Fatalf("USERDATA_PATH = %q — must be the controller's own derivation, never the bundle's", got["USERDATA_PATH"])
|
||||
}
|
||||
if got["IMPORT_PATH"] != "/mnt/sys_drive/felhom-data/userdata/import" || got["X"] != "1" || got["HDD_PATH"] != "/mnt/felhom-drives/d" {
|
||||
t.Fatalf("env = %v", got)
|
||||
}
|
||||
if env["USERDATA_PATH"] != "/mnt/felhom-drives/d/appdata/other" {
|
||||
t.Fatal("the caller's map was changed")
|
||||
}
|
||||
if g := withImportPathVars(map[string]string{"IMPORT_PATH": "/x"}, "", ""); g["USERDATA_PATH"] != "" || g["IMPORT_PATH"] != "" {
|
||||
t.Fatalf("no namespace / no import root must leave neither variable: %v", g)
|
||||
}
|
||||
}
|
||||
@@ -5,11 +5,11 @@ import (
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"fmt"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/appbackup"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/dockerexec"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/util"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
@@ -184,16 +184,6 @@ func (e *Exporter) StartImport(req ImportRequest) error {
|
||||
e.debugf("StartImport rejected: another job is already running")
|
||||
return fmt.Errorf("export or import already in progress")
|
||||
}
|
||||
// R-1187 (d): another operation holds the app-stop marker (a restore, a put-back, a stop-first export) — an import
|
||||
// now would stop and rewrite an app that operation is moving. Refused before anything is read. Pinned by
|
||||
// TestR1187d_AnImportWaitsForARestore.
|
||||
if e.stopGuard != nil {
|
||||
if busy := e.stopGuard.BusyWith(); busy != "" {
|
||||
e.mu.Unlock()
|
||||
e.logger.Printf("[WARN] Import: refused — another operation holds the box (%s) (R-1187 d)", busy)
|
||||
return util.MsgError("err.appexport.import_foglalt", busy)
|
||||
}
|
||||
}
|
||||
|
||||
// Validate file exists
|
||||
info, err := os.Stat(req.FABPath)
|
||||
@@ -441,8 +431,6 @@ func (e *Exporter) executeImport(req ImportRequest, job *Job) {
|
||||
e.failJob(job, step, fmt.Sprintf("Konfiguráció visszaállítása sikertelen: %v", err))
|
||||
return
|
||||
}
|
||||
// R-1187 (found live): compose runs below see the controller's path variables, as at every install and start.
|
||||
env = withImportPathVars(env, e.provider.GetStackNamespaceRoot(manifest.AppName), e.provider.GetImportRoot())
|
||||
e.debugf("config restored: %d env vars", len(env))
|
||||
if e.debug {
|
||||
for k := range env {
|
||||
@@ -686,14 +674,9 @@ func (e *Exporter) restoreHDDData(tmpDir string, manifest *Manifest, composePath
|
||||
|
||||
e.logger.Printf("[INFO] Import: extracting HDD data %s → %s", subdir, destPath)
|
||||
extractStart := time.Now()
|
||||
refused, err := extractTar(tarPath, destPath)
|
||||
if err != nil {
|
||||
if err := extractTar(tarPath, destPath); err != nil {
|
||||
return fmt.Errorf("extracting %s: %w", subdir, err)
|
||||
}
|
||||
if len(refused) > 0 {
|
||||
e.logger.Printf("[WARN] Import: %s — %d link(s) pointing outside the app's folder were NOT restored (R-1187 b): %s",
|
||||
subdir, len(refused), strings.Join(refused, ", "))
|
||||
}
|
||||
e.debugf("restoreHDDData: extracted %s in %v", subdir, time.Since(extractStart))
|
||||
}
|
||||
|
||||
@@ -743,16 +726,6 @@ func resolveHDDMounts(composePath string, env map[string]string) []string {
|
||||
// validateBundleData asserts every manifest-claimed data tar exists non-empty in the extracted
|
||||
// bundle (v0.125.0, scenario C). Pure read — called before ANY destructive import step.
|
||||
func validateBundleData(tmpDir string, manifest *Manifest) error {
|
||||
// R-1187 (f): drive data needs an ABSOLUTE, clean HDD_PATH from the bundle's app.yaml — an empty one made the drive
|
||||
// tars extract into a relative path (the controller's working directory). Refused before anything is touched.
|
||||
// Pinned by TestR1187f_AnEmptyOrRelativeHDDPathIsRefused.
|
||||
if len(manifest.HDDSubdirs) > 0 {
|
||||
env, _ := readPlaintextAppYaml(filepath.Join(tmpDir, "config", "app.yaml"))
|
||||
hdd := strings.TrimSpace(env["HDD_PATH"])
|
||||
if hdd == "" || !filepath.IsAbs(hdd) || filepath.Clean(hdd) != hdd || strings.Contains(hdd, "..") {
|
||||
return util.MsgError("err.appexport.hdd_path_hibas", hdd)
|
||||
}
|
||||
}
|
||||
for _, v := range manifest.VolumeNames {
|
||||
if err := ValidateSegment("volume_name", v); err != nil {
|
||||
return err
|
||||
@@ -883,53 +856,29 @@ func (e *Exporter) restoreDatabase(manifest *Manifest, tmpDir, composePath, stac
|
||||
}
|
||||
e.debugf("restoreDatabase: container ID=%s", containerID)
|
||||
|
||||
// Wait for readiness and import the dump.
|
||||
// Wait for DB readiness
|
||||
e.logger.Printf("[INFO] Import: waiting for DB readiness (container %s)", containerID[:12])
|
||||
e.debugf("restoreDatabase: waiting for DB readiness...")
|
||||
waitStart := time.Now()
|
||||
if err := waitForDB(containerID, dbSvc.DBType, env); err != nil {
|
||||
e.debugf("restoreDatabase: DB not ready after %v: %v", time.Since(waitStart), err)
|
||||
return fmt.Errorf("waiting for DB: %w", err)
|
||||
}
|
||||
e.debugf("restoreDatabase: DB ready in %v", time.Since(waitStart))
|
||||
|
||||
// Import the dump
|
||||
e.logger.Printf("[INFO] Import: importing database dump")
|
||||
e.debugf("restoreDatabase: importing dump into container %s", containerID[:12])
|
||||
importStart := time.Now()
|
||||
if err := e.importIntoContainer(manifest.AppName, containerID, dbSvc.DBType, dumpGzPath, env); err != nil {
|
||||
if err := importDBDump(containerID, dumpGzPath, dbSvc.DBType, env); err != nil {
|
||||
e.debugf("restoreDatabase: import failed after %v: %v", time.Since(importStart), err)
|
||||
return err
|
||||
return fmt.Errorf("importing dump: %w", err)
|
||||
}
|
||||
e.debugf("restoreDatabase: dump imported in %v", time.Since(importStart))
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Seams for importIntoContainer (a test replaces them; production is the backup side's own code).
|
||||
var (
|
||||
discoverDBsFn = appbackup.DiscoverDatabases
|
||||
importDumpFn = appbackup.ImportDump
|
||||
)
|
||||
|
||||
// importIntoContainer waits for the database and loads the dump — through the BACKUP side (R-1187, found live on 9202):
|
||||
// the container's own credentials from appbackup's discovery, and appbackup.ImportDump (mariadb / mariadb-admin, which
|
||||
// the MariaDB 11+ images ship; psql ON_ERROR_STOP + --single-transaction). The import's own waitForDB/importDBDump used
|
||||
// mysqladmin/mysql (absent from mariadb:11.4) and two fixed env names for the password, so every MariaDB 11+ import
|
||||
// timed out. Pinned by TestR1187_TheImportLoadsTheDumpThroughTheBackupSide.
|
||||
func (e *Exporter) importIntoContainer(app, containerID, dbType, dumpPath string, env map[string]string) error {
|
||||
ctx := context.Background()
|
||||
dbs, err := discoverDBsFn(ctx, e.logger, e.debug, []string{app})
|
||||
if err != nil {
|
||||
return fmt.Errorf("finding the database container: %w", err)
|
||||
}
|
||||
for _, db := range dbs {
|
||||
if db.ContainerID == "" || !(strings.HasPrefix(containerID, db.ContainerID) || strings.HasPrefix(db.ContainerID, containerID)) {
|
||||
continue
|
||||
}
|
||||
e.logger.Printf("[INFO] Import: loading the database dump into %s (%s)", db.ContainerName, db.DBType)
|
||||
if err := importDumpFn(ctx, db, dumpPath, e.logger, e.debug); err != nil {
|
||||
return fmt.Errorf("importing dump: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("the started database container %s is not among the databases the box can see", shortID(containerID))
|
||||
}
|
||||
|
||||
func shortID(id string) string {
|
||||
if len(id) > 12 {
|
||||
return id[:12]
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
// findDumpFile finds the gzipped SQL dump in the database directory.
|
||||
func findDumpFile(dbDir string) (string, error) {
|
||||
entries, err := os.ReadDir(dbDir)
|
||||
@@ -1018,12 +967,11 @@ func findDBServiceInCompose(composePath string) *dbServiceInfo {
|
||||
}
|
||||
|
||||
func classifyDBImage(service, image, container string) *dbServiceInfo {
|
||||
// R-1187 (g): the backup side's predicate (appbackup.DBTypeForImage, R-484) — postgis / pgvector / timescaledb are
|
||||
// PostgreSQL too. The import used its own narrower list and skipped adventurelog's and dawarich's database.
|
||||
switch t, ok := appbackup.DBTypeForImage(image); {
|
||||
case ok && t == appbackup.DBTypePostgres:
|
||||
img := strings.ToLower(image)
|
||||
if strings.Contains(img, "postgres") {
|
||||
return &dbServiceInfo{ServiceName: service, DBType: "postgres", ContainerName: container}
|
||||
case ok && t == appbackup.DBTypeMariaDB:
|
||||
}
|
||||
if strings.Contains(img, "mariadb") || strings.Contains(img, "mysql") {
|
||||
return &dbServiceInfo{ServiceName: service, DBType: "mariadb", ContainerName: container}
|
||||
}
|
||||
return nil
|
||||
@@ -1054,43 +1002,118 @@ func getComposeContainerID(stackDir string, env map[string]string, service strin
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("compose ps: %w", err)
|
||||
}
|
||||
id := containerIDFrom(out)
|
||||
id := strings.TrimSpace(string(out))
|
||||
if id == "" {
|
||||
return "", fmt.Errorf("no container found for service %s", service)
|
||||
}
|
||||
// Take the first line in case of multiple
|
||||
if idx := strings.Index(id, "\n"); idx > 0 {
|
||||
id = id[:idx]
|
||||
}
|
||||
return id, nil
|
||||
}
|
||||
|
||||
// containerIDFrom picks the container id out of `compose ps -q` output: the first line that IS an id (12–64 hex). The
|
||||
// output is combined, so a compose WARNING can come first — it was read as the id once (R-1187, found live on 9202).
|
||||
func containerIDFrom(out []byte) string {
|
||||
for _, line := range strings.Split(string(out), "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if n := len(line); n >= 12 && n <= 64 && strings.Trim(line, "0123456789abcdef") == "" {
|
||||
return line
|
||||
// waitForDB polls until the database is ready to accept connections.
|
||||
func waitForDB(containerID, dbType string, env map[string]string) error {
|
||||
timeout := 60 * time.Second
|
||||
start := time.Now()
|
||||
|
||||
for {
|
||||
if time.Since(start) > timeout {
|
||||
return fmt.Errorf("timeout waiting for %s to become ready", dbType)
|
||||
}
|
||||
|
||||
var cmd *exec.Cmd
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
|
||||
switch dbType {
|
||||
case "postgres":
|
||||
user := env["POSTGRES_USER"]
|
||||
if user == "" {
|
||||
user = "postgres"
|
||||
}
|
||||
cmd = dockerexec.CommandContext(ctx, "docker", "exec", containerID,
|
||||
"pg_isready", "-U", user)
|
||||
case "mariadb":
|
||||
password := env["MYSQL_ROOT_PASSWORD"]
|
||||
if password == "" {
|
||||
password = env["MARIADB_ROOT_PASSWORD"]
|
||||
}
|
||||
cmd = dockerexec.CommandContext(ctx, "docker", "exec", containerID,
|
||||
"mysqladmin", "ping", "-u", "root", "-p"+password)
|
||||
default:
|
||||
cancel()
|
||||
return fmt.Errorf("unknown DB type: %s", dbType)
|
||||
}
|
||||
|
||||
err := cmd.Run()
|
||||
cancel()
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
time.Sleep(2 * time.Second)
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// withImportPathVars is the import's env as `compose` must see it: the bundle's values plus the two path variables the
|
||||
// controller itself adds at every install and start (stacks.policyEnv) — USERDATA_PATH always the controller's own
|
||||
// derivation from the namespace root (never a bundle's value), IMPORT_PATH the drop-zone root. Without them compose
|
||||
// printed a warning the DB step read as the container id (R-1187, live on 9202). Returns a new map.
|
||||
func withImportPathVars(env map[string]string, nsRoot, importRoot string) map[string]string {
|
||||
out := make(map[string]string, len(env)+2)
|
||||
for k, v := range env {
|
||||
out[k] = v
|
||||
// importDBDump imports a (possibly gzipped) SQL dump into the running database container.
|
||||
func importDBDump(containerID, dumpPath, dbType string, env map[string]string) error {
|
||||
// Open the dump file
|
||||
f, err := os.Open(dumpPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
delete(out, "USERDATA_PATH")
|
||||
delete(out, "IMPORT_PATH")
|
||||
if nsRoot != "" {
|
||||
out["USERDATA_PATH"] = appbackup.UserdataDir(nsRoot)
|
||||
defer f.Close()
|
||||
|
||||
// If gzipped, wrap in gzip reader
|
||||
var reader io.Reader = f
|
||||
if strings.HasSuffix(dumpPath, ".gz") {
|
||||
gr, err := gzip.NewReader(f)
|
||||
if err != nil {
|
||||
return fmt.Errorf("opening gzip: %w", err)
|
||||
}
|
||||
defer gr.Close()
|
||||
reader = gr
|
||||
}
|
||||
if importRoot != "" {
|
||||
out["IMPORT_PATH"] = importRoot
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
var cmd *exec.Cmd
|
||||
switch dbType {
|
||||
case "postgres":
|
||||
user := env["POSTGRES_USER"]
|
||||
if user == "" {
|
||||
user = "postgres"
|
||||
}
|
||||
dbName := env["POSTGRES_DB"]
|
||||
if dbName == "" {
|
||||
dbName = user
|
||||
}
|
||||
cmd = dockerexec.CommandContext(ctx, "docker", "exec", "-i", containerID,
|
||||
"psql", "-U", user, "-d", dbName)
|
||||
case "mariadb":
|
||||
password := env["MYSQL_ROOT_PASSWORD"]
|
||||
if password == "" {
|
||||
password = env["MARIADB_ROOT_PASSWORD"]
|
||||
}
|
||||
dbName := env["MYSQL_DATABASE"]
|
||||
if dbName == "" {
|
||||
dbName = env["MARIADB_DATABASE"]
|
||||
}
|
||||
cmd = dockerexec.CommandContext(ctx, "docker", "exec", "-i", containerID,
|
||||
"mysql", "-u", "root", "-p"+password, dbName)
|
||||
default:
|
||||
return fmt.Errorf("unknown DB type: %s", dbType)
|
||||
}
|
||||
return out
|
||||
|
||||
cmd.Stdin = reader
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s import failed: %s — %w", dbType, strings.TrimSpace(string(out)), err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// extractTarGz extracts a tar.gz archive to a directory.
|
||||
@@ -1147,142 +1170,50 @@ func extractTarGz(tgzPath, destDir string) error {
|
||||
}
|
||||
}
|
||||
|
||||
// safeDownwardLink is the rule for a restored link: a relative target with no „.." component, and no symbolic link on
|
||||
// the way — not among the folders of the link's own path, not among the existing components of its target.
|
||||
func safeDownwardLink(rt *os.Root, name, target string) bool {
|
||||
if target == "" || filepath.IsAbs(target) {
|
||||
return false
|
||||
}
|
||||
for _, part := range strings.Split(filepath.ToSlash(target), "/") {
|
||||
if part == ".." {
|
||||
return false
|
||||
}
|
||||
}
|
||||
if parentThroughLink(rt, name) {
|
||||
return false
|
||||
}
|
||||
acc := filepath.Dir(name)
|
||||
for _, part := range strings.Split(filepath.Clean(target), string(os.PathSeparator)) {
|
||||
acc = filepath.Join(acc, part)
|
||||
fi, err := rt.Lstat(acc)
|
||||
if err != nil {
|
||||
break // not there (yet): nothing further down can be a link that exists now
|
||||
}
|
||||
if fi.Mode()&os.ModeSymlink != 0 {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// parentThroughLink is true when any folder on name's path (inside the root) is a symbolic link.
|
||||
func parentThroughLink(rt *os.Root, name string) bool {
|
||||
dir := filepath.Dir(name)
|
||||
if dir == "." {
|
||||
return false
|
||||
}
|
||||
acc := ""
|
||||
for _, part := range strings.Split(dir, string(os.PathSeparator)) {
|
||||
acc = filepath.Join(acc, part)
|
||||
fi, err := rt.Lstat(acc)
|
||||
if err != nil {
|
||||
return false // not there yet: MkdirAll creates real folders
|
||||
}
|
||||
if fi.Mode()&os.ModeSymlink != 0 {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// lchownFn is the owner seam (a test records it; production is os.Lchown).
|
||||
var lchownFn = func(r *os.Root, name string, uid, gid int) error { return r.Lchown(name, uid, gid) }
|
||||
|
||||
// extractTar extracts a plain tar archive (one drive folder) into destDir and returns the entries it REFUSED.
|
||||
//
|
||||
// R-1187 (b): owners and modes are kept (Lchown — the controller runs as root on a box), and a symlink is restored when
|
||||
// it stays inside destDir; a link pointing outside (absolute, or climbing out with ..) is refused and listed — never
|
||||
// followed, never created. Only folders, regular files and links exist in a drive tar (the export refuses the rest).
|
||||
// Pinned by TestR1187b_TheImportKeepsOwnersModesAndInsideLinks.
|
||||
func extractTar(tarPath, destDir string) ([]string, error) {
|
||||
// extractTar extracts a plain tar archive to a directory.
|
||||
func extractTar(tarPath, destDir string) error {
|
||||
f, err := os.Open(tarPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
root := filepath.Clean(destDir)
|
||||
// Every write goes through os.Root: no entry can create or write a file outside destDir, not even through a link
|
||||
// already in the folder (left by the old app, or made by an earlier entry) that points out — the security review's
|
||||
// case, pinned by TestR1187b_NoWriteThroughALinkThatLeavesTheFolder.
|
||||
rt, err := os.OpenRoot(root)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rt.Close()
|
||||
var refused []string
|
||||
|
||||
tr := tar.NewReader(f)
|
||||
for {
|
||||
hdr, err := tr.Next()
|
||||
if err == io.EOF {
|
||||
return refused, nil
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return refused, err
|
||||
return err
|
||||
}
|
||||
name := filepath.Clean(hdr.Name)
|
||||
if name == "." || filepath.IsAbs(name) || name == ".." || strings.HasPrefix(name, ".."+string(os.PathSeparator)) {
|
||||
continue // path traversal
|
||||
}
|
||||
switch hdr.Typeflag {
|
||||
case tar.TypeDir:
|
||||
if err := rt.MkdirAll(name, os.FileMode(hdr.Mode).Perm()); err != nil {
|
||||
return refused, err
|
||||
}
|
||||
rt.Chmod(name, os.FileMode(hdr.Mode).Perm())
|
||||
case tar.TypeReg:
|
||||
if dir := filepath.Dir(name); dir != "." {
|
||||
if err := rt.MkdirAll(dir, 0755); err != nil {
|
||||
return refused, err
|
||||
}
|
||||
}
|
||||
out, err := rt.OpenFile(name, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, os.FileMode(hdr.Mode).Perm())
|
||||
if err != nil {
|
||||
refused = append(refused, hdr.Name) // e.g. its path goes through a link that leaves the folder
|
||||
continue
|
||||
}
|
||||
if _, err := io.Copy(out, tr); err != nil {
|
||||
out.Close()
|
||||
return refused, err
|
||||
}
|
||||
out.Close()
|
||||
rt.Chmod(name, os.FileMode(hdr.Mode).Perm())
|
||||
case tar.TypeSymlink:
|
||||
// A restored link may only point DOWNWARD inside the folder, through real folders: no absolute target, no
|
||||
// „..", and no link on the way — neither on the link's own path nor in its target. Such a link can never
|
||||
// resolve outside the app's folder, whatever else the folder holds. Everything else is refused and listed.
|
||||
// (Three rounds of the automated security review: a link written through, a linked parent, a target through
|
||||
// a link. Pinned by TestR1187b_*.)
|
||||
if !safeDownwardLink(rt, name, hdr.Linkname) {
|
||||
refused = append(refused, hdr.Name)
|
||||
continue
|
||||
}
|
||||
dst := hdr.Linkname
|
||||
if dir := filepath.Dir(name); dir != "." {
|
||||
if err := rt.MkdirAll(dir, 0755); err != nil {
|
||||
return refused, err
|
||||
}
|
||||
}
|
||||
rt.Remove(name)
|
||||
if err := rt.Symlink(dst, name); err != nil {
|
||||
refused = append(refused, hdr.Name)
|
||||
continue
|
||||
}
|
||||
default:
|
||||
refused = append(refused, hdr.Name)
|
||||
|
||||
target := filepath.Join(destDir, hdr.Name)
|
||||
|
||||
// Security: prevent path traversal
|
||||
if !strings.HasPrefix(filepath.Clean(target), filepath.Clean(destDir)+string(os.PathSeparator)) {
|
||||
continue
|
||||
}
|
||||
if err := lchownFn(rt, name, hdr.Uid, hdr.Gid); err != nil && !os.IsPermission(err) {
|
||||
return refused, fmt.Errorf("setting the owner of %s: %w", hdr.Name, err)
|
||||
|
||||
switch hdr.Typeflag {
|
||||
case tar.TypeDir:
|
||||
if err := os.MkdirAll(target, os.FileMode(hdr.Mode)); err != nil {
|
||||
return err
|
||||
}
|
||||
case tar.TypeReg:
|
||||
if err := os.MkdirAll(filepath.Dir(target), 0755); err != nil {
|
||||
return err
|
||||
}
|
||||
outFile, err := os.Create(target)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := io.Copy(outFile, tr); err != nil {
|
||||
outFile.Close()
|
||||
return err
|
||||
}
|
||||
outFile.Close()
|
||||
os.Chmod(target, os.FileMode(hdr.Mode))
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -301,12 +301,8 @@ func (m *Manager) downloadKeptOffsiteUnit(ctx context.Context, app, drive string
|
||||
// LoadKeptOffsite is LoadKeptApp from the off-site copy c: download the unit alone, judge it, then
|
||||
// prepare (nil = nothing; a dated folder's files move back here), then the one unit restore, then
|
||||
// after(ok). A download failure or a refusal never reaches prepare, so the kept files stay as they were.
|
||||
// ErrRestoreBusy (nothing started, after not called) when another operation runs (R-1256 H6).
|
||||
func (m *Manager) LoadKeptOffsite(app, drive string, c KeptCopy, prepare func() error, okMsg, failMsg func(err error) string, after func(ok bool)) error {
|
||||
if busy, _ := m.TryBeginRestoreOp("restore", app); busy != "" {
|
||||
m.logger.Printf("[WARN] [backup] kept load %s REFUSED: a %s operation runs (R-1256)", app, busy)
|
||||
return ErrRestoreBusy
|
||||
}
|
||||
func (m *Manager) LoadKeptOffsite(app, drive string, c KeptCopy, prepare func() error, okMsg, failMsg func(err error) string, after func(ok bool)) {
|
||||
m.BeginRestoreOp("restore", app)
|
||||
go func() {
|
||||
start := time.Now()
|
||||
fail := func(err error) {
|
||||
@@ -349,7 +345,6 @@ func (m *Manager) LoadKeptOffsite(app, drive string, c KeptCopy, prepare func()
|
||||
after(true)
|
||||
}
|
||||
}()
|
||||
return nil
|
||||
}
|
||||
|
||||
// RemovedUnitOnDrive is the start-fresh hook (stacks.SetKeptUnitFinder): the removed app's OWN unit when
|
||||
@@ -375,26 +370,11 @@ func (m *Manager) PrimaryUnitHome(app, drive string) string {
|
||||
|
||||
// LoadKeptApp runs the load as a restore operation the backup pages already follow (the poll banner):
|
||||
// Begin → RestoreFromRecoveryUnitAt(app, unitDir) → End, then after(ok) in the same goroutine (the
|
||||
// after_load command, the kept folder's tidy-up). prepare (nil = nothing) runs first, inside the operation — a dated
|
||||
// folder's files move back there. R-1256 H6: the op is begun in the same step as the check for every other operation;
|
||||
// a refusal returns ErrRestoreBusy with nothing started and nothing moved.
|
||||
func (m *Manager) LoadKeptApp(app, unitDir string, prepare func() error, okMsg, failMsg func(err error) string, after func(ok bool)) error {
|
||||
if busy, _ := m.TryBeginRestoreOp("restore", app); busy != "" {
|
||||
m.logger.Printf("[WARN] [backup] kept load %s REFUSED: a %s operation runs (R-1256)", app, busy)
|
||||
return ErrRestoreBusy
|
||||
}
|
||||
// after_load command, the kept folder's tidy-up). The caller has checked RestoreStatus/IsRunning.
|
||||
func (m *Manager) LoadKeptApp(app, unitDir string, okMsg, failMsg func(err error) string, after func(ok bool)) {
|
||||
m.BeginRestoreOp("restore", app)
|
||||
go func() {
|
||||
start := time.Now()
|
||||
if prepare != nil {
|
||||
if perr := prepare(); perr != nil {
|
||||
m.logger.Printf("[ERROR] [backup] kept load %s: preparing the kept files FAILED: %v — nothing was loaded", app, perr)
|
||||
m.EndRestoreOp(false, failMsg(perr))
|
||||
if after != nil {
|
||||
after(false)
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
res, err := m.keptUnitRestore()(app, unitDir)
|
||||
if errors.Is(err, ErrRestoreNotStarted) {
|
||||
m.keptLoadNotStarted(app, err, after)
|
||||
@@ -415,7 +395,6 @@ func (m *Manager) LoadKeptApp(app, unitDir string, prepare func() error, okMsg,
|
||||
after(true)
|
||||
}
|
||||
}()
|
||||
return nil
|
||||
}
|
||||
|
||||
// keptLoadNotStarted (R-1270): the kept data is loaded but the app did not start in time. Never „done": the record says
|
||||
|
||||
@@ -714,14 +714,6 @@ func (m *Manager) OffboxConfigured() bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// OffboxRepoPasswordOnBox reports whether the off-site repository password FILE is on the box — whatever the target's
|
||||
// state. R-1218: ClearOffboxTarget keeps that file on purpose, and a switched-off target keeps it too, so a recovery
|
||||
// ceremony run then must still seal it; OffboxConfigured() is the wrong question for „is there a key to escrow".
|
||||
func (m *Manager) OffboxRepoPasswordOnBox() bool {
|
||||
_, err := os.Stat(m.offboxPwPath())
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// offboxRepoPwPattern matches a valid restic repo password (generateOffboxPassword = 32 rand bytes → 64 hex).
|
||||
var offboxRepoPwPattern = regexp.MustCompile(`^[0-9a-fA-F]{64}$`)
|
||||
|
||||
|
||||
@@ -548,25 +548,6 @@ func (m *Manager) holdAppAfterMixedRestore(stack string, replayErr error, live *
|
||||
}
|
||||
}
|
||||
|
||||
// restartAtLiveDefinition (R-1256 H4) is the failure branch of the definition write, taken while no data has moved.
|
||||
// The writer writes the compose before the app's settings, so a failure can leave the snapshot's compose in the
|
||||
// stack dir: the LIVE definition is written back first, and only then is the app started. When the write-back fails
|
||||
// too, the app is held stopped (restore_mixed) — a start would run the snapshot's definition over the live data.
|
||||
// Pinned by TestR1256_H4_*.
|
||||
func (m *Manager) restartAtLiveDefinition(stack string, live *liveDefinition, cause error, restartStack func() error) {
|
||||
if live != nil {
|
||||
if err := m.stackProvider.RecreateStackDefinitionFromUnit(stack, live.dir, live.env); err != nil {
|
||||
m.logger.Printf("[ERROR] [offbox] %s: the definition write failed (%v) and writing the live definition back failed too: %v — holding the app stopped", stack, cause, err)
|
||||
m.holdAppAfterMixedRestore(stack, cause, nil)
|
||||
return
|
||||
}
|
||||
m.logger.Printf("[INFO] [offbox] %s: the definition write failed (%v); the live definition is written back before the start", stack, cause)
|
||||
}
|
||||
if sErr := restartStack(); sErr != nil {
|
||||
m.logger.Printf("[WARN] [offbox] %s: restart after a failed definition write also failed: %v", stack, sErr)
|
||||
}
|
||||
}
|
||||
|
||||
// SetRestoreHoldNotify wires the operator notification for a held app (cmd/controller/main.go).
|
||||
func (m *Manager) SetRestoreHoldNotify(fn func(stack string, replayErr, rollbackErr error)) {
|
||||
m.restoreHoldNotify = fn
|
||||
@@ -1024,12 +1005,16 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack
|
||||
// yet — a failure here restarts the app as it was.
|
||||
if defineFromSnapshot {
|
||||
if err := m.stackProvider.RecreateStackDefinitionFromUnit(stack, scratchCompose, snapEnv); err != nil {
|
||||
m.restartAtLiveDefinition(stack, liveDef, err, restartStack)
|
||||
if sErr := restartStack(); sErr != nil {
|
||||
m.logger.Printf("[WARN] [offbox] %s: restart after a failed definition write also failed: %v", stack, sErr)
|
||||
}
|
||||
return res, fmt.Errorf("restoring %s: writing the snapshot's definition failed: %w", stack, err)
|
||||
}
|
||||
} else if len(credOverlay) > 0 {
|
||||
if err := m.stackProvider.(secretValueSetter).SetStackSecretValues(stack, credOverlay); err != nil {
|
||||
m.restartAtLiveDefinition(stack, liveDef, err, restartStack)
|
||||
if sErr := restartStack(); sErr != nil {
|
||||
m.logger.Printf("[WARN] [offbox] %s: restart after a failed password write also failed: %v", stack, sErr)
|
||||
}
|
||||
return res, fmt.Errorf("restoring %s: writing the backup's database password failed: %w", stack, err)
|
||||
}
|
||||
m.logger.Printf("[INFO] [offbox] %s: the backup's database password(s) %v written into the app's settings before the data (R-1106)", stack, credNames(credOverlay))
|
||||
|
||||
@@ -226,13 +226,8 @@ func (m *Manager) putBackAfterFailedRestore(stack string, cause error, live *liv
|
||||
m.holdAppAfterMixedRestore(stack, cause, live)
|
||||
return util.MsgError(holdKey, stack)
|
||||
}
|
||||
// R-1256 H5: a put-back under an app that could not be stopped could mix the copy with what the app writes.
|
||||
// Nothing is poured; the copy is kept and the app held (D20: a put-back that fails holds and keeps its copy).
|
||||
if err := m.stackProvider.StopStack(stack); err != nil {
|
||||
c.keepIfAny()
|
||||
m.logger.Printf("[ERROR] [offbox] %s: stopping the app before the put-back FAILED: %v — nothing is put back; the copy stays at %s; holding the app", stack, err, c.dirOrNone())
|
||||
m.holdAppAfterMixedRestore(stack, cause, live)
|
||||
return util.MsgError(holdKey, stack)
|
||||
m.logger.Printf("[WARN] [offbox] %s: stopping the app before the put-back failed: %v", stack, err)
|
||||
}
|
||||
if volumesMoved {
|
||||
if err := m.putBack(stack, c); err != nil {
|
||||
@@ -244,8 +239,7 @@ func (m *Manager) putBackAfterFailedRestore(stack string, cause error, live *liv
|
||||
}
|
||||
if live != nil {
|
||||
if err := m.stackProvider.RecreateStackDefinitionFromUnit(stack, live.dir, live.env); err != nil {
|
||||
c.keepIfAny() // R-1256 H5 (D20): a put-back that fails keeps its copy for support
|
||||
m.logger.Printf("[ERROR] [offbox] %s: writing the live definition back FAILED: %v — the copy stays at %s; holding the app", stack, err, c.dirOrNone())
|
||||
m.logger.Printf("[ERROR] [offbox] %s: writing the live definition back FAILED: %v — holding the app", stack, err)
|
||||
m.holdAppAfterMixedRestore(stack, cause, live)
|
||||
return util.MsgError(holdKey, stack)
|
||||
}
|
||||
@@ -255,19 +249,12 @@ func (m *Manager) putBackAfterFailedRestore(stack string, cause error, live *liv
|
||||
m.restorePutBackNotify(stack, cause)
|
||||
}
|
||||
if err := restart(); err != nil {
|
||||
m.logger.Printf("[ERROR] [offbox] %s: start after the put-back failed: %v (the live volumes and definition are back; the copy %s is removed)", stack, err, c.dirOrNone())
|
||||
m.logger.Printf("[ERROR] [offbox] %s: start after the put-back failed: %v (the copy stays at %s)", stack, err, c.dirOrNone())
|
||||
return util.MsgError("err.backup.restore_failed_put_back_not_started", stack)
|
||||
}
|
||||
return putBackMessage(stack, filesPlaced)
|
||||
}
|
||||
|
||||
// keepIfAny marks the copy as kept for support (no copy: nothing to keep).
|
||||
func (c *liveVolumeCopy) keepIfAny() {
|
||||
if c != nil {
|
||||
c.keep = true
|
||||
}
|
||||
}
|
||||
|
||||
func (c *liveVolumeCopy) dirOrNone() string {
|
||||
if c == nil {
|
||||
return "(no volume copy)"
|
||||
|
||||
@@ -62,53 +62,6 @@ type RestoreOpStatus struct {
|
||||
func (m *Manager) BeginRestoreOp(op, stack string) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.beginRestoreOpLocked(op, stack)
|
||||
}
|
||||
|
||||
// Why a new restore may not begin now (R-1256 H6). "" = it may.
|
||||
const (
|
||||
RestoreBusyRestore = "restore" // a restore op is in flight (the record the pages follow)
|
||||
RestoreBusyBackup = "backup" // the single-flight backup/restore run is held (the night run never records an op)
|
||||
RestoreBusyMirror = "mirror" // a second-drive copy runs, or one a previous process lost may still run
|
||||
)
|
||||
|
||||
// TryBeginRestoreOp is BeginRestoreOp that refuses — in the SAME step, under m.mu — while another restore op, a
|
||||
// backup run or a second-drive copy runs. R-1256 H6: the pages asked (RestoreStatus, IsRunning,
|
||||
// Tier2CopyBlocksRestore) and recorded the op later; a mirror that began in between ran beside the restore. The mirror
|
||||
// asks under the same m.mu (beginTier2Run), so one of the two always sees the other. Lock order m.mu → tier2Copies,
|
||||
// as in beginTier2Run. Returns the busy reason ("" = begun) and, for a mirror, its app ("" when unknown).
|
||||
// Pinned by TestR1256_H6_*.
|
||||
func (m *Manager) TryBeginRestoreOp(op, stack string) (busy, mirrorApp string) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if busy, app := m.restoreBusyLocked(); busy != "" {
|
||||
return busy, app
|
||||
}
|
||||
m.beginRestoreOpLocked(op, stack)
|
||||
return "", ""
|
||||
}
|
||||
|
||||
// RestoreBusy is TryBeginRestoreOp's question alone — for a refusal asked early, before a request is read further.
|
||||
func (m *Manager) RestoreBusy() (busy, mirrorApp string) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return m.restoreBusyLocked()
|
||||
}
|
||||
|
||||
func (m *Manager) restoreBusyLocked() (string, string) {
|
||||
if m.opRunning {
|
||||
return RestoreBusyRestore, m.opStack
|
||||
}
|
||||
if m.running {
|
||||
return RestoreBusyBackup, ""
|
||||
}
|
||||
if app, blocked := m.Tier2CopyBlocksRestore(); blocked {
|
||||
return RestoreBusyMirror, app
|
||||
}
|
||||
return "", ""
|
||||
}
|
||||
|
||||
func (m *Manager) beginRestoreOpLocked(op, stack string) {
|
||||
m.opRunning = true
|
||||
m.opName = op
|
||||
m.opStack = stack
|
||||
|
||||
@@ -1,358 +0,0 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// R-1256 (review round 2, HUNT-1): four restore edges — H4, H5, H6 and H8, each pinned below.
|
||||
//
|
||||
// H4: an off-site restore writes the snapshot's definition before any data moves. The writer
|
||||
// (stackAdapter.RecreateStackDefinitionFromUnit) writes the compose FIRST and the app's settings after, so a
|
||||
// failure can come back with the snapshot's compose already in the stack dir. The failure branch used to start
|
||||
// the app at once — on the snapshot's (older) compose, over the live data. Now the live definition is written
|
||||
// back first; only then is the app started. When the write-back fails too, the app is held, never started.
|
||||
|
||||
// r1256DefFailProvider fails the FIRST definition write (the snapshot's), after recording it — as the real
|
||||
// writer does when its compose write succeeded and its settings write did not.
|
||||
type r1256DefFailProvider struct {
|
||||
*r893Provider
|
||||
failWrites int // how many writes fail, from the first
|
||||
failFrom int // when set, every write from this one (1-based) on fails
|
||||
}
|
||||
|
||||
func (p *r1256DefFailProvider) RecreateStackDefinitionFromUnit(name, composeDir string, env map[string]string) error {
|
||||
if err := p.r893Provider.RecreateStackDefinitionFromUnit(name, composeDir, env); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(p.defs) <= p.failWrites || (p.failFrom > 0 && len(p.defs) >= p.failFrom) {
|
||||
return errors.New("app.yaml write failed")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func r1256DefFailFixture(t *testing.T, failWrites int) (*Manager, *r1256DefFailProvider, *int) {
|
||||
t.Helper()
|
||||
m, vp, _, notified, _ := r893FixtureA(t, true, 0)
|
||||
p := &r1256DefFailProvider{r893Provider: vp, failWrites: failWrites}
|
||||
m.SetStackProvider(p)
|
||||
return m, p, notified
|
||||
}
|
||||
|
||||
// RED-PROOF: remove the live write-back from the definition-write failure branch (restartStack alone) → one
|
||||
// definition write (the snapshot's), then the start → FAILS.
|
||||
func TestR1256_H4_AFailedDefinitionWriteStartsTheAppAtItsLiveDefinition(t *testing.T) {
|
||||
m, p, notified := r1256DefFailFixture(t, 1)
|
||||
_, err := m.ReconstituteFromOffsite(context.Background(), "immich", false)
|
||||
if err == nil {
|
||||
t.Fatal("a failed definition write must be surfaced")
|
||||
}
|
||||
if len(p.defs) != 2 || !strings.Contains(strings.Join(p.defs[1], " "), "postgres:18-alpine") {
|
||||
t.Fatalf("definition writes %v — want the snapshot's, then the LIVE one written back", p.defs)
|
||||
}
|
||||
if !p.fullStarted {
|
||||
t.Fatalf("the app must run again at its live definition — calls %v", p.calls)
|
||||
}
|
||||
// The write-back comes BEFORE the start.
|
||||
lastRecreate, lastStart := -1, -1
|
||||
for i, c := range p.calls {
|
||||
switch c {
|
||||
case "recreate":
|
||||
lastRecreate = i
|
||||
case "start":
|
||||
lastStart = i
|
||||
}
|
||||
}
|
||||
if lastRecreate > lastStart {
|
||||
t.Fatalf("the app was started before the live definition was written back — calls %v", p.calls)
|
||||
}
|
||||
if held, _ := m.RestoreHoldFor("immich"); held {
|
||||
t.Fatal("the live definition is back: the app must not be held")
|
||||
}
|
||||
if *notified != 0 {
|
||||
t.Errorf("hold notices %d, want 0", *notified)
|
||||
}
|
||||
}
|
||||
|
||||
// The write-back fails too: the stack dir may still hold the snapshot's compose, so the app is NOT started; it is
|
||||
// held (restore_mixed: the definition moved and could not be put back) and the operator is told once.
|
||||
// RED-PROOF: start the app when the write-back fails → fullStarted → FAILS.
|
||||
func TestR1256_H4_AFailedWriteBackHoldsTheApp(t *testing.T) {
|
||||
m, p, notified := r1256DefFailFixture(t, 2)
|
||||
_, err := m.ReconstituteFromOffsite(context.Background(), "immich", false)
|
||||
if err == nil {
|
||||
t.Fatal("a failed definition write must be surfaced")
|
||||
}
|
||||
if p.fullStarted {
|
||||
t.Fatalf("the app was STARTED although its live definition could not be written back — calls %v", p.calls)
|
||||
}
|
||||
held, _ := m.RestoreHoldFor("immich")
|
||||
if !held {
|
||||
t.Fatal("no hold: every start path would start the app on the snapshot's definition")
|
||||
}
|
||||
if k := m.HoldKind("immich"); k != settings.HoldReasonRestoreMixed {
|
||||
t.Errorf("hold kind %q, want %q", k, settings.HoldReasonRestoreMixed)
|
||||
}
|
||||
if *notified != 1 {
|
||||
t.Errorf("the operator must be told once, got %d", *notified)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- H5: a put-back whose stop fails must not go on; `volume rm -f` errors count; a failed put-back keeps its copy.
|
||||
|
||||
// r1256StopFailProvider fails the Nth StopStack (1-based).
|
||||
type r1256StopFailProvider struct {
|
||||
*r893Provider
|
||||
failAt, n int
|
||||
}
|
||||
|
||||
func (p *r1256StopFailProvider) StopStack(name string) error {
|
||||
p.n++
|
||||
err := p.r893Provider.StopStack(name)
|
||||
if p.n == p.failAt {
|
||||
return errors.New("compose down: timeout")
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// The off-site put-back: the replay fails after the snapshot's volume replaced the live one; the stop before the
|
||||
// put-back fails. Pouring the copy back under a running app could mix it with what the app writes, so nothing is
|
||||
// poured, the copy is KEPT, and the app is held.
|
||||
// RED-PROOF: restore the „(continuing)" WARN in putBackAfterFailedRestore → the copy is poured and the app started
|
||||
// → FAILS.
|
||||
func TestR1256_H5_APutBackWhoseStopFailsDoesNotPourAndKeepsTheCopy(t *testing.T) {
|
||||
m, vp, rig, _, notified, putBack := r893PutBackFixture(t)
|
||||
p := &r1256StopFailProvider{r893Provider: vp, failAt: 2} // 1 = the restore's stop, 2 = the put-back's
|
||||
m.SetStackProvider(p)
|
||||
_, err := m.ReconstituteFromOffsite(context.Background(), "immich", false)
|
||||
if err == nil {
|
||||
t.Fatal("a failed restore must be surfaced")
|
||||
}
|
||||
if p.n < 2 {
|
||||
t.Fatalf("the put-back's stop never ran (stops %d) — the fixture did not reach the put-back", p.n)
|
||||
}
|
||||
if len(rig.poured) != 0 || len(rig.removed) != 0 {
|
||||
t.Fatalf("the copy was poured (%v) / volumes removed (%v) after a FAILED stop", rig.poured, rig.removed)
|
||||
}
|
||||
if vp.fullStarted {
|
||||
t.Fatalf("the app was started after a failed put-back — calls %v", vp.calls)
|
||||
}
|
||||
if held, _ := m.RestoreHoldFor("immich"); !held {
|
||||
t.Fatal("a put-back that could not run must hold the app (D20)")
|
||||
}
|
||||
if left := undoCopies(t, rig.undoRoot); len(left) == 0 {
|
||||
t.Fatal("the copy must be KEPT after a failed put-back (D20)")
|
||||
}
|
||||
if *notified != 1 || *putBack != 0 {
|
||||
t.Errorf("hold notices %d (want 1), put-back notices %d (want 0)", *notified, *putBack)
|
||||
}
|
||||
}
|
||||
|
||||
// The live definition cannot be written back after the volumes were poured: the app is held and the copy is KEPT
|
||||
// (D20: a put-back that fails keeps its copy for support).
|
||||
// RED-PROOF: drop `c.keep = true` from the definition write-back failure branch → the copy is removed → FAILS.
|
||||
func TestR1256_H5_AFailedDefinitionWriteBackKeepsTheCopy(t *testing.T) {
|
||||
m, vp, rig, _, _, _ := r893PutBackFixture(t)
|
||||
// A version change, so a live definition exists to be written back; the second write (the write-back) fails.
|
||||
if err := os.WriteFile(vp.composePath, []byte(vtCompose("18")), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
vp.livePins = ParseComposeImages(vp.composePath)
|
||||
p := &r1256DefFailProvider{r893Provider: vp, failWrites: 0}
|
||||
p.failFrom = 2
|
||||
m.SetStackProvider(p)
|
||||
_, err := m.ReconstituteFromOffsite(context.Background(), "immich", false)
|
||||
if err == nil {
|
||||
t.Fatal("a failed restore must be surfaced")
|
||||
}
|
||||
if len(p.defs) < 2 {
|
||||
t.Fatalf("the write-back never ran: writes %v", p.defs)
|
||||
}
|
||||
if held, _ := m.RestoreHoldFor("immich"); !held {
|
||||
t.Fatal("a failed write-back must hold the app")
|
||||
}
|
||||
if left := undoCopies(t, rig.undoRoot); len(left) == 0 {
|
||||
t.Fatal("the copy must be KEPT after a failed put-back (D20)")
|
||||
}
|
||||
}
|
||||
|
||||
// The crash put-back (a restore cut by a power loss): the stop before pouring fails → an error (the guard then
|
||||
// holds the app and keeps the copy, R-1251 H2), nothing poured.
|
||||
// RED-PROOF: restore the „(continuing)" WARN in PutBackInterruptedRestore → nil error, the copy poured → FAILS.
|
||||
func TestR1256_H5_ACrashPutBackWhoseStopFailsRefuses(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
undo := filepath.Join(dir, "backups", "restore-undo", "app", "20261028T000000Z")
|
||||
mustWrite(t, filepath.Join(undo, "undo.json"), `{"stack":"app","copied":["app_data"],"definition":false}`)
|
||||
mustWrite(t, filepath.Join(undo, "app_data.tar"), "LIVE")
|
||||
poured := 0
|
||||
cp := NewCrashPutBack(func(string) error { return errors.New("compose down: timeout") }, nil, log.New(io.Discard, "", 0))
|
||||
cp.volReplay = func(string, string) (int, error) { poured++; return 1, nil }
|
||||
if err := cp.PutBackInterruptedRestore("app", undo); err == nil {
|
||||
t.Fatal("a put-back whose stop failed must return an error")
|
||||
}
|
||||
if poured != 0 {
|
||||
t.Fatal("the copy was poured back under an app that could not be stopped")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(undo, "app_data.tar")); err != nil {
|
||||
t.Fatal("the copy must stay on disk")
|
||||
}
|
||||
}
|
||||
|
||||
// restoreDockerVolumesFrom: a `docker volume rm -f` that fails (the volume is still in use) used to be ignored; the
|
||||
// `volume create` that follows is a no-op on an existing volume, so the tar was unpacked ON TOP of the old content —
|
||||
// a merge. Now that volume counts as failed and is not touched.
|
||||
// RED-PROOF: ignore the rm error again → the create and the tar run for the volume → FAILS.
|
||||
func TestR1256_H5_AVolumeThatCannotBeRemovedIsNotMergedInto(t *testing.T) {
|
||||
bin := t.TempDir()
|
||||
calls := filepath.Join(bin, "calls.log")
|
||||
stub := "#!/bin/sh\necho \"$@\" >> " + calls + "\n" +
|
||||
"case \"$*\" in 'compose version --short') echo v2.29.7;; 'volume rm -f app_busy') echo 'volume is in use' >&2; exit 1;; esac\nexit 0\n"
|
||||
if err := os.WriteFile(filepath.Join(bin, "docker"), []byte(stub), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH"))
|
||||
dump := t.TempDir()
|
||||
for _, n := range []string{"app_busy.tar", "app_free.tar"} {
|
||||
mustWrite(t, filepath.Join(dump, n), "tar")
|
||||
}
|
||||
m := &Manager{logger: log.New(io.Discard, "", 0)}
|
||||
n, err := m.restoreDockerVolumesFrom("app", dump)
|
||||
if err == nil || n != 1 {
|
||||
t.Fatalf("restored %d, err %v — want 1 and an error naming app_busy", n, err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "app_busy") {
|
||||
t.Errorf("the error must name the volume: %v", err)
|
||||
}
|
||||
b, _ := os.ReadFile(calls)
|
||||
for _, l := range strings.Split(string(b), "\n") {
|
||||
if (strings.HasPrefix(l, "volume create") || strings.HasPrefix(l, "run ")) && strings.Contains(l, "app_busy") {
|
||||
t.Fatalf("the volume that could not be removed was written into: %q", l)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---- H6: a second-drive mirror must not start between the page's refusal and the restore's own record.
|
||||
|
||||
// The page asked (nothing ran), then a mirror began, then the restore began: the begin itself refuses, records
|
||||
// nothing, and the mirror — asked under the same lock — refuses once a restore is recorded.
|
||||
// RED-PROOF: make TryBeginRestoreOp record the op without asking (the old BeginRestoreOp) → the restore begins beside
|
||||
// the mirror → FAILS.
|
||||
func TestR1256_H6_ARestoreCannotBeginBesideAMirrorThatStartedAfterTheCheck(t *testing.T) {
|
||||
tier2ResetForTest()
|
||||
defer tier2ResetForTest()
|
||||
m, _ := newOffboxManager(t)
|
||||
if busy, _ := m.RestoreBusy(); busy != "" {
|
||||
t.Fatalf("fixture: nothing runs, got %q", busy)
|
||||
}
|
||||
// the window: the page's check passed; now a mirror begins
|
||||
if !m.beginTier2Run("immich") {
|
||||
t.Fatal("fixture: the mirror must begin while no restore is recorded")
|
||||
}
|
||||
if busy, app := m.TryBeginRestoreOp("restore", "immich"); busy != RestoreBusyMirror || app != "immich" {
|
||||
t.Fatalf("the restore began beside a running mirror: busy=%q app=%q", busy, app)
|
||||
}
|
||||
if m.RestoreStatus().Running {
|
||||
t.Fatal("a refused begin left a restore op recorded")
|
||||
}
|
||||
tier2CopyEnd("immich")
|
||||
if busy, _ := m.TryBeginRestoreOp("restore", "immich"); busy != "" {
|
||||
t.Fatalf("with the mirror done the restore must begin, got %q", busy)
|
||||
}
|
||||
if m.beginTier2Run("immich") {
|
||||
t.Fatal("a mirror began while the restore of the same app is recorded")
|
||||
}
|
||||
// a second restore is refused while the first runs; the backup run refuses too
|
||||
if busy, _ := m.TryBeginRestoreOp("restore", "other"); busy != RestoreBusyRestore {
|
||||
t.Fatalf("a second restore began beside the first: %q", busy)
|
||||
}
|
||||
m.EndRestoreOp(true, "ok")
|
||||
if err := m.AcquireRunningForTest(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if busy, _ := m.TryBeginRestoreOp("restore", "immich"); busy != RestoreBusyBackup {
|
||||
t.Fatalf("a restore began beside a backup run: %q", busy)
|
||||
}
|
||||
m.ReleaseRunningForTest()
|
||||
}
|
||||
|
||||
// ---- H8: an unused restore copy is cleaned up; a copy a hold or a pending put-back needs is never touched.
|
||||
|
||||
func r1256Copy(t *testing.T, m *Manager, app string, age time.Duration, now time.Time) string {
|
||||
t.Helper()
|
||||
dir := filepath.Join(restoreUndoDir(m.namespaceRoot(m.systemDataPath), app), now.Add(-age).UTC().Format(restoreCopyStampLayout))
|
||||
mustWrite(t, filepath.Join(dir, app+"_data.tar"), "LIVE")
|
||||
return dir
|
||||
}
|
||||
|
||||
func r1256Exists(p string) bool { _, err := os.Stat(p); return err == nil }
|
||||
|
||||
// RED-PROOF: drop the hold check (or the marker check) from SweepUnusedRestoreCopies → the held / named copy is removed
|
||||
// → FAILS; drop the age check → the young copy is removed → FAILS.
|
||||
func TestR1256_H8_OnlyAnUnusedOldCopyIsRemoved(t *testing.T) {
|
||||
m, sett := newOffboxManager(t)
|
||||
now := time.Now()
|
||||
old := 20 * 24 * time.Hour
|
||||
unused := r1256Copy(t, m, "unused", old, now)
|
||||
young := r1256Copy(t, m, "young", 24*time.Hour, now)
|
||||
held := r1256Copy(t, m, "held", old, now)
|
||||
waiting := r1256Copy(t, m, "waiting", old, now)
|
||||
named := r1256Copy(t, m, "named", old, now)
|
||||
if err := sett.SetRestoreHold(settings.RestoreHold{Stack: "held", At: now.Format(time.RFC3339), Reason: settings.HoldReasonRestoreMixed}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := sett.SetRestoreHold(settings.RestoreHold{Stack: "waiting", At: now.Format(time.RFC3339), Reason: settings.HoldReasonPutBackPending}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// a restore cut by a power loss: the marker (no longer active after a failed recovery's End) still names the copy
|
||||
if err := m.appStop.Begin("unit-restore:named", ReasonUnitRestore, []string{"named"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := m.appStop.SetUndo("named", named); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
got := m.SweepUnusedRestoreCopies(now)
|
||||
if len(got) != 1 || got[0] != unused || r1256Exists(unused) {
|
||||
t.Fatalf("removed %v — want exactly the unused old copy %s", got, unused)
|
||||
}
|
||||
for _, keep := range []string{young, held, waiting, named} {
|
||||
if !r1256Exists(keep) {
|
||||
t.Errorf("a needed copy was removed: %s", keep)
|
||||
}
|
||||
}
|
||||
if r1256Exists(filepath.Dir(unused)) {
|
||||
t.Error("the emptied per-app folder must go too")
|
||||
}
|
||||
}
|
||||
|
||||
// While a restore runs nothing is removed; a marker that cannot be read stops the sweep.
|
||||
// RED-PROOF: drop the RestoreBusy check → the old copy goes during a restore → FAILS.
|
||||
func TestR1256_H8_NothingIsRemovedDuringARestoreOrWithAnUnreadableMarker(t *testing.T) {
|
||||
m, _ := newOffboxManager(t)
|
||||
now := time.Now()
|
||||
c := r1256Copy(t, m, "app", 20*24*time.Hour, now)
|
||||
m.BeginRestoreOp("restore", "other")
|
||||
if got := m.SweepUnusedRestoreCopies(now); len(got) != 0 || !r1256Exists(c) {
|
||||
t.Fatalf("removed %v during a restore", got)
|
||||
}
|
||||
m.EndRestoreOp(true, "ok")
|
||||
mustWrite(t, m.appStop.path, "{not json")
|
||||
if got := m.SweepUnusedRestoreCopies(now); len(got) != 0 || !r1256Exists(c) {
|
||||
t.Fatalf("removed %v with an unreadable marker", got)
|
||||
}
|
||||
if err := os.Remove(m.appStop.path); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := m.SweepUnusedRestoreCopies(now); len(got) != 1 {
|
||||
t.Fatalf("control: with nothing in the way the old copy goes, removed %v", got)
|
||||
}
|
||||
}
|
||||
@@ -224,14 +224,8 @@ func (m *Manager) restoreDockerVolumesFrom(stackName, dumpDir string) (int, erro
|
||||
|
||||
m.logger.Printf("[INFO] [backup] Restoring Docker volume %s for %s", volName, stackName)
|
||||
|
||||
// Remove the existing volume. `rm -f` of a missing volume succeeds, so an error means the volume EXISTS and
|
||||
// could not be removed (still in use). R-1256 H5: it used to be ignored — the `volume create` below is a no-op
|
||||
// on an existing volume, and the tar was unpacked ON TOP of the old content (a merge). That volume fails now.
|
||||
if out, err := dockerexec.Command("docker", "volume", "rm", "-f", volName).CombinedOutput(); err != nil {
|
||||
m.logger.Printf("[ERROR] [backup] Failed to remove volume %s before the restore: %s — %v (not written into)", volName, strings.TrimSpace(string(out)), err)
|
||||
failed = append(failed, volName)
|
||||
continue
|
||||
}
|
||||
// Remove existing volume (ignore errors — may not exist)
|
||||
dockerexec.Command("docker", "volume", "rm", "-f", volName).Run()
|
||||
|
||||
// Create fresh volume — WITH the labels compose gives its own volumes (R-658, v0.268.0). A bare
|
||||
// `volume create` made a volume the update's undo (until v0.267.0) could not find, and one
|
||||
|
||||
@@ -1,129 +0,0 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-1256 H8 (review round 2, HUNT-1): a restore copy (<namespace>/backups/restore-undo/<app>/<stamp>) that nothing
|
||||
// will ever use again used to stay on the drive for good. A copy is KEPT on purpose in three states — a put-back that
|
||||
// failed (the app is held, D20), a put-back waiting for its drive (the putback_pending hold, R-1251) and a restore cut
|
||||
// by a power loss (the app-stop marker names it, D22) — and for 30 minutes after a restore while the box watches for a
|
||||
// stable start (R-1270). Every one of those is still needed. A copy outside all of them — the app runs again, or was
|
||||
// removed, and nothing names the copy — is disk the household cannot see or use.
|
||||
//
|
||||
// The rule (safest reading, decided in the 2026-10-28 night run; operator may change the age): a copy is removed only
|
||||
// when ALL of these hold, and the sweep removes NOTHING when it cannot answer one of them:
|
||||
// - it is older than restoreCopyUnusedAge (its folder's stamp; the folder's time when the stamp does not parse);
|
||||
// - the app has no restore hold of any kind (held apps keep every copy — support decides);
|
||||
// - the app-stop marker does not name the app (a pending put-back), and the marker is readable;
|
||||
// - no restore op is in flight on the box (a new restore's copy and the stable-start watcher are left alone).
|
||||
//
|
||||
// Run every 6 hours from main.go (TestR1256_H8_TheSweepIsWiredAtStartup). Pinned by TestR1256_H8_*.
|
||||
|
||||
// restoreCopyUnusedAge is how long an unused copy stays for support after its restore.
|
||||
const restoreCopyUnusedAge = 14 * 24 * time.Hour
|
||||
|
||||
// restoreCopyStampLayout is copyLiveVolumes' folder name.
|
||||
const restoreCopyStampLayout = "20060102T150405Z"
|
||||
|
||||
// SweepUnusedRestoreCopies removes the restore copies no hold, no pending put-back and no running restore needs, and
|
||||
// older than restoreCopyUnusedAge. Returns the folders removed.
|
||||
func (m *Manager) SweepUnusedRestoreCopies(now time.Time) []string {
|
||||
if busy, _ := m.RestoreBusy(); busy == RestoreBusyRestore {
|
||||
m.logger.Printf("[INFO] [backup] restore-copy sweep skipped: a restore runs")
|
||||
return nil
|
||||
}
|
||||
named, ok := m.appStopNamedApps()
|
||||
if !ok {
|
||||
m.logger.Printf("[WARN] [backup] restore-copy sweep skipped: the app-stop marker cannot be read — a copy it names must not go")
|
||||
return nil
|
||||
}
|
||||
var removed []string
|
||||
for _, nsRoot := range m.primaryUnitRoots() {
|
||||
base := filepath.Join(nsRoot, "backups", "restore-undo")
|
||||
apps, err := os.ReadDir(base)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for _, a := range apps {
|
||||
if !a.IsDir() {
|
||||
continue
|
||||
}
|
||||
app := a.Name()
|
||||
if named[app] {
|
||||
continue
|
||||
}
|
||||
if m.settings == nil {
|
||||
return removed // no hold store: a held app cannot be told apart
|
||||
}
|
||||
if _, held := m.settings.GetRestoreHold(app); held {
|
||||
continue
|
||||
}
|
||||
appDir := filepath.Join(base, app)
|
||||
copies, err := os.ReadDir(appDir)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
sort.Slice(copies, func(i, j int) bool { return copies[i].Name() < copies[j].Name() })
|
||||
for _, c := range copies {
|
||||
if !c.IsDir() {
|
||||
continue
|
||||
}
|
||||
dir := filepath.Join(appDir, c.Name())
|
||||
at, err := time.Parse(restoreCopyStampLayout, c.Name())
|
||||
if err != nil {
|
||||
fi, sErr := os.Stat(dir)
|
||||
if sErr != nil {
|
||||
continue
|
||||
}
|
||||
at = fi.ModTime()
|
||||
}
|
||||
if now.Sub(at) < restoreCopyUnusedAge {
|
||||
continue
|
||||
}
|
||||
if err := os.RemoveAll(dir); err != nil {
|
||||
m.logger.Printf("[WARN] [backup] %s: the unused restore copy %s could not be removed: %v", app, dir, err)
|
||||
continue
|
||||
}
|
||||
m.logger.Printf("[INFO] [backup] %s: the unused restore copy %s (from %s) is removed — no hold, no pending put-back, older than %s (R-1256)",
|
||||
app, dir, at.UTC().Format(time.RFC3339), restoreCopyUnusedAge)
|
||||
removed = append(removed, dir)
|
||||
}
|
||||
_ = os.Remove(appDir) // the per-app folder, when it is now empty
|
||||
}
|
||||
}
|
||||
return removed
|
||||
}
|
||||
|
||||
// appStopNamedApps reads the app-stop marker without side effects: the apps of an ACTIVE marker and every app its Undo
|
||||
// names. ok=false when the marker exists and cannot be read — the caller then touches nothing.
|
||||
func (m *Manager) appStopNamedApps() (map[string]bool, bool) {
|
||||
named := map[string]bool{}
|
||||
if m.appStop == nil || m.appStop.path == "" {
|
||||
return named, true
|
||||
}
|
||||
data, err := os.ReadFile(m.appStop.path)
|
||||
if os.IsNotExist(err) {
|
||||
return named, true
|
||||
}
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
var mk AppStopMarker
|
||||
if err := json.Unmarshal(data, &mk); err != nil {
|
||||
return nil, false
|
||||
}
|
||||
for app := range mk.Undo {
|
||||
named[app] = true
|
||||
}
|
||||
if mk.Active {
|
||||
for _, app := range mk.Stacks {
|
||||
named[app] = true
|
||||
}
|
||||
}
|
||||
return named, true
|
||||
}
|
||||
@@ -195,11 +195,9 @@ func (p *CrashPutBack) PutBackInterruptedRestore(stack, undoDir string) error {
|
||||
if j.Stack != stack {
|
||||
return fmt.Errorf("the restore copy at %s belongs to %q, not %q", undoDir, j.Stack, stack)
|
||||
}
|
||||
// R-1256 H5: never pour the copy back under an app that could not be stopped — a refusal here is a failed
|
||||
// put-back: the guard holds the app and keeps the copy (R-1251 H2).
|
||||
if p.stop != nil {
|
||||
if err := p.stop(stack); err != nil {
|
||||
return fmt.Errorf("stopping %s before the put-back: %w", stack, err)
|
||||
p.logger.Printf("[WARN] [appstop] %s: stopping before the put-back failed: %v (continuing)", stack, err)
|
||||
}
|
||||
}
|
||||
mgr := &Manager{logger: p.logger, volumeReplayFrom: p.volReplay, undoVolumeRemoveFn: p.volRemove}
|
||||
|
||||
@@ -1,171 +0,0 @@
|
||||
// Package holdreport tells the hub which apps are held (D53, felhom.eu `08` §10).
|
||||
//
|
||||
// A held app is stopped on purpose and stays stopped until a person acts. Until D53 the hub knew nothing about it:
|
||||
// the crash-loop stop sent one event, the other holds sent none, and demo-hp's Kimai sat held for 11 days unseen
|
||||
// (R-1291). The legacy report is frozen, so the event channel carries the holds:
|
||||
//
|
||||
// - app_held when a hold appears or changes (stack, reason, since, the hold's sentence);
|
||||
// - app_hold_cleared when it is gone — also when it went while the controller was down, because the set last
|
||||
// reported is kept in settings (ReportedHolds), not in memory;
|
||||
// - every hold still in place is sent again once a day and at every controller start, because an event can be
|
||||
// lost. The hub treats a repeat as idempotent.
|
||||
//
|
||||
// ONE OBSERVER, NOT A HOOK PER WRITER. Seven code paths write a hold and four clear one. A send in each of them is
|
||||
// the shape where the one path nobody found is the next silent hold; this package reads the hold store itself, so a
|
||||
// new writer is covered the day it is added. Both events are severity info: the hub mails nothing on receipt. Its
|
||||
// reminder after the grace per reason is the operator's mail, and it never goes to the household (D53).
|
||||
//
|
||||
// Pinned by holdreport_test.go.
|
||||
package holdreport
|
||||
|
||||
import (
|
||||
"log"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// Event types — the hub's allowedEventTypes carries both (the coupling rule).
|
||||
const (
|
||||
EventHeld = "app_held"
|
||||
EventCleared = "app_hold_cleared"
|
||||
)
|
||||
|
||||
// ResendEvery is how often every hold still in place is sent again.
|
||||
const ResendEvery = 24 * time.Hour
|
||||
|
||||
// Store is what the reporter needs of the settings.
|
||||
type Store interface {
|
||||
ListRestoreHolds() []settings.RestoreHold
|
||||
GetReportedHolds() map[string]string
|
||||
SetReportedHolds(map[string]string) error
|
||||
}
|
||||
|
||||
// Push sends one event to the hub (notify.Notifier.PushEvent).
|
||||
type Push func(eventType, severity, message string, details interface{})
|
||||
|
||||
// Details is the payload of both events.
|
||||
type Details struct {
|
||||
StackName string `json:"stack_name"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
Since string `json:"since,omitempty"`
|
||||
Why string `json:"why,omitempty"`
|
||||
Repeat bool `json:"repeat,omitempty"` // a daily or start-up re-send, not a new hold
|
||||
}
|
||||
|
||||
// Reporter compares the hold store with what was last reported. Tick is called by the scheduler.
|
||||
type Reporter struct {
|
||||
store Store
|
||||
push Push
|
||||
logger *log.Logger
|
||||
lastFull time.Time
|
||||
// installed (R-1298) — nil = every app counts as installed.
|
||||
installed func(stack string) bool
|
||||
skipLogged map[string]bool
|
||||
}
|
||||
|
||||
// New builds a reporter. The first Tick re-sends every hold (a controller start).
|
||||
func New(store Store, push Push, logger *log.Logger) *Reporter {
|
||||
return &Reporter{store: store, push: push, logger: logger}
|
||||
}
|
||||
|
||||
func key(h settings.RestoreHold) string { return h.WireReason() + "|" + h.At }
|
||||
|
||||
// SetInstalled wires the „is this app installed?" check (R-1298). nil = every app counts as installed.
|
||||
func (r *Reporter) SetInstalled(f func(stack string) bool) { r.installed = f }
|
||||
|
||||
// installHold is a hold that belongs to an app's INSTALL (R-704): a removal ends it. A restore hold (restore_failed,
|
||||
// restore_mixed) does not — a person clears it, so it stays reported whatever the app's state (R-379).
|
||||
func installHold(h settings.RestoreHold) bool {
|
||||
switch h.Reason {
|
||||
case settings.HoldReasonUnhealthyStop, settings.HoldReasonUpdateFailed, settings.HoldReasonPutBackPending:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Tick sends what changed since the last tick, and every hold again when a day has passed.
|
||||
func (r *Reporter) Tick(now time.Time) {
|
||||
if r.store == nil || r.push == nil {
|
||||
return
|
||||
}
|
||||
all := r.store.ListRestoreHolds()
|
||||
sort.Slice(all, func(i, j int) bool { return all[i].Stack < all[j].Stack })
|
||||
// R-1298: an install's hold of an app that is not installed any more (removed outside the product) is not a held
|
||||
// app — it is skipped, and reported cleared if it was reported before. Logged once per stack per process.
|
||||
holds := all[:0:0]
|
||||
for _, h := range all {
|
||||
if r.installed != nil && installHold(h) && !r.installed(h.Stack) {
|
||||
if r.skipLogged == nil {
|
||||
r.skipLogged = map[string]bool{}
|
||||
}
|
||||
if !r.skipLogged[h.Stack] {
|
||||
r.skipLogged[h.Stack] = true
|
||||
r.logf("[INFO] [holdreport] %s: a %s hold of an app that is not installed — not reported (R-1298)", h.Stack, h.WireReason())
|
||||
}
|
||||
continue
|
||||
}
|
||||
holds = append(holds, h)
|
||||
}
|
||||
prev := r.store.GetReportedHolds()
|
||||
full := r.lastFull.IsZero() || now.Sub(r.lastFull) >= ResendEvery
|
||||
|
||||
cur := make(map[string]string, len(holds))
|
||||
sent, cleared := 0, 0
|
||||
for _, h := range holds {
|
||||
k := key(h)
|
||||
cur[h.Stack] = k
|
||||
known, seen := prev[h.Stack]
|
||||
if seen && known == k && !full {
|
||||
continue
|
||||
}
|
||||
repeat := seen && known == k
|
||||
r.push(EventHeld, "info", "App "+h.Stack+" is held: "+h.Why(),
|
||||
Details{StackName: h.Stack, Reason: h.WireReason(), Since: h.At, Why: h.Why(), Repeat: repeat})
|
||||
sent++
|
||||
if !repeat {
|
||||
r.logf("[INFO] [holdreport] %s held (%s since %s) — reported to the hub", h.Stack, h.WireReason(), h.At)
|
||||
}
|
||||
}
|
||||
gone := make([]string, 0)
|
||||
for stack := range prev {
|
||||
if _, ok := cur[stack]; !ok {
|
||||
gone = append(gone, stack)
|
||||
}
|
||||
}
|
||||
sort.Strings(gone)
|
||||
for _, stack := range gone {
|
||||
r.push(EventCleared, "info", "App "+stack+" is no longer held", Details{StackName: stack})
|
||||
cleared++
|
||||
r.logf("[INFO] [holdreport] %s hold cleared — reported to the hub", stack)
|
||||
}
|
||||
if full {
|
||||
r.lastFull = now
|
||||
r.logf("[DEBUG] [holdreport] full re-send: %d hold(s) sent, %d cleared", sent, cleared)
|
||||
}
|
||||
if !sameMap(prev, cur) {
|
||||
if err := r.store.SetReportedHolds(cur); err != nil {
|
||||
// Not fatal: the next tick sends the same difference again, which the hub reads as idempotent.
|
||||
r.logf("[WARN] [holdreport] could not save the reported holds: %v — the next tick repeats the difference", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func sameMap(a, b map[string]string) bool {
|
||||
if len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
for k, v := range a {
|
||||
if b[k] != v {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (r *Reporter) logf(format string, args ...interface{}) {
|
||||
if r.logger != nil {
|
||||
r.logger.Printf(format, args...)
|
||||
}
|
||||
}
|
||||
@@ -1,211 +0,0 @@
|
||||
package holdreport
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
type fakeStore struct {
|
||||
holds map[string]settings.RestoreHold
|
||||
reported map[string]string
|
||||
}
|
||||
|
||||
func (f *fakeStore) ListRestoreHolds() []settings.RestoreHold {
|
||||
out := []settings.RestoreHold{}
|
||||
for _, h := range f.holds {
|
||||
out = append(out, h)
|
||||
}
|
||||
return out
|
||||
}
|
||||
func (f *fakeStore) GetReportedHolds() map[string]string {
|
||||
out := map[string]string{}
|
||||
for k, v := range f.reported {
|
||||
out[k] = v
|
||||
}
|
||||
return out
|
||||
}
|
||||
func (f *fakeStore) SetReportedHolds(m map[string]string) error { f.reported = m; return nil }
|
||||
|
||||
type sent struct {
|
||||
typ, sev string
|
||||
d Details
|
||||
}
|
||||
|
||||
func rec(out *[]sent) Push {
|
||||
return func(t, sev, msg string, d interface{}) { *out = append(*out, sent{t, sev, d.(Details)}) }
|
||||
}
|
||||
|
||||
var t0 = time.Date(2026, 10, 26, 18, 0, 0, 0, time.UTC)
|
||||
|
||||
// The report field (D53): a new hold is sent once as app_held with its stack, reason and since, severity info (the hub
|
||||
// mails nothing on receipt; its reminder after the grace is the mail).
|
||||
func TestD53_ANewHoldIsReportedWithStackReasonAndSince(t *testing.T) {
|
||||
st := &fakeStore{holds: map[string]settings.RestoreHold{
|
||||
"kimai": {Stack: "kimai", At: "2026-10-14T08:00:00Z", Reason: settings.HoldReasonUnhealthyStop, UnhealthyKind: "crash_loop"},
|
||||
}}
|
||||
var out []sent
|
||||
r := New(st, rec(&out), nil)
|
||||
r.Tick(t0)
|
||||
if len(out) != 1 || out[0].typ != EventHeld || out[0].sev != "info" {
|
||||
t.Fatalf("want one info app_held, got %+v", out)
|
||||
}
|
||||
d := out[0].d
|
||||
if d.StackName != "kimai" || d.Reason != "unhealthy_stop" || d.Since != "2026-10-14T08:00:00Z" || d.Why == "" {
|
||||
t.Fatalf("payload: %+v", d)
|
||||
}
|
||||
if st.reported["kimai"] != "unhealthy_stop|2026-10-14T08:00:00Z" {
|
||||
t.Fatalf("reported record: %v", st.reported)
|
||||
}
|
||||
// The next tick inside a day sends nothing.
|
||||
out = nil
|
||||
r.Tick(t0.Add(time.Minute))
|
||||
if len(out) != 0 {
|
||||
t.Fatalf("an unchanged hold was sent again inside a day: %+v", out)
|
||||
}
|
||||
}
|
||||
|
||||
// An event can be lost: every hold still in place is sent again after a day, marked as a repeat.
|
||||
func TestD53_AHoldIsSentAgainEveryDay(t *testing.T) {
|
||||
st := &fakeStore{holds: map[string]settings.RestoreHold{"a": {Stack: "a", At: "2026-10-20T00:00:00Z", Reason: settings.HoldReasonUpdateFailed}}}
|
||||
var out []sent
|
||||
r := New(st, rec(&out), nil)
|
||||
r.Tick(t0)
|
||||
out = nil
|
||||
r.Tick(t0.Add(ResendEvery))
|
||||
if len(out) != 1 || out[0].typ != EventHeld || !out[0].d.Repeat {
|
||||
t.Fatalf("want one repeat app_held after a day, got %+v", out)
|
||||
}
|
||||
}
|
||||
|
||||
// A hold cleared by a person ends it: one app_hold_cleared, then nothing.
|
||||
func TestD53_AClearedHoldSendsClearedOnce(t *testing.T) {
|
||||
st := &fakeStore{holds: map[string]settings.RestoreHold{"a": {Stack: "a", At: "2026-10-20T00:00:00Z", Reason: settings.HoldReasonRestoreMixed}}}
|
||||
var out []sent
|
||||
r := New(st, rec(&out), nil)
|
||||
r.Tick(t0)
|
||||
delete(st.holds, "a")
|
||||
out = nil
|
||||
r.Tick(t0.Add(time.Minute))
|
||||
if len(out) != 1 || out[0].typ != EventCleared || out[0].d.StackName != "a" || out[0].sev != "info" {
|
||||
t.Fatalf("want one info app_hold_cleared for a, got %+v", out)
|
||||
}
|
||||
if len(st.reported) != 0 {
|
||||
t.Fatalf("the reported record still names the cleared hold: %v", st.reported)
|
||||
}
|
||||
out = nil
|
||||
r.Tick(t0.Add(2 * time.Minute))
|
||||
r.Tick(t0.Add(ResendEvery + time.Hour))
|
||||
if len(out) != 0 {
|
||||
t.Fatalf("a cleared hold was reported again: %+v", out)
|
||||
}
|
||||
}
|
||||
|
||||
// A hold cleared while the controller was down is still reported cleared: the record lives in settings.
|
||||
func TestD53_AHoldClearedWhileDownIsReportedAtStart(t *testing.T) {
|
||||
st := &fakeStore{holds: map[string]settings.RestoreHold{}, reported: map[string]string{"gone": "update_failed|2026-10-01T00:00:00Z"}}
|
||||
var out []sent
|
||||
New(st, rec(&out), nil).Tick(t0)
|
||||
if len(out) != 1 || out[0].typ != EventCleared || out[0].d.StackName != "gone" {
|
||||
t.Fatalf("want app_hold_cleared for gone at start, got %+v", out)
|
||||
}
|
||||
}
|
||||
|
||||
// A hold whose reason changes (a put-back that failed becomes a mixed restore) is a new hold for the hub.
|
||||
func TestD53_AChangedReasonIsReportedAgain(t *testing.T) {
|
||||
st := &fakeStore{holds: map[string]settings.RestoreHold{"a": {Stack: "a", At: "2026-10-20T00:00:00Z", Reason: settings.HoldReasonPutBackPending}}}
|
||||
var out []sent
|
||||
r := New(st, rec(&out), nil)
|
||||
r.Tick(t0)
|
||||
st.holds["a"] = settings.RestoreHold{Stack: "a", At: "2026-10-20T01:00:00Z", Reason: settings.HoldReasonRestoreMixed}
|
||||
out = nil
|
||||
r.Tick(t0.Add(time.Minute))
|
||||
if len(out) != 1 || out[0].typ != EventHeld || out[0].d.Reason != "restore_mixed" || out[0].d.Repeat {
|
||||
t.Fatalf("want a new (not repeat) app_held restore_mixed, got %+v", out)
|
||||
}
|
||||
}
|
||||
|
||||
// The legacy empty reason is a restore that failed with its rollback: the hub reads it as restore_failed.
|
||||
func TestD53_TheEmptyReasonIsRestoreFailedOnTheWire(t *testing.T) {
|
||||
st := &fakeStore{holds: map[string]settings.RestoreHold{"db": {Stack: "db", At: "2026-10-20T00:00:00Z"}}}
|
||||
var out []sent
|
||||
New(st, rec(&out), nil).Tick(t0)
|
||||
if len(out) != 1 || out[0].d.Reason != "restore_failed" {
|
||||
t.Fatalf("want reason restore_failed, got %+v", out)
|
||||
}
|
||||
}
|
||||
|
||||
// The seam is wired: main builds the reporter on the real settings and notifier and ticks it from the scheduler. A
|
||||
// reporter nobody ticks is the inert-seam shape (four instances in this project). A commented-out line does not count.
|
||||
func TestD53_TheReporterIsWiredInMain(t *testing.T) {
|
||||
raw, err := os.ReadFile(filepath.Join("..", "..", "cmd", "controller", "main.go"))
|
||||
if err != nil {
|
||||
t.Fatalf("cannot read main.go: %v", err)
|
||||
}
|
||||
live := []string{}
|
||||
for _, l := range strings.Split(string(raw), "\n") {
|
||||
if !strings.HasPrefix(strings.TrimSpace(l), "//") {
|
||||
live = append(live, l)
|
||||
}
|
||||
}
|
||||
src := strings.Join(live, "\n")
|
||||
for _, want := range []string{"holdreport.New(sett, notifier.PushEvent, logger)", `sched.Every("hold-report"`, "holdReporter.Tick(", "holdReporter.SetInstalled("} {
|
||||
if !strings.Contains(src, want) {
|
||||
t.Errorf("main.go lacks %q — the hold report is not wired", want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The reported record survives a restart: it is what lets a hold cleared while the controller was down be reported.
|
||||
func TestD53_TheReportedRecordSurvivesAReload(t *testing.T) {
|
||||
p := filepath.Join(t.TempDir(), "settings.json")
|
||||
s, err := settings.Load(p, log.New(io.Discard, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.SetReportedHolds(map[string]string{"kimai": "unhealthy_stop|2026-10-14T08:00:00Z"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s2, err := settings.Load(p, log.New(io.Discard, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := s2.GetReportedHolds()["kimai"]; got != "unhealthy_stop|2026-10-14T08:00:00Z" {
|
||||
t.Fatalf("after a reload the record reads %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// R-1298: an install's hold (crash-loop stop, failed update, waiting put-back) of an app that is NOT installed any more is
|
||||
// not reported — on 9202 four such holds from a hand removal would have mailed the operator weekly about apps that do not
|
||||
// exist. A restore hold stays reported whatever the app's state (R-379: a person clears it). A hold reported earlier and
|
||||
// now skipped is reported cleared. COMPANION RED-PROOF: without the installed check all three holds are reported.
|
||||
func TestR1298_OnlyHoldsOfInstalledAppsAreReported(t *testing.T) {
|
||||
st := &fakeStore{holds: map[string]settings.RestoreHold{
|
||||
"gone": {Stack: "gone", At: "2026-09-24T12:16:24Z", Reason: settings.HoldReasonUnhealthyStop},
|
||||
"gonedb": {Stack: "gonedb", At: "2026-09-24T12:16:24Z", Reason: settings.HoldReasonRestoreFailed},
|
||||
"present": {Stack: "present", At: "2026-10-26T17:37:52Z", Reason: settings.HoldReasonUpdateFailed},
|
||||
}, reported: map[string]string{"gone": "unhealthy_stop|2026-09-24T12:16:24Z"}}
|
||||
var out []sent
|
||||
r := New(st, rec(&out), nil)
|
||||
r.SetInstalled(func(stack string) bool { return stack == "present" })
|
||||
r.Tick(t0)
|
||||
got := map[string]string{}
|
||||
for _, s := range out {
|
||||
got[s.d.StackName] = s.typ
|
||||
}
|
||||
if got["gone"] != EventCleared {
|
||||
t.Errorf("the hold of an app that is not installed must be reported cleared, got %q", got["gone"])
|
||||
}
|
||||
if got["gonedb"] != EventHeld {
|
||||
t.Errorf("a restore hold must stay reported even when the app is gone, got %q", got["gonedb"])
|
||||
}
|
||||
if got["present"] != EventHeld {
|
||||
t.Errorf("an installed app's hold must be reported, got %q", got["present"])
|
||||
}
|
||||
}
|
||||
@@ -1258,9 +1258,6 @@
|
||||
"err.appexport.a_z_kotet_adata_hianyzik_a": "the data of volume %q is missing from the package",
|
||||
"err.appexport.jelszo_hibas_vagy_a_fajl_serult": "wrong password, or the file is damaged",
|
||||
"err.appexport.ket_adatkonyvtar_azonos_nevvel_vegzodik_es": "two data folders end in the same name (%q: %s and %s) — the package could not tell them apart",
|
||||
"err.appexport.kulonleges_fajl": "%s is a special file (a pipe, socket or device) that cannot be saved — the export stopped; nothing was left out silently",
|
||||
"err.appexport.import_foglalt": "a package cannot be loaded right now: another operation is working on the box (%s). Wait until it finishes and try again.",
|
||||
"err.appexport.hdd_path_hibas": "the data-drive path in the package is not valid (“%s”) — the import did not start, nothing changed",
|
||||
"err.appexport.visszafejtes_sikertelen": "decryption failed: %s",
|
||||
"err.backup.whole_restore_space": "There is not enough room on the drive to bring the files back: %.1f GB needed, %.1f GB free. Nothing was changed.",
|
||||
"err.backup.whole_restore_no_proven_copy": "The second drive holds no checked copy that can bring the app back together with its files. Nothing was changed.",
|
||||
@@ -2666,13 +2663,11 @@
|
||||
"api.escrow.too_many_attempts": "Too many failed attempts, try again in 1 minute.",
|
||||
"api.escrow.stage_failed": "Preparing the remote backup password for escrow failed — the process did not start. Try again.",
|
||||
"api.escrow.already_running": "A code is already being created — wait until it finishes.",
|
||||
"api.escrow.recovery_offer_open": "You cannot create a new recovery code right now: first open your old off-site backups with your existing code (Backups page). A new code would replace the key of the old backups.",
|
||||
"api.escrow.code_not_shown": "The code was created but not shown — for security it cannot be fetched again. Start the process again: the new code replaces the old one.",
|
||||
"api.escrow.no_active_job": "There is no active recovery process — start creating the code first.",
|
||||
"api.escrow.confirm_expired": "The time to type the new code back has run out — your previous code still works. Start the process again.",
|
||||
"api.escrow.confirm_lost": "The new code no longer waits for confirmation — your previous code still works. Start the process again.",
|
||||
"api.escrow.confirm_store_failed": "The new code could not be saved — your previous code still works. Try again.",
|
||||
"api.escrow.confirm_store_unknown": "We cannot tell yet whether the new code was saved — the server did not answer in time. Keep both codes (the old one and the new one) and try again.",
|
||||
"api.escrow.confirm_words_mismatch": "The words do not match — check the code you wrote down.",
|
||||
"api.share.form_invalid": "Invalid form — reload the page.",
|
||||
"api.share.too_many_attempts": "Too many failed attempts, try again in 1 minute",
|
||||
|
||||
@@ -1253,9 +1253,6 @@
|
||||
"err.appexport.a_z_kotet_adata_hianyzik_a": "a(z) %q kötet adata hiányzik a csomagból",
|
||||
"err.appexport.jelszo_hibas_vagy_a_fajl_serult": "jelszó hibás vagy a fájl sérült",
|
||||
"err.appexport.ket_adatkonyvtar_azonos_nevvel_vegzodik_es": "két adatkönyvtár azonos névvel végződik (%q: %s és %s) — a csomag nem tudná megkülönböztetni őket",
|
||||
"err.appexport.kulonleges_fajl": "a(z) %s különleges fájl (cső, foglalat vagy eszköz), ezt nem lehet menteni — az exportálás leállt, semmi nem maradt ki csendben",
|
||||
"err.appexport.import_foglalt": "most nem tölthetsz vissza csomagot: egy másik művelet dolgozik a dobozon (%s). Várd meg, míg befejeződik, és próbáld újra.",
|
||||
"err.appexport.hdd_path_hibas": "a csomag adatmeghajtó-útvonala hibás („%s”) — a betöltés el sem indult, semmi nem változott",
|
||||
"err.appexport.visszafejtes_sikertelen": "visszafejtés sikertelen: %s",
|
||||
"err.backup.whole_restore_space": "Nincs elég hely a meghajtón a fájlok visszaállításához: %.1f GB kellene, %.1f GB szabad. Semmi nem változott.",
|
||||
"err.backup.whole_restore_no_proven_copy": "A második meghajtón nincs olyan ellenőrzött másolat, amely az alkalmazást a fájljaival együtt vissza tudná hozni. Semmi nem változott.",
|
||||
@@ -2654,13 +2651,11 @@
|
||||
"api.escrow.too_many_attempts": "Túl sok sikertelen próbálkozás, próbáld újra 1 perc múlva.",
|
||||
"api.escrow.stage_failed": "A távoli mentés jelszavának letéti előkészítése nem sikerült — a folyamat nem indult el. Próbáld újra.",
|
||||
"api.escrow.already_running": "Egy kódkészítés már folyamatban van — várd meg, míg befejeződik.",
|
||||
"api.escrow.recovery_offer_open": "Most nem készíthetsz új helyreállítási kódot: előbb nyisd meg a régi, házon kívüli mentéseidet a meglévő kódoddal (Mentések oldal). Egy új kód lecserélné a régi mentések kulcsát.",
|
||||
"api.escrow.code_not_shown": "A kód létrejött, de nem lett megjelenítve — biztonsági okból újra nem kérhető le. Indítsd újra a folyamatot: az új kód a régit érvényteleníti.",
|
||||
"api.escrow.no_active_job": "Nincs aktív helyreállítási folyamat — előbb indítsd el a kódkészítést.",
|
||||
"api.escrow.confirm_expired": "Lejárt az idő az új kód visszaírására — a régi kódod továbbra is érvényes. Indítsd újra a folyamatot.",
|
||||
"api.escrow.confirm_lost": "Az új kód már nem vár megerősítésre — a régi kódod továbbra is érvényes. Indítsd újra a folyamatot.",
|
||||
"api.escrow.confirm_store_failed": "Az új kódot nem sikerült elmenteni — a régi kódod továbbra is érvényes. Próbáld újra.",
|
||||
"api.escrow.confirm_store_unknown": "Nem tudjuk biztosan, hogy az új kód mentése sikerült-e — a központ nem válaszolt időben. Őrizd meg mindkét kódot (a régit és az újat is), és próbáld újra.",
|
||||
"api.escrow.confirm_words_mismatch": "A megadott szavak nem egyeznek — ellenőrizd a felírt kódot.",
|
||||
"api.share.form_invalid": "Érvénytelen űrlap — töltsd újra az oldalt.",
|
||||
"api.share.too_many_attempts": "Túl sok sikertelen próbálkozás, próbáld újra 1 perc múlva",
|
||||
|
||||
@@ -22,15 +22,11 @@ import (
|
||||
// Pinned image tags — NEVER ":latest" (a floating tag breaks reproducible golden bakes and lets the
|
||||
// deployed version drift). Verified to resolve on Docker Hub before baking. cloudflared follows Cloudflare's security
|
||||
// notices (felhom.eu `01` „Tunnel placement"): 2026.10.3 since the notice of 2026-10-20 — before 2026.10.2 the image
|
||||
// trusts eight eMudhra (emSign) intermediates with weak keys; 2026.10.4 since 2026-10-27 (VULN-176088, the metrics
|
||||
// server's debug endpoints). traefik v3.7.16 and the file manager 1.5.8-stable since 2026-10-27 for their security
|
||||
// notices (felhom.eu R-1304, R-1305; 1.5.7-stable is skipped — it broke existing TOTP sign-ins). Each new image reaches
|
||||
// a running box by the reconcile at start: TestEnsureTraefik_ANewImageRecreatesTraefik,
|
||||
// TestEnsureCloudflared_ANewImageRecreatesTheTunnel, TestReconcileFileBrowserImage_MovesOnlyTheImage.
|
||||
// trusts eight eMudhra (emSign) intermediates with weak keys.
|
||||
const (
|
||||
TraefikImage = "traefik:v3.7.16"
|
||||
CloudflaredImage = "cloudflare/cloudflared:2026.10.4"
|
||||
FileBrowserImage = "gtstef/filebrowser:1.5.8-stable"
|
||||
TraefikImage = "traefik:v3.7.13"
|
||||
CloudflaredImage = "cloudflare/cloudflared:2026.10.3"
|
||||
FileBrowserImage = "gtstef/filebrowser:1.5.6-stable"
|
||||
// FileBrowserImportMount is the in-container mount point NAME for the canonical drop-zone
|
||||
// (R-75): the bind lands at /srv/<this>. ASCII and space-free on purpose — it appears in a
|
||||
// container path, in the generated compose, and (percent-encoded) in the deep-link URL.
|
||||
|
||||
@@ -1,7 +0,0 @@
|
||||
# upstream.txt — where each pin in infra.go publishes its security notices (the security watch, felhom.eu R-1303).
|
||||
# One line per `*Image` const in infra.go: the const name, then its sources. The grammar lives in ONE place:
|
||||
# felhom.eu/scripts/security_watch_inventory.py. A const with no line here refuses the push (gate `upstream`).
|
||||
TraefikImage github:traefik/traefik
|
||||
CloudflaredImage github:cloudflare/cloudflared feed:cloudflare-blog vulnnotes:cloudflared
|
||||
FileBrowserImage github:gtsteffaniak/filebrowser
|
||||
SambaImage self
|
||||
@@ -163,8 +163,6 @@ type UpdateDetails struct {
|
||||
type AppDetails struct {
|
||||
StackName string `json:"stack_name,omitempty"`
|
||||
DisplayName string `json:"display_name,omitempty"`
|
||||
// Source (app_oom only, R-528): what saw the kill — "docker" (State.OOMKilled) or "cgroup" (the host's kernel counter).
|
||||
Source string `json:"source,omitempty"`
|
||||
}
|
||||
|
||||
// CrossDriveDetails holds structured data for cross-drive backup events.
|
||||
@@ -806,12 +804,6 @@ func (n *Notifier) NotifyAppStartFailures(apps []AppRunState) {
|
||||
// that stays OOM-marked does not repeat. "warning" — the hub vocabulary (R-329). Operator-only
|
||||
// hub-side (hub >= v0.114.0 registers it): the household sees the dashboard tag.
|
||||
func (n *Notifier) NotifyAppOOM(stack, container, startedAt string, kills int64, memLimit, peak string) {
|
||||
n.NotifyAppOOMFrom(stack, container, startedAt, kills, memLimit, peak, "")
|
||||
}
|
||||
|
||||
// NotifyAppOOMFrom is NotifyAppOOM with the source that saw the kill (R-528: "docker" or "cgroup"), carried in the
|
||||
// event's details. One event per container run, whichever source saw it.
|
||||
func (n *Notifier) NotifyAppOOMFrom(stack, container, startedAt string, kills int64, memLimit, peak, source string) {
|
||||
key := container + "|" + startedAt
|
||||
n.mu.Lock()
|
||||
if n.oomSeen == nil {
|
||||
@@ -824,7 +816,7 @@ func (n *Notifier) NotifyAppOOMFrom(stack, container, startedAt string, kills in
|
||||
if first {
|
||||
n.emit("app_oom", "warning",
|
||||
fmt.Sprintf("Alkalmazás memóriája elfogyott: %s (%s) — egy folyamatát a memóriakorlát leállította", stack, container),
|
||||
AppDetails{StackName: stack, DisplayName: container, Source: source})
|
||||
AppDetails{StackName: stack, DisplayName: container})
|
||||
}
|
||||
if storm {
|
||||
n.logger.Printf("[ERROR] [notify] %s: container %s OOM STORM — %d kills in %d min (limit %s, peak %s)",
|
||||
|
||||
@@ -232,10 +232,6 @@ type Settings struct {
|
||||
// out-of-memory storm (RFC3339). Survives the hold being lifted by Start, so a second stop within
|
||||
// 24 h is known to be a second one.
|
||||
UnhealthyStops map[string]string `json:"unhealthy_stops,omitempty"`
|
||||
// ReportedHolds (D53, `08` §10) — the holds last told to the hub as app_held: stack → „reason|at". The hold
|
||||
// reporter (internal/holdreport) compares it with RestoreHolds to send app_held for a new or changed hold and
|
||||
// app_hold_cleared for one that is gone — also one cleared while the controller was down, because it is kept here.
|
||||
ReportedHolds map[string]string `json:"reported_holds,omitempty"`
|
||||
|
||||
// Cross-drive restic repo password (auto-generated on first use)
|
||||
CrossDriveResticPassword string `json:"cross_drive_restic_password,omitempty"`
|
||||
@@ -2002,35 +1998,6 @@ const (
|
||||
HoldReasonPutBackPending = "putback_pending"
|
||||
)
|
||||
|
||||
// WireReason is the hold's reason as the hub reads it (D53): the empty legacy value is named restore_failed.
|
||||
func (h RestoreHold) WireReason() string {
|
||||
if h.Reason == HoldReasonRestoreFailed {
|
||||
return "restore_failed"
|
||||
}
|
||||
return h.Reason
|
||||
}
|
||||
|
||||
// Why says which operation put a hold in place and what lifts it — one sentence per reason (R-1286). English,
|
||||
// operator-facing: the controller's own -list-restore-holds output and the hub's „Held apps" list (D53) both use it.
|
||||
func (h RestoreHold) Why() string {
|
||||
switch h.Reason {
|
||||
case HoldReasonUpdateFailed:
|
||||
return "held after a failed update (" + h.At + ") — restore it from its backup to start it"
|
||||
case HoldReasonUnhealthyStop:
|
||||
kind := h.UnhealthyKind
|
||||
if kind == "" {
|
||||
kind = "unhealthy"
|
||||
}
|
||||
return "stopped by the box for a " + strings.ReplaceAll(kind, "_", " ") + " (" + h.At + ") — the household's Start gives it one more try"
|
||||
case HoldReasonRestoreMixed:
|
||||
return "held after an off-site restore failed its database replay (" + h.At + ") — clear the hold to start it"
|
||||
case HoldReasonPutBackPending:
|
||||
return "held until the pre-restore copy is poured back (" + h.At + ") — the put-back starts it"
|
||||
default:
|
||||
return "held after a failed restore whose rollback also failed (" + h.At + ") — clear the hold to start it"
|
||||
}
|
||||
}
|
||||
|
||||
// SetRestoreHold records a hold. Modelled on SetDisconnected: a condition, plus what it is holding.
|
||||
func (s *Settings) SetRestoreHold(h RestoreHold) error {
|
||||
s.mu.Lock()
|
||||
@@ -2064,32 +2031,6 @@ func (s *Settings) ListRestoreHolds() []RestoreHold {
|
||||
return out
|
||||
}
|
||||
|
||||
// GetReportedHolds returns a copy of the holds last reported to the hub (D53): stack → „reason|at".
|
||||
func (s *Settings) GetReportedHolds() map[string]string {
|
||||
s.mu.RLock()
|
||||
defer s.mu.RUnlock()
|
||||
out := make(map[string]string, len(s.ReportedHolds))
|
||||
for k, v := range s.ReportedHolds {
|
||||
out[k] = v
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// SetReportedHolds replaces the record of the holds reported to the hub (D53). An empty map removes the field.
|
||||
func (s *Settings) SetReportedHolds(m map[string]string) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if len(m) == 0 {
|
||||
s.ReportedHolds = nil
|
||||
} else {
|
||||
s.ReportedHolds = make(map[string]string, len(m))
|
||||
for k, v := range m {
|
||||
s.ReportedHolds[k] = v
|
||||
}
|
||||
}
|
||||
return s.save()
|
||||
}
|
||||
|
||||
// ClearRestoreHold removes a hold. Returns false when there was none, so a caller can tell "cleared"
|
||||
// from "there was nothing to clear" rather than reporting success either way.
|
||||
func (s *Settings) ClearRestoreHold(stack string) (bool, error) {
|
||||
|
||||
@@ -3,8 +3,6 @@ package stacks
|
||||
import (
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/util"
|
||||
)
|
||||
|
||||
// Desired-state values for AppConfig.DesiredState (R-166, decision D-b).
|
||||
@@ -59,8 +57,7 @@ func (m *Manager) SetDesiredState(name, desired string) error {
|
||||
|
||||
stack, ok := m.GetStack(name)
|
||||
if !ok {
|
||||
// a KIND (404 at the API) and the household's sentence — not a save failure
|
||||
return util.MsgErrorf(ErrStackNotFound, "err.stacks.alkalmazas_nem_talalhato", name)
|
||||
return fmt.Errorf("stack %q not found", name)
|
||||
}
|
||||
stackDir := filepath.Dir(stack.ComposePath)
|
||||
|
||||
|
||||
@@ -162,42 +162,6 @@ func TestEnsureTraefik_ReconcilesARunningTraefik(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A controller update that changes only traefik's IMAGE recreates the running traefik on the new image (the v3.7.16
|
||||
// security bump of 2026-10-27 rides this, felhom.eu R-1304: traefik.yml is unchanged, only the compose's `image:` line
|
||||
// moves). The control: the same render again is a no-op.
|
||||
func TestEnsureTraefik_ANewImageRecreatesTraefik(t *testing.T) {
|
||||
state := stubDocker(t)
|
||||
touch(t, filepath.Join(state, "running-traefik"), "")
|
||||
m, calls := tunnelTestManager(t, "owner@example.com")
|
||||
dir := t.TempDir()
|
||||
cur, err := infra.RenderTraefik(infra.TraefikData{ACMEEmail: "owner@example.com", Tunnel: true})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old := strings.Replace(cur["docker-compose.yml"].Content, "image: "+infra.TraefikImage, "image: traefik:v3.7.13", 1)
|
||||
if old == cur["docker-compose.yml"].Content {
|
||||
t.Fatal("the render has no `image:` line to age — the test would prove nothing")
|
||||
}
|
||||
for name, f := range cur {
|
||||
touch(t, filepath.Join(dir, name), f.Content)
|
||||
}
|
||||
touch(t, filepath.Join(dir, "docker-compose.yml"), old)
|
||||
|
||||
if err := m.ensureTraefik(dir, true); err != nil {
|
||||
t.Fatalf("ensureTraefik: %v", err)
|
||||
}
|
||||
cmp, _ := os.ReadFile(filepath.Join(dir, "docker-compose.yml"))
|
||||
if !strings.Contains(string(cmp), "image: "+infra.TraefikImage) {
|
||||
t.Fatalf("the compose must carry %s:\n%s", infra.TraefikImage, cmp)
|
||||
}
|
||||
if len(*calls) != 1 || (*calls)[0].args != "up -d --force-recreate" {
|
||||
t.Fatalf("want ONE `up -d --force-recreate` (traefik on the new image), got %+v", *calls)
|
||||
}
|
||||
if err := m.ensureTraefik(dir, true); err != nil || len(*calls) != 1 {
|
||||
t.Fatalf("an unchanged image must not recreate traefik again; err %v calls %+v", err, *calls)
|
||||
}
|
||||
}
|
||||
|
||||
// A rewrite that would DROP the certificate resolver the running file has is refused (no e-mail in the config).
|
||||
func TestEnsureTraefik_RefusesToDropTheCertResolver(t *testing.T) {
|
||||
state := stubDocker(t)
|
||||
|
||||
@@ -217,31 +217,10 @@ func readKeptMarker(dir string) *KeptMarker {
|
||||
}
|
||||
|
||||
// liveDriveBinds is every drive folder an INSTALLED app binds — never kept data.
|
||||
//
|
||||
// R-1220 — FAIL CLOSED ON AN UNREADABLE app.yaml. A parse error gives a nil AppConfig, so ScanStacks reads the app as
|
||||
// „not installed"; before this, its live folder was then listed as kept data and offered for deletion. An app whose
|
||||
// app.yaml EXISTS but cannot be read counts as installed: its HDD_PATH is unknown, so every folder its compose would bind
|
||||
// on ANY of the given drives is live. One WARN per app per listing names it. Pinned by
|
||||
// TestR1220_AnUnreadableAppYamlKeepsTheFolderLive.
|
||||
func (m *Manager) liveDriveBinds(drives []string) []string {
|
||||
func (m *Manager) liveDriveBinds() []string {
|
||||
var out []string
|
||||
for _, st := range m.GetStacks() {
|
||||
if st.ComposePath == "" {
|
||||
continue
|
||||
}
|
||||
if !st.Deployed || st.AppConfig == nil {
|
||||
if !st.Deployed && !appYAMLUnreadable(filepath.Dir(st.ComposePath)) {
|
||||
continue // not installed: no app.yaml, or one that reads as not deployed
|
||||
}
|
||||
m.logger.Printf("[WARN] [stacks] kept: %s has an app.yaml that cannot be read — it counts as INSTALLED, so its drive folders are never listed as kept data (R-1220)", st.Name)
|
||||
for _, d := range drives {
|
||||
if d = strings.TrimSpace(d); d == "" {
|
||||
continue
|
||||
}
|
||||
for _, p := range ParseComposeHDDMounts(st.ComposePath, filepath.Clean(d)) {
|
||||
out = append(out, filepath.Clean(p))
|
||||
}
|
||||
}
|
||||
continue
|
||||
}
|
||||
hdd := strings.TrimSpace(st.AppConfig.Env["HDD_PATH"])
|
||||
@@ -255,15 +234,6 @@ func (m *Manager) liveDriveBinds(drives []string) []string {
|
||||
return out
|
||||
}
|
||||
|
||||
// appYAMLUnreadable is true when the stack folder holds an app.yaml that LoadAppConfig cannot turn into a config — the
|
||||
// R-1220 case. A missing app.yaml is false (a template that was never installed, or a removed app).
|
||||
func appYAMLUnreadable(stackDir string) bool {
|
||||
if _, err := os.Stat(filepath.Join(stackDir, "app.yaml")); err != nil {
|
||||
return false
|
||||
}
|
||||
return LoadAppConfig(stackDir) == nil
|
||||
}
|
||||
|
||||
func overlaps(a, b string) bool {
|
||||
sep := string(filepath.Separator)
|
||||
return a == b || strings.HasPrefix(a, b+sep) || strings.HasPrefix(b, a+sep)
|
||||
@@ -314,7 +284,7 @@ var sizeFn = getDirSizeBytes
|
||||
// ListKept lists every kept item on the given drive roots: the dated folders under <drive>/kept/<app>/,
|
||||
// and each <drive>/appdata/<x> that holds something and that no installed app binds. Sorted newest first.
|
||||
func (m *Manager) ListKept(drives []string) []KeptItem {
|
||||
live := m.liveDriveBinds(drives)
|
||||
live := m.liveDriveBinds()
|
||||
var out []KeptItem
|
||||
seen := map[string]bool{}
|
||||
for _, d := range drives {
|
||||
|
||||
@@ -284,8 +284,6 @@ type Manager struct {
|
||||
// R-514: last OOM scan (oom.go), for the dashboard.
|
||||
oomMu sync.Mutex
|
||||
oomCache map[string][]string
|
||||
// hostOOMKills (R-528) is the host agent's per-container OOM counter; nil = Docker's flag only (SetHostOOMKills).
|
||||
hostOOMKills func() map[string]int64
|
||||
// R-513: FileBrowser admin-password seams (filebrowser_password.go); nil/"" in production.
|
||||
fbHTTP fbHTTPDo
|
||||
fbBaseURL string
|
||||
|
||||
@@ -27,21 +27,6 @@ type OOMContainer struct {
|
||||
Kills int64
|
||||
// MemLimit / Peak are the cgroup's memory.max and memory.peak, as "<n>M" (or "max"); "" unread.
|
||||
MemLimit, Peak string
|
||||
// Source names what flagged it: OOMSourceDocker (State.OOMKilled), OOMSourceCgroup (R-528 — only the host's
|
||||
// per-container kernel counter saw it; Docker's flag stayed false), or "docker,cgroup" (both).
|
||||
Source string
|
||||
}
|
||||
|
||||
const (
|
||||
OOMSourceDocker = "docker"
|
||||
OOMSourceCgroup = "cgroup"
|
||||
)
|
||||
|
||||
// SetHostOOMKills wires the host-side OOM counter (R-528): container id → the kernel's oom_kill for the container's
|
||||
// CURRENT run (its cgroup scope is new at each start), read by the host's agent. nil, or a nil answer (no agent, an
|
||||
// agent without the route, a failed call) = Docker's flag only, as before. INIT-ONLY (main.go).
|
||||
func (m *Manager) SetHostOOMKills(fn func() map[string]int64) {
|
||||
m.hostOOMKills = fn
|
||||
}
|
||||
|
||||
// ScanOOMKilled inspects the containers of every deployed, running-ish stack in ONE docker call and
|
||||
@@ -67,45 +52,20 @@ func (m *Manager) ScanOOMKilled() ([]OOMContainer, error) {
|
||||
return nil, nil
|
||||
}
|
||||
sort.Strings(names)
|
||||
args := append([]string{"inspect", "-f", "{{.Name}}|{{.State.OOMKilled}}|{{.State.StartedAt}}|{{.Id}}"}, names...)
|
||||
args := append([]string{"inspect", "-f", "{{.Name}}|{{.State.OOMKilled}}|{{.State.StartedAt}}"}, names...)
|
||||
out, err := m.execCommand("docker", args...)
|
||||
if err != nil && strings.TrimSpace(out) == "" {
|
||||
return nil, err
|
||||
}
|
||||
// R-528: the host's counter, asked once per scan (an HTTPS call to the agent); nil = Docker's flag only.
|
||||
var host map[string]int64
|
||||
if m.hostOOMKills != nil {
|
||||
host = m.hostOOMKills()
|
||||
}
|
||||
var found []OOMContainer
|
||||
hostKills := map[int]int64{}
|
||||
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
|
||||
f := strings.SplitN(strings.TrimSpace(line), "|", 4)
|
||||
if len(f) < 3 {
|
||||
continue
|
||||
}
|
||||
id := ""
|
||||
if len(f) == 4 {
|
||||
id = strings.TrimSpace(f[3])
|
||||
}
|
||||
hk, counted := host[id]
|
||||
source := ""
|
||||
switch {
|
||||
case f[1] == "true" && id != "" && counted && hk > 0:
|
||||
source = OOMSourceDocker + "," + OOMSourceCgroup // both saw it
|
||||
case f[1] == "true":
|
||||
source = OOMSourceDocker
|
||||
case id != "" && counted && hk > 0:
|
||||
source = OOMSourceCgroup
|
||||
default:
|
||||
f := strings.SplitN(strings.TrimSpace(line), "|", 3)
|
||||
if len(f) != 3 || f[1] != "true" {
|
||||
continue
|
||||
}
|
||||
cname := strings.TrimPrefix(f[0], "/")
|
||||
if st, ok := owner[cname]; ok {
|
||||
if counted && id != "" {
|
||||
hostKills[len(found)] = hk
|
||||
}
|
||||
found = append(found, OOMContainer{Stack: st, Container: cname, StartedAt: f[2], Kills: -1, Source: source})
|
||||
found = append(found, OOMContainer{Stack: st, Container: cname, StartedAt: f[2], Kills: -1})
|
||||
}
|
||||
}
|
||||
// R-636: only for the containers already flagged — one `docker exec` each, and the flagged set is
|
||||
@@ -115,9 +75,6 @@ func (m *Manager) ScanOOMKilled() ([]OOMContainer, error) {
|
||||
out, _ := m.execCommand("docker", "exec", found[i].Container, "cat",
|
||||
"/sys/fs/cgroup/memory.events", "/sys/fs/cgroup/memory.max", "/sys/fs/cgroup/memory.peak")
|
||||
found[i].Kills, found[i].MemLimit, found[i].Peak = parseCgroupMemory(out)
|
||||
if hk, ok := hostKills[i]; ok && hk > found[i].Kills {
|
||||
found[i].Kills = hk // the host's count is the kernel's own; the in-container read can be -1 (no cat)
|
||||
}
|
||||
}
|
||||
m.setOOMCache(found)
|
||||
return found, nil
|
||||
|
||||
@@ -1,36 +0,0 @@
|
||||
package stacks
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1220: an installed app whose app.yaml cannot be read must count as INSTALLED (fail closed). Before the fix a parse
|
||||
// error gave a nil AppConfig, so the app looked „not installed", its live drive folder was listed as kept data — and
|
||||
// the household's Delete was offered on it. Red-proved against the code before the fix (the folder was listed).
|
||||
func TestR1220_AnUnreadableAppYamlKeepsTheFolderLive(t *testing.T) {
|
||||
m, drive := keptManager(t)
|
||||
app := filepath.Join(filepath.Dir(m.stacks["cloudapp"].ComposePath))
|
||||
// An app.yaml that exists but does not parse (a half-written file, a disk error).
|
||||
must(t, os.WriteFile(filepath.Join(app, "app.yaml"), []byte("deployed: true\nenv: [this is: not: yaml\n"), 0o600))
|
||||
must(t, m.ScanStacks())
|
||||
items := m.ListKept([]string{drive})
|
||||
for _, it := range items {
|
||||
if it.App == "cloudapp" || it.Path == filepath.Join(drive, "appdata", "cloudapp") {
|
||||
t.Fatalf("an app with an unreadable app.yaml had its live folder listed as kept: %+v", it)
|
||||
}
|
||||
}
|
||||
if _, err := m.DeleteKept([]string{drive}, filepath.Join(drive, "appdata", "cloudapp")); err == nil {
|
||||
t.Fatal("Delete was allowed on the folder of an app whose app.yaml cannot be read")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(drive, "appdata/cloudapp/user1/file.txt")); err != nil {
|
||||
t.Fatalf("the live folder's file is gone: %v", err)
|
||||
}
|
||||
// The control: with NO app.yaml at all (a real leftover), the folder IS listed.
|
||||
must(t, os.Remove(filepath.Join(app, "app.yaml")))
|
||||
must(t, m.ScanStacks())
|
||||
if items := m.ListKept([]string{drive}); len(items) != 1 || items[0].App != "cloudapp" {
|
||||
t.Fatalf("a real leftover (no app.yaml) is not listed: %+v", items)
|
||||
}
|
||||
}
|
||||
@@ -1,84 +0,0 @@
|
||||
package stacks
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-528 — inside a Felhom LXC guest Docker can leave State.OOMKilled false after a memory-limit kill (measured
|
||||
// 2026-09-15/16). The host's agent reads the kernel's own per-container counter (GET /guest/oom-kills); a counter above
|
||||
// 0 for the RUNNING container's id is an OOM kill in this run, whatever Docker says. The scan flags it with
|
||||
// Source "cgroup" and the host's count, and the existing app_oom line (once per run) fires for it.
|
||||
|
||||
const r528ID = "4f5e6d7c8b9a0f1e2d3c4b5a69788796a5b4c3d2e1f00112233445566778899a"
|
||||
|
||||
// RED-PROOF: ignore the host counter in ScanOOMKilled (only `OOMKilled == true` flags) → the killed container is not
|
||||
// seen → FAILS.
|
||||
func TestR528_AHostCounterAboveZeroIsAnOOMKillWhateverDockerSays(t *testing.T) {
|
||||
var execd []string
|
||||
m := &Manager{logger: log.New(io.Discard, "", 0), stacks: map[string]*Stack{
|
||||
"hog": {Name: "hog", Deployed: true, Containers: []ContainerInfo{{Name: "hog", State: StateRunning}}},
|
||||
"bookstack": {Name: "bookstack", Deployed: true, Containers: []ContainerInfo{{Name: "bookstack", State: StateRunning}}},
|
||||
}}
|
||||
m.execFn = func(name string, args ...string) (string, error) {
|
||||
if args[0] == "exec" {
|
||||
execd = append(execd, args[1])
|
||||
return "oom_kill 1\n67108864\n67100000\n", nil
|
||||
}
|
||||
if !strings.Contains(strings.Join(args, " "), "{{.Id}}") {
|
||||
t.Errorf("the inspect must ask for the container id: %v", args)
|
||||
}
|
||||
return "/bookstack|false|t0|" + strings.Repeat("0", 64) + "\n/hog|false|2026-10-28T23:00:00Z|" + r528ID + "\n", nil
|
||||
}
|
||||
m.SetHostOOMKills(func() map[string]int64 {
|
||||
return map[string]int64{r528ID: 2, strings.Repeat("0", 64): 0}
|
||||
})
|
||||
ooms, err := m.ScanOOMKilled()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(ooms) != 1 || ooms[0].Stack != "hog" || ooms[0].Container != "hog" {
|
||||
t.Fatalf("the host-counted kill was not seen: %+v", ooms)
|
||||
}
|
||||
if ooms[0].Source != OOMSourceCgroup || ooms[0].Kills != 2 || ooms[0].StartedAt != "2026-10-28T23:00:00Z" {
|
||||
t.Fatalf("want source cgroup, the host's count 2 and the run's start: %+v", ooms[0])
|
||||
}
|
||||
if ooms[0].MemLimit != "64M" {
|
||||
t.Errorf("limit/peak are still read for the flagged container: %+v", ooms[0])
|
||||
}
|
||||
if len(execd) != 1 || execd[0] != "hog" {
|
||||
t.Errorf("exec must reach the flagged container only, got %v", execd)
|
||||
}
|
||||
}
|
||||
|
||||
// Docker's flag alone still flags (Source docker); no host source (an old agent, 9202 without an agent) = today's
|
||||
// behaviour; a host counter of 0 never flags.
|
||||
func TestR528_WithoutAHostCounterTheScanIsAsBefore(t *testing.T) {
|
||||
m := &Manager{logger: log.New(io.Discard, "", 0), stacks: map[string]*Stack{
|
||||
"a": {Name: "a", Deployed: true, Containers: []ContainerInfo{{Name: "a", State: StateRunning}}},
|
||||
"b": {Name: "b", Deployed: true, Containers: []ContainerInfo{{Name: "b", State: StateRunning}}},
|
||||
}}
|
||||
m.execFn = func(name string, args ...string) (string, error) {
|
||||
if args[0] == "exec" {
|
||||
return "oom_kill 3\n", nil
|
||||
}
|
||||
return "/a|true|s1|" + strings.Repeat("a", 64) + "\n/b|false|s2|" + strings.Repeat("b", 64) + "\n", nil
|
||||
}
|
||||
ooms, _ := m.ScanOOMKilled()
|
||||
if len(ooms) != 1 || ooms[0].Stack != "a" || ooms[0].Source != OOMSourceDocker || ooms[0].Kills != 3 {
|
||||
t.Fatalf("no host source: %+v", ooms)
|
||||
}
|
||||
m.SetHostOOMKills(func() map[string]int64 { return map[string]int64{strings.Repeat("b", 64): 0} })
|
||||
ooms, _ = m.ScanOOMKilled()
|
||||
if len(ooms) != 1 || ooms[0].Stack != "a" {
|
||||
t.Fatalf("a host counter of 0 flagged a container: %+v", ooms)
|
||||
}
|
||||
// Both saw it: the source names both, the host's count wins over a lower in-container read.
|
||||
m.SetHostOOMKills(func() map[string]int64 { return map[string]int64{strings.Repeat("a", 64): 7} })
|
||||
ooms, _ = m.ScanOOMKilled()
|
||||
if len(ooms) != 1 || ooms[0].Source != OOMSourceDocker+","+OOMSourceCgroup || ooms[0].Kills != 7 {
|
||||
t.Fatalf("both sources: %+v", ooms)
|
||||
}
|
||||
}
|
||||
@@ -21,12 +21,9 @@ import (
|
||||
// 1. reads the app's global upload limit exactly as the app stored it, and writes it to settings.json
|
||||
// (`upwatch.line_test_hold`) BEFORE it touches the app;
|
||||
// 2. sets the app's upload to HoldValue (1 KiB/s);
|
||||
// 3. waits until every held app REPORTS its own upload near zero (waitDrained) — qBittorrent 5.2.4 kept sending at
|
||||
// full speed for ~14 s after its limit fell to 1 KiB/s (measured on demo-hp 2026-10-27, felhom.eu
|
||||
// `audits/linetest-2026-10-27/`), so the agent's quiet check, 5–15 s after the hold, refused every test;
|
||||
// 4. asks the agent for a test on request (POST /host/line-test/run: 5 s settle, 10 s quiet check, three 10-second
|
||||
// 3. asks the agent for a test on request (POST /host/line-test/run: 5 s settle, 10 s quiet check, three 10-second
|
||||
// runs 5 s apart — about 55 s);
|
||||
// 5. puts the old limit back — the household's own limit included, exactly as it was — and clears the record.
|
||||
// 4. puts the old limit back — the household's own limit included, exactly as it was — and clears the record.
|
||||
//
|
||||
// Whatever fails after step 1, the limit is put back first. A controller that stops between 1 and 4 finds the record
|
||||
// at start (RecoverHold) and puts it back then. If the app holds anything but the test value when the limit goes
|
||||
@@ -63,8 +60,6 @@ type Holdable interface {
|
||||
WriteLimit(ctx context.Context, l HeldLimit) (HeldLimit, error)
|
||||
// HoldValue is the test value in prev's slot: very low, never zero (zero is „no limit").
|
||||
HoldValue(prev HeldLimit) HeldLimit
|
||||
// UploadRate is the app's own upload speed now, B/s — read while held, to know when it has really slowed.
|
||||
UploadRate(ctx context.Context) (int64, error)
|
||||
}
|
||||
|
||||
// LineRunResult is the agent's answer to a test on request.
|
||||
@@ -206,76 +201,6 @@ func (s *Service) releaseAll(ctx context.Context, held []Holdable, hs []LineHold
|
||||
}
|
||||
}
|
||||
|
||||
// The drain wait (step 3). The agent refuses a test when the uplink sends 2 Mbit/s or more 5–15 s after the call, so the
|
||||
// controller asks only when every held app reports its own upload at or below drainQuietBps (0.13 Mbit/s) on
|
||||
// drainQuietReadings readings in a row, drainPoll apart. It never waits longer than drainMax, and asks then anyway (the
|
||||
// agent judges the line). An app whose speed cannot be read gets a plain drainFallback wait instead — the measured
|
||||
// drain was ~14 s. Pinned by linehold_r1250_drain_test.go.
|
||||
const (
|
||||
drainPoll = 2 * time.Second
|
||||
drainMax = 90 * time.Second
|
||||
drainFallback = 20 * time.Second
|
||||
drainQuietBps = 16 * 1024
|
||||
drainQuietReadings = 2
|
||||
)
|
||||
|
||||
// drainSleep waits d; false when ctx ends first. A variable so the tests can count the waiting instead of doing it.
|
||||
var drainSleep = func(ctx context.Context, d time.Duration) bool {
|
||||
t := time.NewTimer(d)
|
||||
defer t.Stop()
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return false
|
||||
case <-t.C:
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
// waitDrained returns when every held app has slowed down to the test value, or when it cannot tell (see above).
|
||||
func (s *Service) waitDrained(ctx context.Context, apps []Holdable) {
|
||||
var waited time.Duration
|
||||
quiet := 0
|
||||
for {
|
||||
var worst int64
|
||||
var readErr error
|
||||
for _, app := range apps {
|
||||
r, err := app.UploadRate(ctx)
|
||||
if err != nil {
|
||||
readErr = fmt.Errorf("%s: %v", app.Stack(), err)
|
||||
break
|
||||
}
|
||||
if r > worst {
|
||||
worst = r
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case readErr != nil:
|
||||
logx.Infof(s.Log, "[upwatch] line test: cannot read the upload speed (%v) — waiting %s before asking the agent", readErr, drainFallback)
|
||||
if rest := drainFallback - waited; rest > 0 {
|
||||
drainSleep(ctx, rest)
|
||||
}
|
||||
return
|
||||
case worst <= drainQuietBps:
|
||||
quiet++
|
||||
if quiet >= drainQuietReadings {
|
||||
logx.Infof(s.Log, "[upwatch] line test: the held app(s) slowed down after %s — asking the agent", waited)
|
||||
return
|
||||
}
|
||||
default:
|
||||
quiet = 0
|
||||
logx.Debugf(s.Log, "[upwatch] line test: held, still sending %d B/s after %s", worst, waited)
|
||||
}
|
||||
if waited >= drainMax {
|
||||
logx.Infof(s.Log, "[upwatch] line test: the held app(s) still send %d B/s after %s — asking the agent anyway", worst, waited)
|
||||
return
|
||||
}
|
||||
if !drainSleep(ctx, drainPoll) {
|
||||
return
|
||||
}
|
||||
waited += drainPoll
|
||||
}
|
||||
}
|
||||
|
||||
// SeedLineTestNow runs the hold once when the agent waits for quiet and a torrent app is running (main calls it every
|
||||
// 15 minutes, before QBLimitNow). EVERY running seeder is held — one left seeding keeps the uplink busy and the agent
|
||||
// refuses the test „not quiet" every time (R-1261). A no-op otherwise.
|
||||
@@ -330,7 +255,6 @@ func (s *Service) SeedLineTestNow(ctx context.Context) {
|
||||
// Deferred: a panic in the call puts the limits back too. A killed process does not — that is RecoverHold's.
|
||||
res, err := func() (LineRunResult, error) {
|
||||
defer s.releaseAll(ctx, apps, hs)
|
||||
s.waitDrained(ctx, apps)
|
||||
return s.LineRun(ctx)
|
||||
}()
|
||||
switch {
|
||||
@@ -430,28 +354,6 @@ func (q *QBittorrent) HoldValue(prev HeldLimit) HeldLimit {
|
||||
return HeldLimit{Bytes: 1024, Alt: prev.Alt}
|
||||
}
|
||||
|
||||
// UploadRate: transfer/info `up_info_speed` (B/s, payload and overhead together).
|
||||
func (q *QBittorrent) UploadRate(ctx context.Context) (int64, error) {
|
||||
h, err := q.session(ctx, true)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
code, b, _, err := httpDo(ctx, q.client(), "GET", q.base()+"/api/v2/transfer/info", h, nil, "")
|
||||
if err != nil || code != 200 {
|
||||
return 0, fmt.Errorf("qBittorrent transfer/info: HTTP %d %v", code, err)
|
||||
}
|
||||
var ti struct {
|
||||
UpSpeed *int64 `json:"up_info_speed"`
|
||||
}
|
||||
if err := json.Unmarshal(b, &ti); err != nil {
|
||||
return 0, fmt.Errorf("qBittorrent transfer/info: %v", err)
|
||||
}
|
||||
if ti.UpSpeed == nil {
|
||||
return 0, fmt.Errorf("qBittorrent transfer/info: no up_info_speed")
|
||||
}
|
||||
return *ti.UpSpeed, nil
|
||||
}
|
||||
|
||||
func (t *Transmission) ReadLimit(ctx context.Context) (HeldLimit, error) {
|
||||
var st struct {
|
||||
Arguments struct {
|
||||
@@ -484,23 +386,3 @@ func (t *Transmission) ReadLimitLike(ctx context.Context, _ HeldLimit) (HeldLimi
|
||||
|
||||
// HoldValue: 1 kB/s with the limit switched on.
|
||||
func (t *Transmission) HoldValue(HeldLimit) HeldLimit { return HeldLimit{Enabled: true, KBps: 1} }
|
||||
|
||||
// UploadRate: session-stats `uploadSpeed` (B/s).
|
||||
func (t *Transmission) UploadRate(ctx context.Context) (int64, error) {
|
||||
b, err := t.rpc(ctx, `{"method":"session-stats"}`)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
var st struct {
|
||||
Arguments struct {
|
||||
UploadSpeed *int64 `json:"uploadSpeed"`
|
||||
} `json:"arguments"`
|
||||
}
|
||||
if err := json.Unmarshal(b, &st); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if st.Arguments.UploadSpeed == nil {
|
||||
return 0, fmt.Errorf("transmission session-stats: no uploadSpeed")
|
||||
}
|
||||
return *st.Arguments.UploadSpeed, nil
|
||||
}
|
||||
@@ -1,140 +0,0 @@
|
||||
package upwatch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-1250, measured on demo-hp 2026-10-27: qBittorrent 5.2.4 kept sending at full speed for ~14 s after the hold set its
|
||||
// limit to 1 KiB/s, while the agent read the uplink 5–15 s after the call — every test on request was refused „not
|
||||
// quiet" (four tries in one hour). The controller now asks the agent only after the held apps report their own upload
|
||||
// near zero.
|
||||
|
||||
// slept counts the drain waits instead of doing them (every test in this package; none waits for real).
|
||||
var slept time.Duration
|
||||
|
||||
func init() {
|
||||
drainSleep = func(ctx context.Context, d time.Duration) bool {
|
||||
slept += d
|
||||
return ctx.Err() == nil
|
||||
}
|
||||
}
|
||||
|
||||
// The consequence: the agent is asked only once the held app reported two quiet readings in a row.
|
||||
// RED-PROOF: drop the waitDrained call in SeedLineTestNow → the agent is asked with 0 readings taken → fails.
|
||||
func TestR1250_TheAgentIsAskedOnlyAfterTheHeldAppSlowedDown(t *testing.T) {
|
||||
r := newHoldRig(t)
|
||||
// 5 Mbit/s for three readings (the drain), then quiet; one noisy reading between two quiet ones restarts the count.
|
||||
r.app.rates = []int64{640_000, 640_000, 610_000, 0, 300_000, 0, 0}
|
||||
readsAtCall := -1
|
||||
r.during = func() { readsAtCall = r.app.rateReads }
|
||||
slept = 0
|
||||
r.svc.SeedLineTestNow(context.Background())
|
||||
if r.runs != 1 {
|
||||
t.Fatalf("agent asked %d times, want 1", r.runs)
|
||||
}
|
||||
if readsAtCall != 7 {
|
||||
t.Fatalf("the agent was asked after %d speed readings, want 7 — only after two quiet readings in a row", readsAtCall)
|
||||
}
|
||||
if slept != 6*drainPoll {
|
||||
t.Fatalf("waited %s before asking, want %s", slept, 6*drainPoll)
|
||||
}
|
||||
if r.app.limit != (HeldLimit{Bytes: 2_499_584}) || r.record() != nil {
|
||||
t.Fatalf("after the test the app holds %+v (record %+v), want the household's limit back", r.app.limit, r.record())
|
||||
}
|
||||
}
|
||||
|
||||
// Every held app must be quiet, not just the first.
|
||||
// RED-PROOF: read only apps[0] in waitDrained → asked while the second still sends → fails.
|
||||
func TestR1250_EveryHeldAppMustSlowDown(t *testing.T) {
|
||||
qb := &fakeTorrent{limit: HeldLimit{Bytes: 2_499_584}}
|
||||
tr := &fakeTorrent{name: "transmission", limit: HeldLimit{Enabled: true, KBps: 500}, rates: []int64{400_000, 400_000, 400_000, 0}}
|
||||
runs, trAtCall := 0, -1
|
||||
svc := &Service{Store: &memStore{}, Torrents: []Holdable{qb, tr},
|
||||
TorrentRunning: func(string) bool { return true },
|
||||
LineWaiting: func(context.Context) bool { return true },
|
||||
LineRun: func(context.Context) (LineRunResult, error) {
|
||||
runs++
|
||||
trAtCall = tr.rateReads
|
||||
return LineRunResult{OK: true, Mbps: 92}, nil
|
||||
}}
|
||||
svc.SeedLineTestNow(context.Background())
|
||||
if runs != 1 || trAtCall < 5 {
|
||||
t.Fatalf("asked %d time(s) with %d transmission reading(s), want once and after its two quiet readings (5)", runs, trAtCall)
|
||||
}
|
||||
}
|
||||
|
||||
// An app that never slows down: the agent is asked anyway after drainMax (it judges the line), and the limit comes back.
|
||||
// RED-PROOF: drop the drainMax check → the loop never ends → the test times out.
|
||||
func TestR1250_AnAppThatNeverSlowsIsAskedAboutAfterTheMaximum(t *testing.T) {
|
||||
r := newHoldRig(t)
|
||||
r.app.rates = []int64{640_000}
|
||||
slept = 0
|
||||
r.svc.SeedLineTestNow(context.Background())
|
||||
if r.runs != 1 || slept != drainMax {
|
||||
t.Fatalf("asked %d time(s) after %s, want once after %s", r.runs, slept, drainMax)
|
||||
}
|
||||
if r.app.limit != (HeldLimit{Bytes: 2_499_584}) {
|
||||
t.Fatalf("after the test the app holds %+v, want the household's limit back", r.app.limit)
|
||||
}
|
||||
}
|
||||
|
||||
// An app whose speed cannot be read: a plain drainFallback wait, then the agent.
|
||||
// RED-PROOF: return at once on a read error → slept 0 → fails.
|
||||
func TestR1250_AnUnreadableSpeedGetsTheFallbackWait(t *testing.T) {
|
||||
r := newHoldRig(t)
|
||||
r.app.failRate = true
|
||||
slept = 0
|
||||
r.svc.SeedLineTestNow(context.Background())
|
||||
if r.runs != 1 || slept != drainFallback {
|
||||
t.Fatalf("asked %d time(s) after %s, want once after %s", r.runs, slept, drainFallback)
|
||||
}
|
||||
}
|
||||
|
||||
// The two real readers, against the field names the apps serve (qBittorrent 5.2.4 transfer/info, Transmission 4.x
|
||||
// session-stats); a missing field is an error, never a quiet 0.
|
||||
func TestR1250_UploadRateReadsTheAppsOwnSpeed(t *testing.T) {
|
||||
qbBody := `{"up_info_speed":655360,"up_info_data":1}`
|
||||
qs := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
|
||||
switch req.URL.Path {
|
||||
case "/api/v2/auth/login":
|
||||
http.SetCookie(w, &http.Cookie{Name: "SID", Value: "s"})
|
||||
w.WriteHeader(204)
|
||||
case "/api/v2/transfer/info":
|
||||
_, _ = w.Write([]byte(qbBody))
|
||||
default:
|
||||
w.WriteHeader(404)
|
||||
}
|
||||
}))
|
||||
defer qs.Close()
|
||||
qb := &QBittorrent{Box: envBox{"qbittorrent/QBT_ADMIN_PASSWORD": "pw-canary"}, URL: qs.URL}
|
||||
if v, err := qb.UploadRate(context.Background()); err != nil || v != 655360 {
|
||||
t.Fatalf("qBittorrent UploadRate = %d, %v; want 655360", v, err)
|
||||
}
|
||||
qbBody = `{"up_info_data":1}`
|
||||
if v, err := qb.UploadRate(context.Background()); err == nil {
|
||||
t.Fatalf("qBittorrent without up_info_speed read %d, want an error", v)
|
||||
}
|
||||
|
||||
trBody := `{"arguments":{"uploadSpeed":12345,"downloadSpeed":0},"result":"success"}`
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
|
||||
if req.Header.Get("X-Transmission-Session-Id") == "" {
|
||||
w.Header().Set("X-Transmission-Session-Id", "sid")
|
||||
w.WriteHeader(409)
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte(trBody))
|
||||
}))
|
||||
defer ts.Close()
|
||||
tr := &Transmission{Box: envBox{"transmission/PASS": "tr-canary"}, URL: ts.URL}
|
||||
if v, err := tr.UploadRate(context.Background()); err != nil || v != 12345 {
|
||||
t.Fatalf("Transmission UploadRate = %d, %v; want 12345", v, err)
|
||||
}
|
||||
trBody = `{"arguments":{"downloadSpeed":0},"result":"success"}`
|
||||
if v, err := tr.UploadRate(context.Background()); err == nil {
|
||||
t.Fatalf("Transmission without uploadSpeed read %d, want an error", v)
|
||||
}
|
||||
}
|
||||
@@ -20,24 +20,6 @@ type fakeTorrent struct {
|
||||
writes []HeldLimit
|
||||
failWrite bool
|
||||
failRead bool
|
||||
rates []int64 // UploadRate answers in order, the last one repeating; none = 0 (already quiet)
|
||||
rateReads int
|
||||
failRate bool
|
||||
}
|
||||
|
||||
func (f *fakeTorrent) UploadRate(context.Context) (int64, error) {
|
||||
f.rateReads++
|
||||
if f.failRate {
|
||||
return 0, errors.New("transfer/info: HTTP 404")
|
||||
}
|
||||
if len(f.rates) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
i := f.rateReads - 1
|
||||
if i >= len(f.rates) {
|
||||
i = len(f.rates) - 1
|
||||
}
|
||||
return f.rates[i], nil
|
||||
}
|
||||
|
||||
func (f *fakeTorrent) Stack() string {
|
||||
|
||||
@@ -53,11 +53,6 @@ func (s *Server) escrowBannerVisible(r *http.Request) bool {
|
||||
if r == nil || !s.hasAdminSession(r) || !s.escrowPaused() {
|
||||
return false
|
||||
}
|
||||
// R-1219: during a recovery offer a NEW code would replace the key that opens the old off-site copy (R-199) — the
|
||||
// backups page hides the ceremony then, and so does this bar. Pinned by TestR1219_NoCeremonyDuringARecoveryOffer.
|
||||
if s.recoveryOffer() {
|
||||
return false
|
||||
}
|
||||
// R-546: while the agent says the ceremony cannot run yet, do not urge the household into it.
|
||||
// Unknown readiness keeps the bar (fail loud).
|
||||
if ready, known := s.escrowReadiness(r.Context(), false); known && !ready {
|
||||
|
||||
@@ -213,20 +213,8 @@ func (s *Server) escrowStartAPIHandler(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
|
||||
// (1c) R-1219: no ceremony during a recovery offer — a new code would replace the key that opens the old off-site
|
||||
// copy (R-199). The page hides the form; a direct POST (or the old top-bar link) is refused here, before anything is
|
||||
// staged or started. Pinned by TestR1219_NoCeremonyDuringARecoveryOffer.
|
||||
if s.recoveryOffer() {
|
||||
s.logger.Printf("[INFO] [web] escrow start refused: a recovery offer is open (R-1219)")
|
||||
escrowJSON(w, http.StatusConflict, nil, s.msg(r, "api.escrow.recovery_offer_open"))
|
||||
return
|
||||
}
|
||||
|
||||
// (2) Re-stage-first whenever the repository password FILE is on the box (R-1218) — not only when off-site is
|
||||
// configured: a cleared or switched-off target keeps the file, and a ceremony that seals no password makes the hub
|
||||
// demote the household's good package. A box with no password (Scenario C, R-241's awaiting key) skips it.
|
||||
// Pinned by TestR1218_TheCeremonyStagesWhenThePasswordFileExists.
|
||||
if s.backupMgr != nil && s.backupMgr.OffboxRepoPasswordOnBox() {
|
||||
// (2) Re-stage-first (only when offsite is configured — Scenario C boxes skip it).
|
||||
if s.backupMgr != nil && s.backupMgr.OffboxConfigured() {
|
||||
if err := s.escrowStage(r.Context()); err != nil {
|
||||
s.logger.Printf("[WARN] [web] escrow start: re-stage failed (ceremony NOT started): %v", err) // err carries no secret
|
||||
escrowJSON(w, http.StatusBadGateway, nil, s.msg(r, "api.escrow.stage_failed"))
|
||||
@@ -352,12 +340,6 @@ func (s *Server) escrowConfirmAPIHandler(w http.ResponseWriter, r *http.Request)
|
||||
escrowJSON(w, http.StatusGone, map[string]any{"restart": true}, s.msg(r, "api.escrow.confirm_expired"))
|
||||
case http.StatusNotFound:
|
||||
escrowJSON(w, http.StatusGone, map[string]any{"restart": true}, s.msg(r, "api.escrow.confirm_lost"))
|
||||
case http.StatusGatewayTimeout, 0:
|
||||
// R-1254 H2: the agent could not learn which package the hub holds (504), or this call itself failed (0 — it may
|
||||
// have timed out while the agent still stored the package). Either way the new package MAY be on the hub: no
|
||||
// promise about the previous code. Only 502 (the hub holds another package) keeps the store-failed sentence.
|
||||
s.logger.Printf("[WARN] [web] escrow confirm outcome unknown (status %d)", status)
|
||||
escrowJSON(w, http.StatusGatewayTimeout, nil, s.msg(r, "api.escrow.confirm_store_unknown"))
|
||||
default:
|
||||
s.logger.Printf("[WARN] [web] escrow confirm failed (status %d)", status)
|
||||
escrowJSON(w, http.StatusBadGateway, nil, s.msg(r, "api.escrow.confirm_store_failed"))
|
||||
|
||||
@@ -1894,10 +1894,7 @@ func (s *Server) backupRestoreHandler(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
|
||||
s.logger.Printf("[WARN] [web] Restore requested (async): stack=%s, snapshot=%s from %s", stackName, snapshotID, r.RemoteAddr)
|
||||
if msg, ok := s.beginRestoreOp("restore", stackName); !ok {
|
||||
http.Redirect(w, r, "/backups/restore?flash_error="+url.QueryEscape(msg), http.StatusFound)
|
||||
return
|
||||
}
|
||||
s.backupMgr.BeginRestoreOp("restore", stackName)
|
||||
go func() {
|
||||
start := time.Now()
|
||||
// Phase 2b: restore from the app's recovery unit (recovers secrets from the guest, fail-closed
|
||||
@@ -2239,10 +2236,7 @@ func (s *Server) backupTier2RestoreHandler(w http.ResponseWriter, r *http.Reques
|
||||
}
|
||||
|
||||
s.logger.Printf("[WARN] [web] Tier-2 file restore requested (async): stack=%s from %s", stackName, r.RemoteAddr)
|
||||
if msg, ok := s.beginRestoreOp("tier2-restore", stackName); !ok {
|
||||
http.Redirect(w, r, "/backups/apps?flash_error="+url.QueryEscape(msg), http.StatusFound)
|
||||
return
|
||||
}
|
||||
s.backupMgr.BeginRestoreOp("tier2-restore", stackName)
|
||||
go func() {
|
||||
n, err := s.backupMgr.RestoreTier2Files(stackName)
|
||||
if err != nil {
|
||||
@@ -2317,10 +2311,7 @@ func (s *Server) backupTier2UnitRestoreHandler(w http.ResponseWriter, r *http.Re
|
||||
// (R-538), which is what left such an app with no way back from the second drive.
|
||||
if s.backupMgr.HasDriveFileLegs(stackName) {
|
||||
s.logger.Printf("[WARN] [web] Tier-2 WHOLE restore requested (async, files + database): stack=%s from %s", stackName, r.RemoteAddr)
|
||||
if msg, ok := s.beginRestoreOp("tier2-whole-restore", stackName); !ok {
|
||||
http.Redirect(w, r, "/backups/apps?flash_error="+url.QueryEscape(msg), http.StatusFound)
|
||||
return
|
||||
}
|
||||
s.backupMgr.BeginRestoreOp("tier2-whole-restore", stackName)
|
||||
go func() {
|
||||
start := time.Now()
|
||||
res, err := s.backupMgr.RestoreTier2Whole(stackName)
|
||||
@@ -2355,10 +2346,7 @@ func (s *Server) backupTier2UnitRestoreHandler(w http.ResponseWriter, r *http.Re
|
||||
}
|
||||
|
||||
s.logger.Printf("[WARN] [web] Tier-2 UNIT restore requested (async, OVERWRITES live data): stack=%s from %s", stackName, r.RemoteAddr)
|
||||
if msg, ok := s.beginRestoreOp("tier2-unit-restore", stackName); !ok {
|
||||
http.Redirect(w, r, "/backups/apps?flash_error="+url.QueryEscape(msg), http.StatusFound)
|
||||
return
|
||||
}
|
||||
s.backupMgr.BeginRestoreOp("tier2-unit-restore", stackName)
|
||||
go func() {
|
||||
start := time.Now()
|
||||
res, err := s.backupMgr.RestoreTier2Unit(stackName)
|
||||
|
||||
@@ -106,10 +106,7 @@ func (s *Server) keptUnitRestoreHandler(w http.ResponseWriter, r *http.Request)
|
||||
return
|
||||
}
|
||||
s.logger.Printf("[WARN] [web] Restore of %s from a set-aside unit requested (async): %s from %s", k.Source, k.Path, r.RemoteAddr)
|
||||
if msg, ok := s.beginRestoreOp("restore", k.Source); !ok {
|
||||
http.Redirect(w, r, "/backups/restore?flash_error="+url.QueryEscape(msg), http.StatusFound)
|
||||
return
|
||||
}
|
||||
s.backupMgr.BeginRestoreOp("restore", k.Source)
|
||||
go func() {
|
||||
app, res, err := s.backupMgr.RestoreSetAsideUnit(id)
|
||||
if err != nil {
|
||||
|
||||
@@ -217,23 +217,16 @@ func (s *Server) keptLoadHandler(w http.ResponseWriter, r *http.Request) {
|
||||
_, err := s.stackMgr.RestoreKeptFiles(it)
|
||||
return err
|
||||
}
|
||||
// R-1256 H6: the load begins in the same step as the check for every other operation; a refusal moves nothing
|
||||
// (the dated folder's files move back inside the operation, as prepare).
|
||||
var lErr error
|
||||
if c.Tier == backup.KeptTierOffsite {
|
||||
s.logger.Printf("[INFO] [web] kept LOAD %s: %s from the off-site snapshot %s (from %s)", it.App, it.Path, c.SnapshotID, r.RemoteAddr)
|
||||
lErr = s.backupMgr.LoadKeptOffsite(app, it.Drive, c, moveBack, okMsg, failMsg, after)
|
||||
s.backupMgr.LoadKeptOffsite(app, it.Drive, c, moveBack, okMsg, failMsg, after)
|
||||
} else {
|
||||
s.logger.Printf("[INFO] [web] kept LOAD %s: %s from %s (from %s)", it.App, it.Path, c.UnitDir, r.RemoteAddr)
|
||||
lErr = s.backupMgr.LoadKeptApp(app, c.UnitDir, moveBack, okMsg, failMsg, after)
|
||||
}
|
||||
if lErr != nil {
|
||||
msg, _ := s.restoreOpBlocked()
|
||||
if msg == "" { // the other operation ended in the meantime
|
||||
msg, _ = s.restoreBusyMessage(backup.RestoreBusyRestore, "")
|
||||
if err := moveBack(); err != nil {
|
||||
s.keptRedirect(w, r, "flash_error", s.errText(r, err))
|
||||
return
|
||||
}
|
||||
s.keptRedirect(w, r, "flash_error", msg)
|
||||
return
|
||||
s.logger.Printf("[INFO] [web] kept LOAD %s: %s from %s (from %s)", it.App, it.Path, c.UnitDir, r.RemoteAddr)
|
||||
s.backupMgr.LoadKeptApp(app, c.UnitDir, okMsg, failMsg, after)
|
||||
}
|
||||
http.Redirect(w, r, "/backups/restore?"+flashQuery("flash", "flash.restore.started"), http.StatusFound)
|
||||
}
|
||||
|
||||
@@ -450,10 +450,7 @@ func (s *Server) offboxRestoreHandler(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
full := mode == "full"
|
||||
if msg, ok := s.beginRestoreOp("offbox-restore", app); !ok {
|
||||
offboxRedirectTo(w, r, restoreWizardPath(app), msg, true)
|
||||
return
|
||||
}
|
||||
s.backupMgr.BeginRestoreOp("offbox-restore", app)
|
||||
go func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
|
||||
defer cancel()
|
||||
@@ -541,10 +538,7 @@ func (s *Server) offboxReconstituteHandler(w http.ResponseWriter, r *http.Reques
|
||||
// data"; this one answers "yes, into a different place than the backup recorded". One checkbox
|
||||
// carrying both would be the two-decisions-one-button shape R-48 removed from this surface.
|
||||
ackPlacement := r.FormValue("ack_placement") == "1"
|
||||
if msg, ok := s.beginRestoreOp("offbox-reconstitute", app); !ok {
|
||||
offboxRedirectTo(w, r, restoreWizardPath(app), msg, true)
|
||||
return
|
||||
}
|
||||
s.backupMgr.BeginRestoreOp("offbox-reconstitute", app)
|
||||
go func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Minute)
|
||||
defer cancel()
|
||||
@@ -684,10 +678,7 @@ func (s *Server) offboxPlaceHandler(w http.ResponseWriter, r *http.Request) {
|
||||
offboxRedirectTo(w, r, restoreWizardPath(app), offsiteScratchIncompleteKey, true)
|
||||
return
|
||||
}
|
||||
if msg, ok := s.beginRestoreOp("offbox-place", app); !ok {
|
||||
offboxRedirectTo(w, r, restoreWizardPath(app), msg, true)
|
||||
return
|
||||
}
|
||||
s.backupMgr.BeginRestoreOp("offbox-place", app)
|
||||
go func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
|
||||
defer cancel()
|
||||
@@ -720,10 +711,7 @@ func (s *Server) sharesRestoreHandler(w http.ResponseWriter, r *http.Request) {
|
||||
offboxRedirectTo(w, r, "/backups/restore", msg, true)
|
||||
return
|
||||
}
|
||||
if msg, ok := s.beginRestoreOp("shares-restore", backup.SharesDisplayName); !ok {
|
||||
offboxRedirectTo(w, r, "/backups/restore", msg, true)
|
||||
return
|
||||
}
|
||||
s.backupMgr.BeginRestoreOp("shares-restore", backup.SharesDisplayName)
|
||||
go func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
|
||||
defer cancel()
|
||||
@@ -749,10 +737,7 @@ func (s *Server) sharesPlaceHandler(w http.ResponseWriter, r *http.Request) {
|
||||
offboxRedirectTo(w, r, "/backups/restore", msg, true)
|
||||
return
|
||||
}
|
||||
if msg, ok := s.beginRestoreOp("shares-place", backup.SharesDisplayName); !ok {
|
||||
offboxRedirectTo(w, r, "/backups/restore", msg, true)
|
||||
return
|
||||
}
|
||||
s.backupMgr.BeginRestoreOp("shares-place", backup.SharesDisplayName)
|
||||
go func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
@@ -93,9 +93,7 @@ func TestR1151_ConfirmAnswersInTheBoxLanguage(t *testing.T) {
|
||||
{http.StatusGone, http.StatusGone, true, "Lejárt az idő", "has run out"},
|
||||
{http.StatusNotFound, http.StatusGone, true, "már nem vár megerősítésre", "no longer waits"},
|
||||
{http.StatusBadGateway, http.StatusBadGateway, false, "nem sikerült elmenteni", "could not be saved"},
|
||||
// R-1254 H2 (2026-10-28): a failed call (0) or an agent that cannot learn the outcome (504) promises nothing.
|
||||
{0, http.StatusGatewayTimeout, false, "Nem tudjuk biztosan", "cannot tell yet"},
|
||||
{http.StatusGatewayTimeout, http.StatusGatewayTimeout, false, "Nem tudjuk biztosan", "cannot tell yet"},
|
||||
{0, http.StatusBadGateway, false, "nem sikerült elmenteni", "could not be saved"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
for _, lang := range []string{"hu", "en"} {
|
||||
|
||||
@@ -1,56 +0,0 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1218: a recovery ceremony run while off-site is switched OFF (or its target cleared) must still stage the off-site
|
||||
// repository password when the password FILE is on the box — ClearOffboxTarget keeps that file on purpose. Before the
|
||||
// fix the stage ran only when OffboxConfigured() was true, the agent sealed no password, uploaded sha "", and the hub
|
||||
// demoted the household's good package: the family's code no longer opened its off-site copy. Red-proved against the
|
||||
// OffboxConfigured() gate (order „preflight,start", no stage).
|
||||
func TestR1218_TheCeremonyStagesWhenThePasswordFileExists(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
off func(h *escrowWizardHarness)
|
||||
}{
|
||||
{"off-site switched off", func(h *escrowWizardHarness) {
|
||||
tg := h.sett.GetOffboxTarget()
|
||||
tg.Enabled = false
|
||||
if err := h.sett.SetOffboxTarget(tg); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}},
|
||||
{"off-site target cleared", func(h *escrowWizardHarness) {
|
||||
if err := h.sett.SetOffboxTarget(nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
h := newEscrowWizardHarness(t)
|
||||
h.configureOffbox(t, "escrowed")
|
||||
tc.off(h)
|
||||
if h.m.OffboxConfigured() {
|
||||
t.Fatal("setup: off-site should read as not configured")
|
||||
}
|
||||
w := postStart(t, h.s, wizardPassword)
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("start: %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
if got := strings.Join(h.order, ","); got != "preflight,stage,start" {
|
||||
t.Fatalf("call order = %q — the repository password on the box was NOT staged, so the new package would cover no key", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
// The control: no password file at all (a box that never had off-site, R-241's awaiting-key shape) → nothing staged.
|
||||
h := newEscrowWizardHarness(t)
|
||||
if w := postStart(t, h.s, wizardPassword); w.Code != http.StatusOK {
|
||||
t.Fatalf("start without off-site: %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
if got := strings.Join(h.order, ","); got != "preflight,start" {
|
||||
t.Fatalf("a box with no repository password staged something: %q", got)
|
||||
}
|
||||
}
|
||||
@@ -1,69 +0,0 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// offerShapeC puts the harness box in the recovery offer, shape (c): the hub holds a package for a DIFFERENT repository
|
||||
// key than the one the box uses — the old off-site copy opens only with the household's EXISTING recovery code.
|
||||
func offerShapeC(t *testing.T, h *escrowWizardHarness) {
|
||||
t.Helper()
|
||||
h.configureOffbox(t, "pending")
|
||||
if err := h.sett.SetHubEscrowIdentityPresent(true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := h.sett.SetHubEscrowKeySHA256("0000000000000000000000000000000000000000000000000000000000000000", "2026-10-26T20:00:00Z"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !h.m.OffsiteRecoveryOffer() {
|
||||
t.Fatal("setup: the box should be in the recovery offer")
|
||||
}
|
||||
}
|
||||
|
||||
// R-1219: during a recovery offer a NEW recovery code would replace the key that opens the old off-site copy (R-199).
|
||||
// The backups page already hid the ceremony; the global top bar still linked to it and the start handler ran it. Now
|
||||
// the bar is not shown and the start handler answers 409 with a sentence in the household's language, BEFORE anything
|
||||
// is staged or started. COMPANION RED-PROOF: before the fix the bar was visible and the start answered 200
|
||||
// („preflight,stage,start").
|
||||
func TestR1219_NoCeremonyDuringARecoveryOffer(t *testing.T) {
|
||||
h := newEscrowWizardHarness(t)
|
||||
offerShapeC(t, h)
|
||||
if !h.s.escrowPaused() {
|
||||
t.Fatal("setup: the tier should read as paused (pending) — the bar would otherwise be hidden for another reason")
|
||||
}
|
||||
r := httptest.NewRequest("GET", "/", nil)
|
||||
r.AddCookie(&http.Cookie{Name: sessionCookieName, Value: newTestSession(h.s)})
|
||||
if h.s.escrowBannerVisible(r) {
|
||||
t.Error("the top bar urges the recovery-code ceremony during a recovery offer")
|
||||
}
|
||||
w := postStart(t, h.s, wizardPassword)
|
||||
if w.Code != http.StatusConflict {
|
||||
t.Fatalf("start during an offer = %d (%s), want 409", w.Code, w.Body.String())
|
||||
}
|
||||
if !strings.Contains(w.Body.String(), "helyre") { // ASCII fragment of the Hungarian sentence (ui-hungarian rule)
|
||||
t.Errorf("the refusal does not say why in Hungarian: %s", w.Body.String())
|
||||
}
|
||||
for _, step := range h.order {
|
||||
if step == "stage" || step == "start" {
|
||||
t.Fatalf("the agent was reached during an offer: %v", h.order)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The control for the bar half: the SAME paused box with no recovery offer, the same logged-in request — the bar IS
|
||||
// shown. Without this, a bar hidden for any other reason (no session, readiness) would pass the test above.
|
||||
func TestR1219_TheBarStillShowsWithoutAnOffer(t *testing.T) {
|
||||
h := newEscrowWizardHarness(t)
|
||||
h.configureOffbox(t, "pending")
|
||||
if h.m.OffsiteRecoveryOffer() {
|
||||
t.Fatal("setup: no offer expected")
|
||||
}
|
||||
r := httptest.NewRequest("GET", "/", nil)
|
||||
r.AddCookie(&http.Cookie{Name: sessionCookieName, Value: newTestSession(h.s)})
|
||||
if !h.s.escrowBannerVisible(r) {
|
||||
t.Fatal("the bar is hidden on a paused box with no offer — the offer test above would prove nothing")
|
||||
}
|
||||
}
|
||||
@@ -1,48 +0,0 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1254 H2 (the household's sentence): a confirm whose upload answer is late or lost must not promise „your previous
|
||||
// code still works" — the hub may hold the new package. Since agent 0.168.0 the agent asks the hub and answers 502 only
|
||||
// when the hub holds ANOTHER package; 504 = unknown. A controller-side transport failure (status 0 — the call itself
|
||||
// timed out while the agent may still store) is unknown too.
|
||||
// RED-PROOF: drop the 504/0 case from escrowConfirmAPIHandler → the default sentence („a régi kódod továbbra is
|
||||
// érvényes") is shown → FAILS.
|
||||
func TestR1254_AnUnknownConfirmOutcomePromisesNothing(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
status int
|
||||
}{{"agent 504", http.StatusGatewayTimeout}, {"transport failure", 0}} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
h := newEscrowWizardHarness(t)
|
||||
h.configureOffbox(t, "pending")
|
||||
h.agent.confirmStatus, h.agent.confirmErr = tc.status, errors.New("late")
|
||||
for _, lang := range []string{"hu", "en"} {
|
||||
w := postConfirm(t, h.s, lang, `{"words":["ketto","negy"]}`)
|
||||
low := strings.ToLower(w.Body.String())
|
||||
// ASCII fragments (the accented forms carry these stems): „régi kódod továbbra is érvényes" / „still works".
|
||||
if strings.Contains(low, "tovabbra is") || strings.Contains(low, "továbbra is") || strings.Contains(low, "still works") {
|
||||
t.Fatalf("%s: an unknown outcome promised the previous code: %s", lang, w.Body.String())
|
||||
}
|
||||
if !strings.Contains(low, "mindket") && !strings.Contains(low, "mindkét") && !strings.Contains(low, "both") {
|
||||
t.Fatalf("%s: the sentence must tell the household to keep both codes: %s", lang, w.Body.String())
|
||||
}
|
||||
}
|
||||
if ceremonyStamped(t, h) {
|
||||
t.Fatal("an unknown outcome stamped the ceremony completed")
|
||||
}
|
||||
})
|
||||
}
|
||||
// Control: the agent says the hub holds another package (502) → the old sentence, now true.
|
||||
h := newEscrowWizardHarness(t)
|
||||
h.configureOffbox(t, "pending")
|
||||
h.agent.confirmStatus, h.agent.confirmErr = http.StatusBadGateway, errors.New("not stored")
|
||||
if w := postConfirm(t, h.s, "en", `{"words":["ketto","negy"]}`); !strings.Contains(w.Body.String(), "still works") {
|
||||
t.Fatalf("502 must keep the store-failed sentence: %s", w.Body.String())
|
||||
}
|
||||
}
|
||||
@@ -1,60 +0,0 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
|
||||
)
|
||||
|
||||
// R-1256 H6: every restore handler records its op through beginRestoreOp, which asks about every other operation in
|
||||
// the same step (backup.TryBeginRestoreOp). A handler that calls BeginRestoreOp directly re-opens the window between
|
||||
// the early refusal and the record. RED-PROOF: put one `s.backupMgr.BeginRestoreOp(...)` back → FAILS, naming the file.
|
||||
func TestR1256_H6_NoHandlerBeginsARestoreUnchecked(t *testing.T) {
|
||||
files, _ := filepath.Glob("*.go")
|
||||
for _, f := range files {
|
||||
if strings.HasSuffix(f, "_test.go") {
|
||||
continue
|
||||
}
|
||||
src, err := os.ReadFile(f)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
fs := token.NewFileSet()
|
||||
af, err := parser.ParseFile(fs, f, src, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ast.Inspect(af, func(n ast.Node) bool {
|
||||
if sel, ok := n.(*ast.SelectorExpr); ok && sel.Sel.Name == "BeginRestoreOp" {
|
||||
t.Errorf("%s: a direct BeginRestoreOp — begin through s.beginRestoreOp (R-1256 H6)", fs.Position(sel.Pos()))
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The handler's begin refuses with the household's sentence when a second-drive copy began after the early check.
|
||||
func TestR1256_H6_TheBeginRefusesWithTheCopySentence(t *testing.T) {
|
||||
s, _, _ := newOffboxWebServer(t)
|
||||
if _, blocked := s.restoreOpBlocked(); blocked {
|
||||
t.Fatal("fixture: nothing runs")
|
||||
}
|
||||
done := backup.MarkTier2CopyForTest("immich")
|
||||
msg, ok := s.beginRestoreOp("offbox-reconstitute", "immich")
|
||||
done()
|
||||
if ok || !strings.Contains(msg, "második meghajtóra") {
|
||||
t.Fatalf("the begin must refuse with the copy named; ok=%v %q", ok, msg)
|
||||
}
|
||||
if s.backupMgr.RestoreStatus().Running {
|
||||
t.Fatal("a refused begin recorded a restore")
|
||||
}
|
||||
if _, ok := s.beginRestoreOp("offbox-reconstitute", "immich"); !ok {
|
||||
t.Fatal("after the copy the begin must work")
|
||||
}
|
||||
}
|
||||
@@ -209,50 +209,28 @@ func (s *Server) restoreOpBlocked() (string, bool) {
|
||||
if s.backupMgr == nil {
|
||||
return "", false
|
||||
}
|
||||
busy, app := s.backupMgr.RestoreBusy()
|
||||
return s.restoreBusyMessage(busy, app)
|
||||
}
|
||||
|
||||
// beginRestoreOp records a restore op in the SAME step as the check for every other operation (R-1256 H6): the early
|
||||
// restoreOpBlocked answer can be stale by the time a handler has read its form, and a second-drive copy that began in
|
||||
// between ran beside the restore. A refusal returns the same sentence as restoreOpBlocked; the caller redirects with it
|
||||
// and starts nothing. Every restore handler begins through here — TestR1256_H6_NoHandlerBeginsARestoreUnchecked.
|
||||
func (s *Server) beginRestoreOp(op, stack string) (string, bool) {
|
||||
busy, app := s.backupMgr.TryBeginRestoreOp(op, stack)
|
||||
if busy == "" {
|
||||
return "", true
|
||||
}
|
||||
if s.logger != nil {
|
||||
s.logger.Printf("[WARN] [restore] %s of %s refused at its start — a %s operation runs (R-1256)", op, stack, busy)
|
||||
}
|
||||
msg, _ := s.restoreBusyMessage(busy, app)
|
||||
return msg, false
|
||||
}
|
||||
|
||||
// restoreBusyMessage is the household's sentence for a RestoreBusy reason ("" = not busy).
|
||||
func (s *Server) restoreBusyMessage(busy, app string) (string, bool) {
|
||||
switch busy {
|
||||
case "":
|
||||
return "", false
|
||||
case backup.RestoreBusyRestore:
|
||||
if st := s.backupMgr.RestoreStatus(); restoreOpInFlight(st) {
|
||||
subject := "Egy visszaállítási művelet"
|
||||
if app != "" {
|
||||
subject = "Egy visszaállítási művelet (" + app + ")"
|
||||
if st.Stack != "" {
|
||||
subject = "Egy visszaállítási művelet (" + st.Stack + ")"
|
||||
}
|
||||
return subject + " már fut, ezért most nem indítható újabb. Az állapotát ezen az oldalon " +
|
||||
"követheted; amint befejeződik, újra indíthatsz visszaállítást.", true
|
||||
case backup.RestoreBusyBackup:
|
||||
}
|
||||
if s.backupMgr.IsRunning() {
|
||||
return "Egy mentési művelet már fut, ezért most nem indítható visszaállítás. Az állapotát " +
|
||||
"ezen az oldalon követheted; amint befejeződik, újra indíthatsz visszaállítást.", true
|
||||
default:
|
||||
// The stray-copy fix: a second-drive copy (rsync --delete) runs now, or one a previous controller process lost
|
||||
// may still run — a restore beside it could lose the files it brings back (R-1180's other direction).
|
||||
}
|
||||
// The stray-copy fix: a second-drive copy (rsync --delete) runs now, or one a previous controller process lost may
|
||||
// still run — a restore beside it could lose the files it brings back (R-1180's other direction).
|
||||
if app, blocked := s.backupMgr.Tier2CopyBlocksRestore(); blocked {
|
||||
if s.logger != nil {
|
||||
s.logger.Printf("[WARN] [restore] restore refused — a second-drive copy is running or may still run (app=%q)", app)
|
||||
}
|
||||
return "Éppen fut a második meghajtóra készülő mentési másolat, ezért most nem indítható visszaállítás. " +
|
||||
"Néhány perc múlva próbáld újra.", true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// backupsRestoreWizardHandler renders GET /backups/restore/app?name=<app> — the single entry the
|
||||
|
||||
@@ -21,8 +21,6 @@ Gates, in order (all must pass; **non-zero exit on any failure**):
|
||||
… security-scan govulncheck (called vulnerabilities) + gitleaks on the pushed commits, WARN-FIRST: it
|
||||
lists, it never refuses (R-986; shared, felhom.eu/scripts/security_scan_gate.py). Fast
|
||||
although it reads vuln.go.dev: offline it says NOT CHECKED, so it cannot stop a push
|
||||
… upstream every infra.go pin and every Dockerfile FROM names where its security notices are
|
||||
published (R-1303; shared, felhom.eu/scripts/security_watch_inventory.py)
|
||||
… gate-decoys the decoy suite (test_gate_decoys.py): every gate above still convicts its
|
||||
decoys (R-1116). ~22 s and it edits real files in this tree for a moment, so
|
||||
the full run only, never --fast (never the hook, never CI)
|
||||
@@ -67,8 +65,6 @@ SHARED_INSTRUCTIONS = os.path.join(SHARED_DIR, "instructions_gate.py")
|
||||
SHARED_OBSERVATIONS = os.path.join(SHARED_DIR, "observations_gate.py")
|
||||
# R-986 — shared too: the dependency and secret scanner, warn-first.
|
||||
SHARED_SECURITY = os.path.join(SHARED_DIR, "security_scan_gate.py")
|
||||
# R-1303 — shared too: every pin names where its security notices are published (the nightly watch reads it).
|
||||
SHARED_UPSTREAM = os.path.join(SHARED_DIR, "security_watch_inventory.py")
|
||||
|
||||
# R-404 — the golden NOTICE. Lives here, beside the runner, because it is about THIS repo's
|
||||
# releases; it imports the felhom.eu gate rather than copying its comparison.
|
||||
@@ -118,9 +114,6 @@ GATES = [
|
||||
# although it reads vuln.go.dev: it cannot refuse, so a network blink is a NOT CHECKED line. ~8 s here. CI fetches
|
||||
# the pinned gitleaks (FELHOM_SCAN_FETCH=1) and has no Go, so govulncheck says NOT CHECKED there.
|
||||
("security-scan", SHARED_SECURITY, [REPO], True, True),
|
||||
# R-1303 (2026-10-26) — every infra.go pin (upstream.txt beside it) and every Dockerfile FROM / restic ARG (a
|
||||
# `# upstream:` comment directly above) names its notice source. Fast: stdlib file reads.
|
||||
("upstream", SHARED_UPSTREAM, ["--gate=controller", REPO], True, True),
|
||||
# R-404 — ADVISORY. Reports the golden debt where it is created; never refuses.
|
||||
("golden-notice", GOLDEN_NOTICE, [REPO], True, False),
|
||||
# R-1116 (2026-10-17) — the gates' OWN decoy suite: every gate above, red-proved against its decoys. It was
|
||||
|
||||
@@ -5,7 +5,6 @@
|
||||
"api.escrow.confirm_expired": "BORN AS A KEY (R-1151, 2026-10-19) -- a NEW sentence, never a Go literal: the wizard's typed-back confirm answer (agent >= 0.160.0 keeps the new package until the words come back). Pinned by TestR1151_ConfirmAnswersInTheBoxLanguage.",
|
||||
"api.escrow.confirm_lost": "BORN AS A KEY (R-1151, 2026-10-19) -- a NEW sentence, never a Go literal: the wizard's typed-back confirm answer (agent >= 0.160.0 keeps the new package until the words come back). Pinned by TestR1151_ConfirmAnswersInTheBoxLanguage.",
|
||||
"api.escrow.confirm_store_failed": "BORN AS A KEY (R-1151, 2026-10-19) -- a NEW sentence, never a Go literal: the wizard's typed-back confirm answer (agent >= 0.160.0 keeps the new package until the words come back). Pinned by TestR1151_ConfirmAnswersInTheBoxLanguage.",
|
||||
"api.escrow.confirm_store_unknown": "BORN AS A KEY (R-1254, 2026-10-28 night) -- a NEW sentence, never a Go literal: the confirm's outcome is unknown (agent 504 or the call itself failed) -- no promise about the previous code. Pinned by TestR1254_AnUnknownConfirmOutcomePromisesNothing and TestR1151_ConfirmAnswersInTheBoxLanguage.",
|
||||
"waiting.title": "BORN AS A KEY (R-554, 2026-10-12) -- the title of the page an unconfigured box shows while it retries its configuration, replacing the deleted setup wizard. Pinned by internal/bootstrap TestWaitForConfig_ServesWaitingPageThenStarts.",
|
||||
"waiting.body": "BORN AS A KEY (R-554, 2026-10-12) -- the body of that waiting page. Pinned by internal/bootstrap TestWaitForConfig_ServesWaitingPageThenStarts.",
|
||||
"recovery_info.body": "BORN AS A KEY (R-554, 2026-10-12, `09` §3 decision 222) -- the NEW text of recovery-info.txt, pointing at the dashboard's Restore page and the recovery code instead of the deleted setup wizard. The operator approves the Hungarian before release. Pinned by internal/recovery TestRecoveryInfo_PointsAtDashboardNotWizard.",
|
||||
@@ -329,11 +328,7 @@
|
||||
"err.backup.restore_image_gone": "BORN AS A KEY (D48, operator ruling 2026-10-25; 09 decision 257) -- a restore downloads first: the refusal when an image of the backup's version cannot be downloaded (deleted by its maker, or the registry busy or unreachable); the app untouched. Pinned by internal/backup/restore_pull_test.go TestD48_*.",
|
||||
"err.backup.restore_image_unreachable": "BORN AS A KEY (D48, operator ruling 2026-10-25; 09 decision 257) -- a restore downloads first: the refusal when an image of the backup's version cannot be downloaded (deleted by its maker, or the registry busy or unreachable); the app untouched. Pinned by internal/backup/restore_pull_test.go TestD48_*.",
|
||||
"kept.backup.future_note": "BORN AS A KEY (R-1271, brief 2026-10-25 Part 4) -- a copy dated in the future is never the newest; when it is the only copy the kept-data question and the kept list say its date is in the future. Pinned by internal/backup/kept_pick_test.go and internal/api/r1271_future_copy_test.go.",
|
||||
"kept.choice.use.future": "BORN AS A KEY (R-1271, brief 2026-10-25 Part 4) -- a copy dated in the future is never the newest; when it is the only copy the kept-data question and the kept list say its date is in the future. Pinned by internal/backup/kept_pick_test.go and internal/api/r1271_future_copy_test.go.",
|
||||
"api.escrow.recovery_offer_open": "BORN AS A KEY (R-1219, brief 2026-10-26 night Part 3) -- the escrow start handler refuses during a recovery offer. Pinned by TestR1219_NoCeremonyDuringARecoveryOffer.",
|
||||
"err.appexport.kulonleges_fajl": "BORN AS A KEY (R-1187 a, brief 2026-10-26 night Part 4) -- the export refuses a FIFO, socket or device, naming the path. Pinned by TestR1187a_AFifoFailsTheExportNamingThePath.",
|
||||
"err.appexport.import_foglalt": "BORN AS A KEY (R-1187 d, brief 2026-10-26 night Part 4) -- an import is refused while another operation holds the box. Pinned by TestR1187d_AnImportWaitsForARestore.",
|
||||
"err.appexport.hdd_path_hibas": "BORN AS A KEY (R-1187 f, brief 2026-10-26 night Part 4) -- a bundle with an empty or relative HDD_PATH is refused before anything is touched. Pinned by TestR1187f_AnEmptyOrRelativeHDDPathIsRefused."
|
||||
"kept.choice.use.future": "BORN AS A KEY (R-1271, brief 2026-10-25 Part 4) -- a copy dated in the future is never the newest; when it is the only copy the kept-data question and the kept list say its date is in the future. Pinned by internal/backup/kept_pick_test.go and internal/api/r1271_future_copy_test.go."
|
||||
},
|
||||
"flash.share.already_on": "A megosztás már be van kapcsolva.",
|
||||
"flash.share.enable_failed": "A megosztás bekapcsolása nem sikerült.",
|
||||
|
||||
@@ -141,28 +141,6 @@ run_build_in absent "$CLEAN_REPO" 0.999.0 --push
|
||||
check "ahead of origin/main -> refuses" 1 "not at origin/main" "FAKE-DOCKER-REACHED"
|
||||
rm -rf "$CLEAN_REPO"
|
||||
|
||||
# ── OR AT THE PUSHED TAG v<VERSION> (v0.345.1, 2026-10-26) ───────────────────────────────────────
|
||||
# A patch release off a released commit while main holds unreleased work. The tag must be on the SERVER, name THIS version,
|
||||
# and point at HEAD. RED-PROOF: with the tag branch accepting any tag the three refusals fail; with it removed the
|
||||
# first case fails.
|
||||
TAG_REPO="$(scratch_repo)"; TAG_ORIGIN="$(mktemp -d)"
|
||||
git init -q --bare "$TAG_ORIGIN" 2>/dev/null; git -C "$TAG_REPO" remote add origin "$TAG_ORIGIN"
|
||||
echo y > "$TAG_REPO/controller/f2"
|
||||
git -C "$TAG_REPO" add -A >/dev/null; git -C "$TAG_REPO" commit -qm fix >/dev/null
|
||||
git -C "$TAG_REPO" tag -a v0.999.1 -m t; git -C "$TAG_REPO" push -q origin v0.999.1 2>/dev/null
|
||||
run_build_in absent "$TAG_REPO" 0.999.1 --push
|
||||
check "at the pushed tag v<VERSION> -> proceeds" "$rc" "at the pushed tag v0.999.1" "refusing to push"
|
||||
run_build_in absent "$TAG_REPO" 0.999.2 --push
|
||||
check "a pushed tag of ANOTHER version -> refuses" 1 "nor at a pushed tag v0.999.2" "FAKE-DOCKER-REACHED"
|
||||
git -C "$TAG_REPO" tag v0.999.3
|
||||
run_build_in absent "$TAG_REPO" 0.999.3 --push
|
||||
check "a tag only on this clone -> refuses" 1 "nor at a pushed tag v0.999.3" "FAKE-DOCKER-REACHED"
|
||||
echo z > "$TAG_REPO/controller/f3"
|
||||
git -C "$TAG_REPO" add -A >/dev/null; git -C "$TAG_REPO" commit -qm later >/dev/null
|
||||
run_build_in absent "$TAG_REPO" 0.999.1 --push
|
||||
check "the pushed tag on another commit -> refuses" 1 "nor at a pushed tag v0.999.1" "FAKE-DOCKER-REACHED"
|
||||
rm -rf "$TAG_REPO" "$TAG_ORIGIN"
|
||||
|
||||
NOT_A_REPO="$(mktemp -d)"; mkdir -p "$NOT_A_REPO/controller"
|
||||
run_build_in absent "$NOT_A_REPO" 0.999.0 --push
|
||||
check "not a git checkout -> refuses" 1 "not a git checkout" "FAKE-DOCKER-REACHED"
|
||||
|
||||
@@ -55,10 +55,6 @@ COVERS = {
|
||||
"second secret on another line of an accepted file still listed; a called govulncheck "
|
||||
"finding listed, an imported-only one not; no tool is NOT CHECKED out loud), run from here; "
|
||||
"plus the gate on THIS repo: exit 0 and its summary line (warn-first)"),
|
||||
"upstream": ("R-1303: a scratch controller tree - a NEW *Image const with no upstream.txt line, the line "
|
||||
"only in a COMMENT, a Dockerfile `# upstream:` comment one line too high (not directly above "
|
||||
"its FROM), a new FROM with none; controls: a FROM of an earlier STAGE needs no line, and the "
|
||||
"real repo passes"),
|
||||
"offbox-rename": "R-425: NAS branding in a NEW backups*.html, and in a bundle value an offbox Go file "
|
||||
"names; control: the same token in the network-storage feature's copy is accepted",
|
||||
"docker-v": "R-426: an unallowlisted `-v /etc:/x` in a NEW .go file two directories down "
|
||||
@@ -537,56 +533,6 @@ def security_cases():
|
||||
|
||||
security_cases()
|
||||
|
||||
|
||||
def upstream_cases():
|
||||
"""R-1303: the shared upstream rule (felhom.eu scripts/security_watch_inventory.py --gate=controller) on a SCRATCH
|
||||
controller tree, then on this repo."""
|
||||
global ran
|
||||
inv = os.path.join(SHARED, "security_watch_inventory.py")
|
||||
if not os.path.isfile(inv):
|
||||
fails.append("shared %s is MISSING beside this clone - a failure, never a skip" % inv)
|
||||
return
|
||||
GO = 'package infra\nconst (\n\tTraefikImage = "traefik:v3.7.13"\n\tFooImage = "foo/foo:1.0"\n)\n'
|
||||
TABLE = "# header\nTraefikImage github:traefik/traefik\nFooImage github:foo/foo\n"
|
||||
DOCKER = ("# upstream: osv:Go/stdlib\nFROM golang:1.27.3 AS builder\nRUN go build\n"
|
||||
"# upstream: gap:Debian fixes are DSAs the watch does not read\nFROM debian:13.7-slim\n"
|
||||
"COPY --from=builder /x /x\nFROM builder AS again\n")
|
||||
ws = tempfile.mkdtemp(prefix="ctrl-upstream-")
|
||||
try:
|
||||
def run(name, go, table, docker, expect, must):
|
||||
global ran
|
||||
root = os.path.join(ws, "repo")
|
||||
shutil.rmtree(root, ignore_errors=True)
|
||||
os.makedirs(os.path.join(root, "controller", "internal", "infra"))
|
||||
for rel, body in (("controller/internal/infra/infra.go", go), ("controller/internal/infra/upstream.txt", table),
|
||||
("controller/Dockerfile", docker)):
|
||||
io.open(os.path.join(root, rel), "w", encoding="utf-8").write(body)
|
||||
ran += 1
|
||||
rc, out = _run([sys.executable, inv, "--gate=controller", root], REPO)
|
||||
if rc != expect or must not in out:
|
||||
fails.append("upstream: %s: rc=%d (want %d), %r missing\n%s" % (name, rc, expect, must, out[-500:]))
|
||||
else:
|
||||
print(" ok upstream: %s" % name)
|
||||
run("GENUINE: every const and FROM named (a stage FROM needs none)", GO, TABLE, DOCKER, 0, "upstream gate OK")
|
||||
run("FACT: a new *Image const with no table line", GO.replace(")", '\tBarImage = "bar/bar:2.0"\n)'), TABLE, DOCKER,
|
||||
1, "BarImage")
|
||||
run("FACT: the table line only in a COMMENT", GO, TABLE.replace("FooImage ", "# FooImage "), DOCKER, 1, "FooImage")
|
||||
run("FACT: the upstream comment one line too high", GO, TABLE,
|
||||
DOCKER.replace("# upstream: osv:Go/stdlib\nFROM golang", "# upstream: osv:Go/stdlib\n# builder\nFROM golang"),
|
||||
1, "golang:1.27.3")
|
||||
run("FACT: a new FROM with no upstream comment", GO, TABLE, DOCKER + "FROM alpine:3.24.2\n", 1, "alpine:3.24.2")
|
||||
ran += 1
|
||||
rc, out = _run([sys.executable, inv, "--gate=controller", REPO], REPO)
|
||||
if rc != 0 or "upstream gate OK" not in out:
|
||||
fails.append("upstream: GENUINE: this repo: rc=%d\n%s" % (rc, out[-500:]))
|
||||
else:
|
||||
print(" ok upstream: GENUINE: this repo")
|
||||
finally:
|
||||
shutil.rmtree(ws, ignore_errors=True)
|
||||
|
||||
|
||||
upstream_cases()
|
||||
|
||||
print()
|
||||
if fails:
|
||||
for f in fails:
|
||||
|
||||
Reference in new issue
Block a user