diff --git a/CHANGELOG.md b/CHANGELOG.md index 974040b3..b12db45d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,9 @@ ## Unreleased (2026-10-28 night) — four restore edges (felhom.eu R-1256, review round 2 HUNT-1 H4/H5/H6/H8); commit only, rides the next release **MinAgent: 0.158.1** (unchanged). Each edge: a test seen failing first, then the fix; live proof on 9202 in the felhom.eu -night report (`audits/night-2026-10-28/`). +night report (`audits/night-2026-10-28/`). **Live 2026-10-29 on 9202:** H5's busy volume (not written into, the live data +put back, no hold) and H8 (the 03:29 sweep removed the two old copies, kept the young one). H4, H5's failed stop and H6: +tests only — the attended live proof (or a revert of those two branches) is owed. - **H4 — `internal/backup/offbox_reconstitute.go`:** an off-site restore whose definition write (or database-password write) fails no longer starts the app at once. The writer writes the compose BEFORE the app's settings, so the stack diff --git a/CONTEXT.md b/CONTEXT.md index 0f452242..a641de33 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -1,5 +1,12 @@ # CONTEXT.md — Project Memory +> **2026-10-28 night (unattended) — R-1256, R-1254 (controller half), R-528 and the unknown-app 404 on `main`, NOT +> released (`342a394`, `e33cffa`, `eed23e7`, `20c5877`; MinAgent unchanged).** Restore ops begin through +> `Manager.TryBeginRestoreOp` / `web.beginRestoreOp` only (`TestR1256_H6_NoHandlerBeginsARestoreUnchecked` refuses a direct +> `BeginRestoreOp` in a handler). **Decided by CC unattended — operator may reverse:** an unused restore copy goes after +> 14 days (`restoreCopyUnusedAge`; felhom.eu `07` §6.3). H4 and H5's failed stop: tests only, live proof owed. H5 (busy +> volume) proven live on 9202, R-528 on Tester 1, with the local test tag `0.347.0-rc.n1028` (never pushed). + > **2026-10-26 (evening) — D53 ruled (felhom.eu `09` §3 decision 265): held apps reach the OPERATOR first.** Next here: > the controller reports its holds (stack, reason, since) to the hub as `app_held` / `app_hold_cleared` events plus a > daily re-send (felhom.eu `08` §10). Commit only; the release rides a later attended run. diff --git a/REPORT.md b/REPORT.md index 4c6dfa75..71b5baa8 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,29 +1,39 @@ -# REPORT — controller v0.346.0 RELEASED; the line-test hold fixed the same day (2026-10-27, attended) +# REPORT — restore edges, the escrow timeout, the host OOM count: on main, NOT released (night 2026-10-28 → 29, unattended) -**Plain words first.** Controller 0.346.0 is out on every box (global floor 15:37 CET, MinAgent 0.158.1). It carries -last night's data-safety fixes and one fix made today: a box that seeds torrents all day now gets its weekly line test. -The old hold asked the line test too early — qBittorrent keeps sending for about 14 seconds after it is slowed — so -the test was refused every time. Now the controller waits until the app has really slowed down. +**Plain words first.** Four commits are on main, each with CI green and each new test seen failing first. Nothing is +released: the night brief allowed no release. The controller release goes on the morning list. It needs no new MinAgent: +with an older agent the OOM count answers 404 and the controller waits 10 minutes before it asks again. -## What 0.346.0 carries +**Decided by CC unattended — operator may reverse:** an unused restore copy goes after 14 days (options: 14 days, for +ever, 3 days; `07` §6.3 [DESIGN] 2026-10-28). H4 and H5's failed-stop branch keep only test proofs; their live proof is +owed to an attended session (or revert those two branches of `342a394`). -D53 held-app reports (R-1291), R-1220, R-1218 + R-1219, R-1187 (a)(b)(d)(f)(g) and the two import defects, R-1298, -Go 1.27.3 (R-972), and **R-1250** (`710ceac`). CHANGELOG `## v0.346.0`. +## Commits -## R-1250 (`710ceac`) +| Commit | CI (run id / #) | What | +|---|---|---| +| `342a394` | 2469 / #532 green | **R-1256** four restore edges. H4: an off-site restore whose definition write fails writes the LIVE definition back before the start; a failed write-back holds the app. H5: a put-back whose stop fails pours nothing, keeps the copy, holds the app; a failed definition write-back keeps the copy (D20); a `volume rm -f` that fails fails that volume (no merge on top of the old content). H6: a restore begins in ONE step with the check for a restore, the backup run and a second-drive copy (`TryBeginRestoreOp`); every restore button and both kept loads begin that way. H8: `SweepUnusedRestoreCopies`, every 6 h. | +| `e33cffa` | 2472 / #533 green | **R-1254** controller half: escrow claim/confirm wait 60 s (the agent's own upload wait is longer than the old 15 s); an unknown outcome is 504 `api.escrow.confirm_store_unknown` („keep both codes"), hu + en. | +| `eed23e7` | 2474 / #534 green | **R-528**: the 30-s OOM scan also reads the host's per-container `oom_kill` (agent `GET /guest/oom-kills`); a counted kill raises `app_oom` whatever Docker's flag says; `source` in the details. | +| `20c5877` | 2475 / #535 green | An action on a name that is no app answers 404, not „settings cannot be saved" (found walking the capability map). | -- Before (0.345.1 on demo-hp 9201, a throwaway qBittorrent seeding to DooPlex): four holds 11:36–12:21 CET, each - refused „not quiet" 4.0–4.2 Mbit/s. Per-second reads: qBittorrent reports its limit 1024 within a second but sends - at full speed ~14 s; the forwarder and UDP queues hold nothing (felhom.eu `audits/linetest-2026-10-27/`). -- Fix: `upwatch.waitDrained` — after the hold, poll each held app's own upload (`Holdable.UploadRate`: qBittorrent - `up_info_speed`, Transmission `uploadSpeed`) until ≤ 16 KiB/s twice, 2 s apart; at most 90 s; an unreadable speed - waits 20 s; then ask the agent. Tests `linehold_r1250_drain_test.go` (five), red-proved by four mutations. -- After (0.346.0-rc.1 on 9201): held 13:39:56 → slowed after 22 s → 94.5 Mbit/s → the limit back exactly; the 50 % rule - kept the normal slot (0.15 % move). +Red-proofs: felhom.eu `documentation/audits/night-2026-10-28/redproofs/README.md`. -## Release +## Live proof (test build `0.347.0-rc.n1028`, local tag only, never pushed) -rc.1 by hand on 9201 and Tester 1 (9202 was in another session's use); `b1b9be3` (CI 2423) built as 0.346.0 -(`sha256:3d910b03…`); per-box floors 15:09 CET → all three on 0.346.0 („update successful"); box walk green (start -logs 0 errors, every app healthy, `https_only_check.py` green, held-apps lists match 0 = 0); global floor 15:37 CET, -30-min watch. R-1218's live proof was waived by the operator (felhom.eu `09` §3 decision 270). +- **H5, scratch 9202:** vikunja's first volume held by a created-only container, the household's restore pressed through + the page: „Failed to remove volume vikunja_vikunja_data before the restore … (not written into)", the restore failed, + the put-back poured the live copy of both volumes, the app ran, no hold; a project from before the restore point AND + one from after it read back. +- **H8, scratch 9202:** three copies planted by hand (2026-10-01, 2026-10-05, 2026-10-28); the first sweep at 03:29 UTC + removed the two older than 14 days and kept the young one (8 ms). The young one was then removed by hand. +- **R-528, Tester 1:** one `app_oom` „seen by docker,cgroup", one mail with `"source":"docker,cgroup"`. +- **Not live:** H4 (off-site only; 9202 has no off-site target), H5's failed stop and crash put-back, H6 (the old + code also refused at the early check, so a live press cannot tell the two apart). Tests only. + +Both boxes put back after. **Scratch 9202** (2026-10-29 04:14 UTC): controller 0.346.1 healthy, the test image removed, the +live catalog (config byte-identical to the saved copy), no throwaway app, no planted restore copy, no volume made +tonight. **Tester 1** (2026-10-28 23:30 UTC): controller 0.346.1, agent 0.167.0 (sha256 `28bdd282…`), the test app +removed. The bench is stopped. Hub: nothing provisioned. + +Evidence: felhom.eu `documentation/audits/night-2026-10-28/` (`r1256-restore/`, `r528-oom/`). diff --git a/controller/README.md b/controller/README.md index 45966bc3..b95520dd 100644 --- a/controller/README.md +++ b/controller/README.md @@ -1034,6 +1034,13 @@ Each app can define rich metadata in `.felhom.yml`: `healthy`; the copy then goes after a further 3 min of the same (10 s samples, watched up to 30 min; a new restore of the app ends the watch). A restore whose app does not start reads „…még nem indult el…", keeps the copy and alarms the operator once (D49 = A: nothing is poured back by itself). `restore_verdict.go`. `offbox_restore_undo.go`. + - **Restore edges (unreleased, R-1256, night 2026-10-28).** A put-back whose stop fails pours nothing, keeps the copy + and holds the app; a failed definition write-back keeps the copy; a volume whose `docker volume rm -f` fails is a + failed volume (never unpacked on top of the old content). An off-site restore whose definition write fails writes + the LIVE definition back before the start, or holds the app. Every restore begins in one step with the check for a + restore, the backup run and a second-drive copy (`TryBeginRestoreOp`, also both kept loads). A copy that nothing + needs (no hold of any kind, not named by the app-stop marker, no restore running) goes after 14 days, checked every + 6 h (`restore_copy_sweep.go`; decided by CC unattended — operator may reverse). - **A restore cut by a power loss, on a drive that comes back late (unreleased, R-1251).** The copy of the live data lives on the app's drive. When it cannot be reached at start because the drive is not bound yet, the app is HELD (`putback_pending`; „…A visszaállítás előtti adatai egy meghajtón vannak, amely most nem érhető el…") and no @@ -1488,6 +1495,10 @@ The nightly backup has two phases that run sequentially. All paths are **per-dri > **OOM visibility (v0.243.0, R-514).** The 30 s dead-app check also reads `State.OOMKilled` for running app containers (`Manager.ScanOOMKilled`); the dashboard shows „Memória elfogyott" and the hub gets `app_oom` once per container run. The controller does not restart the app. > **OOM storm (v0.265.0, R-636).** For a flagged container the scan also reads the kernel's own kill counter (`memory.events` `oom_kill`, plus `memory.max` / `memory.peak`) with one `docker exec … cat` — `OOMKilled` is sticky, so it cannot count. When the SAME container run's counter rises by **20 or more within 30 minutes**, the notifier sends **`app_oom_storm`** (severity `error`, operator-only, details `{app, container, kills, window_min, mem_limit, peak}`) — ONCE per container run. `app_oom` itself is unchanged. An unreadable counter (an image without `cat`) never escalates. +> **Host OOM count (unreleased, R-528, night 2026-10-28).** Docker inside an LXC guest misses real OOM kills. The scan +> also asks the agent for the host's per-container `oom_kill` (`GET /guest/oom-kills`, 5 s cap; a 404 from an older +> agent → 10 minutes before the next ask); a count above zero raises `app_oom` whatever `OOMKilled` says, with `source` +> (`docker`, `cgroup` or `docker,cgroup`) in the details. `cmd/controller/hostoom_wiring.go`, `internal/stacks/oom.go`. > **Multi-tier whole-guest backup (v0.174.0, R-82 Slice B).** The agent can serve SEVERAL whole-guest > backup tiers with independent cadences — "local daily + PBS weekly" (agent >= v0.97.0,