Night 2026-10-28 docs: REPORT (H5 + H8 live on 9202, teardown), README (restore edges, host OOM count — unreleased), CONTEXT, CHANGELOG live line
gates / gates (push) Successful in 29s

This commit is contained in:
admin committed 2026-10-29 05:14:05 +01:00
1 parent 20c5877056
commit a815b37fbd
4 files changed
+53 -23

No files matched your search

+3 -1
View File
@@ -1,7 +1,9 @@
## Unreleased (2026-10-28 night) — four restore edges (felhom.eu R-1256, review round 2 HUNT-1 H4/H5/H6/H8); commit only, rides the next release ## Unreleased (2026-10-28 night) — four restore edges (felhom.eu R-1256, review round 2 HUNT-1 H4/H5/H6/H8); commit only, rides the next release
**MinAgent: 0.158.1** (unchanged). Each edge: a test seen failing first, then the fix; live proof on 9202 in the felhom.eu **MinAgent: 0.158.1** (unchanged). Each edge: a test seen failing first, then the fix; live proof on 9202 in the felhom.eu
night report (`audits/night-2026-10-28/`). night report (`audits/night-2026-10-28/`). **Live 2026-10-29 on 9202:** H5's busy volume (not written into, the live data
put back, no hold) and H8 (the 03:29 sweep removed the two old copies, kept the young one). H4, H5's failed stop and H6:
tests only — the attended live proof (or a revert of those two branches) is owed.
- **H4 — `internal/backup/offbox_reconstitute.go`:** an off-site restore whose definition write (or database-password - **H4 — `internal/backup/offbox_reconstitute.go`:** an off-site restore whose definition write (or database-password
write) fails no longer starts the app at once. The writer writes the compose BEFORE the app's settings, so the stack write) fails no longer starts the app at once. The writer writes the compose BEFORE the app's settings, so the stack
+7
View File
@@ -1,5 +1,12 @@
# CONTEXT.md — Project Memory # CONTEXT.md — Project Memory
> **2026-10-28 night (unattended) — R-1256, R-1254 (controller half), R-528 and the unknown-app 404 on `main`, NOT
> released (`342a394`, `e33cffa`, `eed23e7`, `20c5877`; MinAgent unchanged).** Restore ops begin through
> `Manager.TryBeginRestoreOp` / `web.beginRestoreOp` only (`TestR1256_H6_NoHandlerBeginsARestoreUnchecked` refuses a direct
> `BeginRestoreOp` in a handler). **Decided by CC unattended — operator may reverse:** an unused restore copy goes after
> 14 days (`restoreCopyUnusedAge`; felhom.eu `07` §6.3). H4 and H5's failed stop: tests only, live proof owed. H5 (busy
> volume) proven live on 9202, R-528 on Tester 1, with the local test tag `0.347.0-rc.n1028` (never pushed).
> **2026-10-26 (evening) — D53 ruled (felhom.eu `09` §3 decision 265): held apps reach the OPERATOR first.** Next here: > **2026-10-26 (evening) — D53 ruled (felhom.eu `09` §3 decision 265): held apps reach the OPERATOR first.** Next here:
> the controller reports its holds (stack, reason, since) to the hub as `app_held` / `app_hold_cleared` events plus a > the controller reports its holds (stack, reason, since) to the hub as `app_held` / `app_hold_cleared` events plus a
> daily re-send (felhom.eu `08` §10). Commit only; the release rides a later attended run. > daily re-send (felhom.eu `08` §10). Commit only; the release rides a later attended run.
+32 -22
View File
@@ -1,29 +1,39 @@
# REPORT — controller v0.346.0 RELEASED; the line-test hold fixed the same day (2026-10-27, attended) # REPORT — restore edges, the escrow timeout, the host OOM count: on main, NOT released (night 2026-10-28 → 29, unattended)
**Plain words first.** Controller 0.346.0 is out on every box (global floor 15:37 CET, MinAgent 0.158.1). It carries **Plain words first.** Four commits are on main, each with CI green and each new test seen failing first. Nothing is
last night's data-safety fixes and one fix made today: a box that seeds torrents all day now gets its weekly line test. released: the night brief allowed no release. The controller release goes on the morning list. It needs no new MinAgent:
The old hold asked the line test too early — qBittorrent keeps sending for about 14 seconds after it is slowed — so with an older agent the OOM count answers 404 and the controller waits 10 minutes before it asks again.
the test was refused every time. Now the controller waits until the app has really slowed down.
## What 0.346.0 carries **Decided by CC unattended — operator may reverse:** an unused restore copy goes after 14 days (options: 14 days, for
ever, 3 days; `07` §6.3 [DESIGN] 2026-10-28). H4 and H5's failed-stop branch keep only test proofs; their live proof is
owed to an attended session (or revert those two branches of `342a394`).
D53 held-app reports (R-1291), R-1220, R-1218 + R-1219, R-1187 (a)(b)(d)(f)(g) and the two import defects, R-1298, ## Commits
Go 1.27.3 (R-972), and **R-1250** (`710ceac`). CHANGELOG `## v0.346.0`.
## R-1250 (`710ceac`) | Commit | CI (run id / #) | What |
|---|---|---|
| `342a394` | 2469 / #532 green | **R-1256** four restore edges. H4: an off-site restore whose definition write fails writes the LIVE definition back before the start; a failed write-back holds the app. H5: a put-back whose stop fails pours nothing, keeps the copy, holds the app; a failed definition write-back keeps the copy (D20); a `volume rm -f` that fails fails that volume (no merge on top of the old content). H6: a restore begins in ONE step with the check for a restore, the backup run and a second-drive copy (`TryBeginRestoreOp`); every restore button and both kept loads begin that way. H8: `SweepUnusedRestoreCopies`, every 6 h. |
| `e33cffa` | 2472 / #533 green | **R-1254** controller half: escrow claim/confirm wait 60 s (the agent's own upload wait is longer than the old 15 s); an unknown outcome is 504 `api.escrow.confirm_store_unknown` („keep both codes"), hu + en. |
| `eed23e7` | 2474 / #534 green | **R-528**: the 30-s OOM scan also reads the host's per-container `oom_kill` (agent `GET /guest/oom-kills`); a counted kill raises `app_oom` whatever Docker's flag says; `source` in the details. |
| `20c5877` | 2475 / #535 green | An action on a name that is no app answers 404, not „settings cannot be saved" (found walking the capability map). |
- Before (0.345.1 on demo-hp 9201, a throwaway qBittorrent seeding to DooPlex): four holds 11:36–12:21 CET, each Red-proofs: felhom.eu `documentation/audits/night-2026-10-28/redproofs/README.md`.
refused „not quiet" 4.0–4.2 Mbit/s. Per-second reads: qBittorrent reports its limit 1024 within a second but sends
at full speed ~14 s; the forwarder and UDP queues hold nothing (felhom.eu `audits/linetest-2026-10-27/`).
- Fix: `upwatch.waitDrained` — after the hold, poll each held app's own upload (`Holdable.UploadRate`: qBittorrent
`up_info_speed`, Transmission `uploadSpeed`) until ≤ 16 KiB/s twice, 2 s apart; at most 90 s; an unreadable speed
waits 20 s; then ask the agent. Tests `linehold_r1250_drain_test.go` (five), red-proved by four mutations.
- After (0.346.0-rc.1 on 9201): held 13:39:56 → slowed after 22 s → 94.5 Mbit/s → the limit back exactly; the 50 % rule
kept the normal slot (0.15 % move).
## Release ## Live proof (test build `0.347.0-rc.n1028`, local tag only, never pushed)
rc.1 by hand on 9201 and Tester 1 (9202 was in another session's use); `b1b9be3` (CI 2423) built as 0.346.0 - **H5, scratch 9202:** vikunja's first volume held by a created-only container, the household's restore pressed through
(`sha256:3d910b03…`); per-box floors 15:09 CET → all three on 0.346.0 („update successful"); box walk green (start the page: „Failed to remove volume vikunja_vikunja_data before the restore … (not written into)", the restore failed,
logs 0 errors, every app healthy, `https_only_check.py` green, held-apps lists match 0 = 0); global floor 15:37 CET, the put-back poured the live copy of both volumes, the app ran, no hold; a project from before the restore point AND
30-min watch. R-1218's live proof was waived by the operator (felhom.eu `09` §3 decision 270). one from after it read back.
- **H8, scratch 9202:** three copies planted by hand (2026-10-01, 2026-10-05, 2026-10-28); the first sweep at 03:29 UTC
removed the two older than 14 days and kept the young one (8 ms). The young one was then removed by hand.
- **R-528, Tester 1:** one `app_oom` „seen by docker,cgroup", one mail with `"source":"docker,cgroup"`.
- **Not live:** H4 (off-site only; 9202 has no off-site target), H5's failed stop and crash put-back, H6 (the old
code also refused at the early check, so a live press cannot tell the two apart). Tests only.
Both boxes put back after. **Scratch 9202** (2026-10-29 04:14 UTC): controller 0.346.1 healthy, the test image removed, the
live catalog (config byte-identical to the saved copy), no throwaway app, no planted restore copy, no volume made
tonight. **Tester 1** (2026-10-28 23:30 UTC): controller 0.346.1, agent 0.167.0 (sha256 `28bdd282…`), the test app
removed. The bench is stopped. Hub: nothing provisioned.
Evidence: felhom.eu `documentation/audits/night-2026-10-28/` (`r1256-restore/`, `r528-oom/`).
+11
View File
@@ -1034,6 +1034,13 @@ Each app can define rich metadata in `.felhom.yml`:
`healthy`; the copy then goes after a further 3 min of the same (10 s samples, watched up to 30 min; a new restore `healthy`; the copy then goes after a further 3 min of the same (10 s samples, watched up to 30 min; a new restore
of the app ends the watch). A restore whose app does not start reads „…még nem indult el…", keeps the copy and of the app ends the watch). A restore whose app does not start reads „…még nem indult el…", keeps the copy and
alarms the operator once (D49 = A: nothing is poured back by itself). `restore_verdict.go`. `offbox_restore_undo.go`. alarms the operator once (D49 = A: nothing is poured back by itself). `restore_verdict.go`. `offbox_restore_undo.go`.
- **Restore edges (unreleased, R-1256, night 2026-10-28).** A put-back whose stop fails pours nothing, keeps the copy
and holds the app; a failed definition write-back keeps the copy; a volume whose `docker volume rm -f` fails is a
failed volume (never unpacked on top of the old content). An off-site restore whose definition write fails writes
the LIVE definition back before the start, or holds the app. Every restore begins in one step with the check for a
restore, the backup run and a second-drive copy (`TryBeginRestoreOp`, also both kept loads). A copy that nothing
needs (no hold of any kind, not named by the app-stop marker, no restore running) goes after 14 days, checked every
6 h (`restore_copy_sweep.go`; decided by CC unattended — operator may reverse).
- **A restore cut by a power loss, on a drive that comes back late (unreleased, R-1251).** The copy of the live - **A restore cut by a power loss, on a drive that comes back late (unreleased, R-1251).** The copy of the live
data lives on the app's drive. When it cannot be reached at start because the drive is not bound yet, the app is data lives on the app's drive. When it cannot be reached at start because the drive is not bound yet, the app is
HELD (`putback_pending`; „…A visszaállítás előtti adatai egy meghajtón vannak, amely most nem érhető el…") and no HELD (`putback_pending`; „…A visszaállítás előtti adatai egy meghajtón vannak, amely most nem érhető el…") and no
@@ -1488,6 +1495,10 @@ The nightly backup has two phases that run sequentially. All paths are **per-dri
> **OOM visibility (v0.243.0, R-514).** The 30 s dead-app check also reads `State.OOMKilled` for running app containers (`Manager.ScanOOMKilled`); the dashboard shows „Memória elfogyott" and the hub gets `app_oom` once per container run. The controller does not restart the app. > **OOM visibility (v0.243.0, R-514).** The 30 s dead-app check also reads `State.OOMKilled` for running app containers (`Manager.ScanOOMKilled`); the dashboard shows „Memória elfogyott" and the hub gets `app_oom` once per container run. The controller does not restart the app.
> **OOM storm (v0.265.0, R-636).** For a flagged container the scan also reads the kernel's own kill counter (`memory.events` `oom_kill`, plus `memory.max` / `memory.peak`) with one `docker exec … cat` — `OOMKilled` is sticky, so it cannot count. When the SAME container run's counter rises by **20 or more within 30 minutes**, the notifier sends **`app_oom_storm`** (severity `error`, operator-only, details `{app, container, kills, window_min, mem_limit, peak}`) — ONCE per container run. `app_oom` itself is unchanged. An unreadable counter (an image without `cat`) never escalates. > **OOM storm (v0.265.0, R-636).** For a flagged container the scan also reads the kernel's own kill counter (`memory.events` `oom_kill`, plus `memory.max` / `memory.peak`) with one `docker exec … cat` — `OOMKilled` is sticky, so it cannot count. When the SAME container run's counter rises by **20 or more within 30 minutes**, the notifier sends **`app_oom_storm`** (severity `error`, operator-only, details `{app, container, kills, window_min, mem_limit, peak}`) — ONCE per container run. `app_oom` itself is unchanged. An unreadable counter (an image without `cat`) never escalates.
> **Host OOM count (unreleased, R-528, night 2026-10-28).** Docker inside an LXC guest misses real OOM kills. The scan
> also asks the agent for the host's per-container `oom_kill` (`GET /guest/oom-kills`, 5 s cap; a 404 from an older
> agent → 10 minutes before the next ask); a count above zero raises `app_oom` whatever `OOMKilled` says, with `source`
> (`docker`, `cgroup` or `docker,cgroup`) in the details. `cmd/controller/hostoom_wiring.go`, `internal/stacks/oom.go`.
> **Multi-tier whole-guest backup (v0.174.0, R-82 Slice B).** The agent can serve SEVERAL whole-guest > **Multi-tier whole-guest backup (v0.174.0, R-82 Slice B).** The agent can serve SEVERAL whole-guest
> backup tiers with independent cadences — "local daily + PBS weekly" (agent >= v0.97.0, > backup tiers with independent cadences — "local daily + PBS weekly" (agent >= v0.97.0,