Night 2026-10-28 docs: REPORT (H5 + H8 live on 9202, teardown), README (restore edges, host OOM count — unreleased), CONTEXT, CHANGELOG live line
gates / gates (push) Successful in 29s

This commit is contained in:
admin committed 2026-10-29 05:14:05 +01:00
1 parent 20c5877056
commit a815b37fbd
4 files changed
+53 -23

No files matched your search

+3 -1
View File
@@ -1,7 +1,9 @@
## Unreleased (2026-10-28 night) — four restore edges (felhom.eu R-1256, review round 2 HUNT-1 H4/H5/H6/H8); commit only, rides the next release
**MinAgent: 0.158.1** (unchanged). Each edge: a test seen failing first, then the fix; live proof on 9202 in the felhom.eu
night report (`audits/night-2026-10-28/`).
night report (`audits/night-2026-10-28/`). **Live 2026-10-29 on 9202:** H5's busy volume (not written into, the live data
put back, no hold) and H8 (the 03:29 sweep removed the two old copies, kept the young one). H4, H5's failed stop and H6:
tests only — the attended live proof (or a revert of those two branches) is owed.
- **H4 — `internal/backup/offbox_reconstitute.go`:** an off-site restore whose definition write (or database-password
write) fails no longer starts the app at once. The writer writes the compose BEFORE the app's settings, so the stack
+7
View File
@@ -1,5 +1,12 @@
# CONTEXT.md — Project Memory
> **2026-10-28 night (unattended) — R-1256, R-1254 (controller half), R-528 and the unknown-app 404 on `main`, NOT
> released (`342a394`, `e33cffa`, `eed23e7`, `20c5877`; MinAgent unchanged).** Restore ops begin through
> `Manager.TryBeginRestoreOp` / `web.beginRestoreOp` only (`TestR1256_H6_NoHandlerBeginsARestoreUnchecked` refuses a direct
> `BeginRestoreOp` in a handler). **Decided by CC unattended — operator may reverse:** an unused restore copy goes after
> 14 days (`restoreCopyUnusedAge`; felhom.eu `07` §6.3). H4 and H5's failed stop: tests only, live proof owed. H5 (busy
> volume) proven live on 9202, R-528 on Tester 1, with the local test tag `0.347.0-rc.n1028` (never pushed).
> **2026-10-26 (evening) — D53 ruled (felhom.eu `09` §3 decision 265): held apps reach the OPERATOR first.** Next here:
> the controller reports its holds (stack, reason, since) to the hub as `app_held` / `app_hold_cleared` events plus a
> daily re-send (felhom.eu `08` §10). Commit only; the release rides a later attended run.
+32 -22
View File
@@ -1,29 +1,39 @@
# REPORT — controller v0.346.0 RELEASED; the line-test hold fixed the same day (2026-10-27, attended)
# REPORT — restore edges, the escrow timeout, the host OOM count: on main, NOT released (night 2026-10-28 → 29, unattended)
**Plain words first.** Controller 0.346.0 is out on every box (global floor 15:37 CET, MinAgent 0.158.1). It carries
last night's data-safety fixes and one fix made today: a box that seeds torrents all day now gets its weekly line test.
The old hold asked the line test too early — qBittorrent keeps sending for about 14 seconds after it is slowed — so
the test was refused every time. Now the controller waits until the app has really slowed down.
**Plain words first.** Four commits are on main, each with CI green and each new test seen failing first. Nothing is
released: the night brief allowed no release. The controller release goes on the morning list. It needs no new MinAgent:
with an older agent the OOM count answers 404 and the controller waits 10 minutes before it asks again.
## What 0.346.0 carries
**Decided by CC unattended — operator may reverse:** an unused restore copy goes after 14 days (options: 14 days, for
ever, 3 days; `07` §6.3 [DESIGN] 2026-10-28). H4 and H5's failed-stop branch keep only test proofs; their live proof is
owed to an attended session (or revert those two branches of `342a394`).
D53 held-app reports (R-1291), R-1220, R-1218 + R-1219, R-1187 (a)(b)(d)(f)(g) and the two import defects, R-1298,
Go 1.27.3 (R-972), and **R-1250** (`710ceac`). CHANGELOG `## v0.346.0`.
## Commits
## R-1250 (`710ceac`)
| Commit | CI (run id / #) | What |
|---|---|---|
| `342a394` | 2469 / #532 green | **R-1256** four restore edges. H4: an off-site restore whose definition write fails writes the LIVE definition back before the start; a failed write-back holds the app. H5: a put-back whose stop fails pours nothing, keeps the copy, holds the app; a failed definition write-back keeps the copy (D20); a `volume rm -f` that fails fails that volume (no merge on top of the old content). H6: a restore begins in ONE step with the check for a restore, the backup run and a second-drive copy (`TryBeginRestoreOp`); every restore button and both kept loads begin that way. H8: `SweepUnusedRestoreCopies`, every 6 h. |
| `e33cffa` | 2472 / #533 green | **R-1254** controller half: escrow claim/confirm wait 60 s (the agent's own upload wait is longer than the old 15 s); an unknown outcome is 504 `api.escrow.confirm_store_unknown` („keep both codes"), hu + en. |
| `eed23e7` | 2474 / #534 green | **R-528**: the 30-s OOM scan also reads the host's per-container `oom_kill` (agent `GET /guest/oom-kills`); a counted kill raises `app_oom` whatever Docker's flag says; `source` in the details. |
| `20c5877` | 2475 / #535 green | An action on a name that is no app answers 404, not „settings cannot be saved" (found walking the capability map). |
- Before (0.345.1 on demo-hp 9201, a throwaway qBittorrent seeding to DooPlex): four holds 11:36–12:21 CET, each
refused „not quiet" 4.0–4.2 Mbit/s. Per-second reads: qBittorrent reports its limit 1024 within a second but sends
at full speed ~14 s; the forwarder and UDP queues hold nothing (felhom.eu `audits/linetest-2026-10-27/`).
- Fix: `upwatch.waitDrained` — after the hold, poll each held app's own upload (`Holdable.UploadRate`: qBittorrent
`up_info_speed`, Transmission `uploadSpeed`) until ≤ 16 KiB/s twice, 2 s apart; at most 90 s; an unreadable speed
waits 20 s; then ask the agent. Tests `linehold_r1250_drain_test.go` (five), red-proved by four mutations.
- After (0.346.0-rc.1 on 9201): held 13:39:56 → slowed after 22 s → 94.5 Mbit/s → the limit back exactly; the 50 % rule
kept the normal slot (0.15 % move).
Red-proofs: felhom.eu `documentation/audits/night-2026-10-28/redproofs/README.md`.
## Release
## Live proof (test build `0.347.0-rc.n1028`, local tag only, never pushed)
rc.1 by hand on 9201 and Tester 1 (9202 was in another session's use); `b1b9be3` (CI 2423) built as 0.346.0
(`sha256:3d910b03…`); per-box floors 15:09 CET → all three on 0.346.0 („update successful"); box walk green (start
logs 0 errors, every app healthy, `https_only_check.py` green, held-apps lists match 0 = 0); global floor 15:37 CET,
30-min watch. R-1218's live proof was waived by the operator (felhom.eu `09` §3 decision 270).
- **H5, scratch 9202:** vikunja's first volume held by a created-only container, the household's restore pressed through
the page: „Failed to remove volume vikunja_vikunja_data before the restore … (not written into)", the restore failed,
the put-back poured the live copy of both volumes, the app ran, no hold; a project from before the restore point AND
one from after it read back.
- **H8, scratch 9202:** three copies planted by hand (2026-10-01, 2026-10-05, 2026-10-28); the first sweep at 03:29 UTC
removed the two older than 14 days and kept the young one (8 ms). The young one was then removed by hand.
- **R-528, Tester 1:** one `app_oom` „seen by docker,cgroup", one mail with `"source":"docker,cgroup"`.
- **Not live:** H4 (off-site only; 9202 has no off-site target), H5's failed stop and crash put-back, H6 (the old
code also refused at the early check, so a live press cannot tell the two apart). Tests only.
Both boxes put back after. **Scratch 9202** (2026-10-29 04:14 UTC): controller 0.346.1 healthy, the test image removed, the
live catalog (config byte-identical to the saved copy), no throwaway app, no planted restore copy, no volume made
tonight. **Tester 1** (2026-10-28 23:30 UTC): controller 0.346.1, agent 0.167.0 (sha256 `28bdd282…`), the test app
removed. The bench is stopped. Hub: nothing provisioned.
Evidence: felhom.eu `documentation/audits/night-2026-10-28/` (`r1256-restore/`, `r528-oom/`).
+11
View File
@@ -1034,6 +1034,13 @@ Each app can define rich metadata in `.felhom.yml`:
`healthy`; the copy then goes after a further 3 min of the same (10 s samples, watched up to 30 min; a new restore
of the app ends the watch). A restore whose app does not start reads „…még nem indult el…", keeps the copy and
alarms the operator once (D49 = A: nothing is poured back by itself). `restore_verdict.go`. `offbox_restore_undo.go`.
- **Restore edges (unreleased, R-1256, night 2026-10-28).** A put-back whose stop fails pours nothing, keeps the copy
and holds the app; a failed definition write-back keeps the copy; a volume whose `docker volume rm -f` fails is a
failed volume (never unpacked on top of the old content). An off-site restore whose definition write fails writes
the LIVE definition back before the start, or holds the app. Every restore begins in one step with the check for a
restore, the backup run and a second-drive copy (`TryBeginRestoreOp`, also both kept loads). A copy that nothing
needs (no hold of any kind, not named by the app-stop marker, no restore running) goes after 14 days, checked every
6 h (`restore_copy_sweep.go`; decided by CC unattended — operator may reverse).
- **A restore cut by a power loss, on a drive that comes back late (unreleased, R-1251).** The copy of the live
data lives on the app's drive. When it cannot be reached at start because the drive is not bound yet, the app is
HELD (`putback_pending`; „…A visszaállítás előtti adatai egy meghajtón vannak, amely most nem érhető el…") and no
@@ -1488,6 +1495,10 @@ The nightly backup has two phases that run sequentially. All paths are **per-dri
> **OOM visibility (v0.243.0, R-514).** The 30 s dead-app check also reads `State.OOMKilled` for running app containers (`Manager.ScanOOMKilled`); the dashboard shows „Memória elfogyott" and the hub gets `app_oom` once per container run. The controller does not restart the app.
> **OOM storm (v0.265.0, R-636).** For a flagged container the scan also reads the kernel's own kill counter (`memory.events` `oom_kill`, plus `memory.max` / `memory.peak`) with one `docker exec … cat` — `OOMKilled` is sticky, so it cannot count. When the SAME container run's counter rises by **20 or more within 30 minutes**, the notifier sends **`app_oom_storm`** (severity `error`, operator-only, details `{app, container, kills, window_min, mem_limit, peak}`) — ONCE per container run. `app_oom` itself is unchanged. An unreadable counter (an image without `cat`) never escalates.
> **Host OOM count (unreleased, R-528, night 2026-10-28).** Docker inside an LXC guest misses real OOM kills. The scan
> also asks the agent for the host's per-container `oom_kill` (`GET /guest/oom-kills`, 5 s cap; a 404 from an older
> agent → 10 minutes before the next ask); a count above zero raises `app_oom` whatever `OOMKilled` says, with `source`
> (`docker`, `cgroup` or `docker,cgroup`) in the details. `cmd/controller/hostoom_wiring.go`, `internal/stacks/oom.go`.
> **Multi-tier whole-guest backup (v0.174.0, R-82 Slice B).** The agent can serve SEVERAL whole-guest
> backup tiers with independent cadences — "local daily + PBS weekly" (agent >= v0.97.0,