felhom.eu/REPORT.md

ADDENDUM (same session) — R-649, the operator's ruling: a failed install removes what it started (controller v0.266.0)

Ruling: "On failed install, the controller should clean up the containers." Built in 964ae75: runComposeDeploy's failure branch runs compose down (no -v — volumes kept) before recording not-deployed. Red-proof: the down removed → TestR649_AFailedInstallRemovesWhatItStarted fails with calls=["up -d"]. Live on 9202 (audits/r649-2026-09-23/): outline with a never-healthy redis (drill template only) failed after 102.7 s → 0 containers left, 3 volumes kept, log line the failed deploy's containers were removed (volumes kept) — R-649. Floor 0.266.0 read back; both demo boxes on it in 20 s. R-649 closed; open rows 331 → 330. Teardown: outline removed, images by name, config identical to the saved copy, live catalog, drill reset. Nothing else touched.

---

REPORT — clean-up evening: R-634's cause, held apps, the OOM storm (controller v0.265.0, hub v0.121.0)

2026-09-23 (late evening). Baselines verified live: controller 0a3026180ae8 (v0.264.0), agent d9864a94bf62, felhom.eu 21f17ed32bdc (hub v0.120.0), catalog cfcfe5278428 — as the brief said. Repos touched: felhom-controller (v0.265.0 0054d4b), felhom.eu (hub v0.121.0, manifest, docs, register, evidence), admin/app-catalog-drill (drill commits, reset to live main). felhom-agent, app-catalog-felhom.eu: untouched. Architecture read and named: 09-update-architecture.md §6.1/§6.1a/§6.4, 08-alarm-ladder.md §4/§6.2, 02-controller-module-map.md (stacks ↔ backup seam). Evidence: documentation/audits/cleanup-2026-09-23/README.md. Method: endpoint-level (the endpoints the UI invokes; no browser).

---

1. Not done, or changed

item	state
Part 1 — R-634	cause found in ~25 min (cap 2 h), fixed, proven live. One design question left → R-649 (operator)
Part 2 — R-625	done. The Update button was ALREADY hidden for a held app on the list (since v0.238.0); the defect was the badge only. Now pinned by a test with a positive control. The badge applies to both hold kinds (a failed restore holds the app stopped too; the way back is also a restore) — the brief said update_failed only
Part 3 — R-636	done, and the brief's rule changed shape: "the same key re-fires ≥ 20 times" cannot work — OOMKilled is sticky, so the key re-fires every 30 s after ONE kill (a hiccup would storm after 10 min). The controller now counts real kills (kernel oom_kill); the 20-in-30-min threshold is applied to THAT
Part 3 — hub live curl	not done: posting a synthetic event with a real customer's key would put a fabricated alarm in the operator's record; the hub side is proven by unit tests
Part 4 — R-647	done
Part 5 — R-648	done — no per-app endpoint exists, so the drill presses nothing
Part 6 — floor	raised to 0.265.0 — every live proof passed
a mistake of mine	my first draft of the R-634 backup test drove the real dump and created an empty volume outline_outline_data on DooPlex; my inspection of it pulled alpine:3.20. Both verified new and unused, removed by name. Tests rewritten on seams. R-650 filed (the class is unguarded)

Claims in the brief, checked:

1. "sparkyfitness reproduces alone at today's pin" — wrong. Deployed alone in 47.3 s on v0.264.0 (10-*). The 2026-09-22 "alone" walk pressed the whole-box backup itself — the same race.
2. "a whole-box backup touches a DEPLOYING stack at all (never read)" — true, and it is THE cause.
3. "20 kills in 30 minutes separates a storm from a hiccup" — true only if kills are counted, and they were not. RomM's real rate: 4,530 kills in 6 h ≈ 375 per 30 min; a hiccup is 1–3. 20 kept, applied to the kernel's counter. Live: 21 kills 2.5 min after start → one storm.
4. "a per-app backup endpoint exists" — wrong. router.go has /backup/run and /backup/tier2 only; the per-app backup (RunAppBackupNow) is reachable only through the guarded update.

2. R-634 — the mechanism, at file:line

1. stacks/deploy.go:395 — DeployStack sets the in-memory Deployed=true at ACCEPT (no-stale-button UX).
2. cmd/controller/main.go:2556 — ListDeployedStacks filtered on that flag only → a deploying app is in every backup leg's list.
3. backup/backup.go:693 → DumpAppVolumesSafe — StopStack (compose down) in the middle of the deploy, tar of half-made volumes, StartStack (a SECOND compose up -d, backup.go:918).
4. Both up calls fail together; stacks/deploy.go:421-435 writes Deployed=false without asking whether containers exist. Which up wins decides the end: running under „not deployed" (2026-09-22) or Created (tonight, 14:39:48). No deploy time limit exists — shape (i) ruled out.

Fix: deploying apps leave the list; the volume leg re-asks right before the stop (SKIP, never FAIL); StopStack/StartStack refuse a deploying stack (ErrStackDeploying) — the backstop for quiesce, restore, export, storage and the Stop button, which all had the same exposure. Live (32-*, 33-*): outline's image removed so the deploy must pull, backup at +5 s: the backup ran 15:23:07–15:23:31, stopped gokapi, paperless-ngx and privatebin, never outline; deployed successfully (took 48.9s). The first fix run is kept and labelled NOT A RACE (images cached, 6.3 s).

3. Red-proofs (each seen failing, mutation restored, === RUN checked)

#	mutation	test → failure
1	backup skip removed	TestR634_VolumeLegNeverStopsADeployingApp — dumped=[outline]
2	StopStack guard removed	TestR634_StopAndStartRefuseADeployingStack — exit code -1 instead of ErrStackDeploying
3	held badge removed (v0.264.0's shape)	TestR625_AHeldAppSaysStoppedRestoreNeeded — badge missing, „Frissítés elérhető" present
4	reader funcs off	TestR647_AReaderInTheOtherLanguageReadsTheHoldInTheirs — English hold for the Hungarian reader
5	Hungarian phrase wired back	TestR647_HeldEventCarriesTheCopyHoldsKey
6	health status passed as severity	TestR647_DisabledHealthChangeNamesTheSeverity — severity warn
7	escalation removed	TestR636_TwentyKillsInThirtyMinutesIsOneStorm — kills=20 … storms=0, want 1
8	counter never read	TestR636_ScanReadsTheCounterOfFlaggedContainersOnly — Kills:-1
9	hub: not operator-only	TestAppOOMStormIsAllowlistedAndOperatorOnly — "must be operator-only"
10	hub: not allow-listed	same test — "must be in allowedEventTypes"
11	hub: no per-app cooldown	TestR389_TheAllowListHasExactlyOneMember

Also pinned without a mutation run: 19 kills → no storm, 200 → one, one hiccup over 6 h → none, 20 kills over 3 h → none, an unreadable counter → never. Gates: controller go test ./... rc=0 + controller_gates.py rc=0; hub go test ./... rc=0 + repo_gates.py rc=0. Parity: app_info_deployed and stacks_full regenerated — measured diff exactly one line each, the new held badge.

4. Live proofs in both languages (9202)

• R-625 + R-647 (1) (44-*), every box/reader pair: hu reader „Megállítva — visszaállítás szükséges", title „A frissítés nem sikerült, és az automatikus visszaállítás sem."; en reader "Stopped — restore needed", title "The update did not succeed, and the automatic undo did not either."; with the box in ENGLISH the Hungarian reader reads Hungarian (page and API). No vikunja Update button; other apps keep theirs (control). POST …/update → 409 held.
• R-636 (45-*, 46-*): romm at 320M — kills 8, 13, 21 → OOM STORM — 21 kills in 30 min (limit 320M, peak 320M) + DROPPED event app_oom_storm (severity error); one storm line at 49 kills.
• R-648 (43-*): vikunja's update ran its own backing-up phase.

5. Rows

Closed (5): R-634, R-625, R-636, R-647, R-648. Opened (2): R-649 (operator question — a deploy that fails on its own), R-650 (tests that reach real docker act on DooPlex). Open rows 334 → 331. 09 §6.4 parts 8 and 9 marked SHIPPED; 08 §6.2 has the storm rung and the two per-app grains.

6. Floor

0.265.0 / MinAgent 0.131.0, read back from the hub (min_controller_version=0.265.0, min_agent=0.131.0; managed floor SERVED for demo-felhom and demo-hp); both on 0.265.0 in 20 s.

7. Teardown — three layers

• machine (9202): all four test apps removed through the product; romm's drive folder by name (R-442 fence, as always); 0 volumes, 0 undo copies; test images removed by name where unused; config identical to the saved copy; live catalog; language hu; recorders stopped. 9201 not touched.
• host: nothing provisioned. hub: v0.121.0 + floor (planned). DooPlex: the volume and image above, removed. drill repo: reset to live main. Password files deleted from the scratchpad.