felhom-controller/REPORT.md

REPORT — v0.191.0 / .1 / .2: warn before the wall comes down (R-167 · R-158 · R-174)

Overwritten per the standing rule; the prior contents (v0.190.0 / R-157 A · R-170 · R-171, same day) have their durable record in CHANGELOG.md.

Session: 2026-08-02 · Repos: felhom-controller v0.190.0 → v0.191.2, felhom.eu hub v0.88.0 → v0.89.0 + docs. felhom-agent untouched (Part 3 read its golden-build script and changed nothing in it).

---

1. Baselines — one drifted

Repo	§1 said	Found on arrival	Match?
felhom-controller	95eb5c2c1af4, v0.190.0	95eb5c2c1af4, v0.190.0	yes
felhom.eu	d5774d318941, hub v0.87.0	8ef92a3f, hub v0.88.0	NO

hub v0.88.0 had already shipped (R-172, the WAL fix) between the task being written and this session. Target corrected to hub v0.89.0. Highest register ID in use was R-173, not R-171.

2. One decision taken with the operator before any code was written

Reconnaissance found that disk_warning / disk_critical were a complete customer pipeline with no producer — in the hub's allowedEventTypes, carrying Hungarian customerMessages, in settings.DefaultEnabledEvents, with a UI checkbox (event_disk_alerts) — and grep across all four repos found zero emitters. The only thing reacting to guest disk pressure was healthcheck.go:338, folding it into a generic health_degraded at 90% for registered storage paths only.

The task said to mint a new customer-facing type. Doing so would have put a near-duplicate beside an inert pair — and §5's own reason for rejecting backup_failed (it is customer-enabled and carries Hungarian copy) is the argument for reusing this one, because D-c routes the fill warning to the customer. Operator chose: wire the existing pair. So only ONE new hub type was minted (the operator one), and the sixth built-but-never-wired instance was closed rather than joined by a seventh.

3. Files changed and commits

6 modified, 4 added. controller/cmd/controller/main.go, internal/backup/{appstop_marker,backup,recovery_unit}.go, internal/notify/notifier.go, cmd/controller/appstop_wiring_test.go, CHANGELOG.md, REUSE.md · new: internal/fillwatch/{fillwatch.go,fillwatch_test.go}, internal/backup/{appstop_drivegate_test.go,recovery_unit_notify_test.go}

Commit on main	Subject
cf48214	v0.191.0 — warn before the wall comes down (R-167, R-158, R-174)
5adae4d	v0.191.1 — the fill check also runs at startup (R-167)
9a3c485	v0.191.2 — a quiet fill check now says so (R-167)

4. Tests

1157 → 1184 (+27). Full suite green in both repos (go build ./... && go vet ./... && go test ./...); all eight controller gates and all five felhom.eu gates OK. Hub: 574 → 579.

Red-proofs — each mutated, observed failing, restored

#	What was mutated	Result
A1	the errors.Is(err, ErrStartRefused) branch removed from Recover	4 tests FAIL — refusals collapse into Failed and alarm
A2	SetStarter(gatedAppStopStarter{…}) → SetStarter(stackMgr) (the v0.189.0 wiring)	TestMainWiresGatedAppStopStarter FAILS
C	the m.unitNotify(...) call removed from the capture loop	2 tests FAIL — the seam-never-wired shape
E	both edge guards removed	3 tests FAIL — warns twice; the dead zone breaks
G	recovery_unit_capture_failed removed from operatorOnlyEvents	2 tests FAIL, one reading "a customer was emailed the OPERATOR-ONLY …"
H	clear thresholds edited into equality with warn	TestThresholdsKeepTheirHysteresisGap FAILS
J	fillWatcher.SetNotify( commented out (string still in the file)	TestMainWiresTheFillWatcher FAILS — proves AST, not substring

One red-proof was a FALSE GREEN on the first attempt, and it is reported rather than quietly redone. Red-proof E's first mutation removed only if next == prev { continue }; a second guard (next <= prev) still caught the repeat, so every test passed and the "proof" proved nothing. Redone removing both guards. This is exactly the trap CLAUDE.md records — a mutation or a -run filter that yields a green and reads like a completed proof.

One pre-existing test was TIGHTENED, not loosened. TestMainReportsTheInterruptedOperation asserted the guard was if appStopRecovery != nil. R-174 makes that insufficient — Recover now returns non-nil for a refusal-only recovery — so the test now requires Alarming() in the condition.

5. Thresholds and cadence, with their justification

	value	why
warn	used ≥ 85% OR free < 5 GiB	a percentage alone lies at both ends of this fleet's real size range: 85% of a 20 G mp1 leaves 3 G — less than one DB-backed app's unit (~2× its data, §7.5) — while 85% of a 4 TB drive leaves 600 G
critical	used ≥ 95% OR free < 2 GiB	below 2 GiB a volume tar of almost any real app fails: "the next backup will not complete", not "it is getting tight"
clear	used ≤ 75% AND free ≥ 7 GiB	both must hold; the gap to warn is the hysteresis dead zone, so a filesystem on the line does not flap. Pinned with a real margin, not merely an inequality
cadence	daily 03:30 + once 90 s after startup	a fill is slow-moving, so a shorter interval buys no earlier warning; 03:30 precedes the nightly app-data legs, so a customer about to lose a backup hears about it with a night's margin. The startup run exists because neither Daily nor Every fires on registration — §7

The live proof vindicated the two-term design: the critical crossing fired on the free-byte term (1.7 GB) while the disk was only 91% used. A percentage-only rule would have missed it entirely.

6. The exact Hungarian customer copy, for review as copy

disk_warning (severity warning) — as rendered live on 9201:

A(z) „Rendszer- és mentési terület” tároló 85% foglalt — 4,7 GB szabad hely maradt. Kérjük, szabadíts fel helyet, mielőtt megtelik: törölj felesleges fájlokat, vagy csatlakoztass új meghajtót. Ha megtelik, a biztonsági mentések meghiúsulnak.

disk_critical (severity critical) — as rendered live on 9201:

A(z) „Rendszer- és mentési terület” tároló kritikusan megtelt: 1,7 GB szabad hely maradt (91% foglalt). A biztonsági mentések és az alkalmazások írásai bármikor meghiúsulhatnak. Kérjük, mielőbb szabadíts fel helyet: törölj felesleges fájlokat, vagy csatlakoztass új meghajtót.

Decimal comma throughout (4,7 GB); the storage is named by its label, never its path; both end in an action. No emoji (asserted by a test; emoji_gate passes).

7. Two defects found by the validation itself, both fixed and shipped

1. v0.191.1 — the check was reachable only on its daily schedule. Neither sched.Daily nor sched.Every fires on registration; both wait for their first tick. A box that BOOTS already over the line would have stayed silent for up to 24 h — the R-100 shape, and the same gap the hub's own checkers avoid by leaving already-breached keys unseeded at init. Now also runs once 90 s after startup; safe because the check is edge-triggered against persisted state, so an already-warned filesystem stays silent.
2. v0.191.2 — a quiet run was unreadable as evidence. After the customer had been warned, a restart produced zero fillwatch lines — equally consistent with "ran and chose silence" and "never ran". For an edge-triggered check the quiet run is the healthy steady state, so that ambiguity is permanent, not rare. Check now logs a per-run summary (checked N filesystem(s), M unreadable/skipped, K notification(s); bands: …), counting unreadable separately so a silently unreadable drive cannot read as "all fine".

8. Live evidence — three flows, guest 9201 on felhom-pve

Method: endpoint/log-level on real hardware, exercising the production paths — browser automation is not available on DooPlex. The hub's notification_log was read from a WAL-aware copy of /data/hub.db (all three files — R-172's lesson); the copies were deleted afterwards.

Flow 1 — the guard refuses, keeps the marker, does not alarm. Both hdd_1 mounts held unmounted against the agent's ~60 s heal, per the documented method:

[WARN] [appstop] refusing to restart "calibre-web" after an interrupted operation:
       drive /mnt/felhom-drives/hdd_1 is not a live mountpoint
[WARN] [appstop] crash recovery: NOT restarting calibre-web — start refused by a deliberate
       holder: … the marker is KEPT and the holder owns the restart
[WARN] [appstop] crash recovery: 1 app(s) were deliberately NOT restarted (drive absent) —
       KEEPING the marker; this is the gate working, not a fault: [calibre-web]
[WARN] [appstop] …HELD… — not alarming: … restarted=[] held_by_drive=[calibre-web]

Marker retained byte-identical; docker ps -a --filter name=calibre-web → 0; no backup_failed event. Scenario B, on a later boot with the drive live: restarted calibre-web after the interrupted an app-data backup (volume dump) and the marker was cleared.

Flow 2 — the operator event arrives; the customer is refused. Two real capture failures:

Event pushed: recovery_unit_capture_failed (error) — Recovery unit capture FAILED for "calibre-web"
  … /mnt/felhom-drives/hdd_1: 24.5/93.9 GB used (26%), 64.6 GB free. Error: … permission denied

Hub notification_log — the positive observable:

customer | recovery_unit_capture_failed | skipped | operator_only
operator | recovery_unit_capture_failed | sent    |

Flow 3 — the customer is warned, once, in Hungarian. /mnt/sys_drive filled with fallocate:

run	state	fired	log
1	90% / 4.7 GB	disk_warning	ok → warning … notifying the customer
2	unchanged	nothing	(edge trigger held)
3	91% / 1.7 GB	disk_critical	warning → critical …
4	file removed	nothing	critical → ok … cleared silently, re-armed; state emptied

Hub: customer | disk_warning | sent and customer | disk_critical | sent, Hungarian rendered. Exactly two events across four runs. Run 2's silence is meaningful only because run 3 proves the startup check executes every boot — needing that inference is what motivated v0.191.2.

9. Deployed versions

$ ssh felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller …"
gitea.dooplex.hu/admin/felhom-controller:0.191.2 | Up (healthy)

Hub: ArgoCD app felhom Synced / Healthy after a deliberate hard-refresh + sync (never kubectl set image); deploy/hub rolled out; image gitea.dooplex.hu/admin/felhom-hub:0.89.0; startup log clean.

10. Teardown

R167-FILLTEST.bin deleted (/mnt/sys_drive back to 2.0 G used / 45 G free); the app-stop marker cleared by a successful recovery; /root/hold-unmount.sh + /tmp/hold.log removed from felhom-pve; the secret-bearing local hub.db* copies deleted. All 15 containers on 9201 healthy.

One self-inflicted incident, disclosed in full. Restoring the drive by hand I ran mount --bind /mnt/hdd_1 /mnt/felhom-drives/hdd_1, omitting the felhom-data segment the agent uses (felhom-agent/internal/localapi/intermediary.go:15 — mount --bind /mnt/<name>/felhom-data /mnt/felhom-drives/<name>). The guest then saw the drive root instead of the namespace root, which an unprivileged guest cannot write to — producing real permission denied capture failures. They served as flow 2's evidence, but they were caused by me, not found by me, and the report says so. Separately, a manual docker compose up -d from a stack dir started containers without the controller-injected env (there is no .env on disk — the controller injects at exec time), which recreated immich-server with a blank DB password and left it in a 28P01 auth-failure restart loop. Both were repaired through production paths: the bind corrected, then the stacks brought back by the controller's own StartStack and boot reconciler ([bootrecon] … 1 app(s) recovered in 1 attempt(s): [immich]). Lesson, learned twice in one session: on a box the agent manages, restore state by letting the agent and controller do it.

11. Register rows

Opened: R-174 (closed same session), R-175, R-176, R-177 — each ID established free by grep -ro "R-17n\b" documentation/ *.md → 0 hits. Closed: R-158 (by R-167 — no second row was filed for the same wire), R-167, R-174. Updated and still open: R-165 (gains the spike's M1-M5 and the operator question).

12. CI and --no-verify

--no-verify was NOT used. Every push ran .githooks/pre-push (gates --fast) and it passed.

CI checked by PULL, matching head_sha to each commit — CI emails only on failure, so a green that was never looked at is an assumption. All three commits green:

Commit	Task id	Run #	Conclusion
cf48214 (v0.191.0)	31	11	success
5adae4d (v0.191.1)	34	12	success
9a3c485 (v0.191.2)	35	13	success

The felhom.eu side (hub v0.89.0, the manifest bump and the docs) is in that repo's REPORT.md §6 — also all green.

13. Observations — noticed, documented, NOT acted on

• R-177 — no operator-triggerable "run this scheduler job now" path. It cost a controller restart per observation here, and costs the same on a support call.
• R-175 — 07-backup-architecture.md §7.5 states one box's size bound as if it were the fleet's.
• R-176 — two R-165 prerequisites are unmeasured (a pre-merge-archive restore-test; the in-place migration rehearsal).
• The drive gate's return branch failed to restart both apps ([WARN] [gate] restart calibre-web: … exit code 1) at a moment when a manual start succeeded seconds later. Not filed as a row: it happened during my incorrect bind, so the likeliest cause is mine and the evidence is contaminated. Worth a look if it recurs on a clean box.