felhom-controller/REPORT.md

REPORT — controller v0.215.0 → v0.216.0: disk-health severity ladder, escalation, and the alert that never sent

Date: 2026-08-14 · Task class: Implementation · Repos touched: felhom-controller (code), felhom.eu (documentation only — no hub code, no manifest bump, no ArgoCD sync)

---

1. Confirmed baselines used (as read at the start of the run)

Repo	main @ start	Version	→ Shipped
felhom-controller	3e3ee94b7bbe6b66663c468e22aa86616365a45a	v0.214.0	v0.215.0, then v0.216.0 (a defect found live in v0.215.0 — §14)
felhom.eu	e0b56c976f8e4a7352309d78754fec448dd55f99	n/a (docs only)	n/a

Both trees verified clean (git status --porcelain empty, HEAD == origin/main) before any build. The controller hash matched the spec's stated baseline exactly. MinAgent stays 0.129.0 — no agent change; every field read here has been on the wire since agent v0.94.0/v0.95.0.

---

2. Files created / modified

felhom-controller

• controller/internal/agentapi/diskverdict.go — modified (14-row ladder, DiskPrior, UncorrectableSectors, TemperatureFailC)
• controller/internal/agentapi/diskverdict_test.go — modified
• controller/internal/agentapi/diskverdict_ladder_test.go — created
• controller/internal/notify/notifier.go — modified (severity, DiskAlert, DiskAlertKind, Severity(), 5 message shapes)
• controller/internal/notify/disk_health_test.go — rewritten
• controller/internal/web/disk_health_state.go — created (persistence + decision)
• controller/internal/web/disk_health.go — modified
• controller/internal/web/disk_health_test.go — rewritten
• controller/internal/web/server.go — modified (seam signature)
• controller/cmd/controller/main.go — modified (6h → 1h)
• (v0.216.0) controller/internal/web/disk_health.go + disk_health_test.go — the R-335 dedup fix and its test
• CHANGELOG.md, CONTEXT.md, REUSE.md, controller/README.md, REPORT.md

felhom.eu (documentation only)

• documentation/audits/DIAG-smart-passed-trap-2026-08-14.md — created
• documentation/audits/fixtures/smart-ST3000VX010-failing-2026-08-14.json — created (raw smartctl -a -j, verbatim)
• documentation/audits/fixtures/smartd-history-sdg-2026-08-14.txt — created (406 smartd journal lines)
• documentation/architecture/00-capability-map.md, documentation/backlog/ROADMAP.md, documentation/backlog/OPEN-ITEMS.md — modified

---

3. Commits pushed to main

Repo	Hash	What
felhom.eu	848de81	Part 0 — fixtures + findings doc
felhom-controller	bb50e12	Parts 1–3 — ladder, severity, persisted state + tests
felhom-controller	c24f192	Group L strengthened to two post-restart checks
felhom-controller	34d83f5	Part 4 — cadence 6h → 1h (measured)
felhom-controller	8144a70	Part 5 — CHANGELOG / CONTEXT / README / REUSE
felhom.eu	767960b	Part 5 — capability map, ROADMAP, register rows R-328…R-334
felhom-controller	90f2545	v0.216.0 — R-335, one physical disk evaluated once per run
felhom.eu	fa4748d	R-335 register row

---

4. Per-test results — all twelve groups

Group	Scenario	Test	Result
A	real drive, 2nd observation	TestDiskCheck_RealDrive_HibaAndOneCriticalEvent + TestLadder_RealDrive_ReachesHiba	PASS
B	the transient that cleared	TestDiskCheck_FirstSightingIsWarnOnly	PASS
C	sustained → Hiba	TestDiskLadder_SustainDrivesTheEscalation + TestLadder_SustainIsWhatFires	PASS
D	recovered, silent	TestDiskCheck_RecoveryIsSilentAndClearsState	PASS
E	flap damping	TestDiskCheck_FlapDamping	PASS
F	escalation beats damping	TestDiskCheck_EscalationBeatsDamping	PASS
G	still getting worse	TestDiskCheck_RealertWhenStillWorsening	PASS
H	below both bars	TestDiskCheck_NoRealertBelowBothBars	PASS
I	heat	TestLadder_Temperature + TestDiskCheck_TemperatureShape	PASS
J	no data never alarms	TestDiskCheck_UnknownNeverAlarmsNorErasesPrior + TestLadder_UnknownNeverAlarms	PASS
K	severity routes	TestNotifyDiskHealthDegraded_SeverityRoutes	PASS
L	state survives restart (seam)	TestDiskCheck_StateSurvivesRestart_ProductionPath	PASS

Supporting: TestLadder_CountBackstopBoundary, TestLadder_ZeroPriorIsFailSafe, TestUncorrectableSectors, TestDegradedAttributes_NamesFailCounters, TestNotifyDiskHealthDegraded_{WarnShape,FailShapes,CopyDiscipline}, TestDiskAlertDecision_Table, TestDiskState_CorruptFileFallsBackToNoPrior, TestDiskCheck_DisappearedDiskIsForgotten, TestDiskCheck_UnreachableAgentIsInert — all PASS.

---

5. Red-proof outcomes — all twelve, individually

Each mutation was applied by script, asserted present in the source before the run (the harness aborts with MUTATION-NOT-APPLIED if the target text is absent), the named test run, and the file reverted with git checkout --. The tree was confirmed clean after the sweep.

#	Mutation applied	Target test	Outcome
A	remove truth-table row 6 (the sustain rule)	TestDiskCheck_RealDrive_HibaAndOneCriticalEvent	RED-PROOF PASSED — FINDING, see below
B	make row 9 return Fail	TestDiskCheck_FirstSightingIsWarnOnly	failed as required
C	pass a zero DiskPrior in RunDiskHealthCheck	TestDiskLadder_SustainDrivesTheEscalation	failed as required
D	let Rendben fall through the silence guard	TestDiskCheck_RecoveryIsSilentAndClearsState	failed as required
E	compare against last observed verdict, not last alerted	TestDiskCheck_FlapDamping	failed as required
F	let damping cover escalations	TestDiskCheck_EscalationBeatsDamping	failed as required
G	remove the re-alert branch	TestDiskCheck_RealertWhenStillWorsening	failed as required
H	make cooldown/doubling an OR instead of an AND	TestDiskCheck_NoRealertBelowBothBars	failed as required
I	remove truth-table rows 3 and 13	TestLadder_Temperature, TestDiskCheck_TemperatureShape	failed as required
J	let UNKNOWN delete the prior record	TestDiskCheck_UnknownNeverAlarmsNorErasesPrior	failed as required
K	restore severity := "warn"	TestNotifyDiskHealthDegraded_SeverityRoutes	failed as required
L	skip loading the persisted state	TestDiskCheck_StateSurvivesRestart_ProductionPath	failed as required

A thirteenth red-proof, added after the deploy (R-335)

#	Mutation applied	Target test	Outcome
M	delete the if seen[key] { continue } dedup guard	TestDiskCheck_SameDiskTwiceIsEvaluatedOnce	failed as required

Observed failure: first sighting of an aliased disk must be silent, got 1: [{Label:felhom-backup … Kind:2 Sectors:8}] — i.e. Kind:2 is DiskAlertFailSectors, a Hiba on a first sighting of 8 sectors. Reverted.

FINDING — red-proof A passed, and it is the spec's mutation that is at fault, not the code

The task specified group A's red-proof as "remove truth-table row 6 → verdict is Warn". That mutation cannot fail a test built on the real drive's values: the real drive carries 352 unreadable sectors, so with row 6 deleted it still reaches Hiba via row 8 (count ≥ 64). The test correctly stayed green, so the mutation proves nothing about row 6.

This was anticipated while writing the tests and is documented in the test's own comment rather than discovered afterwards. Row 6 is genuinely pinned, by two tests that hold the counters at 8 (far below the 64 backstop) and vary only the prior:

• TestLadder_SustainIsWhatFires (agentapi) — same SmartSummary, DiskPrior{} → Warn, DiskPrior{SawUncorrectable:true} → Fail.
• TestDiskLadder_SustainDrivesTheEscalation (web) — the event-level twin.

Both were run under the row-6-deleted mutation and both failed, as recorded: SAME 8 sectors, now sustained = 2 (Figyelmeztetés), want Fail/Hiba and severity = "warning", want critical / chip = "Figyelmeztetés", want Hiba. So the invariant is covered; only the spec's chosen mutation was invalid.

A second finding, from building red-proof L

The first version of the Group L seam test ran one check after the restart and passed under the mutation — because a controller that has forgotten its state is also silent on its first check. The test was strengthened to run two checks (commit c24f192), after which the mutation fails. This is the exact shape §10 warns about, caught by running the red-proof rather than assuming it.

---

6. Test count and suite state

• Before: 1391 test functions (at 3e3ee94) · After: 1414 (+23)
• go build ./... && go vet ./... && go test ./... — all green, no failures, no skips introduced.
• python3 controller/scripts/controller_gates.py — all 11 gates OK.

---

7. Cadence measurement (Part 4)

Measured on demo-hp (Tier 0, disposable), through fetchDisks' real path — the agent local API GET /disks, not the 60 s card cache. Ten consecutive calls, all HTTP 200:

0.840558  0.817000  0.815783  0.831992  0.809066
0.832899  0.824788  0.804886  0.812539  0.833547   (seconds)

min	median	max	disk count
0.804886 s	0.820894 s	0.840558 s	3 physical rows across 2 devices (SanDisk X600 M.2 SATA SSD; Toshiba KXG50PNV1T02 NVMe, counted twice as c11-scratch + felhom-backup)

Branch taken: median < 5 s → 6*time.Hour → 1*time.Hour. The median is ~6× under the bar. The detection argument is the real one: the observed benign excursion lasted about one hour, so a 6-hourly sampler can land either side of it and then catch the terminal run half a day late.

No spin-up signature appeared in the timings (uniform ~0.82 s; demo-hp is all-flash), so the measurement did not suggest the spun-down-drive concern. That question is recorded as an Observation below and deliberately not acted on.

---

8. Live validation

Deployed to demo-hp guest 9201 via the bootstrap path (docker pull → /etc/felhom-controller-image → systemctl restart felhom-controller-bootstrap.service).

gitea.dooplex.hu/admin/felhom-controller:0.215.0 Up 19 seconds (healthy)   # 06:23Z
gitea.dooplex.hu/admin/felhom-controller:0.216.0 Up 6 seconds (healthy)    # after the R-335 fix

Leg 1 — no over-correction (the load-bearing check)

Method: endpoint-level — authenticated GET /dashboard on the real controller (https://felhom.enkisfelhom.hu/dashboard, HTTP 200, 44 036 bytes), i.e. the exact endpoint the UI invokes; only rendering is skipped. No browser is available on DooPlex.

Card contents, parsed from the response body:

Disk	Chip	Class	Temp
KXG50PNV1T02 NVMe TOSHIBA 1024GB	Rendben	state-text-run	53 °C
KXG50PNV1T02 NVMe TOSHIBA 1024GB	Rendben	state-text-run	53 °C
SanDisk X600 M.2 2280 SATA 128GB	Rendben	state-text-run	44 °C

Figyelmeztetés = 0, Hiba = 0, Nincs adat = 0, state-text-warn = 0, state-text-crit = 0. No healthy disk was over-corrected.

Positive observable, at deploy: [INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s) — the new cadence is in force, not merely compiled.

Positive observable, per cycle — two full hourly cycles observed after the deploy, from the container log:

2026/08/14 06:23:13 [INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s)
2026/08/14 07:23:13 [INFO] [scheduler] Running job: disk-health-check
2026/08/14 07:23:14 [INFO] [web] disk-health check complete: 3 disk(s) evaluated, 0 alert(s)
2026/08/14 07:23:14 [INFO] [scheduler] Job disk-health-check completed (took 849ms)
2026/08/14 08:23:13 [INFO] [scheduler] Running job: disk-health-check
2026/08/14 08:23:14 [INFO] [web] disk-health check complete: 3 disk(s) evaluated, 0 alert(s)
2026/08/14 08:23:14 [INFO] [scheduler] Job disk-health-check completed (took 843ms)

grep -c disk_health_degraded over the whole container log: 0. Both cycles ran (849 ms / 843 ms, matching the §7 measurement), evaluated every disk, and emitted nothing. Zero alerts from a check that demonstrably ran — not silence.

Persisted state written by the first cycle (/opt/docker/felhom-controller/data/disk-health-state.json, 428 bytes, on the felhom-controller-data docker volume, so it survives container recreation):

{"version": 1, "disks": {
  "path:/var/lib/vz":                          {"verdict": 1, "saw_uncorrectable": false, ...},
  "uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae": {"verdict": 1, "saw_uncorrectable": false, ...}}}

verdict: 1 is DiskVerdictOK for both, saw_uncorrectable: false, never alerted.

Reading those two artefacts against each other is what exposed R-335 — see §14.

Leg 2 — the severity fix arrives (the point of the task)

Two synthetic disk_health_degraded events pushed for customer demo-hp through the real hub event endpoint (POST https://hub.felhom.eu/api/v1/event), from the guest's own controller using its own hub credentials — the genuine controller→hub path, not a hand-crafted operator call. Both returned HTTP 200 {"ok":true}. The hub DB was read with its -wal and -shm copied alongside hub.db (a hub.db-only read is stale).

As STORED by the hub (events):

id	severity pushed	severity STORED
2964	warning	warning
2965	warn	info ← coerced

notification_log rows for those two events:

id	event_type	severity	channel	status	error
689	disk_health_degraded	warning	operator	sent	(none)
—	(the "warn" push)	—	—	NO ROW EXISTS	—

That pair is the proof. The identical event, differing only in one word of the severity string, is the difference between delivered to the operator and stored as an informational notice and delivered to nobody. This is the first time this leg has been observed end to end.

Only the operator leg fired because demo-hp has no customer_notifications row at all (no customer email, no enabled_events), so no customer row was possible for either push — verified directly, not assumed. One real email was sent to the operator, as the task anticipated.

---

9. NOT yet live-validated — stated explicitly

The Fail-from-counters path has never fired on real hardware. Everything in §4/§5 exercises it against the committed fixture's values in unit tests only. The live legs above prove the negative (no false alert on three healthy disks) and the severity wire (end to end, through the hub) — they do not prove a live disk reaching Hiba. The fixture tests must not be read as a live proof.

Tracked as R-332 (WATCHING). Closing condition: a live disk reaching Hiba from counters, or a deliberate injection through the real pipeline (agent /disks → controller check → hub event) — not a hand-set verdict.

One item originally listed here has since been proven live and is no longer part of this gap: the persisted state surviving a controller restart. The v0.215.0 → v0.216.0 redeploy destroyed and rebuilt the container, and the new one read back a changed_at written by the previous version rather than re-baselining — see §14. What remains unproven is the stronger half: an already-alerted disk not re-alerting after a restart, which needs a disk that has actually alerted. The drive that produced the fixture lives in DooPlex, which is Tier 2 and never a drill target; the demo boxes are all-flash and healthy.

---

10. Teardown

This run provisioned nothing — no VM, no guest, no hub customer record, no storage. Nothing was formatted, mounted, unmounted, repaired or written on any monitored disk; the only write is the controller's own disk-health-state.json inside its data volume.

Disposition of what the run did create:

• Two synthetic hub events (events id 2964, 2965) and one notification_log row (id 689) on the live hub. Left in place deliberately. Both messages are self-labelling ("R-328 severity probe (…) - synthetic, no real disk fault"), and deleting rows from the production hub DB is a riskier act than leaving two clearly-marked probe rows. Named here so they are not mistaken later for a real disk fault on demo-hp.
• One real operator email resulting from row 689.
• A local copy of hub.db/-wal/-shm in the session scratchpad only (not committed, not exported).

---

11. Register rows

Row	State	Owner
R-328 — the severity drop: "warn" coerced to info, emailed to nobody	CLOSED (controller v0.215.0), proven live side by side	CC
R-329 — app_start_failed carries the identical defect	READY — not fixed here; needs a decision on whether it should notify at all	Viktor
R-330 — Phase 2: collect SMART attrs 187/199/188 + persist samples	READY — a declared wire change, hub models it in the same session under G-1	CC
R-331 — Phase 3: growth-rate detection; revisit the static 64	READY, blocked on R-330	CC
R-332 — the Fail path has never fired on real hardware	WATCHING	CC
R-333 — NVMe temperature bands; agent smartctl has no -n standby	READY (S each)	Viktor decides (a); CC does (b)
R-334 — released with no golden carrying it (gate waiver)	READY — now applies to v0.216.0	CC bakes; Viktor vouches
R-335 — one physical disk walked twice per run, sustaining against itself	CLOSED (controller v0.216.0)	CC

smartd-on-DooPlex-alerts-nobody is recorded in DIAG-smart-passed-trap-2026-08-14.md §8 as the same shape one layer out.

---

12. Observations — noticed, NOT acted on

1. app_start_failed has the identical severity defect (notifier.go ~L546, "warn"). Left untouched per scope. It needs a prior decision — should a stopped app email the customer at all? — because flipping the string alone converts a silent event into a mail flood on a crash-looping box. R-329.

2. The 55/60 °C bands are spinning-disk bands being applied to NVMe, and this is close to biting. Adopted unchanged from the operator's Prometheus config by explicit decision — but demo-hp's healthy Toshiba NVMe idles at 53 °C, i.e. 2 °C below Figyelmeztetés and 7 °C below Hiba, and NVMe routinely passes 60 °C under sustained write with no fault. As shipped, a healthy customer NVMe under load can be reported as Hiba — the single worst outcome this feature can produce, and the one leg 1 exists to guard. Not changed here because the threshold is a stated, settled operator decision; flagged rather than overridden. R-333(a) — recommend splitting the bands by device class, or dropping them for NVMe and relying on critical_warning.

3. The agent runs bare smartctl -a -j with no -n standby (felhom-agent/internal/storage/hostops.go:368), so every poll wakes a spun-down drive, and 6h → 1h multiplies that by six. Recorded, not acted on, per the task's instruction. demo-hp is all-flash so the measurement could not reveal it. Mitigating datum from the fixture: the failing drive logged only 3375 load cycles in 60505 power-on hours (~one per 18 h), so this duty cycle barely spins down at all. R-333(b).

4. source ~/.config/credentials prints two recovery codes to the terminal. The file contains hyphenated keys (R_DEMO-FELHOM, R_DEMO-HP) that bash cannot assign, so sourcing it emits command not found errors containing the secret values. Anything that sources that file leaks them into logs, scrollback and transcripts. Not a code defect and out of scope; worth quoting values from it by other means, or renaming the keys.

5. golden_currency_gate.py has no waiver parser. Its own failure text says "record a waiver in OPEN-ITEMS.md — never a bypass", but nothing reads such a waiver, so the only way past it is the bypass it warns against. See §13.

---

13. Deviations, stated plainly

• git push --no-verify was used once, on the felhom.eu docs push (767960b), and only there. Cause: golden_currency_gate.py correctly convicts the fact that controller v0.215.0 is released and no golden carries it (newest bake 0.214.0), so a newly installed machine would receive 0.214.0 — without the severity fix. A golden bake was out of the task's scope, and its second half (vouching in the hub's day-0 artifact manifest) is operator-password-gated, so CC cannot complete it; a baked-but-unvouched golden is worse than none. Recorded as R-334 with the bake+vouch owners named. CI re-runs the same entry point and will mail the operator. The running fleet is unaffected.
• One pre-existing test changed meaning by design: TestDiskVerdictFor's critical_warning>0 → warn case is now → fail (truth-table row 4 — NVMe's own critical flag is a device declaration, not a drifting counter). TestDiskHealthCheck_DegradationOnce and its siblings were rewritten into the scenario groups because they encoded the pre-v0.215.0 single-alert behaviour the task deliberately replaces (Scenario C).

---

14. R-335 — a defect in v0.215.0, found live, fixed as v0.216.0

How it was found. Not by a test and not by review: by reading the release's own positive observable against the release's own persisted artefact. The hourly check logged "3 disk(s) evaluated"; disk-health-state.json held two records. Two artefacts that should have agreed did not.

Cause. demo-hp's c11-scratch and felhom-backup are the same physical NVMe (/dev/nvme0n1) and resolve to the same diskKey, so one disk was walked twice in a single run.

Why it mattered. RunDiskHealthCheck writes a disk's new record before the next entry reads it, so the second copy of an aliased disk consumed the first copy's write as its prior. The disk therefore sustained against itself and reached Hiba on a first sighting — defeating truth-table row 6, the single rule separating a one-hour benign excursion from a false critical alert — and would have emitted two identical events for one drive.

Severity in practice: latent, not active. Nothing fired on demo-hp because all three entries are healthy with zero counters. But any aliased disk developing one pending sector would have gone straight to Hiba, which is precisely the outcome §8 leg 1 exists to prevent. Aliasing is not exotic — it is the normal shape whenever a box has two PVE storage entries on one physical device.

Fix (v0.216.0, 90f2545). Each diskKey is evaluated once per run. Both entries stay marked seen, so neither is mistaken for a disappeared disk, and the card still renders both storage rows — the dedup is about state and alerts, not display. Pinned by TestDiskCheck_SameDiskTwiceIsEvaluatedOnce, red-proof run and reverted (§5).

Deployed: gitea.dooplex.hu/admin/felhom-controller:0.216.0 Up 6 seconds (healthy).

Confirming cycle on v0.216.0 — CONFIRMED LIVE, 09:31:35Z:

live image: gitea.dooplex.hu/admin/felhom-controller:0.216.0 Up About an hour (healthy)
2026/08/14 09:31:35 [INFO] [web] disk-health check complete: 2 disk(s) evaluated, 0 alert(s)
grep -c disk_health_degraded: 0

2 disk(s) evaluated now matches the 2 persisted records. The count and the artefact agree, which is the disagreement that exposed R-335 in the first place. Still zero alerts, still both card rows.

The redeploy also proved persistence live — a gap §9 had listed as unproven

The 0.215.0 → 0.216.0 redeploy replaced the container, and the state file came back intact:

"path:/var/lib/vz":  {"verdict": 1, "changed_at": "2026-08-14T07:23:14.640216851Z", ...}
"uuid:91d2dc2d-…":   {"verdict": 1, "changed_at": "2026-08-14T07:23:14.640216851Z", ...}

That changed_at was written by v0.215.0's first cycle at 07:23Z, before the container was destroyed and rebuilt. The v0.216.0 container read it back and preserved it rather than stamping a fresh time — so the new container loaded the pre-restart record instead of silently re-baselining. That is Scenario L observed on real hardware, not just through the production-path unit test, and it is exactly the behaviour that was impossible before v0.215.0 (the baseline was in-memory).

It also incidentally confirms the unchanged-verdict path: changed_at is preserved across four checks and two controller versions because the verdict never changed, rather than being churned every cycle.

What this still does NOT prove: these disks are healthy and were never alerted, so the stronger half — an already-ALERTED disk not re-alerting after a restart — remains unit-tested only. R-332 stands.

Process note, recorded because it nearly cost the fix. The red-proof harness reverts with git checkout --, which restores to HEAD. Running a red-proof against an uncommitted fix therefore deletes the fix along with the mutation — which happened here and was caught only by re-grepping the source afterwards. Commit the fix before red-proofing it, or snapshot outside git.