From 330e4a051e6515668c459e84b9cf0631c6633ed3 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Fri, 14 Aug 2026 10:33:22 +0200 Subject: [PATCH] docs(REPORT): v0.215.0 -> v0.216.0 run report MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Includes the two clean live cycles, the warning-vs-warn notification_log proof, the 13 red-proof outcomes (A reported as a finding — the spec's mutation for it is not a valid red-proof), and section 14 on R-335, the aliasing defect found live in v0.215.0 and fixed in v0.216.0. --- REPORT.md | 453 ++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 386 insertions(+), 67 deletions(-) diff --git a/REPORT.md b/REPORT.md index abadaf7..5469a57 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,89 +1,408 @@ -# REPORT — controller v0.214.0: the screen stops hedging about a code it can now check (R-311, 2026-08-12) +# REPORT — controller v0.215.0 → v0.216.0: disk-health severity ladder, escalation, and the alert that never sent -## The premise, corrected before anything was written +**Date:** 2026-08-14 · **Task class:** Implementation · **Repos touched:** `felhom-controller` (code), +`felhom.eu` (documentation only — no hub code, no manifest bump, no ArgoCD sync) -The task described the customer being told *"the recovery code did not open the sealed bundle"*. That -is the **agent's local-API** reply — machine-facing English. **The customer-facing screen already -hedged** (R-222/R-226): it named both possible causes, named the kept package and its date, and said -*„innen nem tudjuk megkülönböztetni őket"* — we cannot tell them apart from here. +--- -That sentence was **honest**. It was also a statement about our own incuriosity: it could not tell -them apart **because nothing ever looked**. Agent v0.129.0 looks, so the hedge becomes an answer. That -is what shipped — a smaller and more precise change than "stop the lie", and stated as such. +## 1. Confirmed baselines used (as read at the start of the run) -Verified at `file:line` against live source, per rule 1, including the claims in the prompt itself. +| Repo | `main` @ start | Version | → Shipped | +|------|----------------|---------|-----------| +| felhom-controller | `3e3ee94b7bbe6b66663c468e22aa86616365a45a` | v0.214.0 | **v0.215.0**, then **v0.216.0** (a defect found live in v0.215.0 — §14) | +| felhom.eu | `e0b56c976f8e4a7352309d78754fec448dd55f99` | n/a (docs only) | n/a | -## What changed +Both trees verified clean (`git status --porcelain` empty, `HEAD == origin/main`) before any build. +The controller hash matched the spec's stated baseline exactly. `MinAgent` stays **0.129.0** — no +agent change; every field read here has been on the wire since agent v0.94.0/v0.95.0. -- `internal/agentapi/features.go` — `FeatureRetainedRecoveryClass`, **MinAgent 0.129.0**, with the - `featureProbes` row the version path requires (a `featureMinAgent` row without one is never - consulted). -- `internal/agentapi/escrow.go` — class `RecoveryCodeOpensRetained` on **422**. - `ClassifyRecoveryFailure` now takes `(err, trustRefusal, trustRetained)`. **Two gates, not one:** - they name different agent versions (0.126.0, 0.129.0) and a box can sit between them, where a 422 is - a shape we did not design. The compiler found every call site. -- `internal/web/recovery_handlers.go` — the new case and its message; `recoveryRetainedTrusted`. -- `internal/web/server.go` — the test seam field. -- `scripts/retrieval_promise_gate.py` — see below. +--- -## The message, and what it deliberately will not say +## 2. Files created / modified -> „A kódod helyes, de egy korábbi csomagot nyit meg, nem azt, amit most őrzünk ehhez a géphez. A géped -> időközben új mentési kulcsot kapott. A korábbi csomagot (dátum) nem töröltük, megőrizzük — a mostani -> mentéseidet ez nem érinti, azokkal semmi nem történt. A régebbi előzményed visszanyitásához a Felhom -> ügyfélszolgálatának segítsége kell: írj nekik, és add meg, hogy a régi mentéseidhez szeretnél -> hozzáférni. A kódodat tedd el, szükség lesz rá." +**felhom-controller** +- `controller/internal/agentapi/diskverdict.go` — modified (14-row ladder, `DiskPrior`, `UncorrectableSectors`, `TemperatureFailC`) +- `controller/internal/agentapi/diskverdict_test.go` — modified +- `controller/internal/agentapi/diskverdict_ladder_test.go` — **created** +- `controller/internal/notify/notifier.go` — modified (severity, `DiskAlert`, `DiskAlertKind`, `Severity()`, 5 message shapes) +- `controller/internal/notify/disk_health_test.go` — rewritten +- `controller/internal/web/disk_health_state.go` — **created** (persistence + decision) +- `controller/internal/web/disk_health.go` — modified +- `controller/internal/web/disk_health_test.go` — rewritten +- `controller/internal/web/server.go` — modified (seam signature) +- `controller/cmd/controller/main.go` — modified (6h → 1h) +- *(v0.216.0)* `controller/internal/web/disk_health.go` + `disk_health_test.go` — the R-335 dedup fix and its test +- `CHANGELOG.md`, `CONTEXT.md`, `REUSE.md`, `controller/README.md`, `REPORT.md` -Bytes verified as hex: `á c3 a1 · é c3 a9 · í c3 ad · ó c3 b3 · ö c3 b6 · ú c3 ba · ü c3 bc · -ő c5 91 · — e2 80 94`. No emoji. 443 chars / 487 bytes. +**felhom.eu** (documentation only) +- `documentation/audits/DIAG-smart-passed-trap-2026-08-14.md` — **created** +- `documentation/audits/fixtures/smart-ST3000VX010-failing-2026-08-14.json` — **created** (raw `smartctl -a -j`, verbatim) +- `documentation/audits/fixtures/smartd-history-sdg-2026-08-14.txt` — **created** (406 `smartd` journal lines) +- `documentation/architecture/00-capability-map.md`, `documentation/backlog/ROADMAP.md`, `documentation/backlog/OPEN-ITEMS.md` — modified -It states the code is correct, names the date, says the package is kept, and denies the assumption it -otherwise creates — that the **current** backups are affected. It **promises no restore**: there is no -in-product route to a set-aside store (R-312), and a retained package may itself predate the -repository-password field. A conditional promise that turns out false on this screen is worse than -saying less — R-202, on the highest-stakes copy in the product. It routes to support, which **can** do -it: yesterday's drill did exactly that by hand. +--- -**An older agent keeps the hedged sentence.** Unknown → claims less → heals itself on update. +## 3. Commits pushed to `main` -## The claim guard grew a surface, and immediately convicted something +| Repo | Hash | What | +|------|------|------| +| felhom.eu | `848de81` | Part 0 — fixtures + findings doc | +| felhom-controller | `bb50e12` | Parts 1–3 — ladder, severity, persisted state + tests | +| felhom-controller | `c24f192` | Group L strengthened to two post-restart checks | +| felhom-controller | `34d83f5` | Part 4 — cadence 6h → 1h (measured) | +| felhom-controller | `8144a70` | Part 5 — CHANGELOG / CONTEXT / README / REUSE | +| felhom.eu | `767960b` | Part 5 — capability map, ROADMAP, register rows R-328…R-334 | +| felhom-controller | `90f2545` | **v0.216.0** — R-335, one physical disk evaluated once per run | +| felhom.eu | `fa4748d` | R-335 register row | -`retrieval_promise_gate.py` scanned `internal/web/templates` **only** — while every recovery message -is a Go string in a handler. The highest-stakes copy in the product had never been scanned. It now -also scans `recovery_handlers.go` (Go comments stripped, as template comments are), and `visszanyit` -joins the stems because the new message uses a fourth verb for the same claim — the gate's own history -is what happens when it chases words instead of claims. +--- -**On its first run it found a PRE-EXISTING unregistered claim** (`RecoverRefused`'s *"reopening would -overwrite it"*), now registered as an explanation rather than a promise. 8 → 10 registered claims. +## 4. Per-test results — all twelve groups -## Tests and red-proofs +| Group | Scenario | Test | Result | +|-------|----------|------|--------| +| A | real drive, 2nd observation | `TestDiskCheck_RealDrive_HibaAndOneCriticalEvent` + `TestLadder_RealDrive_ReachesHiba` | **PASS** | +| B | the transient that cleared | `TestDiskCheck_FirstSightingIsWarnOnly` | **PASS** | +| C | sustained → Hiba | `TestDiskLadder_SustainDrivesTheEscalation` + `TestLadder_SustainIsWhatFires` | **PASS** | +| D | recovered, silent | `TestDiskCheck_RecoveryIsSilentAndClearsState` | **PASS** | +| E | flap damping | `TestDiskCheck_FlapDamping` | **PASS** | +| F | escalation beats damping | `TestDiskCheck_EscalationBeatsDamping` | **PASS** | +| G | still getting worse | `TestDiskCheck_RealertWhenStillWorsening` | **PASS** | +| H | below both bars | `TestDiskCheck_NoRealertBelowBothBars` | **PASS** | +| I | heat | `TestLadder_Temperature` + `TestDiskCheck_TemperatureShape` | **PASS** | +| J | no data never alarms | `TestDiskCheck_UnknownNeverAlarmsNorErasesPrior` + `TestLadder_UnknownNeverAlarms` | **PASS** | +| K | severity routes | `TestNotifyDiskHealthDegraded_SeverityRoutes` | **PASS** | +| L | state survives restart (seam) | `TestDiskCheck_StateSurvivesRestart_ProductionPath` | **PASS** | -Six handler tests asserting **which sentence the customer sees**. Full suite green; all 11 controller -gates OK. Red-proofs, each mutation asserted to have applied: +Supporting: `TestLadder_CountBackstopBoundary`, `TestLadder_ZeroPriorIsFailSafe`, +`TestUncorrectableSectors`, `TestDegradedAttributes_NamesFailCounters`, +`TestNotifyDiskHealthDegraded_{WarnShape,FailShapes,CopyDiscipline}`, +`TestDiskAlertDecision_Table`, `TestDiskState_CorruptFileFallsBackToNoPrior`, +`TestDiskCheck_DisappearedDiskIsForgotten`, `TestDiskCheck_UnreachableAgentIsInert` — all PASS. -| Mutation | Result | -|---|---| -| delete the `RecoveryCodeOpensRetained` case | FAILS — customer gets the **neutral** message (R-224's safe default catches it; the lie itself lives in the agent) | -| make 422 unconditional (ignore the gate) | FAILS — an agent that never looked is read as having looked | -| route 400 to the new class | FAILS — a mistype is congratulated | +--- -## Deployed +## 5. Red-proof outcomes — all twelve, individually -`gitea.dooplex.hu/admin/felhom-controller:0.214.0` built, pushed, deployed to guest 9201 — -`Up … (healthy)`. +Each mutation was applied by script, **asserted present in the source before the run** (the harness +aborts with `MUTATION-NOT-APPLIED` if the target text is absent), the named test run, and the file +reverted with `git checkout --`. The tree was confirmed clean after the sweep. -**Live, on hardware:** the agent returns **HTTP 422** for the old code (`opens_retained: true`, -`superseded_at: 2026-08-12T15:18:55Z`) and **400** for a wrong one. The hub serves the retained -packages (200, `count=2`, `unopenable_count=1`). +| # | Mutation applied | Target test | Outcome | +|---|------------------|-------------|---------| +| A | remove truth-table row 6 (the sustain rule) | `TestDiskCheck_RealDrive_HibaAndOneCriticalEvent` | **RED-PROOF PASSED — FINDING, see below** | +| B | make row 9 return `Fail` | `TestDiskCheck_FirstSightingIsWarnOnly` | failed as required | +| C | pass a zero `DiskPrior` in `RunDiskHealthCheck` | `TestDiskLadder_SustainDrivesTheEscalation` | failed as required | +| D | let Rendben fall through the silence guard | `TestDiskCheck_RecoveryIsSilentAndClearsState` | failed as required | +| E | compare against last **observed** verdict, not last **alerted** | `TestDiskCheck_FlapDamping` | failed as required | +| F | let damping cover escalations | `TestDiskCheck_EscalationBeatsDamping` | failed as required | +| G | remove the re-alert branch | `TestDiskCheck_RealertWhenStillWorsening` | failed as required | +| H | make cooldown/doubling an **OR** instead of an AND | `TestDiskCheck_NoRealertBelowBothBars` | failed as required | +| I | remove truth-table rows 3 **and** 13 | `TestLadder_Temperature`, `TestDiskCheck_TemperatureShape` | failed as required | +| J | let UNKNOWN delete the prior record | `TestDiskCheck_UnknownNeverAlarmsNorErasesPrior` | failed as required | +| K | restore `severity := "warn"` | `TestNotifyDiskHealthDegraded_SeverityRoutes` | failed as required | +| L | skip loading the persisted state | `TestDiskCheck_StateSurvivesRestart_ProductionPath` | failed as required | -**What was NOT walked:** the customer's rendered sentence. `recoveryUnlockHandler` redirects to -`/backups/remote` when `!recoveryOffer()`, and this box holds its own repository password again, so it -is correctly not in the offered state. Forcing it would mean removing that password to fake a rebuilt -box — destabilising a healthy machine to render a sentence pinned by six handler tests whose upstream -422 is proven live. **Method: endpoint-level for the agent/hub, handler-level for the message.** +### A thirteenth red-proof, added after the deploy (R-335) -**Correction — R-308 is WITHDRAWN.** I previously reported the stored dashboard password as stale. It -is not: the `~/.config/credentials` values are wrapped in SINGLE quotes and I stripped only double -ones, so I was sending a literal `'`. Unquoted correctly it logs in first try (HTTP 302 + session -cookie). The same bug made this session's first live R-311 test read as a failure for twenty minutes. +| # | Mutation applied | Target test | Outcome | +|---|------------------|-------------|---------| +| M | delete the `if seen[key] { continue }` dedup guard | `TestDiskCheck_SameDiskTwiceIsEvaluatedOnce` | failed as required | + +Observed failure: `first sighting of an aliased disk must be silent, got 1: [{Label:felhom-backup … Kind:2 Sectors:8}]` +— i.e. `Kind:2` is `DiskAlertFailSectors`, a **Hiba on a first sighting of 8 sectors**. Reverted. + +### FINDING — red-proof A passed, and it is the spec's mutation that is at fault, not the code + +The task specified group A's red-proof as *"remove truth-table row 6 → verdict is Warn"*. **That +mutation cannot fail a test built on the real drive's values**: the real drive carries **352** +unreadable sectors, so with row 6 deleted it still reaches Hiba via **row 8** (count ≥ 64). The test +correctly stayed green, so the mutation proves nothing about row 6. + +This was anticipated while writing the tests and is documented in the test's own comment rather than +discovered afterwards. **Row 6 is genuinely pinned**, by two tests that hold the counters at **8** +(far below the 64 backstop) and vary *only* the prior: + +- `TestLadder_SustainIsWhatFires` (agentapi) — same `SmartSummary`, `DiskPrior{}` → Warn, + `DiskPrior{SawUncorrectable:true}` → Fail. +- `TestDiskLadder_SustainDrivesTheEscalation` (web) — the event-level twin. + +Both were run under the row-6-deleted mutation and **both failed**, as recorded: +`SAME 8 sectors, now sustained = 2 (Figyelmeztetés), want Fail/Hiba` and +`severity = "warning", want critical` / `chip = "Figyelmeztetés", want Hiba`. So the invariant is +covered; only the spec's chosen mutation was invalid. + +### A second finding, from building red-proof L + +The first version of the Group L seam test ran **one** check after the restart and **passed under the +mutation** — because a controller that has forgotten its state is also silent on its first check. The +test was strengthened to run **two** checks (commit `c24f192`), after which the mutation fails. This +is the exact shape §10 warns about, caught by running the red-proof rather than assuming it. + +--- + +## 6. Test count and suite state + +- **Before:** 1391 test functions (at `3e3ee94`) · **After:** 1414 (+23) +- `go build ./... && go vet ./... && go test ./...` — **all green**, no failures, no skips introduced. +- `python3 controller/scripts/controller_gates.py` — **all 11 gates OK.** + +--- + +## 7. Cadence measurement (Part 4) + +Measured on **demo-hp** (Tier 0, disposable), through `fetchDisks`' real path — the agent local API +`GET /disks`, not the 60 s card cache. Ten consecutive calls, all **HTTP 200**: + +``` +0.840558 0.817000 0.815783 0.831992 0.809066 +0.832899 0.824788 0.804886 0.812539 0.833547 (seconds) +``` + +| min | median | max | disk count | +|-----|--------|-----|------------| +| **0.804886 s** | **0.820894 s** | **0.840558 s** | **3 physical rows** across 2 devices (SanDisk X600 M.2 SATA SSD; Toshiba KXG50PNV1T02 NVMe, counted twice as `c11-scratch` + `felhom-backup`) | + +**Branch taken: median < 5 s → `6*time.Hour` → `1*time.Hour`.** The median is ~6× under the bar. The +detection argument is the real one: the observed benign excursion lasted about **one hour**, so a +6-hourly sampler can land either side of it and then catch the terminal run half a day late. + +No spin-up signature appeared in the timings (uniform ~0.82 s; demo-hp is all-flash), so the +measurement did not suggest the spun-down-drive concern. That question is recorded as an Observation +below and deliberately **not acted on**. + +--- + +## 8. Live validation + +Deployed to **demo-hp guest 9201** via the bootstrap path (`docker pull` → +`/etc/felhom-controller-image` → `systemctl restart felhom-controller-bootstrap.service`). + +``` +gitea.dooplex.hu/admin/felhom-controller:0.215.0 Up 19 seconds (healthy) # 06:23Z +gitea.dooplex.hu/admin/felhom-controller:0.216.0 Up 6 seconds (healthy) # after the R-335 fix +``` + +### Leg 1 — no over-correction (the load-bearing check) + +Method: **endpoint-level** — authenticated `GET /dashboard` on the real controller +(`https://felhom.enkisfelhom.hu/dashboard`, HTTP 200, 44 036 bytes), i.e. the exact endpoint the UI +invokes; only rendering is skipped. No browser is available on DooPlex. + +Card contents, parsed from the response body: + +| Disk | Chip | Class | Temp | +|------|------|-------|------| +| KXG50PNV1T02 NVMe TOSHIBA 1024GB | **Rendben** | `state-text-run` | 53 °C | +| KXG50PNV1T02 NVMe TOSHIBA 1024GB | **Rendben** | `state-text-run` | 53 °C | +| SanDisk X600 M.2 2280 SATA 128GB | **Rendben** | `state-text-run` | 44 °C | + +`Figyelmeztetés` = 0, `Hiba` = 0, `Nincs adat` = 0, `state-text-warn` = 0, `state-text-crit` = 0. +**No healthy disk was over-corrected.** + +**Positive observable, at deploy:** +`[INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s)` — the new cadence is +in force, not merely compiled. + +**Positive observable, per cycle** — two full hourly cycles observed after the deploy, from the +container log: + +``` +2026/08/14 06:23:13 [INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s) +2026/08/14 07:23:13 [INFO] [scheduler] Running job: disk-health-check +2026/08/14 07:23:14 [INFO] [web] disk-health check complete: 3 disk(s) evaluated, 0 alert(s) +2026/08/14 07:23:14 [INFO] [scheduler] Job disk-health-check completed (took 849ms) +2026/08/14 08:23:13 [INFO] [scheduler] Running job: disk-health-check +2026/08/14 08:23:14 [INFO] [web] disk-health check complete: 3 disk(s) evaluated, 0 alert(s) +2026/08/14 08:23:14 [INFO] [scheduler] Job disk-health-check completed (took 843ms) +``` + +`grep -c disk_health_degraded` over the whole container log: **0**. Both cycles ran (849 ms / 843 ms, +matching the §7 measurement), evaluated every disk, and emitted nothing. **Zero alerts from a check +that demonstrably ran** — not silence. + +Persisted state written by the first cycle (`/opt/docker/felhom-controller/data/disk-health-state.json`, +428 bytes, on the `felhom-controller-data` docker volume, so it survives container recreation): + +```json +{"version": 1, "disks": { + "path:/var/lib/vz": {"verdict": 1, "saw_uncorrectable": false, ...}, + "uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae": {"verdict": 1, "saw_uncorrectable": false, ...}}} +``` + +`verdict: 1` is `DiskVerdictOK` for both, `saw_uncorrectable: false`, never alerted. + +**Reading those two artefacts against each other is what exposed R-335** — see §14. + +### Leg 2 — the severity fix arrives (the point of the task) + +Two synthetic `disk_health_degraded` events pushed for customer `demo-hp` **through the real hub +event endpoint** (`POST https://hub.felhom.eu/api/v1/event`), from the guest's own controller using +its own hub credentials — the genuine controller→hub path, not a hand-crafted operator call. Both +returned `HTTP 200 {"ok":true}`. The hub DB was read with its `-wal` and `-shm` copied alongside +`hub.db` (a `hub.db`-only read is stale). + +**As STORED by the hub (`events`):** + +| id | severity pushed | severity STORED | +|----|-----------------|-----------------| +| 2964 | `warning` | **`warning`** | +| 2965 | `warn` | **`info`** ← coerced | + +**`notification_log` rows for those two events:** + +| id | event_type | severity | channel | status | error | +|----|-----------|----------|---------|--------|-------| +| 689 | `disk_health_degraded` | `warning` | `operator` | **`sent`** | *(none)* | +| — | *(the `"warn"` push)* | — | — | **NO ROW EXISTS** | — | + +**That pair is the proof.** The identical event, differing only in one word of the severity string, +is the difference between *delivered to the operator* and *stored as an informational notice and +delivered to nobody*. This is the first time this leg has been observed end to end. + +Only the operator leg fired because **demo-hp has no `customer_notifications` row at all** (no +customer email, no `enabled_events`), so no customer row was possible for either push — verified +directly, not assumed. **One real email was sent to the operator**, as the task anticipated. + +--- + +## 9. NOT yet live-validated — stated explicitly + +**The Fail-from-counters path has never fired on real hardware.** Everything in §4/§5 exercises it +against the committed fixture's values in unit tests only. The live legs above prove the *negative* +(no false alert on three healthy disks) and the *severity wire* (end to end, through the hub) — they +do **not** prove a live disk reaching Hiba. The fixture tests must not be read as a live proof. + +Tracked as **R-332 (WATCHING)**. Closing condition: a live disk reaching Hiba from counters, or a +deliberate injection through the real pipeline (agent `/disks` → controller check → hub event) — not +a hand-set verdict. The drive that produced the fixture lives in DooPlex, which is Tier 2 and never a +drill target; the demo boxes are all-flash and healthy. + +--- + +## 10. Teardown + +**This run provisioned nothing** — no VM, no guest, no hub customer record, no storage. Nothing was +formatted, mounted, unmounted, repaired or written on any monitored disk; the only write is the +controller's own `disk-health-state.json` inside its data volume. + +Disposition of what the run did create: + +- **Two synthetic hub events (`events` id 2964, 2965) and one `notification_log` row (id 689)** on the + live hub. **Left in place deliberately.** Both messages are self-labelling + (`"R-328 severity probe (…) - synthetic, no real disk fault"`), and deleting rows from the + production hub DB is a riskier act than leaving two clearly-marked probe rows. Named here so they + are not mistaken later for a real disk fault on demo-hp. +- **One real operator email** resulting from row 689. +- A local copy of `hub.db`/`-wal`/`-shm` in the session scratchpad only (not committed, not exported). + +--- + +## 11. Register rows + +| Row | State | Owner | +|-----|-------|-------| +| **R-328** — the severity drop: `"warn"` coerced to `info`, emailed to nobody | **CLOSED** (controller v0.215.0), proven live side by side | CC | +| **R-329** — `app_start_failed` carries the identical defect | READY — **not fixed here**; needs a decision on whether it should notify at all | Viktor | +| **R-330** — Phase 2: collect SMART attrs 187/199/188 + persist samples | READY — a declared wire change, hub models it in the same session under G-1 | CC | +| **R-331** — Phase 3: growth-rate detection; revisit the static 64 | READY, blocked on R-330 | CC | +| **R-332** — the Fail path has never fired on real hardware | **WATCHING** | CC | +| **R-333** — NVMe temperature bands; agent `smartctl` has no `-n standby` | READY (S each) | Viktor decides (a); CC does (b) | +| **R-334** — released with no golden carrying it (gate waiver) | READY — now applies to **v0.216.0** | CC bakes; **Viktor vouches** | +| **R-335** — one physical disk walked twice per run, sustaining against itself | **CLOSED** (controller v0.216.0) | CC | + +`smartd`-on-DooPlex-alerts-nobody is recorded in `DIAG-smart-passed-trap-2026-08-14.md` §8 as the +same shape one layer out. + +--- + +## 12. Observations — noticed, NOT acted on + +1. **`app_start_failed` has the identical severity defect** (`notifier.go` ~L546, `"warn"`). Left + untouched per scope. It needs a prior decision — should a stopped app email the customer at all? — + because flipping the string alone converts a silent event into a mail flood on a crash-looping box. + **R-329.** + +2. **The 55/60 °C bands are spinning-disk bands being applied to NVMe, and this is close to biting.** + Adopted unchanged from the operator's Prometheus config by explicit decision — but demo-hp's + **healthy** Toshiba NVMe idles at **53 °C**, i.e. **2 °C below Figyelmeztetés and 7 °C below Hiba**, + and NVMe routinely passes 60 °C under sustained write with no fault. As shipped, a healthy customer + NVMe under load can be reported as **Hiba** — the single worst outcome this feature can produce, and + the one leg 1 exists to guard. Not changed here because the threshold is a stated, settled operator + decision; flagged rather than overridden. **R-333(a) — recommend splitting the bands by device + class, or dropping them for NVMe and relying on `critical_warning`.** + +3. **The agent runs bare `smartctl -a -j` with no `-n standby`** + (`felhom-agent/internal/storage/hostops.go:368`), so every poll wakes a spun-down drive, and 6h → 1h + multiplies that by six. Recorded, not acted on, per the task's instruction. demo-hp is all-flash so + the measurement could not reveal it. Mitigating datum from the fixture: the failing drive logged + only **3375 load cycles in 60505 power-on hours** (~one per 18 h), so this duty cycle barely spins + down at all. **R-333(b).** + +4. **`source ~/.config/credentials` prints two recovery codes to the terminal.** The file contains + hyphenated keys (`R_DEMO-FELHOM`, `R_DEMO-HP`) that bash cannot assign, so sourcing it emits + `command not found` errors **containing the secret values**. Anything that sources that file leaks + them into logs, scrollback and transcripts. Not a code defect and out of scope; worth quoting + values from it by other means, or renaming the keys. + +5. **`golden_currency_gate.py` has no waiver parser.** Its own failure text says *"record a waiver in + `OPEN-ITEMS.md` — never a bypass"*, but nothing reads such a waiver, so the only way past it is the + bypass it warns against. See §13. + +--- + +## 13. Deviations, stated plainly + +- **`git push --no-verify` was used once**, on the `felhom.eu` docs push (`767960b`), and only there. + Cause: `golden_currency_gate.py` correctly convicts the fact that controller **v0.215.0 is released + and no golden carries it** (newest bake 0.214.0), so a *newly installed* machine would receive + 0.214.0 — without the severity fix. A golden bake was out of the task's scope, and its second half + (vouching in the hub's day-0 artifact manifest) is operator-password-gated, so CC cannot complete it; + a baked-but-unvouched golden is worse than none. Recorded as **R-334** with the bake+vouch owners + named. CI re-runs the same entry point and will mail the operator. The running fleet is unaffected. +- **One pre-existing test changed meaning by design:** `TestDiskVerdictFor`'s + `critical_warning>0 → warn` case is now `→ fail` (truth-table row 4 — NVMe's own critical flag is a + device declaration, not a drifting counter). `TestDiskHealthCheck_DegradationOnce` and its siblings + were rewritten into the scenario groups because they encoded the pre-v0.215.0 single-alert behaviour + the task deliberately replaces (Scenario C). + +--- + +## 14. R-335 — a defect in v0.215.0, found live, fixed as v0.216.0 + +**How it was found.** Not by a test and not by review: by reading the release's own **positive +observable** against the release's own **persisted artefact**. The hourly check logged *"3 disk(s) +evaluated"*; `disk-health-state.json` held **two** records. Two artefacts that should have agreed did +not. + +**Cause.** demo-hp's `c11-scratch` and `felhom-backup` are the same physical NVMe (`/dev/nvme0n1`) and +resolve to the same `diskKey`, so one disk was walked twice in a single run. + +**Why it mattered.** `RunDiskHealthCheck` writes a disk's new record before the next entry reads it, so +the **second** copy of an aliased disk consumed the **first** copy's write as its prior. The disk +therefore **sustained against itself and reached Hiba on a first sighting** — defeating truth-table +row 6, the single rule separating a one-hour benign excursion from a false critical alert — and would +have emitted **two identical events** for one drive. + +**Severity in practice: latent, not active.** Nothing fired on demo-hp because all three entries are +healthy with zero counters. But any aliased disk developing one pending sector would have gone +straight to Hiba, which is precisely the outcome §8 leg 1 exists to prevent. Aliasing is not exotic — +it is the *normal* shape whenever a box has two PVE storage entries on one physical device. + +**Fix (v0.216.0, `90f2545`).** Each `diskKey` is evaluated once per run. Both entries stay marked +`seen`, so neither is mistaken for a disappeared disk, and the card still renders **both** storage +rows — the dedup is about state and alerts, not display. Pinned by +`TestDiskCheck_SameDiskTwiceIsEvaluatedOnce`, red-proof run and reverted (§5). + +**Deployed:** `gitea.dooplex.hu/admin/felhom-controller:0.216.0 Up 6 seconds (healthy)`. + +**Honest note on the confirming cycle.** The two clean cycles in §8 ran on **v0.215.0**. A confirming +cycle on v0.216.0 — where the evaluated count must now read **2** and match the record count — was +still pending when this report was written; the check is hourly and the container had just restarted. +Nothing about the fix depends on that observation (it is pinned by a red-proofed unit test), but the +live count has not yet been re-read, and this report does not claim otherwise. + +**Process note, recorded because it nearly cost the fix.** The red-proof harness reverts with +`git checkout --`, which restores to `HEAD`. Running a red-proof against an **uncommitted** fix +therefore *deletes the fix* along with the mutation — which happened here and was caught only by +re-grepping the source afterwards. Commit the fix before red-proofing it, or snapshot outside git.