Files
felhom-controller/audits/REPORT-v0.216.0-2026-08-14.md
T
admin f94543ee5c
gates / gates (push) Successful in 10s
v0.217.0: prefill from the app's own backup, where-the-data-goes on deploy, bounded inventory fan-out
Completes R-351 and ships R-352's visibility half. Gates 11/11 OK, suite 28 packages ok,
go vet clean, -race clean on the changed package - all run and read BEFORE this commit.

PART 2 SCENARIO A - the deploy page prefills the address and data folder from the app's OWN
backup. backup.RecordedUnitForStack scans every readable namespace root (the app is NOT
installed in this case, so there is no own drive to ask) and reads manifest.json plus the
captured compose/app.yaml. Local file reads only: no network, no restic, no restore.
RecordedAddress.Known() requires BOTH halves on purpose - an absent SUBDOMAIN makes the live
deploy path substitute the CATALOG default (stacks/deploy.go:88-90), and offering that back as
"what your backup says" would be a fabricated fact. The prefill is labelled as coming from the
backup and stays editable: a memory, not a lock.

PART 1 VISIBILITY (R-352) - the deploy page now states where the app's data will live before
the button is pressed. Measured 2026-08-21: 13 of 53 catalogue templates declare a storage
field; the other 40 have none and their data goes to the system drive, which no screen said.
Metadata.HasDeployField answers "does this app have somewhere to PUT a recorded value?" - for
the 40-class a recorded placement is a fact to state, never a value written into a field that
does not exist. NO PLACEMENT CHANGED. NOTHING MIGRATED. The rest is a filed specification.

PART 4 - measured before theorising, on the live off-site target:
  snapshots --json 2605 ms once; stats 2697 ms PER APP, sequential, 5 app tags
  => 2605 + 5*2697 = ~16.1 s, matching the reported ten-to-fifteen seconds.
The cause is the shape already on file, so the per-app size calls now run concurrently,
BOUNDED TO 4. The bound is the safety property, not the speed one: the repository is a Hetzner
Storage Box with a session cap, and a refused size call returns SizeBytes 0 - a silent
UNDER-REPORT of the customer's data rather than a visible failure. Peak-in-flight is asserted.
OffsiteInventoryList had no test at all before this.

TEMPLATE SAFETY - every Restore* key is set UNCONDITIONALLY in the deploy handler, because a
template doing index/eq against an undefined key errors at RENDER time: green build, green vet,
green suite, 500 on the page. Four render tests, one per branch, because the existing deploy
render test only renders AutoFields and never reaches these blocks.

RED-PROOFS, mutation asserted applied then reverted to 0:
  A   three template guards dropped (count asserted 3) -> the blank form returned
  P4  inventorySizeConcurrency = 1 -> "peak in flight was 1", elapsed 282ms = sequential

DOCS: CHANGELOG v0.217.0 (MinAgent 0.129.0 unchanged), CONTEXT (the restore's own memory +
what is next), controller/README.md (Backup System), REUSE.md (4 new rows), REPORT.md
overwritten - the previous REPORT preserved to audits/REPORT-v0.216.0-2026-08-14.md first.

NOT fixed here, filed as R-353 and named the next session's first item: a restore whose unit
carries no db_dumps and no volume_dumps still reports a bare completion.
2026-08-21 21:29:01 +02:00

442 lines
25 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# REPORT — controller v0.215.0 → v0.216.0: disk-health severity ladder, escalation, and the alert that never sent
**Date:** 2026-08-14 · **Task class:** Implementation · **Repos touched:** `felhom-controller` (code),
`felhom.eu` (documentation only — no hub code, no manifest bump, no ArgoCD sync)
---
## 1. Confirmed baselines used (as read at the start of the run)
| Repo | `main` @ start | Version | → Shipped |
|------|----------------|---------|-----------|
| felhom-controller | `3e3ee94b7bbe6b66663c468e22aa86616365a45a` | v0.214.0 | **v0.215.0**, then **v0.216.0** (a defect found live in v0.215.0 — §14) |
| felhom.eu | `e0b56c976f8e4a7352309d78754fec448dd55f99` | n/a (docs only) | n/a |
Both trees verified clean (`git status --porcelain` empty, `HEAD == origin/main`) before any build.
The controller hash matched the spec's stated baseline exactly. `MinAgent` stays **0.129.0** — no
agent change; every field read here has been on the wire since agent v0.94.0/v0.95.0.
---
## 2. Files created / modified
**felhom-controller**
- `controller/internal/agentapi/diskverdict.go` — modified (14-row ladder, `DiskPrior`, `UncorrectableSectors`, `TemperatureFailC`)
- `controller/internal/agentapi/diskverdict_test.go` — modified
- `controller/internal/agentapi/diskverdict_ladder_test.go` — **created**
- `controller/internal/notify/notifier.go` — modified (severity, `DiskAlert`, `DiskAlertKind`, `Severity()`, 5 message shapes)
- `controller/internal/notify/disk_health_test.go` — rewritten
- `controller/internal/web/disk_health_state.go` — **created** (persistence + decision)
- `controller/internal/web/disk_health.go` — modified
- `controller/internal/web/disk_health_test.go` — rewritten
- `controller/internal/web/server.go` — modified (seam signature)
- `controller/cmd/controller/main.go` — modified (6h → 1h)
- *(v0.216.0)* `controller/internal/web/disk_health.go` + `disk_health_test.go` — the R-335 dedup fix and its test
- `CHANGELOG.md`, `CONTEXT.md`, `REUSE.md`, `controller/README.md`, `REPORT.md`
**felhom.eu** (documentation only)
- `documentation/audits/DIAG-smart-passed-trap-2026-08-14.md` — **created**
- `documentation/audits/fixtures/smart-ST3000VX010-failing-2026-08-14.json` — **created** (raw `smartctl -a -j`, verbatim)
- `documentation/audits/fixtures/smartd-history-sdg-2026-08-14.txt` — **created** (406 `smartd` journal lines)
- `documentation/architecture/00-capability-map.md`, `documentation/backlog/ROADMAP.md`, `documentation/backlog/OPEN-ITEMS.md` — modified
---
## 3. Commits pushed to `main`
| Repo | Hash | What |
|------|------|------|
| felhom.eu | `848de81` | Part 0 — fixtures + findings doc |
| felhom-controller | `bb50e12` | Parts 1–3 — ladder, severity, persisted state + tests |
| felhom-controller | `c24f192` | Group L strengthened to two post-restart checks |
| felhom-controller | `34d83f5` | Part 4 — cadence 6h → 1h (measured) |
| felhom-controller | `8144a70` | Part 5 — CHANGELOG / CONTEXT / README / REUSE |
| felhom.eu | `767960b` | Part 5 — capability map, ROADMAP, register rows R-328…R-334 |
| felhom-controller | `90f2545` | **v0.216.0** — R-335, one physical disk evaluated once per run |
| felhom.eu | `fa4748d` | R-335 register row |
---
## 4. Per-test results — all twelve groups
| Group | Scenario | Test | Result |
|-------|----------|------|--------|
| A | real drive, 2nd observation | `TestDiskCheck_RealDrive_HibaAndOneCriticalEvent` + `TestLadder_RealDrive_ReachesHiba` | **PASS** |
| B | the transient that cleared | `TestDiskCheck_FirstSightingIsWarnOnly` | **PASS** |
| C | sustained → Hiba | `TestDiskLadder_SustainDrivesTheEscalation` + `TestLadder_SustainIsWhatFires` | **PASS** |
| D | recovered, silent | `TestDiskCheck_RecoveryIsSilentAndClearsState` | **PASS** |
| E | flap damping | `TestDiskCheck_FlapDamping` | **PASS** |
| F | escalation beats damping | `TestDiskCheck_EscalationBeatsDamping` | **PASS** |
| G | still getting worse | `TestDiskCheck_RealertWhenStillWorsening` | **PASS** |
| H | below both bars | `TestDiskCheck_NoRealertBelowBothBars` | **PASS** |
| I | heat | `TestLadder_Temperature` + `TestDiskCheck_TemperatureShape` | **PASS** |
| J | no data never alarms | `TestDiskCheck_UnknownNeverAlarmsNorErasesPrior` + `TestLadder_UnknownNeverAlarms` | **PASS** |
| K | severity routes | `TestNotifyDiskHealthDegraded_SeverityRoutes` | **PASS** |
| L | state survives restart (seam) | `TestDiskCheck_StateSurvivesRestart_ProductionPath` | **PASS** |
Supporting: `TestLadder_CountBackstopBoundary`, `TestLadder_ZeroPriorIsFailSafe`,
`TestUncorrectableSectors`, `TestDegradedAttributes_NamesFailCounters`,
`TestNotifyDiskHealthDegraded_{WarnShape,FailShapes,CopyDiscipline}`,
`TestDiskAlertDecision_Table`, `TestDiskState_CorruptFileFallsBackToNoPrior`,
`TestDiskCheck_DisappearedDiskIsForgotten`, `TestDiskCheck_UnreachableAgentIsInert` — all PASS.
---
## 5. Red-proof outcomes — all twelve, individually
Each mutation was applied by script, **asserted present in the source before the run** (the harness
aborts with `MUTATION-NOT-APPLIED` if the target text is absent), the named test run, and the file
reverted with `git checkout --`. The tree was confirmed clean after the sweep.
| # | Mutation applied | Target test | Outcome |
|---|------------------|-------------|---------|
| A | remove truth-table row 6 (the sustain rule) | `TestDiskCheck_RealDrive_HibaAndOneCriticalEvent` | **RED-PROOF PASSED — FINDING, see below** |
| B | make row 9 return `Fail` | `TestDiskCheck_FirstSightingIsWarnOnly` | failed as required |
| C | pass a zero `DiskPrior` in `RunDiskHealthCheck` | `TestDiskLadder_SustainDrivesTheEscalation` | failed as required |
| D | let Rendben fall through the silence guard | `TestDiskCheck_RecoveryIsSilentAndClearsState` | failed as required |
| E | compare against last **observed** verdict, not last **alerted** | `TestDiskCheck_FlapDamping` | failed as required |
| F | let damping cover escalations | `TestDiskCheck_EscalationBeatsDamping` | failed as required |
| G | remove the re-alert branch | `TestDiskCheck_RealertWhenStillWorsening` | failed as required |
| H | make cooldown/doubling an **OR** instead of an AND | `TestDiskCheck_NoRealertBelowBothBars` | failed as required |
| I | remove truth-table rows 3 **and** 13 | `TestLadder_Temperature`, `TestDiskCheck_TemperatureShape` | failed as required |
| J | let UNKNOWN delete the prior record | `TestDiskCheck_UnknownNeverAlarmsNorErasesPrior` | failed as required |
| K | restore `severity := "warn"` | `TestNotifyDiskHealthDegraded_SeverityRoutes` | failed as required |
| L | skip loading the persisted state | `TestDiskCheck_StateSurvivesRestart_ProductionPath` | failed as required |
### A thirteenth red-proof, added after the deploy (R-335)
| # | Mutation applied | Target test | Outcome |
|---|------------------|-------------|---------|
| M | delete the `if seen[key] { continue }` dedup guard | `TestDiskCheck_SameDiskTwiceIsEvaluatedOnce` | failed as required |
Observed failure: `first sighting of an aliased disk must be silent, got 1: [{Label:felhom-backup … Kind:2 Sectors:8}]`
— i.e. `Kind:2` is `DiskAlertFailSectors`, a **Hiba on a first sighting of 8 sectors**. Reverted.
### FINDING — red-proof A passed, and it is the spec's mutation that is at fault, not the code
The task specified group A's red-proof as *"remove truth-table row 6 → verdict is Warn"*. **That
mutation cannot fail a test built on the real drive's values**: the real drive carries **352**
unreadable sectors, so with row 6 deleted it still reaches Hiba via **row 8** (count ≥ 64). The test
correctly stayed green, so the mutation proves nothing about row 6.
This was anticipated while writing the tests and is documented in the test's own comment rather than
discovered afterwards. **Row 6 is genuinely pinned**, by two tests that hold the counters at **8**
(far below the 64 backstop) and vary *only* the prior:
- `TestLadder_SustainIsWhatFires` (agentapi) — same `SmartSummary`, `DiskPrior{}` → Warn,
`DiskPrior{SawUncorrectable:true}` → Fail.
- `TestDiskLadder_SustainDrivesTheEscalation` (web) — the event-level twin.
Both were run under the row-6-deleted mutation and **both failed**, as recorded:
`SAME 8 sectors, now sustained = 2 (Figyelmeztetés), want Fail/Hiba` and
`severity = "warning", want critical` / `chip = "Figyelmeztetés", want Hiba`. So the invariant is
covered; only the spec's chosen mutation was invalid.
### A second finding, from building red-proof L
The first version of the Group L seam test ran **one** check after the restart and **passed under the
mutation** — because a controller that has forgotten its state is also silent on its first check. The
test was strengthened to run **two** checks (commit `c24f192`), after which the mutation fails. This
is the exact shape §10 warns about, caught by running the red-proof rather than assuming it.
---
## 6. Test count and suite state
- **Before:** 1391 test functions (at `3e3ee94`) · **After:** 1414 (+23)
- `go build ./... && go vet ./... && go test ./...` — **all green**, no failures, no skips introduced.
- `python3 controller/scripts/controller_gates.py` — **all 11 gates OK.**
---
## 7. Cadence measurement (Part 4)
Measured on **demo-hp** (Tier 0, disposable), through `fetchDisks`' real path — the agent local API
`GET /disks`, not the 60 s card cache. Ten consecutive calls, all **HTTP 200**:
```
0.840558 0.817000 0.815783 0.831992 0.809066
0.832899 0.824788 0.804886 0.812539 0.833547 (seconds)
```
| min | median | max | disk count |
|-----|--------|-----|------------|
| **0.804886 s** | **0.820894 s** | **0.840558 s** | **3 physical rows** across 2 devices (SanDisk X600 M.2 SATA SSD; Toshiba KXG50PNV1T02 NVMe, counted twice as `c11-scratch` + `felhom-backup`) |
**Branch taken: median < 5 s → `6*time.Hour` → `1*time.Hour`.** The median is ~6× under the bar. The
detection argument is the real one: the observed benign excursion lasted about **one hour**, so a
6-hourly sampler can land either side of it and then catch the terminal run half a day late.
No spin-up signature appeared in the timings (uniform ~0.82 s; demo-hp is all-flash), so the
measurement did not suggest the spun-down-drive concern. That question is recorded as an Observation
below and deliberately **not acted on**.
---
## 8. Live validation
Deployed to **demo-hp guest 9201** via the bootstrap path (`docker pull` →
`/etc/felhom-controller-image` → `systemctl restart felhom-controller-bootstrap.service`).
```
gitea.dooplex.hu/admin/felhom-controller:0.215.0 Up 19 seconds (healthy) # 06:23Z
gitea.dooplex.hu/admin/felhom-controller:0.216.0 Up 6 seconds (healthy) # after the R-335 fix
```
### Leg 1 — no over-correction (the load-bearing check)
Method: **endpoint-level** — authenticated `GET /dashboard` on the real controller
(`https://felhom.enkisfelhom.hu/dashboard`, HTTP 200, 44 036 bytes), i.e. the exact endpoint the UI
invokes; only rendering is skipped. No browser is available on DooPlex.
Card contents, parsed from the response body:
| Disk | Chip | Class | Temp |
|------|------|-------|------|
| KXG50PNV1T02 NVMe TOSHIBA 1024GB | **Rendben** | `state-text-run` | 53 °C |
| KXG50PNV1T02 NVMe TOSHIBA 1024GB | **Rendben** | `state-text-run` | 53 °C |
| SanDisk X600 M.2 2280 SATA 128GB | **Rendben** | `state-text-run` | 44 °C |
`Figyelmeztetés` = 0, `Hiba` = 0, `Nincs adat` = 0, `state-text-warn` = 0, `state-text-crit` = 0.
**No healthy disk was over-corrected.**
**Positive observable, at deploy:**
`[INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s)` — the new cadence is
in force, not merely compiled.
**Positive observable, per cycle** — two full hourly cycles observed after the deploy, from the
container log:
```
2026/08/14 06:23:13 [INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s)
2026/08/14 07:23:13 [INFO] [scheduler] Running job: disk-health-check
2026/08/14 07:23:14 [INFO] [web] disk-health check complete: 3 disk(s) evaluated, 0 alert(s)
2026/08/14 07:23:14 [INFO] [scheduler] Job disk-health-check completed (took 849ms)
2026/08/14 08:23:13 [INFO] [scheduler] Running job: disk-health-check
2026/08/14 08:23:14 [INFO] [web] disk-health check complete: 3 disk(s) evaluated, 0 alert(s)
2026/08/14 08:23:14 [INFO] [scheduler] Job disk-health-check completed (took 843ms)
```
`grep -c disk_health_degraded` over the whole container log: **0**. Both cycles ran (849 ms / 843 ms,
matching the §7 measurement), evaluated every disk, and emitted nothing. **Zero alerts from a check
that demonstrably ran** — not silence.
Persisted state written by the first cycle (`/opt/docker/felhom-controller/data/disk-health-state.json`,
428 bytes, on the `felhom-controller-data` docker volume, so it survives container recreation):
```json
{"version": 1, "disks": {
"path:/var/lib/vz": {"verdict": 1, "saw_uncorrectable": false, ...},
"uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae": {"verdict": 1, "saw_uncorrectable": false, ...}}}
```
`verdict: 1` is `DiskVerdictOK` for both, `saw_uncorrectable: false`, never alerted.
**Reading those two artefacts against each other is what exposed R-335** — see §14.
### Leg 2 — the severity fix arrives (the point of the task)
Two synthetic `disk_health_degraded` events pushed for customer `demo-hp` **through the real hub
event endpoint** (`POST https://hub.felhom.eu/api/v1/event`), from the guest's own controller using
its own hub credentials — the genuine controller→hub path, not a hand-crafted operator call. Both
returned `HTTP 200 {"ok":true}`. The hub DB was read with its `-wal` and `-shm` copied alongside
`hub.db` (a `hub.db`-only read is stale).
**As STORED by the hub (`events`):**
| id | severity pushed | severity STORED |
|----|-----------------|-----------------|
| 2964 | `warning` | **`warning`** |
| 2965 | `warn` | **`info`** ← coerced |
**`notification_log` rows for those two events:**
| id | event_type | severity | channel | status | error |
|----|-----------|----------|---------|--------|-------|
| 689 | `disk_health_degraded` | `warning` | `operator` | **`sent`** | *(none)* |
| — | *(the `"warn"` push)* | — | — | **NO ROW EXISTS** | — |
**That pair is the proof.** The identical event, differing only in one word of the severity string,
is the difference between *delivered to the operator* and *stored as an informational notice and
delivered to nobody*. This is the first time this leg has been observed end to end.
Only the operator leg fired because **demo-hp has no `customer_notifications` row at all** (no
customer email, no `enabled_events`), so no customer row was possible for either push — verified
directly, not assumed. **One real email was sent to the operator**, as the task anticipated.
---
## 9. NOT yet live-validated — stated explicitly
**The Fail-from-counters path has never fired on real hardware.** Everything in §4/§5 exercises it
against the committed fixture's values in unit tests only. The live legs above prove the *negative*
(no false alert on three healthy disks) and the *severity wire* (end to end, through the hub) — they
do **not** prove a live disk reaching Hiba. The fixture tests must not be read as a live proof.
Tracked as **R-332 (WATCHING)**. Closing condition: a live disk reaching Hiba from counters, or a
deliberate injection through the real pipeline (agent `/disks` → controller check → hub event) — not
a hand-set verdict.
**One item originally listed here has since been proven live** and is no longer part of this gap: the
**persisted state surviving a controller restart**. The v0.215.0 → v0.216.0 redeploy destroyed and
rebuilt the container, and the new one read back a `changed_at` written by the previous version rather
than re-baselining — see §14. What remains unproven is the stronger half: an already-**alerted** disk
not re-alerting after a restart, which needs a disk that has actually alerted. The drive that produced the fixture lives in DooPlex, which is Tier 2 and never a
drill target; the demo boxes are all-flash and healthy.
---
## 10. Teardown
**This run provisioned nothing** — no VM, no guest, no hub customer record, no storage. Nothing was
formatted, mounted, unmounted, repaired or written on any monitored disk; the only write is the
controller's own `disk-health-state.json` inside its data volume.
Disposition of what the run did create:
- **Two synthetic hub events (`events` id 2964, 2965) and one `notification_log` row (id 689)** on the
live hub. **Left in place deliberately.** Both messages are self-labelling
(`"R-328 severity probe (…) - synthetic, no real disk fault"`), and deleting rows from the
production hub DB is a riskier act than leaving two clearly-marked probe rows. Named here so they
are not mistaken later for a real disk fault on demo-hp.
- **One real operator email** resulting from row 689.
- A local copy of `hub.db`/`-wal`/`-shm` in the session scratchpad only (not committed, not exported).
---
## 11. Register rows
| Row | State | Owner |
|-----|-------|-------|
| **R-328** — the severity drop: `"warn"` coerced to `info`, emailed to nobody | **CLOSED** (controller v0.215.0), proven live side by side | CC |
| **R-329** — `app_start_failed` carries the identical defect | READY — **not fixed here**; needs a decision on whether it should notify at all | Viktor |
| **R-330** — Phase 2: collect SMART attrs 187/199/188 + persist samples | READY — a declared wire change, hub models it in the same session under G-1 | CC |
| **R-331** — Phase 3: growth-rate detection; revisit the static 64 | READY, blocked on R-330 | CC |
| **R-332** — the Fail path has never fired on real hardware | **WATCHING** | CC |
| **R-333** — NVMe temperature bands; agent `smartctl` has no `-n standby` | READY (S each) | Viktor decides (a); CC does (b) |
| **R-334** — released with no golden carrying it (gate waiver) | READY — now applies to **v0.216.0** | CC bakes; **Viktor vouches** |
| **R-335** — one physical disk walked twice per run, sustaining against itself | **CLOSED** (controller v0.216.0) | CC |
`smartd`-on-DooPlex-alerts-nobody is recorded in `DIAG-smart-passed-trap-2026-08-14.md` §8 as the
same shape one layer out.
---
## 12. Observations — noticed, NOT acted on
1. **`app_start_failed` has the identical severity defect** (`notifier.go` ~L546, `"warn"`). Left
untouched per scope. It needs a prior decision — should a stopped app email the customer at all? —
because flipping the string alone converts a silent event into a mail flood on a crash-looping box.
**R-329.**
2. **The 55/60 °C bands are spinning-disk bands being applied to NVMe, and this is close to biting.**
Adopted unchanged from the operator's Prometheus config by explicit decision — but demo-hp's
**healthy** Toshiba NVMe idles at **53 °C**, i.e. **2 °C below Figyelmeztetés and 7 °C below Hiba**,
and NVMe routinely passes 60 °C under sustained write with no fault. As shipped, a healthy customer
NVMe under load can be reported as **Hiba** — the single worst outcome this feature can produce, and
the one leg 1 exists to guard. Not changed here because the threshold is a stated, settled operator
decision; flagged rather than overridden. **R-333(a) — recommend splitting the bands by device
class, or dropping them for NVMe and relying on `critical_warning`.**
3. **The agent runs bare `smartctl -a -j` with no `-n standby`**
(`felhom-agent/internal/storage/hostops.go:368`), so every poll wakes a spun-down drive, and 6h → 1h
multiplies that by six. Recorded, not acted on, per the task's instruction. demo-hp is all-flash so
the measurement could not reveal it. Mitigating datum from the fixture: the failing drive logged
only **3375 load cycles in 60505 power-on hours** (~one per 18 h), so this duty cycle barely spins
down at all. **R-333(b).**
4. **`source ~/.config/credentials` prints two recovery codes to the terminal.** The file contains
hyphenated keys (`R_DEMO-FELHOM`, `R_DEMO-HP`) that bash cannot assign, so sourcing it emits
`command not found` errors **containing the secret values**. Anything that sources that file leaks
them into logs, scrollback and transcripts. Not a code defect and out of scope; worth quoting
values from it by other means, or renaming the keys.
5. **`golden_currency_gate.py` has no waiver parser.** Its own failure text says *"record a waiver in
`OPEN-ITEMS.md` — never a bypass"*, but nothing reads such a waiver, so the only way past it is the
bypass it warns against. See §13.
---
## 13. Deviations, stated plainly
- **`git push --no-verify` was used once**, on the `felhom.eu` docs push (`767960b`), and only there.
Cause: `golden_currency_gate.py` correctly convicts the fact that controller **v0.215.0 is released
and no golden carries it** (newest bake 0.214.0), so a *newly installed* machine would receive
0.214.0 — without the severity fix. A golden bake was out of the task's scope, and its second half
(vouching in the hub's day-0 artifact manifest) is operator-password-gated, so CC cannot complete it;
a baked-but-unvouched golden is worse than none. Recorded as **R-334** with the bake+vouch owners
named. CI re-runs the same entry point and will mail the operator. The running fleet is unaffected.
- **One pre-existing test changed meaning by design:** `TestDiskVerdictFor`'s
`critical_warning>0 → warn` case is now `→ fail` (truth-table row 4 — NVMe's own critical flag is a
device declaration, not a drifting counter). `TestDiskHealthCheck_DegradationOnce` and its siblings
were rewritten into the scenario groups because they encoded the pre-v0.215.0 single-alert behaviour
the task deliberately replaces (Scenario C).
---
## 14. R-335 — a defect in v0.215.0, found live, fixed as v0.216.0
**How it was found.** Not by a test and not by review: by reading the release's own **positive
observable** against the release's own **persisted artefact**. The hourly check logged *"3 disk(s)
evaluated"*; `disk-health-state.json` held **two** records. Two artefacts that should have agreed did
not.
**Cause.** demo-hp's `c11-scratch` and `felhom-backup` are the same physical NVMe (`/dev/nvme0n1`) and
resolve to the same `diskKey`, so one disk was walked twice in a single run.
**Why it mattered.** `RunDiskHealthCheck` writes a disk's new record before the next entry reads it, so
the **second** copy of an aliased disk consumed the **first** copy's write as its prior. The disk
therefore **sustained against itself and reached Hiba on a first sighting** — defeating truth-table
row 6, the single rule separating a one-hour benign excursion from a false critical alert — and would
have emitted **two identical events** for one drive.
**Severity in practice: latent, not active.** Nothing fired on demo-hp because all three entries are
healthy with zero counters. But any aliased disk developing one pending sector would have gone
straight to Hiba, which is precisely the outcome §8 leg 1 exists to prevent. Aliasing is not exotic —
it is the *normal* shape whenever a box has two PVE storage entries on one physical device.
**Fix (v0.216.0, `90f2545`).** Each `diskKey` is evaluated once per run. Both entries stay marked
`seen`, so neither is mistaken for a disappeared disk, and the card still renders **both** storage
rows — the dedup is about state and alerts, not display. Pinned by
`TestDiskCheck_SameDiskTwiceIsEvaluatedOnce`, red-proof run and reverted (§5).
**Deployed:** `gitea.dooplex.hu/admin/felhom-controller:0.216.0 Up 6 seconds (healthy)`.
**Confirming cycle on v0.216.0 — CONFIRMED LIVE, 09:31:35Z:**
```
live image: gitea.dooplex.hu/admin/felhom-controller:0.216.0 Up About an hour (healthy)
2026/08/14 09:31:35 [INFO] [web] disk-health check complete: 2 disk(s) evaluated, 0 alert(s)
grep -c disk_health_degraded: 0
```
**`2 disk(s) evaluated` now matches the 2 persisted records.** The count and the artefact agree, which
is the disagreement that exposed R-335 in the first place. Still zero alerts, still both card rows.
### The redeploy also proved persistence live — a gap §9 had listed as unproven
The 0.215.0 → 0.216.0 redeploy **replaced the container**, and the state file came back intact:
```json
"path:/var/lib/vz": {"verdict": 1, "changed_at": "2026-08-14T07:23:14.640216851Z", ...}
"uuid:91d2dc2d-…": {"verdict": 1, "changed_at": "2026-08-14T07:23:14.640216851Z", ...}
```
That `changed_at` was written by **v0.215.0's first cycle at 07:23Z**, before the container was
destroyed and rebuilt. The v0.216.0 container read it back and preserved it rather than stamping a
fresh time — so the new container **loaded the pre-restart record instead of silently re-baselining**.
That is Scenario L observed on real hardware, not just through the production-path unit test, and it
is exactly the behaviour that was impossible before v0.215.0 (the baseline was in-memory).
It also incidentally confirms the unchanged-verdict path: `changed_at` is preserved across four checks
and two controller versions because the verdict never changed, rather than being churned every cycle.
**What this still does NOT prove:** these disks are healthy and were never alerted, so the stronger
half — *an already-ALERTED disk not re-alerting after a restart* — remains unit-tested only. R-332
stands.
**Process note, recorded because it nearly cost the fix.** The red-proof harness reverts with
`git checkout --`, which restores to `HEAD`. Running a red-proof against an **uncommitted** fix
therefore *deletes the fix* along with the mutation — which happened here and was caught only by
re-grepping the source afterwards. Commit the fix before red-proofing it, or snapshot outside git.