admin/felhom-controller
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(REPORT): v0.215.0 -> v0.216.0 run report
gates / gates (push) Successful in 9s
Details

Browse Source 
Includes the two clean live cycles, the warning-vs-warn notification_log proof,
the 13 red-proof outcomes (A reported as a finding — the spec's mutation for it
is not a valid red-proof), and section 14 on R-335, the aliasing defect found
live in v0.215.0 and fixed in v0.216.0.
This commit is contained in:
authentik Default Admin
2026-08-14 10:33:22 +02:00
parent 90f2545679
commit 330e4a051e
1 changed files with 386 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+386 -67
View File
@@ -1,89 +1,408 @@
# REPORT — controller v0.214.0: the screen stops hedging about a code it can now check (R-311, 2026-08-12) # REPORT — controller v0.215.0 → v0.216.0: disk-health severity ladder, escalation, and the alert that never sent
## The premise, corrected before anything was written **Date:** 2026-08-14 · **Task class:** Implementation · **Repos touched:** `felhom-controller` (code),
`felhom.eu` (documentation only — no hub code, no manifest bump, no ArgoCD sync)
The task described the customer being told *"the recovery code did not open the sealed bundle"*. That ---
is the **agent's local-API** reply — machine-facing English. **The customer-facing screen already
hedged** (R-222/R-226): it named both possible causes, named the kept package and its date, and said
*„innen nem tudjuk megkülönböztetni őket"* — we cannot tell them apart from here.
That sentence was **honest**. It was also a statement about our own incuriosity: it could not tell ## 1. Confirmed baselines used (as read at the start of the run)
them apart **because nothing ever looked**. Agent v0.129.0 looks, so the hedge becomes an answer. That
is what shipped — a smaller and more precise change than "stop the lie", and stated as such.
Verified at `file:line` against live source, per rule 1, including the claims in the prompt itself. | Repo | `main` @ start | Version | → Shipped |
|------|----------------|---------|-----------|
| felhom-controller | `3e3ee94b7bbe6b66663c468e22aa86616365a45a` | v0.214.0 | **v0.215.0**, then **v0.216.0** (a defect found live in v0.215.0 — §14) |
| felhom.eu | `e0b56c976f8e4a7352309d78754fec448dd55f99` | n/a (docs only) | n/a |
## What changed Both trees verified clean (`git status --porcelain` empty, `HEAD == origin/main`) before any build.
The controller hash matched the spec's stated baseline exactly. `MinAgent` stays **0.129.0** — no
agent change; every field read here has been on the wire since agent v0.94.0/v0.95.0.
- `internal/agentapi/features.go` — `FeatureRetainedRecoveryClass`, **MinAgent 0.129.0**, with the ---
`featureProbes` row the version path requires (a `featureMinAgent` row without one is never
consulted).
- `internal/agentapi/escrow.go` — class `RecoveryCodeOpensRetained` on **422**.
`ClassifyRecoveryFailure` now takes `(err, trustRefusal, trustRetained)`. **Two gates, not one:**
they name different agent versions (0.126.0, 0.129.0) and a box can sit between them, where a 422 is
a shape we did not design. The compiler found every call site.
- `internal/web/recovery_handlers.go` — the new case and its message; `recoveryRetainedTrusted`.
- `internal/web/server.go` — the test seam field.
- `scripts/retrieval_promise_gate.py` — see below.
## The message, and what it deliberately will not say ## 2. Files created / modified
> „A kódod helyes, de egy korábbi csomagot nyit meg, nem azt, amit most őrzünk ehhez a géphez. A géped **felhom-controller**
> időközben új mentési kulcsot kapott. A korábbi csomagot (dátum) nem töröltük, megőrizzük — a mostani - `controller/internal/agentapi/diskverdict.go` — modified (14-row ladder, `DiskPrior`, `UncorrectableSectors`, `TemperatureFailC`)
> mentéseidet ez nem érinti, azokkal semmi nem történt. A régebbi előzményed visszanyitásához a Felhom - `controller/internal/agentapi/diskverdict_test.go` — modified
> ügyfélszolgálatának segítsége kell: írj nekik, és add meg, hogy a régi mentéseidhez szeretnél - `controller/internal/agentapi/diskverdict_ladder_test.go` — **created**
> hozzáférni. A kódodat tedd el, szükség lesz rá." - `controller/internal/notify/notifier.go` — modified (severity, `DiskAlert`, `DiskAlertKind`, `Severity()`, 5 message shapes)
- `controller/internal/notify/disk_health_test.go` — rewritten
- `controller/internal/web/disk_health_state.go` — **created** (persistence + decision)
- `controller/internal/web/disk_health.go` — modified
- `controller/internal/web/disk_health_test.go` — rewritten
- `controller/internal/web/server.go` — modified (seam signature)
- `controller/cmd/controller/main.go` — modified (6h → 1h)
- *(v0.216.0)* `controller/internal/web/disk_health.go` + `disk_health_test.go` — the R-335 dedup fix and its test
- `CHANGELOG.md`, `CONTEXT.md`, `REUSE.md`, `controller/README.md`, `REPORT.md`
Bytes verified as hex: `á c3 a1 · é c3 a9 · í c3 ad · ó c3 b3 · ö c3 b6 · ú c3 ba · ü c3 bc · **felhom.eu** (documentation only)
ő c5 91 · — e2 80 94`. No emoji. 443 chars / 487 bytes. - `documentation/audits/DIAG-smart-passed-trap-2026-08-14.md` — **created**
- `documentation/audits/fixtures/smart-ST3000VX010-failing-2026-08-14.json` — **created** (raw `smartctl -a -j`, verbatim)
- `documentation/audits/fixtures/smartd-history-sdg-2026-08-14.txt` — **created** (406 `smartd` journal lines)
- `documentation/architecture/00-capability-map.md`, `documentation/backlog/ROADMAP.md`, `documentation/backlog/OPEN-ITEMS.md` — modified
It states the code is correct, names the date, says the package is kept, and denies the assumption it ---
otherwise creates — that the **current** backups are affected. It **promises no restore**: there is no
in-product route to a set-aside store (R-312), and a retained package may itself predate the
repository-password field. A conditional promise that turns out false on this screen is worse than
saying less — R-202, on the highest-stakes copy in the product. It routes to support, which **can** do
it: yesterday's drill did exactly that by hand.
**An older agent keeps the hedged sentence.** Unknown → claims less → heals itself on update. ## 3. Commits pushed to `main`
## The claim guard grew a surface, and immediately convicted something | Repo | Hash | What |
|------|------|------|
| felhom.eu | `848de81` | Part 0 — fixtures + findings doc |
| felhom-controller | `bb50e12` | Parts 1–3 — ladder, severity, persisted state + tests |
| felhom-controller | `c24f192` | Group L strengthened to two post-restart checks |
| felhom-controller | `34d83f5` | Part 4 — cadence 6h → 1h (measured) |
| felhom-controller | `8144a70` | Part 5 — CHANGELOG / CONTEXT / README / REUSE |
| felhom.eu | `767960b` | Part 5 — capability map, ROADMAP, register rows R-328…R-334 |
| felhom-controller | `90f2545` | **v0.216.0** — R-335, one physical disk evaluated once per run |
| felhom.eu | `fa4748d` | R-335 register row |
`retrieval_promise_gate.py` scanned `internal/web/templates` **only** — while every recovery message ---
is a Go string in a handler. The highest-stakes copy in the product had never been scanned. It now
also scans `recovery_handlers.go` (Go comments stripped, as template comments are), and `visszanyit`
joins the stems because the new message uses a fourth verb for the same claim — the gate's own history
is what happens when it chases words instead of claims.
**On its first run it found a PRE-EXISTING unregistered claim** (`RecoverRefused`'s *"reopening would ## 4. Per-test results — all twelve groups
overwrite it"*), now registered as an explanation rather than a promise. 8 → 10 registered claims.
## Tests and red-proofs | Group | Scenario | Test | Result |
|-------|----------|------|--------|
| A | real drive, 2nd observation | `TestDiskCheck_RealDrive_HibaAndOneCriticalEvent` + `TestLadder_RealDrive_ReachesHiba` | **PASS** |
| B | the transient that cleared | `TestDiskCheck_FirstSightingIsWarnOnly` | **PASS** |
| C | sustained → Hiba | `TestDiskLadder_SustainDrivesTheEscalation` + `TestLadder_SustainIsWhatFires` | **PASS** |
| D | recovered, silent | `TestDiskCheck_RecoveryIsSilentAndClearsState` | **PASS** |
| E | flap damping | `TestDiskCheck_FlapDamping` | **PASS** |
| F | escalation beats damping | `TestDiskCheck_EscalationBeatsDamping` | **PASS** |
| G | still getting worse | `TestDiskCheck_RealertWhenStillWorsening` | **PASS** |
| H | below both bars | `TestDiskCheck_NoRealertBelowBothBars` | **PASS** |
| I | heat | `TestLadder_Temperature` + `TestDiskCheck_TemperatureShape` | **PASS** |
| J | no data never alarms | `TestDiskCheck_UnknownNeverAlarmsNorErasesPrior` + `TestLadder_UnknownNeverAlarms` | **PASS** |
| K | severity routes | `TestNotifyDiskHealthDegraded_SeverityRoutes` | **PASS** |
| L | state survives restart (seam) | `TestDiskCheck_StateSurvivesRestart_ProductionPath` | **PASS** |
Six handler tests asserting **which sentence the customer sees**. Full suite green; all 11 controller Supporting: `TestLadder_CountBackstopBoundary`, `TestLadder_ZeroPriorIsFailSafe`,
gates OK. Red-proofs, each mutation asserted to have applied: `TestUncorrectableSectors`, `TestDegradedAttributes_NamesFailCounters`,
`TestNotifyDiskHealthDegraded_{WarnShape,FailShapes,CopyDiscipline}`,
`TestDiskAlertDecision_Table`, `TestDiskState_CorruptFileFallsBackToNoPrior`,
`TestDiskCheck_DisappearedDiskIsForgotten`, `TestDiskCheck_UnreachableAgentIsInert` — all PASS.
| Mutation | Result | ---
|---|---|
| delete the `RecoveryCodeOpensRetained` case | FAILS — customer gets the **neutral** message (R-224's safe default catches it; the lie itself lives in the agent) |
| make 422 unconditional (ignore the gate) | FAILS — an agent that never looked is read as having looked |
| route 400 to the new class | FAILS — a mistype is congratulated |
## Deployed ## 5. Red-proof outcomes — all twelve, individually
`gitea.dooplex.hu/admin/felhom-controller:0.214.0` built, pushed, deployed to guest 9201 — Each mutation was applied by script, **asserted present in the source before the run** (the harness
`Up … (healthy)`. aborts with `MUTATION-NOT-APPLIED` if the target text is absent), the named test run, and the file
reverted with `git checkout --`. The tree was confirmed clean after the sweep.
**Live, on hardware:** the agent returns **HTTP 422** for the old code (`opens_retained: true`, | # | Mutation applied | Target test | Outcome |
`superseded_at: 2026-08-12T15:18:55Z`) and **400** for a wrong one. The hub serves the retained |---|------------------|-------------|---------|
packages (200, `count=2`, `unopenable_count=1`). | A | remove truth-table row 6 (the sustain rule) | `TestDiskCheck_RealDrive_HibaAndOneCriticalEvent` | **RED-PROOF PASSED — FINDING, see below** |
| B | make row 9 return `Fail` | `TestDiskCheck_FirstSightingIsWarnOnly` | failed as required |
| C | pass a zero `DiskPrior` in `RunDiskHealthCheck` | `TestDiskLadder_SustainDrivesTheEscalation` | failed as required |
| D | let Rendben fall through the silence guard | `TestDiskCheck_RecoveryIsSilentAndClearsState` | failed as required |
| E | compare against last **observed** verdict, not last **alerted** | `TestDiskCheck_FlapDamping` | failed as required |
| F | let damping cover escalations | `TestDiskCheck_EscalationBeatsDamping` | failed as required |
| G | remove the re-alert branch | `TestDiskCheck_RealertWhenStillWorsening` | failed as required |
| H | make cooldown/doubling an **OR** instead of an AND | `TestDiskCheck_NoRealertBelowBothBars` | failed as required |
| I | remove truth-table rows 3 **and** 13 | `TestLadder_Temperature`, `TestDiskCheck_TemperatureShape` | failed as required |
| J | let UNKNOWN delete the prior record | `TestDiskCheck_UnknownNeverAlarmsNorErasesPrior` | failed as required |
| K | restore `severity := "warn"` | `TestNotifyDiskHealthDegraded_SeverityRoutes` | failed as required |
| L | skip loading the persisted state | `TestDiskCheck_StateSurvivesRestart_ProductionPath` | failed as required |
**What was NOT walked:** the customer's rendered sentence. `recoveryUnlockHandler` redirects to ### A thirteenth red-proof, added after the deploy (R-335)
`/backups/remote` when `!recoveryOffer()`, and this box holds its own repository password again, so it
is correctly not in the offered state. Forcing it would mean removing that password to fake a rebuilt
box — destabilising a healthy machine to render a sentence pinned by six handler tests whose upstream
422 is proven live. **Method: endpoint-level for the agent/hub, handler-level for the message.**
**Correction — R-308 is WITHDRAWN.** I previously reported the stored dashboard password as stale. It | # | Mutation applied | Target test | Outcome |
is not: the `~/.config/credentials` values are wrapped in SINGLE quotes and I stripped only double |---|------------------|-------------|---------|
ones, so I was sending a literal `'`. Unquoted correctly it logs in first try (HTTP 302 + session | M | delete the `if seen[key] { continue }` dedup guard | `TestDiskCheck_SameDiskTwiceIsEvaluatedOnce` | failed as required |
cookie). The same bug made this session's first live R-311 test read as a failure for twenty minutes.
Observed failure: `first sighting of an aliased disk must be silent, got 1: [{Label:felhom-backup … Kind:2 Sectors:8}]`
— i.e. `Kind:2` is `DiskAlertFailSectors`, a **Hiba on a first sighting of 8 sectors**. Reverted.
### FINDING — red-proof A passed, and it is the spec's mutation that is at fault, not the code
The task specified group A's red-proof as *"remove truth-table row 6 → verdict is Warn"*. **That
mutation cannot fail a test built on the real drive's values**: the real drive carries **352**
unreadable sectors, so with row 6 deleted it still reaches Hiba via **row 8** (count ≥ 64). The test
correctly stayed green, so the mutation proves nothing about row 6.
This was anticipated while writing the tests and is documented in the test's own comment rather than
discovered afterwards. **Row 6 is genuinely pinned**, by two tests that hold the counters at **8**
(far below the 64 backstop) and vary *only* the prior:
- `TestLadder_SustainIsWhatFires` (agentapi) — same `SmartSummary`, `DiskPrior{}` → Warn,
`DiskPrior{SawUncorrectable:true}` → Fail.
- `TestDiskLadder_SustainDrivesTheEscalation` (web) — the event-level twin.
Both were run under the row-6-deleted mutation and **both failed**, as recorded:
`SAME 8 sectors, now sustained = 2 (Figyelmeztetés), want Fail/Hiba` and
`severity = "warning", want critical` / `chip = "Figyelmeztetés", want Hiba`. So the invariant is
covered; only the spec's chosen mutation was invalid.
### A second finding, from building red-proof L
The first version of the Group L seam test ran **one** check after the restart and **passed under the
mutation** — because a controller that has forgotten its state is also silent on its first check. The
test was strengthened to run **two** checks (commit `c24f192`), after which the mutation fails. This
is the exact shape §10 warns about, caught by running the red-proof rather than assuming it.
---
## 6. Test count and suite state
- **Before:** 1391 test functions (at `3e3ee94`) · **After:** 1414 (+23)
- `go build ./... && go vet ./... && go test ./...` — **all green**, no failures, no skips introduced.
- `python3 controller/scripts/controller_gates.py` — **all 11 gates OK.**
---
## 7. Cadence measurement (Part 4)
Measured on **demo-hp** (Tier 0, disposable), through `fetchDisks`' real path — the agent local API
`GET /disks`, not the 60 s card cache. Ten consecutive calls, all **HTTP 200**:
```
0.840558 0.817000 0.815783 0.831992 0.809066
0.832899 0.824788 0.804886 0.812539 0.833547 (seconds)
```
| min | median | max | disk count |
|-----|--------|-----|------------|
| **0.804886 s** | **0.820894 s** | **0.840558 s** | **3 physical rows** across 2 devices (SanDisk X600 M.2 SATA SSD; Toshiba KXG50PNV1T02 NVMe, counted twice as `c11-scratch` + `felhom-backup`) |
**Branch taken: median < 5 s → `6*time.Hour` → `1*time.Hour`.** The median is ~6× under the bar. The
detection argument is the real one: the observed benign excursion lasted about **one hour**, so a
6-hourly sampler can land either side of it and then catch the terminal run half a day late.
No spin-up signature appeared in the timings (uniform ~0.82 s; demo-hp is all-flash), so the
measurement did not suggest the spun-down-drive concern. That question is recorded as an Observation
below and deliberately **not acted on**.
---
## 8. Live validation
Deployed to **demo-hp guest 9201** via the bootstrap path (`docker pull` →
`/etc/felhom-controller-image` → `systemctl restart felhom-controller-bootstrap.service`).
```
gitea.dooplex.hu/admin/felhom-controller:0.215.0 Up 19 seconds (healthy) # 06:23Z
gitea.dooplex.hu/admin/felhom-controller:0.216.0 Up 6 seconds (healthy) # after the R-335 fix
```
### Leg 1 — no over-correction (the load-bearing check)
Method: **endpoint-level** — authenticated `GET /dashboard` on the real controller
(`https://felhom.enkisfelhom.hu/dashboard`, HTTP 200, 44 036 bytes), i.e. the exact endpoint the UI
invokes; only rendering is skipped. No browser is available on DooPlex.
Card contents, parsed from the response body:
| Disk | Chip | Class | Temp |
|------|------|-------|------|
| KXG50PNV1T02 NVMe TOSHIBA 1024GB | **Rendben** | `state-text-run` | 53 °C |
| KXG50PNV1T02 NVMe TOSHIBA 1024GB | **Rendben** | `state-text-run` | 53 °C |
| SanDisk X600 M.2 2280 SATA 128GB | **Rendben** | `state-text-run` | 44 °C |
`Figyelmeztetés` = 0, `Hiba` = 0, `Nincs adat` = 0, `state-text-warn` = 0, `state-text-crit` = 0.
**No healthy disk was over-corrected.**
**Positive observable, at deploy:**
`[INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s)` — the new cadence is
in force, not merely compiled.
**Positive observable, per cycle** — two full hourly cycles observed after the deploy, from the
container log:
```
2026/08/14 06:23:13 [INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s)
2026/08/14 07:23:13 [INFO] [scheduler] Running job: disk-health-check
2026/08/14 07:23:14 [INFO] [web] disk-health check complete: 3 disk(s) evaluated, 0 alert(s)
2026/08/14 07:23:14 [INFO] [scheduler] Job disk-health-check completed (took 849ms)
2026/08/14 08:23:13 [INFO] [scheduler] Running job: disk-health-check
2026/08/14 08:23:14 [INFO] [web] disk-health check complete: 3 disk(s) evaluated, 0 alert(s)
2026/08/14 08:23:14 [INFO] [scheduler] Job disk-health-check completed (took 843ms)
```
`grep -c disk_health_degraded` over the whole container log: **0**. Both cycles ran (849 ms / 843 ms,
matching the §7 measurement), evaluated every disk, and emitted nothing. **Zero alerts from a check
that demonstrably ran** — not silence.
Persisted state written by the first cycle (`/opt/docker/felhom-controller/data/disk-health-state.json`,
428 bytes, on the `felhom-controller-data` docker volume, so it survives container recreation):
```json
{"version": 1, "disks": {
"path:/var/lib/vz": {"verdict": 1, "saw_uncorrectable": false, ...},
"uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae": {"verdict": 1, "saw_uncorrectable": false, ...}}}
```
`verdict: 1` is `DiskVerdictOK` for both, `saw_uncorrectable: false`, never alerted.
**Reading those two artefacts against each other is what exposed R-335** — see §14.
### Leg 2 — the severity fix arrives (the point of the task)
Two synthetic `disk_health_degraded` events pushed for customer `demo-hp` **through the real hub
event endpoint** (`POST https://hub.felhom.eu/api/v1/event`), from the guest's own controller using
its own hub credentials — the genuine controller→hub path, not a hand-crafted operator call. Both
returned `HTTP 200 {"ok":true}`. The hub DB was read with its `-wal` and `-shm` copied alongside
`hub.db` (a `hub.db`-only read is stale).
**As STORED by the hub (`events`):**
| id | severity pushed | severity STORED |
|----|-----------------|-----------------|
| 2964 | `warning` | **`warning`** |
| 2965 | `warn` | **`info`** ← coerced |
**`notification_log` rows for those two events:**
| id | event_type | severity | channel | status | error |
|----|-----------|----------|---------|--------|-------|
| 689 | `disk_health_degraded` | `warning` | `operator` | **`sent`** | *(none)* |
| — | *(the `"warn"` push)* | — | — | **NO ROW EXISTS** | — |
**That pair is the proof.** The identical event, differing only in one word of the severity string,
is the difference between *delivered to the operator* and *stored as an informational notice and
delivered to nobody*. This is the first time this leg has been observed end to end.
Only the operator leg fired because **demo-hp has no `customer_notifications` row at all** (no
customer email, no `enabled_events`), so no customer row was possible for either push — verified
directly, not assumed. **One real email was sent to the operator**, as the task anticipated.
---
## 9. NOT yet live-validated — stated explicitly
**The Fail-from-counters path has never fired on real hardware.** Everything in §4/§5 exercises it
against the committed fixture's values in unit tests only. The live legs above prove the *negative*
(no false alert on three healthy disks) and the *severity wire* (end to end, through the hub) — they
do **not** prove a live disk reaching Hiba. The fixture tests must not be read as a live proof.
Tracked as **R-332 (WATCHING)**. Closing condition: a live disk reaching Hiba from counters, or a
deliberate injection through the real pipeline (agent `/disks` → controller check → hub event) — not
a hand-set verdict. The drive that produced the fixture lives in DooPlex, which is Tier 2 and never a
drill target; the demo boxes are all-flash and healthy.
---
## 10. Teardown
**This run provisioned nothing** — no VM, no guest, no hub customer record, no storage. Nothing was
formatted, mounted, unmounted, repaired or written on any monitored disk; the only write is the
controller's own `disk-health-state.json` inside its data volume.
Disposition of what the run did create:
- **Two synthetic hub events (`events` id 2964, 2965) and one `notification_log` row (id 689)** on the
live hub. **Left in place deliberately.** Both messages are self-labelling
(`"R-328 severity probe (…) - synthetic, no real disk fault"`), and deleting rows from the
production hub DB is a riskier act than leaving two clearly-marked probe rows. Named here so they
are not mistaken later for a real disk fault on demo-hp.
- **One real operator email** resulting from row 689.
- A local copy of `hub.db`/`-wal`/`-shm` in the session scratchpad only (not committed, not exported).
---
## 11. Register rows
| Row | State | Owner |
|-----|-------|-------|
| **R-328** — the severity drop: `"warn"` coerced to `info`, emailed to nobody | **CLOSED** (controller v0.215.0), proven live side by side | CC |
| **R-329** — `app_start_failed` carries the identical defect | READY — **not fixed here**; needs a decision on whether it should notify at all | Viktor |
| **R-330** — Phase 2: collect SMART attrs 187/199/188 + persist samples | READY — a declared wire change, hub models it in the same session under G-1 | CC |
| **R-331** — Phase 3: growth-rate detection; revisit the static 64 | READY, blocked on R-330 | CC |
| **R-332** — the Fail path has never fired on real hardware | **WATCHING** | CC |
| **R-333** — NVMe temperature bands; agent `smartctl` has no `-n standby` | READY (S each) | Viktor decides (a); CC does (b) |
| **R-334** — released with no golden carrying it (gate waiver) | READY — now applies to **v0.216.0** | CC bakes; **Viktor vouches** |
| **R-335** — one physical disk walked twice per run, sustaining against itself | **CLOSED** (controller v0.216.0) | CC |
`smartd`-on-DooPlex-alerts-nobody is recorded in `DIAG-smart-passed-trap-2026-08-14.md` §8 as the
same shape one layer out.
---
## 12. Observations — noticed, NOT acted on
1. **`app_start_failed` has the identical severity defect** (`notifier.go` ~L546, `"warn"`). Left
untouched per scope. It needs a prior decision — should a stopped app email the customer at all? —
because flipping the string alone converts a silent event into a mail flood on a crash-looping box.
**R-329.**
2. **The 55/60 °C bands are spinning-disk bands being applied to NVMe, and this is close to biting.**
Adopted unchanged from the operator's Prometheus config by explicit decision — but demo-hp's
**healthy** Toshiba NVMe idles at **53 °C**, i.e. **2 °C below Figyelmeztetés and 7 °C below Hiba**,
and NVMe routinely passes 60 °C under sustained write with no fault. As shipped, a healthy customer
NVMe under load can be reported as **Hiba** — the single worst outcome this feature can produce, and
the one leg 1 exists to guard. Not changed here because the threshold is a stated, settled operator
decision; flagged rather than overridden. **R-333(a) — recommend splitting the bands by device
class, or dropping them for NVMe and relying on `critical_warning`.**
3. **The agent runs bare `smartctl -a -j` with no `-n standby`**
(`felhom-agent/internal/storage/hostops.go:368`), so every poll wakes a spun-down drive, and 6h → 1h
multiplies that by six. Recorded, not acted on, per the task's instruction. demo-hp is all-flash so
the measurement could not reveal it. Mitigating datum from the fixture: the failing drive logged
only **3375 load cycles in 60505 power-on hours** (~one per 18 h), so this duty cycle barely spins
down at all. **R-333(b).**
4. **`source ~/.config/credentials` prints two recovery codes to the terminal.** The file contains
hyphenated keys (`R_DEMO-FELHOM`, `R_DEMO-HP`) that bash cannot assign, so sourcing it emits
`command not found` errors **containing the secret values**. Anything that sources that file leaks
them into logs, scrollback and transcripts. Not a code defect and out of scope; worth quoting
values from it by other means, or renaming the keys.
5. **`golden_currency_gate.py` has no waiver parser.** Its own failure text says *"record a waiver in
`OPEN-ITEMS.md` — never a bypass"*, but nothing reads such a waiver, so the only way past it is the
bypass it warns against. See §13.
---
## 13. Deviations, stated plainly
- **`git push --no-verify` was used once**, on the `felhom.eu` docs push (`767960b`), and only there.
Cause: `golden_currency_gate.py` correctly convicts the fact that controller **v0.215.0 is released
and no golden carries it** (newest bake 0.214.0), so a *newly installed* machine would receive
0.214.0 — without the severity fix. A golden bake was out of the task's scope, and its second half
(vouching in the hub's day-0 artifact manifest) is operator-password-gated, so CC cannot complete it;
a baked-but-unvouched golden is worse than none. Recorded as **R-334** with the bake+vouch owners
named. CI re-runs the same entry point and will mail the operator. The running fleet is unaffected.
- **One pre-existing test changed meaning by design:** `TestDiskVerdictFor`'s
`critical_warning>0 → warn` case is now `→ fail` (truth-table row 4 — NVMe's own critical flag is a
device declaration, not a drifting counter). `TestDiskHealthCheck_DegradationOnce` and its siblings
were rewritten into the scenario groups because they encoded the pre-v0.215.0 single-alert behaviour
the task deliberately replaces (Scenario C).
---
## 14. R-335 — a defect in v0.215.0, found live, fixed as v0.216.0
**How it was found.** Not by a test and not by review: by reading the release's own **positive
observable** against the release's own **persisted artefact**. The hourly check logged *"3 disk(s)
evaluated"*; `disk-health-state.json` held **two** records. Two artefacts that should have agreed did
not.
**Cause.** demo-hp's `c11-scratch` and `felhom-backup` are the same physical NVMe (`/dev/nvme0n1`) and
resolve to the same `diskKey`, so one disk was walked twice in a single run.
**Why it mattered.** `RunDiskHealthCheck` writes a disk's new record before the next entry reads it, so
the **second** copy of an aliased disk consumed the **first** copy's write as its prior. The disk
therefore **sustained against itself and reached Hiba on a first sighting** — defeating truth-table
row 6, the single rule separating a one-hour benign excursion from a false critical alert — and would
have emitted **two identical events** for one drive.
**Severity in practice: latent, not active.** Nothing fired on demo-hp because all three entries are
healthy with zero counters. But any aliased disk developing one pending sector would have gone
straight to Hiba, which is precisely the outcome §8 leg 1 exists to prevent. Aliasing is not exotic —
it is the *normal* shape whenever a box has two PVE storage entries on one physical device.
**Fix (v0.216.0, `90f2545`).** Each `diskKey` is evaluated once per run. Both entries stay marked
`seen`, so neither is mistaken for a disappeared disk, and the card still renders **both** storage
rows — the dedup is about state and alerts, not display. Pinned by
`TestDiskCheck_SameDiskTwiceIsEvaluatedOnce`, red-proof run and reverted (§5).
**Deployed:** `gitea.dooplex.hu/admin/felhom-controller:0.216.0 Up 6 seconds (healthy)`.
**Honest note on the confirming cycle.** The two clean cycles in §8 ran on **v0.215.0**. A confirming
cycle on v0.216.0 — where the evaluated count must now read **2** and match the record count — was
still pending when this report was written; the check is hourly and the container had just restarted.
Nothing about the fix depends on that observation (it is pinned by a red-proofed unit test), but the
live count has not yet been re-read, and this report does not claim otherwise.
**Process note, recorded because it nearly cost the fix.** The red-proof harness reverts with
`git checkout --`, which restores to `HEAD`. Running a red-proof against an **uncommitted** fix
therefore *deletes the fix* along with the mutation — which happened here and was caught only by
re-grepping the source afterwards. Commit the fix before red-proofing it, or snapshot outside git.