Includes the two clean live cycles, the warning-vs-warn notification_log proof, the 13 red-proof outcomes (A reported as a finding — the spec's mutation for it is not a valid red-proof), and section 14 on R-335, the aliasing defect found live in v0.215.0 and fixed in v0.216.0.
This commit is contained in:
@@ -1,89 +1,408 @@
|
||||
# REPORT — controller v0.214.0: the screen stops hedging about a code it can now check (R-311, 2026-08-12)
|
||||
# REPORT — controller v0.215.0 → v0.216.0: disk-health severity ladder, escalation, and the alert that never sent
|
||||
|
||||
## The premise, corrected before anything was written
|
||||
**Date:** 2026-08-14 · **Task class:** Implementation · **Repos touched:** `felhom-controller` (code),
|
||||
`felhom.eu` (documentation only — no hub code, no manifest bump, no ArgoCD sync)
|
||||
|
||||
The task described the customer being told *"the recovery code did not open the sealed bundle"*. That
|
||||
is the **agent's local-API** reply — machine-facing English. **The customer-facing screen already
|
||||
hedged** (R-222/R-226): it named both possible causes, named the kept package and its date, and said
|
||||
*„innen nem tudjuk megkülönböztetni őket"* — we cannot tell them apart from here.
|
||||
---
|
||||
|
||||
That sentence was **honest**. It was also a statement about our own incuriosity: it could not tell
|
||||
them apart **because nothing ever looked**. Agent v0.129.0 looks, so the hedge becomes an answer. That
|
||||
is what shipped — a smaller and more precise change than "stop the lie", and stated as such.
|
||||
## 1. Confirmed baselines used (as read at the start of the run)
|
||||
|
||||
Verified at `file:line` against live source, per rule 1, including the claims in the prompt itself.
|
||||
| Repo | `main` @ start | Version | → Shipped |
|
||||
|------|----------------|---------|-----------|
|
||||
| felhom-controller | `3e3ee94b7bbe6b66663c468e22aa86616365a45a` | v0.214.0 | **v0.215.0**, then **v0.216.0** (a defect found live in v0.215.0 — §14) |
|
||||
| felhom.eu | `e0b56c976f8e4a7352309d78754fec448dd55f99` | n/a (docs only) | n/a |
|
||||
|
||||
## What changed
|
||||
Both trees verified clean (`git status --porcelain` empty, `HEAD == origin/main`) before any build.
|
||||
The controller hash matched the spec's stated baseline exactly. `MinAgent` stays **0.129.0** — no
|
||||
agent change; every field read here has been on the wire since agent v0.94.0/v0.95.0.
|
||||
|
||||
- `internal/agentapi/features.go` — `FeatureRetainedRecoveryClass`, **MinAgent 0.129.0**, with the
|
||||
`featureProbes` row the version path requires (a `featureMinAgent` row without one is never
|
||||
consulted).
|
||||
- `internal/agentapi/escrow.go` — class `RecoveryCodeOpensRetained` on **422**.
|
||||
`ClassifyRecoveryFailure` now takes `(err, trustRefusal, trustRetained)`. **Two gates, not one:**
|
||||
they name different agent versions (0.126.0, 0.129.0) and a box can sit between them, where a 422 is
|
||||
a shape we did not design. The compiler found every call site.
|
||||
- `internal/web/recovery_handlers.go` — the new case and its message; `recoveryRetainedTrusted`.
|
||||
- `internal/web/server.go` — the test seam field.
|
||||
- `scripts/retrieval_promise_gate.py` — see below.
|
||||
---
|
||||
|
||||
## The message, and what it deliberately will not say
|
||||
## 2. Files created / modified
|
||||
|
||||
> „A kódod helyes, de egy korábbi csomagot nyit meg, nem azt, amit most őrzünk ehhez a géphez. A géped
|
||||
> időközben új mentési kulcsot kapott. A korábbi csomagot (dátum) nem töröltük, megőrizzük — a mostani
|
||||
> mentéseidet ez nem érinti, azokkal semmi nem történt. A régebbi előzményed visszanyitásához a Felhom
|
||||
> ügyfélszolgálatának segítsége kell: írj nekik, és add meg, hogy a régi mentéseidhez szeretnél
|
||||
> hozzáférni. A kódodat tedd el, szükség lesz rá."
|
||||
**felhom-controller**
|
||||
- `controller/internal/agentapi/diskverdict.go` — modified (14-row ladder, `DiskPrior`, `UncorrectableSectors`, `TemperatureFailC`)
|
||||
- `controller/internal/agentapi/diskverdict_test.go` — modified
|
||||
- `controller/internal/agentapi/diskverdict_ladder_test.go` — **created**
|
||||
- `controller/internal/notify/notifier.go` — modified (severity, `DiskAlert`, `DiskAlertKind`, `Severity()`, 5 message shapes)
|
||||
- `controller/internal/notify/disk_health_test.go` — rewritten
|
||||
- `controller/internal/web/disk_health_state.go` — **created** (persistence + decision)
|
||||
- `controller/internal/web/disk_health.go` — modified
|
||||
- `controller/internal/web/disk_health_test.go` — rewritten
|
||||
- `controller/internal/web/server.go` — modified (seam signature)
|
||||
- `controller/cmd/controller/main.go` — modified (6h → 1h)
|
||||
- *(v0.216.0)* `controller/internal/web/disk_health.go` + `disk_health_test.go` — the R-335 dedup fix and its test
|
||||
- `CHANGELOG.md`, `CONTEXT.md`, `REUSE.md`, `controller/README.md`, `REPORT.md`
|
||||
|
||||
Bytes verified as hex: `á c3 a1 · é c3 a9 · í c3 ad · ó c3 b3 · ö c3 b6 · ú c3 ba · ü c3 bc ·
|
||||
ő c5 91 · — e2 80 94`. No emoji. 443 chars / 487 bytes.
|
||||
**felhom.eu** (documentation only)
|
||||
- `documentation/audits/DIAG-smart-passed-trap-2026-08-14.md` — **created**
|
||||
- `documentation/audits/fixtures/smart-ST3000VX010-failing-2026-08-14.json` — **created** (raw `smartctl -a -j`, verbatim)
|
||||
- `documentation/audits/fixtures/smartd-history-sdg-2026-08-14.txt` — **created** (406 `smartd` journal lines)
|
||||
- `documentation/architecture/00-capability-map.md`, `documentation/backlog/ROADMAP.md`, `documentation/backlog/OPEN-ITEMS.md` — modified
|
||||
|
||||
It states the code is correct, names the date, says the package is kept, and denies the assumption it
|
||||
otherwise creates — that the **current** backups are affected. It **promises no restore**: there is no
|
||||
in-product route to a set-aside store (R-312), and a retained package may itself predate the
|
||||
repository-password field. A conditional promise that turns out false on this screen is worse than
|
||||
saying less — R-202, on the highest-stakes copy in the product. It routes to support, which **can** do
|
||||
it: yesterday's drill did exactly that by hand.
|
||||
---
|
||||
|
||||
**An older agent keeps the hedged sentence.** Unknown → claims less → heals itself on update.
|
||||
## 3. Commits pushed to `main`
|
||||
|
||||
## The claim guard grew a surface, and immediately convicted something
|
||||
| Repo | Hash | What |
|
||||
|------|------|------|
|
||||
| felhom.eu | `848de81` | Part 0 — fixtures + findings doc |
|
||||
| felhom-controller | `bb50e12` | Parts 1–3 — ladder, severity, persisted state + tests |
|
||||
| felhom-controller | `c24f192` | Group L strengthened to two post-restart checks |
|
||||
| felhom-controller | `34d83f5` | Part 4 — cadence 6h → 1h (measured) |
|
||||
| felhom-controller | `8144a70` | Part 5 — CHANGELOG / CONTEXT / README / REUSE |
|
||||
| felhom.eu | `767960b` | Part 5 — capability map, ROADMAP, register rows R-328…R-334 |
|
||||
| felhom-controller | `90f2545` | **v0.216.0** — R-335, one physical disk evaluated once per run |
|
||||
| felhom.eu | `fa4748d` | R-335 register row |
|
||||
|
||||
`retrieval_promise_gate.py` scanned `internal/web/templates` **only** — while every recovery message
|
||||
is a Go string in a handler. The highest-stakes copy in the product had never been scanned. It now
|
||||
also scans `recovery_handlers.go` (Go comments stripped, as template comments are), and `visszanyit`
|
||||
joins the stems because the new message uses a fourth verb for the same claim — the gate's own history
|
||||
is what happens when it chases words instead of claims.
|
||||
---
|
||||
|
||||
**On its first run it found a PRE-EXISTING unregistered claim** (`RecoverRefused`'s *"reopening would
|
||||
overwrite it"*), now registered as an explanation rather than a promise. 8 → 10 registered claims.
|
||||
## 4. Per-test results — all twelve groups
|
||||
|
||||
## Tests and red-proofs
|
||||
| Group | Scenario | Test | Result |
|
||||
|-------|----------|------|--------|
|
||||
| A | real drive, 2nd observation | `TestDiskCheck_RealDrive_HibaAndOneCriticalEvent` + `TestLadder_RealDrive_ReachesHiba` | **PASS** |
|
||||
| B | the transient that cleared | `TestDiskCheck_FirstSightingIsWarnOnly` | **PASS** |
|
||||
| C | sustained → Hiba | `TestDiskLadder_SustainDrivesTheEscalation` + `TestLadder_SustainIsWhatFires` | **PASS** |
|
||||
| D | recovered, silent | `TestDiskCheck_RecoveryIsSilentAndClearsState` | **PASS** |
|
||||
| E | flap damping | `TestDiskCheck_FlapDamping` | **PASS** |
|
||||
| F | escalation beats damping | `TestDiskCheck_EscalationBeatsDamping` | **PASS** |
|
||||
| G | still getting worse | `TestDiskCheck_RealertWhenStillWorsening` | **PASS** |
|
||||
| H | below both bars | `TestDiskCheck_NoRealertBelowBothBars` | **PASS** |
|
||||
| I | heat | `TestLadder_Temperature` + `TestDiskCheck_TemperatureShape` | **PASS** |
|
||||
| J | no data never alarms | `TestDiskCheck_UnknownNeverAlarmsNorErasesPrior` + `TestLadder_UnknownNeverAlarms` | **PASS** |
|
||||
| K | severity routes | `TestNotifyDiskHealthDegraded_SeverityRoutes` | **PASS** |
|
||||
| L | state survives restart (seam) | `TestDiskCheck_StateSurvivesRestart_ProductionPath` | **PASS** |
|
||||
|
||||
Six handler tests asserting **which sentence the customer sees**. Full suite green; all 11 controller
|
||||
gates OK. Red-proofs, each mutation asserted to have applied:
|
||||
Supporting: `TestLadder_CountBackstopBoundary`, `TestLadder_ZeroPriorIsFailSafe`,
|
||||
`TestUncorrectableSectors`, `TestDegradedAttributes_NamesFailCounters`,
|
||||
`TestNotifyDiskHealthDegraded_{WarnShape,FailShapes,CopyDiscipline}`,
|
||||
`TestDiskAlertDecision_Table`, `TestDiskState_CorruptFileFallsBackToNoPrior`,
|
||||
`TestDiskCheck_DisappearedDiskIsForgotten`, `TestDiskCheck_UnreachableAgentIsInert` — all PASS.
|
||||
|
||||
| Mutation | Result |
|
||||
|---|---|
|
||||
| delete the `RecoveryCodeOpensRetained` case | FAILS — customer gets the **neutral** message (R-224's safe default catches it; the lie itself lives in the agent) |
|
||||
| make 422 unconditional (ignore the gate) | FAILS — an agent that never looked is read as having looked |
|
||||
| route 400 to the new class | FAILS — a mistype is congratulated |
|
||||
---
|
||||
|
||||
## Deployed
|
||||
## 5. Red-proof outcomes — all twelve, individually
|
||||
|
||||
`gitea.dooplex.hu/admin/felhom-controller:0.214.0` built, pushed, deployed to guest 9201 —
|
||||
`Up … (healthy)`.
|
||||
Each mutation was applied by script, **asserted present in the source before the run** (the harness
|
||||
aborts with `MUTATION-NOT-APPLIED` if the target text is absent), the named test run, and the file
|
||||
reverted with `git checkout --`. The tree was confirmed clean after the sweep.
|
||||
|
||||
**Live, on hardware:** the agent returns **HTTP 422** for the old code (`opens_retained: true`,
|
||||
`superseded_at: 2026-08-12T15:18:55Z`) and **400** for a wrong one. The hub serves the retained
|
||||
packages (200, `count=2`, `unopenable_count=1`).
|
||||
| # | Mutation applied | Target test | Outcome |
|
||||
|---|------------------|-------------|---------|
|
||||
| A | remove truth-table row 6 (the sustain rule) | `TestDiskCheck_RealDrive_HibaAndOneCriticalEvent` | **RED-PROOF PASSED — FINDING, see below** |
|
||||
| B | make row 9 return `Fail` | `TestDiskCheck_FirstSightingIsWarnOnly` | failed as required |
|
||||
| C | pass a zero `DiskPrior` in `RunDiskHealthCheck` | `TestDiskLadder_SustainDrivesTheEscalation` | failed as required |
|
||||
| D | let Rendben fall through the silence guard | `TestDiskCheck_RecoveryIsSilentAndClearsState` | failed as required |
|
||||
| E | compare against last **observed** verdict, not last **alerted** | `TestDiskCheck_FlapDamping` | failed as required |
|
||||
| F | let damping cover escalations | `TestDiskCheck_EscalationBeatsDamping` | failed as required |
|
||||
| G | remove the re-alert branch | `TestDiskCheck_RealertWhenStillWorsening` | failed as required |
|
||||
| H | make cooldown/doubling an **OR** instead of an AND | `TestDiskCheck_NoRealertBelowBothBars` | failed as required |
|
||||
| I | remove truth-table rows 3 **and** 13 | `TestLadder_Temperature`, `TestDiskCheck_TemperatureShape` | failed as required |
|
||||
| J | let UNKNOWN delete the prior record | `TestDiskCheck_UnknownNeverAlarmsNorErasesPrior` | failed as required |
|
||||
| K | restore `severity := "warn"` | `TestNotifyDiskHealthDegraded_SeverityRoutes` | failed as required |
|
||||
| L | skip loading the persisted state | `TestDiskCheck_StateSurvivesRestart_ProductionPath` | failed as required |
|
||||
|
||||
**What was NOT walked:** the customer's rendered sentence. `recoveryUnlockHandler` redirects to
|
||||
`/backups/remote` when `!recoveryOffer()`, and this box holds its own repository password again, so it
|
||||
is correctly not in the offered state. Forcing it would mean removing that password to fake a rebuilt
|
||||
box — destabilising a healthy machine to render a sentence pinned by six handler tests whose upstream
|
||||
422 is proven live. **Method: endpoint-level for the agent/hub, handler-level for the message.**
|
||||
### A thirteenth red-proof, added after the deploy (R-335)
|
||||
|
||||
**Correction — R-308 is WITHDRAWN.** I previously reported the stored dashboard password as stale. It
|
||||
is not: the `~/.config/credentials` values are wrapped in SINGLE quotes and I stripped only double
|
||||
ones, so I was sending a literal `'`. Unquoted correctly it logs in first try (HTTP 302 + session
|
||||
cookie). The same bug made this session's first live R-311 test read as a failure for twenty minutes.
|
||||
| # | Mutation applied | Target test | Outcome |
|
||||
|---|------------------|-------------|---------|
|
||||
| M | delete the `if seen[key] { continue }` dedup guard | `TestDiskCheck_SameDiskTwiceIsEvaluatedOnce` | failed as required |
|
||||
|
||||
Observed failure: `first sighting of an aliased disk must be silent, got 1: [{Label:felhom-backup … Kind:2 Sectors:8}]`
|
||||
— i.e. `Kind:2` is `DiskAlertFailSectors`, a **Hiba on a first sighting of 8 sectors**. Reverted.
|
||||
|
||||
### FINDING — red-proof A passed, and it is the spec's mutation that is at fault, not the code
|
||||
|
||||
The task specified group A's red-proof as *"remove truth-table row 6 → verdict is Warn"*. **That
|
||||
mutation cannot fail a test built on the real drive's values**: the real drive carries **352**
|
||||
unreadable sectors, so with row 6 deleted it still reaches Hiba via **row 8** (count ≥ 64). The test
|
||||
correctly stayed green, so the mutation proves nothing about row 6.
|
||||
|
||||
This was anticipated while writing the tests and is documented in the test's own comment rather than
|
||||
discovered afterwards. **Row 6 is genuinely pinned**, by two tests that hold the counters at **8**
|
||||
(far below the 64 backstop) and vary *only* the prior:
|
||||
|
||||
- `TestLadder_SustainIsWhatFires` (agentapi) — same `SmartSummary`, `DiskPrior{}` → Warn,
|
||||
`DiskPrior{SawUncorrectable:true}` → Fail.
|
||||
- `TestDiskLadder_SustainDrivesTheEscalation` (web) — the event-level twin.
|
||||
|
||||
Both were run under the row-6-deleted mutation and **both failed**, as recorded:
|
||||
`SAME 8 sectors, now sustained = 2 (Figyelmeztetés), want Fail/Hiba` and
|
||||
`severity = "warning", want critical` / `chip = "Figyelmeztetés", want Hiba`. So the invariant is
|
||||
covered; only the spec's chosen mutation was invalid.
|
||||
|
||||
### A second finding, from building red-proof L
|
||||
|
||||
The first version of the Group L seam test ran **one** check after the restart and **passed under the
|
||||
mutation** — because a controller that has forgotten its state is also silent on its first check. The
|
||||
test was strengthened to run **two** checks (commit `c24f192`), after which the mutation fails. This
|
||||
is the exact shape §10 warns about, caught by running the red-proof rather than assuming it.
|
||||
|
||||
---
|
||||
|
||||
## 6. Test count and suite state
|
||||
|
||||
- **Before:** 1391 test functions (at `3e3ee94`) · **After:** 1414 (+23)
|
||||
- `go build ./... && go vet ./... && go test ./...` — **all green**, no failures, no skips introduced.
|
||||
- `python3 controller/scripts/controller_gates.py` — **all 11 gates OK.**
|
||||
|
||||
---
|
||||
|
||||
## 7. Cadence measurement (Part 4)
|
||||
|
||||
Measured on **demo-hp** (Tier 0, disposable), through `fetchDisks`' real path — the agent local API
|
||||
`GET /disks`, not the 60 s card cache. Ten consecutive calls, all **HTTP 200**:
|
||||
|
||||
```
|
||||
0.840558 0.817000 0.815783 0.831992 0.809066
|
||||
0.832899 0.824788 0.804886 0.812539 0.833547 (seconds)
|
||||
```
|
||||
|
||||
| min | median | max | disk count |
|
||||
|-----|--------|-----|------------|
|
||||
| **0.804886 s** | **0.820894 s** | **0.840558 s** | **3 physical rows** across 2 devices (SanDisk X600 M.2 SATA SSD; Toshiba KXG50PNV1T02 NVMe, counted twice as `c11-scratch` + `felhom-backup`) |
|
||||
|
||||
**Branch taken: median < 5 s → `6*time.Hour` → `1*time.Hour`.** The median is ~6× under the bar. The
|
||||
detection argument is the real one: the observed benign excursion lasted about **one hour**, so a
|
||||
6-hourly sampler can land either side of it and then catch the terminal run half a day late.
|
||||
|
||||
No spin-up signature appeared in the timings (uniform ~0.82 s; demo-hp is all-flash), so the
|
||||
measurement did not suggest the spun-down-drive concern. That question is recorded as an Observation
|
||||
below and deliberately **not acted on**.
|
||||
|
||||
---
|
||||
|
||||
## 8. Live validation
|
||||
|
||||
Deployed to **demo-hp guest 9201** via the bootstrap path (`docker pull` →
|
||||
`/etc/felhom-controller-image` → `systemctl restart felhom-controller-bootstrap.service`).
|
||||
|
||||
```
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.215.0 Up 19 seconds (healthy) # 06:23Z
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.216.0 Up 6 seconds (healthy) # after the R-335 fix
|
||||
```
|
||||
|
||||
### Leg 1 — no over-correction (the load-bearing check)
|
||||
|
||||
Method: **endpoint-level** — authenticated `GET /dashboard` on the real controller
|
||||
(`https://felhom.enkisfelhom.hu/dashboard`, HTTP 200, 44 036 bytes), i.e. the exact endpoint the UI
|
||||
invokes; only rendering is skipped. No browser is available on DooPlex.
|
||||
|
||||
Card contents, parsed from the response body:
|
||||
|
||||
| Disk | Chip | Class | Temp |
|
||||
|------|------|-------|------|
|
||||
| KXG50PNV1T02 NVMe TOSHIBA 1024GB | **Rendben** | `state-text-run` | 53 °C |
|
||||
| KXG50PNV1T02 NVMe TOSHIBA 1024GB | **Rendben** | `state-text-run` | 53 °C |
|
||||
| SanDisk X600 M.2 2280 SATA 128GB | **Rendben** | `state-text-run` | 44 °C |
|
||||
|
||||
`Figyelmeztetés` = 0, `Hiba` = 0, `Nincs adat` = 0, `state-text-warn` = 0, `state-text-crit` = 0.
|
||||
**No healthy disk was over-corrected.**
|
||||
|
||||
**Positive observable, at deploy:**
|
||||
`[INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s)` — the new cadence is
|
||||
in force, not merely compiled.
|
||||
|
||||
**Positive observable, per cycle** — two full hourly cycles observed after the deploy, from the
|
||||
container log:
|
||||
|
||||
```
|
||||
2026/08/14 06:23:13 [INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s)
|
||||
2026/08/14 07:23:13 [INFO] [scheduler] Running job: disk-health-check
|
||||
2026/08/14 07:23:14 [INFO] [web] disk-health check complete: 3 disk(s) evaluated, 0 alert(s)
|
||||
2026/08/14 07:23:14 [INFO] [scheduler] Job disk-health-check completed (took 849ms)
|
||||
2026/08/14 08:23:13 [INFO] [scheduler] Running job: disk-health-check
|
||||
2026/08/14 08:23:14 [INFO] [web] disk-health check complete: 3 disk(s) evaluated, 0 alert(s)
|
||||
2026/08/14 08:23:14 [INFO] [scheduler] Job disk-health-check completed (took 843ms)
|
||||
```
|
||||
|
||||
`grep -c disk_health_degraded` over the whole container log: **0**. Both cycles ran (849 ms / 843 ms,
|
||||
matching the §7 measurement), evaluated every disk, and emitted nothing. **Zero alerts from a check
|
||||
that demonstrably ran** — not silence.
|
||||
|
||||
Persisted state written by the first cycle (`/opt/docker/felhom-controller/data/disk-health-state.json`,
|
||||
428 bytes, on the `felhom-controller-data` docker volume, so it survives container recreation):
|
||||
|
||||
```json
|
||||
{"version": 1, "disks": {
|
||||
"path:/var/lib/vz": {"verdict": 1, "saw_uncorrectable": false, ...},
|
||||
"uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae": {"verdict": 1, "saw_uncorrectable": false, ...}}}
|
||||
```
|
||||
|
||||
`verdict: 1` is `DiskVerdictOK` for both, `saw_uncorrectable: false`, never alerted.
|
||||
|
||||
**Reading those two artefacts against each other is what exposed R-335** — see §14.
|
||||
|
||||
### Leg 2 — the severity fix arrives (the point of the task)
|
||||
|
||||
Two synthetic `disk_health_degraded` events pushed for customer `demo-hp` **through the real hub
|
||||
event endpoint** (`POST https://hub.felhom.eu/api/v1/event`), from the guest's own controller using
|
||||
its own hub credentials — the genuine controller→hub path, not a hand-crafted operator call. Both
|
||||
returned `HTTP 200 {"ok":true}`. The hub DB was read with its `-wal` and `-shm` copied alongside
|
||||
`hub.db` (a `hub.db`-only read is stale).
|
||||
|
||||
**As STORED by the hub (`events`):**
|
||||
|
||||
| id | severity pushed | severity STORED |
|
||||
|----|-----------------|-----------------|
|
||||
| 2964 | `warning` | **`warning`** |
|
||||
| 2965 | `warn` | **`info`** ← coerced |
|
||||
|
||||
**`notification_log` rows for those two events:**
|
||||
|
||||
| id | event_type | severity | channel | status | error |
|
||||
|----|-----------|----------|---------|--------|-------|
|
||||
| 689 | `disk_health_degraded` | `warning` | `operator` | **`sent`** | *(none)* |
|
||||
| — | *(the `"warn"` push)* | — | — | **NO ROW EXISTS** | — |
|
||||
|
||||
**That pair is the proof.** The identical event, differing only in one word of the severity string,
|
||||
is the difference between *delivered to the operator* and *stored as an informational notice and
|
||||
delivered to nobody*. This is the first time this leg has been observed end to end.
|
||||
|
||||
Only the operator leg fired because **demo-hp has no `customer_notifications` row at all** (no
|
||||
customer email, no `enabled_events`), so no customer row was possible for either push — verified
|
||||
directly, not assumed. **One real email was sent to the operator**, as the task anticipated.
|
||||
|
||||
---
|
||||
|
||||
## 9. NOT yet live-validated — stated explicitly
|
||||
|
||||
**The Fail-from-counters path has never fired on real hardware.** Everything in §4/§5 exercises it
|
||||
against the committed fixture's values in unit tests only. The live legs above prove the *negative*
|
||||
(no false alert on three healthy disks) and the *severity wire* (end to end, through the hub) — they
|
||||
do **not** prove a live disk reaching Hiba. The fixture tests must not be read as a live proof.
|
||||
|
||||
Tracked as **R-332 (WATCHING)**. Closing condition: a live disk reaching Hiba from counters, or a
|
||||
deliberate injection through the real pipeline (agent `/disks` → controller check → hub event) — not
|
||||
a hand-set verdict. The drive that produced the fixture lives in DooPlex, which is Tier 2 and never a
|
||||
drill target; the demo boxes are all-flash and healthy.
|
||||
|
||||
---
|
||||
|
||||
## 10. Teardown
|
||||
|
||||
**This run provisioned nothing** — no VM, no guest, no hub customer record, no storage. Nothing was
|
||||
formatted, mounted, unmounted, repaired or written on any monitored disk; the only write is the
|
||||
controller's own `disk-health-state.json` inside its data volume.
|
||||
|
||||
Disposition of what the run did create:
|
||||
|
||||
- **Two synthetic hub events (`events` id 2964, 2965) and one `notification_log` row (id 689)** on the
|
||||
live hub. **Left in place deliberately.** Both messages are self-labelling
|
||||
(`"R-328 severity probe (…) - synthetic, no real disk fault"`), and deleting rows from the
|
||||
production hub DB is a riskier act than leaving two clearly-marked probe rows. Named here so they
|
||||
are not mistaken later for a real disk fault on demo-hp.
|
||||
- **One real operator email** resulting from row 689.
|
||||
- A local copy of `hub.db`/`-wal`/`-shm` in the session scratchpad only (not committed, not exported).
|
||||
|
||||
---
|
||||
|
||||
## 11. Register rows
|
||||
|
||||
| Row | State | Owner |
|
||||
|-----|-------|-------|
|
||||
| **R-328** — the severity drop: `"warn"` coerced to `info`, emailed to nobody | **CLOSED** (controller v0.215.0), proven live side by side | CC |
|
||||
| **R-329** — `app_start_failed` carries the identical defect | READY — **not fixed here**; needs a decision on whether it should notify at all | Viktor |
|
||||
| **R-330** — Phase 2: collect SMART attrs 187/199/188 + persist samples | READY — a declared wire change, hub models it in the same session under G-1 | CC |
|
||||
| **R-331** — Phase 3: growth-rate detection; revisit the static 64 | READY, blocked on R-330 | CC |
|
||||
| **R-332** — the Fail path has never fired on real hardware | **WATCHING** | CC |
|
||||
| **R-333** — NVMe temperature bands; agent `smartctl` has no `-n standby` | READY (S each) | Viktor decides (a); CC does (b) |
|
||||
| **R-334** — released with no golden carrying it (gate waiver) | READY — now applies to **v0.216.0** | CC bakes; **Viktor vouches** |
|
||||
| **R-335** — one physical disk walked twice per run, sustaining against itself | **CLOSED** (controller v0.216.0) | CC |
|
||||
|
||||
`smartd`-on-DooPlex-alerts-nobody is recorded in `DIAG-smart-passed-trap-2026-08-14.md` §8 as the
|
||||
same shape one layer out.
|
||||
|
||||
---
|
||||
|
||||
## 12. Observations — noticed, NOT acted on
|
||||
|
||||
1. **`app_start_failed` has the identical severity defect** (`notifier.go` ~L546, `"warn"`). Left
|
||||
untouched per scope. It needs a prior decision — should a stopped app email the customer at all? —
|
||||
because flipping the string alone converts a silent event into a mail flood on a crash-looping box.
|
||||
**R-329.**
|
||||
|
||||
2. **The 55/60 °C bands are spinning-disk bands being applied to NVMe, and this is close to biting.**
|
||||
Adopted unchanged from the operator's Prometheus config by explicit decision — but demo-hp's
|
||||
**healthy** Toshiba NVMe idles at **53 °C**, i.e. **2 °C below Figyelmeztetés and 7 °C below Hiba**,
|
||||
and NVMe routinely passes 60 °C under sustained write with no fault. As shipped, a healthy customer
|
||||
NVMe under load can be reported as **Hiba** — the single worst outcome this feature can produce, and
|
||||
the one leg 1 exists to guard. Not changed here because the threshold is a stated, settled operator
|
||||
decision; flagged rather than overridden. **R-333(a) — recommend splitting the bands by device
|
||||
class, or dropping them for NVMe and relying on `critical_warning`.**
|
||||
|
||||
3. **The agent runs bare `smartctl -a -j` with no `-n standby`**
|
||||
(`felhom-agent/internal/storage/hostops.go:368`), so every poll wakes a spun-down drive, and 6h → 1h
|
||||
multiplies that by six. Recorded, not acted on, per the task's instruction. demo-hp is all-flash so
|
||||
the measurement could not reveal it. Mitigating datum from the fixture: the failing drive logged
|
||||
only **3375 load cycles in 60505 power-on hours** (~one per 18 h), so this duty cycle barely spins
|
||||
down at all. **R-333(b).**
|
||||
|
||||
4. **`source ~/.config/credentials` prints two recovery codes to the terminal.** The file contains
|
||||
hyphenated keys (`R_DEMO-FELHOM`, `R_DEMO-HP`) that bash cannot assign, so sourcing it emits
|
||||
`command not found` errors **containing the secret values**. Anything that sources that file leaks
|
||||
them into logs, scrollback and transcripts. Not a code defect and out of scope; worth quoting
|
||||
values from it by other means, or renaming the keys.
|
||||
|
||||
5. **`golden_currency_gate.py` has no waiver parser.** Its own failure text says *"record a waiver in
|
||||
`OPEN-ITEMS.md` — never a bypass"*, but nothing reads such a waiver, so the only way past it is the
|
||||
bypass it warns against. See §13.
|
||||
|
||||
---
|
||||
|
||||
## 13. Deviations, stated plainly
|
||||
|
||||
- **`git push --no-verify` was used once**, on the `felhom.eu` docs push (`767960b`), and only there.
|
||||
Cause: `golden_currency_gate.py` correctly convicts the fact that controller **v0.215.0 is released
|
||||
and no golden carries it** (newest bake 0.214.0), so a *newly installed* machine would receive
|
||||
0.214.0 — without the severity fix. A golden bake was out of the task's scope, and its second half
|
||||
(vouching in the hub's day-0 artifact manifest) is operator-password-gated, so CC cannot complete it;
|
||||
a baked-but-unvouched golden is worse than none. Recorded as **R-334** with the bake+vouch owners
|
||||
named. CI re-runs the same entry point and will mail the operator. The running fleet is unaffected.
|
||||
- **One pre-existing test changed meaning by design:** `TestDiskVerdictFor`'s
|
||||
`critical_warning>0 → warn` case is now `→ fail` (truth-table row 4 — NVMe's own critical flag is a
|
||||
device declaration, not a drifting counter). `TestDiskHealthCheck_DegradationOnce` and its siblings
|
||||
were rewritten into the scenario groups because they encoded the pre-v0.215.0 single-alert behaviour
|
||||
the task deliberately replaces (Scenario C).
|
||||
|
||||
---
|
||||
|
||||
## 14. R-335 — a defect in v0.215.0, found live, fixed as v0.216.0
|
||||
|
||||
**How it was found.** Not by a test and not by review: by reading the release's own **positive
|
||||
observable** against the release's own **persisted artefact**. The hourly check logged *"3 disk(s)
|
||||
evaluated"*; `disk-health-state.json` held **two** records. Two artefacts that should have agreed did
|
||||
not.
|
||||
|
||||
**Cause.** demo-hp's `c11-scratch` and `felhom-backup` are the same physical NVMe (`/dev/nvme0n1`) and
|
||||
resolve to the same `diskKey`, so one disk was walked twice in a single run.
|
||||
|
||||
**Why it mattered.** `RunDiskHealthCheck` writes a disk's new record before the next entry reads it, so
|
||||
the **second** copy of an aliased disk consumed the **first** copy's write as its prior. The disk
|
||||
therefore **sustained against itself and reached Hiba on a first sighting** — defeating truth-table
|
||||
row 6, the single rule separating a one-hour benign excursion from a false critical alert — and would
|
||||
have emitted **two identical events** for one drive.
|
||||
|
||||
**Severity in practice: latent, not active.** Nothing fired on demo-hp because all three entries are
|
||||
healthy with zero counters. But any aliased disk developing one pending sector would have gone
|
||||
straight to Hiba, which is precisely the outcome §8 leg 1 exists to prevent. Aliasing is not exotic —
|
||||
it is the *normal* shape whenever a box has two PVE storage entries on one physical device.
|
||||
|
||||
**Fix (v0.216.0, `90f2545`).** Each `diskKey` is evaluated once per run. Both entries stay marked
|
||||
`seen`, so neither is mistaken for a disappeared disk, and the card still renders **both** storage
|
||||
rows — the dedup is about state and alerts, not display. Pinned by
|
||||
`TestDiskCheck_SameDiskTwiceIsEvaluatedOnce`, red-proof run and reverted (§5).
|
||||
|
||||
**Deployed:** `gitea.dooplex.hu/admin/felhom-controller:0.216.0 Up 6 seconds (healthy)`.
|
||||
|
||||
**Honest note on the confirming cycle.** The two clean cycles in §8 ran on **v0.215.0**. A confirming
|
||||
cycle on v0.216.0 — where the evaluated count must now read **2** and match the record count — was
|
||||
still pending when this report was written; the check is hourly and the container had just restarted.
|
||||
Nothing about the fix depends on that observation (it is pinned by a red-proofed unit test), but the
|
||||
live count has not yet been re-read, and this report does not claim otherwise.
|
||||
|
||||
**Process note, recorded because it nearly cost the fix.** The red-proof harness reverts with
|
||||
`git checkout --`, which restores to `HEAD`. Running a red-proof against an **uncommitted** fix
|
||||
therefore *deletes the fix* along with the mutation — which happened here and was caught only by
|
||||
re-grepping the source afterwards. Commit the fix before red-proofing it, or snapshot outside git.
|
||||
|
||||
Reference in New Issue
Block a user