felhom-controller/REPORT.md

REPORT — R-331: the operator Backup card said every customer had no backups

Controller v0.225.0 + hub v0.109.0 · 2026-08-30

This is Option B of a two-part request. Option A (the nightly false app_start_failed e-mails) shipped earlier today as controller v0.224.0 (R-330) and is recorded in this repo's CHANGELOG.md.

---

1. What was wrong

The hub customer page's Backup card read, for every customer, indefinitely:

Enabled  Yes        Snapshots  0
Repo Size  0 MB     Integrity  Unknown

Measured on demo-hp 2026-08-30, at which moment the truth was:

source	value
the box's own settings.json	snapshot_count: 67, repo_size_bytes: 140829678, stats_known: true
that night's controller log	[offbox] backup OK: 8 app(s) backed up, 67 snapshot(s), 2m14s
the hub's Offsite page	0.1 GB used of a 50 GB quota — read from the stored report

A card that reads "no backups" over a working backup is worse than no card. It is the R-88 direction of failure — degrading to no backup rather than to unknown — on the one screen an operator consults to answer "is this customer protected?".

2. Root cause

The card rendered the report's backup object. Its snapshot_count, repo_size_mb and integrity_ok fields have had no producer since disk-tier restic moved to the host agent (slice 8C) — buildBackupReport leaves them zero deliberately and says so in a comment. So the zeros were not a bug in the controller; they were correct values for dead fields, being rendered as if live.

The data was never missing. The live numbers ride in the report's offsite object (backup.OffboxReportStatus), which the hub already reads for the Offsite page (offsiteUsageBytes) and which monitor.OffsiteChecker already drives fill and staleness alarms from. That the Offsite page rendered demo-hp's real usage from the same stored report, at the same moment the Backup card said 0 MB, is the proof the bytes were arriving.

So the fix is a render change over an existing feed, not a new pipeline.

3. Why it was not a one-line template swap

snapshot_count: 0 means two opposite things — this repository holds nothing and nobody has ever measured this repository.

R-225 already measured that exact confusion one layer down: a rebuilt box rendered „Tarolo meret · 0 pillanatkep" over a store that really held snapshot f3d9cd67, and settings.OffboxTarget.StatsKnown is what fixed it for the controller's own UI. It was never put on the wire, so the hub was free to make the identical mistake one layer up — and rendering the count without consulting it would have moved R-225 to the hub instead of fixing anything.

4. What changed

Controller (v0.225.0) — one field, and the dead ones labelled.

• OffboxReportStatus.StatsKnown now forwards settings.OffboxTarget.StatsKnown. It is omitempty, so a controller below this version sends no key, a reader sees false, and false means "cannot answer", never "the answer is zero". Absence is ignorance, not emptiness.
• The four dead BackupReport fields (SnapshotCount, RepoSizeMB, LastIntegrityCheck, IntegrityOK) are kept on the wire so historical reports in the hub's store keep parsing, but now carry a warning naming R-331 and pointing at Offsite. TestBackupReport_DeadFieldsStayZero fails the moment a producer appears for any of them — the prompt to update the hub card in the same change rather than shipping a field nothing renders.

Hub (v0.109.0) — the card reads offsite, resolved in Go.

backup_card.go builds a typed view, because the card's whole subject is a distinction a template {{if}} chain over map[string]interface{} float64s cannot keep:

report state	card shows
no offsite object at all	"No off-site data reported" — and says explicitly this is not the same as "no backups"
enabled:false + declared state	the blocker by name (needs_credential) — a different operator action from "not enabled"
enabled, stats_known:false	—, plus "never been measured". Never 0
enabled, stats_known:true	the real count and size, including a real 0 — measured empty is knowledge

The Integrity row is deleted, not re-sourced. Nothing produces it: the controller runs no integrity check, and NotifyIntegrityOK / NotifyIntegrityFailed exist and are called from nowhere. A row that can only ever read "Unknown" is not information, and one that could read "OK" from an unwritten field would be a lie.

fmtBytesAuto is new rather than reusing the Offsite page's fmtBytesGB: that one is fixed at GB because it renders against GB quotas, and it turns demo-hp's real 140 829 678 bytes into 0.1 GB — which on a card whose entire defect was under-reporting a real backup reads as "nearly nothing".

5. Tests and red-proofs

The hub tests assert the rendered page, using demo-hp's real reported values, so a regression fails against the same numbers the defect was measured against. The defect lived in the template's choice of source object, so a test one layer below it would have been green against the shipped bug.

test	pins
TestBackupCard_RendersTheRealSnapshotCount (hub)	67 and 134.3 MB are on the page; no 0 MB; no Integrity row
TestBackupCard_ThreeWayRuling (hub)	unmeasured ≠ measured-empty ≠ no-object, all four branches
TestBackupCard_OldControllerDegradesToUnknownNotEmpty (hub)	upgrading the hub ahead of the fleet must not report every customer as empty
TestFmtBytesAuto_... (hub)	the real byte value renders as 134.3 MB, not 0.1 GB
TestOffboxReportStatus_CarriesStatsKnown (controller)	the field is on the wire
TestOffboxReportStatus_MeasuredEmptyIsNotUnmeasured (controller)	asserted on the JSON, not the struct — the hub sees bytes
TestOffboxReportStatus_AbsentStatsKnownParsesAsUnknown (controller)	the fail-safe direction
TestBackupReport_DeadFieldsStayZero (controller)	the dead fields stay dead

Red-proofs, each printing the pre-fix value:

1. Restore the pre-fix card markup → all four hub tests fail, reporting 67 and 134.3 MB absent from the rendered page and the Integrity row present.
2. Drop StatsKnown: t.StatsKnown from OffboxReportStatus() → a MEASURED empty repository reported stats_known=<nil>.

Both restored immediately; git diff clean afterwards.

Green gates: controller go build && go vet && go test ./... — 28 packages, rc 0. Hub, same command in hub/ — 18 packages, rc 0.

6. Deployment and live verification

Recorded in the hub's own REPORT.md; the two halves ship together (the hub renders "unknown" for any box still on 0.224.0, which is correct rather than wrong).

7. Not done, and why

• No staleness verdict on the card. monitor.OffsiteChecker already owns that and alarms on it. A second verdict on the same data is two things that can disagree, which this codebase has been bitten by before (LastRun vs LastSuccess, R-100).
• The dead BackupReport fields were not removed from the wire. Removing them would stop historical reports already in the hub's store from parsing, for no gain — nothing renders them now, and a test fails if anything starts producing them.