felhom.eu/REPORT.md

# REPORT — R-331: the operator Backup card said every customer had no backups

**Hub v0.109.0 (with controller v0.225.0) · 2026-08-30**

---

## 1. What was wrong

The hub customer page's **Backup** card read, for **every customer, indefinitely**:

```
Enabled  Yes        Snapshots  0
Repo Size  0 MB     Integrity  Unknown
```

Measured on `demo-hp` 2026-08-30, at which moment the truth was:

| source | value |
|---|---|
| the box's own `settings.json` | `snapshot_count: 67, repo_size_bytes: 140829678, stats_known: true` |
| that night's controller log | `[offbox] backup OK: 8 app(s) backed up, 67 snapshot(s), 2m14s` |
| **this hub's own Offsite page** | `0.1 GB` used of a `50 GB` quota — read from the same stored report |

**A card that reads "no backups" over a working backup is worse than no card.** It is the R-88
direction of failure — degrading to *no backup* rather than to *unknown* — on the one screen an
operator consults to answer "is this customer protected?".

## 2. Root cause

The card rendered the report's **`backup`** object. Its `snapshot_count`, `repo_size_mb` and
`integrity_ok` fields have had **no producer** since disk-tier restic moved to the host agent (slice
8C) — the controller's `buildBackupReport` leaves them zero *deliberately* and says so in a comment.
The zeros were correct values for dead fields, rendered as if live.

**The data was never missing.** The live numbers ride in the report's **`offsite`** object, which this
package **already** reads for the Offsite page (`offsiteUsageBytes`) and which `monitor.OffsiteChecker`
**already** drives fill and staleness alarms from. That the Offsite page rendered demo-hp's real usage
from the same stored report, at the same moment the Backup card said `0 MB`, is the proof the bytes
were arriving. This is a **render fix over an existing feed**, not a new pipeline.

## 3. Why it was not a one-line template swap

`snapshot_count: 0` means two opposite things — *this repository holds nothing* and *nobody has ever
measured this repository*. **R-225 measured that confusion one layer down**: a rebuilt box rendered
„Tarolo meret · 0 pillanatkep" over a store that really held snapshot `f3d9cd67`, and the controller's
`StatsKnown` fixed it there. It was never on the wire, so rendering the count without it would have
**moved R-225 up to the hub instead of fixing anything**. Controller v0.225.0 now forwards
`stats_known`.

## 4. What changed

`hub/internal/web/backup_card.go` builds a typed `backupCardView` — resolved in Go, because the card's
whole subject is a distinction a template `{{if}}` chain over `map[string]interface{}` float64s cannot
keep:

| report state | card shows |
|---|---|
| no `offsite` object at all | "No off-site data reported" — **and says explicitly this is not the same as "no backups"** |
| `enabled:false` + declared `state` | the blocker by name (`needs_credential`) — a different operator action from "not enabled" |
| enabled, `stats_known:false` | **—**, plus "never been measured". Never `0` |
| enabled, `stats_known:true` | the real count and size, **including a real `0`** — measured empty is knowledge |

**A pre-v0.225.0 controller sends no `stats_known`, which unmarshals to false → "unknown".** That is
the fail-safe direction: upgrading the hub ahead of the fleet must not tell the operator that every
un-upgraded customer has zero backups. Pinned by a test.

**The Integrity row is deleted, not re-sourced.** Nothing produces it: the controller runs no integrity
check, and `NotifyIntegrityOK` / `NotifyIntegrityFailed` exist and are **called from nowhere**. A row
that can only ever read "Unknown" is not information, and one that could read "OK" from an unwritten
field would be a lie.

`fmtBytesAuto` is new rather than reusing `fmtBytesGB`: that one is fixed at GB because it renders
against GB quotas, and it turns demo-hp's real 140 829 678 bytes into `0.1 GB` — which on a card whose
entire defect was under-reporting a real backup reads as "nearly nothing".

## 5. Tests and the red-proof

`r331_backup_card_test.go` asserts the **rendered page**, using demo-hp's real reported values, so a
regression fails against the same numbers the defect was measured against. **The defect lived in the
template's choice of source object, so a test one layer below it would have been green against the
shipped bug** — which is why these drive `handleCustomerUnified` and grep the HTML.

**RED-PROOF (run 2026-08-30):** restoring the pre-fix card markup fails all four tests —
`the rendered Backup card does not contain demo-hp's real snapshot count (67)`,
`the card does not carry the real repository size (134.3 MB ...)`,
`the card still shows an Integrity row`, plus every branch of the three-way ruling. Restored
immediately; `git diff` clean.

**Green gate:** `go build ./... && go vet ./... && go test ./...` in `hub/` — 18 packages, rc 0.

## 6. Deployment and live verification

Hub **0.109.0** built, pushed, manifest bumped, ArgoCD hard-refreshed and synced. Controller
**0.225.0** deployed to both demo boxes. Verified from the live objects, not from a rollout message
(an ArgoCD "rolled out" can name the old image):

```
argocd:  sync=Synced  rev=36f86300209e9ce914f2a97dac16ee9eeea249e2   (== HEAD)
deploy:  gitea.dooplex.hu/admin/felhom-hub:0.109.0
pod:     hub-6795879c4b-pf4lz  ...felhom-hub:0.109.0  Running
boxes:   ...felhom-controller:0.225.0  Up (healthy)   on demo-felhom AND demo-hp
```

**The card, fetched from the live hub** (endpoint-level: the exact URL the operator's browser
requests; the residual is client-side rendering only — there is no browser on DooPlex):

| | demo-hp | demo-felhom |
|---|---|---|
| Off-site snapshots | **67** | **10** |
| Repo size | **134.3 MB** | **132.5 KB** |
| Last successful run | 15h ago | 15h ago |
| Soft quota | 50 GB | 50 GB |
| Integrity row | **absent** (grep count 0) | **absent** (grep count 0) |

Both read `Snapshots 0 · Repo Size 0 MB · Integrity Unknown` before this change.

**Cross-checked against the source, not just against itself** — the numbers on the card are the
numbers on the boxes:

```
demo-hp      settings.json offbox: snapshot_count 67, repo_size_bytes 140829678, stats_known true
demo-felhom  settings.json offbox: snapshot_count 10, repo_size_bytes 135635,    stats_known true
                                   135635 / 1024 = 132.5 KB  → matches the rendered value
```

**One honest detail worth keeping:** demo-felhom's "Last DB dump" reads `—`. That is correct, not a
regression — its only app (`opengist`) has no database, so the box has never taken a DB dump.

**Not verified live: the "never measured" branch.** Both boxes report `stats_known: true`, so the
degradation path could not be exercised on real hardware without falsifying a box's state. It is
covered by `TestBackupCard_ThreeWayRuling` and `TestBackupCard_OldControllerDegradesToUnknownNotEmpty`
at render level, and this is stated rather than implied.

## 7. The push bypassed a gate, deliberately, and here is the declaration

**`git push --no-verify` was used for this change.** `repo_gates.py`'s `golden-currency` gate was
CONVICTED and it was RIGHT: controller **v0.224.0** and **v0.225.0** are released and the newest golden
bake carries **0.223.0**, so a machine installed right now receives neither fix.

**This is a BYPASS, not a waiver.** The gate offers a waiver only for a release that *deliberately needs
no golden*; these need one. **The operator was asked and ruled bypass-now-bake-later**, on the stated
ground that neither fix bites a day-0 box — R-330 is a nightly false alarm about apps a new box has not
installed yet, R-331 is a hub-side display over backups a new box has not taken yet — and both arrive by
self-update afterwards. That ground is recorded on R-242 precisely because it is the thing to re-check:
**it does not extend to a release that changes first-boot behaviour.**

**A golden carrying 0.225.0 is OWED** (`RUNBOOK-manual-build.md` §4.1; the vouch is a three-field change,
`MinAgent 0.129.0`). This is the **fourth** bypass of this gate, and the gap it names is now two releases
wide rather than one.

**The other failing gate was fixed, not bypassed.** `due-checks` was red on R-341's `+7 d` measurement,
five days overdue. It was **taken** during this session — see §8.

## 8. R-341's overdue check was taken, and its premise did not survive

Unrelated to R-331; it blocked the same push, so it was done rather than deferred. Evidence:
`documentation/audits/evidence-r341-plus7d-2026-08-30/step1-fd-and-sockets.txt`.

**Precondition passed**, which is what makes the reading interpretable: `proxmox-backup-proxy` still
`MainPID 551655`, `ps -o lstart=` still `2026-08-18 09:51:04`, `NRestarts=0` — the same proxy generation
as t0, so nothing restarted and re-based the count. (The anchor is `ps`, not `ActiveEnterTimestamp`,
which reads 03:54:54Z here — R-346's trap, avoided.)

**Result: fd = 17.** Not 17 more — seventeen total, exactly the documented baseline, against **405** at
the first check on 2026-08-20. The socket histogram holds **one LISTEN and nothing else**: ESTAB 0,
CLOSE-WAIT 0.

**The verdict is "unanswerable", not "the upgrade fixed it".** R-341 asks whether the PBS 4.2.5-1
upgrade changed the fd slope. Inside this interval **we removed the leak ourselves** (R-344, agent
0.130.0, now live on both boxes). A slope of ~0 measures our fix, not the upgrade — and reading it the
other way would credit a changelog that was read in advance and found to contain no such mechanism. The
perturbation pre-registered for this window was Phase C at ~3%; the actual perturbation was the removal
of the entire phenomenon. The question is now **moot**, and the row is closed as such.

**What it does establish, which is worth more than the original question:** twelve days after the R-344
fix, on the same proxy generation with no restart to hide behind, ep0 sits at baseline with zero
established connections. The 388-descriptor accumulation has not returned, and R-336's ~323-day runway
concern retires with it.

## 9. Not done, and why

- **No staleness verdict on the card.** `monitor.OffsiteChecker` already owns that and alarms on it. A
  second verdict over the same data is two things that can disagree — a shape this codebase has already
  paid for (`LastRun` vs `LastSuccess`, R-100).
- **The dead `backup` fields were not removed from the controller's wire format.** Removing them would
  stop historical reports already in this hub's store from parsing, for no gain — nothing renders them
  now, and a controller-side test fails if anything starts producing them.