felhom.eu/documentation/audits/DIAG-backup-missed-2026-07-26.md

# DIAG — `expected_backup_missed` on demo-felhom / demo-hp / drill-r50 (2026-07-26)

**Class:** read-only diagnostic (R-80). Nothing was fixed, triggered, cleared or restarted.
**Author:** Claude Code, DooPlex. **Window investigated:** 2026-05-30 → 2026-07-26 09:55 UTC.

---

## 0. Headline

**The alarm is a FALSE ALARM for 2 of 3 customers, and premature for the third. It did not fire
nightly — it fired exactly once, at 2026-07-26 03:00 UTC.** No real external customer received
anything.

**Root cause (H2, empty-array variant):** the agent's backup record store is **in-memory** —
`felhom-agent/internal/backup/store.go:15`, whose own doc-comment says *"In-memory (lost on restart;
the cadence re-populates)"*. An agent restart wipes it, so the host-report emits `backups: []` until
the next backup runs. The hub's `assessBackupFreshness` reads an empty array as *no backup exists*.

**Trigger:** the R-50 island migration restarted `felhom-agent` fleet-wide on 2026-07-25
~12:41–12:44 UTC (agent v0.96.0). Backups run on a ~24 h cadence; the next one landed 07-26
07:03 UTC. The 03:00 UTC deadline check fell inside that ~18 h blind window.

**The one genuinely bad finding, unrelated to the alarm text:** the **PBS / offsite-DR tier has no
schedule at all**. demo-felhom's `felhom-offsite` datastore holds exactly one snapshot
(2026-07-18); demo-hp has `pbs_dr` applied since 07-21 and **zero snapshots ever**. See §7.

The premise that ~7.3 days of staleness predates the 07-25 channel outage is **correct but
misleading**: the 176 h figure is the age of the *PBS* snapshot, reached only as a fallback once the
vzdump array went empty. Local vzdump was never 7 days stale — it ran on 07-24, 07-25 and 07-26.

---

## 1. D0 — access inventory

| Target | Reachable | Path | User | Passwordless sudo | Notes |
|---|---|---|---|---|---|
| **DooPlex** (this host) | ✅ | local | `kisfenyo` | ✅ | hosts hub, hub DB, k3s, local PBS |
| **Hub** (k3s `felhom-system`) | ✅ | `sudo kubectl` | — | ✅ | pod `hub-9496fd7c9-kfpkx`, image `felhom-hub:0.74.0` |
| **Hub DB** | ✅ | `kubectl cp` → scratchpad, `sqlite3 -readonly` | — | ✅ | 113 MB copy; **live DB never opened writable** |
| **PVE `demo-felhom`** (N100) | ✅ | `ssh felhom-pve` (Tailscale 100.70.170.35) | `root` | n/a (is root) | full journal + `pvesm` |
| **PVE `demo-hp`** (t740) | ❌ | `ssh demo-hp` → `Permission denied (publickey,password)` | — | — | no baked key. Break-glass root exists in hub `host_recovery` — **not used**, per scope ("do not escalate or hunt for credentials") |
| **PVE `drill-r50`** (host `drill-r50-0a4f9a`) | ❌ | no SSH alias configured | — | — | NOT COLLECTED |
| **PBS `felhom-offsite`** (ep0/Hetzner, via wg `10.77.0.1`) | ⚠️ indirect | read through PVE storage client + agent verify loop | — | — | no direct PBS API/root; free space + prune/GC history NOT COLLECTED |
| **Guest 9201 (demo-felhom)** | ✅ | `ssh felhom-pve "pct exec 9201 …"` | root | n/a | not needed for this diagnosis |
| **DooPlex-local PBS `felhom-spike`** | ✅ | local, `sudo` | — | ✅ | leftover spike store, newest snapshot 2026-07-10 — **not** the live offsite datastore |

**Exit-code discipline.** stderr was inspected on every rc=0. `pvesm status` / `pvesm list` first
emitted perl locale warnings to stderr that truncated the read — re-run with `LC_ALL=C` and stderr
captured to a separate file; all three commands rc=0 with **empty** stderr on the clean run.
`proxmox-backup-manager namespace list` returned rc≠0 (subcommand absent on this PBS version) and
`proxmox-backup-client list` failed with *"no password input mechanism available"* — neither was
retried, because `pvesm list felhom-pbs` yields the same ground truth via the configured credential.

---

## 2. Who the three customers are

| customer_id | Name | Status | DR tier | Customer notify channel | Real external customer? |
|---|---|---|---|---|---|
| `demo-felhom` | Demo Ügyfél | active | 1 | ✅ `doo…@freemail.hu` (operator's own mailbox) | **No** — operator demo box (N100) |
| `demo-hp` | Demo HP | active | 1 | ❌ none — operator channel only | **No** — operator demo box (HP t740) |
| `drill-r50` | drill-r50 | active | 0 | ❌ none — operator channel only | **No** — R-50 drill VM, bound 2026-07-25 09:49 UTC, ~19 h old |

**No real external customer was notified.** The one real external customer in the hub,
**`peti-felhom` ("Peti Proxmox", active, dr_tier 0)**, did **not** fire today and has **no**
`customer_notifications` row (operator channel only, always). It is silent because it has no agent
host-report at all → `deadline.go:176` takes the explicit *"legacy/defunct controller-only
customer"* branch, which deliberately raises no backup alarm.

`customer_notifications` holds exactly **one** real row (`demo-felhom`) plus a
`_resend-rotation-test` stub. So at most one mailbox — the operator's own — can ever receive this
event today.

---

## 3. D1 — the events, verbatim

### 3.1 Every `expected_backup_missed` in the current era (since the 2026-06-16 repoint)

| id | customer | created_at (UTC) | sev | verbatim message |
|---|---|---|---|---|
| 723 | demo-felhom | 2026-06-29 03:00:00 | error | `No fresh verified backup: no PBS snapshot or successful backup in the latest host-report` |
| 1129 | peti-felhom | 2026-07-13 03:00:00 | error | `No fresh verified backup: newest backup is 40h0m0s old (limit 26h0m0s)` |
| 1203 | peti-felhom | 2026-07-14 03:00:00 | error | `No fresh verified backup: no PBS snapshot or successful backup in the latest host-report` |
| 1275 | peti-felhom | 2026-07-15 03:00:00 | error | `No fresh verified backup: no PBS snapshot or successful backup in the latest host-report` |
| **1803** | **demo-felhom** | **2026-07-26 03:00:00** | error | `No fresh verified backup: newest backup is 176h0m0s old (limit 26h0m0s)` |
| **1804** | **demo-hp** | **2026-07-26 03:00:01** | error | `No fresh verified backup: no PBS snapshot or successful backup in the latest host-report` |
| **1805** | **drill-r50** | **2026-07-26 03:00:01** | error | `No fresh verified backup: no PBS snapshot or successful backup in the latest host-report` |

(Rows 339–512, 2026-05-30 → 2026-06-16, are the **pre-repoint** era with the old message
`No backup completed or failed since midnight` — the v0.12.0 CHANGELOG names that as a known daily
false alarm, already fixed. Not relevant.)

### 3.2 Per-customer answers

| | demo-felhom | demo-hp | drill-r50 |
|---|---|---|---|
| **Reason branch fired** | `newest backup is Xh old` | `no PBS snapshot or successful backup` | `no PBS snapshot or successful backup` |
| **First occurrence** | 2026-07-26 03:00:00 UTC | 2026-07-26 03:00:01 UTC | 2026-07-26 03:00:01 UTC |
| **Total count** | 1 (this era: 2, incl. 06-29) | 1 | 1 |
| **Nightly?** | **No — single occurrence** | **No — single occurrence** | **No — single occurrence** |
| **Channels delivered** | operator **and customer** | operator only | operator only |
| **H2 status** | refuted by string alone → H1/H4/H5 | live → D2b decisive | live → D2b decisive |

**The "nightly" premise is refuted.** `notification_log` records **exactly one** customer-channel
send for this event since the 2026-06-16 repoint: id 245, `demo-felhom`, 2026-07-26 03:00:01,
status `sent`, Hungarian subject text **„A mai biztonsági mentés nem készült el a határidőig!"**
(`hub/internal/notify/templates.go:106`), body carrying the English reason string
`No fresh verified backup: newest backup is 176h0m0s old (limit 26h0m0s)`.

### 3.3 Co-firing signals in the same window

`expected_dbdump_missed`: **not fired** for any of the three (all three emitted `db_dump_completed`
at 00:30 UTC today). `host_staleness` / `node_stale` / `offsite_stale` / `pbsdr_*`: **none** on
2026-07-26. The only other events today are `agent_channel_unreachable` (demo-felhom 06:16, demo-hp
06:19, drill-r50 07:23 — transient) and `local_api_endpoint_drift` on drill-r50 (07:21, unrelated,
see §10).

---

## 4. D2 — what the agent reported

Latest host-report per customer (2026-07-26 07:42–07:44 UTC), `pbs_snapshots` and `backups` verbatim:

**demo-felhom** — `pbs_snapshots` 1 entry, `backups` 1 entry:
```json
"pbs_snapshots": [{ "namespace":"root","backup_type":"ct","backup_id":"9201",
  "backup_time":"2026-07-18T18:31:06Z","size_bytes":9744319415,
  "owner":"felhom@pbs!demo-felhom","protected":false,"encrypted":true,
  "verify_state":"ok",
  "verify_upid":"UPID:felhom-hetzner:000003D4:000005BE:00000048:6A65ACDA:verify:felhom\\x2doffsite\\x3ans-demo\\x2dfelhom:felhom@pbs!demo-felhom:" }]
"backups": [{ "target_id":"local","vmid":9201,
  "archive":"local:backup/vzdump-lxc-9201-2026_07_26-09_03_19.tar.zst",
  "mode":"snapshot","crash_consistent":true,"size_bytes":5690147837,
  "success":true,"started_at":"2026-07-26T07:03:19Z","duration_seconds":216.556501101,
  "uncovered_volumes":["/etc/felhom-bootstrap","/mnt/felhom-drives"] }]
```

**demo-hp** — `pbs_snapshots: []` (0 entries), `backups` 1 entry:
```json
"backups": [{ "target_id":"local","vmid":9201,
  "archive":"local:backup/vzdump-lxc-9201-2026_07_26-09_06_02.tar.zst",
  "mode":"snapshot","crash_consistent":true,"size_bytes":1476771458,
  "success":true,"started_at":"2026-07-26T07:06:02Z","duration_seconds":76.320400309,
  "uncovered_volumes":["/etc/felhom-bootstrap","/mnt/felhom-drives"] }]
```

**drill-r50** — `pbs_snapshots: []`, `backups: []`. Both empty. No `pbs_dr` stanza (`null`).

> Note the **timing trap**: these are the *07:44* reports. At **03:00**, when the check ran, all
> three `backups` arrays were **empty** — see §5.

### 4.1 D2b — the parse check (H2 parse variant)

Agent wire struct: `felhom-agent/internal/hub/report.go:27,29` → `Backups []Backup`,
`PBSSnapshots []PBSSnapshot`. Emitted values are `2026-07-26T07:03:19Z` / `2026-07-18T18:31:06Z` —
plain RFC3339 with a `Z` offset. `parseBackupTime` (`deadline.go:107`) accepts exactly
`time.RFC3339`. **No format mismatch.**

**H2's parse variant is REFUTED. H2's empty-array variant is the root cause.** The two are conflated
behind one reason string — see fix #3.

### 4.2 The mechanism, proven

`felhom-agent/internal/backup/store.go:10-19`:

> *"Store holds the agent's LATEST backup result per target … **In-memory (lost on restart; the
> cadence re-populates)** and mutex-guarded…"*

`Collector.collectBackups` (`internal/hub/collect.go:393`) returns `[]Backup{}` when the store is
empty. Hub-side, `assessBackupFreshness` sees `!havePBS && !haveVzdump` → fires *"no PBS snapshot or
successful backup"*; or, if a stale PBS snapshot survives (demo-felhom), falls through to the age
branch and reports **the PBS snapshot's** age — 176 h.

**The alarm cannot distinguish "no backup was taken" from "the agent forgot".**

---

## 5. D4 — the discriminator: agent-reported vs ground truth

Ground truth gathered read-only on `felhom-pve` (`pvesm list`, rc=0, empty stderr — **no verify, no
prune, no GC, no backup triggered**):

```
PBS  felhom-pbs (datastore felhom-offsite, ns demo-felhom, server 10.77.0.1):
  felhom-pbs:backup/ct/9201/2026-07-18T18:31:06Z   pbs-ct  9744319415  9201     <- ONLY entry

LOCAL local (content=backup):
  local:backup/vzdump-lxc-9100-2026_07_18-18_30_52.tar.zst    612397450  9100
  local:backup/vzdump-lxc-9100-2026_07_20-17_50_57.tar.zst    648549366  9100
  local:backup/vzdump-lxc-9201-2026_07_24-21_12_03.tar.zst   5487897588  9201
  local:backup/vzdump-lxc-9201-2026_07_25-08_30_14.tar.zst   5603727891  9201
  local:backup/vzdump-lxc-9201-2026_07_26-09_03_19.tar.zst   5690147837  9201
```

### 5.1 What the check actually saw, night by night (demo-felhom)

The report nearest each 03:00 UTC — this is the decisive table:

| Check at | Report used | `backups` newest vzdump | `pbs_snapshots` newest | Age of newest | Fired? |
|---|---|---|---|---|---|
| 07-19 03:00 | 07-19 02:45 | 2026-07-18T18:20:11Z | none | 8.7 h | no ✅ correct |
| 07-20 03:00 | 07-19 11:30 | 2026-07-18T18:20:11Z | none | 32.7 h | no ⚠️ *(see note)* |
| 07-21 03:00 | 07-21 02:55 | 2026-07-20T06:40:45Z | none | 20.3 h | no ✅ correct |
| 07-22 03:00 | 07-22 02:47 | 2026-07-21T11:18:24Z | 2026-07-18T18:31:06Z | 15.7 h | no ✅ correct |
| 07-23 03:00 | 07-23 02:59 | 2026-07-22T17:21:46Z | 2026-07-18T18:31:06Z | 9.6 h | no ✅ correct |
| 07-24 03:00 | 07-24 02:55 | 2026-07-23T11:41:34Z | 2026-07-18T18:31:06Z | 15.3 h | no ✅ correct |
| 07-25 03:00 | 07-25 02:57 | 2026-07-24T19:12:03Z | 2026-07-18T18:31:06Z | 7.8 h | no ✅ correct |
| **07-26 03:00** | 07-26 02:59 | **NONE (array empty)** | 2026-07-18T18:31:06Z | **176 h** | **YES** ❌ false |

> *07-20 note:* the customer was `down` that morning (`node_down` 07-19 12:38) and would have been
> skipped by the `staleness.GetState(id) == "down"` guard at `deadline.go:160`. Either way no alarm
> was raised, and a real backup had run 07-20 06:40.

**The deadline check ran every night and was correctly silent on all of them.** The 07-26 fire is
caused solely by the array going empty. This also refutes any "the scheduler stopped" theory —
`scheduleDaily` (`cmd/hub/main.go:677`) recomputes the next 05:00 Budapest on every loop iteration
and is sound.

### 5.2 The array-emptying transitions (all three customers)

Only the change-points are shown:

```
demo-felhom   2026-07-24 19:11:58  nb=0            <- agent restart, refilled 30 min later
              2026-07-25 06:23:55  nb=0            <- agent restart, refilled 15 min later
              2026-07-25 12:44:05  nb=0            <- R-50 restart; STAYS EMPTY across 03:00
              2026-07-26 07:12:xx  nb=1 vzdump=2026-07-26T07:03:19Z   <- self-cleared

demo-hp       2026-07-24 18:46:13  nb=1 vzdump=2026-07-24T18:38:30Z
              2026-07-25 10:22:45  nb=0            <- agent restart, refilled 15 min later
              2026-07-25 12:41:28  nb=0            <- R-50 restart; STAYS EMPTY across 03:00
              2026-07-26 07:12:20  nb=1 vzdump=2026-07-26T07:06:02Z   <- self-cleared

drill-r50     2026-07-25 09:50:55  nb=0 npbs=0     <- never had a backup; still nb=0
```

The two earlier restarts (07-24 19:11, 07-25 06:23) also emptied the array, but a backup followed
within 30 minutes, so no 03:00 boundary was crossed. **The R-50 restart is the one that landed in
the blind window.**

### 5.3 Verdict per customer

| Customer | Agent said | PBS/local ground truth | **Verdict** |
|---|---|---|---|
| **demo-felhom** | `backups: []` at 03:00; PBS 07-18 | vzdump ran **07-24 21:12, 07-25 08:30, 07-26 09:03 CEST** (3 archives retained, retention=3); PBS **1 snapshot, verify_state `ok`** | **H2 — FALSE ALARM.** Local tier fully healthy. PBS tier genuinely stale (§7), but that is a *different* fault than the message implies |
| **demo-hp** | `backups: []`, `pbs_snapshots: []` at 03:00 | vzdump **07-25 10:23, 07-26 07:06 UTC** both `success:true`; PBS **0 snapshots ever** | **H2 — FALSE ALARM.** Local tier healthy. PBS tier never ran (§7) |
| **drill-r50** | both empty | nothing, ever | **H1 — TRUE but premature.** Box bound 2026-07-25 09:49 UTC, ~17 h before the check; no backup was yet due |

---

## 6. Verdicts on H1–H6

| | Verdict | The observation that decided it |
|---|---|---|
| **H1** genuine backup outage | **REFUTED for the host/vzdump tier; SUPPORTED for the PBS/offsite-DR tier; SUPPORTED-but-premature for drill-r50** | `pvesm list local` shows three retained vzdump archives (07-24, 07-25, 07-26) and the agent journal logs `backup: completed … target=local` at 09:06:56 CEST today. Conversely `pvesm list felhom-pbs` returns exactly one snapshot, from 07-18 |
| **H2** reporting/parse break | **SUPPORTED (empty-array variant); REFUTED (parse variant)** | `store.go:15` is in-memory by design; §5.2 shows the array emptying at each agent restart and refilling at the next backup. Timestamps are valid RFC3339 `Z`, so `parseBackupTime` never silently drops one |
| **H3** verification failure | **REFUTED** | demo-felhom's only snapshot carries `verify_state: "ok"`; the agent journal logs `pbs: verify cycle complete datastore=felhom-offsite snapshots=1` at 02:44 and 08:44 CEST today. demo-hp/drill-r50 have no snapshots to verify |
| **H4** transport | **REFUTED** | `wg show` on felhom-pve: peer `167.233.158.164:443`, **latest handshake 39 seconds ago**, 753.13 MiB rx / 481.85 MiB tx, keepalive 25 s. All three reports carry `wireguard.active: true` with `last_handshake_age_s` 36–46. `pvesm status` shows `felhom-pbs` **active** |
| **H5** schedule stopped | **PARTIALLY SUPPORTED** | The vzdump cadence is alive but its 24 h phase **resets on agent restart** (`BackupCadence()` default 24 h, `config.go:390`), which opened the window. Separately, **no PBS schedule has ever existed**: `backup.local_backup_target = "local"` and no `/etc/pve/jobs.cfg` (§7) |
| **H6** stale host-report | **REFUTED** | Latest reports 2026-07-26 **07:42:19 / 07:44:41 / 07:44:53** UTC, all agent 0.96.0; 664 / 436 / 96 rows. No `host_staleness` fired |

---

## 7. The real finding — the PBS / offsite-DR tier has no schedule

This is **not** what the alarm text described, and it is the one thing here that is genuinely wrong.

- `felhom-agent`'s backup runner has exactly **one** target, resolved by
  `BackupConfig.BackupTarget()` (`internal/config/config.go:415`): `local_backup_target` if set,
  else the constant `defaultBackupTarget = "felhom-pbs"`.
- On demo-felhom, `/etc/felhom-agent/agent.json` sets `"local_backup_target": "local"`. **Therefore
  no agent-driven backup on that host ever targets `felhom-pbs`.**
- There is **no `/etc/pve/jobs.cfg`** on the host — no PVE-native backup job either.
- The single PBS snapshot (2026-07-18T18:31:06Z) predates that and was not produced by a recurring
  job.
- demo-hp reports `pbs_dr: {state:"applied", storage_id:"felhom-pbs", namespace:"demo-hp",
  applied_at:"2026-07-21T16:35:44Z"}` and **`pbs_snapshots: []`** — the DR tier has been "applied"
  for 5 days and has produced nothing.
- Today's vzdump was **controller-initiated**, not cadence-initiated:
  `local-api: backup reached snapshotted (app may resume) vmid=9201 job=backup-9201-1785049399941273219`
  → `backup: completed … target=local`. The agent exposes `GET /backup/due` with a 24 h cadence
  (`localapi/server.go:92-95`) and the in-guest controller drives it.

**So the offsite-DR promise is currently unbacked on both demo boxes.** Worth its own task; it is
not fixable as a one-liner.

---

## 8. D7 — timeline

| When (UTC) | What |
|---|---|
| 2026-06-16 | hub **v0.12.0** repoints `CheckBackupDeadlines` from `backup_completed` events to host-report freshness. **This is where the empty-array sensitivity is introduced** (`assessBackupFreshness`, commit `0635640`) |
| 2026-07-18 18:20 | last vzdump before the PBS snapshot |
| **2026-07-18 18:31:06** | **the one and only PBS snapshot on demo-felhom.** Nothing has been written to `felhom-offsite` since |
| 2026-07-21 08:39 / 16:35 | `pbs_dr` applied — demo-felhom / demo-hp |
| 2026-07-24 19:11 | agent restart → `backups` empties, refills 19:42 (no 03:00 crossed) |
| 2026-07-25 06:23 | agent restart → `backups` empties, refills 06:38 (no 03:00 crossed) |
| 2026-07-25 09:49 | **drill-r50** appliance bound — a brand-new customer with no backup history |
| **2026-07-25 12:41 / 12:44** | **R-50 island-bridge fleet migration — `felhom-agent` v0.96.0 restarted on every box.** `systemctl show felhom-agent` on felhom-pve: `ExecMainStartTimestamp=Sat 2026-07-25 14:44:37 CEST`, `NRestarts=0`. `backups` empties **and stays empty** |
| 2026-07-26 00:30 / 01:30 | `db_dump_completed` + `crossdrive_completed` — app-data tiers healthy |
| **2026-07-26 03:00:00–03:00:01** | **deadline check fires on all three.** demo-felhom also mails the customer channel |
| 2026-07-26 07:03 / 07:06 | next vzdumps run → arrays repopulate → **condition self-clears** |
| 2026-07-26 07:15:58 | hub pod restarts onto **v0.74.0** — *after* the alarm, not causal |

**What changed at the outage start: the R-50 island migration's fleet-wide agent restart on
2026-07-25 12:44 UTC.** That is the named, correlated change. Nothing else in the window touches
backups.

Host uptime is 3 d 14 h (boot ~2026-07-22 19:14 UTC) — **no host reboot** in the window; the agent
restart was a service restart. hub v0.73.2 was, as stated in the task, a `hostInstallVersion` string
bump never deployed as an artifact (`manifests/hub.yaml` now pins 0.74.0) — **confirmed irrelevant**.
`deadline.go` has not been touched since 2026-06-16, so neither 0.73.x nor 0.74.0 changed this logic.

---

## 9. D8 — is the customer-facing message accurate?

**No, on two independent counts.**

The Hungarian template (`hub/internal/notify/templates.go:106`) is
**„A mai biztonsági mentés nem készült el a határidőig!"** — *"Today's backup was not completed by
the deadline!"*. A customer reads that as *all my backups failed*.

At the moment it was sent to demo-felhom (2026-07-26 03:00:01):

| Tier | State at send time | Healthy? |
|---|---|---|
| Local vzdump (host tier) | `vzdump-lxc-9201-2026_07_25-08_30_14.tar.zst` on disk, 5.6 GB, 20 h old | ✅ |
| App-data cross-drive (tier 2) | `crossdrive_completed` 2026-07-26 01:30:06 — 1.5 h before the mail | ✅ |
| DB dump | `db_dump_completed` 2026-07-26 00:30:56 — 2.5 h before the mail | ✅ |
| PBS / offsite-DR | last snapshot 2026-07-18, `verify_state: ok` | ❌ genuinely stale |

1. **The trigger was an artifact** — the local host-tier backup had run and was on disk; the agent
   had merely forgotten it across a restart.
2. **Even when the signal is true, the scope is wrong** — this check covers only the PBS/vzdump host
   tier. The controller's app-data tiers are separate signals (`backup_failed`, `db_dump_failed`,
   `crossdrive_*`) and were all green.

Recorded as a finding. Not changed.

---

## 10. Proposed fixes — described, NOT applied

| # | Where | Fix | Class |
|---|---|---|---|
| **1** | `felhom-agent/internal/backup/store.go` (`Store`, `RecordBackup`, `Backups`) | Stop losing backup history on restart. Either **persist** the latest-per-target record to the agent's state dir, or have `Collector.collectBackups` **fall back to querying PVE** for the newest archive on the target storage (`pvesm list <target> --content backup` / the API equivalent) when the store is cold. The query path is arguably better — it is ground truth rather than remembered state, and it would also have surfaced the real PBS gap | **needs-a-spike** (persistence vs query-on-demand is a design call) |
| **2** | `felhom.eu/hub/internal/monitor/deadline.go` → `assessBackupFreshness` | **Do not alarm on absence of evidence.** The function's own doc says *"Only POSITIVE evidence of a problem fires an alarm"*, but `!havePBS && !haveVzdump` is precisely *absence* of evidence. Narrow it: empty+empty → **unknown**, log + operator-only, never the customer channel. Optionally require the condition to persist across ≥2 consecutive daily checks before escalating. **This is the same bug shape as the hub v0.73.0 `offsite_stale` fix** (never-ran branch with no time anchor) | **safe-obvious** for the operator-only narrowing; **needs-a-spike** for a full unknown-state model |
| **3** | same file, same function (`deadline.go:90`) | **Split the conflated reason string.** `"no PBS snapshot or successful backup in the latest host-report"` currently covers both *arrays empty* and *every timestamp unparseable*. Emit distinct strings so the next diagnosis is one query instead of a session | **safe-obvious** |
| **4** | `felhom.eu/hub/internal/notify/templates.go:106` | **Narrow the customer copy.** Name the host/DR tier explicitly and state that app data is covered by separate alerts, so the message stops implying total backup failure | **safe-obvious** — run `scripts/hub_confirm_gate.py` + the `felhom-ui-design` copy rules |
| **5** | agent config on both demo hosts + `internal/backup/runner.go` | **Wire an actual PBS/offsite-DR schedule** (§7). Today `local_backup_target: "local"` means nothing ever targets `felhom-pbs`, and the runner supports only one target per run. Decide: second target per run, alternating cadence, or a PVE backup job | **needs-a-spike** — this is the real defect and the largest piece of work |
| **6** | `customer_notifications` for `demo-felhom` | **RECOMMENDATION — operator decision, deliberately NOT applied.** `enabled_events` currently includes `expected_backup_missed`. Given that the check demonstrably false-alarms on any agent restart and the copy overstates scope, consider **removing `expected_backup_missed` from that list until fixes 1–4 land**. The operator channel keeps full visibility. Mitigating factor: the only subscribed mailbox is the operator's own, so the blast radius today is zero — this can wait for the proper fix | operator's call |

**Suppression urgency: LOW.** The condition self-cleared at 07:03 UTC, no real external customer is
subscribed, and the alarm requires an agent restart in the backup blind window to recur.

---

## 11. Not collectable

| Item | Why | What access would be needed |
|---|---|---|
| demo-hp host-level: agent journal, `wg show`, `agent.json` backup stanza, PBS attempt logs | `ssh demo-hp` → `Permission denied (publickey,password)`; break-glass root exists in hub `host_recovery/demo-hp-bb76ea` but was **not used** per scope | a baked SSH key, or explicit authorisation to use the break-glass credential |
| drill-r50 host-level (host `drill-r50-0a4f9a`) | no SSH alias configured; host not identified beyond the report | an SSH route to that host |
| PBS server side: `felhom-offsite` free space, prune/GC job history, per-namespace usage | no direct API/root on the ep0/Hetzner PBS; local `proxmox-backup-client` has no password input mechanism | PBS API token or root on the PBS box |
| Hub logs from 03:00 today | the hub pod restarted at 07:15:58 UTC onto v0.74.0; `kubectl logs` only reaches back to then, and there is no previous-container log | a log sink outside the pod |
| Whether a datastore-full condition contributed | `pvesm status` reports `felhom-pbs` Total/Used/Available all **0 KiB** while Status=`active` — see §12 | PBS-side status |

---

## 12. Observations — noticed, not acted on

1. **`pvesm status` reports `felhom-pbs` with Total = Used = Available = 0 KiB**, yet Status `active`
   and `pvesm list` works. Either the PBS storage plugin isn't returning usage over the wg link, or
   the datastore genuinely reports nothing. Worth a look when §7 is picked up — a full datastore is a
   classic silent-write-refusal cause, and this reading cannot currently rule it out.
2. **`customer_notifications.updated_at` for `demo-felhom` = 2026-07-26 07:17:54** — that row was
   touched **today, after the alarm**. Flagging in case the operator already adjusted something;
   I did not change it and did not investigate further.
3. **drill-r50 is emitting `local_api_endpoint_drift`** (07:21 today):
   `controller.yaml=192.168.0.176:8443 bootstrap.json=169.254.253.1:8443 (pin agrees: true)`, plus
   `agent_channel_unreachable`. Unrelated to this diagnosis, but live and probably R-50 fallout.
4. **`agent_channel_unreachable` fired on demo-felhom (06:16) and demo-hp (06:19) today** and
   recovered. The task's framing treated the 07-25 channel outage as a possible cause; it is not —
   §5.1 shows backups ran normally throughout it. Coincidence confirmed.
5. **The DooPlex-local PBS datastore `felhom-spike`** (`/mnt/5_hdd/backup/demo-felhom`, the only
   entry in `datastore.cfg`) is a leftover spike store with snapshots up to 2026-07-10. It is **not**
   the live offsite datastore — that is `felhom-offsite` on `felhom-hetzner`, reached over wg at
   `10.77.0.1`. Easy to confuse when hunting PBS ground truth on this host.
6. **`restore_test_cadence_seconds: 0`** in the agent config → `RestoreTestCadence()` maps 0 to the
   24 h default, so restore tests should be running; `restore_tests` was not examined in this pass.
7. The pre-repoint era (2026-05-30 → 06-16) shows this same event false-alarming **daily** on
   peti-felhom and demo-felhom for a different reason, fixed in v0.12.0. Combined with the v0.73.0
   `offsite_stale` fix, **this is the third false-alarm in the same monitor family** — all three from
   treating "no signal" as "bad signal". Fix #2 addresses the class, not just the instance.

---

## 13. What was NOT done (prime directive compliance)

No backup was run, triggered or tested. No PBS verify, prune or GC. No agent, hub, PBS or
`wg-felhom` restart. No alert cleared, acknowledged or muted. No notification preference or channel
setting changed. No schedule, timer or config file edited. No code changed, no version bumped.

The hub DB was **copied out** (`kubectl cp`) and every query ran against the **copy** with
`sqlite3 -readonly`; the live `/data/hub.db` was never opened writable. All host commands were
listings (`pvesm list`, `pvesm status`, `wg show`, `systemctl show`, `journalctl`).

The staleness measurement is intact: as of this writing the newest `felhom-offsite` snapshot is still
**2026-07-18T18:31:06Z**.