felhom.eu/REPORT.md

# REPORT — closing the backup arc: seam sweep + live outage validation (2026-07-27)

**Overwritten** per the standing rule. **No version shipped** — the sweep found no defect, so the
witnesses are compile-time-only source changes riding the next release. Agent `9842c52`→`0236553`,
controller `ca013c8`→`8f46495`. Live fleet unchanged: hub v0.79.0, agent v0.105.0, controller v0.178.0.

## Part 1 — the seam table

Every optional-capability type assertion across all four repos. `StackDataProvider`, `BackupReporter`
and `RestoreTestReporter` are **not** in this table: they are interface-typed struct *fields*, which
the compiler already enforces at assignment. The dangerous shape is `x.(Interface)`.

| Interface | Asserted at | On failure | Silent? | Witness now |
|---|---|---|---|---|
| `quiesce.TieredBackend` | `tiers.go:109` (`resolveDueTiers`) | falls back to the untargeted single-tier path | **SILENT** | ✅ moved to production |
| `quiesce.TieredBackend` | `quiesce.go:375` (`allTiersForManualRun`) | manual run covers only the untargeted tier | **SILENT** | ✅ same witness |
| `quiesce.TieredBackend` | `tiers.go:191`, `:199` | **unchecked — would panic** | no (loud) | ✅ same witness |
| `localapi.BackupArchiveLister` | `server.go:1155` (`newestArchiveOn`) | degrades to `archiveAbsent` = pre-R-84 in-memory-only | **SILENT** | ✅ added |
| `agentapi.AgentVersionReporter` | `features.go:157` | version gate → live probe | silent, benign (both decide correctly) | ✅ added |
| anonymous `GuestMemory` iface | `features.go:86` | returns `errNoMemoryProbe` → `SupportUnknown` | **no** — distinct error | n/a (anonymous) |

**Was `TieredBackend` the only one, or the first one? It was the first one.**
`BackupArchiveLister` has the identical shape and a worse blast radius: a silent degrade to
`archiveAbsent` is the R-84 bug returning — a cold store after a restart reading as "no backup ever" —
with nothing in any log. It had **no witness of any kind**, and in fact **zero compile-time witnesses
existed in production code anywhere in either repo** before today.

**No defect found.** Every implementation satisfies its interface today, so per the task's rule there
is no version bump and no deploy. What was missing was the *guard*, not the behaviour.

**The guard is proven, not assumed.** Breaking `BackupRunner.NewestArchiveTime`'s signature now fails
`go build`:

```
cmd/felhom-agent/seam_witnesses.go:28:38: *backup.BackupRunner does not implement
    localapi.BackupArchiveLister (wrong type for method NewestArchiveTime)
```

Before the witness, that same break compiled and vetted clean. The controller's witness also **moved
out of a `_test.go` file** — a test-file witness fires on `go test`/`go vet` but not on `go build`
alone, which is exactly the path the R-88 Part 2 near-miss would have taken.

## Part 2 — the live outage

Pre-flight cleared: 0 tasks in flight on ep0, 0 backup/restore processes on either box (counted by
`COMM`, since `pgrep -f` self-matches and gave a false `1` first), 0 scratch guests, no quiesce marker.

### The headline — `age_state=unknown` on real hardware, first ever

```json
{"vmid":9201,"due":true,
 "reason":"backup storage unreadable and no in-memory record — age UNKNOWN, treating as due",
 "target":"felhom-pbs","age_state":"unknown"}
```

Identical on both boxes. The transition is timestamped on demo-felhom:

```
18:35:30  felhom-pbs/content  200    ← before power-off
18:39:26  felhom-pbs/content  500    ← after
18:39:11  local/content       200    ← local tier, same window, unaffected
```

Agent journal, both boxes: `could not read the backup storage for the due-check … target=felhom-pbs`
— the degrade path genuinely executed. **Positive observables throughout**, per standing rule 3: an
empty quiesce log alone would have been equally consistent with a dead loop.

### The behaviour it gates

demo-felhom's `local` tier was **genuinely due** (25.8 h) for the whole window, so the due set was
`[local(known), felhom-pbs(unknown)]` → `oldestAge` nil → `valveLicensed` false → **defer**. This is
the 09:02 incident's exact condition replayed with the fix in place.

| | demo-felhom | demo-hp |
|---|---|---|
| App containers running | **10**, unchanged | **3**, unchanged |
| Quiesce cycles | **0** | **0** |
| Quiesce marker | absent | absent |
| vzdump attempted | none since 07:13 | none since 09:37 |

**Not one app stack was stopped.** Per-tier isolation demonstrated on the same box in the same
minute: `local` → 200 with a real age, `felhom-pbs` → 500 → unknown.

### Item 4 — conditions did not arise, and that is confirmed, not assumed

`felhom-pbs` was not due on either box (28 h / 7 h against a 168 h cadence), so no backup ran, nothing
failed, **the R-88 breaker did not arm**, and **no `whole_guest_backup_failed` travelled**. I did not
force one. Confirmed positively: the newest hub event is `16:10:33`, *before* the window opened at
16:24, and 0 events exist in the window.

## Part 2.2 — post-boot

| Check | Result |
|---|---|
| Mount | `/mnt/pbs-datastore ext4 rw,relatime,discard` — **relatime present, `noatime` ABSENT** ✅ |
| Swap | `/swapfile 4G` back — **survived its first reboot** (it was added and verified without one) ✅ |
| PBS services | `proxmox-backup` + `proxmox-backup-proxy` both active ✅ |
| Datastore | `felhom-offsite → /mnt/pbs-datastore`, online ✅ |
| Snapshots | demo-felhom **2**, demo-hp **3** — **exact match**, same timestamps ✅ |
| Schedules | `gc-schedule sun 04:30`, `verify-new 1`, 2 prune jobs ✅ |
| Filesystem | `EXT4-fs (sda): recovery complete` — journal replayed after the unclean power-off, then mounted r/w; **0** I/O or EXT4 errors ✅ |

### The unplanned finding: the volume changed device name

```
before:  volume=/dev/sdb   root=/dev/sda1
after:   volume=/dev/sda   root=/dev/sdb1
```

The kernel enumerated the disks in the opposite order across the reboot. The mount survived only
because fstab uses `/dev/disk/by-id/scsi-0HC_Volume_106469259`. **A `/dev/sdb` entry would have failed
to mount — or mounted the root disk's device node.** That choice was load-bearing and had never been
tested; this reboot tested it.

### Recovery

Both boxes returned to `age_state=known` on both tiers within ~60 s of boot; `felhom-pbs` on
demo-felhom back to `due=false, within cadence window, age_seconds=102102`. No breaker was armed, so
none needed clearing; no failure event fired, so no recovery event was expected — and none appeared.

## Part 3 — CX33

**Not available even with the server powered off**, so the powered-on theory is settled: it is the
Cost-Optimized line's "Limited availability", not the power state. No rescale happened. R-90 stays
`BLOCKED`/`WATCHING` with the 4 GiB swap as the interim — now a *proven* interim, since it survived a
reboot. Escape hatch if 8 GB becomes urgent: the **Regular Performance (CPX)** / **General Purpose
(CCX)** lines carry no availability warning, at higher cost. Operator's call.

## Comes off the never-live-validated list

- **`age_state=unknown` observed on real hardware** — the path R-88 Part 2 exists for.
- **The window gate refusing an unknown-driven valve, with a genuinely due tier present** — zero app
  stacks stopped.
- **Per-tier isolation under a real offsite outage.**
- **The by-id mount surviving a reboot** (and surviving a device rename, which was not anticipated).
- **The 4 GiB swap surviving a reboot.**
- **Datastore integrity across an unclean power-off** — journal replay, exact snapshot match.

## Honestly cannot come off it

- **The R-88 breaker has still never armed on real hardware.** No backup was due, so none failed.
- **`whole_guest_backup_failed` has never travelled from a REAL failure** — only from the induced
  POST on 2026-07-27 15:04:57. The quiesce→notify link remains unit-proven.
- **R-97b's app-alarm suppression has never fired live** — it needs a real quiesce cycle.
- **`age_state=absent` has never been observed live** — it needs a genuinely fresh tier.
- **The legacy-agent path (Scenario C) is now unobservable on this fleet** — both agents are v0.105.0.
- **The first in-window scheduled backup** lands 2026-07-28 at the earliest.