felhom.eu/documentation/audits/SPIKE-r82-phase0-2026-07-26.md

# SPIKE — R-82 Phase 0: the three gates before the backup target split (2026-07-26)

**Class:** read-only Phase-0 gate for R-82. **No code written. No backup triggered. No config changed.**
Baselines: felhom-agent `dfd5d73` v0.96.0, felhom-controller `47fda06` v0.173.0, felhom.eu `c73800c`
hub v0.75.0. All three trees clean at `origin/main`.

---

## Verdicts — one gate returns STOP

| Gate | Verdict |
|---|---|
| **P0.1** — what is exposed for seven days | **weekly CONFIRMED**, with the exposed set named and one conditional |
| **P0.2** — the `pvesm status` 0/0/0 anomaly | **RESOLVED — benign PVE-side reporting artifact.** Not a blocker |
| **P0.3** — capacity headroom | **🛑 STOP.** The 80% alert is reached at roughly the **second additional customer**, inside the alpha horizon |

**R-82 stops here pending an operator ruling on P0.3.** P0.1 and P0.2 clear; the blocker is
capacity, not correctness.

---

## P0.1 — What is exposed for seven days?

### Method

The whole-guest vzdump covers what PVE will actually snapshot. Live guest config (`pct config 9201`,
demo-felhom):

```
rootfs: local-lvm:vm-9201-disk-0,size=32G                              → IN the snapshot
mp0:    local-lvm:vm-9201-disk-1,mp=/var/lib/docker,backup=1,size=200G → IN
mp1:    local-lvm:vm-9201-disk-2,mp=/mnt/sys_drive,backup=1,size=50G   → IN
mp8:    /mnt/felhom-drives,mp=/mnt/felhom-drives                       → BIND — NOT in the snapshot
mp9:    /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1 → BIND — NOT in
```

Corroborated by the agent's own report: `"uncovered_volumes": ["/etc/felhom-bootstrap",
"/mnt/felhom-drives"]`.

**The daily tiers are alive** — this is what makes the answer come out the way it does, so it was
verified rather than assumed:

| Customer | Tier-3 offsite (restic, daily) | Evidence |
|---|---|---|
| demo-felhom | **running** | `last_run 2026-07-26T02:17:56Z`, `last_status "ok"`, 28 snapshots, 1.08 GB / 50 GB |
| demo-hp | **running** | `last_run 2026-07-26T02:15:50Z`, `last_status "ok"`, 4 snapshots, 29 MB / 50 GB |
| drill-r50 | **`offsite: null` — not configured** | see the conditional below |

### The exposed-state table

| State | Where it lives | In the vzdump? | Covered by a DAILY tier? | 7-day staleness cost |
|---|---|---|---|---|
| **`encryption.key`** (app-secret key) | controller data dir | ✅ | ❌ none | **NONE — the key is STABLE.** `mtime 2026-07-18 16:32`, unchanged since creation. A 7-day-old copy is byte-identical. *This was the biggest theoretical risk and it is clean.* |
| **`settings.json` — SMB shares** (`smb`, `smb_shares`) | controller data dir | ✅ | ✅ **yes** — R-7b payload in tier-2 **and** offsite (`_shares-manifest.json` + `passdb.tar`, staged 2026-07-26 02:16) | none |
| **`settings.json` — everything else** | controller data dir | ✅ | ❌ **none** | **THE EXPOSED SET.** `mtime 2026-07-26 10:02` — actively changing. Detailed below |
| `controller.yaml` | controller data dir | ✅ | ❌ | **NONE in practice** — the hub is the source of truth (`config_pull` / bootstrap re-fetch overwrites it; it explicitly never touches `settings.json`) |
| offbox credentials (`repo_password`, `ssh_key`, `known_hosts`) | controller data dir | ✅ | ❌ | **NONE — stable** (`mtime 2026-07-18 16:55`) *and* the restic password is escrowed hub-side |
| App definitions (`/opt/docker/stacks/*`) | guest rootfs | ✅ | ✅ **yes** — Tier-1 recovery unit captures `compose/` per app, daily; tier-3 lifts it offsite | none |
| App data (DB dumps, volume tars) | app drives | ✅ (volumes on mp0) | ✅ **yes** — Tier-1/2/3, daily | none |
| Customer bulk data | `/mnt/felhom-drives` | ❌ bind | ✅ Tier-2/3 per class | n/a — never was PBS's job |
| `bootstrap.json` | host, bind at mp9 | ❌ bind | ❌ | none — agent-side state, regenerated at provision |
| `metrics.db` | controller data dir | ✅ | ❌ | metrics history only — cosmetic |
| Guest rootfs / OS / packages | rootfs | ✅ | ❌ | 7 days of OS drift. Re-derivable from the golden image + host-install |

### What is actually lost at 7 days

Only the non-SMB half of `settings.json`, and none of it is catastrophic:

| Key | Consequence at 7 days | Recovery |
|---|---|---|
| `storage_paths` | drive enrollments made in the last week are lost from the registry | drives are **physical and still present** — re-run the drive wizard |
| `app_backup` | per-app backup toggles revert | customer re-selects |
| `notifications` | prefs revert | hub seeds prefs at claim (F11/F12); `customer_notifications` is hub-side |
| `password_hash` | dashboard password reverts | hub-issued claim-reset code — the supported path |
| `launcher_share_token` | shared launcher URLs break | regenerate + reshare (v0.165.0) |
| `claim_*`, `db_validations`, `hub_*` | operational state | re-derived on next hub contact |

### Verdict: **weekly CONFIRMED**

Nothing catastrophic is exposed. The two things that would have overturned this — the app-secret
encryption key and the offsite credentials — are **stable files that have not changed since first
boot**, so a 7-day-old copy is identical to a fresh one. Everything referentially coupled to app
state (definitions, DB dumps, volumes, bulk data, SMB shares) is carried **daily** by Tier-1/2/3, and
Tier-3 offsite was verified running and `ok` on both production boxes this morning.

**⚠️ The conditional — and it is load-bearing.** This verdict depends on the daily offsite tier being
alive. **drill-r50 reports `offsite: null`.** On a box with no Tier-3, PBS-weekly would be the *only*
DR tier, and the 7-day window would then cover **app data and app definitions**, not just settings —
a completely different risk. Recommended ruling to carry into Slice D: **PBS weekly is only
sufficient where Tier-3 offsite is enabled and healthy; a box without it needs either offsite
enabled first, or a shorter PBS cadence.** That is a cadence decision and belongs to the operator.

---

## P0.2 — The `pvesm status` 0/0/0 KiB anomaly: RESOLVED, benign

**It is a PVE-side reporting artifact. The datastore is real, writable, and has capacity.**

Reproduced, then settled from three independent directions:

**1. The anomaly is real and consistent** — both the CLI and the API agree:
```
pvesm status --storage felhom-pbs   → active, Total 0, Used 0, Available 0, 0.00%   (rc=0, empty stderr)
pvesh get /nodes/demo-felhom/storage/felhom-pbs/status
   → {"active":1,"avail":0,"content":"backup","enabled":1,"shared":1,"total":0,"type":"pbs","used":0}
```

**2. PBS itself returns zeros to this token — with HTTP 200, not 403.** Queried directly over the wg
tunnel as `felhom@pbs!demo-felhom`:
```
GET /api2/json/admin/datastore/felhom-offsite/status   → http=200
   {"data":{"avail":0,"backend-type":"filesystem","total":0,"used":0}}
GET /api2/json/status/datastore-usage                  → http=200
   {"data":[{"backend-type":"filesystem","mount-status":"nonremovable","store":"felhom-offsite"}]}   ← usage fields OMITTED
GET /api2/json/admin/datastore/felhom-offsite/snapshots?ns=demo-felhom → http=200, real data
```
The token is **namespace-scoped** — `GET .../namespace` returns exactly `[{"ns":"demo-felhom"}]`. It
holds `DatastoreBackup` on `/datastore/felhom-offsite/demo-felhom` (per the tenantsync `provision`
op's dual-grant), **not** `Datastore.Audit` on the datastore root. PBS answers the status call
rather than refusing it, and zeroes/omits the usage figures it will not disclose. PVE has no way to
distinguish "zero" from "not permitted" and prints 0/0/0.

**3. Ground truth from an independent path says the datastore is healthy.** The hub reads fill over
the ep0 SSH `usage` op (a literal `df` on the datastore path — `scripts/felhom-tenantsync.sh` v1.2.0,
read-only, no admin token):
```
2026/07/26 11:46:44 [INFO] PBS-DR box refreshed: 28.9% full (10.8 GB of 37.2 GB)
2026/07/26 12:02:42 [INFO] PBS-DR box refreshed: 28.9% full (10.8 GB of 37.2 GB)
```

**4. Writes demonstrably work with this exact token.** The 2026-07-18 snapshot is owned by
`felhom@pbs!demo-felhom`, is 9.74 GB, encrypted, and `verification.state == "ok"`.

**Conclusion: cosmetic.** Scheduling recurring writes here is safe on this ground. Two follow-ups
recorded, neither blocking: PVE's storage view will keep showing 0% for this storage (the operator
must read fill from the hub's PBS-DR gauge, not `pvesm status`), and granting `Datastore.Audit` at
the namespace would be the fix if the PVE-side number is ever wanted — a tenantsync `provision`
change, deliberately **not** made here.

---

## P0.3 — Capacity: 🛑 STOP → 🟢 **CLEARED 2026-07-27**

> **Resolution (2026-07-27).** The datastore was moved off ep0's 40 GB root disk onto a dedicated
> **100 GB Hetzner Cloud Volume**. Total **37.2 GB → 98 GB**; used **28.9 % → 13 %**; headroom to the
> 80 % warn **19 GB → ≈65 GB**. The "80 % at roughly the second additional customer" projection below
> becomes **roughly the seventh to thirteenth**. Method, verification and the restore round-trip that
> re-cleared the tier: `runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md`.
>
> **The dedup fact below is unchanged and still governs the slope** — per-tenant encryption still
> means no cross-customer dedup. The volume bought runway, not a better cost model. The weekly
> incremental size remains **unmeasured**.

**The analysis below is the original 2026-07-26 record, retained as written.**

### Measured

| Fact | Value | Source |
|---|---|---|
| `felhom-offsite` datastore total | **37.2 GB** | ep0 `df`, via the hub's usage op |
| Used now | **10.8 GB (28.9%)** | same, 2026-07-26 12:02 |
| 80% alert threshold | **29.8 GB** | hub PBS-DR box checker (`fill warn=80% crit=90%`) |
| **Headroom to the alert** | **19.0 GB** | |
| demo-felhom snapshot, logical | **9.74 GB** (`root.pxar.didx` 9.739 GB) | PBS API |
| demo-hp local vzdump, compressed | 1.48 GB | host-report |
| demo-felhom local vzdump, compressed | 5.69 GB | host-report |
| Snapshots in the datastore today | **1** (demo-felhom); demo-hp namespace exists, **0 snapshots** | PBS API |

The single 9.74 GB logical snapshot accounts for essentially all of the 10.8 GB used — first-snapshot
compression is ≈1:1 here, which is expected when the source is already-compressed Docker layers.

### The dedup fact that makes this worse

**Backups are client-side encrypted per tenant** (`encryption-key` in `storage.cfg`, one key per
customer). PBS derives chunk digests under the crypt key, so **chunks do not dedup across
customers.** Every customer's snapshots cost their full independent size. There is no fleet-scale
dedup dividend to lean on.

### Projection (weekly, `keep-last=3` = three weeks)

Weekly *incremental* cost is **not measured** — there has never been a second PBS snapshot of any
guest to measure it from. Bracketed at 5–20% of full per week and stated as a range rather than a
point estimate:

| Box | First snapshot | 3 retained weekly | Note |
|---|---|---|---|
| demo-felhom | 9.7 GB | **10.7 – 13.7 GB** | already on disk |
| demo-hp | ~2.5 GB | **2.8 – 3.5 GB** | scaled from local-vzdump ratio 5.69→9.74 (×1.71) |
| drill-r50 | ~1.5–3 GB | **1.7 – 4.0 GB** | never backed up; estimate only |
| **Fleet total** | | **≈ 15 – 21 GB (40–57%)** | comfortably under 80% |

The current three boxes fit. The problem is the next ones:

```
Steady state, 3 boxes:        ~18 GB   (48%)
80% alert:                     29.8 GB
Headroom:                      ~12 GB
Per additional customer:       ~5–10 GB  (first snapshot + 2 retained weekly increments,
                                          no cross-tenant dedup)
→ the 80% alert fires at roughly the SECOND additional customer.
```

### Why this is a STOP

The alpha horizon is *"the first remote tester"* (ROADMAP pre-invite checklist) and beyond — that is
squarely within one-to-two additional customers. The gate condition is met: **projected fill crosses
the 80% alert threshold within the alpha horizon.**

This is not a reason to abandon R-82 — it is a reason not to schedule recurring writes into a 37.2 GB
datastore without first ruling on which lever moves:

| Lever | Effect | Cost |
|---|---|---|
| **Grow the ep0 datastore** | the clean fix; 37.2 GB is small for fleet DR | Hetzner resize — money + an operator action |
| **Retention `keep-last=2`** weekly | ~1 snapshot/guest saved, ≈15% | two weeks of recovery depth instead of three |
| **Fortnightly PBS** | halves increment accrual | doubles the P0.1 exposure window to 14 days — would reopen P0.1 |
| **Exclude mp1 `/mnt/sys_drive`** from the snapshot | 1.8 GB used of 50 GB today | needs a ruling on what is on it |
| **Stage the rollout** | drill + demo-hp now, defer real customers | buys time, does not solve it |

**Recommendation (operator's call):** grow the datastore, and set PBS retention to `keep-last=2`
weekly for now. Together they roughly double the runway. But the sizing question deserves an explicit
answer before recurring writes start, because the failure mode — a DR datastore that silently
refuses writes at full — is exactly the class of "applied and empty" fault R-82 exists to fix.

---

## Not collected

| Item | Why | Needed access |
|---|---|---|
| Whether the ep0 datastore filesystem is **dedicated** or shares the root disk | no direct ep0 SSH; the hub's `usage` op returns only total/used/avail for the datastore path | root on ep0 (167.233.158.164) |
| Actual PBS **weekly incremental** size for these guests | no second snapshot has ever existed to diff against | resolved by Slice D's first weekly cycle — the projection should be re-measured then, not trusted |
| Other namespaces' usage in `felhom-offsite` | the PVE token is namespace-scoped (sees only `demo-felhom`) | PBS admin token |
| demo-hp / drill-r50 host-level state | no SSH key; break-glass not used, per standing scope | baked key or explicit break-glass authorisation |

---

## Observations

1. **drill-r50 has no offsite tier at all** (`offsite: null`) and no backups of any kind. It is the
   box the rollout plan puts *first*, which is right — but it is also the box where the P0.1 verdict
   does not hold, so its PBS cadence cannot simply inherit the fleet ruling.
2. **`last_success` is not a field** on the controller's offsite object — the shape is
   `last_run` + `last_status`. An early probe here asked for `last_success`, got `null`, and briefly
   looked like a failing daily tier. Recorded because it is the same shape as the R-80 false alarm:
   *absence of a field read as evidence of failure.* The real values are `last_status: "ok"` on both
   production boxes.
3. **The PVE storage view will permanently under-report this datastore at 0%.** Any future operator
   check of PBS fill must use the hub's PBS-DR gauge. Worth a line in the runbook when R-82 lands.
4. **`/mnt/sys_drive` (mp1, `backup=1`, 50 GB allocated, 1.8 GB used)** is inside the snapshot. Nobody
   asked for it to be; it is worth confirming during Slice D that its contents justify the space at
   weekly cadence.
5. `07-backup-architecture.md`'s tier table already carries a **PBS whole-guest** row saying
   *"bind-mounted drives out of reach"* — which the live `uncovered_volumes` confirms exactly. The
   doc is stale on versions but correct on this mechanism.