docs(ep0): relocate the felhom-offsite PBS datastore onto the 100 GB volume

Supervised runbook execution. No code change, no version bump.

felhom-offsite moved from ep0's 40 GB root disk (/srv/pbs-felhom) to a
dedicated 100 GB Hetzner Cloud Volume (/mnt/pbs-datastore, ext4 -m 0,
by-id fstab, relatime). Datastore NAME unchanged, so the PBS-DR
descriptors, per-box storage ids, ACLs and namespaces are untouched.

Capacity: 37.2 GB -> 98 GB total, 28.9% -> 13% used, headroom to the 80%
warn 19 GB -> ~65 GB. This CLEARS the R-82 Phase 0 P0.3 STOP. Per-tenant
encryption still precludes cross-customer dedup, so the slope is
unchanged - the volume buys runway, not a better cost model.

Verified: byte totals and chunk counts identical (9748), 7/7 snapshots
across all three namespaces, backup:backup ownership, clean itemised
dry-run, full verify job TASK OK with 0 errors, and a restore round-trip
(source_tier pbs, pass true, mount_parity ok, clean teardown).

Nothing deleted - the original 13 GB stays at /srv/pbs-felhom as the
rollback until a new weekly backup lands. GC deliberately not run.

Three findings recorded:
- the `scratch` datastore points at a non-existent path (pre-existing;
  now logs ENOENT every start) - operator decision
- the runbook's S6 guard test proves the wrong proposition:
  RequiresMountsFor re-mounts rather than refusing, so the test only
  bites when the device is genuinely unavailable (re-run that way, and
  the refusal was observed) - amendment recommended
- S11: storage box u629193 has no live backup path, BUT ep0 carries an
  enabled sshfs mount unit against it that must be removed before the
  box is deleted

Deviations: the volume arrived pre-formatted and mounted; S8 ran on
demo-felhom rather than demo-hp (no SSH key for demo-hp); the window was
contended by a stale in-memory 10-minute restore-test cadence whose
config had already been reverted on disk.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018ARoadHBf8rHoscfiqeVZn
This commit is contained in:
2026-07-27 09:36:11 +02:00
parent 5aca709ca1
commit ad8057c4e3
4 changed files with 573 additions and 1 deletions
@@ -342,6 +342,32 @@ backups/secondary/<stack>/
### 9.1 PBS whole-guest tier — measured capacity state (R-82 Phase 0, 2026-07-26)
> ### 🟢 SUPERSEDED 2026-07-27 — the datastore was grown; the P0.3 STOP is CLEARED
>
> The "grown later" in the paragraph below **has happened.** `felhom-offsite` was relocated off
> ep0's 40 GB root disk onto a dedicated **100 GB Hetzner Cloud Volume**
> (`scsi-0HC_Volume_106469259` → `/mnt/pbs-datastore`, ext4 `-m 0`). Evidence and method:
> `runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md`.
>
> | Fact | Before (2026-07-26) | After (2026-07-27) |
> |---|---|---|
> | Datastore total | 37.2 GB | **98 GB** |
> | Used | 10.8 GB (**28.9 %**) | 13 GB (**13 %**) |
> | 80 % warn threshold | 29.8 GB | **78.4 GB** |
> | Headroom to warn | 19.0 GB | **≈ 65 GB** |
> | Additional customers before warn (at ≈510 GB each) | **≈ 2** | **≈ 713** |
>
> **What this does NOT change:** point 1 below still holds in full — per-tenant encryption still
> precludes cross-customer dedup, so cost still scales linearly per customer. The volume buys
> runway, not a better cost model. Point 3 also still holds: `pvesm` still cannot see this
> datastore's fill.
>
> **Re-measure rather than trust this table** once a second weekly snapshot exists — the weekly
> incremental size is still **UNMEASURED**, and it, not the total, decides the real slope.
>
> ⚠️ The hub's PBS-DR gauge had **not** been re-read at the time of writing; if it still reports
> 37.2 GB it is reading the wrong filesystem, which is a real bug worth a roadmap item.
Recorded per the operator's 2026-07-26 ruling: **the datastore will be grown later; R-82 proceeds
meanwhile.** These are measurements, not projections-of-record — re-measure before relying on them.
Full method + evidence: `audits/SPIKE-r82-phase0-2026-07-26.md`.