felhom.eu/REPORT-ep0-datastore-volume-2026-07-27.md

# REPORT — ep0 PBS datastore relocated onto the 100 GB volume (2026-07-27)

**Class:** supervised operational run (RUNBOOK execution). **No code changed. No version bump.**
Written as `REPORT-<topic>.md` per the parallel-session rule — the shared `REPORT.md` was not touched.

**Full record with all evidence:** `documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md`

---

## Outcome: DONE and verified

`felhom-offsite` now lives on a dedicated 100 GB Hetzner Cloud Volume instead of ep0's 40 GB root disk.

| | Before | After |
|---|---|---|
| Path | `/srv/pbs-felhom` (root disk) | **`/mnt/pbs-datastore`** (volume) |
| Datastore total | 37.2 GB | **98 GB** (hub gauge: 97.9 GB) |
| Used | 28.9 % | **13 %** (hub gauge: 12.6 GB, 13 %) |
| Headroom to the 80 % warn | 19 GB | **≈65 GB** |
| Additional customers before warn | ≈2 | **≈7–13** |

Datastore **name unchanged** — the PBS-DR descriptors, per-box storage ids, ACLs and namespace
layout that R-39/R-82 made self-healing are untouched.

**Window:** 06:58 → 07:19 UTC (PBS down 07:00 → 07:17). **Nothing was deleted.**

### Acceptance evidence

| Gate | Result |
|---|---|
| Copy integrity | 13,242,207,822 = 13,242,207,822 B · **9,748 = 9,748 chunks** · 7 = 7 snapshots · `backup:backup` · itemised dry-run **0 lines** |
| Snapshot counts per ns | `demo-felhom` 2=2, `demo-felhom-01` 3=3, `demo-hp` 2=2 |
| atime semantics | `rw,relatime,discard` — **`relatime` present, `noatime` absent** (GC correctness) |
| Verify job | `TASK OK`, 3/3 groups, forced re-verification of every snapshot, **0 errors** |
| §6 mount guard | **refusal observed** — `Job … failed with result 'dependency'`; mountpoint stayed empty |
| §8 restore round-trip | `source_tier: pbs`, `pass: true`, `mount_parity: ok`, clean teardown, 12m1s |

---

## Three findings the operator should act on

1. **`scratch` datastore is configured at a path that does not exist** (`/srv/pbs-scratch`).
   Pre-existing, not caused here, but now logs `ENOENT` on every PBS start. This is the PRIME RISK
   shape ("reports fine, is not there") already live in the config. **Decision needed:** remove the
   stanza or create the directory.

2. **The runbook's §6 acceptance test proves the wrong proposition.** `RequiresMountsFor` is a
   mount-first ordering guarantee, not a refusal — systemd silently *re-mounts* an unmounted volume
   and PBS then starts safely. The test only bites when the device is genuinely unavailable, which
   is how it was re-run and passed. **Amendment recommended in the runbook record.**

3. **§11 — storage box `u629193` is NOT simply unused.** No live backup path references it (no
   datastore, no restic repo, no fstab, no `known_hosts` pin; R-17 already deleted `u629193-sub1`),
   **but ep0 carries an enabled, currently-mounted sshfs unit** `mnt-pbs\x2dstoragebox.mount` →
   `/mnt/pbs-storagebox`, holding spike leftovers. Disable and remove that unit before deleting the
   box, or ep0 logs a failed mount every boot. **The deletion is the operator's console click.**

## Deviations from the runbook as written

- **The volume arrived already formatted and mounted** by Hetzner at `/mnt/HC_Volume_106469259`
  (§2 assumed neither). Operator ruled: reformat + repath. The 5 % reserve was reclaimed (`-m 0`).
- **§8 ran on demo-felhom, not demo-hp** — DooPlex holds no SSH key for demo-hp (the G1 gap). Same
  tier, same relocated datastore, larger archive.
- **The window was contended** by a stale 10-minute restore-test cadence on demo-felhom: the config
  had already been reverted to 3.5 days on disk, but the cadence is read once at daemon start and
  `NRestarts=0`. Restarting the agent applied it (`cadence=84h0m0s`). The in-flight test was allowed
  to finish rather than aborted.

## Process errors made during this run (recorded deliberately)

- `rsync -aHAX` **OOM-killed** ep0 (3.7 GB RAM, no swap). Cause: a PBS `.chunks/` tree pre-creates
  all 65536 shard dirs → 75,341 inodes, and `-H` retains the whole inode map. `-H` was dropped only
  after **proving** no hardlinks exist (`-links +1` → 0; max link count → 1); PBS references chunks
  by digest, never by hardlink.
- `/usr/bin/time -v` is not installed on ep0 → exit 127, rsync never ran, and a `| grep … || true`
  wrapper swallowed it while printing a success-looking line.
- `rsync --version | head -1` reported a working rsync 3.4.1 as "missing" — **the §12 pipe-into-head
  trap, fourth recorded instance in this project.** Both fixed by capturing the command's own `$?`.

## Deferred

1. **Old copy retained** at `/srv/pbs-felhom` (13 GB, 9,748 chunks) as the rollback. Rollback is a
   two-line `datastore.cfg` revert. Reclaim only after a new weekly offsite backup lands on the
   volume, with explicit go-ahead.
2. **GC not run** — now unblocked by the round-trip, but left for a separate deliberate run. No GC
   schedule is configured on this PBS at all.
3. ~~Hub PBS-DR capacity gauge not re-read.~~ **CLOSED — verified correct.** The hub operator UI
   (Offsite → PBS DR) reports `felhom-offsite (ep0)` at **97.9 GB capacity, 12.6 GB used, 13 % full**,
   agreeing with the on-box `df`. The gauge follows the datastore's configured path, so the move
   needed no hub-side change and the suspected "wrong filesystem" bug does not exist.
4. **ep0 has no swap** (temporary 4 GB file removed; box left as found). Worth a small permanent
   swapfile — outside this runbook's scope.