felhom.eu/REPORT-ep0-datastore-volume-2026-07-27.md

REPORT — ep0 PBS datastore relocated onto the 100 GB volume (2026-07-27)

Class: supervised operational run (RUNBOOK execution). No code changed. No version bump. Written as REPORT-<topic>.md per the parallel-session rule — the shared REPORT.md was not touched.

Full record with all evidence: documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md

---

Outcome: DONE and verified

felhom-offsite now lives on a dedicated 100 GB Hetzner Cloud Volume instead of ep0's 40 GB root disk.

	Before	After
Path	/srv/pbs-felhom (root disk)	/mnt/pbs-datastore (volume)
Datastore total	37.2 GB	98 GB (hub gauge: 97.9 GB)
Used	28.9 %	13 % (hub gauge: 12.6 GB, 13 %)
Headroom to the 80 % warn	19 GB	≈65 GB
Additional customers before warn	≈2	≈7–13

Datastore name unchanged — the PBS-DR descriptors, per-box storage ids, ACLs and namespace layout that R-39/R-82 made self-healing are untouched.

Window: 06:58 → 07:19 UTC (PBS down 07:00 → 07:17). Nothing was deleted.

Acceptance evidence

Gate	Result
Copy integrity	13,242,207,822 = 13,242,207,822 B · 9,748 = 9,748 chunks · 7 = 7 snapshots · backup:backup · itemised dry-run 0 lines
Snapshot counts per ns	demo-felhom 2=2, demo-felhom-01 3=3, demo-hp 2=2
atime semantics	rw,relatime,discard — relatime present, noatime absent (GC correctness)
Verify job	TASK OK, 3/3 groups, forced re-verification of every snapshot, 0 errors
§6 mount guard	refusal observed — Job … failed with result 'dependency'; mountpoint stayed empty
§8 restore round-trip	source_tier: pbs, pass: true, mount_parity: ok, clean teardown, 12m1s

---

Three findings the operator should act on

1. scratch datastore is configured at a path that does not exist (/srv/pbs-scratch). Pre-existing, not caused here, but now logs ENOENT on every PBS start. This is the PRIME RISK shape ("reports fine, is not there") already live in the config. Decision needed: remove the stanza or create the directory.

2. The runbook's §6 acceptance test proves the wrong proposition. RequiresMountsFor is a mount-first ordering guarantee, not a refusal — systemd silently re-mounts an unmounted volume and PBS then starts safely. The test only bites when the device is genuinely unavailable, which is how it was re-run and passed. Amendment recommended in the runbook record.

3. §11 — storage box u629193 is NOT simply unused. No live backup path references it (no datastore, no restic repo, no fstab, no known_hosts pin; R-17 already deleted u629193-sub1), but ep0 carries an enabled, currently-mounted sshfs unit mnt-pbs\x2dstoragebox.mount → /mnt/pbs-storagebox, holding spike leftovers. Disable and remove that unit before deleting the box, or ep0 logs a failed mount every boot. The deletion is the operator's console click.

Deviations from the runbook as written

• The volume arrived already formatted and mounted by Hetzner at /mnt/HC_Volume_106469259 (§2 assumed neither). Operator ruled: reformat + repath. The 5 % reserve was reclaimed (-m 0).
• §8 ran on demo-felhom, not demo-hp — DooPlex holds no SSH key for demo-hp (the G1 gap). Same tier, same relocated datastore, larger archive.
• The window was contended by a stale 10-minute restore-test cadence on demo-felhom: the config had already been reverted to 3.5 days on disk, but the cadence is read once at daemon start and NRestarts=0. Restarting the agent applied it (cadence=84h0m0s). The in-flight test was allowed to finish rather than aborted.

Process errors made during this run (recorded deliberately)

• rsync -aHAX OOM-killed ep0 (3.7 GB RAM, no swap). Cause: a PBS .chunks/ tree pre-creates all 65536 shard dirs → 75,341 inodes, and -H retains the whole inode map. -H was dropped only after proving no hardlinks exist (-links +1 → 0; max link count → 1); PBS references chunks by digest, never by hardlink.
• /usr/bin/time -v is not installed on ep0 → exit 127, rsync never ran, and a | grep … || true wrapper swallowed it while printing a success-looking line.
• rsync --version | head -1 reported a working rsync 3.4.1 as "missing" — the §12 pipe-into-head trap, fourth recorded instance in this project. Both fixed by capturing the command's own $?.

Deferred

1. Old copy retained at /srv/pbs-felhom (13 GB, 9,748 chunks) as the rollback. Rollback is a two-line datastore.cfg revert. Reclaim only after a new weekly offsite backup lands on the volume, with explicit go-ahead.
2. GC not run — now unblocked by the round-trip, but left for a separate deliberate run. No GC schedule is configured on this PBS at all.
3. Hub PBS-DR capacity gauge not re-read. CLOSED — verified correct. The hub operator UI (Offsite → PBS DR) reports felhom-offsite (ep0) at 97.9 GB capacity, 12.6 GB used, 13 % full, agreeing with the on-box df. The gauge follows the datastore's configured path, so the move needed no hub-side change and the suspected "wrong filesystem" bug does not exist.
4. ep0 has no swap (temporary 4 GB file removed; box left as found). Worth a small permanent swapfile — outside this runbook's scope.