felhom.eu/documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md

RUNBOOK RECORD — ep0 PBS datastore relocated onto the 100 GB volume (2026-07-27)

Class: supervised operational run. No repo version bump — the only commits are this record and the capacity note. Nothing was deleted.

Host: ep0 / felhom-hetzner, 167.233.158.164, Hetzner CX33 (4 vCPU / 8 GB RAM), Nuremberg.

Rescaled 2026-08-03 from the CX23 (2 vCPU / 3.8 GB) this runbook was written against. The 40 GB local disk did NOT change — this was a CPU/RAM resize — so every disk figure below still stands. The 4 GiB swapfile added on 2026-07-27 survived the resize. Datastore moved: felhom-offsite, /srv/pbs-felhom → /mnt/pbs-datastore (name unchanged). Window: 06:58 → 07:19 UTC. PBS down 07:00 → 07:17 UTC.

---

1. Phase 0 — ground truth

Established read-only before anything was touched. Three of the runbook's premises did not hold; each is recorded below.

Item	Value
PBS	proxmox-backup-server 4.2.3-1, running version 4.2.2
Datastore name / path	felhom-offsite → /srv/pbs-felhom
Size on disk	13,242,207,822 B (13 GB)
Chunk count	9,748
Total entries	75,341 (reg 9,785, dir 65,556 — PBS pre-creates all 65536 .chunks shards)
Snapshots	7 — demo-felhom 2, demo-felhom-01 3, demo-hp 2
Ownership	backup:backup
Root filesystem	/dev/sda1 ext4, 38 G, 16 G used / 21 G avail (43 %), rw,relatime,errors=remount-ro
Volume	/dev/disk/by-id/scsi-0HC_Volume_106469259 → /dev/sdb, 100 G — present
Scheduled jobs	none — no verification.cfg, no prune.cfg, no sync.cfg, no GC schedule
Last weekly backups	2026-07-26 (demo-felhom 12:21, demo-hp 15:42 + 17:03); next ≈ 08-02 — window clear

Deviation ① — a second datastore is configured whose path does not exist

datastore: scratch          → /srv/pbs-scratch    ← No such file or directory
datastore: felhom-offsite   → /srv/pbs-felhom     ← the real one

proxmox-backup-manager datastore list reports scratch without complaint. /srv/ contains only pbs-felhom. This is the PRIME RISK failure mode already live in the config — a datastore that reports fine and is not there. Pre-existing, untouched by this run, and now louder: since the relocation PBS polls filesystem info for both datastores and logs on every start:

read fs info on "/srv/pbs-scratch" failed - ENOENT: No such file or directory
find_mounted_device failed - ENOENT: No such file or directory

Open decision for the operator: remove the stanza, or create the directory. Not actioned here.

Deviation ② — the volume was already formatted and already mounted

Hetzner's provisioning had done both. /etc/fstab already carried:

/dev/disk/by-id/scsi-0HC_Volume_106469259 /mnt/HC_Volume_106469259 ext4 discard,nofail,defaults 0 0

Effective options rw,relatime,discard — relatime present, noatime absent, and by-id already in use, so those two traps were clear on arrival. The filesystem was empty (lost+found only) but carried the default 5 % reserve: 98 G size, 93 G avail.

§2 as written would have run mkfs.ext4 against a mounted filesystem and failed. Operator ruling: reformat and repath per the runbook (the volume was provably empty), reclaiming the 5 GB and aligning the mount path with the §6 guard.

Deviation ③ — the window was contended by a restore-test loop

ep0 showed reader tasks pulling chunks every ~10–20 minutes, each running 10–11 minutes. Root cause was not on ep0 but on demo-felhom:

07:54:06 CEST  felhom-agent restarted  →  scheduler started with cadence=10m0s
08:08   CEST  /etc/felhom-agent/agent.json edited → restore_test_cadence_seconds = 302400 (3.5 d)
              NRestarts=0 — the running process never picked up the revert

The cadence is read once at daemon start. The config was already correct on disk; only the process was stale — the same in-memory-vs-persisted shape as the controller's report floor. The agent was restarted (08:50:16 CEST, cadence=84h0m0s confirmed in the journal), after which the loop stopped and the window was uncontended.

The in-flight restore-test was allowed to finish cleanly (06:44:14 → 06:58:01 OK) rather than being aborted; §1's "do not proceed while a task is running" was held.

---

2. The volume (§2)

fstab line used (replacing Hetzner's; discard retained as appropriate for a Hetzner Cloud Volume):

/dev/disk/by-id/scsi-0HC_Volume_106469259 /mnt/pbs-datastore ext4 defaults,relatime,discard,nofail,x-systemd.device-timeout=30s 0 0

mkfs.ext4 -F -m 0 -L pbs-datastore — guarded by a check that the filesystem held nothing but lost+found before unmounting.

Proof of mount and atime semantics:

$ findmnt /mnt/pbs-datastore
TARGET             SOURCE   FSTYPE OPTIONS
/mnt/pbs-datastore /dev/sdb ext4   rw,relatime,discard

options: rw,relatime,discard
OK: relatime present, noatime absent

df after mkfs: 98 G size, 98 G avail (was 93 G — the 5 GB root reserve reclaimed).

---

3. The copy (§4) — and the two failures on the way

First attempt OOM-killed at 52 %.

oom-kill: … task=rsync,pid=605913
Out of memory: Killed process 605913 (rsync) total-vm:3464728kB, anon-rss:1912224kB

ep0 has 3.7 GB RAM and no swap. A PBS .chunks/ tree pre-creates all 65536 shard directories, so even this 13 GB store is 75,341 inodes; -H makes rsync retain the whole inode map to detect hardlinks, and that is what exhausted memory.

-H was dropped only after proving it was unnecessary — the runbook is right that it is unrecoverable if you guess:

$ find /srv/pbs-felhom -type f -links +1 | wc -l      → 0
$ find /srv/pbs-felhom -type f -printf '%n\n' | sort -rn | head -1  → 1

A PBS chunk store contains no hardlinks — snapshots reference chunks by digest inside .didx/.fidx index files, never by link count. A temporary 4 GB swapfile was added as insurance.

Second attempt failed silently — operator-visible process error, recorded deliberately. /usr/bin/time -v is not installed on ep0 → exit 127, rsync never ran, and the wrapper … | grep … || true swallowed it while printing a success-looking line. A follow-up guard then hit the §12 trap directly — rsync --version | head -1 returns head's status, not rsync's, so a working rsync 3.4.1 was reported as "ABORT: rsync missing". This is the fourth recorded instance of that trap in this project. Both were fixed by capturing the command's own $? with no pipeline in between.

Successful copy (rsync -aAX --numeric-ids --stats, trailing slash on source, no --delete, no -H), 07:12:48 → 07:14:16 UTC, exit 0:

Number of files: 75,341 (reg: 9,785, dir: 65,556)
Number of created files: 8,697
Number of deleted files: 0
Total file size: 13,242,207,822 bytes
sent 11,803,880,151 bytes  received 394,094 bytes  134,905,991.37 bytes/sec

Copy verification — all gates passed

src bytes = 13242207822        dst bytes = 13242207822
src chunks = 9748              dst chunks = 9748
  demo-felhom      src=2 dst=2  OK
  demo-felhom-01   src=3 dst=3  OK
  demo-hp          src=2 dst=2  OK
total snapshots src=7 dst=7

chunks owner        = backup:backup
datastore root owner = backup:backup
.lock present       = -rw-r--r-- 1 backup backup 0

dry-run exit = 0
itemised lines = 0
OK: dry-run clean

---

4. The repoint (§5)

datastore.cfg backed up to datastore.cfg.pre-volume.bak. Exactly one line changed, asserted programmatically to be the felhom-offsite stanza's path and no other:

5c5
< 	path /srv/pbs-felhom
---
> 	path /mnt/pbs-datastore

The datastore name is unchanged and the scratch stanza was not touched.

---

5. The mount guard (§6) — and a correction to the runbook's acceptance test

Drop-ins written for both services:

/etc/systemd/system/proxmox-backup.service.d/10-datastore-mount.conf
/etc/systemd/system/proxmox-backup-proxy.service.d/10-datastore-mount.conf
  [Unit]
  RequiresMountsFor=/mnt/pbs-datastore

The runbook's test as written proves the wrong proposition

§6 says PBS "MUST FAIL" to start with the volume unmounted. It does not — and that is correct behaviour, not a broken guard. RequiresMountsFor is a mount-first ordering guarantee, not a refusal: systemd pulls the mount unit in as a dependency and starts it. Observed:

07:16:10  mnt-pbs\x2ddatastore.mount: Deactivated successfully   ← the test's umount
07:16:11  Mounting mnt-pbs\x2ddatastore.mount...                 ← systemd re-mounted it
07:16:11  Mounted  mnt-pbs\x2ddatastore.mount

PBS then started with the volume correctly mounted. It never saw an unmounted path. Satisfying the dependency by mounting is stronger than refusing.

The corrected test — volume genuinely unavailable

The real hazard is not "unmounted but mountable", it is "volume gone". Simulated by pointing the fstab entry at a non-existent device and attempting a start:

$ systemctl start proxmox-backup
A dependency job for proxmox-backup.service failed.
systemctl start proxmox-backup exit code: 1
is-active: inactive

Jul 27 07:17:43 Dependency failed for proxmox-backup.service - Proxmox Backup API Server.
Jul 27 07:17:43 proxmox-backup.service: Job proxmox-backup.service/start failed with result 'dependency'.

And the bare mountpoint stayed empty — PBS wrote nothing into the root disk underneath the mount, which is the precise PRIME RISK outcome the guard exists to prevent:

$ ls -la /mnt/pbs-datastore
total 8
drwxr-xr-x 2 root root 4096 Jul 27 06:58 .
drwxr-xr-x 4 root root 4096 Jul 27 06:58 ..

fstab was then restored and the real volume remounted. A guard that has not been observed refusing is not a guard — this one has now been observed refusing.

Recommended amendment to the runbook: §6's acceptance test should unmount and make the device unavailable; an unmount alone will always be silently repaired by RequiresMountsFor.

---

6. Bring-up and content verification (§7)

is-active: active / active

+================+====================+=========+
| name           | path               | comment |
+================+====================+=========+
| felhom-offsite | /mnt/pbs-datastore |         |
+----------------+--------------------+---------+
| scratch        | /srv/pbs-scratch   |         |
+================+====================+=========+

/dev/sdb  98G  13G  86G  13%  /mnt/pbs-datastore

Snapshot counts, before vs after — they match exactly:

Namespace	Phase 0	After the move
demo-felhom	2	2
demo-felhom-01	3	3
demo-hp	2	2
total	7	7

Verify job — full datastore, forced re-verification

Run with --ignore-verified false so every snapshot was re-read rather than trusting existing verification state. This reads and digest-checks all 9,748 chunks:

verify datastore felhom-offsite
found 3 groups
…
percentage done: 100.00% (3/3 groups)
TASK OK

0 errors on every snapshot. Read throughput off the new volume ~250 MiB/s.

---

7. Restore round-trip (§8) — PASSED

A tier is only real once a restore round-trip is proven. The datastore moved to different physical storage, so the bar was re-cleared.

Deviation from the runbook: run on demo-felhom, not demo-hp. §8 specifies demo-hp because its archive is smaller, but DooPlex holds no SSH key for demo-hp (ssh demo-hp → Permission denied (publickey) — the known G1 gap). demo-felhom was used instead: same tier, same relocated datastore, larger archive (~12 min instead of ~3). The acceptance criteria are unchanged; only the wall-clock differs.

Triggered on demand via the agent's own harness rather than waiting for a tick (the cadence is now 3.5 days):

sudo -u felhom-agent /usr/local/bin/felhom-agent \
  --config /etc/felhom-agent/agent.json \
  --selftest=restore-test \
  --archive "felhom-pbs:backup/ct/9201/2026-07-26T12:21:48Z"

Result — 09:20:41 → 09:32:41 CEST:

{
  "source_archive": "felhom-pbs:backup/ct/9201/2026-07-26T12:21:48Z",
  "source_tier": "pbs",
  "scratch_vmid": 990000,
  "pass": true,
  "verified": "boot+running",
  "tested_at": "2026-07-27T07:32:41Z",
  "duration_seconds": 720.710414828,
  "mount_parity": "ok",
  "mount_inventory": [
    "mp0=/var/lib/docker (200G)",
    "mp1=/mnt/sys_drive (50G)",
    "mp8=/mnt/felhom-drives (throwaway for the archived bind)",
    "mp9=/etc/felhom-bootstrap (throwaway for the archived bind)"
  ]
}
=== selftest=restore-test OK (scratch 990000 restored+booted+verified+torn-down in 12m1s) ===

Required evidence	Result
source_tier	pbs ✓
pass	true ✓
mount_parity	ok ✓
clean teardown	✓ — restore-test: scratch guest torn down vmid=990000, gate class=guest_destroy disposition=benign allowed=true

Scratch band confirmed clean afterwards: no 9900xx guests, no 990000 LVs.

A full restore → boot → verify → destroy cycle succeeded against the relocated chunk store. The pre-run recover pass reported examined=0 scratch_destroyed=0 scratch_clean=0, i.e. the earlier interrupted test had already been reaped and did not contaminate this run.

---

8. The old copy is still in place (§10 rollback)

Nothing was deleted.

drwxr-xr-x 4 backup backup 4096 Jul  3 21:48 /srv/pbs-felhom
13G     /srv/pbs-felhom          (13,242,207,822 bytes)
old chunks: 9748

Root filesystem is back to 16 G used / 21 G avail (43 %) — unchanged from Phase 0. It peaked at 54 % during the run (retained 13 GB original + the temporary 4 GB swapfile); the swapfile has since been removed, leaving ep0 with no swap exactly as it was found.

Rollback remains a two-line config revert:

systemctl stop proxmox-backup-proxy proxmox-backup
cp -a /etc/proxmox-backup/datastore.cfg.pre-volume.bak /etc/proxmox-backup/datastore.cfg
systemctl start proxmox-backup proxmox-backup-proxy

Reclaiming the old copy is a separate operator decision (§10), gated on at least one new weekly offsite backup landing on the volume.

---

9. §11 — the unused storage box u629193

Finding: not referenced by any live backup path, but ep0 carries a live mount against it.

No live reference:

• no PBS datastore points at it (both stanzas are /srv/… / /mnt/pbs-datastore)
• no restic repo — the live offsite target is storage-box-pool-1 (u629488), a separate project
• no fstab entry, and no known_hosts pin (/root/.ssh/known_hosts holds 0 storagebox lines)
• ROADMAP R-17 already closed 2026-07-22 — sub-account u629193-sub1 deleted at the console
• in-repo hits are documentation plus two test fixtures only: hub/internal/hetznerapi/fake.go (generates the u629193-sub%d pattern) and felhom-controller/controller/internal/report/dr_recipe_test.go (string literal)

But there IS a live mount on ep0:

/etc/systemd/system/mnt-pbs\x2dstoragebox.mount   enabled, active
  What=u629193@u629193.your-storagebox.de:/home  →  /mnt/pbs-storagebox
  IdentityFile=/root/.ssh/storagebox_ed25519
  Options=port=23,idmap=user,uid=34,gid=34,…,StrictHostKeyChecking=yes,_netdev

Mounted now, holding spike leftovers felhom-demo/, felhom-peti-spike/, spike-sub/, .ssh/. lsof +D shows nothing holding files open.

Before the box is deleted, disable and remove that unit, or ep0 logs a failed mount on every boot:

systemctl disable --now 'mnt-pbs\x2dstoragebox.mount'
rm /etc/systemd/system/'mnt-pbs\x2dstoragebox.mount'
systemctl daemon-reload
rm -f /root/.ssh/storagebox_ed25519 /root/.ssh/storagebox_ed25519.pub   # access key, regenerable

The deletion itself is the operator's console click — not performed here.

---

10. Deferred / open

1. scratch datastore stanza points at a non-existent path and now logs ENOENT on every PBS start. Remove the stanza or create the directory — operator decision.
2. Old copy at /srv/pbs-felhom retained (13 GB). Reclaim only after a new weekly offsite backup lands on the volume, with explicit go-ahead (§10.1).
3. GC has still not been run. The round-trip has now passed, so §10.2 is unblocked — but GC is the one operation that deletes, and it is deliberately left for a separate, deliberate run. No GC schedule is configured on this PBS at all, which is worth its own decision.
4. ep0 has no swap (removed the temporary 4 GB file; the box is as found). A 3.7 GB box with zero swap is what turned a routine 13 GB copy into an OOM kill. Worth considering a small permanent swapfile — not done here, as it is outside this runbook's scope.
5. Hub PBS-DR capacity gauge (§10.3) not yet re-read. CLOSED — verified correct 2026-07-27 via the hub operator UI (Offsite → PBS DR): felhom-offsite (ep0) — capacity 97.9 GB, used 12.6 GB, 13 % full, polled 12 min prior. The gauge follows the datastore's configured path, so the relocation required no hub-side change. §10.3's "if it still shows the old size that is a real bug" does not apply — there is no bug. (It could not be queried from DooPlex directly — the hub is a ClusterIP service — so this was read from the operator UI.)
6. x-systemd.device-timeout=30s means a genuinely missing volume delays boot by 30 s before nofail lets boot continue. Accepted trade-off.

11. Observations

• The atime note in §4 holds: the copy gave every chunk a fresh access time. This is safe in the conservative direction — GC only reclaims chunks whose atime predates the cutoff, so fresh times mean nothing is reclaimed early. Not a problem; do not "fix" it.
• -m 0 mattered more than it looks: 5 GB withheld on a 100 GB volume is 5 % of the headroom this whole exercise was about.
• The relocation does not change the per-tenant-encryption growth story. The volume buys headroom (13 GB / 98 GB = 13 %, vs 13 GB / 37 GB = 35 %); it does not introduce cross-customer dedup, which per-tenant encryption still precludes.