@@ -0,0 +1,444 @@
# RUNBOOK RECORD — ep0 PBS datastore relocated onto the 100 GB volume (2026-07-27)
**Class: ** supervised operational run. **No repo version bump ** — the only commits are this record
and the capacity note. **Nothing was deleted. **
**Host: ** `ep0` / `felhom-hetzner` , `167.233.158.164` , Hetzner CX23, Nuremberg.
**Datastore moved: ** `felhom-offsite` , `/srv/pbs-felhom` → * * `/mnt/pbs-datastore` ** (name unchanged).
**Window: ** 06:58 → 07:19 UTC. PBS down 07:00 → 07:17 UTC.
---
## 1. Phase 0 — ground truth
Established read-only before anything was touched. **Three of the runbook's premises did not hold ** ;
each is recorded below.
| Item | Value |
|---|---|
| PBS | `proxmox-backup-server 4.2.3-1` , running version **4.2.2 ** |
| Datastore name / path | * * `felhom-offsite` ** → `/srv/pbs-felhom` |
| Size on disk | **13,242,207,822 B (13 GB) ** |
| Chunk count | **9,748 ** |
| Total entries | 75,341 (reg 9,785, dir 65,556 — PBS pre-creates all 65536 `.chunks` shards) |
| Snapshots | **7 ** — `demo-felhom` 2, `demo-felhom-01` 3, `demo-hp` 2 |
| Ownership | `backup:backup` |
| Root filesystem | `/dev/sda1` ext4, 38 G, **16 G used / 21 G avail (43 %) ** , `rw,relatime,errors=remount-ro` |
| Volume | `/dev/disk/by-id/scsi-0HC_Volume_106469259` → `/dev/sdb` , 100 G — present |
| Scheduled jobs | **none ** — no `verification.cfg` , no `prune.cfg` , no `sync.cfg` , no GC schedule |
| Last weekly backups | 2026-07-26 (demo-felhom 12:21, demo-hp 15:42 + 17:03); next ≈ 08-02 — window clear |
### Deviation ① — a second datastore is configured whose path does not exist
```
datastore: scratch → /srv/pbs-scratch ← No such file or directory
datastore: felhom-offsite → /srv/pbs-felhom ← the real one
```
`proxmox-backup-manager datastore list` reports `scratch` without complaint. `/srv/` contains only
`pbs-felhom` . **This is the PRIME RISK failure mode already live in the config ** — a datastore that
reports fine and is not there. Pre-existing, untouched by this run, and now * louder * : since the
relocation PBS polls filesystem info for both datastores and logs on every start:
```
read fs info on "/srv/pbs-scratch" failed - ENOENT: No such file or directory
find_mounted_device failed - ENOENT: No such file or directory
```
**Open decision for the operator: ** remove the stanza, or create the directory. Not actioned here.
### Deviation ② — the volume was already formatted and already mounted
Hetzner's provisioning had done both. `/etc/fstab` already carried:
```
/dev/disk/by-id/scsi-0HC_Volume_106469259 /mnt/HC_Volume_106469259 ext4 discard,nofail,defaults 0 0
```
Effective options `rw,relatime,discard` — `relatime` present, `noatime` absent, and `by-id` already
in use, so those two traps were clear on arrival. The filesystem was empty (`lost+found` only) but
carried the default 5 % reserve: 98 G size, **93 G avail ** .
§2 as written would have run `mkfs.ext4` against a mounted filesystem and failed. **Operator ruling:
reformat and repath per the runbook** (the volume was provably empty), reclaiming the 5 GB and
aligning the mount path with the §6 guard.
### Deviation ③ — the window was contended by a restore-test loop
ep0 showed `reader` tasks pulling chunks every ~10– –
cause was **not ** on ep0 but on demo-felhom:
```
07:54:06 CEST felhom-agent restarted → scheduler started with cadence=10m0s
08:08 CEST /etc/felhom-agent/agent.json edited → restore_test_cadence_seconds = 302400 (3.5 d)
NRestarts=0 — the running process never picked up the revert
```
**The cadence is read once at daemon start. ** The config was already correct on disk; only the
process was stale — the same in-memory-vs-persisted shape as the controller's report floor. The
agent was restarted (08:50:16 CEST, `cadence=84h0m0s` confirmed in the journal), after which the
loop stopped and the window was uncontended.
The in-flight restore-test was allowed to **finish cleanly ** (`06:44:14 → 06:58:01 OK` ) rather than
being aborted; §1's "do not proceed while a task is running" was held.
---
## 2. The volume (§2)
fstab line used (replacing Hetzner's; `discard` retained as appropriate for a Hetzner Cloud Volume):
```
/dev/disk/by-id/scsi-0HC_Volume_106469259 /mnt/pbs-datastore ext4 defaults,relatime,discard,nofail,x-systemd.device-timeout=30s 0 0
```
`mkfs.ext4 -F -m 0 -L pbs-datastore` — guarded by a check that the filesystem held nothing but
`lost+found` before unmounting.
**Proof of mount and atime semantics: **
```
$ findmnt /mnt/pbs-datastore
TARGET SOURCE FSTYPE OPTIONS
/mnt/pbs-datastore /dev/sdb ext4 rw,relatime,discard
options: rw,relatime,discard
OK: relatime present, noatime absent
```
`df` after mkfs: **98 G size, 98 G avail ** (was 93 G — the 5 GB root reserve reclaimed).
---
## 3. The copy (§4) — and the two failures on the way
**First attempt OOM-killed at 52 %. **
```
oom-kill: … task=rsync,pid=605913
Out of memory: Killed process 605913 (rsync) total-vm:3464728kB, anon-rss:1912224kB
```
ep0 has 3.7 GB RAM and **no swap ** . A PBS `.chunks/` tree pre-creates **all 65536 shard
directories**, so even this 13 GB store is 75,341 inodes; `-H` makes rsync retain the whole inode
map to detect hardlinks, and that is what exhausted memory.
* * `-H` was dropped only after proving it was unnecessary** — the runbook is right that it is
unrecoverable if you guess:
```
$ find /srv/pbs-felhom -type f -links +1 | wc -l → 0
$ find /srv/pbs-felhom -type f -printf '%n\n' | sort -rn | head -1 → 1
```
A PBS chunk store contains no hardlinks — snapshots reference chunks by digest inside `.didx` /`.fidx`
index files, never by link count. A temporary 4 GB swapfile was added as insurance.
**Second attempt failed silently — operator-visible process error, recorded deliberately. **
`/usr/bin/time -v` is **not installed ** on ep0 → exit 127, rsync never ran, and the wrapper
`… | grep … || true` swallowed it while printing a success-looking line. A follow-up guard then hit
the §12 trap directly — `rsync --version | head -1` returns * head's * status, not rsync's, so a
working rsync 3.4.1 was reported as "ABORT: rsync missing". **This is the fourth recorded instance
of that trap in this project.** Both were fixed by capturing the command's own `$?` with no
pipeline in between.
**Successful copy ** (`rsync -aAX --numeric-ids --stats` , trailing slash on source, no `--delete` ,
no `-H` ), 07:12:48 → 07:14:16 UTC, exit **0 ** :
```
Number of files: 75,341 (reg: 9,785, dir: 65,556)
Number of created files: 8,697
Number of deleted files: 0
Total file size: 13,242,207,822 bytes
sent 11,803,880,151 bytes received 394,094 bytes 134,905,991.37 bytes/sec
```
### Copy verification — all gates passed
```
src bytes = 13242207822 dst bytes = 13242207822
src chunks = 9748 dst chunks = 9748
demo-felhom src=2 dst=2 OK
demo-felhom-01 src=3 dst=3 OK
demo-hp src=2 dst=2 OK
total snapshots src=7 dst=7
chunks owner = backup:backup
datastore root owner = backup:backup
.lock present = -rw-r--r-- 1 backup backup 0
dry-run exit = 0
itemised lines = 0
OK: dry-run clean
```
---
## 4. The repoint (§5)
`datastore.cfg` backed up to `datastore.cfg.pre-volume.bak` . Exactly one line changed, asserted
programmatically to be the `felhom-offsite` stanza's `path` and no other:
``` diff
5c5
< path /srv/pbs-felhom
---
> path /mnt/pbs-datastore
```
**The datastore name is unchanged ** and the `scratch` stanza was not touched.
---
## 5. The mount guard (§6) — and a correction to the runbook's acceptance test
Drop-ins written for **both ** services:
```
/etc/systemd/system/proxmox-backup.service.d/10-datastore-mount.conf
/etc/systemd/system/proxmox-backup-proxy.service.d/10-datastore-mount.conf
[Unit]
RequiresMountsFor=/mnt/pbs-datastore
```
### The runbook's test as written proves the wrong proposition
§6 says PBS "MUST FAIL" to start with the volume unmounted. It does not — and that is correct
behaviour, not a broken guard. `RequiresMountsFor` is a **mount-first ordering guarantee ** , not a
refusal: systemd pulls the mount unit in as a dependency and * starts * it. Observed:
```
07:16:10 mnt-pbs\x2ddatastore.mount: Deactivated successfully ← the test's umount
07:16:11 Mounting mnt-pbs\x2ddatastore.mount... ← systemd re-mounted it
07:16:11 Mounted mnt-pbs\x2ddatastore.mount
```
PBS then started **with the volume correctly mounted ** . It never saw an unmounted path. Satisfying
the dependency by mounting is stronger than refusing.
### The corrected test — volume genuinely unavailable
The real hazard is not "unmounted but mountable", it is "volume gone". Simulated by pointing the
fstab entry at a non-existent device and attempting a start:
```
$ systemctl start proxmox-backup
A dependency job for proxmox-backup.service failed.
systemctl start proxmox-backup exit code: 1
is-active: inactive
Jul 27 07:17:43 Dependency failed for proxmox-backup.service - Proxmox Backup API Server.
Jul 27 07:17:43 proxmox-backup.service: Job proxmox-backup.service/start failed with result 'dependency'.
```
**And the bare mountpoint stayed empty ** — PBS wrote nothing into the root disk underneath the
mount, which is the precise PRIME RISK outcome the guard exists to prevent:
```
$ ls -la /mnt/pbs-datastore
total 8
drwxr-xr-x 2 root root 4096 Jul 27 06:58 .
drwxr-xr-x 4 root root 4096 Jul 27 06:58 ..
```
fstab was then restored and the real volume remounted. **A guard that has not been observed refusing
is not a guard — this one has now been observed refusing.**
> **Recommended amendment to the runbook:** §6's acceptance test should unmount *and make the device
> unavailable*; an unmount alone will always be silently repaired by `RequiresMountsFor`.
---
## 6. Bring-up and content verification (§7)
```
is-active: active / active
+================+====================+=========+
| name | path | comment |
+================+====================+=========+
| felhom-offsite | /mnt/pbs-datastore | |
+----------------+--------------------+---------+
| scratch | /srv/pbs-scratch | |
+================+====================+=========+
/dev/sdb 98G 13G 86G 13% /mnt/pbs-datastore
```
**Snapshot counts, before vs after — they match exactly: **
| Namespace | Phase 0 | After the move |
|---|---|---|
| `demo-felhom` | 2 | **2 ** |
| `demo-felhom-01` | 3 | **3 ** |
| `demo-hp` | 2 | **2 ** |
| **total ** | **7 ** | **7 ** |
### Verify job — full datastore, forced re-verification
Run with `--ignore-verified false` so every snapshot was re-read rather than trusting existing
verification state. This reads and digest-checks all 9,748 chunks:
```
verify datastore felhom-offsite
found 3 groups
…
percentage done: 100.00% (3/3 groups)
TASK OK
```
**0 errors on every snapshot. ** Read throughput off the new volume ~250 MiB/s.
---
## 7. Restore round-trip (§8) — PASSED
* A tier is only real once a restore round-trip is proven. * The datastore moved to different physical
storage, so the bar was re-cleared.
**Deviation from the runbook: run on demo-felhom, not demo-hp. ** §8 specifies demo-hp because its
archive is smaller, but **DooPlex holds no SSH key for demo-hp ** (`ssh demo-hp` →
`Permission denied (publickey)` — the known G1 gap). demo-felhom was used instead: same tier, same
relocated datastore, larger archive (~12 min instead of ~3). The acceptance criteria are unchanged;
only the wall-clock differs.
Triggered on demand via the agent's own harness rather than waiting for a tick (the cadence is now
3.5 days):
``` sh
sudo -u felhom-agent /usr/local/bin/felhom-agent \
--config /etc/felhom-agent/agent.json \
--selftest= restore-test \
--archive "felhom-pbs:backup/ct/9201/2026-07-26T12:21:48Z"
```
**Result — 09:20:41 → 09:32:41 CEST: **
``` json
{
"source_archive" : "felhom-pbs:backup/ct/9201/2026-07-26T12:21:48Z" ,
"source_tier" : "pbs" ,
"scratch_vmid" : 990000 ,
"pass" : true ,
"verified" : "boot+running" ,
"tested_at" : "2026-07-27T07:32:41Z" ,
"duration_seconds" : 720.710414828 ,
"mount_parity" : "ok" ,
"mount_inventory" : [
"mp0=/var/lib/docker (200G)" ,
"mp1=/mnt/sys_drive (50G)" ,
"mp8=/mnt/felhom-drives (throwaway for the archived bind)" ,
"mp9=/etc/felhom-bootstrap (throwaway for the archived bind)"
]
}
= = = s e l f t e s t = r e s t o r e - t e s t O K ( s c r a t c h 990000 r e s t o r e d + b o o t e d + v e r i f i e d + t o r n - d o w n i n 12 m 1 s ) = = =
```
| Required evidence | Result |
|---|---|
| `source_tier` | * * `pbs` ** ✓ |
| `pass` | * * `true` ** ✓ |
| `mount_parity` | * * `ok` ** ✓ |
| clean teardown | ✓ — `restore-test: scratch guest torn down vmid=990000` , gate `class=guest_destroy disposition=benign allowed=true` |
Scratch band confirmed clean afterwards: no `9900xx` guests, no `990000` LVs.
**A full restore → boot → verify → destroy cycle succeeded against the relocated chunk store. **
The pre-run `recover` pass reported `examined=0 scratch_destroyed=0 scratch_clean=0` , i.e. the
earlier interrupted test had already been reaped and did not contaminate this run.
---
## 8. The old copy is still in place (§10 rollback)
**Nothing was deleted. **
```
drwxr-xr-x 4 backup backup 4096 Jul 3 21:48 /srv/pbs-felhom
13G /srv/pbs-felhom (13,242,207,822 bytes)
old chunks: 9748
```
Root filesystem is back to **16 G used / 21 G avail (43 %) ** — unchanged from Phase 0. It peaked at
54 % during the run (retained 13 GB original + the temporary 4 GB swapfile); the swapfile has since
been removed, leaving ep0 with no swap exactly as it was found.
**Rollback remains a two-line config revert: **
``` sh
systemctl stop proxmox-backup-proxy proxmox-backup
cp -a /etc/proxmox-backup/datastore.cfg.pre-volume.bak /etc/proxmox-backup/datastore.cfg
systemctl start proxmox-backup proxmox-backup-proxy
```
Reclaiming the old copy is a **separate operator decision ** (§10), gated on at least one new weekly
offsite backup landing on the volume.
---
## 9. §11 — the unused storage box `u629193`
**Finding: not referenced by any live backup path, but ep0 carries a live mount against it. **
**No live reference: **
- no PBS datastore points at it (both stanzas are `/srv/…` / `/mnt/pbs-datastore` )
- no restic repo — the live offsite target is `storage-box-pool-1` (`u629488` ), a separate project
- no fstab entry, and **no `known_hosts` pin ** (`/root/.ssh/known_hosts` holds 0 storagebox lines)
- ROADMAP **R-17 already closed 2026-07-22 ** — sub-account `u629193-sub1` deleted at the console
- in-repo hits are documentation plus two **test fixtures ** only:
`hub/internal/hetznerapi/fake.go` (generates the `u629193-sub%d` pattern) and
`felhom-controller/controller/internal/report/dr_recipe_test.go` (string literal)
**But there IS a live mount on ep0: **
```
/etc/systemd/system/mnt-pbs\x2dstoragebox.mount enabled, active
What=u629193@u629193.your-storagebox.de:/home → /mnt/pbs-storagebox
IdentityFile=/root/.ssh/storagebox_ed25519
Options=port=23,idmap=user,uid=34,gid=34,…,StrictHostKeyChecking=yes,_netdev
```
Mounted now, holding spike leftovers `felhom-demo/` , `felhom-peti-spike/` , `spike-sub/` , `.ssh/` .
`lsof +D` shows nothing holding files open.
**Before the box is deleted ** , disable and remove that unit, or ep0 logs a failed mount on every
boot:
``` sh
systemctl disable --now 'mnt-pbs\x2dstoragebox.mount'
rm /etc/systemd/system/'mnt-pbs\x2dstoragebox.mount'
systemctl daemon-reload
rm -f /root/.ssh/storagebox_ed25519 /root/.ssh/storagebox_ed25519.pub # access key, regenerable
```
**The deletion itself is the operator's console click — not performed here. **
---
## 10. Deferred / open
1. * * `scratch` datastore stanza** points at a non-existent path and now logs ENOENT on every PBS
start. Remove the stanza or create the directory — operator decision.
2. **Old copy at `/srv/pbs-felhom` retained ** (13 GB). Reclaim only after a new weekly offsite
backup lands on the volume, with explicit go-ahead (§10.1).
3. **GC has still not been run. ** The round-trip has now passed, so §10.2 is unblocked — but GC is
the one operation that deletes, and it is deliberately left for a separate, deliberate run. No GC
schedule is configured on this PBS at all, which is worth its own decision.
4. **ep0 has no swap ** (removed the temporary 4 GB file; the box is as found). A 3.7 GB box with
zero swap is what turned a routine 13 GB copy into an OOM kill. Worth considering a small
permanent swapfile — not done here, as it is outside this runbook's scope.
5. **Hub PBS-DR capacity gauge ** (§10.3) not yet re-read — it should now report ~98 GB, not 37.2 GB.
Could not be queried from DooPlex: the hub is a ClusterIP service and was not reachable directly.
6. * * `x-systemd.device-timeout=30s` means a genuinely missing volume delays boot by 30 s** before
`nofail` lets boot continue. Accepted trade-off.
## 11. Observations
- **The atime note in §4 holds:** the copy gave every chunk a fresh access time. This is safe in the
conservative direction — GC only reclaims chunks whose atime predates the cutoff, so fresh times
mean nothing is reclaimed early. Not a problem; do not "fix" it.
- **`-m 0` mattered more than it looks:** 5 GB withheld on a 100 GB volume is 5 % of the headroom
this whole exercise was about.
- **The relocation does not change the per-tenant-encryption growth story.** The volume buys
headroom (13 GB / 98 GB = 13 %, vs 13 GB / 37 GB = 35 %); it does not introduce cross-customer
dedup, which per-tenant encryption still precludes.
