From ad8057c4e339e7a9c61f1088e1f4229846633b61 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 27 Jul 2026 09:36:11 +0200 Subject: [PATCH] docs(ep0): relocate the felhom-offsite PBS datastore onto the 100 GB volume Supervised runbook execution. No code change, no version bump. felhom-offsite moved from ep0's 40 GB root disk (/srv/pbs-felhom) to a dedicated 100 GB Hetzner Cloud Volume (/mnt/pbs-datastore, ext4 -m 0, by-id fstab, relatime). Datastore NAME unchanged, so the PBS-DR descriptors, per-box storage ids, ACLs and namespaces are untouched. Capacity: 37.2 GB -> 98 GB total, 28.9% -> 13% used, headroom to the 80% warn 19 GB -> ~65 GB. This CLEARS the R-82 Phase 0 P0.3 STOP. Per-tenant encryption still precludes cross-customer dedup, so the slope is unchanged - the volume buys runway, not a better cost model. Verified: byte totals and chunk counts identical (9748), 7/7 snapshots across all three namespaces, backup:backup ownership, clean itemised dry-run, full verify job TASK OK with 0 errors, and a restore round-trip (source_tier pbs, pass true, mount_parity ok, clean teardown). Nothing deleted - the original 13 GB stays at /srv/pbs-felhom as the rollback until a new weekly backup lands. GC deliberately not run. Three findings recorded: - the `scratch` datastore points at a non-existent path (pre-existing; now logs ENOENT every start) - operator decision - the runbook's S6 guard test proves the wrong proposition: RequiresMountsFor re-mounts rather than refusing, so the test only bites when the device is genuinely unavailable (re-run that way, and the refusal was observed) - amendment recommended - S11: storage box u629193 has no live backup path, BUT ep0 carries an enabled sshfs mount unit against it that must be removed before the box is deleted Deviations: the volume arrived pre-formatted and mounted; S8 ran on demo-felhom rather than demo-hp (no SSH key for demo-hp); the window was contended by a stale in-memory 10-minute restore-test cadence whose config had already been reverted on disk. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_018ARoadHBf8rHoscfiqeVZn --- REPORT-ep0-datastore-volume-2026-07-27.md | 90 ++++ .../architecture/07-backup-architecture.md | 26 + .../audits/SPIKE-r82-phase0-2026-07-26.md | 14 +- ...RUNBOOK-ep0-datastore-volume-2026-07-27.md | 444 ++++++++++++++++++ 4 files changed, 573 insertions(+), 1 deletion(-) create mode 100644 REPORT-ep0-datastore-volume-2026-07-27.md create mode 100644 documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md diff --git a/REPORT-ep0-datastore-volume-2026-07-27.md b/REPORT-ep0-datastore-volume-2026-07-27.md new file mode 100644 index 0000000..031041e --- /dev/null +++ b/REPORT-ep0-datastore-volume-2026-07-27.md @@ -0,0 +1,90 @@ +# REPORT — ep0 PBS datastore relocated onto the 100 GB volume (2026-07-27) + +**Class:** supervised operational run (RUNBOOK execution). **No code changed. No version bump.** +Written as `REPORT-.md` per the parallel-session rule — the shared `REPORT.md` was not touched. + +**Full record with all evidence:** `documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md` + +--- + +## Outcome: DONE and verified + +`felhom-offsite` now lives on a dedicated 100 GB Hetzner Cloud Volume instead of ep0's 40 GB root disk. + +| | Before | After | +|---|---|---| +| Path | `/srv/pbs-felhom` (root disk) | **`/mnt/pbs-datastore`** (volume) | +| Datastore total | 37.2 GB | **98 GB** | +| Used | 28.9 % | **13 %** | +| Headroom to the 80 % warn | 19 GB | **≈65 GB** | +| Additional customers before warn | ≈2 | **≈7–13** | + +Datastore **name unchanged** — the PBS-DR descriptors, per-box storage ids, ACLs and namespace +layout that R-39/R-82 made self-healing are untouched. + +**Window:** 06:58 → 07:19 UTC (PBS down 07:00 → 07:17). **Nothing was deleted.** + +### Acceptance evidence + +| Gate | Result | +|---|---| +| Copy integrity | 13,242,207,822 = 13,242,207,822 B · **9,748 = 9,748 chunks** · 7 = 7 snapshots · `backup:backup` · itemised dry-run **0 lines** | +| Snapshot counts per ns | `demo-felhom` 2=2, `demo-felhom-01` 3=3, `demo-hp` 2=2 | +| atime semantics | `rw,relatime,discard` — **`relatime` present, `noatime` absent** (GC correctness) | +| Verify job | `TASK OK`, 3/3 groups, forced re-verification of every snapshot, **0 errors** | +| §6 mount guard | **refusal observed** — `Job … failed with result 'dependency'`; mountpoint stayed empty | +| §8 restore round-trip | `source_tier: pbs`, `pass: true`, `mount_parity: ok`, clean teardown, 12m1s | + +--- + +## Three findings the operator should act on + +1. **`scratch` datastore is configured at a path that does not exist** (`/srv/pbs-scratch`). + Pre-existing, not caused here, but now logs `ENOENT` on every PBS start. This is the PRIME RISK + shape ("reports fine, is not there") already live in the config. **Decision needed:** remove the + stanza or create the directory. + +2. **The runbook's §6 acceptance test proves the wrong proposition.** `RequiresMountsFor` is a + mount-first ordering guarantee, not a refusal — systemd silently *re-mounts* an unmounted volume + and PBS then starts safely. The test only bites when the device is genuinely unavailable, which + is how it was re-run and passed. **Amendment recommended in the runbook record.** + +3. **§11 — storage box `u629193` is NOT simply unused.** No live backup path references it (no + datastore, no restic repo, no fstab, no `known_hosts` pin; R-17 already deleted `u629193-sub1`), + **but ep0 carries an enabled, currently-mounted sshfs unit** `mnt-pbs\x2dstoragebox.mount` → + `/mnt/pbs-storagebox`, holding spike leftovers. Disable and remove that unit before deleting the + box, or ep0 logs a failed mount every boot. **The deletion is the operator's console click.** + +## Deviations from the runbook as written + +- **The volume arrived already formatted and mounted** by Hetzner at `/mnt/HC_Volume_106469259` + (§2 assumed neither). Operator ruled: reformat + repath. The 5 % reserve was reclaimed (`-m 0`). +- **§8 ran on demo-felhom, not demo-hp** — DooPlex holds no SSH key for demo-hp (the G1 gap). Same + tier, same relocated datastore, larger archive. +- **The window was contended** by a stale 10-minute restore-test cadence on demo-felhom: the config + had already been reverted to 3.5 days on disk, but the cadence is read once at daemon start and + `NRestarts=0`. Restarting the agent applied it (`cadence=84h0m0s`). The in-flight test was allowed + to finish rather than aborted. + +## Process errors made during this run (recorded deliberately) + +- `rsync -aHAX` **OOM-killed** ep0 (3.7 GB RAM, no swap). Cause: a PBS `.chunks/` tree pre-creates + all 65536 shard dirs → 75,341 inodes, and `-H` retains the whole inode map. `-H` was dropped only + after **proving** no hardlinks exist (`-links +1` → 0; max link count → 1); PBS references chunks + by digest, never by hardlink. +- `/usr/bin/time -v` is not installed on ep0 → exit 127, rsync never ran, and a `| grep … || true` + wrapper swallowed it while printing a success-looking line. +- `rsync --version | head -1` reported a working rsync 3.4.1 as "missing" — **the §12 pipe-into-head + trap, fourth recorded instance in this project.** Both fixed by capturing the command's own `$?`. + +## Deferred + +1. **Old copy retained** at `/srv/pbs-felhom` (13 GB, 9,748 chunks) as the rollback. Rollback is a + two-line `datastore.cfg` revert. Reclaim only after a new weekly offsite backup lands on the + volume, with explicit go-ahead. +2. **GC not run** — now unblocked by the round-trip, but left for a separate deliberate run. No GC + schedule is configured on this PBS at all. +3. **Hub PBS-DR capacity gauge not re-read** — the hub is a ClusterIP service and was not reachable + from DooPlex. If it still reports 37.2 GB it is reading the wrong filesystem: a real bug. +4. **ep0 has no swap** (temporary 4 GB file removed; box left as found). Worth a small permanent + swapfile — outside this runbook's scope. diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index 437e46b..7a7c58d 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -342,6 +342,32 @@ backups/secondary// ### 9.1 PBS whole-guest tier — measured capacity state (R-82 Phase 0, 2026-07-26) +> ### 🟢 SUPERSEDED 2026-07-27 — the datastore was grown; the P0.3 STOP is CLEARED +> +> The "grown later" in the paragraph below **has happened.** `felhom-offsite` was relocated off +> ep0's 40 GB root disk onto a dedicated **100 GB Hetzner Cloud Volume** +> (`scsi-0HC_Volume_106469259` → `/mnt/pbs-datastore`, ext4 `-m 0`). Evidence and method: +> `runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md`. +> +> | Fact | Before (2026-07-26) | After (2026-07-27) | +> |---|---|---| +> | Datastore total | 37.2 GB | **98 GB** | +> | Used | 10.8 GB (**28.9 %**) | 13 GB (**13 %**) | +> | 80 % warn threshold | 29.8 GB | **78.4 GB** | +> | Headroom to warn | 19.0 GB | **≈ 65 GB** | +> | Additional customers before warn (at ≈5–10 GB each) | **≈ 2** | **≈ 7–13** | +> +> **What this does NOT change:** point 1 below still holds in full — per-tenant encryption still +> precludes cross-customer dedup, so cost still scales linearly per customer. The volume buys +> runway, not a better cost model. Point 3 also still holds: `pvesm` still cannot see this +> datastore's fill. +> +> **Re-measure rather than trust this table** once a second weekly snapshot exists — the weekly +> incremental size is still **UNMEASURED**, and it, not the total, decides the real slope. +> +> ⚠️ The hub's PBS-DR gauge had **not** been re-read at the time of writing; if it still reports +> 37.2 GB it is reading the wrong filesystem, which is a real bug worth a roadmap item. + Recorded per the operator's 2026-07-26 ruling: **the datastore will be grown later; R-82 proceeds meanwhile.** These are measurements, not projections-of-record — re-measure before relying on them. Full method + evidence: `audits/SPIKE-r82-phase0-2026-07-26.md`. diff --git a/documentation/audits/SPIKE-r82-phase0-2026-07-26.md b/documentation/audits/SPIKE-r82-phase0-2026-07-26.md index 10116f9..de0e887 100644 --- a/documentation/audits/SPIKE-r82-phase0-2026-07-26.md +++ b/documentation/audits/SPIKE-r82-phase0-2026-07-26.md @@ -139,7 +139,19 @@ change, deliberately **not** made here. --- -## P0.3 — Capacity: 🛑 STOP +## P0.3 — Capacity: 🛑 STOP → 🟢 **CLEARED 2026-07-27** + +> **Resolution (2026-07-27).** The datastore was moved off ep0's 40 GB root disk onto a dedicated +> **100 GB Hetzner Cloud Volume**. Total **37.2 GB → 98 GB**; used **28.9 % → 13 %**; headroom to the +> 80 % warn **19 GB → ≈65 GB**. The "80 % at roughly the second additional customer" projection below +> becomes **roughly the seventh to thirteenth**. Method, verification and the restore round-trip that +> re-cleared the tier: `runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md`. +> +> **The dedup fact below is unchanged and still governs the slope** — per-tenant encryption still +> means no cross-customer dedup. The volume bought runway, not a better cost model. The weekly +> incremental size remains **unmeasured**. + +**The analysis below is the original 2026-07-26 record, retained as written.** ### Measured diff --git a/documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md b/documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md new file mode 100644 index 0000000..3be31d0 --- /dev/null +++ b/documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md @@ -0,0 +1,444 @@ +# RUNBOOK RECORD — ep0 PBS datastore relocated onto the 100 GB volume (2026-07-27) + +**Class:** supervised operational run. **No repo version bump** — the only commits are this record +and the capacity note. **Nothing was deleted.** + +**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner CX23, Nuremberg. +**Datastore moved:** `felhom-offsite`, `/srv/pbs-felhom` → **`/mnt/pbs-datastore`** (name unchanged). +**Window:** 06:58 → 07:19 UTC. PBS down 07:00 → 07:17 UTC. + +--- + +## 1. Phase 0 — ground truth + +Established read-only before anything was touched. **Three of the runbook's premises did not hold**; +each is recorded below. + +| Item | Value | +|---|---| +| PBS | `proxmox-backup-server 4.2.3-1`, running version **4.2.2** | +| Datastore name / path | **`felhom-offsite`** → `/srv/pbs-felhom` | +| Size on disk | **13,242,207,822 B (13 GB)** | +| Chunk count | **9,748** | +| Total entries | 75,341 (reg 9,785, dir 65,556 — PBS pre-creates all 65536 `.chunks` shards) | +| Snapshots | **7** — `demo-felhom` 2, `demo-felhom-01` 3, `demo-hp` 2 | +| Ownership | `backup:backup` | +| Root filesystem | `/dev/sda1` ext4, 38 G, **16 G used / 21 G avail (43 %)**, `rw,relatime,errors=remount-ro` | +| Volume | `/dev/disk/by-id/scsi-0HC_Volume_106469259` → `/dev/sdb`, 100 G — present | +| Scheduled jobs | **none** — no `verification.cfg`, no `prune.cfg`, no `sync.cfg`, no GC schedule | +| Last weekly backups | 2026-07-26 (demo-felhom 12:21, demo-hp 15:42 + 17:03); next ≈ 08-02 — window clear | + +### Deviation ① — a second datastore is configured whose path does not exist + +``` +datastore: scratch → /srv/pbs-scratch ← No such file or directory +datastore: felhom-offsite → /srv/pbs-felhom ← the real one +``` + +`proxmox-backup-manager datastore list` reports `scratch` without complaint. `/srv/` contains only +`pbs-felhom`. **This is the PRIME RISK failure mode already live in the config** — a datastore that +reports fine and is not there. Pre-existing, untouched by this run, and now *louder*: since the +relocation PBS polls filesystem info for both datastores and logs on every start: + +``` +read fs info on "/srv/pbs-scratch" failed - ENOENT: No such file or directory +find_mounted_device failed - ENOENT: No such file or directory +``` + +**Open decision for the operator:** remove the stanza, or create the directory. Not actioned here. + +### Deviation ② — the volume was already formatted and already mounted + +Hetzner's provisioning had done both. `/etc/fstab` already carried: + +``` +/dev/disk/by-id/scsi-0HC_Volume_106469259 /mnt/HC_Volume_106469259 ext4 discard,nofail,defaults 0 0 +``` + +Effective options `rw,relatime,discard` — `relatime` present, `noatime` absent, and `by-id` already +in use, so those two traps were clear on arrival. The filesystem was empty (`lost+found` only) but +carried the default 5 % reserve: 98 G size, **93 G avail**. + +§2 as written would have run `mkfs.ext4` against a mounted filesystem and failed. **Operator ruling: +reformat and repath per the runbook** (the volume was provably empty), reclaiming the 5 GB and +aligning the mount path with the §6 guard. + +### Deviation ③ — the window was contended by a restore-test loop + +ep0 showed `reader` tasks pulling chunks every ~10–20 minutes, each running 10–11 minutes. Root +cause was **not** on ep0 but on demo-felhom: + +``` +07:54:06 CEST felhom-agent restarted → scheduler started with cadence=10m0s +08:08 CEST /etc/felhom-agent/agent.json edited → restore_test_cadence_seconds = 302400 (3.5 d) + NRestarts=0 — the running process never picked up the revert +``` + +**The cadence is read once at daemon start.** The config was already correct on disk; only the +process was stale — the same in-memory-vs-persisted shape as the controller's report floor. The +agent was restarted (08:50:16 CEST, `cadence=84h0m0s` confirmed in the journal), after which the +loop stopped and the window was uncontended. + +The in-flight restore-test was allowed to **finish cleanly** (`06:44:14 → 06:58:01 OK`) rather than +being aborted; §1's "do not proceed while a task is running" was held. + +--- + +## 2. The volume (§2) + +fstab line used (replacing Hetzner's; `discard` retained as appropriate for a Hetzner Cloud Volume): + +``` +/dev/disk/by-id/scsi-0HC_Volume_106469259 /mnt/pbs-datastore ext4 defaults,relatime,discard,nofail,x-systemd.device-timeout=30s 0 0 +``` + +`mkfs.ext4 -F -m 0 -L pbs-datastore` — guarded by a check that the filesystem held nothing but +`lost+found` before unmounting. + +**Proof of mount and atime semantics:** + +``` +$ findmnt /mnt/pbs-datastore +TARGET SOURCE FSTYPE OPTIONS +/mnt/pbs-datastore /dev/sdb ext4 rw,relatime,discard + +options: rw,relatime,discard +OK: relatime present, noatime absent +``` + +`df` after mkfs: **98 G size, 98 G avail** (was 93 G — the 5 GB root reserve reclaimed). + +--- + +## 3. The copy (§4) — and the two failures on the way + +**First attempt OOM-killed at 52 %.** + +``` +oom-kill: … task=rsync,pid=605913 +Out of memory: Killed process 605913 (rsync) total-vm:3464728kB, anon-rss:1912224kB +``` + +ep0 has 3.7 GB RAM and **no swap**. A PBS `.chunks/` tree pre-creates **all 65536 shard +directories**, so even this 13 GB store is 75,341 inodes; `-H` makes rsync retain the whole inode +map to detect hardlinks, and that is what exhausted memory. + +**`-H` was dropped only after proving it was unnecessary** — the runbook is right that it is +unrecoverable if you guess: + +``` +$ find /srv/pbs-felhom -type f -links +1 | wc -l → 0 +$ find /srv/pbs-felhom -type f -printf '%n\n' | sort -rn | head -1 → 1 +``` + +A PBS chunk store contains no hardlinks — snapshots reference chunks by digest inside `.didx`/`.fidx` +index files, never by link count. A temporary 4 GB swapfile was added as insurance. + +**Second attempt failed silently — operator-visible process error, recorded deliberately.** +`/usr/bin/time -v` is **not installed** on ep0 → exit 127, rsync never ran, and the wrapper +`… | grep … || true` swallowed it while printing a success-looking line. A follow-up guard then hit +the §12 trap directly — `rsync --version | head -1` returns *head's* status, not rsync's, so a +working rsync 3.4.1 was reported as "ABORT: rsync missing". **This is the fourth recorded instance +of that trap in this project.** Both were fixed by capturing the command's own `$?` with no +pipeline in between. + +**Successful copy** (`rsync -aAX --numeric-ids --stats`, trailing slash on source, no `--delete`, +no `-H`), 07:12:48 → 07:14:16 UTC, exit **0**: + +``` +Number of files: 75,341 (reg: 9,785, dir: 65,556) +Number of created files: 8,697 +Number of deleted files: 0 +Total file size: 13,242,207,822 bytes +sent 11,803,880,151 bytes received 394,094 bytes 134,905,991.37 bytes/sec +``` + +### Copy verification — all gates passed + +``` +src bytes = 13242207822 dst bytes = 13242207822 +src chunks = 9748 dst chunks = 9748 + demo-felhom src=2 dst=2 OK + demo-felhom-01 src=3 dst=3 OK + demo-hp src=2 dst=2 OK +total snapshots src=7 dst=7 + +chunks owner = backup:backup +datastore root owner = backup:backup +.lock present = -rw-r--r-- 1 backup backup 0 + +dry-run exit = 0 +itemised lines = 0 +OK: dry-run clean +``` + +--- + +## 4. The repoint (§5) + +`datastore.cfg` backed up to `datastore.cfg.pre-volume.bak`. Exactly one line changed, asserted +programmatically to be the `felhom-offsite` stanza's `path` and no other: + +```diff +5c5 +< path /srv/pbs-felhom +--- +> path /mnt/pbs-datastore +``` + +**The datastore name is unchanged** and the `scratch` stanza was not touched. + +--- + +## 5. The mount guard (§6) — and a correction to the runbook's acceptance test + +Drop-ins written for **both** services: + +``` +/etc/systemd/system/proxmox-backup.service.d/10-datastore-mount.conf +/etc/systemd/system/proxmox-backup-proxy.service.d/10-datastore-mount.conf + [Unit] + RequiresMountsFor=/mnt/pbs-datastore +``` + +### The runbook's test as written proves the wrong proposition + +§6 says PBS "MUST FAIL" to start with the volume unmounted. It does not — and that is correct +behaviour, not a broken guard. `RequiresMountsFor` is a **mount-first ordering guarantee**, not a +refusal: systemd pulls the mount unit in as a dependency and *starts* it. Observed: + +``` +07:16:10 mnt-pbs\x2ddatastore.mount: Deactivated successfully ← the test's umount +07:16:11 Mounting mnt-pbs\x2ddatastore.mount... ← systemd re-mounted it +07:16:11 Mounted mnt-pbs\x2ddatastore.mount +``` + +PBS then started **with the volume correctly mounted**. It never saw an unmounted path. Satisfying +the dependency by mounting is stronger than refusing. + +### The corrected test — volume genuinely unavailable + +The real hazard is not "unmounted but mountable", it is "volume gone". Simulated by pointing the +fstab entry at a non-existent device and attempting a start: + +``` +$ systemctl start proxmox-backup +A dependency job for proxmox-backup.service failed. +systemctl start proxmox-backup exit code: 1 +is-active: inactive + +Jul 27 07:17:43 Dependency failed for proxmox-backup.service - Proxmox Backup API Server. +Jul 27 07:17:43 proxmox-backup.service: Job proxmox-backup.service/start failed with result 'dependency'. +``` + +**And the bare mountpoint stayed empty** — PBS wrote nothing into the root disk underneath the +mount, which is the precise PRIME RISK outcome the guard exists to prevent: + +``` +$ ls -la /mnt/pbs-datastore +total 8 +drwxr-xr-x 2 root root 4096 Jul 27 06:58 . +drwxr-xr-x 4 root root 4096 Jul 27 06:58 .. +``` + +fstab was then restored and the real volume remounted. **A guard that has not been observed refusing +is not a guard — this one has now been observed refusing.** + +> **Recommended amendment to the runbook:** §6's acceptance test should unmount *and make the device +> unavailable*; an unmount alone will always be silently repaired by `RequiresMountsFor`. + +--- + +## 6. Bring-up and content verification (§7) + +``` +is-active: active / active + ++================+====================+=========+ +| name | path | comment | ++================+====================+=========+ +| felhom-offsite | /mnt/pbs-datastore | | ++----------------+--------------------+---------+ +| scratch | /srv/pbs-scratch | | ++================+====================+=========+ + +/dev/sdb 98G 13G 86G 13% /mnt/pbs-datastore +``` + +**Snapshot counts, before vs after — they match exactly:** + +| Namespace | Phase 0 | After the move | +|---|---|---| +| `demo-felhom` | 2 | **2** | +| `demo-felhom-01` | 3 | **3** | +| `demo-hp` | 2 | **2** | +| **total** | **7** | **7** | + +### Verify job — full datastore, forced re-verification + +Run with `--ignore-verified false` so every snapshot was re-read rather than trusting existing +verification state. This reads and digest-checks all 9,748 chunks: + +``` +verify datastore felhom-offsite +found 3 groups +… +percentage done: 100.00% (3/3 groups) +TASK OK +``` + +**0 errors on every snapshot.** Read throughput off the new volume ~250 MiB/s. + +--- + +## 7. Restore round-trip (§8) — PASSED + +*A tier is only real once a restore round-trip is proven.* The datastore moved to different physical +storage, so the bar was re-cleared. + +**Deviation from the runbook: run on demo-felhom, not demo-hp.** §8 specifies demo-hp because its +archive is smaller, but **DooPlex holds no SSH key for demo-hp** (`ssh demo-hp` → +`Permission denied (publickey)` — the known G1 gap). demo-felhom was used instead: same tier, same +relocated datastore, larger archive (~12 min instead of ~3). The acceptance criteria are unchanged; +only the wall-clock differs. + +Triggered on demand via the agent's own harness rather than waiting for a tick (the cadence is now +3.5 days): + +```sh +sudo -u felhom-agent /usr/local/bin/felhom-agent \ + --config /etc/felhom-agent/agent.json \ + --selftest=restore-test \ + --archive "felhom-pbs:backup/ct/9201/2026-07-26T12:21:48Z" +``` + +**Result — 09:20:41 → 09:32:41 CEST:** + +```json +{ + "source_archive": "felhom-pbs:backup/ct/9201/2026-07-26T12:21:48Z", + "source_tier": "pbs", + "scratch_vmid": 990000, + "pass": true, + "verified": "boot+running", + "tested_at": "2026-07-27T07:32:41Z", + "duration_seconds": 720.710414828, + "mount_parity": "ok", + "mount_inventory": [ + "mp0=/var/lib/docker (200G)", + "mp1=/mnt/sys_drive (50G)", + "mp8=/mnt/felhom-drives (throwaway for the archived bind)", + "mp9=/etc/felhom-bootstrap (throwaway for the archived bind)" + ] +} +=== selftest=restore-test OK (scratch 990000 restored+booted+verified+torn-down in 12m1s) === +``` + +| Required evidence | Result | +|---|---| +| `source_tier` | **`pbs`** ✓ | +| `pass` | **`true`** ✓ | +| `mount_parity` | **`ok`** ✓ | +| clean teardown | ✓ — `restore-test: scratch guest torn down vmid=990000`, gate `class=guest_destroy disposition=benign allowed=true` | + +Scratch band confirmed clean afterwards: no `9900xx` guests, no `990000` LVs. + +**A full restore → boot → verify → destroy cycle succeeded against the relocated chunk store.** +The pre-run `recover` pass reported `examined=0 scratch_destroyed=0 scratch_clean=0`, i.e. the +earlier interrupted test had already been reaped and did not contaminate this run. + +--- + +## 8. The old copy is still in place (§10 rollback) + +**Nothing was deleted.** + +``` +drwxr-xr-x 4 backup backup 4096 Jul 3 21:48 /srv/pbs-felhom +13G /srv/pbs-felhom (13,242,207,822 bytes) +old chunks: 9748 +``` + +Root filesystem is back to **16 G used / 21 G avail (43 %)** — unchanged from Phase 0. It peaked at +54 % during the run (retained 13 GB original + the temporary 4 GB swapfile); the swapfile has since +been removed, leaving ep0 with no swap exactly as it was found. + +**Rollback remains a two-line config revert:** + +```sh +systemctl stop proxmox-backup-proxy proxmox-backup +cp -a /etc/proxmox-backup/datastore.cfg.pre-volume.bak /etc/proxmox-backup/datastore.cfg +systemctl start proxmox-backup proxmox-backup-proxy +``` + +Reclaiming the old copy is a **separate operator decision** (§10), gated on at least one new weekly +offsite backup landing on the volume. + +--- + +## 9. §11 — the unused storage box `u629193` + +**Finding: not referenced by any live backup path, but ep0 carries a live mount against it.** + +**No live reference:** +- no PBS datastore points at it (both stanzas are `/srv/…` / `/mnt/pbs-datastore`) +- no restic repo — the live offsite target is `storage-box-pool-1` (`u629488`), a separate project +- no fstab entry, and **no `known_hosts` pin** (`/root/.ssh/known_hosts` holds 0 storagebox lines) +- ROADMAP **R-17 already closed 2026-07-22** — sub-account `u629193-sub1` deleted at the console +- in-repo hits are documentation plus two **test fixtures** only: + `hub/internal/hetznerapi/fake.go` (generates the `u629193-sub%d` pattern) and + `felhom-controller/controller/internal/report/dr_recipe_test.go` (string literal) + +**But there IS a live mount on ep0:** + +``` +/etc/systemd/system/mnt-pbs\x2dstoragebox.mount enabled, active + What=u629193@u629193.your-storagebox.de:/home → /mnt/pbs-storagebox + IdentityFile=/root/.ssh/storagebox_ed25519 + Options=port=23,idmap=user,uid=34,gid=34,…,StrictHostKeyChecking=yes,_netdev +``` + +Mounted now, holding spike leftovers `felhom-demo/`, `felhom-peti-spike/`, `spike-sub/`, `.ssh/`. +`lsof +D` shows nothing holding files open. + +**Before the box is deleted**, disable and remove that unit, or ep0 logs a failed mount on every +boot: + +```sh +systemctl disable --now 'mnt-pbs\x2dstoragebox.mount' +rm /etc/systemd/system/'mnt-pbs\x2dstoragebox.mount' +systemctl daemon-reload +rm -f /root/.ssh/storagebox_ed25519 /root/.ssh/storagebox_ed25519.pub # access key, regenerable +``` + +**The deletion itself is the operator's console click — not performed here.** + +--- + +## 10. Deferred / open + +1. **`scratch` datastore stanza** points at a non-existent path and now logs ENOENT on every PBS + start. Remove the stanza or create the directory — operator decision. +2. **Old copy at `/srv/pbs-felhom` retained** (13 GB). Reclaim only after a new weekly offsite + backup lands on the volume, with explicit go-ahead (§10.1). +3. **GC has still not been run.** The round-trip has now passed, so §10.2 is unblocked — but GC is + the one operation that deletes, and it is deliberately left for a separate, deliberate run. No GC + schedule is configured on this PBS at all, which is worth its own decision. +4. **ep0 has no swap** (removed the temporary 4 GB file; the box is as found). A 3.7 GB box with + zero swap is what turned a routine 13 GB copy into an OOM kill. Worth considering a small + permanent swapfile — not done here, as it is outside this runbook's scope. +5. **Hub PBS-DR capacity gauge** (§10.3) not yet re-read — it should now report ~98 GB, not 37.2 GB. + Could not be queried from DooPlex: the hub is a ClusterIP service and was not reachable directly. +6. **`x-systemd.device-timeout=30s` means a genuinely missing volume delays boot by 30 s** before + `nofail` lets boot continue. Accepted trade-off. + +## 11. Observations + +- **The atime note in §4 holds:** the copy gave every chunk a fresh access time. This is safe in the + conservative direction — GC only reclaims chunks whose atime predates the cutoff, so fresh times + mean nothing is reclaimed early. Not a problem; do not "fix" it. +- **`-m 0` mattered more than it looks:** 5 GB withheld on a 100 GB volume is 5 % of the headroom + this whole exercise was about. +- **The relocation does not change the per-tenant-encryption growth story.** The volume buys + headroom (13 GB / 98 GB = 13 %, vs 13 GB / 37 GB = 35 %); it does not introduce cross-customer + dedup, which per-tenant encryption still precludes.