# RUNBOOK — manual guest restore (`pct restore`), and the bind mounts it drags with it **Status:** operational procedure. Written 2026-07-28 to close **F-OPS** (Campaign 8). **Audience:** operator, at a keyboard, usually during a real disaster recovery. > **The hazard in one sentence:** `pct restore` recreates the guest from the archive's **own config**, > which includes the source guest's `mpN` bind mounts — so the restored guest can come up bound to a > host path that does not exist on this host, or, worse, to a path that exists and belongs to > **something else**. This is not hypothetical and it is not rare: the agent's own restore paths neutralise these mountpoints deliberately (`bindMountOverrides` in `restoretest.go`, `RestoreLXC{MountOverrides:…}` — see `audits/SPIKE-dr-bindmount-source-2026-07-07.md`). A **manual** `pct restore` has no such protection. The automation is safe; the human path is the one that needs this page. Timing makes it worse: a manual restore happens during an incident, under pressure, often on a *different* host from the one the archive came from. --- ## 1. What a Felhom guest's mounts actually look like From live guest 9201 (2026-07-28) — the shape you should expect: ``` rootfs: local-lvm:vm-9201-disk-0,size=32G mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/docker,backup=1,size=50G mp1: local-lvm:vm-9201-disk-2,mp=/mnt/sys_drive,backup=1,size=20G mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives ← HOST BIND mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1 ← HOST BIND hookscript: local:snippets/felhom-guest-hook.sh ``` **Two kinds of `mpN`, and only one is dangerous:** | kind | looks like | on restore | |---|---|---| | **Volume mount** (`mp0`, `mp1`) | starts with a storage ID — `local-lvm:vm-…` | restored from the archive. Fine. | | **Bind mount** (`mp8`, `mp9`) | starts with an absolute **host path** — `/mnt/…`, `/var/lib/…` | **NOT in the archive.** The path is taken as-is on the target host. | The bind mounts are **structural constants** of the Felhom topology, not per-customer data — `mp8` is the shared drive parent, `mp9` is the per-guest bootstrap dir. That is what makes them safe to rewrite by hand: you are restoring a known layout, not guessing. **`mp9` is the one that bites.** It embeds the **source** VMID: `/var/lib/felhom-agent/guests//bootstrap`. Restore to a different VMID and the guest either finds nothing there or — if that VMID exists on this host — reads **another guest's bootstrap config, including its credentials**. --- ## 2. Before you start — three checks Run these on the **target** host, and write the answers down. ```bash # 1. What does the archive think its mounts are? (read the config WITHOUT restoring) # PBS: proxmox-backup-client restore pct.conf - --repository 2>/dev/null | grep -E '^(mp|rootfs)' # vzdump tarball: tar -xOf /path/to/vzdump-lxc--.tar.zst ./etc/vzdump/pct.conf 2>/dev/null | grep -E '^(mp|rootfs)' # 2. Does every BIND path exist on THIS host? ls -ld /mnt/felhom-drives /var/lib/felhom-agent/guests//bootstrap # 3. Is the target VMID free, and is the source VMID something else here? pct list | awk '{print $1}' | grep -x -e -e ``` **Stop and think if:** a bind path is missing, or the source VMID is a live guest on this host. --- ## 3. Restore, then fix the binds BEFORE first boot Restore **without starting** the guest. `pct restore` does not auto-start, but never pass anything that would, and do not `pct start` until §4 passes. ```bash pct restore --storage --unprivileged 1 ``` Then strip and re-add the binds. Do this even when the VMID is unchanged — it costs seconds and it is the whole point of this page: ```bash # strip every BIND mountpoint the archive carried (leave the volume mounts mp0/mp1 alone) pct set --delete mp8 pct set --delete mp9 # re-add them for THIS host and THIS vmid mkdir -p /var/lib/felhom-agent/guests//bootstrap pct set -mp8 /mnt/felhom-drives,mp=/mnt/felhom-drives pct set -mp9 /var/lib/felhom-agent/guests//bootstrap,mp=/etc/felhom-bootstrap,ro=1 ``` > The bootstrap dir must be populated for the guest to enrol. On a host running the agent, the > provisioning back-half owns that directory — prefer letting the agent write it to hand-copying a > `bootstrap.json` between guests. **Never** copy one from another customer's guest: it carries that > customer's tokens. Also check the hookscript line: `hookscript: local:snippets/felhom-guest-hook.sh` must point at a snippet that exists **on this host**, or the guest will refuse to start. ```bash ls -l /var/lib/vz/snippets/felhom-guest-hook.sh || pct set --delete hookscript ``` --- ## 4. Verify before starting — the positive check Do not accept "no error" as evidence. Assert what must be **true**: ```bash # every mpN is either a storage volume or a bind whose host path EXISTS on this host pct config | grep -E '^mp[0-9]+:' | while read -r line; do src=${line#*: }; src=${src%%,*} case "$src" in /*) [ -e "$src" ] && echo "OK bind $src" || echo "MISSING bind $src <-- fix before start" ;; *) echo "OK volume $src" ;; esac done # no mpN may still reference the SOURCE vmid pct config | grep -E "guests//" && echo "STILL POINTS AT THE SOURCE GUEST" ``` Only when every line reads `OK` and the source-VMID grep is empty: ```bash pct start pct exec -- docker ps --format '{{.Names}} {{.Status}}' ``` --- ## 5. If the guest is already running with the wrong binds Stop it before touching the mounts — a live bind swap under a running container set is how data ends up half-written to two places. ```bash pct stop # ...§3 fixes... pct start ``` --- ## What this page deliberately does NOT do **It does not build tooling.** The agent already neutralises binds on its own restore paths; wrapping the manual path in a script would create a second implementation of that logic, and two implementations of one invariant is how they drift. If the manual path becomes routine, the correct move is to route it through the agent, not to script around it.