felhom.eu/documentation/runbooks/RUNBOOK-manual-guest-restore.md

RUNBOOK — manual guest restore (pct restore), and the bind mounts it drags with it

Status: operational procedure. Written 2026-07-28 to close F-OPS (Campaign 8). Audience: operator, at a keyboard, usually during a real disaster recovery.

The hazard in one sentence: pct restore recreates the guest from the archive's own config, which includes the source guest's mpN bind mounts — so the restored guest can come up bound to a host path that does not exist on this host, or, worse, to a path that exists and belongs to something else.

This is not hypothetical and it is not rare: the agent's own restore paths neutralise these mountpoints deliberately (bindMountOverrides in restoretest.go, RestoreLXC{MountOverrides:…} — see audits/SPIKE-dr-bindmount-source-2026-07-07.md). A manual pct restore has no such protection. The automation is safe; the human path is the one that needs this page.

Timing makes it worse: a manual restore happens during an incident, under pressure, often on a different host from the one the archive came from.

---

1. What a Felhom guest's mounts actually look like

From live guest 9201 (2026-07-28) — the shape you should expect:

rootfs: local-lvm:vm-9201-disk-0,size=32G
mp0:    local-lvm:vm-9201-disk-1,mp=/var/lib/docker,backup=1,size=50G
mp1:    local-lvm:vm-9201-disk-2,mp=/mnt/sys_drive,backup=1,size=20G
mp8:    /mnt/felhom-drives,mp=/mnt/felhom-drives          ← HOST BIND
mp9:    /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1   ← HOST BIND
hookscript: local:snippets/felhom-guest-hook.sh

Two kinds of mpN, and only one is dangerous:

kind	looks like	on restore
Volume mount (mp0, mp1)	starts with a storage ID — local-lvm:vm-…	restored from the archive. Fine.
Bind mount (mp8, mp9)	starts with an absolute host path — /mnt/…, /var/lib/…	NOT in the archive. The path is taken as-is on the target host.

The bind mounts are structural constants of the Felhom topology, not per-customer data — mp8 is the shared drive parent, mp9 is the per-guest bootstrap dir. That is what makes them safe to rewrite by hand: you are restoring a known layout, not guessing.

mp9 is the one that bites. It embeds the source VMID: /var/lib/felhom-agent/guests/<SOURCE-VMID>/bootstrap. Restore to a different VMID and the guest either finds nothing there or — if that VMID exists on this host — reads another guest's bootstrap config, including its credentials.

---

2. Before you start — three checks

Run these on the target host, and write the answers down.

# 1. What does the archive think its mounts are? (read the config WITHOUT restoring)
#    PBS:
proxmox-backup-client restore <snapshot> pct.conf - --repository <repo> 2>/dev/null | grep -E '^(mp|rootfs)'
#    vzdump tarball:
tar -xOf /path/to/vzdump-lxc-<vmid>-<ts>.tar.zst ./etc/vzdump/pct.conf 2>/dev/null | grep -E '^(mp|rootfs)'

# 2. Does every BIND path exist on THIS host?
ls -ld /mnt/felhom-drives /var/lib/felhom-agent/guests/<TARGET-VMID>/bootstrap

# 3. Is the target VMID free, and is the source VMID something else here?
pct list | awk '{print $1}' | grep -x -e <TARGET-VMID> -e <SOURCE-VMID>

Stop and think if: a bind path is missing, or the source VMID is a live guest on this host.

---

3. Restore, then fix the binds BEFORE first boot

Restore without starting the guest. pct restore does not auto-start, but never pass anything that would, and do not pct start until §4 passes.

pct restore <TARGET-VMID> <volid> --storage <storage> --unprivileged 1

Then strip and re-add the binds. Do this even when the VMID is unchanged — it costs seconds and it is the whole point of this page:

# strip every BIND mountpoint the archive carried (leave the volume mounts mp0/mp1 alone)
pct set <TARGET-VMID> --delete mp8
pct set <TARGET-VMID> --delete mp9

# re-add them for THIS host and THIS vmid
mkdir -p /var/lib/felhom-agent/guests/<TARGET-VMID>/bootstrap
pct set <TARGET-VMID> -mp8 /mnt/felhom-drives,mp=/mnt/felhom-drives
pct set <TARGET-VMID> -mp9 /var/lib/felhom-agent/guests/<TARGET-VMID>/bootstrap,mp=/etc/felhom-bootstrap,ro=1

The bootstrap dir must be populated for the guest to enrol. On a host running the agent, the provisioning back-half owns that directory — prefer letting the agent write it to hand-copying a bootstrap.json between guests. Never copy one from another customer's guest: it carries that customer's tokens.

Also check the hookscript line: hookscript: local:snippets/felhom-guest-hook.sh must point at a snippet that exists on this host, or the guest will refuse to start.

ls -l /var/lib/vz/snippets/felhom-guest-hook.sh || pct set <TARGET-VMID> --delete hookscript

---

4. Verify before starting — the positive check

Do not accept "no error" as evidence. Assert what must be true:

# every mpN is either a storage volume or a bind whose host path EXISTS on this host
pct config <TARGET-VMID> | grep -E '^mp[0-9]+:' | while read -r line; do
  src=${line#*: }; src=${src%%,*}
  case "$src" in
    /*) [ -e "$src" ] && echo "OK   bind $src" || echo "MISSING bind $src  <-- fix before start" ;;
    *)  echo "OK   volume $src" ;;
  esac
done

# no mpN may still reference the SOURCE vmid
pct config <TARGET-VMID> | grep -E "guests/<SOURCE-VMID>/" && echo "STILL POINTS AT THE SOURCE GUEST"

Only when every line reads OK and the source-VMID grep is empty:

pct start <TARGET-VMID>
pct exec <TARGET-VMID> -- docker ps --format '{{.Names}} {{.Status}}'

---

5. If the guest is already running with the wrong binds

Stop it before touching the mounts — a live bind swap under a running container set is how data ends up half-written to two places.

pct stop <TARGET-VMID>
# ...§3 fixes...
pct start <TARGET-VMID>

---

What this page deliberately does NOT do

It does not build tooling. The agent already neutralises binds on its own restore paths; wrapping the manual path in a script would create a second implementation of that logic, and two implementations of one invariant is how they drift. If the manual path becomes routine, the correct move is to route it through the agent, not to script around it.