# RUNBOOK — manual guest restore (`pct restore`), and the bind mounts it drags with it

**Status:** operational procedure. Written 2026-07-28 to close **F-OPS** (Campaign 8).
**Audience:** operator, at a keyboard, usually during a real disaster recovery.

> **The hazard in one sentence:** `pct restore` recreates the guest from the archive's **own config**,
> which includes the source guest's `mpN` bind mounts — so the restored guest can come up bound to a
> host path that does not exist on this host, or, worse, to a path that exists and belongs to
> **something else**.

This is not hypothetical and it is not rare: the agent's own restore paths neutralise these mountpoints
deliberately (`bindMountOverrides` in `restoretest.go`, `RestoreLXC{MountOverrides:…}` — see
`audits/SPIKE-dr-bindmount-source-2026-07-07.md`). A **manual** `pct restore` has no such protection.
The automation is safe; the human path is the one that needs this page.

Timing makes it worse: a manual restore happens during an incident, under pressure, often on a
*different* host from the one the archive came from.

---

## 1. What a Felhom guest's mounts actually look like

From live guest 9201 (2026-07-28) — the shape you should expect:

```
rootfs: local-lvm:vm-9201-disk-0,size=32G
mp0:    local-lvm:vm-9201-disk-1,mp=/var/lib/docker,backup=1,size=50G
mp1:    local-lvm:vm-9201-disk-2,mp=/mnt/sys_drive,backup=1,size=20G
mp8:    /mnt/felhom-drives,mp=/mnt/felhom-drives          ← HOST BIND
mp9:    /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1   ← HOST BIND
hookscript: local:snippets/felhom-guest-hook.sh
```

**Two kinds of `mpN`, and only one is dangerous:**

| kind | looks like | on restore |
|---|---|---|
| **Volume mount** (`mp0`, `mp1`) | starts with a storage ID — `local-lvm:vm-…` | restored from the archive. Fine. |
| **Bind mount** (`mp8`, `mp9`) | starts with an absolute **host path** — `/mnt/…`, `/var/lib/…` | **NOT in the archive.** The path is taken as-is on the target host. |

The bind mounts are **structural constants** of the Felhom topology, not per-customer data — `mp8` is
the shared drive parent, `mp9` is the per-guest bootstrap dir. That is what makes them safe to rewrite
by hand: you are restoring a known layout, not guessing.

**`mp9` is the one that bites.** It embeds the **source** VMID:
`/var/lib/felhom-agent/guests/<SOURCE-VMID>/bootstrap`. Restore to a different VMID and the guest
either finds nothing there or — if that VMID exists on this host — reads **another guest's bootstrap
config, including its credentials**.

---

## 2. Before you start — three checks

Run these on the **target** host, and write the answers down.

```bash
# 1. What does the archive think its mounts are? (read the config WITHOUT restoring)
#    PBS:
proxmox-backup-client restore <snapshot> pct.conf - --repository <repo> 2>/dev/null | grep -E '^(mp|rootfs)'
#    vzdump tarball:
tar -xOf /path/to/vzdump-lxc-<vmid>-<ts>.tar.zst ./etc/vzdump/pct.conf 2>/dev/null | grep -E '^(mp|rootfs)'

# 2. Does every BIND path exist on THIS host?
ls -ld /mnt/felhom-drives /var/lib/felhom-agent/guests/<TARGET-VMID>/bootstrap

# 3. Is the target VMID free, and is the source VMID something else here?
pct list | awk '{print $1}' | grep -x -e <TARGET-VMID> -e <SOURCE-VMID>
```

**Stop and think if:** a bind path is missing, or the source VMID is a live guest on this host.

---

## 3. Restore, then fix the binds BEFORE first boot

Restore **without starting** the guest. `pct restore` does not auto-start, but never pass anything that
would, and do not `pct start` until §4 passes.

```bash
pct restore <TARGET-VMID> <volid> --storage <storage> --unprivileged 1
```

Then strip and re-add the binds. Do this even when the VMID is unchanged — it costs seconds and it is
the whole point of this page:

```bash
# strip every BIND mountpoint the archive carried (leave the volume mounts mp0/mp1 alone)
pct set <TARGET-VMID> --delete mp8
pct set <TARGET-VMID> --delete mp9

# re-add them for THIS host and THIS vmid
mkdir -p /var/lib/felhom-agent/guests/<TARGET-VMID>/bootstrap
pct set <TARGET-VMID> -mp8 /mnt/felhom-drives,mp=/mnt/felhom-drives
pct set <TARGET-VMID> -mp9 /var/lib/felhom-agent/guests/<TARGET-VMID>/bootstrap,mp=/etc/felhom-bootstrap,ro=1
```

> The bootstrap dir must be populated for the guest to enrol. On a host running the agent, the
> provisioning back-half owns that directory — prefer letting the agent write it to hand-copying a
> `bootstrap.json` between guests. **Never** copy one from another customer's guest: it carries that
> customer's tokens.

Also check the hookscript line: `hookscript: local:snippets/felhom-guest-hook.sh` must point at a
snippet that exists **on this host**, or the guest will refuse to start.

```bash
ls -l /var/lib/vz/snippets/felhom-guest-hook.sh || pct set <TARGET-VMID> --delete hookscript
```

---

## 4. Verify before starting — the positive check

Do not accept "no error" as evidence. Assert what must be **true**:

```bash
# every mpN is either a storage volume or a bind whose host path EXISTS on this host
pct config <TARGET-VMID> | grep -E '^mp[0-9]+:' | while read -r line; do
  src=${line#*: }; src=${src%%,*}
  case "$src" in
    /*) [ -e "$src" ] && echo "OK   bind $src" || echo "MISSING bind $src  <-- fix before start" ;;
    *)  echo "OK   volume $src" ;;
  esac
done

# no mpN may still reference the SOURCE vmid
pct config <TARGET-VMID> | grep -E "guests/<SOURCE-VMID>/" && echo "STILL POINTS AT THE SOURCE GUEST"
```

Only when every line reads `OK` and the source-VMID grep is empty:

```bash
pct start <TARGET-VMID>
pct exec <TARGET-VMID> -- docker ps --format '{{.Names}} {{.Status}}'
```

---

## 5. If the guest is already running with the wrong binds

Stop it before touching the mounts — a live bind swap under a running container set is how data ends up
half-written to two places.

```bash
pct stop <TARGET-VMID>
# ...§3 fixes...
pct start <TARGET-VMID>
```

---

## What this page deliberately does NOT do

**It does not build tooling.** The agent already neutralises binds on its own restore paths; wrapping
the manual path in a script would create a second implementation of that logic, and two implementations
of one invariant is how they drift. If the manual path becomes routine, the correct move is to route it
through the agent, not to script around it.
