R-101 + F-DIAG closed, F-OPS documented (manual-restore runbook)
This commit is contained in:
@@ -0,0 +1,156 @@
|
||||
# RUNBOOK — manual guest restore (`pct restore`), and the bind mounts it drags with it
|
||||
|
||||
**Status:** operational procedure. Written 2026-07-28 to close **F-OPS** (Campaign 8).
|
||||
**Audience:** operator, at a keyboard, usually during a real disaster recovery.
|
||||
|
||||
> **The hazard in one sentence:** `pct restore` recreates the guest from the archive's **own config**,
|
||||
> which includes the source guest's `mpN` bind mounts — so the restored guest can come up bound to a
|
||||
> host path that does not exist on this host, or, worse, to a path that exists and belongs to
|
||||
> **something else**.
|
||||
|
||||
This is not hypothetical and it is not rare: the agent's own restore paths neutralise these mountpoints
|
||||
deliberately (`bindMountOverrides` in `restoretest.go`, `RestoreLXC{MountOverrides:…}` — see
|
||||
`audits/SPIKE-dr-bindmount-source-2026-07-07.md`). A **manual** `pct restore` has no such protection.
|
||||
The automation is safe; the human path is the one that needs this page.
|
||||
|
||||
Timing makes it worse: a manual restore happens during an incident, under pressure, often on a
|
||||
*different* host from the one the archive came from.
|
||||
|
||||
---
|
||||
|
||||
## 1. What a Felhom guest's mounts actually look like
|
||||
|
||||
From live guest 9201 (2026-07-28) — the shape you should expect:
|
||||
|
||||
```
|
||||
rootfs: local-lvm:vm-9201-disk-0,size=32G
|
||||
mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/docker,backup=1,size=50G
|
||||
mp1: local-lvm:vm-9201-disk-2,mp=/mnt/sys_drive,backup=1,size=20G
|
||||
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives ← HOST BIND
|
||||
mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1 ← HOST BIND
|
||||
hookscript: local:snippets/felhom-guest-hook.sh
|
||||
```
|
||||
|
||||
**Two kinds of `mpN`, and only one is dangerous:**
|
||||
|
||||
| kind | looks like | on restore |
|
||||
|---|---|---|
|
||||
| **Volume mount** (`mp0`, `mp1`) | starts with a storage ID — `local-lvm:vm-…` | restored from the archive. Fine. |
|
||||
| **Bind mount** (`mp8`, `mp9`) | starts with an absolute **host path** — `/mnt/…`, `/var/lib/…` | **NOT in the archive.** The path is taken as-is on the target host. |
|
||||
|
||||
The bind mounts are **structural constants** of the Felhom topology, not per-customer data — `mp8` is
|
||||
the shared drive parent, `mp9` is the per-guest bootstrap dir. That is what makes them safe to rewrite
|
||||
by hand: you are restoring a known layout, not guessing.
|
||||
|
||||
**`mp9` is the one that bites.** It embeds the **source** VMID:
|
||||
`/var/lib/felhom-agent/guests/<SOURCE-VMID>/bootstrap`. Restore to a different VMID and the guest
|
||||
either finds nothing there or — if that VMID exists on this host — reads **another guest's bootstrap
|
||||
config, including its credentials**.
|
||||
|
||||
---
|
||||
|
||||
## 2. Before you start — three checks
|
||||
|
||||
Run these on the **target** host, and write the answers down.
|
||||
|
||||
```bash
|
||||
# 1. What does the archive think its mounts are? (read the config WITHOUT restoring)
|
||||
# PBS:
|
||||
proxmox-backup-client restore <snapshot> pct.conf - --repository <repo> 2>/dev/null | grep -E '^(mp|rootfs)'
|
||||
# vzdump tarball:
|
||||
tar -xOf /path/to/vzdump-lxc-<vmid>-<ts>.tar.zst ./etc/vzdump/pct.conf 2>/dev/null | grep -E '^(mp|rootfs)'
|
||||
|
||||
# 2. Does every BIND path exist on THIS host?
|
||||
ls -ld /mnt/felhom-drives /var/lib/felhom-agent/guests/<TARGET-VMID>/bootstrap
|
||||
|
||||
# 3. Is the target VMID free, and is the source VMID something else here?
|
||||
pct list | awk '{print $1}' | grep -x -e <TARGET-VMID> -e <SOURCE-VMID>
|
||||
```
|
||||
|
||||
**Stop and think if:** a bind path is missing, or the source VMID is a live guest on this host.
|
||||
|
||||
---
|
||||
|
||||
## 3. Restore, then fix the binds BEFORE first boot
|
||||
|
||||
Restore **without starting** the guest. `pct restore` does not auto-start, but never pass anything that
|
||||
would, and do not `pct start` until §4 passes.
|
||||
|
||||
```bash
|
||||
pct restore <TARGET-VMID> <volid> --storage <storage> --unprivileged 1
|
||||
```
|
||||
|
||||
Then strip and re-add the binds. Do this even when the VMID is unchanged — it costs seconds and it is
|
||||
the whole point of this page:
|
||||
|
||||
```bash
|
||||
# strip every BIND mountpoint the archive carried (leave the volume mounts mp0/mp1 alone)
|
||||
pct set <TARGET-VMID> --delete mp8
|
||||
pct set <TARGET-VMID> --delete mp9
|
||||
|
||||
# re-add them for THIS host and THIS vmid
|
||||
mkdir -p /var/lib/felhom-agent/guests/<TARGET-VMID>/bootstrap
|
||||
pct set <TARGET-VMID> -mp8 /mnt/felhom-drives,mp=/mnt/felhom-drives
|
||||
pct set <TARGET-VMID> -mp9 /var/lib/felhom-agent/guests/<TARGET-VMID>/bootstrap,mp=/etc/felhom-bootstrap,ro=1
|
||||
```
|
||||
|
||||
> The bootstrap dir must be populated for the guest to enrol. On a host running the agent, the
|
||||
> provisioning back-half owns that directory — prefer letting the agent write it to hand-copying a
|
||||
> `bootstrap.json` between guests. **Never** copy one from another customer's guest: it carries that
|
||||
> customer's tokens.
|
||||
|
||||
Also check the hookscript line: `hookscript: local:snippets/felhom-guest-hook.sh` must point at a
|
||||
snippet that exists **on this host**, or the guest will refuse to start.
|
||||
|
||||
```bash
|
||||
ls -l /var/lib/vz/snippets/felhom-guest-hook.sh || pct set <TARGET-VMID> --delete hookscript
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Verify before starting — the positive check
|
||||
|
||||
Do not accept "no error" as evidence. Assert what must be **true**:
|
||||
|
||||
```bash
|
||||
# every mpN is either a storage volume or a bind whose host path EXISTS on this host
|
||||
pct config <TARGET-VMID> | grep -E '^mp[0-9]+:' | while read -r line; do
|
||||
src=${line#*: }; src=${src%%,*}
|
||||
case "$src" in
|
||||
/*) [ -e "$src" ] && echo "OK bind $src" || echo "MISSING bind $src <-- fix before start" ;;
|
||||
*) echo "OK volume $src" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# no mpN may still reference the SOURCE vmid
|
||||
pct config <TARGET-VMID> | grep -E "guests/<SOURCE-VMID>/" && echo "STILL POINTS AT THE SOURCE GUEST"
|
||||
```
|
||||
|
||||
Only when every line reads `OK` and the source-VMID grep is empty:
|
||||
|
||||
```bash
|
||||
pct start <TARGET-VMID>
|
||||
pct exec <TARGET-VMID> -- docker ps --format '{{.Names}} {{.Status}}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. If the guest is already running with the wrong binds
|
||||
|
||||
Stop it before touching the mounts — a live bind swap under a running container set is how data ends up
|
||||
half-written to two places.
|
||||
|
||||
```bash
|
||||
pct stop <TARGET-VMID>
|
||||
# ...§3 fixes...
|
||||
pct start <TARGET-VMID>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What this page deliberately does NOT do
|
||||
|
||||
**It does not build tooling.** The agent already neutralises binds on its own restore paths; wrapping
|
||||
the manual path in a script would create a second implementation of that logic, and two implementations
|
||||
of one invariant is how they drift. If the manual path becomes routine, the correct move is to route it
|
||||
through the agent, not to script around it.
|
||||
Reference in New Issue
Block a user