# Put one host package back by hand (OS updates, host fast lane)

> **When:** the hub mailed `os_update_health_failed` for the **host** layer (the box's base system), or the operator
> sees a host problem that started with a host OS pass, and ONE package is the suspect.
> **Who:** the operator, or CC with the operator's word, as root on the box's Proxmox host (`ssh <host>`).
> **Owner design:** `architecture/11-os-updates.md` §5.6 (host row), §8 step 3. **Proved** on demo-hp 2026-10-04 with
> `tzdata` (2026c → 2026b, held; then released and re-installed by the next ring-0 pass) and again on demo-felhom for
> the ring-1 test — evidence `audits/os-host-lane-2026-10-04/partB/undo/` and `partB/ring1/`.

There is **no automatic undo** for the host. A host cannot be snapshotted the way the old design assumed, and the
fast lane only ever installs Debian / Debian-Security packages, never a kernel, boot or firmware package (wrapper
refusals R14, R12). So the undo is small: install the previous version of the suspect package from
`snapshot.debian.org`, which keeps every version Debian ever published.

## 1. Find the suspect and its previous version

```bash
# the host pass's own apt run (Requested-By: felhom-agent); each line "name:arch (old, new)"
grep -B2 -A6 "Requested-By: felhom-agent" /var/log/apt/history.log | tail -20
PKG=<name>; OLD=<old version from that line>
```

## 2. Pick the snapshot timestamp

- **Normal case:** the timestamp of the **previous host release** — the hub fleet view (`GET /os/fleet`, the box's
  host line names its release; the release's approval time IS its snapshot time, `YYYYMMDDTHHMMSSZ`). At that time
  the old version was the current one.
- **No previous host release** (a ring-0 box, or the first host pass): ask snapshot.debian.org when the **NEW**
  (installed) version first appeared, and use that time. At that moment the suite still carried the old version.
  **Do not use the OLD version's `first_seen`:** that is when it reached Debian *unstable*, and `trixie` may not
  have had it yet — measured on demo-hp 2026-10-04: `eject 2.41-5` at its own `first_seen` → `Version not found`.

```bash
NEW=$(dpkg-query -W -f='${Version}' "$PKG")
curl -s "https://snapshot.debian.org/mr/binary/$PKG/$NEW/binfiles?fileinfo=1" | python3 -c '
import json,sys; d=json.load(sys.stdin)
print(sorted(f["first_seen"] for v in d["fileinfo"].values() for f in v)[0])'
# → e.g. 20260831T204404Z
TS=<that timestamp>
```

## 3. Install the old version from the snapshot, then remove the snapshot source

```bash
. /etc/os-release
cat > /etc/apt/sources.list.d/felhom-undo-snapshot.list <<EOF
deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/$TS $VERSION_CODENAME main
deb [check-valid-until=no] http://snapshot.debian.org/archive/debian-security/$TS $VERSION_CODENAME-security main
EOF
apt-get -q update
apt-get -s install --allow-downgrades "$PKG=$OLD"     # READ the simulation: only $PKG may change. If apt
                                                       # refuses, the package pins its siblings to the SAME
                                                       # version (measured: eject needs libmount1 = 2.41-5) —
                                                       # list them all in one command, or stop.
apt-get -y install --allow-downgrades "$PKG=$OLD"
apt-mark hold "$PKG"                                   # or the next ring-0 night installs the new one again
rm -f /etc/apt/sources.list.d/felhom-undo-snapshot.list
apt-get -q update
dpkg -s "$PKG" | grep -E "^(Status|Version)"
```

**The hold is the important line.** Without it the next night pass (ring 0) or the next approved release (ring 1)
installs the new version again. Write the hold into the register row that tracks the incident; take it off
(`apt-mark unhold "$PKG"`) when a fixed version is out.

## 4. Check the box

- `systemctl is-active pveproxy pvedaemon pvestatd pve-cluster felhom-agent` → all `active`.
- `pct status <customer vmid>` → `running`.
- The host health rule (`11` §8.2) is what the leg checks; run the debug action to see it pass:
  `sudo -u felhom-agent /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=os-update -vmid <vmid>`
  — **note: on ring 0 this also installs every pending fast-lane fix.** A held package is NOT reported as pending
  at all (measured on demo-hp: `pending` stayed 78 with `tzdata` held) — **the hub cannot see a hold**, so the hold
  lives only in the register row (R-848).

## What this runbook does NOT cover

- A kernel: the fast lane never installs one (R14). The kernel lane is `11` §5.6 (not built).
- A Proxmox-origin package (pve-*, qemu-server, …): never installed by the fast lane (R12/origin rule). Proxmox's
  own repository keeps old versions; that undo is not written yet.
- The customer guest: its undo is last night's whole-guest backup (decision 81).