admin
76ff0654ea
gates / gates (push) Successful in 30s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
82 lines
4.9 KiB
Markdown
82 lines
4.9 KiB
Markdown
# Put one host package back by hand (OS updates, host fast lane)
|
|
|
|
> **When:** the hub mailed `os_update_health_failed` for the **host** layer (the box's base system), or the operator
|
|
> sees a host problem that started with a host OS pass, and ONE package is the suspect.
|
|
> **Who:** the operator, or CC with the operator's word, as root on the box's Proxmox host (`ssh <host>`).
|
|
> **Owner design:** `architecture/11-os-updates.md` §5.6 (host row), §8 step 3. **Proved** on demo-hp 2026-10-04 with
|
|
> `tzdata` (2026c → 2026b, held; then released and re-installed by the next ring-0 pass) and again on demo-felhom for
|
|
> the ring-1 test — evidence `audits/os-host-lane-2026-10-04/partB/undo/` and `partB/ring1/`.
|
|
|
|
There is **no automatic undo** for the host. A host cannot be snapshotted the way the old design assumed, and the
|
|
fast lane only ever installs Debian / Debian-Security packages, never a kernel, boot or firmware package (wrapper
|
|
refusals R14, R12). So the undo is small: install the previous version of the suspect package from
|
|
`snapshot.debian.org`, which keeps every version Debian ever published.
|
|
|
|
## 1. Find the suspect and its previous version
|
|
|
|
```bash
|
|
# the host pass's own apt run (Requested-By: felhom-agent); each line "name:arch (old, new)"
|
|
grep -B2 -A6 "Requested-By: felhom-agent" /var/log/apt/history.log | tail -20
|
|
PKG=<name>; OLD=<old version from that line>
|
|
```
|
|
|
|
## 2. Pick the snapshot timestamp
|
|
|
|
- **Normal case:** the timestamp of the **previous host release** — the hub fleet view (`GET /os/fleet`, the box's
|
|
host line names its release; the release's approval time IS its snapshot time, `YYYYMMDDTHHMMSSZ`). At that time
|
|
the old version was the current one.
|
|
- **No previous host release** (a ring-0 box, or the first host pass): ask snapshot.debian.org when the **NEW**
|
|
(installed) version first appeared, and use that time. At that moment the suite still carried the old version.
|
|
**Do not use the OLD version's `first_seen`:** that is when it reached Debian *unstable*, and `trixie` may not
|
|
have had it yet — measured on demo-hp 2026-10-04: `eject 2.41-5` at its own `first_seen` → `Version not found`.
|
|
|
|
```bash
|
|
NEW=$(dpkg-query -W -f='${Version}' "$PKG")
|
|
curl -s "https://snapshot.debian.org/mr/binary/$PKG/$NEW/binfiles?fileinfo=1" | python3 -c '
|
|
import json,sys; d=json.load(sys.stdin)
|
|
print(sorted(f["first_seen"] for v in d["fileinfo"].values() for f in v)[0])'
|
|
# → e.g. 20260831T204404Z
|
|
TS=<that timestamp>
|
|
```
|
|
|
|
## 3. Install the old version from the snapshot, then remove the snapshot source
|
|
|
|
```bash
|
|
. /etc/os-release
|
|
cat > /etc/apt/sources.list.d/felhom-undo-snapshot.list <<EOF
|
|
deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/$TS $VERSION_CODENAME main
|
|
deb [check-valid-until=no] http://snapshot.debian.org/archive/debian-security/$TS $VERSION_CODENAME-security main
|
|
EOF
|
|
apt-get -q update
|
|
apt-get -s install --allow-downgrades "$PKG=$OLD" # READ the simulation: only $PKG may change. If apt
|
|
# refuses, the package pins its siblings to the SAME
|
|
# version (measured: eject needs libmount1 = 2.41-5) —
|
|
# list them all in one command, or stop.
|
|
apt-get -y install --allow-downgrades "$PKG=$OLD"
|
|
apt-mark hold "$PKG" # or the next ring-0 night installs the new one again
|
|
rm -f /etc/apt/sources.list.d/felhom-undo-snapshot.list
|
|
apt-get -q update
|
|
dpkg -s "$PKG" | grep -E "^(Status|Version)"
|
|
```
|
|
|
|
**The hold is the important line.** Without it the next night pass (ring 0) or the next approved release (ring 1)
|
|
installs the new version again. Write the hold into the register row that tracks the incident; take it off
|
|
(`apt-mark unhold "$PKG"`) when a fixed version is out.
|
|
|
|
## 4. Check the box
|
|
|
|
- `systemctl is-active pveproxy pvedaemon pvestatd pve-cluster felhom-agent` → all `active`.
|
|
- `pct status <customer vmid>` → `running`.
|
|
- The host health rule (`11` §8.2) is what the leg checks; run the debug action to see it pass:
|
|
`sudo -u felhom-agent /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=os-update -vmid <vmid>`
|
|
— **note: on ring 0 this also installs every pending fast-lane fix.** A held package is NOT reported as pending
|
|
at all (measured on demo-hp: `pending` stayed 78 with `tzdata` held) — **the hub cannot see a hold**, so the hold
|
|
lives only in the register row (R-848).
|
|
|
|
## What this runbook does NOT cover
|
|
|
|
- A kernel: the fast lane never installs one (R14). The kernel lane is `11` §5.6 (not built).
|
|
- A Proxmox-origin package (pve-*, qemu-server, …): never installed by the fast lane (R12/origin rule). Proxmox's
|
|
own repository keeps old versions; that undo is not written yet.
|
|
- The customer guest: its undo is last night's whole-guest backup (decision 81) — `os-updates-guest-undo.md` (proven 2026-10-04 night, 48 s down).
|