felhom.eu/documentation/runbooks/os-updates-test-waits.md

RUNBOOK — test waits for OS-update approvals, and how they end

Design: architecture/11-os-updates.md §5.3, §5.3.1, §5.8. Row R-859. Hub v0.133.0.

The ruled waits

• Guest and host fast lane: every ring-0 box runs the set healthy for 24 h and through 1 night run → the hub approves it automatically.
• Docker engine set: every ring-0 box ran it in 2 healthy night Docker steps → the operator's "Approve Docker set" button works.

A test wait (only for a test, never to ship)

Set one of these env vars on the hub Deployment (via manifests/hub.yaml, synced), restart: OS_APPROVE_AFTER=<duration>, OS_APPROVE_NIGHTS=<n>, OS_DOCKER_APPROVE_NIGHTS=<n>. The hub logs TEST CONFIGURATION at start.

The rule (R-859): test approvals end with the test. Every approval made while ANY of the three is set is stored with a test mark (amber "TEST approval" on the System page). At the next start WITHOUT the overrides, every test approval that no real approval has superseded is cancelled: no ring-1 box installs it from then on (ring-1 boxes are bumped), and the operator gets an os_release_cancelled mail. What boxes already installed stays. The same set is approved again by the ruled wait, as a real release.

So: remove the override and restart the hub as the last step of every test. Leaving it set leaves the test approvals in force for real boxes.

The 2026-10-04 fact

On 2026-10-04 no release could pass the ruled 24 h + 1 night, so the guest and host sets were approved under the test wait (12:39 / 12:41 UTC, 1.5 h after first seen). Tester 2 (bound 16:06 UTC) installed both on its first night run (16:24 / 16:25 UTC): 49 guest and 106 host packages. Why the risk was small: ring 0 (both demo boxes) had run the same sets healthy since 11:07 / 12:24 UTC and kept running them; the packages are Debian (Security) fixes only; Tester 2 reported both runs applied, healthy. Hub v0.133.0's one-time backfill marked those two automatic approvals as test approvals and cancelled them at its start (2026-10-04 18:20 UTC, with two older ones of the same day). The operator's own Docker button approval of that day stays in force (decided by CC unattended — operator may reverse).