REPORT — day 2026-10-23: the releases and the DooPlex hour (the operator's „go")

Design read: 07-backup-architecture.md §6.3/§8, 09-update-architecture.md §3 decision 250, 03-host-agent.md, 05-hub-architecture.md §16.3, runbooks/total-loss-of-dooplex.md, audits/dooplex-survival-2026-10-22/PLAN.md. Evidence: documentation/audits/day-2026-10-23/01–14.

Five lines

Released: controller 0.340.0 (global floor), agent 0.163.0 (vouched by the operator), installer 1.36.0, hub 0.167.0, the website with D36's two sentences.
Root SSH from the LAN is key-only on all three test boxes; new installs get it from installer 1.36.0.
The hub database now leaves every 6 hours (restored back byte-identical); the released images and agent packages leave nightly to ep0 (pulled back, every sha256 OK, one image loaded on a scratch box).
One false failure mail at 10:28 (my install of the 6-hourly copy) — filed R-1234, fixed and red-proved the same hour.
Register 191 → 184; the operator: print the sheet, Hosts page clean-up, D37 + D38.

Per box — versions before → after, and the SSH checks

Box	controller	agent	root-file bundle	`sshd -T` root	sshd reloaded	key login	felhom-sshd	swap = RAM (9201)
Tester 1	0.339.0 → 0.340.0	0.162.0 → 0.163.0	0.160.0 → 0.163.0 (via 0.163.0-step1)	without-password	yes, 09:27:38	ok	active, `sshd -t` ok	4096/4096
demo-hp	0.339.0 → 0.340.0	0.162.0 → 0.163.0	0.160.0 → 0.163.0	without-password	yes, 09:19:44	ok	active, `sshd -t` ok	25898/25898
demo-felhom	0.339.0 → 0.340.0	0.162.0 → 0.163.0	0.160.0 → 0.163.0	without-password	yes, 09:24:42	ok	active, `sshd -t` ok	12288/12288
hub	0.165.0 → 0.167.0

Not proven (said plainly): the real root password over SSH (CC does not hold it — a dummy one is refused, but OpenSSH logs only „Failed password"); a felhom-sshd connection (it admits only the operator's tunnel peer); the physical console (the drop-in does not touch it; passwd -S root = P).

Digests, sha256, CI by commit

What	Value
controller 0.340.0	`sha256:a8d1c6458da4ea9d667f928498656a291771e0b3756b20f2fcc8c1cf9d516621` — `caaa86e`, CI 2245 success
agent 0.163.0 binary / bundle	`561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c` / `3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8` — `1109092` CI 2246, `b4721ee` CI 2249 success
step bundle 0.163.0-step1	`6cbc066902d5f3294765d5c7dc7113166c1dca25e994dbc36bfc5f139bb61fcf`
installer 1.36.0 (served = tag)	`de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec` — `83388c08` CI 2247 success; pins `2aebf1bd`
hub 0.167.0	`sha256:177860a996cb285d935777d6ffdf85725f6cf4489885f54fd3bf8135c68041a0` — built at `d306b8c2` (hub/ = `be7e97dc`, CI 2250 success; `d306b8c2` CI 2251 success); manifest `34979d5d`
website	`0bb57a4f`
D34 fix (R-1234)	`52af38df`
prometheus rules	homelab-manifests `1bbef892`

CI of the last commits (2aebf1bd, 0bb57a4f, 34979d5d, 52af38df and this close-out): see the final message.

Parts

1 — controller 0.340.0: 9202 smoke, per-customer floors, three boxes green, box walk + https-only rc 0, global floor 06:41 UTC (MinAgent 0.158.1), read back with the WAL (01–06).
2 — agent + installer: release, a step bundle because the D4 drop-in is a new path (R-880), three signed jobs per box (07); the operator's vouch read back from the DB and the live page (10); installer tag, both pins, a webpage-only sync, the served script = the tag (11).
3 — hub 0.167.0: built from a clean clone; on the operator's word it also carries D34's 6-hourly snapshot (asked before the build; not soaked); hub-only sync; 30-minute watch: 0 hub events, 0 errors, all boxes reporting, no hub mail (14).
4 — website: D3/D5 patch + D36 HU + EN; site gates; published; read back live: new text present, old gone (control: the previous commit), screenshots at 390 and 1366 px, no sideways scroll (12-website-readback/).
5 — D34: retention unchanged, server-side on ep0 (prune-operator-hubdb 03:45, keep-daily 14, keep-weekly 8 — the current day's up to 4 copies until the next prune). Hand run 501.6 MiB / 24.5 MiB uploaded in 4 s; the newest copy restored from ep0 with the read-only token = the pod's snapshot by sha256 a9ff4c8a…, integrity ok, 4 hosts, holds today's vouch. Alarm 13 h. Runbook updated (13).
6 — D33: ep0 before: / 14 GB free (the data disk's root IS the PBS datastore — not used). One user felhom-relcopy (locked, rrsync-confined: shell and .. refused), one directory /srv/felhom-release-copy. Cap 3 GB, enforced by the sender (ep0 has no quota). skopeo, not docker save: dir: keeps the registry's bytes, so a push back keeps the digest; no Docker daemon. A read-only registry token of its own, minted in the Gitea pod straight into a 0600 file. First run 676 MB, 91 s; pulled back: 141 files OK, 9 images re-verified (digests match independent records), controller 0.340.0 loaded on 9202 (ID = config digest). Alarm FelhomReleaseCopyStale. Restore steps in total-loss-of-dooplex.md (08).
7 — D32: the sheet is names and places only, HU + EN, one A4 (rendered: 1 page; a copy in the operator's home folder); every place checked to exist. demo-hp: three superseded ISOs removed by name, / 8.5 → 14 GB free (09). R-548: the box half done, the product fix proposed (R-921's pattern).
8 — close-out: CONTEXT rulings, STATUS (decide rows D32–D36 → history, D37/D38 new, the do list, D35 dated 2026-10-30), REPORTs and CHANGELOGs; the system poster's facts file updated, its hub-DB label edited (text), and a new arrow asked for in STATUS („System poster needs a refresh").

Register — 191 → 184; opened 4, closed 11

Opened: R-1234 (closed the same hour), R-1235, R-1236, R-1237. Closed: R-49, R-982, R-985, R-991, R-1204, R-1205, R-1225, R-1226, R-1227, R-1230, R-1234. Updated: R-1214 (released; dated check 2026-10-24), R-1187, R-1189, R-232, R-548, R-923, R-924, R-1228 (D35, dated 2026-10-30), R-1183's check text (four snapshot slots).

Observations

The D34 first push refused the hub's own catch-up and mailed admin@ — FILED: R-1234 (closed, fixed 52af38df).
The release copy is proven to load, not to restore a registry; it follows the newest versions, not the floor/vouch; no golden — FILED: R-1235.
Gitea OOM-killed at its 1 GiB limit, the last at 06:41:10 UTC during the release — FILED: R-1236 (STATUS D37).
The night sweep's mail: one-time test-box codes in one session record — FILED: R-1237.
The hub DB snapshot is 501 MiB, not 24 MB — NOT-A-FINDING: 24 MB is the compressed upload, which is what ep0 stores.
felhom-sshd does not answer from the LAN — NOT-A-FINDING: its firewall admits only the operator peer (H1, by design).
A dummy root password logs „Failed password", not a policy refusal — NOT-A-FINDING: OpenSSH's behaviour under prohibit-password; the policy is proven by sshd -T and the reload.
The vouch left artifact_wrapper_sha256 empty — NOT-A-FINDING: an optional field; the wrapper rides the vouched bundle.
Guests 9202, 9401 (demo-hp) and a transient 990000 (Tester 1) have swap ≠ RAM — NOT-A-FINDING: outside the felhom pool.
known_hosts holds an older key for 192.168.0.155 (demo-hp's guest address) — NOT-A-FINDING: guests are reached by pct exec; nothing was changed.
The D4 drop-in needed a step bundle — NOT-A-FINDING: the documented R-880 path for a new bundle path.

Teardown — three layers

Machine: 9202 — the test image untagged, the tar removed. Host: demo-hp /tmp tar removed; three ISOs removed on purpose (Part 7); DooPlex: the read-back copies and the hub-DB scratch restore shredded/removed. Hub: nothing provisioned; the per-customer floors cleared after the global floor. Left on purpose: the ep0 user + directory, the DooPlex release-copy job, the 6-hourly hub-DB timer, both alarms.

8.0 KiB Raw Blame History Unescape Escape