felhom.eu/REPORT-day-2026-10-23.md

# REPORT — day 2026-10-23: the releases and the DooPlex hour (the operator's „go")

Design read: `07-backup-architecture.md` §6.3/§8, `09-update-architecture.md` §3 decision 250, `03-host-agent.md`,
`05-hub-architecture.md` §16.3, `runbooks/total-loss-of-dooplex.md`, `audits/dooplex-survival-2026-10-22/PLAN.md`.
Evidence: `documentation/audits/day-2026-10-23/01`–`14`.

## Five lines

1. Released: controller 0.340.0 (global floor), agent 0.163.0 (vouched by the operator), installer 1.36.0, hub 0.167.0,
   the website with D36's two sentences.
2. Root SSH from the LAN is key-only on all three test boxes; new installs get it from installer 1.36.0.
3. The hub database now leaves every 6 hours (restored back byte-identical); the released images and agent packages
   leave nightly to ep0 (pulled back, every sha256 OK, one image loaded on a scratch box).
4. One false failure mail at 10:28 (my install of the 6-hourly copy) — filed R-1234, fixed and red-proved the same hour.
5. Register 191 → 184; the operator: print the sheet, Hosts page clean-up, D37 + D38.

## Per box — versions before → after, and the SSH checks

| Box | controller | agent | root-file bundle | `sshd -T` root | sshd reloaded | key login | felhom-sshd | swap = RAM (9201) |
|---|---|---|---|---|---|---|---|---|
| Tester 1 | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 (via 0.163.0-step1) | without-password | yes, 09:27:38 | ok | active, `sshd -t` ok | 4096/4096 |
| demo-hp | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 | without-password | yes, 09:19:44 | ok | active, `sshd -t` ok | 25898/25898 |
| demo-felhom | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 | without-password | yes, 09:24:42 | ok | active, `sshd -t` ok | 12288/12288 |
| hub | 0.165.0 → 0.167.0 | | | | | | | |

Not proven (said plainly): the real root password over SSH (CC does not hold it — a dummy one is refused, but OpenSSH logs
only „Failed password"); a felhom-sshd connection (it admits only the operator's tunnel peer); the physical console (the
drop-in does not touch it; `passwd -S root` = P).

## Digests, sha256, CI by commit

| What | Value |
|---|---|
| controller 0.340.0 | `sha256:a8d1c6458da4ea9d667f928498656a291771e0b3756b20f2fcc8c1cf9d516621` — `caaa86e`, CI 2245 success |
| agent 0.163.0 binary / bundle | `561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c` / `3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8` — `1109092` CI 2246, `b4721ee` CI 2249 success |
| step bundle 0.163.0-step1 | `6cbc066902d5f3294765d5c7dc7113166c1dca25e994dbc36bfc5f139bb61fcf` |
| installer 1.36.0 (served = tag) | `de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec` — `83388c08` CI 2247 success; pins `2aebf1bd` |
| hub 0.167.0 | `sha256:177860a996cb285d935777d6ffdf85725f6cf4489885f54fd3bf8135c68041a0` — built at `d306b8c2` (hub/ = `be7e97dc`, CI 2250 success; `d306b8c2` CI 2251 success); manifest `34979d5d` |
| website | `0bb57a4f` |
| D34 fix (R-1234) | `52af38df` |
| prometheus rules | homelab-manifests `1bbef892` |

CI of the last commits (`2aebf1bd`, `0bb57a4f`, `34979d5d`, `52af38df` and this close-out): see the final message.

## Parts

- **1 — controller 0.340.0:** 9202 smoke, per-customer floors, three boxes green, box walk + https-only rc 0, global floor
  06:41 UTC (MinAgent 0.158.1), read back with the WAL (`01`–`06`).
- **2 — agent + installer:** release, a step bundle because the D4 drop-in is a new path (R-880), three signed jobs per
  box (`07`); the operator's vouch read back from the DB and the live page (`10`); installer tag, both pins, a
  webpage-only sync, the served script = the tag (`11`).
- **3 — hub 0.167.0:** built from a clean clone; on the operator's word it also carries D34's 6-hourly snapshot (asked
  before the build; not soaked); hub-only sync; 30-minute watch: 0 hub events, 0 errors, all boxes reporting, no hub mail
  (`14`).
- **4 — website:** D3/D5 patch + D36 HU + EN; site gates; published; read back live: new text present, old gone (control:
  the previous commit), screenshots at 390 and 1366 px, no sideways scroll (`12-website-readback/`).
- **5 — D34:** retention unchanged, server-side on ep0 (`prune-operator-hubdb` 03:45, keep-daily 14, keep-weekly 8 — the
  current day's up to 4 copies until the next prune). Hand run 501.6 MiB / 24.5 MiB uploaded in 4 s; the newest copy
  restored from ep0 with the read-only token = the pod's snapshot by sha256 `a9ff4c8a…`, integrity ok, 4 hosts, holds
  today's vouch. Alarm 13 h. Runbook updated (`13`).
- **6 — D33:** ep0 before: `/` 14 GB free (the data disk's root IS the PBS datastore — not used). One user
  `felhom-relcopy` (locked, rrsync-confined: shell and `..` refused), one directory `/srv/felhom-release-copy`. Cap 3 GB,
  enforced by the sender (ep0 has no quota). **skopeo, not `docker save`:** `dir:` keeps the registry's bytes, so a push
  back keeps the digest; no Docker daemon. A read-only registry token of its own, minted in the Gitea pod straight into a
  0600 file. First run 676 MB, 91 s; pulled back: 141 files OK, 9 images re-verified (digests match independent records),
  controller 0.340.0 loaded on 9202 (ID = config digest). Alarm `FelhomReleaseCopyStale`. Restore steps in
  `total-loss-of-dooplex.md` (`08`).
- **7 — D32:** the sheet is names and places only, HU + EN, one A4 (rendered: 1 page; a copy in the operator's home
  folder); every place checked to exist. demo-hp: three superseded ISOs removed by name, `/` 8.5 → 14 GB free (`09`).
  R-548: the box half done, the product fix proposed (R-921's pattern).
- **8 — close-out:** CONTEXT rulings, STATUS (decide rows D32–D36 → history, D37/D38 new, the do list, D35 dated
  2026-10-30), REPORTs and CHANGELOGs; the system poster's facts file updated, its hub-DB label edited (text), and a new
  arrow asked for in STATUS („System poster needs a refresh").

## Register — 191 → 184; opened 4, closed 11

Opened: R-1234 (closed the same hour), R-1235, R-1236, R-1237. Closed: R-49, R-982, R-985, R-991, R-1204, R-1205,
R-1225, R-1226, R-1227, R-1230, R-1234. Updated: R-1214 (released; dated check 2026-10-24), R-1187, R-1189, R-232,
R-548, R-923, R-924, R-1228 (D35, dated 2026-10-30), R-1183's check text (four snapshot slots).

## Observations

- The D34 first push refused the hub's own catch-up and mailed admin@ — FILED: R-1234 (closed, fixed `52af38df`).
- The release copy is proven to load, not to restore a registry; it follows the newest versions, not the floor/vouch; no
  golden — FILED: R-1235.
- Gitea OOM-killed at its 1 GiB limit, the last at 06:41:10 UTC during the release — FILED: R-1236 (STATUS D37).
- The night sweep's mail: one-time test-box codes in one session record — FILED: R-1237.
- The hub DB snapshot is 501 MiB, not 24 MB — NOT-A-FINDING: 24 MB is the compressed upload, which is what ep0 stores.
- felhom-sshd does not answer from the LAN — NOT-A-FINDING: its firewall admits only the operator peer (H1, by design).
- A dummy root password logs „Failed password", not a policy refusal — NOT-A-FINDING: OpenSSH's behaviour under
  `prohibit-password`; the policy is proven by `sshd -T` and the reload.
- The vouch left `artifact_wrapper_sha256` empty — NOT-A-FINDING: an optional field; the wrapper rides the vouched bundle.
- Guests 9202, 9401 (demo-hp) and a transient 990000 (Tester 1) have swap ≠ RAM — NOT-A-FINDING: outside the felhom pool.
- `known_hosts` holds an older key for 192.168.0.155 (demo-hp's guest address) — NOT-A-FINDING: guests are reached by
  `pct exec`; nothing was changed.
- The D4 drop-in needed a step bundle — NOT-A-FINDING: the documented R-880 path for a new bundle path.

## Teardown — three layers

Machine: 9202 — the test image untagged, the tar removed. Host: demo-hp `/tmp` tar removed; three ISOs removed on purpose
(Part 7); DooPlex: the read-back copies and the hub-DB scratch restore shredded/removed. Hub: nothing provisioned; the
per-customer floors cleared after the global floor. Left on purpose: the ep0 user + directory, the DooPlex release-copy
job, the 6-hourly hub-DB timer, both alarms.