Day 2026-10-23 close-out: rulings D31-D36 in CONTEXT; STATUS (releases done, do list, D35 dated, D37/D38); register 191 -> 184 (closed R-49 R-982 R-985 R-991 R-1204 R-1205 R-1225 R-1226 R-1227 R-1230 R-1234; opened R-1235 R-1236 R-1237); the break-glass sheet current (D32, one A4, HU+EN); runbooks for D33/D34; poster facts; evidence day-2026-10-23

2026-10-23 11:03:45 +02:00
parent 52af38df6f
commit 2c9152d912
36 changed files with 565 additions and 68 deletions
@@ -101,6 +101,7 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
  - one namespace **per household** (whole-guest copies, each household's own key),
    with a NEUTRAL note under it (not a red gap box): "By design: each household holds the key to its own copies; the operator cannot open them without their recovery code." — *fix (c), 2026-10-09: this is a property of the design, not a gap, and a red box said the opposite of what it means.*
  - the **`operator`** namespace: two groups — **"hub database"** and **"DooPlex copy"** (code, password manager, secrets, signing keys).
+  - outside PBS, a plain folder: the **release copy** (images + agent packages, D33).
  Prune jobs on ep0: households keep one copy per week for 2 weeks (`keep-weekly 2`, R-1162, 2026-10-19; every new household's job is made with its namespace, R-1165); `operator` keeps 14 daily + 8 weekly.
  Mark ep0 with a small shield: "**Protected** — holds the only off-premises copy of real household data."
 - **Hetzner Storage Box** (grey): the shared box for the households' per-app restic copies (Tier 3), one sub-account per household, add-only keys.
@@ -121,8 +122,9 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–

 Arrows from DooPlex (dashed, orange → ep0):
 - **00:20 "DooPlex copy"**: the code (614 MB), Gitea's database and settings, the **password manager**, the nightly secrets export, the **signing keys**. Encrypted on DooPlex with the **DooPlex off-site key**. The key that writes it **cannot delete** old copies. Tested back every Sunday.
- **02:30 "Hub database"**: encrypted with the **hub-DB key**. Tested back every Sunday at 04:30.
- **Left out on purpose:** the container registry (27.7 GB) — "rebuilt from the code".
+- **02:30, 08:30, 14:30, 20:30 "Hub database"** (every 6 hours since 2026-10-23, D34): encrypted with the **hub-DB key**. Tested back every Sunday at 04:30.
+- **01:40 "Release copy"** (since 2026-10-23, D33): the newest 3 released images of the controller, hub and samba, and the newest 5 agent packages — not encrypted (nothing secret), into a plain folder on ep0 (`/srv/felhom-release-copy`, its own limited user, outside PBS), about 0.7 GB, capped at 3 GB.
+- **Left out on purpose:** the rest of the container registry (older tags, test builds, non-Felhom images) — "rebuilt from the code".
 Arrow ep0 → DooPlex (dashed, 05:00): "**ep0-copy** pull: a second copy of ep0 at home. Keeps 8 weekly copies. A daily job removes a deleted customer's copy within 30 days."

 **The test machines** (small, plain tiles near DooPlex, labelled "disposable"):
@@ -133,7 +133,7 @@ code{font-family: ui-monospace, "SFMono-Regular", Menlo, monospace; font-size: 1
 <div style="position: absolute; left: 1720px; top: 706px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: center;">The hub never connects into a box.</div>
 <div style="position: absolute; left: 1720px; top: 784px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">00:20 DooPlex copy</div>
 <div style="position: absolute; left: 1720px; top: 824px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #a85400; text-align: center;">DooPlex off-site key</div>
-<div style="position: absolute; left: 1720px; top: 872px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">02:30 hub database</div>
+<div style="position: absolute; left: 1720px; top: 872px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">every 6 h: hub database</div>
 <div style="position: absolute; left: 1720px; top: 912px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #a85400; text-align: center;">hub-DB key</div>
 <div style="position: absolute; left: 1720px; top: 962px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #0a3d73; text-align: center;">05:00 ep0-copy pull</div>
 <div style="position: absolute; left: 1720px; top: 1002px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #0a3d73; text-align: center;">ciphertext only</div>
@@ -0,0 +1,6 @@
+== 9202 smoke 06:37:25
+switched to gitea.dooplex.hu/admin/felhom-controller:0.340.0 at 06:37:28
+healthy after ~10s: Up 7 seconds (healthy)
+2026/10/23 06:37:28 [INFO] [tier2] start-up: no leftover second-drive copy container
+errors, unhealthy+restarting: 0 0
+login page: 200
@@ -0,0 +1,3 @@
+tester-1: HTTP/1.1 303 See Other Location: /customers/tester-1?flash=floor_set
+demo-hp: HTTP/1.1 303 See Other Location: /customers/demo-hp?flash=floor_set
+demo-felhom: HTTP/1.1 303 See Other Location: /customers/demo-felhom?flash=floor_set
@@ -0,0 +1,15 @@
+### root@192.168.0.154
+containers=10 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.340.0 Up 53 seconds (healthy)
+errors=0
+copy-containers=0
+2026/10/23 06:38:21 [INFO] [tier2] start-up: no leftover second-drive copy container
+### hp
+containers=24 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.340.0 Up 54 seconds (healthy)
+errors=0
+copy-containers=0
+2026/10/23 06:38:22 [INFO] [tier2] start-up: no leftover second-drive copy container
+### felhom-pve
+containers=9 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.340.0 Up 58 seconds (healthy)
+errors=0
+copy-containers=0
+2026/10/23 06:38:05 [INFO] [tier2] start-up: no leftover second-drive copy container
@@ -0,0 +1,33 @@
+control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
+== tester-1 (enkicsifelhom.hu): 6 public name(s)
+   OK   control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
+   OK   http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
+   OK   http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
+   OK   http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
+   OK   http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
+   OK   http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
+== demo-hp (enkisfelhom.hu): 14 public name(s)
+   OK   http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
+   OK   http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
+   OK   http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
+   OK   http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
+   OK   http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
+   OK   http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
+   OK   http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
+   OK   http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
+   OK   http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
+   OK   http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
+   OK   http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
+   OK   http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
+   OK   http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
+   OK   http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
+== demo-felhom (demo-felhom.eu): 8 public name(s)
+   OK   http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
+   OK   http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
+   OK   http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
+   OK   http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
+   OK   http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
+   OK   http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
+   OK   http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
+   OK   http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
+== every public name redirects http -> https
@@ -0,0 +1,13 @@
+== tester-1 (dashboard login with the household password, the press a browser makes)
+python3: can't open file '/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/38fde0c6-e964-4ec1-b0c7-56e62d675225/scratchpad/n23/fileswalk.py': [Errno 2] No such file or directory
+== demo-hp (no session): GET /__files/open -> HTTP/2 302  location: /login
+== demo-felhom (no session): GET /__files/open -> HTTP/2 302  location: /login
+== tester-1 06:40:35
+Fájlok press: GET /__files/open -> ['HTTP/2 302'] https://files.enkicsifelhom.hu/_felhom/sso
+manual second-drive copy: 200 2. mentés elindítva
+2026/10/23 06:40:43 [INFO] [backup] Tier 2 copied bookstack → /mnt/felhom-drives/adatlemez/backups/secondary/bookstack (157.0 MB, 0 leg(s), 1s)
+2026/10/23 06:40:46 [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sys_drive/felhom-data/backups/secondary/paperless-ngx (85.9 MB, 1 leg(s), 2s) [SSD: state-only]
+2026/10/23 06:40:48 [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/adatlemez/backups/secondary/privatebin (16.4 KB, 0 leg(s), 0s)
+2026/10/23 06:40:48 [INFO] [backup] Tier 2 run complete: 3 app(s) processed (incl. volume-only — F6)
+copy-containers-left=0
+[exited with code 0]
@@ -0,0 +1,5 @@
+== global floor 06:41:04: HTTP/1.1 303 See Other Location: /configuration?flash=floor_set
+clear tester-1: HTTP/1.1 303 See Other Location: /customers/tester-1?flash=floor_set
+clear demo-hp: HTTP/1.1 303 See Other Location: /customers/demo-hp?flash=floor_set
+clear demo-felhom: HTTP/1.1 303 See Other Location: /customers/demo-felhom?flash=floor_set
+read back (with WAL): [('min_controller_version', '0.340.0', '2026-10-23 06:41:04'), ('min_controller_version_declared_min_agent', '0.340.0=0.158.1', '2026-10-23 06:41:04')]
@@ -0,0 +1,78 @@
+# Part 2 — agent 0.163.0 (operator go 2026-10-23), read 2026-10-23T07:28:53Z
+
+## release (scripts/release-agent.sh at felhom-agent 1109092, CI 2246 green; CHANGELOG b4721ee CI 2249 green)
+[release-agent] built ok: sha256 561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c
+[release-agent] config bundle built: sha256 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8
+[publish-agent] upload OK (HTTP 201)
+[publish-agent] round-trip GET verified (sha256 matches)
+[release-agent] config bundle published (HTTP 201)
+    sha256  : 561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c
+    bundle  : 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8   (felhom-config-bundle.json — vouch it with the agent)
+ * [new tag]         v0.163.0 -> v0.163.0
+
+## step bundle (R-880: the D4 drop-in is a NEW path; the boxes ran bundle 0.160.0 = 6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f)
+0.163.0-step1 = base 0.160.0 with only /usr/local/sbin/felhom-os-apply replaced; sha256 6cbc066902d5f3294765d5c7dc7113166c1dca25e994dbc36bfc5f139bb61fcf; PUT 201; read back by download: same sha
+
+## delivery: three signed jobs per box (felhom-opsign, key felhom-op-1): agent_update 06:48Z, agent_config_update 0.163.0-step1 06:59Z, agent_config_update 0.163.0 07:19Z
+
+## hp (192.168.0.104) 2026-10-23T07:26:29Z
+agent: felhom-agent 0.163.0
+bundle: 0.163.0 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 2026-10-23T07:19:44Z
+      1 BUNDLE DONE agent=0.163.0 written=1 same=30 self-check=ok
+      2 BUNDLE DONE agent=0.163.0-step1 written=1 same=29 self-check=ok
+guest 9201: memory: 25898 swap: 25898 
+guest 9202: memory: 25898 swap: 512 
+guest 9401: memory: 10240 swap: 0 
+drop-in: -rw-r--r-- 1 root root 690 Oct 23 09:19 /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf
+drop-in content: PermitRootLogin prohibit-password
+sshd -T: permitrootlogin without-password
+root password set (passwd -S): P
+felhom-sshd: active; its root policy: permitrootlogin without-password
+felhom-sshd listens: 0.0.0.0:8822 [::]:8822  config test: ok; firewall admits only: elements = { 10.77.0.250 }
+main sshd reloaded after the drop-in: Oct 23 09:19:44 demo-hp sshd[1010]: Received SIGHUP; restarting. Oct 23 09:19:44 demo-hp systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server. 
+pveproxy: active
+main sshd banner from DooPlex (:22): SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4
+key login: ok
+root password attempt rc=255 (non-zero = refused)
+Oct 23 09:26:37 demo-hp sshd-session[3026730]: Failed password for root from 192.168.0.180 port 34150 ssh2
+## root@192.168.0.154 (192.168.0.154) 2026-10-23T07:28:00Z
+agent: felhom-agent 0.163.0
+bundle: 0.163.0 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 2026-10-23T07:27:38Z
+      2 BUNDLE DONE agent=0.163.0 written=1 same=30 self-check=ok
+      2 BUNDLE DONE agent=0.163.0-step1 written=1 same=29 self-check=ok
+guest 9201: memory: 4096 swap: 4096 
+guest 990000: memory: 4096 swap: 512 
+drop-in: -rw-r--r-- 1 root root 690 Oct 23 09:27 /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf
+drop-in content: PermitRootLogin prohibit-password
+sshd -T: permitrootlogin without-password
+root password set (passwd -S): P
+felhom-sshd: active; its root policy: permitrootlogin without-password
+felhom-sshd listens: 0.0.0.0:8822 [::]:8822  config test: ok; firewall admits only: elements = { 10.77.0.250 }
+main sshd reloaded after the drop-in: Oct 23 09:27:38 felhom sshd[890]: Received SIGHUP; restarting. Oct 23 09:27:38 felhom systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server. 
+pveproxy: active
+main sshd banner from DooPlex (:22): SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4
+key login: ok
+root password attempt rc=255 (non-zero = refused)
+Oct 23 09:28:08 felhom sshd-session[4184378]: Failed password for root from 192.168.0.180 port 33442 ssh2
+## felhom-pve (192.168.0.162) 2026-10-23T07:28:11Z
+agent: felhom-agent 0.163.0
+bundle: 0.163.0 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 2026-10-23T07:24:43Z
+      2 BUNDLE DONE agent=0.163.0 written=1 same=30 self-check=ok
+      2 BUNDLE DONE agent=0.163.0-step1 written=1 same=29 self-check=ok
+guest 9201: memory: 12288 swap: 12288 
+drop-in: -rw-r--r-- 1 root root 690 Oct 23 09:24 /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf
+drop-in content: PermitRootLogin prohibit-password
+sshd -T: permitrootlogin without-password
+root password set (passwd -S): P
+felhom-sshd: active; its root policy: permitrootlogin without-password
+felhom-sshd listens: 0.0.0.0:8822 [::]:8822  config test: ok; firewall admits only: elements = { 10.77.0.250 }
+main sshd reloaded after the drop-in: Oct 23 09:24:42 demo-felhom sshd[1011]: Received SIGHUP; restarting. Oct 23 09:24:42 demo-felhom systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server 
+pveproxy: active
+main sshd banner from DooPlex (:22): SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4
+key login: ok
+root password attempt rc=255 (non-zero = refused)
+
+## notes
+- felhom-sshd :8822 admits only the operator peer 10.77.0.250 over wg-felhom (H1); a connection test from DooPlex times out BY DESIGN — proven: active, listening, sshd -t ok, firewall set.
+- a dummy root password is logged as 'Failed password' (OpenSSH refuses a root password silently under prohibit-password); the policy proof is sshd -T + the SIGHUP reload after the drop-in. The real root password was not tried (CC does not hold it).
+- guests 9202, 9401 (demo-hp) and the transient 990000 (Tester 1, gone a minute later) are outside the felhom pool; swap = RAM applies to pool guests (9201 on all three: swap = memory).
@@ -0,0 +1,53 @@
+# Part 6 — D33: the released images and agent packages to ep0 nightly (operator yes 2026-10-23), 2026-10-23T07:51:56Z
+
+## ep0 — free space first, then ONE user and ONE directory (nothing else on ep0 changed)
+before: /dev/sda1 38G 23G used 14G free 63% (/mnt/pbs-datastore is the PBS datastore root — not used)
+created: user felhom-relcopy (system, uid 999, password locked 'L', shell /bin/sh for the forced command), /srv/felhom-release-copy (0750) with data/ and .ssh/authorized_keys:
+  command="/usr/bin/rrsync /srv/felhom-release-copy/data",restrict ssh-ed25519 <the job key, felhom-release-copy@dooplex>
+fence test: rsync --list-only OK; a shell command → 'rrsync error: SSH_ORIGINAL_COMMAND does not run rsync'; '..' → 'do not use .. in arg'
+after the first copy: /dev/sda1 38G 24G used 13G free 65%; /srv/felhom-release-copy/data 677M
+cap: CAP_MB=3072, enforced by the sender (ep0's root fs has no quota; a quota would change more than one directory)
+
+## DooPlex — the job (felhom.eu d306b8c2): /usr/local/sbin/felhom-release-copy + .service + .timer (01:40), /etc/felhom-release-copy/{env,id_ed25519,known_hosts,registry-token}
+registry token: a NEW Gitea access token for user admin, name felhom-release-copy-d33, scope read:package ONLY — minted inside the gitea pod straight into /etc/felhom-release-copy/registry-token (0600 root); never printed
+known_hosts: ep0's ed25519 host key, SHA256:NGtcS5v3K+9UjMTM6X5Ib0CoChdhqj9bezLz1hOUqgY — equal to the key already pinned for root@ep0
+skopeo, not docker save: skopeo dir: keeps the registry's manifest bytes and compressed blobs, so a push back gives the SAME digest (docker save re-compresses → a new digest); no Docker daemon needed
+
+## the first run (systemctl start, under the unit's sandbox)
+2026-10-23T09:48:33+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-controller: keeping 0.338.0, 0.339.0, 0.340.0
+2026-10-23T09:48:37+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-hub: keeping 0.164.0, 0.165.0, 0.166.0
+2026-10-23T09:48:43+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-samba: keeping 1.0.0, 1.1.0, 1.2.0
+2026-10-23T09:48:47+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-agent: keeping 0.159.2, 0.160.0, 0.161.0, 0.162.0, 0.163.0
+2026-10-23T09:48:51+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: staged and verified: 676 MB, new this run: felhom-controller:0.338.0, felhom-controller:0.339.0, felhom-controller:0.340.0, felhom-hub:0.164.0, felhom-hub:0.165.0, felhom-hub:0.166.0, felhom-samba:1.0.0, felhom-samba:1.1.0, felhom-samba:1.2.0
+2026-10-23T09:49:51+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: pushed to ep0 and compared by checksum: no difference (91 s)
+2026-10-23T09:49:51+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: success signal written
+2026-10-23T09:49:51+02:00 dooplex systemd[1]: felhom-release-copy.service: Consumed 30.416s CPU time, 743.1M memory peak.
+
+## read back from ep0 (rsync pull with the job key → /var/lib/felhom-release-copy/readback)
+sha256sum -c SHA256SUMS: every file OK (141 files), 677 MB
+every image re-verified (top manifest = its digest, every blob = its name):
+  felhom-controller 0.338.0 sha256:e8a529f395bfe8c0e7b87d5d98929a6f8e9834aa13c0ab77bd214ee9e7e53433
+  felhom-controller 0.339.0 sha256:fee71aafceb4a7d5f24ae4a6a273a8d51eb26e0550fe9d12d4d86955f6a343a8
+  felhom-controller 0.340.0 sha256:a8d1c6458da4ea9d667f928498656a291771e0b3756b20f2fcc8c1cf9d516621
+  felhom-hub 0.164.0 sha256:931594e333a4e8a9784eff2fb144d022ab46f89dd6a865a93e6100dc28dbd9dc
+  felhom-hub 0.165.0 sha256:c6180fb79a2ff7d5546a704c68b634b9207afb0a0ecd524c74b0310434532537
+  felhom-hub 0.166.0 sha256:829667bb289c790bc99719e88dad8f7d08faaa9f662df1e9a06b1774915a1af6
+  felhom-samba 1.0.0 sha256:0fa9fae3eaa9d48836c855a4884f0751a90ff050dc9e7d17e41d93bb434499c0
+  felhom-samba 1.1.0 sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10
+  felhom-samba 1.2.0 sha256:da96f25033827fbb77d0ee398e86fe5b744e2f29be9de898994533340c97acf9
+  agent 0.159.2 binary 4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9 bundle 477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9
+  agent 0.160.0 binary 7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1 bundle 6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f
+  agent 0.161.0 binary ff4e6dc52dd70234913445ecdeb88d31dea8e394d77c48704d26ff19b999d3b5 bundle 485f36c957d83f772aa22cac24e5e73f44da5cf8b957b1fd513bec2bc1d12c47
+  agent 0.162.0 binary 5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928 bundle ac772967fd4669045e837a70bf5acca239c58d55f7f00212e472a8facd60b5a5
+  agent 0.163.0 binary 561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c bundle 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8
+independent controls (other channels): controller 0.340.0 = the release digest from build.sh (03-release-0340-boxes.txt); hub 0.165.0 = the running hub pod's imageID; hub 0.164.0 = hub/CHANGELOG; agent 0.163.0 = release-agent.sh today; agent 0.161.0 = night2-2026-10-20/P1/1b; bundle 0.160.0 = the three boxes' installed bundle_sha256
+
+## load on a scratch machine (9202 on demo-hp, Tier 0)
+the ep0 copy of felhom-controller 0.340.0 → skopeo docker-archive (amd64) → pct push 9202 → docker load:
+  Loaded image: felhom-restorecopy-test:0.340.0; ID=sha256:3f34d74519463c338422594b70c31d9c5beabed48847134c6d26b5a95ebbae2b = the copy's amd64 config digest; a container ran (ls /)
+  teardown: image untagged (0 left), the tar removed on 9202, on demo-hp /tmp and on DooPlex
+
+timer: Sat 2026-10-24 01:42:16 CEST  15h -         - felhom-release-copy.timer felhom-release-copy.service
+
+## alarm (homelab-manifests 1bbef892): FelhomReleaseCopyStale (> 26 h or absent, warning) — promtool check: 7 rules OK; promtool test: quiet at 25 h, fires at 27 h
+synced ConfigMap mon-system/prometheus-rules only; POST /-/reload 200; /api/v1/rules: FelhomReleaseCopyStale inactive ok; metric age at read ~3235 s
@@ -0,0 +1,10 @@
+no guest references the three
+BEFORE: /dev/mapper/pve-root   39G   29G  8.5G  77% /
+Removed volume 'local:iso/felhom-installer-1.27.1-pve9.2-1.iso'
+removed local:iso/felhom-installer-1.27.1-pve9.2-1.iso
+Removed volume 'local:iso/felhom-installer-1.28.0-pve9.2-1.iso'
+removed local:iso/felhom-installer-1.28.0-pve9.2-1.iso
+Removed volume 'local:iso/felhom-installer-1.29.0-pve9.2-1.iso'
+removed local:iso/felhom-installer-1.29.0-pve9.2-1.iso
+AFTER: /dev/mapper/pve-root   39G   24G   14G  64% /
+local:iso/felhom-installer-1.30.0-pve9.2-1.iso iso     iso       1705324544
@@ -0,0 +1,16 @@
+# The vouch read-back — agent 0.163.0 (the operator vouched on the hub, answered 'Vouched now'), read 2026-10-23T08:00:42Z
+
+## channel 1: hub DB (hub.db + -wal + -shm in ONE tar from the pod), hub_settings
+artifact_agent_sha256|561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c|2026-10-23 07:58:46
+artifact_agent_version|0.163.0|2026-10-23 07:58:46
+artifact_bundle_sha256|3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8|2026-10-23 07:58:46
+artifact_golden_sha256|3881de62b0986ccbe5e55580e3f1bb9765d65b8d898f67c6339bac105244f84f|2026-10-23 07:58:46
+artifact_golden_version|0.331.0|2026-10-23 07:58:46
+artifact_min_agent|0.158.1|2026-10-23 07:58:46
+artifact_wrapper_sha256||2026-10-23 07:58:46
+
+## channel 2: the running hub's operator page GET /configuration (CC's token), the selected options
+value="0.163.0" data-sha="561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c" selected>0.163.0
+value="0.331.0" data-sha="3881de62b0986ccbe5e55580e3f1bb9765d65b8d898f67c6339bac105244f84f" selected>0.331.0
+
+agent 0.163.0 sha256 = the release (561b5316…); bundle sha256 = the release (3afb88d9…). Golden 0.331.0 unchanged. The floor's MinAgent stays 0.158.1 (artifact_min_agent 0.158.1; controller 0.340.0 declares 0.158.1).
@@ -0,0 +1,13 @@
+# Installer 1.36.0 published (operator yes 2026-10-23), read 2026-10-23T08:16:46Z
+
+1. bump: felhom.eu 83388c08 (SCRIPT_VERSION 1.36.0; scripts/test_hostinstall.py 40 passed; CI 2247 success)
+2. tag: installer-v1.36.0 (annotated d00f868c) -> 83388c08, pushed
+3. pins: manifests/webpage.yaml both --ref=installer-v1.36.0 (sidecar git-sync-scripts + init git-sync-scripts-init), commit 2aebf1bd;
+   ArgoCD: only Deployment felhom-system/felhom-webpage was OutOfSync; synced that ONE resource at revision 2aebf1bd; rollout ok;
+   live args: git-sync-scripts-init ['--ref=installer-v1.36.0'], git-sync-scripts ['--ref=installer-v1.36.0'], website git-sync ['--ref=main'] (unchanged)
+4. read back: curl https://felhom.eu/scripts/felhom-host-install.sh
+   SCRIPT_VERSION="1.36.0"   # the SINGLE version source (F-1): -h and the run banners follow it.
+   sha256 served   = de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec
+   sha256 tag file = de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec
+   the drop-in name 10-felhom-root-key-only appears 2 times (the uninstall's removal)
+Rollback: move the tag back to installer-v1.35.0's commit and wait ~30 s (no sync).
@@ -0,0 +1,4 @@
+# Website publish read-back (felhom.eu 0bb57a4f, operator yes 2026-10-23)
+Method: headless Chromium (playwright 1.61.1 image) on the LIVE site; the FAQ answer opened by a click; 390x844 (phone) and 1366x900 (computer).
+No page scrolls sideways at either size (scrollWidth == viewport on all 10 shots).
+Text: see text-check.txt (new sentences present; the old 'only encrypted (the remote backup)' gone; control: HEAD~1 had it twice).
@@ -0,0 +1,7 @@
+waited ~0 s
+NEW letoltes 'az alapokat futtatja' count 1 (want 2 on the FAQ: visible + JSON-LD; 1 on download)
+NEW en/download 'runs the basics; the photo app needs 16 GB' count 1 (want 2 on the FAQ: visible + JSON-LD; 1 on download)
+NEW gyik 'a Cloudflare h' count 4 (want 2 on the FAQ: visible + JSON-LD; 1 on download)
+NEW en/faq "passes through Cloudflare's network and is readable there" count 2 (want 2 on the FAQ: visible + JSON-LD; 1 on download)
+OLD gyik 'otthonodat (a t' live 0 (want 0); control HEAD~1 2 (want 2)
+OLD en/faq 'leaves your home only encrypted (the remote backup)' live 0 (want 0); control HEAD~1 2 (want 2)
@@ -0,0 +1,43 @@
+# Part 5 — D34: the hub DB off-site every 6 hours (operator yes 2026-10-23), 2026-10-23T08:41:15Z
+
+## what changed
+- hub 0.167.0 (deployed 08:27Z, image sha256:177860a9…): snapshots at 02:00 08:00 14:00 20:00 Budapest; start-up catch-up when the newest > 6 h. Hub log at start:
+    2026/10/23 10:28:12 [INFO] db-snapshot 08:00: next run at 2026-10-24 08:00 CEST (in 21h31m48s)
+    2026/10/23 10:28:12 [INFO] db-snapshot 14:00: next run at 2026-10-23 14:00 CEST (in 3h31m48s)
+    2026/10/23 10:28:12 [INFO] db-snapshot 20:00: next run at 2026-10-23 20:00 CEST (in 9h31m48s)
+    2026/10/23 10:28:12 [INFO] db-snapshot 02:00: next run at 2026-10-24 02:00 CEST (in 15h31m48s)
+    2026/10/23 10:30:06 [INFO] db snapshot written: hub-20261023T082812Z.db (525996032 bytes, 1m54.537s); pruned 1, keeping 2
+- DooPlex felhom-hub-db-backup (felhom.eu 52af38df): timer 02:30 08:30 14:30 20:30; waits ≤ 20 min for a snapshot being written; refuses a newest snapshot older than 7 h
+- R-1234 (closed): the first version (2 h, no wait, be7e97dc) refused the timer's own catch-up at 10:28 CEST and mailed admin@ once — fixed in 52af38df
+
+## retention (ep0, server-side, unchanged): prune-job prune-operator-hubdb, ns operator, 03:45, keep-daily 14, keep-weekly 8
+=> the current day's copies (up to 4) stay until the next 03:45 prune, then the newest of each day for 14 days and one per week for 8 weeks. Size per copy: 501.6 MiB raw, 24.5 MiB uploaded (compressed, dedup'd chunks).
+
+## run once by hand (systemctl start felhom-hub-db-backup.service, 10:39 CEST)
+2026-10-23T10:39:33+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: snapshot hub-20261023T082812Z.db, 11 min old
+2026-10-23T10:39:45+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: checked: 525996032 bytes, integrity ok, 4 host(s)
+2026-10-23T10:39:48+02:00 dooplex felhom-hub-db-backup[2377041]: hubdb.pxar: had to backup 501.629 MiB of 501.629 MiB (compressed 24.471 MiB) in 2.47 s (average 203.437 MiB/s)
+2026-10-23T10:39:49+02:00 dooplex felhom-hub-db-backup[2377041]: Duration: 3.21s
+2026-10-23T10:39:49+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: pushed hub-20261023T082812Z.db to ep0 (ns operator) in 4 s
+2026-10-23T10:39:49+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: success signal written
+
+## read back: restore the NEWEST copy from ep0 with the READ-ONLY token into a scratch dir (shredded after)
+15 copies on ep0, oldest 2026-10-10T00:30:56Z
+newest copy on ep0: host/dooplex-hub/2026-10-23T08:39:45Z
+Using encryption key from '/etc/felhom-hub-backup/enc.key'..    
+Fingerprint: b2:19:bf:36:3b:97:3d:6c    
+restore complete (501.629 MiB processed in 5.4s, average 92.519 MiB/s)    
+restored size: 525996032 bytes
+restored sha256: a9ff4c8a0d0fe34da72411d9c165e573069f5601492d1e625578464f56adbde7
+integrity_check: ok
+SELECT COUNT(*) FROM hosts: 4
+SELECT value FROM hub_settings WHERE key='artifact_agent_version': 0.163.0
+pod snapshot hub-20261023T082812Z.db: 525996032 bytes, sha256 a9ff4c8a0d0fe34da72411d9c165e573069f5601492d1e625578464f56adbde7 (kubectl exec sha256sum — a different channel from the restore)
+=> byte-identical; it holds today's vouch (artifact_agent_version 0.163.0, set 07:58Z) — a nightly copy could not
+
+## timers
+Fri 2026-10-23 14:30:08 CEST 3h 48min Fri 2026-10-23 10:28:30 CEST 12min ago felhom-hub-db-backup.timer felhom-hub-db-backup.service
+
+## alarm (homelab-manifests 1bbef892): HubDBBackupStale 26 h -> 13 h (two missed 6-hourly pushes)
+promtool test: quiet at 12 h, fires at 14 h; RED-PROOF: the rule at 26 h fails the 14 h case
+in force: /api/v1/rules shows the 13 * 3600 expression, inactive ok; metric age at read ~237 s
@@ -0,0 +1,21 @@
+# Part 3 — hub 0.167.0 deploy and the 30-minute watch (operator yes 2026-10-23), read 2026-10-23T08:56:31Z
+
+build: clean clone at d306b8c2 (hub/ unchanged since be7e97dc, CI 2250 success; d306b8c2 CI 2251 success before the deploy); hub tests rc 0 (28 packages)
+image: gitea.dooplex.hu/admin/felhom-hub:0.167.0 sha256:177860a996cb285d935777d6ffdf85725f6cf4489885f54fd3bf8135c68041a0
+manifest 34979d5d; ArgoCD: only Deployment felhom-system/hub OutOfSync; synced that one resource; rollout ok 08:27:34Z; pod imageID = the digest above; Synced/Healthy
+
+## watch 08:27:34Z → 08:56:31Z — channel 1: the hub DB (with WAL) + log
+events since the deploy: 0 (newest event 07:07:10Z Tester4 host_down — the expected dead test box; 161 events since 2026-10-22: the table is live)
+host report|demo-felhom-8363b5|2026-10-23 08:54:54|2
+host report|demo-hp-bb76ea|2026-10-23 08:49:41|2
+host report|tester-1-d70be4|2026-10-23 08:42:35|1
+controller report|demo-felhom|2026-10-23 08:53:20|3
+controller report|demo-hp|2026-10-23 08:53:24|3
+controller report|tester-1|2026-10-23 08:53:23|3
+hub log since deploy: 0 [ERROR]; 'osupdates: alarm check done … 0 alarm(s) sent'; 'Host staleness checker initialized: 3 ok, 1 already alarmed (seeded silently, R-1001)' — Tester4 not re-mailed
+
+## channel 2: the operator mailbox (Gmail connector, to:admin@felhom.eu newer_than:1d)
+after 08:27Z: ONE mail, 08:28:32Z '[DooPlex backup] FAILED: felhom-hub-db-backup.service' — NOT from the hub: CC's install of the 6-hourly timer (R-1234, closed, fixed 52af38df). No hub mail after the deploy.
+before the deploy (control that the channel delivers): 07:07:11Z Tester4 host_down, 07:04:11Z node_down, 06:22Z/06:19Z stale — the hub's mails do arrive
+
+verdict: no new false alarm from the hub in the 30 minutes after the deploy; one CC-caused DooPlex mail, filed and fixed
@@ -30,6 +30,16 @@

 | Row | What | Closed | Evidence |
 |---|---|---|---|
+| **R-49** | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** (P3) | CLOSED 2026-10-23 — released in controller **0.340.0** (D21 = B: a template may exclude a named volume; Immich's model cache is out of every backup); the global floor 0.340.0 (MinAgent 0.158.1) since 06:41 UTC, all three test boxes on it | `audits/day-2026-10-23/` 03, 06 |
+| **R-1204** | **A box that runs once with its clock ahead loses its WireGuard tunnel to ep0 — the PBS whole-box copies and the operator's door — until the real date passes the wrong one, and nothing says why.** (P2) | CLOSED 2026-10-23 — the sign reported and the heal (D30, the agent re-keys on the existing peer path) in agent **0.163.0** (vouched by the operator 07:58 UTC; on Tester 1, demo-hp, demo-felhom by signed jobs), the alarm and its clearing in hub **0.167.0** (deployed 08:27 UTC). The live heal was proven on the test night (`audits/night-2026-10-23/P4-testnight/`) | `audits/day-2026-10-23/` 07, and the hub watch in 14 |
+| **R-1205** | **After a box death, a SECOND install that is deleted before it makes a key leaves the whole-box (PBS) tier waiting for the operator's re-issue.** (P2) | CLOSED 2026-10-23 — D31 in hub **0.167.0** (deployed 08:27 UTC, image sha256:177860a9…): the re-issue reads the newest deletion of a host that held a key | `audits/day-2026-10-23/` 14 |
+| **R-1230** | **A reinstalled box that waits for the household's recovery code was reported „stuck” after an hour: a false operator mail, a needless re-issue of the off-site login, and a controller restart on the box.** (P3) | CLOSED 2026-10-23 — hub **0.167.0** (deployed 08:27 UTC): the delivery checker stands down for `awaiting_recovery_key` | `audits/day-2026-10-23/` 14 |
+| **R-982** | **The Proxmox host accepts root password login over SSH from the LAN (measured on demo-hp), so anyone on the household network can try the root password.** (P3) | CLOSED 2026-10-23 — D4 delivered: the bundle drop-in in agent **0.163.0** (config bundle sha256 3afb88d9…, via a step bundle 0.163.0-step1 because it adds a path, R-880) on all three test boxes — `sshd -T` `permitrootlogin without-password`, sshd reloaded after the drop-in, a key login works, felhom-sshd active; new installs: installer **1.36.0** published (tag `installer-v1.36.0`, served sha256 de8bf5f3… = the tag). The real root password was not tried over SSH (CC does not hold it); the console is untouched by design | `audits/day-2026-10-23/` 07, 11 |
+| **R-985** | **Our zero-knowledge wording is stronger than the product: on a live box the operator can reach the off-site backup key (through the vaulted root credential and the operator tunnel) without the customer's recovery code.** (P3) | CLOSED 2026-10-23 — the D3 wording published on the website (felhom.eu `0bb57a4f`), read back live at phone and computer size | `audits/day-2026-10-23/` 12-website-readback |
+| **R-991** | **Nothing a buyer can read says which machines Felhom installs on, and a quarter of the forum beginners own only machines it cannot install on.** (P3) | CLOSED 2026-10-23 — the D5 minimum machine published (felhom.eu `0bb57a4f`), read back live | `audits/day-2026-10-23/` 12-website-readback |
+| **R-1225** | **The website's minimum machine (8 GB) cannot run the photo app, and the download page does not say so.** (P3) | CLOSED 2026-10-23 — D36: „8 GB runs the basics; the photo app needs 16 GB” on both download pages, published `0bb57a4f`, read back live | `audits/day-2026-10-23/` 12-website-readback |
+| **R-1226** | **Two FAQ answers say the household's data leaves the home only encrypted (the remote backup) — not true for data reached from outside.** (P2) | CLOSED 2026-10-23 — D36: both FAQ answers (visible + JSON-LD, HU + EN) now say what is opened from outside passes Cloudflare's network and is readable there; the backups leave only encrypted. Published `0bb57a4f`; live: the new sentence 2× per FAQ page, the old one 0× (control: the previous commit had it 2×) | `audits/day-2026-10-23/` 12-website-readback |
+| **R-1227** | **The container registry and the agent packages exist only on DooPlex — after its loss no box updates, reinstalls or installs.** (P2) | CLOSED 2026-10-23 — D33 built (felhom.eu `d306b8c2`): nightly 01:40 the newest 3 release tags of controller, hub, samba (skopeo `dir:`, digest-true) and the newest 5 agent versions to ep0 `/srv/felhom-release-copy` (one new user `felhom-relcopy`, rrsync-confined; cap 3 GB by the sender); first run 676 MB in 91 s; pulled back, every sha256 OK, every image re-verified; controller 0.340.0 from the copy loaded on 9202 (ID = config digest); alarm `FelhomReleaseCopyStale`; restore steps in `total-loss-of-dooplex.md`. Follow-ups: R-1235 | `audits/day-2026-10-23/` 08-release-copy-D33.txt |
 | **R-1234** | **D34's first push refused the hub's own catch-up and mailed a false „hub-db-backup failed" to admin@.** Installing the 6-hourly timer at 10:28 made systemd catch up the missed 08:30 slot at once (`Persistent=true`), while the freshly deployed hub 0.167.0 was still writing its start-up snapshot (`hub-*.db.tmp`); the only finished one was 02:00 (8 h), above the new 2-hour limit → „the hub stopped snapshotting", `OnFailure` mail id `01a167e0…`. The 2 h limit was also stricter than the hub's own 6-hour catch-up, so any restart could repeat it. (P3) | CLOSED 2026-10-23 day, the same session — the push waits (≤ 20 min) for a snapshot being written, and the limit is one slot gap + 1 h (7 h); tests `test_d34_waits_for_a_snapshot_being_written` (reproduces the 10:28 run when the wait is removed), `…_a_restart_within_one_slot_is_not_an_alarm`, `…_two_missed_slots_are_refused`, each red-proved | `audits/day-2026-10-23/13-hubdb-6h-D34.txt` |

 ## 2026-10-22 night — the seven decisions put into effect (full text of the rows: `git show 1100a344:documentation/backlog/OPEN-ITEMS.md`)
@@ -5,6 +5,20 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S

 **Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**

+## Moved from STATUS on 2026-10-23 (day)
+
+**Updated 2026-10-23 (morning).** Your three picks are built (the tunnel repairs itself with a new key, the hub skips a keyless deleted install, the controller repository stays public), and the night copy can no longer outlive the controller. Four fresh reviews found 34 points, every real one fixed or a row; the test night PASSED (no failed check at night, every night job passed, nothing lost). Nothing was released or deployed; the morning list has the order, and the DooPlex survey asks four questions (D32–D35).
+
+**Decided 2026-10-23 morning — the operator's „go" for the day brief** (each ruling in its row and in CONTEXT.md): D32 = yes, print the sheet; D33 = yes, the nightly release copy to ep0 (built, R-1227 closed); D34 = yes, the hub DB every 6 hours (built); D35 = yes, after the release week (R-1228, dated 2026-10-30); D36 = yes, both sentences (published). The rows as they stood:
+
+| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
+|---|---|---|---|---|---|---|
+| D36 | R-1225, R-1226 | Two website sentences before the publish: „8 GB runs the basics; the photo app needs 16 GB” and „what you open from outside the home passes Cloudflare's network, readable there; the backups leave only encrypted”? | **Yes, both** (the patch's own aim is the whole truth) | — | the patch publishes with both promises a little too strong | — |
+| D35 | R-1228 | After the release week, serve the website and the installer from Cloudflare, like the ISO? | **Yes, later** (no new installs stop when DooPlex is down) | keep them home: after a DooPlex loss, no install until it is back | the same | — |
+| D34 | R-232 | Copy the hub database to ep0 every 6 hours instead of every night? | **Yes** (0 €, about 30 minutes, one DooPlex change) | keep nightly: up to a day of hub changes lost, e.g. a new box's recovery package | the same | — |
+| D33 | R-1227 | Copy the released images and agent packages to ep0 every night? | **Yes** (0 €, about 2 hours of CC, a few GB on ep0) | a timed rebuild drill instead: about 3 hours once, and rebuilt images are not the same | after a DooPlex loss no box updates, reinstalls or installs until everything is rebuilt | — |
+| D32 | R-923, R-924 | Print the break-glass sheet now (eight keys and logins + ep0's fingerprint) and keep it outside the home? | **Yes** (30 minutes; nothing else opens the off-site copies after a loss) | an encrypted USB stick outside the home: ~10 €, 1 hour, one more passphrase | after a DooPlex loss no off-site copy opens | — |
+
 ## Moved from STATUS on 2026-10-23

 **Updated 2026-10-22 (night).** Your seven picks of 15:47 are in effect or ready: D6 and D8 are live, D21 waits for the controller release, and D3, D5 (the website) and D4 (root SSH) are ready patches. Four controller fixes are built and proven on the scratch box, among them a gentler night copy (the 03:30 memory pressure) and no import over a stopped app. Nothing was released or deployed; the morning list below has the order.
@@ -93,6 +93,17 @@ itself. **Corrected:** one snapshot took 44 s on the live volume (0.63 s on a lo

 ### Step 4 — the push (on DooPlex, root; `felhom-hub-db-backup.service` + `.timer` 02:30) — DONE 2026-10-05

+> **Since 2026-10-23 (D34, operator yes): every 6 hours.** The hub (≥ 0.167.0) writes a snapshot at 02:00, 08:00, 14:00
+> and 20:00 Budapest (`dbsnap.Slots`; start-up catch-up when the newest is older than 6 h); the timer pushes at 02:30,
+> 08:30, 14:30 and 20:30. The push waits (≤ 20 min) for a snapshot being written and refuses a newest snapshot older
+> than **7 h** (one slot gap + 1 h — a 2 h limit refused the hub's own catch-up once and mailed a false failure,
+> R-1234). Alarm `HubDBBackupStale` at **13 h** (two missed pushes). **Retention is unchanged and server-side on ep0**
+> (`prune-operator-hubdb`, 03:45, keep-daily 14, keep-weekly 8): the current day's copies (up to 4) live until the next
+> prune, then the newest of each day for 14 days and one per week for 8 weeks. One copy: ~500 MiB raw, ~25 MiB
+> uploaded. Proven 2026-10-23: a hand run, then the newest copy restored from ep0 with the read-only token was
+> byte-identical to the pod's snapshot, integrity ok, `SELECT COUNT(*) FROM hosts` = 4
+> (`audits/day-2026-10-23/13-hubdb-6h-D34.txt`). The text below is the original nightly design.
+
 **The real script is `scripts/hub-db-backup/felhom-hub-db-backup`** (versioned, R-231; installed by `install.sh`;
 15 tests in `test_hub_db_backup.py`, run by hand — not in CI). It adds to the sketch below: it refuses a snapshot older
 than 26 h (the hub stopped snapshotting), a copy whose size differs from the pod's file, and a copy with no hosts; it
@@ -1,4 +1,4 @@
-# Break-glass sheet — the keys that must exist outside DooPlex (R-923)
+# Break-glass sheet — the keys that must exist outside DooPlex (R-923, D32)

 > **A template. It holds NO key value, and none may ever be written into this file, a commit, a log or a chat.**
 > Print this page, then write or stick each value onto the paper copy only. Keep the paper away from home (DooPlex is
@@ -33,31 +33,43 @@ captured line on some systems; strip it when typing the value back.
 **Check a print later without exposing it:** for S1/S2 the only reliable check is one restore with a key file rebuilt
 from the printed `data` field (hub-DB runbook Step 0 note) — `key show` cannot check a rebuilt key.

+## Public fingerprints — to check you hold the right key (not secret; not on the printed page)
+
+| # | Fingerprint |
+|---|---|
+| S1 | PBS key `93:03:bf:d7:1f:4c:9e:fe…` (`proxmox-backup-client key show`) |
+| S2 | PBS key `b2:19:bf:36:3b:97:3d:6c…` |
+| S6a | `SHA256:/ixgTesZqykAGJpFUUd4kLAiHFgKOkYFLNC3AQXWP+k` |
+| S6b | `SHA256:7YqN4rXO08yixTeOO+UtQ8jHyIGycICuctQgRYVGnWw` |
+| S19 | recipient `age1aw83ej37x9sck89xwt856ry0553kf5sng3w544an4et54yjdx5qq79utrv` |
+
+Every place below was checked to exist on 2026-10-23 (existence only, no value read). The release copy on ep0 (D33)
+and the 6-hourly hub-DB copy (D34) add NO key: the release copy is not encrypted (it holds no secret), and the hub-DB
+copy uses S2 as before.
+
 ---

-## The sheet (print from here)
+## The sheet (print from here — one A4 page)

-**FELHOM — break-glass keys.** Printed on: ____________ Stored at: ______________________________
+**FELHOM — vészhelyzeti kulcsok / break-glass keys.** Nyomtatva / printed: ____________ Hol tárolom / kept at: ______________________

-| # | Key | Public fingerprint (to match the right key) | Value — write or stick here |
+Csak nevek és helyek; az értéket kézzel írd be. *Names and places only; write each value by hand.* Tartsd az otthonon kívül. *Keep it outside the home.*
+
+| # | Név / Name | Hol van most / Where it lives now | Érték / Value |
 |---|---|---|---|
-| S1 | DooPlex off-site key — opens Gitea, the password manager, the k8s Secrets export (ep0 `operator`, `host/dooplex-gitea`) | PBS key `93:03:bf:d7:1f:4c:9e:fe…` | `data`: ______________________________________________ |
-| S2 | Hub-DB off-site key — opens the hub database (ep0 `operator`, `host/dooplex-hub`) | PBS key `b2:19:bf:36:3b:97:3d:6c…` | `data`: ______________________________________________ |
-| S4 | Hub seal key `OFFSITE_SECRET_KEY` (64 hex) | — | ______________________________________________________ |
-| S5 | DooPlex restic / Secrets-export passphrase | — | ______________________________________________________ |
-| S6a | Signing RECOVERY key `felhom-rec-recovery` (also in the S1 copy since 2026-10-09; the paper is the copy that needs nothing else) | `SHA256:/ixgTesZqykAGJpFUUd4kLAiHFgKOkYFLNC3AQXWP+k` | (staple the printout) |
-| S6b | Signing OPERATIONAL key `felhom-op-operational` | `SHA256:7YqN4rXO08yixTeOO+UtQ8jHyIGycICuctQgRYVGnWw` | (staple the printout) |
-| S7 | ep0 read-only token `dooplex-hub@pbs!restore` | — | ______________________________________________________ |
-| S8 | Hetzner account: login e-mail · password · two-factor recovery codes | — | ______________________________________________________ |
-| S11 | Cloudflare account: login · password · two-factor recovery codes | — | ______________________________________________________ |
-| S12 | Gmail `felhom.eu@gmail.com`: password · two-factor recovery codes | — | ______________________________________________________ |
-| S19 | CC secrets age key — opens `admin/cc-secrets` (every secret Claude Code uses on DooPlex); the store is in Gitea, behind S1 | recipient `age1aw83ej37x9sck89xwt856ry0553kf5sng3w544an4et54yjdx5qq79utrv` | `AGE-SECRET-KEY-1`______________________________________ |
-| S3 | Vaultwarden master password — **in your head**; write it here only if you decide to | — | ______________________________________________________ |
+| S1 | DooPlex off-site kulcs / DooPlex off-site key (Gitea, jelszókezelő / password manager, k8s Secrets) | DooPlex `/etc/felhom-dooplex-offsite/enc.key` | `data`: ____________________________ |
+| S2 | Hub-adatbázis off-site kulcs / Hub-DB off-site key | DooPlex `/etc/felhom-hub-backup/enc.key` | `data`: ____________________________ |
+| S4 | Hub pecsétkulcs / hub seal key `OFFSITE_SECRET_KEY` | k8s Secret `felhom-system/offsite-secret-key` | __________________________________ |
+| S5 | DooPlex restic / Secrets-export jelmondat / passphrase | DooPlex `/etc/backup/restic-password` | __________________________________ |
+| S6a | Aláíró HELYREÁLLÍTÓ kulcs / signing RECOVERY key | DooPlex `/mnt/5_hdd/felhom.eu/felhom-rec-recovery` | (tűzd ide / staple) |
+| S6b | Aláíró ÜZEMI kulcs / signing OPERATIONAL key | DooPlex `/mnt/5_hdd/felhom.eu/felhom-op-operational` | (tűzd ide / staple) |
+| S7 | ep0 csak-olvasó token / read-only token `dooplex-hub@pbs!restore` | DooPlex `/etc/felhom-hub-backup/token-restore` | __________________________________ |
+| S8 | Hetzner fiók / account (e-mail · jelszó / password · 2FA kódok / codes) | Vaultwarden | __________________________________ |
+| S11 | Cloudflare fiók / account (login · jelszó / password · 2FA kódok / codes) | Vaultwarden | __________________________________ |
+| S12 | Gmail `felhom.eu@gmail.com` (jelszó / password · 2FA kódok / codes) | Vaultwarden | __________________________________ |
+| S19 | CC secrets age kulcs / key (`admin/cc-secrets`) | DooPlex `~/.config/sops/age/keys.txt`; Vaultwarden | `AGE-SECRET-KEY-1`__________________ |
+| S3 | Vaultwarden mesterjelszó / master password | a fejedben / in your head | (csak ha akarod / only if you choose) |

-**Not secret, needed with the keys:**
- ep0: `167.233.158.164` (Hetzner, `felhom-hetzner`); PBS datastore `felhom-offsite`, namespace `operator`;
-  its certificate fingerprint `c6:07:28:3f:5b:7b:5a:41:90:28:d7:ca:4f:37:14:70:56:39:2e:2f:0b:71:e8:06:ca:60:4a:d5:56:5f:3c:fd`.
- Restore order: `total-loss-of-dooplex.md` (in the restored Gitea, repo `felhom.eu`, `documentation/runbooks/`).
-  **Print that page too** — the runbook itself is inside the copy it explains how to open.
-
-**Re-print when** a key above is rotated, and once a year.
+**Nem titok, de kell / not secret, needed:** ep0 = `167.233.158.164` (Hetzner, `felhom-hetzner`), PBS `felhom-offsite`, névtér / namespace `operator`; tanúsítvány / certificate
+`c6:07:28:3f:5b:7b:5a:41:90:28:d7:ca:4f:37:14:70:56:39:2e:2f:0b:71:e8:06:ca:60:4a:d5:56:5f:3c:fd`. Kiadások másolata / release copy: ep0 `/srv/felhom-release-copy/data`.
+Sorrend / order: `total-loss-of-dooplex.md` — **nyomtasd ki azt is / print it too.** Újranyomtatás / re-print: kulcscserekor és évente / when a key rotates, and yearly.
@@ -19,8 +19,7 @@ build tree. **Assume all of it is gone, and the house with it.**
 | The boxes themselves | they keep running and backing up; they cannot report, update or mail until the hub is back | — |
 | The operator's workstation (if it has them) | the WireGuard peer to ep0 (`10.77.0.250`), maybe one of ep0's two root SSH keys, maybe a Bitwarden client's offline vault cache | the operator |

-**Lost with DooPlex and NOT in any copy** (measured 2026-10-09): the container registry (rebuild the images from
-code); *(the operator signing keys were here until 2026-10-09 afternoon; they now ride the nightly copy — R-924)*; DooPlex's own SSH key (one of ep0's two root keys);
+**Lost with DooPlex and NOT in any copy** (measured 2026-10-09): *(the container registry was here until 2026-10-23: the released images and agent packages now ride a nightly copy to ep0 — D33, „Restore the release copy" below; the rest of the registry — older tags, rc builds, non-Felhom images — is still lost)*; *(the operator signing keys were here until 2026-10-09 afternoon; they now ride the nightly copy — R-924)*; DooPlex's own SSH key (one of ep0's two root keys);
 the build tree and drills (`/mnt/5_hdd/felhom.eu/{build,drills}`); `.claude-memory` (same-disk restic only); the
 local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom; Longhorn on DooPlex only).

@@ -66,16 +65,48 @@ local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom
   copy, use that. Every later login comes from here.
 6. **The Secrets export:** `gpg --batch --passphrase-file <S5 file> -d secrets-<stamp>.yaml.gpg > secrets.yaml` —
   every k8s Secret of the old cluster (shred it after use).
-7. **Gitea:** `gitea-restore.md` (on a normal network this time). Then rebuild the images from the code into the new
-   registry: hub, controller, agent (`RUNBOOK-manual-build.md`).
+7. **Gitea:** `gitea-restore.md` (on a normal network this time). Then put the released images and agent packages
+   back into its registry from ep0 — „Restore the release copy" below (same digests; no rebuild). Rebuild from code
+   (`RUNBOOK-manual-build.md`) only what the copy does not hold.
 8. **The hub:** k3s, then `RUNBOOK-hub-db-offsite-backup.md` §3 with S2 and S4 — **start it with mail held**
   (§3 step 1: the `MAIL-HOLD` marker — shipped in hub 0.145.0, 2026-10-10).
 9. **DNS:** in Cloudflare (S11) point `hub.felhom.eu` (today a CNAME to `dooplex.hopto.org`) and `gitea.dooplex.hu` at
   the new place. The boxes reconnect by themselves: their API keys are in the restored hub DB.
 10. **Signing:** the keys are in the restored copy (`signing/`) and on the sheet (S6) — put them back and sign as before. Without it, no box accepts an agent update or a
    bundle until it is re-enrolled on site (`04` §4).
-11. **Backups again:** reinstall `scripts/hub-db-backup/` and `scripts/dooplex-offsite/` from the restored code, the
-    tunnel and the tokens, and check the next night's copies.
+11. **Backups again:** reinstall `scripts/hub-db-backup/`, `scripts/dooplex-offsite/` and `scripts/release-copy/` from
+    the restored code, the tunnel and the tokens (the release copy needs a new read-only registry token and a new SSH
+    key in ep0's `felhom-relcopy` `authorized_keys`), and check the next night's copies.
+
+## Restore the release copy (D33, since 2026-10-23)
+
+**What it is.** Every night at 01:40 DooPlex copies the newest 3 release tags of `felhom-controller`, `felhom-hub` and
+`felhom-samba`, and the newest 5 agent versions (binary + config bundle), to ep0 `/srv/felhom-release-copy/data`
+(`scripts/release-copy/`). Images are in skopeo's `dir:` layout — the registry's own bytes — so a push back gives the
+**same digest** as the release. Nothing in it is secret; it is not encrypted. Alarm: `FelhomReleaseCopyStale`.
+
+1. **Get it off ep0** (ep0 root — step 2): `rsync -a root@167.233.158.164:/srv/felhom-release-copy/data/ ./release-copy/`
+   (or work on ep0 itself). Then `cd release-copy && sha256sum -c --quiet SHA256SUMS` — every line must pass.
+2. **Images back into the new registry**, one per tag (log in to the new Gitea first: `skopeo login <gitea host>`):
+   ```bash
+   for d in images/*/*; do repo=${d#images/}; repo=${repo%%/*}; tag=${d##*/}
+     skopeo copy --all --preserve-digests "dir:$d" "docker://<gitea host>/admin/$repo:$tag"
+     echo "$repo:$tag $(skopeo inspect --raw "docker://<gitea host>/admin/$repo:$tag" | sha256sum | cut -c1-64) want $(cut -c8- "$d/DIGEST")"
+   done
+   ```
+   The two hashes on each line must be equal.
+3. **Agent packages back** (Gitea's generic registry, a token with package write):
+   ```bash
+   for d in agent/*; do v=${d#agent/}; for f in felhom-agent felhom-config-bundle.json; do
+     curl -fsS -u admin:"$(cat token-file)" -X PUT --upload-file "$d/$f" \
+       "https://<gitea host>/api/packages/admin/generic/felhom-agent/$v/$f"; done; done
+   ```
+   Check each download's sha256 against `agent/<v>/<file>.sha256`.
+4. **Keep the registry's name** (`gitea.dooplex.hu`, step 9's DNS) — the boxes pull `gitea.dooplex.hu/admin/…` by name.
+
+**Proven 2026-10-23** (`audits/day-2026-10-23/08-release-copy-D33.txt`): the copy was pulled back from ep0, every
+sha256 matched, every image re-verified against its digest, and controller 0.340.0 from it loaded on a scratch box
+(image ID = its config digest; a container ran). **Not yet walked:** step 2 into a real new registry, and step 3.

 ## Walked on paper (2026-10-09): what still needs DooPlex if the operator has only the sheet, memory and a new machine

@@ -90,7 +121,7 @@ local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom
 | the hub's seal key | S4 (or S1+S5) | sheet, or via S5 |
 | the hub DB | S2 (or Vaultwarden) | sheet, or via S3 |
 | sign a box update | S6 | **sheet — today exists on DooPlex only** |
-| hub image | the registry (lost) | rebuild from restored code — needs a Go/Docker build machine, no secret |
+| hub, controller, samba images; agent packages | the release copy on ep0 (D33, nightly since 2026-10-23) | ep0 root (S8/S9) → „Restore the release copy" — no secret, no rebuild |
 | Gitea `main` → `felhom.eu` website, ISO host | Cloudflare (S11), the build tree (lost) | S11 on the sheet; the build tree is rebuilt from code |
 | Authentik (S17) | not in any copy | not needed for the business; DooPlex homelab apps that use it are out of scope |