Day 2026-10-23 close-out: rulings D31-D36 in CONTEXT; STATUS (releases done, do list, D35 dated, D37/D38); register 191 -> 184 (closed R-49 R-982 R-985 R-991 R-1204 R-1205 R-1225 R-1226 R-1227 R-1230 R-1234; opened R-1235 R-1236 R-1237); the break-glass sheet current (D32, one A4, HU+EN); runbooks for D33/D34; poster facts; evidence day-2026-10-23
gates / gates (push) Successful in 12m53s
gates / gates (push) Successful in 12m53s
This commit is contained in:
@@ -0,0 +1,6 @@
|
||||
== 9202 smoke 06:37:25
|
||||
switched to gitea.dooplex.hu/admin/felhom-controller:0.340.0 at 06:37:28
|
||||
healthy after ~10s: Up 7 seconds (healthy)
|
||||
2026/10/23 06:37:28 [INFO] [tier2] start-up: no leftover second-drive copy container
|
||||
errors, unhealthy+restarting: 0 0
|
||||
login page: 200
|
||||
@@ -0,0 +1,3 @@
|
||||
tester-1: HTTP/1.1 303 See Other Location: /customers/tester-1?flash=floor_set
|
||||
demo-hp: HTTP/1.1 303 See Other Location: /customers/demo-hp?flash=floor_set
|
||||
demo-felhom: HTTP/1.1 303 See Other Location: /customers/demo-felhom?flash=floor_set
|
||||
@@ -0,0 +1,15 @@
|
||||
### root@192.168.0.154
|
||||
containers=10 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.340.0 Up 53 seconds (healthy)
|
||||
errors=0
|
||||
copy-containers=0
|
||||
2026/10/23 06:38:21 [INFO] [tier2] start-up: no leftover second-drive copy container
|
||||
### hp
|
||||
containers=24 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.340.0 Up 54 seconds (healthy)
|
||||
errors=0
|
||||
copy-containers=0
|
||||
2026/10/23 06:38:22 [INFO] [tier2] start-up: no leftover second-drive copy container
|
||||
### felhom-pve
|
||||
containers=9 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.340.0 Up 58 seconds (healthy)
|
||||
errors=0
|
||||
copy-containers=0
|
||||
2026/10/23 06:38:05 [INFO] [tier2] start-up: no leftover second-drive copy container
|
||||
@@ -0,0 +1,33 @@
|
||||
control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
|
||||
== tester-1 (enkicsifelhom.hu): 6 public name(s)
|
||||
OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
|
||||
== demo-hp (enkisfelhom.hu): 14 public name(s)
|
||||
OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
|
||||
== demo-felhom (demo-felhom.eu): 8 public name(s)
|
||||
OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
|
||||
== every public name redirects http -> https
|
||||
@@ -0,0 +1,13 @@
|
||||
== tester-1 (dashboard login with the household password, the press a browser makes)
|
||||
python3: can't open file '/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/38fde0c6-e964-4ec1-b0c7-56e62d675225/scratchpad/n23/fileswalk.py': [Errno 2] No such file or directory
|
||||
== demo-hp (no session): GET /__files/open -> HTTP/2 302 location: /login
|
||||
== demo-felhom (no session): GET /__files/open -> HTTP/2 302 location: /login
|
||||
== tester-1 06:40:35
|
||||
Fájlok press: GET /__files/open -> ['HTTP/2 302'] https://files.enkicsifelhom.hu/_felhom/sso
|
||||
manual second-drive copy: 200 2. mentés elindítva
|
||||
2026/10/23 06:40:43 [INFO] [backup] Tier 2 copied bookstack → /mnt/felhom-drives/adatlemez/backups/secondary/bookstack (157.0 MB, 0 leg(s), 1s)
|
||||
2026/10/23 06:40:46 [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sys_drive/felhom-data/backups/secondary/paperless-ngx (85.9 MB, 1 leg(s), 2s) [SSD: state-only]
|
||||
2026/10/23 06:40:48 [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/adatlemez/backups/secondary/privatebin (16.4 KB, 0 leg(s), 0s)
|
||||
2026/10/23 06:40:48 [INFO] [backup] Tier 2 run complete: 3 app(s) processed (incl. volume-only — F6)
|
||||
copy-containers-left=0
|
||||
[exited with code 0]
|
||||
@@ -0,0 +1,5 @@
|
||||
== global floor 06:41:04: HTTP/1.1 303 See Other Location: /configuration?flash=floor_set
|
||||
clear tester-1: HTTP/1.1 303 See Other Location: /customers/tester-1?flash=floor_set
|
||||
clear demo-hp: HTTP/1.1 303 See Other Location: /customers/demo-hp?flash=floor_set
|
||||
clear demo-felhom: HTTP/1.1 303 See Other Location: /customers/demo-felhom?flash=floor_set
|
||||
read back (with WAL): [('min_controller_version', '0.340.0', '2026-10-23 06:41:04'), ('min_controller_version_declared_min_agent', '0.340.0=0.158.1', '2026-10-23 06:41:04')]
|
||||
@@ -0,0 +1,78 @@
|
||||
# Part 2 — agent 0.163.0 (operator go 2026-10-23), read 2026-10-23T07:28:53Z
|
||||
|
||||
## release (scripts/release-agent.sh at felhom-agent 1109092, CI 2246 green; CHANGELOG b4721ee CI 2249 green)
|
||||
[release-agent] built ok: sha256 561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c
|
||||
[release-agent] config bundle built: sha256 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8
|
||||
[publish-agent] upload OK (HTTP 201)
|
||||
[publish-agent] round-trip GET verified (sha256 matches)
|
||||
[release-agent] config bundle published (HTTP 201)
|
||||
sha256 : 561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c
|
||||
bundle : 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 (felhom-config-bundle.json — vouch it with the agent)
|
||||
* [new tag] v0.163.0 -> v0.163.0
|
||||
|
||||
## step bundle (R-880: the D4 drop-in is a NEW path; the boxes ran bundle 0.160.0 = 6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f)
|
||||
0.163.0-step1 = base 0.160.0 with only /usr/local/sbin/felhom-os-apply replaced; sha256 6cbc066902d5f3294765d5c7dc7113166c1dca25e994dbc36bfc5f139bb61fcf; PUT 201; read back by download: same sha
|
||||
|
||||
## delivery: three signed jobs per box (felhom-opsign, key felhom-op-1): agent_update 06:48Z, agent_config_update 0.163.0-step1 06:59Z, agent_config_update 0.163.0 07:19Z
|
||||
|
||||
## hp (192.168.0.104) 2026-10-23T07:26:29Z
|
||||
agent: felhom-agent 0.163.0
|
||||
bundle: 0.163.0 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 2026-10-23T07:19:44Z
|
||||
1 BUNDLE DONE agent=0.163.0 written=1 same=30 self-check=ok
|
||||
2 BUNDLE DONE agent=0.163.0-step1 written=1 same=29 self-check=ok
|
||||
guest 9201: memory: 25898 swap: 25898
|
||||
guest 9202: memory: 25898 swap: 512
|
||||
guest 9401: memory: 10240 swap: 0
|
||||
drop-in: -rw-r--r-- 1 root root 690 Oct 23 09:19 /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf
|
||||
drop-in content: PermitRootLogin prohibit-password
|
||||
sshd -T: permitrootlogin without-password
|
||||
root password set (passwd -S): P
|
||||
felhom-sshd: active; its root policy: permitrootlogin without-password
|
||||
felhom-sshd listens: 0.0.0.0:8822 [::]:8822 config test: ok; firewall admits only: elements = { 10.77.0.250 }
|
||||
main sshd reloaded after the drop-in: Oct 23 09:19:44 demo-hp sshd[1010]: Received SIGHUP; restarting. Oct 23 09:19:44 demo-hp systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server.
|
||||
pveproxy: active
|
||||
main sshd banner from DooPlex (:22): SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4
|
||||
key login: ok
|
||||
root password attempt rc=255 (non-zero = refused)
|
||||
Oct 23 09:26:37 demo-hp sshd-session[3026730]: Failed password for root from 192.168.0.180 port 34150 ssh2
|
||||
## root@192.168.0.154 (192.168.0.154) 2026-10-23T07:28:00Z
|
||||
agent: felhom-agent 0.163.0
|
||||
bundle: 0.163.0 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 2026-10-23T07:27:38Z
|
||||
2 BUNDLE DONE agent=0.163.0 written=1 same=30 self-check=ok
|
||||
2 BUNDLE DONE agent=0.163.0-step1 written=1 same=29 self-check=ok
|
||||
guest 9201: memory: 4096 swap: 4096
|
||||
guest 990000: memory: 4096 swap: 512
|
||||
drop-in: -rw-r--r-- 1 root root 690 Oct 23 09:27 /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf
|
||||
drop-in content: PermitRootLogin prohibit-password
|
||||
sshd -T: permitrootlogin without-password
|
||||
root password set (passwd -S): P
|
||||
felhom-sshd: active; its root policy: permitrootlogin without-password
|
||||
felhom-sshd listens: 0.0.0.0:8822 [::]:8822 config test: ok; firewall admits only: elements = { 10.77.0.250 }
|
||||
main sshd reloaded after the drop-in: Oct 23 09:27:38 felhom sshd[890]: Received SIGHUP; restarting. Oct 23 09:27:38 felhom systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server.
|
||||
pveproxy: active
|
||||
main sshd banner from DooPlex (:22): SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4
|
||||
key login: ok
|
||||
root password attempt rc=255 (non-zero = refused)
|
||||
Oct 23 09:28:08 felhom sshd-session[4184378]: Failed password for root from 192.168.0.180 port 33442 ssh2
|
||||
## felhom-pve (192.168.0.162) 2026-10-23T07:28:11Z
|
||||
agent: felhom-agent 0.163.0
|
||||
bundle: 0.163.0 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 2026-10-23T07:24:43Z
|
||||
2 BUNDLE DONE agent=0.163.0 written=1 same=30 self-check=ok
|
||||
2 BUNDLE DONE agent=0.163.0-step1 written=1 same=29 self-check=ok
|
||||
guest 9201: memory: 12288 swap: 12288
|
||||
drop-in: -rw-r--r-- 1 root root 690 Oct 23 09:24 /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf
|
||||
drop-in content: PermitRootLogin prohibit-password
|
||||
sshd -T: permitrootlogin without-password
|
||||
root password set (passwd -S): P
|
||||
felhom-sshd: active; its root policy: permitrootlogin without-password
|
||||
felhom-sshd listens: 0.0.0.0:8822 [::]:8822 config test: ok; firewall admits only: elements = { 10.77.0.250 }
|
||||
main sshd reloaded after the drop-in: Oct 23 09:24:42 demo-felhom sshd[1011]: Received SIGHUP; restarting. Oct 23 09:24:42 demo-felhom systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server
|
||||
pveproxy: active
|
||||
main sshd banner from DooPlex (:22): SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4
|
||||
key login: ok
|
||||
root password attempt rc=255 (non-zero = refused)
|
||||
|
||||
## notes
|
||||
- felhom-sshd :8822 admits only the operator peer 10.77.0.250 over wg-felhom (H1); a connection test from DooPlex times out BY DESIGN — proven: active, listening, sshd -t ok, firewall set.
|
||||
- a dummy root password is logged as 'Failed password' (OpenSSH refuses a root password silently under prohibit-password); the policy proof is sshd -T + the SIGHUP reload after the drop-in. The real root password was not tried (CC does not hold it).
|
||||
- guests 9202, 9401 (demo-hp) and the transient 990000 (Tester 1, gone a minute later) are outside the felhom pool; swap = RAM applies to pool guests (9201 on all three: swap = memory).
|
||||
@@ -0,0 +1,53 @@
|
||||
# Part 6 — D33: the released images and agent packages to ep0 nightly (operator yes 2026-10-23), 2026-10-23T07:51:56Z
|
||||
|
||||
## ep0 — free space first, then ONE user and ONE directory (nothing else on ep0 changed)
|
||||
before: /dev/sda1 38G 23G used 14G free 63% (/mnt/pbs-datastore is the PBS datastore root — not used)
|
||||
created: user felhom-relcopy (system, uid 999, password locked 'L', shell /bin/sh for the forced command), /srv/felhom-release-copy (0750) with data/ and .ssh/authorized_keys:
|
||||
command="/usr/bin/rrsync /srv/felhom-release-copy/data",restrict ssh-ed25519 <the job key, felhom-release-copy@dooplex>
|
||||
fence test: rsync --list-only OK; a shell command → 'rrsync error: SSH_ORIGINAL_COMMAND does not run rsync'; '..' → 'do not use .. in arg'
|
||||
after the first copy: /dev/sda1 38G 24G used 13G free 65%; /srv/felhom-release-copy/data 677M
|
||||
cap: CAP_MB=3072, enforced by the sender (ep0's root fs has no quota; a quota would change more than one directory)
|
||||
|
||||
## DooPlex — the job (felhom.eu d306b8c2): /usr/local/sbin/felhom-release-copy + .service + .timer (01:40), /etc/felhom-release-copy/{env,id_ed25519,known_hosts,registry-token}
|
||||
registry token: a NEW Gitea access token for user admin, name felhom-release-copy-d33, scope read:package ONLY — minted inside the gitea pod straight into /etc/felhom-release-copy/registry-token (0600 root); never printed
|
||||
known_hosts: ep0's ed25519 host key, SHA256:NGtcS5v3K+9UjMTM6X5Ib0CoChdhqj9bezLz1hOUqgY — equal to the key already pinned for root@ep0
|
||||
skopeo, not docker save: skopeo dir: keeps the registry's manifest bytes and compressed blobs, so a push back gives the SAME digest (docker save re-compresses → a new digest); no Docker daemon needed
|
||||
|
||||
## the first run (systemctl start, under the unit's sandbox)
|
||||
2026-10-23T09:48:33+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-controller: keeping 0.338.0, 0.339.0, 0.340.0
|
||||
2026-10-23T09:48:37+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-hub: keeping 0.164.0, 0.165.0, 0.166.0
|
||||
2026-10-23T09:48:43+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-samba: keeping 1.0.0, 1.1.0, 1.2.0
|
||||
2026-10-23T09:48:47+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-agent: keeping 0.159.2, 0.160.0, 0.161.0, 0.162.0, 0.163.0
|
||||
2026-10-23T09:48:51+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: staged and verified: 676 MB, new this run: felhom-controller:0.338.0, felhom-controller:0.339.0, felhom-controller:0.340.0, felhom-hub:0.164.0, felhom-hub:0.165.0, felhom-hub:0.166.0, felhom-samba:1.0.0, felhom-samba:1.1.0, felhom-samba:1.2.0
|
||||
2026-10-23T09:49:51+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: pushed to ep0 and compared by checksum: no difference (91 s)
|
||||
2026-10-23T09:49:51+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: success signal written
|
||||
2026-10-23T09:49:51+02:00 dooplex systemd[1]: felhom-release-copy.service: Consumed 30.416s CPU time, 743.1M memory peak.
|
||||
|
||||
## read back from ep0 (rsync pull with the job key → /var/lib/felhom-release-copy/readback)
|
||||
sha256sum -c SHA256SUMS: every file OK (141 files), 677 MB
|
||||
every image re-verified (top manifest = its digest, every blob = its name):
|
||||
felhom-controller 0.338.0 sha256:e8a529f395bfe8c0e7b87d5d98929a6f8e9834aa13c0ab77bd214ee9e7e53433
|
||||
felhom-controller 0.339.0 sha256:fee71aafceb4a7d5f24ae4a6a273a8d51eb26e0550fe9d12d4d86955f6a343a8
|
||||
felhom-controller 0.340.0 sha256:a8d1c6458da4ea9d667f928498656a291771e0b3756b20f2fcc8c1cf9d516621
|
||||
felhom-hub 0.164.0 sha256:931594e333a4e8a9784eff2fb144d022ab46f89dd6a865a93e6100dc28dbd9dc
|
||||
felhom-hub 0.165.0 sha256:c6180fb79a2ff7d5546a704c68b634b9207afb0a0ecd524c74b0310434532537
|
||||
felhom-hub 0.166.0 sha256:829667bb289c790bc99719e88dad8f7d08faaa9f662df1e9a06b1774915a1af6
|
||||
felhom-samba 1.0.0 sha256:0fa9fae3eaa9d48836c855a4884f0751a90ff050dc9e7d17e41d93bb434499c0
|
||||
felhom-samba 1.1.0 sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10
|
||||
felhom-samba 1.2.0 sha256:da96f25033827fbb77d0ee398e86fe5b744e2f29be9de898994533340c97acf9
|
||||
agent 0.159.2 binary 4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9 bundle 477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9
|
||||
agent 0.160.0 binary 7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1 bundle 6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f
|
||||
agent 0.161.0 binary ff4e6dc52dd70234913445ecdeb88d31dea8e394d77c48704d26ff19b999d3b5 bundle 485f36c957d83f772aa22cac24e5e73f44da5cf8b957b1fd513bec2bc1d12c47
|
||||
agent 0.162.0 binary 5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928 bundle ac772967fd4669045e837a70bf5acca239c58d55f7f00212e472a8facd60b5a5
|
||||
agent 0.163.0 binary 561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c bundle 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8
|
||||
independent controls (other channels): controller 0.340.0 = the release digest from build.sh (03-release-0340-boxes.txt); hub 0.165.0 = the running hub pod's imageID; hub 0.164.0 = hub/CHANGELOG; agent 0.163.0 = release-agent.sh today; agent 0.161.0 = night2-2026-10-20/P1/1b; bundle 0.160.0 = the three boxes' installed bundle_sha256
|
||||
|
||||
## load on a scratch machine (9202 on demo-hp, Tier 0)
|
||||
the ep0 copy of felhom-controller 0.340.0 → skopeo docker-archive (amd64) → pct push 9202 → docker load:
|
||||
Loaded image: felhom-restorecopy-test:0.340.0; ID=sha256:3f34d74519463c338422594b70c31d9c5beabed48847134c6d26b5a95ebbae2b = the copy's amd64 config digest; a container ran (ls /)
|
||||
teardown: image untagged (0 left), the tar removed on 9202, on demo-hp /tmp and on DooPlex
|
||||
|
||||
timer: Sat 2026-10-24 01:42:16 CEST 15h - - felhom-release-copy.timer felhom-release-copy.service
|
||||
|
||||
## alarm (homelab-manifests 1bbef892): FelhomReleaseCopyStale (> 26 h or absent, warning) — promtool check: 7 rules OK; promtool test: quiet at 25 h, fires at 27 h
|
||||
synced ConfigMap mon-system/prometheus-rules only; POST /-/reload 200; /api/v1/rules: FelhomReleaseCopyStale inactive ok; metric age at read ~3235 s
|
||||
@@ -0,0 +1,10 @@
|
||||
no guest references the three
|
||||
BEFORE: /dev/mapper/pve-root 39G 29G 8.5G 77% /
|
||||
Removed volume 'local:iso/felhom-installer-1.27.1-pve9.2-1.iso'
|
||||
removed local:iso/felhom-installer-1.27.1-pve9.2-1.iso
|
||||
Removed volume 'local:iso/felhom-installer-1.28.0-pve9.2-1.iso'
|
||||
removed local:iso/felhom-installer-1.28.0-pve9.2-1.iso
|
||||
Removed volume 'local:iso/felhom-installer-1.29.0-pve9.2-1.iso'
|
||||
removed local:iso/felhom-installer-1.29.0-pve9.2-1.iso
|
||||
AFTER: /dev/mapper/pve-root 39G 24G 14G 64% /
|
||||
local:iso/felhom-installer-1.30.0-pve9.2-1.iso iso iso 1705324544
|
||||
@@ -0,0 +1,16 @@
|
||||
# The vouch read-back — agent 0.163.0 (the operator vouched on the hub, answered 'Vouched now'), read 2026-10-23T08:00:42Z
|
||||
|
||||
## channel 1: hub DB (hub.db + -wal + -shm in ONE tar from the pod), hub_settings
|
||||
artifact_agent_sha256|561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c|2026-10-23 07:58:46
|
||||
artifact_agent_version|0.163.0|2026-10-23 07:58:46
|
||||
artifact_bundle_sha256|3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8|2026-10-23 07:58:46
|
||||
artifact_golden_sha256|3881de62b0986ccbe5e55580e3f1bb9765d65b8d898f67c6339bac105244f84f|2026-10-23 07:58:46
|
||||
artifact_golden_version|0.331.0|2026-10-23 07:58:46
|
||||
artifact_min_agent|0.158.1|2026-10-23 07:58:46
|
||||
artifact_wrapper_sha256||2026-10-23 07:58:46
|
||||
|
||||
## channel 2: the running hub's operator page GET /configuration (CC's token), the selected options
|
||||
value="0.163.0" data-sha="561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c" selected>0.163.0
|
||||
value="0.331.0" data-sha="3881de62b0986ccbe5e55580e3f1bb9765d65b8d898f67c6339bac105244f84f" selected>0.331.0
|
||||
|
||||
agent 0.163.0 sha256 = the release (561b5316…); bundle sha256 = the release (3afb88d9…). Golden 0.331.0 unchanged. The floor's MinAgent stays 0.158.1 (artifact_min_agent 0.158.1; controller 0.340.0 declares 0.158.1).
|
||||
@@ -0,0 +1,13 @@
|
||||
# Installer 1.36.0 published (operator yes 2026-10-23), read 2026-10-23T08:16:46Z
|
||||
|
||||
1. bump: felhom.eu 83388c08 (SCRIPT_VERSION 1.36.0; scripts/test_hostinstall.py 40 passed; CI 2247 success)
|
||||
2. tag: installer-v1.36.0 (annotated d00f868c) -> 83388c08, pushed
|
||||
3. pins: manifests/webpage.yaml both --ref=installer-v1.36.0 (sidecar git-sync-scripts + init git-sync-scripts-init), commit 2aebf1bd;
|
||||
ArgoCD: only Deployment felhom-system/felhom-webpage was OutOfSync; synced that ONE resource at revision 2aebf1bd; rollout ok;
|
||||
live args: git-sync-scripts-init ['--ref=installer-v1.36.0'], git-sync-scripts ['--ref=installer-v1.36.0'], website git-sync ['--ref=main'] (unchanged)
|
||||
4. read back: curl https://felhom.eu/scripts/felhom-host-install.sh
|
||||
SCRIPT_VERSION="1.36.0" # the SINGLE version source (F-1): -h and the run banners follow it.
|
||||
sha256 served = de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec
|
||||
sha256 tag file = de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec
|
||||
the drop-in name 10-felhom-root-key-only appears 2 times (the uninstall's removal)
|
||||
Rollback: move the tag back to installer-v1.35.0's commit and wait ~30 s (no sync).
|
||||
@@ -0,0 +1,4 @@
|
||||
# Website publish read-back (felhom.eu 0bb57a4f, operator yes 2026-10-23)
|
||||
Method: headless Chromium (playwright 1.61.1 image) on the LIVE site; the FAQ answer opened by a click; 390x844 (phone) and 1366x900 (computer).
|
||||
No page scrolls sideways at either size (scrollWidth == viewport on all 10 shots).
|
||||
Text: see text-check.txt (new sentences present; the old 'only encrypted (the remote backup)' gone; control: HEAD~1 had it twice).
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 144 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 122 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 113 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 122 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 124 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 70 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 54 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 59 KiB |
@@ -0,0 +1,7 @@
|
||||
waited ~0 s
|
||||
NEW letoltes 'az alapokat futtatja' count 1 (want 2 on the FAQ: visible + JSON-LD; 1 on download)
|
||||
NEW en/download 'runs the basics; the photo app needs 16 GB' count 1 (want 2 on the FAQ: visible + JSON-LD; 1 on download)
|
||||
NEW gyik 'a Cloudflare h' count 4 (want 2 on the FAQ: visible + JSON-LD; 1 on download)
|
||||
NEW en/faq "passes through Cloudflare's network and is readable there" count 2 (want 2 on the FAQ: visible + JSON-LD; 1 on download)
|
||||
OLD gyik 'otthonodat (a t' live 0 (want 0); control HEAD~1 2 (want 2)
|
||||
OLD en/faq 'leaves your home only encrypted (the remote backup)' live 0 (want 0); control HEAD~1 2 (want 2)
|
||||
@@ -0,0 +1,43 @@
|
||||
# Part 5 — D34: the hub DB off-site every 6 hours (operator yes 2026-10-23), 2026-10-23T08:41:15Z
|
||||
|
||||
## what changed
|
||||
- hub 0.167.0 (deployed 08:27Z, image sha256:177860a9…): snapshots at 02:00 08:00 14:00 20:00 Budapest; start-up catch-up when the newest > 6 h. Hub log at start:
|
||||
2026/10/23 10:28:12 [INFO] db-snapshot 08:00: next run at 2026-10-24 08:00 CEST (in 21h31m48s)
|
||||
2026/10/23 10:28:12 [INFO] db-snapshot 14:00: next run at 2026-10-23 14:00 CEST (in 3h31m48s)
|
||||
2026/10/23 10:28:12 [INFO] db-snapshot 20:00: next run at 2026-10-23 20:00 CEST (in 9h31m48s)
|
||||
2026/10/23 10:28:12 [INFO] db-snapshot 02:00: next run at 2026-10-24 02:00 CEST (in 15h31m48s)
|
||||
2026/10/23 10:30:06 [INFO] db snapshot written: hub-20261023T082812Z.db (525996032 bytes, 1m54.537s); pruned 1, keeping 2
|
||||
- DooPlex felhom-hub-db-backup (felhom.eu 52af38df): timer 02:30 08:30 14:30 20:30; waits ≤ 20 min for a snapshot being written; refuses a newest snapshot older than 7 h
|
||||
- R-1234 (closed): the first version (2 h, no wait, be7e97dc) refused the timer's own catch-up at 10:28 CEST and mailed admin@ once — fixed in 52af38df
|
||||
|
||||
## retention (ep0, server-side, unchanged): prune-job prune-operator-hubdb, ns operator, 03:45, keep-daily 14, keep-weekly 8
|
||||
=> the current day's copies (up to 4) stay until the next 03:45 prune, then the newest of each day for 14 days and one per week for 8 weeks. Size per copy: 501.6 MiB raw, 24.5 MiB uploaded (compressed, dedup'd chunks).
|
||||
|
||||
## run once by hand (systemctl start felhom-hub-db-backup.service, 10:39 CEST)
|
||||
2026-10-23T10:39:33+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: snapshot hub-20261023T082812Z.db, 11 min old
|
||||
2026-10-23T10:39:45+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: checked: 525996032 bytes, integrity ok, 4 host(s)
|
||||
2026-10-23T10:39:48+02:00 dooplex felhom-hub-db-backup[2377041]: hubdb.pxar: had to backup 501.629 MiB of 501.629 MiB (compressed 24.471 MiB) in 2.47 s (average 203.437 MiB/s)
|
||||
2026-10-23T10:39:49+02:00 dooplex felhom-hub-db-backup[2377041]: Duration: 3.21s
|
||||
2026-10-23T10:39:49+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: pushed hub-20261023T082812Z.db to ep0 (ns operator) in 4 s
|
||||
2026-10-23T10:39:49+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: success signal written
|
||||
|
||||
## read back: restore the NEWEST copy from ep0 with the READ-ONLY token into a scratch dir (shredded after)
|
||||
15 copies on ep0, oldest 2026-10-10T00:30:56Z
|
||||
newest copy on ep0: host/dooplex-hub/2026-10-23T08:39:45Z
|
||||
Using encryption key from '/etc/felhom-hub-backup/enc.key'..
|
||||
Fingerprint: b2:19:bf:36:3b:97:3d:6c
|
||||
restore complete (501.629 MiB processed in 5.4s, average 92.519 MiB/s)
|
||||
restored size: 525996032 bytes
|
||||
restored sha256: a9ff4c8a0d0fe34da72411d9c165e573069f5601492d1e625578464f56adbde7
|
||||
integrity_check: ok
|
||||
SELECT COUNT(*) FROM hosts: 4
|
||||
SELECT value FROM hub_settings WHERE key='artifact_agent_version': 0.163.0
|
||||
pod snapshot hub-20261023T082812Z.db: 525996032 bytes, sha256 a9ff4c8a0d0fe34da72411d9c165e573069f5601492d1e625578464f56adbde7 (kubectl exec sha256sum — a different channel from the restore)
|
||||
=> byte-identical; it holds today's vouch (artifact_agent_version 0.163.0, set 07:58Z) — a nightly copy could not
|
||||
|
||||
## timers
|
||||
Fri 2026-10-23 14:30:08 CEST 3h 48min Fri 2026-10-23 10:28:30 CEST 12min ago felhom-hub-db-backup.timer felhom-hub-db-backup.service
|
||||
|
||||
## alarm (homelab-manifests 1bbef892): HubDBBackupStale 26 h -> 13 h (two missed 6-hourly pushes)
|
||||
promtool test: quiet at 12 h, fires at 14 h; RED-PROOF: the rule at 26 h fails the 14 h case
|
||||
in force: /api/v1/rules shows the 13 * 3600 expression, inactive ok; metric age at read ~237 s
|
||||
@@ -0,0 +1,21 @@
|
||||
# Part 3 — hub 0.167.0 deploy and the 30-minute watch (operator yes 2026-10-23), read 2026-10-23T08:56:31Z
|
||||
|
||||
build: clean clone at d306b8c2 (hub/ unchanged since be7e97dc, CI 2250 success; d306b8c2 CI 2251 success before the deploy); hub tests rc 0 (28 packages)
|
||||
image: gitea.dooplex.hu/admin/felhom-hub:0.167.0 sha256:177860a996cb285d935777d6ffdf85725f6cf4489885f54fd3bf8135c68041a0
|
||||
manifest 34979d5d; ArgoCD: only Deployment felhom-system/hub OutOfSync; synced that one resource; rollout ok 08:27:34Z; pod imageID = the digest above; Synced/Healthy
|
||||
|
||||
## watch 08:27:34Z → 08:56:31Z — channel 1: the hub DB (with WAL) + log
|
||||
events since the deploy: 0 (newest event 07:07:10Z Tester4 host_down — the expected dead test box; 161 events since 2026-10-22: the table is live)
|
||||
host report|demo-felhom-8363b5|2026-10-23 08:54:54|2
|
||||
host report|demo-hp-bb76ea|2026-10-23 08:49:41|2
|
||||
host report|tester-1-d70be4|2026-10-23 08:42:35|1
|
||||
controller report|demo-felhom|2026-10-23 08:53:20|3
|
||||
controller report|demo-hp|2026-10-23 08:53:24|3
|
||||
controller report|tester-1|2026-10-23 08:53:23|3
|
||||
hub log since deploy: 0 [ERROR]; 'osupdates: alarm check done … 0 alarm(s) sent'; 'Host staleness checker initialized: 3 ok, 1 already alarmed (seeded silently, R-1001)' — Tester4 not re-mailed
|
||||
|
||||
## channel 2: the operator mailbox (Gmail connector, to:admin@felhom.eu newer_than:1d)
|
||||
after 08:27Z: ONE mail, 08:28:32Z '[DooPlex backup] FAILED: felhom-hub-db-backup.service' — NOT from the hub: CC's install of the 6-hourly timer (R-1234, closed, fixed 52af38df). No hub mail after the deploy.
|
||||
before the deploy (control that the channel delivers): 07:07:11Z Tester4 host_down, 07:04:11Z node_down, 06:22Z/06:19Z stale — the hub's mails do arrive
|
||||
|
||||
verdict: no new false alarm from the hub in the 30 minutes after the deploy; one CC-caused DooPlex mail, filed and fixed
|
||||
Reference in New Issue
Block a user