9 Commits

Author SHA1 Message Date
admin 87dc50743c D39 = B: 09 §3 decision 255 (the security-fix step) + a dated note on decision 13; capability B23; R-1239 closed, R-1240 opened, R-1238 updated (register 186 -> 186); STATUS/CONTEXT; nodes.md (9202 password, Tester 1 DHCP); evidence security-lane-2026-10-23 (controller 0.341.0, global floor 11:24 UTC)
gates / gates (push) Has started running
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-23 13:35:38 +02:00
admin 4b9b0af3cb Plex2Shell: the operator's 10:56 ruling in CONTEXT; demo-hp 9201 checked (clean) and updated to 1.43.5.11029; register 184 -> 186 (opened R-1238 Plex test record/fixture, R-1239 a fast lane for security fixes = D39); evidence plex-demo-boxes-2026-10-23
gates / gates (push) Successful in 13m46s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-23 11:27:52 +02:00
admin 2c9152d912 Day 2026-10-23 close-out: rulings D31-D36 in CONTEXT; STATUS (releases done, do list, D35 dated, D37/D38); register 191 -> 184 (closed R-49 R-982 R-985 R-991 R-1204 R-1205 R-1225 R-1226 R-1227 R-1230 R-1234; opened R-1235 R-1236 R-1237); the break-glass sheet current (D32, one A4, HU+EN); runbooks for D33/D34; poster facts; evidence day-2026-10-23
gates / gates (push) Successful in 12m53s
2026-10-23 11:03:45 +02:00
admin 52af38df6f hub-db-backup: wait for a snapshot being written, limit 7 h (one slot gap + 1 h) — the 2 h limit refused the hub's own catch-up and mailed a false failure at 10:28 (R-1234, closed); tests red-proved
gates / gates (push) Successful in 13m14s
2026-10-23 10:35:16 +02:00
admin 34979d5d25 hub manifest: 0.165.0 -> 0.167.0 (operator yes 2026-10-23: D31, the clock-ahead tunnel alarm, R-1230, D34 6-hourly snapshot; image sha256:177860a9…)
gates / gates (push) Successful in 13m26s
2026-10-23 10:23:29 +02:00
admin 0bb57a4fee website: publish D3/D5 (the whole truth on privacy; the minimum machine) and D36's two sentences, HU + EN (operator yes 2026-10-23; R-985, R-991, R-1225, R-1226)
gates / gates (push) Successful in 13m10s
2026-10-23 10:16:54 +02:00
admin 2aebf1bde1 webpage: serve installer-v1.36.0 (both git-sync pins; operator yes 2026-10-23 — root SSH by key only on new installs, R-982 D4)
gates / gates (push) Successful in 13m12s
2026-10-23 10:08:09 +02:00
admin d306b8c2d7 release-copy: the released images (skopeo dir:, digest-true) and agent packages copied to ep0 nightly, verified by sha256 and a checksum compare, capped at 3 GB (D33, R-1227); tests red-proved
gates / gates (push) Successful in 15m56s
2026-10-23 09:39:31 +02:00
admin be7e97dc1a hub 0.167.0 + DooPlex push: the hub DB snapshot every 6 hours (D34, operator yes 2026-10-23, R-232 G5) — slots 02/08/14/20, catch-up 6 h, push :30 after each, refuse a copy older than 2 h; tests red-proved
gates / gates (push) Successful in 16m20s
2026-10-23 09:12:08 +02:00
110 changed files with 2942 additions and 135 deletions
+25
View File
@@ -16,6 +16,31 @@
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
> **2026-10-23 ~12:00 — operator rulings by starting the D39 brief (`09` §3 decision 255; R-1239 closed).** **D39 = B:** a
> `security_fix` ladder step — box walk on 9202 required, bench optional, taken by the night leg. **No operator word per
> incident:** a public advisory id (CVE or GHSA) + its https:// notice is the authority; the session that writes the step
> reports it to the operator after the push. The design (Claude.ai's recommendation) was taken as written; one fact in the
> brief was checked and held (Plex's box fixture exists but is a `NoRoute` stub, so `seed: "none"` is right). Built:
> catalog `3898593` (`ladder.py` `check_security` + `engine_majors_crossed` importing `check-engine-major.py`; the move gate
> accepts it; `check-test-record.py` WARNs on a head with no bench; `upgrade-test.py --write-ladder --security`;
> `box_walk.py --security` + `refresh_box_mirror()`), controller **0.341.0** (`legCandidate` takes `proven` or
> `security_fix`; global floor 11:24 UTC with MinAgent 0.158.1, the operator's yes in chat). Proof:
> `audits/security-lane-2026-10-23/`. Older controllers skip the step at night and take it on a hand press (proven).
> **2026-10-23 10:56 — operator ruling: Plex to the fixed version now, without the test record (Plex2Shell).**
> Verbatim: *„bump the version to the newest in the app template immediately (as there are no customers yet), I want the
> demo boxes on the newest version asap."* It skips `09` §3 decision 13's test record **once, for Plex only**. Done: catalog
> `6cc317d` (Plex 1.41.4.9463-630c9f557 → 1.43.5.11029-6bf8af3f0, CVE-2026-96652/-96654/-96655/-96656), pushed
> `--no-verify` past `check-test-record-move.py` (its refusal is in `app-catalog-felhom.eu/REPORT.md`); the only box running
> Plex (demo-hp 9201) was checked for the attack's signs (none) and updated by the Update press; the guide re-checked
> (`guide.checked_version` moved). Decision 13 itself is unchanged. Follow-ups: R-1238 (Plex's test record and fixture),
> R-1239 (a fast lane for security fixes — the operator's decision, D39). Evidence: `audits/plex-demo-boxes-2026-10-23/`.
> **2026-10-23 morning — the operator's „go" for the day brief: the releases, D31 as built, D32–D36 (decide rows in `STATUS-HISTORY.md`).**
> **Releases:** controller 0.340.0 (global floor 06:41 UTC, MinAgent 0.158.1); agent 0.163.0 (binary sha256 561b5316…, bundle 3afb88d9…, vouched by the operator 07:58 UTC; delivered to the three test boxes by signed jobs through a step bundle `0.163.0-step1` — the D4 drop-in is a new path, R-880); installer 1.36.0 (tag `installer-v1.36.0` → 83388c08, both webpage pins); hub 0.167.0 (sha256:177860a9…, 08:27 UTC); the website `0bb57a4f`.
> **D31** = as built (the re-issue reads the newest deletion of a KEY-HOLDING host; an install that got the password but no recovery package is rotated without a press). **D32** = yes: the sheet is names and places only, one A4, HU + EN; the operator prints and fills it. **D33** = yes: `scripts/release-copy/` nightly 01:40, newest 3 image tags (controller, hub, samba) + 5 agent versions, skopeo `dir:` (digest-true — why not `docker save`: a re-push re-compresses), ep0 `/srv/felhom-release-copy` under the one limited user `felhom-relcopy` (rrsync), cap 3 GB enforced by the sender, a read-only registry token of its own. **D34** = yes: hub snapshots at 02/08/14/20 (catch-up > 6 h), DooPlex push :30 after, limit 7 h + a wait for a snapshot being written (R-1234: 2 h was wrong), alarm 13 h; ep0 retention unchanged (keep-daily 14, keep-weekly 8). **D35** = yes, after the release week (R-1228, dated 2026-10-30). **D36** = yes, both sentences, published.
> **Added on the operator's word the same day (not soaked):** D34's hub half went into 0.167.0 (asked before the build). Record: `audits/day-2026-10-23/`, `REPORT-day-2026-10-23.md`.
> **2026-10-22 evening — operator rulings D29, D30, D31 (CC's picks accepted; the decide rows now in `STATUS-HISTORY.md`).**
> **D29** = no: the controller repository stays public for now (R-925). **D30** = the agent makes a new WireGuard key and
> registers it on the existing peer path (R-1204). **D31** = the hub skips a deleted install that never made a key when it
+103
View File
@@ -0,0 +1,103 @@
# REPORT — day 2026-10-23: the releases and the DooPlex hour (the operator's „go")
Design read: `07-backup-architecture.md` §6.3/§8, `09-update-architecture.md` §3 decision 250, `03-host-agent.md`,
`05-hub-architecture.md` §16.3, `runbooks/total-loss-of-dooplex.md`, `audits/dooplex-survival-2026-10-22/PLAN.md`.
Evidence: `documentation/audits/day-2026-10-23/01`–`14`.
## Five lines
1. Released: controller 0.340.0 (global floor), agent 0.163.0 (vouched by the operator), installer 1.36.0, hub 0.167.0,
the website with D36's two sentences.
2. Root SSH from the LAN is key-only on all three test boxes; new installs get it from installer 1.36.0.
3. The hub database now leaves every 6 hours (restored back byte-identical); the released images and agent packages
leave nightly to ep0 (pulled back, every sha256 OK, one image loaded on a scratch box).
4. One false failure mail at 10:28 (my install of the 6-hourly copy) — filed R-1234, fixed and red-proved the same hour.
5. Register 191 → 184; the operator: print the sheet, Hosts page clean-up, D37 + D38.
## Per box — versions before → after, and the SSH checks
| Box | controller | agent | root-file bundle | `sshd -T` root | sshd reloaded | key login | felhom-sshd | swap = RAM (9201) |
|---|---|---|---|---|---|---|---|---|
| Tester 1 | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 (via 0.163.0-step1) | without-password | yes, 09:27:38 | ok | active, `sshd -t` ok | 4096/4096 |
| demo-hp | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 | without-password | yes, 09:19:44 | ok | active, `sshd -t` ok | 25898/25898 |
| demo-felhom | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 | without-password | yes, 09:24:42 | ok | active, `sshd -t` ok | 12288/12288 |
| hub | 0.165.0 → 0.167.0 | | | | | | | |
Not proven (said plainly): the real root password over SSH (CC does not hold it — a dummy one is refused, but OpenSSH logs
only „Failed password"); a felhom-sshd connection (it admits only the operator's tunnel peer); the physical console (the
drop-in does not touch it; `passwd -S root` = P).
## Digests, sha256, CI by commit
| What | Value |
|---|---|
| controller 0.340.0 | `sha256:a8d1c6458da4ea9d667f928498656a291771e0b3756b20f2fcc8c1cf9d516621` — `caaa86e`, CI 2245 success |
| agent 0.163.0 binary / bundle | `561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c` / `3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8` — `1109092` CI 2246, `b4721ee` CI 2249 success |
| step bundle 0.163.0-step1 | `6cbc066902d5f3294765d5c7dc7113166c1dca25e994dbc36bfc5f139bb61fcf` |
| installer 1.36.0 (served = tag) | `de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec` — `83388c08` CI 2247 success; pins `2aebf1bd` |
| hub 0.167.0 | `sha256:177860a996cb285d935777d6ffdf85725f6cf4489885f54fd3bf8135c68041a0` — built at `d306b8c2` (hub/ = `be7e97dc`, CI 2250 success; `d306b8c2` CI 2251 success); manifest `34979d5d` |
| website | `0bb57a4f` |
| D34 fix (R-1234) | `52af38df` |
| prometheus rules | homelab-manifests `1bbef892` |
CI of the last commits (`2aebf1bd`, `0bb57a4f`, `34979d5d`, `52af38df` and this close-out): see the final message.
## Parts
- **1 — controller 0.340.0:** 9202 smoke, per-customer floors, three boxes green, box walk + https-only rc 0, global floor
06:41 UTC (MinAgent 0.158.1), read back with the WAL (`01`–`06`).
- **2 — agent + installer:** release, a step bundle because the D4 drop-in is a new path (R-880), three signed jobs per
box (`07`); the operator's vouch read back from the DB and the live page (`10`); installer tag, both pins, a
webpage-only sync, the served script = the tag (`11`).
- **3 — hub 0.167.0:** built from a clean clone; on the operator's word it also carries D34's 6-hourly snapshot (asked
before the build; not soaked); hub-only sync; 30-minute watch: 0 hub events, 0 errors, all boxes reporting, no hub mail
(`14`).
- **4 — website:** D3/D5 patch + D36 HU + EN; site gates; published; read back live: new text present, old gone (control:
the previous commit), screenshots at 390 and 1366 px, no sideways scroll (`12-website-readback/`).
- **5 — D34:** retention unchanged, server-side on ep0 (`prune-operator-hubdb` 03:45, keep-daily 14, keep-weekly 8 — the
current day's up to 4 copies until the next prune). Hand run 501.6 MiB / 24.5 MiB uploaded in 4 s; the newest copy
restored from ep0 with the read-only token = the pod's snapshot by sha256 `a9ff4c8a…`, integrity ok, 4 hosts, holds
today's vouch. Alarm 13 h. Runbook updated (`13`).
- **6 — D33:** ep0 before: `/` 14 GB free (the data disk's root IS the PBS datastore — not used). One user
`felhom-relcopy` (locked, rrsync-confined: shell and `..` refused), one directory `/srv/felhom-release-copy`. Cap 3 GB,
enforced by the sender (ep0 has no quota). **skopeo, not `docker save`:** `dir:` keeps the registry's bytes, so a push
back keeps the digest; no Docker daemon. A read-only registry token of its own, minted in the Gitea pod straight into a
0600 file. First run 676 MB, 91 s; pulled back: 141 files OK, 9 images re-verified (digests match independent records),
controller 0.340.0 loaded on 9202 (ID = config digest). Alarm `FelhomReleaseCopyStale`. Restore steps in
`total-loss-of-dooplex.md` (`08`).
- **7 — D32:** the sheet is names and places only, HU + EN, one A4 (rendered: 1 page; a copy in the operator's home
folder); every place checked to exist. demo-hp: three superseded ISOs removed by name, `/` 8.5 → 14 GB free (`09`).
R-548: the box half done, the product fix proposed (R-921's pattern).
- **8 — close-out:** CONTEXT rulings, STATUS (decide rows D32–D36 → history, D37/D38 new, the do list, D35 dated
2026-10-30), REPORTs and CHANGELOGs; the system poster's facts file updated, its hub-DB label edited (text), and a new
arrow asked for in STATUS („System poster needs a refresh").
## Register — 191 → 184; opened 4, closed 11
Opened: R-1234 (closed the same hour), R-1235, R-1236, R-1237. Closed: R-49, R-982, R-985, R-991, R-1204, R-1205,
R-1225, R-1226, R-1227, R-1230, R-1234. Updated: R-1214 (released; dated check 2026-10-24), R-1187, R-1189, R-232,
R-548, R-923, R-924, R-1228 (D35, dated 2026-10-30), R-1183's check text (four snapshot slots).
## Observations
- The D34 first push refused the hub's own catch-up and mailed admin@ — FILED: R-1234 (closed, fixed `52af38df`).
- The release copy is proven to load, not to restore a registry; it follows the newest versions, not the floor/vouch; no
golden — FILED: R-1235.
- Gitea OOM-killed at its 1 GiB limit, the last at 06:41:10 UTC during the release — FILED: R-1236 (STATUS D37).
- The night sweep's mail: one-time test-box codes in one session record — FILED: R-1237.
- The hub DB snapshot is 501 MiB, not 24 MB — NOT-A-FINDING: 24 MB is the compressed upload, which is what ep0 stores.
- felhom-sshd does not answer from the LAN — NOT-A-FINDING: its firewall admits only the operator peer (H1, by design).
- A dummy root password logs „Failed password", not a policy refusal — NOT-A-FINDING: OpenSSH's behaviour under
`prohibit-password`; the policy is proven by `sshd -T` and the reload.
- The vouch left `artifact_wrapper_sha256` empty — NOT-A-FINDING: an optional field; the wrapper rides the vouched bundle.
- Guests 9202, 9401 (demo-hp) and a transient 990000 (Tester 1) have swap ≠ RAM — NOT-A-FINDING: outside the felhom pool.
- `known_hosts` holds an older key for 192.168.0.155 (demo-hp's guest address) — NOT-A-FINDING: guests are reached by
`pct exec`; nothing was changed.
- The D4 drop-in needed a step bundle — NOT-A-FINDING: the documented R-880 path for a new bundle path.
## Teardown — three layers
Machine: 9202 — the test image untagged, the tar removed. Host: demo-hp `/tmp` tar removed; three ISOs removed on purpose
(Part 7); DooPlex: the read-back copies and the hub-DB scratch restore shredded/removed. Hub: nothing provisioned; the
per-customer floors cleared after the global floor. Left on purpose: the ep0 user + directory, the DooPlex release-copy
job, the 6-hourly hub-DB timer, both alarms.
+42
View File
@@ -0,0 +1,42 @@
# REPORT — Plex to the fixed version in the catalog and on the demo/test boxes (2026-10-23 day, Plex2Shell)
**Plain words first.**
- The catalog now gives Plex 1.43.5.11029-6bf8af3f0 (catalog `6cc317d`, pushed without the update test, on your ruling).
- One box ran Plex: demo-hp. It showed no sign of the attack. CC pressed Update; it runs the new Plex now.
- You have one decision (D39): may a security fix reach the boxes at night without the full update test?
## Per box
| Box | Plex before → after | Signs check | Box walk |
|---|---|---|---|
| demo-hp 9201 | 1.41.4.9463-630c9f557 → 1.43.5.11029-6bf8af3f0 | clean (NOT-A-FINDING; logs reach back only to 00:32 UTC today — CPU history and files cover 10-16 → 10-22) | green: all containers healthy, 0 errors, 0 warnings since the press |
| demo-hp 9202, demo-felhom 9201, tester-1 9201 | no Plex | — | — |
| Tester4 | hub host DOWN, VM gone | — | — |
| Tester 2 | no host on the hub | — | — |
Detail and evidence: `documentation/audits/plex-demo-boxes-2026-10-23/README.md`.
## Records
- Register 184 → 186: opened R-1238 (P2, Plex has no test record or fixture), R-1239 (P2, a fast lane for security
fixes — the operator's decision); closed none.
- `CONTEXT.md`: the 10:56 ruling. `STATUS.md`: decide row D39. No „broken" line: Part 2 was finished before the
records were written, so the line the brief asked for would have been added and removed in the same commit.
The „Updated" paragraph was left as the day session wrote it (it is that session's summary; this run adds one
decide row only).
- Architecture read: `09-update-architecture.md` §3 decisions 13 and 52; `13-threat-model.md` (Plex row);
`runbooks/target-selection.md`. Decision 13 is unchanged; no poster fact changed (none of the three poster actions).
- `--no-verify` was used ONCE, for catalog `6cc317d`, on the ruling. The `check-test-record-move.py` refusal is verbatim
in `app-catalog-felhom.eu/REPORT.md`.
## Fixed without a row
- The memory index (`.claude-memory/MEMORY.md`, outside git) had grown to 201 lines / 25 735 bytes at 11:09 (another
session), over the 200 / 25 600 ceiling, so `instructions_gate` failed and so did two script suites that run it.
Lines 71–72 (two memories about the deleted credentials file, both superseded by `sec`) became one line: 200 lines,
25 590 bytes. `repo_gates.py --fast` then OK. The index sits right at the ceiling; the next added line trips it again.
## Teardown
Machine: Plex stays on the new version; helper scripts removed from guest and host. Host: nothing provisioned.
Hub: reads only.
+17
View File
@@ -0,0 +1,17 @@
# REPORT — D39 = B: the security-fix lane (2026-10-23, attended)
**Plain words first.** An app with a public security hole can now get its fix at night on every box, after one test
on the test box. Controller 0.341.0 does it and is the floor since 13:24 (your yes). Older controllers skip such a fix
at night and take it when a person presses Update — both seen on the test box. Nothing needs you.
- Design: `09-update-architecture.md` §3 decision 255 (+ a dated note on decision 13). Capability map B23.
- Catalog `3898593` (+ a box-walk address fix after it); controller `63dcccf` = 0.341.0; drill commits `8c1d908` …
`17e0da0` (the drill only).
- Evidence and the full table: `documentation/audits/security-lane-2026-10-23/README.md`.
- Register 186 → 186: closed R-1239 (D39 = B); opened R-1240 (9202's 34 saved configs with the catalog token); R-1238
stays open, its text now says the next Plex fix can use the lane.
- `STATUS.md`: D39 out of the decide table, one „what works" line, the „Updated" paragraph replaced (the old one is in
`STATUS-HISTORY.md`). `CONTEXT.md`: the rulings. System poster: no fact changed (none of the three actions).
- Instruction / fact files edited: `app-catalog-felhom.eu/CLAUDE.md` (the lane, added); `operations/nodes.md` line 221
(9202 „same dashboard password" → its own password, measured) and line 46 (Tester 1 `.101` → DHCP, `.148` today).
- No `--no-verify` in this run.
+11 -16
View File
@@ -2,7 +2,7 @@
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
**Updated 2026-10-23 (morning).** Your three picks are built (the tunnel repairs itself with a new key, the hub skips a keyless deleted install, the controller repository stays public), and the night copy can no longer outlive the controller. Four fresh reviews found 34 points, every real one fixed or a row; the test night PASSED (no failed check at night, every night job passed, nothing lost). Nothing was released or deployed; the morning list has the order, and the DooPlex survey asks four questions (D32–D35).
**Updated 2026-10-23 (afternoon).** Your D39 = B is built and live: an app with a public security hole can now get its fix at night on every box, after one test on the test box — controller 0.341.0 is the floor since 13:24 (your yes). The test: Plex moved by itself at night on the test box, and an older controller safely waited for a hand press. Nothing needs you.
## What works
- A box installs from nothing with the public installer, links to the hub, and moves to the floor by itself (map A1, A3, A4).
@@ -12,6 +12,7 @@
- After a power cut the box starts by itself, cleans up a half-written backup, and Immich repairs broken photos (map C10, F12).
- A second drive is offered as the backup target; media apps see every drive; a one-disk box gets its own data area (map D1, D13, D15).
- The three test boxes answer only over https; the hub's key opens nothing on a box; password guesses are counted (map E12, E13, E17).
- An urgent security fix reaches the boxes at night after one test on the test box; the box backs up first and undoes a failed update (map B23).
- The hub mails alarms and recoveries, and keeps one whole-box copy per week on ep0; CC uses its own limited token, not your password (map F1, F11, G15).
## What is broken
@@ -21,7 +22,7 @@
- The scratch box cannot use agent features (R-1083).
## What we work on
- Dated checks, run by CC. 2026-10-23: no app stop when the host is busy (R-1173). 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts).
- Dated checks, run by CC. 2026-10-24: the 8 GB box's first night copy on 0.340.0 (R-1214). 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes).
## Waiting on you — decide
@@ -29,11 +30,8 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|---|---|---|---|---|---|---|
| D36 | R-1225, R-1226 | Two website sentences before the publish: „8 GB runs the basics; the photo app needs 16 GB” and „what you open from outside the home passes Cloudflare's network, readable there; the backups leave only encrypted”? | **Yes, both** (the patch's own aim is the whole truth) | — | the patch publishes with both promises a little too strong | — |
| D35 | R-1228 | After the release week, serve the website and the installer from Cloudflare, like the ISO? | **Yes, later** (no new installs stop when DooPlex is down) | keep them home: after a DooPlex loss, no install until it is back | the same | — |
| D34 | R-232 | Copy the hub database to ep0 every 6 hours instead of every night? | **Yes** (0 €, about 30 minutes, one DooPlex change) | keep nightly: up to a day of hub changes lost, e.g. a new box's recovery package | the same | — |
| D33 | R-1227 | Copy the released images and agent packages to ep0 every night? | **Yes** (0 €, about 2 hours of CC, a few GB on ep0) | a timed rebuild drill instead: about 3 hours once, and rebuilt images are not the same | after a DooPlex loss no box updates, reinstalls or installs until everything is rebuilt | — |
| D32 | R-923, R-924 | Print the break-glass sheet now (eight keys and logins + ep0's fingerprint) and keep it outside the home? | **Yes** (30 minutes; nothing else opens the off-site copies after a loss) | an encrypted USB stick outside the home: ~10 €, 1 hour, one more passphrase | after a DooPlex loss no off-site copy opens | — |
| D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — |
| D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — |
| D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — |
| D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — |
@@ -48,17 +46,14 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14.
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
**Morning, 2026-10-23 — in this order.**
1. Hold or go: CC says go — the test night passed on a fresh 8 GB box (no failed check at night, every night job passed, nothing lost; one gap: no whole-box copy to ep0, R-1232). One question from the review: D31 as built also covers an install that got the whole-box password but no recovery package — its password is rotated without your press (R-1205). CC: go. If not: revert one hub commit before item 4.
2. Release controller 0.340.0 (main after tonight; tested as rc.8 on the scratch box, rc.6 on the test box): the safe night copy, the catch-up fix, R-49, R-1187, R-1189, R-1214. If not: these wait.
3. Release the agent (the tunnel repair, its tunnel field, the root-SSH bundle) and the installer tag with the root-SSH removal (R-1204, R-982); then vouch. If not: a clock-ahead box stays cut off; root keeps its password on the LAN.
4. Build and deploy hub 0.167.0 (D31, the tunnel alarm and its clearing). If not: a dead tunnel stays silent.
5. Publish the website patch (D3, D5), with D36's two sentences if yes. If not: the old wording stays.
6. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-16fdef (tonight's test box). If not: two dead entries alarm.
7. The DooPlex hour: D32–D35 in the table. If not: a DooPlex loss stays as the survey describes.
**Day, 2026-10-23 — your do list.**
1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens.
2. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-16fdef. If not: two dead entries alarm.
3. D37 and D38 in the table (two small yes/no). If not: Gitea may restart in release hours; no golden in the release copy.
**The rest**
- System poster needs a refresh: the new 01:40 „release copy” arrow from DooPlex to ep0 (D33); the hub-database label already says every 6 hours. If not: the drawing misses one copy.
- Pull the failed disk Z6A07P2G out of DooPlex (R-934). If not: its alarms return on 11 November. Since 2026-10-12.
- Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09.
- Two leftovers on DooPlex: the Facebook `first-post/` folder in `leftovers/2026-10-14/` (keep, commit or delete); the `deny`-list edit in homelab-manifests' `.claude/settings.json` (commit or drop). If not: they stay. Since 2026-10-14.
- Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784); print the break-glass sheet and the signing keys (R-923, R-924).
- Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784);.
- Later (yours, not urgent): R-1229 (ArgoCD's sync settings into git), R-862 (Tester 2's bootstrap, when it is back), R-504, R-1071, R-1073, R-1077, R-1042, R-132, R-908, R-779, R-904, R-232, R-433, R-933, R-213, R-832, R-388, R-882, R-883, R-814, R-1056, R-1164, R-930, R-920, R-902.
@@ -124,6 +124,7 @@
| <a id="b20"></a>[B20](capability-map-detail/B.md#b20) | App crashes → customer notified (one event per transition, no flapping spam) | controller v0.120, hub v0.48 | **IMPLEMENTED** | crash→customer mail never live-confirmed | `audits/CAMPAIGN-3-2026-07-11.md` (2026-07-11) |
| <a id="b21"></a>[B21](capability-map-detail/B.md#b21) | Post-deploy optional config (API keys etc.) with restart | controller, catalog `.felhom.yml` | **IMPLEMENTED** | config-save + restart in no campaign | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
| <a id="b22"></a>[B22](capability-map-detail/B.md#b22) | Backup classification: 13 bind-bearing apps carry mandatory/optional/excluded classes | catalog, controller v0.132–133 | **PROVEN-LIVE** | — | `audits/SPIKE-backup-classification-2026-07-14.md` (2026-07-14); `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15) |
| <a id="b23"></a>[B23](capability-map-detail/B.md#b23) | An urgent security fix reaches the boxes at night — a `security_fix` ladder step (box walk required, bench optional, a CVE/GHSA notice), taken by the night update (`09` §3 decision 255, D39 = B) | catalog `3898593`, controller v0.341.0 | **PROVEN-LIVE** | scratch 9202 (drill catalog), Plex 1.43.4 → 1.43.5; an app with no box fixture is checked by health, version and a soak only (`seed: none`) | `audits/security-lane-2026-10-23/` (2026-10-23) |
## C. Protection & recovery (the product promise)
@@ -286,6 +286,9 @@ builds them (`audits/update-rulings-2026-09-23/`).
update night measured the proxy wrong in both directions — minor moves that held
(adventurelog, outline) and a tag shape it cannot read at all (§3b Q3). The test record is the
thing the proxy was guessing at.
**Dated note 2026-10-23 (decision 255, D39 = B):** a second step kind now exists — the `security_fix` step, proven
on the box with the bench optional, for an urgent upstream fix with a public notice. Decision 13's rule is
unchanged for every other move: a step a box takes was still tested, on at least the box venue.
14. **The update ladder** (Q3). A box more than one step behind **climbs one tested step at a time, in
order, and never jumps**. Each step is the full guarded update of §6.1. **Why:** a step was tested
@@ -1031,6 +1034,52 @@ its length, and both fixes cost something the household would notice — operato
Built in controller v0.314.0 (`internal/stacks/restore_gate.go`); pinned by `restore_gate_test.go` and
`r1057_unit_gate_test.go` (ten mutations each seen failing). Proof: `audits/gate-restore-2026-10-15/9202/`.
### 2026-10-23 (~12:00) — D39 = B: the security-fix step (operator ruling by starting the run; R-1239)
255. **An urgent upstream security fix reaches the boxes at night as a `security_fix` ladder step: the box walk is
required, the bench is optional, and a public notice is the authority.** *Operator ruling 2026-10-23 ~12:00 (D39 =
B, R-1239), taken by starting the D39 brief; its design is Claude.ai's recommendation, taken with it.* Why: on
2026-10-23 the only route for the Plex2Shell fix was an operator word, `git push --no-verify` past the move gate,
and a hand press on each box — and the night leg never takes a step with no test record (R-1238). That does not
scale past the first customers and needs the operator awake.
1. **The entry** (`scripts/ladder.py` `check_security`, the one reader): `"verdict": "security_fix"` with `security`
`{ids: [CVE-…|GHSA-…], advisory: "https://…", fixed_in}`, `box_evidence` (required), `seed` `read_back`|`none`,
`memory_peak_pct` from the BOX soak with `memory_basis: "box_soak"`; `evidence` = the bench record or null — a
bench given must be `proven` (`bench_verdict`, `harness_version` ≥ 2 ride with it). Never `backfilled`,
`digest_from` (a same-tag fix stays decision 52's lane), `engine_conversion`, nor a database engine MAJOR (any
engine — MariaDB included; the judgement is `check-engine-major.py`'s own `engine_of`/`major_of`, imported).
2. **No operator word per incident.** The session that writes the step reports it to the operator after the push.
3. **The gates:** the move gate accepts a `security_fix` move exactly as a proven one (from/to per service, digests
the registry still serves); `check-test-record.py` WARNs on every push while a template's head is a security-fix
step with no bench test („<app>: security-fix step <to> has no bench test (since <tested_at>)"). Seven decoys
(`test_gate_decoys.py`), each seen failing with the rules switched off.
4. **The box venue:** `box_walk.py --security` on 9202 through the product's Update; verdict `proven` only if the
update ended `done` with no undo and no hold, the app runs, the four version observables are TO, the seed (if
any) reads back, and a soak (default 600 s, cgroup `anon` against the limit, OOM kills and restarts counted)
sees no kill and no restart. A negative control (a TO image that exits at once) comes back `failed`.
5. **The writer:** `upgrade-test.py --write-ladder --security …`, refusing a box walk not proven, a bench given and
not proven, a template not at the walk's FROM, a digest that moved since the walk, an engine major, a missing
field. The compose move, `catalog_since` and the superseded step's own files are the shared `apply_entry`.
6. **The controller (v0.341.0):** the leg takes `proven` OR `security_fix`; every other guard runs unchanged, in
the same order; it logs `[update-leg] <app>: security fix <ids> taken (<from> → <to>)` and the hub report's
step carries the ids. A person's press and the badge are unchanged. **A controller ≤ 0.340.0 skips the step at
night (`not_proven`) and a hand press takes it** — proven on 9202, below.
7. **The trade, stated.** A security-fix step skipped the bench. For an app with a box fixture the data is still
read back after the update; for an app without one (Plex today) the step is checked only by „it starts, it is
healthy, it runs the new version, and a 10-minute soak sees no kill or restart". `files_may_change` is a bench
measurement: with no bench the step carries `false`. That is accepted because the leg's own guard is not the only
net — the update takes a per-app backup first, the undo puts the pre-update data back on a failed health check
(decision 15), and a security fix that rewrites files without breaking the health check is the case the bench
would catch and this lane accepts the risk of. A session that knows a fix migrates files runs the bench.
**Proof (9202, drill catalog, 2026-10-23):** `audits/security-lane-2026-10-23/` — the negative control (TO
`busybox:1.36.1`) came back `failed` (the box undid it after 335 s) and the writer refused it; Plex 1.43.4.10903 →
1.43.5.11029 `proven` (done in 36.9 s, the four observables at TO, digest `sha256:01bdf0ad…5c5f`, soak 600 s: anon
peak 10.7 % of 2 GiB, 0 kills, 0 restarts, `seed: none`); the writer wrote the drill step (CVE-2026-96652, -96654,
-96655, -96656; Plex's own notice as CISA KEV cites it, added 2026-10-20; fixed in 1.43.3.10861); on 0.340.0 the
night leg logged `plex: skipped — not_proven` and a hand press took the step; on 0.341.0-rc.1 the night leg logged
`plex: security fix CVE-2026-96652,CVE-2026-96654,CVE-2026-96655,CVE-2026-96656 taken` and ended `done` in 30.1 s
after the update's second-drive copy and undo copy. Released as controller 0.341.0, global floor 11:24 UTC.
### 2026-10-15 (07:48–08:20) — bake first; Plex through its own key; HSTS one day (recorded before the media build)
247. ~~**Bake first.** A controller version is RELEASED only when its golden is baked.~~ **REVERSED by decision 250,
@@ -379,3 +379,28 @@
**Gap / roadmap (as written):** Remaining ~39 apps are legacy-class by design (unit-only offsite)
---
## B23 — An urgent security fix reaches the boxes at night (the security-fix step)
*Map row: [B23](../00-capability-map.md#b23) · added 2026-10-23.*
**Scenario:** an app has a published hole (a CVE or GHSA id with its notice) and a fixed upstream version. CC walks the
step on scratch 9202 through the product's Update (`box_walk.py --security`: install at FROM, the guarded Update, the
four version observables at TO, a 600 s memory soak), and the only writer turns a proven walk into a `security_fix`
ladder step. Every box on controller >= 0.341.0 takes it at night with the usual guards (backup first, undo on a
failed health check, held / stopped / needs_person / W+5h respected).
**Components:** catalog `3898593` (`ladder.py check_security`, the two test-record gates, `upgrade-test.py --write-ladder
--security`, `box_walk.py --security`); controller v0.341.0 (`legCandidate`).
**Status:** **PROVEN-LIVE** on 9202 with the drill catalog, 2026-10-23.
**Evidence:** `audits/security-lane-2026-10-23/` — the negative control (TO `busybox:1.36.1`) `failed` and the writer
refused it; Plex 1.43.4.10903 → 1.43.5.11029 `proven`; on 0.340.0 the night leg logged `plex: skipped — not_proven` and
a hand press took the step; on 0.341.0-rc.1 the night leg logged `plex: security fix CVE-2026-96652,… taken` and ended
`done` after a second-drive copy and an undo copy.
**Gap:** an app with no box fixture (Plex today, R-1238) is checked only by health, version and the soak — no data is
read back. `files_may_change` is a bench measurement and reads `false` when no bench ran (decision 255 item 7).
---
@@ -101,6 +101,7 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
- one namespace **per household** (whole-guest copies, each household's own key),
with a NEUTRAL note under it (not a red gap box): "By design: each household holds the key to its own copies; the operator cannot open them without their recovery code." — *fix (c), 2026-10-09: this is a property of the design, not a gap, and a red box said the opposite of what it means.*
- the **`operator`** namespace: two groups — **"hub database"** and **"DooPlex copy"** (code, password manager, secrets, signing keys).
- outside PBS, a plain folder: the **release copy** (images + agent packages, D33).
Prune jobs on ep0: households keep one copy per week for 2 weeks (`keep-weekly 2`, R-1162, 2026-10-19; every new household's job is made with its namespace, R-1165); `operator` keeps 14 daily + 8 weekly.
Mark ep0 with a small shield: "**Protected** — holds the only off-premises copy of real household data."
- **Hetzner Storage Box** (grey): the shared box for the households' per-app restic copies (Tier 3), one sub-account per household, add-only keys.
@@ -121,8 +122,9 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
Arrows from DooPlex (dashed, orange → ep0):
- **00:20 "DooPlex copy"**: the code (614 MB), Gitea's database and settings, the **password manager**, the nightly secrets export, the **signing keys**. Encrypted on DooPlex with the **DooPlex off-site key**. The key that writes it **cannot delete** old copies. Tested back every Sunday.
- **02:30 "Hub database"**: encrypted with the **hub-DB key**. Tested back every Sunday at 04:30.
- **Left out on purpose:** the container registry (27.7 GB) — "rebuilt from the code".
- **02:30, 08:30, 14:30, 20:30 "Hub database"** (every 6 hours since 2026-10-23, D34): encrypted with the **hub-DB key**. Tested back every Sunday at 04:30.
- **01:40 "Release copy"** (since 2026-10-23, D33): the newest 3 released images of the controller, hub and samba, and the newest 5 agent packages — not encrypted (nothing secret), into a plain folder on ep0 (`/srv/felhom-release-copy`, its own limited user, outside PBS), about 0.7 GB, capped at 3 GB.
- **Left out on purpose:** the rest of the container registry (older tags, test builds, non-Felhom images) — "rebuilt from the code".
Arrow ep0 → DooPlex (dashed, 05:00): "**ep0-copy** pull: a second copy of ep0 at home. Keeps 8 weekly copies. A daily job removes a deleted customer's copy within 30 days."
**The test machines** (small, plain tiles near DooPlex, labelled "disposable"):
@@ -133,7 +133,7 @@ code{font-family: ui-monospace, "SFMono-Regular", Menlo, monospace; font-size: 1
<div style="position: absolute; left: 1720px; top: 706px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: center;">The hub never connects into a box.</div>
<div style="position: absolute; left: 1720px; top: 784px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">00:20 DooPlex copy</div>
<div style="position: absolute; left: 1720px; top: 824px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #a85400; text-align: center;">DooPlex off-site key</div>
<div style="position: absolute; left: 1720px; top: 872px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">02:30 hub database</div>
<div style="position: absolute; left: 1720px; top: 872px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">every 6 h: hub database</div>
<div style="position: absolute; left: 1720px; top: 912px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #a85400; text-align: center;">hub-DB key</div>
<div style="position: absolute; left: 1720px; top: 962px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #0a3d73; text-align: center;">05:00 ep0-copy pull</div>
<div style="position: absolute; left: 1720px; top: 1002px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #0a3d73; text-align: center;">ciphertext only</div>
@@ -0,0 +1,6 @@
== 9202 smoke 06:37:25
switched to gitea.dooplex.hu/admin/felhom-controller:0.340.0 at 06:37:28
healthy after ~10s: Up 7 seconds (healthy)
2026/10/23 06:37:28 [INFO] [tier2] start-up: no leftover second-drive copy container
errors, unhealthy+restarting: 0 0
login page: 200
@@ -0,0 +1,3 @@
tester-1: HTTP/1.1 303 See Other Location: /customers/tester-1?flash=floor_set
demo-hp: HTTP/1.1 303 See Other Location: /customers/demo-hp?flash=floor_set
demo-felhom: HTTP/1.1 303 See Other Location: /customers/demo-felhom?flash=floor_set
@@ -0,0 +1,15 @@
### root@192.168.0.154
containers=10 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.340.0 Up 53 seconds (healthy)
errors=0
copy-containers=0
2026/10/23 06:38:21 [INFO] [tier2] start-up: no leftover second-drive copy container
### hp
containers=24 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.340.0 Up 54 seconds (healthy)
errors=0
copy-containers=0
2026/10/23 06:38:22 [INFO] [tier2] start-up: no leftover second-drive copy container
### felhom-pve
containers=9 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.340.0 Up 58 seconds (healthy)
errors=0
copy-containers=0
2026/10/23 06:38:05 [INFO] [tier2] start-up: no leftover second-drive copy container
@@ -0,0 +1,33 @@
control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
== tester-1 (enkicsifelhom.hu): 6 public name(s)
OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
== demo-hp (enkisfelhom.hu): 14 public name(s)
OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
== demo-felhom (demo-felhom.eu): 8 public name(s)
OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
== every public name redirects http -> https
@@ -0,0 +1,13 @@
== tester-1 (dashboard login with the household password, the press a browser makes)
python3: can't open file '/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/38fde0c6-e964-4ec1-b0c7-56e62d675225/scratchpad/n23/fileswalk.py': [Errno 2] No such file or directory
== demo-hp (no session): GET /__files/open -> HTTP/2 302 location: /login
== demo-felhom (no session): GET /__files/open -> HTTP/2 302 location: /login
== tester-1 06:40:35
Fájlok press: GET /__files/open -> ['HTTP/2 302'] https://files.enkicsifelhom.hu/_felhom/sso
manual second-drive copy: 200 2. mentés elindítva
2026/10/23 06:40:43 [INFO] [backup] Tier 2 copied bookstack → /mnt/felhom-drives/adatlemez/backups/secondary/bookstack (157.0 MB, 0 leg(s), 1s)
2026/10/23 06:40:46 [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sys_drive/felhom-data/backups/secondary/paperless-ngx (85.9 MB, 1 leg(s), 2s) [SSD: state-only]
2026/10/23 06:40:48 [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/adatlemez/backups/secondary/privatebin (16.4 KB, 0 leg(s), 0s)
2026/10/23 06:40:48 [INFO] [backup] Tier 2 run complete: 3 app(s) processed (incl. volume-only — F6)
copy-containers-left=0
[exited with code 0]
@@ -0,0 +1,5 @@
== global floor 06:41:04: HTTP/1.1 303 See Other Location: /configuration?flash=floor_set
clear tester-1: HTTP/1.1 303 See Other Location: /customers/tester-1?flash=floor_set
clear demo-hp: HTTP/1.1 303 See Other Location: /customers/demo-hp?flash=floor_set
clear demo-felhom: HTTP/1.1 303 See Other Location: /customers/demo-felhom?flash=floor_set
read back (with WAL): [('min_controller_version', '0.340.0', '2026-10-23 06:41:04'), ('min_controller_version_declared_min_agent', '0.340.0=0.158.1', '2026-10-23 06:41:04')]
@@ -0,0 +1,78 @@
# Part 2 — agent 0.163.0 (operator go 2026-10-23), read 2026-10-23T07:28:53Z
## release (scripts/release-agent.sh at felhom-agent 1109092, CI 2246 green; CHANGELOG b4721ee CI 2249 green)
[release-agent] built ok: sha256 561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c
[release-agent] config bundle built: sha256 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8
[publish-agent] upload OK (HTTP 201)
[publish-agent] round-trip GET verified (sha256 matches)
[release-agent] config bundle published (HTTP 201)
sha256 : 561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c
bundle : 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 (felhom-config-bundle.json — vouch it with the agent)
* [new tag] v0.163.0 -> v0.163.0
## step bundle (R-880: the D4 drop-in is a NEW path; the boxes ran bundle 0.160.0 = 6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f)
0.163.0-step1 = base 0.160.0 with only /usr/local/sbin/felhom-os-apply replaced; sha256 6cbc066902d5f3294765d5c7dc7113166c1dca25e994dbc36bfc5f139bb61fcf; PUT 201; read back by download: same sha
## delivery: three signed jobs per box (felhom-opsign, key felhom-op-1): agent_update 06:48Z, agent_config_update 0.163.0-step1 06:59Z, agent_config_update 0.163.0 07:19Z
## hp (192.168.0.104) 2026-10-23T07:26:29Z
agent: felhom-agent 0.163.0
bundle: 0.163.0 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 2026-10-23T07:19:44Z
1 BUNDLE DONE agent=0.163.0 written=1 same=30 self-check=ok
2 BUNDLE DONE agent=0.163.0-step1 written=1 same=29 self-check=ok
guest 9201: memory: 25898 swap: 25898
guest 9202: memory: 25898 swap: 512
guest 9401: memory: 10240 swap: 0
drop-in: -rw-r--r-- 1 root root 690 Oct 23 09:19 /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf
drop-in content: PermitRootLogin prohibit-password
sshd -T: permitrootlogin without-password
root password set (passwd -S): P
felhom-sshd: active; its root policy: permitrootlogin without-password
felhom-sshd listens: 0.0.0.0:8822 [::]:8822 config test: ok; firewall admits only: elements = { 10.77.0.250 }
main sshd reloaded after the drop-in: Oct 23 09:19:44 demo-hp sshd[1010]: Received SIGHUP; restarting. Oct 23 09:19:44 demo-hp systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server.
pveproxy: active
main sshd banner from DooPlex (:22): SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4
key login: ok
root password attempt rc=255 (non-zero = refused)
Oct 23 09:26:37 demo-hp sshd-session[3026730]: Failed password for root from 192.168.0.180 port 34150 ssh2
## root@192.168.0.154 (192.168.0.154) 2026-10-23T07:28:00Z
agent: felhom-agent 0.163.0
bundle: 0.163.0 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 2026-10-23T07:27:38Z
2 BUNDLE DONE agent=0.163.0 written=1 same=30 self-check=ok
2 BUNDLE DONE agent=0.163.0-step1 written=1 same=29 self-check=ok
guest 9201: memory: 4096 swap: 4096
guest 990000: memory: 4096 swap: 512
drop-in: -rw-r--r-- 1 root root 690 Oct 23 09:27 /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf
drop-in content: PermitRootLogin prohibit-password
sshd -T: permitrootlogin without-password
root password set (passwd -S): P
felhom-sshd: active; its root policy: permitrootlogin without-password
felhom-sshd listens: 0.0.0.0:8822 [::]:8822 config test: ok; firewall admits only: elements = { 10.77.0.250 }
main sshd reloaded after the drop-in: Oct 23 09:27:38 felhom sshd[890]: Received SIGHUP; restarting. Oct 23 09:27:38 felhom systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server.
pveproxy: active
main sshd banner from DooPlex (:22): SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4
key login: ok
root password attempt rc=255 (non-zero = refused)
Oct 23 09:28:08 felhom sshd-session[4184378]: Failed password for root from 192.168.0.180 port 33442 ssh2
## felhom-pve (192.168.0.162) 2026-10-23T07:28:11Z
agent: felhom-agent 0.163.0
bundle: 0.163.0 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 2026-10-23T07:24:43Z
2 BUNDLE DONE agent=0.163.0 written=1 same=30 self-check=ok
2 BUNDLE DONE agent=0.163.0-step1 written=1 same=29 self-check=ok
guest 9201: memory: 12288 swap: 12288
drop-in: -rw-r--r-- 1 root root 690 Oct 23 09:24 /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf
drop-in content: PermitRootLogin prohibit-password
sshd -T: permitrootlogin without-password
root password set (passwd -S): P
felhom-sshd: active; its root policy: permitrootlogin without-password
felhom-sshd listens: 0.0.0.0:8822 [::]:8822 config test: ok; firewall admits only: elements = { 10.77.0.250 }
main sshd reloaded after the drop-in: Oct 23 09:24:42 demo-felhom sshd[1011]: Received SIGHUP; restarting. Oct 23 09:24:42 demo-felhom systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server
pveproxy: active
main sshd banner from DooPlex (:22): SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4
key login: ok
root password attempt rc=255 (non-zero = refused)
## notes
- felhom-sshd :8822 admits only the operator peer 10.77.0.250 over wg-felhom (H1); a connection test from DooPlex times out BY DESIGN — proven: active, listening, sshd -t ok, firewall set.
- a dummy root password is logged as 'Failed password' (OpenSSH refuses a root password silently under prohibit-password); the policy proof is sshd -T + the SIGHUP reload after the drop-in. The real root password was not tried (CC does not hold it).
- guests 9202, 9401 (demo-hp) and the transient 990000 (Tester 1, gone a minute later) are outside the felhom pool; swap = RAM applies to pool guests (9201 on all three: swap = memory).
@@ -0,0 +1,53 @@
# Part 6 — D33: the released images and agent packages to ep0 nightly (operator yes 2026-10-23), 2026-10-23T07:51:56Z
## ep0 — free space first, then ONE user and ONE directory (nothing else on ep0 changed)
before: /dev/sda1 38G 23G used 14G free 63% (/mnt/pbs-datastore is the PBS datastore root — not used)
created: user felhom-relcopy (system, uid 999, password locked 'L', shell /bin/sh for the forced command), /srv/felhom-release-copy (0750) with data/ and .ssh/authorized_keys:
command="/usr/bin/rrsync /srv/felhom-release-copy/data",restrict ssh-ed25519 <the job key, felhom-release-copy@dooplex>
fence test: rsync --list-only OK; a shell command → 'rrsync error: SSH_ORIGINAL_COMMAND does not run rsync'; '..' → 'do not use .. in arg'
after the first copy: /dev/sda1 38G 24G used 13G free 65%; /srv/felhom-release-copy/data 677M
cap: CAP_MB=3072, enforced by the sender (ep0's root fs has no quota; a quota would change more than one directory)
## DooPlex — the job (felhom.eu d306b8c2): /usr/local/sbin/felhom-release-copy + .service + .timer (01:40), /etc/felhom-release-copy/{env,id_ed25519,known_hosts,registry-token}
registry token: a NEW Gitea access token for user admin, name felhom-release-copy-d33, scope read:package ONLY — minted inside the gitea pod straight into /etc/felhom-release-copy/registry-token (0600 root); never printed
known_hosts: ep0's ed25519 host key, SHA256:NGtcS5v3K+9UjMTM6X5Ib0CoChdhqj9bezLz1hOUqgY — equal to the key already pinned for root@ep0
skopeo, not docker save: skopeo dir: keeps the registry's manifest bytes and compressed blobs, so a push back gives the SAME digest (docker save re-compresses → a new digest); no Docker daemon needed
## the first run (systemctl start, under the unit's sandbox)
2026-10-23T09:48:33+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-controller: keeping 0.338.0, 0.339.0, 0.340.0
2026-10-23T09:48:37+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-hub: keeping 0.164.0, 0.165.0, 0.166.0
2026-10-23T09:48:43+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-samba: keeping 1.0.0, 1.1.0, 1.2.0
2026-10-23T09:48:47+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-agent: keeping 0.159.2, 0.160.0, 0.161.0, 0.162.0, 0.163.0
2026-10-23T09:48:51+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: staged and verified: 676 MB, new this run: felhom-controller:0.338.0, felhom-controller:0.339.0, felhom-controller:0.340.0, felhom-hub:0.164.0, felhom-hub:0.165.0, felhom-hub:0.166.0, felhom-samba:1.0.0, felhom-samba:1.1.0, felhom-samba:1.2.0
2026-10-23T09:49:51+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: pushed to ep0 and compared by checksum: no difference (91 s)
2026-10-23T09:49:51+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: success signal written
2026-10-23T09:49:51+02:00 dooplex systemd[1]: felhom-release-copy.service: Consumed 30.416s CPU time, 743.1M memory peak.
## read back from ep0 (rsync pull with the job key → /var/lib/felhom-release-copy/readback)
sha256sum -c SHA256SUMS: every file OK (141 files), 677 MB
every image re-verified (top manifest = its digest, every blob = its name):
felhom-controller 0.338.0 sha256:e8a529f395bfe8c0e7b87d5d98929a6f8e9834aa13c0ab77bd214ee9e7e53433
felhom-controller 0.339.0 sha256:fee71aafceb4a7d5f24ae4a6a273a8d51eb26e0550fe9d12d4d86955f6a343a8
felhom-controller 0.340.0 sha256:a8d1c6458da4ea9d667f928498656a291771e0b3756b20f2fcc8c1cf9d516621
felhom-hub 0.164.0 sha256:931594e333a4e8a9784eff2fb144d022ab46f89dd6a865a93e6100dc28dbd9dc
felhom-hub 0.165.0 sha256:c6180fb79a2ff7d5546a704c68b634b9207afb0a0ecd524c74b0310434532537
felhom-hub 0.166.0 sha256:829667bb289c790bc99719e88dad8f7d08faaa9f662df1e9a06b1774915a1af6
felhom-samba 1.0.0 sha256:0fa9fae3eaa9d48836c855a4884f0751a90ff050dc9e7d17e41d93bb434499c0
felhom-samba 1.1.0 sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10
felhom-samba 1.2.0 sha256:da96f25033827fbb77d0ee398e86fe5b744e2f29be9de898994533340c97acf9
agent 0.159.2 binary 4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9 bundle 477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9
agent 0.160.0 binary 7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1 bundle 6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f
agent 0.161.0 binary ff4e6dc52dd70234913445ecdeb88d31dea8e394d77c48704d26ff19b999d3b5 bundle 485f36c957d83f772aa22cac24e5e73f44da5cf8b957b1fd513bec2bc1d12c47
agent 0.162.0 binary 5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928 bundle ac772967fd4669045e837a70bf5acca239c58d55f7f00212e472a8facd60b5a5
agent 0.163.0 binary 561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c bundle 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8
independent controls (other channels): controller 0.340.0 = the release digest from build.sh (03-release-0340-boxes.txt); hub 0.165.0 = the running hub pod's imageID; hub 0.164.0 = hub/CHANGELOG; agent 0.163.0 = release-agent.sh today; agent 0.161.0 = night2-2026-10-20/P1/1b; bundle 0.160.0 = the three boxes' installed bundle_sha256
## load on a scratch machine (9202 on demo-hp, Tier 0)
the ep0 copy of felhom-controller 0.340.0 → skopeo docker-archive (amd64) → pct push 9202 → docker load:
Loaded image: felhom-restorecopy-test:0.340.0; ID=sha256:3f34d74519463c338422594b70c31d9c5beabed48847134c6d26b5a95ebbae2b = the copy's amd64 config digest; a container ran (ls /)
teardown: image untagged (0 left), the tar removed on 9202, on demo-hp /tmp and on DooPlex
timer: Sat 2026-10-24 01:42:16 CEST 15h - - felhom-release-copy.timer felhom-release-copy.service
## alarm (homelab-manifests 1bbef892): FelhomReleaseCopyStale (> 26 h or absent, warning) — promtool check: 7 rules OK; promtool test: quiet at 25 h, fires at 27 h
synced ConfigMap mon-system/prometheus-rules only; POST /-/reload 200; /api/v1/rules: FelhomReleaseCopyStale inactive ok; metric age at read ~3235 s
@@ -0,0 +1,10 @@
no guest references the three
BEFORE: /dev/mapper/pve-root 39G 29G 8.5G 77% /
Removed volume 'local:iso/felhom-installer-1.27.1-pve9.2-1.iso'
removed local:iso/felhom-installer-1.27.1-pve9.2-1.iso
Removed volume 'local:iso/felhom-installer-1.28.0-pve9.2-1.iso'
removed local:iso/felhom-installer-1.28.0-pve9.2-1.iso
Removed volume 'local:iso/felhom-installer-1.29.0-pve9.2-1.iso'
removed local:iso/felhom-installer-1.29.0-pve9.2-1.iso
AFTER: /dev/mapper/pve-root 39G 24G 14G 64% /
local:iso/felhom-installer-1.30.0-pve9.2-1.iso iso iso 1705324544
@@ -0,0 +1,16 @@
# The vouch read-back — agent 0.163.0 (the operator vouched on the hub, answered 'Vouched now'), read 2026-10-23T08:00:42Z
## channel 1: hub DB (hub.db + -wal + -shm in ONE tar from the pod), hub_settings
artifact_agent_sha256|561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c|2026-10-23 07:58:46
artifact_agent_version|0.163.0|2026-10-23 07:58:46
artifact_bundle_sha256|3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8|2026-10-23 07:58:46
artifact_golden_sha256|3881de62b0986ccbe5e55580e3f1bb9765d65b8d898f67c6339bac105244f84f|2026-10-23 07:58:46
artifact_golden_version|0.331.0|2026-10-23 07:58:46
artifact_min_agent|0.158.1|2026-10-23 07:58:46
artifact_wrapper_sha256||2026-10-23 07:58:46
## channel 2: the running hub's operator page GET /configuration (CC's token), the selected options
value="0.163.0" data-sha="561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c" selected>0.163.0
value="0.331.0" data-sha="3881de62b0986ccbe5e55580e3f1bb9765d65b8d898f67c6339bac105244f84f" selected>0.331.0
agent 0.163.0 sha256 = the release (561b5316…); bundle sha256 = the release (3afb88d9…). Golden 0.331.0 unchanged. The floor's MinAgent stays 0.158.1 (artifact_min_agent 0.158.1; controller 0.340.0 declares 0.158.1).
@@ -0,0 +1,13 @@
# Installer 1.36.0 published (operator yes 2026-10-23), read 2026-10-23T08:16:46Z
1. bump: felhom.eu 83388c08 (SCRIPT_VERSION 1.36.0; scripts/test_hostinstall.py 40 passed; CI 2247 success)
2. tag: installer-v1.36.0 (annotated d00f868c) -> 83388c08, pushed
3. pins: manifests/webpage.yaml both --ref=installer-v1.36.0 (sidecar git-sync-scripts + init git-sync-scripts-init), commit 2aebf1bd;
ArgoCD: only Deployment felhom-system/felhom-webpage was OutOfSync; synced that ONE resource at revision 2aebf1bd; rollout ok;
live args: git-sync-scripts-init ['--ref=installer-v1.36.0'], git-sync-scripts ['--ref=installer-v1.36.0'], website git-sync ['--ref=main'] (unchanged)
4. read back: curl https://felhom.eu/scripts/felhom-host-install.sh
SCRIPT_VERSION="1.36.0" # the SINGLE version source (F-1): -h and the run banners follow it.
sha256 served = de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec
sha256 tag file = de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec
the drop-in name 10-felhom-root-key-only appears 2 times (the uninstall's removal)
Rollback: move the tag back to installer-v1.35.0's commit and wait ~30 s (no sync).
@@ -0,0 +1,4 @@
# Website publish read-back (felhom.eu 0bb57a4f, operator yes 2026-10-23)
Method: headless Chromium (playwright 1.61.1 image) on the LIVE site; the FAQ answer opened by a click; 390x844 (phone) and 1366x900 (computer).
No page scrolls sideways at either size (scrollWidth == viewport on all 10 shots).
Text: see text-check.txt (new sentences present; the old 'only encrypted (the remote backup)' gone; control: HEAD~1 had it twice).
Binary file not shown.

After

Width:  |  Height:  |  Size: 144 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 122 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 113 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 122 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 124 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 55 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 55 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 59 KiB

@@ -0,0 +1,7 @@
waited ~0 s
NEW letoltes 'az alapokat futtatja' count 1 (want 2 on the FAQ: visible + JSON-LD; 1 on download)
NEW en/download 'runs the basics; the photo app needs 16 GB' count 1 (want 2 on the FAQ: visible + JSON-LD; 1 on download)
NEW gyik 'a Cloudflare h' count 4 (want 2 on the FAQ: visible + JSON-LD; 1 on download)
NEW en/faq "passes through Cloudflare's network and is readable there" count 2 (want 2 on the FAQ: visible + JSON-LD; 1 on download)
OLD gyik 'otthonodat (a t' live 0 (want 0); control HEAD~1 2 (want 2)
OLD en/faq 'leaves your home only encrypted (the remote backup)' live 0 (want 0); control HEAD~1 2 (want 2)
@@ -0,0 +1,43 @@
# Part 5 — D34: the hub DB off-site every 6 hours (operator yes 2026-10-23), 2026-10-23T08:41:15Z
## what changed
- hub 0.167.0 (deployed 08:27Z, image sha256:177860a9…): snapshots at 02:00 08:00 14:00 20:00 Budapest; start-up catch-up when the newest > 6 h. Hub log at start:
2026/10/23 10:28:12 [INFO] db-snapshot 08:00: next run at 2026-10-24 08:00 CEST (in 21h31m48s)
2026/10/23 10:28:12 [INFO] db-snapshot 14:00: next run at 2026-10-23 14:00 CEST (in 3h31m48s)
2026/10/23 10:28:12 [INFO] db-snapshot 20:00: next run at 2026-10-23 20:00 CEST (in 9h31m48s)
2026/10/23 10:28:12 [INFO] db-snapshot 02:00: next run at 2026-10-24 02:00 CEST (in 15h31m48s)
2026/10/23 10:30:06 [INFO] db snapshot written: hub-20261023T082812Z.db (525996032 bytes, 1m54.537s); pruned 1, keeping 2
- DooPlex felhom-hub-db-backup (felhom.eu 52af38df): timer 02:30 08:30 14:30 20:30; waits ≤ 20 min for a snapshot being written; refuses a newest snapshot older than 7 h
- R-1234 (closed): the first version (2 h, no wait, be7e97dc) refused the timer's own catch-up at 10:28 CEST and mailed admin@ once — fixed in 52af38df
## retention (ep0, server-side, unchanged): prune-job prune-operator-hubdb, ns operator, 03:45, keep-daily 14, keep-weekly 8
=> the current day's copies (up to 4) stay until the next 03:45 prune, then the newest of each day for 14 days and one per week for 8 weeks. Size per copy: 501.6 MiB raw, 24.5 MiB uploaded (compressed, dedup'd chunks).
## run once by hand (systemctl start felhom-hub-db-backup.service, 10:39 CEST)
2026-10-23T10:39:33+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: snapshot hub-20261023T082812Z.db, 11 min old
2026-10-23T10:39:45+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: checked: 525996032 bytes, integrity ok, 4 host(s)
2026-10-23T10:39:48+02:00 dooplex felhom-hub-db-backup[2377041]: hubdb.pxar: had to backup 501.629 MiB of 501.629 MiB (compressed 24.471 MiB) in 2.47 s (average 203.437 MiB/s)
2026-10-23T10:39:49+02:00 dooplex felhom-hub-db-backup[2377041]: Duration: 3.21s
2026-10-23T10:39:49+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: pushed hub-20261023T082812Z.db to ep0 (ns operator) in 4 s
2026-10-23T10:39:49+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: success signal written
## read back: restore the NEWEST copy from ep0 with the READ-ONLY token into a scratch dir (shredded after)
15 copies on ep0, oldest 2026-10-10T00:30:56Z
newest copy on ep0: host/dooplex-hub/2026-10-23T08:39:45Z
Using encryption key from '/etc/felhom-hub-backup/enc.key'..
Fingerprint: b2:19:bf:36:3b:97:3d:6c
restore complete (501.629 MiB processed in 5.4s, average 92.519 MiB/s)
restored size: 525996032 bytes
restored sha256: a9ff4c8a0d0fe34da72411d9c165e573069f5601492d1e625578464f56adbde7
integrity_check: ok
SELECT COUNT(*) FROM hosts: 4
SELECT value FROM hub_settings WHERE key='artifact_agent_version': 0.163.0
pod snapshot hub-20261023T082812Z.db: 525996032 bytes, sha256 a9ff4c8a0d0fe34da72411d9c165e573069f5601492d1e625578464f56adbde7 (kubectl exec sha256sum — a different channel from the restore)
=> byte-identical; it holds today's vouch (artifact_agent_version 0.163.0, set 07:58Z) — a nightly copy could not
## timers
Fri 2026-10-23 14:30:08 CEST 3h 48min Fri 2026-10-23 10:28:30 CEST 12min ago felhom-hub-db-backup.timer felhom-hub-db-backup.service
## alarm (homelab-manifests 1bbef892): HubDBBackupStale 26 h -> 13 h (two missed 6-hourly pushes)
promtool test: quiet at 12 h, fires at 14 h; RED-PROOF: the rule at 26 h fails the 14 h case
in force: /api/v1/rules shows the 13 * 3600 expression, inactive ok; metric age at read ~237 s
@@ -0,0 +1,21 @@
# Part 3 — hub 0.167.0 deploy and the 30-minute watch (operator yes 2026-10-23), read 2026-10-23T08:56:31Z
build: clean clone at d306b8c2 (hub/ unchanged since be7e97dc, CI 2250 success; d306b8c2 CI 2251 success before the deploy); hub tests rc 0 (28 packages)
image: gitea.dooplex.hu/admin/felhom-hub:0.167.0 sha256:177860a996cb285d935777d6ffdf85725f6cf4489885f54fd3bf8135c68041a0
manifest 34979d5d; ArgoCD: only Deployment felhom-system/hub OutOfSync; synced that one resource; rollout ok 08:27:34Z; pod imageID = the digest above; Synced/Healthy
## watch 08:27:34Z → 08:56:31Z — channel 1: the hub DB (with WAL) + log
events since the deploy: 0 (newest event 07:07:10Z Tester4 host_down — the expected dead test box; 161 events since 2026-10-22: the table is live)
host report|demo-felhom-8363b5|2026-10-23 08:54:54|2
host report|demo-hp-bb76ea|2026-10-23 08:49:41|2
host report|tester-1-d70be4|2026-10-23 08:42:35|1
controller report|demo-felhom|2026-10-23 08:53:20|3
controller report|demo-hp|2026-10-23 08:53:24|3
controller report|tester-1|2026-10-23 08:53:23|3
hub log since deploy: 0 [ERROR]; 'osupdates: alarm check done … 0 alarm(s) sent'; 'Host staleness checker initialized: 3 ok, 1 already alarmed (seeded silently, R-1001)' — Tester4 not re-mailed
## channel 2: the operator mailbox (Gmail connector, to:admin@felhom.eu newer_than:1d)
after 08:27Z: ONE mail, 08:28:32Z '[DooPlex backup] FAILED: felhom-hub-db-backup.service' — NOT from the hub: CC's install of the 6-hourly timer (R-1234, closed, fixed 52af38df). No hub mail after the deploy.
before the deploy (control that the channel delivers): 07:07:11Z Tester4 host_down, 07:04:11Z node_down, 06:22Z/06:19Z stale — the hub's mails do arrive
verdict: no new false alarm from the hub in the 30 minutes after the deploy; one CC-caused DooPlex mail, filed and fixed
@@ -0,0 +1,81 @@
# Plex to the fixed version on the demo/test boxes — 2026-10-23 (Plex2Shell)
Operator ruling 2026-10-23 10:56 (recorded in `CONTEXT.md`): move the catalog's Plex to the newest fixed version at once,
without the test record of `09` §3 decision 13, and put the demo boxes on it. DooPlex and ep0 were not touched; the
incident session handles DooPlex.
## The catalog move
- Catalog `6cc317d`: `plexinc/pms-docker:1.41.4.9463-630c9f557` → `1.43.5.11029-6bf8af3f0` (Docker Hub, pushed
2026-10-20; manifest list `sha256:01bdf0ad6d53198da4331540c1e73ab65db7a72e961fff97ecb0847d46055c5f`, re-resolved at
start — still the newest plain `X.Y.Z.B-hash` tag; `public`/`beta`/`plexpass` moved on 2026-10-22 but carry no plain
tag and are excluded). `catalog_since: "2026-10-23"`.
- Gates green: image-pins, catalog-since (range), test-record, image-resolvable (plex), `catalog_gates.py --fast plex`.
`check-test-record-move.py --range ce33e3b..6cc317d` refused, as expected (verbatim in `app-catalog-felhom.eu/REPORT.md`).
Pushed with `--no-verify`, that commit only.
- Read back from the box's catalog cache on demo-hp 9201 after one sync: the new tag, `catalog_since: "2026-10-23"`,
HEAD `6cc317d`. The badge went to „Frissítés elérhető — ma" / „Update available — today" in 4.7 s.
## Which boxes run Plex
Hub `/hosts` (CC token, read only): four hosts — Tester4 (DOWN), demo-felhom, demo-hp, tester-1. The customer
**Tester 2 has 0 hosts**, so the operator's question was not needed. The hub's `/apps` page counts 2 Plex deployments;
the boxes say ONE runs it (a second channel: `docker ps -a` in every running guest):
| Box | Guest | Plex |
|---|---|---|
| demo-hp | 9201 | yes, 1.41.4.9463-630c9f557 |
| demo-hp | 9202 (scratch) | no |
| demo-felhom (N100) | 9201 | no |
| tester-1 (VM 341 on demo-hp) | 9201 | no |
| Tester4 | — | the hub host is DOWN and its VM no longer exists on demo-hp or felhom-pve (`qm list`); nothing to update. The hub's second Plex deployment is most likely this stale host's last report (not proven). |
## demo-hp 9201 — the signs check (before the update, read only) → **NOT-A-FINDING**
Scripts: `demo-hp-9201/signs.sh`, `demo-hp-9201/signs2.sh` (run inside the guest). Outputs, e-mail and token redacted:
`demo-hp-9201/signs-9201-before-update.txt`, `demo-hp-9201/signs2-9201-before-update.txt`.
| Sign | Result | Control |
|---|---|---|
| CPU, last days | whole guest 0.23–0.34 cores average of 7, max 1.04, per day 10-16 → 10-23 (PVE rrd, the HOST's channel). A miner would hold several cores flat. | — |
| Processes in the container | only s6, Plex Media Server, Plex Script Host, Plex Tuner Service | — |
| `xmrig`, `mylovelyminer`, `x12345`, `myawesomeproxy` anywhere in the guest | none | — |
| ELF files under `Codecs` | none (one file, `.device-id`; one empty codec dir from 2026-10-15) | the ELF test says ELF for `/usr/bin/docker`, not for `Preferences.xml` |
| ELF > 10 kB in the config volume newer than 2026-10-15 | none | same |
| Container layer (`docker diff plex`) | 80 entries, all s6 runtime under `/run` | — |
| `TranscoderH264Options` / `…Override` in `Preferences.xml` | absent | the attribute list printed (27 names) |
| Log lines `searchOne`, `/video/:/transcode/universal`, `?x=http…` | 0, 0, 0 | 633 `Request: [` lines in the same log |
| Open connections from the container | plex.tv / Cloudflare 443 and the box's traefik only | — |
**Limit, stated:** Plex keeps 6 rotated logs and restarted several times in the night, so the logs reach back only to
2026-10-23 00:32 UTC — not to 2026-10-16. The 10-16 → 10-22 window is covered by the CPU history and the files only,
not by the logs. The container was recreated at 04:39 UTC, so its layer has no history before that either.
## The update (the product's Update press, `POST /api/stacks/plex/update`)
`demo-hp-9201/update-press.json`: accepted 202; phases safety-dump → pulling → copying → verifying → done in 28.4 s;
no error, no hold. After (`demo-hp-9201/after-update.txt`):
- `/identity`: version `1.43.5.11029-6bf8af3f0`, `claimed: true`, the SAME `machineIdentifier` as before.
- Library: 0 sections before and after (this Plex never had a library; `demo-hp-9201/library-before-update.txt`). The
server's identity and claim are what carried over.
- Box walk: every container healthy (traefik has no health check, up), none restarting; 0 `[ERROR]` and 0 `[WARN]` lines
in the controller log since the press (control: 99 `[INFO]` lines in the same window); badge „Naprakész" / „Up to date";
pinned = installed = catalog = live compose = the running container, all the new tag.
## The guide (Part 3)
The guide's labels were read from the new image's own web client (`WebClient.bundle/.../translations/hu.json`, plex-web
4.162.1; its keys are the English labels): Home/„Kezdőképernyő", Play/„Lejátszás", Scan Library Files/„Könyvtárfájlok
beolvasása", More/„Több", Fix Match/„Találatok pontosítása", Settings/„Beállítások", Manage Library Access/„Könyvtár
hozzáférés kezelése", and the FAQ's Video Quality/„Videó minőség", Original/„Eredeti" — all present, all equal
(`demo-hp-9201/guide-labels-1.43.5.txt`; ASCII-fragment controls: `beolvas` 5, `kezel` 26, `xqzzyw` 0). The box's own
Plex wiring (`POST /api/storage/media/wire`, the household's „connect now") answered `plex: unchanged`, no error, against
the new Plex (`demo-hp-9201/guide-wiring-after-update.txt`). **Not exercised:** the wiring's PUT that adds a new
drive's folder — this box has one drive and no library, so it had nothing to add. `guide.checked_version` moved.
## Teardown
- Machine: Plex stays on the new version (that is the goal). The helper scripts were removed from the guest and host.
- Host: nothing provisioned.
- Hub: nothing changed (reads only).
@@ -0,0 +1,57 @@
identity: {"MediaContainer":{"size":0,"apiVersion":"1.2.3","claimed":true,"machineIdentifier":"fb56097105b927dd42576cd879499a50c064c9fc","version":"1.43.5.11029-6bf8af3f0"}}
sections: 0
---containers
adventurelog Up 5 hours (healthy)
adventurelog-frontend Up 5 hours (healthy)
adventurelog-postgres Up 5 hours (healthy)
bentopdf Up 5 hours (healthy)
bookstack Up 5 hours (healthy)
bookstack-db Up 5 hours (healthy)
calibre-web Up 5 hours (healthy)
cloudflared Up 2 days (healthy)
docmost Up 4 hours (healthy)
docmost-postgres Up 5 hours (healthy)
docmost-redis Up 5 hours (healthy)
felhom-controller Up 3 hours (healthy)
filebrowser Up 2 days (healthy)
home-assistant Up 4 hours (healthy)
opengist Up 4 hours (healthy)
paperless-postgres Up 4 hours (healthy)
paperless-redis Up 4 hours (healthy)
paperless-webserver Up 4 hours (healthy)
plex Up 48 seconds (healthy)
privatebin Up 4 hours (healthy)
romm Up 4 hours (healthy)
romm-db Up 4 hours (healthy)
romm-redis Up 4 hours (healthy)
traefik Up 30 hours
---not-healthy
---restarting
---ctl-errors-since-update
0
---ctl-plex-lines
2026/10/23 09:08:22 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=true composePath=/opt/docker/stacks/plex/docker-compose.yml
2026/10/23 09:08:22 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping plex — last check 4s ago, effective interval 5m0s, healthy=true
2026/10/23 09:08:22 backup.go:526: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/hdd_1 → [calibre-web, paperless-ngx, plex, romm]
2026/10/23 09:08:23 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: e94565919cd7 plex plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0
f8900ef8917f felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.340.0
2026/10/23 09:08:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container plex (image=plexinc/pms-docker:1.43.5.11029-6bf8af3f0, not a database)
2026/10/23 09:08:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.340.0, not a database)
2026/10/23 09:08:23 sync.go:559: [DEBUG] [sync] plex/docker-compose.yml: hash match, skipped
2026/10/23 09:08:23 sync.go:559: [DEBUG] [sync] plex/.felhom.yml: hash match, skipped
2026/10/23 09:08:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/plex/docker-compose.yml
2026/10/23 09:08:23 recovery_unit.go:248: [INFO] [backup] plex: the app now runs [plexinc/pms-docker:1.43.5.11029-6bf8af3f0]; the unit keeps the definition of its data ([plexinc/pms-docker:1.41.4.9463-630c9f557], written 2026-10-23T02:16:55Z) until the next backup replaces the data
2026/10/23 09:08:23 recovery_unit.go:289: [INFO] [backup] Recovery unit captured for plex → /mnt/felhom-drives/hdd_1/backups/primary/plex (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/10/23 09:08:24 auth.go:144: [DEBUG] [web] auth: valid session for GET /apps/plex
2026/10/23 09:08:24 server.go:637: [DEBUG] [web] ServeHTTP: GET /apps/plex from 172.18.0.10:45638
2026/10/23 09:08:24 handlers.go:421: [DEBUG] [web] deployPageData: stack=plex method=GET
2026/10/23 09:08:24 auth.go:144: [DEBUG] [web] auth: valid session for GET /apps/plex
2026/10/23 09:08:24 server.go:637: [DEBUG] [web] ServeHTTP: GET /apps/plex from 172.18.0.10:45638
2026/10/23 09:08:24 handlers.go:421: [DEBUG] [web] deployPageData: stack=plex method=GET
2026/10/23 09:08:24 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/stacks/plex
2026/10/23 09:08:24 router.go:78: [DEBUG] [api] GET /api/stacks/plex (path=/stacks/plex)
2026/10/23 09:08:25 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned plex: errors=0 warnings=0 issues=0 (took 28ms)
2026/10/23 09:08:32 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping plex — last check 13s ago, effective interval 5m0s, healthy=true
2026/10/23 09:08:42 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping plex — last check 23s ago, effective interval 5m0s, healthy=true
2026/10/23 09:08:52 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping plex — last check 33s ago, effective interval 5m0s, healthy=true
2026/10/23 09:09:02 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping plex — last check 43s ago, effective interval 5m0s, healthy=true
@@ -0,0 +1,10 @@
Plex web client translations/hu.json from plexinc/pms-docker:1.43.5.11029-6bf8af3f0 (WebClient plex-4.162.1), keys = the English UI string
EN key 'Home': present=True; HU in Plex='Kezdőképernyő'; guide HU='Kezdőképernyő'; match=True
EN key 'Play': present=True; HU in Plex='Lejátszás'; guide HU='Lejátszás'; match=True
EN key 'Scan Library Files': present=True; HU in Plex='Könyvtárfájlok beolvasása'; guide HU='Könyvtárfájlok beolvasása'; match=True
EN key 'More': present=True; HU in Plex='Több'; guide HU='Több'; match=True
EN key 'Fix Match': present=True; HU in Plex='Találatok pontosítása'; guide HU='Találatok pontosítása'; match=True
EN key 'Settings': present=True; HU in Plex='Beállítások'; guide HU='Beállítások'; match=True
EN key 'Manage Library Access': present=True; HU in Plex='Könyvtár hozzáférés kezelése'; guide HU='Könyvtár hozzáférés kezelése'; match=True
EN key 'Video Quality': present=True; HU in Plex='Videó minőség'; guide HU='Videó minőség'; match=True
EN key 'Original': present=True; HU in Plex='Eredeti'; guide HU='Eredeti'; match=True
@@ -0,0 +1,2 @@
status 200 {"data": {"pending": false, "drives": null, "apps": ["plex"], "wiring": [{"app": "folders", "state": "done"}, {"app": "plex", "state": "unchanged"}], "wired_at": "2026-10-23T09:10:30Z"}, "ok": true}
wire 200 {"data": [{"app": "folders", "state": "done"}, {"app": "plex", "state": "unchanged"}], "ok": true}
@@ -0,0 +1,2 @@
identity: {"MediaContainer":{"size":0,"apiVersion":"0.1.0","claimed":true,"machineIdentifier":"fb56097105b927dd42576cd879499a50c064c9fc","version":"1.41.4.9463-630c9f557"}}
sections: 0
@@ -0,0 +1,63 @@
### 1. processes in the plex container (docker top)
PID USER ELAPSED TIME %CPU COMMAND
1493203 root 04:25:08 00:00:00 0.0 s6-svscan
1493262 root 04:25:08 00:00:00 0.0 s6-supervise
1493541 root 04:25:08 00:00:00 0.0 s6-supervise
1493544 1000 04:25:08 00:00:21 0.1 Plex Media Serv
1493588 1000 04:25:08 00:00:15 0.0 Plex Script Hos
1494146 1000 04:25:05 00:00:12 0.0 Plex Tuner Serv
### 1b. suspect names anywhere in the guest
none
### 1c. top CPU processes in the guest
PID USER %CPU ELAPSED COMMAND
235 root 1.7 2-06:15:22 dockerd
183 root 0.8 2-06:15:23 containerd
1493858 999 0.7 04:25:07 redis-server
1489649 999 0.6 04:25:33 redis-server
1492193 999 0.6 04:25:19 redis-server
1717354 root 0.3 02:26:14 felhom-controll
1496145 root 0.3 04:24:23 gunicorn
1698 65532 0.3 2-06:15:16 cloudflared
1496141 root 0.3 04:24:23 gunicorn
1492308 1000 0.2 04:25:19 node-MainThread
1495252 1000 0.2 04:24:46 [celeryd: celer
### 2. ELF files under Codecs (path size mtime sha256)
all files under Codecs: 1
total 16
drwxr-xr-x 3 1000 1000 4096 2026-10-15 08:46:05.701113471 +0000 .
drwxr-xr-x 11 1000 1000 4096 2026-10-23 08:39:53.737639828 +0000 ..
-rw-r--r-- 1 1000 1000 36 2026-10-15 08:46:05.701113471 +0000 .device-id
drwxr-xr-x 2 1000 1000 4096 2026-10-15 08:46:05.700807513 +0000 e613bce-97f23d579c1001d8e9cc0d2e-linux-x86_64
### 2b. any ELF in the config volume outside Plex's own folders, newer than 2026-10-15
(end 2b)
### 3. Preferences.xml transcoder attributes
TranscoderH264Options*: absent
attribute names present:
AcceptedEULA AnonymousMachineIdentifier CertificateUUID CertificateVersion DlnaEnabled FSEventLibraryPartialScanEnabled FSEventLibraryUpdatesEnabled GlobalMusicVideoPathMigrated LanguageInCloud LastAutomaticMappedPort MachineIdentifier MetricsEpoch OldestPreviousVersion PlexOnlineHome PlexOnlineMail PlexOnlineToken PlexOnlineUsername ProcessedMachineIdentifier PubSubServer PubSubServerPing PubSubServerRegion PublishServerOnPlexOnlineKey TranscoderTempDirectory customConnections encoding version
### 4. log lines: searchOne / transcode/universal (tokens redacted)
total 940
drwxr-xr-x 3 1000 1000 4096 2026-10-23 04:39:29.306642337 +0000 .
drwxr-xr-x 11 1000 1000 4096 2026-10-23 08:39:53.737639828 +0000 ..
drwxr-xr-x 2 1000 1000 4096 2026-10-23 04:39:30.774964086 +0000 PMS Plugin Logs
-rw-r--r-- 1 1000 1000 880 2026-10-23 04:38:23.878139505 +0000 Plex Crash Uploader.1.log
-rw-r--r-- 1 1000 1000 561 2026-10-23 03:34:34.438792292 +0000 Plex Crash Uploader.2.log
-rw-r--r-- 1 1000 1000 880 2026-10-23 03:33:23.603543601 +0000 Plex Crash Uploader.3.log
-rw-r--r-- 1 1000 1000 561 2026-10-23 02:59:31.142955018 +0000 Plex Crash Uploader.4.log
-rw-r--r-- 1 1000 1000 880 2026-10-23 02:58:23.855206791 +0000 Plex Crash Uploader.5.log
-rw-r--r-- 1 1000 1000 561 2026-10-23 04:39:28.559376452 +0000 Plex Crash Uploader.log
-rw-r--r-- 1 1000 1000 122590 2026-10-23 04:38:23.956828290 +0000 Plex Media Server.1.log
-rw-r--r-- 1 1000 1000 95344 2026-10-23 03:33:23.650544218 +0000 Plex Media Server.2.log
-rw-r--r-- 1 1000 1000 97012 2026-10-23 02:58:23.959207947 +0000 Plex Media Server.3.log
-rw-r--r-- 1 1000 1000 96861 2026-10-23 02:38:23.762270724 +0000 Plex Media Server.4.log
-rw-r--r-- 1 1000 1000 157007 2026-10-23 02:16:45.904202135 +0000 Plex Media Server.5.log
-rw-r--r-- 1 1000 1000 311157 2026-10-23 09:04:06.104368626 +0000 Plex Media Server.log
-rw-r--r-- 1 1000 1000 2769 2026-10-23 04:38:23.939140257 +0000 Plex Tuner Service.1.log
-rw-r--r-- 1 1000 1000 2769 2026-10-23 03:33:23.635544021 +0000 Plex Tuner Service.2.log
-rw-r--r-- 1 1000 1000 2769 2026-10-23 02:58:23.938207713 +0000 Plex Tuner Service.3.log
-rw-r--r-- 1 1000 1000 2769 2026-10-23 02:38:23.745270485 +0000 Plex Tuner Service.4.log
-rw-r--r-- 1 1000 1000 2769 2026-10-23 02:16:45.852201461 +0000 Plex Tuner Service.5.log
-rw-r--r-- 1 1000 1000 1855 2026-10-23 04:39:39.105066764 +0000 Plex Tuner Service.log
count searchOne: 0
count transcode/universal: 0
count lines with http(s):// in a query: 0
oldest log line: Oct 23, 2026 02:32:13.754 [1343875524390
@@ -0,0 +1,34 @@
#!/bin/sh
# Read-only Plex2Shell signs check, run INSIDE the guest. Prints no token.
V=/var/lib/docker/volumes/plex_plex_config/_data
P="$V/Library/Application Support/Plex Media Server"
echo "### 1. processes in the plex container (docker top)"
docker top plex -eo pid,user,etime,time,pcpu,comm
echo "### 1b. suspect names anywhere in the guest"
ps -eo pid,user,comm,args | grep -i -E 'xmrig|mylovelyminer|x12345|myawesomeproxy' | grep -v grep || echo "none"
echo "### 1c. top CPU processes in the guest"
ps -eo pid,user,pcpu,etime,comm --sort=-pcpu | head -12
echo "### 2. ELF files under Codecs (path size mtime sha256)"
if [ -d "$P/Codecs" ]; then
find "$P/Codecs" -type f | while read -r f; do
if head -c 4 "$f" | grep -q 'ELF'; then
printf '%s\t%s\t%s\t%s\n' "$f" "$(stat -c %s "$f")" "$(stat -c %y "$f")" "$(sha256sum "$f" | cut -c1-64)"
fi
done
echo "all files under Codecs: $(find "$P/Codecs" -type f | wc -l)"
ls -la --time-style=full-iso "$P/Codecs"
else echo "no Codecs dir"; fi
echo "### 2b. any ELF in the config volume outside Plex's own folders, newer than 2026-10-15"
find "$V" -type f -newermt 2026-10-15 -size +10k 2>/dev/null | while read -r f; do head -c 4 "$f" | grep -q ELF && printf '%s\t%s\t%s\n' "$f" "$(stat -c %s "$f")" "$(sha256sum "$f"|cut -c1-64)"; done; echo "(end 2b)"
echo "### 3. Preferences.xml transcoder attributes"
grep -o -E 'TranscoderH264Options(Override)?="[^"]*"' "$P/Preferences.xml" || echo "TranscoderH264Options*: absent"
echo "attribute names present:"; grep -o -E ' [A-Za-z]+="' "$P/Preferences.xml" | tr -d ' ="' | sort | tr '\n' ' '; echo
echo "### 4. log lines: searchOne / transcode/universal (tokens redacted)"
ls -la --time-style=full-iso "$P/Logs/" | head -30
for f in "$P/Logs/"Plex\ Media\ Server*.log; do
grep -h -E 'searchOne|/video/:/transcode/universal' "$f" 2>/dev/null
done | sed -E 's/(X-Plex-Token|PlexOnlineToken|token)=[^&" ]*/\1=REDACTED/gI' | head -200
echo "count searchOne: $(cat "$P/Logs/"Plex\ Media\ Server*.log 2>/dev/null | grep -c searchOne)"
echo "count transcode/universal: $(cat "$P/Logs/"Plex\ Media\ Server*.log 2>/dev/null | grep -c '/video/:/transcode/universal')"
echo "count lines with http(s):// in a query: $(cat "$P/Logs/"Plex\ Media\ Server*.log 2>/dev/null | grep -c -E '[?&][a-zA-Z]+=https?(%3A|:)')"
echo "oldest log line: $(head -1 "$P/Logs/Plex Media Server.5.log" 2>/dev/null | cut -c1-40)"
File diff suppressed because one or more lines are too long
@@ -0,0 +1,21 @@
#!/bin/sh
V=/var/lib/docker/volumes/plex_plex_config/_data
P="$V/Library/Application Support/Plex Media Server"
echo "### controls"
printf 'ELF detector on /usr/bin/docker (positive): '; head -c 4 /usr/bin/docker | grep -q ELF && echo ELF || echo not-ELF
printf 'ELF detector on Preferences.xml (negative): '; head -c 4 "$P/Preferences.xml" | grep -q ELF && echo ELF || echo not-ELF
echo "log grep positive control (lines with 'Request: [' in current log): $(grep -c 'Request: \[' "$P/Logs/Plex Media Server.log")"
echo "### docker diff plex (files changed in the container layer)"
docker diff plex | head -60; echo "total changes: $(docker diff plex | wc -l)"
echo "### created / started / restarts"
docker inspect plex --format 'created={{.Created}} started={{.State.StartedAt}} restarts={{.RestartCount}}'
echo "### transcode volume: files"
find /var/lib/docker/volumes/plex_plex_transcode/_data -type f | head; echo "(n=$(find /var/lib/docker/volumes/plex_plex_transcode/_data -type f | wc -l))"
echo "### crash uploader log (latest)"
sed -E 's/(token|Token)[=:" ]+[A-Za-z0-9_-]{8,}/\1=REDACTED/g' "$P/Logs/Plex Crash Uploader.1.log"
echo "### first lines of each PMS log (start time)"
for f in "$P/Logs/"Plex\ Media\ Server*.log; do echo "$(basename "$f"): $(head -1 "$f" | cut -c1-24)"; done
echo "### outbound connections now from plex netns"
PID=$(docker inspect -f '{{.State.Pid}}' plex); nsenter -t $PID -n ss -tnp 2>/dev/null | head -20
echo "### docker logs plex (tail, redacted)"
docker logs --tail 30 plex 2>&1 | sed -E 's/(claim-|token=)[A-Za-z0-9_-]+/\1REDACTED/g'
@@ -0,0 +1,78 @@
{
"update": {
"accepted": true,
"http": "202",
"phases": [
{
"t": 0.0,
"phase": "safety-dump",
"label": "Adatbázis pillanatkép…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 4.1,
"phase": "pulling",
"label": "Új verzió letöltése…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 10.1,
"phase": "copying",
"label": "Az adatok másolása a frissítés előtt…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 22.3,
"phase": "verifying",
"label": "Működés ellenőrzése…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 28.4,
"phase": "done",
"label": "Frissítve",
"updating": false,
"error": null,
"hold": null
}
],
"duration_s": 28.4,
"final_phase": "done",
"update_error": null,
"hold_reason": null,
"state": "running"
},
"observables": {
"pinned_images": {
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
},
"installed_images": {
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
},
"catalog_images": {
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
},
"live_compose_image_lines": [
"image: plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
],
"docker_inspect": [
"plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0 running=true restarts=0"
]
},
"log": [
"11:07:51 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}",
"11:07:51 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None",
"11:07:55 + 4.1s phase=pulling label=Új verzió letöltése… err=None hold=None",
"11:08:01 + 10.1s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None",
"11:08:14 + 22.3s phase=verifying label=Működés ellenőrzése… err=None hold=None",
"11:08:20 + 28.4s phase=done label=Frissítve err=None hold=None"
]
}
@@ -0,0 +1,19 @@
{
"app": "plex",
"walk": "security",
"box_walk_version": 1,
"verdict": "inconclusive",
"venue": "box 9202 (drill catalog), the guarded Update",
"from": {
"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"
},
"to": {
"plex": "busybox:1.36.1"
},
"why": "the install at FROM did not complete — nothing was tested",
"log": [
"12:26:15 [1] made the drive paths this app requires: ['/mnt/felhom-drives/scratch_hdd/userdata/plex']",
"12:26:15 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['PLEX_CLAIM', 'HDD_PATH']",
"12:26:15 [1] deploy -> 409 {'ok': False, 'error': 'Az alkalmazás adatainak helye egy meghajtó lehet, nem egy mappa a meghajtón belül. Válaszd ezt: "
]
}
@@ -0,0 +1,5 @@
12:26:15 [1] made the drive paths this app requires: ['/mnt/felhom-drives/scratch_hdd/userdata/plex']
12:26:15 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['PLEX_CLAIM', 'HDD_PATH']
12:26:15 [1] deploy -> 409 {'ok': False, 'error': 'Az alkalmazás adatainak helye egy meghajtó lehet, nem egy mappa a meghajtón belül. Válaszd ezt:
12:26:15 [9] verdict inconclusive -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/security-lane-2026-10-23/P2-negative/verdict.json
12:26:15 RESULT inconclusive — the install at FROM did not complete — nothing was tested
@@ -0,0 +1,188 @@
{
"app": "plex",
"walk": "security",
"box_walk_version": 1,
"verdict": "failed",
"venue": "box 9202 (drill catalog), the guarded Update",
"from": {
"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"
},
"to": {
"plex": "busybox:1.36.1"
},
"observables_before": {
"pinned_images": {
"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"
},
"installed_images": {
"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"
},
"catalog_images": null,
"live_compose_image_lines": [
"image: plexinc/pms-docker:1.43.4.10903-e5521bd8c"
],
"docker_inspect": [
"plex plexinc/pms-docker:1.43.4.10903-e5521bd8c running=true restarts=0"
]
},
"setup_gate_before": {
"browser": {
"status": "302",
"redirect": "https://felhom.enkisfelhom.hu/__gate/start?…"
},
"script": {
"status": "401",
"redirect": null
}
},
"seed": "none",
"seed_why": "the first-run claim needs a token minted at plex.tv by a real Plex account; no account exists for this venue",
"drill_commit": "577947198bff",
"sync_waited_s": 4.5,
"badge_before": {
"hu": [
{
"title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.",
"text": "Frissítés elérhető — ma"
}
],
"en": [
{
"title": "A newer version of this app is available. Select the Update button to start it.",
"text": "Update available — today"
}
]
},
"measured_at": "2026-10-23T10:27:24Z",
"update": {
"accepted": true,
"http": "202",
"phases": [
{
"t": 0.0,
"phase": "backing-up",
"label": "Biztonsági mentés készül a frissítés előtt…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 11.3,
"phase": "safety-dump",
"label": "Adatbázis pillanatkép…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 12.3,
"phase": "pulling",
"label": "Új verzió letöltése…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 14.4,
"phase": "copying",
"label": "Az adatok másolása a frissítés előtt…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 24.7,
"phase": "starting",
"label": "Indítás az új verzióval…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 25.7,
"phase": "verifying",
"label": "Működés ellenőrzése…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 327.2,
"phase": "undoing",
"label": "Visszaállítás az előző változatra…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 335.4,
"phase": "undone",
"label": "Visszaállítva az előző változatra",
"updating": false,
"error": null,
"hold": null
}
],
"duration_s": 335.4,
"final_phase": "undone",
"update_error": null,
"hold_reason": null,
"state": "running"
},
"observables_after": {
"pinned_images": {
"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"
},
"installed_images": {
"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"
},
"catalog_images": {
"plex": "busybox:1.36.1"
},
"live_compose_image_lines": [
"image: plexinc/pms-docker:1.43.4.10903-e5521bd8c"
],
"docker_inspect": [
"plex plexinc/pms-docker:1.43.4.10903-e5521bd8c running=true restarts=0"
]
},
"badge_after": {
"hu": [
{
"title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.",
"text": "Frissítés elérhető — ma"
}
],
"en": [
{
"title": "A newer version of this app is available. Select the Update button to start it.",
"text": "Update available — today"
}
]
},
"state_after": "running",
"hold_after": null,
"to_digests": {
"plex": "sha256:e0ab27395614a8e1a4fdf84c6bc60ac664915cfdde70c52d030c7728a1c48e14"
},
"why": "the update ended 'undone' (error None, hold None)",
"log": [
"12:26:46 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH', 'PLEX_CLAIM']",
"12:26:46 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}",
"12:27:06 [1] deployed, controller state=running, pinned={'plex': 'plexinc/pms-docker:1.43.4.10903-e5521bd8c'}",
"12:27:06 gate: plex is gated — passed as the household (cookie set)",
"12:27:10 [2] setup gate as installed: {'browser': {'status': '302', 'redirect': 'https://felhom.enkisfelhom.hu/__gate/start?…'}, 'script': {'status': '401', 'redirect': None}}",
"12:27:10 [3] seed: none — the first-run claim needs a token minted at plex.tv by a real Plex account; no account exists for this venue",
"12:27:11 [5] drill commit 577947198bff: plex plexinc/pms-docker:1.43.4.10903-e5521bd8c -> busybox:1.36.1 (push rc=0)",
"12:27:19 [5] the box's drill mirror now at 57794719",
"12:27:24 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}",
"12:27:24 + 0.0s phase=backing-up label=Biztonsági mentés készül a frissítés előtt… err=None hold=None",
"12:27:35 + 11.3s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None",
"12:27:37 + 12.3s phase=pulling label=Új verzió letöltése… err=None hold=None",
"12:27:39 + 14.4s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None",
"12:27:49 + 24.7s phase=starting label=Indítás az új verzióval… err=None hold=None",
"12:27:50 + 25.7s phase=verifying label=Működés ellenőrzése… err=None hold=None",
"12:32:51 + 327.2s phase=undoing label=Visszaállítás az előző változatra… err=None hold=None",
"12:33:00 + 335.4s phase=undone label=Visszaállítva az előző változatra err=None hold=None"
]
}
@@ -0,0 +1,22 @@
12:26:46 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH', 'PLEX_CLAIM']
12:26:46 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
12:27:06 [1] deployed, controller state=running, pinned={'plex': 'plexinc/pms-docker:1.43.4.10903-e5521bd8c'}
12:27:06 gate: plex is gated — passed as the household (cookie set)
12:27:10 [2] setup gate as installed: {'browser': {'status': '302', 'redirect': 'https://felhom.enkisfelhom.hu/__gate/start?…'}, 'script': {'status': '401', 'redirect': None}}
12:27:10 [3] seed: none — the first-run claim needs a token minted at plex.tv by a real Plex account; no account exists for this venue
12:27:11 [5] drill commit 577947198bff: plex plexinc/pms-docker:1.43.4.10903-e5521bd8c -> busybox:1.36.1 (push rc=0)
12:27:19 [5] the box's drill mirror now at 57794719
12:27:24 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}
12:27:24 + 0.0s phase=backing-up label=Biztonsági mentés készül a frissítés előtt… err=None hold=None
12:27:35 + 11.3s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None
12:27:37 + 12.3s phase=pulling label=Új verzió letöltése… err=None hold=None
12:27:39 + 14.4s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None
12:27:49 + 24.7s phase=starting label=Indítás az új verzióval… err=None hold=None
12:27:50 + 25.7s phase=verifying label=Működés ellenőrzése… err=None hold=None
12:32:51 + 327.2s phase=undoing label=Visszaállítás az előző változatra… err=None hold=None
12:33:00 + 335.4s phase=undone label=Visszaállítva az előző változatra err=None hold=None
12:33:18 [9] verdict failed -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/security-lane-2026-10-23/P2-negative/verdict.json
12:33:18 RESULT failed — the update ended 'undone' (error None, hold None)
12:33:27 [X] stop -> 200 {'ok': True, 'message': 'Stack plex stop completed'}
12:33:59 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'plex', 'volumes_removed': ['plex_plex_config', 'plex_plex_transcode'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'us
12:34:08 [X] after remove: deployed=False leftovers='/opt/docker/stacks/plex'
@@ -0,0 +1 @@
REFUSED plex: the box walk is 'security' with verdict 'failed' — only a PROVEN `box_walk.py --security` walk is written as a security-fix step
@@ -0,0 +1,9 @@
12:57:50 [5] drill commit f8b700e1d889: plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0 -> plexinc/pms-docker:1.43.4.10903-e5521bd8c (push rc=0)
12:57:58 [5] the box's drill mirror now at f8b700e1
f8b700e1d889
13:00:27 [sync] the badge NEVER caught up to plexinc/pms-docker:1.43.4.10903-e5521bd8c in 148.8s — catalog_images = None
13:00:27 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH', 'PLEX_CLAIM']
13:00:27 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
13:00:47 [1] deployed, controller state=running, pinned={'plex': 'plexinc/pms-docker:1.43.4.10903-e5521bd8c'}
13:00:47 gate: plex is gated — passed as the household (cookie set)
deployed True {"pinned_images": {"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"}, "installed_images": {"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"}, "catalog_images": null, "live_compose_image_lines": ["image: plexinc/pms-docker:1.43.4.10903-e5521bd8c"], "docker_inspect": ["plex plexinc/pms-docker:1.43.4.10903-e5521bd8c running=true restarts=0"]}
@@ -0,0 +1,15 @@
13:01:23 [5] the box's drill mirror now at f46e1bc6
before: {"pinned_images": {"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"}, "installed_images": {"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"}, "catalog_images": {"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"}, "live_compose_image_lines": ["image: plexinc/pms-docker:1.43.4.10903-e5521bd8c"], "docker_inspect": ["plex plexinc/pms-docker:1.43.4.10903-e5521bd8c running=true restarts=0"]}
badge before: {'hu': [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — ma'}], 'en': [{'title': 'A newer version of this app is available. Select the Update button to start it.', 'text': 'Update available — today'}]}
night chain -> 202 {'data': {'legs': ['db-dump', 'tier2', 'update-leg']}, 'message': 'started', 'ok': True}
the leg's own lines:
2026/10/23 11:01:34 night_chain.go:108: [INFO] [night-chain] manual run started from 172.18.0.6:46254: [db-dump tier2 update-leg]
2026/10/23 11:02:40 night_chain.go:77: [INFO] [night-chain] update-leg: started
2026/10/23 11:02:40 unattended.go:270: [INFO] [update-leg] started (manual-chain): window 02:30, no step starts at or after 16:17
2026/10/23 11:02:41 unattended.go:316: [INFO] [update-leg] plex: skipped — not_proven
2026/10/23 11:02:41 unattended.go:316: [INFO] [update-leg] privatebin: skipped — stopped_by_household
2026/10/23 11:02:41 unattended.go:248: [INFO] [update-leg] update leg (manual-chain): done=0 undone=0 held=0 failed=0 skipped=2 in 0s [skipped: plex=not_proven, privatebin=stopped_by_household]
2026/10/23 11:02:41 night_chain.go:82: [INFO] [night-chain] update-leg: done in 0s
after: {"pinned_images": {"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"}, "installed_images": {"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"}, "catalog_images": {"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"}, "live_compose_image_lines": ["image: plexinc/pms-docker:1.43.4.10903-e5521bd8c"], "docker_inspect": ["plex plexinc/pms-docker:1.43.4.10903-e5521bd8c running=true restarts=0"]}
state running hold None phase None
@@ -0,0 +1,11 @@
13:03:07 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}
13:03:07 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None
13:03:08 + 1.0s phase=pulling label=Új verzió letöltése… err=None hold=None
13:03:16 + 9.2s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None
13:03:26 + 19.5s phase=starting label=Indítás az új verzióval… err=None hold=None
13:03:27 + 20.5s phase=verifying label=Működés ellenőrzése… err=None hold=None
13:03:32 + 25.6s phase=done label=Frissítve err=None hold=None
{"accepted": true, "final_phase": "done", "update_error": null, "hold_reason": null, "state": "running", "duration_s": 25.7}
after: {"pinned_images": {"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"}, "installed_images": {"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"}, "catalog_images": {"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"}, "live_compose_image_lines": ["image: plexinc/pms-docker:1.43.5.11029-6bf8af3f0@sha256:01bdf0ad6d53198da4331540c1e73ab65db7a72e961fff97ecb0847d46055c5f"], "docker_inspect": ["plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0@sha256:01bdf0ad6d53198da4331540c1e73ab65db7a72e961fff97ecb0847d46055c5f running=true restarts=0"]}
13:03:51 gate: plex is gated — passed as the household (cookie set)
identity 200 {"MediaContainer":{"size":0,"apiVersion":"1.2.3","claimed":false,"machineIdentifier":"f34aaee0f637bb3bf168e5b4e5ce2223c7c40c5f","version":"1.43.5.11029-6bf8af3f0"}}
@@ -0,0 +1,227 @@
{
"app": "plex",
"walk": "security",
"box_walk_version": 1,
"verdict": "proven",
"venue": "box 9202 (drill catalog), the guarded Update",
"from": {
"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"
},
"to": {
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
},
"observables_before": {
"pinned_images": {
"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"
},
"installed_images": {
"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"
},
"catalog_images": null,
"live_compose_image_lines": [
"image: plexinc/pms-docker:1.43.4.10903-e5521bd8c"
],
"docker_inspect": [
"plex plexinc/pms-docker:1.43.4.10903-e5521bd8c running=true restarts=0"
]
},
"setup_gate_before": {
"browser": {
"status": "302",
"redirect": "https://felhom.enkisfelhom.hu/__gate/start?…"
},
"script": {
"status": "401",
"redirect": null
}
},
"seed": "none",
"seed_why": "the first-run claim needs a token minted at plex.tv by a real Plex account; no account exists for this venue",
"drill_commit": "b5e2406d3b49",
"sync_waited_s": 4.5,
"badge_before": {
"hu": [
{
"title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.",
"text": "Frissítés elérhető — ma"
}
],
"en": [
{
"title": "A newer version of this app is available. Select the Update button to start it.",
"text": "Update available — today"
}
]
},
"measured_at": "2026-10-23T10:36:22Z",
"update": {
"accepted": true,
"http": "202",
"phases": [
{
"t": 0.0,
"phase": "backing-up",
"label": "Biztonsági mentés készül a frissítés előtt…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 11.3,
"phase": "safety-dump",
"label": "Adatbázis pillanatkép…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 12.3,
"phase": "pulling",
"label": "Új verzió letöltése…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 20.5,
"phase": "copying",
"label": "Az adatok másolása a frissítés előtt…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 30.8,
"phase": "starting",
"label": "Indítás az új verzióval…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 31.8,
"phase": "verifying",
"label": "Működés ellenőrzése…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 36.9,
"phase": "done",
"label": "Frissítve",
"updating": false,
"error": null,
"hold": null
}
],
"duration_s": 36.9,
"final_phase": "done",
"update_error": null,
"hold_reason": null,
"state": "running"
},
"observables_after": {
"pinned_images": {
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
},
"installed_images": {
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
},
"catalog_images": {
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
},
"live_compose_image_lines": [
"image: plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
],
"docker_inspect": [
"plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0 running=true restarts=0"
]
},
"badge_after": {
"hu": [
{
"title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.",
"text": "Naprakész"
}
],
"en": [
{
"title": "This app is running the newest version available.",
"text": "Up to date"
}
]
},
"state_after": "running",
"hold_after": null,
"to_digests": {
"plex": "sha256:01bdf0ad6d53198da4331540c1e73ab65db7a72e961fff97ecb0847d46055c5f"
},
"four_observables": {
"pinned": {
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
},
"installed": {
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
},
"catalog": {
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
},
"live_compose": [
"plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
]
},
"setup_gate_after": {
"browser": {
"status": "302",
"redirect": "https://felhom.enkisfelhom.hu/__gate/start?…"
},
"script": {
"status": "401",
"redirect": null
}
},
"memory": {
"soak_s": 600,
"interval_s": 10,
"samples": 60,
"containers": {
"plex": {
"samples": 60,
"anon_peak_bytes": 229015552,
"limit_bytes": 2147483648,
"anon_peak_pct": 10.7,
"cgroup_peak_pct": 11.5,
"oom_kill_delta": 0,
"oom_killed_flag": false,
"restarts_delta": 0,
"running_all_samples": true,
"cgroup": "/sys/fs/cgroup/system.slice/docker-f699bdec12560064fa6bd64884d941915072974ca27d389bcb3e3b917861ac88.scope"
}
},
"oom_kills": 0,
"restarts": 0,
"peak_pct": 10.7
},
"why": "the security-fix step ran on the box through the guarded Update; healthy, TO on all four observables, no undo, no hold, a clean soak",
"log": [
"12:35:39 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH', 'PLEX_CLAIM']",
"12:35:40 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}",
"12:36:05 [1] deployed, controller state=running, pinned={'plex': 'plexinc/pms-docker:1.43.4.10903-e5521bd8c'}",
"12:36:05 gate: plex is gated — passed as the household (cookie set)",
"12:36:08 [2] setup gate as installed: {'browser': {'status': '302', 'redirect': 'https://felhom.enkisfelhom.hu/__gate/start?…'}, 'script': {'status': '401', 'redirect': None}}",
"12:36:08 [3] seed: none — the first-run claim needs a token minted at plex.tv by a real Plex account; no account exists for this venue",
"12:36:09 [5] drill commit b5e2406d3b49: plex plexinc/pms-docker:1.43.4.10903-e5521bd8c -> plexinc/pms-docker:1.43.5.11029-6bf8af3f0 (push rc=0)",
"12:36:18 [5] the box's drill mirror now at b5e2406d",
"12:36:23 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}",
"12:36:23 + 0.0s phase=backing-up label=Biztonsági mentés készül a frissítés előtt… err=None hold=None",
"12:36:34 + 11.3s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None",
"12:36:35 + 12.3s phase=pulling label=Új verzió letöltése… err=None hold=None",
"12:36:43 + 20.5s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None",
"12:36:53 + 30.8s phase=starting label=Indítás az új verzióval… err=None hold=None",
"12:36:54 + 31.8s phase=verifying label=Működés ellenőrzése… err=None hold=None",
"12:36:59 + 36.9s phase=done label=Frissítve err=None hold=None",
"12:37:18 [7] memory soak 600 s …",
"12:47:24 [7] soak: peak 10.7% of the limit (anon), kills 0, restarts 0"
]
}
@@ -0,0 +1,23 @@
12:35:39 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH', 'PLEX_CLAIM']
12:35:40 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
12:36:05 [1] deployed, controller state=running, pinned={'plex': 'plexinc/pms-docker:1.43.4.10903-e5521bd8c'}
12:36:05 gate: plex is gated — passed as the household (cookie set)
12:36:08 [2] setup gate as installed: {'browser': {'status': '302', 'redirect': 'https://felhom.enkisfelhom.hu/__gate/start?…'}, 'script': {'status': '401', 'redirect': None}}
12:36:08 [3] seed: none — the first-run claim needs a token minted at plex.tv by a real Plex account; no account exists for this venue
12:36:09 [5] drill commit b5e2406d3b49: plex plexinc/pms-docker:1.43.4.10903-e5521bd8c -> plexinc/pms-docker:1.43.5.11029-6bf8af3f0 (push rc=0)
12:36:18 [5] the box's drill mirror now at b5e2406d
12:36:23 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}
12:36:23 + 0.0s phase=backing-up label=Biztonsági mentés készül a frissítés előtt… err=None hold=None
12:36:34 + 11.3s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None
12:36:35 + 12.3s phase=pulling label=Új verzió letöltése… err=None hold=None
12:36:43 + 20.5s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None
12:36:53 + 30.8s phase=starting label=Indítás az új verzióval… err=None hold=None
12:36:54 + 31.8s phase=verifying label=Működés ellenőrzése… err=None hold=None
12:36:59 + 36.9s phase=done label=Frissítve err=None hold=None
12:37:18 [7] memory soak 600 s …
12:47:24 [7] soak: peak 10.7% of the limit (anon), kills 0, restarts 0
12:47:24 [9] verdict proven -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/security-lane-2026-10-23/P2-walk/verdict.json
12:47:24 RESULT proven — the security-fix step ran on the box through the guarded Update; healthy, TO on all four observables, no undo, no hold, a clean soak
12:47:33 [X] stop -> 200 {'ok': True, 'message': 'Stack plex stop completed'}
12:48:05 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'plex', 'volumes_removed': ['plex_plex_config', 'plex_plex_transcode'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'us
12:48:13 [X] after remove: deployed=False leftovers='/opt/docker/stacks/plex'
@@ -0,0 +1 @@
WROTE plex: {'plex': 'plexinc/pms-docker:1.43.4.10903-e5521bd8c'} -> {'plex': 'plexinc/pms-docker:1.43.5.11029-6bf8af3f0'} peak 10.7% marks {'files_may_change': False, 'needs_person': None, 'memory_tight': False}
@@ -0,0 +1,21 @@
13:04:15 [X] stop -> 200 {'ok': True, 'message': 'Stack plex stop completed'}
13:04:47 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'plex', 'volumes_removed': ['plex_plex_config', 'plex_plex_transcode'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'us
13:04:56 [X] after remove: deployed=False leftovers='/opt/docker/stacks/plex'
remote: . Processing 1 references
remote: Processed 1 references in total
13:05:05 [5] the box's drill mirror now at 42945c94
13:07:34 [sync] the badge NEVER caught up to plexinc/pms-docker:1.43.4.10903-e5521bd8c in 149.0s — catalog_images = None
13:07:34 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH', 'PLEX_CLAIM']
13:07:34 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
13:07:54 [1] deployed, controller state=running, pinned={'plex': 'plexinc/pms-docker:1.43.4.10903-e5521bd8c'}
deploy True
13:07:55 gate: plex is gated — passed as the household (cookie set)
remote: . Processing 1 references
remote: Processed 1 references in total
17e0da0 Revert "DRILL plex: compose image line back to 1.43.4 ONLY (the ladder untouched) — a fresh install at FROM for the Part 3 proof; reverted right after"
42945c9 DRILL plex: compose image line back to 1.43.4 ONLY (the ladder untouched) — a fresh install at FROM for the Part 3 proof; reverted right after
f46e1bc DRILL plex: security_fix step 1.43.4 -> 1.43.5 written by upgrade-test.py --write-ladder --security (D39 = B proof)
13:08:04 [5] the box's drill mirror now at 17e0da02
obs {"pinned_images": {"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"}, "installed_images": {"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"}, "catalog_images": {"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"}, "live_compose_image_lines": ["image: plexinc/pms-docker:1.43.4.10903-e5521bd8c"], "docker_inspect": ["plex plexinc/pms-docker:1.43.4.10903-e5521bd8c running=true restarts=0"]}
badge {'hu': [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — ma'}], 'en': [{'title': 'A newer version of this app is available. Select the Update button to start it.', 'text': 'Update available — today'}]}
@@ -0,0 +1,17 @@
13:12:03 [5] the box's drill mirror now at 17e0da02
before: {"pinned_images": {"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"}, "installed_images": {"plex": "plexinc/pms-docker:1.43.4.10903-e5521bd8c"}, "catalog_images": {"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"}, "live_compose_image_lines": ["image: plexinc/pms-docker:1.43.4.10903-e5521bd8c"], "docker_inspect": ["plex plexinc/pms-docker:1.43.4.10903-e5521bd8c running=true restarts=0"]}
badge before: {'hu': [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — ma'}], 'en': [{'title': 'A newer version of this app is available. Select the Update button to start it.', 'text': 'Update available — today'}]}
night chain -> 202 {'data': {'legs': ['db-dump', 'tier2', 'update-leg']}, 'message': 'started', 'ok': True}
the leg's own lines:
2026/10/23 11:12:13 night_chain.go:108: [INFO] [night-chain] manual run started from 172.18.0.6:38520: [db-dump tier2 update-leg]
2026/10/23 11:13:15 night_chain.go:77: [INFO] [night-chain] update-leg: started
2026/10/23 11:13:15 unattended.go:279: [INFO] [update-leg] started (manual-chain): window 02:30, no step starts at or after 16:28
2026/10/23 11:13:15 unattended.go:345: [INFO] [update-leg] plex: step pressed plex=plexinc/pms-docker:1.43.4.10903-e5521bd8c → plex=plexinc/pms-docker:1.43.5.11029-6bf8af3f0
2026/10/23 11:13:15 unattended.go:351: [INFO] [update-leg] plex: security fix CVE-2026-96652,CVE-2026-96654,CVE-2026-96655,CVE-2026-96656 taken (plex=plexinc/pms-docker:1.43.4.10903-e5521bd8c → plex=plexinc/pms-docker:1.43.5.11029-6bf8af3f0)
2026/10/23 11:13:45 unattended.go:359: [INFO] [update-leg] plex: step ended done after 30.1 s
2026/10/23 11:13:45 unattended.go:325: [INFO] [update-leg] privatebin: skipped — stopped_by_household
2026/10/23 11:13:45 unattended.go:257: [INFO] [update-leg] update leg (manual-chain): done=1 undone=0 held=0 failed=0 skipped=1 in 30s [skipped: privatebin=stopped_by_household] [security fix plex: CVE-2026-96652,CVE-2026-96654,CVE-2026-96655,CVE-2026-96656 done]
2026/10/23 11:13:45 night_chain.go:82: [INFO] [night-chain] update-leg: done in 30s
after: {"pinned_images": {"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"}, "installed_images": {"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"}, "catalog_images": {"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"}, "live_compose_image_lines": ["image: plexinc/pms-docker:1.43.5.11029-6bf8af3f0@sha256:01bdf0ad6d53198da4331540c1e73ab65db7a72e961fff97ecb0847d46055c5f"], "docker_inspect": ["plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0@sha256:01bdf0ad6d53198da4331540c1e73ab65db7a72e961fff97ecb0847d46055c5f running=true restarts=0"]}
state running hold None phase done
@@ -0,0 +1,28 @@
2026/10/23 11:13:15 tier2.go:507: [INFO] [backup] Tier 2 copied plex → /mnt/sizing-bench/backups/secondary/plex (2.3 MB, 0 leg(s), 0s)
2026/10/23 11:13:15 update.go:537: [INFO] [stacks] update plex: accepted — guarded update started
2026/10/23 11:13:15 unattended.go:345: [INFO] [update-leg] plex: step pressed plex=plexinc/pms-docker:1.43.4.10903-e5521bd8c → plex=plexinc/pms-docker:1.43.5.11029-6bf8af3f0
2026/10/23 11:13:15 unattended.go:351: [INFO] [update-leg] plex: security fix CVE-2026-96652,CVE-2026-96654,CVE-2026-96655,CVE-2026-96656 taken (plex=plexinc/pms-docker:1.43.4.10903-e5521bd8c → plex=plexinc/pms-docker:1.43.5.11029-6bf8af3f0)
2026/10/23 11:13:15 update.go:1378: [INFO] [stacks] update plex: phase checking
2026/10/23 11:13:15 update.go:795: [INFO] [stacks] update plex: ladder — the last step (1 of 1) — the catalog's current definition
2026/10/23 11:13:15 update.go:823: [INFO] [stacks] update plex: precondition met — Tier 2 (second drive) copy from 2026-10-23T11:13:04Z (0s old, limit 24h0m0s)
2026/10/23 11:13:15 update.go:1378: [INFO] [stacks] update plex: phase safety-dump
2026/10/23 11:13:16 update_guard.go:497: [INFO] [backup] update safety dump for plex: the app has no database — nothing to copy (no-op)
2026/10/23 11:13:16 update.go:855: [INFO] [stacks] update plex: safety dump done (0 file(s)) []
2026/10/23 11:13:17 undo.go:287: [INFO] [stacks] update plex: the undo copy will hold 2 named volume(s), 2.8 MiB
2026/10/23 11:13:17 update.go:1378: [INFO] [stacks] update plex: phase pinning
2026/10/23 11:13:17 pin.go:385: [INFO] [stacks] update plex: pin advanced to /opt/docker/felhom-controller/data/catalog-cache/templates/plex/docker-compose.yml (plex=plexinc/pms-docker:1.43.5.11029-6bf8af3f0)
2026/10/23 11:13:17 update.go:1378: [INFO] [stacks] update plex: phase pulling
2026/10/23 11:13:28 update.go:1378: [INFO] [stacks] update plex: phase copying
2026/10/23 11:13:38 update.go:1378: [INFO] [stacks] update plex: phase copying
2026/10/23 11:13:38 undo.go:437: [INFO] [stacks] update plex: copied plex_plex_config → plex_plex_config.pre-update-20261023T111338Z in 520ms
2026/10/23 11:13:38 update.go:1378: [INFO] [stacks] update plex: phase copying
2026/10/23 11:13:39 undo.go:437: [INFO] [stacks] update plex: copied plex_plex_transcode → plex_plex_transcode.pre-update-20261023T111338Z in 465ms
2026/10/23 11:13:39 update.go:1378: [INFO] [stacks] update plex: phase starting
2026/10/23 11:13:39 update.go:1378: [INFO] [stacks] update plex: phase verifying
2026/10/23 11:13:44 update.go:1006: [INFO] [stacks] update plex: healthy after 5s (the app's health check passed)
2026/10/23 11:13:45 update.go:1043: [INFO] [stacks] update plex: DONE in 29s
2026/10/23 11:13:45 image_retention.go:238: [INFO] [stacks] image retention (update of plex): pass over 40 image(s) of [plexinc/pms-docker] — 0 candidate(s), 0 deleted, the rest kept
2026/10/23 11:13:45 unattended.go:359: [INFO] [update-leg] plex: step ended done after 30.1 s
2026/10/23 11:13:45 unattended.go:257: [INFO] [update-leg] update leg (manual-chain): done=1 undone=0 held=0 failed=0 skipped=1 in 30s [skipped: privatebin=stopped_by_household] [security fix plex: CVE-2026-96652,CVE-2026-96654,CVE-2026-96655,CVE-2026-96656 do
identity 200 {"MediaContainer":{"size":0,"apiVersion":"1.2.3","claimed":false,"machineIdentifier":"de0d1cd3ffa436835af296505a1e2534be391d1f","version":"1.43.5.11029-6bf8af3f
state running health None
@@ -0,0 +1,27 @@
### root@192.168.0.154
containers=10 unhealthy=0 restarting=0 exited=[] gitea.dooplex.hu/admin/felhom-controller:0.341.0 Up 35 seconds (healthy)
started=2026-10-23T11:15:27.242236608Z
errors=0 info=126
2026/10/23 11:15:28 [INFO] Self-update enabled (check every 6h, auto-update: false, auto-update time: 04:30)
2026/10/23 11:15:28 [INFO] [scheduler] Registered periodic job: selfupdate-check (every 6h0m0s)
2026/10/23 11:15:59 [INFO] [selfupdate] Current version 0.341.0 is up to date
2026/10/23 11:15:29 [INFO] [sync] Initial sync: Sablonok frissítve — a katalógus új változata betöltve; a telepített alkalmazások fájljai nem változt
### hp
containers=24 unhealthy=0 restarting=0 exited=[] gitea.dooplex.hu/admin/felhom-controller:0.341.0 Up 36 seconds (healthy)
started=2026-10-23T11:15:28.047379649Z
errors=0 info=135
2026/10/23 11:15:29 deploy.go:1442: [WARN] [stacks] Stack bookstack: new field ADMIN_PASSWORD (type=password) requires manual configuration
2026/10/23 11:15:29 deploy.go:1442: [WARN] [stacks] Stack calibre-web: new field ADMIN_PASSWORD (type=password) requires manual configuration
2026/10/23 11:15:29 deploy.go:1442: [WARN] [stacks] Stack plex: new field PLEX_CLAIM (type=text) requires manual configuration
2026/10/23 11:15:29 main.go:833: [INFO] Self-update enabled (check every 6h, auto-update: false, auto-update time: 04:30)
2026/10/23 11:15:29 scheduler.go:120: [INFO] [scheduler] Registered periodic job: selfupdate-check (every 6h0m0s)
2026/10/23 11:16:01 updater.go:196: [INFO] [selfupdate] Current version 0.341.0 is up to date
2026/10/23 11:15:30 sync.go:142: [INFO] [sync] Initial sync: Sablonok frissítve — a katalógus új változata betöltve; a telepített alkalmazások fájljai
### felhom-pve
containers=9 unhealthy=0 restarting=0 exited=[] gitea.dooplex.hu/admin/felhom-controller:0.341.0 Up 40 seconds (healthy)
started=2026-10-23T11:15:11.64495822Z
errors=0 info=141
2026/10/23 11:15:12 [INFO] Self-update enabled (check every 6h, auto-update: false, auto-update time: 04:30)
2026/10/23 11:15:12 [INFO] [scheduler] Registered periodic job: selfupdate-check (every 6h0m0s)
2026/10/23 11:15:44 [INFO] [selfupdate] Current version 0.341.0 is up to date
2026/10/23 11:15:13 [INFO] [sync] Initial sync: Sablonok frissítve — a katalógus új változata betöltve; a telepített alkalmazások fájljai nem változt
@@ -0,0 +1,10 @@
2026-10-23T11:24:47Z
HTTP/1.1 303 See Other
Location: /configuration?flash=floor_set
Effective floor: v0.341.0 — source: DB (hub_settings) ; env fallback would be v0.120.0 Save global floor… DB override: v
clear tester-1 Location: /customers/tester-1?flash=floor_set
clear demo-hp Location: /customers/demo-hp?flash=floor_set
clear demo-felhom Location: /customers/demo-felhom?flash=floor_set
2026/10/23 13:15:19 [INFO] managed floor SERVED for tester-1: floor 0.341.0, agent requirement "0.158.1" from declared (golden 0.331.0)
2026/10/23 13:15:20 [INFO] managed floor SERVED for demo-felhom: floor 0.341.0, agent requirement "0.158.1" from declared (golden 0.331.0)
2026/10/23 13:15:20 [INFO] managed floor SERVED for demo-hp: floor 0.341.0, agent requirement "0.158.1" from declared (golden 0.331.0)
@@ -0,0 +1,33 @@
control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
== tester-1 (enkicsifelhom.hu): 6 public name(s)
OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
== demo-hp (enkisfelhom.hu): 14 public name(s)
OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
== demo-felhom (demo-felhom.eu): 8 public name(s)
OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
== every public name redirects http -> https
@@ -0,0 +1,3 @@
9201 login ok, /api/stacks 200
login failed: no session cookie
n100 /api/health 200tester-1 (BASE .148) login ok, /api/stacks 200
@@ -0,0 +1,3 @@
tester-1 HTTP/1.1 303 See Other Location: /customers/tester-1?flash=floor_set
demo-hp HTTP/1.1 303 See Other Location: /customers/demo-hp?flash=floor_set
demo-felhom HTTP/1.1 303 See Other Location: /customers/demo-felhom?flash=floor_set
@@ -0,0 +1,74 @@
# D39 = B — the security-fix lane, built and proven (2026-10-23)
Decision: `architecture/09-update-architecture.md` §3 decision 255 (the operator started the D39 brief ~12:00; R-1239
closed). Read before the work: `09` §3 decisions 13–17, 22, 25–26, 52 and §6.4 parts 4–7; `00-capability-map.md` (the
B rows); `13-threat-model.md`; `runbooks/target-selection.md`; `runbooks/publish-train-rules.md`.
Baselines at start: catalog `cde26ab`, controller `32e61d5` (0.340.0, global floor 06:41 UTC — read from the hub before
the first controller commit), felhom.eu `4b9b0af3`. Machines: DooPlex (builds, the writer), demo-hp (9202 scratch; the
release on 9201), Tester 1, demo-felhom (release only). Not touched: ep0, any household box.
## Venue
9202 was pointed at the drill catalog through its LOCAL mirror (`tools/repoint.py`, copied from
`audits/burndown-3-2026-10-17/tools/`, R-992 — no catalog credential on the box): `repoint-drill.txt`. The drill was
reset to live `main` (`cde26ab`) first. 9202's dashboard password is its own (`REHEARSAL_2026_10_20_9202_PW`; the
demo-hp one was refused — five failed logins hit the one-minute limit; `operations/nodes.md` corrected).
## Part 2 — the lane on the old controller (9202 on 0.340.0)
| Step | Result | File |
|---|---|---|
| First try of the negative control | `inconclusive` — the install was refused before anything was made (Plex wants the drive itself as `HDD_PATH`); the walk gained `--value` | `P2-negative-try1-install-refused/` |
| **Negative control**: FROM Plex 1.43.4.10903, TO `busybox:1.36.1` (exits at once) | **`failed`** — backing-up → … → verifying → undoing → **undone** after 335 s; Plex 1.43.4 back | `P2-negative/walk.txt`, `box-verdict-plex.json` |
| The writer on that walk | **refused**: „the box walk is 'security' with verdict 'failed' — only a PROVEN `box_walk.py --security` walk is written" | `P2-negative/writer-refusal.txt` |
| **The walk**: Plex 1.43.4.10903-e5521bd8c → 1.43.5.11029-6bf8af3f0 | **`proven`** — done in 36.9 s, no undo, no hold; pinned = installed = catalog = live compose = running = TO; digest `sha256:01bdf0ad6d53198da4331540c1e73ab65db7a72e961fff97ecb0847d46055c5f` (the registry's, re-read by the writer); soak 600 s, 60 samples, anon peak 10.7 % of 2 GiB (cgroup peak 11.5 %), 0 OOM kills, 0 restarts; `seed: none` (Plex's fixture is a `NoRoute` stub) | `P2-walk/` |
| Setup gate (unclaimed Plex) | a browser (`Accept: text/html`) → 302 to `felhom.<domain>/__gate/start`; a script → 401; the same after the update | `P2-walk/box-verdict-plex.json` |
| The writer, into the DRILL only | wrote the `security_fix` entry: ids CVE-2026-96652/-96654/-96655/-96656; advisory Plex's notice `https://forums.plex.tv/t/important-security-update-for-plex-media-server-v1-43-2-and-earlier/942319` (the one CISA KEV cites, all four added 2026-10-20 — `cisa-kev-plex2shell.json`, from CISA's own feed); `fixed_in` 1.43.3.10861; the gate accepts it with its WARN | `P2-walk/writer.txt`; drill `f46e1bc` |
| Old controller, night (`POST /api/debug/backup/night-chain`) | **`[update-leg] plex: skipped — not_proven`**; Plex stays 1.43.4 (control: the same leg judged privatebin) | `P2-oldctl/2-night-chain-0.340.0.txt` |
| Old controller, hand press | **done** in 25.7 s; Plex 1.43.5 at the step's digest; `/identity` 200 with 1.43.5 | `P2-oldctl/3-hand-press-0.340.0.txt` |
**How Plex was put back at 1.43.4 for Part 3:** removed through the product (with its data); drill commit `42945c9`
moved ONLY the compose's image line back to 1.43.4 (the ladder line untouched); a fresh install pinned 1.43.4; `17e0da0`
reverted that commit, so the step is the head again and the badge read „Frissítés elérhető" (`P3-newctl/0-…`).
## Part 3 — the new controller (0.341.0-rc.1 on 9202)
Controller `63dcccf` (CI run 2264 / #471 success), image `0.341.0-rc.1` set by hand (9202's self-update is off). The
night chain: `[update-leg] plex: step pressed …`, **`[update-leg] plex: security fix
CVE-2026-96652,CVE-2026-96654,CVE-2026-96655,CVE-2026-96656 taken (plex=…1.43.4… → plex=…1.43.5…)`**, `step ended done
after 30.1 s`; the summary line ends `[security fix plex: … done]`. Backup first: the update's precondition met by a
second-drive copy 0 s old, the undo copy of 2 named volumes (2.8 MiB), then the pin. `/identity` 200 with 1.43.5
(`P3-newctl/1-…`, `2-…`).
## Part 4 — release 0.341.0
Image `0.341.0` (`sha256:8aa4ce60…c0bc`) from `63dcccf`; per-box floors on tester-1, demo-hp, demo-felhom (303
`floor_set`); all three ran 0.341.0 by 11:15:31 UTC. Box walk: 0 unhealthy, 0 restarting, 0 exited, 0 `[ERROR]` lines since
the start (126 / 135 / 141 `[INFO]` lines — the control); selfupdate „0.341.0 is up to date"; the https-only check passes
on every public name; the dashboard login works on demo-hp and Tester 1 (`/api/stacks` 200), the N100's `/api/health`
200 (`P4-release/`). **The operator said yes in chat; the global floor 0.341.0 / MinAgent 0.158.1 was set at 11:24:47
UTC** (303 `floor_set`; the running hub's page reads „Effective floor: v0.341.0 — source: DB"); the per-box floors
cleared.
## Teardown (three layers)
- **Machine (9202):** Plex removed through the product (no plex container or volume left); `controller.yaml` restored
byte-identical (`cmp`) and the catalog cache re-cloned from the live catalog (R-615); this run's saved copy deleted
after the `cmp`; image `0.341.0`. 34 older saved copies from earlier sessions stay — filed R-1240. The drill
repository keeps this run's commits as the record.
- **Host (demo-hp):** nothing provisioned.
- **Hub:** the global floor 0.341.0 (the operator's yes); per-box floors cleared.
## Observations
- FILED: R-1240 — 9202 holds 34 saved `controller.yaml` copies with the catalog token.
- FIXED without a row: drill commits never reached a box pointed at the drill's local mirror (`box_walk.refresh_box_mirror`);
two stale catalog tests (9202 address, gate count + missing sibling in the shallow clone); Tester 1's guest moved
`.101` → `.148` and the box walk had it typed (now read from the guest); `operations/nodes.md` (9202's password,
Tester 1's address).
- NOT-A-FINDING: the writer's own engine-major check is redundant with `check_entry` (two layers on purpose; the test
asserts the refusal); `/opt/docker/stacks/plex` keeps the catalog's template files after a removal, as every catalog
app's folder does (72); the demo-hp start WARNs „new field … requires manual configuration" come from code 0.341.0 did
not change; a waiter of mine matched its own command line (a known trap, memory `pgrep-waiter-matches-itself`) and
was stopped.
@@ -0,0 +1,70 @@
{
"catalogVersion": "2026.10.21",
"dateReleased": "2026-10-21T14:04:39.8983Z",
"entries": [
{
"cveID": "CVE-2026-96656",
"vendorProject": "Plex",
"product": "Media Server",
"vulnerabilityName": "Plex Media Server External Control of File Name or Path Vulnerability",
"dateAdded": "2026-10-20",
"shortDescription": "Plex Media Server contains an external control of file name or path vulnerability which can allow an admin user to write arbitrary files that may be executed on load. This vulnerability can be chained with CVE-2026-96652, CVE-2026-96654, and CVE-2026-96655.",
"requiredAction": "Apply mitigations in accordance with vendor instructions, ensuring compliance with CISA\u2019s BOD 26-04 Prioritizing Security Updates Based on Risk (see URL in Notes) guidance and CISA\u2019s \u201cForensics Triage Requirements\u201d (see URL in Notes). Follow applicable BOD 26-04 guidance for cloud services or discontinue use of the product if mitigations are unavailable. Stakeholders are responsible for evaluating each asset's internet exposure and ensuring adherence to BOD 26-04 patching guidelines.",
"dueDate": "2026-10-23",
"knownRansomwareCampaignUse": "Unknown",
"forensicTriage": "Yes",
"notes": "https://forums.plex.tv/t/important-security-update-for-plex-media-server-v1-43-2-and-earlier/942319 ; ; BOD 26-04: https://www.cisa.gov/news-events/directives/bod-26-04-prioritizing-security-updates-based-risk ; Forensics Triage Requirements: https://www.cisa.gov/news-events/directives/bod-26-04-implementation-guidance-prioritizing-security-updates-based-risk ; https://nvd.nist.gov/vuln/detail/CVE-2026-96656",
"cwes": [
"CWE-73"
]
},
{
"cveID": "CVE-2026-96655",
"vendorProject": "Plex",
"product": "Media Server",
"vulnerabilityName": "Plex Media Server Server-Side Request Forgery (SSRF) Vulnerability",
"dateAdded": "2026-10-20",
"shortDescription": "Plex Media Server contains a server-side request forgery (SSRF) vulnerability that could allow an authenticated user to request arbitrary internal or external addresses via the '/video/:/transcode/universal' path parameter. This vulnerability can be chained with CVE-2026-96652, CVE-2026-96654, and CVE-2026-96656.",
"requiredAction": "Apply mitigations in accordance with vendor instructions, ensuring compliance with CISA\u2019s BOD 26-04 Prioritizing Security Updates Based on Risk (see URL in Notes) guidance and CISA\u2019s \u201cForensics Triage Requirements\u201d (see URL in Notes). Follow applicable BOD 26-04 guidance for cloud services or discontinue use of the product if mitigations are unavailable. Stakeholders are responsible for evaluating each asset's internet exposure and ensuring adherence to BOD 26-04 patching guidelines.",
"dueDate": "2026-11-03",
"knownRansomwareCampaignUse": "Unknown",
"forensicTriage": "No",
"notes": "https://forums.plex.tv/t/important-security-update-for-plex-media-server-v1-43-2-and-earlier/942319 ; ; BOD 26-04: https://www.cisa.gov/news-events/directives/bod-26-04-prioritizing-security-updates-based-risk ; Forensics Triage Requirements: https://www.cisa.gov/news-events/directives/bod-26-04-implementation-guidance-prioritizing-security-updates-based-risk ; https://nvd.nist.gov/vuln/detail/CVE-2026-96655",
"cwes": [
"CWE-918"
]
},
{
"cveID": "CVE-2026-96654",
"vendorProject": "Plex",
"product": "Media Server",
"vulnerabilityName": "Plex Media Server Improper Neutralization of Encoded URI Schemes in a Web Page Vulnerability",
"dateAdded": "2026-10-20",
"shortDescription": "Plex Media Server contains an improper neutralization of encoded URI schemes in a web page vulnerability that could allow an attacker to call other plugins' functions and supply their own parameters. This vulnerability can be chained with CVE-2026-96652, CVE-2026-96655, and CVE-2026-96656.",
"requiredAction": "Apply mitigations in accordance with vendor instructions, ensuring compliance with CISA\u2019s BOD 26-04 Prioritizing Security Updates Based on Risk (see URL in Notes) guidance and CISA\u2019s \u201cForensics Triage Requirements\u201d (see URL in Notes). Follow applicable BOD 26-04 guidance for cloud services or discontinue use of the product if mitigations are unavailable. Stakeholders are responsible for evaluating each asset's internet exposure and ensuring adherence to BOD 26-04 patching guidelines.",
"dueDate": "2026-10-23",
"knownRansomwareCampaignUse": "Unknown",
"forensicTriage": "No",
"notes": "https://forums.plex.tv/t/important-security-update-for-plex-media-server-v1-43-2-and-earlier/942319 ; ; BOD 26-04: https://www.cisa.gov/news-events/directives/bod-26-04-prioritizing-security-updates-based-risk ; Forensics Triage Requirements: https://www.cisa.gov/news-events/directives/bod-26-04-implementation-guidance-prioritizing-security-updates-based-risk ; https://nvd.nist.gov/vuln/detail/CVE-2026-96654",
"cwes": [
"CWE-84"
]
},
{
"cveID": "CVE-2026-96652",
"vendorProject": "Plex",
"product": "Media Server",
"vulnerabilityName": "Plex Media Server Server-Side Request Forgery (SSRF) Vulnerability",
"dateAdded": "2026-10-20",
"shortDescription": "Plex Media Server contains a server-side request forgery (SSRF) vulnerability which could allow an attacker using any X-Plex-Token value to include a full URL in the 'protocol' parameter and force the Plex server to POST to the attacker's chosen destination. This vulnerability can be chained with CVE-2026-96654, CVE-2026-96655, and CVE-2026-96656. ",
"requiredAction": "Apply mitigations in accordance with vendor instructions, ensuring compliance with CISA\u2019s BOD 26-04 Prioritizing Security Updates Based on Risk (see URL in Notes) guidance and CISA\u2019s \u201cForensics Triage Requirements\u201d (see URL in Notes). Follow applicable BOD 26-04 guidance for cloud services or discontinue use of the product if mitigations are unavailable. Stakeholders are responsible for evaluating each asset's internet exposure and ensuring adherence to BOD 26-04 patching guidelines.",
"dueDate": "2026-11-03",
"knownRansomwareCampaignUse": "Unknown",
"forensicTriage": "No",
"notes": "https://forums.plex.tv/t/important-security-update-for-plex-media-server-v1-43-2-and-earlier/942319 ; ; BOD 26-04: https://www.cisa.gov/news-events/directives/bod-26-04-prioritizing-security-updates-based-risk ; Forensics Triage Requirements: https://www.cisa.gov/news-events/directives/bod-26-04-implementation-guidance-prioritizing-security-updates-based-risk ; https://nvd.nist.gov/vuln/detail/CVE-2026-96652",
"cwes": [
"CWE-918"
]
}
]
}
@@ -0,0 +1,33 @@
mirror main cde26ab plex: guide checked against 1.43.5.11029-6bf8af3f0 (every HU + EN label read from the new web client, all equal); CHANGELOG + REPORT for the Plex2Shell move
drill repo main: cde26ab
26: repo_url: file:///opt/docker/felhom-controller/drill-catalog.git
26c26
< repo_url: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git
---
> repo_url: file:///opt/docker/felhom-controller/drill-catalog.git
Traceback (most recent call last):
File "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/security-lane-2026-10-23/tools/repoint.py", line 86, in <module>
{"mirror": mirror, "drill": drill, "restore": restore, "status": status}[sys.argv[1]]()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/security-lane-2026-10-23/tools/repoint.py", line 74, in drill
restart_sync()
~~~~~~~~~~~~^^
File "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/security-lane-2026-10-23/tools/repoint.py", line 64, in restart_sync
w.sync_rescan()
~~~~~~~~~~~~~^^
File "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts/box_walk.py", line 466, in sync_rescan
ctl("POST", "/api/sync")
~~~^^^^^^^^^^^^^^^^^^^^^
File "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts/box_walk.py", line 132, in ctl
sess = open(f"{SC}/sess{os.getpid()}.txt").read().strip()
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/63b6c84e-0c09-4657-a160-855d2e4fe326/scratchpad/sc9202/sess3574518.txt'
26: repo_url: file:///opt/docker/felhom-controller/drill-catalog.git
27: sync_interval: 15m
-rw------- 1 root root 1994 Oct 22 14:47 /var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml.pre-seclane1023
mirror main cde26ab plex: guide checked against 1.43.5.11029-6bf8af3f0 (every HU + EN label read from the new web client, all equal); CHANGELOG + REPORT for the Plex2Shell move
cache HEAD cde26ab plex: guide checked against 1.43.5.11029-6bf8af3f0 (every HU + EN label read from the new web client, all equal); CHANGELOG + REPORT for the Plex2Shell move
file:///opt/docker/felhom-controller/drill-catalog.git
@@ -0,0 +1,3 @@
13:25:23 [X] stop -> 200 {'ok': True, 'message': 'Stack plex stop completed'}
13:25:55 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'plex', 'volumes_removed': ['plex_plex_config', 'plex_plex_transcode'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'us
13:26:04 [X] after remove: deployed=False leftovers='/opt/docker/stacks/plex'
@@ -0,0 +1,19 @@
RESTORED-IDENTICAL
26: repo_url: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git
2026/10/23 11:26:11 sync.go:344: [WARN] [sync] git.repo_url changed (cache was cloned from file:///opt/docker/felhom-controller/drill-catalog.git, config says https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git) — re-cloning the catalog cache from the configured repository (R-615)
2026/10/23 11:26:11 sync.go:318: [INFO] [sync] Cloning https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git → /opt/docker/felhom-controller/data/catalog-cache
2026/10/23 11:26:13 sync.go:511: [WARN] [sync] immich: catalog definition REFUSED by the allow-list — the app keeps its last good compose, nothing written: the catalog definition of immich asks for container rights the box does not allow: immich-server: bind /mnt/sizing-bench/appdata/immich -> /usr/src/app/upload rw; immich-server: bind /mnt/sizing-bench/userdata/media/photos -> /external/photos ro
2026/10/23 11:26:13 sync.go:535: [INFO] [sync] Updated plex/docker-compose.yml
2026/10/23 11:26:13 sync.go:535: [INFO] [sync] Updated plex/.felhom.yml
2026/10/23 11:26:13 sync.go:261: [INFO] [sync] catalog moved 17e0da02fd61 -> 389859359491 (0 new, 1 stack dir(s) changed)
2026/10/23 11:26:13 sync.go:299: [INFO] [sync] Catalog sync complete
2026/10/23 11:26:13 sync.go:142: [INFO] [sync] Initial sync: Sablonok frissítve — frissítve: plex
26: repo_url: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git
27: sync_interval: 15m
-rw------- 1 root root 1994 Oct 22 14:47 /var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml.pre-seclane1023
mirror main 17e0da02 Revert "DRILL plex: compose image line back to 1.43.4 ONLY (the ladder untouched) — a fresh install at FROM for the Part 3 proof; reverted right after"
cache HEAD 3898593 D39 = B: the security-fix lane — a `security_fix` ladder step (box walk required, bench optional, CVE/GHSA notice), its gates + 8 decoys, the writer (--write-ladder --security), the box venue (box_walk.py --security, drill-mirror refresh); two stale tests fixed
https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git
@@ -0,0 +1,4 @@
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.341.0 Up 25 seconds (healthy)
0
removed-identical-save
34
@@ -0,0 +1,25 @@
"""chain.py — press "run tonight's chain now" (POST /api/debug/backup/night-chain, R-705) on the box and quote the update
leg's own lines; then read Plex's four observables. D39 = B proof (felhom.eu audits/security-lane-2026-10-23)."""
import json, re, sys, time
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
import box_walk as w
w.LOG = []
w.login()
w.refresh_box_mirror()
w.sync_rescan("plex", "plexinc/pms-docker:1.43.5.11029-6bf8af3f0", tries=20, delay=6)
print("before:", json.dumps(w.observables("plex")))
print("badge before:", w.badges("plex"))
since = w.guest("date -u +%Y-%m-%dT%H:%M:%SZ").strip()
code, d = w.ctl("POST", "/api/debug/backup/night-chain")
print("night chain ->", code, str(d)[:200])
leg = ""
for _ in range(240):
time.sleep(10)
leg = w.guest("docker logs --since %s felhom-controller 2>&1 | grep -E 'update-leg' | grep -v DEBUG | cut -c1-400" % since)
if re.search(r"update leg .*: done=", leg):
break
print("the leg's own lines:\n" + leg)
time.sleep(10)
print("after:", json.dumps(w.observables("plex")))
st = w.stack("plex")
print("state", st.get("state"), "hold", st.get("hold_reason"), "phase", st.get("update_phase"))
@@ -0,0 +1,86 @@
#!/usr/bin/env python3
"""repoint.py mirror|drill|restore|status — point a box (scratch 9202; copied from audits/burndown-3-2026-10-17/tools/ for the D39 security lane, 2026-10-23) at the drill catalog WITHOUT a credential (R-992).
WHY A LOCAL MIRROR. `09` §6.5's route needs a catalog credential on the box: the drill repo is private (R-992). This
session could not mint a READ-ONLY one — the Gitea credential on DooPlex is itself a token, and Gitea answers 403 to a
token asking to list or create tokens (measured 2026-10-14) — and the night of 2026-10-06 put the admin WRITE token
into the box's controller.yaml, which this task forbids. The cache-staging workaround of 2026-10-12 does not hold for a
test that lasts days: every sync (and every controller start) re-applies the live catalog over the staged templates
and over the installed apps' compose files (Syncer.copyTemplates, no deployed check).
So the drill catalog is MIRRORED onto the box: a bare git repository inside the controller's own data volume, and
`git.repo_url: file:///opt/docker/felhom-controller/drill-catalog.git`. The controller's sync runs exactly as for a
remote (clone/fetch/reset, the R-615 re-clone on a changed URL); no secret is on the box. DooPlex updates the mirror
from the drill repo with a git bundle (`mirror`). The drill repo on Gitea stays the record.
SC=~/.felhom-tester1 TARGET=tester-1 python3 repoint.py mirror # (re)load the mirror from the drill repo main
... repoint.py drill # save controller.yaml (once), set repo_url to the mirror, restart, sync, rescan
... repoint.py restore # put the saved controller.yaml back (byte-identical), restart, sync, rescan
... repoint.py status
"""
import os, subprocess, sys, time
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
import box_walk as w
DRILL = "/mnt/5_hdd/felhom.eu/drill/app-catalog-drill"
VOL = "/var/lib/docker/volumes/felhom-controller-data/_data" # guest path of the controller's data volume
CVOL = "/opt/docker/felhom-controller" # the same, inside the controller container
SAVE = f"{VOL}/controller.yaml.pre-seclane1023"
MIRROR_URL = f"file://{CVOL}/drill-catalog.git"
def status():
print(w.guest(f"""grep -nE 'repo_url|sync_interval' {VOL}/controller.yaml; ls -l {SAVE} 2>&1 | tail -1
docker exec felhom-controller sh -c 'git -C {CVOL}/drill-catalog.git log -1 --format="mirror main %h %s" main 2>&1;
git -C {CVOL}/data/catalog-cache log -1 --format="cache HEAD %h %s" 2>&1; git -C {CVOL}/data/catalog-cache remote get-url origin'"""))
def mirror():
b = "/tmp/drill-seclane1023.bundle"
subprocess.run(["git", "-C", DRILL, "bundle", "create", b, "main"], check=True, capture_output=True)
head = subprocess.run(["git", "-C", DRILL, "rev-parse", "--short", "main"], capture_output=True, text=True).stdout.strip()
ssh = w.ssh_args()
# bundle -> PVE host -> guest -> controller volume; the bare repo is made and fetched BY the controller's own user
subprocess.run(["scp", "-q", "-o", "StrictHostKeyChecking=accept-new"] + (["-J", w._T["jump"]] if w._T["jump"] else [])
+ [b, f"{w._T['ssh']}:{b}"], check=True)
subprocess.run(ssh + [f"pct push {w.GUEST} {b} {VOL}/drill-seclane1023.bundle && rm -f {b}"], check=True)
print(w.guest(f"""set -e
docker exec felhom-controller sh -c 'test -d {CVOL}/drill-catalog.git || git init -q --bare -b main {CVOL}/drill-catalog.git
git -C {CVOL}/drill-catalog.git fetch -q {CVOL}/drill-seclane1023.bundle +main:main && rm -f {CVOL}/drill-seclane1023.bundle
git -C {CVOL}/drill-catalog.git log -1 --format="mirror main %h %s" main'"""))
print("drill repo main:", head)
os.remove(b)
def restart_sync():
w.guest("docker restart felhom-controller >/dev/null")
for _ in range(60):
time.sleep(3)
try:
w.login()
break
except SystemExit:
continue
w.sync_rescan()
time.sleep(3)
print(w.guest("docker logs --since 3m felhom-controller 2>&1 | grep -E '\\[sync\\]' | grep -v DEBUG | tail -8"))
def drill():
print(w.guest(f"""set -e
test -f {SAVE} || cp -p {VOL}/controller.yaml {SAVE}
sed -i -E 's#^(\\s+repo_url: ).*$#\\1{MIRROR_URL}#' {VOL}/controller.yaml
grep -n 'repo_url' {VOL}/controller.yaml; diff {SAVE} {VOL}/controller.yaml || true"""))
restart_sync()
status()
def restore():
print(w.guest(f"""set -e
cp -p {SAVE} {VOL}/controller.yaml
cmp {SAVE} {VOL}/controller.yaml && echo RESTORED-IDENTICAL; grep -n repo_url {VOL}/controller.yaml"""))
restart_sync()
status()
{"mirror": mirror, "drill": drill, "restore": restore, "status": status}[sys.argv[1]]()
+22
View File
@@ -26,6 +26,28 @@
---
## 2026-10-23 day (later) — D39 = B: the security-fix lane
| Row | What | Closed | Evidence |
|---|---|---|---|
| **R-1239** | **The catalog has no fast lane for an urgent security fix — a decision for the operator.** (P2) | CLOSED 2026-10-23 — **D39 = B** (operator ruling by starting the D39 brief; `09` §3 decision 255): a `security_fix` ladder step, box walk required, bench optional, a CVE/GHSA notice as the authority, no operator word per incident. Catalog `3898593` (format, gates, 8 decoys, writer `--write-ladder --security`, `box_walk.py --security`); controller **0.341.0** (the leg takes it; global floor 11:24 UTC, MinAgent 0.158.1). **Reasoning kept:** a security-fix step never crosses a database engine major and is never a same-tag fix (decision 52's lane); a controller ≤ 0.340.0 skips it at night and a hand press takes it. | `audits/security-lane-2026-10-23/` (negative control `failed` and refused; Plex 1.43.4 → 1.43.5 proven on 9202; 0.340.0 night `not_proven` + hand press; rc.1 night `security fix … taken`) |
## 2026-10-23 day — the releases and the DooPlex hour (the operator's „go")
| Row | What | Closed | Evidence |
|---|---|---|---|
| **R-49** | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** (P3) | CLOSED 2026-10-23 — released in controller **0.340.0** (D21 = B: a template may exclude a named volume; Immich's model cache is out of every backup); the global floor 0.340.0 (MinAgent 0.158.1) since 06:41 UTC, all three test boxes on it | `audits/day-2026-10-23/` 03, 06 |
| **R-1204** | **A box that runs once with its clock ahead loses its WireGuard tunnel to ep0 — the PBS whole-box copies and the operator's door — until the real date passes the wrong one, and nothing says why.** (P2) | CLOSED 2026-10-23 — the sign reported and the heal (D30, the agent re-keys on the existing peer path) in agent **0.163.0** (vouched by the operator 07:58 UTC; on Tester 1, demo-hp, demo-felhom by signed jobs), the alarm and its clearing in hub **0.167.0** (deployed 08:27 UTC). The live heal was proven on the test night (`audits/night-2026-10-23/P4-testnight/`) | `audits/day-2026-10-23/` 07, and the hub watch in 14 |
| **R-1205** | **After a box death, a SECOND install that is deleted before it makes a key leaves the whole-box (PBS) tier waiting for the operator's re-issue.** (P2) | CLOSED 2026-10-23 — D31 in hub **0.167.0** (deployed 08:27 UTC, image sha256:177860a9…): the re-issue reads the newest deletion of a host that held a key | `audits/day-2026-10-23/` 14 |
| **R-1230** | **A reinstalled box that waits for the household's recovery code was reported „stuck” after an hour: a false operator mail, a needless re-issue of the off-site login, and a controller restart on the box.** (P3) | CLOSED 2026-10-23 — hub **0.167.0** (deployed 08:27 UTC): the delivery checker stands down for `awaiting_recovery_key` | `audits/day-2026-10-23/` 14 |
| **R-982** | **The Proxmox host accepts root password login over SSH from the LAN (measured on demo-hp), so anyone on the household network can try the root password.** (P3) | CLOSED 2026-10-23 — D4 delivered: the bundle drop-in in agent **0.163.0** (config bundle sha256 3afb88d9…, via a step bundle 0.163.0-step1 because it adds a path, R-880) on all three test boxes — `sshd -T` `permitrootlogin without-password`, sshd reloaded after the drop-in, a key login works, felhom-sshd active; new installs: installer **1.36.0** published (tag `installer-v1.36.0`, served sha256 de8bf5f3… = the tag). The real root password was not tried over SSH (CC does not hold it); the console is untouched by design | `audits/day-2026-10-23/` 07, 11 |
| **R-985** | **Our zero-knowledge wording is stronger than the product: on a live box the operator can reach the off-site backup key (through the vaulted root credential and the operator tunnel) without the customer's recovery code.** (P3) | CLOSED 2026-10-23 — the D3 wording published on the website (felhom.eu `0bb57a4f`), read back live at phone and computer size | `audits/day-2026-10-23/` 12-website-readback |
| **R-991** | **Nothing a buyer can read says which machines Felhom installs on, and a quarter of the forum beginners own only machines it cannot install on.** (P3) | CLOSED 2026-10-23 — the D5 minimum machine published (felhom.eu `0bb57a4f`), read back live | `audits/day-2026-10-23/` 12-website-readback |
| **R-1225** | **The website's minimum machine (8 GB) cannot run the photo app, and the download page does not say so.** (P3) | CLOSED 2026-10-23 — D36: „8 GB runs the basics; the photo app needs 16 GB” on both download pages, published `0bb57a4f`, read back live | `audits/day-2026-10-23/` 12-website-readback |
| **R-1226** | **Two FAQ answers say the household's data leaves the home only encrypted (the remote backup) — not true for data reached from outside.** (P2) | CLOSED 2026-10-23 — D36: both FAQ answers (visible + JSON-LD, HU + EN) now say what is opened from outside passes Cloudflare's network and is readable there; the backups leave only encrypted. Published `0bb57a4f`; live: the new sentence 2× per FAQ page, the old one 0× (control: the previous commit had it 2×) | `audits/day-2026-10-23/` 12-website-readback |
| **R-1227** | **The container registry and the agent packages exist only on DooPlex — after its loss no box updates, reinstalls or installs.** (P2) | CLOSED 2026-10-23 — D33 built (felhom.eu `d306b8c2`): nightly 01:40 the newest 3 release tags of controller, hub, samba (skopeo `dir:`, digest-true) and the newest 5 agent versions to ep0 `/srv/felhom-release-copy` (one new user `felhom-relcopy`, rrsync-confined; cap 3 GB by the sender); first run 676 MB in 91 s; pulled back, every sha256 OK, every image re-verified; controller 0.340.0 from the copy loaded on 9202 (ID = config digest); alarm `FelhomReleaseCopyStale`; restore steps in `total-loss-of-dooplex.md`. Follow-ups: R-1235 | `audits/day-2026-10-23/` 08-release-copy-D33.txt |
| **R-1234** | **D34's first push refused the hub's own catch-up and mailed a false „hub-db-backup failed" to admin@.** Installing the 6-hourly timer at 10:28 made systemd catch up the missed 08:30 slot at once (`Persistent=true`), while the freshly deployed hub 0.167.0 was still writing its start-up snapshot (`hub-*.db.tmp`); the only finished one was 02:00 (8 h), above the new 2-hour limit → „the hub stopped snapshotting", `OnFailure` mail id `01a167e0…`. The 2 h limit was also stricter than the hub's own 6-hour catch-up, so any restart could repeat it. (P3) | CLOSED 2026-10-23 day, the same session — the push waits (≤ 20 min) for a snapshot being written, and the limit is one slot gap + 1 h (7 h); tests `test_d34_waits_for_a_snapshot_being_written` (reproduces the 10:28 run when the wait is removed), `…_a_restart_within_one_slot_is_not_an_alarm`, `…_two_missed_slots_are_refused`, each red-proved | `audits/day-2026-10-23/13-hubdb-6h-D34.txt` |
## 2026-10-22 night — the seven decisions put into effect (full text of the rows: `git show 1100a344:documentation/backlog/OPEN-ITEMS.md`)
| Row | What | Closed | Evidence |
File diff suppressed because one or more lines are too long
+24
View File
@@ -5,6 +5,30 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
## Moved from STATUS on 2026-10-23 (afternoon)
**Updated 2026-10-23 (day).** Released on your go: controller 0.340.0 (the floor), agent 0.163.0 (vouched; root SSH by key only on the three test boxes), installer 1.36.0, hub 0.167.0, and the website with your two sentences. The hub database now leaves every 6 hours and the released images and agent packages every night, both read back from ep0; the break-glass sheet is current — please print it. One false failure mail reached you at 10:28 (my install of the 6-hour copy, fixed in the same hour).
**Decided 2026-10-23 ~12:00 — D39 = B** (the operator started the D39 brief; the ruling is in R-1239 in CLOSED-ITEMS.md and in CONTEXT.md):
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|---|---|---|---|---|---|---|
| D39 | R-1239 | When an app has an urgent security hole, may its fix reach the boxes at night without the full update test? Today it needs your word and a hand press on each box (Plex, today). | **Yes — a marked „security fix” step** (tested on one test box only; the night update then takes it; works without you awake) | stay as today: your word and a CC session for every hole; households wait for the press | the same as today | — |
## Moved from STATUS on 2026-10-23 (day)
**Updated 2026-10-23 (morning).** Your three picks are built (the tunnel repairs itself with a new key, the hub skips a keyless deleted install, the controller repository stays public), and the night copy can no longer outlive the controller. Four fresh reviews found 34 points, every real one fixed or a row; the test night PASSED (no failed check at night, every night job passed, nothing lost). Nothing was released or deployed; the morning list has the order, and the DooPlex survey asks four questions (D32–D35).
**Decided 2026-10-23 morning — the operator's „go" for the day brief** (each ruling in its row and in CONTEXT.md): D32 = yes, print the sheet; D33 = yes, the nightly release copy to ep0 (built, R-1227 closed); D34 = yes, the hub DB every 6 hours (built); D35 = yes, after the release week (R-1228, dated 2026-10-30); D36 = yes, both sentences (published). The rows as they stood:
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|---|---|---|---|---|---|---|
| D36 | R-1225, R-1226 | Two website sentences before the publish: „8 GB runs the basics; the photo app needs 16 GB” and „what you open from outside the home passes Cloudflare's network, readable there; the backups leave only encrypted”? | **Yes, both** (the patch's own aim is the whole truth) | — | the patch publishes with both promises a little too strong | — |
| D35 | R-1228 | After the release week, serve the website and the installer from Cloudflare, like the ISO? | **Yes, later** (no new installs stop when DooPlex is down) | keep them home: after a DooPlex loss, no install until it is back | the same | — |
| D34 | R-232 | Copy the hub database to ep0 every 6 hours instead of every night? | **Yes** (0 €, about 30 minutes, one DooPlex change) | keep nightly: up to a day of hub changes lost, e.g. a new box's recovery package | the same | — |
| D33 | R-1227 | Copy the released images and agent packages to ep0 every night? | **Yes** (0 €, about 2 hours of CC, a few GB on ep0) | a timed rebuild drill instead: about 3 hours once, and rebuilt images are not the same | after a DooPlex loss no box updates, reinstalls or installs until everything is rebuilt | — |
| D32 | R-923, R-924 | Print the break-glass sheet now (eight keys and logins + ep0's fingerprint) and keep it outside the home? | **Yes** (30 minutes; nothing else opens the off-site copies after a loss) | an encrypted USB stick outside the home: ~10 €, 1 hour, one more passphrase | after a DooPlex loss no off-site copy opens | — |
## Moved from STATUS on 2026-10-23
**Updated 2026-10-22 (night).** Your seven picks of 15:47 are in effect or ready: D6 and D8 are live, D21 waits for the controller release, and D3, D5 (the website) and D4 (root SSH) are ready patches. Four controller fixes are built and proven on the scratch box, among them a gentler night copy (the 03:30 memory pressure) and no import over a stopped app. Nothing was released or deployed; the morning list below has the order.
+2 -2
View File
@@ -43,7 +43,7 @@ DooPlex while the boxes are away (`felhom-pve-lan` → `No route to host`, 2026-
| What it is | **QEMU VM 341 `night1004-tester1` on demo-hp** (installed from the ISO on 2026-10-04, `audits/night-2026-10-04/tester1/`); disposable (operator) |
| PVE node name | `felhom` (its certificate: `CN=felhom.enkicsifelhom.hu`) |
| LAN address | **DHCP** — read 2026-10-06 as `192.168.0.154` (MAC `bc:24:11:ac:e3:f9`; find it with `ip neigh` after a ping) |
| Customer / guest | `tester-1`; guest LXC 9201 at `192.168.0.101`, dashboard `felhom.enkicsifelhom.hu` |
| Customer / guest | `tester-1`; guest LXC 9201 by DHCP (`192.168.0.101` until 2026-10-23, then `.148` — read it: `pct exec 9201 -- ip -4 -o addr show eth0`; `box_walk.py` reads it itself), dashboard `felhom.enkicsifelhom.hu` |
| SSH | **`ssh root@192.168.0.154` from DooPlex works BY KEY** (measured 2026-10-09: `BatchMode=yes`, used for the release read-back); the old route `ssh -J demo-hp root@<VM address>` stays the fallback; `box_walk.py` `TARGET=tester-1` holds the route |
| Identity checked | 2026-10-06: the agent's own report `host.node=felhom` = the VM's certificate; the guest answers its domain (200) and not demo-hp's (404) — `audits/readback-2026-10-07/E1-identity-match.txt` |
@@ -218,7 +218,7 @@ Details, and the two hard rules that apply to any host node, in `operations/tail
### demo-hp scratch guest — LXC 9202 `demo-hp-scratch` (R-481, built 2026-09-13)
- **Disposition:** the nightly rotation's throwaway host. Under the **demo-hp customer** (same domain,
same dashboard password), **hub reporting OFF, Cloudflare tunnel OFF, agent local API OFF, off-site
a dashboard password of its OWN — `REHEARSAL_2026_10_20_9202_PW` in the CC secret store, named for the 2026-10-20 rehearsal; the demo-hp `PASSWORD` is refused, measured 2026-10-23), **hub reporting OFF, Cloudflare tunnel OFF, agent local API OFF, off-site
OFF, self-update OFF** (the image is set by hand in `/etc/felhom-controller-image`). **Persists across
nights on purpose.** Apps on it are throwaways; nothing on it is a customer promise. Not in the felhom
pool, so the hub never sees it. Tags `scratch,r481`; the same text sits in the guest at
@@ -93,6 +93,17 @@ itself. **Corrected:** one snapshot took 44 s on the live volume (0.63 s on a lo
### Step 4 — the push (on DooPlex, root; `felhom-hub-db-backup.service` + `.timer` 02:30) — DONE 2026-10-05
> **Since 2026-10-23 (D34, operator yes): every 6 hours.** The hub (≥ 0.167.0) writes a snapshot at 02:00, 08:00, 14:00
> and 20:00 Budapest (`dbsnap.Slots`; start-up catch-up when the newest is older than 6 h); the timer pushes at 02:30,
> 08:30, 14:30 and 20:30. The push waits (≤ 20 min) for a snapshot being written and refuses a newest snapshot older
> than **7 h** (one slot gap + 1 h — a 2 h limit refused the hub's own catch-up once and mailed a false failure,
> R-1234). Alarm `HubDBBackupStale` at **13 h** (two missed pushes). **Retention is unchanged and server-side on ep0**
> (`prune-operator-hubdb`, 03:45, keep-daily 14, keep-weekly 8): the current day's copies (up to 4) live until the next
> prune, then the newest of each day for 14 days and one per week for 8 weeks. One copy: ~500 MiB raw, ~25 MiB
> uploaded. Proven 2026-10-23: a hand run, then the newest copy restored from ep0 with the read-only token was
> byte-identical to the pod's snapshot, integrity ok, `SELECT COUNT(*) FROM hosts` = 4
> (`audits/day-2026-10-23/13-hubdb-6h-D34.txt`). The text below is the original nightly design.
**The real script is `scripts/hub-db-backup/felhom-hub-db-backup`** (versioned, R-231; installed by `install.sh`;
15 tests in `test_hub_db_backup.py`, run by hand — not in CI). It adds to the sketch below: it refuses a snapshot older
than 26 h (the hub stopped snapshotting), a copy whose size differs from the pod's file, and a copy with no hosts; it
+35 -23
View File
@@ -1,4 +1,4 @@
# Break-glass sheet — the keys that must exist outside DooPlex (R-923)
# Break-glass sheet — the keys that must exist outside DooPlex (R-923, D32)
> **A template. It holds NO key value, and none may ever be written into this file, a commit, a log or a chat.**
> Print this page, then write or stick each value onto the paper copy only. Keep the paper away from home (DooPlex is
@@ -33,31 +33,43 @@ captured line on some systems; strip it when typing the value back.
**Check a print later without exposing it:** for S1/S2 the only reliable check is one restore with a key file rebuilt
from the printed `data` field (hub-DB runbook Step 0 note) — `key show` cannot check a rebuilt key.
## Public fingerprints — to check you hold the right key (not secret; not on the printed page)
| # | Fingerprint |
|---|---|
| S1 | PBS key `93:03:bf:d7:1f:4c:9e:fe…` (`proxmox-backup-client key show`) |
| S2 | PBS key `b2:19:bf:36:3b:97:3d:6c…` |
| S6a | `SHA256:/ixgTesZqykAGJpFUUd4kLAiHFgKOkYFLNC3AQXWP+k` |
| S6b | `SHA256:7YqN4rXO08yixTeOO+UtQ8jHyIGycICuctQgRYVGnWw` |
| S19 | recipient `age1aw83ej37x9sck89xwt856ry0553kf5sng3w544an4et54yjdx5qq79utrv` |
Every place below was checked to exist on 2026-10-23 (existence only, no value read). The release copy on ep0 (D33)
and the 6-hourly hub-DB copy (D34) add NO key: the release copy is not encrypted (it holds no secret), and the hub-DB
copy uses S2 as before.
---
## The sheet (print from here)
## The sheet (print from here — one A4 page)
**FELHOM — break-glass keys.** Printed on: ____________ Stored at: ______________________________
**FELHOM — vészhelyzeti kulcsok / break-glass keys.** Nyomtatva / printed: ____________ Hol tárolom / kept at: ______________________
| # | Key | Public fingerprint (to match the right key) | Value — write or stick here |
Csak nevek és helyek; az értéket kézzel írd be. *Names and places only; write each value by hand.* Tartsd az otthonon kívül. *Keep it outside the home.*
| # | Név / Name | Hol van most / Where it lives now | Érték / Value |
|---|---|---|---|
| S1 | DooPlex off-site key — opens Gitea, the password manager, the k8s Secrets export (ep0 `operator`, `host/dooplex-gitea`) | PBS key `93:03:bf:d7:1f:4c:9e:fe…` | `data`: ______________________________________________ |
| S2 | Hub-DB off-site key — opens the hub database (ep0 `operator`, `host/dooplex-hub`) | PBS key `b2:19:bf:36:3b:97:3d:6c…` | `data`: ______________________________________________ |
| S4 | Hub seal key `OFFSITE_SECRET_KEY` (64 hex) | — | ______________________________________________________ |
| S5 | DooPlex restic / Secrets-export passphrase | — | ______________________________________________________ |
| S6a | Signing RECOVERY key `felhom-rec-recovery` (also in the S1 copy since 2026-10-09; the paper is the copy that needs nothing else) | `SHA256:/ixgTesZqykAGJpFUUd4kLAiHFgKOkYFLNC3AQXWP+k` | (staple the printout) |
| S6b | Signing OPERATIONAL key `felhom-op-operational` | `SHA256:7YqN4rXO08yixTeOO+UtQ8jHyIGycICuctQgRYVGnWw` | (staple the printout) |
| S7 | ep0 read-only token `dooplex-hub@pbs!restore` | — | ______________________________________________________ |
| S8 | Hetzner account: login e-mail · password · two-factor recovery codes | — | ______________________________________________________ |
| S11 | Cloudflare account: login · password · two-factor recovery codes | — | ______________________________________________________ |
| S12 | Gmail `felhom.eu@gmail.com`: password · two-factor recovery codes | — | ______________________________________________________ |
| S19 | CC secrets age key — opens `admin/cc-secrets` (every secret Claude Code uses on DooPlex); the store is in Gitea, behind S1 | recipient `age1aw83ej37x9sck89xwt856ry0553kf5sng3w544an4et54yjdx5qq79utrv` | `AGE-SECRET-KEY-1`______________________________________ |
| S3 | Vaultwarden master password — **in your head**; write it here only if you decide to | — | ______________________________________________________ |
| S1 | DooPlex off-site kulcs / DooPlex off-site key (Gitea, jelszókezelő / password manager, k8s Secrets) | DooPlex `/etc/felhom-dooplex-offsite/enc.key` | `data`: ____________________________ |
| S2 | Hub-adatbázis off-site kulcs / Hub-DB off-site key | DooPlex `/etc/felhom-hub-backup/enc.key` | `data`: ____________________________ |
| S4 | Hub pecsétkulcs / hub seal key `OFFSITE_SECRET_KEY` | k8s Secret `felhom-system/offsite-secret-key` | __________________________________ |
| S5 | DooPlex restic / Secrets-export jelmondat / passphrase | DooPlex `/etc/backup/restic-password` | __________________________________ |
| S6a | Aláíró HELYREÁLLÍTÓ kulcs / signing RECOVERY key | DooPlex `/mnt/5_hdd/felhom.eu/felhom-rec-recovery` | (tűzd ide / staple) |
| S6b | Aláíró ÜZEMI kulcs / signing OPERATIONAL key | DooPlex `/mnt/5_hdd/felhom.eu/felhom-op-operational` | (tűzd ide / staple) |
| S7 | ep0 csak-olvasó token / read-only token `dooplex-hub@pbs!restore` | DooPlex `/etc/felhom-hub-backup/token-restore` | __________________________________ |
| S8 | Hetzner fiók / account (e-mail · jelszó / password · 2FA kódok / codes) | Vaultwarden | __________________________________ |
| S11 | Cloudflare fiók / account (login · jelszó / password · 2FA kódok / codes) | Vaultwarden | __________________________________ |
| S12 | Gmail `felhom.eu@gmail.com` (jelszó / password · 2FA kódok / codes) | Vaultwarden | __________________________________ |
| S19 | CC secrets age kulcs / key (`admin/cc-secrets`) | DooPlex `~/.config/sops/age/keys.txt`; Vaultwarden | `AGE-SECRET-KEY-1`__________________ |
| S3 | Vaultwarden mesterjelszó / master password | a fejedben / in your head | (csak ha akarod / only if you choose) |
**Not secret, needed with the keys:**
- ep0: `167.233.158.164` (Hetzner, `felhom-hetzner`); PBS datastore `felhom-offsite`, namespace `operator`;
its certificate fingerprint `c6:07:28:3f:5b:7b:5a:41:90:28:d7:ca:4f:37:14:70:56:39:2e:2f:0b:71:e8:06:ca:60:4a:d5:56:5f:3c:fd`.
- Restore order: `total-loss-of-dooplex.md` (in the restored Gitea, repo `felhom.eu`, `documentation/runbooks/`).
**Print that page too** — the runbook itself is inside the copy it explains how to open.
**Re-print when** a key above is rotated, and once a year.
**Nem titok, de kell / not secret, needed:** ep0 = `167.233.158.164` (Hetzner, `felhom-hetzner`), PBS `felhom-offsite`, névtér / namespace `operator`; tanúsítvány / certificate
`c6:07:28:3f:5b:7b:5a:41:90:28:d7:ca:4f:37:14:70:56:39:2e:2f:0b:71:e8:06:ca:60:4a:d5:56:5f:3c:fd`. Kiadások másolata / release copy: ep0 `/srv/felhom-release-copy/data`.
Sorrend / order: `total-loss-of-dooplex.md` — **nyomtasd ki azt is / print it too.** Újranyomtatás / re-print: kulcscserekor és évente / when a key rotates, and yearly.
@@ -19,8 +19,7 @@ build tree. **Assume all of it is gone, and the house with it.**
| The boxes themselves | they keep running and backing up; they cannot report, update or mail until the hub is back | — |
| The operator's workstation (if it has them) | the WireGuard peer to ep0 (`10.77.0.250`), maybe one of ep0's two root SSH keys, maybe a Bitwarden client's offline vault cache | the operator |
**Lost with DooPlex and NOT in any copy** (measured 2026-10-09): the container registry (rebuild the images from
code); *(the operator signing keys were here until 2026-10-09 afternoon; they now ride the nightly copy — R-924)*; DooPlex's own SSH key (one of ep0's two root keys);
**Lost with DooPlex and NOT in any copy** (measured 2026-10-09): *(the container registry was here until 2026-10-23: the released images and agent packages now ride a nightly copy to ep0 — D33, „Restore the release copy" below; the rest of the registry — older tags, rc builds, non-Felhom images — is still lost)*; *(the operator signing keys were here until 2026-10-09 afternoon; they now ride the nightly copy — R-924)*; DooPlex's own SSH key (one of ep0's two root keys);
the build tree and drills (`/mnt/5_hdd/felhom.eu/{build,drills}`); `.claude-memory` (same-disk restic only); the
local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom; Longhorn on DooPlex only).
@@ -66,16 +65,48 @@ local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom
copy, use that. Every later login comes from here.
6. **The Secrets export:** `gpg --batch --passphrase-file <S5 file> -d secrets-<stamp>.yaml.gpg > secrets.yaml` —
every k8s Secret of the old cluster (shred it after use).
7. **Gitea:** `gitea-restore.md` (on a normal network this time). Then rebuild the images from the code into the new
registry: hub, controller, agent (`RUNBOOK-manual-build.md`).
7. **Gitea:** `gitea-restore.md` (on a normal network this time). Then put the released images and agent packages
back into its registry from ep0 — „Restore the release copy" below (same digests; no rebuild). Rebuild from code
(`RUNBOOK-manual-build.md`) only what the copy does not hold.
8. **The hub:** k3s, then `RUNBOOK-hub-db-offsite-backup.md` §3 with S2 and S4 — **start it with mail held**
(§3 step 1: the `MAIL-HOLD` marker — shipped in hub 0.145.0, 2026-10-10).
9. **DNS:** in Cloudflare (S11) point `hub.felhom.eu` (today a CNAME to `dooplex.hopto.org`) and `gitea.dooplex.hu` at
the new place. The boxes reconnect by themselves: their API keys are in the restored hub DB.
10. **Signing:** the keys are in the restored copy (`signing/`) and on the sheet (S6) — put them back and sign as before. Without it, no box accepts an agent update or a
bundle until it is re-enrolled on site (`04` §4).
11. **Backups again:** reinstall `scripts/hub-db-backup/` and `scripts/dooplex-offsite/` from the restored code, the
tunnel and the tokens, and check the next night's copies.
11. **Backups again:** reinstall `scripts/hub-db-backup/`, `scripts/dooplex-offsite/` and `scripts/release-copy/` from
the restored code, the tunnel and the tokens (the release copy needs a new read-only registry token and a new SSH
key in ep0's `felhom-relcopy` `authorized_keys`), and check the next night's copies.
## Restore the release copy (D33, since 2026-10-23)
**What it is.** Every night at 01:40 DooPlex copies the newest 3 release tags of `felhom-controller`, `felhom-hub` and
`felhom-samba`, and the newest 5 agent versions (binary + config bundle), to ep0 `/srv/felhom-release-copy/data`
(`scripts/release-copy/`). Images are in skopeo's `dir:` layout — the registry's own bytes — so a push back gives the
**same digest** as the release. Nothing in it is secret; it is not encrypted. Alarm: `FelhomReleaseCopyStale`.
1. **Get it off ep0** (ep0 root — step 2): `rsync -a root@167.233.158.164:/srv/felhom-release-copy/data/ ./release-copy/`
(or work on ep0 itself). Then `cd release-copy && sha256sum -c --quiet SHA256SUMS` — every line must pass.
2. **Images back into the new registry**, one per tag (log in to the new Gitea first: `skopeo login <gitea host>`):
```bash
for d in images/*/*; do repo=${d#images/}; repo=${repo%%/*}; tag=${d##*/}
skopeo copy --all --preserve-digests "dir:$d" "docker://<gitea host>/admin/$repo:$tag"
echo "$repo:$tag $(skopeo inspect --raw "docker://<gitea host>/admin/$repo:$tag" | sha256sum | cut -c1-64) want $(cut -c8- "$d/DIGEST")"
done
```
The two hashes on each line must be equal.
3. **Agent packages back** (Gitea's generic registry, a token with package write):
```bash
for d in agent/*; do v=${d#agent/}; for f in felhom-agent felhom-config-bundle.json; do
curl -fsS -u admin:"$(cat token-file)" -X PUT --upload-file "$d/$f" \
"https://<gitea host>/api/packages/admin/generic/felhom-agent/$v/$f"; done; done
```
Check each download's sha256 against `agent/<v>/<file>.sha256`.
4. **Keep the registry's name** (`gitea.dooplex.hu`, step 9's DNS) — the boxes pull `gitea.dooplex.hu/admin/…` by name.
**Proven 2026-10-23** (`audits/day-2026-10-23/08-release-copy-D33.txt`): the copy was pulled back from ep0, every
sha256 matched, every image re-verified against its digest, and controller 0.340.0 from it loaded on a scratch box
(image ID = its config digest; a container ran). **Not yet walked:** step 2 into a real new registry, and step 3.
## Walked on paper (2026-10-09): what still needs DooPlex if the operator has only the sheet, memory and a new machine
@@ -90,7 +121,7 @@ local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom
| the hub's seal key | S4 (or S1+S5) | sheet, or via S5 |
| the hub DB | S2 (or Vaultwarden) | sheet, or via S3 |
| sign a box update | S6 | **sheet — today exists on DooPlex only** |
| hub image | the registry (lost) | rebuild from restored code — needs a Go/Docker build machine, no secret |
| hub, controller, samba images; agent packages | the release copy on ep0 (D33, nightly since 2026-10-23) | ep0 root (S8/S9) → „Restore the release copy" — no secret, no rebuild |
| Gitea `main` → `felhom.eu` website, ISO host | Cloudflare (S11), the build tree (lost) | S11 on the sheet; the build tree is rebuilt from code |
| Authentik (S17) | not in any copy | not needed for the business; DooPlex homelab apps that use it are out of scope |
+1
View File
@@ -19,6 +19,7 @@
box's controller). The checker now stands down for that declared state (the wait is the household's, it can take
days); an undeclared consumed shape alarms as before. `TestR1230_AHeldBoxIsNotStuck`, red-proved.
- `scripts/wire_contract_gate.py`: `wireguard.last_handshake_age_s` left the unread list (the hub reads it since 0.166.0).
- **D34 (operator yes 2026-10-23, R-232 G5) — the database snapshot every 6 hours.** The hub writes its `VACUUM INTO` snapshot at 02:00, 08:00, 14:00 and 20:00 Budapest (`dbsnap.Slots`, keep 2 unchanged), and the start-up catch-up fires when the newest is older than 6 h (`dbsnap.CatchUpAge`, was 24 h). DooPlex's push now runs :30 after each slot (`scripts/hub-db-backup`, refuses a newest copy older than 7 h — one slot gap + 1 h — was 26 h; waits for a snapshot being written), so the off-site hub copy is at most about 6.5 h old instead of a day. Added the same day as the build, on the operator's word (not soaked). Tests `TestD34_SlotsAreEverySixHours`, `TestR173_MainSchedulesTheDBSnapshot` (now the loop), and the script's `test_schedules` (the timer read against the hub's slots) and `test_d34_a_missed_slot_is_not_pushed_again` — each red-proved.
## v0.166.0 (unreleased — built 2026-10-22 night, NOT deployed; the deploy is the operator's yes) — the clock-ahead tunnel alarm (R-1204)
+7 -5
View File
@@ -944,9 +944,9 @@ func main() {
}
}()
// Nightly DB snapshot (R-173, v0.136.0, `05` §16.3): VACUUM INTO <data>/snapshots/, keep 2. DooPlex's
// felhom-hub-db-backup.service picks up the newest at 02:30, checks, encrypts and pushes it to ep0. A start-up
// catch-up runs one when the newest is older than 24 h (a pod that was down at 02:00).
// DB snapshot every 6 hours (R-173, v0.136.0, `05` §16.3; D34 v0.167.0): VACUUM INTO <data>/snapshots/, keep 2.
// DooPlex's felhom-hub-db-backup.service picks up the newest at :30 after each slot, checks, encrypts and pushes it
// to ep0. A start-up catch-up runs one when the newest is older than one slot gap (a pod that was down at a slot).
snapMaker := &dbsnap.Maker{Store: dataStore, Dir: filepath.Join(cfg.Server.DataDir, "snapshots"), Logger: logger}
runSnapshot := func() {
sctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
@@ -955,10 +955,12 @@ func main() {
logger.Printf("[ERROR] db snapshot failed: %v", err)
}
}
if dbsnap.NeedsCatchUp(snapMaker.Dir, time.Now(), 24*time.Hour) {
if dbsnap.NeedsCatchUp(snapMaker.Dir, time.Now(), dbsnap.CatchUpAge) {
go runSnapshot()
}
go scheduleDaily(ctx, "db-snapshot", "02:00", runSnapshot, logger)
for _, at := range dbsnap.Slots {
go scheduleDaily(ctx, "db-snapshot "+at, at, runSnapshot, logger)
}
// Backup deadline checker — runs daily at 05:00 Budapest
go scheduleDaily(ctx, "deadline-check", "05:00", func() {
+27 -18
View File
@@ -4,13 +4,13 @@ import (
"go/ast"
"go/parser"
"go/token"
"strconv"
"testing"
)
// R-173 seam wiring: main() must schedule the nightly db snapshot — a scheduleDaily call named "db-snapshot" at
// "02:00", before DooPlex's 02:30 push — and must run the start-up catch-up (dbsnap.NeedsCatchUp). Without them the
// dbsnap package exists and DooPlex pushes nothing new. RED-PROOF: delete either call → this test fails.
// R-173 seam wiring, D34: main() must schedule a db snapshot at every dbsnap.Slots time — a `for _, at := range
// dbsnap.Slots` loop calling scheduleDaily(ctx, <name>, at, …) — and must run the start-up catch-up
// (dbsnap.NeedsCatchUp with dbsnap.CatchUpAge). Without them the dbsnap package exists and DooPlex pushes nothing
// new. RED-PROOF: delete the loop, schedule a literal time instead, or pass 24*time.Hour → this test fails.
func TestR173_MainSchedulesTheDBSnapshot(t *testing.T) {
f, err := parser.ParseFile(token.NewFileSet(), "main.go", nil, 0)
if err != nil {
@@ -18,30 +18,39 @@ func TestR173_MainSchedulesTheDBSnapshot(t *testing.T) {
}
scheduled, catchUp := false, false
ast.Inspect(f, func(n ast.Node) bool {
if r, ok := n.(*ast.RangeStmt); ok {
sel, ok := r.X.(*ast.SelectorExpr)
v, vok := r.Value.(*ast.Ident)
if ok && vok && sel.Sel.Name == "Slots" {
ast.Inspect(r.Body, func(m ast.Node) bool {
c, ok := m.(*ast.CallExpr)
if !ok {
return true
}
if id, ok := c.Fun.(*ast.Ident); ok && id.Name == "scheduleDaily" && len(c.Args) >= 3 {
if a, ok := c.Args[2].(*ast.Ident); ok && a.Name == v.Name {
scheduled = true
}
}
return true
})
}
}
c, ok := n.(*ast.CallExpr)
if !ok {
return true
}
if id, ok := c.Fun.(*ast.Ident); ok && id.Name == "scheduleDaily" && len(c.Args) >= 3 {
name, _ := c.Args[1].(*ast.BasicLit)
at, _ := c.Args[2].(*ast.BasicLit)
if name != nil && at != nil {
n, _ := strconv.Unquote(name.Value)
a, _ := strconv.Unquote(at.Value)
if n == "db-snapshot" && a == "02:00" {
scheduled = true
}
if sel, ok := c.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "NeedsCatchUp" && len(c.Args) == 3 {
if a, ok := c.Args[2].(*ast.SelectorExpr); ok && a.Sel.Name == "CatchUpAge" {
catchUp = true
}
}
if sel, ok := c.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "NeedsCatchUp" {
catchUp = true
}
return true
})
if !scheduled {
t.Error(`cmd/hub/main.go never calls scheduleDaily(ctx, "db-snapshot", "02:00", …) — no nightly snapshot`)
t.Error("cmd/hub/main.go never schedules a snapshot at each dbsnap.Slots time — no 6-hourly snapshot (D34)")
}
if !catchUp {
t.Error("cmd/hub/main.go never calls dbsnap.NeedsCatchUp — a pod down at 02:00 leaves a stale snapshot")
t.Error("cmd/hub/main.go never calls dbsnap.NeedsCatchUp(…, dbsnap.CatchUpAge) — a pod down at a slot leaves a stale snapshot")
}
}
+9
View File
@@ -26,6 +26,15 @@ import (
// grown to 2 GiB for it, operator choice 2026-10-05).
const Keep = 2
// Slots are the Budapest times the hub writes a snapshot — every 6 hours (D34, operator yes 2026-10-23, R-232 G5).
// DooPlex's felhom-hub-db-backup.timer pushes the newest at :30 after each slot, so the off-site copy is at most
// about 6.5 h old instead of a day. Pinned by TestD34_SlotsAreEverySixHours and cmd/hub TestR173_MainSchedulesTheDBSnapshot.
var Slots = []string{"02:00", "08:00", "14:00", "20:00"}
// CatchUpAge is the start-up catch-up limit: a pod that was down at a slot writes one at start when the newest is
// older than one slot gap. Pinned by TestD34_SlotsAreEverySixHours.
const CatchUpAge = 6 * time.Hour
// Prefix and Suffix make a snapshot's file name: hub-20261005T020000Z.db.
const (
Prefix = "hub-"
+40
View File
@@ -206,3 +206,43 @@ func TestNeedsCatchUp(t *testing.T) {
t.Error("25 h old: want catch-up")
}
}
// D34 (operator yes 2026-10-23): a snapshot every 6 hours. The consequence pinned: no gap between two slots — the
// wrap past midnight included — is longer than CatchUpAge, so the off-site copy (pushed :30 after each slot) is never
// more than one slot gap behind; 02:00 stays a slot (the nightly restore test and the runbook read it); the start-up
// catch-up fires for a pod that missed exactly one slot. RED-PROOF: drop a slot, or set CatchUpAge to 24 h → fails.
func TestD34_SlotsAreEverySixHours(t *testing.T) {
if len(Slots) != 4 {
t.Fatalf("Slots = %v, want 4 (every 6 h)", Slots)
}
var mins []int
has0200 := false
for _, s := range Slots {
tm, err := time.Parse("15:04", s)
if err != nil {
t.Fatalf("slot %q: %v", s, err)
}
mins = append(mins, tm.Hour()*60+tm.Minute())
has0200 = has0200 || s == "02:00"
}
if !has0200 {
t.Error("02:00 is not a slot")
}
for i := range mins {
next := mins[(i+1)%len(mins)]
if i == len(mins)-1 {
next += 24 * 60
}
if gap := time.Duration(next-mins[i]) * time.Minute; gap <= 0 || gap > CatchUpAge {
t.Errorf("gap after %s is %s (want 0 < gap <= %s)", Slots[i], gap, CatchUpAge)
}
}
dir := t.TempDir()
_ = os.WriteFile(filepath.Join(dir, "hub-20261023T000000Z.db"), nil, 0o600) // 02:00 Budapest (CEST)
if !NeedsCatchUp(dir, time.Date(2026, 10, 23, 7, 0, 1, 0, time.UTC), CatchUpAge) {
t.Error("a pod up at 09:00 CEST that missed the 08:00 slot (newest 7 h old): want catch-up")
}
if NeedsCatchUp(dir, time.Date(2026, 10, 23, 5, 0, 0, 0, time.UTC), CatchUpAge) {
t.Error("newest 5 h old: want no catch-up")
}
}
+1 -1
View File
@@ -128,7 +128,7 @@ spec:
spec:
containers:
- name: hub
image: gitea.dooplex.hu/admin/felhom-hub:0.165.0
image: gitea.dooplex.hu/admin/felhom-hub:0.167.0
ports:
- containerPort: 8080
name: http
+2 -2
View File
@@ -344,7 +344,7 @@ spec:
image: registry.k8s.io/git-sync/git-sync:v4.4.0
args:
- --repo=http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git
- --ref=installer-v1.35.0
- --ref=installer-v1.36.0
- --root=/git-scripts
- --link=current
- --period=30s
@@ -390,7 +390,7 @@ spec:
image: registry.k8s.io/git-sync/git-sync:v4.4.0
args:
- --repo=http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git
- --ref=installer-v1.35.0
- --ref=installer-v1.36.0
- --root=/git-scripts
- --link=current
- --one-time
+5
View File
@@ -1,3 +1,8 @@
## 2026-10-23 — `release-copy/`: the released images and agent packages leave DooPlex nightly (D33, R-1227); `hub-db-backup/`: every 6 hours (D34)
- **`release-copy/felhom-release-copy`** (new, DooPlex root, timer 01:40): the newest 3 release tags of felhom-controller, felhom-hub and felhom-samba (`skopeo copy --all --preserve-digests` into `dir:` — the registry's own bytes, so a push back gives the SAME digest; a `docker save` tar would not) and the newest 5 agent versions (binary + config bundle). Before anything leaves: the top manifest hashes to the registry's digest, every blob and child manifest to its name, every named blob is present, each agent file matches the registry's sha256; the staged tree ≤ the cap (3072 MB). Then `rsync --delete` to ep0's limited user `felhom-relcopy` (rrsync, one directory), and a `--checksum --dry-run` must show no difference before the success signal. Spoiled tags (0.315.0, 0.316.3) are never copied. 13 tests (`test_release_copy.py`), four red-proved.
- **`hub-db-backup/`** — the push runs at 02:30, 08:30, 14:30 and 20:30 (the hub snapshots every 6 hours since hub 0.167.0) and refuses a newest snapshot older than 7 h (one slot gap + 1 h; was 26 h), and waits (up to 20 min) while the hub is writing one. **Corrected the same day:** 2 h was shipped first, and its first run refused the timer's own catch-up while the hub's start-up snapshot was still being written — one false failure mail to admin@ at 10:28 (R-1234). Tests: the timer read against the hub's slots; two missed slots refused; a restart within one slot pushed; a snapshot being written waited for — each red-proved (the last reproduces the 10:28 run).
## 2026-10-23 — felhom-host-install.sh 1.36.0 (pushed; the tag `installer-v1.36.0` waits for the operator's yes): the uninstall removes the root-SSH drop-in (R-982, D4)
The agent's config bundle (next agent release) gives the host's own sshd `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`
+17 -3
View File
@@ -1,6 +1,7 @@
#!/bin/sh
# felhom-hub-db-backup — push the hub's newest nightly snapshot to ep0's PBS, encrypted (R-173, decision A).
# Runs on DooPlex as root from felhom-hub-db-backup.timer (02:30; the hub writes the snapshot at 02:00).
# felhom-hub-db-backup — push the hub's newest snapshot to ep0's PBS, encrypted (R-173, decision A).
# Runs on DooPlex as root from felhom-hub-db-backup.timer (02:30, 08:30, 14:30, 20:30; the hub writes a snapshot at
# 02:00, 08:00, 14:00 and 20:00 since hub 0.167.0 — D34).
# Runbook: documentation/runbooks/RUNBOOK-hub-db-offsite-backup.md Step 4. Pinned by test_hub_db_backup.py.
#
# Refuses to push — and so never writes the success signal — when: no snapshot exists; the newest is older than
@@ -11,7 +12,13 @@ set -eu
CONF=${FELHOM_HUBBK_CONF:-/etc/felhom-hub-backup}
STATE=${FELHOM_HUBBK_STATE:-/var/lib/felhom-hub-backup}
TEXTFILE_DIR=${FELHOM_HUBBK_TEXTFILE_DIR:-/var/lib/node_exporter/textfile_collector}
MAX_AGE_H=${FELHOM_HUBBK_MAX_AGE_H:-26}
# D34: one slot gap + 1 h. The hub writes a catch-up snapshot at start when its newest is older than 6 h
# (dbsnap.CatchUpAge), so after any restart the newest is either < 7 h old or being written (waited for below). A hub
# that really stopped is refused at the second missed slot. (2 h was tried first, 2026-10-23: it refused the timer's own
# catch-up run while the hub's catch-up snapshot was still being written — a false "stopped snapshotting" mail.)
MAX_AGE_H=${FELHOM_HUBBK_MAX_AGE_H:-7}
WAIT_S=${FELHOM_HUBBK_WAIT_S:-1200}
POLL_S=${FELHOM_HUBBK_POLL_S:-15}
NOW=${FELHOM_HUBBK_NOW:-$(date +%s)}
. "$CONF/env" # PBS_REPOSITORY_PUSH, PBS_FINGERPRINT (no secrets in this file)
@@ -24,6 +31,13 @@ mkdir -p "$STAGE"; chmod 700 "$STATE" "$STAGE"
rm -f "$STAGE"/*
trap 'if [ -f "$STAGE/hub.db" ]; then shred -u "$STAGE/hub.db" 2>/dev/null || rm -f "$STAGE/hub.db"; fi' EXIT
# A snapshot being written (hub-*.db.tmp) is waited for, so a push never takes the older one meanwhile (D34).
WAITED=0
while kubectl -n felhom-system exec deploy/hub -- sh -c 'ls /data/snapshots/hub-*.db.tmp >/dev/null 2>&1'; do
[ "$WAITED" -ge "$WAIT_S" ] && { log "a snapshot is still being written after ${WAIT_S} s — taking the newest finished one"; break; }
[ "$WAITED" -eq 0 ] && log "a snapshot is being written in the hub pod — waiting for it (up to ${WAIT_S} s)"
sleep "$POLL_S"; WAITED=$((WAITED + POLL_S))
done
SNAP=$(kubectl -n felhom-system exec deploy/hub -- sh -c 'ls -1 /data/snapshots/hub-*.db 2>/dev/null | tail -n 1') || die "listing snapshots in the hub pod"
[ -n "$SNAP" ] || die "no snapshot in the hub pod's /data/snapshots"
NAME=${SNAP##*/}
@@ -1,9 +1,13 @@
# Versioned in felhom.eu/scripts/hub-db-backup/ (R-231); installed by install.sh.
[Unit]
Description=Felhom: push the hub DB snapshot to ep0 (R-173) — schedule
Description=Felhom: push the hub DB snapshot to ep0 (R-173, D34) — every 6 hours
[Timer]
# D34 (operator yes 2026-10-23): every 6 h, :30 after each hub snapshot slot (hub dbsnap.Slots 02/08/14/20:00).
OnCalendar=*-*-* 02:30:00
OnCalendar=*-*-* 08:30:00
OnCalendar=*-*-* 14:30:00
OnCalendar=*-*-* 20:30:00
Persistent=true
RandomizedDelaySec=2min
+45 -1
View File
@@ -8,6 +8,7 @@ was written. Run: python3 scripts/hub-db-backup/test_hub_db_backup.py
"""
import json
import os
import re
import shutil
import sqlite3
import stat
@@ -184,6 +185,42 @@ class Push(Base):
self.assertEqual(self.pushed(), [])
self.assertFalse(self.signal("felhom_hub_db_backup.prom"))
def test_d34_two_missed_slots_are_refused(self):
# D34: snapshots every 6 h; the limit is one slot gap + 1 h (7 h) — a newest copy 8 h old means the hub missed
# two slots, or stopped. RED-PROOF: MAX_AGE_H back to 26 → this pushes.
self.snapshot(age_s=8 * 3600)
r = self.run_script(PUSH)
self.assertNotEqual(r.returncode, 0)
self.assertIn("stopped snapshotting", r.stderr)
self.assertEqual(self.pushed(), [])
self.assertFalse(self.signal("felhom_hub_db_backup.prom"))
def test_d34_a_restart_within_one_slot_is_not_an_alarm(self):
# the hub's catch-up fires only past 6 h, so after a restart the newest can be up to 6 h old — that is pushed,
# not refused (2 h refused it: the false mail of 2026-10-23 10:28). RED-PROOF: MAX_AGE_H 2 → this fails.
self.snapshot(age_s=int(5.9 * 3600))
r = self.run_script(PUSH)
self.assertEqual(r.returncode, 0, r.stderr)
self.assertEqual(len(self.pushed()), 1)
def test_d34_waits_for_a_snapshot_being_written(self):
# the 2026-10-23 10:28 case: the hub's catch-up snapshot is being written (hub-*.db.tmp) and the only finished
# one is 8 h old. The push must wait for the new one, not refuse the old. RED-PROOF: drop the wait loop → fails.
import threading
self.snapshot(age_s=8 * 3600)
snaps = self.j("pod", "snapshots")
tmp = os.path.join(snaps, "hub-%s.db.tmp" % stamp(int(time.time())))
src = [os.path.join(snaps, n) for n in os.listdir(snaps) if n.endswith(".db")][0]
shutil.copy(src, tmp)
def finish():
time.sleep(1.5)
os.rename(tmp, tmp[: -len(".tmp")])
threading.Thread(target=finish).start()
r = self.run_script(PUSH, FELHOM_HUBBK_POLL_S="1", FELHOM_HUBBK_WAIT_S="20")
self.assertEqual(r.returncode, 0, r.stderr)
self.assertIn("being written", r.stdout)
self.assertEqual(len(self.pushed()), 1)
def test_truncated_copy_is_not_pushed(self):
self.snapshot()
r = self.run_script(PUSH, FAKE_TRUNCATE="1")
@@ -268,7 +305,14 @@ class Units(unittest.TestCase):
return open(os.path.join(HERE, n)).read()
def test_schedules(self):
self.assertIn("OnCalendar=*-*-* 02:30:00", self.read("felhom-hub-db-backup.timer"))
# D34: the push runs :30 after EVERY hub snapshot slot — read from the hub's own source, so a slot moved on
# one side only fails here (RED-PROOF: drop one OnCalendar line, or change a hub slot → fails).
src = open(os.path.join(HERE, "..", "..", "hub", "internal", "dbsnap", "dbsnap.go")).read()
slots = re.findall(r'"(\d\d):(\d\d)"', re.search(r"var Slots = \[\]string\{([^}]*)\}", src).group(1))
self.assertEqual(len(slots), 4)
want = sorted("OnCalendar=*-*-* %02d:%02d:00" % (int(h), int(m) + 30) for h, m in slots)
got = sorted(l.strip() for l in self.read("felhom-hub-db-backup.timer").splitlines() if l.startswith("OnCalendar="))
self.assertEqual(got, want)
self.assertIn("OnCalendar=Sun *-*-* 04:30:00", self.read("felhom-hub-db-restore-test.timer"))
def test_units_run_the_installed_scripts(self):
+300
View File
@@ -0,0 +1,300 @@
#!/usr/bin/env python3
"""felhom-release-copy — copy the released images and agent packages off DooPlex to ep0, nightly (D33, R-1227).
Runs on DooPlex as root from felhom-release-copy.timer. Runbook: documentation/runbooks/total-loss-of-dooplex.md
(„Restore the release copy"). Pinned by test_release_copy.py.
WHAT. The newest KEEP release tags (X.Y.Z only) of each image in REPOS, and the newest AGENT_KEEP agent versions (the
binary and its config bundle). After a DooPlex loss these are what every box update, reinstall and new install pulls.
HOW, and why skopeo, not `docker save`. Each image tag is copied with `skopeo copy --all --preserve-digests` into
skopeo's `dir:` layout: the registry's own manifest bytes and compressed layer blobs, unchanged. Pushed back into any
registry, the image has THE SAME DIGEST as the release (a `docker save` tar is re-compressed on the next push, so the
digest — what the „a version is pushed once" rule and the hub's spoiled-tag list are about — would differ). It also
needs no Docker daemon on DooPlex.
CHECKS, before anything leaves: the sha256 of the copied top manifest equals the registry's digest; every blob and
child manifest hashes to its own name; every blob a manifest names is present; each agent file's sha256 equals the
registry's record. The staged tree must be ≤ CAP_MB (the cap — the sender enforces it; ep0 has no quota). Then
`rsync --delete` to ep0 (the limited user `felhom-relcopy`, rrsync-confined to one directory), and a second
`rsync --checksum --dry-run` must report NO difference. Only then is the success signal written (CLAUDE.md „presence
is not success"). Versions beyond KEEP / AGENT_KEEP are deleted locally, and so on ep0 by `--delete`.
"""
import base64
import hashlib
import json
import os
import re
import shutil
import subprocess
import sys
import tempfile
import time
import urllib.request
CONF = os.environ.get("FELHOM_RC_CONF", "/etc/felhom-release-copy")
STATE = os.environ.get("FELHOM_RC_STATE", "/var/lib/felhom-release-copy")
TEXTFILE_DIR = os.environ.get("FELHOM_RC_TEXTFILE_DIR", "/var/lib/node_exporter/textfile_collector")
REGISTRY = "gitea.dooplex.hu"
OWNER = "admin"
SPOILED = {"felhom-controller": {"0.315.0", "0.316.3"}} # withdrawn builds (R-1085) — never offered, never copied
AGENT_FILES = ("felhom-agent", "felhom-config-bundle.json")
HEX64 = re.compile(r"^[0-9a-f]{64}$")
RELEASE = re.compile(r"^\d+\.\d+\.\d+$")
class Refused(Exception):
pass
def log(msg):
print("felhom-release-copy: " + msg, flush=True)
def load_env(path):
env = {}
for line in open(path, encoding="utf-8"):
line = line.strip()
if line and not line.startswith("#") and "=" in line:
k, v = line.split("=", 1)
env[k.strip()] = v.strip().strip("'\"")
return env
def newest(versions, n, spoiled=()):
rel = [v for v in versions if RELEASE.match(v) and v not in spoiled]
rel.sort(key=lambda v: tuple(int(x) for x in v.split(".")))
return rel[-n:] if n > 0 else []
def sha256_file(path):
h = hashlib.sha256()
with open(path, "rb") as f:
for chunk in iter(lambda: f.read(1 << 20), b""):
h.update(chunk)
return h.hexdigest()
def verify_image_dir(d, want_digest):
"""Refuses unless the dir: copy in d is exactly the image want_digest names (see the module docstring)."""
top = os.path.join(d, "manifest.json")
if not os.path.isfile(top):
raise Refused(f"{d}: no manifest.json")
got = "sha256:" + sha256_file(top)
if got != want_digest:
raise Refused(f"{d}: manifest digest {got} != registry {want_digest}")
names = set(os.listdir(d))
manifests = [top]
for n in sorted(names):
p = os.path.join(d, n)
if HEX64.match(n):
if sha256_file(p) != n:
raise Refused(f"{d}: blob {n} does not hash to its name")
elif n.endswith(".manifest.json") and HEX64.match(n[: -len(".manifest.json")]):
if sha256_file(p) != n[: -len(".manifest.json")]:
raise Refused(f"{d}: child manifest {n} does not hash to its name")
manifests.append(p)
for m in manifests:
doc = json.load(open(m, encoding="utf-8"))
for ref in doc.get("manifests", []):
hx = ref["digest"].split(":", 1)[1]
if hx + ".manifest.json" not in names:
raise Refused(f"{d}: child manifest {hx[:12]} named by the index is missing")
for ref in ([doc["config"]] if "config" in doc else []) + doc.get("layers", []):
hx = ref["digest"].split(":", 1)[1]
if hx not in names:
raise Refused(f"{d}: blob {hx[:12]} named by a manifest is missing")
def tree_bytes(root):
total = 0
for dp, _, fs in os.walk(root):
for f in fs:
total += os.lstat(os.path.join(dp, f)).st_size
return total
def prune(root, keep):
"""Removes every directory under root whose name is not in keep; returns the removed names."""
gone = []
if os.path.isdir(root):
for n in sorted(os.listdir(root)):
if n not in keep:
shutil.rmtree(os.path.join(root, n))
gone.append(n)
return gone
class Registry:
def __init__(self, user, token):
self.auth = "Basic " + base64.b64encode(f"{user}:{token}".encode()).decode()
def get(self, url, raw=False):
r = urllib.request.urlopen(urllib.request.Request(url, headers={"Authorization": self.auth}), timeout=120)
return r.read() if raw else json.load(r)
def agent_versions(self):
out, page = [], 1
while True:
batch = self.get(f"https://{REGISTRY}/api/v1/packages/{OWNER}?type=generic&q=felhom-agent&limit=50&page={page}")
if not batch:
return out
out += [p["version"] for p in batch if p.get("name") == "felhom-agent"]
page += 1
def agent_files(self, ver):
return {f["name"]: f["sha256"] for f in self.get(
f"https://{REGISTRY}/api/v1/packages/{OWNER}/generic/felhom-agent/{ver}/files")}
def agent_download(self, ver, name, dest):
with open(dest, "wb") as f:
f.write(self.get(f"https://{REGISTRY}/api/packages/{OWNER}/generic/felhom-agent/{ver}/{name}", raw=True))
def run(cmd, **kw):
return subprocess.run(cmd, check=True, text=True, capture_output=True, **kw)
def skopeo(authfile, *args):
return run(["skopeo", *args[:1], "--authfile", authfile, *args[1:]]).stdout
def copy_images(reg_authfile, stage, repos, keep):
images = os.path.join(stage, "images")
os.makedirs(images, exist_ok=True)
copied = []
for repo in repos:
tags = json.loads(skopeo(reg_authfile, "list-tags", f"docker://{REGISTRY}/{OWNER}/{repo}"))["Tags"]
want = newest(tags, keep, SPOILED.get(repo, ()))
if not want:
raise Refused(f"{repo}: no release tag")
rdir = os.path.join(images, repo)
os.makedirs(rdir, exist_ok=True)
for tag in want:
ref = f"docker://{REGISTRY}/{OWNER}/{repo}:{tag}"
digest = "sha256:" + hashlib.sha256(
skopeo(reg_authfile, "inspect", "--raw", ref).encode()).hexdigest()
d = os.path.join(rdir, tag)
if os.path.isfile(os.path.join(d, "DIGEST")) and open(os.path.join(d, "DIGEST")).read().strip() == digest:
verify_image_dir(d, digest) # already here — re-checked every night, not trusted
continue
tmp = d + ".tmp"
shutil.rmtree(tmp, ignore_errors=True)
shutil.rmtree(d, ignore_errors=True)
skopeo(reg_authfile, "copy", "--all", "--preserve-digests", "--quiet", ref, f"dir:{tmp}")
verify_image_dir(tmp, digest)
with open(os.path.join(tmp, "DIGEST"), "w") as f:
f.write(digest + "\n")
os.rename(tmp, d)
copied.append(f"{repo}:{tag}")
gone = prune(rdir, set(want))
log(f"{repo}: keeping {', '.join(want)}" + (f"; removed {', '.join(gone)}" if gone else ""))
prune(images, set(repos))
return copied
def copy_agents(reg, stage, keep):
adir = os.path.join(stage, "agent")
os.makedirs(adir, exist_ok=True)
want = newest(reg.agent_versions(), keep)
if not want:
raise Refused("felhom-agent: no release version")
for ver in want:
d = os.path.join(adir, ver)
os.makedirs(d, exist_ok=True)
recorded = reg.agent_files(ver)
for name in AGENT_FILES:
if name not in recorded:
raise Refused(f"felhom-agent {ver}: the registry has no {name}")
p = os.path.join(d, name)
if not (os.path.isfile(p) and sha256_file(p) == recorded[name]):
reg.agent_download(ver, name, p + ".tmp")
if sha256_file(p + ".tmp") != recorded[name]:
os.remove(p + ".tmp")
raise Refused(f"felhom-agent {ver}/{name}: the download does not match the registry's sha256")
os.rename(p + ".tmp", p)
with open(os.path.join(d, name + ".sha256"), "w") as f:
f.write(f"{recorded[name]} {name}\n")
gone = prune(adir, set(want))
log(f"felhom-agent: keeping {', '.join(want)}" + (f"; removed {', '.join(gone)}" if gone else ""))
def write_sums(stage):
lines = []
for dp, _, fs in os.walk(stage):
for f in fs:
if f == "SHA256SUMS":
continue
p = os.path.join(dp, f)
lines.append(f"{sha256_file(p)} {os.path.relpath(p, stage)}")
with open(os.path.join(stage, "SHA256SUMS"), "w") as f:
f.write("\n".join(sorted(lines, key=lambda l: l[66:])) + "\n")
def rsync_to(env, stage, dry_check=False):
ssh = (f"ssh -i {CONF}/id_ed25519 -o UserKnownHostsFile={CONF}/known_hosts -o StrictHostKeyChecking=yes "
f"-o BatchMode=yes -o ConnectTimeout=30")
cmd = ["rsync", "-a", "--delete", "-e", ssh]
if dry_check:
cmd += ["--checksum", "--dry-run", "--itemize-changes"]
return run(cmd + [stage.rstrip("/") + "/", env["REMOTE"] + ":./"]).stdout
def signal(nbytes):
tmp = os.path.join(TEXTFILE_DIR, f"felhom_release_copy.prom.{os.getpid()}")
with open(tmp, "w") as f:
f.write("# HELP felhom_release_copy_last_success_timestamp_seconds Last verified copy of the releases to ep0 (D33).\n"
"# TYPE felhom_release_copy_last_success_timestamp_seconds gauge\n"
f"felhom_release_copy_last_success_timestamp_seconds {int(time.time())}\n"
f"felhom_release_copy_last_success_bytes {nbytes}\n")
os.chmod(tmp, 0o644)
os.rename(tmp, os.path.join(TEXTFILE_DIR, "felhom_release_copy.prom"))
def main():
env = load_env(os.path.join(CONF, "env"))
keep, akeep, cap = int(env.get("KEEP", "3")), int(env.get("AGENT_KEEP", "5")), int(env["CAP_MB"])
repos = env["REPOS"].split()
token = open(os.path.join(CONF, "registry-token"), encoding="utf-8").read().strip()
user = env.get("REGISTRY_USER", OWNER)
stage = os.path.join(STATE, "stage")
os.makedirs(stage, exist_ok=True)
os.chmod(STATE, 0o700)
with tempfile.TemporaryDirectory(dir=STATE) as td:
authfile = os.path.join(td, "auth.json")
fd = os.open(authfile, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
with os.fdopen(fd, "w") as f:
json.dump({"auths": {REGISTRY: {"auth": base64.b64encode(f"{user}:{token}".encode()).decode()}}}, f)
start = time.time()
copied = copy_images(authfile, stage, repos, keep)
copy_agents(Registry(user, token), stage, akeep)
for n in os.listdir(stage):
if n not in ("images", "agent", "SHA256SUMS", "COPIED_AT"):
p = os.path.join(stage, n)
shutil.rmtree(p) if os.path.isdir(p) else os.remove(p)
with open(os.path.join(stage, "COPIED_AT"), "w") as f:
f.write(time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + "\n")
write_sums(stage)
nbytes = tree_bytes(stage)
if nbytes > cap * 1024 * 1024:
raise Refused(f"the copy is {nbytes // (1024 * 1024)} MB, above the cap {cap} MB — nothing sent")
log(f"staged and verified: {nbytes // (1024 * 1024)} MB, new this run: {', '.join(copied) or 'none'}")
rsync_to(env, stage)
diff = rsync_to(env, stage, dry_check=True).strip()
if diff:
raise Refused("ep0's copy differs from the staged one after the push: " + diff.splitlines()[0])
log(f"pushed to ep0 and compared by checksum: no difference ({int(time.time() - start)} s)")
signal(nbytes)
log("success signal written")
if __name__ == "__main__":
try:
main()
except Refused as e:
print(f"felhom-release-copy: FAILED: {e}", file=sys.stderr)
sys.exit(1)
except subprocess.CalledProcessError as e:
print(f"felhom-release-copy: FAILED: {e.cmd[0]} {e.cmd[1] if len(e.cmd) > 1 else ''} rc={e.returncode}: "
f"{(e.stderr or '').strip()[:300]}", file=sys.stderr)
sys.exit(1)
@@ -0,0 +1,19 @@
# Versioned in felhom.eu/scripts/release-copy/ (D33, R-1227); installed by install.sh. Runbook: total-loss-of-dooplex.md.
[Unit]
Description=Felhom: copy the released images and agent packages to ep0 (D33)
Wants=network-online.target
After=network-online.target
OnFailure=felhom-backup-failmail@%n.service
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/felhom-release-copy
UMask=0077
TimeoutStartSec=2h
Nice=10
IOSchedulingClass=idle
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/felhom-release-copy /var/lib/node_exporter/textfile_collector
NoNewPrivileges=yes
@@ -0,0 +1,12 @@
# Versioned in felhom.eu/scripts/release-copy/ (D33, R-1227); installed by install.sh.
[Unit]
Description=Felhom: copy the released images and agent packages to ep0 (D33) — nightly
[Timer]
# 01:40: after the 00:20 off-site push, before the 02:30 hub DB push and the 03:00 local backup.
OnCalendar=*-*-* 01:40:00
Persistent=true
RandomizedDelaySec=5min
[Install]
WantedBy=timers.target
+30
View File
@@ -0,0 +1,30 @@
#!/bin/sh
# install.sh — install the release copy job on DooPlex (D33, R-1227). Root. Idempotent.
# Installs the script and two units, writes /etc/felhom-release-copy/env (no secrets) when absent, and does NOT enable
# the timer — enable it by hand after the first manual run: systemctl enable --now felhom-release-copy.timer
# The registry token (registry-token), the SSH key (id_ed25519) and ep0's host key (known_hosts) are created
# separately, file to file, never by this script (runbook total-loss-of-dooplex.md, „The release copy").
set -eu
HERE=$(cd "$(dirname "$0")" && pwd)
[ "$(id -u)" = 0 ] || { echo "install.sh: run as root" >&2; exit 1; }
install -m 0755 "$HERE/felhom-release-copy" /usr/local/sbin/felhom-release-copy
for u in felhom-release-copy.service felhom-release-copy.timer; do
install -m 0644 "$HERE/$u" "/etc/systemd/system/$u"
done
install -d -m 0700 /etc/felhom-release-copy /var/lib/felhom-release-copy
if [ ! -f /etc/felhom-release-copy/env ]; then
umask 077
cat > /etc/felhom-release-copy/env <<'ENV'
# Not secret. The token is in registry-token (0600), the SSH key in id_ed25519 (0600).
REMOTE='felhom-relcopy@167.233.158.164'
REGISTRY_USER='admin'
# the newest KEEP release tags of each image; the newest AGENT_KEEP agent versions (the vouched one can lag the newest)
REPOS='felhom-controller felhom-hub felhom-samba'
KEEP=3
AGENT_KEEP=5
# the cap: the sender refuses to send a bigger copy (ep0's / has ~14 GB free; ep0 has no quota)
CAP_MB=3072
ENV
fi
systemctl daemon-reload
echo "install.sh: installed; timer NOT enabled (see the header)"
+165
View File
@@ -0,0 +1,165 @@
#!/usr/bin/env python3
"""Tests for felhom-release-copy (D33). No network, no registry, no ep0: a synthetic skopeo `dir:` copy, and the
end-to-end refusals through main() with the copy steps replaced. Run: python3 test_release_copy.py"""
import hashlib
import importlib.machinery
import importlib.util
import json
import os
import tempfile
import unittest
from unittest import mock
HERE = os.path.dirname(os.path.abspath(__file__))
_loader = importlib.machinery.SourceFileLoader("rc", os.path.join(HERE, "felhom-release-copy"))
_spec = importlib.util.spec_from_loader("rc", _loader)
rc = importlib.util.module_from_spec(_spec)
_loader.exec_module(rc)
def h(b):
return hashlib.sha256(b).hexdigest()
def make_image(d):
"""A two-level image like gitea's: an OCI index → one platform manifest → config + one layer. Returns the digest."""
os.makedirs(d)
layer, config = b"layer-bytes", b'{"config":1}'
child = json.dumps({"schemaVersion": 2, "config": {"digest": "sha256:" + h(config)},
"layers": [{"digest": "sha256:" + h(layer)}]}).encode()
index = json.dumps({"schemaVersion": 2, "manifests": [{"digest": "sha256:" + h(child)}]}).encode()
for name, data in ((h(layer), layer), (h(config), config), (h(child) + ".manifest.json", child),
("manifest.json", index), ("version", b"Directory Transport Version: 1.1\n")):
open(os.path.join(d, name), "wb").write(data)
return "sha256:" + h(index)
class Verify(unittest.TestCase):
def setUp(self):
self.t = tempfile.mkdtemp()
self.d = os.path.join(self.t, "img")
self.digest = make_image(self.d)
def test_a_true_copy_passes(self):
rc.verify_image_dir(self.d, self.digest)
def test_a_changed_layer_is_refused(self):
# RED-PROOF: drop the per-blob hash check in verify_image_dir → this passes silently
blob = [n for n in os.listdir(self.d) if rc.HEX64.match(n)][0]
with open(os.path.join(self.d, blob), "ab") as f:
f.write(b"x")
with self.assertRaisesRegex(rc.Refused, "does not hash to its name"):
rc.verify_image_dir(self.d, self.digest)
def test_another_image_is_refused(self):
with self.assertRaisesRegex(rc.Refused, "!= registry"):
rc.verify_image_dir(self.d, "sha256:" + "0" * 64)
def test_a_missing_layer_is_refused(self):
# RED-PROOF: drop the „named by a manifest is missing" check → a copy without its layer passes
layer = [n for n in os.listdir(self.d) if rc.HEX64.match(n)
and open(os.path.join(self.d, n), "rb").read() == b"layer-bytes"][0]
os.remove(os.path.join(self.d, layer))
with self.assertRaisesRegex(rc.Refused, "missing"):
rc.verify_image_dir(self.d, self.digest)
def test_a_missing_child_manifest_is_refused(self):
child = [n for n in os.listdir(self.d) if n.endswith(".manifest.json")][0]
os.remove(os.path.join(self.d, child))
with self.assertRaisesRegex(rc.Refused, "child manifest .* missing"):
rc.verify_image_dir(self.d, self.digest)
class Select(unittest.TestCase):
def test_newest_releases_by_number_not_text(self):
tags = ["0.9.0", "0.10.0", "0.340.0", "0.339.0", "0.340.0-rc.8", "latest", "0.99.1"]
self.assertEqual(rc.newest(tags, 3), ["0.99.1", "0.339.0", "0.340.0"])
def test_spoiled_tags_are_never_copied(self):
self.assertEqual(rc.newest(["0.314.0", "0.315.0", "0.316.3"], 3, rc.SPOILED["felhom-controller"]), ["0.314.0"])
def test_prune_deletes_beyond_keep(self):
t = tempfile.mkdtemp()
for v in ("0.1.0", "0.2.0", "0.3.0"):
os.makedirs(os.path.join(t, v))
self.assertEqual(rc.prune(t, {"0.3.0"}), ["0.1.0", "0.2.0"])
self.assertEqual(os.listdir(t), ["0.3.0"])
class Main(unittest.TestCase):
"""main() with the registry copies replaced: the cap and the post-push comparison gate the success signal."""
def setUp(self):
self.t = tempfile.mkdtemp()
self.conf, self.state, self.tf = (os.path.join(self.t, n) for n in ("conf", "state", "textfile"))
for d in (self.conf, self.state, self.tf):
os.makedirs(d)
open(os.path.join(self.conf, "env"), "w").write(
"REMOTE=felhom-relcopy@ep0\nKEEP=3\nAGENT_KEEP=5\nCAP_MB=1\nREPOS='felhom-hub'\n")
open(os.path.join(self.conf, "registry-token"), "w").write("not-a-token\n")
self.patches = [mock.patch.object(rc, "CONF", self.conf), mock.patch.object(rc, "STATE", self.state),
mock.patch.object(rc, "TEXTFILE_DIR", self.tf)]
for p in self.patches:
p.start()
self.addCleanup(lambda: [p.stop() for p in self.patches])
self.pushed = []
def fake_images(self, size):
def f(authfile, stage, repos, keep):
d = os.path.join(stage, "images", "felhom-hub")
os.makedirs(d, exist_ok=True)
open(os.path.join(d, "blob"), "wb").write(b"x" * size)
return ["felhom-hub:0.167.0"]
return f
def fake_rsync(self, diff=""):
def f(env, stage, dry_check=False):
self.pushed.append(dry_check)
return diff if dry_check else ""
return f
def go(self, size, diff=""):
with mock.patch.object(rc, "copy_images", self.fake_images(size)), \
mock.patch.object(rc, "copy_agents", lambda reg, stage, keep: None), \
mock.patch.object(rc, "rsync_to", self.fake_rsync(diff)):
rc.main()
def signal(self):
return os.path.exists(os.path.join(self.tf, "felhom_release_copy.prom"))
def test_success_pushes_compares_then_signals(self):
self.go(1000)
self.assertEqual(self.pushed, [False, True])
self.assertTrue(self.signal())
sums = open(os.path.join(self.state, "stage", "SHA256SUMS")).read()
self.assertIn("images/felhom-hub/blob", sums)
def test_above_the_cap_nothing_is_sent(self):
# RED-PROOF: drop the cap check in main() → the push happens and the signal is written
with self.assertRaisesRegex(rc.Refused, "above the cap"):
self.go(2 * 1024 * 1024)
self.assertEqual(self.pushed, [])
self.assertFalse(self.signal())
def test_a_difference_after_the_push_writes_no_signal(self):
# RED-PROOF: ignore the dry-run's output → a copy ep0 does not hold reads as success
with self.assertRaisesRegex(rc.Refused, "differs"):
self.go(1000, diff=">fcsT...... images/felhom-hub/blob\n")
self.assertFalse(self.signal())
class Units(unittest.TestCase):
def read(self, n):
return open(os.path.join(HERE, n)).read()
def test_unit_runs_the_installed_script_and_mails_on_failure(self):
s = self.read("felhom-release-copy.service")
self.assertIn("ExecStart=/usr/local/sbin/felhom-release-copy\n", s)
self.assertIn("OnFailure=felhom-backup-failmail@%n.service", s)
def test_nightly(self):
self.assertIn("OnCalendar=*-*-* 01:40:00", self.read("felhom-release-copy.timer"))
if __name__ == "__main__":
unittest.main(verbosity=2)
+21 -1
View File
@@ -1,4 +1,24 @@
## 2026-10-22 — the download pages point at installer 1.30.0 (W1, W7; operator yes)
## 2026-10-23 — the privacy wording says what is true; the minimum machine is written down (D3 / R-985, D5 / R-991; operator rulings 2026-10-22 15:47) and D36's two sentences (R-1225, R-1226) — published on the operator's yes (day brief)
- **D36 (operator yes 2026-10-23, R-1225, R-1226).** `letoltes` / `en/download`: „A 8 GB az alapokat futtatja; a fotóalkalmazáshoz 16 GB kell.” / „8 GB runs the basics; the photo app needs 16 GB.” `gyik` / `en/faq` („Ti hozzáfértek az adataimhoz?”, the visible answer and its JSON-LD twin): „…csak titkosítva hagyják el az otthonodat (a távoli mentés)” became „Amit az otthonodon kívülről nyitsz meg, az a Cloudflare hálózatán halad át, és ott olvasható; a mentések csak titkosítva hagyják el az otthonodat.” / „What you open from outside your home passes through Cloudflare's network and is readable there; the backups leave your home only encrypted.” (`13-threat-model.md` §6, R-904).
- **D3 (R-985):** every promise that „nobody, not even us" can read the remote backup now says the whole truth.
The box encrypts the backup with a key made on the box; the recovery code is made on the box, shown once, and we keep
no copy; we hold only the locked key package and cannot open it; from the remote storage and the hub nobody can read
the backup. **New, stated plainly:** while the box runs, the operator has root access for support, and with it could
reach the data and the backup key on the box. Pages: `biztonsagimentes` / `en/backups`, `gyik` / `en/faq` („Ti
hozzáfértek az adataimhoz?", JSON-LD too), `szolgaltatasok` / `en/services`, `sajat-felho-tarhely` /
`en/own-cloud-storage`, `adatkezeles` (§6, §7, §8) and `feltetelek` (§6, §7: „a távoli mentésed nem nyitható ki" now
only when the box is lost AND the code is lost). Sources: `documentation/architecture/03-host-agent.md` §8a,
`07-backup-architecture.md` §2, `13-threat-model.md` §5. The volunteer guide (both languages) got the same wording.
- **D5 (R-991, R-993 merged):** the minimum machine — x86-64, 4 cores, 8 GB memory, 256 GB SSD; recommended 16 GB and
512 GB; not supported: Raspberry Pi / ARM, Synology / QNAP, routers, rented cloud servers. New block „Milyen gépre
telepíthető?" / „Which machines can it install on?" in the „Mielőtt letöltöd" card of `letoltes` / `en/download`
(installer version, links and checksum untouched); one paragraph in the FAQ „Milyen gépre van szükségem?" (JSON-LD
too); the estimator page's advice list now names the memory floor (`gepigeny` / `en/sizing`). `sizing-data.json`
unchanged. Nothing enforces the floor at install time (R-993's measurement) — that stays the operator's call.
## 2026-10-22 — the download pages point at installer 1.30.0 (W1, W7; operator yes)
- `letoltes.html` and `en/download.html`: version, both links, the side-file link, the two check commands and the SHA-256
now name `felhom-installer-1.30.0-pve9.2-1.iso` (`590d7b9b…74d5cc1`, the same 1 705 324 544 bytes). Published and
+6 -5
View File
@@ -250,8 +250,8 @@
</ul>
<p>
<strong>A kulcsletétről, egyszerűen:</strong> a mentésed kulcsát csak lezárt formában tároljuk.
Kinyitni csak a <strong>helyreállító kódoddal</strong> lehet, ami nálunk nincs meg. Ha elveszíted
ezt a kódot, a távoli mentésed nem nyitható ki — mi sem tudjuk kinyitni.
Kinyitni csak a <strong>helyreállító kódoddal</strong> lehet, ami nálunk nincs meg. Ha a szervered
tönkremegy, és ezt a kódot is elveszíted, a távoli mentésed nem nyitható ki — mi sem tudjuk kinyitni.
</p>
<!-- source: documentation/architecture/07-backup-architecture.md:67-75 ("genuinely zero-knowledge… R exists in zero system copies by design") -->
<p>
@@ -265,8 +265,9 @@
<h2>7. A szervered adatainak távoli mentése</h2>
<p>
A szervereden lévő adatokról távoli mentés készül. <strong>A mentés nálad, a szerveren
titkosítódik, mielőtt elhagyja az otthonodat.</strong> Sem a tárhely üzemeltetője, sem mi nem
tudjuk elolvasni a tartalmát.
titkosítódik, mielőtt elhagyja az otthonodat.</strong> A tárhelyen lévő tartalmát sem a tárhely
üzemeltetője, sem mi nem tudjuk elolvasni. A működő szervereden lévő kulcsot a rendszergazdai
hozzáférésünkkel elérhetnénk (lásd a 8. pontot).
</p>
<!-- source: documentation/architecture/01-topology-and-trust.md:118 ("ciphertext only (operator can't read)"); 07-backup-architecture.md:378-379 -->
<ul>
@@ -288,7 +289,7 @@
<p>
Őszintén: <strong>rendszergazdai (root) hozzáférésünk van minden szerverhez, amit kezelünk.</strong>
Ez kell a frissítésekhez, a mentések ellenőrzéséhez és a hibák javításához. Ezzel a hozzáféréssel
a szervereden tárolt adatok elérhetők. Csak üzemeltetésre és hibaelhárításra használjuk.
a szervereden tárolt adatok és a távoli mentés kulcsa is elérhetők. Csak üzemeltetésre és hibaelhárításra használjuk.
</p>
<!-- source: documentation/architecture/07-backup-architecture.md:49-52 ("The operator holds root SSH on every box… 'The operator cannot read customer data' was never the security property") -->
<p>
+3 -2
View File
@@ -269,8 +269,9 @@
<p>Egy lemezen minden helyi másolat <strong>ugyanazon a lemezen</strong> van. Ez véd a véletlen törlés és a hibás fájlok ellen, de egy lemezhiba ellen nem. Lemezhiba ellen <strong>egy másik lemez</strong> vagy a <strong>távoli mentés</strong> véd. A Felhom ezt meg is mutatja: csak akkor számol második másolatot, ha biztosan tudja, hogy az egy másik lemezen van, és ha ezt nem tudja ellenőrizni (például egy virtuális gépben), azt ki is írja.</p>
<h3 style="margin-top: 24px;"><svg class="ico-lg" aria-hidden="true"><use href="/assets/icons.svg?v=1#i-lock"/></svg> A helyreállítási kód – amit csak te ismersz</h3>
<p>A távoli mentésed titkosított, és a titkosítás kulcsa egy <strong>helyreállítási kód</strong>. Ez a kód pontosan egyszer jelenik meg: a te képernyődön, a beállításkor. Nálunk nem marad belőle másolat.</p>
<p>Ez tudatos döntés, és két oldala van. Az egyik, hogy a távoli mentésedet rajtad kívül <strong>senki</strong> nem tudja elolvasni – mi sem. A másik, hogy ha a doboz megsemmisül és a kód is elveszett, akkor a távoli mentés <strong>nekünk sem visszaállítható</strong>. Ezért a kód papírra kerül, és jó helyre.</p>
<p>A távoli mentésedet a doboz titkosítja, mielőtt az elhagyja az otthonodat. A kulcs a dobozon készül, és ott is marad. A kulcsról a doboz egy lezárt másolatot is készít, amit a <strong>helyreállítási kód</strong> nyit. Ezt a lezárt másolatot mi őrizzük, de kinyitni nem tudjuk. A kód is a dobozon készül, és pontosan egyszer jelenik meg: a te képernyődön, a beállításkor. Nálunk nem marad belőle másolat.</p>
<p>Ez tudatos döntés, és két oldala van. Az egyik: a távoli tárhelyen csak a titkosított mentés van, nálunk pedig csak a lezárt kulcs. Ezekből <strong>senki</strong> nem tudja elolvasni a mentésedet – mi sem. A másik: ha a doboz megsemmisül és a kód is elveszett, akkor a távoli mentés <strong>nekünk sem visszaállítható</strong>. Ezért a kód papírra kerül, és jó helyre.</p>
<p>Egy dolgot őszintén elmondunk. Amíg a dobozod működik, támogatás céljából <strong>rendszergazdai (root) hozzáférésünk</strong> van hozzá. Ezzel a dobozon lévő adataidat és a mentés kulcsát is elérhetnénk. Ezt a hozzáférést csak üzemeltetésre és hibaelhárításra használjuk. A helyreállítási kód tehát a távoli másolatot védi, nem a működő dobozt.</p>
<h3 style="margin-top: 24px;"><svg class="ico-lg" aria-hidden="true"><use href="/assets/icons.svg?v=1#i-chart-column"/></svg> Monitoring és értesítések</h3>
<p>A mentéseket folyamatosan figyeljük. Ha probléma van – sikertelen mentés, betelő tárhely, kapcsolati hiba – az a felügyeleti rendszerünkben megjelenik, és e-mail értesítés is indul róla. A zárt teszt egyik célja épp az, hogy ezeket az értesítéseket éles körülmények között csiszoljuk.</p>
+3 -2
View File
@@ -269,8 +269,9 @@
<p>On one disk, every local copy is <strong>on the same disk</strong>. That protects against deleting something by mistake and against broken files, but not against a disk failure. A <strong>different disk</strong> or the <strong>remote backup</strong> protects against a disk failure. Felhom shows this too: it counts a second copy only when it knows for sure that the copy is on another disk, and when it cannot check that (for example inside a virtual machine), it says so.</p>
<h3 style="margin-top: 24px;"><svg class="ico-lg" aria-hidden="true"><use href="/assets/icons.svg?v=1#i-lock"/></svg> The recovery code – only you know it</h3>
<p>Your remote backup is encrypted, and the key to the encryption is a <strong>recovery code</strong>. The code appears exactly once: on your screen, when it is set up. We keep no copy of it.</p>
<p>This is a deliberate choice, and it has two sides. One: <strong>nobody</strong> but you can read your remote backup – not even us. The other: if the box is destroyed and the code is lost too, then <strong>we cannot restore</strong> the remote backup either. So the code goes on paper, and somewhere safe.</p>
<p>The box encrypts your remote backup before it leaves your home. The key is made on the box, and it stays there. The box also makes a locked copy of the key, which the <strong>recovery code</strong> opens. We keep that locked copy, but we cannot open it. The code is made on the box too, and it appears exactly once: on your screen, when it is set up. We keep no copy of it.</p>
<p>This is a deliberate choice, and it has two sides. One: the remote storage holds only the encrypted backup, and we hold only the locked key. From these, <strong>nobody</strong> can read your backup – not even us. The other: if the box is destroyed and the code is lost too, then <strong>we cannot restore</strong> the remote backup either. So the code goes on paper, and somewhere safe.</p>
<p>One thing we tell you honestly. While your box runs, we have <strong>administrator (root) access</strong> to it for support. With it, we could reach the data on the box and the backup key too. We use this access only to run the service and to fix faults. So the recovery code protects the remote copy, not the running box.</p>
<h3 style="margin-top: 24px;"><svg class="ico-lg" aria-hidden="true"><use href="/assets/icons.svg?v=1#i-chart-column"/></svg> Monitoring and notifications</h3>
<p>We watch the backups all the time. If there is a problem – a failed backup, storage filling up, a connection fault – it shows up in our monitoring, and an e-mail goes out about it. One aim of the closed test is to polish these notifications in real use.</p>
+5
View File
@@ -92,6 +92,11 @@
<p>Installing <strong>erases every piece of data</strong> on the disk you select. The installer never chooses for you: it lists the disks with their size and type, and overwrites the one you pick.</p>
<p>Unplug the external drive you use for backups before you start. If you are not certain which disk is the right one, <strong>stop and call your operator</strong>.</p>
<p>The guide from your operator describes every step, screen by screen. Do not start without it.</p>
<h3 style="margin-top: 24px;"><svg class="ico-lg" aria-hidden="true"><use href="/assets/icons.svg?v=1#i-cpu"/></svg> Which machines can it install on?</h3>
<p><strong>At least:</strong> an x86-64 processor (Intel or AMD) with 4 cores, N100 class or newer; 8 GB of memory; a 256 GB SSD. 8 GB runs the basics; the photo app needs 16 GB.</p>
<p><strong>Recommended:</strong> 16 GB of memory and a 512 GB SSD. Photos and films go on a separate data drive.</p>
<p><strong>Not supported:</strong> Raspberry Pi and other ARM boards, Synology / QNAP, routers, rented cloud servers.</p>
<p>To estimate how much memory your own apps need, use the <a href="/en/sizing">Which machine do I need?</a> page.</p>
</div>
</div>
</section>

Some files were not shown because too many files have changed in this diff Show More