Author SHA1 Message Date
admin 79ee3e3392 Release close-out (attended, 2026-10-25): hub 0.170.0, agent 0.165.0 (vouched), controller 0.343.0 (global floor), catalog ebbf1f4; D49 = A and D44–D46 recorded (09 §3 258–261, CONTEXT, 07 §6.3/§8.2e); 9 rows closed, R-1254 hub half, R-1250 dated check 2026-10-27; R-1287–R-1290 filed; STATUS, map, REPORT-release-2026-10-26
gates / gates (push) Successful in 12m49s
2026-10-25 19:30:10 +01:00
admin 36ef8dcdec installer 1.36.1: http_code prints one status (R-1284, red-proved; committed, not published); R-1285 (BYO drop-in removed one bundle late), R-1286 (demo-hp kimai held since 10-14); release-2026-10-26 evidence: BYO proof, the three boxes on 0.343.0, Homepage gate on the drill
gates / gates (push) Successful in 12m58s
2026-10-25 18:32:04 +01:00
admin ade357a14e D44/D45 = all picks applied (R-1265, R-1266 closed; R-1280–R-1283 opened); R-232 dated check done; release-2026-10-26 evidence: fix A/B live on rc.6, D47 set-aside alarm on rc.7
gates / gates (push) Successful in 13m3s
2026-10-25 17:57:11 +01:00
admin a86ab64f6c hub v0.170.0 (R-1254: the recovery package in one transaction) — manifest bump, operator yes in the attended release run
gates / gates (push) Successful in 12m32s
2026-10-25 16:46:40 +01:00
admin 1d4abca54a Release run 2026-10-26: hub CHANGELOG names v0.170.0 (R-1254) for the build; the run's evidence folder (fix A/B red-proofs, probe)
gates / gates (push) Successful in 13m10s
2026-10-25 12:59:14 +01:00
admin 9647f1d158 Restore-truth run (2026-10-25): D47 + D48 recorded (09 §3 decisions 256, 257; CONTEXT); 07 §6.5 (R-1271 cause + the future-date rule), §6.6 (D48 order, the R-698 sentence), §8.2d (D47), §8.2e (R-1270); register 217 -> 219 (R-1278, R-1279 opened; R-1269/1270/1271 VERIFY; notes R-1206, R-1243 dated check done read-only, R-232 moved one day); STATUS D49 + Monday list; evidence (9202)
gates / gates (push) Successful in 13m44s
2026-10-25 11:04:46 +01:00
admin 67053335d1 Catalog sweep 2026-10-24: all 65 catalog apps on a fresh box (52 work, 10 with a problem, 0 broken); register 208 -> 217 (R-1269..R-1277 opened; notes on R-739, R-1267); STATUS + D46; map B1 re-checked; evidence
gates / gates (push) Successful in 13m17s
Every catalog app installed, used, backed up, removed keeping data, reinstalled, restored and removed with data on a
fresh box from the public ISO (demo-hp VM 9360, Tester4), endpoint-level. Worst: a reinstall's capture rewrites the
kept unit so its restore breaks the app (R-1269, P2), the restore still reports success (R-1270, P2), and „use my kept
data" loaded another box's future-dated database (R-1271, P2). Drill catalog fixes for the morning list (n8n usage
data; OpenGist 256M, proven by a cgroup probe). VM 9360 and LXC 9361 destroyed; hub host Tester4-aa9466 left for the
operator. REPORT-catalog-sweep-2026-10-24.md.
2026-10-25 00:13:05 +02:00
admin 3add386aa9 Restore-holes run (2026-10-24 night): D43 = A recorded; 07/05/03/11 carry the R-1251, R-1252, R-1254 and D43 rules; register 207 -> 208 (R-1268 opened; R-1251/1252/1253/1257 VERIFY, R-1254 NARROWED); STATUS Monday list + D44/D45 picks; evidence (Tester 1, 9202)
gates / gates (push) Successful in 13m4s
2026-10-24 21:08:38 +02:00
admin dcf4f84cb7 hub R-1254 (P2): the escrow key, its hash and the identity blob in ONE transaction; a bad identity blob is refused before anything is stored. Committed, not deployed.
gates / gates (push) Successful in 13m27s
2026-10-24 20:37:59 +02:00
admin 80f1ab3167 Review round 2: CI numbers by hash in the session report
gates / gates (push) Successful in 12m51s
2026-10-24 16:29:40 +02:00
admin 876d92367e Review round 2 (batch 2 + close-out): 02/07/09/10/11/14/15/16/inventory facts corrected at file:line (EDITS.md), undocumented decisions as notes (R-1265), rulings left for the operator (R-1266); register 190 -> 207 (R-1251..R-1267 opened: R-1251 P1 cut drive-app restore put back over newer data, R-1252 P2 rejected backup block deletes second-drive copies - reproduced on 9202); STATUS D43-D45; CONTEXT; audit README
gates / gates (push) Successful in 13m10s
2026-10-24 16:12:30 +02:00
admin 77015997f8 Review round 2, part 1 (batch 1): 01/03/04/05/06/08/12/13/_hub-review facts corrected against live code (each at file:line, EDITS.md); 20 capability rows marked re-check; poster timeline text; live-probes path; reviewer candidates and the H1/H2 proofs
gates / gates (push) Successful in 12m37s
2026-10-24 15:46:33 +02:00
admin 44d1904286 DooPlex backup alarms: one cause (the dump job's password after the reboot), fixed; alarm mail quieter; R-1250 proven
gates / gates (push) Successful in 12m24s
R-232 dated note + a dated check for tonight's sweep; R-934 closed (the disk was pulled
2026-10-23); R-1250 fixed on main, live proof on demo-hp with a real seed; 08 §9 item 5.
Evidence audits/dooplex-alarms-2026-10-24/.
2026-10-24 13:53:57 +02:00
admin 6018486fcc Attended release run done: controller 0.342.0 the global floor, agent 0.164.0 vouched, hub 0.169.0; R-1244 and R-1247 closed (register 193 -> 191); capability F13; STATUS; the run report
gates / gates (push) Successful in 13m3s
2026-10-24 10:37:27 +02:00
admin 01bf4e83a9 Deploy hub 0.169.0 (the upload watch's four operator-only events; the System page's Upload line) on the operator's yes; agent 0.164.0 delivery + vouch evidence
gates / gates (push) Successful in 12m38s
2026-10-24 09:46:42 +02:00
admin 1a788f332d Attended release run: hub 0.168.0 deployed (R-975 closed; the live file's residue accepted by the operator, 05 16.2); D40 in effect (R-1243 VERIFY, dated check 10-25); D41/D42 closed; the ep0 iperf3 line-test server (scripts/ep0-iperf3, 06 5) and hub 0.169.0 models line_test (System page cell); R-1249, R-1250 filed; register 194 -> 193
gates / gates (push) Successful in 12m46s
2026-10-24 09:18:44 +02:00
admin 3363ffa89a D40 (R-1243): the hub's volume out of DooPlex's Longhorn backup — PVC label default: disabled (operator 2026-10-24)
gates / gates (push) Successful in 12m22s
2026-10-24 07:57:23 +02:00
admin 0003115092 Deploy hub 0.168.0 (R-975: the last plain tokens sealed) on the operator's yes; CONTEXT: the 2026-10-24 07:22/07:26 rulings (D40, D41, D42, the iperf3 line test)
gates / gates (push) Successful in 12m41s
2026-10-24 07:33:36 +02:00
admin 20fa831eaf Upload watch night data: one-second reads of the real push on three boxes (spike 2.1); R-1245 gets the measured floor; R-1214 dated check read (stays WATCHING); weekend-night report
gates / gates (push) Successful in 12m31s
2026-10-24 04:44:25 +02:00
admin a91d0b2421 Upload watch: spike, design (08 section 9, proposed), live proofs on 9202; rows R-1244..R-1248 (register 189 -> 194); STATUS D41/D42 + the Monday list
gates / gates (push) Successful in 13m1s
Spike: per-app normal levels on three boxes (19-31 days), the off-site push as a line measure (a lower bound),
qBittorrent/Transmission/Plex explainers measured on 9202, a tc ceiling measured (not built). Live on 9202 with
0.342.0-rc.1: limit set; real seed explained; CPU burner and the hidden flood alarmed once each and cleared once;
the household's limit left alone. Teardown: apps removed, 9202 back on 0.341.0, receiver 9471 destroyed.
2026-10-23 22:52:41 +02:00
admin af5c005a4e hub 0.169.0 (unreleased, not built): the upload watch's four events — allowed, operator-only, per-app cooldown, one readable Watch line in the operator mail
gates / gates (push) Successful in 12m47s
app_unexplained_upload / app_unexplained_cpu (warning) and their _cleared (info), from controller v0.342.0 (08 section 9,
operator ruling 2026-10-23). Red-proved: operator-only, the render line, the allow-list.
2026-10-23 21:26:36 +02:00
admin 5277d859d5 hub 0.168.0: seal the last plain tokens in config_json (R-975, R-1241's Felhom half); decide row R-1243 for the hub's plain Longhorn copies (register 188 -> 189)
gates / gates (push) Successful in 12m26s
The Cloudflare tunnel token, Cloudflare API token and git token inside customer_configs.config_json now hold the
05 §16.2 seal (enc:v1:, OFFSITE_SECRET_KEY). Old rows sealed once at start-up without a config_version bump, then
VACUUM + WAL checkpoint (an UPDATE leaves the old plaintext in a freed block). -unseal-box-secrets opens them too.
Eight tests, each red-proved. Built, not deployed. Audit: documentation/audits/hub-token-seal-2026-10-23/.
2026-10-23 19:48:17 +02:00
admin 8257ff1cbe Plex2Shell impact on Felhom: verdict NOT REACHED; R-1241 (P1), R-1242 (P3) opened (register 186 -> 188)
gates / gates (push) Successful in 12m37s
Read-only audit of whether the hacked Plex pod on DooPlex (HM-040, 2026-10-17 -> 10-23) reached Felhom's
secrets. Two no-login doors existed (the Longhorn NFS export and the Longhorn API); neither was used:
nfsd accepted 0 new connections in the window (positive control), no off-schedule Longhorn backup or volume.
Evidence in documentation/audits/plex2shell-felhom-impact-2026-10-23/. The NFS export was narrowed from *
to 10.42.0.0/16 on the operator's yes (host change, not in git). 13-threat-model.md: attacker j and one
boundary row. STATUS: verdict, the P1, one do item.
2026-10-23 17:53:43 +02:00
adminandClaude Opus 5.5 87dc50743c D39 = B: 09 §3 decision 255 (the security-fix step) + a dated note on decision 13; capability B23; R-1239 closed, R-1240 opened, R-1238 updated (register 186 -> 186); STATUS/CONTEXT; nodes.md (9202 password, Tester 1 DHCP); evidence security-lane-2026-10-23 (controller 0.341.0, global floor 11:24 UTC)
gates / gates (push) Successful in 13m49s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-23 13:35:38 +02:00
adminandClaude Opus 5.5 4b9b0af3cb Plex2Shell: the operator's 10:56 ruling in CONTEXT; demo-hp 9201 checked (clean) and updated to 1.43.5.11029; register 184 -> 186 (opened R-1238 Plex test record/fixture, R-1239 a fast lane for security fixes = D39); evidence plex-demo-boxes-2026-10-23
gates / gates (push) Successful in 13m46s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-23 11:27:52 +02:00
admin 2c9152d912 Day 2026-10-23 close-out: rulings D31-D36 in CONTEXT; STATUS (releases done, do list, D35 dated, D37/D38); register 191 -> 184 (closed R-49 R-982 R-985 R-991 R-1204 R-1205 R-1225 R-1226 R-1227 R-1230 R-1234; opened R-1235 R-1236 R-1237); the break-glass sheet current (D32, one A4, HU+EN); runbooks for D33/D34; poster facts; evidence day-2026-10-23
gates / gates (push) Successful in 12m53s
2026-10-23 11:03:45 +02:00
admin 52af38df6f hub-db-backup: wait for a snapshot being written, limit 7 h (one slot gap + 1 h) — the 2 h limit refused the hub's own catch-up and mailed a false failure at 10:28 (R-1234, closed); tests red-proved
gates / gates (push) Successful in 13m14s
2026-10-23 10:35:16 +02:00
admin 34979d5d25 hub manifest: 0.165.0 -> 0.167.0 (operator yes 2026-10-23: D31, the clock-ahead tunnel alarm, R-1230, D34 6-hourly snapshot; image sha256:177860a9…)
gates / gates (push) Successful in 13m26s
2026-10-23 10:23:29 +02:00
admin 0bb57a4fee website: publish D3/D5 (the whole truth on privacy; the minimum machine) and D36's two sentences, HU + EN (operator yes 2026-10-23; R-985, R-991, R-1225, R-1226)
gates / gates (push) Successful in 13m10s
2026-10-23 10:16:54 +02:00
admin 2aebf1bde1 webpage: serve installer-v1.36.0 (both git-sync pins; operator yes 2026-10-23 — root SSH by key only on new installs, R-982 D4)
gates / gates (push) Successful in 13m12s
2026-10-23 10:08:09 +02:00
admin d306b8c2d7 release-copy: the released images (skopeo dir:, digest-true) and agent packages copied to ep0 nightly, verified by sha256 and a checksum compare, capped at 3 GB (D33, R-1227); tests red-proved
gates / gates (push) Successful in 15m56s
2026-10-23 09:39:31 +02:00
admin be7e97dc1a hub 0.167.0 + DooPlex push: the hub DB snapshot every 6 hours (D34, operator yes 2026-10-23, R-232 G5) — slots 02/08/14/20, catch-up 6 h, push :30 after each, refuse a copy older than 2 h; tests red-proved
gates / gates (push) Successful in 16m20s
2026-10-23 09:12:08 +02:00
admin 83388c088a felhom-host-install.sh 1.36.0 (pushed, NOT published — the tag waits for the operator's yes): the uninstall removes the root-SSH drop-in, a failed reload warns (R-982, D4, reviewed v2 patch)
gates / gates (push) Successful in 14m17s
2026-10-23 08:29:20 +02:00
admin a408256c7e REPORT-night-2026-10-23: the night report (five lines, parts 1-5, findings -> verdicts, the test night PASS, teardown in three layers, commits + CI); STATUS: the test night's verdict in the morning list; Nextcloud sha256 + teardown evidence
gates / gates (push) Successful in 16m37s
2026-10-23 07:39:27 +02:00
admin 7802da7298 Dated checks 2026-10-23 done: R-32 (Tester4's listing 72 -> 80, none vanished after Tester 2's RESET), R-1173 (demo-hp's PBS tier started with nothing in flight; the busy branch still test-only); R-548: demo-hp's local tier refused 4 times by space tonight, apps stopped before each refusal
gates / gates (push) Successful in 16m23s
2026-10-23 07:30:13 +02:00
admin 1cb3e57133 Register 189 -> 191 (R-1232 a reinstalled box's first ep0 copy may come a week late — likely; R-1233 an error line naming nothing); test-night evidence: night jobs, D21 on the box, no copy container left, off-site leg/proof/integrity, controller log, agent journal (sudo noise trimmed), recorder, outside probe, loader logs, Immich per-user counts, night numbers 2026-10-23 07:13:52 +02:00
admin 5dc17cab03 test night: controller fixture 2 (rc.8) recorded
gates / gates (push) Successful in 15m57s
2026-10-23 00:07:06 +02:00
admin f56f3328f7 Register 187 -> 189: R-1230 (a held box reported stuck — measured on the test night, fixed in hub 0.167.0 source), R-1231 (the recovery page calls an unknown agent older); test-night provenance 2026-10-23 00:07:01 +02:00
admin e94d0256d2 test night evidence: agent fixture 2 (0.162.99 = agent main b9b831f), the off-site unlock with the household's recovery code (first refused: a pre-release fixture version reads as an unknown capability)
gates / gates (push) Successful in 15m44s
2026-10-22 23:56:34 +02:00
admin 14388e218f hub 0.167.0 source: R-1230 — a box that holds for the household's recovery code is not reported stuck and gets no off-site re-issue (measured on the test night); red-proved 2026-10-22 23:56:20 +02:00
admin 1805f8bbf0 STATUS 2026-10-23 morning: D29-D31 to history; D32-D36 (the DooPlex survey and two website sentences); the morning list in the brief's order (hold or go, controller, agent + installer, hub 0.167.0, website, discards, the DooPlex hour); verdict of the test night at 07:00
gates / gates (push) Successful in 15m56s
2026-10-22 23:39:06 +02:00
admin 70a600d0c2 Register 180 -> 187 (opened R-1223..R-1229: the tunnel reboot gap, the per-customer cooldown, two website sentences, the registry and the home line and ArgoCD from the DooPlex survey); R-1214/R-1204/R-1205 READY, R-1187/R-982/R-925/R-232/R-923/R-924 updated; rc.8 re-proof of the stray-copy fix on 9202 2026-10-22 23:25:16 +02:00
admin c6649dd49b hub 0.167.0 source: the stale-tunnel text no longer promises 15 minutes flat (15 min after the agent sees the sign, plus its next desired-state fetch) — review N8
gates / gates (push) Successful in 15m55s
2026-10-22 23:00:46 +02:00
admin 7585443227 night 2026-10-23 so far: D30 live proof (re-key → new-key handshake; ep0-started handshakes heal a short clock jump by themselves), D4 patches v2 after review, test-night install/fixture evidence, DooPlex survival survey (read only) + PLAN, 03/05/07 + CONTEXT, total-loss runbook: MAIL-HOLD shipped
gates / gates (push) Successful in 15m49s
2026-10-22 22:24:48 +02:00
admin 84751cb0ce hub v0.167.0 source (not built, not deployed): D31 — the F-14 PBS auto re-issue reads the newest deletion of a host that held a key, a keyless install between no longer blocks it (R-1205); D30 — the new key's handshake clears wg_tunnel_stale (pinned), wording; wire gate entry; Part 1 stray-copy 9202 evidence
gates / gates (push) Successful in 15m46s
2026-10-22 21:04:42 +02:00
admin 96ac07e168 REPORT-night-2026-10-22: CI 2224 and 2225 green
gates / gates (push) Successful in 15m51s
2026-10-22 18:47:33 +02:00
admin acc78637d9 night 2026-10-22: the seven 15:47 rulings in effect or ready (D6, D8 live; D21 waits for controller 0.340.0; D3/D5 website patch; D4 two patches); data-loss candidates proven/fixed/filed; R-1214 cause + capped copy proven on 9202; R-1204 alarm built; repos-private read; STATUS morning list + D29/D30/D31; CONTEXT; 03/05/07/09; rows 178 -> 180 (opened R-1218..R-1222, closed R-206, R-992, R-1207)
gates / gates (push) Successful in 16m7s
2026-10-22 18:24:30 +02:00
admin 1100a344d5 night 2026-10-22: D4 as two ready patches (agent bundle + installer uninstall) — a push of the installer would publish it
gates / gates (push) Successful in 16m3s
2026-10-22 17:11:07 +02:00
admin e267c5e2cd hub 0.166.0 source: the clock-ahead tunnel alarm wg_tunnel_stale/ok (R-1204, operator-only, edge-triggered); D21 + D4 proof evidence 2026-10-22 17:02:37 +02:00
admin 0639cf0772 night 2026-10-22: D3 wording in the volunteer guides (the website change is a ready patch — a push publishes); D6 drill-reader proof; P3 research; P6 repos-private readers (read only)
gates / gates (push) Successful in 16m8s
2026-10-22 16:22:28 +02:00
admin 1b442bcc2e Tester 2's RESET checked read only: every leg ok, the first live R-32 purge (folder held no repository; sub-account gone), ep0 namespace+copy+prune job+token gone, no other customer touched (Storage Box 303/62/43 and ep0 2/2/2/1/27 before = after); hand-over: tell the friend go; A20 third firing; R-32 result + dated check for Tester4
gates / gates (push) Successful in 16m3s
2026-10-22 15:04:07 +02:00
admin 82ce9d72d6 Tester 2: a new install, not an upgrade (operator ruling 2026-10-22 14:11) — hand-over list rewritten (host delete then RESET, what each removes, the friend's page in Hungarian); R-862, R-1139 close at the reinstall; R-32's first live purge noted; STATUS, CONTEXT
gates / gates (push) Successful in 16m5s
2026-10-22 14:19:26 +02:00
admin ecfc09c3ef day 2026-10-22: R-1199 reopened and closed (hub 0.165.0 + controller 0.339.0; demo-felhom's list restored), R-1216 closed (D28 = A), R-1217 filed; Tester 2 hand-over list; STATUS, CONTEXT, D28 to history; rows 178 -> 178
gates / gates (push) Successful in 16m0s
2026-10-22 13:17:17 +02:00
admin 25dae2f568 hub 0.165.0: deploy (operator yes 2026-10-22) — the household's mail list never overwritten; host_swap_short only when swap is really short
gates / gates (push) Successful in 15m59s
2026-10-22 12:46:27 +02:00
admin 5ae5984cf9 hub 0.165.0 source: a household's mail list is never overwritten by a box default, a box reads it back (R-1199 reopened); host_swap_short only when swap is really short (R-1216, D28 = A); host-swap-rule gate with 4 decoys; 03/05/08 updated
gates / gates (push) Successful in 15m54s
2026-10-22 12:23:07 +02:00
admin a92837410a morning 2026-10-22: ISO 1.30.0 published (W1, W7; gate + proof installs on both entries); download pages point at it; rows 184 -> 178 (closed R-1192, R-1177, R-1178, R-1180, R-1195, R-1199, R-1210; opened R-1216 the host-swap check too strict); STATUS, CONTEXT, report
gates / gates (push) Successful in 15m50s
2026-10-22 10:51:24 +02:00
admin 115f15c89e morning 2026-10-22 evidence: agent 0.162.0 delivered to the 3 test boxes, controller 0.338.0 on all 3 through the floor, https-only + containers + controller errors green
gates / gates (push) Successful in 15m49s
2026-10-22 10:16:49 +02:00
admin 6d5a7f1b92 ISO 1.30.0: version bump for the release build (W1 boot text, W7 banner clears; operator yes 2026-10-22); the gate's G16 command names iso_bootstrap_gate.py (the old mount failed 10 checks on a correct tree)
gates / gates (push) Successful in 15m54s
2026-10-22 10:09:50 +02:00
admin 6b391a639a website: a real download button on the download pages (W2, R-1210; operator yes 2026-10-22)
gates / gates (push) Successful in 15m37s
2026-10-22 09:55:55 +02:00
admin 0b7a5ab3bc hub 0.164.0: deploy (operator yes 2026-10-22 morning) — host_swap_short alarm, first-hour words W3-W5
gates / gates (push) Successful in 15m39s
2026-10-22 09:42:08 +02:00
admin 26dbc987d9 night 2026-10-21: soak result (no freeze in 11 h; memory pressure above 20 % for 3 min in the tier2 job, R-1214; 1 Immich upload lost, R-1215), no-swap test (the freeze comes back without host swap, R-1207), teardown, the night report
gates / gates (push) Successful in 16m4s
Rows 182 -> 184. STATUS morning list item 2 carries the soak result; CONTEXT the two results.
2026-10-22 06:50:29 +02:00
admin 97f088ec02 night evidence: both Nextcloud upload trees 3031/3033 byte-identical against the night manifest (the 2 missing are the uploader's by-design skips)
gates / gates (push) Successful in 15m49s
2026-10-22 01:12:36 +02:00
admin dee26da3fa STATUS-HISTORY: the 2026-10-21 evening Updated paragraph moved here (STATUS rule)
gates / gates (push) Successful in 15m39s
2026-10-22 00:34:26 +02:00
admin c2c130ecbb R-1209 closed: the decoy suite gets 600 s in script-tests (the others keep 300 s; a time-out still fails) — it timed out on every CI push tonight
gates / gates (push) Successful in 15m40s
2026-10-22 00:26:53 +02:00
admin f576d7f7ba register: R-1211 (the household backup_failed mail names no app and no next step), R-1212 (two backup texts ignore the box language), R-1213 (operator detail left on two pages) — the night helper's findings; 10-localisation: W25 narrows the verbatim rule on the off-site page; 180 -> 183
gates / gates (push) Failing after 15m3s
2026-10-22 00:07:57 +02:00
admin 307d25b174 STATUS: the morning list after the night run (controller, agent, hub, Tester 2's install, the publishes); D22 and D26 out of the decide table (decided), D27 in (host swap on a no-swap host, R-1207)
gates / gates (push) Failing after 15m3s
2026-10-21 23:59:18 +02:00
admin 57c06ba99d CONTEXT + 08: tonight's rulings (D22, D26) and the CC-unattended R-1199 decision (the box's page is the list the hub mails by); evidence: rc.2 smoke on 9202, 9202 back as found, no-swap test tools 2026-10-21 23:45:05 +02:00
admin 9d3bf76def register: R-1177 R-1195 R-1178 R-1180 fixed on main and proven (D22), R-1194 cause found and fixed in the agent — all wait for the morning releases; R-1188 (a) fixed; new R-1207 (host swap on a no-swap host), R-1208 (node_down after an operator delete), R-1209 (CI decoy suite near its timeout), R-1210 (first-hour words waiting for a publish); 176 -> 180 2026-10-21 23:18:37 +02:00
admin 2293b252d5 docs: 07 §6.3 the put-back covers every restore and a power cut (D22, as built and proven on 9202); 03 guest swap = RAM and the host's swap behind it; capability map D16; soak provenance
gates / gates (push) Failing after 14m51s
2026-10-21 23:00:45 +02:00
admin 6b0d23c29c night 2026-10-21 evidence so far: soak box install + fixture agent (swap raised live), D22 proof on 9202 (put-back on a damaged unit and after a controller kill), R-1194 reproduced, R-1199 read, red-proofs, the W2 website patch (not applied)
gates / gates (push) Failing after 15m10s
2026-10-21 22:39:25 +02:00
admin 2fa5b8f3d8 ISO source (NOT built, NOT published): boot text in the te-form (W1); the linked banner clears the screen and its scroll-back so the spent pairing code is gone (W7); test with a decoy
gates / gates (push) Failing after 14m52s
2026-10-21 22:32:14 +02:00
admin f90e22293a hub 0.164.0 (unreleased): bind mail true when no box exists (W3), tries left on a wrong try (W4), use the newer mail hint (W5)
gates / gates (push) Failing after 14m55s
2026-10-21 22:18:38 +02:00
admin 3c620f8d2e hub 0.164.0 (unreleased): host_swap_short / host_swap_ok operator alarm from the agent's host and guest swap numbers (R-1192 follow-up); wire gate allowlists guests.spec.swap_bytes
gates / gates (push) Failing after 14m53s
2026-10-21 21:55:15 +02:00
admin dbfe5dac06 fixes 2026-10-21: teardown (demo-hp, hub, ep0 read-only), final evidence
gates / gates (push) Successful in 13m45s
2026-10-21 18:50:23 +02:00
admin 47fa1e621e fixes 2026-10-21: register (7 closed, R-1205/R-1206 opened, R-1183 VERIFY, R-1192 diagnosed), STATUS (not ready: one step — the agent swap fix), A12/A13 proven live, CONTEXT, report, evidence
gates / gates (push) Successful in 13m51s
2026-10-21 17:57:32 +02:00
admin ef4dbcf984 fixes 2026-10-21: evidence so far (R-1176 live proof, R-1192 freeze measured, R-1191 on 9202); design notes 03 §8a / 07 §7.0, §8.2a (D25) / 05 §16.5 (DST)
gates / gates (push) Successful in 13m34s
2026-10-21 16:00:36 +02:00
admin cd3d05981a manifests: hub 0.163.0 (operator yes, 2026-10-21)
gates / gates (push) Successful in 13m21s
2026-10-21 13:42:42 +02:00
admin 9688e76efd hub v0.163.0: a rebuilt box gets the household's sealed package back (R-1176); one run per daily job on clock-change days (R-1183); a scoped abandonment purge (R-1182); the new-link page says the old box is still linked (R-1203)
gates / gates (push) Successful in 13m23s
2026-10-21 13:36:45 +02:00
adminandClaude Opus 5.5 e07b077ac2 system poster refreshed 2026-10-21: one-file HTML + A1 PDF from Claude Design (data area, https only); R-1201 closed
gates / gates (push) Successful in 13m7s
Third export passed all Part A checks (documentation/audits/poster-refresh-2026-10-21/CHECKS.md).
STATUS poster line removed; facts file History line. Register 179 -> 178.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-21 13:25:43 +02:00
admin 7ad17be6bd rehearsal 2026-10-20: final report, STATUS (morning list, D24), capability map A12/A13 → PARTIAL (B7), stuck point 10
gates / gates (push) Successful in 13m35s
2026-10-21 11:47:40 +02:00
admin eaff4d890c teardown: secret count 24 2026-10-21 11:41:58 +02:00
admin 7a287e71fc rehearsal 2026-10-20: Part D teardown (machine done; hub host for the operator; ep0/Storage Box listed, nothing removed) 2026-10-21 11:41:51 +02:00
admin 6559aa6d2e rehearsal 2026-10-20: B7 — re-linked after the operator's press; the household cannot restore from off-site (R-1176 proven live)
gates / gates (push) Successful in 13m11s
2026-10-21 11:36:48 +02:00
admin a2d2569719 register: the ep0 WireGuard row is R-1204 (R-1201 was taken by a parallel session; the earlier commit message says R-1201)
gates / gates (push) Successful in 13m37s
2026-10-21 11:03:32 +02:00
admin ebf142d921 rehearsal 2026-10-20: B7 — box destroyed, reinstalled; the forgotten press is a dead end for the household (R-1203, P2) 2026-10-21 10:59:24 +02:00
admin 6d73110cd0 rehearsal 2026-10-20: B6 #10 + #15 — PASS 2026-10-21 10:59:24 +02:00
admin ff06ab52da rehearsal 2026-10-20: B6 #8 wrong clock — R-1201 (ep0 tunnel dead after a clock ahead, P2), R-1202 (clock behind: silent app stop, blames the hub) 2026-10-21 10:59:24 +02:00
admin a3be7525cd rehearsal 2026-10-20: B6 #7 DST live — PASS (one database copy at the second 02:30); #8/#10 expectations written before 2026-10-21 10:59:24 +02:00
adminandClaude Opus 5.5 3decdf94a8 poster refresh 2026-10-21: export NOT taken in (6-file folder, 4 unasked text rewrites -> R-1201); OMV guide matches Ferenc's PDF (installer >= 1.33.0)
gates / gates (push) Successful in 13m21s
Part A checks + label diff + PNG crops in documentation/audits/poster-refresh-2026-10-21/.
Guide: HDD_PATH trouble item and "remote data area" sentence removed, first-days line added,
operator note dated. Register 178 -> 179.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-21 10:00:54 +02:00
admin 7a773207f8 STATUS: dated checks of 2026-10-21 done (R-895 read, R-1019 closed), R-435 moved
gates / gates (push) Successful in 13m28s
2026-10-21 07:54:41 +02:00
admin 19064ef1a7 register: R-1200 state word 2026-10-21 07:41:59 +02:00
admin 31d975c997 rehearsal 2026-10-20: B5 night (db/second drive/off-site PASS, data identical; whole-box not due → R-1200; host reboot = harness); due checks R-1019 closed, R-895 read, R-435 moved 2026-10-21 07:38:10 +02:00
admin 0d3f72e463 rehearsal 2026-10-20: journal — push held by due-checks gate 2026-10-21 02:58:05 +02:00
admin c4262a6f6c rehearsal 2026-10-20: night db copy OK; watcher logs recovered after an autostash froze them (harness slip) 2026-10-21 02:44:16 +02:00
admin a600d68d50 rehearsal 2026-10-20: night log copier added before the night (read-only)
gates / gates (push) Successful in 13m39s
2026-10-20 23:39:50 +02:00
admin 830ed9714e rehearsal 2026-10-20: report draft (night/morning parts pending); W26 marked fixed (unreleased)
gates / gates (push) Successful in 12m56s
2026-10-20 23:27:47 +02:00
admin 16188df64c rehearsal 2026-10-20: 44 more screenshots off the browser container, gallery rebuilt; A4 browser spike result
gates / gates (push) Successful in 12m48s
2026-10-20 23:16:20 +02:00
admin 0d4bfe7ca6 rehearsal 2026-10-20: pre-night household snapshot (byte compare vs manifest), B7 plan written before
gates / gates (push) Successful in 12m56s
2026-10-20 23:09:29 +02:00
admin 938afb7416 rehearsal 2026-10-20: B10 ten stuck points; capability map A15 updated (still PARTIAL: R-1192); clock tool for #7/#8/#10
gates / gates (push) Successful in 12m57s
2026-10-20 22:41:46 +02:00
admin c0b70b15ac rehearsal 2026-10-20: post-night scenario plan written before any result
gates / gates (push) Successful in 12m51s
2026-10-20 22:24:21 +02:00
admin ce8cf2c477 rehearsal 2026-10-20: B6 #1 README, journal to 20:14, B5 night expectations written before the night
gates / gates (push) Successful in 12m55s
2026-10-20 22:18:53 +02:00
admin 4a557f54a5 rehearsal 2026-10-20: B6 #1 power cut mid install, #6 off-site blocked, #14 kept data — evidence
gates / gates (push) Successful in 12m58s
2026-10-20 22:13:50 +02:00
admin 92de2d324e Rehearsal: B6 #12 DNS dead an hour (as expected; household told nothing); R-1199 (P2 VERIFY) the household's ticked alarms reached only the operator (1 of 9 mails); R-1189 (b) demo-hp journal: three skipped nights
gates / gates (push) Successful in 13m31s
2026-10-20 21:50:47 +02:00
admin 12090df2d9 Rehearsal: R-1197 (Nextcloud 413 over ~1 GB by WebDAV), R-1198 (a later backup disk is never offered as the whole-box target); R-1194 facts
gates / gates (push) Successful in 13m23s
2026-10-20 20:50:36 +02:00
admin 434fc7cc02 Rehearsal: B6 #3 disk full (as expected: warned at 89 %, refused before writing, old copies kept, healed); #12 DNS dead started
gates / gates (push) Successful in 13m23s
2026-10-20 20:43:29 +02:00
admin 53d962d95b Volunteer guide (HU+EN): the console's „linked" text replaces the stale „known bug: the code stays on screen" note; „Mentés most" takes ~2 min with eight apps (measured in the 2026-10-20 rehearsal)
gates / gates (push) Successful in 13m37s
2026-10-20 20:12:20 +02:00
admin 836b47f1ec Rehearsal: B6 #16 English pass, #9 two-at-once (as expected), #13 second drive (not offered as backup target), #4 drive pulled (handled well; last uploads 0 bytes → R-1196 P3); B9 W24-W29
gates / gates (push) Successful in 13m21s
2026-10-20 20:07:30 +02:00
admin bb187f3503 Rehearsal: B8 all seven restore paths pass (NFC/NFD: both names return); B6 #2 power cut mid-restore → R-1195 (P2, proven live: the app runs on the backup's data, the page says only interrupted); B6 #5 slow line; register 173 -> 174
gates / gates (push) Successful in 13m31s
2026-10-20 19:43:31 +02:00
admin 9db8aacdad Rehearsal: R-1193 corrected and narrowed to P3 (the operator mailbox shows host_stale at 13:33 and app_oom; the household was told nothing); R-1192 second freeze + the app_oom hint; R-1194 (P3 VERIFY) fresh box's felhom-sshd config invalid; B6 #5 slow-line result
gates / gates (push) Successful in 13m17s
2026-10-20 19:10:42 +02:00
admin 78278502ac Rehearsal 2026-10-20: second freeze and plug pull, memory watch, Mentés most, B6 #5 started; B9 words W7-W23; tools
gates / gates (push) Successful in 13m11s
2026-10-20 18:32:20 +02:00
admin 2228b9f6b4 Rehearsal: R-1192 (P1) the 8 GB minimum box froze ~1 h on the family's first afternoon, plug pulled to recover; R-1193 (P2) the hub showed it OK throughout; register 170 -> 172; STATUS P1 line and a note on D5
gates / gates (push) Successful in 13m16s
2026-10-20 16:38:27 +02:00
admin ee743477b4 Rehearsal 2026-10-20 B2-B4: bound (operator passphrase), claimed, recovery code, family (accented names refused), second disk initialised, 8 of 9 apps (Jellyfin refused on memory), first uploads froze the box ~1 h (hub said OK); screenshots gallery
gates / gates (push) Successful in 13m8s
2026-10-20 16:34:30 +02:00
admin ad6397308c Rehearsal 2026-10-20: guide §3 quotes the console the box really shows; bind-page screenshots (codes never shown); app upload and gallery tools; R-1189 (d) (e) confirmed in code
gates / gates (push) Successful in 13m17s
2026-10-20 13:17:29 +02:00
admin d750a90d0e Data-loss hunt: R-1188 (c) proven — a leftover .felhom-restoring blocks every retry of the second drive's whole restore
gates / gates (push) Successful in 13m33s
2026-10-20 13:11:32 +02:00
admin a942ae90d6 Data-loss hunt: R-1185 proven (an invalid dump replaces the good one, no error); R-1182 confirmed in code; DST replay of the controller scheduler (once each on 25 Oct; spring 02:30 dump collides with the 03:30 copy)
gates / gates (push) Successful in 13m44s
2026-10-20 13:07:39 +02:00
admin 20a77d367b Data-loss hunt: R-1180 proven against production code (the nightly rsync --delete mirror runs while a restore holds the backup flag and deletes the files being restored)
gates / gates (push) Successful in 13m20s
2026-10-20 13:02:50 +02:00
admin a57a8363df Data-loss hunt: R-1191 (P1, proven live on 9202) — a .fab import makes a family-only app public (stranger 401 -> 200); R-1187 narrowed; register 169 -> 170; STATUS P1 line
gates / gates (push) Successful in 12m52s
2026-10-20 12:58:30 +02:00
admin bffe2d9dea Data-loss hunt 2026-10-20 C4: register 153 -> 169 (opened R-1175..R-1190: two P1 — shares second copy pruned by another drive's group, box rebuilt after the host delete cannot reach its off-site history; R-1177 proven LIVE on 9202 — a failed local restore destroys live data with no copy); STATUS: the P1 line and decisions D22, D23
gates / gates (push) Successful in 13m20s
2026-10-20 12:31:34 +02:00
admin 242584af4e Status view report: CI jobs 2131-2133 green, by head_sha
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-20 12:28:00 +02:00
admin 43e5a5496d Data-loss hunt 2026-10-20: four reviewers' candidates, C2 verdicts, proofs (D1 shares second copy pruned, K1 escrow unreachable after host delete, R5 whole restore leaves app stopped while saying untouched, T9 hub jobs run twice on 25 Oct); rehearsal A5 manifest, B1 notes, B9 words, tools
gates / gates (push) Successful in 12m41s
2026-10-20 12:13:19 +02:00
admin c701b2530b Status view: session report (REPORT-status-view-2026-10-20.md); STATUS points to the rehearsal's screenshot folder
gates / gates (push) Successful in 12m53s
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-20 12:06:50 +02:00
admin 54d1ee8210 Status view, part 4 (the instruction): CLAUDE.md end-of-session checklist — how to update STATUS.md in five lines (replace the Updated paragraph, never prepend; a dated morning list in do; remove what you finished; the decide-row rule; the gate checks it, R-1174). A format rule under PROMPT-TEMPLATE §9 rule 9 (decision 150); no safety rule loosened
gates / gates (push) Successful in 12m43s
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-20 11:59:54 +02:00
admin 1d7071e8fd Status view, part 3 (the gate): scripts/status_view_gate.py in repo_gates.py (fast: pre-push + CI) — size outside decide, one Updated paragraph, English outside quotes, cited rows open, no decided row in decide, the P1 and every operator-owned row on the page; WARN-FIRST until 2026-10-27, then it refuses; 39 decoy cases (red-proved per check and on both sides of the date); filed R-1174 (P4, Process & tooling) + its due check 2026-10-27; R-919 owner cell -> operator (the upload and the look); register 152 -> 153
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-20 11:59:48 +02:00
admin 9ec5f76a88 Status view, part 2 (the page): STATUS.md rebuilt from the register — 107 lines / 31,695 bytes -> one screen (50 lines, about 5,400 characters outside the decide table); decide table: 8 open decisions (13 decided rows out, rulings checked in CLOSED-ITEMS); do list checked against the hub and DooPlex (7 done items out, 7 'still needed? yes/no'); old page byte for byte in STATUS-HISTORY.md
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-20 11:59:41 +02:00
admin f84f94d987 Rehearsal 2026-10-20 (Tester4) + data-loss hunt: baselines, B1 install evidence, A6 drawn order, tools, first two reviewers' candidates
No release, deploy or publish. Off-site is OFF for Tester4 and the owner passphrase is needed from the operator (bind blocked after 2 of 5 tries).
2026-10-20 11:57:55 +02:00
admin 2954b0be3a Status view, part 1 (register pass): register 156 -> 152 (closed R-1121 file gone, R-1118 retirement done, R-231 in git + green run; R-993 merged into R-991); R-132 narrowed (rotation done, hash field left), R-1049 -> WATCHING (fix live since hub 0.151.0); hub + local read evidence
gates / gates (push) Successful in 12m40s
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-20 11:43:58 +02:00
admin 5c487313fa Night brief part 2 (run by day 2026-10-20): register 160 -> 156 (closed R-1001, R-1011, R-1167, R-1070, R-1010, R-893, R-1169; opened R-1171..R-1173; R-1019/R-462 narrowed, R-49 READY -> D21, R-1083 re-stated); 07 §6.3 option A as the design; STATUS morning list; report part 2; P4 Immich plan, P7 release evidence
gates / gates (push) Successful in 12m57s
2026-10-20 11:06:00 +02:00
admin bcaa9576b2 Night 2 evidence: P2 R-893 option A live proof on 9202 (forced failure put back exactly; normal restore OK; teardown); P5 R-462 coverage and currency list
gates / gates (push) Successful in 12m48s
2026-10-20 10:47:37 +02:00
admin 301f4e9e92 Night 2 (2026-10-20) evidence: P1 hub 0.162.0 deploy, installer 1.35.0 publish, agent 0.161.0 release+delivery+host_busy live; P2 R-893 red-proofs; P3 R-1019 classifier before/after
gates / gates (push) Successful in 13m14s
2026-10-20 09:32:53 +02:00
admin b0ae2c574a installer 1.35.0 published (operator yes, 2026-10-20): both webpage git-sync pins -> installer-v1.35.0 (R-1167)
gates / gates (push) Successful in 13m6s
2026-10-20 08:13:57 +02:00
2342 changed files with 184735 additions and 923 deletions

No files matched your search

+1 -1
View File
@@ -38,7 +38,7 @@ to probe a refusal at all.**
## If a probe does create something, remove it through the product
Not by hand, and not by `docker rm`: stop it, then `POST /api/stacks/<name>/remove` with
`remove_hdd_data` and `remove_backups`, and then **verify** — no container, no `/opt/felhom/stacks/<name>`,
`remove_hdd_data` and `remove_backups`, and then **verify** — no container, no `/opt/docker/stacks/<name>/app.yaml` (the folder itself stays — it holds the catalog's template files; path corrected 2026-10-24),
no volume. Say in the report that it happened. A tidy-up nobody is told about is how the next session
learns nothing.
+8 -1
View File
@@ -138,7 +138,14 @@ something, not only sessions that touch `documentation/` — which is why it is
and the eleven names head `OPEN-ITEMS.md`, and `register_shape_gate.py` refuses anything else.
- **Root `STATUS.md`** — at the end of every session in which something shipped, broke or was
decided. It is a **view** of `OPEN-ITEMS.md`; nothing may exist only there. One screen, written for
the operator in plain language, and deliberately **not** `CONTEXT.md`.
the operator in plain language (STE English), and deliberately **not** `CONTEXT.md`. How to update it:
1. **Replace** the „Updated" paragraph (at most 3 sentences, the newest change only) — never prepend; the old one
moves to `documentation/backlog/STATUS-HISTORY.md` under a dated heading.
2. A morning list goes under a dated sub-heading in „Waiting on you — do".
3. Remove what you finished: a done „do" item, a fixed „broken" line, a decided row (its ruling is in its row first).
4. A decide row: the question in one answerable sentence, at most 2 options, CC's pick first with the reason, the
other option's cost, what happens if the operator does nothing, and who is blocked.
5. `scripts/status_view_gate.py` checks it on every push (warn-first until 2026-10-27, then it refuses; R-1174).
- **The golden, weekly** (operator ruling 2026-10-15 11:26, `09` §3 decision 250; decision 7 stands): **a controller
release needs no golden** — the floor carries it, MinAgent declared, after the three test boxes (`09` §3 decision 250
item 2). The golden is baked **every Sunday night** when the floor is above it (`runbooks/nightly-rotation.md`) and
+241
View File
@@ -16,6 +16,247 @@
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
> **2026-10-25 (attended release run) — D49 = A, D44/D45 = all picks, D46 = yes; released: hub 0.170.0, agent 0.165.0,
> controller 0.343.0, catalog `ebbf1f4`** (`09` §3 decisions 258–261; `REPORT-release-2026-10-26.md`). **D49 = A:** a
> restore whose app does not start keeps both copies and never pours back by itself (`07` §8.2e). **Fix A** (controller
> `restore_verdict.go`): „started" = running on every sample of 45 s, no `RestartCount` growth, every healthcheck
> `healthy`; the undo copy is released only after 3 more minutes of the same (10 s samples, 30 min watch, a newer restore
> ends the watch — `restoreBegunAfter`). **Fix B:** the `.fab` import pulls first (`Exporter.SetImagePreparer` →
> `PrepareRestoreImages`). **D47 follow-up:** a set-aside mails the operator (`backup_run_failures`, leg
> `unit-set-aside`) — its hub subject still says „failed to back up" (R-1287). Agent 0.165.0 vouched 18:38 CET; on a
> BYO host our SSH drop-in is removed only on the bundle run AFTER the one that installs the new wrapper (R-1285).
> Installer 1.36.1 on main, not published (R-1284: `http_code` printed the code twice). R-1250's live proof is a dated
> check on 2026-10-27 (the agent tests only on its due Tuesday 10:00–15:59).
> **2026-10-25 08:38 — operator rulings by starting the R-1269 brief: D47 and D48** (`09` §3 decisions 256, 257;
> `audits/restore-truth-2026-10-25/README.md`). **D47:** an existing recovery unit is never changed in part — its
> definition and data are written only together, by a backup of the same install; a unit an earlier install left is set
> aside whole and stays restorable (`07` §8.2d). **D48:** every restore path resolves and pulls every image its
> definition names before the first change to the app; a failed pull is a refusal with the app untouched (`07` §6.6).
> **Confirmed, no change:** `07` §6.6's version rule; R-698 and R-933. Built the same day, controller commits only (rides
> v0.343.0): `02daebe` (D47 / R-1269), `b680db8` (R-1270: a restore whose app does not start is never a success; the
> safest reading — no automatic pour-back, the undo copy kept, a late start turns it into a success), `2d71270` (D48),
> `bd9f4f7` (R-1271: a copy dated more than 10 minutes in the future is never „the newest").
> **2026-10-24 (afternoon, attended) — two DooPlex backup alarms — DONE.** One cause: the 2026-10-23 window reboot
> restarted `postgresql-2`; CNPG 1.29 sets the role `postgres` from the superuser secret, then from the
> `bootstrap.initdb` owner secret (`owner: postgres`), so `postgres-superuser` is in force at every start; the dump
> CronJob read `postgresql-superuser` → 28P01 from 16:00Z → `PostgresqlDumpStale`; the 00:20 Gitea push refused the
> stale dump → `DooplexGiteaOffsiteStale`. **Operator rulings 2026-10-24:** (1) fix the CronJob only (homelab
> `531023d`; the Cluster keeps two writers — homelab list); (2) apply the routing change by restarting Alertmanager
> (its `envsubst` init container makes `/-/reload` useless; homelab `22d7697`: critical 4 h, warning 12 h, one mail per
> backup group and per drive); (3) the R-1250 live proof may use DooPlex's k3s qBittorrent as the downloading peer
> (one private LAN torrent in its container `/tmp`, deleted with its files). R-1250 committed (agent `29bcf44`,
> `1e087bb`; controller `046b808`), not released. R-934 closed (the disk was pulled 2026-10-23). Catalog `7de554e`:
> the box walk sends the drive itself as HDD_PATH (R-839). Report: `REPORT-dooplex-alarms-2026-10-24.md`.
> **2026-10-24 (attended release run) — DONE.** Hub 0.168.0 (05:39Z) and 0.169.0 (07:51Z) deployed; agent 0.164.0
> released, delivered to the three test boxes and vouched by the operator; controller 0.342.0 the global floor (08:29Z,
> MinAgent 0.158.1). ep0 runs `felhom-iperf3` (tunnel only). First line tests: Tester 1 94.3, demo-hp 94.4, demo-felhom
> 92.6 Mbit/s; demo-felhom's qBittorrent limited to 46 Mbit/s. D40 in effect (dated check 2026-10-25, R-1243). Opened
> R-1249 (the start-up VACUUM), R-1250 (a seeding qBittorrent is never tested). Report `REPORT-attended-release-2026-10-24.md`.
> **2026-10-24 07:22 / 07:26 — operator rulings by starting the attended release brief (D40, D41, D42; the line test).**
> **D40 (R-1243) = stop the Longhorn backup of the hub's volume**; the encrypted 6-hourly copy to ep0 stays the hub's
> off-site copy; the operator deleted the two old plain Longhorn backups himself (UI); the Longhorn UI cannot take the
> volume out of the backup job, so the change is in a manifest. **D41 (R-1245) = 50 %** of the measured line for
> qBittorrent's default upload limit. **D42 (R-1246) = the family is told only about the torrent limit**;
> `app_unexplained_upload` / `app_unexplained_cpu` are operator-only. **Line measure (R-1247) = a weekly iperf3 upload
> test from the box's host to an iperf3 server on ep0, inside the WireGuard tunnel only** (operator 07:26; his design of
> 2026-10-23: „during a quiet window the box measures the upload bandwidth"); the passive floor stays as a lower bound.
> This is the one change on ep0 the run may make (Tier 2 — the operator's yes is starting the run). CC still stops
> before each hub deploy, before the global controller floor, and for the agent vouch.
> **Two more answers in the same run (AskUserQuestion, 2026-10-24 ~08:2x CEST):** (1) **the agent speaks iperf3 itself**
> (no host has the iperf3 program; nothing is installed on hosts) — agent `internal/linetest`; (2) **the live hub file's
> old token bytes are accepted, not cleaned** (the start-up VACUUM did not finish on the Longhorn disk, R-1249; a clean-up
> needs an 8–20 min hub stop, a trim and the volume's snapshots deleted; only root on DooPlex can read the volume, and it
> holds the key; every copy that leaves is clean) — written into `05` §16.2 with the rule „a hand copy of hub.db is made
> with VACUUM INTO, never cp/tar".
> **2026-10-23 19:22 — operator rulings by starting the weekend-night brief (R-1241's Felhom half; the upload watch).**
> **Part 1 = R-1241's Felhom half:** seal R-975 (done in hub 0.168.0, not deployed — `05` §16.2,
> `audits/hub-token-seal-2026-10-23/`); the hub's plain Longhorn copies are decide row R-1243 (CC's pick: take the hub
> volume out of the Longhorn backup; ep0's encrypted copy is the backup). **Part 2 = the upload watch, shaped by the
> operator:** the box measures its own upload line (first from what it already sees — the off-site backup's upload
> rate — a short test at a quiet hour only if needed); on a sustained high upload it attributes the bytes to an app's
> container and ASKS THE APP (qBittorrent's seeding, Plex's remote sessions); a mismatch is the alarm
> (`app_unexplained_upload`, the same shape for CPU, `app_unexplained_cpu`) — operator only. qBittorrent gets a default
> upload limit from the measured line; a limit the household set is respected. **Open for the operator, CC's pick in
> STATUS:** the default limit's size (CC: 50 % of the measured upload) and who is told (CC: the family only about the
> qBittorrent limit they can change; the unexplained alarms to the operator only — the R-388 rule).
> **2026-10-24 (night, unattended) — the restore and backup holes; operator ruling D43 = A**
> (`audits/restore-holes-2026-10-24/README.md`). **Ruling D43 = A (operator, 2026-10-24 evening, by starting the run):**
> the root-key-only sshd drop-in applies only in appliance mode; BYO hosts keep their SSH settings (`11` §1, `03`).
> Fixed on main, nothing released: controller `bb2809d`/`062f9b4` (R-1251: a pending put-back blocks every start — a
> restore hold `putback_pending`; the put-back runs first when the drive binds; a failed put-back holds; the copy synced
> before the marker names it), `518a135` (R-1252: reconcile only under a valid classification), `c9b4b82` (R-1253: a
> stop-first export refuses while another operation holds the marker); hub `dcf4f84c` (R-1254: escrow in one
> transaction; image built locally, not pushed); agent `c8d3c17` (R-1257: D43). Proven: R-1251 on Tester 1 (power cut,
> disk off, disk back), R-1252 and R-1253 on 9202. D44/D45 prepared: `audits/review-round-2-2026-10-24/D44-D45-picks.md`.
> **2026-10-24 (evening, unattended) — review round 2** (`audits/review-round-2-2026-10-24/README.md`). The 17 architecture
> documents checked against `main`: facts corrected at `file:line` (EDITS.md), rulings and promises left for the operator
> (R-1266, D45), undocumented decisions written as notes (R-1265, D44). New: **R-1251 (P1)** a cut restore of a drive-backed
> app is put back over newer data (shown with the real `Recover`); **R-1252 (P2)** a rejected backup block makes the
> second-drive reconcile delete the household's copies (reproduced on 9202). Controller `b01bb4b` (unreleased, next
> v0.343.0): a media restart skips household-stopped apps (R-721); the FileBrowser SSO header file appears whole.
> **2026-10-23 ~12:00 — operator rulings by starting the D39 brief (`09` §3 decision 255; R-1239 closed).** **D39 = B:** a
> `security_fix` ladder step — box walk on 9202 required, bench optional, taken by the night leg. **No operator word per
> incident:** a public advisory id (CVE or GHSA) + its https:// notice is the authority; the session that writes the step
> reports it to the operator after the push. The design (Claude.ai's recommendation) was taken as written; one fact in the
> brief was checked and held (Plex's box fixture exists but is a `NoRoute` stub, so `seed: "none"` is right). Built:
> catalog `3898593` (`ladder.py` `check_security` + `engine_majors_crossed` importing `check-engine-major.py`; the move gate
> accepts it; `check-test-record.py` WARNs on a head with no bench; `upgrade-test.py --write-ladder --security`;
> `box_walk.py --security` + `refresh_box_mirror()`), controller **0.341.0** (`legCandidate` takes `proven` or
> `security_fix`; global floor 11:24 UTC with MinAgent 0.158.1, the operator's yes in chat). Proof:
> `audits/security-lane-2026-10-23/`. Older controllers skip the step at night and take it on a hand press (proven).
> **2026-10-23 10:56 — operator ruling: Plex to the fixed version now, without the test record (Plex2Shell).**
> Verbatim: *„bump the version to the newest in the app template immediately (as there are no customers yet), I want the
> demo boxes on the newest version asap."* It skips `09` §3 decision 13's test record **once, for Plex only**. Done: catalog
> `6cc317d` (Plex 1.41.4.9463-630c9f557 → 1.43.5.11029-6bf8af3f0, CVE-2026-96652/-96654/-96655/-96656), pushed
> `--no-verify` past `check-test-record-move.py` (its refusal is in `app-catalog-felhom.eu/REPORT.md`); the only box running
> Plex (demo-hp 9201) was checked for the attack's signs (none) and updated by the Update press; the guide re-checked
> (`guide.checked_version` moved). Decision 13 itself is unchanged. Follow-ups: R-1238 (Plex's test record and fixture),
> R-1239 (a fast lane for security fixes — the operator's decision, D39). Evidence: `audits/plex-demo-boxes-2026-10-23/`.
> **2026-10-23 morning — the operator's „go" for the day brief: the releases, D31 as built, D32–D36 (decide rows in `STATUS-HISTORY.md`).**
> **Releases:** controller 0.340.0 (global floor 06:41 UTC, MinAgent 0.158.1); agent 0.163.0 (binary sha256 561b5316…, bundle 3afb88d9…, vouched by the operator 07:58 UTC; delivered to the three test boxes by signed jobs through a step bundle `0.163.0-step1` — the D4 drop-in is a new path, R-880); installer 1.36.0 (tag `installer-v1.36.0` → 83388c08, both webpage pins); hub 0.167.0 (sha256:177860a9…, 08:27 UTC); the website `0bb57a4f`.
> **D31** = as built (the re-issue reads the newest deletion of a KEY-HOLDING host; an install that got the password but no recovery package is rotated without a press). **D32** = yes: the sheet is names and places only, one A4, HU + EN; the operator prints and fills it. **D33** = yes: `scripts/release-copy/` nightly 01:40, newest 3 image tags (controller, hub, samba) + 5 agent versions, skopeo `dir:` (digest-true — why not `docker save`: a re-push re-compresses), ep0 `/srv/felhom-release-copy` under the one limited user `felhom-relcopy` (rrsync), cap 3 GB enforced by the sender, a read-only registry token of its own. **D34** = yes: hub snapshots at 02/08/14/20 (catch-up > 6 h), DooPlex push :30 after, limit 7 h + a wait for a snapshot being written (R-1234: 2 h was wrong), alarm 13 h; ep0 retention unchanged (keep-daily 14, keep-weekly 8). **D35** = yes, after the release week (R-1228, dated 2026-10-30). **D36** = yes, both sentences, published.
> **Added on the operator's word the same day (not soaked):** D34's hub half went into 0.167.0 (asked before the build). Record: `audits/day-2026-10-23/`, `REPORT-day-2026-10-23.md`.
> **2026-10-22 evening — operator rulings D29, D30, D31 (CC's picks accepted; the decide rows now in `STATUS-HISTORY.md`).**
> **D29** = no: the controller repository stays public for now (R-925). **D30** = the agent makes a new WireGuard key and
> registers it on the existing peer path (R-1204). **D31** = the hub skips a deleted install that never made a key when it
> decides the automatic PBS re-issue (R-1205). If the operator reverses one in the morning, its commit is reverted before
> release (night brief 2026-10-23 §0).
> **2026-10-22/23 night — what was built (night brief; nothing released, deployed, published or vouched).**
> **Stray-copy fix** (R-1214/R-1180): controller `66a69d4` + `912e592` — the capped copy's container has a fixed name +
> label `felhom.tier2-copy=1`, is removed by name on any error, every leftover is removed at start before the scheduler
> and web server, a restore is refused while a copy runs (`backup.Tier2CopyBlocksRestore`, in-memory record + a stray
> flag until the start-up removal succeeds); `--mount`, image ID, `--pull never`, `--no-healthcheck`. Proven on 9202 rc.5.
> **D30:** agent `ae69ad4` (`wgtunnel/rekey.go`, 15 min, once an hour, `private.key.prev`); hub `84751cb0` pins the
> alarm clearing. **D31:** hub `84751cb0` `store.LatestKeyHoldingHostDeletion` (hub 0.167.0 source, NOT built — a hub
> image build is the morning's, with the deploy). **Review fixes:** controller `943c517` (C5 catch-up releases the
> whole-guest hold before its checks; C3 text; C6/C8 wording); the D4 patches v2 (`audits/night-2026-10-23/P3-review/`).
> **Images:** controller `0.340.0-rc.5` (= `66a69d4`) and `0.340.0-rc.6` (= `943c517`); no hub image tonight.
> **Survey:** `audits/dooplex-survival-2026-10-22/PLAN.md` (G1–G10; the three biggest: the sheet, the registry, the
> home line). Report: `REPORT-night-2026-10-23.md`.
> **2026-10-22 15:47 — operator rulings, „all picks" (the decide table, rows now in `STATUS-HISTORY.md`).**
> **D3** = change the zero-knowledge wording now, look again after the first sale (R-985). **D4** = root SSH from the
> LAN by key only; the console password stays (R-982). **D5** = approve the minimum machine, `audits/sizing-2026-10-13/SIZING.md`
> §5 (R-991). **D6** = a read-only login for the scratch box's drill catalog, not a public drill repo (R-992). **D8** =
> DooPlex Docker settings from git plus a weekly build-cache prune (R-206). **D21 = B** — only Immich's model cache
> leaves the backups now; the raw DB folder and Immich's own backups after their restore proof (R-49). **D27** = report
> only: the agent adds no host swap and the installer refuses nobody; reopen at the first BYO box short of swap (R-1207).
> **2026-10-22 evening → night — the rulings put into effect (night brief; nothing released, deployed or published).**
> **D8 live:** homelab-manifests `b35ba83`, role `docker_daemon` + `docker-buildcache-prune.{service,timer}` (Sun 23:30,
> `docker builder prune --force --filter until=168h`, never `-a`); the live `daemon.json` was identical → no Docker
> restart. **D6 live:** Gitea user `drill-reader` (not admin), read collaborator on `admin/app-catalog-drill`, token
> `read:repository` in `sec`; proven (read 200, write/push 403, other private repos 404); `09` §6.5. **D21 built:**
> controller `6a17c95` — `.felhom.yml` `backup.volumes: [{name, class: excluded}]` → `appbackup.ExcludedDockerVolumes`
> (declared names only; refuses to exclude all of an app's named volumes) → `backup.Manager.dumpableVolumes`; catalog
> `ce33e3b` (Immich, live, inert before controller 0.340.0); 9202: unit 1.16 GB → 337 MB, Immich ML answered after the
> cache was removed. **D3/D5:** the volunteer guides committed (`0639cf07`); the website change is a ready patch
> (`audits/night-2026-10-22/P2-D3-D5/website-D3-D5.patch`) — a push to `website/` publishes. **D4:** two ready patches
> (`P2-D4/`): an agent config-bundle drop-in `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`
> (`PermitRootLogin prohibit-password`; Debian's Include comes first and the first value wins; felhom-sshd :8822 has its
> own config) and the installer's uninstall; proven on a fresh ISO 1.30.0 install (VM 9343, destroyed; its appliance 49
> waits for the operator's Discard). **Data-loss candidates** (`audits/night-2026-10-22/P3-candidates/analysis.md`):
> R-1187(c) fixed (`d5f88a2`: an import over an installed, not-running app is refused — 9202 reproduced the merge on
> rc.1 and the refusal on rc.2); R-1189(d) fixed (`nightchain.CatchUp.After` runs the off-site proof + integrity check
> once); new rows R-1218..R-1222 with their proposed fixes. **R-1214** (03:30 pressure): the copy's page cache is
> charged to the guest; `backup/tier2.go` now runs rsync in a 512 MB memory-capped container of the controller's own
> image for clean `/mnt/` paths with no symlink (`03c0a69`, `ae39467`; exit 125 → plain rsync); 9202 stall 5.7 s → 0.37 s.
> **R-1204:** agent `a0c88e5` reports `handshake_future_s`; hub 0.166.0 (`e267c5e2`, built, not deployed) mails
> `wg_tunnel_stale`/`wg_tunnel_ok` to the operator only. **Repos private (R-925):** only felhom-controller can go
> private (after a CI read token); STATUS D29. Picks waiting: D29 (No), D30 (re-key through `RegisterWGPeerForHost`),
> D31 (R-1205: skip an un-acked, key-less deletion). Record: `REPORT-night-2026-10-22.md`.
> **2026-10-22 ~14:50 — operator acts, checked by CC (read only).** Host `Tester-2-be8404` deleted (12:54:47 UTC);
> customer `Tester-2` language set to Magyar; customer `Tester-2` RESET (12:55:56–12:56:04 UTC, journal #25, every leg
> ok). The first live run of R-32's purge-first leg: logged in with the sub-account's own password, found no repository
> (0 paths — the box never backed up off-site), deleted sub-account 322460; ep0 namespace + its 2026-10-04 copy + prune
> job + token gone. No other customer touched: Storage Box counts after = the hub's 05:10 listing (demo-hp 303,
> tester-1 62, demo-felhom 43; Tester4 has no box — compared by the hub on 10-23 05:10), ep0 after = the 03:30 prune
> keep-lists (2/2/2/1, operator 27). No false alarm. `audits/tester2-reset-2026-10-22/`.
> **2026-10-22 14:11 — operator ruling: Tester 2 is a NEW INSTALL, not an upgrade.** The friend reinstalls the laptop
> from the public download (installer 1.30.0 or newer) with the new drive in before the install; the old box's data is
> not needed („he barely used the laptop, no useful data on it"). Read on the hub the same day (read only): no sealed
> key package; off-site provisioned (shared) but its key never deposited; one ep0 whole-box copy (2026-10-04); the mail
> language is English. CC's pick for the operator: host delete, then customer RESET (it empties the Storage Box folder
> and the ep0 namespace — the operator's press). R-862 and R-1139 close at the reinstall; Tester 2's RESET is R-32's
> first live purge. List: `documentation/pilot/TESTER2-handover.md`.
> **2026-10-22 day — operator rulings (11:42) and what was built.** Rulings: **D28 = A** (loosen the host-swap check);
> agent 0.162.0 **vouched** (read back: `artifact_agent_version` 0.162.0 at 07:57 UTC); Tester 2's laptop was installed
> with **our installer**; Tester 2 is off, no date. Built: **R-1199 reopened** — controller 0.338.0's start-up default
> push overwrote demo-felhom's own saved mail list (13 → 17 events, 07:54:55 UTC); hub **0.165.0** marks a list the
> household saved (`household_saved`, backfilled once: Tester-2, Tester4 false; the other four true) and never lets a
> default overwrite it; it answers a box's self-scoped read; controller **0.339.0** reads the hub's list when it has
> none, adopts a household's, pushes its default once only when no household saved one. demo-felhom's list restored
> from the hub's 00:00 UTC snapshot through the hub's own save route; all six rows equal the snapshot after. Found and
> fixed with it: an empty saved list read back as the 17 defaults (`omitempty`; Tester 1 and demo-hp hold that shape),
> and the add-only seeds ran on a new box's second start. **R-1216 (D28 = A):** short host swap = none, or below 90 %
> of min(guests' swap, 8192 MB) — the installer's maximum (`Install.pm compute_swapsize`); one case table in hub and
> controller, `scripts/host_swap_rule_gate.py`. Tester 2 hand-over: `documentation/pilot/TESTER2-handover.md`.
> **2026-10-22 morning — the operator's five yeses, carried out.** Controller **0.338.0** released (image
> `sha256:e8a529f3…`, MinAgent 0.158.1): Tester 1, demo-hp, demo-felhom through per-customer floors, checks green
> (https-only 28/28, 0 unhealthy, 0 errors), then the global floor 0.338.0 (read back with the WAL). Agent **0.162.0**
> released (`5fd7a23b…` = the soak's bytes) and delivered by signed job to the same three; **the vouch is the
> operator's**. Hub **0.164.0** deployed (only `Deployment/hub` synced). ISO **1.30.0** (W1, W7) built, gated (G1–G16,
> G12 not re-read) and proof-installed from both menu entries on demo-hp, published, round trip `590d7b9b…`; both
> download pages point at it and carry the W2 button. Operator answer: Tester 2's laptop was installed with our
> installer (R-1207). **Found:** the host-swap check (agent WARN, hub `host_swap_short`, the controller's install
> sentence) compares host swap with the guests' WHOLE swap allowance, so every box with a guest above ~8 GB trips it
> (demo-hp, demo-felhom mailed) though the soak burst used 2.7 GB — R-1216, decision D28. The runbook's G16 command
> was stale (now `scripts/iso_bootstrap_gate.py`). Record: `REPORT-morning-2026-10-22.md`.
> **2026-10-21 night (unattended; nothing released, deployed, published or vouched).** Operator rulings recorded:
> **D22 = yes (20:05)** — D20's put-back extended to the local unit restore, the second-drive restores and a restore cut by
> a power loss (`07` §6.3, the 2026-10-21 `[DESIGN]` paragraph; controller `d582feb`, proven on 9202 with 0.338.0-rc.1:
> a damaged unit → the live data back and running; a controller killed after the volume step → put back on start);
> **D26 = A (20:03)** — release and vouch the agent with the swap fix in the morning; a host-swap warning follows (agent
> `0da792e` reports `host_swap_mb`/`guest_swap_mb`, one WARN; hub 0.164.0 built, not deployed: operator-only
> `host_swap_short`; controller install-page sentence). **Decided by CC unattended — operator may reverse (R-1199):** the
> box's notification page is the source the hub mails the household by: a box whose household never saved a choice now
> pushes the default list its page shows at start (no address; the hub's no-clobber guard keeps the stored one) —
> `08` (the household decides; „the box's page is the source"). Measured: Tester4's hub list was saved 2026-10-18 and
> lacked disk warning, drive disconnected and second-drive failure, so the household got 1 alarm of 9. Built tonight:
> controller `0.338.0-rc.2` (`sha256:ae52c93e…`, all of tonight's controller work, smoke-tested on 9202, 9202 back on
> 0.337.0), hub `0.164.0` (`sha256:931594e3…`, manifest still 0.163.0), agent candidate `0.162.0` from `0da792e`
> (sha256 `5fd7a23b…`, not published). The 8 GB soak with the swap fix (`audits/night-2026-10-21/P1-soak/`): 11 h, no
> freeze, 0 failed health probes (largest interval 16 s), every night job ran; memory „full" above 20 % for ~3 min in the
> 03:30 tier2 job (R-1214); 1 photo of ~26 000 not stored (R-1215). With `swapoff -a` on the host the freeze came back
> within 20 s (dead up to 289 s; `P2-noswap/`) — the fix depends on host swap (R-1207).
> **2026-10-21 (day brief, before Tester 2) — hub 0.163.0 DEPLOYED (operator yes, 11:46 UTC); controller 0.337.0 released
> (three test boxes 14:06 UTC, global floor after the freeze proof); agent `286a7e3` committed, NOT released.** Operator rulings
> today: **D24 = yes** (the hub's DST double run fixed: `nextDailyRun`, calendar day+1, `05` §16.5); **D25** (a second-drive
> copy whose SOURCE is gone is kept until the household releases it — `07` §8.2a; controller `tier2_kept.go`, the restore
> page's „Megőrzött másolatok"); the **R-1176 shape** = the hub CARRIES the household's retained sealed package to the
> rebuilt host (`store.ReattachRetainedEscrow`: one live host, no package held, the newest escrow-ACKED deletion; moved in
> one tx at the agent's first report and the controller's report; never opened/logged; two live hosts → refused + operator
> mail) — `03` §8a. Proven live on demo-hp (Tester4 from the public ISO): no fresh key minted, recovery offered, wrong code
> refused, right code opened the history, Nextcloud 3 031/3 033 files byte-identical, off-site appended 63 → 64
> (`audits/fixes-2026-10-21/P1-live/`). R-1182: retained rows carry `repo_tag` (NULL legacy / 'live' / the set-aside path
> tagged at the hub's move-aside); the abandonment purges only rows tagged with a path the hub DELETED; the ACK says
> `abandon_purge_done`. R-1203: the new-link page tells the holder of a real spent link that the old box is still linked;
> operator mail `bind_old_host_linked`. **R-1192 diagnosed with numbers:** Proxmox's LXC default swap cap (512 MB) — the
> agent never set `swap` — turned Immich's model load (+1.15 GB) into a code-page refault thrash at the 4096 MB guest limit
> (150 000–600 000 refaults and ~1 GB of reads per 10 s, memory „full" 92–95 %, load 155, dashboard dead 6½ min, 0 OOM
> kills); `pct set --swap 4096` live ended it within a minute. Agent fix (swap = RAM at bring-up, at RAM resize, once at
> start) committed; its release is the operator's. The controller's „Normál használat mellett ez nem okoz problémát" is
> gone (0.337.0). Observed and filed: R-1205 (PBS re-issue after an un-acked second install), R-1206 (future-dated
> snapshots). Report: `REPORT-fixes-2026-10-21.md`.
> **2026-10-19 (night) — controller 0.335.0 is the floor (MinAgent 0.158.1, 18:39 UTC); hub 0.162.0 BUILT, not deployed;
> agent `56aabc7` and installer 1.35.0 committed, not released/published.** R-921 (operator answer „add one field"): the
> agent serves its host-wide heavy-operation gate as `host_busy` on `GET /backup/status`; the controller reads it before
+89
View File
@@ -0,0 +1,89 @@
# REPORT — attended release run 2026-10-24: the hub's tokens locked, the hub out of the open backup, the iperf3 line test, the watch released
Brief: „the hub's tokens locked, the hub out of the open backup, an iperf3 line test via ep0, then the upload watch
released" (operator rulings 07:22 / 07:26, `CONTEXT.md`). Read first: `05` §16.2, `06` §4.5 / §5, `07`, `08` §9, the
spike `audits/SPIKE-upload-watch-2026-10-23.md`, the felhom-build-deploy skill. Evidence: `audits/line-test-2026-10-24/`.
## Five lines
1. **Hub tokens:** sealed in hub 0.168.0 and still working (demo-hp is served the same tunnel and API tokens it runs; the
template fetch works; the next ep0 copy holds 0 of 8 token values). The live file keeps old bytes in free space — your
call: accepted, written down, with the rule „copy hub.db only with VACUUM INTO".
2. **The hub is out of the open backup:** the hub volume reads `default: disabled`; the folder holds no copy of it (your
two deletions worked; nothing else is left). Tonight's job is the last proof (dated check 2026-10-25).
3. **The line test:** demo-hp measured **94.4 Mbit/s** (Tester 1 94.3, demo-felhom 92.6) — the night backup said „at
least 10".
4. **Live where:** hub 0.169.0 and controller 0.342.0 run on all three boxes (the global floor); agent 0.164.0 on all
three (vouched); demo-felhom's qBittorrent is limited to 46 Mbit/s. Only these three boxes report to the hub.
5. **You still do:** nothing for this run. Two rows wait for a design (below).
## What I asked you, and what you said
- Deploy hub 0.168.0 → yes. Deploy hub 0.169.0 → yes. Global floor 0.342.0 → yes. Agent vouch → done.
- The live file's old bytes → accept and write down (my first estimate of „2 minutes" was wrong; the disk writes ~1.3
MB/s, so a clean-up means an 8–20 minute hub stop).
- No host has the iperf3 program → the agent speaks iperf3 itself (nothing installed on hosts).
## Versions, digests, CI (by commit)
| Repo | Commit | What | CI |
|---|---|---|---|
| felhom.eu | `00031150` | hub 0.168.0 manifest + rulings | 2275 success |
| felhom.eu | `3363ffa8` | D40: the PVC label | 2276 success |
| felhom-agent | `f7e1dcb` | the line test | 2277 success |
| felhom-controller | `ed1011b` | the tested line in the watch | 2278 success |
| felhom.eu | `1a788f33` | docs, register, ep0 unit, hub `line_test` model | 2279 success |
| felhom-controller | `42cda07` | the start-line wording | 2283 success |
| felhom-agent | `08ca678` | CHANGELOG released, REPORT | (docs) |
| felhom.eu | `01bf4e83` | hub 0.169.0 manifest | (see the final push) |
| felhom-controller | `bddc477` | CHANGELOG released, REPORT | (docs) |
- **Hub 0.168.0** `sha256:3bfdecaa…` (from `5277d859`), deployed 05:39Z. **Hub 0.169.0** `sha256:fb9b2489…` (from
`1a788f33`), deployed 07:51Z; 30-minute watch: only info events, 0 errors.
- **Agent 0.164.0**: binary `1b803d8b70f8afbe1546f6989f35b6b0e29c9ed63f6acb3843d3c04ef0d24b2b`, bundle
`1d7b54d39d0c1aff682ba0c0a7f91aea0f9407a10d2f5a8bbfdc858d32c13341`, tag `v0.164.0`; signed `agent_update` to the three
boxes, self-check 76/76; vouched (hub page + `Artifact manifest set` line).
- **Controller 0.342.0** `sha256:d002d3b7…` (from `42cda07`); rc.2 on 9202; per-box floors → healthy on all three, box
checks and https-only green; global floor 08:29:19Z, MinAgent 0.158.1, read back on the hub log and page.
- **ep0:** `felhom-iperf3.service` (iperf3 3.18-2+deb13u2, already installed since 2026-07-03), one nftables line;
no answer on the internet (`:22` as the control), an answer in the tunnel. CVEs: four fixed in this build; two open
„minor" server DoS issues bounded by tunnel-only, TCP-only and systemd caps (`scripts/ep0-iperf3/README.md`).
## Rows
**Before 194, after 191; opened 2 (R-1249, R-1250), closed 5 (R-975, R-1244, R-1245, R-1246, R-1247).** R-1243 →
VERIFY (dated check 2026-10-25); R-1241 keeps only the homelab half.
## Observations
- The start-up VACUUM after a legacy seal cannot finish on the Longhorn disk; its comment said 2.8 s — FILED: R-1249.
- A qBittorrent that seeds all day keeps the uplink busy, so that box is never tested and gets no limit — FILED: R-1250.
- The live hub file's old token bytes — NOT-A-FINDING: accepted by the operator 2026-10-24 (`05` §16.2), every copy that
leaves is clean.
- Two pre-deploy hub snapshots in `/data/snapshots` held plain tokens — NOT-A-FINDING: they roll off (one already pruned
at 06:06Z; the other at the 12:00Z snapshot); same volume, same reader as above.
- The cross-repo wire gate refused the agent's new `line_test` field until the hub modelled it — fixed in the session
(hub 0.169.0 `sysfacts.LineTest`, the System page cell).
- My first ArgoCD sync used a wrong commit id — NOT-A-FINDING: it failed to fetch and changed nothing; redone.
- 9202 has no agent local API, so its controller cannot read a line test — NOT-A-FINDING: by design (scratch guest);
the qBittorrent proof moved to demo-felhom, where the product path runs.
- demo-hp's first watch minute could not resolve `plex` — NOT-A-FINDING: the known start-up race (2026-10-23).
- The family's line on demo-felhom's qBittorrent page was not seen live — NOT-A-FINDING: the household's password is not
CC's; the render is pinned by a test, and the text with the live value is in the evidence.
- Old rc controller images (other sessions) sit on 9202 — NOT-A-FINDING: a scratch guest; mine (rc.2) is removed.
- The hub's database is 530 MB, mostly report history (`host_reports` 305 MB, `reports` 128 MB) — NOT-A-FINDING here:
it is why a VACUUM is slow, recorded in R-1249.
## Teardown, three layers
- **Machine:** 9202 — the throwaway qBittorrent removed through the product (0 containers, 0 volumes), back on the
0.342.0 release, my rc.2 image removed. demo-hp's host — the probe binary deleted, nothing running.
- **Host / DooPlex:** the database copies I restored for the checks shredded; the 9202 password copy, the signed job
envelopes and the release clone shredded / removed. ep0 — the one planned change stays (`felhom-iperf3`); the backup
of the firewall file is `/etc/nftables.conf.bak-20261024-iperf3`.
- **Hub:** the three per-box floors cleared; the global floor 0.342.0 stays (the release).
## Instruction files, poster
No instruction file edited. System poster: a text-only change (ep0 also runs the tunnel-only line-test server) — the
facts file and the matching poster text edited in the same commit.
+67
View File
@@ -0,0 +1,67 @@
# REPORT — every catalog app on a fresh box, the way a family uses it (2026-10-24 → 25 night, unattended)
**Plain words first.**
- 65 apps: **52 work, 10 work with a problem, 0 broken**; 2 are not offered for a new install (by design); Wanderer
cannot run on a box without DNS (known, R-739).
- **The three worst:** (1) after „remove — keep backups" and a reinstall, restoring that backup breaks the app — the
household's item is gone from view (Tandoor, AdventureLog; R-1269, P2); (2) that restore still says it worked (R-1270,
P2); (3) „use my kept data" loaded another box's newer-dated database for Paperless-ngx, Nextcloud and Immich, and
Immich never answered again (R-1271, P2). In all three the data is not deleted.
- **Fixes made in the drill catalog, not live:** n8n stops sending usage data out; OpenGist gets 256 MB (at 128 MB one
sign-in hit the limit 443 times and swapped 98 MB). Both proven on the sweep box pointed at the drill.
- **Morning:** say yes to the two catalog fixes (in Monday's release list); decide D46 (Homepage behind the family gate —
CC's pick: yes); delete hub host Tester4-aa9466.
**Safest readings taken (the brief allowed no question):** ISO 1.30.0 (the current public one, not 1.36.0); 12 GiB VM
(the brief's minimum; demo-hp had 14 free); the kept-data question exists only for drive-folder apps, so volume apps
were checked by a unit restore after the reinstall (that made a unit restore on every seeded app, not every fifth);
„Mentés most" is local-only by design; backup/storage findings filed in their own category, not „Apps & catalog". All in
the audit's head.
**Table and evidence:** `documentation/audits/catalog-sweep-2026-10-24/README.md` (the 65-row table, the findings, the
R-1269 proof, D22 put-back PASS on PrivateBin and Vikunja, the drill proofs, teardown).
**Architecture read:** `07-backup-architecture.md` §6.5 (kept data) and §8, `16-storage-physical-identity.md` §5 (the
internal place is the default drive — F3 NOT-A-FINDING), `10-localisation.md`.
**Two mistakes of this session, both caught and kept in the evidence:**
1. The first drill pass ran on the live templates: I had believed the drill repo public; the box's clone failed and the
controller kept its templates. Re-run through the box's local mirror (R-992's method) with a positive check before
any install (`DRILL-CONFIRMED`). Memory note written.
2. I reverted the OpenGist fix on a wrong reading („it grows into its limit"); an automated security review flagged the
lowered cap; a cgroup measurement proved the fix needed and it went back in.
**Commits:** drill catalog (never live) `a601950` (n8n), `fd23571` → `c420053` (wrong revert) → `405077f` + `492a17f`
(OpenGist). felhom.eu: this commit (audit + tools + evidence, register, STATUS, STATUS-HISTORY, capability map B1 +
detail). CI by hash: in the final report.
**Register:** at the sweep's start 190 (other sessions filed 18 since); before this commit 208, after 217; opened 9
(R-1269–R-1277), closed 0. Notes added to R-739 (Wanderer, no DNS) and R-1267 (anonymous volumes: OnlyOffice and RomM
too; 52 left after every app was removed with data).
**Observations:**
- FILED: R-1269 (reinstall's capture rewrites the kept unit), R-1270 (restore says ok while the app never starts),
R-1271 (kept data loads another box's future-dated database), R-1272 (Homepage has no gate — operator, D46), R-1273
(registry rate limit shown as „exit code 1"), R-1274 (restore refusal advice when tiers exist but have not copied),
R-1275 (drive names in Hungarian on English pages), R-1276 (empty parent folder after remove with data), R-1277 (box
fixtures drifted: OpenGist, Calibre-Web, Mealie, Papra, Rallly, Vaultwarden 1.37).
- FIXED (drill, morning list): n8n usage data (`a601950`); OpenGist memory (`405077f`, `492a17f`).
- NOTE on existing rows: R-739 (Wanderer), R-1267 (anonymous volumes).
- NOT-A-FINDING: F3 install default is the internal place (`16` §5, by design); F4 English guide labels (the app's own
words; the Hungarian copy is frozen by the copy gate); F5 Kimai in English (a Hungarian browser lands on `/hu/`); F6
AdventureLog 95 % (docker stats with cache; anon 75 %); database first-start lines in SparkyFitness and Dawarich (no
restart, item back); Radicale IP redirect (curl without Host); NZBGet/Transmission basic-auth prompt in screenshots (the
apps' own login); Vaultwarden stranger 200 (its own login page; sign-up closed).
**Teardown:** machine — VM 9360 and LXC 9361 (both this session's) destroyed with `--purge`, no disk left on
`nvme-scratch`; Tester 1, 9201, 9202, the thin pool and the ISO untouched; the box's logs copied off first. Host —
DooPlex: the session's secret files shredded. Hub — host Tester4-aa9466 left for the operator to delete. ep0 — listed,
nothing removed: the host's PBS link was set up, no whole-box copy went to it; the app copies went to the Tester4 restic
repository on the storage box.
**Secrets:** read by name through `sec`; the evidence was scanned for every known secret value (positive control seen) and
for token shapes; one Actual Budget session token (a throwaway app) was blanked; screenshots spot-checked (passwords sit
behind „Show", never pressed). No password, key or token is in this commit.
**System poster:** no fact on it changed. **Instruction files:** none edited. **No** release, deploy, floor, vouch,
publish or live catalog push.
+70
View File
@@ -0,0 +1,70 @@
# REPORT — the day of 2026-10-22: the family's mail list, the swap check (D28 = A), the Tester 2 hand-over
**Operator rulings (11:42):** D28 = A; agent 0.162.0 vouched (read back: hub `artifact_agent_version` 0.162.0, 07:57
UTC); Tester 2's laptop used our installer; Tester 2 off, no date. Design read: `08-alarm-ladder.md`,
`05-hub-architecture.md`, `03-host-agent.md`. Baselines: controller `08fa440d`, agent `2d5840ba`, felhom.eu `a9283741`.
## Four lines
1. **Mail list fix live** on the hub (0.165.0) and on Tester 1, demo-hp and demo-felhom (controller 0.339.0, global
floor). A box with no list of its own now adopts the family's list from the hub; a default never overwrites it.
2. **One family list restored: demo-felhom** (13 events, from the hub's 00:00 UTC snapshot; this morning's release had
made it 17). After the release demo-felhom adopted it; all six hub rows equal the snapshot.
3. **Swap warning rule:** none, or below 90 % of min(the guests' swap, 8 GB). The two morning alarms cleared at the next
report (demo-hp 11:04, demo-felhom 11:09 UTC), no mail. No agent change.
4. **Hand-over list:** `documentation/pilot/TESTER2-handover.md`, linked from STATUS.
## Versions, digests, CI
| | version | digest | CI (commit → job) |
|---|---|---|---|
| hub | 0.165.0, DEPLOYED (operator yes; only `Deployment/hub` synced, manifest `25dae2f5`) | `sha256:c6180fb79a2ff7d5546a704c68b634b9207afb0a0ecd524c74b0310434532537` | `5ae5984c` → 2207 success; `25dae2f5` → 2208 success |
| controller | 0.339.0-rc.1 (9202) | `sha256:7d4c88528b50adeb891e409cf50343f44fb168a84e890d42864b8124f3051fdd` | `add310c` → 2206 success |
| controller | 0.339.0, released, global floor (MinAgent 0.158.1) | `sha256:fee71aafceb4a7d5f24ae4a6a273a8d51eb26e0550fe9d12d4d86955f6a343a8` | `fc5a20f` → 2209 success |
Agent: no change. Evidence: `documentation/audits/day-2026-10-22/` (rows before, the restore, rows after, the release
on the three boxes, https-only).
## What was built
- **Hub:** `customer_notifications.household_saved` — set only by the household's own save (`source: household`),
never cleared; any other push over a marked list → `{"status":"kept"}` (an older controller's push with neither
address nor clear counts as not the household's — that is 0.338.0's default push); `GET /api/v1/preferences`
(customer-scoped); one-time backfill (written after creation → a box wrote it): Tester-2 false, Tester4 false,
demo-felhom / demo-hp / tester-1 / `_resend-rotation-test` true. `host_swap_short` by the new rule.
- **Controller:** start-up — a saved list as before (R-922); none saved → read the hub, adopt a household's, else push
the default once and save it; unreadable hub → nothing. The swap sentence by the same rule.
- **Contract:** one case table in both repos, byte-identical, each run by a Go test; `scripts/host_swap_rule_gate.py`
with 4 decoys (`test_host_swap_rule_gate.py`).
- **Red-proofs seen failing:** hub A / A-legacy / C (refusal removed), the swap table and the clear (morning rule), the
backfill (marking off); controller A / B / unreadable hub (0.338.0's rule), the empty-list test (getter restored),
the swap table (morning rule).
## Rows
Before 178, after 178. Opened 2: R-1199 (reopened), R-1217 (demo-hp's hub list has 3 app events its page does not
show — P4). Closed 2: R-1199, R-1216.
## Observations
- An empty saved list read back as the 17 defaults (Tester 1, demo-hp) — **NOT-A-FINDING: fixed in this session**
(size rule; controller CHANGELOG „Fixed without a row").
- The add-only seeds re-added a just-switched-off mail on a new box's second start — **NOT-A-FINDING: fixed in this
session** (same entry).
- demo-hp: hub mails 3 app events the page does not show — **FILED: R-1217**.
- The cleared swap alarms sent no mail (`host_swap_ok` is info) — **NOT-A-FINDING: the ladder sends no operator mail
for an info clear; the event row is the record.**
- A docstring test refused my first push (`9c.` is not a gate-list number) — **NOT-A-FINDING: my numbering; fixed
before the push.**
- The safety check blocked one `sh -c` with a temp-file removal; rewritten with no file to remove — **NOT-A-FINDING:
nothing was run.**
- 9202 has no hub link, so the rc there proved start-up only — **NOT-A-FINDING: the mail path was proven on the three
test boxes through the floor.**
## Teardown
- **Machine:** nothing created. 9202 now runs 0.339.0-rc.1 (the scratch box; the floor moves it at its next report if
it reports — it has no hub link, so set it by hand next time it is needed).
- **Host:** the three test boxes on controller 0.339.0 — intended.
- **Hub:** one row written (demo-felhom's restored list, through `POST /api/v1/preferences`); global floor 0.339.0;
no per-customer floor left.
+103
View File
@@ -0,0 +1,103 @@
# REPORT — day 2026-10-23: the releases and the DooPlex hour (the operator's „go")
Design read: `07-backup-architecture.md` §6.3/§8, `09-update-architecture.md` §3 decision 250, `03-host-agent.md`,
`05-hub-architecture.md` §16.3, `runbooks/total-loss-of-dooplex.md`, `audits/dooplex-survival-2026-10-22/PLAN.md`.
Evidence: `documentation/audits/day-2026-10-23/01`–`14`.
## Five lines
1. Released: controller 0.340.0 (global floor), agent 0.163.0 (vouched by the operator), installer 1.36.0, hub 0.167.0,
the website with D36's two sentences.
2. Root SSH from the LAN is key-only on all three test boxes; new installs get it from installer 1.36.0.
3. The hub database now leaves every 6 hours (restored back byte-identical); the released images and agent packages
leave nightly to ep0 (pulled back, every sha256 OK, one image loaded on a scratch box).
4. One false failure mail at 10:28 (my install of the 6-hourly copy) — filed R-1234, fixed and red-proved the same hour.
5. Register 191 → 184; the operator: print the sheet, Hosts page clean-up, D37 + D38.
## Per box — versions before → after, and the SSH checks
| Box | controller | agent | root-file bundle | `sshd -T` root | sshd reloaded | key login | felhom-sshd | swap = RAM (9201) |
|---|---|---|---|---|---|---|---|---|
| Tester 1 | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 (via 0.163.0-step1) | without-password | yes, 09:27:38 | ok | active, `sshd -t` ok | 4096/4096 |
| demo-hp | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 | without-password | yes, 09:19:44 | ok | active, `sshd -t` ok | 25898/25898 |
| demo-felhom | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 | without-password | yes, 09:24:42 | ok | active, `sshd -t` ok | 12288/12288 |
| hub | 0.165.0 → 0.167.0 | | | | | | | |
Not proven (said plainly): the real root password over SSH (CC does not hold it — a dummy one is refused, but OpenSSH logs
only „Failed password"); a felhom-sshd connection (it admits only the operator's tunnel peer); the physical console (the
drop-in does not touch it; `passwd -S root` = P).
## Digests, sha256, CI by commit
| What | Value |
|---|---|
| controller 0.340.0 | `sha256:a8d1c6458da4ea9d667f928498656a291771e0b3756b20f2fcc8c1cf9d516621` — `caaa86e`, CI 2245 success |
| agent 0.163.0 binary / bundle | `561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c` / `3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8` — `1109092` CI 2246, `b4721ee` CI 2249 success |
| step bundle 0.163.0-step1 | `6cbc066902d5f3294765d5c7dc7113166c1dca25e994dbc36bfc5f139bb61fcf` |
| installer 1.36.0 (served = tag) | `de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec` — `83388c08` CI 2247 success; pins `2aebf1bd` |
| hub 0.167.0 | `sha256:177860a996cb285d935777d6ffdf85725f6cf4489885f54fd3bf8135c68041a0` — built at `d306b8c2` (hub/ = `be7e97dc`, CI 2250 success; `d306b8c2` CI 2251 success); manifest `34979d5d` |
| website | `0bb57a4f` |
| D34 fix (R-1234) | `52af38df` |
| prometheus rules | homelab-manifests `1bbef892` |
CI of the last commits (`2aebf1bd`, `0bb57a4f`, `34979d5d`, `52af38df` and this close-out): see the final message.
## Parts
- **1 — controller 0.340.0:** 9202 smoke, per-customer floors, three boxes green, box walk + https-only rc 0, global floor
06:41 UTC (MinAgent 0.158.1), read back with the WAL (`01`–`06`).
- **2 — agent + installer:** release, a step bundle because the D4 drop-in is a new path (R-880), three signed jobs per
box (`07`); the operator's vouch read back from the DB and the live page (`10`); installer tag, both pins, a
webpage-only sync, the served script = the tag (`11`).
- **3 — hub 0.167.0:** built from a clean clone; on the operator's word it also carries D34's 6-hourly snapshot (asked
before the build; not soaked); hub-only sync; 30-minute watch: 0 hub events, 0 errors, all boxes reporting, no hub mail
(`14`).
- **4 — website:** D3/D5 patch + D36 HU + EN; site gates; published; read back live: new text present, old gone (control:
the previous commit), screenshots at 390 and 1366 px, no sideways scroll (`12-website-readback/`).
- **5 — D34:** retention unchanged, server-side on ep0 (`prune-operator-hubdb` 03:45, keep-daily 14, keep-weekly 8 — the
current day's up to 4 copies until the next prune). Hand run 501.6 MiB / 24.5 MiB uploaded in 4 s; the newest copy
restored from ep0 with the read-only token = the pod's snapshot by sha256 `a9ff4c8a…`, integrity ok, 4 hosts, holds
today's vouch. Alarm 13 h. Runbook updated (`13`).
- **6 — D33:** ep0 before: `/` 14 GB free (the data disk's root IS the PBS datastore — not used). One user
`felhom-relcopy` (locked, rrsync-confined: shell and `..` refused), one directory `/srv/felhom-release-copy`. Cap 3 GB,
enforced by the sender (ep0 has no quota). **skopeo, not `docker save`:** `dir:` keeps the registry's bytes, so a push
back keeps the digest; no Docker daemon. A read-only registry token of its own, minted in the Gitea pod straight into a
0600 file. First run 676 MB, 91 s; pulled back: 141 files OK, 9 images re-verified (digests match independent records),
controller 0.340.0 loaded on 9202 (ID = config digest). Alarm `FelhomReleaseCopyStale`. Restore steps in
`total-loss-of-dooplex.md` (`08`).
- **7 — D32:** the sheet is names and places only, HU + EN, one A4 (rendered: 1 page; a copy in the operator's home
folder); every place checked to exist. demo-hp: three superseded ISOs removed by name, `/` 8.5 → 14 GB free (`09`).
R-548: the box half done, the product fix proposed (R-921's pattern).
- **8 — close-out:** CONTEXT rulings, STATUS (decide rows D32–D36 → history, D37/D38 new, the do list, D35 dated
2026-10-30), REPORTs and CHANGELOGs; the system poster's facts file updated, its hub-DB label edited (text), and a new
arrow asked for in STATUS („System poster needs a refresh").
## Register — 191 → 184; opened 4, closed 11
Opened: R-1234 (closed the same hour), R-1235, R-1236, R-1237. Closed: R-49, R-982, R-985, R-991, R-1204, R-1205,
R-1225, R-1226, R-1227, R-1230, R-1234. Updated: R-1214 (released; dated check 2026-10-24), R-1187, R-1189, R-232,
R-548, R-923, R-924, R-1228 (D35, dated 2026-10-30), R-1183's check text (four snapshot slots).
## Observations
- The D34 first push refused the hub's own catch-up and mailed admin@ — FILED: R-1234 (closed, fixed `52af38df`).
- The release copy is proven to load, not to restore a registry; it follows the newest versions, not the floor/vouch; no
golden — FILED: R-1235.
- Gitea OOM-killed at its 1 GiB limit, the last at 06:41:10 UTC during the release — FILED: R-1236 (STATUS D37).
- The night sweep's mail: one-time test-box codes in one session record — FILED: R-1237.
- The hub DB snapshot is 501 MiB, not 24 MB — NOT-A-FINDING: 24 MB is the compressed upload, which is what ep0 stores.
- felhom-sshd does not answer from the LAN — NOT-A-FINDING: its firewall admits only the operator peer (H1, by design).
- A dummy root password logs „Failed password", not a policy refusal — NOT-A-FINDING: OpenSSH's behaviour under
`prohibit-password`; the policy is proven by `sshd -T` and the reload.
- The vouch left `artifact_wrapper_sha256` empty — NOT-A-FINDING: an optional field; the wrapper rides the vouched bundle.
- Guests 9202, 9401 (demo-hp) and a transient 990000 (Tester 1) have swap ≠ RAM — NOT-A-FINDING: outside the felhom pool.
- `known_hosts` holds an older key for 192.168.0.155 (demo-hp's guest address) — NOT-A-FINDING: guests are reached by
`pct exec`; nothing was changed.
- The D4 drop-in needed a step bundle — NOT-A-FINDING: the documented R-880 path for a new bundle path.
## Teardown — three layers
Machine: 9202 — the test image untagged, the tar removed. Host: demo-hp `/tmp` tar removed; three ISOs removed on purpose
(Part 7); DooPlex: the read-back copies and the hub-DB scratch restore shredded/removed. Hub: nothing provisioned; the
per-customer floors cleared after the global floor. Left on purpose: the ep0 user + directory, the DooPlex release-copy
job, the 6-hourly hub-DB timer, both alarms.
+111
View File
@@ -0,0 +1,111 @@
# REPORT — two DooPlex backup alarms, every alarm the mail route delivers, and R-1250 (2026-10-24, attended)
Brief: „two DooPlex backup alarms — find why, fix, and check every other alarm the new mail route now delivers".
Read first: `runbooks/total-loss-of-dooplex.md`, `runbooks/RUNBOOK-hub-db-offsite-backup.md`,
`audits/dooplex-survival-2026-10-22/PLAN.md`, R-232, R-1227, R-1241, the two homelab rules and the Alertmanager
route, homelab HM-040…HM-049 and the window record; design `08` §9. Evidence: `audits/dooplex-alarms-2026-10-24/`.
## Three lines
1. **Both alarms had one cause, and both are fixed.** Yesterday's reboot restarted the database, and it now takes the
password of the other of its two secrets; the dump job could not log in, and the Gitea off-site push then refused
the old dump. The job now reads the right secret. Cleared: the dump alarm 10:44Z, the Gitea alarm 10:51Z.
2. **Nothing else fires now.** In 7 days only the pulled disk's two alarms fired long (silenced by you since 10-12,
ended by the pull); the rest were minutes long or are known.
3. **Fewer mails:** critical every 4 h (was 1 h), warning every 12 h (was 4 h), all backup alarms in one mail.
## What I asked you, and what you said
- The dump fix → **the CronJob only** (the database's own definition untouched).
- The mail change needs an Alertmanager restart, not a reload → **restart**.
- The R-1250 proof needed a downloader on another machine → **use DooPlex's k3s qBittorrent**.
## The alarm table (7 days, Prometheus `ALERTS`; nothing firing at 10:51Z)
| Alarm | Severity | When (UTC) | True? | Cause | Fix / row |
|---|---|---|---|---|---|
| `DooplexGiteaOffsiteStale` | critical | 10-24 00:57 → 10:51 | true | the push refused a 12 h old dump | the dump fix; R-232 note |
| `PostgresqlDumpStale` | warning | 10-23 22:17 → 10-24 10:44 | true | dump job login failed (28P01) | homelab `531023d` |
| `DiskOfflineUncorrectable`, `DiskUncorrectableSectors` (Z6A07P2G) | critical | whole window (≥ 145 h) → 10-23 12:02 | true | the dying HDD4 | silenced by you 10-12; pulled 10-23 — R-934 closed |
| `DiskDisappeared` (Z6A07P2G) | critical | 10-23 12:17 → 18:02 | true | the pull | expected; cleared |
| `DiskInventoryChanged` | warning | 10-23 12:17 → 20:17 | true | the pull | expected; cleared |
| `LonghornVolumeDegraded` (two Plex volumes) | warning | 10-23 08:22 → 11:22 | true | the HM-040 rebuild | cleared |
| `PodRestartingFrequently` (felhom-webpage) | warning | 10-19 09:02 → 10:07 | true | git-syncs stopped on errors | R-1163 (closed 10-19) |
| `WanUploadDegraded`, `WanDownloadDegraded`, `WanUpstreamPacketLoss`, `WanBufferbloat` | warning/info | minutes, 10-17 → 10-20 | true | short WAN dips (the flood days) | none needed |
| `AuthentikHighLatency` | warning | 10-21 08:12, 5 min | true | a short spike | none needed |
## The mail change (homelab `22d7697`, applied by a restart, read back from the running Alertmanager)
| | before | after |
|---|---|---|
| critical | every 1 h | every 4 h |
| warning | every 4 h | every 12 h |
| other (info) | every 4 h | every 12 h |
| backup alarms | one mail stream per alarm | one mail for the group, every 4 h |
| a disk | one stream per alarm | one mail per drive, daily (unchanged) |
| resolved mail | `send_resolved: true` | unchanged; the mail counter rose once after each clear (10:47, and by 10:57) — that they read „RESOLVED" is not seen: the mailbox is not mine |
## Part 4 — R-1250, committed and proven
Agent `29bcf44` + `1e087bb`, controller `046b808` (commit only; release in the next train). A seeding box: the agent
says „waiting for quiet"; the controller records the torrent app's limit, holds it at 1 KiB/s, asks for one test,
puts the exact limit back. **Proven on demo-hp with a real seed** (10 Mbit/s from 9202 to DooPlex's qBittorrent):
refused without the hold (11.1 Mbit/s); with it the uplink fell to 0.02–0.16 Mbit/s, 94.2 Mbit/s, 56.8 s, the limit
back to 1 249 280 B/s; a kill in the middle → the start-up check put it back. The brief said „a nested VM": NOT done —
a nested VM would need its own tunnel to ep0 to run the real test; demo-hp's host with probes did the same with the
real tunnel. The pinned agent client is unit-tested only.
## Commits, CI, undo
| Repo | Commit | What | CI |
|---|---|---|---|
| homelab-manifests | `531023d` | the dump CronJob's secret | no CI in this repo |
| homelab-manifests | `22d7697` | the mail routes, the dump alarm's label | no CI |
| homelab-manifests | `9f399fe` | REPORT | no CI |
| felhom-agent | `29bcf44`, `1e087bb` | R-1250 | 2286, 2288 success |
| felhom-controller | `046b808` | R-1250 | 2287 success |
| app-catalog-felhom.eu | `7de554e`, `5e196e1` | the box walk: HDD_PATH, the leftover check | 2289 success, (final push) |
| felhom-agent / felhom-controller | `5e05627` / `321df1f` | REPORT, README | (final push) |
| felhom.eu | this commit | register, STATUS, CONTEXT, `08` §9, evidence | (final push) |
Undo: `audits/dooplex-alarms-2026-10-24/README.md` § Undo. Backups of the three homelab files: `*.before` there.
## Rows
**Before 191, after 190; opened 0, closed 1 (R-934 — the disk was already pulled).** R-232: a dated note and a dated
check (2026-10-25, tonight's sweep). R-1250: NARROWED (fixed on main, waits for the release train).
## Observations
- The database's two secrets for one role stay (only the CronJob moved) — homelab list item 1 (not a Felhom row).
- A stale dump stops the Gitea + secrets off-site push for as long as it lasts — NOT-A-FINDING: the designed fail-closed
order (`PLAN.md`: the dump first, for consistency); the alarm caught it inside its 26 h, and it now comes in one mail
with its cause.
- Alertmanager cannot reload a ConfigMap change (the brief assumed it could) — NOT-A-FINDING: the homelab manifest's
design (`envsubst` init container); you chose the restart; the fact is in CC's memory and the homelab list.
- The agent said „waiting for quiet" before it had watched a minute — fixed in the session (`1e087bb`, found live).
- The box walk sent a folder inside the drive as HDD_PATH (refused since R-839) — fixed in the session (`7de554e`).
- The box walk called the template folder a leftover after every remove — fixed in the session (`5e196e1`).
- R-934 said the failed disk was still in DooPlex; STATUS still listed R-1214's done check — fixed (row closed, line removed).
- CC's memory said Prometheus has no config-reloader; it has one since 2026-10-12 — fixed (memory note).
- `govulncheck` warns GO-2026-6809 (crypto/x509, stdlib) in the agent and the controller — NOT-A-FINDING here: the Go
toolchain row R-972 owns it.
- My Gmail connector's mailbox gets no Alertmanager mail — NOT-A-FINDING: it is another mailbox than `SMTP_TO`; 60 sent,
0 failed in 7 days. The Alertmanager config did not change yesterday (last 10-12).
- `kubectl exec -i` never closed stdin for `pg_restore` — NOT-A-FINDING: my method; the host's own `pg_restore` read the dumps.
- The 9202 ISO torrent never uploaded (behind NAT, 18 leechers / 1080 seeds) — NOT-A-FINDING: why the LAN peer was used.
- I overwrote homelab-manifests' REPORT.md, which held another session's HM-050 report from today — NOT-A-FINDING: the
repo's overwrite rule; HM-050's entry stays in its CHANGELOG.
## Teardown, three layers
- **Machine:** 9202 — qBittorrent removed through the product (no container, no volume; the template folder stays, as for
every app); the test files, the forwarder and the probe removed. DooPlex's qBittorrent — the test torrent and its
files deleted (by hash), nothing else touched.
- **Host:** demo-hp — the agent probe stopped and deleted, no iperf3 left; the installed agent untouched. DooPlex — the
9202 password copy and the helper's session files shredded; the clones removed.
- **Hub:** nothing provisioned.
## Instruction files, poster
No instruction file edited. No poster fact changed (the mail intervals are not on it).
+131
View File
@@ -0,0 +1,131 @@
# REPORT — the day brief of 2026-10-21: fix the four serious problems from the rehearsal, and the Sunday clock change
Separate from the shared `REPORT.md` on purpose: another session works in this clone (repo rule: a second session writes
`REPORT-<topic>.md`). Evidence: `documentation/audits/fixes-2026-10-21/`. Design read first: `03` §8a, `07` §7.0/§6/§8.2,
`05` (scheduler, escrow store), `04` (self-scoped reads), `09` §3 decisions D19, 78, R-241.
## 1. Five lines
1. **R-1176 (a rebuilt box gets its off-site backups back) — FIXED AND LIVE.** Hub 0.163.0 (operator yes, 11:46 UTC);
proven end to end on demo-hp: no fresh key, recovery offered, wrong code refused, right code opened the history,
Nextcloud 3 031 of 3 033 files byte-identical, the next off-site runs appended (63 → 64).
2. **R-1175 (shares on two drives) — FIXED AND LIVE.** Controller 0.337.0 (global floor 15:54 UTC), D25: a gone source
keeps its copy until the household removes it.
3. **R-1191 (an imported app keeps its family lock) — FIXED AND LIVE.** Controller 0.337.0; proven on 9202: a stranger
gets 401 after export → remove → import (it was 200).
4. **R-1192 (the 8 GB freeze) — CAUSE MEASURED, FIX COMMITTED, NOT RELEASED.** The guest's swap was capped at Proxmox's
512 MB default; the agent fix (swap = RAM) is committed (`286a7e3`); its release and vouch are the operator's (D26).
Proven by applying the same setting by hand for 2 hours of the same uploads.
5. **Ready for Tester 2: NO — one step left:** the agent release with the swap fix. Everything else on the P1 list is live.
## 2. R-1192 — the mechanism, the numbers, the fix
**One sentence:** the guest's swap is capped at 512 MB (Proxmox's LXC default — the agent never set `swap`), so when
Immich loaded its analysis models (+1.15 GB) on top of 8 apps, the 4096 MB guest could evict only program code, and
~25 containers re-read their code from disk without end.
| | the freeze (13:26–13:34 UTC, cap 512 MB) | the proof (13:43–15:43 UTC, cap = RAM by hand) |
|---|---|---|
| guest private memory (anon), max | 3 585 MB | 2 782 MB |
| guest swap used, max | 511 MB of 512 (full) | 2 555 MB |
| refaults per 10 s, max | 2 741 774 | 187 462 |
| disk reads per 10 s, max | 12 GB | 0.9 GB |
| memory „full" pressure (avg10), max | 96 % | 12 % |
| IO pressure, max | 97 % | 53 % |
| load, max | 162 | 33 |
| guest „limit hit" events per 10 s, max | 3 889 | 0 |
| OOM kills | 0 | 0 |
| dashboard (DooPlex probe every 10 s) | dead 13:28:18–13:34:48 | every probe answered, slowest 1.26 s |
The experiment between them: `pct set 9201 --swap 4096` at 13:34:06, uploads still running → memory „full" 96 % → 1.7 %
within 2 minutes, the dashboard back 42 s after the change. The host had 2.2 GB RAM and 7.4 GB swap free throughout.
Load for the proof: the same uploads as fresh work, repeated — Paperless 110 cycles × 53 PDFs (emptied each time), Immich
111 new users × 306 photos (305 accepted each time; one photo is refused every cycle), two Nextcloud trees (WebDAV small
files, ~20 files/min under this load — slow, nothing lost). Method: a recorder on the box (`tools/rec.py`, 10 s, copied to
DooPlex every minute) and the second channel on DooPlex (`tools/outside.sh`: the dashboard's `/api/health` and QEMU's
disk counters from demo-hp).
**Fix:** agent `286a7e3` (commit only): `swap` = memory at bring-up, at a RAM resize, and once at agent start for every
pool guest whose swap is below its RAM (live, no restart, never lowers). Tests red-proved. The controller half (0.337.0):
„Normál használat mellett ez nem okoz problémát." is gone; the install page now says **„Az alkalmazások csúcsterhelése
meghaladhatja a rendelkezésre álló memóriát. Ilyenkor a doboz nagy feltöltések alatt lelassulhat, és akár percekig sem
válaszol. Ha ez gyakran előfordul, távolíts el egy alkalmazást."** No catalog change and no admission-ceiling change:
the measured cause is the swap cap, and the order of preference (catalog → controller → agent) gave way to the proof.
R-1193 (an earlier, plainer alarm): a design note in its row; build it only if a freeze is seen after the agent release.
## 3. R-1176 — the shape, the order check, the live proof
**Shape (operator ruling):** `store.ReattachRetainedEscrow` moves the newest retained row of the customer's newest
escrow-ACKED host deletion into `host_escrow` for its one live host, in one transaction (no other live host, no live
package, the deleted host not live again, an identity blob present). Called on the agent's host report and before the
controller ACK's escrow status. Never opened, re-sealed or logged; audit `escrow_carried` (host ids + hash); two live
hosts → `escrow_carry_refused` mailed once. Self-scope unchanged: the new host reads only rows attached to it (its
retained GET for the old host is 403).
**Order check (measured, not assumed):** the hub carried at the agent's FIRST report (12:04:21 UTC); the controller's
settle gate went GO at 12:06:25 after its first ACK; at 12:06:28 it logged „NOT minting a repository password: the hub
holds a sealed recovery package". So no controller change was needed; 0.337.0 still records the escrow before
`SetFloor` (a pinned belt for a hub that carries later).
**Live proof (demo-hp VM 9321, Tester4, public ISO 1.29.0, no operator press before the install):** bind (one OCR misread
of the pairing code cost one of five tries — harness), claim, the dashboard offered „A korábbi, házon kívüli mentéseid
visszaszerezhetők"; a wrong code: „A megadott helyreállítási kódot nem fogadtuk el … Semmi nem változott"; the right code
unlocked 9 groups; Nextcloud reinstalled („előbb telepítsd újra" is the page's own order), „Teljes visszaállítás" with
the placement confirm → „3113 fájl és 2 adatkötet és az adatbázis visszaállítva"; **sha256 through Nextcloud's own WebDAV
against the rehearsal's A5 manifest: 3 031 of 3 033 byte-identical** — the 2 missing never reached the old box (the 1.5 GB
file Nextcloud refused, a symlink rclone skips); 4 extra files are the rehearsal's later uploads. Off-site: two runs, 63 → 64
snapshots in the same 1.4 GB store, `orphaned: false`; the hub's full log has no set-aside and no key change.
## 4. Versions, commits, tests
| Component | Version | State | Commits |
|---|---|---|---|
| hub | 0.163.0 | **deployed** 11:46 UTC (operator yes), Synced/Healthy | `9688e76e`, manifest `cd3d0598` |
| controller | 0.337.0 (`sha256:264af313…`) | **released**: rc.1 on 9202; per-box floors 13:59 → three test boxes on 0.337.0 by 14:06; box walk green; **global floor 0.337.0 / MinAgent 0.158.1 at 15:54:29 UTC**, read back; per-box floors cleared | `8f5bea0`, `5929b8e`, `7848d4a` |
| agent | (0.162.0 when released) | **committed, not released** | `286a7e3` |
| catalog, installer | — | unchanged | — |
Tests (all pass; each red-proved — seen failing first): hub `TestR1183_*` (3), `TestR1176_*` (store 7, api 3),
`TestR1182_*` (store 2, api 1), `TestR1203_*`; controller `TestR1175_*` (3), `TestR1181_*` (2), `TestD25_*` (3),
`TestR1179_*`, `TestR1188c_*`, `TestR1191_*` (2), `TestR1176_EscrowIsRecordedBeforeTheFloorIsKnown`, `TestR1182_*` (2);
agent `TestR1192_*` (2) + the resize test's swap assertion. CI: green by job id for every push (see §5).
## 5. Morning list, teardown, rows, observations
**Morning list (operator):** D26 — release + vouch the agent with the swap fix; delete host **Tester4-f6b67c** (asked
today); R-1205 (the whole-box tier waits for „Re-issue PBS credentials" after a failed second install); the
`REHEARSAL_2026_10_20_*` / `FIXES_2026_10_21_*` names in sec.
**Operator acts today:** the hub-deploy yes; the root credential reveal for the freeze test (~2 min); the host delete
(done 16:49 UTC — the delete appears 6 minutes after a box goes off, R-30's rule).
**Teardown (three layers):**
- **Machine (demo-hp, Tier 0):** VM 9321 `rh1021-box2` and helper CT 9331 `rh1021-loader` destroyed with `--purge` at
16:27:40 UTC (names checked first); no image left on `nvme-scratch`; 341, 9201, 9202, 9401 untouched. 9202 moved from
`0.337.0-rc.1` to `0.337.0` by hand (it does not take the hub's floor). The public ISO 1.29.0 was only read.
- **Hub:** the operator deleted host Tester4-f6b67c (16:49:37 UTC, escrow acknowledged): no Tester4 host, no current
package, the household's package retained under Tester4-f6b67c with the same hash, `repo_tag` 'live' — a future
rebuild carries it again; the WireGuard peer gone (`P6-release/05-teardown-hub.txt`).
- **ep0 / Storage Box (read only, nothing removed):** Tester4's off-site store gained 2 snapshots (63 → 64, normal
traffic); the hub's registrar pinned the rebuilt box's key and removed the dead box's line (normal enrollment). No PBS
descriptor was made (R-1205), so nothing new on ep0's PBS.
**Rows:** before 182 (181 after a parallel session closed R-1201); after 176; opened 2 (R-1205, R-1206); closed 7 (R-1176,
R-1203, R-1175, R-1179, R-1181, R-1182, R-1191). R-1183 → VERIFY (due 2026-10-26); R-1192 → WAITING-ON-OPERATOR;
R-1188 (c) fixed. Capability map A12, A13 → PROVEN-LIVE. `unproven.py --summary`: unchanged, 35 of 55 not walked.
**Observations:**
- PBS auto re-issue refused for the rebuilt Tester4 (latest deletion un-acked) — **FILED: R-1205**.
- Future-dated snapshots from the clock test show as „Legutóbbi mentés: 2026. 10. 28." — **FILED: R-1206**.
- The pairing-code OCR misread one character — **NOT-A-FINDING:** harness (a human reads the screen); corrected against the hub's record.
- Immich's first admin needs the household's dashboard session (the setup gate) — **NOT-A-FINDING:** designed.
- The full restore asks to confirm a new place (backup on `hdd_1`, now the internal disk) — **NOT-A-FINDING:** designed (R-351).
- Nextcloud's small-file WebDAV upload ~20 files/min under heavy concurrent load — **NOT-A-FINDING:** slow, not lost; the new install-page sentence says big uploads can be slow.
- `kubectl logs -l` reads only 10 lines — **NOT-A-FINDING** (a tooling trap; saved to memory, two checks redone over the full log).
- Tester 2 last reported 2026-10-04 on 0.292.0; the floor reaches it only when it reports — **NOT-A-FINDING** (R-1139 covers the box).
**Secrets:** names only — `FIXES_2026_10_21_PAIRING_CODE`, `FIXES_2026_10_21_BIND_TOKEN`, `FIXES_2026_10_21_SETUP_CODE`
(one-time, spent), `FIXES_2026_10_21_ROOTPW` (the operator's reveal; the box is torn down). The bind link and the setup code
were read from the mailbox by the mail connector, so both one-time values entered this session's record (spent within
minutes, as in the rehearsal). A scan of all 45 evidence files for every secret used today: 0 hits. Nothing secret in any
commit.
+61
View File
@@ -0,0 +1,61 @@
# REPORT — the morning of 2026-10-22: the operator's five yeses
**Operator answers:** release controller 0.338.0 — yes; release and vouch agent 0.162.0 — yes; deploy hub 0.164.0 — yes
(Tester4 host deleted); Tester 2's laptop — installed with our installer; publish items — yes, all. Design read:
`07` §6.3 (restore put-back), `03` §9 (guest sizing), the felhom-build-deploy runbooks, `runbooks/iso-release-gate.md`.
## Decisions taken by CC
None new. The night's R-1199 decision (a box with no saved notification choice pushes its page's default list) now
ships in controller 0.338.0 — *operator may reverse*.
## What was done
| | what | result |
|---|---|---|
| agent | `release-agent.sh 0.162.0` from a clean clone at `b310773` | sha256 `5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928` (= the soak's bytes), bundle `ac772967…60b5a5`, tag `v0.162.0`, round trip OK |
| agent | one signed `agent_update` per box: demo-hp, Tester 1, demo-felhom (07:41 UTC) | all three `update committed`, `ok=76 total=76`, `guest-swap: raised` for 9201 (512 MB → RAM); `pct config` swap = memory on all three |
| hub | manifest `0.163.0 → 0.164.0` (`0b7a5ab3`), only `Deployment/hub` synced | running `sha256:931594e3…` (= the night's build), log `felhom-hub 0.164.0 starting` |
| controller | `build.sh 0.338.0 --push` from `7900a32` | `sha256:e8a529f395bfe8c0e7b87d5d98929a6f8e9834aa13c0ab77bd214ee9e7e53433` |
| controller | per-customer floors (tester-1, demo-hp, demo-felhom), MinAgent 0.158.1 | all three on 0.338.0 within 10 s (agent `controller-swap: new controller healthy`); https-only 28/28 + control; 10/24/9 containers, 0 unhealthy, 0 restarting; 0 `level=ERROR` |
| controller | global floor 0.338.0 / MinAgent 0.158.1 with CC's hub token | `303 → flash=floor_set`; hub DB (with WAL): `min_controller_version=0.338.0`, `0.338.0=0.158.1`; the three per-customer floors cleared |
| website | the W2 patch (`6b391a63`), site gates OK | live on felhom.eu/letoltes.html and /en/download |
| ISO | 1.30.0 `--release` from `115f15c8` | `590d7b9b3ccbc7644536a1971a44deb2d1bf7136f69eae2b4dc42961e74d5cc1`, 1 705 324 544 B; gate G1–G16 PASS (G12 not re-read); proof installs from BOTH entries (VMs 9341, 9342), first boot + one reboot; W1 seen on screen; published, round trip identical; both download pages point at 1.30.0 |
Evidence: `documentation/audits/morning-2026-10-22/release/`, `documentation/tests/iso-release-1.30.0-2026-10-22/`.
## What broke
- **The host-swap check is too strict (R-1216, D28).** Within minutes of agent 0.162.0 demo-hp (host swap 8191 MB vs
guests 25898 MB) and demo-felhom (8191 vs 12288) warned and the hub mailed `host_swap_short`. The check compares host
swap with the guests' whole swap allowance; the installer gives at most 8 GB, the soak's burst used 2.7 GB. The same
comparison drives the controller's install-page sentence. Not fixed: the threshold is the operator's pick.
- **A stale runbook command (fixed).** `iso-release-gate.md` G16 mounted `scripts/iso` as `/work`; the harness needs
`golden/` and `postinst` staged, so 10 checks failed on a correct tree. Now names `scripts/iso_bootstrap_gate.py`
(73 checks green). My first run with the old line left harness scratch files in `scripts/iso/` (root-owned); removed
by name before any commit.
## Rows
Before 184, after 178. Opened 1 (R-1216). Closed 7: R-1192, R-1177, R-1178, R-1180, R-1195, R-1199, R-1210. Updated:
R-1194 (released; WATCHING for the next fresh install; owner CC), R-1207 (Tester 2 = our installer). Section headers
recounted (Backup & restore read 44, held 42).
## Waiting on the operator
1. **Vouch agent 0.162.0** (Configs → Day-0 artifacts). If not: new installs get 0.161.0, without the swap fix.
2. **Discard appliances 47 and 48** (the proof installs). If not: two „unclaimed" rows stay.
3. **D28** — loosen the host-swap check (A, CC's pick) or keep it (B).
4. **Tester 2** is off; when it is back, CC sends it agent 0.162.0.
5. The secret names `REHEARSAL_2026_10_20_*`, `FIXES_2026_10_21_*`, `NIGHT_2026_10_21_*` are KEPT (the proof installs
used the night's root password); say „remove" to drop them.
## Teardown
- **Machine:** VMs 9341, 9342 destroyed (names checked); ISO 1.30.0 kept in demo-hp `local:iso`. No thin-pool write.
- **Host:** demo-hp, Tester 1, demo-felhom now on agent 0.162.0 and controller 0.338.0 — intended, not torn down.
- **Hub:** appliances 47, 48 for the operator to discard; floors: global 0.338.0, no per-customer floor left.
## Instruction files / poster
No instruction file changed. No system-poster fact changed (versions are not poster facts).
+126 -1
View File
@@ -1,4 +1,129 @@
# REPORT — evening/night brief 2026-10-19 (one stop less per night, no false tunnel error, R-893 measured, hub and agent fixes ready, a scanner on every push, ep0's first weekly clean-up)
# REPORT — the night briefs of 2026-10-19 and 2026-10-20 (part 2 run by day): one stop less per night, the safe undo for a failed off-site restore, hub 0.162.0 / agent 0.161.0 / installer 1.35.0 delivered, downloads out of the backups, a scanner on every push
# PART 2 — the night brief part 2 (run by day, 2026-10-20): your four answers, the safe undo, downloads out of the backups, Immich, the update test
**Safest readings taken alone (written first, as the brief asks):**
1. **R-1083 is not in agent 0.161.0.** An agent key would make 9202's controller book the agent's whole-guest backups,
one of which writes to ep0 — a change on ep0 the brief forbids; 9202 is also outside the `felhom` pool.
2. **All five downloaders' folders are `excluded`, not only the two named** (qBittorrent, Transmission; also NZBGet,
SABnzbd, MeTube): the brief's gate rule — a download folder is never optional or mandatory — covers all five.
3. **No update-test edge was run:** the bench (guest 9401) and the drill catalog are outside the brief's machines and
repos. The list done and the list left are written; the ask is on the morning list.
4. The brief was written for a night run and ran by day; its „stop by 05:00" was read as a time box, not a clock.
## Part 2 — in plain words
- **Morning list:** vouch agent 0.161.0 · decide D21 (Immich) · MeTube's videos out of the second-drive copy — keep? ·
the bench and the drill catalog for the update test — yes?
- **Hub 0.162.0 is live.** Its own deploy was the real restart test: Tester-2 (down since 4 October) got no repeat mail
— read in the hub and, separately, in your mailbox.
- **Agent 0.161.0 runs on demo-hp, Tester 1 and demo-felhom.** On demo-hp the agent said „busy" during a backup and
„free" before and after. **Not vouched — that is yours.**
- **Installer 1.35.0 is published.** The served file equals the tag's file.
- **A failed off-site restore now puts the app back exactly as it was** (your D20 = A, controller 0.336.0, the floor).
Proven on the scratch box: after a forced failure the database matched its fingerprints from before, the app ran,
nothing was held; a normal restore then worked.
- **Downloads are out of the backups** (torrent, usenet, MeTube): the controller's own check copies 0 of them now.
- **Immich's off-site copy:** 1.33 GB, of which 102 MB must survive. Two options; pick B (leave out the AI model files).
- **ep0's first weekly clean-up:** done this morning (Part 10 of part 1 below).
## Part 2 — each part
| Part | Done? | Why / how |
|---|---|---|
| 1a hub deploy | **done** | `manifests/hub.yaml` → 0.162.0 (`e1d349f1`), only Deployment/hub synced (the only OutOfSync); image `sha256:69285e64…` = the build; Synced/Healthy; reports from all three boxes; floor read 0.335.0; Day-0 agent list has no `-step1`. R-1001 + R-1011 closed. |
| 1b agent release | **done** | R-1083 left out (above). `release-agent.sh 0.161.0` from a clean clone; signed `agent_update` per box: demo-hp 06:38, Tester 1 06:54, demo-felhom 06:55 UTC, each COMPLETED, 76/76, committed. `host_busy` live; tunnel `absent` = the fixture row (9202's real Docker 29 answer); R-1010 by its tests. R-1070 + R-1010 closed. |
| 1b R-921 live | **not done** | The no-stop case happens only on the night schedule (a press backs up the local tier only). Row R-1173 with a dated check after demo-hp's next off-site night. |
| 1c installer | **done** | tag `installer-v1.35.0`, both webpage pins (`b0ae2c57`), webpage Deployment synced alone; served sha256 `f7a196fd…` = tag. R-1167 closed. |
| 2 safe undo | **done** | below. R-893 + R-1169 closed. |
| 3 downloads | **done, one read left** | catalog `c50adfc`, gate `download-binds`; classifier proof. demo-felhom (qBittorrent) runs its next second-drive copy 2026-10-21 03:30 — dated check on R-1019 (NARROWED). |
| 4 Immich | **done (plan only)** | below. R-49 → READY, D21. |
| 5 update test | **not done** | fixture side complete for every installed app; the edges need the bench / drill catalog (above). R-462 NARROWED. |
| 6 ep0 clean-up | **done** | part 1, Part 10. |
| 7 controller release | **done** | 0.336.0 below. |
## Part 2 — the safe undo, the 9202 proof (`07` §6.3; `audits/night2-2026-10-20/P2/`)
- Code `93fb251`: the fit check before the stop, the copy right after it (`backups/restore-undo/<app>/`), the put-back on
every failure branch, the hold only when the put-back fails, the copy removed on success. 11 tests; nine mutations red.
- **9202, rc.1** (a helper with the fences in full; Rallly + postgres 18; the same forced failure as on 19 October):
| | Live before | After the forced failure |
|---|---|---|
| public tables | 36 | 36 |
| markers | A, B, C, D | A, B, C, D |
| `r893_only_old` (backup only) | absent | **absent** (on 19 October: present) |
| schema / data / row-count md5 | `05828c77` / `e5c1a72f` / `d8bcbc4b` | **equal** |
| app | running | **running**, no hold |
| the copy | — | 69 MB in 0.9 s, removed after |
The sentence: „…visszaállítása sikertelen. Nem változott semmi: az alkalmazás adatai visszakerültek a visszaállítás
előtti állapotba, és az alkalmazás újra fut". Then a normal restore of the same snapshot: the snapshot's content
(A–C, the backup-only table present), the app running, the copy removed. 9202 back on 0.331.0 as found.
- Not measured live (each pinned by a red-proved test): the version-change branch, the `_files` sentence, the refusals
before the stop, a failing put-back, the operator event (9202 has no hub).
## Part 2 — download binds (`15` §7; `audits/night2-2026-10-20/P3/`)
| App | Bind (userdata) | Class now | Copied by a tier |
|---|---|---|---|
| qBittorrent | media/torrents | excluded (was optional) | none (was the second drive) |
| Transmission | media/transmission, media/transmission/watch | excluded, excluded | none |
| NZBGet | media/nzbget | excluded (was optional) | none |
| SABnzbd | media/usenet | excluded (was optional) | none |
| MeTube | media/metube | excluded (was optional) | none |
| Sonarr, Radarr | media (rw) | excluded | none |
| Jellyfin, Plex, Emby | media (:ro) | excluded | none |
| Prowlarr | no userdata bind | — | — |
## Part 2 — Immich (`audits/night2-2026-10-20/P4/immich-offsite-plan.md`)
Model cache 824 MB (62 %) · raw database folder 283 MB (21 %) · Immich's own backups 115 MB (9 %) · logical dump 54 MB
· originals 49 MB · thumbnails 2 MB. **A** (leave out all three re-fetchable parts, −92 %) needs a never-run restore
from the dump alone on the vectorchord image; **B** (only the model cache, −62 %) needs one restore with an empty cache.
**Pick B now**, A's other parts after their proof.
## Part 2 — update-test edges
None run (above). Installed apps: 16; with a fixture: 15; without: bentopdf (keeps no data). Upstream within a major,
waiting for a tested step: PrivateBin 2.0.6→2.1.0, Home Assistant 2026.9.4→2026.10.2, RomM 5.3.1→5.4.0, Plex
1.41.4→1.43.4, Kimai 2.67.0→2.70.0 (`audits/night2-2026-10-20/P5/r462-list.txt`).
## Part 2 — versions, commits, rows
- Hub 0.162.0 deployed (`e1d349f1`). Agent 0.161.0 released (`5152123`, tag `v0.161.0`; notes `4e1621c`). Installer
1.35.0 published (`b0ae2c57`). Controller 0.336.0 the floor (`93fb251`; notes `aa9e09e`, `7ca59e2`), image
`sha256:93cd67d8…8d73`. Catalog `c50adfc`. Evidence `301f4e9e`, `bcaa9576`.
- i18n: no page template changed; seven Go-side keys (HU + EN), one rewritten (R-1169); no fixture changed.
- **System poster:** no listed fact changed — none of the three.
- **Register 160 before, 156 after. Closed 7:** R-1001, R-1011, R-1167, R-1070, R-1010, R-893, R-1169. **Opened 3:**
R-1171 (demo-hp's local copy never restore-tested), R-1172 (the put-back logs at ERROR; an empty folder stays),
R-1173 (R-921's no-stop not yet seen live). Re-stated: R-1019 (NARROWED, dated 2026-10-21), R-1083, R-462 (NARROWED),
R-49 (READY, D21). **Both briefs together: 161 before, 156 after; closed 11, opened 6.**
## Observations — part 2
1. demo-hp's local whole-guest copy is never restore-tested: one copy kept, a 24 h settle, replaced at 24 h. FILED: R-1171
2. A successful put-back is logged at ERROR, and an empty `backups/restore-undo/` folder stays after a restore. FILED: R-1172
3. R-921's no-stop branch has no live observation yet (it needs a night schedule). FILED: R-1173
4. R-462's „only three apps are tested" was out of date: 58 apps have fixtures on both venues. NOT-A-FINDING: the row is re-stated with today's list.
5. The hub deploy at 06:03 UTC removed the old pod's log, which held the 07:10 Budapest off-site audit lines of today (R-895). NOT-A-FINDING: R-895's dated check is 2026-10-21; that run happens in the new pod.
6. In step 4 of the 9202 drill the data-only md5 differed from a read taken before snapshot 2, while the schema and every row count were equal. NOT-A-FINDING: the two reads were at different moments (the read, then the snapshot's own dump); the claim under test — before and after the forced failure — matched on all three fingerprints.
7. The helper's brief expected 37 tables after the normal restore; snapshot 2 has 36. NOT-A-FINDING: last night's 37 was the mixed state, not the snapshot (my arithmetic, corrected).
8. The first background wait loop for demo-hp's agent update printed a time and stopped before the box had updated. NOT-A-FINDING: the version was read directly and a foreground loop used instead.
9. A Python edit of the controller CHANGELOG failed on a quote character, and the commit carried only REPORT.md. NOT-A-FINDING: fixed by the next commit (`7ca59e2`).
## Part 2 — teardown
- Machine: 9202 back on 0.331.0 as found (Rallly removed through the product, the SFTP target, images and files removed
by name, settings and the off-box folder put back); demo-hp 9201's probe script removed.
- Host: demo-hp `/tmp` copies removed; nothing on `pve/data`; no whole-guest backup of 9202.
- Hub: per-box floors set and cleared; the global floor raised; nothing else.
- ep0: read only.
---
# PART 1 — the evening/night brief of 2026-10-19
## In plain words
+209
View File
@@ -0,0 +1,209 @@
# REPORT — the night of 2026-10-21 → 22: make tomorrow's release safe
Unattended, 20:15 CEST → 08:00. Nothing released, deployed, published or vouched. Evidence:
`documentation/audits/night-2026-10-21/`. Design read first: `07` §6.3/§8 (the restore ladder, R-893 option A),
`03` §9 (guest sizing), `08` (notification groups), `10-localisation.md`, the controller CHANGELOG v0.336.0.
**Safest readings taken at the start (unattended):** (1) Tester4 is the soak customer — a new customer would provision on
ep0 and the Storage Box, which are read-only tonight; its old host was already deleted. (2) „One hub image" — so the
`node_down`-after-delete finding is a row, not a second image. (3) The website is published by a push, so W2 is a patch
file, not a commit. (4) The no-swap test ran after the box's last night job, not at 07:00 sharp (see Part 2).
## 1. Five lines
1. **Soak (Part 1): NO FREEZE — 3 of the 4 pass lines met, 1 missed.** 11 h on a fresh 8 GB box: no `/api/health` gap (largest interval 16 s), every night job ran, uploads complete except 1 photo of ~26 000 (R-1215); memory „full" pressure passed 20 % for ~3 min in the 03:30 second-drive job (peak 39 % host / 45 % guest over 10 s — R-1214).
2. **No-swap test (Part 2): THE FREEZE CAME BACK.** Host swap off, the same burst: the dashboard stopped answering within 20 s, dead up to 289 s at a time (24 of 125 probes failed), Immich lost 207 of 306 photos; it ended only when the Immich stream broke. Swap back on: normal at once. The agent warned once.
3. **D22 built and proven: YES** — the put-back covers the local, second-drive and kept restores and a restore cut by a
power loss; proven on 9202 (a damaged backup and a killed controller both kept the newest row and the app running).
4. **First-hour words: 22 fixed** (controller W9–W14, W21, W22, W24, W25, W28, W29; hub W3–W5; catalog W16–W19; ISO
source W1, W7); W6/W8 were already true; W2 is a ready patch (a publish).
5. **R-1199:** the hub mailed the household by a list saved two days before the box existed; the box never sent its own
— fixed in the controller (decided by CC, may be reversed). **R-1194:** a timing false alarm — the first report runs
before the emergency-door setup; fixed in the agent.
## 2. The morning list (copied from STATUS)
**Morning list, 2026-10-22.** CC decided one thing (you may reverse it): a box tells the hub the notification list its page shows, even if never saved (R-1199; the rehearsal household got 1 alarm of 9).
1. Release controller 0.338.0 (restore put-back incl. power cut, first-hour words, swap sentence)? On the scratch box a damaged backup and a killed controller both kept the newest data, app running (R-1177, R-1195, R-1178, R-1180).
2. Release and vouch agent 0.162.0 (swap = RAM, host-swap check, no false door alarm — R-1192, R-1194)? Soak, 11 h on a fresh 8 GB box: no freeze, every check answered, every night job ran; memory pressure passed the 20 % line for 3 min in the 03:30 backup (R-1214). With no host swap the freeze comes back; our installer gives 7.75 GB. I'd release.
3. Deploy hub 0.164.0 (host-swap alarm, bind-page words)? And delete the test host „Tester4-9aca27" (machine gone; my token cannot delete).
4. How was Tester 2's laptop installed — our installer, or its own Proxmox? (Without host swap the fix does not help.)
5. A publish: the website download button (patch ready), and with the next ISO the boot text and the cleared screen (R-1210). Secrets: keep or remove the `REHEARSAL_2026_10_20_*`, `FIXES_2026_10_21_*`, `NIGHT_2026_10_21_*` names in sec.
## 3. Part 1 — the soak (numbers)
Box: VM 9322 on demo-hp (4 vCPU, 8 GiB, 256 GB + 200 GB), installed from the public ISO, customer Tester4; agent fixture
`0.162.0-night.1` then `night.2` by hand (provenance `P1-soak/00-provenance.txt`). Two channels: the box's own 10-s
recorder (`03-rec.jsonl`, 3 632 samples) and a probe from DooPlex (`02-outside.txt`, 3 390 probes + demo-hp's view of the VM's
disk reads).
| | Load window 18:52–20:53 UTC | Night 20:53–04:05 UTC |
|---|---|---|
| health probes / failed | 684 / 0 | 2 449 / 0 |
| worst health gap (between answers) | 10–11 s (slowest answer 1.27 s) | **16 s** at 01:30:36 (slowest answer 4.07 s) |
| worst memory „full" (10 s, host / guest) | 31.2 % / 35.2 % (18:53:38, Immich model load; 1 sample) | **39.4 % / 45.5 %** (01:30:42, tier2 job; 12 samples > 20 %, ~3 min) |
| worst memory „full" (60 s) | — | 20.1 % / 25.0 % |
| OOM kills / guest limit hits | 0 / 0 | 0 / 0 |
| guest swap used (max) | 2 614 MB | 2 720 MB |
- **Guest swap raised:** 512 → 4096 MB by the fixture at 18:37:13 UTC (`04-agent-fixture.txt`).
- **Uploads:** Paperless 99 cycles × 53 PDFs, all accepted; Immich 87 cycles × 305 + the first upload — every user holds
303 photos (the set has 2 duplicates and 1 file Immich refuses by design), except cycle 36: 302 — **one photo hung 10 min
and was not stored** (19:38 UTC, pressure 3.9 %, health 0.26 s; Immich's log lost at its 02:16 restart — R-1215);
Nextcloud: both trees 3 031 of 3 033 files byte-identical to the manifest (sha256; the 2 are the uploader's by-design skips).
- **Night jobs (box log `10-controller-log.txt`):** db-dump 00:30 (4 DBs, 4 m 14 s), tier2 01:30 (4 m 15 s), off-site
02:15 (8 apps, 72 snapshots, 5 m 53 s), abandon-sweep 03:10, proof 03:30 (actualbudget PASSED), integrity 04:00
(1 m 22 s) — all completed. The agent's whole-guest backup ran at 18:41 UTC (24 h cadence; next due after the night).
- **Host swap the installer gives an 8 GB / 256 GB machine: 7.75 GB** (LVM `pve/swap`, SwapTotal 8 126 460 kB;
`02-host-swap-after-install.txt`).
- **Before (rehearsal, no fix):** „full" 96 %, dashboard dead for about an hour. **After:** no gap over 16 s.
## 4. Part 3 — D22, each scenario
| Scenario (9202, Rallly + Postgres, controller 0.338.0-rc.1) | Result |
|---|---|
| a normal local restore (the control) | unchanged: „1 adatkötet és az adatbázis visszaállítva", the expected rewind (A,B → A) |
| the unit's database volume archive damaged (cut to half) | put back: the live volume poured back, markers **A,C** (C = the newest live row), Rallly running, „…visszaállítása sikertelen. Nem változott semmi…" — on 0.331.0 (2026-10-20) the same case crash-looped Postgres and lost C |
| the controller killed (kill -9) right after the volume step, before the replay | on start: „interrupted restore is put back … 1 volume(s) poured back, definition=true", markers **A,C,D** (D newer than the backup), Rallly running, the record: „A visszaállítás félbeszakadt (áramszünet). Az alkalmazás adatai a visszaállítás előtti állapotban vannak." |
| off-site restore cut the same way | by test (`TestR1195_AKilledOffsiteRestoreIsPutBackOnStart`) |
| second drive: the unit refuses after the file half (R-1178) | by test: nothing moves, the app is never stopped |
| the night mirror during a restore (R-1180) | by test with the production rsync: skipped, mirrored the next night |
**Red-proofs seen failing, then passing:** `TestR1177_*` (4 failed on the old `restore_unit.go`: the old „completed with
data errors" text, no copy), `TestR1195_*` (the put-back at start switched off → „nothing poured back"; the off-site
journal not written → no undo), `TestR1195_TheRecordSays…` (old „indítsd el újra"), `TestR1178_*` (1 restored +
1 replaced), `TestR1180_*` ([a.jpg b.jpg]). Three older tests that pinned the old outcome were changed on purpose
(the app started on a half state is now held or put back). 9202 put back as found: controller 0.337.0, the same apps.
## 5. Part 2 — machines with no swap
**The test** (`P2-noswap/`): the soak box after its night, `swapoff -a` on the host at 04:06 UTC (48 s; it held 1.9 GB),
the burst from 04:08:25 to 04:38:27 (Immich machine-learning restarted so its models load again, as on the first afternoon),
then `swapon -a`. The first attempt (04:06–04:08) is VOID and kept: the load script restarted its names at 1, so Immich got
duplicates and Nextcloud skipped existing folders (`02-attempt1-VOID-duplicate-uploads.txt`); the rerun used fresh names.
| | No host swap (04:08–04:38) | Swap back on (04:38–04:44) |
|---|---|---|
| health probes / failed | 125 / **24** | 34 / 0 |
| gaps between answers | **289 s**, 172 s, 68 s | none |
| memory „full" (10 s, host / guest) | **36 % / 47 %** (94 of 180 samples > 20 %) | 1.2 % / 1.3 % |
| VM disk reads (demo-hp's view) | ~40 GB per 25 s at the worst | quiet |
| OOM kills | 0 | 0 |
| uploads | Immich 99 of 306, then its stream broke (a login answer that was not JSON); Paperless 52 of 53 in the first cycle, then 22 × 53; Nextcloud cut at the time limit | — |
**Said plainly:** without host swap the freeze comes back — the same shape as the rehearsal (refaults, disk reads, no OOM
kill), shorter only because the Immich stream died at 04:22 and took the load away. The agent logged its WARN once at the
swapoff („host_swap_mb=938 guest_swap_mb=4096").
**Built (commit only):** agent `0da792e` reports `host.host_swap_mb` (Proxmox node status = the kernel's SwapTotal — the
same number as `/proc/meminfo`; used instead of reading the file because the agent already holds the node status) and
`host.guest_swap_mb`, logs one WARN when short; `GET /guest/memory` carries both. Hub 0.164.0: operator-only
`host_swap_short` / `host_swap_ok`, edge-triggered from the host's newest swap event. Controller install page, when memory
is tight and the host's swap is below the guest's RAM: **HU** „A gépen nincs elég cserehely (swap), ezért nagy
feltöltéskor a doboz lelassulhat, és percekig sem válaszolhat. Szólj a Felhomnak, mi rendbe tesszük." **EN** „The machine
has too little swap space, so during large uploads the box can slow down and not answer for minutes. Tell Felhom; we will
fix it." Creating host swap on a BYO host is a decision → R-1207 (D27).
## 6. Part 5 — the two checks
**R-1199 (the household's mails went to the operator).** Hub DB read with `SELECT` on named non-secret columns only
(`P5-checks/r1199-01-hub-db-read.txt`): Tester4's stored list was saved **2026-10-18 18:18** — before the rehearsal box
existed — and holds `node_down, backup_failed, disk_critical, host_disk_critical, storage_fill_critical,
offbox_repo_orphaned, app_update_undone, app_update_held, app_stopped_unhealthy`: no `disk_warning`, no
`storage_disconnected`, no `crossdrive_failed`. The rehearsal day's log: the one household mail was `backup_failed`.
**Mechanism:** the box shows `settings.DefaultEnabledEvents` ticked when nothing was saved
(`controller/internal/settings/settings.go` `GetNotificationPrefs`), but the start-up sync pushes only when an address or
a clear was saved (`settings.go` `SyncOnStartup`, `cmd/controller/main.go` start-up goroutine), and the hub mails the
household only by its stored list (`hub/internal/notify/dispatcher.go` `processCustomer` → `isEventEnabled`). **Fix
(controller `ba22f9b`):** a box with no saved choice pushes the default list at start, with no address (the hub's
no-clobber guard keeps the stored one). The box's page is the source — `08` („the household decides"). *Decided by CC
unattended — operator may reverse.* `TestR1199_*` red-proved.
**R-1194 (emergency SSH door „broken" on a new box).** On the fresh soak box: the agent ran `sshd -t -f
/etc/felhom-sshd/sshd_config` **through sudo** at 20:33:59, the config was applied at 20:34:00.434, and the hub mailed
`oob_degraded` („config invalid") at 20:34 CEST; as the agent's user the check passes now (rc 0, port 8822 listening);
without sudo it fails on the host key — which the agent never does. **True candidate: the first report runs before the
felhom-sshd loop's first pass** (port 0, config never rendered) — a false alarm on every fresh box, operator key or not.
**Fix (agent `0da792e`):** no OOB stanza until the first pass has run (a failed pass counts). Red-proved.
## 7. Builds, commits, CI
| Artifact | Version | From | Digest / sha256 | State |
|---|---|---|---|---|
| controller | `0.338.0-rc.1` | `d582feb` | — | the D22 proof on 9202 |
| controller | `0.338.0-rc.2` | `f0abdd0` | `sha256:ae52c93ebb83af1c646986119a1b74b07f992ffb03017624d052ece6266ec748` | smoke on 9202 (pages 200, 0 error lines); 9202 back on 0.337.0 |
| hub | `0.164.0` | `6b0d23c2` | `sha256:931594e333a4e8a9784eff2fb144d022ab46f89dd6a865a93e6100dc28dbd9dc` | built, pushed, NOT deployed (manifest 0.163.0) |
| agent | `0.162.0` candidate | `0da792e`, clean clone, release flags | `5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928` | not published; fixtures `night.1` (`286a7e3`) and `night.2` (`0da792e`) ran on the soak host |
**Commits.** controller: `d582feb`, `37b79ca`, `8c1e290`, `28b05d6`, `5776d54` (the helper's three: W9–W29), `ba22f9b`,
`f0abdd0`, `1de8a55`. agent: `0da792e`, `b310773`. felhom.eu: `3c620f8d`, `f90e2229` (hub), `2fa5b8f3` (ISO source),
`6b0d23c2`, `2293b252`, `9d3bf76d`, `57c06ba9`, `307d25b1`, `f576d7f7`, `c2c130ec`, `dee26da3` + this report.
catalog: `13f9c9c` (live through the catalog sync). **CI:** controller 2175, 2178, 2181, 2184, 2186, 2187, 2188, 2191 —
all success; agent 2176, 2190 — success; catalog 2180 — success; felhom.eu 2177–2192 **failed on one suite only**
(`test_gate_decoys.py` hit the 300 s limit on the CI runner; R-1209, fixed in `c2c130ec`) — after the fix: `c2c130ec` 2193, `dee26da3` 2194, `97f088ec` 2195 — all success; the commit carrying this report is checked after its push (the final note)..
## 8. Teardown (three layers)
- **Machine:** VM 9322 `night1021-soak` and CT 9332 `night1021-loader` destroyed on demo-hp (names checked first) at
~04:47 UTC; their disk folders on nvme-scratch are gone (used 301 → 208 GB). The thin pool `local-lvm` read the same before
and after (26 159 563 KiB used) — never written. The public ISO on `local` was there before and stays. demo-hp's own clock
and swap untouched. The night's evidence was copied off first (logs, recorder, Immich counts).
- **Host (hub):** the host `Tester4-9aca27` stays on the hub (agent `0.162.0-night.2`); CC's token cannot delete a host —
**the operator deletes it** (morning list 3). Until then it reads down. The customer Tester4 stays (the kept one).
- **Off-site:** ep0 (read only) holds for Tester4 one copy, `ct/9201/2026-10-20T12:17:26Z`, from the rehearsal box —
nothing from tonight; nothing removed. The Storage Box holds Tester4's off-site repository with tonight's 72 snapshots —
left (a new Tester4 box sets it aside by design).
- **9202:** back on controller 0.337.0 with the apps it had; the D22 test app removed through the product.
## 9. Rows
Before **176**; after **184**; opened **9** (R-1207 host swap on a no-swap host — D27; R-1208 `node_down` after an
operator delete; R-1209 the CI time limit — closed the same night; R-1210 first-hour words waiting for a publish; R-1211
the household `backup_failed` mail; R-1212 two texts ignoring the box language; R-1213 operator detail left on two pages; R-1214 memory pressure in the night's second-drive job; R-1215 one Immich upload hung);
closed **1** (R-1209). Moved to WAITING-ON-OPERATOR (fixed on main, ship with the morning releases): R-1177, R-1178,
R-1180, R-1195 (controller), R-1194 (agent). R-1188 (a) fixed. `unproven.py --summary`: unchanged, 35 of 55 not walked.
## 10. Observations
- The decoy suite timed out on CI on every felhom.eu push of the night — **FILED: R-1209** (fixed, closed).
- `node_down` mailed for a customer whose host the operator had deleted — **FILED: R-1208**.
- A household `backup_failed` mail with no app and no step — **FILED: R-1211** (the helper's).
- Two backup texts ignore the box language — **FILED: R-1212**. Operator detail on two pages — **FILED: R-1213**.
- `TestR603_*` was red on controller main since 0.337.0 (CI runs gates, not `go test`) — **NOT-A-FINDING:** fixed in
the session (two English apostrophes), small.
- The first `resend` press on a spent link before the new box registered sent nothing — **NOT-A-FINDING:** a live link
existed (minted at the host delete, mailed 16:49); the page says „if it was a real link" by design.
- The helper chose different words than the brief for W25 („a következő éjjel", the retry is the same night when pressed
by day) and W29 („Leszerelve. Nem költöztettünk át adatot róla; ami rajta volt, a meghajtón maradt." — „adat nem volt
rajta" would be false) — **NOT-A-FINDING:** the brief's texts were examples; the reasons are in the CHANGELOG.
- On the installed app pages Mealie already showed the right address (R-498); W19 was wrong only on the install page —
**NOT-A-FINDING** beyond the catalog fix.
- An empty `backups/restore-undo/` folder stays after a put-back (only the per-app folder is removed) —
**NOT-A-FINDING:** harmless, no data.
- The soak load script's Nextcloud stream ran past the 2 hours (each stream finishes its current pass) —
**NOT-A-FINDING:** harness.
- Memory „full" above 20 % in the night's second-drive job (~3 min) — **FILED: R-1214**. One Immich upload hung and was
not stored — **FILED: R-1215**.
- Without host swap the freeze comes back — **recorded on R-1207** (the operator's D27), not a new row.
- One `execCommand failed: exit status 1` at 00:30:49 — **NOT-A-FINDING:** the memory check's `docker inspect` of all
containers ran 1 s after actualbudget was stopped for its dump; the code keeps the other answers by design.
- One Radicale health probe timed out at 01:30:24 (the second-drive job starting) — **NOT-A-FINDING:** the next probe passed,
no alarm, the household's door answered throughout.
- The guest hit its memory limit 101 times at 18:39 UTC, during the app installs, before the load — **NOT-A-FINDING:** page
cache reclaim (0 OOM kills, swap 40 KB used), never again in 11 h.
- A 637 MB `vzdump-lxc-9100-…` on the fresh box's `local` — **NOT-A-FINDING:** the installer's golden archive (9100 = the
golden build guest, `felhom-host-install.sh`); the restore test skips it with a reason.
- The byte-check tool `tools/nc_compare.py` turns TLS checking off — **NOT-A-FINDING:** a measurement script against a
throwaway box with a self-signed certificate and a test domain that does not resolve; the box is destroyed.
- The first no-swap attempt reused the soak's names (Immich duplicates, Nextcloud skips) — **NOT-A-FINDING:** harness;
stopped after 2 min, kept as VOID, rerun with fresh names (`tools/load2.py` now takes a name tag).
- CC ran an old one-off register script by mistake at 04:07 UTC; it added 16 duplicate rows to the register file —
**NOT-A-FINDING:** caught at once by the row count, the file restored from git before anything was committed; the
script renamed so it cannot run again.
**Fixed without a row:** the R-603 test, the restore-record log line after a put-back, the CHANGELOG blank line.
**Secrets:** names only — `NIGHT_2026_10_21_ROOTPW`, `NIGHT_2026_10_21_PAIRING_CODE`, `NIGHT_2026_10_21_BIND_TOKEN`,
`NIGHT_2026_10_21_SETUP_CODE` (the last three spent). The bind link and the setup code were read from the mailbox through
the mail connector, so both one-time values entered this session's record (spent within minutes). A scan of the evidence
for every secret used tonight: 0 hits — all 34 secret names of the night, the rehearsal and the fixes run, across the 75 evidence files, this report, STATUS, CONTEXT and the register (positive control: a known word in the evidence was found). **Poster:** no fact in `felhom-system-poster.facts.md` changed (nothing
released). **Instruction files:** none edited.
+111
View File
@@ -0,0 +1,111 @@
# REPORT — night 2026-10-22: seven decisions, the data-loss candidates, the 03:30 pressure, the tunnel alarm, private repositories
Evening → night brief, started ~16:00 CEST. Nothing was released, deployed or published, except the catalog change and
the scratch box's catalog login (both allowed by the brief). Design read: `07-backup-architecture.md` §6.1.1, §6.2,
§10.2; `03-host-agent.md` (tunnel, bundle); `05-hub-architecture.md` §15; `06-offsite-connectivity.md`;
`09-update-architecture.md` §6.5.
## 1. Five lines
1. **D8:** done before 18:00 — DooPlex's Docker settings come from git (no change, no restart), and a weekly build-cache
prune timer runs Sunday 23:30.
2. **The seven decisions:** live — D6, D8, D27 (report only). Ready, waiting for a release or publish — D21
(controller 0.340.0), D3 + D5 (one website patch), D4 (agent bundle patch + installer patch).
3. **Data-loss candidates:** 2 proven and fixed in code (R-1187 c, R-1189 d); 5 real in code, now rows with a written
fix (R-1218, R-1219, R-1220, R-1221, R-1222); 1 closed half (R-1189 b, fixed in 0.304.0); 3 to the operator
(R-1205 as D31, R-1189 f, R-1190 d); the rest stay as named candidates.
4. **03:30 pressure:** the second-drive copy's file cache is charged to the guest and pushes the apps out of memory.
The fix runs the copy in a 512 MB memory-capped container: full stall 5.7 s → 0.37 s on the scratch box.
5. **Private repositories:** only felhom-controller can go private (after one CI change). felhom.eu, the agent and the
catalog must stay public: the website, every install and every box read them with no login.
## 2. The morning list (also in STATUS)
1. Publish the website patch (D3 wording, D5 hardware page).
2. D4: the agent bundle patch with the next agent release (your vouch) + the installer patch with a new installer tag.
Discard appliance 49 on the hub's Hosts page (tonight's test install).
3. Deploy hub 0.166.0 (tunnel alarm) and release the agent with `handshake_future_s` — one agent release with item 2.
4. Release controller 0.340.0 (R-49, R-1187 c, R-1189 d, R-1214), MinAgent 0.158.1.
5. Decide D29 — the controller repository private? CC: No, not now.
6. Decide D30 — the tunnel repair. CC: the agent makes a new key and registers it on the path that exists.
7. (D31, R-1205, when there is time.) Tester 2's „go" from yesterday stays on the list.
## 3. Per candidate
| Item | Result | Reason |
|---|---|---|
| R-1187 a | REAL in code → row R-1187 (now P2) | an export drops a folder with a symlink and still says done; capture code: needs a red-proof + 9202 proof |
| R-1187 b | REAL in code → R-1187 | the import sets no owners and drops symlinks |
| R-1187 c | **PROVEN and FIXED** (`d5f88a2`) | 9202 rc.1: an import over a stopped app merged old + new data; rc.2: refused, app untouched |
| R-1187 d | PLAUSIBLE → R-1187 | no busy check; after (a) |
| R-1187 f, g | REAL in code → R-1187 | HDD_PATH trusted; postgis missed, psql without ON_ERROR_STOP |
| R-1188 a, c | already CLOSED (R-1195; 0.337.0) | — |
| R-1188 b, d, e | R-1188, re-ranked P4 | unlikely or a sentence |
| R-1189 a | REAL, half-live → R-1189 | agent code; rides an agent release |
| R-1189 b | SPLIT: skipped nights CLOSED (R-899, 0.304.0); creep → **R-1221** (P4) | demo-hp journal: a night run after each of 3 off-cycle runs; creep 1.5–5 min/night measured |
| R-1189 c, g, i | REAL in code → R-1189 (P4-shaped) | quiesce vs restore; spring forward 2027; double click |
| R-1189 d | **FIXED** (`d5f88a2`) | the catch-up now runs the proof + integrity once; red-proved |
| R-1189 e | REAL in code → **R-1222** (hub, P3) | an evening backup time gives a false „dump missed" every morning |
| R-1189 f | operator → R-1189 | design question |
| R-1190 a | REAL in code → **R-1218** (P2) | proposed fix, not built: key code needs a live proof, and 9202 has no hub |
| R-1190 b | REAL in code → **R-1219** (P2) | the top bar links to a ceremony the page hides during an offer |
| R-1190 c | PLAUSIBLE → R-1190 | a read-only fingerprint compare decides it |
| R-1190 d | operator → R-1190 | design question |
| R-1190 g | REAL in code → **R-1220** (P2) | an unreadable app.yaml lists a live folder as kept (deletable) |
| R-1205 | operator → **D31** | the F-14 gate is deliberate (ruling 2026-07-13) |
| R-1206 | NARROWED | Felhom tier: no loss, by design (R-822 guard). NAS tier: a unit test is owed |
Research input: `documentation/audits/night-2026-10-22/P3-candidates/analysis.md`.
## 4. Records
**Commits.**
- felhom-controller: `6a17c95` (R-49), `d5f88a2` (R-1187 c, R-1189 d), `03c0a69` + `ae39467` (R-1214), `df78b6c` (docs).
CI jobs 2214, 2220, 2221, 2222, 2223 — all success, attempt 1.
- felhom-agent: `a0c88e5` (R-1204 + D4), `7651461` (D4 moved to a patch), `e803ad2` (docs, a correction). CI 2217,
2218, 2224 success.
- felhom.eu: `0639cf07`, `e267c5e2` + `1100a344` (one push), `acc78637`. CI 2215, 2219, 2225 success (2219 covers `e267c5e2`).
- app-catalog-felhom.eu: `ce33e3b` (Immich `backup.volumes`). CI 2216 success.
- homelab-manifests: `b35ba83` (D8), `c7506df` + `b0cce29` (REPORT). No CI in that repo.
- Drill catalog (scratch, no CI by design): `2ba4b53`, `9ebdba6`, `b0350a3` (tree = live `ce33e3b` again).
**Images pushed.** `felhom-controller:0.340.0-rc.1`..`rc.4` (never offered to a household: not X.Y.Z).
`felhom-hub:0.166.0` (built, NOT deployed: the live hub and `manifests/hub.yaml` are 0.165.0, read 18:2x).
**Teardown, three layers.**
- **Machine (9202):** the throwaway uptime-kuma removed through the product (its `.fab` deleted); the hog container and
19 GB of test files removed; memory 25898 MB / swap 512 MB again; Immich's `.felhom.yml` back to its pre-night file;
controller back on 0.339.0 (healthy, every app up). Kept on purpose: the `drill-reader` catalog login (D6).
Left: the four rc images in 9202's own Docker store, like earlier rc images there.
- **Host (demo-hp):** VM 9343 destroyed (own id); `/tmp/r1214*` removed; two sampler loops stopped. The scratch
storage and VM 341 / CT 9401 are standing items, not tonight's.
- **Hub:** appliance 49 from VM 9343 is still „registered" — Discard is the operator's (session-only). Nothing else.
- **DooPlex:** the D8 timer (intended). Nothing else changed.
**Rows:** before 178, after 180. Opened 5: R-1218, R-1219, R-1220, R-1221, R-1222. Closed 3: R-206 (D8), R-992 (D6),
R-1207 (D27). Updated: R-49, R-925, R-982, R-985, R-991, R-1187 (P3→P2), R-1188 (P3→P4), R-1189, R-1190, R-1204,
R-1205, R-1206, R-1214, R-1215.
**Unproven claims:** 35 of 55 not walked — unchanged. **System poster:** no listed fact changed (nothing that the
poster draws shipped tonight) — none of the three actions was needed.
**Observations.**
- 9202's sync refused Immich's template (R-971: Immich's data sits on a bench mount, not a household drive). —
NOT-A-FINDING: R-971 by design; the file was placed by hand for the proof and put back.
- The local unit restore of Immich was refused (R-538). — NOT-A-FINDING: by design; the proof used an empty-cache start.
- A non-`--fast` catalog gate run started the runtime gate on DooPlex; its canary refused and left nothing. —
NOT-A-FINDING: the canary is the control; the rule (`--fast` on DooPlex) exists.
- Images and packages stay public when a repository goes private. — NOT-A-FINDING: they follow the user `admin`, and
boxes must pull them; recorded in R-925 and D29.
- A CI read token is a precondition of D29. — NOT-A-FINDING: a step of that decision, written in it.
- R-925 said „32 source comments"; the served count is 61 on four pages. — fixed in the row.
- **CC's own mistake 1:** the agent CHANGELOG and felhom.eu commit `1100a344`'s message say a push of the installer
publishes it. Wrong: `/scripts/` is served from the tag `installer-v1.35.0` (R-110). Corrected in the agent CHANGELOG
(`e803ad2`) and R-982; the commit message cannot change (no force-push). Keeping D4 as a patch still holds: both
halves need a release that was forbidden tonight.
- **CC's own mistake 2:** controller commit `df78b6c` carries a „Co-Authored-By" line, which the brief forbids. It stays
(no force-push); no other commit tonight has one.
- **CC's own mistake 3:** homelab-manifests' REPORT first claimed a separate daemon.json backup. There was none — the
file did not change and is byte-identical to the role's copy in git (`cmp`). Corrected in `b0cce29`.
- The guest has no rsync. — NOT-A-FINDING: rsync lives in the controller image; the capped copy uses that image.
- R-1215's evidence is gone (Immich restarted at 02:16). — said in the row; it stays a watch.
+221
View File
@@ -0,0 +1,221 @@
# REPORT — night 2026-10-22 → 23: the stray-copy fix, D30/D31 built, a fresh review, one full test night, the DooPlex survey
Night brief 20:15 → 08:00, unattended. **Nothing was released, deployed, published or vouched.** No change on ep0 or to
DooPlex's configuration. Design read: `07-backup-architecture.md` §6.3, §8 (now §8.2b); `03-host-agent.md` and
`06-offsite-connectivity.md` (the WireGuard peer, the key, the full-list push to ep0); `05-hub-architecture.md`
(re-issue after a host delete, §15.4b); `01-topology-and-trust.md`, `runbooks/total-loss-of-dooplex.md`,
`runbooks/RUNBOOK-hub-db-offsite-backup.md`, `runbooks/gitea-restore.md`, `runbooks/secrets.md`.
**Safest readings taken (unattended):** (1) D29/D30/D31 are the operator's accepted picks (brief §0); built as picked.
(2) The test box binds to Tester4 with the bind link the hub minted at 2026-10-22 07:37 (unconsumed, valid to 10-29) —
no new link was minted. (3) The test box's root password is read from the hub vault (the break-glass path) because the
day-0 install rotates it. (4) No hub image is built tonight: the morning builds and deploys hub 0.167.0 in one step
(a hub image build in an unattended session is not CC's, `09` §3 decision 162). (5) Review fixes to code that copies or
deletes customer data were made only with a red-proof AND a 9202 proof (rc.8).
## 0. Five lines (plain words)
1. **Stray-copy fix proven:** the night copy's container can no longer outlive the controller — removed by name on an
error or timeout, every leftover removed at start, a restore refused while a copy runs; proven twice on the scratch
box (rc.5, and rc.8 after two more review rounds).
2. **D30 and D31 built:** D30 proven live — a box's tunnel that ep0 refused healed itself with a new key in 20 minutes
(and a short clock jump heals by itself, no new key needed); D31 built and red-proved (hub, not deployed).
3. **The review:** four fresh reviews and the commits' own security review — 41 points; every real one is fixed (code,
red-proved; copy code also on the scratch box) or a row; one question for you (D31's reach, morning item 1).
4. **The test night: PASS** — a fresh 8 GB box with the release set, a household and 3.5 h of uploads: at night 0 of 1 833
health checks failed (worst gap 12 s), every night job ran and passed, nothing uploaded was lost, the 03:30 copy
peaked at 14 % memory stall (39 % on 2026-10-21). One gap: no whole-box copy to ep0 (R-1232, likely an old copy counted).
5. **DooPlex's three biggest gaps:** no printed break-glass sheet (without it no off-site copy opens), the image
registry and agent packages exist only on DooPlex (no update, reinstall or install after a loss), and the website,
installer and hub all hang on the home line.
## 4. Part 4 — one full test night (Tester4-16fdef, nested VM 9344)
Installed tonight from ISO 1.30.0; the release set by hand (controller rc.6 → rc.8, agent `0.163.0-night.1` → `0.162.99`
= agent main, the D4 drop-in by the bundle's own steps); the rehearsal's 8 apps; household seed 20261023 (3 414 files,
3.5 GB); uploads 20:04–23:38 UTC; the night chain untouched. Evidence: `audits/night-2026-10-23/P4-testnight/`
(`00-provenance.txt` first — it records the extra load below).
| | Load window 19:56–23:38 UTC | Night 23:38–05:00 UTC |
|---|---|---|
| health probes / failed | 1 175 / 2 (both explained, below) | 1 833 / **0** |
| worst gap between answers | 23 s (22:03, my planned controller swap) | **12 s** (slowest answer 0.27 s) |
| worst memory „full" (10 s, host / guest) | 22.7 % / 24.3 % (1 sample) | 28.2 % / 30.6 % (00:32, the dump leg; 1 sample) |
| samples > 20 % | 2, never two in a row | 4, never two in a row |
| the 03:30 second-drive copy | — | **13.6 % / 17.4 % at most** (2026-10-21: 39.4 % / 45.5 %, ~3 min) |
| OOM kills | 0 | 0 |
- **Night jobs, all on time, all passed:** dumps 00:30 (3 m 55 s; Immich's model cache „excluded by the template — not
dumped" = D21); second drive 01:30 (8 apps, 4 m 47 s, each copy in its named capped container, **none left after**);
off-site 02:15 (8 apps, 80 snapshots, 5 m 27 s); proof 03:30 PASSED; integrity 04:00 PASSED (100 % re-read).
- **Uploads:** Immich — 90 users, each exactly 303 photos (the set's 306 minus 2 duplicates and 1 empty file refused by
design): **nothing lost** (R-1215 did not recur); Paperless — every upload accepted; Nextcloud — 3 031 of 3 033 files byte-identical to the manifest (sha256; the 2 are the uploader's by-design
skips: the 1.5 GB file over its 1 GB limit and the link to it — the same as 2026-10-21) (`18-nextcloud-sha256.txt`).
- **D4:** root password over SSH refused, root key works, the web page's password login works (console not re-run
tonight; proven 2026-10-22). **D30:** proven on this box before the uploads (Part 2).
- **The two failed probes:** 20:16:34 — the hub judged the box „stuck" while it only waited for the household's
recovery code, re-issued the off-site login, and the box's controller restarted (R-1230, fixed in hub source tonight);
22:03:40 — my planned switch to rc.8.
- **Extra load, said plainly:** the first (void) load attempt's loop was not stopped (my `pkill -f` killed its own shell
first) and ran beside the planned load 19:56–22:14 — about twice the planned upload work for two hours. The box
carried it without a gap.
- **Not seen tonight:** the whole-box copy to ep0 did not run; most likely the old Tester4 box's ep0 copy (same guest
number) counts as this box's weekly copy — R-1232 (P2, likely, not proven). The local whole-box tier ran at install
(19:20) and is due again after 24 h, correctly.
- Other: one `[ERROR] execCommand failed` line naming nothing (also on 2026-10-21) — R-1233.
## 3. Part 3 — the fresh review: findings → verdicts
Three read-only reviewers (controller `v0.339.0..main`, agent `v0.162.0..main` + the D4 patches, hub `0.165.0..main` +
the website patch), then a fourth on tonight's own new code, then the commits' automatic security review. Each finding
checked at file:line by the main session.
| # | Finding | Verdict |
|---|---|---|
| C1 | the capped copy's container outlives the controller | REAL → fixed (Part 1) |
| C2 | two more `.fab` import paths merge old and new data | REAL, pre-existing → R-1187 (recovery code) — not a hold |
| C3 | the import refusal also catches an unhealthy app, wrong advice | REAL → text fixed; the path → R-1187 |
| C4 | `-v` creates a missing path on the daemon side | REAL → `--mount` |
| C5 | the catch-up holds the whole-box backup during its slow checks | REAL → fixed, red-proved |
| C6, C8 | comments / README overstate | REAL → fixed |
| C7 | image tag, no `--pull never` | REAL → fixed; „no memory cgroup": NOT-A-FINDING (every guest is an LXC on a cgroup-v2 Proxmox kernel) |
| A1 | `systemctl reload ssh` fails on a stopped / socket-activated sshd → bundle route frozen | REAL → D4 patch v2 (`try-reload-or-restart`) |
| A2 | nothing checks the drop-in is in force | REAL → v2 reads `sshd -T` back, reports `sshd_dropin_not_in_force` |
| A3 | an undone bundle leaves sshd reloaded with the drop-in | REAL → v2 reloads in undo |
| A4 | downgrade / rollout order | NOT-A-FINDING: bundles go forward by signed jobs only; the morning list ships both halves in one step |
| A5 | the installer swallows a failed reload | REAL → installer patch v2 warns |
| A6, A7 | a wrong citation; „may only ever" wording | REAL → v2 |
| A8 / H1 | after a reboot the future stamp is gone, nothing alarms | REAL → R-1223 |
| H2, H7 | „ok" said to be mailed; WARN for ok | REAL → comment and log level fixed |
| H3 | the operator cooldown is per customer, not per host | REAL → R-1224 |
| H4 | the alarm repeats after the 90-day event prune | NOT-A-FINDING: a reminder every 90 days for a tunnel still dead is wanted; host_swap does the same |
| H5 | a wire-gate entry is false now | REAL → fixed |
| H6 | two reports at once could raise two events | NOT-A-FINDING: the agent reports from one loop, one at a time |
| W1 | the website patch applies, no over-promise | — |
| W2 | 8 GB minimum cannot run Immich | REAL → R-1225 (the operator's wording) |
| W3 | FAQ: data leaves home only encrypted (Cloudflare reads tunnel traffic) | REAL, pre-existing → R-1226 (the operator's wording) |
| N1 | D31's „never made a key" = „held no escrow row" — a deleted install that held a PBS secret is skipped too | REAL as a mechanism → **hold or go** (morning item 1); CC: go |
| N2 | one failed `docker rm` refuses every restore until a restart | REAL → fixed (per-name strays) |
| N3 | the start-up retry removes this process's own copies | REAL → fixed |
| N4 | a second copy of the same destination kills the first | REAL → fixed (one per destination) |
| N5 | `RunTier2` from the „Tier 2 on" save ignores a running restore | REAL, pre-existing → fixed |
| N6 | a power cut during the key rotation can leave no key | REAL → fixed (fsync, link, rename over) |
| N7 | rotation can repeat hourly; the first old key overwritten | REAL → fixed (first key kept, bound saved) |
| N8 | „within about 15 minutes" | REAL → text fixed |
| N9 | the import refusal advises removing an unhealthy app | REAL → text fixed |
| S1 | (security review) the start-up removal could hit a copy starting between list and remove | REAL → fixed (the destination's lock) |
| S2 | (security review) a restore beginning between the copy's check and its record | REAL → fixed (one step under the lock; every leg asks again) |
| S3 | (security review) a wait stamp dated after now blocks the re-key | REAL → fixed |
## 1. Part 1 — the stray-copy fix
The capped second-drive copy (R-1214) ran `rsync -a --delete` in its own container; killing the `docker` CLI (the
60-minute limit, a restart, a self-update) left the container deleting outside every guard. Controller `66a69d4`,
`912e592`, then after the reviews `943c517` (C5), `9d9e404` (N2–N5, N9), `1babfe9` (two races): a fixed name + label;
removed by name on any error; every leftover removed at start before the scheduler and the web server; a restore
refused while a copy runs or may run; one copy per destination at a time; `RunTier2` refuses during a restore for every
caller, in one step with its record; `--mount`, the image ID, `--pull never`, `--no-healthcheck`.
**Proof on 9202:** rc.5 and again rc.8 (`audits/night-2026-10-23/P1-straycopy/`): a restart in the middle of a 16 GB
copy → the old container removed at start, by name, logged; a restore probe during the copy refused; the redone copy
equal to the source; a 15-second test build → the timed-out container removed, the partial copy stopped growing; two
runs into the same destination → the second waited, no kill, no failure. 9202 back on 0.339.0.
## 2. Part 2 — D30, D31, D29
- **D30** (agent `ae69ad4`, hardened `39fb9cb`, `b9b831f`; hub `84751cb0`, `c6649dd4`): live on the test box
(`P2-D30/`). Run 1: a 2-hour clock jump healed by itself in 3 minutes — ep0 started a handshake (it had traffic for the
box) — and the agent correctly did not re-key. Run 2 (ep0's handshake starts dropped on the box, the rehearsal's
condition): dead tunnel → 15 min → **RE-KEYED** → the hub took the new key in place (`existed=false generation=3
sync=ok`, same 10.77.0.5) → the box switched at its next desired-state fetch → a handshake with the new key, tunnel
ping ok. The live hub (0.165.0) stored the alarm's input (`handshake_future_s=6851`); the alarm and its clearing are
pinned by hub tests (0.167.0, not deployed). **Nothing new on ep0:** the hub's existing full-list push.
- **D31** (hub `84751cb0`): the F-14 re-issue reads the newest deletion of a host that held a key; red-proved. The
review's N1 is the morning's hold-or-go question.
- **D29**: recorded (CONTEXT, R-925).
## 5. Part 5 — the DooPlex survey (read only)
`documentation/audits/dooplex-survival-2026-10-22/PLAN.md`: every thing on DooPlex with where it lives, what copies it,
where the copy is, who can restore it, how long — measured from unit states, journals, sizes, pods, DNS. Both nightly
off-site jobs and both weekly restore tests are green. `total-loss-of-dooplex.md` marked step by step (works today /
works once / never tried; one stale line fixed: the mail hold shipped in hub 0.145.0). Ten gaps, each with two options and
CC's pick; four go to the operator as D32–D35. **The three biggest:** the paper sheet (without S1 no off-site copy
opens), the registry and agent packages (only on DooPlex — no update, reinstall or install after a loss), and the home
line (website, installer, hub — new installs stop).
## 4. Records
**Commits.**
- felhom-controller: `66a69d4`, `912e592`, `943c517`, `9d9e404`, `1babfe9`, `23f77f9` (docs).
- felhom-agent: `ae69ad4`, `39fb9cb`, `b9b831f`, `b45c47b` (REPORT).
- felhom.eu: `84751cb0` (hub D30/D31 + Part 1 evidence), `75854432` (D30 proof, D4 v2, survey, 03/05/07, CONTEXT),
`c6649dd4` (hub text), `70a600d0` (register), `1805f8bb` (STATUS), + the report commit.
- **CI by commit (Gitea jobs, all success, attempt 1):** controller `66a69d4` 2227, `912e592` 2228, `943c517` 2231,
`9d9e404` 2233, `1babfe9` 2234, `23f77f9` 2240; agent `ae69ad4` 2229, `39fb9cb` 2235, `b9b831f` 2237, `b45c47b` 2241;
felhom.eu `84751cb0` 2230, `75854432` 2232, `c6649dd4` 2236, `1805f8bb` 2238 (with `70a600d0`), `e94d0256` 2239 (with
`14388e21`), `5dc17cab` 2242 (with `f56f3328`), `7802da72` 2243 (with `1cb3e571`) — a commit pushed with a later one
has no job of its own; the last report commit's job is checked after this file.
**Images.** Controller `0.340.0-rc.5` (= `66a69d4`), `rc.6` (= `943c517`), `rc.7` (= `9d9e404`, built, not used: the
security review came first), `rc.8` (= `1babfe9`). No hub image (the morning builds 0.167.0 with its deploy). A
test-only controller image (rc.5 + a 15 s limit) lived only on DooPlex and 9202 and was removed from both.
**Dated checks due today, done:** R-32 (Tester4's listing 72 → 80, none vanished), R-1173 (the other outcome).
**Rows:** before 180, after 191. Opened 11: R-1223 (the tunnel reboot gap), R-1224 (the per-customer cooldown), R-1225 and
R-1226 (two website sentences, D36), R-1227 (the registry, D33), R-1228 (the home line, D35), R-1229 (ArgoCD settings),
R-1230 (a held box reported stuck — fixed in hub source), R-1231 (the recovery page says „older"), R-1232 (the first ep0
copy of a reinstalled box), R-1233 (an error line naming nothing). Closed 0 — the night's fixes close on release
(R-1214, R-1204, R-1205, R-1230 are READY). Updated: R-1214, R-1204, R-1205, R-925, R-1187, R-982, R-232, R-923, R-924.
**Teardown, three layers.**
- **Machine:** 9202 back on 0.339.0, healthy, test files and the test image removed, no copy container (rc.5 / rc.8 stay
in its own image store, as earlier rc images). The test box: VM 9344 destroyed with its disks, after every log was
copied off (`P4-testnight/13`–`21`).
- **Host (demo-hp):** VM 9344 and loader CT 9345 destroyed (own IDs; disks were on nvme-scratch, never the thin pool);
`/tmp` clean. Standing items untouched: VM 341, CT 9401, the ISOs in `local` (see R-548).
- **Hub:** host **Tester4-16fdef** stays registered (it will read „down") — **the operator deletes it** on the Hosts page,
with appliance 49 (morning item 6). Tester4's off-site store holds tonight's 8 new snapshots (80 in all; left, as on
2026-10-21); its PBS token was re-issued automatically at the install (F-14, product behaviour); its WireGuard peer
holds the D30 key until the host is deleted.
- **DooPlex:** nothing changed — the survey read only; builds and the scratchpad only; the local test image removed.
**System poster:** no listed fact changed (no machine, time, tier or custody moved; the WireGuard key rotation is not a
poster fact) — none of the three actions needed. **Instruction files:** none edited.
**Observations.**
- demo-hp (a standing box): its local whole-box tier was refused 4 times tonight by the agent's space check (8.5 GB free,
9.2 GB needed), and the controller stopped 11 apps for about a minute before each try — FILED: R-548 (updated; the
superseded installer ISOs on `local` are the easy half).
- The capped copy inherited the controller's health check („starting") — NOT-A-FINDING: small, fixed in the session
(`--no-healthcheck`, `912e592`).
- One outside health check answered 502 at 20:16:34Z: the hub's false „stuck" re-issue restarted the controller —
FILED: R-1230 (fixed in hub source tonight).
- The Nextcloud tree upload takes ~3 h under the load (same as 2026-10-21: 18:53 → 21:56). — NOT-A-FINDING: the same
load gives the same time; not a regression.
- The soak tools assumed the setup gate takes the dashboard cookie; the gate now hands out its own cookie
(`felhom_gate`) through `/__gate/start`. — NOT-A-FINDING: the product works as designed; the tool was updated.
- The first load attempt (19:55Z) got 404/401 everywhere and was stopped after 2 minutes, logs kept as `void-1955/`
in the loader. — NOT-A-FINDING: the tool's gate cookie (above); nothing on the box changed.
- The VOID 9202 run's restore probe started a restore of a nonexistent app, failed at once. — NOT-A-FINDING: no data;
explained in `P1-straycopy/README.md`.
- A test request's error printed one character of a test password into the session (a Python traceback). —
NOT-A-FINDING for the product; the probe now encodes the password and prints nothing of it.
- The off-site unlock was refused once: the agent fixture's pre-release version string („0.163.0-night.1") is not a
version the controller can compare, so its capability read „unknown" — NOT-A-FINDING for the product (no pre-release
agent is ever released; the fixture was rebuilt as 0.162.99); the page's wording for „unknown" — FILED: R-1231.
- The household's off-site snapshots carry the rehearsal's future date (2026-10-28) — NOT-A-FINDING: known, R-1206;
retention was not due tonight, so its case did not come up.
- The night copy sat at the guest's memory limit while it ran (77 845 limit events, 0 OOM) with almost no stall — NOT-A-
FINDING: the capped copy's cache is reclaimed inside the guest by design; the consequence (pressure, health) is the
measure, and it improved.
- **CC's own mistakes:** (1) a commit (`70a600d0`) was made while its fast gate failed — the command chained the commit
after printing the gate's code; the push hook refused it, STATUS was fixed, and both commits went up together with
green gates. (2) Two gate runs at once (mine and the push hook's) made the decoy test fail; it passes alone. (3) The
void load loop ran on for 2 h 18 min because my `pkill -f load2.py` matched its own shell (a known trap); found at
22:14, stopped by pid, recorded in the provenance; the same trap hit once more at the teardown (a `pkill -f` killed my own
command; the teardown was rerun). (4) The first agent fixture used a version string the controller
cannot compare, which cost one refused unlock.
+42
View File
@@ -0,0 +1,42 @@
# REPORT — Plex to the fixed version in the catalog and on the demo/test boxes (2026-10-23 day, Plex2Shell)
**Plain words first.**
- The catalog now gives Plex 1.43.5.11029-6bf8af3f0 (catalog `6cc317d`, pushed without the update test, on your ruling).
- One box ran Plex: demo-hp. It showed no sign of the attack. CC pressed Update; it runs the new Plex now.
- You have one decision (D39): may a security fix reach the boxes at night without the full update test?
## Per box
| Box | Plex before → after | Signs check | Box walk |
|---|---|---|---|
| demo-hp 9201 | 1.41.4.9463-630c9f557 → 1.43.5.11029-6bf8af3f0 | clean (NOT-A-FINDING; logs reach back only to 00:32 UTC today — CPU history and files cover 10-16 → 10-22) | green: all containers healthy, 0 errors, 0 warnings since the press |
| demo-hp 9202, demo-felhom 9201, tester-1 9201 | no Plex | — | — |
| Tester4 | hub host DOWN, VM gone | — | — |
| Tester 2 | no host on the hub | — | — |
Detail and evidence: `documentation/audits/plex-demo-boxes-2026-10-23/README.md`.
## Records
- Register 184 → 186: opened R-1238 (P2, Plex has no test record or fixture), R-1239 (P2, a fast lane for security
fixes — the operator's decision); closed none.
- `CONTEXT.md`: the 10:56 ruling. `STATUS.md`: decide row D39. No „broken" line: Part 2 was finished before the
records were written, so the line the brief asked for would have been added and removed in the same commit.
The „Updated" paragraph was left as the day session wrote it (it is that session's summary; this run adds one
decide row only).
- Architecture read: `09-update-architecture.md` §3 decisions 13 and 52; `13-threat-model.md` (Plex row);
`runbooks/target-selection.md`. Decision 13 is unchanged; no poster fact changed (none of the three poster actions).
- `--no-verify` was used ONCE, for catalog `6cc317d`, on the ruling. The `check-test-record-move.py` refusal is verbatim
in `app-catalog-felhom.eu/REPORT.md`.
## Fixed without a row
- The memory index (`.claude-memory/MEMORY.md`, outside git) had grown to 201 lines / 25 735 bytes at 11:09 (another
session), over the 200 / 25 600 ceiling, so `instructions_gate` failed and so did two script suites that run it.
Lines 71–72 (two memories about the deleted credentials file, both superseded by `sec`) became one line: 200 lines,
25 590 bytes. `repo_gates.py --fast` then OK. The index sits right at the ceiling; the next added line trips it again.
## Teardown
Machine: Plex stays on the new version; helper scripts removed from guest and host. Host: nothing provisioned.
Hub: reads only.
@@ -0,0 +1,75 @@
# REPORT — did the Plex attack on DooPlex reach Felhom's secrets? (2026-10-23)
## Plain words first
1. **Verdict: NOT REACHED.** Two open doors led from the hacked Plex to Felhom's secrets. Neither was used in the attack
days: the backup folder got no new connection at all, and Longhorn made no backup or disk outside its normal plan.
2. **The shared backup folder: closed to the home network and the tailnet** (your yes). It is **still open to every app
on DooPlex** — Longhorn needs that, so the real fix is the homelab session's.
3. **What you do next:** give the homelab session the hand-over list (`documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md`,
last section). Nothing needs to be rotated.
## The Part 4 table
In `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md` § Part 4. Short form: every Felhom secret on
DooPlex has exposure **none**. The ones a hacked pod *could* have reached: the hub's unencrypted nightly copies (with the
unsealed Cloudflare/tunnel/git tokens, R-975) through the NFS export or a Longhorn backup, and the operator signing keys
through the NFS file-handle escape (documented, not exercised). Sealed hub columns need a key that is not in the copies;
escrow blobs need each household's recovery code; the k8s API, the kubelet and the hub refuse a caller with no credential
(measured).
## The paths and the evidence per path
| path | existed? | evidence of use (window 10-17 04:00Z → 10-23 08:04Z) |
|---|---|---|
| NFS export `longhorn-pvc` (`*`, `no_root_squash`, `no_subtree_check`; `secure` void in pods) | yes | **unused** — `node_nfsd_connections_total` flat at 288 for 13 days, 3902 samples, no gap; positive control: today's three probe connections moved it to 4. nfsd is TCP-only. Reads only in Longhorn's backup hours |
| Longhorn API :9500, no login | yes | **no sign of use** — 0 backups first seen off-schedule (1-min scrape); every night wrote to the local store; no unexplained volume; the backup target is the 2025-11-29 one. Manager logs: lost (reboot) |
| hub `hub:8080` direct | yes, but no secret without a credential | 0 requests from 10.42.0.52 in `/data/access-log` (from 10-18 10:25Z; positive control: the probe's IP is logged). 10-17 04:00Z → 10-18 10:25Z: no record |
| k8s API, kubelet | no (401) | — |
| SSH, Samba on DooPlex | login needed | 0 sshd lines from the pod network (control: 61 from the LAN); no Samba client file for 10.42.0.52 (positive control: the probe made one) |
| Vaultwarden | login needed; vault encrypted | logs lost (reboot) |
**Evidence lost before this session started:** the 13:20Z maintenance reboot erased the pre-reboot container logs of
longhorn-manager, longhorn-ui, the ingress, the hub, Gitea, Vaultwarden, Grafana and Prometheus. Reproduced
independently from records that survived: Prometheus history (nfsd, Longhorn, cAdvisor, node network), the hub's own
access log, the journal (boot -1), Samba per-client files, CoreDNS error lines, the Alertmanager silence list. The
incident session's pre-reboot greps (10.42.0.52 in longhorn-manager, CNPG, Gitea, authentik: 0) are cited as theirs.
## For the homelab session
The seven items at the end of the audit README: the export still open to pods (and `secure` is void there); Longhorn API
and UI with no login; an unattributed Longhorn delete of the old Plex volume at 11:24:15Z; **~1.5 TB sent out by the Plex
pod 10-18 → 10-22** (not in INCIDENT.md) plus `hand.genddos.st` DNS lookups on 10-19; the reboot that erased the logs;
the monitoring stack and Headlamp reachable from every pod; a `python3 main.py` listener on :8443. No HM numbers minted;
`homelab-manifests` not touched.
## What this session changed
- **DooPlex `/etc/exports`** (operator yes, ~15:18Z): `longhorn-pvc` from `*` to `10.42.0.0/16`, `exportfs -ra`; Longhorn
target `available: true` at 15:20:10Z. Backup `/etc/exports.bak-2026-10-23-plex2shell`. The only change on DooPlex.
- **Probe pod** `default/cc-p2s-probe`: created 15:19:46Z, removed 15:33:44Z; before/after listings in the evidence. No
other object created. The incident's NetworkPolicies untouched.
- **felhom.eu:** the audit folder; R-1241 and R-1242 in `OPEN-ITEMS.md`; `13-threat-model.md` (attacker j, one boundary
row); `STATUS.md` (verdict in „What works", the P1 in „What is broken", one „do" item; the old „Updated" paragraph moved to
`STATUS-HISTORY.md`).
- Nothing rotated. No secret value written anywhere (evidence scanned for secret shapes; the only 64-hex string is the
sha256 of `/etc/exports`).
## Rows
Before 186, after 188; opened 2 (R-1241 P1, R-1242 P3), closed 0.
## Observations
- Hacked pod → hub backups + signing keys through the NFS export and the Longhorn API — FILED: R-1241.
- Demo NAS export on the same disk with `no_subtree_check` — FILED: R-1242.
- The hub answers pods without the geo filter — NOT-A-FINDING: the geo filter is not authentication; every no-token path refused.
- demo-felhom's Proxmox UI and SSH reachable from any DooPlex pod — NOT-A-FINDING: the door of R-982; added as attacker j in `13` §3.
- Pod logs erased by the reboot — NOT-A-FINDING for Felhom (the hub's access log persisted, R-1140); handed to the homelab session.
## Notes for the next session
- Teardown, three layers: machine — probe pod removed, export change kept on purpose (operator yes); host — none; hub —
none (read-only access-log aggregation in the running pod).
- System poster: no fact on it changed (none of the three applied).
- `unproven.py`: not run — no capability claim changed.
+48
View File
@@ -0,0 +1,48 @@
# REPORT — system poster refresh check + OMV guide fixes (2026-10-21)
Baseline `main` = `7a773207`. Own clone; documentation only; no machine, box or release touched.
Binding: `documentation/architecture/felhom-system-poster.facts.md`.
## Plain words
- **The new poster is in the repository**, the web page and the PDF. It took three exports.
- The first export was a folder of 6 files, and Claude Design had changed four texts. Nothing was committed then
(commit `3decdf94`, with the OMV guide fixes and R-1201).
- The second export was one file; two texts were still short. The third export passed every check.
- The STATUS line „System poster needs a refresh" is gone. R-1201 is closed.
## Part A — checks
Full table, all three rounds: `documentation/audits/poster-refresh-2026-10-21/CHECKS.md`.
Round 3: file PASS · shape: plain static HTML (better than the old loader) · two changes PASS · nothing lost PASS ·
secrets PASS · look PASS (PDF; HTML render not checked, no browser on DooPlex).
## Part B — committed (second commit)
- `documentation/architecture/felhom-system-poster.html` replaced; `felhom-system-poster.pdf` added (6.25 MB, A1 size).
- Facts file: one History line.
- STATUS: the poster line removed. R-1201 moved to `CLOSED-ITEMS.md`.
- Evidence: round-3 hashes, text diff, three PNG crops.
## Part C — `documentation/pilot/GUIDE-omv-kvm-hu.md` (first commit, `3decdf94`)
1. „Ha valami nem megy" item 7 (HDD_PATH error) removed; 8 → 7.
2. Jellyfin: „Ha a géped egy korábbi telepítővel…" sentence removed.
3. Memory bullet: „Az első napokban kevés alkalmazással indulunk; a nagy fotógyűjtemény feltöltését együtt csináljuk." added.
4. Operator note: dated line, point (4) obsolete since installer 1.33.0; history kept.
Re-checked: `scripts/iso/felhom-bootstrap.sh:165` fetches `https://felhom.eu/scripts/felhom-host-install.sh`, which
serves `SCRIPT_VERSION="1.35.0"` (≥ 1.33.0). `https://felhom.eu/letoltes` lists `felhom-installer-1.29.0-pve9.2-1.iso`
and `dceacae5…4829`, the same as the guide's step 3.
## Rows
Opened R-1201 (first commit), closed R-1201 (second commit). Register 178 → 179 → 178.
## Claims in the task that were wrong
- The PDF is not A2: 2400×1696.5 pt ≈ A1 size.
- `claude/DESIGN-poster-refresh-2026-10-21.md` is not in this repository: `claude/` is the planning project's folder
(as for `claude/MONTHLY-security-retest.md`).
- felhom.eu has no root `CHANGELOG.md`; this repo records sessions in `REPORT-<topic>.md` files, so this is one.
## Notes
- The pre-push hook refuses a clone outside `/mnt/5_hdd/felhom.eu`; the clones were moved under it. No `--no-verify`.
## Teardown
Provisioned nothing. Scratch copies and clones deleted. `~/temp/` untouched.
+158
View File
@@ -0,0 +1,158 @@
# REPORT — Tester-2 rehearsal with a pretend household + an independent data-loss hunt (2026-10-20 → 21)
> Run 2026-10-20 09:30 UTC → 2026-10-21 ~09:50 UTC. Drafted before the night, finished in the morning. This repo's
> shared `REPORT.md` was not touched: a parallel session works in the same clone, and the repo rule says the second
> session writes `REPORT-<topic>.md` only (the brief asked for both; this line is why one was skipped).
> Evidence: `documentation/audits/rehearsal-2026-10-20/` (journal, A0–B10, screens) and
> `documentation/audits/dataloss-hunt-2026-10-20/` (C1–C3). Architecture read for the claims: `07-backup-architecture.md`,
> `10-localisation.md`, `00-capability-map.md` (A15).
## Five sentences
A pretend household installed a box from the public ISO, linked it, claimed it, put eight apps and about 2 GB of family
data on it, used it for a day, and every customer restore path brought its data back. The first night's backups, the
clock change, three days off and power cuts in the middle of installs and catch-ups all worked, and the morning data was
byte-identical to the evening's. But the 8 GB minimum box froze twice on its first afternoon and only a plug pull helped
(R-1192), and the household's warning mails went to the operator instead (R-1199). When the box died, the household could
not even start again without the operator, and after the operator's press the rebuilt box could not reach its off-site
history at all — proven live (R-1176, P1). Nothing was released; 29 new register rows carry the rest.
## The ten places a household gets stuck
`documentation/audits/rehearsal-2026-10-20/B10-stuck/README.md`. In short:
1. The box froze twice on the first afternoon (≈1 h and ≈35 min); only pulling the plug helped (R-1192, P1).
2. The bind page needs an owner passphrase the household does not have — 2 h 4 min waiting for the operator.
3. A power cut during a full off-site restore leaves the app on the backup's older data; the page only says „interrupted" (R-1195).
4. A pulled drive breaks the last uploads (0 bytes or gone) although each was confirmed (R-1196).
5. The family's warning mails went to the operator, not the family (R-1199).
6. „Ödön" and „Írisz" are refused as family names (W12).
7. A new drive needs a „mount name" nobody knows; the button silently does nothing (W13).
8. Sharing waits for ever for a password the page never asks for (W21).
9. Errors arrive as raw technical English (W22, W25).
10. After the box died, the household was promised a mail that never came, and after the operator's press the rebuilt box
could not reach the family's off-site data (R-1203, R-1176). (Close behind: three apps' wrong „first steps", W17–W19.)
## Data-loss findings, ranked
| row | Sev | harm | reproduced | design / defect |
|---|---|---|---|---|
| R-1175 | P1 | shares on two drives sharing one second drive: every night one drive's share copy is pruned; the night a drive dies its copy is deleted | unit test on production code (`C3-D1-shares-tier2/`) | defect |
| R-1176 | P1 | a box rebuilt after „delete the old host" cannot reach its off-site history: no recovery code asked, a new key, the old history set aside on night one | hub store unit test (`C3-K1-…`) + **LIVE in B7** (2026-10-21) | defect |
| R-1191 | P1 | a `.fab` import makes a family-only app open to anyone | LIVE on scratch 9202 | defect (security) |
| R-1192 | P1 | the 8 GB box froze under first-day use; two plug pulls | LIVE on Tester4 (twice) | defect — cause not diagnosed (no shell on the box) |
| R-1177 | P2 | a failed local restore destroys the live data, keeps no copy | LIVE on 9202 | design question (D22 in STATUS) |
| R-1195 | P2 | a power cut mid full off-site restore: app on the backup's data, page says only „interrupted" | LIVE on Tester4 (B6 #2) | defect |
| R-1178 | P2 | second-drive „whole restore" can change files, refuse the DB, leave the app stopped and say „untouched" | unit test | defect |
| R-1180 | P2 | the nightly mirror (`rsync --delete`) can run during a restore from it | unit test | defect |
| R-1181 | P2 | a missing app folder deletes its second-drive copy in one night (the only copy for optional folders) | unit test | defect (D23 in STATUS) |
| R-1179 | P2 | a drive marked „away" can still be the second-drive target | code + unit test | defect |
| R-1182 | P2 | abandoning one set-aside off-site copy deletes every retained key package | code | defect |
| R-1199 | P2 | warning mails reach the operator, not the household | LIVE (two mailboxes) | defect |
| R-1204 | P2 | a box that ran once with its clock ahead loses its WireGuard tunnel to ep0 (PBS copies + operator door) until that date | LIVE (B6 #8, ep0 `wg show`) | defect |
| R-1203 | P2 | after a box death, the household is promised a link mail that never comes until the operator deletes the old host | LIVE (B7) | defect |
| R-1185 | P3 | a dump that fails its own check has already replaced the good one | unit test (docker stub) | defect (order) |
| R-1186 | P3 | the nightly off-site copy is skipped silently when another backup holds the flag | code | defect |
| R-1184 | P3 | a cut-off shares restore can be reported restored with files missing | code | defect |
| R-1196 | P3 | uploads confirmed just before a drive pull are 0 bytes or gone | LIVE (B6 #4) | defect |
| R-1183 | P3 | on 2026-10-25 every hub daily job runs twice (DB snapshot keep-2 pruned a day early) | replay of the hub's arithmetic | defect (D24 in STATUS) |
| R-1200 | P3 | an afternoon install gets no whole-box backup on night one | LIVE (B5) | design question |
| R-1202 | P3 | a box with its clock behind stops an app silently and only says „Hub nem elérhető" | LIVE (B6 #8) | defect |
| R-1187 … R-1190 | P3 | the unproven candidates of the four reviewers, grouped | code reading | mixed |
## Failure scenarios (B6)
| # | scenario | result | evidence |
|---|---|---|---|
| ★1 | power cut mid app install | PASS — clean rollback, honest event, reinstall in 15 s | `B6-failures/s1-…` |
| ★2 | power cut mid off-site restore | 6 s: safe; 14 s: app on the backup's data, page „interrupted" → R-1195 | `s2-…` |
| ★3 | data disk 100 % during backup | PASS — warned at 89 %, refused before writing, healed; SMB error unclear; system disk 97 % not run | `s3-…` |
| ★4 | data drive pulled, put back | apps stopped in 42/34 s, back by themselves; last uploads 0 B/missing → R-1196 | `s4-…` |
| ★5 | slow lossy line, first big off-site | run failed under 1 Mbit + 5 % loss, raw English; second press OK 3 m 38 s | `s5-…` |
| ★6 | off-site blocked | failed in 2.5 min, raw English; catch-up OK 2 m 37 s (shortened, not a night) | `s6-…` |
| ★7 | DST | PASS — the database copy ran once, at the second 02:30; W30 (no summer/winter mark in the repeated hour) | `s7-dst/` |
| ★8 | wrong clock at boot ±2 y | FINDINGS — −2 y became −6 months (the OS floor): „Hub nem elérhető" only, Paperless stopped silently (R-1202); +2 y: a whole-box archive dated 2028 (R-1189 a), then the ep0 tunnel dead (R-1204) | `s8-wrong-clock/` |
| ★9 | two actions at once | PASS — second action refused with a clear sentence | `s9-…` |
| ★10 | box off 3 days (+#15 restart in the catch-up) | PASS — ONE catch-up, a clear note to the household, the interrupted catch-up recovered | `s10-three-days-off/` |
| 11 | hard reset mid app update | NOT RUN — no update was offered to the box (nothing was released during the run) | — |
| 12 | DNS dead 1 h | PASS for data; operator told after ~37 min, household never | `s12-…` |
| 13 | second drive added later | not offered as the backup place → R-1198; safe removal OK; blank texts (W29) | `s13-…` |
| 14 | remove keeping data → reinstall „use kept data" | PASS — 47/47 documents, 47/47 originals | `s14-…` |
| 16 | English browser / English dashboard | follows the household's setting (by design); a few Hungarian strings → fixed W26 (unreleased) | `s16-…` |
## Restore paths (B8)
All seven customer paths PASS on the live box (`B8-restores/README.md`): A.1.1 Mealie unit 17 s; A.1.2 Nextcloud from
the second drive 3/3 sha256; A.1.3 Radicale verify copy 5 s; A.1.4 Nextcloud missing-only 3/3 sha256 (NFC/NFD: both
names come back on the share; Nextcloud itself folds NFD into NFC); A.1.5 BookStack full off-site 30 s; A.1.6 shares
2 files; A.1.7 `.fab` Mealie. Gaps: the messages do not say which point in time or how many files.
## Night chain and DST
**The first night (B5), no hands:** database copy 00:30Z PASS (3 m 27 s); second-drive copy 01:30Z PASS (8 apps + the
share, 58 s); off-site 02:15Z PASS (44 snapshots, 3 m 6 s, nothing deleted; the hub's own listing agrees: 44 files,
newest 02:18Z); whole-box backup NOT RUN — not due under the 24 h rule, because the install's own backup at 12:09Z
started the clock (R-1200, P3: the family's first day reaches a whole-box copy only on the second night). At 02:46Z
demo-hp's own kernel update rebooted the test host and shut the rehearsal VM down (harness, not a Tester4 defect; a BYO
host skips that step by design). Health 200 all night before that; memory steady. **The morning data snapshot is
identical to the evening one, byte for byte** (`B5-night/README.md`).
**DST (#7), live:** the box was started with its clock at 2026-10-25 02:20 CEST and left alone: no database copy at the first 02:30, one at the second (01:30:00Z, 1 m 55 s). The code replay agrees for the box; the hub runs every daily job twice on that day (R-1183, D24) and spring 2027 puts the 02:30 dump on the 03:30 copy (`B6-failures/dst-replay/`).
## Interventions (operator-only acts)
| # | when (UTC) | what | minutes the household waited |
|---|---|---|---|
| 1 | 09:57 → 12:00:57 | the owner passphrase handed over (by design, out of band) | 124 |
| 2 | asked 09:35, done 12:01–12:15 | off-site switched on for Tester4 | 0 (done before the household needed it) |
| 3 | asked 08:59, done ~09:11 (2026-10-21) | delete host Tester4-816e21 before the reinstall (B7) | ~12 (and 17 min of a dead end before it, R-1203) |
Not counted as interventions: the two plug pulls (14:11, 16:13) — a household can do that, and did.
## Versions
controller 0.336.0 (`7ca59e2`, floor), agent 0.161.0 (`4e1621c`), hub 0.162.0, golden 0.331.0, min_agent 0.158.1,
installer 1.35.0, ISO 1.29.0 (sha256 `dceacae5…4829`), catalog `c50adfc6` (65 templates). Nothing was released, deployed,
vouched, baked or published during the run. One text fix was committed, not released: controller `01c967e` (W26).
## Browser spike (A4)
Works: Chromium 155 + Playwright 1.52 in a throwaway LXC on demo-hp, phone and computer, HU and EN, in ~12 minutes.
121 screenshots. Limits: no real DNS/TLS for the household's address; not a human. `A4-browser/README.md`.
## Teardown
Machine: VM 9320, CT 9330 and the ISO removed from demo-hp by their own IDs; no drill firewall rules left; memory back
to the start. DooPlex: three watchers stopped by PID. Hub: **host Tester4-648b80 is left for the operator to delete**
(the customer Tester4 stays). ep0: one PBS snapshot `ns/Tester4/ct/9201/2026-10-20T12:17:26Z` and the WireGuard peer
10.77.0.6 remain; the Storage Box keeps the household's restic history — listed, nothing removed (`D-teardown/`).
## Secrets (names only)
All in `sec`, prefix `REHEARSAL_2026_10_20_`: `9202_PW`, `9202_RESETCODE`, `APP_BOOKSTACK`, `APP_RADICALE`,
`APP_VAULTADMIN`, `BIND_TOKEN`, `BOOKSTACK_TOKEN`, `DASH_PW`, `FAMILY_IRISZ`, `FAMILY_KATA`, `FAMILY_ODON`,
`INSTALL_ROOTPW`, `IRISZ_PW`, `KATA_PW`, `ODON_PW`, `OLD_BIND_TOKEN`, `PAIRING_CODE`, `PASSPHRASE`, `RECOVERY_CODE`,
`SETUP_CODE`, `SMB_PW`. Kept (the operator decides); none written anywhere else. Evidence scanned for their values:
0 hits.
## Register
Before 153 open rows; now 182. Opened by this run: 29 (R-1175 … R-1200, R-1202, R-1203, R-1204). Closed: R-1019 (a dated
check read on demo-felhom, read-only). One row (R-1201) came from a parallel session. Updated: R-1176 (proven live),
R-1189 (a half-live), R-895 (first verdict), R-435 (moved). Capability map: A15 still PARTIAL with new evidence; A12 and
A13 PROVEN-LIVE → PARTIAL (B7). `unproven.py --summary`: unchanged (35 of 55 not walked).
## Observations
FILED: everything in the tables above. NOT-A-FINDING: the dashboard ignores an English browser (by design: the
household's setting wins); Nextcloud folding NFD into NFC (upstream behaviour; both forms survive on the share); the
films' off-site size (CBR filler compresses; nothing missing); demo-hp's own kernel reboot at 02:46 (a harness effect: a
BYO host skips that step by design, `osupdate/leg.go:674`). FILED: everything in the tables above.
**Harness slips (mine), each recorded in the journal:** a screenshot step blanked a form before submit (15 min lost); the
two watcher logs froze at 20:24 when `git pull --autostash` replaced their files (recovered in full from the open files);
my first #10 start used a date the #7 records had made meaningless (restarted, recorded before any result). **Secrets:**
the nightly DooPlex sweep found three rehearsal secret values in my session record (the old and new bind tokens and a
setup code — I had read their mails in full and typed one into a `sec put`); in B7 a second bind link and setup code
entered the record the same way. All are one-time values, spent within minutes; nothing secret is in any committed file
(323 files scanned, 0 hits; pictures checked by eye; the B7 pairing code was blacked out).
+106
View File
@@ -0,0 +1,106 @@
# REPORT — the Monday release, run on Sunday 2026-10-25 (attended)
**Plain words first.**
1. **Fix A and fix B are proven.** A restore whose app does not start says so and keeps both copies (D49 = A); the old
copy goes only after 3 minutes of stable running. A `.fab` import that cannot download its version refuses and
touches nothing.
2. **Hub 0.170.0 is live.** Tester 1's recovery package went to the hub in one transaction; no alarm mail after.
3. **Agent 0.165.0 is vouched** (18:38:49 CET). It runs on all four hosts; on a BYO host our SSH file is removed — one
bundle run late (R-1285).
4. **Controller 0.343.0 is on every box:** 4 of 4 hosts (Tester 1, demo-hp, demo-felhom, the BYO test host) and the
scratch guest 9202. None stayed behind.
5. **The catalog is pushed:** n8n sends no usage data, OpenGist has 256 MiB, Homepage is behind the family gate.
**The D47 upgrade count:** Tester 1 3 of 3 apps got an install id, demo-felhom 5 of 5, demo-hp 11 of 12 (kimai is held
stopped since 2026-10-14 and the start-up step skips a held app — R-1286). **0 units set aside on any box.**
**The dated checks:** R-232 — DONE (last night's DooPlex database copy reached the restic sweep, snapshot `3973450f`,
folder `20261024-220001`, no alarm mail since 2026-10-23 22:20Z). R-1183 — partial today (one hub snapshot file per
slot, `07:00Z` and `13:00Z`; the hub restarted at 15:50Z for 0.170.0, so the morning log is gone); the full read stays
due 2026-10-26.
## What ran, in order
- **Part 1, fix A (R-1270 follow-up):** one helper, „started" = running on every sample of a 45 s window, no
`RestartCount` growth, every healthcheck `healthy`; the undo copy released after 3 more minutes (10 s samples, 30 min
watch; only a restore begun after the watch ends it). Found live and fixed on the way: docker's `-f` template failed on
every app without a healthcheck (now the JSON is read). Live on 9202 rc.6: an image that exits after 20 s → „még nem
indult el", copy kept, one operator line; the original unit back → released 3 min later.
- **Part 2, fix B (R-1279):** the D48 helper before step 1 of the import. Live: a bundle copy with a missing tag →
refused with the D48 sentence, the app running, its volumes untouched.
- **Part 3, hub 0.170.0 (R-1254):** asked → yes → manifest bump, hard refresh, sync of `Deployment/hub` alone,
Synced/Healthy, digest `sha256:6478498d…`. Tester 1 ran the escrow ceremony through the page's endpoints (the new code
stored by the operator as a CC secret); the hub logged „the new package is on the hub". No alarm mail after the
deploy (the mailbox as the second channel; a bind mail to Tester4 arrived, the control).
- **Part 4, agent 0.165.0 (R-1257, R-1250):** `release-agent.sh` from a clean clone; signed jobs to the three boxes
(76/76, the appliance drop-in in force by `sshd -T`). **No BYO host existed, so one was built** on demo-hp: VM 9371, a
plain Proxmox VE 9.2 from an answer-file ISO, Felhom installed with `--mode byo` as Tester4 (the operator deleted the
old host and stored the retrieval passphrase as a CC secret). Our 0.164.0 drop-in planted first; after 0.165.0 still
there; a second config bundle removed it, wrote nothing, `sshd -T` read the owner's own setting. Vouched by the
operator.
- **Part 5, controller 0.343.0:** rc.6/rc.7 on 9202; the set-aside alarm added and red-proved (Part 5.2); `0.343.0`
pushed after CI 2330; per-box floors → the three boxes at 17:17Z; box checks: 0 ERROR lines, all containers up,
https-only green; the D47 count above; on Tester 1 a unit restore and a set-aside restore — the item back, the copy
released after 3m0s each. Asked → yes → global floor 17:53:42Z (MinAgent 0.158.1, read back in the hub log; the BYO
host moved 18 s later); per-box floors cleared; a 30-minute watch: the hub sent 0 alarms (`global-floor-watch.txt`) and no alarm mail arrived (the mailbox as the second channel).
- **Part 6, catalog:** Homepage's family gate tried on the drill on 9202 (stranger 302/401, household 200); asked →
yes → `ebbf1f4` pushed; demo-hp read it 8 minutes later.
- **Part 7, D44/D45 = all picks:** confirm lines tagged, text-follows-code changes made (old words struck through), the
four code picks became R-1280–R-1283; R-1265 and R-1266 closed.
- **R-1250:** released, but its live proof cannot run before the agent's next due window (Tuesday 10:00–15:59 local; all
boxes measured on 2026-10-24) — a dated check on 2026-10-27 with the operator's yes for DooPlex's qBittorrent as peer.
## Versions, digests, CI by commit
- **Hub 0.170.0** `sha256:6478498d…`.
- **Agent 0.165.0** binary `e9d04a47ee85222ce6c472633ecdcebe1baa88d6836848f9a7da6435a9adb7d3`, bundle
`8dea1f34d77ba40aa44c5461c3728c2ca94bab46854a6bd8b86191f520dfe85a`; agent CI by commit: `c8d3c17` 2306 ✓, `4a1a849` 2325 ✓, `4697f8a` 2326 ✓, `6b6ba0c` 2334 ✗ (R-1290 — the registry, not the code).
- **Controller 0.343.0** `sha256:bb3da72a054c45751e3baf6e4bad43d1561a952717c5f426d65a5aeed0413a3b` from `e2a9761`.
CI: `7720439` 2319 ✓, `2149a13` 2321 ✓, `35232de` 2322 ✓, `1ee0353` 2324 ✓, `3a1f09d` 2327 ✓, `935e624` 2328 ✓,
`e2a9761` 2330 ✓, `fe98377` 2333 ✓.
- **Catalog** `ebbf1f4` 2332 ✓, `cd0cf48` (REPORT).
- **felhom.eu** `ade357a1` 2329 ✓, `36ef8dcd` 2331 ✓, this commit (CI by hash in the final message).
- **Installer 1.36.1** on main, NOT published (R-1284).
## Register
Before **219**, after **219**; opened **11** (R-1280–R-1290), closed **11** (R-1251, R-1252, R-1253, R-1269, R-1270,
R-1271, R-1279, R-1257, R-1272, R-1265, R-1266). R-1254: the hub half closed in its row, the ceremony half open. R-1250:
VERIFY with a dated check. `unproven.py --summary`: no number moved (35 of 55 not walked).
## Observations
- FILED: R-1284 — the installer's preflight printed every refusal code twice („401401"), so its 401/404/409 sentences
never showed (R-1167's 409 sentence included); fixed on main (1.36.1, red-proved), the publish is the operator's.
- FILED: R-1285 — BYO: our drop-in is removed one bundle run late (the old wrapper plans the run that installs the new one).
- FILED: R-1286 — demo-hp's kimai held since 2026-10-14, no row named it.
- FILED: R-1287 — a set-aside mails „1 of 1 apps failed to back up" (red); nothing failed.
- FILED: R-1288 — after a set-aside restore, the next backup sets the reinstall's own unit aside too (a second copy and
alarm).
- FILED: R-1289 — the hub's „fleet's newest controller" can be a dev build no box runs (`0.343.0-dev.r1251a`).
- FILED: R-1280–R-1283 — the four D44 code picks.
- FILED: R-1290 — the agent's CI is RED since ~18:00Z: four old agent packages (0.159.0–0.160.0) were removed from the
registry by something outside the repos, and the `published` gate still expects the newest 10 (`6b6ba0c`, job 2334).
Not the release's code (job 2326 on the release commit was green); the keep rule is the operator's.
- NOT-A-FINDING: the BYO host's first bundle read 71/76 (`pve:store-grant:felhom-pbs`) — the PBS storage was being added;
76/76 two seconds later and on every run after.
- NOT-A-FINDING: the `.fab` import status reads `"ok": true` beside its `error` — `ok` is the HTTP envelope, the job's
result is `error` and the steps.
- NOT-A-FINDING: 9202's alarms read „DROPPED … notifier disabled" — 9202 has no hub; the line is the positive observable
that the event was raised (one per case).
## Teardown (three layers)
- **Machine:** every throwaway app removed through the product (9202: Vikunja, Tandoor, Homepage; Tester 1: Tandoor —
no container, volume, `app.yaml` or set-aside left). 9202 back on the released 0.343.0, its drill mirror and my saved
`controller.yaml` copy removed, my `0.343.0-rc.3…7` images removed (older rc images of earlier sessions left as found).
Tester 1 keeps its new escrow package on purpose (the old recovery code no longer works; the new one is a CC secret).
- **Host:** the BYO VM uninstalled through the product's `--uninstall` (rc 0), then VM 9371 destroyed with its disk; its
ISO removed from demo-hp's `local`; demo-hp lists only VM 341 and its old ISO, as found. DooPlex: only the hub deploy;
the BYO ISO work folder deleted; the session's password, session and job files shredded.
- **Hub:** the three per-box floors cleared; the global floor 0.343.0 and the vouched manifest stay (the release). The
host record `Tester4-f967d4` stays until the operator deletes it (session only — in STATUS); customer Tester4 is kept.
**Secrets:** used only through `sec`; the evidence scanned for every stored value used (0 hits) and the BYO root
password (0 hits); one CSRF token blanked in a saved page snippet. **System poster:** no fact changed. **Instruction
files:** none edited.
+29
View File
@@ -0,0 +1,29 @@
# REPORT — the restore and backup holes, D43 = A, and the D44/D45 picks (2026-10-24 night, unattended)
**Plain words first.**
- A power cut during a restore of an app on a drive no longer brings old data back later (R-1251, the P1). Proven on
Tester 1 with a real power-off and a drive that came back late.
- One wrong line in an app's backup settings no longer deletes its second-drive copy (R-1252). Proven on 9202.
- An export no longer overwrites a running restore's record (R-1253, proven on 9202); the hub saves a recovery package
in one step (R-1254, tested, not deployed).
- BYO hosts keep their own SSH settings (your D43 = A, R-1257). Tested in the bundle's harness.
- The review's 32 notes have one line each with a pick: answer „all picks" or name the ones you disagree with.
- Nothing is released. Monday: controller v0.343.0, agent v0.165.0 + your vouch, hub v0.170.0.
**Evidence and tables:** `documentation/audits/restore-holes-2026-10-24/README.md`; the picks:
`documentation/audits/review-round-2-2026-10-24/D44-D45-picks.md`.
**Commits:** controller `bb2809d`, `062f9b4`, `518a135`, `c9b4b82`, `e1613ae`; felhom.eu `dcf4f84c` (hub) and this
commit (documents, register, STATUS, CONTEXT, evidence); felhom-agent `c8d3c17`, `4a1a849`. CI by hash: in the final
report.
**Register:** before 207, after 208; opened 1 (R-1268), closed 0. R-1251/R-1252/R-1253/R-1257 → VERIFY (fixed on main,
waits for the release); R-1254 → NARROWED (the hub half fixed); R-1265/R-1266 carry the picks; R-1267 extended.
**Documents edited:** `07` (the R-1251 rule after D22's paragraph; the R-1252 rule in §8.2a; a pointer in §6.5), `05`
(the escrow transaction), `03` and `11` §1 (D43 = A). **System poster:** no fact on it changed (none of the four).
**Instruction files:** none edited.
**Machines:** Tester 1 and 9202 put back as found except leftover empty anonymous volumes of the R-1267 class (named in
the audit). demo-hp: nothing provisioned; the catalog sweep's VM untouched. DooPlex, ep0, household boxes, the hub: not
changed. No release, deploy, vouch, publish or catalog push.
+59
View File
@@ -0,0 +1,59 @@
# REPORT — a backup is never changed by a new install, a restore downloads first, a restore tells the truth (2026-10-25, unattended, commit only)
**Plain words first.**
- **R-1269 is fixed and proven:** a reinstall no longer writes into the old backup. The old backup is set aside whole,
listed on the restore page, and restores with its own password (Tandoor and AdventureLog on 9202: the account back).
- **D48 download-first on every restore path that starts a definition (the unit restore and with it the second
drive's unit and whole restore, both kept-data loads and a set-aside unit; the off-site restore; the second drive's
file restore; the no-unit fallback).** An unpullable version is refused and the app is never stopped (proven).
- **R-1270:** a restore whose app does not start says so — never „it worked"; the old copy is kept (proven).
- **R-1271:** cause — the other box's off-site copy was dated in the future (its clock), and „newest wins" took it;
now a copy more than 10 minutes in the future never counts as the newest (proven on a forged copy).
- **One question for you:** D49 in STATUS — after a restore whose app does not start, keep the data and wait (CC's
pick, done) or pour the old data back by itself.
**Safest readings taken** (also at the head of `documentation/audits/restore-truth-2026-10-25/README.md`): an install
identity field, not the data stamps; the set-aside unit is a D25 kept copy (remove-with-backups deletes it, as it does
other kept copies); a „partial write" means one half into another install's unit; R-1270 keeps the data and the undo
copy (no new retention rule); D48 tells a deleted version from a busy registry by Docker's answer; 10 minutes of clock
skew; the `.fab` import left to its own row.
**Architecture read:** `07` §6.3, §6.5, §6.6, §8 (D20/D22, §8.2a D25); `03` (drive identity); `09` §3 decisions 14,
40, 53; rows R-1269–R-1271, R-1206, R-698, R-933, R-1267.
**Each proof, before (`e1613ae`, 9202 on 0.342.0) → after (9202 on 0.343.0-rc.2):**
- R-1269: the kept unit's `app.yaml` rewritten (`cbfcb538…` → `e19d2610…`), restore broken → set aside byte for byte,
restored, the account back, 0 password errors (Tandoor, AdventureLog).
- R-1270: `ok: true` over a `starting` app → `ok: false`, `not_started`, the honest sentence, the undo copy kept.
- D48: the first pull after the stop (code order) → an unpullable pin refused, the container's start time unchanged.
- R-1271: the future copy chosen (sweep) → the own copy chosen; alone, the future copy marked and said.
**Commits.** Controller: `02daebe` (D47) CI 2311 ✓, `b680db8` (R-1270) 2312 ✓, `2d71270` (D48) 2313 ✓, `bd9f4f7`
(R-1271) 2314 ✓, `1f9c863` (the kept list without off-site) 2315 ✓, `1f82959` (remove deletes the undo copy) 2316 ✓,
`e8d2151` (REPORT). felhom.eu: this commit (`07` §6.5/§6.6/§8.2d/§8.2e, `09` §3 decisions 256–257, CONTEXT, register,
STATUS, STATUS-HISTORY, the audit with its evidence and tools) — CI by hash in the final message. Test images
`0.343.0-rc.1` (`bd9f4f7`) and `0.343.0-rc.2` (`1f9c863`) in the registry; nothing released.
**Teardown.** Machine: 9202 only; my throwaway apps removed through the product; two empty folders and one undo copy of
mine removed by hand after checking; my rc images removed from its Docker; back on 0.342.0; the apps found running still
run. Host: the session's 9202 password and session files shredded. Hub: provisioned nothing. ep0: nothing.
**Register:** before 217, after 219; opened 2 (R-1278, R-1279), closed 0; R-1269, R-1270, R-1271 → VERIFY (fixed on
main, wait for the release); notes on R-1206 (cross-reference) and R-1243 (its dated check done, read-only); R-232's
dated check moved one day (this brief: no DooPlex work).
**Observations:**
- FILED: R-1278 (a volume app's reinstall does not offer its kept backup).
- FILED: R-1279 (the `.fab` import removes the app's volumes before its pull — D48 does not cover it).
- FIXED without a row (found live, red-proved): the kept-copies list was hidden on a box without off-site (`1f9c863`);
a kept undo copy outlived remove-with-backups (`1f82959`).
- NOT-A-FINDING: Immich's 404 after the sweep's load — the copy brought its own address (`photos.`), as every unit
restore does; the app was healthy there.
- NOT-A-FINDING: „immich: definition REFUSED by the allow-list" on 9202 — present before this run's image; an earlier
session's Immich, not this change.
- NOT-A-FINDING: the D48 refusal reads „Visszaállítás sikertelen: …nem lehet letölteni… Az alkalmazáshoz nem
nyúltunk" — the page's failure frame around an honest sentence; true as written.
**Secrets:** used only through `sec`; the evidence scanned for the box password and the session tokens (the CSRF token
in two saved page snippets was the positive control and is blanked); units compared by sha256 only. **System poster:**
no fact changed. **Instruction files:** none edited.
+33
View File
@@ -0,0 +1,33 @@
# REPORT — review round 2 (2026-10-24, unattended): the documents against the code, data loss and security
**Plain words first.**
- The 17 design documents were checked against the code: 493 claims, 121 out of date. About 130 places are corrected or
marked with a dated note; no rule, fence or promise was changed.
- About 30 things the code does that no document named are now written down for you to confirm (D44), and seven places
where a document's ruling or promise says something the code does not do are waiting for you (D45).
- The worst data-loss finding: a power cut during a restore of an app on a drive can later pour old data over the
household's new data (R-1251, P1). Proven with the real recovery code; it needs a fix in an attended run.
- Also proven, on the scratch box: one wrong line in an app's backup settings deletes that app's second-drive copy
(R-1252, P2).
- The worst security finding: the „root logs in by key only" SSH rule also reaches a host the household owns (BYO),
where we promised to change nothing (R-1257, D43). Nothing serious was found in the hub's or the controller's new routes.
**Evidence and the full tables:** `documentation/audits/review-round-2-2026-10-24/README.md`; every document edit with
its `file:line` in `EDITS.md`; the reviewers' candidates in `reviewers/`; the two proofs in `proofs/`.
**Commits (CI read by hash):** controller `b01bb4b` (two small fixes, red-proved; unreleased) — CI 2294 / #480 success;
controller `2449223` (REUSE.md) — 2298 / #481 success; felhom-agent `f5c3d77` — 2296 / #261 success and catalog
`227a544` — 2297 / #256 success (the rule-file alignment); felhom.eu `77015997` (documents batch 1) — 2295 / #1253
success; felhom.eu `876d9236` (batch 2, the register, STATUS, CONTEXT) — 2299 / #1254 success.
**Register:** before 190, after 207; opened 17 (R-1251–R-1267), closed 0.
**Instruction files edited (facts only, named here):** `.claude/rules/unprompted-work.md` in felhom-agent, the catalog
and the workspace root — the missing „A version number is pushed once" paragraph (before 95 lines, md5 3b071ad2; after
byte-identical to the other two, b1891bd4); `felhom.eu/.claude/rules/live-probes.md:41` — `/opt/felhom/stacks/<name>` →
`/opt/docker/stacks/<name>/app.yaml` (the real path, and the folder stays after a clean remove).
**System poster:** the facts file and the html changed together (text only, the hub-DB push times); no new drawing.
**Machines:** 9202 used for the R-1252 proof and put back (one difference: Immich's new empty volume, R-1267). DooPlex,
ep0, household boxes and the hub: not changed. No release, deploy, vouch or publish.
+17
View File
@@ -0,0 +1,17 @@
# REPORT — D39 = B: the security-fix lane (2026-10-23, attended)
**Plain words first.** An app with a public security hole can now get its fix at night on every box, after one test
on the test box. Controller 0.341.0 does it and is the floor since 13:24 (your yes). Older controllers skip such a fix
at night and take it when a person presses Update — both seen on the test box. Nothing needs you.
- Design: `09-update-architecture.md` §3 decision 255 (+ a dated note on decision 13). Capability map B23.
- Catalog `3898593` (+ a box-walk address fix after it); controller `63dcccf` = 0.341.0; drill commits `8c1d908` …
`17e0da0` (the drill only).
- Evidence and the full table: `documentation/audits/security-lane-2026-10-23/README.md`.
- Register 186 → 186: closed R-1239 (D39 = B); opened R-1240 (9202's 34 saved configs with the catalog token); R-1238
stays open, its text now says the next Plex fix can use the lane.
- `STATUS.md`: D39 out of the decide table, one „what works" line, the „Updated" paragraph replaced (the old one is in
`STATUS-HISTORY.md`). `CONTEXT.md`: the rulings. System poster: no fact changed (none of the three actions).
- Instruction / fact files edited: `app-catalog-felhom.eu/CLAUDE.md` (the lane, added); `operations/nodes.md` line 221
(9202 „same dashboard password" → its own password, measured) and line 46 (Tester 1 `.101` → DHCP, `.148` today).
- No `--no-verify` in this run.
+120
View File
@@ -0,0 +1,120 @@
# REPORT — the status page back to one screen, and a gate that keeps it there (2026-10-20, day)
## In plain words
The status page was 107 lines and 31,695 bytes; now it is 50 lines and about 5,500 characters outside the decide table,
which holds the 8 decisions that are still open. The „do" list went from about 25 items to 7 „still needed? (yes/no)"
questions, 2 morning-list items and 5 other lines (plus one „Later" line of row ids); CC checked the rest in the hub
and on DooPlex, and 7 were already done. **You answer:** the „Still needed? (yes/no)" list below, one word per line.
## Still needed? (yes/no) — copied from STATUS.md
1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). Since 2026-10-16.
2. The N100 sheet, about 30 minutes: Plex, the admins in Prowlarr, Sonarr and Radarr, your download sources
(`documentation/audits/n100-lived-in-2026-10-15/SHEET.md`). Since 2026-10-15.
3. Look at these pages on demo-hp when you have time: Calibre-Web („Adatok és mentés"), the file manager, AdventureLog,
Plex, the app page, the Plex and Sonarr guides („Útmutató"). Since 2026-10-16.
4. Remove the test server „plex-k6" from your Plex account (plex.tv → Settings → Authorized Devices). Since 2026-10-15.
5. Ask Google to index `https://felhom.eu/sajat-felho-tarhely` in Search Console (R-1035). Since 2026-10-14.
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable.
Since 2026-10-14.
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
## What CC checked, and how (method: the hub's GET pages with CC's scoped token, from the running pod; read-only local checks on DooPlex)
| „Do" item | Verdict | Evidence |
|---|---|---|
| Vouch agent 0.161.0 | **done** — the Day-0 manifest names 0.161.0 with the asked sha256 | `documentation/audits/status-view-2026-10-20/hub-reads.txt` |
| Delete the test customer Tester3 | **done** — not in the customer overview | same |
| Discard registrations 40 and 41 | **done** — the Hosts page has no „Unclaimed appliances" section (only `registered`/`bound` rows render) | same |
| Delete Tester4's old host `Tester4-31fa0e` | **done** — not on the Hosts page | same |
| Say „tear it down" for Tester 1's media tests | **done** — R-1022 closed 2026-10-17 | `CLOSED-ITEMS.md` |
| Hub master key into the password manager, then shred the file | **done** — `new.key` is gone; its folder is there (control) | `…/local-checks.txt` |
| Hub 0.153.0 „no token can …" | not a task — a statement; moved to history | — |
| Pull the failed disk Z6A07P2G | **not done** — still in DooPlex (lsblk; its twin is the control) | `…/local-checks.txt` |
| The Facebook `first-post/` leftovers | **not done** — 15 files still there | same |
| The `deny`-list edit in homelab-manifests | **not done** — still uncommitted | same |
| Old „broken" line: a Longhorn volume at 91 % | **gone** — Alertmanager lists only the two R-934 disk alarms | same |
| Old „broken" line: CI jobs get lost (R-887) | **gone** — R-887 closed 2026-10-19 (381 jobs, 0 lost) | `CLOSED-ITEMS.md` |
Every decided row (D1, D2, D10–D20) left the decide table only after its ruling was found in its closed row in
`CLOSED-ITEMS.md`. The whole old page is in `STATUS-HISTORY.md` under „Moved from STATUS on 2026-10-20" (headings
moved down two levels, otherwise unchanged).
## Register
- **Four numbers:** rows before **156**, after **153**, opened **1** (R-1174), closed **4**.
- **Closed:** R-1121 (the rotation file is gone; control `~/.bashrc`), R-1118 (its last part, R-1134, closed
2026-10-17; `sec doctor` all OK), R-231 (scripts in git and installed from it, the first run green; its same-disk
facts live on in R-232) — evidence in `CLOSED-ITEMS.md`, `audits/status-view-2026-10-20/local-checks.txt`,
`audits/answers-2026-10-12/s20-backup-scripts.txt`.
- **Merged:** R-993 into R-991 (one decision, D5; the proposal and the „no host floor at install" fact copied into R-991).
- **Narrowed:** R-132 (the hub password was rotated 2026-10-15; left: the break-glass hash field in the public repo);
R-1049 → WATCHING (the fix is live since hub 0.151.0; it no longer waits on the operator).
- **Owner cell fixed:** R-919 `CC` → `operator (upload + the look), then CC (close)` — its next action is the operator's.
- **Operator-owned rows (43), still waiting on the operator?** yes for all, except: R-933 — **CC first** (the size
measurement), then the operator's go; R-388 — **partly** (slice 1 is built; the scope decision is open);
R-920 — parked by the operator's own choice (c). On the page: decide D3–D9, D21; do (still needed, the rest);
the „Later" line for the remaining 23.
- **Rows over 4.5 KB (not rewritten, §N.7):** R-336 7.6 KB, R-516 7.0, R-462 6.8, R-433 6.7, R-264 5.8, R-919 5.0,
R-440 4.6, R-925 9.0.
## The gate — `scripts/status_view_gate.py` (R-1174)
- **Checks:** size outside the decide table (≤ 50 lines, ≤ 6,000 characters); one „Updated" paragraph (≤ 3 sentences,
no „Earlier" chain); no Hungarian letter outside a quote; every cited R-n open; no decided/closed/done row in the
decide table; the P1 row under „What is broken" or „decide"; every open row whose Owner names the operator under
„decide" or „do". Exit 0 clean, 1 convicted, 2 inconclusive (no sections, missing section, no or empty register).
- **Switch:** warn-first until **2026-10-27** (`SWITCH_DATE`), then a refusal; the date is also a due check for
R-1174, so the next push on or after that date names it.
- **Tests** (`scripts/test_status_view_gate.py`, 39 cases, all PASS): the genuine fixture before and after the switch;
the clean run's „what it looked at" line; per check, decoys that warn before and refuse from the switch — size (50
more lines; few lines and too many characters), updated (a prepended paragraph, an Earlier chain, four sentences, no
paragraph), english (a Hungarian do-line), open-ids (a closed row cited), decide (a DECIDED row; a „done" pick), p1
(the P1 line deleted; moved to What works), operator (the Later line deleted; a row only in „broken"); genuine
controls that must pass (a long decide table, version numbers in a sentence, Hungarian inside a quote); four
inconclusive cases; and the real `STATUS.md` against the real register.
- **Red-proof:** each check removed in turn → exactly its own decoys FAIL (size 4, updated 8, english 2, open-ids 2,
decide 4, p1 4, operator 4), then put back → 39/39. The switch moved to 2099 → the 14 „refuses" cases FAIL; moved
one day earlier → the 14 „warns on 10-26" cases FAIL. Put back → 39/39.
- **Run against the old page** (as it was at `2954b0be`): 75 convictions across all seven checks (81 lines, 25,472
characters outside decide; 13 decided rows; 24 operator rows missing; the P1 only in „do").
- Wired: `repo_gates.py` (fast set, docstring entry 19), `test_gate_decoys.py` (COVERS + runs the suite), found by
`script_tests_gate.py`; `decoy_coverage_gate.py` reports it COVERED.
## The CLAUDE.md edit (repo `felhom.eu/CLAUDE.md`, end-of-session checklist, line 139–141)
- **Before:** „… One screen, written for the operator in plain language, and deliberately **not** `CONTEXT.md`."
- **After:** the same, plus „(STE English) … How to update it:" and five numbered lines — replace the „Updated"
paragraph (never prepend; the old one to `STATUS-HISTORY.md`); a morning list under a dated sub-heading in „do";
remove what you finished; the decide-row rule; the gate checks it (R-1174).
- **Why allowed:** a format rule, not a loosened safety rule (`PROMPT-TEMPLATE.md` §9 rule 9, decision 150).
## Commits and the running session
`2954b0be` register pass · `9ec5f76a` the page · `1d7071e8` the gate · `54d1ee82` the instruction · this report.
**The rehearsal session shares this clone.** Its commit `f84f94d9` (audit evidence only) landed between my parts 1
and 2; it touched neither `STATUS.md` nor the register, so there was nothing to merge. **Its `git push` carried my
parts 2–4 to `origin` with its own** (the shared branch); its hook ran with `--scope=docs`. My own `--fast` run of the
same content was green (rc 0), and `scripts/push_scope.py` classifies the full range correctly as CODE. Its decoy
suite briefly planted changes in `website/gyik.html` and `scripts/felhom-host-install.sh` while I worked (R-928's
shape); I staged only my own paths, and both files were clean again afterwards. `unproven.py --summary`: unchanged
(NOT WALKED 35 of 55).
**Gate runs.** Every push: `repo_gates.py --fast`, rc 0 (the last one on the final tree, this report included). **Not
followed: „the full run before the last push".** The full run adds exactly one gate, `iso-bootstrap`, which starts a
container from an ISO image; on DooPlex that is a Docker run on production (workspace rule, and a standing feedback
memory), and it checks only `scripts/iso/`, which this session did not touch.
## Observations
- „My commits were pushed by the other session's push, under a docs-scope hook" — **NOT-A-FINDING:** CI re-runs the
whole fast set on every push whatever the hook's scope, and CI was green for every push of this session (by `head_sha` on the `jobs` endpoint, attempt 1 each): `2954b0be` job 2131, `54d1ee82` job 2132 (it carried parts 2–4), `c701b253` job 2133 — all `success`; the
shared-clone behaviour is already recorded (R-928, the shared-worktree rule).
- Tester4 shows DOWN with no host on the hub — **NOT-A-FINDING:** the rehearsal session reinstalls Tester4 today.
- „A Longhorn volume at 91 %" had no row — **NOT-A-FINDING:** the alarm is no longer active (Alertmanager, with the two
R-934 alarms as the control).
## Teardown
Provisioned nothing: no machine, no host, no hub write (GET only, CC's scoped token).
+129
View File
@@ -0,0 +1,129 @@
# REPORT — weekend night 2026-10-23: the hub's last plain tokens locked; the upload and CPU watch (spike, design, build)
**Safest readings taken (unattended), stated first:**
- **A test image is not a release.** The upload watch was proven on scratch 9202 with `felhom-controller:0.342.0-rc.1`,
set by hand in 9202's `/etc/felhom-controller-image` (the shape of today's security-lane rc.1). An `-rc` tag is never
offered to a household's update button. No floor was touched; only 9202 ran it, and 9202 is back on 0.341.0.
- **9202's line was seeded by hand** (one 40 Mbit/s record): its off-site backup is off by design, so it can never learn a
line itself. The learning path is pinned by unit tests and by tonight's one-second read of the real pushes on the
three test boxes (read only).
- **The hub 0.168.0 image was built** (the brief asked for it) **and not deployed.** The four new hub event types are in
hub 0.169.0, committed only.
- **Per-app trigger instead of a whole-box trigger.** The brief said „the box's upload above X % of the line". Inside its
container the controller cannot tell LAN from internet bytes, nor see the box's total; it can see each app's exact
bytes, and the per-app view is what the explanation compares. So the trigger is per app (§6.2 of the spike).
- **The first day:** apps the box can ask are judged at once on the 5 Mbit/s floor (a mismatch is evidence whatever the
line); apps it cannot ask are judged only after ~20 h of history. The brief's „first day: no alarm, only learning" holds
for those apps; for the others a mismatch on day one is still an alarm.
## Five lines
1. **Tokens locked:** the Cloudflare tunnel token, the Cloudflare API token and the git token are sealed in **hub 0.168.0**
— built, CI green, **not deployed** (you deploy it on Monday).
2. **Backup copies:** two options — encrypt the hub's Longhorn disk, or take it out of the Longhorn backup because the
hub's own copy to ep0 is already encrypted at rest and in transit (measured). **My pick: take it out** (D40).
3. **The line:** measured from the box's own nightly off-site upload (one-second samples, best 10 seconds) — a lower
bound; most nights the upload is too small to say anything, so a real test is your decision (R-1247).
4. **Apps that explain their traffic:** qBittorrent and Transmission (their own byte counters, within 1–7 %), Plex (its
viewers and transcodes). Jellyfin needs a key the household makes; Nextcloud and Immich give no number.
5. **The fake hacked app was caught: yes** — a flood from inside qBittorrent, hidden behind a real seed: one alarm after
15 minutes, one clear 15 minutes after it stopped. A CPU burner in Plex: the same. A real seed: no alarm.
## The spike's numbers
`documentation/audits/SPIKE-upload-watch-2026-10-23.md` (evidence beside it). Short form:
- **Normal levels** (three boxes, 19–31 days of one-minute samples): no app's own upload p95 reached 6 Mbit/s; the
longest run at ≥ 5 Mbit/s was **3 minutes**, at ≥ 1 Mbit/s 9 minutes. CPU: the longest run at ≥ 150 % of a core was
**7 minutes**, at ≥ 80 % 13 minutes. DooPlex's hacked Plex: ~44 Mbit/s for hours, ~10 cores.
- **Line speed:** at a one-minute sample the push exceeded 2 Mbit/s on 3 of 81 box-nights (the push moves a few MB);
the box keeps no byte count of it. Tonight's one-second read: §Night data below. No active test was run (ep0 changes
forbidden; a public test's terms not read).
- **Apps:** qBittorrent counted 131.1 Mbit/s while its container sent 134.6–140.5 (×1.03–1.07); Transmission ×1.001–1.02;
the limit API works and persists (KiB/s rounding); Plex answers its session lists (with the token it already has when
claimed); Jellyfin 401 without a household-made key.
- **A hard cap** (`tc` from a helper in the app's network namespace) works in an unprivileged guest: a flood held at
20 Mbit/s, the app kept answering. Not built (R-1248).
## The household's line (final text)
- HU: „A qBittorrent feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod."
- EN: "qBittorrent uploads at most N Mbit/s, so the family internet stays fast. You can change this in qBittorrent."
- The system settings page: „Feltöltési sebesség: legalább N Mbit/s (az éjszakai távoli mentésből mérve)" /
"Upload speed: at least N Mbit/s (measured from the nightly off-site backup)"; „még mérjük …" until known.
## Morning list (Monday)
1. Deploy hub 0.168.0 (the last plain tokens locked). Then read its start line `config tokens sealed at rest (8 …)`.
2. D40: the hub's plain Longhorn copies — stop that backup (my pick) or encrypt the disk; and whether to delete the two
plain copies already there.
3. D41: qBittorrent's default limit — 50 % of the measured line (my pick). D42: who is told — the family only about the
limit, the hacked-app alarms only you (my pick).
4. Release the upload watch when you say yes (hub 0.169.0 build + deploy, then controller 0.342.0 through the normal path).
5. For the homelab session (list only, no HM numbers): an upload-saturation alert on DooPlex's `eno1` (the line is
~100 Mbit/s; the flood peaked at 905 Mbit/s).
## Commits, images, CI (by commit)
| Repo | Commit | What | CI |
|---|---|---|---|
| felhom.eu | `5277d859` | hub 0.168.0: the tokens sealed; R-1243; STATUS D40 | run 2269 **success** |
| felhom.eu | `af5c005a` | hub 0.169.0: the four events (allowed, operator-only, mail line) | run 2271 **success** |
| felhom.eu | `a91d0b24` | spike, design (`08` §9), live proofs, rows R-1244..R-1248, STATUS | run 2272 **success** |
| felhom-controller | `a9c1997f` | v0.342.0 (unreleased): the upload and CPU watch | run 2270 **success** |
| felhom-controller | `60e487d` | CONTEXT + REPORT | run 2273 **success** |
Images: `felhom-hub:0.168.0` `sha256:3bfdecaae527d4762059600cdfaad83daf4e4b5b20661fc47acf128180d87cb3` (from `5277d859`,
not deployed); `felhom-controller:0.342.0-rc.1` (from `a9c1997f`, scratch 9202 only, removed from 9202 after).
**Teardown, three layers:** machine — 9202: the four throwaway apps removed through the product (containers and volumes
gone), my test files, probes and the `alpine:3.20` helper image removed, back on 0.341.0, the seeded record removed;
host — demo-hp: the receiver LXC 9471 destroyed with its disk; DooPlex: my private copy of the hub snapshot shredded,
the 9202 password copy shredded; hub — none. ep0: read only (the hub-copy snapshot list). No household box touched.
## Rows
**Before 188, after 194; opened 6 (R-1243 … R-1248), closed 0.** R-975 stays open (WAITING-ON-OPERATOR: deploy 0.168.0).
## Observations
- The hub's Cloudflare/tunnel/git tokens unsealed in `config_json` — FIXED in hub 0.168.0 (R-975, not yet deployed).
- The hub's Longhorn copies are plain and sit in the open folder — FILED: R-1243.
- The upload watch is not released — FILED: R-1244; its limit size — FILED: R-1245; who is told — FILED: R-1246.
- The line is only a lower bound — FILED: R-1247. An app's API can lie — FILED: R-1248.
- `appliance_registrations.pairing_code` is plain — NOT-A-FINDING: it is shown on the box's screen and is one of two
factors; the other is hashed.
- R-879's sealing (0.138.0) ran no VACUUM, so old plaintext of those columns could sit in freed blocks of the live
`hub.db` — NOT-A-FINDING: 0.168.0's one-time VACUUM rewrites the file on its first start.
- The hub CHANGELOG's 0.167.0 header still said „NOT built" though it is deployed — fixed without a row (small).
- The shared memory index went over its size ceiling twice in one day (another session's lines) and blocked the push
gates — fixed without a row (shortened, no link lost); it sits 14 bytes under the ceiling now.
- The watch's very first minute after a start cannot reach apps by name (network joined seconds later) — NOT-A-FINDING:
that minute only sets baselines.
- The media wiring's generated override file stays after an app's removal — NOT-A-FINDING: rewritten before every start.
- `docker stats` text in the stored metrics has three significant digits — NOT-A-FINDING for the watch (it reads the
Engine API); the charts are display only.
- `felhom-samba` runs on the host network and has no counter — NOT-A-FINDING: infrastructure, not judged.
- Transmission's rate under its limit was not observed (no peer contact) — NOT-A-FINDING: the limit API was read back;
qBittorrent's was observed.
- A dated check came due on the push (R-1214, Tester 1's night copy): read read-only — the guest's memory „full" stall
was 444 µs in total over 11 h, 0 OOM kills; the copy was small (0.24 GB) and the capped container is not shown — the
row stays WATCHING, the due line removed.
- The night's line floor is low (10 Mbit/s on demo-hp, one second at 70) — FILED in R-1245's text (D41), no new row.
## Instruction files, poster
No instruction file edited. System poster: no fact on it changed (nothing deployed) — none of the three applied.
## Night data
Read 02:08–02:30Z, once a second, on the three test boxes (read only; spike §2.1):
- **demo-hp: counts.** Best 10 seconds 12.6 MB = **10.0 Mbit/s**. One second reached 70 Mbit/s.
- **demo-felhom: too small** (best 10 seconds 3.2 MB = 2.5 Mbit/s).
- **Tester 1: too small** (3.5 MB = 2.7 Mbit/s).
**What it means:** the nightly push is a burst of a few seconds. So the line it gives is a low floor. On demo-hp,
50 % of that floor is about 5 Mbit/s for qBittorrent — probably much less than the real line. The floor rises only when
a bigger night comes (the box keeps the best of 14 nights). Added to the D41 decision row (R-1245): if you want a
looser limit before a real line test exists (R-1247), pick a higher share or „none".
+1 -1
View File
@@ -96,7 +96,7 @@
| `(*Server).configFormData` (v0.49.0) | hub/internal/web/configs.go (~L430) | `(r, isNew, cfg, overrides, errMsg) configFormView` | The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | `overrides=nil` → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). |
| `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" <configFormView>}}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `</form>` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `<script>`. |
| `handleHostEscrowGet` + `escrowSelfServiceRetrieval` | hub/internal/api/handler.go | `GET /hosts/{id}/escrow` | The box-authenticated MIRROR of the escrow PUT — self-scoped by the per-host key. **`escrowSelfServiceRetrieval` is THE single decision point** for whether a box may read its own blob without operator-armed recovery mode (§8.2 vs §8.3, R-199): flip it, do not scatter the condition. **Never merge this with `dr.go`'s `handleReEnroll`/`handleGetRestoreDirective`** — those rotate the host API key and serve the K-escrow and directive too, and keep their recovery-mode gate (pinned by `TestEscrowGet_OperatorDRPathUnchanged`). Every successful retrieval MUST raise `escrow_blob_served` before the bytes leave; that audit row is the mitigation the trade rests on. |
| `demoteCurrentEscrowTx` (+ `(*Store).SaveHostEscrow`) | hub/internal/store/store.go (~L2597/~L2612) | `(tx, hostID) (int64, error)` / `(hostID, blob, fp, posture, createdAt, resticPwSHA) (superseded bool, prevResticPwSHA string, err error)` | **THE ONE escrow row-copy routine** — used by the re-escrow retention AND by `DeleteHost`'s custody demotion; never write a second one | **It must copy BOTH sealed artifacts** — `blob` (K-escrow / PBS key) and `identity_blob` (the age bundle carrying the offsite restic repo password). Omitting the second is R-198: two months of retaining the wrong key, with the ceremony as the destroying act. **Ordering it depends on:** `SaveHostDRBundle` writes `identity_blob` AFTER `SaveHostEscrow` returns, so the demote sees the PREVIOUS generation — invert that and the retained bytes are the new blob under the old hash. Pinned by `TestSaveHostEscrow_RetainsIdentityBlob` + `TestDeleteHost_DemotesIdentityBlob` (both callers). `prevResticPwSHA` feeds R-197's changed-key signal; it is a hash and never leaves the store. |
| `demoteCurrentEscrowTx` (+ `(*Store).SaveHostEscrow` / `SaveHostEscrowWithIdentity`) | hub/internal/store/store.go (~L2597/~L2612) | `(tx, hostID) (int64, error)` / `(hostID, blob, fp, posture, createdAt, resticPwSHA) (superseded bool, prevResticPwSHA string, err error)` | **THE ONE escrow row-copy routine** — used by the re-escrow retention AND by `DeleteHost`'s custody demotion; never write a second one | **It must copy BOTH sealed artifacts** — `blob` (K-escrow / PBS key) and `identity_blob` (the age bundle carrying the offsite restic repo password). Omitting the second is R-198: two months of retaining the wrong key, with the ceremony as the destroying act. **Ordering it depends on:** since R-1254 (unreleased, next hub v0.170.0) `SaveHostEscrowWithIdentity` writes the key, its hash and `identity_blob` in ONE transaction, with the demote FIRST inside it, so the demote sees the PREVIOUS generation — invert that and the retained bytes are the new blob under the old hash. `SaveHostEscrow` is that function with no identity blob; `SaveHostDRBundle` (a second statement) is no longer called by the escrow PUT. Pinned by `TestR1254_*`. Pinned by `TestSaveHostEscrow_RetainsIdentityBlob` + `TestDeleteHost_DemotesIdentityBlob` (both callers). `prevResticPwSHA` feeds R-197's changed-key signal; it is a hash and never leaves the store. |
| `(*Store).CountHostArtifacts` / `DeleteHost` | hub/internal/store/store.go (~L1640/~L1690) | `(hostID) (HostArtifacts, error)` / `(hostID, deleteEscrow bool) error` | Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (`ErrHostEscrowPresent`, tx never starts). log_bundles die by `scope_id == host_id` ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
| `(*Server).commitCustomerReset` (v0.69.0) | hub/internal/web/customer_reset.go (~L165) | `(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError` | THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the `customer_resets` journal | Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). `purgeEscrow` governs ONLY whether `PurgeCustomerResetDBState` destroys retained custody: standalone RESET passes the operator's `escrow_ack`; the DELETE cascade passes **false** so custody dies exactly once, in its leg 3. Returns a `resetLegError` carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. |
| `(*Store).CustomerResidue` / `PurgeCustomerResidue` (v0.70.0) | hub/internal/store/customer_delete.go | `(customerID) (*CustomerResidue, error)` / `(customerID) error` | Counting + purging the report-derived state and the credential-bearing bindings a deleted customer leaves behind | **`GetCustomers()` is REPORT-derived** — until the reports are gone the customer stays on the Customers list AND stays in the staleness/offsite checkers’ work list, so a deleted customer keeps emailing the operator. Both funcs walk ONE shared `residueQueries` list so a table can never be counted-but-not-purged. Includes `appliance_registrations` + `selfbind_tokens` (credential-bearing, not telemetry). NEVER touches `events`, `notification_log`, `host_deletions`, `customer_resets`. |
+46 -84
View File
@@ -1,99 +1,61 @@
# STATUS — what works, what's broken, what's next
**Ready for the first real tester (Tester-2): yes. Tester 2 (a laptop) is off; nothing was sent to it.**
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
**Updated 2026-10-19 (night).** An off-site backup no longer stops the household's apps for nothing when the agent is busy (controller 0.335.0 is the floor since 20:39 local; it works once the next agent is released — morning list). Three agent fixes and three hub fixes are written and tested, NOT released or deployed; the installer's „delete the old host first" message is written, NOT published (morning list). A security scanner now runs on every push in all four repos (warn-first: it lists, it never refuses); the first full scan found no real secret and no called weakness. The ArgoCD danger to three live hub secrets is gone (ArgoCD no longer tracks them). A failed off-site restore was measured on the scratch box — **decide D20**. ep0's first weekly clean-up ran at 05:30 local: it removed only tester-1's 4 October copy, as predicted, and the hub sees a rule for every household. — `REPORT-night-2026-10-19.md`. **Earlier:** **Updated 2026-10-19 (late afternoon).** The household passphrase no longer gives a live box's keys (your D19): each box now fetches its config with its own key (controller 0.334.0 is the floor since 16:31 local; hub 0.161.0 is live, your yes). A reinstall over a live box needs you to delete the old host first; the installer does not word that refusal yet (row filed). Every new household on ep0 gets its weekly clean-up rule automatically, and the hub alarms on one without; tester-1 and Tester-2 got theirs (your yes; tester-1's 4 October copy goes tonight). demo-hp kept its own address for a 30-minute watch after your router change. — `REPORT-day3-2026-10-19.md`. **Earlier:** **Updated 2026-10-19 (afternoon).** ep0 now keeps one whole-box copy per week for 2 weeks (your yes; the change removed nothing today, the first run is tomorrow 03:30 UTC). Controller 0.333.0 is the floor for every box (12:16 local): signing out on the dashboard now also ends the file manager's own 2-hour pass, and the recovery-code page says the code becomes live after the two words. Hub 0.160.0 is live (your yes): a reinstall within 45 minutes also mails you; a refused backup that repeats is counted, and the alarm repeats at most once a day. The household passphrase was measured: alone it reaches the box's keys and through them every off-site key action — **decide D19**. demo-hp loses its address to another device every few minutes — **fix it in the router** (do-list). — `REPORT-day2-2026-10-19.md`. **Earlier:** **Updated 2026-10-19 (day).** One sign-in for the file manager is live on every box (your D16 = yes; controller 0.332.0 is the floor since 05:24 local): „Fájlok ↗" opens the app's folder with no second login; signing out on the dashboard asks again; no app on a box can pretend to be the household (tested from inside an app). Agent 0.160.0 runs on demo-hp, Tester 1 and demo-felhom (your yes): a new recovery code replaces the old one only after the household types two words back; a broken-into box cannot flood the whole-box backups, and the local copies are one per day for 3 days (your D17); an empty partition no longer asks „data will be erased". **Vouch agent 0.160.0** (morning list). Hub 0.159.0 is live (your yes): a refused backup has its own alarm; a reinstall over an old host now mails you instead of staying silent — a first draft that re-issued the codes by itself was stopped by a security review. The old backup program still reads demo-hp's off-site copies after the first real clean-up with the new one. The document-preview error was a broken test file, not the box — `REPORT-day-2026-10-19.md`. **Earlier:** **Updated 2026-10-18 (evening).** A brand-new box was installed from nothing on demo-hp (your test customer Tester4): the emergency password was saved in the hub first, then set, and the web login took it; the box came up on the approved golden and moved to the floor by itself in under 3 minutes; its agent was the approved one. OnlyOffice now opens AND saves documents from FileBrowser and from Nextcloud (before tonight: FileBrowser could not open one, and Nextcloud lost every save while saying „saved") — controller 0.331.0 is the floor for every box (22:48 local). Immich's and Nextcloud's Redis no longer have a disk volume. Three agent fixes are written and tested, NOT released (your yes). Golden 0.331.0 is baked, NOT vouched — `REPORT-evening-2026-10-18.md`. **Earlier:** **Updated 2026-10-18 (day).** Hub 0.158.0 is live (your yes): its own 90-day access log works and writes no secret (tested with a fake token); you can vouch an agent without a new golden; the off-site snapshot check gives its first verdict on 20 October. Controller 0.330.0 is the floor (13:10 local): if the host refuses „retire anyway", the drive's apps start again. Agent 0.159.2 runs on the three test hosts (your yes; no pool check on a good backup) — **vouch it**. Installer 1.34.0 is published (your yes): the emergency password is saved in the hub first, then set. The calculator shows Immich'Line truncated
`documentation/backlog/STATUS-HISTORY.md`. Reports: `REPORT-night-2026-10-14.md`, `REPORT-immich-rollout-2026-10-14.md`, `REPORT-seo-felho-tarhely-2026-10-14.md`, `REPORT-immich-repair-2026-10-14.md`, `REPORT-power-cut-fixes-2026-10-14.md`, `REPORT-night-power-2026-10-13.md`, `REPORT-poweroff.md`.
**Updated 2026-10-25 (evening, attended release).** Every box runs the new controller: a restore that does not start says so and keeps both copies (D49), a reinstall never touches the old backup, a restore downloads first. The hub stores the recovery package in one step; the new agent is vouched; the catalog has three fixes.
## What works
- **A full disk can no longer leave an app's Redis broken** (catalog, 2026-10-17): RomM, Docmost, paperless-ngx, Immich and Nextcloud save their small Redis store as snapshots. Proven first on the scratch box: 100 of 100 test keys kept through the switch and through a backup and restore. A box takes it at the app's next start (normally that night's backup). Tester 1 waits: it still follows the drill catalog (R-1022).
- **Two more password doors count a guess before checking it** (controller 0.325.0): the share-link password and the off-site wizard's password. Family sign-ins are stored as fingerprints; signed-in family members stay signed in.
- **A restore brings the database password back with the data** (controller 0.324.0, your decision 1A): remove an app, install it again, restore it — the app opens its own data (proven on demo-hp with Rallly and Grimmory, off-site and on-box, and through „use my kept data"). A backup without the password is refused before anything changes, with a plain sentence. Backup program restic 0.19.1 (your 2A): old and new copies read both ways; scanner findings in it 50 → 19.
- **The dashboard is harder to attack** (0.324.0): no sending the browser to another site after sign-in, no framing, parallel password guesses counted, 12-character minimum, reset-code requests limited, sign-out needs a token (the old sign-out link did nothing), a password change ends every family and gate session.
- **The box's backup program is now chosen, not inherited** (controller 0.323.0, 2026-10-16): restic 0.18.1, checked by its signature and fingerprint at every build, on Debian 13. Every way the box reads restic was compared old vs new with real outputs; old copies stay readable by the new program and new copies by the old one (on demo-hp's real store). The file-sharing image has 0 known high holes (was 48), and the same household password still works after the change.
- **The hub's key can no longer open anything on a box** (controller 0.322.0, 2026-10-16): the settings route is gone, and every former door answers 401 even with the box's own key. Every box that reports answers only over https (35 names). The controller and the hub (0.156.0, deployed at your yes) are built on a supported Go (1.27.2): known Go weaknesses in each 62 → 1; the hub has 0 high findings.
- **RomM game saves are in the backup now** (2026-10-16, catalog): a save survived a backup, a removal and a restore on demo-hp, byte for byte. Eight apps no longer claim „a known, shared password" (your D10 = A). If someone changes the file manager's password in the file manager, the box notices and offers a new one (controller 0.321.0).
- **Every installed app says where its login is, and has a „Fájlok ↗" button to its folder** (controller 0.320.0, 2026-10-16). The file manager is shown as part of the box again, with its password; on a box that does not know that password, one press sets a new one (done on demo-hp and, at your yes, on the N100).
- **Every app but three has a user guide and an FAQ, in Hungarian and English** (catalog, 2026-10-15 night): the „Útmutató” tab shows it on boxes with controller 0.315.0 or newer. Every button name in a guide was checked in the app's own files at the version we ship; a guide that could not be checked stays hidden (plant-it, home-assistant), and Emby has none (R-1092).
- **The N100 is the lived-in box** (since 2026-10-15): used like a household's box, changed only through the product. qBittorrent, Prowlarr, Sonarr and Radarr are installed and linked by the box; films never go off-site (checked before any download). Its diary: `documentation/lived-in/n100.md`.
- A restored app comes back exactly as safe as it was at the backup: closed to strangers until its first account is made (0.314.0, proven on the scratch box). A box on a test build now moves with the floor (Tester 1 did, by itself). The box claims Plex again (demo-hp's Plex claimed through the dashboard, 2026-10-15). A brand-new box from the public installer updated itself to the floor 3 minutes after it was linked (test box, since removed).
- Scripts use the hub with a scoped token, not your password (hub 0.153.0): CC sets the floor with its own token; no token can see a secret, delete anything, or send, make or show a household's password or link. Your new hub password is yours alone (changed 2026-10-15).
- Every public name of the three test boxes answers only over https: plain http gets a 301 to the same https address, and browsers are told to remember it for one day (29 names checked, 2026-10-14 night).
- Media apps work together on one drive: a download reaches Sonarr/Radarr as one file, no copy; the download apps see only their own folder (live catalog, proven on demo-hp).
- A power cut in the middle of a backup no longer blocks later backups: the box removes the half-written backup at start-up (proven with a real cut on demo-hp, on all three boxes now).
- After a power cut, Immich repairs the photos the cut left broken, by itself, and the app page lists any lost photo (proven with a real cut on demo-hp: 41 of 41 repaired; on all three boxes now). An Immich update cannot switch it off quietly: the catalog refuses an untested Immich version.
- A household can switch the machine off; the hub holds alarms for 14 days and clears the state by itself (N100).
- A real power cut on demo-hp: it started by itself after all 8 cuts of the night (with the BIOS set), every app back in ~4 min; stale and recovered mails true and paired.
- A page for people searching „felhő tárhely" is live (/sajat-felho-tarhely, /en/own-cloud-storage): what cloud storage is, what to do when Google or iCloud fills up, a five-year cost table with prices read 2026-10-14, and who should NOT buy this.
- The machine estimator is public (/gepigeny, /en/sizing), with figures from two machines (N100 and HP t740; raised only, 2026-10-14).
- 65 apps in the catalog. New today: Prowlarr — you enter your download sources once, and Sonarr and Radarr get them, linked by the box with no button (the website lists 63 apps — R-1036). OS updates, the kernel lane too, run at night.
- The hub is closed again after the 2026-10-12 security review; one box cannot read another's data.
- Felhom runs in a virtual machine on OpenMediaVault 8, tried on an exact copy of the first outside tester's box: install, bind, claim, apps, backup, and back by itself after an OMV restart. The Hungarian guide for him is `documentation/pilot/GUIDE-omv-kvm-hu.md`; open: one-disk Jellyfin (R-1068), NAS films for Jellyfin (R-1071, yours), no disk health in a VM (R-1073, yours). Waits on you: send the guide.
- A second local copy counts only when it is on a different physical disk, and the box can prove it. Inside a virtual machine the box makes the copy but says „A Felhom nem tudja ellenőrizni…" (Tester 1). On real disks nothing changes (demo-hp, N100). A NAS never hides the „only one copy" notice. A box with no drive refuses a media app in words, with a link to the Meghajtók page. Live on the three test boxes; the NAS and no-drive parts are unit-tested only. Next: build B (the internal place on one-disk boxes, R-1079). Your rulings D1–D6 are in the design (`architecture/16-storage-physical-identity.md`).
- **One-disk boxes get their own data area** (2026-10-16): a new install makes a „Belső tárhely" on the disk, outside every whole-box copy, and media apps install there with no extra step (proven on a fresh 512 GB box: 213 GiB area, Jellyfin with no extra step). A disk split into parts can be used part by part. The backup offer no longer promises disk-failure safety when it cannot check the disk, and `/mnt/sys_drive` is no longer offered.
- **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241).
- A box installs from nothing with the public installer, links to the hub, and moves to the floor by itself (map A1, A3, A4).
- 65 catalog apps install, update one tested step at a time, and a failed update undoes itself (map B1, B3, B4, B6).
- Backups: database dumps, a second-drive copy, an off-site copy and whole-box copies; restore tests run by themselves (map C1–C4, A17, A18).
- A restore brings an app back with its data and its database password; a failed or cut restore puts the app back as it was; a restore that does not start says so and keeps both copies (map C5, D6, D7, D16).
- After a power cut the box starts by itself, cleans up a half-written backup, and Immich repairs broken photos (map C10, F12).
- A second drive is offered as the backup target; media apps see every drive; a one-disk box gets its own data area (map D1, D13, D15).
- The three test boxes answer only over https; the hub's key opens nothing on a box; password guesses are counted (map E12, E13, E17).
- A hacked app that uploads or computes more than it explains alarms only you; each box tests its upload weekly; qBittorrent gets half (map F13).
- An urgent security fix reaches the boxes at night after one test on the test box; the box backs up first and undoes a failed update (map B23).
- The hub mails alarms and recoveries and keeps a weekly whole-box copy on ep0; CC uses its own limited token (map F1, F11, G15).
## What is broken
- Two root-level weaknesses wait on your decisions below (R-982, R-985). Tester 2 still has its old emergency password (R-1139, when it is back).
- DooPlex: a failed disk with its alarms silenced (R-934); CI jobs still get lost (R-887); something sets the shared
checkout to „bare" (R-935); a Longhorn volume at 91 %.
- Tester 2 has never made an off-site check (R-862). The scratch box cannot read the drill catalog (R-992).
- **P1:** any hacked app on DooPlex can read the hub's nightly backups, and maybe the signing keys (R-1241, homelab session); about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours).
- Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862). The scratch box cannot use agent features (R-1083).
- DooPlex: something sets the shared checkout to „bare" (R-935).
## What we work on
- **CC's secrets** (R-1118): done 2026-10-17 — the store, `sec`, the guards in every CC session (option B: `bash -x` and `curl -v` allowed inside `sec run`), the nightly check (linger on). The old credentials file and its three copies are deleted.
- **Old restic after the first real 0.19.1 clean-up — DONE 2026-10-19:** demo-hp's window 12 ran at 02:18 UTC (324 → 261 copies, 63 removed as planned); the old restic (image 0.322.0, restic 0.14.0) then listed the same 261 and restored the newest copy (made by 0.19.1) and one from August with the same content hash as the new restic (`audits/evening-2026-10-18/P8/old-restic-check.txt`). R-1105/R-1107: nothing left.
- The power-cut findings still open (R-1004, R-1005, R-1010, R-1013). R-1012 closed: agent 0.159.1 names an orphan thin snapshot in the backup failure.
- Media apps on every drive (R-1023): live with the 0.313.0 floor — every drive in every media app, a new drive made ready by the box, a pulled drive loses nothing, a safe removal lets go first and the drive comes back by itself.
- Dated checks, run by CC. 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the qBittorrent line test on a seeding box, with DooPlex's qBittorrent as the peer (R-1250, your yes); the weekly clean-up window (R-435; the logs of the last one were lost to restarts). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes).
## Waiting on you — decide
| # | Row | Question | Pick | The other option costs | If you do nothing | Blocked |
Newest first. CC's pick is in the „Pick" column. Each row also has its full text in the register.
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|---|---|---|---|---|---|---|
| D1 | R-954 | **DECIDED 2026-10-17: A** — new random emergency root passwords on demo-hp, Tester 1 and demo-felhom (your answer: random); each new one works, each old one is refused. Tester 2: R-1139. | done | — | — | — |
| D2 | R-971 | **DECIDED 2026-10-17: A** — the box refuses a catalog change that asks for dangerous container rights (controller 0.328.0, every box). Only homepage's read-only Docker socket is allowed. A refusal keeps the app running and mails you. | done | — | — | — |
| D3 | R-985 | „Zero-knowledge": change the wording, or remove the operator's root reach to the key? | **Change the wording now**; revisit after the first sale | Remove the reach: a design change to break-glass | the promise stays stronger than the product | the website text |
| D4 | R-982 | Root SSH with a password from the LAN? | **No — key only**; the password stays for the console | Keep it: anyone on the household network can try it | the host keeps accepting root password logins from the LAN | — |
| D5 | R-993 (R-991) | Approve the minimum machine (`audits/sizing-2026-10-13/SIZING.md` §5: x86-64, 4 cores, 8 GB, 256 GB SSD)? | **Approve** | Change it: one edit | no hardware page; a buyer with a Raspberry Pi can still download the installer | the hardware page |
| D6 | R-992 | The scratch box cannot read the private drill catalog: give it a read-only login, or make that copy readable? | **A read-only login** | Readable copy: the drill catalog becomes public | every new-app test needs hand-staging | new-app tests |
| D7 | R-990, R-989 | Rule yes / no / later on the forum ideas (`audits/forum-research-2026-10-12/FELHOM.md` §5) and the selfh.st ideas (`audits/selfhst-survey-2026-10-12/FELHOM.md` §6) | — | — | nothing changes; the research ages | — |
| D8 | R-206 | Install DooPlex's Docker settings from git (restarts Docker), and prune the build cache weekly: yes or no? | **Yes, in a quiet window** | — | a rebuild of DooPlex still has the settings in git | — |
| D13 | R-1140 | **DECIDED 2026-10-17: A** — the hub keeps its own access log, 90 days. Built in hub 0.158.0 (≈ 19 MB for 90 days: the hub serves ≈ 1,870 requests a day, not 500); it waits for your deploy yes (morning list). | done | — | — | — |
| D14 | R-1036 | **DECIDED 2026-10-18: no** — wger and Plant-it stay off the site; it shows only apps a visitor can install. | done | — | — | — |
| D15 | R-1147 | **DECIDED 2026-10-18: yes** — the estimator's Immich peak is 3,983 MB (was 3,290); Immich alone on 8 GB now reads „does not fit". Live. | done | — | — | — |
| D10 | R-1125 | **DECIDED 2026-10-17: A** — the 23 memory requests now match real use (catalog `1af6ea7`); the install refusal and page say what to do (controller 0.327.0). An 8 GB box no longer installs the whole household set. | done | — | — | — |
| D11 | R-1126 | **DECIDED 2026-10-17: A** — FileBrowser is published only after its default password is checked; it never answers on the default (controller 0.327.1). The tunnel still runs before the claim, by design (`01` §5). | done | — | — | — |
| D12 | R-1129 | **CLOSED 2026-10-17: not a defect** — CC misread the CI log; the failure mail always worked. A forced test mail went to admin@felhom.eu at 10:48:54 UTC (CC saw it arrive). Your key change did no harm. | done | — | — | — |
| D9 | R-1062 | The N100's USB drive is also its whole-box backup drive. Split it in two anyway? | **No — keep one drive**; the lived-in box keeps its nightly whole-box backup | Split: the box loses its local whole-box backup (or add a second drive first) | it stays one drive; multi-drive tests stay on Tester 1 | — |
| D16 | R-1100 | **DECIDED 2026-10-19: yes** — built with both safeguards and live on every box (controller 0.332.0): a one-time pass from the dashboard, and a header name only the box knows. | done | — | — | — |
| D17 | R-1149 | **DECIDED 2026-10-19: one per day, 3 days** — agent 0.160.0 on the three test hosts. | done | — | — | — |
| D18 | R-1162 | **DECIDED 2026-10-19: A** — ep0 keeps one whole-box copy per ISO week, 2 weeks (`keep-weekly 2` on demo-hp and demo-felhom; nothing removed today; the first run, 2026-10-20 03:30 UTC, removed only tester-1's 2026-10-04 copy, as predicted). tester-1's missing job is R-1165 (your pick: not by hand today). | done | — | — | — |
| D20 | R-893 | After a failed off-site restore of one app, put it back exactly as it was and start it (A), or keep holding it stopped and only make the database part exact (B)? Measured tonight on the scratch box: the box puts the newer rows over the backup's older volume and a table only the backup had stays; the app is held. Note: `audits/night-2026-10-19/P3/design-R-893.md` | **A — copy the app's volumes first** (your D8 already chose this direction): one copy's disk for the restore's minutes; a fit check refuses before any stop | B: no disk, but a bad undo copy leaves an empty database, and the household still waits for support | a failed off-site restore keeps ending in a stopped app only support can bring back | — |
| D19 | R-1159 | **DECIDED 2026-10-19: box key first, then lock** — a live box pulls its config with its own key (controller 0.334.0, the floor), and the passphrase gives no keys once the box has reported on it (hub 0.161.0). Proven on demo-hp: 409, no key, you got the alarm mail. A reinstall needs your delete of the old host first. | done | — | — | — |
| D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — |
| D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — |
| D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — |
| D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — |
## Waiting on you — do
- **Morning list, 2026-10-20 (from the night of 2026-10-19):**
1. **Deploy hub 0.162.0?** (built `sha256:69285e64…`, from `d92a712e`): a hub restart no longer re-mails a box you already know is down (R-1001); the Day-0 agent list offers only real agent packages (R-1011); a box with no tunnel reads „none set up". **What the deploy does with the three Secrets** (`gitea-creds`, `umami-config`, `healthchecks-config`): **nothing** — ArgoCD no longer tracks them (read 18:21 UTC: none in the app's 25 resources, no tracking mark), so even a whole-app sync with prune leaves them; still sync only the hub, as always (R-953 closed). If not: Tester 2's restart mails continue on every hub restart.
2. **Release agent 0.161.0?** (code `56aabc7`, CHANGELOG „Unreleased"): `host_busy` for the off-site stop (R-921 — without it the controller's new check has nothing to read), a box with no tunnel no longer fails its OS update (R-1070), the janitor clears a half-removed backup snapshot (R-1010). Then your vouch. If not: the extra app stop and the Tester4-type false error stay.
3. **Publish installer 1.35.0?** (tag `installer-v1.35.0` + the `webpage.yaml` ref): a re-run on a live box says „the operator deletes the old host first" in EN and HU instead of „unexpected 409" (R-1167). If not: the next person to hit it reads only „409".
- **Nézd meg a demo-hp-n a Calibre-Web „Adatok és mentés" fülét és a Fájlkezelőt** (`felhom.enkisfelhom.hu`, controller 0.321.0, minden dobozon kiadva). A mentés egy részben, egyszerű szavakkal; a Fájlkezelő nem kér frissítést és nem ajánl „Install app"-ot.
- **Look at `roms/roms/` on the N100's drive in the file manager (R-1099).** RomM now scans `roms/<platform>/`. A game uploaded through RomM before today sits one folder deeper: if there are platform folders in `roms/roms/`, move them up into `roms/` and press „Scan” in RomM. If not: those games show under a platform called „roms”.
- **Nézd meg a Calibre-Web oldalát a demo-hp-n** (`felhom.enkisfelhom.hu`, controller 0.319.0, minden dobozon kiadva). A fejléc, a fülsor és minden rész most külön panel (az „A” stílus). A belépési adatok egy panelben vannak, egymás mellett. A Calibre-Web jelszaváról a doboz nem tud, mert az alkalmazás előbb települt, mint hogy a sablonban jelszó lett volna — ezt most egy mondat mondja ki. Ha nem tetszik, írd meg, mit változtassak. Ha nem nézed meg: semmi nem romlik el.
- **Discard two unclaimed boxes** (hub, browser only): the build B scratch VMs registered as unclaimed boxes — registration ids 40 (`b43afa10…`) and 41 (`9c78ea30…`); both VMs are destroyed. If not: two dead entries stay in the Hosts list. **Tester4 stays** (your word, 2026-10-18) as the fresh-install customer; **before its next install, delete its old host `Tester4-31fa0e`** (Hosts page) — otherwise the new install reuses the old host and skips two re-issues (since hub 0.159.0 you get a mail saying so; the codes are not re-issued by themselves, on purpose).
- System poster needs a refresh: one-disk boxes now have a data area („Belső tárhely") on the system disk, outside the whole-guest backups (facts file updated 2026-10-16).
- **Olvasd el a Plex és a Sonarr útmutatót a demo-hp-n** (az alkalmazás oldalán az „Útmutató” fül). Ha más formát szeretnél, egy menetben átírom mind a 62-t. A robot 2026-10-16-án bejelentkezve megnézte: mindkét oldal megjelenik magyarul és angolul, 4 rész és a GYIK, minden sor egyezik a katalógussal. Ha nem nézed meg: csak a forma marad emberi szemmel nem látott.
- **Nézd meg az AdventureLog és a Plex oldalát a demo-hp-n, a rendes címén** (`felhom.enkisfelhom.hu`). A túl nagy keret javítva: a tartalom most 32 képpontra van a bal oszloptól és 36-ra a tetejétől (előtte 80 és 64 volt). Az alkalmazás oldala mostantól egy hasábban van: egy szélesség, vékony vonalak a szakaszok között, aláhúzott fül, és doboz csak ott, ahol az alkalmazás vár rád (kék) vagy ahol valami kockázatos (sárga). A „sikeres frissítés" sáv kis szürke sorrá vált a név alatt. Ha nem nézed meg: a felület él, csak senki nem látta még böngészőben.
- **Nézd meg az új alkalmazás oldalt a demo-hp-n (egy telepített és egy nem telepített app).** Egy alkalmazás mostantól egy oldal négy füllel (Beállítások · Adatok és mentés · Útmutató · Napló), az Indítás/Újraindítás/Leállítás a „Működés" kártyán van, és a „Vissza" az alkalmazáshoz visz, nem a listához. Mondd meg, jó-e így az elrendezés. Ha nem nézed meg: a felület él, csak senki nem látta még böngészőben — a robot a DooPlexen nem tud kattintani.
- **The N100 sheet (about 30 minutes):** `documentation/audits/n100-lived-in-2026-10-15/SHEET.md` — install Plex and claim it, make the admins in Prowlarr/Sonarr/Radarr, add your download sources in Prowlarr, add one show and one film. Then say „done". If not: the media apps wait, safe behind their gate.
- **Delete the test customer „Tester3” in the hub (browser only).** Its test box is already gone and switched off on purpose, so no alarm is coming. If not: a dead test customer stays in your list.
- **Hub 0.153.0 (your rulings 13:20 + R-1059 „A"): no token can resend a claim code, create or edit a customer, make a passphrase or a link, or re-issue an off-site password.** Live since 13:39 (checked with CC's token: all six refused).
- **Remove the test server „plex-k6” from your Plex account** (plex.tv → Settings → Authorized Devices). The claim test passed and the server is gone from Tester 1. If not: a dead server stays listed in your Plex apps.
- System poster needs a refresh: every public name answers only over https (http → 301).
- **Leftovers moved out of the shared checkout** to `/mnt/5_hdd/felhom.eu/leftovers/2026-10-14/` (nothing deleted): the Facebook `first-post/` folder is not in git — keep, commit or delete it. The rest is already in git. If not: it just stays there.
- **The media tests on Tester 1 are finished** — results in `REPORT-night-2026-10-14.md`. Say „tear it down" when you want Tester 1 back to normal (R-1022; now three test drives and a throwaway Plex).
- **Ask Google to index the new page:** Search Console → URL-ellenőrzés → `https://felhom.eu/sajat-felho-tarhely` → „Indexelés kérése". If not: the page may sit unseen, and the 11 November / 9 December checks (R-1035) read zero for a reason that is not the page.
- **N100 BIOS:** read „After Power Loss" (the HP's „Previous State" made it start after 8 of 8 cuts). **HP BIOS:** „Bypass F1 Prompt on Configuration Changes" → Enable.
- **Pull the failed disk** Z6A07P2G (R-934; the twin Z6A070XZ is `/mnt/3_hdd`). If not: its alarms return on 11 November.
- **Hub master key** into the password manager, then `shred -u` the key file (security review 2026-10-12, in the history).
- **12 reused homelab logins** to rotate, and the repo-visibility decision (R-925, P1).
- **Before the first paying customer:** Tandoor permission (R-789); a lawyer for the licence list (R-802) and the terms (R-813); SparkyFitness (R-784 — its author replied on 2026-10-13); print the break-glass sheet (R-923).
- **Smaller:** pin the first Facebook post (R-917); Tester 2's bootstrap (R-862); look at the uncommitted `deny`-list edit in `homelab-manifests/.claude/settings.json`; the rest of „Your own tasks" (T6–T19) is in the history, 2026-10-10.
**Still needed? (yes/no)** — CC could not check these. Answer yes or no per line.
1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). Since 2026-10-16.
2. The N100 sheet, about 30 minutes: Plex, the admins in Prowlarr, Sonarr and Radarr, your download sources (`documentation/audits/n100-lived-in-2026-10-15/SHEET.md`). Since 2026-10-15.
3. Look at these demo-hp pages when you have time: Calibre-Web, the file manager, AdventureLog, Plex, the app page, the Plex and Sonarr guides. Since 2026-10-16.
4. Remove the test server „plex-k6" from your Plex account (plex.tv → Settings → Authorized Devices). Since 2026-10-15.
5. Ask Google to index `https://felhom.eu/sajat-felho-tarhely` in Search Console (R-1035). Since 2026-10-14.
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14.
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
**Day, 2026-10-23 — your do list.**
1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens.
2. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-f967d4 (the BYO test host, removed 2026-10-25). If not: dead entries alarm.
3. Give the homelab session R-1241, R-1242, the lists at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md` and `documentation/audits/dooplex-alarms-2026-10-24/README.md`, and an upload alarm on DooPlex's `eno1` (~100 Mbit/s line; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups.
**The rest**
- System poster needs a refresh: the new 01:40 „release copy” arrow from DooPlex to ep0 (D33); the hub-database label already says every 6 hours. If not: the drawing misses one copy.
- Tell CC what removed four old agent packages from Gitea on 2026-10-25 (a clean-up rule?): the agent's CI is red until the gate and the rule agree (R-1290).
- Publish installer 1.36.1 (tag + webpage sync): a wrong passphrase then reads „passphrase REJECTED", not a doubled code (R-1284).
- Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09.
- Two DooPlex leftovers: `leftovers/2026-10-14/first-post/` (keep or delete); the `deny`-list edit in homelab-manifests' `.claude/settings.json` (commit or drop). Since 2026-10-14.
- Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784).
- Later (yours, not urgent): R-1250 (a seeding qBittorrent is never tested), R-1248 (a hard upload cap per app), R-1229 (ArgoCD's sync settings into git), R-862 (Tester 2's bootstrap, when it is back), R-504, R-1071, R-1073, R-1077, R-1042, R-132, R-908, R-779, R-904, R-232, R-433, R-933, R-213, R-832, R-388, R-882, R-883, R-814, R-1056, R-1164, R-930, R-920, R-902.
+28 -25
View File
@@ -80,20 +80,20 @@
| <a id="a4"></a>[A4](capability-map-detail/A.md#a4) | Bare-metal Felhom ISO (blank hardware → zero-touch auto-install → first-boot `host-install`); selectable UEFI loader; universal secret-free / operator-bind mode | scripts v1.19.0 (`scripts/iso/`) + hub v0.62.0 + assistant container | **PROVEN-LIVE** | N100 (2026-07-18) and HP t740 (2026-07-21) | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18); `audits/SPIKE-baremetal-iso-2026-07-16.md` (2026-07-16) |
| <a id="a5"></a>[A5](capability-map-detail/A.md#a5) | Box survives a wrong-NIC install: hub-unreachable first boot → legible Hungarian console screen (NIC table + remedy) + NIC sweep self-heal (bounded DHCP + hub probe per NIC, success-only persist), and the baked root password is operator-knowable (`<iso>.rootpw.txt`) | scripts v1.24.0 (`felhom-bootstrap.sh`, `build-felhom-iso.sh`) | **PROVEN-LIVE** | nested drill; metal proof pending | `audits/SPIKE-firstboot-nic-sweep-2026-07-22.md` (2026-07-22) |
| <a id="a6"></a>[A6](capability-map-detail/A.md#a6) | Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** | drill VM; no non-operator human yet (R-3) | `audits/DRILL-day0-vm-2026-07-12.md` (2026-07-12) |
| <a id="a7"></a>[A7](capability-map-detail/A.md#a7) | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** | customer-zero, on metal | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
| <a id="a8"></a>[A8](capability-map-detail/A.md#a8) | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** | drill VM, endpoint-exact; browser wizard first fired 2026-07-18 | `audits/SPIKE-controller-escrow-2026-07-13.md` (2026-07-13); `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
| <a id="a7"></a>[A7](capability-map-detail/A.md#a7) | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** | customer-zero, on metal; **re-check (2026-10-24):** mechanism changed after its evidence — f90e2229 (bind mail W3–W5) | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
| <a id="a8"></a>[A8](capability-map-detail/A.md#a8) | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** | drill VM, endpoint-exact; browser wizard first fired 2026-07-18; **re-check (2026-10-24):** mechanism changed after its evidence — agent c11e3e2 + controller ba4a6f0 (typed-back words, R-1151) | `audits/SPIKE-controller-escrow-2026-07-13.md` (2026-07-13); `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
| <a id="a9"></a>[A9](capability-map-detail/A.md#a9) | One name per secret on every surface — „Beállító kód" (the setup code) and „Tulajdonosi jelmondat" (the owner passphrase) — and the re-enrol mail points at a page a rebuilt box shows | controller v0.211.0, hub v0.104.0 + v0.105.0 (R-295, R-323) | **IMPLEMENTED** | written BUILT: source-read; no customer has typed the passphrase | R-295; R-323 |
| <a id="a10"></a>[A10](capability-map-detail/A.md#a10) | The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE** | demo-felhom, real endpoints | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04); `audits/RECON-offsite-dr-chain-2026-08-04.md` (2026-08-04) |
| <a id="a11"></a>[A11](capability-map-detail/A.md#a11) | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** | demo/drill fleet only | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
| <a id="a12"></a>[A12](capability-map-detail/A.md#a12) | A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE** | the success path; the failure path is not claimed (R-379, R-380) | `audits/DRILL-r201-night-run-2026-08-04.md` (2026-08-04); `audits/DRILL-r356-hot-only-restore-2026-08-22/` (2026-08-22); `audits/DRILL-r356b-driveless-db-restore-2026-08-22/` (2026-08-22) |
| <a id="a13"></a>[A13](capability-map-detail/A.md#a13) | The customer's own UNAIDED recovery journey, end to end | controller v0.206.0, hub v0.98.0, agent v0.127.0 | **PROVEN-LIVE** | fifth walk, scoped — read what it does not claim | `tests/walk5-r201-2026-08-07/journal.md` (2026-08-07); `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md` (2026-08-05) |
| <a id="a10"></a>[A10](capability-map-detail/A.md#a10) | The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE** | demo-felhom, real endpoints; **re-check (2026-10-24):** mechanism changed after its evidence — hub 9688e76e (sealed package back to a rebuilt box, R-1176) | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04); `audits/RECON-offsite-dr-chain-2026-08-04.md` (2026-08-04) |
| <a id="a11"></a>[A11](capability-map-detail/A.md#a11) | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** | demo/drill fleet only; **re-check (2026-10-24):** mechanism changed after its evidence — agent ae69ad4 (D30 WireGuard re-key) | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
| <a id="a12"></a>[A12](capability-map-detail/A.md#a12) | A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE** | 2026-10-21 (fix brief, hub 0.163.0): PASSES — the rebuilt Tester4 box (public ISO) got the household's sealed package from the hub, asked for the recovery code, opened the history; Nextcloud came back with 3 031 of 3 033 manifest files byte-identical (the 2 never reached the old box) and the next off-site runs appended (63 → 64). Earlier the same day it FAILED (R-1176, now closed); the failure paths are not claimed (R-379, R-380) | `audits/fixes-2026-10-21/P1-live/` (2026-10-21); `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `audits/DRILL-r201-night-run-2026-08-04.md` (2026-08-04); `audits/DRILL-r356-hot-only-restore-2026-08-22/` (2026-08-22); `audits/DRILL-r356b-driveless-db-restore-2026-08-22/` (2026-08-22) |
| <a id="a13"></a>[A13](capability-map-detail/A.md#a13) | The customer's own UNAIDED recovery journey, end to end | controller v0.206.0, hub v0.98.0, agent v0.127.0 | **PROVEN-LIVE** | 2026-10-21 (fix brief, hub 0.163.0): PASSES with ONE operator act by design (D19: delete the dead host) — install, bind, claim, the recovery offer, a wrong code refused, the right code, „Teljes visszaállítás", all through the household's own pages; the new-link page now tells a household whose old box is still linked to write to Felhom (R-1203). Earlier the same day it stopped at the rebuilt dashboard (R-1176) | `audits/fixes-2026-10-21/P1-live/` (2026-10-21); `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `tests/walk5-r201-2026-08-07/journal.md` (2026-08-07); `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md` (2026-08-05) |
| <a id="a14"></a>[A14](capability-map-detail/A.md#a14) | A random night of household actions while random things go wrong — twelve rounds, unattended | controller v0.245.0, agent v0.131.0, hub v0.116.0, golden 0.245.0, ISO 1.28.0 | **PROVEN-LIVE** | nested box, twelve chaos rounds | `audits/DRILL-chaos-night-2026-09-17.md` (2026-09-17) |
| <a id="a15"></a>[A15](capability-map-detail/A.md#a15) | A stranger's FIRST HOUR on a fresh install — download → install → claim → two apps → use → backup → remove → restore, plus a power cut and a mistyped code | controller v0.242.0, agent v0.130.0, hub v0.112.0, golden 0.242.0, installer ISO 1.26.1 | **PARTIAL** | every mechanism passed; the unaided journey failed before the first app | `audits/DRILL-fresh-install-0242-2026-09-14.md` (2026-09-14); `audits/DRILL-new-household-2026-09-30.md` (2026-09-30) |
| <a id="a15"></a>[A15](capability-map-detail/A.md#a15) | A stranger's FIRST HOUR on a fresh install — download → install → claim → two apps → use → backup → remove → restore, plus a power cut and a mistyped code | controller v0.336.0, agent v0.161.0, hub v0.162.0, golden 0.331.0, installer 1.35.0, ISO 1.29.0 | **PARTIAL** | 2026-10-20: download → install → bind → claim → 8 apps → use → backup → remove → every restore path → power cut → mistyped codes all worked; the 8 GB box FROZE twice in the first afternoon and only a plug pull brought it back (R-1192), and the owner passphrase needed the operator | `audits/rehearsal-2026-10-20/journal.md` (2026-10-20); `audits/DRILL-fresh-install-0242-2026-09-14.md` (2026-09-14); `audits/DRILL-new-household-2026-09-30.md` (2026-09-30) |
| <a id="a16"></a>[A16](capability-map-detail/A.md#a16) | Off-site app-data capture covers the paths an app declares MANDATORY — on BOTH drive layouts | controller v0.197.0 | **PROVEN-LIVE** | both drive layouts | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04) |
| <a id="a17"></a>[A17](capability-map-detail/A.md#a17) | Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE** | demo-hp | `audits/SPIKE-r82-phase0-2026-07-26.md` (2026-07-26) |
| <a id="a17"></a>[A17](capability-map-detail/A.md#a17) | Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE** | demo-hp; **re-check (2026-10-24):** mechanism changed after its evidence — agent 1b6e316 (local keep-daily, R-1149) | `audits/SPIKE-r82-phase0-2026-07-26.md` (2026-07-26) |
| <a id="a18"></a>[A18](capability-map-detail/A.md#a18) | Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard | agent v0.104.0 → v0.121.0, hub v0.77.0 → v0.91.0 | **PROVEN-LIVE** | — | *(evidence not on file — not re-verifiable)*: the row cites per-repo CHANGELOGs and the spec `backlog/SPEC-r85-phase4-5-2026-07-26.md` — a spec, not a run record |
| <a id="a19"></a>[A19](capability-map-detail/A.md#a19) | A restore-test can never fill the box's disk: it sizes the restore first (uncompressed), keeps off the tested guest's pool when it can, refuses what does not fit, and retries a failed clean-up on a timer | agent v0.133.0 (RELEASED, NOT DELIVERED), hub v0.124.0 | **IMPLEMENTED** | written BUILT: the refusal proven on demo-hp 2026-09-24; the full-test path not live | `audits/r672-2026-09-24/` (2026-09-24); R-672 |
| <a id="a20"></a>[A20](capability-map-detail/A.md#a20) | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE** | two real firings | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
| <a id="a20"></a>[A20](capability-map-detail/A.md#a20) | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0; purge-first hub v0.142.0 | **PROVEN-LIVE** | two real firings (2026-07-18); third 2026-10-22 on a real customer (Tester-2) with the R-32 purge-first leg — all legs ok, others untouched | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18); `audits/tester2-reset-2026-10-22/` |
| <a id="a21"></a>[A21](capability-map-detail/A.md#a21) | Customer DELETE cascade (top lifecycle tier): one guided operator action → `hosts → RESET → residue → purge`; host rows deleted (custody DEMOTED), full external teardown (Hetzner repo destroyed, PBS namespace/token revoked, tunnel + zone removed), then the customer record and ALL escrow ciphertext purged | hub v0.69.0 | **IMPLEMENTED** | written UNIT-PROVEN: live leg pending | hub unit tests `hub/internal/web/customer_delete_test.go` (outside `documentation/`) |
| <a id="a22"></a>[A22](capability-map-detail/A.md#a22) | Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | GL6-F2 open | `pilot/DRILL-GL6-2026-07-08.md` (2026-07-08); R-9 |
| <a id="a23"></a>[A23](capability-map-detail/A.md#a23) | Felhom in a virtual machine on OpenMediaVault 8 (KVM plugin): the whole NVMe passed to the VM, installed from the public ISO, bound, claimed, apps, backup, and back by itself after an OMV restart | ISO 1.29.0, agent 0.156.1, controller 0.314.0, OMV 8.5.11-1 + openmediavault-kvm 8.3.4 | **PARTIAL** | live on an exact OMV copy (VM on demo-hp), not yet on the pilot's real hardware; one-disk Jellyfin not proven (R-1068); NAS films not usable by Jellyfin (R-1071); no SMART inside the VM (R-1073) | `audits/omv-vm-2026-10-15/README.md` (2026-10-15); guide `pilot/GUIDE-omv-kvm-hu.md` |
@@ -102,17 +102,17 @@
| ID | Capability | Component | Status | Scope | Evidence |
|---|---|---|---|---|---|
| <a id="b1"></a>[B1](capability-map-detail/B.md#b1) | Deploy an app from the catalog (env config, memory guard, health-aware progress) | controller, catalog (~52 apps, images pinned) | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/RERUN-p1p3-2026-07-06.md` (2026-07-06) |
| <a id="b2"></a>[B2](capability-map-detail/B.md#b2) | App lifecycle: start/stop/restart/update/logs/remove/redeploy | controller | **PROVEN-LIVE** | the actions; remove-with-data re-proven 2026-09-13 | `audits/R442-2026-09-13/` (2026-09-13); `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
| <a id="b1"></a>[B1](capability-map-detail/B.md#b1) | Deploy an app from the catalog (env config, memory guard, health-aware progress) | controller, catalog (65 apps, images pinned) | **PROVEN-LIVE** | **re-checked 2026-10-24** on controller 0.342.0 (after 6d90374 + 2fe6341): all 65 apps through the deploy request on a fresh box — 62 installed and answered, Wanderer installed but cannot run without DNS (R-739), 2 not offered by lifecycle; the memory guard did not fire (one app at a time) | `audits/catalog-sweep-2026-10-24/README.md` (2026-10-24); `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/RERUN-p1p3-2026-07-06.md` (2026-07-06) |
| <a id="b2"></a>[B2](capability-map-detail/B.md#b2) | App lifecycle: start/stop/restart/update/logs/remove/redeploy | controller | **PROVEN-LIVE** | the actions; remove-with-data re-proven 2026-09-13; **re-check (2026-10-24):** mechanism changed after its evidence — 8ae9178 (removal deletes labelled volumes, R-1117) | `audits/R442-2026-09-13/` (2026-09-13); `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
| <a id="b3"></a>[B3](capability-map-detail/B.md#b3) | Update is GUARDED: it refuses without a restorable backup, backs up first when the copy is stale, and HOLDS an app that does not come up — on ANY backup tier, and the release itself arrives by the managed floor | controller v0.237.0 + v0.238.0 + v0.238.1 + v0.239.0, hub v0.112.0 | **PROVEN-LIVE** | — | `audits/slice4-2026-09-13/` (2026-09-13); `audits/rulings-r472-r475-2026-09-13/` (2026-09-13) |
| <a id="b4"></a>[B4](capability-map-detail/B.md#b4) | A failed update is UNDONE by the box itself — the previous version back with its data from seconds before the update; the app is held only if that undo fails too | controller v0.263.2 (the undo), v0.264.0 + hub v0.120.0 (the mail) | **PROVEN-LIVE** | scratch 9202 | `audits/undo-live-2026-09-23/README.md` (2026-09-23) |
| <a id="b5"></a>[B5](capability-map-detail/B.md#b5) | The catalog holds only TESTED steps — an image move without a proven test record (bench + box, digests the registry still serves) is refused at push time | catalog `6db08a5` (test-record gate, `ladder.py`) | **PROVEN-LIVE** | written PROVEN: the catalog gate against the live registry | `audits/DRILL-night-2026-09-23.md` (2026-09-23) |
| <a id="b6"></a>[B6](capability-map-detail/B.md#b6) | A box behind climbs ONE tested step per press, each with its own definition (`09` §3 decision 14) | controller v0.268.0 + catalog `5ed599c` | **PROVEN-LIVE** | — | `audits/ladder-2026-09-24/partD/10-romm-two-steps.json` (2026-09-24) |
| <a id="b7"></a>[B7](capability-map-detail/B.md#b7) | A file app comes back WHOLE from the second drive — its files by four rules (never delete, never overwrite a newer file, bring back what is missing, keep an older replaced file beside it), then its settings and database (`09` §3 decision 26) | controller v0.269.0 (`backup/tier2_whole.go`) | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A1/21-readback.json` (2026-09-24) |
| <a id="b7"></a>[B7](capability-map-detail/B.md#b7) | A file app comes back WHOLE from the second drive — its files by four rules (never delete, never overwrite a newer file, bring back what is missing, keep an older replaced file beside it), then its settings and database (`09` §3 decision 26) | controller v0.269.0 (`backup/tier2_whole.go`) | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d582feb + 5929b8e (D22, D25) | `audits/night-2026-09-24/A1/21-readback.json` (2026-09-24) |
| <a id="b8"></a>[B8](capability-map-detail/B.md#b8) | A crash-looping or out-of-memory app is STOPPED by the box, the household and operator are told, and Start gives one more try (decision 28) | controller v0.269.0, hub v0.123.0 | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A3/21-gokapi-page-event.txt` (2026-09-24) |
| <a id="b9"></a>[B9](capability-map-detail/B.md#b9) | A held app whose page says support is informed can only be removed KEEPING its data (decision 27) | controller v0.269.0 | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A2/10-held-no-copy-keep-data.json` (2026-09-24) |
| <a id="b10"></a>[B10](capability-map-detail/B.md#b10) | The box runs the exact TESTED image of a floating tag, and an installed app keeps its image until a guarded Update moves it (`09` §6.4 part 6, box half) | controller v0.269.1 (`stacks/digest.go`) | **PROVEN-LIVE** | — | `audits/night-2026-09-24/B/21-floating-tag-0.269.1.json` (2026-09-24) |
| <a id="b11"></a>[B11](capability-map-detail/B.md#b11) | Automatic updates at night — the update leg (`09` §6.4 part 7) | controller v0.271.0 | **PROVEN-LIVE** | scratch 9202, six simulated nights; demo boxes in Part D | `audits/DRILL-night-2026-09-25.md` (2026-09-25) |
| <a id="b11"></a>[B11](capability-map-detail/B.md#b11) | Automatic updates at night — the update leg (`09` §6.4 part 7) | controller v0.271.0 | **PROVEN-LIVE** | scratch 9202, six simulated nights; demo boxes in Part D; **re-check (2026-10-24):** mechanism changed after its evidence — 63dcccf (security_fix steps, 0.341.0) | `audits/DRILL-night-2026-09-25.md` (2026-09-25) |
| <a id="b12"></a>[B12](capability-map-detail/B.md#b12) | What `restart` and `update` do to a deployed app whose compose file the catalog already moved | controller v0.235.0 | **PROVEN-LIVE** | HISTORY: the behaviour up to v0.234.0; changed 2026-09-06 — current behaviour is B17 | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
| <a id="b13"></a>[B13](capability-map-detail/B.md#b13) | Whether the box UPGRADES an app by itself, with nobody pressing anything | controller | **PROVEN-LIVE** | answer: yes, only when an app fails to come back (measured 2026-09-01) | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
| <a id="b14"></a>[B14](capability-map-detail/B.md#b14) | Whether an app UPGRADE can be undone | controller + catalog | **PROVEN-LIVE** | answer: no, once a migration has run | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
@@ -124,6 +124,7 @@
| <a id="b20"></a>[B20](capability-map-detail/B.md#b20) | App crashes → customer notified (one event per transition, no flapping spam) | controller v0.120, hub v0.48 | **IMPLEMENTED** | crash→customer mail never live-confirmed | `audits/CAMPAIGN-3-2026-07-11.md` (2026-07-11) |
| <a id="b21"></a>[B21](capability-map-detail/B.md#b21) | Post-deploy optional config (API keys etc.) with restart | controller, catalog `.felhom.yml` | **IMPLEMENTED** | config-save + restart in no campaign | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
| <a id="b22"></a>[B22](capability-map-detail/B.md#b22) | Backup classification: 13 bind-bearing apps carry mandatory/optional/excluded classes | catalog, controller v0.132–133 | **PROVEN-LIVE** | — | `audits/SPIKE-backup-classification-2026-07-14.md` (2026-07-14); `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15) |
| <a id="b23"></a>[B23](capability-map-detail/B.md#b23) | An urgent security fix reaches the boxes at night — a `security_fix` ladder step (box walk required, bench optional, a CVE/GHSA notice), taken by the night update (`09` §3 decision 255, D39 = B) | catalog `3898593`, controller v0.341.0 | **PROVEN-LIVE** | scratch 9202 (drill catalog), Plex 1.43.4 → 1.43.5; an app with no box fixture is checked by health, version and a soak only (`seed: none`) | `audits/security-lane-2026-10-23/` (2026-10-23) |
## C. Protection & recovery (the product promise)
@@ -146,10 +147,10 @@
|---|---|---|---|---|---|
| <a id="c1"></a>[C1](capability-map-detail/C.md#c1) | Whole-guest backup lands OFF the guest's own physical device; a single-drive box is recorded DEGRADED rather than silently normal (installer Case A/B) | agent v0.113, host-install v1.22.0 | **PROVEN-LIVE** | — | `audits/E2D-fresh-vm-2026-07-29.md` (2026-07-29) |
| <a id="c2"></a>[C2](capability-map-detail/C.md#c2) | Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15) |
| <a id="c3"></a>[C3](capability-map-detail/C.md#c3) | Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14) |
| <a id="c4"></a>[C4](capability-map-detail/C.md#c4) | Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/VALIDATION-offbox-storagebox-2026-07-09.md` (2026-07-09) |
| <a id="c5"></a>[C5](capability-map-detail/C.md#c5) | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/DIAG-immich-restore-round2-2026-07-19.md` (2026-07-19) |
| <a id="c6"></a>[C6](capability-map-detail/C.md#c6) | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6B-2026-07-14.md` (2026-07-14) |
| <a id="c3"></a>[C3](capability-map-detail/C.md#c3) | Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 03c0a69 + 66a69d4 + 5929b8e (capped container R-1214, kept copies D25) | `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14) |
| <a id="c4"></a>[C4](capability-map-detail/C.md#c4) | Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 8f5bea0 + d5f88a2 (abandon R-1182, off-site proof R-1189) | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/VALIDATION-offbox-storagebox-2026-07-09.md` (2026-07-09) |
| <a id="c5"></a>[C5](capability-map-detail/C.md#c5) | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 93fb251 + d582feb (D20, D22) **2026-10-25:** changed again by controller 0.343.0 (D48 pull-first, R-1270 fix A: the verdict needs a stable start, the undo copy goes after 3 more minutes); on 0.343.0 Tester 1's local unit restore and set-aside restore passed and released the copy (`audits/release-2026-10-26/t1-restores-0.343.0/`); this path itself not re-walked. | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/DIAG-immich-restore-round2-2026-07-19.md` (2026-07-19) |
| <a id="c6"></a>[C6](capability-map-detail/C.md#c6) | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d5f88a2 + 5929b8e (R-1187 c, R-1191) | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6B-2026-07-14.md` (2026-07-14) |
| <a id="c7"></a>[C7](capability-map-detail/C.md#c7) | Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `pilot/DRILL-GL6-2026-07-08.md` (2026-07-08) |
| <a id="c8"></a>[C8](capability-map-detail/C.md#c8) | PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | scoped to one host | `audits/SPIKE-pbsdr-selfheal-2026-07-15.md` (2026-07-15) |
| <a id="c9"></a>[C9](capability-map-detail/C.md#c9) | Box survives an unattended app or guest-network failure (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0→v0.190.0, agent v0.92.1 | **PROVEN-LIVE** | demo box; boot-orphan leg 6 of 6 hard resets | `audits/DIAG-bootrecon-drive-absent-2026-08-02.md` (2026-08-02) |
@@ -171,16 +172,17 @@
| <a id="d3"></a>[D3](capability-map-detail/D.md#d3) | Data migration between drives (all / per-app), crash-safe | controller | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14); `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
| <a id="d4"></a>[D4](capability-map-detail/D.md#d4) | NAS (NFS/SMB-client) verify-before-commit, uid-1000 probe, categorized Hungarian errors, DSM-validated | controller v0.113–117, agent v0.81/84/85 | **PROVEN-LIVE** | — | `audits/SPIKE-nas-verify-2026-07-11.md` (2026-07-11); `audits/SPIKE-nas-dsm-2026-07-11.md` (2026-07-11) |
| <a id="d5"></a>[D5](capability-map-detail/D.md#d5) | Network storage (NAS) is browse + bulk-media only — it may NOT host an app's data namespace | controller v0.187.0 | **PROVEN-LIVE** | — | `audits/R108-network-app-namespace-2026-07-30.md` (2026-07-30) |
| <a id="d6"></a>[D6](capability-map-detail/D.md#d6) | A Tier-1/Tier-2 app restore works from the DRIVE ALONE — the recovery unit carries the app's secrets (D5) | controller v0.188.0 | **PROVEN-LIVE** | — | `audits/D5-drive-alone-restore-2026-07-30.md` (2026-07-30) |
| <a id="d7"></a>[D7](capability-map-detail/D.md#d7) | A restore SAYS what it returned, and refuses what it cannot do — the four restore-surface truth defects from the 2026-08-21 drill | controller v0.226.0 (R-353, R-357, R-358, R-360, R-396) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: three of four live; R-357 implemented only | `audits/evidence-r353-r360-live-2026-08-30/live-validation.txt` (2026-08-30) |
| <a id="d6"></a>[D6](capability-map-detail/D.md#d6) | A Tier-1/Tier-2 app restore works from the DRIVE ALONE — the recovery unit carries the app's secrets (D5) | controller v0.188.0 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d582feb (D22, R-1178) **2026-10-25:** changed again by controller 0.343.0 (D48 pull-first, R-1270 fix A: the verdict needs a stable start, the undo copy goes after 3 more minutes); on 0.343.0 Tester 1's local unit restore and set-aside restore passed and released the copy (`audits/release-2026-10-26/t1-restores-0.343.0/`); this path itself not re-walked. | `audits/D5-drive-alone-restore-2026-07-30.md` (2026-07-30) |
| <a id="d7"></a>[D7](capability-map-detail/D.md#d7) | A restore SAYS what it returned, and refuses what it cannot do — the four restore-surface truth defects from the 2026-08-21 drill | controller v0.226.0 (R-353, R-357, R-358, R-360, R-396) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: three of four live; R-357 implemented only **2026-10-25:** controller 0.343.0 adds two truths here — a restore whose app does not start says so and keeps its copy (R-1270, D49 = A; live on 9202 rc.6), and a `.fab` import that cannot download its version refuses before it touches the app (R-1279, live). | `audits/evidence-r353-r360-live-2026-08-30/live-validation.txt` (2026-08-30) |
| <a id="d8"></a>[D8](capability-map-detail/D.md#d8) | The box PROVES its own off-site copy still HOLDS something — a backup that is intact and EMPTY is caught without a person | controller v0.231.0 (R-87) + hub v0.110.0 | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the scheduled firing is implemented only | `tests/r87-offsite-proof-2026-08-31/` (2026-08-31) |
| <a id="d9"></a>[D9](capability-map-detail/D.md#d9) | The off-site store is VERIFIED on a cadence — something checks that the customer's backups are still readable | controller v0.228.0 (R-359, R-397, R-399) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the scheduled firing is implemented only | `tests/r359-integrity-2026-08-30/` (2026-08-30) |
| <a id="d10"></a>[D10](capability-map-detail/D.md#d10) | USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | — | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15); `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13) |
| <a id="d11"></a>[D11](capability-map-detail/D.md#d11) | Decommission (migrate-first and anyway-paths), eject | agent, controller | **PROVEN-LIVE** | — | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
| <a id="d10"></a>[D10](capability-map-detail/D.md#d10) | USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — agent e15d1d3 (R-1124) | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15); `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13) |
| <a id="d11"></a>[D11](capability-map-detail/D.md#d11) | Decommission (migrate-first and anyway-paths), eject | agent, controller | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — agent e15d1d3 + 5c3b218, controller 6c6bd87 (R-1124, R-1144) | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
| <a id="d12"></a>[D12](capability-map-detail/D.md#d12) | Boot ordering: automount + networking survive reboot; appliance self-heal watchdog | agent v0.85 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13); `audits/CAMPAIGN-6A-2026-07-14.md` (2026-07-14) |
| <a id="d13"></a>[D13](capability-map-detail/D.md#d13) | Media apps see EVERY drive; a drive added, pulled, back or safely removed is handled per app (notice, real restart, wiring, torrents paused/resumed) | controller v0.313.0, catalog `0188430`, hub v0.151.0 | **PROVEN-LIVE** | Tier-0 boxes (Tester 1, demo-hp 9201); the fleet with golden 0.313.0 + the floor | `audits/media-build-2026-10-15/G/` (2026-10-15); `audits/media-drive-events-2026-10-14/` (2026-10-14) |
| <a id="d14"></a>[D14](capability-map-detail/D.md#d14) | A second local copy COUNTS only on a proven different physical disk: the agent knows each place's disks; a VM box makes the copy but says it cannot check; a same-disk copy is not made; a NAS never hides the one-copy notice; a box with no drive refuses a data-folder app in words | agent v0.157.0, controller v0.317.0, hub v0.154.0 | **PARTIAL** | live: VM `unknown` (Tester 1) and two real-disk boxes `proven` (demo-hp, N100); unit only: the same-disk skip, the NAS notice, the no-drive sentence; `usb_bridge` never seen on a device | `audits/storage-identity-buildA-2026-10-15/README.md` (2026-10-15); `architecture/16-storage-physical-identity.md` §13 |
| <a id="d15"></a>[D15](capability-map-detail/D.md#d15) | A one-disk box gets its own data area („Belső tárhely") outside every whole-box copy, registered as the default drive with no press, so media and data-folder apps install with no extra step; a free disk's partitions can be formatted or used one by one | installer 1.33.0, agent v0.158.1, controller v0.318.0, golden 0.318.0 | **PROVEN-LIVE** | fresh 512 GB one-disk VM on demo-hp; partitions on a scratch disk; the old-box signed job and the D4 sentence unit-proven only; an empty part asks for confirmation (R-1093) | `audits/storage-identity-buildB-2026-10-15/README.md` (2026-10-16) |
| <a id="d16"></a>[D16](capability-map-detail/D.md#d16) | A failed or cut restore puts the household's data back EXACTLY as it was and starts the app — local unit, second drive, kept load and off-site, and after a power cut or a killed controller (D22) | controller 0.338.0 (D22) → **0.343.0** (released 2026-10-25: R-1251 a copy on an unbound drive waits; fix A) | **IMPLEMENTED** | proven live on 9202 (rc.1): a damaged unit and a controller killed between the volume step and the replay; the second-drive pre-check (R-1178), the mirror skip (R-1180) and the off-site power cut by test only; a database outside the replaced volumes is not covered **2026-10-25 (re-check):** released in 0.343.0; the release run proved the success path's copy release on Tester 1, not a failure put-back on the release build — the status stays. | `audits/night-2026-10-21/P3-restore/02-run.log` (2026-10-21) |
## E. Access, networking & household use
@@ -202,7 +204,7 @@
| <a id="e14"></a>[E14](capability-map-detail/E.md#e14) | Multiple household users / per-person accounts | — | **MISSING** | — | R-15 |
| <a id="e15"></a>[E15](capability-map-detail/E.md#e15) | A second login step for the dashboard (a TOTP code or a passkey) | — | **MISSING** | — | R-811 |
| <a id="e16"></a>[E16](capability-map-detail/E.md#e16) | The household can leave Felhom, or outlive it — the box runs without the hub, the household owns its domain, tunnel and off-site account, and can export everything | — | **MISSING** | as a written answer | R-810 |
| <a id="e17"></a>[E17](capability-map-detail/E.md#e17) | The host agent cannot reach root without the operator key: exact sudo patterns, no agent-written file installed where root reads it without a content check, the agent binary only by an operator-signed update checked as root | agent v0.146.1 (R-861; delivered by a step bundle, R-880) | **PROVEN-LIVE** | both demo boxes; three named residuals | `audits/hub-safety-2026-10-05/partF/` (2026-10-05) |
| <a id="e17"></a>[E17](capability-map-detail/E.md#e17) | The host agent cannot reach root without the operator key: exact sudo patterns, no agent-written file installed where root reads it without a content check, the agent binary only by an operator-signed update checked as root | agent v0.146.1 (R-861; delivered by a step bundle, R-880) | **PROVEN-LIVE** | both demo boxes; three named residuals; **re-check (2026-10-24):** mechanism changed after its evidence — agent e15d1d3 + a0c88e5 (sudoers + priv-apply, D4 drop-in) | `audits/hub-safety-2026-10-05/partF/` (2026-10-05) |
| <a id="e18"></a>[E18](capability-map-detail/E.md#e18) | WireGuard base infra always-on; OOB operator access (felhom-sshd, /32 peer) | agent v0.72, hub v0.35 | **IMPLEMENTED** | — | `audits/SPIKE-oob-wg-operator-peer-2026-07-05.md` (2026-07-05); `audits/SPIKE-felhom-sshd-2026-07-05.md` (2026-07-05) |
| <a id="e19"></a>[E19](capability-map-detail/E.md#e19) | The operator can see WHERE a managed host is — its LAN address and its WireGuard address, on the host page | agent v0.119.0, hub v0.85.0 | **PROVEN-LIVE** | — | `audits/host-addresses-visible-2026-07-31.md` (2026-07-31) |
| <a id="e20"></a>[E20](capability-map-detail/E.md#e20) | The operator can see whether a managed host's guests still have working networking — and how often the watchdog had to repair them | agent v0.92.0 (emitter, 2026-07-21), hub v0.104.0 (reader, 2026-08-13) | **IMPLEMENTED** | not proven live | R-319 |
@@ -221,6 +223,7 @@
| <a id="f7"></a>[F7](capability-map-detail/F.md#f7) | Always-on debug rings + on-demand log-bundle pulls with TTL/custody | controller v0.116, agent v0.83, hub v0.46 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-3-2026-07-11.md` (2026-07-11) |
| <a id="f8"></a>[F8](capability-map-detail/F.md#f8) | Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | no corpus validation doc | operator infra; no evidence document |
| <a id="f9"></a>[F9](capability-map-detail/F.md#f9) | Backup-deadline alerting (`expected_backup_missed`) is ANCHORED — absence of signal is UNKNOWN, not failure | hub v0.75.0 | **IMPLEMENTED** | — | `audits/DIAG-backup-missed-2026-07-26.md` (2026-07-26); R-82 |
| <a id="f13"></a>[F13](capability-map-detail/F.md#f13) | A hacked app is noticed: an app that uploads or computes far more than its own numbers explain alarms the operator only; each box tests its upload line weekly and qBittorrent gets half of it (`08` §9; D41, D42; R-1244, R-1247) | controller v0.342.0, agent v0.164.0, hub v0.169.0, ep0 `felhom-iperf3` | **PROVEN-LIVE** | the alarm path on scratch 9202 (a hidden flood and a CPU burner, 2026-10-23); the line test on the three test boxes; the qBittorrent limit on demo-felhom. **Gaps:** a seeding qBittorrent is never tested (R-1250); an app's API can lie (R-1248) | `audits/SPIKE-upload-watch-2026-10-23.md` §7, `audits/line-test-2026-10-24/` (2026-10-24) |
| <a id="f12"></a>[F12](capability-map-detail/F.md#f12) | After a power cut, an app rebuilds the files the cut left empty, and the household sees any lost photo on the app page | controller v0.311.0, hub v0.150.0, catalog Immich block | **PROVEN-LIVE** | demo-hp 2026-10-14: a real cut mid-upload, 41 of 41 broken photos repaired, 120 controls untouched; the lost-photo note not exercised live (0 lost; unit- and 9202-tested). **Fleet delivery done 2026-10-14:** golden 0.311.0 + global floor; all three demo boxes on 0.311.0; catalog gate `unclean-boot-images` | `audits/immich-repair-2026-10-14/README.md` (2026-10-14); `09` §3 decision 236; R-1007 |
| <a id="f11"></a>[F11](capability-map-detail/F.md#f11) | The household turns the whole machine off („A gép kikapcsolása"); the hub holds every silence alarm for 14 days and shows KIKAPCSOLVA | agent v0.155.0, hub v0.149.0, controller v0.310.0 | **PROVEN-LIVE** | proven on the N100 2026-10-13; fleet delivery waits on the floor | `audits/poweroff-2026-10-13/README.md` (2026-10-13); `architecture/14-host-power-off.md`; R-1000, R-1002 |
| <a id="f10"></a>[F10](capability-map-detail/F.md#f10) | A box that is not always on (off at its backup time) makes the missed night up once when it comes back; the household sees a banner; the operator gets the missed-backup alarm, down or not | controller v0.295.0, hub v0.134.0, agent v0.145.0 | **PARTIAL** | catch-up and banner live; R-872 alarm by tests; suspend not measured | `audits/catchup-2026-10-05/partA/` (2026-10-05); R-872 |
@@ -235,7 +238,7 @@
| <a id="g4"></a>[G4](capability-map-detail/G.md#g4) | Publish train: MinAgent floors, gated auto-Reissue, version channels, floor-field-LAST rules | hub v0.45/0.53, agent | **PARTIAL** | — | `runbooks/publish-train-rules.md` (2026-09-13); R-23 |
| <a id="g5"></a>[G5](capability-map-detail/G.md#g5) | Agent self-update: A/B slots, crash-loop auto-rollback, operator-signed | agent v0.70+ | **PROVEN-LIVE** | demo | `audits/SPIKE-agent-selfupdate-2026-07-05.md` (2026-07-05) |
| <a id="g6"></a>[G6](capability-map-detail/G.md#g6) | Controller self-update: anonymous registry, no credentials in guest | controller v0.112 | **PROVEN-LIVE** | demo; the file found is the floor-driven self-update leg | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
| <a id="g7"></a>[G7](capability-map-detail/G.md#g7) | Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue | hub v0.37–0.39 | **PROVEN-LIVE** | — | `audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md` (2026-07-09); `audits/SPIKE-hetzner-api-provisioning-2026-07-09.md` (2026-07-09) |
| <a id="g7"></a>[G7](capability-map-detail/G.md#g7) | Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue | hub v0.37–0.39 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — hub 9688e76e (scoped abandonment, R-1182) | `audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md` (2026-07-09); `audits/SPIKE-hetzner-api-provisioning-2026-07-09.md` (2026-07-09) |
| <a id="g8"></a>[G8](capability-map-detail/G.md#g8) | Per-customer offsite fill + staleness + freeze lever | hub v0.41 | **IMPLEMENTED** | no live-fired leg | `audits/CAMPAIGN-offsite-overnight-2026-07-10.md` (2026-07-10) |
| <a id="g9"></a>[G9](capability-map-detail/G.md#g9) | Box-level Storage Box aggregate (total fill, Σ quotas, oversubscription alert) | hub v0.64.0 (R-5) | **IMPLEMENTED** | data pipeline live; UI render and alert mail not | hub v0.64.0 code and unit tests (outside `documentation/`) |
| <a id="g10"></a>[G10](capability-map-detail/G.md#g10) | Operator sees PBS DR datastore fill at a glance (Offsite "PBS DR" tab + dashboard gauge) | hub v0.65.0 + tenantsync v1.2.0 (R-5) | **IMPLEMENTED** | data pipeline live; UI render and alert mail not | hub v0.65.0 code and unit tests (outside `documentation/`) |
@@ -243,10 +246,10 @@
| <a id="g12"></a>[G12](capability-map-detail/G.md#g12) | Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | — | the 2026-07-13 hub closing bundle (no evidence document) |
| <a id="g13"></a>[G13](capability-map-detail/G.md#g13) | Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | — | hub v0.54 (2026-07-13; no evidence document) |
| <a id="g14"></a>[G14](capability-map-detail/G.md#g14) | The operator sees boxes left behind: per-customer controller floors with their age (and which a global raise cannot move), each box's agent against the vouched one, a 7-day "agent behind" alarm and a "floor raise skipped boxes" mail | hub v0.135.0 (R-604, R-530) | **IMPLEMENTED** | the page live; the alarm and the mail unit-proven | `audits/hub-safety-2026-10-05/partD/` (2026-10-05) |
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a nightly consistent copy, encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | — | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a consistent copy every 6 hours since D34 (nightly before 2026-10-23; corrected 2026-10-24), encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — be7e97dc + 52af38df (D34 6-hourly) | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
| <a id="g16"></a>[G16](capability-map-detail/G.md#g16) | Gitea (all code), the operator's password manager (Vaultwarden, since 2026-10-09 afternoon, R-923) and DooPlex's secrets survive the loss of DooPlex: a nightly encrypted copy on ep0; restore-tested weekly; an alarm when either stops; a failure mail | `scripts/dooplex-offsite/` (R-232), homelab-manifests rules | **PROVEN-LIVE** | — | `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
| <a id="g17"></a>[G17](capability-map-detail/G.md#g17) | The operator surface refuses a cross-site form post on BOTH login paths (session token; Basic auth + `X-Felhom-Operator`) | hub v0.135.0 (R-135) | **PROVEN-LIVE** | — | `audits/hub-safety-2026-10-05/partA/live.txt` (2026-10-05) |
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo; **re-check (2026-10-24):** mechanism changed after its evidence — agent a0c88e5 (os-apply sshd check, D4) | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
| <a id="g19"></a>[G19](capability-map-detail/G.md#g19) | An ENGLISH-SPEAKING household's first hour: download, install, pair, bind, claim, two apps | controller v0.259.0 + hub v0.119.0 + ISO 1.29.0 + the whole catalog | **PROVEN-LIVE** | walked on 0.258.0 with one blocker; fixed, the walk not repeated | `audits/DRILL-first-hour-en-0258-2026-09-20.md` (2026-09-20); `audits/i18n-closing-2026-09-21/live/` (2026-09-21) |
| <a id="g20"></a>[G20](capability-map-detail/G.md#g20) | A deletion of a customer's off-site history is NOTICED within a day | hub v0.111.0 (R-431) | **IMPLEMENTED** | not yet proven live | R-431 |
@@ -16,15 +16,17 @@
**Status:** draft (decisions from the topology/trust design sessions).
**Platform facts** referenced here live in `docs/proxmox-platform.md`; this document
**Platform facts** referenced here live in `documentation/proxmox-platform.md` (path corrected 2026-10-24); this document
records *Felhom's decisions*, not Proxmox behaviour.
---
## 1. Model at a glance
Three components. **Control is always box-initiated** — the hub never connects *into* a
customer box.
Three components. **Control is always box-initiated** — the hub never ~~connects~~ sends a command *into* a
customer box. **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** The hub may POKE a box: through ep0, one empty UDP datagram to the box's WireGuard
address (port 51822; `hub/internal/poke/client.go`, agent `internal/poke/poke.go`). A poke carries no command; it only
makes the box pull its desired state at once. The box still pulls.
```
operator side customer box (per Proxmox host)
@@ -98,14 +100,18 @@ credentials.
- CGNAT does **not** force this: the Cloudflare Tunnel already makes a box reachable through
Cloudflare's edge. We *choose* box-initiated control for the smallest attack surface — the
box exposes no control endpoint at all.
- The agent and the controller **poll** the hub; the hub never initiates inbound.
box exposes no control endpoint ~~at all~~: no route on the box takes a command from the hub. **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]**
Two listeners sit behind the WireGuard tunnel: the agent's poke listener (a poke carries no command; the box still
pulls) and the operator door, `felhom-sshd` on the host, reachable only from the operator's `/32` peer — the
documented break-glass (`03` §5, `06` §4.5).
- The agent and the controller **poll** the hub; the hub never initiates ~~inbound~~ a command inbound — a poke only
makes them poll at once (corrected 2026-10-25, D45, R-1266).
- Operator actions are delivered as **signed jobs**: the agent verifies an operator signature
before executing, so a compromised hub database alone cannot forge commands.
- **[FACT, 2026-10-16, R-970]** The last contradiction is gone: until controller v0.322.0 the box's own hub API key,
sent as a Bearer token, opened `/api/config/` (read `controller.yaml` with every secret in it, and REPLACE it,
unsigned) and the self-update and geo routes. Nothing called them (the hub never connects to a box). The config
routes are removed and no route accepts the hub key; every `/api/` route needs a dashboard session (pinned by
routes are removed and no route accepts the hub key; every `/api/` route but the read-only `/api/health` needs a dashboard session (pinned by
`TestR970_HubKeyOpensNoDoor` on the real route table). The box exposes no control endpoint, as stated above.
- All operator-initiated actions are recorded in a **customer-visible audit log**.
@@ -132,7 +138,7 @@ of all 53 apps, 2026-09-28).
**[DESIGN, 2026-10-17, R-1126, operator decision D11 = A] The tunnel runs before the claim, and FileBrowser never
answers on a default password.** The box starts cloudflared as soon as it has a tunnel token, before it is claimed,
because the claim itself is reached through it: the claim mail links `https://felhom.<domain>`
(`hub/internal/notify/templates.go:27`). Before the claim a stranger reaches only the claim page (a code: 72 h life, 5
(`hub/internal/notify/templates.go` `FormatClaimEmail`; line corrected 2026-10-24). Before the claim a stranger reaches only the claim page (a code: 72 h life, 5
tries, 15-minute lockout — controller `web/claim.go`) and `files.<domain>`. FileBrowser starts every new database on
admin/admin, and its own admin-password setting cannot replace that (it re-applies at every start and would overwrite
the household's own password — FileBrowser 1.5.6 `backend/cmd/user.go:50-57`, measured). So FileBrowser starts OFF
@@ -219,7 +225,10 @@ app. Controller down → the gated app answers an error, never the app. Measured
This removes any zero-touch remote-enrollment problem.
- A **one-time retrieval code** mints durable identity. Single-use (burned on the successful
config fetch) plus a short *pre-use* TTL; one-click regenerate for the only real failure
case (fetch fails before anything is persisted). After the fetch, the code is irrelevant —
case (fetch fails before anything is persisted). *(As built — corrected 2026-10-24: the passphrase is not burned
and has no TTL; it gives keys until the box has a host record AND a controller ≥ 0.334.0 has reported with its own
key, then `/config`, `/recovery` and `/host-enroll` answer 409 — D19, R-1159, `hub/internal/api/passphrase_scope.go:19-30`;
`05` §8 has the full rule.)* After the fetch, the code is irrelevant —
everything downstream runs on durable credentials, so retries don't need it.
- **Order:** the agent enrolls first (and, running as root at setup, mints its own scoped
operator-tier Proxmox token), then provisions the customer LXC from the golden template and
@@ -242,7 +251,8 @@ app. Controller down → the gated app answers an error, never the app. Measured
2026-10-08, R-900). Replacing it with our own relay is deferred until after the first customers (R-904, `09` §3
decision 184).
- **Outbound only** for control/report/backup (poll to hub, push to PBS). No inbound control
endpoint exists in the chosen model.
endpoint exists in the chosen model. **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** A poke carries no command; the box still pulls. The operator
door (`felhom-sshd` over the WireGuard tunnel) is the documented break-glass (§4, `03` §5).
- **Every customer has their OWN domain — never a name under `felhom.eu`** (operator ruling
2026-09-14). The free Cloudflare tier's certificate covers one level below a zone, so nested names
such as `felhom.<customer>.felhom.eu` are not covered; the customer's own domain is a few thousand
@@ -253,7 +263,7 @@ app. Controller down → the gated app answers an error, never the app. Measured
2026-09-14 first-hour drill used a `*.felhom.eu` customer domain with no tunnel, and the dashboard link
in the setup-code mail did not resolve.
- **A pasted Cloudflare API token must reach exactly the customer's own zone** (R-138 option C, `09` §3 decision 190;
hub main 2026-10-08, unreleased). On save the hub asks Cloudflare `GET /zones` with the token and keeps it only when
released in hub v0.144.0 — corrected 2026-10-24). On save the hub asks Cloudflare `GET /zones` with the token and keeps it only when
the token sees exactly ONE zone and that zone IS the customer's domain (a zone above it is refused: a sibling
customer added later would be reachable). Cloudflare unreachable → the save fails and the old token stays. The token
is never logged. **Limit:** `/zones` lists what the token can READ; a hand-built token with Zone:Read on one zone and
@@ -313,7 +323,8 @@ app. Controller down → the gated app answers an error, never the app. Measured
>
> **What being unmarked cost.** The consequence of this bullet — that 40 of 53 catalogue templates
> declare no configurable path because they are all-hot — is stated as **[FACT]** at
> `07-backup-architecture.md:296-299`. A reader met a marked observation beside an unmarked choice
> `07-backup-architecture.md:296-299`. *(Dated note 2026-10-24: that pointer is dead — those lines are about the local
> plaintext posture now, and no architecture file but this one states „40 of 53 … all-hot".)* A reader met a marked observation beside an unmarked choice
> and reasonably asked whether it *should* be so. **Between 19 and 22 August that reader called this
> decision a defect in four places** (R-370), and one session's work went into correcting the record
> rather than into the product.
@@ -326,7 +337,12 @@ app. Controller down → the gated app answers an error, never the app. Measured
- **Key custody (zero-knowledge default):** three tiers the customer chooses —
*customer-only* / *zero-knowledge escrow (default)* / *operator-managed*. Default escrows
the **PBS passphrase-protected keyfile** in the hub, wrapped under a **customer recovery
code** the operator can't open; DR needs the customer's code. Access-notification is an
code**; DR needs the customer's code. ~~the operator can't open~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** The hub keeps only the
locked key package, and every older one it retained; without the customer's code the operator cannot open any of
them, and from the off-site storage and the hub nobody can read the backup. Two exact limits, as the website says
since R-985: while the box runs, the operator has root access for support and with it could reach the data and the
backup key on the box; and the operator can open an OLDER off-site store from a retained package, by hand, when the
household types its old code on its own box (`runbooks/RUNBOOK-open-retained-package.md`, R-304). Access-notification is an
audit signal, never the primary guard. (Don't build bespoke crypto — use PBS's native
keyfile passphrase.)
@@ -343,14 +359,19 @@ app. Controller down → the gated app answers an error, never the app. Measured
restores from PBS; the controller returns on the same domain (tunnel reused from the hub
record). DR = provisioning + a restore mode, not a separate mechanism.
- **Snapshot-before-deploy:** controller asks the agent to snapshot, deploys, runs its
post-deploy health check, asks the agent to roll back on failure. (Transient snapshot, §8.)
post-deploy health check, asks the agent to roll back on failure. (Transient snapshot, §8.) **[DESIGN — not built,
noted 2026-10-24]:** the agent serves `/snapshot` and `/rollback`, but no controller code calls them
(`felhom-controller/controller/internal/agentapi`); the update's guard is the per-app backup + undo of `09` §3 decision 15.
---
## 10. How this embodies the product values
- **Zero-knowledge offsite** — the operator holds the offsite backup but cannot read it.
- **Box-initiated control + signed jobs** — no standing operator backdoor; a hub compromise
- **Zero-knowledge offsite** — the operator holds the offsite backup but cannot read it without the customer's code
(the two exact limits are in §8 — corrected 2026-10-25, D45, R-1266).
- **Box-initiated control + signed jobs** — ~~no standing operator backdoor~~ no hidden operator door: the one
operator door is `felhom-sshd` over the WireGuard tunnel, the documented break-glass (`03` §5), and the website tells
the household about the operator's root access (R-985) — corrected 2026-10-25, D45, R-1266; a hub compromise
alone can't forge commands.
- **Customer-visible audit log** — every operator action is visible to the customer.
- **Never hold data hostage** — subscriptions cover ongoing labour (monitoring, offsite,
@@ -105,7 +105,9 @@ Tri-state: `""` (unknown) · `"running"` · `"stopped"`.
### In-flight — a marker file, one per owner
Two exist and they are deliberately **separate files**: `quiesce-state.json` (the whole-guest backup
Two exist and they are deliberately **separate files** *(four in-flight records by 2026-10-24: also `update-journal.json`
— the guarded update's crash journal read by `RecoverUpdates`, `internal/stacks/update.go:49,1306` — and
`.felhom-install-pending` — an interrupted install, R-681, `install_interrupted.go:28`; noted 2026-10-24)*: `quiesce-state.json` (the whole-guest backup
window, `internal/quiesce`) and `appstop-state.json` (app-data operations that stop an app —
`backup.AppStopGuard`, covering the volume dump, offbox reconstitution and `.fab` export). **One
file, one writer**: sharing would give one record two lifetimes, and one owner clearing the other's
@@ -169,7 +171,8 @@ A recovery completing after the grace emits a `LATE RECOVERY` warning naming the
**Nothing is started without asking whether it may be.** `bootrecon.StartGate` is the one question
the sweep asks per candidate, and it is **fail-safe: cannot determine ⇒ do not start.** Three holders
answer it, and the last two only became reachable once the window widened past T+5 s:
answer it *(four since slice 4: check „1b", a guarded update moving the app — `IsUpdating`, `cmd/controller/main.go:2459-2477`;
noted 2026-10-24)*, and the last two only became reachable once the window widened past T+5 s:
| Holder | Why starting would be wrong |
|---|---|
@@ -218,6 +221,13 @@ own; every caller that is not the customer must decide for itself whether the ap
---
> *Dated note 2026-10-24 (review round 2): this inventory is v0.33's. On `main` (`321df1f`) 22 packages under
> `controller/internal/` have no row here: `aftercrash`, `agentapi`, `appbackup`, `backupwindow`, `bootrecon`,
> `bootstrap`, `channelhealth`, `crashboot`, `dockerexec`, `family`, `fillwatch`, `i18n`, `infra`, `logx`, `mailrelay`,
> `mediawire`, `nightchain`, `offsiteapply`, `privpolicy`, `quiesce`, `sockheal`, `upwatch` (an `ls` of the tree). Each
> package's own doc comment is its description; `sockheal` (re-heal after the guest's Docker socket is recreated,
> R-860) and `channelhealth` (a periodic proof of the controller→agent channel) are in no architecture document.*
## 2. Classification table (per package/file)
### `cmd/`
@@ -266,6 +276,11 @@ own; every caller that is not the customer must decide for itself whether the ap
|---|---|---|---|
| `client.go`,`zone.go`,`waf.go`,`geosync.go`,`countries.go` | **DELETE (→hub)** | The **hub** holds the CF API token and reconciles geo desired-state → WAF (doc 01 §5, doc 03 §2). The controller no longer calls the Cloudflare API — it reports geo desired-state up. The customer-facing geo *preference UI/data* stays (see `api/geo.go`). | needs-rework |
*As built (noted 2026-10-24, review round 2): the `cloudflare/` package is still wired — with `cf_api_token` set the
controller builds a `GeoSyncManager` and calls the Cloudflare API (`cmd/controller/main.go:1992-2000`; the field is still in
`config.go:160`). The DELETE classification above is the plan, not the state; whether any box sets the token is not
readable from source.*
### `config/`, `crypto/`, `util/`
| File | Class | Reason | Risk |
|---|---|---|---|
@@ -325,6 +340,7 @@ household's file names. The household's lost list is shown on the app page (`los
| File | Class | Reason | Risk |
|---|---|---|---|
| `version.go` | **KEEP** | Semver parse / version string (still used for reporting). | clean |
| *(as built 2026-10-24: both files below are LIVE — the updater decides the version and the floor and hands the container swap to the agent, `AgentSwapper`, `internal/selfupdate/updater.go:38-47`; `state.go` keeps `update-state.json`)* | | | |
| `state.go` | **DELETE (obsolete)** | Self-update audit state — the agent owns controller updates now (doc 03 §11). | clean |
| `updater.go` | **DELETE (→agent)** | Resolved (doc 03 §11): the controller is **agent-managed** — the agent snapshots → redeploys → health-gates → rolls back the controller. The controller's old self-update path (image pull + compose edit) is **removed**. | clean |
@@ -387,7 +403,8 @@ dashboard alert is an `Alert` with two placement fields: `PageOnly` (the pages i
page) and `Inline` (rendered by the page template in place, not by the layout's banner). `GetAlerts` returns
the list (endpoint-drift, agent-channel and dead-app alerts first, then the rest sorted error > warning > info,
capped at five plus an overflow line), and `layout.html` paints only those that are not `Inline` and match the
page; `GetInlineAlerts(page)` hands the dashboard and monitoring pages their inline ones.
page *(as built since R-906 — noted 2026-10-24: pages use `GetBannerAlerts(page, lang)`, which filters per page BEFORE the
cap, `internal/web/alerts.go:353`, `handlers.go:144`)*; `GetInlineAlerts(page)` hands the dashboard and monitoring pages their inline ones.
**Exactly one warning is inline today:** the „storage is not on a separate drive" health warning. It is
`PageOnly: dashboard, monitoring` and `Inline: true`, so it sits quietly under the storage bars; **every other
@@ -821,7 +838,7 @@ R-438. The three facts below are the ones a reader needs before touching any of
`Syncer.copyTemplates` (`sync/sync.go:319`) walks every directory in the catalog cache and copies
`docker-compose.yml` and `.felhom.yml` into the matching stack folder. **There is no test of whether
the app is deployed.** The only guard is a sha256 content compare (`copyIfChanged`) and the only
exclusion is `app.yaml`. Interval is `git.sync_interval`, default `15m` (`config/config.go:351`), plus
exclusion is `app.yaml`. Interval is `git.sync_interval`, default `15m` (`config/config.go:351` then, `:399` now; `copyTemplates` is now `sync.go:436`), plus
one immediate sync at controller start (`sync.go:98`).
**SINCE v0.235.0** this walk still happens and `.felhom.yml` is still copied unconditionally, but the
@@ -835,6 +852,7 @@ and they stay that way until something else acts.
### 2. Every lifecycle path resolves that disagreement, silently, by upgrading
*(This section is the 2026-09 measurement; `UpdateStack` was deleted in v0.237.0 — the button calls `StartGuardedUpdate`, `09` §6.1 — noted 2026-10-24.)*
`StartStack`, `RestartStack` and `UpdateStack` (`stacks/manager.go:1029 / 1133 / 1170`) all end in
`docker compose up -d`. `up -d` makes the container match the file, **and pulls the image itself if it
is absent** — measured at 18.3 s with a pull versus 0.5 s without, against a negative control
@@ -887,7 +905,9 @@ measured; that the restore writes to that path is read, not measured.
beállítások csak olvashatók.", `deploy.html`). This is a design, not an oversight: a changed value would
need a guarded re-deploy that nothing performs. **Finding (2026-09-15):** the catalog field flag
`locked_after_deploy` (`stacks/metadata.go`) is parsed and read by **no** controller code — every field
is read-only after install whatever the catalog says. So a page must never tell a household to change
is read-only after install whatever the catalog says. *(Dated note 2026-10-24: it is now read into the app's
`LockedFields` (`internal/stacks/deploy.go:106,534`) and enforced only by `UpdateStackConfig`, which nothing calls — so
„read-only whatever the catalog says" still holds in practice.)* So a page must never tell a household to change
a value after install. **Vaultwarden (R-512)** follows the design instead of breaking it: registration
is closed by default and the household is invited from the admin panel, measured to work without mail
(stranger 400, invite 200, invited 200).
+119 -10
View File
@@ -47,7 +47,7 @@ Owns:
1. **Proxmox lifecycle** — create/start/stop/destroy guests, snapshots, storage allocation. Via a scoped Proxmox API token (the **`FelhomAgent` operator role** — `proxmox-platform.md` §3.6, validated Phase 3 B3) for everything the API covers; raw host ops only where unavoidable.
2. **Storage management** — attach/classify targets, reconcile the storage manifest, mount USB-by-UUID, present mounts into guests.
3. **Backup/restore orchestration** — vzdump to the tiers, PBS, snapshot management, and the **self-restore-test**.
4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub. **[FACT, 2026-10-04] The "cloudflared health" leg reads a host unit that does not exist:** `internal/hub/cloudflared.go` runs `systemctl is-active cloudflared` on the HOST, but cloudflared is a container in the guest (below), so every box reports `inactive` (`Unit cloudflared.service could not be found`, demo-hp). R-841. **[FACT, FIXED agent v0.141.0 / controller v0.292.0, R-841 CLOSED]** The agent now reads the guest's `cloudflared` container through the existing `pct exec [0-9]* -- docker inspect -f *` sudoers line: state, exit code and the Docker health status of a check the controller adds (`cloudflared tunnel --metrics localhost:20241 ready` → cloudflared's own `/ready`, 200 only with a connection). Three states: `running` (healthy), `not_running` (stopped, absent, or running but NOT connected), `unknown` (could not ask, or the check is still starting) — `unknown` never alarms. No new sudoers line. **[FACT, agent v0.142.0, R-852]** The host report gains `system`: the Proxmox version and kernel from the
4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub. **[FACT, 2026-10-04] The "cloudflared health" leg reads a host unit that does not exist:** `internal/hub/cloudflared.go` runs `systemctl is-active cloudflared` on the HOST, but cloudflared is a container in the guest (below), so every box reports `inactive` (`Unit cloudflared.service could not be found`, demo-hp). R-841. **[FACT, FIXED agent v0.141.0 / controller v0.292.0, R-841 CLOSED]** The agent now reads the guest's `cloudflared` container through the existing `pct exec [0-9]* -- docker inspect -f *` sudoers line: state, exit code and the Docker health status of a check the controller adds (`cloudflared tunnel --metrics localhost:20241 ready` → cloudflared's own `/ready`, 200 only with a connection). Four states (corrected 2026-10-24): `running` (healthy), `not_running` (stopped, or running but NOT connected), `unknown` (could not ask, or the check is still starting) — `unknown` never alarms — and since agent v0.161.0 `absent` (the guest has no cloudflared container at all, R-1070; the hub never alarms on it) — `internal/hub/cloudflared.go:12-23`. No new sudoers line. **[FACT, agent v0.142.0, R-852]** The host report gains `system`: the Proxmox version and kernel from the
Proxmox API, plus the OS wrapper's read-only `facts` (host Debian, next-boot kernel, held packages, taint, the crash
guard; guest Debian, Docker engine, containerd, live-restore), at most every 10 min. **Still no new sudoers line** — the
facts, `live-restore-on` and the Docker layer all ride `FELHOM_OSAPPLY`. New ROOT-owned files instead (installer 1.30.0;
@@ -64,7 +64,8 @@ Explicitly does **not**:
- Serve application traffic or sit in the data path. **Control plane, not data plane**: if the agent dies, apps keep serving (Docker + LXC run without it); only *management* degrades — no new backups, no provisioning, hub loses the heartbeat.
- Hold or proxy customer application data.
- Run inside a guest. It is the thing that recovers guests and the host; it cannot be one of them.
- Manage **geo-restriction / the Cloudflare API**. Geo is hub-owned: the customer sets it in the controller UI, the controller reports the geo desired-state to the hub, and the **hub** (holding the CF API token) reconciles the WAF (S4). The agent manages only the *tunnel* service (`cloudflared`, §3/§5), never WAF rules.
- Manage **geo-restriction / the Cloudflare API**. Geo is hub-owned: the customer sets it in the controller UI, the controller reports the geo desired-state to the hub, and the **hub** (holding the CF API token) reconciles the WAF (S4). ~~The agent manages only the *tunnel* service (`cloudflared`, §3/§5), never WAF rules.~~ The agent manages no
Cloudflare piece: it only reads the guest's `cloudflared` container (§3) — corrected 2026-10-25, D45, R-1266.
## 3. Process model & host integration
@@ -129,6 +130,31 @@ the one trust root; the agent's API token is untouched; a box gets the new rule
`agent_config_update` (the order: signed `agent_update` first, then the bundle — after the bundle, an agent below
0.146.0 cannot update itself on that box).
**[DESIGN, ruled 2026-10-22 15:47 (D4, R-982) — SHIPPED in agent v0.163.0 (`1109092`) and installer-v1.36.0 (`manifests/webpage.yaml:347`); corrected 2026-10-24] Root SSH from the LAN by key only; the console password stays.** The config bundle gains one fixed file, `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`), checked and reloaded by `felhom-os-apply`; the installer's uninstall removes it. Why a drop-in: Debian's `sshd_config` Includes `sshd_config.d/*` FIRST and sshd keeps the first value it reads, so the drop-in wins over the stock line; the out-of-band operator sshd (`FELHOM_SSHD`, port 8822) has its own config with no Include and is not touched. Proven on a fresh ISO 1.30.0 install: password SSH refused after, key SSH and the PVE web password login (200) and the console login still work (`audits/night-2026-10-22/P2-D4/`). It ships with the next agent release + installer tag.
**[DESIGN, operator ruling D43 = A, 2026-10-24 — committed agent `c8d3c17`, unreleased; R-1257]** **Appliances only.** The
drop-in's bundle policy is `appliance`: written only when the root-owned install record (`/var/lib/felhom-install/state.json`)
says `appliance` (the installer records the mode before it installs the bundle); on a `byo` record nothing is written and
a copy WE wrote (this path, beginning with our header line) is removed, then `sshd -t` and a reload; a box with no
readable record is left as it is. The „in force" check asks `sshd -T` for each of the host's own global IPv4 addresses
(a `Match Address <LAN>` block no longer reads as in force); a failing `sshd -t` puts back ONLY the SSH file and keeps
the rest of the bundle (the report says `sshd_step_undone`).
**[FACT, agent `a0c88e5`, released in v0.163.0 — corrected 2026-10-24] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b).
**[DESIGN, operator ruling D30 2026-10-22; agent `ae69ad4`, released in v0.163.0 — corrected 2026-10-24] The tunnel heals itself by a new key.** When
the latest handshake stays stamped in the future for 15 minutes, the agent makes a new WireGuard key pair (the FIRST old
private key kept as `wg/private.key.prev`, later ones as `wg/private.key.prev.<unix>`, the newest three kept, 0600 — the
escrow may hold one; the once-an-hour bound survives a restart in `wg/last-rekey` — `internal/wgtunnel/rekey.go`, corrected 2026-10-24) and registers the new public key on the
existing path, `POST /hosts/<id>/wg`: the hub re-keys the peer in place (same /32, generation bump) and its full-list
push to ep0 (`wgsync`, `wg syncconf`) drops the old key and adds the new one. A new key is a new peer for ep0, with no
stored timestamp. Once an hour at most; not while the unit is down; never during a re-register already in flight. The
interface switches at the agent's next desired-state fetch (up to its report interval). **`[FACT]` 2026-10-23 night
(test box Tester4-16fdef, `audits/night-2026-10-23/P2-D30/`):** with ep0 free to start a handshake of its own, a 2-hour
clock jump healed by itself in ~3 minutes (ep0 had traffic for the box; the box accepts ep0's timestamps) and the agent
correctly did not re-key; with ep0's handshake starts dropped on the box (the rehearsal's condition), the tunnel stayed
dead and the agent re-keyed after 15 minutes. **Known gap:** a reboot after the clock was corrected forgets the
future stamp (R-1223).
## 4. Control model — reconcile + signed destructive ops
Two channels, split by **reversibility**, not by transport.
@@ -215,10 +241,16 @@ alerts the operator when an agent misses its expected check-in window. This is t
failure mode for a managed service, so it gets first-class treatment hub-side.
**Break-glass.** Standing inbound control is off. But when the poll loop *itself* is wedged
(agent hung, host sick) you cannot fix it through the poll loop. So there is an explicit,
**off-by-default, customer-consented, fully-audited** emergency path: SSH to the host via
(agent hung, host sick) you cannot fix it through the poll loop. So there is an explicit emergency path.
~~**off-by-default, customer-consented, fully-audited** emergency path: SSH to the host via
the Cloudflare Tunnel behind Cloudflare Access (or on-site). Enabling it is itself a signed,
logged operation; it auto-expires.
logged operation; it auto-expires.~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** The path is the operator door: the operator's WireGuard
peer reaches a dedicated `felhom-sshd` on the host (a non-22 port, reachable only from the operator's `/32` over
`wg-felhom`, key-only login; `06` §4.5, TASK H1). It does not use the Cloudflare Tunnel. It is installed by default on
an appliance install, and a BYO install refuses it (the owner's call; `scripts/felhom-host-install.sh`). It stays on;
it is not opened per incident. When SSH itself is broken, the hub-vaulted `root@pam` password opens the Proxmox web
console (`runbooks/break-glass.md`). The website tells the household that the operator has root access for support
(R-985).
## 6. Agent ↔ controller local API
@@ -233,8 +265,9 @@ The controller (in its LXC) reaches the agent (on the host) over the local bridg
- `POST /backup` — request a backup-now of *this* guest (enqueued; non-destructive).
- `GET /backup/due` — whether a policy-scheduled backup is due for *this* guest, so the controller can quiesce then call `POST /backup` (the app-consistent path, §8).
- `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI. **`host_busy`
(R-921, agent Unreleased `56aabc7`):** `GET /backup/status` also says whether the host-wide heavy-operation gate is
held (`{busy, what}` — another guest's backup, the restore test, an OS step, the stale-lock sweep, a power-off), the
(R-921, agent `56aabc7`, released in v0.161.0 — corrected 2026-10-24):** `GET /backup/status` also says whether the host-wide heavy-operation gate is
held (`{busy, what}` — another guest's backup, the restore test, an OS step, the stale-lock sweep, a power-off, and since v0.164.0 the
weekly upload line test — gate name `line-test`), the
state in which `POST /backup` answers 409; absent when the gate is not wired. The controller (≥ 0.335.0) reads it
before it stops any app. A read, never a reservation.
- **Crash-boot fact (R-856, agent v0.149.0):** `GET /host/crash-guard` — the host crash guard's last-boot record
@@ -289,6 +322,49 @@ A controller can only `POST /rollback` (or snapshot/backup) **its own** guest
token → guest and authorizes per guest, so a compromised controller's blast radius is
**self-scoped and bounded** to its own guest.
**[DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265); every route listed below since
2026-10-25]:** the surface listed above is the slice-8 design; on `main` the local API registers **41** routes
(`felhom-agent/internal/localapi/server.go:562-634`). The list above names 15 of them; the other 26 are in the table
below (recounted 2026-10-25 — the review counted 24). Every route is behind the same per-guest token (`withGuest`).
Among them the controller can start a HOST action: `POST /host/line-test/run` (about 55 s; bounded by the due
date, the window, 6 tries a day and the heavy-operation gate — R-1250, unreleased). The local API also listens on the
R-50 island bridge (host `169.254.253.1`, guest `net1` `169.254.253.2/30`, `internal/config/config.go:260-269`). The
weekly line test keeps its state in the OOB state dir (`line-test.json`) and has an off switch, `line_test.disable`.
The served WireGuard block carries `oob_peer_ip` and `oob_operator_ssh_key` (`internal/hub/report.go`). The D30 re-key
runs only inside the tunnel watchdog, and its 15-minute count lives in memory — an agent restart starts it again (the
once-an-hour bound survives in `wg/last-rekey`).
The routes the list above does not name (read from `server.go:562-634` and each handler's own comment, 2026-10-25):
| Route | What it does |
|---|---|
| `GET /backup/tiers` | this guest's backup tiers, each with its cadence and whether its Proxmox storage exists now (`present` / `absent` / `unknown`) |
| `POST /backup/target` | moves the primary backup tier to a new storage: create → grant → config; answers `restart_required`, and the caller restarts the agent after its own in-flight check |
| `GET /host/line-test` | HOST: the weekly upload line test's last measurement; `present:false` when never measured (R-1247) |
| `POST /host/line-test/run` | HOST: starts a line test now — about 55 s, bounded as above (R-1250) |
| `GET /disks/candidates` | the host disks free for Felhom to enroll (`initialize`, and `attach` for those with a mountable file system) |
| `POST /disks/same-disk` | whether two of the guest's paths sit on the same physical disk, from the agent's own read (`16` §4) |
| `POST /disks/decommission` | takes the household's own drive out for good: a permanent intent, the guest bind pruned, unmounted; never formats; a system or backup mount is refused (403) |
| `GET /disks/format/status` | the newest or running format job, for a caller whose request timed out |
| `POST /disks/guest-attach` | binds an enrolled drive's `felhom-data` folder into this guest (read-write); the rest of the drive stays hidden |
| `GET /guest/memory` | this guest's RAM and its bounds |
| `POST /guest/memory` | a bounded RAM resize, applied live (R-24) |
| `POST /netstorage/add` | mounts a NAS share on the host and starts a verify job; a failed verify rolls it back |
| `GET /netstorage` | the NAS shares |
| `GET /netstorage/verify-status` | the verify job's state |
| `POST /netstorage/remove` | removes a NAS share |
| `POST /controller/swap` | swaps the controller image (202, then async); the agent verifies and rolls back |
| `GET /controller/swap/status` | the swap's state |
| `POST /escrow/stage-secret` | stages the off-site repository password for the escrow ceremony (0600, never logged) |
| `DELETE /escrow/stage-secret` | wipes the staged password |
| `POST /escrow/recover-offsite-password` | with the recovery code in the body, returns only the off-site repository password (§8a) |
| `GET /escrow/preflight` | the ceremony's checklist |
| `POST /escrow/ceremony` | starts the root ceremony job |
| `GET /escrow/ceremony/status` | the ceremony's state |
| `POST /escrow/ceremony/claim` | the one-shot hand-off of the new recovery code; a later claim is 410 |
| `POST /escrow/ceremony/confirm` | the household's two words; only then does the package reach the hub (R-1151) |
| `GET /debug/logs` | the agent's debug log ring, for the controller's Debug page |
### 6a. Implementation (slice 8A — implemented)
**Status: implemented** (agent v0.10.0 `internal/localapi`; controller v0.35.0 `internal/bootstrap`
@@ -297,7 +373,8 @@ token → guest and authorizes per guest, so a compromised controller's blast ra
quiesce-on-due consumer is 8B), the rest wrap the existing slice-5/6/7 machinery.
- **Transport / pin.** The agent serves a **persisted self-signed leaf** bound to the host bridge IP
on a fixed port (default `:8443`). The controller pins the **leaf-cert SHA-256** (decision:
on a fixed port — `local_api.listen_addr` is REQUIRED, there is no default (`internal/config/config.go:317-319`); an
island install (R-50) listens on `169.254.253.1:8443` on its private bridge (corrected 2026-10-24). The controller pins the **leaf-cert SHA-256** (decision:
consistency with the agent's Proxmox/PBS cert pinning), carried in its bootstrap. The leaf is
generated **once and persisted**, so its fingerprint is stable across agent restarts (a fresh cert
each boot would invalidate every already-issued bootstrap pin). Defense-in-depth: the listener
@@ -403,8 +480,8 @@ per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a ba
> space on …" — so the hub raises `restore_test_failed`; it is never a pass and never dropped.
> - **Leftovers on a timer.** A failed scratch teardown and the stale-lock sweep (R-673) run every 10
> minutes, not only at agent start; the sweep holds the one-heavy-operation gate; after 3 failed
> teardown tries the operator is told. **A half-removed `vzdump` snapshot (R-1010, agent Unreleased
> `56aabc7`):** when the sweep's delete fails ONLY with „Failed to find logical volume …_vzdump" (one disk's
> teardown tries the operator is told. **A half-removed `vzdump` snapshot (R-1010, agent
> `56aabc7`, released in v0.161.0 — corrected 2026-10-24):** when the sweep's delete fails ONLY with „Failed to find logical volume …_vzdump" (one disk's
> snapshot already gone), it retries once with Proxmox's own `force=1` on the same API call (no sudoers line, no
> root command); any other error is returned unchanged, and a failed forced delete is never forced again.
> - **A thin pool ≥ 90 % requests an immediate report**; the hub judges a thin pool on the worse of data
@@ -481,6 +558,18 @@ box. The escrow is how `K` comes back **without** Felhom ever being able to read
(hub v0.8.0, `PUT /api/v1/hosts/{host_id}/escrow`). Validated end-to-end on a throwaway in
`documentation/tests/slice7-escrow-spike-findings.md`. Restore-mode *serving/consumption* is slice 10.
#### The rebuilt box gets the household's package back — the CARRY [DESIGN, operator ruling 2026-10-21, hub v0.163.0, R-1176]
After the operator deletes a dead box's host (escrow acknowledged), the package is demoted to retained custody under
the OLD host id. The hub now **re-attaches** it: when the customer has exactly one live host, no live host holds a
package, and its newest escrow-acknowledged deletion left a retained row with an identity blob, that row becomes the new
host's current escrow row — moved, in one transaction, on the agent's first report (and again on the controller's
report, idempotently). **Zero-knowledge is unchanged:** the hub moves the same ciphertext from one row to another; it
never opens, re-seals or logs it; the audit (`escrow_carried`) carries host ids and the non-reversible hash. Two live
hosts → no carry, `escrow_carry_refused` to the operator. The box then sees „identity present", its mint guard (R-241)
holds, and it offers the recovery screen. Proven live 2026-10-21 (`audits/fixes-2026-10-21/P1-live/`: the right code
opened the household's history, Nextcloud's 3 031 files came back byte-identical, the next off-site run appended).
#### The separation principle (the rule that governs every posture)
Reading customer data needs **BOTH** the encrypted chunks **AND** a usable key. **Zero-knowledge
holds for exactly as long as Felhom never holds both at once.** Every posture below is just a
@@ -643,6 +732,26 @@ run as a **journaled reconcile job**; a mid-flight failure is compensating-rolle
the just-restored guest — allowed unsigned per §4, same-transaction provenance). They diverge in
the *archive* and the *back half*, **and in identity policy** (below).
**Guest swap = guest RAM, and the host's swap behind it (R-1192; agent 0.162.0, released and vouched 2026-10-22).**
`[FACT]` 2026-10-21 (`audits/fixes-2026-10-21/P4-freeze/`): Proxmox gives an LXC a 512 MB swap cap when nothing sets
`swap`, and the agent never set it; on the 8 GB minimum machine (guest 4096 MB) the household's first uploads filled the
guest with its swap full, the kernel could evict only program code, and the box froze (memory „full" pressure 96 %, load
162, the dashboard dead 6½ minutes). `[DESIGN]` The size step sets `swap` = the guest's memory: at bring-up, at a RAM
resize (`POST /guest/memory`), and ONCE, one minute after agent start, for every pool LXC whose swap is below its RAM (`cmd/felhom-agent/main.go`
`EnsureGuestSwap`; „once a minute" corrected 2026-10-24) (live,
no restart, never lowers). Proven on a fresh box the night of 2026-10-21 (`audits/night-2026-10-21/P1-soak/`: 512 →
4096 one minute after the fixture agent started; the 2-hour household load then answered every 10-second probe). **The
host must have swap behind it:** our installer gives an 8 GB / 256 GB machine 7.75 GB (measured the same night); a BYO
host may have none. The agent reports `host.host_swap_mb` and `host.guest_swap_mb` on every host report and logs one WARN
when the first is below the second (its own, strict line — a log, nothing more). **The judgement is the hub's and the
box's, by one rule (R-1216, D28 = A, operator 2026-10-22):** short = no host swap, or less than 90 % of min(the guests'
swap allowance, 8192 MB) — 8192 MB being the most Proxmox's installer gives any machine (`Install.pm compute_swapsize`:
the RAM, at least 4096 MB on a disk ≥ 64 GB, at most 8192 MB). The hub mails the operator once (`host_swap_short`,
operator-only); the controller's install page adds „A gépen nincs elég cserehely (swap)…" when memory is tight and the
rule says short. One case table in both repos, byte-identical (`scripts/host_swap_rule_gate.py`). The first rule (host
swap < the guests' whole allowance) flagged every machine with a guest above 8 GB; the 8 GB soak used 2.7 GB. The
agent never creates or enlarges host swap — changing a host its owner set up is a decision (register).
**Identity reset is scenario-specific — this is a correctness boundary, not a detail.** "Reset
identity" is shorthand for two different operations:
@@ -15,9 +15,9 @@
> Status: design draft (decision content), grounded on `docs/tests/phase4-signing-findings.md`.
> Status: design draft (decision content), grounded on `documentation/tests/phase4-signing-findings.md`.
> To be reviewed by Claude Code against that spike + `03` §4, then placed at
> `docs/architecture/04-control-plane-authorization.md`.
> `documentation/architecture/04-control-plane-authorization.md` — where it is now (paths corrected 2026-10-24).
>
> Builds on Part 1 (enrollment / trust), Part 3 (the agent verifies + the §4 reversibility gate).
> This doc defines the **mechanism** behind `03` §4's "an operator signature the hub can't forge."
@@ -75,7 +75,12 @@ recorded last**, only after everything else passes, so an invalid signature can
nonce (DoS-safe). Each layer is mandatory and was proven to reject independently (Phase 4 §3–4):
- **target binding** — `target.host_id`/`guest_id` must equal *this* box/guest (a signature for box
A cannot be replayed at box B);
- **time window** — `now ∈ [issued_at, expires_at]`;
- **time window** — `now ∈ [issued_at, expires_at]`; as built, `issued_at` may be up to **2 minutes** in the future
(clock skew), `expires_at` is never extended — the agent (`felhom-agent/internal/authz/verifier.go:19-22,148-155`) and
the root wrapper (`configs/felhom-os-apply`) alike (corrected 2026-10-24). **[DESIGN — the operator picked a code change
2026-10-25 (D44); row R-1280]:** neither caps the window's LENGTH (`expires_at − issued_at`); `felhom-opsign` defaults to 30 min
(`cmd/felhom-opsign/main.go:57`), and the nonce store keeps a record until its expiry. The pick: the agent and the
root wrapper refuse a window longer than 24 h. Not built yet;
- **nonce** — unseen within the window (the nonce store **must be persistent across agent restarts**
and expiry-pruned; a non-persistent store reopens the replay window after a restart).
@@ -166,7 +171,7 @@ op the agent verifies** (same pipeline, §2.3) — never unauthenticated config.
queue — then the agent polls, verifies, executes, and audits. One command + passphrase, from the
desk. **Never** a site visit.
### 6.1 Operator actions in the report reply [DESIGN, R-314/R-279 — `09` §3 decision 185, controller + hub main 2026-10-08, unreleased]
### 6.1 Operator actions in the report reply [DESIGN, R-314/R-279 — `09` §3 decision 185; released in controller v0.304.0 + hub v0.144.0 — corrected 2026-10-24]
Routine, non-destructive operator requests need no signature (`03` §4 asks for one only to destroy or overwrite the
only copy). The hub stores a row, bumps the box's intent, and the report reply carries `operator_actions:[{id,action,arg}]`
@@ -22,8 +22,9 @@
> The hub is **not** greenfield — it's a mature service (felhom-hub **v0.11.0** as of 2026-06-13, Go +
> SQLite on k3s, `hub.felhom.eu`). This doc is the **deltas** to evolve it for the Proxmox model, plus
> the new data model. Builds on Part 1 (trust/enrollment), Part 3 (the agent + reconcile), Part 4 (signing).
> NOTE: this remains a design-draft for the deltas; the live hub is v0.11.0 (the prior "v0.6.3" reference
> was stale). Ground specifics against `felhom.eu/hub/` before relying on them.
> NOTE: this remains a design-draft for the deltas. It was written against hub v0.11.0 (2026-06-13); the deployed
> version is read from `manifests/hub.yaml`, never from this line (corrected 2026-10-24, review round 2: the old
> "the live hub is v0.11.0" was a current-state version literal). Ground specifics against `felhom.eu/hub/` before relying on them.
## 1. Source-of-truth model — two drivers, two directions
@@ -91,7 +92,7 @@ Evolves the existing staleness checker (60s **cadence**, a **configured** thresh
- **Primary = host-report recency → `host_stale` / `host_down`.** The agent heartbeat is the box's
liveness signal; a silent agent = the box is gone (the critical alert). **A hub restart does not re-mail a known
silence (R-1001, hub 0.162.0 — built 2026-10-19, not yet deployed):** a host whose newest `host_stale`/`host_down`
silence (R-1001, hub 0.162.0 — deployed since; corrected 2026-10-24):** a host whose newest `host_stale`/`host_down`
event is newer than its last report is seeded with that state at start, silently; `restore_test_stale` reads its edge
from the DB on the first sweep (a signal for the tier inside the read window and newer than its last proof). A failed
lookup lets the alarm through.
@@ -139,7 +140,7 @@ itself to the floor (R-1052: not yet proven end to end). Setting the global floo
operator's password — there is no scoped route for CC (R-1051). **An agent can be vouched mid-week; the golden stays
weekly** (hub v0.158.0): the R-120 golden guard judges only a CHANGED golden selection, so a Day-0 save that keeps the
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet. **The agent list
offers real agent packages only (R-1011, hub 0.162.0, not yet deployed):** the sha lookup asks for exactly the
offers real agent packages only (R-1011, hub 0.162.0, deployed since; corrected 2026-10-24):** the sha lookup asks for exactly the
`felhom-agent` file (the golden: `golden.tar.zst`) and never falls back to another file, so a step bundle
(`0.156.1-step1`, only the config bundle) is left out of the list, and a vouch of one is refused
(`artifact_file_missing`).
@@ -295,6 +296,12 @@ What remains:
PBS repo and namespace in the **DR recipe** (`dr_recipe`, reported every cycle — R-106 fixed it to match the
backup); the endpoint's PBS fingerprint in **tenantsync** (`hub/internal/tenantsync`); the wrapped key in
**`host_escrow.blob`** (and the identity blob beside it).
**[DESIGN] 2026-10-24 (R-1254; committed felhom.eu `dcf4f84c`, hub next v0.170.0 — NOT deployed):** the escrow PUT
stores the blob, its sealed-password hash and the identity blob in ONE transaction (`Store.SaveHostEscrowWithIdentity`;
the superseded copy is made inside it). A failure leaves all of the old package or all of the new — never the new hash
beside the old identity package, which a box would read as „escrowed" while the written code opened nothing. A
malformed identity blob is refused before anything is stored. (The agent ceremony's „the previous code still works"
after a lost answer, R-1254's other half, stays open.)
Both existing infra-backup tables retire — `infra_backup_versions` (the current/live one, all readers
hit it) **and** `infra_backups` (the deprecated legacy mirror). (The planned slim DR record on `hosts` was never
@@ -326,12 +333,15 @@ customer instead of a single controller.
- **DROP `reports` + CREATE `guest_reports`** (under the clean cutover this is drop+create with no data
migration, not an in-place rename); `guest_reports` adds `guest_id`, `host_id`; `cpu/memory` mean
guest-level; `backup_last_snapshot` goes quiet.
- **ADD** desired-state JSON + `desired_generation` to `hosts`; `desired_spec_json` to `guests`; the
slim DR record (PBS namespace + repo fingerprint + wrapped escrow key) onto `hosts`.
- **ADD** desired-state JSON + `desired_generation` to `hosts`; `desired_spec_json` to `guests`. ~~The
slim DR record (PBS namespace + repo fingerprint + wrapped escrow key) onto `hosts`~~ — retired, never built (§9
above, R-105) *(corrected 2026-10-24, review round 2)*.
- **DROP both** `infra_backup_versions` (current/live) **and** `infra_backups` (legacy mirror) — the DR
recipe, tenantsync and the escrow blob replace them (R-105).
- **KEEP** `customer_configs`, `events`, `customer_notifications`, `notification_log`,
`app_telemetry`, `app_log_issues`.
`app_telemetry`, `app_log_issues`. `customer_notifications.household_saved` (hub 0.165.0, R-1199 reopened) marks a
list the household chose on a box; a box's default never overwrites it, and a box reads it back with `GET
/api/v1/preferences` (customer-scoped like the save). The rule is in `08` (the 2026-10-22 `[DESIGN]` paragraph).
- **Authz cleanup the cutover enables:** several endpoints today use global-or-any-customer-key auth
rather than customer-scoped (the infra-backup GETs, `/notify`). Most retire with the infra-backup
push; any that carry over should scope to the resolved host/guest → customer under §2.
@@ -348,7 +358,9 @@ customer instead of a single controller.
The box's console tells the volunteer to open „az e-mailben kapott link". That mail must exist whenever
a customer is **waiting for a box**, without an operator press. `autoMintSelfBindLink` runs on four
events: **customer creation**; **RESET completion**; **an e-mail set or changed on a customer with no
events (a fifth path mails a link too: the household's „new link" on a USED or EXPIRED link — one mail per customer
per hour, R-719; when the customer still has a linked host it shows the „oldbox" page in the customer's language and
tells the operator instead, R-1203 — `internal/web/selfbind.go` `handleBindResend`) *(corrected 2026-10-24, review round 2)*: **customer creation**; **RESET completion**; **an e-mail set or changed on a customer with no
bound host** (config save); **a host delete** that keeps the customer. The last two re-check that no
host is bound, so a customer who has a box is never sent a pairing link. **Appliance registration is
not a trigger** — it knows no customer (`api/appliance.go`). Every send is stored as a hub-internal
@@ -369,6 +381,7 @@ token id, fingerprint, datastore, tunnel IP, `secret_generation`) with its consu
| **re-issue, descriptor ABSENT, DR flag on** (R-511, v0.114.0) | **adopted**: re-keyed | **rebuilt** from the endpoint's answer; `pbsdr_adopted` audit row |
| host delete | **kept** (tenancy survives) | goes with the host |
| host delete acknowledged through escrow-ack, then a new box | re-keyed automatically (F-14) | rebuilt |
| …and a later install deleted before it made a key, then a new box (D31, operator 2026-10-22; hub 0.167.0 source) | re-keyed automatically — the gate reads the newest deletion of a host that HELD A KEY (`LatestKeyHoldingHostDeletion`) | rebuilt |
| RESET / customer delete | **deprovisioned** — namespace, every backup group AND token | purged |
**Why adopt exists:** a rebuilt box's WG hook refused („the endpoint already holds a PBS token … use
@@ -385,7 +398,7 @@ it has seen any box screen, and until v0.118.0 nothing here described that.
| Mail | Trigger | Rendered by | Language source |
|---|---|---|---|
| Event notification (39 event types) | a box event, or a hub checker | `FormatCustomerEmail` | reported → created-with → `hu` |
| Event notification (44 event types — the `mail.event.*` keys of `internal/i18n/locales/hu.json`, counted 2026-10-24) | a box event, or a hub checker | `FormatCustomerEmail` | reported → created-with → `hu` |
| Claim / reset / re-enroll / claimed | the claim arc | `FormatClaimEmail` | created-with (no box has reported yet) |
| Self-bind link | customer creation, or the operator's button | `FormatSelfBindEmail` | created-with |
| The public bind PAGE at `/bind/<token>` | the customer opens the link | one template per language | created-with, **except `expired`** — see 15.4 |
@@ -452,6 +465,17 @@ warning mails the operator (anything lost, still broken, failed or skipped); inf
The operator mail renders the counts as one `Repair:` line. The household's side is NOT a mail: the lost files (name,
date taken) are shown on its own app page, on its box, until it dismisses them — file names never leave the box.
### 15.4b A box whose tunnel to ep0 is dead — the clock-ahead alarm [hub 0.166.0, deployed with 0.167.0; R-1204 — corrected 2026-10-24]
**[FACT, hub `e267c5e2`]** The host report carries the agent's `wireguard` block (`hostWireguard`). `checkWGTunnel` (`api/host_tunnel.go`, called after `checkHostSwap`) calls the tunnel stale when the box reports `handshake_future_s` > 0 (the clock-ahead lock-out) or a last handshake older than 6 h (healthy boxes measured at most 113–124 s over 24 h). It is edge-triggered against the host's latest tunnel event (`store.LatestHostTunnelEvent`): `wg_tunnel_stale` once on the way down, `wg_tunnel_ok` once on the way back; both are allow-listed and operator-only (`notify.operatorOnlyEvents`, `wg_tunnel_operator_only_test.go`) — the household cannot repair a tunnel. A box with no `wireguard` block (an agent before the field, or no tunnel) raises nothing.
**[FACT, hub `84751cb0`, 0.167.0 source] The repair clears it (D30).** An agent with D30 (`03` §R-1204) registers a new key
after 15 minutes of a future-stamped handshake; the new key's first handshake makes the report fresh, and the alarm
records `wg_tunnel_ok` once (`TestD30_TheAlarmClearsWhenTheNewKeyShakesHands`). `wg_tunnel_ok` is an info event:
stored, not mailed. **Known gaps (rows):** a reboot after the clock was corrected sends no handshake at all and is not
judged (R-1223); the operator's mail cooldown is per customer, not per host (R-1224).
### 15.5 How "the Hungarian did not change" is known
56 goldens captured from v0.117.0 before any string moved, in
@@ -541,14 +565,38 @@ record unreadable: its serve paths answer 500, a save refuses it (never blanks i
to a hub before v0.138.0** needs the columns opened first: `felhom-hub -unseal-box-secrets` (all or nothing; exit 1 =
nothing changed) — run in the running pod, then put the old image back. `guests.api_key` is an unused, always-empty
column and is not sealed. *Decided by CC unattended — operator may reverse (`09` §3 decision 132).*
**Since v0.168.0 (R-975, R-1241) the last plain tokens are sealed too:** the Cloudflare tunnel token, the Cloudflare API
token and the git token live INSIDE `customer_configs.config_json` (`infrastructure.cf_tunnel_token`,
`infrastructure.cf_api_token`, `git.token`). Each of those JSON leaves holds the same `enc:v1:` seal; the rest of the JSON
stays readable. `SaveCustomerConfig` seals, every reader opens through `openCustomerSecrets` (a leaf that does not open is
blanked and marks the record unreadable, as above). Legacy rows are sealed at start-up (`SealLegacyConfigTokens`, without
a `config_version` bump — the box's config is byte-for-byte the same, so no box re-pulls or restarts), followed ONCE by
`VACUUM` + a WAL checkpoint, because an `UPDATE` leaves the old plaintext row in a freed block of the file
(`TestR975_DBFileWithoutKeyRevealsNoToken`). `-unseal-box-secrets` opens these leaves too, so the roll-back above also
covers a return to a hub before v0.168.0. The full list of secret-bearing columns, measured on a copy of the live database
2026-10-23: `audits/hub-token-seal-2026-10-23/README.md`.
**Deployed 2026-10-24 — and what the live file still holds (operator ruling 2026-10-24, accepted).** On the live hub the
start-up `VACUUM` did not finish: the Longhorn disk writes ~1.3 MB/s (the 6-hourly `VACUUM INTO` of 530 MB takes ~7 min)
and the liveness probe stopped the first start after ~90 s; the second start found nothing left to seal (R-1249). So the
rows are sealed, but the live `hub.db` and its `-wal` still carry the old token bytes in freed space (8 of 8, measured).
**Every copy that leaves the volume is clean**: the hub's snapshot and the ep0 copy are written by `VACUUM INTO` (the ep0
copy of 06:30Z: 0 of 8), and the volume is out of DooPlex's Longhorn backup (D40). Accepted, not cleaned: whoever can read
the live volume is root on DooPlex and can read `OFFSITE_SECRET_KEY` there too; a full wipe would also need a hub stop of
8–20 minutes, a disk trim and the volume's Longhorn snapshots deleted. **Rule: a hand copy of `hub.db` for analysis is
made with `VACUUM INTO` (or taken from ep0), never `cp`/`tar` of the live file** — a `tar` of the file carries the old
bytes (`audits/line-test-2026-10-24/hub-0168-seal-check.txt`).
**And the key is the other half:** a backup of the database restores a hub that can open the sealed columns only
with the same `OFFSITE_SECRET_KEY`; today that key exists only on DooPlex (the k8s Secret, and the GPG secrets export
on the same machine). The off-site plan for the database and its key: `runbooks/RUNBOOK-hub-db-offsite-backup.md`
(R-173 — option A decided 2026-10-05, `09` decision 125; §16.3).
### 16.3 The nightly database snapshot (R-173, hub v0.136.0)
### 16.3 The database snapshot — every 6 hours since D34 (R-173, hub v0.136.0; D34 in hub 0.167.0)
Every night at **02:00 Budapest** the hub writes `<data>/snapshots/hub-<UTC stamp>.db` with SQLite's `VACUUM INTO`:
**Corrected 2026-10-24 (review round 2):** since D34 (operator yes 2026-10-23) the hub writes a snapshot at **02:00,
08:00, 14:00 and 20:00 Budapest** (`internal/dbsnap/dbsnap.go` `Slots`), catches up at start when the newest is older
than **6 h** (`CatchUpAge`), and DooPlex pushes the newest at **:30 after each slot** (`felhom-hub-db-backup.timer`),
refusing one older than **7 h** (`felhom-hub-db-backup` `MAX_AGE_H`). The paragraph below is the original nightly
design; its mechanics hold per slot. Originally: every night at **02:00 Budapest** the hub writes `<data>/snapshots/hub-<UTC stamp>.db` with SQLite's `VACUUM INTO`:
one statement, one point in time, every committed write included — the rows still in `hub.db-wal` too, which a file copy
of `hub.db` loses (measured in `internal/dbsnap` tests: a plain copy held 0 of 150 fresh rows). Written as `.tmp`, mode
`0600`, then renamed, so a reader never sees half a file. The newest **2** stay (the volume grew from 1 GiB to 2 GiB
@@ -575,6 +623,14 @@ are deleted at start-up and at each new day (`TestAccessLog_Keeps90DaysThenDelet
ingress log: ≈ 1,900 hub requests a day at ≈ 110 bytes a line → ≈ 0.2 MB a day, ≈ 19 MB for 90 days. A logging
failure never fails or delays a request: lines go through a non-blocking buffer to one writer (`internal/accesslog`).
### 16.5 Daily jobs and the clock change [DESIGN — hub v0.163.0, operator ruling D24 2026-10-21, R-1183]
Every daily hub job (`scheduleDaily`: the four db-snapshot slots 02:00/08:00/14:00/20:00, prune at its configured time, deadline-check 05:00, offsite-key-audit 07:10 — slots corrected 2026-10-24) runs at
its Budapest wall-clock time **once per calendar day**: the next run is the calendar day after (`time.Date`, day+1),
never „now + 24 h", and the slot that ran last never runs again. On a 25-hour day (the last Sunday of October) the old
arithmetic ran every job after 02:00 twice; on a 23-hour day it ran them an hour late. Pinned by `TestR1183_*` for
2026-10-25 and 2027-03-28.
## 17. Operator access tokens [DESIGN — hub v0.152.0, operator ruling 2026-10-15 11:38 (`09` §3 decision 251)]
**The problem.** Scripts — Claude Code's floor step first — drove the hub with the operator's own password (HTTP Basic,
@@ -19,7 +19,8 @@
> decisions; grounded against felhom.eu @ `bf099f6` and felhom-agent @ `4ba1b14` (v0.63.0).
> Evidence base: `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` (all
> measurements cited below as "spike Pn"). This document ends in an implementation slice roadmap
> (§8); **no slice is implemented yet** — each is a separate future task. Consistent with and
> (§8); **no slice is implemented yet** — each is a separate future task. *(Dated note 2026-10-24, review round 2: true when
> written; S1–S4 and S5's safe halves have since shipped — §8 and the agent's `internal/wgtunnel` say what is built.)* Consistent with and
> extending Part 01 (`01-topology-and-trust.md`); where this doc names a component's duty it uses
> Part 01 §3 vocabulary.
@@ -140,7 +141,9 @@ before this ships (the drilled `--rescope-acl` posture is unaffected).
4. **Hub → endpoint VM:** adds `{pubkey, /32}` to the WG server peer list (slice-1 surface).
5. **Box (agent):** next heartbeat sees the generation advance → fetches desired-state → writes
the WG config → `install` + `systemctl enable --now wg-felhom` (the sudoers `*.mount` pattern
of `felhom-agent.sudoers:16-18`, extended to this unit — slice-3).
of `felhom-agent.sudoers:16-18`, extended to this unit — slice-3). *(As built since R-861: no `install` sudoers
line; the conf is installed 0600 to its fixed path by the root wrapper `felhom-priv-apply wg` —
`felhom-agent/configs/felhom-agent.sudoers:223-229`; corrected 2026-10-24.)*
6. **Box:** tunnel handshakes (outbound, keepalive); PBS storage is (re)pointed at
`pbs_tunnel_ip:8007` (§3.4); first backup runs.
@@ -169,7 +172,8 @@ tenant ACL (`DatastoreBackup` on `/datastore/<ds>/<ns>` to BOTH the user and the
### 3.5 Key custody + DR re-establishment
The WG private key joins the **escrowed IdentityBundle**
(`internal/escrow/identity.go:24-27`, today `{TunnelToken, PBSToken}` → `+ WGPrivateKey`) — the
(`internal/escrow/identity.go`, then `{TunnelToken, PBSToken}` → `+ WGPrivateKey`; as built the bundle carries FOUR
fields — `TunnelToken`, `PBSToken`, `WGPrivateKey`, `ResticRepoPassword` — `identity.go:30-40`, corrected 2026-10-24) — the
same R-wrapped, zero-knowledge age blob the DR recipe already consumes. On host-loss
re-provisioning (01 §9 restore mode): the box recovers the bundle with the customer's recovery
code, restores the WG key, and re-establishes the tunnel; hub-side the peer entry still exists
@@ -199,7 +203,8 @@ disables the service, KEEPS its marker, and never re-registers on its own — re
revoked until the operator re-adds the peer (the heartbeat report carries the box's pubkey as
the recovery handle; **re-add via `POST /hosts/{id}/wg` with the global key** — the raw registry
add does not bump the host's generation). The only automatic re-registration is the
pubkey-mismatch (DR/re-key) path, backoff-bounded. Absent DATA (a failed fetch) is never a
pubkey-mismatch (DR/re-key) path, backoff-bounded — and, since D30 (agent v0.163.0), a key the agent makes itself after
a handshake stamped in the future for 15 min (`internal/wgtunnel/rekey.go`; corrected 2026-10-24). Absent DATA (a failed fetch) is never a
teardown signal. **Rollout gate:** `wg_tunnel.enabled` defaults FALSE agent-side until the
production endpoint exists.
@@ -285,7 +290,9 @@ production endpoint exists.
`wg-felhom` AllowedIPs so it survives the agent's self-heal ([OF-1]); PBS unaffected.
- **4.6 Tunnel health → hub.** The tunnel is a storage dependency, so it reports like one: no handshake
within ~3 keepalive periods → the offsite target reports unreachable → the existing alerting path
carries it. No new alarm channel.
carries it. No new alarm channel. *(Since hub 0.166.0 there is one: `wg_tunnel_stale` / `wg_tunnel_ok`,
operator-only — a handshake older than 6 h or stamped in the future, D30 / R-1204, `hub/internal/api/host_tunnel.go`;
corrected 2026-10-24.)*
**EXTENDED 2026-07-05 (TASK H1).** The agent's heartbeat now also carries an **`oob` stanza**
(`felhom_sshd_active`, `felhom_sshd_port`, `reachable`, `config_invalid`, `operator_peer_configured`,
`operator_key_configured`, `wg_handshake_age_s`) — the operator's "can I get into this box right
@@ -297,6 +304,11 @@ production endpoint exists.
## 5. The endpoint VM (the one new managed-infra piece)
**AMENDED 2026-10-24 (R-1247, operator ruling):** a fourth, small thing — `felhom-iperf3.service`, an iperf3 server bound to
`10.77.0.1:5201` and opened in nftables for `wg0` from `10.77.0.0/24` only, so each box's agent can measure its upload
line once a week (felhom.eu `scripts/ep0-iperf3/`; agent `internal/linetest`). It never answers on the internet (proven
2026-10-24, `audits/line-test-2026-10-24/`).
Runs three things, nothing else: the WG server (peer list = hub-written), PBS (one datastore,
per-customer namespaces), and the minimal peer-sync surface the hub drives. **How** the hub
mutates the peer list — a tiny authenticated push API on the VM vs the VM pulling a signed peer
@@ -318,12 +330,16 @@ traverse `/root`) — create datastores under a world-traversable parent (`/srv`
## 6. Trust-model conformance (01 §3/§4 preserved)
- The box **only ever dials out** (UDP to the endpoint; the same posture as hub-poll and
PBS-push, 01 §4/§7). No inbound listener is added to any customer box; the hub still never
initiates into a box.
PBS-push, 01 §4/§7). ~~No inbound listener is added to any customer box; the hub still never
initiates into a box.~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** Two listeners sit on the box's tunnel side, reachable only through
the tunnel: the agent's poke listener (UDP 51822 on the box's WireGuard address — a poke carries no command, the box
still pulls) and, on an appliance install, the operator door `felhom-sshd` (only from the operator's `/32`, §4.5) —
the documented break-glass (`03` §5). The hub still never sends a command into a box.
- The endpoint VM cannot reach into boxes either: it terminates tunnels whose per-peer routes
(§4.5) admit only box→PBS traffic; it holds no box credentials.
- **Out of scope, explicitly:** (a) operator-inbound access to boxes — a *separate* future
problem with its own trust decision; nothing here creates or presumes it; (b) game-server
- **Out of scope, explicitly:** (a) ~~operator-inbound access to boxes — a *separate* future
problem with its own trust decision; nothing here creates or presumes it;~~ operator-inbound access to boxes was
added on 2026-07-05 (TASK H1) and rides this tunnel — §4.5 (corrected 2026-10-25, D45, R-1266); (b) game-server
ingress — public players are not tunnel peers; that is a different ingress problem (public
reachability), not a backup-transport one.
@@ -350,7 +366,8 @@ venue; it gates nothing. Runbook:
**Open sub-decisions (deferred by design):**
- Endpoint peer-sync mechanism (push-API vs signed-manifest pull) — **slice-1 design point** (§5).
- WG key rotation cadence/mechanics — the reconcile channel can carry it (new pubkey → hub →
endpoint swap); design when fleet size warrants, possibly the moment the "separable fleet
endpoint swap); design when fleet size warrants *(one automatic rotation exists since D30: after a future-stamped
handshake for 15 min, at most once an hour — `rekey.go`; a scheduled rotation is still open; noted 2026-10-24)*, possibly the moment the "separable fleet
layer" (D1) conversation reopens.
- Endpoint scaling/sharding (N customers per VM, second region, HU-hosted option) — capacity
planning, not architecture; the peer registry already keys per-host.
@@ -373,6 +390,8 @@ venue; it gates nothing. Runbook:
| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; ~~§4.3 MTU decision~~ **RESOLVED (client MTU 1280, agent v0.65.0, §4.3)**; residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially |
**MVP that puts offsite backup in a real customer's hands = S1 → S2 → S3; then S4.** S5/S6
complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — today a
read-only prober that names itself "the seam for the tunnel-management slice") adopts the
S3-piloted agent-managed-service pattern in a later, separate slice.
complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — then a
read-only prober that named itself "the seam for the tunnel-management slice") ~~adopts the
S3-piloted agent-managed-service pattern in a later, separate slice.~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** This plan is withdrawn:
cloudflared is a GUEST container that the controller renders and keeps up; the agent only reads it (four states,
`cloudflared.go:12-23`) and will not manage it — `03` §2/§3 wins.
@@ -253,7 +253,7 @@ It carries guest sizing, `pve_storage[]`, and an app half with per-app `storage_
the identity bundle's shape is `{tunnel_token, pbs_token, wg_private_key, restic_repo_password}`
(`felhom-agent/internal/escrow/identity.go:26-39`).
**[DESIGN] Off-site deletion custody (decisions 68–69, 2026-10-03) — BUILT hub v0.127.0 / controller v0.289.1, live on both demo boxes.** The restic repository password stays on the box only. The box's off-site key is append-only (pinned in `authorized_keys`, written by the hub — the key registrar); the box never receives the sub-account password, which the hub stores encrypted at rest. Old snapshots are pruned by the box itself, only in a weekly window the hub opens, behind a fake-snapshot guard (R-822); the window has run live on both demo boxes since 2026-10-05 (R-95, closed). The guard's residual — past-dated fakes steering the keeps, and a window check that trusts the box's own counts — is R-822 and R-895. **Hub v0.158.0 (built 2026-10-18, deploy waits on the operator) adds the hub's own check:** the 07:10 key audit compares the repository's `snapshots/` listing, over the hub's own login, with yesterday's — files gone that no clean-up window explains raise `offsite_snapshots_vanished` (error), files no reported box run explains raise `offsite_snapshots_unexplained` (warning); both operator-only (R-895 slice 2). A Felhom-side pruner holding repository passwords is rejected.
**[DESIGN] Off-site deletion custody (decisions 68–69, 2026-10-03) — BUILT hub v0.127.0 / controller v0.289.1, live on both demo boxes.** The restic repository password stays on the box only. The box's off-site key is append-only (pinned in `authorized_keys`, written by the hub — the key registrar); the box never receives the sub-account password, which the hub stores encrypted at rest. Old snapshots are pruned by the box itself, only in a weekly window the hub opens, behind a fake-snapshot guard (R-822); the window has run live on both demo boxes since 2026-10-05 (R-95, closed). The guard's residual — past-dated fakes steering the keeps, and a window check that trusts the box's own counts — is R-822 and R-895. **Hub v0.158.0 (built 2026-10-18; deployed the same day on the operator's yes — corrected 2026-10-24) adds the hub's own check:** the 07:10 key audit compares the repository's `snapshots/` listing, over the hub's own login, with yesterday's — files gone that no clean-up window explains raise `offsite_snapshots_vanished` (error), files no reported box run explains raise `offsite_snapshots_unexplained` (warning); both operator-only (R-895 slice 2). A Felhom-side pruner holding repository passwords is rejected.
**[DESIGN, built hub 2026-10-07 — R-32 option A, `09` §3 decision 167] What RESET does to the household's off-site
copy.** On the shared pool box a sub-account is a LOGIN, not the data: deleting it leaves its folder (measured
@@ -380,7 +380,7 @@ never moved by an update or an undo, and the unit restore accepts it.
| Tier | Location | Captures | Cadence (LIVE) | Retention (LIVE) | Encrypted |
|---|---|---|---|---|---|
| **Tier-1** recovery unit | `<nsRoot>/backups/primary/<app>/` on the app's own drive | compose + `.felhom.yml` + **secret-stripped** `app.yaml`, `db-dumps/*.sql`, `volume-dumps/*.tar`, `manifest.json` | nightly at W, plus a checksum-gated refresh on the periodic status pass | one point per app (`restore_points.go:14-18`) | no |
| **Tier-1** recovery unit | `<nsRoot>/backups/primary/<app>/` on the app's own drive | compose + `.felhom.yml` + `app.yaml` (0600 — since D5 / schema 2 it carries the PORTABLE secrets, `recovery_unit.go:25`; „secret-stripped" corrected 2026-10-24), `db-dumps/*.sql`, `volume-dumps/*.tar`, `manifest.json` | nightly at W, plus a checksum-gated refresh on the periodic status pass | one point per app (`restore_points.go:14-18`) | no |
| **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no |
| **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) |
| **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/felhom` (its `docker/` and `sys_drive/` folders, bound in the guest to `/var/lib/docker` and `/mnt/sys_drive` — §7.5.1; `mp1` no longer exists, corrected 2026-10-15) | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` → `--prune-backups keep-daily=3`: one copy per day, 3 days (keep-last until agent 0.160.0, R-1149) | **no** |
@@ -437,7 +437,7 @@ container must keep its id). Ring 1 never takes it in the night leg. Measured: t
**[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed
offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered
(`cmd/controller/main.go:604-607`). Both boxes run W = `02:30`.
(`backupwindow.LegTimes`, called at `cmd/controller/main.go:1258-1264`; line corrected 2026-10-24). Both boxes run W = `02:30`.
**[FACT, controller v0.271.0, 2026-09-25] A fourth leg: automatic app updates, CHAINED after the off-site
leg** (`09` §3 decisions 11 and 20, §6.4 part 7). The `offbox-backup` job runs the update leg when its off-site
@@ -448,8 +448,10 @@ night the whole-guest backup starts later inside its own window; it is never ski
numbers: `audits/DRILL-night-2026-09-25.md` Part D.
**[FACT, 2026-09-25] demo-hp's local whole-guest target is its host ROOT disk; retention there is now
keep-last=1** (operator ruling 2026-09-24, option A; R-684). PVE prunes AFTER a successful backup, so the target
must hold keep-last + 1 archives during the run — measured: a 9201 archive is 8.18 GB, the root disk 39 GB. The
keep-last=1** (operator ruling 2026-09-24, option A; R-684) — *since R-1149 (operator D17, 2026-10-19) the local tier's
spec is `keep-daily=N` (`felhom-agent/internal/config/config.go:553-559`), so demo-hp is keep-daily=1; corrected
2026-10-24*. PVE prunes AFTER a successful backup, so the target
must hold N + 1 archives during the run — measured: a 9201 archive is 8.18 GB, the root disk 39 GB. The
warning that should precede such a failure is R-685.
**[FACT, 2026-10-14] A power cut during the whole-guest backup leaves a half-written archive, and from agent v0.156.1 the
@@ -647,6 +649,8 @@ leg** — LIVE on demo-felhom, `backups/secondary/bookstack/` and `.../docmost/`
`recovery-unit` and nothing else, at 156 MB and 86 MB (INV Part B.2). Since v0.183.0 the button
refuses **before** stopping the app and names the action that works.
**[DESIGN] A template may leave a named volume out of every backup (2026-10-22, operator D21 = B, R-49; controller 0.340.0).** `.felhom.yml` `backup.volumes: [{name: <volume>, class: excluded}]` — the name as the compose file declares it; the controller dumps every named volume of the app EXCEPT `<project>_<name>` (`appbackup.ExcludedDockerVolumes` → `backup.Manager.dumpableVolumes`), so the volume leaves the local unit, the second-drive copy and the off-site copy at once, and the next run's stale clean-up removes its old tar. Only class `excluded` acts; a name the compose file does not declare is ignored and logged; **a list that would exclude every named volume of the app is refused whole** (nothing excluded) — a template typo must not empty a backup (`TestR49_*`). Use it only for data the app rebuilds by itself: the first user is Immich's `immich_ml_cache` (model weights, re-downloaded at start) — 9202: unit 1.16 GB → 337 MB; with the volume removed, Immich started and its ML answered after 9.2 s (`audits/night-2026-10-22/P2-D21/`). A controller older than 0.340.0 ignores the key (inert). The raw database folder and Immich's own `upload/backups/` stay in until a restore from the dump alone is proven (D21's option A).
### 6.3 Where capture and restore disagree
**[FACT]** Three asymmetries, each source-cited:
@@ -729,13 +733,76 @@ definition is written or a named volume replaced — a failed placement, volume
the live definition back and HOLDS the app stopped (`restore_mixed`, operator-cleared); the household reads that it
needs our help; the operator gets a `backup_run_failures` line. The hold is persisted before the stop. Placed files
ALONE (no version change, no volume replaced) still restart after a successful rollback (`TestR379_ScenarioA`): that
mix, and the undo of everything, is option A — „put back exactly as it was", the next slice on R-893. **`[FACT]`
mix, and the undo of everything, is option A — „put back exactly as it was", the next slice on R-893 *(shipped since:
D20 in controller v0.336.0 and D22 in v0.338.0, below — noted 2026-10-24)*. **`[FACT]`
2026-10-19 (measured on 9202, controller 0.335.0, `audits/night-2026-10-19/P3/`):** a forced replay failure after the
volume was replaced left the NEWEST rows (the undo dump) on top of the backup's volume — a table only the backup had
stayed (37 tables, the live app had 36); the definition did not move (same version); the app was held and a start
answered 409. The restore page's „the database is back as it was" is false for it. Design note with the two options and
the pick (A: a copy of the app's live volumes, put back on failure — one copy's disk for the restore's length, a fit
check before the stop): `audits/night-2026-10-19/P3/design-R-893.md`; the decision is on `STATUS.md`.
**`[DESIGN]` 2026-10-20 — „put back exactly as it was" (R-893 option A, operator decision D20 = A, `09` §3 decision 192
second half; controller 0.336.0 is the floor).** Before the stop, every named volume the snapshot will replace is measured
and its copy is added to the restore's own free-space need (R-357's exact rule); too little room or an unmeasurable volume
refuses with nothing stopped. Right after the stop the volumes are tarred into `<namespace>/backups/restore-undo/<app>/`
(outside every capture set). On ANY failure after a volume or the definition moved, the copies are poured back, a volume
the snapshot added is removed, the live definition is written back and the app is STARTED; the household reads that the
restore failed and nothing changed (with placed files: its database and volumes are back, the copied files stay); the
operator gets `backup_run_failures` leg `restore-put-back`. The hold (`restore_mixed`) remains only for a put-back that
fails — then the copy is kept for support — and the double failure (replay AND rollback) holds as before. On success the
copy is removed — **since controller 0.343.0 (fix A, 2026-10-25) only after a STABLE start, and the verdict and the release
are two different tests (§8.2e):** the verdict „started" needs the stack running on every sample of a 45 s window, no
container restart in it and every healthcheck `healthy`; the copy is released only after a further 3 minutes of the same
(10 s samples, watched up to 30 minutes; a new restore of the app ends the watch and leaves the copy to it). A restore that
passes the verdict and then flaps keeps its copy. **`[FACT]` 2026-10-20 (9202, controller 0.336.0-rc.1, `audits/night2-2026-10-20/P2/live/`):** the same
forced failure as on 2026-10-19 → 36 tables, the backup-only table absent, schema/data/row-count md5 equal to before,
the app running, no hold, the copy (69 MB, 0.9 s) gone; a normal restore of the same snapshot then worked. The R-893
known limit above is closed for named volumes; placed FILES are still not taken back (the plain-files mix, as ruled).
**`[DESIGN]` 2026-10-21 — the put-back covers EVERY restore, and a power cut (operator ruling D22 = yes, 2026-10-21
20:05; released in controller v0.338.0 — corrected 2026-10-24).** (1) **The local unit restore** — the app's own unit, the second drive's unit,
the second drive's „Teljes visszaállítás" and a kept load all run `RestoreFromRecoveryUnitAtWith` — now takes the same
measure → copy → put-back (`offbox_restore_undo.go`, one implementation, no second copy of the logic): the copy must fit
beside the unit before anything stops; ANY failure after the stop (volume replay, definition write, DB-only start, DB
replay, start) pours the live volumes back, writes the live definition back and starts the app; a put-back that fails
holds and keeps the copy; with no copy (a unit with no volume archive) a half-done DB replay HOLDS (the R-379 ruling)
instead of starting on it. The window carries the app-stop marker (`unit_restore`). (2) **A restore cut by a power loss
or a controller kill** (R-1195): once the copy is whole the restore writes `undo.json` and the live definition files
beside it and records the folder in the app-stop marker; at start the crash recovery pours it back BEFORE it restarts the
app, and the household reads „A visszaállítás félbeszakadt (áramszünet). Az alkalmazás adatai a visszaállítás előtti
állapotban vannak." A cut before the copy was whole restarts as before (nothing had moved). Local and off-site alike.
(3) **The second drive's „Teljes visszaállítás" asks the unit's refusals before its file half** (R-1178), so a refusal
moves nothing and its „érintetlenek" is true; a later refusal starts the app and names the changed files. (4) **The
night's second-drive mirror skips an app while a restore of it runs** (R-1180). **`[FACT]` 2026-10-21 night (9202,
controller 0.338.0-rc.1, `audits/night-2026-10-21/P3-restore/`):** Rallly + Postgres; a damaged unit volume → the live
volume poured back, the newest row still there (A,C), the app running, „…Nem változott semmi…" (on 0.331.0 the same case
crash-looped Postgres and lost the row, C3 2026-10-20); the controller killed (kill -9) right after the volume step →
on start „interrupted restore is put back … 1 volume(s) poured back, definition=true", markers A,C,D (D = the row newer
than the unit), the app running, the record reads the D22 sentence. Known limit kept: a database whose data is NOT in a
named volume the unit replaces (none in the catalog today — a drive-file app's unit restore is refused by R-538) is not
covered by the volume copy.
**`[DESIGN]` 2026-10-24 — a pending put-back blocks EVERY start (R-1251; review round 2 HUNT-1 H1, H2, H7; committed in
controller `bb2809d` + `062f9b4`, unreleased — rides the next controller release).** For a drive-backed app the copy lives on
the app's drive, and at start the drive is often not bound yet. A copy that cannot be reached while its drive is not
bound is **waiting, never „gone"**: the crash recovery writes a restore hold (`putback_pending`, the copy's folder
in it) and starts nothing. The hold is the one store every start path already honours — the household's
Start/Restart/Update, the boot recreate, the drive return, the media restart, the boot sweep, the night chain, the
update leg — so no path has its own check. When the drive binds, the put-back runs FIRST (the boot recreate and the
drive-return restart; the start and every drive return sweep the waiting holds), the hold is lifted, then the app
starts. The household reads „…A visszaállítás előtti adatai egy meghajtón vannak, amely most nem érhető el…" on the
app and „Ne indítsd újra a visszaállítást…" in the restore record. A put-back that FAILS (at start or when the drive
returns), or a copy gone while its drive is bound, leaves the app **held** with the D20 hold („félúton megszakadt"),
the copy kept, and the marker lets go of the app — never stopped-and-unheld (H2). Stop pressed during the restore still
gets the put-back; only the start is skipped (H7). A removal clears a waiting put-back; a later restore that completes
drops it (the household chose that data). The copy's folders are synced to disk BEFORE the marker names the copy.
**`[FACT]` 2026-10-24 night, Tester 1 (VM 341; MeTube on the data drive, 1.5 GB in its volume;
`audits/restore-holes-2026-10-24/P1-tester1/`):** a local restore cut by powering the VM off 2.1 s after „the restore
copy is recorded", the data disk taken off, the box booted: held as waiting, Start and Update refused (409, the
sentence); the disk plugged back: the agent re-mounted it, the boot recreate put the copy back BEFORE the start, the
household's newest data („gen2") and its download on the drive were there, and a later controller restart poured
nothing. Run 1 of the same proof found the journal left as `undo.json.tmp` (the rename lived only in the page cache) —
the app was held for support, never started on the cut data; the folder sync above fixed it, run 2 passed.
**[DESIGN] 2026-08-22 — the failure ladder of a database restore: replay → rollback → hold.**
Recorded here rather than only in a closed register row, because a decision that survives only inside
@@ -847,6 +914,8 @@ run makes it longer.
### 6.5 Kept data — what a removed app leaves on the drive (controller v0.274.0, `09` §3 decision 36)
*Second-drive copies that are KEPT (D25) and the rule that a rejected backup block never deletes one: §8.2a (R-1252, 2026-10-24).*
**What it is.** "Remove the app, keep my data" leaves the app's drive folder (`<drive>/appdata/<app>`, or the folder
its definition binds through `${HDD_PATH}`) in place. A reinstall over it no longer runs silently into the old files
(R-657): the install asks „A megőrzött adataimat használom" / "Use my kept data" (the database from the newest copy of
@@ -872,6 +941,26 @@ holds no database, or **does not record its data version** (§6.6 — a unit wri
a mixed or mismatched unit is refused by §6.6's own rule. Then a dated folder's files move back and the one unit restore
runs. The downloaded copy is removed on every path. A failed download or a refusal leaves the kept files as they were.
**[FACT] 2026-10-25 — why the „another drive" check let another box's copy through (R-1271; catalog sweep 2026-10-24,
Tester4: Paperless-ngx, Nextcloud, Immich).** The check (`KeptCopyAt`, `internal/backup/kept_load.go`) compares the
unit's `app.yaml` `HDD_PATH` with the drive the kept files sit on. That is a mount PATH, and every box mounts its first
data drive under the same name (`/mnt/felhom-drives/hdd_1`) — so a copy taken of another box's disk matches. It is a
path check, not a physical identity, and it stays one: loading an old box's data on a new box is what off-site is for.
What chose the wrong copy was the DATE: the rehearsal box's clock had been set to 2026-10-28, its off-site snapshot
carried that time, this box had no data time of its own to cap it (`offsiteDataTime`), and „newer than every local
copy" took it over the family's own copy of the same evening. Immich's 20 minutes of 404 were the ROUTING, not a half
load: the unit brought the old box's `SUBDOMAIN` (`photos`), so the app answered at `photos.<domain>` and the probe
asked `immich.<domain>`; its containers were healthy.
**[DESIGN] 2026-10-25 — a copy dated in the future is never „the newest" (R-1271; brief 2026-10-25 Part 4, the safest
reading; controller `bd9f4f7`, unreleased, rides v0.343.0).** A copy whose date lies more than **10 minutes** past this box's clock (`KeptClockSkew`: NTP keeps a box
within a second; ten minutes covers a box that synced late after a power cut) is future-dated. It never counts as the
newest: the newest copy with a true date wins (a tie goes local); the household's own local copy wins over any
off-site copy that is not newer by a true date. A future-dated copy is offered only when no other copy exists — then
the question says its date is in the future and does not recommend it (the „use" button is not the primary one). The
question always names the copy it will use. An off-site copy is never refused only because another box of the
household made it.
**It is NOT backed up.** No tier captures `<drive>/kept` or a leftover `appdata/<x>` of a removed app; the page says
so („Erről nem készül mentés." / "This is not backed up."). What brings it back into an app is the unit it carries
(or the app's own unit on the drive), listed per row as „Visszatölthető innen" / "Can be loaded from".
@@ -935,6 +1024,7 @@ both. Files written under different pins (a failed leg) make the unit `mixed`.
|---|---|---|
| own unit (Tier 1), second-drive mirror (Tier 2), kept-data Load | present, `compose/` names the data's pins | the data starts under that definition; if it differs from what ran, the restore pages' first sentence names the backup and the version (the kept-data Load has its own sentence and does not add this one) |
| same | present, `compose/` names OTHER pins, or `mixed` | **refused before anything is touched** (`ErrUnitVersionMismatch`): „…ezért a visszaállítás nem indult el. Az alkalmazás érintetlen" |
| off-site (Tier 3) | present, `mixed`, or a definition that does not belong to its data | **refused before anything is touched** like the rows above (`unitVersionCheck`, `offbox_reconstitute.go:826-833`; row added 2026-10-24) |
| off-site (Tier 3) | present, the snapshot's version differs from what runs | the snapshot unit's definition is written into the stack dir (and pinned) right after the stop, before any file, volume or database is touched; the database service is resolved from it |
| any | absent — a unit or snapshot written before v0.275.0, or with an unstamped file | restores **as before** (its captured definition; the off-site path the live one), with a WARN naming it |
@@ -961,13 +1051,28 @@ requires a database dump written after the conversion whose recorded engine is t
image is re-pulled on restore. A version its maker has deleted cannot start. Measured 2026-09-26: the 42 ladder
digests all resolve today (`audits/version-travel-2026-09-26/A7/`). **Accepted as a known limit — operator ruling
2026-10-12 (S8, `09` §3 decision 203; R-698 closed).** What a household meets: a restore whose version the maker has
deleted stops at the image pull, the app stays as it was, and the household uses another copy or a newer version.
deleted stops at the image pull, the app stays as it was (since D48 by ORDER — the pull comes before the stop; before it,
by the D22 put-back), and the household uses another copy or a newer version.
The operator's follow-up — *„will need to mirror the apps we support later, just to be sure"* — is **R-933**: mirror
the images of every catalog app we support into our own registry, before scale. Until it is built, this limit holds.
**Since decision 53 (`09` §3, 2026-09-30 evening):** a box keeps only the image each app service runs and the one before
it; older images of the app are deleted. A restore that needs an older version re-pulls it — as before; the limit
above is unchanged, and kept data (decision 40) is not touched by the image clean-up.
**[DESIGN] 2026-10-25 — a restore downloads first (operator ruling D48, `09` §3 decision 257; controller `2d71270`,
unreleased — rides v0.343.0).** „The app stays as it was" in the paragraph above is now true BY ORDER, not by the D22
put-back. After the existing refusals and before the first stop or change, every restore path resolves every image the
definition that will RUN names — present on the box, or pulled now — under the image-work lock held from the first
check until the restore's last start (R-863: the image clean-up cannot delete a pulled image before the start). A pull
that fails is a REFUSAL: the app is not stopped, its data is not moved, and the household reads which version, and
why: „Ennek a mentésnek a verzióját (%s) nem lehet letölteni: a készítője már nem kínálja…" when Docker says the version
does not exist (`manifest unknown`, `not found`, `pull access denied`), „…most nem lehet letölteni… Próbáld újra
később…" for everything else (a busy or unreachable registry). The operator gets one `backup_run_failures` line (leg
`restore-image-unavailable`). The paths: the local unit restore — and with it the second drive's unit restore, the
second drive's whole restore (its `CheckOnly` pass carries the pull, so its file half never runs first), a kept-data
load from a unit and from off-site, and a set-aside unit (D47) — the off-site restore, the second drive's file restore
and the no-unit fallback. Not covered: the `.fab` import — it removes the running app's volumes before its definition is pulled (R-1279).
### 6.7 Why an off-site run failed — the failure classifier (R-571)
**[FACT, read from source 2026-10-05, felhom-controller `114ff27`]** When an off-site (restic) run fails, the
@@ -1038,6 +1143,14 @@ here — §8 has rows where it is the actual state.
### 7.0 What a customer can and cannot do ALONE — the four steps the drill found
> **[DESIGN] 2026-10-21 — the box death, alone (operator ruling, hub v0.163.0, R-1176).** After the operator's one press
> (delete the dead host — D19), the household reinstalls, binds, claims, and the rebuilt box offers „A korábbi, házon
> kívüli mentéseid visszaszerezhetők" and asks for the recovery code: the hub carries the sealed package to the new
> host (03 §8a, „the CARRY"). A wrong code is refused and changes nothing; the right code unlocks the history; each
> app is reinstalled and brought back with „Teljes visszaállítás"; the night's off-site run appends to the same
> history. Measured end to end 2026-10-21 (`audits/fixes-2026-10-21/P1-live/`). Before this ruling the rebuilt box saw
> no package, minted a fresh key and set the history aside on its first night (rehearsal 2026-10-20 B7).
> **[FACT] Added 2026-08-05.** The 2026-08-04 R-201 drill (`audits/DRILL-r201-night-run-2026-08-04.md`)
> is the first end-to-end proof that a customer's file survives a machine rebuild and comes back
> byte-identical. **It passed with an operator present, and four manual interventions stood between
@@ -1295,12 +1408,13 @@ Added 2026-08-02 from `audits/SPIKE-recovery-unit-space-2026-08-02.md` and
which is true and is the whole story for a drive-resident app. **It is not the whole story for an app
with no data drive, and that case was undocumented until now.**
**The unit is RETAINED, not staged.** `GetAppDrivePath` (`felhom-controller/internal/backup/backup.go:245-255`)
**The unit is RETAINED, not staged.** `GetAppDrivePath` (`felhom-controller/internal/backup/backup.go`, now near line 441)
returns the app's `HDD_PATH` if it has one and **`systemDataPath` otherwise** — what
`internal/appbackup/paths.go:26-27` calls *"the SSD-only system-data fallback"*. Nothing deletes a unit
after it is copied onward: the only prune is F5 (`backup.go:1053-1112`), which removes residue on an
after it is copied onward: the only prune is F5 (`pruneStalePrimaryDirs`, `backup.go`, now near line 1514), which removes residue on an
**old** drive when an app **moves**. So for every app without a data drive, `mp1` holds the kept copy
indefinitely.
indefinitely. *(Dated note 2026-10-24: `mp1` no longer exists — §6.1 / line 386; the system data now lives under the
one `mp0` volume's `sys_drive/` folder, `felhom-agent/configs/build-golden.sh`.)*
**What a unit contains, which bounds the problem** (`internal/backup/recovery_unit.go:20-25`):
`compose/` + `db-dumps/` + `volume-dumps/` (named-volume tars) + `manifest.json`. **It never contains
@@ -1449,7 +1563,10 @@ crosses the line — **R-158**.
> Eleven rows carry a blank RTO — 4, 5, 8, 9, 10, 11, 11b, 12, 13, 14, 15 — and only six of those are
> deferred work. **Row 5** has no recovery to time (a derived copy), **row 13** has no route by
> design, **row 14** is proven and merely never stopwatched, **row 11b** is a note not a row, and
> **row 15 is an open DEFECT (R-104) that this line does NOT cover.**
> **row 15 was an open DEFECT (R-104) that this line did NOT cover** — *R-104 was CLOSED 2026-10-05 (controller
v0.297.0: a lock that survives the self-heal gets the class `locked`, its own note and a retry, `offbox.go:156-162`;
CLOSED-ITEMS.md; corrected 2026-10-24).* **Row 16** (added 2026-10-25, D44 #6, R-1265 — after this banner) has a blank RTO
too: a registry rebuild from the ep0 release copy was never timed.
>
> **NO STATUS MOVED ON THE DAY THIS WAS WRITTEN, because nothing was proven that day.** A stopping
> line that promotes a row is a stopping line that lies. Full text, and the conditions that reopen
@@ -1482,13 +1599,14 @@ crosses the line — **R-158**.
| 7 | **Guest stopped and does not come back** | everything | guest-power watchdog (60 s, `onboot` as the deliberate-stop discriminator) | automatic | **120 s** | | **PROVEN** | agent v0.107.0 replay — 120 s unattended vs the incident's 587 s with a human |
| 8 | **Host dies (hardware), drives intact** | the data drives; ep0's PBS namespace; the hub's Recipe + Escrow | **install a new host, then** `--selftest=bring-up -mode dr` per guest, **then** re-attach drives by `durable_id` | **operator** (SSH) | | 7 d | **IMPLEMENTED** **`[BETA-DEFERRED]`** | bring-up code exists and has **never been executed** (`CAMPAIGN-8…:522`); the drive half of the plan is empty on every box (**R-105**) |
| 9 | **Whole box lost (fire/theft) — host and drives gone** | ep0 PBS namespace; the restic repo; the hub's Recipe + Escrow | new hardware → day-0 → escrow-consume with **R** → restore guests from PBS → app data from Tier-3 | **operator + customer** (R) | | 7 d (guest) · 24 h (app data) | **IMPLEMENTED / UNPROVEN** **`[BETA-DEFERRED]`** | every leg exists; the composed path has never been run. The destructive S5 drill is operator-gated and unrun (`06-offsite-connectivity.md:327`) **`[FACT]` 2026-10-03 (decision 70): ep0's datastore now has a nightly copy on DooPlex** (PBS pull-sync `ep0-felhom-offsite`, `remove-vanished false`, weekly verify, failures mailed to the operator); first pull 201 s, 12 GB, 4 of 4 snapshots, matching ep0 namespace for namespace. Restore route: `runbooks/ep0-datastore-copy.md` — never walked. `audits/offsite-lock-build-2026-10-03/partF/`. **`[FACT]` 2026-10-04: the restore route from the DooPlex copy was WALKED** on demo-hp (scratch VMID): listed in 2 s, restored in 186 s (15 GB logical), data read by `pct mount`; the restored config carries `onboot: 1` and the host's real drive binds — strip them on a restore beside the original (R-834). The copy now keeps 8 weekly copies (decision 71). `runbooks/ep0-datastore-copy.md` route 1. |
| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** **`[BETA-DEFERRED]`** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. **2026-09-01 (SPIKE R-95, `audits/SPIKE-r95-offsite-delete-2026-09-01.md`) — THIS ROW'S STATUS IS UNCHANGED, but the seven-day bound beside it is now known to be unverified.** Measured on BOTH boxes over their own SFTP credential, with controls: **no `.snapshots` is visible to either sub-account**, and the account is jailed. Either none exist or a sub-account cannot see them — and a snapshot the box cannot see is one the box cannot restore from, so recovery would be an operator act at the Hetzner panel, not a product capability (R-429; the confirming field needs the provider API, fenced by §11-D). **The PBS shape does NOT transfer**: the sub-account API has one permission axis, `readonly`, and a backup target cannot be read-only — PBS is a server that can refuse, a Storage Box is a filesystem that runs nothing. **What the spike removed as a fear:** withdrawing delete does NOT wedge the store (measured — restic treats a dead owner's lock as stale and proceeds). **What it added:** `unlock --remove-all` reports success while deleting nothing (R-430). **2026-09-01, LATER THE SAME DAY — THE PROBE ABOVE LOOKED FOR THE WRONG NAME AND THIS ROW'S SECOND CLAUSE IS NOW HALF WRONG.** It searched `.snapshots`; the vendor documents `/.zfs/snapshot`. Re-probed at the documented path on BOTH boxes, with controls, the two halves separate cleanly: **(a) the LIVE repository is deletable by the box — unchanged, R-95 stands;** **(b) the daily SNAPSHOTS of it are not writable by anything — PROVEN, not cited:** a write into `/.zfs/snapshot` is refused (`dest open …: Failure`) while the identical write to the account home succeeds. Seven daily snapshots confirmed in the panel (R-429). **So this row's "the restic repo is NOT protected the same way" is true of the repository and FALSE of its snapshots** — a deletion costs at most the day since the last snapshot, recoverable per-file (vendor). **Two limits kept honest:** a sub-account sees the snapshot directory EMPTY, so per-file recovery is operator-only today (R-432); and a panel-driven restore rolls back the WHOLE box. **The status is NOT moved** — evidence (b) is measured, but the recovery ROUTE has never been walked, which is what PARTIAL means. **Detection shipped hub v0.111.0 (R-431):** an unexplained fall in the snapshot count is noticed within a day — **but only a fall of MORE than half (R-435).** **2026-09-01, THE DRILL — CLAUSE (b) OF THE RE-SCOPE ABOVE IS WITHDRAWN; THE STATUS IS STILL NOT MOVED.** The re-scope said a deletion is *recoverable per-file*. **It is not, from the box: no snapshot is reachable by ANY name.** MEASURED on demo-hp, read-only, no delete verb issued: **777,600** exact names in the vendor form `YYYY-MM-DDTHH-MM-SS` (nine full days, second granularity) plus 126 alternative shapes — **zero hits**, against a control where the identical 600-name batch returns a path that exists (6/6). **Structural cause:** `/home` (`u629488-sub3`) is **st_dev 0,82**, `/.zfs/snapshot` is **st_dev 0,276**, and `/home/.zfs` does not exist — a snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding the repository. **Clause (a) — the box cannot WRITE into the snapshot area — is unchanged and re-confirmed.** So this row's *"recoverable per-file (vendor)"* and its limit *"per-file recovery is operator-only today (R-432)"* both overstate what exists: the remaining routes are a panel rollback of the WHOLE Storage Box and the provider API (fenced, and the hub's client has no snapshot method at all). **R-432 ANSWERED negatively; R-433 opened. RTO still blank — nothing was recoveredLine truncated
| 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its Longhorn volume backup | operator (`kubectl`) | | 24 h + weekly (Longhorn `RETAIN 1` each) | **UNPROVEN** **`[BETA-DEFERRED]`** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) |
| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** **`[BETA-DEFERRED]`** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. **2026-09-01 (SPIKE R-95, `audits/SPIKE-r95-offsite-delete-2026-09-01.md`) — THIS ROW'S STATUS IS UNCHANGED, but the seven-day bound beside it is now known to be unverified.** Measured on BOTH boxes over their own SFTP credential, with controls: **no `.snapshots` is visible to either sub-account**, and the account is jailed. Either none exist or a sub-account cannot see them — and a snapshot the box cannot see is one the box cannot restore from, so recovery would be an operator act at the Hetzner panel, not a product capability (R-429; the confirming field needs the provider API, fenced by §11-D). **The PBS shape does NOT transfer**: the sub-account API has one permission axis, `readonly`, and a backup target cannot be read-only — PBS is a server that can refuse, a Storage Box is a filesystem that runs nothing. **What the spike removed as a fear:** withdrawing delete does NOT wedge the store (measured — restic treats a dead owner's lock as stale and proceeds). **What it added:** `unlock --remove-all` reports success while deleting nothing (R-430). **2026-09-01, LATER THE SAME DAY — THE PROBE ABOVE LOOKED FOR THE WRONG NAME AND THIS ROW'S SECOND CLAUSE IS NOW HALF WRONG.** It searched `.snapshots`; the vendor documents `/.zfs/snapshot`. Re-probed at the documented path on BOTH boxes, with controls, the two halves separate cleanly: **(a) the LIVE repository is deletable by the box — unchanged, R-95 stands;** **(b) the daily SNAPSHOTS of it are not writable by anything — PROVEN, not cited:** a write into `/.zfs/snapshot` is refused (`dest open …: Failure`) while the identical write to the account home succeeds. Seven daily snapshots confirmed in the panel (R-429). **So this row's "the restic repo is NOT protected the same way" is true of the repository and FALSE of its snapshots** — a deletion costs at most the day since the last snapshot, recoverable per-file (vendor). **Two limits kept honest:** a sub-account sees the snapshot directory EMPTY, so per-file recovery is operator-only today (R-432); and a panel-driven restore rolls back the WHOLE box. **The status is NOT moved** — evidence (b) is measured, but the recovery ROUTE has never been walked, which is what PARTIAL means. **Detection shipped hub v0.111.0 (R-431):** an unexplained fall in the snapshot count is noticed within a day — **but only a fall of MORE than half (R-435).** **2026-09-01, THE DRILL — CLAUSE (b) OF THE RE-SCOPE ABOVE IS WITHDRAWN; THE STATUS IS STILL NOT MOVED.** The re-scope said a deletion is *recoverable per-file*. **It is not, from the box: no snapshot is reachable by ANY name.** MEASURED on demo-hp, read-only, no delete verb issued: **777,600** exact names in the vendor form `YYYY-MM-DDTHH-MM-SS` (nine full days, second granularity) plus 126 alternative shapes — **zero hits**, against a control where the identical 600-name batch returns a path that exists (6/6). **Structural cause:** `/home` (`u629488-sub3`) is **st_dev 0,82**, `/.zfs/snapshot` is **st_dev 0,276**, and `/home/.zfs` does not exist — a snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding the repository. **Clause (a) — the box cannot WRITE into the snapshot area — is unchanged and re-confirmed.** So this row's *"recoverable per-file (vendor)"* and its limit *"per-file recovery is operator-only today (R-432)"* both overstate what exists: the remaining routes are a panel rollback of the WHOLE Storage Box and the provider API (fenced, and the hub's client has no snapshot method at all). **R-432 ANSWERED negatively; R-433 opened. RTO still blank — nothing was recoveredLine truncated
| 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its encrypted database copy on ep0 (6-hourly, `scripts/hub-db-backup`; **since 2026-10-24 the hub volume is no longer in DooPlex's Longhorn backup**, D40 / R-1243) | operator (`kubectl`) | | 6 h (ep0, 14 daily + 8 weekly) | **UNPROVEN** **`[BETA-DEFERRED]`** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) |
| 11b | *consequences while the hub is gone* | — | — | — | | | **[FACT]** **`[BETA-DEFERRED]`** | **day:** nothing customer-visible breaks; events queue (`settings.go:1466-1485`). **week:** the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. **permanently:** escrow custody and break-glass credentials are gone (INV Part D2.4) |
| 12 | **Offsite provider lost (Hetzner)** | everything on-premises: both drives, both whole-guest tiers | none needed — on-premises recovery is unaffected. Re-provision a new offsite target. | operator | | | **[FACT]** **`[BETA-DEFERRED]`** | **restic and PBS share the provider** (INV Part E.1). Whether they share an account and payment method is **UNKNOWN** → §11-D **`[FACT]` 2026-10-03:** the PBS (whole-guest) off-site copy now also exists OFF Hetzner — DooPlex pulls ep0's datastore nightly (decision 70). The restic tier is still Hetzner-only. |
| 13 | **Customer loses R** | every byte, every tier, the Recipe | **on-premises recovery is unaffected** — Tier-1/2/3 and the whole-guest tiers all work while the guest and host live. What is lost: the ability to re-establish **host identity** and to open the escrowed offsite key material after a host loss | — | | | **NONE for host-loss** | R exists in **zero** system copies by design (INV Part C, row 24). Whether the operator should be able to recover is **open** → §11-A, §11-B. **D5 narrows this further:** Tier-1/2 app recovery needs only the drive, so losing R now costs the offsite route and host identity, never local app recovery (§7.4) |
| 14 | **Host SSH management plane dies** | everything | 3-layer break-glass: tmpfiles → 60 s watchdog → hub-vaulted `root@pam` on the PVE console | operator | | | **PROVEN** | `runbooks/break-glass.md`; the original incident and its fix |
| 15 | **Interrupted offsite run leaves a stale lock** | everything | manual `restic unlock --remove-all` | **operator** (SSH) | | | **DEFECT** | the self-heal exists (`offbox.go:634-648`) but the probe fails first and `classifyResticProbe` has no lock case → fail-fast, operator told *„ismeretlen okból"* → **R-104** |
| 15 | **Interrupted offsite run leaves a stale lock** | everything | manual `restic unlock --remove-all` | **operator** (SSH) | | | **FIXED 2026-10-05 (R-104, v0.297.0; was DEFECT — corrected 2026-10-24)** | the self-heal exists (`offbox.go:634-648`) but the probe fails first and `classifyResticProbe` has no lock case → fail-fast, operator told *„ismeretlen okból"* → **R-104** |
| 16 | **The container registry is lost with DooPlex** (the released images and agent packages) | the release copy on ep0 (D33): the newest 3 release tags of `felhom-controller`, `felhom-hub` and `felhom-samba`, and the newest 5 agent packages (binary + config bundle), in a plain folder outside PBS (`/srv/felhom-release-copy`) | push the copy back into a registry with the same name — `runbooks/total-loss-of-dooplex.md` „Restore the release copy” (same digests, no rebuild; `scripts/release-copy/`) | **operator** (ep0 root) | | 24 h (nightly 01:40) | **PARTIAL** | `audits/day-2026-10-23/08-release-copy-D33.txt`: the copy was pulled back, every sha256 and digest matched, and a controller image from it ran on a scratch box. Not yet walked: the push into a real new registry and the agent-package upload. Added 2026-10-25 (D44 = all picks, R-1265) |
### 8.2 The ONE exception to row 5's derived-copy rule (R-403, controller v0.230.0)
@@ -1531,6 +1649,133 @@ absent or hollow primary unit from the mirror **inside the call, before returnin
observe the hollow state. **The capture itself is deliberately not guarded:** a capture that describes
an empty drive as empty is correct, and guarding it would make the manifest lie.
### 8.2a A second-drive copy whose SOURCE is gone is kept until the household releases it [DESIGN, operator ruling D25, 2026-10-21]
**[DESIGN]** The derived-copy rule (row 5) rebuilds the secondary from the source. When the SOURCE is gone — the drive
is away or decommissioned, a registered share's folder or an app's declared folder is missing — there is nothing to
rebuild from, and the copy may be the only one left. **D25:** such a copy is kept, recorded, and listed on the restore
page („Megőrzött másolatok") with „Eltávolítás"; no night job removes it. This is a different case from „data legs are
not guarded" for a leg that LEAVES the class set (the household's or the catalog's own choice). A share the household
deleted or renamed on a present drive is still pruned. An EMPTIED present folder is not mirrored over a copy that
holds files (the run warns). Removal is by the record (path inside the copy tree, same device + inode), never by size
or pattern. **Why:** the data-loss hunt of 2026-10-20 proved two nights that deleted a household's only second copy
(R-1175: one share group's reconcile removed the other groups' copies, and a dead drive's copy; R-1181: a missing or
emptied folder took its copy within one night). Controller: `internal/backup/tier2_kept.go`.
**`[DESIGN]` 2026-10-24 — a rejected backup block never deletes a copy (R-1252; review round 2 HUNT-2 H1, H2, H9;
committed in controller `518a135`, unreleased).** The night's reconcile (deleting dest dirs the legs no longer name)
runs ONLY under a valid classification. An app whose block is rejected (one bad line rejects the whole block) or absent
falls back to the legacy legs, and nothing is reconciled that night — every existing copy stays. A REJECTED block puts
„Az alkalmazás mentési beállításai most nem olvashatók…" on the app's second-drive status and tells the operator once a
day (the operator-only digest `backup_run_failures`, leg `tier2-settings-unreadable`). A kept copy (D25) is always in
the keep set, also when its leg later leaves the declared set (H2); Remove with „remove backups" drops the kept-copy
records of what it deleted (H9). **`[FACT]` 2026-10-24 night, 9202 (`audits/restore-holes-2026-10-24/P2-9202/`):**
audiobookshelf with a household file in `media/audiobooks`; two control runs kept the copy; one bogus path in the
block → „the backup block is REJECTED … every existing copy on the second drive is KEPT", the copy there, the status
sentence set, the operator digest sent (9202 has no hub: logged as dropped); the review's run on 0.342.0 the same day
deleted that copy (`review-round-2-2026-10-24/proofs/H2-bad-backup-block/`).
### 8.2b The second-drive copy runs in a memory-capped container that never outlives the copy [DESIGN + FACT, R-1214, 2026-10-22/23 nights; rides controller 0.340.0]
The copy (`rsync -a --delete`) runs in a container of the controller's own image, capped at 512 MB, so its page cache
is reclaimed inside the cap instead of from the apps (R-1214: full stall 5.7 s → 0.37 s on 9202). Only for paths the
controller and the Docker daemon see alike (`/mnt/…`, no symlink); the daemon must see the path itself (`--mount`
refuses a missing one — then the in-process copy runs). **The container must never outlive the copy** (the stray-copy
fix, 2026-10-23 night): a fixed name and the label `felhom.tier2-copy=1`; on any error, the timeout included, it is
removed by name; at controller start, before the scheduler and the web server, every labelled container is removed;
and a restore is refused while any second-drive copy runs or a leftover may still run (the other direction of R-1180 —
a `--delete` beside a restore removes what the restore brings back). *(As built — noted 2026-10-24: the refusal is in the
web layer's `restoreOpBlocked` (`internal/web/restore_wizard.go:226`); the install-time „use my kept data" load checks
only `RestoreStatus`/`IsRunning` (`internal/api/kept_install.go:89-92`) — a register row of review round 2.)* **`[FACT]` 9202, rc.5
(`audits/night-2026-10-23/P1-straycopy/`):** a controller restart in the middle of a 16 GB copy → the new controller
removed the old container at start; a restore probe during the copy was refused; the redone copy equals the source; a
15-second test limit → the container removed, the partial copy stopped growing. Controller:
`internal/backup/tier2_copy_lifecycle.go`.
### 8.2d An earlier install's unit is set aside, never rewritten [DESIGN, operator ruling D47, 2026-10-25 08:38; `09` §3 decision 256; R-1269 — controller `02daebe`, unreleased, rides v0.343.0]
**[FACT] Measured before (controller 0.342.0, which is `e1613ae` for these files):** the catalog sweep 2026-10-24
(three times) and 9202 2026-10-25 (`audits/restore-truth-2026-10-25/before/`): Tandoor, one account, „Mentés most",
remove keeping data and backups, install again. Within five minutes the periodic refresh rewrote ONLY the unit's
`compose/app.yaml` (checksum `cbfcb538…` → `e19d2610…`) — the new install's database password — beside the OLD install's
data. Its one „same definition" test was the image pins (`recovery_unit.go:194`, the R-696 `frozen` rule), and a
reinstall of the same version has the same pins. A restore then paired the new password with the old data directory:
`password authentication failed` ×44 (app) / ×23 (postgres), the app stayed `starting`, the account was gone from
view. The next night's dump legs would have overwritten the old data as well, and the stale-drive prune deleted a unit
on another drive whatever install it belonged to.
**[DESIGN] The rule.** A recovery unit's definition and its data are written only together, by a backup of the same
install. Every install has an identity (`install_id` in its `app.yaml`): minted at a fresh install and at a `.fab`
import, carried by every load-then-save, taken FROM the unit by a restore (the app then holds that unit's data), and
minted once for an app installed before D47. The unit records it in its own `app.yaml`. Every writer of a unit asks
first — the capture and its refresh, the night database dump, the pre-backup and the update's pre-backup dumps, the
volume dump, the second drive's unit copy, the stale-drive prune. A unit of ANOTHER install (different identities; for
a unit written before D47, a carried secret that differs — a reinstall mints new ones) is **moved whole** (a rename on
its own drive; never a copy, never an edit) to `<namespace>/backups/set-aside/<app>/<UTC stamp>/` and recorded as a
kept copy (§8.2a's record: kind `unit`, reason `earlier_install`); the new install then gets its own unit from its
first backup. A move that cannot be done **refuses the write** and leaves the old unit byte for byte (an ERROR line
names the app and both identities). Nothing moves a unit while that app is being restored.
**What the household sees.** The restore page lists the set-aside unit under „Megőrzött másolatok" („…· egy korábbi
telepítés mentése") with **„Visszaállítás"** — the one unit-restore body, with the unit's own database password — and
„Eltávolítás". Only those two, and the app's remove with „delete its backups too", ever remove it (the §8.2a / R-1252 H9
rule). After a restore from it, the app is that unit's install; the app's own unit of the install it replaced is then
set aside at the next backup — never lost.
**Known limit, stated.** A box that reinstalled and refreshed BEFORE this release already holds a rewritten unit; its
old password is gone and nothing can bring it back. A unit from before D47 of an app with no carried secret has no
signal: it is treated as this install's (the old behaviour) and gets the identity at its next write.
**Why a volume app's reinstall asks no kept-data question:** the question (§6.5, decision 36) is about the app's drive
FOLDER (`${HDD_PATH}` binds under `appdata/`); a volume app keeps nothing there — „remove, keep my data" removes its named
volumes and keeps only its backups — so the install starts empty, and the kept backup is now the set-aside unit on the
restore page. (The missing offer at install is its own row, R-1278.)
### 8.2e A restore whose app does not start is never „success" [DESIGN, R-1270, 2026-10-25 — controller `b680db8` + fix A; D49 = A (operator 2026-10-25); released in controller 0.343.0]
**[FACT] Measured before:** the health check after a restore was a WARN on every path, so a Tandoor restore whose app
stayed `starting` was recorded `ok: true`, „…visszaállítva — az alkalmazás újraindult." (catalog sweep 2026-10-24; 9202
2026-10-25, `audits/restore-truth-2026-10-25/before/`). **[DESIGN]** Every restore path — the unit restore (so the
second drive's unit and whole restore, a kept load, a set-aside unit), the off-site restore, the second drive's file
restore and the no-unit fallback — returns „not started" when the app does not reach running within the health wait
(90 s). The household reads „A(z) %s: az adatokat visszatöltöttük, de az alkalmazás még nem indult el. Várj pár percet,
majd nyisd meg újra. Ha így marad, szólj nekünk." (the record `ok: false`, `not_started: true`); the operator gets one
`backup_run_failures` line (leg `restore-not-started`). **The safest reading, the operator may change it:** a timeout
does NOT pour the old data back — a slow first start (a database migration) is common, and a put-back would throw away a
correct restore. The data stays restored and the restore's undo copy is KEPT — the same „kept for support" state a failed
put-back leaves (§8 D20/D22), so no new retention rule. A watcher looks 30 more minutes: when the app runs, the record
turns to success („…elindult: a visszaállítás kész.") and the copy is removed, as after any successful restore; if it
never runs, the copy stays for support.
**D49 = A (operator ruling 2026-10-25, attended release run; `09` §3 decision 258):** the safest reading above is the rule —
the box keeps the restored data, keeps the old copy, tells the operator, and never puts the old data back by itself.
**Fix A (controller 0.343.0, R-1270 follow-up):** „started" is one helper on every path — the stack running on every
sample of a 45 s window, no container's `RestartCount` grown in it, every container with a healthcheck `healthy` (a
container without one is judged by running alone). An app that starts and then crash-loops is „not started". The undo
copy's RELEASE is a separate, longer test: 3 minutes of the same after the verdict (10 s samples, watched up to 30
minutes). **[FACT]** 9202 rc.6: an image that exits after 20 s → not started, copy kept, one operator line; the original
unit back → success, copy released 3 minutes later. Tester 1 on 0.343.0: a unit restore and a set-aside restore, each
„ran stably for 3m0s … the restore copy is released" (`audits/release-2026-10-26/`).
### 8.2c Decisions in code that no document recorded [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
Read at `file:line` on `main` (controller `321df1f`, hub `44d19042`) by the main session:
- **A second-drive copy is limited to 60 minutes**; a stray copy container that cannot be removed blocks every
restore and is retried every minute; copies into one destination run one at a time and are never killed by name
while running (`felhom-controller/controller/internal/backup/tier2_copy_lifecycle.go:40,113-121,150-172`).
- **The household's „Eltávolítás" of a kept copy (D25) is refused while any backup, restore or second-drive copy runs**
(`internal/web/kept_copies.go:54-58`).
- **A start that fails AFTER a successful put-back** leaves the app stopped and NOT held, keeps the copy, and tells the
household `restore_failed_put_back_not_started` (`internal/backup/offbox_restore_undo.go:245-247`).
- **The hub's off-site delivery checker** (`hub/internal/monitor/offsite_delivery.go`): a box that took no off-site
credential is reported `offsite_delivery_stuck` (operator) and healed by `maybeHeal` (a burned vs a stuck shape);
a box that declares it waits for the household's recovery code (`awaiting_recovery_key`) is skipped — no event, no
heal (R-1230, `:103-110`).
- **The release copy (D33)** — nightly 01:40, the newest 3 image tags and 5 agent packages to ep0 under its own
limited user, capped by the sender (`felhom.eu/scripts/release-copy/`). ~~has no row in this matrix~~ It is row
**16** of the matrix since 2026-10-25 (D44 = all picks, R-1265); its other homes are the poster facts and
`runbooks/total-loss-of-dooplex.md`.
### 8.1 The blank cells, listed explicitly
Per the rule that a blank is a finding, here they are:
@@ -1548,6 +1793,7 @@ Per the rule that a blank is a finding, here they are:
| 13 | RTO, RPO | not a timed recovery; a capability loss |
| 14 | RTO | the break-glass path is proven but was never timed |
| 15 | RTO | the manual unlock was performed but not timed |
| 16 | RTO | a registry rebuild from the ep0 release copy has never been timed (row added 2026-10-25, D44 #6, R-1265) |
**Also unmeasured, and not representable as a row** (INV Part F.3): any restore larger than 155.5 MB
from restic; any restore of a customer data drive (no tier holds one whole); the weekly PBS
@@ -1801,7 +2047,7 @@ to now *implement* D5 remains an open scheduling decision, not a blocked one.
| host-loss plan → an actual restore | **EXECUTES NOTHING BY CONSTRUCTION** | `felhom-agent/internal/dr/plan.go:1-4` |
| host rebuilt as its former self | **NEVER DONE** | INV Part D1 |
| escrow **consume** in a real recovery | **SPIKE-LEVEL ONLY** | `06-offsite-connectivity.md:327` |
| hub DB restore from its Longhorn backup | **NEVER DONE** | INV Part D2.2 |
| hub DB restore from its ep0 copy into a running hub (the weekly restore-test opens the copy and checks it; a hub has never been brought up on it) | **NEVER DONE** | INV Part D2.2; the Longhorn backup it used to cite is off since 2026-10-24 (D40) |
| a **customer** performing a restore unassisted | **MISSING AS EVIDENCE** | `00-capability-map.md:75` |
---
+91 -7
View File
@@ -135,7 +135,9 @@ live 2026-08-23 — one event across 22 scans.
**The vocabulary is the HUB's and it is exact: `{info, warning, error, critical}`.** Anything else is
**coerced to `info` at ingest**, and `info` is dropped by `severityNotifies` before *both* delivery
legs. So a severity outside the set means the event is stored, answers `200`, shows on the dashboard —
legs — except the paired recoveries in `recoveredPairedDownTypes` (`node_recovered`, `host_recovered`,
`whole_guest_backup_recovered`, `pbsdr_box_recovered`, `offsite_box_recovered`, `storage_reconnected`,
`health_recovered`), which run before the gate and are mailed (`notify/dispatcher.go` — corrected 2026-10-24, review round 2). So a severity outside the set means the event is stored, answers `200`, shows on the dashboard —
and is e-mailed to **nobody**.
**This shipped twice.** `DiskAlertKind.Severity` emitted `"warn"` until controller v0.215.0;
@@ -155,8 +157,10 @@ Three things now hold it:
3. **The dispatcher's `unrecognized severity` branch is kept**, because the hub's own monitor checkers
call `ProcessEvent` directly and never pass the ingest handler. For them it is the only guard.
**Who gets it.** `processOperator` consults only `operatorOn`, the address and a 1-hour cooldown —
**never customer preferences** — so a valid severity always reaches the operator. `processCustomer`
**Who gets it.** `processOperator` consults only `operatorOn`, the address and a cooldown —
1 hour, except 5 min for node/host liveness and 6 h for a storage-fill warning (`operatorCooldownFor`); it records a
`backup_tier_skipped` in a box's FIRST hour as „suppressed" instead of mailing it (R-723), and the mail hold applies —
**never customer preferences** (cooldowns and R-723 corrected 2026-10-24) — so a valid severity always reaches the operator. `processCustomer`
consults `operatorOnlyEvents` and then the customer's `enabled_events`.
**`app_start_failed` is customer-switchable but OFF by default** [DESIGN, operator ruling 2026-08-23]:
@@ -180,7 +184,10 @@ rather than assumed (2026-08-30):
- **not** in `operatorOnlyEvents` — the operator leg ignores customer preferences anyway, so the
operator is always mailed; putting it here would only remove the customer's ability to opt in.
- **not** in `DefaultEnabledEvents` — customer-switchable, default OFF, the same ruling as
`app_start_failed`. The checkbox already exists at `settings_notifications.html:34`.
`app_start_failed`. **Superseded by operator Q1 (2026-10-12, R-388 slice 1):** since controller v0.309.0
`backup_integrity_failed` (with `crossdrive_failed`, `storage_reconnected`, `health_recovered`) IS in
`DefaultEnabledEvents` — the default for a household with no saved list (`internal/settings/settings.go`, pinned by
`TestR388_DefaultListAsGroups`; dated note 2026-10-24, review round 2). The checkbox already exists at `settings_notifications.html:34`.
**And a caveat that belongs in the alarm ladder rather than only in the backup document:** a
`backup_integrity_ok` at the shipped depth means *the index, the pack inventory and the snapshot graph
@@ -293,6 +300,7 @@ bands.
| app down (`app_start_failed`) | **per APP** | `…:<stack_name>` | no digest exists for it — see below |
| update outcome (`app_update_undone`, `app_update_held`) | **per APP**, both legs | `…:<stack_name>` | hub v0.120.0 — one mail per app per outcome; the household leg has its own register (`perAppCustomerCooldownEvents`) |
| OOM storm (`app_oom_storm`) | **per APP**, operator | `…:<stack_name>` | hub v0.121.0 — no digest; the controller already sends it at most once per container run |
| more per-APP types (added 2026-10-24 from `dispatcher.go`'s per-stack set) | **per APP** | `…:<stack_name>` | `app_hold_no_whole_copy`, `app_stopped_unhealthy` (both legs), `app_repaired_after_crash`, `app_unexplained_upload`, `app_unexplained_cpu` |
| backup run (`backup_run_failures`) | per RUN | `…:<run_id>` | a digest already lists every failing app; one per run |
| tiered backup (`whole_guest_backup_failed`, …) | per TIER | `…:<tier>` | the tiers fail independently and mean different things |
| everything else, incl. `crossdrive_failed` and `backup_integrity_failed` | per TYPE, per hour | — | coarse **on purpose** |
@@ -367,6 +375,14 @@ one info line beside `host_crash_restart`; `operatorOnlyEvents`, pinned by
---
**More operator-only box alarms, listed 2026-10-24 (review round 2; each is in `notify/dispatcher.go`'s
`operatorOnlyEvents`, and each has its home elsewhere):** `wg_tunnel_stale` / `wg_tunnel_ok` — the box's tunnel to ep0:
a handshake older than 6 h or stamped in the future, edge-triggered (D30, R-1204; `api/host_tunnel.go`, `05` §15.4b);
`host_swap_short` / `host_swap_ok` — the host's swap below 90 % of min(the guests' swap allowance, 8192 MB) (R-1192;
`api/host_swap.go`); `bind_old_host_linked` — a household asked for a new bind link while its old host is still linked
(R-1203; `05` §13); `golden_not_vouched` — a published golden the hub does not vouch (decision 199, R-242);
`recovery_older_package` — a household's code opens, or may open, an older sealed package (decision 183, R-304 C).
## 6.4 A box that is not always on: the missed-backup deadline and the household's outage mail [DESIGN, hub v0.134.0, 2026-10-05]
Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *operator may reverse*).
@@ -376,7 +392,8 @@ Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *op
never judged (measured 2026-10-05: `1 skipped (down)`). Now a down box is judged on longer lines:
`expected_dbdump_missed` after **48 h** without a `db_dump_completed`, `expected_backup_missed` after **72 h**
without a whole-guest backup in any retained host report — never for a box first seen less than 48 h ago. A box that
died last night still raises only its staleness alarm. A `disabled` box is still skipped (R-321). Pinned by
died last night still raises only its staleness alarm. A `disabled` box is still skipped (R-321), and so is a box the household switched off on purpose — by its staleness
state or by the host row's hold (R-998, `monitor/deadline.go`; corrected 2026-10-24). Pinned by
`monitor/r872_down_box_test.go`.
- **R-873 — "your server cannot be reached" reaches the HOUSEHOLD at most once per 7 days** (`node_stale`,
`node_down`, `host_stale`, `host_down`, read from the persisted notification log, so a hub restart does not reset
@@ -387,7 +404,7 @@ Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *op
---
## 6.5 One lost off-site copy on a pinned tier [DESIGN, R-435 — `09` §3 decision 191, hub main 2026-10-08, unreleased]
## 6.5 One lost off-site copy on a pinned tier [DESIGN, R-435 — `09` §3 decision 191; released in hub v0.144.0 — corrected 2026-10-24]
On a **pinned** off-site tier (the hub holds a confirmed append-only key, decision 69) the only legitimate way the
snapshot count can fall is a clean-up window the hub opened (decision 68). So the hub raises
@@ -470,7 +487,7 @@ was ruled that way on its own merits and does not pre-judge the redesign.
**Filed as a PRODUCT DECISION, not a defect** — see the register. It is the operator's call to take
separately, and no part of it was implemented here.
### 8.1 The design — „only what you can act on" [DESIGN, R-388 — `09` §3 decision 200, 2026-10-12; NOT BUILT]
### 8.1 The design — „only what you can act on" [DESIGN, R-388 — `09` §3 decision 200, 2026-10-12; slice 1 BUILT in controller v0.309.0 — three group switches, details under Részletek (`internal/settings/notif_groups.go`); corrected 2026-10-24]
*Operator ruling 2026-10-12 (S5): design now, build in slices later. Written by CC from the direction above and the
live page (controller, `settings_notifications.html`: **18 event switches** plus the address, the cooldown and the
@@ -499,6 +516,23 @@ app-mail settings).*
**Pick: A.** It follows the direction (the household decides three things, not 18), keeps every existing
mechanism, and each slice is reversible on its own. B can be added later as a fourth choice if households ask.
**`[DESIGN]` 2026-10-21 night — the box's page is the list the hub mails by (R-1199; decided by CC unattended — operator
may reverse).** Measured: a fresh box shows its default list ticked, but pushed nothing until the household saved the
page, so the hub kept mailing by an older stored list (Tester4: saved 2026-10-18, without disk warning, drive
disconnected and second-drive failure — 1 household alarm of 9 on the rehearsal day). Now a box whose household never
saved a choice pushes the default list at start (no address — the hub keeps the stored one); a saved choice is pushed
only as before (R-922). Controller `ba22f9b`, shipped in 0.338.0.
**`[DESIGN]` 2026-10-22 — a household's saved list is never overwritten (R-1199 reopened; the day brief).** `[FACT]`
0.338.0's push overwrote demo-felhom's own saved list (13 events, written by an earlier box) with the 17 defaults at
its first start, and repeated at every start. So the hub now records WHO chose a list (`customer_notifications.
household_saved`, set only by a push the box declares `source: household` — the settings page, or the re-push of a
list saved on the box — and never cleared) and refuses any other push over a household's list (`{"status":"kept"}`;
an older controller's push with neither address nor clear counts as not the household's). A box with no list of its
own READS the hub's first (`GET /api/v1/preferences`, self-scoped): a household's list is adopted and shown, nothing
pushed; only when no household ever saved one does the box push its default, once, and save it. Hub 0.165.0,
controller 0.339.0. Existing rows were marked once: written after creation = a box wrote it (`household_prefs.go`).
**What the operator still gets:** everything, unchanged — the operator channel never reads the household's switches
(`processOperator` is independent of `enabled_events`).
@@ -514,3 +548,53 @@ list is shown as custom and saved back unchanged; the default list maps to group
multi-day backup fault reads as „nobody noticed" if the household ever finds out.
2. **The cooldown field („hány óránként legfeljebb"):** keep it on the page or move it under „Részletek"? Pick:
**under Részletek** — it is an engineering choice, which is what this redesign removes.
## 9. The upload and CPU watch — „does the app explain its own traffic?" [DESIGN — built; released in controller v0.342.0 and hub v0.169.0 (deployed) on the operator's yes, 2026-10-24 — corrected 2026-10-24, review round 2]
**Why** (operator ruling 2026-10-23 19:22, `CONTEXT.md`): DooPlex's hacked Plex sent ~474 GB in a day with no viewer and
used ~10 cores with no film being converted (`audits/plex2shell-felhom-impact-2026-10-23/`). Nothing on a Felhom box
would have noticed: the health check sees whole-box CPU at one instant, and nothing looks at upload at all. **The idea:**
when an app sends a lot (or burns CPU) for a long time, ask the APP whether that is its work. Work the app explains is
normal use; work it does not explain is the sign of a hacked app. Numbers and measurements:
`audits/SPIKE-upload-watch-2026-10-23.md`; the code: controller `internal/upwatch` (README §8e).
1. **The line.** Learnt from what the box already does: while its off-site push runs, the controller samples its own
sent bytes once a second; the best ≥ 10-second rate is kept per night (14 nights, settings.json `upwatch.line`); a
night counts when that window moved ≥ 10 MB; the line is the highest counting night. A **lower bound**, shown so
(„legalább N Mbit/s", system settings). **AMENDED 2026-10-24 (operator ruling 07:26, R-1247): the weekly line TEST.**
The night's one-second read showed how low that floor is (demo-hp „at least 10 Mbit/s"; spike §2.1). So the host's
agent (agent ≥ 0.164.0, `internal/linetest`) runs a 10-second iperf3 upload once a week to ep0's tunnel-only server:
Tuesday 10:00–15:59 local from a per-host slot, only after 10 quiet minutes of the physical uplink (< 2 Mbit/s each
minute), under the one-heavy-op gate, best of 3, never in the night chain; „server busy" → later the same day. The
controller reads it (`GET /host/line-test`) every 15 minutes; the watch judges against the higher of test and floor;
the system page says „N Mbit/s (heti sebességméréssel)". demo-hp measured 90.7 Mbit/s (`audits/line-test-2026-10-24/`).
2. **Trigger, per app** (its compose project's containers summed; exact counters from the Docker Engine API, once a
minute): upload ≥ max(5 Mbit/s, 25 % of the line) for 15 minutes; CPU ≥ max(150 %, 50 % × cores) of one core for 30
minutes. Infrastructure (the controller — its own push is the line measure —, traefik, cloudflared, Samba) is never
judged; the box's own work on an app (a backup run, an update, a deploy) is never judged.
3. **The explanation.** qBittorrent (`up_info_data`) and Transmission (`uploadedBytes`): explained when the container
sent ≤ 1.25 × what the app counted + 1 Mbit/s. Plex: any viewer explains upload; any transcode explains CPU. An app
that does not answer gives no number (a hacked app may refuse its own API). No explainer: upload alarms only at ≥ 10 ×
its own p95 for 60 minutes after ~20 h of history; CPU at ≥ max(200 %, 75 % × cores) for 60 minutes.
4. **Who is told** (the R-388 rule, §8.1): **`app_unexplained_upload` / `app_unexplained_cpu`** (warning) and their
`_cleared` (info, recorded) are **operator-only** (hub `operatorOnlyEvents`), one per episode, per-app cooldown. The
household is told only about the qBittorrent limit, because they can change it (R-1246 asks the operator).
5. **qBittorrent's default upload limit:** 50 % of the TESTED line (D41 = 50 %, operator 2026-10-24; never of the
push floor), every 15 minutes, re-set when the line moves > 20 %; never over a limit the household set (one there
first, or any change from the value qBittorrent stored for the box — it keeps KiB/s). No test → none set. **A box
that seeds all day (R-1250, brief 2026-10-24; committed agent `29bcf44`+`1e087bb`, controller `046b808`, not yet
released):** the 10 quiet minutes never come, so when the agent reports `waiting_for_quiet` (due, in the window,
tunnel up, the full 10-minute window watched and busy) and qBittorrent or Transmission runs, the controller writes
`upwatch.line_test_hold` (the app's limit exactly as stored) FIRST, holds the app at 1 KiB/s (Transmission 1 kB/s),
calls `POST /host/line-test/run` (5 s settle, 10 s quiet check < 2 Mbit/s, three runs 5 s apart — about 55 s), then
puts the old limit back, the household's own included; a change made during the test stays; a killed controller's
start-up check puts it back; the 50 % rule waits while a record exists. Measured on demo-hp with a real seed (10
Mbit/s to DooPlex's qBittorrent): refused `not-quiet` without the hold; with it the uplink fell to 0.02–0.16 Mbit/s,
94.2 Mbit/s measured, the app held 56.8 s and back to 1 249 280 B/s exactly (`audits/dooplex-alarms-2026-10-24/`).
While the box owns it, qBittorrent's page says: „A qBittorrent
feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod."
6. **Limits, stated:** *(added 2026-10-24, review round 2: ANY viewer explains any amount of upload, so a hacked app can send freely while one real playback runs — `upwatch/watch.go:275,299`; R-1262)*; an attacker who knows the check can make the app's API lie — a per-container ceiling (`tc` from
a helper in the app's network namespace, measured to work in an unprivileged guest; spike §5) would cap the flood
whatever the API says; not built. The box cannot tell LAN from internet bytes per app; a host-network container
(Samba) has no counter; the push floor is a lower bound, the weekly test is close to the line.
7. **Release:** released 2026-10-24 on the operator's yes (controller v0.342.0, hub v0.169.0; R-1244) — corrected 2026-10-24.
@@ -65,6 +65,13 @@ build is added to both in the same commit.
### 1.1 The button
> *Dated note 2026-10-24 (review round 2): §1 is the 2026-09 MEASUREMENT before the arc. `UpdateStack` no longer exists
> (deleted in v0.237.0, controller `REUSE.md`); the button calls `StartGuardedUpdate` — backup precondition, safety
> dump, pin journal, undo copy, health wait, undo (`internal/api/router.go:794`; `internal/stacks/update.go`). The line
> numbers below are the measurement's; e.g. `copyTemplates` is now `sync.go:436`, `copyIfChanged` `:700`, the `15m`
> default `config.go:399`, `RestartStack` `manager.go:1454` (it now runs `up -d --force-recreate` and errors when no
> container restarted, R-1026).*
`Manager.UpdateStack` (`felhom-controller/controller/internal/stacks/manager.go:1199`) is two compose
commands and nothing else:
@@ -267,7 +274,8 @@ builds them (`audits/update-rulings-2026-09-23/`).
next night. **Why:** the update then leans on the freshest copy the box ever has, and there is no
second window for anyone to set or misread. **Replaces:** §3b Q1's recommendation of a fixed
02:30–05:00. *For the builder:* `stacks.update_window` (`config.go` L195, default `03:00-05:00`)
exists and **no Go code reads it** — it appears only in `config.go` (field + default),
exists and **no Go code reads it** *(dated note 2026-10-24: the field is now removed — no controller source names
it; `controller/README.md` „`stacks.update_window` is removed")* — it appears only in `config.go` (field + default),
`setup/handlers.go` L482 (written into a new box's `controller.yaml`) and
`configs/controller.yaml.example`. Remove it or fold it into the chain; do not build a second
window.
@@ -278,17 +286,27 @@ builds them (`audits/update-rulings-2026-09-23/`).
13. **The test decides, not the tag** (Q3). **REPLACES decision 3's second half, "never across a
major".** The box applies by itself every step the catalog holds, because **the catalog holds only
tested steps**; the size of the version jump does not matter, the test does. The box does not
parse tags to decide. The tag rule (`stacks.CompareImageRefs`) moves to the **catalog gate** as a
push-time safety net: an image move with no test record is refused there. Two exceptions, both
parse tags to decide. ~~The tag rule (`stacks.CompareImageRefs`) moves to the **catalog gate** as a
push-time safety net:~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** The tag rule (`stacks.CompareImageRefs`) did not move: it
stays in the controller (`internal/stacks/updateorder.go`), where it only tells whether an installed app is
AHEAD of the catalog; it never decides an update. The catalog gate is the push-time safety net, and it asks for a
proven test instead (§6.4 part 4): an image move with no test record is refused there. Two exceptions, both
**marks the test sets on the step**: *files may change* → automatic only when a fresh copy holds
the files, else a person (this is Q2's refinement); *needs a person* → the tester writes why.
**Why:** decision 3 was a proxy. "Within a major" was standing in for "known to work", and the
update night measured the proxy wrong in both directions — minor moves that held
(adventurelog, outline) and a tag shape it cannot read at all (§3b Q3). The test record is the
thing the proxy was guessing at.
**Dated note 2026-10-23 (decision 255, D39 = B):** a second step kind now exists — the `security_fix` step, proven
on the box with the bench optional, for an urgent upstream fix with a public notice. Decision 13's rule is
unchanged for every other move: a step a box takes was still tested, on at least the box venue.
14. **The update ladder** (Q3). A box more than one step behind **climbs one tested step at a time, in
order, and never jumps**. Each step is the full guarded update of §6.1. **Why:** a step was tested
order, and ~~never jumps~~ the night leg never jumps**. Each step is the full guarded update of §6.1.
**[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** An app older than the ladder (its installed version matches no entry's `from`) is never
pressed by the night leg (`internal/stacks/unattended.go`). A PERSON's press on such an app takes the catalog's
current definition, logged by name (`internal/stacks/ladder.go`, `TestLadder_UnknownInstalledJumpsAndSaysSo`); the
guarded update and its undo (decision 15) still run around it. **Why:** a step was tested
from the version before it, not from three versions before it; R-40's multi-major jump is what
this prevents. This is the "stepping" §6.1 already assigned to slice 6.
@@ -421,7 +439,8 @@ R-636's louder repeated alarm.
(a) broke the second. Shipped as controller **v0.269.1** — a second release in the session, instead of
shipping (a) to the fleet with the floor.
*Note 2026-09-30 (R-740) — the cost line above is not what the code does.* „(or the automatic leg)": the
automatic leg never presses a digest-only change (`unattended.go:433–435`, skip `no_test_record`). And the
automatic leg never presses a digest-only change (`unattended.go:433–435` then, `:449-451` now — line corrected
2026-10-24 — skip `no_test_record`). And the
catalog never writes a re-test of a floating tag at a new digest (the writer refuses a step that moves no
image), so a same-name upstream fix reaches no box by either route today. Measured, with the options, in
R-740; the decision is the operator's (STATUS 2026-09-30). This decision's ruling — the sync never moves
@@ -976,8 +995,11 @@ its length, and both fixes cost something the household would notice — operato
installs the vouched golden and updates itself to the floor (not yet proven end to end — R-1052). **The golden
stays weekly:** a CHANGED golden selection is still refused when it is older than the fleet's newest controller
(`TestVouchGoldenChangedToOlderIsStillRefused`).
5. The push gate judges the golden by its AGE (older than 7 days while the floor is above it), not by the newest
CHANGELOG heading (`scripts/golden_currency_gate.py`).
5. The push gate judges the golden by its AGE, not by the newest CHANGELOG heading
(`scripts/golden_currency_gate.py`). **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** ~~(older than 7 days while the floor is above
it)~~ The gate runs offline and cannot read the floor, so „a release is ahead of the newest bake” stands in for
„the floor is above it”: it convicts when a release is ahead AND the newest bake is more than 7 days old; ahead
and at most 7 days old is an advisory. The floor follows a release within hours, so the gate is the stricter one.
251. **Scripts use scoped hub tokens, not the operator password; CC's floor step uses its own.** *Operator ruling
2026-10-15 11:38.* Tokens with groups (read/write per group), managed on the Configuration tab, shown once, stored
hashed, every use logged by name (`05` §17, hub v0.152.0). Never by any token: a secret shown or minted, a delete or
@@ -1031,6 +1053,87 @@ its length, and both fixes cost something the household would notice — operato
Built in controller v0.314.0 (`internal/stacks/restore_gate.go`); pinned by `restore_gate_test.go` and
`r1057_unit_gate_test.go` (ten mutations each seen failing). Proof: `audits/gate-restore-2026-10-15/9202/`.
### 2026-10-23 (~12:00) — D39 = B: the security-fix step (operator ruling by starting the run; R-1239)
255. **An urgent upstream security fix reaches the boxes at night as a `security_fix` ladder step: the box walk is
required, the bench is optional, and a public notice is the authority.** *Operator ruling 2026-10-23 ~12:00 (D39 =
B, R-1239), taken by starting the D39 brief; its design is Claude.ai's recommendation, taken with it.* Why: on
2026-10-23 the only route for the Plex2Shell fix was an operator word, `git push --no-verify` past the move gate,
and a hand press on each box — and the night leg never takes a step with no test record (R-1238). That does not
scale past the first customers and needs the operator awake.
1. **The entry** (`scripts/ladder.py` `check_security`, the one reader): `"verdict": "security_fix"` with `security`
`{ids: [CVE-…|GHSA-…], advisory: "https://…", fixed_in}`, `box_evidence` (required), `seed` `read_back`|`none`,
`memory_peak_pct` from the BOX soak with `memory_basis: "box_soak"`; `evidence` = the bench record or null — a
bench given must be `proven` (`bench_verdict`, `harness_version` ≥ 2 ride with it). Never `backfilled`,
`digest_from` (a same-tag fix stays decision 52's lane), `engine_conversion`, nor a database engine MAJOR (any
engine — MariaDB included; the judgement is `check-engine-major.py`'s own `engine_of`/`major_of`, imported).
2. **No operator word per incident.** The session that writes the step reports it to the operator after the push.
3. **The gates:** the move gate accepts a `security_fix` move exactly as a proven one (from/to per service, digests
the registry still serves); `check-test-record.py` WARNs on every push while a template's head is a security-fix
step with no bench test („<app>: security-fix step <to> has no bench test (since <tested_at>)"). Seven decoys
(`test_gate_decoys.py`), each seen failing with the rules switched off.
4. **The box venue:** `box_walk.py --security` on 9202 through the product's Update; verdict `proven` only if the
update ended `done` with no undo and no hold, the app runs, the four version observables are TO, the seed (if
any) reads back, and a soak (default 600 s, cgroup `anon` against the limit, OOM kills and restarts counted)
sees no kill and no restart. A negative control (a TO image that exits at once) comes back `failed`.
5. **The writer:** `upgrade-test.py --write-ladder --security …`, refusing a box walk not proven, a bench given and
not proven, a template not at the walk's FROM, a digest that moved since the walk, an engine major, a missing
field. The compose move, `catalog_since` and the superseded step's own files are the shared `apply_entry`.
6. **The controller (v0.341.0):** the leg takes `proven` OR `security_fix`; every other guard runs unchanged, in
the same order; it logs `[update-leg] <app>: security fix <ids> taken (<from> → <to>)` and the hub report's
step carries the ids. A person's press and the badge are unchanged. **A controller ≤ 0.340.0 skips the step at
night (`not_proven`) and a hand press takes it** — proven on 9202, below.
7. **The trade, stated.** A security-fix step skipped the bench. For an app with a box fixture the data is still
read back after the update; for an app without one (Plex today) the step is checked only by „it starts, it is
healthy, it runs the new version, and a 10-minute soak sees no kill or restart". `files_may_change` is a bench
measurement: with no bench the step carries `false`. That is accepted because the leg's own guard is not the only
net — the update takes a per-app backup first, the undo puts the pre-update data back on a failed health check
(decision 15), and a security fix that rewrites files without breaking the health check is the case the bench
would catch and this lane accepts the risk of. A session that knows a fix migrates files runs the bench.
**Proof (9202, drill catalog, 2026-10-23):** `audits/security-lane-2026-10-23/` — the negative control (TO
`busybox:1.36.1`) came back `failed` (the box undid it after 335 s) and the writer refused it; Plex 1.43.4.10903 →
1.43.5.11029 `proven` (done in 36.9 s, the four observables at TO, digest `sha256:01bdf0ad…5c5f`, soak 600 s: anon
peak 10.7 % of 2 GiB, 0 kills, 0 restarts, `seed: none`); the writer wrote the drill step (CVE-2026-96652, -96654,
-96655, -96656; Plex's own notice as CISA KEV cites it, added 2026-10-20; fixed in 1.43.3.10861); on 0.340.0 the
night leg logged `plex: skipped — not_proven` and a hand press took the step; on 0.341.0-rc.1 the night leg logged
`plex: security fix CVE-2026-96652,CVE-2026-96654,CVE-2026-96655,CVE-2026-96656 taken` and ended `done` in 30.1 s
after the update's second-drive copy and undo copy. Released as controller 0.341.0, global floor 11:24 UTC.
### 2026-10-25 (attended release run) — D49: a restore that does not start keeps both copies; D44–D46: the review picks
258. **A restore whose app does not start keeps the restored data AND the old copy, tells the operator, and never puts the
old data back by itself (D49 = A).** *Operator ruling 2026-10-25, given by starting the attended release run.* Options:
A) keep and wait — cost: an app that never starts needs a person; B) pour the old data back after the wait — cost: a
slow but correct first start (a database migration) is thrown away. Why A: the slow first start is the common case,
the old copy is kept, a late start turns the record into a success. Built: `07` §8.2e and fix A (the verdict needs a
stable start; the copy goes only after 3 more minutes of it) — controller 0.343.0.
259. **The 21 confirm lines and four code changes of the second review round are accepted as picked (D44 = all picks).**
*Operator, 2026-10-25.* The notes carry a normal `[DESIGN — confirmed …]`; the code changes became R-1280–R-1283
(R-1265 closed).
260. **Where a design document and the code disagree on the seven named points, the text follows the code (D45 = all
picks).** *Operator, 2026-10-25.* Each corrected place is tagged `[DESIGN — corrected 2026-10-25, D45 = all picks,
R-1266]`, the old words struck through, not deleted (R-1266 closed).
261. **Homepage is behind the family gate, like Glance (D46 = yes).** *Operator, 2026-10-25.* Why: it has no login of its
own and showed a stranger the box's CPU, memory and disk. Tried on the drill first; catalog `ebbf1f4` (R-1272 closed).
### 2026-10-25 (08:38) — D47: nothing changes an existing backup; D48: a restore downloads first (operator rulings by starting the run; R-1269)
256. **An existing recovery unit is never changed in part (D47).** A unit's definition (compose, pins, `app.yaml`) and
its data are written only together, by a backup of the same install. A new install never adopts, rewrites or
deletes a unit that an earlier install left; that unit is set aside whole and stays restorable. *Operator ruling
2026-10-25 08:38, taken by starting the R-1269 brief.* Why: on 2026-10-24 the catalog sweep measured three times
that a reinstall's own five-minute refresh wrote the new install's database password into the unit „remove — keep
backups" had kept; a restore then could not open the old data (R-1269). Built: every install has an identity
(`install_id`), the unit records it, and every writer asks first — `07` §8.2a „An earlier install's unit is set
aside" (controller, unreleased; rides v0.343.0).
257. **Every restore path resolves and pulls every image its definition names before the first change to the app; a
failed pull is a refusal with the app untouched (D48).** *Operator ruling 2026-10-25 08:38, taken by starting the
brief.* Why: the restore stopped the app and moved its data first and pulled only inside the database-only start;
a version that could not be downloaded then ended in the D22 put-back, a full stop-and-pour-back for nothing.
Built: `07` §6.6 „The limit — the image is not in the backup" (controller, unreleased; rides v0.343.0).
**Confirmed, no change:** `07` §6.6 — a restore brings the app back at its data's version, then the guarded update
climbs the ladder; R-698 (a deleted version cannot come back) and R-933 (our own image mirror) are unchanged.
### 2026-10-15 (07:48–08:20) — bake first; Plex through its own key; HSTS one day (recorded before the media build)
247. ~~**Bake first.** A controller version is RELEASED only when its golden is baked.~~ **REVERSED by decision 250,
@@ -1932,7 +2035,7 @@ moves a stack dir.
| writer | pin source |
|---|---|
| the deploy path (`runComposeDeploy`) | the template just deployed from |
| **`UpdateStack`** | the catalog's current template, written **BEFORE** the pull |
| **`UpdateStack`** | the catalog's current template, written **BEFORE** the pull — *as built: `UpdateStack` was deleted in v0.237.0; the guarded update writes the pin (`advancePinTo`, the `pinning` phase before `pulling`), and two more paths move a pin BACK — `pinBack` and the undo's `restoreDefinition` (`internal/stacks/update.go`, `undo.go`; noted 2026-10-24)* |
| the restore (`stackAdapter.RecreateStackDefinitionFromUnit`) | the recovery unit's captured compose — **this closes R-441** |
| `Manager.AdoptPins` | the observation, once, and only when complete AND matching |
@@ -1998,7 +2101,7 @@ earlier feature is the failure mode to look for whenever a file changes meaning.
| **3** | **The compose file becomes DERIVED** — the pin in `app.yaml` wins; the syncer renders instead of copying. | **SHIPPED, controller v0.235.0 (2026-09-06)** — operator ruling §3.4 |
| **4** | **A guarded update** — verified-backup precondition, abort-on-failure, and the truth at the moment of action rather than 5m16s later (R-443). | **SHIPPED + PROVEN LIVE, controller v0.237.0 (job) + v0.238.0 (page) + v0.238.1 (2026-09-13); any backup tier since v0.239.0 (§3 decision 8)** — §6.1 |
| **5** | **An upgrade test that runs again** — a harness that upgrades a real app with real data in it and asks the app for the data back. | **SHIPPED, `app-catalog/scripts/upgrade-test.py` (2026-09-06)** — 7 edges, 3 apps; see §4.1 and §10 |
| **6** | **Automatic updates** — ~~within a major, a human across one~~ **the catalog's tested steps, climbed one at a time, as a leg of the backup chain, undone by the box on failure** (§3 decisions 11–17); **an engine change gets its own edge.** | OPEN — R-450; rulings 2026-09-23; build order §6.4 |
| **6** | **Automatic updates** — ~~within a major, a human across one~~ **the catalog's tested steps, climbed one at a time, as a leg of the backup chain, undone by the box on failure** (§3 decisions 11–17); **an engine change gets its own edge.** | ~~OPEN — R-450~~ **BUILT** — the leg in `internal/stacks/unattended.go`, wired into the off-site job; R-450 CLOSED 2026-10-05 (capability map B11) *(noted 2026-10-24, review round 2)* |
| **7** | **A fleet sweep pipeline** — the operator can see, and move, how far behind every box is. | OPEN — R-451; ruled 2026-09-23 (decision 18), built later |
### 6.1 Slice 4 as SHIPPED (controller v0.237.0 + v0.238.0, 2026-09-13)
@@ -2013,12 +2116,13 @@ closed by construction: nothing reports an update complete on the compose exit c
| # | phase | what happens | on failure |
|---|---|---|---|
| 0 | refusals (409, before the intent is recorded) | held (R-439), busy (backup/restore/app-data op/quiesce), migration, already updating, deploying, memory (the deploy's `memoryVerdict`, releasing the app's own request), disk (**fixed 2 GB floor** — image size unknown without a registry), **no copy on ANY tier and no backup can be taken now** (since v0.239.0; before it, no restorable Tier-2 copy) | nothing moves, nothing is recorded |
| 1 | `checking` | walks Tier 2 → Tier 1 → Tier 3 for the first copy younger than `update.backup_max_age` (v0.239.0) | nothing moves |
| 1 | `checking` | walks Tier 2 → Tier 1 → Tier 3 for the first copy younger than `update.backup_max_age` (v0.239.0) — *an app whose data lives in bind-mounted files walks second drive → OFF-SITE → own unit (decision 9, R-479, v0.241.0; `internal/backup/update_guard.go` `updateTierOrderBindData`; noted 2026-10-24)* | nothing moves |
| 2 | `backing-up` — only when no tier holds a fresh copy | `RunAppBackupNow`: this app's DB dump → volume dump → unit capture (marked proven current) → Tier-2 copy, whose failure is a WARN since v0.239.0 | refused with the backup's own error; nothing moves |
| 3 | `safety-dump` | `WriteUpdateSafetyDump` (R-361's undo copy) — **before the pin moves** | refused; nothing moves |
| 4 | `pinning` | the previous definition is copied aside and journaled, then the pin advances | pin put back |
| 5 | `pulling` | `compose pull` | **pin and definition PUT BACK** — nothing ran (Scenario E) |
| 5a | `copying` **(v0.263.0)** | `compose stop`, then each NAMED volume `cp -a` into `<volume>.pre-update-<stamp>` by a helper that writes a finished-marker last (bind folders never) | copies removed, **pin put back, the old version started again** — nothing new ran |
| 5b | `converting` **(v0.273.0, row added 2026-10-24)** | only on a step whose ladder entry carries `engine_conversion`: the box converts the PostgreSQL major (dump from the old engine, load into the new, check) — `internal/stacks/pgconvert.go` | the undo puts the old engine's data back |
| 6 | `starting` | `compose up -d --remove-orphans` | **UNDO** (below), HOLD only if the undo fails |
| 7 | `verifying` | the `.felhom.yml` health check through the existing probe **when it resolves to a container**, else 60 s of every container running and none restarting; bounded by `update.health_timeout` | **UNDO** (below), HOLD only if the undo fails. Before v0.263.0: stop + HOLD, the pin stays (Scenario F) |
| 7a | `undoing` **(v0.263.0)** | every copy validated (marker) BEFORE anything is poured back; volumes emptied and refilled; definition, pin and the pinned version's `.felhom.yml` record put back from the job's own copies; `up`; health with the OLD version's probe | HOLD, the sentence prefixed *„A frissítés nem sikerült, és az automatikus visszaállítás sem."* + the data state (`untouched` / `half` / `not_started`); copies kept |
@@ -2067,7 +2171,7 @@ the harness has proven it, is slice 6's.
> copy taken before the pin moved**, so the old version meets pre-migration data. Spiked by hand on
> 9202 the same day — §6.1a.
**Not gated here:** a multi-major jump (R-40). It fails health and is held honestly; stepping is slice 6.
**Not gated here:** a multi-major jump (R-40). It fails health and is held honestly; stepping is slice 6. *(Noted 2026-10-24: stepping is built — the ladder, one tested step per press, decision 14.)*
### 6.1a The undo (decision 15) — SPIKED BY HAND, then BUILT: controller v0.263.2 (2026-09-23)
@@ -2081,7 +2185,8 @@ only the live box could show, fixed the same day:** the undo's probe was never a
current probe held the app `unhealthy` (v0.263.1), and the "old" `.felhom.yml` taken at update time
was already the new one, because `.felhom.yml` flows in on every catalog sync (v0.263.2: the pinned
version's file is now recorded in `applied-meta/` whenever a version is pinned). **Residual (R-646):**
an app pinned before v0.263.2 has no such record until its next pin. **Two more, found by the chaos hour of
an app pinned before v0.263.2 has no such record until its next pin. *(Closed since: `BackfillAppliedMeta` records it at
start for every app CURRENT with the catalog, and names the rest — `internal/stacks/undo.go:646-652`; noted 2026-10-24.)* **Two more, found by the chaos hour of
2026-09-23 night (`audits/DRILL-night-2026-09-23.md` Part D):** the undo finds an app's volumes by their compose
label, and a restore recreates them WITHOUT it — so after any restore the undo copies nothing (R-658, P1); and a
held file-leg app is pointed at a restore that refuses a database-only copy, with no route left on a box without
@@ -2221,7 +2326,7 @@ breaks.
---
### 6.2 Slice 6, as it will be built (OPEN — R-450; ruled 2026-09-23, decisions 11–17)
### 6.2 Slice 6, as it will be built (~~OPEN — R-450~~ BUILT, R-450 closed 2026-10-05 — noted 2026-10-24; ruled 2026-09-23, decisions 11–17)
**The shape the rulings fix. The build order and costs are §6.4.** Rewritten 2026-09-23; the
earlier "as it would be built" draft keyed on the window of Q1's recommendation and on
@@ -2292,7 +2397,7 @@ retried. A working caller in this exact shape exists as evidence, not product:
**The per-box switch key is `app_update.unattended`**, default **true** (decision 12).
⚠ **NOT `auto_update` — that name is TAKEN, and by the very thing this must not collide with.**
`self_update.auto_update` / `self_update.auto_update_time` (`config/config.go` L280-281, default
`self_update.auto_update` / `self_update.auto_update_time` (`config/config.go` L280-281 then, L278-279 now; default
**04:30** at L422) are the CONTROLLER's own update. Two settings with that name, one meaning the
controller and one meaning apps, is the kind of collision that is only discovered by an operator who
turned off the wrong one. The app-scoped key is `app_update.*`. **And `stacks.update_window` is
@@ -2322,12 +2427,12 @@ what the part can do to a household's data if it is wrong, not how likely that i
| # | part | rulings / rows | cost | depends on | risk to customer data |
|---|---|---|---|---|---|
| **1** | **SHIPPED — controller v0.263.2, proven live on 9202 2026-09-23** (`audits/undo-live-2026-09-23/`). **The undo.** Keep the pre-update copies (compose, applied, pin, **old `.felhom.yml`**) until the undo is over; in `failAndHold`: pin back → DB up alone → **validate the copy's completion marker** → **empty-then-load in one transaction** (PostgreSQL: the dump's schemas dropped and recreated inside the load's transaction; MariaDB: every table dropped first, and a failed load HOLDS with a sentence saying the database is in neither state) → full start → **health with the OLD probe** → `undone`, else HOLD. A volume tar at safety-dump time for apps with no database server. Household page + event; the mail rides part 2. | 15; the audit's 8-point list | **4** | — | **HIGH by nature** — it writes the customer's database. Bounded: it only ever loads the copy taken seconds before, validated first, atomically on PostgreSQL; every failure mode ends in today's hold. **It also makes the manual button safer on its own**, which is why it goes first. |
| **1** | **SHIPPED — controller v0.263.2, proven live on 9202 2026-09-23** (`audits/undo-live-2026-09-23/`). **The undo.** Keep the pre-update copies (compose, applied, pin, **old `.felhom.yml`**) until the undo is over; in `failAndHold` *(as SHIPPED — noted 2026-10-24: the copy is the FOLDER copy decision 19 chose: every named volume `cp -a` into a sibling volume with a finished-marker, put back on failure; no database load — `internal/stacks/undo.go:26-36`; the text that follows is the design before the bake-off)*: pin back → DB up alone → **validate the copy's completion marker** → **empty-then-load in one transaction** (PostgreSQL: the dump's schemas dropped and recreated inside the load's transaction; MariaDB: every table dropped first, and a failed load HOLDS with a sentence saying the database is in neither state) → full start → **health with the OLD probe** → `undone`, else HOLD. A volume tar at safety-dump time for apps with no database server. Household page + event; the mail rides part 2. | 15; the audit's 8-point list | **4** | — | **HIGH by nature** — it writes the customer's database. Bounded: it only ever loads the copy taken seconds before, validated first, atomically on PostgreSQL; every failure mode ends in today's hold. **It also makes the manual button safer on its own**, which is why it goes first. |
| **2** | **SHIPPED — controller v0.264.0 + hub v0.120.0, proven live on 9202 and 9201 2026-09-23** (`audits/undo-fleet-2026-09-23/`). **The update sentences in the household's language** (R-606) and a mail when an automatic update is undone or held: events `app_update_undone` (warning) and `app_update_held` (error), one per app per outcome, on by default, mailed in the household's language with the app named in the subject; per-app cooldown on both legs. Leftovers: R-647. | R-606, 15 | **1** | — | none |
| **3** | **SHIPPED — controller v0.264.0, proven live on 9202 2026-09-23.** **A disabled notifier says so** (R-620), so the mail of part 2 can be measured on a scratch box at all. | R-620 | **0.5** | — | none |
| **4** | **SHIPPED — catalog `6db08a5`, night 2026-09-23** (`audits/DRILL-night-2026-09-23.md`): `update_ladder:` in `.felhom.yml`, one JSON entry per line (spiked on controller v0.266.0 and v0.267.0 first — the controller ignores the key); `check-test-record.py` (static, CI) + `check-test-record-move.py` (history + the registry for moved refs only, decision 23); the only writer `upgrade-test.py --write-ladder`; the 21 moves of 2026-09-22 backfilled from their records (21 proven). Not built: `steps/<to>.yml` (part 5 needs it once an app has two steps); `CompareImageRefs` did NOT move to the gate — the gate asks for a proven test instead, which is decision 13's own test. **The test record + the catalog gate + the memory check.** The harness writes the ladder entry (below) from its verdict record, including the memory watch's peak and marks; the gate refuses an image move with no entry, an entry with a `failed` verdict, or one with no memory watch; `CompareImageRefs`' rule moves here as the push-time safety net. **Backfill:** one entry per current pin — the 21 proven moves from their records, every other pin `needs_person: "never tested"`, which is honest and keeps them manual. A version move re-checks `mem_limit` against the watch's peak (the RomM follow-up: gate, not checklist, because the watch now produces the number). | 13, R-635 follow-up | **2.5** | the memory watch (shipped 2026-09-23) | none on a box — catalog-side only |
| **5** | **SHIPPED — controller v0.268.0 (`206b035`) + catalog `5ed599c`, proven live on 9202 2026-09-24** (`audits/ladder-2026-09-24/partD/`): romm 5.3.0/11.4 → 5.3.1/11.4 (its app step, from `steps/90dd9d68258286ef.yml`) → 5.3.1/11.8 (the engine step; `mariadb-upgrade` ran) in two presses, the seeded account read back after each, the page's „Hátralévő frissítési lépések" 2 → 1 → none. Step files: `templates/<app>/steps/<StepKey(to)>.yml` (sha256 of `to` as canonical JSON, 16 hex; `stacks.StepKey` = `ladder.step_key`), refused absent or wrong by `check-test-record.py` rule 4, written by `--write-ladder` when a step is superseded, 8 backfilled from the NEWEST commit naming each step's images (romm's from `f4eb94f`, not the OOM-looping `15f9ebf`). The box reads them from its `--depth 1` clone (the whole tree is there — measured). One press = one step; a missing step file refuses before anything moves; an installed version matching no entry jumps, logged by name (measured live: vikunja 2.5.0). Limitation: a step has no `.felhom.yml` of its own (R-664). **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/<to>.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape |
| **6** | **BOTH HALVES SHIPPED. BOX HALF — controller v0.269.0 + v0.269.1, proven live on 9202 2026-09-24** (`audits/night-2026-09-24/B/`): the compose that runs pins `tag@sha256` from the ladder entry that tested those refs; pins and records stay digest-free; the badge reads behind for a newer TESTED digest; **an installed app keeps its digest until a guarded Update moves it** (v0.269.0 let the sync move it — found live, fixed in v0.269.1, `CarryDigests`); Compose accepts the form for all 25 ladder apps. **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. **The box half (compare, render `name:tag@sha256`) is not built.** **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low |
| **5** | **SHIPPED — controller v0.268.0 (`206b035`) + catalog `5ed599c`, proven live on 9202 2026-09-24** (`audits/ladder-2026-09-24/partD/`): romm 5.3.0/11.4 → 5.3.1/11.4 (its app step, from `steps/90dd9d68258286ef.yml`) → 5.3.1/11.8 (the engine step; `mariadb-upgrade` ran) in two presses, the seeded account read back after each, the page's „Hátralévő frissítési lépések" 2 → 1 → none. Step files: `templates/<app>/steps/<StepKey(to)>.yml` (sha256 of `to` as canonical JSON, 16 hex; `stacks.StepKey` = `ladder.step_key`), refused absent or wrong by `check-test-record.py` rule 4, written by `--write-ladder` when a step is superseded, 8 backfilled from the NEWEST commit naming each step's images (romm's from `f4eb94f`, not the OOM-looping `15f9ebf`). The box reads them from its `--depth 1` clone (the whole tree is there — measured). One press = one step; a missing step file refuses before anything moves; an installed version matching no entry jumps, logged by name (measured live: vikunja 2.5.0). Limitation: a step has no `.felhom.yml` of its own (R-664). *(Closed since: a step's own `steps/<key>.felhom.yml` is read when present — `ladder.go:127-132`, v0.269.0; noted 2026-10-24.)* **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/<to>.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape |
| **6** | **BOTH HALVES SHIPPED. BOX HALF — controller v0.269.0 + v0.269.1, proven live on 9202 2026-09-24** (`audits/night-2026-09-24/B/`): the compose that runs pins `tag@sha256` from the ladder entry that tested those refs; pins and records stay digest-free; the badge reads behind for a newer TESTED digest; **an installed app keeps its digest until a guarded Update moves it** (v0.269.0 let the sync move it — found live, fixed in v0.269.1, `CarryDigests`); Compose accepts the form for all 25 ladder apps. **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. ~~**The box half (compare, render `name:tag@sha256`) is not built.**~~ *Built — the box renders the tested digest, carries it on sync, and the badge reads the tested digest (`internal/stacks/digest.go:15-24`, `updateorder.go`; noted 2026-10-24).* **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low |
| **7** | **SHIPPED — controller v0.271.0 (`9cf13a3`), proven on 9202 over four simulated nights 2026-09-24/25 and watched through the demo boxes' first real automatic night (`audits/DRILL-night-2026-09-25.md`).** Decisions 31–33 taken unattended. Gaps a scratch box cannot close: R-687; no resume after a restart: R-686. *(Before:)* **SPIKED 2026-09-24 (night, Part C) — the measured build brief is §6.4.2 below.** **The update leg in the chain + the automatic caller + the switch.** A leg that starts when the off-site leg has ENDED (on every exit path, skips included), one app at a time, one step per press, `app_update.unattended` default ON (decision 12), `stacks.update_window` removed, reads `UpdateRefusal.Reason`, remembers a failed step so it never re-presses it, and the full-system backup's gate waits for it until W+5h (decision 20). | 11, 12, 14, 15, 20 | **3** | 1, 2, 5 | medium — the only part that acts with nobody watching; everything above is what makes it safe |
| **8** | **SHIPPED — controller v0.265.0 + hub v0.121.0, proven live on 9202 2026-09-23** (`audits/cleanup-2026-09-23/`; a kernel `oom_kill` counter, not the sticky flag — `08` §6.2). **R-636** — the same OOM key re-firing escalates instead of staying one `warning` for six hours. | R-636 | **1** | — | none |
| **9** | **SHIPPED — controller v0.265.0, proven live on 9202 2026-09-23** (badge „Megállítva — visszaállítás szükséges" / "Stopped — restore needed", no Update button, 409 `held` unchanged). **R-625** — a held app stops offering an Update it will refuse. With the undo, holds become rarer; the lie on the page does not go away by itself. | R-625 | **0.5** | 1 | none |
@@ -2410,7 +2515,7 @@ During the db-dump leg every press was refused `busy` (transient, retried next p
update's own finish should refresh them.
- **An Update pressed on an app already at the head runs the whole guarded update** — backup, pull,
restart, `done` — with no refusal (R-679: navidrome restarted four times for nothing). The leg never
presses an app whose pin equals the head; the preflight should refuse with reason `current`.
presses an app whose pin equals the head; the preflight should refuse with reason `current` (as built the reason key is `already_current`, `update.go:432` — noted 2026-10-24).
- **The product does not remember a failed step.** After vikunja's undo the badge still offers the
same step; only the caller's memory kept it from a re-press. Decision 15 needs it on the box: the
failed `to` is recorded per app and the leg skips it until the catalog's ladder changes.
@@ -2492,6 +2597,24 @@ and E are the ones that unblock a decision; leg A is the one that takes the time
harness on DooPlex for the image-side ones.
### 6.4.x Decisions in code no section records [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
Read at `file:line` on controller `main` (`321df1f`) by the main session:
- **`build.sh --push` also refuses a DIRTY tree, a HEAD that is not `origin/main`, and a directory that is not a git
checkout** — `controller/build.sh:144-166`. §0 names only the tag-exists and unreachable-registry refusals.
- **The leg retries a passing refusal ONCE in the same night** (`busy`, `updating`, `deploying`, `migrating`,
`self_updating`), after `RetryWait` — not „the next pass/night" (`internal/stacks/unattended.go:65-66`, the pass loop).
- **The leg skips an app the household stopped** (`stopped_by_household`, R-721): an update would start it
(`unattended.go:416-418`).
- **A manual run of the chain** (R-705, the debug action) gives the leg a deadline of its start + the leg's normal
length (`backupwindow.UpdateLegLengthMin`), not W+5h (`unattended.go:232-238`).
- **Two preflight refusals are plain skips to the leg:** `already_current` and `engine_no_test`
(`internal/stacks/update.go:432,454`).
- **The render table (§5.4) has one more row:** when the catalog template's images cannot be read, the syncer leaves
the app's compose file alone for that cycle (`internal/sync/sync.go:622-627`); and the box's R-971 allow-list runs
before ANY compose write (`cmd/controller/main.go:421`, `SetAdmitFn`).
## 6.5 The drill catalog and the image store — the standing method for update drills
**Why this section exists.** On 2026-09-21 an afternoon session put a deliberately broken image into
@@ -2549,7 +2672,13 @@ well, or the box goes on following the live catalog and reports success. Filed a
is fixed, the drill procedure is:
1. save `controller.yaml` as `controller.yaml.pre-update-night`;
2. set `git.repo_url` (and `username`/`token` — the drill repo is private);
2. set `git.repo_url` (and `username`/`token` — the drill repo is private). **The box's login is the Gitea user
`drill-reader`** — not an admin, a read collaborator on `admin/app-catalog-drill` only, token scope
`read:repository`; it cannot write or push and sees no other private repo (measured: 200 / 403 / 403 / 404,
`audits/night-2026-10-22/P2-D6/01-token-limits.txt`). The token is `DRILL_CATALOG_TOKEN` and the account's password
`DRILL_READER_PW` in CC's `sec` store — never printed. *Operator ruling 2026-10-22 15:47, D6: a read-only login, not
a public drill repo (R-992, closed).* Since that night 9202 keeps `git.username`/`git.token` while it follows the
live catalog (the public repo ignores them), so pointing it at the drill is the URL plus step 3;
3. **remove `<data>/catalog-cache`**;
4. restart the controller, sync, **rescan** (R-607: a sync can answer „nincs változás" while the
catalog has moved, and the badge answers from the stale value until the rescan);
@@ -2574,7 +2703,9 @@ customer box has credentials for it. The store listens on the guest's loopback o
### 7.1 The record (slice 1)
`Manager.recordInstalledImages` (`felhom-controller/controller/internal/stacks/installed.go`) runs
after a successful compose up from `StartStack`, `RestartStack`, `UpdateStack` and `runComposeDeploy`,
after a successful compose up from `StartStack`, `RestartStack`, `UpdateStack` and `runComposeDeploy`
*(as built — noted 2026-10-24: `UpdateStack` is gone; the callers are `StartStack`, `RestartStack`, the deploy path, the
end of the guarded update and the end of the undo — `manager.go`, `deploy.go`, `update.go`, `undo.go`)*,
and writes `app.yaml`:
```yaml
@@ -2650,7 +2781,7 @@ Version strings stay in the logs, the API and the hub.
The 8th, immich's own ghcr build, is UNMEASURED — ghcr exposes no anonymous last-modified. So on
demo-hp today four apps read „Naprakész" over a database engine image that has demonstrably moved.
Digest-level comparison needs the catalog to record the digest at push time — **R-446**, put to the
operator as §3b **Q6**, recommended YES.
operator as §3b **Q6**, recommended YES. *(Built since for every ref with a ladder entry — `updateorder.go`, `digest.go`; the limitation remains only for a ref no entry records; noted 2026-10-24.)*
2. **~~Nothing enforces `catalog_since`.~~ Enforced by the pre-push hook since 2026-09-13 (R-452, `app-catalog-felhom.eu/scripts/check-catalog-since.py`); CI's shallow clone still skips it out loud.** A commit that moves an `image:` line and forgets the date
under-reports how far behind a box is. The gates runner fetches at `--depth 1` and has no parent
commit to diff against, so the gate needs a deeper fetch — **R-452**.
@@ -2675,13 +2806,18 @@ Version strings stay in the logs, the API and the hub.
refuses without a restorable, proven Tier-2 copy, backs up first when that copy is stale, takes a
safety dump, and holds an app that does not come up. **What stays true:** it still has no automatic
rollback (deliberately, §6.1) and can still attempt a multi-major jump the app will refuse (R-40) —
that now ends HELD rather than crash-looping behind a green button.
that now ends HELD rather than crash-looping behind a green button. *(Dated note 2026-10-24: a copy on ANY tier now
satisfies the precondition (v0.239.0), the box undoes a failed update itself (decision 15, v0.263.2), and the
ladder climbs one tested step per press (decision 14) — `internal/backup/update_guard.go`, `internal/stacks/undo.go`,
`ladder.go`.)*
7. ~~**An engine major can be applied without its datadir upgrade, and nothing notices.**~~ **CLOSED
2026-09-13 for MariaDB (R-459):** every `mariadb:` sidecar carries `MARIADB_AUTO_UPGRADE=1`, and the
harness shows the conversion running on the E3/E3b edges (§3 decision 5). **What stays true:** the
PostgreSQL half (R-463) has no equivalent — the image performs no `pg_upgrade` — and the
engine-major rule (§3 precaution 3, R-469) is what keeps both engines inside their major until
Slice 4 gives the Update button a backup.
Slice 4 gives the Update button a backup. *(Dated note 2026-10-24: MariaDB is lifted — its major is its own edge; a
PostgreSQL major may pass with a two-venue proof and an `engine_conversion` mark, and the box converts it
(v0.273.0) — `app-catalog-felhom.eu/scripts/check-engine-major.py`.)*
8. ~~**Only three of 53 apps have ever had an upgrade measured.**~~ **WIDENED 2026-09-21 to 21
EDGES ACROSS 19 APPS** (`audits/DRILL-update-night-2026-09-21.md`), on scratch guest 9202
through the product's own guarded Update, against a PRIVATE DRILL CATALOG so the live catalog
+13 -7
View File
@@ -128,7 +128,9 @@ the hub's e-mails follow.**
`expired` pinned to Hungarian so the language cannot become the oracle the text refuses to be.
Full design: `05-hub-architecture.md` §15. **R-555 closed** — the `language` allowlist entry is out
of `wire_contract_gate.py` and the gate now checks the field for real.
- **[DESIGN] Slice 3 Part A as planned — now built; the box half (Part B) is the remaining piece:** the hub stores a per-customer language, renders it into
- **[DESIGN] Slice 3 Part A as planned — now built; the box half (Part B) is the remaining piece** *(Part B is built
too — the controller reads `customer.language` as its start language since v0.256.0, `internal/config/config.go:149-154`;
corrected 2026-10-24)*: the hub stores a per-customer language, renders it into
`controller.yaml` next to `customer.id/name/domain/email` (`hub/internal/configgen/configgen.go`),
and the box uses it **only while the household has never chosen** (`settings.json` empty). The
household's own choice always wins; the hub's e-mails follow the language the box **reports**, which
@@ -184,7 +186,8 @@ bundles. They now judge the page **as rendered in each language** (`controller/s
- **hu** — the product speaks in „te". The converted copy carries formal („ön") forms (R-516). They are
not fixed by a localisation release (§1), so the gate counts them against `HU_FORMAL_CEILING` as a
ratchet: a new one convicts, fixing one lowers the ceiling. The ceiling followed the conversion:
6 (spike) → 10 (release A) → 12 (release B) → **16** (release C). **[FACT] Measured limit:** the stem
6 (spike) → 10 (release A) → 12 (release B) → **16** (release C) → … → **0** (R-516 item 12, the last eleven in the
te-form — `felhom-controller/controller/scripts/i18n_missing_gate.py:41`; corrected 2026-10-24). **[FACT] Measured limit:** the stem
list is short — on the release C pages it counts 4 formal forms, while a wider list („írja be",
„adja meg", „válassza", „biztosan eltávolítja", „engedélyezze", …) finds at least 22 keys (R-516).
- **en** — second person, plain: no „please", no „kindly".
@@ -201,7 +204,8 @@ vissza") — R-564. The hub copy gate and the catalog have no language concept
- **[DESIGN] The hub's operator pages** — the operator reads them; they stay English (decision 1).
- **[DESIGN] The first-boot wizard** (`controller/internal/setup/`, 8 pages, 95 strings) — out of scope
and obsolete (decision 4, 2026-09-17; `02-controller-module-map.md` L56). A household still reaches
it when bootstrap ingestion leaves `customer.id` empty (inventory §2.7). Its deletion is **R-554**.
it when bootstrap ingestion leaves `customer.id` empty (inventory §2.7). Its deletion is **R-554**. *(Deleted since:
`internal/setup/` is gone (v0.308.0); `bootstrap.WaitForConfig` shows a waiting page instead — corrected 2026-10-24.)*
- **[FACT]** App containers' own UIs (Uptime Kuma, PrivateBin…) — not Felhom's copy (R-516 items 5-6).
---
@@ -226,7 +230,7 @@ i18n:
```
Why: one app's copy stays in one file a reviewer sees whole; the controller's `yaml.Unmarshal` ignores
unknown keys (`internal/stacks/metadata.go` L316 — **verified: this repo constructs no `yaml.Decoder`
unknown keys (`internal/stacks/metadata.go` L316 then, L475 now — **verified: this repo constructs no `yaml.Decoder`
at all, so there is no `KnownFields` anywhere to turn strict**), so older controllers are unaffected;
a missing English field falls back field by field.
@@ -234,7 +238,7 @@ a missing English field falls back field by field.
reached only through `LocalizeStacks` / `LocalizeStackPtr` / `Stack.MetaFor`.
- **`For("hu")` is the parsed struct with `I18n` cleared and nothing else**, deep-compared against all
53 real catalog files (`TestMetaForHuIsIdentity`, fixtures in `internal/stacks/testdata/catalog/`).
53 real catalog files (the identity fixtures are still those 53; the catalog has 65 apps — noted 2026-10-24) (`TestMetaForHuIsIdentity`, fixtures in `internal/stacks/testdata/catalog/`).
§1 therefore holds by construction for catalog copy, and `LocalizeStacks` returns the INPUT SLICE
for Hungarian rather than a copy, so nothing on the Hungarian path is even touched.
- **Fallback is field by field, and blank counts as absent** (ruling 3). A half-translated app is a
@@ -307,10 +311,10 @@ for the English page test (R-565), arriving again in a different repo. So
`app-catalog-felhom.eu/scripts/check-copy-i18n.py` folds and matches word-bounded stems, with a
positive and two negative controls printed on every run.
**[FACT] The gate**, fifth row of `catalog_gates.py`, static, in the pre-push hook:
**[FACT] The gate**, fifth row of `catalog_gates.py` (the sixth since — corrected 2026-10-24), static, in the pre-push hook:
1. **FREEZE** — every Hungarian copy string equals `scripts/copy_freeze/hu.json`, captured in its own
commit before any translation. Runs on all 53 apps whatever scope is named. A new app must be
commit before any translation. Runs on every app in the freeze file (65 on 2026-10-24) whatever scope is named. A new app must be
admitted with `--add-app NAME --reason "…"`.
2. **STRUCTURE** — the `en` block may carry copy fields and nothing else; every key-matched entry must
have a Hungarian twin, or it is INERT on the box and the translator never learns.
@@ -367,6 +371,8 @@ here; every sentence this product writes is display, and a decision reads a sign
a sentinel for `errors.Is` — never `fmt.Errorf("%w: …")`, which would prepend the sentinel's own text
to the customer's sentence.
**2026-10-21 night (W25, controller `5776d54`, unreleased):** the off-site page's „Utolsó hiba" no longer shows the run's raw text to the household. It is built from the classifier's class and the run's own result (the failed apps by name, „the other apps are fine" when true, „the box tries again the next night" when a retry can help). The raw text — restic's included — stays verbatim in the log and, sanitised, in the operator's event; the „foreign error prints verbatim" rule below now holds for those two places only on this page.
**[FACT] One exception, with an end date:** a box upgraded to 0.251.0 carries the OLD persisted note
with no kind until its next off-site run, so `offboxWarningDisplay` keeps the substring test for
`kind == ""` only. **Slice 2 (R-557) must not translate that producer until R-570 closes.**
+18 -8
View File
@@ -11,7 +11,7 @@
> | | |
> |---|---|
> | **Status** | **NOT RATIFIED — a PROPOSAL with operator rulings, corrected by the 2026-10-04 spike (§7.1, C1–C12); §8 step 2 BUILT 2026-10-04 (§8.1).** Ratification is Viktor's review, not an editor's. |
> | **Status** | **NOT RATIFIED — a PROPOSAL with operator rulings, corrected by the 2026-10-04 spike (§7.1, C1–C12); §8 step 2 BUILT 2026-10-04 (§8.1); steps 2–6 are BUILT by §8's own list (corrected 2026-10-24 — the ratification is still the operator's).** Ratification is Viktor's review, not an editor's. |
> | **Written** | 2026-10-04, by the reviewer (project Claude), before any spike. |
> | **Verified against** | felhom.eu `d07a1a9` · felhom-controller `99a1497` (v0.290.0) · felhom-agent `d766666` (v0.138.0) · hub v0.128.0 |
> | **Freshness** | **CURRENT** as of 2026-10-04. The spike `TASK-backup-close-and-os-updates-spike-2026-10-04` adds measurements here as `[FACT]` and corrects every claim it disproves. Mark this file STALE when it falls behind what the product does. |
@@ -76,10 +76,14 @@ boxes installed. Nobody keeps a hand-written list.
- **A BYO host** (the owner brought their own Proxmox). `[FACT]` The installer leaves its repositories
alone: *"apt repo alignment skipped (byo — the owner manages repos)"* (`scripts/felhom-host-install.sh`,
`align_apt_repos`). `[PROPOSAL]` On a BYO box we update the guest and Docker only, never the host.
**[DESIGN, operator ruling D43 = A, 2026-10-24 — committed agent `c8d3c17`, unreleased]** The host's root-key-only sshd
drop-in (D4, R-982) is an APPLIANCE-only file of the config bundle: written only when the root-owned install record
says `appliance`; on a `byo` record nothing is written and a copy the bundle wrote earlier (this path, our first line)
is removed; a box whose record proves nothing is left as it is (R-1257).
- **The Proxmox MAJOR upgrade** (PVE 9 → 10). It is a later step of its own, drilled first (R-808 item 5).
- **`cloudflared` and the other infrastructure images** (traefik, filebrowser). **[FACT] (C8)** They are containers in
the guest, pinned in the controller (`internal/infra/infra.go:26`: `cloudflare/cloudflared:2026.6.0`, since
2026-06-11) and baked into the golden; upstream was `2026.9.3` on 2026-10-04. They move only by a controller release
2026-06-11 — *`2026.9.3` today, corrected 2026-10-24*) and baked into the golden; upstream was `2026.9.3` on 2026-10-04. They move only by a controller release
— the app-image question (`09`), not an OS package. The gap is R-838.
---
@@ -101,9 +105,10 @@ gets the Docker engine that was current when its golden was baked, and keeps it.
~~**[FACT] The agent may not run `apt` today, with one exception.** Its sudoers allowlist
(`felhom-agent/configs/felhom-agent.sudoers`) holds one apt line: `apt-get install -y -q dnsmasq`.~~
**[FACT] (C1) The agent may run `apt-get install` for two named packages:** `felhom-agent.sudoers:58`
(`apt-get install -y -q dnsmasq`, used by `internal/lanresolver/lanresolver.go:107`) and `:194`
(`apt-get install -y -q wireguard-tools`, used by `internal/wgtunnel/manager.go:628`). Nothing else.
**[FACT] (C1) The agent may run `apt-get install` for two named packages:** `felhom-agent.sudoers:68`
(`apt-get install -y -q dnsmasq`, used by `internal/lanresolver/lanresolver.go:133`) and `:228`
(`apt-get install -y -q wireguard-tools`, used by `internal/wgtunnel/manager.go:659`). Nothing else. *(Line numbers
corrected 2026-10-24.)*
`apt` as root runs package scripts as root, so a broad `apt` grant is a full root grant. §5.4
proposes how to avoid that.
@@ -250,7 +255,7 @@ approval of that day stays in force (decided by CC unattended — operator may r
versions. A broken-into hub can choose an older version or no version. It cannot make a box install a
package that the publisher did not sign.
### 5.4.1 The root wrapper's interface — DRAFT (2026-10-04, design only, nothing installed) `[PROPOSAL]`
### 5.4.1 The root wrapper's interface — DRAFT (2026-10-04, design only, nothing installed) `[PROPOSAL]` — *BUILT and installed since (§8.1); the agent's sudoers holds only the `--plan` line, no `--repair-only` (`felhom-agent/configs/felhom-agent.sudoers:337`); noted 2026-10-24*
`felhom-os-apply` — root-owned (`0755 root:root`), installed by the installer beside `felhom-selfupdate-guarded`;
the agent's sudoers gets exactly `/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json` and
@@ -315,7 +320,7 @@ PID 1 or `lxc-start`, or a kernel was installed.
Decision 96. Before it, the signed `agent_update` replaced only the binary; wrappers, units and sudoers lines reached
an installed box by reinstall or by hand.
- **One source of truth.** `BUNDLE_FILES` in `felhom-os-apply` is the ONE table of root-owned paths (22: sudoers ×2, the
- **One source of truth.** `BUNDLE_FILES` in `felhom-os-apply` is the ONE table of root-owned paths (22 then, **32** counted 2026-10-24 — plus the vzdump-partial-clean, thin-check, internal-place and priv-apply wrappers, the guest hook, the shared-parent unit, the GRUB one-shot pair and the sshd root-key-only drop-in; sudoers ×2, the
five wrappers, the crash guard and its units + config, the agent and rollback units, the start-limit drop-in, the mgmt
watchdog and its tmpfiles/units, the OOB belt's four files). `scripts/build-config-bundle.py` builds the bundle from it
reproducibly; `release-agent.sh` publishes it beside the binary; the hub vouches its sha with the agent (exact-name
@@ -393,6 +398,9 @@ must never overlap a backup, a restore-test or a self-update.~~
- **Operator:** a hub event for each update and each failure; a fleet view showing each box's OS release,
how far behind it is, and whether it needs a reboot. A box that is more than N days behind the newest
approved release raises an alarm (`08`).
- **[FACT, added 2026-10-24 — hub v0.169.0] The System page's „Upload line" cell** shows the best of the last SUCCESSFUL
weekly line test (R-1247), amber when it is older than 15 days or was never measured after an attempt — judged on
`last_ok_at`, never on the attempt time (`hub/internal/web/system.go` `lineStaleAfter`, `lineTestCell`).
- **[FACT, hub v0.132.0] The System page** (`/system`, R-852, decision 89): one row per box — ring and switch with
buttons, the tunnel, host Proxmox / running and next-boot kernel / Debian / release / pending / not covered / held /
reboot needed / `kernel.panic` / oops / crash restarts / the guard, guest Debian / release / pending / restart needed,
@@ -480,7 +488,9 @@ Decision 88 (R-851): *"Yes, but maybe not indefinitely."* Evidence `audits/os-do
**What it updates:** the host's Proxmox USERSPACE packages (pve-manager, qemu-server, pve-container, lxc-pve,
libpve-*, proxmox-widget-toolkit, the ceph client libraries …) — the 77–78 packages behind on the demo boxes on
2026-10-07. **What it never touches:** a kernel, boot, firmware or microcode name (`HOST_SLOW_RE`: `proxmox-kernel*`,
`proxmox-default-kernel`, `pve-kernel*`, `pve-firmware`, `shim*`, `grub*`, `systemd-boot`, `*-microcode`, `efibootmgr`)
`proxmox-default-kernel`, `pve-kernel*`, `pve-firmware`, `shim*`, `grub*`, `systemd-boot`, `*-microcode`, `efibootmgr`,
and — read 2026-10-24 in `felhom-os-apply` — `linux-(image|headers|kbuild|modules|base)*`, `firmware-*`,
`proxmox-secure-boot-support`)
— that is R-836's lane (decision 164). No reboot.
- **Wrapper** (`felhom-agent/configs/felhom-os-apply`): layer `pve`, lane `slow` only. Origin `Proxmox Debian
@@ -20,8 +20,10 @@ workspace-root `CLAUDE.md` § „Artifact taxonomy". **[FACT]** The spec shape i
class). Validation is project Claude's unless CC is asked (`CLAUDE.md`, same section).
**A session with no spec** (a `/goal`, a night brief, „work the register") inherits the discipline a
spec used to carry from `.claude/rules/unprompted-work.md` — **[FACT]** five byte-identical copies:
the workspace root and each of the four repos (the file's own header says „change all five or none").
spec used to carry from `.claude/rules/unprompted-work.md` — **[FACT]** five copies:
the workspace root and each of the four repos (the file's own header says „change all five or none"). *(Measured
2026-10-24: they are NOT byte-identical — the felhom.eu and felhom-controller copies carry one more paragraph, „A
version number is pushed once" (added 2026-10-15); the review round 2 session aligned the other three, see its report.)*
## 2. Where an instruction lives
@@ -29,7 +31,7 @@ the workspace root and each of the four repos (the file's own header says „cha
|---|---|---|
| Cross-repo facts and standing rules (production host, gates, secrets, CHANGELOG/REPORT) | workspace-root `CLAUDE.md` | every session in the workspace |
| One repo's stable orientation | `<repo>/CLAUDE.md` | when a file in that repo is touched |
| Path-scoped rules (a subsystem's traps) | `<repo>/.claude/rules/*.md` | when a matching path is touched |
| Path-scoped rules (a subsystem's traps) | `<repo>/.claude/rules/*.md` | when a matching path is touched — a file with `unconditional: true` (e.g. `live-probes.md`) or without `paths:` (the `unprompted-work.md` copies) loads every session (corrected 2026-10-24) |
| A procedure with steps (build, test, diagnose, write for the operator) | a skill, `felhom.eu/skills/<name>/SKILL.md` | when its trigger matches, or by name |
| Current state, decisions, open work | `CONTEXT.md`, `STATUS.md`, `documentation/backlog/OPEN-ITEMS.md` | read on demand — never copied into an instruction file |
| Durable cross-session facts about the operator and the machines | the memory folder (`.claude-memory/`, index `MEMORY.md`) | the index each session; files on recall |
@@ -38,8 +40,9 @@ Which rung a piece of material belongs on is the doc-authoring skill's §3 („W
material sits") and §6 („One rule, one home"). This page does not repeat them.
**[FACT] Skills** are written in `felhom.eu/skills/`, installed by
`python3 felhom.eu/scripts/install_skills.py` as symlinks into `~/.claude/skills/` (so a repo edit is
live at once), and checked by `python3 felhom.eu/scripts/check_skills.py` (shape, the 150-line limit
`python3 felhom.eu/scripts/install_skills.py` as symlinks into `~/.claude/skills/` (so an edit to an INSTALLED skill is
live at once; a NEW skill folder is live only after `install_skills.py` runs again — measured 2026-10-24:
`skills/felhom-web-design` has no link yet), and checked by `python3 felhom.eu/scripts/check_skills.py` (shape, the 150-line limit
and its one grandfathered entry, R-394). The list is the folder; `skills/SOURCES.md` names where each
skill's material came from.
+19 -9
View File
@@ -36,6 +36,7 @@ decision (R-925).
| g | a compromised Gitea / registry / build | push to the catalog, the images and the installer tag |
| h | a thief with the box | the disks |
| i | Cloudflare | the plaintext of tunnel traffic — **accepted** (R-904) |
| j | **added 2026-10-23:** a hacked homelab app pod on DooPlex (happened: Plex2Shell, HM-040) | the pod network with free egress: every cluster Service, the node's ports (NFS, kubelet, k3s API), the LAN masqueraded as DooPlex; no service-account token |
## 4. Entry points
@@ -46,8 +47,8 @@ scripts are held with the review's working notes.
| component | routes | no credential | listeners |
|---|---|---|---|
| controller | 231 (+8 in setup mode) | 21, of which 7 change state, each behind its own password, code or token | port 8080 on the guest's Docker networks (not published); mail relay 2525/2465/2526 |
| hub | 121 (1 health, 53 API, 67 web — v0.152.0 added the two token routes; both route tables are now pinned to the source by tests, R-952) | appliance register (rate-limited, by design), re-enroll (only while the operator has armed recovery mode), login, health | one ingress, path `/`, geo filter (HU, IE, LAN) |
| agent | 38 local-API routes (`POST /host/poweroff` added 2026-10-13, R-998) | 0 | local API on the island bridge; WireGuard; poke (tunnel only); felhom-sshd (tunnel only, measured) |
| hub | 121 (1 health, 53 API, 67 web — v0.152.0 added the two token routes; both route tables are now pinned to the source by tests, R-952) — *the API table has 54 entries since `/box-config/` (v0.161.0) and `/preferences` (v0.165.0); counted from `hub/internal/api/routes.go`, 2026-10-24* | appliance register (rate-limited, by design), re-enroll (only while the operator has armed recovery mode), login, health, and the public self-bind page `/bind/<token>` (a capability link, added to this list 2026-10-24) | one ingress, path `/`, geo filter (HU, IE, LAN) |
| agent | 41 local-API routes on `main` (counted 2026-10-24 in `felhom-agent/internal/localapi/server.go`; `POST /host/poweroff` added 2026-10-13, R-998; `POST /host/line-test/run` added 2026-10-24, R-1250, unreleased) | 0 | local API on the island bridge; WireGuard; poke (tunnel only); felhom-sshd (tunnel only, measured) |
**Outbound fetches that bring code or config onto a box:** the catalog (git, `main`), app images, the four infra
images, the controller image (by tag), `controller.yaml` from the hub, the agent binary and config bundle (signed),
@@ -57,11 +58,11 @@ OS packages (exact versions, apt-signed), the installer script (by tag, R-980).
| boundary | verdict | evidence |
|---|---|---|
| end-user ↔ apps | holds, with gaps | traefik routing and header stripping hold; homepage has the Docker socket and no gate (R-981) |
| end-user ↔ apps | holds, with gaps | traefik routing and header stripping hold; homepage has the Docker socket and no gate (R-981). **Re-check (2026-10-24, review round 2):** the socket is gone from the homepage template (catalog `templates/homepage/docker-compose.yml:21`) and the box's allow-list is empty, so a socket bind is refused (`felhom-controller/controller/internal/privpolicy/allowlist.json`) |
| **added 2026-10-14 (R-1033, P1):** a visitor ↔ Cloudflare over plain http | holds on boxes with controller ≥ 0.312.0 | before: every public name answered `http://` (login pages too), and Sonarr/Radarr/Prowlarr redirected https visitors to an absolute `http://` login; a password could cross an untrusted network unencrypted (attacker: anyone on the visitor's network). Now traefik sends every `http://` request to https (301) before an app sees it, HSTS one day; boxes below 0.312.0 (Tester 2 at 0.292.0) keep the gap until the floor delivers it. `audits/https-only-2026-10-14/` |
| **added 2026-10-15 (R-1045):** a stranger ↔ a fresh Plex | holds on controller ≥ 0.313.0; on older ones only while the household does not press „Kész" before claiming | an unclaimed Plex answers everything to a private address with no token, and through the tunnel every request comes from one. The setup gate (catalog `9a23ee5`) closes the public path; 0.313.0 opens it only when Plex itself says claimed. A status probe meant to cover older controllers OPENED an unclaimed Plex while it started (`audits/media-build-2026-10-15/G7/`) and was removed |
| customer ↔ controller UI | partly | bcrypt and session hold; small gaps (R-983); family sessions survive a reset (R-976) |
| controller ↔ agent | partly | every route bound to the caller's guest; a compromised controller can still flush restore points (R-973), void the recovery code (R-978), and — since agent v0.155.0 — switch the whole machine off once per power-on, holding the hub's silence alarms for up to 14 days (R-998; mitigations and residual risk: `14-host-power-off.md` §5) |
| customer ↔ controller UI | partly | bcrypt and session hold; small gaps (R-983); family sessions survive a reset (R-976). **Re-check (2026-10-24, review round 2):** both fixed — a password change or a claim/reset ends every dashboard, family and gate session (controller v0.324.0, `internal/web/auth.go:352-366`) |
| controller ↔ agent | partly | every route bound to the caller's guest; a compromised controller can still flush restore points (R-973), void the recovery code (R-978) *(re-check 2026-10-24: both narrowed — a cadence cap on `POST /backup` since agent v0.160.0, `internal/localapi/backup_cap.go`; a new escrow package is uploaded only at the claim since agent v0.160.0, `escrow_ceremony.go`)*, and — since agent v0.155.0 — switch the whole machine off once per power-on, holding the hub's silence alarms for up to 14 days (R-998; mitigations and residual risk: `14-host-power-off.md` §5) |
| **added (R-1007):** controller → an app's own job queue | holds — nothing new | the `after_unclean_boot` commands run inside the app's container through the docker socket the controller already holds, and read the app's database with a read-only session: no key into the household's account, nothing a compromised controller could not already do (exec in any app, read its database). The household's lost-file names stay on the box (`app.yaml`, the app page); the operator event carries counts only, and a failing command's output is never logged (security review of controller `8d862e4`, fixed in `3b85a0c`) |
| **added:** agent ↔ root on the host | holds by the sudoers | every root verb is one regex-anchored line or a root wrapper that re-checks its input (`03` §3); agent v0.156.1 adds one delete: `felhom-vzdump-partial-clean` removes only a DEAD half-written vzdump of the named guest on a `dir` storage (no vzdump on the host, nothing holding it open) — a compromised agent gains the deletion of a leftover that is not a backup |
| agent ↔ hub | holds for destructive ops | signed jobs verified in the documented order; unsigned spec has no size bounds (R-984) |
@@ -69,12 +70,21 @@ OS packages (exact versions, apt-signed), the installer script (by tag, R-980).
| controller ↔ hub | **did not hold until 2026-10-12** | the global key was public (R-950, rotated) and three routes needed no key (R-951, fixed); per-box keys are bound to their box (R-955) |
| box ↔ PBS | holds as written | ciphertext only; the zero-knowledge wording is stronger than a live box allows (R-985) |
| guest ↔ Proxmox host | holds | the guest has no Proxmox credentials (measured) |
| hub ↔ Cloudflare API | holds for the zone check | tokens are stored unsealed in the hub database (R-975); R-137 open |
| hub ↔ Cloudflare API | holds for the zone check | tokens are stored unsealed in the hub database (R-975); R-137 open. **Re-check (2026-10-24, review round 2):** the tokens are sealed (`enc:v1:`) since hub 0.168.0, deployed 2026-10-24 (`hub/internal/store/r975_config_token_seal.go`) |
| **added:** app ↔ app (shared Docker daemon) | partly | separate containers and networks; any docker-socket app is root in the guest (R-981) |
| **added:** app ↔ controller | partly | apps share a network with the controller; dashboard routes still need a session |
| **added:** LAN ↔ box | partly | root password SSH on the host from the LAN (R-982) |
| **added:** hub ↔ box (inbound) | **does not hold** | `01` §4 says the box exposes no control endpoint; a hub-key route can replace the box's settings (R-970) |
| **added:** catalog ↔ every box | **design gap** | a catalog push reaches every box with no check on the box (R-971) |
| **added:** LAN ↔ box | partly | root password SSH on the host from the LAN (R-982). **Re-check (2026-10-24, review round 2):** the config bundle's key-only drop-in shipped in agent v0.163.0 (`felhom-agent/configs/felhom-os-apply:230`) |
| **added:** hub ↔ box (inbound) | **does not hold** | `01` §4 says the box exposes no control endpoint; a hub-key route can replace the box's settings (R-970). **Re-check (2026-10-24, review round 2):** the config routes are removed and no route takes the hub key — pinned by `felhom-controller/controller/cmd/controller/r970_no_hub_key_door_test.go` (`01` §4 says the same) |
| **added:** catalog ↔ every box | **design gap** | a catalog push reaches every box with no check on the box (R-971). **Re-check (2026-10-24, review round 2):** the box checks every rendered compose against its allow-list before any write since controller v0.328.0 (`internal/privpolicy/privpolicy.go`) |
| **added 2026-10-23 (R-1241):** a hacked app pod on DooPlex ↔ Felhom's secrets | **does not hold**; not used in the Plex2Shell window | two no-login doors: the Longhorn NFS export (the hub's unencrypted nightly copies with the unsealed tokens of R-975; by a published file-handle escape the whole sda1 disk, the operator signing keys included — not exercised) and the Longhorn API (backup a volume to any target). Plex2Shell, 2026-10-17 → 10-23: **NOT REACHED** — 0 new NFS connections in the window, no off-schedule backup or volume; the hub, the k8s API and the kubelet refuse a caller with no credential (measured). The export was narrowed to the pod network on 2026-10-23; the pod side stays open (`audits/plex2shell-felhom-impact-2026-10-23/`). **Felhom half, 2026-10-23 night:** the last plain tokens sealed in hub 0.168.0 (deployed 2026-10-24 — corrected 2026-10-24; `audits/hub-token-seal-2026-10-23/`); the two plain Longhorn copies of the hub volume already in the folder are R-1243's decision. A household box's version of this attacker (a hacked app sending or mining) is what the upload watch (R-1244, released 2026-10-24 in controller v0.342.0 + hub v0.169.0 — corrected 2026-10-24) detects |
**[DESIGN — the operator picked a code change 2026-10-25 (D44); row R-1281]:** the file manager's one
sign-in (R-1100) — FileBrowser trusts its proxy header blindly, and every app container can reach `filebrowser:80`
directly; the only guard is that the header NAME is a per-box secret, which traefik's forwardAuth strips from every
client request and sets only for a browser holding the dashboard's files pass
(`felhom-controller/controller/internal/infra/filebrowser_sso.go:15-22`, `internal/web/files_sso.go`). The name file's
creation race was fixed by review round 2 (controller `b01bb4b`). The pick: FileBrowser joins only the network traefik uses, so no app
container reaches it directly. Not built yet.
## 6. What we accept (do not re-report)
@@ -22,7 +22,7 @@ came back (BIOS „last state", inferred), its power button did (dashboard in 42
| 2 | controller `POST /api/server/poweroff` (`internal/web/poweroff_handlers.go`) | dashboard login (`RequireAuth` + CSRF), POST only, the word again, the controller's own backup/restore/app-update check, the `host_poweroff` feature gate (MinAgent 0.155.0) |
| 3 | agent `POST /host/poweroff` (`internal/localapi/hostpoweroff.go`) | `withGuest` token scope; the guards (§3); the record (`internal/poweroff`) |
| 4 | agent → hub | a host report carrying `deliberate_off.requested_at`, waited for until the hub ACCEPTED it (≤ 60 s, `Loop.SetReportedHook`) |
| 5 | agent | `pct shutdown <vmid> --timeout 120`, then `systemctl poweroff` (two exact sudoers vectors, `FELHOM_POWEROFF`) |
| 5 | agent | `pct shutdown <vmid> --timeout 120`, then `systemctl poweroff` (two exact sudoers vectors: `pct ^shutdown [0-9]+ --timeout [0-9]+$` in `FELHOM_GUESTHOOK`, and `systemctl poweroff` in `FELHOM_POWEROFF` — `felhom-agent/configs/felhom-agent.sudoers:81-90`; corrected 2026-10-24) |
| 6 | hub (`store/deliberateoff.go`) | `hosts.deliberate_off_at`; silence alarms held 14 days (§4) |
| 7 | the household | presses the machine's power button; the first report of the new boot has no `deliberate_off` → the hub clears it |
@@ -33,11 +33,11 @@ came back (BIOS „last state", inferred), its power button did (dashboard in 42
| first 10 min after boot | `too_soon_after_boot` | `TooSoonAfterBoot` |
| one accepted request per boot (file keyed by `/proc/stat` btime) | `already_requested_this_boot` | `SecondRequestSameBootRefused`, `poweroff/state_test.go` |
| any other guest or VM running (`/proc`: `lxc-start -n N`, `kvm -id N`) | `other_guests_running` | `RefusedWhileOtherGuestsRun`, `RunningGuestsFromProc` |
| heavy-operation gate held (backup, restore test, OS step, sweep, trim) | `backup_running` / `restore_test_running` / `os_update_running` / `host_busy` | `RefusedDuringBackup` |
| heavy-operation gate held (backup, restore test, OS step, sweep, trim, and since agent v0.164.0 the weekly upload line test — corrected 2026-10-24) | `backup_running` / `restore_test_running` / `os_update_running` / `host_busy` | `RefusedDuringBackup` |
| backup job, controller swap, drive format, NAS verify, escrow ceremony, Proxmox vzdump | `backup_running` / `controller_update_running` / `drive_operation_running` / `escrow_ceremony_running` | `RefusedDuringVzdump`, `FailsClosedWithoutTaskReader` |
| after acceptance: the heavy gate is TAKEN and kept; before each step the jobs and other guests are read again — a job waits ≤ 15 min, another guest cancels at once, a job still running cancels | (log) | `JobStartedAfterAcceptance`, `OtherGuestStartedAfterAcceptance` |
**„A Proxmox vzdump runs” (agent v0.156.1, R-1008):** a vzdump task for the guest in Proxmox's active list whose worker process still lives — Proxmox's own rule (the UPID's pid + pstart against `/proc/<pid>/stat`). After a power cut the dead task stays listed for ~30–60 s; it no longer counts. Doubt (an unparsable task, an unreadable `/proc`) still refuses. The same answer serves the start-up stale-lock pass, the guest-power watchdog and the controller supervisor (`stalelock_inflight_test.go`).
| after acceptance: the heavy gate is TAKEN and kept; before each step the jobs and other guests are read again — a job waits ≤ 15 min, another guest cancels at once, a job still running cancels | (log) | `JobStartedAfterAcceptance`, `OtherGuestStartedAfterAcceptance` |
**[DESIGN]** A cancelled power-off leaves the box on (gate released) — better a box that did not switch off than a
backup, update or format cut in the middle. **[DESIGN]** A failed guest shutdown does not stop the power-off: Proxmox's
@@ -86,5 +86,15 @@ alternative (an out-of-band confirmation) needs a second channel the box does no
## 6. Not done here
- The website FAQ line („to turn it off: Beállítások → Rendszer → A gép kikapcsolása") — a later website task (R-1000).
- ~~The website FAQ line („to turn it off: Beállítások → Rendszer → A gép kikapcsolása") — a later website task (R-1000).~~ Published — `website/gyik.html:128,603` (corrected 2026-10-24).
- The BIOS „after power loss" value of the N100 was not read (no monitor); the household text promises only the button.
## Decisions in code no section records [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
- **A re-enroll while the household power-off hold is active counts as „gone"** (a reinstall over the old host), not
„live" (R-1154; `hub/internal/api/reenroll.go:67-68`).
- **Setting and clearing the deliberate-off time each record an info event** — `host_deliberate_off`,
`host_deliberate_off_ended` — on the customer's event list; neither is mailed (`hub/internal/api/deliberate_off.go:12-13`).
- **The confirmation word is accepted in three forms:** „KIKAPCSOLÁS", the unaccented „KIKAPCSOLAS" (a foreign keyboard)
and „TURN OFF" (`felhom-controller/controller/internal/web/poweroff_handlers.go:28-29`).
+22 -1
View File
@@ -165,7 +165,12 @@ directory) with that tag. Facts the link code needs:
**[FACT]** An unlisted writable `${…_PATH}` bind is MANDATORY (every tier, off-site included); a plain host path is in
no capture set (`stacks/classify_binds.go` `classifyRoot`). Measured over the live templates: Transmission's unlisted
`watch` bind goes off-site; the download folders are `optional`, so a second drive gets a nightly copy of every
download (R-1019). **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's
download (R-1019). *(Fixed since: catalog `c50adfc` marks the download and watch folders `excluded` —
`app-catalog-felhom.eu/templates/transmission/.felhom.yml:46-51`; noted 2026-10-24.)* **[DESIGN — the operator picked a code change 2026-10-25 (D44); row R-1283]:** the same commit also
marks the download folders of NZBGet, SABnzbd and MeTube `excluded` (`templates/{nzbget,sabnzbd,metube}/.felhom.yml`).
NZBGet and SABnzbd stay excluded (their files can be fetched again). The pick for MeTube: `optional`, so the
second-drive copy carries it and the off-site copy does not — a MeTube video can be gone from its source site. Not
built yet. **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's
media is no single app's data; whether the library gets its own household-level backup is a separate decision.
## 8. Removal and export assume one drive per app
@@ -192,6 +197,13 @@ household's existing files under `media/torrents` stay where they are.
Evidence: `audits/media-drive-events-2026-10-14/` (K1–K7). „Idempotent" = a second run changes nothing (K2: run 2 left every entry, folder and library as it
was). „Household" = a step only the household can take.
> *Dated note 2026-10-24 (review round 2): this table is the 2026-10-14 MEASUREMENT („today" = then). As built (§11):
> on a drive event the controller only REWRITES the binds; the media apps restart when the household presses
> „Újraindítás most" or „Később" (03:30) — the file browser alone recreates itself (`internal/stacks/media_library.go`,
> `internal/web/media_events.go`); a safe removal keeps the drive REGISTERED and its return re-binds with no household
> step (`SetEjected`, `media_events.go`); and since controller `b01bb4b` (unreleased) neither restart recreates an app
> the household stopped (R-721).*
| event → app | action | measured call | idempotent | household | evidence |
|---|---|---|---|---|---|
| **added** — every media app | the compose gains the drive's bind → `up -d` recreates it (the compose CHANGED, so today's restart works) | controller compose `up -d` | yes | no | K5-1: 1–10 s per app |
@@ -258,3 +270,12 @@ save path when its client is saved before the wiring; **a Plex status probe (`/l
unclaimed Plex because Plex answers 401 for a moment while it starts — never gate on a status a starting app also
returns** (G7f).
**[DESIGN — the operator picked a code change 2026-10-25 (D44); row R-1282]:** the „Később"
03:30 media restart is an IN-MEMORY timer — a controller restart forgets it and only the notice stays
(`internal/web/media_events.go` `handleMediaRestart`, `time.AfterFunc`). The pick: the controller stores the due
time and runs the restart after a restart of its own. Not built yet.
**[DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]:** the Sonarr/Radarr API key is READ from
`/config/config.xml`; the „set at install" option §6 calls cleaner was not built (`internal/mediawire/mediawire.go:336-345`).
@@ -15,8 +15,8 @@
| Place | Found by | Identified by | Mounted | What lives there | The backup page says |
|---|---|---|---|---|---|
| Host root / PVE `local` | PVE storage `local` [FACT] A/internal/storage/observe.go:424; never a candidate [FACT] A/internal/storage/role.go:53-79 | `path:` / `store:local` [FACT] A/internal/storage/durableid.go:54-67; role system | host `/`, `/var/lib/vz` | the **local whole-guest copies** (vzdump, retention 3) [FACT] E/scripts/felhom-host-install.sh:730-734; `07` §6.1 | `backup.target.degraded` „…ugyanazon a lemezen van, mint a rendszer…" [FACT] C/internal/web/backup_target_offer.go:78-100 |
| Guest `mp0` `/var/lib/felhom` with folders `docker/` and `sys_drive/` (bound to `/var/lib/docker`, `/mnt/sys_drive`) | config default `system_data_path=/mnt/sys_drive` [FACT] C/internal/config/config.go:166 | path only; „same disk" = `st_dev` [FACT] C/internal/system/mounts_linux.go:225-245 | thin LV in `local-lvm`, `backup=1` [FACT] A/configs/build-golden.sh:105, :221-225; measured on all 3 boxes | app volumes; Tier-1 of apps with no drive [FACT] C/internal/backup/backup.go:429-438; Tier-2 state-only fallback [FACT] C/internal/backup/tier2.go:157-178; **inside every whole-guest copy** | „Csak egy másolat készül (nincs második meghajtó)…" — a Go constant, not i18n [FACT] backup.go:359-361, :1387-1410 |
| Enrolled drive (`/mnt/<name>` on the host → `/mnt/felhom-drives/<name>`) | `GET /disks/candidates`: **whole disks only** [FACT] A/internal/storage/candidates.go:33-53 | mount: `uuid:<fs-uuid>`; wipe: `byid:` wwn > nvme-eui > … [FACT] A/internal/storage/durable_device.go:32-51 | systemd unit by fs-UUID through `FELHOM_MOUNT` [FACT] A/internal/storage/mountunit.go:50-65; host bind into `mp8` | `HDD_PATH` apps, media binds [FACT] C/internal/stacks/media_library.go:24-33; Tier-1/Tier-2; optionally the whole-guest target `felhom-backup`; **not in vzdump** (`mp8` is a host bind) [FACT] `07`:477-480 | Tier-2 badges; „Nincs 2. (off-drive) másolat" [FACT] C/internal/i18n/locales/hu.json:303 |
| Guest `mp0` `/var/lib/felhom` with folders `docker/` and `sys_drive/` (bound to `/var/lib/docker`, `/mnt/sys_drive`) | config default `system_data_path=/mnt/sys_drive` [FACT] C/internal/config/config.go:166 | path only; „same disk" = `st_dev` [FACT] C/internal/system/mounts_linux.go:225-245 | thin LV in `local-lvm`, `backup=1` [FACT] A/configs/build-golden.sh:105, :221-225; measured on all 3 boxes | app volumes; Tier-1 of apps with no drive [FACT] C/internal/backup/backup.go:429-438; Tier-2 state-only fallback [FACT] C/internal/backup/tier2.go:157-178; **inside every whole-guest copy** | „Csak egy másolat készül (nincs második meghajtó)…" — *corrected 2026-10-24:* the i18n keys `note.backup.single_copy` / `note.backup.single_copy_unproven`, picked by the agent's verdict [FACT] C/internal/backup/backup.go:1485-1494 |
| Enrolled drive (`/mnt/<name>` on the host → `/mnt/felhom-drives/<name>`) | `GET /disks/candidates`: whole disks, each carrying its nested partition candidates (empty / filesystem / in use; corrected 2026-10-24) [FACT] A/internal/storage/candidates.go:84-90 | mount: `uuid:<fs-uuid>`; wipe: `byid:` wwn > nvme-eui > … [FACT] A/internal/storage/durable_device.go:32-51 | systemd unit by fs-UUID through `FELHOM_MOUNT` [FACT] A/internal/storage/mountunit.go:50-65; host bind into `mp8` | `HDD_PATH` apps, media binds [FACT] C/internal/stacks/media_library.go:24-33; Tier-1/Tier-2; optionally the whole-guest target `felhom-backup`; **not in vzdump** (`mp8` is a host bind) [FACT] `07`:477-480 | Tier-2 badges; „Nincs 2. (off-drive) másolat" [FACT] C/internal/i18n/locales/hu.json:303 |
| Network share | customer input [FACT] A/internal/localapi/netstorage.go:36-40 | none („no durable-id, no SMART") [FACT] netstorage.go:15-19 | `.automount` under `/mnt/felhom-drives/<name>` [FACT] A/internal/storage/netmount.go:18-31 | browse only; never media, never Tier-2 [FACT] media_library.go:83; tier2.go:141-143 | `err.backup.halozati_tarolo_nem_lehet_a_2` [FACT] tier2.go:37-40 |
| USB disk | same as an enrolled drive; transport never decides [FACT] C/internal/settings/settings.go:609-614 | same; the bridge may pass the real identity (N100: yes) [FACT] spike case 4a | same | same | same; SMART may be missing → „Nincs adat" |
@@ -126,7 +126,7 @@ binds both. **Creating or deleting partitions: never from the dashboard in this
## 7. The installer
**[FACT]** Today `step_grows` gives the guest's one data volume (`mp0`) 70 / 130 / 250 GiB by the thin pool's free space
(<150 / ≥150 / ≥300 GiB), and nothing above 250 (E/scripts/felhom-host-install.sh:2261-2275). Measured pools: 128 GB
(<150 / ≥150 / ≥300 GiB), and nothing above 250 (E/scripts/felhom-host-install.sh `step_grows`, now at :2475 — line corrected 2026-10-24). Measured pools: 128 GB
disk → 54 GiB, 200 GB → 113 GiB, 512 GB → 349 GiB (PVE default layout; root capped at 96 GiB).
**[DESIGN] The rule:** `mp0` = 70 GiB (pool < 300), 130 (300–800), 250 (≥ 800). Reserve = **max(16 GiB, 10 % of the
@@ -231,10 +231,25 @@ sizes ruled above in §7):
the public installer 1.33.0, agent 0.158.1 and golden 0.318.0 vouched) → data volume 100 GiB, internal place 213 GiB,
registered as the default drive with no press; Jellyfin installed with no extra step onto it; a whole-box copy holds nothing
of it; partitions formatted and used through the wizard; the honest backup offer on a VM. **Not as written:** an empty part
asks for the typed confirmation (the probe reads the parent's partition table — R-1093); a 200 GB disk gets no internal
asks for the typed confirmation (the probe reads the parent's partition table — R-1093) *(fixed since in agent v0.160.0:
the parent's PTTYPE is set aside when `blkid -p` proves the node is a partition of that scheme — `git tag --contains
f7b7308`; noted 2026-10-24)*; a 200 GB disk gets no internal
place under the ruled sizes (15 GiB < 20).
Also in build B: R-1084 (the whole-box backup offer asks the same-disk question) and R-1082 (`/mnt/sys_drive` is never
offered to attach, by device). A bundle that adds a path needs a step bundle on boxes with an older wrapper (R16, R-880):
0.158.1 was delivered through `0.158.1-step1`.
## Decisions in code no section records [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
Read at `file:line` on `main` by the main session:
- **`systemd-detect-virt --vm` is read once per agent process**; empty output becomes `unknown`, which is not `none`,
so every disk then reads unknown/vm — the fail-closed direction (`felhom-agent/internal/storage/identity.go:195-205`).
- **A path the controller cannot read counts as the SAME disk** (R-668, v0.269.0): no second copy is ever claimed on
it (`felhom-controller/controller/internal/backup/backup.go:1436-1450`).
- **The internal place („Belső tárhely") refuses eject, decommission and wipe** — the controller (`refuseInternalLifecycle`)
and the agent both — and allows moving its data to another drive (`internal/web/internal_place.go:20-26`).
- **The internal place's size subtracts EVERY thin volume already promised in the pool**, not only the 32 GiB rootfs and
the data volume (`felhom-agent/configs/felhom-internal-place:11-14`).
@@ -1,5 +1,11 @@
# `05-hub-architecture.md` — critical review (grounded against felhom-hub v0.6.3 source + Parts 01–04)
> **HISTORICAL (banner added 2026-10-24, review round 2).** A dated review of hub v0.6.3. Many of its facts have moved
> since — for example the stale threshold is 45 min by config (code default 30 min), the operator cooldown is per event
> type (`notify/dispatcher.go` `operatorCooldownFor`), and the hub→controller push was retired in v0.26.0. Read
> `05-hub-architecture.md` and `08-alarm-ladder.md` for the current behaviour; this file is kept as the record of what
> the review found then.
Method: every claim about the existing hub was checked against `felhom.eu/hub/` source; every
cross-doc claim against Parts 01/03/04. Citations are `file:line`. Severity: **blocking** (wrong /
breaks an assumption) · **should-fix** (real gap or contradiction, low blast) · **minor**.
@@ -5,6 +5,21 @@
> No matrix is designed here, no target architecture is proposed, and no gap is ranked.
>
> **This is the input to a redesign, not the redesign.**
>
> **WHAT HAS MOVED SINCE (banner added 2026-10-24, review round 2 — this inventory records 2026-07-28 and is kept as
> it was; each line below was read at `file:line` on `main` that day):**
> - The guest has ONE data volume, `mp0` at `/var/lib/felhom`, whose `docker/` and `sys_drive/` folders are bound in the
> guest; there is no `mp1` (`felhom-agent/configs/build-golden.sh:31-38,99`) — the „rootfs + mp0 + mp1" lines are history.
> - A schema-2 recovery unit (D5) carries the PORTABLE secrets; a restore takes the database password from the unit
> (R-1106) — `felhom-controller/controller/internal/backup/recovery_unit.go:24-27`, `restore_dbcreds.go`.
> - The second-drive copy's `recovery-unit/` IS read: `RestoreTier2Unit` restores the full unit from the mirror (R-102)
> — `internal/backup/tier2_restore.go:197`.
> - The off-site reconstitution replays the snapshot's named-volume tars, volumes first (R-354) —
> `internal/backup/offbox_reconstitute.go`.
> - ep0's per-household prune jobs are keep-weekly 2 at 03:30 (R-1162/R-1165, `scripts/felhom-tenantsync.sh:187-197`);
> the local whole-guest tier is keep-daily=3 (R-1149, `felhom-agent/internal/config/config.go:553-559`).
> - restic is the pinned official release, checked by version and checksum (R-1105, `felhom-controller/controller/Dockerfile:43-46`),
> no longer Debian's 0.14.0.
## 0. Method, baselines, and what "verified" means here
@@ -241,9 +241,20 @@ likewise silent. Evidence: `audits/DRILL-r361-2026-08-22/evidence/06-part3-decis
**NARROWED 2026-08-21 by the backup-truth drill — kept as history; both halves have since closed, see the entry immediately above.** Proven that night on `demo-hp` with planted, hash-recorded files: the **declared-userdata** leg of a **drive-declaring** app does come back byte-identical (`calibre-web`, 5/5 including two Hungarian accented filenames). **Two legs of the same story do NOT:** (a) the off-site restore has **no named-volume leg at all**, so an app's volume tar sits in the unit, in the snapshot and in the checking folder and is never replayed (**R-354**); (b) for the **40 of 53** apps that declare no data drive the off-site restore **refuses outright**, saying a running app „nincs telepítve" (**R-356**). Since the 40-class keeps ALL its data in named volumes, the end-to-end story is **unproven for that class and disproven for the volume leg generally**. The escrow/key half of this row is untouched by that and still stands. Evidence: `audits/DRILL-backup-truth-2026-08-21/evidence/` and `REPORT.md` (2026-08-21).
**Update 2026-10-21 (rehearsal B7; controller 0.336.0, agent 0.161.0, hub 0.162.0): status PARTIAL.** [FACT] Tester4 was destroyed, reinstalled from the public ISO, re-linked after the operator deleted the old host, and claimed. The rebuilt dashboard showed only „A távoli mentés szünetel, amíg nem hozod létre a helyreállítási kódot" with „Helyreállítási kód létrehozása" (create, never enter); the restore page said „Nem tudjuk elolvasni a távoli tárolót". No front door reached the household's off-site history (R-1176, P1; the forgotten-press dead end R-1203). The August evidence stands for the versions it names; it does not hold today. `audits/rehearsal-2026-10-20/B7-died/README.md`.
---
<a id="a13"></a>
**Update 2026-10-21 (fix brief; hub 0.163.0, controller 0.336.0 on the box, agent 0.161.0): status PROVEN-LIVE.** [FACT] The
same journey as the rehearsal's B7, after the hub's carry (R-1176, `03` §8a): Tester4 reinstalled on demo-hp from the public
ISO 1.29.0, bound and claimed; the hub moved the household's sealed package to the new host at the agent's first report;
the box refused to mint a key („NOT minting a repository password"); the dashboard offered „A korábbi, házon kívüli
mentéseid visszaszerezhetők"; a wrong code was refused, the right code unlocked 9 groups; Nextcloud reinstalled and fully
restored: 3 031 of 3 033 manifest files byte-identical by sha256 through Nextcloud's own WebDAV (the 2 others never reached
the old box); two off-site runs appended to the same store (63 → 64 snapshots), nothing set aside. Evidence
`audits/fixes-2026-10-21/P1-live/`.
## A13 — The customer's own UNAIDED recovery journey, end to end
*Map row: [A13](../00-capability-map.md#a13) · former map line 91.*
@@ -258,9 +269,18 @@ likewise silent. Evidence: `audits/DRILL-r361-2026-08-22/evidence/06-part3-decis
**Gap / roadmap (as written):** **WHAT IT DOES NOT CLAIM, and what the v0.201.0/v0.97.1 fixes do and do not change.** Six of the nine findings are fixed and each is red-proofed (R-216, R-218, R-219, R-217, R-222, R-215) — but **fixes are not a journey.** This row goes green only when the walk is repeated end to end and completes with no operator intervention. **Three findings are deliberately still open and each is a live blocker for some flow:** **R-214** (the console never stops showing a stale pairing code), **R-220** (after a rebuild the customer's drives cannot be re-enrolled — the deploy refuses and the wizard's list is empty, so no app can be redeployed onto its own data drive), **R-221** (a rebuilt box cannot run the escrow ceremony at all, because the pbsdr convergence marker survives on the host while the seeded `escrow.pbs_storage_id` is rewritten out of `agent.json`). **And R-216's fix does not by itself make a NEW box work:** with the hub guard corrected but the Day-0 manifest still vouching agent 0.120.0, a new box is **HELD, not served** — it stops being lied to, which is better, but the feature does not work for it until agent 0.125.0 is vouched. **⚠ CAMPAIGN-11 PHASES 2 + 4 RAN OVERNIGHT 2026-08-05/06 AND THIS ROW STAYS **FAIL**.** **What they add:** eleven injected faults against the recovery journey, each with a positive control, and a full unattended scheduled cycle. **The backup promise strengthened** — the off-site tier ran itself at 04:15 on a box rebuilt twice and set aside hours earlier (`snapshot_count` 1 → 2), all five daily jobs fired exactly once, and nothing on the must-not list fired. **R-217's and R-215's fixes were proven live under exactly their faults**, the set-aside proved it does not delete (12 535 KB byte-exact at the far end), and a wrong code was refused three times with nothing written and no lockout. **What they do NOT add: any progress on the JOURNEY.** These faults are **not a re-walk** — no customer route was walked end to end — and four new findings say the journey got no better where it matters: **R-224** (a hub outage and a stopped agent are BOTH reported to the customer as a bad recovery code, in 0.056 s and 0.030 s — no unseal attempted; the agent's own `err` distinguishes them and it is discarded at the HTTP boundary, and R-216's gate answers `source=version` so it cannot see reachability — **Phase 1's headline defect relocated from the version channel to the transport**), **R-226** (M1, the only message telling a customer to check their typing, is unreachable on any box that has re-escrowed), **R-225** (an unread store renders `0 pillanatkép · 0 GB` above a card saying it holds backups — 1 snapshot and 12 535 KB were really there), **R-228** (the set-aside history is recorded in `orphaned_renamed_to` and shown by nobody — `OrphanedRenamedTo` has zero references in any template or handler). **§4.1 is now MEASURED rather than deduced** — the floor IS served, from the box's own rendered `GetFloor()` and a cold-started controller's settle-gate line. **§4.2's positive half is still NOT measured** and needs a rebuild. Evidence: `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md`, `tests/campaign11-evidence-2026-08-05/journal-phase24.md` **2026-08-06 — THE FIVE PHASE-2 FINDINGS ARE FIXED AND THIS ROW STILL SAYS FAIL.** controller **v0.202.0** + agent **v0.126.0** close R-224, R-226, R-225, R-227, R-228. **What changed:** the unlock path now classifies WHY it failed, from the value and never the text, and the message that mentions typing is reachable from exactly one class — a real refusal — with **unknown defaulting to neutral**. Proven live on the venue with the same wrong code and only the hub's reachability changed: `400 → 502 → 400`. **What it does NOT change: this row.** Fixes are not a journey; nothing here walked a customer end to end, and three of the blockers that made Phase 1 fail are still open (**R-220** in particular is worked around BY HAND on the venue — without that unmount no app can be deployed on a rebuilt box). **What still needs proving:** a re-walk with no operator intervention, and specifically the customer-facing messages end-to-end — they were NOT re-driven, because `/recovery` correctly retires itself once the old data is set aside and restoring that state would be the reconfiguration the fix task forbade. Evidence: `felhom-controller/REPORT.md` **⚠ THE RE-WALK RAN 2026-08-06 (R-201, attended) AND THIS ROW STAYS FAIL — but it is a nearer no.** A brand-new appliance was built from the published ISO, given three sentinels, escrowed, backed up off-site, destroyed (guest purged AND both drives wiped), reinstalled and walked. **THE DATA: PASS** — all three sentinels byte-identical, including a 12 MB binary and an accented Hungarian filename **whose NAME BYTES are also identical**, restored in **23 s** out of the pre-destruction snapshot through the customer's own flow. **THE JOURNEY: FAILLine truncated
**Update 2026-10-21 (rehearsal B7; controller 0.336.0, agent 0.161.0, hub 0.162.0): status PARTIAL.** [FACT] Tester4 was destroyed, reinstalled from the public ISO, re-linked after the operator deleted the old host, and claimed. The rebuilt dashboard showed only „A távoli mentés szünetel, amíg nem hozod létre a helyreállítási kódot" with „Helyreállítási kód létrehozása" (create, never enter); the restore page said „Nem tudjuk elolvasni a távoli tárolót". No front door reached the household's off-site history (R-1176, P1; the forgotten-press dead end R-1203). The August evidence stands for the versions it names; it does not hold today. `audits/rehearsal-2026-10-20/B7-died/README.md`.
---
<a id="a14"></a>
**Update 2026-10-21 (fix brief; hub 0.163.0): status PROVEN-LIVE, with one operator act by design.** [FACT] The household's
side ran through its own pages only (bind, claim, the recovery offer, the code, „Teljes visszaállítás"); the operator's
one press is D19's (delete the dead host — done the day before). A household that presses „Új linket kérek" while its old
box is still linked is now told so and the operator is mailed the host name (R-1203, proven live). Evidence
`audits/fixes-2026-10-21/P1-live/`.
## A14 — A random night of household actions while random things go wrong — twelve rounds, unattended
*Map row: [A14](../00-capability-map.md#a14) · former map line 92.*
@@ -292,6 +312,18 @@ likewise silent. Evidence: `audits/DRILL-r361-2026-08-22/evidence/06-part3-decis
**Gap / roadmap (as written):** Rows **R-493 … R-500**, R-534 … R-545, R-719 … R-727
**Update 2026-10-20 (rehearsal, Tester4 on demo-hp; controller 0.336.0, agent 0.161.0, hub 0.162.0, golden 0.331.0,
installer 1.35.0, ISO 1.29.0):** still **PARTIAL**. [FACT] Every mechanism of the first hour worked with a pretend
household in a real browser and through the page endpoints: the ISO from felhom.eu/letoltes (sha256 matched), install
(4 min to the pairing code), bind (two wrong passphrases refused, the third right), claim (a typo refused), the recovery
code ceremony (16 s), eight apps installed (Jellyfin refused for memory, with a clear sentence), a family's use, „Mentés
most", remove keeping data and „use my kept data" (47/47 documents back), all seven customer restore paths
(`audits/rehearsal-2026-10-20/B8-restores/README.md`), a power cut in the middle of an install (clean rollback,
`B6-failures/s1-power-cut-mid-install/`). [FACT] The journey still FAILS unaided: on the minimum 8 GB machine the
first afternoon's ordinary uploads froze the box twice (≈1 h and ≈35 min, a relapse after each return) and only pulling
the plug brought it back (R-1192); the owner passphrase reached the household only through the operator (2 h 4 min);
the ten places a household gets stuck are in `audits/rehearsal-2026-10-20/B10-stuck/README.md`. New rows R-1192 … R-1199.
---
<a id="a16"></a>
@@ -375,6 +407,15 @@ likewise silent. Evidence: `audits/DRILL-r361-2026-08-22/evidence/06-part3-decis
**Evidence (as written):** **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests
**Third live firing — 2026-10-22, a real customer (Tester-2), the first with the R-32 purge-first off-site leg (hub
0.165.0):** host delete 12:54:47 UTC, RESET 12:55:56–12:56:04, every leg `ok`. The off-site leg logged in with the
sub-account's own password, found no repository (the box never ran an off-site backup — so the removal of a folder
WITH data is still not seen live), re-listed and deleted the sub-account; ep0's namespace with its one whole-box copy
(2026-10-04), its prune job and its token are gone; the customer, address, domain, language, tier choice and quota
stayed. **No other customer touched:** the three live boxes' repositories re-listed with their own keys equal the hub's
05:10 listing (303 / 62 / 43); every other ep0 namespace equals today's 03:30 prune keep-list (2 / 2 / 2 / 1; operator
27). Evidence `audits/tester2-reset-2026-10-22/`.
**Gap / roadmap (as written):** External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **R-25b CLOSED (hub v0.69.0, 2026-07-21):** the Danger-zone DELETE is now the guided full-teardown cascade that runs this very sequence as its middle leg — see the row below
---
@@ -25,6 +25,8 @@
**Gap / roadmap (as written):** Memory-guard FIRING is not live-shown (T-RES-MEMGUARD never fired: ample RAM / auth-walled) — implemented + unit-level only
**Re-checked 2026-10-24 (catalog sweep, `audits/catalog-sweep-2026-10-24/README.md`):** a fresh box from the public ISO 1.30.0, controller 0.342.0, Tester4. Every catalog app through `POST /api/stacks/<app>/deploy` one at a time, timed to the front door (median 61 s, Cal.com 278 s), then first login, one item, backup, keep-data remove, reinstall, unit restore, remove with data. 62 installed and answered; Wanderer installed but cannot reach its database without DNS (R-739); plant-it and wger are not offered (lifecycle, by design). 52 work, 10 work with a problem (R-1269–R-1277), 0 broken. The memory guard still did not fire (one app at a time) — the gap above stands.
---
<a id="b2"></a>
@@ -379,3 +381,28 @@
**Gap / roadmap (as written):** Remaining ~39 apps are legacy-class by design (unit-only offsite)
---
## B23 — An urgent security fix reaches the boxes at night (the security-fix step)
*Map row: [B23](../00-capability-map.md#b23) · added 2026-10-23.*
**Scenario:** an app has a published hole (a CVE or GHSA id with its notice) and a fixed upstream version. CC walks the
step on scratch 9202 through the product's Update (`box_walk.py --security`: install at FROM, the guarded Update, the
four version observables at TO, a 600 s memory soak), and the only writer turns a proven walk into a `security_fix`
ladder step. Every box on controller >= 0.341.0 takes it at night with the usual guards (backup first, undo on a
failed health check, held / stopped / needs_person / W+5h respected).
**Components:** catalog `3898593` (`ladder.py check_security`, the two test-record gates, `upgrade-test.py --write-ladder
--security`, `box_walk.py --security`); controller v0.341.0 (`legCandidate`).
**Status:** **PROVEN-LIVE** on 9202 with the drill catalog, 2026-10-23.
**Evidence:** `audits/security-lane-2026-10-23/` — the negative control (TO `busybox:1.36.1`) `failed` and the writer
refused it; Plex 1.43.4.10903 → 1.43.5.11029 `proven`; on 0.340.0 the night leg logged `plex: skipped — not_proven` and
a hand press took the step; on 0.341.0-rc.1 the night leg logged `plex: security fix CVE-2026-96652,… taken` and ended
`done` after a second-drive copy and an undo copy.
**Gap:** an app with no box fixture (Plex today, R-1238) is checked only by health, version and the soak — no data is
read back. `files_may_change` is a bench measurement and reads `false` when no bench ran (decision 255 item 7).
---
@@ -294,3 +294,29 @@ part next to a mounted part can only be used, not formatted (the root wrapper tr
---
## D16 — A failed or cut restore puts the data back exactly (D22)
**What it is:** a restore that fails after it began changing an app — or is cut by a power loss or a killed controller
— leaves the app on the household's own data as it was before the restore, running, and says so. Before (2026-10-20,
C3 on 9202): a damaged local unit removed the live database volume, Postgres crash-looped, the newest row was gone and
nothing could put it back; a power cut mid off-site restore left the app on the backup's data (rehearsal s2).
**Components:** controller `main` (`d582feb`, unreleased): `restore_unit.go` reuses `offbox_restore_undo.go` (measure,
fit, copy, put back, hold on a double failure); `restore_crash_putback.go` + the app-stop marker's `undo` record and
`Recover`'s put-back; `tier2_whole.go` pre-check (R-1178); `RunAllTier2` skip during a restore (R-1180). Design: `07` §6.3
(the 2026-10-21 `[DESIGN]` paragraph), operator ruling D22.
**Status:** **IMPLEMENTED** — proven live on scratch 9202 with 0.338.0-rc.1 (2026-10-21): a damaged unit volume → the
live volume poured back, the newest row kept (A,C), the app running, „…Nem változott semmi…"; the controller killed
(kill -9) right after the volume step → on start the copy poured back (A,C,D — D newer than the unit), the app running,
„A visszaállítás félbeszakadt (áramszünet). Az alkalmazás adatai a visszaállítás előtti állapotban vannak." A normal
restore is unchanged (the control). PROVEN-LIVE when the release reaches the fleet.
**Unit only:** the off-site power cut (`TestR1195_AKilledOffsiteRestoreIsPutBackOnStart`), the second drive's pre-check
(`TestR1178_*`), the mirror skip (`TestR1180_*`), a failed put-back at start (`TestR1195_AFailedPutBackOnStart…`).
**Known gap:** a database whose data is not in a named volume the unit replaces is not covered by the copy (none in the
catalog today; a drive-file app's unit restore is refused by R-538).
**Evidence:** `audits/night-2026-10-21/P3-restore/` (the run log, the red-proofs).
---
@@ -219,3 +219,25 @@ Tester 1 run controller 0.311.0. An Immich image bump the repair was not proven
**Evidence:** `audits/immich-repair-2026-10-14/` — spike A on 9202 (the five candidates), the scripts on real damage
(9202), the live cut on demo-hp 9201 (D/). **Not exercised live:** a lost original (the cut lost none); pinned by unit
tests and the 9202 run.
## F13 — A hacked app is noticed (the upload and CPU watch, the weekly line test)
*Map row: [F13](../00-capability-map.md#f13) · added 2026-10-24.*
**Scenario:** an app on a household box sends far more than it says it sends, or burns CPU with no transcode to explain
it (the Plex2Shell shape). Once a minute the controller reads every app's exact counters and asks the app (qBittorrent,
Transmission, Plex); not explained for 15 / 30 minutes → one operator-only alarm, one clear when it stops. Once a week
the host's agent measures the upload line with a 10-second iperf3 upload to ep0 inside the tunnel; qBittorrent's default
upload limit is half of the tested line, never over the household's own limit.
**Components:** controller v0.342.0 (`internal/upwatch`, `agentapi.LineTest`), agent v0.164.0 (`internal/linetest`),
hub v0.169.0 (four operator-only events; the System page's „Upload line"), ep0 `felhom-iperf3.service`.
**Status:** **PROVEN-LIVE** — the alarm path on scratch 9202 with `0.342.0-rc.1` (2026-10-23: a 50 Mbit/s flood hidden
behind a real seed → one alarm after 15 min, one clear; a CPU burner in Plex → the same; a real seed → none); the line test
on the three test boxes (2026-10-24: 94.3 / 94.4 / 92.6 Mbit/s; „server busy" retried); the qBittorrent limit on
demo-felhom (5 789 696 B/s = 50 % of 92.6 Mbit/s, three channels agree).
**Evidence:** `audits/SPIKE-upload-watch-2026-10-23.md` §7; `audits/line-test-2026-10-24/`. **Not exercised live:** an
alarm reaching the operator's mailbox from a real box (the 9202 proof had no hub); the family's line on qBittorrent's
page on demo-felhom (pinned by a render test). **Gaps:** R-1250, R-1248.
@@ -16,6 +16,8 @@
> renamed here the same day, with the operator's three Claude Design fixes folded in (marked *fix
> (a)/(b)/(c)* below) and one measured correction: felhom.eu is served from DooPlex with
> **Cloudflare DNS only**, not through Cloudflare.
> 2026-10-21: poster regenerated in Claude Design (one-disk data area, https-only); PDF print version added next to it
> (`felhom-system-poster.pdf`).
---
@@ -95,10 +97,11 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
## Zone 2 — the internet
- **Cloudflare** (grey): DNS for the customer domains and for felhom.eu; the **tunnels**; the **geo-block** (only Hungarian visitors); flood protection. The hub holds a Cloudflare key per customer zone and checks that it reaches only that customer's own domain.
- **ep0** (grey box with an orange stripe, a Hetzner cloud server in Germany): runs the **WireGuard endpoint** and the **off-site backup server (PBS)**. Inside, one datastore with **namespaces**:
- **ep0** (grey box with an orange stripe, a Hetzner cloud server in Germany): runs the **WireGuard endpoint**, the **off-site backup server (PBS)** and, since 2026-10-24, a small **line-speed test server** (iperf3) that answers only inside the tunnel — each box measures its upload once a week (R-1247). Inside, one datastore with **namespaces**:
- one namespace **per household** (whole-guest copies, each household's own key),
with a NEUTRAL note under it (not a red gap box): "By design: each household holds the key to its own copies; the operator cannot open them without their recovery code." — *fix (c), 2026-10-09: this is a property of the design, not a gap, and a red box said the opposite of what it means.*
- the **`operator`** namespace: two groups — **"hub database"** and **"DooPlex copy"** (code, password manager, secrets, signing keys).
- outside PBS, a plain folder: the **release copy** (images + agent packages, D33).
Prune jobs on ep0: households keep one copy per week for 2 weeks (`keep-weekly 2`, R-1162, 2026-10-19; every new household's job is made with its namespace, R-1165); `operator` keeps 14 daily + 8 weekly.
Mark ep0 with a small shield: "**Protected** — holds the only off-premises copy of real household data."
- **Hetzner Storage Box** (grey): the shared box for the households' per-app restic copies (Tier 3), one sub-account per household, add-only keys.
@@ -119,8 +122,9 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
Arrows from DooPlex (dashed, orange → ep0):
- **00:20 "DooPlex copy"**: the code (614 MB), Gitea's database and settings, the **password manager**, the nightly secrets export, the **signing keys**. Encrypted on DooPlex with the **DooPlex off-site key**. The key that writes it **cannot delete** old copies. Tested back every Sunday.
- **02:30 "Hub database"**: encrypted with the **hub-DB key**. Tested back every Sunday at 04:30.
- **Left out on purpose:** the container registry (27.7 GB) — "rebuilt from the code".
- **02:30, 08:30, 14:30, 20:30 "Hub database"** (every 6 hours since 2026-10-23, D34): encrypted with the **hub-DB key**. Tested back every Sunday at 04:30.
- **01:40 "Release copy"** (since 2026-10-23, D33): the newest 3 released images of the controller, hub and samba, and the newest 5 agent packages — not encrypted (nothing secret), into a plain folder on ep0 (`/srv/felhom-release-copy`, its own limited user, outside PBS), about 0.7 GB, capped at 3 GB.
- **Left out on purpose:** the rest of the container registry (older tags, test builds, non-Felhom images) — "rebuilt from the code".
Arrow ep0 → DooPlex (dashed, 05:00): "**ep0-copy** pull: a second copy of ep0 at home. Keeps 8 weekly copies. A daily job removes a deleted customer's copy within 30 days."
**The test machines** (small, plain tiles near DooPlex, labelled "disposable"):
@@ -143,7 +147,7 @@ Title: **"Who tells a box what to do"**
## Bottom strip — "One night, hour by hour"
A horizontal timeline from **00:00 to 09:00**, two rows:
- **DooPlex row:** 00:00 database dumps · **00:20 DooPlex copy → ep0** · **02:30 hub database → ep0** · 03:30 ep0 prunes households · 03:45 ep0 prunes `operator` · **05:00 ep0-copy pull → DooPlex** · 07:30 ep0-copy prune · 08:00 deleted-customer clean-up. Sundays: 04:30 restore tests.
- **DooPlex row:** 00:00 database dumps · **00:20 DooPlex copy → ep0** · **02:30 hub database → ep0** (and again at 08:30; every 6 h since D34 — corrected 2026-10-24) · 03:30 ep0 prunes households · 03:45 ep0 prunes `operator` · **05:00 ep0-copy pull → DooPlex** · 07:30 ep0-copy prune · 08:00 deleted-customer clean-up. Sundays: 04:30 restore tests.
- **Customer box row:** **02:30 Tier 1** · **03:30 Tier 2** · **04:15 Tier 3 off-site** → app updates → **04:30–08:30 whole-guest backup** (local daily, off-site weekly) → Debian fixes → (when approved) Proxmox, Docker, kernel restart. 05:00 the hub checks every box made its backup.
- A small note: "Household mails about a kernel restart go out the day before, 09:00–20:00."
@@ -1,391 +1,237 @@
<!DOCTYPE html>
<html>
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Felhom system poster</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { background: #faf9f5; display: flex; align-items: center; justify-content: center; min-height: 100vh; font-family: -apple-system, BlinkMacSystemFont, sans-serif; }
#__bundler_loading { position: fixed; bottom: 20px; right: 20px; font: 13px/1.4 -apple-system, BlinkMacSystemFont, sans-serif; color: #666; background: #fff; padding: 8px 14px; border-radius: 8px; box-shadow: 0 1px 4px rgba(0,0,0,0.12); z-index: 10000; }
#__bundler_thumbnail { position: fixed; inset: 0; width: 100%; height: 100%; display: flex; align-items: center; justify-content: center; background: #faf9f5; z-index: 9999; }
#__bundler_thumbnail svg { width: 100%; height: 100%; object-fit: contain; }
#__bundler_placeholder { color: #999; font-size: 14px; }
</style>
<noscript>
<style>#__bundler_loading { display: none; }</style>
<div style="position:fixed;bottom:12px;left:12px;font:13px/1.4 -apple-system,BlinkMacSystemFont,sans-serif;color:#999;background:rgba(255,255,255,0.9);padding:6px 12px;border-radius:6px;box-shadow:0 1px 4px rgba(0,0,0,0.08);z-index:10000;">
This page requires JavaScript to display.
</div>
</noscript>
<meta charset="utf-8">
<meta name="viewport" content="width=3200">
<title>Felhom system poster</title>
<style>
@font-face{font-family:'Plus Jakarta Sans';font-style:normal;font-weight:400;font-display:block;src:url(data:font/woff2;base64,d09GMgABAAAAAC5kABAAAAAAdIQAAC4CAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGoEUG6sAHIgyBmA/U1RBVEgAhTwRCAqBjRjvDwuEQAABNgIkA4h8BCAFhTQHijUMBxsmYrOibrRa58XIQK2inCpm/8cEOmDsR9uBUk2UUKW7NTYz49ua3u9Pq2XtivoezxOcOTInPB/CpVd/j0JxB/Qa3t6eAuCcHV2olRQryqGkA4cwHNSiOkJjn+RyiZ7fD36z535XS2IepWlSSGSaRyKVEC0UGiR7P89v88+97wEPWkWMekYzRMQqxEi00W0mpVIqLutHl8tyXerauWh11R9U2bIeZT08FcN3b/f+RJsvU6qKTWRpGIHx1YJ2DOO54WlO/9Wgn1TZgCAW04tcLrnYXU5zF7kESUiClRYt+Ao1qDjdnAoTo9uqQiditJ3/1mGuBejvh2v5M4HCvE3xsyISBlgDBq6bLRC5i70o8bHBASZZDs7tp1IqsZOrp5vaxELdkCONNfmKODBsNrGsWHq2YOO6NlMbl8e1JW9LXnc0OBocNqf7058qtaznH2QU9Kwwrs2weQ4yNCCAgkCjTC3TbQ73HiDfcA8ypCz53uTkWcpDzkTOVD47MwB3drhPguCdAO4ZPCnDJU9/xOEdTo4LLqkl7p03uKMM7r2st/GXMgWpok8VZUpi40KV4kRp4q3f68iZpWi07yF6BOOGYN59luq/w1Q9yNXjT44sj2YOTRn9QYww+mfR497tJjuGsekiC5SoH5Qw6kUJsAcYpnnHDsIgQ6N58oQh080GgoSSCONgQvUHlEoH3pG8okW8w0SJgkjlQQrUQrR0EAMDxKQDYjYt5nsfByMGEGEhCojwkDsCAvoJkwXv2FVSDYLbFxxWECAB3Q8DgW9d6baC+HPdBAiAKGiHq8fPmMzioKAes9rx6ZVPAk/X9N8GNdWq67s/7+f79j5vYx/e2/d7wc36bk9p9tbRdK2xaVtlK2nz81pWS2lKGZK1iBbE4eoB820SV9Q46rtJ4v973sNGu9FI58DynLKBjra/nW1pHZ58K/q33/tRft1b+Gnvt+Q4kM/L4c9rBn6WHWUtU7XWpKoFdaSKSt1mMyqpLRWgKJRWWNFViwfMu82Kiet9S9C/nR6NRWGMFaVwk3m9p7u/O7u24Z3ZyXb5w9sLOQ/mkXo25Qadzev7PQZgiOQjBg5C1BCN/u3nTZwF1EGPKlOkgARM3ws4lCurr2c3xdCvDYChXo8bADza7Z4Uoz7FZI/a41JK2Er0RpUyCpTWo0ZB/EmYCgEJ6UsCHe9JNIkmo2SUNJJGAgRIFckhqSSVqIiKrCfrya/kVxLamQ+m6xxKcyoyr2ZeJHamOMh35DsSTIJJASkgc8gc0kf6yEQykYyRMZKHPBej7mQIkAgSSdSkijSTlnwdEfBQyW2hcBXny51E+IQPh7w9EBUKvwFi95CqZ1QzXhdmvQ+IkzzHIxcj7toIEFIxs/85g/uvw4H79CFBv3V9ORDGTxPrbOJ9zggz8aHB+RNPnOfX++0IvsHmB6lTjCD9CpUbaw/qHz1dETaVAjvyEYbbmhu7uclCGIlwvnzACB7sGwBLGFgUf7H75KQ0ezdrZC9uiBh4KMVwjZGSO/aMcpk2Z2LL3nt3FQZ/1q7oHcouy3dcAogKnfCbaJBqV0ddKnbhmUYqps7ZU5URQllHXO8DzoHfEOyoe96Fs2xbl0Sk44yyQJIlHB19jCKssZIVwSL4+MIaX1Wi34Q1a81xa6t2XrNcgMm8Eas0wbm3hBx1MOKiQpN8o+dBzlGMs872Id3FT5ye+LvacbOD3B1wQUfdiEbPD6efPMbYA2JTepp8NXscbMiPOf3VCYblKx8RHWnnNPvIkIBnPPFB9wH83n8UFSFECaySQClJrnj5SuXQqFdkIqMa7cxadeth0GsGk1lmsRpyls2wET2uumqKUeOmeuCBGTAFhI2Fi4fkzh3Jly9PQSKRpKRIMjICMWK4kFPiSJDATZJkIqlUsBw5uNRKIKVKuSpTBtHQ4CpXiVKlGl+NWpQ6dTxoaVHq1eNq0IBtoskYGukhBgYMRkZcJiZIu3YMHToImZnxWHQhdOvmq0cPd72mYppmGonpZnO1yWZi/fqJDRgkNmSIt9PO8HHWWd6GXeJlxAjSVVcJjRqDjBsnkAoipIOS7KtB4agA2VrE+AT5Qm4oYhiB3LfLJeED8YWf4c0/KxHQLEaCRCBFYteUKNGCxJYTYPDJDBcxYXfJiU/WTQoWe1EJD4aXy0M+tslQoARWatA0ipCf8IgIPEoiVH6Sg5A6QA8MIxObsLOuOq8qQCuBIszW/vCkNGgzDJ5ZNk/meiEklIS0DoYrheFhTDE0cfnquUF+RQIHDtdgDL393s635a3LPmAFBbUq/fl+vtpWXt1lpknLAJoEzEqr9iHpy/UY2rCdeME7/O/e3xK9nj1FUZtV/9u/DG5Bjz8zQ9W+CCO+ostqglfCyyLmzl+QYCFChQkXMbxTS00gEyNegsRhfF+p5MhXoFCRYiVKNWrSrEWrNjp6BkYdOjlMN98CS7zlnZqeqt92Ox113Mm6fk1HXHbVmHEPIDE7IXV02GQLMdoSvyEBSUj26uQZt/BAMhrgNPLTYmT+CssiyiVPuUmxGf0YwKCkGsawmDaRAAAAHiDvQKIe6YQlhp9UmaZIRwYykYVsqKS6W+YCechHAQpRhGKUoAyNaEIzWtCKNuighwmjGKtE2sHfMXlOtLbhlAiIg1FxSMTtruAR7ghCMEIQijCEIwKRkI47P7d0naNXJU01x50rOjzzkmWdtBI22NGFbvTI3pacAkyV04EZmIlZmI2R0NWyeXnFVgl1u2EBARIMMMECBbbk9koe4I4gBCMEoQhDOCIQCamUiUp9MkWJdGQgE1nIhqpFvSkXechHAQpRhGKUCJmQKM+WmyiYhMmy0ZJNgma0oBVt0EEvDW2YZHtGh2desxRpFdhgR1dL94oD08aUTG5aJzcT/RjAoBzpk1ezHCXG1HgNS8fzmVg8qB+V2MXidwpvAxDUMm1FRtKClMmJr72u3Rf97Yo61NaTuTSUHLOrk6E8VjaVnGY3NJxrSprm3GhKe+6yXcZdZTH/Wi1h8zskPjt8No5uzQSll5gtJgG1zdTcSxfMDvjwoc1ih0XduU97tNos0P0PgRWC6and2AJviRMzw6Si2pdeSzeUv/k+KPqCzaqBEBA+YREGISHsbKm4DUo4XQIx5z3HO07EtUtiouc7uf4tmCOTUpaUTPLQNYf1myFLGBTlfYFxYE88hmAOBzGBEoSDgEJDXIEAQyHIQdeMqX4H0oyRSjAvCUMYf55c8cyGNZgVjHLTW0rNRLgNZkCVh29QCoFD/45jffnM7ikMeddnwIUMkW1tWJSrLlMD+MbnfmBu+LZ1NX70FzL570T2yGZ7+w8jJvY28YcPO887uSiJW1iihcBDdsFsYUL8BSmwSYPe4IuuGM7F5fAmG6aYJ1c1S04mnoNnSUZ4TvpizM5sEiVcMISEG1ghiNhiqcBVyAxGGNDXDIEkLxmiiAxxxJa71MMoN5A6B8Vgm4vrkJm4DYhqe1N4CGICVFzhsJALHIzc4FBOCGOi8NsKCANEjIXI1wchghEPiZBVmI8xQL1JC+w3556zIqLS7PAFuAbc8g47cVmsge3400aSAmcQ6DkkuimAPgBqTN0GpiAgEV0wotNmPMuzayYHIgDGNksjbmUoktWCwwNgurMJWZCPQf3k0EuN4DFjkPW3u0vWHdDHIl+MZHP0cdZQbKpqb+e6cIggCTYRPAXTfen+9Ajam/ang+gwOo5OodW0g+4LlAR+Enh3UGaQu/MHBqLJpfjZshh65UZANIJabzjtSfvePwWd/JbvA44A6Oqnwf93s2dlAfDfvxHbMkttCPjjm6+tS+8Q3UjmxeOl3Zdsl6zDV4YvAgJ2AOfdCgDylWajfLlvPZp5vy+9r360zbB9xjnLZZccdcxej2x20Cb7bbHVC888t8MIhIOLT/i086YlPPjy4y8ALUjUrEuO91WKkw445Y0j8ZQqh1qeAqXKaJSrU6/BRJM0MjBp18HM8iITjurTHHfHCa+std2o+zSDdZdf8PFwzRznvHY9fm56aaVV8fK/K3bF3wpznfevf/xnJyaMRGFgYeNxI+LClQ9PXrwJBAoTLESEUE+EU4qlECdZtE6Z0qTLliGLSq6Sp9EUqvLFTibfZPKn01IX8bRaFysbuymMpooUdnP69Q9GYmecNWDIaYMQ1DkJAZkMqE0gR8Hwe4BxPwHaC1CnAXrAXcMfSkPICJWGpqVtwepw2ITFl0eCgNli5U92/wm4E7e+vIKmJASfBaJERIfghRHXwe0+SWxOQulGZ+druJPN9wfOl68CLQlOwK9FOYRM0OgB6CKCQZRI0MXgtCL8xfmLLNc1wMSrwUHwheiVglJabhquYaTDEDGvC11l1dfqpJ485mSkfLy0iKjF4EhYs5hJgQaNnsWeUs7Skwa29Oz9WaY+WZ5JdDY4mFJK4ghWDeo2nyJOTdNhtZbv1GdY0WBwjLVT2jSVHOTAytZloXBjE9l5Cx0LbInBlo2ZJs4VYebx+n/2q+c5DGuPjOBEOu5Qi5Ge+I0xJKghpt2c56GU8RzSdnNc28JyPm3QaSpT6CgKEf9jTxR7EvW9itiGTRJVXiotlCXa1CjXt/n72rTAugo/EJBya+O40oXPDRXjgojwMrOoQZcGXLCEvm7P8lGODMA7aDGHjAjzWAAWDAQuG5v+DDN21AMygR1KjHoGFMGJPH6jwu6bcF7FymkY0Y0qaQM5zZM5U0NW8bAaVuwrSUvR1HZnWsn1xRuzDyrILine truncated
@font-face{font-family:'Plus Jakarta Sans';font-style:normal;font-weight:400;font-display:block;src:url(data:font/woff2;base64,d09GMgABAAAAACkIABAAAAAAdFAAACinAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGoIkG6guHIR+BmA/U1RBVEgAiBQRCAr7MOd2C4V0AAE2AiQDiyAEIAWFNAeaLQwHG4tmlw4QC84DhPKf6R8OUZRQzvbZ//8nJCdjIMNioFn1/xBpOyZxZpD2mlZKokwEjnylOOoWtqcFSR64T7zKU8VfDQuSmKCjVg1EutHjLbKgiH4qVm22p3/Ir6nO07xd94P8BTd0e3iLf1yHhLxLpRs+p226eyR8ch0Yqv+nfjd84heLvLwleuEq74EjJJlteR6+zq9zX9KVlj30H5or2juzjui0mb2LUdhJal3ph4S8mb6Y4XHO/s3shtax0harhJonqaeSKqUG2xCTmRs7MePvc2KMg+tEYaJE2YJV3XvsniDBEHAIh5IYHGiNdmDs25//tan/IAwTU4HBoJxsSzbLIEU6KNAwtXu2vG7NWuaVYAIe1hr+eQ73jWkT657XGNuyNQgiTrN9PNwY8MSPAX/++f6e39rnvgSDOMGwaQrCDAMNJGnCDAL8wsckXSkZHjS3T0GCar2+JwC3dhtgrgCoTZmAoHM12m3C0IEw92hibp9rZ2Fk4sKTBhRQIDixzlvdh1YmP8AJbvLmUjPl+3eCQnjP2jaUF+HlZeGkmzpjf3HYuz7yVEBQEgXdKGcoBQAKk9mwB8pD2ITJ8BX/Z9Pqwq7+qhXZs+pr76sF8rz1EgSZvIS9QBgdtKq61WqVyiCpDWrN2jOsgZvRanAH3rRg3NaSlu0DZvuIOEUMMdogO+DsgiC6IL8gDs6WzWZBsZXYMRexPp7nP/ulXw80S5fVcZg8Qi1BJEQRbwwx1vqWr1n25yRAT+v/9gsfiAvx8SEhDWTJCXLlBfnwhwJFQTFioDipULoCqFgNVKcRatYJ6qIfNMBQaLiJ0GTToBnmQwstgZZZDa21FlpvJ7TbEei4M9A5F6HLbkN33IXuuw899BJ6rd04X7wwjnkcAYypVQKmWmuBaUsSTGeyYEbDhZmKGGZFFDAHgsP4ko+KlTAqydSiItOIipMWFAQ6cqVDVlx5zQ2I99j2gL2I0YD6FQWC7L7jfntRdn2OuOAiYEtKx2pgWzJK/4Dq8OkuRHVAhKku39fe1B7JH7DKWwzaT7+zT+noO/ny6H/H3TMvRWk5gLNt/X1G0/TPvO/Pf2cQXb+fq6D0+vkKVM4nng3ZOFcePW8U7zynQ9lyuATXnv1np2UYA1z6zWctOH1OOQtP3RGtlgvu06poFUHk8Cq6OH/LBU67NVuGk1+2Jb80sw/3ZHJ6bQ/v4EPAKTrlxFgp9SzoWGaW/se1sh/v+8DJf7uHL6PP96+7fzaoXpWR5/dHUL/d7xx+3yn37vPKjfvwaXfpHr4kl9xxe6NYLccAfGlYjeL86PDonn/poTqseofF2rqbADWHV9PFlVVzwWmxReU72XUry47iXmhI2+W7el4tC+cBGJQv3PlLq9i62ZxjBj8H9zj3HAHfUeCduP8TQDHHWoDQRHNplIXuGwNmgO68DwJ1hg/Dijmkd9tGsjuSjfksyRmX9+WzCazA/4K/AO8AHwD08T5/hRWAMQl/5b/bFQefz7JMiDEzAAcOh2EzrIbV8CtzlbECzPN5JsuoS3J/PtsZZRnkcjOHITBEHpcP58wN5vgcg1eZHYwV4DQ/zWABFuCf/C9a/Hwb0DaX8JhjcJi7Ie76MjwKapIZHCjABpzADQJQoJ3hLclSIG4eB0mQiwrVvsqhETpAb+gPw2E0TIRJMBeWw1rYCLtgD+yFj+BruAV34Qm8g8/yP8g6mQoy4AgHBxkS7YN2a7YD/FuZ0CbvSRuUgUiCfgNi7oNyYkjwQ/4YSWnURDDDT/TFJVjzSACggQ2S+xApggtxkWoMDdqv2FJzpPF8GtYP3Qa9fYyZg2VB1INWIdHdgpucyRpatrTcGg6B37TmTzdWgS3XfQjUb82lGZCGnk/Qxbzhe5e39lAvYnC1hUNmjvVSzaj16JhSk8njt0wMt3zlQIWtj4M7IzCz7l43+rNmmKzYUG/w3WnAB3P/W/7swFCcV1Kf2AlqMlBHQ3WxBJNPsNz/Xf+n6/2ZihjwaxksZizVQe6po4+przEawA3049gkFEsRFiLLLD4yIJoov8j5ABnopu9hxDksPfTUHMxjPzqNwVjSiJ8mAwOF9nftcLvPtECHNB2pzBSx6KiRFNYPYwCAi3iqMxXYK66CwBOCULBaKA7SoBBKoBraoCsMgCEwBiagaaXNptvaKXPsrJxrt6PmTeNoZectaxnR1tIDOzyGssvzyt1e++SNyRw+gotwFYz8KZPfRHz4jPiaJoif6aL3N2us/ofHyTwuP96MA8KLDOAgTqcUzRnoYVoUczUNXB070JyOGsQgAz7IuTReIL3kB1DdjyLBVJdxXXGDyE1CXNdINY8B7rDAg46zZjOcFztMgnGAwAIJFwIoriAA2Ej5pjQ42RgEna3Ro2mRhyxehMnj44E7rCTrFJqcW8z4hmkacO28z8+lhwsSBqEbCakC62FrceWwJV2nqOkPas0giVpySSJFqpany4Py8ZwkChRCglmCgQhkoAI+WIFCWNBMUws0YI+0pSOIowQsTlEOQpByYbKBBGzAHknKP6KYmFRpd48SD5OABTyQczGPt8XaJRuiZYdKZXTRcaCfC1IR4fo4eZxxEvUIqITVDwq3aTUPF0wbi7vcZisNB2lPy22Qy23K83KbxV7WWXO74LbxWnwDg/Letob1g0VeNiOBP25EqWxsv/2e+3DNKy33WGy/B+W9ywY2M4OFZcqvBSeeS/ZZZJFAUggEFyFG62VwcaFAtB4GFRMyYvxKIS2eSow8GZLoRArmC/HIaegly5Rvp+dMiDeMiIe5siWZmsGIoriOgY7LDCMyi+oxMiR+Q+gm4nItUI/YCEeM2IFXzjPLGKUx8rlDQqgNiIwgpEKgctMNc/eCyq6c+OJjM3mEdumosFQqhdTSWT0Q6q7eRCOCsAk3GX/oy8L7RHVcCVGUxLjJIbFShcSXGiRYizIqS1n3cgiiPNWK/6NBVHwpTSW67Mx6LFBOqx9YQzz7bMU9iOddLwdba+/oc7bB6xrrOEL/EG95regI2nVP0XCA5wNK/0aB4ShItwt659Lo9Q7eeZcDMgHgN5laLzMAKdFWnSTVAAbZjMyKXADlYQ0uvLcdFQrdhO/MPeiweNM7GhRTnOnJumzMgZzPNxnJjVLLqaCS5jfGGaZRYfS+OZOfkl8JECRKmjx12nRxE6ScWjpZwjo2sY29PLGX1jqsUzqrPr2l94Px/69Q0z4Bzowlnre0P+fySYZzPU+bXeiqE1ANo/wz40h+TH7Gj4iQIkeNFh2sBIhQRRvzWcvGK7g/3SEd1Rld0Bt65yr2yzGf9/j/y7yOWVzy/b/5/8fm7r8B+xG+/HxyH+CL+8cHNP//9PNHQIBdnPG18QCkLZ74UpKyVOR0LgLInbkMcneu5J18lE/yWb7IV/km3+WH/Jmbud2JeqVeX0ppgDIQUEaXVaj8yqsGqKxEWYZjFaH1JDaR2kxkA4WtlLaxd5Cl3Wzt5+QIK3u4O8nVcZ5O83CKnwt8nBPplnA3BLkiwk2hAt0W46449z1g8EiCJ+I9luipVC9leSvHe9neKfRJxG95vgj5qayJ0NpkaGkSdGkm9CxWj2J06zUYFNnASAZENKFMY2IYXorxZRgV3bhYpsQ2I8hMC5NanNyiZMr8sKQ0cho76SV7LlM+Ezcn1PhnqtX41qpjBpflWFawsY+dA4JdFeY6xldhv1Q2BTo2DUaUanTpehVncBQlvilpPJT6bmeilKYyvtQS20hlO7UdHB3m4BBre3k5w99Fvs4LcU2SZ1K8kO61XB/k+ahAu6IA7Ng4qPZXgT+qmgqdmg69i9enBF2bBd2bDX1LNCSaoSWZWJaxMU2La3I5psYxq1xLUpiTwOz45iY0P3F0MccaW/B44woaYzyxRx9NDIHjjjOWYHHEFC2EtZknQA4FlJ8BX4FOy4JuZ4Iu00F7AE2kBlWPQrwG31p6dRAPD5C7RCxvRwzWg6+6Wt5qh1OLeeut149lsTXx8i2ovxMWcW2s+kg7NOSkC4M91Qs2vjzMMZZjBS/YS4nCQUZEIyIoCKKhRPIys8ggcJYhiACkBoRATLCsG9oSkX6rqGDUhgHDoGMiED62KUlCMsWu27pwXTZ2Farfsq5vf9JKj+HxOLYcIgTA4K9CNlHfHBr9XAR6/G6sKdKw44B6alXA1rRD2XWX+HggQMeEIMgqKHD/Lu0fOm28TgfLQArTVKHuvfa9pChgGkIYMgvDZBnXvs8PxhLm8zmVWgTXvVpUo1JMTGerLOjSVIpZCmkuLrKlMPy1MQ7+bwTR99cc/fsn4ecn4T0h+vvetSxcbR8pQ2L0J/0cXD/Pks2K4jwEN1mD6Xmj5LBUSpm1b2CfeuuaP7+DuOvN3YMyDzrX2RDGS7wmxuoWC2caS9lwxI5ZbrSbqwl72AHftnlJ3KTtGzX+7W/q3O++pVWnmVqOkkmlkqWp9FtSVB5uONgiWpNVSWGJz8UU3yQuh0qlHOuIO6VEyqY5vlIp7hutglMYVobyeSC6M3tVx0ikybdPGIjEii5tIm/jwGiX/5fN71BE1aMcZEbicJp8RIZ9cnlmoB0tnv8iXHWcXozOlKwdYtl9RQ+dW+h3b+oFu6+tZm/sOjOBfPQ0v8PzPAxXqrw4fW9I6jyfbVh9xL7odQ3N7aJlXNRcgTzkSn4Wy5G86ZVdhkzJRokFrj7ShQpF7aPXe9rmk3754E9PiTIhxdnmZJAvhhIecujrz5JVdl6/YOoLcuZaLine truncated
@font-face{font-family:'Plus Jakarta Sans';font-style:normal;font-weight:500;font-display:block;src:url(data:font/woff2;base64,d09GMgABAAAAADBcABAAAAAAdOAAAC/4AAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGoEUG6swHIgyBmA/U1RBVEQAhTwRCAqBjRzvDAuEQAABNgIkA4h8BCAFhV4HijUMBxt5YhVy80y5HYDz6+vPkSjKpKqKooSRFmb/fzrQxgiR/aFVTbgkEaKHLEqaWRN6iNYoMzFRot6ZE0wUDSy1vx6PBQuWsurTe/dSHo26HsUt1LvM6OmU6d/XtqvsymyNw9N2Pj9q+Jf48CO6NSY3XxEbPjp88LCDO0Jjn+QSfI3x23dfTPHkJpUsIpkMoZmEhofGEOV+Hrf1/t9gGxthNVZhZF9hBwgqXHZ4VV60XiQXrV6Gw9N0/oPiKZSKpo1cVC4md5dLLqZNvG0qmlQseDHtZ8i0EybGVGHaiXgnlI/5/wMoaKzN7N4LEKV/RkqlJLNmFnKAyBFRQ1o9s6cdIQSnkHiEjBojcRTGIJMKH8b/75vqe7uS7czdCaCsINCAvGOSZHPKoJal0y8MEqj5K3wSmqQbcq9Jv6k64sPGyrIC//T7LQiN+HujBqbTTKIl04dZEz+IXXw3/v/vapGXcJhIuCr7fPa67j87FFgNCGAl8MrUMt3GcO8XRxnsQY6yxOlZx3tn8jtPechGipypfHZ6wN3ZIY4wPGmJpcF5gHxDcw6kDECA1BLvvAOPr3+885RzJnI+NjZVEH34afxBJheEuSIR/3XjPyvN2YFZc0W9LAwobBJpAufMbJXTEtc6eC6tS0OEyxhhyPHKK+b8/GTkK03XzLJ3MWYRjmMcx3mIRRyb9K9X2jHmevwnW/NtLJRTWQUOcNd1CWAeUO/5g1uNR0ZKypQpDBlkGBBIKCLABPhQ2gmKXAGmCc4Ulw8aN24QD4mQZP0guSoh1aohteohCgNrToQDIx4QQSE6BThk9D6GgHJOouFZJ4tWER26OnWECAnI/+YCAR+8O3GE4feaJJpIhOEgFV6Ej+TZaahwfX9f+TI4TedYn/nJHvZ2r/diz/SYDx7s7ue8tet/2Cu7uHM7vRM7ukM75vsPtKdtzfDwWIN111YEz5uuCldcLn7YUOtb1eVdQpD4f67gps7lRF5Fl/tpz9W0+dfcqRzJ/uycbHPWZnkWZnamejHeDn9kBuNXeST5dCWbVF4F0pr44/TteEy+0IAiI48w7Fak502aumBFbhGi9i0vwhj99dJIGOZHj/vCx+nuu97E0MueA3kOrrVa1MUPKqVh0JcBZBlhCeuzqLO9CyQ4HyRcodrImpEMIEk7zLgAIHwmyw2YA1iWqgVABwOYpMwmIiVdGyl759OG7g+U/4yelPOJnYjtWUDysrz1NlLixI6dyEQGgQJ99lmmTGXK8PD0FitatDhxDjigSZOttnLe04bhmitNz2TswALqqINFFnHgIEGCoYbabLMCBbp1y67sE6OXI8CFq3i9lSq7V5IAJ8ShlWpPbfvagPeqVC255kFi8VwPhPb52kg+/AvMLAsHguKfGLEIAmT43f8MrtiPAC7dCgnKJ2wtASdWnh3qvrPCHv+lSqRnrI8XbH6JtDOjsSsqpKZ5Gpmw64OYqVE87OGxN+YP6njYo+FUUmj2w0eMFT0mJ28Y4wqyH6uTUUaU9PwsmG4vFnVaqOucHLSBqGSL+fN0TPF0T4IVS51LnK+i1kRhz5wtO78Re30w7b1zVgxcoN/87pUb2++ePevqWRh0dGYffQIh3baU08GbVFtEldFYEOBY5THPRuj2AfWAleyc7XHwmrlyakBptpOZhInCLBTMEJyKsjwX2UrwCnit1DAa9eJ1MJZ7foaHGXv7JDPzIqe+4EMsoxTHpKlHwYMLeaIqMF4fejEn95pZKX3ZLJT3xpol/pnF2C+1o87mS3HRLo+VV9tawUwftkbm9R5NK6aOW2hP69yUJjyT+mPqo3Y4mjaOEplGjEAWyYROOTaSYJHWe0tnkRx79ca+qDD578NEDPEHxgjkL1iCAEkyxMmSJ1WBGn3VUSjXrEU1rcFqDTVUo9vuUXlMp8Vzz/XXqdsAn3wyGOYLYVBYHJIRIyRLlkzZcUXy4IEkIyPixYseb/4EAgUyECyERJhYWJw4rHjpkAwZ9GXKhGTJwsrWC623PoT66ocmJ2ciVy5anjysfPkYBYrwFKuCVKvGU6MGq1YtpE4dnnr1xBQUOEpNCM2aWWrRwojWAHwDDWRskGH0tTrM0DHHGGrXwdBtt5m74y4L99xj7rEnzOjokJ57TqxTF6RbN5FPPpHAIiA0CoNBExCwweIYEhIhiRmgGTJky4gRljELiOULWOnmrNuN2SAXseOC5Ip1aW7c2aktPKELyXQ9Xl4jzYlFu4FQdJuJVR542QQmkrB9nmTpsAxdKksS+hOOSICjRMTSSnMgRCWgCjDUMLUJ066v4SVtnBQR4c9pDaaUdKnBOmeo5HQmDsSROMxpD/oUuokuyXNE73tnAFmNBBZdHIf+cFzNto1T44gWAPaLq7VKW8f6FrYxhQzBYKsPNHEYmZIwgSJSJjx+8nGM/G3cu9mpjn+ZqdYpztxNjKtGJpy+cTjNILN3YcT/fpkl0Aj/jTNkxJodew4cOXHmwo07D55kvAQIFCRYiFhxkiRLkSpNugzFSpQqU65CpSrVatRroDHIKKONN8EkrQ475oRTrrrupnYddJ56rku3TxCvxQEiV4khS/YWLyMgSZWkidBGRCXqbvmkmk85ETgRnKRNOhuaY/0KZdPKeLSuHF45ttK+0rFC2xSzGeON1pbDLcda2ls6WpDtR8gm0RbjTayETcInEZPISdQkehIziZ3EDwlD4pA0JA8pQ+qQNqQPmUPxUDKUDmVD+VAxVA5VQ+3QOXQNiGY3pJOsXZPohkq3uEA0idycIGz+AydgBHZgDw7gCE7gDC7gCh6j7jKUsZvMbG+J+mDUHUN9rmJTykkjQAVqaMpt3k3LzrSe6A8YIAaVYjBgCAyFYULH5Z6vfK70gvL7MAEBJPCADxTQwITYfAIHRmAH9uAAjuAEzuACruCxfdlewjYUDhEQCVEQDTEQG4qfkQCJkATJkAKpkAbpXCcwkEsRKthUIRSJ4pooAZRCGZRDBVRClaj2oVbUKdRnFROliUYGFaihKdQ8aGBgUc1E6zVxmOAYtEOH0LWJ5yo6Cbrc7r20XNfAAoB9ZNBg/JchpJKBIYLWjLo9zTAjQIqhvB4otzX/q1UHmlixCo0mEqTAPTOmT9IVl4J17oUx+YHFWel1a1dkZSRWxUj6ql5p+L9d4nBZwqvDbwKEm7MArJr4VXcEEM8dk6O1RaGB6TsppRrGDuc+cFejSgnNJxA2GGoP1dWUwTZ0ggoozOl3RKtshuxjfQpST2HL8VwMCCIiJBThERLELQWtMAaaiy6BuMaA+OBJ6HdL+OinX3jM0WpM6VVnis9euOyYwaI5QVU+HpgA9s1XCKZxER9I7AgQUEhB9IEA6iDIRWne9HIN0hVjGTj253FizbQllzMMlm9o0rMNCmUYgnDzDSaZXd6QAYFDryWIsvx8CaZTTzYHSBDJu61x8tNXZXol3DvXMiCD5dz6+nIbIMZvbGT/Nm2q7TCGDJN1rOHDjjSpF2ks84h7EGWR0zCF+BBrzhJikq/mF1SXPLE7G56yeQqa0pdD9WTqteArM8M7vH0upqZQyB8uqGUuA5YIvHYrioZj/MAIA/rPISDBJV3ict3QCsvoVQuGbB2RR1ENNu2WIztx8xFmE5xObEiEDn1wUEgvUQgDtNIiiIMSl84tIjwoClKIfEcQIjDikAQypXgBHkjnkwtYSOceW3AJzvPOz4G9wEEftFX3Fi51grDv/rxlPRBQVGvbhtANgNSRWUAvAlLRBY2D0n/Y2Yd9p6IIYFNoySAZQ5LnAArh4ACisesVJejrAyQokI0QFuhk2ujOEV37fD4KeQkx3FY99iAfQ3k39/OwiCAJhrDfzKVt0XZoe7Q6GoUG0dg0mKaj4dKXpXvpEH0vfR9UBlXe+AdUkfIWaqXtLNTgLCAKCPrJu7QaWv1zk9O07/3DwDIA2T5pwP/H19HryPWFAPj367dndY/rHtO9CMD3t7pbdM+nD0qbzvyb5cnbv/Y/yT/+45efAQJmAdu8CQDiMetZcdfFvIRx3fl8vJ477rHzuvXYfvfEVdec88VhF7Vqc8RRv/zw00k6iABLSNzzvG1jJixZsWZDyo7bqNUbfbxC3XTBLf9ciSlh4sRLlCxDpizZ5PLkK1CoWLVadeopKD8zQXPDA133zg1/7HNCp4/kfPIe3/Qx4YXh7vvrZax47bdddseM/545HWt2GuGBzTbZ4hQ+jETjoTA4BiT06LNgyow5EVtO7Dlw4egbZ/58+PITwl2DKOEixIgULVaC9J40hez1udObpEh5j1pUjvC9WpNGKmr91RjAlZgr/f07Fxly1z3tbrujA4LyDSWA6AFIU4BYDjS8BGh/C+A69xzZBABD8hrbUY8B9xLEhbemn5dYU03Q3xMB81OhsOdeacmuQK1SRZLiqKRFhGG0MK/QaHb7ZB/HKS5fuYI1EpnPDpxFnl/axgTlB3zH30cdHJGQNieGjiCgdQGZo4VvV/iW5fIaLqZgIUidW4lAxP7OiIIWznOM/ZY8oph+Vhf62NJK8P79ZVfIjMxJcGtGpwYcOmZjPGrRdvlIYJts/Pr09XGn4uYZhU9JCPXRw0bqctfMSNNhmf0t7R3dTcicSjZ9naYRvTR0P3cpELYKxNJZaOiIe6y3GcL5bC0RbcSb/5UgxexLno+JBGdFmLZnxkk+8dtL+I5WuE+Jak+tbQtpLbsug7Vlzga4rkX9o9nLyVH+Ldm9i/pYMY6NRtQLEygnQ1hPMwpILine truncated
@font-face{font-family:'Plus Jakarta Sans';font-style:normal;font-weight:500;font-display:block;src:url(data:font/woff2;base64,d09GMgABAAAAACp0ABAAAAAAdIAAACoSAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGoIkG6hGHIR+BmA/U1RBVEQAiBQRCAr7IOdiC4V0AAE2AiQDiyAEIAWFXgeaLQwHG7VmNcrtEwWcB5B7zldlaiRC2DgAWcwDUZQyuoPg/z8nQckQl/zwPgCDajs1EBir6uGCZAaVKAzZNiSMJ6mQzFlF2Ja1eFfChTObcGYg+3A/J1XsNNKJbQlpEv5616wTdfqqGHz/3LvrLa3RINnKcacn06aTPnGwusqNaETw8oO84tfRee1ODOL9t1a+4pGjHWgarFji+ZeF5YhPGg2P0NgnuTwPb/fVudXTVb+H5lLKoGhDRAQjRus3PL/N/3MBFcVqjMQCaQkDEAQEpA1Q3AAVFSkD0XW46aJ0Ufmf26tF/xWrdDjX7vUmVLKYO7NHMCQYEhwSvEOhBFJU2upW/S3qSilQkOJHoo51nLt7pK3EWZO1wDwxPVARS1rP3n0WmqQpjEKGbCRGojwqRO1A/9K2iOlDx2VYO0NsjyhYefkWJOMHHIE1LYjW8OjlC+CfgrFh1dD1eEGLgjaFra0tlLiQ0jIAgHHYKRAUeQYR3b9/NJym9JW9k9yPtQEqwgUcNAQG54U8gBe0cBMNPu1a26ZeUHk8ARUExs2tE3rn92Rui8dgZJDvQNiEyfAVm6vFmw5bUBjPNH8r0894/Dm8uZ/XmbWlvr/fRkO7dsg5Ai9B0dlLqJkcAFck/S/5S/r+E1se5WLJ3jhhD5CXwJqxh0LOoeeAceAAuUJogVuk7jqCoqiu6A/+n9Zebd6cEL2EXbik0OfLStmZ8t0t8GbLimBL6Mi2wlSoVkgC4YhUhWRnbCGqtbR3P8/zQEl/LBsbIc9pgg2gAxQ2Om4qaoUMqJgupFAzVlxHdQa87nnmfOt9I73j7gYzCGGEEUZdTAghPb7aY22UsdWAMEQPBub/Ip+eAHp7/V+/ggNBgvjxA/EXBRIrFQQjBwSHAEKSD1KoEISlFERMDVJJD2LUANLICmLXDTLRNJAZFoH06AVZZi1Ivw0gm2wH2WknyG5HQY47C/KLCyCXXAW57h7IfQ9AHnkE8sQbkLc+ia/+E2OQBqAgID8AxR+KACWqeFASywIlp1xQCOWBQi8fFHnkoFREA56a6MBjig08bWkDjzNOcBCgMww22Nr1W3cTeNzBp50gEAKgjBEQsGMPP+UE4T+HnEDDhwDpqzqn8e6i9f4Y1THacoF3tPKPP7KhNb6lbe666pYUaG1sFV+rS6/zbhszr7Ov026dhOtoYLW9r/2z8ClWI4HP3zicz2jzASj/3LwAXunPQw1Wv16u+bC5EX21+fU692Rz4la4t9nTbGOuScCtv6FZA1x3bmgWX3tXNdNv9rzG2dhx8wllv2V1TS1wndurUa5+a8O/dVY2zCvKXTf3ahvQTLxKudaGXkUaN88VTc8rP581uDUQ3j+44fWfDf829Ec9Ur/ZANRf2/pP69tY9md97YR7Fn25/jn6eH3sOnd3w79V7qq31AN41YkxwHsz1o3rznX17OvARXU3mbuxbge2nlDtt6yxNgDXif3qitXuVIuuKF7KQycZzp8OUzjzfoB+fynTHwJC9/m+rXQABLDJ1Q4sdGdCTbZS0HMRSH0zewFsmAchap/O+2AkF52ahPrGZXbqW1W6EDVFQxGFIXNB9ZViFdvyqN8DSMow6TYCSNEzRQAS40v4MlxN5KcOI9NL8uIlL0VFMWzYeLLjkgwccgh72BNa21wPGZsjliitrm4yJsxkZujdHLukCFZrtXA7K5FkMFEThV3s4kP40FbpM7ks5cUwWSBI4HpnyItD+AoTKeiSRxjxpJIB0UirJfKQlBrrqgcfhZX3dqmhgRY6mcAM5rCIxaxmMzvZyzFOcJJneY27POA57/gSvtPJIUEIoLj8jG1kdpIdc6vDcOMBPtvrTXcBkYDhg+IDgNgGiNtDg4L42uS7EkO//IGfeZcwXldNAAi8CZL4JRIMBgmCBKuxEEBrbVMkRZSb62D5w86MijvOf09iCjYiprxghuZ2ZUQYYy/sk0XUnYePdKZfKDPSL4JHlVNj14qb1gNYVffNrrZXkziyS+Pl3pQfRU7EKuClY4fUXDupObAg90n5vC7HNBtPqwm7vj41ykRjZroV3Qsy1VoNfz3DLvyxXKcFvn4l+Itm6qnCLFYmDlDz4ojGGdzj6htLli30K22mXlsFyaRmBhXuEczFhDzTRBTmtkUSO4m7eWw54WbuxPodR+anwC+M+RqwSqsugT8sBo2oQ9PcI1QdrIrdjG/tuR7HZP3HysRFSu6APPLyrAzOs+PFIZGP5TU5tALrUoV7Y+mCrHv4FxsASPPtTRNBkmOQyYZslN5yLESUo2E8ZhxMZCpzWWi9B/brc9D5Kkerqx13zpoz3baO3rWT2w4eeOyIZ/BjXlWPe+uzh092nuUqN3HrhQRPq/dfxWisiv0VLyX+SxKJH8mjMAZtTXkgAJNTDD7mMxoMIQ1EgZBfFFmJImsAIUQBBBKCH6EKnh3YOthfqWrPGeph1yHdcFuAOzRk6RZxpQFZoDx28rrYyLcG2gkQXgTJH8AVVgbeFv4yFp9wUgyRkpRAcL9aSOVoETqdC7IIF+IcIVQxl78CIQqKvgLeiKxKqOUxqGSCaiOuiMN6FrHYziNxDsQpxlfzCwgEESo9UUwJosKDg6iVg8qOy0YAIUTgB5qwPKbyuGZEkWTRB86COUe8BK8ff4Llv3gEEU+SBZ34AdxHH5V/8nu8C7TjhS+hCtTpCQ3Rh9hhS0Q1PF54ku+FUGtp0Co5KimiEHB3iMjRf03sb7amQf3IIshV+6wbSfR81W5N2MVfrNqn+9WOrcOU9+E2fXT5x122rPzroyqyG9DfJI1q26GHHn8STFON/YSHHif8s/QG/GPhxWs5O98nLtecMuBOJQkGAVfSFoFVpymFFjvq8naTCy4GzPmjINF81aqiJMHHxkSRC+IrVBQOgTIqR73yEYyHG8yTxg/E6JVwW77YDLZCXLtDDIxctrXA50HugLmeE+Qxb4N2u3lPUV1jhbm082r+/sgjXQbbDYFqySF6+zZvbWXViE8qUpEu9QatNgU3x+y22OIFH1iM6r0DYXKYHLvDDTFFzPqxbBj+4MIFQsYClpq0gKUiVrDoHNbca05jbRMaq+ytSdW7QOCpiClmiKo0o9qB6pqqCrClvX8OkBnPzeK3AwcCOvtuu73ZssNZnCd9EIFjKUB7z+pGgq4HqIbMAxgJDqq+BPo2dWMvOvPIo04LHID/RHVRUAASDtExCos/AHk+T5cGGqVwtUVFf2wd0Qn8nMdh84LFcS5CwigyOT3pzcYcyJUM5V2FHm7k0aZpg+AIfnBfeOpjTLgm3M0Wm8Pl8falUq2JzQSz9Oiz0oC/q19b2t2+jnaqwSXs/zN0XoAkRKhft4H9/MobhvnHbAkPo0CHw5FfZ0i4JNxMstqdbi8fElq1WrhMtVjvA18etblt7e1gJzrz0F2evvxm1rF8d1luHdp0dAckdy+PNiD/8O3n548A+ObB1xvlf6h8/RQE4CgXjAYDgOwSdSrSkMY0Z08OAZB7cwTI/TmWM/k5v+b3XMilXMm13MiTjORDhYD5wYIXuziAajBANXaYEUYac0UAalvBbeJjG3+7Bdkn2H4B9ghzULhDkvxPrOMSnJbqLLQTMv0G4xfZzsvyO7wrcC5huovuNrIbGO7IQ3JPoQdYHnms2FM8z3E9U+KFUm9IDZMbIfNOuc8O8I89fbWfvxwVAjTnDZryAvb8QUch2gvWWiCYXIxJRZtYlIVlmluaGSVaUIbZpZgfxpKwlkWwXH9062MaiKHany4vWagoR3EIvFJG5aMMv9L7rmU7PzsZjYGxmZct4p2S6AyKm2huqTJqf387Nh9gyRfMLMmcUjkLNaVYGt8cEQxo/eG2qJliTkVsAu0V4bBIR6QYlOwncU7KcQHBVbkuo/o/vpeEXhN7S+E9pQ/UPjksAA4PAuP950D/Oi4ksOYHOgvjKpyjANBWEOgqwk5PFdc08S2S1Tzp9cqtR05L4VqB2Ab5rUJuJVKrUVorD08piTJy5ZS4BNSkSrDwcagoiIjJCDnHFkpXAXg2QO8P8A9lJd2pNPMBogCAWBD5ErjedUOfr7fhOq33jtDlCowtG5dqg3YwuKZsNUggrLZI5t4fQZ1Az73REtu1tgKPi12LJV7bRf3Pxw/2PFkIgm7KdWZAgxSF0CSlcsWwKASXDWGLIS4E1gDdJmF1un6uRJkOpaiU+r1XVTWD1lcvb3UxGp1GGo2sfKLi5GQ5v39r2o1NY7vdTjbTTJLJukCrMvOYpVnbJmrg7S14rbSanW1aL/UUpUFKdq2D035yFCnrrunUCeLwybM98cMuHzPq+9ErtW1UtDT+BylEZbWk2UzULNWlR6qZyTSU/wrCx+A9J9x5mRdJWXJYhFj1hoMFos1LC7NhrNOk91URdP9Xwf67Wikff+Q9K6AchP2/2n7osAnIgKdMSh3/RqP7rRENo4CZTmbTcCbAySWI+HAIaz66TL9aZdSVY2LRr1BPPNCNzbji1ZI12pWx1UIxQUNbQBM6G6Rb2lji1WxtZKTByP6NDW2KxFpSzqU3cfcnt0TPAnnzPOPMUkiC5GsTMO9l6ObspyCZ6XsdnIYAgwDVBXOeFyCzdobm8g0dXawxX+VSvTHpcXF4Dfg7or3Ug0jK2XvDkaRYorZCoR1dPcapq3B1uyB/Bl06iU9Rc48qUfxP9acE7+YdtqsCm6Jk55dBCtmk6dxdeo9uZXQmZKscUI539SnNgWkhD2+8c9DWQ55GEQogYx6ILine truncated
@font-face{font-family:'Plus Jakarta Sans';font-style:normal;font-weight:600;font-display:block;src:url(data:font/woff2;base64,d09GMgABAAAAAC/EABAAAAAAdMwAAC9iAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGoEUG6sMHIgyBmA/U1RBVEQAhTwRCAqBjRzuTQuEQAABNgIkA4h8BCAFhW4HijUMBxsmYhVsm1YzOA9Asdz1Z41ECBsHBIR2nuz/DwnckIGv4bUMBApNkQOattatEBWiYkpyNXiIsBgFRtVJi1ErvaGH73qEVsysNY7wK8qlbMARDg6Gd477tfHfeW6dYpInUwzijINPaz8P3HKEJLM/j5v+n3sTICFBvEJZKvq6qtP6Nn751YlVbKXQUWbuTMQ7dyamDE/T+a+KBK+mcb+INrn45eJNJWk9OSKVVB2xMnFkH9G5UUbRMlPEJxSbAd8ZRf6eOffyBQQjMB5HwBFM84WjqSlpKiq0qjBrUU2ze1cydGkO0HTEPCDvGOHCoK/2e4igxN88w9M/+z3s98x5mFjTpC6RRcSTiIQGVez/Zp48wlXpJNCByJCPJ+PKy/FJJTKpTCKTSCVy8JOXnle0xCWqwrLk1xwZJs5lrpiszZAfWDhS34AACgL/3pZW2n/e9G6P5d2a9izpUKot1Y6XKDWQjjC6iCCk7v961P37q63WjFaenhbg3YxMkuWqFZhm1JJbtNYCQXSo9dFIR2Oz7AOEECnCICEKoosuvDi82uwoCEMmIi7VLOAD/gQobl/STqONG9BjSlVqGIwoBiy9eHNADzSaaRGm5/4ANt/2ZvpJHFlPiA8NS7cKg7FMPP1rv+rzY2n1Y45ezHOe6zqptYYgIiKJRNGS3Hum6e7rAgQgF4Bc+MEV4wlBRVWsWCakjwEAEBKaDMCE+NDaC00nBpnKWBUCPmiC8GUI5oDkKoG0VQmpVg2p1Q1i0lvlxx4PRjwgRoAoIIZFKlkQACE/MHzPblME4neZzQxiJABw/mEACP6usppBua4JEOc4ivsD4G0lPpLOTUHb0Ln78pdapLuEOT3HIvfA7IK7ZeyIrJzFj3zuTP9+T5zRM3T6T990TPNgUdf6CU/ZeKi/Y+DRjWLEePS4Q3eJU4jv97JZSJD4X3/0U9/0GaZz9Hav9nybe0Sc4lnWj7qjm7r21S3vwjk9bbplpkdsql7x5o7Ej/Pg9m1P0002CvSSVjYQOWhtNjNCQXdNoQobxaPHLjVSMh6WP37CJZjCmBidC/tEgv7kWz7kVZ7kfm7mctT3bE7I8AaYS8vjmCWAyvh4pMXDAgALdrCDIqAotmXN4SgHyLxEisY4JwcAAX5wFs6a2qBFvw5CiaSBAgPST72RADhs3jmYuUJFKBdm+ScsHgkeqc5plfl/8IxUfg4APAdP8AeOc4SnAdLgB/yAYiiGSqgEIQihAPSQBVmQCZmwE3aCFaywHbZDYKUxprki+JwDIRtUQiGwcAuBWTALfMEXMiAD+kE/WA2r4T28h/akvZsJ8z0OAOBCIGRCIXSFskwlkALwrAcQj8RW0riwan85j/gyCEBgQXeuiK0AkMfdwmUURIGWs9iLRktc3UyEhwMAkuea/RMGn3YQwgc7EggvRwE/JOFEp9PRN88jechaQnXe4V0fZoZOKpuUdzKWo/NLyCnt2emobRqNLm56Ha3yXJK6OVqCWcHHOGdQ7+gboZJQOPMUPgyyjVNoaLZdxy0n8Cyf3SGXqIXjS9IxZnYldheaMXBnjH/rGzi/9TtETgKxAIxrRUBN9WpX3+egg3P2wXMQ/HJLztg/rc3crRWpAYJq+OBGGGwDm4AjwVxuj7EfXZVMgVbXx5OjI1sIGXoaoQrjvkk3okIBnSWAzKC91yUoFD5MYj0kRgmnuoUzyhrF3Gxx5Oawx2yAhPO1KtFwj3pmrP3oFO9+M5s9Xp2nHvpwq/jaQQtrI38AXWhXX7uCzr+8UsV8NCpMdZ/tlb7mlDpI0rAao6jZerMBxma6I8bAhWxLsxEiya17nPCSW9t63qW8c//5ikkgUQDEUFHQsIvmZJAhXzutdFCjWB2TclZNqvXQV63++jO77ox2913W7bFr+rxxR7+PPuoLi5CBJsBgkVQKZNMiKkbHky2YULYQYkuEklhOSmGhGCp5NLSW0bPAMmRgZGoDMTCQ8/kfJF8+hlEBSqEiIsVKUEqV0ghri9JOO4z22qN10AlPZ1WQatV4atRg1KqF1KnD0003EiYmrAbdEaystLo1UenVC19vvakNGkDuA3spTTtA6Vvfyfe975X4wY8IzjijxEWX4F12WbbHrlnqjdsyvHPHEh99JJUpWQaKAI1GERJqAYeltJgYSUJhPqV8NAUK4KgRZNAiIiIhKUFGVoiCJoMXrmyBeOYLIkC3Nmu5yYuFyMtJ/AU2B4KTR4dGPIvlCS+TRcOJ0/LkagMzmJz9ixDAgpSwCoGEe9icEKwSpIpgUkOtDbQj17LxFgEXE+DPryMuCsmcIZnVn7N2BiawgDk/g6xQIBe5zXMCWP71U8iwFAmgzfYK8t3V+ayb5tWR1b6yBztyvvfMvlo5HYkS3BtbUfPUhCoPUVe99hO0zJasi8i7uZ9G5727tRmfkT/7jagVyqz5y+p8UpbGJtJeKyBnX2HEdd3GJVpH+EWpABkdAxMLGwe3ez8LiYhJKKmou/GfsMiQI1eeVlprw6CzLroqU65CpSrVanRTz6aPIYYaaZQx9tjrgENmfO5LX/vWdy674prb7vgDEtoPUqoSLaQkaVEWYgpxDfENpbK9K/3RwBPdQ9VDU4gvfCwgQ6J5hifrkg+iD6Pp6Nvou2i+4U0juCCllEnMxmROMTxnZC486JMUJEuRKo1eepmZ2Dk4IVxQbh4+9Ro0ahIVg1kp6Zbbv2aw1UiHkX3B4RLMFVzWkNNd1HDRIlKAjoGJhY2Di0d4tpFzb23oJsSoiM42WHO3NWrLzGDRqPvGWmt6ptdKH+nPYJu+0E9/A+LgR9c2vmsFUJBJsmTLkWue+RYEN5pFUICOgYmFjYOLRzgobkusGJQyMoGZWUpbYOfghHBBuXmGdgJdKpQdolp1qWcNGjWJJgaYlYnbkklRmrFpzRo0adGuQ2fZlfQYOGqz7LmXD8m0b32Xy2O5xm657WZTd9q2pFZsFw24D4+NI9FNbm64ZCms9k0K6BkibmrhOuzh9hFH561EXdtncTw3fVGWFbeyYs9D3z78zG9TPPaRb4jvepa8F7TG67ZAo1/C2RXThMH8fEKJWJs1koAuiF498yoz2WDiq76hEYbHB9H702xpAOuXtawCat91NWXw9TgRE3TsVfLTo8EKxl+fglZ/4HaaTR0XA0QgAB6QRFI5ujBLKI3pEog5pogPnpS868pHP3I5BzI0ttTGF11q/emJsw7oC8aGlHw8MCHsqy8QzOY0PlBehAgoHEQOBNgEgpxmLZxqsck2o5THJDPh8aPjQo41QKb2RoJsNFhqMCzLrTYEHpAvM8gAl11OmMnwl28w3XasaYBFUjRjg34hcjGEry8uT7eAsol2Qbm80CqS4as1cUd+/ubLKSm5xXTw5QcbYwpXszETi4g4gxyFqZQP0WmUQxbai+2zlHMes9EoM0+7KeJCbva3J1rPypxVCPgH6u+NNWpVSwEXbEw1NsEcQVNdhTL9MX4wwoCuHBLE2CxLs0xWJt9QbUUwMcoZSq0MFTTfWCXbELe9LGdjbJlaSYyKHI4AyYICpkAVxSSSiDSLFxQDj2RFBIicGUIEIxZJkc1rWA7oUxQBTp4U5DLc2IVnfQJcD7jZB+2c01vnw8dEu25BLnF6gTU2DrYzoGcA3aFzwM4IyIou2DwWdu93//W1NpQFcJuzBMI5BieFSnFYACuRK2ubU09HKdmco0Qmu9o9Dux9Ph9FQsUb6CXOFIUn1BO93buDCJKgCe+dMXUd6nrUzTk3Tsd5cX5cJJfAZXI2zu7p5TnFc5qX3EvldAIb4IRJsNgGFWLZcUAMQR27GefCaR9VBBfvs19NeC8tWTjwy/5P/y/tnwCAn/8djkNzqI4AgN+/7vg4vPx2vR0mR8aDYw8sD9L3r96/Bwg4BnCF60F9XhIelxcz1xOZpzPTx+sLB9130jtODz1w3gUnfLbXaXs022e/n7774TAHRIghIjny3LeahpYHnRY4XoJmVWH4eCW47JQr/joXF4ky2DggvHz8AqpEVKtRq15cUkqzVm3nmaSn8IAvvXLJb9sd8sYHUoPxmr7ko/HEQGf98TQenvtlsy3j6p9HjkZnk0HOWWuNdY7gw0gUHgEaS0FKRs6dC1duxDz58eYjgK+v/EUJFyFSvP/UM0mSTM8ojYWd50haAS8733Gc6ogeUTMskvuFOqW169AnoR+P6Vj67U8mUrrplquuu+EaBC1OEkB2AvQwkDOBzS4CsO01AOYJ0EsArOAlxP2oDyUaSUDT3FlNSGdwQoZjuTFgYQ3Z2h41AVfiKMssrzqBwJknGlNoEWJ+JGrhaFsS+WFI35bGZYXIpPMLgs3VFwPNdKEtgVFsc0RSA2AgKqQJkUNDFU9kfgIQIbKS7A7TwA6G9S2wEGTObUQgYn9lJEELFwVil7VIKNLvwkIvW1kJ3j/f7EpZkD4ILm1MNoChxEzErZHiyEtm+jMTH6+huZ6suV0tzBViQp30uJM6vSQzKfnyzD5bIqz7OX2o2Ax9hXKSJuhhcQ4IO52x1RsYqIsDyMlDOB6tJdBq2v0//vh0KoopleCsCBftWHCQV3mngndKR5srpJSORtsOjbUtthwTw3wa5qZhDVWQXgUqS7yaO9P3In6ZF+1oFZvSHhF0VWdKkYLJ+Z2FgqSK8T8ELeeXl5J8YhFTzhAR9sItXMnaLFnDnKEMY6eaoC8RyaKcouqcnGQuoA5JKNDK0qnq79SzogYm1agboadjDHSgVIbXLine truncated
@font-face{font-family:'Plus Jakarta Sans';font-style:normal;font-weight:600;font-display:block;src:url(data:font/woff2;base64,d09GMgABAAAAACrMABAAAAAAdJgAACppAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGoIkG6hGHIR+BmA/U1RBVEQAiBQRCAr7KOdWC4V0AAE2AiQDiyAEIAWFbgeaLQwHG7lmB6bbPFCgO/BAKjaJbCRC2Dgg8LzPF0UpY3vJ/v9PSE7GEKaHTbWs/1AgUuSsMaZcsMVGZWIJVcNbKNXMhfK+7L2R0YBws8iO1X5ixRkJD72splTbwpIHm9xpBwc2idV/3KbTtOHtygbThMl0hAuaqi3KWQMaX9P8pqtQQdIr6Eem5/XE+MiVQcKXmuqDMtTzcr749x+xOtyo4MSWwsKiq8cdHqGxT3Khorr4j6zuwYPhEz/NK701PL/N3gcDxSwUiypREFSQEkEQo7AGIiGKMMDcpi7D9cUqbh3eKt1aVymbS+fF7twWdcpe8XpHcBYQ1FgBcg0pwT3+n/6e39rnvgSDONAsbJqaJsEEA82iQCz6FK6BzUYkJwo1hXcP/8TcJhr69u7PBEh4W8NTpJJFSyOkxMQ1b/w4UP6KXmSIzxL+/73RG9gD2uQKfNOJjyjSKAh2Jh0fJ8GoGpjnQktrdR2O4EkOsmrbEuI7oo/YIdYYKuksc+symfMFFngxL62qH2W6Fa0JF8Z+S5dJ9l0yI3lDgOUzLdw5SN0Tu7XbR+Aq/AmoIJDv3vebq8WbDFtQGM/k6yfTNjUefw5vLgT+z6bVhl39+z9L6gH1yn61xNrZuwtyaQnLngPgiKSqLpXb1d8k94AkD5KeR3vAls/aaXsWdKw5wAgpRIoQo3sbRAcY3bv4gjSgILwgDo7nv7//8Wn1tzsPKMDASrWo2X34TiA9bBLJNMAw4a50ZisFFcAq6ZCL+tvmtPfy2SfL9suPsh+JuwChXo9YpWKs0qYlKKqkTp+Jr3tTPRuStDexqDc5FwKHzx3mfPvcGbV3skcfwSzGBCOMEcG4Ipi8g/T4av/20xrYXkhDQ9Un3YbR5H/3rn84BKCsL//qG2QQCCC88QbCh3AQURJBYBBBkNFAMHBBZMoEwZMLQq4YRBkNiDp6EEYWEFYdICboATHFPBAL9IJY4lcQv1sDYp3NILbaCmKbAyAO+RPEGRdBXHYNRL/7IAY9APHIIxBPvAEx5DMx4n/iJ3QHILlB3gCSDxQKUngokOLCg0SMAhKtNJDYcUFSSQWkUlUAV5WqgcugZuCyyw5cLWoBDgSgKBh8sP4Dh4/Hb/mc9Svxww0A/hMDIGDLFqxdScjPvo7fqgkBeNiKJvEoVJ7+E1Xk25sEvEgbPvJBNOb6fNzyrFXnxYB8ZC69f/vmGU+Kjo/2CE9I0osFNlr5BT3nAeTFER6I+/H8ft9w4IcbAkotzxdedexZzsw+u3vBW3f5gffcnXw8Hrq9t90oJwBP+KZbAzxoLtySB+zkzXz0vTchawHrtdXyuBoyHeCBNzMr3/jYLO8JGZjxM86zlnkUIGv8aPdf3RvSr58UWfULuwA8Ad6ebgA87LSw/zyRftnhfVN1R1/J9k/2DguFvX7tPQte2tPBY3vwgXfX/vP43LIbdhWsuC5W2d62esaD5vhOfyDs3g7Za/86AJZry/G4jKkW8MCYnpZudHQqOwcx/FmHApkn0IGEimdqOwDbgyF4VVRXcunWvVuJAAJgAkKdm4Bx0B7nLCpnMTj9KiBlF7sBzMwFN2YXnbcAKAU67WR4TaFFWKQqJqSlOUlz4s8pZVGSl9lYXUJx3koqwkuIB2wDDUDZNUQAqAj/uP+caT28bdsq1lHp4G29BQohG8QgBld8iGoF4+14wgE44Fp9+l38nFSU9LluAZ2Zkgrur3NqhqqD5XY54U4spFpBZ+if3SKwBT65n3xrsGoy9X+DMPAgEACK4g0CHwJ5ESyMP1ESBcMgQiGjScTAhZUpEx1PLga5PJmUivGU0ZCoo1fAyKKEVbsqE/TQm2KeRgv0arXErzr9bo0p1tlshq22mmebA+Y75E8rnXHRepdds1W/+3YY9MBBjzxy2BNvHDHks2eM+N+rfkJ34F5ukDd4kA8U6nnhobwvLryvEaMYi1ZaEMSOGwIEAmwEWH3BjWzplc1Pz9TY5B92LIg8MCQghQCIfQDi5VAgBc7g65JzbPclmLBh00jOlwBESKHUwtKMHPE/jKUXEyHvnw2zLAnCfXkg+l/cmRYWV/u/E7ixseX4C7Oqprtyo4kmMZOp67kFO1wcb8Ss4hdmx/lRP/go6QU4sOfXu1rUj9Lhu9rCx6XduyP2H7ovXblcr5W34WaGd922vWe7uxd0x4ga46quV20s14+Jt5pRex3Wbiu7dgzsZklHXIrk+1iSajFq0CoDW/S0lRowd6aWdGP/HVNlrxMasDVKSPOp8IKS8U1uaXUCD/DSvPY63YnRh0z2rAAvdE99ZDu1uHP72A12I61HbVtdd/lj5JWv79mat8bebyW9Q3UaOrC4w5NHejh5n8r8Fz/KG5UZETP4Sw1FgrfHf6UBIMprSBIK3TBSEKQUcyjBI1OiwjhmNhN0m2lu9Y67LLYntRUOvK50yCm/PMBr4+aerQZqz/jYfs/QDnr1esiQLx56jHvGNTcN5AUnrrUPf/EtXgL+SSQx/5enPH6kUgE/oZZKBAEYRDd4lucmgBtJ3LobXE6YDwnrnVvcuuYnkLegBPQLdoP9ZWr/VCG/9UNwwx2+3KVC3AeEnmAgOpLHKe7nmHj1LiEQd/7xAXiCHeBRIb94PJUtAW4SS4wbPYoJrAQrgnaFED1EYGpuBCXyrNfdCEciTnzvCH+CHJiC7hyQV2J6DN69R+GnFiZ7jCEgse0cOfyRyo38nHQqok/+FCsBOXwcML4ChfIWLfgS6fNhtIRDV8T4JxiniHvgnvMREJ9h4w8FXf7PH8H55JO1z34+OC/hzktQ/GQ3dhbB7zBffOLQGZJIIuL9t8hRZNC4Ch0VqADfHYFQiv4L4kiDX7D4s2ZB9Jk6oDay5z7jF3ds8NJnav511PAzmR+nEfZTCT7HDjN/cbT2vl1Af4dk6gzz5q1YDWbpjPo6z1tOyNT0AJRcJNxx1/FFPdXGdUetSrdnCAABzqFZ/DpbdBVUlP3LOkwqoEgw9TcFEcGLGuUK5ZHg42CigPAiSLhsORSKHPDKJzBcBsB4sgS7Sy/kGWBcy4/At4HK9kCrtgkvzO9LQdwFo78WoDzzYND2AR6Nd/5imZnUPdqOR4LCFAbbDgF5QVBy8qCXG7NOT5WqFOgkz6DIkAOzzEGzmuWpHmCyvnCXG4NgbN81wK1x2d5Py4fhA5wQfiDUBExVagSmUlmAqTrLGgpmGctNUMHKClUq3woBd3wlg8y4Wa6V4w6wozoMGPahNRNcYl7f4Spg1un11zb+bdgo/pT9UM7x++mBpHS62kDnALAa4wKoDQ7kdxNQfnHqv8SmRYvXCw7Af596CSEBRAi3ogth8gGAdGmgvA3tsg31skSGwC2C387l+NwhyXYqWlIVaJIW1Nvadne1t72f6CFPyrCmIkBwN7gXPPE/jP3G/hGTUNAxYc6aDeOYxhJWsY0DwODQSLwl/WTIV6RSHQ2T5t9ho/sEUNtkKv5aa9rV2d70ru9DGNrThQAe4IhfThD7if0RIXJaxsxYMTCcqcxkOZvZDQT2BK/PpstSqFQt9Z7y/pTr12Lhxz30+vBfwSxMf3/d3+Gf8M33nAFf33+lsgO+egTi/y1d7KJvwQAAjVWxSqWXUQ3a3l4AQPe0H9B9Hex4pzvb+S52uatd70ZPlHznh4lIUIwwMYBVNYDVxw4PvPBGYGUBQMpGcNbxZBMftvFnpwC7+LJdsD1C7IV2QpRDYh2T6E/RDsM5B+MMggvwzqO6iuwyjnvY7khxQ7q70jDcl+kBnkceE3hK5DmhZ8ReyPWG0jsqw/K9V+KLmb6bZMR0/1icG2jIA5hyB9Z8gLNAjgKMzw9MKtLEIkwo3NxwZpZkSnHmhDW9BLPDWBjJkmiW+j221XGsKl2lvx0oXpBwB2TL8YpCkU+wztIYk7OZN1vV+Qki67mzAcpRcY5juonltnLfzPCvZXmCprzA1NBmlKilIF1FqTBqYTCg9pcLpWaQWaVqxo8dQu0TZr8EfeKdFOMIootorqG4ItUtEi9JvSY3pMAHhT4q9tn8AFgQBMb53yz/WR4CWPIGrQVrK4QtX2DPH7QXqrsYPaHMC29WyXqjWBDRosiWRbcmrhWlWB7Dyph+LS2RcpUnhVQqUaGEylGxlBKLl0TZKlKBZJIrX1LxgYCMd1cKQFsA9gt8S7HeKLUAJTobcgCImFOmN6BuJRU+dedAXTJp87IK9cVsUQvduW6zZbWzDHYZLM/GFNyQjNyB+dcgOLdGWJ5rV8fGNjRkD3BAvX+6ZjKfThS6od4OBjIss8iMAsgy0klr1GsF0gmDLICkQioxNKqZnokhvh9ioHKVM3OW+1EoLJrSiaNTGXU6mw47Hc23wNDbaErXn8RFV9nVw2FtGTAh0iucKq8GeVhOg4KTzsPuUJzAZcLJYKt92ckmkE2MTRlWie/cgUw8st+nvmxl7472mqb5VYt15oRlGRhqz/+e8Uuo4s7BgLEbAjZ21C29fg79V5n1w8yVqjTAEzxG2pEd7LJFxIMDAlEsh9jU5z/ei9F79sHBfapRPnqQf5sEvAEHz2b1QOYw0wLuHGnked/oPaaJOlDAIOY4rDMgAGfi0+MhbsMRok70thRdPoJG7A36caMYoIKTlmAciuscFE3YHQ1L2Bwi27G6g81J8+XYIUYOrvjFN4J/PtpcvE8u28dF0NUoG+FBM0SuBOnBBIz4A/SGHPRBb7AUZnAAImsQug/RCFJRuHWQhtvjeslYS6Ift377Line truncated
@font-face{font-family:'Plus Jakarta Sans';font-style:normal;font-weight:700;font-display:block;src:url(data:font/woff2;base64,d09GMgABAAAAADBMABAAAAAAdKQAAC/oAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGoEUG6s2HIgyBmA/U1RBVEQAhTwRCAqBjRDuNwuEQAABNgIkA4h8BCAFhSgHijUMBxv0YSXs2CsCugMODfWXYhJFyWJtj6KSEzH7/w8JnAwRqguovxhhwoiqWgUnFqtF8Ru7NaIooi9jo98VloqJIywb/wu1gvrhDBHjS7YjVp8dRVEzZu5XqD9nvizj1PO6/207inVHxyM09kkuT/+sAfs78xZRT0QOzVyaeBS1jCcaRIuiIcKXCDtEc9ZsCJogFiAkkIQoJE4SIiKQIMU0AU+CWA2oUpfrtXfXlmtPrFTlvOJfu/YKNTtrH/+/nPpvJJnmSrF66qZO4yD5FxhmNIkn5Djhz5SttS0iuO32bza/wQEmWRag/+KczlzpC6HGfIYxZS3/4N/484gSz1rYkhZYLWrtwwf324AACgLh/2s/FU5Fia98ZnWA5QaIX4BOWr4t//82/Wzn6Xr+H1n+IGs3scM+G58/S9TrL6GCVSqC0795d8aaN09jkAyCD4aAZDtnDR8MAY0t+8vmJWLZS1oC7gJU5bgKlSmalEmZKl2bpgQsc1K0TaoQfH9omgVLAfoToLiV+iYYDFCtWLpScgdWhe1siGq+wyNj0zxxiapimV/b/+39jHOlSV9CigQ7m0r2jrXiWid7ERnSvv/msY9huNWh4z7DSKokapt5fQI4AbCt4dndUjAmHJxkyWAgvQYBAQfFBCZCCKgnQalhyGqOdIn6rMmUCUKnBzEqAClSC1KvHsSjEaRZT8qXHhYYFAyCEgqFgaAgoURBIMCmyJRgxxxndYOoW64MtIIocIB1BQME7OZrfa0gYVcjIGomoLE/0OU3IeB6I2GgZWy54rMhdcyzmTDXdG9uzuX5bc7NCeM7ND98+K9mbK4/ma2zflbO4pk3s2fAdO6a9mmcWulfPkWTN9bR46NTjjiRPUzMJXUIgxnUxExEEBz2f9xs/+6rPumfvdPrvdhfzJfvTI/1QL+7pi+6ay53ezd3reHL/bBHOwe/yjPb10Bb62k10EtaUKeJmattczkokjnllp5F+ujIxfnQ3c4T8huPgoXBYND7gXCbeZcXeZT7uZWry+KcN7Pkp5wCeQP0SnElTVb7riQB7uKKxwGcIBVSXcxS33ajgDgXFHaRPHEWGwAkQUPxLwDChWs2fXWBcXEfAAaP6Qka4OGTsPRBnujhR/ELJ3iweGiKfUIQ1wCjXgROofuEJDIIKJAm0lKAWZgBFrDA/+H/UAZl0AANEA3RkAcKMIEJtKCF7+F78IEPDsNhyAq5Rzbz+E0AIqMaMQjaRBvBSlgJJCCBEpTQC73wCXwCxVAML+AF1JLaBEzYqgQAqEADLbigEqraasUAVRW3GF2R68gVo12E7HN85LTUCSBqoCS1XtcLR8C4lIUOeEICBqEnYILWQgAAFyHF/5qBbfadCLCNx8B1DoiYI0eGNUyf0jT2ly9D2nNiH8+pf5jP6QxflJN2OWXGjPUH0RTZndOYb6Uxc2DjdqzJXBAZLEe36KoFTOVqsRim3QCY1YIKIqNhb7CYYHShTln9OXB8lHGLuadsWLGHpTBhprXEeg3amsiM56dlp0Yv9aTqF4Y9YWzfVyNUxErNVNasa4MOrdmH7kGIcFsKG7NzYvOi2o5MSTDl6rfS2ghpClQC7glWoe1xzG7VijejqpBPdD1BOIN0JriHsSgs0CosQhO5EC8DztBXr4MwwrC3KOyZYpEp6iLMOPZioRUcvQPIUQRAZGojKsGUOvJpZXarKpmvm05+ooyVeviHdbLWDjlYX4sHLTIy9bxfGa0Q9B86RVk/lqn4W1fpaaw3ZRBEmjUJMBrStwEsc40u2IgGRDRv09kRO57kgialukoOv3jDH+X0/ncGJhokGwACAtly6PAZ2Gg4TGNWokE+r2bVfPzqderjMWBAq8uuaXPLHX5/+EOXJ57p9tJLfWC4IOFCISDBJUoEh4GRjIAGjo4OjokpCgtLLLZsEQQE4uUQiSGhBqOhgaBlBbGxiWNnB3FwQHDKE8bFLVK+AmEKFUpSpEiYaaZBKFYsXIkywcrVgdSrF6xBAwQPD4iXV7BGjaI1a4bUokMQHx8MP79EnbpJD9GjB0qvQXHG3RBzyG0xF12S4LLL0K64KtU116DdcluKO+6Ai0SiU09Bnl1FlFTEJTISxsPhDRMh0mUrSxApKhYtPuIJFHgh+yOgpIJgQLBetLQYSnqzN4SACo6G1cNkykIQmzNgRUruEeufCKmxeGKsTKEWngSndJIYsOVgRlbBwNgsnPweQYCEGIKkhGiBlZwE8VqQOgIjDUxsCI/FabqKdCWiCEL800gypRenz4s0IJAyAxFIQPTPIHEU3iRPRbCC2J9ePMg2cMApyyN4by6/pM4sx5YD5kDRd/FL9Gps+WRKp6A99aEi7WaUDw2Ui0zxAeq+mbqXm7kVd3P1E/JwV0YzRxXbY462Uvxx+SXyutu8jNUHYgWBCfoGuNoStUb4SoJBligNQQYiEjIKan0r42cMTCx8AsI6/h+paRgYmZhZWNmUq1CpSrUaterUa9CoSUCvWWabY675xu13yBHH/OI35110yR13/eGpZ16CsIqDFKoVjpnHKoefI8jJKSAqUEjvWd7LAsH4NQhqyMkR5bzMgcxqKdRwz+aE8Zb2t3SopYstXWopzHAFI2FOLeN5+/MO5V3Mu5QHmd8byHACmZNgSRJpIkvkiSJRJqpEnWiDLuiDIRiDKZiDJViDPZSHilAZqkJ1qAm1oS54wpPwNEACPYI7Ks2v9H5X6wIqSMCzXyCEyCBVIoIMRCRkFFQ09IuLTLYe2fsuxdy4uGCN+82ZlqQV2rTr2Pf1yN9JZyu6oDt66+iDfgMG4448+2Px+5IHqg9GQeCChQgVJjwhAlIiggxEJGQUVDT0Umb/JE1IycgpKKmok7YWHT0DIxMzC2tdJi0NZaSSTKmyKI+iAipVqVajVl3Ut3nCaxr7zUmLR6to064j+UJAz0nJYvzTYj8dctGluJOLPyye0NPes/6lXG+B4QPt0xGOBX65jrzZiaByVIm9lGUBQYcgpO3+N7YLx4Y8L2etsCWlw7GuKvfdm73mlVo3VfHi3WEtnX44bOFTxXnPPAt+wwsRXJewO+xwEco/hIRlmN6thAN6aeiaEVe6OQAWX5mWdjA83o2eu7WtBfgeX1UN8LzehipwJB1fMyjtVfB1tviA8zfEgfmPjWvxEhBREAmhEAxwEh2aXOiSMJ1NBAnqcUI8ezHiOqch0N85n0OZnUBKY484pV657yeH9FEig5K8HDARYN56AwITcE4IoAgiBIGCA4kDBLANCOQct91hTQFwIxohsA8+GFmaZHGQBsEUGwDj1AvGpl/QmCjWB+GML7eBgENuF+GkQtZ3YDzlAisAjsj1YgvIbnEyU2we7l1pE+MJZuk472afkoz8LJO4M/c/c6gECd5SaWCHnml+KQLFo2MYEpWCHAdjWQhImq4FcKFYLnYkF8HsSifYnNXJlyzOGpATxQvBumBA9vT8vWHaNSuVjQSwtdZ5AhMQ9NRVkKOGCQkMguE3hwTiyIM3JjHehCSYiVcPGHF6IYU2lILNVxaCF5IoFtRsHGkzJ4gSljiwhEKxoxcGxVeFsSTaF5PIpaMgmAT5QiH4I4EEBQYhoRjwmnpAMNBHWwQ4mc45l6HGYNj2Q8D1gJs9a+ecbnGArSL33QEeNUUANtg6tXsT9BxA9+4kYG9BIBETwPajtH+/O6/3BKAgAA6IFoEQjSHgQWWwIAGwY9pkE9iBGSkZStYx64X3s8RxlKvP2zESi8h0Y6ZSiBNXT/V6bw4UBA8KD8rYG1YNF45D4BJwaFwajoAj43g4MU6LC+DG8AT8MvwKQhwhcWoKzwBbwGET22pPCjlvCQiaoLANtsMl4zAfiYsTPfljgDMAWLPigH/PqaocCgD//Oe/qz0+90OAP9+z1nf3Z6X2joLm9onbbbdbb9279T8AAY4BXGEiACBvGpbk5az3aebZrPZyfeawW057Zspdt/3iV6e8tt8548444KB//OVvR90BiYAQKfqd531ASYKBlSYdDkGmroqNl0vsvLMu+ODnJJPQ0NIzsrFzcCo0TbESpcrV8/Bq1KzlcyYJ5O/xm0d+959vHfHEC2qj+Zg/8Ely33TXvfcgWA/960tfJcX/3XM8ab4www277LTbMSFg4MIECxUOKV6MWHFSJUuBFgWPLAMRFclbFNk4uHhEsjRRkJJRkVNS07G+k6YQeZ93YgZlZO+oee1E3uXr0KpNuy4NutGE7Uu//SsKjMX86JqLLrviEkjWbAXZC3oYOZPtitnVzjxELwGLrsNaasQaCMuF0w32adGKURT7cTRY3zizfeTRUI2jrExzwmCXkTYS2hOWnVC7ZipnPiL50otzDMiyno3Oco3N4BQMSrdojOXIiw4oTEU0mQEyVG0Q4jfOxTkXOdo6kOUBhC13ifPAiETKAQuweZ1tOXlB147FrKlTWcxBFK/5XSTD+WE3j/OmWXZklFuKZKVr1RZM0bhlNeiUilr1cddibtLdsbtUtDGZiK+PjrYLLrbVU3KXHYJFFQhUn5RQMb0f2ra5S4dosLAEycp8O5SI6X1AarSCidLX+pgF68beedlgWoll0YS12GxeXlVVOAwIasdbXhyda1dXx2LFI9y8SNh1bdrWUqvSTfLH8i3e00D/QEmtNR4Mj7GsrQ5b1wF+mqGPXWzH4zxO2eoVs/I7WJR+nOSvc2IIA9yzhUYK3holmxwQFzY7vSmTI+D4y0cCS9mzcXVN3IM/s+GJcwalLine truncated
@font-face{font-family:'Plus Jakarta Sans';font-style:normal;font-weight:700;font-display:block;src:url(data:font/woff2;base64,d09GMgABAAAAACqMABAAAAAAdDwAACoqAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGoIkG6hAHIR+BmA/U1RBVEQAiBQRCAr7HOdLC4V0AAE2AiQDiyAEIAWFKAeaLQwHG2JmB3DniwZQO0G6Smk+e9wopBOcVDZR1O7JSZ39/39PbsgQXQFutuo9p2SSpEAylDCxFmE0GoZheKGKsztYxNWGtwziBpejbkLJTTHh1c4MRsJ2nNEafgYYeKbnddrEYvD5Huxsvm8oFHy+O/wi893QoKJmVJAPWW3GEnbCGvsVGsak9RgGYSZcEL9gJj3aZyYq+f6pyfTDFfaBx2hoJDHteZzz3k/apKIUWtxmghYRnYg5EzFgnRkTkSAMuVrHHTr/0I/1O/f978myZkMS0TW5dGY7icrQSJ7J1fIOwdw6o4j0EXpjwSpYBctm1GAiWQNGSb5EKZjVYMesof/4z2cATE5TJ/eXMQHrnBlj/VxOnI9kK7JeesvW8EiPF5AU0ABauAAtoGsCFKjgnt/c7D1KLPHYYww0LvGkfzs1tyMvd1omT8wQl2T20hIW6cxd6Uw7urdDwCXigemZQ6RWagNITZMSoee93QvubxCGhZtgEWQSYERRWJwEFaxtHnFbROVKbVAwHJsFwUVw0eMFXQxqK6itrWEZK4CABFQQ8J8/93sDPXzoDeQaTngI4wSq5GGU/dLIa/tffpfxocnSHfYLxAWW0u3X7OX8/9Z8bd7eEk0BUKEnVMyqjudlM/l9OwXen7IKqJyVxf3MuvUkJIGwgMaxUVUVUpERovM/l04mw4yvrnKIQk/JGfMh+HM5LKaEuRJfAckNJEPuRgTPVw5YtlOAQk3vTSi3K11lOrOtPtJh6Vz9jKFzrQnnzoBB6HKs1cysdk/qYxkWngC1xrUjGThyOfrsQ0PocohJ8EnmyOZGVVRl36+RRYKIiIRGvNA4M7t9z7/oZgXzVAzJusA/+dlv/g/WN8q3x8p5EY6K5LFw9f8i7wUCGF397jOhiIgoKRE1M+LEG/ETjISqgdQSjkSKRKKlIGnykSItkFbaIu11RbobiAw2AhllCjLNTGS2RcgSy5GV1iEbbEA22YPscww56RxywWXkqlvIbXeQe+4hD1QhL31EvviHSMRA8aQEpSZ7lLkIlHtoVHAEVI3oqAZxUNlloworwdW4Jrja1Q1Xr3rh6ltfHAIo+bFip8/maob23WPlGVo8AJIEAGFvnwvPMH3pjxnaJEHADFvWwBuYXMme/kQp/orlmKBg8KKxoTMxWbQsu1YnKgBKrMhd2noiaUcEiXAxcluCxQCAJaqHBo4Abx+E3/mKUe+Fj8ClJ8JT3HuVRrFYeFHXxMnqgnB++R3J4a3Yme3ZaArFgG1fnaVgxcxO37KNT8vGg4UqoRT5c6p0S5lCGsDyKxZMS04XZNtiFdiiYtfGCgSABAjRS+UiBJmZugN6wWMwhZ0ycO1PWPJn326nKvlL+eOLgOUX56/IL6PHZ0/MuXJ1yINqu1uX32q/bMVSFxfSCQdiMUB6+io0VsxYm5bDQLlDj1pTXgzkzqniLcyS2wC2vV6TW7FGs+XAqQNBvrYbkL7DovWwygA66dc7gfg7cTUA2TOWgw5Q+ZjyKuQhSZIogUv3K8DM3Q04MlxwPC4ZOi2BCxgqY9GqZ9QiWsUNuYnTknCOos9QtcEJxJVUgZssu8QuPsvniLt4iBtyw1QyQAbXznU/9hqlvIKHFy5jUiYFFWwijTR+eG/JCYQI4p3z7hiuAU+9mkRzTDFiZLIWhrPqOHFwGy7lkvjkbZETJJIQV7jCf+d/R6Y8h/v5L8hxZgIREaCkJFIzULDjQMeJNzt+grkKVYO3WsL5ixSppmgpakmTLlKmfNGKtJColbZytNdVge4GaGywEdoaZYrOppmpn9kWGWSJ5UZZaZ1xNthgik32mGqfYxY46ZxVLrhsg6tu2eK2O/a65579HqhywEsfnfDFPxdJxFAZT0rcSU32HmcuwpvcQ/scNoJ/kaI3jhrEaSoMgMGAvq9aqDt1LKs1SjYtvVoxIBdJxySiPAHkKIB8mTZe4OJYOzjUMV+EUkiCkX1jCYCHHLpfiB4jIiI2H4IH8fRCFC9mT8tR+k5Fwixm+X8xrmxs+0B6YpL9z5V5SxPNpbkzbYXuQd5WyxMxCV9EvmdJZb1Z20WhnpZPGlRW58e0bytXt/9yfrlXFNuPiLwbC+o1e6u7NKzInxmpukR/Kp3YKV9SaOudMR6M6aOgfsjrvivxfaIWBZDVcyBPlpJlLuUUp3a7it/M7+ch2xqFPzp4wpzzvJS8sv2RzvtSV1mbrSnEaKZVOkt8bbW2KsYjOG/OuegKlB5lemQGnrgj1SDeqRsHl0du0Nv499eQ7X4d8sc85Z9WX+hHY+E/SorOJmIL8dbS3e4ZofyUxm5f+1p/UIVm5OYAIqbi0KNfzwAQSdHDhz08lJ/aCEJtqtOjQDRSUYASNEcn9MBgDMd4TKaZqW1m2fGs5tsTXGCf4xbu47RmWKUNbLQjdd9uj6D2ehbc56VP9u/mOIHLuA6bPKHzqoN3H8XXBIn8TFKCv2lK9z9DOSTKqyACE+wGBBKG6fF8wPu8cBEHH8RhdPDCDw0tDFDCKPpxwQ6xX0lNf5xU97tKdE05jQqSmLCx/zYGoq9y/1lk38yIYnTIQCDIoBM1wImdA5CT6XkCQjHF4ksoAc9kPkPxaM84bRxE38TwrHhGcfyGz+eZIZZv+0qK/ohqYogSYcgruPguou98J9Zn5SAm5SJWYpw4SRKdZCnZaXKRPNxLJ18BcpgnDBoYYA8lnGFnHX3WswUzPMiSOoY5TsiE8xxq6EU9bdDBFR6ky36P88GHqD76pJ2HDDIoYBStTOvW2pKxZuWOoClJ8Tz9F9UsEsWzEchYDs2GwN46/+qwV0cLW+c/MIp44s7+LamntmuLU1GXts1uV4q5fnVuFvNyu6BuV7LmV43C3s8n+gUkZZB7+rTnG35L0Okdt5lhukvIoWY4kZG58fMT6u+KMkstVYseAYiI0eL1NSxjoQivt6HpC8EAREQsFJoqlitdIqswdVRDFIzMYiTJkGePZz5gXrFhHojaHqtHOhuFixWwiqFw3mDpOtFX0j6NVGCu1hflSU503kbeeXChucYrHVHqHRPWoUDsPCEvLKnyufL0jFhQqLBC9I08UU67RXWqk+pWN/1NPZhu71GBJ1jGFhVsvwbm/t7Wyo8ax0RLrAtW4zpjFdYVq8kxuNSxR6faT98BJVTUo3HFr0K4CmtXJ/xIGYfIgH0ciYA0dnd6ICC5t6mXQzOA0qfLv5YtGsvF7KzzPtI7ZCtlbqNPALDt4QJ44BC8E4BpE/G57P1LV/VxwLeduqxPBWLCg4ax1IClJANaYocyxIgTE0Qzxc/i8+zMSiZEjOPVSFFOQ5vWQGva3fVe9GY4D/ygDsZImiGO5xRc4zCDyWKzO908vHz8mKXIU6LOr/oNWmSlserf+ra1swMd7dQeiF0qoWlC0NFU5b9sdbs6X1Wv+y1mkCbDwM1w4scR8RvNVjlc7p7evjSSpSnk1KTPwAQ/HrSuje1oT4c7PumGu19138b8/9ytuNnM13Ov8a7x16E/8OTDq6d4vPGoSP+HCjzaBQGA9s752ngACJeppXX23Pm3HwHQ3hOg/Wc3T8+/+PKrr7/59ue//Ca7tWwllQCdAvTRM7d0dLVaAYDcGpyVBGupbaKzld42GpvZ2cFkJw+HOdnHzUHejnG2X4DT/JwU5KxAZ1R3SagLwlRqoFxt1zRUoZ5abol0R7R77ov1ULzH4jyS4IkUVTK9lu2tLG8U+KSp34p80dhP7ePRMTk6JEP31OiTQe/0eqbF0BwNyWJwZpNDGR/cqKJMKsbYok0MaXo4syOZY0lMy2JbGksjP4wJyshsjxhJnsmQ5wN/p7Twj2gdpQ1akTCsIrOaqzLuDqnjuvpuKvZVE790SkCXFBgdxLhg+mY0LCclvmkbQ6nvFkYrp04V1o3WFvZ2cbCbl6M8HeHigGDn1HBZNRfVdUOip5I9l+alHO/kei/fR60D2kRo7q9m/uiciK4p0S87/TPpkQa90mFA9oYXbkQRpoQ2IYSZEUwLa0Z4cyNbHsf8qOZFsaBYi6InKaX0MsquoNziSiq/zBKKTlZMeeWUWlpZJWdFaFoaAJoAwP4APAXVcdDdAc0xEAEAeGxKdHoB5I2LPoVCPGSItmINIoFu4k4NtDQSVlYgewpAQwoKpsYAupLhaRHy66GQ60QVDLVZYYuH2lrmQoVfzF6aRXGRMOH3FEfANPHELEcs42gkBXWzTCzdYVjGQFpABOYSTqajbUhzq1Rz4XlV4heqoIBoXaBaRUpP1GajrMqW1DNNJh8T35+IXrVWVPmpPnqlrIlUF+bb6/nKEhCwt1j41aBUyarLWSkCy0oLYtW7n0B5Cqtd+NkWX71ZvVS9WFRqoqJQl7RwhQ8X4JWoFfw0sy2nMvLURzK1boIPi5oAqWfzDQ15la+89Dcq56FKaj813bLDdnDUOBlumWrWWjMX6Uw1bEgX/egVF5kaqoiOIkWkiWoizsaG0pqgxcAxPgBSWLHBfEfKBJQTOQNSXNt+tSHDtZCBEQcoL5pjVClSY00dlEGG06CQ4nx5c5Eajt/0qwSLZeGLK1RLQA33o1ZWQR3/GYKpxicj1m9he9wEOoSSMAqavC2ahnhKBUb6IwffRO6FFvpn8OR92zVHRd1sbYcO3ylVQ9KH09Sd7Cjeq9ItbFpPHZ7bqbxDQkASu0UOyeztkK5tau/RuydzwL79qjWpOfArIt0T275rWqjNzbM3PBG8I85zwFnCulX1P6QfRVrfRHHqLine truncated
@font-face{font-family:'Plus Jakarta Sans';font-style:normal;font-weight:800;font-display:block;src:url(data:font/woff2;base64,d09GMgABAAAAAC8MABAAAAAAdKgAAC6oAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGoEUG6sAHIgyBmA/U1RBVEQAhTwRCAqBjHzuBAuEQAABNgIkA4h8BCAFhXYHijUMBxvZYQXs2DSwcQCzB31hFMHGAY6NX4mikpM3+/+aoIaM/aMdwDatRAhRVpV6NDPjq9Rrs1VN99mM5ukR9SbBXPmWpr0+LlWOYxyl0tBGaQQYHJQgNmkofesntGz/PePDZZwA4+7S4aY3WHc84QhpTCzP//9+0Pa574+74d6oQyWaJgvFGl51dSsBIvOOyM3/2bujHgIiIFHAAxRBscTYClLsqFETa0wRW8doSiNG09o3ryam9G56N8+892N6E1W2rFxZD89/f9DOve/PiqzCytLqhk0TJJB4EMfW5M0bnrbVv98bbWGtpIgMIDXEMFQM0dJGga7gKdb+NcdvLe62bLZb6WZcRXv/rhscYJJl+t8CTm2TzV9IkxIxiQ9xjfuOiOYwO1xbK6RMSsT4og5+Q33iRzrzZyTrYJ42yecQc7Vga33E1UdwK7VBhIP2W3quYM9e4EmRy0tq1R4Mmk/xYVRb8ny4vC4G0nnlogEBFATOv011/V/n6zvFIcUFqSi9VC9KCXY7PBVoQnz71/9n6/59XSwppNMZlYKkkBQVLLskiHtWSAlXUwF4kotBUAF4BIK3d1g7dRzbZexKwzyw/9/SV+qra3ckCdlOK9gngLjyTUV49t0d7Z99f9P008umVPSNNNKqpBYaQoMaaiwMVBaAjUgQC6bVRObiLozpqnxeS4RYo/6/v+b/3YlDl5q77ONxqaUUYwgxxChHjBTq0H4N0x0JcBqYav7YTkTji0KRkMAgo4wDgoCSBsNGh+oylDIJ/INyMtA8Zrx4QXTiIAkGQ3JYISVKIGUqIFVabD79WDBEAzwMxAQ8HCSCQ8BYanrY8ZMpWcB94VFbDXARgP7DgMCef7KhBoTb2hNwgTCIRtTQia753R5MaBjb7sMGPo76CM8qbu2+dObYwd1R2daNu7x6LB7vGVlnkr9YEVuYk5GSW8bpwwN9sds0Sk0qxifNZ7MNCOxXX3rXi/7D/Lq72/Uu1tWxDrQrOvK2tK4VLb6135rfzH7V1sRIGi1ne9ay4VtdNVVWI6qgBlc6qLNlquhIDa2AMtUBIaQuqlyihO02YXHLmNgvCYtva0jJj33am1RokWd7socvPKm/vatofn6nID/AWKjSXBrWHzgan1Kya6QCSgchR+3tYmJ+S+YD2SgOiLkOoytAGHg8zr4Cyx3UbBCHvfxhChSpF41J60hMvOBXZB6AOwc2xf1GEMu25zM8LqmcuzN63i8FTGqme3rKQYNBB4swJkYrrazwCkYhhRzIaKYylWaauZu72chGdrGLPts5zHJHXHFgmFtSWphfQpxqqhnFKDazmSu4grnM5Vu+ZQ1qHsAwOxwCNdTSxEwO4/CwH/LBnAiP4L7TXd+4qFLanYYY6OedO9ZE7h0PMg87dyQBiHkAg3dtBBJO9/gCg43bhQ0bvQEBxu4dwwY1mflH90j752v85U7icpdnVm+kMT3P2fiTdtpYnC70zaJP+jl7q513Gtb+VIllZfNefIOZmDjElpogQq/5LX8ZvLPpns73umbv0f6mrRpkZQ28vDtZ7Nl8PZtLkpXRek73rb6d+H5tbH2H31rqzzcYoP3QX7442P0KS/d4E2Qn677t6bPQ69+lmTASe0/Ch2lH4LvHm7ZrqLlN3Z1h11pRUrewB+xgNxodyULkMV56/o7HsvV3mHyif3tUs7hHWSv2OMKlzOmIM9MVqeBWIHNXWmnKtB6vvTtYdXundXS1S2tXXfECV/2ekTYWNkikfbGnRr3ozUjf2O9fHpCExrPVaH4g+nR5y1HbEyLny4aw3KyfvNxoVcr/XwsPEgjsFCxQKLMg8SyM0uRKkq/UIOWqjNCgUYkmo5UZa6waN91R66FujXr0GOmFV5q99dZomAAICwOJgyAiQpCSklDSIujoEHz54vLjx5G/QGzBgjkJFYYvggHGyIhkkgKxsBBIlQpJk4aULgNTpiwOBhmMKVs2ZzlyMOXKRcqThyXfEDSFiiElStCUKkUqUwYpV46mQgWeKlU4qtXDNWgg1aiRSJNmdC1aiI0yjsA++wkdcojQdTcI3XSTi1tuc3XHHS4eeqSfbt0IPXrwvPAS8sorXG+9xYeJgjAxsLAwsbG5IXEIOeAi8DhhEhJSEBEhiblCpKSkZGRcyMmJuVFAlDQIWlpMXrwpyU4+V3egnI4ni+h6n5NwPNKPgXjQOjNn8TiRJkEKjMUp+omTQz44geS5jHLgyQoUA0OpkU3WfQKVV3fjzmWknyyHJHBIGT3kGOscmulOD7qfrIIgYOjspdM4Hf84J8gyBHD+PMHw5nV1x+7A7dIM2BIQvK9adZ1vcEuvhqpiya0ShDUimD1ZNSSt2KPDzT1c4hXv8+s2c223V9YlKaZSnriuRes9v6oqKNgfxuCv6FpIKBB+ipCInJKKOw9qnjTN++06Pnz5CRIspBl/PwZG8RIkSpIshUWhoYYZboQiVsVKlKpQyWaUSezatJtmn/0OOeKYiy676robuj3W46VX3kL8tkOyWbH4LrJd0FbB6fLQLcK6bHFe+ektE22BQlGD4O13I5N2Y2jd0La0z6T9wCFcx43ElKGZEoowAACAwA6ZttGxFW1DmYgUCUQhGjHQIxaGy01LZsQhHglIRBKSkYJUFGIohmE4RqAIVhSjDC/w8qsIR8ldEudK6QTXaCA2Y7tVCLL/g0OIoIQK7vCAGp7QQAvd0v7BgtQuk1UWlvYVFbur1lVvqAFqUYf61IDU6KcmYiSa06h9Gg2MwViMS92y0bPb2r3dIQPqt2ECHARooIMBJliJjCYOIIISKrjDA2p4QgMtdJ0voBn5JUUCUYhGDPSIheFyU58xIw7xSEAikpCMlIomGGoCLs/vcwUYkgoFQzEMwzECRbCmYqSSMspSuaKiq+oLcUk1glrUod6oSzagZT5lCm4AgMDU7aEHL/DygVcAqAzLixgQAoSPi2IXhjqFTTvFZU70G9eoM6ApIZQqvvba+RjipxasA6YqT0cEKUaVZzUFGl2WlGRVsKvFbrGMz7BbfVkqmSnLZSav1XLAm0U6JDLKz8Lm5emQyZc4qieAqJhbJx9/tMoGs59F1XXQGl95y0tNbTU0/CKweCh7LS8dDh+JU1YFBQmD35qqGyD9vQlA0geYyR7jgnPgGByNIyDP3VyRLGSKUg2Hj0lK8NjxCdoWHX3Z1R2afbYUJDXhKfDOE+cdMpqeGrLyTGDYMB99gGBszqIDRYkNh0JBBECAKRDkrKzXzKANYYQFC06WhkZNTkKAYxxMnrGtKd0om8UYuJZnNFi6ud0CgdlXYGfC9P4JxgtPtwDIYLTobYnaSMD7svH6poX+xnIorSgwLbYapu0aR6w7X77rDEJCpYUcdsaJpvnCxHKSqYwgN4wch9FGh8hFwSDL5fF5SdYZTVqfDsvjZiqTELCxdGLyFGxUpaEf7z4RU6dKgUBoYLIQ5RBjCMR25S0DWTw9GMIAvWYIEKIZUPhmKEIzFFEpDKabkGwF2cD5+myEJ1oePNgeCAdOiIsZASwM5AgLhpxgYUohr4wfh4pcjgbxMgYiVAQPrBgH8RGV2Cg0UK+XA5xLrjwP0STBfPt3wdPA8x6z67fkNDjCYekMgtunBCMmx7oY0JuAWlq9wGI4RKIGpsdgeYtXny6zIRzgvc1aIBIQIULLhoUDoCQaIWNxKgcLNjZdBycmotZoJ+k7n1eiAz9hxtugL4X6J7NOdbf7D+EEzsJVizFqDDWOmk65UHJKSampAVQ4ZaJs1AYSWTGPtIG8RCnq+4EJKP7Cddpkg8/oAXjDmd2nURJKumsBVJhuw6tYD1FPnwL37mLpMQD4/6+7U8jNAv75IhnWuU+6FexHdx/VPko8fP/hA0CAx9d7xNN0zic98t7+cCnzxjrMn6kvHfbQaa/0HY89ctElp7y331n7dDngoK8+++KobggbyQHvnHPxYs6kZOTcUJS8RtX+eKbCXXXGNT9diEQEI5M4CSxSpUmXLVeefAUKlShTrkKV6itMaJtzi8ueueK7nY544Q2drfScr/s4e2K8u374NzL/981W29LPL/84HrktJrhnnbXWO4YOQ2CiYWDhcMLnSMCVRD8uuBTUVNxpePjIU6D+AgwQxlulGJGixIqmZ2CWcg6NwDKudNR4Q4jPpVPJ2uHTn3S9GrXqjFSqmVZxOIgfDAnddsd1N91yA4J6Hx4giwB1EOQKmPYAmHsFaHdBPQAwwPpcz6MbKIkSR7fzs5VWzR03cczsbWDaFXJnN47DtrEo86uvS+Ia9RLRYovrbSLdFoveSzz/BuRu1rtsLqmNLc79jYvFrNaaFYri7uwjKhAjPjEGPsfBbTSdk/F852KSkzlBOhcLF8Ek+INgDEs5jiRIksPWjiccuHT+eOF0ootRCB1IHvmcM3+c72jxaP5ENtvnAIclmkwh5M4nNHM6v4QgCqgLTyK8ZvNVnezLH/FtmKfbKJV87nce329UiDEincwoumwZSVOfBIYcrWfkcNhZK59Dd10Ucp/JYUh9xJQskaBLF5M0giRueEQVjUQEaiZJB7ZFLtdjBZNJcnRnErHguEn68eqVQqHckc9lMfmTOUUj1OA+2be8WTywCeSIYw+TqRHJSKbLsLEeJnGtSEiMb7tyJLOpBDyRIBzEDyMe2MZ8TcGRGTWoEhtrjwQ50ptiUmjUMUt+XqEgUhnpIngtv/wmhcQi+3PpJ8INALDx16X00pVLjEJvK9P0JSop7kTZARmbLine truncated
@font-face{font-family:'Plus Jakarta Sans';font-style:normal;font-weight:800;font-display:block;src:url(data:font/woff2;base64,d09GMgABAAAAACpoABAAAAAAdHgAACoHAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGoIkG6goHIR+BmA/U1RBVEQAiBQRCAr7IOc/C4V0AAE2AiQDiyAEIAWFdgeaLQwHG4xmNWyb9g27HQRynGrTjUTkcTAHjUVRNfmQ/f9nJCdjIMwHc6pZ/Yci22G6Z+UouKbHROXwqqquuaujCnffGKMOSoLxwMGN6QeD6nL3XXTUkvySgr3o5C73PAlD+kwFZ5KiooXf6IVx8F4ww/cHfJ1Of6xvyaJpK5IkTkiyyGSITGpVKDVONacauviLFgY94f7feGkin7BLYWHRbqiGGR6hsU9yiQfjqq+Xnk16Zj9gnXiieRaZI3Lzf/ZARQ4bKraODWxYUMSCFVvAAogQFQsKNsSWqsEkmmii8RlTTGumm9Lqy0955r2Y0n7IK4lJTDURKjhoc7MXBSE+Q5KEEQf4BBVht/zz/cXOfX9XMIgTDJummWYFE4w8zjiDIIDL/Zj4b/N+7bJtY1G6gUP+IogDzKihNfer+4iWSCVbCWvZ1U4Q1wbxCEk/jC1jxdbfq2ouolExinuEVfjx+/WbefKBOXu/NMSlieYr61IxTSuPaJ7wRChuK5UCKLDQzcolQb7wiPnI1GkGArrWAQ7w3if5gT+dymdjpjYtrX1EF0BA5s/z9/u9gPfN1eJNhy0ojGeSv5W/8fhzeHMJqCCAsA2UUXbu/9ksc9g1mu/teV75jYwyAAUZEYbU/atL1d2lL+rpBbWGJM8CoKSR1q1FrQngUEdIqaMDjn0bOXQQ3buXXx5fkJ//39ynzeT+LEyWsvR2SxRjl/UrSyBXV5WX/DmnySsgCgs0vzgysiB7fK0qPM/PvdrenBK9liWBQt9fV+navPJNCpwiql8cIBkHwhHZTREoEmZqQk1ZP+FmNpbNJvbHGZHS9A5TjJpyz2tODO1Ja/alkD7BFCOCcTVXzTxPCyb99n/H2r43NptX6iKMiuRg9OGy8eP5BJjM9L9+wYSQIGRkCIUNxJ4HhM4fwhQECcGBcHEhsfiQdLkQsWJIiXJIhRpInemQmeZA2i2C9OiD9BuErLYOssFWyJAhyA4HIYedgZx3BXLNTchtf0Ae+hPyyCPIY68hb7yHfPQTMhkMUIiIDCgUZA3FJk5QXOILxT8BUIISDoUdDhRBBFBEyUeILHKEKFOLEG20CGlKEwIEjKGLg62xViQDTA/YlbvAFBHQO4HA9t+TucDqZe8B00WAAI0vY4ZgqOSerkGNPhiWI4wWoYRCyh1mDVPnq5Y3jAbDpYZBJ3a+oeclMdPQdki9CBSSIQDWOCiyhGeO0d/tE3b4vr0BxY/bIyn/gcmK9XOZ7MV2LXuundpzD7V9h3Fb29LWucwnwJGvbP1g75zROnfvmDZz/0NbXVOhdKVUh5raJGDPbdYydr5WizuS5VpoCzi+Ps0dtBkabdcmaZ3V9A+MPtHIRkQcE8Nf9SvY+c/1n6P0eX1a/7wD6qv768/XKxCL+hxW7l72WD2U3VW377nr6j+Hc0VdXvu0O34C1PPr+lDsk9EbatsR1yNrA2k1rDVAxUrVHErRToE9sUMV7XSTmrZGTAwU95nio3Q2zOKOK2Mf+M4UCoBNxIyjs+ZcsFj/kY+T07Q0ySSpmoHxO6CIAw4XGBj9uXjhqu4PJANd3YVOc2lRqqTgRKLWJtEmxibkaoSD1NUEMAtHzwpksZEJ9YR6L3BURzsVsCOIL8SEQgy6kcFsTzVbLVZqxQiJIYss9LbVyKgWaLnCFac6we/KZE8yu+xxTiE6vQl6nR0D6WNbxl2PC7ugWZsjm9jEmDOWUGZ5NdRE/3phfCEkYAwZCYUFY5ZozNjzYInOnxOmIB5CcHjj4goWiy9EugxcWXLFEiuWrEQ5oQo18tSZRmamOcq1W6RKjz7N+g2aYbV12m2wVYchQxbZ4aDFDjtjhfOu2Oiam4bc9oddHvrTIY88csRjrx31xnvnfPTTdZPB4EFERIY/oyBrz7KJk79zia8P8U+AnwlKeBBihxMSWACHjLKf7FHfTqdsk7PxTTeDZMAkQwkA4gaQYJg7FP/ZyYl3PSEhEjuGMXgNEoCIIWaL30LMYUgQEqwyEMFwjRbDnY3pZOy/OwbWcv2vfCP944UZV+QfWxmlkKMsq3tl9a06+ZzlIvmx2TT9kT7oiKgHsEnd01GbQmX1K3UFt3GsewOkbfXCWr2W3i5ZP1btdv2nVlCwxZb6RGG9zDeZp+rRDd5R6oY+UM/n8DGXnNKa9gr7TRtnuUb9W8U7rK3HOXyr1hE1oK5mKzNIld1KuhrdnlVd9N9746AYexzZe7ubbvkYGGz9pyQ7dWjj+rEb8o//ucUW2yZ9mD/Hs29O8o2T7fWTPtlGrQOHybPvbeGcNqnC9cn6ctqtBo/cFd0fvx8GgCTG6Z6scTV0LPxgSWh6nljSyCOfItRomEkbnXRL38oevxmOZrmDG1c47KyVd92yOe+BISMyvDLqgKcwh7zceNgbY47cgXOOm9xhRJ+To4/37xfxKbHh+ZrEJJlIRjL8iiBCk8lJXhBg/N3ASIzyzBF5QgyIOKw0H5TWOkQlNo0pFpChqnm7YN3Y+55Czwp+v9tIfnePifskpLER1q+7QApwo1ENXoNR49YxAIEwwEwpgKCWDmAoVs+LZZT48BCTlCREWs5lUR7iUOsmQAqsWERDRFW71/ICIhuo7et6p5L8UbxjBI1FQV7hGDiKDYG9uGho4is74klGk7JiJhW/hKfLhuTgbmZy5UEO2w7DBAusIeOApW/ns9tb2OAqtitnYM4SBkrwHArmSuk2zHDCVczC/0Pwzrti3htznccAA4yhqinXdY6yXYtLHBdstCoxidz8V7wXBbULsCFhhEw2BGvf4X2EmSoro3IfGIO06nZoGtKuN5nZSZnbTSn1Ho/khj5PUT0y5kiF296nqOV8aqD3NMpsHOy+u9ME9Hmj/esp28BqHDMEmvwwYNASf4ZRWtxyzBprhDCHgCDBmCJq0ho19hCiBrMzjx0s+70QW8amksiWIVmcKKECIMaobMRLkSnHQS+9g9EbgXk8m71FpmcYEY7GEXGjFkXiEmvWsaOYPg25D3O7JigthoKmRhhWvnGlAZ2kLepSe8KnZQSbQpAXPhJnH+RDi5SNRhFFBL0mH4EaYw4UdQ6U2n7otXowtzz9PqLwsfmW+0be32vr+9A4dBQEVkyRUg2LLFWwiFIDizwpqnR1KkIryRdxuiySACE4Xk0ZNWLhD1wZgHJKkQDrhztnJz6lmmpfBXYGxnyzLR7hODgj/mypmE4KMG6SlM4C6FxAma3ogVkQIL/HwBRzxBx7Hba3zlsIAN9eaq8WHBArRFUeFgqAn3FNxgiuCyky0Ck0bjDEEfg9DxDHAEM84bNXBgrP3lPaMrz/8q2ADjJlGWF+EIFIMCakBDxmMp0ndp6agJicljFTlhzJoYBKapnBvETJUmXsuxYN7HyCErJKm/v/sVJFQIslLbkvW5c9uZDXeZvvxa8EHeQBYUAgfRnDeQLnKfCJyGgYMWHBigykTKWKFtoSJj3g5UE2TgERafmD1P5UW6aS/89vWsty7ZrTT3Z+/ZtAf/Dx+9kj8NHpwyEx9+EFov+W7uOKT8EAyGbJjSjlqYgqO7MPQO7JAZD7cignci4XcilXci03ciu/53H+yX+FiJEx82JfHAFlOkCZo9BLUAkpUSUGoNqMYAMjW1DsYGY3c3uY2MnSMCv7uDrJ3mHOjvNwhoMjfFxEd56fy3xdEugGpmuiPMB2D8vvIt0XLsQfuP4U65FRPE8keibBU0me43sty1sC/5jib3nGTPWd2EcyX1VEBFWGUJkB1EWBxiw0ZK4+U5idnVnZmpmN7nx05qk9F115m5+7hdH1xtBfkKVWx7a2KGuKVOCLebmhsnFQvBQvZcrxjrcLiv1EshXZkBKT1kYGNnFyjIsTQt0R4S6JT+S+UWcE1RnD3Fx15KEpqtbs5RtXHgZSn60sLMqoI0otU7tY24/mAHenuTnF0VH+rghyU4DrwvxPshdSvZLuDaF/ZftPrvdKA9gzBEUmFPqhKhLURIbmLLVkRZMJaDODaVlrK7mjOZ0s6mtBL30D9PS3JNPAYOtyLM+ybIgVQw0Ofz84MjoxO/Wuf2YsetX3ZnpyaHh8ILQ4COp/b4AcASg/A5+A8auAidcCE64ChgBGYJ1c2cAcnjklw+Ep5jCOxBzgJhmlOBUstY2i0qymemqgCoPabKTiCAzHkkvfiDg3VGpz7cBgxy80xJMcqP7hoI4Yri9Ugh+PJTCCZUZkGBFAWAKdVJJYchB6DYOwEEGYIaSRJLNydO4shq9eA6M1TZgmXQuB8HMtSRKSKfX93sf5s2Hz1KCFPRvG8Se9Dhj7Oea55xgxaGcMthV0QmaNjSxX1efG9p4jjweOaKReA1xdBzltNBv3BhhYEEU5BVbhUze8atc8ycBJ0+awLA3apQ8+aSoGphGIY8XrEJT09ZkwymOf9ovvwW3Z+9goyIiN6RSSKOMmiAIm6M+NsdRjfCn00NDJl/FHQQQHm3L3IemBPuGdIcJb8vbmECYzJOhiDc+fmi+joUiKFZWSjlCsQGC8kMnGjKCIr6ovs+/faPHGObySwlrR/ivlBsvZf044jWUTY7WJ1/N+POUjED/mZZstl6Y9dQBGNOMZcyO+eH1W/AhWxd4rJhzyZ30/25V42VUkqenCn0iRcmERzSnhXQWQL+VxYlQiyTlmDZddo7kkPFCW6qFAxxr7Q7tZPmah56QhB9lVQsg0dxCp2/SthMZLjcjmCrYLe8Sb4ofhgeZNFJFrB5pjtMc1BgBE9pwseSZwLine truncated
html,body{margin:0;background:#eef2f7}
a{color:#0071b3}a:hover{color:#051343}
code{font-family: ui-monospace, "SFMono-Regular", Menlo, monospace; font-size: 15px}
@page{size:3200px 2262px;margin:0}
</style>
</head>
<body>
<div id="__bundler_placeholder">Loading Felhom system poster...</div>
<div id="__bundler_loading">Unpacking...</div>
<div style="position: relative; width: 3200px; height: 2262px; overflow: hidden; background: #eef2f7; font-family: 'Plus Jakarta Sans', system-ui, sans-serif; color: #051343">
<div style="position: absolute; left: 1912px; top: 52px; width: 1216px; height: 160px; box-sizing: border-box; background: #ffffff; border: 1px solid #d5dbe5; border-radius: 12px;"></div>
<div style="position: absolute; left: 72px; top: 232px; width: 1000px; height: 1108px; box-sizing: border-box; background: #ffffff; border: 1px solid #d5dbe5; border-radius: 16px;"></div>
<div style="position: absolute; left: 1232px; top: 232px; width: 480px; height: 1108px; box-sizing: border-box; background: #ffffff; border: 1px solid #d5dbe5; border-radius: 16px;"></div>
<div style="position: absolute; left: 1912px; top: 232px; width: 640px; height: 1108px; box-sizing: border-box; background: #ffffff; border: 1px solid #d5dbe5; border-radius: 16px;"></div>
<div style="position: absolute; left: 104px; top: 416px; width: 944px; height: 908px; box-sizing: border-box; background: #f6f8fb; border: 3px solid #051343; border-radius: 16px;"></div>
<div style="position: absolute; left: 128px; top: 540px; width: 896px; height: 336px; box-sizing: border-box; background: #ffffff; border: 2px solid #1c7a43; border-radius: 12px;"></div>
<div style="position: absolute; left: 148px; top: 744px; width: 856px; height: 122px; box-sizing: border-box; background: #e3f4ea; border: 1px solid #1c7a43; border-radius: 10px;"></div>
<div style="position: absolute; left: 128px; top: 896px; width: 896px; height: 136px; box-sizing: border-box; background: #fdf0e1; border: 2px solid #a85400; border-radius: 12px;"></div>
<div style="position: absolute; left: 1256px; top: 724px; width: 432px; height: 376px; box-sizing: border-box; background: #eef0f3; border: 2px solid #5b6270; border-radius: 12px; border-top: 10px solid #a85400;"></div>
<div style="position: absolute; left: 1936px; top: 296px; width: 592px; height: 728px; box-sizing: border-box; background: #fdf0e1; border: 3px solid #a85400; border-radius: 14px;"></div>
<div style="position: absolute; left: 72px; top: 1364px; width: 1120px; height: 516px; box-sizing: border-box; background: #ffffff; border: 1px solid #d5dbe5; border-radius: 16px;"></div>
<div style="position: absolute; left: 1232px; top: 1364px; width: 660px; height: 516px; box-sizing: border-box; background: #ffffff; border: 1px solid #d5dbe5; border-radius: 16px;"></div>
<div style="position: absolute; left: 1912px; top: 1364px; width: 640px; height: 516px; box-sizing: border-box; background: #ffffff; border: 1px solid #d5dbe5; border-radius: 16px;"></div>
<div style="position: absolute; left: 72px; top: 1904px; width: 2480px; height: 272px; box-sizing: border-box; background: #ffffff; border: 1px solid #d5dbe5; border-radius: 16px;"></div>
<div style="position: absolute; left: 300px; top: 1976px; width: 1px; height: 184px; background: #d5dbe5"></div>
<div style="position: absolute; left: 540px; top: 1976px; width: 1px; height: 184px; background: #d5dbe5"></div>
<div style="position: absolute; left: 780px; top: 1976px; width: 1px; height: 184px; background: #d5dbe5"></div>
<div style="position: absolute; left: 1020px; top: 1976px; width: 1px; height: 184px; background: #d5dbe5"></div>
<div style="position: absolute; left: 1260px; top: 1976px; width: 1px; height: 184px; background: #d5dbe5"></div>
<div style="position: absolute; left: 1500px; top: 1976px; width: 1px; height: 184px; background: #d5dbe5"></div>
<div style="position: absolute; left: 1740px; top: 1976px; width: 1px; height: 184px; background: #d5dbe5"></div>
<div style="position: absolute; left: 1980px; top: 1976px; width: 1px; height: 184px; background: #d5dbe5"></div>
<div style="position: absolute; left: 2220px; top: 1976px; width: 1px; height: 184px; background: #d5dbe5"></div>
<div style="position: absolute; left: 2460px; top: 1976px; width: 1px; height: 184px; background: #d5dbe5"></div>
<div style="position: absolute; left: 300px; top: 2029px; width: 2160px; height: 3px; background: #a85400; opacity: 0.35"></div>
<div style="position: absolute; left: 300px; top: 2113px; width: 2160px; height: 3px; background: #1c7a43; opacity: 0.35"></div>
<div style="position: absolute; left: 1380px; top: 2105px; width: 960px; height: 18px; box-sizing: border-box; background: #e4f1fb; border: 2px solid #0a3d73; border-radius: 9px"></div>
<div style="position: absolute; left: 2592px; top: 232px; width: 536px; height: 740px; box-sizing: border-box; background: #ffffff; border: 1px solid #d5dbe5; border-radius: 16px;"></div>
<div style="position: absolute; left: 2592px; top: 996px; width: 536px; height: 704px; box-sizing: border-box; background: #ffffff; border: 1px solid #d5dbe5; border-radius: 16px;"></div>
<div style="position: absolute; left: 2592px; top: 1724px; width: 536px; height: 452px; box-sizing: border-box; background: #ffffff; border: 1px solid #d5dbe5; border-radius: 16px;"></div>
<svg width="569" height="8" viewBox="0 0 569 8" preserveAspectRatio="none" style="position: absolute; left: 686px; top: 344px; width: 569px; height: 8px; overflow: visible; fill: none; stroke: #1c7a43; stroke-width: 3; stroke-linecap: round; stroke-linejoin: round;"><defs><marker id="dc-arrow-head-filled-1" orient="auto" markerWidth="5" markerHeight="5" refX="2" refY="2" overflow="visible"><path d="M0 0 L4 2 L0 4 Z" fill="#1c7a43" stroke="none" style="stroke: none; fill: #1c7a43; fill: context-stroke"></path></marker></defs><path d="M 0 4 L 563 4" marker-end="url(#dc-arrow-head-filled-1)"></path></svg>
<svg width="251" height="8" viewBox="0 0 251 8" preserveAspectRatio="none" style="position: absolute; left: 1005px; top: 652px; width: 251px; height: 8px; overflow: visible; fill: none; stroke: #1c7a43; stroke-width: 3; stroke-linecap: round; stroke-linejoin: round;"><defs><marker id="dc-arrow-head-filled-2" orient="auto" markerWidth="5" markerHeight="5" refX="2" refY="2" overflow="visible"><path d="M0 0 L4 2 L0 4 Z" fill="#1c7a43" stroke="none" style="stroke: none; fill: #1c7a43; fill: context-stroke"></path></marker></defs><path d="M 251 4 L 6 4" marker-end="url(#dc-arrow-head-filled-2)"></path></svg>
<svg width="8" height="219" viewBox="0 0 8 219" preserveAspectRatio="none" style="position: absolute; left: 572px; top: 384px; width: 8px; height: 219px; overflow: visible; fill: none; stroke: #1c7a43; stroke-width: 3; stroke-linecap: round; stroke-linejoin: round;"><defs><marker id="dc-arrow-head-filled-3" orient="auto" markerWidth="5" markerHeight="5" refX="2" refY="2" overflow="visible"><path d="M0 0 L4 2 L0 4 Z" fill="#1c7a43" stroke="none" style="stroke: none; fill: #1c7a43; fill: context-stroke"></path></marker></defs><path d="M 4 0 L 4 213" marker-end="url(#dc-arrow-head-filled-3)"></path></svg>
<svg width="8" height="35" viewBox="0 0 8 35" preserveAspectRatio="none" style="position: absolute; left: 572px; top: 708px; width: 8px; height: 35px; overflow: visible; fill: none; stroke: #1c7a43; stroke-width: 3; stroke-linecap: round; stroke-linejoin: round;"><defs><marker id="dc-arrow-head-filled-4" orient="auto" markerWidth="5" markerHeight="5" refX="2" refY="2" overflow="visible"><path d="M0 0 L4 2 L0 4 Z" fill="#1c7a43" stroke="none" style="stroke: none; fill: #1c7a43; fill: context-stroke"></path></marker></defs><path d="M 4 0 L 4 29" marker-end="url(#dc-arrow-head-filled-4)"></path></svg>
<svg width="907" height="8" viewBox="0 0 907 8" preserveAspectRatio="none" style="position: absolute; left: 1048px; top: 690px; width: 907px; height: 8px; overflow: visible; fill: none; stroke: #4a5670; stroke-width: 3; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 0.5 8;"><defs><marker id="dc-arrow-head-filled-5" orient="auto" markerWidth="5" markerHeight="5" refX="2" refY="2" overflow="visible"><path d="M0 0 L4 2 L0 4 Z" fill="#4a5670" stroke="none" style="stroke: none; fill: #4a5670; fill: context-stroke"></path></marker></defs><path d="M 0 4 L 901 4" marker-end="url(#dc-arrow-head-filled-5)"></path></svg>
<svg width="247" height="8" viewBox="0 0 247 8" preserveAspectRatio="none" style="position: absolute; left: 1008px; top: 996px; width: 247px; height: 8px; overflow: visible; fill: none; stroke: #0a3d73; stroke-width: 3; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 12 8;"><defs><marker id="dc-arrow-head-filled-6" orient="auto" markerWidth="5" markerHeight="5" refX="2" refY="2" overflow="visible"><path d="M0 0 L4 2 L0 4 Z" fill="#0a3d73" stroke="none" style="stroke: none; fill: #0a3d73; fill: context-stroke"></path></marker></defs><path d="M 0 4 L 241 4" marker-end="url(#dc-arrow-head-filled-6)"></path></svg>
<svg width="207" height="8" viewBox="0 0 207 8" preserveAspectRatio="none" style="position: absolute; left: 1048px; top: 1168px; width: 207px; height: 8px; overflow: visible; fill: none; stroke: #0b5fa5; stroke-width: 3; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 12 8;"><defs><marker id="dc-arrow-head-filled-7" orient="auto" markerWidth="5" markerHeight="5" refX="2" refY="2" overflow="visible"><path d="M0 0 L4 2 L0 4 Z" fill="#0b5fa5" stroke="none" style="stroke: none; fill: #0b5fa5; fill: context-stroke"></path></marker></defs><path d="M 0 4 L 201 4" marker-end="url(#dc-arrow-head-filled-7)"></path></svg>
<svg width="247" height="8" viewBox="0 0 247 8" preserveAspectRatio="none" style="position: absolute; left: 1689px; top: 808px; width: 247px; height: 8px; overflow: visible; fill: none; stroke: #a85400; stroke-width: 3; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 12 8;"><defs><marker id="dc-arrow-head-filled-8" orient="auto" markerWidth="5" markerHeight="5" refX="2" refY="2" overflow="visible"><path d="M0 0 L4 2 L0 4 Z" fill="#a85400" stroke="none" style="stroke: none; fill: #a85400; fill: context-stroke"></path></marker></defs><path d="M 247 4 L 6 4" marker-end="url(#dc-arrow-head-filled-8)"></path></svg>
<svg width="247" height="8" viewBox="0 0 247 8" preserveAspectRatio="none" style="position: absolute; left: 1689px; top: 896px; width: 247px; height: 8px; overflow: visible; fill: none; stroke: #a85400; stroke-width: 3; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 12 8;"><defs><marker id="dc-arrow-head-filled-9" orient="auto" markerWidth="5" markerHeight="5" refX="2" refY="2" overflow="visible"><path d="M0 0 L4 2 L0 4 Z" fill="#a85400" stroke="none" style="stroke: none; fill: #a85400; fill: context-stroke"></path></marker></defs><path d="M 247 4 L 6 4" marker-end="url(#dc-arrow-head-filled-9)"></path></svg>
<svg width="267" height="8" viewBox="0 0 267 8" preserveAspectRatio="none" style="position: absolute; left: 1688px; top: 986px; width: 267px; height: 8px; overflow: visible; fill: none; stroke: #0a3d73; stroke-width: 3; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 12 8;"><defs><marker id="dc-arrow-head-filled-10" orient="auto" markerWidth="5" markerHeight="5" refX="2" refY="2" overflow="visible"><path d="M0 0 L4 2 L0 4 Z" fill="#0a3d73" stroke="none" style="stroke: none; fill: #0a3d73; fill: context-stroke"></path></marker></defs><path d="M 0 4 L 261 4" marker-end="url(#dc-arrow-head-filled-10)"></path></svg>
<svg width="11" height="8" viewBox="0 0 11 8" preserveAspectRatio="none" style="position: absolute; left: 2120px; top: 1454px; width: 11px; height: 8px; overflow: visible; fill: none; stroke: #4a5670; stroke-width: 3; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 0.5 8;"><defs><marker id="dc-arrow-head-filled-11" orient="auto" markerWidth="5" markerHeight="5" refX="2" refY="2" overflow="visible"><path d="M0 0 L4 2 L0 4 Z" fill="#4a5670" stroke="none" style="stroke: none; fill: #4a5670; fill: context-stroke"></path></marker></defs><path d="M 0 4 L 5 4" marker-end="url(#dc-arrow-head-filled-11)"></path></svg>
<svg width="11" height="8" viewBox="0 0 11 8" preserveAspectRatio="none" style="position: absolute; left: 2320px; top: 1454px; width: 11px; height: 8px; overflow: visible; fill: none; stroke: #4a5670; stroke-width: 3; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 0.5 8;"><defs><marker id="dc-arrow-head-filled-12" orient="auto" markerWidth="5" markerHeight="5" refX="2" refY="2" overflow="visible"><path d="M0 0 L4 2 L0 4 Z" fill="#4a5670" stroke="none" style="stroke: none; fill: #4a5670; fill: context-stroke"></path></marker></defs><path d="M 0 4 L 5 4" marker-end="url(#dc-arrow-head-filled-12)"></path></svg>
<svg width="11" height="8" viewBox="0 0 11 8" preserveAspectRatio="none" style="position: absolute; left: 2120px; top: 1560px; width: 11px; height: 8px; overflow: visible; fill: none; stroke: #4a5670; stroke-width: 3; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 0.5 8;"><defs><marker id="dc-arrow-head-filled-13" orient="auto" markerWidth="5" markerHeight="5" refX="2" refY="2" overflow="visible"><path d="M0 0 L4 2 L0 4 Z" fill="#4a5670" stroke="none" style="stroke: none; fill: #4a5670; fill: context-stroke"></path></marker></defs><path d="M 0 4 L 5 4" marker-end="url(#dc-arrow-head-filled-13)"></path></svg>
<svg width="11" height="8" viewBox="0 0 11 8" preserveAspectRatio="none" style="position: absolute; left: 2320px; top: 1560px; width: 11px; height: 8px; overflow: visible; fill: none; stroke: #4a5670; stroke-width: 3; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 0.5 8;"><defs><marker id="dc-arrow-head-filled-14" orient="auto" markerWidth="5" markerHeight="5" refX="2" refY="2" overflow="visible"><path d="M0 0 L4 2 L0 4 Z" fill="#4a5670" stroke="none" style="stroke: none; fill: #4a5670; fill: context-stroke"></path></marker></defs><path d="M 0 4 L 5 4" marker-end="url(#dc-arrow-head-filled-14)"></path></svg>
<img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAoUAAAGYCAYAAAA0iZgMAAAACXBIWXMAAA7DAAAOwwHHb6hkAAAAGXRFWHRTb2Z0d2FyZQB3d3cuaW5rc2NhcGUub3Jnm+48GgAAIABJREFUeJzs3XmcjeX7B/DP9Sxnnf3MMGNNlkqSbCFhNiSJkiKl+iJLdiLabJUl1A9JSmkRLdpk36LsW0J2kt0MZp+zPPfvj6G0MctzznOW6/16fV/fjHOu+5oxM+c693LdJIQAY4wxxhgLbZLRCTDGGGOMMeNxUcgYY4wxxrgoZIwxxhhjXBQyxhhjjDEAitEJMMaCF5VvZDXnm8t6ICdIgsprmognoAwIpQUhlgRiAEQKIAKAiYisBc8UBAEzrn7jKuCEBA9AAkK4BZBLoExAZAqBTBDSCeJ3QdJvJMRZTRNnJSFOujTTcXFpcboRnz9jjAUS4tPHjLGSoKi7o1VFrQZQNYK4WSOqRYJuAiEBGkyCkEsCHkEkEYQVgOrVhARcgpBPgBuABIIZggCI00LgGAF7QbRTkPjVTZ694uyq017NhzHGAgQXhYyxQiEiMsc1qyw8ci0Q1RUCjQWhOpGwQSBPQEgEssNft6UIAPTHH3IhKB8SLABpENphCGk9oK2HJra5LordQqxyG5gtY4z5HBeFjLF/RfEt7IrbfScENSVCSwA1BcENAY0AOwDZ6Bz1JoAsABoJWEA4BIgVBKx0KsqP4vSSs0bnxxhj3sRFIWMMAECVEi1KhtSEBFpDohYQohKIciBgA3l5yddvkQYhsiCRWQiRToSVpNE3TpdYITKWpxmdHWOM6YmLQsZCmDk29RYhtBaQ6EEIqgchckEIRxDOAupFgHIIQgHEcQh8JYT0lftC9Hoh5nuMzo0xxkqCi0LGQggRkepIqgeBjoLoERDCSUAGYDE6twDlhkAOSBCIFpKgT51m6zJx4pscoxNjjLGi4qKQsSBHRKREp9xFEh4VEB0AqARYwS2p9CYAZAIwC8IySdAsZ7S6WBz4Pt/oxBhjrDC4KGQsSFlKJVYRmvykRuhKQlgBsoGXhX2loEAkyID4WkB6x31++RrBv3AZY36Mi0LGggjFNQ5XYekEgd4AqoBIghBmo/MKbaQBIlsAORCYZpKkd7PPLT1pdFaMMfZ3XBQyFgTMjpTqAhgECY+IgpYxYUbnxP5VHgiAEBuFoNfc6SuW8OwhY8xfcFHIWIAiSlRMMXJbQRgqgOpUcFNIiLaOCUhZAsgkIY1zSTnviXPrMo1OiDEW2rgoZCzAUNk2NjUvr5uQxAgSwgIg3OicWHEJCKJsAggC7yvC82pO2qrfjc6KMRaauChkLEBQRIpDUdGXCP0FBBGIi8GgQk6Q0AB8JUF7Oe/cyn1GZ8QYCy1cFDLm5ygiMVZVlBcha10BAgSsRufEvEjADRIuQdIqSfM8m5+2crfRKTHGQgMXhYz5KYpJjTRJ2jBB1AcCEsDFYIjxAHAJYIksSUPyzi49YHRCjLHgxkUhY36GyraxKfk5/QliGCDJgLAZnRMzlAcQLpC0QBHakJzzK04YnRBjLDhxUciYnyi4gi7xMQFpEggWErAbnZPeVFWB1WqCRBI8Hg1utwf5TidMqgJ7mA3hditURYHZrEJVFaiKDJPZBJNJQXZ2HjSt4Dkulxu5eU5kZeUiJzcXublOCAiYTSoURYFEgNPtRn6+Cx6PZvSnrRcnCG4SeMOpyGPF6SXZRifEGAsuXBQy5gdUR3IjInoHQAUEeI9Bq8UMs1lFfp4TIKBsmTiUL1caVSoloErlciiXEIeyZWJRKi4aUZFhiI4KhyxLJR433+lCenoGTp1Jw6kz6ThzJh0nTp3H/oPH8ev+3/DbibPIzMiB1WoGSUB2dl5AFowClAMgnwQGu9Ibvy/ES4H3STDG/BIXhYwZyOZILOciZRogUi7fR0xG51RoBITZrYAA3G43bq5WEXfcXg131KyCm6qWx81VKyAh3mF0ln+R73ThyNFT2L33CH7efQgbtvyK3XuP4MLFTNisZuTm5sPpchudZuEQsiBwDCQed55bsc3odBhjgY+LQsYMQDRSUmPW9QWJMQBMCICm04osw263IDfXico3lsHdDW9Dw/q3ou4dN6Nq5XKQpMCpZ/8uIzMHW3fsw/pNe7Bq7Xbs2HkALo8HgEBOTr7R6V2DEADlQuATl+YZIi6sumh0RoyxwMVFIWM+ZopOuh2y9AmAioD/7hskAsLD7MjLy0eVyuXQKvVOJDWtjUZ31oDNGvzXKR88fAKr1+3Ad4vXY936n6FpAq7L+xT9UJ4A8iSBHvlpy+cZnQxjLDBxUciYj1ClRIuaIY8FoQdAFkCUfCOdzlRFgcmsQJYktGreAB3aNUOzu+8oWCYOcb/sOYLvlmzA51+vwv4Dx2GxmHHxUpbRaf0N5Qho69yS1kWcXXXa6GwYY4GFi0LGfMAUl1wbgr4UhDgS8KsWMyZVhqKqcMREoGP7JLS9927UqVUNRIG7HOxt59MuYcmKTZjz6TKs37gLiqogKyvX6LSucALIB+Q+zvNLPjA6GcZY4OCikDEvIkpUlFj5RRJicMHsoH8cJJFkGXarCSazCY91SMVjjzTHbbfeaHRaASkjMwffLvoRsz9chE3bfoUkScjJzTM6LQDIFoS17nx0FhnL04xOhjHm/7goZMxLLHEp1TRBXwCiEvxk76DdboGmCTxwXxN0ffxeNLrzVp4R1NH5tEv45LPlmPrOApw/n4G8/Hy43R7jEhLkFIRsAB1c55ctNy4Rxlgg4KKQMS8wxyV1EkKaCZDV6L2DsizDYlFRJj4W/Xu1xyMPJiE8zK9WsIPSlu37MOPdb/DZ16tBBENPMQsghzR6z3XhwkAhtvjlSRnGmPG4KGRMR5cPk7wFwkMQsBu5WGw2myARcF+ruzCg10OoU6uaccmEsEsZ2Xjvw+8xaepnyMnNRaZBew8FkEOEg4omt85JW3LckCQYY36Ni0LGdGIplVhF0+TvBKGckVfU2ewWkAC6Pn4vBvZ5GAmlY4xKhV1F0wQWLduAUa/NwYHDJ5CTW3Btn4+5AcoRAh1cacuW+Hpwxph/46KQMR0opZJaSZo07/JysWxEDna7FWaTgmEDOqFrl9bcRsaP/bjhF7w49j1s3bkfeXlOnxeHApQjQUxxpq14XvCLAGPsMi4KGSshtVTKcGgYQTCm1YzVZoZJUfDi0C7o/uR9MJv8/nIUdtnPvxzCsJEzsX7THuTm5EHz5e9jQrYA1rplub04vSTbdwMzxvwVF4WMFRNVSrQomdLHBGoBA04XW60FxeCwgY+iV7f7YbUE/y0jwWrztl8xaMR07Np9GNk5PmxnQ8iFoBOyR0vJvbDimO8GZoz5Iy4KGSsGimuZoAr3cgFRiUA+XadVVQWqIqN3t7Z4blBnXiYOIqvX7cAzg9/AyVPnkJXts+LQI4BMCLRypS1f76tBGWP+h4tCxorIEpd0kwfSGtIQA4LP1mqJAIvFjKQmtfHmuD4oX66Ur4ZmPqRpAnPmLsbQl2bC6XIh22fFocgl4NH88ysW+GhAxpif4aKQsSIwOVIbgLAEEGEAfNZ/MMxuQbkycZj55hA0qFfdV8MyA2Vm5WD0+DmY+d63yHO6fHIYRQA5BDznPL/8Ta8PxhjzO1wUMlZIlrikNpqQPoEP9w8qigyzScXLzz2JZ7q3gywb2gebGeDX/b+hy9Ov4sDh475aUs6GEDOcaSsG+2Iwxpj/4KKQsUIwxTZ/AtCmwWcnjAXsdisa1K2OWf83BGXLxPlmWOaXhBCY/dEiDHn+LeQ73XC6vH4pSTYEPnWlr+jGLWsYCx1cFDJ2HSZHandBYrKvWs6YTSosFhPeeXMI2rZu7IshWYA4c+4CnujxKjZs3uOLU8rZILHQdd7RSYj5Bl7gzBjzFS4KGbsGU2xqH0HiNRK+KQjtNgvq162OD2cOR6nYKF8MyQLQnLlLMOC5acjLd8LlcntvIBK5AtIq9/kLbfnOZMaCHxeFjP0HNS55MASN9MUMoSxLsJhNmDC6J7p2udfbw7Eg8PuJc3jkqVHYvfeIV2cNBSgHhFXu8+62QqzyYgXKGDMaF4WM/QuTI6UfCGPhg0MlVpsZN5SPx1efjEGligneHo4FEU0TGD1hDiZPnY+c3HzvDUTIFkIscac5OvBSMmPBi4tCxv7GFNv8CUHaNK8vGQvAZrfg4XaJeHNCX76ejhXbug278NDjLyErKxf5Tq+t8maD8J3r/N2dhHjJt5c1M8Z8gotCxq5ijk16QIP0obeXjCWJYLWYMWPKQDz8QJI3h2IhIi09A+0efR67dh9Cdo7XZg2zIfCxM235094agDFmHC4KGbtMdaSkEuEreLkgNJtNiIuNxKLPx+OmquW9ORQLMR6PhkEjpuP9TxYjx3v7DLMFiQmucytGemsAxpgxuChkDIApLrk2BP0ACDtAXhvHbrWg5m2V8fXcsYiKDPPaOCy0zf18BXoNmIzsXO8UhgLIIUH9nWnL3vHKAIwxQ3BRyEKezdGivIu0HQQRDS9WhHabBQ/e3xQzJg+EosjeGoYxAMCW7ftwT/uhyMrKgcej/xZAAeQIog7uc8sW6h6cMWYILgpZSCNHqwhVcm6DhoogKN4ax2o1Y9Twp9Cv54PeGoKxfzh67DSS7x+Is2cveOsAShZpuCs/ffnP3gjOGPMtLgpZyCJKVNRYeRUE6gKweGscm9WM96YNxQNtmnhrCMb+04WLmWjVfij2/HoMuXm6H0ARguicWxI1xZnlZ/QOzhjzLcnoBBgzihojTYeg2vBSQUgEhNksWPDJGC4ImWGio8KxauEUNLyzBmxWs97hiTQtWvVgGVVK9NobK8aYb3BRyEKSKbb5YyDqBAivnDSWJEJEuB3LvpmExLvv8MYQjBWaxWzCt5+ORWpSXdhtOtduRKoAqqhZ8mx9AzPGfI2Xj1nIMZVqXguatg5euq1Line truncated
<div style="position: absolute; left: 304px; top: 70px; width: 1500px; font-size: 56px; line-height: 64px; font-weight: 800; color: #051343; text-align: left;">Felhom — the whole system on one page</div>
<div style="position: absolute; left: 306px; top: 146px; width: 900px; font-size: 26px; line-height: 32px; font-weight: 600; color: #4a5670; text-align: left;">State of 21 October 2026</div>
<div style="position: absolute; left: 1936px; top: 64px; width: 200px; font-size: 18px; line-height: 22px; font-weight: 800; color: #051343; text-align: left;">Legend</div>
<div style="position: absolute; left: 1936px; top: 96px; width: 400px; font-size: 16px; line-height: 26px; font-weight: 400; color: #051343; text-align: left;"><span style="display: inline-flex; align-items: center; gap: 10px"><span style="display: inline-block; width: 30px; height: 20px; box-sizing: border-box; background: #e3f4ea; border: 2px solid #1c7a43; border-radius: 4px"></span><span><b>Green</b> household data (apps, files, photos)</span></span></div>
<div style="position: absolute; left: 2344px; top: 96px; width: 380px; font-size: 16px; line-height: 26px; font-weight: 400; color: #051343; text-align: left;"><span style="display: inline-flex; align-items: center; gap: 10px"><span style="display: inline-flex"><span style="display: inline-block; width: 16px; height: 20px; background: #5aaee8; margin-right: 2px"></span><span style="display: inline-block; width: 16px; height: 20px; background: #2f8fd6; margin-right: 2px"></span><span style="display: inline-block; width: 16px; height: 20px; background: #0b5fa5; margin-right: 2px"></span><span style="display: inline-block; width: 16px; height: 20px; background: #0a3d73; margin-right: 2px"></span></span><span><b>Blue</b> backup copies, one shade per tier</span></span></div>
<div style="position: absolute; left: 2740px; top: 96px; width: 380px; font-size: 16px; line-height: 26px; font-weight: 400; color: #051343; text-align: left;"><span style="display: inline-flex; align-items: center; gap: 10px"><span style="display: inline-block; width: 30px; height: 20px; box-sizing: border-box; background: #fdf0e1; border: 2px solid #a85400; border-radius: 4px"></span><span><b>Orange</b> operator infrastructure</span></span></div>
<div style="position: absolute; left: 1936px; top: 132px; width: 400px; font-size: 16px; line-height: 26px; font-weight: 400; color: #051343; text-align: left;"><span style="display: inline-flex; align-items: center; gap: 10px"><span style="display: inline-block; width: 30px; height: 20px; box-sizing: border-box; background: #f1e8fb; border: 2px solid #6f35b0; border-radius: 4px"></span><span><b>Purple</b> keys and secrets (names only)</span></span></div>
<div style="position: absolute; left: 2344px; top: 132px; width: 380px; font-size: 16px; line-height: 26px; font-weight: 400; color: #051343; text-align: left;"><span style="display: inline-flex; align-items: center; gap: 10px"><span style="display: inline-block; width: 30px; height: 20px; box-sizing: border-box; background: #eef0f3; border: 2px solid #5b6270; border-radius: 4px"></span><span><b>Grey</b> outside services</span></span></div>
<div style="position: absolute; left: 2740px; top: 132px; width: 380px; font-size: 16px; line-height: 26px; font-weight: 400; color: #051343; text-align: left;"><span style="display: inline-flex; align-items: center; gap: 10px"><span style="display: inline-block; width: 30px; height: 20px; box-sizing: border-box; background: #ffffff; border: 2px solid #c22d28; border-radius: 4px"></span><span><b>Red outline</b> single point of failure / gap</span></span></div>
<div style="position: absolute; left: 1936px; top: 168px; width: 400px; font-size: 16px; line-height: 26px; font-weight: 400; color: #051343; text-align: left;"><span style="display: inline-flex; align-items: center; gap: 10px"><svg width="56" height="12" viewBox="0 0 56 12" style="width: 56px; height: 12px"><path d="M2 6 H54" stroke="#1c7a43" stroke-width="3" stroke-linecap="round"></path></svg><span>solid = data traffic</span></span></div>
<div style="position: absolute; left: 2344px; top: 168px; width: 380px; font-size: 16px; line-height: 26px; font-weight: 400; color: #051343; text-align: left;"><span style="display: inline-flex; align-items: center; gap: 10px"><svg width="56" height="12" viewBox="0 0 56 12" style="width: 56px; height: 12px"><path d="M2 6 H54" stroke="#0b5fa5" stroke-width="3" stroke-linecap="round" stroke-dasharray="12 8"></path></svg><span>dashed = backup copy (with its time)</span></span></div>
<div style="position: absolute; left: 2740px; top: 168px; width: 380px; font-size: 16px; line-height: 26px; font-weight: 400; color: #051343; text-align: left;"><span style="display: inline-flex; align-items: center; gap: 10px"><svg width="56" height="12" viewBox="0 0 56 12" style="width: 56px; height: 12px"><path d="M2 6 H54" stroke="#4a5670" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.5 8"></path></svg><span>dotted = control (reports, jobs)</span></span></div>
<div style="position: absolute; left: 100px; top: 252px; width: 420px; font-size: 26px; line-height: 32px; font-weight: 800; color: #051343; text-align: left;">1 · The household's home</div>
<div style="position: absolute; left: 1260px; top: 252px; width: 420px; font-size: 26px; line-height: 32px; font-weight: 800; color: #051343; text-align: left;">2 · The internet</div>
<div style="position: absolute; left: 1940px; top: 252px; width: 560px; font-size: 26px; line-height: 32px; font-weight: 800; color: #051343; text-align: left;">3 · The operator's home</div>
<div style="position: absolute; left: 466px; top: 312px; width: 220px; height: 72px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #e3f4ea; border: 2px solid #1c7a43; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><svg width="22" height="22" viewBox="0 0 24 24" style="flex: none; width: 22px; height: 22px; fill: none; stroke: #1c7a43; stroke-width: 2; stroke-linecap: round; stroke-linejoin: round"><rect x="7" y="2.5" width="10" height="19" rx="2"></rect><path d="M11 18.5 H13"></path></svg><span><b>Household's phone / laptop</b></span></div>
<div style="position: absolute; left: 712px; top: 360px; width: 352px; font-size: 16px; line-height: 20px; font-weight: 400; color: #1c7a43; text-align: left;"><b>https only</b> — http:// is sent to https:// before any app answers.</div>
<div style="position: absolute; left: 712px; top: 256px; width: 352px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: left;"><b>Remote access.</b> HTTPS ends at Cloudflare's edge, so Cloudflare could technically see this traffic (later item: our own relay).</div>
<div style="position: absolute; left: 128px; top: 432px; width: 430px; font-size: 26px; line-height: 32px; font-weight: 800; color: #051343; text-align: left;">A Felhom box</div>
<div style="position: absolute; left: 128px; top: 464px; width: 430px; font-size: 18px; line-height: 24px; font-weight: 700; color: #051343; text-align: left;">(example: one household)</div>
<div style="position: absolute; left: 128px; top: 488px; width: 430px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: left;">a small PC running Proxmox, the host operating system</div>
<div style="position: absolute; left: 588px; top: 452px; width: 340px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: left;">At home on the LAN: direct to Traefik</div>
<div style="position: absolute; left: 148px; top: 552px; width: 400px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>Customer guest</b> — one Linux container per household (ID 9201), running Docker</div>
<div style="position: absolute; left: 148px; top: 604px; width: 252px; height: 104px; box-sizing: border-box; padding: 12px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #1c7a43; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Controller</b><br>the household's dashboard and the app manager; it has no Proxmox key</span></div>
<div style="position: absolute; left: 450px; top: 604px; width: 252px; height: 104px; box-sizing: border-box; padding: 12px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #1c7a43; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Traefik</b><br>routes each web address to its app</span></div>
<div style="position: absolute; left: 752px; top: 604px; width: 252px; height: 104px; box-sizing: border-box; padding: 12px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #1c7a43; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>cloudflared</b><br>the tunnel to Cloudflare, so no router setup is needed</span></div>
<div style="position: absolute; left: 588px; top: 716px; width: 120px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: left;">to the app</div>
<div style="position: absolute; left: 164px; top: 750px; width: 600px; font-size: 16px; line-height: 22px; font-weight: 400; color: #1c7a43; text-align: left;"><b>The apps</b> — 56 in the catalog, for example:</div>
<div style="position: absolute; left: 164px; top: 778px; width: 196px; height: 36px; box-sizing: border-box; padding: 4px; display: flex; align-items: center; justify-content: center; gap: 10px; text-align: center; background: #ffffff; border: 2px solid #1c7a43; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Nextcloud</b></span></div>
<div style="position: absolute; left: 372px; top: 778px; width: 196px; height: 36px; box-sizing: border-box; padding: 4px; display: flex; align-items: center; justify-content: center; gap: 10px; text-align: center; background: #ffffff; border: 2px solid #1c7a43; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Immich</b></span></div>
<div style="position: absolute; left: 580px; top: 778px; width: 196px; height: 36px; box-sizing: border-box; padding: 4px; display: flex; align-items: center; justify-content: center; gap: 10px; text-align: center; background: #ffffff; border: 2px solid #1c7a43; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Paperless-ngx</b></span></div>
<div style="position: absolute; left: 788px; top: 778px; width: 196px; height: 36px; box-sizing: border-box; padding: 4px; display: flex; align-items: center; justify-content: center; gap: 10px; text-align: center; background: #ffffff; border: 2px solid #1c7a43; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Vaultwarden</b></span></div>
<div style="position: absolute; left: 164px; top: 822px; width: 196px; height: 36px; box-sizing: border-box; padding: 4px; display: flex; align-items: center; justify-content: center; gap: 10px; text-align: center; background: #ffffff; border: 2px solid #1c7a43; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Jellyfin</b></span></div>
<div style="position: absolute; left: 372px; top: 822px; width: 196px; height: 36px; box-sizing: border-box; padding: 4px; display: flex; align-items: center; justify-content: center; gap: 10px; text-align: center; background: #ffffff; border: 2px solid #1c7a43; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Mealie</b></span></div>
<div style="position: absolute; left: 580px; top: 822px; width: 196px; height: 36px; box-sizing: border-box; padding: 4px; display: flex; align-items: center; justify-content: center; gap: 10px; text-align: center; background: #ffffff; border: 2px dashed #1c7a43; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>+50 more</b></span></div>
<div style="position: absolute; left: 148px; top: 912px; width: 250px; font-size: 17px; line-height: 22px; font-weight: 400; color: #051343; text-align: left;"><b>The Proxmox host</b><br>the host operating system</div>
<div style="position: absolute; left: 420px; top: 908px; width: 588px; height: 112px; box-sizing: border-box; padding: 12px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Host agent</b> — Felhom's operator-tier program. Holds the only Proxmox key; backs up and restores the whole guest; OS updates; drive management. Never accepts a connection from outside: it <b>polls the hub</b> for signed jobs.</span></div>
<div style="position: absolute; left: 128px; top: 1044px; width: 896px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;">The data drives are attached to the guest from the host. <b>Whole-guest backups do not include the data drives</b> — that is what the per-app tiers are for.</div>
<div style="position: absolute; left: 128px; top: 1096px; width: 488px; height: 212px; box-sizing: border-box; background: #ffffff; border: 2px solid #051343; border-radius: 8px;"></div>
<div style="position: absolute; left: 142px; top: 1106px; width: 460px; font-size: 16px; line-height: 22px; font-weight: 400; color: #051343; text-align: left;"><b>System disk</b> — Proxmox + the guest; local whole-guest copies <span style="display: inline-block; min-width: 26px; padding: 0 6px; box-sizing: border-box; height: 26px; border-radius: 13px; background: #0a3d73; color: #ffffff; font-weight: 800; text-align: center; line-height: 26px; font-size: 16px; vertical-align: middle">G</span></div>
<div style="position: absolute; left: 140px; top: 1154px; width: 464px; height: 146px; box-sizing: border-box; padding: 8px 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #e3f4ea; border: 2px dashed #1c7a43; border-radius: 6px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Data area — „Belső tárhely”</b> <span style="display: inline-block; padding: 0 8px; height: 22px; line-height: 20px; border-radius: 11px; border: 2px dashed #1c7a43; background: #ffffff; color: #1c7a43; font-size: 15px; font-weight: 800; vertical-align: middle">one-disk boxes only</span><br><span style="font-size: 15px; line-height: 19px">On a box with one disk, this part of the system disk is data drive&nbsp;1.<br>Not in the whole-guest backups.<br>A copy between it and the system data does not count as a second copy — same disk.</span></span></div>
<div style="position: absolute; left: 632px; top: 1096px; width: 188px; height: 212px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #e3f4ea; border: 2px solid #1c7a43; border-radius: 8px; font-size: 16px; line-height: 22px; font-weight: 400; color: #051343"><span><b>Data drive 1</b><br>household data + the Tier-1 backup folder <span style="display: inline-block; min-width: 26px; padding: 0 6px; box-sizing: border-box; height: 26px; border-radius: 13px; background: #5aaee8; color: #051343; font-weight: 800; text-align: center; line-height: 26px; font-size: 16px; vertical-align: middle">1</span></span></div>
<div style="position: absolute; left: 836px; top: 1096px; width: 188px; height: 212px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #e4f1fb; border: 2px dashed #2f8fd6; border-radius: 8px; font-size: 16px; line-height: 22px; font-weight: 400; color: #051343"><span><b>Data drive 2</b> (optional)<br>the Tier-2 copy <span style="display: inline-block; min-width: 26px; padding: 0 6px; box-sizing: border-box; height: 26px; border-radius: 13px; background: #2f8fd6; color: #051343; font-weight: 800; text-align: center; line-height: 26px; font-size: 16px; vertical-align: middle">2</span></span></div>
<div style="position: absolute; left: 1256px; top: 296px; width: 280px; height: 368px; box-sizing: border-box; padding: 14px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #eef0f3; border: 2px solid #5b6270; border-radius: 8px; font-size: 16px; line-height: 21px; font-weight: 400; color: #051343"><span><b style="font-size: 20px">Cloudflare</b><br>• DNS for the customer domains and for felhom.eu<br>• the tunnels<br>• the geo-block: only Hungarian visitors<br>• flood protection<br><br>The hub holds one Cloudflare key per customer zone and checks that it reaches only that customer's own domain.</span></div>
<div style="position: absolute; left: 1552px; top: 296px; width: 136px; height: 152px; box-sizing: border-box; padding: 8px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #5b6270; border-radius: 6px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>felhom.eu</b><br>public website, served from DooPlex, Cloudflare DNS only</span></div>
<div style="position: absolute; left: 1552px; top: 460px; width: 136px; height: 112px; box-sizing: border-box; padding: 8px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #5b6270; border-radius: 6px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>iso.felhom.eu</b><br>households download the installer</span></div>
<div style="position: absolute; left: 1552px; top: 580px; width: 136px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: center;">web pages</div>
<div style="position: absolute; left: 1272px; top: 738px; width: 400px; font-size: 17px; line-height: 26px; font-weight: 400; color: #051343; text-align: left;"><b style="font-size: 22px">ep0</b> — Hetzner cloud server, Germany</div>
<div style="position: absolute; left: 1272px; top: 766px; width: 400px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;">WireGuard endpoint + off-site backup server (PBS) + a tunnel-only line-speed test (iperf3). One datastore, split into namespaces:</div>
<div style="position: absolute; left: 1272px; top: 818px; width: 192px; height: 150px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #0a3d73; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><span style="display: inline-block; min-width: 26px; padding: 0 6px; box-sizing: border-box; height: 26px; border-radius: 13px; background: #0a3d73; color: #ffffff; font-weight: 800; text-align: center; line-height: 26px; font-size: 16px; vertical-align: middle">G</span> <b>One namespace per household</b><br>whole-guest copies, each household's own key<br><b>keeps one per week, 2 weeks</b></span></div>
<div style="position: absolute; left: 1480px; top: 818px; width: 192px; height: 150px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b><code style="font-size: 16px">operator</code> namespace</b><br>groups “hub database” and “DooPlex copy”<br><b>keeps 14 daily + 8 weekly</b></span></div>
<div style="position: absolute; left: 1272px; top: 974px; width: 400px; font-size: 15px; line-height: 19px; font-weight: 400; color: #4a5670; text-align: left;"><b>By design:</b> each household holds the key to its own copies; the operator cannot open them without the household’s recovery code.</div>
<div style="position: absolute; left: 1272px; top: 1036px; width: 400px; height: 56px; box-sizing: border-box; padding: 8px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #051343; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><svg width="24" height="24" viewBox="0 0 24 24" style="flex: none; width: 24px; height: 24px; fill: none; stroke: #051343; stroke-width: 2; stroke-linecap: round; stroke-linejoin: round"><path d="M12 2 L20 5 V11 C20 16.5 16.5 20.5 12 22 C7.5 20.5 4 16.5 4 11 V5 Z"></path><path d="M8.5 12 L11 14.5 L15.5 9.5"></path></svg><span><b>Protected</b> — holds the only off-premises copy of real household data.</span></div>
<div style="position: absolute; left: 1256px; top: 1110px; width: 432px; height: 130px; box-sizing: border-box; padding: 12px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #eef0f3; border: 2px solid #5b6270; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Hetzner Storage Box</b> <span style="display: inline-block; min-width: 26px; padding: 0 6px; box-sizing: border-box; height: 26px; border-radius: 13px; background: #0b5fa5; color: #ffffff; font-weight: 800; text-align: center; line-height: 26px; font-size: 16px; vertical-align: middle">3</span><br>The shared box for the households' per-app restic copies. One sub-account per household; <b>add-only keys</b>.</span></div>
<div style="position: absolute; left: 1256px; top: 1252px; width: 432px; height: 64px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #eef0f3; border: 2px solid #5b6270; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Mail service</b> (Resend)<br>every mail the hub sends — to the operator and to households</span></div>
<div style="position: absolute; left: 1082px; top: 628px; width: 140px; font-size: 16px; line-height: 20px; font-weight: 700; color: #1c7a43; text-align: center;">via the tunnel</div>
<div style="position: absolute; left: 1082px; top: 856px; width: 140px; font-size: 16px; line-height: 20px; font-weight: 700; color: #0a3d73; text-align: center;">Whole-guest backup — encrypted on the box, sent through WireGuard —</div>
<div style="position: absolute; left: 1090px; top: 982px; width: 124px; height: 36px; box-sizing: border-box; padding: 4px; display: flex; align-items: center; justify-content: center; gap: 6px; text-align: center; background: #ffffff; border: 2px solid #0a3d73; border-radius: 18px; font-size: 16px; line-height: 20px; font-weight: 700; color: #051343"><svg width="18" height="18" viewBox="0 0 24 24" style="flex: none; width: 18px; height: 18px; fill: none; stroke: #0a3d73; stroke-width: 2; stroke-linecap: round; stroke-linejoin: round"><rect x="4" y="11" width="16" height="10" rx="2"></rect><path d="M8 11 V7 a4 4 0 0 1 8 0 V11"></path></svg><span>WireGuard</span></div>
<div style="position: absolute; left: 1082px; top: 1026px; width: 140px; font-size: 16px; line-height: 20px; font-weight: 400; color: #0a3d73; text-align: center;">weekly, 04:30–08:30</div>
<div style="position: absolute; left: 1082px; top: 1124px; width: 140px; font-size: 16px; line-height: 20px; font-weight: 700; color: #0b5fa5; text-align: center;">per-app off-site (restic)</div>
<div style="position: absolute; left: 1082px; top: 1184px; width: 140px; font-size: 16px; line-height: 20px; font-weight: 400; color: #0b5fa5; text-align: center;">nightly 04:15, encrypted on the box</div>
<div style="position: absolute; left: 1720px; top: 626px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 700; color: #4a5670; text-align: center;">Box reports every few minutes; asks for signed jobs.</div>
<div style="position: absolute; left: 1720px; top: 706px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: center;">The hub never connects into a box.</div>
<div style="position: absolute; left: 1720px; top: 784px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">00:20 DooPlex copy</div>
<div style="position: absolute; left: 1720px; top: 824px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #a85400; text-align: center;">DooPlex off-site key</div>
<div style="position: absolute; left: 1720px; top: 872px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">every 6 h: hub database</div>
<div style="position: absolute; left: 1720px; top: 912px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #a85400; text-align: center;">hub-DB key</div>
<div style="position: absolute; left: 1720px; top: 962px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #0a3d73; text-align: center;">05:00 ep0-copy pull</div>
<div style="position: absolute; left: 1720px; top: 1002px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #0a3d73; text-align: center;">ciphertext only</div>
<div style="position: absolute; left: 1956px; top: 308px; width: 552px; font-size: 18px; line-height: 32px; font-weight: 400; color: #051343; text-align: left;"><b style="font-size: 26px">DooPlex</b> — the operator's home server</div>
<div style="position: absolute; left: 1956px; top: 342px; width: 552px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;">a small Kubernetes cluster (k3s) plus host services</div>
<div style="position: absolute; left: 1956px; top: 372px; width: 552px; height: 48px; box-sizing: border-box; padding: 8px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #051343; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><svg width="24" height="24" viewBox="0 0 24 24" style="flex: none; width: 24px; height: 24px; fill: none; stroke: #051343; stroke-width: 2; stroke-linecap: round; stroke-linejoin: round"><path d="M12 2 L20 5 V11 C20 16.5 16.5 20.5 12 22 C7.5 20.5 4 16.5 4 11 V5 Z"></path><path d="M8.5 12 L11 14.5 L15.5 9.5"></path></svg><span><b>Protected</b> — everything else can be rebuilt from it</span></div>
<div style="position: absolute; left: 1956px; top: 436px; width: 268px; height: 140px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Gitea</b><br>all the code (10 repositories) and the container registry (the images the boxes run); the CI runner runs here too</span></div>
<div style="position: absolute; left: 1956px; top: 588px; width: 268px; height: 164px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>The hub</b><br>customer records · box reports · the job queue · alarms · the escrow of household keys · the System page (OS updates, kernel approvals)</span></div>
<div style="position: absolute; left: 1956px; top: 764px; width: 268px; height: 76px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>The website</b> (felhom.eu) and the contact form's mailer</span></div>
<div style="position: absolute; left: 1956px; top: 852px; width: 268px; height: 156px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #e4f1fb; border: 2px solid #0a3d73; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>ep0-copy</b><br>a nightly pull of the whole ep0 datastore back to DooPlex — ciphertext only; DooPlex cannot read households' copies</span></div>
<div style="position: absolute; left: 2240px; top: 436px; width: 268px; height: 80px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Vaultwarden</b><br>the operator's password manager</span></div>
<div style="position: absolute; left: 2240px; top: 528px; width: 268px; height: 76px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Prometheus + Alertmanager</b><br>the operator's alarms</span></div>
<div style="position: absolute; left: 2240px; top: 620px; width: 268px; height: 100px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><svg width="22" height="22" viewBox="0 0 24 24" style="flex: none; width: 22px; height: 22px; fill: none; stroke: #6f35b0; stroke-width: 2; stroke-linecap: round; stroke-linejoin: round"><circle cx="7.5" cy="15.5" r="4.5"></circle><path d="M10.7 12.3 L20 3"></path><path d="M16 7 L19 10"></path><path d="M18 5 L21 8"></path></svg><span><b>The signing keys</b><br>sign every agent update and config bundle a box accepts</span></div>
<div style="position: absolute; left: 2240px; top: 736px; width: 268px; height: 120px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 3px solid #c22d28; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Local backups</b><br>several restic sets on DooPlex's own disks — <b>all on the same machine</b></span></div>
<div style="position: absolute; left: 2240px; top: 872px; width: 268px; height: 136px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px dashed #5b6270; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Left out of the DooPlex copy on purpose:</b> the container registry (27.7 GB) — rebuilt from the code</span></div>
<div style="position: absolute; left: 1940px; top: 1034px; width: 600px; font-size: 20px; line-height: 26px; font-weight: 800; color: #051343; text-align: left;">Test machines <span style="font-weight: 600; color: #4a5670">— disposable</span></div>
<div style="position: absolute; left: 1936px; top: 1068px; width: 288px; height: 164px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #5b6270; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>demo-hp</b> (a mini PC) — ring 0: gets OS updates, Docker and kernels first. Also hosts the <b>scratch guest 9202</b> (no hub link, for tests) and the <b>Tester 1</b> VM (a whole Felhom box inside a VM)</span></div>
<div style="position: absolute; left: 2240px; top: 1068px; width: 288px; height: 60px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #5b6270; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>demo-felhom</b> (Intel N100 mini PC) — ring 0</span></div>
<div style="position: absolute; left: 2240px; top: 1136px; width: 288px; height: 96px; box-sizing: border-box; padding: 8px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #5b6270; border-radius: 8px; font-size: 16px; line-height: 19px; font-weight: 400; color: #051343"><span><b>The test bench</b> — a throwaway guest where app updates are tested before any box gets them</span></div>
<div style="position: absolute; left: 1936px; top: 1244px; width: 592px; height: 88px; box-sizing: border-box; padding: 12px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #e3f4ea; border: 2px solid #1c7a43; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><svg width="26" height="26" viewBox="0 0 24 24" style="flex: none; width: 26px; height: 26px; fill: none; stroke: #1c7a43; stroke-width: 2; stroke-linecap: round; stroke-linejoin: round"><rect x="3" y="6" width="18" height="13" rx="2"></rect><path d="M7 10 H11 M7 14 H9"></path><circle cx="16" cy="12.5" r="1.4"></circle></svg><span><b>Tester 2</b> — the first outside tester's laptop, at his own home: a second, smaller Felhom box. <b>Real household, often off.</b></span></div>
<div style="position: absolute; left: 96px; top: 1370px; width: 900px; font-size: 24px; line-height: 30px; font-weight: 800; color: #051343; text-align: left;">Where a household's data is copied</div>
<div style="position: absolute; left: 84px; top: 1402px; width: 1096px; display: flex; flex-direction: column; gap: 8px"><table style="width: 1096px; border-collapse: collapse; table-layout: fixed; font-size: 16px; line-height: 20px; color: #051343"><thead><tr><th style="width: 134px; padding: 6px 8px; text-align: left; font-weight: 800; border-bottom: 2px solid #051343; vertical-align: bottom">Tier</th><th style="width: 156px; padding: 6px 8px; text-align: left; font-weight: 800; border-bottom: 2px solid #051343; vertical-align: bottom">What it copies</th><th style="width: 280px; padding: 6px 8px; text-align: left; font-weight: 800; border-bottom: 2px solid #051343; vertical-align: bottom">Where</th><th style="width: 144px; padding: 6px 8px; text-align: left; font-weight: 800; border-bottom: 2px solid #051343; vertical-align: bottom">When</th><th style="width: 134px; padding: 6px 8px; text-align: left; font-weight: 800; border-bottom: 2px solid #051343; vertical-align: bottom">Kept</th><th style="width: 152px; padding: 6px 8px; text-align: left; font-weight: 800; border-bottom: 2px solid #051343; vertical-align: bottom">Locked by</th></tr></thead><tbody><tr><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3"><span style="display: inline-block; min-width: 26px; padding: 0 6px; box-sizing: border-box; height: 26px; border-radius: 13px; background: #5aaee8; color: #051343; font-weight: 800; text-align: center; line-height: 26px; font-size: 16px; vertical-align: middle">1</span> <b>Tier 1</b> — recovery unit</td><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3">each app: its settings, database dump, volumes</td><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3">data drive 1, on the box</td><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3">nightly at <b>02:30</b> (W)</td><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3">1 point per app</td><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3">not encrypted (it never leaves the home)</td></tr><tr><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3"><span style="display: inline-block; min-width: 26px; padding: 0 6px; box-sizing: border-box; height: 26px; border-radius: 13px; background: #2f8fd6; color: #051343; font-weight: 800; text-align: center; line-height: 26px; font-size: 16px; vertical-align: middle">2</span> <b>Tier 2</b> — cross-drive</td><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3">a mirror of Tier 1, plus the app's files</td><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3">data drive 2, on the box — counted only on a different physical disk; in a VM made, not counted</td><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3">nightly <b>03:30</b></td><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3">mirror</td><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3">not encrypted</td></tr><tr><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3"><span style="display: inline-block; min-width: 26px; padding: 0 6px; box-sizing: border-box; height: 26px; border-radius: 13px; background: #0b5fa5; color: #ffffff; font-weight: 800; text-align: center; line-height: 26px; font-size: 16px; vertical-align: middle">3</span> <b>Tier 3</b> — off-site, per app</td><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3">Tier 1 + the app's must-keep files</td><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3"><b>Hetzner Storage Box</b> (Germany), one folder per household</td><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3">nightly <b>04:15</b></td><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3">7 daily, 4 weekly, 6 monthly</td><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3">the household's own restic password</td></tr><tr><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3"><span style="display: inline-block; min-width: 26px; padding: 0 6px; box-sizing: border-box; height: 26px; border-radius: 13px; background: #0a3d73; color: #ffffff; font-weight: 800; text-align: center; line-height: 26px; font-size: 16px; vertical-align: middle">G</span> <b>Whole guest</b> — local</td><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3">the guest's system (not the data drives)</td><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3">the box's own system disk</td><td style="padding: 3px 8px; vertical-align: top; border-bottom: 1px solid #b9c3d3">daily, in the window <b>04:30–08:30</b></td><td style="paLine truncated
<div style="position: absolute; left: 1256px; top: 1380px; width: 620px; font-size: 24px; line-height: 30px; font-weight: 800; color: #051343; text-align: left;">Where the operator's data is copied</div>
<div style="position: absolute; left: 1256px; top: 1424px; width: 612px; height: 136px; box-sizing: border-box; padding: 12px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>00:20 · DooPlex copy → ep0</b> (<code>operator</code>): the code (614 MB), Gitea's database and settings, the <b>password manager</b>, the nightly secrets export, the <b>signing keys</b>. Encrypted on DooPlex with the <b>DooPlex off-site key</b>; the key that writes it <b>cannot delete</b> old copies. Tested back every Sunday.</span></div>
<div style="position: absolute; left: 1256px; top: 1572px; width: 612px; height: 60px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>02:30 · Hub database → ep0</b> (<code>operator</code>): encrypted with the <b>hub-DB key</b>. Tested back every Sunday at 04:30.</span></div>
<div style="position: absolute; left: 1256px; top: 1644px; width: 612px; height: 84px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #e4f1fb; border: 2px solid #0a3d73; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>05:00 · ep0-copy: ep0 → DooPlex</b> a second copy of ep0 at home. Keeps 8 weekly copies. A daily job removes a deleted customer's copy within 30 days.</span></div>
<div style="position: absolute; left: 1256px; top: 1740px; width: 612px; height: 60px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px dashed #5b6270; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Not copied:</b> the container registry (27.7 GB) — rebuilt from the code.</span></div>
<div style="position: absolute; left: 1256px; top: 1812px; width: 612px; height: 56px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 3px solid #c22d28; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Local backups</b> on DooPlex's own disks: all on the same machine.</span></div>
<div style="position: absolute; left: 1936px; top: 1380px; width: 600px; font-size: 24px; line-height: 30px; font-weight: 800; color: #051343; text-align: left;">Who tells a box what to do</div>
<div style="position: absolute; left: 1932px; top: 1420px; width: 188px; height: 76px; box-sizing: border-box; padding: 8px; display: flex; align-items: center; justify-content: center; gap: 6px; text-align: center; background: #ffffff; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span>The <b>hub</b> queues a job</span></div>
<div style="position: absolute; left: 2132px; top: 1420px; width: 188px; height: 76px; box-sizing: border-box; padding: 8px; display: flex; align-items: center; justify-content: center; gap: 6px; text-align: center; background: #ffffff; border: 2px solid #6f35b0; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><svg width="20" height="20" viewBox="0 0 24 24" style="flex: none; width: 20px; height: 20px; fill: none; stroke: #6f35b0; stroke-width: 2; stroke-linecap: round; stroke-linejoin: round"><circle cx="7.5" cy="15.5" r="4.5"></circle><path d="M10.7 12.3 L20 3"></path><path d="M16 7 L19 10"></path><path d="M18 5 L21 8"></path></svg><span>The <b>operator signs</b> it</span></div>
<div style="position: absolute; left: 2332px; top: 1420px; width: 188px; height: 76px; box-sizing: border-box; padding: 8px; display: flex; align-items: center; justify-content: center; gap: 6px; text-align: center; background: #ffffff; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span>The <b>agent</b> checks the signature, then acts</span></div>
<div style="position: absolute; left: 1936px; top: 1506px; width: 600px; font-size: 17px; line-height: 22px; font-weight: 800; color: #051343; text-align: left;">Updates, in rings</div>
<div style="position: absolute; left: 1932px; top: 1532px; width: 188px; height: 64px; box-sizing: border-box; padding: 8px; display: flex; align-items: center; justify-content: center; gap: 10px; text-align: center; background: #ffffff; border: 2px solid #5b6270; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Ring 0</b>: the demo boxes, first</span></div>
<div style="position: absolute; left: 2132px; top: 1532px; width: 188px; height: 64px; box-sizing: border-box; padding: 8px; display: flex; align-items: center; justify-content: center; gap: 10px; text-align: center; background: #ffffff; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span>Operator approves on the <b>System page</b></span></div>
<div style="position: absolute; left: 2332px; top: 1532px; width: 188px; height: 64px; box-sizing: border-box; padding: 8px; display: flex; align-items: center; justify-content: center; gap: 10px; text-align: center; background: #e3f4ea; border: 2px solid #1c7a43; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Ring 1</b>: the customers</span></div>
<div style="position: absolute; left: 1936px; top: 1606px; width: 596px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>Apps:</b> each new version is tested on the bench and the scratch box first; boxes update at night, after a fresh backup.<br><b>Debian security fixes:</b> nightly, after the whole-guest backup.<br><b>Proxmox programs, Docker, the kernel:</b> slow lanes, one approved set at a time.<br><b>Kernel:</b> the household gets a mail the day before; the box restarts at night; a crash falls back to the old kernel by itself; a freeze needs someone to unplug the box.</div>
<div style="position: absolute; left: 1932px; top: 1794px; width: 600px; height: 80px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #fdf0e1; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>Hub buttons</b> — no signature needed, a fixed safe list: off-site backup now · run a check now · stop or extend a deletion countdown.</span></div>
<div style="position: absolute; left: 96px; top: 1918px; width: 600px; font-size: 24px; line-height: 30px; font-weight: 800; color: #051343; text-align: left;">One night, hour by hour</div>
<div style="position: absolute; left: 1640px; top: 1922px; width: 888px; font-size: 16px; line-height: 22px; font-weight: 400; color: #4a5670; text-align: right;">Household mails about a kernel restart go out the day before, 09:00–20:00.</div>
<div style="position: absolute; left: 270px; top: 1952px; width: 60px; font-size: 16px; line-height: 20px; font-weight: 700; color: #4a5670; text-align: center;">00:00</div>
<div style="position: absolute; left: 510px; top: 1952px; width: 60px; font-size: 16px; line-height: 20px; font-weight: 700; color: #4a5670; text-align: center;">01:00</div>
<div style="position: absolute; left: 750px; top: 1952px; width: 60px; font-size: 16px; line-height: 20px; font-weight: 700; color: #4a5670; text-align: center;">02:00</div>
<div style="position: absolute; left: 990px; top: 1952px; width: 60px; font-size: 16px; line-height: 20px; font-weight: 700; color: #4a5670; text-align: center;">03:00</div>
<div style="position: absolute; left: 1230px; top: 1952px; width: 60px; font-size: 16px; line-height: 20px; font-weight: 700; color: #4a5670; text-align: center;">04:00</div>
<div style="position: absolute; left: 1470px; top: 1952px; width: 60px; font-size: 16px; line-height: 20px; font-weight: 700; color: #4a5670; text-align: center;">05:00</div>
<div style="position: absolute; left: 1710px; top: 1952px; width: 60px; font-size: 16px; line-height: 20px; font-weight: 700; color: #4a5670; text-align: center;">06:00</div>
<div style="position: absolute; left: 1950px; top: 1952px; width: 60px; font-size: 16px; line-height: 20px; font-weight: 700; color: #4a5670; text-align: center;">07:00</div>
<div style="position: absolute; left: 2190px; top: 1952px; width: 60px; font-size: 16px; line-height: 20px; font-weight: 700; color: #4a5670; text-align: center;">08:00</div>
<div style="position: absolute; left: 2430px; top: 1952px; width: 60px; font-size: 16px; line-height: 20px; font-weight: 700; color: #4a5670; text-align: center;">09:00</div>
<div style="position: absolute; left: 96px; top: 2019px; width: 190px; font-size: 18px; line-height: 22px; font-weight: 800; color: #a85400; text-align: left;">DooPlex</div>
<div style="position: absolute; left: 96px; top: 2103px; width: 190px; font-size: 18px; line-height: 22px; font-weight: 800; color: #1c7a43; text-align: left;">Customer box</div>
<div style="position: absolute; left: 294px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #a85400; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 298px; top: 2000px; width: 190px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>00:00</b> database dumps</div>
<div style="position: absolute; left: 374px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #a85400; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 378px; top: 2040px; width: 220px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>00:20</b> DooPlex copy → ep0</div>
<div style="position: absolute; left: 894px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #a85400; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 898px; top: 2000px; width: 230px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>02:30</b> hub DB → ep0, 6-hourly</div>
<div style="position: absolute; left: 1134px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #0a3d73; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 1138px; top: 2040px; width: 236px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>03:30</b> ep0 prunes households</div>
<div style="position: absolute; left: 1194px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #0a3d73; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 1198px; top: 2000px; width: 260px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>03:45</b> ep0 prunes <code>operator</code></div>
<div style="position: absolute; left: 1374px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #a85400; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 1378px; top: 2040px; width: 260px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>Sundays 04:30</b> restore tests</div>
<div style="position: absolute; left: 1494px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #0a3d73; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 1498px; top: 2000px; width: 270px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>05:00</b> ep0-copy pull → DooPlex</div>
<div style="position: absolute; left: 2094px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #0a3d73; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 2098px; top: 2040px; width: 200px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>07:30</b> ep0-copy prune</div>
<div style="position: absolute; left: 2214px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #4a5670; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 2218px; top: 2000px; width: 280px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>08:00</b> deleted-customer clean-up</div>
<div style="position: absolute; left: 894px; top: 2108px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #5aaee8; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 898px; top: 2084px; width: 140px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>02:30</b> Tier 1</div>
<div style="position: absolute; left: 1134px; top: 2108px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #2f8fd6; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 1138px; top: 2124px; width: 140px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>03:30</b> Tier 2</div>
<div style="position: absolute; left: 1314px; top: 2108px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #0b5fa5; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 1318px; top: 2084px; width: 190px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>04:15</b> Tier 3 off-site</div>
<div style="position: absolute; left: 1334px; top: 2124px; width: 150px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;">→ app updates</div>
<div style="position: absolute; left: 1504px; top: 2084px; width: 1040px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>04:30–08:30</b> whole-guest backup (local daily, off-site weekly) → Debian fixes → (when approved) Proxmox, Docker, kernel restart</div>
<div style="position: absolute; left: 1494px; top: 2108px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #4a5670; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 1498px; top: 2124px; width: 420px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>05:00</b> the hub checks that every box made its backup</div>
<div style="position: absolute; left: 2616px; top: 252px; width: 488px; font-size: 26px; line-height: 32px; font-weight: 800; color: #051343; text-align: left;">If DooPlex is lost</div>
<div style="position: absolute; left: 2616px; top: 300px; width: 488px; display: flex; flex-direction: column; gap: 16px"><div style="display: flex; gap: 16px; align-items: flex-start"><div style="flex: 1 1 0; min-width: 0; box-sizing: border-box; padding: 12px; background: #ffffff; border-radius: 8px; font-size: 16px; line-height: 20px; border: 2px solid #051343"><b>Survives</b> (off DooPlex)<br>• <b>ep0</b>: the DooPlex copy and the hub database, plus every household's whole-guest copies<br>• the <b>Hetzner Storage Box</b>: households' per-app copies<br>• <b>the boxes themselves</b>: they keep running and backing up, but cannot report or update until the hub is back<br>• the operator's workstation</div><div style="flex: 1 1 0; min-width: 0; box-sizing: border-box; padding: 12px; background: #ffffff; border-radius: 8px; font-size: 16px; line-height: 20px; border: 3px solid #c22d28"><b>Lost and not in any copy</b><br>• the container registry (rebuild from code)<br>• DooPlex's own SSH key<br>• local build files<br>• Prometheus history<br>• other homelab apps</div></div><div style="font-size: 19px; line-height: 24px; font-weight: 800; margin-top: 4px">The way back</div><div style="display: flex; flex-direction: column; gap: 8px"><div style="display: flex; align-items: flex-start; gap: 10px; font-size: 16px; line-height: 24px"><span style="flex: none; width: 24px; height: 24px; border-radius: 12px; background: #051343; color: #ffffff; font-weight: 800; text-align: center; line-height: 24px; font-size: 15px">1</span><span>New machine</span></div><div style="display: flex; align-items: flex-start; gap: 10px; font-size: 16px; line-height: 24px"><span style="flex: none; width: 24px; height: 24px; border-radius: 12px; background: #051343; color: #ffffff; font-weight: 800; text-align: center; line-height: 24px; font-size: 15px">2</span><span>Reach ep0</span></div><div style="display: flex; align-items: flex-start; gap: 10px; font-size: 16px; line-height: 24px"><span style="flex: none; width: 24px; height: 24px; border-radius: 12px; background: #051343; color: #ffffff; font-weight: 800; text-align: center; line-height: 24px; font-size: 15px">3</span><span>Open the DooPlex copy with the <b>paper key</b></span></div><div style="display: flex; align-items: flex-start; gap: 10px; font-size: 16px; line-height: 24px"><span style="flex: none; width: 24px; height: 24px; border-radius: 12px; background: #051343; color: #ffffff; font-weight: 800; text-align: center; line-height: 24px; font-size: 15px">4</span><span>Password manager first</span></div><div style="display: flex; align-items: flex-start; gap: 10px; font-size: 16px; line-height: 24px"><span style="flex: none; width: 24px; height: 24px; border-radius: 12px; background: #051343; color: #ffffff; font-weight: 800; text-align: center; line-height: 24px; font-size: 15px">5</span><span>Gitea → rebuild images</span></div><div style="display: flex; align-items: flex-start; gap: 10px; font-size: 16px; line-height: 24px"><span style="flex: none; width: 24px; height: 24px; border-radius: 12px; background: #051343; color: #ffffff; font-weight: 800; text-align: center; line-height: 24px; font-size: 15px">6</span><span>Hub (mail held)</span></div><div style="display: flex; align-items: flex-start; gap: 10px; font-size: 16px; line-height: 24px"><span style="flex: none; width: 24px; height: 24px; border-radius: 12px; background: #051343; color: #ffffff; font-weight: 800; text-align: center; line-height: 24px; font-size: 15px">7</span><span>Point DNS at the new home → boxes reconnect by themselves</span></div></div></div>
<div style="position: absolute; left: 2616px; top: 1014px; width: 488px; font-size: 24px; line-height: 30px; font-weight: 800; color: #6f35b0; text-align: left;">Keys that must exist on paper, away from home</div>
<div style="position: absolute; left: 2616px; top: 1084px; width: 488px; height: 52px; box-sizing: border-box; padding: 6px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #f1e8fb; border: 1px solid #6f35b0; border-radius: 8px; font-size: 16px; line-height: 19px; font-weight: 400; color: #051343"><svg width="22" height="22" viewBox="0 0 24 24" style="flex: none; width: 22px; height: 22px; fill: none; stroke: #6f35b0; stroke-width: 2; stroke-linecap: round; stroke-linejoin: round"><circle cx="7.5" cy="15.5" r="4.5"></circle><path d="M10.7 12.3 L20 3"></path><path d="M16 7 L19 10"></path><path d="M18 5 L21 8"></path></svg><span><b>1. DooPlex off-site key</b> — opens the DooPlex copy — the key to everything else</span></div>
<div style="position: absolute; left: 2616px; top: 1142px; width: 488px; height: 52px; box-sizing: border-box; padding: 6px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #f1e8fb; border: 1px solid #6f35b0; border-radius: 8px; font-size: 16px; line-height: 19px; font-weight: 400; color: #051343"><svg width="22" height="22" viewBox="0 0 24 24" style="flex: none; width: 22px; height: 22px; fill: none; stroke: #6f35b0; stroke-width: 2; stroke-linecap: round; stroke-linejoin: round"><circle cx="7.5" cy="15.5" r="4.5"></circle><path d="M10.7 12.3 L20 3"></path><path d="M16 7 L19 10"></path><path d="M18 5 L21 8"></path></svg><span><b>2. Hub-DB off-site key</b> — opens the hub database copy</span></div>
<div style="position: absolute; left: 2616px; top: 1200px; width: 488px; height: 52px; box-sizing: border-box; padding: 6px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #f1e8fb; border: 1px solid #6f35b0; border-radius: 8px; font-size: 16px; line-height: 19px; font-weight: 400; color: #051343"><svg width="22" height="22" viewBox="0 0 24 24" style="flex: none; width: 22px; height: 22px; fill: none; stroke: #6f35b0; stroke-width: 2; stroke-linecap: round; stroke-linejoin: round"><circle cx="7.5" cy="15.5" r="4.5"></circle><path d="M10.7 12.3 L20 3"></path><path d="M16 7 L19 10"></path><path d="M18 5 L21 8"></path></svg><span><b>3. Hub seal key</b> — opens the console passwords inside the hub</span></div>
<div style="position: absolute; left: 2616px; top: 1258px; width: 488px; height: 52px; box-sizing: border-box; padding: 6px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #f1e8fb; border: 1px solid #6f35b0; border-radius: 8px; font-size: 16px; line-height: 19px; font-weight: 400; color: #051343"><svg width="22" height="22" viewBox="0 0 24 24" style="flex: none; width: 22px; height: 22px; fill: none; stroke: #6f35b0; stroke-width: 2; stroke-linecap: round; stroke-linejoin: round"><circle cx="7.5" cy="15.5" r="4.5"></circle><path d="M10.7 12.3 L20 3"></path><path d="M16 7 L19 10"></path><path d="M18 5 L21 8"></path></svg><span><b>4. DooPlex backup passphrase</b> — opens the secrets export</span></div>
<div style="position: absolute; left: 2616px; top: 1316px; width: 488px; height: 52px; box-sizing: border-box; padding: 6px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #f1e8fb; border: 1px solid #6f35b0; border-radius: 8px; font-size: 16px; line-height: 19px; font-weight: 400; color: #051343"><svg width="22" height="22" viewBox="0 0 24 24" style="flex: none; width: 22px; height: 22px; fill: none; stroke: #6f35b0; stroke-width: 2; stroke-linecap: round; stroke-linejoin: round"><circle cx="7.5" cy="15.5" r="4.5"></circle><path d="M10.7 12.3 L20 3"></path><path d="M16 7 L19 10"></path><path d="M18 5 L21 8"></path></svg><span><b>5. Signing keys (recovery + operational)</b> — let the operator update boxes again</span></div>
<div style="position: absolute; left: 2616px; top: 1374px; width: 488px; height: 52px; box-sizing: border-box; padding: 6px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #f1e8fb; border: 1px solid #6f35b0; border-radius: 8px; font-size: 16px; line-height: 19px; font-weight: 400; color: #051343"><svg width="22" height="22" viewBox="0 0 24 24" style="flex: none; width: 22px; height: 22px; fill: none; stroke: #6f35b0; stroke-width: 2; stroke-linecap: round; stroke-linejoin: round"><circle cx="7.5" cy="15.5" r="4.5"></circle><path d="M10.7 12.3 L20 3"></path><path d="M16 7 L19 10"></path><path d="M18 5 L21 8"></path></svg><span><b>6. ep0 read token</b> — reads the copies</span></div>
<div style="position: absolute; left: 2616px; top: 1432px; width: 488px; height: 52px; box-sizing: border-box; padding: 6px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #f1e8fb; border: 1px solid #6f35b0; border-radius: 8px; font-size: 16px; line-height: 19px; font-weight: 400; color: #051343"><svg width="22" height="22" viewBox="0 0 24 24" style="flex: none; width: 22px; height: 22px; fill: none; stroke: #6f35b0; stroke-width: 2; stroke-linecap: round; stroke-linejoin: round"><circle cx="7.5" cy="15.5" r="4.5"></circle><path d="M10.7 12.3 L20 3"></path><path d="M16 7 L19 10"></path><path d="M18 5 L21 8"></path></svg><span><b>7. Hetzner login</b> — reaches ep0 if DooPlex is gone</span></div>
<div style="position: absolute; left: 2616px; top: 1490px; width: 488px; height: 52px; box-sizing: border-box; padding: 6px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #f1e8fb; border: 1px solid #6f35b0; border-radius: 8px; font-size: 16px; line-height: 19px; font-weight: 400; color: #051343"><svg width="22" height="22" viewBox="0 0 24 24" style="flex: none; width: 22px; height: 22px; fill: none; stroke: #6f35b0; stroke-width: 2; stroke-linecap: round; stroke-linejoin: round"><circle cx="7.5" cy="15.5" r="4.5"></circle><path d="M10.7 12.3 L20 3"></path><path d="M16 7 L19 10"></path><path d="M18 5 L21 8"></path></svg><span><b>8. Cloudflare login</b> — moves DNS to the new home</span></div>
<div style="position: absolute; left: 2616px; top: 1552px; width: 488px; height: 52px; box-sizing: border-box; padding: 6px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px dashed #6f35b0; border-radius: 8px; font-size: 16px; line-height: 19px; font-weight: 400; color: #051343"><svg width="24" height="24" viewBox="0 0 24 24" style="flex: none; width: 24px; height: 24px; fill: none; stroke: #6f35b0; stroke-width: 2; stroke-linecap: round; stroke-linejoin: round"><circle cx="12" cy="8" r="4.5"></circle><path d="M4 21 C4 16 7.5 13.5 12 13.5 C16.5 13.5 20 16 20 21"></path></svg><span><b>The password-manager master password</b> — memorised, never written on the sheet</span></div>
<div style="position: absolute; left: 2616px; top: 1616px; width: 488px; font-size: 17px; line-height: 22px; font-weight: 800; color: #c22d28; text-align: left;">A key kept only in the password manager is not off DooPlex — the password manager runs on DooPlex.</div>
<div style="position: absolute; left: 2616px; top: 1742px; width: 488px; font-size: 24px; line-height: 30px; font-weight: 800; color: #051343; text-align: left;">Known gaps</div>
<div style="position: absolute; left: 2616px; top: 1788px; width: 488px; height: 108px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 3px solid #c22d28; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><svg width="22" height="22" viewBox="0 0 24 24" style="flex: none; width: 22px; height: 22px; fill: none; stroke: #c22d28; stroke-width: 2; stroke-linecap: round; stroke-linejoin: round"><path d="M12 3 L22 20 H2 Z"></path><path d="M12 10 V14"></path><path d="M12 17.5 V17.6"></path></svg><span><b>Cloudflare can read remote-access traffic</b> (HTTPS ends at its edge). Later item: our own relay.</span></div>
<div style="position: absolute; left: 2616px; top: 1912px; width: 488px; height: 108px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 3px solid #c22d28; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><svg width="22" height="22" viewBox="0 0 24 24" style="flex: none; width: 22px; height: 22px; fill: none; stroke: #c22d28; stroke-width: 2; stroke-linecap: round; stroke-linejoin: round"><path d="M12 3 L22 20 H2 Z"></path><path d="M12 10 V14"></path><path d="M12 17.5 V17.6"></path></svg><span><b>DooPlex's local backups all sit on one machine</b>; only the DooPlex copy and the hub database leave it.</span></div>
<script>
document.addEventListener('DOMContentLoaded', async function() {
const loading = document.getElementById('__bundler_loading');
function setStatus(msg) { if (loading) loading.textContent = msg; }
const FONT_MIME = /^(font[/]|application[/](x-)?font-|application[/]vnd\.ms-fontobject)/i;
const MIME_TOKEN = /^[\w.+-]+[/][\w.+-]+$/;
function toBase64(bytes) {
let bin = '';
for (let i = 0; i < bytes.length; i += 0x8000) {
bin += String.fromCharCode.apply(null, bytes.subarray(i, i + 0x8000));
}
return btoa(bin);
}
// Error sink persists across replaceWith since it's on window, not the DOM.
window.addEventListener('error', function(e) {
// Failed resource loads (CSP-blocked links, scripts, images) fire plain
// events at the element — warn only; real JS errors carry message/error.
if (!e.message && !e.error && e.target && e.target !== window) {
console.warn('[bundle] resource failed to load:',
e.target.tagName, String(e.target.src || e.target.href || ''));
return;
}
var p = document.body || document.documentElement;
var d = document.getElementById('__bundler_err') || p.appendChild(document.createElement('div'));
d.id = '__bundler_err';
d.style.cssText = 'position:fixed;bottom:12px;left:12px;right:12px;font:12px/1.4 ui-monospace,monospace;background:#2a1215;color:#ff8a80;padding:10px 14px;border-radius:8px;border:1px solid #5c2b2e;z-index:99999;white-space:pre-wrap;max-height:40vh;overflow:auto';
d.textContent = (d.textContent ? d.textContent + String.fromCharCode(10) : '') +
'[bundle] ' + (e.message || e.type) +
(e.filename ? ' (' + e.filename.slice(0, 60) + ':' + e.lineno + ')' : '');
}, true);
try {
const manifestEl = document.querySelector('script[type="__bundler/manifest"]');
const templateEl = document.querySelector('script[type="__bundler/template"]');
if (!manifestEl || !templateEl) {
setStatus('Error: missing bundle data');
console.error('[bundler] Missing script tags — manifestEl:', !!manifestEl, 'templateEl:', !!templateEl);
return;
}
const manifest = JSON.parse(manifestEl.textContent);
let template = JSON.parse(templateEl.textContent);
// Nested page bundles (iframe targets). Each ships ONCE, in the ROOT
// document's manifest; frame srcs carry an about:blank#<uuid> marker
// instead of a substitutable uuid. Every document mints the blobs for
// its OWN frames — from local text at the root, or text obtained from
// its DIRECT parent over the relay protocol below — so each frame
// navigation stays same-origin with its initiator and a deduped page
// loads at any depth on any host, including a file:// download's
// opaque origins (where a blob minted by one document cannot be
// navigated to by another).
const pageOrderEl = document.querySelector('script[type="__bundler/page_order"]');
const pageOrder = pageOrderEl ? JSON.parse(pageOrderEl.textContent) : [];
const pageSet = new Set(pageOrder);
const pageTexts = {};
// ── Nested-page text protocol (parent-chain relay) ──
// Page bundles ship once, in the ROOT document's manifest. Every
// document mints the blobs for its OWN frames, obtaining page text
// through a strict parent-chain relay: a document only ever SENDS to
// its direct parent (which, for every document the publisher's own
// runtime minted, is another publisher document) and only ACCEPTS
// text from that direct parent; requests from below are honored only
// for the document's own child frames, and the ROOT never sends
// upward at all. That last rule is load-bearing: the artifact host's
// frame-ancestors allows claude.ai (and friends) to frame an
// artifact, so window.top can be a FOREIGN document — a top-addressed
// protocol would hand page uuids to that ancestor and accept forged
// "page text" from it, minting attacker HTML as an artifact-origin
// blob document. The chain never touches the ancestor: requests stop
// at the root, replies are origin-addressed per hop (never '*' when
// an origin exists), and both directions are origin-gated (on https
// a foreign sender's origin can never equal this document's; in
// opaque contexts — file:// — everything serializes as 'null', where
// an embedded EXTERNAL frame still fails the gate by carrying its
// own real origin, and a sandboxed child of such a frame is excluded
// by the own-child-frames check plus uuid unguessability). Shape is
// deny-by-default: one request form, one reply form, no error
// replies.
// window.origin is the SERIALIZED SECURITY origin ('null' in opaque
// contexts such as a sandboxed preview iframe) — location.origin is
// computed from the URL and stays 'https://…' even when the document
// is sandboxed, which would misgate every opaque chain.
const OWN_TARGET = /^https?:[/][/]/.test(window.origin || '')
? window.origin
: '*';
function trustedOrigin(e) {
if (e.origin === window.origin) return true; // https chains; opaque==opaque
// file:// trees: Chrome serializes the file document's origin as
// 'file://' while the blob: documents it mints serialize as 'null',
// so the two legitimate shapes can never string-match — accept the
// pair in both directions. https origins never serialize as either,
// and sender identity (own child frame / direct parent) is checked
// separately.
return (
(e.origin === 'null' &&
(window.origin === 'file://' ||
window.location.protocol === 'file:')) ||
(/^file:/.test(e.origin) && window.origin === 'null')
);
}
// Non-empty only in the root document (nested bundles ship an empty
// island), so it doubles as the root marker: the root answers from
// local text and treats a miss as authoritative — never relaying to
// its (possibly foreign) parent.
const isPageRoot = pageOrder.length > 0;
const blobUrls = {};
const resourceBlobs = {};
const UUID_SHAPE = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
const pendingFrames = {}; // uuid -> [{el, frag}] awaiting text for own frames
const pendingRelays = {}; // uuid -> [childWindow] awaiting text to pass down
function mountPage(el, frag, text) {
const pageBlob = new Blob([text], { type: 'text/html' });
const u = URL.createObjectURL(pageBlob);
resourceBlobs[u] = pageBlob;
el.src = u + frag;
}
function deliverPage(uuid, text) {
const frames = pendingFrames[uuid];
delete pendingFrames[uuid];
if (frames) for (const f of frames) mountPage(f.el, f.frag, text);
const relays = pendingRelays[uuid];
delete pendingRelays[uuid];
if (relays) {
for (const cw of relays) {
try { cw.postMessage({ __bundler_page: uuid, __bundler_text: text }, OWN_TARGET); } catch (err) { /* gone */ }
}
}
}
function ownFrameWindows() {
const out = [];
for (const f of Array.from(document.querySelectorAll('iframe'))) {
try { if (f.contentWindow) out.push(f.contentWindow); } catch (err) { /* detached */ }
}
return out;
}
window.addEventListener('message', function (e) {
const d = e.data;
if (!d || typeof d !== 'object' || !e.source) return;
// Origin gate first; the structural checks below (own-child-frames
// for requests, direct-parent + solicited-uuid for replies, the
// root's never-relay rule) are what hold in fully opaque chains
// where origin strings can't discriminate — a sandboxed root's
// FOREIGN wrapper still fails all three even though both serialize
// as real-vs-'null' rather than matching.
if (!trustedOrigin(e)) return;
if (typeof d.__bundler_need === 'string' && UUID_SHAPE.test(d.__bundler_need)) {
// Requests are honored only for this document's own child frames.
if (ownFrameWindows().indexOf(e.source) < 0) return;
const uuid = d.__bundler_need;
const text = pageTexts[uuid];
if (typeof text === 'string') {
try { e.source.postMessage({ __bundler_page: uuid, __bundler_text: text }, OWN_TARGET); } catch (err) { /* gone */ }
return;
}
if (isPageRoot || window.parent === window) return; // authoritative miss
const waiting = pendingRelays[uuid];
if (waiting) {
if (waiting.indexOf(e.source) < 0) waiting.push(e.source);
return;
}
// Never-answered uuids would otherwise accumulate forever under a
// request spray; past the cap (far above any real page count) new
// relays are dropped rather than remembered.
if (Object.keys(pendingRelays).length >= 128) return;
pendingRelays[uuid] = [e.source];
window.parent.postMessage({ __bundler_need: uuid }, OWN_TARGET);
} else if (
typeof d.__bundler_page === 'string' &&
UUID_SHAPE.test(d.__bundler_page) &&
typeof d.__bundler_text === 'string'
) {
// The shape test is load-bearing beyond tidiness: the solicited
// check below uses the in operator, which matches prototype keys
// ('__proto__', 'constructor') — an unshaped key would reach
// deliverPage and throw, painting the error banner.
// Text is accepted only from the direct parent — the document
// that minted this one.
if (e.source !== window.parent || window.parent === window) return;
if (!(d.__bundler_page in pendingFrames) && !(d.__bundler_page in pendingRelays)) return;
deliverPage(d.__bundler_page, d.__bundler_text);
}
});
const uuids = Object.keys(manifest);
setStatus('Unpacking ' + uuids.length + ' assets...');
await Promise.all(uuids.map(async (uuid) => {
const entry = manifest[uuid];
try {
const binaryStr = atob(entry.data);
const bytes = new Uint8Array(binaryStr.length);
for (let i = 0; i < binaryStr.length; i++) bytes[i] = binaryStr.charCodeAt(i);
let finalBytes = bytes;
if (entry.compressed) {
if (typeof DecompressionStream !== 'undefined') {
const ds = new DecompressionStream('gzip');
const writer = ds.writable.getWriter();
const reader = ds.readable.getReader();
writer.write(bytes);
writer.close();
const chunks = [];
let totalLen = 0;
while (true) {
const { done, value } = await reader.read();
if (done) break;
chunks.push(value);
totalLen += value.length;
}
finalBytes = new Uint8Array(totalLen);
let offset = 0;
for (const chunk of chunks) { finalBytes.set(chunk, offset); offset += chunk.length; }
} else {
console.warn('DecompressionStream not available, asset ' + uuid + ' may not render');
}
}
if (pageSet.has(uuid)) {
pageTexts[uuid] = new TextDecoder().decode(finalBytes);
return;
}
if (FONT_MIME.test(entry.mime) && MIME_TOKEN.test(entry.mime)) {
// Strict artifact hosts allow font-src data: but not blob:.
blobUrls[uuid] = 'data:' + entry.mime + ';base64,' +
(entry.compressed ? toBase64(finalBytes) : entry.data);
} else {
const blob = new Blob([finalBytes], { type: entry.mime });
blobUrls[uuid] = URL.createObjectURL(blob);
resourceBlobs[blobUrls[uuid]] = blob;
}
} catch (err) {
console.error('Failed to decode asset ' + uuid + ':', err);
const blob = new Blob([], { type: entry.mime });
blobUrls[uuid] = URL.createObjectURL(blob);
resourceBlobs[blobUrls[uuid]] = blob;
}
}));
const extResEl = document.querySelector('script[type="__bundler/ext_resources"]');
const extResources = extResEl ? JSON.parse(extResEl.textContent) : [];
const resourceMap = {};
for (const entry of extResources) {
if (blobUrls[entry.uuid]) resourceMap[entry.id] = blobUrls[entry.uuid];
}
// Artifact-host CSP (connect-src 'self') refuses fetch() of blob: URLs —
// consumers read these Blobs directly. Survives the swap like the error sink.
window.__resourceBlobs = resourceBlobs;
setStatus('Rendering...');
// Page uuids have no blob here — they ride inside about:blank#<uuid>
// frame markers the pass below resolves after the swap.
for (const uuid of uuids) {
if (pageSet.has(uuid)) continue;
template = template.split(uuid).join(blobUrls[uuid]);
}
// Strip integrity + crossorigin — blob URLs from a file:// document inherit
// a null origin, so crossorigin forces a CORS fetch that SRI then rejects.
// The manifest bytes are ours; SRI protects against CDN compromise, not this.
// appifacts: only on tags now pointing at OUR blob: URLs. A reference the
// bundler left live-fetched (an absolute x-import, a runtime dependency it
// could not embed, an authored external script) keeps the pin it was
// written with: that fetch is exactly what SRI protects. (A crossorigin
// on a parent whose child carries the blob — <video crossorigin><source>
// — is left alone too; the attribute is then the author's to drop.)
template = template.replace(/<[a-z][^>]*>/gi, (tag) => /\s(?:src|href|srcset|poster|data|xlink:href)="blob:/i.test(tag) ? tag.replace(/\s+integrity="[^"]*"/gi, '').replace(/\s+crossorigin="[^"]*"/gi, '') : tag);
const resourceScript = '<script>window.__resources = ' +
JSON.stringify(resourceMap).replace(/<\//g, '<\\/') +
';</' + 'script>';
// Inject after <head> so the DOCTYPE stays first; prepending the script
// would push the parser into quirks mode. DOMParser always emits a <head>
// (synthesizing one if the source HTML omitted it) but may carry
// attributes through, so match the full opening tag. slice() rather than
// replace() keeps us clear of $-pattern substitution in resourceScript.
const headOpen = template.match(/<head[^>]*>/i);
if (headOpen) {
const i = headOpen.index + headOpen[0].length;
template = template.slice(0, i) + resourceScript + template.slice(i);
}
// Parse the template and swap the root element. Scripts inserted via
// DOMParser/replaceWith are inert per spec — re-create each with
// createElement so they execute, awaiting onload for src scripts to
// preserve ordering (React before ReactDOM before Babel before text/babel).
const doc = new DOMParser().parseFromString(template, 'text/html');
document.documentElement.replaceWith(doc.documentElement);
// Resolve page-frame markers: mint this document's own blob for each
// framed page — from local text when this is the root, otherwise by
// asking the direct parent over the relay protocol above. An
// unanswered request leaves the frame on about:blank, the same
// degraded state as a refused embed.
{
const MARKER = /^about:blank#([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})(#.*)?$/;
const needed = [];
for (const el of Array.from(document.querySelectorAll('iframe'))) {
const m = MARKER.exec(el.getAttribute('src') || '');
if (!m) continue;
const text = pageTexts[m[1]];
if (typeof text === 'string') {
mountPage(el, m[2] || '', text);
} else if (window.parent !== window && !isPageRoot) {
const had = pendingFrames[m[1]];
(pendingFrames[m[1]] = had || []).push({ el: el, frag: m[2] || '' });
if (!had) needed.push(m[1]);
}
}
for (const uuid of needed) {
window.parent.postMessage({ __bundler_need: uuid }, OWN_TARGET);
}
}
const dead = Array.from(document.scripts);
for (const old of dead) {
const s = document.createElement('script');
for (const a of old.attributes) s.setAttribute(a.name, a.value);
s.textContent = old.textContent;
// text/babel scripts with a src: read and inline. transformScriptTags
// does XHR against the src, but blob:null/ from a file:// origin is
// silently dropped. Inlining makes it a plain inline babel script,
// which transformScriptTags handles unconditionally.
if ((s.type === 'text/babel' || s.type === 'text/jsx') && s.src) {
const pre = resourceBlobs[s.src.split('#')[0]];
s.textContent = pre ? await pre.text() : await (await fetch(s.src)).text();
s.removeAttribute('src');
}
const p = s.src ? new Promise(function(r) { s.onload = s.onerror = r; }) : null;
old.replaceWith(s);
if (p) await p;
}
// Babel standalone auto-transforms type=text/babel on DOMContentLoaded,
// which fired before we swapped the document. Trigger manually if present.
if (window.Babel && typeof window.Babel.transformScriptTags === 'function') {
window.Babel.transformScriptTags();
}
} catch (err) {
setStatus('Error unpacking: ' + err.message);
console.error('Bundle unpack error:', err);
}
});
</script>
<script type="__bundler/manifest">
{"d8b6a7d7-6db8-4784-abe3-0772b69b3705":{"mime":"image/png","compressed":false,"data":"iVBORw0KGgoAAAANSUhEUgAAAoUAAAGYCAYAAAA0iZgMAAAACXBIWXMAAA7DAAAOwwHHb6hkAAAAGXRFWHRTb2Z0d2FyZQB3d3cuaW5rc2NhcGUub3Jnm+48GgAAIABJREFUeJzs3XmcjeX7B/DP9Sxnnf3MMGNNlkqSbCFhNiSJkiKl+iJLdiLabJUl1A9JSmkRLdpk36LsW0J2kt0MZp+zPPfvj6G0MctzznOW6/16fV/fjHOu+5oxM+c693LdJIQAY4wxxhgLbZLRCTDGGGOMMeNxUcgYY4wxxrgoZIwxxhhjXBQyxhhjjDEAitEJMMaCF5VvZDXnm8t6ICdIgsprmognoAwIpQUhlgRiAEQKIAKAiYisBc8UBAEzrn7jKuCEBA9AAkK4BZBLoExAZAqBTBDSCeJ3QdJvJMRZTRNnJSFOujTTcXFpcboRnz9jjAUS4tPHjLGSoKi7o1VFrQZQNYK4WSOqRYJuAiEBGkyCkEsCHkEkEYQVgOrVhARcgpBPgBuABIIZggCI00LgGAF7QbRTkPjVTZ694uyq017NhzHGAgQXhYyxQiEiMsc1qyw8ci0Q1RUCjQWhOpGwQSBPQEgEssNft6UIAPTHH3IhKB8SLABpENphCGk9oK2HJra5LordQqxyG5gtY4z5HBeFjLF/RfEt7IrbfScENSVCSwA1BcENAY0AOwDZ6Bz1JoAsABoJWEA4BIgVBKx0KsqP4vSSs0bnxxhj3sRFIWMMAECVEi1KhtSEBFpDohYQohKIciBgA3l5yddvkQYhsiCRWQiRToSVpNE3TpdYITKWpxmdHWOM6YmLQsZCmDk29RYhtBaQ6EEIqgchckEIRxDOAupFgHIIQgHEcQh8JYT0lftC9Hoh5nuMzo0xxkqCi0LGQggRkepIqgeBjoLoERDCSUAGYDE6twDlhkAOSBCIFpKgT51m6zJx4pscoxNjjLGi4qKQsSBHRKREp9xFEh4VEB0AqARYwS2p9CYAZAIwC8IySdAsZ7S6WBz4Pt/oxBhjrDC4KGQsSFlKJVYRmvykRuhKQlgBsoGXhX2loEAkyID4WkB6x31++RrBv3AZY36Mi0LGggjFNQ5XYekEgd4AqoBIghBmo/MKbaQBIlsAORCYZpKkd7PPLT1pdFaMMfZ3XBQyFgTMjpTqAhgECY+IgpYxYUbnxP5VHgiAEBuFoNfc6SuW8OwhY8xfcFHIWIAiSlRMMXJbQRgqgOpUcFNIiLaOCUhZAsgkIY1zSTnviXPrMo1OiDEW2rgoZCzAUNk2NjUvr5uQxAgSwgIg3OicWHEJCKJsAggC7yvC82pO2qrfjc6KMRaauChkLEBQRIpDUdGXCP0FBBGIi8GgQk6Q0AB8JUF7Oe/cyn1GZ8QYCy1cFDLm5ygiMVZVlBcha10BAgSsRufEvEjADRIuQdIqSfM8m5+2crfRKTHGQgMXhYz5KYpJjTRJ2jBB1AcCEsDFYIjxAHAJYIksSUPyzi49YHRCjLHgxkUhY36GyraxKfk5/QliGCDJgLAZnRMzlAcQLpC0QBHakJzzK04YnRBjLDhxUciYnyi4gi7xMQFpEggWErAbnZPeVFWB1WqCRBI8Hg1utwf5TidMqgJ7mA3hditURYHZrEJVFaiKDJPZBJNJQXZ2HjSt4Dkulxu5eU5kZeUiJzcXublOCAiYTSoURYFEgNPtRn6+Cx6PZvSnrRcnCG4SeMOpyGPF6SXZRifEGAsuXBQy5gdUR3IjInoHQAUEeI9Bq8UMs1lFfp4TIKBsmTiUL1caVSoloErlciiXEIeyZWJRKi4aUZFhiI4KhyxLJR433+lCenoGTp1Jw6kz6ThzJh0nTp3H/oPH8ev+3/DbibPIzMiB1WoGSUB2dl5AFowClAMgnwQGu9Ibvy/ES4H3STDG/BIXhYwZyOZILOciZRogUi7fR0xG51RoBITZrYAA3G43bq5WEXfcXg131KyCm6qWx81VKyAh3mF0ln+R73ThyNFT2L33CH7efQgbtvyK3XuP4MLFTNisZuTm5sPpchudZuEQsiBwDCQed55bsc3odBhjgY+LQsYMQDRSUmPW9QWJMQBMCICm04osw263IDfXico3lsHdDW9Dw/q3ou4dN6Nq5XKQpMCpZ/8uIzMHW3fsw/pNe7Bq7Xbs2HkALo8HgEBOTr7R6V2DEADlQuATl+YZIi6sumh0RoyxwMVFIWM+ZopOuh2y9AmAioD/7hskAsLD7MjLy0eVyuXQKvVOJDWtjUZ31oDNGvzXKR88fAKr1+3Ad4vXY936n6FpAq7L+xT9UJ4A8iSBHvlpy+cZnQxjLDBxUciYj1ClRIuaIY8FoQdAFkCUfCOdzlRFgcmsQJYktGreAB3aNUOzu+8oWCYOcb/sOYLvlmzA51+vwv4Dx2GxmHHxUpbRaf0N5Qho69yS1kWcXXXa6GwYY4GFi0LGfMAUl1wbgr4UhDgS8KsWMyZVhqKqcMREoGP7JLS9927UqVUNRIG7HOxt59MuYcmKTZjz6TKs37gLiqogKyvX6LSucALIB+Q+zvNLPjA6GcZY4OCikDEvIkpUlFj5RRJicMHsoH8cJJFkGXarCSazCY91SMVjjzTHbbfeaHRaASkjMwffLvoRsz9chE3bfoUkScjJzTM6LQDIFoS17nx0FhnL04xOhjHm/7goZMxLLHEp1TRBXwCiEvxk76DdboGmCTxwXxN0ffxeNLrzVp4R1NH5tEv45LPlmPrOApw/n4G8/Hy43R7jEhLkFIRsAB1c55ctNy4Rxlgg4KKQMS8wxyV1EkKaCZDV6L2DsizDYlFRJj4W/Xu1xyMPJiE8zK9WsIPSlu37MOPdb/DZ16tBBENPMQsghzR6z3XhwkAhtvjlSRnGmPG4KGRMR5cPk7wFwkMQsBu5WGw2myARcF+ruzCg10OoU6uaccmEsEsZ2Xjvw+8xaepnyMnNRaZBew8FkEOEg4omt85JW3LckCQYY36Ni0LGdGIplVhF0+TvBKGckVfU2ewWkAC6Pn4vBvZ5GAmlY4xKhV1F0wQWLduAUa/NwYHDJ5CTW3Btn4+5AcoRAh1cacuW+Hpwxph/46KQMR0opZJaSZo07/JysWxEDna7FWaTgmEDOqFrl9bcRsaP/bjhF7w49j1s3bkfeXlOnxeHApQjQUxxpq14XvCLAGPsMi4KGSshtVTKcGgYQTCm1YzVZoZJUfDi0C7o/uR9MJv8/nIUdtnPvxzCsJEzsX7THuTm5EHz5e9jQrYA1rplub04vSTbdwMzxvwVF4WMFRNVSrQomdLHBGoBA04XW60FxeCwgY+iV7f7YbUE/y0jwWrztl8xaMR07Np9GNk5PmxnQ8iFoBOyR0vJvbDimO8GZoz5Iy4KGSsGimuZoAr3cgFRiUA+XadVVQWqIqN3t7Z4blBnXiYOIqvX7cAzg9/AyVPnkJXts+LQI4BMCLRypS1f76tBGWP+h4tCxorIEpd0kwfSGtIQA4LP1mqJAIvFjKQmtfHmuD4oX66Ur4ZmPqRpAnPmLsbQl2bC6XIh22fFocgl4NH88ysW+GhAxpif4aKQsSIwOVIbgLAEEGEAfNZ/MMxuQbkycZj55hA0qFfdV8MyA2Vm5WD0+DmY+d63yHO6fHIYRQA5BDznPL/8Ta8PxhjzO1wUMlZIlrikNpqQPoEP9w8qigyzScXLzz2JZ7q3gywb2gebGeDX/b+hy9Ov4sDh475aUs6GEDOcaSsG+2Iwxpj/4KKQsUIwxTZ/AtCmwWcnjAXsdisa1K2OWf83BGXLxPlmWOaXhBCY/dEiDHn+LeQ73XC6vH4pSTYEPnWlr+jGLWsYCx1cFDJ2HSZHandBYrKvWs6YTSosFhPeeXMI2rZu7IshWYA4c+4CnujxKjZs3uOLU8rZILHQdd7RSYj5Bl7gzBjzFS4KGbsGU2xqH0HiNRK+KQjtNgvq162OD2cOR6nYKF8MyQLQnLlLMOC5acjLd8LlcntvIBK5AtIq9/kLbfnOZMaCHxeFjP0HNS55MASN9MUMoSxLsJhNmDC6J7p2udfbw7Eg8PuJc3jkqVHYvfeIV2cNBSgHhFXu8+62QqzyYgXKGDMaF4WM/QuTI6UfCGPhg0MlVpsZN5SPx1efjEGligneHo4FEU0TGD1hDiZPnY+c3HzvDUTIFkIscac5OvBSMmPBi4tCxv7GFNv8CUHaNK8vGQvAZrfg4XaJeHNCX76ejhXbug278NDjLyErKxf5Tq+t8maD8J3r/N2dhHjJt5c1M8Z8gotCxq5ijk16QIP0obeXjCWJYLWYMWPKQDz8QJI3h2IhIi09A+0efR67dh9Cdo7XZg2zIfCxM235094agDFmHC4KGbtMdaSkEuEreLkgNJtNiIuNxKLPx+OmquW9ORQLMR6PhkEjpuP9TxYjx3v7DLMFiQmucytGemsAxpgxuChkDIApLrk2BP0ACDtAXhvHbrWg5m2V8fXcsYiKDPPaOCy0zf18BXoNmIzsXO8UhgLIIUH9nWnL3vHKAIwxQ3BRyEKezdGivIu0HQQRDS9WhHabBQ/e3xQzJg+EosjeGoYxAMCW7ftwT/uhyMrKgcej/xZAAeQIog7uc8sW6h6cMWYILgpZSCNHqwhVcm6DhoogKN4ax2o1Y9Twp9Cv54PeGoKxfzh67DSS7x+Is2cveOsAShZpuCs/ffnP3gjOGPMtLgpZyCJKVNRYeRUE6gKweGscm9WM96YNxQNtmnhrCMb+04WLmWjVfij2/HoMuXm6H0ARguicWxI1xZnlZ/QOzhjzLcnoBBgzihojTYeg2vBSQUgEhNksWPDJGC4ImWGio8KxauEUNLyzBmxWs97hiTQtWvVgGVVK9NobK8aYb3BRyEKSKbb5YyDqBAivnDSWJEJEuB3LvpmExLvv8MYQjBWaxWzCLine truncated
</script>
<script type="__bundler/ext_resources">
[{"id":"https://cdn.jsdelivr.net/npm/react@18.3.1/umd/react.production.min.js","uuid":"bdd67b3c-1f89-443b-843a-65fde192a9fd"},{"id":"https://cdn.jsdelivr.net/npm/react-dom@18.3.1/umd/react-dom.production.min.js","uuid":"fe7f04f2-e97a-40e1-9ad3-034e63dbf740"}]
</script>
<script type="__bundler/page_order">
[]
</script>
<script type="__bundler/template">
"<!DOCTYPE html>\n<html lang=\"en\"><head>\n<meta charset=\"utf-8\">\n<title>Felhom system poster<\u002Ftitle>\n<script src=\"7425d086-bbee-40a4-a354-fbfba43c8a3d\"><\u002Fscript>\n<meta name=\"hz:slide-selector\" content=\"[data-canvas-width]\"><meta name=\"hz:canvas-width\" content=\"3200\"><meta name=\"hz:canvas-height\" content=\"2262\"><\u002Fhead>\n<body>\n<x-dc>\n<helmet>\n<link rel=\"preconnect\" href=\"https://fonts.googleapis.com\">\n<style>/* cyrillic-ext */\n@font-face {\n font-family: 'Plus Jakarta Sans';\n font-style: normal;\n font-weight: 400;\n font-display: swap;\n src: url(https://fonts.gstatic.com/s/plusjakartasans/v12/LDIoaomQNQcsA88c7O9yZ4KMCoOg4Ko70yyygA.woff2) format('woff2');\n unicode-range: U+0460-052F, U+1C80-1C8A, U+20B4, U+2DE0-2DFF, U+A640-A69F, U+FE2E-FE2F;\n}\n/* vietnamese */\n@font-face {\n font-family: 'Plus Jakarta Sans';\n font-style: normal;\n font-weight: 400;\n font-display: swap;\n src: url(https://fonts.gstatic.com/s/plusjakartasans/v12/LDIoaomQNQcsA88c7O9yZ4KMCoOg4Ko50yyygA.woff2) format('woff2');\n unicode-range: U+0102-0103, U+0110-0111, U+0128-0129, U+0168-0169, U+01A0-01A1, U+01AF-01B0, U+0300-0301, U+0303-0304, U+0308-0309, U+0323, U+0329, U+1EA0-1EF9, U+20AB;\n}\n/* latin-ext */\n@font-face {\n font-family: 'Plus Jakarta Sans';\n font-style: normal;\n font-weight: 400;\n font-display: swap;\n src: url(https://fonts.gstatic.com/s/plusjakartasans/v12/LDIoaomQNQcsA88c7O9yZ4KMCoOg4Ko40yyygA.woff2) format('woff2');\n unicode-range: U+0100-02BA, U+02BD-02C5, U+02C7-02CC, U+02CE-02D7, U+02DD-02FF, U+0304, U+0308, U+0329, U+1D00-1DBF, U+1E00-1E9F, U+1EF2-1EFF, U+2020, U+20A0-20AB, U+20AD-20C4, U+2113, U+2C60-2C7F, U+A720-A7FF;\n}\n/* latin */\n@font-face {\n font-family: 'Plus Jakarta Sans';\n font-style: normal;\n font-weight: 400;\n font-display: swap;\n src: url(\"e228819f-c09e-4d10-9162-36e030c24619\") format('woff2');\n unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+0304, U+0308, U+0329, U+2000-206F, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD;\n}\n/* cyrillic-ext */\n@font-face {\n font-family: 'Plus Jakarta Sans';\n font-style: normal;\n font-weight: 500;\n font-display: swap;\n src: url(https://fonts.gstatic.com/s/plusjakartasans/v12/LDIoaomQNQcsA88c7O9yZ4KMCoOg4Ko70yyygA.woff2) format('woff2');\n unicode-range: U+0460-052F, U+1C80-1C8A, U+20B4, U+2DE0-2DFF, U+A640-A69F, U+FE2E-FE2F;\n}\n/* vietnamese */\n@font-face {\n font-family: 'Plus Jakarta Sans';\n font-style: normal;\n font-weight: 500;\n font-display: swap;\n src: url(https://fonts.gstatic.com/s/plusjakartasans/v12/LDIoaomQNQcsA88c7O9yZ4KMCoOg4Ko50yyygA.woff2) format('woff2');\n unicode-range: U+0102-0103, U+0110-0111, U+0128-0129, U+0168-0169, U+01A0-01A1, U+01AF-01B0, U+0300-0301, U+0303-0304, U+0308-0309, U+0323, U+0329, U+1EA0-1EF9, U+20AB;\n}\n/* latin-ext */\n@font-face {\n font-family: 'Plus Jakarta Sans';\n font-style: normal;\n font-weight: 500;\n font-display: swap;\n src: url(https://fonts.gstatic.com/s/plusjakartasans/v12/LDIoaomQNQcsA88c7O9yZ4KMCoOg4Ko40yyygA.woff2) format('woff2');\n unicode-range: U+0100-02BA, U+02BD-02C5, U+02C7-02CC, U+02CE-02D7, U+02DD-02FF, U+0304, U+0308, U+0329, U+1D00-1DBF, U+1E00-1E9F, U+1EF2-1EFF, U+2020, U+20A0-20AB, U+20AD-20C4, U+2113, U+2C60-2C7F, U+A720-A7FF;\n}\n/* latin */\n@font-face {\n font-family: 'Plus Jakarta Sans';\n font-style: normal;\n font-weight: 500;\n font-display: swap;\n src: url(\"e228819f-c09e-4d10-9162-36e030c24619\") format('woff2');\n unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+0304, U+0308, U+0329, U+2000-206F, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD;\n}\n/* cyrillic-ext */\n@font-face {\n font-family: 'Plus Jakarta Sans';\n font-style: normal;\n font-weight: 600;\n font-display: swap;\n src: url(https://fonts.gstatic.com/s/plusjakartasans/v12/LDIoaomQNQcsA88c7O9yZ4KMCoOg4Ko70yyygA.woff2) format('woff2');\n unicode-range: U+0460-052F, U+1C80-1C8A, U+20B4, U+2DE0-2DFF, U+A640-A69F, U+FE2E-FE2F;\n}\n/* vietnamese */\n@font-face {\n font-family: 'Plus Jakarta Sans';\n font-style: normal;\n font-weight: 600;\n font-display: swap;\n src: url(https://fonts.gstatic.com/s/plusjakartasans/v12/LDIoaomQNQcsA88c7O9yZ4KMCoOg4Ko50yyygA.woff2) format('woff2');\n unicode-range: U+0102-0103, U+0110-0111, U+0128-0129, U+0168-0169, U+01A0-01A1, U+01AF-01B0, U+0300-0301, U+0303-0304, U+0308-0309, U+0323, U+0329, U+1EA0-1EF9, U+20AB;\n}\n/* latin-ext */\n@font-face {\n font-family: 'Plus Jakarta Sans';\n font-style: normal;\n font-weight: 600;\n font-display: swap;\n src: url(https://fonts.gstatic.com/s/plusjakartasans/v12/LDIoaomQNQcsA88c7O9yZ4KMCoOg4Ko40yyygA.woff2) format('woff2');\n unicode-range: U+0100-02BA, U+02BD-02C5, U+02C7-02CC, U+02CE-02D7, U+02DD-02FF, U+0304, U+0308, U+0329, U+1D00-1DBF, U+1E00-1E9F, U+1EF2-1EFF, U+2020, U+20A0-20AB, U+20AD-20C4, Line truncated
</script>
<div style="position: absolute; left: 72px; top: 2200px; width: 2400px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: left;">Sources: architecture documents 01, 06, 07, 11 and the total-loss runbook in the felhom.eu repository. Names only; no secret values.</div>
</div>
</body>
</html>
</html>
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,173 @@
# SPIKE — who is using the upload line, and does the app explain it? (2026-10-23 night)
Operator ruling 2026-10-23 19:22 (`CONTEXT.md`): measure the line; on a sustained high upload attribute it to an app
and ask the app; a mismatch is the alarm; the same shape for CPU; qBittorrent gets a default upload limit from the
measured line, household changes respected. Brief Part 2.1. Design written from these numbers: §6 here and
`architecture/08-alarm-ladder.md` §9 [DESIGN — proposed]. Evidence and tools: `SPIKE-upload-watch-2026-10-23/`.
Read first: `08-alarm-ladder.md` (R-388, §8.1), `07-backup-architecture.md` (the night chain: W = 02:30 local, the
off-site push after it, the proof at 03:30 UTC), `15-media-library.md`, `13-threat-model.md` (attacker j),
`audits/plex2shell-felhom-impact-2026-10-23/README.md` (the 1.5 TB Plex upload). Controller: `internal/metrics/`
(`container_metrics` — one row per container per minute, 30 days), `internal/monitor/healthcheck.go` (the instant
whole-box CPU warning), `internal/mediawire/` (the app clients and their credentials, `09` §3 decision 248).
Machines: the three test boxes READ ONLY (their stored metrics; their off-site record; tonight a one-second read of
the controller's own counter during the real push); scratch 9202 on demo-hp (throwaway qBittorrent, Transmission, Plex,
Jellyfin through the product's deploy); a throwaway receiver LXC **9471 `cc-uw-recv`** on demo-hp's scratch storage
(never the thin pool). Every flood went 9202 → 9471 on demo-hp's own bridge; nothing went toward the internet.
## 1. Normal levels (stored metrics, three boxes)
History: **demo-hp** 2026-09-23 02:00 → 10-23 18:09 UTC (30.7 d, 998 216 rows); **demo-felhom** (the N100) same span
(30.7 d, 268 120 rows); **Tester 1** 2026-10-04 19:41 → 10-23 18:08 (18.9 d, 294 433 rows). Per app: upload from the
`net_tx_bytes` deltas (one-minute samples), CPU in percent of ONE core. Full tables: `s1-levels-per-app.txt`.
| box | the busiest uploaders (p50 / p95 / max Mbit/s) | the busiest CPU (p95 / max % of a core) |
|---|---|---|
| demo-hp | qbittorrent 0.02 / 1.2 / 4.0; immich-server 0.03 / 0.13 / 243 (one minute); felhom-controller 0 / 0.003 / 17.8 (the push) | grimmory 272 / 305 (a scan, 94 samples); immich-ML 32 / 548; romm 3 / 442; paperless 0.3 / 404 |
| demo-felhom | immich-server 0.67 / 5.95 / 12.9 (32 samples); jellyfin 0 / 0.85 / 2.9 | immich-ML 295 / 296 (an import); onlyoffice 131 / 148; paperless 137 / 140 |
| Tester 1 | qbittorrent 0.007 / 0.13 / 42.4; transmission 0.001 / 0.013 / 57.4; controller 0 / 0.003 / 37.5 | paperless 0.7 / 508; controller 46 / 423; plex 4.3 / 198 |
**How long a normal high lasts** (`s1-longest-runs.txt` — the longest run of consecutive minutes): upload ≥ 5 Mbit/s:
**3 min** at most (qBittorrent, Tester 1); ≥ 1 Mbit/s: 9 min. CPU ≥ 150 %: **7 min** at most (Immich machine learning);
≥ 80 %: 13 min (Kimai). Busiest upload hours (UTC): demo-hp 03h, 02h; demo-felhom 16h, 05h; Tester 1 07h, 18h — no fixed
pattern beyond the night push. For scale: DooPlex's hacked Plex sent ~474 GB in one day (≈ 44 Mbit/s on average, for
hours) and used ~10 cores.
`felhom-samba` runs with the host network: its counter is always 0 (no own network namespace) — the watch cannot see it.
## 2. The line without a test
**What the box already sees:** the off-site push (restic, inside the controller container) is the box's only big
upload of its own. At the stored one-minute sample (`s2-controller-push-per-night.txt`): the controller's best five
minutes per night exceeded 2 Mbit/s on **2 of 31 nights** on demo-hp (5.1 and 6.9 Mbit/s; best single minute 17.0),
**1 of 19** on Tester 1 (7.6; best minute 37.5) and **0 of 31** on demo-felhom. The off-site record keeps the push's
duration (33 s on demo-felhom, 1 m 8 s on Tester 1, 3 m 19 s on demo-hp on 2026-10-23) and **no byte count**.
**Conclusion:** at a one-minute sample the push shows the night's DATA SIZE (a few MB, incremental), not the line. Not
enough as it stands. Two changes make it usable: sample once a SECOND during the push (the controller knows when its
push runs and can read its own counter), and let a night count only when its best 10 seconds moved ≥ 10 MB. Even then
it is a **lower bound** (restic over SFTP to the Storage Box may not fill the line; the family uploads too).
**Tonight's one-second read of the real push** (2026-10-24 ~02:16Z, three boxes, read only): §2.1 below.
### 2.1 One-second samples of the real push
Read 2026-10-24 02:08–02:30Z, once a second, the controller container's own transmit counter (the counter the watch
reads: `eth0` in the controller's network namespace), on the three test boxes, read only
(`SPIKE-upload-watch-2026-10-23/night-push-1s-<box>.txt`; the reader `tools/nightsample.sh`).
| Box | Sent in the 22 min | Best 10 s | 1-second peak | Counts (≥ 10 MiB in the best 10 s)? |
|---|---|---|---|---|
| demo-hp | 19.3 MB | **12.6 MB in 10.1 s = 10.0 Mbit/s** (02:17:32Z) | 70.3 Mbit/s | **yes** → line „at least 10 Mbit/s" |
| demo-felhom | 3.3 MB | 3.2 MB = 2.5 Mbit/s (02:15:22Z) | 13.7 Mbit/s | no |
| Tester 1 | 3.6 MB | 3.5 MB = 2.7 Mbit/s (02:15:40Z) | 14.7 Mbit/s | no |
**What it says:** one night in three counted. The push is a burst of a few seconds (3–9 seconds above 100 kB/s), so
even the counting night gives a low floor: 10 Mbit/s, while one second of the same push reached 70 Mbit/s. The design
keeps the best counting night of the last 14, so the line rises only when a bigger night comes. **Consequence for the
qBittorrent limit (R-1245):** 50 % of tonight's floor is ~5 Mbit/s on demo-hp — likely far below its real line. This is
the known lower-bound limit (R-1247), now measured; it is written into R-1245 so the operator decides with it.
## 3. The line with a test — NOT run tonight
| Candidate | Ours? | Terms | Cost per test | Tonight |
|---|---|---|---|---|
| ep0 (a receiver on our Hetzner box) | yes | ours | 10 s × line ≈ 125 MB at 100 Mbit/s | **not built** — a change on ep0 is forbidden tonight |
| the box's own Storage Box sub-account, through the same transport as the push | yes (Hetzner, unlimited traffic) | Hetzner's — ours | same | not tried — a test file in the account that holds the restic repository needs its own design (the repo check, the quota) |
| Cloudflare R2 (where the ISO lives) | yes | ours | same + R2 class-A writes | not used — its write key is the operator's ISO-publish key; a test object in a public bucket is wrong |
| public speed tests (speed.cloudflare.com, LibreSpeed servers) | no | **not read** | — | not used (the brief: no third-party test without its terms) |
A quiet hour: never 02:30–05:30 local (dumps 02:30, second drive, off-site push, the 03:30 UTC proof). The family shares
the line, so take the highest of several runs. **Not needed for the first build** — §6 works with the push's lower
bound and an absolute floor; an active test is a decide row (R-1247).
## 4. Can the box ask each app? (measured on 9202 with throwaway installs; a private torrent 9202 → 9471)
Set-up: qBittorrent 5.2.4 and Transmission 4.1.3 installed through the product (`POST /api/stacks/<n>/deploy`; the
first try was refused BEFORE anything was made — the data place must be the drive itself; two empty helper folders
removed). The same 2 GiB file (an AES-CTR keystream of a fixed key) on both sides; a private torrent; a 40-line HTTP
tracker on 9471 (`tools/tracker.py`); `transmission-daemon` on 9471 as the leecher. The admin passwords were read from
the containers at use and never printed.
| App | Its own number | Measured | Limit | Household's limit visible? |
|---|---|---|---|---|
| **qBittorrent** | `/api/v2/transfer/info` → `up_info_data` (cumulative), `up_info_speed` | seeding: app **131.1 Mbit/s** vs container **134.6–140.5** → **×1.03–1.07** (`s4-qb-seed-nolimit.jsonl`) | `transfer/setUploadLimit` (bytes/s) = `preferences.up_limit`; stored in KiB/s (2 500 000 → **2 499 584**); survives a restart (`Session\GlobalUPSpeedLimit=2441`); measured: ~20 Mbit/s held (`s4-qb-seed-limit20.jsonl`) | yes: `transfer/uploadLimit` ≠ the value the box stored |
| **Transmission** | RPC `session-stats.current-stats.uploadedBytes`, `uploadSpeed` | seeding: app 487.1 / 390.3 / 115.8 vs container 487.6 / 390.7 / 117.9 Mbit/s → **×1.001–1.02** (`s4-tr-seed-nolimit.jsonl`) | `session-set speed-limit-up` (kB/s) + `-enabled` (read back); **its rate under the limit was NOT observed** — the re-added leecher got no connection in 2 minutes | yes: `speed-limit-up-enabled` + value |
| **Plex** | `/status/sessions`, `/transcode/sessions` (`MediaContainer.size`) | unclaimed 1.43.5: both 200 without a token from the docker network, `{"size":0}`; a claimed server needs its PlexOnlineToken (read at use from Preferences.xml, as mediawire does — no new secret) | — | — |
| **Jellyfin** | `/Sessions` | **401** without a key; the key named `felhom` exists only when the household made it (mediawire's „one household step") — the box cannot create one at install without the admin login it does not have | — | — |
| Nextcloud, Immich | **no upload-explaining number found** (not measured further); Immich's job queue needs an API key the box does not hold | — | — | → the plain rule (§6) |
**The fake hacked app** (code inside qBittorrent's own container, `docker exec … python3` → 9471:5001, Plex2Shell's
shape): the container sent **~10 100 Mbit/s** (LAN, one bridge) while qBittorrent reported **0** (`s4-qb-flood.jsonl`).
**The counter itself:** the Docker Engine API's one-shot stats (`networks.*.tx_bytes`) equal the kernel's counter in
the container's network namespace to the byte; `docker stats`' text (what `internal/metrics` stores) has three
significant digits — ~10 MB steps at a GB total, ~1.3 Mbit/s at a one-minute sample. The watch reads the Engine API.
**A limit the hacked app controls:** an attacker with code in the app's container can also answer its API. The check
catches an attacker who does not know about it; one who does could lie. §5 is the answer to that.
## 5. A universal upload ceiling — spike only
A helper container sharing the app's network namespace (`docker run --rm --network container:qbittorrent --cap-add
NET_ADMIN alpine:3.20` + `iproute2-tc`) added `tc qdisc … root tbf rate 20mbit burst 64kb latency 400ms` on the app's
`eth0` **inside the unprivileged 9202 guest**. The same flood then ran at **19.9–20.1 Mbit/s** (`s4-qb-flood-tbf.jsonl`),
qBittorrent's API kept answering, and `tc qdisc del` restored it. Costs: a helper image with `tc` (a new external image,
or `tc` shipped in the controller image), `NET_ADMIN` for that helper, re-applied after every app restart (the qdisc
dies with the namespace), upload only. **Not built** (brief). It would cap a hacked app's flood whatever its API says.
## 6. The design (numbers from §1–§5; full text `08-alarm-ladder.md` §9)
1. **Line:** one-second samples of the controller's own sent bytes during the off-site push; the best ≥ 10-second rate
per night; a night counts when that window moved ≥ 10 MB; the line = the highest counting night of the last 14;
settings.json `upwatch.line`; shown on the system settings page as „legalább N Mbit/s" („még mérjük" until known).
2. **Trigger:** an app (its containers summed, exact counters, once a minute) sends ≥ **max(5 Mbit/s, 25 % of the line)**
for **15 minutes** — five times the longest normal run at ≥ 5 Mbit/s (3 min). Per app, not per box: the box cannot
tell LAN from internet bytes inside its controller container, and the per-app view is what the check compares.
3. **Attribution:** the app's compose project; infrastructure (controller, traefik, cloudflared, Samba) never judged.
4. **Explanation:** qBittorrent / Transmission: explained when container bytes ≤ 1.25 × the app's own count + 1 Mbit/s
(measured overhead 1–7 %); Plex: any viewer explains the upload (session bandwidth not measured — the conservative
half; no viewer + a big upload = the Plex2Shell signature). Not explained → **`app_unexplained_upload`**,
operator-only, one per episode, a clear after 15 quiet minutes. No explainer: alarm only at ≥ 10 × its own p95 for
60 minutes, after ~20 h of history, worded „no app number to compare".
5. **CPU:** ≥ **max(150 %, 50 % × cores)** of one core for **30 minutes** (four times the longest normal run at ≥ 150 %),
explained by a Plex transcode or the box's own work on the app (backup, update, deploy) → else
**`app_unexplained_cpu`**. No explainer: ≥ max(200 %, 75 % × cores) for 60 minutes.
6. **qBittorrent limit:** 50 % of the line (CC's pick — R-1245), every 15 minutes, re-set when the line moves > 20 %;
never over a household's limit (a limit there first, or any change from the stored value). No line → nothing set.
The household's line (HU + EN in the report).
7. **Edge cases:** the night push is the controller's (never judged); the first day: apps with an explainer are judged
on the floor (a mismatch is evidence whatever the line), apps without one are not judged before ~20 h of history; a
slow or changing line (a Wi-Fi laptop): the line is a max over 14 nights — a falling line lowers nothing until it
rolls out (R-1247 notes the active test); two apps at once: each judged on its own bytes; the household's own devices:
invisible to the box — the trigger uses the box's own bytes only; an app reinstalled from scratch reads as „the
household removed the limit" and is left alone.
## 7. Live proofs on 9202 (controller `0.342.0-rc.1`, commit `a9c1997`, CI run 2270 success)
Set by hand in 9202's `/etc/felhom-controller-image` (an `-rc` tag is never offered to a household); 9202's hub link
is off by design, so every event was handed to the notifier and DROPPED there — the hub side (allow-list, operator-only,
the mail line) is proven by hub 0.169.0's tests. 9202's off-site is off, so one line record (40 Mbit/s) was seeded by
hand into its settings.json. Logs: `SPIKE-upload-watch-2026-10-23/live/2-9202-watch-log-full.txt`.
| # | What | Result (UTC) |
|---|---|---|
| 1 | qBittorrent unlimited, line 40 Mbit/s → the box sets the limit | **19:47:42** „qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)" — read back from qBittorrent: 2 499 584 |
| 2 | **a real torrent seeding at the limit** (5 GB, private, 9202 → 9471) | **20:04:42** „qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s)"; no alarm (counter ratio 1.04) |
| 3 | **a CPU burner in Plex with no transcode** (4 busy loops, 400 % on a 7-core guest; trigger 350 %) | **20:07:42** „plex used 399% CPU … for 30 min with no work that explains it — operator alarm" → `app_unexplained_cpu`; ONE |
| 4 | **the fake hacked app**: a 50 Mbit/s flood from inside qBittorrent's container to 9471, hidden behind the real seed | **20:19:42** „qbittorrent sent 65.0 Mbit/s for 15 min — NOT explained by the app (qbittorrent: 20.0 Mbit/s) — operator alarm" → `app_unexplained_upload`; ONE, none more while it ran |
| 5 | the burner stopped 20:11 | **20:25:41** „plex: the unexplained CPU stopped … — cleared" → `app_unexplained_cpu_cleared`; ONE |
| 6 | the household changes qBittorrent's limit by hand (1 500 000 → stored 1 499 136) at 20:21 | **20:32:42** „qBittorrent's upload limit is the household's — the box leaves it alone from now on"; read back at 20:43: still 1 499 136 |
| 7 | the flood stopped 20:26 (the seed went on, explained) | **20:41:41** „qbittorrent: the unexplained upload stopped … — cleared" → `app_unexplained_upload_cleared`; ONE |
| 8 | the system settings page (`GET /settings`) | „Feltöltési sebesség: legalább 40 Mbit/s (az éjszakai távoli mentésből mérve)"; qBittorrent's page carried no limit line after step 6 (the household's) — the box-owned branch is pinned by `TestUpwatch_QBLimitLineOnQBittorrentsPageOnly`, not captured live |
**Found live:** the very first minute after a start failed to reach the apps by name (the bootstrap joins the controller
to the app network a few seconds after the start); every later minute answered. That first minute only sets baselines —
**NOT-A-FINDING**. The media wiring's generated `docker-compose.override.yml` stays beside the template after a removal
(Jellyfin tonight; Plex's from the morning session) — rewritten before every start, harmless — **NOT-A-FINDING**.
**Teardown, three layers:** machine — the four throwaway apps removed through the product (containers and volumes gone,
verified), my test files and probes deleted, the `alpine:3.20` helper image removed, 9202 back on 0.341.0 with the seeded
record removed, the rc image removed; host — the receiver LXC 9471 destroyed (`--purge`, its disk gone); hub — none (9202
is not on the hub). The night sampler on the three test boxes is reads only.
@@ -0,0 +1,15 @@
gitea.dooplex.hu/admin/felhom-controller:0.342.0-rc.1
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0-rc.1 Up 6 minutes (healthy)
filebrowser gtstef/filebrowser:1.5.6-stable Up 4 hours (healthy)
immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 Up 4 hours (healthy)
immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 Up 4 hours (healthy)
immich-redis redis:7-alpine Up 4 hours (healthy)
immich-server ghcr.io/immich-app/immich-server:v3.2.4 Up 4 hours (healthy)
jellyfin jellyfin/jellyfin:10.11.11 Up About an hour (healthy)
paperless-postgres postgres:18-alpine Up 4 hours (healthy)
paperless-redis redis:7-alpine Up 4 hours (healthy)
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up 4 hours (healthy)
plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0 Up About an hour (healthy)
qbittorrent qbittorrentofficial/qbittorrent-nox:5.2.4-1 Up About an hour (healthy)
traefik traefik:v3.7.13 Up 4 hours
transmission lscr.io/linuxserver/transmission:version-4.1.3-r0 Up About an hour (healthy)
@@ -0,0 +1,10 @@
VMID Status Lock Name
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
9471 running cc-uw-recv
description: CC throwaway 2026-10-23 night%3A upload-watch spike receiver/leecher. Remove after the session.%0A
hostname: cc-uw-recv
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:04:68:80,ip=dhcp,type=veth
rootfs: nvme-scratch:9471/vm-9471-disk-0.raw,size=8G
tags: cc;scratch;uw-spike
@@ -0,0 +1,9 @@
2026/10/23 19:32:41 upwatch_wiring.go:97: [INFO] [upwatch] upload and CPU watch started (line at least 40 Mbit/s, qBittorrent share 50 %)
2026/10/23 19:32:41 watch.go:237: [DEBUG] [upwatch] plex: the app did not answer (resolving the plex container: lookup plex on 192.168.0.250:53: no such host) — treated as no number this minute
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] transmission: the app did not answer (Post "http://transmission:9091/transmission/rpc": dial tcp: lookup transmission on 192.168.0.250:53: no such host) — treated as no number this minute
2026/10/23 19:32:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_definition_refused (severity warning) — further app_definition_refused events are logged at DEBUG only
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] qbittorrent: the app did not answer (Post "http://qbittorrent:8080/api/v2/auth/login": dial tcp: lookup qbittorrent on 192.168.0.250:53: no such host) — treated as no number this minute
2026/10/23 19:47:42 service.go:108: [INFO] [upwatch] qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)
2026/10/23 20:04:42 watch.go:303: [INFO] [upwatch] qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s, 0 viewer(s)); normal use
2026/10/23 20:07:42 watch.go:343: [WARN] [upwatch] plex used 399% CPU (of one core; 7 cores) for 30 min with no work that explains it (plex transcodes) — operator alarm
2026/10/23 20:07:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_cpu (severity warning) — further app_unexplained_cpu events are logged at DEBUG only
@@ -0,0 +1,16 @@
2026/10/23 19:32:41 upwatch_wiring.go:97: [INFO] [upwatch] upload and CPU watch started (line at least 40 Mbit/s, qBittorrent share 50 %)
2026/10/23 19:32:41 watch.go:237: [DEBUG] [upwatch] plex: the app did not answer (resolving the plex container: lookup plex on 192.168.0.250:53: no such host) — treated as no number this minute
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] transmission: the app did not answer (Post "http://transmission:9091/transmission/rpc": dial tcp: lookup transmission on 192.168.0.250:53: no such host) — treated as no number this minute
2026/10/23 19:32:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_definition_refused (severity warning) — further app_definition_refused events are logged at DEBUG only
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] qbittorrent: the app did not answer (Post "http://qbittorrent:8080/api/v2/auth/login": dial tcp: lookup qbittorrent on 192.168.0.250:53: no such host) — treated as no number this minute
2026/10/23 19:47:42 service.go:108: [INFO] [upwatch] qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)
2026/10/23 20:04:42 watch.go:303: [INFO] [upwatch] qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s, 0 viewer(s)); normal use
2026/10/23 20:07:42 watch.go:343: [WARN] [upwatch] plex used 399% CPU (of one core; 7 cores) for 30 min with no work that explains it (plex transcodes) — operator alarm
2026/10/23 20:07:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_cpu (severity warning) — further app_unexplained_cpu events are logged at DEBUG only
2026/10/23 20:19:42 watch.go:307: [WARN] [upwatch] qbittorrent sent 65.0 Mbit/s for 15 min — NOT explained by the app (qbittorrent: 20.0 Mbit/s, 0 viewer(s)) — operator alarm
2026/10/23 20:19:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_upload (severity warning) — further app_unexplained_upload events are logged at DEBUG only
2026/10/23 20:25:41 watch.go:378: [INFO] [upwatch] plex: the unexplained CPU stopped (15 min below the trigger or explained) — cleared
2026/10/23 20:25:41 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_cpu_cleared (severity info) — further app_unexplained_cpu_cleared events are logged at DEBUG only
2026/10/23 20:32:42 service.go:111: [INFO] [upwatch] qBittorrent's upload limit is the household's — the box leaves it alone from now on
2026/10/23 20:41:41 watch.go:370: [INFO] [upwatch] qbittorrent: the unexplained upload stopped (15 min below the trigger or explained) — cleared
2026/10/23 20:41:41 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_upload_cleared (severity info) — further app_unexplained_upload_cleared events are logged at DEBUG only
@@ -0,0 +1,2 @@
/settings 56743 data-uplink> <span class="settings-label">Feltöltési sebesség</span> <span class="settings-value">legalább <span class="mono">40</span> Mbit/s (az éjszakai távoli mentésből mérve)</span> </div> | qb line: none
/apps/qbittorrent 49281 no uplink row | qb line: none
@@ -0,0 +1,4 @@
version v5.2.4 webapi 2.15.1
transfer/info {'connection_status': 'connected', 'dht_nodes': 375, 'dl_info_data': 0, 'dl_info_speed': 0, 'dl_rate_limit': 0, 'last_external_address_v4': '37.191.56.193', 'last_external_address_v6': '', 'up_info_data': 5566595787, 'up_info_speed': 0, 'up_rate_limit': 1499136}
prefs {'up_limit': 1499136, 'dl_limit': 0, 'alt_up_limit': 10240, 'alt_dl_limit': 10240, 'scheduler_enabled': False, 'limit_tcp_overhead': False, 'limit_lan_peers': True, 'limit_utp_rate': True, 'max_uploads': 20, 'enable_upload_suggestions': False}
uploadLimit 1499136 speedLimitsMode 0
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,186 @@
=== demo-hp: 2026-09-23 02:00 -> 2026-10-23 18:09 UTC (30.7 d), 998216 rows
container n tx p50 p95 max Mbit/s | cpu p50 p95 max %core | notes
qbittorrent 18 0.019 1.199 4.01 | 0.4 5.4 11.3 |
metube 3 0.146 0.146 0.15 | 0.0 0.1 0.1 |
immich-server 318 0.031 0.133 242.93 | 0.1 5.6 138.1 | resets=7
grimmory-db 102 0.000 0.116 0.35 | 0.0 16.9 102.5 | resets=12
immich-postgres 320 0.000 0.066 2.20 | 0.0 8.7 70.2 | resets=7
grimmory 94 0.001 0.047 0.77 | 1.1 272.0 305.0 | resets=9
immich-redis 320 0.013 0.029 0.18 | 0.6 4.3 5.1 | resets=7
romm 26492 0.019 0.027 0.20 | 0.7 3.2 442.4 | resets=97
docmost 43499 0.013 0.027 0.23 | 0.1 0.8 158.1 | resets=147
rallly 71 0.000 0.017 0.12 | 0.0 8.5 152.1 | resets=10
radarr 16 0.002 0.015 0.03 | 0.2 7.2 11.8 |
romm-redis 26523 0.005 0.013 0.10 | 0.7 4.4 61.5 | resets=101
paperless-webserver 43488 0.005 0.013 0.04 | 0.1 0.3 404.2 | resets=144
nextcloud-db 663 0.001 0.013 1.74 | 0.0 6.1 71.8 | resets=11
docmost-redis 43532 0.005 0.013 0.13 | 0.7 4.4 6.6 | resets=147
cloudflared 43604 0.000 0.013 1.21 | 0.3 0.5 13.8 | resets=6
rallly-postgres 71 0.000 0.007 0.01 | 0.0 5.0 8.8 | resets=7
prowlarr 5 0.002 0.004 0.00 | 0.1 0.2 0.2 |
sonarr 18 0.000 0.004 0.01 | 2.5 8.2 8.8 |
felhom-controller 43608 0.000 0.003 17.79 | 1.0 57.9 251.4 | resets=133
paperless-redis 43521 0.001 0.003 0.02 | 0.6 4.3 6.1 | resets=148
bentopdf 43567 0.000 0.003 0.01 | 0.0 0.0 5.2 | resets=58
homepage 78 0.000 0.002 0.01 | 0.0 4.2 4.5 | resets=1
bookstack-db 43565 0.001 0.001 0.01 | 0.0 6.1 102.8 | resets=147
romm-db 26526 0.000 0.001 0.19 | 0.0 6.0 105.3 | resets=89
plex 11974 0.001 0.001 1.10 | 0.3 0.6 124.6 | resets=46
nextcloud 662 0.000 0.001 0.80 | 0.0 5.7 150.0 | resets=11
immich-machine-learning 320 0.000 0.001 0.39 | 0.2 32.0 548.0 | resets=7
docmost-postgres 43533 0.000 0.001 0.02 | 0.0 4.7 29.6 | resets=147
bookstack 43546 0.000 0.001 0.01 | 0.0 0.0 63.0 | resets=148
adventurelog-postgres 43567 0.000 0.001 1.35 | 0.0 4.7 37.9 | resets=138
adventurelog 43549 0.000 0.001 2.16 | 0.0 0.7 204.3 | resets=146
ghost 20 0.000 0.001 0.00 | 0.0 6.8 132.9 | resets=2
glance 20 0.000 0.001 0.00 | 0.0 0.0 0.0 | resets=2
privatebin 43518 0.000 0.001 0.00 | 0.1 0.1 8.1 | resets=140
paperless-postgres 43520 0.000 0.001 0.05 | 0.0 4.7 19.1 | resets=148
kimai 30136 0.000 0.000 0.00 | 0.0 0.0 107.2 | resets=102
home-assistant 14403 0.000 0.000 1.00 | 0.0 0.5 106.9 | resets=62
kimai-db 30150 0.000 0.000 0.05 | 0.0 5.9 100.8 | resets=96
adventurelog-frontend 43559 0.000 0.000 0.21 | 0.0 3.8 22.8 | resets=126
vaultwarden 329 0.000 0.000 0.00 | 0.0 0.1 4.0 | resets=7
calibre-web 43547 0.000 0.000 0.01 | 0.0 0.3 127.1 | resets=145
opengist 43530 0.000 0.000 0.01 | 0.0 0.0 9.8 | resets=140
jellyfin 297 0.000 0.000 5.87 | 0.0 4.4 160.5 | resets=7
vikunja 8 0.000 0.000 0.00 | 0.0 0.0 0.0 | resets=2
a1-echo 113 0.000 0.000 0.00 | 0.0 0.0 0.0 |
upbeat_perlman 1 0.000 0.000 0.00 | 31.2 31.2 31.2 |
traefik 43608 0.000 0.000 1.21 | 0.1 3.5 92.1 | resets=11
practical_johnson 1 0.000 0.000 0.00 | 9.9 9.9 9.9 |
peaceful_ptolemy 1 0.000 0.000 0.00 | 14.1 14.1 14.1 |
nextcloud-redis 663 0.000 0.000 0.03 | 0.6 4.1 7.9 | resets=5
friendly_wilbur 1 0.000 0.000 0.00 | 59.0 59.0 59.0 |
filebrowser 43607 0.000 0.000 0.74 | 0.0 0.0 35.1 | resets=17
felhom-samba 10 0.000 0.000 0.00 | 0.0 13.5 17.6 | tx counter always 0 (host network?)
determined_satoshi 1 0.000 0.000 0.00 | 17.4 17.4 17.4 |
busiest UTC hours by bytes sent: 03h 14%, 02h 7%, 13h 5%, 00h 4%
=== demo-felhom: 2026-09-23 02:00 -> 2026-10-23 18:08 UTC (30.7 d), 268120 rows
container n tx p50 p95 max Mbit/s | cpu p50 p95 max %core | notes
immich-server 32 0.667 5.953 12.90 | 0.6 120.1 130.8 |
jellyfin 16 0.000 0.854 2.85 | 0.0 72.7 290.7 |
immich-postgres 32 0.000 0.766 2.24 | 0.3 8.5 197.6 |
romm 16 0.020 0.339 1.07 | 0.5 25.5 100.5 |
dawarich 20 0.000 0.313 3.01 | 0.1 5.3 104.4 |
immich-machine-learning 32 0.003 0.243 0.42 | 0.2 294.7 295.7 |
immich-redis 32 0.017 0.144 0.16 | 0.5 3.9 4.2 |
nextcloud-db 16 0.000 0.137 0.43 | 0.0 4.9 5.8 |
nextcloud 16 0.000 0.099 0.33 | 0.0 21.7 86.7 |
paperless-webserver 21 0.005 0.073 0.08 | 0.0 137.4 139.7 |
paperless-postgres 22 0.000 0.048 0.06 | 0.0 4.6 5.0 |
kimai-db 19 0.000 0.023 0.15 | 0.0 8.4 38.8 |
dawarich-sidekiq 19 0.020 0.023 0.02 | 0.1 0.2 0.2 |
claper 20 0.019 0.020 0.02 | 0.5 0.7 0.9 |
docmost 22 0.020 0.020 0.02 | 0.0 0.2 0.3 |
wanderer 19 0.002 0.015 0.02 | 0.0 0.0 0.0 |
cloudflared 43863 0.000 0.013 1.33 | 0.2 0.3 8.3 | resets=11
adventurelog 22 0.000 0.013 3.24 | 0.0 0.1 65.8 |
grafana 20 0.000 0.012 0.12 | 1.0 1.9 17.3 |
wanderer-db 19 0.007 0.011 0.01 | 0.0 3.7 3.7 |
dawarich-redis 20 0.009 0.011 0.01 | 0.4 3.7 3.8 |
claper-postgres 20 0.011 0.011 0.01 | 0.2 0.5 4.4 |
outline 19 0.008 0.008 0.01 | 0.0 7.4 7.7 |
nextcloud-redis 16 0.000 0.008 0.02 | 0.4 0.7 1.4 |
romm-redis 16 0.006 0.007 0.01 | 0.5 3.7 3.8 |
docmost-redis 23 0.005 0.005 0.07 | 0.4 0.7 3.9 |
dawarich-db 20 0.001 0.005 0.03 | 0.0 0.1 0.1 |
bookstack-db 20 0.001 0.005 0.04 | 0.0 4.7 4.8 |
grocy 19 0.000 0.005 0.01 | 0.0 0.0 0.0 |
bookstack 20 0.000 0.005 0.04 | 0.0 7.9 47.2 |
zipline 20 0.003 0.003 0.00 | 0.0 0.0 0.0 |
paperless-redis 22 0.002 0.003 0.00 | 0.4 0.4 0.4 |
outline-redis 19 0.003 0.003 0.00 | 0.4 0.4 0.5 |
kimai 19 0.000 0.003 0.02 | 0.0 2.4 24.3 |
zipline-postgres 20 0.003 0.003 0.00 | 0.0 0.0 0.0 |
qbittorrent 11759 0.001 0.003 0.01 | 0.1 0.2 97.5 | resets=25
felhom-controller 43863 0.000 0.003 1.73 | 0.1 15.5 122.0 | resets=92
bentopdf 18 0.000 0.002 0.00 | 0.0 0.0 0.0 |
emby 19 0.001 0.002 0.01 | 0.0 0.1 0.1 |
homepage 20 0.000 0.002 0.00 | 0.0 0.0 0.0 |
sonarr 11756 0.000 0.001 0.01 | 0.1 0.3 74.7 | resets=25
grimmory-db 24 0.000 0.001 0.35 | 0.0 5.0 100.2 |
grimmory 23 0.001 0.001 0.00 | 5.7 25.3 35.0 |
lubelogger 19 0.000 0.001 0.00 | 0.0 0.2 0.6 |
radarr 11756 0.000 0.001 0.17 | 0.0 0.6 79.9 | resets=25
glance 19 0.000 0.001 0.00 | 0.0 0.0 0.0 |
metube 20 0.000 0.001 0.00 | 0.0 1.3 26.1 |
actualbudget 18 0.000 0.001 0.00 | 0.0 0.0 0.0 |
privatebin 19 0.000 0.001 0.00 | 0.0 0.0 0.0 |
plex 18 0.000 0.001 0.00 | 0.7 0.8 0.8 |
papra 20 0.000 0.001 0.00 | 0.0 6.8 7.5 |
home-assistant 16 0.000 0.001 0.00 | 0.0 19.8 79.0 |
wanderer-search 19 0.000 0.001 0.00 | 0.1 0.1 0.1 |
karakeep 19 0.001 0.001 0.00 | 1.8 2.0 2.0 |
romm-db 16 0.001 0.001 0.00 | 0.0 4.6 4.7 |
radicale 19 0.000 0.001 0.00 | 0.0 0.0 0.0 |
gramps-web 20 0.000 0.001 0.00 | 0.0 0.0 0.0 |
tandoor 19 0.000 0.001 0.00 | 0.1 4.7 4.8 |
karakeep-chrome 19 0.001 0.001 0.00 | 0.0 0.0 0.0 |
outline-postgres 19 0.000 0.001 0.00 | 0.0 0.4 0.7 |
seerr 18 0.000 0.001 0.00 | 0.0 0.0 0.0 |
n8n 22 0.000 0.000 0.00 | 0.8 1.0 110.6 |
termix 20 0.000 0.000 0.00 | 0.0 0.4 2.3 |
tandoor-postgres 19 0.000 0.000 0.00 | 0.4 1.5 4.1 |
audiobookshelf 19 0.000 0.000 0.00 | 0.0 0.0 0.1 |
adventurelog-postgres 22 0.000 0.000 0.02 | 0.0 3.8 33.6 |
ghost 21 0.000 0.000 0.00 | 0.0 0.0 0.0 |
transmission 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
komga 19 0.000 0.000 0.00 | 0.3 6.8 10.6 |
calcom 18 0.000 0.000 0.00 | 0.0 3.4 22.3 |
vaultwarden 20 0.000 0.000 0.00 | 0.0 0.0 0.0 |
sparkyfitness 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
homebox 19 0.000 0.000 0.00 | 0.1 0.2 0.2 |
docmost-postgres 23 0.000 0.000 0.01 | 0.0 3.8 73.9 |
vikunja 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
adventurelog-frontend 22 0.000 0.000 0.00 | 0.0 0.0 0.0 |
rallly 20 0.000 0.000 0.00 | 0.0 5.6 110.3 |
calibre-web 17 0.000 0.000 0.00 | 0.0 21.3 106.4 |
wishlist 20 0.000 0.000 0.00 | 0.0 6.3 125.1 |
prowlarr 11738 0.000 0.000 0.00 | 0.0 0.1 77.0 | resets=26
opengist 43862 0.000 0.000 0.01 | 0.0 0.0 6.0 | resets=107
onlyoffice 22 0.000 0.000 0.00 | 0.0 130.5 147.5 |
sabnzbd 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
gitea 19 0.000 0.000 0.00 | 0.0 0.4 4.0 |
navidrome 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
code-server 20 0.000 0.000 0.00 | 0.0 0.0 0.0 |
uptime-kuma 19 0.000 0.000 0.00 | 0.7 0.7 0.7 |
nzbget 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
gokapi 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
recipe-importer 20 0.000 0.000 0.00 | 0.0 0.0 0.0 |
crafty-controller 19 0.000 0.000 0.00 | 0.1 8.3 8.5 |
mealie 20 0.000 0.000 0.00 | 0.1 0.1 0.1 |
rallly-postgres 20 0.000 0.000 0.00 | 0.0 3.7 3.8 |
traefik 43863 0.000 0.000 0.96 | 0.0 3.6 17.9 | resets=13
sparkyfitness-server 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
sparkyfitness-db 19 0.000 0.000 0.00 | 0.0 3.8 3.8 |
karakeep-meilisearch 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
filebrowser 43863 0.000 0.000 0.74 | 0.0 0.0 15.7 | resets=16
calcom-postgres 18 0.000 0.000 0.00 | 0.0 3.9 4.0 |
busiest UTC hours by bytes sent: 16h 16%, 05h 12%, 09h 6%, 02h 5%
=== tester-1: 2026-10-04 19:41 -> 2026-10-23 18:08 UTC (18.9 d), 294433 rows
container n tx p50 p95 max Mbit/s | cpu p50 p95 max %core | notes
qbittorrent 3857 0.007 0.133 42.40 | 0.1 2.6 92.3 | resets=20
transmission 3857 0.001 0.013 57.43 | 0.1 1.6 82.5 | resets=19
paperless-webserver 26710 0.005 0.013 0.09 | 0.1 0.7 508.1 | resets=63
cloudflared 26751 0.000 0.013 1.63 | 0.4 1.2 29.4 | resets=6
prowlarr 3725 0.000 0.012 0.26 | 0.1 0.3 101.8 | resets=8
sonarr 3855 0.001 0.004 1.21 | 0.1 4.5 131.6 | resets=20
radarr 3855 0.001 0.003 1.52 | 0.1 4.3 103.3 | resets=19
paperless-redis 26740 0.001 0.003 0.01 | 0.7 4.2 27.5 | resets=68
felhom-controller 26751 0.000 0.003 37.47 | 0.5 46.4 422.6 | resets=62
plex 3739 0.001 0.002 1.07 | 0.3 4.3 198.4 | resets=17
bookstack-db 26729 0.001 0.001 0.03 | 0.0 5.7 130.6 | resets=66
plex-k6 514 0.001 0.001 0.01 | 0.4 5.4 7.2 |
bookstack 26722 0.000 0.001 0.04 | 0.0 0.1 72.5 | resets=65
privatebin 26744 0.000 0.001 0.00 | 0.0 0.1 14.7 | resets=64
paperless-postgres 26740 0.000 0.001 0.02 | 0.0 4.4 65.8 | resets=68
traefik 26751 0.000 0.000 1.56 | 0.0 3.4 100.6 | resets=3
jellyfin 3648 0.000 0.000 2.83 | 0.0 0.2 181.5 | resets=19
xfp-echo 1 0.000 0.000 0.00 | 0.0 0.0 0.0 |
vaultwarden 1 0.000 0.000 0.00 | 0.0 0.0 0.0 |
filebrowser 26743 0.000 0.000 0.01 | 0.0 0.0 80.1 | resets=22
busiest UTC hours by bytes sent: 07h 13%, 18h 11%, 02h 9%, 17h 9%
@@ -0,0 +1,22 @@
== demo-hp
container cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]
kimai 13 0 (-) 0 0 (-)
paperless-webserver 6 1 (09-23 04:56) 0 0 (-)
immich-machine-learning 4 4 (10-14 00:54) 0 0 (-)
romm 3 2 (10-23 15:36) 0 0 (-)
adventurelog 2 1 (09-30 11:29) 3 0 (-)
adventurelog-postgres 0 0 (-) 3 0 (-)
felhom-controller 3 1 (09-23 11:37) 2 2 (09-28 13:44)
immich-server 2 0 (-) 3 2 (10-14 00:53)
== demo-felhom
container cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]
immich-server 2 0 (-) 8 1 (10-13 05:32)
immich-machine-learning 7 7 (10-13 16:39) 0 0 (-)
paperless-webserver 6 0 (-) 0 0 (-)
onlyoffice 4 0 (-) 0 0 (-)
== tester-1
container cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]
plex 10 5 (10-14 17:20) 1 0 (-)
qbittorrent 2 0 (-) 9 3 (10-14 18:08)
sonarr 9 0 (-) 2 0 (-)
transmission 1 0 (-) 6 2 (10-16 07:17)
@@ -0,0 +1,90 @@
== demo-hp: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak
2026-09-23 max1= 3.14 best5= 0.91 busy_min= 0 peak_at=02:17Z
2026-09-24 max1= 3.04 best5= 0.73 busy_min= 0 peak_at=02:17Z
2026-09-25 max1= 2.13 best5= 0.67 busy_min= 0 peak_at=02:18Z
2026-09-26 max1= 1.77 best5= 0.63 busy_min= 0 peak_at=02:18Z
2026-09-27 max1= 2.04 best5= 0.75 busy_min= 0 peak_at=02:18Z
2026-09-28 max1= 17.79 best5= 6.89 busy_min= 3 peak_at=13:43Z
2026-09-29 max1= 2.45 best5= 0.65 busy_min= 0 peak_at=02:17Z
2026-09-30 max1= 2.62 best5= 0.76 busy_min= 0 peak_at=02:18Z
2026-10-01 max1= 1.02 best5= 0.54 busy_min= 0 peak_at=02:18Z
2026-10-02 max1= 2.02 best5= 0.62 busy_min= 0 peak_at=02:18Z
2026-10-03 max1= 2.01 best5= 0.60 busy_min= 0 peak_at=15:25Z
2026-10-04 max1= 2.08 best5= 0.42 busy_min= 0 peak_at=02:17Z
2026-10-05 max1= 2.36 best5= 0.48 busy_min= 0 peak_at=05:26Z
2026-10-06 max1= 1.58 best5= 0.32 busy_min= 0 peak_at=02:17Z
2026-10-07 max1= 1.05 best5= 0.30 busy_min= 0 peak_at=02:16Z
2026-10-08 max1= 1.43 best5= 0.29 busy_min= 0 peak_at=02:17Z
2026-10-09 max1= 1.47 best5= 0.29 busy_min= 0 peak_at=02:17Z
2026-10-10 max1= 0.90 best5= 0.27 busy_min= 0 peak_at=02:16Z
2026-10-11 max1= 1.34 best5= 0.30 busy_min= 0 peak_at=02:16Z
2026-10-12 max1= 4.96 best5= 0.99 busy_min= 0 peak_at=02:17Z
2026-10-13 max1= 1.31 best5= 0.28 busy_min= 0 peak_at=02:16Z
2026-10-14 max1= 6.31 best5= 1.27 busy_min= 1 peak_at=02:18Z
2026-10-15 max1= 2.25 best5= 0.45 busy_min= 0 peak_at=02:17Z
2026-10-16 max1= 2.78 best5= 0.66 busy_min= 0 peak_at=20:55Z
2026-10-17 max1= 2.26 best5= 0.45 busy_min= 0 peak_at=02:17Z
2026-10-18 max1= 2.08 best5= 0.47 busy_min= 0 peak_at=02:18Z
2026-10-19 max1= 16.95 best5= 5.08 busy_min= 2 peak_at=02:18Z
2026-10-20 max1= 1.20 best5= 0.48 busy_min= 0 peak_at=02:17Z
2026-10-21 max1= 2.27 best5= 0.46 busy_min= 0 peak_at=02:18Z
2026-10-22 max1= 2.13 best5= 0.52 busy_min= 0 peak_at=02:18Z
2026-10-23 max1= 1.37 best5= 0.48 busy_min= 0 peak_at=02:18Z
nights with a push (best5>2): 2; best5 min/median/max = 5.1/6.9/6.9 Mbit/s
== demo-felhom: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak
2026-09-23 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=02:15Z
2026-09-24 max1= 0.06 best5= 0.01 busy_min= 0 peak_at=02:15Z
2026-09-25 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
2026-09-26 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
2026-09-27 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=00:19Z
2026-09-28 max1= 0.03 best5= 0.01 busy_min= 0 peak_at=02:16Z
2026-09-29 max1= 0.03 best5= 0.01 busy_min= 0 peak_at=02:15Z
2026-09-30 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
2026-10-01 max1= 0.07 best5= 0.01 busy_min= 0 peak_at=19:54Z
2026-10-02 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
2026-10-03 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=02:16Z
2026-10-04 max1= 0.02 best5= 0.00 busy_min= 0 peak_at=06:58Z
2026-10-05 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=05:21Z
2026-10-06 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=17:34Z
2026-10-07 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=02:15Z
2026-10-08 max1= 0.09 best5= 0.02 busy_min= 0 peak_at=13:04Z
2026-10-09 max1= 0.10 best5= 0.02 busy_min= 0 peak_at=05:34Z
2026-10-10 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=02:15Z
2026-10-11 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=10:46Z
2026-10-12 max1= 0.08 best5= 0.02 busy_min= 0 peak_at=06:18Z
2026-10-13 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=13:54Z
2026-10-14 max1= 0.07 best5= 0.02 busy_min= 0 peak_at=07:59Z
2026-10-15 max1= 0.14 best5= 0.03 busy_min= 0 peak_at=14:25Z
2026-10-16 max1= 0.84 best5= 0.17 busy_min= 0 peak_at=02:15Z
2026-10-17 max1= 0.49 best5= 0.10 busy_min= 0 peak_at=02:15Z
2026-10-18 max1= 0.33 best5= 0.07 busy_min= 0 peak_at=02:16Z
2026-10-19 max1= 1.73 best5= 0.35 busy_min= 0 peak_at=02:16Z
2026-10-20 max1= 0.23 best5= 0.07 busy_min= 0 peak_at=02:16Z
2026-10-21 max1= 0.36 best5= 0.07 busy_min= 0 peak_at=02:16Z
2026-10-22 max1= 0.52 best5= 0.11 busy_min= 0 peak_at=02:15Z
2026-10-23 max1= 0.48 best5= 0.10 busy_min= 0 peak_at=02:15Z
== tester-1: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak
2026-10-04 max1= 0.17 best5= 0.14 busy_min= 0 peak_at=19:55Z
2026-10-05 max1= 1.44 best5= 0.29 busy_min= 0 peak_at=03:10Z
2026-10-06 max1= 0.64 best5= 0.13 busy_min= 0 peak_at=02:16Z
2026-10-07 max1= 0.64 best5= 0.13 busy_min= 0 peak_at=02:15Z
2026-10-08 max1= 0.42 best5= 0.08 busy_min= 0 peak_at=02:16Z
2026-10-09 max1= 0.48 best5= 0.10 busy_min= 0 peak_at=02:16Z
2026-10-10 max1= 0.84 best5= 0.17 busy_min= 0 peak_at=02:16Z
2026-10-11 max1= 0.46 best5= 0.09 busy_min= 0 peak_at=02:15Z
2026-10-12 max1= 1.71 best5= 0.34 busy_min= 0 peak_at=02:16Z
2026-10-13 max1= 0.99 best5= 0.20 busy_min= 0 peak_at=02:15Z
2026-10-14 max1= 0.45 best5= 0.09 busy_min= 0 peak_at=02:16Z
2026-10-15 max1= 37.47 best5= 7.58 busy_min= 1 peak_at=02:17Z
2026-10-16 max1= 3.85 best5= 0.77 busy_min= 0 peak_at=02:17Z
2026-10-17 max1= 5.11 best5= 1.21 busy_min= 1 peak_at=02:16Z
2026-10-18 max1= 0.50 best5= 0.10 busy_min= 0 peak_at=02:16Z
2026-10-19 max1= 5.14 best5= 1.03 busy_min= 1 peak_at=02:16Z
2026-10-20 max1= 0.52 best5= 0.10 busy_min= 0 peak_at=02:16Z
2026-10-21 max1= 0.61 best5= 0.12 busy_min= 0 peak_at=02:16Z
2026-10-22 max1= 0.53 best5= 0.11 busy_min= 0 peak_at=02:16Z
2026-10-23 max1= 0.80 best5= 0.18 busy_min= 0 peak_at=02:16Z
nights with a push (best5>2): 1; best5 min/median/max = 7.6/7.6/7.6 Mbit/s
@@ -0,0 +1,6 @@
login 204 cookie
{"t": 0.2, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65136369198, "engine_tx": 65136367684, "cli": "65.5MB / 65.1GB"}
{"t": 6.5, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65152041285, "engine_tx": 65152039771, "cli": "65.9MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 19.9, "rate_engine_Mbit": 19.9}
{"t": 12.7, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65167646902, "engine_tx": 65167645388, "cli": "66.2MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 20.14, "rate_engine_Mbit": 20.14}
{"t": 19.0, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65183320503, "engine_tx": 65183318989, "cli": "66.6MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 19.9, "rate_engine_Mbit": 19.9}
{"t": 25.2, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65198925974, "engine_tx": 65198924460, "cli": "66.9MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 20.14, "rate_engine_Mbit": 20.14}
@@ -0,0 +1,10 @@
LISTEN 0 5 0.0.0.0:5001 0.0.0.0:*
login 204 cookie
{"t": 0.0, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 1559315619, "engine_tx": 1558369279, "cli": "2.94MB / 2.85GB"}
{"t": 6.0, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 9161893432, "engine_tx": 9161333668, "cli": "9.34MB / 10.5GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10136.77, "rate_engine_Mbit": 10137.29}
{"t": 12.1, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 17020963755, "engine_tx": 17019975365, "cli": "16.7MB / 18.7GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10306.98, "rate_engine_Mbit": 10306.42}
{"t": 18.1, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 25140960556, "engine_tx": 25139940620, "cli": "23.6MB / 26.5GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10826.66, "rate_engine_Mbit": 10826.62}
{"t": 24.2, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 33049634829, "engine_tx": 33048966257, "cli": "30.6MB / 34.4GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10372.03, "rate_engine_Mbit": 10372.49}
{"t": 30.2, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 41055773948, "engine_tx": 41055105508, "cli": "37.6MB / 42.3GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10674.85, "rate_engine_Mbit": 10674.85}
{"t": 36.3, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 48735514801, "engine_tx": 48734576785, "cli": "44.3MB / 50GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10071.79, "rate_engine_Mbit": 10071.44}
{"t": 42.3, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 56370706217, "engine_tx": 56370119481, "cli": "52.4MB / 58.8GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10180.26, "rate_engine_Mbit": 10180.72}
@@ -0,0 +1,9 @@
login 204 cookie
setUploadLimit (200, '')
uploadLimit now 2499584 prefs.up_limit 2499584
login 204 cookie
{"t": 0.0, "qb_up_speed": 16386315, "qb_up_data": 1346633728, "qb_tor_uploaded": 1346633728, "proc_tx": 1422747831, "engine_tx": 1422747831, "cli": "27.2MB / 1.42GB"}
{"t": 6.0, "qb_up_speed": 0, "qb_up_data": 1374387364, "qb_tor_uploaded": 1374387364, "proc_tx": 1434635707, "engine_tx": 1434635707, "cli": "27.6MB / 1.44GB", "rate_qb_data_Mbit": 37.0, "rate_proc_Mbit": 15.85, "rate_engine_Mbit": 15.85}
{"t": 12.1, "qb_up_speed": 2872275, "qb_up_data": 1392093322, "qb_tor_uploaded": 1391692420, "proc_tx": 1457229455, "engine_tx": 1457229455, "cli": "27.9MB / 1.46GB", "rate_qb_data_Mbit": 23.22, "rate_proc_Mbit": 29.63, "rate_engine_Mbit": 29.63}
{"t": 18.1, "qb_up_speed": 1975559, "qb_up_data": 1406540742, "qb_tor_uploaded": 1406540742, "proc_tx": 1471460371, "engine_tx": 1471460371, "cli": "28.2MB / 1.47GB", "rate_qb_data_Mbit": 19.26, "rate_proc_Mbit": 18.97, "rate_engine_Mbit": 18.97}
{"t": 24.2, "qb_up_speed": 3254241, "qb_up_data": 1423113257, "qb_tor_uploaded": 1420865709, "proc_tx": 1488294067, "engine_tx": 1488294067, "cli": "28.5MB / 1.49GB", "rate_qb_data_Mbit": 21.73, "rate_proc_Mbit": 22.08, "rate_engine_Mbit": 22.08}
@@ -0,0 +1,11 @@
login 204 cookie
{"t": 0.0, "qb_up_speed": 16266049, "qb_up_data": 117833728, "qb_tor_uploaded": 117701484, "proc_tx": 123161204, "engine_tx": 123161204, "cli": "6.74MB / 140MB"}
{"t": 6.0, "qb_up_speed": 16384087, "qb_up_data": 216137728, "qb_tor_uploaded": 216137728, "proc_tx": 225760308, "engine_tx": 225760308, "cli": "8.38MB / 243MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 136.8, "rate_engine_Mbit": 136.8}
{"t": 12.1, "qb_up_speed": 16387266, "qb_up_data": 314441728, "qb_tor_uploaded": 314441728, "proc_tx": 328359626, "engine_tx": 328359626, "cli": "10.1MB / 346MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 134.56, "rate_engine_Mbit": 134.56}
{"t": 18.1, "qb_up_speed": 16391332, "qb_up_data": 412745728, "qb_tor_uploaded": 412745728, "proc_tx": 431576783, "engine_tx": 431540471, "cli": "11.8MB / 450MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 137.62, "rate_engine_Mbit": 137.57}
{"t": 24.2, "qb_up_speed": 16377743, "qb_up_data": 511049728, "qb_tor_uploaded": 511049728, "proc_tx": 536615588, "engine_tx": 536559607, "cli": "13.4MB / 556MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 137.76, "rate_engine_Mbit": 137.73}
{"t": 30.2, "qb_up_speed": 16387474, "qb_up_data": 609353728, "qb_tor_uploaded": 609353728, "proc_tx": 641957804, "engine_tx": 641903336, "cli": "14.9MB / 660MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 140.46, "rate_engine_Mbit": 140.46}
{"t": 36.3, "qb_up_speed": 16383934, "qb_up_data": 707657728, "qb_tor_uploaded": 707657728, "proc_tx": 746552576, "engine_tx": 746552576, "cli": "16.8MB / 764MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 137.17, "rate_engine_Mbit": 137.24}
{"t": 42.3, "qb_up_speed": 16392852, "qb_up_data": 805961728, "qb_tor_uploaded": 805961728, "proc_tx": 849903723, "engine_tx": 849903723, "cli": "18.3MB / 867MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 137.8, "rate_engine_Mbit": 137.8}
{"t": 48.4, "qb_up_speed": 16382132, "qb_up_data": 904265728, "qb_tor_uploaded": 904265728, "proc_tx": 952502020, "engine_tx": 952502020, "cli": "19.9MB / 970MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 134.56, "rate_engine_Mbit": 134.56}
{"t": 54.4, "qb_up_speed": 16376106, "qb_up_data": 1002569728, "qb_tor_uploaded": 1002569728, "proc_tx": 1055099637, "engine_tx": 1055099637, "cli": "21.5MB / 1.07GB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 136.8, "rate_engine_Mbit": 136.8}
@@ -0,0 +1,8 @@
{'arguments': {}, 'result': 'success'}
{'speed-limit-up': 2500, 'speed-limit-up-enabled': True}
{"t": 0.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157242728}
{"t": 5.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157244479, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 10.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157246230, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 15.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157248133, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 20.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157249884, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 25.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157251635, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
@@ -0,0 +1,10 @@
{"t": 0.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157256302}
{"t": 10.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157258205, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 20.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157261594, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 30.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157263345, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 40.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157265096, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 50.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157268485, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 60.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157271188, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 70.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157272939, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 80.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157276328, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 90.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157278588, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
@@ -0,0 +1,15 @@
login 204 cookie
/api/v2/torrents/stop (200, '')
{'arguments': {}, 'result': 'success'}
{"t": 0.0, "tr_upspeed": 40648163, "tr_cur_up": 1528429958, "tr_tor_up": 1528429958, "tx": 1534546958}
{"t": 5.0, "tr_upspeed": 65588000, "tr_cur_up": 1832873057, "tr_tor_up": 1832873057, "tx": 1839315356, "rate_tr_Mbit": 487.11, "rate_container_Mbit": 487.63}
{"t": 10.0, "tr_upspeed": 23587084, "tr_cur_up": 2076811226, "tr_tor_up": 2076811226, "tx": 2083512878, "rate_tr_Mbit": 390.3, "rate_container_Mbit": 390.72}
{"t": 15.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157213873, "rate_tr_Mbit": 115.8, "rate_container_Mbit": 117.92}
{"t": 20.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157215624, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 25.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157218947, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.01}
{"t": 30.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157220698, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 35.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157222449, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 40.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157224200, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 45.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157225951, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 50.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157227702, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 55.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157231243, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.01}
@@ -0,0 +1,2 @@
# usage: gen.sh <dir> ; writes <dir>/uwtest.bin, 2 GiB, identical on every machine (AES-CTR keystream of a fixed key)
mkdir -p "$1"; openssl enc -aes-128-ctr -nosalt -K 00112233445566778899aabbccddeeff -iv 0 -in /dev/zero 2>/dev/null | head -c 2147483648 > "$1/uwtest.bin"; sha256sum "$1/uwtest.bin" | cut -c1-16
@@ -0,0 +1,30 @@
import sqlite3,sys,collections,datetime,statistics
def pct(a,p):
if not a: return 0
a=sorted(a); k=(len(a)-1)*p/100; f=int(k); c=min(f+1,len(a)-1); return a[f]+(a[c]-a[f])*(k-f)
for box in sys.argv[1:]:
db=sqlite3.connect(f'file:{box}/metrics.db?mode=ro',uri=True)
lo,hi,n=db.execute('select min(ts),max(ts),count(*) from container_metrics').fetchone()
print(f'\n=== {box.rsplit("/",1)[1]}: {datetime.datetime.utcfromtimestamp(lo):%Y-%m-%d %H:%M} -> {datetime.datetime.utcfromtimestamp(hi):%Y-%m-%d %H:%M} UTC ({(hi-lo)/86400:.1f} d), {n} rows')
rows=collections.defaultdict(list)
for ts,name,cpu,tx in db.execute('select ts,container_name,cpu_percent,net_tx_bytes from container_metrics order by ts'):
rows[name].append((ts,cpu,tx))
hour=collections.defaultdict(float)
out=[]
for name,r in rows.items():
rates=[];resets=0;zero=all((x[2] or 0)==0 for x in r)
for (t0,_,a),(t1,_,b) in zip(r,r[1:]):
if a is None or b is None or t1-t0<=0 or t1-t0>180: continue
d=b-a
if d<0: resets+=1; continue
mbps=d*8/(t1-t0)/1e6; rates.append(mbps)
hour[datetime.datetime.utcfromtimestamp(t1).hour]+=d
cpus=[x[1] for x in r]
out.append((pct(rates,95),name,len(r),pct(rates,50),pct(rates,95),max(rates or [0]),pct(cpus,50),pct(cpus,95),max(cpus),zero,resets))
print(f'{"container":32} {"n":>6} {"tx p50":>8} {"p95":>8} {"max":>8} Mbit/s | {"cpu p50":>7} {"p95":>7} {"max":>7} %core | notes')
for o in sorted(out,reverse=True):
_,name,nn,a,b,c,d,e,f,zero,res=o
note=('tx counter always 0 (host network?)' if zero else '')+(f' resets={res}' if res else '')
print(f'{name[:32]:32} {nn:6} {a:8.3f} {b:8.3f} {c:8.2f} | {d:7.1f} {e:7.1f} {f:7.1f} | {note}')
tot=sum(hour.values()) or 1
print('busiest UTC hours by bytes sent:',', '.join(f'{h:02d}h {100*v/tot:.0f}%' for h,v in sorted(hour.items(),key=lambda x:-x[1])[:4]))
@@ -0,0 +1,21 @@
import sqlite3,sys,collections,datetime
for box in sys.argv[1:]:
db=sqlite3.connect(f'file:{box}/metrics.db?mode=ro',uri=True)
r=db.execute("select ts,net_tx_bytes from container_metrics where container_name='felhom-controller' order by ts").fetchall()
per=collections.defaultdict(list)
for (t0,a),(t1,b) in zip(r,r[1:]):
if a is None or b is None or b<a or not 0<t1-t0<=180: continue
d=datetime.datetime.utcfromtimestamp(t1)
per[d.date()].append((t1,(b-a)*8/(t1-t0)/1e6))
print(f'\n== {box.rsplit("/",1)[1]}: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak')
peaks=[]
for day in sorted(per):
v=per[day]; rates=[x[1] for x in v]
best5=max((sum(rates[i:i+5])/5 for i in range(max(1,len(rates)-4))),default=0)
m=max(v,key=lambda x:x[1])
busy=sum(1 for x in rates if x>5)
peaks.append(best5)
print(f'{day} max1={m[1]:6.2f} best5={best5:6.2f} busy_min={busy:4d} peak_at={datetime.datetime.utcfromtimestamp(m[0]):%H:%M}Z')
big=[p for p in peaks if p>2]
if big:
big.sort(); print(f'nights with a push (best5>2): {len(big)}; best5 min/median/max = {big[0]:.1f}/{big[len(big)//2]:.1f}/{big[-1]:.1f} Mbit/s')
@@ -0,0 +1,10 @@
#!/bin/bash
# Waits until 02:08Z on 2026-10-24, then reads the controller container's eth0 tx bytes once a second for 22 minutes on
# each of the three test boxes (read only: /proc/<pid>/net/dev inside guest 9201). Output: one file per box.
OUT=$1
target=$(date -u -d '2026-10-24T02:08:00Z' +%s); now=$(date -u +%s); [ $target -gt $now ] && sleep $((target-now))
INNER='P=$(docker inspect -f {{.State.Pid}} felhom-controller); end=$(( $(date +%s) + 1320 )); while [ $(date +%s) -lt $end ]; do echo "$(date +%s.%N | cut -c1-14) $(awk "/eth0:/{print \$10}" /proc/$P/net/dev)"; sleep 1; done'
for pair in "demo-hp:demo-hp" "felhom-pve:demo-felhom" "root@192.168.0.154:tester-1"; do h=${pair%%:*}; n=${pair##*:}
ssh -o BatchMode=yes -o ServerAliveInterval=30 $h "pct exec 9201 -- bash -c '$INNER'" > "$OUT/night-push-1s-$n.txt" 2>&1 &
done
wait; echo "done $(date -u +%FT%TZ)"
@@ -0,0 +1,89 @@
# Runs INSIDE guest 9202. Asks qBittorrent's own Web API and reads the container's exact byte counters.
# The admin password is read from the container env at use (as the controller does) and never printed.
import json, subprocess, sys, time, urllib.request, urllib.parse, http.client, socket, re, uuid
def sh(*a): return subprocess.run(a, capture_output=True, text=True).stdout.strip()
C = sys.argv[1] if len(sys.argv) > 1 else 'qbittorrent'
IP = sh('docker', 'inspect', '-f', '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}', C).split()[0]
PID = sh('docker', 'inspect', '-f', '{{.State.Pid}}', C)
CID = sh('docker', 'inspect', '-f', '{{.Id}}', C)
BASE = f'http://{IP}:8080'
PW = sh('docker', 'exec', C, 'printenv', 'QBT_ADMIN_PASSWORD')
def req(method, path, data=None, hdr=None, raw=False):
h = {'Referer': BASE}
h.update(hdr or {})
if COOKIE: h['Cookie'] = COOKIE
r = urllib.request.Request(BASE + path, data=data, headers=h, method=method)
try:
with urllib.request.urlopen(r, timeout=20) as resp:
b = resp.read(); return resp.status, (b if raw else (json.loads(b) if b and b[:1] in b'[{' else b.decode()))
except urllib.error.HTTPError as e:
return e.code, e.read().decode()[:200]
COOKIE = None
code, _ = None, None
r = urllib.request.Request(BASE + '/api/v2/auth/login', data=urllib.parse.urlencode({'username': 'admin', 'password': PW}).encode(),
headers={'Referer': BASE, 'Content-Type': 'application/x-www-form-urlencoded'}, method='POST')
with urllib.request.urlopen(r, timeout=20) as resp:
m = re.search(r'((?:QBT_)?SID[A-Za-z0-9_]*)=([^;]+)', '; '.join(resp.headers.get_all('Set-Cookie') or []))
print('login', resp.status, 'cookie' if m else 'NO COOKIE')
COOKIE = f'{m.group(1)}={m.group(2)}'
def netdev():
for line in open(f'/proc/{PID}/net/dev'):
if line.strip().startswith('eth0:'):
f = line.split(':', 1)[1].split(); return int(f[8])
def engine_stats():
s = socket.socket(socket.AF_UNIX); s.connect('/var/run/docker.sock')
s.sendall(f'GET /containers/{CID}/stats?stream=false&one-shot=true HTTP/1.0\r\n\r\n'.encode())
b = b''
while True:
x = s.recv(65536)
if not x: break
b += x
j = json.loads(b.split(b'\r\n\r\n', 1)[1])
return sum(n['tx_bytes'] for n in j.get('networks', {}).values()), j['cpu_stats']['cpu_usage']['total_usage']
def cli_stats():
return sh('docker', 'stats', '--no-stream', '--format', '{{.NetIO}}', C)
mode = sys.argv[2] if len(sys.argv) > 2 else 'info'
if mode == 'info':
print('version', req('GET', '/api/v2/app/version')[1], 'webapi', req('GET', '/api/v2/app/webapiVersion')[1])
print('transfer/info', req('GET', '/api/v2/transfer/info')[1])
p = req('GET', '/api/v2/app/preferences')[1]
print('prefs', {k: p.get(k) for k in ('up_limit', 'dl_limit', 'alt_up_limit', 'alt_dl_limit', 'scheduler_enabled', 'limit_tcp_overhead', 'limit_lan_peers', 'limit_utp_rate', 'max_uploads', 'enable_upload_suggestions')})
print('uploadLimit', req('GET', '/api/v2/transfer/uploadLimit')[1], 'speedLimitsMode', req('GET', '/api/v2/transfer/speedLimitsMode')[1])
elif mode == 'add':
bnd = uuid.uuid4().hex
tor = open('/root/uwtest.torrent', 'rb').read()
parts = []
for k, v in (('savepath', '/mnt/felhom-drives/scratch_hdd/userdata/media/torrents'), ('skip_checking', 'true'), ('category', '')):
parts.append(f'--{bnd}\r\nContent-Disposition: form-data; name="{k}"\r\n\r\n{v}\r\n'.encode())
parts.append(f'--{bnd}\r\nContent-Disposition: form-data; name="torrents"; filename="uwtest.torrent"\r\nContent-Type: application/x-bittorrent\r\n\r\n'.encode() + tor + b'\r\n')
parts.append(f'--{bnd}--\r\n'.encode())
print('add', req('POST', '/api/v2/torrents/add', b''.join(parts), {'Content-Type': f'multipart/form-data; boundary={bnd}'}))
time.sleep(3)
print('torrents', [(t['name'], t['state'], t['progress']) for t in req('GET', '/api/v2/torrents/info')[1]])
elif mode == 'setlimit':
print('setUploadLimit', req('POST', '/api/v2/transfer/setUploadLimit', urllib.parse.urlencode({'limit': sys.argv[3]}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
print('uploadLimit now', req('GET', '/api/v2/transfer/uploadLimit')[1], 'prefs.up_limit', req('GET', '/api/v2/app/preferences')[1].get('up_limit'))
elif mode == 'setpref':
print('setPreferences', req('POST', '/api/v2/app/setPreferences', urllib.parse.urlencode({'json': json.dumps({'up_limit': int(sys.argv[3])})}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
print('uploadLimit now', req('GET', '/api/v2/transfer/uploadLimit')[1], 'prefs.up_limit', req('GET', '/api/v2/app/preferences')[1].get('up_limit'))
elif mode == 'peer':
print('addPeers', req('POST', '/api/v2/torrents/addPeers', urllib.parse.urlencode({'hashes': 'all', 'peers': sys.argv[3]}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
print('reannounce', req('POST', '/api/v2/torrents/reannounce', urllib.parse.urlencode({'hashes': 'all'}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
elif mode == 'post':
print(sys.argv[3], req('POST', sys.argv[3], urllib.parse.urlencode(json.loads(sys.argv[4])).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
elif mode == 'sample':
dur, step = int(sys.argv[3]), int(sys.argv[4])
t0 = time.time(); prev = None
while time.time() - t0 <= dur:
ti = req('GET', '/api/v2/transfer/info')[1]
tor = req('GET', '/api/v2/torrents/info')[1]
etx, ecpu = engine_stats()
row = {'t': round(time.time() - t0, 1), 'qb_up_speed': ti['up_info_speed'], 'qb_up_data': ti['up_info_data'],
'qb_tor_uploaded': sum(x.get('uploaded', 0) for x in tor), 'proc_tx': netdev(), 'engine_tx': etx, 'cli': cli_stats()}
if prev:
dt = row['t'] - prev['t']
row['rate_qb_data_Mbit'] = round((row['qb_up_data'] - prev['qb_up_data']) * 8 / dt / 1e6, 2)
row['rate_proc_Mbit'] = round((row['proc_tx'] - prev['proc_tx']) * 8 / dt / 1e6, 2)
row['rate_engine_Mbit'] = round((row['engine_tx'] - prev['engine_tx']) * 8 / dt / 1e6, 2)
print(json.dumps(row), flush=True); prev = row
time.sleep(step)
@@ -0,0 +1,32 @@
import sqlite3,sys,collections
def runs(r, pred):
best=cur=0; bt=None; prev=None
for t,v in r:
if pred(v) and prev is not None and t-prev<=180: cur+=1
elif pred(v): cur=1
else: cur=0
if cur>best: best=cur; bt=t
prev=t
return best,bt
import datetime
for box in sys.argv[1:]:
db=sqlite3.connect(f'file:{box}/metrics.db?mode=ro',uri=True)
nc=None
rows=collections.defaultdict(list)
for ts,name,cpu,tx in db.execute('select ts,container_name,cpu_percent,net_tx_bytes from container_metrics order by ts'):
rows[name].append((ts,cpu,tx))
print('==',box.rsplit('/',1)[1])
out=[]
for name,r in rows.items():
cpu=[(t,c) for t,c,_ in r]
tx=[]
for (t0,_,a),(t1,_,b) in zip(r,r[1:]):
if a is None or b is None or b<a or not 0<t1-t0<=180: continue
tx.append((t1,(b-a)*8/(t1-t0)/1e6))
c150,ct=runs(cpu,lambda v:v>=150); c80,_=runs(cpu,lambda v:v>=80)
u5,ut=runs(tx,lambda v:v>=5); u1,_=runs(tx,lambda v:v>=1)
if max(c80,u1)>=3: out.append((name,c80,c150,ct,u1,u5,ut))
print(f'{"container":28} cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]')
for name,c80,c150,ct,u1,u5,ut in sorted(out,key=lambda x:-max(x[1],x[4])):
f=lambda t: datetime.datetime.utcfromtimestamp(t).strftime('%m-%d %H:%M') if t else '-'
print(f'{name[:28]:28} {c80:8} {c150:9} ({f(ct)}) {u1:9} {u5:9} ({f(ut)})')
@@ -0,0 +1,21 @@
# Minimal BitTorrent HTTP tracker for a closed scratch test (no internet). Compact peer lists.
import http.server, urllib.parse, socket, struct, time
peers = {} # info_hash -> {(ip,port): last_seen}
def benc(o):
if isinstance(o, int): return b"i%de" % o
if isinstance(o, bytes): return b"%d:%s" % (len(o), o)
if isinstance(o, str): return benc(o.encode())
if isinstance(o, dict): return b"d" + b"".join(benc(k) + benc(v) for k, v in sorted(o.items())) + b"e"
class H(http.server.BaseHTTPRequestHandler):
def do_GET(self):
q = urllib.parse.parse_qs(urllib.parse.urlsplit(self.path).query.encode('latin-1').decode('latin-1'), encoding='latin-1')
ih = q.get('info_hash', [''])[0]; port = int(q.get('port', ['0'])[0]); ip = self.client_address[0]
d = peers.setdefault(ih, {}); now = time.time()
if q.get('event', [''])[0] == 'stopped': d.pop((ip, port), None)
else: d[(ip, port)] = now
lst = b"".join(socket.inet_aton(i) + struct.pack(">H", p) for (i, p), t in d.items() if now - t < 600 and (i, p) != (ip, port))
body = benc({"interval": 30, "peers": lst})
print(time.strftime('%H:%M:%S'), 'announce', ip, port, q.get('event', [''])[0], 'peers', len(lst) // 6, flush=True)
self.send_response(200); self.send_header('Content-Length', str(len(body))); self.end_headers(); self.wfile.write(body)
def log_message(self, *a): pass
http.server.ThreadingHTTPServer(('0.0.0.0', 6969), H).serve_forever()
@@ -0,0 +1,47 @@
# Runs INSIDE guest 9202. Transmission's own RPC numbers vs the container's exact tx bytes. Password read at use, never printed.
import json, subprocess, sys, time, urllib.request, base64
def sh(*a): return subprocess.run(a, capture_output=True, text=True).stdout.strip()
C = 'transmission'
IP = sh('docker', 'inspect', '-f', '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}', C).split()[0]
PID = sh('docker', 'inspect', '-f', '{{.State.Pid}}', C)
PW = sh('docker', 'exec', C, 'printenv', 'PASS')
AUTH = 'Basic ' + base64.b64encode(('admin:' + PW).encode()).decode()
SID = ''
def rpc(method, args=None):
global SID
for _ in range(2):
r = urllib.request.Request(f'http://{IP}:9091/transmission/rpc', data=json.dumps({'method': method, 'arguments': args or {}}).encode(),
headers={'Authorization': AUTH, 'Host': 'transmission:9091', 'X-Transmission-Session-Id': SID, 'Content-Type': 'application/json'}, method='POST')
try:
with urllib.request.urlopen(r, timeout=20) as resp: return json.loads(resp.read())
except urllib.error.HTTPError as e:
if e.code == 409: SID = e.headers.get('X-Transmission-Session-Id'); continue
return {'http': e.code}
def tx():
for line in open(f'/proc/{PID}/net/dev'):
if line.strip().startswith('eth0:'): return int(line.split(':', 1)[1].split()[8])
mode = sys.argv[1]
if mode == 'info':
g = rpc('session-get')['arguments']
print('version', g.get('version'), {k: g.get(k) for k in ('speed-limit-up', 'speed-limit-up-enabled', 'alt-speed-up', 'alt-speed-enabled', 'alt-speed-time-enabled', 'units')})
print('session-stats', rpc('session-stats')['arguments'])
elif mode == 'add':
meta = base64.b64encode(open('/root/uwtest.torrent', 'rb').read()).decode()
print(rpc('torrent-add', {'metainfo': meta, 'download-dir': '/mnt/felhom-drives/scratch_hdd/userdata/media/transmission', 'paused': False}))
time.sleep(5); rpc('torrent-verify', {}); time.sleep(20)
print(rpc('torrent-get', {'fields': ['name', 'status', 'percentDone', 'error', 'errorString']}))
elif mode == 'limit':
print(rpc('session-set', {'speed-limit-up': int(sys.argv[2]), 'speed-limit-up-enabled': True}))
g = rpc('session-get')['arguments']; print({k: g.get(k) for k in ('speed-limit-up', 'speed-limit-up-enabled')})
elif mode == 'reannounce':
print(rpc('torrent-reannounce', {}))
elif mode == 'sample':
dur, step = int(sys.argv[2]), int(sys.argv[3]); t0 = time.time(); prev = None
while time.time() - t0 <= dur:
st = rpc('session-stats')['arguments']; tg = rpc('torrent-get', {'fields': ['uploadedEver', 'rateUpload']})['arguments']['torrents']
row = {'t': round(time.time() - t0, 1), 'tr_upspeed': st['uploadSpeed'], 'tr_cur_up': st['current-stats']['uploadedBytes'], 'tr_tor_up': sum(x['uploadedEver'] for x in tg), 'tx': tx()}
if prev:
dt = row['t'] - prev['t']
row['rate_tr_Mbit'] = round((row['tr_cur_up'] - prev['tr_cur_up']) * 8 / dt / 1e6, 2)
row['rate_container_Mbit'] = round((row['tx'] - prev['tx']) * 8 / dt / 1e6, 2)
print(json.dumps(row), flush=True); prev = row; time.sleep(step)
@@ -0,0 +1,236 @@
# Catalog sweep 2026-10-24 — every catalog app on a fresh box, the way a family uses it
Unattended validation sweep (Saturday 12:30Z → Sunday ~00:00Z). Brief: the operator's „every catalog app, one by one, on
a fresh box" prompt. Venue: demo-hp (Tier 0). Method: endpoint-level — every step is the request the dashboard or the
app's own page sends (no browser click-through except the screenshot harness; no `docker exec` to make state).
## Safest readings taken (the brief allows no question)
1. **ISO 1.30.0, not 1.36.0.** The brief said „1.36.0 or the current one"; the current one on felhom.eu/letoltes was
1.30.0 (sha256 checked against the published `.sha256`). See `_box/README.md`.
2. **12 GiB for the VM, not 16.** demo-hp had 14 GiB free beside 9201, 9202 and Tester 1; 12 is the brief's minimum.
The guest itself is 7.7 GiB (the golden's size); the box's own memory admission decided what could run.
3. **„Remove keeping data → install again with use my kept data" is a drive-folder question.** By design (`07` §6.5)
the question is asked only for an app that keeps a folder on a drive (`HDD_PATH`). For the 40+ apps that keep their
data in Docker volumes, „keep my data" keeps the recovery unit; the reinstall starts empty, and the item comes back by
the unit restore („Visszaállítás"). So step 5 + 6 ran as: keep-remove → reinstall → read (empty, expected) → unit
restore → read. That is a unit restore on **every** app with a seed, not only every fifth.
4. **„Mentés most" is the local copy only** (`07` §8, by design); the second-drive and off-site copies run in the night.
The sweep did not wait for a night, so „the right tiers ran" is checked as „the local unit holds the app".
5. **Categories:** the brief names „Apps & catalog" for bigger issues; a row whose harm is in backup/restore or storage is
filed in that category instead (`register_shape_gate` has one category per row, and the owner reads it there).
6. **Sev:** the brief's scale (data at risk P1/P2, a broken app P2, a confusing step P3). R-1269, R-1271 are P2 by that rule (the data is not deleted in either; the household loses access to it).
7. **The sweep box reads the LIVE catalog** (main `5e196e1`, then `227a544` — rules only, no template moved). The drill
catalog was used only for the three apps whose fix needed proving; the box was pointed back after (`_drill-repoint.txt`).
8. **Harness fixes live in the sweep's copy** (`tools/`), not in the catalog's scripts; porting them is R-1277.
## The box (Part 0)
`_box/README.md`: VM **9360** `sweep1024-box` (6 cores, 12 GiB, disks 256 + 200 + 150 GB on `nvme-scratch`), ISO 1.30.0,
customer Tester4 (hub host `Tester4-aa9466`), guest 9201, domain `tester4.test` (no DNS: the browser LXC maps the
names). Bind 12:42:18Z, day-0 SUCCESS 12:45:40Z, claim, recovery code escrowed (stored out-of-band), drives `hdd_1`
(210 GB) and `hdd_2` (157 GB) plus the installer's internal place `belso`. Versions at the start: controller 0.342.0,
agent 0.164.0, hub 0.169.0. Browser: LXC **9361** `sweep1024-browser` (Chromium + Playwright), 60 dashboard screens
(15 pages × phone/computer × HU/EN) in `_box/screens-dashboard/`.
## Method per app (Part 1)
`tools/sweep.py <app>` — one app at a time, small first, heavy last (`tools/order.txt`); the next starts after the last
is removed with its data. Each `<app>/` holds `record.json` (the measured facts), `walk.log`, the page texts
(`page-{install,guide,overview}-{hu,en}.txt`) and `screens/` (phone + computer, HU + EN, install page and the app after
first login). Steps: install through `POST /api/stacks/<app>/deploy` (timed to the front door answering); a stranger
request without the family cookie (must be refused where the template gates); one item through the app's own API (the
catalog's fixture where one exists, a new one in `tools/sweep_fixtures.py` for seven more); „Mentés most"; remove
keeping data; reinstall (the kept-data question answered „use" where it is asked); read; unit restore; read; remove with
data; leftovers checked (containers, volumes, `app.yaml`, `appdata`, `kept`, the subdomain → 404). Memory: the peak of
each container's **anon** memory from its cgroup (`memory.stat`; the file cache is not counted — project rule) against
its compose limit; a `*` in the table means only `docker stats` was read for that app (it includes the cache). The anon reading joined mid-sweep, so 22 apps carry it and 40 carry the `docker stats` upper bound; of those 40 none passed 79 % of its limit (Tandoor 1009/1280) and none restarted. Above 80 % on anon: only OpenGist (F14).
## Results
<!-- TABLE START -->
| App | Verdict | Peak memory / limit (MiB) | Install → answering (s) | What we saw | Row / fix |
|---|---|---|---|---|---|
| actualbudget | **works** | 49* / 256 | 30 | the seeded item back after the restore | |
| adventurelog | **works with a problem** | 381 / 896 | 207 | works; but the keep-backups → reinstall → restore path breaks it (DB password from the new install over the old data) — reproduced in its re-run | R-1269, R-1270 |
| audiobookshelf | **works** | 56* / 512 | 41 | the seeded item back after the restore | drive names: R-1275 |
| bentopdf | **works** | 3* / 384 | 30 | stateless (nothing to keep); installs, answers, removes clean | |
| bookstack | **works** | 167* / 768 | 71 | the seeded item back after the restore | |
| calcom | **works** | 807 / 1792 | 278 | the seeded item back after the restore; 2 start-up messages of the image itself | |
| calibre-web | **works** | 211 / 768 | 76 | the book (its own upload) is listed again after the keep-data reinstall; the fixture's download check gets 401 (its sign-in for /download, not the app); the admin name is the box's generated one (re-run 2) | R-1277 |
| claper | **works** | 267* / 640 | 86 | the seeded item back after the restore | |
| code-server | **works** | 81 / 1024 | 46 | first install refused by the registry's rate limit, told as „exit code 1”; the re-run installed and worked (no seed: the editor has no API) | R-1273 |
| crafty-controller | **works** | 79* / 2048 | 76 | the seeded item back after the restore | |
| dawarich | **works** | 835* / 2688 | 136 | the seeded item back after the restore (every-5th: restore also before the reinstall) | |
| docmost | **works** | 528* / 1152 | 126 | the seeded item back after the restore | |
| emby | **works** | 95 / 2048 | 41 | installs, answers, removes clean; no seed (no fixture for its wizard) | |
| ghost | **works** | 167* / 512 | 121 | the seeded item back after the restore | |
| gitea | **works** | 112 / 512 | 25 | the seeded item back after the restore | |
| glance | **works** | 7* / 128 | 15 | family gate holds a stranger; config-only (nothing to keep); the app's own headings are English | |
| gokapi | **works** | 6 / 128 | 15 | installs, own login, removes clean; no seed (API key only in its UI) | |
| grafana | **works** | 383* / 768 | 61 | the seeded item back after the restore | |
| gramps-web | **works** | 347* / 1024 | 206 | installs, answers, removes clean; no seed fixture | |
| grimmory | **works with a problem** | 594 / 1408 | 96 | first admin, library and an EPUB upload work; setup gate then family gate work as designed; the read-back through the family gate stayed inconclusive (the harness's gate logic, R-1277); remove-with-data leaves the empty parent folder | R-1276, R-1277 |
| grocy | **works** | 34* / 384 | 36 | the seeded item back after the restore | |
| home-assistant | **works** | 311 / 1024 | 161 | the seeded item back after the restore | |
| homebox | **works** | 205 / 256 | 20 | the seeded item back after the restore | |
| homepage | **works with a problem** | 91* / 256 | 25 | no gate at all: a stranger sees the box's CPU/RAM/disk (and later the family's links); still the upstream sample page | R-1272 |
| immich | **works with a problem** | 1845 / 6016 | 121 | album (new fixture) in; „use my kept data” loaded the rehearsal's future-dated database and Immich never answered again (404, 20 min) | R-1271 |
| jellyfin | **works** | 110 / 2048 | 41 | first user (new fixture, its wizard API) back after the restore | |
| karakeep | **works** | 649* / 2816 | 107 | the seeded item back after the restore | |
| kimai | **works** | 303* / 640 | 111 | own login; a Hungarian browser lands on /hu/ (curl without a language gets /en/) | |
| komga | **works** | 473* / 768 | 46 | the seeded item back after the restore | drive names: R-1275 |
| lubelogger | **works** | 60* / 512 | 25 | the seeded item back after the restore | |
| mealie | **works** | 322 / 1000 | 72 | the seeded item back after the restore (the fixture raced after_install on the first run) | |
| metube | **works** | 90* / 768 | 61 | family gate holds a stranger; download queue entry back | |
| n8n | **works with a problem** | 346 / 1536 | 151 | the owner back after the restore; on the live template it tries to send usage data out ([Rudder] lines, 6 in the live re-run) — the drill fix: the variable in the running container, 0 error lines | drill a601950 (morning list) |
| navidrome | **works** | 23* / 256 | 25 | the seeded item back after the restore | drive names: R-1275 |
| nextcloud | **works with a problem** | 298 / 1664 | 122 | user in; „use my kept data” loaded the rehearsal's future-dated database beside today's files — the user was not there | R-1271 |
| nzbget | **works** | 6* / 512 | 25 | the seeded item back after the restore; „no news server” until the family adds one (the guide says so) | |
| onlyoffice | **works** | 464 / 2048 | 71 | document server (no household data to keep); installs, answers, removes clean (every-5th restore) | |
| opengist | **works with a problem** | 125 / 128 | 15 | the gist owner back after the restore; but 128M is too small: one sign-up + sign-in hit the limit 443 times and swapped 98 MiB (memprobe-128M.txt); at 256M (drill) anon 216 MiB | drill 405077f (morning list) |
| outline | **works** | 327 / 1152 | 151 | the seeded item back after the restore | |
| paperless-ngx | **works with a problem** | 669* / 1664 | 146 | account in; „use my kept data” loaded the rehearsal's future-dated database — the account was not there | R-1271, R-1276 |
| papra | **works** | 320 / 768 | 76 | the seeded item back after the restore (Papra 26.6 needs an Origin header — a browser sends it) | |
| plant-it | **not installed** | – | – | lifecycle: abandoned — not offered for a new install (by design) | |
| plex | **works** | 223 / 2048 | 30 | installs, answers, removes clean; no seed (a server is claimed with a plex.tv account) | |
| privatebin | **works** | 31* / 128 | 15 | the seeded item back after the restore; D22 put-back PASS | |
| prowlarr | **works** | 114* / 512 | 21 | the seeded item back after the restore | |
| qbittorrent | **works** | 77* / 1024 | 15 | the seeded item back after the restore | |
| radarr | **works** | 100* / 512 | 41 | the seeded item back after the restore | |
| radicale | **works** | 33* / 128 | 25 | the seeded item back after the restore | |
| rallly | **works** | 346 / 1024 | 71 | installs, answers, removes clean; no seed: Rallly 4.15 closed password sign-up (login by e-mail code); the guide's polls need no account | R-1277 |
| recipe-importer | **works** | 67* / 128 | 25 | stateless helper; installs, answers, removes clean (every-5th restore) | |
| romm | **works with a problem** | 748* / 1536 | 116 | item back through „use my kept data” (its own copy); remove-with-data left the empty parent folder | R-1276 |
| sabnzbd | **works** | 69* / 1024 | 25 | the seeded item back after the restore | |
| seerr | **works** | 166 / 384 | 92 | installs, answers (307 to its setup), removes clean; no seed (needs a media server) | |
| sonarr | **works** | 80* / 512 | 41 | the seeded item back after the restore | |
| sparkyfitness | **works** | 417* / 1792 | 91 | the seeded item back after the restore | |
| tandoor | **works with a problem** | 1052* / 1536 | 177 | works; but the keep-backups → reinstall → restore path breaks it (DB password from the new install over the old data), and the restore still said success — reproduced on its own | R-1269, R-1270 |
| termix | **works** | 111* / 256 | 61 | installs, answers, removes clean (every-5th restore); no seed fixture | |
| transmission | **works** | 6* / 512 | 20 | the seeded item back after the restore | |
| uptime-kuma | **works** | 120* / 384 | 76 | the seeded item back after the restore | |
| vaultwarden | **works** | 37 / 256 | 25 | installs in 25 s, its own login page, sign-up closed (404); the box admin's invite works (run-live-2), but 1.37.4 has no route for the fixture's invited sign-up, so no item was seeded | R-1277 |
| vikunja | **works** | 37* / 256 | 10 | the seeded item back after the restore; D22 put-back PASS | |
| wanderer | **not testable on this box** | 140* / 1024 | 605 | not testable on this box: its web part calls its database at the PUBLIC name, which tester4.test cannot resolve (known, R-739) — unhealthy, 404 | R-739 |
| wger | **not installed** | – | – | lifecycle: hidden — not offered for a new install (by design) | |
| wishlist | **works** | 200* / 512 | 71 | the seeded item back after the restore | |
| zipline | **works** | 233* / 768 | 81 | the seeded item back after the restore | |
works: 52, works with a problem: 10, not installed: 2, not testable on this box: 1 (of 65)
<!-- TABLE END -->
## Findings — each a row, a fix or NOT-A-FINDING
| # | What | Disposition |
|---|---|---|
| F10 | A reinstall's own capture rewrites the unit that „remove — keep backups" kept; restoring it pairs the NEW install's DB password with the OLD data — the app never starts, the item is gone (Tandoor ×2, AdventureLog) | R-1269 (P2) |
| F11 | The unit restore reports `ok: true` „…az alkalmazás újraindult" while the controller logged „health check failed … did not reach running state" | R-1270 (P2) |
| F12 | „Use my kept data" loaded the rehearsal box's future-dated off-site database (2026-10-28) for Paperless-ngx, Nextcloud and Immich — another drive's install, which `07` §6.5 says is refused; Immich then never answered | R-1271 (P2) |
| F1 | Homepage has no gate: a stranger sees the box's CPU/RAM/disk (Glance, same class, is gated) | R-1272 (operator decision) |
| F7 | An install refused by the registry's rate limit (`toomanyrequests`) tells the household only „exit code 1" | R-1273 |
| F8 | The restore refusal says „turn on remote backup, or connect a second drive" when both exist but have not copied yet | R-1274 |
| F2 | English pages show the drive names in Hungarian („Belső tárhely", „Külső tárhely (hdd_1)") | R-1275 |
| F9 | „Remove with data" leaves the empty parent folder `appdata/<app>` (Grimmory, RomM, Paperless-ngx) | R-1276 |
| — | The catalog's box fixtures drifted from the apps (OpenGist, Calibre-Web, Mealie, Papra, Rallly; seven apps had none) | R-1277 |
| F13 | n8n sends usage data (`[Rudder]` lines; no `N8N_DIAGNOSTICS_ENABLED=false`) | **fixed in the drill** `a601950`, proven below; morning list |
| F14 | OpenGist at 125.8 of 128 MiB anon after one sign-up + sign-in; at 128M it hits the limit 443 times and swaps 98 MiB | **fixed in the drill** `405077f` + `492a17f` (256M), proven below; morning list |
| — | 52 anonymous Docker volumes stayed after every app was removed with data (PrivateBin, OnlyOffice, RomM, Immich: the images' own `VOLUME` paths; 4–72 KB, no household data). The sweep's own leftover check filters by the app's label and missed them; found in the end-state read (`_box/anon-volumes.txt`) | known: note on R-1267 (widened to OnlyOffice and RomM) |
| — | Wanderer unhealthy, 404: its web part calls its database by the public name, which a box without DNS cannot resolve | known: note on R-739 |
| F3 | The install page's default place is the internal place `belso`, not the 210 GB drive | NOT-A-FINDING: by design — `16` §5 registers the internal place as the default drive |
| F4 | Guide labels in English (LubeLogger „Settings", NZBGet „Settings / News-Servers") | NOT-A-FINDING: they are the app's own on-screen labels, and the Hungarian catalog copy is frozen by the copy-i18n gate (measured on the drill) |
| F5 | Kimai opens in English | NOT-A-FINDING: a Hungarian browser lands on `/hu/`; only a request with no language gets `/en/` |
| F6 | AdventureLog backend 95 % of its limit | NOT-A-FINDING: that was `docker stats` with the cache; on anon 288.7/384 MiB (75 %), = the ladder's 73 % |
| — | Database error lines (SparkyFitness `database "sparky" does not exist`, Dawarich `database "dawarich" already exists`; SparkyFitness `Invalid password` = the harness's wrong-password control) | NOT-A-FINDING: the images' own first-start and restore-time checks; no restart, the item read back after the restore |
| — | Radicale redirects to an IP | NOT-A-FINDING: a curl artefact (no Host on the redirect probe); the browser lands right |
| — | NZBGet / Transmission screenshots show the browser's basic-auth prompt | NOT-A-FINDING: the app's own login, by design |
| — | code-server's first install failed (`toomanyrequests`) | the failed install was removed through the product (`code-server.failed-remove.txt`); the re-run worked; the message is R-1273 |
## F10 proof — the kept unit is rewritten by the reinstall (`_unitprobe/tandoor/`)
`tools/unitprobe.py tandoor`: install, one account, „Mentés most" (unit 19:45:14–20), remove keeping data and
backups, install again (a volume app: no kept-data question), wait for the controller's next capture, „Visszaállítás".
- 19:50:32 `Recovery unit captured for tandoor` — the unit's `app.yaml` checksum `0f366c97…` → `4438c1c5…`,
`created_at` 19:45:26 → 19:50:32; the data files unchanged (`unit-1/2/3*.txt`).
- The restore: `ok: true`, „A(z) tandoor: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult."
- After: state `starting`, the account not there, `password authentication failed` ×42 (app) and ×22 (postgres).
- The comment at `felhom-controller/controller/internal/backup/recovery_unit.go:101-103` says the refresh never rewrites
the definition away from its data; after a remove + reinstall it does.
Seen three times: the sweep's own Tandoor cycle (17:27:41 reinstall → 17:30:34 capture → 17:30:47 restore), this probe,
and AdventureLog's re-run (20:03:18 → 20:05:32 → restore). AdventureLog's first run worked only because no capture fell
between its reinstall and its restore.
## D22 put-back — PASS on two apps (`_putback/`)
`tools/putback.py <app>`: the largest volume tar of the app's unit truncated to half, then „Visszaállítás". PrivateBin
and Vikunja: the restore failed honestly („Nem változott semmi…"), the pre-restore data was put back, the app ran, and the
seeded item read back through the front door.
## Drill fixes, proven on the sweep box pointed at the drill (Part 2)
Drill catalog `app-catalog-drill` (never the live catalog): `742a354` (merge of live main), then:
- `a601950` **n8n** — `N8N_DIAGNOSTICS_ENABLED=false` in the compose (usage data stays home).
- `405077f` + `492a17f` **OpenGist** — memory 128M → 256M (compose + `.felhom.yml` `mem_limit`), the comment carries the
measured need.
**How the box was pointed at the drill.** The drill repo on Gitea is private (R-992) and the box holds no catalog
credential, so the box reads a local mirror: a bare repository in the controller's data volume, loaded from a git bundle,
`repo_url: file:///opt/docker/felhom-controller/drill-catalog.git` (the method of
`audits/burndown-3-2026-10-17/tools/repoint.py`; loading the mirror is a `docker exec git fetch` into the controller —
catalog plumbing, not app state). Only `repo_url` changes; `tools/repoint.py live` puts the saved `controller.yaml` back
and checks it byte for byte (`RESTORED-IDENTICAL`, twice in `_drill-repoint.txt`). The box ended on the live catalog
`227a544`.
**What went wrong first, and is kept as evidence.** The first drill pass (chain 3, 21:18Z) set `repo_url` to the Gitea
URL in the belief that the repo was public. The box's clone failed (`fatal: could not read Username`), the controller kept
the templates it had, and n8n, OpenGist and Vaultwarden ran on the LIVE templates. The leftover `[Rudder]` lines in that
n8n run showed it. Those three runs are kept as `run-live-2/` (valid live re-runs). The second pass (chain 4) refused to
install anything until the box's catalog cache read the drill's commit and the changed lines were in the box's copy of
both templates: `drill main fd23571, box cache fd23571: DRILL-CONFIRMED`. (The drill also carries a six-line Plex
ladder entry from an earlier session's test; Plex was not installed in this pass.)
**n8n proof.** The running container carried `N8N_DIAGNOSTICS_ENABLED=false` (read at 21:37:58, that line only); error
lines 0 against 6 `[Rudder]` lines in the live re-run just before it (the control); the owner account read back after the
unit restore.
**OpenGist proof — and a wrong turn.** At 256M (drill) OpenGist took 215.8 MiB anon. I first read that as „it grows into
its limit" and reverted the fix (`c420053`); an automated security review flagged the revert as a lowered resource cap.
A measurement settled it (`tools/memprobe-guest.sh`, `opengist/memprobe-128M.txt`): on the live 128M template one
sign-up + sign-in hit `memory.max` 443 times and pushed 98 MiB into swap (anon 125 MiB, 0 `oom_kill`, 0 restarts). The
real need is ~220 MiB; the fix went back in (`405077f`). Where the box has no swap room left, the same sign-up would end in a kill (not measured here).
Drill gates: `catalog_gates.py --fast opengist n8n` from the drill clone — all OK except `security-scan`, which looks for
a `felhom.eu` clone beside the drill and found none; the shared scanner run by hand on the drill clone: 0 hits.
## Teardown
Three layers, 2026-10-25 ~00:05Z:
- **Machine (demo-hp):** VM **9360** `sweep1024-box` and LXC **9361** `sweep1024-browser` — both created by this session —
stopped and destroyed with `--purge`; no disk of either is left on `nvme-scratch`. Untouched: Tester 1 (VM 341), 9201,
9202, 9401, the thin pool, and the 1.30.0 ISO in `local:iso` (there since 2026-10-22). Before the destroy, the box's
controller log, the agent journal, its end state and its anonymous-volume list were copied here (`_box/`).
- **Host (DooPlex):** the session's secret files (`/run/user/1000/sweep-sc`: the dashboard password copy, cookies, CSRF
tokens) shredded. The drill catalog keeps its commits (`a601950`, `405077f`, `492a17f`, and the reverted turn
`c420053`) for the morning list; nothing went to the live catalog.
- **Hub:** host **Tester4-aa9466** stays for the operator to delete (CC holds no delete). Until then it reads as a box
that stopped reporting.
- **ep0:** listed, nothing removed. The host's PBS link to ep0 (`felhom-pbs`) was set up at day-0 (`pbsdr: converged`,
14:44:48 CEST, `_box/agent-journal.log.gz`); no whole-box copy went to it (weekly cadence — only the local copy ran,
14:51). The app copies the box sent off-site went to the Tester4 restic repository on the storage box (unlocked with
the rehearsal's recovery code; a new code escrowed — stored out-of-band), not to ep0.
## Evidence map
- `_box/` — Part 0 (VM, ISO, bind, day-0, claim, recovery, drives, dashboard screens).
- `<app>/` — one per app; re-run apps keep the earlier run in `first-run/`, `run-2/` or `run-drill-pre/`.
- `<app>.console.txt`, `progress.log`, `batch.out`, `chain2.out`, `chain3.out` — the runner's own output.
- `_unitprobe/tandoor/` — F10. `_putback/{privatebin,vikunja}/` — D22.
- `tools/` — the harness (sweep copies of the catalog's `box_walk.py` and `upgrade_fixtures_box.py`, marked SWEEP COPY;
`sweep.py`, `sweep_fixtures.py`, `unitprobe.py`, `putback.py`, `repoint.py`, `table.py`, the rehearsal's VM and browser
helpers). Secrets were read by name through `sec` and never written here; the evidence was scanned for every known
secret value and for token shapes (one app session token found and blanked: Actual Budget's), with a positive control.
- Screenshots were reduced with `pngquant` (lossy palette) to keep the repository small.
@@ -0,0 +1,20 @@
# Part 0 — the sweep box (2026-10-24, from 12:33Z)
| | |
|---|---|
| VM | **9360 `sweep1024-box`** on demo-hp: 6 cores (`cpu host`), **12 GiB** (demo-hp had 14 GiB available of 29; 9201, 9202 and Tester 1 untouched — the brief's minimum, not its 16 GiB aim), 256 GB system disk + 200 GB + 150 GB data disks, all raw on `nvme-scratch` (the dir storage), never the thin pool (`vm-create.txt`) |
| ISO | `felhom-installer-1.30.0-pve9.2-1.iso` — the current one on felhom.eu/letoltes; sha256 `590d7b9b…74d5cc1` = the published `.sha256` |
| Installer | graphical, Hungarian keyboard kept, driven by `qm sendkey` (`tools/vmctl.sh`), screens in `install/`. Install pressed **12:37:33Z**; pairing code on the console **12:41** (blacked out in `install/11-console.png`) |
| Bind | Tester4's old host deleted by the operator (12:32, asked by CC); the hub mailed a new bind link 12:31:57Z; bound through the real page in Chromium 12:42:18Z, first try (`screens/b-bind-*`) |
| Day-0 | 8/8 steps, **SUCCESS 12:45:40Z** — host `Tester4-aa9466`, guest 9201, golden imported and verified; Install → done **8 min 7 s** |
| Claim | setup-code mail 12:43:14Z („Új beállító kód — újratelepült a szervered", right for a customer who had a box before); claimed 302 → `/` (`claim.txt`) |
| Recovery | the box found Tester4's sealed off-site package (2026-10-20 14:16) and asked for the old code first (`/recovery`): the 2026-10-20 code, typed as it was written down (ten words joined by hyphens), unlocked it — controller „the offsite repository key was recovered and placed" 12:47:41Z. Then the dashboard asked for a NEW code: the ceremony in Chromium (words 4 and 9 asked), escrowed; „Távoli mentés: Aktív" (`screens/box-escrow-*`, the code blacked out) |
| Drives | `/dev/sdb` → `hdd_1` (210 GB, the apps' data), `/dev/sdc` → `hdd_2` (157 GB), both through `POST /api/storage/init` (`drive-init.txt`); the installer's internal place `belso` (59 GB) |
| Versions | controller **0.342.0**, agent **0.164.0**, hub **0.169.0**; the box syncs the LIVE catalog (`app-catalog-felhom.eu` main, `5e196e1` at the start; it re-syncs every 15 min) |
| Guest | 9201 at 192.168.0.157, 7.7 GiB RAM (the golden's size), domain `tester4.test` (no DNS — the browser LXC maps the names; TLS not verified) |
| Browser | LXC **9361 `sweep1024-browser`** (2 cores, 2 GB, nvme-scratch): Chromium 156.0.8078.47 + Playwright 1.52.0; the household's session stored in the LXC (`/root/work/state-dash.json`) |
| Dashboard | every menu page at phone and computer size, HU and EN (`screens-dashboard/`, 60): no sideways scroll reported; a signed-in household reads its OWN language setting whatever the browser asks — by design (`langFor`), the page texts were read with `?lang=` |
Secrets used (names only, `sec`): SWEEP_2026_10_24_ROOTPW (the typed root password — day-0 replaced it with the vaulted
one; DooPlex's key was added before the bind), SWEEP_2026_10_24_BIND_TOKEN, _PAIRING_CODE, _SETUP_CODE, _DASH_PW,
_RECOVERY_CODE, and the existing REHEARSAL_2026_10_20_PASSPHRASE / _RECOVERY_CODE.
@@ -0,0 +1,52 @@
2026-10-24T17:15:18Z 28 assets,config,library,
2026-10-24T12:51:57Z 24 client_body,fastcgi,proxy,
2026-10-24T13:08:14Z 24 client_body,fastcgi,proxy,
2026-10-24T19:54:35Z 4
2026-10-24T17:15:18Z 4
2026-10-24T12:51:49Z 24 client_body,fastcgi,proxy,
2026-10-24T19:54:18Z 4
2026-10-24T13:06:43Z 24 client_body,fastcgi,proxy,
2026-10-24T13:08:05Z 72 lock,nginx,nginx.pid,
2026-10-24T19:06:36Z 36 documentserver,documentserver-example,
2026-10-24T13:06:43Z 4
2026-10-24T13:08:05Z 24 client_body,fastcgi,proxy,
2026-10-24T13:06:43Z 72 lock,nginx,nginx.pid,
2026-10-24T19:54:18Z 24 client_body,fastcgi,proxy,
2026-10-24T17:18:02Z 4
2026-10-24T13:07:11Z 24 client_body,fastcgi,proxy,
2026-10-24T12:50:39Z 72 lock,nginx,nginx.pid,
2026-10-24T13:07:11Z 72 lock,nginx,nginx.pid,
2026-10-24T19:09:38Z 36 documentserver,documentserver-example,
2026-10-24T17:20:41Z 28 assets,config,library,
2026-10-24T19:54:35Z 24 client_body,fastcgi,proxy,
2026-10-24T12:50:39Z 24 client_body,fastcgi,proxy,
2026-10-24T12:51:57Z 72 lock,nginx,nginx.pid,
2026-10-24T19:07:30Z 4
2026-10-24T12:50:57Z 24 client_body,fastcgi,proxy,
2026-10-24T19:21:09Z 4
2026-10-24T19:09:38Z 4
2026-10-24T12:50:57Z 72 lock,nginx,nginx.pid,
2026-10-24T19:20:01Z 4
2026-10-24T19:55:03Z 4
2026-10-24T12:51:57Z 4
2026-10-24T19:54:18Z 72 lock,nginx,nginx.pid,
2026-10-24T17:18:02Z 28 assets,config,library,
2026-10-24T19:24:43Z 4
2026-10-24T12:50:39Z 4
2026-10-24T13:08:05Z 4
2026-10-24T12:51:49Z 4
2026-10-24T19:07:30Z 32 documentserver,documentserver-example,
2026-10-24T19:06:36Z 4
2026-10-24T19:07:39Z 4
2026-10-24T13:08:14Z 4
2026-10-24T17:17:24Z 4
2026-10-24T12:50:57Z 4
2026-10-24T17:17:24Z 28 assets,config,library,
2026-10-24T19:55:03Z 24 client_body,fastcgi,proxy,
2026-10-24T19:55:03Z 72 lock,nginx,nginx.pid,
2026-10-24T13:08:14Z 72 lock,nginx,nginx.pid,
2026-10-24T19:07:39Z 36 documentserver,documentserver-example,
2026-10-24T19:54:35Z 72 lock,nginx,nginx.pid,
2026-10-24T17:20:41Z 4
2026-10-24T12:51:49Z 72 lock,nginx,nginx.pid,
2026-10-24T13:07:11Z 4
@@ -0,0 +1,3 @@
GET /claim HTTP/2 200
csrf_len 64 cookie_present yes
POST /claim HTTP/2 302 location: /
@@ -0,0 +1,11 @@
candidates 200 {"data": {"vmid": 9201, "initialize": [{"device": "/dev/sdb", "size_bytes": 214748364800, "model": "QEMU HARDDISK", "data_bearing": false, "mountable": false}, {"device": "/dev/sdc", "size_bytes": 161061273600, "model": "QEMU HARDDISK", "data_bearing": false, "mountable": false}], "attach": []}, "ok": true}
12:50:04Z init /dev/sdb hdd_1 200 {"data": {"phase": "formatting", "started": true}, "ok": true}
status 200 {"device": "/dev/sdb", "durable_id": "", "error": "", "opsign": "", "phase": "done", "reason": "", "started_at": "2026-10-24T12:50:04.496400956Z", "updated_at": "2026-10-24T12:50:06.691612961Z", "where": "/mnt/felhom-drives/hdd_1"}
12:50:09Z init /dev/sdc hdd_2 200 {"data": {"phase": "formatting", "started": true}, "ok": true}
status 200 {"device": "/dev/sdc", "durable_id": "", "error": "", "opsign": "", "phase": "done", "reason": "", "started_at": "2026-10-24T12:50:09.550638661Z", "updated_at": "2026-10-24T12:50:11.762283786Z", "where": "/mnt/felhom-drives/hdd_2"}
9dcbe7bf-098d-4d99-9c43-f39fe2b20f42 disk user-data attached 157 GB /mnt/felhom-drives/hdd_2
belso internal user-data attached 59 GB /mnt/felhom-drives/belso
e8b4f4a0-7aca-4829-9220-b394948b9368 disk user-data attached 210 GB /mnt/felhom-drives/hdd_1
local system system attached 77 GB None backup_target
local-lvm system system attached 165 GB None
felhom-pbs network backup attached 0 GB None
@@ -0,0 +1,8 @@
filebrowser gtstef/filebrowser:1.5.6-stable Up 9 hours (healthy)
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0 Up 6 minutes (healthy)
traefik traefik:v3.7.13 Up 9 hours
54
Filesystem Size Used Avail Use% Mounted on
/dev/sdb 196G 268M 186G 1% /mnt/felhom-drives/hdd_1
/dev/sdc 147G 2.2M 140G 1% /mnt/felhom-drives/hdd_2
/dev/mapper/pve-vm--9201--disk--0 32G 1.2G 29G 4% /
Binary file not shown.

After

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 57 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 61 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 60 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 67 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 55 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 60 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 60 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 55 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.9 KiB

Loaded 100 of 2342 files, more files were not shown because too many files have changed in this diff. Show more