Author SHA1 Message Date
admin 79ee3e3392 Release close-out (attended, 2026-10-25): hub 0.170.0, agent 0.165.0 (vouched), controller 0.343.0 (global floor), catalog ebbf1f4; D49 = A and D44–D46 recorded (09 §3 258–261, CONTEXT, 07 §6.3/§8.2e); 9 rows closed, R-1254 hub half, R-1250 dated check 2026-10-27; R-1287–R-1290 filed; STATUS, map, REPORT-release-2026-10-26
gates / gates (push) Successful in 12m49s
2026-10-25 19:30:10 +01:00
admin 36ef8dcdec installer 1.36.1: http_code prints one status (R-1284, red-proved; committed, not published); R-1285 (BYO drop-in removed one bundle late), R-1286 (demo-hp kimai held since 10-14); release-2026-10-26 evidence: BYO proof, the three boxes on 0.343.0, Homepage gate on the drill
gates / gates (push) Successful in 12m58s
2026-10-25 18:32:04 +01:00
admin ade357a14e D44/D45 = all picks applied (R-1265, R-1266 closed; R-1280–R-1283 opened); R-232 dated check done; release-2026-10-26 evidence: fix A/B live on rc.6, D47 set-aside alarm on rc.7
gates / gates (push) Successful in 13m3s
2026-10-25 17:57:11 +01:00
admin a86ab64f6c hub v0.170.0 (R-1254: the recovery package in one transaction) — manifest bump, operator yes in the attended release run
gates / gates (push) Successful in 12m32s
2026-10-25 16:46:40 +01:00
admin 1d4abca54a Release run 2026-10-26: hub CHANGELOG names v0.170.0 (R-1254) for the build; the run's evidence folder (fix A/B red-proofs, probe)
gates / gates (push) Successful in 13m10s
2026-10-25 12:59:14 +01:00
admin 9647f1d158 Restore-truth run (2026-10-25): D47 + D48 recorded (09 §3 decisions 256, 257; CONTEXT); 07 §6.5 (R-1271 cause + the future-date rule), §6.6 (D48 order, the R-698 sentence), §8.2d (D47), §8.2e (R-1270); register 217 -> 219 (R-1278, R-1279 opened; R-1269/1270/1271 VERIFY; notes R-1206, R-1243 dated check done read-only, R-232 moved one day); STATUS D49 + Monday list; evidence (9202)
gates / gates (push) Successful in 13m44s
2026-10-25 11:04:46 +01:00
admin 67053335d1 Catalog sweep 2026-10-24: all 65 catalog apps on a fresh box (52 work, 10 with a problem, 0 broken); register 208 -> 217 (R-1269..R-1277 opened; notes on R-739, R-1267); STATUS + D46; map B1 re-checked; evidence
gates / gates (push) Successful in 13m17s
Every catalog app installed, used, backed up, removed keeping data, reinstalled, restored and removed with data on a
fresh box from the public ISO (demo-hp VM 9360, Tester4), endpoint-level. Worst: a reinstall's capture rewrites the
kept unit so its restore breaks the app (R-1269, P2), the restore still reports success (R-1270, P2), and „use my kept
data" loaded another box's future-dated database (R-1271, P2). Drill catalog fixes for the morning list (n8n usage
data; OpenGist 256M, proven by a cgroup probe). VM 9360 and LXC 9361 destroyed; hub host Tester4-aa9466 left for the
operator. REPORT-catalog-sweep-2026-10-24.md.
2026-10-25 00:13:05 +02:00
admin 3add386aa9 Restore-holes run (2026-10-24 night): D43 = A recorded; 07/05/03/11 carry the R-1251, R-1252, R-1254 and D43 rules; register 207 -> 208 (R-1268 opened; R-1251/1252/1253/1257 VERIFY, R-1254 NARROWED); STATUS Monday list + D44/D45 picks; evidence (Tester 1, 9202)
gates / gates (push) Successful in 13m4s
2026-10-24 21:08:38 +02:00
admin dcf4f84cb7 hub R-1254 (P2): the escrow key, its hash and the identity blob in ONE transaction; a bad identity blob is refused before anything is stored. Committed, not deployed.
gates / gates (push) Successful in 13m27s
2026-10-24 20:37:59 +02:00
admin 80f1ab3167 Review round 2: CI numbers by hash in the session report
gates / gates (push) Successful in 12m51s
2026-10-24 16:29:40 +02:00
admin 876d92367e Review round 2 (batch 2 + close-out): 02/07/09/10/11/14/15/16/inventory facts corrected at file:line (EDITS.md), undocumented decisions as notes (R-1265), rulings left for the operator (R-1266); register 190 -> 207 (R-1251..R-1267 opened: R-1251 P1 cut drive-app restore put back over newer data, R-1252 P2 rejected backup block deletes second-drive copies - reproduced on 9202); STATUS D43-D45; CONTEXT; audit README
gates / gates (push) Successful in 13m10s
2026-10-24 16:12:30 +02:00
admin 77015997f8 Review round 2, part 1 (batch 1): 01/03/04/05/06/08/12/13/_hub-review facts corrected against live code (each at file:line, EDITS.md); 20 capability rows marked re-check; poster timeline text; live-probes path; reviewer candidates and the H1/H2 proofs
gates / gates (push) Successful in 12m37s
2026-10-24 15:46:33 +02:00
admin 44d1904286 DooPlex backup alarms: one cause (the dump job's password after the reboot), fixed; alarm mail quieter; R-1250 proven
gates / gates (push) Successful in 12m24s
R-232 dated note + a dated check for tonight's sweep; R-934 closed (the disk was pulled
2026-10-23); R-1250 fixed on main, live proof on demo-hp with a real seed; 08 §9 item 5.
Evidence audits/dooplex-alarms-2026-10-24/.
2026-10-24 13:53:57 +02:00
admin 6018486fcc Attended release run done: controller 0.342.0 the global floor, agent 0.164.0 vouched, hub 0.169.0; R-1244 and R-1247 closed (register 193 -> 191); capability F13; STATUS; the run report
gates / gates (push) Successful in 13m3s
2026-10-24 10:37:27 +02:00
admin 01bf4e83a9 Deploy hub 0.169.0 (the upload watch's four operator-only events; the System page's Upload line) on the operator's yes; agent 0.164.0 delivery + vouch evidence
gates / gates (push) Successful in 12m38s
2026-10-24 09:46:42 +02:00
admin 1a788f332d Attended release run: hub 0.168.0 deployed (R-975 closed; the live file's residue accepted by the operator, 05 16.2); D40 in effect (R-1243 VERIFY, dated check 10-25); D41/D42 closed; the ep0 iperf3 line-test server (scripts/ep0-iperf3, 06 5) and hub 0.169.0 models line_test (System page cell); R-1249, R-1250 filed; register 194 -> 193
gates / gates (push) Successful in 12m46s
2026-10-24 09:18:44 +02:00
admin 3363ffa89a D40 (R-1243): the hub's volume out of DooPlex's Longhorn backup — PVC label default: disabled (operator 2026-10-24)
gates / gates (push) Successful in 12m22s
2026-10-24 07:57:23 +02:00
admin 0003115092 Deploy hub 0.168.0 (R-975: the last plain tokens sealed) on the operator's yes; CONTEXT: the 2026-10-24 07:22/07:26 rulings (D40, D41, D42, the iperf3 line test)
gates / gates (push) Successful in 12m41s
2026-10-24 07:33:36 +02:00
admin 20fa831eaf Upload watch night data: one-second reads of the real push on three boxes (spike 2.1); R-1245 gets the measured floor; R-1214 dated check read (stays WATCHING); weekend-night report
gates / gates (push) Successful in 12m31s
2026-10-24 04:44:25 +02:00
admin a91d0b2421 Upload watch: spike, design (08 section 9, proposed), live proofs on 9202; rows R-1244..R-1248 (register 189 -> 194); STATUS D41/D42 + the Monday list
gates / gates (push) Successful in 13m1s
Spike: per-app normal levels on three boxes (19-31 days), the off-site push as a line measure (a lower bound),
qBittorrent/Transmission/Plex explainers measured on 9202, a tc ceiling measured (not built). Live on 9202 with
0.342.0-rc.1: limit set; real seed explained; CPU burner and the hidden flood alarmed once each and cleared once;
the household's limit left alone. Teardown: apps removed, 9202 back on 0.341.0, receiver 9471 destroyed.
2026-10-23 22:52:41 +02:00
admin af5c005a4e hub 0.169.0 (unreleased, not built): the upload watch's four events — allowed, operator-only, per-app cooldown, one readable Watch line in the operator mail
gates / gates (push) Successful in 12m47s
app_unexplained_upload / app_unexplained_cpu (warning) and their _cleared (info), from controller v0.342.0 (08 section 9,
operator ruling 2026-10-23). Red-proved: operator-only, the render line, the allow-list.
2026-10-23 21:26:36 +02:00
admin 5277d859d5 hub 0.168.0: seal the last plain tokens in config_json (R-975, R-1241's Felhom half); decide row R-1243 for the hub's plain Longhorn copies (register 188 -> 189)
gates / gates (push) Successful in 12m26s
The Cloudflare tunnel token, Cloudflare API token and git token inside customer_configs.config_json now hold the
05 §16.2 seal (enc:v1:, OFFSITE_SECRET_KEY). Old rows sealed once at start-up without a config_version bump, then
VACUUM + WAL checkpoint (an UPDATE leaves the old plaintext in a freed block). -unseal-box-secrets opens them too.
Eight tests, each red-proved. Built, not deployed. Audit: documentation/audits/hub-token-seal-2026-10-23/.
2026-10-23 19:48:17 +02:00
admin 8257ff1cbe Plex2Shell impact on Felhom: verdict NOT REACHED; R-1241 (P1), R-1242 (P3) opened (register 186 -> 188)
gates / gates (push) Successful in 12m37s
Read-only audit of whether the hacked Plex pod on DooPlex (HM-040, 2026-10-17 -> 10-23) reached Felhom's
secrets. Two no-login doors existed (the Longhorn NFS export and the Longhorn API); neither was used:
nfsd accepted 0 new connections in the window (positive control), no off-schedule Longhorn backup or volume.
Evidence in documentation/audits/plex2shell-felhom-impact-2026-10-23/. The NFS export was narrowed from *
to 10.42.0.0/16 on the operator's yes (host change, not in git). 13-threat-model.md: attacker j and one
boundary row. STATUS: verdict, the P1, one do item.
2026-10-23 17:53:43 +02:00
adminandClaude Opus 5.5 87dc50743c D39 = B: 09 §3 decision 255 (the security-fix step) + a dated note on decision 13; capability B23; R-1239 closed, R-1240 opened, R-1238 updated (register 186 -> 186); STATUS/CONTEXT; nodes.md (9202 password, Tester 1 DHCP); evidence security-lane-2026-10-23 (controller 0.341.0, global floor 11:24 UTC)
gates / gates (push) Successful in 13m49s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-23 13:35:38 +02:00
adminandClaude Opus 5.5 4b9b0af3cb Plex2Shell: the operator's 10:56 ruling in CONTEXT; demo-hp 9201 checked (clean) and updated to 1.43.5.11029; register 184 -> 186 (opened R-1238 Plex test record/fixture, R-1239 a fast lane for security fixes = D39); evidence plex-demo-boxes-2026-10-23
gates / gates (push) Successful in 13m46s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
2026-10-23 11:27:52 +02:00
admin 2c9152d912 Day 2026-10-23 close-out: rulings D31-D36 in CONTEXT; STATUS (releases done, do list, D35 dated, D37/D38); register 191 -> 184 (closed R-49 R-982 R-985 R-991 R-1204 R-1205 R-1225 R-1226 R-1227 R-1230 R-1234; opened R-1235 R-1236 R-1237); the break-glass sheet current (D32, one A4, HU+EN); runbooks for D33/D34; poster facts; evidence day-2026-10-23
gates / gates (push) Successful in 12m53s
2026-10-23 11:03:45 +02:00
admin 52af38df6f hub-db-backup: wait for a snapshot being written, limit 7 h (one slot gap + 1 h) — the 2 h limit refused the hub's own catch-up and mailed a false failure at 10:28 (R-1234, closed); tests red-proved
gates / gates (push) Successful in 13m14s
2026-10-23 10:35:16 +02:00
admin 34979d5d25 hub manifest: 0.165.0 -> 0.167.0 (operator yes 2026-10-23: D31, the clock-ahead tunnel alarm, R-1230, D34 6-hourly snapshot; image sha256:177860a9…)
gates / gates (push) Successful in 13m26s
2026-10-23 10:23:29 +02:00
admin 0bb57a4fee website: publish D3/D5 (the whole truth on privacy; the minimum machine) and D36's two sentences, HU + EN (operator yes 2026-10-23; R-985, R-991, R-1225, R-1226)
gates / gates (push) Successful in 13m10s
2026-10-23 10:16:54 +02:00
admin 2aebf1bde1 webpage: serve installer-v1.36.0 (both git-sync pins; operator yes 2026-10-23 — root SSH by key only on new installs, R-982 D4)
gates / gates (push) Successful in 13m12s
2026-10-23 10:08:09 +02:00
admin d306b8c2d7 release-copy: the released images (skopeo dir:, digest-true) and agent packages copied to ep0 nightly, verified by sha256 and a checksum compare, capped at 3 GB (D33, R-1227); tests red-proved
gates / gates (push) Successful in 15m56s
2026-10-23 09:39:31 +02:00
admin be7e97dc1a hub 0.167.0 + DooPlex push: the hub DB snapshot every 6 hours (D34, operator yes 2026-10-23, R-232 G5) — slots 02/08/14/20, catch-up 6 h, push :30 after each, refuse a copy older than 2 h; tests red-proved
gates / gates (push) Successful in 16m20s
2026-10-23 09:12:08 +02:00
1607 changed files with 45965 additions and 368 deletions

No files matched your search

+1 -1
View File
@@ -38,7 +38,7 @@ to probe a refusal at all.**
## If a probe does create something, remove it through the product
Not by hand, and not by `docker rm`: stop it, then `POST /api/stacks/<name>/remove` with
`remove_hdd_data` and `remove_backups`, and then **verify** — no container, no `/opt/felhom/stacks/<name>`,
`remove_hdd_data` and `remove_backups`, and then **verify** — no container, no `/opt/docker/stacks/<name>/app.yaml` (the folder itself stays — it holds the catalog's template files; path corrected 2026-10-24),
no volume. Say in the report that it happened. A tidy-up nobody is told about is how the next session
learns nothing.
+111
View File
@@ -16,6 +16,117 @@
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
> **2026-10-25 (attended release run) — D49 = A, D44/D45 = all picks, D46 = yes; released: hub 0.170.0, agent 0.165.0,
> controller 0.343.0, catalog `ebbf1f4`** (`09` §3 decisions 258–261; `REPORT-release-2026-10-26.md`). **D49 = A:** a
> restore whose app does not start keeps both copies and never pours back by itself (`07` §8.2e). **Fix A** (controller
> `restore_verdict.go`): „started" = running on every sample of 45 s, no `RestartCount` growth, every healthcheck
> `healthy`; the undo copy is released only after 3 more minutes of the same (10 s samples, 30 min watch, a newer restore
> ends the watch — `restoreBegunAfter`). **Fix B:** the `.fab` import pulls first (`Exporter.SetImagePreparer` →
> `PrepareRestoreImages`). **D47 follow-up:** a set-aside mails the operator (`backup_run_failures`, leg
> `unit-set-aside`) — its hub subject still says „failed to back up" (R-1287). Agent 0.165.0 vouched 18:38 CET; on a
> BYO host our SSH drop-in is removed only on the bundle run AFTER the one that installs the new wrapper (R-1285).
> Installer 1.36.1 on main, not published (R-1284: `http_code` printed the code twice). R-1250's live proof is a dated
> check on 2026-10-27 (the agent tests only on its due Tuesday 10:00–15:59).
> **2026-10-25 08:38 — operator rulings by starting the R-1269 brief: D47 and D48** (`09` §3 decisions 256, 257;
> `audits/restore-truth-2026-10-25/README.md`). **D47:** an existing recovery unit is never changed in part — its
> definition and data are written only together, by a backup of the same install; a unit an earlier install left is set
> aside whole and stays restorable (`07` §8.2d). **D48:** every restore path resolves and pulls every image its
> definition names before the first change to the app; a failed pull is a refusal with the app untouched (`07` §6.6).
> **Confirmed, no change:** `07` §6.6's version rule; R-698 and R-933. Built the same day, controller commits only (rides
> v0.343.0): `02daebe` (D47 / R-1269), `b680db8` (R-1270: a restore whose app does not start is never a success; the
> safest reading — no automatic pour-back, the undo copy kept, a late start turns it into a success), `2d71270` (D48),
> `bd9f4f7` (R-1271: a copy dated more than 10 minutes in the future is never „the newest").
> **2026-10-24 (afternoon, attended) — two DooPlex backup alarms — DONE.** One cause: the 2026-10-23 window reboot
> restarted `postgresql-2`; CNPG 1.29 sets the role `postgres` from the superuser secret, then from the
> `bootstrap.initdb` owner secret (`owner: postgres`), so `postgres-superuser` is in force at every start; the dump
> CronJob read `postgresql-superuser` → 28P01 from 16:00Z → `PostgresqlDumpStale`; the 00:20 Gitea push refused the
> stale dump → `DooplexGiteaOffsiteStale`. **Operator rulings 2026-10-24:** (1) fix the CronJob only (homelab
> `531023d`; the Cluster keeps two writers — homelab list); (2) apply the routing change by restarting Alertmanager
> (its `envsubst` init container makes `/-/reload` useless; homelab `22d7697`: critical 4 h, warning 12 h, one mail per
> backup group and per drive); (3) the R-1250 live proof may use DooPlex's k3s qBittorrent as the downloading peer
> (one private LAN torrent in its container `/tmp`, deleted with its files). R-1250 committed (agent `29bcf44`,
> `1e087bb`; controller `046b808`), not released. R-934 closed (the disk was pulled 2026-10-23). Catalog `7de554e`:
> the box walk sends the drive itself as HDD_PATH (R-839). Report: `REPORT-dooplex-alarms-2026-10-24.md`.
> **2026-10-24 (attended release run) — DONE.** Hub 0.168.0 (05:39Z) and 0.169.0 (07:51Z) deployed; agent 0.164.0
> released, delivered to the three test boxes and vouched by the operator; controller 0.342.0 the global floor (08:29Z,
> MinAgent 0.158.1). ep0 runs `felhom-iperf3` (tunnel only). First line tests: Tester 1 94.3, demo-hp 94.4, demo-felhom
> 92.6 Mbit/s; demo-felhom's qBittorrent limited to 46 Mbit/s. D40 in effect (dated check 2026-10-25, R-1243). Opened
> R-1249 (the start-up VACUUM), R-1250 (a seeding qBittorrent is never tested). Report `REPORT-attended-release-2026-10-24.md`.
> **2026-10-24 07:22 / 07:26 — operator rulings by starting the attended release brief (D40, D41, D42; the line test).**
> **D40 (R-1243) = stop the Longhorn backup of the hub's volume**; the encrypted 6-hourly copy to ep0 stays the hub's
> off-site copy; the operator deleted the two old plain Longhorn backups himself (UI); the Longhorn UI cannot take the
> volume out of the backup job, so the change is in a manifest. **D41 (R-1245) = 50 %** of the measured line for
> qBittorrent's default upload limit. **D42 (R-1246) = the family is told only about the torrent limit**;
> `app_unexplained_upload` / `app_unexplained_cpu` are operator-only. **Line measure (R-1247) = a weekly iperf3 upload
> test from the box's host to an iperf3 server on ep0, inside the WireGuard tunnel only** (operator 07:26; his design of
> 2026-10-23: „during a quiet window the box measures the upload bandwidth"); the passive floor stays as a lower bound.
> This is the one change on ep0 the run may make (Tier 2 — the operator's yes is starting the run). CC still stops
> before each hub deploy, before the global controller floor, and for the agent vouch.
> **Two more answers in the same run (AskUserQuestion, 2026-10-24 ~08:2x CEST):** (1) **the agent speaks iperf3 itself**
> (no host has the iperf3 program; nothing is installed on hosts) — agent `internal/linetest`; (2) **the live hub file's
> old token bytes are accepted, not cleaned** (the start-up VACUUM did not finish on the Longhorn disk, R-1249; a clean-up
> needs an 8–20 min hub stop, a trim and the volume's snapshots deleted; only root on DooPlex can read the volume, and it
> holds the key; every copy that leaves is clean) — written into `05` §16.2 with the rule „a hand copy of hub.db is made
> with VACUUM INTO, never cp/tar".
> **2026-10-23 19:22 — operator rulings by starting the weekend-night brief (R-1241's Felhom half; the upload watch).**
> **Part 1 = R-1241's Felhom half:** seal R-975 (done in hub 0.168.0, not deployed — `05` §16.2,
> `audits/hub-token-seal-2026-10-23/`); the hub's plain Longhorn copies are decide row R-1243 (CC's pick: take the hub
> volume out of the Longhorn backup; ep0's encrypted copy is the backup). **Part 2 = the upload watch, shaped by the
> operator:** the box measures its own upload line (first from what it already sees — the off-site backup's upload
> rate — a short test at a quiet hour only if needed); on a sustained high upload it attributes the bytes to an app's
> container and ASKS THE APP (qBittorrent's seeding, Plex's remote sessions); a mismatch is the alarm
> (`app_unexplained_upload`, the same shape for CPU, `app_unexplained_cpu`) — operator only. qBittorrent gets a default
> upload limit from the measured line; a limit the household set is respected. **Open for the operator, CC's pick in
> STATUS:** the default limit's size (CC: 50 % of the measured upload) and who is told (CC: the family only about the
> qBittorrent limit they can change; the unexplained alarms to the operator only — the R-388 rule).
> **2026-10-24 (night, unattended) — the restore and backup holes; operator ruling D43 = A**
> (`audits/restore-holes-2026-10-24/README.md`). **Ruling D43 = A (operator, 2026-10-24 evening, by starting the run):**
> the root-key-only sshd drop-in applies only in appliance mode; BYO hosts keep their SSH settings (`11` §1, `03`).
> Fixed on main, nothing released: controller `bb2809d`/`062f9b4` (R-1251: a pending put-back blocks every start — a
> restore hold `putback_pending`; the put-back runs first when the drive binds; a failed put-back holds; the copy synced
> before the marker names it), `518a135` (R-1252: reconcile only under a valid classification), `c9b4b82` (R-1253: a
> stop-first export refuses while another operation holds the marker); hub `dcf4f84c` (R-1254: escrow in one
> transaction; image built locally, not pushed); agent `c8d3c17` (R-1257: D43). Proven: R-1251 on Tester 1 (power cut,
> disk off, disk back), R-1252 and R-1253 on 9202. D44/D45 prepared: `audits/review-round-2-2026-10-24/D44-D45-picks.md`.
> **2026-10-24 (evening, unattended) — review round 2** (`audits/review-round-2-2026-10-24/README.md`). The 17 architecture
> documents checked against `main`: facts corrected at `file:line` (EDITS.md), rulings and promises left for the operator
> (R-1266, D45), undocumented decisions written as notes (R-1265, D44). New: **R-1251 (P1)** a cut restore of a drive-backed
> app is put back over newer data (shown with the real `Recover`); **R-1252 (P2)** a rejected backup block makes the
> second-drive reconcile delete the household's copies (reproduced on 9202). Controller `b01bb4b` (unreleased, next
> v0.343.0): a media restart skips household-stopped apps (R-721); the FileBrowser SSO header file appears whole.
> **2026-10-23 ~12:00 — operator rulings by starting the D39 brief (`09` §3 decision 255; R-1239 closed).** **D39 = B:** a
> `security_fix` ladder step — box walk on 9202 required, bench optional, taken by the night leg. **No operator word per
> incident:** a public advisory id (CVE or GHSA) + its https:// notice is the authority; the session that writes the step
> reports it to the operator after the push. The design (Claude.ai's recommendation) was taken as written; one fact in the
> brief was checked and held (Plex's box fixture exists but is a `NoRoute` stub, so `seed: "none"` is right). Built:
> catalog `3898593` (`ladder.py` `check_security` + `engine_majors_crossed` importing `check-engine-major.py`; the move gate
> accepts it; `check-test-record.py` WARNs on a head with no bench; `upgrade-test.py --write-ladder --security`;
> `box_walk.py --security` + `refresh_box_mirror()`), controller **0.341.0** (`legCandidate` takes `proven` or
> `security_fix`; global floor 11:24 UTC with MinAgent 0.158.1, the operator's yes in chat). Proof:
> `audits/security-lane-2026-10-23/`. Older controllers skip the step at night and take it on a hand press (proven).
> **2026-10-23 10:56 — operator ruling: Plex to the fixed version now, without the test record (Plex2Shell).**
> Verbatim: *„bump the version to the newest in the app template immediately (as there are no customers yet), I want the
> demo boxes on the newest version asap."* It skips `09` §3 decision 13's test record **once, for Plex only**. Done: catalog
> `6cc317d` (Plex 1.41.4.9463-630c9f557 → 1.43.5.11029-6bf8af3f0, CVE-2026-96652/-96654/-96655/-96656), pushed
> `--no-verify` past `check-test-record-move.py` (its refusal is in `app-catalog-felhom.eu/REPORT.md`); the only box running
> Plex (demo-hp 9201) was checked for the attack's signs (none) and updated by the Update press; the guide re-checked
> (`guide.checked_version` moved). Decision 13 itself is unchanged. Follow-ups: R-1238 (Plex's test record and fixture),
> R-1239 (a fast lane for security fixes — the operator's decision, D39). Evidence: `audits/plex-demo-boxes-2026-10-23/`.
> **2026-10-23 morning — the operator's „go" for the day brief: the releases, D31 as built, D32–D36 (decide rows in `STATUS-HISTORY.md`).**
> **Releases:** controller 0.340.0 (global floor 06:41 UTC, MinAgent 0.158.1); agent 0.163.0 (binary sha256 561b5316…, bundle 3afb88d9…, vouched by the operator 07:58 UTC; delivered to the three test boxes by signed jobs through a step bundle `0.163.0-step1` — the D4 drop-in is a new path, R-880); installer 1.36.0 (tag `installer-v1.36.0` → 83388c08, both webpage pins); hub 0.167.0 (sha256:177860a9…, 08:27 UTC); the website `0bb57a4f`.
> **D31** = as built (the re-issue reads the newest deletion of a KEY-HOLDING host; an install that got the password but no recovery package is rotated without a press). **D32** = yes: the sheet is names and places only, one A4, HU + EN; the operator prints and fills it. **D33** = yes: `scripts/release-copy/` nightly 01:40, newest 3 image tags (controller, hub, samba) + 5 agent versions, skopeo `dir:` (digest-true — why not `docker save`: a re-push re-compresses), ep0 `/srv/felhom-release-copy` under the one limited user `felhom-relcopy` (rrsync), cap 3 GB enforced by the sender, a read-only registry token of its own. **D34** = yes: hub snapshots at 02/08/14/20 (catch-up > 6 h), DooPlex push :30 after, limit 7 h + a wait for a snapshot being written (R-1234: 2 h was wrong), alarm 13 h; ep0 retention unchanged (keep-daily 14, keep-weekly 8). **D35** = yes, after the release week (R-1228, dated 2026-10-30). **D36** = yes, both sentences, published.
> **Added on the operator's word the same day (not soaked):** D34's hub half went into 0.167.0 (asked before the build). Record: `audits/day-2026-10-23/`, `REPORT-day-2026-10-23.md`.
> **2026-10-22 evening — operator rulings D29, D30, D31 (CC's picks accepted; the decide rows now in `STATUS-HISTORY.md`).**
> **D29** = no: the controller repository stays public for now (R-925). **D30** = the agent makes a new WireGuard key and
> registers it on the existing peer path (R-1204). **D31** = the hub skips a deleted install that never made a key when it
+89
View File
@@ -0,0 +1,89 @@
# REPORT — attended release run 2026-10-24: the hub's tokens locked, the hub out of the open backup, the iperf3 line test, the watch released
Brief: „the hub's tokens locked, the hub out of the open backup, an iperf3 line test via ep0, then the upload watch
released" (operator rulings 07:22 / 07:26, `CONTEXT.md`). Read first: `05` §16.2, `06` §4.5 / §5, `07`, `08` §9, the
spike `audits/SPIKE-upload-watch-2026-10-23.md`, the felhom-build-deploy skill. Evidence: `audits/line-test-2026-10-24/`.
## Five lines
1. **Hub tokens:** sealed in hub 0.168.0 and still working (demo-hp is served the same tunnel and API tokens it runs; the
template fetch works; the next ep0 copy holds 0 of 8 token values). The live file keeps old bytes in free space — your
call: accepted, written down, with the rule „copy hub.db only with VACUUM INTO".
2. **The hub is out of the open backup:** the hub volume reads `default: disabled`; the folder holds no copy of it (your
two deletions worked; nothing else is left). Tonight's job is the last proof (dated check 2026-10-25).
3. **The line test:** demo-hp measured **94.4 Mbit/s** (Tester 1 94.3, demo-felhom 92.6) — the night backup said „at
least 10".
4. **Live where:** hub 0.169.0 and controller 0.342.0 run on all three boxes (the global floor); agent 0.164.0 on all
three (vouched); demo-felhom's qBittorrent is limited to 46 Mbit/s. Only these three boxes report to the hub.
5. **You still do:** nothing for this run. Two rows wait for a design (below).
## What I asked you, and what you said
- Deploy hub 0.168.0 → yes. Deploy hub 0.169.0 → yes. Global floor 0.342.0 → yes. Agent vouch → done.
- The live file's old bytes → accept and write down (my first estimate of „2 minutes" was wrong; the disk writes ~1.3
MB/s, so a clean-up means an 8–20 minute hub stop).
- No host has the iperf3 program → the agent speaks iperf3 itself (nothing installed on hosts).
## Versions, digests, CI (by commit)
| Repo | Commit | What | CI |
|---|---|---|---|
| felhom.eu | `00031150` | hub 0.168.0 manifest + rulings | 2275 success |
| felhom.eu | `3363ffa8` | D40: the PVC label | 2276 success |
| felhom-agent | `f7e1dcb` | the line test | 2277 success |
| felhom-controller | `ed1011b` | the tested line in the watch | 2278 success |
| felhom.eu | `1a788f33` | docs, register, ep0 unit, hub `line_test` model | 2279 success |
| felhom-controller | `42cda07` | the start-line wording | 2283 success |
| felhom-agent | `08ca678` | CHANGELOG released, REPORT | (docs) |
| felhom.eu | `01bf4e83` | hub 0.169.0 manifest | (see the final push) |
| felhom-controller | `bddc477` | CHANGELOG released, REPORT | (docs) |
- **Hub 0.168.0** `sha256:3bfdecaa…` (from `5277d859`), deployed 05:39Z. **Hub 0.169.0** `sha256:fb9b2489…` (from
`1a788f33`), deployed 07:51Z; 30-minute watch: only info events, 0 errors.
- **Agent 0.164.0**: binary `1b803d8b70f8afbe1546f6989f35b6b0e29c9ed63f6acb3843d3c04ef0d24b2b`, bundle
`1d7b54d39d0c1aff682ba0c0a7f91aea0f9407a10d2f5a8bbfdc858d32c13341`, tag `v0.164.0`; signed `agent_update` to the three
boxes, self-check 76/76; vouched (hub page + `Artifact manifest set` line).
- **Controller 0.342.0** `sha256:d002d3b7…` (from `42cda07`); rc.2 on 9202; per-box floors → healthy on all three, box
checks and https-only green; global floor 08:29:19Z, MinAgent 0.158.1, read back on the hub log and page.
- **ep0:** `felhom-iperf3.service` (iperf3 3.18-2+deb13u2, already installed since 2026-07-03), one nftables line;
no answer on the internet (`:22` as the control), an answer in the tunnel. CVEs: four fixed in this build; two open
„minor" server DoS issues bounded by tunnel-only, TCP-only and systemd caps (`scripts/ep0-iperf3/README.md`).
## Rows
**Before 194, after 191; opened 2 (R-1249, R-1250), closed 5 (R-975, R-1244, R-1245, R-1246, R-1247).** R-1243 →
VERIFY (dated check 2026-10-25); R-1241 keeps only the homelab half.
## Observations
- The start-up VACUUM after a legacy seal cannot finish on the Longhorn disk; its comment said 2.8 s — FILED: R-1249.
- A qBittorrent that seeds all day keeps the uplink busy, so that box is never tested and gets no limit — FILED: R-1250.
- The live hub file's old token bytes — NOT-A-FINDING: accepted by the operator 2026-10-24 (`05` §16.2), every copy that
leaves is clean.
- Two pre-deploy hub snapshots in `/data/snapshots` held plain tokens — NOT-A-FINDING: they roll off (one already pruned
at 06:06Z; the other at the 12:00Z snapshot); same volume, same reader as above.
- The cross-repo wire gate refused the agent's new `line_test` field until the hub modelled it — fixed in the session
(hub 0.169.0 `sysfacts.LineTest`, the System page cell).
- My first ArgoCD sync used a wrong commit id — NOT-A-FINDING: it failed to fetch and changed nothing; redone.
- 9202 has no agent local API, so its controller cannot read a line test — NOT-A-FINDING: by design (scratch guest);
the qBittorrent proof moved to demo-felhom, where the product path runs.
- demo-hp's first watch minute could not resolve `plex` — NOT-A-FINDING: the known start-up race (2026-10-23).
- The family's line on demo-felhom's qBittorrent page was not seen live — NOT-A-FINDING: the household's password is not
CC's; the render is pinned by a test, and the text with the live value is in the evidence.
- Old rc controller images (other sessions) sit on 9202 — NOT-A-FINDING: a scratch guest; mine (rc.2) is removed.
- The hub's database is 530 MB, mostly report history (`host_reports` 305 MB, `reports` 128 MB) — NOT-A-FINDING here:
it is why a VACUUM is slow, recorded in R-1249.
## Teardown, three layers
- **Machine:** 9202 — the throwaway qBittorrent removed through the product (0 containers, 0 volumes), back on the
0.342.0 release, my rc.2 image removed. demo-hp's host — the probe binary deleted, nothing running.
- **Host / DooPlex:** the database copies I restored for the checks shredded; the 9202 password copy, the signed job
envelopes and the release clone shredded / removed. ep0 — the one planned change stays (`felhom-iperf3`); the backup
of the firewall file is `/etc/nftables.conf.bak-20261024-iperf3`.
- **Hub:** the three per-box floors cleared; the global floor 0.342.0 stays (the release).
## Instruction files, poster
No instruction file edited. System poster: a text-only change (ep0 also runs the tunnel-only line-test server) — the
facts file and the matching poster text edited in the same commit.
+67
View File
@@ -0,0 +1,67 @@
# REPORT — every catalog app on a fresh box, the way a family uses it (2026-10-24 → 25 night, unattended)
**Plain words first.**
- 65 apps: **52 work, 10 work with a problem, 0 broken**; 2 are not offered for a new install (by design); Wanderer
cannot run on a box without DNS (known, R-739).
- **The three worst:** (1) after „remove — keep backups" and a reinstall, restoring that backup breaks the app — the
household's item is gone from view (Tandoor, AdventureLog; R-1269, P2); (2) that restore still says it worked (R-1270,
P2); (3) „use my kept data" loaded another box's newer-dated database for Paperless-ngx, Nextcloud and Immich, and
Immich never answered again (R-1271, P2). In all three the data is not deleted.
- **Fixes made in the drill catalog, not live:** n8n stops sending usage data out; OpenGist gets 256 MB (at 128 MB one
sign-in hit the limit 443 times and swapped 98 MB). Both proven on the sweep box pointed at the drill.
- **Morning:** say yes to the two catalog fixes (in Monday's release list); decide D46 (Homepage behind the family gate —
CC's pick: yes); delete hub host Tester4-aa9466.
**Safest readings taken (the brief allowed no question):** ISO 1.30.0 (the current public one, not 1.36.0); 12 GiB VM
(the brief's minimum; demo-hp had 14 free); the kept-data question exists only for drive-folder apps, so volume apps
were checked by a unit restore after the reinstall (that made a unit restore on every seeded app, not every fifth);
„Mentés most" is local-only by design; backup/storage findings filed in their own category, not „Apps & catalog". All in
the audit's head.
**Table and evidence:** `documentation/audits/catalog-sweep-2026-10-24/README.md` (the 65-row table, the findings, the
R-1269 proof, D22 put-back PASS on PrivateBin and Vikunja, the drill proofs, teardown).
**Architecture read:** `07-backup-architecture.md` §6.5 (kept data) and §8, `16-storage-physical-identity.md` §5 (the
internal place is the default drive — F3 NOT-A-FINDING), `10-localisation.md`.
**Two mistakes of this session, both caught and kept in the evidence:**
1. The first drill pass ran on the live templates: I had believed the drill repo public; the box's clone failed and the
controller kept its templates. Re-run through the box's local mirror (R-992's method) with a positive check before
any install (`DRILL-CONFIRMED`). Memory note written.
2. I reverted the OpenGist fix on a wrong reading („it grows into its limit"); an automated security review flagged the
lowered cap; a cgroup measurement proved the fix needed and it went back in.
**Commits:** drill catalog (never live) `a601950` (n8n), `fd23571` → `c420053` (wrong revert) → `405077f` + `492a17f`
(OpenGist). felhom.eu: this commit (audit + tools + evidence, register, STATUS, STATUS-HISTORY, capability map B1 +
detail). CI by hash: in the final report.
**Register:** at the sweep's start 190 (other sessions filed 18 since); before this commit 208, after 217; opened 9
(R-1269–R-1277), closed 0. Notes added to R-739 (Wanderer, no DNS) and R-1267 (anonymous volumes: OnlyOffice and RomM
too; 52 left after every app was removed with data).
**Observations:**
- FILED: R-1269 (reinstall's capture rewrites the kept unit), R-1270 (restore says ok while the app never starts),
R-1271 (kept data loads another box's future-dated database), R-1272 (Homepage has no gate — operator, D46), R-1273
(registry rate limit shown as „exit code 1"), R-1274 (restore refusal advice when tiers exist but have not copied),
R-1275 (drive names in Hungarian on English pages), R-1276 (empty parent folder after remove with data), R-1277 (box
fixtures drifted: OpenGist, Calibre-Web, Mealie, Papra, Rallly, Vaultwarden 1.37).
- FIXED (drill, morning list): n8n usage data (`a601950`); OpenGist memory (`405077f`, `492a17f`).
- NOTE on existing rows: R-739 (Wanderer), R-1267 (anonymous volumes).
- NOT-A-FINDING: F3 install default is the internal place (`16` §5, by design); F4 English guide labels (the app's own
words; the Hungarian copy is frozen by the copy gate); F5 Kimai in English (a Hungarian browser lands on `/hu/`); F6
AdventureLog 95 % (docker stats with cache; anon 75 %); database first-start lines in SparkyFitness and Dawarich (no
restart, item back); Radicale IP redirect (curl without Host); NZBGet/Transmission basic-auth prompt in screenshots (the
apps' own login); Vaultwarden stranger 200 (its own login page; sign-up closed).
**Teardown:** machine — VM 9360 and LXC 9361 (both this session's) destroyed with `--purge`, no disk left on
`nvme-scratch`; Tester 1, 9201, 9202, the thin pool and the ISO untouched; the box's logs copied off first. Host —
DooPlex: the session's secret files shredded. Hub — host Tester4-aa9466 left for the operator to delete. ep0 — listed,
nothing removed: the host's PBS link was set up, no whole-box copy went to it; the app copies went to the Tester4 restic
repository on the storage box.
**Secrets:** read by name through `sec`; the evidence was scanned for every known secret value (positive control seen) and
for token shapes; one Actual Budget session token (a throwaway app) was blanked; screenshots spot-checked (passwords sit
behind „Show", never pressed). No password, key or token is in this commit.
**System poster:** no fact on it changed. **Instruction files:** none edited. **No** release, deploy, floor, vouch,
publish or live catalog push.
+103
View File
@@ -0,0 +1,103 @@
# REPORT — day 2026-10-23: the releases and the DooPlex hour (the operator's „go")
Design read: `07-backup-architecture.md` §6.3/§8, `09-update-architecture.md` §3 decision 250, `03-host-agent.md`,
`05-hub-architecture.md` §16.3, `runbooks/total-loss-of-dooplex.md`, `audits/dooplex-survival-2026-10-22/PLAN.md`.
Evidence: `documentation/audits/day-2026-10-23/01`–`14`.
## Five lines
1. Released: controller 0.340.0 (global floor), agent 0.163.0 (vouched by the operator), installer 1.36.0, hub 0.167.0,
the website with D36's two sentences.
2. Root SSH from the LAN is key-only on all three test boxes; new installs get it from installer 1.36.0.
3. The hub database now leaves every 6 hours (restored back byte-identical); the released images and agent packages
leave nightly to ep0 (pulled back, every sha256 OK, one image loaded on a scratch box).
4. One false failure mail at 10:28 (my install of the 6-hourly copy) — filed R-1234, fixed and red-proved the same hour.
5. Register 191 → 184; the operator: print the sheet, Hosts page clean-up, D37 + D38.
## Per box — versions before → after, and the SSH checks
| Box | controller | agent | root-file bundle | `sshd -T` root | sshd reloaded | key login | felhom-sshd | swap = RAM (9201) |
|---|---|---|---|---|---|---|---|---|
| Tester 1 | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 (via 0.163.0-step1) | without-password | yes, 09:27:38 | ok | active, `sshd -t` ok | 4096/4096 |
| demo-hp | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 | without-password | yes, 09:19:44 | ok | active, `sshd -t` ok | 25898/25898 |
| demo-felhom | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 | without-password | yes, 09:24:42 | ok | active, `sshd -t` ok | 12288/12288 |
| hub | 0.165.0 → 0.167.0 | | | | | | | |
Not proven (said plainly): the real root password over SSH (CC does not hold it — a dummy one is refused, but OpenSSH logs
only „Failed password"); a felhom-sshd connection (it admits only the operator's tunnel peer); the physical console (the
drop-in does not touch it; `passwd -S root` = P).
## Digests, sha256, CI by commit
| What | Value |
|---|---|
| controller 0.340.0 | `sha256:a8d1c6458da4ea9d667f928498656a291771e0b3756b20f2fcc8c1cf9d516621` — `caaa86e`, CI 2245 success |
| agent 0.163.0 binary / bundle | `561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c` / `3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8` — `1109092` CI 2246, `b4721ee` CI 2249 success |
| step bundle 0.163.0-step1 | `6cbc066902d5f3294765d5c7dc7113166c1dca25e994dbc36bfc5f139bb61fcf` |
| installer 1.36.0 (served = tag) | `de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec` — `83388c08` CI 2247 success; pins `2aebf1bd` |
| hub 0.167.0 | `sha256:177860a996cb285d935777d6ffdf85725f6cf4489885f54fd3bf8135c68041a0` — built at `d306b8c2` (hub/ = `be7e97dc`, CI 2250 success; `d306b8c2` CI 2251 success); manifest `34979d5d` |
| website | `0bb57a4f` |
| D34 fix (R-1234) | `52af38df` |
| prometheus rules | homelab-manifests `1bbef892` |
CI of the last commits (`2aebf1bd`, `0bb57a4f`, `34979d5d`, `52af38df` and this close-out): see the final message.
## Parts
- **1 — controller 0.340.0:** 9202 smoke, per-customer floors, three boxes green, box walk + https-only rc 0, global floor
06:41 UTC (MinAgent 0.158.1), read back with the WAL (`01`–`06`).
- **2 — agent + installer:** release, a step bundle because the D4 drop-in is a new path (R-880), three signed jobs per
box (`07`); the operator's vouch read back from the DB and the live page (`10`); installer tag, both pins, a
webpage-only sync, the served script = the tag (`11`).
- **3 — hub 0.167.0:** built from a clean clone; on the operator's word it also carries D34's 6-hourly snapshot (asked
before the build; not soaked); hub-only sync; 30-minute watch: 0 hub events, 0 errors, all boxes reporting, no hub mail
(`14`).
- **4 — website:** D3/D5 patch + D36 HU + EN; site gates; published; read back live: new text present, old gone (control:
the previous commit), screenshots at 390 and 1366 px, no sideways scroll (`12-website-readback/`).
- **5 — D34:** retention unchanged, server-side on ep0 (`prune-operator-hubdb` 03:45, keep-daily 14, keep-weekly 8 — the
current day's up to 4 copies until the next prune). Hand run 501.6 MiB / 24.5 MiB uploaded in 4 s; the newest copy
restored from ep0 with the read-only token = the pod's snapshot by sha256 `a9ff4c8a…`, integrity ok, 4 hosts, holds
today's vouch. Alarm 13 h. Runbook updated (`13`).
- **6 — D33:** ep0 before: `/` 14 GB free (the data disk's root IS the PBS datastore — not used). One user
`felhom-relcopy` (locked, rrsync-confined: shell and `..` refused), one directory `/srv/felhom-release-copy`. Cap 3 GB,
enforced by the sender (ep0 has no quota). **skopeo, not `docker save`:** `dir:` keeps the registry's bytes, so a push
back keeps the digest; no Docker daemon. A read-only registry token of its own, minted in the Gitea pod straight into a
0600 file. First run 676 MB, 91 s; pulled back: 141 files OK, 9 images re-verified (digests match independent records),
controller 0.340.0 loaded on 9202 (ID = config digest). Alarm `FelhomReleaseCopyStale`. Restore steps in
`total-loss-of-dooplex.md` (`08`).
- **7 — D32:** the sheet is names and places only, HU + EN, one A4 (rendered: 1 page; a copy in the operator's home
folder); every place checked to exist. demo-hp: three superseded ISOs removed by name, `/` 8.5 → 14 GB free (`09`).
R-548: the box half done, the product fix proposed (R-921's pattern).
- **8 — close-out:** CONTEXT rulings, STATUS (decide rows D32–D36 → history, D37/D38 new, the do list, D35 dated
2026-10-30), REPORTs and CHANGELOGs; the system poster's facts file updated, its hub-DB label edited (text), and a new
arrow asked for in STATUS („System poster needs a refresh").
## Register — 191 → 184; opened 4, closed 11
Opened: R-1234 (closed the same hour), R-1235, R-1236, R-1237. Closed: R-49, R-982, R-985, R-991, R-1204, R-1205,
R-1225, R-1226, R-1227, R-1230, R-1234. Updated: R-1214 (released; dated check 2026-10-24), R-1187, R-1189, R-232,
R-548, R-923, R-924, R-1228 (D35, dated 2026-10-30), R-1183's check text (four snapshot slots).
## Observations
- The D34 first push refused the hub's own catch-up and mailed admin@ — FILED: R-1234 (closed, fixed `52af38df`).
- The release copy is proven to load, not to restore a registry; it follows the newest versions, not the floor/vouch; no
golden — FILED: R-1235.
- Gitea OOM-killed at its 1 GiB limit, the last at 06:41:10 UTC during the release — FILED: R-1236 (STATUS D37).
- The night sweep's mail: one-time test-box codes in one session record — FILED: R-1237.
- The hub DB snapshot is 501 MiB, not 24 MB — NOT-A-FINDING: 24 MB is the compressed upload, which is what ep0 stores.
- felhom-sshd does not answer from the LAN — NOT-A-FINDING: its firewall admits only the operator peer (H1, by design).
- A dummy root password logs „Failed password", not a policy refusal — NOT-A-FINDING: OpenSSH's behaviour under
`prohibit-password`; the policy is proven by `sshd -T` and the reload.
- The vouch left `artifact_wrapper_sha256` empty — NOT-A-FINDING: an optional field; the wrapper rides the vouched bundle.
- Guests 9202, 9401 (demo-hp) and a transient 990000 (Tester 1) have swap ≠ RAM — NOT-A-FINDING: outside the felhom pool.
- `known_hosts` holds an older key for 192.168.0.155 (demo-hp's guest address) — NOT-A-FINDING: guests are reached by
`pct exec`; nothing was changed.
- The D4 drop-in needed a step bundle — NOT-A-FINDING: the documented R-880 path for a new bundle path.
## Teardown — three layers
Machine: 9202 — the test image untagged, the tar removed. Host: demo-hp `/tmp` tar removed; three ISOs removed on purpose
(Part 7); DooPlex: the read-back copies and the hub-DB scratch restore shredded/removed. Hub: nothing provisioned; the
per-customer floors cleared after the global floor. Left on purpose: the ep0 user + directory, the DooPlex release-copy
job, the 6-hourly hub-DB timer, both alarms.
+111
View File
@@ -0,0 +1,111 @@
# REPORT — two DooPlex backup alarms, every alarm the mail route delivers, and R-1250 (2026-10-24, attended)
Brief: „two DooPlex backup alarms — find why, fix, and check every other alarm the new mail route now delivers".
Read first: `runbooks/total-loss-of-dooplex.md`, `runbooks/RUNBOOK-hub-db-offsite-backup.md`,
`audits/dooplex-survival-2026-10-22/PLAN.md`, R-232, R-1227, R-1241, the two homelab rules and the Alertmanager
route, homelab HM-040…HM-049 and the window record; design `08` §9. Evidence: `audits/dooplex-alarms-2026-10-24/`.
## Three lines
1. **Both alarms had one cause, and both are fixed.** Yesterday's reboot restarted the database, and it now takes the
password of the other of its two secrets; the dump job could not log in, and the Gitea off-site push then refused
the old dump. The job now reads the right secret. Cleared: the dump alarm 10:44Z, the Gitea alarm 10:51Z.
2. **Nothing else fires now.** In 7 days only the pulled disk's two alarms fired long (silenced by you since 10-12,
ended by the pull); the rest were minutes long or are known.
3. **Fewer mails:** critical every 4 h (was 1 h), warning every 12 h (was 4 h), all backup alarms in one mail.
## What I asked you, and what you said
- The dump fix → **the CronJob only** (the database's own definition untouched).
- The mail change needs an Alertmanager restart, not a reload → **restart**.
- The R-1250 proof needed a downloader on another machine → **use DooPlex's k3s qBittorrent**.
## The alarm table (7 days, Prometheus `ALERTS`; nothing firing at 10:51Z)
| Alarm | Severity | When (UTC) | True? | Cause | Fix / row |
|---|---|---|---|---|---|
| `DooplexGiteaOffsiteStale` | critical | 10-24 00:57 → 10:51 | true | the push refused a 12 h old dump | the dump fix; R-232 note |
| `PostgresqlDumpStale` | warning | 10-23 22:17 → 10-24 10:44 | true | dump job login failed (28P01) | homelab `531023d` |
| `DiskOfflineUncorrectable`, `DiskUncorrectableSectors` (Z6A07P2G) | critical | whole window (≥ 145 h) → 10-23 12:02 | true | the dying HDD4 | silenced by you 10-12; pulled 10-23 — R-934 closed |
| `DiskDisappeared` (Z6A07P2G) | critical | 10-23 12:17 → 18:02 | true | the pull | expected; cleared |
| `DiskInventoryChanged` | warning | 10-23 12:17 → 20:17 | true | the pull | expected; cleared |
| `LonghornVolumeDegraded` (two Plex volumes) | warning | 10-23 08:22 → 11:22 | true | the HM-040 rebuild | cleared |
| `PodRestartingFrequently` (felhom-webpage) | warning | 10-19 09:02 → 10:07 | true | git-syncs stopped on errors | R-1163 (closed 10-19) |
| `WanUploadDegraded`, `WanDownloadDegraded`, `WanUpstreamPacketLoss`, `WanBufferbloat` | warning/info | minutes, 10-17 → 10-20 | true | short WAN dips (the flood days) | none needed |
| `AuthentikHighLatency` | warning | 10-21 08:12, 5 min | true | a short spike | none needed |
## The mail change (homelab `22d7697`, applied by a restart, read back from the running Alertmanager)
| | before | after |
|---|---|---|
| critical | every 1 h | every 4 h |
| warning | every 4 h | every 12 h |
| other (info) | every 4 h | every 12 h |
| backup alarms | one mail stream per alarm | one mail for the group, every 4 h |
| a disk | one stream per alarm | one mail per drive, daily (unchanged) |
| resolved mail | `send_resolved: true` | unchanged; the mail counter rose once after each clear (10:47, and by 10:57) — that they read „RESOLVED" is not seen: the mailbox is not mine |
## Part 4 — R-1250, committed and proven
Agent `29bcf44` + `1e087bb`, controller `046b808` (commit only; release in the next train). A seeding box: the agent
says „waiting for quiet"; the controller records the torrent app's limit, holds it at 1 KiB/s, asks for one test,
puts the exact limit back. **Proven on demo-hp with a real seed** (10 Mbit/s from 9202 to DooPlex's qBittorrent):
refused without the hold (11.1 Mbit/s); with it the uplink fell to 0.02–0.16 Mbit/s, 94.2 Mbit/s, 56.8 s, the limit
back to 1 249 280 B/s; a kill in the middle → the start-up check put it back. The brief said „a nested VM": NOT done —
a nested VM would need its own tunnel to ep0 to run the real test; demo-hp's host with probes did the same with the
real tunnel. The pinned agent client is unit-tested only.
## Commits, CI, undo
| Repo | Commit | What | CI |
|---|---|---|---|
| homelab-manifests | `531023d` | the dump CronJob's secret | no CI in this repo |
| homelab-manifests | `22d7697` | the mail routes, the dump alarm's label | no CI |
| homelab-manifests | `9f399fe` | REPORT | no CI |
| felhom-agent | `29bcf44`, `1e087bb` | R-1250 | 2286, 2288 success |
| felhom-controller | `046b808` | R-1250 | 2287 success |
| app-catalog-felhom.eu | `7de554e`, `5e196e1` | the box walk: HDD_PATH, the leftover check | 2289 success, (final push) |
| felhom-agent / felhom-controller | `5e05627` / `321df1f` | REPORT, README | (final push) |
| felhom.eu | this commit | register, STATUS, CONTEXT, `08` §9, evidence | (final push) |
Undo: `audits/dooplex-alarms-2026-10-24/README.md` § Undo. Backups of the three homelab files: `*.before` there.
## Rows
**Before 191, after 190; opened 0, closed 1 (R-934 — the disk was already pulled).** R-232: a dated note and a dated
check (2026-10-25, tonight's sweep). R-1250: NARROWED (fixed on main, waits for the release train).
## Observations
- The database's two secrets for one role stay (only the CronJob moved) — homelab list item 1 (not a Felhom row).
- A stale dump stops the Gitea + secrets off-site push for as long as it lasts — NOT-A-FINDING: the designed fail-closed
order (`PLAN.md`: the dump first, for consistency); the alarm caught it inside its 26 h, and it now comes in one mail
with its cause.
- Alertmanager cannot reload a ConfigMap change (the brief assumed it could) — NOT-A-FINDING: the homelab manifest's
design (`envsubst` init container); you chose the restart; the fact is in CC's memory and the homelab list.
- The agent said „waiting for quiet" before it had watched a minute — fixed in the session (`1e087bb`, found live).
- The box walk sent a folder inside the drive as HDD_PATH (refused since R-839) — fixed in the session (`7de554e`).
- The box walk called the template folder a leftover after every remove — fixed in the session (`5e196e1`).
- R-934 said the failed disk was still in DooPlex; STATUS still listed R-1214's done check — fixed (row closed, line removed).
- CC's memory said Prometheus has no config-reloader; it has one since 2026-10-12 — fixed (memory note).
- `govulncheck` warns GO-2026-6809 (crypto/x509, stdlib) in the agent and the controller — NOT-A-FINDING here: the Go
toolchain row R-972 owns it.
- My Gmail connector's mailbox gets no Alertmanager mail — NOT-A-FINDING: it is another mailbox than `SMTP_TO`; 60 sent,
0 failed in 7 days. The Alertmanager config did not change yesterday (last 10-12).
- `kubectl exec -i` never closed stdin for `pg_restore` — NOT-A-FINDING: my method; the host's own `pg_restore` read the dumps.
- The 9202 ISO torrent never uploaded (behind NAT, 18 leechers / 1080 seeds) — NOT-A-FINDING: why the LAN peer was used.
- I overwrote homelab-manifests' REPORT.md, which held another session's HM-050 report from today — NOT-A-FINDING: the
repo's overwrite rule; HM-050's entry stays in its CHANGELOG.
## Teardown, three layers
- **Machine:** 9202 — qBittorrent removed through the product (no container, no volume; the template folder stays, as for
every app); the test files, the forwarder and the probe removed. DooPlex's qBittorrent — the test torrent and its
files deleted (by hash), nothing else touched.
- **Host:** demo-hp — the agent probe stopped and deleted, no iperf3 left; the installed agent untouched. DooPlex — the
9202 password copy and the helper's session files shredded; the clones removed.
- **Hub:** nothing provisioned.
## Instruction files, poster
No instruction file edited. No poster fact changed (the mail intervals are not on it).
+42
View File
@@ -0,0 +1,42 @@
# REPORT — Plex to the fixed version in the catalog and on the demo/test boxes (2026-10-23 day, Plex2Shell)
**Plain words first.**
- The catalog now gives Plex 1.43.5.11029-6bf8af3f0 (catalog `6cc317d`, pushed without the update test, on your ruling).
- One box ran Plex: demo-hp. It showed no sign of the attack. CC pressed Update; it runs the new Plex now.
- You have one decision (D39): may a security fix reach the boxes at night without the full update test?
## Per box
| Box | Plex before → after | Signs check | Box walk |
|---|---|---|---|
| demo-hp 9201 | 1.41.4.9463-630c9f557 → 1.43.5.11029-6bf8af3f0 | clean (NOT-A-FINDING; logs reach back only to 00:32 UTC today — CPU history and files cover 10-16 → 10-22) | green: all containers healthy, 0 errors, 0 warnings since the press |
| demo-hp 9202, demo-felhom 9201, tester-1 9201 | no Plex | — | — |
| Tester4 | hub host DOWN, VM gone | — | — |
| Tester 2 | no host on the hub | — | — |
Detail and evidence: `documentation/audits/plex-demo-boxes-2026-10-23/README.md`.
## Records
- Register 184 → 186: opened R-1238 (P2, Plex has no test record or fixture), R-1239 (P2, a fast lane for security
fixes — the operator's decision); closed none.
- `CONTEXT.md`: the 10:56 ruling. `STATUS.md`: decide row D39. No „broken" line: Part 2 was finished before the
records were written, so the line the brief asked for would have been added and removed in the same commit.
The „Updated" paragraph was left as the day session wrote it (it is that session's summary; this run adds one
decide row only).
- Architecture read: `09-update-architecture.md` §3 decisions 13 and 52; `13-threat-model.md` (Plex row);
`runbooks/target-selection.md`. Decision 13 is unchanged; no poster fact changed (none of the three poster actions).
- `--no-verify` was used ONCE, for catalog `6cc317d`, on the ruling. The `check-test-record-move.py` refusal is verbatim
in `app-catalog-felhom.eu/REPORT.md`.
## Fixed without a row
- The memory index (`.claude-memory/MEMORY.md`, outside git) had grown to 201 lines / 25 735 bytes at 11:09 (another
session), over the 200 / 25 600 ceiling, so `instructions_gate` failed and so did two script suites that run it.
Lines 71–72 (two memories about the deleted credentials file, both superseded by `sec`) became one line: 200 lines,
25 590 bytes. `repo_gates.py --fast` then OK. The index sits right at the ceiling; the next added line trips it again.
## Teardown
Machine: Plex stays on the new version; helper scripts removed from guest and host. Host: nothing provisioned.
Hub: reads only.
@@ -0,0 +1,75 @@
# REPORT — did the Plex attack on DooPlex reach Felhom's secrets? (2026-10-23)
## Plain words first
1. **Verdict: NOT REACHED.** Two open doors led from the hacked Plex to Felhom's secrets. Neither was used in the attack
days: the backup folder got no new connection at all, and Longhorn made no backup or disk outside its normal plan.
2. **The shared backup folder: closed to the home network and the tailnet** (your yes). It is **still open to every app
on DooPlex** — Longhorn needs that, so the real fix is the homelab session's.
3. **What you do next:** give the homelab session the hand-over list (`documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md`,
last section). Nothing needs to be rotated.
## The Part 4 table
In `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md` § Part 4. Short form: every Felhom secret on
DooPlex has exposure **none**. The ones a hacked pod *could* have reached: the hub's unencrypted nightly copies (with the
unsealed Cloudflare/tunnel/git tokens, R-975) through the NFS export or a Longhorn backup, and the operator signing keys
through the NFS file-handle escape (documented, not exercised). Sealed hub columns need a key that is not in the copies;
escrow blobs need each household's recovery code; the k8s API, the kubelet and the hub refuse a caller with no credential
(measured).
## The paths and the evidence per path
| path | existed? | evidence of use (window 10-17 04:00Z → 10-23 08:04Z) |
|---|---|---|
| NFS export `longhorn-pvc` (`*`, `no_root_squash`, `no_subtree_check`; `secure` void in pods) | yes | **unused** — `node_nfsd_connections_total` flat at 288 for 13 days, 3902 samples, no gap; positive control: today's three probe connections moved it to 4. nfsd is TCP-only. Reads only in Longhorn's backup hours |
| Longhorn API :9500, no login | yes | **no sign of use** — 0 backups first seen off-schedule (1-min scrape); every night wrote to the local store; no unexplained volume; the backup target is the 2025-11-29 one. Manager logs: lost (reboot) |
| hub `hub:8080` direct | yes, but no secret without a credential | 0 requests from 10.42.0.52 in `/data/access-log` (from 10-18 10:25Z; positive control: the probe's IP is logged). 10-17 04:00Z → 10-18 10:25Z: no record |
| k8s API, kubelet | no (401) | — |
| SSH, Samba on DooPlex | login needed | 0 sshd lines from the pod network (control: 61 from the LAN); no Samba client file for 10.42.0.52 (positive control: the probe made one) |
| Vaultwarden | login needed; vault encrypted | logs lost (reboot) |
**Evidence lost before this session started:** the 13:20Z maintenance reboot erased the pre-reboot container logs of
longhorn-manager, longhorn-ui, the ingress, the hub, Gitea, Vaultwarden, Grafana and Prometheus. Reproduced
independently from records that survived: Prometheus history (nfsd, Longhorn, cAdvisor, node network), the hub's own
access log, the journal (boot -1), Samba per-client files, CoreDNS error lines, the Alertmanager silence list. The
incident session's pre-reboot greps (10.42.0.52 in longhorn-manager, CNPG, Gitea, authentik: 0) are cited as theirs.
## For the homelab session
The seven items at the end of the audit README: the export still open to pods (and `secure` is void there); Longhorn API
and UI with no login; an unattributed Longhorn delete of the old Plex volume at 11:24:15Z; **~1.5 TB sent out by the Plex
pod 10-18 → 10-22** (not in INCIDENT.md) plus `hand.genddos.st` DNS lookups on 10-19; the reboot that erased the logs;
the monitoring stack and Headlamp reachable from every pod; a `python3 main.py` listener on :8443. No HM numbers minted;
`homelab-manifests` not touched.
## What this session changed
- **DooPlex `/etc/exports`** (operator yes, ~15:18Z): `longhorn-pvc` from `*` to `10.42.0.0/16`, `exportfs -ra`; Longhorn
target `available: true` at 15:20:10Z. Backup `/etc/exports.bak-2026-10-23-plex2shell`. The only change on DooPlex.
- **Probe pod** `default/cc-p2s-probe`: created 15:19:46Z, removed 15:33:44Z; before/after listings in the evidence. No
other object created. The incident's NetworkPolicies untouched.
- **felhom.eu:** the audit folder; R-1241 and R-1242 in `OPEN-ITEMS.md`; `13-threat-model.md` (attacker j, one boundary
row); `STATUS.md` (verdict in „What works", the P1 in „What is broken", one „do" item; the old „Updated" paragraph moved to
`STATUS-HISTORY.md`).
- Nothing rotated. No secret value written anywhere (evidence scanned for secret shapes; the only 64-hex string is the
sha256 of `/etc/exports`).
## Rows
Before 186, after 188; opened 2 (R-1241 P1, R-1242 P3), closed 0.
## Observations
- Hacked pod → hub backups + signing keys through the NFS export and the Longhorn API — FILED: R-1241.
- Demo NAS export on the same disk with `no_subtree_check` — FILED: R-1242.
- The hub answers pods without the geo filter — NOT-A-FINDING: the geo filter is not authentication; every no-token path refused.
- demo-felhom's Proxmox UI and SSH reachable from any DooPlex pod — NOT-A-FINDING: the door of R-982; added as attacker j in `13` §3.
- Pod logs erased by the reboot — NOT-A-FINDING for Felhom (the hub's access log persisted, R-1140); handed to the homelab session.
## Notes for the next session
- Teardown, three layers: machine — probe pod removed, export change kept on purpose (operator yes); host — none; hub —
none (read-only access-log aggregation in the running pod).
- System poster: no fact on it changed (none of the three applied).
- `unproven.py`: not run — no capability claim changed.
+106
View File
@@ -0,0 +1,106 @@
# REPORT — the Monday release, run on Sunday 2026-10-25 (attended)
**Plain words first.**
1. **Fix A and fix B are proven.** A restore whose app does not start says so and keeps both copies (D49 = A); the old
copy goes only after 3 minutes of stable running. A `.fab` import that cannot download its version refuses and
touches nothing.
2. **Hub 0.170.0 is live.** Tester 1's recovery package went to the hub in one transaction; no alarm mail after.
3. **Agent 0.165.0 is vouched** (18:38:49 CET). It runs on all four hosts; on a BYO host our SSH file is removed — one
bundle run late (R-1285).
4. **Controller 0.343.0 is on every box:** 4 of 4 hosts (Tester 1, demo-hp, demo-felhom, the BYO test host) and the
scratch guest 9202. None stayed behind.
5. **The catalog is pushed:** n8n sends no usage data, OpenGist has 256 MiB, Homepage is behind the family gate.
**The D47 upgrade count:** Tester 1 3 of 3 apps got an install id, demo-felhom 5 of 5, demo-hp 11 of 12 (kimai is held
stopped since 2026-10-14 and the start-up step skips a held app — R-1286). **0 units set aside on any box.**
**The dated checks:** R-232 — DONE (last night's DooPlex database copy reached the restic sweep, snapshot `3973450f`,
folder `20261024-220001`, no alarm mail since 2026-10-23 22:20Z). R-1183 — partial today (one hub snapshot file per
slot, `07:00Z` and `13:00Z`; the hub restarted at 15:50Z for 0.170.0, so the morning log is gone); the full read stays
due 2026-10-26.
## What ran, in order
- **Part 1, fix A (R-1270 follow-up):** one helper, „started" = running on every sample of a 45 s window, no
`RestartCount` growth, every healthcheck `healthy`; the undo copy released after 3 more minutes (10 s samples, 30 min
watch; only a restore begun after the watch ends it). Found live and fixed on the way: docker's `-f` template failed on
every app without a healthcheck (now the JSON is read). Live on 9202 rc.6: an image that exits after 20 s → „még nem
indult el", copy kept, one operator line; the original unit back → released 3 min later.
- **Part 2, fix B (R-1279):** the D48 helper before step 1 of the import. Live: a bundle copy with a missing tag →
refused with the D48 sentence, the app running, its volumes untouched.
- **Part 3, hub 0.170.0 (R-1254):** asked → yes → manifest bump, hard refresh, sync of `Deployment/hub` alone,
Synced/Healthy, digest `sha256:6478498d…`. Tester 1 ran the escrow ceremony through the page's endpoints (the new code
stored by the operator as a CC secret); the hub logged „the new package is on the hub". No alarm mail after the
deploy (the mailbox as the second channel; a bind mail to Tester4 arrived, the control).
- **Part 4, agent 0.165.0 (R-1257, R-1250):** `release-agent.sh` from a clean clone; signed jobs to the three boxes
(76/76, the appliance drop-in in force by `sshd -T`). **No BYO host existed, so one was built** on demo-hp: VM 9371, a
plain Proxmox VE 9.2 from an answer-file ISO, Felhom installed with `--mode byo` as Tester4 (the operator deleted the
old host and stored the retrieval passphrase as a CC secret). Our 0.164.0 drop-in planted first; after 0.165.0 still
there; a second config bundle removed it, wrote nothing, `sshd -T` read the owner's own setting. Vouched by the
operator.
- **Part 5, controller 0.343.0:** rc.6/rc.7 on 9202; the set-aside alarm added and red-proved (Part 5.2); `0.343.0`
pushed after CI 2330; per-box floors → the three boxes at 17:17Z; box checks: 0 ERROR lines, all containers up,
https-only green; the D47 count above; on Tester 1 a unit restore and a set-aside restore — the item back, the copy
released after 3m0s each. Asked → yes → global floor 17:53:42Z (MinAgent 0.158.1, read back in the hub log; the BYO
host moved 18 s later); per-box floors cleared; a 30-minute watch: the hub sent 0 alarms (`global-floor-watch.txt`) and no alarm mail arrived (the mailbox as the second channel).
- **Part 6, catalog:** Homepage's family gate tried on the drill on 9202 (stranger 302/401, household 200); asked →
yes → `ebbf1f4` pushed; demo-hp read it 8 minutes later.
- **Part 7, D44/D45 = all picks:** confirm lines tagged, text-follows-code changes made (old words struck through), the
four code picks became R-1280–R-1283; R-1265 and R-1266 closed.
- **R-1250:** released, but its live proof cannot run before the agent's next due window (Tuesday 10:00–15:59 local; all
boxes measured on 2026-10-24) — a dated check on 2026-10-27 with the operator's yes for DooPlex's qBittorrent as peer.
## Versions, digests, CI by commit
- **Hub 0.170.0** `sha256:6478498d…`.
- **Agent 0.165.0** binary `e9d04a47ee85222ce6c472633ecdcebe1baa88d6836848f9a7da6435a9adb7d3`, bundle
`8dea1f34d77ba40aa44c5461c3728c2ca94bab46854a6bd8b86191f520dfe85a`; agent CI by commit: `c8d3c17` 2306 ✓, `4a1a849` 2325 ✓, `4697f8a` 2326 ✓, `6b6ba0c` 2334 ✗ (R-1290 — the registry, not the code).
- **Controller 0.343.0** `sha256:bb3da72a054c45751e3baf6e4bad43d1561a952717c5f426d65a5aeed0413a3b` from `e2a9761`.
CI: `7720439` 2319 ✓, `2149a13` 2321 ✓, `35232de` 2322 ✓, `1ee0353` 2324 ✓, `3a1f09d` 2327 ✓, `935e624` 2328 ✓,
`e2a9761` 2330 ✓, `fe98377` 2333 ✓.
- **Catalog** `ebbf1f4` 2332 ✓, `cd0cf48` (REPORT).
- **felhom.eu** `ade357a1` 2329 ✓, `36ef8dcd` 2331 ✓, this commit (CI by hash in the final message).
- **Installer 1.36.1** on main, NOT published (R-1284).
## Register
Before **219**, after **219**; opened **11** (R-1280–R-1290), closed **11** (R-1251, R-1252, R-1253, R-1269, R-1270,
R-1271, R-1279, R-1257, R-1272, R-1265, R-1266). R-1254: the hub half closed in its row, the ceremony half open. R-1250:
VERIFY with a dated check. `unproven.py --summary`: no number moved (35 of 55 not walked).
## Observations
- FILED: R-1284 — the installer's preflight printed every refusal code twice („401401"), so its 401/404/409 sentences
never showed (R-1167's 409 sentence included); fixed on main (1.36.1, red-proved), the publish is the operator's.
- FILED: R-1285 — BYO: our drop-in is removed one bundle run late (the old wrapper plans the run that installs the new one).
- FILED: R-1286 — demo-hp's kimai held since 2026-10-14, no row named it.
- FILED: R-1287 — a set-aside mails „1 of 1 apps failed to back up" (red); nothing failed.
- FILED: R-1288 — after a set-aside restore, the next backup sets the reinstall's own unit aside too (a second copy and
alarm).
- FILED: R-1289 — the hub's „fleet's newest controller" can be a dev build no box runs (`0.343.0-dev.r1251a`).
- FILED: R-1280–R-1283 — the four D44 code picks.
- FILED: R-1290 — the agent's CI is RED since ~18:00Z: four old agent packages (0.159.0–0.160.0) were removed from the
registry by something outside the repos, and the `published` gate still expects the newest 10 (`6b6ba0c`, job 2334).
Not the release's code (job 2326 on the release commit was green); the keep rule is the operator's.
- NOT-A-FINDING: the BYO host's first bundle read 71/76 (`pve:store-grant:felhom-pbs`) — the PBS storage was being added;
76/76 two seconds later and on every run after.
- NOT-A-FINDING: the `.fab` import status reads `"ok": true` beside its `error` — `ok` is the HTTP envelope, the job's
result is `error` and the steps.
- NOT-A-FINDING: 9202's alarms read „DROPPED … notifier disabled" — 9202 has no hub; the line is the positive observable
that the event was raised (one per case).
## Teardown (three layers)
- **Machine:** every throwaway app removed through the product (9202: Vikunja, Tandoor, Homepage; Tester 1: Tandoor —
no container, volume, `app.yaml` or set-aside left). 9202 back on the released 0.343.0, its drill mirror and my saved
`controller.yaml` copy removed, my `0.343.0-rc.3…7` images removed (older rc images of earlier sessions left as found).
Tester 1 keeps its new escrow package on purpose (the old recovery code no longer works; the new one is a CC secret).
- **Host:** the BYO VM uninstalled through the product's `--uninstall` (rc 0), then VM 9371 destroyed with its disk; its
ISO removed from demo-hp's `local`; demo-hp lists only VM 341 and its old ISO, as found. DooPlex: only the hub deploy;
the BYO ISO work folder deleted; the session's password, session and job files shredded.
- **Hub:** the three per-box floors cleared; the global floor 0.343.0 and the vouched manifest stay (the release). The
host record `Tester4-f967d4` stays until the operator deletes it (session only — in STATUS); customer Tester4 is kept.
**Secrets:** used only through `sec`; the evidence scanned for every stored value used (0 hits) and the BYO root
password (0 hits); one CSRF token blanked in a saved page snippet. **System poster:** no fact changed. **Instruction
files:** none edited.
+29
View File
@@ -0,0 +1,29 @@
# REPORT — the restore and backup holes, D43 = A, and the D44/D45 picks (2026-10-24 night, unattended)
**Plain words first.**
- A power cut during a restore of an app on a drive no longer brings old data back later (R-1251, the P1). Proven on
Tester 1 with a real power-off and a drive that came back late.
- One wrong line in an app's backup settings no longer deletes its second-drive copy (R-1252). Proven on 9202.
- An export no longer overwrites a running restore's record (R-1253, proven on 9202); the hub saves a recovery package
in one step (R-1254, tested, not deployed).
- BYO hosts keep their own SSH settings (your D43 = A, R-1257). Tested in the bundle's harness.
- The review's 32 notes have one line each with a pick: answer „all picks" or name the ones you disagree with.
- Nothing is released. Monday: controller v0.343.0, agent v0.165.0 + your vouch, hub v0.170.0.
**Evidence and tables:** `documentation/audits/restore-holes-2026-10-24/README.md`; the picks:
`documentation/audits/review-round-2-2026-10-24/D44-D45-picks.md`.
**Commits:** controller `bb2809d`, `062f9b4`, `518a135`, `c9b4b82`, `e1613ae`; felhom.eu `dcf4f84c` (hub) and this
commit (documents, register, STATUS, CONTEXT, evidence); felhom-agent `c8d3c17`, `4a1a849`. CI by hash: in the final
report.
**Register:** before 207, after 208; opened 1 (R-1268), closed 0. R-1251/R-1252/R-1253/R-1257 → VERIFY (fixed on main,
waits for the release); R-1254 → NARROWED (the hub half fixed); R-1265/R-1266 carry the picks; R-1267 extended.
**Documents edited:** `07` (the R-1251 rule after D22's paragraph; the R-1252 rule in §8.2a; a pointer in §6.5), `05`
(the escrow transaction), `03` and `11` §1 (D43 = A). **System poster:** no fact on it changed (none of the four).
**Instruction files:** none edited.
**Machines:** Tester 1 and 9202 put back as found except leftover empty anonymous volumes of the R-1267 class (named in
the audit). demo-hp: nothing provisioned; the catalog sweep's VM untouched. DooPlex, ep0, household boxes, the hub: not
changed. No release, deploy, vouch, publish or catalog push.
+59
View File
@@ -0,0 +1,59 @@
# REPORT — a backup is never changed by a new install, a restore downloads first, a restore tells the truth (2026-10-25, unattended, commit only)
**Plain words first.**
- **R-1269 is fixed and proven:** a reinstall no longer writes into the old backup. The old backup is set aside whole,
listed on the restore page, and restores with its own password (Tandoor and AdventureLog on 9202: the account back).
- **D48 download-first on every restore path that starts a definition (the unit restore and with it the second
drive's unit and whole restore, both kept-data loads and a set-aside unit; the off-site restore; the second drive's
file restore; the no-unit fallback).** An unpullable version is refused and the app is never stopped (proven).
- **R-1270:** a restore whose app does not start says so — never „it worked"; the old copy is kept (proven).
- **R-1271:** cause — the other box's off-site copy was dated in the future (its clock), and „newest wins" took it;
now a copy more than 10 minutes in the future never counts as the newest (proven on a forged copy).
- **One question for you:** D49 in STATUS — after a restore whose app does not start, keep the data and wait (CC's
pick, done) or pour the old data back by itself.
**Safest readings taken** (also at the head of `documentation/audits/restore-truth-2026-10-25/README.md`): an install
identity field, not the data stamps; the set-aside unit is a D25 kept copy (remove-with-backups deletes it, as it does
other kept copies); a „partial write" means one half into another install's unit; R-1270 keeps the data and the undo
copy (no new retention rule); D48 tells a deleted version from a busy registry by Docker's answer; 10 minutes of clock
skew; the `.fab` import left to its own row.
**Architecture read:** `07` §6.3, §6.5, §6.6, §8 (D20/D22, §8.2a D25); `03` (drive identity); `09` §3 decisions 14,
40, 53; rows R-1269–R-1271, R-1206, R-698, R-933, R-1267.
**Each proof, before (`e1613ae`, 9202 on 0.342.0) → after (9202 on 0.343.0-rc.2):**
- R-1269: the kept unit's `app.yaml` rewritten (`cbfcb538…` → `e19d2610…`), restore broken → set aside byte for byte,
restored, the account back, 0 password errors (Tandoor, AdventureLog).
- R-1270: `ok: true` over a `starting` app → `ok: false`, `not_started`, the honest sentence, the undo copy kept.
- D48: the first pull after the stop (code order) → an unpullable pin refused, the container's start time unchanged.
- R-1271: the future copy chosen (sweep) → the own copy chosen; alone, the future copy marked and said.
**Commits.** Controller: `02daebe` (D47) CI 2311 ✓, `b680db8` (R-1270) 2312 ✓, `2d71270` (D48) 2313 ✓, `bd9f4f7`
(R-1271) 2314 ✓, `1f9c863` (the kept list without off-site) 2315 ✓, `1f82959` (remove deletes the undo copy) 2316 ✓,
`e8d2151` (REPORT). felhom.eu: this commit (`07` §6.5/§6.6/§8.2d/§8.2e, `09` §3 decisions 256–257, CONTEXT, register,
STATUS, STATUS-HISTORY, the audit with its evidence and tools) — CI by hash in the final message. Test images
`0.343.0-rc.1` (`bd9f4f7`) and `0.343.0-rc.2` (`1f9c863`) in the registry; nothing released.
**Teardown.** Machine: 9202 only; my throwaway apps removed through the product; two empty folders and one undo copy of
mine removed by hand after checking; my rc images removed from its Docker; back on 0.342.0; the apps found running still
run. Host: the session's 9202 password and session files shredded. Hub: provisioned nothing. ep0: nothing.
**Register:** before 217, after 219; opened 2 (R-1278, R-1279), closed 0; R-1269, R-1270, R-1271 → VERIFY (fixed on
main, wait for the release); notes on R-1206 (cross-reference) and R-1243 (its dated check done, read-only); R-232's
dated check moved one day (this brief: no DooPlex work).
**Observations:**
- FILED: R-1278 (a volume app's reinstall does not offer its kept backup).
- FILED: R-1279 (the `.fab` import removes the app's volumes before its pull — D48 does not cover it).
- FIXED without a row (found live, red-proved): the kept-copies list was hidden on a box without off-site (`1f9c863`);
a kept undo copy outlived remove-with-backups (`1f82959`).
- NOT-A-FINDING: Immich's 404 after the sweep's load — the copy brought its own address (`photos.`), as every unit
restore does; the app was healthy there.
- NOT-A-FINDING: „immich: definition REFUSED by the allow-list" on 9202 — present before this run's image; an earlier
session's Immich, not this change.
- NOT-A-FINDING: the D48 refusal reads „Visszaállítás sikertelen: …nem lehet letölteni… Az alkalmazáshoz nem
nyúltunk" — the page's failure frame around an honest sentence; true as written.
**Secrets:** used only through `sec`; the evidence scanned for the box password and the session tokens (the CSRF token
in two saved page snippets was the positive control and is blanked); units compared by sha256 only. **System poster:**
no fact changed. **Instruction files:** none edited.
+33
View File
@@ -0,0 +1,33 @@
# REPORT — review round 2 (2026-10-24, unattended): the documents against the code, data loss and security
**Plain words first.**
- The 17 design documents were checked against the code: 493 claims, 121 out of date. About 130 places are corrected or
marked with a dated note; no rule, fence or promise was changed.
- About 30 things the code does that no document named are now written down for you to confirm (D44), and seven places
where a document's ruling or promise says something the code does not do are waiting for you (D45).
- The worst data-loss finding: a power cut during a restore of an app on a drive can later pour old data over the
household's new data (R-1251, P1). Proven with the real recovery code; it needs a fix in an attended run.
- Also proven, on the scratch box: one wrong line in an app's backup settings deletes that app's second-drive copy
(R-1252, P2).
- The worst security finding: the „root logs in by key only" SSH rule also reaches a host the household owns (BYO),
where we promised to change nothing (R-1257, D43). Nothing serious was found in the hub's or the controller's new routes.
**Evidence and the full tables:** `documentation/audits/review-round-2-2026-10-24/README.md`; every document edit with
its `file:line` in `EDITS.md`; the reviewers' candidates in `reviewers/`; the two proofs in `proofs/`.
**Commits (CI read by hash):** controller `b01bb4b` (two small fixes, red-proved; unreleased) — CI 2294 / #480 success;
controller `2449223` (REUSE.md) — 2298 / #481 success; felhom-agent `f5c3d77` — 2296 / #261 success and catalog
`227a544` — 2297 / #256 success (the rule-file alignment); felhom.eu `77015997` (documents batch 1) — 2295 / #1253
success; felhom.eu `876d9236` (batch 2, the register, STATUS, CONTEXT) — 2299 / #1254 success.
**Register:** before 190, after 207; opened 17 (R-1251–R-1267), closed 0.
**Instruction files edited (facts only, named here):** `.claude/rules/unprompted-work.md` in felhom-agent, the catalog
and the workspace root — the missing „A version number is pushed once" paragraph (before 95 lines, md5 3b071ad2; after
byte-identical to the other two, b1891bd4); `felhom.eu/.claude/rules/live-probes.md:41` — `/opt/felhom/stacks/<name>` →
`/opt/docker/stacks/<name>/app.yaml` (the real path, and the folder stays after a clean remove).
**System poster:** the facts file and the html changed together (text only, the hub-DB push times); no new drawing.
**Machines:** 9202 used for the R-1252 proof and put back (one difference: Immich's new empty volume, R-1267). DooPlex,
ep0, household boxes and the hub: not changed. No release, deploy, vouch or publish.
+17
View File
@@ -0,0 +1,17 @@
# REPORT — D39 = B: the security-fix lane (2026-10-23, attended)
**Plain words first.** An app with a public security hole can now get its fix at night on every box, after one test
on the test box. Controller 0.341.0 does it and is the floor since 13:24 (your yes). Older controllers skip such a fix
at night and take it when a person presses Update — both seen on the test box. Nothing needs you.
- Design: `09-update-architecture.md` §3 decision 255 (+ a dated note on decision 13). Capability map B23.
- Catalog `3898593` (+ a box-walk address fix after it); controller `63dcccf` = 0.341.0; drill commits `8c1d908` …
`17e0da0` (the drill only).
- Evidence and the full table: `documentation/audits/security-lane-2026-10-23/README.md`.
- Register 186 → 186: closed R-1239 (D39 = B); opened R-1240 (9202's 34 saved configs with the catalog token); R-1238
stays open, its text now says the next Plex fix can use the lane.
- `STATUS.md`: D39 out of the decide table, one „what works" line, the „Updated" paragraph replaced (the old one is in
`STATUS-HISTORY.md`). `CONTEXT.md`: the rulings. System poster: no fact changed (none of the three actions).
- Instruction / fact files edited: `app-catalog-felhom.eu/CLAUDE.md` (the lane, added); `operations/nodes.md` line 221
(9202 „same dashboard password" → its own password, measured) and line 46 (Tester 1 `.101` → DHCP, `.148` today).
- No `--no-verify` in this run.
+129
View File
@@ -0,0 +1,129 @@
# REPORT — weekend night 2026-10-23: the hub's last plain tokens locked; the upload and CPU watch (spike, design, build)
**Safest readings taken (unattended), stated first:**
- **A test image is not a release.** The upload watch was proven on scratch 9202 with `felhom-controller:0.342.0-rc.1`,
set by hand in 9202's `/etc/felhom-controller-image` (the shape of today's security-lane rc.1). An `-rc` tag is never
offered to a household's update button. No floor was touched; only 9202 ran it, and 9202 is back on 0.341.0.
- **9202's line was seeded by hand** (one 40 Mbit/s record): its off-site backup is off by design, so it can never learn a
line itself. The learning path is pinned by unit tests and by tonight's one-second read of the real pushes on the
three test boxes (read only).
- **The hub 0.168.0 image was built** (the brief asked for it) **and not deployed.** The four new hub event types are in
hub 0.169.0, committed only.
- **Per-app trigger instead of a whole-box trigger.** The brief said „the box's upload above X % of the line". Inside its
container the controller cannot tell LAN from internet bytes, nor see the box's total; it can see each app's exact
bytes, and the per-app view is what the explanation compares. So the trigger is per app (§6.2 of the spike).
- **The first day:** apps the box can ask are judged at once on the 5 Mbit/s floor (a mismatch is evidence whatever the
line); apps it cannot ask are judged only after ~20 h of history. The brief's „first day: no alarm, only learning" holds
for those apps; for the others a mismatch on day one is still an alarm.
## Five lines
1. **Tokens locked:** the Cloudflare tunnel token, the Cloudflare API token and the git token are sealed in **hub 0.168.0**
— built, CI green, **not deployed** (you deploy it on Monday).
2. **Backup copies:** two options — encrypt the hub's Longhorn disk, or take it out of the Longhorn backup because the
hub's own copy to ep0 is already encrypted at rest and in transit (measured). **My pick: take it out** (D40).
3. **The line:** measured from the box's own nightly off-site upload (one-second samples, best 10 seconds) — a lower
bound; most nights the upload is too small to say anything, so a real test is your decision (R-1247).
4. **Apps that explain their traffic:** qBittorrent and Transmission (their own byte counters, within 1–7 %), Plex (its
viewers and transcodes). Jellyfin needs a key the household makes; Nextcloud and Immich give no number.
5. **The fake hacked app was caught: yes** — a flood from inside qBittorrent, hidden behind a real seed: one alarm after
15 minutes, one clear 15 minutes after it stopped. A CPU burner in Plex: the same. A real seed: no alarm.
## The spike's numbers
`documentation/audits/SPIKE-upload-watch-2026-10-23.md` (evidence beside it). Short form:
- **Normal levels** (three boxes, 19–31 days of one-minute samples): no app's own upload p95 reached 6 Mbit/s; the
longest run at ≥ 5 Mbit/s was **3 minutes**, at ≥ 1 Mbit/s 9 minutes. CPU: the longest run at ≥ 150 % of a core was
**7 minutes**, at ≥ 80 % 13 minutes. DooPlex's hacked Plex: ~44 Mbit/s for hours, ~10 cores.
- **Line speed:** at a one-minute sample the push exceeded 2 Mbit/s on 3 of 81 box-nights (the push moves a few MB);
the box keeps no byte count of it. Tonight's one-second read: §Night data below. No active test was run (ep0 changes
forbidden; a public test's terms not read).
- **Apps:** qBittorrent counted 131.1 Mbit/s while its container sent 134.6–140.5 (×1.03–1.07); Transmission ×1.001–1.02;
the limit API works and persists (KiB/s rounding); Plex answers its session lists (with the token it already has when
claimed); Jellyfin 401 without a household-made key.
- **A hard cap** (`tc` from a helper in the app's network namespace) works in an unprivileged guest: a flood held at
20 Mbit/s, the app kept answering. Not built (R-1248).
## The household's line (final text)
- HU: „A qBittorrent feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod."
- EN: "qBittorrent uploads at most N Mbit/s, so the family internet stays fast. You can change this in qBittorrent."
- The system settings page: „Feltöltési sebesség: legalább N Mbit/s (az éjszakai távoli mentésből mérve)" /
"Upload speed: at least N Mbit/s (measured from the nightly off-site backup)"; „még mérjük …" until known.
## Morning list (Monday)
1. Deploy hub 0.168.0 (the last plain tokens locked). Then read its start line `config tokens sealed at rest (8 …)`.
2. D40: the hub's plain Longhorn copies — stop that backup (my pick) or encrypt the disk; and whether to delete the two
plain copies already there.
3. D41: qBittorrent's default limit — 50 % of the measured line (my pick). D42: who is told — the family only about the
limit, the hacked-app alarms only you (my pick).
4. Release the upload watch when you say yes (hub 0.169.0 build + deploy, then controller 0.342.0 through the normal path).
5. For the homelab session (list only, no HM numbers): an upload-saturation alert on DooPlex's `eno1` (the line is
~100 Mbit/s; the flood peaked at 905 Mbit/s).
## Commits, images, CI (by commit)
| Repo | Commit | What | CI |
|---|---|---|---|
| felhom.eu | `5277d859` | hub 0.168.0: the tokens sealed; R-1243; STATUS D40 | run 2269 **success** |
| felhom.eu | `af5c005a` | hub 0.169.0: the four events (allowed, operator-only, mail line) | run 2271 **success** |
| felhom.eu | `a91d0b24` | spike, design (`08` §9), live proofs, rows R-1244..R-1248, STATUS | run 2272 **success** |
| felhom-controller | `a9c1997f` | v0.342.0 (unreleased): the upload and CPU watch | run 2270 **success** |
| felhom-controller | `60e487d` | CONTEXT + REPORT | run 2273 **success** |
Images: `felhom-hub:0.168.0` `sha256:3bfdecaae527d4762059600cdfaad83daf4e4b5b20661fc47acf128180d87cb3` (from `5277d859`,
not deployed); `felhom-controller:0.342.0-rc.1` (from `a9c1997f`, scratch 9202 only, removed from 9202 after).
**Teardown, three layers:** machine — 9202: the four throwaway apps removed through the product (containers and volumes
gone), my test files, probes and the `alpine:3.20` helper image removed, back on 0.341.0, the seeded record removed;
host — demo-hp: the receiver LXC 9471 destroyed with its disk; DooPlex: my private copy of the hub snapshot shredded,
the 9202 password copy shredded; hub — none. ep0: read only (the hub-copy snapshot list). No household box touched.
## Rows
**Before 188, after 194; opened 6 (R-1243 … R-1248), closed 0.** R-975 stays open (WAITING-ON-OPERATOR: deploy 0.168.0).
## Observations
- The hub's Cloudflare/tunnel/git tokens unsealed in `config_json` — FIXED in hub 0.168.0 (R-975, not yet deployed).
- The hub's Longhorn copies are plain and sit in the open folder — FILED: R-1243.
- The upload watch is not released — FILED: R-1244; its limit size — FILED: R-1245; who is told — FILED: R-1246.
- The line is only a lower bound — FILED: R-1247. An app's API can lie — FILED: R-1248.
- `appliance_registrations.pairing_code` is plain — NOT-A-FINDING: it is shown on the box's screen and is one of two
factors; the other is hashed.
- R-879's sealing (0.138.0) ran no VACUUM, so old plaintext of those columns could sit in freed blocks of the live
`hub.db` — NOT-A-FINDING: 0.168.0's one-time VACUUM rewrites the file on its first start.
- The hub CHANGELOG's 0.167.0 header still said „NOT built" though it is deployed — fixed without a row (small).
- The shared memory index went over its size ceiling twice in one day (another session's lines) and blocked the push
gates — fixed without a row (shortened, no link lost); it sits 14 bytes under the ceiling now.
- The watch's very first minute after a start cannot reach apps by name (network joined seconds later) — NOT-A-FINDING:
that minute only sets baselines.
- The media wiring's generated override file stays after an app's removal — NOT-A-FINDING: rewritten before every start.
- `docker stats` text in the stored metrics has three significant digits — NOT-A-FINDING for the watch (it reads the
Engine API); the charts are display only.
- `felhom-samba` runs on the host network and has no counter — NOT-A-FINDING: infrastructure, not judged.
- Transmission's rate under its limit was not observed (no peer contact) — NOT-A-FINDING: the limit API was read back;
qBittorrent's was observed.
- A dated check came due on the push (R-1214, Tester 1's night copy): read read-only — the guest's memory „full" stall
was 444 µs in total over 11 h, 0 OOM kills; the copy was small (0.24 GB) and the capped container is not shown — the
row stays WATCHING, the due line removed.
- The night's line floor is low (10 Mbit/s on demo-hp, one second at 70) — FILED in R-1245's text (D41), no new row.
## Instruction files, poster
No instruction file edited. System poster: no fact on it changed (nothing deployed) — none of the three applied.
## Night data
Read 02:08–02:30Z, once a second, on the three test boxes (read only; spike §2.1):
- **demo-hp: counts.** Best 10 seconds 12.6 MB = **10.0 Mbit/s**. One second reached 70 Mbit/s.
- **demo-felhom: too small** (best 10 seconds 3.2 MB = 2.5 Mbit/s).
- **Tester 1: too small** (3.5 MB = 2.7 Mbit/s).
**What it means:** the nightly push is a burst of a few seconds. So the line it gives is a low floor. On demo-hp,
50 % of that floor is about 5 Mbit/s for qBittorrent — probably much less than the real line. The floor rises only when
a bigger night comes (the box keeps the best of 14 nights). Added to the D41 decision row (R-1245): if you want a
looser limit before a real line test exists (R-1247), pick a higher share or „none".
+1 -1
View File
@@ -96,7 +96,7 @@
| `(*Server).configFormData` (v0.49.0) | hub/internal/web/configs.go (~L430) | `(r, isNew, cfg, overrides, errMsg) configFormView` | The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | `overrides=nil` → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). |
| `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" <configFormView>}}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `</form>` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `<script>`. |
| `handleHostEscrowGet` + `escrowSelfServiceRetrieval` | hub/internal/api/handler.go | `GET /hosts/{id}/escrow` | The box-authenticated MIRROR of the escrow PUT — self-scoped by the per-host key. **`escrowSelfServiceRetrieval` is THE single decision point** for whether a box may read its own blob without operator-armed recovery mode (§8.2 vs §8.3, R-199): flip it, do not scatter the condition. **Never merge this with `dr.go`'s `handleReEnroll`/`handleGetRestoreDirective`** — those rotate the host API key and serve the K-escrow and directive too, and keep their recovery-mode gate (pinned by `TestEscrowGet_OperatorDRPathUnchanged`). Every successful retrieval MUST raise `escrow_blob_served` before the bytes leave; that audit row is the mitigation the trade rests on. |
| `demoteCurrentEscrowTx` (+ `(*Store).SaveHostEscrow`) | hub/internal/store/store.go (~L2597/~L2612) | `(tx, hostID) (int64, error)` / `(hostID, blob, fp, posture, createdAt, resticPwSHA) (superseded bool, prevResticPwSHA string, err error)` | **THE ONE escrow row-copy routine** — used by the re-escrow retention AND by `DeleteHost`'s custody demotion; never write a second one | **It must copy BOTH sealed artifacts** — `blob` (K-escrow / PBS key) and `identity_blob` (the age bundle carrying the offsite restic repo password). Omitting the second is R-198: two months of retaining the wrong key, with the ceremony as the destroying act. **Ordering it depends on:** `SaveHostDRBundle` writes `identity_blob` AFTER `SaveHostEscrow` returns, so the demote sees the PREVIOUS generation — invert that and the retained bytes are the new blob under the old hash. Pinned by `TestSaveHostEscrow_RetainsIdentityBlob` + `TestDeleteHost_DemotesIdentityBlob` (both callers). `prevResticPwSHA` feeds R-197's changed-key signal; it is a hash and never leaves the store. |
| `demoteCurrentEscrowTx` (+ `(*Store).SaveHostEscrow` / `SaveHostEscrowWithIdentity`) | hub/internal/store/store.go (~L2597/~L2612) | `(tx, hostID) (int64, error)` / `(hostID, blob, fp, posture, createdAt, resticPwSHA) (superseded bool, prevResticPwSHA string, err error)` | **THE ONE escrow row-copy routine** — used by the re-escrow retention AND by `DeleteHost`'s custody demotion; never write a second one | **It must copy BOTH sealed artifacts** — `blob` (K-escrow / PBS key) and `identity_blob` (the age bundle carrying the offsite restic repo password). Omitting the second is R-198: two months of retaining the wrong key, with the ceremony as the destroying act. **Ordering it depends on:** since R-1254 (unreleased, next hub v0.170.0) `SaveHostEscrowWithIdentity` writes the key, its hash and `identity_blob` in ONE transaction, with the demote FIRST inside it, so the demote sees the PREVIOUS generation — invert that and the retained bytes are the new blob under the old hash. `SaveHostEscrow` is that function with no identity blob; `SaveHostDRBundle` (a second statement) is no longer called by the escrow PUT. Pinned by `TestR1254_*`. Pinned by `TestSaveHostEscrow_RetainsIdentityBlob` + `TestDeleteHost_DemotesIdentityBlob` (both callers). `prevResticPwSHA` feeds R-197's changed-key signal; it is a hash and never leaves the store. |
| `(*Store).CountHostArtifacts` / `DeleteHost` | hub/internal/store/store.go (~L1640/~L1690) | `(hostID) (HostArtifacts, error)` / `(hostID, deleteEscrow bool) error` | Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (`ErrHostEscrowPresent`, tx never starts). log_bundles die by `scope_id == host_id` ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
| `(*Server).commitCustomerReset` (v0.69.0) | hub/internal/web/customer_reset.go (~L165) | `(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError` | THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the `customer_resets` journal | Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). `purgeEscrow` governs ONLY whether `PurgeCustomerResetDBState` destroys retained custody: standalone RESET passes the operator's `escrow_ack`; the DELETE cascade passes **false** so custody dies exactly once, in its leg 3. Returns a `resetLegError` carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. |
| `(*Store).CustomerResidue` / `PurgeCustomerResidue` (v0.70.0) | hub/internal/store/customer_delete.go | `(customerID) (*CustomerResidue, error)` / `(customerID) error` | Counting + purging the report-derived state and the credential-bearing bindings a deleted customer leaves behind | **`GetCustomers()` is REPORT-derived** — until the reports are gone the customer stays on the Customers list AND stays in the staleness/offsite checkers’ work list, so a deleted customer keeps emailing the operator. Both funcs walk ONE shared `residueQueries` list so a table can never be counted-but-not-purged. Includes `appliance_registrations` + `selfbind_tokens` (credential-bearing, not telemetry). NEVER touches `events`, `notification_log`, `host_deletions`, `customer_resets`. |
+23 -26
View File
@@ -2,26 +2,28 @@
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
**Updated 2026-10-23 (morning).** Your three picks are built (the tunnel repairs itself with a new key, the hub skips a keyless deleted install, the controller repository stays public), and the night copy can no longer outlive the controller. Four fresh reviews found 34 points, every real one fixed or a row; the test night PASSED (no failed check at night, every night job passed, nothing lost). Nothing was released or deployed; the morning list has the order, and the DooPlex survey asks four questions (D32–D35).
**Updated 2026-10-25 (evening, attended release).** Every box runs the new controller: a restore that does not start says so and keeps both copies (D49), a reinstall never touches the old backup, a restore downloads first. The hub stores the recovery package in one step; the new agent is vouched; the catalog has three fixes.
## What works
- **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241).
- A box installs from nothing with the public installer, links to the hub, and moves to the floor by itself (map A1, A3, A4).
- 65 catalog apps install, update one tested step at a time, and a failed update undoes itself (map B1, B3, B4, B6).
- Backups: database dumps, a second-drive copy, an off-site copy and whole-box copies; restore tests run by themselves (map C1–C4, A17, A18).
- A restore brings an app back with its data and its database password; a failed or cut restore puts the app back as it was — off-site live, local on the test image (map C5, D6, D16).
- A restore brings an app back with its data and its database password; a failed or cut restore puts the app back as it was; a restore that does not start says so and keeps both copies (map C5, D6, D7, D16).
- After a power cut the box starts by itself, cleans up a half-written backup, and Immich repairs broken photos (map C10, F12).
- A second drive is offered as the backup target; media apps see every drive; a one-disk box gets its own data area (map D1, D13, D15).
- The three test boxes answer only over https; the hub's key opens nothing on a box; password guesses are counted (map E12, E13, E17).
- The hub mails alarms and recoveries, and keeps one whole-box copy per week on ep0; CC uses its own limited token, not your password (map F1, F11, G15).
- A hacked app that uploads or computes more than it explains alarms only you; each box tests its upload weekly; qBittorrent gets half (map F13).
- An urgent security fix reaches the boxes at night after one test on the test box; the box backs up first and undoes a failed update (map B23).
- The hub mails alarms and recoveries and keeps a weekly whole-box copy on ep0; CC uses its own limited token (map F1, F11, G15).
## What is broken
- **P1:** about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours).
- Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862).
- DooPlex: the failed disk is still in the box, its alarms silenced (R-934); something sets the shared checkout to „bare" (R-935).
- The scratch box cannot use agent features (R-1083).
- **P1:** any hacked app on DooPlex can read the hub's nightly backups, and maybe the signing keys (R-1241, homelab session); about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours).
- Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862). The scratch box cannot use agent features (R-1083).
- DooPlex: something sets the shared checkout to „bare" (R-935).
## What we work on
- Dated checks, run by CC. 2026-10-23: no app stop when the host is busy (R-1173). 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts).
- Dated checks, run by CC. 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the qBittorrent line test on a seeding box, with DooPlex's qBittorrent as the peer (R-1250, your yes); the weekly clean-up window (R-435; the logs of the last one were lost to restarts). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes).
## Waiting on you — decide
@@ -29,11 +31,8 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|---|---|---|---|---|---|---|
| D36 | R-1225, R-1226 | Two website sentences before the publish: „8 GB runs the basics; the photo app needs 16 GB” and „what you open from outside the home passes Cloudflare's network, readable there; the backups leave only encrypted”? | **Yes, both** (the patch's own aim is the whole truth) | — | the patch publishes with both promises a little too strong | — |
| D35 | R-1228 | After the release week, serve the website and the installer from Cloudflare, like the ISO? | **Yes, later** (no new installs stop when DooPlex is down) | keep them home: after a DooPlex loss, no install until it is back | the same | — |
| D34 | R-232 | Copy the hub database to ep0 every 6 hours instead of every night? | **Yes** (0 €, about 30 minutes, one DooPlex change) | keep nightly: up to a day of hub changes lost, e.g. a new box's recovery package | the same | — |
| D33 | R-1227 | Copy the released images and agent packages to ep0 every night? | **Yes** (0 €, about 2 hours of CC, a few GB on ep0) | a timed rebuild drill instead: about 3 hours once, and rebuilt images are not the same | after a DooPlex loss no box updates, reinstalls or installs until everything is rebuilt | — |
| D32 | R-923, R-924 | Print the break-glass sheet now (eight keys and logins + ep0's fingerprint) and keep it outside the home? | **Yes** (30 minutes; nothing else opens the off-site copies after a loss) | an encrypted USB stick outside the home: ~10 €, 1 hour, one more passphrase | after a DooPlex loss no off-site copy opens | — |
| D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — |
| D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — |
| D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — |
| D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — |
@@ -42,23 +41,21 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
**Still needed? (yes/no)** — CC could not check these. Answer yes or no per line.
1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). Since 2026-10-16.
2. The N100 sheet, about 30 minutes: Plex, the admins in Prowlarr, Sonarr and Radarr, your download sources (`documentation/audits/n100-lived-in-2026-10-15/SHEET.md`). Since 2026-10-15.
3. Look at these pages on demo-hp when you have time: Calibre-Web („Adatok és mentés"), the file manager, AdventureLog, Plex, the app page, the Plex and Sonarr guides („Útmutató"). Since 2026-10-16.
3. Look at these demo-hp pages when you have time: Calibre-Web, the file manager, AdventureLog, Plex, the app page, the Plex and Sonarr guides. Since 2026-10-16.
4. Remove the test server „plex-k6" from your Plex account (plex.tv → Settings → Authorized Devices). Since 2026-10-15.
5. Ask Google to index `https://felhom.eu/sajat-felho-tarhely` in Search Console (R-1035). Since 2026-10-14.
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14.
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
**Morning, 2026-10-23 — in this order.**
1. Hold or go: CC says go — the test night passed on a fresh 8 GB box (no failed check at night, every night job passed, nothing lost; one gap: no whole-box copy to ep0, R-1232). One question from the review: D31 as built also covers an install that got the whole-box password but no recovery package — its password is rotated without your press (R-1205). CC: go. If not: revert one hub commit before item 4.
2. Release controller 0.340.0 (main after tonight; tested as rc.8 on the scratch box, rc.6 on the test box): the safe night copy, the catch-up fix, R-49, R-1187, R-1189, R-1214. If not: these wait.
3. Release the agent (the tunnel repair, its tunnel field, the root-SSH bundle) and the installer tag with the root-SSH removal (R-1204, R-982); then vouch. If not: a clock-ahead box stays cut off; root keeps its password on the LAN.
4. Build and deploy hub 0.167.0 (D31, the tunnel alarm and its clearing). If not: a dead tunnel stays silent.
5. Publish the website patch (D3, D5), with D36's two sentences if yes. If not: the old wording stays.
6. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-16fdef (tonight's test box). If not: two dead entries alarm.
7. The DooPlex hour: D32–D35 in the table. If not: a DooPlex loss stays as the survey describes.
**Day, 2026-10-23 — your do list.**
1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens.
2. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-f967d4 (the BYO test host, removed 2026-10-25). If not: dead entries alarm.
3. Give the homelab session R-1241, R-1242, the lists at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md` and `documentation/audits/dooplex-alarms-2026-10-24/README.md`, and an upload alarm on DooPlex's `eno1` (~100 Mbit/s line; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups.
**The rest**
- Pull the failed disk Z6A07P2G out of DooPlex (R-934). If not: its alarms return on 11 November. Since 2026-10-12.
- System poster needs a refresh: the new 01:40 „release copy” arrow from DooPlex to ep0 (D33); the hub-database label already says every 6 hours. If not: the drawing misses one copy.
- Tell CC what removed four old agent packages from Gitea on 2026-10-25 (a clean-up rule?): the agent's CI is red until the gate and the rule agree (R-1290).
- Publish installer 1.36.1 (tag + webpage sync): a wrong passphrase then reads „passphrase REJECTED", not a doubled code (R-1284).
- Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09.
- Two leftovers on DooPlex: the Facebook `first-post/` folder in `leftovers/2026-10-14/` (keep, commit or delete); the `deny`-list edit in homelab-manifests' `.claude/settings.json` (commit or drop). If not: they stay. Since 2026-10-14.
- Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784); print the break-glass sheet and the signing keys (R-923, R-924).
- Later (yours, not urgent): R-1229 (ArgoCD's sync settings into git), R-862 (Tester 2's bootstrap, when it is back), R-504, R-1071, R-1073, R-1077, R-1042, R-132, R-908, R-779, R-904, R-232, R-433, R-933, R-213, R-832, R-388, R-882, R-883, R-814, R-1056, R-1164, R-930, R-920, R-902.
- Two DooPlex leftovers: `leftovers/2026-10-14/first-post/` (keep or delete); the `deny`-list edit in homelab-manifests' `.claude/settings.json` (commit or drop). Since 2026-10-14.
- Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784).
- Later (yours, not urgent): R-1250 (a seeding qBittorrent is never tested), R-1248 (a hard upload cap per app), R-1229 (ArgoCD's sync settings into git), R-862 (Tester 2's bootstrap, when it is back), R-504, R-1071, R-1073, R-1077, R-1042, R-132, R-908, R-779, R-904, R-232, R-433, R-933, R-213, R-832, R-388, R-882, R-883, R-814, R-1056, R-1164, R-930, R-920, R-902.
+24 -22
View File
@@ -80,17 +80,17 @@
| <a id="a4"></a>[A4](capability-map-detail/A.md#a4) | Bare-metal Felhom ISO (blank hardware → zero-touch auto-install → first-boot `host-install`); selectable UEFI loader; universal secret-free / operator-bind mode | scripts v1.19.0 (`scripts/iso/`) + hub v0.62.0 + assistant container | **PROVEN-LIVE** | N100 (2026-07-18) and HP t740 (2026-07-21) | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18); `audits/SPIKE-baremetal-iso-2026-07-16.md` (2026-07-16) |
| <a id="a5"></a>[A5](capability-map-detail/A.md#a5) | Box survives a wrong-NIC install: hub-unreachable first boot → legible Hungarian console screen (NIC table + remedy) + NIC sweep self-heal (bounded DHCP + hub probe per NIC, success-only persist), and the baked root password is operator-knowable (`<iso>.rootpw.txt`) | scripts v1.24.0 (`felhom-bootstrap.sh`, `build-felhom-iso.sh`) | **PROVEN-LIVE** | nested drill; metal proof pending | `audits/SPIKE-firstboot-nic-sweep-2026-07-22.md` (2026-07-22) |
| <a id="a6"></a>[A6](capability-map-detail/A.md#a6) | Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** | drill VM; no non-operator human yet (R-3) | `audits/DRILL-day0-vm-2026-07-12.md` (2026-07-12) |
| <a id="a7"></a>[A7](capability-map-detail/A.md#a7) | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** | customer-zero, on metal | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
| <a id="a8"></a>[A8](capability-map-detail/A.md#a8) | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** | drill VM, endpoint-exact; browser wizard first fired 2026-07-18 | `audits/SPIKE-controller-escrow-2026-07-13.md` (2026-07-13); `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
| <a id="a7"></a>[A7](capability-map-detail/A.md#a7) | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** | customer-zero, on metal; **re-check (2026-10-24):** mechanism changed after its evidence — f90e2229 (bind mail W3–W5) | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
| <a id="a8"></a>[A8](capability-map-detail/A.md#a8) | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** | drill VM, endpoint-exact; browser wizard first fired 2026-07-18; **re-check (2026-10-24):** mechanism changed after its evidence — agent c11e3e2 + controller ba4a6f0 (typed-back words, R-1151) | `audits/SPIKE-controller-escrow-2026-07-13.md` (2026-07-13); `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
| <a id="a9"></a>[A9](capability-map-detail/A.md#a9) | One name per secret on every surface — „Beállító kód" (the setup code) and „Tulajdonosi jelmondat" (the owner passphrase) — and the re-enrol mail points at a page a rebuilt box shows | controller v0.211.0, hub v0.104.0 + v0.105.0 (R-295, R-323) | **IMPLEMENTED** | written BUILT: source-read; no customer has typed the passphrase | R-295; R-323 |
| <a id="a10"></a>[A10](capability-map-detail/A.md#a10) | The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE** | demo-felhom, real endpoints | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04); `audits/RECON-offsite-dr-chain-2026-08-04.md` (2026-08-04) |
| <a id="a11"></a>[A11](capability-map-detail/A.md#a11) | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** | demo/drill fleet only | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
| <a id="a10"></a>[A10](capability-map-detail/A.md#a10) | The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE** | demo-felhom, real endpoints; **re-check (2026-10-24):** mechanism changed after its evidence — hub 9688e76e (sealed package back to a rebuilt box, R-1176) | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04); `audits/RECON-offsite-dr-chain-2026-08-04.md` (2026-08-04) |
| <a id="a11"></a>[A11](capability-map-detail/A.md#a11) | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** | demo/drill fleet only; **re-check (2026-10-24):** mechanism changed after its evidence — agent ae69ad4 (D30 WireGuard re-key) | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
| <a id="a12"></a>[A12](capability-map-detail/A.md#a12) | A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE** | 2026-10-21 (fix brief, hub 0.163.0): PASSES — the rebuilt Tester4 box (public ISO) got the household's sealed package from the hub, asked for the recovery code, opened the history; Nextcloud came back with 3 031 of 3 033 manifest files byte-identical (the 2 never reached the old box) and the next off-site runs appended (63 → 64). Earlier the same day it FAILED (R-1176, now closed); the failure paths are not claimed (R-379, R-380) | `audits/fixes-2026-10-21/P1-live/` (2026-10-21); `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `audits/DRILL-r201-night-run-2026-08-04.md` (2026-08-04); `audits/DRILL-r356-hot-only-restore-2026-08-22/` (2026-08-22); `audits/DRILL-r356b-driveless-db-restore-2026-08-22/` (2026-08-22) |
| <a id="a13"></a>[A13](capability-map-detail/A.md#a13) | The customer's own UNAIDED recovery journey, end to end | controller v0.206.0, hub v0.98.0, agent v0.127.0 | **PROVEN-LIVE** | 2026-10-21 (fix brief, hub 0.163.0): PASSES with ONE operator act by design (D19: delete the dead host) — install, bind, claim, the recovery offer, a wrong code refused, the right code, „Teljes visszaállítás", all through the household's own pages; the new-link page now tells a household whose old box is still linked to write to Felhom (R-1203). Earlier the same day it stopped at the rebuilt dashboard (R-1176) | `audits/fixes-2026-10-21/P1-live/` (2026-10-21); `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `tests/walk5-r201-2026-08-07/journal.md` (2026-08-07); `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md` (2026-08-05) |
| <a id="a14"></a>[A14](capability-map-detail/A.md#a14) | A random night of household actions while random things go wrong — twelve rounds, unattended | controller v0.245.0, agent v0.131.0, hub v0.116.0, golden 0.245.0, ISO 1.28.0 | **PROVEN-LIVE** | nested box, twelve chaos rounds | `audits/DRILL-chaos-night-2026-09-17.md` (2026-09-17) |
| <a id="a15"></a>[A15](capability-map-detail/A.md#a15) | A stranger's FIRST HOUR on a fresh install — download → install → claim → two apps → use → backup → remove → restore, plus a power cut and a mistyped code | controller v0.336.0, agent v0.161.0, hub v0.162.0, golden 0.331.0, installer 1.35.0, ISO 1.29.0 | **PARTIAL** | 2026-10-20: download → install → bind → claim → 8 apps → use → backup → remove → every restore path → power cut → mistyped codes all worked; the 8 GB box FROZE twice in the first afternoon and only a plug pull brought it back (R-1192), and the owner passphrase needed the operator | `audits/rehearsal-2026-10-20/journal.md` (2026-10-20); `audits/DRILL-fresh-install-0242-2026-09-14.md` (2026-09-14); `audits/DRILL-new-household-2026-09-30.md` (2026-09-30) |
| <a id="a16"></a>[A16](capability-map-detail/A.md#a16) | Off-site app-data capture covers the paths an app declares MANDATORY — on BOTH drive layouts | controller v0.197.0 | **PROVEN-LIVE** | both drive layouts | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04) |
| <a id="a17"></a>[A17](capability-map-detail/A.md#a17) | Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE** | demo-hp | `audits/SPIKE-r82-phase0-2026-07-26.md` (2026-07-26) |
| <a id="a17"></a>[A17](capability-map-detail/A.md#a17) | Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE** | demo-hp; **re-check (2026-10-24):** mechanism changed after its evidence — agent 1b6e316 (local keep-daily, R-1149) | `audits/SPIKE-r82-phase0-2026-07-26.md` (2026-07-26) |
| <a id="a18"></a>[A18](capability-map-detail/A.md#a18) | Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard | agent v0.104.0 → v0.121.0, hub v0.77.0 → v0.91.0 | **PROVEN-LIVE** | — | *(evidence not on file — not re-verifiable)*: the row cites per-repo CHANGELOGs and the spec `backlog/SPEC-r85-phase4-5-2026-07-26.md` — a spec, not a run record |
| <a id="a19"></a>[A19](capability-map-detail/A.md#a19) | A restore-test can never fill the box's disk: it sizes the restore first (uncompressed), keeps off the tested guest's pool when it can, refuses what does not fit, and retries a failed clean-up on a timer | agent v0.133.0 (RELEASED, NOT DELIVERED), hub v0.124.0 | **IMPLEMENTED** | written BUILT: the refusal proven on demo-hp 2026-09-24; the full-test path not live | `audits/r672-2026-09-24/` (2026-09-24); R-672 |
| <a id="a20"></a>[A20](capability-map-detail/A.md#a20) | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0; purge-first hub v0.142.0 | **PROVEN-LIVE** | two real firings (2026-07-18); third 2026-10-22 on a real customer (Tester-2) with the R-32 purge-first leg — all legs ok, others untouched | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18); `audits/tester2-reset-2026-10-22/` |
@@ -102,17 +102,17 @@
| ID | Capability | Component | Status | Scope | Evidence |
|---|---|---|---|---|---|
| <a id="b1"></a>[B1](capability-map-detail/B.md#b1) | Deploy an app from the catalog (env config, memory guard, health-aware progress) | controller, catalog (~52 apps, images pinned) | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/RERUN-p1p3-2026-07-06.md` (2026-07-06) |
| <a id="b2"></a>[B2](capability-map-detail/B.md#b2) | App lifecycle: start/stop/restart/update/logs/remove/redeploy | controller | **PROVEN-LIVE** | the actions; remove-with-data re-proven 2026-09-13 | `audits/R442-2026-09-13/` (2026-09-13); `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
| <a id="b1"></a>[B1](capability-map-detail/B.md#b1) | Deploy an app from the catalog (env config, memory guard, health-aware progress) | controller, catalog (65 apps, images pinned) | **PROVEN-LIVE** | **re-checked 2026-10-24** on controller 0.342.0 (after 6d90374 + 2fe6341): all 65 apps through the deploy request on a fresh box — 62 installed and answered, Wanderer installed but cannot run without DNS (R-739), 2 not offered by lifecycle; the memory guard did not fire (one app at a time) | `audits/catalog-sweep-2026-10-24/README.md` (2026-10-24); `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/RERUN-p1p3-2026-07-06.md` (2026-07-06) |
| <a id="b2"></a>[B2](capability-map-detail/B.md#b2) | App lifecycle: start/stop/restart/update/logs/remove/redeploy | controller | **PROVEN-LIVE** | the actions; remove-with-data re-proven 2026-09-13; **re-check (2026-10-24):** mechanism changed after its evidence — 8ae9178 (removal deletes labelled volumes, R-1117) | `audits/R442-2026-09-13/` (2026-09-13); `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
| <a id="b3"></a>[B3](capability-map-detail/B.md#b3) | Update is GUARDED: it refuses without a restorable backup, backs up first when the copy is stale, and HOLDS an app that does not come up — on ANY backup tier, and the release itself arrives by the managed floor | controller v0.237.0 + v0.238.0 + v0.238.1 + v0.239.0, hub v0.112.0 | **PROVEN-LIVE** | — | `audits/slice4-2026-09-13/` (2026-09-13); `audits/rulings-r472-r475-2026-09-13/` (2026-09-13) |
| <a id="b4"></a>[B4](capability-map-detail/B.md#b4) | A failed update is UNDONE by the box itself — the previous version back with its data from seconds before the update; the app is held only if that undo fails too | controller v0.263.2 (the undo), v0.264.0 + hub v0.120.0 (the mail) | **PROVEN-LIVE** | scratch 9202 | `audits/undo-live-2026-09-23/README.md` (2026-09-23) |
| <a id="b5"></a>[B5](capability-map-detail/B.md#b5) | The catalog holds only TESTED steps — an image move without a proven test record (bench + box, digests the registry still serves) is refused at push time | catalog `6db08a5` (test-record gate, `ladder.py`) | **PROVEN-LIVE** | written PROVEN: the catalog gate against the live registry | `audits/DRILL-night-2026-09-23.md` (2026-09-23) |
| <a id="b6"></a>[B6](capability-map-detail/B.md#b6) | A box behind climbs ONE tested step per press, each with its own definition (`09` §3 decision 14) | controller v0.268.0 + catalog `5ed599c` | **PROVEN-LIVE** | — | `audits/ladder-2026-09-24/partD/10-romm-two-steps.json` (2026-09-24) |
| <a id="b7"></a>[B7](capability-map-detail/B.md#b7) | A file app comes back WHOLE from the second drive — its files by four rules (never delete, never overwrite a newer file, bring back what is missing, keep an older replaced file beside it), then its settings and database (`09` §3 decision 26) | controller v0.269.0 (`backup/tier2_whole.go`) | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A1/21-readback.json` (2026-09-24) |
| <a id="b7"></a>[B7](capability-map-detail/B.md#b7) | A file app comes back WHOLE from the second drive — its files by four rules (never delete, never overwrite a newer file, bring back what is missing, keep an older replaced file beside it), then its settings and database (`09` §3 decision 26) | controller v0.269.0 (`backup/tier2_whole.go`) | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d582feb + 5929b8e (D22, D25) | `audits/night-2026-09-24/A1/21-readback.json` (2026-09-24) |
| <a id="b8"></a>[B8](capability-map-detail/B.md#b8) | A crash-looping or out-of-memory app is STOPPED by the box, the household and operator are told, and Start gives one more try (decision 28) | controller v0.269.0, hub v0.123.0 | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A3/21-gokapi-page-event.txt` (2026-09-24) |
| <a id="b9"></a>[B9](capability-map-detail/B.md#b9) | A held app whose page says support is informed can only be removed KEEPING its data (decision 27) | controller v0.269.0 | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A2/10-held-no-copy-keep-data.json` (2026-09-24) |
| <a id="b10"></a>[B10](capability-map-detail/B.md#b10) | The box runs the exact TESTED image of a floating tag, and an installed app keeps its image until a guarded Update moves it (`09` §6.4 part 6, box half) | controller v0.269.1 (`stacks/digest.go`) | **PROVEN-LIVE** | — | `audits/night-2026-09-24/B/21-floating-tag-0.269.1.json` (2026-09-24) |
| <a id="b11"></a>[B11](capability-map-detail/B.md#b11) | Automatic updates at night — the update leg (`09` §6.4 part 7) | controller v0.271.0 | **PROVEN-LIVE** | scratch 9202, six simulated nights; demo boxes in Part D | `audits/DRILL-night-2026-09-25.md` (2026-09-25) |
| <a id="b11"></a>[B11](capability-map-detail/B.md#b11) | Automatic updates at night — the update leg (`09` §6.4 part 7) | controller v0.271.0 | **PROVEN-LIVE** | scratch 9202, six simulated nights; demo boxes in Part D; **re-check (2026-10-24):** mechanism changed after its evidence — 63dcccf (security_fix steps, 0.341.0) | `audits/DRILL-night-2026-09-25.md` (2026-09-25) |
| <a id="b12"></a>[B12](capability-map-detail/B.md#b12) | What `restart` and `update` do to a deployed app whose compose file the catalog already moved | controller v0.235.0 | **PROVEN-LIVE** | HISTORY: the behaviour up to v0.234.0; changed 2026-09-06 — current behaviour is B17 | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
| <a id="b13"></a>[B13](capability-map-detail/B.md#b13) | Whether the box UPGRADES an app by itself, with nobody pressing anything | controller | **PROVEN-LIVE** | answer: yes, only when an app fails to come back (measured 2026-09-01) | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
| <a id="b14"></a>[B14](capability-map-detail/B.md#b14) | Whether an app UPGRADE can be undone | controller + catalog | **PROVEN-LIVE** | answer: no, once a migration has run | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
@@ -124,6 +124,7 @@
| <a id="b20"></a>[B20](capability-map-detail/B.md#b20) | App crashes → customer notified (one event per transition, no flapping spam) | controller v0.120, hub v0.48 | **IMPLEMENTED** | crash→customer mail never live-confirmed | `audits/CAMPAIGN-3-2026-07-11.md` (2026-07-11) |
| <a id="b21"></a>[B21](capability-map-detail/B.md#b21) | Post-deploy optional config (API keys etc.) with restart | controller, catalog `.felhom.yml` | **IMPLEMENTED** | config-save + restart in no campaign | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
| <a id="b22"></a>[B22](capability-map-detail/B.md#b22) | Backup classification: 13 bind-bearing apps carry mandatory/optional/excluded classes | catalog, controller v0.132–133 | **PROVEN-LIVE** | — | `audits/SPIKE-backup-classification-2026-07-14.md` (2026-07-14); `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15) |
| <a id="b23"></a>[B23](capability-map-detail/B.md#b23) | An urgent security fix reaches the boxes at night — a `security_fix` ladder step (box walk required, bench optional, a CVE/GHSA notice), taken by the night update (`09` §3 decision 255, D39 = B) | catalog `3898593`, controller v0.341.0 | **PROVEN-LIVE** | scratch 9202 (drill catalog), Plex 1.43.4 → 1.43.5; an app with no box fixture is checked by health, version and a soak only (`seed: none`) | `audits/security-lane-2026-10-23/` (2026-10-23) |
## C. Protection & recovery (the product promise)
@@ -146,10 +147,10 @@
|---|---|---|---|---|---|
| <a id="c1"></a>[C1](capability-map-detail/C.md#c1) | Whole-guest backup lands OFF the guest's own physical device; a single-drive box is recorded DEGRADED rather than silently normal (installer Case A/B) | agent v0.113, host-install v1.22.0 | **PROVEN-LIVE** | — | `audits/E2D-fresh-vm-2026-07-29.md` (2026-07-29) |
| <a id="c2"></a>[C2](capability-map-detail/C.md#c2) | Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15) |
| <a id="c3"></a>[C3](capability-map-detail/C.md#c3) | Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14) |
| <a id="c4"></a>[C4](capability-map-detail/C.md#c4) | Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/VALIDATION-offbox-storagebox-2026-07-09.md` (2026-07-09) |
| <a id="c5"></a>[C5](capability-map-detail/C.md#c5) | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/DIAG-immich-restore-round2-2026-07-19.md` (2026-07-19) |
| <a id="c6"></a>[C6](capability-map-detail/C.md#c6) | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6B-2026-07-14.md` (2026-07-14) |
| <a id="c3"></a>[C3](capability-map-detail/C.md#c3) | Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 03c0a69 + 66a69d4 + 5929b8e (capped container R-1214, kept copies D25) | `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14) |
| <a id="c4"></a>[C4](capability-map-detail/C.md#c4) | Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 8f5bea0 + d5f88a2 (abandon R-1182, off-site proof R-1189) | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/VALIDATION-offbox-storagebox-2026-07-09.md` (2026-07-09) |
| <a id="c5"></a>[C5](capability-map-detail/C.md#c5) | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 93fb251 + d582feb (D20, D22) **2026-10-25:** changed again by controller 0.343.0 (D48 pull-first, R-1270 fix A: the verdict needs a stable start, the undo copy goes after 3 more minutes); on 0.343.0 Tester 1's local unit restore and set-aside restore passed and released the copy (`audits/release-2026-10-26/t1-restores-0.343.0/`); this path itself not re-walked. | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/DIAG-immich-restore-round2-2026-07-19.md` (2026-07-19) |
| <a id="c6"></a>[C6](capability-map-detail/C.md#c6) | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d5f88a2 + 5929b8e (R-1187 c, R-1191) | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6B-2026-07-14.md` (2026-07-14) |
| <a id="c7"></a>[C7](capability-map-detail/C.md#c7) | Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `pilot/DRILL-GL6-2026-07-08.md` (2026-07-08) |
| <a id="c8"></a>[C8](capability-map-detail/C.md#c8) | PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | scoped to one host | `audits/SPIKE-pbsdr-selfheal-2026-07-15.md` (2026-07-15) |
| <a id="c9"></a>[C9](capability-map-detail/C.md#c9) | Box survives an unattended app or guest-network failure (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0→v0.190.0, agent v0.92.1 | **PROVEN-LIVE** | demo box; boot-orphan leg 6 of 6 hard resets | `audits/DIAG-bootrecon-drive-absent-2026-08-02.md` (2026-08-02) |
@@ -171,17 +172,17 @@
| <a id="d3"></a>[D3](capability-map-detail/D.md#d3) | Data migration between drives (all / per-app), crash-safe | controller | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14); `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
| <a id="d4"></a>[D4](capability-map-detail/D.md#d4) | NAS (NFS/SMB-client) verify-before-commit, uid-1000 probe, categorized Hungarian errors, DSM-validated | controller v0.113–117, agent v0.81/84/85 | **PROVEN-LIVE** | — | `audits/SPIKE-nas-verify-2026-07-11.md` (2026-07-11); `audits/SPIKE-nas-dsm-2026-07-11.md` (2026-07-11) |
| <a id="d5"></a>[D5](capability-map-detail/D.md#d5) | Network storage (NAS) is browse + bulk-media only — it may NOT host an app's data namespace | controller v0.187.0 | **PROVEN-LIVE** | — | `audits/R108-network-app-namespace-2026-07-30.md` (2026-07-30) |
| <a id="d6"></a>[D6](capability-map-detail/D.md#d6) | A Tier-1/Tier-2 app restore works from the DRIVE ALONE — the recovery unit carries the app's secrets (D5) | controller v0.188.0 | **PROVEN-LIVE** | — | `audits/D5-drive-alone-restore-2026-07-30.md` (2026-07-30) |
| <a id="d7"></a>[D7](capability-map-detail/D.md#d7) | A restore SAYS what it returned, and refuses what it cannot do — the four restore-surface truth defects from the 2026-08-21 drill | controller v0.226.0 (R-353, R-357, R-358, R-360, R-396) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: three of four live; R-357 implemented only | `audits/evidence-r353-r360-live-2026-08-30/live-validation.txt` (2026-08-30) |
| <a id="d6"></a>[D6](capability-map-detail/D.md#d6) | A Tier-1/Tier-2 app restore works from the DRIVE ALONE — the recovery unit carries the app's secrets (D5) | controller v0.188.0 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d582feb (D22, R-1178) **2026-10-25:** changed again by controller 0.343.0 (D48 pull-first, R-1270 fix A: the verdict needs a stable start, the undo copy goes after 3 more minutes); on 0.343.0 Tester 1's local unit restore and set-aside restore passed and released the copy (`audits/release-2026-10-26/t1-restores-0.343.0/`); this path itself not re-walked. | `audits/D5-drive-alone-restore-2026-07-30.md` (2026-07-30) |
| <a id="d7"></a>[D7](capability-map-detail/D.md#d7) | A restore SAYS what it returned, and refuses what it cannot do — the four restore-surface truth defects from the 2026-08-21 drill | controller v0.226.0 (R-353, R-357, R-358, R-360, R-396) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: three of four live; R-357 implemented only **2026-10-25:** controller 0.343.0 adds two truths here — a restore whose app does not start says so and keeps its copy (R-1270, D49 = A; live on 9202 rc.6), and a `.fab` import that cannot download its version refuses before it touches the app (R-1279, live). | `audits/evidence-r353-r360-live-2026-08-30/live-validation.txt` (2026-08-30) |
| <a id="d8"></a>[D8](capability-map-detail/D.md#d8) | The box PROVES its own off-site copy still HOLDS something — a backup that is intact and EMPTY is caught without a person | controller v0.231.0 (R-87) + hub v0.110.0 | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the scheduled firing is implemented only | `tests/r87-offsite-proof-2026-08-31/` (2026-08-31) |
| <a id="d9"></a>[D9](capability-map-detail/D.md#d9) | The off-site store is VERIFIED on a cadence — something checks that the customer's backups are still readable | controller v0.228.0 (R-359, R-397, R-399) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the scheduled firing is implemented only | `tests/r359-integrity-2026-08-30/` (2026-08-30) |
| <a id="d10"></a>[D10](capability-map-detail/D.md#d10) | USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | — | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15); `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13) |
| <a id="d11"></a>[D11](capability-map-detail/D.md#d11) | Decommission (migrate-first and anyway-paths), eject | agent, controller | **PROVEN-LIVE** | — | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
| <a id="d10"></a>[D10](capability-map-detail/D.md#d10) | USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — agent e15d1d3 (R-1124) | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15); `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13) |
| <a id="d11"></a>[D11](capability-map-detail/D.md#d11) | Decommission (migrate-first and anyway-paths), eject | agent, controller | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — agent e15d1d3 + 5c3b218, controller 6c6bd87 (R-1124, R-1144) | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
| <a id="d12"></a>[D12](capability-map-detail/D.md#d12) | Boot ordering: automount + networking survive reboot; appliance self-heal watchdog | agent v0.85 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13); `audits/CAMPAIGN-6A-2026-07-14.md` (2026-07-14) |
| <a id="d13"></a>[D13](capability-map-detail/D.md#d13) | Media apps see EVERY drive; a drive added, pulled, back or safely removed is handled per app (notice, real restart, wiring, torrents paused/resumed) | controller v0.313.0, catalog `0188430`, hub v0.151.0 | **PROVEN-LIVE** | Tier-0 boxes (Tester 1, demo-hp 9201); the fleet with golden 0.313.0 + the floor | `audits/media-build-2026-10-15/G/` (2026-10-15); `audits/media-drive-events-2026-10-14/` (2026-10-14) |
| <a id="d14"></a>[D14](capability-map-detail/D.md#d14) | A second local copy COUNTS only on a proven different physical disk: the agent knows each place's disks; a VM box makes the copy but says it cannot check; a same-disk copy is not made; a NAS never hides the one-copy notice; a box with no drive refuses a data-folder app in words | agent v0.157.0, controller v0.317.0, hub v0.154.0 | **PARTIAL** | live: VM `unknown` (Tester 1) and two real-disk boxes `proven` (demo-hp, N100); unit only: the same-disk skip, the NAS notice, the no-drive sentence; `usb_bridge` never seen on a device | `audits/storage-identity-buildA-2026-10-15/README.md` (2026-10-15); `architecture/16-storage-physical-identity.md` §13 |
| <a id="d15"></a>[D15](capability-map-detail/D.md#d15) | A one-disk box gets its own data area („Belső tárhely") outside every whole-box copy, registered as the default drive with no press, so media and data-folder apps install with no extra step; a free disk's partitions can be formatted or used one by one | installer 1.33.0, agent v0.158.1, controller v0.318.0, golden 0.318.0 | **PROVEN-LIVE** | fresh 512 GB one-disk VM on demo-hp; partitions on a scratch disk; the old-box signed job and the D4 sentence unit-proven only; an empty part asks for confirmation (R-1093) | `audits/storage-identity-buildB-2026-10-15/README.md` (2026-10-16) |
| <a id="d16"></a>[D16](capability-map-detail/D.md#d16) | A failed or cut restore puts the household's data back EXACTLY as it was and starts the app — local unit, second drive, kept load and off-site, and after a power cut or a killed controller (D22) | controller `main` (0.338.0-rc.1), unreleased | **IMPLEMENTED** | proven live on 9202 (rc.1): a damaged unit and a controller killed between the volume step and the replay; the second-drive pre-check (R-1178), the mirror skip (R-1180) and the off-site power cut by test only; a database outside the replaced volumes is not covered | `audits/night-2026-10-21/P3-restore/02-run.log` (2026-10-21) |
| <a id="d16"></a>[D16](capability-map-detail/D.md#d16) | A failed or cut restore puts the household's data back EXACTLY as it was and starts the app — local unit, second drive, kept load and off-site, and after a power cut or a killed controller (D22) | controller 0.338.0 (D22) → **0.343.0** (released 2026-10-25: R-1251 a copy on an unbound drive waits; fix A) | **IMPLEMENTED** | proven live on 9202 (rc.1): a damaged unit and a controller killed between the volume step and the replay; the second-drive pre-check (R-1178), the mirror skip (R-1180) and the off-site power cut by test only; a database outside the replaced volumes is not covered **2026-10-25 (re-check):** released in 0.343.0; the release run proved the success path's copy release on Tester 1, not a failure put-back on the release build — the status stays. | `audits/night-2026-10-21/P3-restore/02-run.log` (2026-10-21) |
## E. Access, networking & household use
@@ -203,7 +204,7 @@
| <a id="e14"></a>[E14](capability-map-detail/E.md#e14) | Multiple household users / per-person accounts | — | **MISSING** | — | R-15 |
| <a id="e15"></a>[E15](capability-map-detail/E.md#e15) | A second login step for the dashboard (a TOTP code or a passkey) | — | **MISSING** | — | R-811 |
| <a id="e16"></a>[E16](capability-map-detail/E.md#e16) | The household can leave Felhom, or outlive it — the box runs without the hub, the household owns its domain, tunnel and off-site account, and can export everything | — | **MISSING** | as a written answer | R-810 |
| <a id="e17"></a>[E17](capability-map-detail/E.md#e17) | The host agent cannot reach root without the operator key: exact sudo patterns, no agent-written file installed where root reads it without a content check, the agent binary only by an operator-signed update checked as root | agent v0.146.1 (R-861; delivered by a step bundle, R-880) | **PROVEN-LIVE** | both demo boxes; three named residuals | `audits/hub-safety-2026-10-05/partF/` (2026-10-05) |
| <a id="e17"></a>[E17](capability-map-detail/E.md#e17) | The host agent cannot reach root without the operator key: exact sudo patterns, no agent-written file installed where root reads it without a content check, the agent binary only by an operator-signed update checked as root | agent v0.146.1 (R-861; delivered by a step bundle, R-880) | **PROVEN-LIVE** | both demo boxes; three named residuals; **re-check (2026-10-24):** mechanism changed after its evidence — agent e15d1d3 + a0c88e5 (sudoers + priv-apply, D4 drop-in) | `audits/hub-safety-2026-10-05/partF/` (2026-10-05) |
| <a id="e18"></a>[E18](capability-map-detail/E.md#e18) | WireGuard base infra always-on; OOB operator access (felhom-sshd, /32 peer) | agent v0.72, hub v0.35 | **IMPLEMENTED** | — | `audits/SPIKE-oob-wg-operator-peer-2026-07-05.md` (2026-07-05); `audits/SPIKE-felhom-sshd-2026-07-05.md` (2026-07-05) |
| <a id="e19"></a>[E19](capability-map-detail/E.md#e19) | The operator can see WHERE a managed host is — its LAN address and its WireGuard address, on the host page | agent v0.119.0, hub v0.85.0 | **PROVEN-LIVE** | — | `audits/host-addresses-visible-2026-07-31.md` (2026-07-31) |
| <a id="e20"></a>[E20](capability-map-detail/E.md#e20) | The operator can see whether a managed host's guests still have working networking — and how often the watchdog had to repair them | agent v0.92.0 (emitter, 2026-07-21), hub v0.104.0 (reader, 2026-08-13) | **IMPLEMENTED** | not proven live | R-319 |
@@ -222,6 +223,7 @@
| <a id="f7"></a>[F7](capability-map-detail/F.md#f7) | Always-on debug rings + on-demand log-bundle pulls with TTL/custody | controller v0.116, agent v0.83, hub v0.46 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-3-2026-07-11.md` (2026-07-11) |
| <a id="f8"></a>[F8](capability-map-detail/F.md#f8) | Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | no corpus validation doc | operator infra; no evidence document |
| <a id="f9"></a>[F9](capability-map-detail/F.md#f9) | Backup-deadline alerting (`expected_backup_missed`) is ANCHORED — absence of signal is UNKNOWN, not failure | hub v0.75.0 | **IMPLEMENTED** | — | `audits/DIAG-backup-missed-2026-07-26.md` (2026-07-26); R-82 |
| <a id="f13"></a>[F13](capability-map-detail/F.md#f13) | A hacked app is noticed: an app that uploads or computes far more than its own numbers explain alarms the operator only; each box tests its upload line weekly and qBittorrent gets half of it (`08` §9; D41, D42; R-1244, R-1247) | controller v0.342.0, agent v0.164.0, hub v0.169.0, ep0 `felhom-iperf3` | **PROVEN-LIVE** | the alarm path on scratch 9202 (a hidden flood and a CPU burner, 2026-10-23); the line test on the three test boxes; the qBittorrent limit on demo-felhom. **Gaps:** a seeding qBittorrent is never tested (R-1250); an app's API can lie (R-1248) | `audits/SPIKE-upload-watch-2026-10-23.md` §7, `audits/line-test-2026-10-24/` (2026-10-24) |
| <a id="f12"></a>[F12](capability-map-detail/F.md#f12) | After a power cut, an app rebuilds the files the cut left empty, and the household sees any lost photo on the app page | controller v0.311.0, hub v0.150.0, catalog Immich block | **PROVEN-LIVE** | demo-hp 2026-10-14: a real cut mid-upload, 41 of 41 broken photos repaired, 120 controls untouched; the lost-photo note not exercised live (0 lost; unit- and 9202-tested). **Fleet delivery done 2026-10-14:** golden 0.311.0 + global floor; all three demo boxes on 0.311.0; catalog gate `unclean-boot-images` | `audits/immich-repair-2026-10-14/README.md` (2026-10-14); `09` §3 decision 236; R-1007 |
| <a id="f11"></a>[F11](capability-map-detail/F.md#f11) | The household turns the whole machine off („A gép kikapcsolása"); the hub holds every silence alarm for 14 days and shows KIKAPCSOLVA | agent v0.155.0, hub v0.149.0, controller v0.310.0 | **PROVEN-LIVE** | proven on the N100 2026-10-13; fleet delivery waits on the floor | `audits/poweroff-2026-10-13/README.md` (2026-10-13); `architecture/14-host-power-off.md`; R-1000, R-1002 |
| <a id="f10"></a>[F10](capability-map-detail/F.md#f10) | A box that is not always on (off at its backup time) makes the missed night up once when it comes back; the household sees a banner; the operator gets the missed-backup alarm, down or not | controller v0.295.0, hub v0.134.0, agent v0.145.0 | **PARTIAL** | catch-up and banner live; R-872 alarm by tests; suspend not measured | `audits/catchup-2026-10-05/partA/` (2026-10-05); R-872 |
@@ -236,7 +238,7 @@
| <a id="g4"></a>[G4](capability-map-detail/G.md#g4) | Publish train: MinAgent floors, gated auto-Reissue, version channels, floor-field-LAST rules | hub v0.45/0.53, agent | **PARTIAL** | — | `runbooks/publish-train-rules.md` (2026-09-13); R-23 |
| <a id="g5"></a>[G5](capability-map-detail/G.md#g5) | Agent self-update: A/B slots, crash-loop auto-rollback, operator-signed | agent v0.70+ | **PROVEN-LIVE** | demo | `audits/SPIKE-agent-selfupdate-2026-07-05.md` (2026-07-05) |
| <a id="g6"></a>[G6](capability-map-detail/G.md#g6) | Controller self-update: anonymous registry, no credentials in guest | controller v0.112 | **PROVEN-LIVE** | demo; the file found is the floor-driven self-update leg | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
| <a id="g7"></a>[G7](capability-map-detail/G.md#g7) | Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue | hub v0.37–0.39 | **PROVEN-LIVE** | — | `audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md` (2026-07-09); `audits/SPIKE-hetzner-api-provisioning-2026-07-09.md` (2026-07-09) |
| <a id="g7"></a>[G7](capability-map-detail/G.md#g7) | Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue | hub v0.37–0.39 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — hub 9688e76e (scoped abandonment, R-1182) | `audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md` (2026-07-09); `audits/SPIKE-hetzner-api-provisioning-2026-07-09.md` (2026-07-09) |
| <a id="g8"></a>[G8](capability-map-detail/G.md#g8) | Per-customer offsite fill + staleness + freeze lever | hub v0.41 | **IMPLEMENTED** | no live-fired leg | `audits/CAMPAIGN-offsite-overnight-2026-07-10.md` (2026-07-10) |
| <a id="g9"></a>[G9](capability-map-detail/G.md#g9) | Box-level Storage Box aggregate (total fill, Σ quotas, oversubscription alert) | hub v0.64.0 (R-5) | **IMPLEMENTED** | data pipeline live; UI render and alert mail not | hub v0.64.0 code and unit tests (outside `documentation/`) |
| <a id="g10"></a>[G10](capability-map-detail/G.md#g10) | Operator sees PBS DR datastore fill at a glance (Offsite "PBS DR" tab + dashboard gauge) | hub v0.65.0 + tenantsync v1.2.0 (R-5) | **IMPLEMENTED** | data pipeline live; UI render and alert mail not | hub v0.65.0 code and unit tests (outside `documentation/`) |
@@ -244,10 +246,10 @@
| <a id="g12"></a>[G12](capability-map-detail/G.md#g12) | Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | — | the 2026-07-13 hub closing bundle (no evidence document) |
| <a id="g13"></a>[G13](capability-map-detail/G.md#g13) | Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | — | hub v0.54 (2026-07-13; no evidence document) |
| <a id="g14"></a>[G14](capability-map-detail/G.md#g14) | The operator sees boxes left behind: per-customer controller floors with their age (and which a global raise cannot move), each box's agent against the vouched one, a 7-day "agent behind" alarm and a "floor raise skipped boxes" mail | hub v0.135.0 (R-604, R-530) | **IMPLEMENTED** | the page live; the alarm and the mail unit-proven | `audits/hub-safety-2026-10-05/partD/` (2026-10-05) |
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a nightly consistent copy, encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | — | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a consistent copy every 6 hours since D34 (nightly before 2026-10-23; corrected 2026-10-24), encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — be7e97dc + 52af38df (D34 6-hourly) | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
| <a id="g16"></a>[G16](capability-map-detail/G.md#g16) | Gitea (all code), the operator's password manager (Vaultwarden, since 2026-10-09 afternoon, R-923) and DooPlex's secrets survive the loss of DooPlex: a nightly encrypted copy on ep0; restore-tested weekly; an alarm when either stops; a failure mail | `scripts/dooplex-offsite/` (R-232), homelab-manifests rules | **PROVEN-LIVE** | — | `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
| <a id="g17"></a>[G17](capability-map-detail/G.md#g17) | The operator surface refuses a cross-site form post on BOTH login paths (session token; Basic auth + `X-Felhom-Operator`) | hub v0.135.0 (R-135) | **PROVEN-LIVE** | — | `audits/hub-safety-2026-10-05/partA/live.txt` (2026-10-05) |
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo; **re-check (2026-10-24):** mechanism changed after its evidence — agent a0c88e5 (os-apply sshd check, D4) | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
| <a id="g19"></a>[G19](capability-map-detail/G.md#g19) | An ENGLISH-SPEAKING household's first hour: download, install, pair, bind, claim, two apps | controller v0.259.0 + hub v0.119.0 + ISO 1.29.0 + the whole catalog | **PROVEN-LIVE** | walked on 0.258.0 with one blocker; fixed, the walk not repeated | `audits/DRILL-first-hour-en-0258-2026-09-20.md` (2026-09-20); `audits/i18n-closing-2026-09-21/live/` (2026-09-21) |
| <a id="g20"></a>[G20](capability-map-detail/G.md#g20) | A deletion of a customer's off-site history is NOTICED within a day | hub v0.111.0 (R-431) | **IMPLEMENTED** | not yet proven live | R-431 |
@@ -16,15 +16,17 @@
**Status:** draft (decisions from the topology/trust design sessions).
**Platform facts** referenced here live in `docs/proxmox-platform.md`; this document
**Platform facts** referenced here live in `documentation/proxmox-platform.md` (path corrected 2026-10-24); this document
records *Felhom's decisions*, not Proxmox behaviour.
---
## 1. Model at a glance
Three components. **Control is always box-initiated** — the hub never connects *into* a
customer box.
Three components. **Control is always box-initiated** — the hub never ~~connects~~ sends a command *into* a
customer box. **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** The hub may POKE a box: through ep0, one empty UDP datagram to the box's WireGuard
address (port 51822; `hub/internal/poke/client.go`, agent `internal/poke/poke.go`). A poke carries no command; it only
makes the box pull its desired state at once. The box still pulls.
```
operator side customer box (per Proxmox host)
@@ -98,14 +100,18 @@ credentials.
- CGNAT does **not** force this: the Cloudflare Tunnel already makes a box reachable through
Cloudflare's edge. We *choose* box-initiated control for the smallest attack surface — the
box exposes no control endpoint at all.
- The agent and the controller **poll** the hub; the hub never initiates inbound.
box exposes no control endpoint ~~at all~~: no route on the box takes a command from the hub. **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]**
Two listeners sit behind the WireGuard tunnel: the agent's poke listener (a poke carries no command; the box still
pulls) and the operator door, `felhom-sshd` on the host, reachable only from the operator's `/32` peer — the
documented break-glass (`03` §5, `06` §4.5).
- The agent and the controller **poll** the hub; the hub never initiates ~~inbound~~ a command inbound — a poke only
makes them poll at once (corrected 2026-10-25, D45, R-1266).
- Operator actions are delivered as **signed jobs**: the agent verifies an operator signature
before executing, so a compromised hub database alone cannot forge commands.
- **[FACT, 2026-10-16, R-970]** The last contradiction is gone: until controller v0.322.0 the box's own hub API key,
sent as a Bearer token, opened `/api/config/` (read `controller.yaml` with every secret in it, and REPLACE it,
unsigned) and the self-update and geo routes. Nothing called them (the hub never connects to a box). The config
routes are removed and no route accepts the hub key; every `/api/` route needs a dashboard session (pinned by
routes are removed and no route accepts the hub key; every `/api/` route but the read-only `/api/health` needs a dashboard session (pinned by
`TestR970_HubKeyOpensNoDoor` on the real route table). The box exposes no control endpoint, as stated above.
- All operator-initiated actions are recorded in a **customer-visible audit log**.
@@ -132,7 +138,7 @@ of all 53 apps, 2026-09-28).
**[DESIGN, 2026-10-17, R-1126, operator decision D11 = A] The tunnel runs before the claim, and FileBrowser never
answers on a default password.** The box starts cloudflared as soon as it has a tunnel token, before it is claimed,
because the claim itself is reached through it: the claim mail links `https://felhom.<domain>`
(`hub/internal/notify/templates.go:27`). Before the claim a stranger reaches only the claim page (a code: 72 h life, 5
(`hub/internal/notify/templates.go` `FormatClaimEmail`; line corrected 2026-10-24). Before the claim a stranger reaches only the claim page (a code: 72 h life, 5
tries, 15-minute lockout — controller `web/claim.go`) and `files.<domain>`. FileBrowser starts every new database on
admin/admin, and its own admin-password setting cannot replace that (it re-applies at every start and would overwrite
the household's own password — FileBrowser 1.5.6 `backend/cmd/user.go:50-57`, measured). So FileBrowser starts OFF
@@ -219,7 +225,10 @@ app. Controller down → the gated app answers an error, never the app. Measured
This removes any zero-touch remote-enrollment problem.
- A **one-time retrieval code** mints durable identity. Single-use (burned on the successful
config fetch) plus a short *pre-use* TTL; one-click regenerate for the only real failure
case (fetch fails before anything is persisted). After the fetch, the code is irrelevant —
case (fetch fails before anything is persisted). *(As built — corrected 2026-10-24: the passphrase is not burned
and has no TTL; it gives keys until the box has a host record AND a controller ≥ 0.334.0 has reported with its own
key, then `/config`, `/recovery` and `/host-enroll` answer 409 — D19, R-1159, `hub/internal/api/passphrase_scope.go:19-30`;
`05` §8 has the full rule.)* After the fetch, the code is irrelevant —
everything downstream runs on durable credentials, so retries don't need it.
- **Order:** the agent enrolls first (and, running as root at setup, mints its own scoped
operator-tier Proxmox token), then provisions the customer LXC from the golden template and
@@ -242,7 +251,8 @@ app. Controller down → the gated app answers an error, never the app. Measured
2026-10-08, R-900). Replacing it with our own relay is deferred until after the first customers (R-904, `09` §3
decision 184).
- **Outbound only** for control/report/backup (poll to hub, push to PBS). No inbound control
endpoint exists in the chosen model.
endpoint exists in the chosen model. **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** A poke carries no command; the box still pulls. The operator
door (`felhom-sshd` over the WireGuard tunnel) is the documented break-glass (§4, `03` §5).
- **Every customer has their OWN domain — never a name under `felhom.eu`** (operator ruling
2026-09-14). The free Cloudflare tier's certificate covers one level below a zone, so nested names
such as `felhom.<customer>.felhom.eu` are not covered; the customer's own domain is a few thousand
@@ -253,7 +263,7 @@ app. Controller down → the gated app answers an error, never the app. Measured
2026-09-14 first-hour drill used a `*.felhom.eu` customer domain with no tunnel, and the dashboard link
in the setup-code mail did not resolve.
- **A pasted Cloudflare API token must reach exactly the customer's own zone** (R-138 option C, `09` §3 decision 190;
hub main 2026-10-08, unreleased). On save the hub asks Cloudflare `GET /zones` with the token and keeps it only when
released in hub v0.144.0 — corrected 2026-10-24). On save the hub asks Cloudflare `GET /zones` with the token and keeps it only when
the token sees exactly ONE zone and that zone IS the customer's domain (a zone above it is refused: a sibling
customer added later would be reachable). Cloudflare unreachable → the save fails and the old token stays. The token
is never logged. **Limit:** `/zones` lists what the token can READ; a hand-built token with Zone:Read on one zone and
@@ -313,7 +323,8 @@ app. Controller down → the gated app answers an error, never the app. Measured
>
> **What being unmarked cost.** The consequence of this bullet — that 40 of 53 catalogue templates
> declare no configurable path because they are all-hot — is stated as **[FACT]** at
> `07-backup-architecture.md:296-299`. A reader met a marked observation beside an unmarked choice
> `07-backup-architecture.md:296-299`. *(Dated note 2026-10-24: that pointer is dead — those lines are about the local
> plaintext posture now, and no architecture file but this one states „40 of 53 … all-hot".)* A reader met a marked observation beside an unmarked choice
> and reasonably asked whether it *should* be so. **Between 19 and 22 August that reader called this
> decision a defect in four places** (R-370), and one session's work went into correcting the record
> rather than into the product.
@@ -326,7 +337,12 @@ app. Controller down → the gated app answers an error, never the app. Measured
- **Key custody (zero-knowledge default):** three tiers the customer chooses —
*customer-only* / *zero-knowledge escrow (default)* / *operator-managed*. Default escrows
the **PBS passphrase-protected keyfile** in the hub, wrapped under a **customer recovery
code** the operator can't open; DR needs the customer's code. Access-notification is an
code**; DR needs the customer's code. ~~the operator can't open~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** The hub keeps only the
locked key package, and every older one it retained; without the customer's code the operator cannot open any of
them, and from the off-site storage and the hub nobody can read the backup. Two exact limits, as the website says
since R-985: while the box runs, the operator has root access for support and with it could reach the data and the
backup key on the box; and the operator can open an OLDER off-site store from a retained package, by hand, when the
household types its old code on its own box (`runbooks/RUNBOOK-open-retained-package.md`, R-304). Access-notification is an
audit signal, never the primary guard. (Don't build bespoke crypto — use PBS's native
keyfile passphrase.)
@@ -343,14 +359,19 @@ app. Controller down → the gated app answers an error, never the app. Measured
restores from PBS; the controller returns on the same domain (tunnel reused from the hub
record). DR = provisioning + a restore mode, not a separate mechanism.
- **Snapshot-before-deploy:** controller asks the agent to snapshot, deploys, runs its
post-deploy health check, asks the agent to roll back on failure. (Transient snapshot, §8.)
post-deploy health check, asks the agent to roll back on failure. (Transient snapshot, §8.) **[DESIGN — not built,
noted 2026-10-24]:** the agent serves `/snapshot` and `/rollback`, but no controller code calls them
(`felhom-controller/controller/internal/agentapi`); the update's guard is the per-app backup + undo of `09` §3 decision 15.
---
## 10. How this embodies the product values
- **Zero-knowledge offsite** — the operator holds the offsite backup but cannot read it.
- **Box-initiated control + signed jobs** — no standing operator backdoor; a hub compromise
- **Zero-knowledge offsite** — the operator holds the offsite backup but cannot read it without the customer's code
(the two exact limits are in §8 — corrected 2026-10-25, D45, R-1266).
- **Box-initiated control + signed jobs** — ~~no standing operator backdoor~~ no hidden operator door: the one
operator door is `felhom-sshd` over the WireGuard tunnel, the documented break-glass (`03` §5), and the website tells
the household about the operator's root access (R-985) — corrected 2026-10-25, D45, R-1266; a hub compromise
alone can't forge commands.
- **Customer-visible audit log** — every operator action is visible to the customer.
- **Never hold data hostage** — subscriptions cover ongoing labour (monitoring, offsite,
@@ -105,7 +105,9 @@ Tri-state: `""` (unknown) · `"running"` · `"stopped"`.
### In-flight — a marker file, one per owner
Two exist and they are deliberately **separate files**: `quiesce-state.json` (the whole-guest backup
Two exist and they are deliberately **separate files** *(four in-flight records by 2026-10-24: also `update-journal.json`
— the guarded update's crash journal read by `RecoverUpdates`, `internal/stacks/update.go:49,1306` — and
`.felhom-install-pending` — an interrupted install, R-681, `install_interrupted.go:28`; noted 2026-10-24)*: `quiesce-state.json` (the whole-guest backup
window, `internal/quiesce`) and `appstop-state.json` (app-data operations that stop an app —
`backup.AppStopGuard`, covering the volume dump, offbox reconstitution and `.fab` export). **One
file, one writer**: sharing would give one record two lifetimes, and one owner clearing the other's
@@ -169,7 +171,8 @@ A recovery completing after the grace emits a `LATE RECOVERY` warning naming the
**Nothing is started without asking whether it may be.** `bootrecon.StartGate` is the one question
the sweep asks per candidate, and it is **fail-safe: cannot determine ⇒ do not start.** Three holders
answer it, and the last two only became reachable once the window widened past T+5 s:
answer it *(four since slice 4: check „1b", a guarded update moving the app — `IsUpdating`, `cmd/controller/main.go:2459-2477`;
noted 2026-10-24)*, and the last two only became reachable once the window widened past T+5 s:
| Holder | Why starting would be wrong |
|---|---|
@@ -218,6 +221,13 @@ own; every caller that is not the customer must decide for itself whether the ap
---
> *Dated note 2026-10-24 (review round 2): this inventory is v0.33's. On `main` (`321df1f`) 22 packages under
> `controller/internal/` have no row here: `aftercrash`, `agentapi`, `appbackup`, `backupwindow`, `bootrecon`,
> `bootstrap`, `channelhealth`, `crashboot`, `dockerexec`, `family`, `fillwatch`, `i18n`, `infra`, `logx`, `mailrelay`,
> `mediawire`, `nightchain`, `offsiteapply`, `privpolicy`, `quiesce`, `sockheal`, `upwatch` (an `ls` of the tree). Each
> package's own doc comment is its description; `sockheal` (re-heal after the guest's Docker socket is recreated,
> R-860) and `channelhealth` (a periodic proof of the controller→agent channel) are in no architecture document.*
## 2. Classification table (per package/file)
### `cmd/`
@@ -266,6 +276,11 @@ own; every caller that is not the customer must decide for itself whether the ap
|---|---|---|---|
| `client.go`,`zone.go`,`waf.go`,`geosync.go`,`countries.go` | **DELETE (→hub)** | The **hub** holds the CF API token and reconciles geo desired-state → WAF (doc 01 §5, doc 03 §2). The controller no longer calls the Cloudflare API — it reports geo desired-state up. The customer-facing geo *preference UI/data* stays (see `api/geo.go`). | needs-rework |
*As built (noted 2026-10-24, review round 2): the `cloudflare/` package is still wired — with `cf_api_token` set the
controller builds a `GeoSyncManager` and calls the Cloudflare API (`cmd/controller/main.go:1992-2000`; the field is still in
`config.go:160`). The DELETE classification above is the plan, not the state; whether any box sets the token is not
readable from source.*
### `config/`, `crypto/`, `util/`
| File | Class | Reason | Risk |
|---|---|---|---|
@@ -325,6 +340,7 @@ household's file names. The household's lost list is shown on the app page (`los
| File | Class | Reason | Risk |
|---|---|---|---|
| `version.go` | **KEEP** | Semver parse / version string (still used for reporting). | clean |
| *(as built 2026-10-24: both files below are LIVE — the updater decides the version and the floor and hands the container swap to the agent, `AgentSwapper`, `internal/selfupdate/updater.go:38-47`; `state.go` keeps `update-state.json`)* | | | |
| `state.go` | **DELETE (obsolete)** | Self-update audit state — the agent owns controller updates now (doc 03 §11). | clean |
| `updater.go` | **DELETE (→agent)** | Resolved (doc 03 §11): the controller is **agent-managed** — the agent snapshots → redeploys → health-gates → rolls back the controller. The controller's old self-update path (image pull + compose edit) is **removed**. | clean |
@@ -387,7 +403,8 @@ dashboard alert is an `Alert` with two placement fields: `PageOnly` (the pages i
page) and `Inline` (rendered by the page template in place, not by the layout's banner). `GetAlerts` returns
the list (endpoint-drift, agent-channel and dead-app alerts first, then the rest sorted error > warning > info,
capped at five plus an overflow line), and `layout.html` paints only those that are not `Inline` and match the
page; `GetInlineAlerts(page)` hands the dashboard and monitoring pages their inline ones.
page *(as built since R-906 — noted 2026-10-24: pages use `GetBannerAlerts(page, lang)`, which filters per page BEFORE the
cap, `internal/web/alerts.go:353`, `handlers.go:144`)*; `GetInlineAlerts(page)` hands the dashboard and monitoring pages their inline ones.
**Exactly one warning is inline today:** the „storage is not on a separate drive" health warning. It is
`PageOnly: dashboard, monitoring` and `Inline: true`, so it sits quietly under the storage bars; **every other
@@ -821,7 +838,7 @@ R-438. The three facts below are the ones a reader needs before touching any of
`Syncer.copyTemplates` (`sync/sync.go:319`) walks every directory in the catalog cache and copies
`docker-compose.yml` and `.felhom.yml` into the matching stack folder. **There is no test of whether
the app is deployed.** The only guard is a sha256 content compare (`copyIfChanged`) and the only
exclusion is `app.yaml`. Interval is `git.sync_interval`, default `15m` (`config/config.go:351`), plus
exclusion is `app.yaml`. Interval is `git.sync_interval`, default `15m` (`config/config.go:351` then, `:399` now; `copyTemplates` is now `sync.go:436`), plus
one immediate sync at controller start (`sync.go:98`).
**SINCE v0.235.0** this walk still happens and `.felhom.yml` is still copied unconditionally, but the
@@ -835,6 +852,7 @@ and they stay that way until something else acts.
### 2. Every lifecycle path resolves that disagreement, silently, by upgrading
*(This section is the 2026-09 measurement; `UpdateStack` was deleted in v0.237.0 — the button calls `StartGuardedUpdate`, `09` §6.1 — noted 2026-10-24.)*
`StartStack`, `RestartStack` and `UpdateStack` (`stacks/manager.go:1029 / 1133 / 1170`) all end in
`docker compose up -d`. `up -d` makes the container match the file, **and pulls the image itself if it
is absent** — measured at 18.3 s with a pull versus 0.5 s without, against a negative control
@@ -887,7 +905,9 @@ measured; that the restore writes to that path is read, not measured.
beállítások csak olvashatók.", `deploy.html`). This is a design, not an oversight: a changed value would
need a guarded re-deploy that nothing performs. **Finding (2026-09-15):** the catalog field flag
`locked_after_deploy` (`stacks/metadata.go`) is parsed and read by **no** controller code — every field
is read-only after install whatever the catalog says. So a page must never tell a household to change
is read-only after install whatever the catalog says. *(Dated note 2026-10-24: it is now read into the app's
`LockedFields` (`internal/stacks/deploy.go:106,534`) and enforced only by `UpdateStackConfig`, which nothing calls — so
„read-only whatever the catalog says" still holds in practice.)* So a page must never tell a household to change
a value after install. **Vaultwarden (R-512)** follows the design instead of breaking it: registration
is closed by default and the household is invited from the admin panel, measured to work without mail
(stranger 400, invite 200, invited 200).
+77 -16
View File
@@ -47,7 +47,7 @@ Owns:
1. **Proxmox lifecycle** — create/start/stop/destroy guests, snapshots, storage allocation. Via a scoped Proxmox API token (the **`FelhomAgent` operator role** — `proxmox-platform.md` §3.6, validated Phase 3 B3) for everything the API covers; raw host ops only where unavoidable.
2. **Storage management** — attach/classify targets, reconcile the storage manifest, mount USB-by-UUID, present mounts into guests.
3. **Backup/restore orchestration** — vzdump to the tiers, PBS, snapshot management, and the **self-restore-test**.
4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub. **[FACT, 2026-10-04] The "cloudflared health" leg reads a host unit that does not exist:** `internal/hub/cloudflared.go` runs `systemctl is-active cloudflared` on the HOST, but cloudflared is a container in the guest (below), so every box reports `inactive` (`Unit cloudflared.service could not be found`, demo-hp). R-841. **[FACT, FIXED agent v0.141.0 / controller v0.292.0, R-841 CLOSED]** The agent now reads the guest's `cloudflared` container through the existing `pct exec [0-9]* -- docker inspect -f *` sudoers line: state, exit code and the Docker health status of a check the controller adds (`cloudflared tunnel --metrics localhost:20241 ready` → cloudflared's own `/ready`, 200 only with a connection). Three states: `running` (healthy), `not_running` (stopped, absent, or running but NOT connected), `unknown` (could not ask, or the check is still starting) — `unknown` never alarms. No new sudoers line. **[FACT, agent v0.142.0, R-852]** The host report gains `system`: the Proxmox version and kernel from the
4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub. **[FACT, 2026-10-04] The "cloudflared health" leg reads a host unit that does not exist:** `internal/hub/cloudflared.go` runs `systemctl is-active cloudflared` on the HOST, but cloudflared is a container in the guest (below), so every box reports `inactive` (`Unit cloudflared.service could not be found`, demo-hp). R-841. **[FACT, FIXED agent v0.141.0 / controller v0.292.0, R-841 CLOSED]** The agent now reads the guest's `cloudflared` container through the existing `pct exec [0-9]* -- docker inspect -f *` sudoers line: state, exit code and the Docker health status of a check the controller adds (`cloudflared tunnel --metrics localhost:20241 ready` → cloudflared's own `/ready`, 200 only with a connection). Four states (corrected 2026-10-24): `running` (healthy), `not_running` (stopped, or running but NOT connected), `unknown` (could not ask, or the check is still starting) — `unknown` never alarms — and since agent v0.161.0 `absent` (the guest has no cloudflared container at all, R-1070; the hub never alarms on it) — `internal/hub/cloudflared.go:12-23`. No new sudoers line. **[FACT, agent v0.142.0, R-852]** The host report gains `system`: the Proxmox version and kernel from the
Proxmox API, plus the OS wrapper's read-only `facts` (host Debian, next-boot kernel, held packages, taint, the crash
guard; guest Debian, Docker engine, containerd, live-restore), at most every 10 min. **Still no new sudoers line** — the
facts, `live-restore-on` and the Docker layer all ride `FELHOM_OSAPPLY`. New ROOT-owned files instead (installer 1.30.0;
@@ -64,7 +64,8 @@ Explicitly does **not**:
- Serve application traffic or sit in the data path. **Control plane, not data plane**: if the agent dies, apps keep serving (Docker + LXC run without it); only *management* degrades — no new backups, no provisioning, hub loses the heartbeat.
- Hold or proxy customer application data.
- Run inside a guest. It is the thing that recovers guests and the host; it cannot be one of them.
- Manage **geo-restriction / the Cloudflare API**. Geo is hub-owned: the customer sets it in the controller UI, the controller reports the geo desired-state to the hub, and the **hub** (holding the CF API token) reconciles the WAF (S4). The agent manages only the *tunnel* service (`cloudflared`, §3/§5), never WAF rules.
- Manage **geo-restriction / the Cloudflare API**. Geo is hub-owned: the customer sets it in the controller UI, the controller reports the geo desired-state to the hub, and the **hub** (holding the CF API token) reconciles the WAF (S4). ~~The agent manages only the *tunnel* service (`cloudflared`, §3/§5), never WAF rules.~~ The agent manages no
Cloudflare piece: it only reads the guest's `cloudflared` container (§3) — corrected 2026-10-25, D45, R-1266.
## 3. Process model & host integration
@@ -129,13 +130,21 @@ the one trust root; the agent's API token is untouched; a box gets the new rule
`agent_config_update` (the order: signed `agent_update` first, then the bundle — after the bundle, an agent below
0.146.0 cannot update itself on that box).
**[DESIGN, ruled 2026-10-22 15:47 (D4, R-982) — built as a ready patch, NOT shipped] Root SSH from the LAN by key only; the console password stays.** The config bundle gains one fixed file, `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`), checked and reloaded by `felhom-os-apply`; the installer's uninstall removes it. Why a drop-in: Debian's `sshd_config` Includes `sshd_config.d/*` FIRST and sshd keeps the first value it reads, so the drop-in wins over the stock line; the out-of-band operator sshd (`FELHOM_SSHD`, port 8822) has its own config with no Include and is not touched. Proven on a fresh ISO 1.30.0 install: password SSH refused after, key SSH and the PVE web password login (200) and the console login still work (`audits/night-2026-10-22/P2-D4/`). It ships with the next agent release + installer tag.
**[DESIGN, ruled 2026-10-22 15:47 (D4, R-982) — SHIPPED in agent v0.163.0 (`1109092`) and installer-v1.36.0 (`manifests/webpage.yaml:347`); corrected 2026-10-24] Root SSH from the LAN by key only; the console password stays.** The config bundle gains one fixed file, `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`), checked and reloaded by `felhom-os-apply`; the installer's uninstall removes it. Why a drop-in: Debian's `sshd_config` Includes `sshd_config.d/*` FIRST and sshd keeps the first value it reads, so the drop-in wins over the stock line; the out-of-band operator sshd (`FELHOM_SSHD`, port 8822) has its own config with no Include and is not touched. Proven on a fresh ISO 1.30.0 install: password SSH refused after, key SSH and the PVE web password login (200) and the console login still work (`audits/night-2026-10-22/P2-D4/`). It ships with the next agent release + installer tag.
**[DESIGN, operator ruling D43 = A, 2026-10-24 — committed agent `c8d3c17`, unreleased; R-1257]** **Appliances only.** The
drop-in's bundle policy is `appliance`: written only when the root-owned install record (`/var/lib/felhom-install/state.json`)
says `appliance` (the installer records the mode before it installs the bundle); on a `byo` record nothing is written and
a copy WE wrote (this path, beginning with our header line) is removed, then `sshd -t` and a reload; a box with no
readable record is left as it is. The „in force" check asks `sshd -T` for each of the host's own global IPv4 addresses
(a `Match Address <LAN>` block no longer reads as in force); a failing `sshd -t` puts back ONLY the SSH file and keeps
the rest of the bundle (the report says `sshd_step_undone`).
**[FACT, agent main `a0c88e5`, unreleased] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b).
**[FACT, agent `a0c88e5`, released in v0.163.0 — corrected 2026-10-24] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b).
**[DESIGN, operator ruling D30 2026-10-22; agent main `ae69ad4`, unreleased] The tunnel heals itself by a new key.** When
the latest handshake stays stamped in the future for 15 minutes, the agent makes a new WireGuard key pair (the old
private key kept as `wg/private.key.prev`, 0600 — the escrow may hold it) and registers the new public key on the
**[DESIGN, operator ruling D30 2026-10-22; agent `ae69ad4`, released in v0.163.0 — corrected 2026-10-24] The tunnel heals itself by a new key.** When
the latest handshake stays stamped in the future for 15 minutes, the agent makes a new WireGuard key pair (the FIRST old
private key kept as `wg/private.key.prev`, later ones as `wg/private.key.prev.<unix>`, the newest three kept, 0600 — the
escrow may hold one; the once-an-hour bound survives a restart in `wg/last-rekey` — `internal/wgtunnel/rekey.go`, corrected 2026-10-24) and registers the new public key on the
existing path, `POST /hosts/<id>/wg`: the hub re-keys the peer in place (same /32, generation bump) and its full-list
push to ep0 (`wgsync`, `wg syncconf`) drops the old key and adds the new one. A new key is a new peer for ep0, with no
stored timestamp. Once an hour at most; not while the unit is down; never during a re-register already in flight. The
@@ -232,10 +241,16 @@ alerts the operator when an agent misses its expected check-in window. This is t
failure mode for a managed service, so it gets first-class treatment hub-side.
**Break-glass.** Standing inbound control is off. But when the poll loop *itself* is wedged
(agent hung, host sick) you cannot fix it through the poll loop. So there is an explicit,
**off-by-default, customer-consented, fully-audited** emergency path: SSH to the host via
(agent hung, host sick) you cannot fix it through the poll loop. So there is an explicit emergency path.
~~**off-by-default, customer-consented, fully-audited** emergency path: SSH to the host via
the Cloudflare Tunnel behind Cloudflare Access (or on-site). Enabling it is itself a signed,
logged operation; it auto-expires.
logged operation; it auto-expires.~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** The path is the operator door: the operator's WireGuard
peer reaches a dedicated `felhom-sshd` on the host (a non-22 port, reachable only from the operator's `/32` over
`wg-felhom`, key-only login; `06` §4.5, TASK H1). It does not use the Cloudflare Tunnel. It is installed by default on
an appliance install, and a BYO install refuses it (the owner's call; `scripts/felhom-host-install.sh`). It stays on;
it is not opened per incident. When SSH itself is broken, the hub-vaulted `root@pam` password opens the Proxmox web
console (`runbooks/break-glass.md`). The website tells the household that the operator has root access for support
(R-985).
## 6. Agent ↔ controller local API
@@ -250,8 +265,9 @@ The controller (in its LXC) reaches the agent (on the host) over the local bridg
- `POST /backup` — request a backup-now of *this* guest (enqueued; non-destructive).
- `GET /backup/due` — whether a policy-scheduled backup is due for *this* guest, so the controller can quiesce then call `POST /backup` (the app-consistent path, §8).
- `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI. **`host_busy`
(R-921, agent Unreleased `56aabc7`):** `GET /backup/status` also says whether the host-wide heavy-operation gate is
held (`{busy, what}` — another guest's backup, the restore test, an OS step, the stale-lock sweep, a power-off), the
(R-921, agent `56aabc7`, released in v0.161.0 — corrected 2026-10-24):** `GET /backup/status` also says whether the host-wide heavy-operation gate is
held (`{busy, what}` — another guest's backup, the restore test, an OS step, the stale-lock sweep, a power-off, and since v0.164.0 the
weekly upload line test — gate name `line-test`), the
state in which `POST /backup` answers 409; absent when the gate is not wired. The controller (≥ 0.335.0) reads it
before it stops any app. A read, never a reservation.
- **Crash-boot fact (R-856, agent v0.149.0):** `GET /host/crash-guard` — the host crash guard's last-boot record
@@ -306,6 +322,49 @@ A controller can only `POST /rollback` (or snapshot/backup) **its own** guest
token → guest and authorizes per guest, so a compromised controller's blast radius is
**self-scoped and bounded** to its own guest.
**[DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265); every route listed below since
2026-10-25]:** the surface listed above is the slice-8 design; on `main` the local API registers **41** routes
(`felhom-agent/internal/localapi/server.go:562-634`). The list above names 15 of them; the other 26 are in the table
below (recounted 2026-10-25 — the review counted 24). Every route is behind the same per-guest token (`withGuest`).
Among them the controller can start a HOST action: `POST /host/line-test/run` (about 55 s; bounded by the due
date, the window, 6 tries a day and the heavy-operation gate — R-1250, unreleased). The local API also listens on the
R-50 island bridge (host `169.254.253.1`, guest `net1` `169.254.253.2/30`, `internal/config/config.go:260-269`). The
weekly line test keeps its state in the OOB state dir (`line-test.json`) and has an off switch, `line_test.disable`.
The served WireGuard block carries `oob_peer_ip` and `oob_operator_ssh_key` (`internal/hub/report.go`). The D30 re-key
runs only inside the tunnel watchdog, and its 15-minute count lives in memory — an agent restart starts it again (the
once-an-hour bound survives in `wg/last-rekey`).
The routes the list above does not name (read from `server.go:562-634` and each handler's own comment, 2026-10-25):
| Route | What it does |
|---|---|
| `GET /backup/tiers` | this guest's backup tiers, each with its cadence and whether its Proxmox storage exists now (`present` / `absent` / `unknown`) |
| `POST /backup/target` | moves the primary backup tier to a new storage: create → grant → config; answers `restart_required`, and the caller restarts the agent after its own in-flight check |
| `GET /host/line-test` | HOST: the weekly upload line test's last measurement; `present:false` when never measured (R-1247) |
| `POST /host/line-test/run` | HOST: starts a line test now — about 55 s, bounded as above (R-1250) |
| `GET /disks/candidates` | the host disks free for Felhom to enroll (`initialize`, and `attach` for those with a mountable file system) |
| `POST /disks/same-disk` | whether two of the guest's paths sit on the same physical disk, from the agent's own read (`16` §4) |
| `POST /disks/decommission` | takes the household's own drive out for good: a permanent intent, the guest bind pruned, unmounted; never formats; a system or backup mount is refused (403) |
| `GET /disks/format/status` | the newest or running format job, for a caller whose request timed out |
| `POST /disks/guest-attach` | binds an enrolled drive's `felhom-data` folder into this guest (read-write); the rest of the drive stays hidden |
| `GET /guest/memory` | this guest's RAM and its bounds |
| `POST /guest/memory` | a bounded RAM resize, applied live (R-24) |
| `POST /netstorage/add` | mounts a NAS share on the host and starts a verify job; a failed verify rolls it back |
| `GET /netstorage` | the NAS shares |
| `GET /netstorage/verify-status` | the verify job's state |
| `POST /netstorage/remove` | removes a NAS share |
| `POST /controller/swap` | swaps the controller image (202, then async); the agent verifies and rolls back |
| `GET /controller/swap/status` | the swap's state |
| `POST /escrow/stage-secret` | stages the off-site repository password for the escrow ceremony (0600, never logged) |
| `DELETE /escrow/stage-secret` | wipes the staged password |
| `POST /escrow/recover-offsite-password` | with the recovery code in the body, returns only the off-site repository password (§8a) |
| `GET /escrow/preflight` | the ceremony's checklist |
| `POST /escrow/ceremony` | starts the root ceremony job |
| `GET /escrow/ceremony/status` | the ceremony's state |
| `POST /escrow/ceremony/claim` | the one-shot hand-off of the new recovery code; a later claim is 410 |
| `POST /escrow/ceremony/confirm` | the household's two words; only then does the package reach the hub (R-1151) |
| `GET /debug/logs` | the agent's debug log ring, for the controller's Debug page |
### 6a. Implementation (slice 8A — implemented)
**Status: implemented** (agent v0.10.0 `internal/localapi`; controller v0.35.0 `internal/bootstrap`
@@ -314,7 +373,8 @@ token → guest and authorizes per guest, so a compromised controller's blast ra
quiesce-on-due consumer is 8B), the rest wrap the existing slice-5/6/7 machinery.
- **Transport / pin.** The agent serves a **persisted self-signed leaf** bound to the host bridge IP
on a fixed port (default `:8443`). The controller pins the **leaf-cert SHA-256** (decision:
on a fixed port — `local_api.listen_addr` is REQUIRED, there is no default (`internal/config/config.go:317-319`); an
island install (R-50) listens on `169.254.253.1:8443` on its private bridge (corrected 2026-10-24). The controller pins the **leaf-cert SHA-256** (decision:
consistency with the agent's Proxmox/PBS cert pinning), carried in its bootstrap. The leaf is
generated **once and persisted**, so its fingerprint is stable across agent restarts (a fresh cert
each boot would invalidate every already-issued bootstrap pin). Defense-in-depth: the listener
@@ -420,8 +480,8 @@ per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a ba
> space on …" — so the hub raises `restore_test_failed`; it is never a pass and never dropped.
> - **Leftovers on a timer.** A failed scratch teardown and the stale-lock sweep (R-673) run every 10
> minutes, not only at agent start; the sweep holds the one-heavy-operation gate; after 3 failed
> teardown tries the operator is told. **A half-removed `vzdump` snapshot (R-1010, agent Unreleased
> `56aabc7`):** when the sweep's delete fails ONLY with „Failed to find logical volume …_vzdump" (one disk's
> teardown tries the operator is told. **A half-removed `vzdump` snapshot (R-1010, agent
> `56aabc7`, released in v0.161.0 — corrected 2026-10-24):** when the sweep's delete fails ONLY with „Failed to find logical volume …_vzdump" (one disk's
> snapshot already gone), it retries once with Proxmox's own `force=1` on the same API call (no sudoers line, no
> root command); any other error is returned unchanged, and a failed forced delete is never forced again.
> - **A thin pool ≥ 90 % requests an immediate report**; the hub judges a thin pool on the worse of data
@@ -677,7 +737,8 @@ the *archive* and the *back half*, **and in identity policy** (below).
`swap`, and the agent never set it; on the 8 GB minimum machine (guest 4096 MB) the household's first uploads filled the
guest with its swap full, the kernel could evict only program code, and the box froze (memory „full" pressure 96 %, load
162, the dashboard dead 6½ minutes). `[DESIGN]` The size step sets `swap` = the guest's memory: at bring-up, at a RAM
resize (`POST /guest/memory`), and once a minute after agent start for every pool LXC whose swap is below its RAM (live,
resize (`POST /guest/memory`), and ONCE, one minute after agent start, for every pool LXC whose swap is below its RAM (`cmd/felhom-agent/main.go`
`EnsureGuestSwap`; „once a minute" corrected 2026-10-24) (live,
no restart, never lowers). Proven on a fresh box the night of 2026-10-21 (`audits/night-2026-10-21/P1-soak/`: 512 →
4096 one minute after the fixture agent started; the 2-hour household load then answered every 10-second probe). **The
host must have swap behind it:** our installer gives an 8 GB / 256 GB machine 7.75 GB (measured the same night); a BYO
@@ -15,9 +15,9 @@
> Status: design draft (decision content), grounded on `docs/tests/phase4-signing-findings.md`.
> Status: design draft (decision content), grounded on `documentation/tests/phase4-signing-findings.md`.
> To be reviewed by Claude Code against that spike + `03` §4, then placed at
> `docs/architecture/04-control-plane-authorization.md`.
> `documentation/architecture/04-control-plane-authorization.md` — where it is now (paths corrected 2026-10-24).
>
> Builds on Part 1 (enrollment / trust), Part 3 (the agent verifies + the §4 reversibility gate).
> This doc defines the **mechanism** behind `03` §4's "an operator signature the hub can't forge."
@@ -75,7 +75,12 @@ recorded last**, only after everything else passes, so an invalid signature can
nonce (DoS-safe). Each layer is mandatory and was proven to reject independently (Phase 4 §3–4):
- **target binding** — `target.host_id`/`guest_id` must equal *this* box/guest (a signature for box
A cannot be replayed at box B);
- **time window** — `now ∈ [issued_at, expires_at]`;
- **time window** — `now ∈ [issued_at, expires_at]`; as built, `issued_at` may be up to **2 minutes** in the future
(clock skew), `expires_at` is never extended — the agent (`felhom-agent/internal/authz/verifier.go:19-22,148-155`) and
the root wrapper (`configs/felhom-os-apply`) alike (corrected 2026-10-24). **[DESIGN — the operator picked a code change
2026-10-25 (D44); row R-1280]:** neither caps the window's LENGTH (`expires_at − issued_at`); `felhom-opsign` defaults to 30 min
(`cmd/felhom-opsign/main.go:57`), and the nonce store keeps a record until its expiry. The pick: the agent and the
root wrapper refuse a window longer than 24 h. Not built yet;
- **nonce** — unseen within the window (the nonce store **must be persistent across agent restarts**
and expiry-pruned; a non-persistent store reopens the replay window after a restart).
@@ -166,7 +171,7 @@ op the agent verifies** (same pipeline, §2.3) — never unauthenticated config.
queue — then the agent polls, verifies, executes, and audits. One command + passphrase, from the
desk. **Never** a site visit.
### 6.1 Operator actions in the report reply [DESIGN, R-314/R-279 — `09` §3 decision 185, controller + hub main 2026-10-08, unreleased]
### 6.1 Operator actions in the report reply [DESIGN, R-314/R-279 — `09` §3 decision 185; released in controller v0.304.0 + hub v0.144.0 — corrected 2026-10-24]
Routine, non-destructive operator requests need no signature (`03` §4 asks for one only to destroy or overwrite the
only copy). The hub stores a row, bumps the box's intent, and the report reply carries `operator_actions:[{id,action,arg}]`
@@ -22,8 +22,9 @@
> The hub is **not** greenfield — it's a mature service (felhom-hub **v0.11.0** as of 2026-06-13, Go +
> SQLite on k3s, `hub.felhom.eu`). This doc is the **deltas** to evolve it for the Proxmox model, plus
> the new data model. Builds on Part 1 (trust/enrollment), Part 3 (the agent + reconcile), Part 4 (signing).
> NOTE: this remains a design-draft for the deltas; the live hub is v0.11.0 (the prior "v0.6.3" reference
> was stale). Ground specifics against `felhom.eu/hub/` before relying on them.
> NOTE: this remains a design-draft for the deltas. It was written against hub v0.11.0 (2026-06-13); the deployed
> version is read from `manifests/hub.yaml`, never from this line (corrected 2026-10-24, review round 2: the old
> "the live hub is v0.11.0" was a current-state version literal). Ground specifics against `felhom.eu/hub/` before relying on them.
## 1. Source-of-truth model — two drivers, two directions
@@ -91,7 +92,7 @@ Evolves the existing staleness checker (60s **cadence**, a **configured** thresh
- **Primary = host-report recency → `host_stale` / `host_down`.** The agent heartbeat is the box's
liveness signal; a silent agent = the box is gone (the critical alert). **A hub restart does not re-mail a known
silence (R-1001, hub 0.162.0 — built 2026-10-19, not yet deployed):** a host whose newest `host_stale`/`host_down`
silence (R-1001, hub 0.162.0 — deployed since; corrected 2026-10-24):** a host whose newest `host_stale`/`host_down`
event is newer than its last report is seeded with that state at start, silently; `restore_test_stale` reads its edge
from the DB on the first sweep (a signal for the tier inside the read window and newer than its last proof). A failed
lookup lets the alarm through.
@@ -139,7 +140,7 @@ itself to the floor (R-1052: not yet proven end to end). Setting the global floo
operator's password — there is no scoped route for CC (R-1051). **An agent can be vouched mid-week; the golden stays
weekly** (hub v0.158.0): the R-120 golden guard judges only a CHANGED golden selection, so a Day-0 save that keeps the
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet. **The agent list
offers real agent packages only (R-1011, hub 0.162.0, not yet deployed):** the sha lookup asks for exactly the
offers real agent packages only (R-1011, hub 0.162.0, deployed since; corrected 2026-10-24):** the sha lookup asks for exactly the
`felhom-agent` file (the golden: `golden.tar.zst`) and never falls back to another file, so a step bundle
(`0.156.1-step1`, only the config bundle) is left out of the list, and a vouch of one is refused
(`artifact_file_missing`).
@@ -295,6 +296,12 @@ What remains:
PBS repo and namespace in the **DR recipe** (`dr_recipe`, reported every cycle — R-106 fixed it to match the
backup); the endpoint's PBS fingerprint in **tenantsync** (`hub/internal/tenantsync`); the wrapped key in
**`host_escrow.blob`** (and the identity blob beside it).
**[DESIGN] 2026-10-24 (R-1254; committed felhom.eu `dcf4f84c`, hub next v0.170.0 — NOT deployed):** the escrow PUT
stores the blob, its sealed-password hash and the identity blob in ONE transaction (`Store.SaveHostEscrowWithIdentity`;
the superseded copy is made inside it). A failure leaves all of the old package or all of the new — never the new hash
beside the old identity package, which a box would read as „escrowed" while the written code opened nothing. A
malformed identity blob is refused before anything is stored. (The agent ceremony's „the previous code still works"
after a lost answer, R-1254's other half, stays open.)
Both existing infra-backup tables retire — `infra_backup_versions` (the current/live one, all readers
hit it) **and** `infra_backups` (the deprecated legacy mirror). (The planned slim DR record on `hosts` was never
@@ -326,8 +333,9 @@ customer instead of a single controller.
- **DROP `reports` + CREATE `guest_reports`** (under the clean cutover this is drop+create with no data
migration, not an in-place rename); `guest_reports` adds `guest_id`, `host_id`; `cpu/memory` mean
guest-level; `backup_last_snapshot` goes quiet.
- **ADD** desired-state JSON + `desired_generation` to `hosts`; `desired_spec_json` to `guests`; the
slim DR record (PBS namespace + repo fingerprint + wrapped escrow key) onto `hosts`.
- **ADD** desired-state JSON + `desired_generation` to `hosts`; `desired_spec_json` to `guests`. ~~The
slim DR record (PBS namespace + repo fingerprint + wrapped escrow key) onto `hosts`~~ — retired, never built (§9
above, R-105) *(corrected 2026-10-24, review round 2)*.
- **DROP both** `infra_backup_versions` (current/live) **and** `infra_backups` (legacy mirror) — the DR
recipe, tenantsync and the escrow blob replace them (R-105).
- **KEEP** `customer_configs`, `events`, `customer_notifications`, `notification_log`,
@@ -350,7 +358,9 @@ customer instead of a single controller.
The box's console tells the volunteer to open „az e-mailben kapott link". That mail must exist whenever
a customer is **waiting for a box**, without an operator press. `autoMintSelfBindLink` runs on four
events: **customer creation**; **RESET completion**; **an e-mail set or changed on a customer with no
events (a fifth path mails a link too: the household's „new link" on a USED or EXPIRED link — one mail per customer
per hour, R-719; when the customer still has a linked host it shows the „oldbox" page in the customer's language and
tells the operator instead, R-1203 — `internal/web/selfbind.go` `handleBindResend`) *(corrected 2026-10-24, review round 2)*: **customer creation**; **RESET completion**; **an e-mail set or changed on a customer with no
bound host** (config save); **a host delete** that keeps the customer. The last two re-check that no
host is bound, so a customer who has a box is never sent a pairing link. **Appliance registration is
not a trigger** — it knows no customer (`api/appliance.go`). Every send is stored as a hub-internal
@@ -388,7 +398,7 @@ it has seen any box screen, and until v0.118.0 nothing here described that.
| Mail | Trigger | Rendered by | Language source |
|---|---|---|---|
| Event notification (39 event types) | a box event, or a hub checker | `FormatCustomerEmail` | reported → created-with → `hu` |
| Event notification (44 event types — the `mail.event.*` keys of `internal/i18n/locales/hu.json`, counted 2026-10-24) | a box event, or a hub checker | `FormatCustomerEmail` | reported → created-with → `hu` |
| Claim / reset / re-enroll / claimed | the claim arc | `FormatClaimEmail` | created-with (no box has reported yet) |
| Self-bind link | customer creation, or the operator's button | `FormatSelfBindEmail` | created-with |
| The public bind PAGE at `/bind/<token>` | the customer opens the link | one template per language | created-with, **except `expired`** — see 15.4 |
@@ -456,7 +466,7 @@ The operator mail renders the counts as one `Repair:` line. The household's side
date taken) are shown on its own app page, on its box, until it dismisses them — file names never leave the box.
### 15.4b A box whose tunnel to ep0 is dead — the clock-ahead alarm [hub 0.166.0, built, NOT deployed; R-1204]
### 15.4b A box whose tunnel to ep0 is dead — the clock-ahead alarm [hub 0.166.0, deployed with 0.167.0; R-1204 — corrected 2026-10-24]
**[FACT, hub `e267c5e2`]** The host report carries the agent's `wireguard` block (`hostWireguard`). `checkWGTunnel` (`api/host_tunnel.go`, called after `checkHostSwap`) calls the tunnel stale when the box reports `handshake_future_s` > 0 (the clock-ahead lock-out) or a last handshake older than 6 h (healthy boxes measured at most 113–124 s over 24 h). It is edge-triggered against the host's latest tunnel event (`store.LatestHostTunnelEvent`): `wg_tunnel_stale` once on the way down, `wg_tunnel_ok` once on the way back; both are allow-listed and operator-only (`notify.operatorOnlyEvents`, `wg_tunnel_operator_only_test.go`) — the household cannot repair a tunnel. A box with no `wireguard` block (an agent before the field, or no tunnel) raises nothing.
@@ -555,14 +565,38 @@ record unreadable: its serve paths answer 500, a save refuses it (never blanks i
to a hub before v0.138.0** needs the columns opened first: `felhom-hub -unseal-box-secrets` (all or nothing; exit 1 =
nothing changed) — run in the running pod, then put the old image back. `guests.api_key` is an unused, always-empty
column and is not sealed. *Decided by CC unattended — operator may reverse (`09` §3 decision 132).*
**Since v0.168.0 (R-975, R-1241) the last plain tokens are sealed too:** the Cloudflare tunnel token, the Cloudflare API
token and the git token live INSIDE `customer_configs.config_json` (`infrastructure.cf_tunnel_token`,
`infrastructure.cf_api_token`, `git.token`). Each of those JSON leaves holds the same `enc:v1:` seal; the rest of the JSON
stays readable. `SaveCustomerConfig` seals, every reader opens through `openCustomerSecrets` (a leaf that does not open is
blanked and marks the record unreadable, as above). Legacy rows are sealed at start-up (`SealLegacyConfigTokens`, without
a `config_version` bump — the box's config is byte-for-byte the same, so no box re-pulls or restarts), followed ONCE by
`VACUUM` + a WAL checkpoint, because an `UPDATE` leaves the old plaintext row in a freed block of the file
(`TestR975_DBFileWithoutKeyRevealsNoToken`). `-unseal-box-secrets` opens these leaves too, so the roll-back above also
covers a return to a hub before v0.168.0. The full list of secret-bearing columns, measured on a copy of the live database
2026-10-23: `audits/hub-token-seal-2026-10-23/README.md`.
**Deployed 2026-10-24 — and what the live file still holds (operator ruling 2026-10-24, accepted).** On the live hub the
start-up `VACUUM` did not finish: the Longhorn disk writes ~1.3 MB/s (the 6-hourly `VACUUM INTO` of 530 MB takes ~7 min)
and the liveness probe stopped the first start after ~90 s; the second start found nothing left to seal (R-1249). So the
rows are sealed, but the live `hub.db` and its `-wal` still carry the old token bytes in freed space (8 of 8, measured).
**Every copy that leaves the volume is clean**: the hub's snapshot and the ep0 copy are written by `VACUUM INTO` (the ep0
copy of 06:30Z: 0 of 8), and the volume is out of DooPlex's Longhorn backup (D40). Accepted, not cleaned: whoever can read
the live volume is root on DooPlex and can read `OFFSITE_SECRET_KEY` there too; a full wipe would also need a hub stop of
8–20 minutes, a disk trim and the volume's Longhorn snapshots deleted. **Rule: a hand copy of `hub.db` for analysis is
made with `VACUUM INTO` (or taken from ep0), never `cp`/`tar` of the live file** — a `tar` of the file carries the old
bytes (`audits/line-test-2026-10-24/hub-0168-seal-check.txt`).
**And the key is the other half:** a backup of the database restores a hub that can open the sealed columns only
with the same `OFFSITE_SECRET_KEY`; today that key exists only on DooPlex (the k8s Secret, and the GPG secrets export
on the same machine). The off-site plan for the database and its key: `runbooks/RUNBOOK-hub-db-offsite-backup.md`
(R-173 — option A decided 2026-10-05, `09` decision 125; §16.3).
### 16.3 The nightly database snapshot (R-173, hub v0.136.0)
### 16.3 The database snapshot — every 6 hours since D34 (R-173, hub v0.136.0; D34 in hub 0.167.0)
Every night at **02:00 Budapest** the hub writes `<data>/snapshots/hub-<UTC stamp>.db` with SQLite's `VACUUM INTO`:
**Corrected 2026-10-24 (review round 2):** since D34 (operator yes 2026-10-23) the hub writes a snapshot at **02:00,
08:00, 14:00 and 20:00 Budapest** (`internal/dbsnap/dbsnap.go` `Slots`), catches up at start when the newest is older
than **6 h** (`CatchUpAge`), and DooPlex pushes the newest at **:30 after each slot** (`felhom-hub-db-backup.timer`),
refusing one older than **7 h** (`felhom-hub-db-backup` `MAX_AGE_H`). The paragraph below is the original nightly
design; its mechanics hold per slot. Originally: every night at **02:00 Budapest** the hub writes `<data>/snapshots/hub-<UTC stamp>.db` with SQLite's `VACUUM INTO`:
one statement, one point in time, every committed write included — the rows still in `hub.db-wal` too, which a file copy
of `hub.db` loses (measured in `internal/dbsnap` tests: a plain copy held 0 of 150 fresh rows). Written as `.tmp`, mode
`0600`, then renamed, so a reader never sees half a file. The newest **2** stay (the volume grew from 1 GiB to 2 GiB
@@ -591,7 +625,7 @@ failure never fails or delays a request: lines go through a non-blocking buffer
### 16.5 Daily jobs and the clock change [DESIGN — hub v0.163.0, operator ruling D24 2026-10-21, R-1183]
Every daily hub job (`scheduleDaily`: db-snapshot 02:00, prune, deadline-check 05:00, offsite-key-audit 07:10) runs at
Every daily hub job (`scheduleDaily`: the four db-snapshot slots 02:00/08:00/14:00/20:00, prune at its configured time, deadline-check 05:00, offsite-key-audit 07:10 — slots corrected 2026-10-24) runs at
its Budapest wall-clock time **once per calendar day**: the next run is the calendar day after (`time.Date`, day+1),
never „now + 24 h", and the slot that ran last never runs again. On a 25-hour day (the last Sunday of October) the old
arithmetic ran every job after 02:00 twice; on a 23-hour day it ran them an hour late. Pinned by `TestR1183_*` for
@@ -19,7 +19,8 @@
> decisions; grounded against felhom.eu @ `bf099f6` and felhom-agent @ `4ba1b14` (v0.63.0).
> Evidence base: `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` (all
> measurements cited below as "spike Pn"). This document ends in an implementation slice roadmap
> (§8); **no slice is implemented yet** — each is a separate future task. Consistent with and
> (§8); **no slice is implemented yet** — each is a separate future task. *(Dated note 2026-10-24, review round 2: true when
> written; S1–S4 and S5's safe halves have since shipped — §8 and the agent's `internal/wgtunnel` say what is built.)* Consistent with and
> extending Part 01 (`01-topology-and-trust.md`); where this doc names a component's duty it uses
> Part 01 §3 vocabulary.
@@ -140,7 +141,9 @@ before this ships (the drilled `--rescope-acl` posture is unaffected).
4. **Hub → endpoint VM:** adds `{pubkey, /32}` to the WG server peer list (slice-1 surface).
5. **Box (agent):** next heartbeat sees the generation advance → fetches desired-state → writes
the WG config → `install` + `systemctl enable --now wg-felhom` (the sudoers `*.mount` pattern
of `felhom-agent.sudoers:16-18`, extended to this unit — slice-3).
of `felhom-agent.sudoers:16-18`, extended to this unit — slice-3). *(As built since R-861: no `install` sudoers
line; the conf is installed 0600 to its fixed path by the root wrapper `felhom-priv-apply wg` —
`felhom-agent/configs/felhom-agent.sudoers:223-229`; corrected 2026-10-24.)*
6. **Box:** tunnel handshakes (outbound, keepalive); PBS storage is (re)pointed at
`pbs_tunnel_ip:8007` (§3.4); first backup runs.
@@ -169,7 +172,8 @@ tenant ACL (`DatastoreBackup` on `/datastore/<ds>/<ns>` to BOTH the user and the
### 3.5 Key custody + DR re-establishment
The WG private key joins the **escrowed IdentityBundle**
(`internal/escrow/identity.go:24-27`, today `{TunnelToken, PBSToken}` → `+ WGPrivateKey`) — the
(`internal/escrow/identity.go`, then `{TunnelToken, PBSToken}` → `+ WGPrivateKey`; as built the bundle carries FOUR
fields — `TunnelToken`, `PBSToken`, `WGPrivateKey`, `ResticRepoPassword` — `identity.go:30-40`, corrected 2026-10-24) — the
same R-wrapped, zero-knowledge age blob the DR recipe already consumes. On host-loss
re-provisioning (01 §9 restore mode): the box recovers the bundle with the customer's recovery
code, restores the WG key, and re-establishes the tunnel; hub-side the peer entry still exists
@@ -199,7 +203,8 @@ disables the service, KEEPS its marker, and never re-registers on its own — re
revoked until the operator re-adds the peer (the heartbeat report carries the box's pubkey as
the recovery handle; **re-add via `POST /hosts/{id}/wg` with the global key** — the raw registry
add does not bump the host's generation). The only automatic re-registration is the
pubkey-mismatch (DR/re-key) path, backoff-bounded. Absent DATA (a failed fetch) is never a
pubkey-mismatch (DR/re-key) path, backoff-bounded — and, since D30 (agent v0.163.0), a key the agent makes itself after
a handshake stamped in the future for 15 min (`internal/wgtunnel/rekey.go`; corrected 2026-10-24). Absent DATA (a failed fetch) is never a
teardown signal. **Rollout gate:** `wg_tunnel.enabled` defaults FALSE agent-side until the
production endpoint exists.
@@ -285,7 +290,9 @@ production endpoint exists.
`wg-felhom` AllowedIPs so it survives the agent's self-heal ([OF-1]); PBS unaffected.
- **4.6 Tunnel health → hub.** The tunnel is a storage dependency, so it reports like one: no handshake
within ~3 keepalive periods → the offsite target reports unreachable → the existing alerting path
carries it. No new alarm channel.
carries it. No new alarm channel. *(Since hub 0.166.0 there is one: `wg_tunnel_stale` / `wg_tunnel_ok`,
operator-only — a handshake older than 6 h or stamped in the future, D30 / R-1204, `hub/internal/api/host_tunnel.go`;
corrected 2026-10-24.)*
**EXTENDED 2026-07-05 (TASK H1).** The agent's heartbeat now also carries an **`oob` stanza**
(`felhom_sshd_active`, `felhom_sshd_port`, `reachable`, `config_invalid`, `operator_peer_configured`,
`operator_key_configured`, `wg_handshake_age_s`) — the operator's "can I get into this box right
@@ -297,6 +304,11 @@ production endpoint exists.
## 5. The endpoint VM (the one new managed-infra piece)
**AMENDED 2026-10-24 (R-1247, operator ruling):** a fourth, small thing — `felhom-iperf3.service`, an iperf3 server bound to
`10.77.0.1:5201` and opened in nftables for `wg0` from `10.77.0.0/24` only, so each box's agent can measure its upload
line once a week (felhom.eu `scripts/ep0-iperf3/`; agent `internal/linetest`). It never answers on the internet (proven
2026-10-24, `audits/line-test-2026-10-24/`).
Runs three things, nothing else: the WG server (peer list = hub-written), PBS (one datastore,
per-customer namespaces), and the minimal peer-sync surface the hub drives. **How** the hub
mutates the peer list — a tiny authenticated push API on the VM vs the VM pulling a signed peer
@@ -318,12 +330,16 @@ traverse `/root`) — create datastores under a world-traversable parent (`/srv`
## 6. Trust-model conformance (01 §3/§4 preserved)
- The box **only ever dials out** (UDP to the endpoint; the same posture as hub-poll and
PBS-push, 01 §4/§7). No inbound listener is added to any customer box; the hub still never
initiates into a box.
PBS-push, 01 §4/§7). ~~No inbound listener is added to any customer box; the hub still never
initiates into a box.~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** Two listeners sit on the box's tunnel side, reachable only through
the tunnel: the agent's poke listener (UDP 51822 on the box's WireGuard address — a poke carries no command, the box
still pulls) and, on an appliance install, the operator door `felhom-sshd` (only from the operator's `/32`, §4.5) —
the documented break-glass (`03` §5). The hub still never sends a command into a box.
- The endpoint VM cannot reach into boxes either: it terminates tunnels whose per-peer routes
(§4.5) admit only box→PBS traffic; it holds no box credentials.
- **Out of scope, explicitly:** (a) operator-inbound access to boxes — a *separate* future
problem with its own trust decision; nothing here creates or presumes it; (b) game-server
- **Out of scope, explicitly:** (a) ~~operator-inbound access to boxes — a *separate* future
problem with its own trust decision; nothing here creates or presumes it;~~ operator-inbound access to boxes was
added on 2026-07-05 (TASK H1) and rides this tunnel — §4.5 (corrected 2026-10-25, D45, R-1266); (b) game-server
ingress — public players are not tunnel peers; that is a different ingress problem (public
reachability), not a backup-transport one.
@@ -350,7 +366,8 @@ venue; it gates nothing. Runbook:
**Open sub-decisions (deferred by design):**
- Endpoint peer-sync mechanism (push-API vs signed-manifest pull) — **slice-1 design point** (§5).
- WG key rotation cadence/mechanics — the reconcile channel can carry it (new pubkey → hub →
endpoint swap); design when fleet size warrants, possibly the moment the "separable fleet
endpoint swap); design when fleet size warrants *(one automatic rotation exists since D30: after a future-stamped
handshake for 15 min, at most once an hour — `rekey.go`; a scheduled rotation is still open; noted 2026-10-24)*, possibly the moment the "separable fleet
layer" (D1) conversation reopens.
- Endpoint scaling/sharding (N customers per VM, second region, HU-hosted option) — capacity
planning, not architecture; the peer registry already keys per-host.
@@ -373,6 +390,8 @@ venue; it gates nothing. Runbook:
| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; ~~§4.3 MTU decision~~ **RESOLVED (client MTU 1280, agent v0.65.0, §4.3)**; residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially |
**MVP that puts offsite backup in a real customer's hands = S1 → S2 → S3; then S4.** S5/S6
complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — today a
read-only prober that names itself "the seam for the tunnel-management slice") adopts the
S3-piloted agent-managed-service pattern in a later, separate slice.
complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — then a
read-only prober that named itself "the seam for the tunnel-management slice") ~~adopts the
S3-piloted agent-managed-service pattern in a later, separate slice.~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** This plan is withdrawn:
cloudflared is a GUEST container that the controller renders and keeps up; the agent only reads it (four states,
`cloudflared.go:12-23`) and will not manage it — `03` §2/§3 wins.
@@ -253,7 +253,7 @@ It carries guest sizing, `pve_storage[]`, and an app half with per-app `storage_
the identity bundle's shape is `{tunnel_token, pbs_token, wg_private_key, restic_repo_password}`
(`felhom-agent/internal/escrow/identity.go:26-39`).
**[DESIGN] Off-site deletion custody (decisions 68–69, 2026-10-03) — BUILT hub v0.127.0 / controller v0.289.1, live on both demo boxes.** The restic repository password stays on the box only. The box's off-site key is append-only (pinned in `authorized_keys`, written by the hub — the key registrar); the box never receives the sub-account password, which the hub stores encrypted at rest. Old snapshots are pruned by the box itself, only in a weekly window the hub opens, behind a fake-snapshot guard (R-822); the window has run live on both demo boxes since 2026-10-05 (R-95, closed). The guard's residual — past-dated fakes steering the keeps, and a window check that trusts the box's own counts — is R-822 and R-895. **Hub v0.158.0 (built 2026-10-18, deploy waits on the operator) adds the hub's own check:** the 07:10 key audit compares the repository's `snapshots/` listing, over the hub's own login, with yesterday's — files gone that no clean-up window explains raise `offsite_snapshots_vanished` (error), files no reported box run explains raise `offsite_snapshots_unexplained` (warning); both operator-only (R-895 slice 2). A Felhom-side pruner holding repository passwords is rejected.
**[DESIGN] Off-site deletion custody (decisions 68–69, 2026-10-03) — BUILT hub v0.127.0 / controller v0.289.1, live on both demo boxes.** The restic repository password stays on the box only. The box's off-site key is append-only (pinned in `authorized_keys`, written by the hub — the key registrar); the box never receives the sub-account password, which the hub stores encrypted at rest. Old snapshots are pruned by the box itself, only in a weekly window the hub opens, behind a fake-snapshot guard (R-822); the window has run live on both demo boxes since 2026-10-05 (R-95, closed). The guard's residual — past-dated fakes steering the keeps, and a window check that trusts the box's own counts — is R-822 and R-895. **Hub v0.158.0 (built 2026-10-18; deployed the same day on the operator's yes — corrected 2026-10-24) adds the hub's own check:** the 07:10 key audit compares the repository's `snapshots/` listing, over the hub's own login, with yesterday's — files gone that no clean-up window explains raise `offsite_snapshots_vanished` (error), files no reported box run explains raise `offsite_snapshots_unexplained` (warning); both operator-only (R-895 slice 2). A Felhom-side pruner holding repository passwords is rejected.
**[DESIGN, built hub 2026-10-07 — R-32 option A, `09` §3 decision 167] What RESET does to the household's off-site
copy.** On the shared pool box a sub-account is a LOGIN, not the data: deleting it leaves its folder (measured
@@ -380,7 +380,7 @@ never moved by an update or an undo, and the unit restore accepts it.
| Tier | Location | Captures | Cadence (LIVE) | Retention (LIVE) | Encrypted |
|---|---|---|---|---|---|
| **Tier-1** recovery unit | `<nsRoot>/backups/primary/<app>/` on the app's own drive | compose + `.felhom.yml` + **secret-stripped** `app.yaml`, `db-dumps/*.sql`, `volume-dumps/*.tar`, `manifest.json` | nightly at W, plus a checksum-gated refresh on the periodic status pass | one point per app (`restore_points.go:14-18`) | no |
| **Tier-1** recovery unit | `<nsRoot>/backups/primary/<app>/` on the app's own drive | compose + `.felhom.yml` + `app.yaml` (0600 — since D5 / schema 2 it carries the PORTABLE secrets, `recovery_unit.go:25`; „secret-stripped" corrected 2026-10-24), `db-dumps/*.sql`, `volume-dumps/*.tar`, `manifest.json` | nightly at W, plus a checksum-gated refresh on the periodic status pass | one point per app (`restore_points.go:14-18`) | no |
| **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no |
| **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) |
| **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/felhom` (its `docker/` and `sys_drive/` folders, bound in the guest to `/var/lib/docker` and `/mnt/sys_drive` — §7.5.1; `mp1` no longer exists, corrected 2026-10-15) | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` → `--prune-backups keep-daily=3`: one copy per day, 3 days (keep-last until agent 0.160.0, R-1149) | **no** |
@@ -437,7 +437,7 @@ container must keep its id). Ring 1 never takes it in the night leg. Measured: t
**[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed
offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered
(`cmd/controller/main.go:604-607`). Both boxes run W = `02:30`.
(`backupwindow.LegTimes`, called at `cmd/controller/main.go:1258-1264`; line corrected 2026-10-24). Both boxes run W = `02:30`.
**[FACT, controller v0.271.0, 2026-09-25] A fourth leg: automatic app updates, CHAINED after the off-site
leg** (`09` §3 decisions 11 and 20, §6.4 part 7). The `offbox-backup` job runs the update leg when its off-site
@@ -448,8 +448,10 @@ night the whole-guest backup starts later inside its own window; it is never ski
numbers: `audits/DRILL-night-2026-09-25.md` Part D.
**[FACT, 2026-09-25] demo-hp's local whole-guest target is its host ROOT disk; retention there is now
keep-last=1** (operator ruling 2026-09-24, option A; R-684). PVE prunes AFTER a successful backup, so the target
must hold keep-last + 1 archives during the run — measured: a 9201 archive is 8.18 GB, the root disk 39 GB. The
keep-last=1** (operator ruling 2026-09-24, option A; R-684) — *since R-1149 (operator D17, 2026-10-19) the local tier's
spec is `keep-daily=N` (`felhom-agent/internal/config/config.go:553-559`), so demo-hp is keep-daily=1; corrected
2026-10-24*. PVE prunes AFTER a successful backup, so the target
must hold N + 1 archives during the run — measured: a 9201 archive is 8.18 GB, the root disk 39 GB. The
warning that should precede such a failure is R-685.
**[FACT, 2026-10-14] A power cut during the whole-guest backup leaves a half-written archive, and from agent v0.156.1 the
@@ -731,7 +733,8 @@ definition is written or a named volume replaced — a failed placement, volume
the live definition back and HOLDS the app stopped (`restore_mixed`, operator-cleared); the household reads that it
needs our help; the operator gets a `backup_run_failures` line. The hold is persisted before the stop. Placed files
ALONE (no version change, no volume replaced) still restart after a successful rollback (`TestR379_ScenarioA`): that
mix, and the undo of everything, is option A — „put back exactly as it was", the next slice on R-893. **`[FACT]`
mix, and the undo of everything, is option A — „put back exactly as it was", the next slice on R-893 *(shipped since:
D20 in controller v0.336.0 and D22 in v0.338.0, below — noted 2026-10-24)*. **`[FACT]`
2026-10-19 (measured on 9202, controller 0.335.0, `audits/night-2026-10-19/P3/`):** a forced replay failure after the
volume was replaced left the NEWEST rows (the undo dump) on top of the backup's volume — a table only the backup had
stayed (37 tables, the live app had 36); the definition did not move (same version); the app was held and a start
@@ -747,13 +750,17 @@ the snapshot added is removed, the live definition is written back and the app i
restore failed and nothing changed (with placed files: its database and volumes are back, the copied files stay); the
operator gets `backup_run_failures` leg `restore-put-back`. The hold (`restore_mixed`) remains only for a put-back that
fails — then the copy is kept for support — and the double failure (replay AND rollback) holds as before. On success the
copy is removed. **`[FACT]` 2026-10-20 (9202, controller 0.336.0-rc.1, `audits/night2-2026-10-20/P2/live/`):** the same
copy is removed — **since controller 0.343.0 (fix A, 2026-10-25) only after a STABLE start, and the verdict and the release
are two different tests (§8.2e):** the verdict „started" needs the stack running on every sample of a 45 s window, no
container restart in it and every healthcheck `healthy`; the copy is released only after a further 3 minutes of the same
(10 s samples, watched up to 30 minutes; a new restore of the app ends the watch and leaves the copy to it). A restore that
passes the verdict and then flaps keeps its copy. **`[FACT]` 2026-10-20 (9202, controller 0.336.0-rc.1, `audits/night2-2026-10-20/P2/live/`):** the same
forced failure as on 2026-10-19 → 36 tables, the backup-only table absent, schema/data/row-count md5 equal to before,
the app running, no hold, the copy (69 MB, 0.9 s) gone; a normal restore of the same snapshot then worked. The R-893
known limit above is closed for named volumes; placed FILES are still not taken back (the plain-files mix, as ruled).
**`[DESIGN]` 2026-10-21 — the put-back covers EVERY restore, and a power cut (operator ruling D22 = yes, 2026-10-21
20:05; controller `main`, unreleased).** (1) **The local unit restore** — the app's own unit, the second drive's unit,
20:05; released in controller v0.338.0 — corrected 2026-10-24).** (1) **The local unit restore** — the app's own unit, the second drive's unit,
the second drive's „Teljes visszaállítás" and a kept load all run `RestoreFromRecoveryUnitAtWith` — now takes the same
measure → copy → put-back (`offbox_restore_undo.go`, one implementation, no second copy of the logic): the copy must fit
beside the unit before anything stops; ANY failure after the stop (volume replay, definition write, DB-only start, DB
@@ -775,6 +782,28 @@ than the unit), the app running, the record reads the D22 sentence. Known limit
named volume the unit replaces (none in the catalog today — a drive-file app's unit restore is refused by R-538) is not
covered by the volume copy.
**`[DESIGN]` 2026-10-24 — a pending put-back blocks EVERY start (R-1251; review round 2 HUNT-1 H1, H2, H7; committed in
controller `bb2809d` + `062f9b4`, unreleased — rides the next controller release).** For a drive-backed app the copy lives on
the app's drive, and at start the drive is often not bound yet. A copy that cannot be reached while its drive is not
bound is **waiting, never „gone"**: the crash recovery writes a restore hold (`putback_pending`, the copy's folder
in it) and starts nothing. The hold is the one store every start path already honours — the household's
Start/Restart/Update, the boot recreate, the drive return, the media restart, the boot sweep, the night chain, the
update leg — so no path has its own check. When the drive binds, the put-back runs FIRST (the boot recreate and the
drive-return restart; the start and every drive return sweep the waiting holds), the hold is lifted, then the app
starts. The household reads „…A visszaállítás előtti adatai egy meghajtón vannak, amely most nem érhető el…" on the
app and „Ne indítsd újra a visszaállítást…" in the restore record. A put-back that FAILS (at start or when the drive
returns), or a copy gone while its drive is bound, leaves the app **held** with the D20 hold („félúton megszakadt"),
the copy kept, and the marker lets go of the app — never stopped-and-unheld (H2). Stop pressed during the restore still
gets the put-back; only the start is skipped (H7). A removal clears a waiting put-back; a later restore that completes
drops it (the household chose that data). The copy's folders are synced to disk BEFORE the marker names the copy.
**`[FACT]` 2026-10-24 night, Tester 1 (VM 341; MeTube on the data drive, 1.5 GB in its volume;
`audits/restore-holes-2026-10-24/P1-tester1/`):** a local restore cut by powering the VM off 2.1 s after „the restore
copy is recorded", the data disk taken off, the box booted: held as waiting, Start and Update refused (409, the
sentence); the disk plugged back: the agent re-mounted it, the boot recreate put the copy back BEFORE the start, the
household's newest data („gen2") and its download on the drive were there, and a later controller restart poured
nothing. Run 1 of the same proof found the journal left as `undo.json.tmp` (the rename lived only in the page cache) —
the app was held for support, never started on the cut data; the folder sync above fixed it, run 2 passed.
**[DESIGN] 2026-08-22 — the failure ladder of a database restore: replay → rollback → hold.**
Recorded here rather than only in a closed register row, because a decision that survives only inside
a closed work item is a decision nobody will find.
@@ -885,6 +914,8 @@ run makes it longer.
### 6.5 Kept data — what a removed app leaves on the drive (controller v0.274.0, `09` §3 decision 36)
*Second-drive copies that are KEPT (D25) and the rule that a rejected backup block never deletes one: §8.2a (R-1252, 2026-10-24).*
**What it is.** "Remove the app, keep my data" leaves the app's drive folder (`<drive>/appdata/<app>`, or the folder
its definition binds through `${HDD_PATH}`) in place. A reinstall over it no longer runs silently into the old files
(R-657): the install asks „A megőrzött adataimat használom" / "Use my kept data" (the database from the newest copy of
@@ -910,6 +941,26 @@ holds no database, or **does not record its data version** (§6.6 — a unit wri
a mixed or mismatched unit is refused by §6.6's own rule. Then a dated folder's files move back and the one unit restore
runs. The downloaded copy is removed on every path. A failed download or a refusal leaves the kept files as they were.
**[FACT] 2026-10-25 — why the „another drive" check let another box's copy through (R-1271; catalog sweep 2026-10-24,
Tester4: Paperless-ngx, Nextcloud, Immich).** The check (`KeptCopyAt`, `internal/backup/kept_load.go`) compares the
unit's `app.yaml` `HDD_PATH` with the drive the kept files sit on. That is a mount PATH, and every box mounts its first
data drive under the same name (`/mnt/felhom-drives/hdd_1`) — so a copy taken of another box's disk matches. It is a
path check, not a physical identity, and it stays one: loading an old box's data on a new box is what off-site is for.
What chose the wrong copy was the DATE: the rehearsal box's clock had been set to 2026-10-28, its off-site snapshot
carried that time, this box had no data time of its own to cap it (`offsiteDataTime`), and „newer than every local
copy" took it over the family's own copy of the same evening. Immich's 20 minutes of 404 were the ROUTING, not a half
load: the unit brought the old box's `SUBDOMAIN` (`photos`), so the app answered at `photos.<domain>` and the probe
asked `immich.<domain>`; its containers were healthy.
**[DESIGN] 2026-10-25 — a copy dated in the future is never „the newest" (R-1271; brief 2026-10-25 Part 4, the safest
reading; controller `bd9f4f7`, unreleased, rides v0.343.0).** A copy whose date lies more than **10 minutes** past this box's clock (`KeptClockSkew`: NTP keeps a box
within a second; ten minutes covers a box that synced late after a power cut) is future-dated. It never counts as the
newest: the newest copy with a true date wins (a tie goes local); the household's own local copy wins over any
off-site copy that is not newer by a true date. A future-dated copy is offered only when no other copy exists — then
the question says its date is in the future and does not recommend it (the „use" button is not the primary one). The
question always names the copy it will use. An off-site copy is never refused only because another box of the
household made it.
**It is NOT backed up.** No tier captures `<drive>/kept` or a leftover `appdata/<x>` of a removed app; the page says
so („Erről nem készül mentés." / "This is not backed up."). What brings it back into an app is the unit it carries
(or the app's own unit on the drive), listed per row as „Visszatölthető innen" / "Can be loaded from".
@@ -973,6 +1024,7 @@ both. Files written under different pins (a failed leg) make the unit `mixed`.
|---|---|---|
| own unit (Tier 1), second-drive mirror (Tier 2), kept-data Load | present, `compose/` names the data's pins | the data starts under that definition; if it differs from what ran, the restore pages' first sentence names the backup and the version (the kept-data Load has its own sentence and does not add this one) |
| same | present, `compose/` names OTHER pins, or `mixed` | **refused before anything is touched** (`ErrUnitVersionMismatch`): „…ezért a visszaállítás nem indult el. Az alkalmazás érintetlen" |
| off-site (Tier 3) | present, `mixed`, or a definition that does not belong to its data | **refused before anything is touched** like the rows above (`unitVersionCheck`, `offbox_reconstitute.go:826-833`; row added 2026-10-24) |
| off-site (Tier 3) | present, the snapshot's version differs from what runs | the snapshot unit's definition is written into the stack dir (and pinned) right after the stop, before any file, volume or database is touched; the database service is resolved from it |
| any | absent — a unit or snapshot written before v0.275.0, or with an unstamped file | restores **as before** (its captured definition; the off-site path the live one), with a WARN naming it |
@@ -999,13 +1051,28 @@ requires a database dump written after the conversion whose recorded engine is t
image is re-pulled on restore. A version its maker has deleted cannot start. Measured 2026-09-26: the 42 ladder
digests all resolve today (`audits/version-travel-2026-09-26/A7/`). **Accepted as a known limit — operator ruling
2026-10-12 (S8, `09` §3 decision 203; R-698 closed).** What a household meets: a restore whose version the maker has
deleted stops at the image pull, the app stays as it was, and the household uses another copy or a newer version.
deleted stops at the image pull, the app stays as it was (since D48 by ORDER — the pull comes before the stop; before it,
by the D22 put-back), and the household uses another copy or a newer version.
The operator's follow-up — *„will need to mirror the apps we support later, just to be sure"* — is **R-933**: mirror
the images of every catalog app we support into our own registry, before scale. Until it is built, this limit holds.
**Since decision 53 (`09` §3, 2026-09-30 evening):** a box keeps only the image each app service runs and the one before
it; older images of the app are deleted. A restore that needs an older version re-pulls it — as before; the limit
above is unchanged, and kept data (decision 40) is not touched by the image clean-up.
**[DESIGN] 2026-10-25 — a restore downloads first (operator ruling D48, `09` §3 decision 257; controller `2d71270`,
unreleased — rides v0.343.0).** „The app stays as it was" in the paragraph above is now true BY ORDER, not by the D22
put-back. After the existing refusals and before the first stop or change, every restore path resolves every image the
definition that will RUN names — present on the box, or pulled now — under the image-work lock held from the first
check until the restore's last start (R-863: the image clean-up cannot delete a pulled image before the start). A pull
that fails is a REFUSAL: the app is not stopped, its data is not moved, and the household reads which version, and
why: „Ennek a mentésnek a verzióját (%s) nem lehet letölteni: a készítője már nem kínálja…" when Docker says the version
does not exist (`manifest unknown`, `not found`, `pull access denied`), „…most nem lehet letölteni… Próbáld újra
később…" for everything else (a busy or unreachable registry). The operator gets one `backup_run_failures` line (leg
`restore-image-unavailable`). The paths: the local unit restore — and with it the second drive's unit restore, the
second drive's whole restore (its `CheckOnly` pass carries the pull, so its file half never runs first), a kept-data
load from a unit and from off-site, and a set-aside unit (D47) — the off-site restore, the second drive's file restore
and the no-unit fallback. Not covered: the `.fab` import — it removes the running app's volumes before its definition is pulled (R-1279).
### 6.7 Why an off-site run failed — the failure classifier (R-571)
**[FACT, read from source 2026-10-05, felhom-controller `114ff27`]** When an off-site (restic) run fails, the
@@ -1341,12 +1408,13 @@ Added 2026-08-02 from `audits/SPIKE-recovery-unit-space-2026-08-02.md` and
which is true and is the whole story for a drive-resident app. **It is not the whole story for an app
with no data drive, and that case was undocumented until now.**
**The unit is RETAINED, not staged.** `GetAppDrivePath` (`felhom-controller/internal/backup/backup.go:245-255`)
**The unit is RETAINED, not staged.** `GetAppDrivePath` (`felhom-controller/internal/backup/backup.go`, now near line 441)
returns the app's `HDD_PATH` if it has one and **`systemDataPath` otherwise** — what
`internal/appbackup/paths.go:26-27` calls *"the SSD-only system-data fallback"*. Nothing deletes a unit
after it is copied onward: the only prune is F5 (`backup.go:1053-1112`), which removes residue on an
after it is copied onward: the only prune is F5 (`pruneStalePrimaryDirs`, `backup.go`, now near line 1514), which removes residue on an
**old** drive when an app **moves**. So for every app without a data drive, `mp1` holds the kept copy
indefinitely.
indefinitely. *(Dated note 2026-10-24: `mp1` no longer exists — §6.1 / line 386; the system data now lives under the
one `mp0` volume's `sys_drive/` folder, `felhom-agent/configs/build-golden.sh`.)*
**What a unit contains, which bounds the problem** (`internal/backup/recovery_unit.go:20-25`):
`compose/` + `db-dumps/` + `volume-dumps/` (named-volume tars) + `manifest.json`. **It never contains
@@ -1495,7 +1563,10 @@ crosses the line — **R-158**.
> Eleven rows carry a blank RTO — 4, 5, 8, 9, 10, 11, 11b, 12, 13, 14, 15 — and only six of those are
> deferred work. **Row 5** has no recovery to time (a derived copy), **row 13** has no route by
> design, **row 14** is proven and merely never stopwatched, **row 11b** is a note not a row, and
> **row 15 is an open DEFECT (R-104) that this line does NOT cover.**
> **row 15 was an open DEFECT (R-104) that this line did NOT cover** — *R-104 was CLOSED 2026-10-05 (controller
v0.297.0: a lock that survives the self-heal gets the class `locked`, its own note and a retry, `offbox.go:156-162`;
CLOSED-ITEMS.md; corrected 2026-10-24).* **Row 16** (added 2026-10-25, D44 #6, R-1265 — after this banner) has a blank RTO
too: a registry rebuild from the ep0 release copy was never timed.
>
> **NO STATUS MOVED ON THE DAY THIS WAS WRITTEN, because nothing was proven that day.** A stopping
> line that promotes a row is a stopping line that lies. Full text, and the conditions that reopen
@@ -1528,13 +1599,14 @@ crosses the line — **R-158**.
| 7 | **Guest stopped and does not come back** | everything | guest-power watchdog (60 s, `onboot` as the deliberate-stop discriminator) | automatic | **120 s** | | **PROVEN** | agent v0.107.0 replay — 120 s unattended vs the incident's 587 s with a human |
| 8 | **Host dies (hardware), drives intact** | the data drives; ep0's PBS namespace; the hub's Recipe + Escrow | **install a new host, then** `--selftest=bring-up -mode dr` per guest, **then** re-attach drives by `durable_id` | **operator** (SSH) | | 7 d | **IMPLEMENTED** **`[BETA-DEFERRED]`** | bring-up code exists and has **never been executed** (`CAMPAIGN-8…:522`); the drive half of the plan is empty on every box (**R-105**) |
| 9 | **Whole box lost (fire/theft) — host and drives gone** | ep0 PBS namespace; the restic repo; the hub's Recipe + Escrow | new hardware → day-0 → escrow-consume with **R** → restore guests from PBS → app data from Tier-3 | **operator + customer** (R) | | 7 d (guest) · 24 h (app data) | **IMPLEMENTED / UNPROVEN** **`[BETA-DEFERRED]`** | every leg exists; the composed path has never been run. The destructive S5 drill is operator-gated and unrun (`06-offsite-connectivity.md:327`) **`[FACT]` 2026-10-03 (decision 70): ep0's datastore now has a nightly copy on DooPlex** (PBS pull-sync `ep0-felhom-offsite`, `remove-vanished false`, weekly verify, failures mailed to the operator); first pull 201 s, 12 GB, 4 of 4 snapshots, matching ep0 namespace for namespace. Restore route: `runbooks/ep0-datastore-copy.md` — never walked. `audits/offsite-lock-build-2026-10-03/partF/`. **`[FACT]` 2026-10-04: the restore route from the DooPlex copy was WALKED** on demo-hp (scratch VMID): listed in 2 s, restored in 186 s (15 GB logical), data read by `pct mount`; the restored config carries `onboot: 1` and the host's real drive binds — strip them on a restore beside the original (R-834). The copy now keeps 8 weekly copies (decision 71). `runbooks/ep0-datastore-copy.md` route 1. |
| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** **`[BETA-DEFERRED]`** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. **2026-09-01 (SPIKE R-95, `audits/SPIKE-r95-offsite-delete-2026-09-01.md`) — THIS ROW'S STATUS IS UNCHANGED, but the seven-day bound beside it is now known to be unverified.** Measured on BOTH boxes over their own SFTP credential, with controls: **no `.snapshots` is visible to either sub-account**, and the account is jailed. Either none exist or a sub-account cannot see them — and a snapshot the box cannot see is one the box cannot restore from, so recovery would be an operator act at the Hetzner panel, not a product capability (R-429; the confirming field needs the provider API, fenced by §11-D). **The PBS shape does NOT transfer**: the sub-account API has one permission axis, `readonly`, and a backup target cannot be read-only — PBS is a server that can refuse, a Storage Box is a filesystem that runs nothing. **What the spike removed as a fear:** withdrawing delete does NOT wedge the store (measured — restic treats a dead owner's lock as stale and proceeds). **What it added:** `unlock --remove-all` reports success while deleting nothing (R-430). **2026-09-01, LATER THE SAME DAY — THE PROBE ABOVE LOOKED FOR THE WRONG NAME AND THIS ROW'S SECOND CLAUSE IS NOW HALF WRONG.** It searched `.snapshots`; the vendor documents `/.zfs/snapshot`. Re-probed at the documented path on BOTH boxes, with controls, the two halves separate cleanly: **(a) the LIVE repository is deletable by the box — unchanged, R-95 stands;** **(b) the daily SNAPSHOTS of it are not writable by anything — PROVEN, not cited:** a write into `/.zfs/snapshot` is refused (`dest open …: Failure`) while the identical write to the account home succeeds. Seven daily snapshots confirmed in the panel (R-429). **So this row's "the restic repo is NOT protected the same way" is true of the repository and FALSE of its snapshots** — a deletion costs at most the day since the last snapshot, recoverable per-file (vendor). **Two limits kept honest:** a sub-account sees the snapshot directory EMPTY, so per-file recovery is operator-only today (R-432); and a panel-driven restore rolls back the WHOLE box. **The status is NOT moved** — evidence (b) is measured, but the recovery ROUTE has never been walked, which is what PARTIAL means. **Detection shipped hub v0.111.0 (R-431):** an unexplained fall in the snapshot count is noticed within a day — **but only a fall of MORE than half (R-435).** **2026-09-01, THE DRILL — CLAUSE (b) OF THE RE-SCOPE ABOVE IS WITHDRAWN; THE STATUS IS STILL NOT MOVED.** The re-scope said a deletion is *recoverable per-file*. **It is not, from the box: no snapshot is reachable by ANY name.** MEASURED on demo-hp, read-only, no delete verb issued: **777,600** exact names in the vendor form `YYYY-MM-DDTHH-MM-SS` (nine full days, second granularity) plus 126 alternative shapes — **zero hits**, against a control where the identical 600-name batch returns a path that exists (6/6). **Structural cause:** `/home` (`u629488-sub3`) is **st_dev 0,82**, `/.zfs/snapshot` is **st_dev 0,276**, and `/home/.zfs` does not exist — a snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding the repository. **Clause (a) — the box cannot WRITE into the snapshot area — is unchanged and re-confirmed.** So this row's *"recoverable per-file (vendor)"* and its limit *"per-file recovery is operator-only today (R-432)"* both overstate what exists: the remaining routes are a panel rollback of the WHOLE Storage Box and the provider API (fenced, and the hub's client has no snapshot method at all). **R-432 ANSWERED negatively; R-433 opened. RTO still blank — nothing was recoveredLine truncated
| 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its Longhorn volume backup | operator (`kubectl`) | | 24 h + weekly (Longhorn `RETAIN 1` each) | **UNPROVEN** **`[BETA-DEFERRED]`** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) |
| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** **`[BETA-DEFERRED]`** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. **2026-09-01 (SPIKE R-95, `audits/SPIKE-r95-offsite-delete-2026-09-01.md`) — THIS ROW'S STATUS IS UNCHANGED, but the seven-day bound beside it is now known to be unverified.** Measured on BOTH boxes over their own SFTP credential, with controls: **no `.snapshots` is visible to either sub-account**, and the account is jailed. Either none exist or a sub-account cannot see them — and a snapshot the box cannot see is one the box cannot restore from, so recovery would be an operator act at the Hetzner panel, not a product capability (R-429; the confirming field needs the provider API, fenced by §11-D). **The PBS shape does NOT transfer**: the sub-account API has one permission axis, `readonly`, and a backup target cannot be read-only — PBS is a server that can refuse, a Storage Box is a filesystem that runs nothing. **What the spike removed as a fear:** withdrawing delete does NOT wedge the store (measured — restic treats a dead owner's lock as stale and proceeds). **What it added:** `unlock --remove-all` reports success while deleting nothing (R-430). **2026-09-01, LATER THE SAME DAY — THE PROBE ABOVE LOOKED FOR THE WRONG NAME AND THIS ROW'S SECOND CLAUSE IS NOW HALF WRONG.** It searched `.snapshots`; the vendor documents `/.zfs/snapshot`. Re-probed at the documented path on BOTH boxes, with controls, the two halves separate cleanly: **(a) the LIVE repository is deletable by the box — unchanged, R-95 stands;** **(b) the daily SNAPSHOTS of it are not writable by anything — PROVEN, not cited:** a write into `/.zfs/snapshot` is refused (`dest open …: Failure`) while the identical write to the account home succeeds. Seven daily snapshots confirmed in the panel (R-429). **So this row's "the restic repo is NOT protected the same way" is true of the repository and FALSE of its snapshots** — a deletion costs at most the day since the last snapshot, recoverable per-file (vendor). **Two limits kept honest:** a sub-account sees the snapshot directory EMPTY, so per-file recovery is operator-only today (R-432); and a panel-driven restore rolls back the WHOLE box. **The status is NOT moved** — evidence (b) is measured, but the recovery ROUTE has never been walked, which is what PARTIAL means. **Detection shipped hub v0.111.0 (R-431):** an unexplained fall in the snapshot count is noticed within a day — **but only a fall of MORE than half (R-435).** **2026-09-01, THE DRILL — CLAUSE (b) OF THE RE-SCOPE ABOVE IS WITHDRAWN; THE STATUS IS STILL NOT MOVED.** The re-scope said a deletion is *recoverable per-file*. **It is not, from the box: no snapshot is reachable by ANY name.** MEASURED on demo-hp, read-only, no delete verb issued: **777,600** exact names in the vendor form `YYYY-MM-DDTHH-MM-SS` (nine full days, second granularity) plus 126 alternative shapes — **zero hits**, against a control where the identical 600-name batch returns a path that exists (6/6). **Structural cause:** `/home` (`u629488-sub3`) is **st_dev 0,82**, `/.zfs/snapshot` is **st_dev 0,276**, and `/home/.zfs` does not exist — a snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding the repository. **Clause (a) — the box cannot WRITE into the snapshot area — is unchanged and re-confirmed.** So this row's *"recoverable per-file (vendor)"* and its limit *"per-file recovery is operator-only today (R-432)"* both overstate what exists: the remaining routes are a panel rollback of the WHOLE Storage Box and the provider API (fenced, and the hub's client has no snapshot method at all). **R-432 ANSWERED negatively; R-433 opened. RTO still blank — nothing was recoveredLine truncated
| 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its encrypted database copy on ep0 (6-hourly, `scripts/hub-db-backup`; **since 2026-10-24 the hub volume is no longer in DooPlex's Longhorn backup**, D40 / R-1243) | operator (`kubectl`) | | 6 h (ep0, 14 daily + 8 weekly) | **UNPROVEN** **`[BETA-DEFERRED]`** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) |
| 11b | *consequences while the hub is gone* | — | — | — | | | **[FACT]** **`[BETA-DEFERRED]`** | **day:** nothing customer-visible breaks; events queue (`settings.go:1466-1485`). **week:** the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. **permanently:** escrow custody and break-glass credentials are gone (INV Part D2.4) |
| 12 | **Offsite provider lost (Hetzner)** | everything on-premises: both drives, both whole-guest tiers | none needed — on-premises recovery is unaffected. Re-provision a new offsite target. | operator | | | **[FACT]** **`[BETA-DEFERRED]`** | **restic and PBS share the provider** (INV Part E.1). Whether they share an account and payment method is **UNKNOWN** → §11-D **`[FACT]` 2026-10-03:** the PBS (whole-guest) off-site copy now also exists OFF Hetzner — DooPlex pulls ep0's datastore nightly (decision 70). The restic tier is still Hetzner-only. |
| 13 | **Customer loses R** | every byte, every tier, the Recipe | **on-premises recovery is unaffected** — Tier-1/2/3 and the whole-guest tiers all work while the guest and host live. What is lost: the ability to re-establish **host identity** and to open the escrowed offsite key material after a host loss | — | | | **NONE for host-loss** | R exists in **zero** system copies by design (INV Part C, row 24). Whether the operator should be able to recover is **open** → §11-A, §11-B. **D5 narrows this further:** Tier-1/2 app recovery needs only the drive, so losing R now costs the offsite route and host identity, never local app recovery (§7.4) |
| 14 | **Host SSH management plane dies** | everything | 3-layer break-glass: tmpfiles → 60 s watchdog → hub-vaulted `root@pam` on the PVE console | operator | | | **PROVEN** | `runbooks/break-glass.md`; the original incident and its fix |
| 15 | **Interrupted offsite run leaves a stale lock** | everything | manual `restic unlock --remove-all` | **operator** (SSH) | | | **DEFECT** | the self-heal exists (`offbox.go:634-648`) but the probe fails first and `classifyResticProbe` has no lock case → fail-fast, operator told *„ismeretlen okból"* → **R-104** |
| 15 | **Interrupted offsite run leaves a stale lock** | everything | manual `restic unlock --remove-all` | **operator** (SSH) | | | **FIXED 2026-10-05 (R-104, v0.297.0; was DEFECT — corrected 2026-10-24)** | the self-heal exists (`offbox.go:634-648`) but the probe fails first and `classifyResticProbe` has no lock case → fail-fast, operator told *„ismeretlen okból"* → **R-104** |
| 16 | **The container registry is lost with DooPlex** (the released images and agent packages) | the release copy on ep0 (D33): the newest 3 release tags of `felhom-controller`, `felhom-hub` and `felhom-samba`, and the newest 5 agent packages (binary + config bundle), in a plain folder outside PBS (`/srv/felhom-release-copy`) | push the copy back into a registry with the same name — `runbooks/total-loss-of-dooplex.md` „Restore the release copy” (same digests, no rebuild; `scripts/release-copy/`) | **operator** (ep0 root) | | 24 h (nightly 01:40) | **PARTIAL** | `audits/day-2026-10-23/08-release-copy-D33.txt`: the copy was pulled back, every sha256 and digest matched, and a controller image from it ran on a scratch box. Not yet walked: the push into a real new registry and the agent-package upload. Added 2026-10-25 (D44 = all picks, R-1265) |
### 8.2 The ONE exception to row 5's derived-copy rule (R-403, controller v0.230.0)
@@ -1590,6 +1662,19 @@ or pattern. **Why:** the data-loss hunt of 2026-10-20 proved two nights that del
(R-1175: one share group's reconcile removed the other groups' copies, and a dead drive's copy; R-1181: a missing or
emptied folder took its copy within one night). Controller: `internal/backup/tier2_kept.go`.
**`[DESIGN]` 2026-10-24 — a rejected backup block never deletes a copy (R-1252; review round 2 HUNT-2 H1, H2, H9;
committed in controller `518a135`, unreleased).** The night's reconcile (deleting dest dirs the legs no longer name)
runs ONLY under a valid classification. An app whose block is rejected (one bad line rejects the whole block) or absent
falls back to the legacy legs, and nothing is reconciled that night — every existing copy stays. A REJECTED block puts
„Az alkalmazás mentési beállításai most nem olvashatók…" on the app's second-drive status and tells the operator once a
day (the operator-only digest `backup_run_failures`, leg `tier2-settings-unreadable`). A kept copy (D25) is always in
the keep set, also when its leg later leaves the declared set (H2); Remove with „remove backups" drops the kept-copy
records of what it deleted (H9). **`[FACT]` 2026-10-24 night, 9202 (`audits/restore-holes-2026-10-24/P2-9202/`):**
audiobookshelf with a household file in `media/audiobooks`; two control runs kept the copy; one bogus path in the
block → „the backup block is REJECTED … every existing copy on the second drive is KEPT", the copy there, the status
sentence set, the operator digest sent (9202 has no hub: logged as dropped); the review's run on 0.342.0 the same day
deleted that copy (`review-round-2-2026-10-24/proofs/H2-bad-backup-block/`).
### 8.2b The second-drive copy runs in a memory-capped container that never outlives the copy [DESIGN + FACT, R-1214, 2026-10-22/23 nights; rides controller 0.340.0]
The copy (`rsync -a --delete`) runs in a container of the controller's own image, capped at 512 MB, so its page cache
@@ -1599,12 +1684,98 @@ refuses a missing one — then the in-process copy runs). **The container must n
fix, 2026-10-23 night): a fixed name and the label `felhom.tier2-copy=1`; on any error, the timeout included, it is
removed by name; at controller start, before the scheduler and the web server, every labelled container is removed;
and a restore is refused while any second-drive copy runs or a leftover may still run (the other direction of R-1180 —
a `--delete` beside a restore removes what the restore brings back). **`[FACT]` 9202, rc.5
a `--delete` beside a restore removes what the restore brings back). *(As built — noted 2026-10-24: the refusal is in the
web layer's `restoreOpBlocked` (`internal/web/restore_wizard.go:226`); the install-time „use my kept data" load checks
only `RestoreStatus`/`IsRunning` (`internal/api/kept_install.go:89-92`) — a register row of review round 2.)* **`[FACT]` 9202, rc.5
(`audits/night-2026-10-23/P1-straycopy/`):** a controller restart in the middle of a 16 GB copy → the new controller
removed the old container at start; a restore probe during the copy was refused; the redone copy equals the source; a
15-second test limit → the container removed, the partial copy stopped growing. Controller:
`internal/backup/tier2_copy_lifecycle.go`.
### 8.2d An earlier install's unit is set aside, never rewritten [DESIGN, operator ruling D47, 2026-10-25 08:38; `09` §3 decision 256; R-1269 — controller `02daebe`, unreleased, rides v0.343.0]
**[FACT] Measured before (controller 0.342.0, which is `e1613ae` for these files):** the catalog sweep 2026-10-24
(three times) and 9202 2026-10-25 (`audits/restore-truth-2026-10-25/before/`): Tandoor, one account, „Mentés most",
remove keeping data and backups, install again. Within five minutes the periodic refresh rewrote ONLY the unit's
`compose/app.yaml` (checksum `cbfcb538…` → `e19d2610…`) — the new install's database password — beside the OLD install's
data. Its one „same definition" test was the image pins (`recovery_unit.go:194`, the R-696 `frozen` rule), and a
reinstall of the same version has the same pins. A restore then paired the new password with the old data directory:
`password authentication failed` ×44 (app) / ×23 (postgres), the app stayed `starting`, the account was gone from
view. The next night's dump legs would have overwritten the old data as well, and the stale-drive prune deleted a unit
on another drive whatever install it belonged to.
**[DESIGN] The rule.** A recovery unit's definition and its data are written only together, by a backup of the same
install. Every install has an identity (`install_id` in its `app.yaml`): minted at a fresh install and at a `.fab`
import, carried by every load-then-save, taken FROM the unit by a restore (the app then holds that unit's data), and
minted once for an app installed before D47. The unit records it in its own `app.yaml`. Every writer of a unit asks
first — the capture and its refresh, the night database dump, the pre-backup and the update's pre-backup dumps, the
volume dump, the second drive's unit copy, the stale-drive prune. A unit of ANOTHER install (different identities; for
a unit written before D47, a carried secret that differs — a reinstall mints new ones) is **moved whole** (a rename on
its own drive; never a copy, never an edit) to `<namespace>/backups/set-aside/<app>/<UTC stamp>/` and recorded as a
kept copy (§8.2a's record: kind `unit`, reason `earlier_install`); the new install then gets its own unit from its
first backup. A move that cannot be done **refuses the write** and leaves the old unit byte for byte (an ERROR line
names the app and both identities). Nothing moves a unit while that app is being restored.
**What the household sees.** The restore page lists the set-aside unit under „Megőrzött másolatok" („…· egy korábbi
telepítés mentése") with **„Visszaállítás"** — the one unit-restore body, with the unit's own database password — and
„Eltávolítás". Only those two, and the app's remove with „delete its backups too", ever remove it (the §8.2a / R-1252 H9
rule). After a restore from it, the app is that unit's install; the app's own unit of the install it replaced is then
set aside at the next backup — never lost.
**Known limit, stated.** A box that reinstalled and refreshed BEFORE this release already holds a rewritten unit; its
old password is gone and nothing can bring it back. A unit from before D47 of an app with no carried secret has no
signal: it is treated as this install's (the old behaviour) and gets the identity at its next write.
**Why a volume app's reinstall asks no kept-data question:** the question (§6.5, decision 36) is about the app's drive
FOLDER (`${HDD_PATH}` binds under `appdata/`); a volume app keeps nothing there — „remove, keep my data" removes its named
volumes and keeps only its backups — so the install starts empty, and the kept backup is now the set-aside unit on the
restore page. (The missing offer at install is its own row, R-1278.)
### 8.2e A restore whose app does not start is never „success" [DESIGN, R-1270, 2026-10-25 — controller `b680db8` + fix A; D49 = A (operator 2026-10-25); released in controller 0.343.0]
**[FACT] Measured before:** the health check after a restore was a WARN on every path, so a Tandoor restore whose app
stayed `starting` was recorded `ok: true`, „…visszaállítva — az alkalmazás újraindult." (catalog sweep 2026-10-24; 9202
2026-10-25, `audits/restore-truth-2026-10-25/before/`). **[DESIGN]** Every restore path — the unit restore (so the
second drive's unit and whole restore, a kept load, a set-aside unit), the off-site restore, the second drive's file
restore and the no-unit fallback — returns „not started" when the app does not reach running within the health wait
(90 s). The household reads „A(z) %s: az adatokat visszatöltöttük, de az alkalmazás még nem indult el. Várj pár percet,
majd nyisd meg újra. Ha így marad, szólj nekünk." (the record `ok: false`, `not_started: true`); the operator gets one
`backup_run_failures` line (leg `restore-not-started`). **The safest reading, the operator may change it:** a timeout
does NOT pour the old data back — a slow first start (a database migration) is common, and a put-back would throw away a
correct restore. The data stays restored and the restore's undo copy is KEPT — the same „kept for support" state a failed
put-back leaves (§8 D20/D22), so no new retention rule. A watcher looks 30 more minutes: when the app runs, the record
turns to success („…elindult: a visszaállítás kész.") and the copy is removed, as after any successful restore; if it
never runs, the copy stays for support.
**D49 = A (operator ruling 2026-10-25, attended release run; `09` §3 decision 258):** the safest reading above is the rule —
the box keeps the restored data, keeps the old copy, tells the operator, and never puts the old data back by itself.
**Fix A (controller 0.343.0, R-1270 follow-up):** „started" is one helper on every path — the stack running on every
sample of a 45 s window, no container's `RestartCount` grown in it, every container with a healthcheck `healthy` (a
container without one is judged by running alone). An app that starts and then crash-loops is „not started". The undo
copy's RELEASE is a separate, longer test: 3 minutes of the same after the verdict (10 s samples, watched up to 30
minutes). **[FACT]** 9202 rc.6: an image that exits after 20 s → not started, copy kept, one operator line; the original
unit back → success, copy released 3 minutes later. Tester 1 on 0.343.0: a unit restore and a set-aside restore, each
„ran stably for 3m0s … the restore copy is released" (`audits/release-2026-10-26/`).
### 8.2c Decisions in code that no document recorded [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
Read at `file:line` on `main` (controller `321df1f`, hub `44d19042`) by the main session:
- **A second-drive copy is limited to 60 minutes**; a stray copy container that cannot be removed blocks every
restore and is retried every minute; copies into one destination run one at a time and are never killed by name
while running (`felhom-controller/controller/internal/backup/tier2_copy_lifecycle.go:40,113-121,150-172`).
- **The household's „Eltávolítás" of a kept copy (D25) is refused while any backup, restore or second-drive copy runs**
(`internal/web/kept_copies.go:54-58`).
- **A start that fails AFTER a successful put-back** leaves the app stopped and NOT held, keeps the copy, and tells the
household `restore_failed_put_back_not_started` (`internal/backup/offbox_restore_undo.go:245-247`).
- **The hub's off-site delivery checker** (`hub/internal/monitor/offsite_delivery.go`): a box that took no off-site
credential is reported `offsite_delivery_stuck` (operator) and healed by `maybeHeal` (a burned vs a stuck shape);
a box that declares it waits for the household's recovery code (`awaiting_recovery_key`) is skipped — no event, no
heal (R-1230, `:103-110`).
- **The release copy (D33)** — nightly 01:40, the newest 3 image tags and 5 agent packages to ep0 under its own
limited user, capped by the sender (`felhom.eu/scripts/release-copy/`). ~~has no row in this matrix~~ It is row
**16** of the matrix since 2026-10-25 (D44 = all picks, R-1265); its other homes are the poster facts and
`runbooks/total-loss-of-dooplex.md`.
### 8.1 The blank cells, listed explicitly
Per the rule that a blank is a finding, here they are:
@@ -1622,6 +1793,7 @@ Per the rule that a blank is a finding, here they are:
| 13 | RTO, RPO | not a timed recovery; a capability loss |
| 14 | RTO | the break-glass path is proven but was never timed |
| 15 | RTO | the manual unlock was performed but not timed |
| 16 | RTO | a registry rebuild from the ep0 release copy has never been timed (row added 2026-10-25, D44 #6, R-1265) |
**Also unmeasured, and not representable as a row** (INV Part F.3): any restore larger than 155.5 MB
from restic; any restore of a customer data drive (no tier holds one whole); the weekly PBS
@@ -1875,7 +2047,7 @@ to now *implement* D5 remains an open scheduling decision, not a blocked one.
| host-loss plan → an actual restore | **EXECUTES NOTHING BY CONSTRUCTION** | `felhom-agent/internal/dr/plan.go:1-4` |
| host rebuilt as its former self | **NEVER DONE** | INV Part D1 |
| escrow **consume** in a real recovery | **SPIKE-LEVEL ONLY** | `06-offsite-connectivity.md:327` |
| hub DB restore from its Longhorn backup | **NEVER DONE** | INV Part D2.2 |
| hub DB restore from its ep0 copy into a running hub (the weekly restore-test opens the copy and checks it; a hub has never been brought up on it) | **NEVER DONE** | INV Part D2.2; the Longhorn backup it used to cite is off since 2026-10-24 (D40) |
| a **customer** performing a restore unassisted | **MISSING AS EVIDENCE** | `00-capability-map.md:75` |
---
+74 -7
View File
@@ -135,7 +135,9 @@ live 2026-08-23 — one event across 22 scans.
**The vocabulary is the HUB's and it is exact: `{info, warning, error, critical}`.** Anything else is
**coerced to `info` at ingest**, and `info` is dropped by `severityNotifies` before *both* delivery
legs. So a severity outside the set means the event is stored, answers `200`, shows on the dashboard —
legs — except the paired recoveries in `recoveredPairedDownTypes` (`node_recovered`, `host_recovered`,
`whole_guest_backup_recovered`, `pbsdr_box_recovered`, `offsite_box_recovered`, `storage_reconnected`,
`health_recovered`), which run before the gate and are mailed (`notify/dispatcher.go` — corrected 2026-10-24, review round 2). So a severity outside the set means the event is stored, answers `200`, shows on the dashboard —
and is e-mailed to **nobody**.
**This shipped twice.** `DiskAlertKind.Severity` emitted `"warn"` until controller v0.215.0;
@@ -155,8 +157,10 @@ Three things now hold it:
3. **The dispatcher's `unrecognized severity` branch is kept**, because the hub's own monitor checkers
call `ProcessEvent` directly and never pass the ingest handler. For them it is the only guard.
**Who gets it.** `processOperator` consults only `operatorOn`, the address and a 1-hour cooldown —
**never customer preferences** — so a valid severity always reaches the operator. `processCustomer`
**Who gets it.** `processOperator` consults only `operatorOn`, the address and a cooldown —
1 hour, except 5 min for node/host liveness and 6 h for a storage-fill warning (`operatorCooldownFor`); it records a
`backup_tier_skipped` in a box's FIRST hour as „suppressed" instead of mailing it (R-723), and the mail hold applies —
**never customer preferences** (cooldowns and R-723 corrected 2026-10-24) — so a valid severity always reaches the operator. `processCustomer`
consults `operatorOnlyEvents` and then the customer's `enabled_events`.
**`app_start_failed` is customer-switchable but OFF by default** [DESIGN, operator ruling 2026-08-23]:
@@ -180,7 +184,10 @@ rather than assumed (2026-08-30):
- **not** in `operatorOnlyEvents` — the operator leg ignores customer preferences anyway, so the
operator is always mailed; putting it here would only remove the customer's ability to opt in.
- **not** in `DefaultEnabledEvents` — customer-switchable, default OFF, the same ruling as
`app_start_failed`. The checkbox already exists at `settings_notifications.html:34`.
`app_start_failed`. **Superseded by operator Q1 (2026-10-12, R-388 slice 1):** since controller v0.309.0
`backup_integrity_failed` (with `crossdrive_failed`, `storage_reconnected`, `health_recovered`) IS in
`DefaultEnabledEvents` — the default for a household with no saved list (`internal/settings/settings.go`, pinned by
`TestR388_DefaultListAsGroups`; dated note 2026-10-24, review round 2). The checkbox already exists at `settings_notifications.html:34`.
**And a caveat that belongs in the alarm ladder rather than only in the backup document:** a
`backup_integrity_ok` at the shipped depth means *the index, the pack inventory and the snapshot graph
@@ -293,6 +300,7 @@ bands.
| app down (`app_start_failed`) | **per APP** | `…:<stack_name>` | no digest exists for it — see below |
| update outcome (`app_update_undone`, `app_update_held`) | **per APP**, both legs | `…:<stack_name>` | hub v0.120.0 — one mail per app per outcome; the household leg has its own register (`perAppCustomerCooldownEvents`) |
| OOM storm (`app_oom_storm`) | **per APP**, operator | `…:<stack_name>` | hub v0.121.0 — no digest; the controller already sends it at most once per container run |
| more per-APP types (added 2026-10-24 from `dispatcher.go`'s per-stack set) | **per APP** | `…:<stack_name>` | `app_hold_no_whole_copy`, `app_stopped_unhealthy` (both legs), `app_repaired_after_crash`, `app_unexplained_upload`, `app_unexplained_cpu` |
| backup run (`backup_run_failures`) | per RUN | `…:<run_id>` | a digest already lists every failing app; one per run |
| tiered backup (`whole_guest_backup_failed`, …) | per TIER | `…:<tier>` | the tiers fail independently and mean different things |
| everything else, incl. `crossdrive_failed` and `backup_integrity_failed` | per TYPE, per hour | — | coarse **on purpose** |
@@ -367,6 +375,14 @@ one info line beside `host_crash_restart`; `operatorOnlyEvents`, pinned by
---
**More operator-only box alarms, listed 2026-10-24 (review round 2; each is in `notify/dispatcher.go`'s
`operatorOnlyEvents`, and each has its home elsewhere):** `wg_tunnel_stale` / `wg_tunnel_ok` — the box's tunnel to ep0:
a handshake older than 6 h or stamped in the future, edge-triggered (D30, R-1204; `api/host_tunnel.go`, `05` §15.4b);
`host_swap_short` / `host_swap_ok` — the host's swap below 90 % of min(the guests' swap allowance, 8192 MB) (R-1192;
`api/host_swap.go`); `bind_old_host_linked` — a household asked for a new bind link while its old host is still linked
(R-1203; `05` §13); `golden_not_vouched` — a published golden the hub does not vouch (decision 199, R-242);
`recovery_older_package` — a household's code opens, or may open, an older sealed package (decision 183, R-304 C).
## 6.4 A box that is not always on: the missed-backup deadline and the household's outage mail [DESIGN, hub v0.134.0, 2026-10-05]
Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *operator may reverse*).
@@ -376,7 +392,8 @@ Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *op
never judged (measured 2026-10-05: `1 skipped (down)`). Now a down box is judged on longer lines:
`expected_dbdump_missed` after **48 h** without a `db_dump_completed`, `expected_backup_missed` after **72 h**
without a whole-guest backup in any retained host report — never for a box first seen less than 48 h ago. A box that
died last night still raises only its staleness alarm. A `disabled` box is still skipped (R-321). Pinned by
died last night still raises only its staleness alarm. A `disabled` box is still skipped (R-321), and so is a box the household switched off on purpose — by its staleness
state or by the host row's hold (R-998, `monitor/deadline.go`; corrected 2026-10-24). Pinned by
`monitor/r872_down_box_test.go`.
- **R-873 — "your server cannot be reached" reaches the HOUSEHOLD at most once per 7 days** (`node_stale`,
`node_down`, `host_stale`, `host_down`, read from the persisted notification log, so a hub restart does not reset
@@ -387,7 +404,7 @@ Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *op
---
## 6.5 One lost off-site copy on a pinned tier [DESIGN, R-435 — `09` §3 decision 191, hub main 2026-10-08, unreleased]
## 6.5 One lost off-site copy on a pinned tier [DESIGN, R-435 — `09` §3 decision 191; released in hub v0.144.0 — corrected 2026-10-24]
On a **pinned** off-site tier (the hub holds a confirmed append-only key, decision 69) the only legitimate way the
snapshot count can fall is a clean-up window the hub opened (decision 68). So the hub raises
@@ -470,7 +487,7 @@ was ruled that way on its own merits and does not pre-judge the redesign.
**Filed as a PRODUCT DECISION, not a defect** — see the register. It is the operator's call to take
separately, and no part of it was implemented here.
### 8.1 The design — „only what you can act on" [DESIGN, R-388 — `09` §3 decision 200, 2026-10-12; NOT BUILT]
### 8.1 The design — „only what you can act on" [DESIGN, R-388 — `09` §3 decision 200, 2026-10-12; slice 1 BUILT in controller v0.309.0 — three group switches, details under Részletek (`internal/settings/notif_groups.go`); corrected 2026-10-24]
*Operator ruling 2026-10-12 (S5): design now, build in slices later. Written by CC from the direction above and the
live page (controller, `settings_notifications.html`: **18 event switches** plus the address, the cooldown and the
@@ -531,3 +548,53 @@ list is shown as custom and saved back unchanged; the default list maps to group
multi-day backup fault reads as „nobody noticed" if the household ever finds out.
2. **The cooldown field („hány óránként legfeljebb"):** keep it on the page or move it under „Részletek"? Pick:
**under Részletek** — it is an engineering choice, which is what this redesign removes.
## 9. The upload and CPU watch — „does the app explain its own traffic?" [DESIGN — built; released in controller v0.342.0 and hub v0.169.0 (deployed) on the operator's yes, 2026-10-24 — corrected 2026-10-24, review round 2]
**Why** (operator ruling 2026-10-23 19:22, `CONTEXT.md`): DooPlex's hacked Plex sent ~474 GB in a day with no viewer and
used ~10 cores with no film being converted (`audits/plex2shell-felhom-impact-2026-10-23/`). Nothing on a Felhom box
would have noticed: the health check sees whole-box CPU at one instant, and nothing looks at upload at all. **The idea:**
when an app sends a lot (or burns CPU) for a long time, ask the APP whether that is its work. Work the app explains is
normal use; work it does not explain is the sign of a hacked app. Numbers and measurements:
`audits/SPIKE-upload-watch-2026-10-23.md`; the code: controller `internal/upwatch` (README §8e).
1. **The line.** Learnt from what the box already does: while its off-site push runs, the controller samples its own
sent bytes once a second; the best ≥ 10-second rate is kept per night (14 nights, settings.json `upwatch.line`); a
night counts when that window moved ≥ 10 MB; the line is the highest counting night. A **lower bound**, shown so
(„legalább N Mbit/s", system settings). **AMENDED 2026-10-24 (operator ruling 07:26, R-1247): the weekly line TEST.**
The night's one-second read showed how low that floor is (demo-hp „at least 10 Mbit/s"; spike §2.1). So the host's
agent (agent ≥ 0.164.0, `internal/linetest`) runs a 10-second iperf3 upload once a week to ep0's tunnel-only server:
Tuesday 10:00–15:59 local from a per-host slot, only after 10 quiet minutes of the physical uplink (< 2 Mbit/s each
minute), under the one-heavy-op gate, best of 3, never in the night chain; „server busy" → later the same day. The
controller reads it (`GET /host/line-test`) every 15 minutes; the watch judges against the higher of test and floor;
the system page says „N Mbit/s (heti sebességméréssel)". demo-hp measured 90.7 Mbit/s (`audits/line-test-2026-10-24/`).
2. **Trigger, per app** (its compose project's containers summed; exact counters from the Docker Engine API, once a
minute): upload ≥ max(5 Mbit/s, 25 % of the line) for 15 minutes; CPU ≥ max(150 %, 50 % × cores) of one core for 30
minutes. Infrastructure (the controller — its own push is the line measure —, traefik, cloudflared, Samba) is never
judged; the box's own work on an app (a backup run, an update, a deploy) is never judged.
3. **The explanation.** qBittorrent (`up_info_data`) and Transmission (`uploadedBytes`): explained when the container
sent ≤ 1.25 × what the app counted + 1 Mbit/s. Plex: any viewer explains upload; any transcode explains CPU. An app
that does not answer gives no number (a hacked app may refuse its own API). No explainer: upload alarms only at ≥ 10 ×
its own p95 for 60 minutes after ~20 h of history; CPU at ≥ max(200 %, 75 % × cores) for 60 minutes.
4. **Who is told** (the R-388 rule, §8.1): **`app_unexplained_upload` / `app_unexplained_cpu`** (warning) and their
`_cleared` (info, recorded) are **operator-only** (hub `operatorOnlyEvents`), one per episode, per-app cooldown. The
household is told only about the qBittorrent limit, because they can change it (R-1246 asks the operator).
5. **qBittorrent's default upload limit:** 50 % of the TESTED line (D41 = 50 %, operator 2026-10-24; never of the
push floor), every 15 minutes, re-set when the line moves > 20 %; never over a limit the household set (one there
first, or any change from the value qBittorrent stored for the box — it keeps KiB/s). No test → none set. **A box
that seeds all day (R-1250, brief 2026-10-24; committed agent `29bcf44`+`1e087bb`, controller `046b808`, not yet
released):** the 10 quiet minutes never come, so when the agent reports `waiting_for_quiet` (due, in the window,
tunnel up, the full 10-minute window watched and busy) and qBittorrent or Transmission runs, the controller writes
`upwatch.line_test_hold` (the app's limit exactly as stored) FIRST, holds the app at 1 KiB/s (Transmission 1 kB/s),
calls `POST /host/line-test/run` (5 s settle, 10 s quiet check < 2 Mbit/s, three runs 5 s apart — about 55 s), then
puts the old limit back, the household's own included; a change made during the test stays; a killed controller's
start-up check puts it back; the 50 % rule waits while a record exists. Measured on demo-hp with a real seed (10
Mbit/s to DooPlex's qBittorrent): refused `not-quiet` without the hold; with it the uplink fell to 0.02–0.16 Mbit/s,
94.2 Mbit/s measured, the app held 56.8 s and back to 1 249 280 B/s exactly (`audits/dooplex-alarms-2026-10-24/`).
While the box owns it, qBittorrent's page says: „A qBittorrent
feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod."
6. **Limits, stated:** *(added 2026-10-24, review round 2: ANY viewer explains any amount of upload, so a hacked app can send freely while one real playback runs — `upwatch/watch.go:275,299`; R-1262)*; an attacker who knows the check can make the app's API lie — a per-container ceiling (`tc` from
a helper in the app's network namespace, measured to work in an unprivileged guest; spike §5) would cap the flood
whatever the API says; not built. The box cannot tell LAN from internet bytes per app; a host-network container
(Samba) has no counter; the push floor is a lower bound, the weekly test is close to the line.
7. **Release:** released 2026-10-24 on the operator's yes (controller v0.342.0, hub v0.169.0; R-1244) — corrected 2026-10-24.
@@ -65,6 +65,13 @@ build is added to both in the same commit.
### 1.1 The button
> *Dated note 2026-10-24 (review round 2): §1 is the 2026-09 MEASUREMENT before the arc. `UpdateStack` no longer exists
> (deleted in v0.237.0, controller `REUSE.md`); the button calls `StartGuardedUpdate` — backup precondition, safety
> dump, pin journal, undo copy, health wait, undo (`internal/api/router.go:794`; `internal/stacks/update.go`). The line
> numbers below are the measurement's; e.g. `copyTemplates` is now `sync.go:436`, `copyIfChanged` `:700`, the `15m`
> default `config.go:399`, `RestartStack` `manager.go:1454` (it now runs `up -d --force-recreate` and errors when no
> container restarted, R-1026).*
`Manager.UpdateStack` (`felhom-controller/controller/internal/stacks/manager.go:1199`) is two compose
commands and nothing else:
@@ -267,7 +274,8 @@ builds them (`audits/update-rulings-2026-09-23/`).
next night. **Why:** the update then leans on the freshest copy the box ever has, and there is no
second window for anyone to set or misread. **Replaces:** §3b Q1's recommendation of a fixed
02:30–05:00. *For the builder:* `stacks.update_window` (`config.go` L195, default `03:00-05:00`)
exists and **no Go code reads it** — it appears only in `config.go` (field + default),
exists and **no Go code reads it** *(dated note 2026-10-24: the field is now removed — no controller source names
it; `controller/README.md` „`stacks.update_window` is removed")* — it appears only in `config.go` (field + default),
`setup/handlers.go` L482 (written into a new box's `controller.yaml`) and
`configs/controller.yaml.example`. Remove it or fold it into the chain; do not build a second
window.
@@ -278,17 +286,27 @@ builds them (`audits/update-rulings-2026-09-23/`).
13. **The test decides, not the tag** (Q3). **REPLACES decision 3's second half, "never across a
major".** The box applies by itself every step the catalog holds, because **the catalog holds only
tested steps**; the size of the version jump does not matter, the test does. The box does not
parse tags to decide. The tag rule (`stacks.CompareImageRefs`) moves to the **catalog gate** as a
push-time safety net: an image move with no test record is refused there. Two exceptions, both
parse tags to decide. ~~The tag rule (`stacks.CompareImageRefs`) moves to the **catalog gate** as a
push-time safety net:~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** The tag rule (`stacks.CompareImageRefs`) did not move: it
stays in the controller (`internal/stacks/updateorder.go`), where it only tells whether an installed app is
AHEAD of the catalog; it never decides an update. The catalog gate is the push-time safety net, and it asks for a
proven test instead (§6.4 part 4): an image move with no test record is refused there. Two exceptions, both
**marks the test sets on the step**: *files may change* → automatic only when a fresh copy holds
the files, else a person (this is Q2's refinement); *needs a person* → the tester writes why.
**Why:** decision 3 was a proxy. "Within a major" was standing in for "known to work", and the
update night measured the proxy wrong in both directions — minor moves that held
(adventurelog, outline) and a tag shape it cannot read at all (§3b Q3). The test record is the
thing the proxy was guessing at.
**Dated note 2026-10-23 (decision 255, D39 = B):** a second step kind now exists — the `security_fix` step, proven
on the box with the bench optional, for an urgent upstream fix with a public notice. Decision 13's rule is
unchanged for every other move: a step a box takes was still tested, on at least the box venue.
14. **The update ladder** (Q3). A box more than one step behind **climbs one tested step at a time, in
order, and never jumps**. Each step is the full guarded update of §6.1. **Why:** a step was tested
order, and ~~never jumps~~ the night leg never jumps**. Each step is the full guarded update of §6.1.
**[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** An app older than the ladder (its installed version matches no entry's `from`) is never
pressed by the night leg (`internal/stacks/unattended.go`). A PERSON's press on such an app takes the catalog's
current definition, logged by name (`internal/stacks/ladder.go`, `TestLadder_UnknownInstalledJumpsAndSaysSo`); the
guarded update and its undo (decision 15) still run around it. **Why:** a step was tested
from the version before it, not from three versions before it; R-40's multi-major jump is what
this prevents. This is the "stepping" §6.1 already assigned to slice 6.
@@ -421,7 +439,8 @@ R-636's louder repeated alarm.
(a) broke the second. Shipped as controller **v0.269.1** — a second release in the session, instead of
shipping (a) to the fleet with the floor.
*Note 2026-09-30 (R-740) — the cost line above is not what the code does.* „(or the automatic leg)": the
automatic leg never presses a digest-only change (`unattended.go:433–435`, skip `no_test_record`). And the
automatic leg never presses a digest-only change (`unattended.go:433–435` then, `:449-451` now — line corrected
2026-10-24 — skip `no_test_record`). And the
catalog never writes a re-test of a floating tag at a new digest (the writer refuses a step that moves no
image), so a same-name upstream fix reaches no box by either route today. Measured, with the options, in
R-740; the decision is the operator's (STATUS 2026-09-30). This decision's ruling — the sync never moves
@@ -976,8 +995,11 @@ its length, and both fixes cost something the household would notice — operato
installs the vouched golden and updates itself to the floor (not yet proven end to end — R-1052). **The golden
stays weekly:** a CHANGED golden selection is still refused when it is older than the fleet's newest controller
(`TestVouchGoldenChangedToOlderIsStillRefused`).
5. The push gate judges the golden by its AGE (older than 7 days while the floor is above it), not by the newest
CHANGELOG heading (`scripts/golden_currency_gate.py`).
5. The push gate judges the golden by its AGE, not by the newest CHANGELOG heading
(`scripts/golden_currency_gate.py`). **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** ~~(older than 7 days while the floor is above
it)~~ The gate runs offline and cannot read the floor, so „a release is ahead of the newest bake” stands in for
„the floor is above it”: it convicts when a release is ahead AND the newest bake is more than 7 days old; ahead
and at most 7 days old is an advisory. The floor follows a release within hours, so the gate is the stricter one.
251. **Scripts use scoped hub tokens, not the operator password; CC's floor step uses its own.** *Operator ruling
2026-10-15 11:38.* Tokens with groups (read/write per group), managed on the Configuration tab, shown once, stored
hashed, every use logged by name (`05` §17, hub v0.152.0). Never by any token: a secret shown or minted, a delete or
@@ -1031,6 +1053,87 @@ its length, and both fixes cost something the household would notice — operato
Built in controller v0.314.0 (`internal/stacks/restore_gate.go`); pinned by `restore_gate_test.go` and
`r1057_unit_gate_test.go` (ten mutations each seen failing). Proof: `audits/gate-restore-2026-10-15/9202/`.
### 2026-10-23 (~12:00) — D39 = B: the security-fix step (operator ruling by starting the run; R-1239)
255. **An urgent upstream security fix reaches the boxes at night as a `security_fix` ladder step: the box walk is
required, the bench is optional, and a public notice is the authority.** *Operator ruling 2026-10-23 ~12:00 (D39 =
B, R-1239), taken by starting the D39 brief; its design is Claude.ai's recommendation, taken with it.* Why: on
2026-10-23 the only route for the Plex2Shell fix was an operator word, `git push --no-verify` past the move gate,
and a hand press on each box — and the night leg never takes a step with no test record (R-1238). That does not
scale past the first customers and needs the operator awake.
1. **The entry** (`scripts/ladder.py` `check_security`, the one reader): `"verdict": "security_fix"` with `security`
`{ids: [CVE-…|GHSA-…], advisory: "https://…", fixed_in}`, `box_evidence` (required), `seed` `read_back`|`none`,
`memory_peak_pct` from the BOX soak with `memory_basis: "box_soak"`; `evidence` = the bench record or null — a
bench given must be `proven` (`bench_verdict`, `harness_version` ≥ 2 ride with it). Never `backfilled`,
`digest_from` (a same-tag fix stays decision 52's lane), `engine_conversion`, nor a database engine MAJOR (any
engine — MariaDB included; the judgement is `check-engine-major.py`'s own `engine_of`/`major_of`, imported).
2. **No operator word per incident.** The session that writes the step reports it to the operator after the push.
3. **The gates:** the move gate accepts a `security_fix` move exactly as a proven one (from/to per service, digests
the registry still serves); `check-test-record.py` WARNs on every push while a template's head is a security-fix
step with no bench test („<app>: security-fix step <to> has no bench test (since <tested_at>)"). Seven decoys
(`test_gate_decoys.py`), each seen failing with the rules switched off.
4. **The box venue:** `box_walk.py --security` on 9202 through the product's Update; verdict `proven` only if the
update ended `done` with no undo and no hold, the app runs, the four version observables are TO, the seed (if
any) reads back, and a soak (default 600 s, cgroup `anon` against the limit, OOM kills and restarts counted)
sees no kill and no restart. A negative control (a TO image that exits at once) comes back `failed`.
5. **The writer:** `upgrade-test.py --write-ladder --security …`, refusing a box walk not proven, a bench given and
not proven, a template not at the walk's FROM, a digest that moved since the walk, an engine major, a missing
field. The compose move, `catalog_since` and the superseded step's own files are the shared `apply_entry`.
6. **The controller (v0.341.0):** the leg takes `proven` OR `security_fix`; every other guard runs unchanged, in
the same order; it logs `[update-leg] <app>: security fix <ids> taken (<from> → <to>)` and the hub report's
step carries the ids. A person's press and the badge are unchanged. **A controller ≤ 0.340.0 skips the step at
night (`not_proven`) and a hand press takes it** — proven on 9202, below.
7. **The trade, stated.** A security-fix step skipped the bench. For an app with a box fixture the data is still
read back after the update; for an app without one (Plex today) the step is checked only by „it starts, it is
healthy, it runs the new version, and a 10-minute soak sees no kill or restart". `files_may_change` is a bench
measurement: with no bench the step carries `false`. That is accepted because the leg's own guard is not the only
net — the update takes a per-app backup first, the undo puts the pre-update data back on a failed health check
(decision 15), and a security fix that rewrites files without breaking the health check is the case the bench
would catch and this lane accepts the risk of. A session that knows a fix migrates files runs the bench.
**Proof (9202, drill catalog, 2026-10-23):** `audits/security-lane-2026-10-23/` — the negative control (TO
`busybox:1.36.1`) came back `failed` (the box undid it after 335 s) and the writer refused it; Plex 1.43.4.10903 →
1.43.5.11029 `proven` (done in 36.9 s, the four observables at TO, digest `sha256:01bdf0ad…5c5f`, soak 600 s: anon
peak 10.7 % of 2 GiB, 0 kills, 0 restarts, `seed: none`); the writer wrote the drill step (CVE-2026-96652, -96654,
-96655, -96656; Plex's own notice as CISA KEV cites it, added 2026-10-20; fixed in 1.43.3.10861); on 0.340.0 the
night leg logged `plex: skipped — not_proven` and a hand press took the step; on 0.341.0-rc.1 the night leg logged
`plex: security fix CVE-2026-96652,CVE-2026-96654,CVE-2026-96655,CVE-2026-96656 taken` and ended `done` in 30.1 s
after the update's second-drive copy and undo copy. Released as controller 0.341.0, global floor 11:24 UTC.
### 2026-10-25 (attended release run) — D49: a restore that does not start keeps both copies; D44–D46: the review picks
258. **A restore whose app does not start keeps the restored data AND the old copy, tells the operator, and never puts the
old data back by itself (D49 = A).** *Operator ruling 2026-10-25, given by starting the attended release run.* Options:
A) keep and wait — cost: an app that never starts needs a person; B) pour the old data back after the wait — cost: a
slow but correct first start (a database migration) is thrown away. Why A: the slow first start is the common case,
the old copy is kept, a late start turns the record into a success. Built: `07` §8.2e and fix A (the verdict needs a
stable start; the copy goes only after 3 more minutes of it) — controller 0.343.0.
259. **The 21 confirm lines and four code changes of the second review round are accepted as picked (D44 = all picks).**
*Operator, 2026-10-25.* The notes carry a normal `[DESIGN — confirmed …]`; the code changes became R-1280–R-1283
(R-1265 closed).
260. **Where a design document and the code disagree on the seven named points, the text follows the code (D45 = all
picks).** *Operator, 2026-10-25.* Each corrected place is tagged `[DESIGN — corrected 2026-10-25, D45 = all picks,
R-1266]`, the old words struck through, not deleted (R-1266 closed).
261. **Homepage is behind the family gate, like Glance (D46 = yes).** *Operator, 2026-10-25.* Why: it has no login of its
own and showed a stranger the box's CPU, memory and disk. Tried on the drill first; catalog `ebbf1f4` (R-1272 closed).
### 2026-10-25 (08:38) — D47: nothing changes an existing backup; D48: a restore downloads first (operator rulings by starting the run; R-1269)
256. **An existing recovery unit is never changed in part (D47).** A unit's definition (compose, pins, `app.yaml`) and
its data are written only together, by a backup of the same install. A new install never adopts, rewrites or
deletes a unit that an earlier install left; that unit is set aside whole and stays restorable. *Operator ruling
2026-10-25 08:38, taken by starting the R-1269 brief.* Why: on 2026-10-24 the catalog sweep measured three times
that a reinstall's own five-minute refresh wrote the new install's database password into the unit „remove — keep
backups" had kept; a restore then could not open the old data (R-1269). Built: every install has an identity
(`install_id`), the unit records it, and every writer asks first — `07` §8.2a „An earlier install's unit is set
aside" (controller, unreleased; rides v0.343.0).
257. **Every restore path resolves and pulls every image its definition names before the first change to the app; a
failed pull is a refusal with the app untouched (D48).** *Operator ruling 2026-10-25 08:38, taken by starting the
brief.* Why: the restore stopped the app and moved its data first and pulled only inside the database-only start;
a version that could not be downloaded then ended in the D22 put-back, a full stop-and-pour-back for nothing.
Built: `07` §6.6 „The limit — the image is not in the backup" (controller, unreleased; rides v0.343.0).
**Confirmed, no change:** `07` §6.6 — a restore brings the app back at its data's version, then the guarded update
climbs the ladder; R-698 (a deleted version cannot come back) and R-933 (our own image mirror) are unchanged.
### 2026-10-15 (07:48–08:20) — bake first; Plex through its own key; HSTS one day (recorded before the media build)
247. ~~**Bake first.** A controller version is RELEASED only when its golden is baked.~~ **REVERSED by decision 250,
@@ -1932,7 +2035,7 @@ moves a stack dir.
| writer | pin source |
|---|---|
| the deploy path (`runComposeDeploy`) | the template just deployed from |
| **`UpdateStack`** | the catalog's current template, written **BEFORE** the pull |
| **`UpdateStack`** | the catalog's current template, written **BEFORE** the pull — *as built: `UpdateStack` was deleted in v0.237.0; the guarded update writes the pin (`advancePinTo`, the `pinning` phase before `pulling`), and two more paths move a pin BACK — `pinBack` and the undo's `restoreDefinition` (`internal/stacks/update.go`, `undo.go`; noted 2026-10-24)* |
| the restore (`stackAdapter.RecreateStackDefinitionFromUnit`) | the recovery unit's captured compose — **this closes R-441** |
| `Manager.AdoptPins` | the observation, once, and only when complete AND matching |
@@ -1998,7 +2101,7 @@ earlier feature is the failure mode to look for whenever a file changes meaning.
| **3** | **The compose file becomes DERIVED** — the pin in `app.yaml` wins; the syncer renders instead of copying. | **SHIPPED, controller v0.235.0 (2026-09-06)** — operator ruling §3.4 |
| **4** | **A guarded update** — verified-backup precondition, abort-on-failure, and the truth at the moment of action rather than 5m16s later (R-443). | **SHIPPED + PROVEN LIVE, controller v0.237.0 (job) + v0.238.0 (page) + v0.238.1 (2026-09-13); any backup tier since v0.239.0 (§3 decision 8)** — §6.1 |
| **5** | **An upgrade test that runs again** — a harness that upgrades a real app with real data in it and asks the app for the data back. | **SHIPPED, `app-catalog/scripts/upgrade-test.py` (2026-09-06)** — 7 edges, 3 apps; see §4.1 and §10 |
| **6** | **Automatic updates** — ~~within a major, a human across one~~ **the catalog's tested steps, climbed one at a time, as a leg of the backup chain, undone by the box on failure** (§3 decisions 11–17); **an engine change gets its own edge.** | OPEN — R-450; rulings 2026-09-23; build order §6.4 |
| **6** | **Automatic updates** — ~~within a major, a human across one~~ **the catalog's tested steps, climbed one at a time, as a leg of the backup chain, undone by the box on failure** (§3 decisions 11–17); **an engine change gets its own edge.** | ~~OPEN — R-450~~ **BUILT** — the leg in `internal/stacks/unattended.go`, wired into the off-site job; R-450 CLOSED 2026-10-05 (capability map B11) *(noted 2026-10-24, review round 2)* |
| **7** | **A fleet sweep pipeline** — the operator can see, and move, how far behind every box is. | OPEN — R-451; ruled 2026-09-23 (decision 18), built later |
### 6.1 Slice 4 as SHIPPED (controller v0.237.0 + v0.238.0, 2026-09-13)
@@ -2013,12 +2116,13 @@ closed by construction: nothing reports an update complete on the compose exit c
| # | phase | what happens | on failure |
|---|---|---|---|
| 0 | refusals (409, before the intent is recorded) | held (R-439), busy (backup/restore/app-data op/quiesce), migration, already updating, deploying, memory (the deploy's `memoryVerdict`, releasing the app's own request), disk (**fixed 2 GB floor** — image size unknown without a registry), **no copy on ANY tier and no backup can be taken now** (since v0.239.0; before it, no restorable Tier-2 copy) | nothing moves, nothing is recorded |
| 1 | `checking` | walks Tier 2 → Tier 1 → Tier 3 for the first copy younger than `update.backup_max_age` (v0.239.0) | nothing moves |
| 1 | `checking` | walks Tier 2 → Tier 1 → Tier 3 for the first copy younger than `update.backup_max_age` (v0.239.0) — *an app whose data lives in bind-mounted files walks second drive → OFF-SITE → own unit (decision 9, R-479, v0.241.0; `internal/backup/update_guard.go` `updateTierOrderBindData`; noted 2026-10-24)* | nothing moves |
| 2 | `backing-up` — only when no tier holds a fresh copy | `RunAppBackupNow`: this app's DB dump → volume dump → unit capture (marked proven current) → Tier-2 copy, whose failure is a WARN since v0.239.0 | refused with the backup's own error; nothing moves |
| 3 | `safety-dump` | `WriteUpdateSafetyDump` (R-361's undo copy) — **before the pin moves** | refused; nothing moves |
| 4 | `pinning` | the previous definition is copied aside and journaled, then the pin advances | pin put back |
| 5 | `pulling` | `compose pull` | **pin and definition PUT BACK** — nothing ran (Scenario E) |
| 5a | `copying` **(v0.263.0)** | `compose stop`, then each NAMED volume `cp -a` into `<volume>.pre-update-<stamp>` by a helper that writes a finished-marker last (bind folders never) | copies removed, **pin put back, the old version started again** — nothing new ran |
| 5b | `converting` **(v0.273.0, row added 2026-10-24)** | only on a step whose ladder entry carries `engine_conversion`: the box converts the PostgreSQL major (dump from the old engine, load into the new, check) — `internal/stacks/pgconvert.go` | the undo puts the old engine's data back |
| 6 | `starting` | `compose up -d --remove-orphans` | **UNDO** (below), HOLD only if the undo fails |
| 7 | `verifying` | the `.felhom.yml` health check through the existing probe **when it resolves to a container**, else 60 s of every container running and none restarting; bounded by `update.health_timeout` | **UNDO** (below), HOLD only if the undo fails. Before v0.263.0: stop + HOLD, the pin stays (Scenario F) |
| 7a | `undoing` **(v0.263.0)** | every copy validated (marker) BEFORE anything is poured back; volumes emptied and refilled; definition, pin and the pinned version's `.felhom.yml` record put back from the job's own copies; `up`; health with the OLD version's probe | HOLD, the sentence prefixed *„A frissítés nem sikerült, és az automatikus visszaállítás sem."* + the data state (`untouched` / `half` / `not_started`); copies kept |
@@ -2067,7 +2171,7 @@ the harness has proven it, is slice 6's.
> copy taken before the pin moved**, so the old version meets pre-migration data. Spiked by hand on
> 9202 the same day — §6.1a.
**Not gated here:** a multi-major jump (R-40). It fails health and is held honestly; stepping is slice 6.
**Not gated here:** a multi-major jump (R-40). It fails health and is held honestly; stepping is slice 6. *(Noted 2026-10-24: stepping is built — the ladder, one tested step per press, decision 14.)*
### 6.1a The undo (decision 15) — SPIKED BY HAND, then BUILT: controller v0.263.2 (2026-09-23)
@@ -2081,7 +2185,8 @@ only the live box could show, fixed the same day:** the undo's probe was never a
current probe held the app `unhealthy` (v0.263.1), and the "old" `.felhom.yml` taken at update time
was already the new one, because `.felhom.yml` flows in on every catalog sync (v0.263.2: the pinned
version's file is now recorded in `applied-meta/` whenever a version is pinned). **Residual (R-646):**
an app pinned before v0.263.2 has no such record until its next pin. **Two more, found by the chaos hour of
an app pinned before v0.263.2 has no such record until its next pin. *(Closed since: `BackfillAppliedMeta` records it at
start for every app CURRENT with the catalog, and names the rest — `internal/stacks/undo.go:646-652`; noted 2026-10-24.)* **Two more, found by the chaos hour of
2026-09-23 night (`audits/DRILL-night-2026-09-23.md` Part D):** the undo finds an app's volumes by their compose
label, and a restore recreates them WITHOUT it — so after any restore the undo copies nothing (R-658, P1); and a
held file-leg app is pointed at a restore that refuses a database-only copy, with no route left on a box without
@@ -2221,7 +2326,7 @@ breaks.
---
### 6.2 Slice 6, as it will be built (OPEN — R-450; ruled 2026-09-23, decisions 11–17)
### 6.2 Slice 6, as it will be built (~~OPEN — R-450~~ BUILT, R-450 closed 2026-10-05 — noted 2026-10-24; ruled 2026-09-23, decisions 11–17)
**The shape the rulings fix. The build order and costs are §6.4.** Rewritten 2026-09-23; the
earlier "as it would be built" draft keyed on the window of Q1's recommendation and on
@@ -2292,7 +2397,7 @@ retried. A working caller in this exact shape exists as evidence, not product:
**The per-box switch key is `app_update.unattended`**, default **true** (decision 12).
⚠ **NOT `auto_update` — that name is TAKEN, and by the very thing this must not collide with.**
`self_update.auto_update` / `self_update.auto_update_time` (`config/config.go` L280-281, default
`self_update.auto_update` / `self_update.auto_update_time` (`config/config.go` L280-281 then, L278-279 now; default
**04:30** at L422) are the CONTROLLER's own update. Two settings with that name, one meaning the
controller and one meaning apps, is the kind of collision that is only discovered by an operator who
turned off the wrong one. The app-scoped key is `app_update.*`. **And `stacks.update_window` is
@@ -2322,12 +2427,12 @@ what the part can do to a household's data if it is wrong, not how likely that i
| # | part | rulings / rows | cost | depends on | risk to customer data |
|---|---|---|---|---|---|
| **1** | **SHIPPED — controller v0.263.2, proven live on 9202 2026-09-23** (`audits/undo-live-2026-09-23/`). **The undo.** Keep the pre-update copies (compose, applied, pin, **old `.felhom.yml`**) until the undo is over; in `failAndHold`: pin back → DB up alone → **validate the copy's completion marker** → **empty-then-load in one transaction** (PostgreSQL: the dump's schemas dropped and recreated inside the load's transaction; MariaDB: every table dropped first, and a failed load HOLDS with a sentence saying the database is in neither state) → full start → **health with the OLD probe** → `undone`, else HOLD. A volume tar at safety-dump time for apps with no database server. Household page + event; the mail rides part 2. | 15; the audit's 8-point list | **4** | — | **HIGH by nature** — it writes the customer's database. Bounded: it only ever loads the copy taken seconds before, validated first, atomically on PostgreSQL; every failure mode ends in today's hold. **It also makes the manual button safer on its own**, which is why it goes first. |
| **1** | **SHIPPED — controller v0.263.2, proven live on 9202 2026-09-23** (`audits/undo-live-2026-09-23/`). **The undo.** Keep the pre-update copies (compose, applied, pin, **old `.felhom.yml`**) until the undo is over; in `failAndHold` *(as SHIPPED — noted 2026-10-24: the copy is the FOLDER copy decision 19 chose: every named volume `cp -a` into a sibling volume with a finished-marker, put back on failure; no database load — `internal/stacks/undo.go:26-36`; the text that follows is the design before the bake-off)*: pin back → DB up alone → **validate the copy's completion marker** → **empty-then-load in one transaction** (PostgreSQL: the dump's schemas dropped and recreated inside the load's transaction; MariaDB: every table dropped first, and a failed load HOLDS with a sentence saying the database is in neither state) → full start → **health with the OLD probe** → `undone`, else HOLD. A volume tar at safety-dump time for apps with no database server. Household page + event; the mail rides part 2. | 15; the audit's 8-point list | **4** | — | **HIGH by nature** — it writes the customer's database. Bounded: it only ever loads the copy taken seconds before, validated first, atomically on PostgreSQL; every failure mode ends in today's hold. **It also makes the manual button safer on its own**, which is why it goes first. |
| **2** | **SHIPPED — controller v0.264.0 + hub v0.120.0, proven live on 9202 and 9201 2026-09-23** (`audits/undo-fleet-2026-09-23/`). **The update sentences in the household's language** (R-606) and a mail when an automatic update is undone or held: events `app_update_undone` (warning) and `app_update_held` (error), one per app per outcome, on by default, mailed in the household's language with the app named in the subject; per-app cooldown on both legs. Leftovers: R-647. | R-606, 15 | **1** | — | none |
| **3** | **SHIPPED — controller v0.264.0, proven live on 9202 2026-09-23.** **A disabled notifier says so** (R-620), so the mail of part 2 can be measured on a scratch box at all. | R-620 | **0.5** | — | none |
| **4** | **SHIPPED — catalog `6db08a5`, night 2026-09-23** (`audits/DRILL-night-2026-09-23.md`): `update_ladder:` in `.felhom.yml`, one JSON entry per line (spiked on controller v0.266.0 and v0.267.0 first — the controller ignores the key); `check-test-record.py` (static, CI) + `check-test-record-move.py` (history + the registry for moved refs only, decision 23); the only writer `upgrade-test.py --write-ladder`; the 21 moves of 2026-09-22 backfilled from their records (21 proven). Not built: `steps/<to>.yml` (part 5 needs it once an app has two steps); `CompareImageRefs` did NOT move to the gate — the gate asks for a proven test instead, which is decision 13's own test. **The test record + the catalog gate + the memory check.** The harness writes the ladder entry (below) from its verdict record, including the memory watch's peak and marks; the gate refuses an image move with no entry, an entry with a `failed` verdict, or one with no memory watch; `CompareImageRefs`' rule moves here as the push-time safety net. **Backfill:** one entry per current pin — the 21 proven moves from their records, every other pin `needs_person: "never tested"`, which is honest and keeps them manual. A version move re-checks `mem_limit` against the watch's peak (the RomM follow-up: gate, not checklist, because the watch now produces the number). | 13, R-635 follow-up | **2.5** | the memory watch (shipped 2026-09-23) | none on a box — catalog-side only |
| **5** | **SHIPPED — controller v0.268.0 (`206b035`) + catalog `5ed599c`, proven live on 9202 2026-09-24** (`audits/ladder-2026-09-24/partD/`): romm 5.3.0/11.4 → 5.3.1/11.4 (its app step, from `steps/90dd9d68258286ef.yml`) → 5.3.1/11.8 (the engine step; `mariadb-upgrade` ran) in two presses, the seeded account read back after each, the page's „Hátralévő frissítési lépések" 2 → 1 → none. Step files: `templates/<app>/steps/<StepKey(to)>.yml` (sha256 of `to` as canonical JSON, 16 hex; `stacks.StepKey` = `ladder.step_key`), refused absent or wrong by `check-test-record.py` rule 4, written by `--write-ladder` when a step is superseded, 8 backfilled from the NEWEST commit naming each step's images (romm's from `f4eb94f`, not the OOM-looping `15f9ebf`). The box reads them from its `--depth 1` clone (the whole tree is there — measured). One press = one step; a missing step file refuses before anything moves; an installed version matching no entry jumps, logged by name (measured live: vikunja 2.5.0). Limitation: a step has no `.felhom.yml` of its own (R-664). **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/<to>.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape |
| **6** | **BOTH HALVES SHIPPED. BOX HALF — controller v0.269.0 + v0.269.1, proven live on 9202 2026-09-24** (`audits/night-2026-09-24/B/`): the compose that runs pins `tag@sha256` from the ladder entry that tested those refs; pins and records stay digest-free; the badge reads behind for a newer TESTED digest; **an installed app keeps its digest until a guarded Update moves it** (v0.269.0 let the sync move it — found live, fixed in v0.269.1, `CarryDigests`); Compose accepts the form for all 25 ladder apps. **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. **The box half (compare, render `name:tag@sha256`) is not built.** **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low |
| **5** | **SHIPPED — controller v0.268.0 (`206b035`) + catalog `5ed599c`, proven live on 9202 2026-09-24** (`audits/ladder-2026-09-24/partD/`): romm 5.3.0/11.4 → 5.3.1/11.4 (its app step, from `steps/90dd9d68258286ef.yml`) → 5.3.1/11.8 (the engine step; `mariadb-upgrade` ran) in two presses, the seeded account read back after each, the page's „Hátralévő frissítési lépések" 2 → 1 → none. Step files: `templates/<app>/steps/<StepKey(to)>.yml` (sha256 of `to` as canonical JSON, 16 hex; `stacks.StepKey` = `ladder.step_key`), refused absent or wrong by `check-test-record.py` rule 4, written by `--write-ladder` when a step is superseded, 8 backfilled from the NEWEST commit naming each step's images (romm's from `f4eb94f`, not the OOM-looping `15f9ebf`). The box reads them from its `--depth 1` clone (the whole tree is there — measured). One press = one step; a missing step file refuses before anything moves; an installed version matching no entry jumps, logged by name (measured live: vikunja 2.5.0). Limitation: a step has no `.felhom.yml` of its own (R-664). *(Closed since: a step's own `steps/<key>.felhom.yml` is read when present — `ladder.go:127-132`, v0.269.0; noted 2026-10-24.)* **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/<to>.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape |
| **6** | **BOTH HALVES SHIPPED. BOX HALF — controller v0.269.0 + v0.269.1, proven live on 9202 2026-09-24** (`audits/night-2026-09-24/B/`): the compose that runs pins `tag@sha256` from the ladder entry that tested those refs; pins and records stay digest-free; the badge reads behind for a newer TESTED digest; **an installed app keeps its digest until a guarded Update moves it** (v0.269.0 let the sync move it — found live, fixed in v0.269.1, `CarryDigests`); Compose accepts the form for all 25 ladder apps. **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. ~~**The box half (compare, render `name:tag@sha256`) is not built.**~~ *Built — the box renders the tested digest, carries it on sync, and the badge reads the tested digest (`internal/stacks/digest.go:15-24`, `updateorder.go`; noted 2026-10-24).* **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low |
| **7** | **SHIPPED — controller v0.271.0 (`9cf13a3`), proven on 9202 over four simulated nights 2026-09-24/25 and watched through the demo boxes' first real automatic night (`audits/DRILL-night-2026-09-25.md`).** Decisions 31–33 taken unattended. Gaps a scratch box cannot close: R-687; no resume after a restart: R-686. *(Before:)* **SPIKED 2026-09-24 (night, Part C) — the measured build brief is §6.4.2 below.** **The update leg in the chain + the automatic caller + the switch.** A leg that starts when the off-site leg has ENDED (on every exit path, skips included), one app at a time, one step per press, `app_update.unattended` default ON (decision 12), `stacks.update_window` removed, reads `UpdateRefusal.Reason`, remembers a failed step so it never re-presses it, and the full-system backup's gate waits for it until W+5h (decision 20). | 11, 12, 14, 15, 20 | **3** | 1, 2, 5 | medium — the only part that acts with nobody watching; everything above is what makes it safe |
| **8** | **SHIPPED — controller v0.265.0 + hub v0.121.0, proven live on 9202 2026-09-23** (`audits/cleanup-2026-09-23/`; a kernel `oom_kill` counter, not the sticky flag — `08` §6.2). **R-636** — the same OOM key re-firing escalates instead of staying one `warning` for six hours. | R-636 | **1** | — | none |
| **9** | **SHIPPED — controller v0.265.0, proven live on 9202 2026-09-23** (badge „Megállítva — visszaállítás szükséges" / "Stopped — restore needed", no Update button, 409 `held` unchanged). **R-625** — a held app stops offering an Update it will refuse. With the undo, holds become rarer; the lie on the page does not go away by itself. | R-625 | **0.5** | 1 | none |
@@ -2410,7 +2515,7 @@ During the db-dump leg every press was refused `busy` (transient, retried next p
update's own finish should refresh them.
- **An Update pressed on an app already at the head runs the whole guarded update** — backup, pull,
restart, `done` — with no refusal (R-679: navidrome restarted four times for nothing). The leg never
presses an app whose pin equals the head; the preflight should refuse with reason `current`.
presses an app whose pin equals the head; the preflight should refuse with reason `current` (as built the reason key is `already_current`, `update.go:432` — noted 2026-10-24).
- **The product does not remember a failed step.** After vikunja's undo the badge still offers the
same step; only the caller's memory kept it from a re-press. Decision 15 needs it on the box: the
failed `to` is recorded per app and the leg skips it until the catalog's ladder changes.
@@ -2492,6 +2597,24 @@ and E are the ones that unblock a decision; leg A is the one that takes the time
harness on DooPlex for the image-side ones.
### 6.4.x Decisions in code no section records [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
Read at `file:line` on controller `main` (`321df1f`) by the main session:
- **`build.sh --push` also refuses a DIRTY tree, a HEAD that is not `origin/main`, and a directory that is not a git
checkout** — `controller/build.sh:144-166`. §0 names only the tag-exists and unreachable-registry refusals.
- **The leg retries a passing refusal ONCE in the same night** (`busy`, `updating`, `deploying`, `migrating`,
`self_updating`), after `RetryWait` — not „the next pass/night" (`internal/stacks/unattended.go:65-66`, the pass loop).
- **The leg skips an app the household stopped** (`stopped_by_household`, R-721): an update would start it
(`unattended.go:416-418`).
- **A manual run of the chain** (R-705, the debug action) gives the leg a deadline of its start + the leg's normal
length (`backupwindow.UpdateLegLengthMin`), not W+5h (`unattended.go:232-238`).
- **Two preflight refusals are plain skips to the leg:** `already_current` and `engine_no_test`
(`internal/stacks/update.go:432,454`).
- **The render table (§5.4) has one more row:** when the catalog template's images cannot be read, the syncer leaves
the app's compose file alone for that cycle (`internal/sync/sync.go:622-627`); and the box's R-971 allow-list runs
before ANY compose write (`cmd/controller/main.go:421`, `SetAdmitFn`).
## 6.5 The drill catalog and the image store — the standing method for update drills
**Why this section exists.** On 2026-09-21 an afternoon session put a deliberately broken image into
@@ -2580,7 +2703,9 @@ customer box has credentials for it. The store listens on the guest's loopback o
### 7.1 The record (slice 1)
`Manager.recordInstalledImages` (`felhom-controller/controller/internal/stacks/installed.go`) runs
after a successful compose up from `StartStack`, `RestartStack`, `UpdateStack` and `runComposeDeploy`,
after a successful compose up from `StartStack`, `RestartStack`, `UpdateStack` and `runComposeDeploy`
*(as built — noted 2026-10-24: `UpdateStack` is gone; the callers are `StartStack`, `RestartStack`, the deploy path, the
end of the guarded update and the end of the undo — `manager.go`, `deploy.go`, `update.go`, `undo.go`)*,
and writes `app.yaml`:
```yaml
@@ -2656,7 +2781,7 @@ Version strings stay in the logs, the API and the hub.
The 8th, immich's own ghcr build, is UNMEASURED — ghcr exposes no anonymous last-modified. So on
demo-hp today four apps read „Naprakész" over a database engine image that has demonstrably moved.
Digest-level comparison needs the catalog to record the digest at push time — **R-446**, put to the
operator as §3b **Q6**, recommended YES.
operator as §3b **Q6**, recommended YES. *(Built since for every ref with a ladder entry — `updateorder.go`, `digest.go`; the limitation remains only for a ref no entry records; noted 2026-10-24.)*
2. **~~Nothing enforces `catalog_since`.~~ Enforced by the pre-push hook since 2026-09-13 (R-452, `app-catalog-felhom.eu/scripts/check-catalog-since.py`); CI's shallow clone still skips it out loud.** A commit that moves an `image:` line and forgets the date
under-reports how far behind a box is. The gates runner fetches at `--depth 1` and has no parent
commit to diff against, so the gate needs a deeper fetch — **R-452**.
@@ -2681,13 +2806,18 @@ Version strings stay in the logs, the API and the hub.
refuses without a restorable, proven Tier-2 copy, backs up first when that copy is stale, takes a
safety dump, and holds an app that does not come up. **What stays true:** it still has no automatic
rollback (deliberately, §6.1) and can still attempt a multi-major jump the app will refuse (R-40) —
that now ends HELD rather than crash-looping behind a green button.
that now ends HELD rather than crash-looping behind a green button. *(Dated note 2026-10-24: a copy on ANY tier now
satisfies the precondition (v0.239.0), the box undoes a failed update itself (decision 15, v0.263.2), and the
ladder climbs one tested step per press (decision 14) — `internal/backup/update_guard.go`, `internal/stacks/undo.go`,
`ladder.go`.)*
7. ~~**An engine major can be applied without its datadir upgrade, and nothing notices.**~~ **CLOSED
2026-09-13 for MariaDB (R-459):** every `mariadb:` sidecar carries `MARIADB_AUTO_UPGRADE=1`, and the
harness shows the conversion running on the E3/E3b edges (§3 decision 5). **What stays true:** the
PostgreSQL half (R-463) has no equivalent — the image performs no `pg_upgrade` — and the
engine-major rule (§3 precaution 3, R-469) is what keeps both engines inside their major until
Slice 4 gives the Update button a backup.
Slice 4 gives the Update button a backup. *(Dated note 2026-10-24: MariaDB is lifted — its major is its own edge; a
PostgreSQL major may pass with a two-venue proof and an `engine_conversion` mark, and the box converts it
(v0.273.0) — `app-catalog-felhom.eu/scripts/check-engine-major.py`.)*
8. ~~**Only three of 53 apps have ever had an upgrade measured.**~~ **WIDENED 2026-09-21 to 21
EDGES ACROSS 19 APPS** (`audits/DRILL-update-night-2026-09-21.md`), on scratch guest 9202
through the product's own guarded Update, against a PRIVATE DRILL CATALOG so the live catalog
+11 -7
View File
@@ -128,7 +128,9 @@ the hub's e-mails follow.**
`expired` pinned to Hungarian so the language cannot become the oracle the text refuses to be.
Full design: `05-hub-architecture.md` §15. **R-555 closed** — the `language` allowlist entry is out
of `wire_contract_gate.py` and the gate now checks the field for real.
- **[DESIGN] Slice 3 Part A as planned — now built; the box half (Part B) is the remaining piece:** the hub stores a per-customer language, renders it into
- **[DESIGN] Slice 3 Part A as planned — now built; the box half (Part B) is the remaining piece** *(Part B is built
too — the controller reads `customer.language` as its start language since v0.256.0, `internal/config/config.go:149-154`;
corrected 2026-10-24)*: the hub stores a per-customer language, renders it into
`controller.yaml` next to `customer.id/name/domain/email` (`hub/internal/configgen/configgen.go`),
and the box uses it **only while the household has never chosen** (`settings.json` empty). The
household's own choice always wins; the hub's e-mails follow the language the box **reports**, which
@@ -184,7 +186,8 @@ bundles. They now judge the page **as rendered in each language** (`controller/s
- **hu** — the product speaks in „te". The converted copy carries formal („ön") forms (R-516). They are
not fixed by a localisation release (§1), so the gate counts them against `HU_FORMAL_CEILING` as a
ratchet: a new one convicts, fixing one lowers the ceiling. The ceiling followed the conversion:
6 (spike) → 10 (release A) → 12 (release B) → **16** (release C). **[FACT] Measured limit:** the stem
6 (spike) → 10 (release A) → 12 (release B) → **16** (release C) → … → **0** (R-516 item 12, the last eleven in the
te-form — `felhom-controller/controller/scripts/i18n_missing_gate.py:41`; corrected 2026-10-24). **[FACT] Measured limit:** the stem
list is short — on the release C pages it counts 4 formal forms, while a wider list („írja be",
„adja meg", „válassza", „biztosan eltávolítja", „engedélyezze", …) finds at least 22 keys (R-516).
- **en** — second person, plain: no „please", no „kindly".
@@ -201,7 +204,8 @@ vissza") — R-564. The hub copy gate and the catalog have no language concept
- **[DESIGN] The hub's operator pages** — the operator reads them; they stay English (decision 1).
- **[DESIGN] The first-boot wizard** (`controller/internal/setup/`, 8 pages, 95 strings) — out of scope
and obsolete (decision 4, 2026-09-17; `02-controller-module-map.md` L56). A household still reaches
it when bootstrap ingestion leaves `customer.id` empty (inventory §2.7). Its deletion is **R-554**.
it when bootstrap ingestion leaves `customer.id` empty (inventory §2.7). Its deletion is **R-554**. *(Deleted since:
`internal/setup/` is gone (v0.308.0); `bootstrap.WaitForConfig` shows a waiting page instead — corrected 2026-10-24.)*
- **[FACT]** App containers' own UIs (Uptime Kuma, PrivateBin…) — not Felhom's copy (R-516 items 5-6).
---
@@ -226,7 +230,7 @@ i18n:
```
Why: one app's copy stays in one file a reviewer sees whole; the controller's `yaml.Unmarshal` ignores
unknown keys (`internal/stacks/metadata.go` L316 — **verified: this repo constructs no `yaml.Decoder`
unknown keys (`internal/stacks/metadata.go` L316 then, L475 now — **verified: this repo constructs no `yaml.Decoder`
at all, so there is no `KnownFields` anywhere to turn strict**), so older controllers are unaffected;
a missing English field falls back field by field.
@@ -234,7 +238,7 @@ a missing English field falls back field by field.
reached only through `LocalizeStacks` / `LocalizeStackPtr` / `Stack.MetaFor`.
- **`For("hu")` is the parsed struct with `I18n` cleared and nothing else**, deep-compared against all
53 real catalog files (`TestMetaForHuIsIdentity`, fixtures in `internal/stacks/testdata/catalog/`).
53 real catalog files (the identity fixtures are still those 53; the catalog has 65 apps — noted 2026-10-24) (`TestMetaForHuIsIdentity`, fixtures in `internal/stacks/testdata/catalog/`).
§1 therefore holds by construction for catalog copy, and `LocalizeStacks` returns the INPUT SLICE
for Hungarian rather than a copy, so nothing on the Hungarian path is even touched.
- **Fallback is field by field, and blank counts as absent** (ruling 3). A half-translated app is a
@@ -307,10 +311,10 @@ for the English page test (R-565), arriving again in a different repo. So
`app-catalog-felhom.eu/scripts/check-copy-i18n.py` folds and matches word-bounded stems, with a
positive and two negative controls printed on every run.
**[FACT] The gate**, fifth row of `catalog_gates.py`, static, in the pre-push hook:
**[FACT] The gate**, fifth row of `catalog_gates.py` (the sixth since — corrected 2026-10-24), static, in the pre-push hook:
1. **FREEZE** — every Hungarian copy string equals `scripts/copy_freeze/hu.json`, captured in its own
commit before any translation. Runs on all 53 apps whatever scope is named. A new app must be
commit before any translation. Runs on every app in the freeze file (65 on 2026-10-24) whatever scope is named. A new app must be
admitted with `--add-app NAME --reason "…"`.
2. **STRUCTURE** — the `en` block may carry copy fields and nothing else; every key-matched entry must
have a Hungarian twin, or it is INERT on the box and the translator never learns.
+18 -8
View File
@@ -11,7 +11,7 @@
> | | |
> |---|---|
> | **Status** | **NOT RATIFIED — a PROPOSAL with operator rulings, corrected by the 2026-10-04 spike (§7.1, C1–C12); §8 step 2 BUILT 2026-10-04 (§8.1).** Ratification is Viktor's review, not an editor's. |
> | **Status** | **NOT RATIFIED — a PROPOSAL with operator rulings, corrected by the 2026-10-04 spike (§7.1, C1–C12); §8 step 2 BUILT 2026-10-04 (§8.1); steps 2–6 are BUILT by §8's own list (corrected 2026-10-24 — the ratification is still the operator's).** Ratification is Viktor's review, not an editor's. |
> | **Written** | 2026-10-04, by the reviewer (project Claude), before any spike. |
> | **Verified against** | felhom.eu `d07a1a9` · felhom-controller `99a1497` (v0.290.0) · felhom-agent `d766666` (v0.138.0) · hub v0.128.0 |
> | **Freshness** | **CURRENT** as of 2026-10-04. The spike `TASK-backup-close-and-os-updates-spike-2026-10-04` adds measurements here as `[FACT]` and corrects every claim it disproves. Mark this file STALE when it falls behind what the product does. |
@@ -76,10 +76,14 @@ boxes installed. Nobody keeps a hand-written list.
- **A BYO host** (the owner brought their own Proxmox). `[FACT]` The installer leaves its repositories
alone: *"apt repo alignment skipped (byo — the owner manages repos)"* (`scripts/felhom-host-install.sh`,
`align_apt_repos`). `[PROPOSAL]` On a BYO box we update the guest and Docker only, never the host.
**[DESIGN, operator ruling D43 = A, 2026-10-24 — committed agent `c8d3c17`, unreleased]** The host's root-key-only sshd
drop-in (D4, R-982) is an APPLIANCE-only file of the config bundle: written only when the root-owned install record
says `appliance`; on a `byo` record nothing is written and a copy the bundle wrote earlier (this path, our first line)
is removed; a box whose record proves nothing is left as it is (R-1257).
- **The Proxmox MAJOR upgrade** (PVE 9 → 10). It is a later step of its own, drilled first (R-808 item 5).
- **`cloudflared` and the other infrastructure images** (traefik, filebrowser). **[FACT] (C8)** They are containers in
the guest, pinned in the controller (`internal/infra/infra.go:26`: `cloudflare/cloudflared:2026.6.0`, since
2026-06-11) and baked into the golden; upstream was `2026.9.3` on 2026-10-04. They move only by a controller release
2026-06-11 — *`2026.9.3` today, corrected 2026-10-24*) and baked into the golden; upstream was `2026.9.3` on 2026-10-04. They move only by a controller release
— the app-image question (`09`), not an OS package. The gap is R-838.
---
@@ -101,9 +105,10 @@ gets the Docker engine that was current when its golden was baked, and keeps it.
~~**[FACT] The agent may not run `apt` today, with one exception.** Its sudoers allowlist
(`felhom-agent/configs/felhom-agent.sudoers`) holds one apt line: `apt-get install -y -q dnsmasq`.~~
**[FACT] (C1) The agent may run `apt-get install` for two named packages:** `felhom-agent.sudoers:58`
(`apt-get install -y -q dnsmasq`, used by `internal/lanresolver/lanresolver.go:107`) and `:194`
(`apt-get install -y -q wireguard-tools`, used by `internal/wgtunnel/manager.go:628`). Nothing else.
**[FACT] (C1) The agent may run `apt-get install` for two named packages:** `felhom-agent.sudoers:68`
(`apt-get install -y -q dnsmasq`, used by `internal/lanresolver/lanresolver.go:133`) and `:228`
(`apt-get install -y -q wireguard-tools`, used by `internal/wgtunnel/manager.go:659`). Nothing else. *(Line numbers
corrected 2026-10-24.)*
`apt` as root runs package scripts as root, so a broad `apt` grant is a full root grant. §5.4
proposes how to avoid that.
@@ -250,7 +255,7 @@ approval of that day stays in force (decided by CC unattended — operator may r
versions. A broken-into hub can choose an older version or no version. It cannot make a box install a
package that the publisher did not sign.
### 5.4.1 The root wrapper's interface — DRAFT (2026-10-04, design only, nothing installed) `[PROPOSAL]`
### 5.4.1 The root wrapper's interface — DRAFT (2026-10-04, design only, nothing installed) `[PROPOSAL]` — *BUILT and installed since (§8.1); the agent's sudoers holds only the `--plan` line, no `--repair-only` (`felhom-agent/configs/felhom-agent.sudoers:337`); noted 2026-10-24*
`felhom-os-apply` — root-owned (`0755 root:root`), installed by the installer beside `felhom-selfupdate-guarded`;
the agent's sudoers gets exactly `/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json` and
@@ -315,7 +320,7 @@ PID 1 or `lxc-start`, or a kernel was installed.
Decision 96. Before it, the signed `agent_update` replaced only the binary; wrappers, units and sudoers lines reached
an installed box by reinstall or by hand.
- **One source of truth.** `BUNDLE_FILES` in `felhom-os-apply` is the ONE table of root-owned paths (22: sudoers ×2, the
- **One source of truth.** `BUNDLE_FILES` in `felhom-os-apply` is the ONE table of root-owned paths (22 then, **32** counted 2026-10-24 — plus the vzdump-partial-clean, thin-check, internal-place and priv-apply wrappers, the guest hook, the shared-parent unit, the GRUB one-shot pair and the sshd root-key-only drop-in; sudoers ×2, the
five wrappers, the crash guard and its units + config, the agent and rollback units, the start-limit drop-in, the mgmt
watchdog and its tmpfiles/units, the OOB belt's four files). `scripts/build-config-bundle.py` builds the bundle from it
reproducibly; `release-agent.sh` publishes it beside the binary; the hub vouches its sha with the agent (exact-name
@@ -393,6 +398,9 @@ must never overlap a backup, a restore-test or a self-update.~~
- **Operator:** a hub event for each update and each failure; a fleet view showing each box's OS release,
how far behind it is, and whether it needs a reboot. A box that is more than N days behind the newest
approved release raises an alarm (`08`).
- **[FACT, added 2026-10-24 — hub v0.169.0] The System page's „Upload line" cell** shows the best of the last SUCCESSFUL
weekly line test (R-1247), amber when it is older than 15 days or was never measured after an attempt — judged on
`last_ok_at`, never on the attempt time (`hub/internal/web/system.go` `lineStaleAfter`, `lineTestCell`).
- **[FACT, hub v0.132.0] The System page** (`/system`, R-852, decision 89): one row per box — ring and switch with
buttons, the tunnel, host Proxmox / running and next-boot kernel / Debian / release / pending / not covered / held /
reboot needed / `kernel.panic` / oops / crash restarts / the guard, guest Debian / release / pending / restart needed,
@@ -480,7 +488,9 @@ Decision 88 (R-851): *"Yes, but maybe not indefinitely."* Evidence `audits/os-do
**What it updates:** the host's Proxmox USERSPACE packages (pve-manager, qemu-server, pve-container, lxc-pve,
libpve-*, proxmox-widget-toolkit, the ceph client libraries …) — the 77–78 packages behind on the demo boxes on
2026-10-07. **What it never touches:** a kernel, boot, firmware or microcode name (`HOST_SLOW_RE`: `proxmox-kernel*`,
`proxmox-default-kernel`, `pve-kernel*`, `pve-firmware`, `shim*`, `grub*`, `systemd-boot`, `*-microcode`, `efibootmgr`)
`proxmox-default-kernel`, `pve-kernel*`, `pve-firmware`, `shim*`, `grub*`, `systemd-boot`, `*-microcode`, `efibootmgr`,
and — read 2026-10-24 in `felhom-os-apply` — `linux-(image|headers|kbuild|modules|base)*`, `firmware-*`,
`proxmox-secure-boot-support`)
— that is R-836's lane (decision 164). No reboot.
- **Wrapper** (`felhom-agent/configs/felhom-os-apply`): layer `pve`, lane `slow` only. Origin `Proxmox Debian
@@ -20,8 +20,10 @@ workspace-root `CLAUDE.md` § „Artifact taxonomy". **[FACT]** The spec shape i
class). Validation is project Claude's unless CC is asked (`CLAUDE.md`, same section).
**A session with no spec** (a `/goal`, a night brief, „work the register") inherits the discipline a
spec used to carry from `.claude/rules/unprompted-work.md` — **[FACT]** five byte-identical copies:
the workspace root and each of the four repos (the file's own header says „change all five or none").
spec used to carry from `.claude/rules/unprompted-work.md` — **[FACT]** five copies:
the workspace root and each of the four repos (the file's own header says „change all five or none"). *(Measured
2026-10-24: they are NOT byte-identical — the felhom.eu and felhom-controller copies carry one more paragraph, „A
version number is pushed once" (added 2026-10-15); the review round 2 session aligned the other three, see its report.)*
## 2. Where an instruction lives
@@ -29,7 +31,7 @@ the workspace root and each of the four repos (the file's own header says „cha
|---|---|---|
| Cross-repo facts and standing rules (production host, gates, secrets, CHANGELOG/REPORT) | workspace-root `CLAUDE.md` | every session in the workspace |
| One repo's stable orientation | `<repo>/CLAUDE.md` | when a file in that repo is touched |
| Path-scoped rules (a subsystem's traps) | `<repo>/.claude/rules/*.md` | when a matching path is touched |
| Path-scoped rules (a subsystem's traps) | `<repo>/.claude/rules/*.md` | when a matching path is touched — a file with `unconditional: true` (e.g. `live-probes.md`) or without `paths:` (the `unprompted-work.md` copies) loads every session (corrected 2026-10-24) |
| A procedure with steps (build, test, diagnose, write for the operator) | a skill, `felhom.eu/skills/<name>/SKILL.md` | when its trigger matches, or by name |
| Current state, decisions, open work | `CONTEXT.md`, `STATUS.md`, `documentation/backlog/OPEN-ITEMS.md` | read on demand — never copied into an instruction file |
| Durable cross-session facts about the operator and the machines | the memory folder (`.claude-memory/`, index `MEMORY.md`) | the index each session; files on recall |
@@ -38,8 +40,9 @@ Which rung a piece of material belongs on is the doc-authoring skill's §3 („W
material sits") and §6 („One rule, one home"). This page does not repeat them.
**[FACT] Skills** are written in `felhom.eu/skills/`, installed by
`python3 felhom.eu/scripts/install_skills.py` as symlinks into `~/.claude/skills/` (so a repo edit is
live at once), and checked by `python3 felhom.eu/scripts/check_skills.py` (shape, the 150-line limit
`python3 felhom.eu/scripts/install_skills.py` as symlinks into `~/.claude/skills/` (so an edit to an INSTALLED skill is
live at once; a NEW skill folder is live only after `install_skills.py` runs again — measured 2026-10-24:
`skills/felhom-web-design` has no link yet), and checked by `python3 felhom.eu/scripts/check_skills.py` (shape, the 150-line limit
and its one grandfathered entry, R-394). The list is the folder; `skills/SOURCES.md` names where each
skill's material came from.
+19 -9
View File
@@ -36,6 +36,7 @@ decision (R-925).
| g | a compromised Gitea / registry / build | push to the catalog, the images and the installer tag |
| h | a thief with the box | the disks |
| i | Cloudflare | the plaintext of tunnel traffic — **accepted** (R-904) |
| j | **added 2026-10-23:** a hacked homelab app pod on DooPlex (happened: Plex2Shell, HM-040) | the pod network with free egress: every cluster Service, the node's ports (NFS, kubelet, k3s API), the LAN masqueraded as DooPlex; no service-account token |
## 4. Entry points
@@ -46,8 +47,8 @@ scripts are held with the review's working notes.
| component | routes | no credential | listeners |
|---|---|---|---|
| controller | 231 (+8 in setup mode) | 21, of which 7 change state, each behind its own password, code or token | port 8080 on the guest's Docker networks (not published); mail relay 2525/2465/2526 |
| hub | 121 (1 health, 53 API, 67 web — v0.152.0 added the two token routes; both route tables are now pinned to the source by tests, R-952) | appliance register (rate-limited, by design), re-enroll (only while the operator has armed recovery mode), login, health | one ingress, path `/`, geo filter (HU, IE, LAN) |
| agent | 38 local-API routes (`POST /host/poweroff` added 2026-10-13, R-998) | 0 | local API on the island bridge; WireGuard; poke (tunnel only); felhom-sshd (tunnel only, measured) |
| hub | 121 (1 health, 53 API, 67 web — v0.152.0 added the two token routes; both route tables are now pinned to the source by tests, R-952) — *the API table has 54 entries since `/box-config/` (v0.161.0) and `/preferences` (v0.165.0); counted from `hub/internal/api/routes.go`, 2026-10-24* | appliance register (rate-limited, by design), re-enroll (only while the operator has armed recovery mode), login, health, and the public self-bind page `/bind/<token>` (a capability link, added to this list 2026-10-24) | one ingress, path `/`, geo filter (HU, IE, LAN) |
| agent | 41 local-API routes on `main` (counted 2026-10-24 in `felhom-agent/internal/localapi/server.go`; `POST /host/poweroff` added 2026-10-13, R-998; `POST /host/line-test/run` added 2026-10-24, R-1250, unreleased) | 0 | local API on the island bridge; WireGuard; poke (tunnel only); felhom-sshd (tunnel only, measured) |
**Outbound fetches that bring code or config onto a box:** the catalog (git, `main`), app images, the four infra
images, the controller image (by tag), `controller.yaml` from the hub, the agent binary and config bundle (signed),
@@ -57,11 +58,11 @@ OS packages (exact versions, apt-signed), the installer script (by tag, R-980).
| boundary | verdict | evidence |
|---|---|---|
| end-user ↔ apps | holds, with gaps | traefik routing and header stripping hold; homepage has the Docker socket and no gate (R-981) |
| end-user ↔ apps | holds, with gaps | traefik routing and header stripping hold; homepage has the Docker socket and no gate (R-981). **Re-check (2026-10-24, review round 2):** the socket is gone from the homepage template (catalog `templates/homepage/docker-compose.yml:21`) and the box's allow-list is empty, so a socket bind is refused (`felhom-controller/controller/internal/privpolicy/allowlist.json`) |
| **added 2026-10-14 (R-1033, P1):** a visitor ↔ Cloudflare over plain http | holds on boxes with controller ≥ 0.312.0 | before: every public name answered `http://` (login pages too), and Sonarr/Radarr/Prowlarr redirected https visitors to an absolute `http://` login; a password could cross an untrusted network unencrypted (attacker: anyone on the visitor's network). Now traefik sends every `http://` request to https (301) before an app sees it, HSTS one day; boxes below 0.312.0 (Tester 2 at 0.292.0) keep the gap until the floor delivers it. `audits/https-only-2026-10-14/` |
| **added 2026-10-15 (R-1045):** a stranger ↔ a fresh Plex | holds on controller ≥ 0.313.0; on older ones only while the household does not press „Kész" before claiming | an unclaimed Plex answers everything to a private address with no token, and through the tunnel every request comes from one. The setup gate (catalog `9a23ee5`) closes the public path; 0.313.0 opens it only when Plex itself says claimed. A status probe meant to cover older controllers OPENED an unclaimed Plex while it started (`audits/media-build-2026-10-15/G7/`) and was removed |
| customer ↔ controller UI | partly | bcrypt and session hold; small gaps (R-983); family sessions survive a reset (R-976) |
| controller ↔ agent | partly | every route bound to the caller's guest; a compromised controller can still flush restore points (R-973), void the recovery code (R-978), and — since agent v0.155.0 — switch the whole machine off once per power-on, holding the hub's silence alarms for up to 14 days (R-998; mitigations and residual risk: `14-host-power-off.md` §5) |
| customer ↔ controller UI | partly | bcrypt and session hold; small gaps (R-983); family sessions survive a reset (R-976). **Re-check (2026-10-24, review round 2):** both fixed — a password change or a claim/reset ends every dashboard, family and gate session (controller v0.324.0, `internal/web/auth.go:352-366`) |
| controller ↔ agent | partly | every route bound to the caller's guest; a compromised controller can still flush restore points (R-973), void the recovery code (R-978) *(re-check 2026-10-24: both narrowed — a cadence cap on `POST /backup` since agent v0.160.0, `internal/localapi/backup_cap.go`; a new escrow package is uploaded only at the claim since agent v0.160.0, `escrow_ceremony.go`)*, and — since agent v0.155.0 — switch the whole machine off once per power-on, holding the hub's silence alarms for up to 14 days (R-998; mitigations and residual risk: `14-host-power-off.md` §5) |
| **added (R-1007):** controller → an app's own job queue | holds — nothing new | the `after_unclean_boot` commands run inside the app's container through the docker socket the controller already holds, and read the app's database with a read-only session: no key into the household's account, nothing a compromised controller could not already do (exec in any app, read its database). The household's lost-file names stay on the box (`app.yaml`, the app page); the operator event carries counts only, and a failing command's output is never logged (security review of controller `8d862e4`, fixed in `3b85a0c`) |
| **added:** agent ↔ root on the host | holds by the sudoers | every root verb is one regex-anchored line or a root wrapper that re-checks its input (`03` §3); agent v0.156.1 adds one delete: `felhom-vzdump-partial-clean` removes only a DEAD half-written vzdump of the named guest on a `dir` storage (no vzdump on the host, nothing holding it open) — a compromised agent gains the deletion of a leftover that is not a backup |
| agent ↔ hub | holds for destructive ops | signed jobs verified in the documented order; unsigned spec has no size bounds (R-984) |
@@ -69,12 +70,21 @@ OS packages (exact versions, apt-signed), the installer script (by tag, R-980).
| controller ↔ hub | **did not hold until 2026-10-12** | the global key was public (R-950, rotated) and three routes needed no key (R-951, fixed); per-box keys are bound to their box (R-955) |
| box ↔ PBS | holds as written | ciphertext only; the zero-knowledge wording is stronger than a live box allows (R-985) |
| guest ↔ Proxmox host | holds | the guest has no Proxmox credentials (measured) |
| hub ↔ Cloudflare API | holds for the zone check | tokens are stored unsealed in the hub database (R-975); R-137 open |
| hub ↔ Cloudflare API | holds for the zone check | tokens are stored unsealed in the hub database (R-975); R-137 open. **Re-check (2026-10-24, review round 2):** the tokens are sealed (`enc:v1:`) since hub 0.168.0, deployed 2026-10-24 (`hub/internal/store/r975_config_token_seal.go`) |
| **added:** app ↔ app (shared Docker daemon) | partly | separate containers and networks; any docker-socket app is root in the guest (R-981) |
| **added:** app ↔ controller | partly | apps share a network with the controller; dashboard routes still need a session |
| **added:** LAN ↔ box | partly | root password SSH on the host from the LAN (R-982) |
| **added:** hub ↔ box (inbound) | **does not hold** | `01` §4 says the box exposes no control endpoint; a hub-key route can replace the box's settings (R-970) |
| **added:** catalog ↔ every box | **design gap** | a catalog push reaches every box with no check on the box (R-971) |
| **added:** LAN ↔ box | partly | root password SSH on the host from the LAN (R-982). **Re-check (2026-10-24, review round 2):** the config bundle's key-only drop-in shipped in agent v0.163.0 (`felhom-agent/configs/felhom-os-apply:230`) |
| **added:** hub ↔ box (inbound) | **does not hold** | `01` §4 says the box exposes no control endpoint; a hub-key route can replace the box's settings (R-970). **Re-check (2026-10-24, review round 2):** the config routes are removed and no route takes the hub key — pinned by `felhom-controller/controller/cmd/controller/r970_no_hub_key_door_test.go` (`01` §4 says the same) |
| **added:** catalog ↔ every box | **design gap** | a catalog push reaches every box with no check on the box (R-971). **Re-check (2026-10-24, review round 2):** the box checks every rendered compose against its allow-list before any write since controller v0.328.0 (`internal/privpolicy/privpolicy.go`) |
| **added 2026-10-23 (R-1241):** a hacked app pod on DooPlex ↔ Felhom's secrets | **does not hold**; not used in the Plex2Shell window | two no-login doors: the Longhorn NFS export (the hub's unencrypted nightly copies with the unsealed tokens of R-975; by a published file-handle escape the whole sda1 disk, the operator signing keys included — not exercised) and the Longhorn API (backup a volume to any target). Plex2Shell, 2026-10-17 → 10-23: **NOT REACHED** — 0 new NFS connections in the window, no off-schedule backup or volume; the hub, the k8s API and the kubelet refuse a caller with no credential (measured). The export was narrowed to the pod network on 2026-10-23; the pod side stays open (`audits/plex2shell-felhom-impact-2026-10-23/`). **Felhom half, 2026-10-23 night:** the last plain tokens sealed in hub 0.168.0 (deployed 2026-10-24 — corrected 2026-10-24; `audits/hub-token-seal-2026-10-23/`); the two plain Longhorn copies of the hub volume already in the folder are R-1243's decision. A household box's version of this attacker (a hacked app sending or mining) is what the upload watch (R-1244, released 2026-10-24 in controller v0.342.0 + hub v0.169.0 — corrected 2026-10-24) detects |
**[DESIGN — the operator picked a code change 2026-10-25 (D44); row R-1281]:** the file manager's one
sign-in (R-1100) — FileBrowser trusts its proxy header blindly, and every app container can reach `filebrowser:80`
directly; the only guard is that the header NAME is a per-box secret, which traefik's forwardAuth strips from every
client request and sets only for a browser holding the dashboard's files pass
(`felhom-controller/controller/internal/infra/filebrowser_sso.go:15-22`, `internal/web/files_sso.go`). The name file's
creation race was fixed by review round 2 (controller `b01bb4b`). The pick: FileBrowser joins only the network traefik uses, so no app
container reaches it directly. Not built yet.
## 6. What we accept (do not re-report)
@@ -22,7 +22,7 @@ came back (BIOS „last state", inferred), its power button did (dashboard in 42
| 2 | controller `POST /api/server/poweroff` (`internal/web/poweroff_handlers.go`) | dashboard login (`RequireAuth` + CSRF), POST only, the word again, the controller's own backup/restore/app-update check, the `host_poweroff` feature gate (MinAgent 0.155.0) |
| 3 | agent `POST /host/poweroff` (`internal/localapi/hostpoweroff.go`) | `withGuest` token scope; the guards (§3); the record (`internal/poweroff`) |
| 4 | agent → hub | a host report carrying `deliberate_off.requested_at`, waited for until the hub ACCEPTED it (≤ 60 s, `Loop.SetReportedHook`) |
| 5 | agent | `pct shutdown <vmid> --timeout 120`, then `systemctl poweroff` (two exact sudoers vectors, `FELHOM_POWEROFF`) |
| 5 | agent | `pct shutdown <vmid> --timeout 120`, then `systemctl poweroff` (two exact sudoers vectors: `pct ^shutdown [0-9]+ --timeout [0-9]+$` in `FELHOM_GUESTHOOK`, and `systemctl poweroff` in `FELHOM_POWEROFF` — `felhom-agent/configs/felhom-agent.sudoers:81-90`; corrected 2026-10-24) |
| 6 | hub (`store/deliberateoff.go`) | `hosts.deliberate_off_at`; silence alarms held 14 days (§4) |
| 7 | the household | presses the machine's power button; the first report of the new boot has no `deliberate_off` → the hub clears it |
@@ -33,11 +33,11 @@ came back (BIOS „last state", inferred), its power button did (dashboard in 42
| first 10 min after boot | `too_soon_after_boot` | `TooSoonAfterBoot` |
| one accepted request per boot (file keyed by `/proc/stat` btime) | `already_requested_this_boot` | `SecondRequestSameBootRefused`, `poweroff/state_test.go` |
| any other guest or VM running (`/proc`: `lxc-start -n N`, `kvm -id N`) | `other_guests_running` | `RefusedWhileOtherGuestsRun`, `RunningGuestsFromProc` |
| heavy-operation gate held (backup, restore test, OS step, sweep, trim) | `backup_running` / `restore_test_running` / `os_update_running` / `host_busy` | `RefusedDuringBackup` |
| heavy-operation gate held (backup, restore test, OS step, sweep, trim, and since agent v0.164.0 the weekly upload line test — corrected 2026-10-24) | `backup_running` / `restore_test_running` / `os_update_running` / `host_busy` | `RefusedDuringBackup` |
| backup job, controller swap, drive format, NAS verify, escrow ceremony, Proxmox vzdump | `backup_running` / `controller_update_running` / `drive_operation_running` / `escrow_ceremony_running` | `RefusedDuringVzdump`, `FailsClosedWithoutTaskReader` |
| after acceptance: the heavy gate is TAKEN and kept; before each step the jobs and other guests are read again — a job waits ≤ 15 min, another guest cancels at once, a job still running cancels | (log) | `JobStartedAfterAcceptance`, `OtherGuestStartedAfterAcceptance` |
**„A Proxmox vzdump runs” (agent v0.156.1, R-1008):** a vzdump task for the guest in Proxmox's active list whose worker process still lives — Proxmox's own rule (the UPID's pid + pstart against `/proc/<pid>/stat`). After a power cut the dead task stays listed for ~30–60 s; it no longer counts. Doubt (an unparsable task, an unreadable `/proc`) still refuses. The same answer serves the start-up stale-lock pass, the guest-power watchdog and the controller supervisor (`stalelock_inflight_test.go`).
| after acceptance: the heavy gate is TAKEN and kept; before each step the jobs and other guests are read again — a job waits ≤ 15 min, another guest cancels at once, a job still running cancels | (log) | `JobStartedAfterAcceptance`, `OtherGuestStartedAfterAcceptance` |
**[DESIGN]** A cancelled power-off leaves the box on (gate released) — better a box that did not switch off than a
backup, update or format cut in the middle. **[DESIGN]** A failed guest shutdown does not stop the power-off: Proxmox's
@@ -86,5 +86,15 @@ alternative (an out-of-band confirmation) needs a second channel the box does no
## 6. Not done here
- The website FAQ line („to turn it off: Beállítások → Rendszer → A gép kikapcsolása") — a later website task (R-1000).
- ~~The website FAQ line („to turn it off: Beállítások → Rendszer → A gép kikapcsolása") — a later website task (R-1000).~~ Published — `website/gyik.html:128,603` (corrected 2026-10-24).
- The BIOS „after power loss" value of the N100 was not read (no monitor); the household text promises only the button.
## Decisions in code no section records [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
- **A re-enroll while the household power-off hold is active counts as „gone"** (a reinstall over the old host), not
„live" (R-1154; `hub/internal/api/reenroll.go:67-68`).
- **Setting and clearing the deliberate-off time each record an info event** — `host_deliberate_off`,
`host_deliberate_off_ended` — on the customer's event list; neither is mailed (`hub/internal/api/deliberate_off.go:12-13`).
- **The confirmation word is accepted in three forms:** „KIKAPCSOLÁS", the unaccented „KIKAPCSOLAS" (a foreign keyboard)
and „TURN OFF" (`felhom-controller/controller/internal/web/poweroff_handlers.go:28-29`).
+22 -1
View File
@@ -165,7 +165,12 @@ directory) with that tag. Facts the link code needs:
**[FACT]** An unlisted writable `${…_PATH}` bind is MANDATORY (every tier, off-site included); a plain host path is in
no capture set (`stacks/classify_binds.go` `classifyRoot`). Measured over the live templates: Transmission's unlisted
`watch` bind goes off-site; the download folders are `optional`, so a second drive gets a nightly copy of every
download (R-1019). **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's
download (R-1019). *(Fixed since: catalog `c50adfc` marks the download and watch folders `excluded` —
`app-catalog-felhom.eu/templates/transmission/.felhom.yml:46-51`; noted 2026-10-24.)* **[DESIGN — the operator picked a code change 2026-10-25 (D44); row R-1283]:** the same commit also
marks the download folders of NZBGet, SABnzbd and MeTube `excluded` (`templates/{nzbget,sabnzbd,metube}/.felhom.yml`).
NZBGet and SABnzbd stay excluded (their files can be fetched again). The pick for MeTube: `optional`, so the
second-drive copy carries it and the off-site copy does not — a MeTube video can be gone from its source site. Not
built yet. **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's
media is no single app's data; whether the library gets its own household-level backup is a separate decision.
## 8. Removal and export assume one drive per app
@@ -192,6 +197,13 @@ household's existing files under `media/torrents` stay where they are.
Evidence: `audits/media-drive-events-2026-10-14/` (K1–K7). „Idempotent" = a second run changes nothing (K2: run 2 left every entry, folder and library as it
was). „Household" = a step only the household can take.
> *Dated note 2026-10-24 (review round 2): this table is the 2026-10-14 MEASUREMENT („today" = then). As built (§11):
> on a drive event the controller only REWRITES the binds; the media apps restart when the household presses
> „Újraindítás most" or „Később" (03:30) — the file browser alone recreates itself (`internal/stacks/media_library.go`,
> `internal/web/media_events.go`); a safe removal keeps the drive REGISTERED and its return re-binds with no household
> step (`SetEjected`, `media_events.go`); and since controller `b01bb4b` (unreleased) neither restart recreates an app
> the household stopped (R-721).*
| event → app | action | measured call | idempotent | household | evidence |
|---|---|---|---|---|---|
| **added** — every media app | the compose gains the drive's bind → `up -d` recreates it (the compose CHANGED, so today's restart works) | controller compose `up -d` | yes | no | K5-1: 1–10 s per app |
@@ -258,3 +270,12 @@ save path when its client is saved before the wiring; **a Plex status probe (`/l
unclaimed Plex because Plex answers 401 for a moment while it starts — never gate on a status a starting app also
returns** (G7f).
**[DESIGN — the operator picked a code change 2026-10-25 (D44); row R-1282]:** the „Később"
03:30 media restart is an IN-MEMORY timer — a controller restart forgets it and only the notice stays
(`internal/web/media_events.go` `handleMediaRestart`, `time.AfterFunc`). The pick: the controller stores the due
time and runs the restart after a restart of its own. Not built yet.
**[DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]:** the Sonarr/Radarr API key is READ from
`/config/config.xml`; the „set at install" option §6 calls cleaner was not built (`internal/mediawire/mediawire.go:336-345`).
@@ -15,8 +15,8 @@
| Place | Found by | Identified by | Mounted | What lives there | The backup page says |
|---|---|---|---|---|---|
| Host root / PVE `local` | PVE storage `local` [FACT] A/internal/storage/observe.go:424; never a candidate [FACT] A/internal/storage/role.go:53-79 | `path:` / `store:local` [FACT] A/internal/storage/durableid.go:54-67; role system | host `/`, `/var/lib/vz` | the **local whole-guest copies** (vzdump, retention 3) [FACT] E/scripts/felhom-host-install.sh:730-734; `07` §6.1 | `backup.target.degraded` „…ugyanazon a lemezen van, mint a rendszer…" [FACT] C/internal/web/backup_target_offer.go:78-100 |
| Guest `mp0` `/var/lib/felhom` with folders `docker/` and `sys_drive/` (bound to `/var/lib/docker`, `/mnt/sys_drive`) | config default `system_data_path=/mnt/sys_drive` [FACT] C/internal/config/config.go:166 | path only; „same disk" = `st_dev` [FACT] C/internal/system/mounts_linux.go:225-245 | thin LV in `local-lvm`, `backup=1` [FACT] A/configs/build-golden.sh:105, :221-225; measured on all 3 boxes | app volumes; Tier-1 of apps with no drive [FACT] C/internal/backup/backup.go:429-438; Tier-2 state-only fallback [FACT] C/internal/backup/tier2.go:157-178; **inside every whole-guest copy** | „Csak egy másolat készül (nincs második meghajtó)…" — a Go constant, not i18n [FACT] backup.go:359-361, :1387-1410 |
| Enrolled drive (`/mnt/<name>` on the host → `/mnt/felhom-drives/<name>`) | `GET /disks/candidates`: **whole disks only** [FACT] A/internal/storage/candidates.go:33-53 | mount: `uuid:<fs-uuid>`; wipe: `byid:` wwn > nvme-eui > … [FACT] A/internal/storage/durable_device.go:32-51 | systemd unit by fs-UUID through `FELHOM_MOUNT` [FACT] A/internal/storage/mountunit.go:50-65; host bind into `mp8` | `HDD_PATH` apps, media binds [FACT] C/internal/stacks/media_library.go:24-33; Tier-1/Tier-2; optionally the whole-guest target `felhom-backup`; **not in vzdump** (`mp8` is a host bind) [FACT] `07`:477-480 | Tier-2 badges; „Nincs 2. (off-drive) másolat" [FACT] C/internal/i18n/locales/hu.json:303 |
| Guest `mp0` `/var/lib/felhom` with folders `docker/` and `sys_drive/` (bound to `/var/lib/docker`, `/mnt/sys_drive`) | config default `system_data_path=/mnt/sys_drive` [FACT] C/internal/config/config.go:166 | path only; „same disk" = `st_dev` [FACT] C/internal/system/mounts_linux.go:225-245 | thin LV in `local-lvm`, `backup=1` [FACT] A/configs/build-golden.sh:105, :221-225; measured on all 3 boxes | app volumes; Tier-1 of apps with no drive [FACT] C/internal/backup/backup.go:429-438; Tier-2 state-only fallback [FACT] C/internal/backup/tier2.go:157-178; **inside every whole-guest copy** | „Csak egy másolat készül (nincs második meghajtó)…" — *corrected 2026-10-24:* the i18n keys `note.backup.single_copy` / `note.backup.single_copy_unproven`, picked by the agent's verdict [FACT] C/internal/backup/backup.go:1485-1494 |
| Enrolled drive (`/mnt/<name>` on the host → `/mnt/felhom-drives/<name>`) | `GET /disks/candidates`: whole disks, each carrying its nested partition candidates (empty / filesystem / in use; corrected 2026-10-24) [FACT] A/internal/storage/candidates.go:84-90 | mount: `uuid:<fs-uuid>`; wipe: `byid:` wwn > nvme-eui > … [FACT] A/internal/storage/durable_device.go:32-51 | systemd unit by fs-UUID through `FELHOM_MOUNT` [FACT] A/internal/storage/mountunit.go:50-65; host bind into `mp8` | `HDD_PATH` apps, media binds [FACT] C/internal/stacks/media_library.go:24-33; Tier-1/Tier-2; optionally the whole-guest target `felhom-backup`; **not in vzdump** (`mp8` is a host bind) [FACT] `07`:477-480 | Tier-2 badges; „Nincs 2. (off-drive) másolat" [FACT] C/internal/i18n/locales/hu.json:303 |
| Network share | customer input [FACT] A/internal/localapi/netstorage.go:36-40 | none („no durable-id, no SMART") [FACT] netstorage.go:15-19 | `.automount` under `/mnt/felhom-drives/<name>` [FACT] A/internal/storage/netmount.go:18-31 | browse only; never media, never Tier-2 [FACT] media_library.go:83; tier2.go:141-143 | `err.backup.halozati_tarolo_nem_lehet_a_2` [FACT] tier2.go:37-40 |
| USB disk | same as an enrolled drive; transport never decides [FACT] C/internal/settings/settings.go:609-614 | same; the bridge may pass the real identity (N100: yes) [FACT] spike case 4a | same | same | same; SMART may be missing → „Nincs adat" |
@@ -126,7 +126,7 @@ binds both. **Creating or deleting partitions: never from the dashboard in this
## 7. The installer
**[FACT]** Today `step_grows` gives the guest's one data volume (`mp0`) 70 / 130 / 250 GiB by the thin pool's free space
(<150 / ≥150 / ≥300 GiB), and nothing above 250 (E/scripts/felhom-host-install.sh:2261-2275). Measured pools: 128 GB
(<150 / ≥150 / ≥300 GiB), and nothing above 250 (E/scripts/felhom-host-install.sh `step_grows`, now at :2475 — line corrected 2026-10-24). Measured pools: 128 GB
disk → 54 GiB, 200 GB → 113 GiB, 512 GB → 349 GiB (PVE default layout; root capped at 96 GiB).
**[DESIGN] The rule:** `mp0` = 70 GiB (pool < 300), 130 (300–800), 250 (≥ 800). Reserve = **max(16 GiB, 10 % of the
@@ -231,10 +231,25 @@ sizes ruled above in §7):
the public installer 1.33.0, agent 0.158.1 and golden 0.318.0 vouched) → data volume 100 GiB, internal place 213 GiB,
registered as the default drive with no press; Jellyfin installed with no extra step onto it; a whole-box copy holds nothing
of it; partitions formatted and used through the wizard; the honest backup offer on a VM. **Not as written:** an empty part
asks for the typed confirmation (the probe reads the parent's partition table — R-1093); a 200 GB disk gets no internal
asks for the typed confirmation (the probe reads the parent's partition table — R-1093) *(fixed since in agent v0.160.0:
the parent's PTTYPE is set aside when `blkid -p` proves the node is a partition of that scheme — `git tag --contains
f7b7308`; noted 2026-10-24)*; a 200 GB disk gets no internal
place under the ruled sizes (15 GiB < 20).
Also in build B: R-1084 (the whole-box backup offer asks the same-disk question) and R-1082 (`/mnt/sys_drive` is never
offered to attach, by device). A bundle that adds a path needs a step bundle on boxes with an older wrapper (R16, R-880):
0.158.1 was delivered through `0.158.1-step1`.
## Decisions in code no section records [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
Read at `file:line` on `main` by the main session:
- **`systemd-detect-virt --vm` is read once per agent process**; empty output becomes `unknown`, which is not `none`,
so every disk then reads unknown/vm — the fail-closed direction (`felhom-agent/internal/storage/identity.go:195-205`).
- **A path the controller cannot read counts as the SAME disk** (R-668, v0.269.0): no second copy is ever claimed on
it (`felhom-controller/controller/internal/backup/backup.go:1436-1450`).
- **The internal place („Belső tárhely") refuses eject, decommission and wipe** — the controller (`refuseInternalLifecycle`)
and the agent both — and allows moving its data to another drive (`internal/web/internal_place.go:20-26`).
- **The internal place's size subtracts EVERY thin volume already promised in the pool**, not only the 32 GiB rootfs and
the data volume (`felhom-agent/configs/felhom-internal-place:11-14`).
@@ -1,5 +1,11 @@
# `05-hub-architecture.md` — critical review (grounded against felhom-hub v0.6.3 source + Parts 01–04)
> **HISTORICAL (banner added 2026-10-24, review round 2).** A dated review of hub v0.6.3. Many of its facts have moved
> since — for example the stale threshold is 45 min by config (code default 30 min), the operator cooldown is per event
> type (`notify/dispatcher.go` `operatorCooldownFor`), and the hub→controller push was retired in v0.26.0. Read
> `05-hub-architecture.md` and `08-alarm-ladder.md` for the current behaviour; this file is kept as the record of what
> the review found then.
Method: every claim about the existing hub was checked against `felhom.eu/hub/` source; every
cross-doc claim against Parts 01/03/04. Citations are `file:line`. Severity: **blocking** (wrong /
breaks an assumption) · **should-fix** (real gap or contradiction, low blast) · **minor**.
@@ -5,6 +5,21 @@
> No matrix is designed here, no target architecture is proposed, and no gap is ranked.
>
> **This is the input to a redesign, not the redesign.**
>
> **WHAT HAS MOVED SINCE (banner added 2026-10-24, review round 2 — this inventory records 2026-07-28 and is kept as
> it was; each line below was read at `file:line` on `main` that day):**
> - The guest has ONE data volume, `mp0` at `/var/lib/felhom`, whose `docker/` and `sys_drive/` folders are bound in the
> guest; there is no `mp1` (`felhom-agent/configs/build-golden.sh:31-38,99`) — the „rootfs + mp0 + mp1" lines are history.
> - A schema-2 recovery unit (D5) carries the PORTABLE secrets; a restore takes the database password from the unit
> (R-1106) — `felhom-controller/controller/internal/backup/recovery_unit.go:24-27`, `restore_dbcreds.go`.
> - The second-drive copy's `recovery-unit/` IS read: `RestoreTier2Unit` restores the full unit from the mirror (R-102)
> — `internal/backup/tier2_restore.go:197`.
> - The off-site reconstitution replays the snapshot's named-volume tars, volumes first (R-354) —
> `internal/backup/offbox_reconstitute.go`.
> - ep0's per-household prune jobs are keep-weekly 2 at 03:30 (R-1162/R-1165, `scripts/felhom-tenantsync.sh:187-197`);
> the local whole-guest tier is keep-daily=3 (R-1149, `felhom-agent/internal/config/config.go:553-559`).
> - restic is the pinned official release, checked by version and checksum (R-1105, `felhom-controller/controller/Dockerfile:43-46`),
> no longer Debian's 0.14.0.
## 0. Method, baselines, and what "verified" means here
@@ -25,6 +25,8 @@
**Gap / roadmap (as written):** Memory-guard FIRING is not live-shown (T-RES-MEMGUARD never fired: ample RAM / auth-walled) — implemented + unit-level only
**Re-checked 2026-10-24 (catalog sweep, `audits/catalog-sweep-2026-10-24/README.md`):** a fresh box from the public ISO 1.30.0, controller 0.342.0, Tester4. Every catalog app through `POST /api/stacks/<app>/deploy` one at a time, timed to the front door (median 61 s, Cal.com 278 s), then first login, one item, backup, keep-data remove, reinstall, unit restore, remove with data. 62 installed and answered; Wanderer installed but cannot reach its database without DNS (R-739); plant-it and wger are not offered (lifecycle, by design). 52 work, 10 work with a problem (R-1269–R-1277), 0 broken. The memory guard still did not fire (one app at a time) — the gap above stands.
---
<a id="b2"></a>
@@ -379,3 +381,28 @@
**Gap / roadmap (as written):** Remaining ~39 apps are legacy-class by design (unit-only offsite)
---
## B23 — An urgent security fix reaches the boxes at night (the security-fix step)
*Map row: [B23](../00-capability-map.md#b23) · added 2026-10-23.*
**Scenario:** an app has a published hole (a CVE or GHSA id with its notice) and a fixed upstream version. CC walks the
step on scratch 9202 through the product's Update (`box_walk.py --security`: install at FROM, the guarded Update, the
four version observables at TO, a 600 s memory soak), and the only writer turns a proven walk into a `security_fix`
ladder step. Every box on controller >= 0.341.0 takes it at night with the usual guards (backup first, undo on a
failed health check, held / stopped / needs_person / W+5h respected).
**Components:** catalog `3898593` (`ladder.py check_security`, the two test-record gates, `upgrade-test.py --write-ladder
--security`, `box_walk.py --security`); controller v0.341.0 (`legCandidate`).
**Status:** **PROVEN-LIVE** on 9202 with the drill catalog, 2026-10-23.
**Evidence:** `audits/security-lane-2026-10-23/` — the negative control (TO `busybox:1.36.1`) `failed` and the writer
refused it; Plex 1.43.4.10903 → 1.43.5.11029 `proven`; on 0.340.0 the night leg logged `plex: skipped — not_proven` and
a hand press took the step; on 0.341.0-rc.1 the night leg logged `plex: security fix CVE-2026-96652,… taken` and ended
`done` after a second-drive copy and an undo copy.
**Gap:** an app with no box fixture (Plex today, R-1238) is checked only by health, version and the soak — no data is
read back. `files_may_change` is a bench measurement and reads `false` when no bench ran (decision 255 item 7).
---
@@ -219,3 +219,25 @@ Tester 1 run controller 0.311.0. An Immich image bump the repair was not proven
**Evidence:** `audits/immich-repair-2026-10-14/` — spike A on 9202 (the five candidates), the scripts on real damage
(9202), the live cut on demo-hp 9201 (D/). **Not exercised live:** a lost original (the cut lost none); pinned by unit
tests and the 9202 run.
## F13 — A hacked app is noticed (the upload and CPU watch, the weekly line test)
*Map row: [F13](../00-capability-map.md#f13) · added 2026-10-24.*
**Scenario:** an app on a household box sends far more than it says it sends, or burns CPU with no transcode to explain
it (the Plex2Shell shape). Once a minute the controller reads every app's exact counters and asks the app (qBittorrent,
Transmission, Plex); not explained for 15 / 30 minutes → one operator-only alarm, one clear when it stops. Once a week
the host's agent measures the upload line with a 10-second iperf3 upload to ep0 inside the tunnel; qBittorrent's default
upload limit is half of the tested line, never over the household's own limit.
**Components:** controller v0.342.0 (`internal/upwatch`, `agentapi.LineTest`), agent v0.164.0 (`internal/linetest`),
hub v0.169.0 (four operator-only events; the System page's „Upload line"), ep0 `felhom-iperf3.service`.
**Status:** **PROVEN-LIVE** — the alarm path on scratch 9202 with `0.342.0-rc.1` (2026-10-23: a 50 Mbit/s flood hidden
behind a real seed → one alarm after 15 min, one clear; a CPU burner in Plex → the same; a real seed → none); the line test
on the three test boxes (2026-10-24: 94.3 / 94.4 / 92.6 Mbit/s; „server busy" retried); the qBittorrent limit on
demo-felhom (5 789 696 B/s = 50 % of 92.6 Mbit/s, three channels agree).
**Evidence:** `audits/SPIKE-upload-watch-2026-10-23.md` §7; `audits/line-test-2026-10-24/`. **Not exercised live:** an
alarm reaching the operator's mailbox from a real box (the 9202 proof had no hub); the family's line on qBittorrent's
page on demo-felhom (pinned by a render test). **Gaps:** R-1250, R-1248.
@@ -97,10 +97,11 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
## Zone 2 — the internet
- **Cloudflare** (grey): DNS for the customer domains and for felhom.eu; the **tunnels**; the **geo-block** (only Hungarian visitors); flood protection. The hub holds a Cloudflare key per customer zone and checks that it reaches only that customer's own domain.
- **ep0** (grey box with an orange stripe, a Hetzner cloud server in Germany): runs the **WireGuard endpoint** and the **off-site backup server (PBS)**. Inside, one datastore with **namespaces**:
- **ep0** (grey box with an orange stripe, a Hetzner cloud server in Germany): runs the **WireGuard endpoint**, the **off-site backup server (PBS)** and, since 2026-10-24, a small **line-speed test server** (iperf3) that answers only inside the tunnel — each box measures its upload once a week (R-1247). Inside, one datastore with **namespaces**:
- one namespace **per household** (whole-guest copies, each household's own key),
with a NEUTRAL note under it (not a red gap box): "By design: each household holds the key to its own copies; the operator cannot open them without their recovery code." — *fix (c), 2026-10-09: this is a property of the design, not a gap, and a red box said the opposite of what it means.*
- the **`operator`** namespace: two groups — **"hub database"** and **"DooPlex copy"** (code, password manager, secrets, signing keys).
- outside PBS, a plain folder: the **release copy** (images + agent packages, D33).
Prune jobs on ep0: households keep one copy per week for 2 weeks (`keep-weekly 2`, R-1162, 2026-10-19; every new household's job is made with its namespace, R-1165); `operator` keeps 14 daily + 8 weekly.
Mark ep0 with a small shield: "**Protected** — holds the only off-premises copy of real household data."
- **Hetzner Storage Box** (grey): the shared box for the households' per-app restic copies (Tier 3), one sub-account per household, add-only keys.
@@ -121,8 +122,9 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
Arrows from DooPlex (dashed, orange → ep0):
- **00:20 "DooPlex copy"**: the code (614 MB), Gitea's database and settings, the **password manager**, the nightly secrets export, the **signing keys**. Encrypted on DooPlex with the **DooPlex off-site key**. The key that writes it **cannot delete** old copies. Tested back every Sunday.
- **02:30 "Hub database"**: encrypted with the **hub-DB key**. Tested back every Sunday at 04:30.
- **Left out on purpose:** the container registry (27.7 GB) — "rebuilt from the code".
- **02:30, 08:30, 14:30, 20:30 "Hub database"** (every 6 hours since 2026-10-23, D34): encrypted with the **hub-DB key**. Tested back every Sunday at 04:30.
- **01:40 "Release copy"** (since 2026-10-23, D33): the newest 3 released images of the controller, hub and samba, and the newest 5 agent packages — not encrypted (nothing secret), into a plain folder on ep0 (`/srv/felhom-release-copy`, its own limited user, outside PBS), about 0.7 GB, capped at 3 GB.
- **Left out on purpose:** the rest of the container registry (older tags, test builds, non-Felhom images) — "rebuilt from the code".
Arrow ep0 → DooPlex (dashed, 05:00): "**ep0-copy** pull: a second copy of ep0 at home. Keeps 8 weekly copies. A daily job removes a deleted customer's copy within 30 days."
**The test machines** (small, plain tiles near DooPlex, labelled "disposable"):
@@ -145,7 +147,7 @@ Title: **"Who tells a box what to do"**
## Bottom strip — "One night, hour by hour"
A horizontal timeline from **00:00 to 09:00**, two rows:
- **DooPlex row:** 00:00 database dumps · **00:20 DooPlex copy → ep0** · **02:30 hub database → ep0** · 03:30 ep0 prunes households · 03:45 ep0 prunes `operator` · **05:00 ep0-copy pull → DooPlex** · 07:30 ep0-copy prune · 08:00 deleted-customer clean-up. Sundays: 04:30 restore tests.
- **DooPlex row:** 00:00 database dumps · **00:20 DooPlex copy → ep0** · **02:30 hub database → ep0** (and again at 08:30; every 6 h since D34 — corrected 2026-10-24) · 03:30 ep0 prunes households · 03:45 ep0 prunes `operator` · **05:00 ep0-copy pull → DooPlex** · 07:30 ep0-copy prune · 08:00 deleted-customer clean-up. Sundays: 04:30 restore tests.
- **Customer box row:** **02:30 Tier 1** · **03:30 Tier 2** · **04:15 Tier 3 off-site** → app updates → **04:30–08:30 whole-guest backup** (local daily, off-site weekly) → Debian fixes → (when approved) Proxmox, Docker, kernel restart. 05:00 the hub checks every box made its backup.
- A small note: "Household mails about a kernel restart go out the day before, 09:00–20:00."
@@ -116,7 +116,7 @@ code{font-family: ui-monospace, "SFMono-Regular", Menlo, monospace; font-size: 1
<div style="position: absolute; left: 1552px; top: 460px; width: 136px; height: 112px; box-sizing: border-box; padding: 8px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #5b6270; border-radius: 6px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>iso.felhom.eu</b><br>households download the installer</span></div>
<div style="position: absolute; left: 1552px; top: 580px; width: 136px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: center;">web pages</div>
<div style="position: absolute; left: 1272px; top: 738px; width: 400px; font-size: 17px; line-height: 26px; font-weight: 400; color: #051343; text-align: left;"><b style="font-size: 22px">ep0</b> — Hetzner cloud server, Germany</div>
<div style="position: absolute; left: 1272px; top: 766px; width: 400px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;">WireGuard endpoint + off-site backup server (PBS). One datastore, split into namespaces:</div>
<div style="position: absolute; left: 1272px; top: 766px; width: 400px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;">WireGuard endpoint + off-site backup server (PBS) + a tunnel-only line-speed test (iperf3). One datastore, split into namespaces:</div>
<div style="position: absolute; left: 1272px; top: 818px; width: 192px; height: 150px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #0a3d73; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><span style="display: inline-block; min-width: 26px; padding: 0 6px; box-sizing: border-box; height: 26px; border-radius: 13px; background: #0a3d73; color: #ffffff; font-weight: 800; text-align: center; line-height: 26px; font-size: 16px; vertical-align: middle">G</span> <b>One namespace per household</b><br>whole-guest copies, each household's own key<br><b>keeps one per week, 2 weeks</b></span></div>
<div style="position: absolute; left: 1480px; top: 818px; width: 192px; height: 150px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b><code style="font-size: 16px">operator</code> namespace</b><br>groups “hub database” and “DooPlex copy”<br><b>keeps 14 daily + 8 weekly</b></span></div>
<div style="position: absolute; left: 1272px; top: 974px; width: 400px; font-size: 15px; line-height: 19px; font-weight: 400; color: #4a5670; text-align: left;"><b>By design:</b> each household holds the key to its own copies; the operator cannot open them without the household’s recovery code.</div>
@@ -133,7 +133,7 @@ code{font-family: ui-monospace, "SFMono-Regular", Menlo, monospace; font-size: 1
<div style="position: absolute; left: 1720px; top: 706px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: center;">The hub never connects into a box.</div>
<div style="position: absolute; left: 1720px; top: 784px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">00:20 DooPlex copy</div>
<div style="position: absolute; left: 1720px; top: 824px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #a85400; text-align: center;">DooPlex off-site key</div>
<div style="position: absolute; left: 1720px; top: 872px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">02:30 hub database</div>
<div style="position: absolute; left: 1720px; top: 872px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">every 6 h: hub database</div>
<div style="position: absolute; left: 1720px; top: 912px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #a85400; text-align: center;">hub-DB key</div>
<div style="position: absolute; left: 1720px; top: 962px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #0a3d73; text-align: center;">05:00 ep0-copy pull</div>
<div style="position: absolute; left: 1720px; top: 1002px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #0a3d73; text-align: center;">ciphertext only</div>
@@ -191,7 +191,7 @@ code{font-family: ui-monospace, "SFMono-Regular", Menlo, monospace; font-size: 1
<div style="position: absolute; left: 374px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #a85400; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 378px; top: 2040px; width: 220px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>00:20</b> DooPlex copy → ep0</div>
<div style="position: absolute; left: 894px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #a85400; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 898px; top: 2000px; width: 230px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>02:30</b> hub database → ep0</div>
<div style="position: absolute; left: 898px; top: 2000px; width: 230px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>02:30</b> hub DB → ep0, 6-hourly</div>
<div style="position: absolute; left: 1134px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #0a3d73; border: 2px solid #ffffff"></div>
<div style="position: absolute; left: 1138px; top: 2040px; width: 236px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>03:30</b> ep0 prunes households</div>
<div style="position: absolute; left: 1194px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #0a3d73; border: 2px solid #ffffff"></div>
@@ -0,0 +1,173 @@
# SPIKE — who is using the upload line, and does the app explain it? (2026-10-23 night)
Operator ruling 2026-10-23 19:22 (`CONTEXT.md`): measure the line; on a sustained high upload attribute it to an app
and ask the app; a mismatch is the alarm; the same shape for CPU; qBittorrent gets a default upload limit from the
measured line, household changes respected. Brief Part 2.1. Design written from these numbers: §6 here and
`architecture/08-alarm-ladder.md` §9 [DESIGN — proposed]. Evidence and tools: `SPIKE-upload-watch-2026-10-23/`.
Read first: `08-alarm-ladder.md` (R-388, §8.1), `07-backup-architecture.md` (the night chain: W = 02:30 local, the
off-site push after it, the proof at 03:30 UTC), `15-media-library.md`, `13-threat-model.md` (attacker j),
`audits/plex2shell-felhom-impact-2026-10-23/README.md` (the 1.5 TB Plex upload). Controller: `internal/metrics/`
(`container_metrics` — one row per container per minute, 30 days), `internal/monitor/healthcheck.go` (the instant
whole-box CPU warning), `internal/mediawire/` (the app clients and their credentials, `09` §3 decision 248).
Machines: the three test boxes READ ONLY (their stored metrics; their off-site record; tonight a one-second read of
the controller's own counter during the real push); scratch 9202 on demo-hp (throwaway qBittorrent, Transmission, Plex,
Jellyfin through the product's deploy); a throwaway receiver LXC **9471 `cc-uw-recv`** on demo-hp's scratch storage
(never the thin pool). Every flood went 9202 → 9471 on demo-hp's own bridge; nothing went toward the internet.
## 1. Normal levels (stored metrics, three boxes)
History: **demo-hp** 2026-09-23 02:00 → 10-23 18:09 UTC (30.7 d, 998 216 rows); **demo-felhom** (the N100) same span
(30.7 d, 268 120 rows); **Tester 1** 2026-10-04 19:41 → 10-23 18:08 (18.9 d, 294 433 rows). Per app: upload from the
`net_tx_bytes` deltas (one-minute samples), CPU in percent of ONE core. Full tables: `s1-levels-per-app.txt`.
| box | the busiest uploaders (p50 / p95 / max Mbit/s) | the busiest CPU (p95 / max % of a core) |
|---|---|---|
| demo-hp | qbittorrent 0.02 / 1.2 / 4.0; immich-server 0.03 / 0.13 / 243 (one minute); felhom-controller 0 / 0.003 / 17.8 (the push) | grimmory 272 / 305 (a scan, 94 samples); immich-ML 32 / 548; romm 3 / 442; paperless 0.3 / 404 |
| demo-felhom | immich-server 0.67 / 5.95 / 12.9 (32 samples); jellyfin 0 / 0.85 / 2.9 | immich-ML 295 / 296 (an import); onlyoffice 131 / 148; paperless 137 / 140 |
| Tester 1 | qbittorrent 0.007 / 0.13 / 42.4; transmission 0.001 / 0.013 / 57.4; controller 0 / 0.003 / 37.5 | paperless 0.7 / 508; controller 46 / 423; plex 4.3 / 198 |
**How long a normal high lasts** (`s1-longest-runs.txt` — the longest run of consecutive minutes): upload ≥ 5 Mbit/s:
**3 min** at most (qBittorrent, Tester 1); ≥ 1 Mbit/s: 9 min. CPU ≥ 150 %: **7 min** at most (Immich machine learning);
≥ 80 %: 13 min (Kimai). Busiest upload hours (UTC): demo-hp 03h, 02h; demo-felhom 16h, 05h; Tester 1 07h, 18h — no fixed
pattern beyond the night push. For scale: DooPlex's hacked Plex sent ~474 GB in one day (≈ 44 Mbit/s on average, for
hours) and used ~10 cores.
`felhom-samba` runs with the host network: its counter is always 0 (no own network namespace) — the watch cannot see it.
## 2. The line without a test
**What the box already sees:** the off-site push (restic, inside the controller container) is the box's only big
upload of its own. At the stored one-minute sample (`s2-controller-push-per-night.txt`): the controller's best five
minutes per night exceeded 2 Mbit/s on **2 of 31 nights** on demo-hp (5.1 and 6.9 Mbit/s; best single minute 17.0),
**1 of 19** on Tester 1 (7.6; best minute 37.5) and **0 of 31** on demo-felhom. The off-site record keeps the push's
duration (33 s on demo-felhom, 1 m 8 s on Tester 1, 3 m 19 s on demo-hp on 2026-10-23) and **no byte count**.
**Conclusion:** at a one-minute sample the push shows the night's DATA SIZE (a few MB, incremental), not the line. Not
enough as it stands. Two changes make it usable: sample once a SECOND during the push (the controller knows when its
push runs and can read its own counter), and let a night count only when its best 10 seconds moved ≥ 10 MB. Even then
it is a **lower bound** (restic over SFTP to the Storage Box may not fill the line; the family uploads too).
**Tonight's one-second read of the real push** (2026-10-24 ~02:16Z, three boxes, read only): §2.1 below.
### 2.1 One-second samples of the real push
Read 2026-10-24 02:08–02:30Z, once a second, the controller container's own transmit counter (the counter the watch
reads: `eth0` in the controller's network namespace), on the three test boxes, read only
(`SPIKE-upload-watch-2026-10-23/night-push-1s-<box>.txt`; the reader `tools/nightsample.sh`).
| Box | Sent in the 22 min | Best 10 s | 1-second peak | Counts (≥ 10 MiB in the best 10 s)? |
|---|---|---|---|---|
| demo-hp | 19.3 MB | **12.6 MB in 10.1 s = 10.0 Mbit/s** (02:17:32Z) | 70.3 Mbit/s | **yes** → line „at least 10 Mbit/s" |
| demo-felhom | 3.3 MB | 3.2 MB = 2.5 Mbit/s (02:15:22Z) | 13.7 Mbit/s | no |
| Tester 1 | 3.6 MB | 3.5 MB = 2.7 Mbit/s (02:15:40Z) | 14.7 Mbit/s | no |
**What it says:** one night in three counted. The push is a burst of a few seconds (3–9 seconds above 100 kB/s), so
even the counting night gives a low floor: 10 Mbit/s, while one second of the same push reached 70 Mbit/s. The design
keeps the best counting night of the last 14, so the line rises only when a bigger night comes. **Consequence for the
qBittorrent limit (R-1245):** 50 % of tonight's floor is ~5 Mbit/s on demo-hp — likely far below its real line. This is
the known lower-bound limit (R-1247), now measured; it is written into R-1245 so the operator decides with it.
## 3. The line with a test — NOT run tonight
| Candidate | Ours? | Terms | Cost per test | Tonight |
|---|---|---|---|---|
| ep0 (a receiver on our Hetzner box) | yes | ours | 10 s × line ≈ 125 MB at 100 Mbit/s | **not built** — a change on ep0 is forbidden tonight |
| the box's own Storage Box sub-account, through the same transport as the push | yes (Hetzner, unlimited traffic) | Hetzner's — ours | same | not tried — a test file in the account that holds the restic repository needs its own design (the repo check, the quota) |
| Cloudflare R2 (where the ISO lives) | yes | ours | same + R2 class-A writes | not used — its write key is the operator's ISO-publish key; a test object in a public bucket is wrong |
| public speed tests (speed.cloudflare.com, LibreSpeed servers) | no | **not read** | — | not used (the brief: no third-party test without its terms) |
A quiet hour: never 02:30–05:30 local (dumps 02:30, second drive, off-site push, the 03:30 UTC proof). The family shares
the line, so take the highest of several runs. **Not needed for the first build** — §6 works with the push's lower
bound and an absolute floor; an active test is a decide row (R-1247).
## 4. Can the box ask each app? (measured on 9202 with throwaway installs; a private torrent 9202 → 9471)
Set-up: qBittorrent 5.2.4 and Transmission 4.1.3 installed through the product (`POST /api/stacks/<n>/deploy`; the
first try was refused BEFORE anything was made — the data place must be the drive itself; two empty helper folders
removed). The same 2 GiB file (an AES-CTR keystream of a fixed key) on both sides; a private torrent; a 40-line HTTP
tracker on 9471 (`tools/tracker.py`); `transmission-daemon` on 9471 as the leecher. The admin passwords were read from
the containers at use and never printed.
| App | Its own number | Measured | Limit | Household's limit visible? |
|---|---|---|---|---|
| **qBittorrent** | `/api/v2/transfer/info` → `up_info_data` (cumulative), `up_info_speed` | seeding: app **131.1 Mbit/s** vs container **134.6–140.5** → **×1.03–1.07** (`s4-qb-seed-nolimit.jsonl`) | `transfer/setUploadLimit` (bytes/s) = `preferences.up_limit`; stored in KiB/s (2 500 000 → **2 499 584**); survives a restart (`Session\GlobalUPSpeedLimit=2441`); measured: ~20 Mbit/s held (`s4-qb-seed-limit20.jsonl`) | yes: `transfer/uploadLimit` ≠ the value the box stored |
| **Transmission** | RPC `session-stats.current-stats.uploadedBytes`, `uploadSpeed` | seeding: app 487.1 / 390.3 / 115.8 vs container 487.6 / 390.7 / 117.9 Mbit/s → **×1.001–1.02** (`s4-tr-seed-nolimit.jsonl`) | `session-set speed-limit-up` (kB/s) + `-enabled` (read back); **its rate under the limit was NOT observed** — the re-added leecher got no connection in 2 minutes | yes: `speed-limit-up-enabled` + value |
| **Plex** | `/status/sessions`, `/transcode/sessions` (`MediaContainer.size`) | unclaimed 1.43.5: both 200 without a token from the docker network, `{"size":0}`; a claimed server needs its PlexOnlineToken (read at use from Preferences.xml, as mediawire does — no new secret) | — | — |
| **Jellyfin** | `/Sessions` | **401** without a key; the key named `felhom` exists only when the household made it (mediawire's „one household step") — the box cannot create one at install without the admin login it does not have | — | — |
| Nextcloud, Immich | **no upload-explaining number found** (not measured further); Immich's job queue needs an API key the box does not hold | — | — | → the plain rule (§6) |
**The fake hacked app** (code inside qBittorrent's own container, `docker exec … python3` → 9471:5001, Plex2Shell's
shape): the container sent **~10 100 Mbit/s** (LAN, one bridge) while qBittorrent reported **0** (`s4-qb-flood.jsonl`).
**The counter itself:** the Docker Engine API's one-shot stats (`networks.*.tx_bytes`) equal the kernel's counter in
the container's network namespace to the byte; `docker stats`' text (what `internal/metrics` stores) has three
significant digits — ~10 MB steps at a GB total, ~1.3 Mbit/s at a one-minute sample. The watch reads the Engine API.
**A limit the hacked app controls:** an attacker with code in the app's container can also answer its API. The check
catches an attacker who does not know about it; one who does could lie. §5 is the answer to that.
## 5. A universal upload ceiling — spike only
A helper container sharing the app's network namespace (`docker run --rm --network container:qbittorrent --cap-add
NET_ADMIN alpine:3.20` + `iproute2-tc`) added `tc qdisc … root tbf rate 20mbit burst 64kb latency 400ms` on the app's
`eth0` **inside the unprivileged 9202 guest**. The same flood then ran at **19.9–20.1 Mbit/s** (`s4-qb-flood-tbf.jsonl`),
qBittorrent's API kept answering, and `tc qdisc del` restored it. Costs: a helper image with `tc` (a new external image,
or `tc` shipped in the controller image), `NET_ADMIN` for that helper, re-applied after every app restart (the qdisc
dies with the namespace), upload only. **Not built** (brief). It would cap a hacked app's flood whatever its API says.
## 6. The design (numbers from §1–§5; full text `08-alarm-ladder.md` §9)
1. **Line:** one-second samples of the controller's own sent bytes during the off-site push; the best ≥ 10-second rate
per night; a night counts when that window moved ≥ 10 MB; the line = the highest counting night of the last 14;
settings.json `upwatch.line`; shown on the system settings page as „legalább N Mbit/s" („még mérjük" until known).
2. **Trigger:** an app (its containers summed, exact counters, once a minute) sends ≥ **max(5 Mbit/s, 25 % of the line)**
for **15 minutes** — five times the longest normal run at ≥ 5 Mbit/s (3 min). Per app, not per box: the box cannot
tell LAN from internet bytes inside its controller container, and the per-app view is what the check compares.
3. **Attribution:** the app's compose project; infrastructure (controller, traefik, cloudflared, Samba) never judged.
4. **Explanation:** qBittorrent / Transmission: explained when container bytes ≤ 1.25 × the app's own count + 1 Mbit/s
(measured overhead 1–7 %); Plex: any viewer explains the upload (session bandwidth not measured — the conservative
half; no viewer + a big upload = the Plex2Shell signature). Not explained → **`app_unexplained_upload`**,
operator-only, one per episode, a clear after 15 quiet minutes. No explainer: alarm only at ≥ 10 × its own p95 for
60 minutes, after ~20 h of history, worded „no app number to compare".
5. **CPU:** ≥ **max(150 %, 50 % × cores)** of one core for **30 minutes** (four times the longest normal run at ≥ 150 %),
explained by a Plex transcode or the box's own work on the app (backup, update, deploy) → else
**`app_unexplained_cpu`**. No explainer: ≥ max(200 %, 75 % × cores) for 60 minutes.
6. **qBittorrent limit:** 50 % of the line (CC's pick — R-1245), every 15 minutes, re-set when the line moves > 20 %;
never over a household's limit (a limit there first, or any change from the stored value). No line → nothing set.
The household's line (HU + EN in the report).
7. **Edge cases:** the night push is the controller's (never judged); the first day: apps with an explainer are judged
on the floor (a mismatch is evidence whatever the line), apps without one are not judged before ~20 h of history; a
slow or changing line (a Wi-Fi laptop): the line is a max over 14 nights — a falling line lowers nothing until it
rolls out (R-1247 notes the active test); two apps at once: each judged on its own bytes; the household's own devices:
invisible to the box — the trigger uses the box's own bytes only; an app reinstalled from scratch reads as „the
household removed the limit" and is left alone.
## 7. Live proofs on 9202 (controller `0.342.0-rc.1`, commit `a9c1997`, CI run 2270 success)
Set by hand in 9202's `/etc/felhom-controller-image` (an `-rc` tag is never offered to a household); 9202's hub link
is off by design, so every event was handed to the notifier and DROPPED there — the hub side (allow-list, operator-only,
the mail line) is proven by hub 0.169.0's tests. 9202's off-site is off, so one line record (40 Mbit/s) was seeded by
hand into its settings.json. Logs: `SPIKE-upload-watch-2026-10-23/live/2-9202-watch-log-full.txt`.
| # | What | Result (UTC) |
|---|---|---|
| 1 | qBittorrent unlimited, line 40 Mbit/s → the box sets the limit | **19:47:42** „qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)" — read back from qBittorrent: 2 499 584 |
| 2 | **a real torrent seeding at the limit** (5 GB, private, 9202 → 9471) | **20:04:42** „qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s)"; no alarm (counter ratio 1.04) |
| 3 | **a CPU burner in Plex with no transcode** (4 busy loops, 400 % on a 7-core guest; trigger 350 %) | **20:07:42** „plex used 399% CPU … for 30 min with no work that explains it — operator alarm" → `app_unexplained_cpu`; ONE |
| 4 | **the fake hacked app**: a 50 Mbit/s flood from inside qBittorrent's container to 9471, hidden behind the real seed | **20:19:42** „qbittorrent sent 65.0 Mbit/s for 15 min — NOT explained by the app (qbittorrent: 20.0 Mbit/s) — operator alarm" → `app_unexplained_upload`; ONE, none more while it ran |
| 5 | the burner stopped 20:11 | **20:25:41** „plex: the unexplained CPU stopped … — cleared" → `app_unexplained_cpu_cleared`; ONE |
| 6 | the household changes qBittorrent's limit by hand (1 500 000 → stored 1 499 136) at 20:21 | **20:32:42** „qBittorrent's upload limit is the household's — the box leaves it alone from now on"; read back at 20:43: still 1 499 136 |
| 7 | the flood stopped 20:26 (the seed went on, explained) | **20:41:41** „qbittorrent: the unexplained upload stopped … — cleared" → `app_unexplained_upload_cleared`; ONE |
| 8 | the system settings page (`GET /settings`) | „Feltöltési sebesség: legalább 40 Mbit/s (az éjszakai távoli mentésből mérve)"; qBittorrent's page carried no limit line after step 6 (the household's) — the box-owned branch is pinned by `TestUpwatch_QBLimitLineOnQBittorrentsPageOnly`, not captured live |
**Found live:** the very first minute after a start failed to reach the apps by name (the bootstrap joins the controller
to the app network a few seconds after the start); every later minute answered. That first minute only sets baselines —
**NOT-A-FINDING**. The media wiring's generated `docker-compose.override.yml` stays beside the template after a removal
(Jellyfin tonight; Plex's from the morning session) — rewritten before every start, harmless — **NOT-A-FINDING**.
**Teardown, three layers:** machine — the four throwaway apps removed through the product (containers and volumes gone,
verified), my test files and probes deleted, the `alpine:3.20` helper image removed, 9202 back on 0.341.0 with the seeded
record removed, the rc image removed; host — the receiver LXC 9471 destroyed (`--purge`, its disk gone); hub — none (9202
is not on the hub). The night sampler on the three test boxes is reads only.
@@ -0,0 +1,15 @@
gitea.dooplex.hu/admin/felhom-controller:0.342.0-rc.1
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0-rc.1 Up 6 minutes (healthy)
filebrowser gtstef/filebrowser:1.5.6-stable Up 4 hours (healthy)
immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 Up 4 hours (healthy)
immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 Up 4 hours (healthy)
immich-redis redis:7-alpine Up 4 hours (healthy)
immich-server ghcr.io/immich-app/immich-server:v3.2.4 Up 4 hours (healthy)
jellyfin jellyfin/jellyfin:10.11.11 Up About an hour (healthy)
paperless-postgres postgres:18-alpine Up 4 hours (healthy)
paperless-redis redis:7-alpine Up 4 hours (healthy)
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up 4 hours (healthy)
plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0 Up About an hour (healthy)
qbittorrent qbittorrentofficial/qbittorrent-nox:5.2.4-1 Up About an hour (healthy)
traefik traefik:v3.7.13 Up 4 hours
transmission lscr.io/linuxserver/transmission:version-4.1.3-r0 Up About an hour (healthy)
@@ -0,0 +1,10 @@
VMID Status Lock Name
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
9471 running cc-uw-recv
description: CC throwaway 2026-10-23 night%3A upload-watch spike receiver/leecher. Remove after the session.%0A
hostname: cc-uw-recv
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:04:68:80,ip=dhcp,type=veth
rootfs: nvme-scratch:9471/vm-9471-disk-0.raw,size=8G
tags: cc;scratch;uw-spike
@@ -0,0 +1,9 @@
2026/10/23 19:32:41 upwatch_wiring.go:97: [INFO] [upwatch] upload and CPU watch started (line at least 40 Mbit/s, qBittorrent share 50 %)
2026/10/23 19:32:41 watch.go:237: [DEBUG] [upwatch] plex: the app did not answer (resolving the plex container: lookup plex on 192.168.0.250:53: no such host) — treated as no number this minute
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] transmission: the app did not answer (Post "http://transmission:9091/transmission/rpc": dial tcp: lookup transmission on 192.168.0.250:53: no such host) — treated as no number this minute
2026/10/23 19:32:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_definition_refused (severity warning) — further app_definition_refused events are logged at DEBUG only
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] qbittorrent: the app did not answer (Post "http://qbittorrent:8080/api/v2/auth/login": dial tcp: lookup qbittorrent on 192.168.0.250:53: no such host) — treated as no number this minute
2026/10/23 19:47:42 service.go:108: [INFO] [upwatch] qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)
2026/10/23 20:04:42 watch.go:303: [INFO] [upwatch] qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s, 0 viewer(s)); normal use
2026/10/23 20:07:42 watch.go:343: [WARN] [upwatch] plex used 399% CPU (of one core; 7 cores) for 30 min with no work that explains it (plex transcodes) — operator alarm
2026/10/23 20:07:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_cpu (severity warning) — further app_unexplained_cpu events are logged at DEBUG only
@@ -0,0 +1,16 @@
2026/10/23 19:32:41 upwatch_wiring.go:97: [INFO] [upwatch] upload and CPU watch started (line at least 40 Mbit/s, qBittorrent share 50 %)
2026/10/23 19:32:41 watch.go:237: [DEBUG] [upwatch] plex: the app did not answer (resolving the plex container: lookup plex on 192.168.0.250:53: no such host) — treated as no number this minute
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] transmission: the app did not answer (Post "http://transmission:9091/transmission/rpc": dial tcp: lookup transmission on 192.168.0.250:53: no such host) — treated as no number this minute
2026/10/23 19:32:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_definition_refused (severity warning) — further app_definition_refused events are logged at DEBUG only
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] qbittorrent: the app did not answer (Post "http://qbittorrent:8080/api/v2/auth/login": dial tcp: lookup qbittorrent on 192.168.0.250:53: no such host) — treated as no number this minute
2026/10/23 19:47:42 service.go:108: [INFO] [upwatch] qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)
2026/10/23 20:04:42 watch.go:303: [INFO] [upwatch] qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s, 0 viewer(s)); normal use
2026/10/23 20:07:42 watch.go:343: [WARN] [upwatch] plex used 399% CPU (of one core; 7 cores) for 30 min with no work that explains it (plex transcodes) — operator alarm
2026/10/23 20:07:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_cpu (severity warning) — further app_unexplained_cpu events are logged at DEBUG only
2026/10/23 20:19:42 watch.go:307: [WARN] [upwatch] qbittorrent sent 65.0 Mbit/s for 15 min — NOT explained by the app (qbittorrent: 20.0 Mbit/s, 0 viewer(s)) — operator alarm
2026/10/23 20:19:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_upload (severity warning) — further app_unexplained_upload events are logged at DEBUG only
2026/10/23 20:25:41 watch.go:378: [INFO] [upwatch] plex: the unexplained CPU stopped (15 min below the trigger or explained) — cleared
2026/10/23 20:25:41 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_cpu_cleared (severity info) — further app_unexplained_cpu_cleared events are logged at DEBUG only
2026/10/23 20:32:42 service.go:111: [INFO] [upwatch] qBittorrent's upload limit is the household's — the box leaves it alone from now on
2026/10/23 20:41:41 watch.go:370: [INFO] [upwatch] qbittorrent: the unexplained upload stopped (15 min below the trigger or explained) — cleared
2026/10/23 20:41:41 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_upload_cleared (severity info) — further app_unexplained_upload_cleared events are logged at DEBUG only
@@ -0,0 +1,2 @@
/settings 56743 data-uplink> <span class="settings-label">Feltöltési sebesség</span> <span class="settings-value">legalább <span class="mono">40</span> Mbit/s (az éjszakai távoli mentésből mérve)</span> </div> | qb line: none
/apps/qbittorrent 49281 no uplink row | qb line: none
@@ -0,0 +1,4 @@
version v5.2.4 webapi 2.15.1
transfer/info {'connection_status': 'connected', 'dht_nodes': 375, 'dl_info_data': 0, 'dl_info_speed': 0, 'dl_rate_limit': 0, 'last_external_address_v4': '37.191.56.193', 'last_external_address_v6': '', 'up_info_data': 5566595787, 'up_info_speed': 0, 'up_rate_limit': 1499136}
prefs {'up_limit': 1499136, 'dl_limit': 0, 'alt_up_limit': 10240, 'alt_dl_limit': 10240, 'scheduler_enabled': False, 'limit_tcp_overhead': False, 'limit_lan_peers': True, 'limit_utp_rate': True, 'max_uploads': 20, 'enable_upload_suggestions': False}
uploadLimit 1499136 speedLimitsMode 0
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,186 @@
=== demo-hp: 2026-09-23 02:00 -> 2026-10-23 18:09 UTC (30.7 d), 998216 rows
container n tx p50 p95 max Mbit/s | cpu p50 p95 max %core | notes
qbittorrent 18 0.019 1.199 4.01 | 0.4 5.4 11.3 |
metube 3 0.146 0.146 0.15 | 0.0 0.1 0.1 |
immich-server 318 0.031 0.133 242.93 | 0.1 5.6 138.1 | resets=7
grimmory-db 102 0.000 0.116 0.35 | 0.0 16.9 102.5 | resets=12
immich-postgres 320 0.000 0.066 2.20 | 0.0 8.7 70.2 | resets=7
grimmory 94 0.001 0.047 0.77 | 1.1 272.0 305.0 | resets=9
immich-redis 320 0.013 0.029 0.18 | 0.6 4.3 5.1 | resets=7
romm 26492 0.019 0.027 0.20 | 0.7 3.2 442.4 | resets=97
docmost 43499 0.013 0.027 0.23 | 0.1 0.8 158.1 | resets=147
rallly 71 0.000 0.017 0.12 | 0.0 8.5 152.1 | resets=10
radarr 16 0.002 0.015 0.03 | 0.2 7.2 11.8 |
romm-redis 26523 0.005 0.013 0.10 | 0.7 4.4 61.5 | resets=101
paperless-webserver 43488 0.005 0.013 0.04 | 0.1 0.3 404.2 | resets=144
nextcloud-db 663 0.001 0.013 1.74 | 0.0 6.1 71.8 | resets=11
docmost-redis 43532 0.005 0.013 0.13 | 0.7 4.4 6.6 | resets=147
cloudflared 43604 0.000 0.013 1.21 | 0.3 0.5 13.8 | resets=6
rallly-postgres 71 0.000 0.007 0.01 | 0.0 5.0 8.8 | resets=7
prowlarr 5 0.002 0.004 0.00 | 0.1 0.2 0.2 |
sonarr 18 0.000 0.004 0.01 | 2.5 8.2 8.8 |
felhom-controller 43608 0.000 0.003 17.79 | 1.0 57.9 251.4 | resets=133
paperless-redis 43521 0.001 0.003 0.02 | 0.6 4.3 6.1 | resets=148
bentopdf 43567 0.000 0.003 0.01 | 0.0 0.0 5.2 | resets=58
homepage 78 0.000 0.002 0.01 | 0.0 4.2 4.5 | resets=1
bookstack-db 43565 0.001 0.001 0.01 | 0.0 6.1 102.8 | resets=147
romm-db 26526 0.000 0.001 0.19 | 0.0 6.0 105.3 | resets=89
plex 11974 0.001 0.001 1.10 | 0.3 0.6 124.6 | resets=46
nextcloud 662 0.000 0.001 0.80 | 0.0 5.7 150.0 | resets=11
immich-machine-learning 320 0.000 0.001 0.39 | 0.2 32.0 548.0 | resets=7
docmost-postgres 43533 0.000 0.001 0.02 | 0.0 4.7 29.6 | resets=147
bookstack 43546 0.000 0.001 0.01 | 0.0 0.0 63.0 | resets=148
adventurelog-postgres 43567 0.000 0.001 1.35 | 0.0 4.7 37.9 | resets=138
adventurelog 43549 0.000 0.001 2.16 | 0.0 0.7 204.3 | resets=146
ghost 20 0.000 0.001 0.00 | 0.0 6.8 132.9 | resets=2
glance 20 0.000 0.001 0.00 | 0.0 0.0 0.0 | resets=2
privatebin 43518 0.000 0.001 0.00 | 0.1 0.1 8.1 | resets=140
paperless-postgres 43520 0.000 0.001 0.05 | 0.0 4.7 19.1 | resets=148
kimai 30136 0.000 0.000 0.00 | 0.0 0.0 107.2 | resets=102
home-assistant 14403 0.000 0.000 1.00 | 0.0 0.5 106.9 | resets=62
kimai-db 30150 0.000 0.000 0.05 | 0.0 5.9 100.8 | resets=96
adventurelog-frontend 43559 0.000 0.000 0.21 | 0.0 3.8 22.8 | resets=126
vaultwarden 329 0.000 0.000 0.00 | 0.0 0.1 4.0 | resets=7
calibre-web 43547 0.000 0.000 0.01 | 0.0 0.3 127.1 | resets=145
opengist 43530 0.000 0.000 0.01 | 0.0 0.0 9.8 | resets=140
jellyfin 297 0.000 0.000 5.87 | 0.0 4.4 160.5 | resets=7
vikunja 8 0.000 0.000 0.00 | 0.0 0.0 0.0 | resets=2
a1-echo 113 0.000 0.000 0.00 | 0.0 0.0 0.0 |
upbeat_perlman 1 0.000 0.000 0.00 | 31.2 31.2 31.2 |
traefik 43608 0.000 0.000 1.21 | 0.1 3.5 92.1 | resets=11
practical_johnson 1 0.000 0.000 0.00 | 9.9 9.9 9.9 |
peaceful_ptolemy 1 0.000 0.000 0.00 | 14.1 14.1 14.1 |
nextcloud-redis 663 0.000 0.000 0.03 | 0.6 4.1 7.9 | resets=5
friendly_wilbur 1 0.000 0.000 0.00 | 59.0 59.0 59.0 |
filebrowser 43607 0.000 0.000 0.74 | 0.0 0.0 35.1 | resets=17
felhom-samba 10 0.000 0.000 0.00 | 0.0 13.5 17.6 | tx counter always 0 (host network?)
determined_satoshi 1 0.000 0.000 0.00 | 17.4 17.4 17.4 |
busiest UTC hours by bytes sent: 03h 14%, 02h 7%, 13h 5%, 00h 4%
=== demo-felhom: 2026-09-23 02:00 -> 2026-10-23 18:08 UTC (30.7 d), 268120 rows
container n tx p50 p95 max Mbit/s | cpu p50 p95 max %core | notes
immich-server 32 0.667 5.953 12.90 | 0.6 120.1 130.8 |
jellyfin 16 0.000 0.854 2.85 | 0.0 72.7 290.7 |
immich-postgres 32 0.000 0.766 2.24 | 0.3 8.5 197.6 |
romm 16 0.020 0.339 1.07 | 0.5 25.5 100.5 |
dawarich 20 0.000 0.313 3.01 | 0.1 5.3 104.4 |
immich-machine-learning 32 0.003 0.243 0.42 | 0.2 294.7 295.7 |
immich-redis 32 0.017 0.144 0.16 | 0.5 3.9 4.2 |
nextcloud-db 16 0.000 0.137 0.43 | 0.0 4.9 5.8 |
nextcloud 16 0.000 0.099 0.33 | 0.0 21.7 86.7 |
paperless-webserver 21 0.005 0.073 0.08 | 0.0 137.4 139.7 |
paperless-postgres 22 0.000 0.048 0.06 | 0.0 4.6 5.0 |
kimai-db 19 0.000 0.023 0.15 | 0.0 8.4 38.8 |
dawarich-sidekiq 19 0.020 0.023 0.02 | 0.1 0.2 0.2 |
claper 20 0.019 0.020 0.02 | 0.5 0.7 0.9 |
docmost 22 0.020 0.020 0.02 | 0.0 0.2 0.3 |
wanderer 19 0.002 0.015 0.02 | 0.0 0.0 0.0 |
cloudflared 43863 0.000 0.013 1.33 | 0.2 0.3 8.3 | resets=11
adventurelog 22 0.000 0.013 3.24 | 0.0 0.1 65.8 |
grafana 20 0.000 0.012 0.12 | 1.0 1.9 17.3 |
wanderer-db 19 0.007 0.011 0.01 | 0.0 3.7 3.7 |
dawarich-redis 20 0.009 0.011 0.01 | 0.4 3.7 3.8 |
claper-postgres 20 0.011 0.011 0.01 | 0.2 0.5 4.4 |
outline 19 0.008 0.008 0.01 | 0.0 7.4 7.7 |
nextcloud-redis 16 0.000 0.008 0.02 | 0.4 0.7 1.4 |
romm-redis 16 0.006 0.007 0.01 | 0.5 3.7 3.8 |
docmost-redis 23 0.005 0.005 0.07 | 0.4 0.7 3.9 |
dawarich-db 20 0.001 0.005 0.03 | 0.0 0.1 0.1 |
bookstack-db 20 0.001 0.005 0.04 | 0.0 4.7 4.8 |
grocy 19 0.000 0.005 0.01 | 0.0 0.0 0.0 |
bookstack 20 0.000 0.005 0.04 | 0.0 7.9 47.2 |
zipline 20 0.003 0.003 0.00 | 0.0 0.0 0.0 |
paperless-redis 22 0.002 0.003 0.00 | 0.4 0.4 0.4 |
outline-redis 19 0.003 0.003 0.00 | 0.4 0.4 0.5 |
kimai 19 0.000 0.003 0.02 | 0.0 2.4 24.3 |
zipline-postgres 20 0.003 0.003 0.00 | 0.0 0.0 0.0 |
qbittorrent 11759 0.001 0.003 0.01 | 0.1 0.2 97.5 | resets=25
felhom-controller 43863 0.000 0.003 1.73 | 0.1 15.5 122.0 | resets=92
bentopdf 18 0.000 0.002 0.00 | 0.0 0.0 0.0 |
emby 19 0.001 0.002 0.01 | 0.0 0.1 0.1 |
homepage 20 0.000 0.002 0.00 | 0.0 0.0 0.0 |
sonarr 11756 0.000 0.001 0.01 | 0.1 0.3 74.7 | resets=25
grimmory-db 24 0.000 0.001 0.35 | 0.0 5.0 100.2 |
grimmory 23 0.001 0.001 0.00 | 5.7 25.3 35.0 |
lubelogger 19 0.000 0.001 0.00 | 0.0 0.2 0.6 |
radarr 11756 0.000 0.001 0.17 | 0.0 0.6 79.9 | resets=25
glance 19 0.000 0.001 0.00 | 0.0 0.0 0.0 |
metube 20 0.000 0.001 0.00 | 0.0 1.3 26.1 |
actualbudget 18 0.000 0.001 0.00 | 0.0 0.0 0.0 |
privatebin 19 0.000 0.001 0.00 | 0.0 0.0 0.0 |
plex 18 0.000 0.001 0.00 | 0.7 0.8 0.8 |
papra 20 0.000 0.001 0.00 | 0.0 6.8 7.5 |
home-assistant 16 0.000 0.001 0.00 | 0.0 19.8 79.0 |
wanderer-search 19 0.000 0.001 0.00 | 0.1 0.1 0.1 |
karakeep 19 0.001 0.001 0.00 | 1.8 2.0 2.0 |
romm-db 16 0.001 0.001 0.00 | 0.0 4.6 4.7 |
radicale 19 0.000 0.001 0.00 | 0.0 0.0 0.0 |
gramps-web 20 0.000 0.001 0.00 | 0.0 0.0 0.0 |
tandoor 19 0.000 0.001 0.00 | 0.1 4.7 4.8 |
karakeep-chrome 19 0.001 0.001 0.00 | 0.0 0.0 0.0 |
outline-postgres 19 0.000 0.001 0.00 | 0.0 0.4 0.7 |
seerr 18 0.000 0.001 0.00 | 0.0 0.0 0.0 |
n8n 22 0.000 0.000 0.00 | 0.8 1.0 110.6 |
termix 20 0.000 0.000 0.00 | 0.0 0.4 2.3 |
tandoor-postgres 19 0.000 0.000 0.00 | 0.4 1.5 4.1 |
audiobookshelf 19 0.000 0.000 0.00 | 0.0 0.0 0.1 |
adventurelog-postgres 22 0.000 0.000 0.02 | 0.0 3.8 33.6 |
ghost 21 0.000 0.000 0.00 | 0.0 0.0 0.0 |
transmission 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
komga 19 0.000 0.000 0.00 | 0.3 6.8 10.6 |
calcom 18 0.000 0.000 0.00 | 0.0 3.4 22.3 |
vaultwarden 20 0.000 0.000 0.00 | 0.0 0.0 0.0 |
sparkyfitness 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
homebox 19 0.000 0.000 0.00 | 0.1 0.2 0.2 |
docmost-postgres 23 0.000 0.000 0.01 | 0.0 3.8 73.9 |
vikunja 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
adventurelog-frontend 22 0.000 0.000 0.00 | 0.0 0.0 0.0 |
rallly 20 0.000 0.000 0.00 | 0.0 5.6 110.3 |
calibre-web 17 0.000 0.000 0.00 | 0.0 21.3 106.4 |
wishlist 20 0.000 0.000 0.00 | 0.0 6.3 125.1 |
prowlarr 11738 0.000 0.000 0.00 | 0.0 0.1 77.0 | resets=26
opengist 43862 0.000 0.000 0.01 | 0.0 0.0 6.0 | resets=107
onlyoffice 22 0.000 0.000 0.00 | 0.0 130.5 147.5 |
sabnzbd 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
gitea 19 0.000 0.000 0.00 | 0.0 0.4 4.0 |
navidrome 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
code-server 20 0.000 0.000 0.00 | 0.0 0.0 0.0 |
uptime-kuma 19 0.000 0.000 0.00 | 0.7 0.7 0.7 |
nzbget 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
gokapi 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
recipe-importer 20 0.000 0.000 0.00 | 0.0 0.0 0.0 |
crafty-controller 19 0.000 0.000 0.00 | 0.1 8.3 8.5 |
mealie 20 0.000 0.000 0.00 | 0.1 0.1 0.1 |
rallly-postgres 20 0.000 0.000 0.00 | 0.0 3.7 3.8 |
traefik 43863 0.000 0.000 0.96 | 0.0 3.6 17.9 | resets=13
sparkyfitness-server 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
sparkyfitness-db 19 0.000 0.000 0.00 | 0.0 3.8 3.8 |
karakeep-meilisearch 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
filebrowser 43863 0.000 0.000 0.74 | 0.0 0.0 15.7 | resets=16
calcom-postgres 18 0.000 0.000 0.00 | 0.0 3.9 4.0 |
busiest UTC hours by bytes sent: 16h 16%, 05h 12%, 09h 6%, 02h 5%
=== tester-1: 2026-10-04 19:41 -> 2026-10-23 18:08 UTC (18.9 d), 294433 rows
container n tx p50 p95 max Mbit/s | cpu p50 p95 max %core | notes
qbittorrent 3857 0.007 0.133 42.40 | 0.1 2.6 92.3 | resets=20
transmission 3857 0.001 0.013 57.43 | 0.1 1.6 82.5 | resets=19
paperless-webserver 26710 0.005 0.013 0.09 | 0.1 0.7 508.1 | resets=63
cloudflared 26751 0.000 0.013 1.63 | 0.4 1.2 29.4 | resets=6
prowlarr 3725 0.000 0.012 0.26 | 0.1 0.3 101.8 | resets=8
sonarr 3855 0.001 0.004 1.21 | 0.1 4.5 131.6 | resets=20
radarr 3855 0.001 0.003 1.52 | 0.1 4.3 103.3 | resets=19
paperless-redis 26740 0.001 0.003 0.01 | 0.7 4.2 27.5 | resets=68
felhom-controller 26751 0.000 0.003 37.47 | 0.5 46.4 422.6 | resets=62
plex 3739 0.001 0.002 1.07 | 0.3 4.3 198.4 | resets=17
bookstack-db 26729 0.001 0.001 0.03 | 0.0 5.7 130.6 | resets=66
plex-k6 514 0.001 0.001 0.01 | 0.4 5.4 7.2 |
bookstack 26722 0.000 0.001 0.04 | 0.0 0.1 72.5 | resets=65
privatebin 26744 0.000 0.001 0.00 | 0.0 0.1 14.7 | resets=64
paperless-postgres 26740 0.000 0.001 0.02 | 0.0 4.4 65.8 | resets=68
traefik 26751 0.000 0.000 1.56 | 0.0 3.4 100.6 | resets=3
jellyfin 3648 0.000 0.000 2.83 | 0.0 0.2 181.5 | resets=19
xfp-echo 1 0.000 0.000 0.00 | 0.0 0.0 0.0 |
vaultwarden 1 0.000 0.000 0.00 | 0.0 0.0 0.0 |
filebrowser 26743 0.000 0.000 0.01 | 0.0 0.0 80.1 | resets=22
busiest UTC hours by bytes sent: 07h 13%, 18h 11%, 02h 9%, 17h 9%
@@ -0,0 +1,22 @@
== demo-hp
container cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]
kimai 13 0 (-) 0 0 (-)
paperless-webserver 6 1 (09-23 04:56) 0 0 (-)
immich-machine-learning 4 4 (10-14 00:54) 0 0 (-)
romm 3 2 (10-23 15:36) 0 0 (-)
adventurelog 2 1 (09-30 11:29) 3 0 (-)
adventurelog-postgres 0 0 (-) 3 0 (-)
felhom-controller 3 1 (09-23 11:37) 2 2 (09-28 13:44)
immich-server 2 0 (-) 3 2 (10-14 00:53)
== demo-felhom
container cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]
immich-server 2 0 (-) 8 1 (10-13 05:32)
immich-machine-learning 7 7 (10-13 16:39) 0 0 (-)
paperless-webserver 6 0 (-) 0 0 (-)
onlyoffice 4 0 (-) 0 0 (-)
== tester-1
container cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]
plex 10 5 (10-14 17:20) 1 0 (-)
qbittorrent 2 0 (-) 9 3 (10-14 18:08)
sonarr 9 0 (-) 2 0 (-)
transmission 1 0 (-) 6 2 (10-16 07:17)
@@ -0,0 +1,90 @@
== demo-hp: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak
2026-09-23 max1= 3.14 best5= 0.91 busy_min= 0 peak_at=02:17Z
2026-09-24 max1= 3.04 best5= 0.73 busy_min= 0 peak_at=02:17Z
2026-09-25 max1= 2.13 best5= 0.67 busy_min= 0 peak_at=02:18Z
2026-09-26 max1= 1.77 best5= 0.63 busy_min= 0 peak_at=02:18Z
2026-09-27 max1= 2.04 best5= 0.75 busy_min= 0 peak_at=02:18Z
2026-09-28 max1= 17.79 best5= 6.89 busy_min= 3 peak_at=13:43Z
2026-09-29 max1= 2.45 best5= 0.65 busy_min= 0 peak_at=02:17Z
2026-09-30 max1= 2.62 best5= 0.76 busy_min= 0 peak_at=02:18Z
2026-10-01 max1= 1.02 best5= 0.54 busy_min= 0 peak_at=02:18Z
2026-10-02 max1= 2.02 best5= 0.62 busy_min= 0 peak_at=02:18Z
2026-10-03 max1= 2.01 best5= 0.60 busy_min= 0 peak_at=15:25Z
2026-10-04 max1= 2.08 best5= 0.42 busy_min= 0 peak_at=02:17Z
2026-10-05 max1= 2.36 best5= 0.48 busy_min= 0 peak_at=05:26Z
2026-10-06 max1= 1.58 best5= 0.32 busy_min= 0 peak_at=02:17Z
2026-10-07 max1= 1.05 best5= 0.30 busy_min= 0 peak_at=02:16Z
2026-10-08 max1= 1.43 best5= 0.29 busy_min= 0 peak_at=02:17Z
2026-10-09 max1= 1.47 best5= 0.29 busy_min= 0 peak_at=02:17Z
2026-10-10 max1= 0.90 best5= 0.27 busy_min= 0 peak_at=02:16Z
2026-10-11 max1= 1.34 best5= 0.30 busy_min= 0 peak_at=02:16Z
2026-10-12 max1= 4.96 best5= 0.99 busy_min= 0 peak_at=02:17Z
2026-10-13 max1= 1.31 best5= 0.28 busy_min= 0 peak_at=02:16Z
2026-10-14 max1= 6.31 best5= 1.27 busy_min= 1 peak_at=02:18Z
2026-10-15 max1= 2.25 best5= 0.45 busy_min= 0 peak_at=02:17Z
2026-10-16 max1= 2.78 best5= 0.66 busy_min= 0 peak_at=20:55Z
2026-10-17 max1= 2.26 best5= 0.45 busy_min= 0 peak_at=02:17Z
2026-10-18 max1= 2.08 best5= 0.47 busy_min= 0 peak_at=02:18Z
2026-10-19 max1= 16.95 best5= 5.08 busy_min= 2 peak_at=02:18Z
2026-10-20 max1= 1.20 best5= 0.48 busy_min= 0 peak_at=02:17Z
2026-10-21 max1= 2.27 best5= 0.46 busy_min= 0 peak_at=02:18Z
2026-10-22 max1= 2.13 best5= 0.52 busy_min= 0 peak_at=02:18Z
2026-10-23 max1= 1.37 best5= 0.48 busy_min= 0 peak_at=02:18Z
nights with a push (best5>2): 2; best5 min/median/max = 5.1/6.9/6.9 Mbit/s
== demo-felhom: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak
2026-09-23 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=02:15Z
2026-09-24 max1= 0.06 best5= 0.01 busy_min= 0 peak_at=02:15Z
2026-09-25 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
2026-09-26 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
2026-09-27 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=00:19Z
2026-09-28 max1= 0.03 best5= 0.01 busy_min= 0 peak_at=02:16Z
2026-09-29 max1= 0.03 best5= 0.01 busy_min= 0 peak_at=02:15Z
2026-09-30 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
2026-10-01 max1= 0.07 best5= 0.01 busy_min= 0 peak_at=19:54Z
2026-10-02 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
2026-10-03 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=02:16Z
2026-10-04 max1= 0.02 best5= 0.00 busy_min= 0 peak_at=06:58Z
2026-10-05 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=05:21Z
2026-10-06 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=17:34Z
2026-10-07 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=02:15Z
2026-10-08 max1= 0.09 best5= 0.02 busy_min= 0 peak_at=13:04Z
2026-10-09 max1= 0.10 best5= 0.02 busy_min= 0 peak_at=05:34Z
2026-10-10 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=02:15Z
2026-10-11 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=10:46Z
2026-10-12 max1= 0.08 best5= 0.02 busy_min= 0 peak_at=06:18Z
2026-10-13 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=13:54Z
2026-10-14 max1= 0.07 best5= 0.02 busy_min= 0 peak_at=07:59Z
2026-10-15 max1= 0.14 best5= 0.03 busy_min= 0 peak_at=14:25Z
2026-10-16 max1= 0.84 best5= 0.17 busy_min= 0 peak_at=02:15Z
2026-10-17 max1= 0.49 best5= 0.10 busy_min= 0 peak_at=02:15Z
2026-10-18 max1= 0.33 best5= 0.07 busy_min= 0 peak_at=02:16Z
2026-10-19 max1= 1.73 best5= 0.35 busy_min= 0 peak_at=02:16Z
2026-10-20 max1= 0.23 best5= 0.07 busy_min= 0 peak_at=02:16Z
2026-10-21 max1= 0.36 best5= 0.07 busy_min= 0 peak_at=02:16Z
2026-10-22 max1= 0.52 best5= 0.11 busy_min= 0 peak_at=02:15Z
2026-10-23 max1= 0.48 best5= 0.10 busy_min= 0 peak_at=02:15Z
== tester-1: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak
2026-10-04 max1= 0.17 best5= 0.14 busy_min= 0 peak_at=19:55Z
2026-10-05 max1= 1.44 best5= 0.29 busy_min= 0 peak_at=03:10Z
2026-10-06 max1= 0.64 best5= 0.13 busy_min= 0 peak_at=02:16Z
2026-10-07 max1= 0.64 best5= 0.13 busy_min= 0 peak_at=02:15Z
2026-10-08 max1= 0.42 best5= 0.08 busy_min= 0 peak_at=02:16Z
2026-10-09 max1= 0.48 best5= 0.10 busy_min= 0 peak_at=02:16Z
2026-10-10 max1= 0.84 best5= 0.17 busy_min= 0 peak_at=02:16Z
2026-10-11 max1= 0.46 best5= 0.09 busy_min= 0 peak_at=02:15Z
2026-10-12 max1= 1.71 best5= 0.34 busy_min= 0 peak_at=02:16Z
2026-10-13 max1= 0.99 best5= 0.20 busy_min= 0 peak_at=02:15Z
2026-10-14 max1= 0.45 best5= 0.09 busy_min= 0 peak_at=02:16Z
2026-10-15 max1= 37.47 best5= 7.58 busy_min= 1 peak_at=02:17Z
2026-10-16 max1= 3.85 best5= 0.77 busy_min= 0 peak_at=02:17Z
2026-10-17 max1= 5.11 best5= 1.21 busy_min= 1 peak_at=02:16Z
2026-10-18 max1= 0.50 best5= 0.10 busy_min= 0 peak_at=02:16Z
2026-10-19 max1= 5.14 best5= 1.03 busy_min= 1 peak_at=02:16Z
2026-10-20 max1= 0.52 best5= 0.10 busy_min= 0 peak_at=02:16Z
2026-10-21 max1= 0.61 best5= 0.12 busy_min= 0 peak_at=02:16Z
2026-10-22 max1= 0.53 best5= 0.11 busy_min= 0 peak_at=02:16Z
2026-10-23 max1= 0.80 best5= 0.18 busy_min= 0 peak_at=02:16Z
nights with a push (best5>2): 1; best5 min/median/max = 7.6/7.6/7.6 Mbit/s
@@ -0,0 +1,6 @@
login 204 cookie
{"t": 0.2, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65136369198, "engine_tx": 65136367684, "cli": "65.5MB / 65.1GB"}
{"t": 6.5, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65152041285, "engine_tx": 65152039771, "cli": "65.9MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 19.9, "rate_engine_Mbit": 19.9}
{"t": 12.7, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65167646902, "engine_tx": 65167645388, "cli": "66.2MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 20.14, "rate_engine_Mbit": 20.14}
{"t": 19.0, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65183320503, "engine_tx": 65183318989, "cli": "66.6MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 19.9, "rate_engine_Mbit": 19.9}
{"t": 25.2, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65198925974, "engine_tx": 65198924460, "cli": "66.9MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 20.14, "rate_engine_Mbit": 20.14}
@@ -0,0 +1,10 @@
LISTEN 0 5 0.0.0.0:5001 0.0.0.0:*
login 204 cookie
{"t": 0.0, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 1559315619, "engine_tx": 1558369279, "cli": "2.94MB / 2.85GB"}
{"t": 6.0, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 9161893432, "engine_tx": 9161333668, "cli": "9.34MB / 10.5GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10136.77, "rate_engine_Mbit": 10137.29}
{"t": 12.1, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 17020963755, "engine_tx": 17019975365, "cli": "16.7MB / 18.7GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10306.98, "rate_engine_Mbit": 10306.42}
{"t": 18.1, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 25140960556, "engine_tx": 25139940620, "cli": "23.6MB / 26.5GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10826.66, "rate_engine_Mbit": 10826.62}
{"t": 24.2, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 33049634829, "engine_tx": 33048966257, "cli": "30.6MB / 34.4GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10372.03, "rate_engine_Mbit": 10372.49}
{"t": 30.2, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 41055773948, "engine_tx": 41055105508, "cli": "37.6MB / 42.3GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10674.85, "rate_engine_Mbit": 10674.85}
{"t": 36.3, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 48735514801, "engine_tx": 48734576785, "cli": "44.3MB / 50GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10071.79, "rate_engine_Mbit": 10071.44}
{"t": 42.3, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 56370706217, "engine_tx": 56370119481, "cli": "52.4MB / 58.8GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10180.26, "rate_engine_Mbit": 10180.72}
@@ -0,0 +1,9 @@
login 204 cookie
setUploadLimit (200, '')
uploadLimit now 2499584 prefs.up_limit 2499584
login 204 cookie
{"t": 0.0, "qb_up_speed": 16386315, "qb_up_data": 1346633728, "qb_tor_uploaded": 1346633728, "proc_tx": 1422747831, "engine_tx": 1422747831, "cli": "27.2MB / 1.42GB"}
{"t": 6.0, "qb_up_speed": 0, "qb_up_data": 1374387364, "qb_tor_uploaded": 1374387364, "proc_tx": 1434635707, "engine_tx": 1434635707, "cli": "27.6MB / 1.44GB", "rate_qb_data_Mbit": 37.0, "rate_proc_Mbit": 15.85, "rate_engine_Mbit": 15.85}
{"t": 12.1, "qb_up_speed": 2872275, "qb_up_data": 1392093322, "qb_tor_uploaded": 1391692420, "proc_tx": 1457229455, "engine_tx": 1457229455, "cli": "27.9MB / 1.46GB", "rate_qb_data_Mbit": 23.22, "rate_proc_Mbit": 29.63, "rate_engine_Mbit": 29.63}
{"t": 18.1, "qb_up_speed": 1975559, "qb_up_data": 1406540742, "qb_tor_uploaded": 1406540742, "proc_tx": 1471460371, "engine_tx": 1471460371, "cli": "28.2MB / 1.47GB", "rate_qb_data_Mbit": 19.26, "rate_proc_Mbit": 18.97, "rate_engine_Mbit": 18.97}
{"t": 24.2, "qb_up_speed": 3254241, "qb_up_data": 1423113257, "qb_tor_uploaded": 1420865709, "proc_tx": 1488294067, "engine_tx": 1488294067, "cli": "28.5MB / 1.49GB", "rate_qb_data_Mbit": 21.73, "rate_proc_Mbit": 22.08, "rate_engine_Mbit": 22.08}
@@ -0,0 +1,11 @@
login 204 cookie
{"t": 0.0, "qb_up_speed": 16266049, "qb_up_data": 117833728, "qb_tor_uploaded": 117701484, "proc_tx": 123161204, "engine_tx": 123161204, "cli": "6.74MB / 140MB"}
{"t": 6.0, "qb_up_speed": 16384087, "qb_up_data": 216137728, "qb_tor_uploaded": 216137728, "proc_tx": 225760308, "engine_tx": 225760308, "cli": "8.38MB / 243MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 136.8, "rate_engine_Mbit": 136.8}
{"t": 12.1, "qb_up_speed": 16387266, "qb_up_data": 314441728, "qb_tor_uploaded": 314441728, "proc_tx": 328359626, "engine_tx": 328359626, "cli": "10.1MB / 346MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 134.56, "rate_engine_Mbit": 134.56}
{"t": 18.1, "qb_up_speed": 16391332, "qb_up_data": 412745728, "qb_tor_uploaded": 412745728, "proc_tx": 431576783, "engine_tx": 431540471, "cli": "11.8MB / 450MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 137.62, "rate_engine_Mbit": 137.57}
{"t": 24.2, "qb_up_speed": 16377743, "qb_up_data": 511049728, "qb_tor_uploaded": 511049728, "proc_tx": 536615588, "engine_tx": 536559607, "cli": "13.4MB / 556MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 137.76, "rate_engine_Mbit": 137.73}
{"t": 30.2, "qb_up_speed": 16387474, "qb_up_data": 609353728, "qb_tor_uploaded": 609353728, "proc_tx": 641957804, "engine_tx": 641903336, "cli": "14.9MB / 660MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 140.46, "rate_engine_Mbit": 140.46}
{"t": 36.3, "qb_up_speed": 16383934, "qb_up_data": 707657728, "qb_tor_uploaded": 707657728, "proc_tx": 746552576, "engine_tx": 746552576, "cli": "16.8MB / 764MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 137.17, "rate_engine_Mbit": 137.24}
{"t": 42.3, "qb_up_speed": 16392852, "qb_up_data": 805961728, "qb_tor_uploaded": 805961728, "proc_tx": 849903723, "engine_tx": 849903723, "cli": "18.3MB / 867MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 137.8, "rate_engine_Mbit": 137.8}
{"t": 48.4, "qb_up_speed": 16382132, "qb_up_data": 904265728, "qb_tor_uploaded": 904265728, "proc_tx": 952502020, "engine_tx": 952502020, "cli": "19.9MB / 970MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 134.56, "rate_engine_Mbit": 134.56}
{"t": 54.4, "qb_up_speed": 16376106, "qb_up_data": 1002569728, "qb_tor_uploaded": 1002569728, "proc_tx": 1055099637, "engine_tx": 1055099637, "cli": "21.5MB / 1.07GB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 136.8, "rate_engine_Mbit": 136.8}
@@ -0,0 +1,8 @@
{'arguments': {}, 'result': 'success'}
{'speed-limit-up': 2500, 'speed-limit-up-enabled': True}
{"t": 0.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157242728}
{"t": 5.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157244479, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 10.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157246230, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 15.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157248133, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 20.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157249884, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 25.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157251635, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
@@ -0,0 +1,10 @@
{"t": 0.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157256302}
{"t": 10.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157258205, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 20.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157261594, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 30.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157263345, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 40.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157265096, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 50.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157268485, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 60.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157271188, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 70.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157272939, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 80.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157276328, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 90.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157278588, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
@@ -0,0 +1,15 @@
login 204 cookie
/api/v2/torrents/stop (200, '')
{'arguments': {}, 'result': 'success'}
{"t": 0.0, "tr_upspeed": 40648163, "tr_cur_up": 1528429958, "tr_tor_up": 1528429958, "tx": 1534546958}
{"t": 5.0, "tr_upspeed": 65588000, "tr_cur_up": 1832873057, "tr_tor_up": 1832873057, "tx": 1839315356, "rate_tr_Mbit": 487.11, "rate_container_Mbit": 487.63}
{"t": 10.0, "tr_upspeed": 23587084, "tr_cur_up": 2076811226, "tr_tor_up": 2076811226, "tx": 2083512878, "rate_tr_Mbit": 390.3, "rate_container_Mbit": 390.72}
{"t": 15.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157213873, "rate_tr_Mbit": 115.8, "rate_container_Mbit": 117.92}
{"t": 20.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157215624, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 25.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157218947, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.01}
{"t": 30.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157220698, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 35.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157222449, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 40.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157224200, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 45.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157225951, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 50.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157227702, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 55.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157231243, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.01}
@@ -0,0 +1,2 @@
# usage: gen.sh <dir> ; writes <dir>/uwtest.bin, 2 GiB, identical on every machine (AES-CTR keystream of a fixed key)
mkdir -p "$1"; openssl enc -aes-128-ctr -nosalt -K 00112233445566778899aabbccddeeff -iv 0 -in /dev/zero 2>/dev/null | head -c 2147483648 > "$1/uwtest.bin"; sha256sum "$1/uwtest.bin" | cut -c1-16
@@ -0,0 +1,30 @@
import sqlite3,sys,collections,datetime,statistics
def pct(a,p):
if not a: return 0
a=sorted(a); k=(len(a)-1)*p/100; f=int(k); c=min(f+1,len(a)-1); return a[f]+(a[c]-a[f])*(k-f)
for box in sys.argv[1:]:
db=sqlite3.connect(f'file:{box}/metrics.db?mode=ro',uri=True)
lo,hi,n=db.execute('select min(ts),max(ts),count(*) from container_metrics').fetchone()
print(f'\n=== {box.rsplit("/",1)[1]}: {datetime.datetime.utcfromtimestamp(lo):%Y-%m-%d %H:%M} -> {datetime.datetime.utcfromtimestamp(hi):%Y-%m-%d %H:%M} UTC ({(hi-lo)/86400:.1f} d), {n} rows')
rows=collections.defaultdict(list)
for ts,name,cpu,tx in db.execute('select ts,container_name,cpu_percent,net_tx_bytes from container_metrics order by ts'):
rows[name].append((ts,cpu,tx))
hour=collections.defaultdict(float)
out=[]
for name,r in rows.items():
rates=[];resets=0;zero=all((x[2] or 0)==0 for x in r)
for (t0,_,a),(t1,_,b) in zip(r,r[1:]):
if a is None or b is None or t1-t0<=0 or t1-t0>180: continue
d=b-a
if d<0: resets+=1; continue
mbps=d*8/(t1-t0)/1e6; rates.append(mbps)
hour[datetime.datetime.utcfromtimestamp(t1).hour]+=d
cpus=[x[1] for x in r]
out.append((pct(rates,95),name,len(r),pct(rates,50),pct(rates,95),max(rates or [0]),pct(cpus,50),pct(cpus,95),max(cpus),zero,resets))
print(f'{"container":32} {"n":>6} {"tx p50":>8} {"p95":>8} {"max":>8} Mbit/s | {"cpu p50":>7} {"p95":>7} {"max":>7} %core | notes')
for o in sorted(out,reverse=True):
_,name,nn,a,b,c,d,e,f,zero,res=o
note=('tx counter always 0 (host network?)' if zero else '')+(f' resets={res}' if res else '')
print(f'{name[:32]:32} {nn:6} {a:8.3f} {b:8.3f} {c:8.2f} | {d:7.1f} {e:7.1f} {f:7.1f} | {note}')
tot=sum(hour.values()) or 1
print('busiest UTC hours by bytes sent:',', '.join(f'{h:02d}h {100*v/tot:.0f}%' for h,v in sorted(hour.items(),key=lambda x:-x[1])[:4]))
@@ -0,0 +1,21 @@
import sqlite3,sys,collections,datetime
for box in sys.argv[1:]:
db=sqlite3.connect(f'file:{box}/metrics.db?mode=ro',uri=True)
r=db.execute("select ts,net_tx_bytes from container_metrics where container_name='felhom-controller' order by ts").fetchall()
per=collections.defaultdict(list)
for (t0,a),(t1,b) in zip(r,r[1:]):
if a is None or b is None or b<a or not 0<t1-t0<=180: continue
d=datetime.datetime.utcfromtimestamp(t1)
per[d.date()].append((t1,(b-a)*8/(t1-t0)/1e6))
print(f'\n== {box.rsplit("/",1)[1]}: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak')
peaks=[]
for day in sorted(per):
v=per[day]; rates=[x[1] for x in v]
best5=max((sum(rates[i:i+5])/5 for i in range(max(1,len(rates)-4))),default=0)
m=max(v,key=lambda x:x[1])
busy=sum(1 for x in rates if x>5)
peaks.append(best5)
print(f'{day} max1={m[1]:6.2f} best5={best5:6.2f} busy_min={busy:4d} peak_at={datetime.datetime.utcfromtimestamp(m[0]):%H:%M}Z')
big=[p for p in peaks if p>2]
if big:
big.sort(); print(f'nights with a push (best5>2): {len(big)}; best5 min/median/max = {big[0]:.1f}/{big[len(big)//2]:.1f}/{big[-1]:.1f} Mbit/s')
@@ -0,0 +1,10 @@
#!/bin/bash
# Waits until 02:08Z on 2026-10-24, then reads the controller container's eth0 tx bytes once a second for 22 minutes on
# each of the three test boxes (read only: /proc/<pid>/net/dev inside guest 9201). Output: one file per box.
OUT=$1
target=$(date -u -d '2026-10-24T02:08:00Z' +%s); now=$(date -u +%s); [ $target -gt $now ] && sleep $((target-now))
INNER='P=$(docker inspect -f {{.State.Pid}} felhom-controller); end=$(( $(date +%s) + 1320 )); while [ $(date +%s) -lt $end ]; do echo "$(date +%s.%N | cut -c1-14) $(awk "/eth0:/{print \$10}" /proc/$P/net/dev)"; sleep 1; done'
for pair in "demo-hp:demo-hp" "felhom-pve:demo-felhom" "root@192.168.0.154:tester-1"; do h=${pair%%:*}; n=${pair##*:}
ssh -o BatchMode=yes -o ServerAliveInterval=30 $h "pct exec 9201 -- bash -c '$INNER'" > "$OUT/night-push-1s-$n.txt" 2>&1 &
done
wait; echo "done $(date -u +%FT%TZ)"
@@ -0,0 +1,89 @@
# Runs INSIDE guest 9202. Asks qBittorrent's own Web API and reads the container's exact byte counters.
# The admin password is read from the container env at use (as the controller does) and never printed.
import json, subprocess, sys, time, urllib.request, urllib.parse, http.client, socket, re, uuid
def sh(*a): return subprocess.run(a, capture_output=True, text=True).stdout.strip()
C = sys.argv[1] if len(sys.argv) > 1 else 'qbittorrent'
IP = sh('docker', 'inspect', '-f', '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}', C).split()[0]
PID = sh('docker', 'inspect', '-f', '{{.State.Pid}}', C)
CID = sh('docker', 'inspect', '-f', '{{.Id}}', C)
BASE = f'http://{IP}:8080'
PW = sh('docker', 'exec', C, 'printenv', 'QBT_ADMIN_PASSWORD')
def req(method, path, data=None, hdr=None, raw=False):
h = {'Referer': BASE}
h.update(hdr or {})
if COOKIE: h['Cookie'] = COOKIE
r = urllib.request.Request(BASE + path, data=data, headers=h, method=method)
try:
with urllib.request.urlopen(r, timeout=20) as resp:
b = resp.read(); return resp.status, (b if raw else (json.loads(b) if b and b[:1] in b'[{' else b.decode()))
except urllib.error.HTTPError as e:
return e.code, e.read().decode()[:200]
COOKIE = None
code, _ = None, None
r = urllib.request.Request(BASE + '/api/v2/auth/login', data=urllib.parse.urlencode({'username': 'admin', 'password': PW}).encode(),
headers={'Referer': BASE, 'Content-Type': 'application/x-www-form-urlencoded'}, method='POST')
with urllib.request.urlopen(r, timeout=20) as resp:
m = re.search(r'((?:QBT_)?SID[A-Za-z0-9_]*)=([^;]+)', '; '.join(resp.headers.get_all('Set-Cookie') or []))
print('login', resp.status, 'cookie' if m else 'NO COOKIE')
COOKIE = f'{m.group(1)}={m.group(2)}'
def netdev():
for line in open(f'/proc/{PID}/net/dev'):
if line.strip().startswith('eth0:'):
f = line.split(':', 1)[1].split(); return int(f[8])
def engine_stats():
s = socket.socket(socket.AF_UNIX); s.connect('/var/run/docker.sock')
s.sendall(f'GET /containers/{CID}/stats?stream=false&one-shot=true HTTP/1.0\r\n\r\n'.encode())
b = b''
while True:
x = s.recv(65536)
if not x: break
b += x
j = json.loads(b.split(b'\r\n\r\n', 1)[1])
return sum(n['tx_bytes'] for n in j.get('networks', {}).values()), j['cpu_stats']['cpu_usage']['total_usage']
def cli_stats():
return sh('docker', 'stats', '--no-stream', '--format', '{{.NetIO}}', C)
mode = sys.argv[2] if len(sys.argv) > 2 else 'info'
if mode == 'info':
print('version', req('GET', '/api/v2/app/version')[1], 'webapi', req('GET', '/api/v2/app/webapiVersion')[1])
print('transfer/info', req('GET', '/api/v2/transfer/info')[1])
p = req('GET', '/api/v2/app/preferences')[1]
print('prefs', {k: p.get(k) for k in ('up_limit', 'dl_limit', 'alt_up_limit', 'alt_dl_limit', 'scheduler_enabled', 'limit_tcp_overhead', 'limit_lan_peers', 'limit_utp_rate', 'max_uploads', 'enable_upload_suggestions')})
print('uploadLimit', req('GET', '/api/v2/transfer/uploadLimit')[1], 'speedLimitsMode', req('GET', '/api/v2/transfer/speedLimitsMode')[1])
elif mode == 'add':
bnd = uuid.uuid4().hex
tor = open('/root/uwtest.torrent', 'rb').read()
parts = []
for k, v in (('savepath', '/mnt/felhom-drives/scratch_hdd/userdata/media/torrents'), ('skip_checking', 'true'), ('category', '')):
parts.append(f'--{bnd}\r\nContent-Disposition: form-data; name="{k}"\r\n\r\n{v}\r\n'.encode())
parts.append(f'--{bnd}\r\nContent-Disposition: form-data; name="torrents"; filename="uwtest.torrent"\r\nContent-Type: application/x-bittorrent\r\n\r\n'.encode() + tor + b'\r\n')
parts.append(f'--{bnd}--\r\n'.encode())
print('add', req('POST', '/api/v2/torrents/add', b''.join(parts), {'Content-Type': f'multipart/form-data; boundary={bnd}'}))
time.sleep(3)
print('torrents', [(t['name'], t['state'], t['progress']) for t in req('GET', '/api/v2/torrents/info')[1]])
elif mode == 'setlimit':
print('setUploadLimit', req('POST', '/api/v2/transfer/setUploadLimit', urllib.parse.urlencode({'limit': sys.argv[3]}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
print('uploadLimit now', req('GET', '/api/v2/transfer/uploadLimit')[1], 'prefs.up_limit', req('GET', '/api/v2/app/preferences')[1].get('up_limit'))
elif mode == 'setpref':
print('setPreferences', req('POST', '/api/v2/app/setPreferences', urllib.parse.urlencode({'json': json.dumps({'up_limit': int(sys.argv[3])})}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
print('uploadLimit now', req('GET', '/api/v2/transfer/uploadLimit')[1], 'prefs.up_limit', req('GET', '/api/v2/app/preferences')[1].get('up_limit'))
elif mode == 'peer':
print('addPeers', req('POST', '/api/v2/torrents/addPeers', urllib.parse.urlencode({'hashes': 'all', 'peers': sys.argv[3]}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
print('reannounce', req('POST', '/api/v2/torrents/reannounce', urllib.parse.urlencode({'hashes': 'all'}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
elif mode == 'post':
print(sys.argv[3], req('POST', sys.argv[3], urllib.parse.urlencode(json.loads(sys.argv[4])).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
elif mode == 'sample':
dur, step = int(sys.argv[3]), int(sys.argv[4])
t0 = time.time(); prev = None
while time.time() - t0 <= dur:
ti = req('GET', '/api/v2/transfer/info')[1]
tor = req('GET', '/api/v2/torrents/info')[1]
etx, ecpu = engine_stats()
row = {'t': round(time.time() - t0, 1), 'qb_up_speed': ti['up_info_speed'], 'qb_up_data': ti['up_info_data'],
'qb_tor_uploaded': sum(x.get('uploaded', 0) for x in tor), 'proc_tx': netdev(), 'engine_tx': etx, 'cli': cli_stats()}
if prev:
dt = row['t'] - prev['t']
row['rate_qb_data_Mbit'] = round((row['qb_up_data'] - prev['qb_up_data']) * 8 / dt / 1e6, 2)
row['rate_proc_Mbit'] = round((row['proc_tx'] - prev['proc_tx']) * 8 / dt / 1e6, 2)
row['rate_engine_Mbit'] = round((row['engine_tx'] - prev['engine_tx']) * 8 / dt / 1e6, 2)
print(json.dumps(row), flush=True); prev = row
time.sleep(step)
@@ -0,0 +1,32 @@
import sqlite3,sys,collections
def runs(r, pred):
best=cur=0; bt=None; prev=None
for t,v in r:
if pred(v) and prev is not None and t-prev<=180: cur+=1
elif pred(v): cur=1
else: cur=0
if cur>best: best=cur; bt=t
prev=t
return best,bt
import datetime
for box in sys.argv[1:]:
db=sqlite3.connect(f'file:{box}/metrics.db?mode=ro',uri=True)
nc=None
rows=collections.defaultdict(list)
for ts,name,cpu,tx in db.execute('select ts,container_name,cpu_percent,net_tx_bytes from container_metrics order by ts'):
rows[name].append((ts,cpu,tx))
print('==',box.rsplit('/',1)[1])
out=[]
for name,r in rows.items():
cpu=[(t,c) for t,c,_ in r]
tx=[]
for (t0,_,a),(t1,_,b) in zip(r,r[1:]):
if a is None or b is None or b<a or not 0<t1-t0<=180: continue
tx.append((t1,(b-a)*8/(t1-t0)/1e6))
c150,ct=runs(cpu,lambda v:v>=150); c80,_=runs(cpu,lambda v:v>=80)
u5,ut=runs(tx,lambda v:v>=5); u1,_=runs(tx,lambda v:v>=1)
if max(c80,u1)>=3: out.append((name,c80,c150,ct,u1,u5,ut))
print(f'{"container":28} cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]')
for name,c80,c150,ct,u1,u5,ut in sorted(out,key=lambda x:-max(x[1],x[4])):
f=lambda t: datetime.datetime.utcfromtimestamp(t).strftime('%m-%d %H:%M') if t else '-'
print(f'{name[:28]:28} {c80:8} {c150:9} ({f(ct)}) {u1:9} {u5:9} ({f(ut)})')
@@ -0,0 +1,21 @@
# Minimal BitTorrent HTTP tracker for a closed scratch test (no internet). Compact peer lists.
import http.server, urllib.parse, socket, struct, time
peers = {} # info_hash -> {(ip,port): last_seen}
def benc(o):
if isinstance(o, int): return b"i%de" % o
if isinstance(o, bytes): return b"%d:%s" % (len(o), o)
if isinstance(o, str): return benc(o.encode())
if isinstance(o, dict): return b"d" + b"".join(benc(k) + benc(v) for k, v in sorted(o.items())) + b"e"
class H(http.server.BaseHTTPRequestHandler):
def do_GET(self):
q = urllib.parse.parse_qs(urllib.parse.urlsplit(self.path).query.encode('latin-1').decode('latin-1'), encoding='latin-1')
ih = q.get('info_hash', [''])[0]; port = int(q.get('port', ['0'])[0]); ip = self.client_address[0]
d = peers.setdefault(ih, {}); now = time.time()
if q.get('event', [''])[0] == 'stopped': d.pop((ip, port), None)
else: d[(ip, port)] = now
lst = b"".join(socket.inet_aton(i) + struct.pack(">H", p) for (i, p), t in d.items() if now - t < 600 and (i, p) != (ip, port))
body = benc({"interval": 30, "peers": lst})
print(time.strftime('%H:%M:%S'), 'announce', ip, port, q.get('event', [''])[0], 'peers', len(lst) // 6, flush=True)
self.send_response(200); self.send_header('Content-Length', str(len(body))); self.end_headers(); self.wfile.write(body)
def log_message(self, *a): pass
http.server.ThreadingHTTPServer(('0.0.0.0', 6969), H).serve_forever()
@@ -0,0 +1,47 @@
# Runs INSIDE guest 9202. Transmission's own RPC numbers vs the container's exact tx bytes. Password read at use, never printed.
import json, subprocess, sys, time, urllib.request, base64
def sh(*a): return subprocess.run(a, capture_output=True, text=True).stdout.strip()
C = 'transmission'
IP = sh('docker', 'inspect', '-f', '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}', C).split()[0]
PID = sh('docker', 'inspect', '-f', '{{.State.Pid}}', C)
PW = sh('docker', 'exec', C, 'printenv', 'PASS')
AUTH = 'Basic ' + base64.b64encode(('admin:' + PW).encode()).decode()
SID = ''
def rpc(method, args=None):
global SID
for _ in range(2):
r = urllib.request.Request(f'http://{IP}:9091/transmission/rpc', data=json.dumps({'method': method, 'arguments': args or {}}).encode(),
headers={'Authorization': AUTH, 'Host': 'transmission:9091', 'X-Transmission-Session-Id': SID, 'Content-Type': 'application/json'}, method='POST')
try:
with urllib.request.urlopen(r, timeout=20) as resp: return json.loads(resp.read())
except urllib.error.HTTPError as e:
if e.code == 409: SID = e.headers.get('X-Transmission-Session-Id'); continue
return {'http': e.code}
def tx():
for line in open(f'/proc/{PID}/net/dev'):
if line.strip().startswith('eth0:'): return int(line.split(':', 1)[1].split()[8])
mode = sys.argv[1]
if mode == 'info':
g = rpc('session-get')['arguments']
print('version', g.get('version'), {k: g.get(k) for k in ('speed-limit-up', 'speed-limit-up-enabled', 'alt-speed-up', 'alt-speed-enabled', 'alt-speed-time-enabled', 'units')})
print('session-stats', rpc('session-stats')['arguments'])
elif mode == 'add':
meta = base64.b64encode(open('/root/uwtest.torrent', 'rb').read()).decode()
print(rpc('torrent-add', {'metainfo': meta, 'download-dir': '/mnt/felhom-drives/scratch_hdd/userdata/media/transmission', 'paused': False}))
time.sleep(5); rpc('torrent-verify', {}); time.sleep(20)
print(rpc('torrent-get', {'fields': ['name', 'status', 'percentDone', 'error', 'errorString']}))
elif mode == 'limit':
print(rpc('session-set', {'speed-limit-up': int(sys.argv[2]), 'speed-limit-up-enabled': True}))
g = rpc('session-get')['arguments']; print({k: g.get(k) for k in ('speed-limit-up', 'speed-limit-up-enabled')})
elif mode == 'reannounce':
print(rpc('torrent-reannounce', {}))
elif mode == 'sample':
dur, step = int(sys.argv[2]), int(sys.argv[3]); t0 = time.time(); prev = None
while time.time() - t0 <= dur:
st = rpc('session-stats')['arguments']; tg = rpc('torrent-get', {'fields': ['uploadedEver', 'rateUpload']})['arguments']['torrents']
row = {'t': round(time.time() - t0, 1), 'tr_upspeed': st['uploadSpeed'], 'tr_cur_up': st['current-stats']['uploadedBytes'], 'tr_tor_up': sum(x['uploadedEver'] for x in tg), 'tx': tx()}
if prev:
dt = row['t'] - prev['t']
row['rate_tr_Mbit'] = round((row['tr_cur_up'] - prev['tr_cur_up']) * 8 / dt / 1e6, 2)
row['rate_container_Mbit'] = round((row['tx'] - prev['tx']) * 8 / dt / 1e6, 2)
print(json.dumps(row), flush=True); prev = row; time.sleep(step)
@@ -0,0 +1,236 @@
# Catalog sweep 2026-10-24 — every catalog app on a fresh box, the way a family uses it
Unattended validation sweep (Saturday 12:30Z → Sunday ~00:00Z). Brief: the operator's „every catalog app, one by one, on
a fresh box" prompt. Venue: demo-hp (Tier 0). Method: endpoint-level — every step is the request the dashboard or the
app's own page sends (no browser click-through except the screenshot harness; no `docker exec` to make state).
## Safest readings taken (the brief allows no question)
1. **ISO 1.30.0, not 1.36.0.** The brief said „1.36.0 or the current one"; the current one on felhom.eu/letoltes was
1.30.0 (sha256 checked against the published `.sha256`). See `_box/README.md`.
2. **12 GiB for the VM, not 16.** demo-hp had 14 GiB free beside 9201, 9202 and Tester 1; 12 is the brief's minimum.
The guest itself is 7.7 GiB (the golden's size); the box's own memory admission decided what could run.
3. **„Remove keeping data → install again with use my kept data" is a drive-folder question.** By design (`07` §6.5)
the question is asked only for an app that keeps a folder on a drive (`HDD_PATH`). For the 40+ apps that keep their
data in Docker volumes, „keep my data" keeps the recovery unit; the reinstall starts empty, and the item comes back by
the unit restore („Visszaállítás"). So step 5 + 6 ran as: keep-remove → reinstall → read (empty, expected) → unit
restore → read. That is a unit restore on **every** app with a seed, not only every fifth.
4. **„Mentés most" is the local copy only** (`07` §8, by design); the second-drive and off-site copies run in the night.
The sweep did not wait for a night, so „the right tiers ran" is checked as „the local unit holds the app".
5. **Categories:** the brief names „Apps & catalog" for bigger issues; a row whose harm is in backup/restore or storage is
filed in that category instead (`register_shape_gate` has one category per row, and the owner reads it there).
6. **Sev:** the brief's scale (data at risk P1/P2, a broken app P2, a confusing step P3). R-1269, R-1271 are P2 by that rule (the data is not deleted in either; the household loses access to it).
7. **The sweep box reads the LIVE catalog** (main `5e196e1`, then `227a544` — rules only, no template moved). The drill
catalog was used only for the three apps whose fix needed proving; the box was pointed back after (`_drill-repoint.txt`).
8. **Harness fixes live in the sweep's copy** (`tools/`), not in the catalog's scripts; porting them is R-1277.
## The box (Part 0)
`_box/README.md`: VM **9360** `sweep1024-box` (6 cores, 12 GiB, disks 256 + 200 + 150 GB on `nvme-scratch`), ISO 1.30.0,
customer Tester4 (hub host `Tester4-aa9466`), guest 9201, domain `tester4.test` (no DNS: the browser LXC maps the
names). Bind 12:42:18Z, day-0 SUCCESS 12:45:40Z, claim, recovery code escrowed (stored out-of-band), drives `hdd_1`
(210 GB) and `hdd_2` (157 GB) plus the installer's internal place `belso`. Versions at the start: controller 0.342.0,
agent 0.164.0, hub 0.169.0. Browser: LXC **9361** `sweep1024-browser` (Chromium + Playwright), 60 dashboard screens
(15 pages × phone/computer × HU/EN) in `_box/screens-dashboard/`.
## Method per app (Part 1)
`tools/sweep.py <app>` — one app at a time, small first, heavy last (`tools/order.txt`); the next starts after the last
is removed with its data. Each `<app>/` holds `record.json` (the measured facts), `walk.log`, the page texts
(`page-{install,guide,overview}-{hu,en}.txt`) and `screens/` (phone + computer, HU + EN, install page and the app after
first login). Steps: install through `POST /api/stacks/<app>/deploy` (timed to the front door answering); a stranger
request without the family cookie (must be refused where the template gates); one item through the app's own API (the
catalog's fixture where one exists, a new one in `tools/sweep_fixtures.py` for seven more); „Mentés most"; remove
keeping data; reinstall (the kept-data question answered „use" where it is asked); read; unit restore; read; remove with
data; leftovers checked (containers, volumes, `app.yaml`, `appdata`, `kept`, the subdomain → 404). Memory: the peak of
each container's **anon** memory from its cgroup (`memory.stat`; the file cache is not counted — project rule) against
its compose limit; a `*` in the table means only `docker stats` was read for that app (it includes the cache). The anon reading joined mid-sweep, so 22 apps carry it and 40 carry the `docker stats` upper bound; of those 40 none passed 79 % of its limit (Tandoor 1009/1280) and none restarted. Above 80 % on anon: only OpenGist (F14).
## Results
<!-- TABLE START -->
| App | Verdict | Peak memory / limit (MiB) | Install → answering (s) | What we saw | Row / fix |
|---|---|---|---|---|---|
| actualbudget | **works** | 49* / 256 | 30 | the seeded item back after the restore | |
| adventurelog | **works with a problem** | 381 / 896 | 207 | works; but the keep-backups → reinstall → restore path breaks it (DB password from the new install over the old data) — reproduced in its re-run | R-1269, R-1270 |
| audiobookshelf | **works** | 56* / 512 | 41 | the seeded item back after the restore | drive names: R-1275 |
| bentopdf | **works** | 3* / 384 | 30 | stateless (nothing to keep); installs, answers, removes clean | |
| bookstack | **works** | 167* / 768 | 71 | the seeded item back after the restore | |
| calcom | **works** | 807 / 1792 | 278 | the seeded item back after the restore; 2 start-up messages of the image itself | |
| calibre-web | **works** | 211 / 768 | 76 | the book (its own upload) is listed again after the keep-data reinstall; the fixture's download check gets 401 (its sign-in for /download, not the app); the admin name is the box's generated one (re-run 2) | R-1277 |
| claper | **works** | 267* / 640 | 86 | the seeded item back after the restore | |
| code-server | **works** | 81 / 1024 | 46 | first install refused by the registry's rate limit, told as „exit code 1”; the re-run installed and worked (no seed: the editor has no API) | R-1273 |
| crafty-controller | **works** | 79* / 2048 | 76 | the seeded item back after the restore | |
| dawarich | **works** | 835* / 2688 | 136 | the seeded item back after the restore (every-5th: restore also before the reinstall) | |
| docmost | **works** | 528* / 1152 | 126 | the seeded item back after the restore | |
| emby | **works** | 95 / 2048 | 41 | installs, answers, removes clean; no seed (no fixture for its wizard) | |
| ghost | **works** | 167* / 512 | 121 | the seeded item back after the restore | |
| gitea | **works** | 112 / 512 | 25 | the seeded item back after the restore | |
| glance | **works** | 7* / 128 | 15 | family gate holds a stranger; config-only (nothing to keep); the app's own headings are English | |
| gokapi | **works** | 6 / 128 | 15 | installs, own login, removes clean; no seed (API key only in its UI) | |
| grafana | **works** | 383* / 768 | 61 | the seeded item back after the restore | |
| gramps-web | **works** | 347* / 1024 | 206 | installs, answers, removes clean; no seed fixture | |
| grimmory | **works with a problem** | 594 / 1408 | 96 | first admin, library and an EPUB upload work; setup gate then family gate work as designed; the read-back through the family gate stayed inconclusive (the harness's gate logic, R-1277); remove-with-data leaves the empty parent folder | R-1276, R-1277 |
| grocy | **works** | 34* / 384 | 36 | the seeded item back after the restore | |
| home-assistant | **works** | 311 / 1024 | 161 | the seeded item back after the restore | |
| homebox | **works** | 205 / 256 | 20 | the seeded item back after the restore | |
| homepage | **works with a problem** | 91* / 256 | 25 | no gate at all: a stranger sees the box's CPU/RAM/disk (and later the family's links); still the upstream sample page | R-1272 |
| immich | **works with a problem** | 1845 / 6016 | 121 | album (new fixture) in; „use my kept data” loaded the rehearsal's future-dated database and Immich never answered again (404, 20 min) | R-1271 |
| jellyfin | **works** | 110 / 2048 | 41 | first user (new fixture, its wizard API) back after the restore | |
| karakeep | **works** | 649* / 2816 | 107 | the seeded item back after the restore | |
| kimai | **works** | 303* / 640 | 111 | own login; a Hungarian browser lands on /hu/ (curl without a language gets /en/) | |
| komga | **works** | 473* / 768 | 46 | the seeded item back after the restore | drive names: R-1275 |
| lubelogger | **works** | 60* / 512 | 25 | the seeded item back after the restore | |
| mealie | **works** | 322 / 1000 | 72 | the seeded item back after the restore (the fixture raced after_install on the first run) | |
| metube | **works** | 90* / 768 | 61 | family gate holds a stranger; download queue entry back | |
| n8n | **works with a problem** | 346 / 1536 | 151 | the owner back after the restore; on the live template it tries to send usage data out ([Rudder] lines, 6 in the live re-run) — the drill fix: the variable in the running container, 0 error lines | drill a601950 (morning list) |
| navidrome | **works** | 23* / 256 | 25 | the seeded item back after the restore | drive names: R-1275 |
| nextcloud | **works with a problem** | 298 / 1664 | 122 | user in; „use my kept data” loaded the rehearsal's future-dated database beside today's files — the user was not there | R-1271 |
| nzbget | **works** | 6* / 512 | 25 | the seeded item back after the restore; „no news server” until the family adds one (the guide says so) | |
| onlyoffice | **works** | 464 / 2048 | 71 | document server (no household data to keep); installs, answers, removes clean (every-5th restore) | |
| opengist | **works with a problem** | 125 / 128 | 15 | the gist owner back after the restore; but 128M is too small: one sign-up + sign-in hit the limit 443 times and swapped 98 MiB (memprobe-128M.txt); at 256M (drill) anon 216 MiB | drill 405077f (morning list) |
| outline | **works** | 327 / 1152 | 151 | the seeded item back after the restore | |
| paperless-ngx | **works with a problem** | 669* / 1664 | 146 | account in; „use my kept data” loaded the rehearsal's future-dated database — the account was not there | R-1271, R-1276 |
| papra | **works** | 320 / 768 | 76 | the seeded item back after the restore (Papra 26.6 needs an Origin header — a browser sends it) | |
| plant-it | **not installed** | – | – | lifecycle: abandoned — not offered for a new install (by design) | |
| plex | **works** | 223 / 2048 | 30 | installs, answers, removes clean; no seed (a server is claimed with a plex.tv account) | |
| privatebin | **works** | 31* / 128 | 15 | the seeded item back after the restore; D22 put-back PASS | |
| prowlarr | **works** | 114* / 512 | 21 | the seeded item back after the restore | |
| qbittorrent | **works** | 77* / 1024 | 15 | the seeded item back after the restore | |
| radarr | **works** | 100* / 512 | 41 | the seeded item back after the restore | |
| radicale | **works** | 33* / 128 | 25 | the seeded item back after the restore | |
| rallly | **works** | 346 / 1024 | 71 | installs, answers, removes clean; no seed: Rallly 4.15 closed password sign-up (login by e-mail code); the guide's polls need no account | R-1277 |
| recipe-importer | **works** | 67* / 128 | 25 | stateless helper; installs, answers, removes clean (every-5th restore) | |
| romm | **works with a problem** | 748* / 1536 | 116 | item back through „use my kept data” (its own copy); remove-with-data left the empty parent folder | R-1276 |
| sabnzbd | **works** | 69* / 1024 | 25 | the seeded item back after the restore | |
| seerr | **works** | 166 / 384 | 92 | installs, answers (307 to its setup), removes clean; no seed (needs a media server) | |
| sonarr | **works** | 80* / 512 | 41 | the seeded item back after the restore | |
| sparkyfitness | **works** | 417* / 1792 | 91 | the seeded item back after the restore | |
| tandoor | **works with a problem** | 1052* / 1536 | 177 | works; but the keep-backups → reinstall → restore path breaks it (DB password from the new install over the old data), and the restore still said success — reproduced on its own | R-1269, R-1270 |
| termix | **works** | 111* / 256 | 61 | installs, answers, removes clean (every-5th restore); no seed fixture | |
| transmission | **works** | 6* / 512 | 20 | the seeded item back after the restore | |
| uptime-kuma | **works** | 120* / 384 | 76 | the seeded item back after the restore | |
| vaultwarden | **works** | 37 / 256 | 25 | installs in 25 s, its own login page, sign-up closed (404); the box admin's invite works (run-live-2), but 1.37.4 has no route for the fixture's invited sign-up, so no item was seeded | R-1277 |
| vikunja | **works** | 37* / 256 | 10 | the seeded item back after the restore; D22 put-back PASS | |
| wanderer | **not testable on this box** | 140* / 1024 | 605 | not testable on this box: its web part calls its database at the PUBLIC name, which tester4.test cannot resolve (known, R-739) — unhealthy, 404 | R-739 |
| wger | **not installed** | – | – | lifecycle: hidden — not offered for a new install (by design) | |
| wishlist | **works** | 200* / 512 | 71 | the seeded item back after the restore | |
| zipline | **works** | 233* / 768 | 81 | the seeded item back after the restore | |
works: 52, works with a problem: 10, not installed: 2, not testable on this box: 1 (of 65)
<!-- TABLE END -->
## Findings — each a row, a fix or NOT-A-FINDING
| # | What | Disposition |
|---|---|---|
| F10 | A reinstall's own capture rewrites the unit that „remove — keep backups" kept; restoring it pairs the NEW install's DB password with the OLD data — the app never starts, the item is gone (Tandoor ×2, AdventureLog) | R-1269 (P2) |
| F11 | The unit restore reports `ok: true` „…az alkalmazás újraindult" while the controller logged „health check failed … did not reach running state" | R-1270 (P2) |
| F12 | „Use my kept data" loaded the rehearsal box's future-dated off-site database (2026-10-28) for Paperless-ngx, Nextcloud and Immich — another drive's install, which `07` §6.5 says is refused; Immich then never answered | R-1271 (P2) |
| F1 | Homepage has no gate: a stranger sees the box's CPU/RAM/disk (Glance, same class, is gated) | R-1272 (operator decision) |
| F7 | An install refused by the registry's rate limit (`toomanyrequests`) tells the household only „exit code 1" | R-1273 |
| F8 | The restore refusal says „turn on remote backup, or connect a second drive" when both exist but have not copied yet | R-1274 |
| F2 | English pages show the drive names in Hungarian („Belső tárhely", „Külső tárhely (hdd_1)") | R-1275 |
| F9 | „Remove with data" leaves the empty parent folder `appdata/<app>` (Grimmory, RomM, Paperless-ngx) | R-1276 |
| — | The catalog's box fixtures drifted from the apps (OpenGist, Calibre-Web, Mealie, Papra, Rallly; seven apps had none) | R-1277 |
| F13 | n8n sends usage data (`[Rudder]` lines; no `N8N_DIAGNOSTICS_ENABLED=false`) | **fixed in the drill** `a601950`, proven below; morning list |
| F14 | OpenGist at 125.8 of 128 MiB anon after one sign-up + sign-in; at 128M it hits the limit 443 times and swaps 98 MiB | **fixed in the drill** `405077f` + `492a17f` (256M), proven below; morning list |
| — | 52 anonymous Docker volumes stayed after every app was removed with data (PrivateBin, OnlyOffice, RomM, Immich: the images' own `VOLUME` paths; 4–72 KB, no household data). The sweep's own leftover check filters by the app's label and missed them; found in the end-state read (`_box/anon-volumes.txt`) | known: note on R-1267 (widened to OnlyOffice and RomM) |
| — | Wanderer unhealthy, 404: its web part calls its database by the public name, which a box without DNS cannot resolve | known: note on R-739 |
| F3 | The install page's default place is the internal place `belso`, not the 210 GB drive | NOT-A-FINDING: by design — `16` §5 registers the internal place as the default drive |
| F4 | Guide labels in English (LubeLogger „Settings", NZBGet „Settings / News-Servers") | NOT-A-FINDING: they are the app's own on-screen labels, and the Hungarian catalog copy is frozen by the copy-i18n gate (measured on the drill) |
| F5 | Kimai opens in English | NOT-A-FINDING: a Hungarian browser lands on `/hu/`; only a request with no language gets `/en/` |
| F6 | AdventureLog backend 95 % of its limit | NOT-A-FINDING: that was `docker stats` with the cache; on anon 288.7/384 MiB (75 %), = the ladder's 73 % |
| — | Database error lines (SparkyFitness `database "sparky" does not exist`, Dawarich `database "dawarich" already exists`; SparkyFitness `Invalid password` = the harness's wrong-password control) | NOT-A-FINDING: the images' own first-start and restore-time checks; no restart, the item read back after the restore |
| — | Radicale redirects to an IP | NOT-A-FINDING: a curl artefact (no Host on the redirect probe); the browser lands right |
| — | NZBGet / Transmission screenshots show the browser's basic-auth prompt | NOT-A-FINDING: the app's own login, by design |
| — | code-server's first install failed (`toomanyrequests`) | the failed install was removed through the product (`code-server.failed-remove.txt`); the re-run worked; the message is R-1273 |
## F10 proof — the kept unit is rewritten by the reinstall (`_unitprobe/tandoor/`)
`tools/unitprobe.py tandoor`: install, one account, „Mentés most" (unit 19:45:14–20), remove keeping data and
backups, install again (a volume app: no kept-data question), wait for the controller's next capture, „Visszaállítás".
- 19:50:32 `Recovery unit captured for tandoor` — the unit's `app.yaml` checksum `0f366c97…` → `4438c1c5…`,
`created_at` 19:45:26 → 19:50:32; the data files unchanged (`unit-1/2/3*.txt`).
- The restore: `ok: true`, „A(z) tandoor: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult."
- After: state `starting`, the account not there, `password authentication failed` ×42 (app) and ×22 (postgres).
- The comment at `felhom-controller/controller/internal/backup/recovery_unit.go:101-103` says the refresh never rewrites
the definition away from its data; after a remove + reinstall it does.
Seen three times: the sweep's own Tandoor cycle (17:27:41 reinstall → 17:30:34 capture → 17:30:47 restore), this probe,
and AdventureLog's re-run (20:03:18 → 20:05:32 → restore). AdventureLog's first run worked only because no capture fell
between its reinstall and its restore.
## D22 put-back — PASS on two apps (`_putback/`)
`tools/putback.py <app>`: the largest volume tar of the app's unit truncated to half, then „Visszaállítás". PrivateBin
and Vikunja: the restore failed honestly („Nem változott semmi…"), the pre-restore data was put back, the app ran, and the
seeded item read back through the front door.
## Drill fixes, proven on the sweep box pointed at the drill (Part 2)
Drill catalog `app-catalog-drill` (never the live catalog): `742a354` (merge of live main), then:
- `a601950` **n8n** — `N8N_DIAGNOSTICS_ENABLED=false` in the compose (usage data stays home).
- `405077f` + `492a17f` **OpenGist** — memory 128M → 256M (compose + `.felhom.yml` `mem_limit`), the comment carries the
measured need.
**How the box was pointed at the drill.** The drill repo on Gitea is private (R-992) and the box holds no catalog
credential, so the box reads a local mirror: a bare repository in the controller's data volume, loaded from a git bundle,
`repo_url: file:///opt/docker/felhom-controller/drill-catalog.git` (the method of
`audits/burndown-3-2026-10-17/tools/repoint.py`; loading the mirror is a `docker exec git fetch` into the controller —
catalog plumbing, not app state). Only `repo_url` changes; `tools/repoint.py live` puts the saved `controller.yaml` back
and checks it byte for byte (`RESTORED-IDENTICAL`, twice in `_drill-repoint.txt`). The box ended on the live catalog
`227a544`.
**What went wrong first, and is kept as evidence.** The first drill pass (chain 3, 21:18Z) set `repo_url` to the Gitea
URL in the belief that the repo was public. The box's clone failed (`fatal: could not read Username`), the controller kept
the templates it had, and n8n, OpenGist and Vaultwarden ran on the LIVE templates. The leftover `[Rudder]` lines in that
n8n run showed it. Those three runs are kept as `run-live-2/` (valid live re-runs). The second pass (chain 4) refused to
install anything until the box's catalog cache read the drill's commit and the changed lines were in the box's copy of
both templates: `drill main fd23571, box cache fd23571: DRILL-CONFIRMED`. (The drill also carries a six-line Plex
ladder entry from an earlier session's test; Plex was not installed in this pass.)
**n8n proof.** The running container carried `N8N_DIAGNOSTICS_ENABLED=false` (read at 21:37:58, that line only); error
lines 0 against 6 `[Rudder]` lines in the live re-run just before it (the control); the owner account read back after the
unit restore.
**OpenGist proof — and a wrong turn.** At 256M (drill) OpenGist took 215.8 MiB anon. I first read that as „it grows into
its limit" and reverted the fix (`c420053`); an automated security review flagged the revert as a lowered resource cap.
A measurement settled it (`tools/memprobe-guest.sh`, `opengist/memprobe-128M.txt`): on the live 128M template one
sign-up + sign-in hit `memory.max` 443 times and pushed 98 MiB into swap (anon 125 MiB, 0 `oom_kill`, 0 restarts). The
real need is ~220 MiB; the fix went back in (`405077f`). Where the box has no swap room left, the same sign-up would end in a kill (not measured here).
Drill gates: `catalog_gates.py --fast opengist n8n` from the drill clone — all OK except `security-scan`, which looks for
a `felhom.eu` clone beside the drill and found none; the shared scanner run by hand on the drill clone: 0 hits.
## Teardown
Three layers, 2026-10-25 ~00:05Z:
- **Machine (demo-hp):** VM **9360** `sweep1024-box` and LXC **9361** `sweep1024-browser` — both created by this session —
stopped and destroyed with `--purge`; no disk of either is left on `nvme-scratch`. Untouched: Tester 1 (VM 341), 9201,
9202, 9401, the thin pool, and the 1.30.0 ISO in `local:iso` (there since 2026-10-22). Before the destroy, the box's
controller log, the agent journal, its end state and its anonymous-volume list were copied here (`_box/`).
- **Host (DooPlex):** the session's secret files (`/run/user/1000/sweep-sc`: the dashboard password copy, cookies, CSRF
tokens) shredded. The drill catalog keeps its commits (`a601950`, `405077f`, `492a17f`, and the reverted turn
`c420053`) for the morning list; nothing went to the live catalog.
- **Hub:** host **Tester4-aa9466** stays for the operator to delete (CC holds no delete). Until then it reads as a box
that stopped reporting.
- **ep0:** listed, nothing removed. The host's PBS link to ep0 (`felhom-pbs`) was set up at day-0 (`pbsdr: converged`,
14:44:48 CEST, `_box/agent-journal.log.gz`); no whole-box copy went to it (weekly cadence — only the local copy ran,
14:51). The app copies the box sent off-site went to the Tester4 restic repository on the storage box (unlocked with
the rehearsal's recovery code; a new code escrowed — stored out-of-band), not to ep0.
## Evidence map
- `_box/` — Part 0 (VM, ISO, bind, day-0, claim, recovery, drives, dashboard screens).
- `<app>/` — one per app; re-run apps keep the earlier run in `first-run/`, `run-2/` or `run-drill-pre/`.
- `<app>.console.txt`, `progress.log`, `batch.out`, `chain2.out`, `chain3.out` — the runner's own output.
- `_unitprobe/tandoor/` — F10. `_putback/{privatebin,vikunja}/` — D22.
- `tools/` — the harness (sweep copies of the catalog's `box_walk.py` and `upgrade_fixtures_box.py`, marked SWEEP COPY;
`sweep.py`, `sweep_fixtures.py`, `unitprobe.py`, `putback.py`, `repoint.py`, `table.py`, the rehearsal's VM and browser
helpers). Secrets were read by name through `sec` and never written here; the evidence was scanned for every known
secret value and for token shapes (one app session token found and blanked: Actual Budget's), with a positive control.
- Screenshots were reduced with `pngquant` (lossy palette) to keep the repository small.
@@ -0,0 +1,20 @@
# Part 0 — the sweep box (2026-10-24, from 12:33Z)
| | |
|---|---|
| VM | **9360 `sweep1024-box`** on demo-hp: 6 cores (`cpu host`), **12 GiB** (demo-hp had 14 GiB available of 29; 9201, 9202 and Tester 1 untouched — the brief's minimum, not its 16 GiB aim), 256 GB system disk + 200 GB + 150 GB data disks, all raw on `nvme-scratch` (the dir storage), never the thin pool (`vm-create.txt`) |
| ISO | `felhom-installer-1.30.0-pve9.2-1.iso` — the current one on felhom.eu/letoltes; sha256 `590d7b9b…74d5cc1` = the published `.sha256` |
| Installer | graphical, Hungarian keyboard kept, driven by `qm sendkey` (`tools/vmctl.sh`), screens in `install/`. Install pressed **12:37:33Z**; pairing code on the console **12:41** (blacked out in `install/11-console.png`) |
| Bind | Tester4's old host deleted by the operator (12:32, asked by CC); the hub mailed a new bind link 12:31:57Z; bound through the real page in Chromium 12:42:18Z, first try (`screens/b-bind-*`) |
| Day-0 | 8/8 steps, **SUCCESS 12:45:40Z** — host `Tester4-aa9466`, guest 9201, golden imported and verified; Install → done **8 min 7 s** |
| Claim | setup-code mail 12:43:14Z („Új beállító kód — újratelepült a szervered", right for a customer who had a box before); claimed 302 → `/` (`claim.txt`) |
| Recovery | the box found Tester4's sealed off-site package (2026-10-20 14:16) and asked for the old code first (`/recovery`): the 2026-10-20 code, typed as it was written down (ten words joined by hyphens), unlocked it — controller „the offsite repository key was recovered and placed" 12:47:41Z. Then the dashboard asked for a NEW code: the ceremony in Chromium (words 4 and 9 asked), escrowed; „Távoli mentés: Aktív" (`screens/box-escrow-*`, the code blacked out) |
| Drives | `/dev/sdb` → `hdd_1` (210 GB, the apps' data), `/dev/sdc` → `hdd_2` (157 GB), both through `POST /api/storage/init` (`drive-init.txt`); the installer's internal place `belso` (59 GB) |
| Versions | controller **0.342.0**, agent **0.164.0**, hub **0.169.0**; the box syncs the LIVE catalog (`app-catalog-felhom.eu` main, `5e196e1` at the start; it re-syncs every 15 min) |
| Guest | 9201 at 192.168.0.157, 7.7 GiB RAM (the golden's size), domain `tester4.test` (no DNS — the browser LXC maps the names; TLS not verified) |
| Browser | LXC **9361 `sweep1024-browser`** (2 cores, 2 GB, nvme-scratch): Chromium 156.0.8078.47 + Playwright 1.52.0; the household's session stored in the LXC (`/root/work/state-dash.json`) |
| Dashboard | every menu page at phone and computer size, HU and EN (`screens-dashboard/`, 60): no sideways scroll reported; a signed-in household reads its OWN language setting whatever the browser asks — by design (`langFor`), the page texts were read with `?lang=` |
Secrets used (names only, `sec`): SWEEP_2026_10_24_ROOTPW (the typed root password — day-0 replaced it with the vaulted
one; DooPlex's key was added before the bind), SWEEP_2026_10_24_BIND_TOKEN, _PAIRING_CODE, _SETUP_CODE, _DASH_PW,
_RECOVERY_CODE, and the existing REHEARSAL_2026_10_20_PASSPHRASE / _RECOVERY_CODE.
@@ -0,0 +1,52 @@
2026-10-24T17:15:18Z 28 assets,config,library,
2026-10-24T12:51:57Z 24 client_body,fastcgi,proxy,
2026-10-24T13:08:14Z 24 client_body,fastcgi,proxy,
2026-10-24T19:54:35Z 4
2026-10-24T17:15:18Z 4
2026-10-24T12:51:49Z 24 client_body,fastcgi,proxy,
2026-10-24T19:54:18Z 4
2026-10-24T13:06:43Z 24 client_body,fastcgi,proxy,
2026-10-24T13:08:05Z 72 lock,nginx,nginx.pid,
2026-10-24T19:06:36Z 36 documentserver,documentserver-example,
2026-10-24T13:06:43Z 4
2026-10-24T13:08:05Z 24 client_body,fastcgi,proxy,
2026-10-24T13:06:43Z 72 lock,nginx,nginx.pid,
2026-10-24T19:54:18Z 24 client_body,fastcgi,proxy,
2026-10-24T17:18:02Z 4
2026-10-24T13:07:11Z 24 client_body,fastcgi,proxy,
2026-10-24T12:50:39Z 72 lock,nginx,nginx.pid,
2026-10-24T13:07:11Z 72 lock,nginx,nginx.pid,
2026-10-24T19:09:38Z 36 documentserver,documentserver-example,
2026-10-24T17:20:41Z 28 assets,config,library,
2026-10-24T19:54:35Z 24 client_body,fastcgi,proxy,
2026-10-24T12:50:39Z 24 client_body,fastcgi,proxy,
2026-10-24T12:51:57Z 72 lock,nginx,nginx.pid,
2026-10-24T19:07:30Z 4
2026-10-24T12:50:57Z 24 client_body,fastcgi,proxy,
2026-10-24T19:21:09Z 4
2026-10-24T19:09:38Z 4
2026-10-24T12:50:57Z 72 lock,nginx,nginx.pid,
2026-10-24T19:20:01Z 4
2026-10-24T19:55:03Z 4
2026-10-24T12:51:57Z 4
2026-10-24T19:54:18Z 72 lock,nginx,nginx.pid,
2026-10-24T17:18:02Z 28 assets,config,library,
2026-10-24T19:24:43Z 4
2026-10-24T12:50:39Z 4
2026-10-24T13:08:05Z 4
2026-10-24T12:51:49Z 4
2026-10-24T19:07:30Z 32 documentserver,documentserver-example,
2026-10-24T19:06:36Z 4
2026-10-24T19:07:39Z 4
2026-10-24T13:08:14Z 4
2026-10-24T17:17:24Z 4
2026-10-24T12:50:57Z 4
2026-10-24T17:17:24Z 28 assets,config,library,
2026-10-24T19:55:03Z 24 client_body,fastcgi,proxy,
2026-10-24T19:55:03Z 72 lock,nginx,nginx.pid,
2026-10-24T13:08:14Z 72 lock,nginx,nginx.pid,
2026-10-24T19:07:39Z 36 documentserver,documentserver-example,
2026-10-24T19:54:35Z 72 lock,nginx,nginx.pid,
2026-10-24T17:20:41Z 4
2026-10-24T12:51:49Z 72 lock,nginx,nginx.pid,
2026-10-24T13:07:11Z 4
@@ -0,0 +1,3 @@
GET /claim HTTP/2 200
csrf_len 64 cookie_present yes
POST /claim HTTP/2 302 location: /
@@ -0,0 +1,11 @@
candidates 200 {"data": {"vmid": 9201, "initialize": [{"device": "/dev/sdb", "size_bytes": 214748364800, "model": "QEMU HARDDISK", "data_bearing": false, "mountable": false}, {"device": "/dev/sdc", "size_bytes": 161061273600, "model": "QEMU HARDDISK", "data_bearing": false, "mountable": false}], "attach": []}, "ok": true}
12:50:04Z init /dev/sdb hdd_1 200 {"data": {"phase": "formatting", "started": true}, "ok": true}
status 200 {"device": "/dev/sdb", "durable_id": "", "error": "", "opsign": "", "phase": "done", "reason": "", "started_at": "2026-10-24T12:50:04.496400956Z", "updated_at": "2026-10-24T12:50:06.691612961Z", "where": "/mnt/felhom-drives/hdd_1"}
12:50:09Z init /dev/sdc hdd_2 200 {"data": {"phase": "formatting", "started": true}, "ok": true}
status 200 {"device": "/dev/sdc", "durable_id": "", "error": "", "opsign": "", "phase": "done", "reason": "", "started_at": "2026-10-24T12:50:09.550638661Z", "updated_at": "2026-10-24T12:50:11.762283786Z", "where": "/mnt/felhom-drives/hdd_2"}
9dcbe7bf-098d-4d99-9c43-f39fe2b20f42 disk user-data attached 157 GB /mnt/felhom-drives/hdd_2
belso internal user-data attached 59 GB /mnt/felhom-drives/belso
e8b4f4a0-7aca-4829-9220-b394948b9368 disk user-data attached 210 GB /mnt/felhom-drives/hdd_1
local system system attached 77 GB None backup_target
local-lvm system system attached 165 GB None
felhom-pbs network backup attached 0 GB None
@@ -0,0 +1,8 @@
filebrowser gtstef/filebrowser:1.5.6-stable Up 9 hours (healthy)
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0 Up 6 minutes (healthy)
traefik traefik:v3.7.13 Up 9 hours
54
Filesystem Size Used Avail Use% Mounted on
/dev/sdb 196G 268M 186G 1% /mnt/felhom-drives/hdd_1
/dev/sdc 147G 2.2M 140G 1% /mnt/felhom-drives/hdd_2
/dev/mapper/pve-vm--9201--disk--0 32G 1.2G 29G 4% /
Binary file not shown.

After

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 57 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 61 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 60 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 67 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 55 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 60 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 60 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 55 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.9 KiB

@@ -0,0 +1 @@
Install pressed 12:37:33
Binary file not shown.

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 42 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 65 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 65 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 141 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 141 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 29 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 29 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 50 KiB

Loaded 100 of 1607 files, more files were not shown because too many files have changed in this diff. Show more