From 1a788f332dc7b2a21ce144207db9a9808a11326f Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sat, 24 Oct 2026 09:18:44 +0200 Subject: [PATCH] Attended release run: hub 0.168.0 deployed (R-975 closed; the live file's residue accepted by the operator, 05 16.2); D40 in effect (R-1243 VERIFY, dated check 10-25); D41/D42 closed; the ep0 iperf3 line-test server (scripts/ep0-iperf3, 06 5) and hub 0.169.0 models line_test (System page cell); R-1249, R-1250 filed; register 194 -> 193 --- CONTEXT.md | 6 +++ STATUS.md | 7 +-- .../architecture/05-hub-architecture.md | 10 ++++ .../architecture/06-offsite-connectivity.md | 5 ++ .../architecture/07-backup-architecture.md | 4 +- documentation/architecture/08-alarm-ladder.md | 18 +++++-- .../felhom-system-poster.facts.md | 2 +- .../architecture/felhom-system-poster.html | 2 +- .../audits/line-test-2026-10-24/README.md | 14 ++++++ .../line-test-2026-10-24/ep0-install.txt | 45 +++++++++++++++++ .../hub-0168-seal-check.txt | 18 +++++++ documentation/backlog/CLOSED-ITEMS.md | 8 +++ documentation/backlog/OPEN-ITEMS.md | 12 ++--- documentation/backlog/STATUS-HISTORY.md | 10 ++++ hub/CHANGELOG.md | 5 ++ hub/internal/sysfacts/sysfacts.go | 20 +++++++- hub/internal/sysfacts/sysfacts_test.go | 14 ++++++ hub/internal/web/system.go | 32 ++++++++++++ hub/internal/web/system_line_test.go | 35 +++++++++++++ hub/internal/web/templates/system.html | 1 + scripts/CHANGELOG.md | 9 ++++ scripts/ep0-iperf3/README.md | 22 +++++++++ scripts/ep0-iperf3/felhom-iperf3.service | 49 +++++++++++++++++++ scripts/wire_contract_gate.py | 1 + 24 files changed, 328 insertions(+), 21 deletions(-) create mode 100644 documentation/audits/line-test-2026-10-24/README.md create mode 100644 documentation/audits/line-test-2026-10-24/ep0-install.txt create mode 100644 documentation/audits/line-test-2026-10-24/hub-0168-seal-check.txt create mode 100644 hub/internal/web/system_line_test.go create mode 100644 scripts/ep0-iperf3/README.md create mode 100644 scripts/ep0-iperf3/felhom-iperf3.service diff --git a/CONTEXT.md b/CONTEXT.md index cfa45a41..b0002b3d 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -26,6 +26,12 @@ > 2026-10-23: „during a quiet window the box measures the upload bandwidth"); the passive floor stays as a lower bound. > This is the one change on ep0 the run may make (Tier 2 — the operator's yes is starting the run). CC still stops > before each hub deploy, before the global controller floor, and for the agent vouch. +> **Two more answers in the same run (AskUserQuestion, 2026-10-24 ~08:2x CEST):** (1) **the agent speaks iperf3 itself** +> (no host has the iperf3 program; nothing is installed on hosts) — agent `internal/linetest`; (2) **the live hub file's +> old token bytes are accepted, not cleaned** (the start-up VACUUM did not finish on the Longhorn disk, R-1249; a clean-up +> needs an 8–20 min hub stop, a trim and the volume's snapshots deleted; only root on DooPlex can read the volume, and it +> holds the key; every copy that leaves is clean) — written into `05` §16.2 with the rule „a hand copy of hub.db is made +> with VACUUM INTO, never cp/tar". > **2026-10-23 19:22 — operator rulings by starting the weekend-night brief (R-1241's Felhom half; the upload watch).** > **Part 1 = R-1241's Felhom half:** seal R-975 (done in hub 0.168.0, not deployed — `05` §16.2, diff --git a/STATUS.md b/STATUS.md index 47b0dbf5..42d734d6 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-10-23 (night).** The hub's last unlocked tokens are now locked in a new hub version — built and tested, you deploy it on Monday. The new upload watch is built and tested on the scratch box, not released: it finds an app that uploads or computes far more than the app itself explains (a hacked app) and tells only you; qBittorrent gets an upload limit. +**Updated 2026-10-24 (day, attended).** Hub 0.168.0 is live: the last plain tokens are locked, and every copy of the hub database that leaves DooPlex is clean. The hub is out of DooPlex's open backup folder. Each box can now test its upload once a week (demo-hp: 90.7 Mbit/s, not 10); the releases follow on your yes. ## What works - **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241). @@ -30,9 +30,6 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked | |---|---|---|---|---|---|---| -| D42 | R-1246 | When the box limits or flags something, who is told? | **The family only about the qBittorrent limit (they can change it); the hacked-app alarms only you** (the family can do nothing about a hacked app) | tell the family too: a worrying mail they cannot act on | the same as my pick | — | -| D41 | R-1245 | How much of the measured upload may qBittorrent use by default? | **50 %** (the family keeps half the line; seeding took the whole line on the test bench) | 30 %: slower seeding; none: only the alarm | 50 % when you release the watch | — | -| D40 | R-1243 | The hub's backups in DooPlex's shared folder are not encrypted (two there now hold the tunnel tokens). Encrypt the hub's disk, or stop that backup? | **Stop it** (the hub's own copy to ep0 is encrypted, every 6 h, tested weekly) | encrypt: 5–10 min hub downtime and one more key to keep | the plain copies stay readable by any hacked DooPlex app | a homelab change | | D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — | | D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — | | D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — | @@ -54,7 +51,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te 2. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-16fdef. If not: two dead entries alarm. 3. D37 and D38 in the table (two small yes/no). If not: Gitea may restart in release hours; no golden in the release copy. 4. Give the homelab session R-1241, R-1242, the 7-item list at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md`, and an upload alarm on DooPlex's `eno1` (the line is ~100 Mbit/s; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups. -5. Monday: deploy hub 0.168.0 (the last plain tokens locked, R-975); then release the upload watch on your yes (R-1244; D41, D42 first). If not: a hub copy still holds the tokens, and a hacked app on a family's box stays unseen. +5. Today (2026-10-24, attended): hub 0.168.0 deployed; D40–D42 ruled and put into effect. Still open: the agent vouch, hub 0.169.0, the controller floor (R-1244). If not: a hacked app stays unseen. **The rest** - System poster needs a refresh: the new 01:40 „release copy” arrow from DooPlex to ep0 (D33); the hub-database label already says every 6 hours. If not: the drawing misses one copy. - Pull the failed disk Z6A07P2G out of DooPlex (R-934). If not: its alarms return on 11 November. Since 2026-10-12. diff --git a/documentation/architecture/05-hub-architecture.md b/documentation/architecture/05-hub-architecture.md index 3359c9d9..74b1f652 100644 --- a/documentation/architecture/05-hub-architecture.md +++ b/documentation/architecture/05-hub-architecture.md @@ -565,6 +565,16 @@ a `config_version` bump — the box's config is byte-for-byte the same, so no bo (`TestR975_DBFileWithoutKeyRevealsNoToken`). `-unseal-box-secrets` opens these leaves too, so the roll-back above also covers a return to a hub before v0.168.0. The full list of secret-bearing columns, measured on a copy of the live database 2026-10-23: `audits/hub-token-seal-2026-10-23/README.md`. +**Deployed 2026-10-24 — and what the live file still holds (operator ruling 2026-10-24, accepted).** On the live hub the +start-up `VACUUM` did not finish: the Longhorn disk writes ~1.3 MB/s (the 6-hourly `VACUUM INTO` of 530 MB takes ~7 min) +and the liveness probe stopped the first start after ~90 s; the second start found nothing left to seal (R-1249). So the +rows are sealed, but the live `hub.db` and its `-wal` still carry the old token bytes in freed space (8 of 8, measured). +**Every copy that leaves the volume is clean**: the hub's snapshot and the ep0 copy are written by `VACUUM INTO` (the ep0 +copy of 06:30Z: 0 of 8), and the volume is out of DooPlex's Longhorn backup (D40). Accepted, not cleaned: whoever can read +the live volume is root on DooPlex and can read `OFFSITE_SECRET_KEY` there too; a full wipe would also need a hub stop of +8–20 minutes, a disk trim and the volume's Longhorn snapshots deleted. **Rule: a hand copy of `hub.db` for analysis is +made with `VACUUM INTO` (or taken from ep0), never `cp`/`tar` of the live file** — a `tar` of the file carries the old +bytes (`audits/line-test-2026-10-24/hub-0168-seal-check.txt`). **And the key is the other half:** a backup of the database restores a hub that can open the sealed columns only with the same `OFFSITE_SECRET_KEY`; today that key exists only on DooPlex (the k8s Secret, and the GPG secrets export on the same machine). The off-site plan for the database and its key: `runbooks/RUNBOOK-hub-db-offsite-backup.md` diff --git a/documentation/architecture/06-offsite-connectivity.md b/documentation/architecture/06-offsite-connectivity.md index bf017634..1cd99c09 100644 --- a/documentation/architecture/06-offsite-connectivity.md +++ b/documentation/architecture/06-offsite-connectivity.md @@ -297,6 +297,11 @@ production endpoint exists. ## 5. The endpoint VM (the one new managed-infra piece) +**AMENDED 2026-10-24 (R-1247, operator ruling):** a fourth, small thing — `felhom-iperf3.service`, an iperf3 server bound to +`10.77.0.1:5201` and opened in nftables for `wg0` from `10.77.0.0/24` only, so each box's agent can measure its upload +line once a week (felhom.eu `scripts/ep0-iperf3/`; agent `internal/linetest`). It never answers on the internet (proven +2026-10-24, `audits/line-test-2026-10-24/`). + Runs three things, nothing else: the WG server (peer list = hub-written), PBS (one datastore, per-customer namespaces), and the minimal peer-sync surface the hub drives. **How** the hub mutates the peer list — a tiny authenticated push API on the VM vs the VM pulling a signed peer diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index 526d6559..5dbef462 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -1529,7 +1529,7 @@ crosses the line — **R-158**. | 8 | **Host dies (hardware), drives intact** | the data drives; ep0's PBS namespace; the hub's Recipe + Escrow | **install a new host, then** `--selftest=bring-up -mode dr` per guest, **then** re-attach drives by `durable_id` | **operator** (SSH) | | 7 d | **IMPLEMENTED** **`[BETA-DEFERRED]`** | bring-up code exists and has **never been executed** (`CAMPAIGN-8…:522`); the drive half of the plan is empty on every box (**R-105**) | | 9 | **Whole box lost (fire/theft) — host and drives gone** | ep0 PBS namespace; the restic repo; the hub's Recipe + Escrow | new hardware → day-0 → escrow-consume with **R** → restore guests from PBS → app data from Tier-3 | **operator + customer** (R) | | 7 d (guest) · 24 h (app data) | **IMPLEMENTED / UNPROVEN** **`[BETA-DEFERRED]`** | every leg exists; the composed path has never been run. The destructive S5 drill is operator-gated and unrun (`06-offsite-connectivity.md:327`) **`[FACT]` 2026-10-03 (decision 70): ep0's datastore now has a nightly copy on DooPlex** (PBS pull-sync `ep0-felhom-offsite`, `remove-vanished false`, weekly verify, failures mailed to the operator); first pull 201 s, 12 GB, 4 of 4 snapshots, matching ep0 namespace for namespace. Restore route: `runbooks/ep0-datastore-copy.md` — never walked. `audits/offsite-lock-build-2026-10-03/partF/`. **`[FACT]` 2026-10-04: the restore route from the DooPlex copy was WALKED** on demo-hp (scratch VMID): listed in 2 s, restored in 186 s (15 GB logical), data read by `pct mount`; the restored config carries `onboot: 1` and the host's real drive binds — strip them on a restore beside the original (R-834). The copy now keeps 8 weekly copies (decision 71). `runbooks/ep0-datastore-copy.md` route 1. | | 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** **`[BETA-DEFERRED]`** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. **2026-09-01 (SPIKE R-95, `audits/SPIKE-r95-offsite-delete-2026-09-01.md`) — THIS ROW'S STATUS IS UNCHANGED, but the seven-day bound beside it is now known to be unverified.** Measured on BOTH boxes over their own SFTP credential, with controls: **no `.snapshots` is visible to either sub-account**, and the account is jailed. Either none exist or a sub-account cannot see them — and a snapshot the box cannot see is one the box cannot restore from, so recovery would be an operator act at the Hetzner panel, not a product capability (R-429; the confirming field needs the provider API, fenced by §11-D). **The PBS shape does NOT transfer**: the sub-account API has one permission axis, `readonly`, and a backup target cannot be read-only — PBS is a server that can refuse, a Storage Box is a filesystem that runs nothing. **What the spike removed as a fear:** withdrawing delete does NOT wedge the store (measured — restic treats a dead owner's lock as stale and proceeds). **What it added:** `unlock --remove-all` reports success while deleting nothing (R-430). **2026-09-01, LATER THE SAME DAY — THE PROBE ABOVE LOOKED FOR THE WRONG NAME AND THIS ROW'S SECOND CLAUSE IS NOW HALF WRONG.** It searched `.snapshots`; the vendor documents `/.zfs/snapshot`. Re-probed at the documented path on BOTH boxes, with controls, the two halves separate cleanly: **(a) the LIVE repository is deletable by the box — unchanged, R-95 stands;** **(b) the daily SNAPSHOTS of it are not writable by anything — PROVEN, not cited:** a write into `/.zfs/snapshot` is refused (`dest open …: Failure`) while the identical write to the account home succeeds. Seven daily snapshots confirmed in the panel (R-429). **So this row's "the restic repo is NOT protected the same way" is true of the repository and FALSE of its snapshots** — a deletion costs at most the day since the last snapshot, recoverable per-file (vendor). **Two limits kept honest:** a sub-account sees the snapshot directory EMPTY, so per-file recovery is operator-only today (R-432); and a panel-driven restore rolls back the WHOLE box. **The status is NOT moved** — evidence (b) is measured, but the recovery ROUTE has never been walked, which is what PARTIAL means. **Detection shipped hub v0.111.0 (R-431):** an unexplained fall in the snapshot count is noticed within a day — **but only a fall of MORE than half (R-435).** **2026-09-01, THE DRILL — CLAUSE (b) OF THE RE-SCOPE ABOVE IS WITHDRAWN; THE STATUS IS STILL NOT MOVED.** The re-scope said a deletion is *recoverable per-file*. **It is not, from the box: no snapshot is reachable by ANY name.** MEASURED on demo-hp, read-only, no delete verb issued: **777,600** exact names in the vendor form `YYYY-MM-DDTHH-MM-SS` (nine full days, second granularity) plus 126 alternative shapes — **zero hits**, against a control where the identical 600-name batch returns a path that exists (6/6). **Structural cause:** `/home` (`u629488-sub3`) is **st_dev 0,82**, `/.zfs/snapshot` is **st_dev 0,276**, and `/home/.zfs` does not exist — a snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding the repository. **Clause (a) — the box cannot WRITE into the snapshot area — is unchanged and re-confirmed.** So this row's *"recoverable per-file (vendor)"* and its limit *"per-file recovery is operator-only today (R-432)"* both overstate what exists: the remaining routes are a panel rollback of the WHOLE Storage Box and the provider API (fenced, and the hub's client has no snapshot method at all). **R-432 ANSWERED negatively; R-433 opened. RTO still blank — nothing was recovered, so nothing was timed.** `audits/evidence-drill-r95-recovery-2026-09-01/`. **`[FACT]` 2026-10-03 (decisions 68–69, hub v0.127.0, controller v0.289.1): THE BOX CAN NO LONGER DELETE ITS RESTIC HISTORY.** Both demo boxes run on an append-only key the hub pinned; a delete from the box is refused (`403 Forbidden`, measured on each, counts 13→13 and 100→100); the box never receives the sub-account password (the old endpoint answers 410, measured with demo-hp's own key); the hub reads every key file daily. Retention runs only in a hub-opened window behind a fake-snapshot guard — mechanics proven live (window 1 on demo-hp: opened, guard refused, closed in 3 s, operator mailed); **a real prune inside a window is NOT yet proven** (R-824). Residual: an add-only attacker can still fill the quota and plant past-dated snapshots (R-822). `audits/offsite-lock-build-2026-10-03/`. **`[FACT]` 2026-10-04 (controller v0.290.0, hub v0.128.0):** the guard now skips same-day-superseded young snapshots instead of refusing; window 2 on demo-hp ran without refusal and removed nothing (127→127 — every candidate was a young same-day copy); weekly windows are ON fleet-wide. **A window that actually removes snapshots has still not been observed** (first candidates age past 8 days around 2026-10-11). The household's set-aside deletion is the hub's after 7 days (decision 74), proven live on a planted scratch dir. **`[FACT]` 2026-10-05 (controller v0.294.0, R-867 CLOSED): A WINDOW THAT REMOVES SNAPSHOTS IS OBSERVED, on both demo boxes.** The guard's line is now built from the policy's own constants (keep-daily 7 CALENDAR days, `09` decision 104) — the 8-day age line sat inside the keep window and refused every honest window (demo-felhom 2026-10-05 02:15 UTC, an error mail). One window each, opened by the operator's one-shot grant and run by the household's "run now": demo-felhom 16 → 14, demo-hp 145 → 127 — the removed snapshots are exactly those the policy predicted; the hub's rows read `pruned` with the same counts, no drop event, no mail, the key files clean. The UNATTENDED weekly window has not yet been observed removing (next due ~2026-10-11/12). Residual unchanged: R-822 (past-dated gap-fills steer older keeps). `audits/night-fixes-2026-10-05/partA/`. **`[DESIGN]` 2026-10-08 (R-435, `09` §3 decision 191; hub main, unreleased): on a pinned tier even ONE snapshot that disappears outside a hub-opened window raises an error mail** — each window explains at most its hub-set cap; NAS tiers keep the half-rule; the count is the box's own and net (`08` §6.5). **2026-10-08 (R-893, decision 192; controller main, unreleased):** a failed off-site replay of one app after a moved definition or a replaced volume now HOLDS the app stopped for support instead of starting it on mixed data (§6.3); „put back exactly as it was" is the next slice. **`[DESIGN]` 2026-10-18 (R-973; agent `main`, unreleased):** re-running the whole-guest backup to push out older restore points is capped per guest and tier (§6.1, the R-973 note); the local tier stays floodable within a day (R-1149). | -| 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its Longhorn volume backup | operator (`kubectl`) | | 24 h + weekly (Longhorn `RETAIN 1` each) | **UNPROVEN** **`[BETA-DEFERRED]`** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) | +| 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its encrypted database copy on ep0 (6-hourly, `scripts/hub-db-backup`; **since 2026-10-24 the hub volume is no longer in DooPlex's Longhorn backup**, D40 / R-1243) | operator (`kubectl`) | | 6 h (ep0, 14 daily + 8 weekly) | **UNPROVEN** **`[BETA-DEFERRED]`** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) | | 11b | *consequences while the hub is gone* | — | — | — | | | **[FACT]** **`[BETA-DEFERRED]`** | **day:** nothing customer-visible breaks; events queue (`settings.go:1466-1485`). **week:** the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. **permanently:** escrow custody and break-glass credentials are gone (INV Part D2.4) | | 12 | **Offsite provider lost (Hetzner)** | everything on-premises: both drives, both whole-guest tiers | none needed — on-premises recovery is unaffected. Re-provision a new offsite target. | operator | | | **[FACT]** **`[BETA-DEFERRED]`** | **restic and PBS share the provider** (INV Part E.1). Whether they share an account and payment method is **UNKNOWN** → §11-D **`[FACT]` 2026-10-03:** the PBS (whole-guest) off-site copy now also exists OFF Hetzner — DooPlex pulls ep0's datastore nightly (decision 70). The restic tier is still Hetzner-only. | | 13 | **Customer loses R** | every byte, every tier, the Recipe | **on-premises recovery is unaffected** — Tier-1/2/3 and the whole-guest tiers all work while the guest and host live. What is lost: the ability to re-establish **host identity** and to open the escrowed offsite key material after a host loss | — | | | **NONE for host-loss** | R exists in **zero** system copies by design (INV Part C, row 24). Whether the operator should be able to recover is **open** → §11-A, §11-B. **D5 narrows this further:** Tier-1/2 app recovery needs only the drive, so losing R now costs the offsite route and host identity, never local app recovery (§7.4) | @@ -1875,7 +1875,7 @@ to now *implement* D5 remains an open scheduling decision, not a blocked one. | host-loss plan → an actual restore | **EXECUTES NOTHING BY CONSTRUCTION** | `felhom-agent/internal/dr/plan.go:1-4` | | host rebuilt as its former self | **NEVER DONE** | INV Part D1 | | escrow **consume** in a real recovery | **SPIKE-LEVEL ONLY** | `06-offsite-connectivity.md:327` | -| hub DB restore from its Longhorn backup | **NEVER DONE** | INV Part D2.2 | +| hub DB restore from its ep0 copy into a running hub (the weekly restore-test opens the copy and checks it; a hub has never been brought up on it) | **NEVER DONE** | INV Part D2.2; the Longhorn backup it used to cite is off since 2026-10-24 (D40) | | a **customer** performing a restore unassisted | **MISSING AS EVIDENCE** | `00-capability-map.md:75` | --- diff --git a/documentation/architecture/08-alarm-ladder.md b/documentation/architecture/08-alarm-ladder.md index 0b06a6dc..fc90d93b 100644 --- a/documentation/architecture/08-alarm-ladder.md +++ b/documentation/architecture/08-alarm-ladder.md @@ -544,7 +544,13 @@ normal use; work it does not explain is the sign of a hacked app. Numbers and me 1. **The line.** Learnt from what the box already does: while its off-site push runs, the controller samples its own sent bytes once a second; the best ≥ 10-second rate is kept per night (14 nights, settings.json `upwatch.line`); a night counts when that window moved ≥ 10 MB; the line is the highest counting night. A **lower bound**, shown so - („legalább N Mbit/s", system settings). An active test is not built (R-1247). + („legalább N Mbit/s", system settings). **AMENDED 2026-10-24 (operator ruling 07:26, R-1247): the weekly line TEST.** + The night's one-second read showed how low that floor is (demo-hp „at least 10 Mbit/s"; spike §2.1). So the host's + agent (agent ≥ 0.164.0, `internal/linetest`) runs a 10-second iperf3 upload once a week to ep0's tunnel-only server: + Tuesday 10:00–15:59 local from a per-host slot, only after 10 quiet minutes of the physical uplink (< 2 Mbit/s each + minute), under the one-heavy-op gate, best of 3, never in the night chain; „server busy" → later the same day. The + controller reads it (`GET /host/line-test`) every 15 minutes; the watch judges against the higher of test and floor; + the system page says „N Mbit/s (heti sebességméréssel)". demo-hp measured 90.7 Mbit/s (`audits/line-test-2026-10-24/`). 2. **Trigger, per app** (its compose project's containers summed; exact counters from the Docker Engine API, once a minute): upload ≥ max(5 Mbit/s, 25 % of the line) for 15 minutes; CPU ≥ max(150 %, 50 % × cores) of one core for 30 minutes. Infrastructure (the controller — its own push is the line measure —, traefik, cloudflared, Samba) is never @@ -556,12 +562,14 @@ normal use; work it does not explain is the sign of a hacked app. Numbers and me 4. **Who is told** (the R-388 rule, §8.1): **`app_unexplained_upload` / `app_unexplained_cpu`** (warning) and their `_cleared` (info, recorded) are **operator-only** (hub `operatorOnlyEvents`), one per episode, per-app cooldown. The household is told only about the qBittorrent limit, because they can change it (R-1246 asks the operator). -5. **qBittorrent's default upload limit:** 50 % of the line (R-1245), every 15 minutes, re-set when the line moves - > 20 %; never over a limit the household set (one there first, or any change from the value qBittorrent stored for - the box — it keeps KiB/s). No line → none set. While the box owns it, qBittorrent's page says: „A qBittorrent +5. **qBittorrent's default upload limit:** 50 % of the TESTED line (D41 = 50 %, operator 2026-10-24; never of the + push floor), every 15 minutes, re-set when the line moves > 20 %; never over a limit the household set (one there + first, or any change from the value qBittorrent stored for the box — it keeps KiB/s). No test → none set. **Known + gap (R-1250):** a qBittorrent that seeds all day keeps the uplink from ever being quiet for 10 minutes, so that box is + never tested and gets no limit — the case the limit exists for. While the box owns it, qBittorrent's page says: „A qBittorrent feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod." 6. **Limits, stated:** an attacker who knows the check can make the app's API lie — a per-container ceiling (`tc` from a helper in the app's network namespace, measured to work in an unprivileged guest; spike §5) would cap the flood whatever the API says; not built. The box cannot tell LAN from internet bytes per app; a host-network container - (Samba) has no counter; the line is a lower bound. + (Samba) has no counter; the push floor is a lower bound, the weekly test is close to the line. 7. **Release:** not released; the operator's yes (R-1244). diff --git a/documentation/architecture/felhom-system-poster.facts.md b/documentation/architecture/felhom-system-poster.facts.md index 452cb1b0..8839621c 100644 --- a/documentation/architecture/felhom-system-poster.facts.md +++ b/documentation/architecture/felhom-system-poster.facts.md @@ -97,7 +97,7 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1– ## Zone 2 — the internet - **Cloudflare** (grey): DNS for the customer domains and for felhom.eu; the **tunnels**; the **geo-block** (only Hungarian visitors); flood protection. The hub holds a Cloudflare key per customer zone and checks that it reaches only that customer's own domain. -- **ep0** (grey box with an orange stripe, a Hetzner cloud server in Germany): runs the **WireGuard endpoint** and the **off-site backup server (PBS)**. Inside, one datastore with **namespaces**: +- **ep0** (grey box with an orange stripe, a Hetzner cloud server in Germany): runs the **WireGuard endpoint**, the **off-site backup server (PBS)** and, since 2026-10-24, a small **line-speed test server** (iperf3) that answers only inside the tunnel — each box measures its upload once a week (R-1247). Inside, one datastore with **namespaces**: - one namespace **per household** (whole-guest copies, each household's own key), with a NEUTRAL note under it (not a red gap box): "By design: each household holds the key to its own copies; the operator cannot open them without their recovery code." — *fix (c), 2026-10-09: this is a property of the design, not a gap, and a red box said the opposite of what it means.* - the **`operator`** namespace: two groups — **"hub database"** and **"DooPlex copy"** (code, password manager, secrets, signing keys). diff --git a/documentation/architecture/felhom-system-poster.html b/documentation/architecture/felhom-system-poster.html index 3ab0e626..5ddd84be 100644 --- a/documentation/architecture/felhom-system-poster.html +++ b/documentation/architecture/felhom-system-poster.html @@ -116,7 +116,7 @@ code{font-family: ui-monospace, "SFMono-Regular", Menlo, monospace; font-size: 1
iso.felhom.eu
households download the installer
web pages
ep0 — Hetzner cloud server, Germany
-
WireGuard endpoint + off-site backup server (PBS). One datastore, split into namespaces:
+
WireGuard endpoint + off-site backup server (PBS) + a tunnel-only line-speed test (iperf3). One datastore, split into namespaces:
G One namespace per household
whole-guest copies, each household's own key
keeps one per week, 2 weeks
operator namespace
groups “hub database” and “DooPlex copy”
keeps 14 daily + 8 weekly
By design: each household holds the key to its own copies; the operator cannot open them without the household’s recovery code.
diff --git a/documentation/audits/line-test-2026-10-24/README.md b/documentation/audits/line-test-2026-10-24/README.md new file mode 100644 index 00000000..ff0d001a --- /dev/null +++ b/documentation/audits/line-test-2026-10-24/README.md @@ -0,0 +1,14 @@ +# The attended release run, 2026-10-24 — evidence + +Brief: „the hub's tokens locked, the hub out of the open backup, an iperf3 line test via ep0, then the upload watch +released" (operator rulings 07:22 / 07:26, `CONTEXT.md`). Read first: `05` §16.2, `06` §4.5/§5, `07` (hub row), +`08` §9, `audits/SPIKE-upload-watch-2026-10-23.md`. + +| File | What it shows | +|---|---| +| `hub-0168-seal-check.txt` | Hub 0.168.0 deploy: the first start killed in the VACUUM, the token search (counts only) on the pre-deploy ep0 copy (8 of 8), the live file (8 of 8), the post-deploy ep0 copy (0 of 8); demo-hp's served config equal to what it runs | +| `ep0-install.txt` | ep0: iperf3 package, the unit's caps, the listener on 10.77.0.1:5201 only, the nftables diff, the reachability proof (internet: no answer, `:22` control open; tunnel: open), the protocol proof (90.7 Mbit/s; busy in 0.1 s) | + +Longhorn (D40): the hub PVC and volume read `recurring-job-group.longhorn.io/default: disabled`; the NFS folder holds no +copy of `pvc-486c9809…` (no directory, no backup config naming it); Longhorn lists no backup volume for it; 53 other +volumes were backed up this morning. The dated check 2026-10-25 reads tonight's job (R-1243). diff --git a/documentation/audits/line-test-2026-10-24/ep0-install.txt b/documentation/audits/line-test-2026-10-24/ep0-install.txt new file mode 100644 index 00000000..1f18e2be --- /dev/null +++ b/documentation/audits/line-test-2026-10-24/ep0-install.txt @@ -0,0 +1,45 @@ +# ep0 iperf3 install, 2026-10-24 ~06:12Z (CC, attended release run; operator ruling 07:26) +Sat Oct 24 06:25:34 AM UTC 2026 +ii iperf3 3.18-2+deb13u2 amd64 Internet Protocol bandwidth measuring tool +disabled +enabled +active +RuntimeMaxUSec=1d +CPUQuotaPerSecUSec=500ms +MemoryMax=134217728 +TasksMax=16 +User=felhom-iperf3 +LISTEN 0 4096 10.77.0.1:5201 0.0.0.0:* users:(("iperf3",pid=3011910,fd=3)) +table inet filter { + chain input { # handle 1 + type filter hook input priority filter; policy drop; + ct state established,related accept # handle 3 + iif "lo" accept # handle 4 + tcp dport 22 accept # handle 5 + udp dport 443 accept # handle 6 + tcp dport 8007 iifname "wg0" accept # handle 7 + tcp dport 5201 iifname "wg0" ip saddr 10.77.0.0/24 accept # handle 13 + ip protocol icmp accept # handle 8 + meta l4proto ipv6-icmp accept # handle 9 + } +} +10a11 +> tcp dport 5201 iifname "wg0" ip saddr 10.77.0.0/24 accept # R-1247: felhom-iperf3 line test, tunnel only + +# Reachability, 2026-10-24 ~06:20Z (bash /dev/tcp connect, 5-8 s timeout) +From DooPlex over the internet (its public address 37.191.56.x) to 167.233.158.164: + 22 open (positive control: the method sees an open port) + 5201 no answer + 8007 no answer +From demo-hp's host (10.77.0.3) through wg-felhom to 10.77.0.1: + 5201 open + 8007 open (PBS, the existing tunnel service) + route: 10.77.0.1 dev wg-felhom src 10.77.0.3 + +# Protocol proof, 2026-10-24 ~06:31Z: a throwaway probe built from felhom-agent internal/linetest (Client, 4 streams, +# 10 s), copied to demo-hp's host /tmp, run against 10.77.0.1:5201 through wg-felhom, then deleted. +took 10.4s err= mbps=90.7 server_bytes=114032640 client_bytes=116046000 server_s=10.05 +# Two at once (the second started 2 s later): +took 0.1s err=iperf3 server busy mbps=0.0 server_bytes=0 client_bytes=0 server_s=0.00 +took 10.4s err= mbps=90.6 server_bytes=113901568 client_bytes=116629300 server_s=10.06 +# After: 0 ltprobe processes; /tmp/ltprobe removed. diff --git a/documentation/audits/line-test-2026-10-24/hub-0168-seal-check.txt b/documentation/audits/line-test-2026-10-24/hub-0168-seal-check.txt new file mode 100644 index 00000000..ae0c35cd --- /dev/null +++ b/documentation/audits/line-test-2026-10-24/hub-0168-seal-check.txt @@ -0,0 +1,18 @@ +# Hub 0.168.0 deploy, 2026-10-24 (attended release run). Counts only — no value was printed. +05:38:10Z first start: three seal lines, then no "config tokens" line; killed by the liveness probe at 05:39:39Z (exit 143) +05:39:39Z second start: "config tokens sealed at rest (0 legacy plaintext token(s) sealed now)"; Listening 05:40:18Z + -> the UPDATE committed in the first start; the VACUUM after it did not finish (538 MB on Longhorn; 2.8 s on local disk) + +Search of the 8 plain token values (read from the pre-deploy ep0 copy 2026-10-24T00:3xZ) as raw bytes: + pre-deploy ep0 copy hub.db 8 of 8 (positive control) + live hub.db (copied 05:41Z) 8 of 8 integrity ok, freelist 312 pages -> the plaintext sits in free space + live hub.db-wal 8 of 8 + live copy after a local VACUUM 0 of 8 (2.8 s) + post-deploy ep0 copy 06:30:57Z 0 of 8 integrity ok; 4 customer rows carry enc:v1: tokens + /data/snapshots: hub-20261023T180000Z.db, hub-20261024T000000Z.db are pre-deploy (plain); they roll off. + +Box side (demo-hp, 05:53Z): GET /api/v1/box-config with the box's own key: HTTP 200, 0 "enc:v1:" in the served config; + infrastructure.cf_tunnel_token and cf_api_token fingerprints EQUAL to the box's applied controller.yaml; + hub log "Config downloaded for customer demo-hp (by box key, 2ms)"; cloudflared Up 14 hours (healthy). +Gitea: hub log "Template fetched (5881 bytes)" at start. +All copies shredded after the check. diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index 7c068580..e82eea9c 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -26,6 +26,14 @@ --- +## 2026-10-24 — the hub's tokens deployed; D40–D42 ruled; the line test (full text of the rows: `git show 3363ffa8:documentation/backlog/OPEN-ITEMS.md`) + +| Row | What | Closed | Evidence | +|---|---|---|---| +| **R-975** | **The hub database kept every customer's tunnel token, Cloudflare API token and git token unsealed.** (P3) | CLOSED 2026-10-24 — hub **0.168.0** deployed on the operator's yes (image sha256:3bfdecaa…; ArgoCD sync of the hub Deployment only, `00031150`): the three `config_json` leaves are sealed; demo-hp's `GET /box-config` serves the same plain tokens its box runs (fingerprints equal, 0 `enc:v1:`); the ep0 copy of 06:30:57Z holds **0 of 8** token values (positive control: the pre-deploy copy, 8 of 8). The live file's old bytes in freed space (the start-up VACUUM did not finish, R-1249) are **accepted by the operator** (2026-10-24: only root on DooPlex can read the volume, and it holds the key; every copy that leaves is clean; rule in `05` §16.2) | `audits/line-test-2026-10-24/hub-0168-seal-check.txt` | +| **R-1245** | **DECIDE: how large is qBittorrent's default upload limit?** (P3) | CLOSED 2026-10-24 — **D41 = 50 %** (operator ruling 07:22), of the host's TESTED line (the weekly iperf3 test, R-1247 — never the push floor); built in controller 0.342.0 (`ed1011b`); released with R-1244 | `08` §9.5 | +| **R-1246** | **DECIDE: who is told when the box limits or flags something?** (P3) | CLOSED 2026-10-24 — **D42** (operator ruling 07:22): the family only about the qBittorrent limit (one line on its page); `app_unexplained_upload` / `app_unexplained_cpu` are operator-only (hub 0.169.0); released with R-1244 | `08` §9.4 | + ## 2026-10-23 day (later) — D39 = B: the security-fix lane | Row | What | Closed | Evidence | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 809e0fc4..2912f635 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -222,7 +222,7 @@ stopping line that lies. | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| -| **R-1241** | Security & access | P1 | **A hacked app pod on DooPlex can read the hub's nightly database copies — with every customer's unsealed Cloudflare, tunnel and git token (R-975) — through the Longhorn NFS export, and by a published file-handle escape the operator signing keys on the same disk: whoever breaks one homelab app can take over the boxes' tunnels and sign agent updates.** MEASURED 2026-10-23 (Plex2Shell impact audit, `audits/plex2shell-felhom-impact-2026-10-23/`): export `/mnt/5_hdd/backup/longhorn-pvc` rw, `no_root_squash`, `no_subtree_check`, a sub-directory of sda1, which also holds `/mnt/5_hdd/felhom.eu/felhom-op-*` (no passphrase, R-924); `secure` is void in pods (containerd sets `ip_unprivileged_port_start=0`; a uid-1000 probe with no capabilities bound port 700); the four `felhom-system` volumes have `encrypted=false`; Longhorn's API on :9500 answers any pod with no login (backups, targets, volumes). In the Plex2Shell window neither door was used — verdict NOT REACHED (nfsd accepted 0 new connections 10-10 → 10-23, positive control; no off-schedule Longhorn backup or volume). 2026-10-23 ~15:18Z, operator yes: the export was narrowed from `*` to `10.42.0.0/16` — the home network and the tailnet are closed, pods are not (Longhorn connects from its pod address). Not exercised: the escape itself (Tier-2 machine). | OPEN | the homelab session (DooPlex host + cluster); the operator (R-924) | Homelab: a NetworkPolicy so only `longhorn-system` reaches :2049/:111 and :9500, or the backup store on its own filesystem. Felhom: R-975 sealed in hub 0.168.0 (deploy pending); the hub's plain Longhorn copies → R-1243; a passphrase on the signing keys (R-924) | operator | +| **R-1241** | Security & access | P1 | **2026-10-24: the Felhom half is DONE** — hub 0.168.0 deployed (the tokens sealed; every copy leaving the volume is clean, the live file's old bytes accepted by the operator, R-975 closed), the hub volume out of the Longhorn backup (D40, R-1243 VERIFY), the two old plain copies deleted by the operator. **What stays here is the homelab half:** the open NFS export and Longhorn's login (the homelab session). **A hacked app pod on DooPlex can read the hub's nightly database copies — with every customer's unsealed Cloudflare, tunnel and git token (R-975) — through the Longhorn NFS export, and by a published file-handle escape the operator signing keys on the same disk: whoever breaks one homelab app can take over the boxes' tunnels and sign agent updates.** MEASURED 2026-10-23 (Plex2Shell impact audit, `audits/plex2shell-felhom-impact-2026-10-23/`): export `/mnt/5_hdd/backup/longhorn-pvc` rw, `no_root_squash`, `no_subtree_check`, a sub-directory of sda1, which also holds `/mnt/5_hdd/felhom.eu/felhom-op-*` (no passphrase, R-924); `secure` is void in pods (containerd sets `ip_unprivileged_port_start=0`; a uid-1000 probe with no capabilities bound port 700); the four `felhom-system` volumes have `encrypted=false`; Longhorn's API on :9500 answers any pod with no login (backups, targets, volumes). In the Plex2Shell window neither door was used — verdict NOT REACHED (nfsd accepted 0 new connections 10-10 → 10-23, positive control; no off-schedule Longhorn backup or volume). 2026-10-23 ~15:18Z, operator yes: the export was narrowed from `*` to `10.42.0.0/16` — the home network and the tailnet are closed, pods are not (Longhorn connects from its pod address). Not exercised: the escape itself (Tier-2 machine). | OPEN | the homelab session (DooPlex host + cluster); the operator (R-924) | Homelab: a NetworkPolicy so only `longhorn-system` reaches :2049/:111 and :9500, or the backup store on its own filesystem. Felhom: R-975 sealed in hub 0.168.0 (deploy pending); the hub's plain Longhorn copies → R-1243; a passphrase on the signing keys (R-924) | operator | | **R-1139** | Security & access | P2 | **Tester 2's break-glass root password was not rotated with the others (R-954, operator decision D1 = A, 2026-10-17): the box has been offline since 2026-10-04.** demo-hp, Tester 1 and demo-felhom were rotated and proven on 2026-10-17 (`audits/agent-allowlist-rotation-2026-10-17/P3/`). | OPEN — filed 2026-10-17 **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a fresh install sets and vaults a new console password).** | Tester 2 reporting again | When it reports: the installer's step 4b alone (`P3/rotate-step4b.sh` shape), then the vaulted value logs in (PVE API 200) and the old one is refused | CC | | **R-1042** | Security & access | P3 | **The boxes' Cloudflare keys cannot change zone settings, so Cloudflare's own „Always Use HTTPS” cannot be switched on.** Measured 2026-10-14 on all three zones (`audits/https-only-2026-10-14/``1A-1-zones-and-keys.txt`): each `cf_api_token` is valid and sees only its own zone, but GET/PATCH `settings/always_use_https`, `ssl`, `security_header` answer 403; the redirect and transform rule phases too. The https-only fix therefore lives on the box (decision 246). A second layer at Cloudflare needs „Zone Settings: Edit” (and, for headers, „Transform Rules”) added to each customer's key — the hub's zone-reach check stays. | **WAITING-ON-OPERATOR** — filed 2026-10-14 | a key permission only the operator can grant | Optional: widen each customer key, then switch „Always Use HTTPS” on per zone | operator | | **R-1242** | Security & access | P3 | **The demo NAS export `/mnt/5_hdd/felhom-data` → 192.168.0.162 (demo-felhom) is a sub-directory of DooPlex's sda1 with `no_subtree_check`, so a compromised demo-felhom host could walk the whole disk as uid 101000 and read every world-readable file there (the GPG secrets exports are 0644, every repo, the public keys).** Measured 2026-10-23 (`audits/plex2shell-felhom-impact-2026-10-23/` Part 1; `exportfs -v`: `all_squash,anonuid=101000`). 0600 files (the signing keys) stay out of reach. Not exercised. | OPEN | a DooPlex host change (fence) | Give the demo share its own filesystem, or `subtree_check`; or remove other-read from `/mnt/5_hdd/backup/secrets/exports` | operator | @@ -236,7 +236,7 @@ stopping line that lies. | **R-908** | Security & access | P3 | **An old Resend API key is still in the history of the `homelab-manifests` repository; it was replaced on 2026-06-29, but whether it is also revoked at Resend is unknown.** FOUND 2026-10-08 by the contact-mailer source search (`audits/mailer-source-2026-10-08/SEARCH.md`, „A side finding"): commit c9648cd (2026-02-05, „added mailer pod") put a key literal in the old `felhom-system/contact-mailer.yaml` comment; the file was deleted in ee93b50 but the history keeps it. Compared by hash only, never printed: it is NOT today's key (`Secret/resend-api`, rotated 2026-06-29, felhom.eu feea0606). If the old key still works at Resend, anyone with read access to that repository can send mail as felhom.eu. | **OPEN — owner: operator** **Note 2026-10-12 (security review):** gitleaks 8.24.3 with its default rules did NOT flag this key in the full history scan, so a scanner gate alone would not have caught it (R-986). | — | In the Resend console, check that the old key is revoked (revoke it if not); rewriting the repository's history is NOT needed once it is revoked | operator | | **R-525** | Security & access | P4 | **[P3-LOW] FileBrowser has its own login; putting it behind the dashboard session (traefik forwardAuth or Quantum proxy auth) is a new mechanism nobody has measured.** Filed 2026-09-15 by the P1-fixes task (B.5). R-513 closed the default-password hole with a generated password; a household still has two logins. **What it needs:** a spike on a scratch guest — forwardAuth to the controller session, and what FileBrowser Quantum does with a trusted header. | **READY — rank P3-LOW; owner: CC (spike)** **Re-ranked 2026-10-03: P3->P4: comfort feature needing a new unmeasured mechanism; the default password hole is closed.** | — | — | CC | | **R-925** | Security & access | P1 | **`manifests/felhom.secret.yaml` carries REAL secret values, and the Gitea repo that holds it is readable by anyone on the internet with no login.** FOUND 2026-10-09 by a background security review of the legal-pages commit, which flagged the served `` comments; chasing what those comments pointed at found this instead. **MEASURED, in this order:** (1) `scripts/manifest_bearer_gate.py` itself prints `manifests/felhom.secret.yaml:39 KNOWN-BACKLOG committed secret 65cee3c4…7a86`; (2) the file holds **live-shaped values, not placeholders** — `SECRET_KEY` (69 chars), `SUPERUSER_EMAIL`/`SUPERUSER_PASSWORD` (18), Umami `APP_SECRET` (66) and `POSTGRES_PASSWORD` (34), and a `username`/`password` pair (18); values were never printed, only measured by length; (3) `gitea.dooplex.hu` resolves to **37.191.56.193**, the same public address as the website; (4) an **anonymous** `curl` returns HTTP 200 and 1686 bytes for that exact file, and 200 for `hub/internal/store/store.go` and `manifests/hub.yaml`; (5) **the off-network control**, which is what makes (4) mean anything — fetched from outside the operator's network entirely: a real path returns the file's first line, a nonsense path returns 404, so the 200 is genuine anonymous read from the internet and not a LAN-only ACL. **This contradicts the project's own written model**: `documentation/runbooks/secrets.md:3-4` says *“Secret values are never committed to git. The manifests in `manifests/` carry only placeholders + comments that point here.”* That promise is false today, which is the P1 wording in this register's own scale — see the ranking note below. The same runbook already states the rule that governs the fix: *“The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing.”* **Known neighbours, none of which cover this:** R-887 records that an outside crawler walks the public Gitea pages and leaves *“Gitea's exposure to the crawler”* as the operator's undone item; R-580 still owes a Gitea admin token rotation. Neither says a secret file is world-readable. **Why P2 and not P1, stated so the operator can overrule it:** the exposed credentials guard **analytics and an undeployed healthchecks instance, not household data** — measured: `umami-db` is a **ClusterIP** service on 5432 with no external IP, so the Postgres password is not reachable from the internet; the realistic harm is session forgery against the public `stats.felhom.eu` via `APP_SECRET`, and reuse of those passwords anywhere else. No customer box, hub token or escrow key is in this file. **CC did NOT change anything**: making the repo private could break the public day-0 path (the installer is fetched from a public tag, R-110), and rotation plus repository visibility are operator decisions on production infrastructure. **PARTLY FIXED 2026-10-09, and RE-RANKED P2 -> P1 on what it turned out to be.** The exposed value was not an analytics password: it was the **Gitea `admin` account password** (`is_admin: true`; `/api/v1/admin/users` answered 200), published in a repo `gitea.dooplex.hu` serves anonymously to the internet. That is push access to every repo — including the one whose `website/` is git-synced live and whose `scripts/` is published by tag to **every new box installer** (R-110), i.e. a supply-chain path onto customer hardware. My first ranking said P2 because I had only measured the analytics blast radius; the credential test is what corrected it. **DONE (CC, operator-instructed):** (1) `umami-config` `APP_SECRET` + `POSTGRES_PASSWORD` rotated — the password changed **inside Postgres** too (`ALTER USER`), because the env var is only read at first init; verified by a real beacon returning 200 with a bogus-site-id control still returning 400. (2) `healthchecks-config` `SECRET_KEY` + `SUPERUSER_PASSWORD` rotated (nothing consumes them — there is no healthchecks Deployment). (3) `gitea-creds` no longer holds the admin password at all: it holds a **scoped token** (`read:package` + `read:repository`). **Both scopes are load-bearing and the second was found by breaking it** — a package-only token made the hub log `Template fetch: unexpected status 403`, because the template fetcher reads a raw file out of the `felhom-controller` repo. (4) The Gitea **admin password** was changed and `gitea-system/gitea-admin` updated; **verified: new password 200, old published password 401 — the leak is dead.** (5) The file is removed from git and `.gitignore`'s `*secret*` now applies to it; `manifest_bearer_gate.py`'s `KNOWN_BACKLOG` carve-out is gone, red-proofed with a decoy (exit 1 with, 0 without). **AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part:** the `rollout restart` needed to pick up the new umami secret put umami into **CrashLoopBackOff and took `stats.felhom.eu` down (503)**. Cause was not the rotation — at `memory: 512Mi` the pod runs for months but **cannot restart**, because startup (Prisma + Next.js) peaks over the limit and is OOMKilled (exit 137). Raised to 1Gi, **in the manifest, not just live** (the `.claude/rules/manifests.md` rule: a bare `kubectl set` is reverted by the next sync and the fix silently disappears). Service restored and verified. New values are in a 0600 file on DooPlex (`~/rotated-secrets-2026-10-09.txt`), never echoed; the operator moves them to the password manager and deletes it. **WHAT REMAINS, and it is bigger than what was fixed:** that one password is still the admin password in about **12 other namespaces** — `nextcloud`, `paperless`, `bookstack` (a **database root** password), `tandoor`, `calibre`, `adventurelog`, `gokapi`, `qbittorrent` (x2), `servarr`, `homepage` (x2). Rotating Gitea does not touch them: each is its own login and each is still the string that was published. **Also owed:** a Gitea token for user `kisfenyo` sits in plaintext in the `origin` URL of the local `homelab-manifests` clone (`.git/config`); CC printed it to a session transcript while investigating, so it should be rotated regardless — this is R-580's shape (store the remote without credentials). **Not a finding:** `homelab-manifests` itself is private (404 anonymously) and does not contain the password; ArgoCD's repo credential is a separate token, so the rotation did not touch it. | **NARROWED 2026-10-09 — the felhom.eu half is DONE and verified (Gitea admin password rotated, old one now 401; umami + healthchecks rotated; file de-gitted; gate carve-out removed). What remains is the ~12 reused logins elsewhere in the homelab and the repo-visibility decision; owner: operator** **2026-10-10 (release session), two consequences measured:** (a) DooPlex's own Docker login for `gitea.dooplex.hu` (`~/.docker/config.json`) still holds the OLD admin password — the registry token endpoint answers 401 for it and 200 for the current one — so every image push from DooPlex fails `unauthorized` until it is refreshed; the 0.145.0/0.305.0 pushes used a one-off login in a scratch `DOCKER_CONFIG`, shredded after. (b) The `felhom` ArgoCD app reads OutOfSync for `Secret/gitea-creds`, `Secret/healthchecks-config`, `Secret/umami-config` and `Deployment/umami`: a whole-app sync would push git's (de-gitted) view over the rotated live Secrets — the hub deploy synced ONLY `Deployment/hub` (`audits/release-2026-10-10/hub-0.145.0-deploy.txt`). Refreshing the DooPlex login is a DooPlex change (operator's word). **2026-10-22 night (Part 6, read only, `audits/night-2026-10-22/P6-repos-private/01-readers.md`):** homelab-manifests is already private; felhom-controller can go private after one change — CI clones with a read token (check first that the hub's `gitea-creds` token user can read a private `admin` repo); felhom.eu must stay public now (the website git-sync has no credential), felhom-agent too (every install fetches 16 config files from `raw/tag` with no login), and the catalog too (every box clones it with no login, and a token on a customer box is readable by its owner). Making a repository private does NOT hide its images or packages: they follow the user `admin`, which is public — and boxes need that (NOT-A-FINDING). **Count corrected:** the served `` comments are 61 on four pages (adatkezeles 33, feltetelek 15, teszteles 7, en/testing 6), not „32 on /adatkezeles”. The visibility question is decision D29 in STATUS. **2026-10-22 evening — D29 = no (operator):** the controller repository stays public for now; this row's secret rotation is unchanged. | — | **OPERATOR RULED 2026-10-09: (a) he rotates the remaining services himself — CC's scope stopped at the Felhom boundary; (b) the repo STAYS anonymously readable for now**, because making it private breaks the website git-sync and the installer tag fetch, which both clone with no credentials (R-110). The standing consequence of (b): no secret may ever enter this repo again, which `manifest_bearer_gate.py` now enforces with no exemption; and if (b) is reversed, the 61 served `` comments (four pages, 33 on /adatkezeles) must be stripped in the same change. **The exact checklist of the 12 remaining secrets** (namespace / secret / key, re-measured after the rotation, with the two traps that bite — a DB password is not changed by editing the Secret, and a pod that has run for months may not restart) is in `documentation/runbooks/secrets.md`. Operator: **(1)** work that checklist, `bookstack-db/root-password` first because it is a database root password and every one of these hosts answers on the public internet; (`bookstack` first — it is a database root password); **(2)** rotate the `kisfenyo` Gitea token embedded in the local `homelab-manifests` remote URL, and store the remote without credentials; **(3)** decide whether `gitea.dooplex.hu` should answer anonymously at all — remembering the website git-sync clones it with no credentials and the installer is fetched from a public tag, so making it private breaks both unless they get credentials first, and the 61 served `` comments become a map of it; **(4)** move `~/rotated-secrets-2026-10-09.txt` into the password manager and delete it. If nothing is done: the published password keeps opening a dozen services, even though Gitea itself is now safe | operator | -| **R-1243** | Security & access | P2 | **The hub's Longhorn backups sit unencrypted in the NFS folder any DooPlex pod can read (R-1241) — the two there now (2026-10-18, 2026-10-23) hold every household's Cloudflare and tunnel tokens in plaintext, because they predate hub 0.168.0.** Measured 2026-10-23 night (`audits/hub-token-seal-2026-10-23/` §3): the hub volume is `encrypted=false` and in Longhorn's `default` backup group (daily 04:00 + weekly Sunday 05:00, retain 1 each); the hub's own copy to ep0 is encrypted at rest (16/16 snapshots `crypt-mode: encrypt`, key on DooPlex's root disk and on paper) and in transit (SSH tunnel + pinned PBS TLS), every 6 h, restore-tested weekly. DECIDE: (a) an encrypted Longhorn volume for the hub (~5–10 min downtime, a homelab StorageClass + key in the cluster) or (b) take the hub volume out of the Longhorn backup — ep0's copy is the backup; the access log then has no backup until it rides the ep0 push. **CC's pick: (b).** And: delete the two plain copies (the operator's call; with (b) they never rotate out). | WAITING-ON-OPERATOR | the operator's pick; a homelab change for either option | Operator: pick (a) or (b), and say whether to delete the two plain copies after hub 0.168.0 is live | operator | +| **R-1243** | Security & access | P2 | **The hub's Longhorn backups sit unencrypted in the NFS folder any DooPlex pod can read (R-1241) — the two there now (2026-10-18, 2026-10-23) hold every household's Cloudflare and tunnel tokens in plaintext, because they predate hub 0.168.0.** Measured 2026-10-23 night (`audits/hub-token-seal-2026-10-23/` §3): the hub volume is `encrypted=false` and in Longhorn's `default` backup group (daily 04:00 + weekly Sunday 05:00, retain 1 each); the hub's own copy to ep0 is encrypted at rest (16/16 snapshots `crypt-mode: encrypt`, key on DooPlex's root disk and on paper) and in transit (SSH tunnel + pinned PBS TLS), every 6 h, restore-tested weekly. DECIDE: (a) an encrypted Longhorn volume for the hub (~5–10 min downtime, a homelab StorageClass + key in the cluster) or (b) take the hub volume out of the Longhorn backup — ep0's copy is the backup; the access log then has no backup until it rides the ep0 push. **CC's pick: (b).** And: delete the two plain copies (the operator's call; with (b) they never rotate out). | **VERIFY** — **2026-10-24: D40 put into effect** (operator ruling 07:22): the hub PVC `hub-data` carries `recurring-job-group.longhorn.io/default: disabled` (felhom.eu `3363ffa8`, ArgoCD sync of the PVC only) and the Longhorn volume `pvc-486c9809…` reads `disabled` too (labelled live to match, unchanged after 2 min); the NFS folder holds no copy of the volume (no `pvc-486c9809*` directory, no backup config naming it; Longhorn lists no backup volume for it; 53 other volumes backed up this morning). Closes after the dated check: tonight's 04:00 job makes no new hub backup while the others run. WAITING-ON-OPERATOR | the operator's pick; a homelab change for either option | Operator: pick (a) or (b), and say whether to delete the two plain copies after hub 0.168.0 is live | CC (the dated check) | | **R-779** | Security & access | P4 | **[P3-LOW] Part A's "two outside addresses seen as two" is proven through the simulated tunnel only; on the REAL tunnel the second outside address (ep0, one request allowed) was refused by Cloudflare's edge with 403 and never reached the box.** Measured 2026-10-01 19:51 UTC (`audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt`): no log line on demo-hp; demo-hp's box has no geo restriction in its settings, so a Cloudflare ZONE rule (country or bot, not read) refused a German datacenter address. DooPlex's own address on the real tunnel was seen as itself. **Needs:** one sign-in from a second Hungarian address (the operator's phone off wifi) while DooPlex is locked out — 2 minutes; and say which Cloudflare rule refused ep0. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (a phone), CC reads the logs** **Re-ranked 2026-10-03: P3→P4: a proof gap on the real tunnel; operator-only follow-up.** | — | — | CC + operator | | **R-904** | Security & access | P4 | **Cloudflare can read every household's app traffic; replacing it with our own relay is a later item.** Facts (reviewer discussion 2026-10-08; `01`): app traffic and the dashboard reach the box through the Cloudflare Tunnel (`01` §5 trust table, rows end-user ↔ apps and customer ↔ controller UI; §7), and the tunnel's public end is Cloudflare's edge, where TLS ends — so Cloudflare can technically read that traffic (the FAQ says so since 2026-10-08, R-900; „TLS ends at the edge" is not written in `01` — add it there). What Cloudflare gives today, free: inbound reach with no router setup, the CGNAT answer (`01` §4, §7); certificates (`01` §7, the free tier covers one level below a zone); the geo-WAF the hub enforces (`01` §5 last row, §7); flood protection (not in `01`). The alternative named: our own EU relay over WireGuard with TLS passthrough by SNI, certificates on the box, the geo-block on the relay. Its costs: one more machine the operator keeps up, and a single point of reach for every box; weaker flood protection; about a week of work after a spike. Operator ruling 2026-10-08 09:07 (`09` §3 decision 184): a later item. | **DEFERRED** — **2026-10-10: the asked sentence is in `01` §7** („TLS ends at Cloudflare's edge …”); the spike stays deferred. **DEFERRED — after the first customers (operator ruling 2026-10-08 09:07)** | — | A spike after the first customers (the relay's reach, cost and flood behaviour, measured) | operator | | **R-1040** | Security & access | P4 | **Until the household makes Sonarr's or Radarr's first admin, any container on the box can read the app's API key.** Measured on the bench 2026-10-14 with the R-1030 seed in place: `AuthenticationRequired=Enabled`, `AuthenticationMethod=None` — `/initialize.json` from another container answers 200 with `apiKey`. Outsiders are blocked by the box's setup gate in that window; a compromised other app is not. | **OPEN** — filed 2026-10-14 | — | Seed the API key at install (`SONARR__AUTH__APIKEY`, measured to work) so the box knows it, and close the window by telling the household in the first steps; or seed a Forms user | CC | @@ -270,6 +270,7 @@ stopping line that lies. | **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC | | **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC | | **R-1244** | Monitoring & notifications | P2 | **The upload and CPU watch is built and not released: until it is, a hacked app on a household box can send hundreds of GB or mine for weeks and nobody is told.** Operator ruling 2026-10-23 19:22; built 2026-10-23 night — controller v0.342.0 (`internal/upwatch`) + hub v0.169.0 (the four operator-only events), both committed, NOT released; proven on scratch 9202 with `0.342.0-rc.1` (`audits/SPIKE-upload-watch-2026-10-23.md` §7; `08` §9 [DESIGN — proposed]). | WAITING-ON-OPERATOR | the operator's yes; R-1245 and R-1246 | Operator: yes → build hub 0.169.0 + deploy, then the controller through the normal release path (Tester 1, demo-hp, demo-felhom, box walk) | operator | +| **R-1250** | Monitoring & notifications | P3 | **A qBittorrent that seeds all day is never line-tested, so it never gets the limit that exists for it.** The weekly test (R-1247) starts only after 10 minutes in which the host's uplink sent < 2 Mbit/s each minute; a seed at the line's speed keeps it busy through every window, so no test, no tested line, and no qBittorrent limit (D41 uses the tested line only). Found 2026-10-24 by reading the rule against the case it serves; not yet seen on a box (no test box seeds). | OPEN | — | Design: on a box with qBittorrent and no tested line, the controller asks qBittorrent to pause uploads (its alternative speed limit) for the test window the agent announces, then restores it; or the watch reports „line unknown, a seeding qBittorrent" to the operator. Pick, red-proof, prove on 9202 | CC | | **R-1193** | Monitoring & notifications | P3 | **When the box froze (R-1192), the operator heard only „no report for 45 m" — 28 minutes in — and the household heard nothing at all.** Measured 2026-10-20 against the operator mailbox (the second channel): the freeze began ~13:05 UTC; the hub page read ONLINE / OK at 13:22 (correct inside the 45-minute threshold, R-549); `host_stale` at 13:33, `app_oom` at 13:34 (immich-machine-learning killed by its memory limit), `host_crash_restart` at 14:26 and 16:29 (the two plug pulls), `node_stale` 15:59 / `node_recovered` 16:03. No mail or banner tells the household that its box is not answering, and no operator mail says „the box answers ping but not its dashboard". CORRECTED the same day: the first version of this row said the hub showed OK „throughout" and „nobody was told" — that read only the hub page, not the mailbox. Evidence `audits/rehearsal-2026-10-20/journal.md`. | OPEN | — | Design note (2026-10-21, after R-1192's diagnosis): the freeze's signature is cheap to see from the HOST side — the guest cgroup's memory PSI „full" above ~50 % for a minute, or its `memory.events max` climbing, while the host still reports; the agent could raise `guest_memory_thrash` (operator mail in its own words, not „no report for 45 m") and the household banner „A dobozod most nagyon lassú…". With the swap fix (R-1192) the measured thrash ends within a minute, so this alarm should be built only if a freeze is seen again after the agent release; until then it stays a design. Decide whether a dashboard that does not answer for N minutes while the host still reports should alarm sooner and in its own words (guest-side liveness), and whether the household is told | CC | | **R-1202** | Monitoring & notifications | P3 | **A box whose clock is behind blames the hub and stops an app without telling anyone.** Rehearsal 2026-10-20 B6 #8: RTC set 2 years back (the OS raised it to 2026-04-13): the dashboard shows only „Hub nem elérhető" while the log says „x509: certificate … not yet valid: current time 2026-04-13…"; Paperless crash-looped and the box stopped it with a restore hold; the `app_stopped_unhealthy` event failed (hub unreachable), so neither the household nor the operator heard. After the real time came back the app stayed stopped until a Start (clear sentence, 1 min). | **OPEN** | — | say „the box's clock is wrong (it says )" when TLS fails with not-yet-valid/expired against a hub that answers; keep the stop event queued until the hub is reachable | CC | | **R-340** | Monitoring & notifications | P3 | **The new reachability check does not touch the surface that actually failed.** R-339 reports when the hub cannot READ ep0 — but the read it performs is the `usage` op, which is `proxmox-backup-manager` plus `df` over SSH, and therefore rides the **local API daemon**. The 2026-08-18 incident explicitly CLEARED that daemon: `proxmox-backup.service` was healthy throughout, and it was the **HTTPS proxy on 8007** that was wedged with a full accept queue. **So R-339's check would have returned green for all 9 h 37 m of that outage.** It closes the case where ep0 is unreachable *as a host*; it does not close the case that actually happened. This is not a defect in R-339 — it is the honest boundary of what it watches, recorded so a future reader does not mistake a green box gauge for a working off-site tier | **READY (M) — NEW 2026-08-18** | a tenantsync endpoint-script version bump (the op is added on ep0, so it needs the same version-gated rollout `ErrUsageUnsupported` already models) | Add a **health op** to `scripts/felhom-tenantsync.sh` that probes `https://127.0.0.1:8007/` on ep0 and reports the proxy's fd count and listen-queue depth, then surface it as a third signal. **Overlaps the connections spike (R-336's remaining half): both want the same observations from ep0, so whichever runs SECOND must reuse the first's evidence rather than re-measuring a protected machine twice** **REUSE, per this row's own instruction — the connections spike ran FIRST (2026-08-20) and already produced most of what the health op wants; do not re-measure a protected machine a third time.** Available in `audits/evidence-ep0-established-connections-2026-08-20/`: the proxy **fd count** and its type breakdown (`lsof` + `/proc//fd`), the **listen-queue depth** (`ss -lnt` — `Recv-Q 0`, `Send-Q 1024`), the **ESTAB/CLOSE-WAIT split**, the **per-peer** connection histogram, a **31-minute persistence diff** of full 4-tuples, and a **46.18 h** slope with Poisson bounds. What the health op would still add beyond these: a loopback `GET https://127.0.0.1:8007/` probe — the observation that distinguished "process problem" from "network problem" on 2026-08-18 and the one thing this spike did NOT take, because it is the surface R-339 cannot see. **And this spike sharpens what the op should report:** a rising **ESTAB** count is the live signal (CLOSE-WAIT was **0**, not merely flat), and per **R-344** the fd ceiling that matters may be the **agent's**, not only ep0's. | CC | @@ -285,18 +286,16 @@ stopping line that lies. | **R-1211** | Monitoring & notifications | P3 | **The household's `backup_failed` mail says only „A biztonsági mentés sikertelen! Kérjük, ellenőrizd a rendszert." — no app named, no next step, and the formal „Kérjük".** Rehearsal 2026-10-20 (the one alarm the household got, `audits/rehearsal-2026-10-20/B9-words/words.md` W25); the box's own page now says which app and that the box retries the next night (controller `5776d54`, W25), but the mail is the hub's `customerMessages` entry. Found by the night helper 2026-10-21. | OPEN | — | Hub: build the household mail from the event's app names and the classifier's class, in the te-form, with „a doboz a következő éjjel újra próbálkozik"; mail golden | CC | | **R-1222** | Monitoring & notifications | P3 | **A household that sets an EVENING backup time gets a false „database dump missed” error every morning.** REAL in code (night 2026-10-22; split from R-1189 (e)): the hub's 05:00 check wants a dump since Budapest midnight (`hub/internal/monitor/deadline.go:416-419, 555-565`), so an evening dump (the banner suggests evening times) is always „yesterday”; on 2026-10-25 the check also runs at 04:00 (R-1183). Evidence `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN | — | Fix: the check wants a dump within the last 26 h (or since the box's own scheduled time); a hub test with an evening schedule; rides a hub deploy | CC | | **R-1223** | Monitoring & notifications | P3 | **After a reboot, a box whose tunnel ep0 refuses is invisible: the future stamp is gone, so neither the D30 re-key nor the hub's alarm sees it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, A8 + H1): the kernel forgets `latest-handshake` when the interface restarts (0 = never); the agent then sends neither `last_handshake_age_s` nor `handshake_future_s`, and `checkWGTunnel` (`hub/internal/api/host_tunnel.go`) does not judge a registered, active tunnel with no handshake at all. Shape: the clock ran ahead, was corrected, the box rebooted (some households switch off at night), ep0 still holds the future stamp and starts nothing. | OPEN | — | Agent: report `tunnel_up_s` (how long the unit has been active) with no handshake; hub: a registered, active tunnel with no handshake for > 30 min while reporting → `wg_tunnel_stale`; the agent re-keys the same way after 30 min with no handshake. Tests both sides; rides an agent release + a hub deploy | CC | -| **R-1245** | Monitoring & notifications | P3 | **DECIDE: how large is qBittorrent's default upload limit?** The box sets it from the measured line and never over the household's own limit. CC's pick: **50 % of the measured upload** (the family keeps half the line; a seeding qBittorrent measured 131 Mbit/s on the bench — the whole line). Other: 30 % (gentler on the family, slower seeding) or none (only the alarm). `audits/SPIKE-upload-watch-2026-10-23.md` §4, §6.6. **Measured 2026-10-24 night (spike §2.1):** of three boxes one push counted (demo-hp: at least 10 Mbit/s from its best 10 s; one second reached 70) — so 50 % of the lower bound would hold qBittorrent at ~5 Mbit/s there until a bigger night comes (best of 14). | WAITING-ON-OPERATOR | — | Operator: say a percentage (or none) | operator | -| **R-1246** | Monitoring & notifications | P3 | **DECIDE: who is told when the box limits or flags something?** CC's pick: the family only about the qBittorrent limit (one line on its page; they can change it); the unexplained-upload and unexplained-CPU alarms reach the operator only (R-388: the household can do nothing about a hacked app but wait for support). Other: tell the family too (that support was told). `audits/SPIKE-upload-watch-2026-10-23.md` §6.4. | WAITING-ON-OPERATOR | — | Operator: pick | operator | | **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC | | **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC | -| **R-1247** | Monitoring & notifications | P4 | **The box's upload line is only a lower bound, learnt from its own off-site push; most nights the push is too small to say anything.** Measured: at a one-minute sample the push exceeded 2 Mbit/s on 3 of 81 box-nights; a one-second sample during the push is now built (a night counts when its best 10 s moved ≥ 10 MB). An ACTIVE test (a timed upload to the box's own Storage Box sub-account, or to ep0) would give the real line — not built: a design for the test file beside the restic repository, a quiet hour outside 02:30–05:30, ~125 MB per test at 100 Mbit/s. `audits/SPIKE-upload-watch-2026-10-23.md` §2, §3. | OPEN | an operator decision (data cost, where the test writes) | Decide whether an active test is wanted; if yes, CC designs it | operator | +| **R-1247** | Monitoring & notifications | P4 | **The box's upload line is only a lower bound, learnt from its own off-site push; most nights the push is too small to say anything.** Measured: at a one-minute sample the push exceeded 2 Mbit/s on 3 of 81 box-nights; a one-second sample during the push is now built (a night counts when its best 10 s moved ≥ 10 MB). An ACTIVE test (a timed upload to the box's own Storage Box sub-account, or to ep0) would give the real line — not built: a design for the test file beside the restic repository, a quiet hour outside 02:30–05:30, ~125 MB per test at 100 Mbit/s. `audits/SPIKE-upload-watch-2026-10-23.md` §2, §3. | **READY** — **2026-10-24 (operator ruling 07:26): the weekly iperf3 line test, built; release in progress.** ep0 `felhom-iperf3.service` live (tunnel only, proven closed on the internet); agent `internal/linetest` (`f7e1dcb`, unreleased, next 0.164.0); controller reads it (`ed1011b`, in 0.342.0). demo-hp measured **90.7 Mbit/s** (the push floor said 10). Closes when the agent and the controller are released and each test box reports a `line_test` value. OPEN | an operator decision (data cost, where the test writes) | Decide whether an active test is wanted; if yes, CC designs it | operator | ## Hub & operator — 12 rows (P3 5, P4 7) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| -| **R-975** | Hub & operator | P3 | **The hub database keeps every customer's tunnel token, Cloudflare API token and git token unsealed in one column, so anyone with a copy of the database (or its nightly snapshots) gets them.** R-879 sealed the other secret columns but not this one. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). 2026-10-23 night: **sealed in hub 0.168.0** — the three JSON leaves of `customer_configs.config_json` hold the `05` §16.2 seal, old rows sealed once at start-up + VACUUM, eight tests red-proved (`audits/hub-token-seal-2026-10-23/`). Built, not deployed. | WAITING-ON-OPERATOR | the deploy of hub 0.168.0 (operator yes) | Deploy; read the start-up line `config tokens sealed at rest (8 …)`, then a second start `(0 …)`; then close | operator | +| **R-1249** | Hub & operator | P4 | **The hub's start-up VACUUM after a legacy seal cannot finish on the live Longhorn disk, and its code comment says 2.8 s.** Measured 2026-10-24 at the hub 0.168.0 deploy: the first start sealed the tokens, then the liveness probe stopped it ~90 s into the VACUUM (exit 143); the second start found nothing to seal and skipped it. The disk writes ~1.3 MB/s (the 6-hourly `VACUUM INTO` of 530 MB: 6 min 57 s); 2.8 s was measured on a local-disk copy. It runs only when a start seals plaintext (a roll-back and a roll-forward), and then costs one restart. `hub/internal/store/r975_config_token_seal.go` `SealLegacyConfigTokens`. | OPEN | — | Remove the start-up VACUUM (the operator accepted the live file's residue, `05` §16.2) and correct the comment; or move it behind a start-up probe with a measured budget. Rides the next hub change | CC | | **R-31** | Hub & operator | P3 | **[P2-HIGH] Offsite provisioning is synchronous with no status affordance.** Save runs the Hetzner sync in-request, so the request can hit the nginx 504 **while succeeding server-side**: the operator cannot tell failed from slow, and a retry races the first attempt. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **Re-ranked 2026-10-03: P2→P3: operator-only; a known workaround (click once, wait, verify) exists.** **2026-10-06 night: the race half fixed on felhom.eu main (hub, unreleased):** a second Save while the first still provisions is refused with 409 („already running — wait about a minute, then reload; do not save again"); nothing saved, nothing created; per customer, in memory. `TestProvision_R31_*`, red-proof `audits/night-burndown-2026-10-06/hub/R-31-red.txt`. LEFT: the async save with a status card (the escrow-card idiom). | — | Direction: make it async + a status card, reusing the proven **awaiting-card/poll idiom** (v0.138.0 escrow card). **Interim mitigation belongs in R-3 as an operator note: click once, wait, verify — do not re-click.** | CC | | **R-882** | Hub & operator | P3 | **Longhorn on DooPlex could not grow a volume online: its `instance-manager` (116 days up) called a host process that no longer existed** — `nsenter: cannot open /host/proc/196610/ns/mnt` on every expansion retry, and an offline growth was blocked by the expansion's own attachment ticket (found 2026-10-05 growing `hub-data` to 2 Gi). A restart of the instance-manager (operator-approved) fixed it: 77/77 volumes back `attached/healthy` in 110 s. **Why the cached PID went stale was not established** (likely a containerd/k3s or iscsid restart after the instance-manager started), so it will recur after the next such restart and stay invisible until a volume needs to grow. `audits/hub-db-offsite-2026-10-05/partA/step1-*.txt` | **OPEN** | — | Find which host process the PID was and whether Longhorn 1.10.x re-resolves it; until then, before growing any volume, check the instance-manager's age against the last k3s/containerd restart | operator | | **R-883** | Hub & operator | P3 | **8 DooPlex workloads run an image by a moving tag (`:latest` or none), so any pod restart is a silent upgrade.** Measured 2026-10-05: the Longhorn restart restarted zipline on `ghcr.io/diced/zipline:latest` (pull Always), which pulled 4.8.0; 4.8.0 refused its database (`cannot safely migrate from prisma to drizzle: expected migration 20260508022000 … was not applied`) and crash-looped. Fixed for zipline by pinning `4.7.0` (homelab-manifests `90f60e4`, `4c8ec7a`; 4.7.0 applied the four missing migrations; a dump from before is kept out-of-band). The other 7 were counted, not named or changed. **Checked from source 2026-10-05 (burn-down round 2):** homelab-manifests@87dfc29 still has moving tags (grep image lines without a numeric tag): admin-system/toolbox.yaml:12 nicolaka/netshoot:latest (a bare Pod); calibre-system/cwa.yaml:826 calibre-web-automated:dev; outline-system/outline.yaml:270 minio/minio:latest; tandoor-system/recipe-importer.yaml:26 gitea.dooplex.hu/admin/recipe-importer:latest (pull Always :27); adventurelog-system/adventurelog.yaml:100 and :256 adventurelog-backend/frontend:latest (pull Always :101,:257); jarrs-system/jarr- | **OPEN** | — | List the 7 (`kubectl get deploy,sts -A` images without a fixed tag), pin each to the running version, and let Renovate move them | operator | @@ -373,6 +372,7 @@ stopping line that lies. the R-row. Duplicating them here would create the second source this design avoids. --> | item | due (UTC) | what to measure | |---|---|---| +| R-1243 | 2026-10-25 | no new Longhorn backup of the hub volume pvc-486c9809 after the 02:00Z daily job while the other volumes got theirs; detail in the R-1243 row | | R-435 | 2026-10-27 | Tester 1's weekly off-site clean-up ran inside a hub window with no drop alarm (box log + mailbox); detail in the R-435 row | | R-1183 | 2026-10-26 | every hub daily job ran ONCE on 2026-10-25 (hub log + one snapshot file per slot — four slots since hub 0.167.0, D34, the 02:00 one not twice); detail in the R-1183 row | | R-1228 | 2026-10-30 | D35 (yes, after the release week): start moving the website + installer to Cloudflare; detail in the R-1228 row | diff --git a/documentation/backlog/STATUS-HISTORY.md b/documentation/backlog/STATUS-HISTORY.md index 2bf29932..4fcd1df4 100644 --- a/documentation/backlog/STATUS-HISTORY.md +++ b/documentation/backlog/STATUS-HISTORY.md @@ -5,6 +5,16 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S **Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.** +## Moved from STATUS on 2026-10-24 (day) + +**Updated 2026-10-23 (night).** The hub's last unlocked tokens are now locked in a new hub version — built and tested, you deploy it on Monday. The new upload watch is built and tested on the scratch box, not released: it finds an app that uploads or computes far more than the app itself explains (a hacked app) and tells only you; qBittorrent gets an upload limit. + +**Decided 2026-10-24 07:22 (the operator started the attended release brief; the rulings are in R-1243, R-1245 and R-1246 and in CONTEXT.md):** + +| D42 | R-1246 | When the box limits or flags something, who is told? | **The family only about the qBittorrent limit (they can change it); the hacked-app alarms only you** (the family can do nothing about a hacked app) | tell the family too: a worrying mail they cannot act on | the same as my pick | — | +| D41 | R-1245 | How much of the measured upload may qBittorrent use by default? | **50 %** (the family keeps half the line; seeding took the whole line on the test bench) | 30 %: slower seeding; none: only the alarm | 50 % when you release the watch | — | +| D40 | R-1243 | The hub's backups in DooPlex's shared folder are not encrypted (two there now hold the tunnel tokens). Encrypt the hub's disk, or stop that backup? | **Stop it** (the hub's own copy to ep0 is encrypted, every 6 h, tested weekly) | encrypt: 5–10 min hub downtime and one more key to keep | the plain copies stay readable by any hacked DooPlex app | a homelab change | + ## Moved from STATUS on 2026-10-23 (night) **Updated 2026-10-23 (evening).** The Plex attack on DooPlex did NOT reach Felhom's secrets: the two open doors to them (a shared backup folder and Longhorn's page) were not used in the attack days. The backup folder is now closed to the home network (your yes); it is still open to every app on DooPlex, so a hacked app there could read the hub's backups (R-1241, the homelab session's fix). diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index b5d9a65f..e79b4626 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -10,6 +10,11 @@ `TestUpwatchEvents_OperatorOnlyWithAReadableLine`, `TestUpwatchEventTypesAreAllowedAndOperatorOnly` — red-proved (dropped from operatorOnlyEvents → the household was mailed; the render call removed → no Watch line; dropped from the allow-list → fails). +- **2026-10-24 (attended release run): the host's weekly line test reaches the hub.** Agent 0.164.0 sends a top-level + `line_test` stanza (R-1247); `sysfacts.LineTest` mirrors it field by field (the wire-contract gate checks both ways, + `SUBTREE_MIRRORS`), and the System page shows one cell per host, „Upload line": the best of the last successful + weekly test and its age, amber when never measured after an attempt or older than 15 days — judged on `last_ok_at`, + never on the attempt time. Tests `TestParse_LineTest`, `TestLineTestCell` — red-proved. ## v0.168.0 (DEPLOYED 2026-10-24 on the operator's yes — built 2026-10-23 night) — the last plain tokens sealed (R-975, R-1241) diff --git a/hub/internal/sysfacts/sysfacts.go b/hub/internal/sysfacts/sysfacts.go index 625a9819..5eeba271 100644 --- a/hub/internal/sysfacts/sysfacts.go +++ b/hub/internal/sysfacts/sysfacts.go @@ -98,6 +98,21 @@ type GuestTrim struct { Error string `json:"error"` } +// LineTest is the agent's weekly upload line test stanza (R-1247, agent ≥ 0.164.0): the host report's TOP-LEVEL +// `line_test` object (agent internal/hub LineTestStatus) — a 10-second iperf3 upload to ep0 inside the tunnel, best of +// the runs, as ep0's server counted it. `last_ok_at` + `mbps` are the last SUCCESSFUL measurement; `last_attempt_at` + +// `last_error` the last attempt, whatever its verdict. A field-by-field mirror, checked both ways by the wire-contract +// gate (SUBTREE_MIRRORS). +type LineTest struct { + Schedule string `json:"schedule"` + LastOKAt string `json:"last_ok_at"` // RFC3339, "" = never + Mbps float64 `json:"mbps"` + RunsMbps []float64 `json:"runs_mbps"` + Server string `json:"server"` + LastAttemptAt string `json:"last_attempt_at"` // RFC3339 + LastError string `json:"last_error"` +} + // System is the whole stanza. Present is false for a report from an agent older than v0.142.0. type System struct { Present bool @@ -112,10 +127,13 @@ type System struct { // DiskTrim is the top-level `guest_disk_trim` stanza (R-444). Independent of the `system` stanza: it is read even // when Present is false. nil = the agent sent none (an agent with no trimmer). DiskTrim *DiskTrim + // LineTest is the top-level `line_test` stanza (R-1247), read like DiskTrim. nil = the agent sent none. + LineTest *LineTest } type wire struct { DiskTrim *DiskTrim `json:"guest_disk_trim"` + LineTest *LineTest `json:"line_test"` System *struct { PVEVersion string `json:"pve_version"` KernelVersion string `json:"kernel_version"` @@ -143,7 +161,7 @@ func Parse(reportJSON string) System { if json.Unmarshal([]byte(reportJSON), &w) != nil { return out } - out.DiskTrim = w.DiskTrim + out.DiskTrim, out.LineTest = w.DiskTrim, w.LineTest if w.System == nil { return out } diff --git a/hub/internal/sysfacts/sysfacts_test.go b/hub/internal/sysfacts/sysfacts_test.go index b98a5188..8afb065b 100644 --- a/hub/internal/sysfacts/sysfacts_test.go +++ b/hub/internal/sysfacts/sysfacts_test.go @@ -70,3 +70,17 @@ func TestParse_DiskTrim(t *testing.T) { t.Errorf("with a system stanza the trim must still be read: %+v", s) } } + +// R-1247: the top-level `line_test` stanza is read on its own, and an absent one stays nil. The fixture is the agent's +// TestStateSurvivesARestartAndReportsIt output shape. +// RED-PROOF: drop `out.LineTest = w.LineTest` → nil → fails. +func TestParse_LineTest(t *testing.T) { + s := Parse(`{"host":{},"line_test":{"schedule":"weekly, due Tuesday","last_ok_at":"2026-10-24T06:31:00Z","mbps":90.7,"runs_mbps":[88,90.7],"server":"10.77.0.1:5201","last_attempt_at":"2026-10-24T06:31:00Z"}}`) + lt := s.LineTest + if lt == nil || lt.Mbps != 90.7 || len(lt.RunsMbps) != 2 || lt.Server != "10.77.0.1:5201" || lt.LastOKAt != "2026-10-24T06:31:00Z" { + t.Fatalf("line_test misread: %+v", lt) + } + if s = Parse(full); s.LineTest != nil { + t.Errorf("a report with no line_test must read nil, got %+v", s.LineTest) + } +} diff --git a/hub/internal/web/system.go b/hub/internal/web/system.go index a530b00e..27fc2d2f 100644 --- a/hub/internal/web/system.go +++ b/hub/internal/web/system.go @@ -43,6 +43,7 @@ type systemRow struct { // guest GuestDebian, GuestRelease, GuestPending, GuestRestart cell Trim cell // R-444: the last `pct fstrim` of the guest + Line cell // R-1247: the host's weekly upload line test // docker Engine, Containerd, LiveRestore, DockerRelease cell // last leg @@ -168,6 +169,36 @@ func buildFloorRows(ovs []store.CustomerFloorOverride, global string, now time.T func plain(s string) cell { return cell{Text: s} } +// lineStaleAfter is when the last SUCCESSFUL line test reads amber: the hosts test weekly (R-1247), so two missed weeks +// are worth a look (a seeding qBittorrent can keep a host from ever being quiet, R-1250). +const lineStaleAfter = 15 * 24 * time.Hour + +// lineTestCell is the "Upload line" cell (R-1247): the best of the last successful weekly test and how old it is, judged +// on `last_ok_at` — never on the attempt time. Amber when never measured after an attempt, or older than 15 days. "—" +// when the agent sends no stanza (an agent before 0.164.0, or no tunnel). +func lineTestCell(lt *sysfacts.LineTest, now time.Time) cell { + if lt == nil { + return cell{Text: "—", Title: "the agent reports no line test (agent before 0.164.0, or no offsite tunnel)"} + } + title := "schedule: " + lt.Schedule + if lt.LastError != "" { + title = "last attempt " + lt.LastAttemptAt + ": " + lt.LastError + " — " + title + } + okAt, err := time.Parse(time.RFC3339, lt.LastOKAt) + if lt.LastOKAt == "" || err != nil || lt.Mbps <= 0 { + if lt.LastAttemptAt == "" { + return cell{Text: "not yet", Title: title} + } + return cell{Text: "not measured", Class: "warn", Title: title} + } + out := cell{Text: fmt.Sprintf("%.1f Mbit/s · %s", lt.Mbps, ago(okAt, now)), Title: title} + if now.Sub(okAt) > lineStaleAfter { + out.Class = "warn" + out.Title = "the last successful test is older than 15 days (the hosts test weekly) — " + out.Title + } + return out +} + // trimStaleAfter is when a guest's last SUCCESSFUL disk trim reads amber: the boxes trim weekly (`09` §3 decision // 139), so two missed weeks are worth a look. const trimStaleAfter = 14 * 24 * time.Hour @@ -371,6 +402,7 @@ func buildSystemRows(lines []osupdates.FleetLine, facts map[string]sysfacts.Syst r.GuestPending = plain(fmt.Sprint(l.Guest.Pending)) r.GuestRestart = plain(fmt.Sprint(l.Guest.RestartNeeded)) r.Trim = trimCell(f.DiskTrim, now) + r.Line = lineTestCell(f.LineTest, now) r.Engine, r.Containerd = unknownCell(f.Guest.DockerEngine), unknownCell(f.Guest.Containerd) r.LiveRestore = unknownCell(f.Guest.LiveRestore) if f.Guest.LiveRestore == "off" { diff --git a/hub/internal/web/system_line_test.go b/hub/internal/web/system_line_test.go new file mode 100644 index 00000000..42105f62 --- /dev/null +++ b/hub/internal/web/system_line_test.go @@ -0,0 +1,35 @@ +package web + +import ( + "strings" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/sysfacts" +) + +// R-1247: the operator's "Upload line" cell is judged on the last SUCCESSFUL test, never on the attempt time — an +// attempt is not a result. +// RED-PROOF: judge staleness on last_attempt_at → the 20-day-old success with a fresh failed attempt reads green → fails. +func TestLineTestCell(t *testing.T) { + now := time.Date(2026, 10, 24, 12, 0, 0, 0, time.UTC) + cases := []struct { + name string + lt *sysfacts.LineTest + text string + class string + }{ + {"no stanza", nil, "—", ""}, + {"never tried", &sysfacts.LineTest{Schedule: "weekly"}, "not yet", ""}, + {"tried, never measured", &sysfacts.LineTest{Schedule: "weekly", LastAttemptAt: "2026-10-24T10:00:00Z", LastError: "iperf3 server busy"}, "not measured", "warn"}, + {"fresh", &sysfacts.LineTest{Schedule: "weekly", LastOKAt: "2026-10-24T06:31:00Z", Mbps: 90.7}, "90.7 Mbit/s", ""}, + {"old success, fresh failed attempt", &sysfacts.LineTest{Schedule: "weekly", LastOKAt: "2026-10-04T06:31:00Z", Mbps: 90.7, + LastAttemptAt: "2026-10-24T10:00:00Z", LastError: "connect: timeout"}, "90.7 Mbit/s", "warn"}, + } + for _, c := range cases { + got := lineTestCell(c.lt, now) + if !strings.HasPrefix(got.Text, c.text) || got.Class != c.class { + t.Errorf("%s: cell = %+v, want text %q class %q", c.name, got, c.text, c.class) + } + } +} diff --git a/hub/internal/web/templates/system.html b/hub/internal/web/templates/system.html index 2da88321..7a6d09dc 100644 --- a/hub/internal/web/templates/system.html +++ b/hub/internal/web/templates/system.html @@ -206,6 +206,7 @@
Pending
{{template "sys_dd" .GuestPending}}
Restart needed
{{template "sys_dd" .GuestRestart}}
Last disk trim
{{template "sys_dd" .Trim}} +
Upload line
{{template "sys_dd" .Line}} diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index abd96f36..fe751163 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,3 +1,12 @@ +## 2026-10-24 — `ep0-iperf3/`: the tunnel-only line-speed test server on ep0 (R-1247; operator ruling 2026-10-24 07:26) + +- **`ep0-iperf3/felhom-iperf3.service`** (new, installed by hand on ep0 2026-10-24): Debian's `iperf3 --server` bound to + `10.77.0.1:5201` as a DynamicUser with systemd hardening and caps (MemoryMax 128M, TasksMax 16, CPUQuota 50 %, a daily + restart); one nftables line admits TCP from `wg0` `10.77.0.0/24` only. Debian's own `iperf3.service` stays disabled. + `README.md`: install, the firewall line, the proof, the two open no-DSA CVEs of 3.18 and how they are bounded, removal. + Proven: no answer from the internet (`:22` as the control), an answer through the tunnel from demo-hp's host, and a + 10-second test of 90.7 Mbit/s (`documentation/audits/line-test-2026-10-24/`). + ## 2026-10-23 — `release-copy/`: the released images and agent packages leave DooPlex nightly (D33, R-1227); `hub-db-backup/`: every 6 hours (D34) - **`release-copy/felhom-release-copy`** (new, DooPlex root, timer 01:40): the newest 3 release tags of felhom-controller, felhom-hub and felhom-samba (`skopeo copy --all --preserve-digests` into `dir:` — the registry's own bytes, so a push back gives the SAME digest; a `docker save` tar would not) and the newest 5 agent versions (binary + config bundle). Before anything leaves: the top manifest hashes to the registry's digest, every blob and child manifest to its name, every named blob is present, each agent file matches the registry's sha256; the staged tree ≤ the cap (3072 MB). Then `rsync --delete` to ep0's limited user `felhom-relcopy` (rrsync, one directory), and a `--checksum --dry-run` must show no difference before the success signal. Spoiled tags (0.315.0, 0.316.3) are never copied. 13 tests (`test_release_copy.py`), four red-proved. diff --git a/scripts/ep0-iperf3/README.md b/scripts/ep0-iperf3/README.md new file mode 100644 index 00000000..8160d0d5 --- /dev/null +++ b/scripts/ep0-iperf3/README.md @@ -0,0 +1,22 @@ +# ep0 — the line-speed test server (R-1247) + +Operator ruling 2026-10-24 07:26: a box's agent runs a weekly 10-second upload test to an iperf3 server on ep0, +**inside the WireGuard tunnel only**. Installed 2026-10-24 (evidence `documentation/audits/line-test-2026-10-24/`). + +- Package: Debian `iperf3` (3.18-2+deb13u2 at install; it was already present since 2026-07-03). Debian's own + `iperf3.service` stays **disabled**. +- Unit: `felhom-iperf3.service` (this folder) → `/etc/systemd/system/`, `systemctl enable --now felhom-iperf3`. + Listens on `10.77.0.1:5201` only, as a DynamicUser, with memory/task/CPU caps. +- Firewall: one line in `/etc/nftables.conf`'s `input` chain, after the PBS line, and the same rule live: + + tcp dport 5201 iifname "wg0" ip saddr 10.77.0.0/24 accept + + Live: `nft add rule inet filter input position tcp dport 5201 iifname "wg0" ip saddr 10.77.0.0/24 accept`. +- Proof that it never answers on the internet: a TCP connect from any public address to `167.233.158.164:5201` gets + no answer (with `:22` as the control); from a box host through the tunnel it answers. +- Known server issues of 3.18 (no Debian advisory, „minor"): CVE-2026-71217 (resource exhaustion by oversized + parameters) and CVE-2026-102253 (a UDP worker spins). Bounded by: tunnel peers only, TCP only, the unit's caps, a daily + restart. Re-check when Debian ships 3.22. +- Remove: `systemctl disable --now felhom-iperf3; rm /etc/systemd/system/felhom-iperf3.service`; delete the nft line + (live `nft delete rule inet filter input handle `, and in the file). The backup of the file before the change is + `/etc/nftables.conf.bak-20261024-iperf3`. diff --git a/scripts/ep0-iperf3/felhom-iperf3.service b/scripts/ep0-iperf3/felhom-iperf3.service new file mode 100644 index 00000000..51adcb98 --- /dev/null +++ b/scripts/ep0-iperf3/felhom-iperf3.service @@ -0,0 +1,49 @@ +# The line-speed test server on ep0 (R-1247; operator ruling 2026-10-24 07:26). A box's agent runs a 10-second +# upload test against it once a week, inside the WireGuard tunnel. Installed by hand on ep0 as +# /etc/systemd/system/felhom-iperf3.service; the firewall line that opens it is in README.md beside this file. +# +# Bound to the tunnel address only, TCP only (the nftables line admits TCP from wg0 10.77.0.0/24). Debian's own +# iperf3.service stays disabled. The caps bound the two open no-DSA server issues of iperf3 3.18 (CVE-2026-71217: +# oversized parameters exhaust memory and threads; CVE-2026-102253: a UDP worker spins — UDP never reaches it): +# MemoryMax/TasksMax/CPUQuota keep PBS safe, RuntimeMaxSec restarts a stuck server once a day. +[Unit] +Description=Felhom: iperf3 line-speed test server, tunnel only (R-1247) +Documentation=https://gitea.dooplex.hu/admin/felhom.eu/src/branch/main/scripts/ep0-iperf3/README.md +After=wg-quick@wg0.service +Wants=wg-quick@wg0.service + +[Service] +Type=simple +ExecStart=/usr/bin/iperf3 --server --bind 10.77.0.1 --port 5201 --interval 0 --idle-timeout 120 --rcv-timeout 15000 +SuccessExitStatus=1 +Restart=always +RestartSec=10 +RuntimeMaxSec=1d +DynamicUser=yes +NoNewPrivileges=yes +CapabilityBoundingSet= +AmbientCapabilities= +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=yes +PrivateDevices=yes +ProtectKernelTunables=yes +ProtectKernelModules=yes +ProtectKernelLogs=yes +ProtectControlGroups=yes +ProtectClock=yes +ProtectHostname=yes +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX +RestrictNamespaces=yes +RestrictRealtime=yes +RestrictSUIDSGID=yes +LockPersonality=yes +MemoryDenyWriteExecute=yes +SystemCallArchitectures=native +SystemCallFilter=@system-service +MemoryMax=128M +TasksMax=16 +CPUQuota=50% + +[Install] +WantedBy=multi-user.target diff --git a/scripts/wire_contract_gate.py b/scripts/wire_contract_gate.py index 28af22c7..f217f2b5 100644 --- a/scripts/wire_contract_gate.py +++ b/scripts/wire_contract_gate.py @@ -135,6 +135,7 @@ MIRRORS = { # built first, against a provisional shape — and is neither a pass nor a conviction of the subtree. SUBTREE_MIRRORS = { ("agent -> hub (POST /host-report)", "guest_disk_trim"): ("internal/sysfacts", "DiskTrim"), + ("agent -> hub (POST /host-report)", "line_test"): ("internal/sysfacts", "LineTest"), } # Tag names whose literal string carries no information in a repo-wide search. NOT CHECKED.