Attended release run: hub 0.168.0 deployed (R-975 closed; the live file's residue accepted by the operator, 05 16.2); D40 in effect (R-1243 VERIFY, dated check 10-25); D41/D42 closed; the ep0 iperf3 line-test server (scripts/ep0-iperf3, 06 5) and hub 0.169.0 models line_test (System page cell); R-1249, R-1250 filed; register 194 -> 193
gates / gates (push) Successful in 12m46s
gates / gates (push) Successful in 12m46s
This commit is contained in:
@@ -26,6 +26,12 @@
|
||||
> 2026-10-23: „during a quiet window the box measures the upload bandwidth"); the passive floor stays as a lower bound.
|
||||
> This is the one change on ep0 the run may make (Tier 2 — the operator's yes is starting the run). CC still stops
|
||||
> before each hub deploy, before the global controller floor, and for the agent vouch.
|
||||
> **Two more answers in the same run (AskUserQuestion, 2026-10-24 ~08:2x CEST):** (1) **the agent speaks iperf3 itself**
|
||||
> (no host has the iperf3 program; nothing is installed on hosts) — agent `internal/linetest`; (2) **the live hub file's
|
||||
> old token bytes are accepted, not cleaned** (the start-up VACUUM did not finish on the Longhorn disk, R-1249; a clean-up
|
||||
> needs an 8–20 min hub stop, a trim and the volume's snapshots deleted; only root on DooPlex can read the volume, and it
|
||||
> holds the key; every copy that leaves is clean) — written into `05` §16.2 with the rule „a hand copy of hub.db is made
|
||||
> with VACUUM INTO, never cp/tar".
|
||||
|
||||
> **2026-10-23 19:22 — operator rulings by starting the weekend-night brief (R-1241's Felhom half; the upload watch).**
|
||||
> **Part 1 = R-1241's Felhom half:** seal R-975 (done in hub 0.168.0, not deployed — `05` §16.2,
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
|
||||
|
||||
**Updated 2026-10-23 (night).** The hub's last unlocked tokens are now locked in a new hub version — built and tested, you deploy it on Monday. The new upload watch is built and tested on the scratch box, not released: it finds an app that uploads or computes far more than the app itself explains (a hacked app) and tells only you; qBittorrent gets an upload limit.
|
||||
**Updated 2026-10-24 (day, attended).** Hub 0.168.0 is live: the last plain tokens are locked, and every copy of the hub database that leaves DooPlex is clean. The hub is out of DooPlex's open backup folder. Each box can now test its upload once a week (demo-hp: 90.7 Mbit/s, not 10); the releases follow on your yes.
|
||||
|
||||
## What works
|
||||
- **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241).
|
||||
@@ -30,9 +30,6 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
|
||||
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|
||||
|---|---|---|---|---|---|---|
|
||||
| D42 | R-1246 | When the box limits or flags something, who is told? | **The family only about the qBittorrent limit (they can change it); the hacked-app alarms only you** (the family can do nothing about a hacked app) | tell the family too: a worrying mail they cannot act on | the same as my pick | — |
|
||||
| D41 | R-1245 | How much of the measured upload may qBittorrent use by default? | **50 %** (the family keeps half the line; seeding took the whole line on the test bench) | 30 %: slower seeding; none: only the alarm | 50 % when you release the watch | — |
|
||||
| D40 | R-1243 | The hub's backups in DooPlex's shared folder are not encrypted (two there now hold the tunnel tokens). Encrypt the hub's disk, or stop that backup? | **Stop it** (the hub's own copy to ep0 is encrypted, every 6 h, tested weekly) | encrypt: 5–10 min hub downtime and one more key to keep | the plain copies stay readable by any hacked DooPlex app | a homelab change |
|
||||
| D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — |
|
||||
| D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — |
|
||||
| D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — |
|
||||
@@ -54,7 +51,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
2. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-16fdef. If not: two dead entries alarm.
|
||||
3. D37 and D38 in the table (two small yes/no). If not: Gitea may restart in release hours; no golden in the release copy.
|
||||
4. Give the homelab session R-1241, R-1242, the 7-item list at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md`, and an upload alarm on DooPlex's `eno1` (the line is ~100 Mbit/s; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups.
|
||||
5. Monday: deploy hub 0.168.0 (the last plain tokens locked, R-975); then release the upload watch on your yes (R-1244; D41, D42 first). If not: a hub copy still holds the tokens, and a hacked app on a family's box stays unseen.
|
||||
5. Today (2026-10-24, attended): hub 0.168.0 deployed; D40–D42 ruled and put into effect. Still open: the agent vouch, hub 0.169.0, the controller floor (R-1244). If not: a hacked app stays unseen.
|
||||
**The rest**
|
||||
- System poster needs a refresh: the new 01:40 „release copy” arrow from DooPlex to ep0 (D33); the hub-database label already says every 6 hours. If not: the drawing misses one copy.
|
||||
- Pull the failed disk Z6A07P2G out of DooPlex (R-934). If not: its alarms return on 11 November. Since 2026-10-12.
|
||||
|
||||
@@ -565,6 +565,16 @@ a `config_version` bump — the box's config is byte-for-byte the same, so no bo
|
||||
(`TestR975_DBFileWithoutKeyRevealsNoToken`). `-unseal-box-secrets` opens these leaves too, so the roll-back above also
|
||||
covers a return to a hub before v0.168.0. The full list of secret-bearing columns, measured on a copy of the live database
|
||||
2026-10-23: `audits/hub-token-seal-2026-10-23/README.md`.
|
||||
**Deployed 2026-10-24 — and what the live file still holds (operator ruling 2026-10-24, accepted).** On the live hub the
|
||||
start-up `VACUUM` did not finish: the Longhorn disk writes ~1.3 MB/s (the 6-hourly `VACUUM INTO` of 530 MB takes ~7 min)
|
||||
and the liveness probe stopped the first start after ~90 s; the second start found nothing left to seal (R-1249). So the
|
||||
rows are sealed, but the live `hub.db` and its `-wal` still carry the old token bytes in freed space (8 of 8, measured).
|
||||
**Every copy that leaves the volume is clean**: the hub's snapshot and the ep0 copy are written by `VACUUM INTO` (the ep0
|
||||
copy of 06:30Z: 0 of 8), and the volume is out of DooPlex's Longhorn backup (D40). Accepted, not cleaned: whoever can read
|
||||
the live volume is root on DooPlex and can read `OFFSITE_SECRET_KEY` there too; a full wipe would also need a hub stop of
|
||||
8–20 minutes, a disk trim and the volume's Longhorn snapshots deleted. **Rule: a hand copy of `hub.db` for analysis is
|
||||
made with `VACUUM INTO` (or taken from ep0), never `cp`/`tar` of the live file** — a `tar` of the file carries the old
|
||||
bytes (`audits/line-test-2026-10-24/hub-0168-seal-check.txt`).
|
||||
**And the key is the other half:** a backup of the database restores a hub that can open the sealed columns only
|
||||
with the same `OFFSITE_SECRET_KEY`; today that key exists only on DooPlex (the k8s Secret, and the GPG secrets export
|
||||
on the same machine). The off-site plan for the database and its key: `runbooks/RUNBOOK-hub-db-offsite-backup.md`
|
||||
|
||||
@@ -297,6 +297,11 @@ production endpoint exists.
|
||||
|
||||
## 5. The endpoint VM (the one new managed-infra piece)
|
||||
|
||||
**AMENDED 2026-10-24 (R-1247, operator ruling):** a fourth, small thing — `felhom-iperf3.service`, an iperf3 server bound to
|
||||
`10.77.0.1:5201` and opened in nftables for `wg0` from `10.77.0.0/24` only, so each box's agent can measure its upload
|
||||
line once a week (felhom.eu `scripts/ep0-iperf3/`; agent `internal/linetest`). It never answers on the internet (proven
|
||||
2026-10-24, `audits/line-test-2026-10-24/`).
|
||||
|
||||
Runs three things, nothing else: the WG server (peer list = hub-written), PBS (one datastore,
|
||||
per-customer namespaces), and the minimal peer-sync surface the hub drives. **How** the hub
|
||||
mutates the peer list — a tiny authenticated push API on the VM vs the VM pulling a signed peer
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -544,7 +544,13 @@ normal use; work it does not explain is the sign of a hacked app. Numbers and me
|
||||
1. **The line.** Learnt from what the box already does: while its off-site push runs, the controller samples its own
|
||||
sent bytes once a second; the best ≥ 10-second rate is kept per night (14 nights, settings.json `upwatch.line`); a
|
||||
night counts when that window moved ≥ 10 MB; the line is the highest counting night. A **lower bound**, shown so
|
||||
(„legalább N Mbit/s", system settings). An active test is not built (R-1247).
|
||||
(„legalább N Mbit/s", system settings). **AMENDED 2026-10-24 (operator ruling 07:26, R-1247): the weekly line TEST.**
|
||||
The night's one-second read showed how low that floor is (demo-hp „at least 10 Mbit/s"; spike §2.1). So the host's
|
||||
agent (agent ≥ 0.164.0, `internal/linetest`) runs a 10-second iperf3 upload once a week to ep0's tunnel-only server:
|
||||
Tuesday 10:00–15:59 local from a per-host slot, only after 10 quiet minutes of the physical uplink (< 2 Mbit/s each
|
||||
minute), under the one-heavy-op gate, best of 3, never in the night chain; „server busy" → later the same day. The
|
||||
controller reads it (`GET /host/line-test`) every 15 minutes; the watch judges against the higher of test and floor;
|
||||
the system page says „N Mbit/s (heti sebességméréssel)". demo-hp measured 90.7 Mbit/s (`audits/line-test-2026-10-24/`).
|
||||
2. **Trigger, per app** (its compose project's containers summed; exact counters from the Docker Engine API, once a
|
||||
minute): upload ≥ max(5 Mbit/s, 25 % of the line) for 15 minutes; CPU ≥ max(150 %, 50 % × cores) of one core for 30
|
||||
minutes. Infrastructure (the controller — its own push is the line measure —, traefik, cloudflared, Samba) is never
|
||||
@@ -556,12 +562,14 @@ normal use; work it does not explain is the sign of a hacked app. Numbers and me
|
||||
4. **Who is told** (the R-388 rule, §8.1): **`app_unexplained_upload` / `app_unexplained_cpu`** (warning) and their
|
||||
`_cleared` (info, recorded) are **operator-only** (hub `operatorOnlyEvents`), one per episode, per-app cooldown. The
|
||||
household is told only about the qBittorrent limit, because they can change it (R-1246 asks the operator).
|
||||
5. **qBittorrent's default upload limit:** 50 % of the line (R-1245), every 15 minutes, re-set when the line moves
|
||||
> 20 %; never over a limit the household set (one there first, or any change from the value qBittorrent stored for
|
||||
the box — it keeps KiB/s). No line → none set. While the box owns it, qBittorrent's page says: „A qBittorrent
|
||||
5. **qBittorrent's default upload limit:** 50 % of the TESTED line (D41 = 50 %, operator 2026-10-24; never of the
|
||||
push floor), every 15 minutes, re-set when the line moves > 20 %; never over a limit the household set (one there
|
||||
first, or any change from the value qBittorrent stored for the box — it keeps KiB/s). No test → none set. **Known
|
||||
gap (R-1250):** a qBittorrent that seeds all day keeps the uplink from ever being quiet for 10 minutes, so that box is
|
||||
never tested and gets no limit — the case the limit exists for. While the box owns it, qBittorrent's page says: „A qBittorrent
|
||||
feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod."
|
||||
6. **Limits, stated:** an attacker who knows the check can make the app's API lie — a per-container ceiling (`tc` from
|
||||
a helper in the app's network namespace, measured to work in an unprivileged guest; spike §5) would cap the flood
|
||||
whatever the API says; not built. The box cannot tell LAN from internet bytes per app; a host-network container
|
||||
(Samba) has no counter; the line is a lower bound.
|
||||
(Samba) has no counter; the push floor is a lower bound, the weekly test is close to the line.
|
||||
7. **Release:** not released; the operator's yes (R-1244).
|
||||
|
||||
@@ -97,7 +97,7 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
|
||||
## Zone 2 — the internet
|
||||
|
||||
- **Cloudflare** (grey): DNS for the customer domains and for felhom.eu; the **tunnels**; the **geo-block** (only Hungarian visitors); flood protection. The hub holds a Cloudflare key per customer zone and checks that it reaches only that customer's own domain.
|
||||
- **ep0** (grey box with an orange stripe, a Hetzner cloud server in Germany): runs the **WireGuard endpoint** and the **off-site backup server (PBS)**. Inside, one datastore with **namespaces**:
|
||||
- **ep0** (grey box with an orange stripe, a Hetzner cloud server in Germany): runs the **WireGuard endpoint**, the **off-site backup server (PBS)** and, since 2026-10-24, a small **line-speed test server** (iperf3) that answers only inside the tunnel — each box measures its upload once a week (R-1247). Inside, one datastore with **namespaces**:
|
||||
- one namespace **per household** (whole-guest copies, each household's own key),
|
||||
with a NEUTRAL note under it (not a red gap box): "By design: each household holds the key to its own copies; the operator cannot open them without their recovery code." — *fix (c), 2026-10-09: this is a property of the design, not a gap, and a red box said the opposite of what it means.*
|
||||
- the **`operator`** namespace: two groups — **"hub database"** and **"DooPlex copy"** (code, password manager, secrets, signing keys).
|
||||
|
||||
@@ -116,7 +116,7 @@ code{font-family: ui-monospace, "SFMono-Regular", Menlo, monospace; font-size: 1
|
||||
<div style="position: absolute; left: 1552px; top: 460px; width: 136px; height: 112px; box-sizing: border-box; padding: 8px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #5b6270; border-radius: 6px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>iso.felhom.eu</b><br>households download the installer</span></div>
|
||||
<div style="position: absolute; left: 1552px; top: 580px; width: 136px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: center;">web pages</div>
|
||||
<div style="position: absolute; left: 1272px; top: 738px; width: 400px; font-size: 17px; line-height: 26px; font-weight: 400; color: #051343; text-align: left;"><b style="font-size: 22px">ep0</b> — Hetzner cloud server, Germany</div>
|
||||
<div style="position: absolute; left: 1272px; top: 766px; width: 400px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;">WireGuard endpoint + off-site backup server (PBS). One datastore, split into namespaces:</div>
|
||||
<div style="position: absolute; left: 1272px; top: 766px; width: 400px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;">WireGuard endpoint + off-site backup server (PBS) + a tunnel-only line-speed test (iperf3). One datastore, split into namespaces:</div>
|
||||
<div style="position: absolute; left: 1272px; top: 818px; width: 192px; height: 150px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #0a3d73; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><span style="display: inline-block; min-width: 26px; padding: 0 6px; box-sizing: border-box; height: 26px; border-radius: 13px; background: #0a3d73; color: #ffffff; font-weight: 800; text-align: center; line-height: 26px; font-size: 16px; vertical-align: middle">G</span> <b>One namespace per household</b><br>whole-guest copies, each household's own key<br><b>keeps one per week, 2 weeks</b></span></div>
|
||||
<div style="position: absolute; left: 1480px; top: 818px; width: 192px; height: 150px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b><code style="font-size: 16px">operator</code> namespace</b><br>groups “hub database” and “DooPlex copy”<br><b>keeps 14 daily + 8 weekly</b></span></div>
|
||||
<div style="position: absolute; left: 1272px; top: 974px; width: 400px; font-size: 15px; line-height: 19px; font-weight: 400; color: #4a5670; text-align: left;"><b>By design:</b> each household holds the key to its own copies; the operator cannot open them without the household’s recovery code.</div>
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
# The attended release run, 2026-10-24 — evidence
|
||||
|
||||
Brief: „the hub's tokens locked, the hub out of the open backup, an iperf3 line test via ep0, then the upload watch
|
||||
released" (operator rulings 07:22 / 07:26, `CONTEXT.md`). Read first: `05` §16.2, `06` §4.5/§5, `07` (hub row),
|
||||
`08` §9, `audits/SPIKE-upload-watch-2026-10-23.md`.
|
||||
|
||||
| File | What it shows |
|
||||
|---|---|
|
||||
| `hub-0168-seal-check.txt` | Hub 0.168.0 deploy: the first start killed in the VACUUM, the token search (counts only) on the pre-deploy ep0 copy (8 of 8), the live file (8 of 8), the post-deploy ep0 copy (0 of 8); demo-hp's served config equal to what it runs |
|
||||
| `ep0-install.txt` | ep0: iperf3 package, the unit's caps, the listener on 10.77.0.1:5201 only, the nftables diff, the reachability proof (internet: no answer, `:22` control open; tunnel: open), the protocol proof (90.7 Mbit/s; busy in 0.1 s) |
|
||||
|
||||
Longhorn (D40): the hub PVC and volume read `recurring-job-group.longhorn.io/default: disabled`; the NFS folder holds no
|
||||
copy of `pvc-486c9809…` (no directory, no backup config naming it); Longhorn lists no backup volume for it; 53 other
|
||||
volumes were backed up this morning. The dated check 2026-10-25 reads tonight's job (R-1243).
|
||||
@@ -0,0 +1,45 @@
|
||||
# ep0 iperf3 install, 2026-10-24 ~06:12Z (CC, attended release run; operator ruling 07:26)
|
||||
Sat Oct 24 06:25:34 AM UTC 2026
|
||||
ii iperf3 3.18-2+deb13u2 amd64 Internet Protocol bandwidth measuring tool
|
||||
disabled
|
||||
enabled
|
||||
active
|
||||
RuntimeMaxUSec=1d
|
||||
CPUQuotaPerSecUSec=500ms
|
||||
MemoryMax=134217728
|
||||
TasksMax=16
|
||||
User=felhom-iperf3
|
||||
LISTEN 0 4096 10.77.0.1:5201 0.0.0.0:* users:(("iperf3",pid=3011910,fd=3))
|
||||
table inet filter {
|
||||
chain input { # handle 1
|
||||
type filter hook input priority filter; policy drop;
|
||||
ct state established,related accept # handle 3
|
||||
iif "lo" accept # handle 4
|
||||
tcp dport 22 accept # handle 5
|
||||
udp dport 443 accept # handle 6
|
||||
tcp dport 8007 iifname "wg0" accept # handle 7
|
||||
tcp dport 5201 iifname "wg0" ip saddr 10.77.0.0/24 accept # handle 13
|
||||
ip protocol icmp accept # handle 8
|
||||
meta l4proto ipv6-icmp accept # handle 9
|
||||
}
|
||||
}
|
||||
10a11
|
||||
> tcp dport 5201 iifname "wg0" ip saddr 10.77.0.0/24 accept # R-1247: felhom-iperf3 line test, tunnel only
|
||||
|
||||
# Reachability, 2026-10-24 ~06:20Z (bash /dev/tcp connect, 5-8 s timeout)
|
||||
From DooPlex over the internet (its public address 37.191.56.x) to 167.233.158.164:
|
||||
22 open (positive control: the method sees an open port)
|
||||
5201 no answer
|
||||
8007 no answer
|
||||
From demo-hp's host (10.77.0.3) through wg-felhom to 10.77.0.1:
|
||||
5201 open
|
||||
8007 open (PBS, the existing tunnel service)
|
||||
route: 10.77.0.1 dev wg-felhom src 10.77.0.3
|
||||
|
||||
# Protocol proof, 2026-10-24 ~06:31Z: a throwaway probe built from felhom-agent internal/linetest (Client, 4 streams,
|
||||
# 10 s), copied to demo-hp's host /tmp, run against 10.77.0.1:5201 through wg-felhom, then deleted.
|
||||
took 10.4s err=<nil> mbps=90.7 server_bytes=114032640 client_bytes=116046000 server_s=10.05
|
||||
# Two at once (the second started 2 s later):
|
||||
took 0.1s err=iperf3 server busy mbps=0.0 server_bytes=0 client_bytes=0 server_s=0.00
|
||||
took 10.4s err=<nil> mbps=90.6 server_bytes=113901568 client_bytes=116629300 server_s=10.06
|
||||
# After: 0 ltprobe processes; /tmp/ltprobe removed.
|
||||
@@ -0,0 +1,18 @@
|
||||
# Hub 0.168.0 deploy, 2026-10-24 (attended release run). Counts only — no value was printed.
|
||||
05:38:10Z first start: three seal lines, then no "config tokens" line; killed by the liveness probe at 05:39:39Z (exit 143)
|
||||
05:39:39Z second start: "config tokens sealed at rest (0 legacy plaintext token(s) sealed now)"; Listening 05:40:18Z
|
||||
-> the UPDATE committed in the first start; the VACUUM after it did not finish (538 MB on Longhorn; 2.8 s on local disk)
|
||||
|
||||
Search of the 8 plain token values (read from the pre-deploy ep0 copy 2026-10-24T00:3xZ) as raw bytes:
|
||||
pre-deploy ep0 copy hub.db 8 of 8 (positive control)
|
||||
live hub.db (copied 05:41Z) 8 of 8 integrity ok, freelist 312 pages -> the plaintext sits in free space
|
||||
live hub.db-wal 8 of 8
|
||||
live copy after a local VACUUM 0 of 8 (2.8 s)
|
||||
post-deploy ep0 copy 06:30:57Z 0 of 8 integrity ok; 4 customer rows carry enc:v1: tokens
|
||||
/data/snapshots: hub-20261023T180000Z.db, hub-20261024T000000Z.db are pre-deploy (plain); they roll off.
|
||||
|
||||
Box side (demo-hp, 05:53Z): GET /api/v1/box-config with the box's own key: HTTP 200, 0 "enc:v1:" in the served config;
|
||||
infrastructure.cf_tunnel_token and cf_api_token fingerprints EQUAL to the box's applied controller.yaml;
|
||||
hub log "Config downloaded for customer demo-hp (by box key, 2ms)"; cloudflared Up 14 hours (healthy).
|
||||
Gitea: hub log "Template fetched (5881 bytes)" at start.
|
||||
All copies shredded after the check.
|
||||
@@ -26,6 +26,14 @@
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-24 — the hub's tokens deployed; D40–D42 ruled; the line test (full text of the rows: `git show 3363ffa8:documentation/backlog/OPEN-ITEMS.md`)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|---|---|---|---|
|
||||
| **R-975** | **The hub database kept every customer's tunnel token, Cloudflare API token and git token unsealed.** (P3) | CLOSED 2026-10-24 — hub **0.168.0** deployed on the operator's yes (image sha256:3bfdecaa…; ArgoCD sync of the hub Deployment only, `00031150`): the three `config_json` leaves are sealed; demo-hp's `GET /box-config` serves the same plain tokens its box runs (fingerprints equal, 0 `enc:v1:`); the ep0 copy of 06:30:57Z holds **0 of 8** token values (positive control: the pre-deploy copy, 8 of 8). The live file's old bytes in freed space (the start-up VACUUM did not finish, R-1249) are **accepted by the operator** (2026-10-24: only root on DooPlex can read the volume, and it holds the key; every copy that leaves is clean; rule in `05` §16.2) | `audits/line-test-2026-10-24/hub-0168-seal-check.txt` |
|
||||
| **R-1245** | **DECIDE: how large is qBittorrent's default upload limit?** (P3) | CLOSED 2026-10-24 — **D41 = 50 %** (operator ruling 07:22), of the host's TESTED line (the weekly iperf3 test, R-1247 — never the push floor); built in controller 0.342.0 (`ed1011b`); released with R-1244 | `08` §9.5 |
|
||||
| **R-1246** | **DECIDE: who is told when the box limits or flags something?** (P3) | CLOSED 2026-10-24 — **D42** (operator ruling 07:22): the family only about the qBittorrent limit (one line on its page); `app_unexplained_upload` / `app_unexplained_cpu` are operator-only (hub 0.169.0); released with R-1244 | `08` §9.4 |
|
||||
|
||||
## 2026-10-23 day (later) — D39 = B: the security-fix lane
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -5,6 +5,16 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
|
||||
|
||||
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
|
||||
|
||||
## Moved from STATUS on 2026-10-24 (day)
|
||||
|
||||
**Updated 2026-10-23 (night).** The hub's last unlocked tokens are now locked in a new hub version — built and tested, you deploy it on Monday. The new upload watch is built and tested on the scratch box, not released: it finds an app that uploads or computes far more than the app itself explains (a hacked app) and tells only you; qBittorrent gets an upload limit.
|
||||
|
||||
**Decided 2026-10-24 07:22 (the operator started the attended release brief; the rulings are in R-1243, R-1245 and R-1246 and in CONTEXT.md):**
|
||||
|
||||
| D42 | R-1246 | When the box limits or flags something, who is told? | **The family only about the qBittorrent limit (they can change it); the hacked-app alarms only you** (the family can do nothing about a hacked app) | tell the family too: a worrying mail they cannot act on | the same as my pick | — |
|
||||
| D41 | R-1245 | How much of the measured upload may qBittorrent use by default? | **50 %** (the family keeps half the line; seeding took the whole line on the test bench) | 30 %: slower seeding; none: only the alarm | 50 % when you release the watch | — |
|
||||
| D40 | R-1243 | The hub's backups in DooPlex's shared folder are not encrypted (two there now hold the tunnel tokens). Encrypt the hub's disk, or stop that backup? | **Stop it** (the hub's own copy to ep0 is encrypted, every 6 h, tested weekly) | encrypt: 5–10 min hub downtime and one more key to keep | the plain copies stay readable by any hacked DooPlex app | a homelab change |
|
||||
|
||||
## Moved from STATUS on 2026-10-23 (night)
|
||||
|
||||
**Updated 2026-10-23 (evening).** The Plex attack on DooPlex did NOT reach Felhom's secrets: the two open doors to them (a shared backup folder and Longhorn's page) were not used in the attack days. The backup folder is now closed to the home network (your yes); it is still open to every app on DooPlex, so a hacked app there could read the hub's backups (R-1241, the homelab session's fix).
|
||||
|
||||
@@ -10,6 +10,11 @@
|
||||
`TestUpwatchEvents_OperatorOnlyWithAReadableLine`, `TestUpwatchEventTypesAreAllowedAndOperatorOnly` — red-proved
|
||||
(dropped from operatorOnlyEvents → the household was mailed; the render call removed → no Watch line; dropped from
|
||||
the allow-list → fails).
|
||||
- **2026-10-24 (attended release run): the host's weekly line test reaches the hub.** Agent 0.164.0 sends a top-level
|
||||
`line_test` stanza (R-1247); `sysfacts.LineTest` mirrors it field by field (the wire-contract gate checks both ways,
|
||||
`SUBTREE_MIRRORS`), and the System page shows one cell per host, „Upload line": the best of the last successful
|
||||
weekly test and its age, amber when never measured after an attempt or older than 15 days — judged on `last_ok_at`,
|
||||
never on the attempt time. Tests `TestParse_LineTest`, `TestLineTestCell` — red-proved.
|
||||
|
||||
## v0.168.0 (DEPLOYED 2026-10-24 on the operator's yes — built 2026-10-23 night) — the last plain tokens sealed (R-975, R-1241)
|
||||
|
||||
|
||||
@@ -98,6 +98,21 @@ type GuestTrim struct {
|
||||
Error string `json:"error"`
|
||||
}
|
||||
|
||||
// LineTest is the agent's weekly upload line test stanza (R-1247, agent ≥ 0.164.0): the host report's TOP-LEVEL
|
||||
// `line_test` object (agent internal/hub LineTestStatus) — a 10-second iperf3 upload to ep0 inside the tunnel, best of
|
||||
// the runs, as ep0's server counted it. `last_ok_at` + `mbps` are the last SUCCESSFUL measurement; `last_attempt_at` +
|
||||
// `last_error` the last attempt, whatever its verdict. A field-by-field mirror, checked both ways by the wire-contract
|
||||
// gate (SUBTREE_MIRRORS).
|
||||
type LineTest struct {
|
||||
Schedule string `json:"schedule"`
|
||||
LastOKAt string `json:"last_ok_at"` // RFC3339, "" = never
|
||||
Mbps float64 `json:"mbps"`
|
||||
RunsMbps []float64 `json:"runs_mbps"`
|
||||
Server string `json:"server"`
|
||||
LastAttemptAt string `json:"last_attempt_at"` // RFC3339
|
||||
LastError string `json:"last_error"`
|
||||
}
|
||||
|
||||
// System is the whole stanza. Present is false for a report from an agent older than v0.142.0.
|
||||
type System struct {
|
||||
Present bool
|
||||
@@ -112,10 +127,13 @@ type System struct {
|
||||
// DiskTrim is the top-level `guest_disk_trim` stanza (R-444). Independent of the `system` stanza: it is read even
|
||||
// when Present is false. nil = the agent sent none (an agent with no trimmer).
|
||||
DiskTrim *DiskTrim
|
||||
// LineTest is the top-level `line_test` stanza (R-1247), read like DiskTrim. nil = the agent sent none.
|
||||
LineTest *LineTest
|
||||
}
|
||||
|
||||
type wire struct {
|
||||
DiskTrim *DiskTrim `json:"guest_disk_trim"`
|
||||
LineTest *LineTest `json:"line_test"`
|
||||
System *struct {
|
||||
PVEVersion string `json:"pve_version"`
|
||||
KernelVersion string `json:"kernel_version"`
|
||||
@@ -143,7 +161,7 @@ func Parse(reportJSON string) System {
|
||||
if json.Unmarshal([]byte(reportJSON), &w) != nil {
|
||||
return out
|
||||
}
|
||||
out.DiskTrim = w.DiskTrim
|
||||
out.DiskTrim, out.LineTest = w.DiskTrim, w.LineTest
|
||||
if w.System == nil {
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -70,3 +70,17 @@ func TestParse_DiskTrim(t *testing.T) {
|
||||
t.Errorf("with a system stanza the trim must still be read: %+v", s)
|
||||
}
|
||||
}
|
||||
|
||||
// R-1247: the top-level `line_test` stanza is read on its own, and an absent one stays nil. The fixture is the agent's
|
||||
// TestStateSurvivesARestartAndReportsIt output shape.
|
||||
// RED-PROOF: drop `out.LineTest = w.LineTest` → nil → fails.
|
||||
func TestParse_LineTest(t *testing.T) {
|
||||
s := Parse(`{"host":{},"line_test":{"schedule":"weekly, due Tuesday","last_ok_at":"2026-10-24T06:31:00Z","mbps":90.7,"runs_mbps":[88,90.7],"server":"10.77.0.1:5201","last_attempt_at":"2026-10-24T06:31:00Z"}}`)
|
||||
lt := s.LineTest
|
||||
if lt == nil || lt.Mbps != 90.7 || len(lt.RunsMbps) != 2 || lt.Server != "10.77.0.1:5201" || lt.LastOKAt != "2026-10-24T06:31:00Z" {
|
||||
t.Fatalf("line_test misread: %+v", lt)
|
||||
}
|
||||
if s = Parse(full); s.LineTest != nil {
|
||||
t.Errorf("a report with no line_test must read nil, got %+v", s.LineTest)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -43,6 +43,7 @@ type systemRow struct {
|
||||
// guest
|
||||
GuestDebian, GuestRelease, GuestPending, GuestRestart cell
|
||||
Trim cell // R-444: the last `pct fstrim` of the guest
|
||||
Line cell // R-1247: the host's weekly upload line test
|
||||
// docker
|
||||
Engine, Containerd, LiveRestore, DockerRelease cell
|
||||
// last leg
|
||||
@@ -168,6 +169,36 @@ func buildFloorRows(ovs []store.CustomerFloorOverride, global string, now time.T
|
||||
|
||||
func plain(s string) cell { return cell{Text: s} }
|
||||
|
||||
// lineStaleAfter is when the last SUCCESSFUL line test reads amber: the hosts test weekly (R-1247), so two missed weeks
|
||||
// are worth a look (a seeding qBittorrent can keep a host from ever being quiet, R-1250).
|
||||
const lineStaleAfter = 15 * 24 * time.Hour
|
||||
|
||||
// lineTestCell is the "Upload line" cell (R-1247): the best of the last successful weekly test and how old it is, judged
|
||||
// on `last_ok_at` — never on the attempt time. Amber when never measured after an attempt, or older than 15 days. "—"
|
||||
// when the agent sends no stanza (an agent before 0.164.0, or no tunnel).
|
||||
func lineTestCell(lt *sysfacts.LineTest, now time.Time) cell {
|
||||
if lt == nil {
|
||||
return cell{Text: "—", Title: "the agent reports no line test (agent before 0.164.0, or no offsite tunnel)"}
|
||||
}
|
||||
title := "schedule: " + lt.Schedule
|
||||
if lt.LastError != "" {
|
||||
title = "last attempt " + lt.LastAttemptAt + ": " + lt.LastError + " — " + title
|
||||
}
|
||||
okAt, err := time.Parse(time.RFC3339, lt.LastOKAt)
|
||||
if lt.LastOKAt == "" || err != nil || lt.Mbps <= 0 {
|
||||
if lt.LastAttemptAt == "" {
|
||||
return cell{Text: "not yet", Title: title}
|
||||
}
|
||||
return cell{Text: "not measured", Class: "warn", Title: title}
|
||||
}
|
||||
out := cell{Text: fmt.Sprintf("%.1f Mbit/s · %s", lt.Mbps, ago(okAt, now)), Title: title}
|
||||
if now.Sub(okAt) > lineStaleAfter {
|
||||
out.Class = "warn"
|
||||
out.Title = "the last successful test is older than 15 days (the hosts test weekly) — " + out.Title
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// trimStaleAfter is when a guest's last SUCCESSFUL disk trim reads amber: the boxes trim weekly (`09` §3 decision
|
||||
// 139), so two missed weeks are worth a look.
|
||||
const trimStaleAfter = 14 * 24 * time.Hour
|
||||
@@ -371,6 +402,7 @@ func buildSystemRows(lines []osupdates.FleetLine, facts map[string]sysfacts.Syst
|
||||
r.GuestPending = plain(fmt.Sprint(l.Guest.Pending))
|
||||
r.GuestRestart = plain(fmt.Sprint(l.Guest.RestartNeeded))
|
||||
r.Trim = trimCell(f.DiskTrim, now)
|
||||
r.Line = lineTestCell(f.LineTest, now)
|
||||
r.Engine, r.Containerd = unknownCell(f.Guest.DockerEngine), unknownCell(f.Guest.Containerd)
|
||||
r.LiveRestore = unknownCell(f.Guest.LiveRestore)
|
||||
if f.Guest.LiveRestore == "off" {
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/sysfacts"
|
||||
)
|
||||
|
||||
// R-1247: the operator's "Upload line" cell is judged on the last SUCCESSFUL test, never on the attempt time — an
|
||||
// attempt is not a result.
|
||||
// RED-PROOF: judge staleness on last_attempt_at → the 20-day-old success with a fresh failed attempt reads green → fails.
|
||||
func TestLineTestCell(t *testing.T) {
|
||||
now := time.Date(2026, 10, 24, 12, 0, 0, 0, time.UTC)
|
||||
cases := []struct {
|
||||
name string
|
||||
lt *sysfacts.LineTest
|
||||
text string
|
||||
class string
|
||||
}{
|
||||
{"no stanza", nil, "—", ""},
|
||||
{"never tried", &sysfacts.LineTest{Schedule: "weekly"}, "not yet", ""},
|
||||
{"tried, never measured", &sysfacts.LineTest{Schedule: "weekly", LastAttemptAt: "2026-10-24T10:00:00Z", LastError: "iperf3 server busy"}, "not measured", "warn"},
|
||||
{"fresh", &sysfacts.LineTest{Schedule: "weekly", LastOKAt: "2026-10-24T06:31:00Z", Mbps: 90.7}, "90.7 Mbit/s", ""},
|
||||
{"old success, fresh failed attempt", &sysfacts.LineTest{Schedule: "weekly", LastOKAt: "2026-10-04T06:31:00Z", Mbps: 90.7,
|
||||
LastAttemptAt: "2026-10-24T10:00:00Z", LastError: "connect: timeout"}, "90.7 Mbit/s", "warn"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
got := lineTestCell(c.lt, now)
|
||||
if !strings.HasPrefix(got.Text, c.text) || got.Class != c.class {
|
||||
t.Errorf("%s: cell = %+v, want text %q class %q", c.name, got, c.text, c.class)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -206,6 +206,7 @@
|
||||
<dt>Pending</dt>{{template "sys_dd" .GuestPending}}
|
||||
<dt>Restart needed</dt>{{template "sys_dd" .GuestRestart}}
|
||||
<dt title="The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.">Last disk trim</dt>{{template "sys_dd" .Trim}}
|
||||
<dt title="The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.">Upload line</dt>{{template "sys_dd" .Line}}
|
||||
</dl>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -1,3 +1,12 @@
|
||||
## 2026-10-24 — `ep0-iperf3/`: the tunnel-only line-speed test server on ep0 (R-1247; operator ruling 2026-10-24 07:26)
|
||||
|
||||
- **`ep0-iperf3/felhom-iperf3.service`** (new, installed by hand on ep0 2026-10-24): Debian's `iperf3 --server` bound to
|
||||
`10.77.0.1:5201` as a DynamicUser with systemd hardening and caps (MemoryMax 128M, TasksMax 16, CPUQuota 50 %, a daily
|
||||
restart); one nftables line admits TCP from `wg0` `10.77.0.0/24` only. Debian's own `iperf3.service` stays disabled.
|
||||
`README.md`: install, the firewall line, the proof, the two open no-DSA CVEs of 3.18 and how they are bounded, removal.
|
||||
Proven: no answer from the internet (`:22` as the control), an answer through the tunnel from demo-hp's host, and a
|
||||
10-second test of 90.7 Mbit/s (`documentation/audits/line-test-2026-10-24/`).
|
||||
|
||||
## 2026-10-23 — `release-copy/`: the released images and agent packages leave DooPlex nightly (D33, R-1227); `hub-db-backup/`: every 6 hours (D34)
|
||||
|
||||
- **`release-copy/felhom-release-copy`** (new, DooPlex root, timer 01:40): the newest 3 release tags of felhom-controller, felhom-hub and felhom-samba (`skopeo copy --all --preserve-digests` into `dir:` — the registry's own bytes, so a push back gives the SAME digest; a `docker save` tar would not) and the newest 5 agent versions (binary + config bundle). Before anything leaves: the top manifest hashes to the registry's digest, every blob and child manifest to its name, every named blob is present, each agent file matches the registry's sha256; the staged tree ≤ the cap (3072 MB). Then `rsync --delete` to ep0's limited user `felhom-relcopy` (rrsync, one directory), and a `--checksum --dry-run` must show no difference before the success signal. Spoiled tags (0.315.0, 0.316.3) are never copied. 13 tests (`test_release_copy.py`), four red-proved.
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
# ep0 — the line-speed test server (R-1247)
|
||||
|
||||
Operator ruling 2026-10-24 07:26: a box's agent runs a weekly 10-second upload test to an iperf3 server on ep0,
|
||||
**inside the WireGuard tunnel only**. Installed 2026-10-24 (evidence `documentation/audits/line-test-2026-10-24/`).
|
||||
|
||||
- Package: Debian `iperf3` (3.18-2+deb13u2 at install; it was already present since 2026-07-03). Debian's own
|
||||
`iperf3.service` stays **disabled**.
|
||||
- Unit: `felhom-iperf3.service` (this folder) → `/etc/systemd/system/`, `systemctl enable --now felhom-iperf3`.
|
||||
Listens on `10.77.0.1:5201` only, as a DynamicUser, with memory/task/CPU caps.
|
||||
- Firewall: one line in `/etc/nftables.conf`'s `input` chain, after the PBS line, and the same rule live:
|
||||
|
||||
tcp dport 5201 iifname "wg0" ip saddr 10.77.0.0/24 accept
|
||||
|
||||
Live: `nft add rule inet filter input position <handle of the 8007 rule> tcp dport 5201 iifname "wg0" ip saddr 10.77.0.0/24 accept`.
|
||||
- Proof that it never answers on the internet: a TCP connect from any public address to `167.233.158.164:5201` gets
|
||||
no answer (with `:22` as the control); from a box host through the tunnel it answers.
|
||||
- Known server issues of 3.18 (no Debian advisory, „minor"): CVE-2026-71217 (resource exhaustion by oversized
|
||||
parameters) and CVE-2026-102253 (a UDP worker spins). Bounded by: tunnel peers only, TCP only, the unit's caps, a daily
|
||||
restart. Re-check when Debian ships 3.22.
|
||||
- Remove: `systemctl disable --now felhom-iperf3; rm /etc/systemd/system/felhom-iperf3.service`; delete the nft line
|
||||
(live `nft delete rule inet filter input handle <n>`, and in the file). The backup of the file before the change is
|
||||
`/etc/nftables.conf.bak-20261024-iperf3`.
|
||||
@@ -0,0 +1,49 @@
|
||||
# The line-speed test server on ep0 (R-1247; operator ruling 2026-10-24 07:26). A box's agent runs a 10-second
|
||||
# upload test against it once a week, inside the WireGuard tunnel. Installed by hand on ep0 as
|
||||
# /etc/systemd/system/felhom-iperf3.service; the firewall line that opens it is in README.md beside this file.
|
||||
#
|
||||
# Bound to the tunnel address only, TCP only (the nftables line admits TCP from wg0 10.77.0.0/24). Debian's own
|
||||
# iperf3.service stays disabled. The caps bound the two open no-DSA server issues of iperf3 3.18 (CVE-2026-71217:
|
||||
# oversized parameters exhaust memory and threads; CVE-2026-102253: a UDP worker spins — UDP never reaches it):
|
||||
# MemoryMax/TasksMax/CPUQuota keep PBS safe, RuntimeMaxSec restarts a stuck server once a day.
|
||||
[Unit]
|
||||
Description=Felhom: iperf3 line-speed test server, tunnel only (R-1247)
|
||||
Documentation=https://gitea.dooplex.hu/admin/felhom.eu/src/branch/main/scripts/ep0-iperf3/README.md
|
||||
After=wg-quick@wg0.service
|
||||
Wants=wg-quick@wg0.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/bin/iperf3 --server --bind 10.77.0.1 --port 5201 --interval 0 --idle-timeout 120 --rcv-timeout 15000
|
||||
SuccessExitStatus=1
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
RuntimeMaxSec=1d
|
||||
DynamicUser=yes
|
||||
NoNewPrivileges=yes
|
||||
CapabilityBoundingSet=
|
||||
AmbientCapabilities=
|
||||
ProtectSystem=strict
|
||||
ProtectHome=yes
|
||||
PrivateTmp=yes
|
||||
PrivateDevices=yes
|
||||
ProtectKernelTunables=yes
|
||||
ProtectKernelModules=yes
|
||||
ProtectKernelLogs=yes
|
||||
ProtectControlGroups=yes
|
||||
ProtectClock=yes
|
||||
ProtectHostname=yes
|
||||
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
|
||||
RestrictNamespaces=yes
|
||||
RestrictRealtime=yes
|
||||
RestrictSUIDSGID=yes
|
||||
LockPersonality=yes
|
||||
MemoryDenyWriteExecute=yes
|
||||
SystemCallArchitectures=native
|
||||
SystemCallFilter=@system-service
|
||||
MemoryMax=128M
|
||||
TasksMax=16
|
||||
CPUQuota=50%
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -135,6 +135,7 @@ MIRRORS = {
|
||||
# built first, against a provisional shape — and is neither a pass nor a conviction of the subtree.
|
||||
SUBTREE_MIRRORS = {
|
||||
("agent -> hub (POST /host-report)", "guest_disk_trim"): ("internal/sysfacts", "DiskTrim"),
|
||||
("agent -> hub (POST /host-report)", "line_test"): ("internal/sysfacts", "LineTest"),
|
||||
}
|
||||
|
||||
# Tag names whose literal string carries no information in a repo-wide search. NOT CHECKED.
|
||||
|
||||
Reference in New Issue
Block a user